[PATCH bpf-next 00/12] bpf: Support 16-byte return values in the R0:R2 register pair

[PATCH bpf-next 00/12] bpf: Support 16-byte return values in the R0:R2 register pair

[Yonghong Song]

Until now a BPF subprogram or kfunc could only return a scalar value of up to 8
bytes, passed back in R0. LLVM 23 added the ability to return a value in
two registers for the BPF target [1][2]: an __int128, or a struct/union
whose size is greater than 8 and up to 16 bytes, is returned in the
R0:R2 register pair, with the low 64 bits in R0 and the high 64 bits in
R2. (Before LLVM 23 the BPF backend could not return such values at all:
a by-value aggregate return was rejected with "aggregate returns are not
supported", and an __int128 return failed in the backend with "unable to
allocate function return #1".)

This series teaches the kernel about that convention so that a BPF-to-BPF
subprogram call and a kfunc call can return up to 16 bytes. A returned
struct or union must be composed only of scalars (recursively); the bytes
are handed back as the raw contents of R0:R2. Otherwise allowing pointer
members would launder a pointer into a scalar and escape the verifier's
provenance and reference tracking. A value larger than 16 bytes remains
unsupported, as does a <=8 byte return, which continues to use R0 alone.

Support is built up incrementally so each step is reviewable on its own,
and the feature is only turned on end to end by the last enabling patch.

  - Patch 1 refactors the global-subprog return check so the per-register
    scalar validation can be reused for R2.
  - Patch 2 to Patch 7 are preparation patches for core verifier handling
    of R0:R2 returns, JIT support, precision backtracking, liveness
    analysis, and limitation on trampoline bpf programs.
  - Patch 8 relaxes btf_distill_func_proto() and btf_validate_return_type()
    to accept <=16 byte aggregates, enabling the feature end to end.
  - Patches 9-11 add selftests including C and inline asm tests.
  - Patch 12 documents the kfunc return-value convention in kfuncs.rst.

Tested on x86 and arm64. The patch set is built on top of the following commit:
    Merge branch 'verify-bpf-signed-loader-at-load-time'

    [1] https://github.com/llvm/llvm-project/pull/190894
    [2] https://github.com/llvm/llvm-project/pull/206876

Yonghong Song (12):
  bpf: Factor check_global_ret_scalar_reg() out of the global return
    check
  bpf: Add verifier support for 16-byte returns in R0:R2
  bpf: Wire up JIT support for 16-byte kfunc returns
  bpf: Track R2 of register-pair returns in precision backtracking
  bpf: Account R2 of register-pair returns in live register analysis
  bpf: Force JIT for programs using the R0:R2 register pair
  bpf: Reject >8 byte return values on return-reading trampoline paths
  bpf: Enable 16-byte aggregate return types
  selftests/bpf: Add C tests for 16-byte returns in R0:R2
  selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
  selftests/bpf: Cover tracing on >8 and <=16 byte return targets
  Documentation/bpf: Document 16-byte kfunc return values in R0:R2

 Documentation/bpf/kfuncs.rst                  |  36 ++
 arch/arm64/net/bpf_jit_comp.c                 |   5 +
 arch/riscv/net/bpf_jit_comp64.c               |   5 +
 arch/x86/net/bpf_jit_comp.c                   |  19 +
 include/linux/bpf.h                           |   1 +
 include/linux/bpf_verifier.h                  |   4 +
 include/linux/filter.h                        |   1 +
 kernel/bpf/backtrack.c                        |  32 +-
 kernel/bpf/bpf_struct_ops.c                   |  12 +
 kernel/bpf/btf.c                              |  14 +-
 kernel/bpf/core.c                             |   7 +-
 kernel/bpf/liveness.c                         |  13 +-
 kernel/bpf/verifier.c                         | 179 ++++++++-
 .../selftests/bpf/prog_tests/aggregate_ret.c  | 249 ++++++++++++
 .../selftests/bpf/progs/aggregate_ret_func.c  | 361 ++++++++++++++++++
 .../bpf/progs/aggregate_ret_int128_c.c        |  48 +++
 .../selftests/bpf/progs/aggregate_ret_kfunc.c |  81 ++++
 .../selftests/bpf/progs/aggregate_ret_run.c   | 160 ++++++++
 .../bpf/progs/aggregate_ret_struct_c.c        |  98 +++++
 .../selftests/bpf/progs/aggregate_ret_trace.c |  44 +++
 .../bpf/progs/aggregate_ret_union_c.c         |  51 +++
 .../selftests/bpf/progs/exceptions_fail.c     |   2 +-
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  80 ++++
 .../selftests/bpf/test_kmods/bpf_testmod.h    |   5 +
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  38 ++
 25 files changed, 1517 insertions(+), 28 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_run.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_trace.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c

-- 
2.53.0-Meta

[PATCH bpf-next 01/12] bpf: Factor check_global_ret_scalar_reg() out of the global return check

[Yonghong Song]

Function check_global_subprog_return_code() verifies that a global subprogram
returns void, an arena pointer, or register R0 which holds a scalar value.
Later patches in this series add 16-byte aggregate return support, whose
second half is returned in R2 and needs similar validation.

Factor the per-register check into check_global_ret_scalar_reg(env, regno) so
it can be reused for R2.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/verifier.c | 32 ++++++++++++++++++++------------
 1 file changed, 20 insertions(+), 12 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 001ac53825da..3f34ce1a3107 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -16679,37 +16679,45 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 	return 0;
 }
 
-static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
 {
-	struct bpf_reg_state *reg = reg_state(env, BPF_REG_0);
-	struct bpf_func_state *cur_frame = cur_func(env);
+	struct bpf_reg_state *reg;
 	int err;
 
-	if (subprog_returns_void(env, cur_frame->subprogno))
-		return 0;
-
-	err = check_reg_arg(env, BPF_REG_0, SRC_OP);
+	err = check_reg_arg(env, regno, SRC_OP);
 	if (err)
 		return err;
 
 	/* Pointers to arena are safe to pass between subprograms. */
-	if (is_arena_reg(env, BPF_REG_0))
+	if (is_arena_reg(env, regno))
 		return 0;
 
-	if (is_pointer_value(env, BPF_REG_0)) {
-		verbose(env, "R%d leaks addr as return value\n", BPF_REG_0);
+	if (is_pointer_value(env, regno)) {
+		verbose(env, "R%d leaks addr as return value\n", regno);
 		return -EACCES;
 	}
 
+	reg = reg_state(env, regno);
 	if (reg->type != SCALAR_VALUE) {
-		verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n",
-			reg_type_str(env, reg->type));
+		verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n",
+			regno, reg_type_str(env, reg->type));
 		return -EINVAL;
 	}
 
 	return 0;
 }
 
+static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+{
+	struct bpf_func_state *cur_frame = cur_func(env);
+	u32 subprog = cur_frame->subprogno;
+
+	if (subprog_returns_void(env, subprog))
+		return 0;
+
+	return check_global_ret_scalar_reg(env, BPF_REG_0);
+}
+
 /* Bitmask with 1s for all caller saved registers */
 #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
 
-- 
2.53.0-Meta

[PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2

[Yonghong Song]

LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more
than 16 bytes. See LLVM patches [1] and [2].

Before LLVM 23 the BPF backend could not return these values at all. A
by-value struct or union return (of any size) was rejected at compile
time with:

  error: aggregate returns are not supported

and an __int128 return failed later in the backend with:

  fatal error: error in backend: unable to allocate function return #1

Both are resolved in LLVM 23, which lowers such returns into the R0:R2
register pair.

This patch is one of several preparation patches that build up
<=16 byte return support incrementally:
  - the verifier R0:R2 modelling in this patch,
  - the x86 JIT move and JIT-capability gate,
  - precision-backtracking and live-register tracking of R2,
  - forcing the JIT for such programs, and
  - guarding the trampoline paths.
Finally, the patch "bpf: Enable 16-byte aggregate return types" enables
support for 16-byte returns.

This patch adds handling for '>8' byte returns in several places: BPF
subprogram returns (the main program, and both global and static
subprograms) and kfunc returns.

  [1] https://github.com/llvm/llvm-project/pull/190894
  [2] https://github.com/llvm/llvm-project/pull/206876

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/verifier.c | 93 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 88 insertions(+), 5 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 3f34ce1a3107..03ffb5f839fa 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
 	return btf_type_is_void(type);
 }
 
+static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+	const struct btf_type *type, *func, *func_proto;
+	const struct btf *btf = env->prog->aux->btf;
+	u32 btf_id;
+
+	if (!btf || !env->prog->aux->func_info)
+		return false;
+
+	btf_id = env->prog->aux->func_info[subprog].type_id;
+
+	func = btf_type_by_id(btf, btf_id);
+	if (!func)
+		return false;
+
+	func_proto = btf_type_by_id(btf, func->type);
+	if (!func_proto)
+		return false;
+
+	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
+	if (!type)
+		return false;
+
+	if (btf_type_is_struct(type) || btf_type_is_scalar(type))
+		return type->size > 8 && type->size <= 16;
+
+	return false;
+}
+
 static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
 {
 	struct bpf_func_info *info;
@@ -9459,10 +9488,17 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		clear_caller_saved_regs(env, caller->regs);
 		invalidate_outgoing_stack_args(env, cur_func(env));
 
-		/* All non-void global functions return a 64-bit SCALAR_VALUE. */
+		/*
+		 * A non-void global function returns a 64-bit SCALAR_VALUE in
+		 * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
+		 */
 		if (!subprog_returns_void(env, subprog)) {
 			mark_reg_unknown(env, caller->regs, BPF_REG_0);
 			caller->regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG;
+			if (bpf_ret_reg_pair(env, subprog)) {
+				mark_reg_unknown(env, caller->regs, BPF_REG_2);
+				caller->regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
+			}
 		}
 
 		if (env->subprog_info[subprog].might_throw) {
@@ -9825,6 +9861,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	} else {
 		/* return to the caller whatever r0 had in the callee */
 		caller->regs[BPF_REG_0] = *r0;
+		if (bpf_ret_reg_pair(env, callee->subprogno)) {
+			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
+				verbose(env, "cannot return stack pointer to the caller\n");
+				return -EINVAL;
+			}
+			caller->regs[BPF_REG_2] = callee->regs[BPF_REG_2];
+		}
 	}
 
 	/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
@@ -10745,6 +10788,18 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno,
 	return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size);
 }
 
+static void mark_kfunc_ret_reg_size(struct bpf_verifier_env *env,
+				    struct bpf_reg_state *regs, u32 size)
+{
+	if (size > 8) {
+		mark_btf_func_reg_size(env, BPF_REG_0, 8);
+		mark_reg_unknown(env, regs, BPF_REG_2);
+		regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
+	} else {
+		mark_btf_func_reg_size(env, BPF_REG_0, size);
+	}
+}
+
 static bool is_kfunc_acquire(struct bpf_kfunc_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ACQUIRE;
@@ -13208,7 +13263,22 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
 		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
 			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
-		mark_btf_func_reg_size(env, BPF_REG_0, t->size);
+		mark_kfunc_ret_reg_size(env, regs, t->size);
+	} else if (btf_type_is_struct(t)) {
+		/*
+		 * The returned struct comes back as raw register bits modeled
+		 * as an unknown scalar, so it must contain only scalars:
+		 * otherwise a pointer field would be laundered into a scalar
+		 * and escape provenance and reference tracking.
+		 */
+		if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+			verbose(env, "kernel function %s returns %s %s that is not composed of scalars\n",
+				func_name, btf_type_str(t),
+				btf_name_by_offset(desc_btf, t->name_off));
+			return -EINVAL;
+		}
+		mark_reg_unknown(env, regs, BPF_REG_0);
+		mark_kfunc_ret_reg_size(env, regs, t->size);
 	} else if (btf_type_is_ptr(t)) {
 		ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
 		err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
@@ -16711,11 +16781,19 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
 {
 	struct bpf_func_state *cur_frame = cur_func(env);
 	u32 subprog = cur_frame->subprogno;
+	int err;
 
 	if (subprog_returns_void(env, subprog))
 		return 0;
 
-	return check_global_ret_scalar_reg(env, BPF_REG_0);
+	err = check_global_ret_scalar_reg(env, BPF_REG_0);
+	if (err)
+		return err;
+
+	if (!bpf_ret_reg_pair(env, subprog))
+		return 0;
+
+	return check_global_ret_scalar_reg(env, BPF_REG_2);
 }
 
 /* Bitmask with 1s for all caller saved registers */
@@ -17205,10 +17283,15 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
 	 */
 	if (cur_frame->subprogno &&
 	    !cur_frame->in_async_callback_fn &&
-	    !cur_frame->in_exception_callback_fn)
+	    !cur_frame->in_exception_callback_fn) {
 		err = check_global_subprog_return_code(env);
-	else
+	} else {
+		if (!cur_frame->subprogno && bpf_ret_reg_pair(env, 0)) {
+			verbose(env, "return value larger than 8 bytes is not supported at program exit\n");
+			return -EINVAL;
+		}
 		err = check_return_code(env, BPF_REG_0, "R0");
+	}
 	if (err)
 		return err;
 	return PROCESS_BPF_EXIT;
-- 
2.53.0-Meta

[PATCH bpf-next 03/12] bpf: Wire up JIT support for 16-byte kfunc returns

[Yonghong Song]

Continue preparing <=16 byte aggregate return support by wiring up the JIT
side of the R0:R2 return convention for kfuncs.

A kfunc returning more than 8 bytes (a small aggregate or an __int128)
hands the second half of the result back in RDX, the native x86-64 ABI's
second return register. BPF instead expects a 16-byte return in the
R0:R2 register pair. BPF R0 maps to RAX so it needs no move, but BPF R2
maps to RSI, so emit a RDX->RSI move after a BPF_PSEUDO_KFUNC_CALL whose
function model reports ret_size > 8.

Placing the second return half into R2 is possible on any JIT, but it
needs architecture-specific JIT work. Rather than requiring every JIT to
implement it at once, add a bpf_jit_supports_kfunc_ret_reg_pair()
capability, defaulting to false in the generic core; an architecture opts
in once its JIT handles the R0:R2 pair, and the remaining ones are left
for future work. The verifier enforces it in bpf_add_kfunc_call(),
rejecting a kfunc whose return is larger than 8 bytes with -EOPNOTSUPP
when the JIT lacks the capability. Only x86, arm64 and riscv are
supported so far.

On arm64 and riscv the native second return register is already BPF R2
(x1 in bpf2a64[] and a1 in regmap[] respectively), so the value is in the
R0:R2 register pair on return with no extra move, unlike x86 (RDX->RSI).
This has been tested on x86 and arm64. The riscv path is expected to work
by the same register-mapping reasoning as arm64 but has not been tested.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 arch/arm64/net/bpf_jit_comp.c   |  5 +++++
 arch/riscv/net/bpf_jit_comp64.c |  5 +++++
 arch/x86/net/bpf_jit_comp.c     | 19 +++++++++++++++++++
 include/linux/filter.h          |  1 +
 kernel/bpf/core.c               |  5 +++++
 kernel/bpf/verifier.c           |  5 +++++
 6 files changed, 40 insertions(+)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index b0075ece4a6e..a4292e9ec075 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2312,6 +2312,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index f9d5347ba966..68faa7016d30 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -2107,6 +2107,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_ptr_xchg(void)
 {
 	return true;
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index de7515ea1bea..0c685c93df15 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2592,6 +2592,20 @@ st:			insn_off = insn->off;
 				return -EINVAL;
 			if (priv_frame_ptr)
 				pop_r9(&prog);
+			if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
+				const struct btf_func_model *fm;
+
+				/*
+				 * A kfunc returning a >8 byte aggregate hands the
+				 * second half back in RDX (the native ABI's second
+				 * return reg), but BPF expects it in R0:R2. BPF R0
+				 * is RAX (no move needed), while BPF R2 is RSI, so
+				 * copy RDX into RSI.
+				 */
+				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
+				if (fm && fm->ret_size > 8)
+					emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
+			}
 			break;
 		}
 
@@ -4045,6 +4059,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 14acb2455746..308a0e2d5988 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1182,6 +1182,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_kfunc_ret_reg_pair(void);
 bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 6e19a030da6f..73dc3ee879de 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3294,6 +3294,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_stack_args(void)
 {
 	return false;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 03ffb5f839fa..e3f934cf702d 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2851,6 +2851,11 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 	err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
 	if (err)
 		return err;
+	if (func_model.ret_size > 8 && !bpf_jit_supports_kfunc_ret_reg_pair()) {
+		verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
+			kfunc.name);
+		return -EOPNOTSUPP;
+	}
 
 	desc = &tab->descs[tab->nr_descs++];
 	desc->func_id = func_id;
-- 
2.53.0-Meta

[PATCH bpf-next 04/12] bpf: Track R2 of register-pair returns in precision backtracking

[Yonghong Song]

A function returning a value larger than 8 bytes (a struct/union, or an
__int128) uses R2 as a second return register alongside R0. Previous
precision backtracking only treated R0 as a return register at a
call/return boundary, so marking the second half of such a return precise
would trip the "unexpected regs" checks in backtrack_insn() and reject a
valid program with -EFAULT.

Handle R2 like R0 in the three boundaries where a call defines the return
registers:

 - static subprog exit (BPF_EXIT): R2 is a return register, not a leftover
   argument; clear it before the args check and propagate its precision
   into the callee.
 - global subprog call: a global subprog returning >8 bytes also sets R2;
   clear it before the args check.
 - kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8)
   also sets R2; clear it like R0.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 include/linux/bpf_verifier.h |  2 ++
 kernel/bpf/backtrack.c       | 32 +++++++++++++++++++++++++++++++-
 kernel/bpf/verifier.c        | 13 ++++++++++++-
 3 files changed, 45 insertions(+), 2 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 317e99b9acc0..fa255f326240 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1431,6 +1431,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);
 struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
+bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog);
+int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 2e4ae0ef0860..c46c2778bf60 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -424,6 +424,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				 */
 				verifier_bug_if(idx + 1 != subseq_idx, env,
 						"extra insn from subprog");
+				if (bpf_ret_reg_pair(env, subprog))
+					bt_clear_reg(bt, BPF_REG_2);
 				/* r1-r5 are invalidated after subprog call,
 				 * so for global func call it shouldn't be set
 				 * anymore
@@ -507,6 +509,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				return -ENOTSUPP;
 			/* regular helper call sets R0 */
 			bt_clear_reg(bt, BPF_REG_0);
+			/* On error the return is negative, R2 stays set, and the
+			 * check below catches it as an unexpected register.
+			 */
+			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+			    bpf_get_kfunc_ret_size(env->prog, insn->imm, insn->off) > 8)
+				bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				/* if backtracking was looking for registers R1-R5
 				 * they should have been found already.
@@ -521,7 +529,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 					return -EFAULT;
 			}
 		} else if (opcode == BPF_EXIT) {
-			bool r0_precise;
+			bool r0_precise, r2_precise = false;
+
+			/* A subprog returning a value larger than 8 bytes (a
+			 * struct/union or an __int128) uses R2 as a second
+			 * return register alongside R0. When exiting such a
+			 * subprog, R2 is a return register rather than a
+			 * leftover argument, so handle it like R0 below.
+			 */
+			if (subseq_idx - 1 >= 0 &&
+			    bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1])) {
+				int call_idx = subseq_idx - 1;
+				int tgt = call_idx + env->prog->insnsi[call_idx].imm + 1;
+				int subprog = bpf_find_subprog(env, tgt);
+
+				if (subprog >= 0 &&
+				    bpf_ret_reg_pair(env, subprog) &&
+				    bt_is_reg_set(bt, BPF_REG_2))
+					r2_precise = true;
+			}
 
 			/* Backtracking to a nested function call, 'idx' is a part of
 			 * the inner frame 'subseq_idx' is a part of the outer frame.
@@ -534,6 +560,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 			if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
 				for (i = BPF_REG_1; i <= BPF_REG_5; i++)
 					bt_clear_reg(bt, i);
+			if (r2_precise)
+				bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				verifier_bug(env, "backtracking exit unexpected regs %x",
 					     bt_reg_mask(bt));
@@ -558,6 +586,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 
 			if (r0_precise)
 				bt_set_reg(bt, BPF_REG_0);
+			if (r2_precise)
+				bt_set_reg(bt, BPF_REG_2);
 			/* r6-r9 and stack slots will stay set in caller frame
 			 * bitmasks until we return back from callee(s)
 			 */
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e3f934cf702d..fcbf532159a5 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -424,7 +424,7 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
 	return btf_type_is_void(type);
 }
 
-static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
 {
 	const struct btf_type *type, *func, *func_proto;
 	const struct btf *btf = env->prog->aux->btf;
@@ -2521,6 +2521,17 @@ int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
 	return 0;
 }
 
+int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset)
+{
+	const struct bpf_kfunc_desc *desc;
+
+	desc = find_kfunc_desc(prog, func_id, offset);
+	if (!desc)
+		return -EFAULT;
+
+	return desc->func_model.ret_size;
+}
+
 #define BPF_FD_SLOT_BTF	1UL
 
 static void fd_slot_set_map(struct bpf_fd_array *slot, struct bpf_map *map)
-- 
2.53.0-Meta

[PATCH bpf-next 05/12] bpf: Account R2 of register-pair returns in live register analysis

[Yonghong Song]

A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a
struct/union or an __int128) reads R2 as well as R0, since the second half
of the return value is passed back in R2. compute_insn_live_regs() only
marked R0 used at exit, so the callee's R2 could be considered dead and
cleaned from checkpointed states, which would allow unsound state pruning.

Mark R2 used at a BPF_EXIT when the enclosing subprogram returns via the
R0:R2 register pair. The call site already treats the caller-saved
registers R0-R5 as clobbered, so R2 as a result register is covered there.

For example, for a subprogram returning an __int128, the live-registers
dump (log level 2) shows both R0 and R2 live before the callee's exit
(insn 5):

  Live regs before insn:
        0: .12345.... (85) call pc+2
        1: ..2....... (bf) r0 = r2
        2: 0......... (95) exit
        3: .......... (b7) r0 = 1
        4: 0......... (b7) r2 = 2
        5: 0.2....... (95) exit

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/liveness.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 0aadfbae0acc..0977d32fcf01 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2062,7 +2062,8 @@ struct insn_live_regs {
 /* Compute info->{use,def} fields for the instruction */
 static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				   struct bpf_insn *insn,
-				   struct insn_live_regs *info)
+				   struct insn_live_regs *info,
+				   int subprog)
 {
 	struct bpf_call_summary cs;
 	u8 class = BPF_CLASS(insn->code);
@@ -2175,6 +2176,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 		case BPF_EXIT:
 			def = 0;
 			use = r0;
+			if (bpf_ret_reg_pair(env, subprog))
+				use |= BIT(BPF_REG_2);
 			break;
 		case BPF_CALL:
 			def = ALL_CALLER_SAVED_REGS;
@@ -2237,8 +2240,12 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		goto out;
 	}
 
-	for (i = 0; i < insn_cnt; ++i)
-		compute_insn_live_regs(env, &insns[i], &state[i]);
+	for (i = 0, j = 0; i < insn_cnt; ++i) {
+		/* Advance to the subprog that contains instruction i. */
+		while (j + 1 < env->subprog_cnt && env->subprog_info[j + 1].start <= i)
+			j++;
+		compute_insn_live_regs(env, &insns[i], &state[i], j);
+	}
 
 	/* Forward pass: resolve stack access through FP-derived pointers */
 	err = bpf_compute_subprog_arg_access(env);
-- 
2.53.0-Meta

[PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

[Yonghong Song]

The R0:R2 return convention introduced in this series is only implemented
in the JIT. The BPF interpreter has no notion of a second return register,
so a program relying on the R0:R2 convention would compute wrong results
under the interpreter. Force the JIT when bpf_prog_aux->ret_reg_pair is
set.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 include/linux/bpf.h   |  1 +
 kernel/bpf/core.c     |  2 +-
 kernel/bpf/verifier.c | 13 +++++++++++--
 3 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index c1a98fa36738..10966aba69a2 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1780,6 +1780,7 @@ struct bpf_prog_aux {
 	bool changes_pkt_data;
 	bool might_sleep;
 	bool kprobe_write_ctx;
+	bool ret_reg_pair;
 	struct {
 		s32 keyring_serial;
 		u8 keyring_type;
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 73dc3ee879de..819a97d25173 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
 		goto finalize;
 
 	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
-	    bpf_prog_has_kfunc_call(fp))
+	    bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)
 		jit_needed = true;
 
 	if (!bpf_prog_select_interpreter(fp))
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index fcbf532159a5..ca8b2f436c20 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
 	return false;
 }
 
+static bool bpf_ret_reg_pair_mark(struct bpf_verifier_env *env, int subprog)
+{
+	if (!bpf_ret_reg_pair(env, subprog))
+		return false;
+
+	env->prog->aux->ret_reg_pair = true;
+	return true;
+}
+
 static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
 {
 	struct bpf_func_info *info;
@@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	} else {
 		/* return to the caller whatever r0 had in the callee */
 		caller->regs[BPF_REG_0] = *r0;
-		if (bpf_ret_reg_pair(env, callee->subprogno)) {
+		if (bpf_ret_reg_pair_mark(env, callee->subprogno)) {
 			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
 				verbose(env, "cannot return stack pointer to the caller\n");
 				return -EINVAL;
@@ -16806,7 +16815,7 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
 	if (err)
 		return err;
 
-	if (!bpf_ret_reg_pair(env, subprog))
+	if (!bpf_ret_reg_pair_mark(env, subprog))
 		return 0;
 
 	return check_global_ret_scalar_reg(env, BPF_REG_2);
-- 
2.53.0-Meta

[PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

[Yonghong Song]

btf_distill_func_proto() builds the function model used for kfunc calls,
the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following
patch relaxes it to accept a >8 and <=16 byte return value, which for the
BPF target is passed back in the R0:R2 register pair.

But the BPF trampoline cannot preserve such a return. It saves and restores
only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half
(RDX / R2) is not preserved across a program that observes the return
value. A program attached to a function returning a 16-byte value (e.g.
current_time() or ns_to_timespec64(), which return struct timespec64)
would corrupt the value seen by the real caller and read a partial return
value itself. struct_ops trampolines have the same limitation.

This only matters for the attach types that actually read the target's
return value: fexit, fmod_ret and fsession (plus the _multi variants of
fexit and fsession). fentry and fentry_multi run before the target returns
and never touch the return value, so they can attach to a >8 byte-returning
function safely.

Reject a >8 byte return value for the return-reading attach types in
bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for
struct_ops in bpf_struct_ops_desc_init(), ahead of the following patch that
would otherwise let such a return through. kfunc and BPF-to-BPF subprogram
calls, which the JIT does handle, are unaffected.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/bpf_struct_ops.c | 12 ++++++++++++
 kernel/bpf/verifier.c       | 25 +++++++++++++++++++++++++
 2 files changed, 37 insertions(+)

diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c
index 51b16e5f5534..bbb44ef8f87a 100644
--- a/kernel/bpf/bpf_struct_ops.c
+++ b/kernel/bpf/bpf_struct_ops.c
@@ -445,6 +445,18 @@ int bpf_struct_ops_desc_init(struct bpf_struct_ops_desc *st_ops_desc,
 			goto errout;
 		}
 
+		/*
+		 * A >8 byte return value is passed back in the R0:R2 register
+		 * pair, which the struct_ops trampoline does not preserve (only
+		 * 8 bytes of the return value are saved and restored).
+		 */
+		if (st_ops->func_models[i].ret_size > 8) {
+			pr_warn("func ptr %s in struct %s has a >8 byte return value, which is not supported\n",
+				mname, st_ops->name);
+			err = -EOPNOTSUPP;
+			goto errout;
+		}
+
 		stub_func_addr = *(void **)(st_ops->cfi_stubs + moff);
 		err = prepare_arg_info(btf, st_ops->name, mname,
 				       func_proto, stub_func_addr,
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index ca8b2f436c20..121f56e3fb16 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -19112,6 +19112,20 @@ static int btf_id_allow_sleepable(u32 btf_id, unsigned long addr, const struct b
 	return -EINVAL;
 }
 
+static bool attach_uses_trampoline_retval(enum bpf_attach_type type)
+{
+	switch (type) {
+	case BPF_MODIFY_RETURN:
+	case BPF_TRACE_FEXIT:
+	case BPF_TRACE_FEXIT_MULTI:
+	case BPF_TRACE_FSESSION:
+	case BPF_TRACE_FSESSION_MULTI:
+		return true;
+	default:
+		return false;
+	}
+}
+
 int bpf_check_attach_target(struct bpf_verifier_log *log,
 			    const struct bpf_prog *prog,
 			    const struct bpf_prog *tgt_prog,
@@ -19376,6 +19390,14 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 		if (ret < 0)
 			return ret;
 
+		if (tgt_info->fmodel.ret_size > 8 &&
+		    attach_uses_trampoline_retval(prog->expected_attach_type)) {
+			bpf_log(log,
+				"Attach to function %s with a >8 byte return value is not supported for this attach type\n",
+				tname);
+			return -EOPNOTSUPP;
+		}
+
 		/*
 		 * *.multi programs don't need an address during program
 		 * verification, we just take the module ref if needed.
@@ -19652,6 +19674,9 @@ int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 bt
 	err = btf_distill_func_proto(NULL, btf, t, tname, &tgt_info->fmodel);
 	if (err < 0)
 		return err;
+	if (tgt_info->fmodel.ret_size > 8 &&
+	    attach_uses_trampoline_retval(prog->expected_attach_type))
+		return -EOPNOTSUPP;
 	if (btf_is_module(btf)) {
 		/* The bpf program already holds reference to module. */
 		if (WARN_ON_ONCE(!prog->aux->mod))
-- 
2.53.0-Meta

[PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types

[Yonghong Song]

Relax btf_distill_func_proto() to accept a by-value struct or union that
the R0:R2 convention added in earlier patches can carry:

 - a struct or union larger than 8 and up to 16 bytes, returned in the
   R0:R2 register pair, matching what LLVM emits for the BPF target;
 - a struct or union up to 8 bytes, returned in R0 alone.

A >8 byte scalar (__int128) was already accepted and is unchanged.
Everything else stays rejected: a return type larger than 16 bytes, and any
type that __get_type_size() cannot return in registers at all (e.g. an
array), which it already reports as ret < 0.

btf_distill_func_proto() also builds the trampoline (fentry/fexit/fmod_ret)
and struct_ops function models. Allowing up to 16 bytes here is safe: the
previous patch already rejects >8 byte returns on the return-reading paths,
where the trampoline preserves only 8 bytes of the return value.

btf_validate_return_type() is relaxed as it accepts a by-value struct or
union up to 16 bytes in addition to void and scalars.

With btf_distill_func_proto() and btf_validate_return_type() relaxed, the
verifier, JIT, precision-backtracking, live-register and trampoline-guard
preparation from the earlier patches becomes reachable: this final patch
enables <=16 byte aggregate return values end to end.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 include/linux/bpf_verifier.h                       |  2 ++
 kernel/bpf/btf.c                                   | 14 +++++++++++---
 kernel/bpf/verifier.c                              |  6 +++---
 .../testing/selftests/bpf/progs/exceptions_fail.c  |  2 +-
 4 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index fa255f326240..ae4dae8e57f9 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1433,6 +1433,8 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
 bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog);
 int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset);
+bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
+				 const struct btf_type *t, int rec);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index dff5c0d91641..bab82d1c8cb9 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
 		return -EINVAL;
 	}
 	ret = __get_type_size(btf, func->type, &t);
-	if (ret < 0 || btf_type_is_struct(t)) {
+	/*
+	 * __get_type_size() already restricts a non-negative ret to void, a
+	 * pointer, an int, an enum or a struct/union, so only the size is checked
+	 * here.
+	 */
+	if (ret < 0 || ret > 16) {
 		bpf_log(log,
 			"The function %s return type %s is unsupported.\n",
 			tname, btf_type_str(t));
@@ -7968,6 +7973,9 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
 	if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
 		return 0;
 
+	if (btf_type_is_struct(t) && t->size <= 16 && __btf_type_is_scalar_struct(env, btf, t, 0))
+		return 0;
+
 	return -EOPNOTSUPP;
 }
 
@@ -8059,8 +8067,8 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 	if (err) {
 		if (is_global) {
 			bpf_log(log,
-				"Global function %s() return value not void or scalar. "
-				"Only those are supported.\n",
+				"Global function %s() has unsupported return type. "
+				"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n",
 				tname);
 		}
 		return err;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 121f56e3fb16..bf451a2d7095 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11082,9 +11082,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_kfunc_call_arg_meta *meta, u3
 }
 
 /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */
-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
-					const struct btf *btf,
-					const struct btf_type *t, int rec)
+bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
+				 const struct btf *btf,
+				 const struct btf_type *t, int rec)
 {
 	const struct btf_type *member_type;
 	const struct btf_member *member;
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index ac44d60e5066..9708efb93683 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)
 
 SEC("?tc")
 __exception_cb(exception_cb_bad_ret_type1)
-__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.")
+__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.")
 int reject_exception_cb_type_1(struct __sk_buff *ctx)
 {
 	bpf_throw(0);
-- 
2.53.0-Meta

[PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

[Yonghong Song]

Add selftests that exercise a 16-byte return value passed in the R0:R2
register pair, expressed in C so they rely on the compiler lowering the
register-pair return. Tests include an __int128 return (from a subprogram
and a kfunc), a 16-byte struct return (from static and global subprograms
and a kfunc) and a 16-byte union return.

The R0:R2 return convention is only emitted by LLVM 23 and newer, so the
programs are gated on __clang_major__ >= 23; with an older compiler they
return a sentinel and the "int128_c", "struct_c", "global_struct_c",
"kfunc_struct_c" and "union_c" subtests are skipped.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 .../selftests/bpf/prog_tests/aggregate_ret.c  | 96 ++++++++++++++++++
 .../bpf/progs/aggregate_ret_int128_c.c        | 48 +++++++++
 .../bpf/progs/aggregate_ret_struct_c.c        | 98 +++++++++++++++++++
 .../bpf/progs/aggregate_ret_union_c.c         | 51 ++++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    | 14 +++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        | 11 +++
 6 files changed, 318 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c

diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
new file mode 100644
index 000000000000..d4ecc19684ce
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -0,0 +1,96 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include <network_helpers.h>
+#include "aggregate_ret_int128_c.skel.h"
+#include "aggregate_ret_struct_c.skel.h"
+#include "aggregate_ret_union_c.skel.h"
+
+/*
+ * Run one program and check its result. The program returns 0 on success, or
+ * 1..4 identifying which half of which aggregate (R0:R2) return value came back
+ * wrong. A return of -1 means the program was built with an LLVM that lacks
+ * R0:R2 16-byte return support, in which case the subtest is skipped.
+ */
+static void run_prog(struct bpf_program *prog, bool may_skip)
+{
+	char buf[64] = {};
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		.data_in = buf,
+		.data_size_in = sizeof(buf),
+		.repeat = 1,
+	);
+	int err;
+
+	err = bpf_prog_test_run_opts(bpf_program__fd(prog), &topts);
+	if (!ASSERT_OK(err, "test_run"))
+		return;
+
+	if (may_skip && (int)topts.retval == -1) {
+		test__skip();
+		return;
+	}
+
+	ASSERT_EQ(topts.retval, 0, "aggregate_ret_result");
+}
+
+void test_aggregate_ret(void)
+{
+	struct aggregate_ret_struct_c *skel_struct_c;
+	struct aggregate_ret_union_c *skel_union_c;
+	struct aggregate_ret_int128_c *skel_c;
+
+	skel_c = aggregate_ret_int128_c__open_and_load();
+	if (!ASSERT_OK_PTR(skel_c, "skel_c_open_load"))
+		return;
+
+	/* C variant (__int128): relies on LLVM 23 R0:R2 codegen, skipped otherwise. */
+	if (test__start_subtest("int128_c"))
+		run_prog(skel_c->progs.aggregate_ret_test, true);
+
+	aggregate_ret_int128_c__destroy(skel_c);
+
+	skel_struct_c = aggregate_ret_struct_c__open_and_load();
+	if (!ASSERT_OK_PTR(skel_struct_c, "skel_struct_c_open_load"))
+		return;
+
+	/*
+	 * C variant returning a 16-byte struct by value: exercises the struct
+	 * branch of bpf_ret_reg_pair(). Also LLVM 23 only.
+	 */
+	if (test__start_subtest("struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_struct_c_test, true);
+
+	/*
+	 * Same 16-byte struct return, but from a global (non-static) callee:
+	 * exercises the compiler lowering a global struct-returning function to
+	 * R0:R2 and the verifier's global return path end to end. Also LLVM 23
+	 * only.
+	 */
+	if (test__start_subtest("global_struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_global_struct_c_test, true);
+
+	/*
+	 * Same 16-byte struct, but returned by value from a kfunc: exercises
+	 * the struct branch of check_kfunc_call() and the JIT RDX->R2 move.
+	 * Also LLVM 23 only.
+	 */
+	if (test__start_subtest("kfunc_struct_c"))
+		run_prog(skel_struct_c->progs.aggregate_ret_kfunc_struct_c_test, true);
+
+	aggregate_ret_struct_c__destroy(skel_struct_c);
+
+	skel_union_c = aggregate_ret_union_c__open_and_load();
+	if (!ASSERT_OK_PTR(skel_union_c, "skel_union_c_open_load"))
+		return;
+
+	/*
+	 * C variant returning a 16-byte union by value: exercises the union BTF
+	 * kind through the same struct branch of bpf_ret_reg_pair().
+	 * Also LLVM 23 only.
+	 */
+	if (test__start_subtest("union_c"))
+		run_prog(skel_union_c->progs.aggregate_ret_union_c_test, true);
+
+	aggregate_ret_union_c__destroy(skel_union_c);
+}
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
new file mode 100644
index 000000000000..0b0c4fa52b62
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
@@ -0,0 +1,48 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+#if __clang_major__ >= 23
+
+typedef unsigned __int128 u128;
+
+static __attribute__((noinline)) u128 make_i128(__u64 a, __u64 b)
+{
+	return ((u128)a << 64) | b;
+}
+
+SEC("tc")
+int aggregate_ret_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	u128 v;
+
+	v = make_i128(a, b);
+	if ((__u64)(v >> 64) != a)
+		return 1;
+	if ((__u64)v != b)
+		return 2;
+
+	v = bpf_kfunc_call_test_i128(a, b);
+	if ((__u64)(v >> 64) != a)
+		return 3;
+	if ((__u64)v != b)
+		return 4;
+
+	return 0;
+}
+
+#else
+
+SEC("tc")
+int aggregate_ret_test(struct __sk_buff *skb)
+{
+	return -1;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
new file mode 100644
index 000000000000..ad8d44755d24
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
@@ -0,0 +1,98 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+#if __clang_major__ >= 23
+
+struct pair {
+	__u64 hi;
+	__u64 lo;
+};
+
+static __attribute__((noinline)) struct pair make_pair(__u64 a, __u64 b)
+{
+	struct pair p = { .hi = a, .lo = b };
+
+	return p;
+}
+
+SEC("tc")
+int aggregate_ret_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	struct pair p;
+
+	p = make_pair(a, b);
+	if (p.hi != a)
+		return 1;
+	if (p.lo != b)
+		return 2;
+
+	return 0;
+}
+
+__attribute__((noinline)) struct pair make_pair_global(__u64 a, __u64 b)
+{
+	struct pair p = { .hi = a, .lo = b };
+
+	return p;
+}
+
+SEC("tc")
+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xcafef00dULL;
+	struct pair p;
+
+	p = make_pair_global(a, b);
+	if (p.hi != a)
+		return 1;
+	if (p.lo != b)
+		return 2;
+
+	return 0;
+}
+
+SEC("tc")
+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xfeedfaceULL;
+	struct bpf_kfunc_ret_ll p;
+
+	p = bpf_kfunc_call_test_ret_ll(a, b);
+	if (p.a != a)
+		return 1;
+	if (p.b != b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+SEC("tc")
+int aggregate_ret_struct_c_test(struct __sk_buff *skb)
+{
+	return -1;
+}
+
+SEC("tc")
+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)
+{
+	return -1;
+}
+
+SEC("tc")
+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)
+{
+	return -1;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c
new file mode 100644
index 000000000000..53e21753291b
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c
@@ -0,0 +1,51 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+#if __clang_major__ >= 23
+
+union pair {
+	__u64 halves[2];
+	struct {
+		__u64 lo;
+		__u64 hi;
+	} parts;
+};
+
+static __attribute__((noinline)) union pair make_pair(__u64 a, __u64 b)
+{
+	union pair p;
+
+	p.parts.lo = a;
+	p.parts.hi = b;
+	return p;
+}
+
+SEC("tc")
+int aggregate_ret_union_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	union pair p;
+
+	p = make_pair(a, b);
+	if (p.parts.lo != a)
+		return 1;
+	if (p.parts.hi != b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+SEC("tc")
+int aggregate_ret_union_c_test(struct __sk_buff *skb)
+{
+	return -1;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 30f1cd23093c..3511a4873d6c 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -825,6 +825,18 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
 	return 0;
 }
 
+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
+{
+	return ((__int128)a << 64) | b;
+}
+
+__bpf_kfunc struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(u64 a, u64 b)
+{
+	struct bpf_kfunc_ret_ll r = { .a = a, .b = b };
+
+	return r;
+}
+
 __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
 					 u64 e, u64 f, u64 g, u64 h,
 					 u64 i, u64 j)
@@ -1358,6 +1370,8 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ll)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index c36bb911defa..34d0a352e70d 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -55,6 +55,15 @@ struct prog_test_big_arg {
 	__u64 b;
 };
 
+/*
+ * A 16-byte struct returned by value from a kfunc, returned in the R0:R2
+ * register pair.
+ */
+struct bpf_kfunc_ret_ll {	/* 16 bytes: R0:R2 */
+	__u64 a;
+	__u64 b;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -118,6 +127,8 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
+struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(__u64 a, __u64 b) __ksym;
 __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
 			       __u64 e, __u64 f, __u64 g, __u64 h,
 			       __u64 i, __u64 j) __ksym;
-- 
2.53.0-Meta

[PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

[Yonghong Song]

Add inline-asm and subprogram tests for more coverage, including
BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive
and negative tests are added.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 .../selftests/bpf/prog_tests/aggregate_ret.c  |  25 ++
 .../selftests/bpf/progs/aggregate_ret_func.c  | 361 ++++++++++++++++++
 .../selftests/bpf/progs/aggregate_ret_kfunc.c |  81 ++++
 .../selftests/bpf/progs/aggregate_ret_run.c   | 160 ++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  34 ++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  27 ++
 6 files changed, 688 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_run.c

diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
index d4ecc19684ce..2f1e5c704959 100644
--- a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -5,6 +5,9 @@
 #include "aggregate_ret_int128_c.skel.h"
 #include "aggregate_ret_struct_c.skel.h"
 #include "aggregate_ret_union_c.skel.h"
+#include "aggregate_ret_run.skel.h"
+#include "aggregate_ret_func.skel.h"
+#include "aggregate_ret_kfunc.skel.h"
 
 /*
  * Run one program and check its result. The program returns 0 on success, or
@@ -39,6 +42,7 @@ void test_aggregate_ret(void)
 	struct aggregate_ret_struct_c *skel_struct_c;
 	struct aggregate_ret_union_c *skel_union_c;
 	struct aggregate_ret_int128_c *skel_c;
+	struct aggregate_ret_run *skel_run;
 
 	skel_c = aggregate_ret_int128_c__open_and_load();
 	if (!ASSERT_OK_PTR(skel_c, "skel_c_open_load"))
@@ -93,4 +97,25 @@ void test_aggregate_ret(void)
 		run_prog(skel_union_c->progs.aggregate_ret_union_c_test, true);
 
 	aggregate_ret_union_c__destroy(skel_union_c);
+
+	skel_run = aggregate_ret_run__open_and_load();
+	if (!ASSERT_OK_PTR(skel_run, "skel_run_open_load"))
+		return;
+
+	/* Inline-asm variant: compiler-version independent, always runs. */
+	if (test__start_subtest("asm"))
+		run_prog(skel_run->progs.aggregate_ret_asm_test, false);
+
+	/* Struct-by-value kfunc returns, also compiler-version independent. */
+	if (test__start_subtest("struct"))
+		run_prog(skel_run->progs.aggregate_ret_struct_test, false);
+
+	/* Union-by-value kfunc return, compiler-version independent. */
+	if (test__start_subtest("union"))
+		run_prog(skel_run->progs.aggregate_ret_union_test, false);
+
+	aggregate_ret_run__destroy(skel_run);
+
+	RUN_TESTS(aggregate_ret_func);
+	RUN_TESTS(aggregate_ret_kfunc);
 }
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
new file mode 100644
index 000000000000..31e176103374
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
@@ -0,0 +1,361 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+typedef unsigned __int128 u128;
+
+__naked u128 global_agg_good(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"	/* low 64 bits */
+	"r2 = 0x5678;"	/* high 64 bits */
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__success __retval(0)
+int aggregate_ret_global(void *ctx)
+{
+	__u64 lo, hi;
+
+	asm volatile (
+	"call %[global_agg_good];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: __imm(global_agg_good)
+	: "r0", "r1", "r2", "r3", "r4", "r5");
+	if (lo != 0x1234)
+		return 1;
+	if (hi != 0x5678)
+		return 2;
+	return 0;
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_global_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad)
+	: __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("At subprogram exit the register R2 is not a scalar value")
+__naked int aggregate_ret_global_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad_ptr];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __attribute__((noinline)) u128 static_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"	/* stack pointer placed in the second return register */
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("cannot return stack pointer to the caller")
+__naked int aggregate_ret_static_ptr_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_bad_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __attribute__((noinline)) u128 static_agg_no_r2(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_static_uninit_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_no_r2];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(static_agg_no_r2)
+	: __clobber_all);
+}
+
+static __naked __attribute__((noinline)) u128 static_agg_precise(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 4;"	/* second half; its value is made precise below */
+	"exit;"
+	);
+}
+
+SEC("tc")
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
+__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
+__naked int aggregate_ret_static_precise(void)
+{
+	asm volatile (
+	"call %[static_agg_precise];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_agg_precise)
+	: __clobber_all);
+}
+
+SEC("tc")
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
+__naked int aggregate_ret_global_precise(void)
+{
+	asm volatile (
+	"call %[global_agg_good];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_agg_good)
+	: __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("return value larger than 8 bytes is not supported at program exit")
+__naked u128 aggregate_ret_entry_fail(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+#if __clang_major__ >= 23
+
+struct pair {
+	__u64 hi;
+	__u64 lo;
+};
+
+union upair {
+	__u64 halves[2];
+	struct {
+		__u64 lo;
+		__u64 hi;
+	} parts;
+};
+
+/* A by-value struct that smuggles a pointer, which must be rejected. */
+struct with_ptr {
+	void *p;
+	__u64 x;
+};
+
+/* A by-value union that smuggles a pointer, which must be rejected too. */
+union upair_with_ptr {
+	void *p;
+	__u64 halves[2];
+};
+
+/* Global subprogram returning a scalar-only 16-byte struct in R0:R2. */
+__naked struct pair global_ret_struct(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"	/* struct's first half */
+	"r2 = 0x5678;"	/* struct's second half */
+	"exit;"
+	);
+}
+
+/* Global subprogram returning a scalar-only 16-byte union in R0:R2. */
+__naked union upair global_ret_union(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"
+	"r2 = 0x5678;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__success __retval(0)
+int aggregate_ret_global_struct(void *ctx)
+{
+	__u64 lo, hi;
+
+	asm volatile (
+	"call %[global_ret_struct];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: __imm(global_ret_struct)
+	: "r0", "r1", "r2", "r3", "r4", "r5");
+	if (lo != 0x1234)
+		return 1;
+	if (hi != 0x5678)
+		return 2;
+	return 0;
+}
+
+SEC("tc")
+__success __retval(0)
+int aggregate_ret_global_union(void *ctx)
+{
+	__u64 lo, hi;
+
+	asm volatile (
+	"call %[global_ret_union];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: __imm(global_ret_union)
+	: "r0", "r1", "r2", "r3", "r4", "r5");
+	if (lo != 0x1234)
+		return 1;
+	if (hi != 0x5678)
+		return 2;
+	return 0;
+}
+
+__naked struct with_ptr global_ret_struct_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_struct_ptr() has unsupported return type")
+__naked int aggregate_ret_global_struct_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_struct_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_struct_ptr)
+	: __clobber_all);
+}
+
+__naked union upair_with_ptr global_ret_union_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_union_ptr() has unsupported return type")
+__naked int aggregate_ret_global_union_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_union_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_union_ptr)
+	: __clobber_all);
+}
+
+#endif /* __clang_major__ >= 23 */
+
+static __naked u128 agg_callee(void)
+{
+	asm volatile (
+	"r0 = 1;"
+	"r2 = 2;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__log_level(2)
+__msg("Live regs before insn:")
+__msg("0: .12345.... (85) call pc+2")
+__msg("1: ..2....... (bf) r0 = r2")
+__msg("2: 0......... (95) exit")
+__msg("3: .......... (b7) r0 = 1")
+__msg("4: 0......... (b7) r2 = 2")
+__msg("5: 0.2....... (95) exit")
+__naked int aggregate_ret_live(void)
+{
+	asm volatile (
+	"call %[agg_callee];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: [agg_callee]"i"(agg_callee)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
new file mode 100644
index 000000000000..d73d30abed55
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
@@ -0,0 +1,81 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+/*
+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address
+ * (rather than calling) avoids any dependence on the compiler lowering an
+ * __int128 or struct return value, which the BPF backend only supports from
+ * LLVM 23 on.
+ */
+void __kfunc_btf_root(void)
+{
+	asm volatile (""
+	:
+	: "r"(&bpf_kfunc_call_test_i128),
+	  "r"(&bpf_kfunc_call_test_ret_ptr),
+	  "r"(&bpf_kfunc_call_test_ret_ii));
+}
+
+SEC("tc")
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
+__naked int aggregate_ret_kfunc_precise(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"r2 = 2;"
+	"call %[bpf_kfunc_call_test_i128];"
+	"r6 = r2;"		/* second return half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_i128)
+	: __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("is not composed of scalars")
+__naked int aggregate_ret_kfunc_ptr_fail(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ptr)
+	: __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_kfunc_small_no_r2(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"r2 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ii];"
+	"r0 = r2;"	/* R2 is not a return register for a <=8 byte struct */
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ii)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_run.c b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c
new file mode 100644
index 000000000000..d6300b31733d
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c
@@ -0,0 +1,160 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+typedef unsigned __int128 u128;
+
+/*
+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address
+ * (rather than calling) avoids any dependence on the compiler lowering an __int128
+ * or struct return value, which the BPF backend only supports from LLVM 23 on.
+ */
+void __kfunc_btf_root(void)
+{
+	asm volatile (""
+	:
+	: "r"(&bpf_kfunc_call_test_i128),
+	  "r"(&bpf_kfunc_call_test_ret_ll),
+	  "r"(&bpf_kfunc_call_test_ret_li),
+	  "r"(&bpf_kfunc_call_test_ret_ii),
+	  "r"(&bpf_kfunc_call_test_ret_uu));
+}
+
+#define I128_ASM_LO 0xABCDabcd12345678ULL
+#define I128_ASM_HI 0x1234567890abcdefULL
+
+static __naked __attribute__((noinline)) u128 make_i128_asm(void)
+{
+	asm volatile (
+	"r0 = %[lo] ll;"	/* low 64 bits */
+	"r2 = %[hi] ll;"	/* high 64 bits */
+	"exit;"
+	:
+	: __imm_const(lo, I128_ASM_LO), __imm_const(hi, I128_ASM_HI)
+	);
+}
+
+SEC("tc")
+int aggregate_ret_asm_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	__u64 lo, hi;
+
+	asm volatile (
+	"call %[callee];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [callee]"i"(make_i128_asm)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (lo != I128_ASM_LO)
+		return 1;
+	if (hi != I128_ASM_HI)
+		return 2;
+
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_i128)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (hi != a)
+		return 3;
+	if (lo != b)
+		return 4;
+
+	return 0;
+}
+
+SEC("tc")
+int aggregate_ret_struct_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	__u64 lo, hi;
+
+	/* struct { u64 a; u64 b; }: R0 = a, R2 = b. */
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_ll)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (lo != a)
+		return 1;
+	if (hi != b)
+		return 2;
+
+	/* struct { u64 a; int b; }: R0 = a, low 32 bits of R2 = b. */
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_li)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (lo != a)
+		return 3;
+	if ((int)hi != (int)b)
+		return 4;
+
+	/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not used. */
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	: [lo]"=r"(lo)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_ii)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if ((int)lo != (int)a)
+		return 5;
+	if ((int)(lo >> 32) != (int)b)
+		return 6;
+
+	return 0;
+}
+
+SEC("tc")
+int aggregate_ret_union_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	__u64 lo, hi;
+
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_uu)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (lo != a)
+		return 1;
+	if (hi != b)
+		return 2;
+
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 3511a4873d6c..c9e211d8a719 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -837,6 +837,36 @@ __bpf_kfunc struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(u64 a, u64 b)
 	return r;
 }
 
+__bpf_kfunc struct bpf_kfunc_ret_li bpf_kfunc_call_test_ret_li(u64 a, int b)
+{
+	struct bpf_kfunc_ret_li r = { .a = a, .b = b };
+
+	return r;
+}
+
+__bpf_kfunc struct bpf_kfunc_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)
+{
+	struct bpf_kfunc_ret_ii r = { .a = a, .b = b };
+
+	return r;
+}
+
+__bpf_kfunc union bpf_kfunc_ret_uu bpf_kfunc_call_test_ret_uu(u64 a, u64 b)
+{
+	union bpf_kfunc_ret_uu r;
+
+	r.halves[0] = a;
+	r.halves[1] = b;
+	return r;
+}
+
+__bpf_kfunc struct bpf_kfunc_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)
+{
+	struct bpf_kfunc_ret_ptr r = { .p = NULL, .tag = tag };
+
+	return r;
+}
+
 __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
 					 u64 e, u64 f, u64 g, u64 h,
 					 u64 i, u64 j)
@@ -1372,6 +1402,10 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ll)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_li)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_uu)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index 34d0a352e70d..9ac31d68e2c8 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -64,6 +64,29 @@ struct bpf_kfunc_ret_ll {	/* 16 bytes: R0:R2 */
 	__u64 b;
 };
 
+struct bpf_kfunc_ret_li {	/* 16 bytes: R0:R2 */
+	__u64 a;
+	int b;
+};
+
+struct bpf_kfunc_ret_ii {	/* 8 bytes: R0 only */
+	int a;
+	int b;
+};
+
+union bpf_kfunc_ret_uu {	/* 16 bytes: R0:R2 */
+	__u64 halves[2];
+	struct {
+		__u64 lo;
+		__u64 hi;
+	} parts;
+};
+
+struct bpf_kfunc_ret_ptr {	/* 16 bytes: contains a pointer */
+	void *p;
+	__u64 tag;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -129,6 +152,10 @@ long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
 __int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
 struct bpf_kfunc_ret_ll bpf_kfunc_call_test_ret_ll(__u64 a, __u64 b) __ksym;
+struct bpf_kfunc_ret_li bpf_kfunc_call_test_ret_li(__u64 a, int b) __ksym;
+struct bpf_kfunc_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
+union bpf_kfunc_ret_uu bpf_kfunc_call_test_ret_uu(__u64 a, __u64 b) __ksym;
+struct bpf_kfunc_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;
 __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
 			       __u64 e, __u64 f, __u64 g, __u64 h,
 			       __u64 i, __u64 j) __ksym;
-- 
2.53.0-Meta

[PATCH bpf-next 11/12] selftests/bpf: Cover tracing on >8 and <=16 byte return targets

[Yonghong Song]

Tracing attach types differ in whether they need the target's return value
preserved across the BPF trampoline, which saves and restores only 8 bytes
of it. fentry runs before the target returns and never observes the return
value, so it can attach to a function returning a >8 byte value; fexit,
fmod_ret and fsession read the return value and must keep rejecting such a
target.

Add fentry, fexit, fentry_multi and fexit_multi tests that attach to
functions returning a >8 and <=16 byte value (a 16-byte struct and an
__int128).

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 .../selftests/bpf/prog_tests/aggregate_ret.c  | 128 ++++++++++++++++++
 .../selftests/bpf/progs/aggregate_ret_trace.c |  44 ++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  32 +++++
 .../selftests/bpf/test_kmods/bpf_testmod.h    |   5 +
 4 files changed, 209 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_trace.c

diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
index 2f1e5c704959..7caf815a2c06 100644
--- a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -8,6 +8,7 @@
 #include "aggregate_ret_run.skel.h"
 #include "aggregate_ret_func.skel.h"
 #include "aggregate_ret_kfunc.skel.h"
+#include "aggregate_ret_trace.skel.h"
 
 /*
  * Run one program and check its result. The program returns 0 on success, or
@@ -37,6 +38,131 @@ static void run_prog(struct bpf_program *prog, bool may_skip)
 	ASSERT_EQ(topts.retval, 0, "aggregate_ret_result");
 }
 
+/*
+ * Multi-attach (BPF_TRACE_*_MULTI) requires CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS,
+ * which today only x86_64 selects. Otherwise bpf_tracing_multi_attach() returns
+ * -EOPNOTSUPP unconditionally.
+ */
+static bool tracing_multi_supported(struct bpf_program *prog)
+{
+	const char *small_pat = "bpf_testmod:bpf_testmod_trampoline_count_test";
+	struct bpf_link *link;
+
+	link = bpf_program__attach_tracing_multi(prog, small_pat, NULL);
+	if (libbpf_get_error(link) == -EOPNOTSUPP)
+		return false;
+	bpf_link__destroy(link);
+	return true;
+}
+
+static void test_tracing_attach_types(void)
+{
+	const char *i128_pat = "bpf_testmod:bpf_testmod_aggregate_ret_i128_fn";
+	int err;
+
+	if (test__start_subtest("trace_fentry")) {
+		struct aggregate_ret_trace *trace = NULL;
+
+		trace = aggregate_ret_trace__open();
+		if (!ASSERT_OK_PTR(trace, "trace_fentry_open"))
+			return;
+
+		bpf_program__set_autoload(trace->progs.fexit_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fentry_multi, false);
+		bpf_program__set_autoload(trace->progs.fexit_multi, false);
+
+		err = aggregate_ret_trace__load(trace);
+		if (!ASSERT_OK(err, "trace_fentry_load"))
+			goto destroy_trace_fentry;
+
+		err = aggregate_ret_trace__attach(trace);
+		if (!ASSERT_OK(err, "trace_fentry_attach"))
+			goto destroy_trace_fentry;
+
+		ASSERT_OK(trigger_module_test_read(1), "trace_fentry_trigger");
+		ASSERT_EQ(trace->bss->fentry_hit, 1, "trace_fentry_hit");
+
+destroy_trace_fentry:
+		aggregate_ret_trace__destroy(trace);
+	}
+
+	if (test__start_subtest("trace_fexit_reject")) {
+		struct aggregate_ret_trace *trace = NULL;
+
+		trace = aggregate_ret_trace__open();
+		if (!ASSERT_OK_PTR(trace, "trace_fexit_open"))
+			return;
+
+		bpf_program__set_autoload(trace->progs.fentry_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fentry_multi, false);
+		bpf_program__set_autoload(trace->progs.fexit_multi, false);
+
+		err = aggregate_ret_trace__load(trace);
+		ASSERT_EQ(err, -EOPNOTSUPP, "trace_fexit_load");
+
+		aggregate_ret_trace__destroy(trace);
+	}
+
+	if (test__start_subtest("trace_fentry_multi")) {
+		struct aggregate_ret_trace *trace = NULL;
+		struct bpf_link *link;
+
+		trace = aggregate_ret_trace__open();
+		if (!ASSERT_OK_PTR(trace, "trace_fentry_multi_open"))
+			return;
+
+		bpf_program__set_autoload(trace->progs.fentry_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fexit_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fexit_multi, false);
+
+		if (!ASSERT_OK(aggregate_ret_trace__load(trace), "trace_fentry_multi_load"))
+			goto destroy_fentry_multi;
+
+		if (!tracing_multi_supported(trace->progs.fentry_multi)) {
+			test__skip();
+			goto destroy_fentry_multi;
+		}
+
+		link = bpf_program__attach_tracing_multi(trace->progs.fentry_multi, i128_pat, NULL);
+		if (!ASSERT_OK_PTR(link, "trace_fentry_multi_attach"))
+			goto destroy_fentry_multi;
+
+		ASSERT_OK(trigger_module_test_read(1), "trace_fentry_multi_trigger");
+		ASSERT_EQ(trace->bss->fentry_multi_hit, 1, "trace_fentry_multi_hit");
+
+		bpf_link__destroy(link);
+destroy_fentry_multi:
+		aggregate_ret_trace__destroy(trace);
+	}
+
+	if (test__start_subtest("trace_fexit_multi_reject")) {
+		struct aggregate_ret_trace *trace = NULL;
+		struct bpf_link *link;
+
+		trace = aggregate_ret_trace__open();
+		if (!ASSERT_OK_PTR(trace, "trace_fexit_multi_open"))
+			return;
+
+		bpf_program__set_autoload(trace->progs.fentry_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fexit_bpf_testmod_aggregate_ret_fn, false);
+		bpf_program__set_autoload(trace->progs.fentry_multi, false);
+
+		if (!ASSERT_OK(aggregate_ret_trace__load(trace), "trace_fexit_multi_load"))
+			goto destroy_fexit_multi;
+
+		if (!tracing_multi_supported(trace->progs.fexit_multi)) {
+			test__skip();
+			goto destroy_fexit_multi;
+		}
+
+		link = bpf_program__attach_tracing_multi(trace->progs.fexit_multi, i128_pat, NULL);
+		ASSERT_EQ(libbpf_get_error(link), -EOPNOTSUPP, "trace_fexit_multi_reject");
+		bpf_link__destroy(link);
+destroy_fexit_multi:
+		aggregate_ret_trace__destroy(trace);
+	}
+}
+
 void test_aggregate_ret(void)
 {
 	struct aggregate_ret_struct_c *skel_struct_c;
@@ -118,4 +244,6 @@ void test_aggregate_ret(void)
 
 	RUN_TESTS(aggregate_ret_func);
 	RUN_TESTS(aggregate_ret_kfunc);
+
+	test_tracing_attach_types();
 }
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_trace.c b/tools/testing/selftests/bpf/progs/aggregate_ret_trace.c
new file mode 100644
index 000000000000..ff9fd4036121
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_trace.c
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+struct bpf_testmod_aggregate_ret {
+	__u64 lo;
+	__u64 hi;
+};
+
+int fentry_hit;
+int fexit_hit;
+int fentry_multi_hit;
+
+SEC("fentry/bpf_testmod:bpf_testmod_aggregate_ret_fn")
+int BPF_PROG2(fentry_bpf_testmod_aggregate_ret_fn, __u64, lo, __u64, hi)
+{
+	fentry_hit++;
+	return 0;
+}
+
+SEC("fexit/bpf_testmod:bpf_testmod_aggregate_ret_fn")
+int BPF_PROG2(fexit_bpf_testmod_aggregate_ret_fn, __u64, lo, __u64, hi,
+	      struct bpf_testmod_aggregate_ret, ret)
+{
+	fexit_hit += ret.lo == lo && ret.hi == hi;
+	return 0;
+}
+
+SEC("fentry.multi/bpf_testmod:bpf_testmod_aggregate_ret_i128_fn")
+int BPF_PROG(fentry_multi)
+{
+	fentry_multi_hit++;
+	return 0;
+}
+
+SEC("fexit.multi/bpf_testmod:bpf_testmod_aggregate_ret_i128_fn")
+int BPF_PROG(fexit_multi)
+{
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index c9e211d8a719..b558c538400e 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -469,12 +469,34 @@ noinline void bpf_testmod_stacktrace_test_1(void)
 }
 
 int bpf_testmod_fentry_ok;
+u64 bpf_testmod_aggregate_ret_seen;
 
 noinline int bpf_testmod_trampoline_count_test(void)
 {
 	return 0;
 }
 
+__noclone noinline struct bpf_testmod_aggregate_ret
+bpf_testmod_aggregate_ret_fn(u64 lo, u64 hi)
+{
+	struct bpf_testmod_aggregate_ret ret = {
+		.lo = lo,
+		.hi = hi,
+	};
+
+	bpf_testmod_aggregate_ret_seen += lo + hi;
+	return ret;
+}
+
+__noclone noinline __int128
+bpf_testmod_aggregate_ret_i128_fn(u64 lo, u64 hi)
+{
+	__int128 ret = ((__int128)hi << 64) | lo;
+
+	bpf_testmod_aggregate_ret_seen += lo + hi;
+	return ret;
+}
+
 noinline ssize_t
 bpf_testmod_test_read(struct file *file, struct kobject *kobj,
 		      const struct bin_attribute *bin_attr,
@@ -492,6 +514,8 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj,
 	struct bpf_testmod_struct_arg_5 struct_arg5 = {23, 24, 25, 26};
 	union bpf_testmod_union_arg_1 union_arg1 = { .arg = {1} };
 	union bpf_testmod_union_arg_2 union_arg2 = { .arg = {2, 3} };
+	struct bpf_testmod_aggregate_ret agg_ret;
+	__int128 agg_i128;
 	int i = 1;
 
 	while (bpf_testmod_return_ptr(i))
@@ -534,6 +558,14 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj,
 
 	trace_bpf_testmod_test_nullable_bare_tp(NULL);
 
+	agg_ret = bpf_testmod_aggregate_ret_fn(len, off);
+	if (agg_ret.lo == U64_MAX && agg_ret.hi == U64_MAX)
+		goto out;
+
+	agg_i128 = bpf_testmod_aggregate_ret_i128_fn(len, off);
+	if (agg_i128 == (__int128)-1)
+		goto out;
+
 	/* Magic number to enable writable tp */
 	if (len == 64) {
 		struct bpf_testmod_test_writable_ctx writable = {
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h
index 863fd10f1619..4dcbfd64de62 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.h
@@ -25,6 +25,11 @@ struct bpf_testmod_test_writable_ctx {
 	int val;
 };
 
+struct bpf_testmod_aggregate_ret {
+	__u64 lo;
+	__u64 hi;
+};
+
 /* BPF iter that returns *value* *n* times in a row */
 struct bpf_iter_testmod_seq {
 	s64 value;
-- 
2.53.0-Meta

[PATCH bpf-next 12/12] Documentation/bpf: Document 16-byte kfunc return values in R0:R2

[Yonghong Song]

kfuncs may now return a value larger than 8 bytes and up to 16 bytes (a
16-byte struct or union, or an __int128), passed back in the R0:R2 register
pair. Add a kfunc return-value section documenting this, including that a
struct or union up to 8 bytes is returned in R0 alone, that the R0:R2
register pair requires JIT support
(bpf_jit_supports_kfunc_ret_reg_pair()), and that a return value larger
than 16 bytes is unsupported.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 Documentation/bpf/kfuncs.rst | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index c801a330aece..be4f489a9f16 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -501,6 +501,42 @@ is also covered by this recovery. A kfunc handed an arena pointer may
 therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
 against the arena. Larger accesses must verify the range explicitly.
 
+2.9 kfunc Return Values
+-----------------------
+
+A kfunc may return a scalar, a pointer, or a small struct or union by
+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.
+
+A struct or union returned by value must be composed only of scalars
+(recursively). Its bytes are handed back to the program as the raw contents
+of R0 (and R2), so a pointer field would otherwise be laundered into a scalar
+and escape the verifier's pointer provenance and reference tracking. Such a
+struct or union is therefore rejected at load time.
+
+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a
+16-byte scalar-only struct or union, or an ``__int128``. Such a value is
+returned in the register pair R0:R2, matching the convention LLVM uses for
+the BPF target: the low 64 bits in R0 and the high 64 bits in R2. A struct
+or union of 8 bytes or less is returned in R0 alone.
+
+::
+
+        struct bpf_pair { __u64 a, b; };   /* 16 bytes */
+
+        __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)
+        {
+                struct bpf_pair p = { .a = 1, .b = 2 };
+
+                return p;      /* p.a in R0, p.b in R2 */
+        }
+
+Returning a value in the R0:R2 pair requires the JIT to place the second
+half of the return value into R2, which not every architecture supports
+right now. A kfunc with a return value larger than 8 bytes is therefore
+rejected at load time on a JIT that does not advertise this capability (see
+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run
+by the interpreter. A return value larger than 16 bytes is not supported.
+
 .. _BPF_kfunc_lifecycle_expectations:
 
 3. kfunc lifecycle expectations
-- 
2.53.0-Meta

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Unconditional use of `__int128` in `bpf_testmod` breaks compilation on 32-bit host architectures.
--

--- Patch [9]: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2 ---
commit 5ed3844739f04c920bb1c37cdf474501b3ca9fef
Author: Yonghong Song <yonghong.song@linux.dev>

selftests/bpf: Add C tests for 16-byte returns in R0:R2

This commit adds selftests to exercise 16-byte return values passed in the
R0:R2 register pair. The tests include an __int128 return, a 16-byte struct
return, and a 16-byte union return.

[ ... ]

> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c

[ ... ]

> @@ -825,6 +825,18 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
>  	return 0;
>  }
>  
> +__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
> +{
> +	return ((__int128)a << 64) | b;
> +}

[Severity: Low]
Will this unconditional use of __int128 break the build on 32-bit host
architectures? 

The __int128 type is a compiler extension that is typically only available
on 64-bit systems.

[ ... ]

> @@ -1358,6 +1370,8 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
>  BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
>  BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
>  BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
> +BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)

[Severity: Low]
If bpf_kfunc_call_test_i128 needs to be conditionally excluded for 32-bit
builds to fix the issue above, should this BTF ID registration also be
conditionally excluded to prevent a missing symbol error during the build?

[ ... ]

> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h

[ ... ]

> @@ -118,6 +127,8 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
>  struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
>  long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
>  int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
> +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;

[Severity: Low]
Does this declaration also need to be guarded for 64-bit architectures,
so that including this header doesn't fail on 32-bit host compilers?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260708200939.2153664-1-yonghong.song@linux.dev?part=9

Re: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Hardcoded little-endian assumptions in struct and `__int128` register unpacking will cause selftest failures on big-endian architectures like s390x.
--

--- Patch [10]: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns ---
commit dcf423ead16608630f40807ad0cde33d495c941d
Author: Yonghong Song <yonghong.song@linux.dev>

selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

Add inline-asm and subprogram tests for more coverage, including
BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive
and negative tests are added.

> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_run.c b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c
> new file mode 100644
> index 0000000000000..d6300b31733d8
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c

[ ... ]

> @@ -0,0 +1,160 @@

[ ... ]

> +SEC("tc")
> +int aggregate_ret_asm_test(struct __sk_buff *skb)
> +{
> +	__u64 a = skb->len;
> +	__u64 b = skb->len ^ 0xdeadbeefULL;
> +	__u64 lo, hi;

[ ... ]

> +	asm volatile (
> +	"r1 = %[a];"
> +	"r2 = %[b];"
> +	"call %[kfunc];"
> +	"%[lo] = r0;"
> +	"%[hi] = r2;"
> +	: [lo]"=r"(lo), [hi]"=r"(hi)
> +	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_i128)
> +	: "r0", "r1", "r2", "r3", "r4", "r5"
> +	);
> +	if (hi != a)
> +		return 3;
> +	if (lo != b)
> +		return 4;

[Severity: Medium]
The test expects the high 64 bits (a) in r2 and the low 64 bits (b) in
r0. On a big-endian ABI, the most significant 64 bits (a) will be returned
in the first register (r0).

Will this cause the test to fail on big-endian architectures like s390x?

> +
> +	return 0;
> +}
> +
> +SEC("tc")
> +int aggregate_ret_struct_test(struct __sk_buff *skb)
> +{
> +	__u64 a = skb->len;
> +	__u64 b = skb->len ^ 0xdeadbeefULL;
> +	__u64 lo, hi;

[ ... ]

> +	/* struct { u64 a; int b; }: R0 = a, low 32 bits of R2 = b. */
> +	asm volatile (
> +	"r1 = %[a];"
> +	"r2 = %[b];"
> +	"call %[kfunc];"
> +	"%[lo] = r0;"
> +	"%[hi] = r2;"
> +	: [lo]"=r"(lo), [hi]"=r"(hi)
> +	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_li)
> +	: "r0", "r1", "r2", "r3", "r4", "r5"
> +	);
> +	if (lo != a)
> +		return 3;
> +	if ((int)hi != (int)b)
> +		return 4;

[Severity: Medium]
The bitwise extraction (int)hi assumes the 4-byte b is in the lower 32
bits of r2. On big-endian, wouldn't b be placed in the upper 32 bits of
the register, causing this assertion to fail?

> +
> +	/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not used. */
> +	asm volatile (
> +	"r1 = %[a];"
> +	"r2 = %[b];"
> +	"call %[kfunc];"
> +	"%[lo] = r0;"
> +	: [lo]"=r"(lo)
> +	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_ii)
> +	: "r0", "r1", "r2", "r3", "r4", "r5"
> +	);
> +	if ((int)lo != (int)a)
> +		return 5;
> +	if ((int)(lo >> 32) != (int)b)
> +		return 6;

[Severity: Medium]
The bitwise extraction (int)lo and (lo >> 32) assumes a little-endian
register layout for the returned 8-byte struct. On big-endian, wouldn't a
be in the upper 32 bits and b in the lower 32 bits?

> +
> +	return 0;
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260708200939.2153664-1-yonghong.song@linux.dev?part=10

Re: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Leaking uninitialized kernel memory via kfunc R2 return register for structs of 9-15 bytes.
--

--- Patch [8]: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types ---
commit 5b4b6cb320ef1b1f82ac557b6f90945992d8a0c5
Author: Yonghong Song <yonghong.song@linux.dev>
Subject: bpf: Enable 16-byte aggregate return types

This patch relaxes btf_distill_func_proto() and btf_validate_return_type()
to allow kfuncs to return by-value structs or unions up to 16 bytes.
This enables the <=16 byte aggregate return values end to end.

> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> index dff5c0d916412..bab82d1c8cb98 100644
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c
> @@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
>  		return -EINVAL;
>  	}
>  	ret = __get_type_size(btf, func->type, &t);
> -	if (ret < 0 || btf_type_is_struct(t)) {
> +	/*
> +	 * __get_type_size() already restricts a non-negative ret to void, a
> +	 * pointer, an int, an enum or a struct/union, so only the size is checked
> +	 * here.
> +	 */
> +	if (ret < 0 || ret > 16) {

[Severity: High]
Can this leak uninitialized kernel memory for structs between 9 and 15 bytes?

The x86_64 ABI leaves the unused upper bytes of the %rdx register undefined
when returning a struct that is not a multiple of 8 bytes.

Looking at mark_kfunc_ret_reg_size() in kernel/bpf/verifier.c, if the struct
size is greater than 8:

	if (size > 8) {
		mark_btf_func_reg_size(env, BPF_REG_0, 8);
		mark_reg_unknown(env, regs, BPF_REG_2);
		regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
	}

The verifier explicitly marks BPF_REG_2 as fully initialized, bypassing subreg
zero-extension.

And in the x86 JIT do_jit() in arch/x86/net/bpf_jit_comp.c, it moves the full
64-bit register:

				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
				if (fm && fm->ret_size > 8)
					emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);

Could a BPF program read the undefined bytes from R2, resulting in an
information leak?

>  		bpf_log(log,
>  			"The function %s return type %s is unsupported.\n",
>  			tname, btf_type_str(t));

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260708200939.2153664-1-yonghong.song@linux.dev?part=8

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 3471 bytes --]

> diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
> new file mode 100644
> index 0000000000000..d4ecc19684cef
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
> @@ -0,0 +1,96 @@

[ ... ]

> +/*
> + * Run one program and check its result. The program returns 0 on success, or
> + * 1..4 identifying which half of which aggregate (R0:R2) return value came back
> + * wrong. A return of -1 means the program was built with an LLVM that lacks
> + * R0:R2 16-byte return support, in which case the subtest is skipped.
> + */

[ ... ]

> +	/* C variant (__int128): relies on LLVM 23 R0:R2 codegen, skipped otherwise. */
> +	if (test__start_subtest("int128_c"))
> +		run_prog(skel_c->progs.aggregate_ret_test, true);
> +
> +	aggregate_ret_int128_c__destroy(skel_c);
> +
> +	skel_struct_c = aggregate_ret_struct_c__open_and_load();
> +	if (!ASSERT_OK_PTR(skel_struct_c, "skel_struct_c_open_load"))
> +		return;
> +
> +	/*
> +	 * C variant returning a 16-byte struct by value: exercises the struct
> +	 * branch of bpf_ret_reg_pair(). Also LLVM 23 only.
> +	 */
> +	if (test__start_subtest("struct_c"))
> +		run_prog(skel_struct_c->progs.aggregate_ret_struct_c_test, true);
> +
> +	/*
> +	 * Same 16-byte struct return, but from a global (non-static) callee:
> +	 * exercises the compiler lowering a global struct-returning function to
> +	 * R0:R2 and the verifier's global return path end to end. Also LLVM 23
> +	 * only.
> +	 */
> +	if (test__start_subtest("global_struct_c"))
> +		run_prog(skel_struct_c->progs.aggregate_ret_global_struct_c_test, true);
> +
> +	/*
> +	 * Same 16-byte struct, but returned by value from a kfunc: exercises
> +	 * the struct branch of check_kfunc_call() and the JIT RDX->R2 move.
> +	 * Also LLVM 23 only.
> +	 */
> +	if (test__start_subtest("kfunc_struct_c"))
> +		run_prog(skel_struct_c->progs.aggregate_ret_kfunc_struct_c_test, true);

this isn't a bug, but the "Also LLVM 23 only." note is repeated on each
of these per-subtest comments, and the run_prog() comment above already
states that programs built without R0:R2 support return -1 and get
skipped.  Could that note be stated once rather than on every block?

Related, do the per-subtest comments need to name the specific internals
(bpf_ret_reg_pair(), check_kfunc_call(), "the JIT RDX->R2 move"), or
would a shorter note serve here?

> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> index c36bb911defa7..34d0a352e70dd 100644
> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> @@ -55,6 +55,15 @@ struct prog_test_big_arg {
>  	__u64 b;
>  };
>
> +/*
> + * A 16-byte struct returned by value from a kfunc, returned in the R0:R2
> + * register pair.
> + */
> +struct bpf_kfunc_ret_ll {	/* 16 bytes: R0:R2 */

this isn't a bug, but the block comment already says the struct is 16
bytes and returned in the R0:R2 register pair, and the trailing
/* 16 bytes: R0:R2 */ on the next line repeats that.  Could one of the
two be dropped?

> +	__u64 a;
> +	__u64 b;
> +};


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

Re: [PATCH bpf-next 05/12] bpf: Account R2 of register-pair returns in live register analysis

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1757 bytes --]

> commit 29d6a87c34e27de90ba01d71870930baf27d4b33
> Author: Yonghong Song <yonghong.song@linux.dev>
> Date:   Wed Jul 8 13:10:05 2026 -0700
>
>     bpf: Account R2 of register-pair returns in live register analysis
>
>     A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a
>     struct/union or an __int128) reads R2 as well as R0, since the second half
>     of the return value is passed back in R2. compute_insn_live_regs() only
>     marked R0 used at exit, so the callee's R2 could be considered dead and
>     cleaned from checkpointed states, which would allow unsound state pruning.
>
>     Mark R2 used at a BPF_EXIT when the enclosing subprogram returns via the
>     R0:R2 register pair. The call site already treats the caller-saved
>     registers R0-R5 as clobbered, so R2 as a result register is covered there.
>
>     For example, for a subprogram returning an __int128, the live-registers
>     dump (log level 2) shows both R0 and R2 live before the callee's exit
>     (insn 5):
>
>       Live regs before insn:
>             0: .12345.... (85) call pc+2
>             1: ..2....... (bf) r0 = r2
>             2: 0......... (95) exit
>             3: .......... (b7) r0 = 1
>             4: 0......... (b7) r2 = 2
>             5: 0.2....... (95) exit
>
>     Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

This looks like a bug fix for 16-byte return type support. Should this
include:

  Fixes: ec219322a9a71 ("bpf: Add verifier support for 16-byte returns in R0:R2")


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

Re: [PATCH bpf-next 04/12] bpf: Track R2 of register-pair returns in precision backtracking

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1477 bytes --]

> A function returning a value larger than 8 bytes (a struct/union, or an
> __int128) uses R2 as a second return register alongside R0. Previous
> precision backtracking only treated R0 as a return register at a
> call/return boundary, so marking the second half of such a return precise
> would trip the "unexpected regs" checks in backtrack_insn() and reject a
> valid program with -EFAULT.
>
> Handle R2 like R0 in the three boundaries where a call defines the return
> registers:
>
>  - static subprog exit (BPF_EXIT): R2 is a return register, not a leftover
>    argument; clear it before the args check and propagate its precision
>    into the callee.
>  - global subprog call: a global subprog returning >8 bytes also sets R2;
>    clear it before the args check.
>  - kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8)
>    also sets R2; clear it like R0.
>
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

This looks like a fix for the R0:R2 register-pair return handling that
went in earlier in this series, where backtrack_insn() was not updated to
treat R2 as a return register.  Should this carry a Fixes: tag?

  Fixes: ec219322a9a7 ("bpf: Add verifier support for 16-byte returns in R0:R2")


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

>     bpf: Force JIT for programs using the R0:R2 register pair
>
>     The R0:R2 return convention introduced in this series is only implemented
>     in the JIT. The BPF interpreter has no notion of a second return register,
>     so a program relying on the R0:R2 convention would compute wrong results
>     under the interpreter. Force the JIT when bpf_prog_aux->ret_reg_pair is
>     set.
>
>     Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

This looks like a bug fix for the R0:R2 return convention, where the
verifier accepted such programs without gating them to JIT-only.  The
bpf_ret_reg_pair() support that introduced this was added earlier in the
series.  Should this include:

  Fixes: ec219322a9a7 ("bpf: Add verifier support for 16-byte returns in R0:R2")


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

Re: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1938 bytes --]

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 3f34ce1a3107d..03ffb5f839fa0 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
>  	return btf_type_is_void(type);
>  }
>
> +static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
> +{
> +	const struct btf_type *type, *func, *func_proto;
> +	const struct btf *btf = env->prog->aux->btf;
> +	u32 btf_id;
> +
> +	if (!btf || !env->prog->aux->func_info)
> +		return false;
> +
> +	btf_id = env->prog->aux->func_info[subprog].type_id;
> +
> +	func = btf_type_by_id(btf, btf_id);
> +	if (!func)
> +		return false;
> +
> +	func_proto = btf_type_by_id(btf, func->type);
> +	if (!func_proto)
> +		return false;
> +
> +	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
> +	if (!type)
> +		return false;
> +
> +	if (btf_type_is_struct(type) || btf_type_is_scalar(type))
> +		return type->size > 8 && type->size <= 16;
> +
> +	return false;
> +}

This isn't a bug, but could bpf_ret_reg_pair() and subprog_returns_void()
share a helper for the func_info -> func -> func_proto -> return type walk?

Both functions repeat the same ~8 line sequence:

subprog_returns_void():
	btf_id = env->prog->aux->func_info[subprog].type_id;
	func = btf_type_by_id(btf, btf_id);
	if (!func)
		return false;
	func_proto = btf_type_by_id(btf, func->type);
	if (!func_proto)
		return false;
	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);

They only differ in the final predicate: subprog_returns_void() checks
btf_type_is_void(type), while bpf_ret_reg_pair() checks the struct/scalar
size.

[ ... ]


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

[Leon Hwang]

On 9/7/26 04:10, Yonghong Song wrote:
> The R0:R2 return convention introduced in this series is only implemented
> in the JIT. The BPF interpreter has no notion of a second return register,
> so a program relying on the R0:R2 convention would compute wrong results
> under the interpreter. Force the JIT when bpf_prog_aux->ret_reg_pair is
> set.
> 
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> ---
>  include/linux/bpf.h   |  1 +
>  kernel/bpf/core.c     |  2 +-
>  kernel/bpf/verifier.c | 13 +++++++++++--
>  3 files changed, 13 insertions(+), 3 deletions(-)
> 
> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> index c1a98fa36738..10966aba69a2 100644
> --- a/include/linux/bpf.h
> +++ b/include/linux/bpf.h
> @@ -1780,6 +1780,7 @@ struct bpf_prog_aux {
>  	bool changes_pkt_data;
>  	bool might_sleep;
>  	bool kprobe_write_ctx;
> +	bool ret_reg_pair;
>  	struct {
>  		s32 keyring_serial;
>  		u8 keyring_type;
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index 73dc3ee879de..819a97d25173 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
> @@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
>  		goto finalize;
>  
>  	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
> -	    bpf_prog_has_kfunc_call(fp))
> +	    bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)
>  		jit_needed = true;
>  
>  	if (!bpf_prog_select_interpreter(fp))
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index fcbf532159a5..ca8b2f436c20 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
>  	return false;
>  }
>  
> +static bool bpf_ret_reg_pair_mark(struct bpf_verifier_env *env, int subprog)
> +{
> +	if (!bpf_ret_reg_pair(env, subprog))
> +		return false;
> +
> +	env->prog->aux->ret_reg_pair = true;


With 'jit_required' introduction [1], it has no need to introduce
'ret_reg_pair'. 'env->prog->jit_required = true;' is enough.

[1]
https://lore.kernel.org/bpf/20260708101806.18885-1-yangtiezhu@loongson.cn/

Thanks,
Leon
> +	return true;
> +}
> +
>  static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
>  {
>  	struct bpf_func_info *info;
> @@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
>  	} else {
>  		/* return to the caller whatever r0 had in the callee */
>  		caller->regs[BPF_REG_0] = *r0;
> -		if (bpf_ret_reg_pair(env, callee->subprogno)) {
> +		if (bpf_ret_reg_pair_mark(env, callee->subprogno)) {
>  			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
>  				verbose(env, "cannot return stack pointer to the caller\n");
>  				return -EINVAL;
> @@ -16806,7 +16815,7 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
>  	if (err)
>  		return err;
>  
> -	if (!bpf_ret_reg_pair(env, subprog))
> +	if (!bpf_ret_reg_pair_mark(env, subprog))
>  		return 0;
>  
>  	return check_global_ret_scalar_reg(env, BPF_REG_2);

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

[Leon Hwang]

On 9/7/26 04:10, Yonghong Song wrote:
> btf_distill_func_proto() builds the function model used for kfunc calls,
> the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following
> patch relaxes it to accept a >8 and <=16 byte return value, which for the
> BPF target is passed back in the R0:R2 register pair.


Do you have a plan to add >8 byte return value support for them?

Thanks,
Leon

> [...]

Re: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2

[Yonghong Song]

On 7/8/26 2:17 PM, bot+bpf-ci@kernel.org wrote:
>> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
>> index 3f34ce1a3107d..03ffb5f839fa0 100644
>> --- a/kernel/bpf/verifier.c
>> +++ b/kernel/bpf/verifier.c
>> @@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
>>   	return btf_type_is_void(type);
>>   }
>>
>> +static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
>> +{
>> +	const struct btf_type *type, *func, *func_proto;
>> +	const struct btf *btf = env->prog->aux->btf;
>> +	u32 btf_id;
>> +
>> +	if (!btf || !env->prog->aux->func_info)
>> +		return false;
>> +
>> +	btf_id = env->prog->aux->func_info[subprog].type_id;
>> +
>> +	func = btf_type_by_id(btf, btf_id);
>> +	if (!func)
>> +		return false;
>> +
>> +	func_proto = btf_type_by_id(btf, func->type);
>> +	if (!func_proto)
>> +		return false;
>> +
>> +	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
>> +	if (!type)
>> +		return false;
>> +
>> +	if (btf_type_is_struct(type) || btf_type_is_scalar(type))
>> +		return type->size > 8 && type->size <= 16;
>> +
>> +	return false;
>> +}
> This isn't a bug, but could bpf_ret_reg_pair() and subprog_returns_void()
> share a helper for the func_info -> func -> func_proto -> return type walk?
>
> Both functions repeat the same ~8 line sequence:
>
> subprog_returns_void():
> 	btf_id = env->prog->aux->func_info[subprog].type_id;
> 	func = btf_type_by_id(btf, btf_id);
> 	if (!func)
> 		return false;
> 	func_proto = btf_type_by_id(btf, func->type);
> 	if (!func_proto)
> 		return false;
> 	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
>
> They only differ in the final predicate: subprog_returns_void() checks
> btf_type_is_void(type), while bpf_ret_reg_pair() checks the struct/scalar
> size.
>
> [ ... ]

Indeed, I think this probably worthwhile to factor a new helper.

>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

Re: [PATCH bpf-next 04/12] bpf: Track R2 of register-pair returns in precision backtracking

[Yonghong Song]

On 7/8/26 2:10 PM, bot+bpf-ci@kernel.org wrote:
>> A function returning a value larger than 8 bytes (a struct/union, or an
>> __int128) uses R2 as a second return register alongside R0. Previous
>> precision backtracking only treated R0 as a return register at a
>> call/return boundary, so marking the second half of such a return precise
>> would trip the "unexpected regs" checks in backtrack_insn() and reject a
>> valid program with -EFAULT.
>>
>> Handle R2 like R0 in the three boundaries where a call defines the return
>> registers:
>>
>>   - static subprog exit (BPF_EXIT): R2 is a return register, not a leftover
>>     argument; clear it before the args check and propagate its precision
>>     into the callee.
>>   - global subprog call: a global subprog returning >8 bytes also sets R2;
>>     clear it before the args check.
>>   - kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8)
>>     also sets R2; clear it like R0.
>>
>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> This looks like a fix for the R0:R2 register-pair return handling that
> went in earlier in this series, where backtrack_insn() was not updated to
> treat R2 as a return register.  Should this carry a Fixes: tag?
>
>    Fixes: ec219322a9a7 ("bpf: Add verifier support for 16-byte returns in R0:R2")

This is still a preparation patch, so Fixes is not necessary. The same
for a few later patches.

>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

[Yonghong Song]

On 7/8/26 8:15 PM, Leon Hwang wrote:
> On 9/7/26 04:10, Yonghong Song wrote:
>> The R0:R2 return convention introduced in this series is only implemented
>> in the JIT. The BPF interpreter has no notion of a second return register,
>> so a program relying on the R0:R2 convention would compute wrong results
>> under the interpreter. Force the JIT when bpf_prog_aux->ret_reg_pair is
>> set.
>>
>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>> ---
>>   include/linux/bpf.h   |  1 +
>>   kernel/bpf/core.c     |  2 +-
>>   kernel/bpf/verifier.c | 13 +++++++++++--
>>   3 files changed, 13 insertions(+), 3 deletions(-)
>>
>> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
>> index c1a98fa36738..10966aba69a2 100644
>> --- a/include/linux/bpf.h
>> +++ b/include/linux/bpf.h
>> @@ -1780,6 +1780,7 @@ struct bpf_prog_aux {
>>   	bool changes_pkt_data;
>>   	bool might_sleep;
>>   	bool kprobe_write_ctx;
>> +	bool ret_reg_pair;
>>   	struct {
>>   		s32 keyring_serial;
>>   		u8 keyring_type;
>> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
>> index 73dc3ee879de..819a97d25173 100644
>> --- a/kernel/bpf/core.c
>> +++ b/kernel/bpf/core.c
>> @@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
>>   		goto finalize;
>>   
>>   	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
>> -	    bpf_prog_has_kfunc_call(fp))
>> +	    bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)
>>   		jit_needed = true;
>>   
>>   	if (!bpf_prog_select_interpreter(fp))
>> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
>> index fcbf532159a5..ca8b2f436c20 100644
>> --- a/kernel/bpf/verifier.c
>> +++ b/kernel/bpf/verifier.c
>> @@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
>>   	return false;
>>   }
>>   
>> +static bool bpf_ret_reg_pair_mark(struct bpf_verifier_env *env, int subprog)
>> +{
>> +	if (!bpf_ret_reg_pair(env, subprog))
>> +		return false;
>> +
>> +	env->prog->aux->ret_reg_pair = true;
>
> With 'jit_required' introduction [1], it has no need to introduce
> 'ret_reg_pair'. 'env->prog->jit_required = true;' is enough.
>
> [1]
> https://lore.kernel.org/bpf/20260708101806.18885-1-yangtiezhu@loongson.cn/

Thanks! I will use jit_required in the next revision. I guess this will
be useful for many other cases who do not want to support interpreter.

>
> Thanks,
> Leon
>> +	return true;
>> +}
>> +
>>   static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
>>   {
>>   	struct bpf_func_info *info;
>> @@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
>>   	} else {
>>   		/* return to the caller whatever r0 had in the callee */
>>   		caller->regs[BPF_REG_0] = *r0;
>> -		if (bpf_ret_reg_pair(env, callee->subprogno)) {
>> +		if (bpf_ret_reg_pair_mark(env, callee->subprogno)) {
>>   			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
>>   				verbose(env, "cannot return stack pointer to the caller\n");
>>   				return -EINVAL;
>> @@ -16806,7 +16815,7 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
>>   	if (err)
>>   		return err;
>>   
>> -	if (!bpf_ret_reg_pair(env, subprog))
>> +	if (!bpf_ret_reg_pair_mark(env, subprog))
>>   		return 0;
>>   
>>   	return check_global_ret_scalar_reg(env, BPF_REG_2);

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

[Yonghong Song]

On 7/8/26 8:16 PM, Leon Hwang wrote:
> On 9/7/26 04:10, Yonghong Song wrote:
>> btf_distill_func_proto() builds the function model used for kfunc calls,
>> the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following
>> patch relaxes it to accept a >8 and <=16 byte return value, which for the
>> BPF target is passed back in the R0:R2 register pair.
>
> Do you have a plan to add >8 byte return value support for them?

I tested with vmlinux BTF built from the latest bpf-next. I got (from vmlinux BTF):

    total number of functions: 67500
    return type size (> 8 && <= 16): 51
    return type size (> 16): 10

So in the initial version, I didn't support >8 byte for trampoline related
bpf programs.

But I can certainly do this, we just need to allocate another 8 byte for return value
in trampoline.

>
> Thanks,
> Leon
>
>> [...]

Re: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types

[Yonghong Song]

On 7/8/26 1:28 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [High] Leaking uninitialized kernel memory via kfunc R2 return register for structs of 9-15 bytes.
> --
>
> --- Patch [8]: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types ---
> commit 5b4b6cb320ef1b1f82ac557b6f90945992d8a0c5
> Author: Yonghong Song <yonghong.song@linux.dev>
> Subject: bpf: Enable 16-byte aggregate return types
>
> This patch relaxes btf_distill_func_proto() and btf_validate_return_type()
> to allow kfuncs to return by-value structs or unions up to 16 bytes.
> This enables the <=16 byte aggregate return values end to end.
>
>> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
>> index dff5c0d916412..bab82d1c8cb98 100644
>> --- a/kernel/bpf/btf.c
>> +++ b/kernel/bpf/btf.c
>> @@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
>>   		return -EINVAL;
>>   	}
>>   	ret = __get_type_size(btf, func->type, &t);
>> -	if (ret < 0 || btf_type_is_struct(t)) {
>> +	/*
>> +	 * __get_type_size() already restricts a non-negative ret to void, a
>> +	 * pointer, an int, an enum or a struct/union, so only the size is checked
>> +	 * here.
>> +	 */
>> +	if (ret < 0 || ret > 16) {
> [Severity: High]
> Can this leak uninitialized kernel memory for structs between 9 and 15 bytes?
>
> The x86_64 ABI leaves the unused upper bytes of the %rdx register undefined
> when returning a struct that is not a multiple of 8 bytes.
>
> Looking at mark_kfunc_ret_reg_size() in kernel/bpf/verifier.c, if the struct
> size is greater than 8:
>
> 	if (size > 8) {
> 		mark_btf_func_reg_size(env, BPF_REG_0, 8);
> 		mark_reg_unknown(env, regs, BPF_REG_2);
> 		regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
> 	}
>
> The verifier explicitly marks BPF_REG_2 as fully initialized, bypassing subreg
> zero-extension.
>
> And in the x86 JIT do_jit() in arch/x86/net/bpf_jit_comp.c, it moves the full
> 64-bit register:
>
> 				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
> 				if (fm && fm->ret_size > 8)
> 					emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
>
> Could a BPF program read the undefined bytes from R2, resulting in an
> information leak?

We should be fine here. See llvm patch https://reviews.llvm.org/D131598
which specifically tries to address this issue. The following is an example
for R2:

$ cat t.c
struct foo { long a; short b; };
struct foo bar(void);

int tar(void) {
   struct foo t = bar();
   return t.a + t.b;
}
$ clang --target=bpf -O2 -S t.c
$ cat t.s
         .file   "t.c"
         .text
         .globl  tar                             # -- Begin function tar
         .p2align        3
         .type   tar,@function
tar:                                    # @tar
# %bb.0:
         call bar
         w2 <<= 16
         w2 s>>= 16
         w2 += w0
         w0 = w2
         exit
.Lfunc_end0:
         .size   tar, .Lfunc_end0-tar
                                         # -- End function
         .addrsig

The caller will do proper narrowing immediately after call return.

>
>>   		bpf_log(log,
>>   			"The function %s return type %s is unsupported.\n",
>>   			tname, btf_type_str(t));

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

[Yonghong Song]

On 7/8/26 1:24 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Low] Unconditional use of `__int128` in `bpf_testmod` breaks compilation on 32-bit host architectures.
> --
>
> --- Patch [9]: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2 ---
> commit 5ed3844739f04c920bb1c37cdf474501b3ca9fef
> Author: Yonghong Song <yonghong.song@linux.dev>
>
> selftests/bpf: Add C tests for 16-byte returns in R0:R2
>
> This commit adds selftests to exercise 16-byte return values passed in the
> R0:R2 register pair. The tests include an __int128 return, a 16-byte struct
> return, and a 16-byte union return.
>
> [ ... ]
>
>> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
>> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
>> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
> [ ... ]
>
>> @@ -825,6 +825,18 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
>>   	return 0;
>>   }
>>   
>> +__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
>> +{
>> +	return ((__int128)a << 64) | b;
>> +}
> [Severity: Low]
> Will this unconditional use of __int128 break the build on 32-bit host
> architectures?
>
> The __int128 type is a compiler extension that is typically only available
> on 64-bit systems.
>
> [ ... ]
>
>> @@ -1358,6 +1370,8 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
>>   BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
>>   BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
>>   BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
>> +BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
> [Severity: Low]
> If bpf_kfunc_call_test_i128 needs to be conditionally excluded for 32-bit
> builds to fix the issue above, should this BTF ID registration also be
> conditionally excluded to prevent a missing symbol error during the build?
>
> [ ... ]
>
>> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
>> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
>> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> [ ... ]
>
>> @@ -118,6 +127,8 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
>>   struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
>>   long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
>>   int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
>> +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
> [Severity: Low]
> Does this declaration also need to be guarded for 64-bit architectures,
> so that including this header doesn't fail on 32-bit host compilers?

We may need to guard with __SIZEOF_INT128__ in selftests. But current CI all supports 64bit,
so I am not sure whether we should do it or not.

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

[Yonghong Song]

On 7/8/26 2:10 PM, bot+bpf-ci@kernel.org wrote:
>> diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
>> new file mode 100644
>> index 0000000000000..d4ecc19684cef
>> --- /dev/null
>> +++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
>> @@ -0,0 +1,96 @@
> [ ... ]
>
>> +/*
>> + * Run one program and check its result. The program returns 0 on success, or
>> + * 1..4 identifying which half of which aggregate (R0:R2) return value came back
>> + * wrong. A return of -1 means the program was built with an LLVM that lacks
>> + * R0:R2 16-byte return support, in which case the subtest is skipped.
>> + */
> [ ... ]
>
>> +	/* C variant (__int128): relies on LLVM 23 R0:R2 codegen, skipped otherwise. */
>> +	if (test__start_subtest("int128_c"))
>> +		run_prog(skel_c->progs.aggregate_ret_test, true);
>> +
>> +	aggregate_ret_int128_c__destroy(skel_c);
>> +
>> +	skel_struct_c = aggregate_ret_struct_c__open_and_load();
>> +	if (!ASSERT_OK_PTR(skel_struct_c, "skel_struct_c_open_load"))
>> +		return;
>> +
>> +	/*
>> +	 * C variant returning a 16-byte struct by value: exercises the struct
>> +	 * branch of bpf_ret_reg_pair(). Also LLVM 23 only.
>> +	 */
>> +	if (test__start_subtest("struct_c"))
>> +		run_prog(skel_struct_c->progs.aggregate_ret_struct_c_test, true);
>> +
>> +	/*
>> +	 * Same 16-byte struct return, but from a global (non-static) callee:
>> +	 * exercises the compiler lowering a global struct-returning function to
>> +	 * R0:R2 and the verifier's global return path end to end. Also LLVM 23
>> +	 * only.
>> +	 */
>> +	if (test__start_subtest("global_struct_c"))
>> +		run_prog(skel_struct_c->progs.aggregate_ret_global_struct_c_test, true);
>> +
>> +	/*
>> +	 * Same 16-byte struct, but returned by value from a kfunc: exercises
>> +	 * the struct branch of check_kfunc_call() and the JIT RDX->R2 move.
>> +	 * Also LLVM 23 only.
>> +	 */
>> +	if (test__start_subtest("kfunc_struct_c"))
>> +		run_prog(skel_struct_c->progs.aggregate_ret_kfunc_struct_c_test, true);
> this isn't a bug, but the "Also LLVM 23 only." note is repeated on each
> of these per-subtest comments, and the run_prog() comment above already
> states that programs built without R0:R2 support return -1 and get
> skipped.  Could that note be stated once rather than on every block?

Ack. Will do.

>
> Related, do the per-subtest comments need to name the specific internals
> (bpf_ret_reg_pair(), check_kfunc_call(), "the JIT RDX->R2 move"), or
> would a shorter note serve here?

I guess I can rename these comments, function names themselves already have
enough information.

>
>> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
>> index c36bb911defa7..34d0a352e70dd 100644
>> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
>> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
>> @@ -55,6 +55,15 @@ struct prog_test_big_arg {
>>   	__u64 b;
>>   };
>>
>> +/*
>> + * A 16-byte struct returned by value from a kfunc, returned in the R0:R2
>> + * register pair.
>> + */
>> +struct bpf_kfunc_ret_ll {	/* 16 bytes: R0:R2 */
> this isn't a bug, but the block comment already says the struct is 16
> bytes and returned in the R0:R2 register pair, and the trailing
> /* 16 bytes: R0:R2 */ on the next line repeats that.  Could one of the
> two be dropped?

Yes, will drop.

>
>> +	__u64 a;
>> +	__u64 b;
>> +};
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/28973410219

Re: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

[Yonghong Song]

On 7/8/26 1:28 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Medium] Hardcoded little-endian assumptions in struct and `__int128` register unpacking will cause selftest failures on big-endian architectures like s390x.
> --
>
> --- Patch [10]: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns ---
> commit dcf423ead16608630f40807ad0cde33d495c941d
> Author: Yonghong Song <yonghong.song@linux.dev>
>
> selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
>
> Add inline-asm and subprogram tests for more coverage, including
> BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive
> and negative tests are added.
>
>> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_run.c b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c
>> new file mode 100644
>> index 0000000000000..d6300b31733d8
>> --- /dev/null
>> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c
> [ ... ]
>
>> @@ -0,0 +1,160 @@
> [ ... ]
>
>> +SEC("tc")
>> +int aggregate_ret_asm_test(struct __sk_buff *skb)
>> +{
>> +	__u64 a = skb->len;
>> +	__u64 b = skb->len ^ 0xdeadbeefULL;
>> +	__u64 lo, hi;
> [ ... ]
>
>> +	asm volatile (
>> +	"r1 = %[a];"
>> +	"r2 = %[b];"
>> +	"call %[kfunc];"
>> +	"%[lo] = r0;"
>> +	"%[hi] = r2;"
>> +	: [lo]"=r"(lo), [hi]"=r"(hi)
>> +	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_i128)
>> +	: "r0", "r1", "r2", "r3", "r4", "r5"
>> +	);
>> +	if (hi != a)
>> +		return 3;
>> +	if (lo != b)
>> +		return 4;
> [Severity: Medium]
> The test expects the high 64 bits (a) in r2 and the low 64 bits (b) in
> r0. On a big-endian ABI, the most significant 64 bits (a) will be returned
> in the first register (r0).
>
> Will this cause the test to fail on big-endian architectures like s390x?

Good point. I need to test it for s390x.

>
>> +
>> +	return 0;
>> +}
>> +
>> +SEC("tc")
>> +int aggregate_ret_struct_test(struct __sk_buff *skb)
>> +{
>> +	__u64 a = skb->len;
>> +	__u64 b = skb->len ^ 0xdeadbeefULL;
>> +	__u64 lo, hi;
> [ ... ]
>
>> +	/* struct { u64 a; int b; }: R0 = a, low 32 bits of R2 = b. */
>> +	asm volatile (
>> +	"r1 = %[a];"
>> +	"r2 = %[b];"
>> +	"call %[kfunc];"
>> +	"%[lo] = r0;"
>> +	"%[hi] = r2;"
>> +	: [lo]"=r"(lo), [hi]"=r"(hi)
>> +	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_li)
>> +	: "r0", "r1", "r2", "r3", "r4", "r5"
>> +	);
>> +	if (lo != a)
>> +		return 3;
>> +	if ((int)hi != (int)b)
>> +		return 4;
> [Severity: Medium]
> The bitwise extraction (int)hi assumes the 4-byte b is in the lower 32
> bits of r2. On big-endian, wouldn't b be placed in the upper 32 bits of
> the register, causing this assertion to fail?

The same as above. Need tests with s390x.

>
>> +
>> +	/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not used. */
>> +	asm volatile (
>> +	"r1 = %[a];"
>> +	"r2 = %[b];"
>> +	"call %[kfunc];"
>> +	"%[lo] = r0;"
>> +	: [lo]"=r"(lo)
>> +	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_ii)
>> +	: "r0", "r1", "r2", "r3", "r4", "r5"
>> +	);
>> +	if ((int)lo != (int)a)
>> +		return 5;
>> +	if ((int)(lo >> 32) != (int)b)
>> +		return 6;
> [Severity: Medium]
> The bitwise extraction (int)lo and (lo >> 32) assumes a little-endian
> register layout for the returned 8-byte struct. On big-endian, wouldn't a
> be in the upper 32 bits and b in the lower 32 bits?

Will test as the above.

>
>> +
>> +	return 0;
>> +}

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

[Eduard Zingerman]

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:

[...]

> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index 73dc3ee879de..819a97d25173 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
> @@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
>  		goto finalize;
>  
>  	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
> -	    bpf_prog_has_kfunc_call(fp))
> +	    bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)

This has a conflict with one of the very recent bpf-next commits.
While reading locally I fixed the conflict by removing this hunk
and adjusting bpf_ret_reg_pair_mark() as below.

>  		jit_needed = true;
>  
>  	if (!bpf_prog_select_interpreter(fp))
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index fcbf532159a5..ca8b2f436c20 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
>  	return false;
>  }
>  
> +static bool bpf_ret_reg_pair_mark(struct bpf_verifier_env *env, int subprog)
> +{
> +	if (!bpf_ret_reg_pair(env, subprog))
> +		return false;
> +
> +	env->prog->aux->ret_reg_pair = true;

Added:
	env->prog->jit_required = true;

> +	return true;
> +}
> +

You probably would want to do the same in v2.

[...]

Re: [PATCH bpf-next 05/12] bpf: Account R2 of register-pair returns in live register analysis

[Eduard Zingerman]

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
> A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a
> struct/union or an __int128) reads R2 as well as R0, since the second half
> of the return value is passed back in R2. compute_insn_live_regs() only
> marked R0 used at exit, so the callee's R2 could be considered dead and
> cleaned from checkpointed states, which would allow unsound state pruning.
> 
> Mark R2 used at a BPF_EXIT when the enclosing subprogram returns via the
> R0:R2 register pair. The call site already treats the caller-saved
> registers R0-R5 as clobbered, so R2 as a result register is covered there.
> 
> For example, for a subprogram returning an __int128, the live-registers
> dump (log level 2) shows both R0 and R2 live before the callee's exit
> (insn 5):
> 
>   Live regs before insn:
>         0: .12345.... (85) call pc+2
>         1: ..2....... (bf) r0 = r2
>         2: 0......... (95) exit
>         3: .......... (b7) r0 = 1
>         4: 0......... (b7) r2 = 2
>         5: 0.2....... (95) exit
> 
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> ---
>  kernel/bpf/liveness.c | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
> index 0aadfbae0acc..0977d32fcf01 100644
> --- a/kernel/bpf/liveness.c
> +++ b/kernel/bpf/liveness.c
> @@ -2062,7 +2062,8 @@ struct insn_live_regs {
>  /* Compute info->{use,def} fields for the instruction */
>  static void compute_insn_live_regs(struct bpf_verifier_env *env,
>  				   struct bpf_insn *insn,
> -				   struct insn_live_regs *info)
> +				   struct insn_live_regs *info,
> +				   int subprog)
>  {
>  	struct bpf_call_summary cs;
>  	u8 class = BPF_CLASS(insn->code);
> @@ -2175,6 +2176,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
>  		case BPF_EXIT:
>  			def = 0;
>  			use = r0;
> +			if (bpf_ret_reg_pair(env, subprog))
> +				use |= BIT(BPF_REG_2);
>  			break;
>  		case BPF_CALL:
>  			def = ALL_CALLER_SAVED_REGS;
> @@ -2237,8 +2240,12 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
>  		goto out;
>  	}
>  
> -	for (i = 0; i < insn_cnt; ++i)
> -		compute_insn_live_regs(env, &insns[i], &state[i]);
> +	for (i = 0, j = 0; i < insn_cnt; ++i) {
> +		/* Advance to the subprog that contains instruction i. */
> +		while (j + 1 < env->subprog_cnt && env->subprog_info[j + 1].start <= i)
> +			j++;

This loop is very in-elegant. The outer loop can be changed to iterate
subprograms, with inner loop iterating instructions. Or just use
bpf_find_subprog() when processing EXIT.

> +		compute_insn_live_regs(env, &insns[i], &state[i], j);
> +	}
>  
>  	/* Forward pass: resolve stack access through FP-derived pointers */
>  	err = bpf_compute_subprog_arg_access(env);

Re: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2

[Eduard Zingerman]

On Wed, 2026-07-08 at 13:09 -0700, Yonghong Song wrote:
> LLVM 23 added support for returning a value in two registers for an
> __int128, or a struct/union whose size is greater than 8 but not more
> than 16 bytes. See LLVM patches [1] and [2].
> 
> Before LLVM 23 the BPF backend could not return these values at all. A
> by-value struct or union return (of any size) was rejected at compile
> time with:
> 
>   error: aggregate returns are not supported
> 
> and an __int128 return failed later in the backend with:
> 
>   fatal error: error in backend: unable to allocate function return #1
> 
> Both are resolved in LLVM 23, which lowers such returns into the R0:R2
> register pair.
> 
> This patch is one of several preparation patches that build up
> <=16 byte return support incrementally:
>   - the verifier R0:R2 modelling in this patch,
>   - the x86 JIT move and JIT-capability gate,
>   - precision-backtracking and live-register tracking of R2,
>   - forcing the JIT for such programs, and
>   - guarding the trampoline paths.
> Finally, the patch "bpf: Enable 16-byte aggregate return types" enables
> support for 16-byte returns.

Nit: I think the above paragraphs belong to a cover letter.

> This patch adds handling for '>8' byte returns in several places: BPF
> subprogram returns (the main program, and both global and static
> subprograms) and kfunc returns.
> 
>   [1] https://github.com/llvm/llvm-project/pull/190894
>   [2] https://github.com/llvm/llvm-project/pull/206876
> 
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> ---
>  kernel/bpf/verifier.c | 93 ++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 88 insertions(+), 5 deletions(-)
> 
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 3f34ce1a3107..03ffb5f839fa 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
>  	return btf_type_is_void(type);
>  }
>  
> +static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
> +{
> +	const struct btf_type *type, *func, *func_proto;
> +	const struct btf *btf = env->prog->aux->btf;
> +	u32 btf_id;
> +
> +	if (!btf || !env->prog->aux->func_info)
> +		return false;
> +
> +	btf_id = env->prog->aux->func_info[subprog].type_id;
> +
> +	func = btf_type_by_id(btf, btf_id);
> +	if (!func)
> +		return false;

`if (!func) return false` checks here and below are not necessary
after BTF validation.

> +
> +	func_proto = btf_type_by_id(btf, func->type);
> +	if (!func_proto)
> +		return false;
> +
> +	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
> +	if (!type)
> +		return false;
> +
> +	if (btf_type_is_struct(type) || btf_type_is_scalar(type))
> +		return type->size > 8 && type->size <= 16;
> +
> +	return false;
> +}
> +
>  static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
>  {
>  	struct bpf_func_info *info;
> @@ -9459,10 +9488,17 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
>  		clear_caller_saved_regs(env, caller->regs);
>  		invalidate_outgoing_stack_args(env, cur_func(env));
>  
> -		/* All non-void global functions return a 64-bit SCALAR_VALUE. */
> +		/*
> +		 * A non-void global function returns a 64-bit SCALAR_VALUE in
> +		 * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
> +		 */
>  		if (!subprog_returns_void(env, subprog)) {
>  			mark_reg_unknown(env, caller->regs, BPF_REG_0);
>  			caller->regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG;
> +			if (bpf_ret_reg_pair(env, subprog)) {
> +				mark_reg_unknown(env, caller->regs, BPF_REG_2);
> +				caller->regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
> +			}
>  		}
>  
>  		if (env->subprog_info[subprog].might_throw) {
> @@ -9825,6 +9861,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
>  	} else {
>  		/* return to the caller whatever r0 had in the callee */
>  		caller->regs[BPF_REG_0] = *r0;
> +		if (bpf_ret_reg_pair(env, callee->subprogno)) {
> +			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {

Let's consolidate all such checks in one place and not copy-paste them.

> +				verbose(env, "cannot return stack pointer to the caller\n");
> +				return -EINVAL;
> +			}
> +			caller->regs[BPF_REG_2] = callee->regs[BPF_REG_2];
> +		}
>  	}
>  
>  	/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
> @@ -10745,6 +10788,18 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno,
>  	return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size);
>  }
>  
> +static void mark_kfunc_ret_reg_size(struct bpf_verifier_env *env,
> +				    struct bpf_reg_state *regs, u32 size)
> +{
> +	if (size > 8) {
> +		mark_btf_func_reg_size(env, BPF_REG_0, 8);
> +		mark_reg_unknown(env, regs, BPF_REG_2);
> +		regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
> +	} else {
> +		mark_btf_func_reg_size(env, BPF_REG_0, size);
> +	}
> +}

This function makes R2 handling very asymmetric compared to R0:
- mark_reg_unknown() for r0 is done in check_kfunc_call()
- DEF_NOT_SUBREG for r0 is done in mark_btf_func_reg_size()

I think the code should be structured to keep R0 and R2 processing
uniform.

> +
>  static bool is_kfunc_acquire(struct bpf_kfunc_call_arg_meta *meta)
>  {
>  	return meta->kfunc_flags & KF_ACQUIRE;
> @@ -13208,7 +13263,22 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
>  		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
>  		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
>  			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
> -		mark_btf_func_reg_size(env, BPF_REG_0, t->size);
> +		mark_kfunc_ret_reg_size(env, regs, t->size);
> +	} else if (btf_type_is_struct(t)) {
> +		/*
> +		 * The returned struct comes back as raw register bits modeled
> +		 * as an unknown scalar, so it must contain only scalars:
> +		 * otherwise a pointer field would be laundered into a scalar
> +		 * and escape provenance and reference tracking.
> +		 */
> +		if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
> +			verbose(env, "kernel function %s returns %s %s that is not composed of scalars\n",
> +				func_name, btf_type_str(t),
> +				btf_name_by_offset(desc_btf, t->name_off));

Should this also check the size of the struct?

> +			return -EINVAL;
> +		}
> +		mark_reg_unknown(env, regs, BPF_REG_0);
> +		mark_kfunc_ret_reg_size(env, regs, t->size);
>  	} else if (btf_type_is_ptr(t)) {
>  		ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
>  		err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);

[...]

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

[Eduard Zingerman]

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:

[...]

> @@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
>  	} else {
>  		/* return to the caller whatever r0 had in the callee */
>  		caller->regs[BPF_REG_0] = *r0;
> -		if (bpf_ret_reg_pair(env, callee->subprogno)) {
> +		if (bpf_ret_reg_pair_mark(env, callee->subprogno)) {

Also, doing this while processing exit is a bit strange.
I think this should be done in do_check_common().

>  			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
>  				verbose(env, "cannot return stack pointer to the caller\n");
>  				return -EINVAL;

[...]

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

[Eduard Zingerman]

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
> btf_distill_func_proto() builds the function model used for kfunc calls,
> the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following
> patch relaxes it to accept a >8 and <=16 byte return value, which for the
> BPF target is passed back in the R0:R2 register pair.
> 
> But the BPF trampoline cannot preserve such a return. It saves and restores
> only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half
> (RDX / R2) is not preserved across a program that observes the return
> value. A program attached to a function returning a 16-byte value (e.g.
> current_time() or ns_to_timespec64(), which return struct timespec64)
> would corrupt the value seen by the real caller and read a partial return
> value itself. struct_ops trampolines have the same limitation.
> 
> This only matters for the attach types that actually read the target's
> return value: fexit, fmod_ret and fsession (plus the _multi variants of
> fexit and fsession). fentry and fentry_multi run before the target returns
> and never touch the return value, so they can attach to a >8 byte-returning
> function safely.
> 
> Reject a >8 byte return value for the return-reading attach types in
> bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for
> struct_ops in bpf_struct_ops_desc_init(), ahead of the following patch that
> would otherwise let such a return through. kfunc and BPF-to-BPF subprogram
> calls, which the JIT does handle, are unaffected.
> 
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> ---

Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>

Maybe post this independently from the current patch-set?
Seem to be a standalone fix.

Re: [PATCH bpf-next 03/12] bpf: Wire up JIT support for 16-byte kfunc returns

[Eduard Zingerman]

On Wed, 2026-07-08 at 13:09 -0700, Yonghong Song wrote:
> Continue preparing <=16 byte aggregate return support by wiring up the JIT
> side of the R0:R2 return convention for kfuncs.
> 
> A kfunc returning more than 8 bytes (a small aggregate or an __int128)
> hands the second half of the result back in RDX, the native x86-64 ABI's
> second return register. BPF instead expects a 16-byte return in the
> R0:R2 register pair. BPF R0 maps to RAX so it needs no move, but BPF R2
> maps to RSI, so emit a RDX->RSI move after a BPF_PSEUDO_KFUNC_CALL whose
> function model reports ret_size > 8.
> 
> Placing the second return half into R2 is possible on any JIT, but it
> needs architecture-specific JIT work. Rather than requiring every JIT to
> implement it at once, add a bpf_jit_supports_kfunc_ret_reg_pair()
> capability, defaulting to false in the generic core; an architecture opts
> in once its JIT handles the R0:R2 pair, and the remaining ones are left
> for future work. The verifier enforces it in bpf_add_kfunc_call(),
> rejecting a kfunc whose return is larger than 8 bytes with -EOPNOTSUPP
> when the JIT lacks the capability. Only x86, arm64 and riscv are
> supported so far.
> 
> On arm64 and riscv the native second return register is already BPF R2
> (x1 in bpf2a64[] and a1 in regmap[] respectively), so the value is in the
> R0:R2 register pair on return with no extra move, unlike x86 (RDX->RSI).
> This has been tested on x86 and arm64. The riscv path is expected to work
> by the same register-mapping reasoning as arm64 but has not been tested.
> 
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> ---

Acked-by: Eduard Zingerman <eddyz87@gmail.com>

[...]

Re: [PATCH bpf-next 04/12] bpf: Track R2 of register-pair returns in precision backtracking

[Eduard Zingerman]

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
> A function returning a value larger than 8 bytes (a struct/union, or an
> __int128) uses R2 as a second return register alongside R0. Previous
> precision backtracking only treated R0 as a return register at a
> call/return boundary, so marking the second half of such a return precise
> would trip the "unexpected regs" checks in backtrack_insn() and reject a
> valid program with -EFAULT.
> 
> Handle R2 like R0 in the three boundaries where a call defines the return
> registers:
> 
>  - static subprog exit (BPF_EXIT): R2 is a return register, not a leftover
>    argument; clear it before the args check and propagate its precision
>    into the callee.
>  - global subprog call: a global subprog returning >8 bytes also sets R2;
>    clear it before the args check.
>  - kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8)
>    also sets R2; clear it like R0.
> 
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> ---

Acked-by: Eduard Zingerman <eddyz87@gmail.com>

> diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
> index 2e4ae0ef0860..c46c2778bf60 100644
> --- a/kernel/bpf/backtrack.c
> +++ b/kernel/bpf/backtrack.c
> @@ -424,6 +424,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
>  				 */
>  				verifier_bug_if(idx + 1 != subseq_idx, env,
>  						"extra insn from subprog");
> +				if (bpf_ret_reg_pair(env, subprog))
> +					bt_clear_reg(bt, BPF_REG_2);

Nit: please move this a few lines below, so that r0 and r2 are handled nearby.

>  				/* r1-r5 are invalidated after subprog call,
>  				 * so for global func call it shouldn't be set
>  				 * anymore
> @@ -507,6 +509,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
>  				return -ENOTSUPP;
>  			/* regular helper call sets R0 */
>  			bt_clear_reg(bt, BPF_REG_0);
> +			/* On error the return is negative, R2 stays set, and the
> +			 * check below catches it as an unexpected register.
> +			 */

Nit: I don't understand the above comment (the code is fine). Maybe drop it altogether?

> +			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
> +			    bpf_get_kfunc_ret_size(env->prog, insn->imm, insn->off) > 8)
> +				bt_clear_reg(bt, BPF_REG_2);
>  			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
>  				/* if backtracking was looking for registers R1-R5
>  				 * they should have been found already.
> @@ -521,7 +529,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
>  					return -EFAULT;
>  			}
>  		} else if (opcode == BPF_EXIT) {
> -			bool r0_precise;
> +			bool r0_precise, r2_precise = false;
> +
> +			/* A subprog returning a value larger than 8 bytes (a
> +			 * struct/union or an __int128) uses R2 as a second
> +			 * return register alongside R0. When exiting such a
> +			 * subprog, R2 is a return register rather than a
> +			 * leftover argument, so handle it like R0 below.
> +			 */

Nit: the comment is too verbose.

> +			if (subseq_idx - 1 >= 0 &&
> +			    bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1])) {
> +				int call_idx = subseq_idx - 1;
> +				int tgt = call_idx + env->prog->insnsi[call_idx].imm + 1;
> +				int subprog = bpf_find_subprog(env, tgt);
> +
> +				if (subprog >= 0 &&
> +				    bpf_ret_reg_pair(env, subprog) &&
> +				    bt_is_reg_set(bt, BPF_REG_2))
> +					r2_precise = true;
> +			}
>  
>  			/* Backtracking to a nested function call, 'idx' is a part of
>  			 * the inner frame 'subseq_idx' is a part of the outer frame.

[...]

Re: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types

[Eduard Zingerman]

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
> Relax btf_distill_func_proto() to accept a by-value struct or union that
> the R0:R2 convention added in earlier patches can carry:
> 
>  - a struct or union larger than 8 and up to 16 bytes, returned in the
>    R0:R2 register pair, matching what LLVM emits for the BPF target;
>  - a struct or union up to 8 bytes, returned in R0 alone.
> 
> A >8 byte scalar (__int128) was already accepted and is unchanged.
> Everything else stays rejected: a return type larger than 16 bytes, and any
> type that __get_type_size() cannot return in registers at all (e.g. an
> array), which it already reports as ret < 0.
> 
> btf_distill_func_proto() also builds the trampoline (fentry/fexit/fmod_ret)
> and struct_ops function models. Allowing up to 16 bytes here is safe: the
> previous patch already rejects >8 byte returns on the return-reading paths,
> where the trampoline preserves only 8 bytes of the return value.
> 
> btf_validate_return_type() is relaxed as it accepts a by-value struct or
> union up to 16 bytes in addition to void and scalars.
> 
> With btf_distill_func_proto() and btf_validate_return_type() relaxed, the
> verifier, JIT, precision-backtracking, live-register and trampoline-guard
> preparation from the earlier patches becomes reachable: this final patch
> enables <=16 byte aggregate return values end to end.
> 
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> ---

Acked-by: Eduard Zingerman <eddyz87@gmail.com>

>  include/linux/bpf_verifier.h                       |  2 ++
>  kernel/bpf/btf.c                                   | 14 +++++++++++---
>  kernel/bpf/verifier.c                              |  6 +++---
>  .../testing/selftests/bpf/progs/exceptions_fail.c  |  2 +-
>  4 files changed, 17 insertions(+), 7 deletions(-)
> 
> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
> index fa255f326240..ae4dae8e57f9 100644
> --- a/include/linux/bpf_verifier.h
> +++ b/include/linux/bpf_verifier.h
> @@ -1433,6 +1433,8 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
>  bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
>  bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog);
>  int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset);
> +bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
> +				 const struct btf_type *t, int rec);

Nit: maybe remove the "__" prefix?

>  
>  int bpf_find_subprog(struct bpf_verifier_env *env, int off);
>  bool bpf_is_throw_kfunc(struct bpf_insn *insn);
> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> index dff5c0d91641..bab82d1c8cb9 100644
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c
> @@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
>  		return -EINVAL;
>  	}
>  	ret = __get_type_size(btf, func->type, &t);
> -	if (ret < 0 || btf_type_is_struct(t)) {
> +	/*
> +	 * __get_type_size() already restricts a non-negative ret to void, a
> +	 * pointer, an int, an enum or a struct/union, so only the size is checked
> +	 * here.
> +	 */
> +	if (ret < 0 || ret > 16) {
>  		bpf_log(log,
>  			"The function %s return type %s is unsupported.\n",
>  			tname, btf_type_str(t));
> @@ -7968,6 +7973,9 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
>  	if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
>  		return 0;
>  
> +	if (btf_type_is_struct(t) && t->size <= 16 && __btf_type_is_scalar_struct(env, btf, t, 0))

Nit:	if (__btf_type_is_scalar_struct(env, btf, t, 0) && t->size <= 16)

> +		return 0;
> +
>  	return -EOPNOTSUPP;
>  }
>  

[...]

Re: [PATCH bpf-next 00/12] bpf: Support 16-byte return values in the R0:R2 register pair

[Eduard Zingerman]

> On Wed, 2026-07-08 at 13:09 -0700, Yonghong Song wrote:

A design question regarding to how this series handles local
functions:
- It relies on the BTF gate to decide if function returns multiple values.
- It also prohibits local functions from returning pointers as a part
  of an aggregate return.

I think this unnecessarily adds an additional discrepancy between how
the return values are handled. The practical experience shows that
existing differences between local and global functions are already a
constant source of pain and confusion.

I'd also argue that lifting the above limitations for local functions
would make the implementation somewhat simpler.

Therefore, I suggest to:
- forgo the BTF check for local functions;
- always copy r2 from callee to caller at EXIT
- forgo the scalar struct requirement for local function
  return types.

Wdyt?

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

[Eduard Zingerman]

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
> Add selftests that exercise a 16-byte return value passed in the R0:R2
> register pair, expressed in C so they rely on the compiler lowering the
> register-pair return. Tests include an __int128 return (from a subprogram
> and a kfunc), a 16-byte struct return (from static and global subprograms
> and a kfunc) and a 16-byte union return.
> 
> The R0:R2 return convention is only emitted by LLVM 23 and newer, so the
> programs are gated on __clang_major__ >= 23; with an older compiler they
> return a sentinel and the "int128_c", "struct_c", "global_struct_c",
> "kfunc_struct_c" and "union_c" subtests are skipped.
> 
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> ---

[...]

> +static void run_prog(struct bpf_program *prog, bool may_skip)
> +{
> +	char buf[64] = {};
> +	LIBBPF_OPTS(bpf_test_run_opts, topts,
> +		.data_in = buf,
> +		.data_size_in = sizeof(buf),
> +		.repeat = 1,
> +	);
> +	int err;
> +
> +	err = bpf_prog_test_run_opts(bpf_program__fd(prog), &topts);
> +	if (!ASSERT_OK(err, "test_run"))
> +		return;
> +
> +	if (may_skip && (int)topts.retval == -1) {
> +		test__skip();
> +		return;
> +	}
> +
> +	ASSERT_EQ(topts.retval, 0, "aggregate_ret_result");
> +}
> +
> +void test_aggregate_ret(void)
> +{

The test_aggregate_ret()/run_prog() duplicate functionality provided
by the __retval tag from test_loader.c. To achieve the same skipping
mechanics I suggest to:
- extend test_loader.c / bpf_misc.h with a new tag: __skip("reason")
- use this tag as:

  #ifdef __clang_major__ >= 23

  SEC("tc")
  int aggregate_ret_test(struct __sk_buff *skb) { ... }

  #else

  SEC("tc") __skip("... need clang23 ...")
  int aggregate_ret_test(struct __sk_buff *skb) { ... }

  #endif

[...]

> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c ...
> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c ...
> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c ...

Nit: all three files can be merged into one.

[...]

Re: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

[Eduard Zingerman]

On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
> Add inline-asm and subprogram tests for more coverage, including
> BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive
> and negative tests are added.
> 
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> ---

[...]

> @@ -93,4 +97,25 @@ void test_aggregate_ret(void)
>  		run_prog(skel_union_c->progs.aggregate_ret_union_c_test, true);
>  
>  	aggregate_ret_union_c__destroy(skel_union_c);
> +
> +	skel_run = aggregate_ret_run__open_and_load();
> +	if (!ASSERT_OK_PTR(skel_run, "skel_run_open_load"))
> +		return;
> +
> +	/* Inline-asm variant: compiler-version independent, always runs. */
> +	if (test__start_subtest("asm"))
> +		run_prog(skel_run->progs.aggregate_ret_asm_test, false);
> +
> +	/* Struct-by-value kfunc returns, also compiler-version independent. */
> +	if (test__start_subtest("struct"))
> +		run_prog(skel_run->progs.aggregate_ret_struct_test, false);
> +
> +	/* Union-by-value kfunc return, compiler-version independent. */
> +	if (test__start_subtest("union"))
> +		run_prog(skel_run->progs.aggregate_ret_union_test, false);
> +
> +	aggregate_ret_run__destroy(skel_run);
> +
> +	RUN_TESTS(aggregate_ret_func);
> +	RUN_TESTS(aggregate_ret_kfunc);

This can be folded into tools/testing/selftests/bpf/prog_tests/verifier.c.

>  }
> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
> new file mode 100644
> index 000000000000..31e176103374
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
> @@ -0,0 +1,361 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
> +#include <linux/bpf.h>
> +#include <bpf/bpf_helpers.h>
> +#include "bpf_misc.h"
> +
> +typedef unsigned __int128 u128;
> +
> +__naked u128 global_agg_good(void)
> +{
> +	asm volatile (
> +	"r0 = 0x1234;"	/* low 64 bits */
> +	"r2 = 0x5678;"	/* high 64 bits */
> +	"exit;"
> +	);
> +}
> +
> +__naked u128 global_agg_bad(void)
> +{
> +	asm volatile (
> +	"r0 = 0;"
> +	"exit;"
> +	);
> +}

C version of tests from a previous patch already checks global vs.
local functions. I suggest dropping that version from C tests and keep
in assembly only to reduce the duplication.

[...]

> +SEC("tc")
> +__success __retval(0)
> +int aggregate_ret_global_union(void *ctx)
> +{
> +	__u64 lo, hi;
> +
> +	asm volatile (
> +	"call %[global_ret_union];"
> +	"%[lo] = r0;"
> +	"%[hi] = r2;"
> +	: [lo]"=r"(lo), [hi]"=r"(hi)
> +	: __imm(global_ret_union)
> +	: "r0", "r1", "r2", "r3", "r4", "r5");
> +	if (lo != 0x1234)
> +		return 1;
> +	if (hi != 0x5678)
> +		return 2;
> +	return 0;
> +}

And here I'd only keep the C version.

> +__naked struct with_ptr global_ret_struct_ptr(void)
> +{
> +	asm volatile (
> +	"r0 = 0;"
> +	"r2 = 0;"
> +	"exit;"
> +	);
> +}

[...]

> +static __naked u128 agg_callee(void)
> +{
> +	asm volatile (
> +	"r0 = 1;"
> +	"r2 = 2;"
> +	"exit;"
> +	);
> +}
> +
> +SEC("tc")
> +__log_level(2)
> +__msg("Live regs before insn:")
> +__msg("0: .12345.... (85) call pc+2")
> +__msg("1: ..2....... (bf) r0 = r2")
> +__msg("2: 0......... (95) exit")
> +__msg("3: .......... (b7) r0 = 1")
> +__msg("4: 0......... (b7) r2 = 2")
> +__msg("5: 0.2....... (95) exit")
> +__naked int aggregate_ret_live(void)
> +{
> +	asm volatile (
> +	"call %[agg_callee];"
> +	"r0 = r2;"
> +	"exit;"
> +	:
> +	: [agg_callee]"i"(agg_callee)
> +	: __clobber_all);
> +}

Nit: usually we add such tests to compute_live_registers.c.

[...]

> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_run.c b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c

I think the test cases in this file are by a large degree redundant.
A single test case calling a kfunc returning a small struct and
checking returned values would suffice.

[...]

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

[Leon Hwang]

On 10/7/26 00:52, Yonghong Song wrote:
> 
> 
> On 7/8/26 8:16 PM, Leon Hwang wrote:
>> On 9/7/26 04:10, Yonghong Song wrote:
>>> btf_distill_func_proto() builds the function model used for kfunc calls,
>>> the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A
>>> following
>>> patch relaxes it to accept a >8 and <=16 byte return value, which for
>>> the
>>> BPF target is passed back in the R0:R2 register pair.
>>
>> Do you have a plan to add >8 byte return value support for them?
> 
> I tested with vmlinux BTF built from the latest bpf-next. I got (from
> vmlinux BTF):
> 
>    total number of functions: 67500
>    return type size (> 8 && <= 16): 51
>    return type size (> 16): 10
> 
> So in the initial version, I didn't support >8 byte for trampoline related
> bpf programs.
> 
> But I can certainly do this, we just need to allocate another 8 byte for
> return value
> in trampoline.
> 

The following support should be mentioned in the cover letter.

Thanks,
Leon

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

[Leon Hwang]

On 10/7/26 07:08, Eduard Zingerman wrote:
> On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
>> btf_distill_func_proto() builds the function model used for kfunc calls,
>> the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following
>> patch relaxes it to accept a >8 and <=16 byte return value, which for the
>> BPF target is passed back in the R0:R2 register pair.
>>
>> But the BPF trampoline cannot preserve such a return. It saves and restores
>> only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half
>> (RDX / R2) is not preserved across a program that observes the return
>> value. A program attached to a function returning a 16-byte value (e.g.
>> current_time() or ns_to_timespec64(), which return struct timespec64)
>> would corrupt the value seen by the real caller and read a partial return
>> value itself. struct_ops trampolines have the same limitation.
>>
>> This only matters for the attach types that actually read the target's
>> return value: fexit, fmod_ret and fsession (plus the _multi variants of
>> fexit and fsession). fentry and fentry_multi run before the target returns
>> and never touch the return value, so they can attach to a >8 byte-returning
>> function safely.
>>
>> Reject a >8 byte return value for the return-reading attach types in
>> bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for
>> struct_ops in bpf_struct_ops_desc_init(), ahead of the following patch that
>> would otherwise let such a return through. kfunc and BPF-to-BPF subprogram
>> calls, which the JIT does handle, are unaffected.
>>
>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>> ---
> 
> Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
> 
> Maybe post this independently from the current patch-set?
> Seem to be a standalone fix.
> 


Agreed.

A Fixes tag should be added.

Thanks,
Leon

Re: [PATCH bpf-next 02/12] bpf: Add verifier support for 16-byte returns in R0:R2

[Yonghong Song]

On 7/9/26 3:21 PM, Eduard Zingerman wrote:
> On Wed, 2026-07-08 at 13:09 -0700, Yonghong Song wrote:
>> LLVM 23 added support for returning a value in two registers for an
>> __int128, or a struct/union whose size is greater than 8 but not more
>> than 16 bytes. See LLVM patches [1] and [2].
>>
>> Before LLVM 23 the BPF backend could not return these values at all. A
>> by-value struct or union return (of any size) was rejected at compile
>> time with:
>>
>>    error: aggregate returns are not supported
>>
>> and an __int128 return failed later in the backend with:
>>
>>    fatal error: error in backend: unable to allocate function return #1
>>
>> Both are resolved in LLVM 23, which lowers such returns into the R0:R2
>> register pair.
>>
>> This patch is one of several preparation patches that build up
>> <=16 byte return support incrementally:
>>    - the verifier R0:R2 modelling in this patch,
>>    - the x86 JIT move and JIT-capability gate,
>>    - precision-backtracking and live-register tracking of R2,
>>    - forcing the JIT for such programs, and
>>    - guarding the trampoline paths.
>> Finally, the patch "bpf: Enable 16-byte aggregate return types" enables
>> support for 16-byte returns.
> Nit: I think the above paragraphs belong to a cover letter.

Will add this to the cover letter.

>
>> This patch adds handling for '>8' byte returns in several places: BPF
>> subprogram returns (the main program, and both global and static
>> subprograms) and kfunc returns.
>>
>>    [1] https://github.com/llvm/llvm-project/pull/190894
>>    [2] https://github.com/llvm/llvm-project/pull/206876
>>
>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>> ---
>>   kernel/bpf/verifier.c | 93 ++++++++++++++++++++++++++++++++++++++++---
>>   1 file changed, 88 insertions(+), 5 deletions(-)
>>
>> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
>> index 3f34ce1a3107..03ffb5f839fa 100644
>> --- a/kernel/bpf/verifier.c
>> +++ b/kernel/bpf/verifier.c
>> @@ -424,6 +424,35 @@ static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
>>   	return btf_type_is_void(type);
>>   }
>>   
>> +static bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
>> +{
>> +	const struct btf_type *type, *func, *func_proto;
>> +	const struct btf *btf = env->prog->aux->btf;
>> +	u32 btf_id;
>> +
>> +	if (!btf || !env->prog->aux->func_info)
>> +		return false;
>> +
>> +	btf_id = env->prog->aux->func_info[subprog].type_id;
>> +
>> +	func = btf_type_by_id(btf, btf_id);
>> +	if (!func)
>> +		return false;
> `if (!func) return false` checks here and below are not necessary
> after BTF validation.

Ack. Will remove them.

>
>> +
>> +	func_proto = btf_type_by_id(btf, func->type);
>> +	if (!func_proto)
>> +		return false;
>> +
>> +	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
>> +	if (!type)
>> +		return false;
>> +
>> +	if (btf_type_is_struct(type) || btf_type_is_scalar(type))
>> +		return type->size > 8 && type->size <= 16;
>> +
>> +	return false;
>> +}
>> +
>>   static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
>>   {
>>   	struct bpf_func_info *info;
>> @@ -9459,10 +9488,17 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
>>   		clear_caller_saved_regs(env, caller->regs);
>>   		invalidate_outgoing_stack_args(env, cur_func(env));
>>   
>> -		/* All non-void global functions return a 64-bit SCALAR_VALUE. */
>> +		/*
>> +		 * A non-void global function returns a 64-bit SCALAR_VALUE in
>> +		 * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
>> +		 */
>>   		if (!subprog_returns_void(env, subprog)) {
>>   			mark_reg_unknown(env, caller->regs, BPF_REG_0);
>>   			caller->regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG;
>> +			if (bpf_ret_reg_pair(env, subprog)) {
>> +				mark_reg_unknown(env, caller->regs, BPF_REG_2);
>> +				caller->regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
>> +			}
>>   		}
>>   
>>   		if (env->subprog_info[subprog].might_throw) {
>> @@ -9825,6 +9861,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
>>   	} else {
>>   		/* return to the caller whatever r0 had in the callee */
>>   		caller->regs[BPF_REG_0] = *r0;
>> +		if (bpf_ret_reg_pair(env, callee->subprogno)) {
>> +			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
> Let's consolidate all such checks in one place and not copy-paste them.

Will try to consolidate them together.

>
>> +				verbose(env, "cannot return stack pointer to the caller\n");
>> +				return -EINVAL;
>> +			}
>> +			caller->regs[BPF_REG_2] = callee->regs[BPF_REG_2];
>> +		}
>>   	}
>>   
>>   	/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
>> @@ -10745,6 +10788,18 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno,
>>   	return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size);
>>   }
>>   
>> +static void mark_kfunc_ret_reg_size(struct bpf_verifier_env *env,
>> +				    struct bpf_reg_state *regs, u32 size)
>> +{
>> +	if (size > 8) {
>> +		mark_btf_func_reg_size(env, BPF_REG_0, 8);
>> +		mark_reg_unknown(env, regs, BPF_REG_2);
>> +		regs[BPF_REG_2].subreg_def = DEF_NOT_SUBREG;
>> +	} else {
>> +		mark_btf_func_reg_size(env, BPF_REG_0, size);
>> +	}
>> +}
> This function makes R2 handling very asymmetric compared to R0:
> - mark_reg_unknown() for r0 is done in check_kfunc_call()
> - DEF_NOT_SUBREG for r0 is done in mark_btf_func_reg_size()
>
> I think the code should be structured to keep R0 and R2 processing
> uniform.

Ack

>
>> +
>>   static bool is_kfunc_acquire(struct bpf_kfunc_call_arg_meta *meta)
>>   {
>>   	return meta->kfunc_flags & KF_ACQUIRE;
>> @@ -13208,7 +13263,22 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
>>   		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
>>   		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
>>   			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
>> -		mark_btf_func_reg_size(env, BPF_REG_0, t->size);
>> +		mark_kfunc_ret_reg_size(env, regs, t->size);
>> +	} else if (btf_type_is_struct(t)) {
>> +		/*
>> +		 * The returned struct comes back as raw register bits modeled
>> +		 * as an unknown scalar, so it must contain only scalars:
>> +		 * otherwise a pointer field would be laundered into a scalar
>> +		 * and escape provenance and reference tracking.
>> +		 */
>> +		if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
>> +			verbose(env, "kernel function %s returns %s %s that is not composed of scalars\n",
>> +				func_name, btf_type_str(t),
>> +				btf_name_by_offset(desc_btf, t->name_off));
> Should this also check the size of the struct?

Later on, there is a size condition for struct (<= 16), so we should be okay here.

>
>> +			return -EINVAL;
>> +		}
>> +		mark_reg_unknown(env, regs, BPF_REG_0);
>> +		mark_kfunc_ret_reg_size(env, regs, t->size);
>>   	} else if (btf_type_is_ptr(t)) {
>>   		ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
>>   		err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
> [...]

Re: [PATCH bpf-next 05/12] bpf: Account R2 of register-pair returns in live register analysis

[Yonghong Song]

On 7/9/26 2:07 PM, Eduard Zingerman wrote:
> On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
>> A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a
>> struct/union or an __int128) reads R2 as well as R0, since the second half
>> of the return value is passed back in R2. compute_insn_live_regs() only
>> marked R0 used at exit, so the callee's R2 could be considered dead and
>> cleaned from checkpointed states, which would allow unsound state pruning.
>>
>> Mark R2 used at a BPF_EXIT when the enclosing subprogram returns via the
>> R0:R2 register pair. The call site already treats the caller-saved
>> registers R0-R5 as clobbered, so R2 as a result register is covered there.
>>
>> For example, for a subprogram returning an __int128, the live-registers
>> dump (log level 2) shows both R0 and R2 live before the callee's exit
>> (insn 5):
>>
>>    Live regs before insn:
>>          0: .12345.... (85) call pc+2
>>          1: ..2....... (bf) r0 = r2
>>          2: 0......... (95) exit
>>          3: .......... (b7) r0 = 1
>>          4: 0......... (b7) r2 = 2
>>          5: 0.2....... (95) exit
>>
>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>> ---
>>   kernel/bpf/liveness.c | 13 ++++++++++---
>>   1 file changed, 10 insertions(+), 3 deletions(-)
>>
>> diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
>> index 0aadfbae0acc..0977d32fcf01 100644
>> --- a/kernel/bpf/liveness.c
>> +++ b/kernel/bpf/liveness.c
>> @@ -2062,7 +2062,8 @@ struct insn_live_regs {
>>   /* Compute info->{use,def} fields for the instruction */
>>   static void compute_insn_live_regs(struct bpf_verifier_env *env,
>>   				   struct bpf_insn *insn,
>> -				   struct insn_live_regs *info)
>> +				   struct insn_live_regs *info,
>> +				   int subprog)
>>   {
>>   	struct bpf_call_summary cs;
>>   	u8 class = BPF_CLASS(insn->code);
>> @@ -2175,6 +2176,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
>>   		case BPF_EXIT:
>>   			def = 0;
>>   			use = r0;
>> +			if (bpf_ret_reg_pair(env, subprog))
>> +				use |= BIT(BPF_REG_2);
>>   			break;
>>   		case BPF_CALL:
>>   			def = ALL_CALLER_SAVED_REGS;
>> @@ -2237,8 +2240,12 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
>>   		goto out;
>>   	}
>>   
>> -	for (i = 0; i < insn_cnt; ++i)
>> -		compute_insn_live_regs(env, &insns[i], &state[i]);
>> +	for (i = 0, j = 0; i < insn_cnt; ++i) {
>> +		/* Advance to the subprog that contains instruction i. */
>> +		while (j + 1 < env->subprog_cnt && env->subprog_info[j + 1].start <= i)
>> +			j++;
> This loop is very in-elegant. The outer loop can be changed to iterate
> subprograms, with inner loop iterating instructions. Or just use
> bpf_find_subprog() when processing EXIT.

Ack. I prefer to have iterating subprogram in outer loop and iterating insns
in the inner loop.

>
>> +		compute_insn_live_regs(env, &insns[i], &state[i], j);
>> +	}
>>   
>>   	/* Forward pass: resolve stack access through FP-derived pointers */
>>   	err = bpf_compute_subprog_arg_access(env);

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

[Yonghong Song]

On 7/9/26 1:30 PM, Eduard Zingerman wrote:
> On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
>
> [...]
>
>> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
>> index 73dc3ee879de..819a97d25173 100644
>> --- a/kernel/bpf/core.c
>> +++ b/kernel/bpf/core.c
>> @@ -2676,7 +2676,7 @@ struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct
>>   		goto finalize;
>>   
>>   	if (IS_ENABLED(CONFIG_BPF_JIT_ALWAYS_ON) ||
>> -	    bpf_prog_has_kfunc_call(fp))
>> +	    bpf_prog_has_kfunc_call(fp) || fp->aux->ret_reg_pair)
> This has a conflict with one of the very recent bpf-next commits.
> While reading locally I fixed the conflict by removing this hunk
> and adjusting bpf_ret_reg_pair_mark() as below.

Indeed, once jit_required is available in fp->aux, I can remove
ret_reg_pair things.

>
>>   		jit_needed = true;
>>   
>>   	if (!bpf_prog_select_interpreter(fp))
>> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
>> index fcbf532159a5..ca8b2f436c20 100644
>> --- a/kernel/bpf/verifier.c
>> +++ b/kernel/bpf/verifier.c
>> @@ -453,6 +453,15 @@ bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
>>   	return false;
>>   }
>>   
>> +static bool bpf_ret_reg_pair_mark(struct bpf_verifier_env *env, int subprog)
>> +{
>> +	if (!bpf_ret_reg_pair(env, subprog))
>> +		return false;
>> +
>> +	env->prog->aux->ret_reg_pair = true;
> Added:
> 	env->prog->jit_required = true;
>
>> +	return true;
>> +}
>> +
> You probably would want to do the same in v2.
>
> [...]

Re: [PATCH bpf-next 06/12] bpf: Force JIT for programs using the R0:R2 register pair

[Yonghong Song]

On 7/9/26 3:27 PM, Eduard Zingerman wrote:
> On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
>
> [...]
>
>> @@ -9877,7 +9886,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
>>   	} else {
>>   		/* return to the caller whatever r0 had in the callee */
>>   		caller->regs[BPF_REG_0] = *r0;
>> -		if (bpf_ret_reg_pair(env, callee->subprogno)) {
>> +		if (bpf_ret_reg_pair_mark(env, callee->subprogno)) {
> Also, doing this while processing exit is a bit strange.
> I think this should be done in do_check_common().

Before sending this patch, I found two places to do this, prepare_func_exit() and
do_check_common(). Since you mentioned, I will do do_check_common() then.

>
>>   			if (callee->regs[BPF_REG_2].type == PTR_TO_STACK) {
>>   				verbose(env, "cannot return stack pointer to the caller\n");
>>   				return -EINVAL;
> [...]

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

[Yonghong Song]

On 7/9/26 4:08 PM, Eduard Zingerman wrote:
> On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
>> btf_distill_func_proto() builds the function model used for kfunc calls,
>> the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following
>> patch relaxes it to accept a >8 and <=16 byte return value, which for the
>> BPF target is passed back in the R0:R2 register pair.
>>
>> But the BPF trampoline cannot preserve such a return. It saves and restores
>> only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half
>> (RDX / R2) is not preserved across a program that observes the return
>> value. A program attached to a function returning a 16-byte value (e.g.
>> current_time() or ns_to_timespec64(), which return struct timespec64)
>> would corrupt the value seen by the real caller and read a partial return
>> value itself. struct_ops trampolines have the same limitation.
>>
>> This only matters for the attach types that actually read the target's
>> return value: fexit, fmod_ret and fsession (plus the _multi variants of
>> fexit and fsession). fentry and fentry_multi run before the target returns
>> and never touch the return value, so they can attach to a >8 byte-returning
>> function safely.
>>
>> Reject a >8 byte return value for the return-reading attach types in
>> bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for
>> struct_ops in bpf_struct_ops_desc_init(), ahead of the following patch that
>> would otherwise let such a return through. kfunc and BPF-to-BPF subprogram
>> calls, which the JIT does handle, are unaffected.
>>
>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>> ---
> Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
>
> Maybe post this independently from the current patch-set?
> Seem to be a standalone fix.

Okay, I will submit this patch separately.

Re: [PATCH bpf-next 07/12] bpf: Reject >8 byte return values on return-reading trampoline paths

[Yonghong Song]

On 7/9/26 7:02 PM, Leon Hwang wrote:
> On 10/7/26 07:08, Eduard Zingerman wrote:
>> On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
>>> btf_distill_func_proto() builds the function model used for kfunc calls,
>>> the fentry/fexit/fmod_ret/fsession trampolines and struct_ops. A following
>>> patch relaxes it to accept a >8 and <=16 byte return value, which for the
>>> BPF target is passed back in the R0:R2 register pair.
>>>
>>> But the BPF trampoline cannot preserve such a return. It saves and restores
>>> only 8 bytes of the return value (RAX on x86, i.e. R0), so the second half
>>> (RDX / R2) is not preserved across a program that observes the return
>>> value. A program attached to a function returning a 16-byte value (e.g.
>>> current_time() or ns_to_timespec64(), which return struct timespec64)
>>> would corrupt the value seen by the real caller and read a partial return
>>> value itself. struct_ops trampolines have the same limitation.
>>>
>>> This only matters for the attach types that actually read the target's
>>> return value: fexit, fmod_ret and fsession (plus the _multi variants of
>>> fexit and fsession). fentry and fentry_multi run before the target returns
>>> and never touch the return value, so they can attach to a >8 byte-returning
>>> function safely.
>>>
>>> Reject a >8 byte return value for the return-reading attach types in
>>> bpf_check_attach_target() and bpf_check_attach_btf_id_multi(), and for
>>> struct_ops in bpf_struct_ops_desc_init(), ahead of the following patch that
>>> would otherwise let such a return through. kfunc and BPF-to-BPF subprogram
>>> calls, which the JIT does handle, are unaffected.
>>>
>>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>>> ---
>> Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
>>
>> Maybe post this independently from the current patch-set?
>> Seem to be a standalone fix.
>>
>
> Agreed.
>
> A Fixes tag should be added.

Ack.

>
> Thanks,
> Leon
>

Re: [PATCH bpf-next 08/12] bpf: Enable 16-byte aggregate return types

[Yonghong Song]

On 7/9/26 5:45 PM, Eduard Zingerman wrote:
> On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
>> Relax btf_distill_func_proto() to accept a by-value struct or union that
>> the R0:R2 convention added in earlier patches can carry:
>>
>>   - a struct or union larger than 8 and up to 16 bytes, returned in the
>>     R0:R2 register pair, matching what LLVM emits for the BPF target;
>>   - a struct or union up to 8 bytes, returned in R0 alone.
>>
>> A >8 byte scalar (__int128) was already accepted and is unchanged.
>> Everything else stays rejected: a return type larger than 16 bytes, and any
>> type that __get_type_size() cannot return in registers at all (e.g. an
>> array), which it already reports as ret < 0.
>>
>> btf_distill_func_proto() also builds the trampoline (fentry/fexit/fmod_ret)
>> and struct_ops function models. Allowing up to 16 bytes here is safe: the
>> previous patch already rejects >8 byte returns on the return-reading paths,
>> where the trampoline preserves only 8 bytes of the return value.
>>
>> btf_validate_return_type() is relaxed as it accepts a by-value struct or
>> union up to 16 bytes in addition to void and scalars.
>>
>> With btf_distill_func_proto() and btf_validate_return_type() relaxed, the
>> verifier, JIT, precision-backtracking, live-register and trampoline-guard
>> preparation from the earlier patches becomes reachable: this final patch
>> enables <=16 byte aggregate return values end to end.
>>
>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>> ---
> Acked-by: Eduard Zingerman <eddyz87@gmail.com>
>
>>   include/linux/bpf_verifier.h                       |  2 ++
>>   kernel/bpf/btf.c                                   | 14 +++++++++++---
>>   kernel/bpf/verifier.c                              |  6 +++---
>>   .../testing/selftests/bpf/progs/exceptions_fail.c  |  2 +-
>>   4 files changed, 17 insertions(+), 7 deletions(-)
>>
>> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
>> index fa255f326240..ae4dae8e57f9 100644
>> --- a/include/linux/bpf_verifier.h
>> +++ b/include/linux/bpf_verifier.h
>> @@ -1433,6 +1433,8 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
>>   bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
>>   bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog);
>>   int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id, u16 offset);
>> +bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
>> +				 const struct btf_type *t, int rec);
> Nit: maybe remove the "__" prefix?

Good point. Since it becomes a global function, we probably should remove "__" prefix.

>
>>   
>>   int bpf_find_subprog(struct bpf_verifier_env *env, int off);
>>   bool bpf_is_throw_kfunc(struct bpf_insn *insn);
>> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
>> index dff5c0d91641..bab82d1c8cb9 100644
>> --- a/kernel/bpf/btf.c
>> +++ b/kernel/bpf/btf.c
>> @@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
>>   		return -EINVAL;
>>   	}
>>   	ret = __get_type_size(btf, func->type, &t);
>> -	if (ret < 0 || btf_type_is_struct(t)) {
>> +	/*
>> +	 * __get_type_size() already restricts a non-negative ret to void, a
>> +	 * pointer, an int, an enum or a struct/union, so only the size is checked
>> +	 * here.
>> +	 */
>> +	if (ret < 0 || ret > 16) {
>>   		bpf_log(log,
>>   			"The function %s return type %s is unsupported.\n",
>>   			tname, btf_type_str(t));
>> @@ -7968,6 +7973,9 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
>>   	if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
>>   		return 0;
>>   
>> +	if (btf_type_is_struct(t) && t->size <= 16 && __btf_type_is_scalar_struct(env, btf, t, 0))
> Nit:	if (__btf_type_is_scalar_struct(env, btf, t, 0) && t->size <= 16)

Yes, this is simpler.

>
>> +		return 0;
>> +
>>   	return -EOPNOTSUPP;
>>   }
>>   
> [...]

Re: [PATCH bpf-next 09/12] selftests/bpf: Add C tests for 16-byte returns in R0:R2

[Yonghong Song]

On 7/9/26 6:19 PM, Eduard Zingerman wrote:
> On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
>> Add selftests that exercise a 16-byte return value passed in the R0:R2
>> register pair, expressed in C so they rely on the compiler lowering the
>> register-pair return. Tests include an __int128 return (from a subprogram
>> and a kfunc), a 16-byte struct return (from static and global subprograms
>> and a kfunc) and a 16-byte union return.
>>
>> The R0:R2 return convention is only emitted by LLVM 23 and newer, so the
>> programs are gated on __clang_major__ >= 23; with an older compiler they
>> return a sentinel and the "int128_c", "struct_c", "global_struct_c",
>> "kfunc_struct_c" and "union_c" subtests are skipped.
>>
>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>> ---
> [...]
>
>> +static void run_prog(struct bpf_program *prog, bool may_skip)
>> +{
>> +	char buf[64] = {};
>> +	LIBBPF_OPTS(bpf_test_run_opts, topts,
>> +		.data_in = buf,
>> +		.data_size_in = sizeof(buf),
>> +		.repeat = 1,
>> +	);
>> +	int err;
>> +
>> +	err = bpf_prog_test_run_opts(bpf_program__fd(prog), &topts);
>> +	if (!ASSERT_OK(err, "test_run"))
>> +		return;
>> +
>> +	if (may_skip && (int)topts.retval == -1) {
>> +		test__skip();
>> +		return;
>> +	}
>> +
>> +	ASSERT_EQ(topts.retval, 0, "aggregate_ret_result");
>> +}
>> +
>> +void test_aggregate_ret(void)
>> +{
> The test_aggregate_ret()/run_prog() duplicate functionality provided
> by the __retval tag from test_loader.c. To achieve the same skipping
> mechanics I suggest to:
> - extend test_loader.c / bpf_misc.h with a new tag: __skip("reason")
> - use this tag as:
>
>    #ifdef __clang_major__ >= 23
>
>    SEC("tc")
>    int aggregate_ret_test(struct __sk_buff *skb) { ... }
>
>    #else
>
>    SEC("tc") __skip("... need clang23 ...")
>    int aggregate_ret_test(struct __sk_buff *skb) { ... }
>
>    #endif

Ack. We have quite some cases like this (guarded by compiler or feature flag,
with __skip makes things simpler.

>
> [...]
>
>> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c ...
>> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c ...
>> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c ...
> Nit: all three files can be merged into one.

Yes, I can merge them.

>
> [...]

Re: [PATCH bpf-next 10/12] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

[Yonghong Song]

On 7/9/26 6:38 PM, Eduard Zingerman wrote:
> On Wed, 2026-07-08 at 13:10 -0700, Yonghong Song wrote:
>> Add inline-asm and subprogram tests for more coverage, including
>> BPF-to-BPF cases, kfunc cases, backtracking and liveness. Both positive
>> and negative tests are added.
>>
>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>> ---
> [...]
>
>> @@ -93,4 +97,25 @@ void test_aggregate_ret(void)
>>   		run_prog(skel_union_c->progs.aggregate_ret_union_c_test, true);
>>   
>>   	aggregate_ret_union_c__destroy(skel_union_c);
>> +
>> +	skel_run = aggregate_ret_run__open_and_load();
>> +	if (!ASSERT_OK_PTR(skel_run, "skel_run_open_load"))
>> +		return;
>> +
>> +	/* Inline-asm variant: compiler-version independent, always runs. */
>> +	if (test__start_subtest("asm"))
>> +		run_prog(skel_run->progs.aggregate_ret_asm_test, false);
>> +
>> +	/* Struct-by-value kfunc returns, also compiler-version independent. */
>> +	if (test__start_subtest("struct"))
>> +		run_prog(skel_run->progs.aggregate_ret_struct_test, false);
>> +
>> +	/* Union-by-value kfunc return, compiler-version independent. */
>> +	if (test__start_subtest("union"))
>> +		run_prog(skel_run->progs.aggregate_ret_union_test, false);
>> +
>> +	aggregate_ret_run__destroy(skel_run);
>> +
>> +	RUN_TESTS(aggregate_ret_func);
>> +	RUN_TESTS(aggregate_ret_kfunc);
> This can be folded into tools/testing/selftests/bpf/prog_tests/verifier.c.
>
>>   }
>> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
>> new file mode 100644
>> index 000000000000..31e176103374
>> --- /dev/null
>> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
>> @@ -0,0 +1,361 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
>> +#include <linux/bpf.h>
>> +#include <bpf/bpf_helpers.h>
>> +#include "bpf_misc.h"
>> +
>> +typedef unsigned __int128 u128;
>> +
>> +__naked u128 global_agg_good(void)
>> +{
>> +	asm volatile (
>> +	"r0 = 0x1234;"	/* low 64 bits */
>> +	"r2 = 0x5678;"	/* high 64 bits */
>> +	"exit;"
>> +	);
>> +}
>> +
>> +__naked u128 global_agg_bad(void)
>> +{
>> +	asm volatile (
>> +	"r0 = 0;"
>> +	"exit;"
>> +	);
>> +}
> C version of tests from a previous patch already checks global vs.
> local functions. I suggest dropping that version from C tests and keep
> in assembly only to reduce the duplication.
>
> [...]
>
>> +SEC("tc")
>> +__success __retval(0)
>> +int aggregate_ret_global_union(void *ctx)
>> +{
>> +	__u64 lo, hi;
>> +
>> +	asm volatile (
>> +	"call %[global_ret_union];"
>> +	"%[lo] = r0;"
>> +	"%[hi] = r2;"
>> +	: [lo]"=r"(lo), [hi]"=r"(hi)
>> +	: __imm(global_ret_union)
>> +	: "r0", "r1", "r2", "r3", "r4", "r5");
>> +	if (lo != 0x1234)
>> +		return 1;
>> +	if (hi != 0x5678)
>> +		return 2;
>> +	return 0;
>> +}
> And here I'd only keep the C version.
>
>> +__naked struct with_ptr global_ret_struct_ptr(void)
>> +{
>> +	asm volatile (
>> +	"r0 = 0;"
>> +	"r2 = 0;"
>> +	"exit;"
>> +	);
>> +}
> [...]
>
>> +static __naked u128 agg_callee(void)
>> +{
>> +	asm volatile (
>> +	"r0 = 1;"
>> +	"r2 = 2;"
>> +	"exit;"
>> +	);
>> +}
>> +
>> +SEC("tc")
>> +__log_level(2)
>> +__msg("Live regs before insn:")
>> +__msg("0: .12345.... (85) call pc+2")
>> +__msg("1: ..2....... (bf) r0 = r2")
>> +__msg("2: 0......... (95) exit")
>> +__msg("3: .......... (b7) r0 = 1")
>> +__msg("4: 0......... (b7) r2 = 2")
>> +__msg("5: 0.2....... (95) exit")
>> +__naked int aggregate_ret_live(void)
>> +{
>> +	asm volatile (
>> +	"call %[agg_callee];"
>> +	"r0 = r2;"
>> +	"exit;"
>> +	:
>> +	: [agg_callee]"i"(agg_callee)
>> +	: __clobber_all);
>> +}
> Nit: usually we add such tests to compute_live_registers.c.

I can move the test to compute_live_registers.c.

>
> [...]
>
>> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_run.c b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c
> I think the test cases in this file are by a large degree redundant.
> A single test case calling a kfunc returning a small struct and
> checking returned values would suffice.

Ack. I tried to test for both C and asm test cases. They are indeed some duplicates.
I will try to remove the redundant ones in next revision.

>
> [...]

Re: [PATCH bpf-next 00/12] bpf: Support 16-byte return values in the R0:R2 register pair

[Yonghong Song]

On 7/9/26 5:56 PM, Eduard Zingerman wrote:
>> On Wed, 2026-07-08 at 13:09 -0700, Yonghong Song wrote:
> A design question regarding to how this series handles local
> functions:
> - It relies on the BTF gate to decide if function returns multiple values.
> - It also prohibits local functions from returning pointers as a part
>    of an aggregate return.
>
> I think this unnecessarily adds an additional discrepancy between how
> the return values are handled. The practical experience shows that
> existing differences between local and global functions are already a
> constant source of pain and confusion.
>
> I'd also argue that lifting the above limitations for local functions
> would make the implementation somewhat simpler.
>
> Therefore, I suggest to:
> - forgo the BTF check for local functions;
> - always copy r2 from callee to caller at EXIT

Not sure about this. If r2 is always copied from callee to caller, and
interpreter still only supports return r0, then this will imply interpreter
will never be used? I know interpreter will be eventually removed, but
at this point, people still use interpreter?

> - forgo the scalar struct requirement for local function
>    return types.

I think this is doable.

>
> Wdyt?

Re: [PATCH bpf-next 00/12] bpf: Support 16-byte return values in the R0:R2 register pair

[Eduard Zingerman]

On Thu, 2026-07-09 at 23:00 -0700, Yonghong Song wrote:
> 
> 
> On 7/9/26 5:56 PM, Eduard Zingerman wrote:
> > > On Wed, 2026-07-08 at 13:09 -0700, Yonghong Song wrote:
> > A design question regarding to how this series handles local
> > functions:
> > - It relies on the BTF gate to decide if function returns multiple values.
> > - It also prohibits local functions from returning pointers as a part
> >    of an aggregate return.
> > 
> > I think this unnecessarily adds an additional discrepancy between how
> > the return values are handled. The practical experience shows that
> > existing differences between local and global functions are already a
> > constant source of pain and confusion.
> > 
> > I'd also argue that lifting the above limitations for local functions
> > would make the implementation somewhat simpler.
> > 
> > Therefore, I suggest to:
> > - forgo the BTF check for local functions;
> > - always copy r2 from callee to caller at EXIT
> 
> Not sure about this. If r2 is always copied from callee to caller, and
> interpreter still only supports return r0, then this will imply interpreter
> will never be used? I know interpreter will be eventually removed, but
> at this point, people still use interpreter?

We can detect this statically from compute_live_registers().
If any r2's are alive after a function call the jit_required flag can be set.

> > - forgo the scalar struct requirement for local function
> >    return types.
> 
> I think this is doable.
> 
> > 
> > Wdyt?

Re: [PATCH bpf-next 00/12] bpf: Support 16-byte return values in the R0:R2 register pair

[Yonghong Song]

On 7/9/26 11:13 PM, Eduard Zingerman wrote:
> On Thu, 2026-07-09 at 23:00 -0700, Yonghong Song wrote:
>>
>> On 7/9/26 5:56 PM, Eduard Zingerman wrote:
>>>> On Wed, 2026-07-08 at 13:09 -0700, Yonghong Song wrote:
>>> A design question regarding to how this series handles local
>>> functions:
>>> - It relies on the BTF gate to decide if function returns multiple values.
>>> - It also prohibits local functions from returning pointers as a part
>>>     of an aggregate return.
>>>
>>> I think this unnecessarily adds an additional discrepancy between how
>>> the return values are handled. The practical experience shows that
>>> existing differences between local and global functions are already a
>>> constant source of pain and confusion.
>>>
>>> I'd also argue that lifting the above limitations for local functions
>>> would make the implementation somewhat simpler.
>>>
>>> Therefore, I suggest to:
>>> - forgo the BTF check for local functions;
>>> - always copy r2 from callee to caller at EXIT
>> Not sure about this. If r2 is always copied from callee to caller, and
>> interpreter still only supports return r0, then this will imply interpreter
>> will never be used? I know interpreter will be eventually removed, but
>> at this point, people still use interpreter?
> We can detect this statically from compute_live_registers().
> If any r2's are alive after a function call the jit_required flag can be set.

Sounds good, I will do some investigation on this.

>
>>> - forgo the scalar struct requirement for local function
>>>     return types.
>> I think this is doable.
>>
>>> Wdyt?