[PATCH v2 00/13] bpf: Support aggregate return values up to 16 bytes

[PATCH v2 00/13] bpf: Support aggregate return values up to 16 bytes

[Yonghong Song]

LLVM 23 can return an __int128, or a struct/union larger than 8 bytes and
no larger than 16 bytes, in the BPF R0:R2 register pair [1][2]. Before
that the BPF backend could not return such values at all: a by-value
aggregate return was rejected at compile time with "aggregate returns are
not supported", and an __int128 return failed in the backend with "unable
to allocate function return #1".

This series teaches the kernel the same convention, so that BPF programs
and kfuncs can return these values. The first 8 bytes of the value come
back in R0 and the second 8 bytes in R2. It applies to kfunc returns and
to BPF-to-BPF subprogram returns, both global and static. The main program
is unchanged: its return value is the program's exit code, so a return
larger than 8 bytes is still rejected at BPF_EXIT.

Patches 1-2 are preparation: patch 1 factors out the per-register check
used by the global return path, and patch 2 adds the shared helpers that
answer "does this subprogram return a register pair", so that the patches
which follow can be ordered independently. Patch 3 wires up the JIT side.
Patches 4-5 teach precision backtracking and live register analysis about
R2 as a second return register, ahead of the patch that starts modeling it.
Patch 6 rejects callbacks returning more than 8 bytes, since neither
bpf_callback_t nor bpf_exception_cb has a second return register. Patch 7
adds the verifier support proper, patch 8 rejects a pair return once the
subprogram's BTF has been marked unreliable, and patch 9 relaxes
btf_distill_func_proto() and btf_validate_return_type(), which is what
makes the whole thing reachable. Patches 10-12 add selftests and patch 13
documents the convention.

Constraints worth calling out:

 - A by-value struct or union returned by a kfunc or by a global subprogram
   must be composed only of scalars. The verifier models the returned
   register bits as an unknown scalar, so a pointer member would be
   laundered into one and escape provenance and reference tracking. A
   static subprogram is verified inline and is not restricted this way.

 - Returning the pair from a kfunc needs the JIT to place the second half
   into R2, which is architecture-specific work. Architectures opt in
   through bpf_jit_supports_kfunc_ret_reg_pair(); x86_64, arm64 and riscv64
   do so here, and elsewhere bpf_add_kfunc_call() rejects such a kfunc with
   -EOPNOTSUPP. A register-pair return from a BPF subprogram needs no such
   capability.

 - The interpreter propagates R0 alone out of a subprogram, so the JIT is
   forced wherever a caller can observe the pair.

 - The compiler side requires LLVM 23 or newer. The selftests written in C
   record which compiler built them in a read-only flag and report a skip
   rather than a pass when built by anything older; the inline-asm tests do
   not depend on the compiler and run everywhere.

  [1] https://github.com/llvm/llvm-project/pull/190894
  [2] https://github.com/llvm/llvm-project/pull/206876

Changelog:
  v1 -> v2:
    - v1: https://lore.kernel.org/bpf/20260708200939.2153664-1-yonghong.song@linux.dev/
    - Split the R0:R2 helpers out of the verifier patch into their own
      preparation patch, and reordered the series so the core verifier patch
      comes after the infrastructure it depends on.
    - New patch rejecting callbacks that return more than 8 bytes, both
      helper/kfunc callbacks and exception callbacks, with selftests.
    - New patch rejecting a register-pair return once btf_check_subprog_call()
      has marked the subprogram's BTF unreliable, rather than silently
      mistracking R2.
    - Folded "bpf: Force JIT for programs using the R0:R2 register pair" into
      the verifier patch.
    - Dropped "bpf: Reject >8 byte return values on return-reading trampoline
      paths" and its selftests; that went in separately as commit
      c48796aa6c39.
    - Described the register mapping as the first and second 8 bytes rather
      than the low and high 64 bits, which is only correct on little-endian,
      and reworded "16-byte" to "up to 16 bytes" where the range 9..16 was
      meant.

Yonghong Song (13):
  bpf: Factor check_global_ret_scalar_reg() out of the global return
    check
  bpf: Add helpers to describe the R0:R2 return register pair
  bpf: Wire up JIT support for 16-byte kfunc returns
  bpf: Track R2 of register-pair returns in precision backtracking
  bpf: Account R2 of register-pair returns in live register analysis
  bpf: Reject callbacks returning more than 8 bytes
  bpf: Add verifier support for 16-byte returns in R0:R2
  bpf: Reject register-pair returns when the subprog BTF is unreliable
  bpf: Enable aggregate return types up to 16 bytes
  selftests/bpf: Add C tests for 16-byte returns in R0:R2
  selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
  selftests/bpf: Add tests for callbacks returning more than 8 bytes
  Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2

 Documentation/bpf/kfuncs.rst                  |  62 +++
 arch/arm64/net/bpf_jit_comp.c                 |   5 +
 arch/riscv/net/bpf_jit_comp64.c               |   5 +
 arch/x86/net/bpf_jit_comp.c                   |  21 +
 include/linux/bpf_verifier.h                  |  16 +
 include/linux/filter.h                        |   1 +
 kernel/bpf/backtrack.c                        |  59 ++-
 kernel/bpf/btf.c                              |  44 +-
 kernel/bpf/core.c                             |   5 +
 kernel/bpf/liveness.c                         |  25 +-
 kernel/bpf/verifier.c                         | 287 ++++++++++--
 .../selftests/bpf/prog_tests/aggregate_ret.c  | 176 ++++++++
 .../selftests/bpf/prog_tests/exceptions.c     |   2 +
 .../selftests/bpf/prog_tests/fexit_bpf2bpf.c  |  15 +
 .../testing/selftests/bpf/prog_tests/timer.c  |   2 +
 .../selftests/bpf/progs/aggregate_ret_func.c  | 420 ++++++++++++++++++
 .../bpf/progs/aggregate_ret_int128_c.c        |  48 ++
 .../selftests/bpf/progs/aggregate_ret_kfunc.c | 127 ++++++
 .../bpf/progs/aggregate_ret_kfunc_c.c         |  66 +++
 .../selftests/bpf/progs/aggregate_ret_run.c   | 168 +++++++
 .../bpf/progs/aggregate_ret_struct_c.c        |  82 ++++
 .../bpf/progs/aggregate_ret_target.c          |  29 ++
 .../bpf/progs/aggregate_ret_union_c.c         |  58 +++
 .../bpf/progs/btf__exceptions_ret_pair_fail.c |  10 +
 .../bpf/progs/btf__timer_ret_pair_fail.c      |  10 +
 .../selftests/bpf/progs/exceptions_fail.c     |   2 +-
 .../bpf/progs/exceptions_ret_pair_fail.c      |  30 ++
 .../selftests/bpf/progs/freplace_ret_pair.c   |  20 +
 .../selftests/bpf/progs/timer_ret_pair_fail.c |  49 ++
 .../selftests/bpf/progs/verifier_arena.c      |  38 ++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  64 +++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  46 ++
 32 files changed, 1930 insertions(+), 62 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_run.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_target.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__exceptions_ret_pair_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__timer_ret_pair_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_ret_pair_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/freplace_ret_pair.c
 create mode 100644 tools/testing/selftests/bpf/progs/timer_ret_pair_fail.c


base-commit: 457d4ecb47aaf7a2cb46aaadd76e8c812e4f3c9e
-- 
2.53.0-Meta

[PATCH bpf-next v2 01/13] bpf: Factor check_global_ret_scalar_reg() out of the global return check

[Yonghong Song]

check_global_subprog_return_code() verifies that a global subprogram
returns void, an arena pointer, or register R0 holding a scalar value.
Later patches in this series add 16-byte aggregate return support, whose
second half is returned in R2 and needs the same validation.

Factor the per-register check into check_global_ret_scalar_reg(env, regno)
so that it can be reused for R2.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/verifier.c | 32 ++++++++++++++++++++------------
 1 file changed, 20 insertions(+), 12 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b274004fccfd..05a8f9907c3f 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -16739,37 +16739,45 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 	return 0;
 }
 
-static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
 {
-	struct bpf_reg_state *reg = reg_state(env, BPF_REG_0);
-	struct bpf_func_state *cur_frame = cur_func(env);
+	struct bpf_reg_state *reg;
 	int err;
 
-	if (subprog_returns_void(env, cur_frame->subprogno))
-		return 0;
-
-	err = check_reg_arg(env, BPF_REG_0, SRC_OP);
+	err = check_reg_arg(env, regno, SRC_OP);
 	if (err)
 		return err;
 
 	/* Pointers to arena are safe to pass between subprograms. */
-	if (is_arena_reg(env, BPF_REG_0))
+	if (is_arena_reg(env, regno))
 		return 0;
 
-	if (is_pointer_value(env, BPF_REG_0)) {
-		verbose(env, "R%d leaks addr as return value\n", BPF_REG_0);
+	if (is_pointer_value(env, regno)) {
+		verbose(env, "R%d leaks addr as return value\n", regno);
 		return -EACCES;
 	}
 
+	reg = reg_state(env, regno);
 	if (reg->type != SCALAR_VALUE) {
-		verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n",
-			reg_type_str(env, reg->type));
+		verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n",
+			regno, reg_type_str(env, reg->type));
 		return -EINVAL;
 	}
 
 	return 0;
 }
 
+static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+{
+	struct bpf_func_state *cur_frame = cur_func(env);
+	u32 subprog = cur_frame->subprogno;
+
+	if (subprog_returns_void(env, subprog))
+		return 0;
+
+	return check_global_ret_scalar_reg(env, BPF_REG_0);
+}
+
 /* Bitmask with 1s for all caller saved registers */
 #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
 
-- 
2.53.0-Meta

[PATCH bpf-next v2 02/13] bpf: Add helpers to describe the R0:R2 return register pair

[Yonghong Song]

LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more than
16 bytes: such a value comes back in the R0:R2 register pair, with R2
holding the upper half. See LLVM patches [1] and [2].

Later patches teach the JIT, precision backtracking, live register analysis
and the verifier itself about that convention. All of them need to answer
the same question: does this subprogram return its value in a register
pair? Add the shared helpers up front so that those patches can be ordered
independently of each other:

 - subprog_ret_type() resolves a subprogram's BTF return type. It is
   factored out of subprog_returns_void(). The verifier_bug_if(!func) and
   !func_proto checks it replaces are redundant, since
   check_btf_func_early() already rejects a func_info record whose type_id
   is not a BTF_KIND_FUNC pointing at a BTF_KIND_FUNC_PROTO. A check on
   prog->aux->{btf,func_info} is added instead: unlike
   subprog_returns_void(), which is only used for global subprograms, later
   callers ask about static subprograms too, and those may belong to a
   program loaded without BTF.

 - ret_regs_cnt() maps the size of a return value to the number of
   registers holding it.

 - bpf_ret_reg_pair() answers the question above. Its users query it at
   every subprogram call and at every subprogram exit, that is once per
   verifier state rather than once per subprogram, so the answer is
   precomputed into bpf_subprog_info->ret_reg_pair by
   bpf_compute_subprog_ret_regs() and the helper itself is a flag test.
   It lives in bpf_verifier.h because kernel/bpf/backtrack.c and
   kernel/bpf/liveness.c need it as well.

bpf_compute_subprog_ret_regs() runs in bpf_check() right before
bpf_compute_live_registers(), which is the first of those users: by then
BTF func_info has been validated and the subprogram list is final.

No functional change, bpf_ret_reg_pair() has no callers yet.

  [1] https://github.com/llvm/llvm-project/pull/190894
  [2] https://github.com/llvm/llvm-project/pull/206876

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 include/linux/bpf_verifier.h | 12 ++++++++
 kernel/bpf/verifier.c        | 55 ++++++++++++++++++++++++++++--------
 2 files changed, 56 insertions(+), 11 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index a2a40caca0a0..18a6ecff39c5 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -816,6 +816,8 @@ struct bpf_subprog_info {
 	bool is_async_cb: 1;
 	bool is_exception_cb: 1;
 	bool args_cached: 1;
+	/* true if the return value is passed in the R0:R2 register pair */
+	bool ret_reg_pair: 1;
 	/* true if bpf_fastcall stack region is used by functions that can't be inlined */
 	bool keep_fastcall_stack: 1;
 	bool changes_pkt_data: 1;
@@ -1049,6 +1051,16 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
 	return &env->subprog_info[subprog];
 }
 
+/*
+ * True if @subprog returns its value in the R0:R2 register pair. Cached by
+ * bpf_compute_subprog_ret_regs(), since this is queried on hot paths: at
+ * every subprogram call and at every subprogram exit.
+ */
+static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+	return subprog_info(env, subprog)->ret_reg_pair;
+}
+
 struct bpf_call_summary {
 	u8 num_params;
 	bool is_void;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 05a8f9907c3f..681dbb4f9e29 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -381,27 +381,57 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
 	return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
 }
 
-static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+/* Return type of a subprogram, NULL if it cannot be resolved */
+static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)
 {
-	const struct btf_type *type, *func, *func_proto;
+	const struct btf_type *func, *func_proto;
 	const struct btf *btf = env->prog->aux->btf;
 	u32 btf_id;
 
+	if (!btf || !env->prog->aux->func_info)
+		return NULL;
+
 	btf_id = env->prog->aux->func_info[subprog].type_id;
 
+	/* Both already validated by check_btf_func_early() at prog load. */
 	func = btf_type_by_id(btf, btf_id);
-	if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id))
-		return false;
-
 	func_proto = btf_type_by_id(btf, func->type);
-	if (!func_proto)
-		return false;
 
-	type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
-	if (!type)
-		return false;
+	return btf_type_skip_modifiers(btf, func_proto->type, NULL);
+}
+
+static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+{
+	const struct btf_type *type = subprog_ret_type(env, subprog);
 
-	return btf_type_is_void(type);
+	return type && btf_type_is_void(type);
+}
+
+/*
+ * Number of registers holding a function return value: a value of up to 8
+ * bytes is returned in R0, a value of more than 8 bytes and no more than 16
+ * bytes (an __int128 or a struct/union of such size) is returned in the R0:R2
+ * register pair, with R2 holding the upper half.
+ */
+static u32 ret_regs_cnt(u32 size)
+{
+	return size > 8 && size <= 16 ? 2 : 1;
+}
+
+/*
+ * Resolve the return convention of every subprogram once, so that
+ * bpf_ret_reg_pair() is a plain flag test on the hot paths that use it.
+ */
+static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
+{
+	const struct btf_type *type;
+	int subprog;
+
+	for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+		type = subprog_ret_type(env, subprog);
+		if (type && (btf_type_is_struct(type) || btf_type_is_scalar(type)))
+			subprog_info(env, subprog)->ret_reg_pair = ret_regs_cnt(type->size) > 1;
+	}
 }
 
 static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
@@ -20330,6 +20360,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
 	if (ret < 0)
 		goto skip_full_check;
 
+	/* must precede the first bpf_ret_reg_pair() user below */
+	bpf_compute_subprog_ret_regs(env);
+
 	ret = bpf_compute_live_registers(env);
 	if (ret < 0)
 		goto skip_full_check;
-- 
2.53.0-Meta

[PATCH bpf-next v2 03/13] bpf: Wire up JIT support for 16-byte kfunc returns

[Yonghong Song]

LLVM 23 returns an __int128, or a struct/union larger than 8 bytes and no
larger than 16 bytes, in the BPF R0:R2 register pair. The previous patch
taught the verifier about that convention; wire up the JIT side so that the
second half of the return value actually lands in R2.

A kfunc returning more than 8 bytes hands the second half of the result
back in RDX, the native x86-64 ABI's second return register. BPF R0 maps to
RAX so it needs no move, but BPF R2 maps to RSI, so emit a RDX->RSI move
after a BPF_PSEUDO_KFUNC_CALL whose function model reports ret_size > 8.

Placing the second return half into R2 is possible on any JIT, but it needs
architecture-specific JIT work. Rather than requiring every JIT to
implement it at once, add a bpf_jit_supports_kfunc_ret_reg_pair()
capability, defaulting to false in the generic core; an architecture opts
in once its JIT handles the R0:R2 pair, and the remaining ones are left for
future work. The verifier enforces it in bpf_add_kfunc_call(), rejecting a
kfunc whose return is larger than 8 bytes with -EOPNOTSUPP when the JIT
lacks the capability. Only x86, arm64 and riscv are supported so far.

On arm64 and riscv the native second return register is already BPF R2 (x1
in bpf2a64[] and a1 in regmap[] respectively), so the value is in the R0:R2
register pair on return with no extra move, unlike x86 (RDX->RSI). This has
been tested on x86 and arm64. The riscv path is expected to work by the
same register-mapping reasoning as arm64 but has not been tested.

bpf_add_kfunc_call() also rejects a kfunc that is marked KF_FASTCALL and
returns more than 8 bytes. The bpf_fastcall contract implemented by
mark_fastcall_pattern_for_call() assumes a call clobbers R0 plus the
registers holding its arguments, so a return in the R0:R2 pair would
clobber an R2 the caller expects the fastcall pattern to preserve. Such
a kfunc is rejected with -EOPNOTSUPP as well.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 arch/arm64/net/bpf_jit_comp.c   |  5 +++++
 arch/riscv/net/bpf_jit_comp64.c |  5 +++++
 arch/x86/net/bpf_jit_comp.c     | 21 +++++++++++++++++++++
 include/linux/filter.h          |  1 +
 kernel/bpf/core.c               |  5 +++++
 kernel/bpf/verifier.c           | 13 +++++++++++++
 6 files changed, 50 insertions(+)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 4cdc7dfb05ba..2e9640975f55 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2325,6 +2325,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index 8fe8969fb8a0..b234d4f54b65 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -2111,6 +2111,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_ptr_xchg(void)
 {
 	return true;
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 01e7ce569c1e..f7c15f7d61d3 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2592,6 +2592,22 @@ st:			insn_off = insn->off;
 				return -EINVAL;
 			if (priv_frame_ptr)
 				pop_r9(&prog);
+			if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
+				const struct btf_func_model *fm;
+
+				/*
+				 * A kfunc returning a >8 byte aggregate hands the
+				 * second half back in RDX (the native ABI's second
+				 * return reg), but BPF expects it in R0:R2. BPF R0
+				 * is RAX (no move needed), while BPF R2 is RSI, so
+				 * copy RDX into RSI.
+				 */
+				fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
+				if (!fm)
+					return -EFAULT;
+				if (fm->ret_size > 8)
+					emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
+			}
 			break;
 		}
 
@@ -4041,6 +4057,11 @@ bool bpf_jit_supports_kfunc_call(void)
 	return true;
 }
 
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return true;
+}
+
 bool bpf_jit_supports_stack_args(void)
 {
 	return true;
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 32d5297c557e..b8f70422c207 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1182,6 +1182,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_kfunc_ret_reg_pair(void);
 bool bpf_jit_supports_stack_args(void);
 bool bpf_jit_supports_far_kfunc_call(void);
 bool bpf_jit_supports_exceptions(void);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index e2076667b245..1afc21663c49 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3303,6 +3303,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
 	return false;
 }
 
+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+	return false;
+}
+
 bool __weak bpf_jit_supports_stack_args(void)
 {
 	return false;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 681dbb4f9e29..4010575d6715 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2841,6 +2841,19 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
 	err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
 	if (err)
 		return err;
+	if (func_model.ret_size > 8) {
+		if (kfunc.flags && (*kfunc.flags & KF_FASTCALL)) {
+			verbose(env,
+				"kfunc %s with >8-byte return is not supported with KF_FASTCALL\n",
+				kfunc.name);
+			return -EOPNOTSUPP;
+		}
+		if (!bpf_jit_supports_kfunc_ret_reg_pair()) {
+			verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
+				kfunc.name);
+			return -EOPNOTSUPP;
+		}
+	}
 
 	memset(&meta, 0, sizeof(meta));
 	meta.btf = kfunc.btf;
-- 
2.53.0-Meta

[PATCH bpf-next v2 04/13] bpf: Track R2 of register-pair returns in precision backtracking

[Yonghong Song]

A function returning a value larger than 8 bytes (a struct/union, or an
__int128) uses R2 as a second return register alongside R0. Precision
backtracking treats only R0 as a return register at a call/return boundary,
so once the verifier starts modeling R2 that way, marking the second half
of such a return precise would trip the "unexpected regs" checks in
backtrack_insn() and reject a valid program with -EFAULT. Handle it here,
ahead of the patch that introduces the modeling.

Marking the upper half precise, for example by branching on it after a
call to a static subprogram, walks backtracking into the callee and
reaches its BPF_EXIT with R2 still set in the mask. R2 is part of
BPF_REGMASK_ARGS, so this hits "backtracking exit unexpected regs".
Returning the pair from a global subprogram or from a kfunc instead hits
the equivalent check at the call site.

Handle R2 like R0 in the three boundaries where a call defines the return
registers:

 - static subprog exit (BPF_EXIT): when the callee returns a pair, R2 is a
   return register rather than a clobbered argument, so its precision has to
   cross the frame boundary just like R0's: clear it from the caller's mask
   before the R1-R5 check, then set it again in the callee's mask after
   bt_subprog_enter().

   The clear has to be conditional, which is why the subprogram containing
   the exit insn is looked up and queried. For a callee that does not return
   a pair, check_func_call() has already invalidated the caller's R1-R5 and
   prepare_func_exit() copies back only R0, so nothing after the call can
   depend on R2 and backtracking should never still be asking for it here.
   Clearing it unconditionally would turn that into a silent no-op instead
   of reporting it through the existing "backtracking exit unexpected regs"
   check.
 - global subprog call: a global subprog returning >8 bytes also sets R2;
   clear it before the args check.
 - kfunc call (BPF_CALL): a kfunc returning >8 bytes (model ret_size > 8)
   also sets R2; clear it like R0.

All three are gated on R2 actually being in the mask, so the extra BTF and
kfunc descriptor lookups stay off the common backtracking path.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 include/linux/bpf_verifier.h |  2 ++
 kernel/bpf/backtrack.c       | 59 ++++++++++++++++++++++++++++--------
 kernel/bpf/verifier.c        | 13 ++++++++
 3 files changed, 62 insertions(+), 12 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 18a6ecff39c5..adb3f3019a98 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1442,6 +1442,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);
 struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
 void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
+int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id,
+			   u16 btf_fd_idx, u8 *ret_size);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 2f473ad4fd7c..498b15082801 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -424,6 +424,14 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				 */
 				verifier_bug_if(idx + 1 != subseq_idx, env,
 						"extra insn from subprog");
+				/* a global subprog returning more than 8 bytes
+				 * sets R2 as well. R2 is part of the args mask
+				 * checked just below, so it has to be cleared
+				 * here rather than next to R0.
+				 */
+				if (bt_is_reg_set(bt, BPF_REG_2) &&
+				    bpf_ret_reg_pair(env, subprog))
+					bt_clear_reg(bt, BPF_REG_2);
 				/* r1-r5 are invalidated after subprog call,
 				 * so for global func call it shouldn't be set
 				 * anymore
@@ -507,6 +515,17 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 				return -ENOTSUPP;
 			/* regular helper call sets R0 */
 			bt_clear_reg(bt, BPF_REG_0);
+			/* a kfunc returning more than 8 bytes also sets R2 */
+			if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+			    bt_is_reg_set(bt, BPF_REG_2)) {
+				u8 ret_size;
+
+				if (bpf_get_kfunc_ret_size(env->prog, insn->imm, insn->off,
+							   &ret_size))
+					return -ENOTSUPP;
+				if (ret_size > 8)
+					bt_clear_reg(bt, BPF_REG_2);
+			}
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				/* if backtracking was looking for registers R1-R5
 				 * they should have been found already.
@@ -521,7 +540,29 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 					return -EFAULT;
 			}
 		} else if (opcode == BPF_EXIT) {
-			bool r0_precise;
+			bool from_subprog_call, r0_precise, r2_precise = false;
+
+			/* BPF_EXIT in subprog or callback always returns
+			 * right after the call instruction, so by checking
+			 * whether the instruction at subseq_idx-1 is subprog
+			 * call or not we can distinguish actual exit from
+			 * *subprog* from exit from *callback*. In the former
+			 * case, we need to propagate the precision of the
+			 * return registers, if necessary. In the latter we
+			 * never do that.
+			 */
+			from_subprog_call = subseq_idx - 1 >= 0 &&
+					    bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
+			if (from_subprog_call && bt_is_reg_set(bt, BPF_REG_2)) {
+				struct bpf_subprog_info *callee;
+
+				/* 'idx' is the exit insn, so it is in the callee */
+				callee = bpf_find_containing_subprog(env, idx);
+				if (verifier_bug_if(!callee, env,
+						    "no subprog contains exit insn %d", idx))
+					return -EFAULT;
+				r2_precise = bpf_ret_reg_pair(env, callee - env->subprog_info);
+			}
 
 			/* Backtracking to a nested function call, 'idx' is a part of
 			 * the inner frame 'subseq_idx' is a part of the outer frame.
@@ -534,23 +575,15 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 			if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
 				for (i = BPF_REG_1; i <= BPF_REG_5; i++)
 					bt_clear_reg(bt, i);
+			if (r2_precise)
+				bt_clear_reg(bt, BPF_REG_2);
 			if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
 				verifier_bug(env, "backtracking exit unexpected regs %x",
 					     bt_reg_mask(bt));
 				return -EFAULT;
 			}
 
-			/* BPF_EXIT in subprog or callback always returns
-			 * right after the call instruction, so by checking
-			 * whether the instruction at subseq_idx-1 is subprog
-			 * call or not we can distinguish actual exit from
-			 * *subprog* from exit from *callback*. In the former
-			 * case, we need to propagate r0 precision, if
-			 * necessary. In the former we never do that.
-			 */
-			r0_precise = subseq_idx - 1 >= 0 &&
-				     bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
-				     bt_is_reg_set(bt, BPF_REG_0);
+			r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
 
 			bt_clear_reg(bt, BPF_REG_0);
 			if (bt_subprog_enter(bt))
@@ -558,6 +591,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
 
 			if (r0_precise)
 				bt_set_reg(bt, BPF_REG_0);
+			if (r2_precise)
+				bt_set_reg(bt, BPF_REG_2);
 			/* r6-r9 and stack slots will stay set in caller frame
 			 * bitmasks until we return back from callee(s)
 			 */
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 4010575d6715..282aee7fc44c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2504,6 +2504,19 @@ int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
 	return 0;
 }
 
+int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id,
+			   u16 btf_fd_idx, u8 *ret_size)
+{
+	const struct bpf_kfunc_desc *desc;
+
+	desc = find_kfunc_desc(prog, func_id, btf_fd_idx);
+	if (!desc)
+		return -EFAULT;
+
+	*ret_size = desc->func_model.ret_size;
+	return 0;
+}
+
 #define BPF_FD_SLOT_BTF	1UL
 
 static void fd_slot_set_map(struct bpf_fd_array *slot, struct bpf_map *map)
-- 
2.53.0-Meta

[PATCH bpf-next v2 05/13] bpf: Account R2 of register-pair returns in live register analysis

[Yonghong Song]

A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a
struct/union or an __int128) reads R2 as well as R0, since the second half
of the return value is passed back in R2. compute_insn_live_regs() only
marked R0 used at exit, so a callee's R2 could be considered dead and
cleaned from checkpointed states, which would allow unsound state pruning.

Mark R2 as read at the BPF_EXIT of a subprogram that does return a register
pair. bpf_compute_live_registers() walks the instructions in order and
env->subprog_info[] is sorted by subprogram start, so the containing
subprogram is tracked with a running index and its return convention is
queried once per subprogram through bpf_ret_reg_pair().

Marking R2 at every exit instead would be simpler, but R2 would then stay
live backwards across any call that is not followed by a write to R2, which
is nearly every program, and would needlessly hurt state pruning.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/liveness.c | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 0aadfbae0acc..68df6d8db4bd 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2059,10 +2059,15 @@ struct insn_live_regs {
 /* Bitmask with 1s for all caller saved registers */
 #define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
 
-/* Compute info->{use,def} fields for the instruction */
+/*
+ * Compute info->{use,def} fields for the instruction. @ret_reg_pair tells
+ * whether the subprogram containing @insn returns its value in the R0:R2
+ * register pair, which matters for BPF_EXIT.
+ */
 static void compute_insn_live_regs(struct bpf_verifier_env *env,
 				   struct bpf_insn *insn,
-				   struct insn_live_regs *info)
+				   struct insn_live_regs *info,
+				   bool ret_reg_pair)
 {
 	struct bpf_call_summary cs;
 	u8 class = BPF_CLASS(insn->code);
@@ -2174,7 +2179,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
 			break;
 		case BPF_EXIT:
 			def = 0;
-			use = r0;
+			use = ret_reg_pair ? (r0 | BIT(BPF_REG_2)) : r0;
 			break;
 		case BPF_CALL:
 			def = ALL_CALLER_SAVED_REGS;
@@ -2209,8 +2214,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 	struct bpf_insn *insns = env->prog->insnsi;
 	struct insn_live_regs *state;
 	int insn_cnt = env->prog->len;
-	int err = 0, i, j;
-	bool changed;
+	int err = 0, i, j, subprog, start, end;
+	bool changed, ret_reg_pair;
 
 	/* Use the following algorithm:
 	 * - define the following:
@@ -2237,8 +2242,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
 		goto out;
 	}
 
-	for (i = 0; i < insn_cnt; ++i)
-		compute_insn_live_regs(env, &insns[i], &state[i]);
+	for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+		start = env->subprog_info[subprog].start;
+		end = env->subprog_info[subprog + 1].start;
+		ret_reg_pair = bpf_ret_reg_pair(env, subprog);
+
+		for (i = start; i < end; ++i)
+			compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
+	}
 
 	/* Forward pass: resolve stack access through FP-derived pointers */
 	err = bpf_compute_subprog_arg_access(env);
-- 
2.53.0-Meta

[PATCH bpf-next v2 06/13] bpf: Reject callbacks returning more than 8 bytes

[Yonghong Song]

A callback handed to a helper or a kfunc (bpf_loop(),
bpf_timer_set_callback(), bpf_for_each_map_elem(), ...) is invoked
through bpf_callback_t, and an exception callback is invoked by
bpf_throw() through

  u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);

Both prototypes yield a single u64 in R0, and neither caller has any
notion of a second return register, so a callback returning a value in
the R0:R2 pair would have the upper half of its return value silently
dropped.

Reject both at load time:

 - check_ld_imm(): a callback is materialized as PTR_TO_FUNC by an
   ld_imm64 pointing at its subprogram, so the subprogram's return
   convention can be checked where the callback pointer is created,
   before it ever reaches a helper or kfunc argument.

 - do_check_common(): an exception callback is not referenced by a
   PTR_TO_FUNC, it is named by a BTF decl_tag and verified on its own,
   so check it as its frame is set up, next to the existing "cannot
   return void" and single-argument checks.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/verifier.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 282aee7fc44c..5584178a0e1c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -16426,6 +16426,11 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
 			verbose(env, "callback function not static\n");
 			return -EINVAL;
 		}
+		if (bpf_ret_reg_pair(env, subprogno)) {
+			verbose(env,
+				"callback function with >8-byte return value is not supported\n");
+			return -EINVAL;
+		}
 
 		dst_reg->type = PTR_TO_FUNC;
 		dst_reg->subprogno = subprogno;
@@ -18650,6 +18655,12 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog)
 				ret = -EINVAL;
 				goto out;
 			}
+			if (bpf_ret_reg_pair(env, subprog)) {
+				verbose(env,
+					"exception cb cannot return value larger than 8 bytes\n");
+				ret = -EINVAL;
+				goto out;
+			}
 
 			/* Also ensure the callback only has a single scalar argument. */
 			if (sub->arg_cnt != 1 || sub->args[0].arg_type != ARG_ANYTHING) {
-- 
2.53.0-Meta

[PATCH bpf-next v2 07/13] bpf: Add verifier support for 16-byte returns in R0:R2

[Yonghong Song]

LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more than
16 bytes. See LLVM patches [1] and [2].

Before LLVM 23 the BPF backend could not return these values at all. A
by-value struct or union return (of any size) was rejected at compile time
with:

  error: aggregate returns are not supported

and an __int128 return failed later in the backend with:

  fatal error: error in backend: unable to allocate function return #1

Both are resolved in LLVM 23, which lowers such returns into the R0:R2
register pair.

This patch adds handling for returns greater than 8 bytes in several
places: BPF subprogram returns (the main program, and both global and
static subprograms) and kfunc returns.

The R0:R2 convention is only implemented in the JIT. The BPF interpreter
has no notion of a second return register: a BPF-to-BPF call goes through
JMP_CALL_ARGS and a BPF_EXIT hands back BPF_R0 alone, so a caller reading
R2 would see a stale value. Force the JIT wherever a caller can observe the
pair, that is at the call to a global subprogram in check_func_call() and
at the return from a static subprogram in prepare_func_exit(). Kfunc calls
need no separate handling since bpf_add_kfunc_call() already sets
jit_required for every kfunc call.

A by-value struct or union returned by a kfunc must be composed only of
scalars, since the verifier models the returned register bits as an unknown
scalar and a pointer field would otherwise be laundered into one, escaping
provenance and reference tracking.

A global subprogram must return a scalar in every return register. The
existing exemption for arena pointers now applies only when the return
value fits in R0 alone: both halves of a register pair carry a piece of a
>8 byte scalar, so an arena pointer in either of them is a leak rather than
a legitimate return value. A subprogram whose whole return value is an
arena pointer is unaffected.

A static subprogram is handled differently. The verifier walks into its
frame, so prepare_func_exit() propagates the return register(s) to the
caller. R0 holding a stack pointer has long been rejected outright there,
but R2 is deliberately not treated the same way. LLVM owns both sides of a
static call and is not bound by the ABI, so even with a 9..16 byte declared
return type it may leave R2 untouched when the caller only consumes the low
half; R2 can then hold an incidental stack pointer that is not a return
value at all, and rejecting the program would be a false positive.
Propagating the register as is would be worse: the callee frame is freed
immediately afterwards, leaving the caller with a PTR_TO_STACK that refers
to a frame which no longer exists. So the caller's R2 is marked
uninitialized instead, and only a caller that actually reads the returned
upper half fails. As with R0, a pointer into the caller's own frame is
scrubbed too, which is conservative but keeps the two registers consistent.

Once callers read R0:R2, an extension program can no longer replace a
function with a >8 byte return value: an extension's own return is
capped at 8 bytes by the program-exit check above, so it would leave R2
stale for the target's callers. btf_check_type_match() cannot catch
this, as it compares return types by btf_type->info only and an int
carries no vlen, so a 16-byte __int128 and an 8-byte long compare equal.
Reject such an attach in bpf_check_attach_target() instead.

  [1] https://github.com/llvm/llvm-project/pull/190894
  [2] https://github.com/llvm/llvm-project/pull/206876

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/verifier.c | 138 +++++++++++++++++++++++++++++++++++++-----
 1 file changed, 124 insertions(+), 14 deletions(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 5584178a0e1c..60b9e587e094 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -418,6 +418,9 @@ static u32 ret_regs_cnt(u32 size)
 	return size > 8 && size <= 16 ? 2 : 1;
 }
 
+/* Registers holding a function return value, in order. See ret_regs_cnt(). */
+static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };
+
 /*
  * Resolve the return convention of every subprogram once, so that
  * bpf_ret_reg_pair() is a plain flag test on the hot paths that use it.
@@ -9528,6 +9531,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	u16 callee_incoming, stack_arg_cnt;
 	struct bpf_func_state *caller;
 	int err, subprog, target_insn;
+	u32 i, nregs;
 
 	target_insn = *insn_idx + insn->imm + 1;
 	subprog = bpf_find_subprog(env, target_insn);
@@ -9570,10 +9574,24 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 		clear_caller_saved_regs(env, caller->regs);
 		invalidate_outgoing_stack_args(env, cur_func(env));
 
-		/* All non-void global functions return a 64-bit SCALAR_VALUE. */
+		/*
+		 * A non-void global function returns a 64-bit SCALAR_VALUE in
+		 * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
+		 */
 		if (!subprog_returns_void(env, subprog)) {
-			mark_reg_unknown(env, caller->regs, BPF_REG_0);
-			caller->regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG;
+			nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+			/*
+			 * The R0:R2 return convention is only implemented in the
+			 * JIT: the interpreter propagates BPF_R0 alone out of a
+			 * subprogram, so a caller reading R2 would see a stale
+			 * value. Force the JIT once a caller can observe the pair.
+			 */
+			if (nregs > 1)
+				env->prog->jit_required = 1;
+			for (i = 0; i < nregs; i++) {
+				mark_reg_unknown(env, caller->regs, ret_regs[i]);
+				caller->regs[ret_regs[i]].subreg_def = DEF_NOT_SUBREG;
+			}
 		}
 
 		if (env->subprog_info[subprog].might_throw) {
@@ -9895,10 +9913,14 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 	struct bpf_func_state *caller, *callee;
 	struct bpf_reg_state *r0;
 	bool in_callback_fn;
+	u32 i, nregs;
 	int err;
 
 	callee = state->frame[state->curframe];
 	r0 = &callee->regs[BPF_REG_0];
+	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+	if (nregs > 1)
+		env->prog->jit_required = 1;
 	if (r0->type == PTR_TO_STACK) {
 		/* technically it's ok to return caller's stack pointer
 		 * (or caller's caller's pointer) back to the caller,
@@ -9934,8 +9956,21 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 			return -EFAULT;
 		}
 	} else {
-		/* return to the caller whatever r0 had in the callee */
-		caller->regs[BPF_REG_0] = *r0;
+		/* return to the caller whatever the callee had in the
+		 * return register(s)
+		 */
+		for (i = 0; i < nregs; i++)
+			caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]];
+
+		/* R2 carries only the upper half of a register pair return
+		 * value. A stack pointer must not escape the callee (see the
+		 * R0 case above), but there is no need to reject the whole
+		 * program for it: hand the caller an uninitialized R2 instead,
+		 * so that only a caller actually using the returned pointer
+		 * fails.
+		 */
+		if (nregs > 1 && caller->regs[BPF_REG_2].type == PTR_TO_STACK)
+			bpf_mark_reg_not_init(env, &caller->regs[BPF_REG_2]);
 	}
 
 	/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
@@ -10835,6 +10870,14 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
 	return 0;
 }
 
+/* Mark a register holding a @reg_size byte part of a function return value */
+static void mark_ret_reg_size(struct bpf_verifier_env *env, struct bpf_reg_state *regs,
+			      u32 regno, size_t reg_size)
+{
+	regs[regno].subreg_def = reg_size == sizeof(u64) ?
+		DEF_NOT_SUBREG : env->insn_idx + 1;
+}
+
 /* mark_btf_func_reg_size() is used when the reg size is determined by
  * the BTF func_proto's return value size and argument.
  */
@@ -10845,8 +10888,7 @@ static void __mark_btf_func_reg_size(struct bpf_verifier_env *env, struct bpf_re
 
 	if (regno == BPF_REG_0) {
 		/* Function return value */
-		reg->subreg_def = reg_size == sizeof(u64) ?
-			DEF_NOT_SUBREG : env->insn_idx + 1;
+		mark_ret_reg_size(env, regs, regno, reg_size);
 	} else if (reg_size == sizeof(u64)) {
 		/* Function argument */
 		mark_insn_zext(env, reg);
@@ -10859,6 +10901,22 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno,
 	return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size);
 }
 
+/* Mark the register(s) holding a @size byte kfunc return value as unknown
+ * scalars. All of them are processed the same way, only the size differs:
+ * a single register may hold a sub-register sized value, while both halves
+ * of a register pair are treated as 64-bit wide.
+ */
+static void mark_kfunc_ret_regs(struct bpf_verifier_env *env,
+				struct bpf_reg_state *regs, u32 size)
+{
+	u32 i, nregs = ret_regs_cnt(size);
+
+	for (i = 0; i < nregs; i++) {
+		mark_reg_unknown(env, regs, ret_regs[i]);
+		mark_ret_reg_size(env, regs, ret_regs[i], nregs == 1 ? size : sizeof(u64));
+	}
+}
+
 static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta)
 {
 	return meta->kfunc_flags & KF_ACQUIRE;
@@ -13316,11 +13374,25 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	}
 
 	if (btf_type_is_scalar(t)) {
-		mark_reg_unknown(env, regs, BPF_REG_0);
+		mark_kfunc_ret_regs(env, regs, t->size);
 		if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
 		    meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
 			__mark_reg_const_zero(env, &regs[BPF_REG_0]);
-		mark_btf_func_reg_size(env, BPF_REG_0, t->size);
+	} else if (btf_type_is_struct(t)) {
+		/*
+		 * The returned struct comes back as raw register bits modeled
+		 * as an unknown scalar, so it must contain only scalars:
+		 * otherwise a pointer field would be laundered into a scalar
+		 * and escape provenance and reference tracking.
+		 */
+		if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+			verbose(env,
+				"kernel function %s returns %s %s that is not composed of scalars\n",
+				func_name, btf_type_str(t),
+				btf_name_by_offset(desc_btf, t->name_off));
+			return -EINVAL;
+		}
+		mark_kfunc_ret_regs(env, regs, t->size);
 	} else if (btf_type_is_ptr(t)) {
 		ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
 		err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
@@ -16800,7 +16872,8 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
 	return 0;
 }
 
-static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno,
+				       bool allow_arena_ptr_return)
 {
 	struct bpf_reg_state *reg;
 	int err;
@@ -16810,7 +16883,7 @@ static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
 		return err;
 
 	/* Pointers to arena are safe to pass between subprograms. */
-	if (is_arena_reg(env, regno))
+	if (allow_arena_ptr_return && is_arena_reg(env, regno))
 		return 0;
 
 	if (is_pointer_value(env, regno)) {
@@ -16832,11 +16905,26 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
 {
 	struct bpf_func_state *cur_frame = cur_func(env);
 	u32 subprog = cur_frame->subprogno;
+	u32 i, nregs;
+	int err;
 
 	if (subprog_returns_void(env, subprog))
 		return 0;
 
-	return check_global_ret_scalar_reg(env, BPF_REG_0);
+	/*
+	 * An arena pointer is only a legitimate return value when it is the
+	 * whole of it, that is when it is returned in R0 alone. Both halves of
+	 * a register pair carry a piece of a >8 byte scalar, so an arena
+	 * pointer in either of them is a leak.
+	 */
+	nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+	for (i = 0; i < nregs; i++) {
+		err = check_global_ret_scalar_reg(env, ret_regs[i], nregs == 1);
+		if (err)
+			return err;
+	}
+
+	return 0;
 }
 
 /* Bitmask with 1s for all caller saved registers */
@@ -17326,10 +17414,16 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
 	 */
 	if (cur_frame->subprogno &&
 	    !cur_frame->in_async_callback_fn &&
-	    !cur_frame->in_exception_callback_fn)
+	    !cur_frame->in_exception_callback_fn) {
 		err = check_global_subprog_return_code(env);
-	else
+	} else {
+		if (!cur_frame->subprogno && bpf_ret_reg_pair(env, 0)) {
+			verbose(env,
+				"return value larger than 8 bytes is not supported at program exit\n");
+			return -EINVAL;
+		}
 		err = check_return_code(env, BPF_REG_0, "R0");
+	}
 	if (err)
 		return err;
 	return PROCESS_BPF_EXIT;
@@ -19462,6 +19556,22 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 			return -EOPNOTSUPP;
 		}
 
+		/*
+		 * An extension replaces the target outright, so it has to match
+		 * the target's return convention. Its own return value is capped
+		 * at 8 bytes (a >8 byte program return is rejected at BPF_EXIT),
+		 * so it can never fill the R0:R2 pair the target's callers read.
+		 * This cannot be left to btf_check_type_match() above, which
+		 * compares return types by btf_type->info only: an int carries no
+		 * vlen, so a 16-byte __int128 and an 8-byte long compare equal.
+		 */
+		if (prog_extension && tgt_info->fmodel.ret_size > 8) {
+			bpf_log(log,
+				"Cannot replace function %s with a >8 byte return value\n",
+				tname);
+			return -EOPNOTSUPP;
+		}
+
 		/*
 		 * *.multi programs don't need an address during program
 		 * verification, we just take the module ref if needed.
-- 
2.53.0-Meta

[PATCH bpf-next v2 08/13] bpf: Reject register-pair returns when the subprog BTF is unreliable

[Yonghong Song]

The R0:R2 return convention is derived from the BTF function prototype:
bpf_compute_subprog_ret_regs() inspects the return type of every
subprogram and records whether its value comes back in a register pair.

btf_check_subprog_call() can decide, at a call site, that this BTF is
not to be trusted and mark the subprogram unreliable, which happens when
compiler optimizations remove arguments from a static function or when a
mismatched type is passed to a global one. From that point on the
verifier falls back to conservative, R0-only, semantics for the
subprogram, while the compiled code keeps returning a pair and leaves
the upper half in R2 behind the verifier's back.

Rather than silently mistracking R2, reject a return value larger than
8 bytes as soon as the prototype it was derived from becomes unreliable.
Add subprog_ret_pair_unreliable() and test it at the two places that can
observe the flag: check_func_call(), for the call itself, and
prepare_func_exit(), for the return from an inlined static subprogram.

Note that the main program needs no such check: a >8 byte return from
subprog 0 is rejected at BPF_EXIT regardless of whether its BTF is
reliable. Callbacks need none either: a callback address only becomes a
PTR_TO_FUNC through check_ld_imm(), which already rejects any callback
returning more than 8 bytes.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 kernel/bpf/verifier.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 60b9e587e094..4bf4e855d0e3 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -437,6 +437,21 @@ static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
 	}
 }
 
+/*
+ * A >8 byte BPF return changes the calling convention to R0:R2, so the
+ * verifier can only allow it while the subprogram's prototype remains
+ * reliable. Once BTF is marked unreliable, reject the feature instead of
+ * silently falling back to R0-only semantics.
+ */
+static bool subprog_ret_pair_unreliable(struct bpf_verifier_env *env, int subprog)
+{
+	struct bpf_prog_aux *aux = env->prog->aux;
+
+	return bpf_ret_reg_pair(env, subprog) &&
+	       aux->func_info_aux &&
+	       aux->func_info_aux[subprog].unreliable;
+}
+
 static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
 {
 	struct bpf_func_info *info;
@@ -9543,6 +9558,11 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
 	err = btf_check_subprog_call(env, subprog, caller->regs);
 	if (err == -EFAULT)
 		return err;
+	if (subprog_ret_pair_unreliable(env, subprog)) {
+		verbose(env, "Func#%d ('%s') returns >8 bytes, which requires reliable BTF\n",
+			subprog, subprog_name(env, subprog));
+		return -EINVAL;
+	}
 	if (bpf_subprog_is_global(env, subprog)) {
 		const char *sub_name = subprog_name(env, subprog);
 
@@ -9918,6 +9938,11 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
 
 	callee = state->frame[state->curframe];
 	r0 = &callee->regs[BPF_REG_0];
+	if (subprog_ret_pair_unreliable(env, callee->subprogno)) {
+		verbose(env, "Func#%d ('%s') returns >8 bytes, which requires reliable BTF\n",
+			callee->subprogno, subprog_name(env, callee->subprogno));
+		return -EINVAL;
+	}
 	nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
 	if (nregs > 1)
 		env->prog->jit_required = 1;
-- 
2.53.0-Meta

[PATCH bpf-next v2 09/13] bpf: Enable aggregate return types up to 16 bytes

[Yonghong Song]

Relax btf_distill_func_proto() to accept a by-value struct or union that
the R0:R2 convention added in earlier patches can carry:

 - a struct or union larger than 8 and up to 16 bytes, returned in the
   R0:R2 register pair, matching what LLVM emits for the BPF target;
 - a struct or union up to 8 bytes, returned in R0 alone.

A >8 byte scalar (__int128) was already accepted and is unchanged.
Everything else stays rejected: a return type larger than 16 bytes, and any
type that __get_type_size() cannot return in registers at all (e.g. an
array), which it already reports as ret < 0.

btf_distill_func_proto() also builds the trampoline (fentry/fexit/fmod_ret)
and struct_ops function models, so relaxing it widens what those can attach
to. A >8 byte return stays rejected on every path that reads the target's
return value: commit c48796aa6c39 ("bpf: Reject >8 byte return values on
return-reading trampoline paths") covers fexit, fmod_ret and fsession plus
their _multi variants, and struct_ops, and an fentry-only trampoline never
sets BPF_TRAMP_F_CALL_ORIG so it does not touch the return value at all. A
struct or union of 8 bytes or less is newly accepted for those paths; its
single eightbyte is returned in R0 like any other scalar.

btf_validate_return_type() is relaxed as well, so that it accepts a
by-value struct or union up to 16 bytes in addition to void and scalars.

With btf_distill_func_proto() and btf_validate_return_type() relaxed, the
verifier, JIT, precision-backtracking and live-register support from the
earlier patches becomes reachable: this final patch enables <=16 byte
aggregate return values end to end.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 include/linux/bpf_verifier.h                  |  2 +
 kernel/bpf/btf.c                              | 44 ++++++++++++++++---
 kernel/bpf/verifier.c                         |  6 +--
 .../selftests/bpf/progs/exceptions_fail.c     |  2 +-
 4 files changed, 45 insertions(+), 9 deletions(-)

diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index adb3f3019a98..7b36ff32301c 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1444,6 +1444,8 @@ void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
 bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
 int bpf_get_kfunc_ret_size(const struct bpf_prog *prog, u32 func_id,
 			   u16 btf_fd_idx, u8 *ret_size);
+bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
+				 const struct btf_type *t, int rec);
 
 int bpf_find_subprog(struct bpf_verifier_env *env, int off);
 bool bpf_is_throw_kfunc(struct bpf_insn *insn);
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 42414633cf26..28da02ea1e91 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -7572,7 +7572,12 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
 		return -EINVAL;
 	}
 	ret = __get_type_size(btf, func->type, &t);
-	if (ret < 0 || btf_type_is_struct(t)) {
+	/*
+	 * __get_type_size() already restricts a non-negative ret to void, a
+	 * pointer, an int, an enum or a struct/union, so only the size is checked
+	 * here.
+	 */
+	if (ret < 0 || ret > 16) {
 		bpf_log(log,
 			"The function %s return type %s is unsupported.\n",
 			tname, btf_type_str(t));
@@ -7945,7 +7950,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env *env,
 
 /* Check whether the type is a valid return type. */
 static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,
-		const struct btf_type *t, int subprog)
+		const struct btf_type *t, int subprog, bool is_global)
 {
 	u32 tags = 0;
 	int err;
@@ -7968,6 +7973,35 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
 	if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
 		return 0;
 
+	if (btf_type_is_struct(t) && t->size <= 16) {
+		/*
+		 * A >8 byte struct/union is returned in the R0:R2 register pair.
+		 * A global function is verified in isolation, so its caller models
+		 * the return as an opaque R0:R2 scalar pair; it must therefore
+		 * contain only scalars, otherwise a pointer field would be
+		 * laundered into a scalar and escape provenance and reference
+		 * tracking. That requirement is enforced here: do_check_common()
+		 * propagates the error for global functions and for the main
+		 * program.
+		 *
+		 * A local (static) function is verified inline and its R0:R2 are
+		 * copied as precise register state (with the JIT forced on when
+		 * the pair is consumed), so a pointer field stays tracked and needs
+		 * no such restriction. Accepting it here is not by itself what
+		 * makes it legal: btf_check_subprog_call() drops any error other
+		 * than -EFAULT. What it avoids is needlessly marking the
+		 * subprogram's BTF unreliable.
+		 *
+		 * The main program (subprog 0) takes the scalar-only path as well,
+		 * but its return value is the program's exit code, so a >8 byte
+		 * return is rejected separately at BPF_EXIT.
+		 */
+		bool local_func = subprog && !is_global;
+
+		if (local_func || __btf_type_is_scalar_struct(env, btf, t, 0))
+			return 0;
+	}
+
 	return -EOPNOTSUPP;
 }
 
@@ -8055,12 +8089,12 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
 		return -EINVAL;
 	}
 
-	err = btf_validate_return_type(env, btf, t, subprog);
+	err = btf_validate_return_type(env, btf, t, subprog, is_global);
 	if (err) {
 		if (is_global) {
 			bpf_log(log,
-				"Global function %s() return value not void or scalar. "
-				"Only those are supported.\n",
+				"Global function %s() has unsupported return type. "
+				"Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n",
 				tname);
 		}
 		return err;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 4bf4e855d0e3..2c0a9ec6f02e 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11202,9 +11202,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_
 }
 
 /* Returns true if struct is composed of scalars, 4 levels of nesting allowed */
-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
-					const struct btf *btf,
-					const struct btf_type *t, int rec)
+bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
+				 const struct btf *btf,
+				 const struct btf_type *t, int rec)
 {
 	const struct btf_type *member_type;
 	const struct btf_member *member;
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index ac44d60e5066..9708efb93683 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)
 
 SEC("?tc")
 __exception_cb(exception_cb_bad_ret_type1)
-__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.")
+__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.")
 int reject_exception_cb_type_1(struct __sk_buff *ctx)
 {
 	bpf_throw(0);
-- 
2.53.0-Meta

[PATCH bpf-next v2 10/13] selftests/bpf: Add C tests for 16-byte returns in R0:R2

[Yonghong Song]

Add selftests that exercise a 16-byte return value passed in the R0:R2
register pair, written in C so that they depend on the compiler lowering
the register-pair return. Covered are an __int128 return, a 16-byte struct
return (from a static and from a global subprogram) and a 16-byte union
return, plus __int128 and 16-byte struct returns from a kfunc, for which
bpf_kfunc_call_test_i128() and bpf_kfunc_call_test_ret_pair() are added to
bpf_testmod.

The R0:R2 convention is only emitted by LLVM 23 and newer. Each object
records in a read-only has_reg_pair_ret flag which compiler built it; where
that is false the programs are stubs and subtests report a skip rather than
a pass.

The two kfunc subtests further depend on the JIT: bpf_add_kfunc_call()
rejects a kfunc returning more than 8 bytes with -EOPNOTSUPP where
bpf_jit_supports_kfunc_ret_reg_pair() is false. Those calls therefore live
in an object of their own, and that load failing with -EOPNOTSUPP is what
turns the two subtests into skips, so no list of the JITs implementing the
pair needs to be kept here. A register-pair return from a BPF subprogram
needs no JIT support, so the remaining subtests run everywhere.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 .../selftests/bpf/prog_tests/aggregate_ret.c  | 125 ++++++++++++++++++
 .../bpf/progs/aggregate_ret_int128_c.c        |  48 +++++++
 .../bpf/progs/aggregate_ret_kfunc_c.c         |  66 +++++++++
 .../bpf/progs/aggregate_ret_struct_c.c        |  82 ++++++++++++
 .../bpf/progs/aggregate_ret_union_c.c         |  58 ++++++++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  14 ++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  11 ++
 7 files changed, 404 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c

diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
new file mode 100644
index 000000000000..42017f89c4da
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -0,0 +1,125 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include "aggregate_ret_int128_c.skel.h"
+#include "aggregate_ret_struct_c.skel.h"
+#include "aggregate_ret_union_c.skel.h"
+#include "aggregate_ret_kfunc_c.skel.h"
+
+static void run_prog(struct bpf_program *prog, bool supported)
+{
+	char buf[64] = {};
+	int err, prog_fd;
+	LIBBPF_OPTS(bpf_test_run_opts, topts,
+		    .data_in = buf,
+		    .data_size_in = sizeof(buf),
+		    .repeat = 1,
+	);
+
+	if (!supported) {
+		test__skip();
+		return;
+	}
+
+	prog_fd = bpf_program__fd(prog);
+	if (!ASSERT_GE(prog_fd, 0, "prog_fd"))
+		return;
+
+	err = bpf_prog_test_run_opts(prog_fd, &topts);
+	if (!ASSERT_OK(err, "test_run"))
+		return;
+
+	ASSERT_EQ(topts.retval, 0, "aggregate_ret_result");
+}
+
+static void test_int128_c(void)
+{
+	struct aggregate_ret_int128_c *skel;
+
+	skel = aggregate_ret_int128_c__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "skel_int128_c_open_load"))
+		return;
+
+	if (test__start_subtest("int128_c"))
+		run_prog(skel->progs.aggregate_ret_int128_c_test,
+			 skel->rodata->has_reg_pair_ret);
+
+	aggregate_ret_int128_c__destroy(skel);
+}
+
+static void test_struct_c(void)
+{
+	struct aggregate_ret_struct_c *skel;
+
+	skel = aggregate_ret_struct_c__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "skel_struct_c_open_load"))
+		return;
+
+	if (test__start_subtest("struct_c"))
+		run_prog(skel->progs.aggregate_ret_struct_c_test,
+			 skel->rodata->has_reg_pair_ret);
+
+	if (test__start_subtest("global_struct_c"))
+		run_prog(skel->progs.aggregate_ret_global_struct_c_test,
+			 skel->rodata->has_reg_pair_ret);
+
+	aggregate_ret_struct_c__destroy(skel);
+}
+
+static void test_union_c(void)
+{
+	struct aggregate_ret_union_c *skel;
+
+	skel = aggregate_ret_union_c__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "skel_union_c_open_load"))
+		return;
+
+	if (test__start_subtest("union_c"))
+		run_prog(skel->progs.aggregate_ret_union_c_test,
+			 skel->rodata->has_reg_pair_ret);
+
+	aggregate_ret_union_c__destroy(skel);
+}
+
+static void test_kfunc_c(void)
+{
+	struct aggregate_ret_kfunc_c *skel;
+	bool supported;
+	int err;
+
+	skel = aggregate_ret_kfunc_c__open();
+	if (!ASSERT_OK_PTR(skel, "skel_kfunc_c_open"))
+		return;
+
+	supported = skel->rodata->has_reg_pair_ret;
+
+	/*
+	 * Where the JIT cannot hand the second half of a >8-byte kfunc return
+	 * back in R0:R2, bpf_add_kfunc_call() rejects the call with
+	 * -EOPNOTSUPP. Asking the kernel keeps this test free of a list of the
+	 * JITs that can, which would have to be updated as the rest of them
+	 * learn.
+	 */
+	err = aggregate_ret_kfunc_c__load(skel);
+	if (err == -EOPNOTSUPP)
+		supported = false;
+	else if (!ASSERT_OK(err, "skel_kfunc_c_load"))
+		goto out;
+
+	if (test__start_subtest("kfunc_int128_c"))
+		run_prog(skel->progs.aggregate_ret_kfunc_int128_c_test, supported);
+
+	if (test__start_subtest("kfunc_struct_c"))
+		run_prog(skel->progs.aggregate_ret_kfunc_struct_c_test, supported);
+
+out:
+	aggregate_ret_kfunc_c__destroy(skel);
+}
+
+void test_aggregate_ret(void)
+{
+	test_int128_c();
+	test_struct_c();
+	test_union_c();
+	test_kfunc_c();
+}
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
new file mode 100644
index 000000000000..f2e09c8be0be
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
@@ -0,0 +1,48 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+const volatile bool has_reg_pair_ret = true;
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+typedef unsigned __int128 u128;
+
+static __noinline u128 make_i128(__u64 a, __u64 b)
+{
+	return ((u128)(a + b) << 64) | (a - b);
+}
+
+SEC("tc")
+int aggregate_ret_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v;
+
+	v = make_i128(a, b);
+	if ((__u64)(v >> 64) != a + b)
+		return 1;
+	if ((__u64)v != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+const volatile bool has_reg_pair_ret = false;
+
+SEC("tc")
+int aggregate_ret_int128_c_test(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
new file mode 100644
index 000000000000..fd000620f314
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
@@ -0,0 +1,66 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+const volatile bool has_reg_pair_ret = true;
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+typedef unsigned __int128 u128;
+
+SEC("tc")
+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	u128 v;
+
+	v = bpf_kfunc_call_test_i128(a, b);
+	if ((__u64)(v >> 64) != a + b)
+		return 1;
+	if ((__u64)v != a - b)
+		return 2;
+
+	return 0;
+}
+
+SEC("tc")
+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct prog_test_ret_pair p;
+
+	p = bpf_kfunc_call_test_ret_pair(a, b);
+	if (p.hi != a + b)
+		return 1;
+	if (p.lo != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+const volatile bool has_reg_pair_ret = false;
+
+SEC("tc")
+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+SEC("tc")
+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
new file mode 100644
index 000000000000..5296e41da3f0
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
@@ -0,0 +1,82 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+const volatile bool has_reg_pair_ret = true;
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+struct pair {
+	__u64 hi;	/* R0 */
+	__u64 lo;	/* R2 */
+};
+
+static __noinline struct pair make_pair(__u64 a, __u64 b)
+{
+	struct pair p = { .hi = a + b, .lo = a - b };
+
+	return p;
+}
+
+SEC("tc")
+int aggregate_ret_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p;
+
+	p = make_pair(a, b);
+	if (p.hi != a + b)
+		return 1;
+	if (p.lo != a - b)
+		return 2;
+
+	return 0;
+}
+
+__noinline struct pair make_pair_global(__u64 a, __u64 b)
+{
+	struct pair p = { .hi = a + b, .lo = a - b };
+
+	return p;
+}
+
+SEC("tc")
+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	struct pair p;
+
+	p = make_pair_global(a, b);
+	if (p.hi != a + b)
+		return 1;
+	if (p.lo != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+const volatile bool has_reg_pair_ret = false;
+
+SEC("tc")
+int aggregate_ret_struct_c_test(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+SEC("tc")
+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c
new file mode 100644
index 000000000000..5547fa6cbd49
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c
@@ -0,0 +1,58 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+const volatile bool has_reg_pair_ret = true;
+
+#define MIX_A	0xdeadbeefcafef00dULL
+#define MIX_B	0x0123456789abcdefULL
+
+union pair {
+	__u64 halves[2];
+	struct {
+		__u64 lo;	/* R0 */
+		__u64 hi;	/* R2 */
+	} parts;
+};
+
+static __noinline union pair make_pair(__u64 a, __u64 b)
+{
+	union pair p;
+
+	p.halves[0] = a + b;
+	p.halves[1] = a - b;
+	return p;
+}
+
+SEC("tc")
+int aggregate_ret_union_c_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len ^ MIX_A;
+	__u64 b = skb->len ^ MIX_B;
+	union pair p;
+
+	p = make_pair(a, b);
+	if (p.parts.lo != a + b)
+		return 1;
+	if (p.parts.hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+#else
+
+const volatile bool has_reg_pair_ret = false;
+
+SEC("tc")
+int aggregate_ret_union_c_test(struct __sk_buff *skb)
+{
+	return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index eb0f9b5e18d8..2ceb34df472e 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -857,6 +857,18 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
 	return 0;
 }
 
+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
+{
+	return (__int128)(((unsigned __int128)(a + b) << 64) | (a - b));
+}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)
+{
+	struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };
+
+	return r;
+}
+
 __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
 					 u64 e, u64 f, u64 g, u64 h,
 					 u64 i, u64 j)
@@ -1390,6 +1402,8 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index c36bb911defa..e6e59fdce33c 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -55,6 +55,15 @@ struct prog_test_big_arg {
 	__u64 b;
 };
 
+/*
+ * A 16-byte struct returned by value from a kfunc: .hi comes back in R0 and
+ * .lo in R2.
+ */
+struct prog_test_ret_pair {
+	__u64 hi;
+	__u64 lo;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -118,6 +127,8 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
 struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
 long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
 __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
 			       __u64 e, __u64 f, __u64 g, __u64 h,
 			       __u64 i, __u64 j) __ksym;
-- 
2.53.0-Meta

[PATCH bpf-next v2 11/13] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

[Yonghong Song]

Add inline-asm tests, which do not depend on the compiler lowering a
register-pair return and so run regardless of the LLVM version, covering
what the C tests cannot reach. aggregate_ret_func.c exercises BPF-to-BPF
returns: a global subprogram whose R2 the caller may read, ones that leave
R2 uninitialised or holding a pointer, a static subprogram whose R2 stays
precise under backtracking, R2 liveness across a call, and a >8 byte return
at program exit. Six kfuncs returning aggregates by value are added to
bpf_testmod, and aggregate_ret_run.c calls them from inline asm to check
what comes back in R0:R2.

A negative arena test is added as well: a global subprogram with a
register-pair return that leaves an arena pointer in R2 is rejected, since
an arena pointer is only a valid return value when it is returned in R0
alone.

Three cases cover the boundaries of the new convention:

 - A return value larger than 16 bytes does not fit in R0:R2 and is
   rejected by btf_distill_func_proto(), ahead of the KF_FASTCALL and
   JIT-capability checks; one of the new kfuncs returns a 24-byte struct
   for this. The equivalent for a BPF subprogram cannot be written in C:
   from LLVM 23 on, a by-value return larger than 16 bytes is lowered to an
   sret pointer argument and the BTF the verifier reads says the function
   returns void, so the size bound in btf_validate_return_type() only
   guards hand-crafted BTF.

 - A static subprogram returning a struct that contains a pointer is
   accepted, and the caller can use the returned pointer. Unlike a global
   subprogram, whose caller models the return as an opaque scalar pair, a
   static one is verified inline, so prepare_func_exit() hands the caller
   real register state and the pointer stays tracked.

 - An extension cannot replace a function returning more than 8 bytes.
   btf_check_type_match() does not catch this, since it compares return
   types by btf_type->info alone and both an __int128 and a __u64 are
   BTF_KIND_INT with no vlen, so the rejection has to come from
   bpf_check_attach_target(). The test reuses the freplace failure harness
   in fexit_bpf2bpf.c, with aggregate_ret_target.c providing a target whose
   global subprogram returns in R0:R2.

The kfunc tests need the JIT to place the second half of a return value
into R2, which bpf_add_kfunc_call() only allows where
bpf_jit_supports_kfunc_ret_reg_pair() is true. In aggregate_ret_kfunc.c the
two tests that depend on getting past that check are tagged
__arch_x86_64/__arch_arm64/__arch_riscv64; the others are rejected earlier
(on KF_FASTCALL, on a >16 byte return, and on reading R2 after an 8-byte
struct return) and run everywhere. In aggregate_ret_run.c the kfunc-calling
programs are dropped from the object when the load reports -EOPNOTSUPP and
their subtests are skipped, and the __int128 inline-asm test is split into
a BPF-to-BPF half, which needs no JIT capability and runs everywhere, and a
kfunc half.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 .../selftests/bpf/prog_tests/aggregate_ret.c  |  51 +++
 .../selftests/bpf/prog_tests/fexit_bpf2bpf.c  |  15 +
 .../selftests/bpf/progs/aggregate_ret_func.c  | 420 ++++++++++++++++++
 .../selftests/bpf/progs/aggregate_ret_kfunc.c | 127 ++++++
 .../selftests/bpf/progs/aggregate_ret_run.c   | 168 +++++++
 .../bpf/progs/aggregate_ret_target.c          |  29 ++
 .../selftests/bpf/progs/freplace_ret_pair.c   |  20 +
 .../selftests/bpf/progs/verifier_arena.c      |  38 ++
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  50 +++
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  35 ++
 10 files changed, 953 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_run.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_target.c
 create mode 100644 tools/testing/selftests/bpf/progs/freplace_ret_pair.c

diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
index 42017f89c4da..2a0c6996f0b9 100644
--- a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -5,6 +5,9 @@
 #include "aggregate_ret_struct_c.skel.h"
 #include "aggregate_ret_union_c.skel.h"
 #include "aggregate_ret_kfunc_c.skel.h"
+#include "aggregate_ret_run.skel.h"
+#include "aggregate_ret_func.skel.h"
+#include "aggregate_ret_kfunc.skel.h"
 
 static void run_prog(struct bpf_program *prog, bool supported)
 {
@@ -116,10 +119,58 @@ static void test_kfunc_c(void)
 	aggregate_ret_kfunc_c__destroy(skel);
 }
 
+static void test_run(void)
+{
+	struct aggregate_ret_run *skel;
+	bool kfunc_ok = true;
+	int err;
+
+	skel = aggregate_ret_run__open();
+	if (!ASSERT_OK_PTR(skel, "skel_run_open"))
+		return;
+
+	err = aggregate_ret_run__load(skel);
+	if (err == -EOPNOTSUPP) {
+		kfunc_ok = false;
+		aggregate_ret_run__destroy(skel);
+
+		skel = aggregate_ret_run__open();
+		if (!ASSERT_OK_PTR(skel, "skel_run_reopen"))
+			return;
+
+		bpf_program__set_autoload(skel->progs.aggregate_ret_asm_kfunc_test, false);
+		bpf_program__set_autoload(skel->progs.aggregate_ret_struct_test, false);
+		bpf_program__set_autoload(skel->progs.aggregate_ret_union_test, false);
+
+		err = aggregate_ret_run__load(skel);
+	}
+	if (!ASSERT_OK(err, "skel_run_load"))
+		goto out;
+
+	if (test__start_subtest("asm"))
+		run_prog(skel->progs.aggregate_ret_asm_test, true);
+
+	if (test__start_subtest("asm_kfunc"))
+		run_prog(skel->progs.aggregate_ret_asm_kfunc_test, kfunc_ok);
+
+	if (test__start_subtest("struct"))
+		run_prog(skel->progs.aggregate_ret_struct_test, kfunc_ok);
+
+	if (test__start_subtest("union"))
+		run_prog(skel->progs.aggregate_ret_union_test, kfunc_ok);
+
+out:
+	aggregate_ret_run__destroy(skel);
+}
+
 void test_aggregate_ret(void)
 {
 	test_int128_c();
 	test_struct_c();
 	test_union_c();
 	test_kfunc_c();
+	test_run();
+
+	RUN_TESTS(aggregate_ret_func);
+	RUN_TESTS(aggregate_ret_kfunc);
 }
diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
index 4a87d7163c8c..a40d63a87d7f 100644
--- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
+++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
@@ -440,6 +440,19 @@ static void test_func_replace_int_with_void(void)
 				     " doesn't match type INT of global_func2()");
 }
 
+static void test_func_replace_ret_pair(void)
+{
+	const char *msg = "Cannot replace function agg_ret_target_func with a >8 byte return";
+
+	/*
+	 * An extension cannot replace a function whose return value comes back
+	 * in the R0:R2 pair: the extension's own return is capped at 8 bytes,
+	 * so it would leave R2 stale for the target's callers.
+	 */
+	test_obj_load_failure_common("freplace_ret_pair.bpf.o",
+				     "./aggregate_ret_target.bpf.o", msg);
+}
+
 static int find_prog_btf_id(const char *name, __u32 attach_prog_fd)
 {
 	struct bpf_prog_info info = {};
@@ -605,6 +618,8 @@ void serial_test_fexit_bpf2bpf(void)
 		test_func_replace_progmap();
 	if (test__start_subtest("freplace_int_with_void"))
 		test_func_replace_int_with_void();
+	if (test__start_subtest("freplace_ret_pair"))
+		test_func_replace_ret_pair();
 	if (test__start_subtest("freplace_void"))
 		test_func_replace_void();
 }
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
new file mode 100644
index 000000000000..0bc18450a46e
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
@@ -0,0 +1,420 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+typedef unsigned __int128 u128;
+
+__naked u128 global_agg_good(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"	/* low 64 bits */
+	"r2 = 0x5678;"	/* high 64 bits */
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+__naked u128 global_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__success __retval(0)
+int aggregate_ret_global(void *ctx)
+{
+	__u64 lo, hi;
+
+	asm volatile (
+	"call %[global_agg_good];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: __imm(global_agg_good)
+	: "r0", "r1", "r2", "r3", "r4", "r5");
+	if (lo != 0x1234)
+		return 1;
+	if (hi != 0x5678)
+		return 2;
+	return 0;
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_global_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad)
+	: __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("At subprogram exit the register R2 is not a scalar value")
+__naked int aggregate_ret_global_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_agg_bad_ptr];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(global_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_bad_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r10;"	/* stack pointer placed in the second return register */
+	"exit;"
+	);
+}
+
+/*
+ * R2 is caller-saved and only copied from the callee at exit; a PTR_TO_STACK
+ * left in it is turned into an uninitialized R2 in the caller. A caller that
+ * never reads R2 is therefore unaffected and loads fine.
+ */
+SEC("tc")
+__success __retval(0)
+__naked int aggregate_ret_static_ptr_unused(void)
+{
+	asm volatile (
+	"call %[static_agg_bad_ptr];"
+	"r0 = 0;"		/* R2 holds a stack pointer but is never read */
+	"exit;"
+	:
+	: __imm(static_agg_bad_ptr)
+	: __clobber_all);
+}
+
+/* But a caller that does read the returned stack pointer is rejected. */
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_static_ptr_read_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_bad_ptr];"
+	"r0 = r2;"		/* using the returned stack pointer is rejected */
+	"exit;"
+	:
+	: __imm(static_agg_bad_ptr)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_no_r2(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_static_uninit_fail(void)
+{
+	asm volatile (
+	"call %[static_agg_no_r2];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: __imm(static_agg_no_r2)
+	: __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_precise(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 4;"	/* second half; its value is made precise below */
+	"exit;"
+	);
+}
+
+SEC("tc")
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
+__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
+__naked int aggregate_ret_static_precise(void)
+{
+	asm volatile (
+	"call %[static_agg_precise];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_agg_precise)
+	: __clobber_all);
+}
+
+SEC("tc")
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
+__naked int aggregate_ret_global_precise(void)
+{
+	asm volatile (
+	"call %[global_agg_good];"
+	"r6 = r2;"		/* derived from the aggregate's second half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_agg_good)
+	: __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("return value larger than 8 bytes is not supported at program exit")
+__naked u128 aggregate_ret_entry_fail(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+#if __clang_major__ >= 23
+
+struct pair {
+	__u64 hi;
+	__u64 lo;
+};
+
+union upair {
+	__u64 halves[2];
+	struct {
+		__u64 lo;
+		__u64 hi;
+	} parts;
+};
+
+/* A by-value struct that smuggles a pointer, which must be rejected. */
+struct with_ptr {
+	void *p;
+	__u64 x;
+};
+
+/* A by-value union that smuggles a pointer, which must be rejected too. */
+union upair_with_ptr {
+	void *p;
+	__u64 halves[2];
+};
+
+/* Global subprogram returning a scalar-only 16-byte struct in R0:R2. */
+__naked struct pair global_ret_struct(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"	/* struct's first half */
+	"r2 = 0x5678;"	/* struct's second half */
+	"exit;"
+	);
+}
+
+/* Global subprogram returning a scalar-only 16-byte union in R0:R2. */
+__naked union upair global_ret_union(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"
+	"r2 = 0x5678;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__success __retval(0)
+int aggregate_ret_global_struct(void *ctx)
+{
+	__u64 lo, hi;
+
+	asm volatile (
+	"call %[global_ret_struct];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: __imm(global_ret_struct)
+	: "r0", "r1", "r2", "r3", "r4", "r5");
+	if (lo != 0x1234)
+		return 1;
+	if (hi != 0x5678)
+		return 2;
+	return 0;
+}
+
+SEC("tc")
+__success __retval(0)
+int aggregate_ret_global_union(void *ctx)
+{
+	__u64 lo, hi;
+
+	asm volatile (
+	"call %[global_ret_union];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: __imm(global_ret_union)
+	: "r0", "r1", "r2", "r3", "r4", "r5");
+	if (lo != 0x1234)
+		return 1;
+	if (hi != 0x5678)
+		return 2;
+	return 0;
+}
+
+__naked struct with_ptr global_ret_struct_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_struct_ptr() has unsupported return type")
+__naked int aggregate_ret_global_struct_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_struct_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_struct_ptr)
+	: __clobber_all);
+}
+
+__naked union upair_with_ptr global_ret_union_ptr(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = 0;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_union_ptr() has unsupported return type")
+__naked int aggregate_ret_global_union_ptr_fail(void)
+{
+	asm volatile (
+	"call %[global_ret_union_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(global_ret_union_ptr)
+	: __clobber_all);
+}
+
+#endif /* __clang_major__ >= 23 */
+
+static __naked u128 agg_callee(void)
+{
+	asm volatile (
+	"r0 = 1;"
+	"r2 = 2;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__log_level(2)
+__msg("Live regs before insn:")
+/*
+ * R2 is read at the exit of agg_callee() (insn 5), which returns a pair, but
+ * not at the exit of this program (insn 2), which returns an int.
+ */
+__msg("0: .12345.... (85) call pc+2")
+__msg("1: ..2....... (bf) r0 = r2")
+__msg("2: 0......... (95) exit")
+__msg("3: .......... (b7) r0 = 1")
+__msg("4: 0......... (b7) r2 = 2")
+__msg("5: 0.2....... (95) exit")
+__naked int aggregate_ret_live(void)
+{
+	asm volatile (
+	"call %[agg_callee];"
+	"r0 = r2;"
+	"exit;"
+	:
+	: [agg_callee]"i"(agg_callee)
+	: __clobber_all);
+}
+
+/*
+ * A static subprogram is verified inline, so prepare_func_exit() hands the
+ * caller the callee's actual R0:R2 register state rather than an opaque scalar
+ * pair. A pointer in the returned struct therefore stays tracked and is usable
+ * by the caller, which is why btf_validate_return_type() does not apply the
+ * scalar-only restriction to a local function. Return the context pointer as
+ * the upper half and dereference it in the caller.
+ */
+struct ptr_pair {
+	void *p;
+	__u64 x;
+};
+
+static __naked __noinline struct ptr_pair static_ret_ptr_pair(void)
+{
+	asm volatile (
+	"r0 = 0;"
+	"r2 = r1;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__success __retval(0)
+__naked int aggregate_ret_static_ptr_pair(void)
+{
+	asm volatile (
+	"call %[static_ret_ptr_pair];"
+	"r1 = *(u32 *)(r2 + 0);"	/* deref the returned ctx pointer */
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(static_ret_ptr_pair)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
new file mode 100644
index 000000000000..ef10d765f738
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
@@ -0,0 +1,127 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+/*
+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address
+ * (rather than calling) avoids any dependence on the compiler lowering an
+ * __int128 or struct return value, which the BPF backend only supports from
+ * LLVM 23 on.
+ */
+void __kfunc_btf_root(void)
+{
+	asm volatile (""
+	:
+	: "r"(&bpf_kfunc_call_test_i128),
+	  "r"(&bpf_kfunc_call_test_ret_fastcall),
+	  "r"(&bpf_kfunc_call_test_ret_ptr),
+	  "r"(&bpf_kfunc_call_test_ret_ii),
+	  "r"(&bpf_kfunc_call_test_ret_big));
+}
+
+/*
+ * bpf_add_kfunc_call() rejects a kfunc returning more than 8 bytes unless the
+ * JIT advertises bpf_jit_supports_kfunc_ret_reg_pair(), so a test that has to
+ * get past it is tagged with the architectures implementing it. The two tests
+ * below that are rejected earlier, on KF_FASTCALL or on reading R2 after an
+ * 8-byte struct return, behave the same everywhere and are not tagged.
+ */
+
+SEC("tc")
+__arch_x86_64 __arch_arm64 __arch_riscv64
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
+__naked int aggregate_ret_kfunc_precise(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"r2 = 2;"
+	"call %[bpf_kfunc_call_test_i128];"
+	"r6 = r2;"		/* second return half */
+	"r6 &= 7;"		/* keep it in [0, 7] to index the stack */
+	"r1 = r10;"
+	"r1 += -8;"
+	"r1 += r6;"		/* ptr += scalar marks r6 (hence R2) precise */
+	"r0 = 0;"
+	"*(u8 *)(r1 + 0) = r0;"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_i128)
+	: __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL")
+__naked int aggregate_ret_kfunc_fastcall_fail(void)
+{
+	asm volatile (
+	"r1 = 1;"
+	"r2 = 2;"
+	"call %[bpf_kfunc_call_test_ret_fastcall];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_fastcall)
+	: __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64 __arch_riscv64
+__failure __msg("is not composed of scalars")
+__naked int aggregate_ret_kfunc_ptr_fail(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ptr];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ptr)
+	: __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_kfunc_small_no_r2(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"r2 = 0;"
+	"call %[bpf_kfunc_call_test_ret_ii];"
+	"r0 = r2;"	/* R2 is not a return register for a <=8 byte struct */
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_ii)
+	: __clobber_all);
+}
+
+/*
+ * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by
+ * btf_distill_func_proto(), before the KF_FASTCALL and JIT-capability checks,
+ * so this behaves the same on every architecture.
+ */
+SEC("tc")
+__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported")
+__naked int aggregate_ret_kfunc_too_big_fail(void)
+{
+	asm volatile (
+	"r1 = 0;"
+	"call %[bpf_kfunc_call_test_ret_big];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(bpf_kfunc_call_test_ret_big)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_run.c b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c
new file mode 100644
index 000000000000..fac813531b4f
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c
@@ -0,0 +1,168 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+typedef unsigned __int128 u128;
+
+/*
+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address
+ * (rather than calling) avoids any dependence on the compiler lowering an __int128
+ * or struct return value, which the BPF backend only supports from LLVM 23 on.
+ */
+void __kfunc_btf_root(void)
+{
+	asm volatile (""
+	:
+	: "r"(&bpf_kfunc_call_test_i128),
+	  "r"(&bpf_kfunc_call_test_ret_pair),
+	  "r"(&bpf_kfunc_call_test_ret_li),
+	  "r"(&bpf_kfunc_call_test_ret_ii),
+	  "r"(&bpf_kfunc_call_test_ret_uu));
+}
+
+#define I128_ASM_LO 0xABCDabcd12345678ULL
+#define I128_ASM_HI 0x1234567890abcdefULL
+
+static __naked __noinline u128 make_i128_asm(void)
+{
+	asm volatile (
+	"r0 = %[lo] ll;"	/* low 64 bits */
+	"r2 = %[hi] ll;"	/* high 64 bits */
+	"exit;"
+	:
+	: __imm_const(lo, I128_ASM_LO), __imm_const(hi, I128_ASM_HI)
+	);
+}
+
+SEC("tc")
+int aggregate_ret_asm_test(struct __sk_buff *skb)
+{
+	__u64 lo, hi;
+
+	asm volatile (
+	"call %[callee];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [callee]"i"(make_i128_asm)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (lo != I128_ASM_LO)
+		return 1;
+	if (hi != I128_ASM_HI)
+		return 2;
+
+	return 0;
+}
+
+SEC("tc")
+int aggregate_ret_asm_kfunc_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	__u64 lo, hi;
+
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_i128)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (hi != a + b)
+		return 1;
+	if (lo != a - b)
+		return 2;
+
+	return 0;
+}
+
+SEC("tc")
+int aggregate_ret_struct_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	__u64 lo, hi;
+
+	/* struct { u64 hi; u64 lo; }: R0 = hi, R2 = lo. */
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_pair)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (lo != a + b)
+		return 1;
+	if (hi != a - b)
+		return 2;
+
+	/* struct { u64 a; int b; }: R0 = a, low 32 bits of R2 = b. */
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_li)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (lo != a)
+		return 3;
+	if ((int)hi != ~(int)b)
+		return 4;
+
+	/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not used. */
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	: [lo]"=r"(lo)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_ii)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if ((int)lo != (int)a)
+		return 5;
+	if ((int)(lo >> 32) != (int)b)
+		return 6;
+
+	return 0;
+}
+
+SEC("tc")
+int aggregate_ret_union_test(struct __sk_buff *skb)
+{
+	__u64 a = skb->len;
+	__u64 b = skb->len ^ 0xdeadbeefULL;
+	__u64 lo, hi;
+
+	asm volatile (
+	"r1 = %[a];"
+	"r2 = %[b];"
+	"call %[kfunc];"
+	"%[lo] = r0;"
+	"%[hi] = r2;"
+	: [lo]"=r"(lo), [hi]"=r"(hi)
+	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_uu)
+	: "r0", "r1", "r2", "r3", "r4", "r5"
+	);
+	if (lo != a + b)
+		return 1;
+	if (hi != a - b)
+		return 2;
+
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
new file mode 100644
index 000000000000..cffd8d7d3241
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* freplace target: a global subprogram returning 16 bytes in R0:R2. */
+__naked unsigned __int128 agg_ret_target_func(void)
+{
+	asm volatile (
+	"r0 = 0x1234;"
+	"r2 = 0x5678;"
+	"exit;"
+	);
+}
+
+SEC("tc")
+__naked int agg_ret_target(void)
+{
+	asm volatile (
+	"call %[agg_ret_target_func];"
+	"r0 = 0;"
+	"exit;"
+	:
+	: __imm(agg_ret_target_func)
+	: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
new file mode 100644
index 000000000000..84b701402ca6
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
@@ -0,0 +1,20 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+/*
+ * An extension replaces its target outright, so it has to match the target's
+ * return convention. Its own return value is capped at 8 bytes, so it can
+ * never fill the R0:R2 pair that the target's callers read, and the attach is
+ * rejected. btf_check_type_match() cannot catch this: it compares return types
+ * by btf_type->info only, and an int carries no vlen, so the __u64 here and
+ * the target's __int128 compare equal.
+ */
+SEC("freplace/agg_ret_target_func")
+__u64 new_agg_ret_target_func(void)
+{
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_arena.c b/tools/testing/selftests/bpf/progs/verifier_arena.c
index b241bbcf54a8..455b55296f35 100644
--- a/tools/testing/selftests/bpf/progs/verifier_arena.c
+++ b/tools/testing/selftests/bpf/progs/verifier_arena.c
@@ -704,4 +704,42 @@ int check_arena_arg_ret(void *ctx)
 	return 0;
 }
 
+struct arena_ret_pair {
+	__u64 lo;
+	__u64 hi;
+};
+
+/*
+ * A 16-byte value is returned in the R0:R2 register pair. A global subprogram
+ * may return an arena pointer in R0, but R2 holds the upper half of a scalar
+ * pair, so an arena pointer there is not a valid return value. The ld_imm64 of
+ * the arena map is what links the arena to the program, without which the
+ * addr_space_cast insn is not allowed.
+ */
+__naked struct arena_ret_pair global_ret_arena_ptr_in_r2(void)
+{
+	asm volatile (
+		"r1 = %[arena] ll;"
+		"r2 = 8192;"
+		"r2 = addr_space_cast(r2, 0x0, 0x1);"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm_addr(arena)
+		: __clobber_all);
+}
+
+SEC("syscall")
+__failure __msg("At subprogram exit the register R2 is not a scalar value (arena)")
+__naked int check_global_ret_arena_ptr_in_r2(void)
+{
+	asm volatile (
+		"call %[global_ret_arena_ptr_in_r2];"
+		"r0 = 0;"
+		"exit;"
+		:
+		: __imm(global_ret_arena_ptr_in_r2)
+		: __clobber_all);
+}
+
 char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 2ceb34df472e..5d13482b0d2e 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -869,6 +869,50 @@ __bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)
 	return r;
 }
 
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)
+{
+	struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_li bpf_kfunc_call_test_ret_li(u64 a, int b)
+{
+	struct prog_test_ret_li r = { .a = a, .b = ~b };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)
+{
+	struct prog_test_ret_ii r = { .a = a, .b = b };
+
+	return r;
+}
+
+__bpf_kfunc union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(u64 a, u64 b)
+{
+	union prog_test_ret_uu r;
+
+	r.halves[0] = a + b;
+	r.halves[1] = a - b;
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)
+{
+	struct prog_test_ret_ptr r = { .p = NULL, .tag = tag };
+
+	return r;
+}
+
+__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(u64 a)
+{
+	struct prog_test_ret_big r = { .a = a, .b = a, .c = a };
+
+	return r;
+}
+
 __bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
 					 u64 e, u64 f, u64 g, u64 h,
 					 u64 i, u64 j)
@@ -1404,6 +1448,12 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
 BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_li)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_uu)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
 BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index e6e59fdce33c..917e943bd963 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -64,6 +64,35 @@ struct prog_test_ret_pair {
 	__u64 lo;
 };
 
+struct prog_test_ret_li {	/* 16 bytes: R0:R2 */
+	__u64 a;
+	int b;
+};
+
+struct prog_test_ret_ii {	/* 8 bytes: R0 only */
+	int a;
+	int b;
+};
+
+union prog_test_ret_uu {	/* 16 bytes: R0:R2 */
+	__u64 halves[2];
+	struct {
+		__u64 lo;
+		__u64 hi;
+	} parts;
+};
+
+struct prog_test_ret_ptr {	/* 16 bytes: contains a pointer */
+	void *p;
+	__u64 tag;
+};
+
+struct prog_test_ret_big {	/* 24 bytes: too large for R0:R2 */
+	__u64 a;
+	__u64 b;
+	__u64 c;
+};
+
 struct prog_test_fail1 {
 	void *p;
 	int x;
@@ -129,6 +158,12 @@ long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
 int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
 __int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
 struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_li bpf_kfunc_call_test_ret_li(__u64 a, int b) __ksym;
+struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
+union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;
+struct prog_test_ret_big bpf_kfunc_call_test_ret_big(__u64 a) __ksym;
 __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
 			       __u64 e, __u64 f, __u64 g, __u64 h,
 			       __u64 i, __u64 j) __ksym;
-- 
2.53.0-Meta

[PATCH bpf-next v2 12/13] selftests/bpf: Add tests for callbacks returning more than 8 bytes

[Yonghong Song]

Add two __failure tests covering the callback return-size checks:

 - timer_ret_pair_fail: a bpf_timer callback declared to return more than
   8 bytes, rejected by check_ld_imm() where the callback's PTR_TO_FUNC is
   created, with "callback function with >8-byte return value is not
   supported".

 - exceptions_ret_pair_fail: an exception callback declared to return more
   than 8 bytes, rejected by do_check_common() when the callback
   subprogram is verified, with "exception cb cannot return value larger
   than 8 bytes".

Both callback bodies are written in inline asm so that the tests do not
depend on LLVM 23 R0:R2 codegen and run on any compiler. The verifier reads
the return type from BTF rather than from the instructions, so the >8 byte
return prototype is supplied through __btf_func_path(), pointing at a
companion btf__*.c program that exists only to carry that BTF.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 .../selftests/bpf/prog_tests/exceptions.c     |  2 +
 .../testing/selftests/bpf/prog_tests/timer.c  |  2 +
 .../bpf/progs/btf__exceptions_ret_pair_fail.c | 10 ++++
 .../bpf/progs/btf__timer_ret_pair_fail.c      | 10 ++++
 .../bpf/progs/exceptions_ret_pair_fail.c      | 30 ++++++++++++
 .../selftests/bpf/progs/timer_ret_pair_fail.c | 49 +++++++++++++++++++
 6 files changed, 103 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/btf__exceptions_ret_pair_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__timer_ret_pair_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/exceptions_ret_pair_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/timer_ret_pair_fail.c

diff --git a/tools/testing/selftests/bpf/prog_tests/exceptions.c b/tools/testing/selftests/bpf/prog_tests/exceptions.c
index 3588d6f97fd4..71d00c568d80 100644
--- a/tools/testing/selftests/bpf/prog_tests/exceptions.c
+++ b/tools/testing/selftests/bpf/prog_tests/exceptions.c
@@ -5,6 +5,7 @@
 #include "exceptions.skel.h"
 #include "exceptions_ext.skel.h"
 #include "exceptions_fail.skel.h"
+#include "exceptions_ret_pair_fail.skel.h"
 #include "exceptions_assert.skel.h"
 
 static char log_buf[1024 * 1024];
@@ -12,6 +13,7 @@ static char log_buf[1024 * 1024];
 static void test_exceptions_failure(void)
 {
 	RUN_TESTS(exceptions_fail);
+	RUN_TESTS(exceptions_ret_pair_fail);
 }
 
 static void test_exceptions_success(void)
diff --git a/tools/testing/selftests/bpf/prog_tests/timer.c b/tools/testing/selftests/bpf/prog_tests/timer.c
index 09ff21e1ad2f..593e56d8964e 100644
--- a/tools/testing/selftests/bpf/prog_tests/timer.c
+++ b/tools/testing/selftests/bpf/prog_tests/timer.c
@@ -6,6 +6,7 @@
 #include <sys/syscall.h>
 #include "timer.skel.h"
 #include "timer_failure.skel.h"
+#include "timer_ret_pair_fail.skel.h"
 #include "timer_interrupt.skel.h"
 
 #define NUM_THR 8
@@ -285,6 +286,7 @@ void serial_test_timer(void)
 	test_timer(timer);
 
 	RUN_TESTS(timer_failure);
+	RUN_TESTS(timer_ret_pair_fail);
 }
 
 void serial_test_timer_stress(void)
diff --git a/tools/testing/selftests/bpf/progs/btf__exceptions_ret_pair_fail.c b/tools/testing/selftests/bpf/progs/btf__exceptions_ret_pair_fail.c
new file mode 100644
index 000000000000..a45db5d9c1d4
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/btf__exceptions_ret_pair_fail.c
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+unsigned __int128 exception_cb_bad_ret_type3(u64 cookie)
+{
+	for (;;)
+		;
+}
diff --git a/tools/testing/selftests/bpf/progs/btf__timer_ret_pair_fail.c b/tools/testing/selftests/bpf/progs/btf__timer_ret_pair_fail.c
new file mode 100644
index 000000000000..35506c7c5a91
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/btf__timer_ret_pair_fail.c
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+unsigned __int128 timer_cb_ret_pair(void *map, int *key, struct bpf_timer *timer)
+{
+	for (;;)
+		;
+}
diff --git a/tools/testing/selftests/bpf/progs/exceptions_ret_pair_fail.c b/tools/testing/selftests/bpf/progs/exceptions_ret_pair_fail.c
new file mode 100644
index 000000000000..842f86ad8659
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/exceptions_ret_pair_fail.c
@@ -0,0 +1,30 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_helpers.h>
+
+#include "bpf_misc.h"
+#include "bpf_experimental.h"
+
+__naked __noinline __used
+unsigned __int128 exception_cb_bad_ret_type3(u64 cookie)
+{
+	asm volatile (
+	"r0 = r1;"
+	"r2 = 0;"
+	"exit;"
+	::: __clobber_all);
+}
+
+SEC("?tc")
+__exception_cb(exception_cb_bad_ret_type3)
+__failure __msg("exception cb cannot return value larger than 8 bytes")
+__btf_func_path("btf__exceptions_ret_pair_fail.bpf.o")
+int reject_exception_cb_ret_pair(void *ctx)
+{
+	bpf_throw(0);
+	return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/timer_ret_pair_fail.c b/tools/testing/selftests/bpf/progs/timer_ret_pair_fail.c
new file mode 100644
index 000000000000..29fd294dfd49
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/timer_ret_pair_fail.c
@@ -0,0 +1,49 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+
+#include <linux/bpf.h>
+#include <time.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "bpf_misc.h"
+
+char _license[] SEC("license") = "GPL";
+
+struct elem {
+	struct bpf_timer t;
+};
+
+struct {
+	__uint(type, BPF_MAP_TYPE_ARRAY);
+	__uint(max_entries, 1);
+	__type(key, int);
+	__type(value, struct elem);
+} timer_map SEC(".maps");
+
+__naked __noinline __used
+static unsigned __int128 timer_cb_ret_pair(void *map, int *key, struct bpf_timer *timer)
+{
+	asm volatile (
+		"r0 = 0;"
+		"r2 = 0;"
+		"exit;"
+		::: __clobber_all
+	);
+}
+
+SEC("fentry/bpf_fentry_test1")
+__failure __msg("callback function with >8-byte return value is not supported")
+__btf_func_path("btf__timer_ret_pair_fail.bpf.o")
+long BPF_PROG2(test_bad_ret_pair, int, a)
+{
+	int key = 0;
+	struct bpf_timer *timer;
+
+	timer = bpf_map_lookup_elem(&timer_map, &key);
+	if (timer) {
+		bpf_timer_init(timer, &timer_map, CLOCK_BOOTTIME);
+		bpf_timer_set_callback(timer, timer_cb_ret_pair);
+	}
+
+	return 0;
+}
-- 
2.53.0-Meta

[PATCH bpf-next v2 13/13] Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2

[Yonghong Song]

kfuncs may now return a value larger than 8 bytes and up to 16 bytes (a
scalar-only struct or union, or an __int128), passed back in the R0:R2
register pair. Add a kfunc return-value section documenting this,
including that a struct or union up to 8 bytes is returned in R0 alone,
which struct and union members are accepted, that the R0:R2 register pair
requires JIT support (bpf_jit_supports_kfunc_ret_reg_pair()), and that a
return value larger than 16 bytes is unsupported.

Also note that the same convention applies to BPF subprogram returns, and
document the consequence for a global subprogram: it must assign both
halves of a register-pair return, since an unassigned R2 may be left
holding a pointer argument and is then rejected as a leak.

Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
 Documentation/bpf/kfuncs.rst | 62 ++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index cbde86d082cc..e3306f2f9d70 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -529,6 +529,68 @@ is also covered by this recovery. A kfunc handed an arena pointer may
 therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
 against the arena. Larger accesses must verify the range explicitly.
 
+2.9 kfunc Return Values
+-----------------------
+
+A kfunc may return a scalar, a pointer, or a small struct or union by
+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.
+
+A struct or union returned by value must be composed only of scalars
+(recursively), where a scalar is an integer or an enum; arrays of scalars are
+allowed as members. Its bytes are handed back to the program as the raw
+contents of R0 (and R2), so a pointer field would be laundered into a scalar
+and escape the verifier's pointer provenance and reference tracking. A struct
+or union with a pointer member is therefore rejected at load time, and so is
+one with a floating-point member, which the ABI may not return in R0:R2 at
+all.
+
+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a
+scalar-only struct or union, or an ``__int128``. Such a value is returned
+in the register pair R0:R2, matching the convention LLVM uses for the BPF
+target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or
+union of 8 bytes or less is returned in R0 alone.
+
+::
+
+        struct bpf_pair { __u64 a, b; };   /* 16 bytes */
+
+        __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)
+        {
+                struct bpf_pair p = { .a = 1, .b = 2 };
+
+                return p;      /* p.a in R0, p.b in R2 */
+        }
+
+Returning a value in the R0:R2 pair requires the JIT to place the second
+half of the return value into R2, which not every architecture supports
+right now. A kfunc with a return value larger than 8 bytes is therefore
+rejected at load time on a JIT that does not advertise this capability (see
+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run
+by the interpreter. A return value larger than 16 bytes is not supported.
+
+The same R0:R2 convention applies to a BPF subprogram, global or static,
+that returns an ``__int128`` or a struct or union larger than 8 bytes. Such a
+program also requires the JIT, since the interpreter propagates only R0 out
+of a subprogram. A global subprogram is verified in isolation, so its
+by-value struct or union return is restricted to scalars just like a kfunc's;
+a static subprogram is verified inline and has no such restriction. The main
+program cannot return more than 8 bytes, as its return value is the program's
+exit code.
+
+A global subprogram must leave a scalar in *every* register of the pair, so
+both halves of the returned value have to be assigned. Leaving the upper half
+uninitialized is not merely untidy: the compiler is then free to leave R2
+holding whatever it happened to hold, which for a subprogram taking a pointer
+argument is typically that pointer. Handing the caller an unknown scalar built
+from a pointer is a leak, so the verifier rejects it with::
+
+        At subprogram exit the register R2 is not a scalar value (...)
+
+Initialize the whole return value, for example ``struct pair p = {};``, to
+avoid this. A static subprogram is exempt: it is verified inline, so an
+unassigned R2 is simply passed back to the caller as uninitialized and only a
+caller that reads it fails.
+
 .. _BPF_kfunc_lifecycle_expectations:
 
 3. kfunc lifecycle expectations
-- 
2.53.0-Meta

Re: [PATCH bpf-next v2 10/13] selftests/bpf: Add C tests for 16-byte returns in R0:R2

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Unconditional use of `__int128` in the `bpf_testmod` kernel module causes build failures on architectures lacking 128-bit integer support.
- [Low] Redundant BPF program FD validation after successful skeleton load.
--

--- Patch [10]: [PATCH bpf-next v2 10/13] selftests/bpf: Add C tests for 16-byte returns in R0:R2 ---
commit 9e3dc1e0617b6a9acfcd02529cae781809d11bbc
Author: Yonghong Song <yonghong.song@linux.dev>

selftests/bpf: Add C tests for 16-byte returns in R0:R2

Add selftests that exercise a 16-byte return value passed in the R0:R2
register pair, written in C so that they depend on the compiler lowering
the register-pair return. Covered are an __int128 return, a 16-byte struct
return (from a static and from a global subprogram) and a 16-byte union
return, plus __int128 and 16-byte struct returns from a kfunc, for which
bpf_kfunc_call_test_i128() and bpf_kfunc_call_test_ret_pair() are added to
bpf_testmod.

> diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
> new file mode 100644
> index 0000000000000..42017f89c4da6
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
> @@ -0,0 +1,125 @@
[ ... ]
> +static void run_prog(struct bpf_program *prog, bool supported)
> +{
> +	char buf[64] = {};
> +	int err, prog_fd;
> +	LIBBPF_OPTS(bpf_test_run_opts, topts,
> +		    .data_in = buf,
> +		    .data_size_in = sizeof(buf),
> +		    .repeat = 1,
> +	);
> +
> +	if (!supported) {
> +		test__skip();
> +		return;
> +	}
> +
> +	prog_fd = bpf_program__fd(prog);
> +	if (!ASSERT_GE(prog_fd, 0, "prog_fd"))
> +		return;

[Severity: Low]
Is this manual FD validation necessary here? According to the BPF skeleton API
guarantees, after a successful skeleton load, skeleton-generated program
fields are guaranteed to have valid FDs. Since run_prog() is called using
programs from successfully loaded skeletons, the extra validation appears to
be redundant.

[ ... ]
> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
> index eb0f9b5e18d85..2ceb34df472e6 100644
> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
> @@ -857,6 +857,18 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
>  	return 0;
>  }
>  
> +__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
> +{
> +	return (__int128)(((unsigned __int128)(a + b) << 64) | (a - b));
> +}

[Severity: Medium]
Will this unconditional use of __int128 cause build failures on 32-bit
architectures? It appears bpf_kfunc_call_test_i128 is not protected by
an #ifdef __SIZEOF_INT128__ guard, which might break compilation of the
bpf_testmod module on platforms lacking 128-bit integer support.

[ ... ]
> @@ -1390,6 +1402,8 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
>  BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
>  BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
>  BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
> +BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)

Should this BTF ID registration also be guarded to prevent compilation
or pahole errors when the function is omitted on 32-bit builds?

> +BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)
>  BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
[ ... ]
> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> index c36bb911defa7..e6e59fdce33cf 100644
> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> @@ -118,6 +127,8 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
>  struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
>  long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
>  int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
> +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;

Does this unconditionally declared prototype also need to be wrapped
with an #ifdef __SIZEOF_INT128__ guard?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260804203522.1869244-1-yonghong.song@linux.dev?part=10

Re: [PATCH bpf-next v2 07/13] bpf: Add verifier support for 16-byte returns in R0:R2

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Kfuncs returning 9-15 byte structs leak uninitialized kernel memory via R2 because the verifier bypasses zero-extension.
--

--- Patch [7]: [PATCH bpf-next v2 07/13] bpf: Add verifier support for 16-byte returns in R0:R2 ---
commit 5c235db7cbd600a1b91b24fbad96652d702b6b3a
Author: Yonghong Song <yonghong.song@linux.dev>

bpf: Add verifier support for 16-byte returns in R0:R2

This commit adds verifier support for returning values in two registers
(R0 and R2) for structures between 9 and 16 bytes or __int128. It adds
handling for subprogram and kfunc returns to map the second return
register to BPF_REG_2.

[ ... ]

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 5584178a0e1c7..60b9e587e0947 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -10859,6 +10901,22 @@ static void mark_btf_func_reg_size(struct bpf_verifier_env *env, u32 regno,
>  	return __mark_btf_func_reg_size(env, cur_regs(env), regno, reg_size);
>  }
>  
> +/* Mark the register(s) holding a @size byte kfunc return value as unknown
> + * scalars. All of them are processed the same way, only the size differs:
> + * a single register may hold a sub-register sized value, while both halves
> + * of a register pair are treated as 64-bit wide.
> + */
> +static void mark_kfunc_ret_regs(struct bpf_verifier_env *env,
> +				struct bpf_reg_state *regs, u32 size)
> +{
> +	u32 i, nregs = ret_regs_cnt(size);
> +
> +	for (i = 0; i < nregs; i++) {
> +		mark_reg_unknown(env, regs, ret_regs[i]);
> +		mark_ret_reg_size(env, regs, ret_regs[i], nregs == 1 ? size : sizeof(u64));

[Severity: Medium]
For kfuncs returning a struct between 9 and 15 bytes in size, does
forcing sizeof(u64) for both return registers bypass zero-extension for R2?

If the native compiler only populates the lower bytes of the second return
register (e.g. RDX on x86_64) for a <16 byte struct, the upper bits will be
left uninitialized. However, setting the size to sizeof(u64) here sets
subreg_def to DEF_NOT_SUBREG for R2, which treats R2 as a fully initialized
64-bit scalar.

Since the verifier's zero-extension logic only supports zero-extending the
destination register (R0 for calls), couldn't a BPF program performing a
64-bit read of R2 leak up to 7 bytes of uninitialized kernel register state?

> +	}
> +}

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260804203522.1869244-1-yonghong.song@linux.dev?part=7

Re: [PATCH bpf-next v2 11/13] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The inline assembly tests verifying kfunc struct returns and __int128 values incorrectly assume little-endian register packing, which will cause deterministic test failures on big-endian architectures.
--

--- Patch [11]: [PATCH bpf-next v2 11/13] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns ---
commit 43d5815a2f48399212b5d17d74ef9350c219fc0e
Author: Yonghong Song <yonghong.song@linux.dev>

selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns

Add inline-asm tests, which do not depend on the compiler lowering a
register-pair return and so run regardless of the LLVM version, covering
what the C tests cannot reach.

> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_run.c b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c
> new file mode 100644
> index 0000000000000..fac813531b4fe
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_run.c

[ ... ]

> +SEC("tc")
> +int aggregate_ret_asm_kfunc_test(struct __sk_buff *skb)
> +{
> +	__u64 a = skb->len;
> +	__u64 b = skb->len ^ 0xdeadbeefULL;
> +	__u64 lo, hi;
> +
> +	asm volatile (
> +	"r1 = %[a];"
> +	"r2 = %[b];"
> +	"call %[kfunc];"
> +	"%[lo] = r0;"
> +	"%[hi] = r2;"
> +	: [lo]"=r"(lo), [hi]"=r"(hi)
> +	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_i128)
> +	: "r0", "r1", "r2", "r3", "r4", "r5"
> +	);
> +	if (hi != a + b)
> +		return 1;
> +	if (lo != a - b)
> +		return 2;

[Severity: Medium]
Does this assume little-endian register packing for the __int128 return value?

On big-endian architectures, the high bits typically occupy the first register
(R0) and the low bits occupy R2, which would cause this check to fail when
testing kfunc __int128 returns.

[ ... ]

> +SEC("tc")
> +int aggregate_ret_struct_test(struct __sk_buff *skb)
> +{
> +	__u64 a = skb->len;
> +	__u64 b = skb->len ^ 0xdeadbeefULL;
> +	__u64 lo, hi;
> +
> +	/* struct { u64 hi; u64 lo; }: R0 = hi, R2 = lo. */
> +	asm volatile (
> +	"r1 = %[a];"
> +	"r2 = %[b];"
> +	"call %[kfunc];"
> +	"%[lo] = r0;"
> +	"%[hi] = r2;"
> +	: [lo]"=r"(lo), [hi]"=r"(hi)
> +	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_pair)
> +	: "r0", "r1", "r2", "r3", "r4", "r5"
> +	);
> +	if (lo != a + b)
> +		return 1;
> +	if (hi != a - b)
> +		return 2;
> +
> +	/* struct { u64 a; int b; }: R0 = a, low 32 bits of R2 = b. */
> +	asm volatile (
> +	"r1 = %[a];"
> +	"r2 = %[b];"
> +	"call %[kfunc];"
> +	"%[lo] = r0;"
> +	"%[hi] = r2;"
> +	: [lo]"=r"(lo), [hi]"=r"(hi)
> +	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_li)
> +	: "r0", "r1", "r2", "r3", "r4", "r5"
> +	);
> +	if (lo != a)
> +		return 3;
> +	if ((int)hi != ~(int)b)
> +		return 4;

[Severity: Medium]
Will this check fail deterministically on big-endian systems?

The test expects the 32-bit integer b to be in the lower 32 bits of R2. On
big-endian platforms, b would occupy the upper 32 bits of R2, leaving the
lower 32 bits as padding.

> +	/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not used. */
> +	asm volatile (
> +	"r1 = %[a];"
> +	"r2 = %[b];"
> +	"call %[kfunc];"
> +	"%[lo] = r0;"
> +	: [lo]"=r"(lo)
> +	: [a]"r"(a), [b]"r"(b), [kfunc]"i"(bpf_kfunc_call_test_ret_ii)
> +	: "r0", "r1", "r2", "r3", "r4", "r5"
> +	);
> +	if ((int)lo != (int)a)
> +		return 5;
> +	if ((int)(lo >> 32) != (int)b)
> +		return 6;
> +
> +	return 0;
> +}

[Severity: Medium]
Could the layout of this 8-byte struct in R0 differ depending on endianness?

The hardcoded cast to (int)lo reads the lower 32 bits of R0. On big-endian
architectures, the first member of the struct (a) is located in the upper 32
bits, so reading the lower 32 bits will yield b instead, causing a spurious
test failure.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260804203522.1869244-1-yonghong.song@linux.dev?part=11

Re: [PATCH bpf-next v2 05/13] bpf: Account R2 of register-pair returns in live register analysis

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] The verifier fails to check if BPF_REG_2 is initialized at BPF_EXIT for global subprograms returning > 8 bytes, which can lead to leaking uninitialized memory.
- [High] The verifier fails to copy the callee's R2 to the caller's R2 in prepare_func_exit() for static subprograms returning > 8 bytes.
--

--- Patch [5]: [PATCH bpf-next v2 05/13] bpf: Account R2 of register-pair returns in live register analysis ---
commit 149c9afee6775ff575a63e08693b340337d96491
Author: Yonghong Song <yonghong.song@linux.dev>

bpf: Account R2 of register-pair returns in live register analysis

This commit updates the live register analysis in the BPF verifier to mark R2
as read at BPF_EXIT for subprograms returning a value larger than 8 bytes. It
uses bpf_ret_reg_pair() to determine if a subprogram returns its value in the
R0:R2 register pair.

> diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
> index 0aadfbae0acc5..68df6d8db4bda 100644
> --- a/kernel/bpf/liveness.c
> +++ b/kernel/bpf/liveness.c
[ ... ]
> @@ -2174,7 +2179,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
>  			break;
>  		case BPF_EXIT:
>  			def = 0;
> -			use = r0;
> +			use = ret_reg_pair ? (r0 | BIT(BPF_REG_2)) : r0;

[Severity: High]
Does check_global_subprog_return_code() also need to check BPF_REG_2 when the
subprogram returns a register pair?

If a global BPF subprogram returns a 16-byte value (e.g., an __int128 or
struct) but does not write to R2 before exiting, the verifier only validates
that BPF_REG_0 is initialized:

kernel/bpf/verifier.c:check_global_subprog_return_code() {
    ...
	return check_global_ret_scalar_reg(env, BPF_REG_0);
}

Although check_global_ret_scalar_reg() was refactored earlier in this series
so that it can be reused for R2, there appears to be no call added to check R2
in check_global_subprog_return_code().

Can this allow a global subprogram to leak uninitialized kernel memory or
pointers in the CPU register corresponding to R2 back to the kernel caller?

[Severity: High]
Additionally, should prepare_func_exit() copy the callee's R2 to the caller's
R2 for static subprograms returning > 8 bytes?

Currently, prepare_func_exit() only copies R0:

kernel/bpf/verifier.c:prepare_func_exit() {
    ...
	} else {
		/* return to the caller whatever r0 had in the callee */
		caller->regs[BPF_REG_0] = *r0;
	}
    ...
}

If a BPF program calls a static subprogram that returns a 16-byte value, and
the caller attempts to use the upper 8 bytes returned in R2, won't the verifier
reject the valid program with a read of uninitialized register error? 

This happens because the caller's R2 was marked NOT_INIT prior to the call by
clear_caller_saved_regs(), but prepare_func_exit() does not propagate the
callee's R2 value back to the caller.

>  			break;
>  		case BPF_CALL:
>  			def = ALL_CALLER_SAVED_REGS;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260804203522.1869244-1-yonghong.song@linux.dev?part=5

Re: [PATCH bpf-next v2 06/13] bpf: Reject callbacks returning more than 8 bytes

[bot+bpf-ci]

[-- Attachment #1: Type: text/plain, Size: 1739 bytes --]

> commit 9c43eef7a91393c06822d44c361ee806f383cf97
> Author: Yonghong Song <yonghong.song@linux.dev>
>
>     bpf: Reject callbacks returning more than 8 bytes
>
>     A callback handed to a helper or a kfunc (bpf_loop(),
>     bpf_timer_set_callback(), bpf_for_each_map_elem(), ...) is invoked
>     through bpf_callback_t, and an exception callback is invoked by
>     bpf_throw() through
>
>       u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
>
>     Both prototypes yield a single u64 in R0, and neither caller has any
>     notion of a second return register, so a callback returning a value in
>     the R0:R2 pair would have the upper half of its return value silently
>     dropped.
>
>     Reject both at load time:
>
>      - check_ld_imm(): a callback is materialized as PTR_TO_FUNC by an
>        ld_imm64 pointing at its subprogram, so the subprogram's return
>        convention can be checked where the callback pointer is created,
>        before it ever reaches a helper or kfunc argument.
>
>      - do_check_common(): an exception callback is not referenced by a
>        PTR_TO_FUNC, it is named by a BTF decl_tag and verified on its own,
>        so check it as its frame is set up, next to the existing "cannot
>        return void" and single-argument checks.
>
>     Signed-off-by: Yonghong Song <yonghong.song@linux.dev>

This looks like a bug fix. Should this include:

  Fixes: 63b5669f8a0d ("bpf: Add helpers to describe the R0:R2 return register pair")


---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/30949310795