* [PATCH bpf-next v5 01/11] bpf: Factor check_global_ret_scalar_reg() out of the global return check
2026-08-13 20:02 [PATCH bpf-next v5 00/11] bpf: Support aggregate return values up to 16 bytes Yonghong Song
@ 2026-08-13 20:02 ` Yonghong Song
2026-08-13 20:02 ` [PATCH bpf-next v5 02/11] bpf: Add helpers to describe the R0:R2 return register pair Yonghong Song
` (9 subsequent siblings)
10 siblings, 0 replies; 18+ messages in thread
From: Yonghong Song @ 2026-08-13 20:02 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
check_global_subprog_return_code() verifies that a global subprogram
returns void, an arena pointer, or register R0 holding a scalar value.
Later patches in this series add 16-byte aggregate return support, whose
second half is returned in R2 and needs the same validation.
Factor the per-register check into check_global_ret_scalar_reg(env, regno)
so that it can be reused for R2. No functional change.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/verifier.c | 32 ++++++++++++++++++++------------
1 file changed, 20 insertions(+), 12 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 6ac1afced20b..03570c693d35 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -16654,37 +16654,45 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
return 0;
}
-static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
{
- struct bpf_reg_state *reg = reg_state(env, BPF_REG_0);
- struct bpf_func_state *cur_frame = cur_func(env);
+ struct bpf_reg_state *reg;
int err;
- if (subprog_returns_void(env, cur_frame->subprogno))
- return 0;
-
- err = check_reg_arg(env, BPF_REG_0, SRC_OP);
+ err = check_reg_arg(env, regno, SRC_OP);
if (err)
return err;
/* Pointers to arena are safe to pass between subprograms. */
- if (is_arena_reg(env, BPF_REG_0))
+ if (is_arena_reg(env, regno))
return 0;
- if (is_pointer_value(env, BPF_REG_0)) {
- verbose(env, "R%d leaks addr as return value\n", BPF_REG_0);
+ if (is_pointer_value(env, regno)) {
+ verbose(env, "R%d leaks addr as return value\n", regno);
return -EACCES;
}
+ reg = reg_state(env, regno);
if (reg->type != SCALAR_VALUE) {
- verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n",
- reg_type_str(env, reg->type));
+ verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n",
+ regno, reg_type_str(env, reg->type));
return -EINVAL;
}
return 0;
}
+static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+{
+ struct bpf_func_state *cur_frame = cur_func(env);
+ u32 subprog = cur_frame->subprogno;
+
+ if (subprog_returns_void(env, subprog))
+ return 0;
+
+ return check_global_ret_scalar_reg(env, BPF_REG_0);
+}
+
/* Bitmask with 1s for all caller saved registers */
#define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH bpf-next v5 02/11] bpf: Add helpers to describe the R0:R2 return register pair
2026-08-13 20:02 [PATCH bpf-next v5 00/11] bpf: Support aggregate return values up to 16 bytes Yonghong Song
2026-08-13 20:02 ` [PATCH bpf-next v5 01/11] bpf: Factor check_global_ret_scalar_reg() out of the global return check Yonghong Song
@ 2026-08-13 20:02 ` Yonghong Song
2026-08-13 21:11 ` bot+bpf-ci
2026-08-13 20:02 ` [PATCH bpf-next v5 03/11] bpf: Wire up JIT support for 16-byte kfunc returns Yonghong Song
` (8 subsequent siblings)
10 siblings, 1 reply; 18+ messages in thread
From: Yonghong Song @ 2026-08-13 20:02 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more than
16 bytes: such a value comes back in the R0:R2 register pair, with R2
holding the upper half. See LLVM patches [1] and [2].
Later patches teach the JIT, precision backtracking, live register analysis
and the verifier itself about that convention. All of them need to answer
the same question: does this call return its value in a register pair? Add
the shared helpers up front so they can be used in subsequent patches:
- bpf_ret_reg_pair() for a BPF subprogram, answered from a per-subprogram
flag that bpf_compute_subprog_ret_regs() derives once from the BTF
prototype;
- bpf_kfunc_ret_reg_pair() for a kfunc call site, answered from the
btf_func_model that bpf_add_kfunc_call() already built, which is the
same ret_size the JIT keys the second return register off, so the
verifier and the generated code cannot disagree.
It also sets jit_required, from the same place the convention is decided
rather than from each site that later comes to model R2, so that no such
site can be missed.
[1] https://github.com/llvm/llvm-project/pull/190894
[2] https://github.com/llvm/llvm-project/pull/206876
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
include/linux/bpf_verifier.h | 9 +++++
kernel/bpf/verifier.c | 73 ++++++++++++++++++++++++++++++------
2 files changed, 71 insertions(+), 11 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 27b43fda9b17..bffd32dca068 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -814,6 +814,8 @@ struct bpf_subprog_info {
bool is_async_cb: 1;
bool is_exception_cb: 1;
bool args_cached: 1;
+ /* true if the return value is passed in the R0:R2 register pair */
+ bool ret_reg_pair: 1;
/* true if bpf_fastcall stack region is used by functions that can't be inlined */
bool keep_fastcall_stack: 1;
bool changes_pkt_data: 1;
@@ -1048,6 +1050,13 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
return &env->subprog_info[subprog];
}
+static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+ return subprog_info(env, subprog)->ret_reg_pair;
+}
+
+bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env *env, struct bpf_insn *insn);
+
struct bpf_call_summary {
u8 num_params;
bool is_void;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 03570c693d35..57d14480ded2 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -382,27 +382,60 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
}
-static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)
{
- const struct btf_type *type, *func, *func_proto;
+ const struct btf_type *func, *func_proto;
const struct btf *btf = env->prog->aux->btf;
u32 btf_id;
+ if (!btf || !env->prog->aux->func_info)
+ return NULL;
+
btf_id = env->prog->aux->func_info[subprog].type_id;
+ /* Both already validated by prepare_btf_func() at prog load. */
func = btf_type_by_id(btf, btf_id);
- if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id))
- return false;
-
func_proto = btf_type_by_id(btf, func->type);
- if (!func_proto)
- return false;
- type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
- if (!type)
- return false;
+ return btf_type_skip_modifiers(btf, func_proto->type, NULL);
+}
+
+static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+{
+ const struct btf_type *type = subprog_ret_type(env, subprog);
+
+ return type && btf_type_is_void(type);
+}
+
+static u32 ret_regs_cnt(u32 size)
+{
+ return size > 8 && size <= 16 ? 2 : 1;
+}
+
+static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
+{
+ const struct btf *btf = env->prog->aux->btf;
+ const struct btf_type *type;
+ int subprog;
+ u32 size;
- return btf_type_is_void(type);
+ for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+ type = subprog_ret_type(env, subprog);
+ if (!type || !(btf_type_is_struct(type) || btf_type_is_scalar(type)))
+ continue;
+ if (IS_ERR(btf_resolve_size(btf, type, &size)))
+ continue;
+ if (ret_regs_cnt(size) > 1) {
+ subprog_info(env, subprog)->ret_reg_pair = true;
+ /*
+ * The R0:R2 return convention is only implemented in
+ * the JIT: the interpreter propagates BPF_R0 alone out
+ * of a subprogram, so a caller reading R2 would see a
+ * stale value.
+ */
+ env->prog->jit_required = 1;
+ }
+ }
}
static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
@@ -2456,6 +2489,21 @@ find_kfunc_desc(const struct bpf_prog *prog, u32 func_id, u16 offset)
sizeof(tab->descs[0]), kfunc_desc_cmp_by_id_off);
}
+/*
+ * True if the kfunc called by @insn returns its value in the R0:R2 pair.
+ * Reads the same btf_func_model.ret_size that bpf_add_kfunc_call() validated
+ * and that the JIT keys the second return register off, so the verifier and
+ * the generated code cannot disagree about the convention.
+ */
+bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+ const struct bpf_kfunc_desc *desc;
+
+ desc = find_kfunc_desc(env->prog, insn->imm, insn->off);
+
+ return desc && ret_regs_cnt(desc->func_model.ret_size) > 1;
+}
+
int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
u16 btf_fd_idx, u8 **func_addr)
{
@@ -20333,6 +20381,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
+ /* must precede the first bpf_ret_reg_pair() user below */
+ bpf_compute_subprog_ret_regs(env);
+
ret = bpf_compute_live_registers(env);
if (ret < 0)
goto skip_full_check;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 18+ messages in thread* Re: [PATCH bpf-next v5 02/11] bpf: Add helpers to describe the R0:R2 return register pair
2026-08-13 20:02 ` [PATCH bpf-next v5 02/11] bpf: Add helpers to describe the R0:R2 return register pair Yonghong Song
@ 2026-08-13 21:11 ` bot+bpf-ci
0 siblings, 0 replies; 18+ messages in thread
From: bot+bpf-ci @ 2026-08-13 21:11 UTC (permalink / raw)
To: yonghong.song, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 4900 bytes --]
> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
> --- a/include/linux/bpf_verifier.h
> +++ b/include/linux/bpf_verifier.h
> @@ -814,6 +814,8 @@ struct bpf_subprog_info {
> bool is_async_cb: 1;
> bool is_exception_cb: 1;
> bool args_cached: 1;
> + /* true if the return value is passed in the R0:R2 register pair */
> + bool ret_reg_pair: 1;
> /* true if bpf_fastcall stack region is used by functions that can't be inlined */
> bool keep_fastcall_stack: 1;
> bool changes_pkt_data: 1;
> @@ -1048,6 +1050,13 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env, int subprog)
> return &env->subprog_info[subprog];
> }
>
> +static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
> +{
> + return subprog_info(env, subprog)->ret_reg_pair;
> +}
> +
> +bool bpf_kfunc_ret_reg_pair(struct bpf_verifier_env *env, struct bpf_insn *insn);
> +
> struct bpf_call_summary {
> u8 num_params;
> bool is_void;
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -382,29 +382,62 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
> return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
> }
>
> -static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
> +static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)
> {
> - const struct btf_type *type, *func, *func_proto;
> + const struct btf_type *func, *func_proto;
> const struct btf *btf = env->prog->aux->btf;
> u32 btf_id;
>
> + if (!btf || !env->prog->aux->func_info)
> + return NULL;
> +
> btf_id = env->prog->aux->func_info[subprog].type_id;
>
> + /* Both already validated by prepare_btf_func() at prog load. */
> func = btf_type_by_id(btf, btf_id);
> - if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id))
> - return false;
> -
> func_proto = btf_type_by_id(btf, func->type);
> - if (!func_proto)
> - return false;
>
> - type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
> - if (!type)
> - return false;
> + return btf_type_skip_modifiers(btf, func_proto->type, NULL);
> +}
>
> - return btf_type_is_void(type);
> +static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
> +{
> + const struct btf_type *type = subprog_ret_type(env, subprog);
> +
> + return type && btf_type_is_void(type);
> +}
> +
> +static u32 ret_regs_cnt(u32 size)
> +{
> + return size > 8 && size <= 16 ? 2 : 1;
> }
>
> +static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
> +{
> + const struct btf *btf = env->prog->aux->btf;
> + const struct btf_type *type;
> + int subprog;
> + u32 size;
> +
> + for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
Does this loop need to start at subprog 0? The main program (subprog 0)
delivers its exit code in R0 alone and never uses the R0:R2 calling
convention, so checking its BTF return type can incorrectly trigger
jit_required even though no R2 propagation is needed.
> + type = subprog_ret_type(env, subprog);
> + if (!type || !(btf_type_is_struct(type) || btf_type_is_scalar(type)))
> + continue;
> + if (IS_ERR(btf_resolve_size(btf, type, &size)))
> + continue;
> + if (ret_regs_cnt(size) > 1) {
> + subprog_info(env, subprog)->ret_reg_pair = true;
> + /*
> + * The R0:R2 return convention is only implemented in
> + * the JIT: the interpreter propagates BPF_R0 alone out
> + * of a subprogram, so a caller reading R2 would see a
> + * stale value.
> + */
> + env->prog->jit_required = 1;
> + }
> + }
> +}
Additionally, should jit_required be set for subprogs whose BTF prototypes
are marked unreliable? btf_check_subprog_call() can flag a mismatch with
prog->aux->func_info_aux[subprog].unreliable = true, and check_func_call()
only aborts on -EFAULT, so a static callee with an aggregate return can
load today without using the BTF return type. Subprogs subsequently deleted
by bpf_opt_remove_dead_code() (which runs after this pass) also don't need
jit_required set.
This changes behavior for existing programs: on CONFIG_BPF_JIT=n or
net.core.bpf_jit_enable=0, such programs would now fail in
bpf_fixup_call_args() with "program requires BPF JIT compiler but it is
not available", whereas they previously verified and ran on the
interpreter.
[ ... ]
> @@ -20333,6 +20381,9 @@
> if (ret < 0)
> goto skip_full_check;
>
> + /* must precede the first bpf_ret_reg_pair() user below */
> + bpf_compute_subprog_ret_regs(env);
> +
> ret = bpf_compute_live_registers(env);
> if (ret < 0)
> goto skip_full_check;
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31740414277
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH bpf-next v5 03/11] bpf: Wire up JIT support for 16-byte kfunc returns
2026-08-13 20:02 [PATCH bpf-next v5 00/11] bpf: Support aggregate return values up to 16 bytes Yonghong Song
2026-08-13 20:02 ` [PATCH bpf-next v5 01/11] bpf: Factor check_global_ret_scalar_reg() out of the global return check Yonghong Song
2026-08-13 20:02 ` [PATCH bpf-next v5 02/11] bpf: Add helpers to describe the R0:R2 return register pair Yonghong Song
@ 2026-08-13 20:02 ` Yonghong Song
2026-08-13 20:02 ` [PATCH bpf-next v5 04/11] bpf: Track R2 of register-pair returns in precision backtracking Yonghong Song
` (7 subsequent siblings)
10 siblings, 0 replies; 18+ messages in thread
From: Yonghong Song @ 2026-08-13 20:02 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
LLVM 23 returns an __int128, or a struct/union larger than 8 bytes and no
larger than 16 bytes, in the BPF R0:R2 register pair. The previous patch
added the shared helpers describing that convention; wire up the JIT side
so that the second half of the return value actually lands in R2.
Placing the second return half into R2 is possible on any JIT, but it needs
architecture-specific JIT work. Rather than requiring every JIT to
implement it at once, add a bpf_jit_supports_kfunc_ret_reg_pair()
capability, defaulting to false in the generic core; an architecture opts
in once its JIT handles the R0:R2 pair, and the remaining ones are left for
future work. Only the x86-64, arm64 and riscv64 JITs opt in so far.
On arm64 and riscv64 the native second return register is already BPF R2
(x1 in bpf2a64[] and a1 in regmap[] respectively), so the upper half needs
no move at all, unlike x86-64's RDX->RSI. The lower half is covered by the
move into BPF R0 that those JITs already emit after every call, from x0
into x8 and from a0 into a5. This has been tested on x86-64 and arm64. The
riscv64 path is expected to work by the same register-mapping reasoning as
arm64 but has not been tested.
bpf_add_kfunc_call() also rejects a kfunc that is marked KF_FASTCALL and
returns more than 8 bytes. The bpf_fastcall contract implemented by
mark_fastcall_pattern_for_call() assumes a call clobbers R0 plus the
registers holding its arguments, so a return in the R0:R2 pair would
clobber an R2 the caller expects the fastcall pattern to preserve. Such
a kfunc is rejected with -EOPNOTSUPP as well.
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
arch/arm64/net/bpf_jit_comp.c | 5 +++++
arch/riscv/net/bpf_jit_comp64.c | 5 +++++
arch/x86/net/bpf_jit_comp.c | 27 ++++++++++++++++++++-------
include/linux/filter.h | 1 +
kernel/bpf/core.c | 5 +++++
kernel/bpf/verifier.c | 12 ++++++++++++
6 files changed, 48 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 74b4083791da..7a4c5968976a 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2346,6 +2346,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_stack_args(void)
{
return true;
diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index 2504df1fa111..49220765e96d 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -2120,6 +2120,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_ptr_xchg(void)
{
return true;
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d920772af7d5..2ec09dd02b18 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
* arena NULL is offset 0. Return the number of emitted bytes.
*/
static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
- const struct bpf_insn *insn, u8 **pprog)
+ const struct btf_func_model *fm, u8 **pprog)
{
- const struct btf_func_model *fm;
u8 *prog = *pprog;
u8 *start = prog;
int i;
- fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
- if (!fm)
- return -EINVAL;
-
for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
u8 flags = fm->arg_flags[i];
u32 reg = BPF_REG_1 + i;
@@ -2644,6 +2639,8 @@ st: insn_off = insn->off;
/* call */
case BPF_JMP | BPF_CALL: {
+ const struct btf_func_model *fm = NULL;
+
func = (u8 *) __bpf_call_base + imm32;
if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
LOAD_TAIL_CALL_CNT_PTR(stack_depth);
@@ -2652,7 +2649,10 @@ st: insn_off = insn->off;
if (!imm32)
return -EINVAL;
if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
- err = emit_kfunc_arena_args(bpf_prog, insn, &prog);
+ fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
+ if (!fm)
+ return -EINVAL;
+ err = emit_kfunc_arena_args(bpf_prog, fm, &prog);
if (err < 0)
return err;
ip += err;
@@ -2666,6 +2666,14 @@ st: insn_off = insn->off;
return -EINVAL;
if (priv_frame_ptr)
pop_r9(&prog);
+ /*
+ * A kfunc returning more than 8 bytes hands the second
+ * half back in RDX (the native ABI's second return reg),
+ * but BPF expects it in R0:R2. BPF R0 is RAX (no move
+ * needed), while BPF R2 is RSI, so copy RDX into RSI.
+ */
+ if (fm && fm->ret_size > 8)
+ emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
break;
}
@@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_stack_args(void)
{
return true;
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 4a9bc6a848f2..6e746b0a0930 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
bool bpf_jit_supports_subprog_tailcalls(void);
bool bpf_jit_supports_percpu_insn(void);
bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_kfunc_ret_reg_pair(void);
bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 6a94370a2448..cb66a2ef52b1 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
return false;
}
+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+ return false;
+}
+
bool __weak bpf_jit_supports_stack_args(void)
{
return false;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 57d14480ded2..161d77791bc6 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2854,6 +2854,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
if (err)
return err;
+ if (func_model.ret_size > 8) {
+ if (kfunc.flags && (*kfunc.flags & KF_FASTCALL)) {
+ verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n",
+ kfunc.name);
+ return -EOPNOTSUPP;
+ }
+ if (!bpf_jit_supports_kfunc_ret_reg_pair()) {
+ verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
+ kfunc.name);
+ return -EOPNOTSUPP;
+ }
+ }
memset(&meta, 0, sizeof(meta));
meta.btf = kfunc.btf;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH bpf-next v5 04/11] bpf: Track R2 of register-pair returns in precision backtracking
2026-08-13 20:02 [PATCH bpf-next v5 00/11] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (2 preceding siblings ...)
2026-08-13 20:02 ` [PATCH bpf-next v5 03/11] bpf: Wire up JIT support for 16-byte kfunc returns Yonghong Song
@ 2026-08-13 20:02 ` Yonghong Song
2026-08-13 20:49 ` bot+bpf-ci
2026-08-13 20:02 ` [PATCH bpf-next v5 05/11] bpf: Account R2 of register-pair returns in live register analysis Yonghong Song
` (6 subsequent siblings)
10 siblings, 1 reply; 18+ messages in thread
From: Yonghong Song @ 2026-08-13 20:02 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
Precision backtracking treats only R0 as a return register at a
call/return boundary, so once the verifier starts modeling R2 that way,
marking the second half of such a return precise would trip the
"unexpected regs" checks in backtrack_insn() and reject a valid
program.
Marking the upper half precise, for example by branching on it after a
call to a static subprogram, walks backtracking into the callee and
reaches its BPF_EXIT with R2 still set in the mask. Handle R2 like R0
in boundaries where a call defines the return registers.
R2 differs from R0 in that it is an argument register as well, which
makes two things worth spelling out:
- R2 is only excused from the BPF_REGMASK_ARGS check where the callee
or kfunc really does return a pair, as reported by
bpf_ret_reg_pair()/bpf_kfunc_ret_reg_pair(). Any other call leaves R2
uninitialized, so a request for its precision cannot be legitimate,
and clearing it unconditionally would turn that verifier bug from a
loud -EFAULT into a silently dropped precision request.
- at BPF_EXIT the return registers are sampled before the callback path
clears R1-R5. That clear does not touch R0, but it does cover R2, and
running it first would drop a pair return whenever the instruction
following the call happens to be one that invokes a callback.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/backtrack.c | 78 +++++++++++++++++++++++++++++++++++-------
1 file changed, 65 insertions(+), 13 deletions(-)
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index a2b18a9f1694..0c8e05a7e175 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -423,6 +423,16 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
*/
verifier_bug_if(idx + 1 != subseq_idx, env,
"extra insn from subprog");
+ /* a global subprog returning more than 8 bytes
+ * sets R2 as well. R2 is part of the args mask
+ * checked just below, so clear it here rather
+ * than next to R0. Only a subprog that does
+ * return a pair defines R2, so leave the mask
+ * alone otherwise and let the check below catch
+ * an R2 that has no business being set.
+ */
+ if (bpf_ret_reg_pair(env, subprog))
+ bt_clear_reg(bt, BPF_REG_2);
/* r1-r5 are invalidated after subprog call,
* so for global func call it shouldn't be set
* anymore
@@ -506,6 +516,17 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return -ENOTSUPP;
/* regular helper call sets R0 */
bt_clear_reg(bt, BPF_REG_0);
+ /* a kfunc returning more than 8 bytes also sets R2.
+ * R2 is part of the args mask checked just below, so
+ * clear it here rather than next to R0. The prototype
+ * lookup is only worth doing when R2 is requested at
+ * all; any other call leaves R2 uninitialized, so a
+ * request for it is caught by the check below.
+ */
+ if (bt_is_reg_set(bt, BPF_REG_2) &&
+ insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
+ bpf_kfunc_ret_reg_pair(env, insn))
+ bt_clear_reg(bt, BPF_REG_2);
if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
/* if backtracking was looking for registers R1-R5
* they should have been found already.
@@ -520,7 +541,41 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return -EFAULT;
}
} else if (opcode == BPF_EXIT) {
- bool r0_precise;
+ bool from_subprog_call, r0_precise, r2_precise;
+ struct bpf_insn *call;
+ int subprog;
+
+ /* BPF_EXIT in subprog or callback always returns
+ * right after the call instruction, so by checking
+ * whether the instruction at subseq_idx-1 is subprog
+ * call or not we can distinguish actual exit from
+ * *subprog* from exit from *callback*. In the former
+ * case, we need to propagate the precision of the
+ * return registers, if necessary. In the latter we
+ * never do that.
+ */
+ from_subprog_call = subseq_idx - 1 >= 0 &&
+ bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
+
+ /* Sample the return registers before the callback
+ * handling below clears R1-R5: unlike R0, R2 is an
+ * argument register as well, so that clear would drop
+ * a pair return on the floor.
+ */
+ r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
+ r2_precise = false;
+ if (from_subprog_call && bt_is_reg_set(bt, BPF_REG_2)) {
+ call = &env->prog->insnsi[subseq_idx - 1];
+ subprog = bpf_find_subprog(env, subseq_idx + call->imm);
+ if (subprog < 0)
+ return -EFAULT;
+ /* Only a callee that does return a pair defines
+ * R2. Leave the mask alone otherwise, so that
+ * the check below still catches an R2 that has
+ * no business being set.
+ */
+ r2_precise = bpf_ret_reg_pair(env, subprog);
+ }
/* Backtracking to a nested function call, 'idx' is a part of
* the inner frame 'subseq_idx' is a part of the outer frame.
@@ -533,30 +588,27 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
for (i = BPF_REG_1; i <= BPF_REG_5; i++)
bt_clear_reg(bt, i);
+
+ /* a callee returning more than 8 bytes sets R2 as well;
+ * R2 is part of the args mask checked just below, so
+ * clear it here rather than next to R0.
+ */
+ if (r2_precise)
+ bt_clear_reg(bt, BPF_REG_2);
if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
verifier_bug(env, "backtracking exit unexpected regs %x",
bt_reg_mask(bt));
return -EFAULT;
}
- /* BPF_EXIT in subprog or callback always returns
- * right after the call instruction, so by checking
- * whether the instruction at subseq_idx-1 is subprog
- * call or not we can distinguish actual exit from
- * *subprog* from exit from *callback*. In the former
- * case, we need to propagate r0 precision, if
- * necessary. In the former we never do that.
- */
- r0_precise = subseq_idx - 1 >= 0 &&
- bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
- bt_is_reg_set(bt, BPF_REG_0);
-
bt_clear_reg(bt, BPF_REG_0);
if (bt_subprog_enter(bt))
return -EFAULT;
if (r0_precise)
bt_set_reg(bt, BPF_REG_0);
+ if (r2_precise)
+ bt_set_reg(bt, BPF_REG_2);
/* r6-r9 and stack slots will stay set in caller frame
* bitmasks until we return back from callee(s)
*/
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 18+ messages in thread* Re: [PATCH bpf-next v5 04/11] bpf: Track R2 of register-pair returns in precision backtracking
2026-08-13 20:02 ` [PATCH bpf-next v5 04/11] bpf: Track R2 of register-pair returns in precision backtracking Yonghong Song
@ 2026-08-13 20:49 ` bot+bpf-ci
0 siblings, 0 replies; 18+ messages in thread
From: bot+bpf-ci @ 2026-08-13 20:49 UTC (permalink / raw)
To: yonghong.song, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3469 bytes --]
> diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
> index a2b18a9f1694..0c8e05a7e175 100644
> --- a/kernel/bpf/backtrack.c
> +++ b/kernel/bpf/backtrack.c
> @@ -423,6 +423,16 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
> */
> verifier_bug_if(idx + 1 != subseq_idx, env,
> "extra insn from subprog");
> + /* a global subprog returning more than 8 bytes
> + * sets R2 as well. R2 is part of the args mask
> + * checked just below, so clear it here rather
> + * than next to R0. Only a subprog that does
> + * return a pair defines R2, so leave the mask
> + * alone otherwise and let the check below catch
> + * an R2 that has no business being set.
> + */
> + if (bpf_ret_reg_pair(env, subprog))
> + bt_clear_reg(bt, BPF_REG_2);
[ ... ]
> @@ -506,6 +516,17 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
> return -ENOTSUPP;
> /* regular helper call sets R0 */
> bt_clear_reg(bt, BPF_REG_0);
> + /* a kfunc returning more than 8 bytes also sets R2.
> + * R2 is part of the args mask checked just below, so
> + * clear it here rather than next to R0. The prototype
> + * lookup is only worth doing when R2 is requested at
> + * all; any other call leaves R2 uninitialized, so a
> + * request for it is caught by the check below.
> + */
> + if (bt_is_reg_set(bt, BPF_REG_2) &&
> + insn->src_reg == BPF_PSEUDO_KFUNC_CALL &&
> + bpf_kfunc_ret_reg_pair(env, insn))
> + bt_clear_reg(bt, BPF_REG_2);
[ ... ]
> @@ -520,7 +541,41 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
> return -EFAULT;
> }
> } else if (opcode == BPF_EXIT) {
[ ... ]
> + r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
> + r2_precise = false;
> + if (from_subprog_call && bt_is_reg_set(bt, BPF_REG_2)) {
> + call = &env->prog->insnsi[subseq_idx - 1];
> + subprog = bpf_find_subprog(env, subseq_idx + call->imm);
> + if (subprog < 0)
> + return -EFAULT;
> + /* Only a callee that does return a pair defines
> + * R2. Leave the mask alone otherwise, so that
> + * the check below still catches an R2 that has
> + * no business being set.
> + */
> + r2_precise = bpf_ret_reg_pair(env, subprog);
> + }
[ ... ]
> @@ -533,30 +588,27 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
> if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
> for (i = BPF_REG_1; i <= BPF_REG_5; i++)
> bt_clear_reg(bt, i);
> +
> + /* a callee returning more than 8 bytes sets R2 as well;
> + * R2 is part of the args mask checked just below, so
> + * clear it here rather than next to R0.
> + */
> + if (r2_precise)
> + bt_clear_reg(bt, BPF_REG_2);
This isn't a bug, but the "R2 is part of the args mask checked just
below, so clear it here rather than next to R0" rationale appears at
each of the three call boundaries (global subprog, kfunc, and BPF_EXIT).
Would stating it once - say at the first site, or in a note above the
BPF_EXIT block - and keeping the other two to a one-liner read more
easily?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31740414277
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH bpf-next v5 05/11] bpf: Account R2 of register-pair returns in live register analysis
2026-08-13 20:02 [PATCH bpf-next v5 00/11] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (3 preceding siblings ...)
2026-08-13 20:02 ` [PATCH bpf-next v5 04/11] bpf: Track R2 of register-pair returns in precision backtracking Yonghong Song
@ 2026-08-13 20:02 ` Yonghong Song
2026-08-13 20:02 ` [PATCH bpf-next v5 06/11] bpf: Add verifier support for 16-byte returns in R0:R2 Yonghong Song
` (5 subsequent siblings)
10 siblings, 0 replies; 18+ messages in thread
From: Yonghong Song @ 2026-08-13 20:02 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a
struct/union or an __int128) reads R2 as well as R0, since the second half
of the return value is passed back in R2. compute_insn_live_regs() only
marked R0 used at exit, so a callee's R2 could be considered dead and
cleaned from checkpointed states, which would allow unsound state pruning.
Mark R2 as read at the BPF_EXIT of a subprogram that does return a register
pair. bpf_compute_live_registers() now loops over the subprograms and, for
each, over the [start, end) instruction range from env->subprog_info[], so
the return convention is queried once per subprogram through
bpf_ret_reg_pair() rather than once per instruction.
Marking R2 at every exit instead would be simpler, but R2 would then stay
live backwards across any call that is not followed by a write to R2, which
is nearly every program, and would needlessly hurt state pruning.
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/liveness.c | 20 ++++++++++++++------
1 file changed, 14 insertions(+), 6 deletions(-)
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 74fc4b3f80d6..71f998c6eb88 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2060,7 +2060,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); }
/* Compute info->{use,def} fields for the instruction */
static void compute_insn_live_regs(struct bpf_verifier_env *env,
struct bpf_insn *insn,
- struct insn_live_regs *info)
+ struct insn_live_regs *info,
+ bool ret_reg_pair)
{
struct bpf_call_summary cs;
const u8 class = BPF_CLASS(insn->code);
@@ -2072,6 +2073,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
const u32 src32 = mask_lo(src);
const u32 dst32 = mask_lo(dst);
const u32 r0 = reg64_mask(0);
+ const u32 r2 = reg64_mask(BPF_REG_2);
u32 def = 0;
u32 use = U32_MAX;
@@ -2191,7 +2193,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
break;
case BPF_EXIT:
def = 0;
- use = r0;
+ use = ret_reg_pair ? (r0 | r2) : r0;
break;
case BPF_CALL:
def = ALL_CALLER_SAVED_REGS;
@@ -2228,8 +2230,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
struct insn_live_regs *state;
int insn_cnt = env->prog->len;
u64 pos, insn_pos;
- int err = 0, i, j;
- bool changed;
+ int err = 0, i, j, subprog, start, end;
+ bool changed, ret_reg_pair;
/* Use the following algorithm:
* - define the following:
@@ -2256,8 +2258,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
goto out;
}
- for (i = 0; i < insn_cnt; ++i)
- compute_insn_live_regs(env, &insns[i], &state[i]);
+ for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+ start = env->subprog_info[subprog].start;
+ end = env->subprog_info[subprog + 1].start;
+ ret_reg_pair = bpf_ret_reg_pair(env, subprog);
+
+ for (i = start; i < end; ++i)
+ compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
+ }
/* Forward pass: resolve stack access through FP-derived pointers */
err = bpf_compute_subprog_arg_access(env);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH bpf-next v5 06/11] bpf: Add verifier support for 16-byte returns in R0:R2
2026-08-13 20:02 [PATCH bpf-next v5 00/11] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (4 preceding siblings ...)
2026-08-13 20:02 ` [PATCH bpf-next v5 05/11] bpf: Account R2 of register-pair returns in live register analysis Yonghong Song
@ 2026-08-13 20:02 ` Yonghong Song
2026-08-13 21:11 ` bot+bpf-ci
2026-08-13 20:02 ` [PATCH bpf-next v5 07/11] bpf: Enable aggregate return types up to 16 bytes Yonghong Song
` (4 subsequent siblings)
10 siblings, 1 reply; 18+ messages in thread
From: Yonghong Song @ 2026-08-13 20:02 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more than
16 bytes. See LLVM patches [1] and [2].
Before LLVM 23 the BPF backend could not return these values at all. A
by-value struct or union return (of any size) was rejected at compile time
with:
error: aggregate returns are not supported
and an __int128 return failed later in the backend with:
fatal error: error in backend: unable to allocate function return #1
Both are resolved in LLVM 23, which lowers such returns into the R0:R2
register pair.
This patch models that pair at calls to global and static BPF subprograms
and at kfunc calls: R2 is marked alongside R0 at the call, propagated out
of a callee at its exit, and held to the same scalar-only and no-stack-
pointer rules that R0 already is. A struct returned by a kfunc must be
composed of scalars, since its bytes reach the program as raw register
contents and a pointer field would otherwise be laundered into a scalar.
An extension program is the one caller of the convention that cannot take
part in it: its own return value is the program exit code, read out of R0
alone, so it has no way to hand back an upper half. Replacing a function
whose return value is larger than 8 bytes is therefore rejected with
-EOPNOTSUPP rather than supported.
[1] https://github.com/llvm/llvm-project/pull/190894
[2] https://github.com/llvm/llvm-project/pull/206876
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/verifier.c | 75 +++++++++++++++++++++++++++++++++++++++----
1 file changed, 68 insertions(+), 7 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 161d77791bc6..0c68ab3bd6ce 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -412,6 +412,9 @@ static u32 ret_regs_cnt(u32 size)
return size > 8 && size <= 16 ? 2 : 1;
}
+/* Registers holding a function return value, in order. See ret_regs_cnt(). */
+static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };
+
static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
{
const struct btf *btf = env->prog->aux->btf;
@@ -9456,6 +9459,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
u16 callee_incoming, stack_arg_cnt;
struct bpf_func_state *caller;
int err, subprog, target_insn;
+ u32 i, nregs;
target_insn = *insn_idx + insn->imm + 1;
subprog = bpf_find_subprog(env, target_insn);
@@ -9498,9 +9502,14 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
clear_caller_saved_regs(env, caller->regs);
invalidate_outgoing_stack_args(env, cur_func(env));
- /* All non-void global functions return a 64-bit SCALAR_VALUE. */
+ /*
+ * A non-void global function returns a 64-bit SCALAR_VALUE in
+ * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
+ */
if (!subprog_returns_void(env, subprog)) {
- mark_reg_unknown(env, caller->regs, BPF_REG_0);
+ nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+ for (i = 0; i < nregs; i++)
+ mark_reg_unknown(env, caller->regs, ret_regs[i]);
}
if (env->subprog_info[subprog].might_throw) {
@@ -9858,11 +9867,15 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
struct bpf_func_state *caller, *callee;
struct bpf_reg_state *r0;
bool in_callback_fn;
+ u32 i, nregs;
int err;
callee = state->frame[state->curframe];
r0 = &callee->regs[BPF_REG_0];
- if (r0->type == PTR_TO_STACK) {
+ nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+ for (i = 0; i < nregs; i++) {
+ if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
+ continue;
/* technically it's ok to return caller's stack pointer
* (or caller's caller's pointer) back to the caller,
* since these pointers are valid. Only current stack
@@ -9897,8 +9910,12 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
return -EFAULT;
}
} else {
- /* return to the caller whatever r0 had in the callee */
- caller->regs[BPF_REG_0] = *r0;
+ /*
+ * return to the caller whatever the callee had in the
+ * return register(s)
+ */
+ for (i = 0; i < nregs; i++)
+ caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]];
}
/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
@@ -10796,6 +10813,19 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
return 0;
}
+/*
+ * Mark the register(s) holding a @size byte kfunc return value as unknown
+ * scalars. Both halves of a register pair are treated the same way.
+ */
+static void mark_kfunc_ret_regs(struct bpf_verifier_env *env,
+ struct bpf_reg_state *regs, u32 size)
+{
+ u32 i, nregs = ret_regs_cnt(size);
+
+ for (i = 0; i < nregs; i++)
+ mark_reg_unknown(env, regs, ret_regs[i]);
+}
+
static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta)
{
return meta->kfunc_flags & KF_ACQUIRE;
@@ -13265,10 +13295,25 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
}
if (btf_type_is_scalar(t)) {
- mark_reg_unknown(env, regs, BPF_REG_0);
+ mark_kfunc_ret_regs(env, regs, t->size);
if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
__mark_reg_const_zero(env, ®s[BPF_REG_0]);
+ } else if (btf_type_is_struct(t)) {
+ /*
+ * The returned struct comes back as raw register bits modeled
+ * as an unknown scalar, so it must contain only scalars:
+ * otherwise a pointer field would be laundered into a scalar
+ * and escape provenance and reference tracking.
+ */
+ if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+ verbose(env,
+ "kernel function %s returns %s %s that is not composed of scalars\n",
+ func_name, btf_type_str(t),
+ btf_name_by_offset(desc_btf, t->name_off));
+ return -EINVAL;
+ }
+ mark_kfunc_ret_regs(env, regs, t->size);
} else if (btf_type_is_ptr(t)) {
ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
@@ -16746,11 +16791,20 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
{
struct bpf_func_state *cur_frame = cur_func(env);
u32 subprog = cur_frame->subprogno;
+ u32 i, nregs;
+ int err;
if (subprog_returns_void(env, subprog))
return 0;
- return check_global_ret_scalar_reg(env, BPF_REG_0);
+ nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+ for (i = 0; i < nregs; i++) {
+ err = check_global_ret_scalar_reg(env, ret_regs[i]);
+ if (err)
+ return err;
+ }
+
+ return 0;
}
/* Bitmask with 1s for all caller saved registers */
@@ -19404,6 +19458,13 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
return -EOPNOTSUPP;
}
+ if (prog_extension && tgt_info->fmodel.ret_size > 8) {
+ bpf_log(log,
+ "Cannot replace function %s with a >8 byte return value\n",
+ tname);
+ return -EOPNOTSUPP;
+ }
+
/*
* *.multi programs don't need an address during program
* verification, we just take the module ref if needed.
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 18+ messages in thread* Re: [PATCH bpf-next v5 06/11] bpf: Add verifier support for 16-byte returns in R0:R2
2026-08-13 20:02 ` [PATCH bpf-next v5 06/11] bpf: Add verifier support for 16-byte returns in R0:R2 Yonghong Song
@ 2026-08-13 21:11 ` bot+bpf-ci
0 siblings, 0 replies; 18+ messages in thread
From: bot+bpf-ci @ 2026-08-13 21:11 UTC (permalink / raw)
To: yonghong.song, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 5501 bytes --]
> bpf: Add verifier support for 16-byte returns in R0:R2
>
> LLVM 23 added support for returning a value in two registers for an
> __int128, or a struct/union whose size is greater than 8 but not more than
> 16 bytes. See LLVM patches [1] and [2].
>
> Before LLVM 23 the BPF backend could not return these values at all. A
> by-value struct or union return (of any size) was rejected at compile time
> with:
>
> error: aggregate returns are not supported
>
> and an __int128 return failed later in the backend with:
>
> fatal error: error in backend: unable to allocate function return #1
>
> Both are resolved in LLVM 23, which lowers such returns into the R0:R2
> register pair.
>
> This patch models that pair at calls to global and static BPF subprograms
> and at kfunc calls: R2 is marked alongside R0 at the call, propagated out
> of a callee at its exit, and held to the same scalar-only and no-stack-
> pointer rules that R0 already is.
This isn't a bug, but could the changelog be reworded in imperative mood
to match Documentation/process/submitting-patches.rst? The other commits
in this series follow that convention, so this one reads inconsistently.
> @@ -9858,11 +9867,15 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
> struct bpf_func_state *caller, *callee;
> struct bpf_reg_state *r0;
> bool in_callback_fn;
> + u32 i, nregs;
> int err;
>
> callee = state->frame[state->curframe];
> r0 = &callee->regs[BPF_REG_0];
> - if (r0->type == PTR_TO_STACK) {
> + nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
> + for (i = 0; i < nregs; i++) {
> + if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
> + continue;
> /* technically it's ok to return caller's stack pointer
> * (or caller's caller's pointer) back to the caller,
> * since these pointers are valid. Only current stack
The new R0:R2 return-register loop is placed before the
'if (callee->in_callback_fn)' branch, so it runs for callback frames as
well. A callback follows bpf_callback_t (return value in R0 only) and R2
holds an incoming argument: set_loop_callback_state() and
set_user_ringbuf_callback_state() both do
'callee->regs[BPF_REG_2] = caller->regs[BPF_REG_3]', i.e. the
callback_ctx, which is PTR_TO_STACK for the common
'bpf_loop(n, cb, &local_var, 0)' idiom.
bpf_compute_subprog_ret_regs() sets ret_reg_pair purely from the
subprog's BTF return type and iterates every subprog, with no exclusion
for subprogs used as callbacks. So a static subprog whose BTF declares a
9..16 byte return and which is passed to bpf_loop() or
bpf_user_ringbuf_drain() gets ret_reg_pair = true, and at its BPF_EXIT
the still-live callback_ctx in R2 is inspected as if it were the upper
half of a return value.
Consequence: the callback exit is rejected with 'cannot return stack
pointer to the caller', a diagnostic that points at a register holding an
argument rather than a return value, and it fires before the
in_callback_fn R0 checks that would otherwise produce the accurate
message. Impact is limited (any such program is already ill-typed with
respect to bpf_callback_t, and compiler-generated code for a 16-byte
return does write R2), so this is a misleading-rejection /
modelling-scope defect rather than a memory-safety bug.
Would it make sense to compute nregs after the in_callback_fn branch, or
force nregs = 1 when callee->in_callback_fn, since a callback frame has
no R0:R2 return convention?
> @@ -9897,8 +9910,12 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
> return -EFAULT;
> }
> } else {
> - /* return to the caller whatever r0 had in the callee */
> - caller->regs[BPF_REG_0] = *r0;
> + /*
> + * return to the caller whatever the callee had in the
> + * return register(s)
> + */
> + for (i = 0; i < nregs; i++)
> + caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]];
> }
prepare_func_exit() has three call sites, and only the real BPF_EXIT one
(process_bpf_exit_full()) establishes the callee's return registers. The
other two are synthetic exits that explicitly set up R0 but not R2,
because R1-R5 were already reset to NOT_INIT above them:
In check_helper_call(), BPF_FUNC_tail_call inside a subprog: the
caller-saved reset loop sets R1-R5 to NOT_INIT, then only
'mark_reg_unknown(env, regs, BPF_REG_0)' runs before
'prepare_func_exit(env, &env->insn_idx)'.
In check_ld_abs() hidden BPF_EXIT inside a subprog: reset loop, then only
'mark_reg_known_zero(env, regs, BPF_REG_0)' before prepare_func_exit().
For a static subprog with ret_reg_pair set, the new copy loop therefore
propagates a NOT_INIT R2 into caller->regs[BPF_REG_2] on those paths.
This is fail-safe (no uninitialised value is treated as readable), but it
means a caller that reads the upper half of the 16-byte return is
rejected on that path with an opaque 'R2 !read_ok', so a subprog with a
>8 byte return that contains a bpf_tail_call() or LD_ABS is unverifiable,
and only R0 gets the conservative unknown-scalar treatment the commit
message describes.
Would symmetric handling be to mark R2 unknown next to R0 at both sites
(the tail-called program only defines R0, so R2 is genuinely undefined
there), or to reject the combination with an explicit message?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31740414277
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH bpf-next v5 07/11] bpf: Enable aggregate return types up to 16 bytes
2026-08-13 20:02 [PATCH bpf-next v5 00/11] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (5 preceding siblings ...)
2026-08-13 20:02 ` [PATCH bpf-next v5 06/11] bpf: Add verifier support for 16-byte returns in R0:R2 Yonghong Song
@ 2026-08-13 20:02 ` Yonghong Song
2026-08-13 20:02 ` [PATCH bpf-next v5 08/11] bpf: Reject register-pair returns when the subprog BTF is unreliable Yonghong Song
` (3 subsequent siblings)
10 siblings, 0 replies; 18+ messages in thread
From: Yonghong Song @ 2026-08-13 20:02 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
Relax btf_distill_func_proto() to accept a by-value struct or union that
the R0:R2 convention added in earlier patches can carry:
- a struct or union larger than 8 and up to 16 bytes, returned in the
R0:R2 register pair, matching what LLVM emits for the BPF target;
- a struct or union up to 8 bytes, returned in R0 alone.
A >8 byte scalar (__int128) was already accepted and is unchanged.
Everything else stays rejected: a return type larger than 16 bytes, and any
type that __get_type_size() cannot return in registers at all (e.g. an
array), which it already reports as ret < 0.
btf_validate_return_type() is relaxed as well, so that it accepts a
by-value struct or union up to 16 bytes in addition to void and scalars.
For a local (static) function this also stops btf_check_subprog_call() from
marking the subprogram's BTF unreliable just because it returns an
aggregate.
With btf_distill_func_proto() and btf_validate_return_type() relaxed, the
verifier, JIT, precision-backtracking and live-register support from the
earlier patches becomes reachable: <=16 byte aggregate return values now
work end to end.
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
include/linux/bpf_verifier.h | 2 ++
kernel/bpf/btf.c | 23 +++++++++++++++----
kernel/bpf/verifier.c | 18 +++++++--------
.../selftests/bpf/progs/exceptions_fail.c | 2 +-
4 files changed, 30 insertions(+), 15 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index bffd32dca068..899e5ce98643 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1448,6 +1448,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);
struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
+ const struct btf_type *t, int rec);
int bpf_find_subprog(struct bpf_verifier_env *env, int off);
bool bpf_is_throw_kfunc(struct bpf_insn *insn);
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 87ffde865a50..e59c00f78b9a 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -7591,7 +7591,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
return -EINVAL;
}
ret = __get_type_size(btf, func->type, &t);
- if (ret < 0 || btf_type_is_struct(t)) {
+ if (ret < 0 || ret > 16) {
bpf_log(log,
"The function %s return type %s is unsupported.\n",
tname, btf_type_str(t));
@@ -7964,7 +7964,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env *env,
/* Check whether the type is a valid return type. */
static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,
- const struct btf_type *t, int subprog)
+ const struct btf_type *t, int subprog, bool is_global)
{
u32 tags = 0;
int err;
@@ -7987,6 +7987,19 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
return 0;
+ if (btf_type_is_struct(t) && t->size <= 16) {
+ /*
+ * A global function's caller models the return as an opaque
+ * scalar pair, so it may only return scalars by value. A local
+ * function is verified inline, so a pointer field stays tracked
+ * and needs no such restriction.
+ */
+ bool local_func = subprog && !is_global;
+
+ if (local_func || btf_type_is_scalar_struct(env, btf, t, 0))
+ return 0;
+ }
+
return -EOPNOTSUPP;
}
@@ -8074,12 +8087,12 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
return -EINVAL;
}
- err = btf_validate_return_type(env, btf, t, subprog);
+ err = btf_validate_return_type(env, btf, t, subprog, is_global);
if (err) {
if (is_global) {
bpf_log(log,
- "Global function %s() return value not void or scalar. "
- "Only those are supported.\n",
+ "Global function %s() has unsupported return type. "
+ "Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n",
tname);
}
return err;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 0c68ab3bd6ce..b23be0fa95af 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11093,9 +11093,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_
}
/* Returns true if struct is composed of scalars, 4 levels of nesting allowed */
-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
- const struct btf *btf,
- const struct btf_type *t, int rec)
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env,
+ const struct btf *btf,
+ const struct btf_type *t, int rec)
{
const struct btf_type *member_type;
const struct btf_member *member;
@@ -11113,7 +11113,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
verbose(env, "max struct nesting depth exceeded\n");
return false;
}
- if (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
+ if (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
return false;
continue;
}
@@ -11512,7 +11512,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
(is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) ||
is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {
if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) &&
- !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+ !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n",
reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
return -EINVAL;
@@ -11528,7 +11528,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
* scalars. The access size is derived from the pointed-to BTF type.
*/
if (!btf_type_is_scalar(ref_t) &&
- !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+ !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n",
reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
return -EINVAL;
@@ -12498,7 +12498,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
break;
}
- if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+ if (!btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id);
verbose(env, "%s is %s expected %s %s",
@@ -12962,7 +12962,7 @@ static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg
struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id);
if (is_bpf_percpu_obj_new_kfunc(meta->func_id)) {
- if (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
+ if (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
verbose(env, "bpf_percpu_obj_new type ID argument must be of a struct of scalars\n");
return -EINVAL;
}
@@ -13306,7 +13306,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
* otherwise a pointer field would be laundered into a scalar
* and escape provenance and reference tracking.
*/
- if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+ if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
verbose(env,
"kernel function %s returns %s %s that is not composed of scalars\n",
func_name, btf_type_str(t),
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index ac44d60e5066..9708efb93683 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)
SEC("?tc")
__exception_cb(exception_cb_bad_ret_type1)
-__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.")
+__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.")
int reject_exception_cb_type_1(struct __sk_buff *ctx)
{
bpf_throw(0);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 18+ messages in thread* [PATCH bpf-next v5 08/11] bpf: Reject register-pair returns when the subprog BTF is unreliable
2026-08-13 20:02 [PATCH bpf-next v5 00/11] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (6 preceding siblings ...)
2026-08-13 20:02 ` [PATCH bpf-next v5 07/11] bpf: Enable aggregate return types up to 16 bytes Yonghong Song
@ 2026-08-13 20:02 ` Yonghong Song
2026-08-13 20:49 ` bot+bpf-ci
2026-08-13 20:02 ` [PATCH bpf-next v5 09/11] selftests/bpf: Add C tests for 16-byte returns in R0:R2 Yonghong Song
` (2 subsequent siblings)
10 siblings, 1 reply; 18+ messages in thread
From: Yonghong Song @ 2026-08-13 20:02 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
btf_check_subprog_call() can decide, at a call site, that this BTF is not
to be trusted and mark the subprogram unreliable, which happens when
compiler optimizations remove arguments from a static function or when a
mismatched type is passed to a global one. Verification carries on, but
the prototype the return convention was read from is one the verifier has
already declared not to describe the compiled code.
Rather than keep tracking R2 on the strength of a discarded signature,
reject a return value larger than 8 bytes as soon as the prototype it was
derived from becomes unreliable. Add subprog_ret_pair_unreliable() and
test it in check_func_call() on the path to a static subprogram, which is
where the flag can be observed while the call still proceeds. For a static
callee this only triggers on a genuine argument mismatch, since the
previous patch stopped btf_validate_return_type() from marking a local
function unreliable for returning an aggregate.
No check is needed anywhere else:
- a global subprogram is already rejected by the existing "Caller passes
invalid args into func#N" path, because btf_check_subprog_call()
returns an error both when it marks the BTF unreliable and on every
later call;
- the main program does not use the convention at all: its return value
is the program's exit code, read out of R0, so nothing looks at R2
there;
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/verifier.c | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b23be0fa95af..0ffb3bed1649 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -441,6 +441,23 @@ static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
}
}
+/*
+ * A >8 byte BPF return changes the calling convention to R0:R2, and the
+ * verifier derives that convention from the subprogram's BTF prototype
+ * alone. Once that prototype is marked unreliable it is known not to
+ * describe the compiled code, so the convention read from it cannot be
+ * trusted either: reject the call rather than keep tracking R2 on the
+ * strength of a signature the verifier has already discarded.
+ */
+static bool subprog_ret_pair_unreliable(struct bpf_verifier_env *env, int subprog)
+{
+ struct bpf_prog_aux *aux = env->prog->aux;
+
+ return bpf_ret_reg_pair(env, subprog) &&
+ aux->func_info_aux &&
+ aux->func_info_aux[subprog].unreliable;
+}
+
static const char *subprog_name(const struct bpf_verifier_env *env, int subprog)
{
struct bpf_func_info *info;
@@ -9527,6 +9544,12 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
return 0;
}
+ if (subprog_ret_pair_unreliable(env, subprog)) {
+ verbose(env, "Func#%d ('%s') returns >8 bytes, which requires reliable BTF\n",
+ subprog, subprog_name(env, subprog));
+ return -EINVAL;
+ }
+
/*
* Track caller's total stack arg count (incoming + max outgoing).
* This is needed so the JIT knows how much stack arg space to allocate.
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 18+ messages in thread* Re: [PATCH bpf-next v5 08/11] bpf: Reject register-pair returns when the subprog BTF is unreliable
2026-08-13 20:02 ` [PATCH bpf-next v5 08/11] bpf: Reject register-pair returns when the subprog BTF is unreliable Yonghong Song
@ 2026-08-13 20:49 ` bot+bpf-ci
0 siblings, 0 replies; 18+ messages in thread
From: bot+bpf-ci @ 2026-08-13 20:49 UTC (permalink / raw)
To: yonghong.song, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 7285 bytes --]
> btf_check_subprog_call() can decide, at a call site, that this BTF is not
> to be trusted and mark the subprogram unreliable, which happens when
> compiler optimizations remove arguments from a static function or when a
> mismatched type is passed to a global one. Verification carries on, but
> the prototype the return convention was read from is one the verifier has
> already declared not to describe the compiled code.
>
> Rather than keep tracking R2 on the strength of a discarded signature,
> reject a return value larger than 8 bytes as soon as the prototype it was
> derived from becomes unreliable. Add subprog_ret_pair_unreliable() and
> test it in check_func_call() on the path to a static subprogram, which is
> where the flag can be observed while the call still proceeds. For a static
> callee this only triggers on a genuine argument mismatch, since the
> previous patch stopped btf_validate_return_type() from marking a local
> function unreliable for returning an aggregate.
>
> No check is needed anywhere else:
> - a global subprogram is already rejected by the existing "Caller passes
> invalid args into func#N" path, because btf_check_subprog_call()
> returns an error both when it marks the BTF unreliable and on every
> later call;
> - the main program does not use the convention at all: its return value
> is the program's exit code, read out of R0, so nothing looks at R2
> there;
Does this enumeration cover all call sites? btf_check_subprog_call() has
two callers: check_func_call() and push_callback_call() in verifier.c.
push_callback_call() also swallows every non-EFAULT error:
err = btf_check_subprog_call(env, subprog, caller->regs);
if (err == -EFAULT)
return err;
so a callback subprogram's BTF can be marked unreliable while the call
still proceeds, exactly the situation described above for the static
check_func_call() path. The two bullets given (global subprogram, main
program) do not cover it.
It interacts with the sticky nature of the flag: a subprogram marked
unreliable by a callback call site will subsequently be rejected by the
new check at an unrelated direct call site whose own arguments were fine,
since btf_check_subprog_call() returns -EINVAL for every later call once
the flag is set (the early return in btf_check_subprog_call()).
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index b23be0fa95af..0ffb3bed1649 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -441,6 +441,23 @@ static void bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
> }
> }
>
> +/*
> + * A >8 byte BPF return changes the calling convention to R0:R2, and the
> + * verifier derives that convention from the subprogram's BTF prototype
> + * alone. Once that prototype is marked unreliable it is known not to
> + * describe the compiled code, so the convention read from it cannot be
> + * trusted either: reject the call rather than keep tracking R2 on the
> + * strength of a signature the verifier has already discarded.
> + */
> +static bool subprog_ret_pair_unreliable(struct bpf_verifier_env *env, int subprog)
> +{
> + struct bpf_prog_aux *aux = env->prog->aux;
> +
> + return bpf_ret_reg_pair(env, subprog) &&
> + aux->func_info_aux &&
> + aux->func_info_aux[subprog].unreliable;
> +}
> +
[ ... ]
> @@ -9527,6 +9544,12 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
> return 0;
> }
>
> + if (subprog_ret_pair_unreliable(env, subprog)) {
> + verbose(env, "Func#%d ('%s') returns >8 bytes, which requires reliable BTF\n",
> + subprog, subprog_name(env, subprog));
> + return -EINVAL;
> + }
> +
Can this gate reject valid programs? For a static callee, func_info_aux[]
.unreliable is set for reasons that have nothing to do with an argument
mismatch.
The path is: do_check_insn() calls check_func_call(), which calls
btf_check_subprog_call(), which calls btf_check_func_arg_match().
btf_check_func_arg_match() in kernel/bpf/verifier.c starts with:
ret = btf_prepare_func_args(env, subprog);
if (ret) {
if (bpf_in_stack_arg_cnt(sub) > 0) ...
return ret;
}
Then btf_check_subprog_call() does unconditionally on any non-zero err:
if (err)
prog->aux->func_info_aux[subprog].unreliable = true;
btf_prepare_func_args() returns -EINVAL for a non-global subprogram in the
ordinary case of an untagged pointer argument. In the argument loop
(kernel/bpf/btf.c), a pointer arg that is not the prog ctx type, not a
dynptr, and carries no __arg_trusted/__arg_untrusted/__arg_arena tag skips
the generic-user-data-pointer branch and falls through to:
skip_pointer:
if (tags) { ... return -EINVAL; }
if (btf_type_is_int(t) || btf_is_any_enum(t)) { ... }
if (!is_global)
return -EINVAL;
A by-value struct argument takes the same exit. This is expected,
documented behaviour for static callees since btf_prepare_func_args() even
comments "not fatal for static funcs" at kernel/bpf/btf.c, because a
static subprogram is verified inline so the verifier never needs a usable
BTF argument model for it.
Concrete example that verifies fine at the parent commit but is now
rejected:
struct pair { __u64 a, b; };
static __noinline struct pair make_pair(__u64 *p)
{
struct pair r = { *p, 1 };
return r;
}
SEC("tc") int prog(struct __sk_buff *ctx)
{
__u64 v = 1;
struct pair p = make_pair(&v);
return p.a + p.b;
}
This program is rejected: "Func#1 ('make_pair') returns >8 bytes, which
requires reliable BTF".
The flow is: bpf_compute_subprog_ret_regs() sees struct pair, size 16,
and sets ret_reg_pair = true. btf_validate_return_type() accepts it via
the local_func path. The argument loop then returns -EINVAL for __u64 *p,
setting unreliable = true. The new check fires.
The rejection is also unnecessary for soundness on this path: a static
callee is verified inline, so prepare_func_exit() copies the callee's
actual tracked R0 and R2 into the caller. R2 tracking does not rest on the
BTF prototype the way it does for a global callee.
The commit message premise appears incorrect: "For a static callee this
only triggers on a genuine argument mismatch, since the previous patch
stopped btf_validate_return_type() from marking a local function
unreliable for returning an aggregate." Relaxing btf_validate_return_type()
removed only one of two independent -EINVAL exits inside
btf_prepare_func_args(); the argument-conversion exit at kernel/bpf/btf.c
is untouched and is the common one. The new comment in verifier.c encodes
the same premise: "Once that prototype is marked unreliable it is known
not to describe the compiled code" - but the flag also means merely "the
verifier cannot express these argument types".
Should the gate be restricted to cases where the return convention itself
is untrustworthy, rather than whenever any argument cannot be expressed in
BTF?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31740414277
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH bpf-next v5 09/11] selftests/bpf: Add C tests for 16-byte returns in R0:R2
2026-08-13 20:02 [PATCH bpf-next v5 00/11] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (7 preceding siblings ...)
2026-08-13 20:02 ` [PATCH bpf-next v5 08/11] bpf: Reject register-pair returns when the subprog BTF is unreliable Yonghong Song
@ 2026-08-13 20:02 ` Yonghong Song
2026-08-13 21:11 ` bot+bpf-ci
2026-08-13 20:03 ` [PATCH bpf-next v5 10/11] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns Yonghong Song
2026-08-13 20:03 ` [PATCH bpf-next v5 11/11] Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2 Yonghong Song
10 siblings, 1 reply; 18+ messages in thread
From: Yonghong Song @ 2026-08-13 20:02 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
Add selftests that exercise a 16-byte return value passed in the R0:R2
register pair, written in C so that they depend on the compiler lowering
the register-pair return. Covered are an __int128 return, a 16-byte struct
return (from a static and from a global subprogram) and a 16-byte union
return, plus __int128 and 16-byte struct returns from a kfunc. The union
program shares an object with the struct ones.
The R0:R2 convention is only emitted by LLVM 23 and newer, and a by-value
aggregate return does not compile at all before that, so the programs sit
behind a __clang_major__ guard. An older compiler builds the dummy test in
the #else branch instead, which keeps each object non-empty and says in
its description why nothing was exercised.
The kfunc tests are tagged __arch_x86_64/__arch_arm64 and skip elsewhere.
Those are the architectures whose JIT advertises
bpf_jit_supports_kfunc_ret_reg_pair(), which bpf_add_kfunc_call() requires
before it accepts a kfunc returning more than 8 bytes, and they are also
the only ones building the kfuncs.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
.../selftests/bpf/prog_tests/aggregate_ret.c | 13 ++
.../bpf/progs/aggregate_ret_int128_c.c | 49 ++++++++
.../bpf/progs/aggregate_ret_kfunc_c.c | 65 ++++++++++
.../bpf/progs/aggregate_ret_struct_c.c | 114 ++++++++++++++++++
.../selftests/bpf/test_kmods/bpf_testmod.c | 18 +++
.../bpf/test_kmods/bpf_testmod_kfunc.h | 9 ++
6 files changed, 268 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
new file mode 100644
index 000000000000..979536f3c89c
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include "aggregate_ret_int128_c.skel.h"
+#include "aggregate_ret_struct_c.skel.h"
+#include "aggregate_ret_kfunc_c.skel.h"
+
+void test_aggregate_ret(void)
+{
+ RUN_TESTS(aggregate_ret_int128_c);
+ RUN_TESTS(aggregate_ret_struct_c);
+ RUN_TESTS(aggregate_ret_kfunc_c);
+}
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
new file mode 100644
index 000000000000..913cc374215d
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
@@ -0,0 +1,49 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A 0xdeadbeefcafef00dULL
+#define MIX_B 0x0123456789abcdefULL
+
+typedef unsigned __int128 u128;
+
+static __noinline u128 make_i128(__u64 a, __u64 b)
+{
+ return ((u128)(a + b) << 64) | (a - b);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_int128_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ u128 v;
+
+ v = make_i128(a, b);
+ if ((__u64)(v >> 64) != a + b)
+ return 1;
+ if ((__u64)v != a - b)
+ return 2;
+
+ return 0;
+}
+
+#else
+
+SEC("socket")
+__description("aggregate_ret_int128_c: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
new file mode 100644
index 000000000000..b66576307f73
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
@@ -0,0 +1,65 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A 0xdeadbeefcafef00dULL
+#define MIX_B 0x0123456789abcdefULL
+
+typedef unsigned __int128 u128;
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ u128 v;
+
+ v = bpf_kfunc_call_test_i128(a, b);
+ if ((__u64)(v >> 64) != a + b)
+ return 1;
+ if ((__u64)v != a - b)
+ return 2;
+
+ return 0;
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ struct prog_test_ret_pair p;
+
+ p = bpf_kfunc_call_test_ret_pair(a, b);
+ if (p.hi != a + b)
+ return 1;
+ if (p.lo != a - b)
+ return 2;
+
+ return 0;
+}
+
+#else
+
+SEC("socket")
+__description("aggregate_ret_kfunc_c: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
new file mode 100644
index 000000000000..83b1a3751988
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
@@ -0,0 +1,114 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A 0xdeadbeefcafef00dULL
+#define MIX_B 0x0123456789abcdefULL
+
+struct pair {
+ __u64 hi; /* R0 */
+ __u64 lo; /* R2 */
+};
+
+union upair {
+ __u64 halves[2];
+ struct {
+ __u64 lo; /* R0 */
+ __u64 hi; /* R2 */
+ } parts;
+};
+
+static __noinline struct pair make_pair(__u64 a, __u64 b)
+{
+ struct pair p = { .hi = a + b, .lo = a - b };
+
+ return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_struct_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ struct pair p;
+
+ p = make_pair(a, b);
+ if (p.hi != a + b)
+ return 1;
+ if (p.lo != a - b)
+ return 2;
+
+ return 0;
+}
+
+__noinline struct pair make_pair_global(__u64 a, __u64 b)
+{
+ struct pair p = { .hi = a + b, .lo = a - b };
+
+ return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ struct pair p;
+
+ p = make_pair_global(a, b);
+ if (p.hi != a + b)
+ return 1;
+ if (p.lo != a - b)
+ return 2;
+
+ return 0;
+}
+
+static __noinline union upair make_upair(__u64 a, __u64 b)
+{
+ union upair p;
+
+ p.halves[0] = a + b;
+ p.halves[1] = a - b;
+ return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_union_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ union upair p;
+
+ p = make_upair(a, b);
+ if (p.parts.lo != a + b)
+ return 1;
+ if (p.parts.hi != a - b)
+ return 2;
+
+ return 0;
+}
+
+#else
+
+SEC("socket")
+__description("aggregate_ret_struct_c: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index a6133f7521f3..fc1e43c5038b 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -939,6 +939,20 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
return 0;
}
+#if defined(__x86_64__) || defined(__aarch64__)
+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
+{
+ return (__int128)(((unsigned __int128)(a + b) << 64) | (a - b));
+}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)
+{
+ struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };
+
+ return r;
+}
+#endif /* __x86_64__ || __aarch64__ */
+
__bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
u64 e, u64 f, u64 g, u64 h,
u64 i, u64 j)
@@ -1472,6 +1486,10 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+#if defined(__x86_64__) || defined(__aarch64__)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)
+#endif
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index c4383acb53c1..682f7793d0c2 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -55,6 +55,11 @@ struct prog_test_big_arg {
__u64 b;
};
+struct prog_test_ret_pair {
+ __u64 hi;
+ __u64 lo;
+};
+
struct prog_test_fail1 {
void *p;
int x;
@@ -130,6 +135,10 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+#ifdef __SIZEOF_INT128__
+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
+#endif
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
__u64 e, __u64 f, __u64 g, __u64 h,
__u64 i, __u64 j) __ksym;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 18+ messages in thread* Re: [PATCH bpf-next v5 09/11] selftests/bpf: Add C tests for 16-byte returns in R0:R2
2026-08-13 20:02 ` [PATCH bpf-next v5 09/11] selftests/bpf: Add C tests for 16-byte returns in R0:R2 Yonghong Song
@ 2026-08-13 21:11 ` bot+bpf-ci
0 siblings, 0 replies; 18+ messages in thread
From: bot+bpf-ci @ 2026-08-13 21:11 UTC (permalink / raw)
To: yonghong.song, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1953 bytes --]
> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
> new file mode 100644
> index 000000000000..83b1a3751988
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
[ ... ]
> +struct pair {
> + __u64 hi; /* R0 */
> + __u64 lo; /* R2 */
> +};
> +
> +union upair {
> + __u64 halves[2];
> + struct {
> + __u64 lo; /* R0 */
> + __u64 hi; /* R2 */
> + } parts;
> +};
This isn't a bug, but struct pair puts hi in R0 while union upair's
parts puts lo in R0 - was the inverted naming in the union deliberate,
or would matching the struct's order (or first/second) make the R0:R2
mapping easier to follow?
> +
> +static __noinline struct pair make_pair(__u64 a, __u64 b)
> +{
> + struct pair p = { .hi = a + b, .lo = a - b };
> +
> + return p;
> +}
> +
> +SEC("tc")
> +__load_if_JITed()
> +__success __retval(0)
> +int aggregate_ret_struct_c_test(struct __sk_buff *skb)
> +{
> + __u64 a = skb->len ^ MIX_A;
> + __u64 b = skb->len ^ MIX_B;
> + struct pair p;
> +
> + p = make_pair(a, b);
> + if (p.hi != a + b)
> + return 1;
> + if (p.lo != a - b)
> + return 2;
> +
> + return 0;
> +}
[ ... ]
> +static __noinline union upair make_upair(__u64 a, __u64 b)
> +{
> + union upair p;
> +
> + p.halves[0] = a + b;
> + p.halves[1] = a - b;
> + return p;
> +}
> +
> +SEC("tc")
> +__load_if_JITed()
> +__success __retval(0)
> +int aggregate_ret_union_c_test(struct __sk_buff *skb)
> +{
> + __u64 a = skb->len ^ MIX_A;
> + __u64 b = skb->len ^ MIX_B;
> + union upair p;
> +
> + p = make_upair(a, b);
> + if (p.parts.lo != a + b)
> + return 1;
> + if (p.parts.hi != a - b)
> + return 2;
> +
> + return 0;
> +}
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31740414277
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH bpf-next v5 10/11] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
2026-08-13 20:02 [PATCH bpf-next v5 00/11] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (8 preceding siblings ...)
2026-08-13 20:02 ` [PATCH bpf-next v5 09/11] selftests/bpf: Add C tests for 16-byte returns in R0:R2 Yonghong Song
@ 2026-08-13 20:03 ` Yonghong Song
2026-08-13 21:11 ` bot+bpf-ci
2026-08-13 20:03 ` [PATCH bpf-next v5 11/11] Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2 Yonghong Song
10 siblings, 1 reply; 18+ messages in thread
From: Yonghong Song @ 2026-08-13 20:03 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
Add inline-asm tests covering what the C tests cannot reach.
aggregate_ret_func.c exercises BPF-to-BPF returns: subprograms that leave
R2 uninitialised or holding a stack pointer, ones returning a struct or a
union that smuggles a pointer, static and global subprograms whose R2
stays precise under backtracking, and liveness at the exit of a
subprogram returning a pair.
In addition, a negative freplace test checks that an extension cannot
replace a function returning R0:R2: the extension's own return is capped
at 8 bytes, so it would leave R2 stale for the target's callers, and the
load is expected to fail.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
.../selftests/bpf/prog_tests/aggregate_ret.c | 4 +
.../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 15 +
.../selftests/bpf/progs/aggregate_ret_func.c | 260 ++++++++++++++++++
.../selftests/bpf/progs/aggregate_ret_kfunc.c | 122 ++++++++
.../bpf/progs/aggregate_ret_kfunc_c.c | 60 ++++
.../bpf/progs/aggregate_ret_target.c | 29 ++
.../bpf/progs/compute_live_registers.c | 30 ++
.../selftests/bpf/progs/freplace_ret_pair.c | 12 +
.../selftests/bpf/test_kmods/bpf_testmod.c | 55 ++++
.../bpf/test_kmods/bpf_testmod_kfunc.h | 35 +++
10 files changed, 622 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c
create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_target.c
create mode 100644 tools/testing/selftests/bpf/progs/freplace_ret_pair.c
diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
index 979536f3c89c..c295adedbae8 100644
--- a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -4,10 +4,14 @@
#include "aggregate_ret_int128_c.skel.h"
#include "aggregate_ret_struct_c.skel.h"
#include "aggregate_ret_kfunc_c.skel.h"
+#include "aggregate_ret_func.skel.h"
+#include "aggregate_ret_kfunc.skel.h"
void test_aggregate_ret(void)
{
RUN_TESTS(aggregate_ret_int128_c);
RUN_TESTS(aggregate_ret_struct_c);
RUN_TESTS(aggregate_ret_kfunc_c);
+ RUN_TESTS(aggregate_ret_func);
+ RUN_TESTS(aggregate_ret_kfunc);
}
diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
index 2523c07a16c6..0b54f911015c 100644
--- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
+++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
@@ -441,6 +441,19 @@ static void test_func_replace_int_with_void(void)
" doesn't match type INT of global_func2()");
}
+static void test_func_replace_ret_pair(void)
+{
+ const char *msg = "Cannot replace function agg_ret_target_func with a >8 byte return";
+
+ /*
+ * An extension cannot replace a function whose return value comes back
+ * in the R0:R2 pair: the extension's own return is capped at 8 bytes,
+ * so it would leave R2 stale for the target's callers.
+ */
+ test_obj_load_failure_common("freplace_ret_pair.bpf.o",
+ "./aggregate_ret_target.bpf.o", msg);
+}
+
static int find_prog_btf_id(const char *name, __u32 attach_prog_fd)
{
struct bpf_prog_info info = {};
@@ -660,6 +673,8 @@ void serial_test_fexit_bpf2bpf(void)
test_func_replace_progmap();
if (test__start_subtest("freplace_int_with_void"))
test_func_replace_int_with_void();
+ if (test__start_subtest("freplace_ret_pair"))
+ test_func_replace_ret_pair();
if (test__start_subtest("freplace_void"))
test_func_replace_void();
if (test__start_subtest("sleepable_fentry_to_xdp"))
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
new file mode 100644
index 000000000000..cfb21bcf704b
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
@@ -0,0 +1,260 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+typedef unsigned __int128 u128;
+
+__naked u128 global_agg_good(void)
+{
+ asm volatile (
+ "r0 = 0x1234;" /* low 64 bits */
+ "r2 = 0x5678;" /* high 64 bits */
+ "exit;"
+ );
+}
+
+__naked u128 global_agg_bad(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "exit;"
+ );
+}
+
+__naked u128 global_agg_bad_ptr(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r2 = r10;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_global_fail(void)
+{
+ asm volatile (
+ "call %[global_agg_bad];"
+ "r0 = r2;"
+ "exit;"
+ :
+ : __imm(global_agg_bad)
+ : __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("At subprogram exit the register R2 is not a scalar value")
+__naked int aggregate_ret_global_ptr_fail(void)
+{
+ asm volatile (
+ "call %[global_agg_bad_ptr];"
+ "r0 = r2;"
+ "exit;"
+ :
+ : __imm(global_agg_bad_ptr)
+ : __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_bad_ptr(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r2 = r10;" /* stack pointer placed in the second return register */
+ "exit;"
+ );
+}
+
+/*
+ * R2 is a return register once the subprogram returns a pair, so a stack
+ * pointer left in it is rejected at the callee's exit exactly as one in R0
+ * is: the callee frame is gone by the time the caller could use it.
+ */
+SEC("tc")
+__failure __msg("cannot return stack pointer to the caller")
+__naked int aggregate_ret_static_ptr_fail(void)
+{
+ asm volatile (
+ "call %[static_agg_bad_ptr];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(static_agg_bad_ptr)
+ : __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_no_r2(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_static_uninit_fail(void)
+{
+ asm volatile (
+ "call %[static_agg_no_r2];"
+ "r0 = r2;"
+ "exit;"
+ :
+ : __imm(static_agg_no_r2)
+ : __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_precise(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r2 = 4;" /* second half; its value is made precise below */
+ "exit;"
+ );
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
+__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
+__naked int aggregate_ret_static_precise(void)
+{
+ asm volatile (
+ "call %[static_agg_precise];"
+ "r6 = r2;" /* derived from the aggregate's second half */
+ "r6 &= 7;" /* keep it in [0, 7] to index the stack */
+ "r1 = r10;"
+ "r1 += -8;"
+ "r1 += r6;" /* ptr += scalar marks r6 (hence R2) precise */
+ "r0 = 0;"
+ "*(u8 *)(r1 + 0) = r0;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(static_agg_precise)
+ : __clobber_all);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
+__naked int aggregate_ret_global_precise(void)
+{
+ asm volatile (
+ "call %[global_agg_good];"
+ "r6 = r2;" /* derived from the aggregate's second half */
+ "r6 &= 7;" /* keep it in [0, 7] to index the stack */
+ "r1 = r10;"
+ "r1 += -8;"
+ "r1 += r6;" /* ptr += scalar marks r6 (hence R2) precise */
+ "r0 = 0;"
+ "*(u8 *)(r1 + 0) = r0;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(global_agg_good)
+ : __clobber_all);
+}
+
+/* A by-value struct that smuggles a pointer, which must be rejected. */
+struct with_ptr {
+ void *p;
+ __u64 x;
+};
+
+/* A by-value union that smuggles a pointer, which must be rejected too. */
+union upair_with_ptr {
+ void *p;
+ __u64 halves[2];
+};
+
+__naked struct with_ptr global_ret_struct_ptr(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r2 = 0;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_struct_ptr() has unsupported return type")
+__naked int aggregate_ret_global_struct_ptr_fail(void)
+{
+ asm volatile (
+ "call %[global_ret_struct_ptr];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(global_ret_struct_ptr)
+ : __clobber_all);
+}
+
+__naked union upair_with_ptr global_ret_union_ptr(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r2 = 0;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_union_ptr() has unsupported return type")
+__naked int aggregate_ret_global_union_ptr_fail(void)
+{
+ asm volatile (
+ "call %[global_ret_union_ptr];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(global_ret_union_ptr)
+ : __clobber_all);
+}
+
+struct ptr_pair {
+ void *p;
+ __u64 x;
+};
+
+static __naked __noinline struct ptr_pair static_ret_ptr_pair(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r2 = r1;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__naked int aggregate_ret_static_ptr_pair(void)
+{
+ asm volatile (
+ "call %[static_ret_ptr_pair];"
+ "r1 = *(u32 *)(r2 + 0);" /* deref the returned ctx pointer */
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(static_ret_ptr_pair)
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
new file mode 100644
index 000000000000..c23b4beb1773
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
@@ -0,0 +1,122 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+/*
+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address
+ * (rather than calling) avoids any dependence on the compiler lowering an
+ * __int128 or struct return value, which the BPF backend only supports from
+ * LLVM 23 on.
+ */
+void __kfunc_btf_root(void)
+{
+ asm volatile (""
+ :
+ : "r"(&bpf_kfunc_call_test_i128),
+ "r"(&bpf_kfunc_call_test_ret_fastcall),
+ "r"(&bpf_kfunc_call_test_ret_ptr),
+ "r"(&bpf_kfunc_call_test_ret_ii),
+ "r"(&bpf_kfunc_call_test_ret_big));
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
+__naked int aggregate_ret_kfunc_precise(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "call %[bpf_kfunc_call_test_i128];"
+ "r6 = r2;" /* second return half */
+ "r6 &= 7;" /* keep it in [0, 7] to index the stack */
+ "r1 = r10;"
+ "r1 += -8;"
+ "r1 += r6;" /* ptr += scalar marks r6 (hence R2) precise */
+ "r0 = 0;"
+ "*(u8 *)(r1 + 0) = r0;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_test_i128)
+ : __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL")
+__naked int aggregate_ret_kfunc_fastcall_fail(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "call %[bpf_kfunc_call_test_ret_fastcall];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_test_ret_fastcall)
+ : __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("is not composed of scalars")
+__naked int aggregate_ret_kfunc_ptr_fail(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "call %[bpf_kfunc_call_test_ret_ptr];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_test_ret_ptr)
+ : __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_kfunc_small_no_r2(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "r2 = 0;"
+ "call %[bpf_kfunc_call_test_ret_ii];"
+ "r0 = r2;" /* R2 is not a return register for a <=8 byte struct */
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_test_ret_ii)
+ : __clobber_all);
+}
+
+/*
+ * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by
+ * btf_distill_func_proto(), before the KF_FASTCALL and JIT-capability checks,
+ * so this behaves the same on every architecture.
+ */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported")
+__naked int aggregate_ret_kfunc_too_big_fail(void)
+{
+ asm volatile (
+ "call %[bpf_kfunc_call_test_ret_big];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_test_ret_big)
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
index b66576307f73..2c1889fc28ef 100644
--- a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc_c.c
@@ -50,6 +50,66 @@ int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)
return 0;
}
+/* struct { u64 a; int b; }: 16 bytes, R0 = a, R2 = b. */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_li_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ int b = skb->len ^ MIX_B;
+ struct prog_test_ret_li r;
+
+ r = bpf_kfunc_call_test_ret_li(a, b);
+ if (r.a != a)
+ return 1;
+ if (r.b != ~b)
+ return 2;
+
+ return 0;
+}
+
+/* struct { int a; int b; }: 8 bytes, packed into R0; R2 is not a return reg. */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_ii_c_test(struct __sk_buff *skb)
+{
+ int a = skb->len ^ MIX_A;
+ int b = skb->len ^ MIX_B;
+ struct prog_test_ret_ii r;
+
+ r = bpf_kfunc_call_test_ret_ii(a, b);
+ if (r.a != a)
+ return 1;
+ if (r.b != b)
+ return 2;
+
+ return 0;
+}
+
+/* A union of 16 bytes takes the same R0:R2 path as a struct. */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_uu_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ union prog_test_ret_uu r;
+
+ r = bpf_kfunc_call_test_ret_uu(a, b);
+ if (r.parts.lo != a + b)
+ return 1;
+ if (r.parts.hi != a - b)
+ return 2;
+
+ return 0;
+}
+
#else
SEC("socket")
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
new file mode 100644
index 000000000000..cffd8d7d3241
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* freplace target: a global subprogram returning 16 bytes in R0:R2. */
+__naked unsigned __int128 agg_ret_target_func(void)
+{
+ asm volatile (
+ "r0 = 0x1234;"
+ "r2 = 0x5678;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__naked int agg_ret_target(void)
+{
+ asm volatile (
+ "call %[agg_ret_target_func];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(agg_ret_target_func)
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
index d055fc7b3b95..0be9441ec273 100644
--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
@@ -431,6 +431,36 @@ __naked void subprog1(void)
::: __clobber_all);
}
+static __used __naked unsigned __int128 aux2(void)
+{
+ asm volatile (
+ "r0 = 1;"
+ "r2 = 2;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+SEC("socket")
+/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */
+__load_if_JITed()
+__log_level(2)
+__msg("0: .12345.... (85) call pc+2")
+__msg("1: ..2....... (bf) r0 = r2")
+/* R2 is not read at the exit of this program, which returns an int, ... */
+__msg("2: 0......... (95) exit")
+__msg("3: .......... (b7) r0 = 1")
+__msg("4: 0......... (b7) r2 = 2")
+/* ... but it is at the exit of aux2(), which returns a register pair. */
+__msg("5: 0.2....... (95) exit")
+__naked void subprog_ret_reg_pair(void)
+{
+ asm volatile (
+ "call aux2;"
+ "r0 = r2;"
+ "exit;"
+ ::: __clobber_all);
+}
+
#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
SEC("socket")
diff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
new file mode 100644
index 000000000000..12c15d293bd7
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+SEC("freplace/agg_ret_target_func")
+__u64 new_agg_ret_target_func(void)
+{
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index fc1e43c5038b..ad36d583d2e7 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -951,8 +951,57 @@ __bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)
return r;
}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)
+{
+ struct prog_test_ret_pair r = { .hi = a + b, .lo = a - b };
+
+ return r;
+}
+
+__bpf_kfunc struct prog_test_ret_li bpf_kfunc_call_test_ret_li(u64 a, int b)
+{
+ struct prog_test_ret_li r = { .a = a, .b = ~b };
+
+ return r;
+}
+
+__bpf_kfunc union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(u64 a, u64 b)
+{
+ union prog_test_ret_uu r;
+
+ r.halves[0] = a + b;
+ r.halves[1] = a - b;
+ return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)
+{
+ struct prog_test_ret_ptr r = { .p = NULL, .tag = tag };
+
+ return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)
+{
+ struct prog_test_ret_ii r = { .a = a, .b = b };
+
+ return r;
+}
#endif /* __x86_64__ || __aarch64__ */
+/*
+ * Takes no argument on purpose: with no arguments there is nothing for the sret
+ * pointer to displace, so this needs no architecture guard even though it
+ * returns 24 bytes. See the comment on bpf_kfunc_call_test_i128() above.
+ */
+__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)
+{
+ struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };
+
+ return r;
+}
+
__bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
u64 e, u64 f, u64 g, u64 h,
u64 i, u64 j)
@@ -1489,7 +1538,13 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
#if defined(__x86_64__) || defined(__aarch64__)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_li)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_uu)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)
#endif
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index 682f7793d0c2..c7be973cd286 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -60,6 +60,35 @@ struct prog_test_ret_pair {
__u64 lo;
};
+struct prog_test_ret_li { /* 16 bytes: R0:R2 */
+ __u64 a;
+ int b;
+};
+
+struct prog_test_ret_ii { /* 8 bytes: R0 only */
+ int a;
+ int b;
+};
+
+union prog_test_ret_uu { /* 16 bytes: R0:R2 */
+ __u64 halves[2];
+ struct {
+ __u64 lo;
+ __u64 hi;
+ } parts;
+};
+
+struct prog_test_ret_ptr { /* 16 bytes: contains a pointer */
+ void *p;
+ __u64 tag;
+};
+
+struct prog_test_ret_big { /* 24 bytes: too large for R0:R2 */
+ __u64 a;
+ __u64 b;
+ __u64 c;
+};
+
struct prog_test_fail1 {
void *p;
int x;
@@ -139,6 +168,12 @@ int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
#endif
struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_li bpf_kfunc_call_test_ret_li(__u64 a, int b) __ksym;
+struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
+union prog_test_ret_uu bpf_kfunc_call_test_ret_uu(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;
+struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym;
__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
__u64 e, __u64 f, __u64 g, __u64 h,
__u64 i, __u64 j) __ksym;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 18+ messages in thread* Re: [PATCH bpf-next v5 10/11] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
2026-08-13 20:03 ` [PATCH bpf-next v5 10/11] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns Yonghong Song
@ 2026-08-13 21:11 ` bot+bpf-ci
0 siblings, 0 replies; 18+ messages in thread
From: bot+bpf-ci @ 2026-08-13 21:11 UTC (permalink / raw)
To: yonghong.song, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 6092 bytes --]
> diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
> index 979536f3c89c..c295adedbae8 100644
> --- a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
> +++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
> @@ -4,10 +4,14 @@
> #include "aggregate_ret_int128_c.skel.h"
> #include "aggregate_ret_struct_c.skel.h"
> #include "aggregate_ret_kfunc_c.skel.h"
> +#include "aggregate_ret_func.skel.h"
> +#include "aggregate_ret_kfunc.skel.h"
>
> void test_aggregate_ret(void)
> {
> RUN_TESTS(aggregate_ret_int128_c);
> RUN_TESTS(aggregate_ret_struct_c);
> RUN_TESTS(aggregate_ret_kfunc_c);
> + RUN_TESTS(aggregate_ret_func);
> + RUN_TESTS(aggregate_ret_kfunc);
> }
[ ... ]
> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
> new file mode 100644
> index 000000000000..cfb21bcf704b
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
> @@ -0,0 +1,260 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
> +#include <linux/bpf.h>
> +#include <bpf/bpf_helpers.h>
> +#include "bpf_misc.h"
[ ... ]
> +SEC("tc")
> +__load_if_JITed()
> +__success __retval(0)
> +__log_level(2)
> +__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
> +__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
> +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
> +__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
> +__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
> +__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
> +__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
> +__naked int aggregate_ret_static_precise(void)
[ ... ]
> +SEC("tc")
> +__load_if_JITed()
> +__success __retval(0)
> +__log_level(2)
> +__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
> +__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
> +__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
> +__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
> +__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
> +__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
> +__naked int aggregate_ret_global_precise(void)
A subsystem pattern flags these two tests as potentially concerning.
The aggregate_ret_static_precise() and aggregate_ret_global_precise()
tests in the new aggregate_ret_func.c file are the R2 variant of
behavior that tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
already covers: subprog_result_precise() and global_subprog_result_precise()
in that file assert the same mark_precise log shape across the call/exit
boundary for static and global subprogs. The two new tests differ only in
which register (R2 vs R0) is being tracked, and in the final one or two
__msg lines.
This same commit did extend an existing file for its liveness test case
rather than adding a new one (it added aux2() and subprog_ret_reg_pair()
to compute_live_registers.c).
Would it be more natural to add these two precision tests as new cases
in verifier_subprog_precision.c, where a reader looks for precision-
backtracking coverage? The rest of aggregate_ret_func.c (the __failure
cases and the accepted pointer-pair case) is genuinely new-feature
coverage with no existing home, so perhaps that part stays separate.
[ ... ]
> diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
> index d055fc7b3b95..0be9441ec273 100644
> --- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
> +++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
> @@ -431,6 +431,36 @@ __naked void subprog1(void)
> ::: __clobber_all);
> }
>
> +static __used __naked unsigned __int128 aux2(void)
> +{
> + asm volatile (
> + "r0 = 1;"
> + "r2 = 2;"
> + "exit;"
> + ::: __clobber_all);
> +}
> +
> +SEC("socket")
> +/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */
> +__load_if_JITed()
> +__log_level(2)
> +__msg("0: .12345.... (85) call pc+2")
> +__msg("1: ..2....... (bf) r0 = r2")
> +/* R2 is not read at the exit of this program, which returns an int, ... */
> +__msg("2: 0......... (95) exit")
> +__msg("3: .......... (b7) r0 = 1")
> +__msg("4: 0......... (b7) r2 = 2")
> +/* ... but it is at the exit of aux2(), which returns a register pair. */
> +__msg("5: 0.2....... (95) exit")
> +__naked void subprog_ret_reg_pair(void)
The changelog describes aggregate_ret_func.c covering "liveness at the exit
of a subprogram returning a pair," but that test is not in that file.
aggregate_ret_func.c has no liveness test at all - its only __log_level(2)
tests match "mark_precise: ..." lines.
The liveness-at-exit test is here, in compute_live_registers.c: aux2() and
subprog_ret_reg_pair(), with the register-mask __msg expectations like
"0: .12345.... (85) call pc+2" and "5: 0.2....... (95) exit".
Conversely, the changelog does not mention the accepted-case test
aggregate_ret_static_ptr_pair() in aggregate_ret_func.c (a static subprog
legitimately returning a pointer in R2, whose result is then dereferenced),
and it never mentions the aggregate_ret_kfunc.c / aggregate_ret_kfunc_c.c
additions that make up roughly 180 of the commit's added lines. New kfuncs
in the test module are the kind of addition a reviewer looks for in the
changelog.
Could the message be adjusted so that aggregate_ret_func.c's description
matches what the file actually contains, the liveness test is attributed to
compute_live_registers.c, and the kfunc side of the patch gets a sentence?
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31740414277
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH bpf-next v5 11/11] Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2
2026-08-13 20:02 [PATCH bpf-next v5 00/11] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (9 preceding siblings ...)
2026-08-13 20:03 ` [PATCH bpf-next v5 10/11] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns Yonghong Song
@ 2026-08-13 20:03 ` Yonghong Song
10 siblings, 0 replies; 18+ messages in thread
From: Yonghong Song @ 2026-08-13 20:03 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
kfuncs may now return a value larger than 8 bytes and up to 16 bytes (a
scalar-only struct or union, or an __int128), passed back in the R0:R2
register pair. Add a kfunc return-value section documenting this,
including that a struct or union up to 8 bytes is returned in R0 alone,
which struct and union members are accepted, that the R0:R2 register pair
requires JIT support (bpf_jit_supports_kfunc_ret_reg_pair()), and that a
return value larger than 16 bytes is unsupported.
Also note that the same convention applies to BPF subprogram returns, and
document the consequence for a global subprogram: it must assign both
halves of a register-pair return, since an unassigned R2 may be left
holding a pointer argument and is then rejected as a leak.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
Documentation/bpf/kfuncs.rst | 63 ++++++++++++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index 10e725cbe64c..5d840d300033 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -575,6 +575,69 @@ is also covered by this recovery. A kfunc handed an arena pointer may
therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
against the arena. Larger accesses must verify the range explicitly.
+2.9 kfunc Return Values
+-----------------------
+
+A kfunc may return a scalar, a pointer, or a small struct or union by
+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.
+
+A struct or union returned by value must be composed only of scalars
+(recursively), where a scalar is an integer or an enum; arrays of scalars are
+allowed as members. Its bytes are handed back to the program as the raw
+contents of R0 (and R2), so a pointer field would be laundered into a scalar
+and escape the verifier's pointer provenance and reference tracking. A struct
+or union with a pointer member is therefore rejected at load time, and so is
+one with a floating-point member, which the ABI may not return in R0:R2 at
+all.
+
+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a
+scalar-only struct or union, or an ``__int128``. Such a value is returned
+in the register pair R0:R2, matching the convention LLVM uses for the BPF
+target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or
+union of 8 bytes or less is returned in R0 alone.
+
+::
+
+ struct bpf_pair { __u64 a, b; }; /* 16 bytes */
+
+ __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)
+ {
+ struct bpf_pair p = { .a = 1, .b = 2 };
+
+ return p; /* p.a in R0, p.b in R2 */
+ }
+
+Returning a value in the R0:R2 pair requires the JIT to place the second
+half of the return value into R2, which not every architecture supports
+right now. A kfunc with a return value larger than 8 bytes is therefore
+rejected at load time on a JIT that does not advertise this capability (see
+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run
+by the interpreter. A return value larger than 16 bytes is not supported.
+
+The same R0:R2 convention applies to a BPF subprogram, global or static,
+that returns an ``__int128`` or a struct or union larger than 8 bytes. Such a
+program also requires the JIT, since the interpreter propagates only R0 out
+of a subprogram. A global subprogram is verified in isolation, so its
+by-value struct or union return is restricted to scalars just like a kfunc's;
+a static subprogram is verified inline and has no such restriction. The main
+program is not covered: its return value is the program's exit code, read out
+of R0 alone, so a declared upper half is never looked at.
+
+A global subprogram must leave a scalar in *every* register of the pair, so
+both halves of the returned value have to be assigned. Leaving the upper half
+uninitialized is not merely untidy: the compiler is then free to leave R2
+holding whatever it happened to hold, which for a subprogram taking a pointer
+argument is typically that pointer. Handing the caller an unknown scalar built
+from a pointer is a leak, so the verifier rejects it with::
+
+ At subprogram exit the register R2 is not a scalar value (...)
+
+Initialize the whole return value, for example ``struct pair p = {};``, to
+avoid this. A static subprogram is exempt from the scalar-only rule: it is
+verified inline, so an unassigned R2 is simply passed back to the caller as
+uninitialized and only a caller that reads it fails. A stack pointer left in
+R2 is still rejected there, just as one in R0 is.
+
.. _BPF_kfunc_lifecycle_expectations:
3. kfunc lifecycle expectations
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 18+ messages in thread