* [PATCH bpf-next v6 01/10] bpf: Factor check_global_ret_scalar_reg() out of the global return check
2026-08-17 4:21 [PATCH bpf-next v6 00/10] bpf: Support aggregate return values up to 16 bytes Yonghong Song
@ 2026-08-17 4:21 ` Yonghong Song
2026-08-17 4:21 ` [PATCH bpf-next v6 02/10] bpf: Add helpers to describe the R0:R2 return register pair Yonghong Song
` (8 subsequent siblings)
9 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 4:21 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
check_global_subprog_return_code() verifies that a global subprogram
returns void, an arena pointer, or register R0 holding a scalar value.
Later patches in this series add 16-byte aggregate return support, whose
second half is returned in R2 and needs the same validation.
Factor the per-register check into check_global_ret_scalar_reg(env, regno)
so that it can be reused for R2. No functional change.
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/verifier.c | 32 ++++++++++++++++++++------------
1 file changed, 20 insertions(+), 12 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index d17f14b35b79..b3c474ba7140 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -17451,37 +17451,45 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
return 0;
}
-static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+static int check_global_ret_scalar_reg(struct bpf_verifier_env *env, u32 regno)
{
- struct bpf_reg_state *reg = reg_state(env, BPF_REG_0);
- struct bpf_func_state *cur_frame = cur_func(env);
+ struct bpf_reg_state *reg;
int err;
- if (subprog_returns_void(env, cur_frame->subprogno))
- return 0;
-
- err = check_reg_arg(env, BPF_REG_0, SRC_OP);
+ err = check_reg_arg(env, regno, SRC_OP);
if (err)
return err;
/* Pointers to arena are safe to pass between subprograms. */
- if (is_arena_reg(env, BPF_REG_0))
+ if (is_arena_reg(env, regno))
return 0;
- if (is_pointer_value(env, BPF_REG_0)) {
- verbose(env, "R%d leaks addr as return value\n", BPF_REG_0);
+ if (is_pointer_value(env, regno)) {
+ verbose(env, "R%d leaks addr as return value\n", regno);
return -EACCES;
}
+ reg = reg_state(env, regno);
if (reg->type != SCALAR_VALUE) {
- verbose(env, "At subprogram exit the register R0 is not a scalar value (%s)\n",
- reg_type_str(env, reg->type));
+ verbose(env, "At subprogram exit the register R%d is not a scalar value (%s)\n",
+ regno, reg_type_str(env, reg->type));
return -EINVAL;
}
return 0;
}
+static int check_global_subprog_return_code(struct bpf_verifier_env *env)
+{
+ struct bpf_func_state *cur_frame = cur_func(env);
+ u32 subprog = cur_frame->subprogno;
+
+ if (subprog_returns_void(env, subprog))
+ return 0;
+
+ return check_global_ret_scalar_reg(env, BPF_REG_0);
+}
+
/* Bitmask with 1s for all caller saved registers */
#define ALL_CALLER_SAVED_REGS ((1u << CALLER_SAVED_REGS) - 1)
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 31+ messages in thread* [PATCH bpf-next v6 02/10] bpf: Add helpers to describe the R0:R2 return register pair
2026-08-17 4:21 [PATCH bpf-next v6 00/10] bpf: Support aggregate return values up to 16 bytes Yonghong Song
2026-08-17 4:21 ` [PATCH bpf-next v6 01/10] bpf: Factor check_global_ret_scalar_reg() out of the global return check Yonghong Song
@ 2026-08-17 4:21 ` Yonghong Song
2026-08-17 5:17 ` bot+bpf-ci
2026-08-17 4:21 ` [PATCH bpf-next v6 03/10] bpf: Wire up JIT support for 16-byte kfunc returns Yonghong Song
` (7 subsequent siblings)
9 siblings, 1 reply; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 4:21 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more than
16 bytes: such a value comes back in the R0:R2 register pair, with R2
holding the upper half. See LLVM patches [1] and [2].
Later patches teach the JIT, live register analysis and the verifier itself
about that convention. They need to answer the same question: does this
subprogram return its value in a register pair? Add bpf_ret_reg_pair() up
front so it can be used in subsequent patches. It is answered from a
per-subprogram flag that bpf_compute_subprog_ret_regs() derives once from
the BTF prototype.
jit_requested only says that the JIT is enabled, not that it succeeded:
bpf_fixup_call_args() falls back to the interpreter when bpf_jit_subprogs()
fails with anything other than -EFAULT. So jit_required is still set once
the pair is modelled, which turns that fallback into a load failure rather
than a silent divergence from what was verified.
[1] https://github.com/llvm/llvm-project/pull/190894
[2] https://github.com/llvm/llvm-project/pull/206876
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
include/linux/bpf_verifier.h | 7 ++++
kernel/bpf/verifier.c | 70 ++++++++++++++++++++++++++++++------
2 files changed, 66 insertions(+), 11 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index bc2af02547fe..f70d5878fbff 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -819,6 +819,8 @@ struct bpf_subprog_info {
bool is_async_cb: 1;
bool is_exception_cb: 1;
bool args_cached: 1;
+ /* true if the return value is passed in the R0:R2 register pair */
+ bool ret_reg_pair: 1;
/* true if bpf_fastcall stack region is used by functions that can't be inlined */
bool keep_fastcall_stack: 1;
bool changes_pkt_data: 1;
@@ -1055,6 +1057,11 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
return &env->subprog_info[subprog];
}
+static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
+{
+ return subprog_info(env, subprog)->ret_reg_pair;
+}
+
struct bpf_call_summary {
u8 num_params;
bool is_void;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b3c474ba7140..f1f1268d29c6 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -385,27 +385,70 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
}
-static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)
{
- const struct btf_type *type, *func, *func_proto;
+ const struct btf_type *func, *func_proto;
const struct btf *btf = env->prog->aux->btf;
u32 btf_id;
+ if (!btf || !env->prog->aux->func_info)
+ return NULL;
+
btf_id = env->prog->aux->func_info[subprog].type_id;
+ /* Both already validated by prepare_btf_func() at prog load. */
func = btf_type_by_id(btf, btf_id);
- if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id))
- return false;
-
func_proto = btf_type_by_id(btf, func->type);
- if (!func_proto)
- return false;
- type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
- if (!type)
- return false;
+ return btf_type_skip_modifiers(btf, func_proto->type, NULL);
+}
- return btf_type_is_void(type);
+static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
+{
+ const struct btf_type *type = subprog_ret_type(env, subprog);
+
+ return type && btf_type_is_void(type);
+}
+
+static u32 ret_regs_cnt(u32 size)
+{
+ return size > 8 && size <= 16 ? 2 : 1;
+}
+
+static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
+{
+ const struct btf *btf = env->prog->aux->btf;
+ const struct btf_type *type;
+ int subprog;
+ u32 size;
+
+ if (!env->prog->jit_requested || bpf_prog_is_offloaded(env->prog->aux))
+ return 0;
+
+ /*
+ * Skip the main program: its return value is the program's exit code,
+ * read out of R0, so it never uses the register pair. An extension does
+ * have a real prototype for subprog 0, but bpf_check_attach_target()
+ * refuses to replace a function returning more than 8 bytes.
+ */
+ for (subprog = 1; subprog < env->subprog_cnt; subprog++) {
+ type = subprog_ret_type(env, subprog);
+ /*
+ * This runs before btf_validate_return_type(), so apply the same
+ * type filter here.
+ */
+ if (!type || !(btf_type_is_struct(type) || btf_type_is_scalar(type)))
+ continue;
+ if (verifier_bug_if(IS_ERR(btf_resolve_size(btf, type, &size)), env,
+ "cannot size return type of subprog %d", subprog))
+ return -EFAULT;
+ if (ret_regs_cnt(size) > 1) {
+ subprog_info(env, subprog)->ret_reg_pair = true;
+ env->prog->jit_required = 1;
+ }
+ }
+
+ return 0;
}
const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog)
@@ -21161,6 +21204,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
+ /* must precede the first bpf_ret_reg_pair() user below */
+ ret = bpf_compute_subprog_ret_regs(env);
+ if (ret < 0)
+ goto skip_full_check;
+
ret = bpf_compute_live_registers(env);
if (ret < 0)
goto skip_full_check;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 02/10] bpf: Add helpers to describe the R0:R2 return register pair
2026-08-17 4:21 ` [PATCH bpf-next v6 02/10] bpf: Add helpers to describe the R0:R2 return register pair Yonghong Song
@ 2026-08-17 5:17 ` bot+bpf-ci
2026-08-17 15:20 ` Yonghong Song
0 siblings, 1 reply; 31+ messages in thread
From: bot+bpf-ci @ 2026-08-17 5:17 UTC (permalink / raw)
To: yonghong.song, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 5450 bytes --]
>8 but <=16 bytes). It introduces bpf_compute_subprog_ret_regs() to
derive this from BTF prototypes.
> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
> index bc2af02547fe..f70d5878fbff 100644
> --- a/include/linux/bpf_verifier.h
> +++ b/include/linux/bpf_verifier.h
> @@ -819,6 +819,8 @@ struct bpf_subprog_info {
> bool is_async_cb: 1;
> bool is_exception_cb: 1;
> bool args_cached: 1;
> + /* true if the return value is passed in the R0:R2 register pair */
> + bool ret_reg_pair: 1;
> /* true if bpf_fastcall stack region is used by functions that can't be inlined */
> bool keep_fastcall_stack: 1;
> bool changes_pkt_data: 1;
> @@ -1055,6 +1057,11 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
> return &env->subprog_info[subprog];
> }
>
> +static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
> +{
> + return subprog_info(env, subprog)->ret_reg_pair;
> +}
> +
[ ... ]
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index b3c474ba7140..f1f1268d29c6 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -385,27 +385,70 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
> return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
> }
>
> -static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
> +static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)
> {
> - const struct btf_type *type, *func, *func_proto;
> + const struct btf_type *func, *func_proto;
> const struct btf *btf = env->prog->aux->btf;
> u32 btf_id;
>
> + if (!btf || !env->prog->aux->func_info)
> + return NULL;
> +
> btf_id = env->prog->aux->func_info[subprog].type_id;
>
> + /* Both already validated by prepare_btf_func() at prog load. */
> func = btf_type_by_id(btf, btf_id);
> - if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id))
> - return false;
> -
> func_proto = btf_type_by_id(btf, func->type);
> - if (!func_proto)
> - return false;
>
> - type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
> - if (!type)
> - return false;
> + return btf_type_skip_modifiers(btf, func_proto->type, NULL);
> +}
>
> - return btf_type_is_void(type);
> +static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
> +{
> + const struct btf_type *type = subprog_ret_type(env, subprog);
> +
> + return type && btf_type_is_void(type);
> +}
> +
> +static u32 ret_regs_cnt(u32 size)
> +{
> + return size > 8 && size <= 16 ? 2 : 1;
> +}
> +
> +static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
> +{
> + const struct btf *btf = env->prog->aux->btf;
> + const struct btf_type *type;
> + int subprog;
> + u32 size;
> +
> + if (!env->prog->jit_requested || bpf_prog_is_offloaded(env->prog->aux))
> + return 0;
> +
> + /*
> + * Skip the main program: its return value is the program's exit code,
> + * read out of R0, so it never uses the register pair. An extension does
> + * have a real prototype for subprog 0, but bpf_check_attach_target()
> + * refuses to replace a function returning more than 8 bytes.
> + */
> + for (subprog = 1; subprog < env->subprog_cnt; subprog++) {
> + type = subprog_ret_type(env, subprog);
> + /*
> + * This runs before btf_validate_return_type(), so apply the same
> + * type filter here.
> + */
> + if (!type || !(btf_type_is_struct(type) || btf_type_is_scalar(type)))
> + continue;
Does the comment's claim about applying "the same type filter" hold?
btf_validate_return_type() accepts void, int, or enum types via:
kernel/bpf/btf.c:btf_validate_return_type() {
/* We always accept void or scalars. */
if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
return 0;
return -EOPNOTSUPP;
}
The new filter instead accepts struct/union or scalar (int or enum, but
not ENUM64):
if (!type || !(btf_type_is_struct(type) || btf_type_is_scalar(type)))
btf_type_is_scalar() is defined as btf_type_is_int() || btf_type_is_enum(),
and btf_type_is_enum() excludes BTF_KIND_ENUM64.
The filters differ in several ways. btf_validate_return_type() accepts
void and ENUM64 (via btf_is_any_enum()) but rejects struct/union at this
commit. The new filter accepts struct/union but filters out void and
ENUM64.
The divergence is harmless since every dropped type is at most 8 bytes and
would need only one register anyway. But the comment implies an equivalence
that does not exist, and specifically suggests structs are validated
elsewhere when btf_validate_return_type() at this commit still rejects
them with -EOPNOTSUPP.
Would describing what the filter actually needs be clearer? Something like
"only struct/union and integer/enum returns can exceed 8 bytes and need
sizing" rather than claiming parity with btf_validate_return_type()?
> + if (verifier_bug_if(IS_ERR(btf_resolve_size(btf, type, &size)), env,
> + "cannot size return type of subprog %d", subprog))
> + return -EFAULT;
> + if (ret_regs_cnt(size) > 1) {
> + subprog_info(env, subprog)->ret_reg_pair = true;
> + env->prog->jit_required = 1;
> + }
> + }
> +
> + return 0;
> }
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31995069373
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 02/10] bpf: Add helpers to describe the R0:R2 return register pair
2026-08-17 5:17 ` bot+bpf-ci
@ 2026-08-17 15:20 ` Yonghong Song
0 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 15:20 UTC (permalink / raw)
To: bot+bpf-ci, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, martin.lau, clm,
ihor.solodrai
On 8/16/26 10:17 PM, bot+bpf-ci@kernel.org wrote:
>> 8 but <=16 bytes). It introduces bpf_compute_subprog_ret_regs() to
> derive this from BTF prototypes.
>
>> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
>> index bc2af02547fe..f70d5878fbff 100644
>> --- a/include/linux/bpf_verifier.h
>> +++ b/include/linux/bpf_verifier.h
>> @@ -819,6 +819,8 @@ struct bpf_subprog_info {
>> bool is_async_cb: 1;
>> bool is_exception_cb: 1;
>> bool args_cached: 1;
>> + /* true if the return value is passed in the R0:R2 register pair */
>> + bool ret_reg_pair: 1;
>> /* true if bpf_fastcall stack region is used by functions that can't be inlined */
>> bool keep_fastcall_stack: 1;
>> bool changes_pkt_data: 1;
>> @@ -1055,6 +1057,11 @@ static inline struct bpf_subprog_info *subprog_info(struct bpf_verifier_env *env
>> return &env->subprog_info[subprog];
>> }
>>
>> +static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
>> +{
>> + return subprog_info(env, subprog)->ret_reg_pair;
>> +}
>> +
> [ ... ]
>
>> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
>> index b3c474ba7140..f1f1268d29c6 100644
>> --- a/kernel/bpf/verifier.c
>> +++ b/kernel/bpf/verifier.c
> [ ... ]
>
>> @@ -385,27 +385,70 @@ bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog)
>> return aux && aux[subprog].linkage == BTF_FUNC_GLOBAL;
>> }
>>
>> -static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
>> +static const struct btf_type *subprog_ret_type(struct bpf_verifier_env *env, int subprog)
>> {
>> - const struct btf_type *type, *func, *func_proto;
>> + const struct btf_type *func, *func_proto;
>> const struct btf *btf = env->prog->aux->btf;
>> u32 btf_id;
>>
>> + if (!btf || !env->prog->aux->func_info)
>> + return NULL;
>> +
>> btf_id = env->prog->aux->func_info[subprog].type_id;
>>
>> + /* Both already validated by prepare_btf_func() at prog load. */
>> func = btf_type_by_id(btf, btf_id);
>> - if (verifier_bug_if(!func, env, "btf_id %u not found", btf_id))
>> - return false;
>> -
>> func_proto = btf_type_by_id(btf, func->type);
>> - if (!func_proto)
>> - return false;
>>
>> - type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
>> - if (!type)
>> - return false;
>> + return btf_type_skip_modifiers(btf, func_proto->type, NULL);
>> +}
>>
>> - return btf_type_is_void(type);
>> +static bool subprog_returns_void(struct bpf_verifier_env *env, int subprog)
>> +{
>> + const struct btf_type *type = subprog_ret_type(env, subprog);
>> +
>> + return type && btf_type_is_void(type);
>> +}
>> +
>> +static u32 ret_regs_cnt(u32 size)
>> +{
>> + return size > 8 && size <= 16 ? 2 : 1;
>> +}
>> +
>> +static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
>> +{
>> + const struct btf *btf = env->prog->aux->btf;
>> + const struct btf_type *type;
>> + int subprog;
>> + u32 size;
>> +
>> + if (!env->prog->jit_requested || bpf_prog_is_offloaded(env->prog->aux))
>> + return 0;
>> +
>> + /*
>> + * Skip the main program: its return value is the program's exit code,
>> + * read out of R0, so it never uses the register pair. An extension does
>> + * have a real prototype for subprog 0, but bpf_check_attach_target()
>> + * refuses to replace a function returning more than 8 bytes.
>> + */
>> + for (subprog = 1; subprog < env->subprog_cnt; subprog++) {
>> + type = subprog_ret_type(env, subprog);
>> + /*
>> + * This runs before btf_validate_return_type(), so apply the same
>> + * type filter here.
>> + */
>> + if (!type || !(btf_type_is_struct(type) || btf_type_is_scalar(type)))
>> + continue;
> Does the comment's claim about applying "the same type filter" hold?
>
> btf_validate_return_type() accepts void, int, or enum types via:
>
> kernel/bpf/btf.c:btf_validate_return_type() {
> /* We always accept void or scalars. */
> if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
> return 0;
>
> return -EOPNOTSUPP;
> }
>
> The new filter instead accepts struct/union or scalar (int or enum, but
> not ENUM64):
>
> if (!type || !(btf_type_is_struct(type) || btf_type_is_scalar(type)))
>
> btf_type_is_scalar() is defined as btf_type_is_int() || btf_type_is_enum(),
> and btf_type_is_enum() excludes BTF_KIND_ENUM64.
>
> The filters differ in several ways. btf_validate_return_type() accepts
> void and ENUM64 (via btf_is_any_enum()) but rejects struct/union at this
> commit. The new filter accepts struct/union but filters out void and
> ENUM64.
>
> The divergence is harmless since every dropped type is at most 8 bytes and
> would need only one register anyway. But the comment implies an equivalence
> that does not exist, and specifically suggests structs are validated
> elsewhere when btf_validate_return_type() at this commit still rejects
> them with -EOPNOTSUPP.
>
> Would describing what the filter actually needs be clearer? Something like
> "only struct/union and integer/enum returns can exceed 8 bytes and need
> sizing" rather than claiming parity with btf_validate_return_type()?
You are right. I missed enum64. I think the above 'if' condition should be
if (!type || !(btf_type_is_struct(type) || btf_type_is_int(type) || btf_is_any_enum(type)))
>
>> + if (verifier_bug_if(IS_ERR(btf_resolve_size(btf, type, &size)), env,
>> + "cannot size return type of subprog %d", subprog))
>> + return -EFAULT;
>> + if (ret_regs_cnt(size) > 1) {
>> + subprog_info(env, subprog)->ret_reg_pair = true;
>> + env->prog->jit_required = 1;
>> + }
>> + }
>> +
>> + return 0;
>> }
> [ ... ]
>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31995069373
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH bpf-next v6 03/10] bpf: Wire up JIT support for 16-byte kfunc returns
2026-08-17 4:21 [PATCH bpf-next v6 00/10] bpf: Support aggregate return values up to 16 bytes Yonghong Song
2026-08-17 4:21 ` [PATCH bpf-next v6 01/10] bpf: Factor check_global_ret_scalar_reg() out of the global return check Yonghong Song
2026-08-17 4:21 ` [PATCH bpf-next v6 02/10] bpf: Add helpers to describe the R0:R2 return register pair Yonghong Song
@ 2026-08-17 4:21 ` Yonghong Song
2026-08-17 4:37 ` sashiko-bot
2026-08-17 4:22 ` [PATCH bpf-next v6 04/10] bpf: Handle R2 as a return register in precision backtracking Yonghong Song
` (6 subsequent siblings)
9 siblings, 1 reply; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 4:21 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
LLVM 23 returns an __int128, or a struct/union larger than 8 bytes and no
larger than 16 bytes, in the BPF R0:R2 register pair. The previous patch
added the shared helpers describing that convention; wire up the JIT side
so that the second half of the return value actually lands in R2.
Placing the second return half into R2 is possible on any JIT, but it needs
architecture-specific JIT work. Rather than requiring every JIT to
implement it at once, add a bpf_jit_supports_kfunc_ret_reg_pair()
capability, defaulting to false in the generic core; an architecture opts
in once its JIT handles the R0:R2 pair, and the remaining ones are left for
future work. Only the x86-64, arm64 and riscv64 JITs opt in so far.
On arm64 and riscv64 the native second return register is already BPF R2
(x1 in bpf2a64[] and a1 in regmap[] respectively), so the upper half needs
no move at all, unlike x86-64's RDX->RSI. The lower half is covered by the
move into BPF R0 that those JITs already emit after every call, from x0
into x8 and from a0 into a5. This has been tested on x86-64 and arm64. The
riscv64 path is expected to work by the same register-mapping reasoning as
arm64 but has not been tested.
bpf_add_kfunc_call() also rejects a kfunc that is marked KF_FASTCALL and
returns more than 8 bytes. The bpf_fastcall contract implemented by
mark_fastcall_pattern_for_call() assumes a call clobbers R0 plus the
registers holding its arguments, so a return in the R0:R2 pair would
clobber an R2 the caller expects the fastcall pattern to preserve. Such
a kfunc is rejected with -EOPNOTSUPP as well.
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
arch/arm64/net/bpf_jit_comp.c | 5 +++++
arch/riscv/net/bpf_jit_comp64.c | 5 +++++
arch/x86/net/bpf_jit_comp.c | 27 ++++++++++++++++++++-------
include/linux/filter.h | 1 +
kernel/bpf/core.c | 5 +++++
kernel/bpf/verifier.c | 12 ++++++++++++
6 files changed, 48 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index c18e005a41db..3aa3ea0bc30b 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -2388,6 +2388,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_stack_args(void)
{
return true;
diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index 74efe4b138d2..47c7bf431ba8 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -2121,6 +2121,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_ptr_xchg(void)
{
return true;
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 1a9fb530adc3..48429fae0641 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1689,17 +1689,12 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
* arena NULL is offset 0. Return the number of emitted bytes.
*/
static int emit_kfunc_arena_args(struct bpf_prog *bpf_prog,
- const struct bpf_insn *insn, u8 **pprog)
+ const struct btf_func_model *fm, u8 **pprog)
{
- const struct btf_func_model *fm;
u8 *prog = *pprog;
u8 *start = prog;
int i;
- fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
- if (!fm)
- return -EINVAL;
-
for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
u8 flags = fm->arg_flags[i];
u32 reg = BPF_REG_1 + i;
@@ -2644,6 +2639,8 @@ st: insn_off = insn->off;
/* call */
case BPF_JMP | BPF_CALL: {
+ const struct btf_func_model *fm = NULL;
+
func = (u8 *) __bpf_call_base + imm32;
if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) {
LOAD_TAIL_CALL_CNT_PTR(stack_depth);
@@ -2652,7 +2649,10 @@ st: insn_off = insn->off;
if (!imm32)
return -EINVAL;
if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
- err = emit_kfunc_arena_args(bpf_prog, insn, &prog);
+ fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
+ if (!fm)
+ return -EINVAL;
+ err = emit_kfunc_arena_args(bpf_prog, fm, &prog);
if (err < 0)
return err;
ip += err;
@@ -2666,6 +2666,14 @@ st: insn_off = insn->off;
return -EINVAL;
if (priv_frame_ptr)
pop_r9(&prog);
+ /*
+ * A kfunc returning more than 8 bytes hands the second
+ * half back in RDX (the native ABI's second return reg),
+ * but BPF expects it in R0:R2. BPF R0 is RAX (no move
+ * needed), while BPF R2 is RSI, so copy RDX into RSI.
+ */
+ if (fm && fm->ret_size > 8)
+ emit_mov_reg(&prog, true, BPF_REG_2, BPF_REG_3);
break;
}
@@ -4156,6 +4164,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
+bool bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_stack_args(void)
{
return true;
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 4a9bc6a848f2..6e746b0a0930 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1237,6 +1237,7 @@ bool bpf_jit_inlines_helper_call(s32 imm);
bool bpf_jit_supports_subprog_tailcalls(void);
bool bpf_jit_supports_percpu_insn(void);
bool bpf_jit_supports_kfunc_call(void);
+bool bpf_jit_supports_kfunc_ret_reg_pair(void);
bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index d55e737ed75a..5db77d7915df 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3287,6 +3287,11 @@ bool __weak bpf_jit_supports_kfunc_call(void)
return false;
}
+bool __weak bpf_jit_supports_kfunc_ret_reg_pair(void)
+{
+ return false;
+}
+
bool __weak bpf_jit_supports_stack_args(void)
{
return false;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index f1f1268d29c6..54aca6c30506 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -2920,6 +2920,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
if (err)
return err;
+ if (func_model.ret_size > 8) {
+ if (kfunc.flags && (*kfunc.flags & KF_FASTCALL)) {
+ verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n",
+ kfunc.name);
+ return -EOPNOTSUPP;
+ }
+ if (!bpf_jit_supports_kfunc_ret_reg_pair()) {
+ verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
+ kfunc.name);
+ return -EOPNOTSUPP;
+ }
+ }
memset(&meta, 0, sizeof(meta));
meta.btf = kfunc.btf;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 03/10] bpf: Wire up JIT support for 16-byte kfunc returns
2026-08-17 4:21 ` [PATCH bpf-next v6 03/10] bpf: Wire up JIT support for 16-byte kfunc returns Yonghong Song
@ 2026-08-17 4:37 ` sashiko-bot
2026-08-17 15:25 ` Yonghong Song
0 siblings, 1 reply; 31+ messages in thread
From: sashiko-bot @ 2026-08-17 4:37 UTC (permalink / raw)
To: Yonghong Song; +Cc: bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The BPF verifier fails to initialize BPF_REG_2 for 16-byte scalar kfunc returns, breaking the feature.
--
--- Patch [3]: [PATCH bpf-next v6 03/10] bpf: Wire up JIT support for 16-byte kfunc returns ---
commit 70077e00d2d0c46ef0daa85eb2612ed7fbf545ef
Author: Yonghong Song <yonghong.song@linux.dev>
bpf: Wire up JIT support for 16-byte kfunc returns
This commit wires up JIT support for 16-byte kfunc returns in BPF, such as
large structs or 128-bit integers. It adds support to the x86-64, arm64, and
riscv64 JITs to return the second half of the value in BPF_REG_2, and adds
verifier checks to reject larger returns on unsupported architectures.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index f1f1268d29c66..54aca6c305061 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -2920,6 +2920,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
> err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
> if (err)
> return err;
> + if (func_model.ret_size > 8) {
> + if (kfunc.flags && (*kfunc.flags & KF_FASTCALL)) {
> + verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n",
> + kfunc.name);
> + return -EOPNOTSUPP;
> + }
> + if (!bpf_jit_supports_kfunc_ret_reg_pair()) {
> + verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
> + kfunc.name);
> + return -EOPNOTSUPP;
> + }
> + }
[Severity: Medium]
While this patch adds JIT support and basic verifier checks for >8-byte
returns, does the verifier correctly initialize BPF_REG_2 for 16-byte scalar
returns?
Looking at check_kfunc_call() in kernel/bpf/verifier.c, it seems only
BPF_REG_0 is marked as initialized for scalar returns:
if (btf_type_is_scalar(t)) {
mark_reg_unknown(env, regs, BPF_REG_0);
if (meta.btf == btf_vmlinux &&
(meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] || ...
Since BPF caller-saved registers (including BPF_REG_2) are marked as NOT_INIT
earlier in check_kfunc_call(), a BPF program attempting to access the upper
64 bits of a 16-byte scalar (like an __int128) in BPF_REG_2 would be rejected
with a verifier read error.
Should check_kfunc_call() also initialize BPF_REG_2 when the return size is
greater than 8 bytes?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260817042141.2286086-1-yonghong.song@linux.dev?part=3
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 03/10] bpf: Wire up JIT support for 16-byte kfunc returns
2026-08-17 4:37 ` sashiko-bot
@ 2026-08-17 15:25 ` Yonghong Song
0 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 15:25 UTC (permalink / raw)
To: sashiko-reviews; +Cc: bpf
On 8/16/26 9:37 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Medium] The BPF verifier fails to initialize BPF_REG_2 for 16-byte scalar kfunc returns, breaking the feature.
> --
>
> --- Patch [3]: [PATCH bpf-next v6 03/10] bpf: Wire up JIT support for 16-byte kfunc returns ---
> commit 70077e00d2d0c46ef0daa85eb2612ed7fbf545ef
> Author: Yonghong Song <yonghong.song@linux.dev>
> bpf: Wire up JIT support for 16-byte kfunc returns
>
> This commit wires up JIT support for 16-byte kfunc returns in BPF, such as
> large structs or 128-bit integers. It adds support to the x86-64, arm64, and
> riscv64 JITs to return the second half of the value in BPF_REG_2, and adds
> verifier checks to reject larger returns on unsupported architectures.
>
>> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
>> index f1f1268d29c66..54aca6c305061 100644
>> --- a/kernel/bpf/verifier.c
>> +++ b/kernel/bpf/verifier.c
> [ ... ]
>> @@ -2920,6 +2920,18 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset)
>> err = btf_distill_func_proto(&env->log, kfunc.btf, kfunc.proto, kfunc.name, &func_model);
>> if (err)
>> return err;
>> + if (func_model.ret_size > 8) {
>> + if (kfunc.flags && (*kfunc.flags & KF_FASTCALL)) {
>> + verbose(env, "kfunc %s with >8-byte return is not supported with KF_FASTCALL\n",
>> + kfunc.name);
>> + return -EOPNOTSUPP;
>> + }
>> + if (!bpf_jit_supports_kfunc_ret_reg_pair()) {
>> + verbose(env, "kfunc %s with >8-byte return is not supported by JIT\n",
>> + kfunc.name);
>> + return -EOPNOTSUPP;
>> + }
>> + }
> [Severity: Medium]
> While this patch adds JIT support and basic verifier checks for >8-byte
> returns, does the verifier correctly initialize BPF_REG_2 for 16-byte scalar
> returns?
>
> Looking at check_kfunc_call() in kernel/bpf/verifier.c, it seems only
> BPF_REG_0 is marked as initialized for scalar returns:
>
> if (btf_type_is_scalar(t)) {
> mark_reg_unknown(env, regs, BPF_REG_0);
> if (meta.btf == btf_vmlinux &&
> (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] || ...
>
> Since BPF caller-saved registers (including BPF_REG_2) are marked as NOT_INIT
> earlier in check_kfunc_call(), a BPF program attempting to access the upper
> 64 bits of a 16-byte scalar (like an __int128) in BPF_REG_2 would be rejected
> with a verifier read error.
>
> Should check_kfunc_call() also initialize BPF_REG_2 when the return size is
> greater than 8 bytes?
We should be okay here. The actual BPF_REG_2 will be used in
commit "bpf: Add verifier support for 16-byte returns in R0:R2"
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH bpf-next v6 04/10] bpf: Handle R2 as a return register in precision backtracking
2026-08-17 4:21 [PATCH bpf-next v6 00/10] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (2 preceding siblings ...)
2026-08-17 4:21 ` [PATCH bpf-next v6 03/10] bpf: Wire up JIT support for 16-byte kfunc returns Yonghong Song
@ 2026-08-17 4:22 ` Yonghong Song
2026-08-17 5:17 ` bot+bpf-ci
2026-08-17 4:22 ` [PATCH bpf-next v6 05/10] bpf: Account R2 of register-pair returns in live register analysis Yonghong Song
` (5 subsequent siblings)
9 siblings, 1 reply; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 4:22 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
Precision backtracking treats only R0 as a return register at a
call/return boundary, so once the verifier starts modeling R2 that way,
marking the second half of such a return precise would trip the
"unexpected regs" checks in backtrack_insn() and reject a valid
program.
Marking the upper half precise, for example by branching on it after a
call to a static subprogram, walks backtracking into the callee and
reaches its BPF_EXIT with R2 still set in the mask. Handle R2 like R0
in boundaries where a call defines the return registers.
R2 differs from R0 in that it is an argument register as well, so it is
part of the BPF_REGMASK_ARGS check and has to be cleared before that check
rather than next to R0. Clear it unconditionally, rather than only where
the callee or the kfunc really does return a pair. That gives up the
"unexpected regs" assertion for R2, and in exchange keeps backtracking
free of any BTF lookup. Nothing is lost: a callee that does not return
a pair leaves the caller's R2 uninitialized, so the main verification
pass has already rejected any program that reads it, and backtracking
is never asked for its precision.
At BPF_EXIT the return registers are sampled before the callback path
clears R1-R5. That clear does not touch R0, but it does cover R2, and
running it first would drop a pair return whenever the instruction
following the call happens to be one that invokes a callback.
Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/backtrack.c | 46 +++++++++++++++++++++++++++---------------
1 file changed, 30 insertions(+), 16 deletions(-)
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index a2b18a9f1694..653db80bcc47 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -423,6 +423,10 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
*/
verifier_bug_if(idx + 1 != subseq_idx, env,
"extra insn from subprog");
+ /* global subprog always sets R0 */
+ bt_clear_reg(bt, BPF_REG_0);
+ /* and if it does not set R2, main pass would catch it */
+ bt_clear_reg(bt, BPF_REG_2);
/* r1-r5 are invalidated after subprog call,
* so for global func call it shouldn't be set
* anymore
@@ -432,8 +436,6 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
bt_reg_mask(bt));
return -EFAULT;
}
- /* global subprog always sets R0 */
- bt_clear_reg(bt, BPF_REG_0);
return 0;
} else {
/* static subprog call instruction, which
@@ -506,6 +508,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return -ENOTSUPP;
/* regular helper call sets R0 */
bt_clear_reg(bt, BPF_REG_0);
+ /* kfunc might also set R2 */
+ bt_clear_reg(bt, BPF_REG_2);
if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
/* if backtracking was looking for registers R1-R5
* they should have been found already.
@@ -520,7 +524,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return -EFAULT;
}
} else if (opcode == BPF_EXIT) {
- bool r0_precise;
+ bool from_subprog_call, r0_precise, r2_precise;
+
+ /* BPF_EXIT in subprog or callback always returns
+ * right after the call instruction, so by checking
+ * whether the instruction at subseq_idx-1 is subprog
+ * call or not we can distinguish actual exit from
+ * *subprog* from exit from *callback*. In the former
+ * case, we need to propagate the precision of the
+ * return registers, if necessary. In the latter we
+ * never do that.
+ */
+ from_subprog_call = subseq_idx - 1 >= 0 &&
+ bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
+
+ /* Sample the return registers before the callback
+ * handling below clears R1-R5.
+ */
+ r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
+ r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
/* Backtracking to a nested function call, 'idx' is a part of
* the inner frame 'subseq_idx' is a part of the outer frame.
@@ -533,30 +555,22 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx))
for (i = BPF_REG_1; i <= BPF_REG_5; i++)
bt_clear_reg(bt, i);
+
+ bt_clear_reg(bt, BPF_REG_0);
+ bt_clear_reg(bt, BPF_REG_2);
if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
verifier_bug(env, "backtracking exit unexpected regs %x",
bt_reg_mask(bt));
return -EFAULT;
}
- /* BPF_EXIT in subprog or callback always returns
- * right after the call instruction, so by checking
- * whether the instruction at subseq_idx-1 is subprog
- * call or not we can distinguish actual exit from
- * *subprog* from exit from *callback*. In the former
- * case, we need to propagate r0 precision, if
- * necessary. In the former we never do that.
- */
- r0_precise = subseq_idx - 1 >= 0 &&
- bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) &&
- bt_is_reg_set(bt, BPF_REG_0);
-
- bt_clear_reg(bt, BPF_REG_0);
if (bt_subprog_enter(bt))
return -EFAULT;
if (r0_precise)
bt_set_reg(bt, BPF_REG_0);
+ if (r2_precise)
+ bt_set_reg(bt, BPF_REG_2);
/* r6-r9 and stack slots will stay set in caller frame
* bitmasks until we return back from callee(s)
*/
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 04/10] bpf: Handle R2 as a return register in precision backtracking
2026-08-17 4:22 ` [PATCH bpf-next v6 04/10] bpf: Handle R2 as a return register in precision backtracking Yonghong Song
@ 2026-08-17 5:17 ` bot+bpf-ci
2026-08-17 15:27 ` Yonghong Song
0 siblings, 1 reply; 31+ messages in thread
From: bot+bpf-ci @ 2026-08-17 5:17 UTC (permalink / raw)
To: yonghong.song, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3948 bytes --]
> bpf: Handle R2 as a return register in precision backtracking
>
> Precision backtracking treats only R0 as a return register at a
> call/return boundary, so once the verifier starts modeling R2 that way,
> marking the second half of such a return precise would trip the
> "unexpected regs" checks in backtrack_insn() and reject a valid
> program.
>
> Marking the upper half precise, for example by branching on it after a
> call to a static subprogram, walks backtracking into the callee and
> reaches its BPF_EXIT with R2 still set in the mask. Handle R2 like R0
> in boundaries where a call defines the return registers.
>
> R2 differs from R0 in that it is an argument register as well, so it is
> part of the BPF_REGMASK_ARGS check and has to be cleared before that check
> rather than next to R0. Clear it unconditionally, rather than only where
> the callee or the kfunc really does return a pair. That gives up the
> "unexpected regs" assertion for R2, and in exchange keeps backtracking
> free of any BTF lookup. Nothing is lost: a callee that does not return
> a pair leaves the caller's R2 uninitialized, so the main verification
> pass has already rejected any program that reads it, and backtracking
> is never asked for its precision.
>
> At BPF_EXIT the return registers are sampled before the callback path
> clears R1-R5. That clear does not touch R0, but it does cover R2, and
> running it first would drop a pair return whenever the instruction
> following the call happens to be one that invokes a callback.
>
> Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
This looks like a bug fix for 16-byte return support. Should this
include:
Fixes: 162762023c59 ("bpf: Add helpers to describe the R0:R2 return register pair")
> diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
> index a2b18a9f1694..653db80bcc47 100644
> --- a/kernel/bpf/backtrack.c
> +++ b/kernel/bpf/backtrack.c
[ ... ]
> @@ -520,7 +524,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
> return -EFAULT;
> }
> } else if (opcode == BPF_EXIT) {
> - bool r0_precise;
> + bool from_subprog_call, r0_precise, r2_precise;
> +
> + /* BPF_EXIT in subprog or callback always returns
> + * right after the call instruction, so by checking
> + * whether the instruction at subseq_idx-1 is subprog
> + * call or not we can distinguish actual exit from
> + * *subprog* from exit from *callback*. In the former
> + * case, we need to propagate the precision of the
> + * return registers, if necessary. In the latter we
> + * never do that.
> + */
> + from_subprog_call = subseq_idx - 1 >= 0 &&
> + bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
> +
> + /* Sample the return registers before the callback
> + * handling below clears R1-R5.
> + */
> + r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
> + r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
A subsystem pattern flags this as potentially concerning: the multi-line
comment above starts its text on the opening /* line:
/* Sample the return registers before the callback
* handling below clears R1-R5.
*/
The BPF subsystem guide suggests putting /* on its own line for
multi-line comments under kernel/bpf/:
/*
* Sample the return registers before the callback
* handling below clears R1-R5.
*/
Every existing multi-line comment in backtrack.c uses the older form,
including the comment block that this same hunk moves, so matching the
file's dominant style is a plausible reason to write it this way. Does
the newer form matter here?
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31995069373
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 04/10] bpf: Handle R2 as a return register in precision backtracking
2026-08-17 5:17 ` bot+bpf-ci
@ 2026-08-17 15:27 ` Yonghong Song
0 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 15:27 UTC (permalink / raw)
To: bot+bpf-ci, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, martin.lau, clm,
ihor.solodrai
On 8/16/26 10:17 PM, bot+bpf-ci@kernel.org wrote:
>> bpf: Handle R2 as a return register in precision backtracking
>>
>> Precision backtracking treats only R0 as a return register at a
>> call/return boundary, so once the verifier starts modeling R2 that way,
>> marking the second half of such a return precise would trip the
>> "unexpected regs" checks in backtrack_insn() and reject a valid
>> program.
>>
>> Marking the upper half precise, for example by branching on it after a
>> call to a static subprogram, walks backtracking into the callee and
>> reaches its BPF_EXIT with R2 still set in the mask. Handle R2 like R0
>> in boundaries where a call defines the return registers.
>>
>> R2 differs from R0 in that it is an argument register as well, so it is
>> part of the BPF_REGMASK_ARGS check and has to be cleared before that check
>> rather than next to R0. Clear it unconditionally, rather than only where
>> the callee or the kfunc really does return a pair. That gives up the
>> "unexpected regs" assertion for R2, and in exchange keeps backtracking
>> free of any BTF lookup. Nothing is lost: a callee that does not return
>> a pair leaves the caller's R2 uninitialized, so the main verification
>> pass has already rejected any program that reads it, and backtracking
>> is never asked for its precision.
>>
>> At BPF_EXIT the return registers are sampled before the callback path
>> clears R1-R5. That clear does not touch R0, but it does cover R2, and
>> running it first would drop a pair return whenever the instruction
>> following the call happens to be one that invokes a callback.
>>
>> Suggested-by: Eduard Zingerman <eddyz87@gmail.com>
>> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> This looks like a bug fix for 16-byte return support. Should this
> include:
>
> Fixes: 162762023c59 ("bpf: Add helpers to describe the R0:R2 return register pair")
This Fixes tag is not needed, esp. it is in the same series.
>
>> diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
>> index a2b18a9f1694..653db80bcc47 100644
>> --- a/kernel/bpf/backtrack.c
>> +++ b/kernel/bpf/backtrack.c
> [ ... ]
>
>> @@ -520,7 +524,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
>> return -EFAULT;
>> }
>> } else if (opcode == BPF_EXIT) {
>> - bool r0_precise;
>> + bool from_subprog_call, r0_precise, r2_precise;
>> +
>> + /* BPF_EXIT in subprog or callback always returns
>> + * right after the call instruction, so by checking
>> + * whether the instruction at subseq_idx-1 is subprog
>> + * call or not we can distinguish actual exit from
>> + * *subprog* from exit from *callback*. In the former
>> + * case, we need to propagate the precision of the
>> + * return registers, if necessary. In the latter we
>> + * never do that.
>> + */
>> + from_subprog_call = subseq_idx - 1 >= 0 &&
>> + bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
>> +
>> + /* Sample the return registers before the callback
>> + * handling below clears R1-R5.
>> + */
>> + r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
>> + r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
> A subsystem pattern flags this as potentially concerning: the multi-line
> comment above starts its text on the opening /* line:
>
> /* Sample the return registers before the callback
> * handling below clears R1-R5.
> */
>
> The BPF subsystem guide suggests putting /* on its own line for
> multi-line comments under kernel/bpf/:
>
> /*
> * Sample the return registers before the callback
> * handling below clears R1-R5.
> */
>
> Every existing multi-line comment in backtrack.c uses the older form,
> including the comment block that this same hunk moves, so matching the
> file's dominant style is a plausible reason to write it this way. Does
> the newer form matter here?
I think it is okay. The new comment format is from the old one.
>
> [ ... ]
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31995069373
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH bpf-next v6 05/10] bpf: Account R2 of register-pair returns in live register analysis
2026-08-17 4:21 [PATCH bpf-next v6 00/10] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (3 preceding siblings ...)
2026-08-17 4:22 ` [PATCH bpf-next v6 04/10] bpf: Handle R2 as a return register in precision backtracking Yonghong Song
@ 2026-08-17 4:22 ` Yonghong Song
2026-08-17 4:43 ` sashiko-bot
2026-08-17 4:22 ` [PATCH bpf-next v6 06/10] bpf: Add verifier support for 16-byte returns in R0:R2 Yonghong Song
` (4 subsequent siblings)
9 siblings, 1 reply; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 4:22 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
A BPF_EXIT of a subprogram returning a value larger than 8 bytes (a
struct/union or an __int128) reads R2 as well as R0, since the second half
of the return value is passed back in R2. compute_insn_live_regs() only
marked R0 used at exit, so a callee's R2 could be considered dead and
cleaned from checkpointed states, which would allow unsound state pruning.
Mark R2 as read at the BPF_EXIT of a subprogram that does return a register
pair. bpf_compute_live_registers() now loops over the subprograms and, for
each, over the [start, end) instruction range from env->subprog_info[], so
the return convention is queried once per subprogram through
bpf_ret_reg_pair() rather than once per instruction.
Marking R2 at every exit instead would be simpler, but R2 would then stay
live backwards across any call that is not followed by a write to R2, which
is nearly every program, and would needlessly hurt state pruning.
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/liveness.c | 20 ++++++++++++++------
1 file changed, 14 insertions(+), 6 deletions(-)
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 74fc4b3f80d6..71f998c6eb88 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2060,7 +2060,8 @@ static inline u16 mask_hi(u32 m) { return (u16)(m >> 16); }
/* Compute info->{use,def} fields for the instruction */
static void compute_insn_live_regs(struct bpf_verifier_env *env,
struct bpf_insn *insn,
- struct insn_live_regs *info)
+ struct insn_live_regs *info,
+ bool ret_reg_pair)
{
struct bpf_call_summary cs;
const u8 class = BPF_CLASS(insn->code);
@@ -2072,6 +2073,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
const u32 src32 = mask_lo(src);
const u32 dst32 = mask_lo(dst);
const u32 r0 = reg64_mask(0);
+ const u32 r2 = reg64_mask(BPF_REG_2);
u32 def = 0;
u32 use = U32_MAX;
@@ -2191,7 +2193,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
break;
case BPF_EXIT:
def = 0;
- use = r0;
+ use = ret_reg_pair ? (r0 | r2) : r0;
break;
case BPF_CALL:
def = ALL_CALLER_SAVED_REGS;
@@ -2228,8 +2230,8 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
struct insn_live_regs *state;
int insn_cnt = env->prog->len;
u64 pos, insn_pos;
- int err = 0, i, j;
- bool changed;
+ int err = 0, i, j, subprog, start, end;
+ bool changed, ret_reg_pair;
/* Use the following algorithm:
* - define the following:
@@ -2256,8 +2258,14 @@ int bpf_compute_live_registers(struct bpf_verifier_env *env)
goto out;
}
- for (i = 0; i < insn_cnt; ++i)
- compute_insn_live_regs(env, &insns[i], &state[i]);
+ for (subprog = 0; subprog < env->subprog_cnt; subprog++) {
+ start = env->subprog_info[subprog].start;
+ end = env->subprog_info[subprog + 1].start;
+ ret_reg_pair = bpf_ret_reg_pair(env, subprog);
+
+ for (i = start; i < end; ++i)
+ compute_insn_live_regs(env, &insns[i], &state[i], ret_reg_pair);
+ }
/* Forward pass: resolve stack access through FP-derived pointers */
err = bpf_compute_subprog_arg_access(env);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 05/10] bpf: Account R2 of register-pair returns in live register analysis
2026-08-17 4:22 ` [PATCH bpf-next v6 05/10] bpf: Account R2 of register-pair returns in live register analysis Yonghong Song
@ 2026-08-17 4:43 ` sashiko-bot
2026-08-17 15:40 ` Yonghong Song
0 siblings, 1 reply; 31+ messages in thread
From: sashiko-bot @ 2026-08-17 4:43 UTC (permalink / raw)
To: Yonghong Song; +Cc: bpf
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Critical] When a BPF program with 16-byte returns falls back to the interpreter (e.g., if JIT is disabled), the interpreter silently discards the upper 8 bytes (R2) of the return value, causing runtime data corruption.
- [High] The verifier fails to propagate or initialize BPF_REG_2 back to the caller for subprograms and kfuncs returning 16-byte values, rendering the feature unusable.
- [High] The verifier fails to validate the safety of BPF_REG_2 at the exit of global subprograms returning 16-byte values, allowing potential pointer leaks or returning uninitialized memory.
--
--- Patch [5]: [PATCH bpf-next v6 05/10] bpf: Account R2 of register-pair returns in live register analysis ---
commit 1e84baf022beac38fd11fa1a392f1c5622154586
Author: Yonghong Song <yonghong.song@linux.dev>
bpf: Account R2 of register-pair returns in live register analysis
This commit updates the live register analysis to account for subprograms
returning a 16-byte register pair. It modifies compute_insn_live_regs()
to mark BPF_REG_2 as read during BPF_EXIT when a subprogram returns a
register pair.
> diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
> index 74fc4b3f80d6e..71f998c6eb888 100644
> --- a/kernel/bpf/liveness.c
> +++ b/kernel/bpf/liveness.c
[ ... ]
> @@ -2191,7 +2193,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
> break;
> case BPF_EXIT:
> def = 0;
> - use = r0;
> + use = ret_reg_pair ? (r0 | r2) : r0;
> break;
[Severity: High]
How does the verifier propagate BPF_REG_2 back to the caller for subprograms
and kfuncs returning 16-byte values?
In kernel/bpf/verifier.c:prepare_func_exit(), only BPF_REG_0 is copied from
the callee's state to the caller's state:
kernel/bpf/verifier.c:prepare_func_exit() {
...
/* return to the caller whatever r0 had in the callee */
bpf_diag_mod_begin(env, &caller->regs[BPF_REG_0], r0, BPF_DIAG_MOD_WRITE);
caller->regs[BPF_REG_0] = *r0;
bpf_diag_mod_end(env);
...
}
Similarly, when returning from a non-void global subprogram or a kfunc, only
BPF_REG_0 is initialized as a scalar. For instance, in check_func_call():
kernel/bpf/verifier.c:check_func_call() {
...
/* All non-void global functions return a 64-bit SCALAR_VALUE. */
if (!returns_void) {
mark_reg_unknown(env, caller->regs, BPF_REG_0);
bpf_diag_mod_end(env);
}
...
}
And in check_kfunc_call():
kernel/bpf/verifier.c:check_kfunc_call() {
...
if (btf_type_is_scalar(t)) {
mark_reg_unknown(env, regs, BPF_REG_0);
...
}
Since clear_caller_saved_regs() marks R1-R5 as NOT_INIT prior to the call,
would this leave the caller's BPF_REG_2 uninitialized, causing the verifier
to reject legitimate reads of the upper 8 bytes of the return value?
[Severity: High]
Is the safety of BPF_REG_2 validated at the exit of global subprograms
returning 16-byte values?
In kernel/bpf/verifier.c:check_global_subprog_return_code(), the return code
validation is only performed for BPF_REG_0:
kernel/bpf/verifier.c:check_global_subprog_return_code() {
...
return check_global_ret_scalar_reg(env, BPF_REG_0);
}
Could this omit BPF_REG_2 for register-pair returns, potentially allowing a
global subprogram to return an unreferenced pointer, a map value pointer, or
uninitialized memory without being caught by the verifier?
[Severity: Critical]
What happens if a BPF program with 16-byte returns falls back to the
interpreter?
If JIT is not requested, bpf_compute_subprog_ret_regs() exits early without
setting env->prog->jit_required = 1:
kernel/bpf/verifier.c:bpf_compute_subprog_ret_regs() {
...
if (!env->prog->jit_requested || bpf_prog_is_offloaded(env->prog->aux))
return 0;
...
}
This permits the program to fall back to the interpreter. However, the
interpreter handles subprogram calls via JMP_CALL_ARGS, which only propagates
BPF_R0:
kernel/bpf/core.c:___bpf_prog_run() {
...
JMP_CALL_ARGS:
BPF_R0 = interpreters_args[insn->off](BPF_R1, BPF_R2, BPF_R3,
BPF_R4, BPF_R5,
insn + insn->imm + 1);
CONT;
...
}
Since the PROG_NAME_ARGS macro generates functions that return a single u64,
does the interpreter silently discard the upper 8 bytes (R2) of the return
value, causing data corruption at runtime?
> case BPF_CALL:
> def = ALL_CALLER_SAVED_REGS;
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260817042141.2286086-1-yonghong.song@linux.dev?part=5
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 05/10] bpf: Account R2 of register-pair returns in live register analysis
2026-08-17 4:43 ` sashiko-bot
@ 2026-08-17 15:40 ` Yonghong Song
0 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 15:40 UTC (permalink / raw)
To: sashiko-reviews; +Cc: bpf
On 8/16/26 9:43 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
> - [Critical] When a BPF program with 16-byte returns falls back to the interpreter (e.g., if JIT is disabled), the interpreter silently discards the upper 8 bytes (R2) of the return value, causing runtime data corruption.
> - [High] The verifier fails to propagate or initialize BPF_REG_2 back to the caller for subprograms and kfuncs returning 16-byte values, rendering the feature unusable.
> - [High] The verifier fails to validate the safety of BPF_REG_2 at the exit of global subprograms returning 16-byte values, allowing potential pointer leaks or returning uninitialized memory.
> --
>
> --- Patch [5]: [PATCH bpf-next v6 05/10] bpf: Account R2 of register-pair returns in live register analysis ---
> commit 1e84baf022beac38fd11fa1a392f1c5622154586
> Author: Yonghong Song <yonghong.song@linux.dev>
>
> bpf: Account R2 of register-pair returns in live register analysis
>
> This commit updates the live register analysis to account for subprograms
> returning a 16-byte register pair. It modifies compute_insn_live_regs()
> to mark BPF_REG_2 as read during BPF_EXIT when a subprogram returns a
> register pair.
>
>> diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
>> index 74fc4b3f80d6e..71f998c6eb888 100644
>> --- a/kernel/bpf/liveness.c
>> +++ b/kernel/bpf/liveness.c
> [ ... ]
>> @@ -2191,7 +2193,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
>> break;
>> case BPF_EXIT:
>> def = 0;
>> - use = r0;
>> + use = ret_reg_pair ? (r0 | r2) : r0;
>> break;
> [Severity: High]
> How does the verifier propagate BPF_REG_2 back to the caller for subprograms
> and kfuncs returning 16-byte values?
>
> In kernel/bpf/verifier.c:prepare_func_exit(), only BPF_REG_0 is copied from
> the callee's state to the caller's state:
>
> kernel/bpf/verifier.c:prepare_func_exit() {
> ...
> /* return to the caller whatever r0 had in the callee */
> bpf_diag_mod_begin(env, &caller->regs[BPF_REG_0], r0, BPF_DIAG_MOD_WRITE);
> caller->regs[BPF_REG_0] = *r0;
> bpf_diag_mod_end(env);
> ...
> }
>
> Similarly, when returning from a non-void global subprogram or a kfunc, only
> BPF_REG_0 is initialized as a scalar. For instance, in check_func_call():
>
> kernel/bpf/verifier.c:check_func_call() {
> ...
> /* All non-void global functions return a 64-bit SCALAR_VALUE. */
> if (!returns_void) {
> mark_reg_unknown(env, caller->regs, BPF_REG_0);
> bpf_diag_mod_end(env);
> }
> ...
> }
>
> And in check_kfunc_call():
>
> kernel/bpf/verifier.c:check_kfunc_call() {
> ...
> if (btf_type_is_scalar(t)) {
> mark_reg_unknown(env, regs, BPF_REG_0);
> ...
> }
>
> Since clear_caller_saved_regs() marks R1-R5 as NOT_INIT prior to the call,
> would this leave the caller's BPF_REG_2 uninitialized, causing the verifier
> to reject legitimate reads of the upper 8 bytes of the return value?
This should be okay. The BPF_REG_2 is introduced in later patch
bpf: Add verifier support for 16-byte returns in R0:R2
>
> [Severity: High]
> Is the safety of BPF_REG_2 validated at the exit of global subprograms
> returning 16-byte values?
>
> In kernel/bpf/verifier.c:check_global_subprog_return_code(), the return code
> validation is only performed for BPF_REG_0:
>
> kernel/bpf/verifier.c:check_global_subprog_return_code() {
> ...
> return check_global_ret_scalar_reg(env, BPF_REG_0);
> }
>
> Could this omit BPF_REG_2 for register-pair returns, potentially allowing a
> global subprogram to return an unreferenced pointer, a map value pointer, or
> uninitialized memory without being caught by the verifier?
The same as above. The BPF_REG_2 is introduced in later patch
bpf: Add verifier support for 16-byte returns in R0:R2
>
> [Severity: Critical]
> What happens if a BPF program with 16-byte returns falls back to the
> interpreter?
>
> If JIT is not requested, bpf_compute_subprog_ret_regs() exits early without
> setting env->prog->jit_required = 1:
>
> kernel/bpf/verifier.c:bpf_compute_subprog_ret_regs() {
> ...
> if (!env->prog->jit_requested || bpf_prog_is_offloaded(env->prog->aux))
> return 0;
> ...
> }
>
> This permits the program to fall back to the interpreter. However, the
> interpreter handles subprogram calls via JMP_CALL_ARGS, which only propagates
> BPF_R0:
>
> kernel/bpf/core.c:___bpf_prog_run() {
> ...
> JMP_CALL_ARGS:
> BPF_R0 = interpreters_args[insn->off](BPF_R1, BPF_R2, BPF_R3,
> BPF_R4, BPF_R5,
> insn + insn->imm + 1);
> CONT;
> ...
> }
>
> Since the PROG_NAME_ARGS macro generates functions that return a single u64,
> does the interpreter silently discard the upper 8 bytes (R2) of the return
> value, causing data corruption at runtime?
>
>> case BPF_CALL:
>> def = ALL_CALLER_SAVED_REGS;
In such case (jit not requested), verifier will fail. For example,
suppose the subprog will return R0:R2, but in interpreter mode, only
R0 is returned. R2 is not initialized. In caller, when trying to
return R2, but R2 is not initialized, so verifier will fail.
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH bpf-next v6 06/10] bpf: Add verifier support for 16-byte returns in R0:R2
2026-08-17 4:21 [PATCH bpf-next v6 00/10] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (4 preceding siblings ...)
2026-08-17 4:22 ` [PATCH bpf-next v6 05/10] bpf: Account R2 of register-pair returns in live register analysis Yonghong Song
@ 2026-08-17 4:22 ` Yonghong Song
2026-08-17 4:42 ` sashiko-bot
2026-08-17 5:17 ` bot+bpf-ci
2026-08-17 4:22 ` [PATCH bpf-next v6 07/10] bpf: Enable aggregate return types up to 16 bytes Yonghong Song
` (3 subsequent siblings)
9 siblings, 2 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 4:22 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
LLVM 23 added support for returning a value in two registers for an
__int128, or a struct/union whose size is greater than 8 but not more than
16 bytes. See LLVM patches [1] and [2].
Before LLVM 23 the BPF backend could not return these values at all. A
by-value struct or union return (of any size) was rejected at compile time
with:
error: aggregate returns are not supported
and an __int128 return failed later in the backend with:
fatal error: error in backend: unable to allocate function return #1
Both are resolved in LLVM 23, which lowers such returns into the R0:R2
register pair.
Model that pair at calls to global and static BPF subprograms and at kfunc
calls: R2 is marked alongside R0 at the call, propagated out of a callee
at its exit, and held to the same scalar-only and no-stack-pointer rules
that already apply to R0. A struct returned by a kfunc must be composed of
scalars, since its bytes reach the program as raw register contents and a
pointer field would otherwise be laundered into a scalar.
An extension program is the one caller of the convention that cannot take
part in it: only R0 is checked at its exit, since that is the program exit
code, so it has no way to hand back an upper half. Replacing a function
whose return value is larger than 8 bytes is therefore rejected in
btf_check_func_type_match(). That function compares btf_type->info, which
carries no size for an int, so an 8 byte long and a 16 byte __int128
compared equal; sizes up to 8 bytes stay interchangeable, as they always
have been, so this only rejects what the R0:R2 convention newly makes
incompatible.
[1] https://github.com/llvm/llvm-project/pull/190894
[2] https://github.com/llvm/llvm-project/pull/206876
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/btf.c | 6 ++++
kernel/bpf/verifier.c | 68 ++++++++++++++++++++++++++++++++++++++-----
2 files changed, 67 insertions(+), 7 deletions(-)
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 5b9d767895c9..2ae7cb9b30f2 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -7684,6 +7684,12 @@ static int btf_check_func_type_match(struct bpf_verifier_log *log,
btf_type_str(t2), fn2);
return -EINVAL;
}
+ if (btf_type_has_size(t1) && (t1->size > 8 || t2->size > 8)) {
+ bpf_log(log,
+ "Return type of %s() has size %u while %s() has size %u, and a size above 8 bytes cannot be replaced\n",
+ fn1, t1->size, fn2, t2->size);
+ return -EINVAL;
+ }
for (i = 0; i < nargs1; i++) {
t1 = btf_type_skip_modifiers(btf1, args1[i].type, NULL);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 54aca6c30506..e371b7e27ec9 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -415,6 +415,9 @@ static u32 ret_regs_cnt(u32 size)
return size > 8 && size <= 16 ? 2 : 1;
}
+/* Registers holding a function return value, in order. See ret_regs_cnt(). */
+static const int ret_regs[] = { BPF_REG_0, BPF_REG_2 };
+
static int bpf_compute_subprog_ret_regs(struct bpf_verifier_env *env)
{
const struct btf *btf = env->prog->aux->btf;
@@ -9910,6 +9913,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
u16 callee_incoming, stack_arg_cnt;
struct bpf_func_state *caller;
int err, subprog, target_insn;
+ u32 i, nregs;
target_insn = *insn_idx + insn->imm + 1;
subprog = bpf_find_subprog(env, target_insn);
@@ -9965,9 +9969,14 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
clear_caller_saved_regs(env, caller->regs);
invalidate_outgoing_stack_args(env, cur_func(env));
- /* All non-void global functions return a 64-bit SCALAR_VALUE. */
+ /*
+ * A non-void global function returns a 64-bit SCALAR_VALUE in
+ * R0, or a >8 byte SCALAR_VALUE in the R0:R2 register pair.
+ */
if (!returns_void) {
- mark_reg_unknown(env, caller->regs, BPF_REG_0);
+ nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+ for (i = 0; i < nregs; i++)
+ mark_reg_unknown(env, caller->regs, ret_regs[i]);
bpf_diag_mod_end(env);
}
@@ -10327,11 +10336,15 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
struct bpf_func_state *caller, *callee;
struct bpf_reg_state *r0;
bool in_callback_fn;
+ u32 i, nregs;
int err;
callee = state->frame[state->curframe];
r0 = &callee->regs[BPF_REG_0];
- if (r0->type == PTR_TO_STACK) {
+ nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+ for (i = 0; i < nregs; i++) {
+ if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
+ continue;
/* technically it's ok to return caller's stack pointer
* (or caller's caller's pointer) back to the caller,
* since these pointers are valid. Only current stack
@@ -10366,9 +10379,13 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
return -EFAULT;
}
} else {
- /* return to the caller whatever r0 had in the callee */
+ /*
+ * return to the caller whatever the callee had in the
+ * return register(s)
+ */
bpf_diag_mod_begin(env, &caller->regs[BPF_REG_0], r0, BPF_DIAG_MOD_WRITE);
- caller->regs[BPF_REG_0] = *r0;
+ for (i = 0; i < nregs; i++)
+ caller->regs[ret_regs[i]] = callee->regs[ret_regs[i]];
bpf_diag_mod_end(env);
}
@@ -11303,6 +11320,19 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
return 0;
}
+/*
+ * Mark the register(s) holding a @size byte kfunc return value as unknown
+ * scalars. Both halves of a register pair are treated the same way.
+ */
+static void mark_kfunc_ret_regs(struct bpf_verifier_env *env,
+ struct bpf_reg_state *regs, u32 size)
+{
+ u32 i, nregs = ret_regs_cnt(size);
+
+ for (i = 0; i < nregs; i++)
+ mark_reg_unknown(env, regs, ret_regs[i]);
+}
+
static bool is_kfunc_acquire(struct bpf_call_arg_meta *meta)
{
return meta->kfunc_flags & KF_ACQUIRE;
@@ -13967,10 +13997,25 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
}
if (btf_type_is_scalar(t)) {
- mark_reg_unknown(env, regs, BPF_REG_0);
+ mark_kfunc_ret_regs(env, regs, t->size);
if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
__mark_reg_const_zero(env, ®s[BPF_REG_0]);
+ } else if (btf_type_is_struct(t)) {
+ /*
+ * The returned struct comes back as raw register bits modeled
+ * as an unknown scalar, so it must contain only scalars:
+ * otherwise a pointer field would be laundered into a scalar
+ * and escape provenance and reference tracking.
+ */
+ if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+ verbose(env,
+ "kernel function %s returns %s %s that is not composed of scalars\n",
+ func_name, btf_type_str(t),
+ btf_name_by_offset(desc_btf, t->name_off));
+ return -EINVAL;
+ }
+ mark_kfunc_ret_regs(env, regs, t->size);
} else if (btf_type_is_ptr(t)) {
ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
err = check_special_kfunc(env, &meta, regs, insn_aux, ptr_type, desc_btf);
@@ -17538,11 +17583,20 @@ static int check_global_subprog_return_code(struct bpf_verifier_env *env)
{
struct bpf_func_state *cur_frame = cur_func(env);
u32 subprog = cur_frame->subprogno;
+ u32 i, nregs;
+ int err;
if (subprog_returns_void(env, subprog))
return 0;
- return check_global_ret_scalar_reg(env, BPF_REG_0);
+ nregs = bpf_ret_reg_pair(env, subprog) ? 2 : 1;
+ for (i = 0; i < nregs; i++) {
+ err = check_global_ret_scalar_reg(env, ret_regs[i]);
+ if (err)
+ return err;
+ }
+
+ return 0;
}
/* Bitmask with 1s for all caller saved registers */
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 06/10] bpf: Add verifier support for 16-byte returns in R0:R2
2026-08-17 4:22 ` [PATCH bpf-next v6 06/10] bpf: Add verifier support for 16-byte returns in R0:R2 Yonghong Song
@ 2026-08-17 4:42 ` sashiko-bot
2026-08-17 15:46 ` Yonghong Song
2026-08-17 5:17 ` bot+bpf-ci
1 sibling, 1 reply; 31+ messages in thread
From: sashiko-bot @ 2026-08-17 4:42 UTC (permalink / raw)
To: Yonghong Song; +Cc: bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The newly added support for kfuncs returning structs by value is mathematically unreachable.
--
--- Patch [6]: [PATCH bpf-next v6 06/10] bpf: Add verifier support for 16-byte returns in R0:R2 ---
commit 438f13b526c411cf77d7536a3436a0cc4625dbe4
Author: Yonghong Song <yonghong.song@linux.dev>
bpf: Add verifier support for 16-byte returns in R0:R2
This patch adds BPF verifier support for handling 16-byte returns in the
R0:R2 register pair, utilizing LLVM 23 capabilities. It models this
register pair at calls to global/static subprograms and kfuncs, allowing
kfuncs to return structs composed of scalars.
Link: https://github.com/llvm/llvm-project/pull/190894
Link: https://github.com/llvm/llvm-project/pull/206876
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -13967,10 +13997,25 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
> }
>
> if (btf_type_is_scalar(t)) {
> - mark_reg_unknown(env, regs, BPF_REG_0);
> + mark_kfunc_ret_regs(env, regs, t->size);
> if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
> meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
> __mark_reg_const_zero(env, ®s[BPF_REG_0]);
> + } else if (btf_type_is_struct(t)) {
[Severity: Medium]
Is this newly added code for kfunc struct returns actually reachable
during program load?
Looking at how kfuncs are processed earlier in the verifier,
bpf_add_kfunc_call() calls btf_distill_func_proto(), which appears to
still explicitly reject struct return types:
kernel/bpf/btf.c:btf_distill_func_proto() {
...
if (ret < 0 || btf_type_is_struct(t)) {
bpf_log(log,
"The function %s return type %s is unsupported.\n",
tname, btf_type_str(t));
return -EINVAL;
}
...
}
Since btf_distill_func_proto() is called during program load before
check_kfunc_call() is ever reached, won't any BPF program calling a
struct-returning kfunc be unconditionally rejected, making this new
branch dead code?
> + /*
> + * The returned struct comes back as raw register bits modeled
> + * as an unknown scalar, so it must contain only scalars:
> + * otherwise a pointer field would be laundered into a scalar
> + * and escape provenance and reference tracking.
> + */
> + if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
> + verbose(env,
> + "kernel function %s returns %s %s that is not composed of scalars\n",
> + func_name, btf_type_str(t),
> + btf_name_by_offset(desc_btf, t->name_off));
> + return -EINVAL;
> + }
> + mark_kfunc_ret_regs(env, regs, t->size);
> } else if (btf_type_is_ptr(t)) {
> ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260817042141.2286086-1-yonghong.song@linux.dev?part=6
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 06/10] bpf: Add verifier support for 16-byte returns in R0:R2
2026-08-17 4:42 ` sashiko-bot
@ 2026-08-17 15:46 ` Yonghong Song
0 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 15:46 UTC (permalink / raw)
To: sashiko-reviews; +Cc: bpf
On 8/16/26 9:42 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Medium] The newly added support for kfuncs returning structs by value is mathematically unreachable.
> --
>
> --- Patch [6]: [PATCH bpf-next v6 06/10] bpf: Add verifier support for 16-byte returns in R0:R2 ---
> commit 438f13b526c411cf77d7536a3436a0cc4625dbe4
> Author: Yonghong Song <yonghong.song@linux.dev>
>
> bpf: Add verifier support for 16-byte returns in R0:R2
>
> This patch adds BPF verifier support for handling 16-byte returns in the
> R0:R2 register pair, utilizing LLVM 23 capabilities. It models this
> register pair at calls to global/static subprograms and kfuncs, allowing
> kfuncs to return structs composed of scalars.
>
> Link: https://github.com/llvm/llvm-project/pull/190894
> Link: https://github.com/llvm/llvm-project/pull/206876
>
>> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
>> --- a/kernel/bpf/verifier.c
>> +++ b/kernel/bpf/verifier.c
> [ ... ]
>
>> @@ -13967,10 +13997,25 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
>> }
>>
>> if (btf_type_is_scalar(t)) {
>> - mark_reg_unknown(env, regs, BPF_REG_0);
>> + mark_kfunc_ret_regs(env, regs, t->size);
>> if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
>> meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
>> __mark_reg_const_zero(env, ®s[BPF_REG_0]);
>> + } else if (btf_type_is_struct(t)) {
> [Severity: Medium]
> Is this newly added code for kfunc struct returns actually reachable
> during program load?
>
> Looking at how kfuncs are processed earlier in the verifier,
> bpf_add_kfunc_call() calls btf_distill_func_proto(), which appears to
> still explicitly reject struct return types:
>
> kernel/bpf/btf.c:btf_distill_func_proto() {
> ...
> if (ret < 0 || btf_type_is_struct(t)) {
> bpf_log(log,
> "The function %s return type %s is unsupported.\n",
> tname, btf_type_str(t));
> return -EINVAL;
> }
> ...
> }
>
> Since btf_distill_func_proto() is called during program load before
> check_kfunc_call() is ever reached, won't any BPF program calling a
> struct-returning kfunc be unconditionally rejected, making this new
> branch dead code?
The next patch "bpf: Enable aggregate return types up to 16 bytes"
will enable to have successful prog load.
>
>> + /*
>> + * The returned struct comes back as raw register bits modeled
>> + * as an unknown scalar, so it must contain only scalars:
>> + * otherwise a pointer field would be laundered into a scalar
>> + * and escape provenance and reference tracking.
>> + */
>> + if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
>> + verbose(env,
>> + "kernel function %s returns %s %s that is not composed of scalars\n",
>> + func_name, btf_type_str(t),
>> + btf_name_by_offset(desc_btf, t->name_off));
>> + return -EINVAL;
>> + }
>> + mark_kfunc_ret_regs(env, regs, t->size);
>> } else if (btf_type_is_ptr(t)) {
>> ptr_type = btf_type_skip_modifiers(desc_btf, t->type, &ptr_type_id);
> [ ... ]
>
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH bpf-next v6 06/10] bpf: Add verifier support for 16-byte returns in R0:R2
2026-08-17 4:22 ` [PATCH bpf-next v6 06/10] bpf: Add verifier support for 16-byte returns in R0:R2 Yonghong Song
2026-08-17 4:42 ` sashiko-bot
@ 2026-08-17 5:17 ` bot+bpf-ci
2026-08-17 16:02 ` Yonghong Song
1 sibling, 1 reply; 31+ messages in thread
From: bot+bpf-ci @ 2026-08-17 5:17 UTC (permalink / raw)
To: yonghong.song, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]
> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> index 5b9d767895c9..2ae7cb9b30f2 100644
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c
> @@ -7684,6 +7684,12 @@ static int btf_check_func_type_match(struct bpf_verifier_log *log,
> btf_type_str(t2), fn2);
> return -EINVAL;
> }
> + if (btf_type_has_size(t1) && (t1->size > 8 || t2->size > 8)) {
> + bpf_log(log,
> + "Return type of %s() has size %u while %s() has size %u, and a size above 8 bytes cannot be replaced\n",
> + fn1, t1->size, fn2, t2->size);
> + return -EINVAL;
> + }
This isn't a bug, but since this check fires on the absolute size rather
than on a mismatch, the message can end up reading "has size 16 while g()
has size 16". Would it be clearer to state which side exceeds 8 bytes
instead of using the "X while Y" mismatch phrasing the neighbouring
diagnostics use?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31995069373
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 06/10] bpf: Add verifier support for 16-byte returns in R0:R2
2026-08-17 5:17 ` bot+bpf-ci
@ 2026-08-17 16:02 ` Yonghong Song
0 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 16:02 UTC (permalink / raw)
To: bot+bpf-ci, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, martin.lau, clm,
ihor.solodrai
On 8/16/26 10:17 PM, bot+bpf-ci@kernel.org wrote:
>> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
>> index 5b9d767895c9..2ae7cb9b30f2 100644
>> --- a/kernel/bpf/btf.c
>> +++ b/kernel/bpf/btf.c
>> @@ -7684,6 +7684,12 @@ static int btf_check_func_type_match(struct bpf_verifier_log *log,
>> btf_type_str(t2), fn2);
>> return -EINVAL;
>> }
>> + if (btf_type_has_size(t1) && (t1->size > 8 || t2->size > 8)) {
>> + bpf_log(log,
>> + "Return type of %s() has size %u while %s() has size %u, and a size above 8 bytes cannot be replaced\n",
>> + fn1, t1->size, fn2, t2->size);
>> + return -EINVAL;
>> + }
> This isn't a bug, but since this check fires on the absolute size rather
> than on a mismatch, the message can end up reading "has size 16 while g()
> has size 16". Would it be clearer to state which side exceeds 8 bytes
> instead of using the "X while Y" mismatch phrasing the neighbouring
> diagnostics use?
Okay, I see. Maybe
"Return type of %s() has size %u and of %s() size %u, and a size above 8 bytes cannot be replaced\n"
?
I guess the intention is to explicitly mention which function exceeds 8 bytes.
User should be able to know this by checking original func and freplace prog?
>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31995069373
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH bpf-next v6 07/10] bpf: Enable aggregate return types up to 16 bytes
2026-08-17 4:21 [PATCH bpf-next v6 00/10] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (5 preceding siblings ...)
2026-08-17 4:22 ` [PATCH bpf-next v6 06/10] bpf: Add verifier support for 16-byte returns in R0:R2 Yonghong Song
@ 2026-08-17 4:22 ` Yonghong Song
2026-08-17 4:42 ` sashiko-bot
2026-08-17 4:22 ` [PATCH bpf-next v6 08/10] selftests/bpf: Add C tests for 16-byte returns in R0:R2 Yonghong Song
` (2 subsequent siblings)
9 siblings, 1 reply; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 4:22 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
Relax btf_distill_func_proto() to accept a by-value struct or union that
the R0:R2 convention added in earlier patches can carry:
- a struct or union larger than 8 and up to 16 bytes, returned in the
R0:R2 register pair, matching what LLVM emits for the BPF target;
- a struct or union up to 8 bytes, returned in R0 alone.
A >8 byte scalar (__int128) was already accepted and is unchanged.
Everything else stays rejected: a return type larger than 16 bytes, and any
type that __get_type_size() cannot return in registers at all (e.g. an
array), which it already reports as ret < 0.
btf_validate_return_type() is relaxed as well, so that it accepts a
by-value struct or union up to 16 bytes in addition to void and scalars.
With btf_distill_func_proto() and btf_validate_return_type() relaxed, the
verifier, JIT, precision-backtracking and live-register support from the
earlier patches becomes reachable: <=16 byte aggregate return values now
work end to end.
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
include/linux/bpf_verifier.h | 2 ++
kernel/bpf/btf.c | 23 +++++++++++++++----
kernel/bpf/verifier.c | 18 +++++++--------
.../selftests/bpf/progs/exceptions_fail.c | 2 +-
4 files changed, 30 insertions(+), 15 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index f70d5878fbff..938c9a9eb9d2 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1467,6 +1467,8 @@ int bpf_jmp_offset(struct bpf_insn *insn);
struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask);
bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env, const struct btf *btf,
+ const struct btf_type *t, int rec);
int bpf_find_subprog(struct bpf_verifier_env *env, int off);
bool bpf_is_throw_kfunc(struct bpf_insn *insn);
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 2ae7cb9b30f2..ba83fa1d52a3 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -7591,7 +7591,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
return -EINVAL;
}
ret = __get_type_size(btf, func->type, &t);
- if (ret < 0 || btf_type_is_struct(t)) {
+ if (ret < 0 || ret > 16) {
bpf_log(log,
"The function %s return type %s is unsupported.\n",
tname, btf_type_str(t));
@@ -7970,7 +7970,7 @@ static int btf_scan_type_tags(struct bpf_verifier_env *env,
/* Check whether the type is a valid return type. */
static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf,
- const struct btf_type *t, int subprog)
+ const struct btf_type *t, int subprog, bool is_global)
{
u32 tags = 0;
int err;
@@ -7993,6 +7993,19 @@ static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *bt
if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t))
return 0;
+ if (btf_type_is_struct(t) && t->size <= 16) {
+ /*
+ * A global function's caller models the return as an opaque
+ * scalar pair, so it may only return scalars by value. A local
+ * function is verified inline, so a pointer field stays tracked
+ * and needs no such restriction.
+ */
+ bool local_func = subprog && !is_global;
+
+ if (local_func || btf_type_is_scalar_struct(env, btf, t, 0))
+ return 0;
+ }
+
return -EOPNOTSUPP;
}
@@ -8080,12 +8093,12 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
return -EINVAL;
}
- err = btf_validate_return_type(env, btf, t, subprog);
+ err = btf_validate_return_type(env, btf, t, subprog, is_global);
if (err) {
if (is_global) {
bpf_log(log,
- "Global function %s() return value not void or scalar. "
- "Only those are supported.\n",
+ "Global function %s() has unsupported return type. "
+ "Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.\n",
tname);
}
return err;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e371b7e27ec9..e5c8d8f7d474 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11600,9 +11600,9 @@ static bool is_kfunc_arg_implicit(const struct bpf_call_arg_meta *meta, u32 arg_
}
/* Returns true if struct is composed of scalars, 4 levels of nesting allowed */
-static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
- const struct btf *btf,
- const struct btf_type *t, int rec)
+bool btf_type_is_scalar_struct(struct bpf_verifier_env *env,
+ const struct btf *btf,
+ const struct btf_type *t, int rec)
{
const struct btf_type *member_type;
const struct btf_member *member;
@@ -11620,7 +11620,7 @@ static bool __btf_type_is_scalar_struct(struct bpf_verifier_env *env,
verbose(env, "max struct nesting depth exceeded\n");
return false;
}
- if (!__btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
+ if (!btf_type_is_scalar_struct(env, btf, member_type, rec + 1))
return false;
continue;
}
@@ -12019,7 +12019,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
(is_kfunc_arg_mem_size(meta->btf, &args[arg + 1]) ||
is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1]))) {
if (!btf_type_is_void(ref_t) && !btf_type_is_scalar(ref_t) &&
- !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+ !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
verbose(env, "%s pointer type %s %s must point to void, scalar, or struct with scalar\n",
reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
return -EINVAL;
@@ -12035,7 +12035,7 @@ get_kfunc_arg_type(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
* scalars. The access size is derived from the pointed-to BTF type.
*/
if (!btf_type_is_scalar(ref_t) &&
- !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+ !btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
verbose(env, "%s pointer type %s %s must point to scalar, or struct with scalar\n",
reg_arg_name(env, argno), btf_type_str(ref_t), ref_tname);
return -EINVAL;
@@ -13091,7 +13091,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me
break;
}
- if (!__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
+ if (!btf_type_is_scalar_struct(env, meta->btf, ref_t, 0)) {
enum bpf_reg_type reg2btf_type = lookup_reg2btf_ids(ref_id);
const char *expected_type;
@@ -13629,7 +13629,7 @@ static int check_special_kfunc(struct bpf_verifier_env *env, struct bpf_call_arg
struct_meta = btf_find_struct_meta(ret_btf, ret_btf_id);
if (is_bpf_percpu_obj_new_kfunc(meta->func_id)) {
- if (!__btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
+ if (!btf_type_is_scalar_struct(env, ret_btf, ret_t, 0)) {
verbose(env, "bpf_percpu_obj_new type ID argument must be of a struct of scalars\n");
return -EINVAL;
}
@@ -14008,7 +14008,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
* otherwise a pointer field would be laundered into a scalar
* and escape provenance and reference tracking.
*/
- if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
+ if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
verbose(env,
"kernel function %s returns %s %s that is not composed of scalars\n",
func_name, btf_type_str(t),
diff --git a/tools/testing/selftests/bpf/progs/exceptions_fail.c b/tools/testing/selftests/bpf/progs/exceptions_fail.c
index ac44d60e5066..9708efb93683 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_fail.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_fail.c
@@ -60,7 +60,7 @@ __noinline int exception_cb_ok_arg_small(int a)
SEC("?tc")
__exception_cb(exception_cb_bad_ret_type1)
-__failure __msg("Global function exception_cb_bad_ret_type1() return value not void or scalar.")
+__failure __msg("Only void, scalar, or a scalar-only struct/union up to 16 bytes is supported.")
int reject_exception_cb_type_1(struct __sk_buff *ctx)
{
bpf_throw(0);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 07/10] bpf: Enable aggregate return types up to 16 bytes
2026-08-17 4:22 ` [PATCH bpf-next v6 07/10] bpf: Enable aggregate return types up to 16 bytes Yonghong Song
@ 2026-08-17 4:42 ` sashiko-bot
2026-08-17 16:12 ` Yonghong Song
0 siblings, 1 reply; 31+ messages in thread
From: sashiko-bot @ 2026-08-17 4:42 UTC (permalink / raw)
To: Yonghong Song; +Cc: bpf
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] Attaching a BPF trampoline to a function returning a 16-byte struct silently corrupts the return value because the trampoline fails to preserve the second return register.
Pre-existing issues:
- [Medium] Kfuncs returning structs with padding or sizes that are not a multiple of 8 may leak uninitialized kernel memory to BPF programs.
--
--- Patch [7]: [PATCH bpf-next v6 07/10] bpf: Enable aggregate return types up to 16 bytes ---
commit 5c9b0fdbdfee0f07315dc190bc0c039af9bce8bd
Author: Yonghong Song <yonghong.song@linux.dev>
bpf: Enable aggregate return types up to 16 bytes
This commit relaxes the verifier and BTF distillation to accept by-value
struct or union return types up to 16 bytes. These are returned in the
R0:R2 register pair.
> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
> --- a/kernel/bpf/btf.c
> +++ b/kernel/bpf/btf.c
> @@ -7591,7 +7591,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
> return -EINVAL;
> }
> ret = __get_type_size(btf, func->type, &t);
> - if (ret < 0 || btf_type_is_struct(t)) {
> + if (ret < 0 || ret > 16) {
[Severity: High]
By relaxing this check to allow up to 16-byte returns, does this silently
break fentry and freplace programs attached to such functions?
While bpf_check_attach_target() rejects programs if they explicitly use the
trampoline retval:
bpf_check_attach_target() {
if (tgt_info->fmodel.ret_size > 8 &&
attach_uses_trampoline_retval(prog->expected_attach_type)) {
...
}
}
It appears to still allow fentry and freplace programs to attach. When a
BPF program attaches to a kernel function returning a 16-byte struct, the
target function returns its value in RAX and RDX (on x86-64).
The BPF trampoline epilogue calls __bpf_tramp_exit(), which clobbers
caller-saved registers like RDX.
If the trampoline only restores RAX from the stack before returning, won't
RDX be left with garbage, corrupting the upper 8 bytes of the original
function's return value?
> bpf_log(log,
> "The function %s return type %s is unsupported.\n",
> tname, btf_type_str(t));
> [ ... ]
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> [ ... ]
> @@ -14008,7 +14008,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
> * otherwise a pointer field would be laundered into a scalar
> * and escape provenance and reference tracking.
> */
> - if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
> + if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
[Severity: Medium]
This isn't a bug introduced by this patch, but since the scope of by-value
returns is expanded from 8 bytes to 16 bytes, it might be worth verifying.
In check_kfunc_call(), the verifier immediately calls mark_kfunc_ret_regs()
after this check:
check_kfunc_call() {
if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
...
return -EINVAL;
}
mark_kfunc_ret_regs(env, regs, t->size);
}
If a kfunc returns a struct containing padding bytes, the C compiler might
leave those padding bytes uninitialized. Since the verifier marks the entire
return registers as fully initialized scalars, could this allow a BPF
program to read uninitialized kernel memory via the padding bytes?
> verbose(env,
> "kernel function %s returns %s %s that is not composed of scalars\n",
> func_name, btf_type_str(t),
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260817042141.2286086-1-yonghong.song@linux.dev?part=7
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 07/10] bpf: Enable aggregate return types up to 16 bytes
2026-08-17 4:42 ` sashiko-bot
@ 2026-08-17 16:12 ` Yonghong Song
0 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 16:12 UTC (permalink / raw)
To: sashiko-reviews; +Cc: bpf
On 8/16/26 9:42 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
>
> New issues:
> - [High] Attaching a BPF trampoline to a function returning a 16-byte struct silently corrupts the return value because the trampoline fails to preserve the second return register.
>
> Pre-existing issues:
> - [Medium] Kfuncs returning structs with padding or sizes that are not a multiple of 8 may leak uninitialized kernel memory to BPF programs.
> --
>
> --- Patch [7]: [PATCH bpf-next v6 07/10] bpf: Enable aggregate return types up to 16 bytes ---
> commit 5c9b0fdbdfee0f07315dc190bc0c039af9bce8bd
> Author: Yonghong Song <yonghong.song@linux.dev>
>
> bpf: Enable aggregate return types up to 16 bytes
>
> This commit relaxes the verifier and BTF distillation to accept by-value
> struct or union return types up to 16 bytes. These are returned in the
> R0:R2 register pair.
>
>> diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
>> --- a/kernel/bpf/btf.c
>> +++ b/kernel/bpf/btf.c
>> @@ -7591,7 +7591,7 @@ int btf_distill_func_proto(struct bpf_verifier_log *log,
>> return -EINVAL;
>> }
>> ret = __get_type_size(btf, func->type, &t);
>> - if (ret < 0 || btf_type_is_struct(t)) {
>> + if (ret < 0 || ret > 16) {
> [Severity: High]
> By relaxing this check to allow up to 16-byte returns, does this silently
> break fentry and freplace programs attached to such functions?
>
> While bpf_check_attach_target() rejects programs if they explicitly use the
> trampoline retval:
>
> bpf_check_attach_target() {
> if (tgt_info->fmodel.ret_size > 8 &&
> attach_uses_trampoline_retval(prog->expected_attach_type)) {
> ...
> }
> }
>
> It appears to still allow fentry and freplace programs to attach. When a
> BPF program attaches to a kernel function returning a 16-byte struct, the
> target function returns its value in RAX and RDX (on x86-64).
>
> The BPF trampoline epilogue calls __bpf_tramp_exit(), which clobbers
> caller-saved registers like RDX.
>
> If the trampoline only restores RAX from the stack before returning, won't
> RDX be left with garbage, corrupting the upper 8 bytes of the original
> function's return value?
For fentry, we should be okay as there are no return values.
For freplace, in patch "bpf: Add verifier support for 16-byte returns in R0:R2",
the return value cannot be more than 8 bytes.
So we should be okay here.
>
>> bpf_log(log,
>> "The function %s return type %s is unsupported.\n",
>> tname, btf_type_str(t));
>> [ ... ]
>> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
>> --- a/kernel/bpf/verifier.c
>> +++ b/kernel/bpf/verifier.c
>> [ ... ]
>> @@ -14008,7 +14008,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
>> * otherwise a pointer field would be laundered into a scalar
>> * and escape provenance and reference tracking.
>> */
>> - if (!__btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
>> + if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
> [Severity: Medium]
> This isn't a bug introduced by this patch, but since the scope of by-value
> returns is expanded from 8 bytes to 16 bytes, it might be worth verifying.
>
> In check_kfunc_call(), the verifier immediately calls mark_kfunc_ret_regs()
> after this check:
>
> check_kfunc_call() {
> if (!btf_type_is_scalar_struct(env, desc_btf, t, 0)) {
> ...
> return -EINVAL;
> }
> mark_kfunc_ret_regs(env, regs, t->size);
> }
>
> If a kfunc returns a struct containing padding bytes, the C compiler might
> leave those padding bytes uninitialized. Since the verifier marks the entire
> return registers as fully initialized scalars, could this allow a BPF
> program to read uninitialized kernel memory via the padding bytes?
This should not be a problem. For kfunc, the return value (if size is less
than 8), the compiler will do proper narrowing.
>
>> verbose(env,
>> "kernel function %s returns %s %s that is not composed of scalars\n",
>> func_name, btf_type_str(t),
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH bpf-next v6 08/10] selftests/bpf: Add C tests for 16-byte returns in R0:R2
2026-08-17 4:21 [PATCH bpf-next v6 00/10] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (6 preceding siblings ...)
2026-08-17 4:22 ` [PATCH bpf-next v6 07/10] bpf: Enable aggregate return types up to 16 bytes Yonghong Song
@ 2026-08-17 4:22 ` Yonghong Song
2026-08-17 4:45 ` sashiko-bot
2026-08-17 4:22 ` [PATCH bpf-next v6 09/10] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns Yonghong Song
2026-08-17 4:22 ` [PATCH bpf-next v6 10/10] Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2 Yonghong Song
9 siblings, 1 reply; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 4:22 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
Add selftests that exercise a 16-byte return value passed in the R0:R2
register pair, written in C so that they depend on the compiler lowering
the register-pair return.
The R0:R2 convention is only emitted by LLVM 23 and newer, and a by-value
aggregate return does not compile at all before that, so the programs sit
behind a __clang_major__ guard. An older compiler builds the dummy test in
the #else branch instead, which keeps the object non-empty and says in its
description why nothing was exercised.
The kfunc tests are tagged __arch_x86_64/__arch_arm64 and skip elsewhere.
Those are the architectures whose JIT advertises
bpf_jit_supports_kfunc_ret_reg_pair(), which bpf_add_kfunc_call() requires
before it accepts a kfunc returning more than 8 bytes, and they are also
the only ones building the kfuncs.
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../bpf/progs/verifier_aggregate_ret.c | 178 ++++++++++++++++++
.../selftests/bpf/test_kmods/bpf_testmod.c | 18 ++
.../bpf/test_kmods/bpf_testmod_kfunc.h | 9 +
4 files changed, 207 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 64ac49ad67e6..f7f94ccebce2 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -5,6 +5,7 @@
#include "arena_kfunc.skel.h"
#include "arena_kfunc_jit.skel.h"
#include "cap_helpers.h"
+#include "verifier_aggregate_ret.skel.h"
#include "verifier_align.skel.h"
#include "verifier_and.skel.h"
#include "verifier_arena.skel.h"
@@ -170,6 +171,7 @@ void test_arena_kfunc(void) { RUN_TESTS(arena_kfunc); }
void test_arena_kfunc_jit(void) { RUN_TESTS(arena_kfunc_jit); }
+void test_verifier_aggregate_ret(void) { RUN_TESTS(verifier_aggregate_ret); }
void test_verifier_align(void) { RUN(verifier_align); }
void test_verifier_and(void) { RUN(verifier_and); }
void test_verifier_arena(void) { RUN(verifier_arena); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c
new file mode 100644
index 000000000000..7851bade2b40
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_aggregate_ret.c
@@ -0,0 +1,178 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "../test_kmods/bpf_testmod_kfunc.h"
+#include "bpf_misc.h"
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+#define MIX_A 0xdeadbeefcafef00dULL
+#define MIX_B 0x0123456789abcdefULL
+
+typedef unsigned __int128 u128;
+
+struct pair {
+ __u64 lo; /* R0 */
+ __u64 hi; /* R2 */
+};
+
+union upair {
+ __u64 halves[2];
+ struct {
+ __u64 lo; /* R0 */
+ __u64 hi; /* R2 */
+ } parts;
+};
+
+static __noinline u128 make_i128(__u64 a, __u64 b)
+{
+ return ((u128)(a + b) << 64) | (a - b);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_int128_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ u128 v;
+
+ v = make_i128(a, b);
+ if ((__u64)(v >> 64) != a + b)
+ return 1;
+ if ((__u64)v != a - b)
+ return 2;
+
+ return 0;
+}
+
+static __noinline struct pair make_pair(__u64 a, __u64 b)
+{
+ struct pair p = { .lo = a + b, .hi = a - b };
+
+ return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_struct_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ struct pair p;
+
+ p = make_pair(a, b);
+ if (p.lo != a + b)
+ return 1;
+ if (p.hi != a - b)
+ return 2;
+
+ return 0;
+}
+
+__noinline struct pair make_pair_global(__u64 a, __u64 b)
+{
+ struct pair p = { .lo = a + b, .hi = a - b };
+
+ return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_global_struct_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ struct pair p;
+
+ p = make_pair_global(a, b);
+ if (p.lo != a + b)
+ return 1;
+ if (p.hi != a - b)
+ return 2;
+
+ return 0;
+}
+
+static __noinline union upair make_upair(__u64 a, __u64 b)
+{
+ union upair p;
+
+ p.halves[0] = a + b;
+ p.halves[1] = a - b;
+ return p;
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_union_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ union upair p;
+
+ p = make_upair(a, b);
+ if (p.parts.lo != a + b)
+ return 1;
+ if (p.parts.hi != a - b)
+ return 2;
+
+ return 0;
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_int128_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ u128 v;
+
+ v = bpf_kfunc_call_test_i128(a, b);
+ if ((__u64)(v >> 64) != a + b)
+ return 1;
+ if ((__u64)v != a - b)
+ return 2;
+
+ return 0;
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+int aggregate_ret_kfunc_struct_c_test(struct __sk_buff *skb)
+{
+ __u64 a = skb->len ^ MIX_A;
+ __u64 b = skb->len ^ MIX_B;
+ struct prog_test_ret_pair p;
+
+ p = bpf_kfunc_call_test_ret_pair(a, b);
+ if (p.lo != a + b)
+ return 1;
+ if (p.hi != a - b)
+ return 2;
+
+ return 0;
+}
+
+#else
+
+SEC("socket")
+__description("verifier_aggregate_ret: needs LLVM 23, dummy test")
+__success
+int dummy_test(void)
+{
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index 9366a3c578f1..c4bc8e11c6e1 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -954,6 +954,20 @@ __bpf_kfunc int bpf_kfunc_call_test5(u8 a, u16 b, u32 c)
return 0;
}
+#if defined(__x86_64__) || defined(__aarch64__)
+__bpf_kfunc __int128 bpf_kfunc_call_test_i128(u64 a, u64 b)
+{
+ return (__int128)(((unsigned __int128)(a + b) << 64) | (a - b));
+}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)
+{
+ struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };
+
+ return r;
+}
+#endif /* __x86_64__ || __aarch64__ */
+
__bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
u64 e, u64 f, u64 g, u64 h,
u64 i, u64 j)
@@ -1487,6 +1501,10 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test2)
BTF_ID_FLAGS(func, bpf_kfunc_call_test3)
BTF_ID_FLAGS(func, bpf_kfunc_call_test4)
BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
+#if defined(__x86_64__) || defined(__aarch64__)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)
+#endif
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index 7d81070eefe7..426000f9a14f 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -55,6 +55,11 @@ struct prog_test_big_arg {
__u64 b;
};
+struct prog_test_ret_pair { /* 16 bytes: R0:R2 */
+ __u64 lo;
+ __u64 hi;
+};
+
struct prog_test_fail1 {
void *p;
int x;
@@ -131,6 +136,10 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
+#ifdef __SIZEOF_INT128__
+__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
+#endif
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
__u64 e, __u64 f, __u64 g, __u64 h,
__u64 i, __u64 j) __ksym;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 08/10] selftests/bpf: Add C tests for 16-byte returns in R0:R2
2026-08-17 4:22 ` [PATCH bpf-next v6 08/10] selftests/bpf: Add C tests for 16-byte returns in R0:R2 Yonghong Song
@ 2026-08-17 4:45 ` sashiko-bot
2026-08-17 16:32 ` Yonghong Song
0 siblings, 1 reply; 31+ messages in thread
From: sashiko-bot @ 2026-08-17 4:45 UTC (permalink / raw)
To: Yonghong Song; +Cc: bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The kfuncs `bpf_kfunc_call_test_i128` and `bpf_kfunc_call_test_ret_pair` are conditionally compiled out of `bpf_testmod.ko` on architectures other than x86_64 and arm64, but are unconditionally compiled and declared as strong `__ksym` externs in the selftest object `verifier_aggregate_ret.c`. This will cause libbpf to fail loading the entire object on unsupported architectures (when compiled with LLVM 23+), failing the entire test suite even for programs that don't use the kfuncs.
--
--- Patch [8]: [PATCH bpf-next v6 08/10] selftests/bpf: Add C tests for 16-byte returns in R0:R2 ---
commit 00f68ada7c423dbadb8514b3c5f6c309f446bc99
Author: Yonghong Song <yonghong.song@linux.dev>
selftests/bpf: Add C tests for 16-byte returns in R0:R2
Add selftests that exercise a 16-byte return value passed in the R0:R2
register pair, written in C so that they depend on the compiler lowering
the register-pair return.
The R0:R2 convention is only emitted by LLVM 23 and newer, and a by-value
aggregate return does not compile at all before that, so the programs sit
behind a __clang_major__ guard. An older compiler builds the dummy test in
the #else branch instead, which keeps the object non-empty and says in its
description why nothing was exercised.
The kfunc tests are tagged __arch_x86_64/__arch_arm64 and skip elsewhere.
Those are the architectures whose JIT advertises
bpf_jit_supports_kfunc_ret_reg_pair(), which bpf_add_kfunc_call() requires
before it accepts a kfunc returning more than 8 bytes, and they are also
the only ones building the kfuncs.
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> index 7d81070eefe7f..426000f9a14fa 100644
> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
[ ... ]
> @@ -131,6 +136,10 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
> struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
> long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
> int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
> +#ifdef __SIZEOF_INT128__
> +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
> +#endif
> +struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
[Severity: Medium]
Should these kfuncs be declared with the __weak attribute?
While the test programs in verifier_aggregate_ret.c use the __arch_x86_64 and
__arch_arm64 tags, they still unconditionally reference these kfuncs in the
C code. Without the __weak attribute, Clang emits them as strong externs in
the BPF object file.
When libbpf attempts to load the verifier_aggregate_ret object on unsupported
architectures (when compiled with LLVM 23+), won't it fail to resolve these
strong externs because they are conditionally compiled out of bpf_testmod.ko?
This could cause the entire object load to fail, preventing even the
non-kfunc tests in the same object from running.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260817042141.2286086-1-yonghong.song@linux.dev?part=8
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH bpf-next v6 08/10] selftests/bpf: Add C tests for 16-byte returns in R0:R2
2026-08-17 4:45 ` sashiko-bot
@ 2026-08-17 16:32 ` Yonghong Song
0 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 16:32 UTC (permalink / raw)
To: sashiko-reviews; +Cc: bpf
On 8/16/26 9:45 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Medium] The kfuncs `bpf_kfunc_call_test_i128` and `bpf_kfunc_call_test_ret_pair` are conditionally compiled out of `bpf_testmod.ko` on architectures other than x86_64 and arm64, but are unconditionally compiled and declared as strong `__ksym` externs in the selftest object `verifier_aggregate_ret.c`. This will cause libbpf to fail loading the entire object on unsupported architectures (when compiled with LLVM 23+), failing the entire test suite even for programs that don't use the kfuncs.
> --
>
> --- Patch [8]: [PATCH bpf-next v6 08/10] selftests/bpf: Add C tests for 16-byte returns in R0:R2 ---
> commit 00f68ada7c423dbadb8514b3c5f6c309f446bc99
> Author: Yonghong Song <yonghong.song@linux.dev>
>
> selftests/bpf: Add C tests for 16-byte returns in R0:R2
>
> Add selftests that exercise a 16-byte return value passed in the R0:R2
> register pair, written in C so that they depend on the compiler lowering
> the register-pair return.
>
> The R0:R2 convention is only emitted by LLVM 23 and newer, and a by-value
> aggregate return does not compile at all before that, so the programs sit
> behind a __clang_major__ guard. An older compiler builds the dummy test in
> the #else branch instead, which keeps the object non-empty and says in its
> description why nothing was exercised.
>
> The kfunc tests are tagged __arch_x86_64/__arch_arm64 and skip elsewhere.
> Those are the architectures whose JIT advertises
> bpf_jit_supports_kfunc_ret_reg_pair(), which bpf_add_kfunc_call() requires
> before it accepts a kfunc returning more than 8 bytes, and they are also
> the only ones building the kfuncs.
>
> Acked-by: Eduard Zingerman <eddyz87@gmail.com>
> Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
>
>> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
>> index 7d81070eefe7f..426000f9a14fa 100644
>> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
>> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
> [ ... ]
>> @@ -131,6 +136,10 @@ int bpf_kfunc_call_test2(struct sock *sk, __u32 a, __u32 b) __ksym;
>> struct sock *bpf_kfunc_call_test3(struct sock *sk) __ksym;
>> long bpf_kfunc_call_test4(signed char a, short b, int c, long d) __ksym;
>> int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
>> +#ifdef __SIZEOF_INT128__
>> +__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
>> +#endif
>> +struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
> [Severity: Medium]
> Should these kfuncs be declared with the __weak attribute?
>
> While the test programs in verifier_aggregate_ret.c use the __arch_x86_64 and
> __arch_arm64 tags, they still unconditionally reference these kfuncs in the
> C code. Without the __weak attribute, Clang emits them as strong externs in
> the BPF object file.
>
> When libbpf attempts to load the verifier_aggregate_ret object on unsupported
> architectures (when compiled with LLVM 23+), won't it fail to resolve these
> strong externs because they are conditionally compiled out of bpf_testmod.ko?
> This could cause the entire object load to fail, preventing even the
> non-kfunc tests in the same object from running.
I think we are fine here. We can revisit this if newer architecture (e.g., riscv, s390x)
needs __weak attribute in the future.
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH bpf-next v6 09/10] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
2026-08-17 4:21 [PATCH bpf-next v6 00/10] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (7 preceding siblings ...)
2026-08-17 4:22 ` [PATCH bpf-next v6 08/10] selftests/bpf: Add C tests for 16-byte returns in R0:R2 Yonghong Song
@ 2026-08-17 4:22 ` Yonghong Song
2026-08-17 4:42 ` sashiko-bot
2026-08-17 5:17 ` bot+bpf-ci
2026-08-17 4:22 ` [PATCH bpf-next v6 10/10] Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2 Yonghong Song
9 siblings, 2 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 4:22 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
Add inline-asm tests covering what the C tests cannot reach, since they
need a callee that violates the convention on purpose.
The coverage includes BPF-to-BPF returns, kfunc calls, backtracking and
liveness. In addition, a negative freplace test checks that an extension
cannot replace a function returning R0:R2.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
.../selftests/bpf/prog_tests/aggregate_ret.c | 11 +
.../selftests/bpf/prog_tests/fexit_bpf2bpf.c | 16 ++
.../selftests/bpf/progs/aggregate_ret_func.c | 235 ++++++++++++++++++
.../selftests/bpf/progs/aggregate_ret_kfunc.c | 122 +++++++++
.../bpf/progs/aggregate_ret_target.c | 29 +++
.../bpf/progs/compute_live_registers.c | 30 +++
.../selftests/bpf/progs/freplace_ret_pair.c | 12 +
.../selftests/bpf/test_kmods/bpf_testmod.c | 37 +++
.../bpf/test_kmods/bpf_testmod_kfunc.h | 20 ++
9 files changed, 512 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c
create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_target.c
create mode 100644 tools/testing/selftests/bpf/progs/freplace_ret_pair.c
diff --git a/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
new file mode 100644
index 000000000000..e0b94ed10f94
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <test_progs.h>
+#include "aggregate_ret_func.skel.h"
+#include "aggregate_ret_kfunc.skel.h"
+
+void test_aggregate_ret(void)
+{
+ RUN_TESTS(aggregate_ret_func);
+ RUN_TESTS(aggregate_ret_kfunc);
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
index 2523c07a16c6..6c438df380bb 100644
--- a/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
+++ b/tools/testing/selftests/bpf/prog_tests/fexit_bpf2bpf.c
@@ -441,6 +441,20 @@ static void test_func_replace_int_with_void(void)
" doesn't match type INT of global_func2()");
}
+static void test_func_replace_ret_pair(void)
+{
+ const char *msg = "Return type of new_agg_ret_target_func() has size 8 "
+ "while agg_ret_target_func() has size 16";
+
+ /*
+ * An extension cannot replace a function whose return value comes back
+ * in the R0:R2 pair: only R0 is checked at the extension's exit, so it
+ * would leave R2 stale for the target's callers.
+ */
+ test_obj_load_failure_common("freplace_ret_pair.bpf.o",
+ "./aggregate_ret_target.bpf.o", msg);
+}
+
static int find_prog_btf_id(const char *name, __u32 attach_prog_fd)
{
struct bpf_prog_info info = {};
@@ -660,6 +674,8 @@ void serial_test_fexit_bpf2bpf(void)
test_func_replace_progmap();
if (test__start_subtest("freplace_int_with_void"))
test_func_replace_int_with_void();
+ if (test__start_subtest("freplace_ret_pair"))
+ test_func_replace_ret_pair();
if (test__start_subtest("freplace_void"))
test_func_replace_void();
if (test__start_subtest("sleepable_fentry_to_xdp"))
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
new file mode 100644
index 000000000000..e35baaa10fea
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
@@ -0,0 +1,235 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+typedef unsigned __int128 u128;
+
+__naked u128 global_agg_good(void)
+{
+ asm volatile (
+ "r0 = 0x1234;" /* low 64 bits */
+ "r2 = 0x5678;" /* high 64 bits */
+ "exit;"
+ );
+}
+
+__naked u128 global_agg_bad(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "exit;"
+ );
+}
+
+__naked u128 global_agg_bad_ptr(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r2 = r10;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_global_fail(void)
+{
+ asm volatile (
+ "call %[global_agg_bad];"
+ "r0 = r2;"
+ "exit;"
+ :
+ : __imm(global_agg_bad)
+ : __clobber_all);
+}
+
+SEC("tc")
+__failure __msg("At subprogram exit the register R2 is not a scalar value")
+__naked int aggregate_ret_global_ptr_fail(void)
+{
+ asm volatile (
+ "call %[global_agg_bad_ptr];"
+ "r0 = r2;"
+ "exit;"
+ :
+ : __imm(global_agg_bad_ptr)
+ : __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_bad_ptr(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r2 = r10;" /* stack pointer placed in the second return register */
+ "exit;"
+ );
+}
+
+/*
+ * R2 is a return register once the subprogram returns a pair, so a stack
+ * pointer left in it is rejected at the callee's exit exactly as one in R0
+ * is: the callee frame is gone by the time the caller could use it.
+ */
+SEC("tc")
+__failure __msg("cannot return stack pointer to the caller")
+__naked int aggregate_ret_static_ptr_fail(void)
+{
+ asm volatile (
+ "call %[static_agg_bad_ptr];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(static_agg_bad_ptr)
+ : __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_no_r2(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_static_uninit_fail(void)
+{
+ asm volatile (
+ "call %[static_agg_no_r2];"
+ "r0 = r2;"
+ "exit;"
+ :
+ : __imm(static_agg_no_r2)
+ : __clobber_all);
+}
+
+static __naked __noinline u128 static_agg_precise(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r2 = 4;" /* second half; its value is made precise below */
+ "exit;"
+ );
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
+__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
+__naked int aggregate_ret_static_precise(void)
+{
+ asm volatile (
+ "call %[static_agg_precise];"
+ "r6 = r2;" /* derived from the aggregate's second half */
+ "r6 &= 7;" /* keep it in [0, 7] to index the stack */
+ "r1 = r10;"
+ "r1 += -8;"
+ "r1 += r6;" /* ptr += scalar marks r6 (hence R2) precise */
+ "r0 = 0;"
+ "*(u8 *)(r1 + 0) = r0;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(static_agg_precise)
+ : __clobber_all);
+}
+
+SEC("tc")
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
+__naked int aggregate_ret_global_precise(void)
+{
+ asm volatile (
+ "call %[global_agg_good];"
+ "r6 = r2;" /* derived from the aggregate's second half */
+ "r6 &= 7;" /* keep it in [0, 7] to index the stack */
+ "r1 = r10;"
+ "r1 += -8;"
+ "r1 += r6;" /* ptr += scalar marks r6 (hence R2) precise */
+ "r0 = 0;"
+ "*(u8 *)(r1 + 0) = r0;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(global_agg_good)
+ : __clobber_all);
+}
+
+#if defined(__clang_major__) && __clang_major__ >= 23
+
+/* A by-value struct that smuggles a pointer, which must be rejected. */
+struct with_ptr {
+ void *p;
+ __u64 x;
+};
+
+/* A by-value union that smuggles a pointer, which must be rejected too. */
+union upair_with_ptr {
+ void *p;
+ __u64 halves[2];
+};
+
+__naked struct with_ptr global_ret_struct_ptr(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r2 = 0;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_struct_ptr() has unsupported return type")
+__naked int aggregate_ret_global_struct_ptr_fail(void)
+{
+ asm volatile (
+ "call %[global_ret_struct_ptr];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(global_ret_struct_ptr)
+ : __clobber_all);
+}
+
+__naked union upair_with_ptr global_ret_union_ptr(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "r2 = 0;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__failure __msg("Global function global_ret_union_ptr() has unsupported return type")
+__naked int aggregate_ret_global_union_ptr_fail(void)
+{
+ asm volatile (
+ "call %[global_ret_union_ptr];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(global_ret_union_ptr)
+ : __clobber_all);
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
new file mode 100644
index 000000000000..c23b4beb1773
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
@@ -0,0 +1,122 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../test_kmods/bpf_testmod_kfunc.h"
+
+/*
+ * Reference kfunc addresses to force those BTF to be emitted. Taking the address
+ * (rather than calling) avoids any dependence on the compiler lowering an
+ * __int128 or struct return value, which the BPF backend only supports from
+ * LLVM 23 on.
+ */
+void __kfunc_btf_root(void)
+{
+ asm volatile (""
+ :
+ : "r"(&bpf_kfunc_call_test_i128),
+ "r"(&bpf_kfunc_call_test_ret_fastcall),
+ "r"(&bpf_kfunc_call_test_ret_ptr),
+ "r"(&bpf_kfunc_call_test_ret_ii),
+ "r"(&bpf_kfunc_call_test_ret_big));
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__load_if_JITed()
+__success __retval(0)
+__log_level(2)
+__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
+__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
+__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
+__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
+__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
+__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
+__naked int aggregate_ret_kfunc_precise(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "call %[bpf_kfunc_call_test_i128];"
+ "r6 = r2;" /* second return half */
+ "r6 &= 7;" /* keep it in [0, 7] to index the stack */
+ "r1 = r10;"
+ "r1 += -8;"
+ "r1 += r6;" /* ptr += scalar marks r6 (hence R2) precise */
+ "r0 = 0;"
+ "*(u8 *)(r1 + 0) = r0;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_test_i128)
+ : __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("kfunc bpf_kfunc_call_test_ret_fastcall with >8-byte return is not supported with KF_FASTCALL")
+__naked int aggregate_ret_kfunc_fastcall_fail(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "call %[bpf_kfunc_call_test_ret_fastcall];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_test_ret_fastcall)
+ : __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("is not composed of scalars")
+__naked int aggregate_ret_kfunc_ptr_fail(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "call %[bpf_kfunc_call_test_ret_ptr];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_test_ret_ptr)
+ : __clobber_all);
+}
+
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("R2 !read_ok")
+__naked int aggregate_ret_kfunc_small_no_r2(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "r2 = 0;"
+ "call %[bpf_kfunc_call_test_ret_ii];"
+ "r0 = r2;" /* R2 is not a return register for a <=8 byte struct */
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_test_ret_ii)
+ : __clobber_all);
+}
+
+/*
+ * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by
+ * btf_distill_func_proto(), before the KF_FASTCALL and JIT-capability checks,
+ * so this behaves the same on every architecture.
+ */
+SEC("tc")
+__arch_x86_64 __arch_arm64
+__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported")
+__naked int aggregate_ret_kfunc_too_big_fail(void)
+{
+ asm volatile (
+ "call %[bpf_kfunc_call_test_ret_big];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_test_ret_big)
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_target.c b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
new file mode 100644
index 000000000000..cffd8d7d3241
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_target.c
@@ -0,0 +1,29 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/* freplace target: a global subprogram returning 16 bytes in R0:R2. */
+__naked unsigned __int128 agg_ret_target_func(void)
+{
+ asm volatile (
+ "r0 = 0x1234;"
+ "r2 = 0x5678;"
+ "exit;"
+ );
+}
+
+SEC("tc")
+__naked int agg_ret_target(void)
+{
+ asm volatile (
+ "call %[agg_ret_target_func];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(agg_ret_target_func)
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
index d055fc7b3b95..0be9441ec273 100644
--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
@@ -431,6 +431,36 @@ __naked void subprog1(void)
::: __clobber_all);
}
+static __used __naked unsigned __int128 aux2(void)
+{
+ asm volatile (
+ "r0 = 1;"
+ "r2 = 2;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+SEC("socket")
+/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */
+__load_if_JITed()
+__log_level(2)
+__msg("0: .12345.... (85) call pc+2")
+__msg("1: ..2....... (bf) r0 = r2")
+/* R2 is not read at the exit of this program, which returns an int, ... */
+__msg("2: 0......... (95) exit")
+__msg("3: .......... (b7) r0 = 1")
+__msg("4: 0......... (b7) r2 = 2")
+/* ... but it is at the exit of aux2(), which returns a register pair. */
+__msg("5: 0.2....... (95) exit")
+__naked void subprog_ret_reg_pair(void)
+{
+ asm volatile (
+ "call aux2;"
+ "r0 = r2;"
+ "exit;"
+ ::: __clobber_all);
+}
+
#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
SEC("socket")
diff --git a/tools/testing/selftests/bpf/progs/freplace_ret_pair.c b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
new file mode 100644
index 000000000000..12c15d293bd7
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/freplace_ret_pair.c
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+SEC("freplace/agg_ret_target_func")
+__u64 new_agg_ret_target_func(void)
+{
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
index c4bc8e11c6e1..20a9b9f20e96 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
@@ -966,8 +966,41 @@ __bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(u64 a, u64 b)
return r;
}
+
+__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)
+{
+ struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };
+
+ return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)
+{
+ struct prog_test_ret_ptr r = { .p = NULL, .tag = tag };
+
+ return r;
+}
+
+__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)
+{
+ struct prog_test_ret_ii r = { .a = a, .b = b };
+
+ return r;
+}
#endif /* __x86_64__ || __aarch64__ */
+/*
+ * Takes no argument on purpose: with no arguments there is nothing for the sret
+ * pointer to displace, so this needs no architecture guard even though it
+ * returns 24 bytes. See the comment on bpf_kfunc_call_test_i128() above.
+ */
+__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)
+{
+ struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };
+
+ return r;
+}
+
__bpf_kfunc u64 bpf_kfunc_call_stack_arg(u64 a, u64 b, u64 c, u64 d,
u64 e, u64 f, u64 g, u64 h,
u64 i, u64 j)
@@ -1504,7 +1537,11 @@ BTF_ID_FLAGS(func, bpf_kfunc_call_test5)
#if defined(__x86_64__) || defined(__aarch64__)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_pair)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_fastcall, KF_FASTCALL)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ptr)
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_ii)
#endif
+BTF_ID_FLAGS(func, bpf_kfunc_call_test_ret_big)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_ptr)
BTF_ID_FLAGS(func, bpf_kfunc_call_stack_arg_mix)
diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
index 426000f9a14f..65e693ada736 100644
--- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
+++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod_kfunc.h
@@ -60,6 +60,22 @@ struct prog_test_ret_pair { /* 16 bytes: R0:R2 */
__u64 hi;
};
+struct prog_test_ret_ii { /* 8 bytes: R0 only */
+ int a;
+ int b;
+};
+
+struct prog_test_ret_ptr { /* 16 bytes: contains a pointer */
+ void *p;
+ __u64 tag;
+};
+
+struct prog_test_ret_big { /* 24 bytes: too large for R0:R2 */
+ __u64 a;
+ __u64 b;
+ __u64 c;
+};
+
struct prog_test_fail1 {
void *p;
int x;
@@ -140,6 +156,10 @@ int bpf_kfunc_call_test5(__u8 a, __u16 b, __u32 c) __ksym;
__int128 bpf_kfunc_call_test_i128(__u64 a, __u64 b) __ksym;
#endif
struct prog_test_ret_pair bpf_kfunc_call_test_ret_pair(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(__u64 a, __u64 b) __ksym;
+struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b) __ksym;
+struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(__u64 tag) __ksym;
+struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void) __ksym;
__u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d,
__u64 e, __u64 f, __u64 g, __u64 h,
__u64 i, __u64 j) __ksym;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 09/10] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
2026-08-17 4:22 ` [PATCH bpf-next v6 09/10] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns Yonghong Song
@ 2026-08-17 4:42 ` sashiko-bot
2026-08-17 16:53 ` Yonghong Song
2026-08-17 5:17 ` bot+bpf-ci
1 sibling, 1 reply; 31+ messages in thread
From: sashiko-bot @ 2026-08-17 4:42 UTC (permalink / raw)
To: Yonghong Song; +Cc: bpf
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The test `aggregate_ret_kfunc_too_big_fail` is incorrectly restricted to only run on x86_64 and arm64 architectures.
--
--- Patch [9]: [PATCH bpf-next v6 09/10] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns ---
commit b44fbeb59afa99f6be0f33037220d124e0fb4e12
Author: Yonghong Song <yonghong.song@linux.dev>
selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
This commit adds BPF inline-assembly tests to cover BPF-to-BPF returns, kfunc
calls, backtracking, and liveness for functions returning values in the R0:R2
register pair. It also includes an extension test to ensure a function returning
R0:R2 cannot be replaced.
> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
> new file mode 100644
> index 0000000000000..c23b4beb1773c
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
[ ... ]
> +/*
> + * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by
> + * btf_distill_func_proto(), before the KF_FASTCALL and JIT-capability checks,
> + * so this behaves the same on every architecture.
> + */
> +SEC("tc")
> +__arch_x86_64 __arch_arm64
[Severity: Medium]
Are these architecture restriction tags intended here?
The comment right above mentions that this test behaves the same on every
architecture, but the __arch_x86_64 and __arch_arm64 tags artificially limit
aggregate_ret_kfunc_too_big_fail() to only run on those two architectures,
which prevents the test from running on others.
> +__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported")
> +__naked int aggregate_ret_kfunc_too_big_fail(void)
> +{
> + asm volatile (
> + "call %[bpf_kfunc_call_test_ret_big];"
> + "r0 = 0;"
> + "exit;"
> + :
> + : __imm(bpf_kfunc_call_test_ret_big)
> + : __clobber_all);
> +}
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260817042141.2286086-1-yonghong.song@linux.dev?part=9
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 09/10] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
2026-08-17 4:42 ` sashiko-bot
@ 2026-08-17 16:53 ` Yonghong Song
0 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 16:53 UTC (permalink / raw)
To: sashiko-reviews; +Cc: bpf
On 8/16/26 9:42 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Medium] The test `aggregate_ret_kfunc_too_big_fail` is incorrectly restricted to only run on x86_64 and arm64 architectures.
> --
>
> --- Patch [9]: [PATCH bpf-next v6 09/10] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns ---
> commit b44fbeb59afa99f6be0f33037220d124e0fb4e12
> Author: Yonghong Song <yonghong.song@linux.dev>
>
> selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
>
> This commit adds BPF inline-assembly tests to cover BPF-to-BPF returns, kfunc
> calls, backtracking, and liveness for functions returning values in the R0:R2
> register pair. It also includes an extension test to ensure a function returning
> R0:R2 cannot be replaced.
>
>> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
>> new file mode 100644
>> index 0000000000000..c23b4beb1773c
>> --- /dev/null
>> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
> [ ... ]
>
>> +/*
>> + * A return value larger than 16 bytes does not fit in R0:R2 and is rejected by
>> + * btf_distill_func_proto(), before the KF_FASTCALL and JIT-capability checks,
>> + * so this behaves the same on every architecture.
>> + */
>> +SEC("tc")
>> +__arch_x86_64 __arch_arm64
> [Severity: Medium]
> Are these architecture restriction tags intended here?
>
> The comment right above mentions that this test behaves the same on every
> architecture, but the __arch_x86_64 and __arch_arm64 tags artificially limit
> aggregate_ret_kfunc_too_big_fail() to only run on those two architectures,
> which prevents the test from running on others.
Okay, I don't have 100% confidence about every architecture. So let us
change to
/*
* A return value larger than 16 bytes does not fit in R0:R2 and is rejected by
* btf_distill_func_proto() before the KF_FASTCALL and JIT-capability checks.
*/
>
>> +__failure __msg("The function bpf_kfunc_call_test_ret_big return type STRUCT is unsupported")
>> +__naked int aggregate_ret_kfunc_too_big_fail(void)
>> +{
>> + asm volatile (
>> + "call %[bpf_kfunc_call_test_ret_big];"
>> + "r0 = 0;"
>> + "exit;"
>> + :
>> + : __imm(bpf_kfunc_call_test_ret_big)
>> + : __clobber_all);
>> +}
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: [PATCH bpf-next v6 09/10] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
2026-08-17 4:22 ` [PATCH bpf-next v6 09/10] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns Yonghong Song
2026-08-17 4:42 ` sashiko-bot
@ 2026-08-17 5:17 ` bot+bpf-ci
2026-08-17 17:12 ` Yonghong Song
1 sibling, 1 reply; 31+ messages in thread
From: bot+bpf-ci @ 2026-08-17 5:17 UTC (permalink / raw)
To: yonghong.song, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, ast, andrii, daniel,
martin.lau, eddyz87, yonghong.song, clm, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 7982 bytes --]
> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
> new file mode 100644
> index 000000000000..e35baaa10fea
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
[ ... ]
> +SEC("tc")
> +__failure __msg("At subprogram exit the register R2 is not a scalar value")
> +__naked int aggregate_ret_global_ptr_fail(void)
> +{
> + asm volatile (
> + "call %[global_agg_bad_ptr];"
> + "r0 = r2;"
> + "exit;"
> + :
> + : __imm(global_agg_bad_ptr)
> + : __clobber_all);
> +}
Does this test need __load_if_JITed()? Without it, the test runs on
kernels where the JIT is disabled. When env->prog->jit_requested is
false, bpf_compute_subprog_ret_regs() returns early without marking any
ret_reg_pair flag, so global_agg_bad_ptr is not treated as returning a
register pair.
Main verification happens first, and with nregs = 1 at the call site,
check_func_call() marks only R0 (the callee's R2 is never propagated to
the caller). The next instruction "r0 = r2" then reads the caller's
NOT_INIT R2 from clear_caller_saved_regs() and dies in
__check_reg_arg() with "R2 !read_ok", never reaching
do_check_subprogs() where the asserted message "At subprogram exit the
register R2 is not a scalar value" would appear.
The program still fails to load, but with a different message than the
test asserts. aggregate_ret_static_precise and
aggregate_ret_global_precise both carry __load_if_JITed() to ensure the
R0:R2 convention is active when they run. Would the same gate keep this
test meaningful?
> +/*
> + * R2 is a return register once the subprogram returns a pair, so a stack
> + * pointer left in it is rejected at the callee's exit exactly as one in R0
> + * is: the callee frame is gone by the time the caller could use it.
> + */
> +SEC("tc")
> +__failure __msg("cannot return stack pointer to the caller")
> +__naked int aggregate_ret_static_ptr_fail(void)
> +{
> + asm volatile (
> + "call %[static_agg_bad_ptr];"
> + "r0 = 0;"
> + "exit;"
> + :
> + : __imm(static_agg_bad_ptr)
> + : __clobber_all);
> +}
Does this test also need __load_if_JITed()? When the JIT is off,
bpf_compute_subprog_ret_regs() bails out before setting ret_reg_pair
for any subprogram, so bpf_ret_reg_pair() returns false for
static_agg_bad_ptr and prepare_func_exit() computes nregs = 1.
The loop "for (i = 0; i < nregs; i++)" then only inspects R0. Since
static_agg_bad_ptr leaves R0 as a scalar and the stack pointer only in
R2, the loop finds nothing and the program verifies cleanly. The test
body "r0 = 0; exit;" never reads R2, so the caller's R2 is already
NOT_INIT from clear_caller_saved_regs() and stays that way.
The __failure tag would then cause the subtest to report failure,
because the program loads successfully instead of failing with the
asserted message.
[ ... ]
> diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
> index d055fc7b3b95..0be9441ec273 100644
> --- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
> +++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
[ ... ]
> +SEC("socket")
> +/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */
> +__load_if_JITed()
> +__log_level(2)
> +__msg("0: .12345.... (85) call pc+2")
> +__msg("1: ..2....... (bf) r0 = r2")
> +/* R2 is not read at the exit of this program, which returns an int, ... */
> +__msg("2: 0......... (95) exit")
> +__msg("3: .......... (b7) r0 = 1")
> +__msg("4: 0......... (b7) r2 = 2")
> +/* ... but it is at the exit of aux2(), which returns a register pair. */
> +__msg("5: 0.2....... (95) exit")
> +__naked void subprog_ret_reg_pair(void)
The comment says R2 is not live at the program's exit because "this
program ... returns an int". The program is declared "__naked void
subprog_ret_reg_pair(void)" - it does not return an int.
Is the reason actually that bpf_compute_subprog_ret_regs() skips the
main program?
In kernel/bpf/verifier.c:437, the loop starts at subprog 1:
/*
* Skip the main program: its return value is the program's exit
* code, read out of R0, so it never uses the register pair. ...
*/
for (subprog = 1; subprog < env->subprog_cnt; subprog++) {
So R2 would not be live at the main program's exit even if subprog 0
were declared to return 16 bytes. The program's return type is not what
decides this.
[ ... ]
> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
> index c4bc8e11c6e1..20a9b9f20e96 100644
> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
[ ... ]
> +__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)
> +{
> + struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };
> +
> + return r;
> +}
> +
> +__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)
> +{
> + struct prog_test_ret_ptr r = { .p = NULL, .tag = tag };
> +
> + return r;
> +}
> +
> +__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)
> +{
> + struct prog_test_ret_ii r = { .a = a, .b = b };
> +
> + return r;
> +}
> #endif /* __x86_64__ || __aarch64__ */
Would riscv64 benefit from test coverage here? Commit 3bfe2cb81326
("bpf: Wire up JIT support for 16-byte kfunc returns") added
bpf_jit_supports_kfunc_ret_reg_pair() returning true for riscv64, so
bpf_add_kfunc_call() accepts 16-byte kfunc returns there.
But the test infrastructure excludes riscv64:
- bpf_kfunc_call_test_ret_pair(), _ret_fastcall(), _ret_ptr(), and
_ret_ii() are not compiled into bpf_testmod on riscv64 (arch guard)
- every test in progs/aggregate_ret_kfunc.c is tagged "__arch_x86_64
__arch_arm64", so test_loader.c skips them on riscv64
The riscv64 LP64D ABI returns a 16-byte two-word struct in a0:a1,
matching the x86-64 rax:rdx and arm64 x0:x1 cases. bpf_misc.h already
defines "__arch_riscv64", and test_loader.c maps ARCH_RISCV64.
Could the arch guard be extended to "defined(__riscv) && __riscv_xlen
== 64" and the tests tagged with "__arch_riscv64" to cover the newly
enabled JIT path?
> +/*
> + * Takes no argument on purpose: with no arguments there is nothing for the sret
> + * pointer to displace, so this needs no architecture guard even though it
> + * returns 24 bytes. See the comment on bpf_kfunc_call_test_i128() above.
> + */
> +__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)
> +{
> + struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };
> +
> + return r;
> +}
The comment has two issues. First, it references "the comment on
bpf_kfunc_call_test_i128() above", but there is no comment on
bpf_kfunc_call_test_i128(). That function appears without explanation a
few lines above.
Second, "with no arguments there is nothing for the sret pointer to
displace" is not why this function is safe without an arch guard. On
SysV x86-64, AAPCS64, and riscv64 LP64D, a 24-byte struct return is
returned via memory through a hidden pointer passed in the first
argument register.
With no declared arguments, the hidden sret pointer would still be taken
from the first argument register (BPF R1), which for this kfunc is never
set up. But that is not what makes this safe: bpf_add_kfunc_call() calls
btf_distill_func_proto() which rejects any return larger than 16 bytes
with "The function %s return type %s is unsupported."
The function is unreachable from BPF on every architecture, which is
what aggregate_ret_kfunc_too_big_fail asserts. The test in
progs/aggregate_ret_kfunc.c states this correct reason; the comment here
states a different one.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31995069373
^ permalink raw reply [flat|nested] 31+ messages in thread* Re: [PATCH bpf-next v6 09/10] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
2026-08-17 5:17 ` bot+bpf-ci
@ 2026-08-17 17:12 ` Yonghong Song
0 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 17:12 UTC (permalink / raw)
To: bot+bpf-ci, bpf
Cc: ast, andrii, daniel, eddyz87, kernel-team, martin.lau, clm,
ihor.solodrai
On 8/16/26 10:17 PM, bot+bpf-ci@kernel.org wrote:
>> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
>> new file mode 100644
>> index 000000000000..e35baaa10fea
>> --- /dev/null
>> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
> [ ... ]
>
>> +SEC("tc")
>> +__failure __msg("At subprogram exit the register R2 is not a scalar value")
>> +__naked int aggregate_ret_global_ptr_fail(void)
>> +{
>> + asm volatile (
>> + "call %[global_agg_bad_ptr];"
>> + "r0 = r2;"
>> + "exit;"
>> + :
>> + : __imm(global_agg_bad_ptr)
>> + : __clobber_all);
>> +}
> Does this test need __load_if_JITed()? Without it, the test runs on
> kernels where the JIT is disabled. When env->prog->jit_requested is
> false, bpf_compute_subprog_ret_regs() returns early without marking any
> ret_reg_pair flag, so global_agg_bad_ptr is not treated as returning a
> register pair.
>
> Main verification happens first, and with nregs = 1 at the call site,
> check_func_call() marks only R0 (the callee's R2 is never propagated to
> the caller). The next instruction "r0 = r2" then reads the caller's
> NOT_INIT R2 from clear_caller_saved_regs() and dies in
> __check_reg_arg() with "R2 !read_ok", never reaching
> do_check_subprogs() where the asserted message "At subprogram exit the
> register R2 is not a scalar value" would appear.
>
> The program still fails to load, but with a different message than the
> test asserts. aggregate_ret_static_precise and
> aggregate_ret_global_precise both carry __load_if_JITed() to ensure the
> R0:R2 convention is active when they run. Would the same gate keep this
> test meaningful?
Right, we should have __load_if_JITed() in the asm test.
>
>> +/*
>> + * R2 is a return register once the subprogram returns a pair, so a stack
>> + * pointer left in it is rejected at the callee's exit exactly as one in R0
>> + * is: the callee frame is gone by the time the caller could use it.
>> + */
>> +SEC("tc")
>> +__failure __msg("cannot return stack pointer to the caller")
>> +__naked int aggregate_ret_static_ptr_fail(void)
>> +{
>> + asm volatile (
>> + "call %[static_agg_bad_ptr];"
>> + "r0 = 0;"
>> + "exit;"
>> + :
>> + : __imm(static_agg_bad_ptr)
>> + : __clobber_all);
>> +}
> Does this test also need __load_if_JITed()? When the JIT is off,
> bpf_compute_subprog_ret_regs() bails out before setting ret_reg_pair
> for any subprogram, so bpf_ret_reg_pair() returns false for
> static_agg_bad_ptr and prepare_func_exit() computes nregs = 1.
>
> The loop "for (i = 0; i < nregs; i++)" then only inspects R0. Since
> static_agg_bad_ptr leaves R0 as a scalar and the stack pointer only in
> R2, the loop finds nothing and the program verifies cleanly. The test
> body "r0 = 0; exit;" never reads R2, so the caller's R2 is already
> NOT_INIT from clear_caller_saved_regs() and stays that way.
>
> The __failure tag would then cause the subtest to report failure,
> because the program loads successfully instead of failing with the
> asserted message.
The same as above, we should have __load_if_JITed() in the above asm.
> [ ... ]
>
>> diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
>> index d055fc7b3b95..0be9441ec273 100644
>> --- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
>> +++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
> [ ... ]
>
>> +SEC("socket")
>> +/* A program observing the pair needs the JIT; see bpf_compute_subprog_ret_regs(). */
>> +__load_if_JITed()
>> +__log_level(2)
>> +__msg("0: .12345.... (85) call pc+2")
>> +__msg("1: ..2....... (bf) r0 = r2")
>> +/* R2 is not read at the exit of this program, which returns an int, ... */
>> +__msg("2: 0......... (95) exit")
>> +__msg("3: .......... (b7) r0 = 1")
>> +__msg("4: 0......... (b7) r2 = 2")
>> +/* ... but it is at the exit of aux2(), which returns a register pair. */
>> +__msg("5: 0.2....... (95) exit")
>> +__naked void subprog_ret_reg_pair(void)
> The comment says R2 is not live at the program's exit because "this
> program ... returns an int". The program is declared "__naked void
> subprog_ret_reg_pair(void)" - it does not return an int.
You are right, the comment is not correct.
>
> Is the reason actually that bpf_compute_subprog_ret_regs() skips the
> main program?
>
> In kernel/bpf/verifier.c:437, the loop starts at subprog 1:
>
> /*
> * Skip the main program: its return value is the program's exit
> * code, read out of R0, so it never uses the register pair. ...
> */
> for (subprog = 1; subprog < env->subprog_cnt; subprog++) {
>
> So R2 would not be live at the main program's exit even if subprog 0
> were declared to return 16 bytes. The program's return type is not what
> decides this.
We skip R2 for main program (subprog 0). It is user's resposibility to
test return value for main program.
>
> [ ... ]
>
>> diff --git a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
>> index c4bc8e11c6e1..20a9b9f20e96 100644
>> --- a/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
>> +++ b/tools/testing/selftests/bpf/test_kmods/bpf_testmod.c
> [ ... ]
>
>> +__bpf_kfunc struct prog_test_ret_pair bpf_kfunc_call_test_ret_fastcall(u64 a, u64 b)
>> +{
>> + struct prog_test_ret_pair r = { .lo = a + b, .hi = a - b };
>> +
>> + return r;
>> +}
>> +
>> +__bpf_kfunc struct prog_test_ret_ptr bpf_kfunc_call_test_ret_ptr(u64 tag)
>> +{
>> + struct prog_test_ret_ptr r = { .p = NULL, .tag = tag };
>> +
>> + return r;
>> +}
>> +
>> +__bpf_kfunc struct prog_test_ret_ii bpf_kfunc_call_test_ret_ii(int a, int b)
>> +{
>> + struct prog_test_ret_ii r = { .a = a, .b = b };
>> +
>> + return r;
>> +}
>> #endif /* __x86_64__ || __aarch64__ */
> Would riscv64 benefit from test coverage here? Commit 3bfe2cb81326
> ("bpf: Wire up JIT support for 16-byte kfunc returns") added
> bpf_jit_supports_kfunc_ret_reg_pair() returning true for riscv64, so
> bpf_add_kfunc_call() accepts 16-byte kfunc returns there.
>
> But the test infrastructure excludes riscv64:
>
> - bpf_kfunc_call_test_ret_pair(), _ret_fastcall(), _ret_ptr(), and
> _ret_ii() are not compiled into bpf_testmod on riscv64 (arch guard)
> - every test in progs/aggregate_ret_kfunc.c is tagged "__arch_x86_64
> __arch_arm64", so test_loader.c skips them on riscv64
>
> The riscv64 LP64D ABI returns a 16-byte two-word struct in a0:a1,
> matching the x86-64 rax:rdx and arm64 x0:x1 cases. bpf_misc.h already
> defines "__arch_riscv64", and test_loader.c maps ARCH_RISCV64.
>
> Could the arch guard be extended to "defined(__riscv) && __riscv_xlen
> == 64" and the tests tagged with "__arch_riscv64" to cover the newly
> enabled JIT path?
I didn't test riscv64 as I don't have infrastructure for this.
riscv64 can be implemented later.
>
>> +/*
>> + * Takes no argument on purpose: with no arguments there is nothing for the sret
>> + * pointer to displace, so this needs no architecture guard even though it
>> + * returns 24 bytes. See the comment on bpf_kfunc_call_test_i128() above.
>> + */
>> +__bpf_kfunc struct prog_test_ret_big bpf_kfunc_call_test_ret_big(void)
>> +{
>> + struct prog_test_ret_big r = { .a = 1, .b = 2, .c = 3 };
>> +
>> + return r;
>> +}
> The comment has two issues. First, it references "the comment on
> bpf_kfunc_call_test_i128() above", but there is no comment on
> bpf_kfunc_call_test_i128(). That function appears without explanation a
> few lines above.
Okay, we can remove 'See the comment on bpf_kfunc_call_test_i128() above'.
>
> Second, "with no arguments there is nothing for the sret pointer to
> displace" is not why this function is safe without an arch guard. On
> SysV x86-64, AAPCS64, and riscv64 LP64D, a 24-byte struct return is
> returned via memory through a hidden pointer passed in the first
> argument register.
>
> With no declared arguments, the hidden sret pointer would still be taken
> from the first argument register (BPF R1), which for this kfunc is never
> set up. But that is not what makes this safe: bpf_add_kfunc_call() calls
> btf_distill_func_proto() which rejects any return larger than 16 bytes
> with "The function %s return type %s is unsupported."
>
> The function is unreachable from BPF on every architecture, which is
> what aggregate_ret_kfunc_too_big_fail asserts. The test in
> progs/aggregate_ret_kfunc.c states this correct reason; the comment here
> states a different one.
Okay, I may just remove the above comments.
>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/31995069373
^ permalink raw reply [flat|nested] 31+ messages in thread
* [PATCH bpf-next v6 10/10] Documentation/bpf: Document up to 16-byte kfunc return values in R0:R2
2026-08-17 4:21 [PATCH bpf-next v6 00/10] bpf: Support aggregate return values up to 16 bytes Yonghong Song
` (8 preceding siblings ...)
2026-08-17 4:22 ` [PATCH bpf-next v6 09/10] selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns Yonghong Song
@ 2026-08-17 4:22 ` Yonghong Song
9 siblings, 0 replies; 31+ messages in thread
From: Yonghong Song @ 2026-08-17 4:22 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Andrii Nakryiko, Daniel Borkmann,
Eduard Zingerman, kernel-team
kfuncs may now return a value larger than 8 bytes and up to 16 bytes (a
scalar-only struct or union, or an __int128), passed back in the R0:R2
register pair. Add a kfunc return-value section documenting this,
including that a struct or union up to 8 bytes is returned in R0 alone,
which struct and union members are accepted, that the R0:R2 register pair
requires JIT support (bpf_jit_supports_kfunc_ret_reg_pair()), and that a
return value larger than 16 bytes is unsupported.
Also note that the same convention applies to BPF subprogram returns, and
document the consequence for a global subprogram: it must assign both
halves of a register-pair return, since an unassigned R2 may be left
holding a pointer argument and is then rejected as a leak.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
Documentation/bpf/kfuncs.rst | 65 ++++++++++++++++++++++++++++++++++++
1 file changed, 65 insertions(+)
diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index 85f73e0bbd0f..89dea6b0b024 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -575,6 +575,71 @@ is also covered by this recovery. A kfunc handed an arena pointer may
therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
against the arena. Larger accesses must verify the range explicitly.
+2.9 kfunc Return Values
+-----------------------
+
+A kfunc may return a scalar, a pointer, or a small struct or union by
+value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.
+
+A struct or union returned by value must be composed only of scalars
+(recursively), where a scalar is an integer or an enum; arrays of scalars are
+allowed as members. Its bytes are handed back to the program as the raw
+contents of R0 (and R2), so a pointer field would be laundered into a scalar
+and escape the verifier's pointer provenance and reference tracking. A struct
+or union with a pointer member is therefore rejected at load time, and so is
+one with a floating-point member, which the ABI may not return in R0:R2 at
+all.
+
+A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a
+scalar-only struct or union, or an ``__int128``. Such a value is returned
+in the register pair R0:R2, matching the convention LLVM uses for the BPF
+target: the first 8 bytes in R0 and the second 8 bytes in R2. A struct or
+union of 8 bytes or less is returned in R0 alone.
+
+::
+
+ struct bpf_pair { __u64 a, b; }; /* 16 bytes */
+
+ __bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)
+ {
+ struct bpf_pair p = { .a = 1, .b = 2 };
+
+ return p; /* p.a in R0, p.b in R2 */
+ }
+
+Returning a value in the R0:R2 pair requires the JIT to place the second
+half of the return value into R2, which not every architecture supports
+right now. A kfunc with a return value larger than 8 bytes is therefore
+rejected at load time on a JIT that does not advertise this capability (see
+``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run
+by the interpreter. A return value larger than 16 bytes is not supported.
+
+The same R0:R2 convention applies to a BPF subprogram, global or static, that
+returns an ``__int128`` or a struct or union larger than 8 bytes. It is only
+used when the program is JITed, since the interpreter propagates only R0 out of
+a subprogram: without a JIT the return value stays in R0 alone, and a caller
+reading R2 is rejected for reading an uninitialized register. A global
+subprogram is verified in isolation, so its by-value struct or union return is
+restricted to scalars just like a kfunc's; a static subprogram is verified
+inline and has no such restriction. The main program is not covered: its return
+value is the program's exit code, read out of R0 alone, so a declared upper
+half is never looked at.
+
+A global subprogram must leave a scalar in *every* register of the pair, so
+both halves of the returned value have to be assigned. Leaving the upper half
+uninitialized is not merely untidy: the compiler is then free to leave R2
+holding whatever it happened to hold, which for a subprogram taking a pointer
+argument is typically that pointer. Handing the caller an unknown scalar built
+from a pointer is a leak, so the verifier rejects it with::
+
+ At subprogram exit the register R2 is not a scalar value (...)
+
+Initialize the whole return value, for example ``struct pair p = {};``, to
+avoid this. A static subprogram is exempt from the scalar-only rule: it is
+verified inline, so an unassigned R2 is simply passed back to the caller as
+uninitialized and only a caller that reads it fails. A stack pointer left in
+R2 is still rejected there, just as one in R0 is.
+
.. _BPF_kfunc_lifecycle_expectations:
3. kfunc lifecycle expectations
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 31+ messages in thread