From: Yonghong Song <yonghong.song@linux.dev>
To: bpf@vger.kernel.org
Cc: Alexei Starovoitov <ast@kernel.org>,
Andrii Nakryiko <andrii@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
"Jose E . Marchesi" <jose.marchesi@oracle.com>,
kernel-team@fb.com, Martin KaFai Lau <martin.lau@kernel.org>
Subject: [PATCH bpf-next v4 14/25] bpf: Support stack arguments for kfunc calls
Date: Tue, 12 May 2026 21:51:04 -0700 [thread overview]
Message-ID: <20260513045104.2391543-1-yonghong.song@linux.dev> (raw)
In-Reply-To: <20260513044949.2382019-1-yonghong.song@linux.dev>
Extend the stack argument mechanism to kfunc calls, allowing kfuncs
with more than 5 parameters to receive additional arguments via the
r11-based stack arg area.
For kfuncs, the caller is a BPF program and the callee is a kernel
function. The BPF program writes outgoing args at negative r11
offsets, following the same convention as BPF-to-BPF calls:
Outgoing: r11 - 8 (arg6), ..., r11 - N*8 (last arg)
The following is an example:
int foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7) {
...
kfunc1(a1, a2, a3, a4, a5, a6, a7, a8);
...
kfunc2(a1, a2, a3, a4, a5, a6, a7, a8, a9);
...
}
Caller (foo), generated by llvm
===============================
Incoming (positive offsets):
r11+8: [incoming arg 6]
r11+16: [incoming arg 7]
Outgoing for kfunc1 (negative offsets):
r11-8: [outgoing arg 6]
r11-16: [outgoing arg 7]
r11-24: [outgoing arg 8]
Outgoing for kfunc2 (negative offsets):
r11-8: [outgoing arg 6]
r11-16: [outgoing arg 7]
r11-24: [outgoing arg 8]
r11-32: [outgoing arg 9]
Later JIT will marshal outgoing arguments to the native calling
convention for kfunc1() and kfunc2().
For kfunc calls where stack args are used as constant or size
parameters, a mark_stack_arg_precision() helper is used to propagate
precision and do proper backtracking.
There are two places where meta->release_regno needs to keep
regno for later releasing the reference. Also, 'cur_aux(env)->arg_prog = regno'
is also keeping regno for later fixup. Since stack arguments don't have a valid
register number (regno is negative), these three cases are rejected for now
if the argument is on the stack.
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
---
kernel/bpf/verifier.c | 77 +++++++++++++++++++++++++++++++++----------
1 file changed, 60 insertions(+), 17 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 18bbc5b6c2d2..a363811aec7a 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11160,14 +11160,12 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta)
}
static enum kfunc_ptr_arg_type
-get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
- struct bpf_kfunc_call_arg_meta *meta,
+get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_func_state *caller,
+ struct bpf_reg_state *regs, struct bpf_kfunc_call_arg_meta *meta,
const struct btf_type *t, const struct btf_type *ref_t,
const char *ref_tname, const struct btf_param *args,
int arg, int nargs, argno_t argno, struct bpf_reg_state *reg)
{
- u32 regno = arg + 1;
- struct bpf_reg_state *regs = cur_regs(env);
bool arg_mem_size = false;
if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] ||
@@ -11176,8 +11174,8 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
return KF_ARG_PTR_TO_CTX;
if (arg + 1 < nargs &&
- (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1], ®s[regno + 1]) ||
- is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1], ®s[regno + 1])))
+ (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1)) ||
+ is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1))))
arg_mem_size = true;
/* In this function, we verify the kfunc's BTF as per the argument type,
@@ -11842,6 +11840,8 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
int insn_idx)
{
const char *func_name = meta->func_name, *ref_tname;
+ struct bpf_func_state *caller = cur_func(env);
+ struct bpf_reg_state *regs = cur_regs(env);
const struct btf *btf = meta->btf;
const struct btf_param *args;
struct btf_record *rec;
@@ -11850,21 +11850,31 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
args = (const struct btf_param *)(meta->func_proto + 1);
nargs = btf_type_vlen(meta->func_proto);
- if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+ if (nargs > MAX_BPF_FUNC_ARGS) {
verbose(env, "Function %s has %d > %d args\n", func_name, nargs,
- MAX_BPF_FUNC_REG_ARGS);
+ MAX_BPF_FUNC_ARGS);
return -EINVAL;
}
+ if (nargs > MAX_BPF_FUNC_REG_ARGS && !bpf_jit_supports_stack_args()) {
+ verbose(env, "JIT does not support kfunc %s() with %d args\n",
+ func_name, nargs);
+ return -ENOTSUPP;
+ }
+
+ ret = check_outgoing_stack_args(env, caller, nargs);
+ if (ret)
+ return ret;
/* Check that BTF function arguments match actual types that the
* verifier sees.
*/
for (i = 0; i < nargs; i++) {
- struct bpf_reg_state *regs = cur_regs(env), *reg = ®s[i + 1];
+ struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, i);
const struct btf_type *t, *ref_t, *resolve_ret;
enum bpf_arg_type arg_type = ARG_DONTCARE;
argno_t argno = argno_from_arg(i + 1);
- u32 regno = i + 1, ref_id, type_size;
+ int regno = reg_from_argno(argno);
+ u32 ref_id, type_size;
bool is_ret_buf_sz = false;
int kf_arg_type;
@@ -11874,6 +11884,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
verifier_bug(env, "Only 1 prog->aux argument supported per-kfunc");
return -EFAULT;
}
+ if (regno < 0) {
+ verbose(env, "%s prog->aux cannot be a stack argument\n",
+ reg_arg_name(env, argno));
+ return -EINVAL;
+ }
meta->arg_prog = true;
cur_aux(env)->arg_prog = regno;
continue;
@@ -11900,7 +11915,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
reg_arg_name(env, argno));
return -EINVAL;
}
- ret = mark_chain_precision(env, regno);
+ if (regno >= 0)
+ ret = mark_chain_precision(env, regno);
+ else
+ ret = mark_stack_arg_precision(env, i);
if (ret < 0)
return ret;
meta->arg_constant.found = true;
@@ -11925,7 +11943,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
}
meta->r0_size = reg->var_off.value;
- ret = mark_chain_precision(env, regno);
+ if (regno >= 0)
+ ret = mark_chain_precision(env, regno);
+ else
+ ret = mark_stack_arg_precision(env, i);
if (ret)
return ret;
}
@@ -11953,14 +11974,21 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
return -EFAULT;
}
meta->ref_obj_id = reg->ref_obj_id;
- if (is_kfunc_release(meta))
+ if (is_kfunc_release(meta)) {
+ if (regno < 0) {
+ verbose(env, "%s release arg cannot be a stack argument\n",
+ reg_arg_name(env, argno));
+ return -EINVAL;
+ }
meta->release_regno = regno;
+ }
}
ref_t = btf_type_skip_modifiers(btf, t->type, &ref_id);
ref_tname = btf_name_by_offset(btf, ref_t->name_off);
- kf_arg_type = get_kfunc_ptr_arg_type(env, meta, t, ref_t, ref_tname, args, i, nargs, argno, reg);
+ kf_arg_type = get_kfunc_ptr_arg_type(env, caller, regs, meta, t, ref_t, ref_tname,
+ args, i, nargs, argno, reg);
if (kf_arg_type < 0)
return kf_arg_type;
@@ -12110,6 +12138,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
dynptr_arg_type |= DYNPTR_TYPE_FILE;
} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_file_discard]) {
dynptr_arg_type |= DYNPTR_TYPE_FILE | OBJ_RELEASE;
+ if (regno < 0) {
+ verbose(env, "%s release arg cannot be a stack argument\n",
+ reg_arg_name(env, argno));
+ return -EINVAL;
+ }
meta->release_regno = regno;
} else if (meta->func_id == special_kfunc_list[KF_bpf_dynptr_clone] &&
(dynptr_arg_type & MEM_UNINIT)) {
@@ -12264,9 +12297,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
break;
case KF_ARG_PTR_TO_MEM_SIZE:
{
- struct bpf_reg_state *buff_reg = ®s[regno];
+ struct bpf_reg_state *buff_reg = reg;
const struct btf_param *buff_arg = &args[i];
- struct bpf_reg_state *size_reg = ®s[regno + 1];
+ struct bpf_reg_state *size_reg = get_func_arg_reg(caller, regs, i + 1);
const struct btf_param *size_arg = &args[i + 1];
argno_t next_argno = argno_from_arg(i + 2);
@@ -13171,8 +13204,18 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
clear_all_pkt_pointers(env);
nargs = btf_type_vlen(meta.func_proto);
+ if (nargs > MAX_BPF_FUNC_REG_ARGS) {
+ struct bpf_func_state *caller = cur_func(env);
+ struct bpf_subprog_info *caller_info = &env->subprog_info[caller->subprogno];
+ u16 out_stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS;
+ u16 stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) + out_stack_arg_cnt;
+
+ if (stack_arg_cnt > caller_info->stack_arg_cnt)
+ caller_info->stack_arg_cnt = stack_arg_cnt;
+ }
+
args = (const struct btf_param *)(meta.func_proto + 1);
- for (i = 0; i < nargs; i++) {
+ for (i = 0; i < min_t(int, nargs, MAX_BPF_FUNC_REG_ARGS); i++) {
u32 regno = i + 1;
t = btf_type_skip_modifiers(desc_btf, args[i].type, NULL);
--
2.53.0-Meta
next prev parent reply other threads:[~2026-05-13 4:51 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-05-13 4:49 [PATCH bpf-next v4 00/25] bpf: Support stack arguments for BPF functions and kfuncs Yonghong Song
2026-05-13 4:49 ` [PATCH bpf-next v4 01/25] bpf: Convert bpf_get_spilled_reg macro to static inline function Yonghong Song
2026-05-13 4:50 ` [PATCH bpf-next v4 02/25] bpf: Remove copy_register_state wrapper function Yonghong Song
2026-05-13 4:50 ` [PATCH bpf-next v4 03/25] bpf: Add helper functions for r11-based stack argument insns Yonghong Song
2026-05-13 4:50 ` [PATCH bpf-next v4 04/25] bpf: Set sub->arg_cnt earlier in btf_prepare_func_args() Yonghong Song
2026-05-13 4:50 ` [PATCH bpf-next v4 05/25] bpf: Support stack arguments for bpf functions Yonghong Song
2026-05-14 10:46 ` sashiko-bot
2026-05-14 16:07 ` Yonghong Song
2026-05-13 4:50 ` [PATCH bpf-next v4 06/25] bpf: Refactor jmp history to use dedicated spi/frame fields Yonghong Song
2026-05-13 4:50 ` [PATCH bpf-next v4 07/25] bpf: Add precision marking and backtracking for stack argument slots Yonghong Song
2026-05-13 5:44 ` bot+bpf-ci
2026-05-13 4:50 ` [PATCH bpf-next v4 08/25] bpf: Refactor record_call_access() to extract per-arg logic Yonghong Song
2026-05-13 4:50 ` [PATCH bpf-next v4 09/25] bpf: Use arg_is_fp() in has_fp_args() Yonghong Song
2026-05-13 4:50 ` [PATCH bpf-next v4 10/25] bpf: Extend liveness analysis to track stack argument slots Yonghong Song
2026-05-13 5:44 ` bot+bpf-ci
2026-05-14 22:53 ` sashiko-bot
2026-05-13 4:50 ` [PATCH bpf-next v4 11/25] bpf: Reject stack arguments in non-JITed programs Yonghong Song
2026-05-13 5:33 ` bot+bpf-ci
2026-05-14 23:59 ` sashiko-bot
2026-05-13 4:50 ` [PATCH bpf-next v4 12/25] bpf: Prepare architecture JIT support for stack arguments Yonghong Song
2026-05-13 5:33 ` bot+bpf-ci
2026-05-15 0:30 ` sashiko-bot
2026-05-13 4:50 ` [PATCH bpf-next v4 13/25] bpf: Enable r11 based insns Yonghong Song
2026-05-13 4:51 ` Yonghong Song [this message]
2026-05-13 4:51 ` [PATCH bpf-next v4 15/25] bpf: Reject stack arguments if tail call reachable Yonghong Song
2026-05-13 5:33 ` bot+bpf-ci
2026-05-15 3:23 ` sashiko-bot
2026-05-13 4:51 ` [PATCH bpf-next v4 16/25] bpf: Disable private stack for x86_64 if stack arguments used Yonghong Song
2026-05-13 5:33 ` bot+bpf-ci
2026-05-13 4:51 ` [PATCH bpf-next v4 17/25] bpf,x86: Implement JIT support for stack arguments Yonghong Song
2026-05-13 4:51 ` [PATCH bpf-next v4 18/25] selftests/bpf: Add tests for BPF function " Yonghong Song
2026-05-13 4:51 ` [PATCH bpf-next v4 19/25] selftests/bpf: Add tests for stack argument validation Yonghong Song
2026-05-13 4:51 ` [PATCH bpf-next v4 20/25] selftests/bpf: Add BTF fixup for __naked subprog parameter names Yonghong Song
2026-05-13 4:51 ` [PATCH bpf-next v4 21/25] selftests/bpf: Add verifier tests for stack argument validation Yonghong Song
2026-05-13 4:51 ` [PATCH bpf-next v4 22/25] selftests/bpf: Add precision backtracking test for stack arguments Yonghong Song
2026-05-13 4:51 ` [PATCH bpf-next v4 23/25] bpf, arm64: Map BPF_REG_0 to x8 instead of x7 Yonghong Song
2026-05-13 4:51 ` [PATCH bpf-next v4 24/25] bpf, arm64: Add JIT support for stack arguments Yonghong Song
2026-05-13 4:52 ` [PATCH bpf-next v4 25/25] selftests/bpf: Enable stack argument tests for arm64 Yonghong Song
2026-05-13 16:33 ` [PATCH bpf-next v4 00/25] bpf: Support stack arguments for BPF functions and kfuncs Alexei Starovoitov
2026-05-13 17:41 ` Yonghong Song
2026-05-13 17:51 ` Alexei Starovoitov
2026-05-13 18:11 ` Yonghong Song
2026-05-13 16:40 ` patchwork-bot+netdevbpf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260513045104.2391543-1-yonghong.song@linux.dev \
--to=yonghong.song@linux.dev \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=jose.marchesi@oracle.com \
--cc=kernel-team@fb.com \
--cc=martin.lau@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.