* [PATCH bpf 01/81] selftests/bpf: Remove tests for global data in arena
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 02/81] libbpf: Remove .arena.data Alexei Starovoitov
` (81 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The .arena.data section and the BPF_F_ARENA_SCALAR flag let a prog in Rust
keep its global data in arena. The subsequent patches remove the two,
because the compiler can do this work. First remove the tests for
.arena.data.
One of the tests is a prog in Rust. It has no replacement, because LLVM
does not put the globals of Rust in arena at this time. No other selftest
is in Rust. Thus also remove the makefile rules that build Rust.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/Makefile | 12 +-
.../testing/selftests/bpf/Makefile.buildvars | 25 --
tools/testing/selftests/bpf/Makefile.skel | 18 +-
.../selftests/bpf/prog_tests/data_in_arena.c | 216 ------------------
.../selftests/bpf/progs/data_in_arena.c | 112 ---------
.../selftests/bpf/progs/data_in_arena_decl.c | 37 ---
.../bpf/progs/data_in_arena_extern.c | 20 --
.../selftests/bpf/progs/data_in_arena_fail.c | 20 --
.../selftests/bpf/progs/data_in_arena_nomap.c | 20 --
.../selftests/bpf/progs/data_in_arena_rust.rs | 73 ------
10 files changed, 2 insertions(+), 551 deletions(-)
delete mode 100644 tools/testing/selftests/bpf/prog_tests/data_in_arena.c
delete mode 100644 tools/testing/selftests/bpf/progs/data_in_arena.c
delete mode 100644 tools/testing/selftests/bpf/progs/data_in_arena_decl.c
delete mode 100644 tools/testing/selftests/bpf/progs/data_in_arena_extern.c
delete mode 100644 tools/testing/selftests/bpf/progs/data_in_arena_fail.c
delete mode 100644 tools/testing/selftests/bpf/progs/data_in_arena_nomap.c
delete mode 100644 tools/testing/selftests/bpf/progs/data_in_arena_rust.rs
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 1e4af62e8312..c99e0f623e10 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -423,16 +423,6 @@ $(LIBARENA_ASAN_SKEL): $(INCLUDE_DIR)/vmlinux.h $(BPFOBJ) $(LIBARENA_BPF_DEPS)
+$(MAKE) -C libarena libarena_asan.skel.h $(LIBARENA_MAKE_ARGS)
endif
-# #![no_std] looks for compiler_builtins too. Nothing of it is used.
-ifneq ($(RUST_CORE),)
-$(RUST_CORE): $(RUST_CORE_SRC)
- $(call msg,RUSTC,,$@)
- $(Q)mkdir -p $(@D)
- +$(Q)$(RUSTC_BPF) -A warnings --edition 2024 --crate-name core --out-dir $(@D) $<
- +$(Q)echo '#![feature(compiler_builtins)] #![compiler_builtins] #![no_std]' | \
- $(RUSTC_BPF) -A warnings --crate-name compiler_builtins --out-dir $(@D) -
-endif
-
# Generated test list headers
define gen_tests_hdr
@@ -468,7 +458,7 @@ RUNNER_PREREQS := $(INCLUDE_DIR)/vmlinux.h $(BPFOBJ) $(BPFTOOL) \
$(VERIFY_SIG_HDR) $(PRIVATE_KEY) $(VERIFICATION_CERT) \
$(LIBARENA_SKEL) $(LIBARENA_ASAN_SKEL) \
prog_tests/tests.h map_tests/tests.h \
- $(RUNNER_OBJS) $(RUST_CORE)
+ $(RUNNER_OBJS)
# Runtime fixtures for each test_progs flavor.
RUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read \
diff --git a/tools/testing/selftests/bpf/Makefile.buildvars b/tools/testing/selftests/bpf/Makefile.buildvars
index ff3476bc40a4..d2a0c0031b87 100644
--- a/tools/testing/selftests/bpf/Makefile.buildvars
+++ b/tools/testing/selftests/bpf/Makefile.buildvars
@@ -109,31 +109,6 @@ HOST_INCLUDE_DIR := $(INCLUDE_DIR)
endif
RESOLVE_BTFIDS := $(HOST_BUILD_DIR)/resolve_btfids/resolve_btfids
-# Programs in Rust are built by upstream rustc. It has no prebuilt core for
-# the bpf target, so it has to come with the source of core:
-# rustup component add rust-src
-# core is built as edition 2024, which it is since rustc 1.87.
-# rustc emits LLVM bitcode and clang makes the object of it, so clang has to be
-# 23 or newer and not older than LLVM of rustc.
-# Otherwise RUST_CORE is empty and the tests are skipped.
-RUSTC ?= rustc
-RUST_CORE_SRC := $(wildcard $(shell $(RUSTC) --print sysroot 2>/dev/null)$\
- /lib/rustlib/src/rust/library/core/src/lib.rs)
-ifneq ($(RUST_CORE_SRC),)
-ifeq ($(shell { clang=$$(echo __clang_major__ | $(CLANG) -E -P -x c -) && \
- llvm=$$($(srctree)/scripts/rustc-llvm-version.sh $(RUSTC)) && \
- [ $$($(srctree)/scripts/rustc-version.sh $(RUSTC)) -ge 108700 ] && \
- [ $$clang -ge 23 ] && [ $$clang -ge $$((llvm / 10000)) ]; } \
- 2>/dev/null && echo y),y)
-RUST_CORE := $(BUILD_DIR)/rust/libcore.rlib
-endif
-endif
-# RUSTC_BOOTSTRAP=1 is to build core with a stable rustc, like the kernel does.
-# panic=abort is a stop gap until panic=unwind is supported.
-RUSTC_BPF = RUSTC_BOOTSTRAP=1 $(RUSTC) -O -C panic=abort --crate-type rlib \
- --target $(if $(IS_LITTLE_ENDIAN),bpfel,bpfeb)-unknown-none \
- -L $(dir $(RUST_CORE))
-
DEFAULT_BPFTOOL := $(HOST_SCRATCH_DIR)/sbin/bpftool
ifneq ($(CROSS_COMPILE),)
CROSS_BPFTOOL := $(SCRATCH_DIR)/sbin/bpftool
diff --git a/tools/testing/selftests/bpf/Makefile.skel b/tools/testing/selftests/bpf/Makefile.skel
index 6570c48f92b0..e8699903feef 100644
--- a/tools/testing/selftests/bpf/Makefile.skel
+++ b/tools/testing/selftests/bpf/Makefile.skel
@@ -20,8 +20,7 @@ ifneq ($(BPF_CC),)
BPF_SRCS := $(notdir $(wildcard progs/*.c))
BPF_OBJS := $(patsubst %.c,$(RDIR)/%.bpf.o,$(BPF_SRCS))
-SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c \
- data_in_arena_extern.c data_in_arena_nomap.c
+SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c
LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
linked_vars.skel.h linked_maps.skel.h linked_arena.skel.h \
@@ -138,21 +137,6 @@ $(LINKED_SKELS_H): $(RDIR)/%.skel.h: $$(addprefix $(RDIR)/,$$($$*.skel.h-deps))
$(BPFTOOL) $(GEN_SKEL) | $(RDIR)
$(Q)$(cmd_bpf_link_skel)
-# Programs in Rust, see Makefile.buildvars. No skeletons: the objects may be absent.
-ifneq ($(RUST_CORE),)
-ifeq ($(BPF_CC),$(CLANG))
-RUST_OBJS := $(patsubst progs/%.rs,$(RDIR)/%.bpf.o,$(wildcard progs/*.rs))
-BPF_OBJS += $(RUST_OBJS)
-
-$(RUST_OBJS): $(RDIR)/%.bpf.o: progs/%.rs $(RUST_CORE) | $(RDIR)
- $(call msg,RUSTC,$(BINARY),$@)
- +$(Q)$(RUSTC_BPF) --edition 2021 -C debuginfo=2 --emit=llvm-bc \
- $(patsubst -mcpu=%,-C target-cpu=%,$(filter -mcpu=%,$(BPF_CC_FLAGS))) \
- -o $(dir $(RUST_CORE))$(BINARY)-$*.bc $< && \
- $(BPF_CC) $(BPF_CC_FLAGS) -c $(dir $(RUST_CORE))$(BINARY)-$*.bc -o $@ $(call skip_on_fail,BPF)
-endif
-endif
-
# Programs in C++. No skeletons: gcc doesn't build them.
ifeq ($(BPF_CC),$(CLANG))
CXX_OBJS := $(patsubst progs/%.cpp,$(RDIR)/%.bpf.o,$(wildcard progs/*.cpp))
diff --git a/tools/testing/selftests/bpf/prog_tests/data_in_arena.c b/tools/testing/selftests/bpf/prog_tests/data_in_arena.c
deleted file mode 100644
index cb1023507c01..000000000000
--- a/tools/testing/selftests/bpf/prog_tests/data_in_arena.c
+++ /dev/null
@@ -1,216 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-#include <test_progs.h>
-#include "data_in_arena.skel.h"
-#include "data_in_arena_decl.skel.h"
-#include "data_in_arena_fail.skel.h"
-
-static int run_prog(struct bpf_program *prog)
-{
- LIBBPF_OPTS(bpf_test_run_opts, topts);
-
- if (!ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(prog), &topts), "test_run"))
- return -1;
- return topts.retval;
-}
-
-static void run(struct data_in_arena *skel, int counter)
-{
- int i;
-
- ASSERT_EQ(run_prog(skel->progs.use_data), counter + 7 + 1 + 2, "retval");
- ASSERT_EQ(skel->bss->sum, counter + 7 + 1 + 2, "sum");
- ASSERT_EQ(skel->data->counter, counter + 1, "counter");
- ASSERT_EQ(skel->data->pair[1], 5, "pair[1]");
- for (i = 0; i < 4; i++)
- ASSERT_EQ(skel->bss->table[i], 10 * (i + 1) + i, "table");
-}
-
-static void test_in_arena(void)
-{
- struct bpf_map_info info = {};
- struct data_in_arena *skel;
- __u32 len = sizeof(info);
- struct bpf_map *arena;
- size_t sz;
-
- skel = data_in_arena__open();
- if (!ASSERT_OK_PTR(skel, "open"))
- return;
-
- arena = bpf_object__find_map_by_name(skel->obj, "arena");
- if (!ASSERT_OK_PTR(arena, "arena"))
- goto out;
- ASSERT_EQ(bpf_map__type(arena), BPF_MAP_TYPE_ARENA, "arena type");
- ASSERT_EQ(bpf_map__max_entries(arena), 1, "arena pages");
- ASSERT_OK(bpf_map__set_max_entries(arena, 8), "arena resize");
- ASSERT_FALSE(bpf_map__autocreate(skel->maps.data), "data autocreate");
- ASSERT_FALSE(bpf_map__autocreate(skel->maps.bss), "bss autocreate");
- ASSERT_FALSE(bpf_map__autocreate(skel->maps.rodata), "rodata autocreate");
- ASSERT_EQ(bpf_map__set_autocreate(skel->maps.data, true), -EOPNOTSUPP, "set_autocreate");
- ASSERT_EQ(bpf_map__set_value_size(skel->maps.bss, 4096), -EOPNOTSUPP, "set_value_size");
- ASSERT_EQ(bpf_map__initial_value(skel->maps.data, &sz), skel->data, "initial_value");
- ASSERT_EQ(sz, sizeof(*skel->data), "initial_value size");
-
- /* initial values are set the usual way */
- skel->data->counter = 100;
-
- if (!ASSERT_OK(data_in_arena__load(skel), "load"))
- goto out;
- /* there are no maps behind the sections */
- ASSERT_ERR(bpf_map_get_info_by_fd(bpf_map__fd(skel->maps.data), &info, &len), "data map");
- ASSERT_ERR(bpf_map_get_info_by_fd(bpf_map__fd(skel->maps.bss), &info, &len), "bss map");
- ASSERT_ERR(bpf_map_get_info_by_fd(bpf_map__fd(skel->maps.rodata), &info, &len),
- "rodata map");
- run(skel, 100);
-
- /* pointers to data next to pointers to functions */
- ASSERT_EQ(run_prog(skel->progs.use_ops), 42 + 1 + 'e', "use_ops");
- ASSERT_EQ(skel->data->counter, 102, "counter");
-
- /* alignment of sections and pointers to data in data */
- ASSERT_EQ((unsigned long)&skel->bss->aligned64 % 64, 0, "alignment");
- ASSERT_EQ(run_prog(skel->progs.use_ptrs), 0, "use_ptrs");
- ASSERT_EQ(skel->data->x, 43, "x");
- ASSERT_EQ(skel->bss->aligned64.v[7], 7, "aligned64");
- ASSERT_EQ(skel->data->px, &skel->data->x, "px");
-
- /* format strings of bpf_printk() and BPF_SNPRINTF() */
- ASSERT_EQ(run_prog(skel->progs.use_printk), sizeof("43-7"), "use_printk");
- ASSERT_STREQ(skel->bss->out, "43-7", "out");
-out:
- data_in_arena__destroy(skel);
-}
-
-/* The object has an arena map and __arena variables */
-static void test_declared_arena(void)
-{
- struct data_in_arena_decl *skel;
- struct bpf_map *map;
- int arenas = 0;
-
- skel = data_in_arena_decl__open();
- if (!ASSERT_OK_PTR(skel, "open"))
- return;
- bpf_object__for_each_map(map, skel->obj)
- arenas += bpf_map__type(map) == BPF_MAP_TYPE_ARENA;
- ASSERT_EQ(arenas, 1, "no second arena");
- skel->data->counter = 6;
- if (!ASSERT_OK(data_in_arena_decl__load(skel), "load"))
- goto out;
- ASSERT_EQ(run_prog(skel->progs.use_data), 6 + 7 + 11, "retval");
- ASSERT_EQ(run_prog(skel->progs.use_data), 7 + 7 + 12, "retval");
- ASSERT_EQ(skel->bss->sum, 7 + 7 + 12, "sum");
- ASSERT_EQ(skel->data->counter, 8, "counter");
-out:
- data_in_arena_decl__destroy(skel);
-}
-
-/* A pointer in data that can't be made an address of arena fails the load */
-static void test_ptr_to_map(void)
-{
- struct data_in_arena_fail *skel;
-
- skel = data_in_arena_fail__open();
- if (!ASSERT_OK_PTR(skel, "open"))
- return;
- ASSERT_ERR(data_in_arena_fail__load(skel), "load");
- data_in_arena_fail__destroy(skel);
-}
-
-/* So does a pointer to a variable of the kernel. There is no skeleton: the open fails. */
-static void test_ptr_to_extern(void)
-{
- struct bpf_object *obj;
-
- obj = bpf_object__open_file("./data_in_arena_extern.bpf.o", NULL);
- if (!ASSERT_ERR_PTR(obj, "open"))
- bpf_object__close(obj);
-}
-
-/* __arena variables and no arena map. There is no skeleton: the open fails. */
-static void test_arena_var_no_map(void)
-{
- struct bpf_object *obj;
-
- obj = bpf_object__open_file("./data_in_arena_nomap.bpf.o", NULL);
- if (!ASSERT_ERR_PTR(obj, "open"))
- bpf_object__close(obj);
-}
-
-/* The address of an arena that libbpf doesn't create is not known. No pointers in data then. */
-static void test_not_my_arena(bool pin)
-{
- LIBBPF_OPTS(bpf_map_create_opts, opts, .map_flags = BPF_F_MMAPABLE);
- struct data_in_arena *skel;
- struct bpf_map *arena;
- int fd = -1;
-
- skel = data_in_arena__open();
- if (!ASSERT_OK_PTR(skel, "open"))
- return;
- arena = bpf_object__find_map_by_name(skel->obj, "arena");
- if (!ASSERT_OK_PTR(arena, "arena"))
- goto out;
- if (pin) {
- ASSERT_OK(bpf_map__set_pin_path(arena, "/sys/fs/bpf/data_in_arena"), "pin_path");
- } else {
- fd = bpf_map_create(BPF_MAP_TYPE_ARENA, "arena", 0, 0, 1, &opts);
- if (!ASSERT_GE(fd, 0, "map_create"))
- goto out;
- ASSERT_OK(bpf_map__reuse_fd(arena, fd), "reuse_fd");
- }
- ASSERT_EQ(data_in_arena__load(skel), -ENOTSUP, "load");
-out:
- if (fd >= 0)
- close(fd);
- data_in_arena__destroy(skel);
-}
-
-/* The object is there if rustc and clang can build it, see Makefile.buildvars */
-static void test_rust(void)
-{
- const char *file = "./data_in_arena_rust.bpf.o";
- struct bpf_object *obj;
-
- if (access(file, R_OK)) {
- test__skip();
- return;
- }
- obj = bpf_object__open_file(file, NULL);
- if (!ASSERT_OK_PTR(obj, "open"))
- return;
- if (!ASSERT_OK(bpf_object__load(obj), "load"))
- goto out;
- /* libbpf goes on without BTF when the kernel doesn't take it */
- ASSERT_GE(bpf_object__btf_fd(obj), 0, "btf_fd");
- ASSERT_EQ(run_prog(bpf_object__find_program_by_name(obj, "list_in_data")),
- 100 + 20 + 3, "retval");
-out:
- bpf_object__close(obj);
-}
-
-void test_data_in_arena(void)
-{
-#if !defined(__x86_64__) && !defined(__aarch64__)
- /* other JITs don't take BPF_F_ARENA_SCALAR */
- test__skip();
- return;
-#endif
- if (test__start_subtest("arena"))
- test_in_arena();
- if (test__start_subtest("declared_arena"))
- test_declared_arena();
- if (test__start_subtest("ptr_to_map"))
- test_ptr_to_map();
- if (test__start_subtest("ptr_to_extern"))
- test_ptr_to_extern();
- if (test__start_subtest("arena_var_no_map"))
- test_arena_var_no_map();
- if (test__start_subtest("pinned_arena"))
- test_not_my_arena(true);
- if (test__start_subtest("reused_arena"))
- test_not_my_arena(false);
- if (test__start_subtest("rust"))
- test_rust();
-}
diff --git a/tools/testing/selftests/bpf/progs/data_in_arena.c b/tools/testing/selftests/bpf/progs/data_in_arena.c
deleted file mode 100644
index 18f38103c0fc..000000000000
--- a/tools/testing/selftests/bpf/progs/data_in_arena.c
+++ /dev/null
@@ -1,112 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-#include <linux/bpf.h>
-#include <bpf/bpf_helpers.h>
-
-/* global data of the object is in arena */
-char data_in_arena SEC(".arena.data");
-
-int counter = 5;
-long pair[2] = { 1, 2 };
-int x = 42;
-long sum;
-long table[4];
-struct {
- long v[8];
-} aligned64 __attribute__((aligned(64)));
-const volatile int ro = 7;
-const volatile long ro_table[4] = { 10, 20, 30, 40 };
-
-/* const strings stay in a map for helpers and kfuncs, a copy of them is in arena */
-const char hello[] SEC(".rodata.str.hello") = "hello";
-
-/* pointers to data that are stored in data */
-int *px = &x;
-const char *str = hello;
-int *const volatile cpx SEC(".data.rel.ro") = &x;
-
-SEC("syscall")
-int use_data(void *ctx)
-{
- int i;
-
- for (i = 0; i < 4; i++)
- table[i] = ro_table[i] + i;
- sum = counter + ro + pair[0] + pair[1];
- counter++;
- __sync_fetch_and_add(&pair[1], 3);
- return sum;
-}
-
-typedef int (*op_fn)(int);
-
-static __noinline int add1(int v)
-{
- return v + 1;
-}
-
-/*
- * Pointers to functions and to data in read-only data of a program with callx.
- * Volatile, so that the compiler doesn't replace the pointers with what
- * they point to.
- */
-static const volatile struct {
- op_fn fn;
- int *data;
- const char *name;
-} ops SEC(".data.rel.ro") = { add1, &x, hello };
-
-SEC("syscall")
-int use_ops(void *ctx)
-{
- /* a program has the arena when its code refers to it */
- counter++;
-#ifdef __clang__
- return ops.fn(*ops.data) + ops.name[1];
-#else
- /* gcc doesn't support indirect calls */
- return add1(*ops.data) + ops.name[1];
-#endif
-}
-
-SEC("syscall")
-int use_ptrs(void *ctx)
-{
- unsigned long addr = (unsigned long)&aligned64;
- char local[4] = "abc";
-
- /* hide the address from the compiler, it knows that '& 63' is 0 */
- asm volatile ("" : "+r"(addr));
- if (addr & 63)
- return 1;
- aligned64.v[7] = 7;
- if (*px != 42)
- return 2;
- *px = 43;
- if (x != 43 || *cpx != 43)
- return 3;
- if (str[0] != 'h' || str[4] != 'o' || str[5])
- return 4;
- /* the literal is in a map */
- if (bpf_strncmp(local, sizeof(local), "abc"))
- return 5;
- return 0;
-}
-
-char out[16];
-
-/* format strings are in a map */
-SEC("syscall")
-int use_printk(void *ctx)
-{
- char buf[sizeof(out)];
- int i, n;
-
- bpf_printk("counter %d", counter);
- n = BPF_SNPRINTF(buf, sizeof(buf), "%d-%d", x, ro);
- for (i = 0; i < sizeof(out); i++)
- out[i] = buf[i];
- return n;
-}
-
-char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/data_in_arena_decl.c b/tools/testing/selftests/bpf/progs/data_in_arena_decl.c
deleted file mode 100644
index 01bc4fea8f0c..000000000000
--- a/tools/testing/selftests/bpf/progs/data_in_arena_decl.c
+++ /dev/null
@@ -1,37 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-#define BPF_NO_KFUNC_PROTOTYPES
-#include <vmlinux.h>
-#include <bpf/bpf_helpers.h>
-#include "bpf_experimental.h"
-#include <bpf_arena_common.h>
-
-struct {
- __uint(type, BPF_MAP_TYPE_ARENA);
- __uint(map_flags, BPF_F_MMAPABLE);
- __uint(max_entries, 4);
-} arena SEC(".maps");
-
-/* global data of the object is in arena */
-char data_in_arena SEC(".arena.data");
-
-int counter = 5;
-long sum;
-const volatile int ro = 7;
-
-#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
-int __arena avar = 11;
-#else
-int avar = 11;
-#endif
-
-SEC("syscall")
-int use_data(void *ctx)
-{
- sum = counter + ro + avar;
- counter++;
- avar++;
- return sum;
-}
-
-char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/data_in_arena_extern.c b/tools/testing/selftests/bpf/progs/data_in_arena_extern.c
deleted file mode 100644
index dc1ad5ca3bdd..000000000000
--- a/tools/testing/selftests/bpf/progs/data_in_arena_extern.c
+++ /dev/null
@@ -1,20 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-#include <linux/bpf.h>
-#include <bpf/bpf_helpers.h>
-
-/* global data of the object is in arena */
-char data_in_arena SEC(".arena.data");
-
-extern const int bpf_prog_active __ksym;
-
-/* the variable of the kernel is not in arena */
-const void *kp = &bpf_prog_active;
-
-SEC("syscall")
-int ptr_to_extern(void *ctx)
-{
- return *(int *)kp;
-}
-
-char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/data_in_arena_fail.c b/tools/testing/selftests/bpf/progs/data_in_arena_fail.c
deleted file mode 100644
index ad8afe9afc96..000000000000
--- a/tools/testing/selftests/bpf/progs/data_in_arena_fail.c
+++ /dev/null
@@ -1,20 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-#include <linux/bpf.h>
-#include <bpf/bpf_helpers.h>
-
-/* global data of the object is in arena */
-char data_in_arena SEC(".arena.data");
-
-int x = 42;
-/* the table is read-only data with pointers. It's not in arena. */
-int *const volatile tbl[1] SEC(".data.rel.ro") = { &x };
-int *const volatile *pp = tbl;
-
-SEC("syscall")
-int ptr_to_map(void *ctx)
-{
- return **pp;
-}
-
-char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/data_in_arena_nomap.c b/tools/testing/selftests/bpf/progs/data_in_arena_nomap.c
deleted file mode 100644
index ccf66321eb36..000000000000
--- a/tools/testing/selftests/bpf/progs/data_in_arena_nomap.c
+++ /dev/null
@@ -1,20 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-#include <vmlinux.h>
-#include <bpf/bpf_helpers.h>
-#include "bpf_arena_common.h"
-
-/* global data of the object is in arena */
-char data_in_arena SEC(".arena.data");
-
-int counter = 5;
-/* needs an arena map that is declared. The one that libbpf creates won't do. */
-int __arena_global avar = 11;
-
-SEC("syscall")
-int arena_var(void *ctx)
-{
- return avar + counter;
-}
-
-char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/data_in_arena_rust.rs b/tools/testing/selftests/bpf/progs/data_in_arena_rust.rs
deleted file mode 100644
index 3dd9d4208232..000000000000
--- a/tools/testing/selftests/bpf/progs/data_in_arena_rust.rs
+++ /dev/null
@@ -1,73 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-// Why .data, .bss and .rodata of a program in Rust are in arena.
-//
-// A reference in Rust is an address. It doesn't say what it points to and it
-// can be stored in data. The list below has a node in each of the sections.
-// The nodes are linked by references that are in the data:
-// IN_BSS.next is stored by the program,
-// IN_DATA.next is a relocation in .data against .rodata.
-// sum() loads the references back and reads the three nodes with the same insn.
-//
-// When the sections are array maps libbpf skips the relocation in .data, and
-// what sum() loads from a node is a number that can't be dereferenced:
-// R1 invalid mem access 'scalar'
-// In arena the address of a node is a number to begin with.
-
-#![no_std]
-#![no_main]
-
-// Tell libbpf to keep .data, .bss and .rodata in arena.
-#[used]
-#[link_section = ".arena.data"]
-static DATA_IN_ARENA: u8 = 0;
-
-#[used]
-#[link_section = "license"]
-static LICENSE: [u8; 4] = *b"GPL\0";
-
-// panic=abort is a stop gap until panic=unwind is supported.
-// Nothing here panics, so the handler is not a part of the program.
-#[panic_handler]
-fn panic(_info: &core::panic::PanicInfo) -> ! {
- loop {}
-}
-
-pub struct Node {
- val: u32,
- next: Option<&'static Node>,
-}
-
-// no_mangle makes them visible outside, so LLVM can't fold the list into a constant.
-#[no_mangle]
-static IN_RODATA: Node = Node { val: 3, next: None };
-#[no_mangle]
-static mut IN_DATA: Node = Node {
- val: 20,
- next: Some(&IN_RODATA),
-};
-#[no_mangle]
-static mut IN_BSS: Node = Node { val: 0, next: None };
-
-#[inline(never)]
-fn sum(mut node: Option<&Node>) -> u32 {
- let mut sum = 0;
- // The verifier wants a bound.
- for _ in 0..8 {
- let Some(n) = node else { break };
- sum += n.val;
- node = n.next;
- }
- sum
-}
-
-#[no_mangle]
-#[link_section = "syscall"]
-pub extern "C" fn list_in_data(_ctx: *mut u8) -> u32 {
- unsafe {
- let head = &mut *&raw mut IN_BSS;
- head.val = 100;
- head.next = Some(&*&raw const IN_DATA);
- sum(Some(head))
- }
-}
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 02/81] libbpf: Remove .arena.data
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 01/81] selftests/bpf: Remove tests for global data in arena Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 03/81] libbpf: Keep format strings of bpf_printk() in .rodata Alexei Starovoitov
` (80 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Rust has no address spaces. Thus a global variable cannot have the __arena
attribute. The .arena.data section was a workaround. It told libbpf to do
two things:
- Move .data, .bss and .rodata into arena.
- Load the progs with BPF_F_ARENA_SCALAR.
The compiler must select the location of a variable, not libbpf. libbpf
already puts the .addr_space.1 section in arena. The __arena variables of C
are in this section, and LLVM can use it for Rust too. Until LLVM does
that, the global data of Rust progs stays in array maps.
This reverts commit 8ab58ed42b08 ("libbpf: Keep global data in arena when
the object has .arena.data"). libbpf 1.8.0 does not have the section.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/lib/bpf/libbpf.c | 433 ++---------------------------------------
1 file changed, 20 insertions(+), 413 deletions(-)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 8fcb8cf39cc0..fb8a1f07c5b6 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -552,26 +552,6 @@ struct bpf_struct_ops {
#define STRUCT_OPS_SEC ".struct_ops"
#define STRUCT_OPS_LINK_SEC ".struct_ops.link"
#define ARENA_SEC ".addr_space.1"
-/*
- * An object with this section keeps its global data in arena instead of
- * array maps. What is in the section doesn't matter. .data, .bss and .rodata
- * sections become a part of the arena map of the object, like __arena global
- * variables are. If the object doesn't declare an arena map libbpf creates
- * one that is just large enough for the data. bpf_object__find_map_by_name()
- * finds it by the name "arena" and bpf_map__set_max_entries() changes its
- * size before the object is loaded.
- * Read-only sections with pointers to functions and sections with constant
- * strings stay frozen array maps. A constant that a helper or a kfunc takes
- * by pointer has to be in such section.
- * Pointers to data that are stored in data become addresses of arena.
- * The load fails if such pointer points to a section that is not in arena,
- * or if the arena is pinned or reused.
- *
- * It's for programs written in Rust. There are no address spaces in Rust,
- * the program accesses arena through plain numbers, so programs of the object
- * are loaded with BPF_F_ARENA_SCALAR.
- */
-#define ARENA_DATA_SEC ".arena.data"
enum libbpf_map_type {
LIBBPF_MAP_UNSPEC,
@@ -627,13 +607,6 @@ struct bpf_map {
/* pointers to functions in the data of an internal map, see obj->func_ptrs */
struct func_ptr *func_ptrs;
size_t func_ptr_cnt;
- /*
- * Data of the internal map is a part of arena data at arena_off.
- * The map is not created, unless it's a copy that is in arena.
- */
- bool in_arena;
- bool arena_copy;
- size_t arena_off;
};
enum extern_type {
@@ -808,17 +781,6 @@ struct bpf_object {
void *arena_data;
size_t arena_data_sz;
size_t arena_data_off;
- bool data_in_arena;
- bool arena_mapped;
- /* pointers to data in the data of internal maps, when data is in arena */
- struct data_ptr {
- int sec_idx;
- size_t sec_off;
- int targ_sec_idx;
- size_t targ_off;
- char *sym_name;
- } *data_ptrs;
- size_t data_ptr_cnt;
void *jumptables_data;
size_t jumptables_data_sz;
@@ -1644,7 +1606,6 @@ static struct bpf_object *bpf_object__new(const char *path,
obj->efile.obj_buf = obj_buf;
obj->efile.obj_buf_sz = obj_buf_sz;
obj->efile.btf_maps_shndx = -1;
- obj->efile.arena_data_shndx = -1;
obj->kconfig_map_idx = -1;
obj->arena_map_idx = -1;
@@ -3126,136 +3087,6 @@ static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map,
return 0;
}
-static bool map_is_const_str(const struct bpf_map *map)
-{
- return map->libbpf_type == LIBBPF_MAP_RODATA &&
- str_has_pfx(map->real_name, RODATA_SEC ".str");
-}
-
-static bool map_data_goes_to_arena(const struct bpf_object *obj, const struct bpf_map *map)
-{
- int i;
-
- switch (map->libbpf_type) {
- case LIBBPF_MAP_DATA:
- case LIBBPF_MAP_BSS:
- return true;
- case LIBBPF_MAP_RODATA:
- break;
- default:
- return false;
- }
-
- /*
- * Const strings that kfuncs and helpers take have to be in a frozen
- * map. Pointers to them that are stored in data point to a copy of
- * the strings that is in arena.
- */
- if (map_is_const_str(map))
- return true;
- /*
- * Read-only data with pointers stays a frozen map: the kernel
- * recognizes pointers to functions in it.
- */
- if (str_has_pfx(map->real_name, DATA_REL_RO_SEC))
- return false;
- for (i = 0; i < obj->efile.sec_cnt; i++) {
- const struct elf_sec_desc *sec = &obj->efile.secs[i];
-
- if (sec->sec_type == SEC_RELO && sec->shdr->sh_info == map->sec_idx)
- return false;
- }
- return true;
-}
-
-/* Returns the offset of the data of the map in arena data that is sz bytes so far */
-static size_t map_arena_off(const struct bpf_object *obj, const struct bpf_map *map, size_t sz)
-{
- size_t align = obj->efile.secs[map->sec_idx].shdr->sh_addralign;
-
- return roundup(sz, max(align, sizeof(__u64)));
-}
-
-/*
- * Make .data, .bss and .rodata a part of arena data. They follow __arena
- * variables of the object, if there are any.
- */
-static int bpf_object__init_arena_data(struct bpf_object *obj)
-{
- const size_t page_sz = sysconf(_SC_PAGE_SIZE);
- size_t sz = obj->arena_data_sz;
- struct bpf_map *map;
- void *data;
- int i;
-
- if (!obj->data_in_arena)
- return 0;
-
- for (i = 0; i < obj->nr_maps; i++) {
- map = &obj->maps[i];
- if (map_data_goes_to_arena(obj, map))
- sz = map_arena_off(obj, map, sz) + map->def.value_size;
- }
- if (sz == obj->arena_data_sz)
- return 0;
-
- if (obj->arena_map_idx < 0) {
- /* bpf_object__init_user_btf_maps() didn't look: there is no .maps section */
- if (obj->efile.arena_data) {
- pr_warn("elf: sec '%s': to use global __arena variables the ARENA map should be explicitly declared in SEC(\".maps\")\n",
- ARENA_SEC);
- return -ENOENT;
- }
-
- map = bpf_object__add_map(obj);
- if (IS_ERR(map))
- return PTR_ERR(map);
-
- map->real_name = strdup("arena");
- map->name = strdup("arena");
- if (!map->real_name || !map->name) {
- zfree(&map->real_name);
- zfree(&map->name);
- return -ENOMEM;
- }
- map->sec_idx = -1;
- map->def.type = BPF_MAP_TYPE_ARENA;
- map->def.max_entries = roundup(sz, page_sz) / page_sz;
- map->def.map_flags = BPF_F_MMAPABLE;
- obj->arena_map_idx = map - obj->maps;
- }
-
- data = realloc(obj->arena_data, sz);
- if (!data)
- return -ENOMEM;
- memset(data + obj->arena_data_sz, 0, sz - obj->arena_data_sz);
- sz = obj->arena_data_sz;
- obj->arena_data = data;
- obj->maps[obj->arena_map_idx].mmaped = data;
-
- for (i = 0; i < obj->nr_maps; i++) {
- map = &obj->maps[i];
- if (!map_data_goes_to_arena(obj, map))
- continue;
- sz = map_arena_off(obj, map, sz);
- map->arena_off = sz;
- memcpy(data + sz, map->mmaped, map->def.value_size);
- sz += map->def.value_size;
- pr_debug("map '%s': data is in arena at offset %zu\n", map->name, map->arena_off);
- if (map_is_const_str(map)) {
- map->arena_copy = true;
- continue;
- }
- munmap(map->mmaped, bpf_map_mmap_sz(map));
- /* make bpf_map__initial_value() and bpf_map__set_initial_value() work */
- map->mmaped = data + map->arena_off;
- map->autocreate = false;
- map->in_arena = true;
- }
- obj->arena_data_sz = sz;
- return 0;
-}
-
static int bpf_object__init_user_btf_maps(struct bpf_object *obj, bool strict,
const char *pin_root_path)
{
@@ -3348,7 +3179,6 @@ static int bpf_object__init_maps(struct bpf_object *obj,
err = err ?: bpf_object__init_global_data_maps(obj);
err = err ?: bpf_object__init_kconfig_map(obj);
err = err ?: bpf_object_init_struct_ops(obj);
- err = err ?: bpf_object__init_arena_data(obj);
return err;
}
@@ -4163,11 +3993,6 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
if (!sh)
return -LIBBPF_ERRNO__FORMAT;
- /* the second pass has to know it when it sees relocations of data */
- name = elf_sec_str(obj, sh->sh_name);
- if (name && strcmp(name, ARENA_DATA_SEC) == 0)
- obj->data_in_arena = true;
-
if (sh->sh_type == SHT_SYMTAB) {
if (obj->efile.symbols) {
pr_warn("elf: multiple symbol tables in %s\n", obj->path);
@@ -4284,8 +4109,6 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
} else if (strcmp(name, ARENA_SEC) == 0) {
obj->efile.arena_data = data;
obj->efile.arena_data_shndx = idx;
- } else if (strcmp(name, ARENA_DATA_SEC) == 0) {
- /* the marker, see the first pass */
} else if (strcmp(name, JUMPTABLES_SEC) == 0) {
obj->jumptables_data = malloc(data->d_size);
if (!obj->jumptables_data)
@@ -4310,9 +4133,6 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
* have pointers to functions.
*/
if (!section_have_execinstr(obj, targ_sec_idx) &&
- !(obj->data_in_arena &&
- (!strcmp(name, ".rel" DATA_SEC) ||
- str_has_pfx(name, ".rel" DATA_SEC "."))) &&
strcmp(name, ".rel" RODATA_SEC) &&
!str_has_pfx(name, ".rel" RODATA_SEC ".") &&
strcmp(name, ".rel" DATA_REL_RO_SEC) &&
@@ -5064,10 +4884,6 @@ static int bpf_program__record_reloc(struct bpf_program *prog,
reloc_desc->insn_idx = insn_idx;
reloc_desc->map_idx = map_idx;
reloc_desc->sym_off = sym->st_value;
- if (map->in_arena) {
- reloc_desc->map_idx = obj->arena_map_idx;
- reloc_desc->sym_off += map->arena_off;
- }
return 0;
}
@@ -5286,9 +5102,6 @@ int bpf_map__set_autocreate(struct bpf_map *map, bool autocreate)
if (map_is_created(map))
return libbpf_err(-EBUSY);
- if (map->in_arena)
- return libbpf_err(-EOPNOTSUPP);
-
map->autocreate = autocreate;
return 0;
}
@@ -5569,45 +5382,6 @@ bpf_object__reuse_map(struct bpf_map *map)
return 0;
}
-static struct bpf_map *bpf_object__sec_map(struct bpf_object *obj, int sec_idx)
-{
- int i;
-
- for (i = 0; i < obj->nr_maps; i++)
- if (bpf_map__is_internal(&obj->maps[i]) && obj->maps[i].sec_idx == sec_idx)
- return &obj->maps[i];
- return NULL;
-}
-
-static int bpf_object__mmap_arena(struct bpf_object *obj, struct bpf_map *map)
-{
- int i, err;
-
- map->mmaped = mmap((void *)(long)map->map_extra,
- bpf_map_mmap_sz(map), PROT_READ | PROT_WRITE,
- map->map_extra ? MAP_SHARED | MAP_FIXED : MAP_SHARED,
- map->fd, 0);
- if (map->mmaped == MAP_FAILED) {
- err = -errno;
- map->mmaped = NULL;
- pr_warn("map '%s': failed to mmap arena: %s\n",
- map->name, errstr(err));
- return err;
- }
- if (obj->arena_data) {
- memcpy(map->mmaped + obj->arena_data_off, obj->arena_data,
- obj->arena_data_sz);
- zfree(&obj->arena_data);
- }
- for (i = 0; i < obj->nr_maps; i++) {
- struct bpf_map *m = &obj->maps[i];
-
- if (m->in_arena)
- m->mmaped = map->mmaped + obj->arena_data_off + m->arena_off;
- }
- return 0;
-}
-
static int
bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
{
@@ -5968,10 +5742,6 @@ bpf_object__create_maps(struct bpf_object *obj)
continue;
}
- /* see bpf_object__relocate_data_ptrs() */
- if (map->def.type == BPF_MAP_TYPE_ARENA && obj->arena_mapped)
- continue;
-
err = map_set_def_max_entries(map);
if (err)
goto err_out;
@@ -6009,9 +5779,22 @@ bpf_object__create_maps(struct bpf_object *obj)
if (err < 0)
goto err_out;
} else if (map->def.type == BPF_MAP_TYPE_ARENA) {
- err = bpf_object__mmap_arena(obj, map);
- if (err)
+ map->mmaped = mmap((void *)(long)map->map_extra,
+ bpf_map_mmap_sz(map), PROT_READ | PROT_WRITE,
+ map->map_extra ? MAP_SHARED | MAP_FIXED : MAP_SHARED,
+ map->fd, 0);
+ if (map->mmaped == MAP_FAILED) {
+ err = -errno;
+ map->mmaped = NULL;
+ pr_warn("map '%s': failed to mmap arena: %s\n",
+ map->name, errstr(err));
return err;
+ }
+ if (obj->arena_data) {
+ memcpy(map->mmaped + obj->arena_data_off, obj->arena_data,
+ obj->arena_data_sz);
+ zfree(&obj->arena_data);
+ }
}
if (map->init_slots_sz && map->def.type != BPF_MAP_TYPE_PROG_ARRAY) {
err = init_map_in_map_slots(obj, map);
@@ -8026,81 +7809,6 @@ static int bpf_program_fixup_func_info(struct bpf_object *obj, struct bpf_progra
return err;
}
-/*
- * Turn pointers to data that are stored in data into addresses of arena.
- * The arena is created here, ahead of the other maps: the address has to be
- * known before the maps that hold the pointers are created.
- */
-static int bpf_object__relocate_data_ptrs(struct bpf_object *obj)
-{
- struct bpf_map *arena, *map, *targ;
- __u64 addr, val;
- size_t i;
- int err;
-
- if (!obj->data_ptr_cnt)
- return 0;
-
- for (i = 0; i < obj->data_ptr_cnt; i++) {
- struct data_ptr *p = &obj->data_ptrs[i];
-
- if (p->targ_sec_idx < 0)
- continue;
- map = bpf_object__sec_map(obj, p->sec_idx);
- targ = bpf_object__sec_map(obj, p->targ_sec_idx);
- if (map && (map->in_arena || map->autocreate) &&
- (!targ || (!targ->in_arena && !targ->arena_copy))) {
- pr_warn("sec '%s': pointer to '%s' at offset %zu can't be resolved: sec '%s' is not in arena\n",
- map->real_name, p->sym_name, p->sec_off,
- targ ? targ->real_name : "<?>");
- return -LIBBPF_ERRNO__RELOC;
- }
- }
-
- if (obj->gen_loader) {
- pr_warn("pointers to data in data are not supported by light skeleton\n");
- return -ENOTSUP;
- }
-
- if (obj->arena_map_idx < 0)
- return 0;
-
- arena = &obj->maps[obj->arena_map_idx];
- if (!arena->autocreate)
- return 0;
- /* libbpf doesn't mmap an arena that it didn't create, so the address is not known */
- if (arena->reused || arena->pin_path) {
- pr_warn("map '%s': pointers to data in data are not supported with pinned or reused arena\n",
- arena->name);
- return -ENOTSUP;
- }
-
- err = bpf_object__create_map(obj, arena, false);
- err = err ?: bpf_object__mmap_arena(obj, arena);
- if (err) {
- pr_warn("map '%s': failed to create: %s\n", arena->name, errstr(err));
- return err;
- }
- obj->arena_mapped = true;
-
- for (i = 0; i < obj->data_ptr_cnt; i++) {
- struct data_ptr *p = &obj->data_ptrs[i];
-
- map = bpf_object__sec_map(obj, p->sec_idx);
- if (!map || (!map->in_arena && !map->autocreate))
- continue;
-
- addr = (__u64)(unsigned long)arena->mmaped + obj->arena_data_off + p->targ_off;
- if (p->targ_sec_idx >= 0)
- addr += bpf_object__sec_map(obj, p->targ_sec_idx)->arena_off;
-
- memcpy(&val, map->mmaped + p->sec_off, sizeof(val));
- val += addr;
- memcpy(map->mmaped + p->sec_off, &val, sizeof(val));
- }
- return 0;
-}
-
static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_path)
{
struct bpf_program *prog;
@@ -8123,10 +7831,8 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
size_t mmap_sz = bpf_map_mmap_sz(arena_map);
if (data_sz > mmap_sz) {
- pr_warn("map '%s': declared ARENA map size (%zu) is too small to hold global %s of size %zu\n",
- arena_map->name, mmap_sz,
- obj->data_in_arena ? "data" : "__arena variables",
- obj->arena_data_sz);
+ pr_warn("map '%s': declared ARENA map size (%zu) is too small to hold global __arena variables of size %zu\n",
+ arena_map->name, mmap_sz, obj->arena_data_sz);
return -E2BIG;
}
@@ -8135,10 +7841,6 @@ static int bpf_object__relocate(struct bpf_object *obj, const char *targ_btf_pat
obj->arena_data_off = mmap_sz - data_sz;
}
- err = bpf_object__relocate_data_ptrs(obj);
- if (err)
- return err;
-
/* Before relocating calls pre-process relocations and mark
* few ld_imm64 instructions that points to subprogs.
* Otherwise bpf_object__reloc_code() later would have to consider
@@ -8371,85 +8073,12 @@ static int bpf_object__collect_map_relos(struct bpf_object *obj,
return 0;
}
-/*
- * A pointer to data that is stored in data. When data is in arena
- * the program dereferences what it loads from data, so the pointer has
- * to be the address of the target in arena.
- */
-static int bpf_object__collect_data_ptr(struct bpf_object *obj, const char *relo_sec_name,
- int relo_idx, size_t sec_idx, const Elf64_Rel *rel,
- const Elf64_Sym *sym)
-{
- Elf_Data *scn_data = obj->efile.secs[sec_idx].data;
- const char *sym_name = elf_sym_str(obj, sym->st_name) ?: "<?>";
- struct data_ptr *ptrs;
-
- if (ELF64_ST_TYPE(sym->st_info) == STT_SECTION && sym->st_shndx < obj->efile.sec_cnt)
- sym_name = elf_sec_name(obj, elf_sec_by_idx(obj, sym->st_shndx)) ?: "<?>";
-
- if (ELF64_R_TYPE(rel->r_info) != R_BPF_64_ABS64 ||
- sym->st_shndx >= obj->efile.sec_cnt ||
- (sym->st_shndx != obj->efile.arena_data_shndx &&
- !bpf_object__shndx_is_data(obj, sym->st_shndx)) ||
- rel->r_offset >= scn_data->d_size ||
- scn_data->d_size - rel->r_offset < sizeof(__u64)) {
- pr_warn("sec '%s': relo #%d: can't resolve pointer to '%s' at offset %zu when data is in arena\n",
- relo_sec_name, relo_idx, sym_name, (size_t)rel->r_offset);
- return -LIBBPF_ERRNO__RELOC;
- }
-
- ptrs = libbpf_reallocarray(obj->data_ptrs, obj->data_ptr_cnt + 1, sizeof(*ptrs));
- if (!ptrs)
- return -ENOMEM;
- obj->data_ptrs = ptrs;
-
- ptrs[obj->data_ptr_cnt].sec_idx = sec_idx;
- ptrs[obj->data_ptr_cnt].sec_off = rel->r_offset;
- /* __arena variables are at the start of arena data */
- ptrs[obj->data_ptr_cnt].targ_sec_idx =
- sym->st_shndx == obj->efile.arena_data_shndx ? -1 : sym->st_shndx;
- ptrs[obj->data_ptr_cnt].targ_off = sym->st_value;
- ptrs[obj->data_ptr_cnt].sym_name = strdup(sym_name);
- if (!ptrs[obj->data_ptr_cnt].sym_name)
- return -ENOMEM;
- obj->data_ptr_cnt++;
-
- pr_debug("sec '%s': relo #%d: pointer at offset %zu to '%s'\n",
- relo_sec_name, relo_idx, (size_t)rel->r_offset, sym_name);
- return 0;
-}
-
-/* Collect pointers in a data section that went to arena */
-static int bpf_object__collect_data_relos(struct bpf_object *obj,
- Elf64_Shdr *shdr, Elf_Data *data)
-{
- int i, err, nrels = shdr->sh_size / shdr->sh_entsize;
- const char *relo_sec_name;
- Elf64_Sym *sym;
- Elf64_Rel *rel;
-
- relo_sec_name = elf_sec_str(obj, shdr->sh_name) ?: "<?>";
- for (i = 0; i < nrels; i++) {
- rel = elf_rel_by_idx(data, i);
- sym = rel ? elf_sym_by_idx(obj, ELF64_R_SYM(rel->r_info)) : NULL;
- if (!sym) {
- pr_warn("sec '%s': failed to get relo #%d\n", relo_sec_name, i);
- return -LIBBPF_ERRNO__FORMAT;
- }
- err = bpf_object__collect_data_ptr(obj, relo_sec_name, i, shdr->sh_info, rel, sym);
- if (err)
- return err;
- }
- return 0;
-}
-
/*
* Collect pointers to functions in a read-only data section. They are
* R_BPF_64_ABS64 relocations against .text section, where the offset of
* a static function in the section is stored in place. Relocations in data
* sections were ignored before pointers to functions were supported. Those
- * that are something else, e.g. pointers to data, still are, unless data
- * is in arena.
+ * that are something else, e.g. pointers to data, still are.
*/
static int bpf_object__collect_rodata_relos(struct bpf_object *obj,
Elf64_Shdr *shdr, Elf_Data *data)
@@ -8485,15 +8114,6 @@ static int bpf_object__collect_rodata_relos(struct bpf_object *obj,
if (ELF64_R_TYPE(rel->r_info) != R_BPF_64_ABS64 ||
!sym_is_subprog(sym, obj->efile.text_shndx)) {
- int err;
-
- if (obj->data_in_arena) {
- err = bpf_object__collect_data_ptr(obj, relo_sec_name, i,
- sec_idx, rel, sym);
- if (err)
- return err;
- continue;
- }
pr_debug("sec '%s': relo #%d: not a pointer to a function, skipping...\n",
relo_sec_name, i);
continue;
@@ -8569,8 +8189,6 @@ static int bpf_object__collect_relos(struct bpf_object *obj)
if (obj->efile.secs[idx].sec_type == SEC_RODATA)
err = bpf_object__collect_rodata_relos(obj, shdr, data);
- else if (obj->efile.secs[idx].sec_type == SEC_DATA)
- err = bpf_object__collect_data_relos(obj, shdr, data);
else if (obj->efile.secs[idx].sec_type == SEC_ST_OPS)
err = bpf_object__collect_st_ops_relos(obj, shdr, data);
else if (idx == obj->efile.btf_maps_shndx)
@@ -8914,9 +8532,6 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
}
load_attr.log_level = log_level;
load_attr.prog_flags = prog->prog_flags;
- /* the program accesses its data in arena through plain numbers */
- if (obj->data_in_arena)
- load_attr.prog_flags |= BPF_F_ARENA_SCALAR;
load_attr.fd_array = obj->fd_array;
err = bpf_object__bind_arena(obj, prog, &load_attr);
if (err)
@@ -8991,7 +8606,7 @@ static int bpf_object_load_prog(struct bpf_object *obj, struct bpf_program *prog
for (i = 0; i < obj->nr_maps; i++) {
map = &prog->obj->maps[i];
- if (map->libbpf_type != LIBBPF_MAP_RODATA || map->in_arena)
+ if (map->libbpf_type != LIBBPF_MAP_RODATA)
continue;
if (bpf_prog_bind_map(ret, map->fd, NULL)) {
@@ -10557,7 +10172,7 @@ static void bpf_map__destroy(struct bpf_map *map)
zfree(&map->init_slots);
map->init_slots_sz = 0;
- if (map->mmaped && map->mmaped != map->obj->arena_data && !map->in_arena)
+ if (map->mmaped && map->mmaped != map->obj->arena_data)
munmap(map->mmaped, bpf_map_mmap_sz(map));
map->mmaped = NULL;
@@ -10642,9 +10257,6 @@ void bpf_object__close(struct bpf_object *obj)
close(obj->func_ptr_maps[i].fd);
zfree(&obj->func_ptr_maps);
zfree(&obj->func_ptrs);
- for (i = 0; i < obj->data_ptr_cnt; i++)
- zfree(&obj->data_ptrs[i].sym_name);
- zfree(&obj->data_ptrs);
if (obj->btf_module_allowlist) {
for (i = 0; i < obj->btf_module_allowlist_cnt; i++)
@@ -11011,8 +10623,6 @@ int bpf_program__clone(struct bpf_program *prog, const struct bpf_prog_load_opts
attr.token_fd = OPTS_GET(opts, token_fd, 0) ?: obj->token_fd;
if (attr.token_fd)
attr.prog_flags |= BPF_F_TOKEN_FD;
- if (obj->data_in_arena)
- attr.prog_flags |= BPF_F_ARENA_SCALAR;
prog_btf_fd = OPTS_GET(opts, prog_btf_fd, 0);
if (!prog_btf_fd && obj->btf)
@@ -12019,9 +11629,6 @@ int bpf_map__set_value_size(struct bpf_map *map, __u32 size)
if (map_is_created(map))
return libbpf_err(-EBUSY);
- if (map->in_arena)
- return libbpf_err(-EOPNOTSUPP);
-
if (map->mmaped) {
size_t mmap_old_sz, mmap_new_sz;
int err;
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 03/81] libbpf: Keep format strings of bpf_printk() in .rodata
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 01/81] selftests/bpf: Remove tests for global data in arena Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 02/81] libbpf: Remove .arena.data Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 04/81] selftests/bpf: Remove tests for arena access through numbers Alexei Starovoitov
` (79 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The format strings of bpf_printk() moved from .rodata to .rodata.str for
one reason. .arena.data moved .rodata into arena, and helpers cannot read
strings from arena.
.arena.data is removed. Move the strings back to .rodata. As a result, an
object that has its own .rodata has one map less.
This reverts commit 373e1223ff00
("libbpf: Keep format strings of bpf_printk() in .rodata.str"). libbpf
1.8.0 has the strings in .rodata.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/lib/bpf/bpf_helpers.h | 18 +++++-------------
1 file changed, 5 insertions(+), 13 deletions(-)
diff --git a/tools/lib/bpf/bpf_helpers.h b/tools/lib/bpf/bpf_helpers.h
index b37b727ef216..9d160b5b9c0e 100644
--- a/tools/lib/bpf/bpf_helpers.h
+++ b/tools/lib/bpf/bpf_helpers.h
@@ -248,21 +248,13 @@ enum libbpf_tristate {
#define ___bpf_fill(arr, args...) \
___bpf_apply(___bpf_fill, ___bpf_narg(args))(arr, 0, args)
-/*
- * Format strings are in a section of their own. When global data of the object
- * is in arena (the object has .arena.data section) libbpf keeps .rodata.str*
- * sections in read-only maps, which is where helpers and kfuncs take strings
- * from.
- */
-#define ___bpf_fmt_sec __attribute__((section(".rodata.str")))
-
/*
* BPF_SEQ_PRINTF to wrap bpf_seq_printf to-be-printed values
* in a structure.
*/
#define BPF_SEQ_PRINTF(seq, fmt, args...) \
({ \
- static const char ___fmt[] ___bpf_fmt_sec = fmt; \
+ static const char ___fmt[] = fmt; \
unsigned long long ___param[___bpf_narg(args)]; \
\
_Pragma("GCC diagnostic push") \
@@ -280,7 +272,7 @@ enum libbpf_tristate {
*/
#define BPF_SNPRINTF(out, out_size, fmt, args...) \
({ \
- static const char ___fmt[] ___bpf_fmt_sec = fmt; \
+ static const char ___fmt[] = fmt; \
unsigned long long ___param[___bpf_narg(args)]; \
\
_Pragma("GCC diagnostic push") \
@@ -295,7 +287,7 @@ enum libbpf_tristate {
#ifdef BPF_NO_GLOBAL_DATA
#define BPF_PRINTK_FMT_MOD
#else
-#define BPF_PRINTK_FMT_MOD static const ___bpf_fmt_sec
+#define BPF_PRINTK_FMT_MOD static const
#endif
#define __bpf_printk(fmt, ...) \
@@ -311,7 +303,7 @@ enum libbpf_tristate {
*/
#define __bpf_vprintk(fmt, args...) \
({ \
- static const char ___fmt[] ___bpf_fmt_sec = fmt; \
+ static const char ___fmt[] = fmt; \
unsigned long long ___param[___bpf_narg(args)]; \
\
_Pragma("GCC diagnostic push") \
@@ -325,7 +317,7 @@ enum libbpf_tristate {
#define bpf_stream_printk(stream_id, fmt, args...) \
({ \
- static const char ___fmt[] ___bpf_fmt_sec = fmt; \
+ static const char ___fmt[] = fmt; \
unsigned long long ___param[___bpf_narg(args)]; \
\
_Pragma("GCC diagnostic push") \
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 04/81] selftests/bpf: Remove tests for arena access through numbers
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (2 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 03/81] libbpf: Keep format strings of bpf_printk() in .rodata Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 05/81] bpf: Remove BPF_F_ARENA_SCALAR Alexei Starovoitov
` (78 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
With BPF_F_ARENA_SCALAR the verifier accepts a load or a store through a
scalar as an access to arena. The next patch removes the flag. Remove its
tests.
Keep one test. Without the flag the verifier must reject such a load, also
in a prog that has an arena. Move this test to verifier_arena.c.
The removed tests were the only tests of a special path in the x86 JIT for
the fetching AND/OR/XOR atomics. Only the flag used that path.
arena_atomics tests the usual path.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../bpf/prog_tests/arena_scalar_blinded.c | 21 -
.../selftests/bpf/prog_tests/verifier.c | 2 -
.../selftests/bpf/progs/arena_noarena.c | 15 -
.../selftests/bpf/progs/verifier_arena.c | 15 +
.../bpf/progs/verifier_arena_scalar.c | 1155 -----------------
tools/testing/selftests/bpf/test_loader.c | 2 -
6 files changed, 15 insertions(+), 1195 deletions(-)
delete mode 100644 tools/testing/selftests/bpf/prog_tests/arena_scalar_blinded.c
delete mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_scalar.c
diff --git a/tools/testing/selftests/bpf/prog_tests/arena_scalar_blinded.c b/tools/testing/selftests/bpf/prog_tests/arena_scalar_blinded.c
deleted file mode 100644
index 2ac2e9a652fe..000000000000
--- a/tools/testing/selftests/bpf/prog_tests/arena_scalar_blinded.c
+++ /dev/null
@@ -1,21 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-#include <test_progs.h>
-#include "sysctl_helpers.h"
-#include "verifier_arena_scalar.skel.h"
-
-/* The same tests with constants of the programs blinded */
-void serial_test_arena_scalar_blinded(void)
-{
- const char *harden = "/proc/sys/net/core/bpf_jit_harden";
- char old[16] = {};
-
- if (!is_jit_enabled()) {
- test__skip();
- return;
- }
- if (sysctl_set_or_fail(harden, old, "2"))
- return;
- RUN_TESTS(verifier_arena_scalar);
- sysctl_set_or_fail(harden, NULL, old);
-}
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 04740ba37e91..6a494c381266 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -13,7 +13,6 @@
#include "verifier_and.skel.h"
#include "verifier_arena.skel.h"
#include "verifier_arena_large.skel.h"
-#include "verifier_arena_scalar.skel.h"
#include "verifier_arena_globals1.skel.h"
#include "verifier_arena_globals2.skel.h"
#include "verifier_array_access.skel.h"
@@ -207,7 +206,6 @@ void test_verifier_align(void) { RUN(verifier_align); }
void test_verifier_and(void) { RUN(verifier_and); }
void test_verifier_arena(void) { RUN(verifier_arena); }
void test_verifier_arena_large(void) { RUN(verifier_arena_large); }
-void test_verifier_arena_scalar(void) { RUN(verifier_arena_scalar); }
void test_verifier_arena_globals1(void) { RUN(verifier_arena_globals1); }
void test_verifier_arena_globals2(void) { RUN(verifier_arena_globals2); }
void test_verifier_basic_stack(void) { RUN(verifier_basic_stack); }
diff --git a/tools/testing/selftests/bpf/progs/arena_noarena.c b/tools/testing/selftests/bpf/progs/arena_noarena.c
index edbd40cd5be8..1f46e5aa403b 100644
--- a/tools/testing/selftests/bpf/progs/arena_noarena.c
+++ b/tools/testing/selftests/bpf/progs/arena_noarena.c
@@ -21,19 +21,4 @@ int arena_arg_no_arena(void *ctx)
return 0;
}
-SEC("syscall")
-__flag(BPF_F_ARENA_SCALAR)
-__arch_x86_64
-__arch_arm64
-__description("arena_scalar: no arena, no access through a number")
-__failure __msg("R6 invalid mem access 'scalar'")
-__naked void no_arena(void)
-{
- asm volatile (" \
- r6 = 0x100000000000 ll; \
- r0 = *(u64 *)(r6 + 0); \
- exit; \
-" ::: __clobber_all);
-}
-
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_arena.c b/tools/testing/selftests/bpf/progs/verifier_arena.c
index 1ce9c4b78481..b17a6dfc100d 100644
--- a/tools/testing/selftests/bpf/progs/verifier_arena.c
+++ b/tools/testing/selftests/bpf/progs/verifier_arena.c
@@ -610,6 +610,21 @@ int mixed_arena_scalar_alu64_arena_first(void *ctx)
return 0;
}
+SEC("syscall")
+__description("arena: a number is not an address in arena")
+__failure __msg("R6 invalid mem access 'scalar'")
+__naked void access_through_number(void)
+{
+ asm volatile (" \
+ r1 = %[arena] ll; \
+ r6 = 0x100000000000 ll; \
+ r0 = *(u64 *)(r6 + 0); \
+ exit; \
+" :
+ : __imm_addr(arena)
+ : __clobber_all);
+}
+
SEC("syscall")
__success __retval(0)
int scalar_xor_arena_ptr(void *ctx)
diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_scalar.c b/tools/testing/selftests/bpf/progs/verifier_arena_scalar.c
deleted file mode 100644
index 6f0efd9202ca..000000000000
--- a/tools/testing/selftests/bpf/progs/verifier_arena_scalar.c
+++ /dev/null
@@ -1,1155 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-#include <linux/bpf.h>
-#include <bpf/bpf_helpers.h>
-#include "../../../include/linux/filter.h"
-#include "bpf_misc.h"
-
-void *bpf_arena_alloc_pages(void *map, void *addr, __u32 page_cnt, int node_id,
- __u64 flags) __ksym;
-void bpf_arena_free_pages(void *map, void *ptr, __u32 page_cnt) __ksym;
-
-#ifdef __TARGET_ARCH_arm64
-#define ARENA_VM_START (1ull << 32)
-#else
-#define ARENA_VM_START (1ull << 44)
-#endif
-
-struct {
- __uint(type, BPF_MAP_TYPE_ARENA);
- __uint(map_flags, BPF_F_MMAPABLE);
- __uint(max_entries, 4);
- __ulong(map_extra, ARENA_VM_START);
-} arena SEC(".maps");
-
-struct {
- __uint(type, BPF_MAP_TYPE_HASH);
- __uint(max_entries, 1);
- __type(key, int);
- __type(value, long long);
-} hash SEC(".maps");
-
-/* JITs that take BPF_F_ARENA_SCALAR */
-#define __arena_scalar __flag(BPF_F_ARENA_SCALAR) __arch_x86_64 __arch_arm64
-
-/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
-void __kfunc_btf_root(void)
-{
- bpf_arena_alloc_pages(0, 0, 0, 0, 0);
- bpf_arena_free_pages(0, 0, 0);
-}
-
-/* Tests start with r6 = address of a new page as the user space sees it, a number */
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: load and store of every size")
-__success __retval(0)
-__load_if_JITed()
-__naked void ld_st_sizes(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r7 = r6; \
- r1 = 0x1122334455667788 ll; \
- *(u64 *)(r6 + 0) = r1; \
- *(u32 *)(r6 + 8) = r1; \
- *(u16 *)(r6 + 12) = r1; \
- *(u8 *)(r6 + 14) = r1; \
- *(u64 *)(r6 + 16) = 0x1234; \
- *(u32 *)(r6 + 24) = 0x5678; \
- *(u16 *)(r6 + 28) = 0x9a; \
- *(u8 *)(r6 + 30) = 0xbc; \
- r0 = 1; \
- r2 = *(u64 *)(r6 + 0); \
- if r2 != r1 goto 9f; \
- r0 = 2; \
- r2 = *(u32 *)(r6 + 8); \
- if r2 != 0x55667788 goto 9f; \
- r0 = 3; \
- r2 = *(u16 *)(r6 + 12); \
- if r2 != 0x7788 goto 9f; \
- r0 = 4; \
- r2 = *(u8 *)(r6 + 14); \
- if r2 != 0x88 goto 9f; \
- r0 = 5; \
- r2 = *(u64 *)(r6 + 16); \
- if r2 != 0x1234 goto 9f; \
- r0 = 6; \
- r2 = *(u32 *)(r6 + 24); \
- if r2 != 0x5678 goto 9f; \
- r0 = 7; \
- r2 = *(u16 *)(r6 + 28); \
- if r2 != 0x9a goto 9f; \
- r0 = 8; \
- r2 = *(u8 *)(r6 + 30); \
- if r2 != 0xbc goto 9f; \
- /* the address is what it was */ \
- r0 = 9; \
- if r6 != r7 goto 9f; \
- r0 = 10; \
- r7 >>= 32; \
- if r7 == 0 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: load into the register that holds the address")
-__success __retval(0)
-__load_if_JITed()
-__naked void ld_into_base(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r1 = 77; \
- *(u64 *)(r6 + 0) = r1; \
- r6 = *(u64 *)(r6 + 0); \
- r0 = 1; \
- if r6 != 77 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: sign extending load")
-__success __retval(0)
-__load_if_JITed()
-__naked void ldsx(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r7 = r6; \
- *(u64 *)(r6 + 0) = 0x80; \
- .8byte %[ldsx_insn]; /* r2 = *(s8 *)(r6 + 0) */ \
- r0 = 1; \
- if r2 != -128 goto 9f; \
- r0 = 2; \
- if r6 != r7 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm_insn(ldsx_insn, BPF_RAW_INSN(BPF_LDX | BPF_MEMSX | BPF_B,
- BPF_REG_2, BPF_REG_6, 0, 0))
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: address loaded from arena")
-__success __retval(0)
-__load_if_JITed()
-__naked void ptr_chase(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- /* page[0] = &page[64]; page[64] = 5; */ \
- r1 = r6; \
- r1 += 64; \
- *(u64 *)(r6 + 0) = r1; \
- *(u64 *)(r1 + 0) = 5; \
- r2 = *(u64 *)(r6 + 0); \
- r0 = 1; \
- if r2 != r1 goto 9f; \
- r3 = *(u64 *)(r2 + 0); \
- r0 = 2; \
- if r3 != 5 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: atomics")
-__success __retval(0)
-__load_if_JITed()
-__naked void atomics(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r7 = r6; \
- *(u64 *)(r6 + 8) = 1; \
- r1 = 2; \
- lock *(u64 *)(r6 + 8) += r1; \
- r1 = 4; \
- .8byte %[fetch_add_insn]; /* r1 = atomic_fetch_add((u64 *)(r6 + 8), r1) */ \
- r0 = 1; \
- if r1 != 3 goto 9f; \
- r1 = 8; \
- .8byte %[xchg_insn]; /* r1 = xchg_64(r6 + 8, r1) */ \
- r0 = 2; \
- if r1 != 7 goto 9f; \
- r0 = 8; \
- r1 = 16; \
- .8byte %[cmpxchg_insn]; /* r0 = cmpxchg_64(r6 + 8, r0, r1) */ \
- r2 = r0; \
- r0 = 3; \
- if r2 != 8 goto 9f; \
- r2 = *(u64 *)(r6 + 8); \
- r0 = 4; \
- if r2 != 16 goto 9f; \
- r0 = 5; \
- if r6 != r7 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm_insn(fetch_add_insn, BPF_ATOMIC_OP(BPF_DW, BPF_ADD | BPF_FETCH,
- BPF_REG_6, BPF_REG_1, 8)),
- __imm_insn(xchg_insn, BPF_ATOMIC_OP(BPF_DW, BPF_XCHG, BPF_REG_6, BPF_REG_1, 8)),
- __imm_insn(cmpxchg_insn, BPF_ATOMIC_OP(BPF_DW, BPF_CMPXCHG, BPF_REG_6, BPF_REG_1, 8))
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: fetching and/or/xor")
-__success __retval(0)
-__load_if_JITed()
-__naked void fetch_and_or_xor(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r7 = 0x11000000110 ll; \
- *(u64 *)(r6 + 8) = r7; \
- r0 = r6; \
- r0 ^= 0x55; \
- r1 = 0x1100000011 ll; \
- .8byte %[fetch_or_dw]; /* r1 = atomic_fetch_or((u64 *)(r6 + 8), r1) */ \
- r2 = r6; \
- r2 ^= 0x55; \
- if r0 == r2 goto 1f; \
- r0 = 99; \
- goto 9f; \
-1: r0 = 1; \
- if r1 != r7 goto 9f; \
- r7 = 0x11100000111 ll; \
- r1 = 0x11000000111 ll; \
- .8byte %[fetch_and_dw]; /* r1 = atomic_fetch_and((u64 *)(r6 + 8), r1) */ \
- r0 = 2; \
- if r1 != r7 goto 9f; \
- r7 = 0x11000000111 ll; \
- r1 = 0x101 ll; \
- .8byte %[fetch_xor_dw]; /* r1 = atomic_fetch_xor((u64 *)(r6 + 8), r1) */ \
- r0 = 3; \
- if r1 != r7 goto 9f; \
- r7 = 0x11000000010 ll; \
- r2 = *(u64 *)(r6 + 8); \
- r0 = 4; \
- if r2 != r7 goto 9f; \
- *(u32 *)(r6 + 16) = 0x110; \
- r1 = 0xdead000000000011 ll; \
- .8byte %[fetch_or_w]; /* r1 = atomic_fetch_or((u32 *)(r6 + 16), r1) */ \
- r0 = 5; \
- if r1 != 0x110 goto 9f; \
- r1 = 0xdead000000000101 ll; \
- .8byte %[fetch_and_w]; /* r1 = atomic_fetch_and((u32 *)(r6 + 16), r1) */ \
- r0 = 6; \
- if r1 != 0x111 goto 9f; \
- r1 = 0xdead000000000011 ll; \
- .8byte %[fetch_xor_w]; /* r1 = atomic_fetch_xor((u32 *)(r6 + 16), r1) */ \
- r0 = 7; \
- if r1 != 0x101 goto 9f; \
- r2 = *(u32 *)(r6 + 16); \
- r0 = 8; \
- if r2 != 0x110 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm_insn(fetch_or_dw, BPF_ATOMIC_OP(BPF_DW, BPF_OR | BPF_FETCH,
- BPF_REG_6, BPF_REG_1, 8)),
- __imm_insn(fetch_and_dw, BPF_ATOMIC_OP(BPF_DW, BPF_AND | BPF_FETCH,
- BPF_REG_6, BPF_REG_1, 8)),
- __imm_insn(fetch_xor_dw, BPF_ATOMIC_OP(BPF_DW, BPF_XOR | BPF_FETCH,
- BPF_REG_6, BPF_REG_1, 8)),
- __imm_insn(fetch_or_w, BPF_ATOMIC_OP(BPF_W, BPF_OR | BPF_FETCH,
- BPF_REG_6, BPF_REG_1, 16)),
- __imm_insn(fetch_and_w, BPF_ATOMIC_OP(BPF_W, BPF_AND | BPF_FETCH,
- BPF_REG_6, BPF_REG_1, 16)),
- __imm_insn(fetch_xor_w, BPF_ATOMIC_OP(BPF_W, BPF_XOR | BPF_FETCH,
- BPF_REG_6, BPF_REG_1, 16))
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: fetching and/or/xor with the operand in r0")
-__success __retval(0)
-__load_if_JITed()
-__naked void fetch_and_or_xor_r0(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r7 = 0x11000000110 ll; \
- *(u64 *)(r6 + 8) = r7; \
- r0 = 0x1100000011 ll; \
- .8byte %[fetch_or_dw]; /* r0 = atomic_fetch_or((u64 *)(r6 + 8), r0) */ \
- r2 = r0; \
- r0 = 1; \
- if r2 != r7 goto 9f; \
- r7 = 0x11100000111 ll; \
- r0 = 0x11000000111 ll; \
- .8byte %[fetch_and_dw]; /* r0 = atomic_fetch_and((u64 *)(r6 + 8), r0) */ \
- r2 = r0; \
- r0 = 2; \
- if r2 != r7 goto 9f; \
- r7 = 0x11000000111 ll; \
- r0 = 0x101 ll; \
- .8byte %[fetch_xor_dw]; /* r0 = atomic_fetch_xor((u64 *)(r6 + 8), r0) */ \
- r2 = r0; \
- r0 = 3; \
- if r2 != r7 goto 9f; \
- r7 = 0x11000000010 ll; \
- r2 = *(u64 *)(r6 + 8); \
- r0 = 4; \
- if r2 != r7 goto 9f; \
- *(u32 *)(r6 + 16) = 0x110; \
- r0 = 0xdead000000000011 ll; \
- .8byte %[fetch_or_w]; /* r0 = atomic_fetch_or((u32 *)(r6 + 16), r0) */ \
- r2 = r0; \
- r0 = 5; \
- if r2 != 0x110 goto 9f; \
- r0 = 0xdead000000000101 ll; \
- .8byte %[fetch_and_w]; /* r0 = atomic_fetch_and((u32 *)(r6 + 16), r0) */ \
- r2 = r0; \
- r0 = 6; \
- if r2 != 0x111 goto 9f; \
- r0 = 0xdead000000000011 ll; \
- .8byte %[fetch_xor_w]; /* r0 = atomic_fetch_xor((u32 *)(r6 + 16), r0) */ \
- r2 = r0; \
- r0 = 7; \
- if r2 != 0x101 goto 9f; \
- r2 = *(u32 *)(r6 + 16); \
- r0 = 8; \
- if r2 != 0x110 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm_insn(fetch_or_dw, BPF_ATOMIC_OP(BPF_DW, BPF_OR | BPF_FETCH,
- BPF_REG_6, BPF_REG_0, 8)),
- __imm_insn(fetch_and_dw, BPF_ATOMIC_OP(BPF_DW, BPF_AND | BPF_FETCH,
- BPF_REG_6, BPF_REG_0, 8)),
- __imm_insn(fetch_xor_dw, BPF_ATOMIC_OP(BPF_DW, BPF_XOR | BPF_FETCH,
- BPF_REG_6, BPF_REG_0, 8)),
- __imm_insn(fetch_or_w, BPF_ATOMIC_OP(BPF_W, BPF_OR | BPF_FETCH,
- BPF_REG_6, BPF_REG_0, 16)),
- __imm_insn(fetch_and_w, BPF_ATOMIC_OP(BPF_W, BPF_AND | BPF_FETCH,
- BPF_REG_6, BPF_REG_0, 16)),
- __imm_insn(fetch_xor_w, BPF_ATOMIC_OP(BPF_W, BPF_XOR | BPF_FETCH,
- BPF_REG_6, BPF_REG_0, 16))
- : __clobber_all);
-}
-
-/*
- * The page is gone, so the access faults. The fetched register has to come back
- * 0 and everything else has to be left alone, r0 included. Reading the address
- * again has to give 0 as well: had the fault been recovered by the arena mapping
- * its zeroed scratch page instead of by the JIT's exception table entry, the
- * access would have succeeded and left a 1 there.
- */
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: fetching or through a number to a freed page")
-__success __retval(0)
-__load_if_JITed()
-__naked void fetch_or_freed_page(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- *(u64 *)(r6 + 8) = 0x55; \
- r1 = %[arena] ll; \
- r2 = r6; \
- r3 = 1; \
- call %[bpf_arena_free_pages]; \
- r0 = r6; \
- r0 ^= 0x55; \
- r1 = 1; \
- .8byte %[fetch_or_dw]; /* faults: r1 = atomic_fetch_or((u64 *)(r6 + 8), r1) */ \
- r2 = r6; \
- r2 ^= 0x55; \
- if r0 == r2 goto 1f; \
- r0 = 1; \
- goto 9f; \
-1: r0 = 2; \
- if r1 != 0 goto 9f; \
- r3 = *(u64 *)(r6 + 8); \
- r0 = 3; \
- if r3 != 0 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm(bpf_arena_free_pages),
- __imm_insn(fetch_or_dw, BPF_ATOMIC_OP(BPF_DW, BPF_OR | BPF_FETCH,
- BPF_REG_6, BPF_REG_1, 8))
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: fetching or with the operand in r0 to a freed page")
-__success __retval(0)
-__load_if_JITed()
-__naked void fetch_or_r0_freed_page(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- *(u64 *)(r6 + 8) = 0x55; \
- r1 = %[arena] ll; \
- r2 = r6; \
- r3 = 1; \
- call %[bpf_arena_free_pages]; \
- r0 = 1; \
- .8byte %[fetch_or_dw]; /* faults: r0 = atomic_fetch_or((u64 *)(r6 + 8), r0) */ \
- r2 = r0; \
- r0 = 1; \
- if r2 != 0 goto 9f; \
- r3 = *(u64 *)(r6 + 8); \
- r0 = 2; \
- if r3 != 0 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm(bpf_arena_free_pages),
- __imm_insn(fetch_or_dw, BPF_ATOMIC_OP(BPF_DW, BPF_OR | BPF_FETCH,
- BPF_REG_6, BPF_REG_0, 8))
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: cmpxchg through r0")
-__success __retval(0)
-__load_if_JITed()
-__naked void cmpxchg_r0(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- /* The address is in r0. The value at the address is not equal to it. */ \
- *(u64 *)(r6 + 0) = 3; \
- r0 = r6; \
- r1 = 5; \
- .8byte %[cmpxchg_insn]; /* r0 = cmpxchg_64(r0 + 0, r0, r1) */ \
- r2 = r0; \
- r0 = 1; \
- if r2 != 3 goto 9f; \
- r2 = *(u64 *)(r6 + 0); \
- r0 = 2; \
- if r2 != 3 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm_insn(cmpxchg_insn, BPF_ATOMIC_OP(BPF_DW, BPF_CMPXCHG, BPF_REG_0, BPF_REG_1, 0))
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: xchg into the register that holds the address")
-__success __retval(0)
-__load_if_JITed()
-__naked void xchg_into_base(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- *(u64 *)(r6 + 0) = 3; \
- r1 = r6; \
- .8byte %[xchg_insn]; /* r1 = xchg_64(r1 + 0, r1) */ \
- r0 = 1; \
- if r1 != 3 goto 9f; \
- r2 = *(u64 *)(r6 + 0); \
- r0 = 2; \
- if r2 != r6 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm_insn(xchg_insn, BPF_ATOMIC_OP(BPF_DW, BPF_XCHG, BPF_REG_1, BPF_REG_1, 0))
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: xchg into the register that holds a pointer to stack")
-__failure __msg("misaligned access off (0x0; 0xffffffffffffffff)+0 size 8")
-__naked void xchg_into_stack_ptr(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r1 = 0; \
- *(u64 *)(r10 - 8) = r1; \
- r1 = r10; \
- r1 += -8; \
- .8byte %[xchg_insn]; /* r1 = xchg_64(r1 + 0, r1) */ \
- r0 = 0; \
- exit; \
-" :
- : __imm_addr(arena),
- __imm_insn(xchg_insn, BPF_ATOMIC_OP(BPF_DW, BPF_XCHG, BPF_REG_1, BPF_REG_1, 0))
- : __clobber_all);
-}
-
-#ifdef CAN_USE_LOAD_ACQ_STORE_REL
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: load-acquire and store-release")
-__success __retval(0)
-__load_if_JITed()
-__naked void load_acq_store_rel(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r7 = r6; \
- r1 = 0x1234; \
- .8byte %[store_release_insn]; /* store_release((u64 *)(r6 + 8), r1) */ \
- .8byte %[load_acquire_insn]; /* r2 = load_acquire((u64 *)(r6 + 8)) */ \
- r0 = 1; \
- if r2 != 0x1234 goto 9f; \
- .8byte %[load_acquire8_insn]; /* w2 = load_acquire((u8 *)(r6 + 8)) */ \
- r0 = 2; \
- if r2 != 0x34 goto 9f; \
- r0 = 3; \
- if r6 != r7 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm_insn(store_release_insn,
- BPF_ATOMIC_OP(BPF_DW, BPF_STORE_REL, BPF_REG_6, BPF_REG_1, 8)),
- __imm_insn(load_acquire_insn,
- BPF_ATOMIC_OP(BPF_DW, BPF_LOAD_ACQ, BPF_REG_2, BPF_REG_6, 8)),
- __imm_insn(load_acquire8_insn,
- BPF_ATOMIC_OP(BPF_B, BPF_LOAD_ACQ, BPF_REG_2, BPF_REG_6, 8))
- : __clobber_all);
-}
-
-#endif /* CAN_USE_LOAD_ACQ_STORE_REL */
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: number that is not an address in arena")
-__success __retval(0)
-__load_if_JITed()
-__naked void not_in_arena(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- /* nothing is allocated: all loads read 0, stores are dropped */ \
- r6 = 0xdeadbeef00000000 ll; \
- r0 = 1; \
- r2 = *(u64 *)(r6 + 0); \
- if r2 != 0 goto 9f; \
- r0 = 2; \
- r2 = *(u8 *)(r6 - 32768); \
- if r2 != 0 goto 9f; \
- r6 = 0x12345678ffffffff ll; \
- r0 = 3; \
- r2 = *(u64 *)(r6 + 32760); \
- if r2 != 0 goto 9f; \
- *(u64 *)(r6 + 32760) = 1; \
- *(u8 *)(r6 + 32767) = r2; \
- r1 = 1; \
- lock *(u64 *)(r6 + 32760) += r1; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: store through the address of the stack as a number")
-__success __retval(0)
-__load_if_JITed()
-__naked void stack_addr_as_number(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- *(u64 *)(r10 - 8) = 5; \
- r6 = r10; \
- r6 |= 0; \
- /* r6 is a number now. The store goes to arena, not to the stack. */ \
- *(u64 *)(r6 - 8) = 7; \
- r1 = 9; \
- *(u64 *)(r6 - 8) = r1; \
- lock *(u64 *)(r6 - 8) += r1; \
- r2 = *(u64 *)(r10 - 8); \
- r0 = 1; \
- if r2 != 5 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: 64-bit math on the address stays 64-bit")
-__success __retval(0)
-__load_if_JITed()
-__naked void alu64_after_access(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r2 = *(u64 *)(r6 + 0); \
- r7 = r6; \
- r7 += 8; \
- r7 -= r6; \
- r0 = 1; \
- if r7 != 8 goto 9f; \
- r7 = r6; \
- r7 >>= 32; \
- r0 = 2; \
- if r7 == 0 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: st of an immediate changes no register")
-__success __retval(0)
-__load_if_JITed()
-__naked void st_keeps_regs(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r0 = r6; \
- r1 = 0x1111; \
- r2 = 0x2222; \
- *(u64 *)(r0 + 0) = 5; \
- r3 = r0; \
- r0 = 1; \
- if r3 != r6 goto 9f; \
- r0 = 2; \
- if r1 != 0x1111 goto 9f; \
- r0 = 3; \
- if r2 != 0x2222 goto 9f; \
- r0 = 0x3333; \
- r1 = r6; \
- *(u32 *)(r1 + 8) = -7; \
- r3 = r0; \
- r0 = 4; \
- if r3 != 0x3333 goto 9f; \
- r0 = 5; \
- if r1 != r6 goto 9f; \
- r0 = 6; \
- r3 = *(u64 *)(r6 + 0); \
- if r3 != 5 goto 9f; \
- r0 = 7; \
- r3 = *(u64 *)(r6 + 8); \
- r4 = 0xfffffff9 ll; \
- if r3 != r4 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: access through a number with garbage in the upper half")
-__success __retval(0)
-__load_if_JITed()
-__naked void st_value_garbage(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r1 = 0xdeadbeef00001000 ll; \
- *(u64 *)(r6 + 2048) = r1; \
- r7 = *(u64 *)(r6 + 2048); \
- r8 = *(u64 *)(r6 + 2048); \
- r1 = 3; \
- *(u64 *)(r8 + 0) = 1; \
- r0 = 1; \
- if r8 != r7 goto 9f; \
- *(u8 *)(r8 + 1) = 1; \
- r0 = 2; \
- if r8 != r7 goto 9f; \
- *(u32 *)(r8 + 4) = r1; \
- r0 = 3; \
- if r8 != r7 goto 9f; \
- r2 = *(u16 *)(r8 + 2); \
- r0 = 4; \
- if r8 != r7 goto 9f; \
- r0 = 5; \
- if r1 != 3 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: access through a number without the upper half")
-__success __retval(0)
-__load_if_JITed()
-__naked void st_value_small(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r1 = 0x1000 ll; \
- *(u64 *)(r6 + 2048) = r1; \
- r7 = *(u64 *)(r6 + 2048); \
- r8 = *(u64 *)(r6 + 2048); \
- r1 = 3; \
- *(u64 *)(r8 + 0) = 1; \
- r0 = 1; \
- if r8 != r7 goto 9f; \
- *(u8 *)(r8 + 1) = 1; \
- r0 = 2; \
- if r8 != r7 goto 9f; \
- *(u32 *)(r8 + 4) = r1; \
- r0 = 3; \
- if r8 != r7 goto 9f; \
- r2 = *(u16 *)(r8 + 2); \
- r0 = 4; \
- if r8 != r7 goto 9f; \
- r0 = 5; \
- if r1 != 3 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: atomics through a number that is not an address in arena")
-__success __retval(0)
-__load_if_JITed()
-__naked void atomic_value(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r1 = 0xdeadbeef00001000 ll; \
- *(u64 *)(r6 + 2048) = r1; \
- r7 = *(u64 *)(r6 + 2048); \
- r8 = *(u64 *)(r6 + 2048); \
- r1 = 3; \
- lock *(u64 *)(r8 + 0) += r1; \
- r0 = 1; \
- if r8 != r7 goto 9f; \
- .8byte %[fetch_add_insn]; \
- r0 = 2; \
- if r8 != r7 goto 9f; \
- .8byte %[xchg_insn]; \
- r0 = 3; \
- if r8 != r7 goto 9f; \
- r0 = 0; \
- .8byte %[cmpxchg_insn]; \
- r0 = 4; \
- if r8 != r7 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm_insn(fetch_add_insn, BPF_ATOMIC_OP(BPF_DW, BPF_ADD | BPF_FETCH,
- BPF_REG_8, BPF_REG_1, 0)),
- __imm_insn(xchg_insn, BPF_ATOMIC_OP(BPF_DW, BPF_XCHG, BPF_REG_8, BPF_REG_1, 0)),
- __imm_insn(cmpxchg_insn, BPF_ATOMIC_OP(BPF_DW, BPF_CMPXCHG, BPF_REG_8, BPF_REG_1, 0))
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: number and pointer to arena at the same insn")
-__success __retval(0)
-__load_if_JITed()
-__naked void mixed_number_arena(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- *(u64 *)(r6 + 8) = 0x1234; \
- /* a number that the verifier does not know, 0 at run time */ \
- r7 = *(u64 *)(r6 + 16); \
- r8 = r6; \
- if r7 != 0 goto 1f; \
- .8byte %[cast_kern_insn]; \
-1: *(u8 *)(r8 + 0) = 1; \
- *(u8 *)(r8 + 1) = r7; \
- r2 = *(u8 *)(r8 + 1); \
- lock *(u64 *)(r8 + 24) += r7; \
- r0 = 0; \
- if r7 != 0 goto 9f; \
- /* pointer to arena only: JIT adds all 64 bits of r8 to the base */ \
- r0 = 2; \
- r2 = *(u64 *)(r8 + 8); \
- if r2 != 0x1234 goto 9f; \
- r0 = 3; \
- r2 = *(u8 *)(r6 + 0); \
- if r2 != 1 goto 9f; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm_insn(cast_kern_insn, BPF_RAW_INSN(BPF_ALU64 | BPF_MOV | BPF_X,
- BPF_REG_8, BPF_REG_8, 1, 1))
- : __clobber_all);
-}
-
-SEC("syscall")
-__description("arena_scalar: no flag, no access through a number")
-__failure __msg("R6 invalid mem access 'scalar'")
-__naked void no_flag(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r6 = 0x100000000000 ll; \
- r0 = *(u64 *)(r6 + 0); \
- exit; \
-" :
- : __imm_addr(arena)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: pointer that is NULL is an address in arena")
-__success __retval(0)
-__load_if_JITed()
-__naked void null_ptr(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r1 = 0; \
- *(u32 *)(r10 - 4) = r1; \
- r2 = r10; \
- r2 += -4; \
- r1 = %[hash] ll; \
- call %[bpf_map_lookup_elem]; \
- r1 = r0; \
- r0 = 1; \
- if r1 != 0 goto 9f; \
- /* nothing is allocated: the load reads 0 */ \
- r0 = *(u64 *)(r1 + 0); \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm_addr(hash),
- __imm(bpf_map_lookup_elem)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: pointer that is NULL with an offset is an address in arena")
-__success __retval(0)
-__load_if_JITed()
-__naked void null_ptr_off(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r1 = 0; \
- *(u32 *)(r10 - 4) = r1; \
- r2 = r10; \
- r2 += -4; \
- r1 = %[hash] ll; \
- call %[bpf_map_lookup_elem]; \
- r1 = r0; \
- r0 = 1; \
- if r1 != 0 goto 9f; \
- r1 += 8; \
- *(u64 *)(r1 + 0) = 5; \
- r0 = *(u64 *)(r1 + 0); \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm_addr(hash),
- __imm(bpf_map_lookup_elem)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: number that is less than a page is an address in arena")
-__success __retval(0)
-__load_if_JITed()
-__naked void small_number(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- call %[bpf_get_prandom_u32]; \
- r1 = r0; \
- r1 &= 0xfff; \
- r0 = *(u64 *)(r1 + 0); \
- exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_get_prandom_u32)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: number is not a pointer for a helper")
-__failure __msg("R2 type=scalar expected=")
-__naked void helper_arg(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- r1 = %[hash] ll; \
- r2 = r6; \
- call %[bpf_map_lookup_elem]; \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm_addr(hash),
- __imm(bpf_arena_alloc_pages),
- __imm(bpf_map_lookup_elem)
- : __clobber_all);
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: number and pointer to stack at the same insn")
-__failure __msg("same insn cannot be used with different pointers")
-__load_if_JITed()
-__naked void mixed_number_stack(void)
-{
- asm volatile (" \
- r1 = %[arena] ll; \
- r2 = 0; \
- r3 = 1; \
- r4 = -1; \
- r5 = 0; \
- call %[bpf_arena_alloc_pages]; \
- r6 = r0; \
- r0 = 100; \
- if r6 == 0 goto 9f; \
- *(u64 *)(r10 - 8) = 0; \
- call %[bpf_get_prandom_u32]; \
- if w0 != 0 goto 1f; \
- r6 = r10; \
- r6 += -8; \
-1: r0 = *(u64 *)(r6 + 0); \
- r0 = 0; \
-9: exit; \
-" :
- : __imm_addr(arena),
- __imm(bpf_arena_alloc_pages),
- __imm(bpf_get_prandom_u32)
- : __clobber_all);
-}
-
-static int st_cb(__u64 idx, void *ctx)
-{
- volatile long *p = *(volatile long **)ctx;
-
- p[idx] = 7;
- return 0;
-}
-
-SEC("syscall")
-__arena_scalar
-__description("arena_scalar: store through a number in a callback")
-__success __retval(0)
-__load_if_JITed()
-int st_in_callback(void *unused)
-{
- volatile long *p = bpf_arena_alloc_pages(&arena, NULL, 1, -1, 0);
-
- if (!p)
- return 100;
- bpf_loop(4, st_cb, &p, 0);
- return p[0] + p[3] + p[4] - 14;
-}
-
-char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/test_loader.c b/tools/testing/selftests/bpf/test_loader.c
index 824e5912ebcb..d24e4fedae46 100644
--- a/tools/testing/selftests/bpf/test_loader.c
+++ b/tools/testing/selftests/bpf/test_loader.c
@@ -590,8 +590,6 @@ static int parse_test_spec(struct test_loader *tester,
update_flags(&spec->prog_flags, BPF_F_XDP_HAS_FRAGS, clear);
} else if (strcmp(val, "BPF_F_TEST_REG_INVARIANTS") == 0) {
update_flags(&spec->prog_flags, BPF_F_TEST_REG_INVARIANTS, clear);
- } else if (strcmp(val, "BPF_F_ARENA_SCALAR") == 0) {
- update_flags(&spec->prog_flags, BPF_F_ARENA_SCALAR, clear);
} else /* assume numeric value */ {
err = parse_int(val, &flags, "test prog flags");
if (err)
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 05/81] bpf: Remove BPF_F_ARENA_SCALAR
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (3 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 04/81] selftests/bpf: Remove tests for arena access through numbers Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 06/81] bpf, x86: Remove r0_on_stack from fetching atomics in arena Alexei Starovoitov
` (77 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Rust has no address spaces. Thus a prog in Rust accesses arena through
scalars. With BPF_F_ARENA_SCALAR the verifier accepts each load and each
store through a scalar as an access to arena.
The verifier rewrites one insn at a time. Thus it must add one insn to each
access. The compiler knows the source of each pointer. It can emit one
addr_space_cast for each pointer, as it does for the __arena pointers of C.
flag: one move for each access cast: one cast for each pointer
ax = (u32)r1 r1 = addr_space_cast(r1, 0, 1)
r2 = *(u64 *)(ax + 0) r2 = *(u64 *)(r1 + 0)
ax = (u32)r1 r3 = *(u64 *)(r1 + 8)
r3 = *(u64 *)(ax + 8)
20 progs of rust-bpf, with the flag and with the casts that a prototype of
an LLVM pass emits:
flag casts
moves or casts 1568 571
insns 14864 13938
jited bytes 72351 69111
No released kernel has the flag. LLVM does not emit the casts for Rust at
this time. Until it does, Rust progs that access arena do not load.
This reverts commit 4c651a91bdfc
("bpf: Treat load and store through a number as arena access"). One part
stays: BPF_REG_AX in reg2pt_regs[] of the x86 JIT. The fetching AND/OR/XOR
atomics in arena use it now.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/arm64/net/bpf_jit_comp.c | 5 ---
arch/x86/net/bpf_jit_comp.c | 5 ---
include/linux/bpf_verifier.h | 2 --
include/linux/filter.h | 1 -
include/uapi/linux/bpf.h | 6 ----
kernel/bpf/core.c | 9 +----
kernel/bpf/fixups.c | 39 ---------------------
kernel/bpf/syscall.c | 4 ---
kernel/bpf/verifier.c | 64 ++++++++--------------------------
tools/include/uapi/linux/bpf.h | 6 ----
10 files changed, 16 insertions(+), 125 deletions(-)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index aaa34c739004..33632fb20997 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -3502,11 +3502,6 @@ bool bpf_jit_supports_arena(void)
return true;
}
-bool bpf_jit_supports_arena_scalar(void)
-{
- return true;
-}
-
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena)
{
if (!in_arena)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index fd65ac84cfc7..7bfb7210484e 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -5240,11 +5240,6 @@ bool bpf_jit_supports_arena(void)
return true;
}
-bool bpf_jit_supports_arena_scalar(void)
-{
- return true;
-}
-
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena)
{
return true;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 5926f13ff43c..de301b32015d 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -782,7 +782,6 @@ struct bpf_insn_aux_data {
u64 non_sleepable:1; /* helper/kfunc may be called from non-sleepable context */
u64 is_iter_next:1; /* bpf_iter_<type>_next() kfunc call */
u64 call_with_percpu_alloc_ptr:1; /* {this,per}_cpu_ptr() with prog percpu alloc */
- u64 arena_scalar:1; /* ldx/stx/st/atomic through a number, it's an address in arena */
u64 bb_end:1; /* last instruction of a basic block */
u64 need_scev:1;
u64 alu_state:5; /* used in combination with alu_limit */
@@ -1074,7 +1073,6 @@ struct bpf_verifier_env {
bool strict_alignment; /* perform strict pointer alignment checks */
bool test_state_freq; /* test verifier with different pruning frequency */
bool test_reg_invariants; /* fail verification on register invariants violations */
- bool arena_scalar; /* load and store through a number is arena access */
struct bpf_verifier_state *cur_state; /* current verifier state */
/* Search pruning optimization, array of list_heads for
* lists of struct bpf_verifier_state_list.
diff --git a/include/linux/filter.h b/include/linux/filter.h
index e0218e677a1d..98254742ae37 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1287,7 +1287,6 @@ bool bpf_jit_supports_far_kfunc_call(void);
bool bpf_jit_supports_exceptions(void);
bool bpf_jit_supports_ptr_xchg(void);
bool bpf_jit_supports_arena(void);
-bool bpf_jit_supports_arena_scalar(void);
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
bool bpf_jit_supports_private_stack(void);
bool bpf_jit_supports_large_stack(void);
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index f29b0aad440c..8d7228834305 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1357,12 +1357,6 @@ enum bpf_perf_event_type {
/* The verifier internal test flag. Behavior is undefined */
#define BPF_F_TEST_REG_INVARIANTS (1U << 7)
-/*
- * Load and store through a number is an access to the arena of the program
- * at the low 32 bits of the number. It's for programs written in Rust.
- */
-#define BPF_F_ARENA_SCALAR (1U << 8)
-
/* link_create.kprobe_multi.flags used in LINK_CREATE command for
* BPF_TRACE_KPROBE_MULTI attach type to create return probe.
*/
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 0a8ba603f44b..7cf37690d41f 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -2639,8 +2639,7 @@ static bool bpf_prog_select_interpreter(struct bpf_prog *fp)
/*
* Whether a function has an insn that works with the arena base, which a JIT
* sets up in the prologue when bpf_prog_uses_arena_insns() says so. Decided on
- * the final insns of the function: an access through an arena pointer, or
- * through a number in a program loaded with BPF_F_ARENA_SCALAR, has been
+ * the final insns of the function: an access through an arena pointer has been
* rewritten to a BPF_PROBE_MEM32 form by then, a cast to the user address
* space keeps BPF_ADDR_SPACE_CAST, and a kfunc call with an __arena argument,
* which the JIT rebases, has a function model that says so. A load of an
@@ -3518,12 +3517,6 @@ bool __weak bpf_jit_supports_arena(void)
return false;
}
-/* Whether JIT takes BPF_REG_AX as the address of arena access */
-bool __weak bpf_jit_supports_arena_scalar(void)
-{
- return false;
-}
-
bool __weak bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena)
{
return false;
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index f062ef7dd943..41bc37e205e0 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -44,31 +44,6 @@ static bool is_addr_space_cast32(struct bpf_prog *prog, const struct bpf_insn *i
return false;
}
-/*
- * The insn accesses arena through a number. JITs add the base of arena
- * to the register as it is, so the access goes through the low 32 bits of
- * the number in BPF_REG_AX. Registers of the program are not changed: the
- * number may be compared or stored later, and on another path the register
- * may be PTR_TO_ARENA.
- *
- * Constant blinding leaves insns that use BPF_REG_AX alone, so the immediate
- * of st through a number is not blinded.
- */
-static int arena_scalar_access(const struct bpf_insn *insn, struct bpf_insn *buf)
-{
- bool load = BPF_CLASS(insn->code) == BPF_LDX || bpf_atomic_is_load_acq(insn);
- struct bpf_insn *patch = buf;
-
- *patch++ = BPF_MOV32_REG(BPF_REG_AX, load ? insn->src_reg : insn->dst_reg);
- *patch = *insn;
- if (load)
- patch->src_reg = BPF_REG_AX;
- else
- patch->dst_reg = BPF_REG_AX;
- patch++;
- return patch - buf;
-}
-
/* Return the regno defined by the insn, or -1. */
static int insn_def_regno(const struct bpf_insn *insn)
{
@@ -1880,20 +1855,6 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
/* Convert BPF_CLASS(insn->code) == BPF_ALU64 to 32-bit ALU */
insn->code = BPF_ALU | BPF_OP(insn->code) | BPF_SRC(insn->code);
- if (env->insn_aux_data[i + delta].arena_scalar) {
- cnt = arena_scalar_access(insn, insn_buf);
-
- new_prog = bpf_patch_insn_data(env, i + delta, insn_buf, cnt);
- if (!new_prog)
- return -ENOMEM;
-
- delta += cnt - 1;
- prog = new_prog;
- env->prog = prog;
- insn = prog->insnsi + i + delta;
- goto next_insn;
- }
-
/* Make sdiv/smod divide-by-minus-one exceptions impossible. */
if ((insn->code == (BPF_ALU64 | BPF_MOD | BPF_K) ||
insn->code == (BPF_ALU64 | BPF_DIV | BPF_K) ||
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 90bc7230afa8..60c4d84b9aba 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -2971,13 +2971,9 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_at
BPF_F_XDP_HAS_FRAGS |
BPF_F_XDP_DEV_BOUND_ONLY |
BPF_F_TEST_REG_INVARIANTS |
- BPF_F_ARENA_SCALAR |
BPF_F_TOKEN_FD))
return -EINVAL;
- if ((attr->prog_flags & BPF_F_ARENA_SCALAR) && !bpf_jit_supports_arena_scalar())
- return -EOPNOTSUPP;
-
bpf_prog_load_fixup_attach_type(attr);
if (attr->prog_flags & BPF_F_TOKEN_FD) {
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index c223bf3bc9cd..c93ef399c420 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -5495,19 +5495,6 @@ static bool is_arena_reg(struct bpf_verifier_env *env, int regno)
return reg->type == PTR_TO_ARENA;
}
-/*
- * There are no address spaces in Rust, addresses of arena are plain numbers.
- * When the program is loaded with BPF_F_ARENA_SCALAR and has an arena a load or
- * a store through a number is an access to arena at the low 32 bits of it,
- * like the access through PTR_TO_ARENA is. The register stays a number.
- */
-static bool is_arena_scalar(struct bpf_verifier_env *env, int regno)
-{
- const struct bpf_reg_state *reg = reg_state(env, regno);
-
- return reg->type == SCALAR_VALUE && env->arena_scalar && env->prog->aux->arena;
-}
-
static bool is_load_acq_unsafe(struct bpf_verifier_env *env, int regno,
struct bpf_insn *insn)
{
@@ -5538,7 +5525,7 @@ static bool atomic_ptr_type_ok(struct bpf_verifier_env *env, int regno,
return false;
if (is_sk_reg(env, regno))
return false;
- if (is_arena_reg(env, regno) || is_arena_scalar(env, regno))
+ if (is_arena_reg(env, regno))
return bpf_jit_supports_insn(insn, true);
if (is_load_acq_unsafe(env, regno, insn))
return false;
@@ -7519,22 +7506,6 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct b
static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type type,
bool allow_trust_mismatch);
-/*
- * Returns the register to check the access of the current insn with.
- * For a number in a program with an arena that is 'arena'.
- */
-static struct bpf_reg_state *mem_access_reg(struct bpf_verifier_env *env, int regno,
- struct bpf_reg_state *arena)
-{
- if (!is_arena_scalar(env, regno))
- return cur_regs(env) + regno;
-
- memset(arena, 0, sizeof(*arena));
- arena->type = PTR_TO_ARENA;
- env->insn_aux_data[env->insn_idx].arena_scalar = true;
- return arena;
-}
-
static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,
bool strict_alignment_once, bool is_ldsx,
bool allow_trust_mismatch, const char *ctx)
@@ -7542,7 +7513,6 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,
struct bpf_verifier_state *vstate = env->cur_state;
struct bpf_func_state *state = vstate->frame[vstate->curframe];
struct bpf_reg_state *regs = cur_regs(env);
- struct bpf_reg_state arena, *src_reg;
enum bpf_reg_type src_reg_type;
int err;
@@ -7564,16 +7534,16 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn,
if (err)
return err;
- src_reg = mem_access_reg(env, insn->src_reg, &arena);
- src_reg_type = src_reg->type;
+ src_reg_type = regs[insn->src_reg].type;
/*
* check_stack_read_fixed_off() may refine the modification's origin to
* the source stack slot.
*/
bpf_diag_mod_begin(env, ®s[insn->dst_reg], NULL, BPF_DIAG_MOD_WRITE);
- err = check_mem_access(env, env->insn_idx, src_reg, argno_from_reg(insn->src_reg),
- insn->off, BPF_SIZE(insn->code), BPF_READ, insn->dst_reg,
+ err = check_mem_access(env, env->insn_idx, regs + insn->src_reg,
+ argno_from_reg(insn->src_reg), insn->off,
+ BPF_SIZE(insn->code), BPF_READ, insn->dst_reg,
strict_alignment_once, is_ldsx);
/*
* Paths through a kernel struct pointer and through other pointers
@@ -7601,7 +7571,6 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,
struct bpf_verifier_state *vstate = env->cur_state;
struct bpf_func_state *state = vstate->frame[vstate->curframe];
struct bpf_reg_state *regs = cur_regs(env);
- struct bpf_reg_state arena, *dst_reg;
enum bpf_reg_type dst_reg_type;
int err;
@@ -7623,12 +7592,12 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,
if (err)
return err;
- dst_reg = mem_access_reg(env, insn->dst_reg, &arena);
- dst_reg_type = dst_reg->type;
+ dst_reg_type = regs[insn->dst_reg].type;
/* Check if (dst_reg + off) is writeable. */
- err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg),
- insn->off, BPF_SIZE(insn->code), BPF_WRITE, insn->src_reg,
+ err = check_mem_access(env, env->insn_idx, regs + insn->dst_reg,
+ argno_from_reg(insn->dst_reg), insn->off,
+ BPF_SIZE(insn->code), BPF_WRITE, insn->src_reg,
strict_alignment_once, false);
err = err ?: save_aux_ptr_type(env, dst_reg_type, false);
@@ -7638,7 +7607,7 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn,
static int check_atomic_rmw(struct bpf_verifier_env *env,
struct bpf_insn *insn)
{
- struct bpf_reg_state arena, *dst_reg;
+ struct bpf_reg_state *dst_reg;
int load_reg;
int err;
@@ -7683,9 +7652,6 @@ static int check_atomic_rmw(struct bpf_verifier_env *env,
return -EACCES;
}
- /* load_reg may be dst_reg. Look at dst_reg before it's marked as unknown. */
- dst_reg = mem_access_reg(env, insn->dst_reg, &arena);
-
load_reg = bpf_atomic_load_reg(insn);
if (load_reg >= 0) {
/* check and record load of old value */
@@ -7694,6 +7660,8 @@ static int check_atomic_rmw(struct bpf_verifier_env *env,
return err;
}
+ dst_reg = cur_regs(env) + insn->dst_reg;
+
/* Check whether we can read the memory, with second call for fetch
* case to simulate the register fill.
*/
@@ -20280,17 +20248,16 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
return check_stack_arg_write(env, state, insn->off, NULL);
}
- struct bpf_reg_state arena, *dst_reg;
enum bpf_reg_type dst_reg_type;
err = check_reg_arg(env, insn->dst_reg, SRC_OP);
if (err)
return err;
- dst_reg = mem_access_reg(env, insn->dst_reg, &arena);
- dst_reg_type = dst_reg->type;
+ dst_reg_type = cur_regs(env)[insn->dst_reg].type;
- err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg),
+ err = check_mem_access(env, env->insn_idx, cur_regs(env) + insn->dst_reg,
+ argno_from_reg(insn->dst_reg),
insn->off, BPF_SIZE(insn->code),
BPF_WRITE, -1, false, false);
if (err)
@@ -23926,7 +23893,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (is_priv)
env->test_state_freq = attr->prog_flags & BPF_F_TEST_STATE_FREQ;
env->test_reg_invariants = attr->prog_flags & BPF_F_TEST_REG_INVARIANTS;
- env->arena_scalar = attr->prog_flags & BPF_F_ARENA_SCALAR;
env->explored_states = kvzalloc_objs(struct list_head,
state_htab_size(env),
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index f29b0aad440c..8d7228834305 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1357,12 +1357,6 @@ enum bpf_perf_event_type {
/* The verifier internal test flag. Behavior is undefined */
#define BPF_F_TEST_REG_INVARIANTS (1U << 7)
-/*
- * Load and store through a number is an access to the arena of the program
- * at the low 32 bits of the number. It's for programs written in Rust.
- */
-#define BPF_F_ARENA_SCALAR (1U << 8)
-
/* link_create.kprobe_multi.flags used in LINK_CREATE command for
* BPF_TRACE_KPROBE_MULTI attach type to create return probe.
*/
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 06/81] bpf, x86: Remove r0_on_stack from fetching atomics in arena
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (4 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 05/81] bpf: Remove BPF_F_ARENA_SCALAR Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog Alexei Starovoitov
` (76 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
x86 has no fetching AND/OR/XOR. Thus the JIT emits a loop with cmpxchg.
cmpxchg uses RAX, and RAX holds BPF R0. Thus the JIT saves R0 before the
loop.
prog the address is in the JIT saves R0 in
-------------------------- ------------------ -------------------
without the flag a BPF register BPF_REG_AX
with BPF_F_ARENA_SCALAR BPF_REG_AX the stack
The flag is removed. Remove the second variant.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/x86/net/bpf_jit_comp.c | 58 +++++++++----------------------------
1 file changed, 14 insertions(+), 44 deletions(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 7bfb7210484e..d90b6d238b3a 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1745,26 +1745,20 @@ static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size,
/*
* A fetching AND/OR/XOR can't be implemented with a single x86 insn, so do a
* CMPXCHG loop. @index_reg is X86_REG_R12 for an arena access or -1 otherwise.
- * Unless @r0_on_stack, the caller has substituted BPF_REG_AX for R0 in
- * @dst_reg and @src_reg.
+ * The caller has saved R0 in BPF_REG_AX and has substituted BPF_REG_AX for R0
+ * in @dst_reg and @src_reg.
*
* Both the load and the CMPXCHG can fault when accessing the arena, so their
* addresses are handed back in @fault[0] and @fault[1]. A fault has to resume
* at @resume, which is past the loop and past the move that delivers the old
* value, but before R0 is restored. Both are NULL for the non-arena case,
* which needs no fixup.
- *
- * R0 is expected to be saved in BPF_REG_AX by the caller, or pushed on the
- * stack if @r0_on_stack. In that case an R0 operand stays in the stack slot,
- * RAX ends up holding the old value that R0 is to return, and the slot is
- * dropped rather than popped.
*/
static int emit_atomic_fetch_rmw(u8 **pprog, struct bpf_insn *insn, u32 dst_reg,
- u32 src_reg, int index_reg, bool r0_on_stack,
+ u32 src_reg, int index_reg,
u8 **fault, u8 **resume)
{
bool is64 = BPF_SIZE(insn->code) == BPF_DW;
- bool operand_on_stack = r0_on_stack && src_reg == BPF_REG_0;
u8 *branch_target, *prog = *pprog;
int err;
@@ -1783,15 +1777,7 @@ static int emit_atomic_fetch_rmw(u8 **pprog, struct bpf_insn *insn, u32 dst_reg,
* Perform the (commutative) operation locally, put the result in
* the AUX_REG.
*/
- if (operand_on_stack) {
- /* mov r11, [rsp]: the operand is the R0 saved there, RAX is the old value */
- if (is64)
- EMIT4(0x4C, 0x8B, 0x1C, 0x24);
- else
- EMIT4(0x44, 0x8B, 0x1C, 0x24);
- } else {
- emit_mov_reg(&prog, is64, AUX_REG, BPF_REG_0);
- }
+ emit_mov_reg(&prog, is64, AUX_REG, BPF_REG_0);
maybe_emit_mod(&prog, AUX_REG, src_reg, is64);
EMIT2(simple_alu_opcodes[BPF_OP(insn->imm)],
add_2reg(0xC0, AUX_REG, src_reg));
@@ -1810,20 +1796,14 @@ static int emit_atomic_fetch_rmw(u8 **pprog, struct bpf_insn *insn, u32 dst_reg,
/* ZF tells us whether we won the race. If it's cleared we need to try again. */
EMIT2(X86_JNE, -(prog - branch_target) - 2);
- /* Return the pre-modification value, which RAX already is for R0 */
- if (!operand_on_stack)
- emit_mov_reg(&prog, is64, src_reg, BPF_REG_0);
+ /* Return the pre-modification value */
+ emit_mov_reg(&prog, is64, src_reg, BPF_REG_0);
if (resume)
*resume = prog;
/* Restore R0 after clobbering RAX */
- if (!r0_on_stack)
- emit_mov_reg(&prog, true, BPF_REG_0, BPF_REG_AX);
- else if (operand_on_stack)
- EMIT2(0x41, 0x5B); /* pop r11, dropping the operand */
- else
- EMIT1(0x58); /* pop rax */
+ emit_mov_reg(&prog, true, BPF_REG_0, BPF_REG_AX);
*pprog = prog;
return 0;
@@ -3354,7 +3334,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
}
if (is_atomic_fetch) {
err = emit_atomic_fetch_rmw(&prog, insn, real_dst_reg,
- real_src_reg, -1, false,
+ real_src_reg, -1,
NULL, NULL);
if (err)
return err;
@@ -3383,29 +3363,19 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
case BPF_STX | BPF_PROBE_ATOMIC | BPF_DW:
if (is_atomic_fetch_op(insn)) {
u32 real_src_reg = src_reg, real_dst_reg = dst_reg;
- /*
- * An access through a number has its address in
- * BPF_REG_AX (see arena_scalar_access()), so R0 is
- * saved on the stack instead, as div and mod do.
- */
- bool r0_on_stack = dst_reg == BPF_REG_AX;
u8 *fault[2], *resume;
int j;
/* Will need RAX as a CMPXCHG operand so save R0 */
- if (r0_on_stack) {
- EMIT1(0x50); /* push rax */
- } else {
- emit_mov_reg(&prog, true, BPF_REG_AX, BPF_REG_0);
- if (src_reg == BPF_REG_0)
- real_src_reg = BPF_REG_AX;
- if (dst_reg == BPF_REG_0)
- real_dst_reg = BPF_REG_AX;
- }
+ emit_mov_reg(&prog, true, BPF_REG_AX, BPF_REG_0);
+ if (src_reg == BPF_REG_0)
+ real_src_reg = BPF_REG_AX;
+ if (dst_reg == BPF_REG_0)
+ real_dst_reg = BPF_REG_AX;
err = emit_atomic_fetch_rmw(&prog, insn, real_dst_reg,
real_src_reg, X86_REG_R12,
- r0_on_stack, fault, &resume);
+ fault, &resume);
if (err)
return err;
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (5 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 06/81] bpf, x86: Remove r0_on_stack from fetching atomics in arena Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 12:50 ` kernel test robot
` (2 more replies)
2026-10-26 2:58 ` [PATCH bpf 08/81] selftests/bpf: Test may_goto in a loop that calls a subprog that spins Alexei Starovoitov
` (75 subsequent siblings)
82 siblings, 3 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
may_goto stops a loop that the verifier cannot bound: after 250 ms the loop
must exit. The count and the timestamp are in the stack frame of the
function that contains the loop. Thus the limit applies to one call of a
subprog, not to one run of the prog. Loops that nest across calls multiply
the limit.
main: sub:
1: call sub 1: may_goto 2f
may_goto 2f goto 1b
goto 1b 2: r0 = 0
2: r0 = 1 exit
exit
main | sub: 250 ms | sub: 250 ms | ... | sub: 250 ms |
1 2 65535 ^
main reads the clock here
Each call of sub spins for 250 ms. main reads the clock one time in
65535 iterations. Thus the prog runs for 65535 * 250 ms: 9 insns cause a
soft lockup. bpf_loop(1 << 23, cb) with such a loop in cb has the same
result.
The count has the same problem. A subprog that makes fewer than
65535 iterations in each call never reads the clock. An endless loop that
calls a subprog with 60000 iterations runs for 3.5 sec. Three levels of
such loops lock up a cpu.
Give one time slice and one count to the full run. A run is one call of the
main prog, or of an async callback, by the kernel, together with all the
subprogs that it calls. The main prog gets a per-cpu budget.
the kernel calls the prog budget.start = 0
| budget.count = 65535
v
entry of a subprog frame.count = budget.count
|
v
may_goto frame.count -= 1
| At 0, read the clock:
| < 250 ms refill frame.count, budget.count
| >= 250 ms budget.count = 0, exit the loop
v
exit of the subprog budget.count = min(budget.count, frame.count)
Iterations now add up across calls. After 250 ms the budget is empty:
- A subprog that starts after this time exits its loops immediately.
- A frame that is already on the stack exits its loop when it reads the
clock again.
The progs above return after 250 ms.
Only JITed progs that have may_goto in a subprog or in a callback get
different code: 0 of 144 progs of Cilium, 53 of 495 progs of scx. The cost
on x86-64 (Zen 5) with hot caches, in nsec:
entry of the main prog, once per run 4 insns, 34 bytes +0.52
entry of a subprog with may_goto 3 insns, 19 bytes +0.03
each exit of such a subprog 6 insns, 36 bytes +0.15
async callback (bpf_timer), all progs reset the budget +0.35 on 34.6
read of the clock, once in 65535 stack walk to find 26 -> 179
iterations, all progs the budget
An iteration has no added cost. A run with one call of such a subprog costs
+0.97 nsec. It costs +90 nsec when the budget is not in the cache.
Limitations:
- A second run of the same prog on the same cpu, in an interrupt or after
preemption, starts the time slice again. The timestamp in the frame
continues to limit each frame to 250 ms.
- The JIT must support a stack walk and per-cpu insns. x86-64 with ORC,
arm64 and powerpc64 do. Nothing changes on riscv, loongarch and s390.
- Nothing changes for extension progs. They cannot find the start of a
run.
Fixes: e723608bf428 ("bpf: Add verifier support for timed may_goto")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 12 +++++++
kernel/bpf/core.c | 72 ++++++++++++++++++++++++++++++++++----
kernel/bpf/fixups.c | 82 ++++++++++++++++++++++++++++++++++++++++++--
kernel/bpf/helpers.c | 4 +++
4 files changed, 160 insertions(+), 10 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index ff0436781293..4c7c3f86ddbd 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1851,6 +1851,15 @@ enum bpf_sig_keyring {
BPF_SIG_KEYRING_BPF,
};
+/*
+ * What all frames of a run of a prog on this cpu share, see bpf_check_timed_may_goto().
+ * It's reset when a run begins, see bpf_prog_begin_run().
+ */
+struct bpf_may_goto_budget {
+ u64 start; /* when may_goto looked at the clock for the first time, 0: it didn't yet */
+ u64 count; /* what is left until the next look at the clock, 0: the time is up */
+};
+
struct bpf_prog_aux {
atomic64_t refcnt;
u32 used_map_cnt;
@@ -1916,6 +1925,8 @@ struct bpf_prog_aux {
const char *attach_func_name;
struct bpf_prog **func;
struct bpf_prog_aux *main_prog_aux;
+ /* main prog: there when a subprog has may_goto */
+ struct bpf_may_goto_budget __percpu *may_goto_budget;
void *jit_data; /* JIT specific data. arch dependent */
struct bpf_jit_poke_descriptor *poke_tab;
struct bpf_kfunc_desc_tab *kfunc_tab;
@@ -3415,6 +3426,7 @@ void bpf_dynptr_init(struct bpf_dynptr_kern *ptr, void *data,
void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr);
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr);
void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip);
+void bpf_prog_begin_run(const struct bpf_prog *prog);
static __always_inline u32
bpf_prog_run_array_sleepable(const struct bpf_prog_array *array,
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 7cf37690d41f..472efced9e40 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)
mutex_destroy(&fp->aux->used_maps_mutex);
mutex_destroy(&fp->aux->dst_mutex);
kfree(fp->aux->poke_tab);
+ free_percpu(fp->aux->may_goto_budget);
kfree(fp->aux);
}
free_percpu(fp->stats);
@@ -3625,21 +3626,78 @@ static noinline void bpf_prog_report_may_goto_violation(void)
#endif
}
+/*
+ * The kernel is about to call an async callback of @prog on this cpu: a run begins.
+ * The main prog does the same in its first insns.
+ */
+void bpf_prog_begin_run(const struct bpf_prog *prog)
+{
+ struct bpf_prog_aux *aux;
+
+ if (!prog)
+ return;
+ aux = prog->aux->main_prog_aux;
+ if (aux->may_goto_budget) {
+ this_cpu_write(aux->may_goto_budget->start, 0);
+ this_cpu_write(aux->may_goto_budget->count, BPF_MAX_TIMED_LOOPS);
+ }
+}
+
+static struct bpf_may_goto_budget *bpf_may_goto_budget(void)
+{
+#ifdef CONFIG_BPF_SYSCALL
+ struct bpf_prog *prog = bpf_prog_find_from_stack();
+
+ if (prog && prog->aux->may_goto_budget)
+ return this_cpu_ptr(prog->aux->may_goto_budget);
+#endif
+ return NULL;
+}
+
+/*
+ * The count and the timestamp are those of a stack frame. With nothing else every call of
+ * a subprog has a time slice of its own, and a loop that calls a subprog that uses up its slice
+ * goes on for BPF_MAX_TIMED_LOOPS slices before it looks at the clock itself.
+ *
+ * So there is one time slice for a run of the prog. It starts when the first frame gets here.
+ * When it's over the frames that are entered from then on start to count from 0: they leave
+ * their loops at once. The frames that are there find out when they get here.
+ *
+ * The count is shared as well. A subprog that makes fewer than BPF_MAX_TIMED_LOOPS iterations
+ * per call would never get here, and neither would a loop that calls it, for that many times as
+ * many iterations. Three levels of that take days. So a subprog starts from what is left for
+ * the run, and what it has left when it returns is what is left for the run. No more than
+ * BPF_MAX_TIMED_LOOPS iterations for every frame on the stack pass between two looks at the clock.
+ *
+ * Another run of the prog on this cpu, in an interrupt or after preemption, starts the slice
+ * anew. The slice of the frame is still there for that.
+ */
u64 bpf_check_timed_may_goto(struct bpf_timed_may_goto *p)
{
- u64 time = ktime_get_mono_fast_ns();
+ struct bpf_may_goto_budget *run = bpf_may_goto_budget();
+ u64 time = ktime_get_mono_fast_ns(), start;
- /* Populate the timestamp for this stack frame, and refresh count. */
- if (!p->timestamp) {
+ /* Populate the timestamp for this stack frame. */
+ if (!p->timestamp)
p->timestamp = time;
- return BPF_MAX_TIMED_LOOPS;
+ start = p->timestamp;
+ if (run) {
+ if (!run->start)
+ run->start = time;
+ start = min(start, run->start);
}
/* Check if we've exhausted our time slice, and zero count. */
- if (unlikely(time - p->timestamp >= (NSEC_PER_SEC / 4))) {
- bpf_prog_report_may_goto_violation();
+ if (unlikely(time - start >= (NSEC_PER_SEC / 4))) {
+ /* once per run */
+ if (!run || run->count)
+ bpf_prog_report_may_goto_violation();
+ if (run)
+ run->count = 0;
return 0;
}
- /* Refresh the count for the stack frame. */
+ /* Refresh the count for the stack frame, and for the frames to come. */
+ if (run)
+ run->count = BPF_MAX_TIMED_LOOPS;
return BPF_MAX_TIMED_LOOPS;
}
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 41bc37e205e0..fe8d51f8c855 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -1821,6 +1821,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
struct bpf_map *map_ptr;
int i, ret, cnt, delta = 0, cur_subprog = 0;
struct bpf_subprog_info *subprogs = env->subprog_info;
+ struct bpf_may_goto_budget __percpu *budget = NULL;
u16 stack_depth = subprogs[cur_subprog].stack_depth;
u16 stack_depth_extra = 0;
@@ -2729,13 +2730,69 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
}
env->prog->aux->stack_depth = subprogs[0].stack_depth;
+ /*
+ * One time slice for all frames of a run, see bpf_check_timed_may_goto(). It finds
+ * the prog with a walk of the stack, like bpf_throw() does. An extension is entered by
+ * the prog that it extends, it has no way to tell one run from another.
+ */
+ for (i = 1; i < env->subprog_cnt; i++) {
+ if (!subprogs[i].stack_extra || !bpf_jit_supports_timed_may_goto() ||
+ !bpf_jit_supports_percpu_insn() || !bpf_jit_supports_exceptions() ||
+ !prog->jit_requested || prog->type == BPF_PROG_TYPE_EXT)
+ continue;
+ budget = alloc_percpu_gfp(struct bpf_may_goto_budget, GFP_KERNEL_ACCOUNT);
+ if (!budget)
+ return -ENOMEM;
+ prog->aux->may_goto_budget = budget;
+ prog->jit_required = 1;
+ break;
+ }
+ /*
+ * What a subprog has left of the count when it returns is what is left for the run,
+ * unless its callees have left less, or the time is up. R1-R3 are dead at exit.
+ */
+ for (i = 1; budget && i < env->subprog_cnt; i++) {
+ int off_count = offsetof(struct bpf_may_goto_budget, count);
+ struct bpf_insn ld[] = { BPF_LD_IMM64(BPF_REG_1, (long)budget) };
+ int j, cnt;
+
+ if (!subprogs[i].stack_extra)
+ continue;
+ for (j = subprogs[i].start; j < subprogs[i + 1].start; j++) {
+ if (bpf_is_ldimm64(&prog->insnsi[j])) {
+ j++;
+ continue;
+ }
+ if (prog->insnsi[j].code != (BPF_JMP | BPF_EXIT))
+ continue;
+ cnt = 0;
+ insn_buf[cnt++] = ld[0];
+ insn_buf[cnt++] = ld[1];
+ insn_buf[cnt++] = BPF_MOV64_PERCPU_REG(BPF_REG_1, BPF_REG_1);
+ insn_buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_2, BPF_REG_FP,
+ -subprogs[i].stack_depth);
+ insn_buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_1, off_count);
+ insn_buf[cnt++] = BPF_JMP_REG(BPF_JLE, BPF_REG_3, BPF_REG_2, 1);
+ insn_buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_2, off_count);
+ insn_buf[cnt++] = prog->insnsi[j];
+ new_prog = bpf_patch_insn_data(env, j, insn_buf, cnt);
+ if (!new_prog)
+ return -ENOMEM;
+ env->prog = prog = new_prog;
+ j += cnt - 1;
+ }
+ }
for (i = 0; i < env->subprog_cnt; i++) {
int delta = bpf_jit_supports_timed_may_goto() ? 2 : 1;
int subprog_start = subprogs[i].start;
int stack_slots = subprogs[i].stack_extra / 8;
+ int off_start = offsetof(struct bpf_may_goto_budget, start);
+ int off_count = offsetof(struct bpf_may_goto_budget, count);
+ /* a run begins. For async callbacks see bpf_prog_begin_run() */
+ bool entry = budget && !i;
int slots = delta, cnt = 0;
- if (!stack_slots)
+ if (!stack_slots && !entry)
continue;
/* We need two slots in case timed may_goto is supported. */
if (stack_slots > slots) {
@@ -2744,7 +2801,26 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
}
stack_depth = subprogs[i].stack_depth;
- if (bpf_jit_supports_timed_may_goto()) {
+ if (budget) {
+ struct bpf_insn ld[] = { BPF_LD_IMM64(BPF_REG_0, (long)budget) };
+
+ /* R0 has nothing at the entry */
+ insn_buf[cnt++] = ld[0];
+ insn_buf[cnt++] = ld[1];
+ insn_buf[cnt++] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
+ }
+ if (entry) {
+ insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_0, off_start, 0);
+ insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_0, off_count,
+ BPF_MAX_TIMED_LOOPS);
+ }
+ if (!stack_slots) {
+ /* nothing to count in this one */
+ } else if (budget && !entry) {
+ insn_buf[cnt++] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, off_count);
+ insn_buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_FP, BPF_REG_0, -stack_depth);
+ insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, -stack_depth + 8, 0);
+ } else if (bpf_jit_supports_timed_may_goto()) {
insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, -stack_depth,
BPF_MAX_TIMED_LOOPS);
insn_buf[cnt++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP, -stack_depth + 8, 0);
@@ -2774,7 +2850,7 @@ int bpf_do_misc_fixups(struct bpf_verifier_env *env)
* to insn after BPF_ST that inits may_goto count.
* Adjustment will succeed because bpf_patch_insn_data() didn't fail.
*/
- WARN_ON(adjust_jmp_off(env->prog, subprog_start, delta));
+ WARN_ON(adjust_jmp_off(env->prog, subprog_start, cnt - 1));
}
/* Since poke tab is now finalized, publish aux to tracker. */
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 648abe1ba63a..cd960f65093d 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -1242,6 +1242,7 @@ static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer)
key = map_key_from_value(map, value, &idx);
+ bpf_prog_begin_run(READ_ONCE(t->cb.prog));
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
/* The verifier checked that return value is zero. */
@@ -1273,6 +1274,7 @@ static void bpf_wq_work(struct work_struct *work)
migrate_disable();
+ bpf_prog_begin_run(READ_ONCE(cb->prog));
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
migrate_enable();
@@ -4611,6 +4613,7 @@ static void bpf_task_work_callback(struct callback_head *cb)
key = (void *)map_key_from_value(ctx->map, ctx->map_val, &idx);
migrate_disable();
+ bpf_prog_begin_run(ctx->prog);
ctx->callback_fn((u64)(long)ctx->map, (u64)(long)key,
(u64)(long)ctx->map_val, 0, 0);
migrate_enable();
@@ -4863,6 +4866,7 @@ static void bpf_rcu_run_callback(struct rcu_head *rcu)
smp_store_release(&rh->armed, 0);
rcu_read_lock_dont_migrate();
+ bpf_prog_begin_run(prog);
callback_fn((u64)(long)map, (u64)(long)key, (u64)(long)value, 0, 0);
rcu_read_unlock_migrate();
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog
2026-10-26 2:58 ` [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog Alexei Starovoitov
@ 2026-10-26 12:50 ` kernel test robot
2026-10-26 14:43 ` kernel test robot
2026-10-26 17:35 ` Kumar Kartikeya Dwivedi
2 siblings, 0 replies; 144+ messages in thread
From: kernel test robot @ 2026-10-26 12:50 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: oe-kbuild-all, daniel, andrii, eddyz87, memxor
Hi Alexei,
kernel test robot noticed the following build warnings:
[auto build test WARNING on bpf/master]
url: https://github.com/intel-lab-lkp/linux/commits/Alexei-Starovoitov/selftests-bpf-Remove-tests-for-global-data-in-arena/20261026-025822
base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git master
patch link: https://lore.kernel.org/r/20261026025943.404450-8-alexei.starovoitov%40gmail.com
patch subject: [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog
config: csky-defconfig (https://download.01.org/0day-ci/archive/20261026/202610262044.OQsV3p74-lkp@intel.com/config)
compiler: csky-linux-gcc (GCC) 16.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261026/202610262044.OQsV3p74-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202610262044.OQsV3p74-lkp@intel.com/
All warnings (new ones prefixed by >>):
>> kernel/bpf/core.c:3633:6: warning: no previous prototype for 'bpf_prog_begin_run' [-Wmissing-prototypes]
3633 | void bpf_prog_begin_run(const struct bpf_prog *prog)
| ^~~~~~~~~~~~~~~~~~
vim +/bpf_prog_begin_run +3633 kernel/bpf/core.c
3628
3629 /*
3630 * The kernel is about to call an async callback of @prog on this cpu: a run begins.
3631 * The main prog does the same in its first insns.
3632 */
> 3633 void bpf_prog_begin_run(const struct bpf_prog *prog)
3634 {
3635 struct bpf_prog_aux *aux;
3636
3637 if (!prog)
3638 return;
3639 aux = prog->aux->main_prog_aux;
3640 if (aux->may_goto_budget) {
3641 this_cpu_write(aux->may_goto_budget->start, 0);
3642 this_cpu_write(aux->may_goto_budget->count, BPF_MAX_TIMED_LOOPS);
3643 }
3644 }
3645
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 144+ messages in thread* Re: [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog
2026-10-26 2:58 ` [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog Alexei Starovoitov
2026-10-26 12:50 ` kernel test robot
@ 2026-10-26 14:43 ` kernel test robot
2026-10-26 17:35 ` Kumar Kartikeya Dwivedi
2 siblings, 0 replies; 144+ messages in thread
From: kernel test robot @ 2026-10-26 14:43 UTC (permalink / raw)
To: Alexei Starovoitov, bpf
Cc: llvm, oe-kbuild-all, daniel, andrii, eddyz87, memxor
Hi Alexei,
kernel test robot noticed the following build warnings:
[auto build test WARNING on bpf/master]
url: https://github.com/intel-lab-lkp/linux/commits/Alexei-Starovoitov/selftests-bpf-Remove-tests-for-global-data-in-arena/20261026-025822
base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git master
patch link: https://lore.kernel.org/r/20261026025943.404450-8-alexei.starovoitov%40gmail.com
patch subject: [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog
config: um-defconfig (https://download.01.org/0day-ci/archive/20261026/202610262241.4DF09B1A-lkp@intel.com/config)
compiler: clang version 24.0.0git (https://github.com/llvm/llvm-project 186e70f4e6a1521991d2340d3d49b23b1beff11f)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261026/202610262241.4DF09B1A-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202610262241.4DF09B1A-lkp@intel.com/
All warnings (new ones prefixed by >>):
>> kernel/bpf/core.c:3633:6: warning: no previous prototype for function 'bpf_prog_begin_run' [-Wmissing-prototypes]
3633 | void bpf_prog_begin_run(const struct bpf_prog *prog)
| ^
kernel/bpf/core.c:3633:1: note: declare 'static' if the function is not intended to be used outside of this translation unit
3633 | void bpf_prog_begin_run(const struct bpf_prog *prog)
| ^
| static
1 warning generated.
vim +/bpf_prog_begin_run +3633 kernel/bpf/core.c
3628
3629 /*
3630 * The kernel is about to call an async callback of @prog on this cpu: a run begins.
3631 * The main prog does the same in its first insns.
3632 */
> 3633 void bpf_prog_begin_run(const struct bpf_prog *prog)
3634 {
3635 struct bpf_prog_aux *aux;
3636
3637 if (!prog)
3638 return;
3639 aux = prog->aux->main_prog_aux;
3640 if (aux->may_goto_budget) {
3641 this_cpu_write(aux->may_goto_budget->start, 0);
3642 this_cpu_write(aux->may_goto_budget->count, BPF_MAX_TIMED_LOOPS);
3643 }
3644 }
3645
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 144+ messages in thread* Re: [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog
2026-10-26 2:58 ` [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog Alexei Starovoitov
2026-10-26 12:50 ` kernel test robot
2026-10-26 14:43 ` kernel test robot
@ 2026-10-26 17:35 ` Kumar Kartikeya Dwivedi
2026-10-27 16:39 ` Alexei Starovoitov
2 siblings, 1 reply; 144+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-10-26 17:35 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: bpf, daniel, andrii, eddyz87
On Mon, 26 Oct 2026 at 04:00, Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> From: Alexei Starovoitov <ast@kernel.org>
>
> may_goto stops a loop that the verifier cannot bound: after 250 ms the loop
> must exit. The count and the timestamp are in the stack frame of the
> function that contains the loop. Thus the limit applies to one call of a
> subprog, not to one run of the prog. Loops that nest across calls multiply
> the limit.
>
> main: sub:
> 1: call sub 1: may_goto 2f
> may_goto 2f goto 1b
> goto 1b 2: r0 = 0
> 2: r0 = 1 exit
> exit
>
> main | sub: 250 ms | sub: 250 ms | ... | sub: 250 ms |
> 1 2 65535 ^
> main reads the clock here
>
> Each call of sub spins for 250 ms. main reads the clock one time in
> 65535 iterations. Thus the prog runs for 65535 * 250 ms: 9 insns cause a
> soft lockup. bpf_loop(1 << 23, cb) with such a loop in cb has the same
> result.
>
> The count has the same problem. A subprog that makes fewer than
> 65535 iterations in each call never reads the clock. An endless loop that
> calls a subprog with 60000 iterations runs for 3.5 sec. Three levels of
> such loops lock up a cpu.
>
> Give one time slice and one count to the full run. A run is one call of the
> main prog, or of an async callback, by the kernel, together with all the
> subprogs that it calls. The main prog gets a per-cpu budget.
>
> the kernel calls the prog budget.start = 0
> | budget.count = 65535
> v
> entry of a subprog frame.count = budget.count
> |
> v
> may_goto frame.count -= 1
> | At 0, read the clock:
> | < 250 ms refill frame.count, budget.count
> | >= 250 ms budget.count = 0, exit the loop
> v
> exit of the subprog budget.count = min(budget.count, frame.count)
>
> Iterations now add up across calls. After 250 ms the budget is empty:
> - A subprog that starts after this time exits its loops immediately.
> - A frame that is already on the stack exits its loop when it reads the
> clock again.
>
> The progs above return after 250 ms.
>
> Only JITed progs that have may_goto in a subprog or in a callback get
> different code: 0 of 144 progs of Cilium, 53 of 495 progs of scx. The cost
> on x86-64 (Zen 5) with hot caches, in nsec:
>
> entry of the main prog, once per run 4 insns, 34 bytes +0.52
> entry of a subprog with may_goto 3 insns, 19 bytes +0.03
> each exit of such a subprog 6 insns, 36 bytes +0.15
> async callback (bpf_timer), all progs reset the budget +0.35 on 34.6
> read of the clock, once in 65535 stack walk to find 26 -> 179
> iterations, all progs the budget
>
> An iteration has no added cost. A run with one call of such a subprog costs
> +0.97 nsec. It costs +90 nsec when the budget is not in the cache.
>
> Limitations:
> - A second run of the same prog on the same cpu, in an interrupt or after
> preemption, starts the time slice again. The timestamp in the frame
> continues to limit each frame to 250 ms.
> - The JIT must support a stack walk and per-cpu insns. x86-64 with ORC,
> arm64 and powerpc64 do. Nothing changes on riscv, loongarch and s390.
> - Nothing changes for extension progs. They cannot find the start of a
> run.
>
> Fixes: e723608bf428 ("bpf: Add verifier support for timed may_goto")
> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
> ---
Why not copy the state from the caller's frame into the callee iff it
uses may_goto? At least it will work correctly without various holes
listed here in the limitations.
^ permalink raw reply [flat|nested] 144+ messages in thread* Re: [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog
2026-10-26 17:35 ` Kumar Kartikeya Dwivedi
@ 2026-10-27 16:39 ` Alexei Starovoitov
0 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-27 16:39 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi; +Cc: bpf, daniel, andrii, eddyz87
On Mon, Oct 26, 2026 at 10:35 AM Kumar Kartikeya Dwivedi <memxor@gmail.com> wrote:
>> Limitations:
>> - A second run of the same prog on the same cpu, in an interrupt or after
>> preemption, starts the time slice again. The timestamp in the frame
>> continues to limit each frame to 250 ms.
>> - The JIT must support a stack walk and per-cpu insns. x86-64 with ORC,
>> arm64 and powerpc64 do. Nothing changes on riscv, loongarch and s390.
>> - Nothing changes for extension progs. They cannot find the start of a
>> run.
[...]
> Why not copy the state from the caller's frame into the callee iff it
> uses may_goto? At least it will work correctly without various holes
> listed here in the limitations.
Because of callbacks. The caller of cb is C code. There is no bpf frame to
copy from.
Direct calls are hard too. Would need some hidden arg to be passed,
so that callee knows where to copy from.
We've been sitting on this issue for long time. I felt we have to
close this hole now and improve later if we can think of a better way.
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 08/81] selftests/bpf: Test may_goto in a loop that calls a subprog that spins
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (6 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 07/81] bpf: One time slice of may_goto for a run of a prog Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 09/81] bpf: Clear delta of a register that scalar evolution widens Alexei Starovoitov
` (74 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
verifier_may_goto_1 gets the two cases of
"bpf: One time slice of may_goto for a run of a prog":
- A loop calls a subprog that spins.
- Three levels of loops. The subprogs make 60000 iterations in each call.
Thus they never read the clock.
Without the fix the two tests lock up a cpu. The fix has no effect on s390,
riscv, loongarch and x86-64 without ORC. Enable the tests on x86-64 and
arm64 only.
may_goto_budget makes sure that an async callback is a new run. The prog
uses its full time slice and starts a timer. The timer callback calls a
subprog, and this subprog must loop.
cond_break5 expected that a subprog loops after the main prog used the full
time slice. The fix changes that. Call the subprog first.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../bpf/prog_tests/may_goto_budget.c | 31 ++++++
.../selftests/bpf/progs/may_goto_budget.c | 73 ++++++++++++++
.../bpf/progs/verifier_iterating_callbacks.c | 8 +-
.../selftests/bpf/progs/verifier_may_goto_1.c | 97 +++++++++++++++++++
4 files changed, 207 insertions(+), 2 deletions(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/may_goto_budget.c
create mode 100644 tools/testing/selftests/bpf/progs/may_goto_budget.c
diff --git a/tools/testing/selftests/bpf/prog_tests/may_goto_budget.c b/tools/testing/selftests/bpf/prog_tests/may_goto_budget.c
new file mode 100644
index 000000000000..c69af272644f
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/may_goto_budget.c
@@ -0,0 +1,31 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include <network_helpers.h>
+#include "may_goto_budget.skel.h"
+
+void test_may_goto_budget(void)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, opts,
+ .data_in = &pkt_v4,
+ .data_size_in = sizeof(pkt_v4),
+ .repeat = 1,
+ );
+ struct may_goto_budget *skel;
+ int i;
+
+ skel = may_goto_budget__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+
+ ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(skel->progs.timer_after_timeout), &opts),
+ "test_run");
+ ASSERT_EQ(opts.retval, 0, "retval");
+ ASSERT_GT(skel->bss->spun, 1, "spun");
+
+ for (i = 0; i < 100 && !skel->bss->cb_done; i++)
+ usleep(10000);
+ ASSERT_EQ(skel->bss->cb_done, 1, "cb_done");
+ ASSERT_EQ(skel->bss->cb_cnt, 1000, "cb_cnt");
+
+ may_goto_budget__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/may_goto_budget.c b/tools/testing/selftests/bpf/progs/may_goto_budget.c
new file mode 100644
index 000000000000..28960aeb5c05
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/may_goto_budget.c
@@ -0,0 +1,73 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+
+char _license[] SEC("license") = "GPL";
+
+#define CLOCK_MONOTONIC 1
+
+struct elem {
+ struct bpf_timer t;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, struct elem);
+} timers SEC(".maps");
+
+int zero, spun, cb_cnt, cb_done;
+
+static __noinline int spins(void)
+{
+ int cnt = zero;
+
+ for (;;) {
+ cond_break;
+ cnt++;
+ }
+ return cnt;
+}
+
+static __noinline int counts_to_1000(void)
+{
+ int i, cnt = zero;
+
+ for (i = zero; i < 1000; i++) {
+ cond_break;
+ cnt++;
+ }
+ return cnt;
+}
+
+static int timer_cb(void *map, int *key, struct elem *e)
+{
+ cb_cnt = counts_to_1000();
+ cb_done = 1;
+ return 0;
+}
+
+/*
+ * The time slice of this run is used up when the timer is started. The callback is another
+ * run on the same cpu, with a time slice of its own.
+ */
+SEC("tc")
+int timer_after_timeout(void *ctx)
+{
+ struct elem *e;
+ int key = 0;
+
+ e = bpf_map_lookup_elem(&timers, &key);
+ if (!e)
+ return 1;
+ if (bpf_timer_init(&e->t, &timers, CLOCK_MONOTONIC))
+ return 2;
+ if (bpf_timer_set_callback(&e->t, timer_cb))
+ return 3;
+ spun = spins();
+ if (bpf_timer_start(&e->t, 0, BPF_F_TIMER_CPU_PIN))
+ return 4;
+ return 0;
+}
diff --git a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
index 1fbcc5228306..a76bce7278e7 100644
--- a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
+++ b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
@@ -553,13 +553,17 @@ int cond_break5(const void *ctx)
{
int cnt1 = zero, cnt2;
+ /*
+ * The subprog first. There is one time slice for a run of the prog: a subprog that
+ * is called when it's used up doesn't loop at all.
+ */
+ cnt2 = static_subprog();
+
for (;;) {
cond_break;
cnt1++;
}
- cnt2 = static_subprog();
-
/* main and subprog have to loop a bit */
return cnt1 > 1 && cnt2 > 1 ? 1 : 0;
}
diff --git a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c
index faf9eec0a63e..6fe602c8c7f3 100644
--- a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c
+++ b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c
@@ -294,4 +294,101 @@ __naked void may_goto_subprog_func_first(void)
: __clobber_all);
}
+/*
+ * There is one time slice for a run of the prog, not one for every call of a subprog.
+ * The subprog uses it up and returns. Its caller calls it again, BPF_MAX_TIMED_LOOPS times
+ * before it looks at the clock.
+ */
+__naked __noinline __used
+static unsigned long spins_on_may_goto(void)
+{
+ asm volatile (
+ "1:"
+ ".8byte %[may_goto_1];"
+ "goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(may_goto_1, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 1, 0))
+ );
+}
+
+SEC("socket")
+__description("may_goto in a loop that calls a subprog that spins on may_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_calls_subprog_with_may_goto(void)
+{
+ asm volatile (
+ "1:"
+ "call spins_on_may_goto;"
+ ".8byte %[may_goto_1];"
+ "goto 1b;"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_insn(may_goto_1, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 1, 0))
+ : __clobber_all
+ );
+}
+
+/*
+ * The count is shared too. These subprogs make 60000 iterations per call, fewer than
+ * BPF_MAX_TIMED_LOOPS: with a count of its own for every call none of them looks at the clock,
+ * and their caller does after 65535 * 60000 * 60000 iterations.
+ */
+__naked __noinline __used
+static unsigned long loops_60000(void)
+{
+ asm volatile (
+ "r1 = 60000;"
+ "1:"
+ ".8byte %[may_goto_2];"
+ "r1 += -1;"
+ "if r1 != 0 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(may_goto_2, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 2, 0))
+ );
+}
+
+__naked __noinline __used
+static unsigned long calls_60000_times(void)
+{
+ asm volatile (
+ "r6 = 60000;"
+ "1:"
+ "call loops_60000;"
+ ".8byte %[may_goto_2];"
+ "r6 += -1;"
+ "if r6 != 0 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(may_goto_2, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 2, 0))
+ );
+}
+
+SEC("socket")
+__description("may_goto in three levels of loops that end before they look at the clock")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void three_levels_of_short_loops(void)
+{
+ asm volatile (
+ "1:"
+ "call calls_60000_times;"
+ ".8byte %[may_goto_1];"
+ "goto 1b;"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_insn(may_goto_1, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 1, 0))
+ : __clobber_all
+ );
+}
+
char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 09/81] bpf: Clear delta of a register that scalar evolution widens
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (7 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 08/81] selftests/bpf: Test may_goto in a loop that calls a subprog that spins Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 20:09 ` Daniel Borkmann
2026-10-26 2:58 ` [PATCH bpf 10/81] selftests/bpf: Test that a widened register is not linked Alexei Starovoitov
` (73 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
After 'r3 = r8; r3 += 4' the verifier knows that r3 == r8 + 4. The two
registers have the same id, and r3 keeps the difference in reg->delta. When
the prog compares one of them with a constant, sync_linked_regs()
calculates the range of the other register from the difference of the two
deltas.
At a loop head scalar evolution widens a register: it replaces the register
with its range over all iterations. Then the register is not linked to
other registers. scratch_widened_reg_id() clears reg->id, but it does not
clear reg->delta. A subsequent copy of the register gets the stale delta.
Then the range of the copy is incorrect by this delta.
r3 = r8
r3 += 4 ; R3=scalar(id=1+4)
1: ; r3 is widened: no id, delta is still 4
r4 = r3 ; id=2 and delta 4 in both
r4 += 1 ; R4=scalar(id=2+1)
if r3 > 10 goto 3f ; R3 <= 10, R4 = R3 + 1 - 4 <= 7
if r4 <= 7 goto 2f
... ; not verified, r4 is up to 11 here
The verifier does not verify code that can run. Clear delta together with
id.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/scev.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index fea77772ca96..b321bfee36c5 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2390,6 +2390,7 @@ static void scratch_widened_reg_id(struct bpf_verifier_env *env, struct bpf_reg_
switch (base_type(reg->type)) {
case SCALAR_VALUE:
reg->id = 0;
+ reg->delta = 0;
break;
case PTR_TO_PACKET:
case PTR_TO_PACKET_META:
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 09/81] bpf: Clear delta of a register that scalar evolution widens
2026-10-26 2:58 ` [PATCH bpf 09/81] bpf: Clear delta of a register that scalar evolution widens Alexei Starovoitov
@ 2026-10-26 20:09 ` Daniel Borkmann
2026-10-27 21:23 ` Alexei Starovoitov
0 siblings, 1 reply; 144+ messages in thread
From: Daniel Borkmann @ 2026-10-26 20:09 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: andrii, eddyz87, memxor
On 10/26/26 3:58 AM, Alexei Starovoitov wrote:
> From: Alexei Starovoitov <ast@kernel.org>
>
> After 'r3 = r8; r3 += 4' the verifier knows that r3 == r8 + 4. The two
> registers have the same id, and r3 keeps the difference in reg->delta. When
> the prog compares one of them with a constant, sync_linked_regs()
> calculates the range of the other register from the difference of the two
> deltas.
>
> At a loop head scalar evolution widens a register: it replaces the register
> with its range over all iterations. Then the register is not linked to
> other registers. scratch_widened_reg_id() clears reg->id, but it does not
> clear reg->delta. A subsequent copy of the register gets the stale delta.
> Then the range of the copy is incorrect by this delta.
>
> r3 = r8
> r3 += 4 ; R3=scalar(id=1+4)
> 1: ; r3 is widened: no id, delta is still 4
> r4 = r3 ; id=2 and delta 4 in both
> r4 += 1 ; R4=scalar(id=2+1)
> if r3 > 10 goto 3f ; R3 <= 10, R4 = R3 + 1 - 4 <= 7
> if r4 <= 7 goto 2f
> ... ; not verified, r4 is up to 11 here
>
> The verifier does not verify code that can run. Clear delta together with
> id.
>
> Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
> ---
> kernel/bpf/scev.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index fea77772ca96..b321bfee36c5 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c
> @@ -2390,6 +2390,7 @@ static void scratch_widened_reg_id(struct bpf_verifier_env *env, struct bpf_reg_
> switch (base_type(reg->type)) {
> case SCALAR_VALUE:
> reg->id = 0;
> + reg->delta = 0;
small nit when you respin for bpf tree: clear_scalar_id
> break;
> case PTR_TO_PACKET:
> case PTR_TO_PACKET_META:
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 10/81] selftests/bpf: Test that a widened register is not linked
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (8 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 09/81] bpf: Clear delta of a register that scalar evolution widens Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 11/81] selftests/bpf: Wait for the last maps of map_kptr to be freed Alexei Starovoitov
` (72 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Scalar evolution widens a register that has delta 4. Then the prog copies
the register. With the stale delta the verifier finds that a read of
uninitialized r9 is dead code, and it loads the prog. See
"bpf: Clear delta of a register that scalar evolution widens".
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 34 +++++++++++++++++++
1 file changed, 34 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 661ce9ca8f93..c41f33ee6c70 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -1377,6 +1377,40 @@ __naked __noinline __used static int probe_read_stack(void)
: __clobber_all);
}
+/*
+ * r3 = r8 + 4 is linked to r8 when it enters the loop. Widening drops the link. If
+ * it leaves the 4 behind then r4 = r3 + 1 is taken for r3 - 3 when r3 is compared:
+ * r4 <= 7 instead of r4 <= 11, and the read of r9 looks like dead code.
+ */
+SEC("socket")
+__failure __msg("R9 !read_ok")
+__naked void widened_reg_is_not_linked(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "r8 = r0;"
+ "r8 &= 0xff;"
+ "r3 = r8;"
+ "r3 += 4;"
+ "r1 = 0;"
+"1:"
+ "r4 = r3;"
+ "r4 += 1;"
+ "if r3 > 10 goto 3f;"
+ "if r4 <= 7 goto 2f;"
+ "r0 = *(u64 *)(r9 + 0);"
+"2:"
+ "r3 += 1;"
+ "r1 += 1;"
+ "if r1 < 2 goto 1b;"
+"3:"
+ "r0 = r8;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
/* The unknown SCEV of an unbound STACK_MISC slot must not prevent widening. */
SEC("tracepoint")
__success
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 11/81] selftests/bpf: Wait for the last maps of map_kptr to be freed
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (9 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 10/81] selftests/bpf: Test that a widened register is not linked Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 12/81] bpf: Don't widen a pointer to a buffer in scalar evolution Alexei Starovoitov
` (71 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
map_kptr_race/htab_leak fails occasionally in a complete run of test_progs:
test_htab_leak:FAIL:htab refcount unexpected htab refcount:
actual 6 != expected 2
This is a race between two tests. It is not a leak in the kernel.
- map_kptr runs immediately before htab_leak. The two tests examine the
refcount of the same test object.
- The kernel frees maps asynchronously. Thus the maps drop their
references some time after close().
- map_kptr waits for that after each subtest, but not after the last one.
Wait after the last subtest too.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/prog_tests/map_kptr.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/tools/testing/selftests/bpf/prog_tests/map_kptr.c b/tools/testing/selftests/bpf/prog_tests/map_kptr.c
index 17e707dddda8..f103f8cd458b 100644
--- a/tools/testing/selftests/bpf/prog_tests/map_kptr.c
+++ b/tools/testing/selftests/bpf/prog_tests/map_kptr.c
@@ -225,6 +225,11 @@ void serial_test_map_kptr(void)
/* Observe refcount dropping to 1 on map release. */
test_map_kptr_success(true);
+
+ /* The next test counts the same references */
+ ASSERT_OK(kern_sync_rcu_tasks_trace(skel), "sync rcu_tasks_trace");
+ ASSERT_OK(kern_sync_rcu(), "sync rcu");
+ wait_for_map_release();
}
rcu_tasks_trace_gp__destroy(skel);
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 12/81] bpf: Don't widen a pointer to a buffer in scalar evolution
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (10 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 11/81] selftests/bpf: Wait for the last maps of map_kptr to be freed Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 13/81] selftests/bpf: Test a loop that moves a pointer to a buffer Alexei Starovoitov
` (70 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Scalar evolution widens counted loops. Since that change the verifier
rejects a loop over the per-cpu values of a map element:
SEC("iter/bpf_map_elem")
...
for (i = 0; i < n; i++) {
out[i] = *(__u32 *)pptr;
pptr += 8;
}
loop header at 12, widening r2 to 0..56 step 8
R2 invalid variable buffer offset: off=0, var_off=(0x0; 0x38)
pptr is a pointer to a buffer (PTR_TO_BUF). Scalar evolution replaces its
offset with the range over all iterations. But __check_buffer_access()
accepts only a constant offset. Thus a prog cannot use a widened
PTR_TO_BUF.
Do not widen it. Then the verifier verifies the loop one iteration at a
time, as it did before.
loop before SCEV with SCEV this patch
-------------------------------- ----------- ---------- ----------
counts in i, reads through pptr loads rejected loads
counts in p ('p < end'), no read 1M insns loads 1M insns
counts in p, reads through p 1M insns rejected 1M insns
The verifier cannot decide a comparison of two pointers to a buffer. Thus
it follows 'p < end' until the limit of 1M insns.
bpf_iter_bpf_percpu_array_map.c and bpf_iter_bpf_percpu_hash_map.c have the
first loop. They load today only because they also add up the values.
Scalar evolution has no formula for the sum, and in that case it widens
nothing.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/scev.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index b321bfee36c5..4266651e3fe4 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2377,7 +2377,6 @@ static bool is_widenable_reg_type(const struct bpf_reg_state *reg)
case PTR_TO_PACKET:
case PTR_TO_PACKET_META:
case PTR_TO_MEM:
- case PTR_TO_BUF:
case PTR_TO_BTF_ID:
return true;
default:
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 13/81] selftests/bpf: Test a loop that moves a pointer to a buffer
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (11 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 12/81] bpf: Don't widen a pointer to a buffer in scalar evolution Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 14/81] bpf: Fix a false verifier bug in the clamp of scalar evolution Alexei Starovoitov
` (69 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
ptr_buf moves one pointer to a buffer until it gets to a second pointer. It
never reads through the pointer. It loads only if scalar evolution widens
the pointer, and that is no longer the case.
Change it to the loop that real progs have, for example
bpf_iter_bpf_percpu_array_map.c. It counts in a register and reads through
the pointer in each iteration. Without
"bpf: Don't widen a pointer to a buffer in scalar evolution" the verifier
rejects it with "invalid variable buffer offset".
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/progs/verifier_scev.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index c41f33ee6c70..504cf8230625 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -3255,21 +3255,23 @@ __naked void ptr_mem_other_reservation(void)
: __clobber_all);
}
+/* An access to a buffer at a variable offset is rejected: the pointer must not be widened. */
SEC("iter/bpf_map_elem")
__success
__log_level(2)
__flag(BPF_F_TEST_STATE_FREQ)
__msg("header_count is 4 ")
-__msg("widening r6 to 0..3 step 1")
+__msg("can't widen r6, expr is (+ r6 8)")
__naked void ptr_buf(void)
{
asm volatile (" \
r6 = *(u64 *)(r1 + %[value]); \
if r6 == 0 goto 2f; \
- r7 = r6; \
- r7 += 4; \
-1: r6 += 1; \
- if r6 < r7 goto 1b; \
+ r7 = 0; \
+1: r0 = *(u32 *)(r6 + 0); \
+ r6 += 8; \
+ r7 += 1; \
+ if r7 < 4 goto 1b; \
2: r0 = 0; \
exit; \
" :
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 14/81] bpf: Fix a false verifier bug in the clamp of scalar evolution
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (12 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 13/81] selftests/bpf: Test a loop that moves a pointer to a buffer Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 23:45 ` Eduard Zingerman
2026-10-26 2:58 ` [PATCH bpf 15/81] selftests/bpf: Test induction variables that come back out of range Alexei Starovoitov
` (68 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier rejects a valid prog with -EFAULT and a WARN:
r6 = 0
r7 = 0
1:
if r7 != 9 goto 2f
r7 += 1
r6 += 1
if r6 < 10 goto 1b
2:
exit
verifier bug: scev clamp produced empty range for r7
Scalar evolution describes each variable of a loop separately. It does not
know that r6 and r7 are always equal.
r6 r7
a run of the prog, at 1: k k
the verifier, at 1: 0..9 0..9
after 'if r7 != 9': 0..9 9 r6 < 9 never occurs
back at 1: 1..9 10 not in 0..9
When a state comes back to the loop head, bpf_clamp_scev_regs() intersects
each variable with its calculated range. For r7 the result is empty, and
the verifier reports a bug.
A range has three parts: 64 bits, 32 bits and the step. The verifier
examines only the 64 bits. When a different part makes the result empty,
the prog gets a second message:
verifier bug: REG INVARIANTS VIOLATION (bpf_set_reg_range)
This is not a bug. No run of the prog gets to this state. Thus each range
is correct for r7. Give r7 the range that it has at the loop head, and
continue. The verifier continues to compare the other parts of the state
with the state at the loop head. Thus it does not hide a real problem on
such a path.
Each pair of the three parts can have numbers in common when all three have
none. Thus new bpf_reg_range_is_empty() examines the three parts together.
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 1 +
kernel/bpf/scev.c | 19 ++++++++++++++-----
kernel/bpf/verifier.c | 19 +++++++++++++++++++
3 files changed, 34 insertions(+), 5 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index de301b32015d..7030b997ce2c 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1895,6 +1895,7 @@ bool bpf_same_memory_origin(const struct bpf_reg_state *reg_a,
const struct bpf_reg_state *reg_b);
int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
struct cnum64 r64, struct cnum32 r32, u16 base, u16 step);
+bool bpf_reg_range_is_empty(struct cnum64 r64, struct cnum32 r32, u16 base, u16 step);
bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_reg_state *b);
int bpf_reg_union(struct bpf_verifier_env *env, struct bpf_reg_state *acc,
const struct bpf_reg_state *src);
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 4266651e3fe4..6330deb2d3f2 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -2869,6 +2869,8 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
struct linear_coefs coefs;
struct env *header_env;
struct bounds bounds;
+ struct cnum64 r64;
+ struct cnum32 r32;
u32 r;
int err;
@@ -2892,11 +2894,18 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
continue;
linear_bounds(entry_reg, max_header_count, &coefs, &bounds);
- bounds.r64 = cnum64_intersect(reg->r64, bounds.r64);
- bounds.r32 = cnum32_intersect(reg->r32, bounds.r32);
- if (cnum64_is_empty(bounds.r64)) {
- verifier_bug(env, "scev clamp produced empty range for r%d", r);
- return -EFAULT;
+ r64 = cnum64_intersect(reg->r64, bounds.r64);
+ r32 = cnum32_intersect(reg->r32, bounds.r32);
+ /*
+ * The state at the head doesn't know how the variables depend on each other.
+ * It lets r6 = 0, r7 = 9 into the loop when both count from 0, and r7 = 10
+ * comes back. No run gets here like that, so any range is right for r7.
+ * Take the one it has at the head: the rest of the state is still compared
+ * with the head, and what was pruned on the way here relies on that.
+ */
+ if (!bpf_reg_range_is_empty(r64, r32, bounds.base, bounds.step)) {
+ bounds.r64 = r64;
+ bounds.r32 = r32;
}
if (log->level & BPF_LOG_LEVEL2) {
bpf_log(log, "loop header at %d, clamping ", insn_idx);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index c93ef399c420..84f722bac539 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -17739,6 +17739,25 @@ int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
return reg_bounds_sanity_check(env, reg, "bpf_set_reg_range");
}
+/* Is there no number that is in @r64, in @r32 and equal to @base + k * @step? */
+bool bpf_reg_range_is_empty(struct cnum64 r64, struct cnum32 r32, u16 base, u16 step)
+{
+ struct bpf_reg_state reg = {
+ .type = SCALAR_VALUE,
+ .r64 = cnum64_intersect_linear(r64, base, step),
+ .r32 = r32,
+ .base = base,
+ .step = step,
+ .var_off = tnum_unknown,
+ };
+
+ if (range_bounds_violation(®))
+ return true;
+ /* Each pair of the three can have numbers in common when all three have none. */
+ reg_bounds_sync(®);
+ return range_bounds_violation(®);
+}
+
/*
* Checks if pointer and packet metadata matches between two registers.
* For use with bpf_reg_union().
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 14/81] bpf: Fix a false verifier bug in the clamp of scalar evolution
2026-10-26 2:58 ` [PATCH bpf 14/81] bpf: Fix a false verifier bug in the clamp of scalar evolution Alexei Starovoitov
@ 2026-10-26 23:45 ` Eduard Zingerman
2026-10-27 16:32 ` Alexei Starovoitov
0 siblings, 1 reply; 144+ messages in thread
From: Eduard Zingerman @ 2026-10-26 23:45 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: daniel, andrii, memxor
On Mon, 2026-10-26 at 02:58 +0000, Alexei Starovoitov wrote:
...
> @@ -2869,6 +2869,8 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
> struct linear_coefs coefs;
> struct env *header_env;
> struct bounds bounds;
> + struct cnum64 r64;
> + struct cnum32 r32;
> u32 r;
> int err;
>
> @@ -2892,11 +2894,18 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
> continue;
>
> linear_bounds(entry_reg, max_header_count, &coefs, &bounds);
> - bounds.r64 = cnum64_intersect(reg->r64, bounds.r64);
> - bounds.r32 = cnum32_intersect(reg->r32, bounds.r32);
> - if (cnum64_is_empty(bounds.r64)) {
> - verifier_bug(env, "scev clamp produced empty range for r%d", r);
> - return -EFAULT;
> + r64 = cnum64_intersect(reg->r64, bounds.r64);
> + r32 = cnum32_intersect(reg->r32, bounds.r32);
> + /*
> + * The state at the head doesn't know how the variables depend on each other.
> + * It lets r6 = 0, r7 = 9 into the loop when both count from 0, and r7 = 10
> + * comes back. No run gets here like that, so any range is right for r7.
> + * Take the one it has at the head: the rest of the state is still compared
> + * with the head, and what was pruned on the way here relies on that.
> + */
> + if (!bpf_reg_range_is_empty(r64, r32, bounds.base, bounds.step)) {
> + bounds.r64 = r64;
> + bounds.r32 = r32;
> }
> if (log->level & BPF_LOG_LEVEL2) {
> bpf_log(log, "loop header at %d, clamping ", insn_idx);
I'd suggest addressing this as in [1,2]. I'd expect that selective
clamping would be a subject to counter-examples as well, just a bit
more convoluted.
[1] https://lore.kernel.org/bpf/20261026-scev-simpler-clamp-v2-2-f373aec9825f@gmail.com/
[2] https://lore.kernel.org/bpf/20261026-scev-simpler-clamp-v2-3-f373aec9825f@gmail.com/
...
^ permalink raw reply [flat|nested] 144+ messages in thread* Re: [PATCH bpf 14/81] bpf: Fix a false verifier bug in the clamp of scalar evolution
2026-10-26 23:45 ` Eduard Zingerman
@ 2026-10-27 16:32 ` Alexei Starovoitov
0 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-27 16:32 UTC (permalink / raw)
To: Eduard Zingerman, bpf; +Cc: daniel, andrii, memxor
On Mon, Oct 26, 2026 at 04:45 PM Eduard Zingerman <eddyz87@gmail.com> wrote:
>> + if (!bpf_reg_range_is_empty(r64, r32, bounds.base, bounds.step)) {
>> + bounds.r64 = r64;
>> + bounds.r32 = r32;
>> }
>
> I'd suggest addressing this as in [1,2]. I'd expect that selective
> clamping would be a subject to counter-examples as well, just a bit
> more convoluted.
fine. applied yours and rebase on top.
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 15/81] selftests/bpf: Test induction variables that come back out of range
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (13 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 14/81] bpf: Fix a false verifier bug in the clamp of scalar evolution Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 16/81] libbpf: Make arena 1 Gbyte when max_entries is not set Alexei Starovoitov
` (67 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
In each test two variables of a loop count together, and a jump in the loop
tests one of them. See
"bpf: Fix a false verifier bug in the clamp of scalar evolution".
clamp_out_of_range... the variable, or the problem result
---------------------- --------------------------------------- ------
(no suffix) a register loads
_1m 1M iterations loads
_step has a step loads
_stack a stack slot loads
_index an index into a map value loads
_32 out of range in 32 bits only loads
_off_step in range, but does not fit the step loads
_const_off_step the same, and it is a constant loads
_64_and_32 no number is in the two ranges loads
_64_32_and_step only the three parts together are empty loads
_leak the prog leaks a reference reject
_two_entries the prog writes to freed memory reject
The last two progs have a problem in a part of the state that is not
clamped. The verifier must find it, although the path to the problem goes
through a state that never occurs.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 331 ++++++++++++++++++
1 file changed, 331 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 504cf8230625..61bb39a12de2 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -3484,4 +3484,335 @@ __naked void dominating_exit_bounds_count(void)
" ::: __clobber_all);
}
+/*
+ * r6 and r7 count together. At the head of the loop both are 0..9 and nothing says
+ * that they are equal: r7 = 9 gets through the first jump with r6 = 0..9 and
+ * r7 = 10 comes back with r6 = 1..9. There is no such iteration.
+ */
+SEC("socket")
+__success __retval(0)
+__log_level(2)
+__msg("loop header at 2, widening r7 to 0..9 step 1")
+__msg("loop header at 2, clamping r7 to 0..9 step 1")
+__naked void clamp_out_of_range(void)
+{
+ asm volatile (
+ "r6 = 0;"
+ "r7 = 0;"
+"1:"
+ "if r7 != 9 goto 2f;"
+ "r7 += 1;"
+ "r6 += 1;"
+ "if r6 < 10 goto 1b;"
+"2:"
+ "r0 = r7;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* If r7 is left as it is it's out of range in every trip that follows. */
+SEC("socket")
+__success __retval(0)
+__naked void clamp_out_of_range_1m(void)
+{
+ asm volatile (
+ "r6 = 0;"
+ "r7 = 0;"
+"1:"
+ "if r7 < 999999 goto 2f;"
+ "r7 += 1;"
+ "r6 += 1;"
+ "if r6 < 1000000 goto 1b;"
+"2:"
+ "r0 = r7;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* r7 is 0 or 256 at the head, 256 goes on, 512 comes back. */
+SEC("socket")
+__success __retval(0)
+__naked void clamp_out_of_range_step(void)
+{
+ asm volatile (
+ "r7 = 0;"
+ "r1 = 0;"
+"1:"
+ "if r7 s<= 18 goto 2f;"
+ "r7 += 256;"
+ "r1 += 1;"
+ "if r1 < 2 goto 1b;"
+"2:"
+ "r0 = r7;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* The variable is in the stack. */
+SEC("socket")
+__success __retval(0)
+__naked void clamp_out_of_range_stack(void)
+{
+ asm volatile (
+ "r6 = 0;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+"1:"
+ "r1 = *(u64 *)(r10 - 8);"
+ "if r1 != 9 goto 2f;"
+ "r1 += 1;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r6 += 1;"
+ "if r6 < 10 goto 1b;"
+"2:"
+ "r0 = *(u64 *)(r10 - 8);"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* The 64-bit ranges have 1..0x7ffffff9 in common, the 32-bit ranges nothing. */
+SEC("socket")
+__success __retval(0)
+__flag(BPF_F_TEST_REG_INVARIANTS)
+__naked void clamp_out_of_range_32(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "r1 = r0;"
+ "if r1 > 3 goto 2f;"
+ "r3 = 0xffffffff7ffffffc ll;"
+ "r1 += r3;"
+ "r6 = 0;"
+"1:"
+ "if r1 & 0x80000007 goto 2f;"
+ "w1 += 1;"
+ "r6 += 1;"
+ "if r6 < 4 goto 1b;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/*
+ * r7 is 0, 3 .. 27 at the head. 'if r7 & 0x1a' forgets the step, 4 and 5 go on, 7 and 8 come
+ * back. They are in 0..27, but they are not a multiple of 3.
+ */
+SEC("socket")
+__success __retval(0)
+__flag(BPF_F_TEST_REG_INVARIANTS)
+__naked void clamp_out_of_range_off_step(void)
+{
+ asm volatile (
+ "r7 = 0;"
+ "r1 = 0;"
+"1:"
+ "if r7 & 0x1a goto 2f;"
+ "if r7 < 4 goto 2f;"
+ "r7 += 3;"
+ "r1 += 1;"
+ "if r1 < 10 goto 1b;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/*
+ * r7 is 0xfffffffb .. 0x100000004 at the head. 101 and 2^32 + 101 come back. The 64-bit ranges
+ * have numbers in common, the 32-bit ranges too, but no number is in both.
+ */
+SEC("socket")
+__success __retval(0)
+__flag(BPF_F_TEST_REG_INVARIANTS)
+__naked void clamp_out_of_range_64_and_32(void)
+{
+ asm volatile (
+ "r7 = 0xfffffffb ll;"
+ "r1 = 0;"
+"1:"
+ "if r7 & 0x200 goto 2f;"
+ "if w7 != 100 goto 2f;"
+ "r7 += 1;"
+ "r1 += 1;"
+ "if r1 < 10 goto 1b;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/*
+ * r7 is 2^33 - 5 + 3 * k at the head. 2^33 + 5 and 2^33 + 6 are in the 64-bit range and in the
+ * 32-bit range. 2^33 + 4 and 2^33 + 7 fit the step. Only all three together have no number.
+ */
+SEC("socket")
+__success __retval(0)
+__flag(BPF_F_TEST_REG_INVARIANTS)
+__naked void clamp_out_of_range_64_32_and_step(void)
+{
+ asm volatile (
+ "r7 = 0x1fffffffb ll;"
+ "r1 = 0;"
+"1:"
+ "if r7 & 0x200 goto 2f;"
+ "if w7 > 3 goto 2f;"
+ "if w7 < 2 goto 2f;"
+ "r7 += 3;"
+ "r1 += 1;"
+ "if r1 < 10 goto 1b;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* 8 comes back. It is in the range of the head and not a multiple of 3. It must not go on alone. */
+SEC("socket")
+__success __retval(0)
+__flag(BPF_F_TEST_REG_INVARIANTS)
+__naked void clamp_out_of_range_const_off_step(void)
+{
+ asm volatile (
+ "r7 = 0;"
+ "r1 = 0;"
+ "r6 = 0;"
+"1:"
+ "if r7 & 16 goto 3f;"
+ "if r7 != 5 goto 3f;"
+ "r6 = 1;"
+"3:"
+ "r7 += 3;"
+ "r1 += 1;"
+ "if r1 < 1000000 goto 1b;"
+ "r0 = r6;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/*
+ * for (i = 0, j = 0; i < 8; i++, j++)
+ * val[j] = j == 7;
+ * val[8] is out of bounds: j = 8 must not go through the loop again.
+ */
+SEC("socket")
+__success __retval(1)
+__naked void clamp_out_of_range_index(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u32 *)(r10 - 4) = r1;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "r1 = %[small_map] ll;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto 3f;"
+ "r8 = r0;"
+ "r6 = 0;"
+ "r7 = 0;"
+"1:"
+ "r2 = 0;"
+ "if r7 != 7 goto 2f;"
+ "r2 = 1;"
+"2:"
+ "r1 = r8;"
+ "r1 += r7;"
+ "*(u8 *)(r1 + 0) = r2;"
+ "r7 += 1;"
+ "r6 += 1;"
+ "if r6 < 8 goto 1b;"
+ "r0 = *(u8 *)(r8 + 7);"
+"3:"
+ "exit;"
+ :
+ : __imm(bpf_map_lookup_elem), __imm_addr(small_map)
+ : __clobber_all);
+}
+
+/*
+ * Every iteration takes a reference, only the last one is released. The path with
+ * r7 = 9 is looked at first. If it ended at the head of the loop the path with
+ * r7 != 9 would be pruned at 2: and nobody would see a reference come back to the head.
+ */
+SEC("socket")
+__failure __msg("Unreleased reference")
+__flag(BPF_F_TEST_STATE_FREQ)
+__naked void clamp_out_of_range_leak(void)
+{
+ asm volatile (
+ "r6 = 0;"
+ "r7 = 0;"
+"1:"
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 3f;"
+ "r8 = r0;"
+ "if r7 != 9 goto 2f;"
+ "r0 = 0;"
+"2:"
+ "r7 += 1;"
+ "r6 += 1;"
+ "if r6 < 10 goto 1b;"
+ "r1 = r8;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+"3:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_ringbuf_reserve), __imm(bpf_ringbuf_discard), __imm_addr(ringbuf)
+ : __clobber_all);
+}
+
+/*
+ * r7 = 9 releases the memory. That is the last iteration when r7 starts from 0, which
+ * is looked at first, and the first one when it starts from 9: the other nine write to
+ * memory that is gone. If the first path ended at the head of the loop, because 10 is not
+ * in 0..9, the second one would be pruned at 2: with the same r7 = 9, although 10 is in 9..18.
+ */
+SEC("socket")
+__failure __msg("R8 invalid mem access 'scalar'")
+__naked void clamp_out_of_range_two_entries(void)
+{
+ asm volatile (
+ "r1 = %[ringbuf] ll;"
+ "r2 = 8;"
+ "r3 = 0;"
+ "call %[bpf_ringbuf_reserve];"
+ "if r0 == 0 goto 3f;"
+ "r8 = r0;"
+ "call %[bpf_get_prandom_u32];"
+ "r7 = 9;"
+ "if r0 == 0 goto 4f;"
+ "r7 = 0;"
+"4:"
+ "r6 = 0;"
+"1:"
+ "r1 = 0;"
+ "*(u8 *)(r8 + 0) = r1;"
+ "if r7 != 9 goto 2f;"
+ "r1 = r8;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+"2:"
+ "r7 += 1;"
+ "r6 += 1;"
+ "if r6 < 10 goto 1b;"
+ "if r7 == 10 goto 3f;"
+ "r1 = r8;"
+ "r2 = 0;"
+ "call %[bpf_ringbuf_discard];"
+"3:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_ringbuf_reserve), __imm(bpf_ringbuf_discard), __imm(bpf_get_prandom_u32),
+ __imm_addr(ringbuf)
+ : __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 16/81] libbpf: Make arena 1 Gbyte when max_entries is not set
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (14 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 15/81] selftests/bpf: Test induction variables that come back out of range Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 16:54 ` Andrii Nakryiko
2026-10-26 2:58 ` [PATCH bpf 17/81] selftests/bpf: Test the default size of arena Alexei Starovoitov
` (66 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
max_entries of an arena is its size in pages. The kernel rejects 0. Thus
each prog must select a number, and the same number is a different size on
a kernel with a different page size.
A large arena has almost no cost. The kernel reserves 4 Gbyte of kernel
addresses for each arena, small or large. Only allocated pages use memory.
When max_entries is not set, make the arena 1 Gbyte. libbpf calculates
max_entries from the page size.
libbpf maps the full arena into the process that loads the object. Thus
VmSize of this process grows by 1 Gbyte. If the process called
mlockall(MCL_FUTURE), the kernel populates all these pages.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/lib/bpf/libbpf.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index fb8a1f07c5b6..1d4f04aa35f1 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -552,6 +552,12 @@ struct bpf_struct_ops {
#define STRUCT_OPS_SEC ".struct_ops"
#define STRUCT_OPS_LINK_SEC ".struct_ops.link"
#define ARENA_SEC ".addr_space.1"
+/*
+ * The size of the arena map when max_entries is not set. It's in bytes:
+ * max_entries of arena is in pages and the size of a page depends on
+ * the kernel. Only the pages that are allocated take memory.
+ */
+#define ARENA_DEFAULT_SZ ((size_t)1 << 30)
enum libbpf_map_type {
LIBBPF_MAP_UNSPEC,
@@ -3148,6 +3154,8 @@ static int bpf_object__init_user_btf_maps(struct bpf_object *obj, bool strict,
return -EINVAL;
}
obj->arena_map_idx = i;
+ if (!map->def.max_entries)
+ map->def.max_entries = ARENA_DEFAULT_SZ / sysconf(_SC_PAGE_SIZE);
if (obj->efile.arena_data) {
err = init_arena_map_data(obj, map, obj->efile.arena_data->d_buf,
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 16/81] libbpf: Make arena 1 Gbyte when max_entries is not set
2026-10-26 2:58 ` [PATCH bpf 16/81] libbpf: Make arena 1 Gbyte when max_entries is not set Alexei Starovoitov
@ 2026-10-26 16:54 ` Andrii Nakryiko
2026-10-26 16:59 ` Kumar Kartikeya Dwivedi
0 siblings, 1 reply; 144+ messages in thread
From: Andrii Nakryiko @ 2026-10-26 16:54 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: bpf, daniel, andrii, eddyz87, memxor
On Sun, Oct 25, 2026 at 8:01 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> From: Alexei Starovoitov <ast@kernel.org>
>
> max_entries of an arena is its size in pages. The kernel rejects 0. Thus
> each prog must select a number, and the same number is a different size on
> a kernel with a different page size.
>
> A large arena has almost no cost. The kernel reserves 4 Gbyte of kernel
> addresses for each arena, small or large. Only allocated pages use memory.
>
> When max_entries is not set, make the arena 1 Gbyte. libbpf calculates
> max_entries from the page size.
Why 1GB then, why not default to full 4GB size? And also why not teach
kernel to accept max_entries == 0 and treat it as "max allowed size"
and set it to 4GB (as of today)?
>
> libbpf maps the full arena into the process that loads the object. Thus
> VmSize of this process grows by 1 Gbyte. If the process called
> mlockall(MCL_FUTURE), the kernel populates all these pages.
>
> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
> ---
> tools/lib/bpf/libbpf.c | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index fb8a1f07c5b6..1d4f04aa35f1 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
> @@ -552,6 +552,12 @@ struct bpf_struct_ops {
> #define STRUCT_OPS_SEC ".struct_ops"
> #define STRUCT_OPS_LINK_SEC ".struct_ops.link"
> #define ARENA_SEC ".addr_space.1"
> +/*
> + * The size of the arena map when max_entries is not set. It's in bytes:
> + * max_entries of arena is in pages and the size of a page depends on
> + * the kernel. Only the pages that are allocated take memory.
> + */
> +#define ARENA_DEFAULT_SZ ((size_t)1 << 30)
>
> enum libbpf_map_type {
> LIBBPF_MAP_UNSPEC,
> @@ -3148,6 +3154,8 @@ static int bpf_object__init_user_btf_maps(struct bpf_object *obj, bool strict,
> return -EINVAL;
> }
> obj->arena_map_idx = i;
> + if (!map->def.max_entries)
> + map->def.max_entries = ARENA_DEFAULT_SZ / sysconf(_SC_PAGE_SIZE);
>
> if (obj->efile.arena_data) {
> err = init_arena_map_data(obj, map, obj->efile.arena_data->d_buf,
> --
> 2.55.0
>
^ permalink raw reply [flat|nested] 144+ messages in thread* Re: [PATCH bpf 16/81] libbpf: Make arena 1 Gbyte when max_entries is not set
2026-10-26 16:54 ` Andrii Nakryiko
@ 2026-10-26 16:59 ` Kumar Kartikeya Dwivedi
2026-10-27 0:13 ` Alexei Starovoitov
0 siblings, 1 reply; 144+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-10-26 16:59 UTC (permalink / raw)
To: Andrii Nakryiko; +Cc: Alexei Starovoitov, bpf, daniel, andrii, eddyz87
On Mon, 26 Oct 2026 at 17:54, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
>
> On Sun, Oct 25, 2026 at 8:01 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > From: Alexei Starovoitov <ast@kernel.org>
> >
> > max_entries of an arena is its size in pages. The kernel rejects 0. Thus
> > each prog must select a number, and the same number is a different size on
> > a kernel with a different page size.
> >
> > A large arena has almost no cost. The kernel reserves 4 Gbyte of kernel
> > addresses for each arena, small or large. Only allocated pages use memory.
> >
> > When max_entries is not set, make the arena 1 Gbyte. libbpf calculates
> > max_entries from the page size.
>
> Why 1GB then, why not default to full 4GB size? And also why not teach
> kernel to accept max_entries == 0 and treat it as "max allowed size"
> and set it to 4GB (as of today)?
>
I would prefer this too. Also, commit log is confusing, it's arguing
that there is not much additional cost for 4GB region, but then talks
about the 1GB limit instead...
> > [...]
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 16/81] libbpf: Make arena 1 Gbyte when max_entries is not set
2026-10-26 16:59 ` Kumar Kartikeya Dwivedi
@ 2026-10-27 0:13 ` Alexei Starovoitov
2026-10-27 21:11 ` Kumar Kartikeya Dwivedi
0 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-27 0:13 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, Andrii Nakryiko; +Cc: bpf, daniel, andrii, eddyz87
On Mon Oct 26, 2026 at 4:59 PM UTC, Kumar Kartikeya Dwivedi wrote:
> On Mon, 26 Oct 2026 at 17:54, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
>>
>> On Sun, Oct 25, 2026 at 8:01 PM Alexei Starovoitov
>> <alexei.starovoitov@gmail.com> wrote:
>> >
>> > From: Alexei Starovoitov <ast@kernel.org>
>> >
>> > max_entries of an arena is its size in pages. The kernel rejects 0. Thus
>> > each prog must select a number, and the same number is a different size on
>> > a kernel with a different page size.
>> >
>> > A large arena has almost no cost. The kernel reserves 4 Gbyte of kernel
>> > addresses for each arena, small or large. Only allocated pages use memory.
>> >
>> > When max_entries is not set, make the arena 1 Gbyte. libbpf calculates
>> > max_entries from the page size.
>>
>> Why 1GB then, why not default to full 4GB size? And also why not teach
>> kernel to accept max_entries == 0 and treat it as "max allowed size"
>> and set it to 4GB (as of today)?
>>
>
> I would prefer this too. Also, commit log is confusing, it's arguing
> that there is not much additional cost for 4GB region, but then talks
> about the 1GB limit instead...
Yeah. commit log could have been more clear.
The reason I didn't go with 4G since 4k pages on x86 need 8Mbyte
of kernel memory for empty arena while 1G needs 2Mbyte, which felt ok-ish.
I didn't go lower than 1G, because poor man per-cpu in arena
implementation might use default 4K cpus, so just 256k bytes of
per-cpu space would need 1G of arena.
4k is a dumb approach, of course, bur CONFIG_NR_CPUS is what we have.
btw we also have:
static __always_inline int arena_spin_lock(arena_spinlock_t __arena *lock)
{
int val = 0;
if (CONFIG_NR_CPUS > 1024)
return -EOPNOTSUPP;
For slightlt different reasons.
I'm fine going with 4G if folks feel that 8Mbyte per arena (that could be many)
is an ok default.
^ permalink raw reply [flat|nested] 144+ messages in thread* Re: [PATCH bpf 16/81] libbpf: Make arena 1 Gbyte when max_entries is not set
2026-10-27 0:13 ` Alexei Starovoitov
@ 2026-10-27 21:11 ` Kumar Kartikeya Dwivedi
2026-10-27 21:22 ` Alexei Starovoitov
0 siblings, 1 reply; 144+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-10-27 21:11 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: Andrii Nakryiko, bpf, daniel, andrii, eddyz87
On Tue, 27 Oct 2026 at 01:13, Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Mon Oct 26, 2026 at 4:59 PM UTC, Kumar Kartikeya Dwivedi wrote:
> > On Mon, 26 Oct 2026 at 17:54, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> >>
> >> On Sun, Oct 25, 2026 at 8:01 PM Alexei Starovoitov
> >> <alexei.starovoitov@gmail.com> wrote:
> >> >
> >> > From: Alexei Starovoitov <ast@kernel.org>
> >> >
> >> > max_entries of an arena is its size in pages. The kernel rejects 0. Thus
> >> > each prog must select a number, and the same number is a different size on
> >> > a kernel with a different page size.
> >> >
> >> > A large arena has almost no cost. The kernel reserves 4 Gbyte of kernel
> >> > addresses for each arena, small or large. Only allocated pages use memory.
> >> >
> >> > When max_entries is not set, make the arena 1 Gbyte. libbpf calculates
> >> > max_entries from the page size.
> >>
> >> Why 1GB then, why not default to full 4GB size? And also why not teach
> >> kernel to accept max_entries == 0 and treat it as "max allowed size"
> >> and set it to 4GB (as of today)?
> >>
> >
> > I would prefer this too. Also, commit log is confusing, it's arguing
> > that there is not much additional cost for 4GB region, but then talks
> > about the 1GB limit instead...
>
> Yeah. commit log could have been more clear.
>
> The reason I didn't go with 4G since 4k pages on x86 need 8Mbyte
> of kernel memory for empty arena while 1G needs 2Mbyte, which felt ok-ish.
I re-read the current code (even with your patches in this set), my
understanding is that regardless of max_entries, we always commit 8MB
of page tables, no? I don't see anything that uses max_entries to
decide that bit. So I think this patch is noop regardless.
> [...]
>
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 16/81] libbpf: Make arena 1 Gbyte when max_entries is not set
2026-10-27 21:11 ` Kumar Kartikeya Dwivedi
@ 2026-10-27 21:22 ` Alexei Starovoitov
2026-10-27 21:28 ` Andrii Nakryiko
0 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-27 21:22 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi; +Cc: Andrii Nakryiko, bpf, daniel, andrii, eddyz87
On Tue, Oct 27, 2026 at 02:11 PM Kumar Kartikeya Dwivedi <memxor@gmail.com> wrote:
>> The reason I didn't go with 4G since 4k pages on x86 need 8Mbyte
>> of kernel memory for empty arena while 1G needs 2Mbyte, which felt ok-ish.
>
> I re-read the current code (even with your patches in this set), my
> understanding is that regardless of max_entries, we always commit 8MB
> of page tables, no? I don't see anything that uses max_entries to
> decide that bit. So I think this patch is noop regardless.
ohh. populate_pgtable_except_pte() doesn't look at max_entries :(
sigh. ok let's do 4G then.
The only remaining question is whether to do it libbpf or kernel.
I lean toward libbpf, since it's easier to revisit
if/when we fix populate_pgtable_except_pte() not to waste memory.
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 16/81] libbpf: Make arena 1 Gbyte when max_entries is not set
2026-10-27 21:22 ` Alexei Starovoitov
@ 2026-10-27 21:28 ` Andrii Nakryiko
2026-10-27 21:29 ` Kumar Kartikeya Dwivedi
0 siblings, 1 reply; 144+ messages in thread
From: Andrii Nakryiko @ 2026-10-27 21:28 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: Kumar Kartikeya Dwivedi, bpf, daniel, andrii, eddyz87
On Tue, Oct 27, 2026 at 2:22 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Tue, Oct 27, 2026 at 02:11 PM Kumar Kartikeya Dwivedi <memxor@gmail.com> wrote:
> >> The reason I didn't go with 4G since 4k pages on x86 need 8Mbyte
> >> of kernel memory for empty arena while 1G needs 2Mbyte, which felt ok-ish.
> >
> > I re-read the current code (even with your patches in this set), my
> > understanding is that regardless of max_entries, we always commit 8MB
> > of page tables, no? I don't see anything that uses max_entries to
> > decide that bit. So I think this patch is noop regardless.
>
> ohh. populate_pgtable_except_pte() doesn't look at max_entries :(
> sigh. ok let's do 4G then.
> The only remaining question is whether to do it libbpf or kernel.
> I lean toward libbpf, since it's easier to revisit
> if/when we fix populate_pgtable_except_pte() not to waste memory.
I'd do both. libbpf side for backwards compat and to not care about
kernel version, but kernel should just assume that zero means "no
extra limitations"
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 16/81] libbpf: Make arena 1 Gbyte when max_entries is not set
2026-10-27 21:28 ` Andrii Nakryiko
@ 2026-10-27 21:29 ` Kumar Kartikeya Dwivedi
0 siblings, 0 replies; 144+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-10-27 21:29 UTC (permalink / raw)
To: Andrii Nakryiko; +Cc: Alexei Starovoitov, bpf, daniel, andrii, eddyz87
On Tue, 27 Oct 2026 at 22:28, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
>
> On Tue, Oct 27, 2026 at 2:22 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Tue, Oct 27, 2026 at 02:11 PM Kumar Kartikeya Dwivedi <memxor@gmail.com> wrote:
> > >> The reason I didn't go with 4G since 4k pages on x86 need 8Mbyte
> > >> of kernel memory for empty arena while 1G needs 2Mbyte, which felt ok-ish.
> > >
> > > I re-read the current code (even with your patches in this set), my
> > > understanding is that regardless of max_entries, we always commit 8MB
> > > of page tables, no? I don't see anything that uses max_entries to
> > > decide that bit. So I think this patch is noop regardless.
> >
> > ohh. populate_pgtable_except_pte() doesn't look at max_entries :(
> > sigh. ok let's do 4G then.
> > The only remaining question is whether to do it libbpf or kernel.
> > I lean toward libbpf, since it's easier to revisit
> > if/when we fix populate_pgtable_except_pte() not to waste memory.
>
I can follow up on this (not sure which approach you'll prefer, but we
can iterate later), since something similar is necessary for typed
arenas which may be numerous per program. The other thing I noticed is
that we don't remove populated page tables when arena is freed, so we
also "leak" 8MB, until something reuses that VA region.
> I'd do both. libbpf side for backwards compat and to not care about
> kernel version, but kernel should just assume that zero means "no
> extra limitations"
+1.
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 17/81] selftests/bpf: Test the default size of arena
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (15 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 16/81] libbpf: Make arena 1 Gbyte when max_entries is not set Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 18/81] bpf: Treat addr_space_cast of a pointer that is not in arena as a move Alexei Starovoitov
` (65 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The test declares an arena without max_entries. The kernel rejects such a
map with -EINVAL, thus libbpf must set the size. See
"libbpf: Make arena 1 Gbyte when max_entries is not set".
The test makes sure that:
- After open, max_entries is 1 Gbyte divided by the page size.
- The object loads.
- The prog can allocate pages in the arena.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../bpf/prog_tests/arena_default_size.c | 22 +++++++++++++++++++
.../selftests/bpf/progs/arena_default_size.c | 21 ++++++++++++++++++
2 files changed, 43 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_default_size.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_default_size.c
diff --git a/tools/testing/selftests/bpf/prog_tests/arena_default_size.c b/tools/testing/selftests/bpf/prog_tests/arena_default_size.c
new file mode 100644
index 000000000000..0c7efb5c9e5a
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/arena_default_size.c
@@ -0,0 +1,22 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+#include <linux/sizes.h>
+#include "arena_default_size.skel.h"
+
+void test_arena_default_size(void)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, opts);
+ struct arena_default_size *skel;
+
+ skel = arena_default_size__open();
+ if (!ASSERT_OK_PTR(skel, "open"))
+ return;
+ ASSERT_EQ(bpf_map__max_entries(skel->maps.arena), SZ_1G / getpagesize(), "max_entries");
+ if (!ASSERT_OK(arena_default_size__load(skel), "load"))
+ goto out;
+ ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(skel->progs.alloc), &opts), "run");
+ ASSERT_OK(opts.retval, "retval");
+out:
+ arena_default_size__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/arena_default_size.c b/tools/testing/selftests/bpf/progs/arena_default_size.c
new file mode 100644
index 000000000000..1fae38126094
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_default_size.c
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define BPF_NO_KFUNC_PROTOTYPES
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+#include <bpf_arena_common.h>
+
+/* max_entries is not set */
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+} arena SEC(".maps");
+
+SEC("syscall")
+int alloc(void *ctx)
+{
+ return bpf_arena_alloc_pages(&arena, NULL, 2, NUMA_NO_NODE, 0) ? 0 : 1;
+}
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 18/81] bpf: Treat addr_space_cast of a pointer that is not in arena as a move
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (16 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 17/81] selftests/bpf: Test the default size of arena Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 19/81] selftests/bpf: Test addr_space_cast of pointers that are not in arena Alexei Starovoitov
` (64 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
'rX = addr_space_cast(rY, 0, 1)' makes a pointer to arena from an address
in arena. The verifier gives rX the type PTR_TO_ARENA and replaces the insn
with 'wX = wY', which keeps the low 32 bits. The JIT adds the base of the
arena in each access. Thus an access through PTR_TO_ARENA cannot leave the
arena, and the verifier does not check it.
In C the __arena attribute marks the pointers to arena, and clang casts
only these pointers. Rust has no address spaces. The compiler follows each
pointer back to its origin, and it does not cast a pointer that comes from
the kernel. But it cannot always find the origin. For example, two progs of
an object call the same function. One prog passes a ring buffer record, and
the other prog passes an object in arena:
fn fill(e: &mut Event)
r6 = addr_space_cast(r1, 0, 1) ; a ring buffer record, or a number
*(u64 *)(r6 + 0) = r2
When the compiler does not know the origin, it casts the pointer. The
verifier does not examine the type of r1. r6 becomes PTR_TO_ARENA in the
two progs. The first prog loads, and it writes to arena instead of the
record.
The verifier knows the type of rY. Let the type select what the cast does:
rY the cast becomes rX
----------------------------- -------------------- ------------
number (scalar), PTR_TO_ARENA wX = wY, as before PTR_TO_ARENA
all other pointers rX = rY, or nothing a copy of rY
when X == Y
All other pointers are kernel pointers, and they are in address space 0
already. Thus a move is the correct cast. rX is a copy of rY. The verifier
applies the same checks to the two registers: bounds, NULL, references.
The verifier replaces the cast one time, for all paths. Thus it rejects a
prog in which the same cast gets a number on one path and a kernel pointer
on a different path:
same addr_space_cast insn cannot be used with a number and with a pointer
The analysis of live stack slots follows a pointer to the stack through a
move. Make arg_track_xfer() follow it through the cast too. Otherwise the
analysis does not see a read of the stack through rX, and the verifier
forgets a slot that the prog reads.
The cast continues to be one insn at most. Progs in C get the same code.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 3 ++
kernel/bpf/fixups.c | 18 ++++++++++++
kernel/bpf/liveness.c | 6 ++--
kernel/bpf/verifier.c | 54 ++++++++++++++++++++++++++++++++++--
4 files changed, 76 insertions(+), 5 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 7030b997ce2c..ce3f0e275d9c 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -782,6 +782,8 @@ struct bpf_insn_aux_data {
u64 non_sleepable:1; /* helper/kfunc may be called from non-sleepable context */
u64 is_iter_next:1; /* bpf_iter_<type>_next() kfunc call */
u64 call_with_percpu_alloc_ptr:1; /* {this,per}_cpu_ptr() with prog percpu alloc */
+ u64 cast_arena:1; /* addr_space_cast of an address in arena */
+ u64 cast_mov:1; /* addr_space_cast of a pointer that is not in arena, it's a move */
u64 bb_end:1; /* last instruction of a basic block */
u64 need_scev:1;
u64 alu_state:5; /* used in combination with alu_limit */
@@ -2007,6 +2009,7 @@ int bpf_optimize_bpf_loop(struct bpf_verifier_env *env);
void bpf_opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env);
int bpf_opt_remove_dead_code(struct bpf_verifier_env *env);
int bpf_opt_remove_nops(struct bpf_verifier_env *env);
+void bpf_lower_cast_mov(struct bpf_verifier_env *env);
int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, const union bpf_attr *attr);
int bpf_convert_ctx_accesses(struct bpf_verifier_env *env);
int bpf_jit_subprogs(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index fe8d51f8c855..ae3347e724ca 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -723,6 +723,24 @@ int bpf_opt_remove_dead_code(struct bpf_verifier_env *env)
return 0;
}
+/* addr_space_cast of a pointer that is not in arena is a move, see is_cast_mov() */
+void bpf_lower_cast_mov(struct bpf_verifier_env *env)
+{
+ struct bpf_insn_aux_data *aux = env->insn_aux_data;
+ struct bpf_insn *insn = env->prog->insnsi;
+ int i;
+
+ for (i = 0; i < env->prog->len; i++, insn++) {
+ if (!aux[i].cast_mov)
+ continue;
+ if (insn->dst_reg == insn->src_reg)
+ /* bpf_opt_remove_nops() removes it */
+ *insn = NOP;
+ else
+ *insn = BPF_MOV64_REG(insn->dst_reg, insn->src_reg);
+ }
+}
+
int bpf_opt_remove_nops(struct bpf_verifier_env *env)
{
struct bpf_insn_aux_data *aux = env->insn_aux_data;
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index e657e6d95408..f2989a4fd9a7 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1426,10 +1426,12 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
} /* else if dst->frame is imprecise it stays so */
} else if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_X) {
if (code == BPF_MOV) {
- if (insn->off == 0) {
+ if (insn->off == 0 ||
+ (insn->off == BPF_ADDR_SPACE_CAST && insn->imm == 1)) {
+ /* the cast of a pointer to stack to as(0) is a move */
*dst = *src;
} else {
- /* addr_space_cast destroys a pointer */
+ /* the cast to as(1) and sign extension destroy a pointer */
*dst = none;
}
} else {
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 84f722bac539..203edbe41397 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -17803,6 +17803,51 @@ int bpf_reg_union(struct bpf_verifier_env *env, struct bpf_reg_state *acc,
return reg_bounds_sanity_check(env, acc, "bpf_reg_union");
}
+/*
+ * There are no address spaces in Rust. The compiler doesn't cast a pointer that
+ * comes from the kernel. But it can't always tell, and then it casts the pointer
+ * to as(0) before the access, like an address in arena.
+ * A pointer that the verifier knows is in as(0) already, so the cast is a move.
+ * arg_track_xfer() does the same for pointers to stack.
+ */
+static bool is_cast_mov(const struct bpf_insn *insn, const struct bpf_reg_state *src_reg)
+{
+ /* cast from as(1) to as(0) */
+ if (insn->imm != 1)
+ return false;
+
+ switch (base_type(src_reg->type)) {
+ case SCALAR_VALUE:
+ case PTR_TO_ARENA:
+ return false;
+ default:
+ return true;
+ }
+}
+
+static int check_addr_space_cast(struct bpf_verifier_env *env, struct bpf_insn *insn,
+ struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg)
+{
+ struct bpf_insn_aux_data *aux = cur_aux(env);
+
+ if (is_cast_mov(insn, src_reg)) {
+ aux->cast_mov = true;
+ *dst_reg = *src_reg;
+ } else {
+ aux->cast_arena = true;
+ mark_reg_unknown(env, cur_regs(env), insn->dst_reg);
+ if (insn->imm == 1) /* cast from as(1) to as(0) */
+ dst_reg->type = PTR_TO_ARENA;
+ }
+
+ /* it's either wX = wY or rX = rY, see bpf_lower_cast_mov() */
+ if (aux->cast_mov && aux->cast_arena) {
+ verbose(env, "same addr_space_cast insn cannot be used with a number and with a pointer\n");
+ return -EINVAL;
+ }
+ return 0;
+}
+
/* check validity of 32-bit and 64-bit arithmetic operations */
static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
{
@@ -17864,9 +17909,9 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)
if (BPF_CLASS(insn->code) == BPF_ALU64) {
if (insn->imm) {
/* off == BPF_ADDR_SPACE_CAST */
- mark_reg_unknown(env, regs, insn->dst_reg);
- if (insn->imm == 1) /* cast from as(1) to as(0) */
- dst_reg->type = PTR_TO_ARENA;
+ err = check_addr_space_cast(env, insn, dst_reg, src_reg);
+ if (err)
+ return err;
} else if (insn->off == 0) {
/* case: R1 = R2
* copy register state to dest reg
@@ -24069,6 +24114,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret == 0)
ret = bpf_optimize_bpf_loop(env);
+ if (ret == 0)
+ bpf_lower_cast_mov(env);
+
if (is_priv) {
if (ret == 0)
bpf_opt_hard_wire_dead_code_branches(env);
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 19/81] selftests/bpf: Test addr_space_cast of pointers that are not in arena
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (17 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 18/81] bpf: Treat addr_space_cast of a pointer that is not in arena as a move Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 20/81] libbpf: Support .bss.addr_space.1 section Alexei Starovoitov
` (63 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier does not check an access through a pointer to arena. Thus a
kernel pointer must keep its type after the cast to address space 0. See
"bpf: Treat addr_space_cast of a pointer that is not in arena as a move".
test what the prog does or shows result
--------------------------- ------------------------------------ ------
map_value the cast is 'r6 = r0' loads
map_value_same_reg the cast of r0 to r0 is removed loads
map_value_bounds access beyond the map value reject
map_value_or_null access without a NULL check reject
map_value_null_check_of_src NULL check of r0 applies to r6 loads
ctx r6 is ctx loads
btf_id r6 is trusted_ptr_task_struct loads
ref_released releases a reference through r1 loads
ref_leaked does not release a reference reject
stack r6 is fp-8 loads
spill_through_cast spills ctx through r6, fills it loads
callee_writes callee writes the caller's stack loads
callee_reads callee reads the caller's stack loads
number r6 is arena, the cast is 'w6 = w6' loads
number_and_pointer one cast gets a number and ctx reject
pointer_and_number the same, the paths are exchanged reject
cast_user cast of ctx to as(1) gives a scalar loads
The progs that return the content of a map value or of a stack slot also
run. The result shows that they read the object and not arena. callee_reads
needs the change in arg_track_xfer().
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/verifier_arena_cast.c | 456 ++++++++++++++++++
2 files changed, 458 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_cast.c
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 6a494c381266..643d67782619 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -12,6 +12,7 @@
#include "verifier_align.skel.h"
#include "verifier_and.skel.h"
#include "verifier_arena.skel.h"
+#include "verifier_arena_cast.skel.h"
#include "verifier_arena_large.skel.h"
#include "verifier_arena_globals1.skel.h"
#include "verifier_arena_globals2.skel.h"
@@ -205,6 +206,7 @@ void test_verifier_aggregate_ret(void) { RUN_TESTS(verifier_aggregate_ret
void test_verifier_align(void) { RUN(verifier_align); }
void test_verifier_and(void) { RUN(verifier_and); }
void test_verifier_arena(void) { RUN(verifier_arena); }
+void test_verifier_arena_cast(void) { RUN(verifier_arena_cast); }
void test_verifier_arena_large(void) { RUN(verifier_arena_large); }
void test_verifier_arena_globals1(void) { RUN(verifier_arena_globals1); }
void test_verifier_arena_globals2(void) { RUN(verifier_arena_globals2); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_cast.c b/tools/testing/selftests/bpf/progs/verifier_arena_cast.c
new file mode 100644
index 000000000000..3208f1009534
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_arena_cast.c
@@ -0,0 +1,456 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+struct task_struct {};
+struct task_struct *bpf_task_acquire(struct task_struct *p) __ksym;
+void bpf_task_release(struct task_struct *p) __ksym;
+
+#ifdef __TARGET_ARCH_arm64
+#define ARENA_VM_START (1ull << 32)
+#else
+#define ARENA_VM_START (1ull << 44)
+#endif
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 4);
+ __ulong(map_extra, ARENA_VM_START);
+} arena SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, long long);
+} hash SEC(".maps");
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+ bpf_task_release(bpf_task_acquire(0));
+}
+
+/* dst = addr_space_cast(src, 0, 1) */
+#define CAST_KERN(dst, src) \
+ BPF_RAW_INSN(BPF_ALU64 | BPF_MOV | BPF_X, dst, src, BPF_ADDR_SPACE_CAST, 1)
+/* dst = addr_space_cast(src, 1, 0) */
+#define CAST_USER(dst, src) \
+ BPF_RAW_INSN(BPF_ALU64 | BPF_MOV | BPF_X, dst, src, BPF_ADDR_SPACE_CAST, 1 << 16)
+
+/* r0 = the element of the hash map or NULL. There is one after the update. */
+#define UPDATE_AND_LOOKUP \
+ "r1 = 0;" \
+ "*(u32 *)(r10 - 4) = r1;" \
+ "r1 = 7;" \
+ "*(u64 *)(r10 - 16) = r1;" \
+ "r1 = %[hash] ll;" \
+ "r2 = r10;" \
+ "r2 += -4;" \
+ "r3 = r10;" \
+ "r3 += -16;" \
+ "r4 = 0;" \
+ "call %[bpf_map_update_elem];" \
+ "r1 = %[hash] ll;" \
+ "r2 = r10;" \
+ "r2 += -4;" \
+ "call %[bpf_map_lookup_elem];"
+
+SEC("syscall")
+__description("arena cast: a pointer to map value, it's a move")
+__success __retval(7)
+__xlated("r6 = r0")
+__xlated("r0 = *(u64 *)(r6 +0)")
+__naked void map_value(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 1f;"
+ ".8byte %[cast_r6_r0];"
+ "r0 = *(u64 *)(r6 + 0);"
+"1:"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm_insn(cast_r6_r0, CAST_KERN(BPF_REG_6, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a pointer to map value into the same register, it's removed")
+__success __retval(7)
+__xlated("r7 = r0")
+__xlated("r0 = *(u64 *)(r0 +0)")
+__xlated("exit")
+__naked void map_value_same_reg(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 1f;"
+ "r7 = r0;"
+ ".8byte %[cast_r0_r0];"
+ "r0 = *(u64 *)(r0 + 0);"
+"1:"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm_insn(cast_r0_r0, CAST_KERN(BPF_REG_0, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: bounds of map value are checked after the cast")
+__failure __msg("invalid access to map value, value_size=8 off=8 size=8")
+__naked void map_value_bounds(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ "if r0 == 0 goto 1f;"
+ ".8byte %[cast_r6_r0];"
+ "r0 = *(u64 *)(r6 + 8);"
+"1:"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm_insn(cast_r6_r0, CAST_KERN(BPF_REG_6, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a pointer that may be NULL has to be checked after the cast")
+__failure __msg("R6 invalid mem access 'map_value_or_null'")
+__naked void map_value_or_null(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ ".8byte %[cast_r6_r0];"
+ "r0 = *(u64 *)(r6 + 0);"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm_insn(cast_r6_r0, CAST_KERN(BPF_REG_6, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: NULL check of the pointer is for its copy too")
+__success __retval(7)
+__naked void map_value_null_check_of_src(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ UPDATE_AND_LOOKUP
+ ".8byte %[cast_r6_r0];"
+ "if r0 == 0 goto 1f;"
+ "r0 = *(u64 *)(r6 + 0);"
+"1:"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_addr(hash),
+ __imm(bpf_map_update_elem),
+ __imm(bpf_map_lookup_elem),
+ __imm_insn(cast_r6_r0, CAST_KERN(BPF_REG_6, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena cast: a pointer to ctx")
+__success __log_level(2)
+__msg("R6=ctx()")
+__naked void ctx(void)
+{
+ asm volatile (
+ "r2 = %[arena] ll;"
+ ".8byte %[cast_r6_r1];"
+ "r0 = *(u32 *)(r6 + %[len]);"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_const(len, offsetof(struct __sk_buff, len)),
+ __imm_insn(cast_r6_r1, CAST_KERN(BPF_REG_6, BPF_REG_1))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a pointer to task_struct")
+__success __log_level(2)
+__msg("R6=trusted_ptr_task_struct()")
+__naked void btf_id(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "call %[bpf_get_current_task_btf];"
+ ".8byte %[cast_r6_r0];"
+ "r0 = *(u32 *)(r6 + 0);"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm(bpf_get_current_task_btf),
+ __imm_insn(cast_r6_r0, CAST_KERN(BPF_REG_6, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a reference is released through the copy")
+__success __retval(0)
+__naked void ref_released(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "call %[bpf_get_current_task_btf];"
+ "r1 = r0;"
+ "call %[bpf_task_acquire];"
+ "if r0 == 0 goto 1f;"
+ ".8byte %[cast_r1_r0];"
+ "call %[bpf_task_release];"
+"1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm(bpf_get_current_task_btf),
+ __imm(bpf_task_acquire),
+ __imm(bpf_task_release),
+ __imm_insn(cast_r1_r0, CAST_KERN(BPF_REG_1, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a reference doesn't go away")
+__failure __msg("Unreleased reference")
+__naked void ref_leaked(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "call %[bpf_get_current_task_btf];"
+ "r1 = r0;"
+ "call %[bpf_task_acquire];"
+ "if r0 == 0 goto 1f;"
+ ".8byte %[cast_r0_r0];"
+"1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm(bpf_get_current_task_btf),
+ __imm(bpf_task_acquire),
+ __imm_insn(cast_r0_r0, CAST_KERN(BPF_REG_0, BPF_REG_0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a number")
+__success __log_level(2)
+__msg("R6=arena")
+__xlated("w6 = w6")
+__naked void number(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "r6 = %[addr] ll;"
+ ".8byte %[cast_r6_r6];"
+ "r0 = *(u64 *)(r6 + 0);"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_const(addr, ARENA_VM_START),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+/* static analysis of the stack doesn't follow the pointer through the cast */
+SEC("syscall")
+__description("arena cast: a pointer to stack")
+__success __retval(7) __log_level(2)
+__msg("R6=fp-8")
+__naked void stack(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "r6 = r10;"
+ "r6 += -8;"
+ ".8byte %[cast_r6_r6];"
+ "r0 = 7;"
+ "*(u64 *)(r6 + 0) = r0;"
+ "r0 = *(u64 *)(r10 - 8);"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: a pointer is spilled through the cast of a pointer to stack")
+__success
+__naked void spill_through_cast(void)
+{
+ asm volatile (
+ "r2 = %[arena] ll;"
+ "r6 = r10;"
+ "r6 += -8;"
+ ".8byte %[cast_r6_r6];"
+ "*(u64 *)(r6 + 0) = r1;"
+ "r2 = *(u64 *)(r10 - 8);"
+ /* it's ctx */
+ "r0 = *(u32 *)(r2 + 0);"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long writes_through_cast(long p)
+{
+ asm volatile (
+ ".8byte %[cast_r1_r1];"
+ "r0 = 7;"
+ "*(u64 *)(r1 + 0) = r0;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(cast_r1_r1, CAST_KERN(BPF_REG_1, BPF_REG_1))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: the callee writes to the stack of the caller")
+__success __retval(7)
+__naked void callee_writes(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "r0 = 0;"
+ "*(u64 *)(r10 - 8) = r0;"
+ "r1 = r10;"
+ "r1 += -8;"
+ "call writes_through_cast;"
+ "r0 = *(u64 *)(r10 - 8);"
+ "exit;"
+ :
+ : __imm_addr(arena)
+ : __clobber_all);
+}
+
+static __naked __noinline __used long reads_through_cast(long p)
+{
+ asm volatile (
+ "r6 = r1;"
+ /* a place where what is not live is forgotten */
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 0 goto +0;"
+ ".8byte %[cast_r6_r6];"
+ "r0 = *(u64 *)(r6 + 0);"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena cast: the callee reads the stack of the caller")
+__success __retval(5)
+__naked void callee_reads(void)
+{
+ asm volatile (
+ "r1 = %[arena] ll;"
+ "r0 = 5;"
+ "*(u64 *)(r10 - 8) = r0;"
+ "r1 = r10;"
+ "r1 += -8;"
+ "call reads_through_cast;"
+ "exit;"
+ :
+ : __imm_addr(arena)
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena cast: a number and a pointer at the same insn")
+__failure __msg("same addr_space_cast insn cannot be used with a number and with a pointer")
+__naked void number_and_pointer(void)
+{
+ asm volatile (
+ "r2 = %[arena] ll;"
+ "r6 = r1;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 0 goto 1f;"
+ "r6 = 4096;"
+"1:"
+ ".8byte %[cast_r6_r6];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm(bpf_get_prandom_u32),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena cast: a pointer and a number at the same insn")
+__failure __msg("same addr_space_cast insn cannot be used with a number and with a pointer")
+__naked void pointer_and_number(void)
+{
+ asm volatile (
+ "r2 = %[arena] ll;"
+ "r6 = 4096;"
+ "r7 = r1;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 0 goto 1f;"
+ "r6 = r7;"
+"1:"
+ ".8byte %[cast_r6_r6];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm(bpf_get_prandom_u32),
+ __imm_insn(cast_r6_r6, CAST_KERN(BPF_REG_6, BPF_REG_6))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena cast: the cast to as(1) of a pointer is a number")
+__success __log_level(2)
+__msg("R6=scalar()")
+__naked void cast_user(void)
+{
+ asm volatile (
+ "r2 = %[arena] ll;"
+ ".8byte %[cast_r6_r1];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(arena),
+ __imm_insn(cast_r6_r1, CAST_USER(BPF_REG_6, BPF_REG_1))
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 20/81] libbpf: Support .bss.addr_space.1 section
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (18 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 19/81] selftests/bpf: Test addr_space_cast of pointers that are not in arena Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 21/81] libbpf: Relocate pointers that are stored in arena Alexei Starovoitov
` (62 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Global variables in arena are in the section .addr_space.1. Its type is
SHT_PROGBITS. Thus the variables that are zero use space in the object file
too.
Support .bss.addr_space.1 as the .bss of arena. The name starts with .bss.
Thus compilers make the section SHT_NOBITS, and it has no bytes in the
file:
long __arena big[8192] SEC(".bss.addr_space.1");
An object with 8 Mbyte of zeros shrinks from 8410296 to 21696 bytes.
libbpf puts the new section after .addr_space.1 in arena and fills it with
zeros. It adds the offset of the section to the relocations that refer to
it.
Only the file becomes smaller. At load the new variables get pages in
arena, as the variables of .addr_space.1 do. The layout of skel->arena does
not change, and the new variables are not in it.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/lib/bpf/libbpf.c | 41 ++++++++++++++++++++++++++++++++---------
1 file changed, 32 insertions(+), 9 deletions(-)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 1d4f04aa35f1..78ff9d653191 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -552,6 +552,11 @@ struct bpf_struct_ops {
#define STRUCT_OPS_SEC ".struct_ops"
#define STRUCT_OPS_LINK_SEC ".struct_ops.link"
#define ARENA_SEC ".addr_space.1"
+/*
+ * Variables in arena that are zero. The name starts with .bss, so compilers make
+ * it SHT_NOBITS and it takes no space in the file. It's placed after ARENA_SEC.
+ */
+#define ARENA_BSS_SEC ".bss.addr_space.1"
/*
* The size of the arena map when max_entries is not set. It's in bytes:
* max_entries of arena is in pages and the size of a page depends on
@@ -697,6 +702,8 @@ struct elf_state {
Elf64_Ehdr *ehdr;
Elf_Data *symbols;
Elf_Data *arena_data;
+ Elf_Data *arena_bss;
+ size_t arena_bss_align;
size_t shstrndx; /* section index for section name strings */
size_t strtabidx;
struct elf_sec_desc *secs;
@@ -707,6 +714,7 @@ struct elf_state {
int symbols_shndx;
bool has_st_ops;
int arena_data_shndx;
+ int arena_bss_shndx;
int jumptables_data_shndx;
};
@@ -787,6 +795,7 @@ struct bpf_object {
void *arena_data;
size_t arena_data_sz;
size_t arena_data_off;
+ size_t arena_bss_off; /* from the start of arena_data */
void *jumptables_data;
size_t jumptables_data_sz;
@@ -1612,6 +1621,7 @@ static struct bpf_object *bpf_object__new(const char *path,
obj->efile.obj_buf = obj_buf;
obj->efile.obj_buf_sz = obj_buf_sz;
obj->efile.btf_maps_shndx = -1;
+ obj->efile.arena_bss_shndx = -1;
obj->kconfig_map_idx = -1;
obj->arena_map_idx = -1;
@@ -1631,6 +1641,7 @@ static void bpf_object__elf_finish(struct bpf_object *obj)
obj->efile.ehdr = NULL;
obj->efile.symbols = NULL;
obj->efile.arena_data = NULL;
+ obj->efile.arena_bss = NULL;
zfree(&obj->efile.secs);
obj->efile.sec_cnt = 0;
@@ -3078,13 +3089,20 @@ static int bpf_object__init_user_btf_map(struct bpf_object *obj,
return 0;
}
-static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map,
- void *data, size_t data_sz)
+static int init_arena_map_data(struct bpf_object *obj, struct bpf_map *map)
{
- obj->arena_data = malloc(data_sz);
+ Elf_Data *data = obj->efile.arena_data, *bss = obj->efile.arena_bss;
+ size_t data_sz = data ? data->d_size : 0;
+
+ if (bss) {
+ obj->arena_bss_off = roundup(data_sz, obj->efile.arena_bss_align);
+ data_sz = obj->arena_bss_off + bss->d_size;
+ }
+ obj->arena_data = calloc(1, data_sz);
if (!obj->arena_data)
return -ENOMEM;
- memcpy(obj->arena_data, data, data_sz);
+ if (data)
+ memcpy(obj->arena_data, data->d_buf, data->d_size);
obj->arena_data_sz = data_sz;
/* make bpf_map__init_value() work for ARENA maps */
@@ -3157,14 +3175,13 @@ static int bpf_object__init_user_btf_maps(struct bpf_object *obj, bool strict,
if (!map->def.max_entries)
map->def.max_entries = ARENA_DEFAULT_SZ / sysconf(_SC_PAGE_SIZE);
- if (obj->efile.arena_data) {
- err = init_arena_map_data(obj, map, obj->efile.arena_data->d_buf,
- obj->efile.arena_data->d_size);
+ if (obj->efile.arena_data || obj->efile.arena_bss) {
+ err = init_arena_map_data(obj, map);
if (err)
return err;
}
}
- if (obj->efile.arena_data && obj->arena_map_idx < 0) {
+ if ((obj->efile.arena_data || obj->efile.arena_bss) && obj->arena_map_idx < 0) {
pr_warn("elf: sec '%s': to use global __arena variables the ARENA map should be explicitly declared in SEC(\".maps\")\n",
ARENA_SEC);
return -ENOENT;
@@ -4159,6 +4176,10 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
sec_desc->sec_type = SEC_RELO;
sec_desc->shdr = sh;
sec_desc->data = data;
+ } else if (sh->sh_type == SHT_NOBITS && strcmp(name, ARENA_BSS_SEC) == 0) {
+ obj->efile.arena_bss = data;
+ obj->efile.arena_bss_align = sh->sh_addralign ?: 1;
+ obj->efile.arena_bss_shndx = idx;
} else if (sh->sh_type == SHT_NOBITS && (strcmp(name, BSS_SEC) == 0 ||
str_has_pfx(name, BSS_SEC "."))) {
sec_desc->sec_type = SEC_BSS;
@@ -4809,7 +4830,7 @@ static int bpf_program__record_reloc(struct bpf_program *prog,
sym_sec_name = elf_sec_name(obj, elf_sec_by_idx(obj, shdr_idx));
/* arena data relocation */
- if (shdr_idx == obj->efile.arena_data_shndx) {
+ if (shdr_idx == obj->efile.arena_data_shndx || shdr_idx == obj->efile.arena_bss_shndx) {
if (obj->arena_map_idx < 0) {
pr_warn("prog '%s': bad arena data relocation at insn %u, no arena maps defined\n",
prog->name, insn_idx);
@@ -4819,6 +4840,8 @@ static int bpf_program__record_reloc(struct bpf_program *prog,
reloc_desc->insn_idx = insn_idx;
reloc_desc->map_idx = obj->arena_map_idx;
reloc_desc->sym_off = sym->st_value;
+ if (shdr_idx == obj->efile.arena_bss_shndx)
+ reloc_desc->sym_off += obj->arena_bss_off;
map = &obj->maps[obj->arena_map_idx];
pr_debug("prog '%s': found arena map %d (%s, sec %d, off %zu) for insn %u\n",
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 21/81] libbpf: Relocate pointers that are stored in arena
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (19 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 20/81] libbpf: Support .bss.addr_space.1 section Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 22/81] selftests/bpf: Test pointers in arena and .bss.addr_space.1 Alexei Starovoitov
` (61 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A variable in arena that points to a different variable in arena gets an
incorrect value:
int __arena x = 42;
int __arena * __arena px = &x;
The compiler emits an R_BPF_64_ABS64 relocation for px. libbpf ignores the
relocations of .addr_space.1. The object loads without an error, and px is
0.
The address of x is not known until libbpf maps the arena. Thus libbpf does
the work in two steps:
- At open it records where each such pointer is and which variable it
points to.
- At load it maps the arena, copies the variables, and adds the address of
the variable to each pointer.
The pointer gets the address in user space. The prog can use the same
value, because an access to arena from a prog uses only the low 32 bits.
A pointer in arena to a variable that is not in arena has no correct value.
Fail the open.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/lib/bpf/libbpf.c | 83 +++++++++++++++++++++++++++++++++++++++++-
1 file changed, 81 insertions(+), 2 deletions(-)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 78ff9d653191..cc4a7d1f10da 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -796,6 +796,12 @@ struct bpf_object {
size_t arena_data_sz;
size_t arena_data_off;
size_t arena_bss_off; /* from the start of arena_data */
+ /* pointers to variables in arena that are stored in variables in arena */
+ struct arena_ptr {
+ size_t off; /* where the pointer is, from the start of arena_data */
+ size_t targ_off; /* the symbol it points to, from the start of arena_data */
+ } *arena_ptrs;
+ size_t arena_ptr_cnt;
void *jumptables_data;
size_t jumptables_data_sz;
@@ -4154,10 +4160,11 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
/*
* Only do relo for section with exec instructions,
- * struct_ops, maps, and read-only data that might
- * have pointers to functions.
+ * struct_ops, maps, read-only data that might
+ * have pointers to functions, and arena.
*/
if (!section_have_execinstr(obj, targ_sec_idx) &&
+ strcmp(name, ".rel" ARENA_SEC) &&
strcmp(name, ".rel" RODATA_SEC) &&
!str_has_pfx(name, ".rel" RODATA_SEC ".") &&
strcmp(name, ".rel" DATA_REL_RO_SEC) &&
@@ -5413,6 +5420,21 @@ bpf_object__reuse_map(struct bpf_map *map)
return 0;
}
+/* The arena is mapped. The address of a variable is the same for the program and for user space. */
+static void relocate_arena_ptrs(struct bpf_object *obj, const struct bpf_map *arena)
+{
+ void *data = arena->mmaped + obj->arena_data_off;
+ struct arena_ptr *p = obj->arena_ptrs;
+ __u64 val;
+ size_t i;
+
+ for (i = 0; i < obj->arena_ptr_cnt; i++, p++) {
+ memcpy(&val, data + p->off, sizeof(val));
+ val += (__u64)(unsigned long)data + p->targ_off;
+ memcpy(data + p->off, &val, sizeof(val));
+ }
+}
+
static int
bpf_object__populate_internal_map(struct bpf_object *obj, struct bpf_map *map)
{
@@ -5825,6 +5847,7 @@ bpf_object__create_maps(struct bpf_object *obj)
memcpy(map->mmaped + obj->arena_data_off, obj->arena_data,
obj->arena_data_sz);
zfree(&obj->arena_data);
+ relocate_arena_ptrs(obj, map);
}
}
if (map->init_slots_sz && map->def.type != BPF_MAP_TYPE_PROG_ARRAY) {
@@ -8184,6 +8207,59 @@ static int bpf_object__collect_rodata_relos(struct bpf_object *obj,
return 0;
}
+/*
+ * Collect pointers that are stored in variables in arena, like
+ * int __arena *p = &var;
+ * They are R_BPF_64_ABS64 relocations. The addend is stored in place.
+ */
+static int bpf_object__collect_arena_relos(struct bpf_object *obj,
+ Elf64_Shdr *shdr, Elf_Data *data)
+{
+ int i, nrels = shdr->sh_size / shdr->sh_entsize;
+ struct arena_ptr *ptrs;
+ const char *sym_name;
+ size_t sym_idx;
+ Elf64_Sym *sym;
+ Elf64_Rel *rel;
+
+ ptrs = libbpf_reallocarray(NULL, nrels, sizeof(*ptrs));
+ if (!ptrs)
+ return -ENOMEM;
+ obj->arena_ptrs = ptrs;
+
+ for (i = 0; i < nrels; i++, ptrs++) {
+ rel = elf_rel_by_idx(data, i);
+ if (!rel) {
+ pr_warn("sec '.rel%s': failed to get relo #%d\n", ARENA_SEC, i);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+
+ sym_idx = ELF64_R_SYM(rel->r_info);
+ sym = elf_sym_by_idx(obj, sym_idx);
+ if (!sym) {
+ pr_warn("sec '.rel%s': symbol #%zu not found for relo #%d\n",
+ ARENA_SEC, sym_idx, i);
+ return -LIBBPF_ERRNO__FORMAT;
+ }
+
+ if (ELF64_R_TYPE(rel->r_info) != R_BPF_64_ABS64 ||
+ (sym->st_shndx != obj->efile.arena_data_shndx &&
+ sym->st_shndx != obj->efile.arena_bss_shndx)) {
+ sym_name = elf_sym_str(obj, sym->st_name) ?: "<?>";
+ pr_warn("sec '%s': pointer to '%s' at offset %zu: it's not in arena\n",
+ ARENA_SEC, sym_name, (size_t)rel->r_offset);
+ return -LIBBPF_ERRNO__RELOC;
+ }
+
+ ptrs->off = rel->r_offset;
+ ptrs->targ_off = sym->st_value;
+ if (sym->st_shndx == obj->efile.arena_bss_shndx)
+ ptrs->targ_off += obj->arena_bss_off;
+ obj->arena_ptr_cnt++;
+ }
+ return 0;
+}
+
static int cmp_func_ptrs(const void *_a, const void *_b)
{
const struct func_ptr *a = _a;
@@ -8224,6 +8300,8 @@ static int bpf_object__collect_relos(struct bpf_object *obj)
err = bpf_object__collect_st_ops_relos(obj, shdr, data);
else if (idx == obj->efile.btf_maps_shndx)
err = bpf_object__collect_map_relos(obj, shdr, data);
+ else if (idx == obj->efile.arena_data_shndx)
+ err = bpf_object__collect_arena_relos(obj, shdr, data);
else
err = bpf_object__collect_prog_relos(obj, shdr, data);
if (err)
@@ -10275,6 +10353,7 @@ void bpf_object__close(struct bpf_object *obj)
close(obj->token_fd);
zfree(&obj->arena_data);
+ zfree(&obj->arena_ptrs);
zfree(&obj->jumptables_data);
obj->jumptables_data_sz = 0;
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 22/81] selftests/bpf: Test pointers in arena and .bss.addr_space.1
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (20 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 21/81] libbpf: Relocate pointers that are stored in arena Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 23/81] bpf: Introduce bpf register BPF_REG_ARENA_SP Alexei Starovoitov
` (60 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The initial values of variables in arena point to other variables in arena:
pointer points to
------------ -------------------------------------------
px an int
parr the third element of an array
head, next the elements of a list
pbig an element of an array in .bss.addr_space.1
The prog and user space must see the same pointers:
- The prog follows each pointer and examines the value that it finds.
- User space compares px, parr, head and next with the addresses in the
skeleton.
- User space reads through pbig the value that the prog stored there.
The variables in .bss.addr_space.1 must be zero and aligned, and the object
file must be smaller than these variables.
A second object has a pointer in arena to a variable that is not in arena.
The open of this object must fail.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/Makefile.skel | 3 +-
.../bpf/prog_tests/arena_data_ptrs.c | 67 ++++++++++
.../selftests/bpf/progs/arena_data_ptrs.c | 117 ++++++++++++++++++
.../bpf/progs/arena_data_ptrs_fail.c | 25 ++++
4 files changed, 211 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_data_ptrs.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_data_ptrs.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_data_ptrs_fail.c
diff --git a/tools/testing/selftests/bpf/Makefile.skel b/tools/testing/selftests/bpf/Makefile.skel
index e8699903feef..b8eaecc7f429 100644
--- a/tools/testing/selftests/bpf/Makefile.skel
+++ b/tools/testing/selftests/bpf/Makefile.skel
@@ -20,7 +20,8 @@ ifneq ($(BPF_CC),)
BPF_SRCS := $(notdir $(wildcard progs/*.c))
BPF_OBJS := $(patsubst %.c,$(RDIR)/%.bpf.o,$(BPF_SRCS))
-SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c
+SKEL_BLACKLIST := btf__% test_pinning_invalid.c test_sk_assign.c \
+ arena_data_ptrs_fail.c
LINKED_SKELS := test_static_linked.skel.h linked_funcs.skel.h \
linked_vars.skel.h linked_maps.skel.h linked_arena.skel.h \
diff --git a/tools/testing/selftests/bpf/prog_tests/arena_data_ptrs.c b/tools/testing/selftests/bpf/prog_tests/arena_data_ptrs.c
new file mode 100644
index 000000000000..79bc324f0909
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/arena_data_ptrs.c
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+
+struct elem {
+ struct elem *next;
+ int val;
+};
+
+#include "arena_data_ptrs.skel.h"
+
+static int run_prog(struct bpf_program *prog)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, opts);
+
+ if (!ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(prog), &opts), "run"))
+ return -1;
+ return opts.retval;
+}
+
+static void test_ptrs_and_bss(void)
+{
+ struct arena_data_ptrs *skel;
+ size_t elf_sz;
+
+ skel = arena_data_ptrs__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+
+ ASSERT_OK(run_prog(skel->progs.use_ptrs), "use_ptrs");
+ if (skel->bss->skip) {
+ printf("%s:SKIP:compiler doesn't support arena_cast\n", __func__);
+ test__skip();
+ goto out;
+ }
+ ASSERT_OK(run_prog(skel->progs.use_bss), "use_bss");
+
+ /* addresses are the same in user space */
+ ASSERT_EQ((void *)skel->arena->px, (void *)&skel->arena->x, "px");
+ ASSERT_EQ((void *)skel->arena->parr, (void *)&skel->arena->arr[2], "parr");
+ ASSERT_EQ((void *)skel->arena->head, (void *)&skel->arena->n1, "head");
+ ASSERT_EQ((void *)skel->arena->n1.next, (void *)&skel->arena->n2, "n1.next");
+ ASSERT_EQ(*(long *)skel->arena->pbig, 7, "*pbig");
+
+ /* 'big' is not in the file */
+ arena_data_ptrs__elf_bytes(&elf_sz);
+ ASSERT_LT(elf_sz, 8192 * sizeof(long), "elf_sz");
+out:
+ arena_data_ptrs__destroy(skel);
+}
+
+static void test_ptr_to_not_arena(void)
+{
+ struct bpf_object *obj;
+
+ obj = bpf_object__open_file("./arena_data_ptrs_fail.bpf.o", NULL);
+ if (!ASSERT_ERR_PTR(obj, "open"))
+ bpf_object__close(obj);
+}
+
+void test_arena_data_ptrs(void)
+{
+ if (test__start_subtest("ptrs_and_bss"))
+ test_ptrs_and_bss();
+ if (test__start_subtest("ptr_to_not_arena"))
+ test_ptr_to_not_arena();
+}
diff --git a/tools/testing/selftests/bpf/progs/arena_data_ptrs.c b/tools/testing/selftests/bpf/progs/arena_data_ptrs.c
new file mode 100644
index 000000000000..76e564b54e49
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_data_ptrs.c
@@ -0,0 +1,117 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define BPF_NO_KFUNC_PROTOTYPES
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+#include <bpf_arena_common.h>
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 64);
+} arena SEC(".maps");
+
+bool skip;
+
+#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
+
+struct elem {
+ struct elem __arena *next;
+ int val;
+};
+
+int __arena x = 42;
+int __arena arr[4] = { 1, 2, 3, 4 };
+struct elem __arena n3 = { NULL, 3 };
+struct elem __arena n2 = { &n3, 20 };
+struct elem __arena n1 = { &n2, 100 };
+
+/* variables that are zero and take no space in the file */
+#define __arena_bss __arena SEC(".bss.addr_space.1")
+char __arena_bss one;
+long __arena_bss big[8192];
+char __arena_bss aligned64[64] __attribute__((aligned(64)));
+
+/* pointers to variables in arena that are stored in variables in arena */
+int __arena * __arena px = &x;
+int __arena * __arena parr = &arr[2];
+struct elem __arena * __arena head = &n1;
+long __arena * __arena pbig = &big[8000];
+
+SEC("syscall")
+int use_ptrs(void *ctx)
+{
+ struct elem __arena *n;
+ int i, sum = 0;
+
+ if (*px != 42)
+ return 1;
+ *px = 43;
+ if (x != 43)
+ return 2;
+ if (parr[0] != 3 || parr[1] != 4)
+ return 3;
+ for (n = head, i = 0; n && i < 10; n = n->next, i++)
+ sum += n->val;
+ if (sum != 123)
+ return 4;
+ return 0;
+}
+
+SEC("syscall")
+int use_bss(void *ctx)
+{
+ unsigned long addr = (unsigned long)&aligned64;
+
+ if (one || big[0] || big[8000] || big[8191] || aligned64[63])
+ return 1;
+ *pbig = 7;
+ if (big[8000] != 7 || big[7999] || big[8001])
+ return 2;
+ big[8191] = 8;
+ one = 1;
+ /* variables that are not zero are where they were */
+ if (arr[3] != 4)
+ return 3;
+ /* hide the address from the compiler, it knows that '& 63' is 0 */
+ asm volatile ("" : "+r"(addr));
+ if (addr & 63)
+ return 4;
+ return 0;
+}
+
+#else
+
+/* The skeleton has to have what user space looks at */
+struct elem {
+ struct elem *next;
+ int val;
+};
+
+int __arena_global x;
+int __arena_global arr[4];
+struct elem __arena_global n2;
+struct elem __arena_global n1;
+int * __arena_global px;
+int * __arena_global parr;
+struct elem * __arena_global head;
+long * __arena_global pbig;
+
+SEC("syscall")
+int use_ptrs(void *ctx)
+{
+ skip = true;
+ return 0;
+}
+
+SEC("syscall")
+int use_bss(void *ctx)
+{
+ skip = true;
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_data_ptrs_fail.c b/tools/testing/selftests/bpf/progs/arena_data_ptrs_fail.c
new file mode 100644
index 000000000000..93533dea5f71
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_data_ptrs_fail.c
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define BPF_NO_KFUNC_PROTOTYPES
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_experimental.h"
+#include <bpf_arena_common.h>
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 1);
+} arena SEC(".maps");
+
+/* the pointer is in arena, what it points to is not */
+int y = 1;
+int *py SEC(".addr_space.1") = &y;
+
+SEC("syscall")
+int prog(void *ctx)
+{
+ return y;
+}
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 23/81] bpf: Introduce bpf register BPF_REG_ARENA_SP
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (21 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 22/81] selftests/bpf: Test pointers in arena and .bss.addr_space.1 Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 24/81] bpf: Add stacks for programs to arena Alexei Starovoitov
` (59 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A Rust function that gets &T does not know where T is. T can be on the
heap, a static, or a local variable of the caller. The same insn accesses
all three. The heap and the statics of a Rust prog are in arena. Thus a
local variable whose address is taken must be in arena too, and the prog
needs a second stack there.
Introduce BPF_REG_ARENA_SP, r12, the stack pointer of the stack in arena.
BPF_REG_AX stays the last register and becomes r13:
register before after
---------------- ------ -----
BPF_REG_PARAMS r11 r11
BPF_REG_ARENA_SP r12
BPF_REG_AX r12 r13
This patch only reserves the number. The verifier continues to reject insns
with r12.
BPF_REG_AX is visible in xlated insns. Adjust the selftests that match it
there.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/filter.h | 5 +-
kernel/bpf/core.c | 3 +-
.../selftests/bpf/prog_tests/ctx_rewrite.c | 14 ++--
.../bpf/progs/verifier_bpf_fastcall.c | 24 +++----
.../selftests/bpf/progs/verifier_may_goto_1.c | 12 ++--
.../selftests/bpf/progs/verifier_sdiv.c | 64 +++++++++----------
6 files changed, 62 insertions(+), 60 deletions(-)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 98254742ae37..f57916e95b94 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -60,8 +60,9 @@ struct ctl_table_header;
/* Kernel hidden auxiliary/helper register. */
#define BPF_REG_PARAMS MAX_BPF_REG
-#define BPF_REG_AX (MAX_BPF_REG + 1)
-#define MAX_BPF_EXT_REG (MAX_BPF_REG + 2)
+#define BPF_REG_ARENA_SP (MAX_BPF_REG + 1)
+#define BPF_REG_AX (MAX_BPF_REG + 2)
+#define MAX_BPF_EXT_REG (MAX_BPF_REG + 3)
#define MAX_BPF_JIT_REG MAX_BPF_EXT_REG
/* unused opcode to mark special call to bpf_tail_call() helper */
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 472efced9e40..94ee44ad88f7 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1368,7 +1368,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,
u32 imm_rnd = get_random_u32();
int off;
- BUILD_BUG_ON(BPF_REG_PARAMS + 2 != MAX_BPF_JIT_REG);
+ BUILD_BUG_ON(BPF_REG_PARAMS + 3 != MAX_BPF_JIT_REG);
+ BUILD_BUG_ON(BPF_REG_ARENA_SP + 2 != MAX_BPF_JIT_REG);
BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG);
/* Constraints on AX register:
diff --git a/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c b/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c
index 2c3124092b73..2d0bb771626d 100644
--- a/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c
+++ b/tools/testing/selftests/bpf/prog_tests/ctx_rewrite.c
@@ -69,19 +69,19 @@ static struct test_case test_cases[] = {
#if defined(__x86_64__) || defined(__aarch64__)
{
N(SCHED_CLS, struct __sk_buff, tstamp),
- .read = "r12 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);"
- "if w12 & 0x4 goto pc+1;"
+ .read = "r13 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);"
+ "if w13 & 0x4 goto pc+1;"
"goto pc+4;"
- "if w12 & 0x3 goto pc+1;"
+ "if w13 & 0x3 goto pc+1;"
"goto pc+2;"
"$dst = 0;"
"goto pc+1;"
"$dst = *(u64 *)($ctx + sk_buff::tstamp);",
- .write = "r12 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);"
- "if w12 & 0x4 goto pc+1;"
+ .write = "r13 = *(u8 *)($ctx + sk_buff::__mono_tc_offset);"
+ "if w13 & 0x4 goto pc+1;"
"goto pc+2;"
- "w12 &= -4;"
- "*(u8 *)($ctx + sk_buff::__mono_tc_offset) = r12;"
+ "w13 &= -4;"
+ "*(u8 *)($ctx + sk_buff::__mono_tc_offset) = r13;"
"*(u64 *)($ctx + sk_buff::tstamp) = $src;",
},
#endif
diff --git a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
index e0c389102db9..5cd503e87df9 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c
@@ -913,13 +913,13 @@ __xlated("...")
__xlated("4: r0 = &(void __percpu *)(r0)")
__xlated("...")
/* may_goto expansion starts */
-__xlated("6: r12 = *(u64 *)(r10 -24)")
-__xlated("7: if r12 == 0x0 goto pc+6")
-__xlated("8: r12 -= 1")
-__xlated("9: if r12 != 0x0 goto pc+2")
-__xlated("10: r12 = -24")
+__xlated("6: r13 = *(u64 *)(r10 -24)")
+__xlated("7: if r13 == 0x0 goto pc+6")
+__xlated("8: r13 -= 1")
+__xlated("9: if r13 != 0x0 goto pc+2")
+__xlated("10: r13 = -24")
__xlated("11: call unknown")
-__xlated("12: *(u64 *)(r10 -24) = r12")
+__xlated("12: *(u64 *)(r10 -24) = r13")
/* may_goto expansion ends */
__xlated("13: *(u64 *)(r10 -8) = r1")
__xlated("14: exit")
@@ -954,13 +954,13 @@ __xlated("1: *(u64 *)(r10 -16) =")
__xlated("2: r1 = 1")
__xlated("3: call bpf_get_smp_processor_id")
/* may_goto expansion starts */
-__xlated("4: r12 = *(u64 *)(r10 -24)")
-__xlated("5: if r12 == 0x0 goto pc+6")
-__xlated("6: r12 -= 1")
-__xlated("7: if r12 != 0x0 goto pc+2")
-__xlated("8: r12 = -24")
+__xlated("4: r13 = *(u64 *)(r10 -24)")
+__xlated("5: if r13 == 0x0 goto pc+6")
+__xlated("6: r13 -= 1")
+__xlated("7: if r13 != 0x0 goto pc+2")
+__xlated("8: r13 = -24")
__xlated("9: call unknown")
-__xlated("10: *(u64 *)(r10 -24) = r12")
+__xlated("10: *(u64 *)(r10 -24) = r13")
/* may_goto expansion ends */
__xlated("11: *(u64 *)(r10 -8) = r1")
__xlated("12: exit")
diff --git a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c
index 6fe602c8c7f3..49753e89cce9 100644
--- a/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c
+++ b/tools/testing/selftests/bpf/progs/verifier_may_goto_1.c
@@ -89,13 +89,13 @@ __arch_riscv64
__arch_loongarch
__xlated("0: *(u64 *)(r10 -16) = 65535")
__xlated("1: *(u64 *)(r10 -8) = 0")
-__xlated("2: r12 = *(u64 *)(r10 -16)")
-__xlated("3: if r12 == 0x0 goto pc+6")
-__xlated("4: r12 -= 1")
-__xlated("5: if r12 != 0x0 goto pc+2")
-__xlated("6: r12 = -16")
+__xlated("2: r13 = *(u64 *)(r10 -16)")
+__xlated("3: if r13 == 0x0 goto pc+6")
+__xlated("4: r13 -= 1")
+__xlated("5: if r13 != 0x0 goto pc+2")
+__xlated("6: r13 = -16")
__xlated("7: call unknown")
-__xlated("8: *(u64 *)(r10 -16) = r12")
+__xlated("8: *(u64 *)(r10 -16) = r13")
__xlated("9: r0 = 1")
__xlated("10: r0 = 2")
__xlated("11: exit")
diff --git a/tools/testing/selftests/bpf/progs/verifier_sdiv.c b/tools/testing/selftests/bpf/progs/verifier_sdiv.c
index d6cbe1e27fd6..0efbfa726764 100644
--- a/tools/testing/selftests/bpf/progs/verifier_sdiv.c
+++ b/tools/testing/selftests/bpf/progs/verifier_sdiv.c
@@ -778,10 +778,10 @@ __arch_x86_64
__xlated("0: r2 = 0x8000000000000000")
__xlated("2: r3 = -1")
__xlated("3: r4 = r2")
-__xlated("4: r12 = r3")
-__xlated("5: r12 += 1")
-__xlated("6: if r12 > 0x1 goto pc+4")
-__xlated("7: if r12 == 0x0 goto pc+1")
+__xlated("4: r13 = r3")
+__xlated("5: r13 += 1")
+__xlated("6: if r13 > 0x1 goto pc+4")
+__xlated("7: if r13 == 0x0 goto pc+1")
__xlated("8: r2 = 0")
__xlated("9: r2 = -r2")
__xlated("10: goto pc+1")
@@ -812,10 +812,10 @@ __success __retval(-5)
__arch_x86_64
__xlated("0: r2 = 5")
__xlated("1: r3 = -1")
-__xlated("2: r12 = r3")
-__xlated("3: r12 += 1")
-__xlated("4: if r12 > 0x1 goto pc+4")
-__xlated("5: if r12 == 0x0 goto pc+1")
+__xlated("2: r13 = r3")
+__xlated("3: r13 += 1")
+__xlated("4: if r13 > 0x1 goto pc+4")
+__xlated("5: if r13 == 0x0 goto pc+1")
__xlated("6: r2 = 0")
__xlated("7: r2 = -r2")
__xlated("8: goto pc+1")
@@ -890,10 +890,10 @@ __arch_x86_64
__xlated("0: w2 = -2147483648")
__xlated("1: w3 = -1")
__xlated("2: w4 = w2")
-__xlated("3: r12 = r3")
-__xlated("4: w12 += 1")
-__xlated("5: if w12 > 0x1 goto pc+4")
-__xlated("6: if w12 == 0x0 goto pc+1")
+__xlated("3: r13 = r3")
+__xlated("4: w13 += 1")
+__xlated("5: if w13 > 0x1 goto pc+4")
+__xlated("6: if w13 == 0x0 goto pc+1")
__xlated("7: w2 = 0")
__xlated("8: w2 = -w2")
__xlated("9: goto pc+1")
@@ -925,10 +925,10 @@ __arch_x86_64
__xlated("0: w2 = -5")
__xlated("1: w3 = -1")
__xlated("2: w4 = w2")
-__xlated("3: r12 = r3")
-__xlated("4: w12 += 1")
-__xlated("5: if w12 > 0x1 goto pc+4")
-__xlated("6: if w12 == 0x0 goto pc+1")
+__xlated("3: r13 = r3")
+__xlated("4: w13 += 1")
+__xlated("5: if w13 > 0x1 goto pc+4")
+__xlated("6: if w13 == 0x0 goto pc+1")
__xlated("7: w2 = 0")
__xlated("8: w2 = -w2")
__xlated("9: goto pc+1")
@@ -1004,10 +1004,10 @@ __arch_x86_64
__xlated("0: r2 = 0x8000000000000000")
__xlated("2: r3 = -1")
__xlated("3: r4 = r2")
-__xlated("4: r12 = r3")
-__xlated("5: r12 += 1")
-__xlated("6: if r12 > 0x1 goto pc+3")
-__xlated("7: if r12 == 0x1 goto pc+3")
+__xlated("4: r13 = r3")
+__xlated("5: r13 += 1")
+__xlated("6: if r13 > 0x1 goto pc+3")
+__xlated("7: if r13 == 0x1 goto pc+3")
__xlated("8: w2 = 0")
__xlated("9: goto pc+1")
__xlated("10: r2 s%= r3")
@@ -1034,10 +1034,10 @@ __arch_x86_64
__xlated("0: r2 = 5")
__xlated("1: r3 = -1")
__xlated("2: r4 = r2")
-__xlated("3: r12 = r3")
-__xlated("4: r12 += 1")
-__xlated("5: if r12 > 0x1 goto pc+3")
-__xlated("6: if r12 == 0x1 goto pc+3")
+__xlated("3: r13 = r3")
+__xlated("4: r13 += 1")
+__xlated("5: if r13 > 0x1 goto pc+3")
+__xlated("6: if r13 == 0x1 goto pc+3")
__xlated("7: w2 = 0")
__xlated("8: goto pc+1")
__xlated("9: r2 s%= r3")
@@ -1108,10 +1108,10 @@ __arch_x86_64
__xlated("0: w2 = -2147483648")
__xlated("1: w3 = -1")
__xlated("2: w4 = w2")
-__xlated("3: r12 = r3")
-__xlated("4: w12 += 1")
-__xlated("5: if w12 > 0x1 goto pc+3")
-__xlated("6: if w12 == 0x1 goto pc+4")
+__xlated("3: r13 = r3")
+__xlated("4: w13 += 1")
+__xlated("5: if w13 > 0x1 goto pc+3")
+__xlated("6: if w13 == 0x1 goto pc+4")
__xlated("7: w2 = 0")
__xlated("8: goto pc+1")
__xlated("9: w2 s%= w3")
@@ -1140,10 +1140,10 @@ __arch_x86_64
__xlated("0: w2 = -5")
__xlated("1: w3 = -1")
__xlated("2: w4 = w2")
-__xlated("3: r12 = r3")
-__xlated("4: w12 += 1")
-__xlated("5: if w12 > 0x1 goto pc+3")
-__xlated("6: if w12 == 0x1 goto pc+4")
+__xlated("3: r13 = r3")
+__xlated("4: w13 += 1")
+__xlated("5: if w13 > 0x1 goto pc+3")
+__xlated("6: if w13 == 0x1 goto pc+4")
__xlated("7: w2 = 0")
__xlated("8: goto pc+1")
__xlated("9: w2 s%= w3")
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 24/81] bpf: Add stacks for programs to arena
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (22 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 23/81] bpf: Introduce bpf register BPF_REG_ARENA_SP Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 25/81] bpf, x86: Add BPF_ATOMIC_LOCAL Alexei Starovoitov
` (58 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A prog that keeps local variables in arena needs a stack in arena while it
runs. r12 (BPF_REG_ARENA_SP) is the stack pointer of this stack.
One stack for each cpu is not sufficient. A run of a prog is interrupted,
is preempted or sleeps, and a second run starts on the same cpu. The second
run cannot put its frames after the frames of the first run, because the
first run can continue before the second run ends:
cpu 0 run 1: f() run 2: h() run 1: f() calls g()
stack | f | | f | h | | f | g |
^ g overwrites h
Thus each run takes a stack when it starts and gives it back when it exits.
Usually there is only one run on a cpu. Make this case a few insns without
a call:
stack that a run gets condition nsec per run
------------------------ ----------------------------- ------------
the per-cpu stack no other run is on this cpu 2.2
one of 16 more per cpu the per-cpu stack is taken 9.1 more
new pages for this run all 17 stacks are taken 1130 (1 page)
The times are for x86-64 (Zen 5). The first time is the difference to a
prog that has the same variables in the r10 stack.
Each run increments the per-cpu counter 'busy'. The run that finds 0 has
the per-cpu stack, and it stores 0 when it exits. The other runs call
bpf_arena_stack_get() and bpf_arena_stack_put(). One bit of r12,
BPF_ARENA_STACK_SLOW, shows at the exit which of the two ways is necessary.
This patch adds the data and the two functions.
"bpf: Support BPF_REG_ARENA_SP in the verifier" emits the insns that use
them. Progs without r12 get the same code.
Memory:
- A stack gets pages of the arena at its first use, in
bpf_arena_stack_get(), and keeps them.
- The bookkeeping has 152 bytes per cpu. It is in kernel memory, because
the prog and user space can write to arena.
- When there is no memory for a stack, the prog does not run. The kernel
counts a miss and writes a message to the stderr of the prog.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 30 ++++++++++
kernel/bpf/arena.c | 133 ++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 163 insertions(+)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 4c7c3f86ddbd..7437b42e0563 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -673,6 +673,31 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root,
struct bpf_spin_lock *spin_lock);
u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena);
u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena);
+/*
+ * The stack in arena of a cpu. The invocation of a program that finds it free
+ * takes it. The others, that interrupt or preempt that one, take a stack from
+ * bpf_arena_stack_get(). bpf_insert_arena_stack() emits the code.
+ */
+struct bpf_arena_cpu_stack {
+ /*
+ * Every invocation increments it. The one that finds 0 has the stack
+ * and sets it to 0 when it exits.
+ */
+ u64 busy;
+ u32 base; /* r12 when the main program starts, the low 32 bits of the address */
+ u32 size;
+};
+
+/*
+ * r12 is a multiple of 32 when the main program starts in the stack of the cpu.
+ * It has this bit set in a stack from bpf_arena_stack_get().
+ */
+#define BPF_ARENA_STACK_SLOW 16
+
+int bpf_arena_stack_init(struct bpf_arena *arena);
+void __percpu *bpf_arena_stack_percpu(struct bpf_arena *arena);
+u64 bpf_arena_stack_get(struct bpf_prog_aux *aux, u64 busy);
+void bpf_arena_stack_put(struct bpf_prog_aux *aux, u32 base);
u64 bpf_arena_map_kern_vm_start(struct bpf_map *map);
struct bpf_map *bpf_prog_arena(struct bpf_prog *prog);
int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
@@ -1918,6 +1943,11 @@ struct bpf_prog_aux {
u64 prog_array_member_cnt; /* counts how many times as member of prog_array */
struct mutex ext_mutex; /* mutex for freplace_link_cnt and prog_array_member_cnt */
struct bpf_arena *arena;
+ /*
+ * The size of the stack in arena that the program needs, a multiple of
+ * the page size. 0 when r12 is not used.
+ */
+ u32 arena_stack_size;
void (*recursion_detected)(struct bpf_prog *prog); /* callback if recursion is detected */
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
const struct btf_type *attach_func_proto;
diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
index 868589f07330..8b99b3b6fa02 100644
--- a/kernel/bpf/arena.c
+++ b/kernel/bpf/arena.c
@@ -67,6 +67,33 @@ struct bpf_arena {
struct irq_work free_irq;
struct work_struct free_work;
struct llist_head free_spans;
+ struct arena_stacks __percpu *stacks;
+};
+
+/*
+ * The stack in arena, see BPF_REG_ARENA_SP. An invocation of a program takes one
+ * when it starts and gives it back when it exits.
+ *
+ * Nothing here is in arena: the program and user space can write there.
+ *
+ * Invocations on a cpu don't nest: a program that is preempted or sleeps exits
+ * after the one that started later. Hence a bit per stack and not a counter.
+ * Programs don't migrate, so the stack is returned on the cpu it was taken on.
+ *
+ * Most of the time there is one invocation on a cpu. It takes the stack of
+ * the cpu, see struct bpf_arena_cpu_stack, and doesn't need these.
+ */
+#define ARENA_STACK_SLOTS 16
+
+struct arena_stack {
+ u32 base; /* low 32 bits of r12 when the main program starts */
+ u32 size; /* 0 until the first use */
+};
+
+struct arena_stacks {
+ struct bpf_arena_cpu_stack cpu;
+ unsigned long busy;
+ struct arena_stack slot[ARENA_STACK_SLOTS];
};
static void arena_free_worker(struct work_struct *work);
@@ -400,6 +427,7 @@ static void arena_map_free(struct bpf_map *map)
free_vm_area(arena->kern_vm);
range_tree_destroy(&arena->rt);
__free_page(arena->scratch_page);
+ free_percpu(arena->stacks);
bpf_map_area_free(arena);
}
@@ -1028,6 +1056,111 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,
irq_work_queue(&arena->free_irq);
}
+/* Called when a program that uses BPF_REG_ARENA_SP is loaded */
+int bpf_arena_stack_init(struct bpf_arena *arena)
+{
+ struct arena_stacks __percpu *stacks;
+
+ if (READ_ONCE(arena->stacks))
+ return 0;
+ stacks = bpf_map_alloc_percpu(&arena->map, sizeof(*stacks), __alignof__(*stacks),
+ GFP_USER | __GFP_NOWARN);
+ if (!stacks)
+ return -ENOMEM;
+ if (cmpxchg(&arena->stacks, NULL, stacks))
+ free_percpu(stacks);
+ return 0;
+}
+
+void __percpu *bpf_arena_stack_percpu(struct bpf_arena *arena)
+{
+ return arena->stacks;
+}
+
+/*
+ * Called at the start of the main program, in any context. When @busy is 0
+ * the program has taken the stack of the cpu and it's too small.
+ * Returns the low 32 bits of r12 with bit 32 set, since they may be 0, or 0 when
+ * there is no memory. The program doesn't run then.
+ */
+u64 bpf_arena_stack_get(struct bpf_prog_aux *aux, u64 busy)
+{
+ struct arena_stacks *stacks = this_cpu_ptr(aux->arena->stacks);
+ struct bpf_arena_cpu_stack *cpu = &stacks->cpu;
+ struct bpf_arena *arena = aux->arena;
+ u32 size = aux->arena_stack_size;
+ struct arena_stack *st = NULL;
+ struct bpf_stream_stage ss;
+ long addr;
+ int i;
+
+ if (!busy) {
+ /* It's not there yet, or programs that used it needed less */
+ addr = arena_alloc_pages(arena, 0, size >> PAGE_SHIFT, NUMA_NO_NODE, false);
+ if (!addr) {
+ WRITE_ONCE(cpu->busy, 0);
+ goto err;
+ }
+ if (cpu->size)
+ arena_free_pages(arena, cpu->base, (cpu->size + 64) >> PAGE_SHIFT, false);
+ /* r12 is not 0: neither at the start of the stack nor at the end */
+ cpu->base = addr + 32;
+ cpu->size = size - 64;
+ return BIT_ULL(32) | cpu->base;
+ }
+
+ while ((i = ffz(READ_ONCE(stacks->busy))) < ARENA_STACK_SLOTS) {
+ if (test_and_set_bit(i, &stacks->busy))
+ continue;
+ st = &stacks->slot[i];
+ if (st->size >= size)
+ return BIT_ULL(32) | st->base;
+ break;
+ }
+
+ /*
+ * It's the first use of the slot, or programs that used it needed less,
+ * or all slots are busy and bpf_arena_stack_put() is going to free the pages.
+ */
+ addr = arena_alloc_pages(arena, 0, size >> PAGE_SHIFT, NUMA_NO_NODE, false);
+ if (!addr) {
+ if (st)
+ clear_bit(i, &stacks->busy);
+ goto err;
+ }
+ if (!st)
+ return BIT_ULL(32) | (u32)(addr + BPF_ARENA_STACK_SLOW);
+
+ if (st->size)
+ arena_free_pages(arena, st->base, st->size >> PAGE_SHIFT, false);
+ st->base = addr + BPF_ARENA_STACK_SLOW;
+ st->size = size;
+ return BIT_ULL(32) | st->base;
+
+err:
+ bpf_prog_inc_misses_counter(aux->prog);
+ bpf_stream_stage(ss, aux->prog, BPF_STDERR, ({
+ bpf_stream_printk(ss, "ERROR: No memory for the stack in arena. The program didn't run\n");
+ }));
+ return 0;
+}
+
+/* Called before the main program exits with a stack from bpf_arena_stack_get() */
+void bpf_arena_stack_put(struct bpf_prog_aux *aux, u32 base)
+{
+ struct arena_stacks *stacks = this_cpu_ptr(aux->arena->stacks);
+ u32 size = aux->arena_stack_size;
+ int i;
+
+ for (i = 0; i < ARENA_STACK_SLOTS; i++) {
+ if (stacks->slot[i].base != base || !stacks->slot[i].size)
+ continue;
+ clear_bit(i, &stacks->busy);
+ return;
+ }
+ arena_free_pages(aux->arena, base, size >> PAGE_SHIFT, false);
+}
+
/*
* Reserve an arena virtual address range without populating it. This call stops
* bpf_arena_alloc_pages from adding pages to this range.
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 25/81] bpf, x86: Add BPF_ATOMIC_LOCAL
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (23 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 24/81] bpf: Add stacks for programs to arena Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 26/81] bpf: Support BPF_REG_ARENA_SP in the verifier Alexei Starovoitov
` (57 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A prog that has a stack in arena takes the per-cpu stack each time it
starts. It does this with fetch_add on a per-cpu word. Only this cpu
accesses the word. Thus the insn must be atomic only for an interrupt or
for a different task on the same cpu. But the fetch_add of BPF is atomic
for all cpus. On x86 it has the lock prefix, which waits for the stores in
the store buffer.
Add BPF_ATOMIC_LOCAL, a flag in imm of fetch_add. It means that the insn is
atomic only for code that runs on this cpu. One x86 insn is atomic for such
code. Thus the x86 JIT does not emit the lock prefix:
BPF_ADD | BPF_FETCH lock xadd
BPF_ADD | BPF_FETCH | BPF_ATOMIC_LOCAL xadd
The flag is for per-cpu memory only. It is correct there because progs do
not migrate.
Only the verifier emits the flag, and only when the JIT reports
bpf_jit_supports_local_atomics(). The other JITs get the usual fetch_add.
The first user is "bpf: Support BPF_REG_ARENA_SP in the verifier". The
verifier continues to reject the flag in the insns of a prog. The atomics
of progs get the same code.
The time that the flag saves depends on the stores that are in the store
buffer when the prog starts. On x86-64 (Zen 5), in nsec per run:
hot caches 0.33
after a store to 16 Mbyte of data (L3) 1.3
after a store that misses all caches 47
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/x86/net/bpf_jit_comp.c | 10 +++++++++-
include/linux/filter.h | 8 ++++++++
kernel/bpf/core.c | 6 ++++++
kernel/bpf/disasm.c | 7 +++++++
4 files changed, 30 insertions(+), 1 deletion(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d90b6d238b3a..a398c50b58ea 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -1593,7 +1593,10 @@ static int emit_atomic_rmw(u8 **pprog, u32 atomic_op,
{
u8 *prog = *pprog;
- if (atomic_op != BPF_XCHG)
+ /* one insn is atomic for what runs on this cpu */
+ if (atomic_op & BPF_ATOMIC_LOCAL)
+ atomic_op &= ~BPF_ATOMIC_LOCAL;
+ else if (atomic_op != BPF_XCHG)
EMIT1(0xF0); /* lock prefix */
maybe_emit_mod(&prog, dst_reg, src_reg, bpf_size == BPF_DW);
@@ -5141,6 +5144,11 @@ bool bpf_jit_supports_private_stack(void)
return true;
}
+bool bpf_jit_supports_local_atomics(void)
+{
+ return true;
+}
+
void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie)
{
#if defined(CONFIG_UNWINDER_ORC)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index f57916e95b94..01a0e9e5f399 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -80,6 +80,13 @@ struct ctl_table_header;
/* unused opcode to mark special atomic instruction */
#define BPF_PROBE_ATOMIC 0xe0
+/*
+ * Flag in imm of BPF_ATOMIC fetch_add: atomic for what runs on this cpu only,
+ * an interrupt or another task. The memory is per-cpu and the program
+ * doesn't migrate. It's cheaper where one insn without a lock does it.
+ */
+#define BPF_ATOMIC_LOCAL 0x1000
+
/* unused opcode to mark special ldsx instruction. Same as BPF_NOSPEC */
#define BPF_PROBE_MEM32SX 0xc0
@@ -1290,6 +1297,7 @@ bool bpf_jit_supports_ptr_xchg(void);
bool bpf_jit_supports_arena(void);
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
bool bpf_jit_supports_private_stack(void);
+bool bpf_jit_supports_local_atomics(void);
bool bpf_jit_supports_large_stack(void);
bool bpf_jit_supports_timed_may_goto(void);
bool bpf_jit_supports_fsession(void);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 94ee44ad88f7..364a3fa7cad5 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3584,6 +3584,12 @@ bool __weak bpf_jit_supports_private_stack(void)
return false;
}
+/* Whether JIT supports BPF_ATOMIC_LOCAL */
+bool __weak bpf_jit_supports_local_atomics(void)
+{
+ return false;
+}
+
/*
* Return TRUE if the JIT lays out frames of up to MAX_BPF_STACK_JIT bytes.
* Its prologue, epilogue and tail call sequences must encode such frame
diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c
index 889fc074496f..25517632c3ff 100644
--- a/kernel/bpf/disasm.c
+++ b/kernel/bpf/disasm.c
@@ -9,6 +9,7 @@
/* Only defined by the non-UAPI linux/filter.h, which this file cannot use. */
#define BPF_PROBE_ATOMIC 0xe0
+#define BPF_ATOMIC_LOCAL 0x1000
#define __BPF_FUNC_STR_FN(x) [BPF_FUNC_ ## x] = __stringify(bpf_ ## x)
static const char * const func_id_str[] = {
@@ -239,6 +240,12 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
insn->dst_reg,
insn->off, insn->src_reg);
+ else if (atomic && insn->imm == (BPF_ADD | BPF_FETCH | BPF_ATOMIC_LOCAL))
+ verbose(cbs->private_data, "(%02x) r%d = local%s_fetch_add((%s *)(r%d %+d), r%d)",
+ insn->code, insn->src_reg,
+ BPF_SIZE(insn->code) == BPF_DW ? "64" : "",
+ bpf_ldst_string[BPF_SIZE(insn->code) >> 3],
+ insn->dst_reg, insn->off, insn->src_reg);
else if (atomic &&
(insn->imm == BPF_ADD || insn->imm == BPF_AND ||
insn->imm == BPF_OR || insn->imm == BPF_XOR)) {
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 26/81] bpf: Support BPF_REG_ARENA_SP in the verifier
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (24 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 25/81] bpf, x86: Add BPF_ATOMIC_LOCAL Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 17:47 ` Kumar Kartikeya Dwivedi
2026-10-26 2:58 ` [PATCH bpf 27/81] bpf, x86: Support BPF_REG_ARENA_SP Alexei Starovoitov
` (56 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
In Rust &local, &heap and &static are the same type of pointer, and the
same insn dereferences all of them. The heap and the statics of a Rust prog
are in arena. Thus a local variable whose address is taken must be in arena
too. Spills and all other local variables stay in the r10 stack.
Add a second stack, in arena. r12 (BPF_REG_ARENA_SP) is its stack pointer.
The stack grows up, and r12 points to the end of the frame of the current
function:
f:
r12 += 32 ; the frame of f, 32 bytes in arena
*(u64 *)(r12 - 8) = r1 ; a local variable
r1 = r12
r1 += -8 ; its address, a number (scalar)
call g ; g accesses the variable through r1
exit ; the verifier adds 'r12 += -32' here
The verifier tracks neither r12 nor the content of this stack. It does not
track the rest of arena either. r12 holds the low 32 bits of the address,
and the JIT adds the base of the arena. Thus an access through r12 stays in
arena. But the code at the exit of the prog needs r12 to find the stack
that it must give back. Thus accept r12 in 4 insns only:
r12 += N first insn of a function, N % 16 == 0
rX = r12 number (scalar): user space address
rX = *(size *)(r12 - off) off is not checked against N
*(size *)(r12 - off) = rX or imm
Only the compiler knows N, the size of the frame. The verifier cannot
calculate it, because callees access the frame through numbers. The frames
of a call chain have 32 Kbyte at most.
Each run takes a stack, see "bpf: Add stacks for programs to arena". The
verifier adds this code to the main prog. busy, size and base are per-cpu:
entry before each exit
old = busy++ if (r12 & BPF_ARENA_STACK_SLOW)
if (old == 0 && size is sufficient) bpf_arena_stack_put()
r12 = base else
else busy = 0
r12 = bpf_arena_stack_get()
no memory: return 0
The usual path has 8 insns at the entry and 4 insns at the exit. The kernel
does not pass r12 to a callback. Thus a callback that uses r12 gets the
same code and takes a stack too.
Progs without r12 get the same code. A prog that uses r12 in one of its
functions pays in each run: up to 22 more x86 insns, no call, no lock
prefix. The cost on x86-64 (Zen 5), in nsec more than the same prog with
r10:
each run, hot caches 0.23 to 2.12
each run, cold caches and TLB 73
each run that finds the per-cpu stack taken 8.45
iteration of bpf_loop(), the callback uses r12 17.5 (1.13 -> 18.6)
call of a function with a frame, load, store below 0.23
A callback of bpf_loop() always finds the per-cpu stack taken, because the
main prog has it. The verifier also does not inline such a loop.
Limitations:
- The JIT must report bpf_jit_supports_arena_stack(). No JIT does in this
patch.
- The verifier rejects tail calls, because the prog would not give the
stack back.
- It rejects extension progs. The code at the entry uses r2-r5, and an
extension prog has arguments there.
- It rejects a jump to 'r12 += N', because the insn would run again.
- It rejects a direct call of a callback that takes a stack, because the
caller would get the r12 of the callee.
- Nothing can attach to a subprog, because trampolines do not keep r12.
- The private stack is off, because the x86 JIT has one register for the
two.
- bpf_throw() does not give the stack back. The stack stays taken until
the arena is freed, for all progs of the arena on that cpu. After 17
throws on a cpu each run there allocates a stack.
The test "invalid src register in LDX" of test_verifier used r12. Use r13.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 2 +
include/linux/bpf_verifier.h | 10 +
include/linux/filter.h | 44 ++++
kernel/bpf/backtrack.c | 2 +-
kernel/bpf/const_fold.c | 7 +
kernel/bpf/core.c | 25 ++
kernel/bpf/fixups.c | 213 +++++++++++++++++-
kernel/bpf/liveness.c | 8 +
kernel/bpf/scev.c | 9 +-
kernel/bpf/verifier.c | 157 ++++++++++++-
.../selftests/bpf/verifier/basic_stx_ldx.c | 4 +-
11 files changed, 474 insertions(+), 7 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 7437b42e0563..4ca06a217055 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1948,6 +1948,8 @@ struct bpf_prog_aux {
* the page size. 0 when r12 is not used.
*/
u32 arena_stack_size;
+ /* a callback that takes a stack in arena, like the main program does */
+ bool arena_stack_entry;
void (*recursion_detected)(struct bpf_prog *prog); /* callback if recursion is detected */
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
const struct btf_type *attach_func_proto;
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index ce3f0e275d9c..2c19c3aea1d5 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -939,6 +939,9 @@ enum priv_stack_mode {
PRIV_STACK_ADAPTIVE,
};
+/* The most that the frames of a call chain take in the stack in arena */
+#define BPF_ARENA_STACK_MAX (32 << 10)
+
struct bpf_subprog_info {
const char *name; /* name extracted from BTF */
u32 start; /* insn idx of function entry point */
@@ -952,6 +955,7 @@ struct bpf_subprog_info {
struct bpf_iarray *jt; /* jump table shared by all gotox of this subprogram */
u16 stack_depth; /* max. stack depth used by this function */
u16 stack_extra;
+ u16 arena_stack_depth; /* size of the frame in the stack in arena */
u32 insns_total;
u32 insns_self;
/* offsets in range [stack_depth .. fastcall_stack_off)
@@ -973,6 +977,9 @@ struct bpf_subprog_info {
bool keep_fastcall_stack: 1;
bool changes_pkt_data: 1;
bool might_sleep: 1;
+ bool uses_arena_sp: 1;
+ /* a callback that takes a stack in arena, like the main program does */
+ bool arena_stack_entry: 1;
u8 arg_slot_cnt:4;
enum priv_stack_mode priv_stack_mode;
@@ -1075,6 +1082,8 @@ struct bpf_verifier_env {
bool strict_alignment; /* perform strict pointer alignment checks */
bool test_state_freq; /* test verifier with different pruning frequency */
bool test_reg_invariants; /* fail verification on register invariants violations */
+ bool uses_arena_sp; /* the program has insns with BPF_REG_ARENA_SP */
+ u32 max_arena_stack_depth; /* of all call chains */
struct bpf_verifier_state *cur_state; /* current verifier state */
/* Search pruning optimization, array of list_heads for
* lists of struct bpf_verifier_state_list.
@@ -2010,6 +2019,7 @@ void bpf_opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env);
int bpf_opt_remove_dead_code(struct bpf_verifier_env *env);
int bpf_opt_remove_nops(struct bpf_verifier_env *env);
void bpf_lower_cast_mov(struct bpf_verifier_env *env);
+int bpf_insert_arena_stack(struct bpf_verifier_env *env);
int bpf_opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, const union bpf_attr *attr);
int bpf_convert_ctx_accesses(struct bpf_verifier_env *env);
int bpf_jit_subprogs(struct bpf_verifier_env *env);
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 01a0e9e5f399..9ee85cd48e6d 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -862,6 +862,49 @@ static inline bool is_stack_arg_stx(const struct bpf_insn *insn)
insn->off < 0 && insn->off % 8 == 0;
}
+/*
+ * BPF_REG_ARENA_SP, r12, points to the end of the frame of the function in
+ * the stack in arena, like r10 does in the stack of the program. The stack
+ * in arena grows up. r12 is used in four insns:
+ * r12 += N the first insn of the function
+ * N is a multiple of 16, so is r12
+ * rX = r12
+ * rX = *(size *)(r12 - off)
+ * *(size *)(r12 - off) = rX or imm
+ */
+static inline bool is_arena_sp_adj(const struct bpf_insn *insn)
+{
+ return insn->code == (BPF_ALU64 | BPF_ADD | BPF_K) &&
+ insn->dst_reg == BPF_REG_ARENA_SP;
+}
+
+static inline bool is_arena_sp_mov(const struct bpf_insn *insn)
+{
+ return insn->code == (BPF_ALU64 | BPF_MOV | BPF_X) &&
+ insn->src_reg == BPF_REG_ARENA_SP && !insn->off && !insn->imm;
+}
+
+static inline bool is_arena_sp_ldx(const struct bpf_insn *insn)
+{
+ return BPF_CLASS(insn->code) == BPF_LDX &&
+ (BPF_MODE(insn->code) == BPF_MEM || BPF_MODE(insn->code) == BPF_MEMSX) &&
+ insn->src_reg == BPF_REG_ARENA_SP;
+}
+
+/* st and stx */
+static inline bool is_arena_sp_st(const struct bpf_insn *insn)
+{
+ return (BPF_CLASS(insn->code) == BPF_ST || BPF_CLASS(insn->code) == BPF_STX) &&
+ BPF_MODE(insn->code) == BPF_MEM &&
+ insn->dst_reg == BPF_REG_ARENA_SP;
+}
+
+static inline bool is_arena_sp_insn(const struct bpf_insn *insn)
+{
+ return is_arena_sp_adj(insn) || is_arena_sp_mov(insn) ||
+ is_arena_sp_ldx(insn) || is_arena_sp_st(insn);
+}
+
#define BPF_SKB_CB_LEN QDISC_CB_PRIV_LEN
struct bpf_skb_data_end {
@@ -1297,6 +1340,7 @@ bool bpf_jit_supports_ptr_xchg(void);
bool bpf_jit_supports_arena(void);
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
bool bpf_jit_supports_private_stack(void);
+bool bpf_jit_supports_arena_stack(void);
bool bpf_jit_supports_local_atomics(void);
bool bpf_jit_supports_large_stack(void);
bool bpf_jit_supports_timed_may_goto(void);
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 43d5740034c0..47452ac7277a 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -325,7 +325,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
* sreg needs precision before this insn
*/
bt_clear_reg(bt, dreg);
- if (sreg != BPF_REG_FP)
+ if (sreg != BPF_REG_FP && sreg != BPF_REG_ARENA_SP)
bt_set_reg(bt, sreg);
} else {
/* dreg = K
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index f9edfde965dc..0e0cad2b4b8e 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -65,6 +65,13 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info *
ci_out[insn->dst_reg] = unknown;
return;
}
+ /* r12 is outside of it too */
+ if (is_arena_sp_adj(insn) || is_arena_sp_st(insn))
+ return;
+ if (is_arena_sp_mov(insn) || is_arena_sp_ldx(insn)) {
+ ci_out[insn->dst_reg] = unknown;
+ return;
+ }
switch (class) {
case BPF_ALU:
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 364a3fa7cad5..21f85c800f0c 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3590,6 +3590,12 @@ bool __weak bpf_jit_supports_local_atomics(void)
return false;
}
+/* Whether JIT supports BPF_REG_ARENA_SP */
+bool __weak bpf_jit_supports_arena_stack(void)
+{
+ return false;
+}
+
/*
* Return TRUE if the JIT lays out frames of up to MAX_BPF_STACK_JIT bytes.
* Its prologue, epilogue and tail call sequences must encode such frame
@@ -3719,6 +3725,25 @@ __weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena)
return 0;
}
+__weak int bpf_arena_stack_init(struct bpf_arena *arena)
+{
+ return -EOPNOTSUPP;
+}
+
+__weak void __percpu *bpf_arena_stack_percpu(struct bpf_arena *arena)
+{
+ return NULL;
+}
+
+__weak u64 bpf_arena_stack_get(struct bpf_prog_aux *aux, u64 busy)
+{
+ return 0;
+}
+
+__weak void bpf_arena_stack_put(struct bpf_prog_aux *aux, u32 base)
+{
+}
+
#ifdef CONFIG_BPF_SYSCALL
__weak bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write,
unsigned long fault_ip)
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index ae3347e724ca..c37325af659a 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -741,6 +741,213 @@ void bpf_lower_cast_mov(struct bpf_verifier_env *env)
}
}
+/*
+ * Make BPF_REG_ARENA_SP work. It holds the low 32 bits of the address, which is
+ * what JITs add to the base of arena.
+ * - the main program takes a stack when it starts and returns it when it exits:
+ * the stack of the cpu when it's free, without a call.
+ * See struct bpf_arena_cpu_stack. So does a callback.
+ * bpf_throw() doesn't return it: the stack stays taken until the arena is freed.
+ * - a function that has a frame drops it before exit.
+ * - 'rX = r12' gives the address like user space sees it, as other addresses
+ * in arena are.
+ * It's done after bpf_do_misc_fixups(), that takes a call for a call of a helper.
+ */
+
+#define CPU_STACK_OFF(field) offsetof(struct bpf_arena_cpu_stack, field)
+
+/* reg = this_cpu_ptr(stacks) */
+static struct bpf_insn *emit_arena_stack_percpu(struct bpf_verifier_env *env,
+ struct bpf_insn *patch, int reg)
+{
+ struct bpf_insn ld_stacks[2] = {
+ BPF_LD_IMM64(reg, (long)bpf_arena_stack_percpu(env->prog->aux->arena))
+ };
+
+ *patch++ = ld_stacks[0];
+ *patch++ = ld_stacks[1];
+ *patch++ = BPF_MOV64_PERCPU_REG(reg, reg);
+ return patch;
+}
+
+/*
+ * Take a stack at the start of the main program or of a callback.
+ *
+ * cnt = busy++;
+ * if (!cnt)
+ * goto free;
+ * slow:
+ * r12 = bpf_arena_stack_get(aux, cnt);
+ * if (r12)
+ * goto done;
+ * return 0;
+ * free:
+ * if (size < what the program needs)
+ * goto slow;
+ * r12 = base;
+ * done:
+ *
+ * r0 and r6-r9 are not initialized when a function starts. Neither are r2-r5 in
+ * the main program. A callback has arguments there.
+ */
+static int insert_arena_stack_get(struct bpf_verifier_env *env, int s)
+{
+ struct bpf_insn ld_aux[2] = { BPF_LD_IMM64(BPF_REG_1, (long)env->prog->aux) };
+ int local = bpf_jit_supports_local_atomics() ? BPF_ATOMIC_LOCAL : 0;
+ int ptr = s ? BPF_REG_6 : BPF_REG_2, cnt = s ? BPF_REG_7 : BPF_REG_3;
+ int size = s ? BPF_REG_8 : BPF_REG_4;
+ struct bpf_subprog_info *sub = &env->subprog_info[s];
+ struct bpf_insn *buf = env->insn_buf, *patch;
+ int jfree, jdone, jslow, slow, free, done, i, spill = 0, start = sub->start;
+ struct bpf_prog *new_prog;
+
+ if (s) {
+ spill = round_up(sub->stack_depth, BPF_REG_SIZE);
+ sub->stack_depth = spill + MAX_BPF_FUNC_REG_ARGS * BPF_REG_SIZE;
+ if (sub->stack_depth > env->stack_limit) {
+ verbose(env, "no room in the stack of %s() for its arguments\n",
+ bpf_subprog_name(env, s));
+ return -EACCES;
+ }
+ }
+
+ patch = emit_arena_stack_percpu(env, buf, ptr);
+ *patch++ = BPF_MOV32_IMM(cnt, 1);
+ *patch++ = BPF_ATOMIC_OP(BPF_DW, BPF_ADD | BPF_FETCH | local, ptr, cnt,
+ CPU_STACK_OFF(busy));
+ jfree = patch++ - buf;
+ slow = patch - buf;
+ if (!s)
+ *patch++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1);
+ for (i = BPF_REG_1; s && i <= BPF_REG_5; i++)
+ *patch++ = BPF_STX_MEM(BPF_DW, BPF_REG_FP, i, -(spill + i * BPF_REG_SIZE));
+ *patch++ = ld_aux[0];
+ *patch++ = ld_aux[1];
+ *patch++ = BPF_MOV64_REG(BPF_REG_2, cnt);
+ *patch++ = BPF_EMIT_CALL(bpf_arena_stack_get);
+ *patch++ = BPF_MOV32_REG(BPF_REG_ARENA_SP, BPF_REG_0);
+ if (!s)
+ *patch++ = BPF_MOV64_REG(BPF_REG_1, BPF_REG_6);
+ for (i = BPF_REG_1; s && i <= BPF_REG_5; i++)
+ *patch++ = BPF_LDX_MEM(BPF_DW, i, BPF_REG_FP, -(spill + i * BPF_REG_SIZE));
+ jdone = patch++ - buf;
+ *patch++ = BPF_MOV64_IMM(BPF_REG_0, 0);
+ *patch++ = BPF_EXIT_INSN();
+ free = patch - buf;
+ *patch++ = BPF_LDX_MEM(BPF_W, size, ptr, CPU_STACK_OFF(size));
+ jslow = patch++ - buf;
+ *patch++ = BPF_LDX_MEM(BPF_W, BPF_REG_ARENA_SP, ptr, CPU_STACK_OFF(base));
+ done = patch - buf;
+ *patch++ = env->prog->insnsi[start];
+
+ buf[jfree] = BPF_JMP_IMM(BPF_JEQ, cnt, 0, free - jfree - 1);
+ buf[jdone] = BPF_JMP_IMM(BPF_JNE, BPF_REG_0, 0, done - jdone - 1);
+ /* size is 0 until the stack is allocated */
+ buf[jslow] = BPF_JMP32_IMM(BPF_JLT, size, max(env->max_arena_stack_depth, 1),
+ slow - jslow - 1);
+
+ new_prog = bpf_patch_insn_data(env, start, buf, patch - buf);
+ if (!new_prog)
+ return -ENOMEM;
+ env->prog = new_prog;
+ /* a jump to the first insn doesn't take the stack again */
+ return adjust_jmp_off(env->prog, start, done);
+}
+
+int bpf_insert_arena_stack(struct bpf_verifier_env *env)
+{
+ struct bpf_map *arena = (struct bpf_map *)env->prog->aux->arena;
+ struct bpf_subprog_info *subprogs = env->subprog_info;
+ struct bpf_insn *insn_buf = env->insn_buf, *patch, *insn;
+ struct bpf_prog_aux *aux = env->prog->aux;
+ struct bpf_insn ld_aux[2] = { BPF_LD_IMM64(BPF_REG_1, (long)aux) };
+ struct bpf_prog *new_prog;
+ int i, cnt, cur = 0, err;
+
+ if (!env->uses_arena_sp)
+ return 0;
+
+ /* the caller would get r12 of the callee back */
+ for (i = 0; i < env->prog->len; i++) {
+ insn = env->prog->insnsi + i;
+ if (!bpf_pseudo_call(insn))
+ continue;
+ cur = bpf_find_subprog(env, i + insn->imm + 1);
+ if (cur < 0 || !subprogs[cur].arena_stack_entry)
+ continue;
+ verbose(env, "%s() uses R%d and is both called and a callback\n",
+ bpf_subprog_name(env, cur), BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+ cur = 0;
+
+ /*
+ * With spare bytes r12 is neither the start nor the end of the stack,
+ * which may be at 0, so 'rX = r12' is not NULL.
+ */
+ aux->arena_stack_size = round_up(env->max_arena_stack_depth + 64, PAGE_SIZE);
+
+ for (i = 0; i < env->prog->len; i++) {
+ insn = env->prog->insnsi + i;
+ if (subprogs[cur + 1].start == i)
+ cur++;
+ patch = insn_buf;
+
+ if (is_arena_sp_adj(insn)) {
+ insn->code = BPF_ALU | BPF_ADD | BPF_K;
+ env->insn_aux_data[i].zext_dst = true;
+ continue;
+ } else if (is_arena_sp_mov(insn)) {
+ /* rX = addr_space_cast(r12, 0, 1) */
+ if (!(arena->map_flags & BPF_F_NO_USER_CONV)) {
+ insn->off = BPF_ADDR_SPACE_CAST;
+ insn->imm = 1 << 16;
+ }
+ continue;
+ } else if (insn->code == (BPF_JMP | BPF_EXIT)) {
+ if (subprogs[cur].arena_stack_depth)
+ *patch++ = BPF_ALU32_IMM(BPF_ADD, BPF_REG_ARENA_SP,
+ -(int)subprogs[cur].arena_stack_depth);
+ if (!cur || subprogs[cur].arena_stack_entry) {
+ /* r1-r5 are free */
+ *patch++ = BPF_JMP32_IMM(BPF_JSET, BPF_REG_ARENA_SP,
+ BPF_ARENA_STACK_SLOW, 5);
+ patch = emit_arena_stack_percpu(env, patch, BPF_REG_2);
+ *patch++ = BPF_ST_MEM(BPF_DW, BPF_REG_2, CPU_STACK_OFF(busy), 0);
+ *patch++ = *insn;
+ *patch++ = BPF_MOV64_REG(BPF_REG_6, BPF_REG_0);
+ *patch++ = ld_aux[0];
+ *patch++ = ld_aux[1];
+ *patch++ = BPF_MOV64_REG(BPF_REG_2, BPF_REG_ARENA_SP);
+ *patch++ = BPF_EMIT_CALL(bpf_arena_stack_put);
+ *patch++ = BPF_MOV64_REG(BPF_REG_0, BPF_REG_6);
+ }
+ if (patch == insn_buf)
+ continue;
+ *patch++ = *insn;
+ } else {
+ continue;
+ }
+
+ cnt = patch - insn_buf;
+ new_prog = bpf_patch_insn_data(env, i, insn_buf, cnt);
+ if (!new_prog)
+ return -ENOMEM;
+ env->prog = new_prog;
+ i += cnt - 1;
+ }
+
+ /* From the last one, so that nothing moves in those that are yet to be done. */
+ for (i = env->subprog_cnt - 1; i >= 0; i--) {
+ if (i && !subprogs[i].arena_stack_entry)
+ continue;
+ err = insert_arena_stack_get(env, i);
+ if (err)
+ return err;
+ }
+ return 0;
+}
+
int bpf_opt_remove_nops(struct bpf_verifier_env *env)
{
struct bpf_insn_aux_data *aux = env->insn_aux_data;
@@ -1456,6 +1663,8 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->jited_linfo = prog->aux->jited_linfo;
func[i]->aux->linfo_idx = env->subprog_info[i].linfo_idx;
func[i]->aux->arena = prog->aux->arena;
+ func[i]->aux->arena_stack_size = prog->aux->arena_stack_size;
+ func[i]->aux->arena_stack_entry = env->subprog_info[i].arena_stack_entry;
func[i]->aux->uses_arena_insns = bpf_prog_has_arena_insns(func[i]);
func[i]->aux->used_maps = env->used_maps;
func[i]->aux->used_map_cnt = env->used_map_cnt;
@@ -2997,7 +3206,9 @@ int bpf_optimize_bpf_loop(struct bpf_verifier_env *env)
struct bpf_loop_inline_state *inline_state =
&env->insn_aux_data[i + delta].loop_inline_state;
- if (is_bpf_loop_call(insn) && inline_state->fit_for_inline) {
+ /* the callback that takes a stack in arena is not for a call from the program */
+ if (is_bpf_loop_call(insn) && inline_state->fit_for_inline &&
+ !subprogs[inline_state->callback_subprogno].arena_stack_entry) {
struct bpf_prog *new_prog;
stack_depth_extra = BPF_REG_SIZE * 3 + stack_depth_roundup;
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index f2989a4fd9a7..044f275d28e9 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -1412,6 +1412,11 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn,
} else if (is_stack_arg_ldx(insn)) {
slot = stack_arg_off_to_slot(insn->off);
at_out[insn->dst_reg] = (slot >= 0) ? at_stack_arg_entry[slot] : none;
+ } else if (is_arena_sp_adj(insn) || is_arena_sp_st(insn)) {
+ /* at_out[12] is a stack arg slot, not r12 */
+ } else if (is_arena_sp_mov(insn) || is_arena_sp_ldx(insn)) {
+ /* an address in arena, or what is loaded from arena */
+ *dst = none;
} else if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) {
if (code == BPF_MOV) {
*dst = none;
@@ -1637,6 +1642,9 @@ static int record_load_store_access(struct bpf_verifier_env *env,
*/
if (is_stack_arg_stx(insn) || is_stack_arg_st(insn) || is_stack_arg_ldx(insn))
return 0;
+ /* Likewise at[12] is not r12, and the access is to arena. */
+ if (is_arena_sp_st(insn) || is_arena_sp_ldx(insn))
+ return 0;
switch (class) {
case BPF_LDX:
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 6330deb2d3f2..44d524c966cd 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -670,6 +670,12 @@ static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
int i, id;
s64 imm;
+ /* reg2expr[12] is a stack slot, not r12 */
+ if (is_arena_sp_adj(insn) || is_arena_sp_st(insn))
+ return 0;
+ if (is_arena_sp_mov(insn) || is_arena_sp_ldx(insn))
+ goto mark_dst_unknown;
+
switch (class) {
case BPF_ALU:
case BPF_ALU64:
@@ -1156,7 +1162,8 @@ static void collect_store_base_regs(struct bpf_verifier_env *env,
u16 base_regs = 0;
u32 r;
- if (size == BPF_W || size == BPF_DW) {
+ if ((size == BPF_W || size == BPF_DW) &&
+ !is_arena_sp_st(insn) && !is_arena_sp_ldx(insn)) {
if ((class == BPF_STX || class == BPF_ST) && insn->dst_reg != BPF_REG_FP)
base_regs |= BIT(insn->dst_reg);
else if (class == BPF_LDX && insn->src_reg != BPF_REG_FP)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 203edbe41397..92b563a9b158 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -5760,6 +5760,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
struct bpf_insn *insn = env->prog->insnsi;
int depth = 0, frame = 0, i, subprog_end, subprog_depth;
bool tail_call_reachable = false;
+ int arena_depth = 0;
int total;
int tmp;
@@ -5800,6 +5801,24 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
* tailcall will unwind the current stack frame but it will not get rid
* of caller's stack as shown on the example above.
*/
+ /* The kernel calls a callback. It knows nothing about r12. */
+ if (subprog[idx].uses_arena_sp) {
+ for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
+ if (!subprog[tmp].is_cb)
+ continue;
+ subprog[tmp].arena_stack_entry = true;
+ break;
+ }
+ }
+ arena_depth += subprog[idx].arena_stack_depth;
+ if (arena_depth > env->max_arena_stack_depth)
+ env->max_arena_stack_depth = arena_depth;
+ if (arena_depth > BPF_ARENA_STACK_MAX) {
+ verbose(env, "combined size of the frames in the stack in arena is %d. Too large\n",
+ arena_depth);
+ return -EACCES;
+ }
+
if (idx && subprog[idx].has_tail_call && depth >= 256) {
verbose(env,
"tail_calls are not allowed when call stack of previous frames is %d bytes. Too large\n",
@@ -5988,6 +6007,7 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
return 0;
if (subprog[idx].priv_stack_mode != PRIV_STACK_ADAPTIVE)
depth -= round_up_stack_depth(env, subprog[idx].stack_depth);
+ arena_depth -= subprog[idx].arena_stack_depth;
/* pop caller idx from callee */
idx = dinfo[idx].caller;
@@ -6016,10 +6036,20 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
for (int i = 0; i < env->subprog_cnt; i++) {
if (si[i].has_tail_call) {
+ /* the stack in arena wouldn't be returned */
+ if (env->uses_arena_sp) {
+ verbose(env, "tail_calls are not allowed in programs that use R%d\n",
+ BPF_REG_ARENA_SP);
+ kvfree(dinfo);
+ return -EINVAL;
+ }
priv_stack_mode = NO_PRIV_STACK;
break;
}
}
+ /* x86 JIT has one register for both */
+ if (env->uses_arena_sp)
+ priv_stack_mode = NO_PRIV_STACK;
if (priv_stack_mode == PRIV_STACK_UNKNOWN)
priv_stack_mode = bpf_enable_priv_stack(env->prog);
@@ -20282,12 +20312,52 @@ static int check_indirect_jump(struct bpf_verifier_env *env, struct bpf_insn *in
return INSN_IDX_UPDATED;
}
+/*
+ * r12 is not in regs[]. Only the kernel and 'r12 += N' change it, so it points
+ * to the stack in arena. What is there is not tracked, like the rest of arena.
+ */
+static int check_arena_sp_insn(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+ struct bpf_reg_state arena = { .type = PTR_TO_ARENA };
+ argno_t argno = argno_from_reg(BPF_REG_ARENA_SP);
+ int size = BPF_SIZE(insn->code);
+ int err, value_regno = -1;
+
+ if (is_arena_sp_adj(insn))
+ return 0;
+
+ /* the address of the frame is a number */
+ if (is_arena_sp_mov(insn))
+ return check_reg_arg(env, insn->dst_reg, DST_OP);
+
+ if (is_arena_sp_ldx(insn)) {
+ err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK);
+ err = err ?: check_mem_access(env, env->insn_idx, &arena, argno, insn->off, size,
+ BPF_READ, insn->dst_reg, false,
+ BPF_MODE(insn->code) == BPF_MEMSX);
+ return err ?: save_aux_ptr_type(env, PTR_TO_ARENA, false);
+ }
+
+ if (BPF_CLASS(insn->code) == BPF_STX) {
+ err = check_reg_arg(env, insn->src_reg, SRC_OP);
+ if (err)
+ return err;
+ value_regno = insn->src_reg;
+ }
+ err = check_mem_access(env, env->insn_idx, &arena, argno, insn->off, size,
+ BPF_WRITE, value_regno, false, false);
+ return err ?: save_aux_ptr_type(env, PTR_TO_ARENA, false);
+}
+
static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
{
int err;
struct bpf_insn *insn = &env->prog->insnsi[env->insn_idx];
u8 class = BPF_CLASS(insn->code);
+ if (is_arena_sp_insn(insn))
+ return check_arena_sp_insn(env, insn);
+
switch (class) {
case BPF_ALU:
case BPF_ALU64:
@@ -21461,6 +21531,74 @@ static int check_insn_fields(struct bpf_verifier_env *env, struct bpf_insn *insn
}
}
+static bool jumps_to_arena_sp_adj(struct bpf_insn *insn)
+{
+ u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+
+ if (class != BPF_JMP && class != BPF_JMP32)
+ return false;
+ if (op == BPF_CALL || op == BPF_EXIT || insn->code == (BPF_JMP | BPF_JA | BPF_X))
+ return false;
+ return is_arena_sp_adj(insn + bpf_jmp_offset(insn) + 1);
+}
+
+/*
+ * Find the insns with BPF_REG_ARENA_SP. The sizes of the frames are in the text
+ * of the program.
+ */
+static int check_arena_stack(struct bpf_verifier_env *env)
+{
+ struct bpf_subprog_info *sub = env->subprog_info;
+ struct bpf_insn *insn = env->prog->insnsi;
+ int i;
+
+ for (i = 0; i < env->prog->len; i++, insn++) {
+ if (i == (sub + 1)->start)
+ sub++;
+ /* it would be executed again */
+ if (jumps_to_arena_sp_adj(insn)) {
+ verbose(env, "jump to the insn that changes R%d\n", BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+ if (!is_arena_sp_insn(insn))
+ continue;
+ sub->uses_arena_sp = true;
+ env->uses_arena_sp = true;
+ if (!is_arena_sp_adj(insn))
+ continue;
+ if (i != sub->start) {
+ verbose(env, "R%d can be changed by the first insn of a function only\n",
+ BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+ if (insn->imm <= 0 || insn->imm % 16 || insn->imm > BPF_ARENA_STACK_MAX) {
+ verbose(env, "invalid size of the frame in the stack in arena %d\n",
+ insn->imm);
+ return -EINVAL;
+ }
+ sub->arena_stack_depth = insn->imm;
+ }
+ if (!env->uses_arena_sp)
+ return 0;
+
+ if (!env->prog->aux->arena) {
+ verbose(env, "R%d can only be used in a program that has an associated arena\n",
+ BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+ if (!bpf_jit_supports_arena_stack() || !bpf_jit_supports_percpu_insn() ||
+ bpf_prog_is_offloaded(env->prog->aux)) {
+ verbose(env, "JIT doesn't support R%d\n", BPF_REG_ARENA_SP);
+ return -EOPNOTSUPP;
+ }
+ /* bpf_insert_arena_stack() relies on r2-r5 being free when the program starts */
+ if (env->prog->type == BPF_PROG_TYPE_EXT) {
+ verbose(env, "R%d is not supported in extension programs\n", BPF_REG_ARENA_SP);
+ return -EINVAL;
+ }
+ return bpf_arena_stack_init(env->prog->aux->arena);
+}
+
/*
* Check that insns are sane and rewrite pseudo imm in ld_imm64 instructions:
*
@@ -21481,11 +21619,13 @@ static int check_and_resolve_insns(struct bpf_verifier_env *env)
for (i = 0; i < insn_cnt; i++, insn++) {
if (insn->dst_reg >= MAX_BPF_REG &&
- !is_stack_arg_st(insn) && !is_stack_arg_stx(insn)) {
+ !is_stack_arg_st(insn) && !is_stack_arg_stx(insn) &&
+ !is_arena_sp_adj(insn) && !is_arena_sp_st(insn)) {
verbose(env, "R%d is invalid\n", insn->dst_reg);
return -EINVAL;
}
- if (insn->src_reg >= MAX_BPF_REG && !is_stack_arg_ldx(insn)) {
+ if (insn->src_reg >= MAX_BPF_REG && !is_stack_arg_ldx(insn) &&
+ !is_arena_sp_mov(insn) && !is_arena_sp_ldx(insn)) {
verbose(env, "R%d is invalid\n", insn->src_reg);
return -EINVAL;
}
@@ -22618,6 +22758,12 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
bpf_log(log, "Cannot attach to a target with arena context arguments\n");
return -EOPNOTSUPP;
}
+ /* the trampoline and the extension don't preserve BPF_REG_ARENA_SP */
+ if (subprog && aux->arena_stack_size) {
+ bpf_log(log, "Cannot attach to a subprog of a program that uses R%d\n",
+ BPF_REG_ARENA_SP);
+ return -EOPNOTSUPP;
+ }
if (aux->func && aux->func[subprog]->aux->exception_cb) {
bpf_log(log,
"%s programs cannot attach to exception callback\n",
@@ -24000,6 +24146,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
+ ret = check_arena_stack(env);
+ if (ret < 0)
+ goto skip_full_check;
+
/* Find pointers to functions in the read-only maps of the program. */
ret = resolve_func_ptrs(env);
if (ret < 0)
@@ -24136,6 +24286,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret == 0)
ret = bpf_do_misc_fixups(env);
+ if (ret == 0)
+ ret = bpf_insert_arena_stack(env);
+
/* do 32-bit optimization after insn patching has done so those patched
* insns could be handled correctly.
*/
diff --git a/tools/testing/selftests/bpf/verifier/basic_stx_ldx.c b/tools/testing/selftests/bpf/verifier/basic_stx_ldx.c
index 7a0aab3f2cd2..66fde558df33 100644
--- a/tools/testing/selftests/bpf/verifier/basic_stx_ldx.c
+++ b/tools/testing/selftests/bpf/verifier/basic_stx_ldx.c
@@ -28,10 +28,10 @@
{
"invalid src register in LDX",
.insns = {
- BPF_LDX_MEM(BPF_B, BPF_REG_0, 12, 0),
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, 13, 0),
BPF_EXIT_INSN(),
},
- .errstr = "R12 is invalid",
+ .errstr = "R13 is invalid",
.result = REJECT,
},
{
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 26/81] bpf: Support BPF_REG_ARENA_SP in the verifier
2026-10-26 2:58 ` [PATCH bpf 26/81] bpf: Support BPF_REG_ARENA_SP in the verifier Alexei Starovoitov
@ 2026-10-26 17:47 ` Kumar Kartikeya Dwivedi
2026-10-26 23:51 ` Alexei Starovoitov
0 siblings, 1 reply; 144+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-10-26 17:47 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: bpf, daniel, andrii, eddyz87
On Mon, 26 Oct 2026 at 04:02, Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> From: Alexei Starovoitov <ast@kernel.org>
>
> In Rust &local, &heap and &static are the same type of pointer, and the
> same insn dereferences all of them. The heap and the statics of a Rust prog
> are in arena. Thus a local variable whose address is taken must be in arena
> too. Spills and all other local variables stay in the r10 stack.
>
> Add a second stack, in arena. r12 (BPF_REG_ARENA_SP) is its stack pointer.
> The stack grows up, and r12 points to the end of the frame of the current
> function:
>
> f:
> r12 += 32 ; the frame of f, 32 bytes in arena
> *(u64 *)(r12 - 8) = r1 ; a local variable
> r1 = r12
> r1 += -8 ; its address, a number (scalar)
> call g ; g accesses the variable through r1
> exit ; the verifier adds 'r12 += -32' here
>
> The verifier tracks neither r12 nor the content of this stack. It does not
> track the rest of arena either. r12 holds the low 32 bits of the address,
> and the JIT adds the base of the arena. Thus an access through r12 stays in
> arena. But the code at the exit of the prog needs r12 to find the stack
> that it must give back. Thus accept r12 in 4 insns only:
>
> r12 += N first insn of a function, N % 16 == 0
> rX = r12 number (scalar): user space address
> rX = *(size *)(r12 - off) off is not checked against N
> *(size *)(r12 - off) = rX or imm
>
> Only the compiler knows N, the size of the frame. The verifier cannot
> calculate it, because callees access the frame through numbers. The frames
> of a call chain have 32 Kbyte at most.
>
> Each run takes a stack, see "bpf: Add stacks for programs to arena". The
> verifier adds this code to the main prog. busy, size and base are per-cpu:
>
> entry before each exit
>
> old = busy++ if (r12 & BPF_ARENA_STACK_SLOW)
> if (old == 0 && size is sufficient) bpf_arena_stack_put()
> r12 = base else
> else busy = 0
> r12 = bpf_arena_stack_get()
> no memory: return 0
>
> The usual path has 8 insns at the entry and 4 insns at the exit. The kernel
> does not pass r12 to a callback. Thus a callback that uses r12 gets the
> same code and takes a stack too.
>
> Progs without r12 get the same code. A prog that uses r12 in one of its
> functions pays in each run: up to 22 more x86 insns, no call, no lock
> prefix. The cost on x86-64 (Zen 5), in nsec more than the same prog with
> r10:
>
> each run, hot caches 0.23 to 2.12
> each run, cold caches and TLB 73
> each run that finds the per-cpu stack taken 8.45
> iteration of bpf_loop(), the callback uses r12 17.5 (1.13 -> 18.6)
> call of a function with a frame, load, store below 0.23
>
> A callback of bpf_loop() always finds the per-cpu stack taken, because the
> main prog has it. The verifier also does not inline such a loop.
>
> Limitations:
> - The JIT must report bpf_jit_supports_arena_stack(). No JIT does in this
> patch.
> - The verifier rejects tail calls, because the prog would not give the
> stack back.
> - It rejects extension progs. The code at the entry uses r2-r5, and an
> extension prog has arguments there.
> - It rejects a jump to 'r12 += N', because the insn would run again.
> - It rejects a direct call of a callback that takes a stack, because the
> caller would get the r12 of the callee.
> - Nothing can attach to a subprog, because trampolines do not keep r12.
> - The private stack is off, because the x86 JIT has one register for the
> two.
> - bpf_throw() does not give the stack back. The stack stays taken until
> the arena is freed, for all progs of the arena on that cpu. After 17
> throws on a cpu each run there allocates a stack.
>
> The test "invalid src register in LDX" of test_verifier used r12. Use r13.
>
> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
> ---
Some initial questions without reading the code:
How will this split stack model work with trusted objects whose
pointers on the stack are taken and passed around (dynptrs, iterators,
etc.)?
Also, will this lead to miscompiled cases where ptr to ptr to trusted is
passed to supbprog as r12 relative access? (As a way for the callee to
fill a trusted pointer for the caller, etc.)?
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 26/81] bpf: Support BPF_REG_ARENA_SP in the verifier
2026-10-26 17:47 ` Kumar Kartikeya Dwivedi
@ 2026-10-26 23:51 ` Alexei Starovoitov
2026-10-27 0:18 ` Kumar Kartikeya Dwivedi
0 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 23:51 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi; +Cc: bpf, daniel, andrii, eddyz87
On Mon Oct 26, 2026 at 5:47 PM UTC, Kumar Kartikeya Dwivedi wrote:
>
> Some initial questions without reading the code:
> How will this split stack model work with trusted objects whose
> pointers on the stack are taken and passed around (dynptrs, iterators,
> etc.)?
dynptrs and iterators won't work in rust. No need to drag old concepts
into rust ecosystem.
r12 is opt-in. rust will compile with extra llvm backend flag.
It's not for old C code.
> Also, will this lead to miscompiled cases where ptr to ptr to trusted is
> passed to supbprog as r12 relative access? (As a way for the callee to
> fill a trusted pointer for the caller, etc.)?
If somebody will use this flag for C code the ptr to ptr will be arena ptr
and won't become trusted.
Not sure I made it clear, but some rust crates need your typed-arena feature.
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 26/81] bpf: Support BPF_REG_ARENA_SP in the verifier
2026-10-26 23:51 ` Alexei Starovoitov
@ 2026-10-27 0:18 ` Kumar Kartikeya Dwivedi
2026-10-27 18:45 ` Alexei Starovoitov
0 siblings, 1 reply; 144+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-10-27 0:18 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: bpf, daniel, andrii, eddyz87
On Tue, 27 Oct 2026 at 00:51, Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Mon Oct 26, 2026 at 5:47 PM UTC, Kumar Kartikeya Dwivedi wrote:
> >
> > Some initial questions without reading the code:
> > How will this split stack model work with trusted objects whose
> > pointers on the stack are taken and passed around (dynptrs, iterators,
> > etc.)?
>
> dynptrs and iterators won't work in rust. No need to drag old concepts
> into rust ecosystem.
I am less optimistic about this. For certain things, like open coded
iterators on kernel collections, there is no other choice. This is
also true for sched-ext (where we iterate over tasks in a DSQ in
various places). We also have bpf_local_irq_save(). Likewise for
certain dynptr types that touch actual kernel objects, even though we
could replace them with slices/fat pointers in Rust for normal memory
accesses happening in arena.
It might require teaching the compiler not to move certain types on
the stack to r12 relative accesses, but it just feels ugly. Also,
unsure how various assumptions around address contiguity get affected
when stack reuse occurs, e.g. where a buffer created in same slots
previously holding trusted contents whose scope ended get reused for
an address taken buffer. Perhaps the compiler pass works around those.
Overall, I think it is less clear to me (or anyone looking at patches)
as to why you did things this way. You've mentioned the problem with
conflicting JIT for the same insn seeing PTR_TO_STACK/PTR_TO_ARENA
before. Are there real patterns that existing stack tracking simply
cannot salvage? More information around the motivation and examples
would be useful in understanding all this.
There are a lot of patches, so perhaps it is explained somewhere that
I missed, but otherwise existing explanations are very terse. They
tell how the implementation works but not why certain choices were
made.
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 26/81] bpf: Support BPF_REG_ARENA_SP in the verifier
2026-10-27 0:18 ` Kumar Kartikeya Dwivedi
@ 2026-10-27 18:45 ` Alexei Starovoitov
2026-10-27 20:23 ` Kumar Kartikeya Dwivedi
0 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-27 18:45 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi; +Cc: bpf, daniel, andrii, eddyz87
On Mon, Oct 26, 2026 at 5:18 PM Kumar Kartikeya Dwivedi <memxor@gmail.com> wrote:
> Overall, I think it is less clear to me (or anyone looking at patches)
> as to why you did things this way.
point taken.
I was wrong in my previous email. An iterator over kernel objects
stays in the r10 stack, and it works today.
> Are there real patterns that existing stack tracking simply cannot
> salvage?
Yes.
Pattern 1: one insn, two kinds of memory
sum(&local_array) r1 = fp-64 --+
+--> r2 = *(u64 *)(r1 + 0)
sum(&vec) r1 = arena --+
one insn, one JIT form
We discussed an idea to let llvm clone such function.
I tried it, and it didn't work.
Short summary:
case why a copy does not help
--------------------------------- ------------------------------------
the pointer comes from memory no kind for the compiler, a scalar
for the verifier
an indirect call: &dyn Trait, fmt one function for the two kinds
if c { &local } else { &*boxed } one pointer, two kinds, one function
n pointer arguments up to 2^n copies
and real case in <&PanicInfo as Display>::fmt().
Pattern 2: the address of a local goes to memory, a load gives a scalar
That's typical rust heap.
Pattern 3: LLVM removes a bounds check, here in core::fmt::num
loop { remain /= 10000; offset -= 4; buf[offset] = ...; }
and 'buf' is on stack.
run-time clamp won't help, since real tracked spill/fills are in r10.
Pattern 4: fp in an argument
...
So an address-taken local must be where the heap is... in arena.
experiments for arena-stack
where the local is what breaks it
-------------------------------------- ---------------------------------
- structs by value, more inlining an optimization. Pattern 2 stays
- a second vm area or arena, a map value a different base: pattern 1
- arena, a fixed frame for each function recursion, a second run on the
and cpu, as in the private stack same cpu
- arena stack, sp in a per-cpu cell preemption, see below
- arena stack, sp in bpf_run_ctx load, add, store in each function
- arena stack, sp in a hidden argument an argument and a callee saved
register. BTF does not match
- arena stack, sp in a reserved r9 a verifier change too, and one
register less for the prog
- arena stack, sp in r12, the kernel the series. It costs verifier
supplies the stack code and a register in each JIT
The choices made for this arena-stack design:
in the series what was rejected and why
------------------------------- ----------------------------------------
- one 'r12 += N', the first insn 'r12 -= rX': the depth has no bound
of a function several 'r12 += imm': verifier state
- the verifier tracks nothing tracked slots: a store through a scalar
invalidates them
- 'rX = r12' gives a scalar PTR_TO_ARENA: after a load from memory
it is a scalar. Two types at one insn
- the cast of a kernel pointer is an error: two progs cannot share
a move (patch 18) fill(&mut Event), ring buffer and Box
- the kernel supplies the stacks, compiler, libbpf: the entry code is in
one set for each arena each object file
- contiguous, 32 Kbyte at most, fixed, 4 per cpu: 512 Mbyte of addresses
pages at the first use segments: each function can fail
pages on a fault: can fail in mid-run
- regular arena pages hidden from user space: no gain. It can
corrupt all other data of the prog
- a run takes the per-cpu stack: preemption off: no sleepable progs
a counter, local xadd. If it is two calls, get and put: 4 nsec
taken: one of 16 more stacks, split, sleepable or not: two mechanisms
then new pages skip the second run: not for a scheduler
- grows up: r12 - off, like r10 down, r12 + off: harder to read
- x86-64: R9. Arg 6 of a BPF call reject 6 arguments: Rust has them
is on the stack r12 in a frame slot: a load per access
- a callback takes its own stack reject r12 there: scx_central needs it
- bpf_throw() keeps the stack: a slot in the frame of main: an irq
the kernel frees the arena as overwrites it after rsp = rbp
a whole, and Rust unwinds give back first: no record of new pages
> It might require teaching the compiler not to move certain types on
> the stack to r12 relative accesses, but it just feels ugly.
it might feel fragile, but it worked for huge corpus of rust crates:
https://github.com/llvm/llvm-project/pull/235577
Your dsq iterator exapmle is 'alloca [48 x i8]' in the IR, so it works
(I replied too soon yesteday).
llvm selects the stack by use:
rule 1 r10 The function that owns the local passes its address to
a helper or a kfunc: iterator, irq flag, dynptr, map key.
rule 2 r10 The prog stores a kernel pointer in it and loads it again.
Also a local that holds the address of such a local.
else r12
> For certain things, like open coded iterators on kernel collections,
> there is no other choice.
Of the 16 rust sched that I hacked 4 walk dsq and 13 call
bpf_local_irq_save(). In testing they loaded and *run*.
But I agree that passing them around is fragile.
The hope is for typed arena to save the day.
Then rule 2 will go away.
> unsure how various assumptions around address contiguity get affected
> when stack reuse occurs [...]
A local never changes its stack, and it is contiguous. The compiler selects
the stack after all IR optimizations, for the full alloca and
its full life. r12 locals have a stack id of their own, and stack coloring merges
only objects with the same id. An iterator in one scope and a buffer in the
next are two allocas, one in each stack.
^ permalink raw reply [flat|nested] 144+ messages in thread* Re: [PATCH bpf 26/81] bpf: Support BPF_REG_ARENA_SP in the verifier
2026-10-27 18:45 ` Alexei Starovoitov
@ 2026-10-27 20:23 ` Kumar Kartikeya Dwivedi
2026-10-28 4:47 ` Alexei Starovoitov
0 siblings, 1 reply; 144+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-10-27 20:23 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: bpf, daniel, andrii, eddyz87
On Tue, 27 Oct 2026 at 19:45, Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Mon, Oct 26, 2026 at 5:18 PM Kumar Kartikeya Dwivedi <memxor@gmail.com> wrote:
> > Overall, I think it is less clear to me (or anyone looking at patches)
> > as to why you did things this way.
>
> point taken.
>
> I was wrong in my previous email. An iterator over kernel objects
> stays in the r10 stack, and it works today.
>
> > Are there real patterns that existing stack tracking simply cannot
> > salvage?
>
> Yes.
>
> Pattern 1: one insn, two kinds of memory
>
> sum(&local_array) r1 = fp-64 --+
> +--> r2 = *(u64 *)(r1 + 0)
> sum(&vec) r1 = arena --+
> one insn, one JIT form
>
> We discussed an idea to let llvm clone such function.
> I tried it, and it didn't work.
> Short summary:
>
> case why a copy does not help
> --------------------------------- ------------------------------------
> the pointer comes from memory no kind for the compiler, a scalar
> for the verifier
> an indirect call: &dyn Trait, fmt one function for the two kinds
> if c { &local } else { &*boxed } one pointer, two kinds, one function
> n pointer arguments up to 2^n copies
>
> and real case in <&PanicInfo as Display>::fmt().
There are other ways to address this, like ensuring r1 holds fully
formed pointer pointing into arena at the point of dereference. I
don't think we sufficiently explored alternatives to this problem,
which is much broader than just stack and arena mismatch, e.g. ringbuf
memory buffer and arena memory buffer as same argument to the same
function has similar problem. The other approach would be testing the
value and deciding the form of access based on that. There must be
some way to distinguish arena ptr from other kernel ptrs at runtime.
>
> Pattern 2: the address of a local goes to memory, a load gives a scalar
>
> That's typical rust heap.
>
So storing a stack pointer in the heap and then reloading and dereferencing it?
Do you have real examples like you showed for other cases?
Normal Rust code probably doesn't even allow that due to lifetime
issues, a Box<T> outlives the stack, but perhaps I'm missing some case
you hit.
We can have more complex cases, like an object going into the arena
stack but storing a kernel pointer into it, that the compiler
heuristics miss. That is the other direction that will not be covered.
> Pattern 3: LLVM removes a bounds check, here in core::fmt::num
>
> loop { remain /= 10000; offset -= 4; buf[offset] = ...; }
>
> and 'buf' is on stack.
> run-time clamp won't help, since real tracked spill/fills are in r10.
>
Well, since you're changing the compiler, we can always inject clamps
on the index.
The backend will know the size of the `buf` stack alloca, and can
clamp the offset into the right range.
Many assumptions have changed since we no longer worry about not
having to change LLVM.
> Pattern 4: fp in an argument
>
> ...
>
This is same as pattern 1, unless I'm mistaken.
> So an address-taken local must be where the heap is... in arena.
>
> [...]
>
> > It might require teaching the compiler not to move certain types on
> > the stack to r12 relative accesses, but it just feels ugly.
>
> it might feel fragile, but it worked for huge corpus of rust crates:
> https://github.com/llvm/llvm-project/pull/235577
>
> Your dsq iterator exapmle is 'alloca [48 x i8]' in the IR, so it works
> (I replied too soon yesteday).
>
> llvm selects the stack by use:
>
> rule 1 r10 The function that owns the local passes its address to
> a helper or a kfunc: iterator, irq flag, dynptr, map key.
> rule 2 r10 The prog stores a kernel pointer in it and loads it again.
> Also a local that holds the address of such a local.
> else r12
>
> > For certain things, like open coded iterators on kernel collections,
> > there is no other choice.
>
> Of the 16 rust sched that I hacked 4 walk dsq and 13 call
> bpf_local_irq_save(). In testing they loaded and *run*.
>
> But I agree that passing them around is fragile.
> The hope is for typed arena to save the day.
> Then rule 2 will go away.
>
I don't think typed arena will help here. It is just a slab of similar
types, it does not solve any of the problems we're discussing, esp.
with the stack. If you're assuming it will help here, and that it may
be necessary or useful for some part of this proposal, then there is a
misunderstanding of the problem it is supposed to solve. It will only
work for Box<T>.
(Unless you have some enhancement in mind you haven't described yet).
The other concern I did not voice before was mixing C BPF objects with
Rust ones, e.g. for supplying the allocator. I don't know how such
mixing will work in practice with all this.
> [...]
^ permalink raw reply [flat|nested] 144+ messages in thread* Re: [PATCH bpf 26/81] bpf: Support BPF_REG_ARENA_SP in the verifier
2026-10-27 20:23 ` Kumar Kartikeya Dwivedi
@ 2026-10-28 4:47 ` Alexei Starovoitov
2026-10-28 11:00 ` Kumar Kartikeya Dwivedi
0 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-28 4:47 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi; +Cc: bpf, daniel, andrii, eddyz87
On Tue Oct 27, 2026 at 8:23 PM UTC, Kumar Kartikeya Dwivedi wrote:
>
> There are other ways to address this, like ensuring r1 holds fully
> formed pointer pointing into arena at the point of dereference. I
> don't think we sufficiently explored alternatives to this problem,
> which is much broader than just stack and arena mismatch, e.g. ringbuf
> memory buffer and arena memory buffer as same argument to the same
> function has similar problem. The other approach would be testing the
> value and deciding the form of access based on that. There must be
> some way to distinguish arena ptr from other kernel ptrs at runtime.
agree. there is likely another option.
Please find one.
but I was working on this since March and what you see is the best
I could come up with.
Once you find it I will be more than happy to delete this approach.
Until then we have to move forward.
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 26/81] bpf: Support BPF_REG_ARENA_SP in the verifier
2026-10-28 4:47 ` Alexei Starovoitov
@ 2026-10-28 11:00 ` Kumar Kartikeya Dwivedi
0 siblings, 0 replies; 144+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2026-10-28 11:00 UTC (permalink / raw)
To: Alexei Starovoitov; +Cc: bpf, daniel, andrii, eddyz87
On Wed, 28 Oct 2026 at 05:47, Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Tue Oct 27, 2026 at 8:23 PM UTC, Kumar Kartikeya Dwivedi wrote:
> >
> > There are other ways to address this, like ensuring r1 holds fully
> > formed pointer pointing into arena at the point of dereference. I
> > don't think we sufficiently explored alternatives to this problem,
> > which is much broader than just stack and arena mismatch, e.g. ringbuf
> > memory buffer and arena memory buffer as same argument to the same
> > function has similar problem. The other approach would be testing the
> > value and deciding the form of access based on that. There must be
> > some way to distinguish arena ptr from other kernel ptrs at runtime.
>
> agree. there is likely another option.
> Please find one.
> but I was working on this since March and what you see is the best
> I could come up with.
> Once you find it I will be more than happy to delete this approach.
> Until then we have to move forward.
Sigh. This is not a good response. If you do not want to have a
discussion, what is the point of posting patches and asking for
feedback?
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 27/81] bpf, x86: Support BPF_REG_ARENA_SP
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (25 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 26/81] bpf: Support BPF_REG_ARENA_SP in the verifier Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 28/81] bpf, arm64: " Alexei Starovoitov
` (55 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
BPF r12 (BPF_REG_ARENA_SP) is the stack pointer of the stack in arena. It
needs an x86 register for the full run of the prog. R9 is the only x86
register that has no fixed function. Map r12 to R9.
R9 has other uses too. In a prog that uses r12 the JIT changes them:
R9 is also in a prog with r12
-------------------------------- -------------------------------------
clobbered by helpers and kfuncs store R9 in the frame before the call,
load it after the call
arg 6 of a BPF-to-BPF call arg 6 is on the stack, before arg 7
arg 6 of a kfunc load arg 6 from the stack into R9,
move rsp to arg 7 for the call
frame pointer of private stack the private stack is off
Only JITed code sees the arguments of a BPF-to-BPF call, because no
trampoline attaches to a subprog of a prog with r12. Thus the JIT can
select the location of arg 6.
Progs without r12 get the same JITed code. The cost in a prog with r12, in
a hot loop on x86-64 (Zen 5), in nsec:
call of a helper or a kfunc store and load of R9 0.012
function that has such a call push r9, add rsp, 8 < 0.01
BPF-to-BPF call with 6 or 7 args arg 6 is on the stack < 0.007
call of a kfunc with 10 args load of R9, moves of rsp 0.26
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/x86/net/bpf_jit_comp.c | 120 +++++++++++++++++++++++++++++++-----
1 file changed, 105 insertions(+), 15 deletions(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index a398c50b58ea..e417ac4ce749 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -155,8 +155,9 @@ static int bpf_size_to_x86_bytes(int bpf_size)
* register in load/store instructions, it always needs an
* extra byte of encoding and is callee saved.
*
- * x86-64 register R9 is not used by BPF programs, but can be used by BPF
- * trampoline. x86-64 register R10 is used for blinding (if enabled).
+ * x86-64 register R9 is the 6th argument, the frame pointer of the private
+ * stack, or BPF_REG_ARENA_SP. It can be used by BPF trampoline.
+ * x86-64 register R10 is used for blinding (if enabled).
*/
static const int reg2hex[] = {
[BPF_REG_0] = 0, /* RAX */
@@ -170,6 +171,7 @@ static const int reg2hex[] = {
[BPF_REG_8] = 6, /* R14 callee saved */
[BPF_REG_9] = 7, /* R15 callee saved */
[BPF_REG_FP] = 5, /* RBP readonly */
+ [BPF_REG_ARENA_SP] = 1, /* R9 */
[BPF_REG_AX] = 2, /* R10 temp register */
[AUX_REG] = 3, /* R11 temp register */
[X86_REG_R9] = 1, /* R9 register, 6th function argument */
@@ -187,6 +189,7 @@ static const int reg2pt_regs[] = {
[BPF_REG_7] = offsetof(struct pt_regs, r13),
[BPF_REG_8] = offsetof(struct pt_regs, r14),
[BPF_REG_9] = offsetof(struct pt_regs, r15),
+ [BPF_REG_ARENA_SP] = offsetof(struct pt_regs, r9),
[BPF_REG_AX] = offsetof(struct pt_regs, r10),
};
@@ -211,6 +214,7 @@ static bool is_ereg(u32 reg)
BIT(BPF_REG_9) |
BIT(X86_REG_R9) |
BIT(X86_REG_R12) |
+ BIT(BPF_REG_ARENA_SP) |
BIT(BPF_REG_AX));
}
@@ -2294,6 +2298,27 @@ static int emit_spectre_bhb_barrier(u8 **pprog, u8 *ip,
return 0;
}
+/*
+ * Helpers and kfuncs clobber R9, which is BPF_REG_ARENA_SP. It's set after
+ * bpf_arena_stack_get() and is not used after bpf_arena_stack_put().
+ */
+static bool call_clobbers_arena_sp(const struct bpf_insn *insn)
+{
+ return insn->code == (BPF_JMP | BPF_CALL) && insn->src_reg != BPF_PSEUDO_CALL &&
+ insn->imm != BPF_CALL_IMM(bpf_arena_stack_get) &&
+ insn->imm != BPF_CALL_IMM(bpf_arena_stack_put);
+}
+
+static bool calls_clobber_arena_sp(const struct bpf_prog *prog)
+{
+ int i;
+
+ for (i = 0; i < prog->len; i++)
+ if (call_clobbers_arena_sp(&prog->insnsi[i]))
+ return true;
+ return false;
+}
+
static const struct bpf_jit_arg_abi x86_arg_abi = {
.nr_arg_regs = 6,
.backfill_after_stack = true,
@@ -2501,16 +2526,23 @@ static u64 bpf_unwind_target(u64 ret_addr)
return e ? ret_addr + e->fixup : ret_addr;
}
-/* unwind_exit: fix the return address of this frame. r0 and r2 are returned. */
-static int emit_unwind_ret_addr(u8 **pprog, u8 *ip)
+/*
+ * unwind_exit: fix the return address of this frame. r0 and r2 are returned.
+ * So is BPF_REG_ARENA_SP, when it's used.
+ */
+static int emit_unwind_ret_addr(u8 **pprog, u8 *ip, bool arena_sp)
{
u8 *prog = *pprog, *start = prog;
EMIT2(0x50, 0x56); /* push rax; push rsi */
+ if (arena_sp)
+ push_r9(&prog);
EMIT4(0x48, 0x8B, 0x7D, 0x08); /* mov rdi, [rbp + 8] */
if (emit_rsb_call(&prog, bpf_unwind_target, ip + (prog - start)))
return -EINVAL;
EMIT4(0x48, 0x89, 0x45, 0x08); /* mov [rbp + 8], rax */
+ if (arena_sp)
+ pop_r9(&prog);
EMIT2(0x5E, 0x58); /* pop rsi; pop rax */
*pprog = prog;
return 0;
@@ -2535,6 +2567,11 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
u32 stack_depth;
int callee_saved_size;
s32 outgoing_arg_base;
+ /* the program uses BPF_REG_ARENA_SP, which is R9 */
+ bool arena_sp = bpf_prog->aux->arena_stack_size;
+ /* it has to be saved around calls of helpers and kfuncs */
+ bool arena_sp_save = arena_sp && calls_clobber_arena_sp(bpf_prog);
+ s32 arena_sp_off = 0;
int err;
stack_depth = bpf_prog->aux->stack_depth;
@@ -2559,12 +2596,21 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
* is disabled by the JIT when stack args exist, so R9 is always
* available.
*
+ * Unless the program uses BPF_REG_ARENA_SP. Then for BPF-to-BPF calls
+ * arg 6 is on the stack too: incoming args 6+ are at [rbp + 16], ...
+ * and outgoing args 6+ are at [rsp], ... No trampoline is attached to
+ * subprogs of such program. For a kfunc call arg 6 is moved to R9 and
+ * rsp is moved to arg 7.
+ * Helpers and kfuncs clobber R9. BPF_REG_ARENA_SP is stored in the frame
+ * before the call and is loaded from there after it.
+ *
* Stack layout (high to low):
* [rbp + 16 + ...] incoming stack args 7+ (from caller)
* [rbp + 8] return address
* [rbp] saved rbp
* [rbp - prog_stack] program stack
* [below] callee-saved regs
+ * [below] BPF_REG_ARENA_SP, when it's used and there are calls
* [below] outgoing args 7+ (= rsp)
*/
if (bpf_prog_uses_arena_insns(bpf_prog)) {
@@ -2623,14 +2669,22 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
* stack args exist, so tcc pushes need not be accounted for.
*/
outgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size);
+ if (arena_sp_save) {
+ /* make the slot */
+ push_r9(&prog);
+ outgoing_arg_base -= 8;
+ arena_sp_off = outgoing_arg_base;
+ }
/*
* Allocate outgoing stack arg area for args 7+ only.
* Arg 6 goes into r9 register, not on stack.
*/
outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0;
+ if (arena_sp)
+ outgoing_rsp = out_stack_arg_cnt * 8;
if (bpf_prog->aux->exception_boundary)
- bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp;
+ bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp + (arena_sp_save ? 8 : 0);
emit_sub_rsp(&prog, outgoing_rsp);
if (arena_vm_start)
@@ -3036,7 +3090,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
case BPF_ST | BPF_MEM | BPF_H:
case BPF_ST | BPF_MEM | BPF_W:
case BPF_ST | BPF_MEM | BPF_DW:
- if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
+ if (dst_reg == BPF_REG_PARAMS && insn->off == -8 && !arena_sp) {
/* Arg 6: store immediate in r9 register */
emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31,
imm32);
@@ -3050,7 +3104,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
* BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ...
*/
insn_off = outgoing_arg_base - outgoing_rsp -
- insn_off - 16;
+ insn_off - (arena_sp ? 8 : 16);
dst_reg = BPF_REG_FP;
}
if (!accesses_stack_only) {
@@ -3070,14 +3124,15 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
case BPF_STX | BPF_MEM | BPF_H:
case BPF_STX | BPF_MEM | BPF_W:
case BPF_STX | BPF_MEM | BPF_DW:
- if (dst_reg == BPF_REG_PARAMS && insn->off == -8) {
+ if (dst_reg == BPF_REG_PARAMS && insn->off == -8 && !arena_sp) {
/* Arg 6: store register value in r9 */
EMIT_mov(X86_REG_R9, src_reg);
break;
}
insn_off = insn->off;
if (dst_reg == BPF_REG_PARAMS) {
- insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16;
+ insn_off = outgoing_arg_base - outgoing_rsp - insn_off -
+ (arena_sp ? 8 : 16);
dst_reg = BPF_REG_FP;
}
if (!accesses_stack_only) {
@@ -3186,11 +3241,13 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
insn_off = insn->off;
if (src_reg == BPF_REG_PARAMS) {
- if (insn_off == 8) {
+ if (insn_off == 8 && !arena_sp) {
/* Incoming arg 6: read from r9 */
EMIT_mov(dst_reg, X86_REG_R9);
break;
}
+ if (arena_sp)
+ insn_off += 8;
src_reg = BPF_REG_FP;
/*
* Incoming args 7+: native_off == bpf_off
@@ -3428,6 +3485,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
}
if (!imm32)
return -EINVAL;
+ if (arena_sp && call_clobbers_arena_sp(insn)) {
+ u8 *start = prog;
+
+ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_ARENA_SP, arena_sp_off);
+ ip += prog - start;
+ }
if (src_reg == BPF_PSEUDO_KFUNC_CALL) {
fm = bpf_jit_find_kfunc_model(bpf_prog, insn);
if (!fm)
@@ -3436,8 +3499,26 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
if (err < 0)
return err;
ip += err;
- ip += emit_kfunc_arg_moves(fm, outgoing_arg_base -
- outgoing_rsp, &prog);
+ if (arena_sp && out_stack_arg_cnt) {
+ u8 *start = prog;
+
+ /*
+ * Args 6+ are at [rsp]. Put arg 6 where
+ * emit_kfunc_arg_moves() expects it.
+ */
+ emit_ldx(&prog, BPF_DW, X86_REG_R9, BPF_REG_FP,
+ outgoing_arg_base - outgoing_rsp);
+ ip += prog - start;
+ }
+ ip += emit_kfunc_arg_moves(fm, outgoing_arg_base - outgoing_rsp +
+ (arena_sp ? 8 : 0), &prog);
+ if (arena_sp && out_stack_arg_cnt) {
+ u8 *start = prog;
+
+ /* arg 7 is at [rsp] */
+ emit_add_rsp(&prog, 8);
+ ip += prog - start;
+ }
}
if (func == (u8 *)arch_bpf_timed_may_goto) {
u32 fp = priv_frame_ptr ? X86_REG_R9 : BPF_REG_FP;
@@ -3460,6 +3541,10 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
return -EINVAL;
if (priv_frame_ptr)
pop_r9(&prog);
+ if (fm && arena_sp && out_stack_arg_cnt)
+ emit_sub_rsp(&prog, 8);
+ if (arena_sp && call_clobbers_arena_sp(insn))
+ emit_ldx(&prog, BPF_DW, BPF_REG_ARENA_SP, BPF_REG_FP, arena_sp_off);
/*
* A kfunc returning more than 8 bytes hands the second
* half back in RDX (the native ABI's second return reg),
@@ -3786,7 +3871,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
case BPF_JMP | BPF_EXIT:
if (src_reg == BPF_UNWIND_EXIT && bpf_prog_can_unwind(bpf_prog) &&
- emit_unwind_ret_addr(&prog, ip))
+ emit_unwind_ret_addr(&prog, ip, arena_sp))
return -EINVAL;
if (seen_exit) {
jmp_offset = ctx->cleanup_addr - addrs[i] + moved;
@@ -3800,8 +3885,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog))
return -EINVAL;
}
- /* Deallocate outgoing args 7+ area. */
- emit_add_rsp(&prog, outgoing_rsp);
+ /* Deallocate outgoing args 7+ area and the slot of BPF_REG_ARENA_SP. */
+ emit_add_rsp(&prog, outgoing_rsp + (arena_sp_save ? 8 : 0));
if (bpf_prog->aux->exception_boundary) {
pop_callee_regs(&prog, all_callee_regs_used);
pop_r12(&prog);
@@ -5144,6 +5229,11 @@ bool bpf_jit_supports_private_stack(void)
return true;
}
+bool bpf_jit_supports_arena_stack(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_local_atomics(void)
{
return true;
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 28/81] bpf, arm64: Support BPF_REG_ARENA_SP
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (26 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 27/81] bpf, x86: Support BPF_REG_ARENA_SP Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 29/81] selftests/bpf: Add tests for BPF_REG_ARENA_SP Alexei Starovoitov
` (54 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
r12 (BPF_REG_ARENA_SP) is the stack pointer of the second stack of a prog.
This stack is in arena. The verifier replaces the insns that use r12 with
insns that the JIT already knows: a 32-bit add, addr_space_cast, and loads
and stores in arena. Thus the JIT only has to supply a register.
Map r12 to x23. The JIT does not use x23 for other data. x23 is callee
saved, thus helpers and kfuncs keep its value. It is not an argument
register, thus calls do not change.
The kernel calls the main prog and callbacks, and it does not know about
r12. The main prog, and a callback that uses r12, take a stack at entry and
set r12. Thus they save x23 at entry and restore it at exit. Other subprogs
get x23 from the caller.
Progs without r12 get the same code. The cost in progs with r12:
save and restore of x23 0 or 2 insns (stp, ldp). The JIT saves
registers in pairs.
load or store through r12 3 insns: add of the base of arena, mov of
the negative offset, ldr or str.
load or store through r10 1 insn, for reference.
Compile tested only.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/arm64/net/bpf_jit_comp.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 33632fb20997..58f30911bdab 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -61,6 +61,8 @@ static const int bpf2a64[] = {
[BPF_REG_9] = A64_R(22),
/* read-only frame pointer to access stack */
[BPF_REG_FP] = A64_R(25),
+ /* callee saved register for the pointer to the stack in arena */
+ [BPF_REG_ARENA_SP] = A64_R(23),
/* temporary registers for BPF JIT */
[TMP_REG_1] = A64_R(10),
[TMP_REG_2] = A64_R(11),
@@ -419,6 +421,11 @@ static void find_used_callee_regs(struct jit_ctx *ctx)
if (ctx->arena_vm_start)
ctx->used_callee_reg[i++] = bpf2a64[ARENA_VM_START];
+ /* subprogs get it from the caller and leave it as it was, unless the kernel calls them */
+ if (prog->aux->arena_stack_size &&
+ (!bpf_is_subprog(prog) || prog->aux->arena_stack_entry))
+ ctx->used_callee_reg[i++] = bpf2a64[BPF_REG_ARENA_SP];
+
ctx->nr_used_callee_reg = i;
}
@@ -2537,6 +2544,11 @@ bool bpf_jit_supports_private_stack(void)
return true;
}
+bool bpf_jit_supports_arena_stack(void)
+{
+ return true;
+}
+
bool bpf_jit_supports_kfunc_call(void)
{
return true;
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 29/81] selftests/bpf: Add tests for BPF_REG_ARENA_SP
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (27 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 28/81] bpf, arm64: " Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 30/81] bpf: Add bpf_arena_memcmp() and bpf_arena_memcpy() kfuncs Alexei Starovoitov
` (53 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
r12 (BPF_REG_ARENA_SP) is the stack pointer of the second stack of a prog.
This stack is in arena. Each run of a prog takes a stack at entry and gives
it back at exit. Runs that preempt or interrupt each other on one cpu must
not get the same stack.
arena_stack runs progs:
subtest what it does
------------------ ---------------------------------------------------
preemption 8 threads on one cpu run a prog that fills its
frames, spins, and then examines the frames.
nesting The same with preemption disabled. A prog in the
timer interrupt writes to its own frames.
timer A timer callback takes a stack of its own.
throw bpf_throw() does not give the stack back. After many
throws other progs continue to run correctly.
no_memory The arena is too small for the stack. The prog does
not run, returns 0, and its stderr has a message.
attach_to_subprog fentry to the main prog loads. fentry to a subprog
is rejected.
jump_to_first_insn A jump to the first insn does not take a stack again.
The first two subtests fail if all runs on a cpu get the same stack.
verifier_arena_stack has the tests of the verifier:
- Progs with the four insns that can use r12 load and return correct
values.
- All other insns with r12 are rejected. So are a frame of an incorrect
size, call chains with more than 32 Kbyte of frames, tail calls, and
progs without an arena.
- r12 and the frame stay correct after a helper, a subprog with or without
a frame, a callback and unwind_exit of a callee.
- On x86 r12 is in the register of the 6th argument. Thus a subprog with 7
arguments and a kfunc with 10 arguments are tested too.
Compilers do not emit r12. Thus the tests encode these insns with .8byte.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/arena_stack.c | 249 +++++++
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/arena_noarena.c | 14 +
.../testing/selftests/bpf/progs/arena_stack.c | 334 +++++++++
.../selftests/bpf/progs/arena_stack_fentry.c | 13 +
.../selftests/bpf/progs/arena_stack_loop.c | 52 ++
.../selftests/bpf/progs/arena_stack_nomem.c | 32 +
.../bpf/progs/verifier_arena_stack.c | 635 ++++++++++++++++++
8 files changed, 1331 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/arena_stack.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_stack.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_stack_fentry.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_stack_loop.c
create mode 100644 tools/testing/selftests/bpf/progs/arena_stack_nomem.c
create mode 100644 tools/testing/selftests/bpf/progs/verifier_arena_stack.c
diff --git a/tools/testing/selftests/bpf/prog_tests/arena_stack.c b/tools/testing/selftests/bpf/prog_tests/arena_stack.c
new file mode 100644
index 000000000000..aa567eb48353
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/arena_stack.c
@@ -0,0 +1,249 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#define _GNU_SOURCE
+#include <test_progs.h>
+#include <pthread.h>
+#include <sched.h>
+#include <linux/perf_event.h>
+#include <sys/syscall.h>
+#include "arena_stack.skel.h"
+#include "arena_stack_nomem.skel.h"
+#include "arena_stack_fentry.skel.h"
+#include "arena_stack_loop.skel.h"
+
+static int run_prog(int fd)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, opts);
+
+ if (bpf_prog_test_run_opts(fd, &opts))
+ return -1;
+ return opts.retval;
+}
+
+static int run_tc_prog(int fd)
+{
+ char pkt[64] = {};
+ LIBBPF_OPTS(bpf_test_run_opts, opts, .data_in = pkt, .data_size_in = sizeof(pkt));
+
+ if (bpf_prog_test_run_opts(fd, &opts))
+ return -1;
+ return opts.retval;
+}
+
+static int pin_to_cpu0(cpu_set_t *old)
+{
+ cpu_set_t set;
+
+ if (sched_getaffinity(0, sizeof(*old), old))
+ return -1;
+ CPU_ZERO(&set);
+ CPU_SET(0, &set);
+ return sched_setaffinity(0, sizeof(set), &set);
+}
+
+#define NR_THREADS 8
+#define NR_RUNS 20
+
+static int preempted_fd;
+
+static void *preempted_thread(void *arg)
+{
+ long bad = 0;
+ int i;
+
+ for (i = 0; i < NR_RUNS; i++)
+ bad += run_prog(preempted_fd) != 0;
+ return (void *)bad;
+}
+
+/* Programs on one cpu preempt each other. They don't exit in the order they started. */
+static void test_preemption(struct arena_stack *skel)
+{
+ pthread_t thread[NR_THREADS];
+ long bad = 0;
+ cpu_set_t old;
+ void *ret;
+ int i, n;
+
+ if (!ASSERT_OK(pin_to_cpu0(&old), "pin"))
+ return;
+ preempted_fd = bpf_program__fd(skel->progs.preempted);
+ /* threads inherit the affinity */
+ for (n = 0; n < NR_THREADS; n++)
+ if (!ASSERT_OK(pthread_create(&thread[n], NULL, preempted_thread, NULL), "thread"))
+ break;
+ for (i = 0; i < n; i++) {
+ pthread_join(thread[i], &ret);
+ bad += (long)ret;
+ }
+ ASSERT_EQ(bad, 0, "runs with a corrupted frame");
+ sched_setaffinity(0, sizeof(old), &old);
+}
+
+/* A program that runs with preemption disabled is interrupted by a program in the timer interrupt */
+static void test_nesting(struct arena_stack *skel)
+{
+ struct perf_event_attr attr = {
+ .size = sizeof(attr),
+ .type = PERF_TYPE_SOFTWARE,
+ .config = PERF_COUNT_SW_CPU_CLOCK,
+ .sample_period = 50000, /* nsec */
+ };
+ struct bpf_link *link = NULL;
+ int i, pfd, bad = 0;
+ cpu_set_t old;
+
+ if (!ASSERT_OK(pin_to_cpu0(&old), "pin"))
+ return;
+ pfd = syscall(__NR_perf_event_open, &attr, -1, 0, -1, 0);
+ if (!ASSERT_GE(pfd, 0, "perf_event_open"))
+ goto out;
+ link = bpf_program__attach_perf_event(skel->progs.in_interrupt, pfd);
+ if (!ASSERT_OK_PTR(link, "attach_perf_event")) {
+ close(pfd);
+ goto out;
+ }
+ for (i = 0; i < 200; i++)
+ bad += run_tc_prog(bpf_program__fd(skel->progs.interrupted)) != 0;
+ ASSERT_EQ(bad, 0, "runs with a corrupted frame");
+ /* the program was interrupted */
+ ASSERT_GT(skel->bss->interrupts, 10, "interrupts");
+ bpf_link__destroy(link);
+out:
+ sched_setaffinity(0, sizeof(old), &old);
+}
+
+/* bpf_throw() doesn't return the stack. Programs run when all stacks of the cpu are taken. */
+static void test_throw(struct arena_stack *skel)
+{
+ cpu_set_t old;
+ int i;
+
+ if (!ASSERT_OK(pin_to_cpu0(&old), "pin"))
+ return;
+ for (i = 0; i < 40; i++)
+ run_prog(bpf_program__fd(skel->progs.throws));
+ for (i = 0; i < 40; i++)
+ if (!ASSERT_EQ(run_prog(bpf_program__fd(skel->progs.prog_returns_7)), 7, "returns_7"))
+ break;
+ /* the same with preemption disabled */
+ for (i = 0; i < 200; i++)
+ run_tc_prog(bpf_program__fd(skel->progs.throws_nested));
+ for (i = 0; i < 40; i++)
+ if (!ASSERT_EQ(run_tc_prog(bpf_program__fd(skel->progs.returns_8_nested)), 8,
+ "returns_8_nested"))
+ break;
+ sched_setaffinity(0, sizeof(old), &old);
+}
+
+/* A callback that the kernel calls takes a stack of its own */
+static void test_timer(struct arena_stack *skel)
+{
+ int i, j;
+
+ for (i = 0; i < 100; i++) {
+ if (!ASSERT_OK(run_prog(bpf_program__fd(skel->progs.start_timer)), "start_timer"))
+ return;
+ for (j = 0; j < 1000 && skel->bss->timer_runs <= i; j++)
+ usleep(100);
+ }
+ ASSERT_EQ(skel->bss->timer_runs, 100, "runs of the callback");
+ ASSERT_EQ(skel->bss->timer_bad, 0, "runs with a wrong frame");
+}
+
+static void test_attach_to_subprog(struct arena_stack *skel)
+{
+ struct arena_stack_fentry *fentry;
+ int tgt_fd = bpf_program__fd(skel->progs.preempted);
+
+ /* the main program is fine */
+ fentry = arena_stack_fentry__open();
+ if (!ASSERT_OK_PTR(fentry, "open"))
+ return;
+ ASSERT_OK(bpf_program__set_attach_target(fentry->progs.on_entry, tgt_fd, "preempted"),
+ "set_attach_target");
+ ASSERT_OK(arena_stack_fentry__load(fentry), "load for the main program");
+ arena_stack_fentry__destroy(fentry);
+
+ fentry = arena_stack_fentry__open();
+ if (!ASSERT_OK_PTR(fentry, "open"))
+ return;
+ ASSERT_OK(bpf_program__set_attach_target(fentry->progs.on_entry, tgt_fd, "spin"),
+ "set_attach_target");
+ ASSERT_ERR(arena_stack_fentry__load(fentry), "load for a subprog");
+ arena_stack_fentry__destroy(fentry);
+}
+
+/* The insns that take the stack are in front of the first insn. A jump to it doesn't run them. */
+static void test_jump_to_first_insn(void)
+{
+ struct arena_stack_loop *skel;
+ cpu_set_t old;
+ int i;
+
+ skel = arena_stack_loop__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+ if (!ASSERT_OK(pin_to_cpu0(&old), "pin"))
+ goto out;
+ for (i = 0; i < 1000; i++) {
+ if (!ASSERT_EQ(run_tc_prog(bpf_program__fd(skel->progs.loop)), 7, "loop"))
+ break;
+ if (!ASSERT_EQ(run_tc_prog(bpf_program__fd(skel->progs.nop_loop)), 7, "nop_loop"))
+ break;
+ }
+ sched_setaffinity(0, sizeof(old), &old);
+out:
+ arena_stack_loop__destroy(skel);
+}
+
+static void test_no_memory(void)
+{
+ struct bpf_prog_info info = {};
+ __u32 info_len = sizeof(info);
+ struct arena_stack_nomem *skel;
+ char buf[256] = {};
+ int fd;
+
+ skel = arena_stack_nomem__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+ fd = bpf_program__fd(skel->progs.returns_1);
+ ASSERT_EQ(run_prog(fd), 0, "retval");
+ ASSERT_EQ(skel->bss->runs, 0, "runs");
+ ASSERT_OK(bpf_prog_get_info_by_fd(fd, &info, &info_len), "info");
+ ASSERT_EQ(info.recursion_misses, 1, "misses");
+ ASSERT_GT(bpf_prog_stream_read(fd, BPF_STREAM_STDERR, buf, sizeof(buf) - 1, NULL), 0,
+ "stream_read");
+ ASSERT_HAS_SUBSTR(buf, "No memory for the stack in arena", "stderr");
+ arena_stack_nomem__destroy(skel);
+}
+
+void test_arena_stack(void)
+{
+ struct arena_stack *skel;
+
+#if !defined(__x86_64__) && !defined(__aarch64__)
+ /* other JITs don't support r12 */
+ test__skip();
+ return;
+#endif
+ skel = arena_stack__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+ if (test__start_subtest("preemption"))
+ test_preemption(skel);
+ if (test__start_subtest("nesting"))
+ test_nesting(skel);
+ if (test__start_subtest("attach_to_subprog"))
+ test_attach_to_subprog(skel);
+ if (test__start_subtest("timer"))
+ test_timer(skel);
+ if (test__start_subtest("throw"))
+ test_throw(skel);
+ arena_stack__destroy(skel);
+ if (test__start_subtest("jump_to_first_insn"))
+ test_jump_to_first_insn();
+ if (test__start_subtest("no_memory"))
+ test_no_memory();
+}
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 643d67782619..089d8ea3f651 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -13,6 +13,7 @@
#include "verifier_and.skel.h"
#include "verifier_arena.skel.h"
#include "verifier_arena_cast.skel.h"
+#include "verifier_arena_stack.skel.h"
#include "verifier_arena_large.skel.h"
#include "verifier_arena_globals1.skel.h"
#include "verifier_arena_globals2.skel.h"
@@ -207,6 +208,7 @@ void test_verifier_align(void) { RUN(verifier_align); }
void test_verifier_and(void) { RUN(verifier_and); }
void test_verifier_arena(void) { RUN(verifier_arena); }
void test_verifier_arena_cast(void) { RUN(verifier_arena_cast); }
+void test_verifier_arena_stack(void) { RUN_TESTS(verifier_arena_stack); }
void test_verifier_arena_large(void) { RUN(verifier_arena_large); }
void test_verifier_arena_globals1(void) { RUN(verifier_arena_globals1); }
void test_verifier_arena_globals2(void) { RUN(verifier_arena_globals2); }
diff --git a/tools/testing/selftests/bpf/progs/arena_noarena.c b/tools/testing/selftests/bpf/progs/arena_noarena.c
index 1f46e5aa403b..8a7ce3e36703 100644
--- a/tools/testing/selftests/bpf/progs/arena_noarena.c
+++ b/tools/testing/selftests/bpf/progs/arena_noarena.c
@@ -21,4 +21,18 @@ int arena_arg_no_arena(void *ctx)
return 0;
}
+SEC("syscall")
+__arch_x86_64
+__arch_arm64
+__description("arena stack: no arena")
+__failure __msg("R12 can only be used in a program that has an associated arena")
+__naked void r12_no_arena(void)
+{
+ asm volatile (
+ ".8byte 0xc1bf;" /* r1 = r12 */
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_stack.c b/tools/testing/selftests/bpf/progs/arena_stack.c
new file mode 100644
index 000000000000..8c93e4cea71c
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_stack.c
@@ -0,0 +1,334 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+extern void bpf_throw(__u64 cookie) __ksym;
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 4096);
+} arena SEC(".maps");
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+ bpf_throw(0);
+}
+
+/* Compilers don't know r12 yet */
+#define R12 12
+#define ADJ(n) BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, R12, 0, 0, n)
+#define LDX(dst, off) BPF_RAW_INSN(BPF_LDX | BPF_MEM | BPF_DW, dst, R12, off, 0)
+#define STX(off, src) BPF_RAW_INSN(BPF_STX | BPF_MEM | BPF_DW, R12, src, off, 0)
+#define ST(off, imm) BPF_RAW_INSN(BPF_ST | BPF_MEM | BPF_DW, R12, 0, off, imm)
+
+/*
+ * Fill the frame, spin long enough to be preempted, check the frame.
+ * Returns the number of words that have changed.
+ */
+static __naked __noinline __used long spin(long seed)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[stx32];"
+ "r1 += 1;"
+ ".8byte %[stx24];"
+ "r1 += 1;"
+ ".8byte %[stx16];"
+ "r1 += 1;"
+ ".8byte %[stx8];"
+ "r2 = 3000000;"
+"1:"
+ "r2 += -1;"
+ "if r2 != 0 goto 1b;"
+ "r0 = 0;"
+ ".8byte %[ldx8];"
+ "if r2 == r1 goto +1;"
+ "r0 += 1;"
+ "r1 -= 1;"
+ ".8byte %[ldx16];"
+ "if r2 == r1 goto +1;"
+ "r0 += 1;"
+ "r1 -= 1;"
+ ".8byte %[ldx24];"
+ "if r2 == r1 goto +1;"
+ "r0 += 1;"
+ "r1 -= 1;"
+ ".8byte %[ldx32];"
+ "if r2 == r1 goto +1;"
+ "r0 += 1;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(32)),
+ __imm_insn(stx32, STX(-32, BPF_REG_1)),
+ __imm_insn(stx24, STX(-24, BPF_REG_1)),
+ __imm_insn(stx16, STX(-16, BPF_REG_1)),
+ __imm_insn(stx8, STX(-8, BPF_REG_1)),
+ __imm_insn(ldx32, LDX(BPF_REG_2, -32)),
+ __imm_insn(ldx24, LDX(BPF_REG_2, -24)),
+ __imm_insn(ldx16, LDX(BPF_REG_2, -16)),
+ __imm_insn(ldx8, LDX(BPF_REG_2, -8))
+ : __clobber_all);
+}
+
+/* the same, one frame deeper */
+static __naked __noinline __used long spin_deeper(long seed)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[stx];"
+ "r6 = r1;"
+ "r1 += 100;"
+ "call spin;"
+ ".8byte %[ldx];"
+ "if r1 == r6 goto +1;"
+ "r0 += 1;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(128)),
+ __imm_insn(stx, STX(-64, BPF_REG_1)),
+ __imm_insn(ldx, LDX(BPF_REG_1, -64))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__naked int preempted(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call %[bpf_get_prandom_u32];"
+ "r6 = r0;"
+ ".8byte %[stx];"
+ "r1 = r6;"
+ "if r6 & 1 goto 1f;"
+ "call spin;"
+ "goto 2f;"
+"1:"
+ "call spin_deeper;"
+"2:"
+ ".8byte %[ldx];"
+ "if r1 == r6 goto +1;"
+ "r0 += 1;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx, STX(-16, BPF_REG_6)),
+ __imm_insn(ldx, LDX(BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+static long thrower(void);
+
+/* The same with preemption disabled. Programs in interrupts nest. */
+SEC("tc")
+__naked int interrupted(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call %[bpf_get_prandom_u32];"
+ "r6 = r0;"
+ ".8byte %[stx];"
+ "r1 = r6;"
+ "call spin_deeper;"
+ ".8byte %[ldx];"
+ "if r1 == r6 goto +1;"
+ "r0 += 1;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx, STX(-16, BPF_REG_6)),
+ __imm_insn(ldx, LDX(BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+long interrupts;
+
+static __naked __noinline __used long scribble(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st128];"
+ ".8byte %[st64];"
+ ".8byte %[st8];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(128)),
+ __imm_insn(st128, ST(-128, -1)),
+ __imm_insn(st64, ST(-64, -1)),
+ __imm_insn(st8, ST(-8, -1))
+ : __clobber_all);
+}
+
+SEC("perf_event")
+__naked int in_interrupt(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st32];"
+ ".8byte %[st8];"
+ "call scribble;"
+ "r1 = %[interrupts] ll;"
+ "r2 = 1;"
+ "lock *(u64 *)(r1 + 0) += r2;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(interrupts),
+ __imm_insn(adj, ADJ(32)),
+ __imm_insn(st32, ST(-32, -1)),
+ __imm_insn(st8, ST(-8, -1))
+ : __clobber_all);
+}
+
+SEC("tc")
+__naked int throws_nested(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call thrower;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+SEC("tc")
+__naked int returns_8_nested(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(-16, 8)),
+ __imm_insn(ldx, LDX(BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long thrower(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = 5;"
+ "call %[bpf_throw];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_throw),
+ __imm_insn(adj, ADJ(64))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__naked int throws(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call thrower;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long returns_7(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(-16, 7)),
+ __imm_insn(ldx, LDX(BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__naked int prog_returns_7(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(-16, 7)),
+ __imm_insn(ldx, LDX(BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+struct elem {
+ struct bpf_timer t;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, struct elem);
+} timers SEC(".maps");
+
+long timer_runs, timer_bad;
+
+/* The kernel calls it. Fill the frame, call a function that has a frame, check the frame. */
+static __naked __noinline __used int timer_cb(void *map, int *key, struct elem *val)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st32];"
+ ".8byte %[st8];"
+ "call returns_7;"
+ ".8byte %[ldx32];"
+ ".8byte %[ldx8];"
+ "r1 += r2;"
+ "r1 += r0;"
+ "r2 = %[timer_runs] ll;"
+ "r3 = 1;"
+ "lock *(u64 *)(r2 + 0) += r3;"
+ "if r1 == 49 goto 1f;"
+ "r2 = %[timer_bad] ll;"
+ "lock *(u64 *)(r2 + 0) += r3;"
+"1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(timer_runs),
+ __imm_addr(timer_bad),
+ __imm_insn(adj, ADJ(32)),
+ __imm_insn(st32, ST(-32, 11)),
+ __imm_insn(st8, ST(-8, 31)),
+ __imm_insn(ldx32, LDX(BPF_REG_1, -32)),
+ __imm_insn(ldx8, LDX(BPF_REG_2, -8))
+ : __clobber_all);
+}
+
+SEC("syscall")
+int start_timer(void *ctx)
+{
+ struct elem *e;
+ int key = 0;
+
+ /* the program has to have an arena */
+ asm volatile ("" :: "r"(&arena));
+ e = bpf_map_lookup_elem(&timers, &key);
+ if (!e)
+ return 2;
+ bpf_timer_init(&e->t, &timers, 1 /* CLOCK_MONOTONIC */);
+ if (bpf_timer_set_callback(&e->t, timer_cb))
+ return 3;
+ return bpf_timer_start(&e->t, 0, 0) ? 4 : 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_stack_fentry.c b/tools/testing/selftests/bpf/progs/arena_stack_fentry.c
new file mode 100644
index 000000000000..4dd870c51166
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_stack_fentry.c
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+SEC("fentry")
+int BPF_PROG(on_entry)
+{
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_stack_loop.c b/tools/testing/selftests/bpf/progs/arena_stack_loop.c
new file mode 100644
index 000000000000..da88baf64cee
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_stack_loop.c
@@ -0,0 +1,52 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+/* A stack that is not returned is noticed soon */
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 8);
+} arena SEC(".maps");
+
+static __naked __noinline __used long returns_7(void)
+{
+ asm volatile (
+ ".8byte %[adj];" /* r12 += 16 */
+ ".8byte %[st];" /* *(u64 *)(r12 - 8) = 7 */
+ ".8byte %[ldx];" /* r0 = *(u64 *)(r12 - 8) */
+ "exit;"
+ :
+ : __imm_insn(adj, BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, 12, 0, 0, 16)),
+ __imm_insn(st, BPF_RAW_INSN(BPF_ST | BPF_MEM | BPF_DW, 12, 0, -8, 7)),
+ __imm_insn(ldx, BPF_RAW_INSN(BPF_LDX | BPF_MEM | BPF_DW, 0, 12, -8, 0))
+ : __clobber_all);
+}
+
+/* r1 is ctx, then 0: one jump to the first insn of the program */
+#define LOOP \
+"1:" \
+ "if r1 == 0 goto 2f;" \
+ "r1 = 0;" \
+ "goto 1b;" \
+"2:" \
+ "call returns_7;" \
+ "exit;"
+
+SEC("tc")
+__naked int loop(void)
+{
+ asm volatile (LOOP ::: __clobber_all);
+}
+
+/* the same after the nop is removed */
+SEC("tc")
+__naked int nop_loop(void)
+{
+ asm volatile ("goto +0;" LOOP ::: __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/arena_stack_nomem.c b/tools/testing/selftests/bpf/progs/arena_stack_nomem.c
new file mode 100644
index 000000000000..dddeae3ea7ce
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/arena_stack_nomem.c
@@ -0,0 +1,32 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+/* one page, the program needs two */
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 1);
+} arena SEC(".maps");
+
+int runs;
+
+SEC("syscall")
+__naked int returns_1(void)
+{
+ asm volatile (
+ ".8byte %[adj];" /* r12 += 32768 */
+ "r1 = %[runs] ll;"
+ "r0 = 1;"
+ "*(u32 *)(r1 + 0) = r0;"
+ "exit;"
+ :
+ : __imm_addr(runs),
+ __imm_insn(adj, BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, 12, 0, 0, 32768))
+ : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_stack.c b/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
new file mode 100644
index 000000000000..79a4552f1a93
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
@@ -0,0 +1,635 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 64);
+} arena SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+ __uint(max_entries, 1);
+ __uint(key_size, sizeof(int));
+ __uint(value_size, sizeof(int));
+} jmp_table SEC(".maps");
+
+/* Compilers don't know r12 yet */
+#define R12 12
+/* r12 += n */
+#define ADJ(n) BPF_RAW_INSN(BPF_ALU64 | BPF_ADD | BPF_K, R12, 0, 0, n)
+/* dst = r12 */
+#define MOV(dst) BPF_RAW_INSN(BPF_ALU64 | BPF_MOV | BPF_X, dst, R12, 0, 0)
+/* dst = *(sz *)(r12 + off) */
+#define LDX(sz, dst, off) BPF_RAW_INSN(BPF_LDX | BPF_MEM | sz, dst, R12, off, 0)
+#define LDSX(sz, dst, off) BPF_RAW_INSN(BPF_LDX | BPF_MEMSX | sz, dst, R12, off, 0)
+/* *(sz *)(r12 + off) = src */
+#define STX(sz, off, src) BPF_RAW_INSN(BPF_STX | BPF_MEM | sz, R12, src, off, 0)
+/* *(sz *)(r12 + off) = imm */
+#define ST(sz, off, imm) BPF_RAW_INSN(BPF_ST | BPF_MEM | sz, R12, 0, off, imm)
+/* dst = addr_space_cast(dst, 0, 1) */
+#define CAST_KERN(dst) BPF_RAW_INSN(BPF_ALU64 | BPF_MOV | BPF_X, dst, dst, BPF_ADDR_SPACE_CAST, 1)
+
+SEC("syscall")
+__description("arena stack: load and store")
+__success __retval(12)
+__xlated("r0 = *(u64 *)(r12 -16)")
+__xlated("r2 = *(u64 *)(r12 -8)")
+__naked void load_store(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 16) = 7 */
+ "r1 = 5;"
+ ".8byte %[stx];" /* *(u64 *)(r12 - 8) = r1 */
+ ".8byte %[ldx0];" /* r0 = *(u64 *)(r12 - 16) */
+ ".8byte %[ldx1];" /* r2 = *(u64 *)(r12 - 8) */
+ "r0 += r2;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 7)),
+ __imm_insn(stx, STX(BPF_DW, -8, BPF_REG_1)),
+ __imm_insn(ldx0, LDX(BPF_DW, BPF_REG_0, -16)),
+ __imm_insn(ldx1, LDX(BPF_DW, BPF_REG_2, -8))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: every size, sign extension")
+__success __retval(0)
+__naked void sizes(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = 0x8899aabb ll;"
+ ".8byte %[stx];" /* *(u64 *)(r12 - 16) = r1 */
+ "r0 = 1;"
+ ".8byte %[ldx_b];"
+ "if r2 != 0xbb goto 1f;"
+ "r0 = 2;"
+ ".8byte %[ldx_h];"
+ "if r2 != 0xaabb goto 1f;"
+ "r0 = 3;"
+ ".8byte %[ldx_w];"
+ "r3 = 0x8899aabb ll;"
+ "if r2 != r3 goto 1f;"
+ "r0 = 4;"
+ ".8byte %[ldsx_w];"
+ "if r2 s>= 0 goto 1f;"
+ "r0 = 5;"
+ ".8byte %[st_b];" /* *(u8 *)(r12 - 15) = 0 */
+ ".8byte %[ldx_h];"
+ "if r2 != 0xbb goto 1f;"
+ "r0 = 0;"
+"1:"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx, STX(BPF_DW, -16, BPF_REG_1)),
+ __imm_insn(ldx_b, LDX(BPF_B, BPF_REG_2, -16)),
+ __imm_insn(ldx_h, LDX(BPF_H, BPF_REG_2, -16)),
+ __imm_insn(ldx_w, LDX(BPF_W, BPF_REG_2, -16)),
+ __imm_insn(ldsx_w, LDSX(BPF_W, BPF_REG_2, -16)),
+ __imm_insn(st_b, ST(BPF_B, -15, 0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: rX = r12 is the address of the frame")
+__success __retval(9)
+__naked void address(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[mov];" /* r1 = r12 */
+ "r1 += -8;"
+ ".8byte %[cast];"
+ "r2 = 9;"
+ "*(u64 *)(r1 + 0) = r2;"
+ ".8byte %[ldx];" /* r0 = *(u64 *)(r12 - 8) */
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(mov, MOV(BPF_REG_1)),
+ __imm_insn(cast, CAST_KERN(BPF_REG_1)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -8))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long callee_with_frame(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st0];"
+ ".8byte %[st1];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st0, ST(BPF_DW, -16, 10)),
+ __imm_insn(st1, ST(BPF_DW, -8, 10)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: the frame of the callee is above")
+__success __retval(21)
+__naked void call(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 16) = 1 */
+ "call callee_with_frame;"
+ "r6 = r0;"
+ "call callee_with_frame;"
+ "r0 += r6;"
+ ".8byte %[ldx];" /* r1 = *(u64 *)(r12 - 16) */
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 1)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+/* no frame of its own, r12 goes through */
+static __naked __noinline __used long callee_in_the_middle(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "call callee_with_frame;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: a function without a frame in the middle")
+__success __retval(11)
+__naked void call_through(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ "call callee_in_the_middle;"
+ ".8byte %[ldx];"
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 1)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: helpers don't change r12")
+__success __retval(3)
+__naked void helper(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ "call %[bpf_get_prandom_u32];"
+ "call %[bpf_ktime_get_ns];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm(bpf_ktime_get_ns),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 3)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long store_through(long addr)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ ".8byte %[cast];"
+ "r2 = 5;"
+ "*(u64 *)(r1 + 0) = r2;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(64)),
+ __imm_insn(st, ST(BPF_DW, -64, 77)),
+ __imm_insn(cast, CAST_KERN(BPF_REG_1))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: the callee writes to the frame of the caller")
+__success __retval(5)
+__naked void pass_address(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[mov];"
+ "r1 += -16;"
+ "call store_through;"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(mov, MOV(BPF_REG_1)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long unwinds(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ "r0 = 0;"
+ ".8byte %[unwind_exit];"
+ :
+ : __imm_insn(adj, ADJ(32)),
+ __imm_insn(st, ST(BPF_DW, -32, -1)),
+ __imm_insn(unwind_exit, BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0))
+ : __clobber_all);
+}
+
+/* its pad only carries on */
+static __naked __noinline __used long unwinds_through(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call unwinds;"
+ ".8byte %[unwind_goto];"
+ "r0 = 0;"
+ "exit;"
+ ".8byte %[unwind_exit];"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(unwind_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, BPF_UNWIND_GOTO, 2, 0)),
+ __imm_insn(unwind_exit, BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: r12 in the landing pad")
+__success __retval(5)
+__naked void unwind(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 16) = 5 */
+ "call unwinds_through;"
+ ".8byte %[unwind_goto];"
+ "r0 = 1;"
+ "exit;"
+ ".8byte %[ldx];" /* r0 = *(u64 *)(r12 - 16) */
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 5)),
+ __imm_insn(unwind_goto, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, BPF_UNWIND_GOTO, 2, 0)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -16))
+ : __clobber_all);
+}
+
+#ifdef __BPF_FEATURE_STACK_ARGUMENT
+
+extern __u64 bpf_kfunc_call_stack_arg(__u64 a, __u64 b, __u64 c, __u64 d, __u64 e, __u64 f,
+ __u64 g, __u64 h, __u64 i, __u64 j) __ksym;
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+ bpf_kfunc_call_stack_arg(0, 0, 0, 0, 0, 0, 0, 0, 0, 0);
+}
+
+static __naked __noinline __used
+long callee_7args(long a, long b, long c, long d, long e, long f, long g)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r0 = *(u64 *)(r11 + 8);"
+ ".8byte %[stx0];" /* *(u64 *)(r12 - 16) = f */
+ "r0 = *(u64 *)(r11 + 16);"
+ ".8byte %[stx8];" /* *(u64 *)(r12 - 8) = g */
+ "r6 = r5;"
+ "call %[bpf_get_prandom_u32];"
+ ".8byte %[ldx0];"
+ ".8byte %[ldx8];"
+ "r0 *= 10;"
+ "r0 += r1;"
+ "r0 *= 10;"
+ "r0 += r6;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx0, STX(BPF_DW, -16, BPF_REG_0)),
+ __imm_insn(stx8, STX(BPF_DW, -8, BPF_REG_0)),
+ __imm_insn(ldx0, LDX(BPF_DW, BPF_REG_0, -16)),
+ __imm_insn(ldx8, LDX(BPF_DW, BPF_REG_1, -8))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: a function with 7 arguments")
+__success __retval(1675)
+__naked void stack_args(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 16) = 1000 */
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 16) = 7;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call callee_7args;"
+ ".8byte %[ldx];"
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 1000)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena stack: a kfunc with 10 arguments")
+__success __retval(1055)
+__naked void kfunc_stack_args(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 16) = 1000 */
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 40) = 10;"
+ "*(u64 *)(r11 - 32) = 9;"
+ "*(u64 *)(r11 - 24) = 8;"
+ "*(u64 *)(r11 - 16) = 7;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call %[bpf_kfunc_call_stack_arg];"
+ ".8byte %[ldx];"
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm(bpf_kfunc_call_stack_arg),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -16, 1000)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+#endif /* __BPF_FEATURE_STACK_ARGUMENT */
+
+SEC("syscall")
+__description("arena stack: what is loaded is a number")
+__failure __msg("R1 invalid mem access 'scalar'")
+__naked void loaded_is_number(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r1 = r10;"
+ ".8byte %[stx];"
+ ".8byte %[ldx];"
+ "r0 = *(u64 *)(r1 - 8);"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx, STX(BPF_DW, -16, BPF_REG_1)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -16))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: rX = r12 is a number")
+__failure __msg("R1 invalid mem access 'scalar'")
+__naked void address_is_number(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[mov];"
+ "r0 = *(u64 *)(r1 + 0);"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(mov, MOV(BPF_REG_1))
+ : __clobber_all);
+}
+
+#define BAD_INSN(name, desc, msg, first, second) \
+SEC("syscall") \
+__description("arena stack: " desc) \
+__failure __msg(msg) \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ ".8byte %[first_insn];" \
+ ".8byte %[second_insn];" \
+ "r0 = 0;" \
+ "exit;" \
+ : \
+ : __imm_insn(first_insn, first), \
+ __imm_insn(second_insn, second) \
+ : __clobber_all); \
+}
+
+#define NOP BPF_MOV64_IMM(BPF_REG_1, 0)
+
+BAD_INSN(adj_not_first, "r12 += 16 is not the first insn",
+ "R12 can be changed by the first insn of a function only", NOP, ADJ(16))
+BAD_INSN(adj_twice, "r12 += 16 twice",
+ "R12 can be changed by the first insn of a function only", ADJ(16), ADJ(16))
+BAD_INSN(adj_negative, "r12 += -16", "invalid size of the frame", ADJ(-16), NOP)
+BAD_INSN(adj_zero, "r12 += 0", "invalid size of the frame", ADJ(0), NOP)
+BAD_INSN(adj_unaligned, "r12 += 8", "invalid size of the frame", ADJ(8), NOP)
+BAD_INSN(adj_large, "r12 += 32784", "invalid size of the frame", ADJ(32784), NOP)
+BAD_INSN(mov_to, "r12 = r1", "R12 is invalid", NOP, BPF_MOV64_REG(R12, BPF_REG_1))
+BAD_INSN(mov_imm, "r12 = 0", "R12 is invalid", NOP, BPF_MOV64_IMM(R12, 0))
+BAD_INSN(add_reg, "r12 += r1", "R12 is invalid", NOP, BPF_ALU64_REG(BPF_ADD, R12, BPF_REG_1))
+BAD_INSN(adj32, "w12 += 16", "R12 is invalid", BPF_ALU32_IMM(BPF_ADD, R12, 16), NOP)
+BAD_INSN(sub_imm, "r12 -= 16", "R12 is invalid", BPF_ALU64_IMM(BPF_SUB, R12, 16), NOP)
+BAD_INSN(add_from, "r1 += r12", "R12 is invalid", NOP, BPF_ALU64_REG(BPF_ADD, BPF_REG_1, R12))
+BAD_INSN(mov32_from, "w1 = w12", "R12 is invalid", NOP, BPF_MOV32_REG(BPF_REG_1, R12))
+BAD_INSN(load_to, "r12 = *(u64 *)(r10 - 8)", "R12 is invalid", NOP,
+ BPF_LDX_MEM(BPF_DW, R12, BPF_REG_10, -8))
+BAD_INSN(store_from, "*(u64 *)(r10 - 8) = r12", "R12 is invalid", NOP,
+ BPF_STX_MEM(BPF_DW, BPF_REG_10, R12, -8))
+BAD_INSN(store_itself, "*(u64 *)(r12 - 8) = r12", "R12 is invalid", NOP,
+ BPF_STX_MEM(BPF_DW, R12, R12, -8))
+BAD_INSN(atomic, "lock *(u64 *)(r12 - 8) += r1", "R12 is invalid", NOP,
+ BPF_ATOMIC_OP(BPF_DW, BPF_ADD, R12, BPF_REG_1, -8))
+BAD_INSN(jump, "if r12 == 0 goto", "R12 is invalid", NOP, BPF_JMP_IMM(BPF_JEQ, R12, 0, 0))
+BAD_INSN(r13, "r1 = r13", "R13 is invalid", NOP, BPF_MOV64_REG(BPF_REG_1, 13))
+BAD_INSN(local_atomic, "atomic that is local to the cpu", "BPF_ATOMIC uses invalid atomic opcode",
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_ATOMIC_OP(BPF_DW, BPF_ADD | BPF_FETCH | 0x1000, BPF_REG_10, BPF_REG_1, -8))
+
+SEC("syscall")
+__description("arena stack: jump to r12 += 16")
+__failure __msg("jump to the insn that changes R12")
+__naked void jump_to_adj(void)
+{
+ asm volatile (
+"1:"
+ ".8byte %[adj];"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 != 0 goto 1b;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+/* adds the index and what the context points to, through its frame */
+static __naked __noinline __used long loop_cb(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[stx16];" /* *(u64 *)(r12 - 16) = r1 */
+ "r4 = *(u64 *)(r2 + 0);"
+ ".8byte %[ldx16];" /* r5 = *(u64 *)(r12 - 16) */
+ "r4 += r5;"
+ "*(u64 *)(r2 + 0) = r4;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(stx16, STX(BPF_DW, -16, BPF_REG_1)),
+ __imm_insn(ldx16, LDX(BPF_DW, BPF_REG_5, -16))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: r12 in a callback")
+__success __retval(145)
+__naked void callback(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 8) = 100 */
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = 10;"
+ "r2 = loop_cb ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ /* 0 + 1 + ... + 9 and what is in the frame of the main program */
+ "r0 = *(u64 *)(r10 - 8);"
+ ".8byte %[ldx];" /* r1 = *(u64 *)(r12 - 8) */
+ "r0 += r1;"
+ "exit;"
+ :
+ : __imm(bpf_loop),
+ __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -8, 100)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -8))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("arena stack: tail call")
+__failure __msg("tail_calls are not allowed in programs that use R12")
+__naked void tail_call(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r2 = %[jmp_table] ll;"
+ "r3 = 0;"
+ "call %[bpf_tail_call];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_tail_call),
+ __imm_addr(jmp_table),
+ __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long frame_16(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long frame_16k_calls_16(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call frame_16;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16384))
+ : __clobber_all);
+}
+
+static __naked __noinline __used long frame_16k(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];"
+ ".8byte %[ldx];"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16384)),
+ __imm_insn(st, ST(BPF_DW, -16384, 4)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_0, -16384))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: 32K")
+__success __retval(4)
+__naked void depth_max(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call frame_16k;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16384))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: more than 32K")
+__failure __msg("combined size of the frames in the stack in arena is 32784. Too large")
+__naked void depth_too_large(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ "call frame_16k_calls_16;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16384))
+ : __clobber_all);
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 30/81] bpf: Add bpf_arena_memcmp() and bpf_arena_memcpy() kfuncs
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (28 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 29/81] selftests/bpf: Add tests for BPF_REG_ARENA_SP Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 31/81] bpf: Add bpf_alloc() and bpf_free() kfuncs Alexei Starovoitov
` (52 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
LLVM replaces a comparison or a copy of memory with a call to memcmp(),
memcpy() or memmove(). BPF has none of these functions. Rust progs keep all
their memory in arena. Add two kfuncs that take arena pointers:
int bpf_arena_memcmp(const void *a__arena, const void *b__arena,
u64 size);
u64 bpf_arena_memcpy(void *dst__arena, const void *src__arena,
u64 size);
An arena pointer can point to a page that is not allocated. When the kernel
accesses such a page, the fault handler maps a scratch page there, and the
access continues. Thus a plain memcmp() or memmove() would compare or copy
the scratch page, and the prog would not see the error.
Walk the page tables before the access, and stop at the first page that is
not allocated. Report this page to the BPF_STDERR stream of the prog, the
same as an access of the prog to this page. Then the fault handler is
necessary only for a page that is freed after the walk.
The kfuncs can run in NMI. Thus one call processes 64 Kbyte at most, and
the prog copies more in a loop. The verifier rejects a call if size is
always larger.
kfunc return value
------------------ ---------------------------------------------------
bpf_arena_memcmp() 0 or the difference of two bytes, as memcmp() does.
-E2BIG if it stopped early and found no difference.
bpf_arena_memcpy() The number of bytes that it copied.
The areas can overlap.
The cost of a call on x86-64 (Zen 5) with hot caches:
copy of 1 to 64 bytes in one page 12.0 nsec
compare of 1 to 64 bytes in one page up to 12.3 nsec
the two walks of the page tables, included 5.2 to 5.5 nsec
the same copy with inline loads and stores less than 2 nsec
64 Kbyte 0.58 to 2.0 usec
Inline code is faster up to 1024 bytes. Thus it is better to keep small
copies inline.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
Documentation/bpf/kfuncs.rst | 10 ++++
include/linux/bpf.h | 2 +
kernel/bpf/arena.c | 91 ++++++++++++++++++++++++++++++++++++
kernel/bpf/verifier.c | 14 ++++++
4 files changed, 117 insertions(+)
diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index 73578a4c3b1f..47d3bed73b83 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -965,3 +965,13 @@ the verifier. bpf_cgroup_ancestor() can be used as follows:
BPF provides a set of kfuncs that can be used to query, allocate, mutate, and
destroy struct cpumask * objects. Please refer to :ref:`cpumasks-header-label`
for more details.
+
+4.4 Arena kfuncs
+----------------
+
+Compilers turn copies and comparisons of memory into calls to memcpy(),
+memmove() and memcmp(). Programs that keep all of their memory in arena,
+like the ones written in Rust, call the following kfuncs instead:
+
+.. kernel-doc:: kernel/bpf/arena.c
+ :identifiers: bpf_arena_memcmp bpf_arena_memcpy
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 4ca06a217055..d62dc3f7f148 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -698,6 +698,8 @@ int bpf_arena_stack_init(struct bpf_arena *arena);
void __percpu *bpf_arena_stack_percpu(struct bpf_arena *arena);
u64 bpf_arena_stack_get(struct bpf_prog_aux *aux, u64 busy);
void bpf_arena_stack_put(struct bpf_prog_aux *aux, u32 base);
+/* bpf_arena_memcmp() and bpf_arena_memcpy() work on that many bytes per call at most */
+#define BPF_ARENA_MEM_MAX SZ_64K
u64 bpf_arena_map_kern_vm_start(struct bpf_map *map);
struct bpf_map *bpf_prog_arena(struct bpf_prog *prog);
int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size);
diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
index 8b99b3b6fa02..c6c13dfa1102 100644
--- a/kernel/bpf/arena.c
+++ b/kernel/bpf/arena.c
@@ -47,6 +47,8 @@
#define KERN_VM_SZ (SZ_4G + GUARD_SZ)
static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable);
+static void __bpf_prog_report_arena_violation(struct bpf_prog *prog, bool write,
+ unsigned long addr, unsigned long fault_ip);
struct bpf_arena {
struct bpf_map map;
@@ -1296,12 +1298,101 @@ __bpf_kfunc int bpf_arena_reserve_pages(void *p__map, void *ptr__ign, u32 page_c
return arena_reserve_pages(arena, (long)ptr__ign, page_cnt);
}
+
+/*
+ * The number of bytes at @ptr, @size at most, that are in pages of arena
+ * that are allocated. @ptr is what JIT made of __arena argument.
+ */
+static u64 arena_span(struct bpf_arena *arena, const void *ptr, u64 size)
+{
+ unsigned long start = (unsigned long)ptr, end;
+ struct page *page;
+
+ size = min(size, SZ_4G - (start - bpf_arena_get_kern_vm_start(arena)));
+ for (end = start; end < start + size; end = (end | ~PAGE_MASK) + 1) {
+ page = vmalloc_to_page((void *)end);
+ if (!page || page == arena->scratch_page)
+ break;
+ }
+ return min(size, end - start);
+}
+
+/*
+ * @size bytes at @a and at @b were asked for. Returns what there is,
+ * BPF_ARENA_MEM_MAX at most, and reports the page that is missing.
+ */
+static u64 arena_span2(struct bpf_prog_aux *aux, const void *a, const void *b, u64 size,
+ bool write)
+{
+ const void *bad;
+ u64 span;
+
+ size = min_t(u64, size, BPF_ARENA_MEM_MAX);
+ span = min(arena_span(aux->arena, a, size), arena_span(aux->arena, b, size));
+ if (span < size) {
+ bad = arena_span(aux->arena, a, size) == span ? a : b;
+ __bpf_prog_report_arena_violation(aux->prog, write && bad == a,
+ (unsigned long)bad + span -
+ bpf_arena_get_kern_vm_start(aux->arena), 0);
+ }
+ return span;
+}
+
+/**
+ * bpf_arena_memcmp() - Compare two areas of arena
+ * @a__arena: One area
+ * @b__arena: Another area
+ * @size: The number of bytes to compare, BPF_ARENA_MEM_MAX at most
+ * @aux: Implicit argument, set by the verifier
+ *
+ * The comparison stops at the first page of either area that is not
+ * allocated. It's reported to the stream of the program as an access to
+ * that page.
+ *
+ * Return: 0 when the areas are equal. The difference between the first two
+ * bytes that differ, like memcmp() does, when they are not. -E2BIG when
+ * the areas are equal as far as they were compared and that is less than
+ * @size.
+ */
+__bpf_kfunc int bpf_arena_memcmp(const void *a__arena, const void *b__arena, u64 size,
+ struct bpf_prog_aux *aux)
+{
+ u64 span = arena_span2(aux, a__arena, b__arena, size, false);
+ int ret = memcmp(a__arena, b__arena, span);
+
+ return ret || span == size ? ret : -E2BIG;
+}
+
+/**
+ * bpf_arena_memcpy() - Copy one area of arena to another
+ * @dst__arena: Where to copy to
+ * @src__arena: Where to copy from
+ * @size: The number of bytes to copy, BPF_ARENA_MEM_MAX at most
+ * @aux: Implicit argument, set by the verifier
+ *
+ * The areas may overlap. The copy stops at the first page of either area
+ * that is not allocated. It's reported to the stream of the program as
+ * an access to that page.
+ *
+ * Return: The number of bytes copied. The program copies more than
+ * BPF_ARENA_MEM_MAX in a loop.
+ */
+__bpf_kfunc u64 bpf_arena_memcpy(void *dst__arena, const void *src__arena, u64 size,
+ struct bpf_prog_aux *aux)
+{
+ u64 span = arena_span2(aux, dst__arena, src__arena, size, true);
+
+ memmove(dst__arena, src__arena, span);
+ return span;
+}
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(arena_kfuncs)
BTF_ID_FLAGS(func, bpf_arena_alloc_pages, KF_ARENA_RET | KF_ARENA_ARG2 | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_arena_free_pages, KF_ARENA_ARG2 | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_arena_reserve_pages, KF_ARENA_ARG2 | KF_SPINLOCK_SAFE)
+BTF_ID_FLAGS(func, bpf_arena_memcmp, KF_IMPLICIT_ARGS)
+BTF_ID_FLAGS(func, bpf_arena_memcpy, KF_IMPLICIT_ARGS)
BTF_KFUNCS_END(arena_kfuncs)
static const struct btf_kfunc_id_set common_kfunc_set = {
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 92b563a9b158..fb80bc4d2d01 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -13719,6 +13719,8 @@ enum special_kfunc_type {
KF_bpf_call_rcu_tasks_trace,
KF_bpf_arena_alloc_pages,
KF_bpf_arena_free_pages,
+ KF_bpf_arena_memcmp,
+ KF_bpf_arena_memcpy,
KF_bpf_session_is_return,
};
@@ -13812,9 +13814,13 @@ BTF_ID(func, bpf_call_rcu_tasks_trace)
#if defined(CONFIG_MMU) && defined(CONFIG_64BIT)
BTF_ID(func, bpf_arena_alloc_pages)
BTF_ID(func, bpf_arena_free_pages)
+BTF_ID(func, bpf_arena_memcmp)
+BTF_ID(func, bpf_arena_memcpy)
#else
BTF_ID_UNUSED
BTF_ID_UNUSED
+BTF_ID_UNUSED
+BTF_ID_UNUSED
#endif
#ifdef CONFIG_BPF_EVENTS
BTF_ID(func, bpf_session_is_return)
@@ -15551,6 +15557,14 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
if (err < 0)
return err;
+ if ((meta.func_id == special_kfunc_list[KF_bpf_arena_memcmp] ||
+ meta.func_id == special_kfunc_list[KF_bpf_arena_memcpy]) &&
+ reg_umin(®s[BPF_REG_3]) > BPF_ARENA_MEM_MAX) {
+ verbose(env, "R3 is more than %u bytes that %s() takes per call\n",
+ BPF_ARENA_MEM_MAX, func_name);
+ return -EINVAL;
+ }
+
err = mark_raw_stack(env, &meta, insn_idx);
if (err)
return err;
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 31/81] bpf: Add bpf_alloc() and bpf_free() kfuncs
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (29 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 30/81] bpf: Add bpf_arena_memcmp() and bpf_arena_memcpy() kfuncs Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 32/81] bpf: Count stack arguments of a static subprog from its code Alexei Starovoitov
` (51 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
This patch is a placeholder to start the discussion about the allocator for
Rust. It is not for merging as is.
Rust needs a global allocator for Box, Vec and BTreeMap:
fn bpf_alloc(size: u64, flags: u64) -> *mut u8;
fn bpf_free(ptr: *mut u8);
This is the simplest implementation on top of the pages of arena. Each
allocation gets full pages. A header of 16 bytes before the memory holds
the number of pages. The prog can write to the header. Then bpf_free()
frees other pages of the same arena, and nothing else.
The problems of this implementation:
- A small allocation uses a full page.
- bpf_free() only queues the pages, and a worker frees them later. Thus a
loop of bpf_alloc() and bpf_free() can use all of the arena.
- Each bpf_free() causes a TLB flush on all cpus.
The cost of an allocation of 16 bytes on x86-64 (Zen 5), in nsec:
alloc free
bpf_alloc() and bpf_free() 250 780
page_frag allocator written in BPF 3 7 to 8
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/arena.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)
diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c
index c6c13dfa1102..d6c0d816849c 100644
--- a/kernel/bpf/arena.c
+++ b/kernel/bpf/arena.c
@@ -1385,6 +1385,62 @@ __bpf_kfunc u64 bpf_arena_memcpy(void *dst__arena, const void *src__arena, u64 s
memmove(dst__arena, src__arena, span);
return span;
}
+
+/* bpf_alloc() keeps the number of pages in front of the memory it returns */
+#define ARENA_ALLOC_HDR 16
+
+/**
+ * bpf_alloc() - Allocate memory in arena of the program
+ * @size: The number of bytes
+ * @flags: Must be 0
+ * @aux: Implicit argument, set by the verifier
+ *
+ * The memory is zeroed and aligned to 16 bytes. Every allocation takes
+ * whole pages.
+ *
+ * Return: The address of the memory in arena or NULL.
+ */
+__bpf_kfunc void *bpf_alloc(u64 size, u64 flags, struct bpf_prog_aux *aux)
+{
+ struct bpf_arena *arena = aux->arena;
+ u64 page_cnt;
+ long uaddr;
+
+ if (!arena || flags || !size || size > SZ_4G)
+ return NULL;
+
+ page_cnt = DIV_ROUND_UP(size + ARENA_ALLOC_HDR, PAGE_SIZE);
+ uaddr = arena_alloc_pages(arena, 0, page_cnt, NUMA_NO_NODE, false);
+ if (!uaddr)
+ return NULL;
+
+ WRITE_ONCE(*(u64 *)(long)(bpf_arena_get_kern_vm_start(arena) + (u32)uaddr), page_cnt);
+ return (void *)(uaddr + ARENA_ALLOC_HDR);
+}
+
+/**
+ * bpf_free() - Free memory allocated by bpf_alloc()
+ * @ptr__arena__nullable: The address that bpf_alloc() returned or NULL
+ * @aux: Implicit argument, set by the verifier
+ *
+ * The program can write to the header that bpf_alloc() left. Then pages
+ * of arena that the header tells about are freed, nothing else.
+ */
+__bpf_kfunc void bpf_free(void *ptr__arena__nullable, struct bpf_prog_aux *aux)
+{
+ u64 *hdr;
+ u64 off;
+
+ if (!ptr__arena__nullable)
+ return;
+
+ hdr = ptr__arena__nullable - ARENA_ALLOC_HDR;
+ off = (u64)(long)hdr - bpf_arena_get_kern_vm_start(aux->arena);
+ if (off >= SZ_4G || (off & ~PAGE_MASK))
+ return;
+
+ arena_free_pages(aux->arena, off, (u32)READ_ONCE(*hdr), false);
+}
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(arena_kfuncs)
@@ -1393,6 +1449,8 @@ BTF_ID_FLAGS(func, bpf_arena_free_pages, KF_ARENA_ARG2 | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_arena_reserve_pages, KF_ARENA_ARG2 | KF_SPINLOCK_SAFE)
BTF_ID_FLAGS(func, bpf_arena_memcmp, KF_IMPLICIT_ARGS)
BTF_ID_FLAGS(func, bpf_arena_memcpy, KF_IMPLICIT_ARGS)
+BTF_ID_FLAGS(func, bpf_alloc, KF_IMPLICIT_ARGS)
+BTF_ID_FLAGS(func, bpf_free, KF_IMPLICIT_ARGS)
BTF_KFUNCS_END(arena_kfuncs)
static const struct btf_kfunc_id_set common_kfunc_set = {
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 32/81] bpf: Count stack arguments of a static subprog from its code
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (30 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 31/81] bpf: Add bpf_alloc() and bpf_free() kfuncs Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:58 ` [PATCH bpf 33/81] selftests/bpf: Test static subprogs that don't take what BTF says Alexei Starovoitov
` (50 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A subprog gets its first 5 arguments in r1-r5. The caller stores the 6th
and subsequent arguments through r11, and the subprog loads them through
r11. The verifier takes the number of these stack arguments from the BTF
prototype of the subprog.
But BTF describes the source. It does not describe what the compiler passes
to a static function. rustc returns Option<i32> through a hidden pointer,
and it passes Option<i32> in two registers:
fn pick_idle_cpu(p: Task, prev_cpu: i32, this_cpu: Option<i32>,
wake_flags: u64) -> Option<i32>
BTF code
pointer to the result - 1
p 1 1
prev_cpu 1 1
this_cpu 1 2
wake_flags 1 1
total 4 6
The verifier rejects the load of the 6th argument,
'r3 = *(u64 *)(r11 + 8)':
invalid read from stack arg off 8 depth 0
For a static subprog use the code. The number of its stack arguments is the
highest slot that it loads through r11.
This is safe, because the verifier verifies a static subprog with the state
of its caller. A load through r11 gives the value that the caller stored,
as r1-r5 give the values that the caller set. At each call and callx the
verifier checks that the caller stored all the slots that the subprog
loads.
Global subprogs do not change. The verifier verifies them without a caller,
thus BTF is the only source.
What else changes:
- A caller gets space only for the slots that its callees load. Thus the
verifier can reject a caller that stores an argument that no callee
loads.
- Helpers and kfuncs pass arguments to callbacks in registers only. Reject
a callback that loads through r11.
- A trampoline knows only the arguments that BTF shows, and it would lose
the others. Refuse fentry and fexit to a static subprog that has stack
arguments when its code and its BTF do not agree.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 2 +
include/linux/bpf_verifier.h | 9 +++++
kernel/bpf/btf.c | 3 +-
kernel/bpf/verifier.c | 71 ++++++++++++++++++++++++++----------
4 files changed, 65 insertions(+), 20 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index d62dc3f7f148..3e4c317e0c7a 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1785,6 +1785,8 @@ struct bpf_func_info_aux {
bool called[BPF_FUNC_CTX_CNT];
bool verified[BPF_FUNC_CTX_CNT];
bool called_with_pkt_ptrs;
+ /* The code of a static function loads stack arguments */
+ bool stack_args;
};
enum bpf_jit_poke_reason {
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 2c19c3aea1d5..806282c39c57 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -980,6 +980,13 @@ struct bpf_subprog_info {
bool uses_arena_sp: 1;
/* a callback that takes a stack in arena, like the main program does */
bool arena_stack_entry: 1;
+ /*
+ * Static subprog: the number of arguments on the stack is what the code
+ * loads. BTF describes the source. The compiler may pass something else:
+ * a hidden pointer to the result, a pair in two registers, no dead arguments.
+ */
+ bool stack_args_from_code: 1;
+ u8 code_stack_args:4;
u8 arg_slot_cnt:4;
enum priv_stack_mode priv_stack_mode;
@@ -990,6 +997,8 @@ struct bpf_subprog_info {
static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub)
{
+ if (sub->stack_args_from_code)
+ return sub->code_stack_args;
if (sub->arg_slot_cnt > MAX_BPF_FUNC_REG_ARGS)
return sub->arg_slot_cnt - MAX_BPF_FUNC_REG_ARGS;
return 0;
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 7ed1ff71a089..274ef78b6e1c 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -8751,7 +8751,8 @@ static int btf_check_arg_slots(struct bpf_verifier_log *log, const char *tname,
tname, slot_cnt);
return -EFAULT;
}
- sub->stack_arg_cnt = slot_cnt - MAX_BPF_FUNC_REG_ARGS;
+ if (!sub->stack_args_from_code)
+ sub->stack_arg_cnt = slot_cnt - MAX_BPF_FUNC_REG_ARGS;
return 0;
}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index fb80bc4d2d01..0f2bae1c2305 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11362,7 +11362,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
const struct btf_type *func, *func_proto;
struct bpf_call_arg_meta meta;
struct bpf_func_proto *fn;
- int ret, err;
+ int ret;
u32 i;
memset(&meta, 0, sizeof(meta));
@@ -11371,16 +11371,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
meta.func_name = bpf_subprog_name(env, subprog);
ret = btf_prepare_func_args(env, subprog);
- if (ret) {
- if (bpf_in_stack_arg_cnt(sub) > 0) {
- err = check_outgoing_stack_args(env, caller, sub->arg_slot_cnt,
- bpf_subprog_name(env, subprog),
- NULL, NULL);
- if (err)
- return err;
- }
+ if (ret)
return ret;
- }
+ /* BTF doesn't describe what the code does */
+ if (sub->stack_args_from_code &&
+ max_t(int, sub->arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS) !=
+ MAX_BPF_FUNC_REG_ARGS + bpf_in_stack_arg_cnt(sub))
+ return -EINVAL;
func = btf_type_by_id(btf, env->prog->aux->func_info[subprog].type_id);
func_proto = btf_type_by_id(btf, func->type);
@@ -11446,6 +11443,12 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
int err;
caller = state->frame[state->curframe];
+ /* helpers and kfuncs pass arguments to callbacks in registers */
+ if (bpf_in_stack_arg_cnt(&env->subprog_info[subprog])) {
+ verbose(env, "callback %s() cannot have stack arguments\n",
+ bpf_subprog_name(env, subprog));
+ return -EINVAL;
+ }
err = btf_check_subprog_call(env, subprog);
if (err == -EFAULT)
return err;
@@ -11546,6 +11549,11 @@ static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
if (stack_arg_cnt > caller_info->stack_arg_cnt)
caller_info->stack_arg_cnt = stack_arg_cnt;
+ err = check_outgoing_stack_args(env, caller, MAX_BPF_FUNC_REG_ARGS + callee_incoming,
+ bpf_subprog_name(env, subprog), NULL, NULL);
+ if (err)
+ return err;
+
/*
* For regular function entry setup new frame and continue
* from that frame.
@@ -15204,12 +15212,31 @@ static bool try_btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
/* Prepare argument metadata for liveness; the main program is handled separately. */
static void prepare_subprog_args(struct bpf_verifier_env *env)
{
- int i;
+ struct bpf_subprog_info *sub;
+ struct bpf_insn *insn;
+ int i, j, cnt;
for (i = 1; i < env->subprog_cnt; i++) {
+ sub = &env->subprog_info[i];
if (!try_btf_prepare_func_args(env, i) && bpf_subprog_is_global(env, i))
/* Keep all argument registers live if the prototype is unavailable. */
- env->subprog_info[i].arg_slot_cnt = MAX_BPF_FUNC_REG_ARGS;
+ sub->arg_slot_cnt = MAX_BPF_FUNC_REG_ARGS;
+ if (bpf_subprog_is_global(env, i))
+ continue;
+ cnt = 0;
+ for (j = sub->start; j < (sub + 1)->start; j++) {
+ insn = &env->prog->insnsi[j];
+ if (is_stack_arg_ldx(insn))
+ cnt = max(cnt, insn->off / BPF_REG_SIZE);
+ if (bpf_is_ldimm64(insn))
+ j++;
+ }
+ sub->stack_args_from_code = true;
+ sub->code_stack_args = min(cnt, MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS);
+ /* incoming + max outgoing. Calls add the latter. */
+ sub->stack_arg_cnt = sub->code_stack_args;
+ if (env->prog->aux->func_info_aux)
+ env->prog->aux->func_info_aux[i].stack_args = sub->code_stack_args;
}
}
@@ -22630,18 +22657,21 @@ static int flatten_bpf_func_model(struct bpf_verifier_log *log, const char *tnam
}
/*
- * The verifier lets a static function use stack arguments counted from its
- * BTF, even if that BTF turns out unreliable. A trampoline on the function is
- * then given only MAX_BPF_FUNC_REG_ARGS u64 arguments and would lose the
- * stack arguments, so such a function cannot be attached to.
+ * The verifier lets a static function use the stack arguments that its code
+ * loads, whatever its BTF says. When BTF is unreliable a trampoline on
+ * the function is given only MAX_BPF_FUNC_REG_ARGS u64 arguments and would
+ * lose the stack arguments, so such a function cannot be attached to.
*/
static int check_unreliable_func_args(struct bpf_verifier_log *log, struct btf *btf,
- const struct btf_type *func_proto, const char *tname)
+ const struct btf_type *func_proto, const char *tname,
+ bool stack_args)
{
struct btf_func_model m;
u32 i, nslots = 0;
- if (btf_distill_func_proto(NULL, btf, func_proto, tname, &m)) {
+ if (stack_args) {
+ nslots = MAX_BPF_FUNC_REG_ARGS + 1;
+ } else if (btf_distill_func_proto(NULL, btf, func_proto, tname, &m)) {
/* at least a slot for each argument */
nslots = btf_type_vlen(func_proto);
} else {
@@ -23012,7 +23042,10 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
if (tgt_prog && conservative) {
if (subprog > 0) {
- ret = check_unreliable_func_args(log, btf, t, tname);
+ struct bpf_func_info_aux *fa = tgt_prog->aux->func_info_aux;
+
+ ret = check_unreliable_func_args(log, btf, t, tname,
+ fa[subprog].stack_args);
if (ret)
return ret;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 32/81] bpf: Count stack arguments of a static subprog from its code
2026-10-26 2:58 ` [PATCH bpf 32/81] bpf: Count stack arguments of a static subprog from its code Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 4412 bytes --]
> bpf: Count stack arguments of a static subprog from its code
TL;DR: Counting stack args from r11 loads may let programs use them on JITs
lacking support (s390, powerpc, loongarch, mips), possibly emitting
absolute-address stores and loads; a bpf_jit_supports_stack_args() check seems
missing.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index fb80bc4d2d01..0f2bae1c2305 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -15204,12 +15212,31 @@ static bool try_btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
> /* Prepare argument metadata for liveness; the main program is handled separately. */
> static void prepare_subprog_args(struct bpf_verifier_env *env)
> {
> - int i;
> + struct bpf_subprog_info *sub;
> + struct bpf_insn *insn;
> + int i, j, cnt;
>
> for (i = 1; i < env->subprog_cnt; i++) {
> + sub = &env->subprog_info[i];
> if (!try_btf_prepare_func_args(env, i) && bpf_subprog_is_global(env, i))
> /* Keep all argument registers live if the prototype is unavailable. */
> - env->subprog_info[i].arg_slot_cnt = MAX_BPF_FUNC_REG_ARGS;
> + sub->arg_slot_cnt = MAX_BPF_FUNC_REG_ARGS;
> + if (bpf_subprog_is_global(env, i))
> + continue;
> + cnt = 0;
> + for (j = sub->start; j < (sub + 1)->start; j++) {
> + insn = &env->prog->insnsi[j];
> + if (is_stack_arg_ldx(insn))
> + cnt = max(cnt, insn->off / BPF_REG_SIZE);
> + if (bpf_is_ldimm64(insn))
> + j++;
> + }
> + sub->stack_args_from_code = true;
> + sub->code_stack_args = min(cnt, MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS);
Can this let a program use r11 stack arguments on a JIT that does not
support them?
Before this patch, a static subprog got a non-zero incoming stack argument
count only through btf_prepare_func_args() -> btf_check_arg_slots(), and that
function rejects the case when the JIT lacks support:
kernel/bpf/btf.c:btf_check_arg_slots() {
...
if (!bpf_jit_supports_stack_args()) {
bpf_log(log, "JIT does not support function %s() with %d argument slots\n",
tname, slot_cnt);
return -EFAULT;
}
...
}
Here code_stack_args comes from the r11 loads in the instructions, for every
non-global subprog. That includes a subprog whose BTF has five or fewer
argument slots (the rustc case in the commit message), and a program with
no func_info at all, since bpf_subprog_is_global() needs func_info_aux.
Nothing on this path calls bpf_jit_supports_stack_args(). Besides
btf_check_arg_slots(), the only other caller is in the kfunc call path.
check_stack_arg_read(), check_stack_arg_write(), check_static_func_call()
and bpf_fixup_call_args() do not check it.
For example:
main: *(u64 *)(r11 - 8) = r1; call f; exit
f: r0 = *(u64 *)(r11 + 8); exit
This looks like it now passes check_static_func_call() ->
check_outgoing_stack_args(), check_stack_arg_read() (incoming count is 1),
and the max_out_stack_arg_cnt check in bpf_fixup_call_args() (outgoing count
is 1). It then reaches bpf_jit_subprogs() on architectures where
bpf_jit_supports_stack_args() is the __weak version that returns false.
Those JITs know nothing about BPF_REG_PARAMS (MAX_BPF_REG). For example,
reg2hex[] in arch/s390/net/bpf_jit_comp.c has no [BPF_REG_PARAMS] entry, so
the r11 store and load are emitted using hardware register 0. On s390,
powerpc, loongarch and mips a base register of 0 means an absolute address
equal to the displacement, so the JIT would emit a store to address -8 and a
load from address 8.
The "stack args are not supported in non-JITed programs" fallback in
bpf_fixup_call_args() only runs when the JIT fails to compile the program,
and these JITs would compile the r11 instructions without complaint.
Should prepare_subprog_args() (or check_stack_arg_read() and
check_stack_arg_write()) reject code_stack_args > 0 when
!bpf_jit_supports_stack_args()? The later patches in this series don't
appear to add such a check.
> + /* incoming + max outgoing. Calls add the latter. */
> + sub->stack_arg_cnt = sub->code_stack_args;
> + if (env->prog->aux->func_info_aux)
> + env->prog->aux->func_info_aux[i].stack_args = sub->code_stack_args;
> }
> }
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 33/81] selftests/bpf: Test static subprogs that don't take what BTF says
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (31 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 32/81] bpf: Count stack arguments of a static subprog from its code Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 34/81] bpf: Let scalar evolution see loads from read-only maps Alexei Starovoitov
` (49 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The compiler can pass more or fewer arguments to a static subprog than its
BTF prototype has parameters. See
"bpf: Count stack arguments of a static subprog from its code".
test the static subprog result
------------------------------- ---------------------------- ----------
more_than_prototype 4 in BTF, loads the 6th returns 11
more_than_prototype_not_written the same, 6th is not stored reject
less_than_prototype 7 in BTF, reads r1 only returns 1
callback_takes_6 a callback, loads the 6th reject
tgt_prog_takes_more 4 in BTF, loads the 6th, reject
fexit attaches to it
more_than_prototype also examines the x86 code of the subprog. The subprog
passes no stack arguments to other functions. Thus the JIT must not move
rsp in it.
The verifier now rejects the test
"stack_arg: read with not-initialized caller write" at the call. Update the
message that it expects.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/tracing_struct.c | 15 ++-
.../bpf/progs/tracing_struct_int128_tgt.c | 26 ++++-
.../bpf/progs/tracing_struct_int128_tracer.c | 7 ++
.../selftests/bpf/progs/verifier_stack_arg.c | 105 ++++++++++++++++++
.../bpf/progs/verifier_stack_arg_order.c | 2 +-
5 files changed, 147 insertions(+), 8 deletions(-)
diff --git a/tools/testing/selftests/bpf/prog_tests/tracing_struct.c b/tools/testing/selftests/bpf/prog_tests/tracing_struct.c
index de358f335d98..628b63b406ad 100644
--- a/tools/testing/selftests/bpf/prog_tests/tracing_struct.c
+++ b/tools/testing/selftests/bpf/prog_tests/tracing_struct.c
@@ -253,13 +253,13 @@ static void test_int128_tgt_prog(void)
* A static function whose BTF is unreliable gets a trampoline that only knows
* of the register arguments, so one taking stack arguments must be refused.
*/
-static void test_tgt_prog_unreliable(void)
+static void test_tgt_prog_unreliable(bool takes_more)
{
#if defined(__x86_64__) || defined(__aarch64__)
struct tracing_struct_int128_tracer *tracer = NULL;
struct tracing_struct_int128_tgt *tgt;
+ struct bpf_program *prog, *entry;
char log_buf[4096] = {};
- struct bpf_program *prog;
int err, fd;
tgt = tracing_struct_int128_tgt__open_and_load();
@@ -277,9 +277,10 @@ static void test_tgt_prog_unreliable(void)
bpf_object__for_each_program(prog, tracer->obj)
bpf_program__set_autoload(prog, false);
- bpf_program__set_autoload(tracer->progs.unreliable_entry, true);
- bpf_program__set_log_buf(tracer->progs.unreliable_entry, log_buf, sizeof(log_buf));
- err = bpf_program__set_attach_target(tracer->progs.unreliable_entry, fd,
+ entry = takes_more ? tracer->progs.takes_more_exit : tracer->progs.unreliable_entry;
+ bpf_program__set_autoload(entry, true);
+ bpf_program__set_log_buf(entry, log_buf, sizeof(log_buf));
+ err = bpf_program__set_attach_target(entry, fd, takes_more ? "callee_takes_more" :
"callee_unreliable");
if (!ASSERT_OK(err, "set_attach_target"))
goto out;
@@ -389,7 +390,9 @@ void test_tracing_struct(void)
if (test__start_subtest("int128_tgt_prog"))
test_int128_tgt_prog();
if (test__start_subtest("tgt_prog_unreliable"))
- test_tgt_prog_unreliable();
+ test_tgt_prog_unreliable(false);
+ if (test__start_subtest("tgt_prog_takes_more"))
+ test_tgt_prog_unreliable(true);
if (test__start_subtest("stack_arg_tailcall"))
test_stack_arg_tailcall();
if (test__start_subtest("union_args"))
diff --git a/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c b/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c
index 2561422c25dd..35c9a0c9b5cf 100644
--- a/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c
+++ b/tools/testing/selftests/bpf/progs/tracing_struct_int128_tgt.c
@@ -8,7 +8,7 @@ struct int128_arg {
};
__u64 base;
-__u64 ret_odd, ret_stack, ret_split, ret_unreliable;
+__u64 ret_odd, ret_stack, ret_split, ret_unreliable, ret_takes_more;
#if defined(__clang__)
@@ -68,6 +68,29 @@ static __noinline __u64 callee_unreliable(__u64 *p, __u64 b, __u64 c, __u64 d,
return *p + b * 2 + c * 3 + d * 4 + e * 5 + f * 6;
}
+/* BTF says 4. The code takes 6, the last one on the stack. */
+static __attribute__((naked)) __noinline __attribute__((used))
+__u64 callee_takes_more(__u64 a, __u64 b, __u64 c, __u64 d)
+{
+ asm volatile (
+ "r0 = *(u64 *)(r11 + 8);"
+ "r0 += r5;"
+ "exit;");
+}
+
+static __attribute__((naked)) __noinline __u64 call_takes_more(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call callee_takes_more;"
+ "exit;");
+}
+
#else
const volatile bool has_stack_arg = false;
@@ -93,6 +116,7 @@ int run(void *ctx)
ret_unreliable = callee_unreliable(&x, base + 2, base + 3,
base + 4, base + 5, base + 6);
}
+ ret_takes_more = call_takes_more();
#endif
#endif
return 0;
diff --git a/tools/testing/selftests/bpf/progs/tracing_struct_int128_tracer.c b/tools/testing/selftests/bpf/progs/tracing_struct_int128_tracer.c
index fc81c2eea4a8..a6d57e6b4798 100644
--- a/tools/testing/selftests/bpf/progs/tracing_struct_int128_tracer.c
+++ b/tools/testing/selftests/bpf/progs/tracing_struct_int128_tracer.c
@@ -74,4 +74,11 @@ int unreliable_entry(unsigned long long *ctx)
return 0;
}
+/* The same, but only the code tells that there is a stack argument. */
+SEC("?fexit/callee_takes_more")
+int takes_more_exit(unsigned long long *ctx)
+{
+ return 0;
+}
+
char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
index b9f799f6e405..5062c5bab5f0 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
@@ -433,6 +433,111 @@ __naked void stack_arg_sequential_calls(void)
);
}
+/*
+ * The prototype is what the source says. A compiler may pass a pair in two
+ * registers, add a pointer to the result, drop arguments that are not used.
+ */
+static __naked __noinline __used long takes_6_says_4(long a, long b, long c, long d)
+{
+ asm volatile (
+ "r0 = *(u64 *)(r11 + 8);"
+ "r0 += r5;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: static subprog takes more than its prototype says")
+__success __retval(11)
+__arch_x86_64
+/* takes_6_says_4() passes no stack arguments: rsp stays where it is */
+__jited("func #1")
+__jited(" endbr64")
+__jited(" nopl (%rax,%rax)")
+__jited(" nopl (%rax)")
+__jited(" pushq %rbp")
+__jited(" movq %rsp, %rbp")
+__jited(" endbr64")
+__jited(" movq %r9, %rax")
+__naked void more_than_prototype(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call takes_6_says_4;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+static __naked __noinline __used long cb_takes_6(long i, void *ctx)
+{
+ asm volatile (
+ "r0 = *(u64 *)(r11 + 8);"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: helpers pass no stack arguments to a callback")
+__failure __msg("callback cb_takes_6() cannot have stack arguments")
+__naked void callback_takes_6(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = %[cb_takes_6];"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_ptr(cb_takes_6),
+ __imm(bpf_loop)
+ : __clobber_common);
+}
+
+static __naked __noinline __used long takes_1_says_7(long a, long b, long c, long d, long e,
+ long f, long g)
+{
+ asm volatile (
+ "r0 = r1;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: static subprog takes less than its prototype says")
+__success __retval(1)
+__naked void less_than_prototype(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "call takes_1_says_7;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+SEC("tc")
+__description("stack_arg: what the static subprog takes is not written")
+__failure __msg("callee expects 6 args, stack arg1 is not initialized")
+__naked void more_than_prototype_not_written(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "call takes_6_says_4;"
+ "exit;"
+ ::: __clobber_all);
+}
+
#else
SEC("socket")
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c
index ab1955852233..42b6d4a4b4d4 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg_order.c
@@ -152,7 +152,7 @@ __naked void stack_arg_read_without_write_1(void)
SEC("tc")
__description("stack_arg: read with not-initialized caller write")
__failure
-__msg("R0 !read_ok")
+__msg("callee expects 7 args, stack arg1 is not initialized")
__btf_func_path("btf__verifier_stack_arg_order.bpf.o")
__naked void stack_arg_read_without_write_2(void)
{
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 34/81] bpf: Let scalar evolution see loads from read-only maps
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (32 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 33/81] selftests/bpf: Test static subprogs that don't take what BTF says Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 35/81] selftests/bpf: Test scalar evolution with a loop bound in .rodata Alexei Starovoitov
` (48 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
User space often sets the bound of a loop in .rodata before it loads the
prog. The variable is volatile, thus the prog loads it in each iteration:
const volatile int nr_cpus;
for (i = 0; i < nr_cpus; i++)
Scalar evolution calculates how many times a loop runs. Then the verifier
does not have to verify the loop one iteration at a time. But scalar
evolution does not examine maps. For it the result of the load is unknown,
and it cannot calculate the number of iterations:
loop header at 3, non-linear latch at 5
The map is read-only, thus the load always gives the same constant.
bpf_compute_const_regs() runs before scalar evolution and already knows
this constant. Use it. insn_aux_data of an insn describes the registers
before this insn. Thus the result of the load is in insn_aux_data of the
next insn.
loop header at 3, header_count is 10
A loop with a large bound in .rodata reached the limit of 1M insns. Now it
loads, as it does with the same bound in a register.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/scev.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 44d524c966cd..5c2150928ff3 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -775,6 +775,15 @@ static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
switch (mode) {
case BPF_MEM:
case BPF_MEMSX:
+ /*
+ * for (i = 0; i < nr_cpus; i++) with "const volatile int nr_cpus":
+ * bpf_compute_const_regs() knows what is loaded from a read-only map.
+ */
+ if (insn->src_reg != BPF_REG_FP && idx + 1 < env->prog->len &&
+ (env->insn_aux_data[idx + 1].const_reg_mask & BIT(dst))) {
+ imm = env->insn_aux_data[idx + 1].const_reg_vals[dst];
+ return replace_reg(scev, e, dst, imm_expr(scev, imm));
+ }
return maybe_load_fp(scev, e, insn);
default:
goto mark_dst_unknown;
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 35/81] selftests/bpf: Test scalar evolution with a loop bound in .rodata
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (33 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 34/81] bpf: Let scalar evolution see loads from read-only maps Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 36/81] bpf: Helpers don't write through ARG_ANYTHING Alexei Starovoitov
` (47 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The bound of the loop is 10. It is in .rodata, and the prog loads it in
each iteration, as clang does for a 'const volatile' variable. The test
expects that scalar evolution calculates the number of iterations:
loop header at 3, header_count is 10
Without "bpf: Let scalar evolution see loads from read-only maps" the
message is 'non-linear latch at 5'.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 23 +++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 61bb39a12de2..c242724686fc 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -1411,6 +1411,29 @@ __naked void widened_reg_is_not_linked(void)
: __clobber_all);
}
+const volatile __u32 rodata_bound = 10;
+
+/* for (i = 0; i < rodata_bound; i++): the bound is loaded in every iteration. */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop header at 3, header_count is 10")
+__naked void bound_from_rodata(void)
+{
+ asm volatile (
+ "r6 = 0;"
+ "r7 = %[rodata_bound] ll;"
+"1:"
+ "r6 += 1;"
+ "r1 = *(u32 *)(r7 + 0);"
+ "if r6 < r1 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(rodata_bound)
+ : __clobber_all);
+}
+
/* The unknown SCEV of an unbound STACK_MISC slot must not prevent widening. */
SEC("tracepoint")
__success
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 36/81] bpf: Helpers don't write through ARG_ANYTHING
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (34 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 35/81] selftests/bpf: Test scalar evolution with a loop bound in .rodata Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 37/81] selftests/bpf: Test scalar evolution with stack address as ARG_ANYTHING Alexei Starovoitov
` (46 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A loop can keep its counter in a stack slot. Scalar evolution calculates
how many times such a loop runs only if no insn in the loop can write to
the slot through a pointer. For a call of a helper,
bpf_helper_stack_access_bytes() tells what the helper does through each
argument. For an argument type that it does not list, it assumes a read and
a write of any size. ARG_ANYTHING is such a type.
strobemeta_nounroll1 calls bpf_probe_read_user_str() in a loop. r3 is the
address to read from, and its type is ARG_ANYTHING. The liveness analysis
cannot tell if r3 points to the stack. Thus it assumes that the call can
write to all stack slots. One of them is the counter of the outer loop:
loop header at 239, can't compute iterations count
ARG_ANYTHING is a number that the prog selects freely, and the verifier
does not check it. Thus no helper writes to kernel memory through it.
bpf_probe_write_user() writes to user space only.
Clear may_write for ARG_ANYTHING. Only scalar evolution uses may_write.
Keep may_read, because bpf_probe_read_kernel() can read the slot.
loop header at 239, header_count is 13
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/verifier.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 0f2bae1c2305..4752a4619035 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -15088,6 +15088,14 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
* analyzed separately.
*/
return (struct arg_access_info) {};
+ case ARG_ANYTHING:
+ /*
+ * A number that the program is free to choose. bpf_probe_read_kernel() and
+ * the like may read what it points to. Nothing in the kernel is written
+ * through it: bpf_probe_write_user() takes addresses of user space only.
+ */
+ info.may_write = false;
+ return info;
default:
return info;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 37/81] selftests/bpf: Test scalar evolution with stack address as ARG_ANYTHING
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (35 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 36/81] bpf: Helpers don't write through ARG_ANYTHING Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:58 ` [PATCH bpf 38/81] bpf: Simplify (zext32 (zext32 x)) in scalar evolution Alexei Starovoitov
` (45 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The loop keeps its counter in fp-8. In each iteration it calls
bpf_probe_read_kernel() to read from the address of fp-16. This argument is
ARG_ANYTHING. The verifier assumed that the helper can write through it,
and thus can change the counter.
The test expects that scalar evolution calculates the number of iterations:
loop header at 2, header_count is 10
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 31 +++++++++++++++++++
1 file changed, 31 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index c242724686fc..bd632951492a 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -1434,6 +1434,37 @@ __naked void bound_from_rodata(void)
: __clobber_all);
}
+/*
+ * The address of a stack slot is passed as a number. The helper may read what is
+ * there. It doesn't write, so the counter in the slot next to it is intact.
+ */
+SEC("tracepoint")
+__success
+__log_level(2)
+__msg("loop header at 2, header_count is 10")
+__naked void stack_address_as_a_number(void)
+{
+ asm volatile (
+ "*(u64 *)(r10 - 8) = 0;"
+ "*(u64 *)(r10 - 16) = 0;"
+"1:"
+ "r1 = r10;"
+ "r1 += -24;"
+ "r2 = 8;"
+ "r3 = r10;"
+ "r3 += -16;"
+ "call %[bpf_probe_read_kernel];"
+ "r1 = *(u64 *)(r10 - 8);"
+ "r1 += 1;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "if r1 < 10 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_probe_read_kernel)
+ : __clobber_all);
+}
+
/* The unknown SCEV of an unbound STACK_MISC slot must not prevent widening. */
SEC("tracepoint")
__success
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 38/81] bpf: Simplify (zext32 (zext32 x)) in scalar evolution
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (36 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 37/81] selftests/bpf: Test scalar evolution with stack address as ARG_ANYTHING Alexei Starovoitov
@ 2026-10-26 2:58 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 39/81] selftests/bpf: Test scalar evolution with a latch on a 32-bit copy Alexei Starovoitov
` (44 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:58 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
clang compiles 'for (i = zero; i < 17 && can_loop; i++)' in
test_buddy_alignment() of libarena into
w1 = w6
w6 += 1
if w1 < 16 goto ...
Scalar evolution describes the value of a register as an expression.
(linear r6 1) is a counter that starts at r6 and grows by 1 in each
iteration. A 32-bit insn clears the upper half of its result, and zext32
shows that in the expression.
the 32-bit counter w6 (zext32 (linear r6 1))
its 32-bit copy w1 (zext32 (zext32 (linear r6 1)))
The latch (the jump back to the loop head) tests the copy. Scalar evolution
recognizes a counter inside one zext32 only. Thus it cannot calculate how
many times the loop runs:
loop header at 1, non-linear latch at 3
The outer zext32 does not change the value. Add the rule:
(zext32 (zext32 x)) -> (zext32 x)
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/scev.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 5c2150928ff3..d0ba709694db 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1507,6 +1507,10 @@ static int simplify(struct scev *scev, u32 id, void *priv)
if (is_zext32(scev, id, &l) && is_imm(scev, l, &imm1))
return imm_expr(scev, (u64)(u32)imm1);
+ /* (zext32 (zext32 x)) -> (zext32 x) */
+ if (is_zext32(scev, id, &l) && is_zext32(scev, l, &a))
+ return l;
+
/*
* (zext32 (+ (zext32 $a) $b)) -> (zext32 (+ $a $b))
* (zext32 (+ $a (zext32 $b))) -> (zext32 (+ $a $b))
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 39/81] selftests/bpf: Test scalar evolution with a latch on a 32-bit copy
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (37 preceding siblings ...)
2026-10-26 2:58 ` [PATCH bpf 38/81] bpf: Simplify (zext32 (zext32 x)) in scalar evolution Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 40/81] bpf: Keep the range of a packet pointer when a number >= 0 is added Alexei Starovoitov
` (43 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The prog is what clang emits for 'for (i = 0; i < 17; i++)'. The latch (the
jump back to the loop head) tests a 32-bit copy of the counter. The prog
makes the copy before the increment:
1:
w1 = w6
w6 += 1
if w1 < 16 goto 1b
The test expects 'header_count is 17'. Without
"bpf: Simplify (zext32 (zext32 x)) in scalar evolution" the message is
'non-linear latch'.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_scev.c | 21 +++++++++++++++++++
1 file changed, 21 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index bd632951492a..5a7c32ce1c6c 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -1411,6 +1411,27 @@ __naked void widened_reg_is_not_linked(void)
: __clobber_all);
}
+/*
+ * for (i = 0; i < 17; i++), the way clang does it: the copy that is compared is made before
+ * the increment, all in 32 bit: (zext32 (zext32 (linear r6 1))).
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop header at 1, header_count is 17")
+__naked void latch_tests_a_32bit_copy(void)
+{
+ asm volatile (
+ "w6 = 0;"
+"1:"
+ "w1 = w6;"
+ "w6 += 1;"
+ "if w1 < 16 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
const volatile __u32 rodata_bound = 10;
/* for (i = 0; i < rodata_bound; i++): the bound is loaded in every iteration. */
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 40/81] bpf: Keep the range of a packet pointer when a number >= 0 is added
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (38 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 39/81] selftests/bpf: Test scalar evolution with a latch on a 32-bit copy Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 41/81] selftests/bpf: Test range of a packet pointer after a number " Alexei Starovoitov
` (42 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
After a prog compares 'pkt + 54' with pkt_end, the verifier knows that
54 bytes are in the packet. It keeps this number in the pointer as the
range (r=54 in the log). The range stays when the prog adds a constant to
the pointer. But the verifier clears the range when the prog adds a scalar
that is not a constant, although the scalar has an upper bound:
if (pkt + 54 > pkt_end || off > 38)
return 0;
*(u64 *)(pkt + off) rejected: pkt has r=54, the sum has r=0
A prog with 'off = c ? 38 : 22' loads only because the verifier follows the
two paths separately, and each path adds a constant.
"bpf: Add join of verifier states" makes one state from the two, with
off=[22,38]. Then the verifier rejects this prog too.
Keep the range when the scalar cannot be negative. The verifier accepts a
load if umax + insn->off + size <= range, where umax is the largest offset
of the pointer. Thus it accepts range - umax bytes after the pointer. A
scalar in [0, N] adds N to umax, and it moves the pointer by N at most:
pkt pkt + off pkt + 38 pkt + 54
| | | |
the sum its umax its range
|<-- 16 bytes ->| accepted
|<------- 54 - off bytes -------->| in the packet
The verifier accepts 54 - 38 bytes after the sum. 54 - off bytes are in the
packet, and off <= 38.
This is correct only if umax of the sum is exactly the sum of the two umax.
reg_bounds_sync() can make umax smaller because of var_off. Commit
b88822d2584f ("bpf: Fix packet range of pointers sharing an id") skips
reg_bounds_sync() for a packet pointer that has a range, thus also for the
sum.
The sum gets a new id, as before. Thus a comparison of the sum with pkt_end
does not change the range of pkt.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/verifier.c | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 4752a4619035..8f5aa77e4ecc 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -16546,12 +16546,16 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
if (!known)
dst_reg->id = ++env->id_gen;
/*
- * Clear range for unknown addends since we can't know
- * where the pkt pointer ended up. Also clear AT_PKT_END /
- * BEYOND_PKT_END from prior comparison as any pointer
- * arithmetic invalidates them.
+ * The range is what umax of the pointer may get to, see
+ * check_packet_access(). umax of the addend was added to it
+ * and the pointer moved by that much at most, so the range
+ * holds. Clear it if the addend may be negative: umax says
+ * nothing then, or if nothing is left of it. Also clear
+ * AT_PKT_END / BEYOND_PKT_END from prior comparison as any
+ * pointer arithmetic invalidates them.
*/
- if (!known || dst_reg->range < 0)
+ if (dst_reg->range < 0 ||
+ (!known && (smin_val < 0 || reg_umax(dst_reg) >= dst_reg->range)))
memset(&dst_reg->raw, 0, sizeof(dst_reg->raw));
}
break;
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 41/81] selftests/bpf: Test range of a packet pointer after a number is added
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (39 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 40/81] bpf: Keep the range of a packet pointer when a number >= 0 is added Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 42/81] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions Alexei Starovoitov
` (41 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier keeps the range of a packet pointer when the prog adds a
scalar >= 0. The range must permit exactly the bytes that the prog checked.
In these tests the prog checks 54 bytes:
test (pkt_add_...) added 8-byte load result
---------------------- -------------------- ----------- ------------
range_keeps_range [0, 38] +0, +8 loads
range_load_past_range [0, 38] +9 reject, r=54
range_twice [0, 20], 1, [0, 20] +5 fits, +6 reject, r=54
negative_drops_range 20, [-4, 4] +0 reject, r=0
one_of_two_offsets 38 or 22 +0 loads
A comparison with pkt_end sets the range of all pointers that have the same
id. This is correct only if the distance between these pointers is a
constant. In these tests the prog has two equal pointers and moves one of
them by a value that is not a constant:
test what moves the pointer compared used result
---------------------- ------------------------- -------- ---- ------
kept_range_old_pointer r7 += [0, 7] r7 + 100 r8 reject
kept_range_new_check r7 += [0, 7] r7 + 100 r7 loads
loop_other_pointer r7 += 1 in a loop r7 r8 reject
loop_same_pointer r7 += 1 in a loop r7 r7 loads
loop_one_of_two r2 = pkt + 1000 in a loop r2 r9 reject
merge_const r7 += 8 on one path r7 r8 reject
merge_displacement the same, both have an id r7 r8 reject
merge_number r7 += [0, 15] on one path r7 r8 reject
Two more progs must be rejected:
- kept_range_merge adds [0, 7] on one path and [0, 15] on the other path.
The load fits only [0, 7].
- kept_range_var_off is the prog from commit b88822d2584f
("bpf: Fix packet range of pointers sharing an id"), with a pointer that
keeps its range.
loop_other_pointer is from
https://lore.kernel.org/bpf/21a8e3dfacb4269e348b10544c43cd05f785737b.camel@gmail.com/
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../bpf/progs/verifier_direct_packet_access.c | 375 ++++++++++++++++++
1 file changed, 375 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c b/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c
index 139ff019d87d..aa4abe22655f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c
+++ b/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c
@@ -1098,4 +1098,379 @@ l0_%=: r0 = 0; \
: __clobber_all);
}
+/* 54 bytes are checked. What is added is 38 at most, so there are 16 left. */
+SEC("tc")
+__description("direct packet access: a number in a range is added to a checked pointer")
+__success __retval(0)
+__naked void pkt_add_range_keeps_range(void)
+{
+ asm volatile (" \
+ r2 = *(u32*)(r1 + %[__sk_buff_data]); \
+ r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \
+ r4 = *(u32*)(r1 + %[__sk_buff_mark]); \
+ r5 = r2; \
+ r5 += 54; \
+ if r5 > r3 goto l0_%=; \
+ if r4 > 38 goto l0_%=; \
+ r2 += r4; \
+ r0 = *(u64*)(r2 + 0); \
+ r0 = *(u64*)(r2 + 8); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)),
+ __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("direct packet access: a number in a range is added, load past range")
+__failure __msg("invalid access to packet, off=47 size=8, R2(id={{[0-9]+}},off=47,r=54)")
+__naked void pkt_add_range_load_past_range(void)
+{
+ asm volatile (" \
+ r2 = *(u32*)(r1 + %[__sk_buff_data]); \
+ r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \
+ r4 = *(u32*)(r1 + %[__sk_buff_mark]); \
+ r5 = r2; \
+ r5 += 54; \
+ if r5 > r3 goto l0_%=; \
+ if r4 > 38 goto l0_%=; \
+ r2 += r4; \
+ r0 = *(u64*)(r2 + 9); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)),
+ __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark))
+ : __clobber_all);
+}
+
+/* twice: 20 + 20, and a constant in between */
+SEC("tc")
+__description("direct packet access: two numbers in a range are added to a checked pointer")
+__failure __msg("invalid access to packet, off=47 size=8, R2(id={{[0-9]+}},off=47,r=54)")
+__naked void pkt_add_range_twice(void)
+{
+ asm volatile (" \
+ r2 = *(u32*)(r1 + %[__sk_buff_data]); \
+ r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \
+ r4 = *(u32*)(r1 + %[__sk_buff_mark]); \
+ r5 = r2; \
+ r5 += 54; \
+ if r5 > r3 goto l0_%=; \
+ if r4 > 20 goto l0_%=; \
+ r2 += r4; \
+ r2 += 1; \
+ r2 += r4; \
+ r0 = *(u64*)(r2 + 5); \
+ r0 = *(u64*)(r2 + 6); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)),
+ __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("direct packet access: a number that may be negative is added to a checked pointer")
+__failure __msg("invalid access to packet, off=16 size=8, R2(id={{[0-9]+}},off=16,r=0)")
+__naked void pkt_add_negative_drops_range(void)
+{
+ asm volatile (" \
+ r2 = *(u32*)(r1 + %[__sk_buff_data]); \
+ r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \
+ r4 = *(u32*)(r1 + %[__sk_buff_mark]); \
+ r5 = r2; \
+ r5 += 54; \
+ if r5 > r3 goto l0_%=; \
+ r2 += 20; \
+ if r4 > 8 goto l0_%=; \
+ r4 += -4; \
+ r2 += r4; \
+ r0 = *(u64*)(r2 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end)),
+ __imm_const(__sk_buff_mark, offsetof(struct __sk_buff, mark))
+ : __clobber_all);
+}
+
+/* off = x ? 38 : 22; the compiler makes one load of it */
+SEC("tc")
+__description("direct packet access: one of two offsets is added to a checked pointer")
+__success __retval(0)
+__naked void pkt_add_one_of_two_offsets(void)
+{
+ asm volatile (" \
+ r6 = *(u32*)(r1 + %[__sk_buff_data]); \
+ r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \
+ r5 = r6; \
+ r5 += 54; \
+ if r5 > r3 goto l0_%=; \
+ call %[bpf_get_prandom_u32]; \
+ r1 = 38; \
+ if r0 != 0 goto l1_%=; \
+ r1 = 22; \
+l1_%=: r6 += r1; \
+ r0 = *(u64*)(r6 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
+/*
+ * See commit b88822d2584f ("bpf: Fix packet range of pointers sharing an id").
+ * r5 keeps the range of r7, r6 is r5 + 8. umax of r6 has to be 58 and not 56.
+ */
+SEC("tc")
+__description("direct packet access: a range that is kept, var_off and umax of two pointers")
+__failure __msg("invalid access to packet")
+__naked void kept_range_var_off(void)
+{
+ asm volatile (
+ "r2 = *(u32 *)(r1 + %[__sk_buff_data]);"
+ "r3 = *(u32 *)(r1 + %[__sk_buff_data_end]);"
+ "r9 = r3; r7 = r2; r4 = r2; r4 += 51;"
+ "if r4 > r3 goto 1f;" /* range 51 */
+ "call %[bpf_get_prandom_u32];"
+ "r4 = r0; r4 &= 0x38;"
+ "if r4 > 50 goto 1f;" /* umax 50, var_off (0x0; 0x38) */
+ "r5 = r7; r5 += r4;" /* umax 50 < 51: the range is kept */
+ "r6 = r5; r6 += 8;" /* umax 58, or 56 */
+ "if r5 > r9 goto 1f;" /* sets the range of the id from umax of r5 */
+ "r0 = *(u8 *)(r6 - 7);" /* r5 + 1 */
+"1: r0 = 0; exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("direct packet access: a range that is kept, 0..7 or 0..15 are added")
+__failure __msg("invalid access to packet")
+__naked void kept_range_merge(void)
+{
+ asm volatile (
+ "r2 = *(u32 *)(r1 + %[__sk_buff_data]);"
+ "r3 = *(u32 *)(r1 + %[__sk_buff_data_end]);"
+ "r9 = r3; r7 = r2; r4 = r2; r4 += 64;"
+ "if r4 > r3 goto 3f;"
+ "call %[bpf_get_prandom_u32];"
+ "r6 = r0; r6 &= 15;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 0 goto 1f;"
+ "r6 &= 7;"
+"1: r7 += r6;" /* [0,7] or [0,15], range 64 */
+ "r0 = *(u8 *)(r7 + 49);"
+"3: r0 = 0; exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("direct packet access: a range that is kept, the sum is checked, the pointer is used")
+__failure __msg("invalid access to packet")
+__naked void kept_range_old_pointer(void)
+{
+ asm volatile (
+ "r2 = *(u32 *)(r1 + %[__sk_buff_data]);"
+ "r3 = *(u32 *)(r1 + %[__sk_buff_data_end]);"
+ "r9 = r3; r7 = r2; r8 = r2; r4 = r2; r4 += 16;"
+ "if r4 > r3 goto 1f;" /* range 16 */
+ "call %[bpf_get_prandom_u32];"
+ "r0 &= 7;"
+ "r7 += r0;" /* new id, range 16 */
+ "r4 = r7; r4 += 100;"
+ "if r4 > r9 goto 1f;" /* r7 + 100 is there */
+ "r0 = *(u8 *)(r8 + 106);" /* r8 + 100 is, r8 + 107 is not */
+"1: r0 = 0; exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("direct packet access: a range that is kept, a check of the sum")
+__success
+__naked void kept_range_new_check(void)
+{
+ asm volatile (
+ "r2 = *(u32 *)(r1 + %[__sk_buff_data]);"
+ "r3 = *(u32 *)(r1 + %[__sk_buff_data_end]);"
+ "r9 = r3; r7 = r2; r8 = r2; r4 = r2; r4 += 16;"
+ "if r4 > r3 goto 1f;"
+ "call %[bpf_get_prandom_u32];"
+ "r0 &= 7;"
+ "r7 += r0;"
+ "r4 = r7; r4 += 100;"
+ "if r4 > r9 goto 1f;"
+ "r0 = *(u8 *)(r7 + 99);"
+"1: r0 = 0; exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
+/* Nobody knows how many times the loop ran, so r7 < data_end is not about r8 */
+SEC("tc")
+__description("direct packet access: one pointer moves in a loop, the other one is used")
+__failure __msg("invalid access to packet")
+__naked void loop_other_pointer(void)
+{
+ asm volatile (
+ "r2 = *(u32 *)(r1 + %[__sk_buff_data]);"
+ "r3 = *(u32 *)(r1 + %[__sk_buff_data_end]);"
+ "r7 = r2; r8 = r2; r9 = r3; r6 = 0;"
+"1: if r6 >= 10 goto 2f;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 42 goto 2f;"
+ "r7 += 1; r6 += 1; goto 1b;"
+"2: if r7 >= r9 goto 3f;"
+ "r0 = *(u8 *)(r8 + 4);"
+"3: r0 = 0; exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("direct packet access: a pointer that moved in a loop is checked and used")
+__success
+__naked void loop_same_pointer(void)
+{
+ asm volatile (
+ "r2 = *(u32 *)(r1 + %[__sk_buff_data]);"
+ "r3 = *(u32 *)(r1 + %[__sk_buff_data_end]);"
+ "r7 = r2; r8 = r2; r9 = r3; r6 = 0;"
+"1: if r6 >= 10 goto 2f;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 42 goto 2f;"
+ "r7 += 1; r6 += 1; goto 1b;"
+"2: if r7 >= r9 goto 3f;"
+ "r0 = *(u8 *)(r7 + 0);"
+"3: r0 = 0; exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("direct packet access: a pointer that becomes another one in a loop")
+__failure __msg("invalid access to packet")
+__naked void loop_one_of_two(void)
+{
+ asm volatile (
+ "r2 = *(u32 *)(r1 + %[__sk_buff_data]);"
+ "r3 = *(u32 *)(r1 + %[__sk_buff_data_end]);"
+ "r9 = r2; r7 = r2; r7 += 1000; r6 = 0;"
+"1: if r6 >= 3 goto 3f;"
+ "if r2 > r3 goto 3f;"
+ "r0 = *(u8 *)(r9 + 999);"
+ "if r6 != 1 goto 2f;"
+ "r2 = r7;"
+"2: r6 += 1; goto 1b;"
+"3: r0 = 0; exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("direct packet access: 8 are added on one path, the other pointer is used")
+__failure __msg("invalid access to packet")
+__naked void merge_const(void)
+{
+ asm volatile (
+ "r2 = *(u32 *)(r1 + %[__sk_buff_data]);"
+ "r3 = *(u32 *)(r1 + %[__sk_buff_data_end]);"
+ "r7 = r2; r8 = r2; r9 = r3;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 0 goto 1f;"
+ "r7 += 8;"
+"1: if r7 > r9 goto 2f;"
+ "r0 = *(u8 *)(r8 + 7);"
+"2: r0 = 0; exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("direct packet access: 8 are added on one path, pointers that share an id")
+__failure __msg("invalid access to packet")
+__naked void merge_displacement(void)
+{
+ asm volatile (
+ "r2 = *(u32 *)(r1 + %[__sk_buff_data]);"
+ "r3 = *(u32 *)(r1 + %[__sk_buff_data_end]);"
+ "r9 = r3; r6 = r2;"
+ "call %[bpf_get_prandom_u32];"
+ "r0 &= 15;"
+ "r6 += r0;" /* id N */
+ "r7 = r6; r8 = r6;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 0 goto 1f;"
+ "r7 += 8;"
+"1: if r7 > r9 goto 2f;"
+ "r0 = *(u8 *)(r8 + 7);"
+"2: r0 = 0; exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("direct packet access: a number is added on one path, the other pointer is used")
+__failure __msg("invalid access to packet")
+__naked void merge_number(void)
+{
+ asm volatile (
+ "r2 = *(u32 *)(r1 + %[__sk_buff_data]);"
+ "r3 = *(u32 *)(r1 + %[__sk_buff_data_end]);"
+ "r9 = r3; r7 = r2; r8 = r2;"
+ "call %[bpf_get_prandom_u32];"
+ "r6 = r0; r6 &= 15;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 0 goto 1f;"
+ "r7 += r6;"
+"1: if r7 > r9 goto 2f;"
+ "r0 = *(u8 *)(r8 + 7);"
+"2: r0 = 0; exit;"
+ :
+ : __imm(bpf_get_prandom_u32),
+ __imm_const(__sk_buff_data, offsetof(struct __sk_buff, data)),
+ __imm_const(__sk_buff_data_end, offsetof(struct __sk_buff, data_end))
+ : __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 41/81] selftests/bpf: Test range of a packet pointer after a number is added
2026-10-26 2:59 ` [PATCH bpf 41/81] selftests/bpf: Test range of a packet pointer after a number " Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3852 bytes --]
> selftests/bpf: Test range of a packet pointer after a number is added
TL;DR: Test issue: the new packet-pointer selftests lack
__flag(BPF_F_ANY_ALIGNMENT), so on kernels without efficient unaligned access
(e.g. riscv64) they may be rejected for misaligned access, and the __msg()
checks may not match.
> diff --git a/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c b/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c
> index 139ff019d87d..aa4abe22655f 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c
[ ... ]
> @@ -1098,4 +1098,379 @@ l0_%=: r0 = 0; \
> : __clobber_all);
> }
>
> +/* 54 bytes are checked. What is added is 38 at most, so there are 16 left. */
> +SEC("tc")
> +__description("direct packet access: a number in a range is added to a checked pointer")
> +__success __retval(0)
> +__naked void pkt_add_range_keeps_range(void)
> +{
> + asm volatile (" \
> + r2 = *(u32*)(r1 + %[__sk_buff_data]); \
> + r3 = *(u32*)(r1 + %[__sk_buff_data_end]); \
> + r4 = *(u32*)(r1 + %[__sk_buff_mark]); \
> + r5 = r2; \
> + r5 += 54; \
> + if r5 > r3 goto l0_%=; \
> + if r4 > 38 goto l0_%=; \
> + r2 += r4; \
> + r0 = *(u64*)(r2 + 0); \
> + r0 = *(u64*)(r2 + 8); \
Without __flag(BPF_F_ANY_ALIGNMENT), can this test be loaded on kernels
that don't have CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS, such as riscv64 with
its default config?
bpf_check() sets strict alignment in that case:
env->strict_alignment = !!(attr->prog_flags & BPF_F_STRICT_ALIGNMENT);
if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS))
env->strict_alignment = true;
if (attr->prog_flags & BPF_F_ANY_ALIGNMENT)
env->strict_alignment = false;
check_mem_access() calls check_ptr_alignment() before check_packet_access(),
and check_pkt_ptr_alignment() adds NET_IP_ALIGN (2), var_off and the
instruction offset before checking the alignment against the access size:
reg_off = tnum_add(reg->var_off, tnum_const(ip_align + off));
if (!tnum_is_aligned(reg_off, size)) {
...
"misaligned packet access off %d+%s+%d size %d\n",
Here r2 = pkt + r4, and r4 is only bounded by "if r4 > 38 goto", so var_off
is (0x0; 0x3f) and the 8-byte loads at r2 + 0 and r2 + 8 are not 8-aligned.
Wouldn't the verifier reject this __success test with:
misaligned packet access off 2+(0x0; 0x3f)+0 size 8
[ ... ]
> +SEC("tc")
> +__description("direct packet access: a number in a range is added, load past range")
> +__failure __msg("invalid access to packet, off=47 size=8, R2(id={{[0-9]+}},off=47,r=54)")
> +__naked void pkt_add_range_load_past_range(void)
[ ... ]
> + r0 = *(u64*)(r2 + 9); \
The same applies to this test, and to pkt_add_range_twice and
pkt_add_negative_drops_range, which also do 8-byte loads at unaligned
offsets from a pointer with a variable offset (r2 + 9, r2 + 5, r2 + 6 and
r2 + 0 with r4 in [-4, 4]).
Those three are __failure tests, so on such kernels would they be rejected at
the alignment check instead? The expected message, for example:
invalid access to packet, off=47 size=8, R2(id={{[0-9]+}},off=47,r=54)
would then never appear in the log, and the __msg() would not match.
Other tests in this file that do multi-byte packet loads at a variable or
unaligned offset, such as the "x += pkt_ptr" tests test20 to test24 and
"checked pointer minus non-negative unknown keeps range", set
__flag(BPF_F_ANY_ALIGNMENT). Should these four tests set it as well?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 42/81] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (40 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 41/81] selftests/bpf: Test range of a packet pointer after a number " Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 43/81] bpf: Tell user space that a run of a prog was cut short Alexei Starovoitov
` (40 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
'tbl[i]()' loads a pointer from a table of functions in .rodata. When i is
not a constant, the verifier continues with one state for each pointer that
the load can give. It rejects a load that can give more than 64 pointers.
Rust needs more. In one prog of rust-bpf, fmt() of a trait object has 245
implementations.
The limit of 64 is arbitrary. Raise it to BPF_MAX_SUBPROGS. A table that
contains each function one time cannot have more entries. The array of
targets is now too large for the kernel stack, thus allocate it.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/verifier.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 8f5aa77e4ecc..e179514c3b59 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -7104,7 +7104,7 @@ static void mark_reg_func_ptr(struct bpf_verifier_env *env, struct bpf_reg_state
}
/* a read from a table of functions branches into that many states at most */
-#define BPF_MAX_FUNC_PTR_TARGETS 64
+#define BPF_MAX_FUNC_PTR_TARGETS BPF_MAX_SUBPROGS
/* and that many pointers at most are next to what is read at variable offset */
#define BPF_MAX_FUNC_PTR_RANGE 4096
@@ -7150,17 +7150,20 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
struct bpf_reg_state *regs = cur_regs(env);
struct bpf_map *map = reg->map_ptr;
struct bpf_verifier_state *branch;
- int subprog, targets[BPF_MAX_FUNC_PTR_TARGETS];
+ u16 *targets __free(kfree) = NULL;
struct bpf_func_ptr *ptrs;
u32 i, cnt, n = 0;
u64 o, lo, hi;
- int err;
+ int err, subprog;
ptrs = bpf_map_range_func_ptrs(env, map, min_off, max_off - min_off + size, &cnt);
if (!ptrs)
return -ENOENT;
if (cnt > BPF_MAX_FUNC_PTR_RANGE)
goto overlap;
+ targets = kmalloc_array(BPF_MAX_FUNC_PTR_TARGETS, sizeof(*targets), GFP_KERNEL_ACCOUNT);
+ if (!targets)
+ return -ENOMEM;
for (i = 0; i < cnt; i++) {
/* the offsets at which the read overlaps with the pointer */
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 42/81] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions
2026-10-26 2:59 ` [PATCH bpf 42/81] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1680 bytes --]
> bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions
TL;DR: Nit only: check_func_ptr_read() uses kmalloc_array(), which
deprecated.rst discourages; kmalloc_objs() might read better, though the
review calls it a style point and not a bug.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 8f5aa77e4ecc..e179514c3b59 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -7150,17 +7150,20 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
[ ... ]
> ptrs = bpf_map_range_func_ptrs(env, map, min_off, max_off - min_off + size, &cnt);
> if (!ptrs)
> return -ENOENT;
> if (cnt > BPF_MAX_FUNC_PTR_RANGE)
> goto overlap;
> + targets = kmalloc_array(BPF_MAX_FUNC_PTR_TARGETS, sizeof(*targets), GFP_KERNEL_ACCOUNT);
> + if (!targets)
> + return -ENOMEM;
this isn't a bug, but Documentation/process/deprecated.rst lists the
open-coded form:
ptr = kmalloc_array(count, sizeof(*ptr), gfp);
as deprecated in favor of:
ptr = kmalloc_objs(*ptr, count [, gfp]);
verifier.c already uses kmalloc_objs() for used_maps and used_btfs, and this
appears to be the only kmalloc_array() left under kernel/bpf/.
The deprecated.rst text is written for struct objects and targets is an
array of u16, so this is a style point only. Would this read better as:
targets = kmalloc_objs(*targets, BPF_MAX_FUNC_PTR_TARGETS, GFP_KERNEL_ACCOUNT);
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 43/81] bpf: Tell user space that a run of a prog was cut short
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (41 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 42/81] bpf: Allow BPF_MAX_SUBPROGS targets of a read from a table of functions Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 44/81] bpf: Mask the count of a 32-bit shift in scalar evolution Alexei Starovoitov
` (39 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A run of a prog can end before the prog completes its work:
- A loop with may_goto runs for more than 250 ms. Then may_goto exits the
loop.
- The prog leaves with unwind_exit, for example after a Rust panic, and no
caller catches it. 'call f; unwind_goto pad' catches it: the caller
continues at pad when f leaves with unwind_exit.
The hook gets a return value, as after a usual run. No value is correct for
XDP, LSM and sched-ext at the same time. Thus user space must know about
the event and decide what to do.
Add the bit 'unwound' to struct bpf_prog_info. The kernel sets the bit when
a run is cut short, and does not clear it. bpftool prints it. The stderr
stream of the prog has the cause.
cause who reports it
----------------------------- ----------------------------------------
may_goto times out the function that prints the timeout
unwind_exit in the main prog a call that the verifier adds before it
unwind_exit in an async the JIT code of unwind_exit, when it
callback finds no unwind_goto after the call
The kernel calls the main prog and the async callbacks. Thus nothing can
catch their unwind_exit. The message for these two causes is new:
ERROR: The program was unwound
The verifier adds 'r1 = r0; call bpf_prog_report_unwound'. The function
returns its argument. Thus the return value of the prog does not change. An
extension prog gets no call: it unwinds into the caller of the subprog that
it replaces.
Progs without unwind_exit in the main prog get the same code. A run that
leaves with exit does not execute the added insns. A run that unwinds to
the kernel costs about 165 nsec more on x86-64 (Zen 5).
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
arch/arm64/net/bpf_jit_comp.c | 2 +-
arch/x86/net/bpf_jit_comp.c | 2 +-
include/linux/bpf.h | 2 ++
include/linux/bpf_verifier.h | 1 +
include/uapi/linux/bpf.h | 7 ++++++-
kernel/bpf/core.c | 23 +++++++++++++++++++++++
kernel/bpf/fixups.c | 27 +++++++++++++++++++++++++++
kernel/bpf/syscall.c | 1 +
kernel/bpf/verifier.c | 3 +++
tools/bpf/bpftool/prog.c | 2 ++
tools/include/uapi/linux/bpf.h | 7 ++++++-
11 files changed, 73 insertions(+), 4 deletions(-)
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 58f30911bdab..ea4102af7ead 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -1282,7 +1282,7 @@ static long bpf_unwind_fixup(u64 lr)
* The verifier made sure that unwind_goto follows a call in a prog. When
* the kernel called an async callback, it's exit.
*/
- return e ? e->fixup : 0;
+ return e ? e->fixup : bpf_prog_report_unwound(0);
}
/*
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index e417ac4ce749..8b7b330baa61 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2523,7 +2523,7 @@ static u64 bpf_unwind_target(u64 ret_addr)
* The verifier made sure that unwind_goto follows a call in a prog. When
* the kernel called an async callback, it's exit.
*/
- return e ? ret_addr + e->fixup : ret_addr;
+ return e ? ret_addr + e->fixup : bpf_prog_report_unwound(ret_addr);
}
/*
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 3e4c317e0c7a..1b1740d8ca8d 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1936,6 +1936,7 @@ struct bpf_prog_aux {
bool has_prologue;
bool might_sleep;
bool can_unwind;
+ bool unwound; /* main prog: see bpf_prog_info */
bool kprobe_write_ctx;
/* has insns that work with the arena base, see bpf_prog_has_arena_insns() */
bool uses_arena_insns;
@@ -3463,6 +3464,7 @@ void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr);
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr);
void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip);
void bpf_prog_begin_run(const struct bpf_prog *prog);
+u64 bpf_prog_report_unwound(u64 ret);
static __always_inline u32
bpf_prog_run_array_sleepable(const struct bpf_prog_array *array,
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 806282c39c57..5c5851cb048c 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1417,6 +1417,7 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
int mark_chain_precision(struct bpf_verifier_env *env, int regno);
int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
+int bpf_insert_unwound_reports(struct bpf_verifier_env *env);
int bpf_split_cur_state(struct bpf_verifier_env *env);
int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 8d7228834305..b0fccdf8c51f 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -6837,7 +6837,12 @@ struct bpf_prog_info {
char name[BPF_OBJ_NAME_LEN];
__u32 ifindex;
__u32 gpl_compatible:1;
- __u32 :31; /* alignment pad */
+ /*
+ * A run of the prog was cut short: it unwound all the way, or a loop or
+ * a recursion went on for too long. See stderr stream of the prog.
+ */
+ __u32 unwound:1;
+ __u32 :30; /* alignment pad */
__u64 netns_dev;
__u64 netns_ino;
__u32 nr_jited_ksyms;
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 21f85c800f0c..bb0aeda9f9cc 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -3632,6 +3632,7 @@ static noinline void bpf_prog_report_may_goto_violation(void)
prog = bpf_prog_find_from_stack();
if (!prog)
return;
+ WRITE_ONCE(prog->aux->unwound, true);
bpf_stream_stage(ss, prog, BPF_STDERR, ({
bpf_stream_printk(ss, "ERROR: Timeout detected for may_goto instruction\n");
bpf_stream_dump_stack(ss);
@@ -3639,6 +3640,28 @@ static noinline void bpf_prog_report_may_goto_violation(void)
#endif
}
+/*
+ * Nothing caught the unwinding. The prog or the async callback returns to the kernel.
+ * There is no return value that is right for every hook, so what to do about it is up
+ * to user space. Returns @ret: the caller has its return value there.
+ */
+u64 bpf_prog_report_unwound(u64 ret)
+{
+#ifdef CONFIG_BPF_SYSCALL
+ struct bpf_stream_stage ss;
+ struct bpf_prog *prog;
+
+ prog = bpf_prog_find_from_stack();
+ if (!prog)
+ return ret;
+ WRITE_ONCE(prog->aux->unwound, true);
+ bpf_stream_stage(ss, prog, BPF_STDERR, ({
+ bpf_stream_printk(ss, "ERROR: The program was unwound\n");
+ }));
+#endif
+ return ret;
+}
+
/*
* The kernel is about to call an async callback of @prog on this cpu: a run begins.
* The main prog does the same in its first insns.
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index c37325af659a..6a591c954c31 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -431,6 +431,33 @@ static int adjust_jmp_off(struct bpf_prog *prog, u32 tgt_idx, u32 delta)
return 0;
}
+/*
+ * Say so when the main prog unwinds. JITs make exit of that, see bpf_prog_can_unwind().
+ * It's done after bpf_do_misc_fixups(), that takes a call for a call of a helper.
+ */
+int bpf_insert_unwound_reports(struct bpf_verifier_env *env)
+{
+ struct bpf_prog *new_prog;
+ struct bpf_insn buf[3];
+ int i;
+
+ if (env->prog->type == BPF_PROG_TYPE_EXT)
+ return 0;
+ for (i = 0; i < env->subprog_info[1].start; i++) {
+ if (!bpf_is_unwind_exit(&env->prog->insnsi[i]))
+ continue;
+ buf[0] = BPF_MOV64_REG(BPF_REG_1, BPF_REG_0);
+ buf[1] = BPF_EMIT_CALL(bpf_prog_report_unwound);
+ buf[2] = env->prog->insnsi[i];
+ new_prog = bpf_patch_insn_data(env, i, buf, 3);
+ if (!new_prog)
+ return -ENOMEM;
+ env->prog = new_prog;
+ i += 2;
+ }
+ return 0;
+}
+
static int adjust_subprog_starts_after_remove(struct bpf_verifier_env *env,
u32 off, u32 cnt)
{
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 60c4d84b9aba..9ba53a3db422 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -5188,6 +5188,7 @@ static int bpf_prog_get_info_by_fd(struct file *file,
info.created_by_uid = from_kuid_munged(current_user_ns(),
prog->aux->user->uid);
info.gpl_compatible = prog->gpl_compatible;
+ info.unwound = READ_ONCE(prog->aux->unwound);
memcpy(info.tag, prog->tag, sizeof(prog->tag));
memcpy(info.name, prog->aux->name, sizeof(prog->aux->name));
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e179514c3b59..f065e7e829c6 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -24348,6 +24348,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret == 0)
ret = bpf_do_misc_fixups(env);
+ if (ret == 0)
+ ret = bpf_insert_unwound_reports(env);
+
if (ret == 0)
ret = bpf_insert_arena_stack(env);
diff --git a/tools/bpf/bpftool/prog.c b/tools/bpf/bpftool/prog.c
index 43599c79c93a..da39d2f0ab2d 100644
--- a/tools/bpf/bpftool/prog.c
+++ b/tools/bpf/bpftool/prog.c
@@ -435,6 +435,7 @@ static void print_prog_header_json(struct bpf_prog_info *info, int fd)
info->tag[4], info->tag[5], info->tag[6], info->tag[7]);
jsonw_bool_field(json_wtr, "gpl_compatible", info->gpl_compatible);
+ jsonw_bool_field(json_wtr, "unwound", info->unwound);
if (info->run_time_ns) {
jsonw_uint_field(json_wtr, "run_time_ns", info->run_time_ns);
jsonw_uint_field(json_wtr, "run_cnt", info->run_cnt);
@@ -521,6 +522,7 @@ static void print_prog_header_plain(struct bpf_prog_info *info, int fd)
fprint_hex(stdout, info->tag, BPF_TAG_SIZE, "");
print_dev_plain(info->ifindex, info->netns_dev, info->netns_ino);
printf("%s", info->gpl_compatible ? " gpl" : "");
+ printf("%s", info->unwound ? " unwound" : "");
if (info->run_time_ns)
printf(" run_time_ns %llu run_cnt %llu",
info->run_time_ns, info->run_cnt);
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 8d7228834305..b0fccdf8c51f 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -6837,7 +6837,12 @@ struct bpf_prog_info {
char name[BPF_OBJ_NAME_LEN];
__u32 ifindex;
__u32 gpl_compatible:1;
- __u32 :31; /* alignment pad */
+ /*
+ * A run of the prog was cut short: it unwound all the way, or a loop or
+ * a recursion went on for too long. See stderr stream of the prog.
+ */
+ __u32 unwound:1;
+ __u32 :30; /* alignment pad */
__u64 netns_dev;
__u64 netns_ino;
__u32 nr_jited_ksyms;
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 43/81] bpf: Tell user space that a run of a prog was cut short
2026-10-26 2:59 ` [PATCH bpf 43/81] bpf: Tell user space that a run of a prog was cut short Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 7743 bytes --]
> bpf: Tell user space that a run of a prog was cut short
TL;DR:
- Possible build failure with CONFIG_BPF_JIT=y and BPF_SYSCALL=n:
bpf_prog_report_unwound lacks a prototype
- The unwound bit may stay unset on x86 without ORC unwinder
- Unwinds in exception callbacks and epilogue-generated progs may go
unreported
> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> --- a/include/linux/bpf.h
> +++ b/include/linux/bpf.h
[ ... ]
> @@ -3463,6 +3464,7 @@ void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr);
> void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr);
> void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip);
> void bpf_prog_begin_run(const struct bpf_prog *prog);
> +u64 bpf_prog_report_unwound(u64 ret);
This prototype sits inside the large '#ifdef CONFIG_BPF_SYSCALL' block, and
there is no stub in the '#else /* !CONFIG_BPF_SYSCALL */' branch. Both JITs
call the function unconditionally:
arch/x86/net/bpf_jit_comp.c:bpf_unwind_target()
return e ? ret_addr + e->fixup : bpf_prog_report_unwound(ret_addr);
arch/arm64/net/bpf_jit_comp.c:bpf_unwind_fixup()
return e ? e->fixup : bpf_prog_report_unwound(0);
Those files are built for CONFIG_BPF_JIT, which does not depend on
CONFIG_BPF_SYSCALL. Doesn't an x86_64 or arm64 config with CONFIG_NET=y,
CONFIG_BPF_JIT=y and CONFIG_BPF_SYSCALL=n (cBPF JIT only) now fail with
'implicit declaration of function bpf_prog_report_unwound'?
kernel/bpf/core.c is obj-y and defines the function unconditionally, so with
CONFIG_BPF_SYSCALL=n it also has no prototype in scope, which
-Wmissing-prototypes would flag.
A static inline stub in the #else branch would clash with that definition.
Should the prototype move outside the CONFIG_BPF_SYSCALL block, for example
next to bpf_jit_bypass_spec_v4()?
I don't see a later patch in the series that changes this.
[ ... ]
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
[ ... ]
> @@ -3639,6 +3640,28 @@ static noinline void bpf_prog_report_may_goto_violation(void)
> #endif
> }
>
> +/*
> + * Nothing caught the unwinding. The prog or the async callback returns to the kernel.
> + * There is no return value that is right for every hook, so what to do about it is up
> + * to user space. Returns @ret: the caller has its return value there.
> + */
> +u64 bpf_prog_report_unwound(u64 ret)
> +{
> +#ifdef CONFIG_BPF_SYSCALL
> + struct bpf_stream_stage ss;
> + struct bpf_prog *prog;
> +
> + prog = bpf_prog_find_from_stack();
> + if (!prog)
> + return ret;
> + WRITE_ONCE(prog->aux->unwound, true);
Can this early return leave 'unwound' clear on valid x86 configs?
bpf_prog_find_from_stack() relies on arch_bpf_stack_walk(), which on x86 does
nothing unless CONFIG_UNWINDER_ORC is set:
arch/x86/net/bpf_jit_comp.c:arch_bpf_stack_walk() {
#if defined(CONFIG_UNWINDER_ORC)
...
#endif
}
On x86, bpf_jit_supports_unwind() and bpf_jit_supports_timed_may_goto()
return true unconditionally, and check_unwind_insns() only needs
bpf_jit_supports_unwind(). So with CONFIG_UNWINDER_FRAME_POINTER=y, progs
with unwind_exit still load and run, but every report path returns early:
this function for the main prog, the async callback path in
bpf_unwind_target(), and bpf_prog_report_may_goto_violation().
bpf_prog_info.unwound stays 0 and stderr stays empty.
Other users of the stack walk are gated on bpf_jit_supports_exceptions(),
which is IS_ENABLED(CONFIG_UNWINDER_ORC) on x86. For example, the timed
may_goto budget setup in bpf_do_misc_fixups() checks it:
if (!subprogs[i].stack_extra || !bpf_jit_supports_timed_may_goto() ||
!bpf_jit_supports_percpu_insn() || !bpf_jit_supports_exceptions() ||
...
For the main prog the verifier already knows which prog it is.
bpf_insert_arena_stack() passes aux directly with
BPF_LD_IMM64(BPF_REG_1, (long)aux). Could the main prog report do the same
and avoid the stack walk?
> + bpf_stream_stage(ss, prog, BPF_STDERR, ({
> + bpf_stream_printk(ss, "ERROR: The program was unwound\n");
> + }));
> +#endif
> + return ret;
> +}
[ ... ]
> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c
[ ... ]
> +/*
> + * Say so when the main prog unwinds. JITs make exit of that, see bpf_prog_can_unwind().
> + * It's done after bpf_do_misc_fixups(), that takes a call for a call of a helper.
> + */
> +int bpf_insert_unwound_reports(struct bpf_verifier_env *env)
> +{
> + struct bpf_prog *new_prog;
> + struct bpf_insn buf[3];
> + int i;
> +
> + if (env->prog->type == BPF_PROG_TYPE_EXT)
> + return 0;
> + for (i = 0; i < env->subprog_info[1].start; i++) {
Is an unwind_exit in an exception callback left unreported here?
The exception callback is subprog >= 1, so this loop over the range up to
subprog_info[1].start never looks at it. The JITs don't report it either:
bpf_prog_can_unwind() returns false when aux->exception_cb is set, so x86
do_jit() and arm64 build_insn() emit a plain exit and bpf_unwind_target() /
bpf_unwind_fixup() never run.
The verifier accepts unwind_exit there, since it is checked at frame 0
through process_bpf_exit_full() -> check_return_code() with no
unwind-specific restriction. bpf_throw() calls aux->bpf_exception_cb(), and
as the comment on bpf_prog_can_unwind() says, 'An exception callback returns
from the frame of the main prog'. So nothing can catch that unwind_exit,
which is the same situation as for the main prog.
The commit message says 'The kernel sets the bit when a run is cut short'
and lists only the main prog and async callbacks. With a panic in an
exception callback, the hook gets an arbitrary r0, bpf_prog_info.unwound
stays 0, and stderr is empty. Should this case be covered, or called out in
the commit message?
> + if (!bpf_is_unwind_exit(&env->prog->insnsi[i]))
> + continue;
Can this test miss the main prog's unwind_exit when an epilogue is
generated?
This runs after bpf_convert_ctx_accesses(), and that pass can already have
replaced the main prog's exit insns, unwind_exit included, with a generated
epilogue. The check there looks only at the code, not src_reg:
kernel/bpf/fixups.c:bpf_convert_ctx_accesses() {
} else if (insn->code == (BPF_JMP | BPF_EXIT) &&
epilogue_cnt &&
i + delta < subprogs[1].start) {
/* Generate epilogue for the main prog */
...
memcpy(insn_buf, epilogue_buf,
epilogue_cnt * sizeof(*epilogue_buf));
For a later exit it emits only a BPF_JMP32_A back to the first epilogue.
The epilogue from bpf_qdisc_gen_epilogue() (net/sched/bpf_qdisc.c), used for
the Qdisc_ops reset and destroy progs, ends with BPF_EXIT_INSN(), so its
src_reg is 0 and bpf_is_unwind_exit() is false. bpf_testmod's
st_ops_gen_epilogue behaves the same way.
The verifier accepts unwind_exit in those progs, since check_unwind_insns()
has no prog-type limit and void return values are not checked. In that case
no 'r1 = r0; call bpf_prog_report_unwound' is inserted, so a run that
unwinds to the kernel never sets bpf_prog_info.unwound and prints nothing to
stderr. The commit message says 'unwind_exit in the main prog: a call that
the verifier adds before it', so is this case expected to be covered?
> + buf[0] = BPF_MOV64_REG(BPF_REG_1, BPF_REG_0);
> + buf[1] = BPF_EMIT_CALL(bpf_prog_report_unwound);
> + buf[2] = env->prog->insnsi[i];
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 44/81] bpf: Mask the count of a 32-bit shift in scalar evolution
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (42 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 43/81] bpf: Tell user space that a run of a prog was cut short Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:52 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 45/81] selftests/bpf: Expect the masked count of 32-bit shifts Alexei Starovoitov
` (38 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Scalar evolution describes 'w1 >>= w2' as (zext32 (>> r1 r2)). The cpu does
something different: a 32-bit shift uses only the low 5 bits of the count.
w2 the cpu shifts by the expression shows
-- ----------------- --------------------
1 1 1
32 0 32
33 1 33
No rule examines the count of a shift today. Thus no prog is affected. But
the first rule that does will calculate an incorrect number of iterations
or an incorrect range.
Describe what the cpu does: (zext32 (>> r1 (& r2 31))). Shifts by an
immediate and 64-bit shifts do not change.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/scev.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index d0ba709694db..46ccce4ab130 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -731,6 +731,14 @@ static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
else
goto mark_dst_unknown;
+ /*
+ * The cpu takes the low 5 bits of the count: 'w1 >>= w2' with 32 in w2
+ * shifts by 0. It must not look like 'r1 >>= 32'.
+ */
+ if (class == BPF_ALU && x_or_k == BPF_X &&
+ (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH))
+ id = expr2(scev, BPF_AND, id, imm_expr(scev, 31));
+
id = expr2(scev, op, reg2expr[dst], id);
if (class == BPF_ALU)
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 44/81] bpf: Mask the count of a 32-bit shift in scalar evolution
2026-10-26 2:59 ` [PATCH bpf 44/81] bpf: Mask the count of a 32-bit shift in scalar evolution Alexei Starovoitov
@ 2026-10-26 3:52 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:52 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 2815 bytes --]
> bpf: Mask the count of a 32-bit shift in scalar evolution
TL;DR: Scalar evolution's 32-bit shift expression still shifts the full 64-bit
dst, so it may not match what the cpu does for w1 >>= w2 or s>>=. No program
is affected today.
> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index d0ba709694db..46ccce4ab130 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c
[ ... ]
> @@ -731,6 +731,14 @@ static int transfer(struct bpf_verifier_env *env, struct env *e, int idx)
> else
> goto mark_dst_unknown;
>
> + /*
> + * The cpu takes the low 5 bits of the count: 'w1 >>= w2' with 32 in w2
> + * shifts by 0. It must not look like 'r1 >>= 32'.
> + */
> + if (class == BPF_ALU && x_or_k == BPF_X &&
> + (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH))
> + id = expr2(scev, BPF_AND, id, imm_expr(scev, 31));
> +
> id = expr2(scev, op, reg2expr[dst], id);
^^^^
Is (zext32 (>> r1 (& r2 31))) really what the cpu does for 'w1 >>= w2' and
'w1 s>>= w2'?
The interpreter in kernel/bpf/core.c truncates the destination before it
shifts:
ALU_##OPCODE##_X:
DST = (u32) DST OP ((u32) SRC & 31);
...
ALU_ARSH_X:
DST = (u64) (u32) (((s32) DST) >> (SRC & 31));
transfer() still uses the full 64-bit reg2expr[dst] as the operand that gets
shifted, and the ZEXT32 applied afterwards only keeps the low 32 bits of the
result. So the upper 32 bits of r1 can shift into the result:
r1 = 0x1_0000_0000, w2 = 1, 'w1 >>= w2'
cpu: 0
expression: 0x8000_0000
r1 = 0x8000_0000, w2 = 1, 'w1 s>>= w2'
cpu: 0xc000_0000
expression: 0x4000_0000
The same gap exists for 32-bit right shifts by an immediate, (zext32 (>> r1
K)), and for 32-bit BPF_DIV/BPF_MOD including the signed variants. BPF_ARSH
would additionally need a sign extension of w1. BPF_LSH is fine, since the
low 32 bits of a left shift depend only on the low 32 bits of the input.
No rule in this commit or later in the series evaluates a 32-bit right
shift (af2fd429b9f6 adds only the 64-bit rule (>> (<< x 32) 32) -> (zext32
x)), so no program is affected today. But a rule that trusts (zext32 (>> a
b)) to describe 'w1 >>= w2' would still be unsound for the dst operand after
this patch.
Should the 32-bit RSH/ARSH description use the 32-bit dst, for example
(zext32 (>> (zext32 r1) (& r2 31))), with a sign-extended operand for
ARSH? Otherwise, should the commit message stop saying that the expression
describes what the cpu does?
>
> if (class == BPF_ALU)
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 45/81] selftests/bpf: Expect the masked count of 32-bit shifts
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (43 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 44/81] bpf: Mask the count of a 32-bit shift in scalar evolution Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 46/81] bpf: Finish the insn in the state that maybe_fork_scalars() splits off Alexei Starovoitov
` (37 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
alu_ops_exprs examines the expressions that scalar evolution makes for
'w1 <<= w7', 'w1 >>= w7' and 'w1 s>>= w7'. These expressions now contain
the mask.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/progs/verifier_scev.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index 5a7c32ce1c6c..a68a2368aa71 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -362,15 +362,15 @@ __msg("w1 ^= w7 {{.*}}; r1 r6 -> (zext32 (^ r6 r7))")
__msg("r1 <<= 2 {{.*}}; r1 r6 -> (<< r6 2)")
__msg("r1 <<= r7 {{.*}}; r1 r6 -> (<< r6 r7)")
__msg("w1 <<= 2 {{.*}}; r1 r6 -> (zext32 (<< r6 2))")
-__msg("w1 <<= w7 {{.*}}; r1 r6 -> (zext32 (<< r6 r7))")
+__msg("w1 <<= w7 {{.*}}; r1 r6 -> (zext32 (<< r6 (& r7 31)))")
__msg("r1 >>= 2 {{.*}}; r1 r6 -> (>> r6 2)")
__msg("r1 >>= r7 {{.*}}; r1 r6 -> (>> r6 r7)")
__msg("w1 >>= 2 {{.*}}; r1 r6 -> (zext32 (>> r6 2))")
-__msg("w1 >>= w7 {{.*}}; r1 r6 -> (zext32 (>> r6 r7))")
+__msg("w1 >>= w7 {{.*}}; r1 r6 -> (zext32 (>> r6 (& r7 31)))")
__msg("r1 s>>= 2 {{.*}}; r1 r6 -> (s>> r6 2)")
__msg("r1 s>>= r7 {{.*}}; r1 r6 -> (s>> r6 r7)")
__msg("w1 s>>= 2 {{.*}}; r1 r6 -> (zext32 (s>> r6 2))")
-__msg("w1 s>>= w7 {{.*}}; r1 r6 -> (zext32 (s>> r6 r7))")
+__msg("w1 s>>= w7 {{.*}}; r1 r6 -> (zext32 (s>> r6 (& r7 31)))")
__msg("r1 = -r1 {{.*}}; r1 r6 -> (- r6)")
__msg("w1 = -w1 {{.*}}; r1 r6 -> (zext32 (- r6))")
__msg("r1 = 42 {{.*}}; r1 r6 -> 42")
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 46/81] bpf: Finish the insn in the state that maybe_fork_scalars() splits off
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (44 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 45/81] selftests/bpf: Expect the masked count of 32-bit shifts Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 47/81] bpf: Add join of verifier states Alexei Starovoitov
` (36 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
When a register is -1 or 0, maybe_fork_scalars() splits the state in two
before an 'and' or an 'or' with a constant. The result is more accurate for
each value alone. The verifier continues with -1. It pushes a copy of the
state with 0 and verifies the copy later, from the same insn.
today this patch
N: r1 &= 8 r1 is -1 or 0 N: r1 &= 8 r1 is -1 or 0
/ \ / \
r1 = -1 r1 = 0 r1 = -1 r1 = 0
r1 &= 8 push to N r1 &= 8 r1 &= 8
| | push to N+1
N+1: ... N+1: ...
At insn N the copy is a part of the state that it came from. If pruning
compared the two states, it would drop the copy, and no one would verify
the case r1 = 0. Today that does not occur for one reason only: the first
state has branches that are not explored, and pruning does not use such
states.
Do not depend on that. Do the insn in the copy too, and push the copy to
the next insn. All other code that pushes a state does the same.
No verdict changes in selftests, Cilium and scx. Each fork processes one
insn less, and the sequence in which the verifier explores states changes
slightly. 68 of 144 progs of Cilium process a different number of insns,
from -4.8% to +3.3%. The total is 1151838 -> 1151493.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/verifier.c | 27 ++++++++++++++++++---------
1 file changed, 18 insertions(+), 9 deletions(-)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index f065e7e829c6..b44a5c63a922 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -17372,11 +17372,14 @@ static bool is_safe_to_compute_dst_reg_range(struct bpf_insn *insn,
}
}
+static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn *insn,
+ struct bpf_reg_state *dst_reg, struct bpf_reg_state src_reg);
+
static int maybe_fork_scalars(struct bpf_verifier_env *env, struct bpf_insn *insn,
- struct bpf_reg_state *dst_reg)
+ struct bpf_reg_state *dst_reg, const struct bpf_reg_state *src_reg)
{
struct bpf_verifier_state *branch;
- struct bpf_reg_state *regs;
+ struct bpf_reg_state *reg;
bool alu32;
if (reg_smin(dst_reg) == -1 && reg_smax(dst_reg) == 0)
@@ -17386,19 +17389,25 @@ static int maybe_fork_scalars(struct bpf_verifier_env *env, struct bpf_insn *ins
else
return 0;
- branch = push_stack(env, env->insn_idx, env->insn_idx, false);
+ branch = push_stack(env, env->insn_idx + 1, env->insn_idx, false);
if (IS_ERR(branch))
return PTR_ERR(branch);
- regs = branch->frame[branch->curframe]->regs;
+ reg = &branch->frame[branch->curframe]->regs[insn->dst_reg];
if (alu32) {
- __mark_reg32_known(®s[insn->dst_reg], 0);
+ __mark_reg32_known(reg, 0);
__mark_reg32_known(dst_reg, -1ull);
} else {
- __mark_reg_known(®s[insn->dst_reg], 0);
+ __mark_reg_known(reg, 0);
__mark_reg_known(dst_reg, -1ull);
}
- return 0;
+ /*
+ * The insn is done with in the branch too, and what gets to the next insn is a state
+ * like any other. At this insn the branch is a part of the state that is looked at:
+ * it must not be compared with it.
+ */
+ clear_scalar_id(reg);
+ return adjust_scalar_min_max_vals(env, insn, reg, *src_reg);
}
/* WARNING: This function does calculations on 64-bit values, but the actual
@@ -17503,7 +17512,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
break;
case BPF_AND:
if (tnum_is_const(src_reg.var_off)) {
- ret = maybe_fork_scalars(env, insn, dst_reg);
+ ret = maybe_fork_scalars(env, insn, dst_reg, &src_reg);
if (ret)
return ret;
}
@@ -17514,7 +17523,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
break;
case BPF_OR:
if (tnum_is_const(src_reg.var_off)) {
- ret = maybe_fork_scalars(env, insn, dst_reg);
+ ret = maybe_fork_scalars(env, insn, dst_reg, &src_reg);
if (ret)
return ret;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 46/81] bpf: Finish the insn in the state that maybe_fork_scalars() splits off
2026-10-26 2:59 ` [PATCH bpf 46/81] bpf: Finish the insn in the state that maybe_fork_scalars() splits off Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 8506 bytes --]
> bpf: Finish the insn in the state that maybe_fork_scalars() splits off
TL;DR:
- Changelog may understate: pruning a forked r1=0 state in a terminating loop
looks reachable, so the verifier could accept unsafe programs; maybe needs
Fixes: tag
- Forked path loses its verifier log record of the insn (log-only regression)
> bpf: Finish the insn in the state that maybe_fork_scalars() splits off
>
> When a register is -1 or 0, maybe_fork_scalars() splits the state in two
> before an 'and' or an 'or' with a constant. The result is more accurate for
> each value alone. The verifier continues with -1. It pushes a copy of the
> state with 0 and verifies the copy later, from the same insn.
>
> today this patch
>
> N: r1 &= 8 r1 is -1 or 0 N: r1 &= 8 r1 is -1 or 0
> / \ / \
> r1 = -1 r1 = 0 r1 = -1 r1 = 0
> r1 &= 8 push to N r1 &= 8 r1 &= 8
> | | push to N+1
> N+1: ... N+1: ...
>
> At insn N the copy is a part of the state that it came from. If pruning
> compared the two states, it would drop the copy, and no one would verify
> the case r1 = 0. Today that does not occur for one reason only: the first
> state has branches that are not explored, and pruning does not use such
> states.
>
> Do not depend on that. Do the insn in the copy too, and push the copy to
> the next insn. All other code that pushes a state does the same.
>
> No verdict changes in selftests, Cilium and scx. Each fork processes one
> insn less, and the sequence in which the verifier explores states changes
> slightly. 68 of 144 progs of Cilium process a different number of insns,
> from -4.8% to +3.3%. The total is 1151838 -> 1151493.
>
> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
The changelog says the copy is not dropped today because "the first state has
branches that are not explored, and pruning does not use such states". Is
that accurate for the parent commit?
bpf_is_state_visited() can prune against a stored state that still has
branches > 0, when both states are in the same loop and SCEV has proved
that loop terminates:
kernel/bpf/states.c:bpf_is_state_visited() {
...
old_loop = current_loop(&sl->state);
cur_loop = current_loop(cur);
if (old_loop && cur_loop &&
old_loop->terminates && cur_loop->terminates &&
old_loop->entry_state == cur_loop->entry_state) {
if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
loop = true;
goto hit;
}
...
}
The sequence below needs a loop for which bpf_compute_loop_iters() returned 1,
so that handle_loop_entry_exit() set entry->terminates and
entry->entry_state:
(1) cur reaches a prune point N inside the loop body that holds 'r1 |= 8',
with r1 in [-1, 0]. bpf_is_state_visited() adds a checkpoint S at N
through bpf_split_cur_state(), so S->branches == 1.
(2) maybe_fork_scalars() pushes the copy B at insn N, so S->branches == 2.
(3) cur, with r1 = -1, takes 'if r1 s< 0 goto exit' and ends.
bpf_update_branch_counts() brings S->branches down to 1.
(4) B is popped at N. handle_loop_entry_exit() changes nothing, because N is
not the loop header. bpf_is_state_visited() then finds S at N with
branches == 1. current_loop() of S and of B is the same terminating loop
with the same entry_state, because B is a copy. states_equal(S, B,
RANGE_WITHIN) holds, because B equals S except that r1 = 0, which is
within [-1, 0], and RANGE_WITHIN compares scalar ranges whatever the
precision.
B is then pruned, and the r1 = 8 path is never verified. That path could be
the out-of-bounds map access of or_scalar_fork_rejects_oob in
progs/verifier_bounds.c, placed inside such a loop.
So in this tree the problem looks reachable, and this patch fixes a real
soundness hole. The other paths that use states with branches > 0
(is_iter_next_insn(), may_goto, bpf_calls_callback()) cannot match an ALU
insn.
Should the changelog describe the bug as reachable? Should it also carry a
Fixes: tag? The candidate would be:
Fixes: 6ccf49f79327 ("bpf: Use SCEV to widen bounded loops")
That commit added pruning against states with branches > 0 in terminating
loops. Before it, c845894ebd6f ("bpf: Fix unsound scalar forking in
maybe_fork_scalars() for BPF_OR") had already made the copy resume at the
same insn.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index f065e7e829c6..b44a5c63a922 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -17386,19 +17389,25 @@ static int maybe_fork_scalars(struct bpf_verifier_env *env, struct bpf_insn *ins
> else
> return 0;
>
> - branch = push_stack(env, env->insn_idx, env->insn_idx, false);
> + branch = push_stack(env, env->insn_idx + 1, env->insn_idx, false);
> if (IS_ERR(branch))
> return PTR_ERR(branch);
>
> - regs = branch->frame[branch->curframe]->regs;
> + reg = &branch->frame[branch->curframe]->regs[insn->dst_reg];
> if (alu32) {
> - __mark_reg32_known(®s[insn->dst_reg], 0);
> + __mark_reg32_known(reg, 0);
> __mark_reg32_known(dst_reg, -1ull);
> } else {
> - __mark_reg_known(®s[insn->dst_reg], 0);
> + __mark_reg_known(reg, 0);
> __mark_reg_known(dst_reg, -1ull);
> }
> - return 0;
> + /*
> + * The insn is done with in the branch too, and what gets to the next insn is a state
> + * like any other. At this insn the branch is a part of the state that is looked at:
> + * it must not be compared with it.
> + */
> + clear_scalar_id(reg);
> + return adjust_scalar_min_max_vals(env, insn, reg, *src_reg);
> }
Does the branch pushed here lose the verifier log record of insn N?
maybe_fork_scalars() runs inside the diagnostic modification scope of
check_alu_op(). push_stack() saves the log position at the fork, which is
before bpf_diag_mod_end() adds the BPF_DIAG_HISTORY_MOD event for insn N, and
that event is added on the current path only:
check_alu_op()
bpf_diag_mod_begin(env, ®s[insn->dst_reg], NULL, BPF_DIAG_MOD_WRITE)
adjust_reg_min_max_vals()
adjust_scalar_min_max_vals()
maybe_fork_scalars()
push_stack()
elem->diag_log_pos = bpf_diag_event_log_save(env)
bpf_diag_mod_end(env) <- event for insn N, current path only
When the branch is popped, pop_stack() calls
bpf_diag_event_log_restore(env, head->diag_log_pos), which cuts the log back
to before that event.
Before this patch the branch resumed at insn N, and check_alu_op() ran again
and recorded an event of its own. For 'r6 |= 8' that was R6 0 -> 8, and for
a 32-bit fork the old and new snapshots also differ. Now the branch resumes
at N+1, and its event log has no change of dst_reg at insn N.
This would show up with log_level >= 1 when that path later fails and the
report follows dst_reg's lineage through diag_build_lineage(). For example:
r6 s>>= 63
r6 |= 8
if r6 s< 0 goto out
r0 = r6
exit
or 'r2 = r6' before a helper call. In such a report the Causal path would
say that r6 last changed at the 's>>= 63' insn, with signed range [-1, 0],
and leave out the insn that made r6 equal to 8.
The new comment says "The insn is done with in the branch too", and the
commit message says "Do the insn in the copy too". Neither holds for this
part of the insn. The same applies to reg_bounds_sanity_check(), which
check_alu_op() runs only on regs[] of the current state.
The cast fork in check_helper_call() shows how to keep the history there:
kernel/bpf/verifier.c:check_helper_call() {
...
bpf_diag_mod_end(env);
...
branch = push_stack(env, env->insn_idx + 1, env->insn_idx, false);
...
bpf_diag_mod_begin(env, ®s[BPF_REG_0], NULL, BPF_DIAG_MOD_WRITE);
...
}
Would calling bpf_diag_mod_end() before push_stack() and bpf_diag_mod_begin()
again after it work here too?
Verdicts are not affected, this is a verifier log regression only. Later in
the series, b3ab9c9b1713 ("bpf: Remove prune points, levels of state
comparison and pop_stack()") removes diag_log_pos and the per-path restore,
which replaces the mechanism this depends on.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 47/81] bpf: Add join of verifier states
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (45 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 46/81] bpf: Finish the insn in the state that maybe_fork_scalars() splits off Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 48/81] bpf: Describe loops that scalar evolution cannot count exactly Alexei Starovoitov
` (35 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier follows each path of a prog. When two paths meet at an insn,
it compares the new state with the states that it saw at this insn before.
If none of them covers the new state (states_equal()), the verifier
verifies the remaining part of the prog again. Thus each 'if' can double
the work.
Add the join of two states: one state that covers the two. A caller that
replaces two states with their join verifies the subsequent code one time.
follow each path join
3: if r0 > 100 goto 5 3: if r0 > 100 goto 5
| \ | \
| 4: r7 += 20 | 4: r7 += 20
r7=[0,7] | r7=[0,7] |
| r7=[20,27] | r7=[20,27]
v v v v
5: r1 += r7 5: r1 += r7
verified 2 times verified 1 time, r7=[0,27]
The rule is to join only numbers: scalars, and the variable offsets of
pointers. All other parts of the two states must be equal. If they are not,
there is no join, and the caller keeps the two states.
state a state b join
------------------------ ------------------------ -------------------
scalar [0,7] scalar [20,27] scalar [0,27]
map value + [0,7] same map value + [20,27] map value + [0,27]
map value + 8 same map value + 16 no join
scalar pointer no join
pointer to map A pointer to map B no join
not initialized scalar or pointer not initialized
a lock or a reference no lock, no reference no join
called from insn 2 called from insn 7 no join
A prog can do nothing with a register that is 'a pointer or a scalar'. Two
constant offsets in one object are two fields, not a range. Only live
registers and stack slots count, thus a dead register cannot prevent a
join.
A join loses information. In the example it loses that r7 is never 8..19. A
join can also leave open a jump that each path decides. Then dead code
stays in the prog.
Registers with the same id have the same value, or point to the same
object. An id in the join represents a pair: the id in state a and the id
in state b. The pairs must be one to one, as in check_ids(). If they are
not, a scalar loses its id, and a pointer or a reference has no join.
At a loop head a range can grow by a small step each time the verifier goes
through the loop. With the argument 'widen' the join moves the bound of a
scalar that grew to the next of these stops:
lower bound: S64_MIN <- S32_MIN <- 0
upper bound: 0 -> S32_MAX -> U32_MAX -> S64_MAX
Thus a range can grow only a few times. The offset of a pointer that grew
becomes unknown. This includes two constant offsets, which have a join in
this mode.
bpf_states_join() examines each result with the comparison that pruning
uses: the join must cover state a and state b. If it does not, the verifier
reports a verifier bug and rejects the prog. Thus an error in join.c does
not cause an unsafe prog to load.
The Makefile does not build join.c in this patch.
"bpf: Verify programs as a fixpoint over the control flow graph" builds it
and calls it.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 10 +
kernel/bpf/join.c | 650 +++++++++++++++++++++++++++++++++++
kernel/bpf/verifier.c | 67 ++++
3 files changed, 727 insertions(+)
create mode 100644 kernel/bpf/join.c
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 5c5851cb048c..8001be4c3626 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -1236,6 +1236,7 @@ struct bpf_verifier_env {
struct bpf_iarray *gotox_tmp_buf;
int *idoms;
struct scev *scev;
+ struct bpf_join_state *join;
};
static inline struct bpf_func_info_aux *subprog_aux(struct bpf_verifier_env *env, int subprog)
@@ -1417,6 +1418,12 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
int mark_chain_precision(struct bpf_verifier_env *env, int regno);
int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
+/* join.c */
+int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst,
+ const struct bpf_verifier_state *a, const struct bpf_verifier_state *b,
+ bool widen);
+void bpf_join_free(struct bpf_verifier_env *env);
+
int bpf_insert_unwound_reports(struct bpf_verifier_env *env);
int bpf_split_cur_state(struct bpf_verifier_env *env);
int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
@@ -1918,6 +1925,9 @@ int bpf_set_reg_range(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
struct cnum64 r64, struct cnum32 r32, u16 base, u16 step);
bool bpf_reg_range_is_empty(struct cnum64 r64, struct cnum32 r32, u16 base, u16 step);
bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_reg_state *b);
+void bpf_reg_union_range(struct bpf_reg_state *acc, const struct bpf_reg_state *src);
+void bpf_reg_forget_range(struct bpf_reg_state *reg);
+void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old);
int bpf_reg_union(struct bpf_verifier_env *env, struct bpf_reg_state *acc,
const struct bpf_reg_state *src);
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
new file mode 100644
index 000000000000..dde40dc4ff4f
--- /dev/null
+++ b/kernel/bpf/join.c
@@ -0,0 +1,650 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Join of verifier states.
+ *
+ * bpf_states_join(dst, a, b) computes the state that describes every concrete
+ * state described by 'a' or by 'b'. With @widen what is larger in the result
+ * than in 'a' is not known at all in the result, so that a sequence of states
+ * at a loop head ends.
+ *
+ * Numbers and offsets of pointers are joined. A pointer and a number, pointers
+ * to different objects, different references and locks are not: there is no
+ * join and the states stay apart.
+ *
+ * The contract is checked on every call: both inputs have to be within the
+ * result, see bpf_state_subsumes(). A violation is a verifier bug.
+ *
+ * Inputs have to be at the same insn and cleaned with
+ * bpf_clean_verifier_state(): dead registers and stack slots are the same on
+ * both sides then.
+ */
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/cnum.h>
+#include <linux/tnum.h>
+#include <linux/filter.h>
+#include <linux/slab.h>
+
+struct join_idpair {
+ u32 a;
+ u32 b;
+ bool bad;
+};
+
+struct bpf_join_state {
+ struct bpf_verifier_env *env;
+ struct join_idpair pairs[BPF_ID_MAP_SIZE];
+ u32 npairs;
+ bool pairs_overflow;
+ bool widen;
+};
+
+static struct bpf_join_state *join_state(struct bpf_verifier_env *env)
+{
+ if (!env->join) {
+ env->join = kvzalloc_obj(*env->join, GFP_KERNEL_ACCOUNT);
+ if (env->join)
+ env->join->env = env;
+ }
+ return env->join;
+}
+
+void bpf_join_free(struct bpf_verifier_env *env)
+{
+ kvfree(env->join);
+ env->join = NULL;
+}
+
+/*
+ * An id in the result stands for the pair (id in 'a', id in 'b'). For the
+ * result to be within both inputs the pairs have to form a bijection,
+ * see check_ids(). The first pass over all locations records the pairs. An id
+ * seen with two partners makes all pairs it takes part in bad.
+ * The second pass keeps the id of 'a' for good pairs.
+ */
+static void jid_mark_bad(struct bpf_join_state *js, u32 a, u32 b)
+{
+ u32 i;
+
+ for (i = 0; i < js->npairs; i++)
+ if ((a && js->pairs[i].a == a) || (b && js->pairs[i].b == b))
+ js->pairs[i].bad = true;
+}
+
+static void jid_note(struct bpf_join_state *js, u32 ida, u32 idb, bool consistent)
+{
+ u32 a = ida & ~BPF_ADD_CONST, b = idb & ~BPF_ADD_CONST;
+ struct join_idpair *p;
+ bool other = false;
+ u32 i;
+
+ if (!a || !b)
+ return;
+ for (i = 0; i < js->npairs; i++) {
+ p = &js->pairs[i];
+ if (p->a == a && p->b == b) {
+ if (!consistent)
+ jid_mark_bad(js, a, b);
+ return;
+ }
+ if (p->a == a || p->b == b)
+ other = true;
+ }
+ if (other) {
+ jid_mark_bad(js, a, b);
+ consistent = false;
+ }
+ if (js->npairs == ARRAY_SIZE(js->pairs)) {
+ js->pairs_overflow = true;
+ return;
+ }
+ p = &js->pairs[js->npairs++];
+ p->a = a;
+ p->b = b;
+ p->bad = !consistent;
+}
+
+static bool jid_good(struct bpf_join_state *js, u32 ida, u32 idb)
+{
+ u32 a = ida & ~BPF_ADD_CONST, b = idb & ~BPF_ADD_CONST;
+ u32 i;
+
+ if (!a || !b || js->pairs_overflow)
+ return false;
+ if ((ida ^ idb) & BPF_ADD_CONST)
+ return false;
+ for (i = 0; i < js->npairs; i++)
+ if (js->pairs[i].a == a && js->pairs[i].b == b)
+ return !js->pairs[i].bad;
+ return false;
+}
+
+/* Ids that have to match: both zero, or a good pair */
+static bool jid_strict_ok(struct bpf_join_state *js, u32 ida, u32 idb)
+{
+ if (!ida && !idb)
+ return true;
+ return jid_good(js, ida, idb);
+}
+
+static bool range_equal(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+ return ra->r64.base == rb->r64.base && ra->r64.size == rb->r64.size &&
+ ra->r32.base == rb->r32.base && ra->r32.size == rb->r32.size &&
+ ra->var_off.value == rb->var_off.value && ra->var_off.mask == rb->var_off.mask &&
+ ra->base == rb->base && ra->step == rb->step;
+}
+
+/*
+ * Join of the range of two registers of the same type, numbers or pointers
+ * with variable offset. @dst is a copy of @ra.
+ */
+static void join_range(struct bpf_join_state *js, struct bpf_reg_state *dst,
+ const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+ if (range_equal(ra, rb))
+ return;
+ bpf_reg_union_range(dst, rb);
+ if (!js->widen || range_equal(dst, ra))
+ return;
+ if (dst->type == SCALAR_VALUE)
+ bpf_reg_widen_range(dst, ra);
+ else
+ bpf_reg_forget_range(dst);
+}
+
+static bool reg_fixed_part_equal(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+ return !memcmp(ra, rb, offsetof(struct bpf_reg_state, var_off));
+}
+
+static bool reg_value_equal(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+ return !memcmp(ra, rb, offsetof(struct bpf_reg_state, id));
+}
+
+static bool scalar_link_consistent(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+ if ((ra->id ^ rb->id) & BPF_ADD_CONST)
+ return false;
+ if ((ra->id & BPF_ADD_CONST) && ra->delta != rb->delta)
+ return false;
+ return true;
+}
+
+/*
+ * Join two registers, regsafe() documents what each type needs.
+ * Pass 1 records id pairs, pass 2 computes @dst, which is a copy of @ra on entry.
+ * Returns false if there is no join.
+ */
+static bool join_reg(struct bpf_join_state *js, struct bpf_reg_state *dst,
+ const struct bpf_reg_state *ra, const struct bpf_reg_state *rb, int pass)
+{
+ if (ra->type == NOT_INIT || rb->type == NOT_INIT) {
+ if (pass == 2 && ra->type != NOT_INIT)
+ *dst = *rb;
+ return true;
+ }
+
+ if (ra->type != rb->type)
+ return false;
+
+ switch (base_type(ra->type)) {
+ case SCALAR_VALUE:
+ if (js->env->explore_alu_limits && !reg_value_equal(ra, rb))
+ return false;
+ if (pass == 1) {
+ jid_note(js, ra->id, rb->id, scalar_link_consistent(ra, rb));
+ return true;
+ }
+ join_range(js, dst, ra, rb);
+ /* a link that exists on one side only is dropped */
+ if (ra->id && !jid_good(js, ra->id, rb->id)) {
+ dst->id = 0;
+ dst->delta = 0;
+ }
+ return true;
+ case PTR_TO_MAP_KEY:
+ case PTR_TO_MAP_VALUE:
+ case PTR_TO_MEM:
+ case PTR_TO_BUF:
+ case PTR_TO_TP_BUFFER:
+ if (!reg_fixed_part_equal(ra, rb))
+ return false;
+ /*
+ * Two places in an object are two pointers: what is at a place
+ * and how much is left behind it is lost in a range of them.
+ */
+ if (!js->widen && tnum_is_const(ra->var_off) && tnum_is_const(rb->var_off) &&
+ ra->var_off.value != rb->var_off.value)
+ return false;
+ break;
+ case PTR_TO_PACKET_META:
+ case PTR_TO_PACKET:
+ if ((ra->range < 0 || rb->range < 0) && ra->range != rb->range)
+ return false;
+ if (!js->widen && !ra->id && !rb->id && ra->range >= 0 &&
+ tnum_is_const(ra->var_off) && tnum_is_const(rb->var_off) &&
+ ra->var_off.value != rb->var_off.value &&
+ ra->var_off.value <= MAX_PACKET_OFF && rb->var_off.value <= MAX_PACKET_OFF)
+ return false;
+ /* how much of the packet was checked goes with numbers elsewhere in the state */
+ if (!js->widen && ra->range != rb->range)
+ return false;
+ /* displacements between pointers that share an id are kept */
+ if ((ra->id || rb->id) && ra->r64.base != rb->r64.base)
+ return false;
+ if (pass == 1) {
+ jid_note(js, ra->id, rb->id, true);
+ return true;
+ }
+ if (!jid_strict_ok(js, ra->id, rb->id))
+ return false;
+ if (rb->range != ra->range)
+ dst->range = min(ra->range, rb->range);
+ join_range(js, dst, ra, rb);
+ return true;
+ case PTR_TO_ARENA:
+ /* The offset is not tracked. */
+ return true;
+ case PTR_TO_INSN:
+ if (!reg_fixed_part_equal(ra, rb))
+ return false;
+ if (pass == 2)
+ join_range(js, dst, ra, rb);
+ return true;
+ default:
+ if (!reg_value_equal(ra, rb))
+ return false;
+ break;
+ }
+
+ if (pass == 1) {
+ jid_note(js, ra->id, rb->id, true);
+ jid_note(js, ra->parent_id, rb->parent_id, true);
+ jid_note(js, ra->map_uid, rb->map_uid, true);
+ return true;
+ }
+ if (!jid_strict_ok(js, ra->id, rb->id) ||
+ !jid_strict_ok(js, ra->parent_id, rb->parent_id) ||
+ !jid_strict_ok(js, ra->map_uid, rb->map_uid))
+ return false;
+ join_range(js, dst, ra, rb);
+ return true;
+}
+
+static bool slot_is_special(const struct bpf_stack_state *s)
+{
+ u8 t = s->slot_type[BPF_REG_SIZE - 1];
+
+ return t == STACK_DYNPTR || t == STACK_ITER || t == STACK_IRQ_FLAG;
+}
+
+static bool stype_is_invalid(u8 t)
+{
+ return t == STACK_INVALID || t == STACK_POISON;
+}
+
+static bool slot_has_spill(const struct bpf_stack_state *s)
+{
+ int i;
+
+ for (i = 0; i < BPF_REG_SIZE; i++)
+ if (s->slot_type[i] == STACK_SPILL)
+ return true;
+ return false;
+}
+
+static bool slot_is_full_spill(const struct bpf_stack_state *s)
+{
+ int i;
+
+ for (i = 0; i < BPF_REG_SIZE; i++)
+ if (s->slot_type[i] != STACK_SPILL)
+ return false;
+ return true;
+}
+
+/* All bytes are initialized and none of them is a part of a spill */
+static bool slot_is_data(const struct bpf_stack_state *s)
+{
+ int i;
+
+ for (i = 0; i < BPF_REG_SIZE; i++)
+ if (s->slot_type[i] != STACK_MISC && s->slot_type[i] != STACK_ZERO)
+ return false;
+ return true;
+}
+
+/* what a byte that is not initialized in one of the states becomes */
+static u8 stype_uninit(struct bpf_join_state *js)
+{
+ return js->env->allow_uninit_stack ? STACK_INVALID : STACK_POISON;
+}
+
+static void slot_poison(struct bpf_join_state *js, struct bpf_stack_state *ds)
+{
+ int i;
+
+ for (i = 0; i < BPF_REG_SIZE; i++)
+ if (ds->slot_type[i] != STACK_POISON)
+ ds->slot_type[i] = stype_uninit(js);
+ bpf_mark_reg_not_init(js->env, &ds->spilled_ptr);
+}
+
+static bool join_special_slot(struct bpf_join_state *js, const struct bpf_stack_state *sa,
+ const struct bpf_stack_state *sb, int pass)
+{
+ const struct bpf_reg_state *ra = &sa->spilled_ptr, *rb = &sb->spilled_ptr;
+
+ if (memcmp(sa->slot_type, sb->slot_type, sizeof(sa->slot_type)))
+ return false;
+ switch (sa->slot_type[BPF_REG_SIZE - 1]) {
+ case STACK_DYNPTR:
+ if (ra->dynptr.type != rb->dynptr.type ||
+ ra->dynptr.first_slot != rb->dynptr.first_slot)
+ return false;
+ break;
+ case STACK_ITER:
+ /* iter.depth is not a part of the state, see stacksafe() */
+ if (ra->type != rb->type || ra->iter.btf != rb->iter.btf ||
+ ra->iter.btf_id != rb->iter.btf_id || ra->iter.state != rb->iter.state)
+ return false;
+ break;
+ case STACK_IRQ_FLAG:
+ if (ra->irq.kfunc_class != rb->irq.kfunc_class)
+ return false;
+ break;
+ }
+ if (pass == 1) {
+ jid_note(js, ra->id, rb->id, true);
+ jid_note(js, ra->parent_id, rb->parent_id, true);
+ return true;
+ }
+ return jid_strict_ok(js, ra->id, rb->id) &&
+ jid_strict_ok(js, ra->parent_id, rb->parent_id);
+}
+
+/*
+ * Join of one 8 byte stack slot, @ds is a copy of @sa on entry.
+ * stacksafe() documents which slot is within which.
+ */
+static bool join_slot(struct bpf_join_state *js, struct bpf_stack_state *ds,
+ const struct bpf_stack_state *sa, const struct bpf_stack_state *sb, int pass)
+{
+ static const struct bpf_stack_state invalid_slot;
+ bool a_ptr, b_ptr, had_spill;
+ u8 res[BPF_REG_SIZE];
+ int i;
+
+ if (!sb)
+ sb = &invalid_slot;
+
+ if (slot_is_special(sa) || slot_is_special(sb))
+ return join_special_slot(js, sa, sb, pass);
+
+ if (!memcmp(sa->slot_type, sb->slot_type, sizeof(sa->slot_type)) &&
+ sa->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL)
+ return join_reg(js, &ds->spilled_ptr, &sa->spilled_ptr, &sb->spilled_ptr, pass);
+
+ a_ptr = slot_has_spill(sa) && sa->spilled_ptr.type != SCALAR_VALUE;
+ b_ptr = slot_has_spill(sb) && sb->spilled_ptr.type != SCALAR_VALUE;
+ /* a pointer and something else */
+ if (a_ptr || b_ptr)
+ return false;
+
+ /* scalar spills, MISC, ZERO, INVALID, POISON: no ids are kept */
+ if (pass == 1)
+ return true;
+
+ had_spill = slot_has_spill(sa) || slot_has_spill(sb);
+ for (i = 0; i < BPF_REG_SIZE; i++) {
+ u8 x = sa->slot_type[i], y = sb->slot_type[i];
+
+ if (x == STACK_SPILL)
+ x = STACK_MISC;
+ if (y == STACK_SPILL)
+ y = STACK_MISC;
+ if (stype_is_invalid(x) && stype_is_invalid(y))
+ res[i] = sa->slot_type[i];
+ else if (stype_is_invalid(x) || stype_is_invalid(y))
+ res[i] = stype_uninit(js);
+ else if (x == y)
+ res[i] = x;
+ else
+ res[i] = STACK_MISC;
+ }
+ /*
+ * When reads of uninitialized stack are not allowed stacksafe() accepts
+ * MISC in place of a spilled scalar only for whole slots.
+ */
+ if (had_spill && !js->env->allow_uninit_stack) {
+ bool ok = (slot_is_full_spill(sa) || slot_is_data(sa)) &&
+ (slot_is_full_spill(sb) || slot_is_data(sb));
+
+ for (i = 0; ok && i < BPF_REG_SIZE; i++)
+ ok = res[i] == STACK_MISC;
+ if (!ok) {
+ slot_poison(js, ds);
+ return true;
+ }
+ }
+ memcpy(ds->slot_type, res, sizeof(res));
+ bpf_mark_reg_not_init(js->env, &ds->spilled_ptr);
+ return true;
+}
+
+static bool join_loop_stack(const struct bpf_func_state *fa, const struct bpf_func_state *fb)
+{
+ u32 i;
+
+ if (fa->loop_stack_cnt != fb->loop_stack_cnt)
+ return false;
+ /* what is known about a counted loop comes from the state it was entered with */
+ for (i = 0; i < fa->loop_stack_cnt; i++)
+ if (fa->loop_stack[i].loop_id != fb->loop_stack[i].loop_id ||
+ fa->loop_stack[i].terminates != fb->loop_stack[i].terminates ||
+ fa->loop_stack[i].apart != fb->loop_stack[i].apart ||
+ fa->loop_stack[i].trip != fb->loop_stack[i].trip ||
+ fa->loop_stack[i].max_header_count != fb->loop_stack[i].max_header_count ||
+ fa->loop_stack[i].entry_state != fb->loop_stack[i].entry_state)
+ return false;
+ return true;
+}
+
+static bool join_func(struct bpf_join_state *js, struct bpf_func_state *fd,
+ const struct bpf_func_state *fa, const struct bpf_func_state *fb,
+ u16 live_regs, int pass)
+{
+ static const struct bpf_reg_state not_init = { .type = NOT_INIT };
+ struct bpf_verifier_env *env = js->env;
+ u32 i, nslots_b;
+
+ if (pass == 1) {
+ if (fa->callsite != fb->callsite || fa->frameno != fb->frameno ||
+ fa->subprogno != fb->subprogno)
+ return false;
+ if (fa->in_callback_fn != fb->in_callback_fn ||
+ fa->in_async_callback_fn != fb->in_async_callback_fn ||
+ fa->in_exception_callback_fn != fb->in_exception_callback_fn ||
+ fa->in_rcu_callback_fn != fb->in_rcu_callback_fn ||
+ fa->async_entry_cnt != fb->async_entry_cnt ||
+ memcmp(&fa->callback_ret_range, &fb->callback_ret_range,
+ sizeof(fa->callback_ret_range)))
+ return false;
+ if (!join_loop_stack(fa, fb))
+ return false;
+ }
+
+ for (i = 0; i < MAX_BPF_REG; i++) {
+ if (!(live_regs & BIT(i))) {
+ if (pass == 2 && i < BPF_REG_FP)
+ bpf_mark_reg_not_init(env, &fd->regs[i]);
+ continue;
+ }
+ if (!join_reg(js, &fd->regs[i], &fa->regs[i], &fb->regs[i], pass))
+ return false;
+ }
+
+ nslots_b = bpf_stack_nr_slots(fb);
+ for (i = 0; i < bpf_stack_nr_slots(fa); i++) {
+ const struct bpf_stack_state *sb = i < nslots_b ? bpf_stack_slot(fb, i) : NULL;
+
+ if (!join_slot(js, bpf_stack_slot(fd, i), bpf_stack_slot(fa, i), sb, pass))
+ return false;
+ }
+ /* Slots that exist only in 'b' are STACK_INVALID in the result. */
+ for (i = bpf_stack_nr_slots(fa); pass == 1 && i < nslots_b; i++) {
+ const struct bpf_stack_state *sb = bpf_stack_slot(fb, i);
+
+ if (slot_is_special(sb) ||
+ (slot_has_spill(sb) && sb->spilled_ptr.type != SCALAR_VALUE))
+ return false;
+ }
+
+ if (fa->out_stack_arg_cnt < fb->out_stack_arg_cnt)
+ return false;
+ for (i = 0; i < fa->out_stack_arg_cnt; i++) {
+ const struct bpf_reg_state *rb;
+
+ rb = i < fb->out_stack_arg_cnt ? &fb->stack_arg_regs[i] : ¬_init;
+ if (!join_reg(js, &fd->stack_arg_regs[i], &fa->stack_arg_regs[i], rb, pass))
+ return false;
+ }
+
+ if (pass == 2) {
+ fd->no_stack_arg_load = fa->no_stack_arg_load || fb->no_stack_arg_load;
+ fd->callback_depth = min(fa->callback_depth, fb->callback_depth);
+ }
+ return true;
+}
+
+/* Things that can't be different in two states that are joined */
+static bool join_check_state(const struct bpf_verifier_state *a,
+ const struct bpf_verifier_state *b)
+{
+ int i;
+
+ if (a->curframe != b->curframe || a->insn_idx != b->insn_idx)
+ return false;
+ if (a->speculative != b->speculative || a->in_sleepable != b->in_sleepable ||
+ a->in_callback != b->in_callback || a->callers_unknown != b->callers_unknown ||
+ a->unwound != b->unwound)
+ return false;
+ if (a->active_locks != b->active_locks ||
+ a->active_preempt_locks != b->active_preempt_locks ||
+ a->active_rcu_locks != b->active_rcu_locks ||
+ a->active_lock_ptr != b->active_lock_ptr ||
+ !a->active_irq_id != !b->active_irq_id ||
+ !a->active_lock_id != !b->active_lock_id)
+ return false;
+ if (a->acquired_refs != b->acquired_refs)
+ return false;
+ for (i = 0; i < a->acquired_refs; i++) {
+ if (a->refs[i].type != b->refs[i].type)
+ return false;
+ switch (a->refs[i].type) {
+ case REF_TYPE_PTR:
+ case REF_TYPE_IRQ:
+ break;
+ default:
+ if (a->refs[i].ptr != b->refs[i].ptr)
+ return false;
+ }
+ }
+ return true;
+}
+
+static void join_refs_note(struct bpf_join_state *js, const struct bpf_verifier_state *a,
+ const struct bpf_verifier_state *b)
+{
+ int i;
+
+ jid_note(js, a->active_irq_id, b->active_irq_id, true);
+ jid_note(js, a->active_lock_id, b->active_lock_id, true);
+ for (i = 0; i < a->acquired_refs; i++) {
+ jid_note(js, a->refs[i].id, b->refs[i].id, true);
+ if (a->refs[i].type == REF_TYPE_PTR)
+ jid_note(js, a->refs[i].parent_id, b->refs[i].parent_id, true);
+ }
+}
+
+static bool join_refs_check(struct bpf_join_state *js, const struct bpf_verifier_state *a,
+ const struct bpf_verifier_state *b)
+{
+ int i;
+
+ if (!jid_strict_ok(js, a->active_irq_id, b->active_irq_id) ||
+ !jid_strict_ok(js, a->active_lock_id, b->active_lock_id))
+ return false;
+ for (i = 0; i < a->acquired_refs; i++) {
+ if (!jid_strict_ok(js, a->refs[i].id, b->refs[i].id))
+ return false;
+ if (a->refs[i].type == REF_TYPE_PTR &&
+ !jid_strict_ok(js, a->refs[i].parent_id, b->refs[i].parent_id))
+ return false;
+ }
+ return true;
+}
+
+/* The result has to describe everything that an input describes */
+static int join_check(struct bpf_join_state *js, struct bpf_verifier_state *dst,
+ const struct bpf_verifier_state *in, const char *which)
+{
+ struct bpf_verifier_env *env = js->env;
+
+ if (bpf_state_subsumes(env, dst, (struct bpf_verifier_state *)in))
+ return 0;
+ bpf_log(&env->log, "join:");
+ print_verifier_state(env, dst, dst->curframe, true);
+ bpf_log(&env->log, "input %s:", which);
+ print_verifier_state(env, in, in->curframe, true);
+ verifier_bug(env, "join of states at insn %u does not subsume input '%s'",
+ dst->insn_idx, which);
+ return -EFAULT;
+}
+
+/*
+ * Compute in @dst the state that describes everything that @a or @b describe.
+ * @dst has to be a valid state (zeroed is fine) different from @a and @b, its
+ * previous content is released. Bookkeeping fields are copied from @a.
+ *
+ * Returns 0 on success, 1 if there is no join, negative error otherwise.
+ * When 1 is returned the content of @dst is undefined, but it's a valid state.
+ */
+int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst,
+ const struct bpf_verifier_state *a, const struct bpf_verifier_state *b,
+ bool widen)
+{
+ struct bpf_join_state *js = join_state(env);
+ int pass, fr, err;
+
+ if (!js)
+ return -ENOMEM;
+ js->widen = widen;
+ js->npairs = 0;
+ js->pairs_overflow = false;
+
+ if (!join_check_state(a, b))
+ return 1;
+ err = bpf_copy_verifier_state(dst, a);
+ if (err)
+ return err;
+
+ for (pass = 1; pass <= 2; pass++) {
+ if (pass == 1)
+ join_refs_note(js, a, b);
+ else if (!join_refs_check(js, a, b))
+ return 1;
+ for (fr = 0; fr <= a->curframe; fr++) {
+ u32 insn_idx = bpf_frame_insn_idx((struct bpf_verifier_state *)a, fr);
+ u16 live_regs = env->insn_aux_data[insn_idx].live_regs_before;
+
+ if (!join_func(js, dst->frame[fr], a->frame[fr], b->frame[fr],
+ live_regs, pass))
+ return 1;
+ }
+ }
+
+ return join_check(js, dst, a, "a") ?: join_check(js, dst, b, "b");
+}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index b44a5c63a922..bb42c2707db2 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -17867,6 +17867,73 @@ bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_re
a->id == b->id && a->parent_id == b->parent_id && a->map_uid == b->map_uid;
}
+/* Nothing is known about the number or the variable part of the offset anymore. */
+void bpf_reg_forget_range(struct bpf_reg_state *reg)
+{
+ reg->var_off = tnum_unknown;
+ __mark_reg_unbounded(reg);
+ reg_step_reset(reg);
+}
+
+/*
+ * @reg is what @old became after a trip around a cycle, @old included. The end
+ * of the range that moved is going to move again. Put it where it may come
+ * to rest. There are few such places, so this is done a few times at most.
+ */
+void bpf_reg_widen_range(struct bpf_reg_state *reg, const struct bpf_reg_state *old)
+{
+ s64 min = reg_smin(reg), max = reg_smax(reg);
+
+ if (max > reg_smax(old))
+ max = max <= 0 ? 0 :
+ max <= S32_MAX ? S32_MAX :
+ max <= U32_MAX ? U32_MAX : S64_MAX;
+ if (min < reg_smin(old))
+ min = min >= 0 ? 0 :
+ min >= S32_MIN ? S32_MIN : S64_MIN;
+ if (min == S64_MIN && max == S64_MAX) {
+ bpf_reg_forget_range(reg);
+ return;
+ }
+ /* every part has to include what it did before, hence no reg_bounds_sync() */
+ reg->r64 = cnum64_union(reg->r64, cnum64_from_srange(min, max));
+ if (min >= 0 && max <= U32_MAX) {
+ reg->r32 = cnum32_union(reg->r32, cnum32_from_urange(min, max));
+ reg->var_off = tnum_union(reg->var_off, tnum_range(min, max));
+ } else if (min >= S32_MIN && max <= S32_MAX) {
+ reg->r32 = cnum32_union(reg->r32, cnum32_from_srange(min, max));
+ reg->var_off = tnum_unknown;
+ } else {
+ reg->r32 = CNUM32_UNBOUNDED;
+ reg->var_off = min >= 0 ? tnum_union(reg->var_off, tnum_range(min, max))
+ : tnum_unknown;
+ }
+ reg_step_reset(reg);
+}
+
+void bpf_reg_union_range(struct bpf_reg_state *acc, const struct bpf_reg_state *src)
+{
+ u16 base, step;
+
+ acc->r64 = cnum64_union(acc->r64, src->r64);
+ acc->r32 = cnum32_union(acc->r32, src->r32);
+ acc->var_off = tnum_union(acc->var_off, src->var_off);
+
+ /* Retain a common congruence if the bases agree modulo the gcd. */
+ step = gcd(acc->step, src->step);
+ base = acc->base % step;
+ if (base != src->base % step) {
+ reg_step_reset(acc);
+ } else {
+ acc->base = base;
+ acc->step = step;
+ }
+ /*
+ * No reg_bounds_sync(): every part of the result has to include that
+ * part of both inputs, and an input is not always in sync.
+ */
+}
+
/*
* Extends acc's scalar range to include src's scalar range.
* Caller must ensure that other kinds of metadata is compatible.
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 47/81] bpf: Add join of verifier states
2026-10-26 2:59 ` [PATCH bpf 47/81] bpf: Add join of verifier states Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 12021 bytes --]
> bpf: Add join of verifier states
TL;DR:
- Widened join of packet pointers with ids may fail the subsume check, so
valid programs could be rejected with a verifier bug warning
- Same possible failure in explore_alu_limits mode and stack slot poisoning
- Header comment on @widen is inaccurate
> diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
> new file mode 100644
> index 000000000000..dde40dc4ff4f
> --- /dev/null
> +++ b/kernel/bpf/join.c
> @@ -0,0 +1,650 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Join of verifier states.
> + *
> + * bpf_states_join(dst, a, b) computes the state that describes every concrete
> + * state described by 'a' or by 'b'. With @widen what is larger in the result
> + * than in 'a' is not known at all in the result, so that a sequence of states
> + * at a loop head ends.
Is this description of @widen accurate for scalars?
It holds only for pointers, where join_range() calls
bpf_reg_forget_range(). For scalars, join_range() calls
bpf_reg_widen_range(dst, ra), which moves the bound that grew to the next
stop (0, S32_MAX, U32_MAX, S64_MAX, or S32_MIN/S64_MIN). That keeps a finite
range such as [0, S32_MAX] and does not make the value unknown.
The commit message describes this correctly ("the join moves the bound of a
scalar that grew to the next of these stops"), so the header comment
contradicts both the code and the changelog.
[ ... ]
> +/*
> + * Join of the range of two registers of the same type, numbers or pointers
> + * with variable offset. @dst is a copy of @ra.
> + */
> +static void join_range(struct bpf_join_state *js, struct bpf_reg_state *dst,
> + const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
> +{
> + if (range_equal(ra, rb))
> + return;
> + bpf_reg_union_range(dst, rb);
> + if (!js->widen || range_equal(dst, ra))
> + return;
> + if (dst->type == SCALAR_VALUE)
> + bpf_reg_widen_range(dst, ra);
> + else
> + bpf_reg_forget_range(dst);
> +}
Can this produce a result that does not subsume input 'a' when widen is set
and the registers are PTR_TO_PACKET pointers that carry an id?
join_reg() lets two such pointers through when their r64.base values are
equal, and dst keeps the id of a:
/* displacements between pointers that share an id are kept */
if ((ra->id || rb->id) && ra->r64.base != rb->r64.base)
return false;
...
if (!jid_strict_ok(js, ra->id, rb->id))
return false;
...
join_range(js, dst, ra, rb);
If b is not within a, join_range() then calls bpf_reg_forget_range(dst).
That calls __mark_reg_unbounded(), which sets dst->r64 to CNUM64_UNBOUNDED,
{ .base = 0, .size = U64_MAX }, while dst->id keeps the id of a. regsafe()
for PTR_TO_PACKET rejects that state when the old state is dst:
/* Preserve displacements between pointers sharing an ID. */
if (rold->id && rold->r64.base != rcur->r64.base)
return false;
So when the r64.base of a is non-zero, join_check(js, dst, a) ->
bpf_state_subsumes() fails. bpf_states_join() then calls verifier_bug()
(a WARN_ONCE under CONFIG_DEBUG_KERNEL) and returns -EFAULT, and
flow_merge() aborts the load. The same thing happens to any second pointer
with the same id at a non-zero displacement, e.g. r4 = r3 + 8.
For example, p = data + 14 gives id 0 and r64 [14,14]. Then
p += *(u8 *)p, an unknown addend >= 0, gives a fresh id and r64 [14,269].
One more trip gives another id and r64 [14,524]. The bases are both 14, the
ids form a good pair, and the ranges differ.
At the end of the series, flow_merge() (kernel/bpf/flow.c) passes widen=true
at a loop head (cycle == FLOW_CYCLE), and also at any insn once
s->joins > BPF_FLOW_MAX_JOINS (16). bpf_flow_exit() passes widen=true for
recursive subprogs. A valid program is then rejected with a "verifier bug:
join of states ... does not subsume input" splat, and the comment
"displacements between pointers that share an id are kept" does not hold in
widen mode.
Neither join.c nor bpf_reg_forget_range() changes later in the series
(through 51224d16c11b) in a way that fixes this.
Should bpf_reg_forget_range() keep r64.base for a packet pointer that has an
id (a full arc may use any base), or should join_reg() return 'no join' for
that case?
[ ... ]
> +static bool join_reg(struct bpf_join_state *js, struct bpf_reg_state *dst,
> + const struct bpf_reg_state *ra, const struct bpf_reg_state *rb, int pass)
> +{
[ ... ]
> + switch (base_type(ra->type)) {
> + case SCALAR_VALUE:
> + if (js->env->explore_alu_limits && !reg_value_equal(ra, rb))
> + return false;
> + if (pass == 1) {
> + jid_note(js, ra->id, rb->id, scalar_link_consistent(ra, rb));
> + return true;
> + }
> + join_range(js, dst, ra, rb);
> + /* a link that exists on one side only is dropped */
> + if (ra->id && !jid_good(js, ra->id, rb->id)) {
> + dst->id = 0;
> + dst->delta = 0;
> + }
> + return true;
When env->explore_alu_limits is set, can clearing dst->delta here give a
result that regsafe() does not accept?
In that mode two scalars are joined only if reg_value_equal(ra, rb) holds.
That is a memcmp() up to 'id', which also covers 'delta', so
ra->delta == rb->delta. If the link is then dropped because the id pair is
bad, dst->delta is set to 0. regsafe() in explore_alu_limits mode compares
the same bytes:
if (env->explore_alu_limits) {
return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
check_scalar_ids(rold->id, rcur->id, idmap);
A scalar with a BPF_ADD_CONST link has a non-zero delta, e.g.
r7 = r6; r7 += 1. If its pair is marked bad, dst->delta (0) no longer
equals a->delta or b->delta. The pair goes bad when the same id has two
partners. For example:
a: r6.id=N, r7.id=N|BPF_ADD_CONST64 with delta=1, r8.id=N
b: r6.id=M, r7.id=M|BPF_ADD_CONST64 with delta=1, r8.id=K
with all three being unknown scalars with the same bounds. Then
join_check(js, dst, a) -> bpf_state_subsumes() -> regsafe() fails, and
bpf_states_join() reports verifier_bug() (WARN_ONCE under
CONFIG_DEBUG_KERNEL) and returns -EFAULT.
explore_alu_limits is set by sanitize_ptr_alu() for any pointer arithmetic
with a non-constant scalar when the Spectre v1 mitigation applies, i.e. for a
loader without bypass_spec_v1. At the end of the series flow_merge() and
bpf_flow_exit() call bpf_states_join() whatever explore_alu_limits is, and no
fix was found through 51224d16c11b.
Would it be better to leave dst->delta alone when the id is dropped in this
mode, or to return 'no join'?
[ ... ]
> + had_spill = slot_has_spill(sa) || slot_has_spill(sb);
[ ... ]
> + /*
> + * When reads of uninitialized stack are not allowed stacksafe() accepts
> + * MISC in place of a spilled scalar only for whole slots.
> + */
> + if (had_spill && !js->env->allow_uninit_stack) {
> + bool ok = (slot_is_full_spill(sa) || slot_is_data(sa)) &&
> + (slot_is_full_spill(sb) || slot_is_data(sb));
> +
> + for (i = 0; ok && i < BPF_REG_SIZE; i++)
> + ok = res[i] == STACK_MISC;
> + if (!ok) {
> + slot_poison(js, ds);
> + return true;
> + }
> + }
Is it correct to poison the whole 8-byte slot here whenever the slot is not a
full spill or plain data on both sides?
There seem to be two problems.
First, the comment does not match stacksafe(), which also accepts MISC in
place of a spilled scalar for half a slot. At im == 4,
scalar_reg_for_stack() returns unbound_reg for a MISC run in the old (joined)
slot and spilled_ptr for a slot whose slot_type[4] is STACK_SPILL with a
scalar, and regsafe(unbound, spill) is true.
For example, a = [M,M,M,M,S,S,S,S] (a 4-byte scalar spill at fp-8 over MISC)
and b = [M x8]. A result of [M x8] is within both inputs: bytes 0..3 are
MISC against MISC, and bytes 4..7 pass the im == 4 shortcut. The same holds
for a = [S x8] against b = [P,P,P,P,M,M,M,M] with a result of
[P,P,P,P,M,M,M,M].
Second, when ok is false, slot_poison() also sets STACK_POISON on bytes that
are initialized (MISC/ZERO) in both inputs. Those bytes cannot be read after
the join. check_stack_read_fixed_off() then rejects a read that is valid on
each path, with the message "slot poisoned by dead code elimination", which
says nothing about the real cause. In the allow_uninit_stack case the same
inputs give per-byte MISC/INVALID and the reads stay valid.
Once flow_merge() and bpf_flow_exit() call bpf_states_join(), a loader with
CAP_BPF but without CAP_PERFMON (bpf_capable && !allow_uninit_stack, the only
mode that has scalar spills and this branch) can have a valid prog rejected.
flow_merge() does not keep such states apart: flow_unset_differs() looks only
at registers, and flow_join_makes_up() looks only at tested numbers.
Could this return false (no join) for the cases that really cannot be
represented, or poison only the bytes that need it, so that bytes both inputs
initialized are not lost? The comment should also describe what stacksafe()
accepts.
[ ... ]
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index b44a5c63a922..bb42c2707db2 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -17867,6 +17867,73 @@ bool bpf_reg_union_compatible(const struct bpf_reg_state *a, const struct bpf_re
> a->id == b->id && a->parent_id == b->parent_id && a->map_uid == b->map_uid;
> }
>
> +/* Nothing is known about the number or the variable part of the offset anymore. */
> +void bpf_reg_forget_range(struct bpf_reg_state *reg)
> +{
> + reg->var_off = tnum_unknown;
> + __mark_reg_unbounded(reg);
> + reg_step_reset(reg);
> +}
Does this make the join of two packet pointers with a variable offset fail
the check in bpf_states_join() when widen is set?
This is the same problem as noted above for join_range(). join_reg() joins
PTR_TO_PACKET registers that have ids only when the bases are equal, and
keeps the id of a in dst. With js->widen, join_range() calls
bpf_reg_forget_range() for every non-scalar whose range grew.
__mark_reg_unbounded() sets r64 to CNUM64_UNBOUNDED, i.e.
{ .base = 0, .size = U64_MAX }. The r64 of a pointer is the whole offset, so
this also drops the fixed displacement while dst->id is still the non-zero id
of a. bpf_state_subsumes(dst, a) then runs regsafe():
/* Preserve displacements between pointers sharing an ID. */
if (rold->id && rold->r64.base != rcur->r64.base)
return false;
That fails (0 != a->r64.base) whenever the shared base is not 0. join_check()
then prints "join of states at insn %u does not subsume input 'a'" through
verifier_bug() and returns -EFAULT.
For example, p = data + 14 + ihl*4 is a variable add, so p has an id and
r64.base 34. The loop body does p += p[1] (adds [0,255]), and scev cannot
count the loop or the walk runs out of patience (bpf_flow_loop_enter() sets
apart=false). At each back edge flow_merge() runs with FLOW_CYCLE, so
widen=true. The head state has p {id X, r64 {34, n}} and the next trip has
p {id Y, r64 {34, n+255}}. The pair (X, Y) is good and the bases are equal,
so join_reg() goes on to join_range() -> bpf_reg_forget_range(), and
join_check() fails.
Widening also happens in flow_merge() when s->joins > BPF_FLOW_MAX_JOINS, and
in bpf_flow_exit() for recursive subprogs, so enough paths meeting at one
insn are also enough, without a loop.
The commit message says only that "The offset of a pointer that grew becomes
unknown." Users then see a kernel warning and a "verifier bug" message for a
program that should get a normal rejection, or keep a usable pointer. join.c
is built and called from "bpf: Verify programs as a fixpoint over the control
flow graph", and the code is unchanged through 51224d16c11b, so no later fix
was found in the series.
Could bpf_reg_forget_range() keep r64.base and forget only the size for a
PTR_TO_PACKET with an id, or could the id be cleared, or no join be returned?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 48/81] bpf: Describe loops that scalar evolution cannot count exactly
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (46 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 47/81] bpf: Add join of verifier states Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 49/81] selftests/bpf: Adjust tests to loops that are counted now Alexei Starovoitov
` (34 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Scalar evolution examines each loop before the main pass. For
'for (i = 0; i < 10; i++)' it calculates the number of iterations. Then the
verifier gives each variable of the loop its range over all iterations, and
verifies the body one time. Such a loop is a counted loop. The verifier
unrolls all other loops: it verifies them one iteration at a time.
Few loops are counted. The first value of the variable must be a constant.
The bound in the latch, the jump back to the loop head, must be a constant.
Scalar evolution must describe each live register that the loop changes.
"bpf: Verify programs as a fixpoint over the control flow graph" needs, for
each loop, one state at the loop head that covers all iterations. Thus
scalar evolution must describe more loops. The patch makes three changes.
Change 1: count with ranges. When the first value or the bound is a range,
calculate the count for the two values that are furthest apart. The result
is an upper bound.
for (i = up_to_16; i < 17; i++) header_count is [0..17]
There is no count when the value that ends the loop can wrap around. The
variable that the latch tests gets the values that the latch lets through,
because 'first value + step * count' is too wide.
Change 2: describe more registers of a counted loop. Before this patch one
register without a description made the full loop not counted.
what the loop does with a live register the register at the head
--- ------------------------------------------ ------------------------
adds a constant range from the count
sets it to one of a few invariant values union of the values
new loads it again in each iteration unknown scalar
new only clears bits of [0, max]: no change
'idx &= ~(1 << order)', zero extension
new moves a pointer to arena it has no offset
new sets it to the same constant each time no change
something else no description
In the last case the count is valid, and the verifier unrolls the loop.
Change 3: make a guess for a loop that has no count. bpf_scev_forget_regs()
changes the state that enters the loop:
- A counter, 'r += c', gets its range at the entry together with the
values that the latch lets through. A counter that the latch does not
test becomes an unknown scalar, or [0, U32_MAX] for a 32-bit counter.
- A register that the loop sets to one of a few invariant values gets the
union of the values.
The caller must compare the state that returns to the loop head with the
guess, as with each other state. Thus an incorrect guess cannot hide an
unsafe iteration. bpf_scev_latch_decided() tells the caller when it is
better to unroll: constants decide an exit of the loop in the first
iteration. No code calls these two functions in this patch.
The verifier uses changes 1 and 2 immediately. Nothing stops a counted loop
at run time. Before this patch 'i < (n & 0x7fffffff)' hit the limit of
1M insns. Now it loads and can run for 0.48 sec, like 'i < 0x7fffffff'.
Three smaller changes make more loops agree with the table:
- 'r += 1; r += 1' is a step of 2.
- 'r <<= 32; r >>= 32' is a zero extension. This rule depends on
"bpf: Mask the count of a 32-bit shift in scalar evolution".
- The register of an implicit kfunc argument is not live before the call,
because the verifier fills in this argument. A live register that the
prog does not set has no description.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 32 +-
kernel/bpf/const_fold.c | 7 +-
kernel/bpf/liveness.c | 3 +-
kernel/bpf/loops.c | 24 ++
kernel/bpf/scev.c | 795 ++++++++++++++++++++++++++++++++++-
kernel/bpf/verifier.c | 14 +-
6 files changed, 850 insertions(+), 25 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 8001be4c3626..f0f054f98dc1 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -335,6 +335,18 @@ struct bpf_loop_iters {
u32 min_header_count; /* min number of times header is executed */
u32 max_header_count; /* max number of times header is executed */
unsigned long precise_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+ /* live, changed by the loop, and there is no summary of how */
+ unsigned long forget_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+ /* [0, max] at the entry, and the loop only clears bits */
+ unsigned long mask_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
+ /*
+ * The variable that the latch tests, or -1, and what it can be at the
+ * loop header: what it is at the entry or what the latch lets through.
+ */
+ int iv_reg;
+ struct cnum64 iv_range;
+ /* a pointer to the stack that stores go through moves in the loop */
+ bool spill_base;
};
struct loop_stack_entry {
@@ -343,6 +355,8 @@ struct loop_stack_entry {
u32 max_header_count;
u32 loop_id:31;
u32 terminates:1;
+ int iv_reg;
+ struct cnum64 iv_range;
};
#define LOOP_STACK_SIZE 16
@@ -724,6 +738,14 @@ struct bpf_loop {
bool irreducible;
bool backedges_overflow;
bool exits_overflow;
+ /* every trip goes through may_goto or bpf_iter_*_next() */
+ bool rt_bounded;
+ /* insns in the loop, nested loops included */
+ u32 nr_insns;
+ /* functions of the program are called in the loop */
+ bool has_calls;
+ /* otherwise: there are places for may_goto, see guard_to in bpf_insn_aux_data */
+ bool has_guards;
/*
* Loop-entry registers used to compute stack addresses for loads,
* stores and calls requiring fixed stack offsets, in this and nested
@@ -832,12 +854,14 @@ struct bpf_insn_aux_data {
/*
* Bitmask of R0-R9 that hold known values at this instruction.
* const_reg_mask: scalar constants that fit in 32 bits.
+ * const_reg_wide_mask: scalar constants that don't, there is no val.
* const_reg_map_mask: map pointers, val is map_index into used_maps[].
* const_reg_subprog_mask: subprog pointers, val is subprog number.
* const_reg_vals[i] holds the 32-bit value for register i.
* Populated by compute_const_regs() pre-pass.
*/
u16 const_reg_mask;
+ u16 const_reg_wide_mask;
u16 const_reg_map_mask;
u16 const_reg_subprog_mask;
u32 const_reg_vals[10];
@@ -1256,6 +1280,8 @@ static inline bool bpf_ret_reg_pair(struct bpf_verifier_env *env, int subprog)
struct bpf_call_summary {
u8 arg_slot_cnt;
+ /* the last of them are filled in by the verifier, the prog doesn't pass them */
+ u8 implicit_slot_cnt;
bool is_void;
bool fastcall;
};
@@ -1425,6 +1451,8 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
void bpf_join_free(struct bpf_verifier_env *env);
int bpf_insert_unwound_reports(struct bpf_verifier_env *env);
+bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n);
+bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
int bpf_split_cur_state(struct bpf_verifier_env *env);
int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
@@ -2058,12 +2086,14 @@ bool bpf_is_nested_loop(struct bpf_verifier_env *env, int inner_header, int oute
int bpf_init_scev(struct bpf_verifier_env *env);
void bpf_free_scev(struct bpf_verifier_env *env);
int bpf_compute_scev(struct bpf_verifier_env *env);
+int bpf_scev_forget_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
+bool bpf_scev_latch_decided(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
struct bpf_loop_iters *iters);
int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
struct bpf_verifier_state *loop_entry, struct bpf_loop_iters *iters);
int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 insn_idx,
- struct bpf_verifier_state *entry_state, u32 max_header_count);
+ const struct loop_stack_entry *entry);
#endif /* _LINUX_BPF_VERIFIER_H */
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index 0e0cad2b4b8e..bd251c8e82ee 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -285,7 +285,7 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
/* Save computed constants into insn_aux[] if they fit into 32-bit */
for (i = 0; i < insn_cnt; i++) {
- u16 mask = 0, map_mask = 0, subprog_mask = 0;
+ u16 mask = 0, wide_mask = 0, map_mask = 0, subprog_mask = 0;
struct bpf_insn_aux_data *aux = &insn_aux[i];
struct const_arg_info *ci = ci_in[i];
@@ -296,8 +296,10 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
case CONST_ARG_CONST: {
u64 val = c->val;
- if (val != (u32)val)
+ if (val != (u32)val) {
+ wide_mask |= BIT(r);
break;
+ }
mask |= BIT(r);
aux->const_reg_vals[r] = val;
break;
@@ -315,6 +317,7 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
}
}
aux->const_reg_mask = mask;
+ aux->const_reg_wide_mask = wide_mask;
aux->const_reg_map_mask = map_mask;
aux->const_reg_subprog_mask = subprog_mask;
}
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 044f275d28e9..985cd39acf3e 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -2631,7 +2631,8 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env,
def = ALL_CALLER_SAVED_REGS;
use = def & ~BIT(BPF_REG_0);
if (bpf_get_call_summary(env, insn, &cs)) {
- use = GENMASK(min_t(u8, cs.arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS), 1);
+ use = GENMASK(min_t(u8, cs.arg_slot_cnt - cs.implicit_slot_cnt,
+ MAX_BPF_FUNC_REG_ARGS), 1);
} else if (bpf_pseudo_call(insn)) {
/* a static callee's reads are added in bpf_compute_live_registers() */
subprog = callee_subprog(env, insn);
diff --git a/kernel/bpf/loops.c b/kernel/bpf/loops.c
index 2568079c2945..2e54dddb3236 100644
--- a/kernel/bpf/loops.c
+++ b/kernel/bpf/loops.c
@@ -457,6 +457,30 @@ bool bpf_is_nested_loop(struct bpf_verifier_env *env, int inner_header, int oute
return false;
}
+/* Whether the conditional jump at @n, in the loop @h or in a loop nested in it, leaves @h */
+bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n)
+{
+ struct bpf_insn *insn = &env->prog->insnsi[n];
+ u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+ int t, f;
+
+ if ((class != BPF_JMP && class != BPF_JMP32) || op == BPF_JA || op == BPF_CALL ||
+ op == BPF_EXIT || op == BPF_JCOND)
+ return false;
+ t = bpf_loop_at_index(env, n + insn->off + 1);
+ f = bpf_loop_at_index(env, n + 1);
+ return (t != h && !bpf_is_nested_loop(env, t, h)) ||
+ (f != h && !bpf_is_nested_loop(env, f, h));
+}
+
+/* The same for an insn that is in the loop @h itself */
+bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n)
+{
+ if (bpf_loop_at_index(env, n) != h)
+ return false;
+ return bpf_jump_leaves_loop(env, h, n);
+}
+
int bpf_compute_loops(struct bpf_verifier_env *env)
{
int i, j, k, s, t, iloop, sloop, err = 0, len = env->prog->len;
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 46ccce4ab130..853aac03d519 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1499,6 +1499,25 @@ static int simplify(struct scev *scev, u32 id, void *priv)
int a, b, l, r, t, base, slope;
s64 imm1, imm2;
+ /* (- x imm) -> (+ x -imm) */
+ if (is_binop(scev, BPF_SUB, id, &l, &r) &&
+ is_imm(scev, r, &imm1) && imm1 != S64_MIN)
+ return expr2(scev, BPF_ADD, l, imm_expr(scev, -imm1));
+
+ /* (+ (+ x imm) imm) -> (+ x imm) */
+ if (is_add(scev, id, &l, &r) &&
+ is_imm(scev, r, &imm2) &&
+ is_add(scev, l, &a, &b) &&
+ is_imm(scev, b, &imm1))
+ return expr2(scev, BPF_ADD, a, imm_expr(scev, (s64)((u64)imm1 + (u64)imm2)));
+
+ /* (>> (<< x 32) 32) -> (zext32 x) */
+ if (is_binop(scev, BPF_RSH, id, &l, &r) &&
+ is_imm(scev, r, &imm2) && imm2 == 32 &&
+ is_binop(scev, BPF_LSH, l, &a, &b) &&
+ is_imm(scev, b, &imm1) && imm1 == 32)
+ return expr1(scev, ZEXT32, a);
+
/* (+ (linear base slope) imm) -> (linear (+ base imm) slope) */
if (is_add(scev, id, &l, &r) &&
is_linear(scev, l, &base, &slope) &&
@@ -2021,11 +2040,20 @@ static int match_linear_latch(struct bpf_verifier_env *env,
return false;
}
+/* What is known about a scalar or about the offset of a pointer. */
struct scev_value {
- u64 value; /* scalar value or a pointer offset */
+ u64 umin, umax;
+ s64 smin, smax;
+ u32 u32_min, u32_max;
+ s32 s32_min, s32_max;
int ptr_reg; /* SCEV register identifying the pointer origin, or -1. */
};
+static bool scev_value_is_const(const struct scev_value *v)
+{
+ return v->umin == v->umax;
+}
+
static void log_scev_value(struct bpf_verifier_env *env, struct bpf_func_state *st,
const struct scev_value *v)
{
@@ -2035,8 +2063,10 @@ static void log_scev_value(struct bpf_verifier_env *env, struct bpf_func_state *
reg = scev_reg_for_read(env, st, v->ptr_reg);
log_reg(env, v->ptr_reg);
bpf_log(&env->log, " (%s)", reg ? reg_type_str(env, reg->type) : "unavailable");
+ } else if (scev_value_is_const(v)) {
+ bpf_log(&env->log, "scalar %llu", v->umin);
} else {
- bpf_log(&env->log, "scalar %llu", v->value);
+ bpf_log(&env->log, "scalar %llu..%llu", v->umin, v->umax);
}
}
@@ -2076,13 +2106,38 @@ static bool scev_value_add(struct bpf_verifier_env *env, struct bpf_func_state *
const struct scev_value *a, const struct scev_value *b,
struct scev_value *result)
{
+ bool min_wraps, max_wraps;
+
if (a->ptr_reg >= 0 && b->ptr_reg >= 0) {
if (env->log.level & BPF_LOG_LEVEL2)
log_incompatible_scev_values(env, "can't add two pointers", st, a, b);
return false;
}
- result->value = a->value + b->value;
result->ptr_reg = a->ptr_reg >= 0 ? a->ptr_reg : b->ptr_reg;
+
+ /* when both ends wrap around it is a range still */
+ min_wraps = check_add_overflow(a->umin, b->umin, &result->umin);
+ max_wraps = check_add_overflow(a->umax, b->umax, &result->umax);
+ if (min_wraps != max_wraps) {
+ result->umin = 0;
+ result->umax = U64_MAX;
+ }
+ if (check_add_overflow(a->smin, b->smin, &result->smin) ||
+ check_add_overflow(a->smax, b->smax, &result->smax)) {
+ result->smin = S64_MIN;
+ result->smax = S64_MAX;
+ }
+ min_wraps = check_add_overflow(a->u32_min, b->u32_min, &result->u32_min);
+ max_wraps = check_add_overflow(a->u32_max, b->u32_max, &result->u32_max);
+ if (min_wraps != max_wraps) {
+ result->u32_min = 0;
+ result->u32_max = U32_MAX;
+ }
+ if (check_add_overflow(a->s32_min, b->s32_min, &result->s32_min) ||
+ check_add_overflow(a->s32_max, b->s32_max, &result->s32_max)) {
+ result->s32_min = S32_MIN;
+ result->s32_max = S32_MAX;
+ }
return true;
}
@@ -2096,13 +2151,23 @@ static bool eval_expr(struct bpf_verifier_env *env, struct scev *scev,
if (is_reg(scev, id, ®no)) {
reg = scev_reg_for_read(env, st, regno);
- if (!reg || reg->type == NOT_INIT || !tnum_is_const(reg->var_off))
+ if (!reg || reg->type == NOT_INIT)
return false;
- result->value = reg->var_off.value;
+ result->umin = reg_umin(reg);
+ result->umax = reg_umax(reg);
+ result->smin = reg_smin(reg);
+ result->smax = reg_smax(reg);
+ result->u32_min = reg_u32_min(reg);
+ result->u32_max = reg_u32_max(reg);
+ result->s32_min = reg_s32_min(reg);
+ result->s32_max = reg_s32_max(reg);
result->ptr_reg = reg->type == SCALAR_VALUE ? -1 : (int)regno;
return true;
} else if (is_imm(scev, id, &imm)) {
- result->value = imm;
+ result->umin = result->umax = imm;
+ result->smin = result->smax = imm;
+ result->u32_min = result->u32_max = imm;
+ result->s32_min = result->s32_max = imm;
result->ptr_reg = -1;
return true;
} else if (is_add(scev, id, &l, &r) &&
@@ -2244,6 +2309,234 @@ static bool compute_backedge_taken_count(u8 op, u64 initial, u64 step, u64 bound
return true;
}
+/*
+ * The first value of the induction variable at the latch and the bound it is
+ * compared with, as ranges of unsigned numbers. For a signed comparison the sign
+ * bit is flipped: that keeps the order and the differences.
+ * [lo, hi] is what the variable can be without wrapping around.
+ */
+struct latch_domain {
+ u64 imin, imax;
+ u64 bmin, bmax;
+ u64 lo, hi;
+};
+
+static void latch_domain(const struct linear_latch *latch, const struct scev_value *initial,
+ const struct scev_value *bound, bool sign, struct latch_domain *d)
+{
+ u64 bias;
+
+ if (latch->jmp32) {
+ bias = sign ? 1U << 31 : 0;
+ d->imin = (sign ? (u32)initial->s32_min : initial->u32_min) ^ bias;
+ d->imax = (sign ? (u32)initial->s32_max : initial->u32_max) ^ bias;
+ d->bmin = (sign ? (u32)bound->s32_min : bound->u32_min) ^ bias;
+ d->bmax = (sign ? (u32)bound->s32_max : bound->u32_max) ^ bias;
+ d->lo = 0;
+ d->hi = U32_MAX;
+ return;
+ }
+ bias = sign ? 1ULL << 63 : 0;
+ d->bmin = (sign ? (u64)bound->smin : bound->umin) ^ bias;
+ d->bmax = (sign ? (u64)bound->smax : bound->umax) ^ bias;
+ if (latch->zext32) {
+ /* zero extended, so it is not negative */
+ d->imin = initial->u32_min ^ bias;
+ d->imax = initial->u32_max ^ bias;
+ d->lo = bias;
+ d->hi = bias + U32_MAX;
+ return;
+ }
+ d->imin = (sign ? (u64)initial->smin : initial->umin) ^ bias;
+ d->imax = (sign ? (u64)initial->smax : initial->umax) ^ bias;
+ d->lo = 0;
+ d->hi = U64_MAX;
+}
+
+/*
+ * '!=' with a step of one ends when the variable gets to the bound, given that
+ * it is on the right side of the bound to begin with.
+ */
+static bool count_ne_ranges(const struct latch_domain *d, bool down, u64 *cnt)
+{
+ if (d->bmin < d->lo || d->bmax > d->hi)
+ return false;
+ if (down ? d->imin < d->bmax : d->imax > d->bmin)
+ return false;
+ *cnt = down ? d->imax - d->bmin : d->bmax - d->imin;
+ return true;
+}
+
+static bool domain_add(const struct latch_domain *d, u64 v, s64 delta, u64 *res)
+{
+ if (delta >= 0 ? (u64)delta > d->hi - v : -(u64)delta > v - d->lo)
+ return false;
+ *res = v + delta;
+ return true;
+}
+
+/*
+ * The number of iterations depends on the first value, so 'first + slope * count'
+ * is far off when the first value is a range. There is a better bound for
+ * the variable that the latch tests. At the loop header it is
+ * - what it is at the entry, or
+ * - a value that the latch let through, @last is the last of them, plus what
+ * is added between the latch and the header.
+ */
+static void latch_iv_range(struct bpf_verifier_env *env, struct bpf_func_state *st,
+ const struct linear_latch *latch, const struct latch_domain *d,
+ bool sign, bool down, u64 step, u64 last, struct bpf_loop_iters *iters)
+{
+ struct scev *scev = env->scev;
+ struct scev_value off, reg_val;
+ u64 bias, min, max, next;
+ u32 l, r, regno;
+ s64 c = 0;
+
+ l = latch->base_expr;
+ if (!is_reg(scev, l, ®no)) {
+ if (!is_add(scev, latch->base_expr, &l, &r) || !is_reg(scev, l, ®no) ||
+ !eval_expr(env, scev, st, r, &off) || !scev_value_is_const(&off) ||
+ off.ptr_reg >= 0)
+ return;
+ c = latch->jmp32 || latch->zext32 ? (s32)off.smin : off.smin;
+ if (c == S64_MIN)
+ return;
+ }
+ /* the lower half says nothing about a value that does not fit in it */
+ if (latch->jmp32 &&
+ (!latch->zext32 || !eval_expr(env, scev, st, l, ®_val) || reg_val.umax > U32_MAX))
+ return;
+ if (last < d->lo || last > d->hi ||
+ !domain_add(d, d->imin, -c, &min) ||
+ !domain_add(d, d->imax, -c, &max) ||
+ !domain_add(d, last, -c, &next) ||
+ !domain_add(d, next, down ? -(s64)step : (s64)step, &next))
+ return;
+ if (down)
+ min = min(min, next);
+ else
+ max = max(max, next);
+
+ if (latch->jmp32) {
+ bias = sign ? 1U << 31 : 0;
+ min ^= bias;
+ max ^= bias;
+ /* negative as s32 is large as zero extended u64 */
+ if (min > max)
+ return;
+ iters->iv_range = cnum64_from_urange(min, max);
+ } else if (sign) {
+ bias = 1ULL << 63;
+ iters->iv_range = cnum64_from_srange(min ^ bias, max ^ bias);
+ } else {
+ iters->iv_range = cnum64_from_urange(min, max);
+ }
+ iters->iv_reg = regno;
+}
+
+/*
+ * The first value or the bound is not a constant. The count is for the pair that is
+ * the furthest apart.
+ */
+static bool compute_backedge_taken_max(struct bpf_verifier_env *env, struct bpf_func_state *st,
+ const struct linear_latch *latch,
+ const struct scev_value *initial,
+ const struct scev_value *step_val,
+ const struct scev_value *bound,
+ struct bpf_loop_iters *iters, u64 *result)
+{
+ struct scev *scev = env->scev;
+ bool sign = is_signed_op(latch->op), down;
+ struct scev_value reg_val;
+ struct latch_domain d;
+ /* the last value that the latch lets through */
+ u64 step, cnt, last;
+ u32 l, r;
+
+ /* halves of pointers are not compared */
+ if (initial->ptr_reg >= 0 && (latch->jmp32 || latch->zext32))
+ return false;
+ /*
+ * (zext32 (linear ...)) is zext32() of something in all iterations but
+ * the first one at the loop header, where it is what the register has.
+ */
+ if (latch->zext32 && !latch->jmp32) {
+ l = latch->base_expr;
+ if (!is_reg(scev, l, &r) && !is_add(scev, latch->base_expr, &l, &r))
+ return false;
+ if (!eval_expr(env, scev, st, l, ®_val) || reg_val.umax > U32_MAX)
+ return false;
+ }
+
+ step = latch->jmp32 || latch->zext32 ? (u64)(s64)(s32)step_val->umin : step_val->umin;
+ if (!step)
+ return false;
+ down = (s64)step < 0;
+ if (down)
+ step = -step;
+
+ latch_domain(latch, initial, bound, sign, &d);
+
+ /*
+ * The value that ends the loop must not wrap around. It is less than a step
+ * away from the bound whatever the first value is.
+ */
+ switch (latch->op) {
+ case BPF_JLT:
+ case BPF_JSLT:
+ if (down || d.imin >= d.bmax ||
+ check_add_overflow(div64_u64(d.bmax - d.imin - 1, step), 1, &cnt) ||
+ d.bmax > d.hi || step - 1 > d.hi - d.bmax)
+ return false;
+ last = d.bmax - 1;
+ break;
+ case BPF_JLE:
+ case BPF_JSLE:
+ if (down || d.imin > d.bmax ||
+ check_add_overflow(div64_u64(d.bmax - d.imin, step), 1, &cnt) ||
+ d.bmax > d.hi || step > d.hi - d.bmax)
+ return false;
+ last = d.bmax;
+ break;
+ case BPF_JGT:
+ case BPF_JSGT:
+ if (!down || d.imax <= d.bmin ||
+ check_add_overflow(div64_u64(d.imax - d.bmin - 1, step), 1, &cnt) ||
+ d.bmin < d.lo || step - 1 > d.bmin - d.lo)
+ return false;
+ last = d.bmin + 1;
+ break;
+ case BPF_JGE:
+ case BPF_JSGE:
+ if (!down || d.imax < d.bmin ||
+ check_add_overflow(div64_u64(d.imax - d.bmin, step), 1, &cnt) ||
+ d.bmin < d.lo || step > d.bmin - d.lo)
+ return false;
+ last = d.bmin;
+ break;
+ case BPF_JNE:
+ if (step != 1)
+ return false;
+ if (!count_ne_ranges(&d, down, &cnt)) {
+ sign = true;
+ latch_domain(latch, initial, bound, true, &d);
+ if (!count_ne_ranges(&d, down, &cnt))
+ return false;
+ }
+ if (!cnt)
+ return false;
+ last = down ? d.bmin + 1 : d.bmax - 1;
+ break;
+ default:
+ return false;
+ }
+
+ latch_iv_range(env, st, latch, &d, sign, down, step, last, iters);
+ *result = cnt;
+ return true;
+}
+
/*
* Loop with post-condition:
*
@@ -2280,8 +2573,11 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
struct scev *scev = env->scev;
struct scev_value initial_val, step_val, bound_val;
u64 step, bound, initial, total_change, backedge_taken_count;
+ bool konst;
u8 op = latch->op;
+ iters->iv_reg = -1;
+
if (!eval_expr(env, scev, st, latch->base_expr, &initial_val) ||
!eval_expr(env, scev, st, latch->step_expr, &step_val) ||
!eval_expr(env, scev, st, latch->bound_expr, &bound_val))
@@ -2298,9 +2594,19 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
if (!scev_value_compatible(env, st, &initial_val, &bound_val))
return false;
- initial = initial_val.value;
- step = step_val.value;
- bound = bound_val.value;
+ if (!scev_value_is_const(&step_val))
+ return false;
+ konst = scev_value_is_const(&initial_val) && scev_value_is_const(&bound_val);
+ if (!konst) {
+ if (!compute_backedge_taken_max(env, st, latch, &initial_val, &step_val, &bound_val,
+ iters, &backedge_taken_count))
+ return false;
+ goto counted;
+ }
+
+ initial = initial_val.umin;
+ step = step_val.umin;
+ bound = bound_val.umin;
/*
* The code below only handles mixed 64-bit/32-bit comparisons/recurrences
@@ -2321,9 +2627,9 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
step = (s64)(s32)(u32)step;
}
}
- if (latch->zext32 ^ latch->jmp32 && (initial != initial_val.value ||
- bound != bound_val.value ||
- step != step_val.value)) {
+ if (latch->zext32 ^ latch->jmp32 && (initial != initial_val.umin ||
+ bound != bound_val.umin ||
+ step != step_val.umin)) {
if (env->log.level & BPF_LOG_LEVEL2)
bpf_log(&env->log,
"scev: mixed-width latch has operands outside 32-bit ranges\n");
@@ -2349,6 +2655,7 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
}
}
+counted:
/*
* max_header_count is u32 and U32_MAX means "infinite",
* check to avoid overflow below.
@@ -2365,7 +2672,7 @@ static bool compute_max_iters(struct bpf_verifier_env *env,
* In such case, conservatively assume that non-latch exit can happen
* at any iteration, thus setting minimal number of iterations as 0.
*/
- iters->min_header_count = loop->exits_cnt == 1 ? iters->max_header_count : 0;
+ iters->min_header_count = konst && loop->exits_cnt == 1 ? iters->max_header_count : 0;
return true;
}
@@ -2391,6 +2698,19 @@ static bool is_unbound_scev_reg(struct bpf_verifier_env *env, const struct bpf_f
reg->r64.size == U64_MAX && reg->r32.size == U32_MAX;
}
+/*
+ * bpf_compute_const_regs() has @r as a constant at @insn_idx, the head of a loop: it's the same in
+ * every iteration. do_check() takes a value that a range adds to it, or to what is computed
+ * from it, for a bug.
+ */
+static bool is_const_scev_reg(struct bpf_verifier_env *env, int insn_idx, u32 r)
+{
+ struct bpf_insn_aux_data *aux = &env->insn_aux_data[insn_idx];
+
+ return r < ARRAY_SIZE(aux->const_reg_vals) &&
+ ((aux->const_reg_mask | aux->const_reg_wide_mask) & BIT(r));
+}
+
/* Main logic in verifier.c forbids varying offsets for certain register types. */
static bool is_widenable_reg_type(const struct bpf_reg_state *reg)
{
@@ -2571,6 +2891,14 @@ static void linear_bounds(struct bpf_reg_state *reg, u32 max_header_count,
out->r32 = r32;
}
+static void iv_bounds(struct bounds *bounds, struct cnum64 iv_range)
+{
+ struct cnum64 r64 = cnum64_intersect(bounds->r64, iv_range);
+
+ if (!cnum64_is_empty(r64))
+ bounds->r64 = r64;
+}
+
static int mark_expr_precise(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 id,
unsigned long *precise_regs)
{
@@ -2595,6 +2923,71 @@ static int mark_expr_precise(struct bpf_verifier_env *env, struct bpf_func_state
return 0;
}
+/*
+ * Nothing that @id is computed from is what a register had at the head of the loop, and
+ * some of it is not known: it's made anew in every iteration.
+ */
+static bool expr_is_fresh(struct bpf_verifier_env *env, u32 id)
+{
+ struct scev *scev = env->scev;
+ bool opaque = false;
+ u32 order, regno;
+
+ scev->stack_sz = 0;
+ expr_stack_push(scev, id);
+ while (expr_next(scev, &id, &order)) {
+ if (order & DEPTH_LIMIT)
+ return false;
+ if (!(order & PRE))
+ continue;
+ if (is_reg(scev, id, ®no) || is_op(scev, id, UNKNOWN))
+ return false;
+ if (is_opaque(scev, id))
+ opaque = true;
+ }
+ return opaque;
+}
+
+/*
+ * Does @id have no bit that register @r doesn't have at the head of the loop? 'x & e' and the
+ * low bits of x lose bits only. 'x >> e', 'x / e' and 'x % e' are not greater than x as well, but
+ * iterations one by one tell more about them: a number that is halved 8 times.
+ */
+static bool expr_loses_bits_only(struct scev *scev, u32 id, u32 r, int depth)
+{
+ struct expr *e = &scev->exprs[id];
+ u32 regno;
+
+ if (is_reg(scev, id, ®no))
+ return regno == r;
+ if (depth > 8)
+ return false;
+ switch (e->op) {
+ case BPF_AND:
+ return expr_loses_bits_only(scev, e->params[0], r, depth + 1) ||
+ expr_loses_bits_only(scev, e->params[1], r, depth + 1);
+ case ANY:
+ return expr_loses_bits_only(scev, e->params[0], r, depth + 1) &&
+ expr_loses_bits_only(scev, e->params[1], r, depth + 1);
+ case ZEXT8:
+ case ZEXT16:
+ case ZEXT32:
+ return expr_loses_bits_only(scev, e->params[0], r, depth + 1);
+ default:
+ return false;
+ }
+}
+
+/* Nothing but [0, max] is known about @reg. */
+static bool reg_is_zero_to_max(const struct bpf_reg_state *reg)
+{
+ struct cnum64 r64 = cnum64_from_urange(0, reg_umax(reg));
+
+ return cnum64_is_subset(reg->r64, r64) &&
+ cnum32_is_subset(reg->r32, cnum32_from_cnum64(r64)) &&
+ tnum_in(reg->var_off, tnum_range(0, reg_umax(reg))) && reg->step <= 1;
+}
+
int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
struct bpf_loop_iters *iters)
{
@@ -2609,7 +3002,7 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
struct bpf_loop_iters tmp;
struct bpf_loop *loop;
int insn_idx = st->insn_idx;
- bool found = false;
+ bool found = false, walk = false;
int linear_latch;
int latch_idx;
int err, j;
@@ -2665,6 +3058,8 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
}
if (!found || tmp.max_header_count < iters->max_header_count) {
iters->max_header_count = tmp.max_header_count;
+ iters->iv_reg = tmp.iv_reg;
+ iters->iv_range = tmp.iv_range;
best = latch;
}
iters->min_header_count = found ? min(iters->min_header_count, tmp.min_header_count)
@@ -2703,6 +3098,9 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
if (err)
return err;
+ bitmap_zero(iters->forget_regs, REGS_NUM);
+ bitmap_zero(iters->mask_regs, REGS_NUM);
+ iters->spill_base = false;
header_env = find_header_env(scev, insn_idx);
for (r = 0; r < REGS_NUM; r++) {
struct bpf_reg_state *reg;
@@ -2724,10 +3122,23 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
if (is_unbound_scev_reg(env, cur_func, r))
continue;
+ if (is_const_scev_reg(env, insn_idx, r))
+ continue;
+
r_scev = header_env->reg2scev[r];
reg = scev_reg_for_linear(cur_func, r);
+ /*
+ * The offset of a pointer to arena is not tracked, so there is nothing to lose.
+ * If the loop makes something else of it the state that returns to the head
+ * is not covered by the state at the head.
+ */
+ if (reg && reg->type == PTR_TO_ARENA) {
+ __set_bit(r, iters->forget_regs);
+ continue;
+ }
/* If SCEV for r is (linear <reg> <slope>) */
- if (reg && is_simple_linear(scev, r_scev, &coefs) && is_widenable_reg_type(reg)) {
+ if (reg && is_simple_linear(scev, r_scev, &coefs) && is_widenable_reg_type(reg) &&
+ (!coefs.zext32 || reg->type == SCALAR_VALUE)) {
/* Spills at varying offsets lose precision */
if (test_bit(r, loop->store_base_regs)) {
spill_base = true;
@@ -2748,6 +3159,18 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
}
continue;
}
+
+ /*
+ * [0, max] that loses bits is in [0, max]: what enters the loop stands for all
+ * iterations as it is. For 64 or [64, 127] that range would have values in it that
+ * are made up.
+ */
+ if (reg && reg->type == SCALAR_VALUE && !test_bit(r, loop->store_base_regs) &&
+ reg_is_zero_to_max(reg) &&
+ expr_loses_bits_only(scev, header_env->reg2expr[r], r, 0)) {
+ __set_bit(r, iters->mask_regs);
+ continue;
+ }
cant_widen:
if (log->level & BPF_LOG_LEVEL2) {
r_expr = header_env->reg2expr[r];
@@ -2759,7 +3182,17 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
bpf_log(log, ", requires exact stack-offset tracking");
bpf_log(log, "\n");
}
- return 0;
+ /* the count stands, it is this one that is not known in the loop */
+ if (spill_base)
+ iters->spill_base = true;
+ else
+ __set_bit(r, iters->forget_regs);
+ /*
+ * What is loaded from memory or returned by a call in every iteration is
+ * as little known when iterations are looked at one by one.
+ */
+ if (spill_base || !expr_is_fresh(env, header_env->reg2expr[r]))
+ walk = true;
}
bitmap_zero(iters->precise_regs, REGS_NUM);
@@ -2773,7 +3206,47 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
if (err)
return err;
- return 1;
+ /* 2: the count is known, what some of the live registers are is not */
+ return walk ? 2 : 1;
+}
+
+/* The loop changes @r in a way that has no summary. If it is a number it is not known. */
+static void scev_forget_reg(struct bpf_verifier_env *env, struct bpf_func_state *func, u32 r)
+{
+ struct bpf_stack_state *stack;
+ int i;
+
+ if (r < MAX_BPF_REG) {
+ if (func->regs[r].type == SCALAR_VALUE) {
+ bpf_mark_reg_unknown_imprecise(&func->regs[r]);
+ mark_scev_reg_scratched(env, r);
+ } else if (func->regs[r].type == PTR_TO_ARENA) {
+ /* NULL or not is all that is known about it */
+ bpf_reg_forget_range(&func->regs[r]);
+ mark_scev_reg_scratched(env, r);
+ }
+ return;
+ }
+ stack = scev_regno_to_stack(func, r);
+ if (!stack)
+ return;
+ if (bpf_is_spilled_reg(stack) && stack->spilled_ptr.type == PTR_TO_ARENA) {
+ bpf_reg_forget_range(&stack->spilled_ptr);
+ mark_scev_reg_scratched(env, r);
+ return;
+ }
+ if (bpf_is_spilled_reg(stack) && stack->spilled_ptr.type != SCALAR_VALUE)
+ return;
+ for (i = 0; i < BPF_REG_SIZE; i++)
+ if (stack->slot_type[i] == STACK_SPILL || stack->slot_type[i] == STACK_ZERO)
+ break;
+ if (i == BPF_REG_SIZE)
+ return;
+ for (i = 0; i < BPF_REG_SIZE; i++)
+ if (stack->slot_type[i] == STACK_SPILL || stack->slot_type[i] == STACK_ZERO)
+ stack->slot_type[i] = STACK_MISC;
+ bpf_mark_reg_not_init(env, &stack->spilled_ptr);
+ mark_scev_reg_scratched(env, r);
}
/* bpf_compute_loop_iters() checked that every leaf can be unioned into acc. */
@@ -2841,8 +3314,29 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
header_env = find_header_env(scev, insn_idx);
for (r = 0; r < REGS_NUM; r++) {
- if (!scev_reg_alive(env, st, r) || is_unbound_scev_reg(env, cur_func, r))
+ if (!scev_reg_alive(env, st, r) || is_unbound_scev_reg(env, cur_func, r) ||
+ is_const_scev_reg(env, insn_idx, r))
+ continue;
+
+ if (test_bit(r, iters->forget_regs)) {
+ scev_forget_reg(env, cur_func, r);
+ continue;
+ }
+ if (test_bit(r, iters->mask_regs)) {
+ /* The range stays. It's no longer equal to the registers with its id. */
+ err = scev_reg_for_write(env, cur_func, r, ®);
+ if (err)
+ return err;
+ if (log->level & BPF_LOG_LEVEL2) {
+ bpf_log(log, "loop header at %d, ", insn_idx);
+ log_reg(env, r);
+ bpf_log(log, " stays in 0..%llu, it loses bits only\n",
+ reg_umax(reg));
+ }
+ scratch_widened_reg_id(env, reg);
+ mark_scev_reg_scratched(env, r);
continue;
+ }
r_expr = header_env->reg2scev[r];
/* If SCEV for r is (linear <reg> <slope>)*/
@@ -2851,6 +3345,8 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
if (err)
return err;
linear_bounds(reg, iters->max_header_count, &coefs, &bounds);
+ if (r == iters->iv_reg)
+ iv_bounds(&bounds, iters->iv_range);
if (log->level & BPF_LOG_LEVEL2) {
bpf_log(log, "loop header at %d, widening ", insn_idx);
log_reg(env, r);
@@ -2888,9 +3384,11 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
}
int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur_func_state, u32 insn_idx,
- struct bpf_verifier_state *entry_state, u32 max_header_count)
+ const struct loop_stack_entry *entry)
{
+ struct bpf_verifier_state *entry_state = entry->entry_state;
struct bpf_func_state *entry_st = entry_state->frame[entry_state->curframe];
+ u32 max_header_count = entry->max_header_count;
struct bpf_verifier_log *log = &env->log;
struct bpf_reg_state *reg, *entry_reg;
struct scev *scev = env->scev;
@@ -2921,7 +3419,14 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
is_unbound_scev_reg(env, entry_st, r))
continue;
+ /* what was not summarized at the entry is not to be trusted here */
+ if (!is_widenable_reg_type(entry_reg) || entry_reg->type != reg->type ||
+ (coefs.zext32 && entry_reg->type != SCALAR_VALUE))
+ continue;
+
linear_bounds(entry_reg, max_header_count, &coefs, &bounds);
+ if (r == entry->iv_reg)
+ iv_bounds(&bounds, entry->iv_range);
r64 = cnum64_intersect(reg->r64, bounds.r64);
r32 = cnum32_intersect(reg->r32, bounds.r32);
/*
@@ -2951,3 +3456,255 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
}
return 0;
}
+
+/*
+ * The number of iterations of the loop that @st enters is not known. Make @st
+ * stand for all of them as far as there is a summary of what the loop does:
+ * - a variable that a constant is added to can be anything, anything that
+ * fits in 32 bits if it is computed in 32 bits,
+ * - a variable that is set to one of several values is one of them.
+ * The rest is left as it is: the state that comes back to the loop head tells.
+ * Guessing here would make up values that the program never has.
+ */
+
+/*
+ * do { arr[i] = i; i--; } while (i < 1000);
+ * What register @r may be at the head of the loop: what it is at the entry, or what the
+ * jump that closes the loop lets through. It's a guess. The state that comes back to
+ * the head is compared with it like with any other state.
+ */
+static bool latch_range(struct bpf_verifier_env *env, struct bpf_loop *loop, int header, u32 r,
+ const struct bpf_reg_state *entry, struct cnum64 *range)
+{
+ struct bpf_insn *insn;
+ u64 uk;
+ s64 sk;
+ int idx;
+ u8 op;
+
+ if (r >= MAX_BPF_REG ||
+ loop->backedges_cnt != 1 || loop->backedges_overflow)
+ return false;
+ idx = loop->backedges[0].from;
+ insn = &env->prog->insnsi[idx];
+ if (BPF_CLASS(insn->code) != BPF_JMP || BPF_SRC(insn->code) != BPF_K || insn->dst_reg != r)
+ return false;
+ op = BPF_OP(insn->code);
+ sk = insn->imm;
+ uk = sk;
+ if (idx + 1 == header && idx + insn->off + 1 != header) {
+ /* the loop goes on when the jump is not taken */
+ op = bpf_rev_opcode(op);
+ } else if (idx + insn->off + 1 != header) {
+ return false;
+ }
+ switch (op) {
+ case BPF_JLT:
+ if (!uk)
+ return false;
+ *range = cnum64_from_urange(0, uk - 1);
+ break;
+ case BPF_JLE:
+ *range = cnum64_from_urange(0, uk);
+ break;
+ case BPF_JGT:
+ if (uk == U64_MAX)
+ return false;
+ *range = cnum64_from_urange(uk + 1, U64_MAX);
+ break;
+ case BPF_JGE:
+ *range = cnum64_from_urange(uk, U64_MAX);
+ break;
+ case BPF_JSLT:
+ if (sk == S64_MIN)
+ return false;
+ *range = cnum64_from_srange(S64_MIN, sk - 1);
+ break;
+ case BPF_JSLE:
+ *range = cnum64_from_srange(S64_MIN, sk);
+ break;
+ case BPF_JSGT:
+ if (sk == S64_MAX)
+ return false;
+ *range = cnum64_from_srange(sk + 1, S64_MAX);
+ break;
+ case BPF_JSGE:
+ *range = cnum64_from_srange(sk, S64_MAX);
+ break;
+ default:
+ return false;
+ }
+ *range = cnum64_union(*range, entry->r64);
+ return true;
+}
+
+/*
+ * elem = NULL; found = false;
+ * loop { if (...) { found = true; elem = v; } }
+ * if (found) use(*elem);
+ * The union of what "found" may be is known at the entry of the loop, what "elem" may be is
+ * not. [0, 1] next to NULL is something that the program never gets to. Leave such a
+ * loop alone: (1, pointer) that comes back has no join with (0, 0).
+ */
+static bool loop_looks_for_something(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ struct env *header_env)
+{
+ struct bpf_func_state *func = st->frame[st->curframe];
+ const struct bpf_reg_state *reg;
+ u32 r, left, right;
+
+ for (r = 0; r < REGS_NUM; r++) {
+ if (r == BPF_REG_FP || !scev_reg_alive(env, st, r) ||
+ is_unbound_scev_reg(env, func, r))
+ continue;
+ reg = scev_reg_for_read(env, func, r);
+ if (reg && reg->type == SCALAR_VALUE && tnum_equals_const(reg->var_off, 0) &&
+ is_any(env->scev, header_env->reg2scev[r], &left, &right) &&
+ !is_widenable_any(env, func, header_env, r, header_env->reg2scev[r]))
+ return true;
+ }
+ return false;
+}
+
+/*
+ * @st enters a loop and the number of iterations is not known. Make @st stand for all of them
+ * ahead of time, it saves trips through the loop:
+ * - a counter, r = r + c, is what the jump that closes the loop lets through, see latch_range(),
+ * 32 bit if it's computed in 32 bit, otherwise it's not known,
+ * - a register that gets one of a few values that don't change in the loop is their union.
+ * It's a guess. The state that comes back to the head is compared with it like with any other.
+ */
+int bpf_scev_forget_regs(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
+{
+ struct bpf_func_state *func = st->frame[st->curframe];
+ struct bpf_loop *loop = env->insn_aux_data[st->insn_idx].loop;
+ struct scev *scev = env->scev;
+ struct bpf_reg_state *reg;
+ struct env *header_env;
+ struct linear_coefs coefs;
+ struct cnum64 range;
+ u32 r, r_scev;
+ bool search;
+ int err;
+
+ err = bpf_live_stack_query_init(env, st);
+ if (err)
+ return err;
+
+ header_env = find_header_env(scev, st->insn_idx);
+ if (!header_env)
+ return 0;
+ search = loop_looks_for_something(env, st, header_env);
+ for (r = 0; r < REGS_NUM; r++) {
+ if (r == BPF_REG_FP || !scev_reg_alive(env, st, r) ||
+ is_unbound_scev_reg(env, func, r))
+ continue;
+ r_scev = header_env->reg2scev[r];
+ reg = scev_reg_for_linear(func, r);
+ if (is_simple_linear(scev, r_scev, &coefs) && coefs.base_reg == r &&
+ reg && reg->type == SCALAR_VALUE) {
+ if (latch_range(env, loop, st->insn_idx, r, reg, &range)) {
+ err = scev_reg_for_write(env, func, r, ®);
+ if (err)
+ return err;
+ scratch_widened_reg_id(env, reg);
+ err = bpf_set_reg_range(env, reg, range, CNUM32_UNBOUNDED, 0, 1);
+ if (err)
+ return err;
+ mark_scev_reg_scratched(env, r);
+ continue;
+ }
+ if (!coefs.zext32 || reg_smin(reg) < 0 || reg_smax(reg) > U32_MAX) {
+ scev_forget_reg(env, func, r);
+ continue;
+ }
+ err = scev_reg_for_write(env, func, r, ®);
+ if (err)
+ return err;
+ scratch_widened_reg_id(env, reg);
+ err = bpf_set_reg_range(env, reg, cnum64_from_urange(0, U32_MAX),
+ CNUM32_UNBOUNDED, 0, 1);
+ if (err)
+ return err;
+ mark_scev_reg_scratched(env, r);
+ } else if (!search && is_widenable_any(env, func, header_env, r, r_scev) &&
+ !test_bit(r, loop->store_base_regs)) {
+ err = scev_reg_for_write(env, func, r, ®);
+ if (err)
+ return err;
+ /* the leaves are invariant, so @func is as good as a copy of it */
+ err = union_any_reg(env, func, reg, r_scev);
+ if (err)
+ return err;
+ scratch_widened_reg_id(env, reg);
+ mark_scev_reg_scratched(env, r);
+ }
+ }
+ return 0;
+}
+
+/* Everything that @id is computed from is a constant in @func. */
+static bool expr_is_const(struct bpf_verifier_env *env, struct bpf_func_state *func, u32 id)
+{
+ const struct bpf_reg_state *reg;
+ struct scev *scev = env->scev;
+ u32 order, regno, l, r;
+ s64 imm;
+
+ scev->stack_sz = 0;
+ expr_stack_push(scev, id);
+ while (expr_next(scev, &id, &order)) {
+ if (order & DEPTH_LIMIT)
+ return false;
+ if (!(order & PRE) || is_imm(scev, id, &imm))
+ continue;
+ if (is_reg(scev, id, ®no)) {
+ reg = scev_reg_for_read(env, func, regno);
+ if (!reg || reg->type != SCALAR_VALUE || !tnum_is_const(reg->var_off))
+ return false;
+ continue;
+ }
+ if (is_opaque(scev, id) || is_op(scev, id, UNKNOWN) || is_any(scev, id, &l, &r))
+ return false;
+ }
+ return true;
+}
+
+/*
+ * @st enters a loop. Is it known which way one of the jumps that leave it goes in the first
+ * iteration? Then the loop counts something in a way that has no summary, and going
+ * through its iterations one by one is likely to get to its end.
+ */
+bool bpf_scev_latch_decided(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
+{
+ struct bpf_func_state *func = st->frame[st->curframe];
+ struct bpf_loop *loop = env->insn_aux_data[st->insn_idx].loop;
+ struct bpf_insn *insn;
+ struct env *latch_env;
+ int latch_idx, i;
+
+ if (!env->allow_uninit_stack)
+ return false;
+ if (!loop || loop->irreducible || loop->backedges_cnt != 1 || loop->backedges_overflow)
+ return false;
+ /* and the head of the loop: it's not among the latches of a back edge from a nested loop */
+ for (i = 0; i <= loop->backedges[0].latches_cnt; i++) {
+ if (i < loop->backedges[0].latches_cnt)
+ latch_idx = loop->backedges[0].latches[i];
+ else if (bpf_insn_leaves_loop(env, st->insn_idx, st->insn_idx))
+ latch_idx = st->insn_idx;
+ else
+ break;
+ insn = &env->prog->insnsi[latch_idx];
+ if (bpf_is_may_goto_insn(insn))
+ return false;
+ latch_env = find_loop_env(env->scev, st->insn_idx, latch_idx);
+ if (!latch_env)
+ continue;
+ if (expr_is_const(env, func, latch_env->reg2expr[insn->dst_reg]) &&
+ (BPF_SRC(insn->code) == BPF_K ||
+ expr_is_const(env, func, latch_env->reg2expr[insn->src_reg])))
+ return true;
+ }
+ return false;
+}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index bb42c2707db2..32b256bc1513 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -19862,6 +19862,7 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
bpf_jit_inlines_helper_call(call->imm));
cs->is_void = fn->ret_type == RET_VOID;
cs->arg_slot_cnt = 0;
+ cs->implicit_slot_cnt = 0;
for (i = 0; i < ARRAY_SIZE(fn->arg_type); ++i) {
if (fn->arg_type[i] == ARG_UNUSED)
break;
@@ -19878,6 +19879,14 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
/* error would be reported later */
return false;
cs->arg_slot_cnt = btf_proto_slots(meta.btf, meta.func_proto);
+ cs->implicit_slot_cnt = 0;
+ if (meta.kfunc_flags & KF_IMPLICIT_ARGS) {
+ const struct btf_type *func = btf_type_by_id(meta.btf, meta.func_id);
+
+ /* the prototype without implicit args, see is_kfunc_arg_implicit() */
+ cs->implicit_slot_cnt = cs->arg_slot_cnt -
+ btf_proto_slots(meta.btf, btf_type_by_id(meta.btf, func->type));
+ }
cs->fastcall = meta.kfunc_flags & KF_FASTCALL;
cs->is_void = btf_type_is_void(btf_type_by_id(meta.btf, meta.func_proto->type));
return true;
@@ -20716,8 +20725,7 @@ static int maybe_clamp_scev_regs(struct bpf_verifier_env *env)
if (!entry->terminates)
return 0;
- return bpf_clamp_scev_regs(env, frame, env->insn_idx, entry->entry_state,
- entry->max_header_count);
+ return bpf_clamp_scev_regs(env, frame, env->insn_idx, entry);
}
static int handle_loop_entry_exit(struct bpf_verifier_env *env)
@@ -20763,6 +20771,8 @@ static int handle_loop_entry_exit(struct bpf_verifier_env *env)
entry->min_header_count = iters.min_header_count;
entry->max_header_count = iters.max_header_count;
+ entry->iv_reg = iters.iv_reg;
+ entry->iv_range = iters.iv_range;
entry->entry_state = cur->parent;
entry->terminates = true;
err = bpf_widen_scev_regs(env, cur, cur->parent, &iters);
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 48/81] bpf: Describe loops that scalar evolution cannot count exactly
2026-10-26 2:59 ` [PATCH bpf 48/81] bpf: Describe loops that scalar evolution cannot count exactly Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 14115 bytes --]
> bpf: Describe loops that scalar evolution cannot count exactly
TL;DR:
- Pointer compares in scev may undercount iterations, so the verifier could
accept an out-of-bounds map access
- Forgetting "fresh" registers may reject previously loading programs or leave
uninitialized/pointer values at the loop head
- Stray leftover comment (nit)
> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index 46ccce4ab130..853aac03d519 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c
[ ... ]
> @@ -2244,6 +2309,234 @@ static bool compute_backedge_taken_count(u8 op, u64 initial, u64 step, u64 bound
[ ... ]
> + bias = sign ? 1ULL << 63 : 0;
> + d->bmin = (sign ? (u64)bound->smin : bound->umin) ^ bias;
> + d->bmax = (sign ? (u64)bound->smax : bound->umax) ^ bias;
> + if (latch->zext32) {
> + /* zero extended, so it is not negative */
> + d->imin = initial->u32_min ^ bias;
> + d->imax = initial->u32_max ^ bias;
> + d->lo = bias;
> + d->hi = bias + U32_MAX;
> + return;
> + }
> + d->imin = (sign ? (u64)initial->smin : initial->umin) ^ bias;
> + d->imax = (sign ? (u64)initial->smax : initial->umax) ^ bias;
> + d->lo = 0;
> + d->hi = U64_MAX;
> +}
Can this undercount the iterations when the latch compares two pointers of
the same origin with an unsigned op?
For a pointer, eval_expr() fills scev_value from reg_umin()/reg_umax() of
r64, which holds the offset of the pointer. With sign == false,
latch_domain() orders the first value and the bound by those u64 offsets.
At run time the jump compares addresses, and for two pointers into the same
object address order is the signed order of the offsets. The two orders
differ whenever a range holds both a negative and a non-negative offset.
cnum64_umin()/cnum64_umax() then return [0, U64_MAX], so d->imin = 0 or
d->bmax = U64_MAX is taken as the furthest value, when in address order the
real extreme is the negative offset or offset 0.
The old code could not get here. eval_expr() required tnum_is_const(), and
with constants a disagreement in order either fails 'initial >= bound' or
gives a count of about 2^64, which the U32_MAX check rejects. The new range
path is the first place where a straddling range reaches the count.
Example (privileged, so env->allow_uninit_stack is set):
r6 &= 63 // r6 in [0, 63]
r7 = r10
r7 -= r6 // PTR_TO_STACK, off in [-63, 0]
r9 = 0
loop:
... body uses r9 as an index ...
r7 += 1
r9 += 1
if r7 <= r10 goto loop // BPF_JLE, BPF_X, both PTR_TO_STACK
base_expr is (+ r7 1). Its offsets [-62, 1] straddle 0, so
scev_value_add() gives umin = 0, umax = U64_MAX. The bound r10 has offset
0, and scev_value_compatible() accepts it (same frameno). konst is false,
so compute_backedge_taken_max() runs. In the BPF_JLE case d.imin = 0 <=
d.bmax = 0, so cnt = div64_u64(0 - 0, 1) + 1 = 1, and the wrap check passes.
compute_max_iters() then sets max_header_count = 2.
At run time, with r6 = 63 so that r7 starts at fp-63, the latch lets
fp-62 .. fp+0 through: 63 backedges and 64 trips through the header.
bpf_widen_scev_regs() gives r9 the range [0, 1] (linear_bounds() with
max_header_count 2). When the body comes back to the head with r9 in
[1, 2], bpf_clamp_scev_regs() narrows it to [1, 1], and the state is pruned
against the head. The body is only verified for r9 <= 1, while r9 really
reaches 63. An access like *(u8 *)(map_value + r9) on a 2-byte value passes
the verifier and writes out of bounds at run time.
The same thing happens with a straddling bound and BPF_JLT, step 1 (bmax =
U64_MAX stands for offset -1, while offset 0 is the real largest bound), and
with a straddling first value in the BPF_JGT/BPF_JGE down case. BPF_JNE is
not affected: its unsigned attempt either fails or yields a count of about
2^64, which is rejected, and its signed retry orders pointers correctly.
The only pointer check in compute_backedge_taken_max() is:
/* halves of pointers are not compared */
if (initial->ptr_reg >= 0 && (latch->jmp32 || latch->zext32))
return false;
For full-width unsigned compares the pointer case falls through to the u64
domain. Should pointer operands use the signed view of the offsets whatever
the op, or be rejected when umin/umax and smin/smax disagree?
Nothing in af2fd429b9f6..51224d16c11b changes compute_backedge_taken_max()
or latch_domain(). At the end of the series bpf_flow_loop_enter() still
widens and clamps from this count, and in the 'counted' walk it also drops
trips beyond max_header_count ('there is no such iteration at run time').
[ ... ]
> @@ -2595,6 +2923,71 @@ static int mark_expr_precise(struct bpf_verifier_env *env, struct bpf_func_state
> +/*
> + * Nothing that @id is computed from is what a register had at the head of the loop, and
> + * some of it is not known: it's made anew in every iteration.
> + */
> +static bool expr_is_fresh(struct bpf_verifier_env *env, u32 id)
> +{
> + struct scev *scev = env->scev;
> + bool opaque = false;
> + u32 order, regno;
> +
> + scev->stack_sz = 0;
> + expr_stack_push(scev, id);
> + while (expr_next(scev, &id, &order)) {
> + if (order & DEPTH_LIMIT)
> + return false;
> + if (!(order & PRE))
> + continue;
> + if (is_reg(scev, id, ®no) || is_op(scev, id, UNKNOWN))
> + return false;
> + if (is_opaque(scev, id))
> + opaque = true;
> + }
> + return opaque;
> +}
Is it right to treat any expression built only from opaque leaves as fresh
here? The comment in bpf_compute_loop_iters() justifies it with:
What is loaded from memory or returned by a call in every iteration is
as little known when iterations are looked at one by one.
The code does not seem to support that. OPAQUE_EXPR_ID is produced by
maybe_load_fp() for every load whose base is not r10, whatever its size
('else id = OPAQUE_EXPR_ID;'). It is also produced for BPF_LD_IMM64 of a
map, map value, BTF id or function, and for atomic fetch results. The
verifier knows more about many of these values:
- check_mem_access() runs coerce_reg_to_size() on a u8/u16/u32 load,
which gives [0,255], [0,65535] or [0,U32_MAX]
- a ctx load can give a pointer type
- a load through a non-r10 pointer to the stack gives the exact spilled
value
Before this patch such a register reached 'cant_widen: ... return 0', and
the loop was unrolled with the exact value in each iteration. Now
expr_is_fresh() returns true, bpf_compute_loop_iters() returns 1, and
bpf_widen_scev_regs() replaces the register with a fully unknown 64-bit
scalar.
For example:
r7 = <non-NULL map value, value_size 256>
r6 = 0
r8 = 0
l0:
r1 = r7
r1 += r6
r0 = *(u8 *)(r1 + 0)
r6 = *(u8 *)(r7 + 0)
r8 += 1
if r8 < 10 goto l0
At the head, r6 has scev (any r6 OPAQUE). That is not widenable, and
expr_loses_bits_only() is false, so r6 goes to forget_regs.
expr_is_fresh() returns true for the opaque leaf, so the function returns 1
and r6 becomes an unknown scalar. The map access with r6 as an unbounded
offset is then rejected, although the same program loaded before this patch
(r6 is in [0,255] in every iteration).
The comment is also wrong about calls: forget_call_regs() sets r0-r5 to
UNKNOWN_EXPR_ID, and expr_is_fresh() returns false for UNKNOWN, so a value
returned by a helper makes the loop walk instead of being forgotten.
The same logic is still in place at the end of the series (51224d16c11b):
bpf_flow_loop_enter() in 0535bc235aed still widens directly on err == 1, and
expr_is_fresh()/maybe_load_fp() are unchanged.
Could the opaque leaves carry their load width (for example zext8/16/32 of
an opaque leaf), or could a register be treated as fresh only when what the
verifier knows at the entry is no better than an unknown scalar of the same
width?
[ ... ]
> @@ -2759,7 +3182,17 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
> bpf_log(log, ", requires exact stack-offset tracking");
> bpf_log(log, "\n");
> }
> - return 0;
> + /* the count stands, it is this one that is not known in the loop */
> + if (spill_base)
> + iters->spill_base = true;
> + else
> + __set_bit(r, iters->forget_regs);
> + /*
> + * What is loaded from memory or returned by a call in every iteration is
> + * as little known when iterations are looked at one by one.
> + */
> + if (spill_base || !expr_is_fresh(env, header_env->reg2expr[r]))
> + walk = true;
> }
Can this make programs that loaded before this patch fail to load? A live
register that is reloaded on every iteration now ends up less precise at the
head than it was in any iteration, while the loop is still counted and
widened.
Before the patch, a live register with no summary went to cant_widen and the
function returned 0, so the verifier walked the loop. Now, when
header_env->reg2expr[r] is fresh, walk stays false. The function returns 1,
and bpf_widen_scev_regs() calls scev_forget_reg() on the register:
if (r < MAX_BPF_REG) {
if (func->regs[r].type == SCALAR_VALUE) {
bpf_mark_reg_unknown_imprecise(&func->regs[r]);
...
} else if (func->regs[r].type == PTR_TO_ARENA) {
...
}
return;
}
An expression is fresh when it contains an opaque leaf and no register, and
maybe_load_fp() gives OPAQUE_EXPR_ID to every load that is not a spill fill,
whatever its size. That causes two problems.
1) The entry value is uninitialized or a pointer. scev_forget_reg() leaves
it as it is, so the widened head still holds the entry value, while the
counted IV is already a range that includes the exit value. A hand-written
example:
r9 = <non-NULL map value>
r7 = 0
1: if r7 >= 10 goto 2f
r6 = *(u8 *)(r9 + 0)
r7 += 1
goto 1b
2: r0 = r6
exit
r6 is live at the head because of the exit path, and its expression is
opaque. The loop is counted (header_count 11), and r7 is widened to [0, 10]
while r6 stays uninitialized. From the head state the branch r7 >= 10 is
now taken, and 'r0 = r6' fails with R6 !read_ok. Before the patch the loop
was walked: the exit is reached only with r7 == 10, where r6 is set, and the
program loaded.
A pointer entry value has the same problem. For example, an entry pointer
that the loop replaces with a scalar is still a pointer on the exit path.
2) The entry value is a scalar. It becomes a fully unknown 64-bit scalar,
but a narrow load gives a bounded value: coerce_reg_to_size() bounds BPF_B,
BPF_H and BPF_W loads to [0, 255], [0, 65535] and [0, U32_MAX]. Take:
prev = 0;
for (i = 0; i < 16; i++) {
tbl[prev] = 1;
prev = in[i];
}
with u8 'in' and a 256-byte map value as 'tbl'. Walking the loop proved the
access safe. Now prev is unknown at the head, and 'tbl + prev' fails as an
unbounded memory access.
The new comment claims that a value loaded again in each iteration is 'as
little known when iterations are looked at one by one'. That is not true
for sized loads, or for an uninitialized or pointer entry value.
The comment also mentions values 'returned by a call', but
forget_call_regs() sets r0-r5 to UNKNOWN_EXPR_ID, and expr_is_fresh()
returns false for UNKNOWN, so a call result always walks the loop and is
never forgotten.
The commit message says such a register becomes an 'unknown scalar' at the
head, which does not match the uninitialized and pointer entry cases either.
In 51224d16c11b, at the end of the series, handle_loop_entry() still calls
bpf_widen_scev_regs() for a return value of 1 with the same forget logic, so
no later commit in the series fixes this.
Could a fresh register be forgotten only when the result covers all
iterations, and the loop be walked otherwise (walk = true)? Here that would
mean forgetting only when the entry value is a SCALAR_VALUE and the unknown
result does not lose a range that the loads give.
[ ... ]
> @@ -2951,3 +3456,255 @@ int bpf_clamp_scev_regs(struct bpf_verifier_env *env, struct bpf_func_state *cur
[ ... ]
> +/*
> + * The number of iterations of the loop that @st enters is not known. Make @st
> + * stand for all of them as far as there is a summary of what the loop does:
> + * - a variable that a constant is added to can be anything, anything that
> + * fits in 32 bits if it is computed in 32 bits,
> + * - a variable that is set to one of several values is one of them.
> + * The rest is left as it is: the state that comes back to the loop head tells.
> + * Guessing here would make up values that the program never has.
> + */
> +
> +/*
> + * do { arr[i] = i; i--; } while (i < 1000);
> + * What register @r may be at the head of the loop: what it is at the entry, or what the
> + * jump that closes the loop lets through. It's a guess. The state that comes back to
> + * the head is compared with it like with any other state.
> + */
Is the first comment block above, the one before latch_range(), meant to
stay? It is not attached to any function, and it describes
bpf_scev_forget_regs(), which has its own comment further down.
The two comments disagree about counters. The detached block says:
a variable that a constant is added to can be anything, anything that
fits in 32 bits if it is computed in 32 bits
The code does something else for a counter that the closing jump tests:
if (latch_range(env, loop, st->insn_idx, r, reg, &range)) {
...
err = bpf_set_reg_range(env, reg, range, CNUM32_UNBOUNDED, 0, 1);
That range is the union of the entry range and what the back edge lets
through. It is not unknown and not [0, U32_MAX]. Only counters that
latch_range() rejects get scev_forget_reg() or [0, U32_MAX].
The comment on bpf_scev_forget_regs() says this correctly ("is what the jump
that closes the loop lets through, see latch_range()"), so the detached
block looks like a leftover of an earlier version. It is still present at
the end of the series (51224d16c11b).
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 49/81] selftests/bpf: Adjust tests to loops that are counted now
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (47 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 48/81] bpf: Describe loops that scalar evolution cannot count exactly Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 50/81] bpf: Add may_goto to the loops that the verifier marks Alexei Starovoitov
` (33 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
"bpf: Describe loops that scalar evolution cannot count exactly" counts
more loops. Three tests expected a loop that is not counted. All progs
load.
test the test expects now
---------------------------- -------------------------------------------
no_widen_small_spill_counter header_count is [0..266]
uninit_slot_counter header_count is [0..11]
state_loop_first_last_equal the prog loads
kfunc_implicit_arg (new) r5 is not live before the call of the kfunc
The first two loops count from a stack slot with unknown contents. The
count has an upper bound for each first value.
state_loop_first_last_equal examined the log of precision backtracking in a
loop that was not counted. 'r0 += 1; r0 += 1' is now a step of 2, and the
loop is counted. Keep the prog as a load test:
"bpf: Remove precision tracking" removes backtracking.
The verifier fills in r5, the implicit argument of bpf_stream_vprintk().
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../bpf/progs/compute_live_registers.c | 13 ++++++++++
.../selftests/bpf/progs/verifier_precision.c | 25 ++-----------------
.../selftests/bpf/progs/verifier_scev.c | 6 +++--
3 files changed, 19 insertions(+), 25 deletions(-)
diff --git a/tools/testing/selftests/bpf/progs/compute_live_registers.c b/tools/testing/selftests/bpf/progs/compute_live_registers.c
index 1ff24bbc7beb..de0496ce7d5e 100644
--- a/tools/testing/selftests/bpf/progs/compute_live_registers.c
+++ b/tools/testing/selftests/bpf/progs/compute_live_registers.c
@@ -538,6 +538,19 @@ void gotox(void)
#endif /* gotox: x86, arm64, riscv64 */
+extern int bpf_stream_vprintk(int stream_id, const char *fmt__str, const void *args,
+ __u32 len__sz) __weak __ksym;
+
+/* The verifier fills in the implicit argument of the kfunc. The prog does not pass r5. */
+SEC("syscall")
+__log_level(2)
+__msg(": .1234..... (85) call bpf_stream_vprintk")
+int kfunc_implicit_arg(void *ctx)
+{
+ bpf_stream_printk(1 /* BPF_STDOUT */, "x");
+ return 0;
+}
+
/* to retain debug info for BTF generation */
void kfunc_root(void)
{
diff --git a/tools/testing/selftests/bpf/progs/verifier_precision.c b/tools/testing/selftests/bpf/progs/verifier_precision.c
index f4459561bf39..2ec3ee7821c0 100644
--- a/tools/testing/selftests/bpf/progs/verifier_precision.c
+++ b/tools/testing/selftests/bpf/progs/verifier_precision.c
@@ -150,24 +150,7 @@ __naked int bpf_store_release(void)
#endif /* v4 instruction */
SEC("?raw_tp")
-__success __log_level(2)
-/*
- * Without the bug fix there will be no history between "last_idx 3 first_idx 3"
- * and "parent state regs=" lines. "R0=6" parts are here to help anchor
- * expected log messages to the one specific mark_chain_precision operation.
- *
- * This is quite fragile: if verifier checkpointing heuristic changes, this
- * might need adjusting.
- */
-__msg("2: (07) r0 += 1 ; R0=6")
-__msg("3: (35) if r0 >= 0xa goto pc+1")
-__msg("mark_precise: frame0: last_idx 3 first_idx 3 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 2: (07) r0 += 1")
-__msg("mark_precise: frame0: regs=r0 stack= before 1: (07) r0 += 1")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (05) goto pc-4")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (35) if r0 >= 0xa goto pc+1")
-__msg("mark_precise: frame0: parent state regs= stack=: R0=P4")
-__msg("3: R0=6")
+__success
__naked int state_loop_first_last_equal(void)
{
asm volatile (
@@ -175,11 +158,7 @@ __naked int state_loop_first_last_equal(void)
"l0_%=:"
"r0 += 1;"
"r0 += 1;"
- /* every few iterations we'll have a checkpoint here with
- * first_idx == last_idx, potentially confusing precision
- * backtracking logic
- */
- "if r0 >= 10 goto l1_%=;" /* checkpoint + mark_precise */
+ "if r0 >= 10 goto l1_%=;"
"goto l0_%=;"
"l1_%=:"
"exit;"
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index a68a2368aa71..b30802c9e9e3 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -1188,7 +1188,8 @@ __success
__log_level(2)
__msg("scev at header 2:")
__msg_next(" *fp-8=(+ *fp-8 1) / (linear *fp-8 1)")
-__msg("loop header at 2, can't compute iterations count")
+/* whatever is in the other bytes of the slot */
+__msg("loop header at 2, header_count is [0..266] ")
__not_msg("widening")
__naked void no_widen_small_spill_counter(void)
{
@@ -2405,7 +2406,8 @@ __naked void sibling_inner_loops(void)
SEC("socket")
__success
__log_level(2)
-__msg("loop header at 0, can't compute iterations count")
+/* whatever is in the slot */
+__msg("loop header at 0, header_count is [0..11] ")
__naked void uninit_slot_counter(void)
{
asm volatile (" \
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 50/81] bpf: Add may_goto to the loops that the verifier marks
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (48 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 49/81] selftests/bpf: Adjust tests to loops that are counted now Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 51/81] bpf: Check stack and count calls of recursive subprogs at run time Alexei Starovoitov
` (32 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
"bpf: Verify programs as a fixpoint over the control flow graph" verifies a
loop with one state that covers all iterations. This shows that each
iteration is safe. It does not show that the loop ends. Thus the kernel
must stop such a loop at run time.
may_goto is the insn for that: after 250 ms it jumps to its target. The
verifier adds may_goto to each loop that it cannot count. It adds nothing
when each iteration already executes may_goto or bpf_iter_*_next().
The jump of the added may_goto is a new edge in the prog. It must pass the
verifier like each other edge. One target for all loops is not possible:
do { lock();
p = lookup(); do { ... } while (...);
} while (!p); unlock();
x = *p;
It cannot exit from the loop: It cannot exit from the prog:
p is NULL after the loop. only the code after the loop
unlocks.
Thus the verifier selects the target for each added may_goto:
Does the frame hold a lock or a reference
that it did not hold at its entry?
| no | yes
v v
exit from the subprog exit from the loop
may_goto is in exit from the subprog
------------------------------------------ ---------------------------
the main prog r0 = 0; exit
a subprog, unwind_goto follows each call r0 = 0; unwind_exit
of it
all other subprogs, thus all subprogs in C not possible: exit from the
loop
'call f; unwind_goto pad' continues at pad, the cleanup code of the caller,
when f leaves with unwind_exit. Thus the callers release what they hold.
The verifier rejects a prog in which the selected edge is not safe.
The main pass selects and verifies the edge, and records it in
insn_aux_data. bpf_insert_loop_guards() adds the insns after the main pass.
No code sets loop_guard in this patch, thus no prog changes.
short form long form
may_goto target may_goto +1
insn goto +N the usual path: skip N insns
rX = 0, ...
goto target or: exit
insn
'r0 = 0; exit' goes after the last insn of the subprog, one time for all
its added may_goto. The long form has one more jump in each iteration. It
has two causes:
- may_goto has a 16-bit offset. A may_goto that is more than 4096 insns
before the end of the subprog gets its own 'r0 = 0; exit'.
- The usual place of may_goto is before a conditional jump that exits the
loop, with the same target. When no such jump is in each iteration,
may_goto is at the loop head and its edge skips the body. Then the code
after the loop can read registers and stack slots that have no value at
the loop head. The long form sets them to 0.
Progs without such a loop get the same code. An added may_goto costs the
same as a may_goto of the prog. On x86-64 (Zen 5):
each iteration, tight loop 7 insns +0.17 .. +0.24 nsec
once in 65535 iterations read of the clock 150 nsec
each run 2 stores, 16 bytes of stack in each frame
may_goto in a subprog 4 insns in each run, 16 bytes per cpu,
5 insns at the entry and 7 at the exit of
each call
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 13 +++
kernel/bpf/fixups.c | 216 +++++++++++++++++++++++++++++++++++
kernel/bpf/verifier.c | 3 +
3 files changed, 232 insertions(+)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index f0f054f98dc1..31b1fa866e49 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -807,6 +807,11 @@ struct bpf_insn_aux_data {
u64 cast_arena:1; /* addr_space_cast of an address in arena */
u64 cast_mov:1; /* addr_space_cast of a pointer that is not in arena, it's a move */
u64 bb_end:1; /* last instruction of a basic block */
+ u64 loop_guard:1; /* may_goto is added in front of this instruction */
+ u64 guard_place:1; /* it's the place for may_goto if the loop needs one */
+ u64 guard_at_header:1; /* there is no better place than the head of the loop */
+ u64 guard_unwind:1; /* the subprog unwinds when may_goto is out of budget */
+ u64 guard_no_unwind:1; /* it cannot unwind from here, at least sometimes */
u64 need_scev:1;
u64 alu_state:5; /* used in combination with alu_limit */
/* true if STX or LDX instruction is a part of a spill/fill
@@ -870,6 +875,11 @@ struct bpf_insn_aux_data {
* For a loop header, identifies the parent loop instead.
*/
s32 loop_header;
+ /* where may_goto in front of this instruction goes */
+ s32 guard_to;
+ /* registers and stack slots that are set to 0 on the way: read there, not written here */
+ u16 guard_zero;
+ u64 guard_zero_slots;
/* additional information about the loop if this instruction is a loop header */
struct bpf_loop *loop;
};
@@ -1010,6 +1020,8 @@ struct bpf_subprog_info {
* a hidden pointer to the result, a pair in two registers, no dead arguments.
*/
bool stack_args_from_code: 1;
+ /* it can leave with unwind_exit: it has one, or unwind_goto follows every call of it */
+ bool bail_unwinds: 1;
u8 code_stack_args:4;
u8 arg_slot_cnt:4;
@@ -1450,6 +1462,7 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
bool widen);
void bpf_join_free(struct bpf_verifier_env *env);
+int bpf_insert_loop_guards(struct bpf_verifier_env *env);
int bpf_insert_unwound_reports(struct bpf_verifier_env *env);
bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n);
bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 6a591c954c31..937274c164e8 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -393,6 +393,222 @@ static int bpf_adj_moved_insn(struct bpf_insn *insn, u32 delta)
return 0;
}
+static bool jumps_to_itself(const struct bpf_insn *insn)
+{
+ u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+
+ return (class == BPF_JMP || class == BPF_JMP32) && op != BPF_CALL && op != BPF_EXIT &&
+ insn->code != (BPF_JMP32 | BPF_JA) && insn->off == -1;
+}
+
+/* How a subprog leaves when it is out of budget, see push_budget_exit() */
+static struct bpf_insn budget_exit_insn(struct bpf_verifier_env *env, struct bpf_subprog_info *sub)
+{
+ if (!sub->bail_unwinds)
+ return BPF_EXIT_INSN();
+ sub->can_unwind = true;
+ env->prog->jit_required = 1;
+ return BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0);
+}
+
+/*
+ * may_goto has 16 bits for the offset, and what is between it and its target is going
+ * to grow: may_goto itself becomes 7 insns.
+ */
+#define BPF_GUARD_NEAR 4096
+
+static bool guard_leaves_subprog(const struct bpf_insn_aux_data *aux)
+{
+ return aux->loop_guard && !aux->guard_no_unwind;
+}
+
+/*
+ * 'r0 = 0; exit' behind the last insn of the subprog, for all its guards that are near.
+ * Returns the number of insns that were added.
+ */
+static int append_budget_exit(struct bpf_verifier_env *env, int s)
+{
+ struct bpf_subprog_info *sub = &env->subprog_info[s];
+ int last = (sub + 1)->start - 1, n = 0, i;
+ struct bpf_prog *new_prog;
+ struct bpf_insn buf[4];
+
+ for (i = max_t(int, sub->start, last + 1 - BPF_GUARD_NEAR); i <= last; i++)
+ if (guard_leaves_subprog(&env->insn_aux_data[i]))
+ break;
+ if (i > last)
+ return 0;
+
+ buf[n++] = env->prog->insnsi[last];
+ buf[n++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+ if (bpf_ret_reg_pair(env, s))
+ buf[n++] = BPF_MOV64_IMM(BPF_REG_2, 0);
+ buf[n++] = budget_exit_insn(env, sub);
+ new_prog = bpf_patch_insn_data(env, last, buf, n);
+ if (!new_prog)
+ return -ENOMEM;
+ env->prog = new_prog;
+ /* the original insn is the first one of the patch, not the last one */
+ swap(env->insn_aux_data[last], env->insn_aux_data[last + n - 1]);
+ for (i = last + 1; i < last + n; i++)
+ env->insn_aux_data[i].seen = 1;
+ return n - 1;
+}
+
+/*
+ * Add may_goto in front of the insns that the main pass marked, see
+ * check_loop_guard(). It verified the edge from the insn to where may_goto goes.
+ */
+int bpf_insert_loop_guards(struct bpf_verifier_env *env)
+{
+ struct bpf_insn_aux_data *aux = env->insn_aux_data;
+ int len = env->prog->len, i, r, s, to, n, jmp, delta = 0, err = 0;
+ struct bpf_insn *patch;
+ struct bpf_prog *new_prog;
+ u32 *before;
+ int *target, *leave, *moved;
+ u16 *zero;
+ u64 *slots;
+ s64 off;
+
+ for (i = 0; i < len; i++)
+ if (aux[i].loop_guard)
+ break;
+ if (i == len)
+ return 0;
+
+ /*
+ * Where the subprog has 'r0 = 0; exit', 0 if it has none. And how far that moved
+ * the subprog: guard_to is from the time before.
+ */
+ leave = kvcalloc(2 * env->subprog_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
+ if (!leave)
+ return -ENOMEM;
+ moved = leave + env->subprog_cnt;
+ for (s = env->subprog_cnt - 1; s >= 0; s--) {
+ leave[s] = append_budget_exit(env, s);
+ if (leave[s] < 0) {
+ err = leave[s];
+ kvfree(leave);
+ return err;
+ }
+ }
+ for (s = 0, n = 0; s < env->subprog_cnt; s++) {
+ moved[s] = n;
+ n += leave[s];
+ if (leave[s])
+ leave[s] = env->subprog_info[s + 1].start - leave[s];
+ }
+ aux = env->insn_aux_data;
+ len = env->prog->len;
+
+ /* the number of insns added in front of the insn, where the guard of the insn goes */
+ before = kvcalloc(len + 1, sizeof(u32), GFP_KERNEL_ACCOUNT);
+ target = kvcalloc(len, sizeof(int), GFP_KERNEL_ACCOUNT);
+ zero = kvcalloc(len, sizeof(u16), GFP_KERNEL_ACCOUNT);
+ slots = kvcalloc(len, sizeof(u64), GFP_KERNEL_ACCOUNT);
+ patch = kvcalloc(80, sizeof(*patch), GFP_KERNEL_ACCOUNT);
+ if (!before || !target || !zero || !slots || !patch) {
+ err = -ENOMEM;
+ goto out;
+ }
+ for (i = 0; i < len; i++) {
+ s = bpf_find_containing_subprog(env, i) - env->subprog_info;
+ target[i] = aux[i].loop_guard ? aux[i].guard_to + moved[s] : -1;
+ zero[i] = aux[i].loop_guard ? aux[i].guard_zero : 0;
+ slots[i] = aux[i].loop_guard ? aux[i].guard_zero_slots : 0;
+ if (guard_leaves_subprog(&aux[i])) {
+ /* see push_budget_exit() */
+ slots[i] = 0;
+ if (leave[s] && leave[s] - i <= BPF_GUARD_NEAR) {
+ target[i] = leave[s];
+ zero[i] = 0;
+ } else {
+ target[i] = -2;
+ zero[i] = BIT(BPF_REG_0);
+ if (bpf_ret_reg_pair(env, s))
+ zero[i] |= BIT(BPF_REG_2);
+ }
+ }
+ before[i + 1] = before[i];
+ if (aux[i].loop_guard)
+ before[i + 1] += zero[i] || slots[i] ?
+ 3 + hweight16(zero[i]) + hweight64(slots[i]) : 1;
+ }
+
+ for (i = 0; i < len; i++) {
+ to = target[i];
+ if (to == -1)
+ continue;
+ /*
+ * may_goto to or may_goto +1
+ * insn goto +N
+ * rX = 0, ...
+ * goto to, or exit
+ * insn
+ */
+ n = 0;
+ if (zero[i] || slots[i]) {
+ patch[n++] = BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 1, 0);
+ patch[n++] = BPF_JMP_A(hweight16(zero[i]) + hweight64(slots[i]) + 1);
+ for (r = BPF_REG_0; r < BPF_REG_FP; r++)
+ if (zero[i] & BIT(r))
+ patch[n++] = BPF_MOV64_IMM(r, 0);
+ for (r = 0; r < 64; r++)
+ if (slots[i] & BIT_ULL(r))
+ patch[n++] = BPF_ST_MEM(BPF_DW, BPF_REG_FP,
+ -(r + 1) * BPF_REG_SIZE, 0);
+ }
+ jmp = n++;
+ /*
+ * What is behind moves with this patch and is adjusted by
+ * the patches to come. What is in front is where it is going to be.
+ */
+ if (to == -2)
+ off = 0;
+ else if (to > i)
+ off = to - i + n - jmp - 1;
+ else
+ off = (s64)to + before[to] - (i + delta + jmp) - 1;
+ if (!zero[i] && !slots[i] && (off > S16_MAX || off < S16_MIN)) {
+ verbose(env, "exit of the loop is too far from insn %d for may_goto\n", i);
+ err = -ERANGE;
+ break;
+ }
+ patch[jmp] = zero[i] || slots[i] ? BPF_JMP32_A(off) :
+ BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, off, 0);
+ if (to == -2)
+ patch[jmp] = budget_exit_insn(env, bpf_find_containing_subprog(env,
+ i + delta));
+ patch[n] = env->prog->insnsi[i + delta];
+ /* A jump to itself is a back edge. It goes through may_goto like the others. */
+ if (jumps_to_itself(&patch[n]))
+ patch[n].off -= n;
+ else
+ err = bpf_adj_moved_insn(&patch[n], n);
+ if (err) {
+ verbose(env, "insn %d cannot be patched due to 16-bit range\n", i);
+ break;
+ }
+ n++;
+ new_prog = bpf_patch_insn_data(env, i + delta, patch, n);
+ if (!new_prog) {
+ err = -ENOMEM;
+ break;
+ }
+ env->prog = new_prog;
+ delta += n - 1;
+ }
+out:
+ kvfree(leave);
+ kvfree(before);
+ kvfree(target);
+ kvfree(zero);
+ kvfree(slots);
+ kvfree(patch);
+ return err;
+}
+
/*
* For all jmp insns in a given 'prog' that point to 'tgt_idx' insn adjust the
* jump offset by 'delta'.
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 32b256bc1513..c3fac4586c92 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -24393,6 +24393,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret == 0 && env->callx_edges)
ret = sort_subprogs_topo(env);
+ if (ret == 0)
+ ret = bpf_insert_loop_guards(env);
+
if (ret == 0 && bpf_prog_is_offloaded(env->prog->aux))
ret = bpf_prog_offload_finalize(env);
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 50/81] bpf: Add may_goto to the loops that the verifier marks
2026-10-26 2:59 ` [PATCH bpf 50/81] bpf: Add may_goto to the loops that the verifier marks Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 7662 bytes --]
> bpf: Add may_goto to the loops that the verifier marks
TL;DR:
- append_budget_exit() swap() may clear indirect_target on a gotox target,
possibly causing a missing-ENDBR/BTI fault
- Loop guards inserted for offloaded (nfp) programs never reach the NIC, so
unbounded loops may run unguarded
> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> index 6a591c954c31..937274c164e8 100644
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c
> @@ -393,6 +393,222 @@ static int bpf_adj_moved_insn(struct bpf_insn *insn, u32 delta)
[ ... ]
> +static int append_budget_exit(struct bpf_verifier_env *env, int s)
> +{
> + struct bpf_subprog_info *sub = &env->subprog_info[s];
> + int last = (sub + 1)->start - 1, n = 0, i;
> + struct bpf_prog *new_prog;
> + struct bpf_insn buf[4];
> +
> + for (i = max_t(int, sub->start, last + 1 - BPF_GUARD_NEAR); i <= last; i++)
> + if (guard_leaves_subprog(&env->insn_aux_data[i]))
> + break;
> + if (i > last)
> + return 0;
> +
> + buf[n++] = env->prog->insnsi[last];
> + buf[n++] = BPF_MOV64_IMM(BPF_REG_0, 0);
> + if (bpf_ret_reg_pair(env, s))
> + buf[n++] = BPF_MOV64_IMM(BPF_REG_2, 0);
> + buf[n++] = budget_exit_insn(env, sub);
> + new_prog = bpf_patch_insn_data(env, last, buf, n);
> + if (!new_prog)
> + return -ENOMEM;
> + env->prog = new_prog;
> + /* the original insn is the first one of the patch, not the last one */
> + swap(env->insn_aux_data[last], env->insn_aux_data[last + n - 1]);
> + for (i = last + 1; i < last + n; i++)
> + env->insn_aux_data[i].seen = 1;
> + return n - 1;
> +}
Can this swap() move indirect_target off the original last insn of the
subprog and onto the appended exit?
bpf_patch_insn_data() uses BPF_PATCH_KEEP_TARGET, and adjust_insn_aux_data()
has already put the original aux into the last slot of the patch. It also
already relocates the flag that belongs to the first slot:
if (mode == BPF_PATCH_KEEP_TARGET && data[off + cnt - 1].indirect_target) {
data[off].indirect_target = 1;
data[off + cnt - 1].indirect_target = 0;
}
non_stack_access is moved the same way, through the memcmp() against
original_insn.
So before the swap(), insn_aux_data[last] is the new entry with
indirect_target = 1, and insn_aux_data[last + n - 1] is the original aux
with indirect_target = 0. The swap() exchanges them as whole structs, so
the original insn at 'last' ends up with indirect_target cleared, and the
appended 'exit' or 'unwind_exit' ends up with it set.
A gotox jump table entry that pointed at 'last' still points there, because
adjust_insn_arrays() is called with first = off + 1. Nothing marks indirect
targets again after the main pass, since mark_indirect_target() is only
called from check_indirect_jump(). The JITs emit ENDBR (x86) or BTI J
(arm64) only where bpf_insn_is_indirect_target() is true, and gotox is
emitted as a plain indirect jump.
This looks reachable once check_loop_guard() starts setting loop_guard:
- A subprog, for example the main prog, has a loop that is not counted.
- The guard leaves the subprog (loop_guard && !guard_no_unwind) and is
within BPF_GUARD_NEAR insns of the end, so append_budget_exit() appends
'r0 = 0; exit'.
- The last insn of the subprog (exit, ja or gotol, see check_subprogs()) is
one of the targets of a gotox jump table, for example a switch case that
returns with r0 already set and jumps straight to the final exit.
When gotox takes that entry on a CPU with kernel IBT, the jump lands on an
insn without ENDBR, which would be a #CP fault ("Missing ENDBR" and BUG() in
do_kernel_cp_fault()). With arm64 BTI it would be a BTI exception in the
kernel. I did not see a later patch in the series that changes
append_budget_exit() or adjust_insn_aux_data().
Should indirect_target (and non_stack_access) stay on 'last' after the swap,
for example by swapping them back, or by copying them from
data[last + n - 1] and clearing them there? Alternatively, only the fields
that adjust_insn_aux_data() did not already relocate could be moved.
[ ... ]
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 32b256bc1513..c3fac4586c92 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -24393,6 +24393,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
> if (ret == 0 && env->callx_edges)
> ret = sort_subprogs_topo(env);
>
> + if (ret == 0)
> + ret = bpf_insert_loop_guards(env);
> +
> if (ret == 0 && bpf_prog_is_offloaded(env->prog->aux))
> ret = bpf_prog_offload_finalize(env);
>
Is it ok to run bpf_insert_loop_guards() for offloaded programs, and before
bpf_prog_offload_finalize()?
For an offloaded (nfp) program, nfp_bpf_verifier_prep() calls
nfp_prog_prepare(nfp_prog, prog->insnsi, prog->len) before the main pass.
That copies every insn into the device's own meta list with meta->n set to
the original index. nfp_bpf_translate() -> nfp_bpf_jit() then compiles only
from that meta list, so insns added later to env->prog->insnsi never reach
the NIC. instr_cb[] in nfp/bpf/jit.c has no entry for BPF_JMP | BPF_JCOND,
so the nfp_verify_insn() opcode check that would reject may_goto never runs
on the added insns.
nfp_bpf_finalize() ends with nfp_bpf_insn_flag_zext(nfp_prog,
env->insn_aux_data), which tests aux[meta->n].zext_dst.
bpf_insert_loop_guards() calls bpf_patch_insn_data(), and
append_budget_exit() adds 'r0 = 0; exit' at the end of subprogs. Both move
insn_aux_data, so meta->n no longer indexes the right entry for any insn
after the first insertion.
check_loop_guard(), added by 0535bc235aed ("bpf: Verify programs as a
fixpoint over the control flow graph"), sets aux->loop_guard and has no
bpf_prog_is_offloaded() check. The series does consider offload when it
computes bail_unwinds:
si[s].bail_unwinds = si[s].can_unwind ||
(s && ... && bpf_jit_supports_unwind() &&
!bpf_prog_is_offloaded(env->prog->aux));
so an offloaded program is expected to get the 'r0 = 0; exit' form of the
guard. bpf_check() at the series tip (51224d16c11b) still calls
bpf_insert_loop_guards() right before bpf_prog_offload_finalize().
Once 0535bc235aed is applied, an offloaded XDP/TC program (nfp supports
backward jumps since 854dc87d1a7b "nfp: bpf: support backward jump") with a
loop that the verifier cannot count was rejected before the series. Now it
looks like it is accepted because the verifier relies on the added may_goto
to stop the loop at run time. On the NIC it runs with no guard, so a loop
that never ends there could stall the device datapath. That conflicts with
the commit message premise that "the kernel must stop such a loop at run
time". Independently, the zext flags that the NIC JIT uses would come from
the wrong insns.
The comment further down in bpf_check(), "instruction rewrites happen after
this point", also no longer holds, since it marks the point up to which
insn indices match the loaded program.
At this commit the call is a no-op because nothing sets loop_guard yet. It
becomes reachable with d42f1667f490 / 0535bc235aed, and I did not see a
later commit in the series that addresses it.
Should a loop that needs a guard be rejected when bpf_prog_is_offloaded()
(for example in check_loop_guard(), or by failing in
bpf_insert_loop_guards())? Or at least, should the insertion move after
bpf_prog_offload_finalize() so that the aux indices nfp reads stay valid?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 51/81] bpf: Check stack and count calls of recursive subprogs at run time
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (49 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 50/81] bpf: Add may_goto to the loops that the verifier marks Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 4:09 ` bot+bpf-ci
2026-10-26 12:51 ` kernel test robot
2026-10-26 2:59 ` [PATCH bpf 52/81] bpf: Add the main pass as a fixpoint over the control flow graph Alexei Starovoitov
` (31 subsequent siblings)
82 siblings, 2 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier rejects recursion, because it cannot know at load time how
deep the calls go and how much stack they use.
Limit recursion at run time, as may_goto limits loops. Each recursive
subprog, that is each subprog on a cycle of the call graph, starts with a
check. The check fails when the stack has no room for one more frame, or
when this run of the prog has no calls left. Then the subprog returns 0
immediately. At its first insn the subprog holds no lock and no reference
of its own, thus it can always exit there.
entry: depth += weight of the frame
depth > limit ? --- yes ---+
calls += 1 |
calls > BPF_MAX_LOOPS ? --- yes ---+
| no v
v report, one time in a run
body of the subprog r0 = 0
| |
v v
exit: depth -= weight of the frame
depth and calls are per-cpu counters of the prog. The main prog clears
calls when a run starts. bpf_throw() subtracts the weights of the frames
that it removes, because they do not get to their exits. If unwind_goto
follows each call of the subprog, a failed check leaves with unwind_exit.
Then the callers continue at their cleanup code.
limit, for one prog on one cpu value
----------------------------------------- ------------------------------
kernel stack of recursive frames 4096 bytes
stack in arena (r12) of recursive frames the part that is free
calls of recursive subprogs in one run BPF_MAX_LOOPS
The stack limits are in addition to the deepest chain of calls without
recursion. One counter serves the two stacks: the weight of a frame is the
larger of its shares of the two limits. The check counts calls because a
subprog that calls itself two times can run for a very long time at a small
depth.
Progs without a recursive subprog get the same code. A recursive subprog
has the cost in each call, also when it does not recurse. The three updates
of the counters are fetch_add insns. They are BPF_ATOMIC_LOCAL where the
JIT supports it: on x86 they have no lock prefix. On x86-64 (Zen 5) a call
without the check costs 0.65-0.95 nsec. The check adds:
each run 3 insns in the main prog +0.44 nsec
each call 8 insns at the entry, 4 at each exit +2.66 nsec
first failed check report to stderr about 5 usec + 0.22 usec
of a run per frame
The first failed check of a run sets 'unwound' in bpf_prog_info. It ends
the 250 ms of may_goto for this run, thus the loops of the run exit too. It
prints the stack to the stderr stream of the prog after one of these lines:
ERROR: No room in the stack for a call of a recursive function
ERROR: Too many calls of recursive functions
Limitations:
- The check does not read the stack pointer. Thus recursive_frame_size()
must be an upper bound of the frame that the JIT makes. The verifier
calculates it after it inlines bpf_loop(), which adds 3 stack slots and
3 registers.
- Only fentry can attach to a recursive subprog. The trampoline of fexit
stays on the stack at each level, and the check does not count it.
- Progs that nest on a cpu do not share the limits: their stack use adds
up. Two runs of the same prog that nest on a cpu share the stack limits.
No code marks a subprog as recursive in this patch.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 12 ++
include/linux/bpf_verifier.h | 5 +
kernel/bpf/core.c | 31 +++++
kernel/bpf/fixups.c | 263 +++++++++++++++++++++++++++++++++++
kernel/bpf/helpers.c | 8 ++
kernel/bpf/verifier.c | 16 +++
6 files changed, 335 insertions(+)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 1b1740d8ca8d..2ff1a2c97328 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1889,6 +1889,13 @@ struct bpf_may_goto_budget {
u64 count; /* what is left until the next look at the clock, 0: the time is up */
};
+/* What the recursive subprogs of a prog have used on this cpu, see bpf_insert_recursion_checks() */
+struct bpf_recursion_budget {
+ u32 calls; /* the number of calls in this run of the prog */
+ u32 reported; /* it was said that the run is out of budget */
+ u32 depth; /* the weights of the frames that are there */
+};
+
struct bpf_prog_aux {
atomic64_t refcnt;
u32 used_map_cnt;
@@ -1937,6 +1944,8 @@ struct bpf_prog_aux {
bool might_sleep;
bool can_unwind;
bool unwound; /* main prog: see bpf_prog_info */
+ /* on a cycle of the call graph: what it adds to call_depth of the main prog */
+ u32 recursive;
bool kprobe_write_ctx;
/* has insns that work with the arena base, see bpf_prog_has_arena_insns() */
bool uses_arena_insns;
@@ -1962,6 +1971,8 @@ struct bpf_prog_aux {
const char *attach_func_name;
struct bpf_prog **func;
struct bpf_prog_aux *main_prog_aux;
+ /* frames of recursive subprogs on this cpu, see bpf_insert_recursion_checks() */
+ struct bpf_recursion_budget __percpu *call_depth;
/* main prog: there when a subprog has may_goto */
struct bpf_may_goto_budget __percpu *may_goto_budget;
void *jit_data; /* JIT specific data. arch dependent */
@@ -3464,6 +3475,7 @@ void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr);
void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr);
void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip);
void bpf_prog_begin_run(const struct bpf_prog *prog);
+u64 bpf_prog_report_recursion(void);
u64 bpf_prog_report_unwound(u64 ret);
static __always_inline u32
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 31b1fa866e49..16987fa282b3 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -812,6 +812,7 @@ struct bpf_insn_aux_data {
u64 guard_at_header:1; /* there is no better place than the head of the loop */
u64 guard_unwind:1; /* the subprog unwinds when may_goto is out of budget */
u64 guard_no_unwind:1; /* it cannot unwind from here, at least sometimes */
+ u64 recursion_bail:1; /* the first insn of the way out, bpf_insert_recursion_reports() */
u64 need_scev:1;
u64 alu_state:5; /* used in combination with alu_limit */
/* true if STX or LDX instruction is a part of a spill/fill
@@ -999,6 +1000,7 @@ struct bpf_subprog_info {
bool has_tail_call: 1;
bool might_throw: 1;
bool can_unwind: 1; /* has unwind_exit */
+ bool recursive: 1; /* on a cycle of the call graph */
bool tail_call_reachable: 1;
bool has_ld_abs: 1;
bool is_cb: 1;
@@ -1027,6 +1029,7 @@ struct bpf_subprog_info {
enum priv_stack_mode priv_stack_mode;
struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
+ u32 recursion_weight; /* see bpf_insert_recursion_checks() */
u16 stack_arg_cnt; /* incoming + max outgoing */
u16 max_out_stack_arg_cnt;
};
@@ -1463,9 +1466,11 @@ int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst
void bpf_join_free(struct bpf_verifier_env *env);
int bpf_insert_loop_guards(struct bpf_verifier_env *env);
+int bpf_insert_recursion_reports(struct bpf_verifier_env *env);
int bpf_insert_unwound_reports(struct bpf_verifier_env *env);
bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n);
bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
+int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
int bpf_split_cur_state(struct bpf_verifier_env *env);
int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index bb0aeda9f9cc..19d8191053b9 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -292,6 +292,7 @@ void __bpf_prog_free(struct bpf_prog *fp)
mutex_destroy(&fp->aux->used_maps_mutex);
mutex_destroy(&fp->aux->dst_mutex);
kfree(fp->aux->poke_tab);
+ free_percpu(fp->aux->call_depth);
free_percpu(fp->aux->may_goto_budget);
kfree(fp->aux);
}
@@ -3640,6 +3641,32 @@ static noinline void bpf_prog_report_may_goto_violation(void)
#endif
}
+/* Called by a recursive subprog that is out of budget, see bpf_insert_recursion_checks() */
+u64 bpf_prog_report_recursion(void)
+{
+#ifdef CONFIG_BPF_SYSCALL
+ struct bpf_stream_stage ss;
+ struct bpf_prog *prog;
+ bool many;
+
+ prog = bpf_prog_find_from_stack();
+ if (!prog)
+ return 0;
+ WRITE_ONCE(prog->aux->unwound, true);
+ this_cpu_write(prog->aux->call_depth->reported, 1);
+ many = this_cpu_read(prog->aux->call_depth->calls) >= BPF_MAX_LOOPS;
+ /* the loops of the run come to an end as well */
+ if (prog->aux->may_goto_budget)
+ this_cpu_write(prog->aux->may_goto_budget->count, 0);
+ bpf_stream_stage(ss, prog, BPF_STDERR, ({
+ bpf_stream_printk(ss, many ? "ERROR: Too many calls of recursive functions\n" :
+ "ERROR: No room in the stack for a call of a recursive function\n");
+ bpf_stream_dump_stack(ss);
+ }));
+#endif
+ return 0;
+}
+
/*
* Nothing caught the unwinding. The prog or the async callback returns to the kernel.
* There is no return value that is right for every hook, so what to do about it is up
@@ -3677,6 +3704,10 @@ void bpf_prog_begin_run(const struct bpf_prog *prog)
this_cpu_write(aux->may_goto_budget->start, 0);
this_cpu_write(aux->may_goto_budget->count, BPF_MAX_TIMED_LOOPS);
}
+ if (aux->call_depth) {
+ this_cpu_write(aux->call_depth->calls, 0);
+ this_cpu_write(aux->call_depth->reported, 0);
+ }
}
static struct bpf_may_goto_budget *bpf_may_goto_budget(void)
diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
index 937274c164e8..4a0b55c5fe35 100644
--- a/kernel/bpf/fixups.c
+++ b/kernel/bpf/fixups.c
@@ -647,6 +647,268 @@ static int adjust_jmp_off(struct bpf_prog *prog, u32 tgt_idx, u32 delta)
return 0;
}
+/*
+ * The room that the frame of a subprog takes besides its stack: what JIT saves at most.
+ * arm64: FP, LR, 8 registers, the tail call count: 96. loongarch: 96, riscv: 80.
+ * powerpc: 112 with ELF ABI v2, 192 with v1.
+ */
+#define BPF_FRAME_OVERHEAD (IS_ENABLED(CONFIG_PPC64) ? 192 : 96)
+
+/* the room for the frames of recursive subprogs */
+#define BPF_RECURSION_STACK 4096
+/* the frames of the helper or the kfunc that is between a callback and its caller */
+#define BPF_CALLBACK_OVERHEAD 256
+
+static u32 recursive_frame_size(struct bpf_verifier_env *env, int s)
+{
+ struct bpf_subprog_info *sub = &env->subprog_info[s];
+ struct bpf_insn *insn = env->prog->insnsi;
+ u32 size, used = 0, may_goto = 0, cb = sub->is_cb ? BPF_CALLBACK_OVERHEAD : 0;
+ int i;
+
+ /* may_goto is yet to add its 16 bytes */
+ if (!IS_ENABLED(CONFIG_X86_64))
+ return round_up(sub->stack_depth + 16, 32) + BPF_FRAME_OVERHEAD + cb +
+ sub->max_out_stack_arg_cnt * BPF_REG_SIZE;
+
+ for (i = sub->start; i < (sub + 1)->start; i++) {
+ if (insn[i].dst_reg >= BPF_REG_6 && insn[i].dst_reg <= BPF_REG_9)
+ used |= BIT(insn[i].dst_reg);
+ if (insn[i].src_reg >= BPF_REG_6 && insn[i].src_reg <= BPF_REG_9)
+ used |= BIT(insn[i].src_reg);
+ if (bpf_is_may_goto_insn(&insn[i]))
+ may_goto = 16;
+ }
+ /* the return address, rbp, the registers that are saved */
+ size = 16 + hweight32(used) * 8;
+ /* the base of arena, the slot for r12 */
+ if (env->prog->aux->arena)
+ size += 16;
+ /* the tail call count and the pointer to it */
+ if (sub->tail_call_reachable)
+ size += 16;
+ return size + cb + round_up(sub->stack_depth + may_goto, 8) +
+ sub->max_out_stack_arg_cnt * BPF_REG_SIZE;
+}
+
+/* what the frames of recursive subprogs add up to at most */
+#define BPF_RECURSION_UNITS 65536
+
+/*
+ * How deep a recursion goes is not known, so it is counted at run time. In the same
+ * spirit as may_goto: the program goes on for as long as there is budget and
+ * takes a way out that was verified when there is none.
+ *
+ * Every subprog on a cycle of the call graph adds its weight to a per-cpu
+ * counter of the program when it is entered and takes it back at every exit.
+ * The weight is the part of the budget that its frame takes: of the stack
+ * room, or of the room in the stack in arena, whichever is larger. So these
+ * frames take no more than BPF_RECURSION_STACK, on top of the chain of calls
+ * without recursion that check_max_stack_depth() looked at.
+ *
+ * A subprog that calls itself twice doesn't have to go deep to go on forever. So
+ * the calls are counted too: BPF_MAX_LOOPS of them in a run of the prog, like the
+ * callbacks of bpf_loop(). Once they are used up every call returns at once, and
+ * the frames that are there come to an end.
+ *
+ * Another run of the program on the same cpu, in interrupt or after
+ * preemption, counts in the same counter. Then there is less for both.
+ */
+int bpf_insert_recursion_checks(struct bpf_verifier_env *env)
+{
+ struct bpf_subprog_info *si = env->subprog_info;
+ u32 frame, min_frame = U32_MAX, arena_frame = 0, arena_room = 0, weight;
+ u32 limit = BPF_RECURSION_STACK;
+ struct bpf_insn buf[24], first;
+ struct bpf_prog *new_prog;
+ struct bpf_recursion_budget __percpu *depth;
+ /* Only this cpu touches the counters */
+ int fetch_add = BPF_ADD | BPF_FETCH |
+ (bpf_jit_supports_local_atomics() ? BPF_ATOMIC_LOCAL : 0);
+ int i, s, n, bail, bail_len, err;
+
+ for (s = 0; s < env->subprog_cnt; s++) {
+ if (!si[s].recursive)
+ continue;
+ frame = recursive_frame_size(env, s);
+ si[s].recursion_weight = DIV_ROUND_UP(frame * BPF_RECURSION_UNITS, limit);
+ min_frame = min(min_frame, frame);
+ arena_frame = max(arena_frame, si[s].arena_stack_depth);
+ }
+ if (min_frame == U32_MAX)
+ return 0;
+ if (arena_frame) {
+ /*
+ * One more frame is for the subprog that finds that there is no
+ * budget: it has its frame by then.
+ */
+ u32 room = BPF_ARENA_STACK_MAX - env->max_arena_stack_depth;
+
+ if (room < 2 * arena_frame) {
+ verbose(env, "no room in the stack in arena for recursion\n");
+ return -EACCES;
+ }
+ /* no more than the frames that fit in the stack can take */
+ arena_room = min(room - arena_frame, limit / min_frame * arena_frame);
+ env->max_arena_stack_depth += arena_room + arena_frame;
+ }
+ for (s = 0; s < env->subprog_cnt; s++) {
+ if (!si[s].recursive)
+ continue;
+ if (si[s].arena_stack_depth)
+ si[s].recursion_weight = max(si[s].recursion_weight,
+ DIV_ROUND_UP(si[s].arena_stack_depth *
+ BPF_RECURSION_UNITS, arena_room));
+ if (si[s].recursion_weight > BPF_RECURSION_UNITS) {
+ verbose(env, "the frame of recursive %s() is too large\n",
+ bpf_subprog_name(env, s));
+ return -EACCES;
+ }
+ }
+
+ depth = alloc_percpu_gfp(struct bpf_recursion_budget, GFP_KERNEL_ACCOUNT);
+ if (!depth)
+ return -ENOMEM;
+ env->prog->aux->call_depth = depth;
+ env->prog->jit_required = 1;
+
+ /* From the last one, so that nothing moves in those that are yet to be done. */
+ for (s = env->subprog_cnt - 1; s > 0; s--) {
+ if (!si[s].recursive)
+ continue;
+ /*
+ * R0 has nothing at the entry. The address is not for AX:
+ * blinding of constants takes ld_imm64 apart with the help of
+ * AX. It leaves alone the insns that have AX in them.
+ */
+ n = 0;
+ first = env->prog->insnsi[si[s].start];
+ /* The exit drops the frame in the stack in arena. Take it before the check. */
+ if (is_arena_sp_adj(&first)) {
+ buf[n++] = first;
+ first = BPF_JMP_A(0);
+ }
+ memcpy(buf + n, (struct bpf_insn[]){ BPF_LD_IMM64(BPF_REG_0, (long)depth) },
+ 2 * sizeof(buf[0]));
+ n += 2;
+ buf[n++] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
+ weight = si[s].recursion_weight;
+ bail_len = bpf_ret_reg_pair(env, s) ? 3 : 2;
+ buf[n++] = BPF_MOV64_IMM(BPF_REG_AX, weight);
+ buf[n++] = BPF_ATOMIC_OP(BPF_W, fetch_add, BPF_REG_0, BPF_REG_AX,
+ offsetof(struct bpf_recursion_budget, depth));
+ buf[n++] = BPF_JMP_IMM(BPF_JGT, BPF_REG_AX, BPF_RECURSION_UNITS - weight, 3);
+ buf[n++] = BPF_MOV64_IMM(BPF_REG_AX, 1);
+ buf[n++] = BPF_ATOMIC_OP(BPF_W, fetch_add, BPF_REG_0, BPF_REG_AX,
+ offsetof(struct bpf_recursion_budget, calls));
+ buf[n++] = BPF_JMP_IMM(BPF_JLT, BPF_REG_AX, BPF_MAX_LOOPS, bail_len);
+ bail = n;
+ buf[n++] = BPF_MOV64_IMM(BPF_REG_0, 0);
+ if (bpf_ret_reg_pair(env, s))
+ buf[n++] = BPF_MOV64_IMM(BPF_REG_2, 0);
+ buf[n++] = budget_exit_insn(env, &si[s]);
+ buf[n] = first;
+ err = bpf_adj_moved_insn(&buf[n], n);
+ if (err) {
+ verbose(env, "insn %d cannot be patched due to 16-bit range\n",
+ env->insn_aux_data[si[s].start].orig_idx);
+ return err;
+ }
+ n++;
+ new_prog = __bpf_patch_insn_data(env, si[s].start, buf, n, BPF_PATCH_MOVE_TARGET);
+ if (!new_prog)
+ return -ENOMEM;
+ env->prog = new_prog;
+ env->insn_aux_data[si[s].start + bail].recursion_bail = true;
+ /* calls get to the check, jumps to the first insn do not */
+ err = adjust_jmp_off(env->prog, si[s].start, n - 1);
+ if (err)
+ return err;
+ }
+
+ /*
+ * The calls are counted anew when a run begins. For async callbacks see
+ * bpf_prog_begin_run(). R0 has nothing at the entry.
+ */
+ memcpy(buf, (struct bpf_insn[]){ BPF_LD_IMM64(BPF_REG_0, (long)depth) },
+ 2 * sizeof(buf[0]));
+ buf[2] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
+ /* calls and reported */
+ buf[3] = BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 0);
+ buf[4] = env->prog->insnsi[0];
+ err = bpf_adj_moved_insn(&buf[4], 4);
+ if (err) {
+ verbose(env, "insn %d cannot be patched due to 16-bit range\n",
+ env->insn_aux_data[0].orig_idx);
+ return err;
+ }
+ new_prog = __bpf_patch_insn_data(env, 0, buf, 5, BPF_PATCH_MOVE_TARGET);
+ if (!new_prog)
+ return -ENOMEM;
+ env->prog = new_prog;
+ /* a jump to the first insn doesn't begin a run */
+ err = adjust_jmp_off(env->prog, 0, 4);
+ if (err)
+ return err;
+
+ /* R1 is nothing to the caller. Jumps to exit get to what is added. */
+ for (i = 0; i < env->prog->len; i++) {
+ if (bpf_is_ldimm64(&env->prog->insnsi[i])) {
+ i++;
+ continue;
+ }
+ if (env->prog->insnsi[i].code != (BPF_JMP | BPF_EXIT) ||
+ !bpf_find_containing_subprog(env, i)->recursive)
+ continue;
+ n = 0;
+ memcpy(buf, (struct bpf_insn[]){ BPF_LD_IMM64(BPF_REG_1, (long)depth) },
+ 2 * sizeof(buf[0]));
+ n += 2;
+ buf[n++] = BPF_MOV64_PERCPU_REG(BPF_REG_1, BPF_REG_1);
+ weight = bpf_find_containing_subprog(env, i)->recursion_weight;
+ buf[n++] = BPF_MOV64_IMM(BPF_REG_AX, -(int)weight);
+ buf[n++] = BPF_ATOMIC_OP(BPF_W, fetch_add, BPF_REG_1, BPF_REG_AX,
+ offsetof(struct bpf_recursion_budget, depth));
+ buf[n++] = env->prog->insnsi[i];
+ new_prog = bpf_patch_insn_data(env, i, buf, n);
+ if (!new_prog)
+ return -ENOMEM;
+ env->prog = new_prog;
+ i += n - 1;
+ }
+ return 0;
+}
+
+/*
+ * Say so when a recursive subprog is out of budget, once in a run: from then on every call
+ * comes here. R0 is what bpf_insert_recursion_checks() left in it. The arguments are not
+ * needed any more.
+ * It's done after bpf_do_misc_fixups(), that takes a call for a call of a helper.
+ */
+int bpf_insert_recursion_reports(struct bpf_verifier_env *env)
+{
+ struct bpf_prog *new_prog;
+ struct bpf_insn buf[4];
+ int i;
+
+ for (i = 0; i < env->prog->len; i++) {
+ if (!env->insn_aux_data[i].recursion_bail)
+ continue;
+ env->insn_aux_data[i].recursion_bail = false;
+ buf[0] = BPF_LDX_MEM(BPF_W, BPF_REG_AX, BPF_REG_0,
+ offsetof(struct bpf_recursion_budget, reported));
+ buf[1] = BPF_JMP_IMM(BPF_JNE, BPF_REG_AX, 0, 1);
+ buf[2] = BPF_EMIT_CALL(bpf_prog_report_recursion);
+ buf[3] = env->prog->insnsi[i];
+ new_prog = bpf_patch_insn_data(env, i, buf, 4);
+ if (!new_prog)
+ return -ENOMEM;
+ env->prog = new_prog;
+ i += 3;
+ }
+ return 0;
+}
+
/*
* Say so when the main prog unwinds. JITs make exit of that, see bpf_prog_can_unwind().
* It's done after bpf_do_misc_fixups(), that takes a call for a call of a helper.
@@ -1936,6 +2198,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->changes_pkt_data = env->subprog_info[i].changes_pkt_data;
func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
func[i]->aux->can_unwind = env->subprog_info[i].can_unwind;
+ func[i]->aux->recursive = env->subprog_info[i].recursion_weight;
func[i]->aux->token = prog->aux->token;
if (!i)
func[i]->aux->exception_boundary = env->seen_exception;
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index cd960f65093d..45e85e7d54bf 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -3403,6 +3403,7 @@ struct bpf_throw_ctx {
u64 sp;
u64 bp;
int cnt;
+ u32 recursive;
};
static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
@@ -3421,6 +3422,7 @@ static bool bpf_stack_walker(void *cookie, u64 ip, u64 sp, u64 bp)
if (!prog)
return !ctx->cnt;
ctx->cnt++;
+ ctx->recursive += prog->aux->recursive;
if (bpf_is_subprog(prog))
return true;
ctx->aux = prog->aux;
@@ -3439,6 +3441,12 @@ __bpf_kfunc void bpf_throw(u64 cookie)
WARN_ON_ONCE(!ctx.aux->exception_boundary);
WARN_ON_ONCE(!ctx.bp);
WARN_ON_ONCE(!ctx.cnt);
+ /*
+ * The frames that go away do not get to their exits. Only these: another
+ * run of the program may be in the middle of its recursion on this cpu.
+ */
+ if (ctx.aux && ctx.recursive)
+ this_cpu_sub(ctx.aux->main_prog_aux->call_depth->depth, ctx.recursive);
/*
* Prevent KASAN false positives for CONFIG_KASAN_STACK by unpoisoning
* deeper stack depths than ctx.sp as we do not return from bpf_throw,
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index c3fac4586c92..a9628f5045fe 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -22925,6 +22925,15 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
bpf_log(log, "Only fentry can attach to %s, which can unwind\n", tname);
return -EINVAL;
}
+ /*
+ * A trampoline that calls the target stays on the stack at every level.
+ * The check of the stack in the target does not count it.
+ */
+ if (aux->func && aux->func[subprog]->aux->recursive && !prog_extension &&
+ prog->expected_attach_type != BPF_TRACE_FENTRY) {
+ bpf_log(log, "Only fentry can attach to %s, which is recursive\n", tname);
+ return -EINVAL;
+ }
conservative = aux->func_info_aux[subprog].unreliable;
if (prog_extension) {
if (aux->func_info_aux[subprog].linkage != BTF_FUNC_GLOBAL ||
@@ -24415,6 +24424,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret == 0)
ret = bpf_optimize_bpf_loop(env);
+ /* after what makes frames larger: inlined bpf_loop() takes 3 slots and 3 registers */
+ if (ret == 0)
+ ret = bpf_insert_recursion_checks(env);
+
if (ret == 0)
bpf_lower_cast_mov(env);
@@ -24437,6 +24450,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret == 0)
ret = bpf_do_misc_fixups(env);
+ if (ret == 0)
+ ret = bpf_insert_recursion_reports(env);
+
if (ret == 0)
ret = bpf_insert_unwound_reports(env);
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 51/81] bpf: Check stack and count calls of recursive subprogs at run time
2026-10-26 2:59 ` [PATCH bpf 51/81] bpf: Check stack and count calls of recursive subprogs at run time Alexei Starovoitov
@ 2026-10-26 4:09 ` bot+bpf-ci
2026-10-26 12:51 ` kernel test robot
1 sibling, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 4:09 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 9984 bytes --]
> bpf: Check stack and count calls of recursive subprogs at run time
TL;DR:
- Nested run may reset the call budget, so recursion may never stop: possible
RCU stall or soft lockup
- Frame size estimate may be too small, so the stack limit may be exceeded
- Possible misaligned 8-byte store; may_goto comment unclear
> diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
> index bb0aeda9f9cc..19d8191053b9 100644
> --- a/kernel/bpf/core.c
> +++ b/kernel/bpf/core.c
[ ... ]
> @@ -3640,6 +3641,32 @@ static noinline void bpf_prog_report_may_goto_violation(void)
> #endif
> }
>
> +/* Called by a recursive subprog that is out of budget, see bpf_insert_recursion_checks() */
> +u64 bpf_prog_report_recursion(void)
> +{
[ ... ]
> + /* the loops of the run come to an end as well */
> + if (prog->aux->may_goto_budget)
> + this_cpu_write(prog->aux->may_goto_budget->count, 0);
This isn't a bug, but does this end the loops of the run as the comment and
the commit message say ("It ends the 250 ms of may_goto for this run, thus
the loops of the run exit too")?
The code only sets run->count to 0 and leaves run->start alone, so the time
slice of the run does not end. Only frames entered after the report copy
count == 0 into their may_goto slot. Frames that are already on the stack
keep their own counts and go on looping.
When one of those reaches 1 -> 0, it calls bpf_check_timed_may_goto(). If
250 ms have not passed yet, that function does:
if (run)
run->count = BPF_MAX_TIMED_LOOPS;
return BPF_MAX_TIMED_LOOPS;
which puts the shared count back, and frames entered later loop normally
again.
For example, if the main prog has a may_goto loop that calls a recursive
subprog that runs out of budget, the loop still runs to the end of its
250 ms slice, and only the bail of each call gets cheaper.
Nothing becomes unsafe, since every frame is still bounded by its own
slice. Should this also make bpf_check_timed_may_goto() see that the time
is up, for example through run->start, or should the comment and the commit
message be reworded?
[ ... ]
> @@ -3677,6 +3704,10 @@ void bpf_prog_begin_run(const struct bpf_prog *prog)
> this_cpu_write(aux->may_goto_budget->start, 0);
> this_cpu_write(aux->may_goto_budget->count, BPF_MAX_TIMED_LOOPS);
> }
> + if (aux->call_depth) {
> + this_cpu_write(aux->call_depth->calls, 0);
> + this_cpu_write(aux->call_depth->reported, 0);
> + }
> }
Can a nested run of the same prog give the interrupted run a fresh call
budget? bpf_prog_begin_run() clears calls and reported without checking
whether another run of the prog is already in progress on this cpu. The
main prog does the same in its first insns, through the
BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 0) that bpf_insert_recursion_checks() adds.
The comment above bpf_insert_recursion_checks() says:
* Another run of the program on the same cpu, in interrupt or after
* preemption, counts in the same counter. Then there is less for both.
That holds for depth, which is never reset. For calls, the nested run sets
the counter back to 0, so the interrupted run gets more budget, not less.
For example, prog P recurses in process context, and a pinned bpf_timer of P
(BPF_F_TIMER_CPU_PIN) re-arms itself every millisecond:
bpf_timer_cb()
bpf_prog_begin_run(t->cb.prog)
this_cpu_write(aux->call_depth->calls, 0)
The interrupted run of P continues with a full BPF_MAX_LOOPS on every
expiry. A second run that preempts a sleepable prog on the same cpu does
the same through the inline store at the main prog entry. The same prog
can also nest on one cpu as a struct_ops, LSM or SYSCALL prog, or as a
networking prog that runs in process context with BH enabled.
The commit message says "a subprog that calls itself two times can run for
a very long time at a small depth". With f() { f(); f(); } and 16 byte
frames, the depth budget allows 256 levels, which is about 2^256 calls. If
a run of the same prog starts on that cpu more often than every
BPF_MAX_LOOPS calls (8M calls at about 2.7 nsec each is about 20 ms), the
outer run never runs out of calls. It holds rcu_read_lock() and
migrate_disable() the whole time, which leads to an RCU stall or soft lockup
for a non-sleepable prog, and an unkillable busy task for a sleepable one.
The nested run also clears reported, so the interrupted run reports again.
may_goto has the same reset in bpf_prog_begin_run(), but
bpf_check_timed_may_goto() still applies the timestamp of each frame. The
recursion budget has no such fallback.
Subprogs are first marked recursive by 0535bc235aed ("bpf: Verify programs
as a fixpoint over the control flow graph") later in this series. Neither
this reset nor the one at the main prog entry is changed through the end of
the series.
Should calls be reset only when no recursive frames of this prog are on the
cpu (depth == 0), or should a nested run share what the interrupted run has
left?
> diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c
> index 937274c164e8..4a0b55c5fe35 100644
> --- a/kernel/bpf/fixups.c
> +++ b/kernel/bpf/fixups.c
[ ... ]
> @@ -647,6 +647,268 @@ static int adjust_jmp_off(struct bpf_prog *prog, u32 tgt_idx, u32 delta)
> return 0;
> }
>
> +/*
> + * The room that the frame of a subprog takes besides its stack: what JIT saves at most.
[ ... ]
> +static u32 recursive_frame_size(struct bpf_verifier_env *env, int s)
> +{
> + struct bpf_subprog_info *sub = &env->subprog_info[s];
> + struct bpf_insn *insn = env->prog->insnsi;
> + u32 size, used = 0, may_goto = 0, cb = sub->is_cb ? BPF_CALLBACK_OVERHEAD : 0;
> + int i;
[ ... ]
> + for (i = sub->start; i < (sub + 1)->start; i++) {
> + if (insn[i].dst_reg >= BPF_REG_6 && insn[i].dst_reg <= BPF_REG_9)
> + used |= BIT(insn[i].dst_reg);
> + if (insn[i].src_reg >= BPF_REG_6 && insn[i].src_reg <= BPF_REG_9)
> + used |= BIT(insn[i].src_reg);
> + if (bpf_is_may_goto_insn(&insn[i]))
> + may_goto = 16;
> + }
> + /* the return address, rbp, the registers that are saved */
> + size = 16 + hweight32(used) * 8;
Is this an upper bound of the frame that the JIT builds on x86-64, as the
commit message requires ("recursive_frame_size() must be an upper bound of
the frame that the JIT makes")?
The function counts R6-R9 from the insns as they are when
bpf_insert_recursion_checks() runs, and reads stack_depth at that time.
The weight is fixed into si[s].recursion_weight and is not computed again.
bpf_check() calls it before the passes that can still make the frame
larger:
/* after what makes frames larger: inlined bpf_loop() takes 3 slots and 3 registers */
if (ret == 0)
ret = bpf_insert_recursion_checks(env);
...
if (ret == 0)
/* program is valid, convert *(u32*)(ctx + off) accesses */
ret = bpf_convert_ctx_accesses(env);
The x86 JIT's detect_reg_usage() scans the final insns, and
push_callee_regs() pushes every callee-saved register that it finds. Later
passes add registers and stack:
- bpf_convert_ctx_accesses() picks temporaries starting at BPF_REG_9 (then
R8, R7) when dst_reg == src_reg. Examples are SOCK_OPS_GET_FIELD in
sock_ops_convert_ctx_access(), bpf_convert_data_end_access() for SK_SKB,
SOCK_ADDR_STORE_NESTED_FIELD_OFF for sock_addr, and the treg = BPF_REG_9
conversions for cgroup sysctl and sockopt.
- insert_arena_stack_get() for an arena_stack_entry callback sets
sub->stack_depth = spill + MAX_BPF_FUNC_REG_ARGS * BPF_REG_SIZE (40 more
bytes). It also uses R6/R7/R8, and the arena exit path uses R6.
For example, a recursive static subprog that uses no R6-R9 and no stack has
an estimated frame of 16 bytes, so a weight of 256 and 256 frames for 65536
units. If ctx conversion adds R9, each frame is really 24 bytes, so the
recursion uses 6144 bytes instead of the 4096 that BPF_RECURSION_STACK
promises. The check never reads the stack pointer, so nothing else limits
this.
The call site comment "after what makes frames larger" is not accurate
either. mark_recursive_subprogs() in 0535bc235aed does not exclude these
prog types, and at the end of the series bpf_insert_recursion_checks() is
still called before bpf_convert_ctx_accesses() and bpf_insert_arena_stack(),
with recursive_frame_size() unchanged.
Could the checks run after bpf_convert_ctx_accesses() (they insert no helper
calls, so they could go before bpf_do_misc_fixups(), with the may_goto bytes
already counted in stack_depth), or could recursive_frame_size() always
count all of R6-R9 on x86-64?
[ ... ]
> + /*
> + * The calls are counted anew when a run begins. For async callbacks see
> + * bpf_prog_begin_run(). R0 has nothing at the entry.
> + */
> + memcpy(buf, (struct bpf_insn[]){ BPF_LD_IMM64(BPF_REG_0, (long)depth) },
> + 2 * sizeof(buf[0]));
> + buf[2] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
> + /* calls and reported */
> + buf[3] = BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 0);
Is this 8-byte store guaranteed to be 8-byte aligned? struct
bpf_recursion_budget has three u32 members, so alloc_percpu_gfp() asks for
4-byte alignment, and pcpu_alloc() keeps it because PCPU_MIN_ALLOC_SIZE is
4. The per-cpu address can then be 4 mod 8, and this becomes a misaligned
64-bit store on every run of the prog.
Some JITs that bpf_jit_supports_percpu_insn() allows have no
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS: riscv64 in its default
configuration, and loongarch with the default ARCH_STRICT_ALIGN=y. On those
arches the verifier sets env->strict_alignment and would reject such an
access from a program, but here the kernel inserts the access itself. It
traps and is emulated on every run, and on loongarch do_ale() calls
die_if_kernel() when unaligned emulation is disabled.
Would __aligned(8) on struct bpf_recursion_budget, or two BPF_W stores, be
better?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread* Re: [PATCH bpf 51/81] bpf: Check stack and count calls of recursive subprogs at run time
2026-10-26 2:59 ` [PATCH bpf 51/81] bpf: Check stack and count calls of recursive subprogs at run time Alexei Starovoitov
2026-10-26 4:09 ` bot+bpf-ci
@ 2026-10-26 12:51 ` kernel test robot
1 sibling, 0 replies; 144+ messages in thread
From: kernel test robot @ 2026-10-26 12:51 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: oe-kbuild-all, daniel, andrii, eddyz87, memxor
Hi Alexei,
kernel test robot noticed the following build errors:
[auto build test ERROR on bpf/master]
url: https://github.com/intel-lab-lkp/linux/commits/Alexei-Starovoitov/selftests-bpf-Remove-tests-for-global-data-in-arena/20261026-025822
base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git master
patch link: https://lore.kernel.org/r/20261026025943.404450-52-alexei.starovoitov%40gmail.com
patch subject: [PATCH bpf 51/81] bpf: Check stack and count calls of recursive subprogs at run time
config: powerpc-randconfig-1000-20261026 (https://download.01.org/0day-ci/archive/20261026/202610262057.WlLdhg8d-lkp@intel.com/config)
compiler: powerpc-linux-gcc (GCC) 8.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261026/202610262057.WlLdhg8d-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202610262057.WlLdhg8d-lkp@intel.com/
All errors (new ones prefixed by >>):
kernel/bpf/fixups.c: In function 'bpf_insert_recursion_checks':
>> kernel/bpf/fixups.c:792:28: error: macro "__fortify_memcpy_chk" passed 9 arguments, but takes just 8
2 * sizeof(buf[0]));
^
In file included from include/linux/string.h:396,
from include/crypto/sha2.h:10,
from include/linux/bpf.h:11,
from kernel/bpf/fixups.c:3:
>> include/linux/fortify-string.h:638:26: error: '__fortify_memcpy_chk' undeclared (first use in this function); did you mean 'fortify_memcpy_chk'?
#define memcpy(p, q, s) __fortify_memcpy_chk(p, q, s, \
^~~~~~~~~~~~~~~~~~~~
kernel/bpf/fixups.c:791:3: note: in expansion of macro 'memcpy'
memcpy(buf + n, (struct bpf_insn[]){ BPF_LD_IMM64(BPF_REG_0, (long)depth) },
^~~~~~
include/linux/fortify-string.h:638:26: note: each undeclared identifier is reported only once for each function it appears in
#define memcpy(p, q, s) __fortify_memcpy_chk(p, q, s, \
^~~~~~~~~~~~~~~~~~~~
kernel/bpf/fixups.c:791:3: note: in expansion of macro 'memcpy'
memcpy(buf + n, (struct bpf_insn[]){ BPF_LD_IMM64(BPF_REG_0, (long)depth) },
^~~~~~
kernel/bpf/fixups.c:834:27: error: macro "__fortify_memcpy_chk" passed 9 arguments, but takes just 8
2 * sizeof(buf[0]));
^
kernel/bpf/fixups.c:865:28: error: macro "__fortify_memcpy_chk" passed 9 arguments, but takes just 8
2 * sizeof(buf[0]));
^
vim +/__fortify_memcpy_chk +792 kernel/bpf/fixups.c
696
697 /*
698 * How deep a recursion goes is not known, so it is counted at run time. In the same
699 * spirit as may_goto: the program goes on for as long as there is budget and
700 * takes a way out that was verified when there is none.
701 *
702 * Every subprog on a cycle of the call graph adds its weight to a per-cpu
703 * counter of the program when it is entered and takes it back at every exit.
704 * The weight is the part of the budget that its frame takes: of the stack
705 * room, or of the room in the stack in arena, whichever is larger. So these
706 * frames take no more than BPF_RECURSION_STACK, on top of the chain of calls
707 * without recursion that check_max_stack_depth() looked at.
708 *
709 * A subprog that calls itself twice doesn't have to go deep to go on forever. So
710 * the calls are counted too: BPF_MAX_LOOPS of them in a run of the prog, like the
711 * callbacks of bpf_loop(). Once they are used up every call returns at once, and
712 * the frames that are there come to an end.
713 *
714 * Another run of the program on the same cpu, in interrupt or after
715 * preemption, counts in the same counter. Then there is less for both.
716 */
717 int bpf_insert_recursion_checks(struct bpf_verifier_env *env)
718 {
719 struct bpf_subprog_info *si = env->subprog_info;
720 u32 frame, min_frame = U32_MAX, arena_frame = 0, arena_room = 0, weight;
721 u32 limit = BPF_RECURSION_STACK;
722 struct bpf_insn buf[24], first;
723 struct bpf_prog *new_prog;
724 struct bpf_recursion_budget __percpu *depth;
725 /* Only this cpu touches the counters */
726 int fetch_add = BPF_ADD | BPF_FETCH |
727 (bpf_jit_supports_local_atomics() ? BPF_ATOMIC_LOCAL : 0);
728 int i, s, n, bail, bail_len, err;
729
730 for (s = 0; s < env->subprog_cnt; s++) {
731 if (!si[s].recursive)
732 continue;
733 frame = recursive_frame_size(env, s);
734 si[s].recursion_weight = DIV_ROUND_UP(frame * BPF_RECURSION_UNITS, limit);
735 min_frame = min(min_frame, frame);
736 arena_frame = max(arena_frame, si[s].arena_stack_depth);
737 }
738 if (min_frame == U32_MAX)
739 return 0;
740 if (arena_frame) {
741 /*
742 * One more frame is for the subprog that finds that there is no
743 * budget: it has its frame by then.
744 */
745 u32 room = BPF_ARENA_STACK_MAX - env->max_arena_stack_depth;
746
747 if (room < 2 * arena_frame) {
748 verbose(env, "no room in the stack in arena for recursion\n");
749 return -EACCES;
750 }
751 /* no more than the frames that fit in the stack can take */
752 arena_room = min(room - arena_frame, limit / min_frame * arena_frame);
753 env->max_arena_stack_depth += arena_room + arena_frame;
754 }
755 for (s = 0; s < env->subprog_cnt; s++) {
756 if (!si[s].recursive)
757 continue;
758 if (si[s].arena_stack_depth)
759 si[s].recursion_weight = max(si[s].recursion_weight,
760 DIV_ROUND_UP(si[s].arena_stack_depth *
761 BPF_RECURSION_UNITS, arena_room));
762 if (si[s].recursion_weight > BPF_RECURSION_UNITS) {
763 verbose(env, "the frame of recursive %s() is too large\n",
764 bpf_subprog_name(env, s));
765 return -EACCES;
766 }
767 }
768
769 depth = alloc_percpu_gfp(struct bpf_recursion_budget, GFP_KERNEL_ACCOUNT);
770 if (!depth)
771 return -ENOMEM;
772 env->prog->aux->call_depth = depth;
773 env->prog->jit_required = 1;
774
775 /* From the last one, so that nothing moves in those that are yet to be done. */
776 for (s = env->subprog_cnt - 1; s > 0; s--) {
777 if (!si[s].recursive)
778 continue;
779 /*
780 * R0 has nothing at the entry. The address is not for AX:
781 * blinding of constants takes ld_imm64 apart with the help of
782 * AX. It leaves alone the insns that have AX in them.
783 */
784 n = 0;
785 first = env->prog->insnsi[si[s].start];
786 /* The exit drops the frame in the stack in arena. Take it before the check. */
787 if (is_arena_sp_adj(&first)) {
788 buf[n++] = first;
789 first = BPF_JMP_A(0);
790 }
791 memcpy(buf + n, (struct bpf_insn[]){ BPF_LD_IMM64(BPF_REG_0, (long)depth) },
> 792 2 * sizeof(buf[0]));
793 n += 2;
794 buf[n++] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
795 weight = si[s].recursion_weight;
796 bail_len = bpf_ret_reg_pair(env, s) ? 3 : 2;
797 buf[n++] = BPF_MOV64_IMM(BPF_REG_AX, weight);
798 buf[n++] = BPF_ATOMIC_OP(BPF_W, fetch_add, BPF_REG_0, BPF_REG_AX,
799 offsetof(struct bpf_recursion_budget, depth));
800 buf[n++] = BPF_JMP_IMM(BPF_JGT, BPF_REG_AX, BPF_RECURSION_UNITS - weight, 3);
801 buf[n++] = BPF_MOV64_IMM(BPF_REG_AX, 1);
802 buf[n++] = BPF_ATOMIC_OP(BPF_W, fetch_add, BPF_REG_0, BPF_REG_AX,
803 offsetof(struct bpf_recursion_budget, calls));
804 buf[n++] = BPF_JMP_IMM(BPF_JLT, BPF_REG_AX, BPF_MAX_LOOPS, bail_len);
805 bail = n;
806 buf[n++] = BPF_MOV64_IMM(BPF_REG_0, 0);
807 if (bpf_ret_reg_pair(env, s))
808 buf[n++] = BPF_MOV64_IMM(BPF_REG_2, 0);
809 buf[n++] = budget_exit_insn(env, &si[s]);
810 buf[n] = first;
811 err = bpf_adj_moved_insn(&buf[n], n);
812 if (err) {
813 verbose(env, "insn %d cannot be patched due to 16-bit range\n",
814 env->insn_aux_data[si[s].start].orig_idx);
815 return err;
816 }
817 n++;
818 new_prog = __bpf_patch_insn_data(env, si[s].start, buf, n, BPF_PATCH_MOVE_TARGET);
819 if (!new_prog)
820 return -ENOMEM;
821 env->prog = new_prog;
822 env->insn_aux_data[si[s].start + bail].recursion_bail = true;
823 /* calls get to the check, jumps to the first insn do not */
824 err = adjust_jmp_off(env->prog, si[s].start, n - 1);
825 if (err)
826 return err;
827 }
828
829 /*
830 * The calls are counted anew when a run begins. For async callbacks see
831 * bpf_prog_begin_run(). R0 has nothing at the entry.
832 */
833 memcpy(buf, (struct bpf_insn[]){ BPF_LD_IMM64(BPF_REG_0, (long)depth) },
834 2 * sizeof(buf[0]));
835 buf[2] = BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0);
836 /* calls and reported */
837 buf[3] = BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 0);
838 buf[4] = env->prog->insnsi[0];
839 err = bpf_adj_moved_insn(&buf[4], 4);
840 if (err) {
841 verbose(env, "insn %d cannot be patched due to 16-bit range\n",
842 env->insn_aux_data[0].orig_idx);
843 return err;
844 }
845 new_prog = __bpf_patch_insn_data(env, 0, buf, 5, BPF_PATCH_MOVE_TARGET);
846 if (!new_prog)
847 return -ENOMEM;
848 env->prog = new_prog;
849 /* a jump to the first insn doesn't begin a run */
850 err = adjust_jmp_off(env->prog, 0, 4);
851 if (err)
852 return err;
853
854 /* R1 is nothing to the caller. Jumps to exit get to what is added. */
855 for (i = 0; i < env->prog->len; i++) {
856 if (bpf_is_ldimm64(&env->prog->insnsi[i])) {
857 i++;
858 continue;
859 }
860 if (env->prog->insnsi[i].code != (BPF_JMP | BPF_EXIT) ||
861 !bpf_find_containing_subprog(env, i)->recursive)
862 continue;
863 n = 0;
864 memcpy(buf, (struct bpf_insn[]){ BPF_LD_IMM64(BPF_REG_1, (long)depth) },
865 2 * sizeof(buf[0]));
866 n += 2;
867 buf[n++] = BPF_MOV64_PERCPU_REG(BPF_REG_1, BPF_REG_1);
868 weight = bpf_find_containing_subprog(env, i)->recursion_weight;
869 buf[n++] = BPF_MOV64_IMM(BPF_REG_AX, -(int)weight);
870 buf[n++] = BPF_ATOMIC_OP(BPF_W, fetch_add, BPF_REG_1, BPF_REG_AX,
871 offsetof(struct bpf_recursion_budget, depth));
872 buf[n++] = env->prog->insnsi[i];
873 new_prog = bpf_patch_insn_data(env, i, buf, n);
874 if (!new_prog)
875 return -ENOMEM;
876 env->prog = new_prog;
877 i += n - 1;
878 }
879 return 0;
880 }
881
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 52/81] bpf: Add the main pass as a fixpoint over the control flow graph
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (50 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 51/81] bpf: Check stack and count calls of recursive subprogs at run time Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 53/81] bpf: Verify programs " Alexei Starovoitov
` (30 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier follows each path of a prog from the entry to exit. Each
branch can double the number of paths.
if (a) r7 |= 1; 2 paths
if (b) r7 |= 2; 4 paths
...
if (t) r7 |= 1 << 19; 2^20 paths
x = arr[r7 & 63];
This prog has 75 insns. r7 is different on each path, thus the verifier
cannot prune a path. It hits the limit of 1M insns.
Add a data-flow analysis over the control flow graph as the main pass.
Where two paths meet, it joins their states into one state that covers the
two: r1=1 and r1=5 give r1=[1,5]. It verifies the subsequent code one time.
A join loses relations between registers, and progs depend on them:
'p is valid when ret is 0'. Thus a basic block has a set of states, not one
state.
state S arrives at block B
|
v
+------------------------------+ yes
| a state of set(B) covers S ? |------> drop S
+------------------------------+
| no
v
+------------------------------+ yes
| S has a join with a state T |------> replace T with the join,
| of set(B), and no rule below | the join is a new state
| refuses it ? |
+------------------------------+
| no
v
add S to set(B), S is a new state
'Covers' is the comparison that pruning uses, states_equal(). The verifier
verifies a block with each new state of its set, and sends the results to
the successors of the block. It takes the blocks in reverse post order,
with the insns of a loop kept together. Thus the states from all
predecessors meet before the verifier uses them. The pass ends when no set
has a new state. The verifier drops a state only when the set has a state
that covers it. Thus a verified state covers each run of the prog.
Two scalars always have a join. The difficult decision is when not to use
it, because a join can contain values that no input has:
if (ret < 0) goto err;
if (ret > 0) goto skip;
use(p);
The states ret=-2 and ret=1 meet before this code, and p is not valid in
these states. The join ret=[-2,1] contains 0. It passes the two jumps and
arrives at use(p) without p. Thus the verifier keeps two states that have a
join when they differ in:
a register that only one state initializes ret = get(&p)
a tested register, and its ranges have a gap ret=-2 and ret=1
two tested registers, each without a common err < 0 and ret=DROP,
value err >= 0 and ret=0
a scalar that is 0 in one state only if (h->v4) else if (h->v6)
a constant that a subprog returns true or false
a constant that becomes a stack offset a[n++] = p
a constant that a helper or a kfunc must know size for a ring buffer
the iteration of bpf_loop(), for 8 or fewer bpf_loop(4, cb, ...)
A tested register is a register or a stack slot whose unchanged value
arrives at a later jump that compares it with a constant. -2 and 1 have a
gap, 0 and 1 do not. flow_compute_need_const() finds them, and the
constants of the table, before the pass. It goes backward through the
graph, like the calculation of live registers.
The rules only refuse joins. Thus they cannot make the verifier accept an
unsafe prog. But the states that it keeps can multiply. Thus the first six
rules apply only while less than half of each limit below is in use.
Some progs that loaded before do not load with this pass. In
'ok = idx < 8; ... if (ok) buf[idx]' the values 0 and 1 of ok have no gap.
The verifier joins the two states and loses the range of idx.
The verifier uses scalar evolution to change the state that enters a loop,
so that it covers all iterations. Then it verifies the body one time. If a
state returns to the loop head and is not covered, the verifier widens: it
joins the two states and removes the bound that grew. Thus the pass ends.
This does not prove that a loop ends. A loop gets may_goto unless the
number of iterations is known, or each iteration executes may_goto or
bpf_iter_*_next(). flow_place_guards() puts may_goto before a conditional
jump that exits the loop, with the same target. The prog can arrive there
with the same state through that jump.
bpf_iter_*_next() stops a loop only if all iterations use the same
iterator. When 8 states with different iterator ids meet in a loop, the
verifier assumes that the loop creates its iterator again. Then it starts
the pass again and adds may_goto to the loops around this insn. Nested
bpf_for() loops can be such a case.
The verifier verifies a static subprog for the states at its entry, not
again for each chain of calls. The subprog must get only scalars and
pointers to objects that are not on the stack.
2: call f --- { r1=scalar } ---> +---------------------+
7: call f --- covered: drop | f: verified 1 time |
3: ... <-- { r0=[0,7] } ----- | |
8: ... <-- { r0=[0,7] } ----- +---------------------+
The verifier saves the state of the caller until the states at the exits of
f arrive. Recursion uses the same mechanism, and the entry of the subprog
is like a loop head.
The verifier rejects a prog of N insns with -E2BIG when it hits a limit:
states at one insn 1024
states at one time 16 * N + 4096
states in total 256 * N + 64K
flow.c only decides where states go. The check of an insn does not change.
"bpf: Verify programs as a fixpoint over the control flow graph" builds
flow.c and calls it.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
kernel/bpf/flow.c | 1817 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 1817 insertions(+)
create mode 100644 kernel/bpf/flow.c
diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
new file mode 100644
index 000000000000..fc5272ed2af8
--- /dev/null
+++ b/kernel/bpf/flow.c
@@ -0,0 +1,1817 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * The main pass of the verifier as a fixpoint over the control flow graph.
+ *
+ * Every insn that starts a basic block has a set of states. The state that
+ * arrives there is
+ * - dropped, if it is within a state of the set,
+ * - joined with a state of the set, if there is a join, see bpf_states_join(),
+ * that doesn't make up what neither of them has, see flow_join_makes_up(),
+ * - added to the set otherwise.
+ * What is new in the set is pending. Pending states are processed in the order
+ * of the program in which a loop is done with before what follows it, see
+ * flow_compute_order(). The pass ends when nothing is pending.
+ *
+ * Loops. The state that enters a loop is made to stand for all iterations with
+ * what scalar evolution knows about the loop, see bpf_flow_loop_enter().
+ * The state that comes back to the loop head is expected to be within it.
+ * If it is not, the two are joined, what grew is forgotten, and the loop is
+ * done again.
+ *
+ * Functions. A static function that gets nothing but numbers and pointers to
+ * what is not the stack is verified on its own, for the states at its entry
+ * and not for every chain of calls. The call waits. What comes out of the
+ * function goes to all calls that wait for the states it comes from.
+ */
+#include <linux/bpf.h>
+#include <linux/bpf_verifier.h>
+#include <linux/filter.h>
+#include <linux/jhash.h>
+#include <linux/rbtree.h>
+#include <linux/slab.h>
+
+#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
+
+/*
+ * States at an insn that have no join: kinds of values, references, chains of calls.
+ * A state that arrives is compared with all of them, so the work is the square of it.
+ */
+#define BPF_FLOW_MAX_STATES 1024
+/* All states that are there at a time: that many for every insn of the prog, and some. */
+#define BPF_FLOW_LIVE_PER_INSN 16
+#define BPF_FLOW_LIVE_BASE 4096
+/* All states that were ever there. It bounds the work by the size of the prog. */
+#define BPF_FLOW_TOTAL_PER_INSN 256
+#define BPF_FLOW_TOTAL_BASE 65536
+/* A state that was joined that many times is in a cycle that has no loop head. */
+#define BPF_FLOW_MAX_JOINS 16
+/* bpf_loop() with that many iterations at most is walked. */
+#define BPF_FLOW_MAX_CALLBACKS 8
+/* What a callback returns to the helper with is joined that many times before it is widened. */
+#define BPF_FLOW_CALLBACK_JOINS 4
+/* That many states in a loop that differ in their iterators: the loop makes new ones. */
+#define BPF_FLOW_MAX_ITERS 8
+
+enum flow_cycle {
+ FLOW_NO_CYCLE,
+ /* what grew is forgotten */
+ FLOW_CYCLE,
+ /* after a while. A flag that a callback sets doesn't grow any further. */
+ FLOW_CALLBACK_CYCLE,
+};
+
+struct flow_state {
+ struct bpf_verifier_state st;
+ struct list_head node; /* states at the insn */
+ struct rb_node rb; /* pending states */
+ int prev_insn_idx;
+ u32 seq;
+ /* see flow_kind() */
+ u32 kind;
+ u16 joins;
+ bool pending;
+ /* the order of pending states, see flow_set_pending() */
+ u8 key_len;
+ u32 key[MAX_CALL_FRAMES + 1];
+};
+
+/* the call that waits for what comes out of the function */
+struct flow_waiter {
+ struct list_head node;
+ struct bpf_verifier_state st;
+ u32 entry;
+ int call_idx;
+ /* the call is verified in the context of the caller */
+ bool dead;
+};
+
+struct flow_exit {
+ struct list_head node;
+ struct bpf_verifier_state st;
+ u32 entry;
+ u16 joins;
+};
+
+struct flow_func {
+ struct list_head waiters;
+ struct list_head exits;
+ int exit_idx; /* all exits are compared as if they were here */
+};
+
+struct flow_united {
+ u32 subprog;
+ u32 a, b;
+};
+
+struct flow_kept {
+ struct list_head node;
+ struct bpf_verifier_state st;
+};
+
+/* What has to stay a known constant at an insn */
+struct flow_need {
+ u64 slots; /* of the stack, fp-8 is bit 0 */
+ u16 regs;
+};
+
+struct bpf_flow {
+ /* computed once */
+ u32 len; /* of the prog during the pass */
+ u32 nr_funcs;
+ u32 *pos; /* place of the insn in the order */
+ u32 *rank; /* callers go first */
+ struct flow_need *need_const; /* see flow_compute_need_const() */
+ struct flow_need *stack_off;
+ struct flow_need *tested;
+ unsigned long *point; /* insn starts a basic block */
+ unsigned long *keep; /* states that were processed are compared with */
+
+ /* of the pass */
+ struct list_head *states; /* per insn */
+ struct list_head *enter; /* per insn: the states that enter the loop at its head */
+ u32 *nr_states;
+ struct rb_root_cached pending;
+ struct flow_func *funcs;
+ u32 *entry_up; /* entries that met are one set */
+ u32 entry_cnt, entry_cap;
+ struct flow_united *united;
+ u32 united_cnt, united_cap;
+ struct list_head kept;
+ /* what leaves a loop if its guard is going to, see bpf_flow_park(), and what does */
+ struct list_head parked;
+ struct list_head unparked;
+ struct bpf_verifier_state tmp;
+ u32 seq;
+ u32 live_states;
+};
+
+/*
+ * The order. Reverse post order is a topological order of the graph without
+ * back edges, but it may put what follows a loop between the insns of
+ * the loop. Keep the insns of a loop together, the head first: in the list of
+ * its parent the loop takes the place of the insn of it that comes first.
+ *
+ * Every cycle has a back edge of the depth first search in it and
+ * bpf_compute_loops() makes the target of such edge a loop head. So every edge
+ * that goes backwards in the order is in a loop whose head is at or before its
+ * target, loops that are not reducible included.
+ */
+static int flow_compute_order(struct bpf_verifier_env *env, struct bpf_flow *flow)
+{
+ struct bpf_insn_aux_data *aux = env->insn_aux_data;
+ int len = env->prog->len, i, s, n, h, top, depth;
+ int *next, *head, *tail, *stack;
+ unsigned long *linked;
+ u32 pos = 0;
+ int err = -ENOMEM;
+
+ /* lists of members are indexed by loop head, len is the function */
+ next = kvmalloc_array(len, sizeof(int), GFP_KERNEL_ACCOUNT);
+ head = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
+ tail = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
+ stack = kvmalloc_array(len + 1, sizeof(int), GFP_KERNEL_ACCOUNT);
+ linked = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
+ if (!next || !head || !tail || !stack || !linked)
+ goto out;
+
+ for (i = 0; i <= len; i++)
+ head[i] = tail[i] = -1;
+
+ for (s = 0; s < env->subprog_cnt; s++) {
+ int first = env->subprog_info[s].postorder_start;
+ int last = env->subprog_info[s + 1].postorder_start - 1;
+
+ head[len] = tail[len] = -1;
+ for (i = last; i >= first; i--) {
+ n = env->cfg.insn_postorder[i];
+ /* n, then the heads of the loops around it that are not in a list yet */
+ depth = 0;
+ for (h = n; h >= 0 && !test_bit(h, linked); h = aux[h].loop_header)
+ stack[depth++] = h;
+ while (depth) {
+ h = stack[--depth];
+ top = aux[h].loop_header >= 0 ? aux[h].loop_header : len;
+ __set_bit(h, linked);
+ next[h] = -1;
+ if (tail[top] < 0)
+ head[top] = h;
+ else
+ next[tail[top]] = h;
+ tail[top] = h;
+ }
+ }
+
+ depth = 0;
+ n = head[len];
+ while (n >= 0 || depth) {
+ if (n < 0) {
+ n = next[stack[--depth]];
+ continue;
+ }
+ flow->pos[n] = pos++;
+ if (aux[n].loop && head[n] >= 0) {
+ stack[depth++] = n;
+ n = head[n];
+ } else {
+ n = next[n];
+ }
+ }
+ }
+ err = 0;
+out:
+ kvfree(next);
+ kvfree(head);
+ kvfree(tail);
+ kvfree(stack);
+ kvfree(linked);
+ return err;
+}
+
+/* Does the insn at @idx stop the loop @h? */
+static bool flow_is_bound(struct bpf_verifier_env *env, int h, int idx)
+{
+ struct bpf_insn *insn = &env->prog->insnsi[idx];
+ int t;
+
+ /* Once it is out of budget it's a jump. It has to leave the loop. */
+ if (bpf_is_may_goto_insn(insn)) {
+ t = bpf_loop_at_index(env, idx + insn->off + 1);
+ return t != h && !bpf_is_nested_loop(env, t, h);
+ }
+ /* as long as the program doesn't go on when there is nothing left, see iter_is_drained() */
+ return bpf_insn_is_iter_next(env, insn);
+}
+
+/*
+ * What stops a loop at run time is a fact about the program: may_goto or
+ * bpf_iter_*_next() that every trip goes through. A loop without it gets
+ * may_goto unless the number of its trips turns out to be known.
+ *
+ * The place for it is in front of a conditional jump that leaves the loop, with
+ * the same target. The registers and the stack are what they are when
+ * the program leaves the loop there. One such jump that every trip goes
+ * through is enough. Otherwise every back edge needs one.
+ */
+static void flow_set_guard(struct bpf_verifier_env *env, int h, int n)
+{
+ struct bpf_insn_aux_data *aux = env->insn_aux_data;
+ int t = n + env->prog->insnsi[n].off + 1;
+ int i = bpf_loop_at_index(env, t);
+
+ if (i == h || bpf_is_nested_loop(env, i, h))
+ t = n + 1;
+ aux[n].guard_place = true;
+ aux[n].guard_to = t;
+ aux[h].loop->has_guards = true;
+}
+
+static void flow_place_guards(struct bpf_verifier_env *env, int h, u32 *seen)
+{
+ struct bpf_insn_aux_data *aux = env->insn_aux_data;
+ struct bpf_loop *loop = aux[h].loop;
+ int i, n, o, found = -1;
+
+ if (loop->backedges_overflow)
+ goto header;
+
+ /*
+ * The head of a loop with more than one entry doesn't dominate what is in the loop.
+ * Then an insn that dominates a back edge can be passed once, on the way in, and
+ * never again. Every trip goes through the jump that is the back edge, and
+ * through the head.
+ */
+ if (loop->irreducible) {
+ for (i = 0; i < loop->backedges_cnt; i++)
+ if (!loop->backedges[i].latches_cnt ||
+ loop->backedges[i].latches[0] != loop->backedges[i].from)
+ goto header;
+ for (i = 0; i < loop->backedges_cnt; i++)
+ flow_set_guard(env, h, loop->backedges[i].from);
+ return;
+ }
+
+ /* seen[n]: the number of back edges that n dominates */
+ for (i = 0; i < loop->backedges_cnt; i++)
+ for (n = loop->backedges[i].from; n >= 0; n = n == h ? -1 : env->idoms[n])
+ seen[n]++;
+ for (n = loop->backedges[0].from; n >= 0 && found < 0; n = n == h ? -1 : env->idoms[n])
+ if (seen[n] == loop->backedges_cnt && bpf_insn_leaves_loop(env, h, n))
+ found = n;
+ for (i = 0; i < loop->backedges_cnt; i++)
+ for (n = loop->backedges[i].from; n >= 0; n = n == h ? -1 : env->idoms[n])
+ seen[n] = 0;
+ if (found >= 0) {
+ flow_set_guard(env, h, found);
+ return;
+ }
+
+ for (i = 0; i < loop->backedges_cnt; i++)
+ if (!loop->backedges[i].latches_cnt)
+ goto header;
+ for (i = 0; i < loop->backedges_cnt; i++)
+ flow_set_guard(env, h, loop->backedges[i].latches[0]);
+ return;
+
+header:
+ /*
+ * In front of a jump may_goto goes where the jump goes. From the head to where the loop
+ * is left is not an edge of the program: it skips what is in the loop. If that is what
+ * stops a loop around this one, bpf_iter_*_next() or i++, and the exit leads back to
+ * the head of that loop, the program goes round and round from may_goto to may_goto.
+ * So leave all loops.
+ */
+ for (o = h; aux[o].loop_header >= 0; o = aux[o].loop_header)
+ ;
+ /* a loop that has no exit can only be stopped together with the program */
+ if (!aux[o].loop->exits_cnt)
+ return;
+ aux[h].guard_place = true;
+ aux[h].guard_at_header = true;
+ aux[h].guard_to = aux[o].loop->exits[0].to;
+ loop->has_guards = true;
+}
+
+static int flow_compute_loop_bounds(struct bpf_verifier_env *env)
+{
+ struct bpf_insn_aux_data *aux = env->insn_aux_data;
+ int len = env->prog->len, h, i, n;
+ struct bpf_loop *loop;
+ u32 *seen;
+
+ seen = kvcalloc(len, sizeof(*seen), GFP_KERNEL_ACCOUNT);
+ if (!seen)
+ return -ENOMEM;
+
+ for (i = 0; i < len; i++) {
+ for (h = bpf_loop_at_index(env, i); h >= 0; h = aux[h].loop_header) {
+ aux[h].loop->nr_insns++;
+ if (bpf_pseudo_call(&env->prog->insnsi[i]) ||
+ bpf_is_callx(&env->prog->insnsi[i]))
+ aux[h].loop->has_calls = true;
+ }
+ }
+
+ for (h = 0; h < len; h++) {
+ loop = aux[h].loop;
+ if (!loop)
+ continue;
+ /* see flow_place_guards() about loops with more than one entry */
+ loop->rt_bounded = !loop->backedges_overflow && !loop->irreducible;
+ for (i = 0; i < loop->backedges_cnt && loop->rt_bounded; i++) {
+ for (n = loop->backedges[i].from; n >= 0 && n != h; n = env->idoms[n])
+ if (flow_is_bound(env, h, n))
+ break;
+ if (n < 0 || !flow_is_bound(env, h, n))
+ loop->rt_bounded = false;
+ }
+ /* also when it's bounded: the main pass may find that it's not */
+ flow_place_guards(env, h, seen);
+ }
+ kvfree(seen);
+ return 0;
+}
+
+/* The slot that @insn spills to or fills from, if it's one that struct flow_need has a bit for */
+static int flow_fp_slot(const struct bpf_insn *insn, u8 base)
+{
+ u8 mode = BPF_MODE(insn->code);
+
+ if (base != BPF_REG_FP || (mode != BPF_MEM && mode != BPF_MEMSX) ||
+ insn->off >= 0 || insn->off < -64 * BPF_REG_SIZE || insn->off % BPF_REG_SIZE)
+ return -1;
+ return -insn->off / BPF_REG_SIZE - 1;
+}
+
+/*
+ * A number is added to a pointer to the stack. A store at an offset that is not known
+ * forgets what the slots it may hit hold, pointers above all.
+ */
+static u16 flow_stack_off_regs(struct bpf_verifier_env *env, int idx)
+{
+ u16 ptrs = env->insn_aux_data[idx].stack_ptrs;
+ struct bpf_insn *insn = &env->prog->insnsi[idx];
+ u8 op = BPF_OP(insn->code);
+
+ if (BPF_CLASS(insn->code) != BPF_ALU64 || BPF_SRC(insn->code) != BPF_X ||
+ (op != BPF_ADD && op != BPF_SUB))
+ return 0;
+ if ((ptrs & BIT(insn->dst_reg)) && !(ptrs & BIT(insn->src_reg)))
+ return BIT(insn->src_reg);
+ if (op == BPF_ADD && (ptrs & BIT(insn->src_reg)) && !(ptrs & BIT(insn->dst_reg)))
+ return BIT(insn->dst_reg);
+ return 0;
+}
+
+/* A jump that compares a register with a constant */
+static u16 flow_tested_regs(const struct bpf_insn *insn)
+{
+ u8 class = BPF_CLASS(insn->code), op = BPF_OP(insn->code);
+
+ if ((class != BPF_JMP && class != BPF_JMP32) || BPF_SRC(insn->code) != BPF_K ||
+ op == BPF_JA || op == BPF_CALL || op == BPF_EXIT || op == BPF_JCOND)
+ return 0;
+ return BIT(insn->dst_reg);
+}
+
+static bool flow_is_copy(const struct bpf_insn *insn)
+{
+ return BPF_OP(insn->code) == BPF_MOV && BPF_SRC(insn->code) == BPF_X &&
+ !insn->off && !insn->imm;
+}
+
+enum flow_need_kind {
+ FLOW_NEED_ARG,
+ FLOW_NEED_STACK_OFF,
+ FLOW_NEED_JUMP,
+};
+
+/*
+ * Some arguments of kfuncs and helpers have to be known constants: the id of a type,
+ * the size of what is reserved in a ring buffer.
+ * A compiler makes one call of
+ * if (x) p = bpf_rdonly_cast(q, ID); else p = bpf_rdonly_cast(q, 0);
+ * and one store of
+ * if (x) a[n++] = p; else b[m++] = p;
+ * where a and b are on the stack.
+ * The join of two constants is not a constant. Find the registers and the slots of the stack
+ * that end up in such arguments, or in offsets in the stack, like live
+ * registers are found: backwards, until nothing changes.
+ * States in which they differ are not joined.
+ * Nothing depends on it but what is known: it's fine to miss a store through a pointer.
+ *
+ * FLOW_NEED_JUMP: those that a jump ahead compares with a constant, as they are, see
+ * flow_join_makes_up().
+ */
+static int flow_compute_need_const(struct bpf_verifier_env *env, struct flow_need **pneed,
+ enum flow_need_kind kind)
+{
+ struct bpf_insn *insns = env->prog->insnsi, *insn;
+ struct flow_need in, out, *need;
+ struct bpf_iarray *succ;
+ bool changed = true, any = false;
+ int i, s, idx, slot;
+ u16 *gen;
+ u8 class;
+
+ need = kvcalloc(env->prog->len, sizeof(*need), GFP_KERNEL_ACCOUNT);
+ gen = kvcalloc(env->prog->len, sizeof(u16), GFP_KERNEL_ACCOUNT);
+ *pneed = need;
+ if (!need || !gen) {
+ kvfree(gen);
+ return -ENOMEM;
+ }
+ for (i = 0; i < env->prog->len; i++) {
+ if (kind == FLOW_NEED_JUMP)
+ gen[i] = flow_tested_regs(&insns[i]);
+ else if (kind == FLOW_NEED_STACK_OFF)
+ gen[i] = flow_stack_off_regs(env, i);
+ else if (bpf_pseudo_kfunc_call(&insns[i]))
+ gen[i] = bpf_kfunc_const_arg_regs(env, &insns[i]);
+ else if (bpf_helper_call(&insns[i]))
+ gen[i] = bpf_helper_const_arg_regs(env, &insns[i]);
+ any |= gen[i];
+ }
+ while (any && changed) {
+ changed = false;
+ for (i = 0; i < env->cfg.cur_postorder; i++) {
+ idx = env->cfg.insn_postorder[i];
+ insn = &insns[idx];
+ class = BPF_CLASS(insn->code);
+ succ = bpf_insn_successors(env, idx);
+ out.regs = 0;
+ out.slots = 0;
+ for (s = 0; s < succ->cnt; s++) {
+ out.regs |= need[succ->items[s]].regs;
+ out.slots |= need[succ->items[s]].slots;
+ }
+ in = out;
+ if (class == BPF_JMP && BPF_OP(insn->code) == BPF_CALL) {
+ in.regs &= ~GENMASK(BPF_REG_5, BPF_REG_0);
+ s = idx + insn->imm + 1;
+ if (bpf_pseudo_call(insn) &&
+ !bpf_subprog_is_global(env, bpf_find_subprog(env, s)))
+ in.regs |= need[s].regs & GENMASK(BPF_REG_5, BPF_REG_1);
+ } else if (class == BPF_LDX || class == BPF_LD) {
+ slot = class == BPF_LDX ? flow_fp_slot(insn, insn->src_reg) : -1;
+ if (slot >= 0 && (out.regs & BIT(insn->dst_reg)))
+ in.slots |= BIT_ULL(slot);
+ in.regs &= ~BIT(insn->dst_reg);
+ } else if (class == BPF_STX || class == BPF_ST) {
+ slot = flow_fp_slot(insn, insn->dst_reg);
+ if (slot >= 0 && (out.slots & BIT_ULL(slot))) {
+ in.slots &= ~BIT_ULL(slot);
+ if (class == BPF_STX)
+ in.regs |= BIT(insn->src_reg);
+ }
+ } else if ((class == BPF_ALU || class == BPF_ALU64) &&
+ (out.regs & BIT(insn->dst_reg)) && kind == FLOW_NEED_JUMP) {
+ in.regs &= ~BIT(insn->dst_reg);
+ if (flow_is_copy(insn))
+ in.regs |= BIT(insn->src_reg);
+ } else if ((class == BPF_ALU || class == BPF_ALU64) &&
+ (out.regs & BIT(insn->dst_reg))) {
+ /* the result is known when what it is made of is */
+ if (BPF_OP(insn->code) == BPF_MOV)
+ in.regs &= ~BIT(insn->dst_reg);
+ if (BPF_SRC(insn->code) == BPF_X)
+ in.regs |= BIT(insn->src_reg);
+ }
+ in.regs |= gen[idx];
+ if (in.regs != need[idx].regs || in.slots != need[idx].slots) {
+ need[idx] = in;
+ changed = true;
+ }
+ }
+ }
+ kvfree(gen);
+ if (!any) {
+ kvfree(need);
+ *pneed = NULL;
+ }
+ return 0;
+}
+
+int bpf_flow_prepare(struct bpf_verifier_env *env)
+{
+ struct bpf_insn_aux_data *aux = env->insn_aux_data;
+ struct bpf_insn *insns = env->prog->insnsi;
+ int len = env->prog->len, i, s, err;
+ struct bpf_iarray *succ;
+ struct bpf_flow *flow;
+ u8 *preds;
+
+ flow = kzalloc_obj(*flow, GFP_KERNEL_ACCOUNT);
+ if (!flow)
+ return -ENOMEM;
+ env->flow = flow;
+ INIT_LIST_HEAD(&flow->kept);
+ INIT_LIST_HEAD(&flow->parked);
+ INIT_LIST_HEAD(&flow->unparked);
+ flow->pending = RB_ROOT_CACHED;
+
+ flow->pos = kvcalloc(len, sizeof(u32), GFP_KERNEL_ACCOUNT);
+ flow->rank = kvcalloc(env->subprog_cnt + 1, sizeof(u32), GFP_KERNEL_ACCOUNT);
+ flow->point = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
+ flow->keep = kvcalloc(BITS_TO_LONGS(len), sizeof(long), GFP_KERNEL_ACCOUNT);
+ flow->states = kvmalloc_array(len, sizeof(*flow->states), GFP_KERNEL_ACCOUNT);
+ flow->enter = kvmalloc_array(len, sizeof(*flow->enter), GFP_KERNEL_ACCOUNT);
+ flow->nr_states = kvcalloc(len, sizeof(u32), GFP_KERNEL_ACCOUNT);
+ flow->funcs = kvcalloc(env->subprog_cnt + 1, sizeof(*flow->funcs), GFP_KERNEL_ACCOUNT);
+ preds = kvcalloc(len, 1, GFP_KERNEL_ACCOUNT);
+ if (!flow->pos || !flow->rank || !flow->point || !flow->keep || !flow->states ||
+ !flow->enter ||
+ !flow->nr_states || !flow->funcs || !preds) {
+ kvfree(preds);
+ return -ENOMEM;
+ }
+ for (i = 0; i < len; i++) {
+ INIT_LIST_HEAD(&flow->states[i]);
+ INIT_LIST_HEAD(&flow->enter[i]);
+ }
+ for (i = 0; i <= env->subprog_cnt; i++) {
+ INIT_LIST_HEAD(&flow->funcs[i].waiters);
+ INIT_LIST_HEAD(&flow->funcs[i].exits);
+ flow->funcs[i].exit_idx = -1;
+ }
+ /* from here on there is something for bpf_flow_reset() to look at */
+ flow->len = len;
+ flow->nr_funcs = env->subprog_cnt + 1;
+
+ /* subprog_topo_order[] has callees first */
+ for (i = 0; i < env->subprog_cnt; i++)
+ flow->rank[env->subprog_topo_order[i]] = env->subprog_cnt - i;
+
+ for (i = 0; i < len; i++) {
+ succ = bpf_insn_successors(env, i);
+ iarray_for_each(s, succ) {
+ if (preds[s] < 2)
+ preds[s]++;
+ if (aux[i].bb_end)
+ __set_bit(s, flow->point);
+ }
+ /* where the callees come back to */
+ if (bpf_pseudo_call(&insns[i]) || bpf_is_callx(&insns[i])) {
+ if (i + 1 < len) {
+ __set_bit(i + 1, flow->point);
+ __set_bit(i + 1, flow->keep);
+ }
+ }
+ if (aux[i].calls_callback) {
+ __set_bit(i, flow->point);
+ __set_bit(i, flow->keep);
+ }
+ if (aux[i].loop) {
+ __set_bit(i, flow->point);
+ __set_bit(i, flow->keep);
+ }
+ if (bpf_is_ldimm64(&insns[i]))
+ i++;
+ }
+ for (i = 0; i < len; i++)
+ if (preds[i] > 1)
+ __set_bit(i, flow->keep);
+ for (i = 0; i < env->subprog_cnt; i++) {
+ __set_bit(env->subprog_info[i].start, flow->point);
+ __set_bit(env->subprog_info[i].start, flow->keep);
+ }
+ kvfree(preds);
+
+ err = flow_compute_order(env, flow);
+ if (err)
+ return err;
+ err = flow_compute_need_const(env, &flow->need_const, FLOW_NEED_ARG);
+ err = err ?: flow_compute_need_const(env, &flow->stack_off, FLOW_NEED_STACK_OFF);
+ err = err ?: flow_compute_need_const(env, &flow->tested, FLOW_NEED_JUMP);
+ if (err)
+ return err;
+ return flow_compute_loop_bounds(env);
+}
+
+static void flow_free_state(struct bpf_verifier_env *env, struct flow_state *s)
+{
+ bpf_free_verifier_state(&s->st, false);
+ kfree(s);
+ env->flow->live_states--;
+}
+
+/* What a pass leaves behind. */
+void bpf_flow_reset(struct bpf_verifier_env *env)
+{
+ struct bpf_flow *flow = env->flow;
+ struct flow_waiter *w, *wn;
+ struct flow_state *s, *sn;
+ struct flow_kept *k, *kn;
+ struct flow_exit *x, *xn;
+ int i;
+
+ if (!flow || !flow->states || !flow->enter)
+ return;
+ for (i = 0; i < flow->len; i++) {
+ list_for_each_entry_safe(s, sn, &flow->states[i], node) {
+ list_del(&s->node);
+ flow_free_state(env, s);
+ }
+ list_for_each_entry_safe(s, sn, &flow->enter[i], node) {
+ list_del(&s->node);
+ flow_free_state(env, s);
+ }
+ flow->nr_states[i] = 0;
+ }
+ flow->pending = RB_ROOT_CACHED;
+ for (i = 0; i < flow->nr_funcs && flow->funcs; i++) {
+ list_for_each_entry_safe(w, wn, &flow->funcs[i].waiters, node) {
+ bpf_free_verifier_state(&w->st, false);
+ kfree(w);
+ }
+ list_for_each_entry_safe(x, xn, &flow->funcs[i].exits, node) {
+ bpf_free_verifier_state(&x->st, false);
+ kfree(x);
+ }
+ INIT_LIST_HEAD(&flow->funcs[i].waiters);
+ INIT_LIST_HEAD(&flow->funcs[i].exits);
+ flow->funcs[i].exit_idx = -1;
+ }
+ list_for_each_entry_safe(k, kn, &flow->kept, node) {
+ bpf_free_verifier_state(&k->st, false);
+ kfree(k);
+ }
+ INIT_LIST_HEAD(&flow->kept);
+ list_splice_init(&flow->unparked, &flow->parked);
+ list_for_each_entry_safe(s, sn, &flow->parked, node) {
+ list_del(&s->node);
+ flow_free_state(env, s);
+ }
+ bpf_free_verifier_state(&flow->tmp, false);
+ memset(&flow->tmp, 0, sizeof(flow->tmp));
+ flow->entry_cnt = 0;
+ flow->united_cnt = 0;
+}
+
+void bpf_flow_free(struct bpf_verifier_env *env)
+{
+ struct bpf_flow *flow = env->flow;
+
+ if (!flow)
+ return;
+ bpf_flow_reset(env);
+ kvfree(flow->pos);
+ kvfree(flow->rank);
+ kvfree(flow->need_const);
+ kvfree(flow->stack_off);
+ kvfree(flow->tested);
+ kvfree(flow->point);
+ kvfree(flow->keep);
+ kvfree(flow->states);
+ kvfree(flow->enter);
+ kvfree(flow->nr_states);
+ kvfree(flow->funcs);
+ kvfree(flow->entry_up);
+ kvfree(flow->united);
+ kfree(flow);
+ env->flow = NULL;
+}
+
+bool bpf_flow_is_point(struct bpf_verifier_env *env, int insn_idx)
+{
+ return test_bit(insn_idx, env->flow->point);
+}
+
+/* A copy of @st that lives as long as the pass. */
+struct bpf_verifier_state *bpf_flow_keep_state(struct bpf_verifier_env *env,
+ const struct bpf_verifier_state *st)
+{
+ struct flow_kept *k = kzalloc_obj(*k, GFP_KERNEL_ACCOUNT);
+
+ if (!k)
+ return NULL;
+ list_add(&k->node, &env->flow->kept);
+ if (bpf_copy_verifier_state(&k->st, st))
+ return NULL;
+ return &k->st;
+}
+
+/*
+ * Both are in a loop whose iterations are kept apart: an iteration is done with before
+ * the next one begins. A back edge in the middle of the loop would run ahead otherwise and
+ * leave the rest of every iteration behind, see flow_earlier_trip(). The state that stands
+ * for all iterations comes last.
+ */
+static int flow_cmp_trips(const struct bpf_func_state *fa, const struct bpf_func_state *fb)
+{
+ const struct loop_stack_entry *ea, *eb;
+ u32 i, ta, tb;
+
+ for (i = 0; i < fa->loop_stack_cnt && i < fb->loop_stack_cnt; i++) {
+ ea = &fa->loop_stack[i];
+ eb = &fb->loop_stack[i];
+ if (ea->loop_id != eb->loop_id)
+ break;
+ ta = ea->apart ? ea->trip : U32_MAX;
+ tb = eb->apart ? eb->trip : U32_MAX;
+ if (ta != tb)
+ return ta < tb ? -1 : 1;
+ }
+ return 0;
+}
+
+static bool flow_less(struct rb_node *na, const struct rb_node *nb)
+{
+ const struct flow_state *a = rb_entry(na, struct flow_state, rb);
+ const struct flow_state *b = rb_entry(nb, struct flow_state, rb);
+ int cmp;
+ u32 i;
+
+ for (i = 0; i < a->key_len && i < b->key_len; i++) {
+ /* key[i] is the place in frame[i - 1] */
+ if (i) {
+ cmp = flow_cmp_trips(a->st.frame[i - 1], b->st.frame[i - 1]);
+ if (cmp)
+ return cmp < 0;
+ }
+ if (a->key[i] != b->key[i])
+ return a->key[i] < b->key[i];
+ }
+ if (a->key_len != b->key_len)
+ return a->key_len < b->key_len;
+ return a->seq < b->seq;
+}
+
+/*
+ * Callers first, so that the calls of a function get to it before it is looked
+ * at. Then by the place of the call in every frame: what is called is done
+ * with before what follows the call.
+ */
+static void flow_set_pending(struct bpf_flow *flow, struct flow_state *s)
+{
+ const struct bpf_verifier_state *st = &s->st;
+ u32 i;
+
+ s->key[0] = flow->rank[st->frame[0]->subprogno];
+ /* the place in the function makes the rank of the function redundant below */
+ for (i = 0; i < st->curframe && i + 2 < ARRAY_SIZE(s->key); i++)
+ s->key[i + 1] = flow->pos[st->frame[i + 1]->callsite];
+ s->key[i + 1] = flow->pos[st->insn_idx];
+ s->key_len = i + 2;
+ s->pending = true;
+ rb_add_cached(&s->rb, &flow->pending, flow_less);
+}
+
+static void flow_unlink(struct bpf_verifier_env *env, struct flow_state *s)
+{
+ struct bpf_flow *flow = env->flow;
+
+ if (s->pending)
+ rb_erase_cached(&s->rb, &flow->pending);
+ s->pending = false;
+ list_del(&s->node);
+ flow->nr_states[s->st.insn_idx]--;
+}
+
+/* Entries of a function whose states met are one set: what comes out is for all of them. */
+static u32 flow_entry_find(struct bpf_flow *flow, u32 e)
+{
+ while (flow->entry_up[e] != e) {
+ flow->entry_up[e] = flow->entry_up[flow->entry_up[e]];
+ e = flow->entry_up[e];
+ }
+ return e;
+}
+
+static int flow_entry_new(struct bpf_flow *flow)
+{
+ u32 *up, cap;
+
+ if (flow->entry_cnt + 1 >= flow->entry_cap) {
+ cap = max(64U, flow->entry_cap * 2);
+ up = kvrealloc(flow->entry_up, cap * sizeof(u32), GFP_KERNEL_ACCOUNT);
+ if (!up)
+ return -ENOMEM;
+ flow->entry_up = up;
+ flow->entry_cap = cap;
+ }
+ flow->entry_cnt++;
+ flow->entry_up[flow->entry_cnt] = flow->entry_cnt;
+ return flow->entry_cnt;
+}
+
+static bool flow_same_entry(struct bpf_flow *flow, const struct bpf_verifier_state *a,
+ const struct bpf_verifier_state *b)
+{
+ if (!a->flow_entry || !b->flow_entry)
+ return a->flow_entry == b->flow_entry;
+ return flow_entry_find(flow, a->flow_entry) == flow_entry_find(flow, b->flow_entry);
+}
+
+/* It delivers states, so it is done when nothing is in the middle of that. */
+static int flow_unite(struct bpf_flow *flow, const struct bpf_verifier_state *st, u32 a, u32 b)
+{
+ struct flow_united *u;
+ u32 cap;
+
+ if (!a || !b || flow_entry_find(flow, a) == flow_entry_find(flow, b))
+ return 0;
+ if (flow->united_cnt == flow->united_cap) {
+ cap = max(16U, flow->united_cap * 2);
+ u = kvrealloc(flow->united, cap * sizeof(*u), GFP_KERNEL_ACCOUNT);
+ if (!u)
+ return -ENOMEM;
+ flow->united = u;
+ flow->united_cap = cap;
+ }
+ u = &flow->united[flow->united_cnt++];
+ u->subprog = st->frame[0]->subprogno;
+ u->a = a;
+ u->b = b;
+ return 0;
+}
+
+/*
+ * What is known about a counted loop comes from the state it was entered
+ * with. @t stands for @s only if that holds for these states too.
+ */
+static bool flow_loops_within(struct bpf_verifier_env *env, const struct bpf_verifier_state *t,
+ const struct bpf_verifier_state *s)
+{
+ const struct loop_stack_entry *lt, *ls;
+ u32 fr, i;
+
+ for (fr = 0; fr <= t->curframe; fr++) {
+ if (t->frame[fr]->loop_stack_cnt != s->frame[fr]->loop_stack_cnt)
+ return false;
+ for (i = 0; i < t->frame[fr]->loop_stack_cnt; i++) {
+ lt = &t->frame[fr]->loop_stack[i];
+ ls = &s->frame[fr]->loop_stack[i];
+ if (!lt->terminates || lt->entry_state == ls->entry_state)
+ continue;
+ if (!ls->terminates ||
+ !bpf_state_subsumes(env, lt->entry_state, ls->entry_state))
+ return false;
+ }
+ }
+ return true;
+}
+
+/*
+ * bpf_iter_*_next() stops the loop if it is the same iterator in every trip.
+ * The one that was destroyed and created again starts from the beginning.
+ */
+static bool flow_same_iters(const struct bpf_verifier_state *t, const struct bpf_verifier_state *s)
+{
+ const struct bpf_stack_state *a, *b;
+ u32 fr, i;
+
+ if (t->curframe != s->curframe)
+ return false;
+ for (fr = 0; fr <= t->curframe; fr++) {
+ for (i = 0; i < bpf_stack_nr_slots(t->frame[fr]) &&
+ i < bpf_stack_nr_slots(s->frame[fr]); i++) {
+ a = bpf_stack_slot(t->frame[fr], i);
+ b = bpf_stack_slot(s->frame[fr], i);
+ if (a->slot_type[BPF_REG_SIZE - 1] == STACK_ITER &&
+ b->slot_type[BPF_REG_SIZE - 1] == STACK_ITER &&
+ a->spilled_ptr.id != b->spilled_ptr.id)
+ return false;
+ }
+ }
+ return true;
+}
+
+/* In a loop that has nothing but may_goto or bpf_iter_*_next() of the program to stop it */
+static bool flow_in_bounded_loop(struct bpf_verifier_env *env, const struct bpf_verifier_state *st)
+{
+ u32 fr, i, h;
+
+ for (fr = 0; fr <= st->curframe; fr++) {
+ for (i = 0; i < st->frame[fr]->loop_stack_cnt; i++) {
+ h = st->frame[fr]->loop_stack[i].loop_id;
+ if (env->insn_aux_data[h].loop->rt_bounded)
+ return true;
+ }
+ }
+ return false;
+}
+
+/* The loops around @st make new iterators. So it's not an iterator that stops them. */
+static int flow_iters_are_remade(struct bpf_verifier_env *env, const struct bpf_verifier_state *st)
+{
+ u32 fr, i, h;
+
+ for (fr = 0; fr <= st->curframe; fr++) {
+ for (i = 0; i < st->frame[fr]->loop_stack_cnt; i++) {
+ h = st->frame[fr]->loop_stack[i].loop_id;
+ env->insn_aux_data[h].loop->rt_bounded = false;
+ }
+ }
+ env->restart = true;
+ return -EINVAL;
+}
+
+static bool flow_within(struct bpf_verifier_env *env, struct bpf_verifier_state *t,
+ struct bpf_verifier_state *s)
+{
+ return t->curframe == s->curframe &&
+ t->frame[0]->subprogno == s->frame[0]->subprogno &&
+ bpf_state_subsumes(env, t, s) && flow_loops_within(env, t, s);
+}
+
+/*
+ * @t is of an earlier iteration of a loop whose iterations are kept apart than @s.
+ * What comes after @s is of later iterations too, so @t is not going to be
+ * of use. If something of its iteration does come it is looked at again.
+ */
+static bool flow_earlier_trip(const struct bpf_verifier_state *t,
+ const struct bpf_verifier_state *s)
+{
+ const struct bpf_func_state *ft, *fs;
+ u32 fr, i;
+
+ if (t->curframe != s->curframe)
+ return false;
+ for (fr = 0; fr <= t->curframe; fr++) {
+ ft = t->frame[fr];
+ fs = s->frame[fr];
+ if (ft->subprogno != fs->subprogno || ft->callsite != fs->callsite ||
+ ft->loop_stack_cnt != fs->loop_stack_cnt)
+ return false;
+ for (i = 0; i < ft->loop_stack_cnt; i++) {
+ if (ft->loop_stack[i].loop_id != fs->loop_stack[i].loop_id ||
+ ft->loop_stack[i].apart != fs->loop_stack[i].apart)
+ return false;
+ if (ft->loop_stack[i].trip != fs->loop_stack[i].trip)
+ return ft->loop_stack[i].trip + 1 < fs->loop_stack[i].trip;
+ }
+ }
+ return false;
+}
+
+static bool flow_two_consts(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb)
+{
+ return ra->type == SCALAR_VALUE && rb->type == SCALAR_VALUE &&
+ tnum_is_const(ra->var_off) && tnum_is_const(rb->var_off) &&
+ ra->var_off.value != rb->var_off.value;
+}
+
+/*
+ * A function returns one of a few numbers, true or false, less, equal or greater,
+ * the variant of an enum, and the caller is about to look at it. It may have
+ * no code for a number that is never returned.
+ */
+static bool flow_rets_differ(const struct bpf_verifier_state *a, const struct bpf_verifier_state *b)
+{
+ return flow_two_consts(&a->frame[a->curframe]->regs[BPF_REG_0],
+ &b->frame[b->curframe]->regs[BPF_REG_0]);
+}
+
+/* Constants that have to stay constants, see flow_compute_need_const() */
+static bool flow_needed_consts_differ(const struct flow_need *needs,
+ const struct bpf_verifier_state *a,
+ const struct bpf_verifier_state *b)
+{
+ const struct bpf_stack_state *sa, *sb;
+ const struct bpf_func_state *fa, *fb;
+ const struct flow_need *need;
+ u32 fr, i, nr;
+
+ if (!needs || a->curframe != b->curframe)
+ return false;
+ for (fr = 0; fr <= a->curframe; fr++) {
+ /* the callers need what they need when the call returns */
+ need = &needs[fr == a->curframe ? a->insn_idx : a->frame[fr + 1]->callsite + 1];
+ fa = a->frame[fr];
+ fb = b->frame[fr];
+ for (i = BPF_REG_0; need->regs && i < BPF_REG_FP; i++)
+ if ((need->regs & BIT(i)) && flow_two_consts(&fa->regs[i], &fb->regs[i]))
+ return true;
+ nr = min(bpf_stack_nr_slots(fa), bpf_stack_nr_slots(fb));
+ for (i = 0; i < nr && i < 64 && need->slots >> i; i++) {
+ sa = bpf_stack_slot(fa, i);
+ sb = bpf_stack_slot(fb, i);
+ if ((need->slots & BIT_ULL(i)) &&
+ bpf_is_spilled_scalar_reg(sa) && bpf_is_spilled_scalar_reg(sb) &&
+ flow_two_consts(&sa->spilled_ptr, &sb->spilled_ptr))
+ return true;
+ }
+ }
+ return false;
+}
+
+/*
+ * Put *@ps into @set, one of the sets of states at its insn. Returns
+ * 1 if it is within a state of the set. It is gone then.
+ * 0 if it is in the set now, as it was or joined with states of the set.
+ * @cycle: it came around a cycle, what grew is forgotten.
+ */
+/*
+ * A state is within another one, or has a join with it, only when the chains of calls and the
+ * loops they are in are the same, iterations that are kept apart included. A walk leaves
+ * a state per iteration at every point of the loop and of what it calls. Tell
+ * them apart before the registers are compared or, for a join, copied.
+ */
+static u32 flow_kind(const struct bpf_verifier_state *st)
+{
+ const struct loop_stack_entry *entry;
+ const struct bpf_func_state *frame;
+ u32 kind = st->curframe, fr, i;
+
+ for (fr = 0; fr <= st->curframe; fr++) {
+ frame = st->frame[fr];
+ kind = jhash_3words(frame->subprogno, frame->callsite, frame->loop_stack_cnt, kind);
+ for (i = 0; i < frame->loop_stack_cnt; i++) {
+ entry = &frame->loop_stack[i];
+ kind = jhash_3words(entry->loop_id << 2 | entry->terminates << 1 |
+ entry->apart,
+ entry->max_header_count, entry->trip, kind);
+ }
+ }
+ return kind;
+}
+
+/*
+ * Two things of which one is there: "if (hdr->ipv4) ... else if (hdr->ipv6)". The one that
+ * is not there is 0. What the other one is may be unknown, e.g. a pointer to the packet
+ * after the packet was changed. (0, x) is not within (y, 0) and the join (?, ?) forgets that
+ * it's one or the other.
+ * The same for a number that was compared with 0: 0 on one path, not 0 on the other.
+ */
+/*
+ * bpf_loop(N, ...) with a small N: the calls of the callback are looked at one by one, like
+ * the iterations of a loop that is walked. The call stops to enter the callback when
+ * callback_depth gets to N.
+ */
+static bool flow_callback_depth_differs(struct bpf_verifier_env *env,
+ const struct bpf_verifier_state *a,
+ const struct bpf_verifier_state *b)
+{
+ u32 max = BPF_FLOW_MAX_CALLBACKS, fr;
+ const struct bpf_func_state *fa, *fb;
+ struct bpf_insn *insn;
+
+ if (a->curframe != b->curframe)
+ return false;
+ /* at the call, and in the callback and what it calls */
+ for (fr = 0; fr <= a->curframe; fr++) {
+ insn = &env->prog->insnsi[fr == a->curframe ? a->insn_idx :
+ a->frame[fr + 1]->callsite];
+ if (!bpf_helper_call(insn) || insn->imm != BPF_FUNC_loop)
+ continue;
+ fa = a->frame[fr];
+ fb = b->frame[fr];
+ if (fa->callback_depth != fb->callback_depth &&
+ fa->regs[BPF_REG_1].type == SCALAR_VALUE &&
+ reg_umax(&fa->regs[BPF_REG_1]) <= max &&
+ fb->regs[BPF_REG_1].type == SCALAR_VALUE &&
+ reg_umax(&fb->regs[BPF_REG_1]) <= max)
+ return true;
+ }
+ return false;
+}
+
+static const struct bpf_reg_state *flow_slot_number(const struct bpf_func_state *frame, u32 i)
+{
+ const struct bpf_stack_state *slot = bpf_stack_slot(frame, i);
+
+ if (slot->slot_type[0] != STACK_SPILL || slot->slot_type[BPF_REG_SIZE - 1] != STACK_SPILL ||
+ slot->spilled_ptr.type != SCALAR_VALUE)
+ return NULL;
+ return &slot->spilled_ptr;
+}
+
+/* Returns true when the answer is known */
+static bool flow_null_pair(const struct bpf_reg_state *ra, const struct bpf_reg_state *rb,
+ bool *a_null, bool *b_null)
+{
+ if (tnum_equals_const(ra->var_off, 0) && !tnum_is_const(rb->var_off)) {
+ *a_null = true;
+ if (reg_umin(rb) == 1 && reg_umax(rb) == U64_MAX)
+ return true;
+ }
+ if (tnum_equals_const(rb->var_off, 0) && !tnum_is_const(ra->var_off)) {
+ *b_null = true;
+ if (reg_umin(ra) == 1 && reg_umax(ra) == U64_MAX)
+ return true;
+ }
+ return *a_null && *b_null;
+}
+
+static bool flow_null_differs(const struct bpf_verifier_state *a,
+ const struct bpf_verifier_state *b)
+{
+ const struct bpf_func_state *fa, *fb;
+ const struct bpf_reg_state *ra, *rb;
+ bool a_null = false, b_null = false;
+ u32 i;
+
+ if (a->curframe != b->curframe)
+ return false;
+ fa = a->frame[a->curframe];
+ fb = b->frame[b->curframe];
+ for (i = BPF_REG_0; i < BPF_REG_FP; i++) {
+ ra = &fa->regs[i];
+ rb = &fb->regs[i];
+ if (ra->type == SCALAR_VALUE && rb->type == SCALAR_VALUE &&
+ flow_null_pair(ra, rb, &a_null, &b_null))
+ return true;
+ }
+ for (i = 0; i < bpf_stack_nr_slots(fa) && i < bpf_stack_nr_slots(fb); i++) {
+ ra = flow_slot_number(fa, i);
+ rb = flow_slot_number(fb, i);
+ if (ra && rb && flow_null_pair(ra, rb, &a_null, &b_null))
+ return true;
+ }
+ return false;
+}
+
+static const struct bpf_reg_state *flow_number_at(const struct bpf_func_state *frame, u8 loc)
+{
+ const struct bpf_reg_state *reg;
+
+ if (loc <= BPF_REG_FP)
+ reg = &frame->regs[loc];
+ else if (loc - BPF_REG_FP - 1 < bpf_stack_nr_slots(frame) &&
+ bpf_is_spilled_scalar_reg(bpf_stack_slot(frame, loc - BPF_REG_FP - 1)))
+ reg = &bpf_stack_slot(frame, loc - BPF_REG_FP - 1)->spilled_ptr;
+ else
+ return NULL;
+ return reg->type == SCALAR_VALUE ? reg : NULL;
+}
+
+/*
+ * A register is set in one of the states and not in the other. Only live registers are in
+ * a state, so something ahead may read it. The state in which it's not set doesn't get there:
+ * a jump sends it another way. The join would not know which way to go, and would get
+ * there without the register. There is nothing to gain from such a join but a state less.
+ */
+static bool flow_unset_differs(const struct bpf_verifier_state *a,
+ const struct bpf_verifier_state *b)
+{
+ u32 fr, i;
+
+ for (fr = 0; fr <= a->curframe; fr++)
+ for (i = BPF_REG_0; i < BPF_REG_FP; i++)
+ if ((a->frame[fr]->regs[i].type == NOT_INIT) !=
+ (b->frame[fr]->regs[i].type == NOT_INIT))
+ return true;
+ return false;
+}
+
+static bool flow_disjoint(const struct bpf_reg_state *a, const struct bpf_reg_state *b)
+{
+ return cnum64_is_empty(cnum64_intersect(a->r64, b->r64)) ||
+ cnum32_is_empty(cnum32_intersect(a->r32, b->r32)) ||
+ ((a->var_off.value ^ b->var_off.value) & ~(a->var_off.mask | b->var_off.mask));
+}
+
+/* The smallest range that has both has numbers that are in neither */
+static bool flow_gap(const struct bpf_reg_state *a, const struct bpf_reg_state *b)
+{
+ if (cnum64_is_empty(cnum64_intersect(a->r64, b->r64)) &&
+ a->r64.size + b->r64.size + 1 > a->r64.size &&
+ cnum64_union(a->r64, b->r64).size > a->r64.size + b->r64.size + 1)
+ return true;
+ return cnum32_is_empty(cnum32_intersect(a->r32, b->r32)) &&
+ cnum32_union(a->r32, b->r32).size > (u64)a->r32.size + b->r32.size + 1;
+}
+
+/*
+ * Numbers always have a join. It must not make up what neither state has where a jump ahead
+ * can see it. flow->tested has the registers and the stack slots that a jump ahead compares
+ * with a constant.
+ *
+ * if (ret < 0) goto err; if (ret > 0) goto skip; use(p);
+ * ret = -2 and ret = 1, both without p, meet in front of that. [-2, 1] has 0, gets through both
+ * jumps, and there is no p. So two ranges that leave a gap are not joined. Ranges that
+ * touch, 0 and 1, -1 and 0, or have something in common lose nothing.
+ *
+ * err = update(); ret = err < 0 ? DROP : 0; ... if (err < 0) goto out; ... if (ret < 0) ...
+ * (err >= 0, ret = 0) and (err < 0, ret = DROP) meet. The join has err < 0 with ret = 0. So
+ * two numbers, each with nothing in common between the states, are not joined either.
+ *
+ * The states go on to the jumps on their own. Behind the last one the numbers are dead, and what
+ * is left is joined.
+ */
+static bool flow_join_makes_up(const struct bpf_flow *flow, const struct bpf_verifier_state *a,
+ const struct bpf_verifier_state *b)
+{
+ const struct bpf_reg_state *ra, *rb;
+ const struct flow_need *need;
+ u32 fr, i, id = 0;
+ bool one = false;
+
+ if (!flow->tested)
+ return false;
+ for (fr = 0; fr <= a->curframe; fr++) {
+ /* the callers go on when the call returns */
+ need = &flow->tested[fr == a->curframe ? a->insn_idx :
+ a->frame[fr + 1]->callsite + 1];
+ for (i = BPF_REG_0; i < BPF_REG_FP + 1 + 64; i++) {
+ if (i <= BPF_REG_FP ? !(need->regs & BIT(i)) :
+ !(need->slots & BIT_ULL(i - BPF_REG_FP - 1)))
+ continue;
+ ra = flow_number_at(a->frame[fr], i);
+ rb = flow_number_at(b->frame[fr], i);
+ if (!ra || !rb || !flow_disjoint(ra, rb))
+ continue;
+ if (flow_gap(ra, rb))
+ return true;
+ /* copies of a number are one number */
+ if (one && (!id || id != (ra->id & ~BPF_ADD_CONST)))
+ return true;
+ one = true;
+ id = ra->id & ~BPF_ADD_CONST;
+ }
+ }
+ return false;
+}
+
+/*
+ * States that have a join are kept apart while there is room: half of every limit.
+ * After that they are joined, which may lose what the prog needs, and does not run into the limit.
+ */
+static bool flow_has_room(const struct bpf_verifier_env *env, u32 nr_states)
+{
+ u64 len = env->prog->len;
+
+ return nr_states < BPF_FLOW_MAX_STATES / 2 &&
+ env->flow->live_states < (BPF_FLOW_LIVE_PER_INSN * len + BPF_FLOW_LIVE_BASE) / 2 &&
+ env->total_states < (BPF_FLOW_TOTAL_PER_INSN * len + BPF_FLOW_TOTAL_BASE) / 2;
+}
+
+static int flow_merge(struct bpf_verifier_env *env, struct list_head *set,
+ struct flow_state **ps, enum flow_cycle cycle)
+{
+ struct bpf_flow *flow = env->flow;
+ struct flow_state *s = *ps, *t, *tn;
+ int insn_idx = s->st.insn_idx, err;
+ bool widen_all, widen, iters;
+ /* the set of the states that enter a loop is a stop on the way, see bpf_flow_deliver() */
+ bool log = (env->log.level & BPF_LOG_LEVEL2) && set != &flow->enter[insn_idx];
+ const char *what = NULL;
+ int prev = s->prev_insn_idx;
+ u32 other_iters;
+
+ /*
+ * Not only where the cycle is closed: the state with a new iterator must not
+ * disappear in the one with the old iterator on the way there.
+ */
+ iters = flow_in_bounded_loop(env, &s->st);
+again:
+ other_iters = 0;
+ if (signal_pending(current)) {
+ err = -EAGAIN;
+ goto err_free;
+ }
+ if (need_resched())
+ cond_resched();
+
+ s->kind = flow_kind(&s->st);
+ list_for_each_entry(t, set, node) {
+ if (t->kind != s->kind ||
+ !flow_within(env, &t->st, &s->st))
+ continue;
+ if (iters && !flow_same_iters(&t->st, &s->st))
+ continue;
+ if (env->log.level & BPF_LOG_LEVEL2)
+ verbose(env, "from %d to %d: within\n", prev, insn_idx);
+ err = flow_unite(flow, &s->st, s->st.flow_entry, t->st.flow_entry);
+ flow_free_state(env, s);
+ return err ?: 1;
+ }
+
+ widen_all = cycle == FLOW_CYCLE || s->joins > BPF_FLOW_MAX_JOINS;
+ list_for_each_entry_safe(t, tn, set, node) {
+ if (t->st.frame[0]->subprogno != s->st.frame[0]->subprogno)
+ continue;
+ widen = widen_all || (cycle == FLOW_CALLBACK_CYCLE &&
+ max(s->joins, t->joins) >= BPF_FLOW_CALLBACK_JOINS);
+ if (!t->pending && flow_earlier_trip(&t->st, &s->st)) {
+ flow_unlink(env, t);
+ flow_free_state(env, t);
+ continue;
+ }
+ if (t->kind != s->kind)
+ continue;
+ if (iters && !flow_same_iters(&t->st, &s->st)) {
+ if (++other_iters < BPF_FLOW_MAX_ITERS)
+ continue;
+ err = flow_iters_are_remade(env, &s->st);
+ goto err_free;
+ }
+ if (!widen && flow_has_room(env, flow->nr_states[insn_idx]) &&
+ flow_null_differs(&t->st, &s->st))
+ continue;
+ if (!widen && flow_needed_consts_differ(flow->need_const, &t->st, &s->st))
+ continue;
+ if (flow_callback_depth_differs(env, &t->st, &s->st))
+ continue;
+ /* what grows in a loop is not replaced with what is larger, it is forgotten */
+ if (!widen && flow_within(env, &s->st, &t->st)) {
+ err = flow_unite(flow, &s->st, s->st.flow_entry, t->st.flow_entry);
+ if (err)
+ goto err_free;
+ s->joins = max(s->joins, t->joins);
+ flow_unlink(env, t);
+ flow_free_state(env, t);
+ what = what ?: "joined";
+ continue;
+ }
+ if (!widen && flow_has_room(env, flow->nr_states[insn_idx]) &&
+ (flow_unset_differs(&t->st, &s->st) ||
+ flow_join_makes_up(flow, &t->st, &s->st)))
+ continue;
+ /*
+ * Around a loop too:
+ * u8 stack[10], depth = 0;
+ * loop {
+ * if (...) { if (depth >= 10) break; stack[depth++] = x; }
+ * else { depth--; if (depth >= 10) break; x = stack[depth]; }
+ * }
+ * depth is in a 32-bit register. A compiler tests 'depth & 255' and computes it
+ * anew for the index. What is found out about a copy says nothing about
+ * a range, and everything about a constant.
+ */
+ if (flow_has_room(env, flow->nr_states[insn_idx]) &&
+ flow_needed_consts_differ(flow->stack_off, &t->st, &s->st))
+ continue;
+ err = bpf_states_join(env, &flow->tmp, &t->st, &s->st, widen);
+ if (err < 0)
+ goto err_free;
+ if (err)
+ continue;
+ what = widen ? "widened" : "joined";
+ err = flow_unite(flow, &s->st, s->st.flow_entry, t->st.flow_entry);
+ if (err)
+ goto err_free;
+ flow->tmp.flow_entry = t->st.flow_entry;
+ flow->tmp.last_insn_idx = s->prev_insn_idx;
+ s->joins = max(s->joins, t->joins) + 1;
+ bpf_free_verifier_state(&s->st, false);
+ s->st = flow->tmp;
+ memset(&flow->tmp, 0, sizeof(flow->tmp));
+ flow_unlink(env, t);
+ flow_free_state(env, t);
+ /* the join may be within, or have a join with, what is left in the set */
+ goto again;
+ }
+
+ if (flow->nr_states[insn_idx] >= BPF_FLOW_MAX_STATES) {
+ verbose(env, "more than %d states that cannot be joined at insn %d\n",
+ BPF_FLOW_MAX_STATES, insn_idx);
+ err = -E2BIG;
+ goto err_free;
+ }
+ if (flow->live_states > BPF_FLOW_LIVE_PER_INSN * env->prog->len + BPF_FLOW_LIVE_BASE ||
+ env->total_states >
+ BPF_FLOW_TOTAL_PER_INSN * (u64)env->prog->len + BPF_FLOW_TOTAL_BASE) {
+ verbose(env, "too many states for %d insns: %d, %d at a time\n",
+ env->prog->len, env->total_states, flow->live_states);
+ err = -E2BIG;
+ goto err_free;
+ }
+ /*
+ * The state is printed when its turn comes. That says enough about a state that
+ * is new, and about a state that goes on to the next insn.
+ */
+ if (prev >= 0 && insn_idx == prev + 1 + bpf_is_ldimm64(&env->prog->insnsi[prev]) &&
+ what && !strcmp(what, "joined"))
+ what = NULL;
+ if (log && what)
+ verbose(env, "from %d to %d: %s\n", prev, insn_idx, what);
+ list_add_tail(&s->node, set);
+ flow->nr_states[insn_idx]++;
+ env->max_states_per_insn = max(env->max_states_per_insn, flow->nr_states[insn_idx]);
+ return 0;
+
+err_free:
+ flow_free_state(env, s);
+ return err;
+}
+
+static struct flow_state *flow_new_state(struct bpf_verifier_env *env, int prev_insn_idx)
+{
+ struct flow_state *s = kzalloc_obj(*s, GFP_KERNEL_ACCOUNT);
+ struct bpf_flow *flow = env->flow;
+
+ if (!s)
+ return NULL;
+ s->prev_insn_idx = prev_insn_idx;
+ s->seq = ++flow->seq;
+ env->total_states++;
+ flow->live_states++;
+ env->peak_states = max(env->peak_states, flow->live_states);
+ return s;
+}
+
+/* What @st holds is taken and @st is left empty. What doesn't matter at @insn_idx goes. */
+static struct flow_state *flow_arrive(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ int insn_idx, int prev_insn_idx, bool *enters)
+{
+ struct bpf_verifier_state *cur = env->cur_state;
+ int cur_idx = env->insn_idx, err;
+ struct flow_state *s;
+
+ if (insn_idx < 0 || insn_idx >= env->prog->len) {
+ verbose(env, "invalid insn idx %d insn_cnt %d\n", insn_idx, env->prog->len);
+ return ERR_PTR(-EFAULT);
+ }
+
+ s = flow_new_state(env, prev_insn_idx);
+ if (!s)
+ return ERR_PTR(-ENOMEM);
+ s->st = *st;
+ memset(st, 0, sizeof(*st));
+ s->st.insn_idx = insn_idx;
+ s->st.last_insn_idx = prev_insn_idx;
+
+ env->cur_state = &s->st;
+ env->insn_idx = insn_idx;
+ err = bpf_flow_loop_leave(env, enters);
+ if (!err)
+ err = bpf_clean_verifier_state(env, &s->st);
+ env->cur_state = cur;
+ env->insn_idx = cur_idx;
+ if (err) {
+ flow_free_state(env, s);
+ return ERR_PTR(err);
+ }
+ /* when the registers that it was shared with are dead, or are those of the caller */
+ bpf_clear_singular_ids(env, &s->st);
+ return s;
+}
+
+/*
+ * @st gets to @insn_idx if the guard at @prev_insn_idx is going to leave the loop, see
+ * check_loop_guard(). Nothing is joined: the states may never meet.
+ */
+int bpf_flow_park(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ int insn_idx, int prev_insn_idx)
+{
+ struct bpf_flow *flow = env->flow;
+ struct flow_state *s, *t, *tn;
+ bool enters;
+
+ s = flow_arrive(env, st, insn_idx, prev_insn_idx, &enters);
+ if (IS_ERR(s))
+ return PTR_ERR(s);
+ list_for_each_entry_safe(t, tn, &flow->parked, node) {
+ if (t->prev_insn_idx != prev_insn_idx || !flow_same_entry(flow, &t->st, &s->st))
+ continue;
+ if (flow_within(env, &t->st, &s->st)) {
+ flow_free_state(env, s);
+ return 0;
+ }
+ if (flow_within(env, &s->st, &t->st)) {
+ list_del(&t->node);
+ flow_free_state(env, t);
+ }
+ }
+ list_add_tail(&s->node, &flow->parked);
+ return 0;
+}
+
+/* The guard at @prev_insn_idx leaves the loop */
+void bpf_flow_unpark(struct bpf_verifier_env *env, int prev_insn_idx)
+{
+ struct bpf_flow *flow = env->flow;
+ struct flow_state *s, *sn;
+
+ list_for_each_entry_safe(s, sn, &flow->parked, node)
+ if (s->prev_insn_idx == prev_insn_idx)
+ list_move_tail(&s->node, &flow->unparked);
+}
+
+int bpf_flow_deliver_unparked(struct bpf_verifier_env *env)
+{
+ struct bpf_flow *flow = env->flow;
+ struct flow_state *s;
+ int err = 0;
+
+ while (!err && !list_empty(&flow->unparked)) {
+ s = list_first_entry(&flow->unparked, struct flow_state, node);
+ list_del(&s->node);
+ err = bpf_flow_deliver(env, &s->st, s->st.insn_idx, s->prev_insn_idx);
+ flow_free_state(env, s);
+ }
+ return err;
+}
+
+/*
+ * @st gets to @insn_idx. What it holds is taken and @st is left empty.
+ */
+int bpf_flow_deliver(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ int insn_idx, int prev_insn_idx)
+{
+ struct bpf_verifier_state *cur = env->cur_state;
+ bool backedge = false, enters = false, drop = false;
+ struct bpf_flow *flow = env->flow;
+ enum flow_cycle cycle;
+ int cur_idx = env->insn_idx, err;
+ struct bpf_subprog_info *sub;
+ struct flow_state *s, *e;
+
+ s = flow_arrive(env, st, insn_idx, prev_insn_idx, &enters);
+ if (IS_ERR(s))
+ return PTR_ERR(s);
+
+ /*
+ * Cycles that have no loop head: the callback that came back to the helper
+ * that calls it, and calls that go through the entry of a function.
+ */
+ sub = bpf_find_containing_subprog(env, insn_idx);
+ cycle = FLOW_NO_CYCLE;
+ if (sub->recursive && sub->start == insn_idx)
+ cycle = FLOW_CYCLE;
+ else if (env->insn_aux_data[insn_idx].calls_callback && prev_insn_idx >= 0 &&
+ env->prog->insnsi[prev_insn_idx].code == (BPF_JMP | BPF_EXIT))
+ cycle = FLOW_CALLBACK_CYCLE;
+
+ /*
+ * What is known about a loop is derived from the state that enters it. So
+ * the states that enter are a set of their own, and what gets to
+ * the set at the head of the loop is derived from what is new in it.
+ */
+ if (enters) {
+ err = flow_merge(env, &flow->enter[insn_idx], &s, cycle);
+ if (err)
+ return err < 0 ? err : 0;
+ cycle = FLOW_NO_CYCLE;
+ e = s;
+ s = flow_new_state(env, prev_insn_idx);
+ if (!s)
+ return -ENOMEM;
+ err = bpf_copy_verifier_state(&s->st, &e->st);
+ if (err)
+ goto err_free;
+ }
+
+ env->cur_state = &s->st;
+ env->insn_idx = insn_idx;
+ err = bpf_flow_loop_enter(env, &backedge, &drop);
+ env->cur_state = cur;
+ env->insn_idx = cur_idx;
+ if (err || drop)
+ goto err_free;
+
+ err = flow_merge(env, &flow->states[insn_idx], &s, backedge ? FLOW_CYCLE : cycle);
+ if (err)
+ return err < 0 ? err : 0;
+ flow_set_pending(flow, s);
+ return 0;
+
+err_free:
+ flow_free_state(env, s);
+ return err;
+}
+
+static int flow_apply_exit(struct bpf_verifier_env *env, struct flow_waiter *w,
+ struct flow_exit *x)
+{
+ struct bpf_verifier_state st = {};
+ int err;
+
+ if (w->dead)
+ return 0;
+ err = bpf_copy_verifier_state(&st, &w->st);
+ if (!err)
+ err = bpf_flow_return(env, &st, &x->st, w->call_idx);
+ if (!err)
+ err = bpf_flow_deliver(env, &st, w->call_idx + 1, w->call_idx);
+ if (err == 1) {
+ struct flow_state *s;
+
+ /*
+ * Do the call again. Not with bpf_flow_deliver(): the state was
+ * at the insn before, it may be there still.
+ */
+ w->dead = true;
+ s = flow_new_state(env, w->st.last_insn_idx);
+ if (!s) {
+ err = -ENOMEM;
+ goto out;
+ }
+ err = bpf_copy_verifier_state(&s->st, &w->st);
+ if (err) {
+ flow_free_state(env, s);
+ goto out;
+ }
+ s->st.insn_idx = w->call_idx;
+ s->kind = flow_kind(&s->st);
+ list_add_tail(&s->node, &env->flow->states[w->call_idx]);
+ env->flow->nr_states[w->call_idx]++;
+ flow_set_pending(env->flow, s);
+ }
+out:
+ bpf_free_verifier_state(&st, false);
+ return err;
+}
+
+static int flow_do_united(struct bpf_verifier_env *env)
+{
+ struct bpf_flow *flow = env->flow;
+ struct flow_waiter *w;
+ struct flow_united u;
+ struct flow_func *f;
+ struct flow_exit *x;
+ u32 a, b, we, xe;
+ int err;
+
+ while (flow->united_cnt) {
+ u = flow->united[--flow->united_cnt];
+ a = flow_entry_find(flow, u.a);
+ b = flow_entry_find(flow, u.b);
+ if (a == b)
+ continue;
+ f = &flow->funcs[u.subprog];
+ /* what came out for one goes to the calls that wait for the other */
+ list_for_each_entry(x, &f->exits, node) {
+ xe = flow_entry_find(flow, x->entry);
+ if (xe != a && xe != b)
+ continue;
+ list_for_each_entry(w, &f->waiters, node) {
+ we = flow_entry_find(flow, w->entry);
+ if (we == xe || (we != a && we != b))
+ continue;
+ err = flow_apply_exit(env, w, x);
+ if (err)
+ return err;
+ }
+ }
+ flow->entry_up[a] = b;
+ }
+ return 0;
+}
+
+/*
+ * The call at @call_idx in @caller of a function that is verified on its own,
+ * @entry is the state at its first insn. Both are taken.
+ */
+int bpf_flow_call(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+ struct bpf_verifier_state *entry, int subprog, int call_idx)
+{
+ struct bpf_flow *flow = env->flow;
+ struct flow_func *f = &flow->funcs[subprog];
+ struct flow_waiter *w, *o, *on;
+ int id;
+
+ caller->insn_idx = call_idx;
+ list_for_each_entry_safe(o, on, &f->waiters, node) {
+ if (o->call_idx != call_idx || o->dead)
+ continue;
+ /* it gets what the call that is there gets */
+ if (o->st.flow_entry == caller->flow_entry && flow_within(env, &o->st, caller))
+ return 0;
+ }
+
+ id = flow_entry_new(flow);
+ if (id < 0)
+ return id;
+ w = kzalloc_obj(*w, GFP_KERNEL_ACCOUNT);
+ if (!w)
+ return -ENOMEM;
+ w->st = *caller;
+ memset(caller, 0, sizeof(*caller));
+ w->call_idx = call_idx;
+ w->entry = id;
+ list_add_tail(&w->node, &f->waiters);
+
+ entry->flow_entry = id;
+ if (env->log.level & BPF_LOG_LEVEL2)
+ verbose(env, "call at %d waits for func#%d\n", call_idx, subprog);
+ return bpf_flow_deliver(env, entry, env->subprog_info[subprog].start, call_idx);
+}
+
+/* @st came to exit of the function that is verified on its own. It is taken. */
+int bpf_flow_exit(struct bpf_verifier_env *env, struct bpf_verifier_state *st, int exit_idx)
+{
+ struct bpf_flow *flow = env->flow;
+ struct flow_func *f = &flow->funcs[st->frame[0]->subprogno];
+ struct flow_exit *x, *o, *on;
+ struct flow_waiter *w;
+ u16 joins = 0;
+ u32 entry, cnt = 0;
+ int err;
+
+ st->insn_idx = exit_idx;
+ err = bpf_clean_verifier_state(env, st);
+ if (err)
+ return err;
+ if (f->exit_idx < 0)
+ f->exit_idx = exit_idx;
+ st->insn_idx = f->exit_idx;
+ entry = flow_entry_find(flow, st->flow_entry);
+
+ list_for_each_entry_safe(o, on, &f->exits, node) {
+ if (flow_entry_find(flow, o->entry) != entry)
+ continue;
+ if (bpf_state_subsumes(env, &o->st, st))
+ return 0;
+ if (flow_has_room(env, ++cnt) && flow_rets_differ(&o->st, st))
+ continue;
+ /* what comes out of a recursive function gets into it again */
+ err = bpf_states_join(env, &flow->tmp, &o->st, st,
+ env->subprog_info[st->frame[0]->subprogno].recursive ||
+ o->joins > BPF_FLOW_MAX_JOINS);
+ if (err < 0)
+ return err;
+ if (err)
+ continue;
+ joins = max(joins, o->joins) + 1;
+ bpf_free_verifier_state(st, false);
+ *st = flow->tmp;
+ memset(&flow->tmp, 0, sizeof(flow->tmp));
+ st->flow_entry = entry;
+ list_del(&o->node);
+ bpf_free_verifier_state(&o->st, false);
+ kfree(o);
+ }
+
+ x = kzalloc_obj(*x, GFP_KERNEL_ACCOUNT);
+ if (!x)
+ return -ENOMEM;
+ x->st = *st;
+ memset(st, 0, sizeof(*st));
+ x->entry = entry;
+ x->joins = joins;
+ list_add_tail(&x->node, &f->exits);
+
+ list_for_each_entry(w, &f->waiters, node) {
+ if (flow_entry_find(flow, w->entry) != entry)
+ continue;
+ err = flow_apply_exit(env, w, x);
+ if (err)
+ return err;
+ }
+ return 0;
+}
+
+/*
+ * Make the next pending state the current one. env->cur_state has to be empty.
+ * Returns -ENOENT when nothing is pending.
+ */
+int bpf_flow_next(struct bpf_verifier_env *env, int *prev_insn_idx)
+{
+ struct bpf_flow *flow = env->flow;
+ struct flow_state *s;
+ struct rb_node *n;
+ int err;
+
+ err = flow_do_united(env);
+ if (err)
+ return err;
+ n = rb_first_cached(&flow->pending);
+ if (!n)
+ return -ENOENT;
+ s = rb_entry(n, struct flow_state, rb);
+ rb_erase_cached(&s->rb, &flow->pending);
+ s->pending = false;
+
+ env->insn_idx = s->st.insn_idx;
+ *prev_insn_idx = s->prev_insn_idx;
+ if (test_bit(s->st.insn_idx, flow->keep))
+ return bpf_copy_verifier_state(env->cur_state, &s->st);
+ *env->cur_state = s->st;
+ memset(&s->st, 0, sizeof(s->st));
+ s->st.insn_idx = env->insn_idx;
+ flow_unlink(env, s);
+ flow_free_state(env, s);
+ return 0;
+}
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 52/81] bpf: Add the main pass as a fixpoint over the control flow graph
2026-10-26 2:59 ` [PATCH bpf 52/81] bpf: Add the main pass as a fixpoint over the control flow graph Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 16535 bytes --]
> bpf: Add the main pass as a fixpoint over the control flow graph
TL;DR:
- Verifier may accept never-ending loops: a destroyed or nested iterator's
next() counts as a bound
- Possible NULL frame dereference in flow_unset_differs() on curframe mismatch
- Subprog return merges may reject valid programs
- Rest: comment and changelog nits
> bpf: Add the main pass as a fixpoint over the control flow graph
>
> [ ... ]
>
> This does not prove that a loop ends. A loop gets may_goto unless the
> number of iterations is known, or each iteration executes may_goto or
> bpf_iter_*_next(). flow_place_guards() puts may_goto before a conditional
> jump that exits the loop, with the same target. The prog can arrive there
> with the same state through that jump.
flow_place_guards() also has a 'header:' fallback. It is taken when the back
edges overflow, for an irreducible loop whose back edges are not exiting
jumps, or for a back edge that has no latch in the loop itself.
There, may_goto is placed at the loop head, and its target is exits[0].to of
the outermost loop around it. That is not an edge of the program, so the
state at the head may not match any state that reaches that target, which is
why check_loop_guard() sets the live NOT_INIT registers to 0 for
guard_at_header. The fallback also places no may_goto at all when the
outermost loop has no recorded exit.
Could the changelog describe the head fallback and how the verifier handles
its edge?
> diff --git a/kernel/bpf/flow.c b/kernel/bpf/flow.c
> new file mode 100644
> index 000000000000..fc5272ed2af8
> --- /dev/null
> +++ b/kernel/bpf/flow.c
> @@ -0,0 +1,1817 @@
[ ... ]
> +/* Does the insn at @idx stop the loop @h? */
> +static bool flow_is_bound(struct bpf_verifier_env *env, int h, int idx)
> +{
> + struct bpf_insn *insn = &env->prog->insnsi[idx];
> + int t;
> +
> + /* Once it is out of budget it's a jump. It has to leave the loop. */
> + if (bpf_is_may_goto_insn(insn)) {
> + t = bpf_loop_at_index(env, idx + insn->off + 1);
> + return t != h && !bpf_is_nested_loop(env, t, h);
> + }
> + /* as long as the program doesn't go on when there is nothing left, see iter_is_drained() */
> + return bpf_insn_is_iter_next(env, insn);
> +}
[ ... ]
> + /* see flow_place_guards() about loops with more than one entry */
> + loop->rt_bounded = !loop->backedges_overflow && !loop->irreducible;
> + for (i = 0; i < loop->backedges_cnt && loop->rt_bounded; i++) {
> + for (n = loop->backedges[i].from; n >= 0 && n != h; n = env->idoms[n])
> + if (flow_is_bound(env, h, n))
> + break;
> + if (n < 0 || !flow_is_bound(env, h, n))
> + loop->rt_bounded = false;
> + }
> + /* also when it's bounded: the main pass may find that it's not */
> + flow_place_guards(env, h, seen);
> + }
Can flow_compute_loop_bounds() mark a loop rt_bounded when the
bpf_iter_*_next() call that dominates its back edge belongs to an iterator
that is created and destroyed within one trip?
flow_is_bound() only checks bpf_insn_is_iter_next(), so a call in a nested
loop counts as a bound for the outer loop, and nothing checks that the same
iterator is alive across trips. An rt_bounded loop never gets may_goto from
check_loop_guard(), and the "never exits" rejection in bpf_flow_loop_enter()
is skipped for it:
if (!loop->rt_bounded && !loop->has_guards)
For example, the usual bpf_for() pattern inside an unbounded loop:
r6 = &one ll ; global in .data, unknown to the verifier
1: r1 = r10; r1 += -8; r2 = 0; r3 = 2
call bpf_iter_num_new
2: r1 = r10; r1 += -8
call bpf_iter_num_next
if r0 != 0 goto 2b
r1 = r10; r1 += -8
call bpf_iter_num_destroy
r1 = *(u64 *)(r6 + 0)
if r1 != 0 goto 1b ; back edge of the loop at 1
r0 = 0; exit
The idom chain from the back edge reaches the bpf_iter_num_next() call of
the inner loop, so the loop at 1 gets rt_bounded = true. Scalar evolution
cannot count the loop, and because rt_bounded is set, bpf_flow_loop_enter()
checks the body once.
bpf_iter_num_destroy() makes unmark_stack_slots_iter() set the iterator slot
to STACK_INVALID, so the state that comes back to 1 has no iterator.
flow_same_iters() is true and flow_merge() drops the state as within the
entry state, and the pass ends after one trip. Only one iterator id ever
reaches the inner loop, so flow_iters_are_remade() never runs, and
iter_is_drained() is never reached because every trip calls next() on a new
active iterator.
The program then loads with no may_goto in the outer loop and never ends
while 'one' is not 0. With 'for (;;)' and no exit, flow_place_guards()
places no guard either, and the "never exits" rejection is skipped, so a
loop that never ends is accepted. The same happens without a nested loop:
while (x) {
bpf_iter_num_new(&it, 0, 1);
bpf_iter_num_next(&it);
bpf_iter_num_destroy(&it);
}
Before the series, bpf_is_state_visited() rejected these programs with
"infinite loop detected". The commit message says "bpf_iter_*_next() stops
a loop only if all iterations use the same iterator", but nothing enforces
that when the iterator does not survive to the loop head.
flow.c is first built by 0535bc235aed ("bpf: Verify programs as a fixpoint
over the control flow graph"), and flow_is_bound(), check_loop_guard(),
iter_is_drained() and flow_iters_are_remade() are unchanged up to the end of
the series, so no later fix was found.
Should an iter_next count as a bound only if its iterator slot is still
STACK_ITER in the state at the loop head, and otherwise clear rt_bounded and
restart, as iter_is_drained() does?
[ ... ]
> + if (class == BPF_JMP && BPF_OP(insn->code) == BPF_CALL) {
> + in.regs &= ~GENMASK(BPF_REG_5, BPF_REG_0);
> + s = idx + insn->imm + 1;
> + if (bpf_pseudo_call(insn) &&
> + !bpf_subprog_is_global(env, bpf_find_subprog(env, s)))
> + in.regs |= need[s].regs & GENMASK(BPF_REG_5, BPF_REG_1);
Does this miss the case where a callee that runs in its own frame returns a
value that the caller tests against a constant, or needs as a constant?
Needs go into a callee only through its arguments: need[s] is added to r1-r5
at the call. A BPF_EXIT has no successors in bpf_insn_successors(), so
need[exit] is 0 for all three kinds, and nothing carries a need from the
insn after a call back into the callee's exit. The callee's r0 is never
"tested" (or a "needed constant") at its exit, even when every caller
compares it with a constant right after the call.
This matters for a static subprog that is not verified on its own.
func_is_on_its_own() (wired in 0535bc235aed) returns false for a callee that
gets a PTR_TO_STACK argument, a nullable pointer, packet_end, or that is
called from a callback. The consumers then look at:
need = &flow->tested[fr == a->curframe ? a->insn_idx :
a->frame[fr + 1]->callsite + 1];
For the callee frame this is tested[exit], which is 0. For the caller
frame, tested[callsite + 1] does have r0, but the caller's r0 is NOT_INIT
while the callee runs, because check_func_call() calls
clear_caller_saved_regs() after setup_func_entry(). So flow_number_at()
returns NULL and the register is skipped. flow_rets_differ(), the rule for
"a constant that a subprog returns", is used only in bpf_flow_exit(), the
path for functions that are verified on their own.
Take the motivating example from the commit message, with the lookup
written as a subprog that takes &p:
static __noinline int get(struct val **pp) /* pp is on the caller's stack */
{
struct val *v = bpf_map_lookup_elem(&map, &key);
if (!v)
return -2;
if (v->skip)
return 1;
*pp = v;
return 0;
}
...
struct val *p;
int ret = get(&p);
if (ret < 0) goto err;
if (ret > 0) goto skip;
use(p);
The callee runs in a frame because of the PTR_TO_STACK argument. Its return
paths jump to one shared exit insn, which is a point, and the states
(r0=-2, p unset) and (r0=1, p unset) meet there. flow_unset_differs() and
flow_null_differs() do not refuse them, and flow_join_makes_up() sees
tested[exit] == 0, so flow_merge() joins them into r0=[-2,1] with p unset.
That state returns to the caller, takes the r0 == 0 path through both jumps,
and reaches use(p) without p. The program is rejected, although the old
path-by-path verifier accepted it.
The same gap affects FLOW_NEED_ARG. A callee in a frame that returns 16 or
32, with the result passed as the size of bpf_ringbuf_reserve(), has the two
constants joined at its exit, and the helper then sees a non-constant size.
The commit message says the verifier keeps such states apart ("a tested
register, and its ranges have a gap ret=-2 and ret=1"), which does not hold
when the states meet inside a callee that runs in a frame.
No later commit in the series changes flow.c. Would it work to give each
exit of a static subprog the union of need[call + 1] over its call sites,
limited to r0 (and r1 for bpf_ret_reg_pair())?
[ ... ]
> +/*
> + * Put *@ps into @set, one of the sets of states at its insn. Returns
> + * 1 if it is within a state of the set. It is gone then.
> + * 0 if it is in the set now, as it was or joined with states of the set.
> + * @cycle: it came around a cycle, what grew is forgotten.
> + */
> +/*
> + * A state is within another one, or has a join with it, only when the chains of calls and the
> + * loops they are in are the same, iterations that are kept apart included. A walk leaves
> + * a state per iteration at every point of the loop and of what it calls. Tell
> + * them apart before the registers are compared or, for a join, copied.
> + */
> +static u32 flow_kind(const struct bpf_verifier_state *st)
[ ... ]
> +/*
> + * Two things of which one is there: "if (hdr->ipv4) ... else if (hdr->ipv6)". The one that
> + * is not there is 0. What the other one is may be unknown, e.g. a pointer to the packet
> + * after the packet was changed. (0, x) is not within (y, 0) and the join (?, ?) forgets that
> + * it's one or the other.
> + * The same for a number that was compared with 0: 0 on one path, not 0 on the other.
> + */
> +/*
> + * bpf_loop(N, ...) with a small N: the calls of the callback are looked at one by one, like
> + * the iterations of a loop that is walked. The call stops to enter the callback when
> + * callback_depth gets to N.
> + */
> +static bool flow_callback_depth_differs(struct bpf_verifier_env *env,
This isn't a bug, but the comment that documents flow_merge() (@ps, @set,
@cycle and the 1/0 return values) sits directly above flow_kind(), which
takes one const state and returns a hash. flow_merge() itself is much
further down and its only header is the flow_has_room() comment.
Could this block move to flow_merge()? It could also mention the negative
error returns (-EAGAIN, -E2BIG, -EINVAL with env->restart set, -ENOMEM), and
that *ps has been freed on every non-zero return, since bpf_flow_deliver()
relies on that with:
return err < 0 ? err : 0;
The same thing happens a second time here: the comment starting "Two things
of which one is there" describes flow_null_pair() and flow_null_differs(),
but it is stacked above flow_callback_depth_differs() and its own comment,
well before flow_null_differs(). Could it move next to the function it
describes?
[ ... ]
> +static bool flow_unset_differs(const struct bpf_verifier_state *a,
> + const struct bpf_verifier_state *b)
> +{
> + u32 fr, i;
> +
> + for (fr = 0; fr <= a->curframe; fr++)
> + for (i = BPF_REG_0; i < BPF_REG_FP; i++)
> + if ((a->frame[fr]->regs[i].type == NOT_INIT) !=
> + (b->frame[fr]->regs[i].type == NOT_INIT))
> + return true;
> + return false;
> +}
Can flow_unset_differs() dereference a NULL frame here?
It walks frames 0 to a->curframe and reads b->frame[fr], and
flow_join_makes_up() does the same through flow_number_at(b->frame[fr], i).
Neither checks that a->curframe == b->curframe. In flow_merge() the only
filter in front of them that relates to the call chain is:
if (t->kind != s->kind)
continue;
kind is a 32-bit jhash from flow_kind() with a fixed seed, so equal kinds do
not prove equal curframe. flow_within() does compare curframe, but when it
fails, flow_merge() falls through to the next check instead of skipping the
state.
Frames above curframe are NULL: prepare_func_exit() sets
state->frame[state->curframe--] = NULL, and bpf_copy_verifier_state() frees
and NULLs frames above src->curframe. So if a state t in the set has more
frames than the arriving s, and their kinds collide,
flow_unset_differs(&t->st, &s->st) reads a NULL b->frame[fr]. This needs
!widen and flow_has_room(), and frame[0]->subprogno has to match, which
happens when one function is reached through call chains of different depth.
The other helpers that flow_merge() calls in the same place
(flow_null_differs, flow_needed_consts_differ, flow_callback_depth_differs,
flow_earlier_trip, flow_same_iters, flow_within) all start with an explicit
curframe check, and bpf_states_join() checks it in join_check_state().
Should flow_unset_differs() and flow_join_makes_up() return false when
a->curframe != b->curframe, as the sibling helpers do? flow.c is unchanged
after this patch in the series, so no later fix was found.
[ ... ]
> + env->cur_state = &s->st;
> + env->insn_idx = insn_idx;
> + err = bpf_flow_loop_enter(env, &backedge, &drop);
> + env->cur_state = cur;
> + env->insn_idx = cur_idx;
> + if (err || drop)
> + goto err_free;
> +
> + err = flow_merge(env, &flow->states[insn_idx], &s, backedge ? FLOW_CYCLE : cycle);
> + if (err)
> + return err < 0 ? err : 0;
> + flow_set_pending(flow, s);
> + return 0;
This isn't a bug, but the commit message argues soundness with a universal
claim:
The verifier drops a state only when the set has a state that
covers it. Thus a verified state covers each run of the prog.
The comment at the top of flow.c says the same, that an arriving state is
"dropped, if it is within a state of the set".
Here there is a second way to drop a state. When bpf_flow_loop_enter() sets
*drop, the arriving state is freed through err_free and bpf_flow_deliver()
returns 0, and no state in any set covers it. At the end of the series
(51224d16c11b) this happens on a back edge of a counted loop whose
iterations are walked apart, once ++entry->trip reaches
entry->max_header_count ("there is no such iteration at run time").
Soundness on that path does not come from coverage. It depends on scalar
evolution giving a correct upper bound on the number of header visits.
Could the commit message and the comment at the top of flow.c name this
exception, and the assumption it relies on, so that the soundness argument
is complete?
[ ... ]
> +/*
> + * The call at @call_idx in @caller of a function that is verified on its own,
> + * @entry is the state at its first insn. Both are taken.
> + */
> +int bpf_flow_call(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
> + struct bpf_verifier_state *entry, int subprog, int call_idx)
> +{
> + struct bpf_flow *flow = env->flow;
> + struct flow_func *f = &flow->funcs[subprog];
> + struct flow_waiter *w, *o, *on;
> + int id;
> +
> + caller->insn_idx = call_idx;
> + list_for_each_entry_safe(o, on, &f->waiters, node) {
> + if (o->call_idx != call_idx || o->dead)
> + continue;
> + /* it gets what the call that is there gets */
> + if (o->st.flow_entry == caller->flow_entry && flow_within(env, &o->st, caller))
> + return 0;
> + }
This isn't a bug today, but the comment says both @caller and @entry are
taken, and that does not hold on every path. When an existing waiter at the
same call_idx covers the caller, the function returns 0 before
w->st = *caller;
memset(caller, 0, sizeof(*caller));
and before bpf_flow_deliver() takes @entry. The flow_entry_new() and
kzalloc_obj() error paths leave both states untouched too.
The only caller, call_func_on_its_own(), frees 'entry' with
bpf_free_verifier_state(&entry, false) in every case, and process_bpf_exit
frees env->cur_state, so nothing leaks now. A caller that trusted the
comment and skipped the free would leak the callee frame on the covered
path.
Could the comment say that the states are taken only when the call has to
wait, or could bpf_flow_call() empty both states on every path?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 53/81] bpf: Verify programs as a fixpoint over the control flow graph
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (51 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 52/81] bpf: Add the main pass as a fixpoint over the control flow graph Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 4:25 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 54/81] bpf: Allow 24 arguments of static subprogs Alexei Starovoitov
` (29 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier follows each path of a prog from the first insn to exit. It
stops a path when a state that it saw before at the same insn covers the
current state. Each branch can double the number of paths. Thus the
verifier has limits of 1M insns and 8K jumps, and large progs hit them.
Do not follow paths. Where two paths meet, make one state that covers the
two states. This operation is a join. What is safe for the join is safe for
each of the two states.
follow each path join
[if] [if]
/ \ / \
r1=1 r1=5 r1=1 r1=5
\ / \ /
[next] [next]
verified 2 times: verified 1 time:
r1=1, then r1=5 r1=[1,5]
The work depends on the size of the prog, not on the number of paths.
Remove the two limits. flow.c limits the number of states, in proportion to
the size of the prog.
Some states have no join, for example a pointer and a scalar in the same
register. flow.c also refuses a join that loses a fact that progs depend
on. Thus each basic block has a set of states, not one state.
a state arrives at a basic block the verifier
-------------------------------------- ---------------------------------
a state of the set covers it drops it
it has a join with a state of the set replaces that state with the join
all other cases adds it to the set
The verifier verifies a block with each new state of its set. It stops when
no set has a new state. This is the fixpoint.
flow.c and join.c contain the algorithm. Build them and call them from
do_check(). The check of one insn does not change.
bpf_flow_loop_enter() makes the state that enters a loop cover all
iterations. Then one pass through the body is usually sufficient. If a
state comes back to the loop head and is not covered, the verifier joins
the two states and removes the bound that grows. Thus the pass ends.
Such a state shows that each iteration is safe, but not that the loop ends.
Thus the verifier adds may_goto to a loop that it cannot bound. may_goto
exits a loop when a run of the prog takes too long.
loop state at the loop head added
------------------------------ ----------------------------- --------
scalar evolution counts the each induction variable has nothing
iterations its range over all iterations
the verifier can unroll it one state for each iteration nothing
to its end
each iteration has may_goto the scalars that the loop nothing
or bpf_iter_*_next() changes are unknown
all other loops the same may_goto
no exit rejected: loop at insn N never exits
Recursion has the same problem. A subprog on a cycle of the call graph gets
a check at its entry. The check limits the stack and the number of calls in
a run. A callback that passes its own address to a helper is on a cycle
too. mark_recursive_subprogs() has the cases that stay rejected, for
example recursion of the main prog.
Before this patch the verifier rejected recursion, and loops that it could
neither count nor follow to the end. They load now.
The jump of an added check is a new edge of the prog. The verifier verifies
it like each other edge. It selects the target at load time:
the frame holds only the references and locks that it held at its entry
main prog, async callback r0 = 0; exit
subprog, unwind_goto follows each call of it r0 = 0; unwind_exit
all other cases
loop exit from the loop
recursive subprog r0 = 0; exit
unwind_exit continues in the cleanup code of the callers.
The verifier verifies a static subprog for each state at its entry, not for
each chain of calls. The state of the caller waits for the exit states.
This is correct only if the subprog cannot access the frames of its
callers. func_is_on_its_own() has the exceptions, for example a pointer to
the stack. Then the frames of the callers stay in the state, as before.
Most progs that loaded before get the same code. The usual causes of
different code are added may_goto, added recursion checks, and a jump that
a join leaves open. Each path decides such a jump, the joined state does
not, and dead code stays in the prog.
The cost of added may_goto on x86-64 (Zen 5) with hot caches, in nsec:
each iteration, the body only counts 0.29 -> 0.45
each iteration, the body does a hash lookup +0.15 (+6%)
each run of the prog +0.43
each call of a subprog that has may_goto +0.28 .. +0.34
A join loses the relation between scalars. The verifier now rejects this
prog:
if (c) { i = 3; n = 4; } else { i = 7; n = 8; }
arr[n - i]
{ i=3 n=4 } { i=7 n=8 } n - i is 1 in each state
\ /
{ i=[3,7] n=[4,8] } n - i is [-3, 5]
Other changes:
- may_goto at a loop head skips the body. The code after the loop can read
registers and stack slots that only the body sets. The verifier sets
them to 0 on this edge.
- Such a register can be a pointer to arena. Thus, in a prog that has an
arena, a load or a store through the scalar 0 is an access to arena.
This is safe, because an access to arena cannot leave the arena.
- If the pass finds that an iterator does not stop its loop, the pass
starts again, and the loop gets may_goto.
- Pruning, jmp_history and precision marks have no function now. This
patch only disables them. "bpf: Remove is_state_visited()" and
subsequent patches remove the code.
- The verifier log shows the states of basic blocks, not paths.
A: the previous patch. B: this patch. Progs load in the two runs. Insns is
the number of insns that the verifier processes.
Corpus Progs Insns (A) Insns (B) Insns (DIFF)
----------------- ----- --------- --------- ------------------
selftests 3714 6843490 2676019 -4167471 (-60.90%)
cilium 144 1151493 282948 -868545 (-75.43%)
scx, as in BPF CI 394 1296250 253946 -1042304 (-80.41%)
scx, master 468 3415944 475390 -2940554 (-86.08%)
tools/sched_ext 240 118790 37454 -81336 (-68.47%)
No prog of Cilium or sched-ext/scx changes its verdict.
One prog with 5,000 insns or more regresses. mlfq_dispatch of scx_mlfq has
nested loops that are not counted: 12902 -> 15673 (+21.48%).
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 71 +-
kernel/bpf/Makefile | 2 +-
kernel/bpf/backtrack.c | 7 +
kernel/bpf/const_fold.c | 6 +
kernel/bpf/log.c | 5 +
kernel/bpf/states.c | 19 +-
kernel/bpf/verifier.c | 1400 +++++++++++++++++++++++++++++-----
7 files changed, 1308 insertions(+), 202 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 16987fa282b3..41ffe9b694a6 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -353,8 +353,26 @@ struct loop_stack_entry {
struct bpf_verifier_state *entry_state;
u32 min_header_count;
u32 max_header_count;
- u32 loop_id:31;
+ u32 loop_id:27;
u32 terminates:1;
+ /*
+ * The iterations are kept apart: states with different @trip are not
+ * compared and not joined for as long as they are in the loop.
+ */
+ u32 apart:1;
+ /* max_header_count follows from the state at the entry, not from patience */
+ u32 counted:1;
+ /* the walk goes on for as long as the jumps that leave the loop are decided */
+ u32 probe:1;
+ /* one of them went both ways */
+ u32 undecided:1;
+ u32 trip;
+ /*
+ * A walk on patience: env->insn_processed at its start and how much it may grow.
+ * The size of the loop says nothing about the functions that it calls.
+ */
+ u32 walk_start;
+ u32 walk_patience;
int iv_reg;
struct cnum64 iv_range;
};
@@ -416,6 +434,10 @@ struct bpf_func_state {
* subprogram (loops never cross subprogram boundaries).
*/
u32 loop_stack_cnt;
+ /* What was held when the frame was entered. More than that is for this frame to release. */
+ u32 entry_refs;
+ u32 entry_rcu_locks;
+ u32 entry_preempt_locks;
/* The following fields should be last. See copy_func_state() */
/* The state of the stack. Each element of the array describes BPF_REG_SIZE
@@ -546,8 +568,17 @@ struct bpf_verifier_state {
bool speculative;
bool in_sleepable;
+ /* a function that is verified on its own and is called from a callback */
+ bool in_callback;
+ /* a global function or an extension, and what it calls: exit of frame 0 is not the end */
+ bool callers_unknown;
/* The callee left with unwind_exit. Next insn is unwind_goto. */
bool unwound;
+ /*
+ * Not zero in a function that is verified on its own: the call that the
+ * state at its entry came from, see bpf_flow_call().
+ */
+ u32 flow_entry;
/*
* First and last insn idx of this verifier state.
@@ -1022,14 +1053,21 @@ struct bpf_subprog_info {
* a hidden pointer to the result, a pair in two registers, no dead arguments.
*/
bool stack_args_from_code: 1;
+ /* it releases what its caller holds, so it's verified in the context of such caller */
+ bool releases_inherited: 1;
+ /* a call of it was verified on its own: its insns are not in insns_total of the caller */
+ bool on_its_own: 1;
/* it can leave with unwind_exit: it has one, or unwind_goto follows every call of it */
bool bail_unwinds: 1;
+ /* some call of it is not followed by unwind_goto */
+ bool plain_call: 1;
u8 code_stack_args:4;
u8 arg_slot_cnt:4;
enum priv_stack_mode priv_stack_mode;
struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
u32 recursion_weight; /* see bpf_insert_recursion_checks() */
+ int bail_insn; /* an exit of it, for the main pass. -1 if there is none */
u16 stack_arg_cnt; /* incoming + max outgoing */
u16 max_out_stack_arg_cnt;
};
@@ -1131,6 +1169,7 @@ struct bpf_verifier_env {
bool test_state_freq; /* test verifier with different pruning frequency */
bool test_reg_invariants; /* fail verification on register invariants violations */
bool uses_arena_sp; /* the program has insns with BPF_REG_ARENA_SP */
+ bool restart; /* do the main pass again with what was found out */
u32 max_arena_stack_depth; /* of all call chains */
struct bpf_verifier_state *cur_state; /* current verifier state */
/* Search pruning optimization, array of list_heads for
@@ -1275,6 +1314,7 @@ struct bpf_verifier_env {
struct bpf_iarray *gotox_tmp_buf;
int *idoms;
struct scev *scev;
+ struct bpf_flow *flow;
struct bpf_join_state *join;
};
@@ -1459,15 +1499,44 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
int mark_chain_precision(struct bpf_verifier_env *env, int regno);
int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
+int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
+bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
+ struct bpf_verifier_state *cur);
+
/* join.c */
int bpf_states_join(struct bpf_verifier_env *env, struct bpf_verifier_state *dst,
const struct bpf_verifier_state *a, const struct bpf_verifier_state *b,
bool widen);
void bpf_join_free(struct bpf_verifier_env *env);
+/* flow.c */
+int bpf_flow_prepare(struct bpf_verifier_env *env);
+void bpf_flow_reset(struct bpf_verifier_env *env);
+void bpf_flow_free(struct bpf_verifier_env *env);
+bool bpf_flow_is_point(struct bpf_verifier_env *env, int insn_idx);
+struct bpf_verifier_state *bpf_flow_keep_state(struct bpf_verifier_env *env,
+ const struct bpf_verifier_state *st);
+int bpf_flow_deliver(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ int insn_idx, int prev_insn_idx);
+int bpf_flow_park(struct bpf_verifier_env *env, struct bpf_verifier_state *st,
+ int insn_idx, int prev_insn_idx);
+void bpf_flow_unpark(struct bpf_verifier_env *env, int prev_insn_idx);
+int bpf_flow_deliver_unparked(struct bpf_verifier_env *env);
+int bpf_flow_next(struct bpf_verifier_env *env, int *prev_insn_idx);
+int bpf_flow_call(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+ struct bpf_verifier_state *entry, int subprog, int call_idx);
+int bpf_flow_exit(struct bpf_verifier_env *env, struct bpf_verifier_state *st, int exit_idx);
+/* verifier.c, for flow.c */
+int bpf_flow_loop_leave(struct bpf_verifier_env *env, bool *enters);
+int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop);
+int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+ const struct bpf_verifier_state *exit, int call_idx);
int bpf_insert_loop_guards(struct bpf_verifier_env *env);
int bpf_insert_recursion_reports(struct bpf_verifier_env *env);
int bpf_insert_unwound_reports(struct bpf_verifier_env *env);
+u16 bpf_kfunc_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
+u16 bpf_helper_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn);
+bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn);
bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n);
bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 269810b128c4..d5a79e1ea933 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o flow.o join.o
obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 47452ac7277a..98539c7e42c5 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -16,6 +16,9 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
struct bpf_jmp_history_entry *p;
size_t alloc_size;
+ /* FLOW: nothing looks at the history */
+ return 0;
+
/* combine instruction flags if we already recorded this instruction */
if (env->cur_hist_ent) {
/* atomic instructions push insn_flags twice, for READ and
@@ -850,6 +853,10 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
struct bpf_func_state *func;
bool tmp, skip_first = true;
struct bpf_stack_state *ss;
+
+ /* FLOW: there are no precision marks */
+ bt_reset(bt);
+ return 0;
struct bpf_reg_state *reg;
int i, fr, err;
diff --git a/kernel/bpf/const_fold.c b/kernel/bpf/const_fold.c
index bd251c8e82ee..a2dbf03601ae 100644
--- a/kernel/bpf/const_fold.c
+++ b/kernel/bpf/const_fold.c
@@ -263,6 +263,12 @@ int bpf_compute_const_regs(struct bpf_verifier_env *env)
for (r = 0; r < MAX_BPF_REG; r++)
ci_in[start][r] = unknown;
+ /*
+ * The subprog gets there from where it is out of budget too, see
+ * push_budget_exit(). That is not an edge of the graph.
+ */
+ for (r = 0; r < MAX_BPF_REG && env->subprog_info[i].bail_insn >= 0; r++)
+ ci_in[env->subprog_info[i].bail_insn][r] = unknown;
}
redo:
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index 5432c0fc442b..82edff722dea 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -792,6 +792,11 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
verbose(env, " cb");
if (state->in_async_callback_fn)
verbose(env, " async_cb");
+ /* the iterations of the loops in which they are kept apart */
+ for (i = 0; i < state->loop_stack_cnt; i++)
+ if (state->loop_stack[i].apart)
+ verbose(env, " iter(%d)=%u", state->loop_stack[i].loop_id,
+ state->loop_stack[i].trip);
if (state->loop_stack_cnt) {
verbose(env, " loop_stack=");
for (i = 0; i < state->loop_stack_cnt; i++)
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index d7df6632a5c4..d55adade5b1c 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -949,7 +949,9 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
for (i = 0; i < old->loop_stack_cnt; i++) {
if (old_stack[i].loop_id == cur_stack[i].loop_id &&
old_stack[i].max_header_count == cur_stack[i].max_header_count &&
- old_stack[i].terminates == cur_stack[i].terminates)
+ old_stack[i].terminates == cur_stack[i].terminates &&
+ old_stack[i].apart == cur_stack[i].apart &&
+ old_stack[i].trip == cur_stack[i].trip)
continue;
return false;
}
@@ -1043,6 +1045,9 @@ static bool states_equal(struct bpf_verifier_env *env,
if (old->speculative && !cur->speculative)
return false;
+ if (old->in_callback != cur->in_callback || old->callers_unknown != cur->callers_unknown)
+ return false;
+
if (old->in_sleepable != cur->in_sleepable)
return false;
@@ -1731,3 +1736,15 @@ int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
*/
return bpf_split_cur_state(env);
}
+
+int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
+{
+ return clean_verifier_state(env, st);
+}
+
+/* Everything that @cur describes is described by @old. */
+bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
+ struct bpf_verifier_state *cur)
+{
+ return states_equal(env, old, cur, RANGE_WITHIN);
+}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index a9628f5045fe..0c1de0f24b85 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -194,13 +194,14 @@ struct bpf_verifier_stack_elem {
struct bpf_verifier_state st;
int insn_idx;
int prev_insn_idx;
+ /* it goes there only if the guard at prev_insn_idx has to, see check_loop_guard() */
+ bool parked;
struct bpf_verifier_stack_elem *next;
/* length of verifier log at the time this state was pushed on stack */
u32 log_pos;
u64 diag_log_pos;
};
-#define BPF_COMPLEXITY_LIMIT_JMP_SEQ 8192
#define BPF_COMPLEXITY_LIMIT_STATES 64
#define BPF_GLOBAL_PERCPU_MA_MAX_SIZE 512
@@ -1781,7 +1782,11 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
return err;
dst_state->speculative = src->speculative;
dst_state->in_sleepable = src->in_sleepable;
+ dst_state->in_callback = src->in_callback;
+ dst_state->callers_unknown = src->callers_unknown;
dst_state->unwound = src->unwound;
+ dst_state->flow_entry = src->flow_entry;
+ dst_state->insn_idx = src->insn_idx;
dst_state->curframe = src->curframe;
dst_state->branches = src->branches;
dst_state->parent = src->parent;
@@ -1910,11 +1915,6 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
if (err)
return ERR_PTR(-ENOMEM);
elem->st.speculative |= speculative;
- if (env->stack_size > BPF_COMPLEXITY_LIMIT_JMP_SEQ) {
- verbose(env, "The sequence of %d jumps is too complex.\n",
- env->stack_size);
- return ERR_PTR(-E2BIG);
- }
if (elem->st.parent) {
++elem->st.parent->branches;
/* WARN_ON(branches > 2) technically makes sense here,
@@ -2593,12 +2593,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
elem->diag_log_pos = bpf_diag_event_log_save(env);
env->head = elem;
env->stack_size++;
- if (env->stack_size > BPF_COMPLEXITY_LIMIT_JMP_SEQ) {
- verbose(env,
- "The sequence of %d jumps is too complex for async cb.\n",
- env->stack_size);
- return ERR_PTR(-E2BIG);
- }
/* Unlike push_stack() do not bpf_copy_verifier_state().
* The caller state doesn't matter.
* This is async callback. It starts in a fresh stack.
@@ -3454,7 +3448,17 @@ static int check_unwind_insns(struct bpf_verifier_env *env)
bool seen = false;
int i;
+ for (i = 0; i < env->subprog_cnt; i++)
+ env->subprog_info[i].bail_insn = -1;
+
for (i = 0; i < env->prog->len; i++, insn++) {
+ if (bpf_is_ldimm64(insn)) {
+ i++;
+ insn++;
+ continue;
+ }
+ if (insn->code == (BPF_JMP | BPF_EXIT))
+ bpf_find_containing_subprog(env, i)->bail_insn = i;
if (bpf_is_unwind_exit(insn)) {
bpf_find_containing_subprog(env, i)->can_unwind = true;
seen = true;
@@ -3552,20 +3556,19 @@ static int sort_subprogs_topo(struct bpf_verifier_env *env)
}
if (color[callee] == 2)
continue;
- if (color[callee] == 1) {
- if (bpf_pseudo_func(&insn[idx]))
- continue;
- verbose(env, "recursive call from %s() to %s()\n",
- bpf_subprog_name(env, cur),
- bpf_subprog_name(env, callee));
- bpf_diag_program_structure(
- env, idx, "recursive subprogram call",
- "Rewrite the recursion as an explicit bounded loop, or split the logic so subprogram calls do not form a cycle.",
- "This bpf2bpf call would make the subprogram call graph recursive. "
- "The verifier requires a finite, acyclic call graph so it can bound stack depth and analysis.");
+ /*
+ * The call that is its own target never returns. It is
+ * also what libbpf leaves in place of a call that it
+ * could not resolve.
+ */
+ if (bpf_pseudo_call(&insn[idx]) && insn[idx].imm == -1) {
+ verbose(env, "call at insn %d calls itself\n", idx);
ret = -EINVAL;
goto out;
}
+ /* recursion: the order within the cycle is arbitrary */
+ if (color[callee] == 1)
+ continue;
color[callee] = 1;
dfs_stack[top++] = callee;
pushed = true;
@@ -4598,6 +4601,23 @@ static int check_stack_write(struct bpf_verifier_env *env,
return err;
}
+/*
+ * The function that is verified on its own has no caller to look at. Its incoming
+ * arguments are in its stack_arg_regs until it is done with loading them.
+ */
+static void end_stack_arg_loads(struct bpf_verifier_env *env, struct bpf_func_state *state)
+{
+ int i;
+
+ if (state->no_stack_arg_load)
+ return;
+ state->no_stack_arg_load = true;
+ if (state->frameno)
+ return;
+ for (i = 0; i < state->out_stack_arg_cnt; i++)
+ bpf_mark_reg_not_init(env, &state->stack_arg_regs[i]);
+}
+
/*
* Write a value to the outgoing stack arg area.
* off is a negative offset from r11 (e.g. -8 for arg6, -16 for arg7).
@@ -4617,6 +4637,7 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
return -EINVAL;
}
+ end_stack_arg_loads(env, state);
err = grow_stack_arg_slots(env, state, spi + 1);
if (err)
return err;
@@ -4636,7 +4657,6 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
__mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm);
}
bpf_diag_mod_end(env);
- state->no_stack_arg_load = true;
return bpf_push_jmp_history(env, env->cur_state,
INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
}
@@ -4665,7 +4685,9 @@ static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_st
return -EACCES;
}
- caller = vstate->frame[vstate->curframe - 1];
+ caller = vstate->frame[vstate->curframe ? vstate->curframe - 1 : 0];
+ if (verifier_bug_if(spi >= caller->out_stack_arg_cnt, env, "no stack arg%d", spi + 1))
+ return -EFAULT;
arg = &caller->stack_arg_regs[spi];
cur = vstate->frame[vstate->curframe];
bpf_diag_mod_begin(env, &cur->regs[dst_regno], arg, BPF_DIAG_MOD_WRITE);
@@ -5746,6 +5768,9 @@ struct bpf_subprog_call_depth_info {
int callx_insn; /* callx instruction whose callees are being walked */
int callx_next; /* next callee of callx_insn to walk */
bool via_callx; /* the subprogram is entered via callx */
+ /* recursive subprog: the deepest it was entered, out of a callback and in one, plus 1 */
+ int seen_depth[2];
+ int seen_arena_depth[2];
};
/* starting from main bpf function walk all instructions of the function
@@ -5810,7 +5835,9 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
break;
}
}
- arena_depth += subprog[idx].arena_stack_depth;
+ /* frames of recursive subprogs are counted at run time */
+ if (!subprog[idx].recursive)
+ arena_depth += subprog[idx].arena_stack_depth;
if (arena_depth > env->max_arena_stack_depth)
env->max_arena_stack_depth = arena_depth;
if (arena_depth > BPF_ARENA_STACK_MAX) {
@@ -5849,7 +5876,8 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
return -EACCES;
}
} else {
- depth += subprog_depth;
+ if (!subprog[idx].recursive)
+ depth += subprog_depth;
if (depth > env->max_stack_depth)
env->max_stack_depth = depth;
if (depth > env->stack_limit) {
@@ -5930,17 +5958,37 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
* The main verification pass limits the depth of the recursion,
* but it doesn't follow calls of global functions.
*/
- for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
- if (tmp != sidx)
+ for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller)
+ if (tmp == sidx)
+ break;
+ /*
+ * Recursion. How deep it goes is not known here. It's counted
+ * at run time.
+ */
+ if (tmp >= 0) {
+ if (bpf_is_callx(insn + i) || bpf_calls_callback(env, i))
+ i--;
+ continue;
+ }
+ /*
+ * Any recursive subprog may follow any other when they call each other
+ * through pointers. Don't walk all such chains: what is called from
+ * here was looked at with at least as much on the stack.
+ */
+ if (subprog[sidx].recursive) {
+ bool cb = subprog[sidx].is_cb;
+
+ for (tmp = idx; tmp >= 0 && !cb; tmp = dinfo[tmp].caller)
+ cb = subprog[tmp].is_cb;
+ if (depth < dinfo[sidx].seen_depth[cb] &&
+ arena_depth < dinfo[sidx].seen_arena_depth[cb]) {
+ if (bpf_is_callx(insn + i) || bpf_calls_callback(env, i))
+ i--;
continue;
- verbose(env, "recursive call from %s() to %s()\n",
- bpf_subprog_name(env, idx), bpf_subprog_name(env, sidx));
- bpf_diag_program_structure(
- env, i, "recursive subprogram call",
- "Rewrite the recursion as an explicit bounded loop, or split the logic so subprogram calls do not form a cycle.",
- "This call or callback would make the subprogram call graph recursive. "
- "The verifier requires a finite, acyclic call graph so it can bound stack depth and analysis.");
- return -EINVAL;
+ }
+ dinfo[sidx].seen_depth[cb] = max(dinfo[sidx].seen_depth[cb], depth + 1);
+ dinfo[sidx].seen_arena_depth[cb] = max(dinfo[sidx].seen_arena_depth[cb],
+ arena_depth + 1);
}
/* store caller info for after we return from callee */
@@ -5959,7 +6007,10 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
/* sync tail_call_reachable with callee state on entry */
tail_call_reachable = subprog[idx].has_tail_call;
- frame = bpf_subprog_is_global(env, idx) ? 0 : frame + 1;
+ if (bpf_subprog_is_global(env, idx))
+ frame = 0;
+ else if (!subprog[idx].recursive)
+ frame++;
if (frame >= MAX_CALL_FRAMES) {
verbose(env, "the call stack of %d frames is too deep !\n",
frame);
@@ -6003,11 +6054,12 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
/* end of for() loop means the last insn of the 'subprog'
* was reached. Doesn't matter whether it was JA or EXIT
*/
- if (frame == 0 && dinfo[idx].caller < 0)
+ if (dinfo[idx].caller < 0)
return 0;
- if (subprog[idx].priv_stack_mode != PRIV_STACK_ADAPTIVE)
+ if (subprog[idx].priv_stack_mode != PRIV_STACK_ADAPTIVE && !subprog[idx].recursive)
depth -= round_up_stack_depth(env, subprog[idx].stack_depth);
- arena_depth -= subprog[idx].arena_stack_depth;
+ if (!subprog[idx].recursive)
+ arena_depth -= subprog[idx].arena_stack_depth;
/* pop caller idx from callee */
idx = dinfo[idx].caller;
@@ -6022,6 +6074,111 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
goto continue_func;
}
+/*
+ * Find the subprogs on cycles of the call graph. It is not known yet what callx
+ * calls: it may be any subprog whose address is taken. A subprog that is
+ * marked for no reason counts its frames for no reason, that is all.
+ */
+static int mark_recursive_subprogs(struct bpf_verifier_env *env)
+{
+ struct bpf_subprog_info *si = env->subprog_info;
+ struct bpf_insn *insn = env->prog->insnsi;
+ int cnt = env->subprog_cnt, words = BITS_TO_LONGS(cnt);
+ unsigned long *calls, *taken, *callx;
+ bool plain_callx = false;
+ int i, j, s, err = 0;
+
+ /* calls + s * words: the subprogs that s calls */
+ calls = kvcalloc((cnt + 2) * words, sizeof(long), GFP_KERNEL_ACCOUNT);
+ if (!calls)
+ return -ENOMEM;
+ taken = calls + cnt * words;
+ callx = taken + words;
+
+ for (s = 0; s < cnt; s++) {
+ for (i = si[s].start; i < si[s + 1].start; i++) {
+ if (bpf_is_callx(&insn[i])) {
+ __set_bit(s, callx);
+ if (!bpf_is_unwind_goto(&insn[i + 1]))
+ plain_callx = true;
+ }
+ if (bpf_pseudo_call(&insn[i]) || bpf_pseudo_func(&insn[i])) {
+ j = bpf_find_subprog(env, i + insn[i].imm + 1);
+ if (j < 0)
+ continue;
+ /* the helper that calls a callback doesn't expect it to unwind */
+ if (bpf_pseudo_func(&insn[i]) || !bpf_is_unwind_goto(&insn[i + 1]))
+ si[j].plain_call = true;
+ /* the helper that takes a callback calls it */
+ __set_bit(j, calls + s * words);
+ if (bpf_pseudo_func(&insn[i]))
+ __set_bit(j, taken);
+ }
+ if (bpf_is_ldimm64(&insn[i]))
+ i++;
+ }
+ }
+ for (i = 0; i < env->func_ptr_cnt; i++) {
+ j = bpf_find_subprog(env, env->func_ptrs[i].xlated_off);
+ if (j >= 0)
+ __set_bit(j, taken);
+ }
+ for_each_set_bit(s, callx, cnt)
+ bitmap_or(calls + s * words, calls + s * words, taken, cnt);
+
+ /*
+ * Callers of a global subprog look at can_unwind before the subprog is verified.
+ * The main prog returns to the kernel.
+ */
+ for (s = 0; s < cnt; s++) {
+ if (plain_callx && test_bit(s, taken))
+ si[s].plain_call = true;
+ si[s].bail_unwinds = si[s].can_unwind ||
+ (s && !si[s].plain_call && !bpf_subprog_is_global(env, s) &&
+ bpf_jit_supports_unwind() &&
+ !bpf_prog_is_offloaded(env->prog->aux));
+ }
+
+ for (j = 0; j < cnt; j++)
+ for (s = 0; s < cnt; s++)
+ if (test_bit(j, calls + s * words))
+ bitmap_or(calls + s * words, calls + s * words,
+ calls + j * words, cnt);
+
+ for (s = 0; s < cnt; s++) {
+ if (!test_bit(s, calls + s * words))
+ continue;
+ err = -EINVAL;
+ if (!s) {
+ verbose(env, "recursive call of the main program\n");
+ break;
+ }
+ /* it has no way to tell one run from another, see bpf_prog_begin_run() */
+ if (env->prog->type == BPF_PROG_TYPE_EXT) {
+ verbose(env, "recursive %s() in an extension\n", bpf_subprog_name(env, s));
+ break;
+ }
+ if (si[s].has_tail_call || si[s].has_ld_abs) {
+ verbose(env, "recursive %s() cannot have %s\n", bpf_subprog_name(env, s),
+ si[s].has_tail_call ? "tail calls" : "LD_ABS");
+ break;
+ }
+ if (!bpf_jit_supports_percpu_insn()) {
+ verbose(env, "JIT does not support recursive calls\n");
+ err = -EOPNOTSUPP;
+ break;
+ }
+ if (si[s].bail_insn < 0) {
+ verbose(env, "recursive %s() has no exit\n", bpf_subprog_name(env, s));
+ break;
+ }
+ si[s].recursive = true;
+ err = 0;
+ }
+ kvfree(calls);
+ return err;
+}
+
static int check_max_stack_depth(struct bpf_verifier_env *env)
{
enum priv_stack_mode priv_stack_mode = PRIV_STACK_UNKNOWN;
@@ -6035,14 +6192,15 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
return -ENOMEM;
for (int i = 0; i < env->subprog_cnt; i++) {
- if (si[i].has_tail_call) {
- /* the stack in arena wouldn't be returned */
- if (env->uses_arena_sp) {
- verbose(env, "tail_calls are not allowed in programs that use R%d\n",
- BPF_REG_ARENA_SP);
- kvfree(dinfo);
- return -EINVAL;
- }
+ /* the stack in arena wouldn't be returned */
+ if (si[i].has_tail_call && env->uses_arena_sp) {
+ verbose(env, "tail_calls are not allowed in programs that use R%d\n",
+ BPF_REG_ARENA_SP);
+ kvfree(dinfo);
+ return -EINVAL;
+ }
+ /* the private stack has room for one frame of a subprog */
+ if (si[i].has_tail_call || si[i].recursive) {
priv_stack_mode = NO_PRIV_STACK;
break;
}
@@ -8962,12 +9120,33 @@ static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_
* }
* bpf_iter_num_destroy(&it);
*/
+/*
+ * The program asks for the next element after it was told that there is none. So it didn't
+ * leave the loops around the call, and the iterator is not what stops them.
+ */
+static int iter_is_drained(struct bpf_verifier_env *env, int insn_idx)
+{
+ struct bpf_loop *loop;
+ int h;
+
+ for (h = bpf_loop_at_index(env, insn_idx); h >= 0; h = env->insn_aux_data[h].loop_header) {
+ loop = env->insn_aux_data[h].loop;
+ if (!loop->rt_bounded)
+ continue;
+ loop->rt_bounded = false;
+ env->restart = true;
+ }
+ return env->restart ? -EINVAL : 0;
+}
+
static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
struct bpf_call_arg_meta *meta)
{
- struct bpf_verifier_state *cur_st = env->cur_state, *queued_st, *prev_st;
+ struct bpf_verifier_state *cur_st = env->cur_state, *queued_st;
+ struct bpf_func_state *fr;
struct bpf_func_state *cur_fr = cur_st->frame[cur_st->curframe], *queued_fr;
struct bpf_reg_state *cur_iter, *queued_iter;
+ int err;
BTF_TYPE_EMIT(struct bpf_iter);
@@ -8980,21 +9159,18 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
return -EFAULT;
}
+ if (cur_iter->iter.state == BPF_ITER_STATE_DRAINED) {
+ err = iter_is_drained(env, insn_idx);
+ if (err)
+ return err;
+ }
+
if (cur_iter->iter.state == BPF_ITER_STATE_ACTIVE) {
- /* Because iter_next() call is a checkpoint is_state_visitied()
- * should guarantee parent state with same call sites and insn_idx.
- */
- if (!cur_st->parent || cur_st->parent->insn_idx != insn_idx ||
- !same_callsites(cur_st->parent, cur_st)) {
- verifier_bug(env, "bad parent state for iter next call");
- return -EFAULT;
- }
- /* Note cur_st->parent in the call below, it is necessary to skip
- * checkpoint created for cur_st by is_state_visited()
- * right at this instruction.
- */
- prev_st = find_prev_entry(env, cur_st->parent, insn_idx);
/* branch out active iter state */
+ /* there is an element or there is none: a jump out of the loop, both ways */
+ fr = cur_st->frame[cur_st->curframe];
+ if (fr->loop_stack_cnt && fr->loop_stack[fr->loop_stack_cnt - 1].probe)
+ fr->loop_stack[fr->loop_stack_cnt - 1].undecided = true;
queued_st = push_stack(env, insn_idx + 1, insn_idx, false);
if (IS_ERR(queued_st))
return PTR_ERR(queued_st);
@@ -9002,8 +9178,6 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
queued_iter = get_iter_from_state(queued_st, meta);
queued_iter->iter.state = BPF_ITER_STATE_ACTIVE;
queued_iter->iter.depth++;
- if (prev_st)
- widen_imprecise_scalars(env, prev_st, queued_st);
queued_fr = queued_st->frame[queued_st->curframe];
mark_ptr_not_null_reg(&queued_fr->regs[BPF_REG_0]);
@@ -11311,6 +11485,9 @@ static int setup_func_entry(struct bpf_verifier_env *env, int subprog, int calls
callsite,
state->curframe + 1 /* frameno within this callchain */,
subprog /* subprog number within this prog */);
+ callee->entry_refs = state->acquired_refs;
+ callee->entry_rcu_locks = state->active_rcu_locks;
+ callee->entry_preempt_locks = state->active_preempt_locks;
err = set_callee_state_cb(env, caller, callee, callsite);
if (err)
goto err_out;
@@ -11365,7 +11542,7 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
const struct btf_type *func, *func_proto;
struct bpf_call_arg_meta meta;
struct bpf_func_proto *fn;
- int ret;
+ int ret, err;
u32 i;
memset(&meta, 0, sizeof(meta));
@@ -11374,13 +11551,22 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, s
meta.func_name = bpf_subprog_name(env, subprog);
ret = btf_prepare_func_args(env, subprog);
- if (ret)
- return ret;
- /* BTF doesn't describe what the code does */
- if (sub->stack_args_from_code &&
- max_t(int, sub->arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS) !=
- MAX_BPF_FUNC_REG_ARGS + bpf_in_stack_arg_cnt(sub))
- return -EINVAL;
+ if (ret || sub->stack_args_from_code) {
+ if (bpf_in_stack_arg_cnt(sub) > 0) {
+ err = check_outgoing_stack_args(env, caller, MAX_BPF_FUNC_REG_ARGS +
+ bpf_in_stack_arg_cnt(sub),
+ bpf_subprog_name(env, subprog),
+ NULL, NULL);
+ if (err)
+ return err;
+ }
+ if (ret)
+ return ret;
+ /* BTF doesn't describe what the code does */
+ if (max_t(int, sub->arg_slot_cnt, MAX_BPF_FUNC_REG_ARGS) !=
+ MAX_BPF_FUNC_REG_ARGS + bpf_in_stack_arg_cnt(sub))
+ return -EINVAL;
+ }
func = btf_type_by_id(btf, env->prog->aux->func_info[subprog].type_id);
func_proto = btf_type_by_id(btf, func->type);
@@ -11523,13 +11709,362 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
return 0;
}
+enum {
+ PROCESS_BPF_EXIT = 1,
+ INSN_IDX_UPDATED = 2,
+};
+
static int process_bpf_exit_full(struct bpf_verifier_env *env,
bool *do_print_state, bool exception_exit);
+static void account_current_path(struct bpf_verifier_env *env);
+
+/*
+ * The function can be verified on its own, for the state at its entry, when
+ * what it does depends on its arguments only and it cannot get to what its
+ * callers have: nothing points to their stack.
+ *
+ * What the callers hold, references, locks and so on, is a part of that state:
+ * it matters for what the function may do. Calls that differ in ids only
+ * have to get to the same state, so the ids of what is held are replaced with
+ * BPF_INHERITED_ID + the position for the function, and put back at the return.
+ * The same for other ids that the arguments have: BPF_INHERITED_ARG_ID + the position.
+ */
+#define BPF_INHERITED_ID 0x20000000
+#define BPF_INHERITED_ARG_ID (BPF_INHERITED_ID + 0x10000)
+#define BPF_MAX_ARG_IDS 32
+
+static u32 inherited_id(const struct bpf_verifier_state *state, u32 id)
+{
+ int i;
+
+ /* the first one: the same lock can't be taken twice, the rest have ids of their own */
+ for (i = 0; id && i < state->acquired_refs; i++)
+ if (state->refs[i].id == id)
+ return BPF_INHERITED_ID + i;
+ return id;
+}
+
+static bool state_in_callback(const struct bpf_verifier_state *st)
+{
+ u32 fr;
+
+ if (st->in_callback)
+ return true;
+ for (fr = 0; fr <= st->curframe; fr++)
+ if (st->frame[fr]->in_callback_fn || st->frame[fr]->in_async_callback_fn ||
+ st->frame[fr]->in_exception_callback_fn || st->frame[fr]->in_rcu_callback_fn)
+ return true;
+ return false;
+}
+
+static u32 add_arg_id(const struct bpf_verifier_state *state, u32 *ids, u32 n, u32 id)
+{
+ u32 i;
+
+ if (!id || n == BPF_MAX_ARG_IDS || inherited_id(state, id) != id)
+ return n;
+ for (i = 0; i < n; i++)
+ if (ids[i] == id)
+ return n;
+ ids[n] = id;
+ return n + 1;
+}
+
+/* The ids that the arguments of the call of @subprog in @state have, in the order of arguments */
+static u32 collect_arg_ids(struct bpf_verifier_env *env, const struct bpf_verifier_state *state,
+ int subprog, u32 *ids)
+{
+ struct bpf_subprog_info *sub = &env->subprog_info[subprog];
+ u16 live = env->insn_aux_data[sub->start].live_regs_before;
+ const struct bpf_func_state *caller = state->frame[state->curframe];
+ const struct bpf_reg_state *reg;
+ u32 i, n = 0;
+
+ for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+ if (!(live & BIT(i)))
+ continue;
+ reg = &caller->regs[i];
+ n = add_arg_id(state, ids, n, reg->id & ~BPF_ADD_CONST);
+ n = add_arg_id(state, ids, n, reg->parent_id);
+ }
+ for (i = 0; i < bpf_in_stack_arg_cnt(sub) && i < caller->out_stack_arg_cnt; i++) {
+ reg = &caller->stack_arg_regs[i];
+ n = add_arg_id(state, ids, n, reg->id & ~BPF_ADD_CONST);
+ n = add_arg_id(state, ids, n, reg->parent_id);
+ }
+ return n;
+}
+
+static u32 callee_id(const struct bpf_verifier_state *state, const u32 *ids, u32 n, u32 id)
+{
+ u32 flags = id & BPF_ADD_CONST, i;
+
+ id &= ~BPF_ADD_CONST;
+ if (inherited_id(state, id) != id)
+ return inherited_id(state, id) | flags;
+ for (i = 0; i < n; i++)
+ if (ids[i] == id)
+ return (BPF_INHERITED_ARG_ID + i) | flags;
+ return id | flags;
+}
+
+static bool func_is_on_its_own(struct bpf_verifier_env *env, struct bpf_verifier_state *state,
+ int subprog)
+{
+ struct bpf_func_state *caller = state->frame[state->curframe];
+ struct bpf_subprog_info *sub = &env->subprog_info[subprog];
+ u16 live = env->insn_aux_data[sub->start].live_regs_before;
+ u32 fr, i;
+
+ /*
+ * Iterations that are kept apart would meet in the function.
+ * There is no other way for a recursive one.
+ */
+ for (fr = 0; fr <= state->curframe && !sub->recursive; fr++)
+ for (i = 0; i < state->frame[fr]->loop_stack_cnt; i++)
+ if (state->frame[fr]->loop_stack[i].apart)
+ return false;
+
+ if (sub->changes_pkt_data || sub->has_tail_call || sub->has_ld_abs ||
+ sub->is_cb || sub->is_async_cb || sub->is_exception_cb)
+ return false;
+ if (state->speculative)
+ return false;
+ if (sub->releases_inherited &&
+ (state->acquired_refs || state->active_rcu_locks || state->active_preempt_locks))
+ return false;
+ /* there is no other way for a recursive function */
+ if (state_in_callback(state) && !sub->recursive)
+ return false;
+ for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+ if (!(live & BIT(i)))
+ continue;
+ if (caller->regs[i].type == PTR_TO_STACK ||
+ base_type(caller->regs[i].type) == CONST_PTR_TO_DYNPTR)
+ return false;
+ /* it checks how much of the packet is there, for the caller */
+ if (caller->regs[i].type == PTR_TO_PACKET_END && !sub->recursive)
+ return false;
+ /* or whether there is something to point to */
+ if (type_may_be_null(caller->regs[i].type) && !sub->recursive)
+ return false;
+ }
+ for (i = 0; i < bpf_in_stack_arg_cnt(sub); i++) {
+ if (caller->stack_arg_regs[i].type == PTR_TO_STACK ||
+ base_type(caller->stack_arg_regs[i].type) == CONST_PTR_TO_DYNPTR)
+ return false;
+ }
+ return true;
+}
+
+static int call_func_on_its_own(struct bpf_verifier_env *env, int subprog, int insn_idx)
+{
+ u16 live = env->insn_aux_data[env->subprog_info[subprog].start].live_regs_before;
+ struct bpf_verifier_state *state = env->cur_state;
+ struct bpf_func_state *caller = state->frame[state->curframe], *callee;
+ struct bpf_verifier_state entry = {};
+ u32 ids[BPF_MAX_ARG_IDS], nids;
+ int i, err;
+
+ callee = kzalloc_obj(*callee, GFP_KERNEL_ACCOUNT);
+ if (!callee)
+ return -ENOMEM;
+ entry.frame[0] = callee;
+ entry.in_sleepable = state->in_sleepable;
+ entry.in_callback = state_in_callback(state);
+ entry.callers_unknown = state->callers_unknown;
+ init_func_state(env, callee, BPF_MAIN_FUNC, 0, subprog);
+ for (i = BPF_REG_1; i <= BPF_REG_5; i++)
+ if (live & BIT(i))
+ callee->regs[i] = caller->regs[i];
+ err = grow_stack_arg_slots(env, callee, bpf_in_stack_arg_cnt(&env->subprog_info[subprog]));
+ if (err) {
+ bpf_free_verifier_state(&entry, false);
+ return err;
+ }
+ for (i = 0; i < callee->out_stack_arg_cnt; i++)
+ callee->stack_arg_regs[i] = caller->stack_arg_regs[i];
+
+ err = copy_reference_state(&entry, state);
+ if (err) {
+ bpf_free_verifier_state(&entry, false);
+ return err;
+ }
+ for (i = 0; i < entry.acquired_refs; i++) {
+ /* the id of a lock is the id of the pointer to it, which may have none */
+ entry.refs[i].id = inherited_id(state, entry.refs[i].id);
+ if (entry.refs[i].type == REF_TYPE_PTR)
+ entry.refs[i].parent_id = inherited_id(state, entry.refs[i].parent_id);
+ }
+ entry.active_irq_id = inherited_id(state, entry.active_irq_id);
+ entry.active_lock_id = inherited_id(state, entry.active_lock_id);
+ callee->entry_refs = entry.acquired_refs;
+ callee->entry_rcu_locks = entry.active_rcu_locks;
+ callee->entry_preempt_locks = entry.active_preempt_locks;
+ nids = collect_arg_ids(env, state, subprog, ids);
+ for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+ callee->regs[i].id = callee_id(state, ids, nids, callee->regs[i].id);
+ callee->regs[i].parent_id = callee_id(state, ids, nids, callee->regs[i].parent_id);
+ }
+ for (i = 0; i < callee->out_stack_arg_cnt; i++) {
+ callee->stack_arg_regs[i].id =
+ callee_id(state, ids, nids, callee->stack_arg_regs[i].id);
+ callee->stack_arg_regs[i].parent_id =
+ callee_id(state, ids, nids, callee->stack_arg_regs[i].parent_id);
+ }
+
+ account_current_path(env);
+ env->subprog_info[subprog].on_its_own = true;
+ err = bpf_flow_call(env, state, &entry, subprog, insn_idx);
+ bpf_free_verifier_state(&entry, false);
+ /* the call waits, there is nothing to go on with */
+ return err ?: PROCESS_BPF_EXIT;
+}
+
+/* The current state is at exit of a function that is verified on its own. */
+static int exit_func_on_its_own(struct bpf_verifier_env *env)
+{
+ struct bpf_verifier_state *state = env->cur_state;
+ struct bpf_func_state *callee = state->frame[0];
+ u32 i, nregs;
+ int err;
+
+ nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+ for (i = 0; i < nregs; i++) {
+ if (callee->regs[ret_regs[i]].type != PTR_TO_STACK)
+ continue;
+ verbose(env, "cannot return stack pointer to the caller\n");
+ return -EINVAL;
+ }
+ state->unwound |= bpf_is_unwind_exit(&env->prog->insnsi[env->insn_idx]);
+
+ /* Invalidate callee-local dynptrs and their slices before the frame goes away. */
+ err = destroy_dynptrs_in_stack_slots(env, callee, 0,
+ callee->allocated_stack / BPF_REG_SIZE - 1);
+ if (err)
+ return err;
+
+ account_current_path(env);
+ err = bpf_flow_exit(env, state, env->insn_idx);
+ return err ?: PROCESS_BPF_EXIT;
+}
+
+static u32 fresh_id(struct bpf_verifier_env *env, struct bpf_idmap *idmap, u32 id)
+{
+ u32 flags = id & BPF_ADD_CONST, i;
+
+ id &= ~BPF_ADD_CONST;
+ if (!id)
+ return 0;
+ for (i = 0; i < idmap->cnt; i++)
+ if (idmap->map[i].old == id)
+ return idmap->map[i].cur | flags;
+ if (idmap->cnt == BPF_ID_MAP_SIZE)
+ return 0;
+ idmap->map[idmap->cnt].old = id;
+ idmap->map[idmap->cnt].cur = ++env->id_gen;
+ return idmap->map[idmap->cnt++].cur | flags;
+}
+
+/*
+ * @caller waits at the call at @call_idx and @exit is what came out of the
+ * function. Make @caller what it is after the call.
+ * Returns 1 if the call has to be verified in the context of the caller.
+ */
+int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
+ const struct bpf_verifier_state *exit, int call_idx)
+{
+ struct bpf_func_state *frame = caller->frame[caller->curframe];
+ const struct bpf_func_state *callee = exit->frame[0];
+ struct bpf_idmap *idmap = &env->idmap_scratch;
+ u32 caller_rcu = caller->active_rcu_locks, caller_preempt = caller->active_preempt_locks;
+ u32 ids[BPF_MAX_ARG_IDS], nids;
+ struct bpf_reference_state *held;
+ struct bpf_reg_state *reg;
+ u32 i, nregs, nheld, kept = 0, id;
+ int err;
+
+ if (exit->unwound && !bpf_is_unwind_goto(&env->prog->insnsi[call_idx + 1])) {
+ verbose(env, "unwind_exit, but no unwind_goto after the call at insn %d\n",
+ call_idx);
+ return -EINVAL;
+ }
+ caller->unwound = exit->unwound;
+
+ nids = collect_arg_ids(env, caller, callee->subprogno, ids);
+ for (i = 0; i < CALLER_SAVED_REGS; i++)
+ bpf_mark_reg_not_init(env, &frame->regs[caller_saved[i]]);
+ for (i = 0; i < frame->out_stack_arg_cnt; i++)
+ bpf_mark_reg_not_init(env, &frame->stack_arg_regs[i]);
+
+ /* what is held now is what was held at the call and what the function took */
+ held = caller->refs;
+ nheld = caller->acquired_refs;
+ caller->refs = NULL;
+ caller->acquired_refs = 0;
+ err = copy_reference_state(caller, exit);
+ if (err) {
+ kfree(held);
+ return err;
+ }
+ if (nheld > BPF_ID_MAP_SIZE / 2) {
+ kfree(held);
+ return -E2BIG;
+ }
+
+ /*
+ * The same state comes out for every call. Ids in it have to be new
+ * every time, or two calls in a row return the same object.
+ */
+ idmap->cnt = 0;
+ for (i = 0; i < nheld; i++) {
+ idmap->map[idmap->cnt].old = BPF_INHERITED_ID + i;
+ idmap->map[idmap->cnt++].cur = held[i].id;
+ }
+ for (i = 0; i < nids; i++) {
+ idmap->map[idmap->cnt].old = BPF_INHERITED_ARG_ID + i;
+ idmap->map[idmap->cnt++].cur = ids[i];
+ }
+ /* What is taken is added at the end. What is released leaves no hole. */
+ for (i = 0; i < nheld && i < caller->acquired_refs; i++) {
+ id = held[i].id ? BPF_INHERITED_ID + i : 0;
+ if (caller->refs[i].id != id || caller->refs[i].type != held[i].type)
+ break;
+ caller->refs[i] = held[i];
+ kept++;
+ }
+ kfree(held);
+ /* What it means for the caller is found out in the context of the caller. */
+ if (kept != nheld || exit->active_rcu_locks < caller_rcu ||
+ exit->active_preempt_locks < caller_preempt) {
+ if (env->log.level & BPF_LOG_LEVEL2)
+ verbose(env, "func#%d releases what its caller holds at insn %d\n",
+ callee->subprogno, call_idx);
+ env->subprog_info[callee->subprogno].releases_inherited = true;
+ return 1;
+ }
+ nregs = bpf_ret_reg_pair(env, callee->subprogno) ? 2 : 1;
+ for (i = 0; i < nregs; i++) {
+ reg = &frame->regs[ret_regs[i]];
+ *reg = callee->regs[ret_regs[i]];
+ reg->id = fresh_id(env, idmap, reg->id);
+ reg->parent_id = fresh_id(env, idmap, reg->parent_id);
+ }
+ /* those that were held are the first */
+ for (i = nheld; i < caller->acquired_refs; i++) {
+ caller->refs[i].id = fresh_id(env, idmap, caller->refs[i].id);
+ if (caller->refs[i].type == REF_TYPE_PTR)
+ caller->refs[i].parent_id = fresh_id(env, idmap, caller->refs[i].parent_id);
+ }
+ caller->active_irq_id = fresh_id(env, idmap, caller->active_irq_id);
+ caller->active_lock_id = fresh_id(env, idmap, caller->active_lock_id);
+ return 0;
+}
/*
- * Call of a static subprog. The callee is verified in the context of
- * the caller, hence set up a new frame and continue from the first
- * instruction of the callee.
+ * Call of a static subprog. Unless the callee is verified on its own it is
+ * verified in the context of the caller, hence set up a new frame and
+ * continue from the first instruction of the callee.
*/
static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
int *insn_idx)
@@ -11551,12 +12086,15 @@ static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) + callee_incoming;
if (stack_arg_cnt > caller_info->stack_arg_cnt)
caller_info->stack_arg_cnt = stack_arg_cnt;
-
+ /* btf_check_subprog_call() doesn't get to it when BTF is not reliable */
err = check_outgoing_stack_args(env, caller, MAX_BPF_FUNC_REG_ARGS + callee_incoming,
bpf_subprog_name(env, subprog), NULL, NULL);
if (err)
return err;
+ if (func_is_on_its_own(env, state, subprog))
+ return call_func_on_its_own(env, subprog, *insn_idx);
+
/*
* For regular function entry setup new frame and continue
* from that frame.
@@ -12162,7 +12700,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
}
caller = state->frame[state->curframe - 1];
- if (bpf_is_unwind_exit(&env->prog->insnsi[*insn_idx])) {
+ if (bpf_is_unwind_exit(&env->prog->insnsi[*insn_idx]) || state->unwound) {
/* the helper or the kfunc that called it doesn't expect that */
if (callee->in_callback_fn) {
verbose(env, "callback cannot unwind\n");
@@ -12181,6 +12719,16 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
return -EACCES;
}
+ /* It's called any number of times. Nobody can release what it leaves behind. */
+ for (i = callee->entry_refs; i < state->acquired_refs; i++) {
+ if (state->refs[i].type != REF_TYPE_PTR)
+ continue;
+ verbose(env, "Unreleased reference id=%d alloc_insn=%d\n",
+ state->refs[i].id, state->refs[i].insn_idx);
+ bpf_diag_leak(env, state->refs[i].id, state->refs[i].insn_idx, *insn_idx);
+ return -EINVAL;
+ }
+
/* we are going to rely on register's precise value */
err = mark_chain_precision(env, BPF_REG_0);
if (err)
@@ -15115,14 +15663,59 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
return info;
}
-/*
- * Describe a kfunc's stack access through argument slot @arg (0-based).
- * As for helpers, must_write describes destruction of prior verifier state,
- * rather than a guarantee that the callee writes every byte at runtime.
- */
-struct arg_access_info
-bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
- int arg, int insn_idx)
+/* insn_aux_data->is_iter_next is not there before the main pass */
+bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+ struct bpf_call_arg_meta meta;
+
+ return bpf_pseudo_kfunc_call(insn) &&
+ !bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta) &&
+ bpf_is_iter_next_kfunc(&meta);
+}
+
+/* The registers in which the kfunc takes nothing but a known constant ("__k") */
+u16 bpf_kfunc_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+ struct bpf_call_arg_meta meta;
+ const struct btf_param *args;
+ u32 i, slot, nargs;
+ u16 mask = 0;
+
+ if (bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta) < 0)
+ return 0;
+ args = btf_params(meta.func_proto);
+ nargs = btf_type_vlen(meta.func_proto);
+ for (i = 0, slot = 0; i < nargs && slot < MAX_BPF_FUNC_REG_ARGS; i++) {
+ if (is_kfunc_arg_constant(meta.btf, &args[i]))
+ mask |= BIT(BPF_REG_1 + slot);
+ slot += btf_arg_slots(btf_type_skip_modifiers(meta.btf, args[i].type, NULL));
+ }
+ return mask;
+}
+
+/* The same for a helper */
+u16 bpf_helper_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+ const struct bpf_func_proto *fn;
+ u16 mask = 0;
+ u32 i;
+
+ if (bpf_get_helper_proto(env, insn->imm, &fn) < 0)
+ return 0;
+ for (i = 0; i < MAX_BPF_FUNC_REG_ARGS; i++)
+ if (fn->arg_type[i] == ARG_CONST_ALLOC_SIZE_OR_ZERO)
+ mask |= BIT(BPF_REG_1 + i);
+ return mask;
+}
+
+/*
+ * Describe a kfunc's stack access through argument slot @arg (0-based).
+ * As for helpers, must_write describes destruction of prior verifier state,
+ * rather than a guarantee that the callee writes every byte at runtime.
+ */
+struct arg_access_info
+bpf_kfunc_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
+ int arg, int insn_idx)
{
struct bpf_insn_aux_data *aux = &env->insn_aux_data[insn_idx];
struct arg_access_info info = {
@@ -19043,6 +19636,21 @@ static void sync_linked_regs(struct bpf_verifier_env *env, struct bpf_verifier_s
}
}
+/* The jump at env->insn_idx goes both ways */
+static void mark_loop_exit_undecided(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
+{
+ struct bpf_func_state *frame = st->frame[st->curframe];
+ struct loop_stack_entry *entry;
+ u32 i;
+
+ for (i = 0; i < frame->loop_stack_cnt; i++) {
+ entry = &frame->loop_stack[i];
+ if (entry->apart && entry->probe &&
+ bpf_jump_leaves_loop(env, entry->loop_id, env->insn_idx))
+ entry->undecided = true;
+ }
+}
+
static int check_cond_jmp_op(struct bpf_verifier_env *env,
struct bpf_insn *insn, int *insn_idx)
{
@@ -19212,6 +19820,7 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
return err;
}
+ mark_loop_exit_undecided(env, this_branch);
other_branch = push_stack(env, *insn_idx + insn->off + 1, *insn_idx, false);
if (IS_ERR(other_branch))
return PTR_ERR(other_branch);
@@ -20263,10 +20872,6 @@ static int save_aux_ptr_type(struct bpf_verifier_env *env, enum bpf_reg_type typ
return 0;
}
-enum {
- PROCESS_BPF_EXIT = 1,
- INSN_IDX_UPDATED = 2,
-};
static int process_bpf_exit_full(struct bpf_verifier_env *env,
bool *do_print_state,
@@ -20274,6 +20879,9 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
{
struct bpf_func_state *cur_frame = cur_func(env);
+ if (!exception_exit && !env->cur_state->curframe && env->cur_state->flow_entry)
+ return exit_func_on_its_own(env);
+
/* We must do check_reference_leak here before
* prepare_func_exit to handle the case when
* state->curframe > 0, it may be a callback function,
@@ -20490,6 +21098,24 @@ static int check_arena_sp_insn(struct bpf_verifier_env *env, struct bpf_insn *in
return err ?: save_aux_ptr_type(env, PTR_TO_ARENA, false);
}
+/*
+ * NULL is dereferenced. In a program that has arena take it for a pointer to arena:
+ * the compiler may have moved the cast to where the number was not known to be 0 yet,
+ * and what check_loop_guard() sets to 0 may be a pointer that was cast in the loop.
+ */
+static void null_to_arena(struct bpf_verifier_env *env, int regno)
+{
+ struct bpf_reg_state *reg;
+
+ if (!env->prog->aux->arena || regno >= BPF_REG_FP)
+ return;
+ reg = &cur_regs(env)[regno];
+ if (reg->type != SCALAR_VALUE || !tnum_equals_const(reg->var_off, 0))
+ return;
+ __mark_reg_const_zero(env, reg);
+ reg->type = PTR_TO_ARENA;
+}
+
static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
{
int err;
@@ -20499,6 +21125,11 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
if (is_arena_sp_insn(insn))
return check_arena_sp_insn(env, insn);
+ if (class == BPF_LDX)
+ null_to_arena(env, insn->src_reg);
+ else if (class == BPF_STX || class == BPF_ST)
+ null_to_arena(env, insn->dst_reg);
+
switch (class) {
case BPF_ALU:
case BPF_ALU64:
@@ -20565,7 +21196,7 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
}
mark_reg_scratched(env, BPF_REG_0);
if (bpf_in_stack_arg_cnt(&env->subprog_info[cur_func(env)->subprogno]))
- cur_func(env)->no_stack_arg_load = true;
+ end_stack_arg_loads(env, cur_func(env));
if (bpf_is_callx(insn))
return check_func_callx(env, insn, &env->insn_idx);
if (insn->src_reg == BPF_PSEUDO_CALL)
@@ -20659,6 +21290,11 @@ static int loop_stack_push(struct bpf_verifier_env *env, bool *pushed)
for (; num_missing; num_missing--, cnt++) {
h = missing_headers[num_missing - 1];
+ /* wherever it is entered: a loop can have more than one entry */
+ if (!aux[h].loop->exits_cnt) {
+ verbose(env, "loop at insn %d never exits\n", h);
+ return -EINVAL;
+ }
loop_stack[cnt] = (struct loop_stack_entry){ .loop_id = h };
if (env->log.level & BPF_LOG_LEVEL2)
verbose(env, "entering loop %d\n", h);
@@ -20728,18 +21364,66 @@ static int maybe_clamp_scev_regs(struct bpf_verifier_env *env)
return bpf_clamp_scev_regs(env, frame, env->insn_idx, entry);
}
-static int handle_loop_entry_exit(struct bpf_verifier_env *env)
+/* How many insns it may take to go through the iterations of a loop one by one. */
+#define BPF_FLOW_MAX_WALK 100000
+/* The same when it is not known that the walk gets to the end of the loop. */
+#define BPF_FLOW_MAX_TRY 4096
+
+/* How many times the current insn is looked at because of the loops around it that are walked. */
+static u64 flow_walk_factor(const struct bpf_verifier_state *st)
+{
+ const struct loop_stack_entry *e;
+ u64 factor = 1;
+ u32 fr, i;
+
+ for (fr = 0; fr <= st->curframe; fr++) {
+ for (i = 0; i < st->frame[fr]->loop_stack_cnt; i++) {
+ e = &st->frame[fr]->loop_stack[i];
+ if (e->apart)
+ factor = min_t(u64, factor * max(e->max_header_count, 1U), U32_MAX);
+ }
+ }
+ return factor;
+}
+
+/*
+ * The current state gets to env->insn_idx. Leave the loops that the insn is not in.
+ * @enters: the insn is the head of a loop that the state is not in yet.
+ */
+int bpf_flow_loop_leave(struct bpf_verifier_env *env, bool *enters)
{
- struct bpf_verifier_state *cur = env->cur_state;
struct bpf_func_state *frame = cur_func(env);
- struct loop_stack_entry *entry;
- struct bpf_loop_iters iters;
- bool pushed, terminates;
int err;
err = loop_stack_pop(env);
if (err)
return err;
+ *enters = env->insn_aux_data[env->insn_idx].loop &&
+ (!frame->loop_stack_cnt ||
+ frame->loop_stack[frame->loop_stack_cnt - 1].loop_id != env->insn_idx);
+ return 0;
+}
+
+/*
+ * Enter the loops that env->insn_idx is in.
+ *
+ * The state that enters a loop at its head is made to stand for all iterations:
+ * - When the number of iterations follows from the state, with the ranges that
+ * induction variables have in that many iterations.
+ * - Otherwise the numbers that the loop changes are forgotten.
+ * The state that comes back to the head is then expected to be within it.
+ */
+int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop)
+{
+ struct bpf_verifier_state *cur = env->cur_state, *entry_state;
+ const struct bpf_insn *insn = &env->prog->insnsi[env->insn_idx];
+ struct bpf_func_state *frame = cur_func(env);
+ struct loop_stack_entry *entry;
+ struct bpf_loop_iters iters;
+ struct bpf_loop *loop;
+ bool pushed;
+ u32 patience;
+ int err;
err = loop_stack_push(env, &pushed);
if (err)
@@ -20752,43 +21436,336 @@ static int handle_loop_entry_exit(struct bpf_verifier_env *env)
if (env->insn_idx != entry->loop_id)
return 0;
+ /*
+ * No edge goes from a call to itself: the call goes on with one function of a set, see
+ * split_func_set(), in the same iteration.
+ */
+ if (!pushed && cur->last_insn_idx == env->insn_idx &&
+ BPF_CLASS(insn->code) == BPF_JMP && BPF_OP(insn->code) == BPF_CALL)
+ return 0;
+
+ /* The callback came back to the helper that calls it, in the same iteration. */
+ if (frame->callback_depth)
+ return 0;
+
/* If nothing was pushed and env->insn_idx is a loop header, we've taken a backedge. */
- if (!pushed)
- return maybe_clamp_scev_regs(env);
+ if (!pushed) {
+ if (!entry->apart) {
+ *backedge = true;
+ return maybe_clamp_scev_regs(env);
+ }
+ /* the next iteration is another kind of state, there is no cycle */
+ if (!entry->undecided && ++entry->trip < entry->max_header_count &&
+ (entry->counted ||
+ env->insn_processed - entry->walk_start <= entry->walk_patience))
+ return 0;
+ /* there is no such iteration at run time */
+ if (entry->counted) {
+ *drop = true;
+ return 0;
+ }
+ /*
+ * The walk doesn't get to the end of the loop. Make this state stand for
+ * all iterations. It stands for those that were looked at as well.
+ */
+ loop = env->insn_aux_data[env->insn_idx].loop;
+ if (!loop->rt_bounded && !loop->has_guards) {
+ verbose(env, "loop at insn %d never exits\n", env->insn_idx);
+ return -EINVAL;
+ }
+ entry->apart = false;
+ entry->probe = false;
+ entry->undecided = false;
+ entry->terminates = false;
+ entry->trip = 0;
+ entry->max_header_count = 0;
+ return bpf_scev_forget_regs(env, cur);
+ }
- err = bpf_compute_loop_iters(env, env->cur_state, &iters);
+ loop = env->insn_aux_data[env->insn_idx].loop;
+ err = bpf_compute_loop_iters(env, cur, &iters);
if (err < 0)
return err;
- terminates = err == 1;
- if (!terminates)
+ /*
+ * There is no summary of what the loop does to something that is live.
+ * Go through the iterations one by one if that is not too long.
+ */
+ if (err == 2 && flow_walk_factor(cur) * iters.max_header_count * loop->nr_insns <=
+ BPF_FLOW_MAX_WALK) {
+ entry->max_header_count = iters.max_header_count;
+ entry->terminates = true;
+ entry->counted = true;
+ entry->apart = true;
+ return 0;
+ }
+ if (err == 2 && iters.spill_base)
+ err = 0;
+ /* The walk has to get to the end of the loop, so it is known to have one. */
+ /*
+ * Or it's found out on the way: the walk goes on for as long as the jumps that
+ * leave the loop are decided.
+ */
+ patience = bpf_scev_latch_decided(env, cur) ? BPF_FLOW_MAX_TRY :
+ env->allow_uninit_stack && !loop->rt_bounded && !loop->irreducible &&
+ !loop->has_calls ? BPF_FLOW_MAX_TRY : 0;
+ /* Two trips for a loop of any length. Many a loop is left in the second one. */
+ if (patience && flow_walk_factor(cur) == 1)
+ patience = max_t(u64, patience, 2ULL * loop->nr_insns);
+ if (!err && flow_walk_factor(cur) * loop->nr_insns <= patience) {
+ entry->max_header_count = max_t(u32, patience /
+ (flow_walk_factor(cur) * loop->nr_insns), 1U);
+ entry->terminates = true;
+ entry->probe = !bpf_scev_latch_decided(env, cur);
+ entry->walk_start = env->insn_processed;
+ entry->walk_patience = patience;
+ entry->apart = true;
return 0;
+ }
- /* Create a loop_entry checkpoint before widening */
- err = bpf_split_cur_state(env);
- if (err)
- return err;
+ if (!err) {
+ if (!loop->rt_bounded && !loop->has_guards) {
+ verbose(env, "loop at insn %d never exits\n", env->insn_idx);
+ return -EINVAL;
+ }
+ return bpf_scev_forget_regs(env, cur);
+ }
+
+ /* What the ranges are derived from for as long as the loop is looked at. */
+ entry_state = bpf_flow_keep_state(env, cur);
+ if (!entry_state)
+ return -ENOMEM;
entry->min_header_count = iters.min_header_count;
entry->max_header_count = iters.max_header_count;
entry->iv_reg = iters.iv_reg;
entry->iv_range = iters.iv_range;
- entry->entry_state = cur->parent;
+ entry->entry_state = entry_state;
entry->terminates = true;
- err = bpf_widen_scev_regs(env, cur, cur->parent, &iters);
- if (err < 0)
+ err = bpf_widen_scev_regs(env, cur, entry_state, &iters);
+ return err < 0 ? err : 0;
+}
+
+/* Can the main program return 0? */
+static bool zero_is_valid_retval(struct bpf_verifier_env *env)
+{
+ struct bpf_retval_range range;
+
+ if (program_returns_void(env) || !return_retval_range(env, &range))
+ return true;
+ return range.minval <= 0 && range.maxval >= 0;
+}
+
+/*
+ * A loop or a recursion is out of budget in the current frame of @st. What the
+ * program computed so far is not what it was meant to compute, so it's better
+ * not to go on. The way to stop is to unwind: 'r0 = 0; unwind_exit'. The landing
+ * pads of the callers are code like any other. They release what the callers
+ * hold, or the program is rejected.
+ *
+ * False when it cannot:
+ * - There is no landing pad for what this frame took itself.
+ * - There is a call of the subprog that unwind_goto doesn't follow. That is every
+ * program that is compiled without unwinding.
+ * - A helper that calls a callback doesn't expect it to unwind.
+ * Then the program goes on along a path that it has, like after may_goto of
+ * its own. It releases what it holds there.
+ */
+static bool budget_can_unwind(struct bpf_verifier_env *env, const struct bpf_verifier_state *st)
+{
+ const struct bpf_func_state *frame = st->frame[st->curframe];
+ const struct bpf_subprog_info *sub = &env->subprog_info[frame->subprogno];
+ bool top;
+
+ if (sub->bail_insn < 0)
+ return false;
+ if (st->acquired_refs != frame->entry_refs ||
+ st->active_rcu_locks != frame->entry_rcu_locks ||
+ st->active_preempt_locks != frame->entry_preempt_locks)
+ return false;
+ if (frame->in_callback_fn || frame->in_exception_callback_fn)
+ return false;
+ /* the kernel called it: either exit ends it */
+ top = !st->curframe && !st->callers_unknown &&
+ (!frame->subprogno || frame->in_async_callback_fn);
+ if (top && !frame->subprogno && !zero_is_valid_retval(env))
+ return false;
+ return top || sub->bail_unwinds;
+}
+
+/*
+ * The current frame leaves with 0 in place of the result: with unwind_exit if the subprog
+ * can do that, with exit otherwise. The insns are added when the main pass is over. Until
+ * then an exit that the subprog has stands for them.
+ */
+static int push_budget_exit(struct bpf_verifier_env *env)
+{
+ struct bpf_subprog_info *sub = &env->subprog_info[cur_func(env)->subprogno];
+ struct bpf_verifier_state *branch;
+ struct bpf_reg_state *regs;
+
+ branch = push_stack(env, sub->bail_insn, env->insn_idx, false);
+ if (IS_ERR(branch))
+ return PTR_ERR(branch);
+ branch->unwound = sub->bail_unwinds;
+ regs = branch->frame[branch->curframe]->regs;
+ __mark_reg_const_zero(env, ®s[BPF_REG_0]);
+ if (bpf_ret_reg_pair(env, cur_func(env)->subprogno))
+ __mark_reg_const_zero(env, ®s[BPF_REG_2]);
+ return 0;
+}
+
+/*
+ * The loop that is not counted and has nothing that stops it at run time gets
+ * may_goto in front of this insn. When it is out of budget the program goes
+ * where may_goto goes with the registers and the stack as they are here.
+ */
+static int check_loop_guard(struct bpf_verifier_env *env)
+{
+ struct bpf_insn_aux_data *aux = &env->insn_aux_data[env->insn_idx];
+ struct bpf_func_state *frame = cur_func(env);
+ struct bpf_verifier_state *branch;
+ struct loop_stack_entry *entry;
+ struct bpf_func_state *func;
+ struct bpf_stack_state *slot;
+ struct bpf_reg_state *regs;
+ struct bpf_loop *loop;
+ int h, i, j, err;
+
+ h = bpf_loop_at_index(env, env->insn_idx);
+ if (h < 0 || !frame->loop_stack_cnt)
+ return 0;
+ loop = env->insn_aux_data[h].loop;
+ entry = &frame->loop_stack[frame->loop_stack_cnt - 1];
+ if (loop->rt_bounded || !env->insn_aux_data[env->insn_idx].guard_place ||
+ entry->loop_id != h || entry->terminates)
+ return 0;
+
+ /*
+ * A guard has one way out, and only that one is looked at.
+ *
+ * Leave the subprog, if that is possible every time the program gets here:
+ * do { p = lookup(); } while (!p); *p
+ * The code that follows the loop counts on what the jump that leaves it has found out.
+ *
+ * Otherwise leave the loop:
+ * lock(); do { ... } while (...); unlock();
+ * The code that follows the loop releases what is held.
+ *
+ * It's the same way for all states that get here. As long as they leave the subprog
+ * what would leave the loop is put aside.
+ */
+ if (!budget_can_unwind(env, env->cur_state))
+ aux->guard_no_unwind = true;
+ if (!aux->loop_guard && !aux->guard_no_unwind) {
+ aux->loop_guard = true;
+ aux->guard_unwind = true;
+ }
+ if (aux->guard_unwind && aux->guard_no_unwind) {
+ aux->guard_unwind = false;
+ bpf_flow_unpark(env, env->insn_idx);
+ }
+ if (!aux->guard_no_unwind) {
+ err = push_budget_exit(env);
+ if (err)
+ return err;
+ }
+ if (aux->guard_to < 0) {
+ if (aux->guard_unwind)
+ return 0;
+ verbose(env, "loop at insn %d never exits\n", h);
+ return -EINVAL;
+ }
+
+ branch = push_stack(env, env->insn_aux_data[env->insn_idx].guard_to, env->insn_idx, false);
+ if (IS_ERR(branch))
+ return PTR_ERR(branch);
+ env->head->parked = aux->guard_unwind;
+ aux->loop_guard = true;
+ if (!aux->guard_at_header)
+ return 0;
+ /*
+ * The program doesn't leave the loop from here. What it computes in the loop
+ * for the code that follows is not there. Nothing good comes out of a loop
+ * that is cut short anyway. Let it be 0.
+ */
+ regs = branch->frame[branch->curframe]->regs;
+ for (i = BPF_REG_0; i < BPF_REG_FP; i++)
+ if (regs[i].type == NOT_INIT &&
+ (env->insn_aux_data[aux->guard_to].live_regs_before & BIT(i)))
+ aux->guard_zero |= BIT(i);
+ for (i = BPF_REG_0; i < BPF_REG_FP; i++)
+ if (aux->guard_zero & BIT(i))
+ __mark_reg_const_zero(env, ®s[i]);
+
+ func = branch->frame[branch->curframe];
+ branch->insn_idx = aux->guard_to;
+ err = bpf_live_stack_query_init(env, branch);
+ if (err)
return err;
+ for (i = 0; i < min_t(int, bpf_stack_nr_slots(func), 64); i++) {
+ if (!bpf_stack_slot_alive(env, branch->curframe, i * 2) &&
+ !bpf_stack_slot_alive(env, branch->curframe, i * 2 + 1))
+ continue;
+ slot = bpf_stack_slot(func, i);
+ for (j = 0; j < BPF_REG_SIZE; j++)
+ if (slot->slot_type[j] == STACK_INVALID ||
+ slot->slot_type[j] == STACK_POISON)
+ aux->guard_zero_slots |= BIT_ULL(i);
+ }
+ for (i = 0; i < min_t(int, bpf_stack_nr_slots(func), 64); i++) {
+ if (!(aux->guard_zero_slots & BIT_ULL(i)))
+ continue;
+ slot = bpf_stack_slot(func, i);
+ bpf_mark_reg_not_init(env, &slot->spilled_ptr);
+ for (j = 0; j < BPF_REG_SIZE; j++)
+ slot->slot_type[j] = STACK_ZERO;
+ }
return 0;
}
-static int do_check(struct bpf_verifier_env *env)
+/* The states that the insn pushed go where they were pushed to. */
+static int flow_drain(struct bpf_verifier_env *env)
+{
+ struct bpf_verifier_stack_elem *elem;
+ int err = 0;
+
+ while ((elem = env->head)) {
+ env->head = elem->next;
+ env->stack_size--;
+ if (!err && elem->parked)
+ err = bpf_flow_park(env, &elem->st, elem->insn_idx, elem->prev_insn_idx);
+ else if (!err)
+ err = bpf_flow_deliver(env, &elem->st, elem->insn_idx,
+ elem->prev_insn_idx);
+ bpf_free_verifier_state(&elem->st, false);
+ kfree(elem);
+ }
+ return err ?: bpf_flow_deliver_unparked(env);
+}
+
+/*
+ * A subprog on a cycle of the call graph counts its frames at run time, see
+ * bpf_insert_recursion_checks(). When there are too many it leaves at once
+ * with 0 in place of the result: it unwinds if it has unwind_exit, otherwise it
+ * returns and callers have to be fine with that.
+ */
+static int check_recursion_guard(struct bpf_verifier_env *env)
+{
+ struct bpf_subprog_info *sub = &env->subprog_info[cur_func(env)->subprogno];
+
+ if (!sub->recursive || env->insn_idx != sub->start)
+ return 0;
+ return push_budget_exit(env);
+}
+
+/* Process the current state until it gets to the start of a basic block. */
+static int do_check_block(struct bpf_verifier_env *env, int prev_insn_idx)
{
- bool pop_log = !(env->log.level & BPF_LOG_LEVEL2);
struct bpf_verifier_state *state = env->cur_state;
struct bpf_insn *insns = env->prog->insnsi;
int insn_cnt = env->prog->len;
- bool do_print_state = false;
- int prev_insn_idx = -1;
+ bool do_print_state = true;
for (;;) {
struct bpf_insn *insn;
@@ -20801,9 +21778,6 @@ static int do_check(struct bpf_verifier_env *env)
if (need_resched())
cond_resched();
- /* reset current history entry on each new instruction */
- env->cur_hist_ent = NULL;
-
env->prev_insn_idx = prev_insn_idx;
if (env->insn_idx >= insn_cnt) {
verbose(env, "invalid insn idx %d insn_cnt %d\n",
@@ -20816,18 +21790,10 @@ static int do_check(struct bpf_verifier_env *env)
account_processed_insn(env);
- if (env->insn_processed > BPF_COMPLEXITY_LIMIT_INSNS) {
- verbose(env,
- "BPF program is too large. Processed %d insn\n",
- env->insn_processed);
- return -E2BIG;
- }
-
state->last_insn_idx = env->prev_insn_idx;
state->insn_idx = env->insn_idx;
/*
- * Record the incoming edge so active and queued paths use the same
- * branch-recording path. A zero-offset conditional has identical
+ * Record the incoming edge. A zero-offset conditional has identical
* successors, so its outcome cannot be reconstructed from the edge.
*/
if (!state->speculative && prev_insn_idx >= 0 && prev_insn_idx < insn_cnt) {
@@ -20847,40 +21813,6 @@ static int do_check(struct bpf_verifier_env *env)
}
}
- /*
- * Possibly widen the registers before creating a checkpoint
- * in bpf_is_state_visited(). The next loop iteration will
- * have a chance to hit this checkpoint and converge.
- */
- err = handle_loop_entry_exit(env);
- if (err)
- return err;
-
- if (bpf_is_prune_point(env, env->insn_idx)) {
- err = bpf_is_state_visited(env, env->insn_idx);
- if (err < 0)
- return err;
- if (err == 1) {
- /* found equivalent state, can prune the search */
- if (env->log.level & BPF_LOG_LEVEL) {
- if (do_print_state)
- verbose(env, "\nfrom %d to %d%s: safe\n",
- env->prev_insn_idx, env->insn_idx,
- env->cur_state->speculative ?
- " (speculative execution)" : "");
- else
- verbose(env, "%d: safe\n", env->insn_idx);
- }
- goto process_bpf_exit;
- }
- }
-
- if (bpf_is_jmp_point(env, env->insn_idx)) {
- err = bpf_push_jmp_history(env, state, 0, 0, 0, NULL, 0);
- if (err)
- return err;
- }
-
if (env->log.level & BPF_LOG_LEVEL2 && do_print_state) {
verbose(env, "\nfrom %d to %d%s:",
env->prev_insn_idx, env->insn_idx,
@@ -20890,13 +21822,6 @@ static int do_check(struct bpf_verifier_env *env)
do_print_state = false;
}
- if (bpf_loop_at_index(env, env->insn_idx) >= 0 &&
- cur_func(env)->loop_stack_cnt == 0) {
- verifier_bug(env, "loop stack empty at %d, while inside the loop %d\n",
- env->insn_idx, bpf_loop_at_index(env, env->insn_idx));
- return -EFAULT;
- }
-
if (env->log.level & BPF_LOG_LEVEL) {
if (verifier_state_scratched(env))
print_insn_state(env, state, state->curframe);
@@ -20947,6 +21872,14 @@ static int do_check(struct bpf_verifier_env *env)
if (state->speculative && insn_aux->nospec)
goto process_bpf_exit;
+ err = check_loop_guard(env);
+ if (err)
+ return err;
+
+ err = check_recursion_guard(env);
+ if (err)
+ return err;
+
err = do_check_insn(env, &do_print_state);
if (error_recoverable_with_nospec(err) && state->speculative) {
/* Prevent this speculative path from ever reaching the
@@ -20968,19 +21901,8 @@ static int do_check(struct bpf_verifier_env *env)
}
if (state->speculative && insn_aux->nospec_result) {
- /* If we are on a path that performed a jump-op, this
- * may skip a nospec patched-in after the jump. This can
- * currently never happen because nospec_result is only
- * used for the write-ops
- * `*(size*)(dst_reg+off)=src_reg|imm32` and helper
- * calls. These must never skip the following insn
- * (i.e., bpf_insn_successors()'s opcode_info.can_jump
- * is false). Still, add a warning to document this in
- * case nospec_result is used elsewhere in the future.
- *
- * All non-branch instructions have a single
- * fall-through edge. For these, nospec_result should
- * already work.
+ /* All non-branch instructions have a single fall-through
+ * edge. For these, nospec_result should already work.
*/
if (verifier_bug_if((BPF_CLASS(insn->code) == BPF_JMP ||
BPF_CLASS(insn->code) == BPF_JMP32) &&
@@ -20990,23 +21912,40 @@ static int do_check(struct bpf_verifier_env *env)
process_bpf_exit:
account_current_path(env);
mark_verifier_state_scratched(env);
- err = bpf_update_branch_counts(env, env->cur_state);
- if (err)
- return err;
- err = pop_stack(env, &prev_insn_idx, &env->insn_idx,
- pop_log);
- if (err < 0) {
- if (err != -ENOENT)
- return err;
- break;
- } else {
- do_print_state = true;
- continue;
- }
+ bpf_free_verifier_state(state, false);
+ memset(state, 0, sizeof(*state));
+ return flow_drain(env);
+ }
+
+ err = flow_drain(env);
+ if (err)
+ return err;
+
+ if (env->insn_idx < insn_cnt && bpf_flow_is_point(env, env->insn_idx)) {
+ account_current_path(env);
+ mark_verifier_state_scratched(env);
+ return bpf_flow_deliver(env, state, env->insn_idx, prev_insn_idx);
}
}
+}
- return 0;
+static int do_check(struct bpf_verifier_env *env)
+{
+ int prev_insn_idx = -1;
+ int err;
+
+ err = bpf_flow_deliver(env, env->cur_state, env->insn_idx, -1);
+ while (!err) {
+ err = bpf_flow_next(env, &prev_insn_idx);
+ if (err == -ENOENT)
+ return 0;
+ if (err)
+ break;
+ /* What is known about how the state came to be is not kept with it. */
+ bpf_diag_event_log_restore(env, 0);
+ err = do_check_block(env, prev_insn_idx);
+ }
+ return err;
}
static int find_btf_percpu_datasec(struct btf *btf)
@@ -22069,6 +23008,7 @@ static void free_states(struct bpf_verifier_env *env)
bpf_free_verifier_state(env->cur_state, true);
env->cur_state = NULL;
while (!pop_stack(env, NULL, NULL, false));
+ bpf_flow_reset(env);
list_for_each_safe(pos, tmp, &env->free_list) {
sl = container_of(pos, struct bpf_verifier_state_list, node);
@@ -22102,8 +23042,8 @@ static void free_states(struct bpf_verifier_env *env)
}
}
-static int do_check_common(struct bpf_verifier_env *env, int subprog,
- enum bpf_func_context context)
+static int __do_check_common(struct bpf_verifier_env *env, int subprog,
+ enum bpf_func_context context)
{
bool pop_log = !(env->log.level & BPF_LOG_LEVEL2);
struct bpf_subprog_info *sub = subprog_info(env, subprog);
@@ -22124,6 +23064,8 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
state->speculative = false;
state->branches = 1;
state->in_sleepable = context == BPF_FUNC_CTX_SLEEPABLE;
+ /* nobody knows what the callers of a global function or of an extension hold */
+ state->callers_unknown = subprog || env->prog->type == BPF_PROG_TYPE_EXT;
env->no_rcu_read_lock = context == BPF_FUNC_CTX_NO_RCU_LOCK;
state->frame[0] = kzalloc_obj(struct bpf_func_state, GFP_KERNEL_ACCOUNT);
if (!state->frame[0]) {
@@ -22230,7 +23172,7 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
* it's matching expected signature, and otherwise mark BTF
* info for main program as unreliable
*/
- if (env->prog->aux->func_info_aux) {
+ if (env->prog->aux->func_info_aux && !env->prog->aux->func_info_aux[0].unreliable) {
ret = btf_prepare_func_args(env, 0);
if (ret || sub->arg_slot_cnt != 1 ||
sub->args[0].arg_type != ARG_PTR_TO_CTX) {
@@ -22279,6 +23221,26 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
return ret;
}
+static int do_check_common(struct bpf_verifier_env *env, int subprog,
+ enum bpf_func_context context)
+{
+ int insn_idx = env->insn_idx, tries = 0, ret;
+ u64 log_pos = env->log.end_pos;
+
+ for (;;) {
+ env->restart = false;
+ ret = __do_check_common(env, subprog, context);
+ if (!ret || !env->restart)
+ return ret;
+ if (++tries == 8) {
+ verbose(env, "verification was started again too many times\n");
+ return ret;
+ }
+ bpf_vlog_reset(&env->log, log_pos);
+ env->insn_idx = insn_idx;
+ }
+}
+
/* Lazily verify all global functions based on their BTF, if they are called
* from main BPF program or any of subprograms transitively.
* BPF global subprogs called from dead code are not validated.
@@ -22364,6 +23326,37 @@ static int do_check_main(struct bpf_verifier_env *env)
return ret;
}
+/*
+ * What it took to verify a static subprog on its own is not in insns_total of its callers.
+ * Add it, once per caller. The main prog and global subprogs have all of it already.
+ */
+static void add_insns_of_callees(struct bpf_verifier_env *env)
+{
+ struct bpf_subprog_info *si = env->subprog_info;
+ struct bpf_insn *insn = env->prog->insnsi;
+ int *added, i, j, s, c;
+
+ added = kvcalloc(env->subprog_cnt, sizeof(*added), GFP_KERNEL_ACCOUNT);
+ if (!added)
+ return;
+ /* callees first */
+ for (i = 0; i < env->subprog_cnt; i++) {
+ s = env->subprog_topo_order[i];
+ if (!s || bpf_subprog_is_global(env, s))
+ continue;
+ for (j = si[s].start; j < si[s + 1].start; j++) {
+ if (!bpf_pseudo_call(&insn[j]))
+ continue;
+ c = bpf_find_subprog(env, j + insn[j].imm + 1);
+ if (c <= 0 || c == s || !si[c].on_its_own || added[c] == s)
+ continue;
+ added[c] = s;
+ si[s].insns_total += si[c].insns_total;
+ }
+ }
+ kvfree(added);
+}
+
static void print_verification_stats(struct bpf_verifier_env *env)
{
/* Skip over hidden subprogs which are not verified. */
@@ -24351,6 +25344,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
+ ret = mark_recursive_subprogs(env);
+ if (ret < 0)
+ goto skip_full_check;
+
ret = sort_subprogs_topo(env);
if (ret < 0)
goto skip_full_check;
@@ -24389,6 +25386,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
+ ret = bpf_flow_prepare(env);
+ if (ret < 0)
+ goto skip_full_check;
+
bpf_free_subprog_jts(env);
ret = mark_fastcall_patterns(env);
@@ -24397,10 +25398,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
ret = do_check_main(env);
ret = ret ?: do_check_subprogs(env);
-
- /* reject recursion through the callx edges found by the main pass */
- if (ret == 0 && env->callx_edges)
- ret = sort_subprogs_topo(env);
+ /* while the calls are what they are in the program */
+ if (env->log.level & BPF_LOG_STATS)
+ add_insns_of_callees(env);
if (ret == 0)
ret = bpf_insert_loop_guards(env);
@@ -24560,6 +25560,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
vfree(env->insn_aux_data);
kvfree(env->fd_array);
bpf_free_scev(env);
+ bpf_flow_free(env);
+ bpf_join_free(env);
bpf_stack_liveness_free(env);
kvfree(env->cfg.postorder_nums);
kvfree(env->cfg.insn_postorder);
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 53/81] bpf: Verify programs as a fixpoint over the control flow graph
2026-10-26 2:59 ` [PATCH bpf 53/81] bpf: Verify programs " Alexei Starovoitov
@ 2026-10-26 4:25 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 4:25 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 50682 bytes --]
> bpf: Verify programs as a fixpoint over the control flow graph
TL;DR:
- Tail calls below recursive subprogs may overflow the kernel stack or skip
tail-call counter setup, causing wild pointer access
- Possible stack-arg, lock and RCU-context tracking holes in subprogs verified
on their own
- Some valid programs may now be rejected
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index a9628f5045fe..0c1de0f24b85 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -4598,6 +4601,23 @@ static int check_stack_write(struct bpf_verifier_env *env,
> return err;
> }
>
> +/*
> + * The function that is verified on its own has no caller to look at. Its incoming
> + * arguments are in its stack_arg_regs until it is done with loading them.
> + */
> +static void end_stack_arg_loads(struct bpf_verifier_env *env, struct bpf_func_state *state)
> +{
> + int i;
> +
> + if (state->no_stack_arg_load)
> + return;
> + state->no_stack_arg_load = true;
> + if (state->frameno)
> + return;
> + for (i = 0; i < state->out_stack_arg_cnt; i++)
> + bpf_mark_reg_not_init(env, &state->stack_arg_regs[i]);
> +}
Can an unwritten incoming stack arg survive as an outgoing one after a join?
For frame 0 of a static subprog that is verified on its own,
stack_arg_regs[0..in_stack_arg_cnt) hold the incoming stack args.
end_stack_arg_loads() wipes them on the first r11 store or the first call
insn, but only for the state that executes it.
That step is not applied when states are compared or joined. In
func_states_equal():
if (!old->no_stack_arg_load && cur->no_stack_arg_load)
return false;
only rejects the case where the old state has the flag clear. An old state
that already stored an outgoing value V in slot 0 (flag set) therefore covers
a current state whose slot 0 still holds the unwritten incoming arg A,
whenever the range of V contains A. In join_func():
fd->no_stack_arg_load = fa->no_stack_arg_load || fb->no_stack_arg_load;
the two slots are joined with join_reg() and the flag is ORed, so at the next
call insn end_stack_arg_loads() returns at once and the joined slot is never
invalidated.
Example, with F a static subprog with one incoming stack arg that is verified
on its own, and G a callee with one stack arg:
F: r0 = *(u64 *)(r11 + 8)
if r1 == 0 goto L1
*(u64 *)(r11 - 8) = r0
L1: r1..r5 = 0
call G
exit
Verified alone, the path with r1 == 0 is rejected with 'callee expects 6 args,
stack arg1 is not initialized'. When the two states meet at L1 and one covers
or joins the other, the call is verified once with slot 0 as a scalar and that
path is accepted.
At run time on that path the outgoing area of the frame of F is uninitialized
stack memory, but G is verified assuming arg6 is in the joined range. A
pointer type with joinable fixed parts (PTR_TO_MAP_VALUE or PTR_TO_CTX in both
slots) is accepted the same way, so an unwritten, possibly attacker controlled
stack word can be passed as a typed argument. This needs a JIT with stack arg
support (x86-64) and a static subprog whose code loads r11+off.
Before this patch frame 0 had no incoming stack args, stack_arg_regs was
outgoing only, and joining 'written' with 'unwritten' gave NOT_INIT.
Should func_states_equal() treat a frame 0 pair with differing
no_stack_arg_load (and in_stack_arg_cnt > 0) as incomparable, and join_func()
refuse to join it? Alternatively the end_stack_arg_loads() wipe could be
applied to the side whose flag is clear before comparing or joining. I did
not find a later patch in the series that touches no_stack_arg_load.
[ ... ]
> @@ -5930,17 +5958,37 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
> * The main verification pass limits the depth of the recursion,
> * but it doesn't follow calls of global functions.
> */
Is this comment still accurate?
It explained the -EINVAL that this loop used to return for a subprog that is
already on the chain. The loop now accepts the cycle instead ("Recursion.
How deep it goes is not known here. It's counted at run time.").
sort_subprogs_topo() now tolerates every cycle, not only the ones that go
through the address of a function. And "The main verification pass limits the
depth of the recursion" does not hold for a subprog that
mark_recursive_subprogs() marked: such a subprog is verified on its own, and
its depth is limited by the check at its entry at run time
(bpf_insert_recursion_checks()).
A reader who relies on the comment will think that the case is rejected or
bounded by the main pass. Could the comment describe the new behaviour, or be
dropped?
> - for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) {
> - if (tmp != sidx)
> + for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller)
> + if (tmp == sidx)
> + break;
> + /*
> + * Recursion. How deep it goes is not known here. It's counted
> + * at run time.
> + */
> + if (tmp >= 0) {
> + if (bpf_is_callx(insn + i) || bpf_calls_callback(env, i))
> + i--;
> + continue;
> + }
Can this cut leave a subprog on a cycle without tail_call_reachable?
Before this patch a call edge whose target was already on the chain was an
error, so every subprog that could reach a tail call was on a walked chain.
Now the edge is cut and the walk goes on. The DFS visits a member of a cycle
only below the member where it entered the cycle, so a member whose only way
to the tail call goes through the cut edge is never marked. The marking is
done only for the subprogs on the chain from the root to the subprog that has
the tail call:
for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller)
...
subprog[tmp].tail_call_reachable = true;
Call graph: main calls Q only. Q calls P and then T. P calls Q. T contains
bpf_tail_call(). mark_recursive_subprogs() marks P and Q recursive and does
not reject them: it rejects a tail call only inside the recursive subprog
itself, and T is not on the cycle.
The walk goes main -> Q -> P. The call P -> Q is on the chain and is cut, so
P is left without a mark. Then Q -> T marks T, Q and main. The result is
P.tail_call_reachable == false although P -> Q -> T is a run time path.
On x86-64 a call to a subprog loads the pointer to the tail call counter into
rax only if the caller is tail_call_reachable (arch/x86/net/bpf_jit_comp.c,
"if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable)"). A
tail_call_reachable subprog saves rax as that pointer in its prologue
(emit_prologue_tail_call(), is_subprog case). Here P calls Q without loading
it, so Q saves whatever P left in rax (BPF R0). T then does 'cmp qword ptr
[rax], MAX_TAIL_CALL_CNT' and 'add qword ptr [rax], 1' on the pointer it
reloads from the frame of Q. That is a read and an increment through a value
that the verifier never constrained.
The combination was impossible before this patch because sort_subprogs_topo()
rejected every cycle. recursive_frame_size() in fixups.c reserves room for
tail_call_reachable in the frame of a recursive subprog, so the combination
looks meant to load.
Should every subprog from which a tail call is reachable be marked
(transitively, so all members of a cycle that reaches T), or should tail calls
below recursive subprogs be rejected? I did not find a later patch in the
series that changes this.
> + /*
> + * Any recursive subprog may follow any other when they call each other
> + * through pointers. Don't walk all such chains: what is called from
> + * here was looked at with at least as much on the stack.
> + */
> + if (subprog[sidx].recursive) {
> + bool cb = subprog[sidx].is_cb;
> +
> + for (tmp = idx; tmp >= 0 && !cb; tmp = dinfo[tmp].caller)
> + cb = subprog[tmp].is_cb;
> + if (depth < dinfo[sidx].seen_depth[cb] &&
> + arena_depth < dinfo[sidx].seen_arena_depth[cb]) {
> + if (bpf_is_callx(insn + i) || bpf_calls_callback(env, i))
> + i--;
> continue;
> - verbose(env, "recursive call from %s() to %s()\n",
> - bpf_subprog_name(env, idx), bpf_subprog_name(env, sidx));
> - bpf_diag_program_structure(
> - env, i, "recursive subprogram call",
> - "Rewrite the recursion as an explicit bounded loop, or split the logic so subprogram calls do not form a cycle.",
> - "This call or callback would make the subprogram call graph recursive. "
> - "The verifier requires a finite, acyclic call graph so it can bound stack depth and analysis.");
> - return -EINVAL;
> + }
> + dinfo[sidx].seen_depth[cb] = max(dinfo[sidx].seen_depth[cb], depth + 1);
> + dinfo[sidx].seen_arena_depth[cb] = max(dinfo[sidx].seen_arena_depth[cb],
> + arena_depth + 1);
> }
Does this skip also drop effects that depend on the chain and not only on the
depth?
The comment says "what is called from here was looked at with at least as much
on the stack". That holds for the summed stack depth. The walk computes more
than that:
- tail_call_reachable is set for each subprog on the chain that leads to a
tail call
- the checks at the tail call (stack args, via_callx, is_cb) look at the
chain
- uses_arena_sp sets arena_stack_entry on the nearest callback of the chain
A second chain that reaches the recursive subprog with the same or less stack
skips all of it.
Call graph, for tail_call_reachable only: main calls X and then Y. X and Y
have the same stack depth (for example both 0). Both call rec. rec is
recursive (it calls itself) and calls T. T has bpf_tail_call().
The walk of main -> X -> rec sets seen_depth[0] of rec and walks T, so T, rec,
X and main become tail_call_reachable. In the walk of main -> Y -> rec the
test
depth < dinfo[sidx].seen_depth[cb] &&
arena_depth < dinfo[sidx].seen_arena_depth[cb]
is true, so the walk takes the continue. Y is never marked, although Y -> rec
-> T is a run time path. Which of X and Y is skipped depends only on the
order of the calls in main, and a chain with less stack is skipped as well.
The JIT then compiles Y without tail call counter handling: on x86-64 a call
from an unmarked caller does not load the counter pointer into rax, and rec
(marked) saves rax in its prologue as that pointer. The tail call in T
dereferences it, so the value is whatever Y left in R0 at the call (see the
previous comment for the JIT side).
The same skip leaves the nearest callback of a second chain without
arena_stack_entry when a subprog below rec uses the stack in arena. That
callback is then not given its stack in arena by bpf_insert_arena_stack()
although code below it uses it.
The old code walked every chain and rejected every cycle. The skip is new in
this patch, and it is needed so that the walk does not take exponential time,
but should it keep the effects that depend on the chain? For example the
chain of the skipped walk could be marked with the flags that the first walk
found below rec, by remembering whether a tail call was reachable below rec.
I did not find a later patch in the series that changes the skip.
[ ... ]
> @@ -5959,7 +6007,10 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
> /* sync tail_call_reachable with callee state on entry */
> tail_call_reachable = subprog[idx].has_tail_call;
>
> - frame = bpf_subprog_is_global(env, idx) ? 0 : frame + 1;
> + if (bpf_subprog_is_global(env, idx))
> + frame = 0;
> + else if (!subprog[idx].recursive)
> + frame++;
> if (frame >= MAX_CALL_FRAMES) {
Does the skip of a recursive subprog above also need to compare 'frame'?
It compares only depth and arena_depth. Zero stack frames add nothing to
depth, so chains of different lengths can have the same depth, and the chain
with more frames is skipped.
Example: main calls rec first and then P1. rec is recursive and calls Q1;
Q1..Q14 call each other in a line (each has stack depth 0). P1..P14 are a
second line of calls that ends with a call of rec.
First walk: main -> rec -> Q1..Q14 reaches frame 14, which is below
MAX_CALL_FRAMES (16 in this tree). Second walk: main -> P1..P14 -> rec, frame
is 14 at rec and depth is equal, so rec is skipped. The checked chains are 14
frames each, but the chain P1..P14, rec, Q1..Q14 has 29 native frames at run
time. The test 'if (frame >= MAX_CALL_FRAMES)' would have rejected that chain
before this patch.
Nothing else seems to limit it. The main pass does not follow it:
func_is_on_its_own() lets each of these static subprogs be verified on its
own, so the frames of the callers are not in the state and setup_func_entry()
does not see them. Recursive frames are bounded at run time by
BPF_RECURSION_STACK, but the weight of a recursive frame is its own frame
only, not the non-recursive frames between two recursive ones. With several
recursive subprogs in a row the skip can be used at each one, so the number of
native frames grows with the number of subprogs (BPF_MAX_SUBPROGS) and not
with MAX_CALL_FRAMES. The JIT frames of zero-stack subprogs are not counted
anywhere (round_up_stack_depth() is 0 for them with the JIT), so this is the
only bound on kernel stack use of such chains.
A chain through global subprogs is not bounded by MAX_CALL_FRAMES either
(frame restarts at 0), so the bound on static chains is the only one that is
left for the common case, and the patch removes it for chains that pass a
recursive subprog. Later commits in the series raise MAX_CALL_FRAMES to 32
and BPF_MAX_SUBPROGS to 1024, which makes it worse; I did not find a change to
the skip.
Should 'frame' be part of what the skip compares?
[ ... ]
> @@ -6022,6 +6074,111 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx,
> goto continue_func;
> }
>
> +/*
> + * Find the subprogs on cycles of the call graph. It is not known yet what callx
> + * calls: it may be any subprog whose address is taken. A subprog that is
> + * marked for no reason counts its frames for no reason, that is all.
> + */
> +static int mark_recursive_subprogs(struct bpf_verifier_env *env)
[ ... ]
> + if (bpf_pseudo_call(&insn[i]) || bpf_pseudo_func(&insn[i])) {
> + j = bpf_find_subprog(env, i + insn[i].imm + 1);
> + if (j < 0)
> + continue;
> + /* the helper that calls a callback doesn't expect it to unwind */
> + if (bpf_pseudo_func(&insn[i]) || !bpf_is_unwind_goto(&insn[i + 1]))
> + si[j].plain_call = true;
> + /* the helper that takes a callback calls it */
> + __set_bit(j, calls + s * words);
> + if (bpf_pseudo_func(&insn[i]))
> + __set_bit(j, taken);
> + }
[ ... ]
> + for (s = 0; s < cnt; s++) {
> + if (!test_bit(s, calls + s * words))
> + continue;
> + err = -EINVAL;
> + if (!s) {
> + verbose(env, "recursive call of the main program\n");
> + break;
> + }
> + /* it has no way to tell one run from another, see bpf_prog_begin_run() */
> + if (env->prog->type == BPF_PROG_TYPE_EXT) {
> + verbose(env, "recursive %s() in an extension\n", bpf_subprog_name(env, s));
> + break;
> + }
> + if (si[s].has_tail_call || si[s].has_ld_abs) {
> + verbose(env, "recursive %s() cannot have %s\n", bpf_subprog_name(env, s),
> + si[s].has_tail_call ? "tail calls" : "LD_ABS");
> + break;
> + }
> + if (!bpf_jit_supports_percpu_insn()) {
> + verbose(env, "JIT does not support recursive calls\n");
> + err = -EOPNOTSUPP;
> + break;
> + }
> + if (si[s].bail_insn < 0) {
> + verbose(env, "recursive %s() has no exit\n", bpf_subprog_name(env, s));
> + break;
> + }
> + si[s].recursive = true;
> + err = 0;
> + }
> + kvfree(calls);
> + return err;
> +}
Can an async callback that re-arms itself be rejected here?
Every BPF_PSEUDO_FUNC reference becomes a call edge s -> j, including
references whose target is an async callback (bpf_timer_set_callback(),
bpf_wq_set_callback(), task work). An async callback does not run on the
stack of the function that registered it, and check_max_stack_depth_subprog()
deliberately skips it (the 'subprog[sidx].is_async_cb ... continue' branch),
but this function has no such exception.
A callback that re-arms itself, directly or through a helper subprog, is put
on a cycle and gets si[s].recursive = true. For callx the edges are
over-approximated the same way: every subprog with callx 'calls' every
address-taken subprog, itself included.
The comment says that a subprog marked this way "counts its frames for no
reason, that is all", but the loop above rejects recursive subprogs:
- it fails with -EOPNOTSUPP ("JIT does not support recursive calls") when
!bpf_jit_supports_percpu_insn() (only x86, arm64, powerpc, riscv and
loongarch define it; s390x, mips and others do not)
- it fails with -EINVAL for BPF_PROG_TYPE_EXT, for has_tail_call/has_ld_abs,
and for a subprog without an exit insn
Such programs loaded before this patch, because sort_subprogs_topo() tolerated
pseudo_func cycles and the stack walk skipped async callbacks. The existing
selftest timer.c (loop_cb() -> rearm() -> bpf_timer_set_callback(...,
loop_cb)) is an example: loop_cb and rearm are marked recursive, so on a JIT
without percpu insn support the whole 'timer' skeleton fails to load with
EOPNOTSUPP (prog_tests/timer.c turns that into a skip, which hides it), and it
would be rejected for a freplace program.
Should async callbacks not add edges to 'calls' (they are separate roots for
the budget), or should the rejection conditions apply only to subprogs that
are on a cycle of synchronous calls?
[ ... ]
> @@ -6035,14 +6192,15 @@ static int check_max_stack_depth(struct bpf_verifier_env *env)
> return -ENOMEM;
>
> for (int i = 0; i < env->subprog_cnt; i++) {
> - if (si[i].has_tail_call) {
> - /* the stack in arena wouldn't be returned */
> - if (env->uses_arena_sp) {
> - verbose(env, "tail_calls are not allowed in programs that use R%d\n",
> - BPF_REG_ARENA_SP);
> - kvfree(dinfo);
> - return -EINVAL;
> - }
> + /* the stack in arena wouldn't be returned */
> + if (si[i].has_tail_call && env->uses_arena_sp) {
> + verbose(env, "tail_calls are not allowed in programs that use R%d\n",
> + BPF_REG_ARENA_SP);
> + kvfree(dinfo);
> + return -EINVAL;
> + }
> + /* the private stack has room for one frame of a subprog */
> + if (si[i].has_tail_call || si[i].recursive) {
> priv_stack_mode = NO_PRIV_STACK;
> break;
> }
Can a program with both a tail call and a recursive subprog overflow the
kernel stack?
check_max_stack_depth() now accepts such a program. The only change it makes
is to turn off the private stack. The one bound on the stack at a tail call
is still the 256 byte check in check_max_stack_depth_subprog():
if (idx && subprog[idx].has_tail_call && depth >= 256) {
That walk never adds the frames of recursive subprogs to depth:
if (!subprog[idx].recursive)
depth += subprog_depth;
Those frames are counted at run time instead. bpf_insert_recursion_checks()
lets them use up to BPF_RECURSION_STACK (4096) bytes for each program.
mark_recursive_subprogs() rejects a tail call only inside the recursive
subprog itself, and recursive_frame_size() even reserves room for
tail_call_reachable, so a tail call reached from a recursive subprog looks
meant to be supported.
Example: main() -> rec() (recursive, about 4 KB deep at run time) -> t() ->
bpf_tail_call(). The verifier sees only the frames of main() at t(), which is
well below 256, and accepts the program. A tail call removes only the frame
of t(). The frames of main() and of the whole recursion stay on the kernel
stack while the next program runs, which is the reason for the 256 limit.
Each program has its own percpu recursion counter (prog->aux->call_depth), so
a chain of tail calls through programs built this way adds about 4.3 KB per
program, up to MAX_TAIL_CALL_CNT times. Two programs that tail call each
other already exceed 16 KB (2 * 4096 + 33 * 256). That overflows the kernel
stack and hits the guard page, or corrupts memory without VMAP_STACK. The old
analysis (32 * 256 = 8 KB at worst) assumed that every frame below a tail call
is counted in depth.
Before this patch sort_subprogs_topo() rejected recursion, so this combination
could not occur. Should the tail call check account for the run time budget
of recursive frames, or should tail calls below a recursive subprog be
rejected? I did not find a later patch in the series that changes this.
Also in this loop: does the new break on a recursive subprog skip the
BPF_REG_ARENA_SP check for later subprogs?
The loop now stops at the first subprog that is recursive, not only at the
first one that has a tail call. The check
if (si[i].has_tail_call && env->uses_arena_sp)
runs only for the subprogs that the loop visits before the break. Before the
patch the loop stopped only at a subprog with has_tail_call, so the check
always ran for the first tail-calling subprog, and since env->uses_arena_sp is
program-wide that was enough.
Take a program with an arena stack where subprog 1 is recursive and subprog 2
does bpf_tail_call(). At i == 1 the arena check does not fire (no tail call
there), then 'si[1].recursive' breaks the loop. Subprog 2 is never examined,
so "tail_calls are not allowed in programs that use R%d" is never printed. No
other check catches this: mark_recursive_subprogs() rejects tail calls only in
the recursive subprog itself, and check_max_stack_depth_subprog() and
bpf_insert_arena_stack() have no tail call/arena check.
The program loads. Per the comments in fixups.c and arena.c, the main program
takes a stack in arena when it starts and returns it at exit. A tail call
skips that exit, so the per-cpu arena stack (cpu->busy / stacks->busy) stays
taken. Later runs on that cpu fall back to bpf_arena_stack_get() and leak
slots, until the program no longer runs on that cpu.
Could the arena check be done in a separate loop, or could the loop avoid
breaking before every subprog is checked?
[ ... ]
> @@ -8962,12 +9120,33 @@ static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_
> * }
> * bpf_iter_num_destroy(&it);
> */
> +/*
> + * The program asks for the next element after it was told that there is none. So it didn't
> + * leave the loops around the call, and the iterator is not what stops them.
> + */
> +static int iter_is_drained(struct bpf_verifier_env *env, int insn_idx)
> +{
> + struct bpf_loop *loop;
> + int h;
> +
> + for (h = bpf_loop_at_index(env, insn_idx); h >= 0; h = env->insn_aux_data[h].loop_header) {
> + loop = env->insn_aux_data[h].loop;
> + if (!loop->rt_bounded)
> + continue;
> + loop->rt_bounded = false;
> + env->restart = true;
> + }
> + return env->restart ? -EINVAL : 0;
> +}
> +
> static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
This isn't a bug, but is iter_is_drained() placed where the comment of
process_iter_next_call() should be?
iter_is_drained() is inserted between the long block comment that describes
process_iter_next_call() (the "Besides that, we also fork current state and
enqueue it ..." text and the DRAINED/ACTIVE state machine) and
process_iter_next_call() itself. The documentation now sits above an
unrelated helper that has a comment block of its own, and
process_iter_next_call() is left without the comment that explains it.
Could the helper be placed above that block, or the block moved down? The
placement is kept in the last commit of the series, so it is not fixed later.
[ ... ]
> @@ -11523,13 +11709,362 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
[ ... ]
> +int bpf_flow_return(struct bpf_verifier_env *env, struct bpf_verifier_state *caller,
> + const struct bpf_verifier_state *exit, int call_idx)
> +{
[ ... ]
> + /* What is taken is added at the end. What is released leaves no hole. */
> + for (i = 0; i < nheld && i < caller->acquired_refs; i++) {
> + id = held[i].id ? BPF_INHERITED_ID + i : 0;
> + if (caller->refs[i].id != id || caller->refs[i].type != held[i].type)
> + break;
> + caller->refs[i] = held[i];
> + kept++;
> + }
> + kfree(held);
> + /* What it means for the caller is found out in the context of the caller. */
> + if (kept != nheld || exit->active_rcu_locks < caller_rcu ||
> + exit->active_preempt_locks < caller_preempt) {
> + if (env->log.level & BPF_LOG_LEVEL2)
> + verbose(env, "func#%d releases what its caller holds at insn %d\n",
> + callee->subprogno, call_idx);
> + env->subprog_info[callee->subprogno].releases_inherited = true;
> + return 1;
> + }
In bpf_flow_return(), can this loop mistake a different lock for the one that
was held at the call?
The loop that decides whether a function verified on its own kept what its
caller held compares only the id and the type of each entry. For lock entries
(REF_TYPE_LOCK, REF_TYPE_RES_LOCK, REF_TYPE_RES_LOCK_IRQ) the identity of the
lock is (type, id, ptr); find_lock_state(), release_lock_state() and
process_spin_lock() all match on ptr too. Global-data locks always have id 0
(see the comment above check_reg_allocation_locked()), and inherited_id(state,
0) returns 0.
Calls to static subprogs are allowed with a lock held, and
func_is_on_its_own() does not exclude active_locks. Suppose the entry state
of the callee holds {LOCK, id 0, ptr A}, the callee releases it and then takes
another lock of the same type that also has id 0 but a different ptr B. Its
exit entry is then {LOCK, id 0, ptr B}.
The loop sees the same id (0) and the same type, counts the entry as kept, and
'caller->refs[i] = held[i]' sets refs[i].ptr back to A.
copy_reference_state(caller, exit) has already set active_lock_ptr to B.
After the return, refs[] says that the lock on allocation A is held, but the
lock the code actually holds is B, and active_lock_ptr disagrees with refs[].
check_reg_allocation_locked() checks only refs[] through find_lock_state(), so
after this return the verifier treats the bpf_list_head/bpf_rb_root fields of
allocation A as protected by a held lock, although that lock was released in
the callee. Before this patch the callee ran in the context of the caller and
refs[] stayed exact.
Should the loop also compare ptr for lock types, for example:
(held[i].type & REF_TYPE_LOCK_MASK) && caller->refs[i].ptr != held[i].ptr
so that the mismatch takes the existing 'releases what its caller holds'
fallback? The loop is unchanged at the end of the series.
[ ... ]
> @@ -11551,12 +12086,15 @@ static int check_static_func_call(struct bpf_verifier_env *env, int subprog,
> stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) + callee_incoming;
> if (stack_arg_cnt > caller_info->stack_arg_cnt)
> caller_info->stack_arg_cnt = stack_arg_cnt;
> -
> + /* btf_check_subprog_call() doesn't get to it when BTF is not reliable */
> err = check_outgoing_stack_args(env, caller, MAX_BPF_FUNC_REG_ARGS + callee_incoming,
> bpf_subprog_name(env, subprog), NULL, NULL);
> if (err)
> return err;
>
> + if (func_is_on_its_own(env, state, subprog))
> + return call_func_on_its_own(env, subprog, *insn_idx);
> +
Does this lose the RCU callback context of the caller?
func_is_on_its_own() lets a recursive subprog through even inside a callback:
if (state_in_callback(state) && !sub->recursive)
return false;
call_func_on_its_own() then builds a fresh entry state whose frame[0] is the
callee. It copies in_sleepable, the reference/lock/rcu counters and a generic
in_callback bool, but not the in_rcu_callback_fn flag of the caller frames.
That flag is set by set_map_elem_callback_state() for hash maps
(bpf_for_each_hash_elem() calls the callback under rcu_read_lock()) and by
set_find_vma_callback_state().
Inside the subprog the sleepable test is:
static inline bool in_sleepable_context(struct bpf_verifier_env *env)
{
return !in_rcu_cs(env) && !in_rcu_callback(env);
}
in_rcu_callback() scans only the frames of the current state, so it returns
false. In a sleepable program in_rcu_cs() is also false, because
active_rcu_locks is 0 (the RCU of the callback is implicit).
Example: a sleepable program does bpf_for_each_map_elem(&hash, cb, ...), and
cb calls a recursive rec(int n) that calls bpf_copy_from_user() or a
KF_SLEEPABLE kfunc. check_helper_call() tests 'fn->might_sleep &&
!in_sleepable_context(env)', the insn is not marked non_sleepable, and the
program loads.
At run time the helper can sleep inside the rcu_read_lock() of
bpf_for_each_hash_elem(). That is a sleeping-in-atomic bug, and the hash
element being walked can be freed while the program still uses it.
has_rcu_read_lock() and the checks of global calls (sub_aux->called[]) see the
same wrong context.
Before this patch recursion was rejected, and callees verified in the context
of the caller kept the callback frame, so in_rcu_callback() saw the flag.
Should call_func_on_its_own() carry this context into the entry state, for
example as a state-level flag that in_rcu_callback() also tests? I did not
find a later patch in the series that changes this.
[ ... ]
> @@ -12181,6 +12719,16 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
> return -EACCES;
> }
>
> + /* It's called any number of times. Nobody can release what it leaves behind. */
> + for (i = callee->entry_refs; i < state->acquired_refs; i++) {
> + if (state->refs[i].type != REF_TYPE_PTR)
> + continue;
> + verbose(env, "Unreleased reference id=%d alloc_insn=%d\n",
> + state->refs[i].id, state->refs[i].insn_idx);
> + bpf_diag_leak(env, state->refs[i].id, state->refs[i].insn_idx, *insn_idx);
> + return -EINVAL;
> + }
> +
> /* we are going to rely on register's precise value */
> err = mark_chain_precision(env, BPF_REG_0);
> if (err)
Can this reject callbacks that hand the reference to their caller?
The new check at callback exit rejects any synchronous callback that returns
while holding a REF_TYPE_PTR acquired after entry. The comment says nobody
can release it, but the caller can: the callback can store the reference
through its ctx pointer into the stack of the caller, and the caller releases
it after the helper returns. For example:
static int cb(u32 i, struct ctx *c) {
if (!c->p)
c->p = bpf_kfunc_call_test_acquire(&sl);
return 0;
}
... bpf_loop(n, cb, &c, 0); if (c.p) release(c.p);
Before this patch check_reference_leak() returned 0 for frame > 0, and the
callback was verified as a loop. On the NULL branch mark_ptr_or_null_regs()
drops the reference. On the non-NULL branch the guard skips the second
acquire. The states at the call site converge, and the caller releases the
pointer on every path, so the program loaded.
Now the first callback exit has refs[entry_refs] of type REF_TYPE_PTR and
fails with "Unreleased reference". The fixpoint does not seem to need this
check for soundness: those states cannot be joined since their refs differ,
and a real leak is still caught at the frame 0 exit, as for the cb2/leak_prog
test in cb_refs.c.
The commit message lists the rejections it introduces (the join that loses the
relation between scalars), but not this one. Is this rejection intended?
[ ... ]
> @@ -15115,14 +15663,59 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
[ ... ]
> +/* The registers in which the kfunc takes nothing but a known constant ("__k") */
> +u16 bpf_kfunc_const_arg_regs(struct bpf_verifier_env *env, struct bpf_insn *insn)
> +{
> + struct bpf_call_arg_meta meta;
> + const struct btf_param *args;
> + u32 i, slot, nargs;
> + u16 mask = 0;
> +
> + if (bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta) < 0)
> + return 0;
> + args = btf_params(meta.func_proto);
> + nargs = btf_type_vlen(meta.func_proto);
> + for (i = 0, slot = 0; i < nargs && slot < MAX_BPF_FUNC_REG_ARGS; i++) {
> + if (is_kfunc_arg_constant(meta.btf, &args[i]))
> + mask |= BIT(BPF_REG_1 + slot);
> + slot += btf_arg_slots(btf_type_skip_modifiers(meta.btf, args[i].type, NULL));
> + }
> + return mask;
> +}
Should this mask also cover '__szk' and the rdonly_buf_size/rdwr_buf_size
arguments?
bpf_kfunc_const_arg_regs() only reports "__k" arguments, but
check_kfunc_args() also requires a known constant for "__szk"
(ARG_CONST_MEM_SIZE -> process_const_arg(): "%s must be a known constant") and
for "rdonly_buf_size"/"rdwr_buf_size" (ARG_CONST_ALLOC_SIZE_OR_ZERO ->
process_const_alloc_mem_size(): "%s is not a const"); see
get_kfunc_arg_type().
The mask is the only input of flow_compute_need_const(FLOW_NEED_ARG) for kfunc
calls, and flow_merge() uses flow->need_const to refuse joins of states whose
needed registers hold different constants:
if (!widen && flow_needed_consts_differ(flow->need_const, &t->st, &s->st))
continue;
Without a bit for the __szk register, two states with different constant sizes
are joined at the block head of the call, and the joined range fails
process_const_arg(). This is the same compiler pattern that the comment above
flow_compute_need_const() mentions for bpf_rdonly_cast(): LLVM sinks two calls
that differ in one operand into one call with a phi.
Example, which loads before this commit because each path has its own constant
in R4:
if (skb->protocol == bpf_htons(ETH_P_IPV6))
p = bpf_dynptr_slice(&d, ETH_HLEN, buf, 40);
else
p = bpf_dynptr_slice(&d, ETH_HLEN, buf, 20);
Once the call is merged, the states {r4=20} and {r4=40} meet at the call.
flow_null_differs(), flow_unset_differs() and flow_join_makes_up() do not
apply, so bpf_states_join() gives r4=[20,40], and the verifier rejects the
program with "R4 must be a known constant". bpf_dynptr_slice() and
bpf_dynptr_slice_rdwr() take buffer__szk.
The helper variant bpf_helper_const_arg_regs() does cover
ARG_CONST_ALLOC_SIZE_OR_ZERO; the kfunc variant misses the kfunc arguments
that map to that type and to ARG_CONST_MEM_SIZE. Could it test
is_kfunc_arg_const_mem_size() and the rdonly_buf_size/rdwr_buf_size names the
way get_kfunc_arg_type() does? No later commit in the series changes this
function.
[ ... ]
> @@ -20274,6 +20879,9 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
> {
> struct bpf_func_state *cur_frame = cur_func(env);
>
> + if (!exception_exit && !env->cur_state->curframe && env->cur_state->flow_entry)
> + return exit_func_on_its_own(env);
> +
Can a speculative state reach this early return and lose its speculative flag?
With !env->bypass_spec_v1 (CAP_BPF without CAP_PERFMON, which is enough for
bpf2bpf calls), check_cond_jmp_op() pushes the dead side of every decided
branch with speculative = true through sanitize_speculative_path(). Inside a
function verified on its own, branches on constant arguments copied from the
caller are decided, so such states are common.
When one reaches the exit, the new early return calls exit_func_on_its_own()
-> bpf_flow_exit() without testing state->speculative:
- join.c refuses to join it with non-speculative exits
- states.c refuses to let it cover them
- so it stays in f->exits, and flow_apply_exit() applies it to every waiting
caller
bpf_flow_return() does not carry exit->speculative into the caller, so the
caller continues non-speculatively with values from a path that cannot happen.
Example: 'static int f(int x) { if (x) return 1; return 100; }' called as
f(1), followed by an access indexed by the result. The old code returned
through prepare_func_exit() with the state still speculative, and the bad
access got a nospec barrier (error_recoverable_with_nospec()). Now the same
access is checked as a real path and is a hard rejection.
This is not a safety hole, since the caller is checked more strictly, but
programs that loaded before can be rejected, and the speculative continuation
in the caller is no longer modeled as speculative. Should
exit_func_on_its_own() drop speculative states, or should bpf_flow_return()
propagate 'speculative' to the caller?
[ ... ]
> @@ -20490,6 +21098,24 @@ static int check_arena_sp_insn(struct bpf_verifier_env *env, struct bpf_insn *in
> return err ?: save_aux_ptr_type(env, PTR_TO_ARENA, false);
> }
>
> +/*
> + * NULL is dereferenced. In a program that has arena take it for a pointer to arena:
> + * the compiler may have moved the cast to where the number was not known to be 0 yet,
> + * and what check_loop_guard() sets to 0 may be a pointer that was cast in the loop.
> + */
> +static void null_to_arena(struct bpf_verifier_env *env, int regno)
> +{
> + struct bpf_reg_state *reg;
> +
> + if (!env->prog->aux->arena || regno >= BPF_REG_FP)
> + return;
> + reg = &cur_regs(env)[regno];
> + if (reg->type != SCALAR_VALUE || !tnum_equals_const(reg->var_off, 0))
> + return;
> + __mark_reg_const_zero(env, reg);
> + reg->type = PTR_TO_ARENA;
> +}
> +
> static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
> {
> int err;
> @@ -20499,6 +21125,11 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
> if (is_arena_sp_insn(insn))
> return check_arena_sp_insn(env, insn);
>
> + if (class == BPF_LDX)
> + null_to_arena(env, insn->src_reg);
> + else if (class == BPF_STX || class == BPF_ST)
> + null_to_arena(env, insn->dst_reg);
> +
Does this miss the load-acquire form?
The commit message says "in a prog that has an arena, a load or a store
through the scalar 0 is an access to arena", and the comment of
null_to_arena() gives as a reason the registers that check_loop_guard() sets
to 0 on the may_goto edge.
For BPF_STX | BPF_ATOMIC with imm == BPF_LOAD_ACQ the address is in src_reg,
not dst_reg: check_atomic_load() tests atomic_ptr_type_ok(env, insn->src_reg,
insn) and check_load_mem() accesses regs + insn->src_reg. null_to_arena() is
applied to dst_reg (the destination, which the load overwrites), so the
pointer register stays SCALAR_VALUE 0 and check_mem_access() rejects the
load-acquire with "invalid mem access 'scalar'". A plain BPF_LDX through the
same register is accepted.
Scenario: an arena pointer is set only in a loop body and is read with
smp_load_acquire() after the loop. The bpf_atomic.h of libarena lowers
smp_load_acquire() to __atomic_load_n(..., __ATOMIC_ACQUIRE), which is
BPF_LOAD_ACQ when __BPF_FEATURE_LOAD_ACQ_STORE_REL is set. When the loop gets
a guard at its header, check_loop_guard() zeroes the NOT_INIT live register on
the guard edge. The plain-load version of the program loads. The
load-acquire version is rejected, including programs whose loops the old
verifier followed to the end and which now get may_goto.
Should null_to_arena() be given src_reg when the insn is BPF_LOAD_ACQ?
[ ... ]
> @@ -20728,18 +21364,66 @@ static int maybe_clamp_scev_regs(struct bpf_verifier_env *env)
[ ... ]
> +int bpf_flow_loop_enter(struct bpf_verifier_env *env, bool *backedge, bool *drop)
> +{
[ ... ]
> @@ -20752,43 +21436,336 @@ static int handle_loop_entry_exit(struct bpf_verifier_env *env)
[ ... ]
> return err;
>
> - terminates = err == 1;
> - if (!terminates)
> + /*
> + * There is no summary of what the loop does to something that is live.
> + * Go through the iterations one by one if that is not too long.
> + */
> + if (err == 2 && flow_walk_factor(cur) * iters.max_header_count * loop->nr_insns <=
> + BPF_FLOW_MAX_WALK) {
> + entry->max_header_count = iters.max_header_count;
> + entry->terminates = true;
> + entry->counted = true;
> + entry->apart = true;
> + return 0;
> + }
Can this product wrap?
BPF_FLOW_MAX_WALK (100000) is meant to cap the insns that a counted walk
processes. Now that BPF_COMPLEXITY_LIMIT_INSNS is no longer enforced, this
cap and the flow.c state counts are what bound the verification time. The
product is computed in u64 and can wrap: flow_walk_factor() can reach about
1e5, iters.max_header_count can be up to U32_MAX - 2 (compute_max_iters()),
and loop->nr_insns can be up to the program length.
Inside one frame the nr_insns of the outer loop includes the inner loop, so
the product stays below 2^64. Across frames it does not: flow_walk_factor()
multiplies the loop stacks of all frames, and the frames of the callers stay
in the state when the callee is not on its own (for example when it gets a
pointer to the stack of the caller). For example:
- A caller loop of 3 insns (call; add; jlt) walked as counted with
max_header_count 32768 gives factor = 2^15.
- The callee has a loop with err == 2, max_header_count = 2^31 and
nr_insns = 2^18.
- 2^15 * 2^31 * 2^18 = 2^64 wraps to 0, which is <= BPF_FLOW_MAX_WALK, so
the loop is walked trip by trip for up to 2^31 trips of 262144 insns.
The walk ends only when env->total_states reaches 256 * len + 65536, after on
the order of 10^13 processed insns. That is hours to days of verifier CPU.
The task can be killed (signal_pending), and the loader needs CAP_BPF.
Could the bound be computed without the overflow, for example by comparing
iters.max_header_count against BPF_FLOW_MAX_WALK / (flow_walk_factor(cur) *
loop->nr_insns)?
[ ... ]
> +/*
> + * The loop that is not counted and has nothing that stops it at run time gets
> + * may_goto in front of this insn. When it is out of budget the program goes
> + * where may_goto goes with the registers and the stack as they are here.
> + */
> +static int check_loop_guard(struct bpf_verifier_env *env)
> +{
[ ... ]
> + branch = push_stack(env, env->insn_aux_data[env->insn_idx].guard_to, env->insn_idx, false);
> + if (IS_ERR(branch))
> + return PTR_ERR(branch);
> + env->head->parked = aux->guard_unwind;
> + aux->loop_guard = true;
> + if (!aux->guard_at_header)
> + return 0;
> + /*
> + * The program doesn't leave the loop from here. What it computes in the loop
> + * for the code that follows is not there. Nothing good comes out of a loop
> + * that is cut short anyway. Let it be 0.
> + */
> + regs = branch->frame[branch->curframe]->regs;
> + for (i = BPF_REG_0; i < BPF_REG_FP; i++)
> + if (regs[i].type == NOT_INIT &&
> + (env->insn_aux_data[aux->guard_to].live_regs_before & BIT(i)))
> + aux->guard_zero |= BIT(i);
> + for (i = BPF_REG_0; i < BPF_REG_FP; i++)
> + if (aux->guard_zero & BIT(i))
> + __mark_reg_const_zero(env, ®s[i]);
Can a guard branch that was pushed earlier be verified with guard_zero masks
that are out of date?
check_loop_guard() collects aux->guard_zero and aux->guard_zero_slots one
state at a time. It applies the bits collected so far only to the guard
branch of the state it is processing at that moment. bpf_insert_loop_guards()
in fixups.c then emits 'rX = 0' and '*(u64 *)(fp - off) = 0' on the may_goto
edge for every bit in the final value, and that code runs on every execution.
A state that reaches the guard insn before a later state adds a bit has its
guard branch pushed and verified with that register or slot unchanged (for
example, still a pointer). At run time the same edge writes 0 there. Nothing
re-verifies the earlier branch when guard_zero or guard_zero_slots grows.
The fixpoint processes states in a fixed order (flow_set_pending() sorts by
subprog rank and insn position, FIFO for ties), so two states that cannot be
joined at the guard insn can be handled in that order. In that case the code
after the loop is verified with a register or slot type that differs from the
value the patched program loads at run time.
Should the main pass re-verify the guard edges at that insn whenever the zero
masks grow? Alternatively the masks could be fixed before any guard branch is
delivered, for example by restarting the pass when they change.
[ ... ]
> @@ -22279,6 +23221,26 @@ static int do_check_common(struct bpf_verifier_env *env, int subprog,
> return ret;
> }
>
> +static int do_check_common(struct bpf_verifier_env *env, int subprog,
> + enum bpf_func_context context)
> +{
> + int insn_idx = env->insn_idx, tries = 0, ret;
> + u64 log_pos = env->log.end_pos;
> +
> + for (;;) {
> + env->restart = false;
> + ret = __do_check_common(env, subprog, context);
> + if (!ret || !env->restart)
> + return ret;
> + if (++tries == 8) {
> + verbose(env, "verification was started again too many times\n");
> + return ret;
> + }
> + bpf_vlog_reset(&env->log, log_pos);
> + env->insn_idx = insn_idx;
> + }
> +}
Can the pass finish without a restart although env->restart was set on a
speculative path?
do_check_common() starts the pass again only when __do_check_common() fails
and env->restart is set:
if (!ret || !env->restart)
return ret;
The restart requests come from iter_is_drained() (reached from
process_iter_next_call() through do_check_insn()) and from
flow_iters_are_remade() (reachable through bpf_flow_call() from
do_check_insn()). Both set env->restart, clear loop->rt_bounded and return
-EINVAL.
In do_check_block() an -EINVAL from do_check_insn() on a speculative state
counts as recoverable: error_recoverable_with_nospec() includes -EINVAL, so
the code sets nospec on the insn and goes to process_bpf_exit. When
env->bypass_spec_v1 is false, check_cond_jmp_op() pushes the branch that the
state does not take as a speculative path. That path can carry a drained
iterator back to bpf_iter_*_next(), so iter_is_drained() can run on it.
The pass then goes on with rt_bounded already cleared and finishes with ret ==
0, and do_check_common() returns 0 without starting again. check_loop_guard()
handled the states before the change as bounded: it returned early, verified
no guard edge and set no aux->loop_guard. States after the change set
aux->loop_guard and push guard branches. bpf_insert_loop_guards() then adds
may_goto, and its exit edge applies to all executions, including those that
match states whose guard edge was never verified.
Should the pass start again whenever env->restart is set, whether or not it
succeeded? Another option is to not treat the -EINVAL as recoverable with
nospec once env->restart is set.
[ ... ]
> @@ -22364,6 +23326,37 @@ static int do_check_main(struct bpf_verifier_env *env)
> return ret;
> }
>
> +/*
> + * What it took to verify a static subprog on its own is not in insns_total of its callers.
> + * Add it, once per caller. The main prog and global subprogs have all of it already.
> + */
> +static void add_insns_of_callees(struct bpf_verifier_env *env)
> +{
> + struct bpf_subprog_info *si = env->subprog_info;
> + struct bpf_insn *insn = env->prog->insnsi;
> + int *added, i, j, s, c;
> +
> + added = kvcalloc(env->subprog_cnt, sizeof(*added), GFP_KERNEL_ACCOUNT);
> + if (!added)
> + return;
> + /* callees first */
> + for (i = 0; i < env->subprog_cnt; i++) {
> + s = env->subprog_topo_order[i];
> + if (!s || bpf_subprog_is_global(env, s))
> + continue;
> + for (j = si[s].start; j < si[s + 1].start; j++) {
> + if (!bpf_pseudo_call(&insn[j]))
> + continue;
> + c = bpf_find_subprog(env, j + insn[j].imm + 1);
> + if (c <= 0 || c == s || !si[c].on_its_own || added[c] == s)
> + continue;
> + added[c] = s;
> + si[s].insns_total += si[c].insns_total;
> + }
> + }
> + kvfree(added);
> +}
Does this add the right total for a callee that is verified in both ways?
The comment says that only what was verified on its own is missing from the
insns_total of the callers, and the function adds si[c].insns_total, but that
total is not only the on-its-own part. account_processed_insns() adds the
subtotal of every frame of c to subprog_info[c].insns_total, in-context frames
included, and those insns were already propagated into the total of the caller
(caller->insns_subtotal += insns).
(1) Double count: if s calls c in the context of its own frame at one site
(for example c receives a PTR_TO_STACK argument, so func_is_on_its_own() is
false) and on its own at another, c.insns_total holds both parts, and
s.insns_total gets the in-context part twice.
(2) Omission: the gate is si[c].on_its_own, so a callee c that is only ever
verified in context is skipped even though its own on-its-own callees were
added to c.insns_total. With q -> p(&local) -> c(scalars), p.insns_total is
p.self + c.total but q.insns_total is q.self + p.self, so a caller reports
less than its callee.
Only the BPF_LOG_STATS output ('subprog N ... insns_total') is affected, not
verification, but the output differs from the per-chain totals that
verifier_subprog_insn_stats.c documents. The sum can also wrap in a u32 for
deep call DAGs.
Should the callee total added for a callee that is not on its own be its full
total minus what the caller frame already accumulated, or should the totals be
kept per category?
[ ... ]
> @@ -24397,10 +25398,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
>
> ret = do_check_main(env);
> ret = ret ?: do_check_subprogs(env);
> -
> - /* reject recursion through the callx edges found by the main pass */
> - if (ret == 0 && env->callx_edges)
> - ret = sort_subprogs_topo(env);
> + /* while the calls are what they are in the program */
> + if (env->log.level & BPF_LOG_STATS)
> + add_insns_of_callees(env);
Are the comments about the repeated sort still correct?
This hunk removes the second sort_subprogs_topo() call from bpf_check() (the
old "reject recursion through the callx edges found by the main pass").
sort_subprogs_topo() now has one call site, which runs before the main pass.
env->callx_edges is only allocated by record_callx_edge() during the main
pass, so it is always NULL there. next_callx_callee() then always returns -1,
and the bpf_is_callx() branch in sort_subprogs_topo() never finds a callee.
Two comments are now wrong. The header comment of sort_subprogs_topo() still
says:
The callees of callx are not known before the main verification pass.
When callx is used the sort is repeated after it with the recorded callx
edges added to the call graph to reject recursion through indirect calls.
and the comment of record_callx_edge() still says that the edges are recorded
for the checks that follow the main pass: "absence of recursion and the
maximum stack depth". Recursion through callx is now accepted and checked at
run time (mark_recursive_subprogs()), and only check_max_stack_depth_subprog()
still reads the callx edges.
Neither comment is changed later in the series. Could the comments be
updated, and the dead callx branch in sort_subprogs_topo() dropped?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 54/81] bpf: Allow 24 arguments of static subprogs
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (52 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 53/81] bpf: Verify programs " Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 55/81] selftests/bpf: Test a static subprog with 24 arguments Alexei Starovoitov
` (28 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A BPF function takes 12 arguments at most: 5 in registers and 7 in stack
slots. Rust needs more, because it passes a slice or a trait object in two
slots. For a struct with 5 fields derive(Debug) calls
core::fmt::Formatter::debug_struct_field5_finish(), which takes up to
23 slots.
The limit of 12 is for calls in which the kernel takes part. Only BPF code
calls a static subprog. Thus a static subprog can have a different limit:
MAX_BPF_STATIC_FUNC_ARGS.
callee arguments
------------------- ----------------------------------------
kfunc, trampoline 12, no change
global subprog 5, no change
static subprog 12 -> 24: 5 registers and 19 stack slots
The verifier counts the stack arguments of a static subprog from its code,
and JITs get the number of stack slots from the verifier. Thus JITs do not
change.
BTF of a static subprog can now show more than 12 parameters. Before this
patch that was an error. Now the verifier marks such BTF as unreliable and
continues.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 9 ++++++++-
include/linux/bpf_verifier.h | 8 ++++----
kernel/bpf/backtrack.c | 2 +-
kernel/bpf/btf.c | 5 ++++-
kernel/bpf/verifier.c | 4 ++--
5 files changed, 19 insertions(+), 9 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 2ff1a2c97328..d8cd7c63f157 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1035,13 +1035,20 @@ static_assert(__BPF_RET_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
/* The longest tracepoint has 12 args.
* See include/trace/bpf_probe.h
*
- * Also reuse this macro for maximum number of arguments a BPF function
+ * Also reuse this macro for maximum number of arguments a global BPF function
* or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
* stack arg slots. The JIT may map some stack arg slots to registers based
* on the native calling convention (e.g., arg 6 to R9 on x86-64).
*/
#define MAX_BPF_FUNC_ARGS 12
+/*
+ * A static BPF function is called by BPF code only, with what the compiler
+ * makes of its arguments. A slice or a trait object of Rust takes two slots. The
+ * function that derive(Debug) calls for a struct with five fields takes 23.
+ */
+#define MAX_BPF_STATIC_FUNC_ARGS 24
+
/* The maximum number of arguments passed through registers
* a single function may have.
*/
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 41ffe9b694a6..c0227e74899b 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -499,7 +499,7 @@ static_assert(MAX_CALL_FRAMES <= (1 << 4));
static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
/* Maximum number of bpf_reg_state objects that can exist at once */
-#define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
+#define MAX_STACK_ARG_SLOTS (MAX_BPF_STATIC_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK_SLOTS + MAX_STACK_ARG_SLOTS) * \
MAX_CALL_FRAMES)
@@ -1061,8 +1061,8 @@ struct bpf_subprog_info {
bool bail_unwinds: 1;
/* some call of it is not followed by unwind_goto */
bool plain_call: 1;
- u8 code_stack_args:4;
- u8 arg_slot_cnt:4;
+ u8 code_stack_args;
+ u8 arg_slot_cnt;
enum priv_stack_mode priv_stack_mode;
struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS];
@@ -1089,7 +1089,7 @@ struct backtrack_state {
u32 frame;
u32 reg_masks[MAX_CALL_FRAMES];
unsigned long stack_masks[MAX_CALL_FRAMES][BITS_TO_LONGS(MAX_BPF_STACK_SLOTS)];
- u8 stack_arg_masks[MAX_CALL_FRAMES];
+ u32 stack_arg_masks[MAX_CALL_FRAMES];
};
struct bpf_id_pair {
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
index 98539c7e42c5..a42b8b830b0b 100644
--- a/kernel/bpf/backtrack.c
+++ b/kernel/bpf/backtrack.c
@@ -226,7 +226,7 @@ static inline unsigned long *bt_stack_mask(struct backtrack_state *bt)
return bt_frame_stack_mask(bt, bt->frame);
}
-static inline u8 bt_stack_arg_mask(struct backtrack_state *bt)
+static inline u32 bt_stack_arg_mask(struct backtrack_state *bt)
{
return bt->stack_arg_masks[bt->frame];
}
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index 274ef78b6e1c..955bb65a71f6 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -8819,12 +8819,15 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog)
}
args = (const struct btf_param *)(t + 1);
nargs = btf_type_vlen(t);
- sub->arg_slot_cnt = nargs;
if (nargs > MAX_BPF_FUNC_ARGS) {
+ /* what a static function takes is seen in its code */
+ if (!is_global)
+ return -EINVAL;
bpf_log(log, "kernel supports at most %d parameters, function %s has %d\n",
MAX_BPF_FUNC_ARGS, tname, nargs);
return -EFAULT;
}
+ sub->arg_slot_cnt = nargs;
err = btf_check_arg_slots(log, tname, is_global, nargs, sub);
if (err)
return err;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 0c1de0f24b85..50e24237a6b7 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -4625,7 +4625,7 @@ static void end_stack_arg_loads(struct bpf_verifier_env *env, struct bpf_func_st
static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state,
int off, struct bpf_reg_state *value_reg)
{
- int max_stack_arg_regs = MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS;
+ int max_stack_arg_regs = MAX_STACK_ARG_SLOTS;
struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno];
int spi = -off / BPF_REG_SIZE - 1;
struct bpf_reg_state *arg;
@@ -15836,7 +15836,7 @@ static void prepare_subprog_args(struct bpf_verifier_env *env)
j++;
}
sub->stack_args_from_code = true;
- sub->code_stack_args = min(cnt, MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS);
+ sub->code_stack_args = min(cnt, MAX_STACK_ARG_SLOTS);
/* incoming + max outgoing. Calls add the latter. */
sub->stack_arg_cnt = sub->code_stack_args;
if (env->prog->aux->func_info_aux)
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 54/81] bpf: Allow 24 arguments of static subprogs
2026-10-26 2:59 ` [PATCH bpf 54/81] bpf: Allow 24 arguments of static subprogs Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 5174 bytes --]
> bpf: Allow 24 arguments of static subprogs
TL;DR:
- BPF_MAX_ARG_IDS (32) may be too small for 24 args; dropped ids could lose a
returned pointer's parent link, allowing writes to a ringbuf record after
submit
- MAX_BPF_FUNC_ARGS comment inaccurate for global functions
> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> index 2ff1a2c97328..d8cd7c63f157 100644
> --- a/include/linux/bpf.h
> +++ b/include/linux/bpf.h
> @@ -1035,13 +1035,20 @@ static_assert(__BPF_RET_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
> /* The longest tracepoint has 12 args.
> * See include/trace/bpf_probe.h
> *
> - * Also reuse this macro for maximum number of arguments a BPF function
> + * Also reuse this macro for maximum number of arguments a global BPF function
> * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via
> * stack arg slots. The JIT may map some stack arg slots to registers based
> * on the native calling convention (e.g., arg 6 to R9 on x86-64).
> */
> #define MAX_BPF_FUNC_ARGS 12
The updated comment says a global BPF function can have up to
MAX_BPF_FUNC_ARGS arguments, with args 6-12 passed via stack arg slots.
Is that accurate?
btf_check_arg_slots() rejects any global function with more than 5 slots:
if (is_global) {
bpf_log(log,
"global function %s() needs %d > %d argument slots, "
"stack args not supported\n", ...);
return -EINVAL;
}
The commit message table also says "global subprog 5, no change".
MAX_BPF_FUNC_ARGS also remains the limit for the BTF-described arguments of
static subprogs (bpf_subprog_info.args[MAX_BPF_FUNC_ARGS] and the slot checks
in btf_prepare_func_args()).
Could the comment say that kfuncs and trampolines get 12, global subprogs
get 5, and static subprogs get MAX_BPF_STATIC_FUNC_ARGS?
[ ... ]
> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
> index 41ffe9b694a6..c0227e74899b 100644
> --- a/include/linux/bpf_verifier.h
> +++ b/include/linux/bpf_verifier.h
> @@ -499,7 +499,7 @@ static_assert(MAX_CALL_FRAMES <= (1 << 4));
> static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
>
> /* Maximum number of bpf_reg_state objects that can exist at once */
> -#define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
> +#define MAX_STACK_ARG_SLOTS (MAX_BPF_STATIC_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
> #define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK_SLOTS + MAX_STACK_ARG_SLOTS) * \
> MAX_CALL_FRAMES)
With MAX_STACK_ARG_SLOTS going from 7 to 19, can the id renaming for
functions verified on their own run out of room and lose the link between a
returned pointer and its parent object?
The previous patch in the series (0535bc235aed "bpf: Verify programs as a
fixpoint over the control flow graph") added:
#define BPF_MAX_ARG_IDS 32
static u32 add_arg_id(const struct bpf_verifier_state *state, u32 *ids,
u32 n, u32 id)
{
...
if (!id || n == BPF_MAX_ARG_IDS || inherited_id(state, id) != id)
return n;
collect_arg_ids() adds up to two ids, id and parent_id, for each live R1-R5
and for each of the bpf_in_stack_arg_cnt(sub) stack args.
With 5 + 7 arguments that was at most 24 ids, so the cap of 32 was never
hit. With 5 + 19 it can be 48, and add_arg_id() silently drops everything
after the 32nd id.
For a dropped id, callee_id() in call_func_on_its_own() does not find it, so
the callee's copy keeps the caller's raw id. At return, bpf_flow_return()
rebuilds idmap from the same truncated ids[] list, and then:
reg->id = fresh_id(env, idmap, reg->id);
reg->parent_id = fresh_id(env, idmap, reg->parent_id);
gives R0 a brand-new id instead of mapping it back to the caller's id.
For parent_id that looks unsafe, because release invalidates children by
matching parent_id (in release_reference(), "if (reg->parent_id == id)
idstack_push(...)").
Consider a caller that holds a ringbuf dynptr D and has a slice S from
bpf_dynptr_data(&D), so S.parent_id == D.id. D.id is not an acquired-ref
id, so add_arg_id() does not skip it as inherited.
The caller then calls a static subprog with about 22 arguments.
func_is_on_its_own() rejects only PTR_TO_STACK and CONST_PTR_TO_DYNPTR in
stack args, so the arguments can be 16 maybe-null slices from 16 local
dynptrs (id + parent_id each, 32 ids), followed by S as stack arg 17.
If the subprog returns S, R0.parent_id in the caller is a fresh id. Then
bpf_ringbuf_submit_dynptr(&D) does not invalidate R0, and the program could
write to the ringbuf record after it was submitted.
This cannot happen with 12 arguments, and nothing later in the series
(fcfc283e9ac3..51224d16c11b) changes BPF_MAX_ARG_IDS or add_arg_id().
Should BPF_MAX_ARG_IDS be 2 * MAX_BPF_STATIC_FUNC_ARGS, or should
func_is_on_its_own() refuse the call when collect_arg_ids() hits the cap?
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 55/81] selftests/bpf: Test a static subprog with 24 arguments
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (53 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 54/81] bpf: Allow 24 arguments of static subprogs Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 56/81] bpf: Bump MAX_CALL_FRAMES to 32 Alexei Starovoitov
` (27 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A prog passes 1 .. 24 to a static subprog, in r1 - r5 and in all 19 stack
slots. The subprog adds them up. The test runs the prog and expects 300.
Thus it also shows that the JIT has room for each slot.
The verifier must reject a store to slot 20:
stack arg write offset -160 exceeds max 19 stack args
The subprog is in asm and its prototype has no parameters. Thus no test
covers BTF with more than 12 parameters.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_stack_arg.c | 95 ++++++++++++++++++-
1 file changed, 92 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
index 5062c5bab5f0..42c241ebe796 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_arg.c
@@ -359,10 +359,99 @@ __naked void stack_arg_shared_store(void)
);
}
+/* r1 - r5 and all 19 slots. There is no BTF for it: what it takes is seen in its code. */
+__naked __noinline __used
+static unsigned long subprog_24args(void)
+{
+ asm volatile (
+ "r0 = r1;"
+ "r0 += r2;"
+ "r0 += r3;"
+ "r0 += r4;"
+ "r0 += r5;"
+ "r1 = *(u64 *)(r11 + 8);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 16);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 24);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 32);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 40);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 48);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 56);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 64);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 72);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 80);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 88);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 96);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 104);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 112);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 120);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 128);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 136);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 144);"
+ "r0 += r1;"
+ "r1 = *(u64 *)(r11 + 152);"
+ "r0 += r1;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("tc")
+__description("stack_arg: static subprog with 24 args")
+__success __retval(300)
+__naked void stack_arg_24args(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "r2 = 2;"
+ "r3 = 3;"
+ "r4 = 4;"
+ "r5 = 5;"
+ "*(u64 *)(r11 - 152) = 24;"
+ "*(u64 *)(r11 - 144) = 23;"
+ "*(u64 *)(r11 - 136) = 22;"
+ "*(u64 *)(r11 - 128) = 21;"
+ "*(u64 *)(r11 - 120) = 20;"
+ "*(u64 *)(r11 - 112) = 19;"
+ "*(u64 *)(r11 - 104) = 18;"
+ "*(u64 *)(r11 - 96) = 17;"
+ "*(u64 *)(r11 - 88) = 16;"
+ "*(u64 *)(r11 - 80) = 15;"
+ "*(u64 *)(r11 - 72) = 14;"
+ "*(u64 *)(r11 - 64) = 13;"
+ "*(u64 *)(r11 - 56) = 12;"
+ "*(u64 *)(r11 - 48) = 11;"
+ "*(u64 *)(r11 - 40) = 10;"
+ "*(u64 *)(r11 - 32) = 9;"
+ "*(u64 *)(r11 - 24) = 8;"
+ "*(u64 *)(r11 - 16) = 7;"
+ "*(u64 *)(r11 - 8) = 6;"
+ "call subprog_24args;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
SEC("tc")
__description("stack_arg: write beyond max outgoing depth")
__failure
-__msg("stack arg write offset -80 exceeds max 7 stack args")
+__msg("stack arg write offset -160 exceeds max 19 stack args")
__naked void stack_arg_write_beyond_max(void)
{
asm volatile (
@@ -371,8 +460,8 @@ __naked void stack_arg_write_beyond_max(void)
"r3 = 3;"
"r4 = 4;"
"r5 = 5;"
- /* Write to offset -80, way beyond any callee's needs */
- "*(u64 *)(r11 - 80) = 99;"
+ /* Write to offset -160, way beyond any callee's needs */
+ "*(u64 *)(r11 - 160) = 99;"
"*(u64 *)(r11 - 16) = 20;"
"*(u64 *)(r11 - 8) = 10;"
"call subprog_7args;"
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 56/81] bpf: Bump MAX_CALL_FRAMES to 32
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (54 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 55/81] selftests/bpf: Test a static subprog with 24 arguments Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 57/81] selftests/bpf: Test 32 and 33 call frames Alexei Starovoitov
` (26 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Progs that use Rust crates nest calls deeply and hit the limit of
16 frames. Bump the limit to 32.
A frame number now needs 5 bits. Widen it in bpf_jmp_history_entry and in
the packed linked registers. There the index of a register or a stack slot
loses one bit. The static_assert next to it shows that the index fits.
Widen the mask of frames in liveness.c to 32 bits.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 8 ++++----
kernel/bpf/liveness.c | 4 ++--
kernel/bpf/verifier.c | 4 ++--
3 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index c0227e74899b..6139e889c86a 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -458,7 +458,7 @@ struct bpf_func_state {
struct loop_stack_entry *loop_stack;
};
-#define MAX_CALL_FRAMES 16
+#define MAX_CALL_FRAMES 32
/* instruction history flags, used in bpf_jmp_history_entry.flags field.
* Frame number and SPI are stored in dedicated fields of bpf_jmp_history_entry.
@@ -478,10 +478,10 @@ enum {
struct bpf_jmp_history_entry {
/* insn idx can't be bigger than 1 million */
u32 idx : 20;
- u32 frame : 4; /* stack access frame number */
+ u32 frame : 5; /* stack access frame number */
/* special INSN_F_xxx flags */
u32 flags : 4;
- u32 : 4;
+ u32 : 3;
u32 prev_idx : 20;
u32 spi : 12; /* stack slot index */
/*
@@ -495,7 +495,7 @@ struct bpf_jmp_history_entry {
u8 linked_regs_cnt;
};
-static_assert(MAX_CALL_FRAMES <= (1 << 4));
+static_assert(MAX_CALL_FRAMES <= (1 << 5));
static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
/* Maximum number of bpf_reg_state objects that can exist at once */
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index 985cd39acf3e..8c7629c99400 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -814,7 +814,7 @@ static int print_instances(struct bpf_verifier_env *env)
struct arg_track {
union {
s16 off[MAX_ARG_OFFSETS]; /* byte offsets; off_cnt says how many */
- u16 mask; /* arg bitmask when arg == ARG_IMPRECISE */
+ u32 mask; /* arg bitmask when arg == ARG_IMPRECISE */
};
s8 frame; /* absolute frame index, or enum arg_track_state */
s8 off_cnt; /* 0 = offset-imprecise, 1-4 = # of precise offsets */
@@ -1175,7 +1175,7 @@ static struct arg_track fill_from_stack(struct bpf_insn *insn,
int depth)
{
struct arg_track imp = {
- .mask = (1u << (depth + 1)) - 1,
+ .mask = GENMASK(depth, 0),
.frame = ARG_IMPRECISE
};
struct arg_track result = { .frame = ARG_NONE };
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 50e24237a6b7..e55ba1db2e1f 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3654,8 +3654,8 @@ static void mark_non_stack_access(struct bpf_verifier_env *env, int idx)
}
/* Layout of one packed linked register in the jump history, see linked_regs_pack() */
-#define LR_FRAMENO_BITS 4
-#define LR_INDEX_BITS 11
+#define LR_FRAMENO_BITS 5
+#define LR_INDEX_BITS 10
#define LR_FRAMENO_MASK ((1u << LR_FRAMENO_BITS) - 1)
#define LR_IS_REG BIT(LR_FRAMENO_BITS)
#define LR_INDEX_OFF (LR_FRAMENO_BITS + 1)
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 56/81] bpf: Bump MAX_CALL_FRAMES to 32
2026-10-26 2:59 ` [PATCH bpf 56/81] bpf: Bump MAX_CALL_FRAMES to 32 Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 5817 bytes --]
> bpf: Bump MAX_CALL_FRAMES to 32
TL;DR:
- Possible kernel stack overflow at program load: liveness analyze_subprog()
recursion now reaches 32 levels (estimates, unmeasured)
- Stale packed-layout comments (nit)
> diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
> index c0227e74899b..6139e889c86a 100644
> --- a/include/linux/bpf_verifier.h
> +++ b/include/linux/bpf_verifier.h
> @@ -458,7 +458,7 @@ struct bpf_func_state {
> struct loop_stack_entry *loop_stack;
> };
>
> -#define MAX_CALL_FRAMES 16
> +#define MAX_CALL_FRAMES 32
>
> /* instruction history flags, used in bpf_jmp_history_entry.flags field.
> * Frame number and SPI are stored in dedicated fields of bpf_jmp_history_entry.
Can this bump overflow the kernel stack while the program is being loaded?
analyze_subprog() in kernel/bpf/liveness.c calls itself once for each static
callee (or callback) that receives a frame-pointer-derived argument. Its
only depth limit is:
if (depth == MAX_CALL_FRAMES - 1)
return -EINVAL;
so the deepest recursion now goes from 16 to 32 nested calls. Each level
holds this array, whose address is passed to the recursive call:
struct arg_track callee_args[MAX_AT_TRACK_REGS] = {};
MAX_AT_TRACK_REGS is MAX_BPF_REG + MAX_STACK_ARG_SLOTS = 11 + (24 - 5) = 30.
With the u32 mask change in this patch, sizeof(struct arg_track) grows from
10 to 12 bytes (4-byte alignment plus 2 bytes of tail padding), so
callee_args is 360 bytes per level. 32 levels need 11,520 bytes for this
array alone, compared to 16 * 300 = 4,800 bytes before the patch.
That lower bound does not count saved registers, return addresses, the
64-byte acc bitmap in merge_may_write() (static with one caller, so probably
inlined), the bpf() syscall path above bpf_compute_subprog_arg_access()
(which has its own u32 callsites[MAX_CALL_FRAMES], 128 bytes), or the frame
of compute_subprog_args() at the deepest level. That frame holds
at_out[MAX_AT_TRACK_REGS] plus at_stack_arg_entry[MAX_STACK_ARG_SLOTS], 588
bytes, and its kvmalloc() calls can enter reclaim.
On 32-bit configs with 8KB kernel stacks (arm and x86_32 without KASAN),
11.5KB is larger than THREAD_SIZE. On x86-64 and arm64 with 16KB stacks, at
roughly 450-500 bytes per level this comes to about 14-16KB before reaching
the leaf, which leaves little or no headroom. These per-frame numbers are
estimates, not measurements.
The call path is:
bpf_check()
bpf_compute_live_registers()
bpf_compute_subprog_arg_access()
analyze_subprog() <- recurses up to 32 levels
This runs before do_check_main() and check_max_stack_depth(), and
sort_subprogs_topo() only rejects recursion, so nothing earlier limits call
depth. A chain of 32 static subprogs that each forward a stack pointer
(for example r1 = r10 - 8 passed down), loaded by a CAP_BPF loader or a BPF
token holder, reaches 32 levels. That is the Rust use case named in the
commit message. It also happens on configs that reject the program later,
such as the interpreter, where the 32-byte minimum frame limits
check_max_stack_depth() to 16 frames.
The result would be a kernel stack overflow during program load (a guard
page oops/panic with CONFIG_VMAP_STACK, silent corruption without it). The
verifier otherwise avoids recursion (see the "non-recursive
depth-first-search" comment in bpf_check_cfg()), and the commit message
does not mention verifier stack use.
The selftest added later in the series (08f1453d610f) passes only skb
through its 32-frame chain, so has_fp_args() is false and this path is not
exercised. analyze_subprog() is unchanged up to 51224d16c11b, and no later
commit in that range addresses this.
Should the walk be made iterative with a heap-allocated stack, should
callee_args move off the kernel stack (for example as per-depth scratch in
env), or should the liveness recursion limit be kept independent of
MAX_CALL_FRAMES?
[ ... ]
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 50e24237a6b7..e55ba1db2e1f 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -3654,8 +3654,8 @@ static void mark_non_stack_access(struct bpf_verifier_env *env, int idx)
> }
>
> /* Layout of one packed linked register in the jump history, see linked_regs_pack() */
> -#define LR_FRAMENO_BITS 4
> -#define LR_INDEX_BITS 11
> +#define LR_FRAMENO_BITS 5
> +#define LR_INDEX_BITS 10
> #define LR_FRAMENO_MASK ((1u << LR_FRAMENO_BITS) - 1)
> #define LR_IS_REG BIT(LR_FRAMENO_BITS)
> #define LR_INDEX_OFF (LR_FRAMENO_BITS + 1)
This isn't a bug, but the comments describing this packed layout still show
the old widths. The comment above linked_regs_pack() says:
* Pack linked registers for a jump history entry, one u16 each:
* - 4 bits frameno
* - 1 bit is_reg
* - 11 bits register or stack slot index
and the linked_regs comment in struct bpf_jmp_history_entry in
include/linux/bpf_verifier.h says:
Each is packed as 4 bits of frame number, one bit telling a register
from a stack slot and 11 bits of register or slot index, see
linked_regs_pack().
Should both be updated to 5 bits of frame number and 10 bits of index to
match LR_FRAMENO_BITS and LR_INDEX_BITS, and the commit message's own "the
index of a register or a stack slot loses one bit"?
There is no functional impact, since bpf_push_jmp_history() returns 0 before
it records anything in this tree. Both comments are deleted later in the
series by 1dc2b8088ff9 ("bpf: Remove precision tracking"), so this only
matters if this patch is taken without that one.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 57/81] selftests/bpf: Test 32 and 33 call frames
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (55 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 56/81] bpf: Bump MAX_CALL_FRAMES to 32 Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 58/81] bpf: Bump BPF_MAX_SUBPROGS to 1024 Alexei Starovoitov
` (25 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
test_global_func3 has a chain of static subprogs that is one frame longer
than the limit. The limit is 32 now. Make the chain longer, and add a prog
that has exactly 32 frames.
prog frames result
----------------------- -------------------------- -----------------
global_func3_32_frames main prog, f31 .. f1: 32 loads
global_func3 main prog, f32 .. f1: 33 rejected
The message for 33 frames shows 32:
the call stack of 32 frames
Two places in the verifier check the limit. The main pass counts the frames
in a state. But it verifies these subprogs for the states at their entry,
without the frames of the callers. Thus it does not see the chain.
check_max_stack_depth() finds the chain, and it does not count the main
prog.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/test_global_func3.c | 107 +++++++++++++++++-
1 file changed, 105 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/bpf/progs/test_global_func3.c b/tools/testing/selftests/bpf/progs/test_global_func3.c
index b66abb350fb0..3b816afaa278 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func3.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func3.c
@@ -101,9 +101,112 @@ int f16(struct __sk_buff *skb)
return f15(skb);
}
+static __attribute__ ((noinline))
+int f17(struct __sk_buff *skb)
+{
+ return f16(skb);
+}
+
+static __attribute__ ((noinline))
+int f18(struct __sk_buff *skb)
+{
+ return f17(skb);
+}
+
+static __attribute__ ((noinline))
+int f19(struct __sk_buff *skb)
+{
+ return f18(skb);
+}
+
+static __attribute__ ((noinline))
+int f20(struct __sk_buff *skb)
+{
+ return f19(skb);
+}
+
+static __attribute__ ((noinline))
+int f21(struct __sk_buff *skb)
+{
+ return f20(skb);
+}
+
+static __attribute__ ((noinline))
+int f22(struct __sk_buff *skb)
+{
+ return f21(skb);
+}
+
+static __attribute__ ((noinline))
+int f23(struct __sk_buff *skb)
+{
+ return f22(skb);
+}
+
+static __attribute__ ((noinline))
+int f24(struct __sk_buff *skb)
+{
+ return f23(skb);
+}
+
+static __attribute__ ((noinline))
+int f25(struct __sk_buff *skb)
+{
+ return f24(skb);
+}
+
+static __attribute__ ((noinline))
+int f26(struct __sk_buff *skb)
+{
+ return f25(skb);
+}
+
+static __attribute__ ((noinline))
+int f27(struct __sk_buff *skb)
+{
+ return f26(skb);
+}
+
+static __attribute__ ((noinline))
+int f28(struct __sk_buff *skb)
+{
+ return f27(skb);
+}
+
+static __attribute__ ((noinline))
+int f29(struct __sk_buff *skb)
+{
+ return f28(skb);
+}
+
+static __attribute__ ((noinline))
+int f30(struct __sk_buff *skb)
+{
+ return f29(skb);
+}
+
+static __attribute__ ((noinline))
+int f31(struct __sk_buff *skb)
+{
+ return f30(skb);
+}
+
+static __attribute__ ((noinline))
+int f32(struct __sk_buff *skb)
+{
+ return f31(skb);
+}
+
+SEC("tc")
+__success
+int global_func3_32_frames(struct __sk_buff *skb)
+{
+ return f31(skb);
+}
+
SEC("tc")
-__failure __msg("the call stack of 17 frames")
+__failure __msg("the call stack of 32 frames")
int global_func3(struct __sk_buff *skb)
{
- return f16(skb);
+ return f32(skb);
}
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 57/81] selftests/bpf: Test 32 and 33 call frames
2026-10-26 2:59 ` [PATCH bpf 57/81] selftests/bpf: Test 32 and 33 call frames Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 2951 bytes --]
> selftests/bpf: Test 32 and 33 call frames
TL;DR: Test issue: with the JIT disabled, the new 32-frame selftests may hit
the combined stack size limit (512 bytes) at frame 17 first, so
global_func3_32_frames may fail to load and global_func3's message may not
match.
> diff --git a/tools/testing/selftests/bpf/progs/test_global_func3.c b/tools/testing/selftests/bpf/progs/test_global_func3.c
> index b66abb350fb0..3b816afaa278 100644
> --- a/tools/testing/selftests/bpf/progs/test_global_func3.c
> +++ b/tools/testing/selftests/bpf/progs/test_global_func3.c
[ ... ]
> +static __attribute__ ((noinline))
> +int f32(struct __sk_buff *skb)
> +{
> + return f31(skb);
> +}
> +
> +SEC("tc")
> +__success
> +int global_func3_32_frames(struct __sk_buff *skb)
> +{
> + return f31(skb);
> +}
> +
> SEC("tc")
> -__failure __msg("the call stack of 17 frames")
> +__failure __msg("the call stack of 32 frames")
> int global_func3(struct __sk_buff *skb)
> {
> - return f16(skb);
> + return f32(skb);
> }
Can these two programs fail when the JIT is disabled?
With bpf_jit_enable=0 on a kernel without CONFIG_BPF_JIT_ALWAYS_ON,
round_up_stack_depth() gives every frame at least 32 bytes:
if (env->prog->jit_requested)
return round_up(stack_depth, 16);
...
return round_up(max_t(u32, stack_depth, 1), 32);
and env->stack_limit is MAX_BPF_STACK (512). check_max_stack_depth_subprog()
adds the frame's depth before it descends to the next call, so the walk goes
main=32, f31=64, ..., f15=544, and fails at the 17th frame:
combined stack size of 17 calls is 544. Too large
That happens before the MAX_CALL_FRAMES (32) check is reached. So
global_func3_32_frames, which is marked __success, would be rejected, and
global_func3 would fail with the combined stack size message instead of "the
call stack of 32 frames", so its __msg() would not match.
The main pass cannot catch the chain first. func_is_on_its_own() is true
for every fN (r1 is PTR_TO_CTX), so each callee is verified in a one-frame
state and setup_func_entry() never sees the chain.
Before this patch the 17 frame chain was rejected in the main pass by
setup_func_entry() ("the call stack of 17 frames") before
check_max_stack_depth() ran, so the test also passed without the JIT.
Would adding __load_if_JITed() to both programs be appropriate?
test_global_func1.c and verifier_unwind.c already handle JIT vs interpreter
differences with __load_if_JITed() or with a {{[0-9]+}} regex in the combined
stack size message.
No later commit in the series changes this file or round_up_stack_depth().
The tools/testing/selftests/bpf/config.* files set CONFIG_BPF_JIT_ALWAYS_ON=y,
so only runs with the JIT off are affected.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 58/81] bpf: Bump BPF_MAX_SUBPROGS to 1024
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (56 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 57/81] selftests/bpf: Test 32 and 33 call frames Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 59/81] selftests/bpf: Make room in veristat for the stats of 1024 subprogs Alexei Starovoitov
` (24 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A prog that uses Rust crates can have more than 256 functions that the
compiler does not inline. The verifier rejects a prog with 777 subprogs:
too many subprograms
Bump the limit to 1024.
Two arrays in struct bpf_verifier_env grow: subprog_info[] and
subprog_topo_order[]. The verifier allocates the struct with kvzalloc(),
thus the size is not a problem. A load from a table of functions can give
BPF_MAX_SUBPROGS different pointers. This limit grows too.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 6139e889c86a..51abcaead6b9 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -974,7 +974,7 @@ struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log,
u32 size);
int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log);
-#define BPF_MAX_SUBPROGS 256
+#define BPF_MAX_SUBPROGS 1024
/*
* A pointer to a static subprog in the value of a frozen read-only array map:
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 59/81] selftests/bpf: Make room in veristat for the stats of 1024 subprogs
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (57 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 58/81] bpf: Bump BPF_MAX_SUBPROGS to 1024 Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 60/81] bpf: Remove is_state_visited() Alexei Starovoitov
` (23 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
veristat reads the statistics at the end of the verifier log. They have one
line for each subprog. Two sizes in veristat are too small now that a prog
can have 1024 subprogs.
size old new problem with the old size
---------------------- ---- ----- ----------------------------------
log buffer 64K 1M 777 subprogs make 82K of stats.
The load fails with ENOSPC.
lines that veristat 300 1100 With more than 296 subprogs it
parses, from the end does not find 'verification time'.
The duration is 0.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
tools/testing/selftests/bpf/veristat.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/tools/testing/selftests/bpf/veristat.c b/tools/testing/selftests/bpf/veristat.c
index 9cfc9b4b41c1..6b295e7d6f32 100644
--- a/tools/testing/selftests/bpf/veristat.c
+++ b/tools/testing/selftests/bpf/veristat.c
@@ -1013,10 +1013,11 @@ static void free_verif_stats(struct verif_stats *stats, size_t stat_cnt)
free(stats);
}
-static char verif_log_buf[64 * 1024];
+/* the stats have a line for every subprog, with its name */
+static char verif_log_buf[1024 * 1024];
-/* Keep room for all 256 subprogram records and trailing statistics. */
-#define MAX_PARSED_LOG_LINES 300
+/* Keep room for all 1024 subprogram records and trailing statistics. */
+#define MAX_PARSED_LOG_LINES 1100
static int parse_verif_log(char * const buf, size_t buf_sz, struct verif_stats *s)
{
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 60/81] bpf: Remove is_state_visited()
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (58 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 59/81] selftests/bpf: Make room in veristat for the stats of 1024 subprogs Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 61/81] bpf: Remove precision tracking Alexei Starovoitov
` (22 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
is_state_visited() did the pruning when the verifier followed each path of
a prog. It saved states at selected insns. It stopped a path when a saved
state at the same insn covered the current state.
Since "bpf: Verify programs as a fixpoint over the control flow graph"
flow.c keeps a set of states for each basic block. Nothing calls
is_state_visited(). Remove it, together with the code and the fields that
only it uses.
removed task in the old pass
---------------------------- -------------------------------------------
explored_states, free_list Hold the saved states.
'branches' in a state Count the paths from the state that are
not explored.
SCC visits, backedges Propagate precision marks over the back
edges of a loop.
states_maybe_looping() Find an infinite loop.
widen_imprecise_scalars() Make the states of a loop with an iterator,
a callback or may_goto converge.
BPF_F_TEST_STATE_FREQ told is_state_visited() to save a state each time.
The flag is UAPI, and selftests set it. The kernel continues to accept the
flag, and the flag has no effect.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 116 +----
kernel/bpf/cfg.c | 7 -
kernel/bpf/states.c | 957 -----------------------------------
kernel/bpf/verifier.c | 227 +--------
4 files changed, 5 insertions(+), 1302 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 51abcaead6b9..d704b3776b68 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -509,52 +509,6 @@ struct bpf_verifier_state {
struct bpf_verifier_state *parent;
/* Acquired reference states */
struct bpf_reference_state *refs;
- /*
- * 'branches' field is the number of branches left to explore:
- * 0 - all possible paths from this state reached bpf_exit or
- * were safely pruned
- * 1 - at least one path is being explored.
- * This state hasn't reached bpf_exit
- * 2 - at least two paths are being explored.
- * This state is an immediate parent of two children.
- * One is fallthrough branch with branches==1 and another
- * state is pushed into stack (to be explored later) also with
- * branches==1. The parent of this state has branches==1.
- * The verifier state tree connected via 'parent' pointer looks like:
- * 1
- * 1
- * 2 -> 1 (first 'if' pushed into stack)
- * 1
- * 2 -> 1 (second 'if' pushed into stack)
- * 1
- * 1
- * 1 bpf_exit.
- *
- * Once do_check() reaches bpf_exit, it calls update_branch_counts()
- * and the verifier state tree will look:
- * 1
- * 1
- * 2 -> 1 (first 'if' pushed into stack)
- * 1
- * 1 -> 1 (second 'if' pushed into stack)
- * 0
- * 0
- * 0 bpf_exit.
- * After pop_stack() the do_check() will resume at second 'if'.
- *
- * If is_state_visited() sees a state with branches > 0 it means
- * there is a loop. If such state is exactly equal to the current state
- * it's an infinite loop. Note states_equal() checks for states
- * equivalency, so two states being 'states_equal' does not mean
- * infinite loop. The exact comparison is provided by
- * states_maybe_looping() function. It's a stronger pre-check and
- * much faster than states_equal().
- *
- * This algorithm may not find all possible infinite loops or
- * loop iteration count may be too high.
- * In such cases BPF_COMPLEXITY_LIMIT_INSNS limit kicks in.
- */
- u32 branches;
u32 insn_idx;
u32 curframe;
@@ -586,10 +540,6 @@ struct bpf_verifier_state {
*/
u32 first_insn_idx;
u32 last_insn_idx;
- /* if this state is a backedge state then equal_state
- * records cached state to which this state is equal.
- */
- struct bpf_verifier_state *equal_state;
/* jmp history recorded from first to last.
* backtracking is using it to go from last to first.
* For most states jmp_history_cnt is [0-3].
@@ -597,9 +547,6 @@ struct bpf_verifier_state {
*/
struct bpf_jmp_history_entry *jmp_history;
u32 jmp_history_cnt;
- u32 dfs_depth;
- u32 callback_unroll_depth;
- u32 may_goto_depth;
};
/* Number of BPF_REG_SIZE stack slots tracked for the frame so far. */
@@ -689,15 +636,6 @@ bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame)
1 << STACK_SPILL, __expr); \
})
-/* linked list of verifier states used to prune search */
-struct bpf_verifier_state_list {
- struct bpf_verifier_state state;
- struct list_head node;
- u32 miss_cnt;
- u32 hit_cnt:31;
- u32 in_free_list:1;
-};
-
struct bpf_loop_inline_state {
unsigned int initialized:1; /* set to true upon first entry */
unsigned int fit_for_inline:1; /* true if callback function is the same
@@ -1111,38 +1049,6 @@ struct bpf_idset {
} entries[BPF_ID_MAP_SIZE];
};
-/* see verifier.c:compute_scc_callchain() */
-struct bpf_scc_callchain {
- /* call sites from bpf_verifier_state->frame[*]->callsite leading to this SCC */
- u32 callsites[MAX_CALL_FRAMES - 1];
- /* last frame in a chain is identified by SCC id */
- u32 scc;
-};
-
-/* verifier state waiting for propagate_backedges() */
-struct bpf_scc_backedge {
- struct bpf_scc_backedge *next;
- struct bpf_verifier_state state;
-};
-
-struct bpf_scc_visit {
- struct bpf_scc_callchain callchain;
- /* first state in current verification path that entered SCC
- * identified by the callchain
- */
- struct bpf_verifier_state *entry_state;
- struct bpf_scc_backedge *backedges; /* list of backedges */
- u32 num_backedges;
-};
-
-/* An array of bpf_scc_visit structs sharing tht same bpf_scc_callchain->scc
- * but having different bpf_scc_callchain->callsites.
- */
-struct bpf_scc_info {
- u32 num_visits;
- struct bpf_scc_visit visits[];
-};
-
struct bpf_liveness;
struct scev;
@@ -1166,17 +1072,11 @@ struct bpf_verifier_env {
struct bpf_verifier_stack_elem *head; /* stack of verifier states to be processed */
int stack_size; /* number of states to be processed */
bool strict_alignment; /* perform strict pointer alignment checks */
- bool test_state_freq; /* test verifier with different pruning frequency */
bool test_reg_invariants; /* fail verification on register invariants violations */
bool uses_arena_sp; /* the program has insns with BPF_REG_ARENA_SP */
bool restart; /* do the main pass again with what was found out */
u32 max_arena_stack_depth; /* of all call chains */
struct bpf_verifier_state *cur_state; /* current verifier state */
- /* Search pruning optimization, array of list_heads for
- * lists of struct bpf_verifier_state_list.
- */
- struct list_head *explored_states;
- struct list_head free_list; /* list of struct bpf_verifier_state_list */
struct bpf_map *used_maps[MAX_USED_MAPS]; /* array of map's used by eBPF program */
struct btf_mod_pair used_btfs[MAX_USED_BTFS]; /* array of BTF's used by BPF program */
struct bpf_map *insn_array_maps[MAX_USED_MAPS]; /* array of INSN_ARRAY map's to be relocated */
@@ -1250,9 +1150,7 @@ struct bpf_verifier_env {
u32 pass_cnt; /* number of times do_check() was called */
u32 subprog_cnt;
/* number of instructions analyzed by the verifier */
- u32 prev_insn_processed, insn_processed;
- /* number of jmps, calls, exits analyzed so far */
- u32 prev_jmps_processed, jmps_processed;
+ u32 insn_processed;
/* maximum combined stack depth */
u32 max_stack_depth;
/* stack budget of the program, see bpf_prog_stack_limit() */
@@ -1270,9 +1168,6 @@ struct bpf_verifier_env {
u32 peak_states;
/* longest register parentage chain walked for liveness marking */
u32 longest_mark_read_walk;
- u32 free_list_size;
- u32 explored_states_size;
- u32 num_backedges;
/*
* The program's fd_array comes in two shapes, told apart by whether
* the caller passed fd_array_cnt. They are mutually exclusive:
@@ -1305,11 +1200,7 @@ struct bpf_verifier_env {
char tmp_arg_name[32];
struct bpf_insn insn_buf[INSN_BUF_SIZE];
struct bpf_insn epilogue_buf[INSN_BUF_SIZE];
- struct bpf_scc_callchain callchain_buf;
struct bpf_liveness *liveness;
- /* array of pointers to bpf_scc_info indexed by SCC id */
- struct bpf_scc_info **scc_info;
- u32 scc_cnt;
struct bpf_iarray *succ;
struct bpf_iarray *gotox_tmp_buf;
int *idoms;
@@ -1498,7 +1389,6 @@ void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
int mark_chain_precision(struct bpf_verifier_env *env, int regno);
-int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx);
int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
struct bpf_verifier_state *cur);
@@ -1540,15 +1430,11 @@ bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn);
bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n);
bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
-int bpf_split_cur_state(struct bpf_verifier_env *env);
-int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
void bpf_clear_jmp_history(struct bpf_verifier_state *state);
int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
const struct bpf_verifier_state *src);
-struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx);
void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
-void bpf_free_backedges(struct bpf_scc_visit *visit);
int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
int insn_flags, int spi, int frame, const u16 *linked_regs,
u8 linked_regs_cnt);
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 2ee950e9ac82..f5705ea0bcb9 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -1016,13 +1016,6 @@ int bpf_compute_scc(struct bpf_verifier_env *env)
dfs_sz--;
}
}
- env->scc_info = kvzalloc_objs(*env->scc_info, next_scc_id,
- GFP_KERNEL_ACCOUNT);
- if (!env->scc_info) {
- err = -ENOMEM;
- goto exit;
- }
- env->scc_cnt = next_scc_id;
exit:
kvfree(stack);
kvfree(pre);
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index d55adade5b1c..7cf81ba411eb 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -7,297 +7,6 @@
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
-#define BPF_COMPLEXITY_LIMIT_STATES 64
-
-static bool is_may_goto_insn_at(struct bpf_verifier_env *env, int insn_idx)
-{
- return bpf_is_may_goto_insn(&env->prog->insnsi[insn_idx]);
-}
-
-static bool is_iter_next_insn(struct bpf_verifier_env *env, int insn_idx)
-{
- return env->insn_aux_data[insn_idx].is_iter_next;
-}
-
-static void update_peak_states(struct bpf_verifier_env *env)
-{
- u32 cur_states;
-
- cur_states = env->explored_states_size + env->free_list_size + env->num_backedges;
- env->peak_states = max(env->peak_states, cur_states);
-}
-
-/* struct bpf_verifier_state->parent refers to states
- * that are in either of env->{expored_states,free_list}.
- * In both cases the state is contained in struct bpf_verifier_state_list.
- */
-static struct bpf_verifier_state_list *state_parent_as_list(struct bpf_verifier_state *st)
-{
- if (st->parent)
- return container_of(st->parent, struct bpf_verifier_state_list, state);
- return NULL;
-}
-
-static bool incomplete_read_marks(struct bpf_verifier_env *env,
- struct bpf_verifier_state *st);
-
-/* A state can be freed if it is no longer referenced:
- * - is in the env->free_list;
- * - has no children states;
- */
-static void maybe_free_verifier_state(struct bpf_verifier_env *env,
- struct bpf_verifier_state_list *sl)
-{
- if (!sl->in_free_list
- || sl->state.branches != 0
- || incomplete_read_marks(env, &sl->state))
- return;
- list_del(&sl->node);
- bpf_free_verifier_state(&sl->state, false);
- kfree(sl);
- env->free_list_size--;
-}
-
-/* For state @st look for a topmost frame with frame_insn_idx() in some SCC,
- * if such frame exists form a corresponding @callchain as an array of
- * call sites leading to this frame and SCC id.
- * E.g.:
- *
- * void foo() { A: loop {... SCC#1 ...}; }
- * void bar() { B: loop { C: foo(); ... SCC#2 ... }
- * D: loop { E: foo(); ... SCC#3 ... } }
- * void main() { F: bar(); }
- *
- * @callchain at (A) would be either (F,SCC#2) or (F,SCC#3) depending
- * on @st frame call sites being (F,C,A) or (F,E,A).
- */
-static bool compute_scc_callchain(struct bpf_verifier_env *env,
- struct bpf_verifier_state *st,
- struct bpf_scc_callchain *callchain)
-{
- u32 i, scc, insn_idx;
-
- memset(callchain, 0, sizeof(*callchain));
- for (i = 0; i <= st->curframe; i++) {
- insn_idx = bpf_frame_insn_idx(st, i);
- scc = env->insn_aux_data[insn_idx].scc;
- if (scc) {
- callchain->scc = scc;
- break;
- } else if (i < st->curframe) {
- callchain->callsites[i] = insn_idx;
- } else {
- return false;
- }
- }
- return true;
-}
-
-/* Check if bpf_scc_visit instance for @callchain exists. */
-static struct bpf_scc_visit *scc_visit_lookup(struct bpf_verifier_env *env,
- struct bpf_scc_callchain *callchain)
-{
- struct bpf_scc_info *info = env->scc_info[callchain->scc];
- struct bpf_scc_visit *visits = info->visits;
- u32 i;
-
- if (!info)
- return NULL;
- for (i = 0; i < info->num_visits; i++)
- if (memcmp(callchain, &visits[i].callchain, sizeof(*callchain)) == 0)
- return &visits[i];
- return NULL;
-}
-
-/* Allocate a new bpf_scc_visit instance corresponding to @callchain.
- * Allocated instances are alive for a duration of the do_check_common()
- * call and are freed by free_states().
- */
-static struct bpf_scc_visit *scc_visit_alloc(struct bpf_verifier_env *env,
- struct bpf_scc_callchain *callchain)
-{
- struct bpf_scc_visit *visit;
- struct bpf_scc_info *info;
- u32 scc, num_visits;
- u64 new_sz;
-
- scc = callchain->scc;
- info = env->scc_info[scc];
- num_visits = info ? info->num_visits : 0;
- new_sz = sizeof(*info) + sizeof(struct bpf_scc_visit) * (num_visits + 1);
- info = kvrealloc(env->scc_info[scc], new_sz, GFP_KERNEL_ACCOUNT);
- if (!info)
- return NULL;
- env->scc_info[scc] = info;
- info->num_visits = num_visits + 1;
- visit = &info->visits[num_visits];
- memset(visit, 0, sizeof(*visit));
- memcpy(&visit->callchain, callchain, sizeof(*callchain));
- return visit;
-}
-
-/* Form a string '(callsite#1,callsite#2,...,scc)' in env->tmp_str_buf */
-static char *format_callchain(struct bpf_verifier_env *env, struct bpf_scc_callchain *callchain)
-{
- char *buf = env->tmp_str_buf;
- int i, delta = 0;
-
- delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "(");
- for (i = 0; i < ARRAY_SIZE(callchain->callsites); i++) {
- if (!callchain->callsites[i])
- break;
- delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "%u,",
- callchain->callsites[i]);
- }
- delta += snprintf(buf + delta, TMP_STR_BUF_LEN - delta, "%u)", callchain->scc);
- return env->tmp_str_buf;
-}
-
-/* If callchain for @st exists (@st is in some SCC), ensure that
- * bpf_scc_visit instance for this callchain exists.
- * If instance does not exist or is empty, assign visit->entry_state to @st.
- */
-static int maybe_enter_scc(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
- struct bpf_scc_callchain *callchain = &env->callchain_buf;
- struct bpf_scc_visit *visit;
-
- if (!compute_scc_callchain(env, st, callchain))
- return 0;
- visit = scc_visit_lookup(env, callchain);
- visit = visit ?: scc_visit_alloc(env, callchain);
- if (!visit)
- return -ENOMEM;
- if (!visit->entry_state) {
- visit->entry_state = st;
- if (env->log.level & BPF_LOG_LEVEL2)
- verbose(env, "SCC enter %s\n", format_callchain(env, callchain));
- }
- return 0;
-}
-
-static int propagate_backedges(struct bpf_verifier_env *env, struct bpf_scc_visit *visit);
-
-/* If callchain for @st exists (@st is in some SCC), make it empty:
- * - set visit->entry_state to NULL;
- * - flush accumulated backedges.
- */
-static int maybe_exit_scc(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
- struct bpf_scc_callchain *callchain = &env->callchain_buf;
- struct bpf_scc_visit *visit;
-
- if (!compute_scc_callchain(env, st, callchain))
- return 0;
- visit = scc_visit_lookup(env, callchain);
- if (!visit) {
- /*
- * If path traversal stops inside an SCC, corresponding bpf_scc_visit
- * must exist for non-speculative paths. For non-speculative paths
- * traversal stops when:
- * a. Verification error is found, maybe_exit_scc() is not called.
- * b. Top level BPF_EXIT is reached. Top level BPF_EXIT is not a member
- * of any SCC.
- * c. A checkpoint is reached and matched. Checkpoints are created by
- * is_state_visited(), which calls maybe_enter_scc(), which allocates
- * bpf_scc_visit instances for checkpoints within SCCs.
- * (c) is the only case that can reach this point.
- */
- if (!st->speculative) {
- verifier_bug(env, "scc exit: no visit info for call chain %s",
- format_callchain(env, callchain));
- return -EFAULT;
- }
- return 0;
- }
- if (visit->entry_state != st)
- return 0;
- if (env->log.level & BPF_LOG_LEVEL2)
- verbose(env, "SCC exit %s\n", format_callchain(env, callchain));
- visit->entry_state = NULL;
- env->num_backedges -= visit->num_backedges;
- visit->num_backedges = 0;
- update_peak_states(env);
- return propagate_backedges(env, visit);
-}
-
-/* Lookup an bpf_scc_visit instance corresponding to @st callchain
- * and add @backedge to visit->backedges. @st callchain must exist.
- */
-static int add_scc_backedge(struct bpf_verifier_env *env,
- struct bpf_verifier_state *st,
- struct bpf_scc_backedge *backedge)
-{
- struct bpf_scc_callchain *callchain = &env->callchain_buf;
- struct bpf_scc_visit *visit;
-
- if (!compute_scc_callchain(env, st, callchain)) {
- verifier_bug(env, "add backedge: no SCC in verification path, insn_idx %d",
- st->insn_idx);
- return -EFAULT;
- }
- visit = scc_visit_lookup(env, callchain);
- if (!visit) {
- verifier_bug(env, "add backedge: no visit info for call chain %s",
- format_callchain(env, callchain));
- return -EFAULT;
- }
- if (env->log.level & BPF_LOG_LEVEL2)
- verbose(env, "SCC backedge %s\n", format_callchain(env, callchain));
- backedge->next = visit->backedges;
- visit->backedges = backedge;
- visit->num_backedges++;
- env->num_backedges++;
- update_peak_states(env);
- return 0;
-}
-
-/* bpf_reg_state->live marks for registers in a state @st are incomplete,
- * if state @st is in some SCC and not all execution paths starting at this
- * SCC are fully explored.
- */
-static bool incomplete_read_marks(struct bpf_verifier_env *env,
- struct bpf_verifier_state *st)
-{
- struct bpf_scc_callchain *callchain = &env->callchain_buf;
- struct bpf_scc_visit *visit;
-
- if (!compute_scc_callchain(env, st, callchain))
- return false;
- visit = scc_visit_lookup(env, callchain);
- if (!visit)
- return false;
- return !!visit->backedges;
-}
-
-int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
- struct bpf_verifier_state_list *sl = NULL, *parent_sl;
- struct bpf_verifier_state *parent;
- int err;
-
- while (st) {
- u32 br = --st->branches;
-
- /* verifier_bug_if(br > 1, ...) technically makes sense here,
- * but see comment in push_stack(), hence:
- */
- verifier_bug_if((int)br < 0, env, "%s:branches_to_explore=%d", __func__, br);
- if (br)
- break;
- err = maybe_exit_scc(env, st);
- if (err)
- return err;
- parent = st->parent;
- parent_sl = state_parent_as_list(st);
- if (sl)
- maybe_free_verifier_state(env, sl);
- st = parent;
- sl = parent_sl;
- }
- return 0;
-}
-
/* check %cur's range satisfies %old's */
static bool range_within(const struct bpf_reg_state *old,
const struct bpf_reg_state *cur)
@@ -1071,672 +780,6 @@ static bool states_equal(struct bpf_verifier_env *env,
return true;
}
-/* find precise scalars in the previous equivalent state and
- * propagate them into the current state
- */
-static int propagate_precision(struct bpf_verifier_env *env,
- const struct bpf_verifier_state *old,
- struct bpf_verifier_state *cur,
- bool *changed)
-{
- struct bpf_reg_state *state_reg;
- struct bpf_func_state *state;
- int i, err = 0, fr;
- bool first;
-
- for (fr = old->curframe; fr >= 0; fr--) {
- state = old->frame[fr];
- state_reg = state->regs;
- first = true;
- for (i = 0; i < BPF_REG_FP; i++, state_reg++) {
- if (state_reg->type != SCALAR_VALUE ||
- !state_reg->precise)
- continue;
- if (env->log.level & BPF_LOG_LEVEL2) {
- if (first)
- verbose(env, "frame %d: propagating r%d", fr, i);
- else
- verbose(env, ",r%d", i);
- }
- bpf_bt_set_frame_reg(&env->bt, fr, i);
- first = false;
- }
-
- for (i = 0; i < bpf_stack_nr_slots(state); i++) {
- struct bpf_stack_state *ss = bpf_stack_slot(state, i);
-
- if (!bpf_is_spilled_reg(ss))
- continue;
- state_reg = &ss->spilled_ptr;
- if (state_reg->type != SCALAR_VALUE ||
- !state_reg->precise)
- continue;
- if (env->log.level & BPF_LOG_LEVEL2) {
- if (first)
- verbose(env, "frame %d: propagating fp%d",
- fr, (-i - 1) * BPF_REG_SIZE);
- else
- verbose(env, ",fp%d", (-i - 1) * BPF_REG_SIZE);
- }
- bpf_bt_set_frame_slot(&env->bt, fr, i);
- first = false;
- }
- if (!first && (env->log.level & BPF_LOG_LEVEL2))
- verbose(env, "\n");
- }
-
- err = bpf_mark_chain_precision(env, cur, -1, changed);
- if (err < 0)
- return err;
-
- return 0;
-}
-
-#define MAX_BACKEDGE_ITERS 64
-
-/* Propagate read and precision marks from visit->backedges[*].state->equal_state
- * to corresponding parent states of visit->backedges[*].state until fixed point is reached,
- * then free visit->backedges.
- * After execution of this function incomplete_read_marks() will return false
- * for all states corresponding to @visit->callchain.
- */
-static int propagate_backedges(struct bpf_verifier_env *env, struct bpf_scc_visit *visit)
-{
- struct bpf_scc_backedge *backedge;
- struct bpf_verifier_state *st;
- bool changed;
- int i, err;
-
- i = 0;
- do {
- if (i++ > MAX_BACKEDGE_ITERS) {
- if (env->log.level & BPF_LOG_LEVEL2)
- verbose(env, "%s: too many iterations\n", __func__);
- for (backedge = visit->backedges; backedge; backedge = backedge->next)
- bpf_mark_all_scalars_precise(env, &backedge->state);
- break;
- }
- changed = false;
- for (backedge = visit->backedges; backedge; backedge = backedge->next) {
- st = &backedge->state;
- err = propagate_precision(env, st->equal_state, st, &changed);
- if (err)
- return err;
- }
- } while (changed);
-
- bpf_free_backedges(visit);
- return 0;
-}
-
-static bool states_maybe_looping(struct bpf_verifier_state *old,
- struct bpf_verifier_state *cur)
-{
- struct bpf_func_state *fold, *fcur;
- int i, fr = cur->curframe;
-
- if (old->curframe != fr)
- return false;
-
- fold = old->frame[fr];
- fcur = cur->frame[fr];
- for (i = 0; i < MAX_BPF_REG; i++)
- if (memcmp(&fold->regs[i], &fcur->regs[i],
- offsetofend(struct bpf_reg_state, step)))
- return false;
- return true;
-}
-
-/* is_state_visited() handles iter_next() (see process_iter_next_call() for
- * terminology) calls specially: as opposed to bounded BPF loops, it *expects*
- * states to match, which otherwise would look like an infinite loop. So while
- * iter_next() calls are taken care of, we still need to be careful and
- * prevent erroneous and too eager declaration of "infinite loop", when
- * iterators are involved.
- *
- * Here's a situation in pseudo-BPF assembly form:
- *
- * 0: again: ; set up iter_next() call args
- * 1: r1 = &it ; <CHECKPOINT HERE>
- * 2: call bpf_iter_num_next ; this is iter_next() call
- * 3: if r0 == 0 goto done
- * 4: ... something useful here ...
- * 5: goto again ; another iteration
- * 6: done:
- * 7: r1 = &it
- * 8: call bpf_iter_num_destroy ; clean up iter state
- * 9: exit
- *
- * This is a typical loop. Let's assume that we have a prune point at 1:,
- * before we get to `call bpf_iter_num_next` (e.g., because of that `goto
- * again`, assuming other heuristics don't get in a way).
- *
- * When we first time come to 1:, let's say we have some state X. We proceed
- * to 2:, fork states, enqueue ACTIVE, validate NULL case successfully, exit.
- * Now we come back to validate that forked ACTIVE state. We proceed through
- * 3-5, come to goto, jump to 1:. Let's assume our state didn't change, so we
- * are converging. But the problem is that we don't know that yet, as this
- * convergence has to happen at iter_next() call site only. So if nothing is
- * done, at 1: verifier will use bounded loop logic and declare infinite
- * looping (and would be *technically* correct, if not for iterator's
- * "eventual sticky NULL" contract, see process_iter_next_call()). But we
- * don't want that. So what we do in process_iter_next_call() when we go on
- * another ACTIVE iteration, we bump slot->iter.depth, to mark that it's
- * a different iteration. So when we suspect an infinite loop, we additionally
- * check if any of the *ACTIVE* iterator states depths differ. If yes, we
- * pretend we are not looping and wait for next iter_next() call.
- *
- * This only applies to ACTIVE state. In DRAINED state we don't expect to
- * loop, because that would actually mean infinite loop, as DRAINED state is
- * "sticky", and so we'll keep returning into the same instruction with the
- * same state (at least in one of possible code paths).
- *
- * This approach allows to keep infinite loop heuristic even in the face of
- * active iterator. E.g., C snippet below is and will be detected as
- * infinitely looping:
- *
- * struct bpf_iter_num it;
- * int *p, x;
- *
- * bpf_iter_num_new(&it, 0, 10);
- * while ((p = bpf_iter_num_next(&t))) {
- * x = p;
- * while (x--) {} // <<-- infinite loop here
- * }
- *
- */
-static bool iter_active_depths_differ(struct bpf_verifier_state *old, struct bpf_verifier_state *cur)
-{
- struct bpf_reg_state *slot, *cur_slot;
- struct bpf_func_state *state;
- int i, fr;
-
- for (fr = old->curframe; fr >= 0; fr--) {
- state = old->frame[fr];
- for (i = 0; i < bpf_stack_nr_slots(state); i++) {
- struct bpf_stack_state *ss = bpf_stack_slot(state, i);
-
- if (ss->slot_type[0] != STACK_ITER)
- continue;
-
- slot = &ss->spilled_ptr;
- if (slot->iter.state != BPF_ITER_STATE_ACTIVE)
- continue;
-
- cur_slot = &bpf_stack_slot(cur->frame[fr], i)->spilled_ptr;
- if (cur_slot->iter.depth != slot->iter.depth)
- return true;
- }
- }
- return false;
-}
-
-static void mark_all_scalars_imprecise(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
- struct bpf_func_state *func;
- struct bpf_reg_state *reg;
- int i, j;
-
- for (i = 0; i <= st->curframe; i++) {
- func = st->frame[i];
- for (j = 0; j < BPF_REG_FP; j++) {
- reg = &func->regs[j];
- if (reg->type != SCALAR_VALUE)
- continue;
- reg->precise = false;
- }
- for (j = 0; j < bpf_stack_nr_slots(func); j++) {
- struct bpf_stack_state *ss = bpf_stack_slot(func, j);
-
- if (!bpf_is_spilled_reg(ss))
- continue;
- reg = &ss->spilled_ptr;
- if (reg->type != SCALAR_VALUE)
- continue;
- reg->precise = false;
- }
- }
-}
-
-int bpf_split_cur_state(struct bpf_verifier_env *env)
-{
- struct bpf_verifier_state *cur = env->cur_state, *new;
- struct bpf_verifier_state_list *new_sl;
- struct list_head *head;
- int insn_idx = cur->insn_idx;
- int err;
-
- head = bpf_explored_state(env, insn_idx);
- new_sl = kzalloc_obj(struct bpf_verifier_state_list, GFP_KERNEL_ACCOUNT);
- if (!new_sl)
- return -ENOMEM;
- env->total_states++;
- env->explored_states_size++;
- update_peak_states(env);
- env->prev_jmps_processed = env->jmps_processed;
- env->prev_insn_processed = env->insn_processed;
-
- /* forget precise markings we inherited, see __mark_chain_precision */
- if (env->bpf_capable)
- mark_all_scalars_imprecise(env, cur);
-
- bpf_clear_singular_ids(env, cur);
-
- /* add new state to the head of linked list */
- new = &new_sl->state;
- err = bpf_copy_verifier_state(new, cur);
- if (err) {
- bpf_free_verifier_state(new, false);
- kfree(new_sl);
- return err;
- }
- new->insn_idx = insn_idx;
- verifier_bug_if(new->branches != 1, env,
- "%s:branches_to_explore=%d insn %d",
- __func__, new->branches, insn_idx);
- err = maybe_enter_scc(env, new);
- if (err) {
- bpf_free_verifier_state(new, false);
- kfree(new_sl);
- return err;
- }
-
- cur->parent = new;
- cur->last_insn_idx = -1;
- cur->first_insn_idx = insn_idx;
- cur->dfs_depth = new->dfs_depth + 1;
- bpf_clear_jmp_history(cur);
- list_add(&new_sl->node, head);
- return 0;
-}
-
-/* Force a checkpoint upon reaching a loop header for a terminating loop. */
-static bool need_loop_checkpoint(struct bpf_verifier_env *env, int insn_idx)
-{
- struct bpf_verifier_state *cur = env->cur_state;
- struct bpf_func_state *frame = cur->frame[cur->curframe];
- struct loop_stack_entry *top;
-
- if (bpf_loop_at_index(env, insn_idx) != insn_idx || !frame->loop_stack_cnt)
- return false;
- top = &frame->loop_stack[frame->loop_stack_cnt - 1];
- return top->loop_id == insn_idx && top->terminates;
-}
-
-static struct loop_stack_entry *current_loop(struct bpf_verifier_state *st)
-{
- struct bpf_func_state *frame = st->frame[st->curframe];
-
- if (frame->loop_stack_cnt == 0)
- return NULL;
- return &frame->loop_stack[frame->loop_stack_cnt - 1];
-}
-
-int bpf_is_state_visited(struct bpf_verifier_env *env, int insn_idx)
-{
- struct loop_stack_entry *old_loop, *cur_loop;
- struct bpf_verifier_state_list *sl;
- struct bpf_verifier_state *cur = env->cur_state;
- bool force_new_state, add_new_state, loop, loop_checkpoint;
- int n, err, states_cnt = 0;
- struct list_head *pos, *tmp, *head;
-
- loop_checkpoint = need_loop_checkpoint(env, insn_idx);
- force_new_state = env->test_state_freq || bpf_is_force_checkpoint(env, insn_idx) ||
- loop_checkpoint ||
- /* Avoid accumulating infinitely long jmp history */
- cur->jmp_history_cnt > 40;
-
- /* bpf progs typically have pruning point every 4 instructions
- * http://vger.kernel.org/bpfconf2019.html#session-1
- * Do not add new state for future pruning if the verifier hasn't seen
- * at least 2 jumps and at least 8 instructions.
- * This heuristics helps decrease 'total_states' and 'peak_states' metric.
- * In tests that amounts to up to 50% reduction into total verifier
- * memory consumption and 20% verifier time speedup.
- */
- add_new_state = force_new_state;
- if (env->jmps_processed - env->prev_jmps_processed >= 2 &&
- env->insn_processed - env->prev_insn_processed >= 8)
- add_new_state = true;
-
- /* keep cleaning the current state as registers/stack become dead */
- err = clean_verifier_state(env, cur);
- if (err)
- return err;
-
- loop = false;
- head = bpf_explored_state(env, insn_idx);
- list_for_each_safe(pos, tmp, head) {
- sl = container_of(pos, struct bpf_verifier_state_list, node);
- states_cnt++;
- if (sl->state.insn_idx != insn_idx)
- continue;
-
- if (sl->state.branches) {
- struct bpf_func_state *old_top_frame = sl->state.frame[0];
-
- if (old_top_frame->in_async_callback_fn &&
- old_top_frame->async_entry_cnt != cur->frame[0]->async_entry_cnt) {
- /* Different async_entry_cnt means that the verifier is
- * processing another entry into async callback.
- * Seeing the same state is not an indication of infinite
- * loop or infinite recursion.
- * But finding the same state doesn't mean that it's safe
- * to stop processing the current state. The previous state
- * hasn't yet reached bpf_exit, since state.branches > 0.
- * Checking in_async_callback_fn alone is not enough either.
- * Since the verifier still needs to catch infinite loops
- * inside async callbacks.
- */
- goto skip_inf_loop_check;
- }
- /* BPF open-coded iterators loop detection is special.
- * states_maybe_looping() logic is too simplistic in detecting
- * states that *might* be equivalent, because it doesn't know
- * about ID remapping, so don't even perform it.
- * See process_iter_next_call() and iter_active_depths_differ()
- * for overview of the logic. When current and one of parent
- * states are detected as equivalent, it's a good thing: we prove
- * convergence and can stop simulating further iterations.
- * It's safe to assume that iterator loop will finish, taking into
- * account iter_next() contract of eventually returning
- * sticky NULL result.
- *
- * Note, that states have to be compared exactly in this case because
- * read and precision marks might not be finalized inside the loop.
- * E.g. as in the program below:
- *
- * 1. r7 = -16
- * 2. r6 = bpf_get_prandom_u32()
- * 3. while (bpf_iter_num_next(&fp[-8])) {
- * 4. if (r6 != 42) {
- * 5. r7 = -32
- * 6. r6 = bpf_get_prandom_u32()
- * 7. continue
- * 8. }
- * 9. r0 = r10
- * 10. r0 += r7
- * 11. r8 = *(u64 *)(r0 + 0)
- * 12. r6 = bpf_get_prandom_u32()
- * 13. }
- *
- * Here verifier would first visit path 1-3, create a checkpoint at 3
- * with r7=-16, continue to 4-7,3. Existing checkpoint at 3 does
- * not have read or precision mark for r7 yet, thus inexact states
- * comparison would discard current state with r7=-32
- * => unsafe memory access at 11 would not be caught.
- */
- if (is_iter_next_insn(env, insn_idx)) {
- if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
- struct bpf_func_state *cur_frame, *iter_frame, *old_frame;
- struct bpf_reg_state *iter_state, *iter_reg;
- struct bpf_reg_state *old_iter;
- int spi;
-
- cur_frame = cur->frame[cur->curframe];
- /* btf_check_iter_kfuncs() enforces that
- * iter state pointer is always the first arg
- */
- iter_reg = &cur_frame->regs[BPF_REG_1];
- /* current state is valid due to states_equal(),
- * so we can assume valid iter and reg state,
- * no need for extra (re-)validations
- */
- spi = bpf_get_spi(iter_reg->var_off.value);
- iter_frame = bpf_func(env, iter_reg);
- iter_state = &bpf_stack_slot(iter_frame, spi)->spilled_ptr;
- old_frame = sl->state.frame[iter_reg->frameno];
- old_iter = &bpf_stack_slot(old_frame, spi)->spilled_ptr;
- /*
- * states_equal() matches ids through idmap.
- * The loop converged only if it's the same
- * iterator. The one that was destroyed and
- * created again starts from the beginning.
- */
- if (iter_state->iter.state == BPF_ITER_STATE_ACTIVE &&
- iter_state->id == old_iter->id) {
- loop = true;
- goto hit;
- }
- }
- goto skip_inf_loop_check;
- }
- if (is_may_goto_insn_at(env, insn_idx)) {
- if (sl->state.may_goto_depth != cur->may_goto_depth &&
- states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
- loop = true;
- goto hit;
- }
- }
- if (bpf_calls_callback(env, insn_idx)) {
- if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
- loop = true;
- goto hit;
- }
- goto skip_inf_loop_check;
- }
- /*
- * If old state belongs to a control flow loop that we know terminates,
- * it should be safe to prune current state. However, account for the
- * following situation:
- *
- * for (;;) {
- * for (i = 0; i < 10; i++)
- * ...
- * }
- *
- * Here a backedge to a non-terminating outer loop is dominated by an exit
- * from the inner loop. Pruning at a re-entry to the inner loop therefore
- * would hide the fact that outer loop is non-terminating.
- * Hence, only allow pruning states within the same entry state.
- */
- old_loop = current_loop(&sl->state);
- cur_loop = current_loop(cur);
- if (old_loop && cur_loop &&
- old_loop->terminates && cur_loop->terminates &&
- old_loop->entry_state == cur_loop->entry_state) {
- if (states_equal(env, &sl->state, cur, RANGE_WITHIN)) {
- loop = true;
- goto hit;
- }
- goto skip_inf_loop_check;
- }
-
- /* attempt to detect infinite loop to avoid unnecessary doomed work */
- if (states_maybe_looping(&sl->state, cur) &&
- states_equal(env, &sl->state, cur, EXACT) &&
- !iter_active_depths_differ(&sl->state, cur) &&
- sl->state.may_goto_depth == cur->may_goto_depth &&
- sl->state.callback_unroll_depth == cur->callback_unroll_depth) {
- verbose_linfo(env, insn_idx, "; ");
- verbose(env, "infinite loop detected at insn %d\n", insn_idx);
- verbose(env, "cur state:");
- print_verifier_state(env, cur, cur->curframe, true);
- verbose(env, "old state:");
- print_verifier_state(env, &sl->state, cur->curframe, true);
- return -EINVAL;
- }
- /* if the verifier is processing a loop, avoid adding new state
- * too often, since different loop iterations have distinct
- * states and may not help future pruning.
- * This threshold shouldn't be too low to make sure that
- * a loop with large bound will be rejected quickly.
- * The most abusive loop will be:
- * r1 += 1
- * if r1 < 1000000 goto pc-2
- * 1M insn_procssed limit / 100 == 10k peak states.
- * This threshold shouldn't be too high either, since states
- * at the end of the loop are likely to be useful in pruning.
- */
-skip_inf_loop_check:
- if (!force_new_state &&
- env->jmps_processed - env->prev_jmps_processed < 20 &&
- env->insn_processed - env->prev_insn_processed < 100)
- add_new_state = false;
- goto miss;
- }
- /* See comments for mark_all_regs_read_and_precise() */
- loop = incomplete_read_marks(env, &sl->state);
- if (states_equal(env, &sl->state, cur, loop ? RANGE_WITHIN : NOT_EXACT)) {
-hit:
- sl->hit_cnt++;
-
- /* if previous state reached the exit with precision and
- * current state is equivalent to it (except precision marks)
- * the precision needs to be propagated back in
- * the current state.
- */
- err = 0;
- if (bpf_is_jmp_point(env, env->insn_idx))
- err = bpf_push_jmp_history(env, cur, 0, 0, 0, NULL, 0);
- err = err ? : propagate_precision(env, &sl->state, cur, NULL);
- if (err)
- return err;
- /* When processing iterator based loops above propagate_liveness and
- * propagate_precision calls are not sufficient to transfer all relevant
- * read and precision marks. E.g. consider the following case:
- *
- * .-> A --. Assume the states are visited in the order A, B, C.
- * | | | Assume that state B reaches a state equivalent to state A.
- * | v v At this point, state C is not processed yet, so state A
- * '-- B C has not received any read or precision marks from C.
- * Thus, marks propagated from A to B are incomplete.
- *
- * The verifier mitigates this by performing the following steps:
- *
- * - Prior to the main verification pass, strongly connected components
- * (SCCs) are computed over the program's control flow graph,
- * intraprocedurally.
- *
- * - During the main verification pass, `maybe_enter_scc()` checks
- * whether the current verifier state is entering an SCC. If so, an
- * instance of a `bpf_scc_visit` object is created, and the state
- * entering the SCC is recorded as the entry state.
- *
- * - This instance is associated not with the SCC itself, but with a
- * `bpf_scc_callchain`: a tuple consisting of the call sites leading to
- * the SCC and the SCC id. See `compute_scc_callchain()`.
- *
- * - When a verification path encounters a `states_equal(...,
- * RANGE_WITHIN)` condition, there exists a call chain describing the
- * current state and a corresponding `bpf_scc_visit` instance. A copy
- * of the current state is created and added to
- * `bpf_scc_visit->backedges`.
- *
- * - When a verification path terminates, `maybe_exit_scc()` is called
- * from `bpf_update_branch_counts()`. For states with `branches == 0`, it
- * checks whether the state is the entry state of any `bpf_scc_visit`
- * instance. If it is, this indicates that all paths originating from
- * this SCC visit have been explored. `propagate_backedges()` is then
- * called, which propagates read and precision marks through the
- * backedges until a fixed point is reached.
- * (In the earlier example, this would propagate marks from A to B,
- * from C to A, and then again from A to B.)
- *
- * A note on callchains
- * --------------------
- *
- * Consider the following example:
- *
- * void foo() { loop { ... SCC#1 ... } }
- * void main() {
- * A: foo();
- * B: ...
- * C: foo();
- * }
- *
- * Here, there are two distinct callchains leading to SCC#1:
- * - (A, SCC#1)
- * - (C, SCC#1)
- *
- * Each callchain identifies a separate `bpf_scc_visit` instance that
- * accumulates backedge states. The `propagate_{liveness,precision}()`
- * functions traverse the parent state of each backedge state, which
- * means these parent states must remain valid (i.e., not freed) while
- * the corresponding `bpf_scc_visit` instance exists.
- *
- * Associating `bpf_scc_visit` instances directly with SCCs instead of
- * callchains would break this invariant:
- * - States explored during `C: foo()` would contribute backedges to
- * SCC#1, but SCC#1 would only be exited once the exploration of
- * `A: foo()` completes.
- * - By that time, the states explored between `A: foo()` and `C: foo()`
- * (i.e., `B: ...`) may have already been freed, causing the parent
- * links for states from `C: foo()` to become invalid.
- */
- if (loop) {
- struct bpf_scc_backedge *backedge;
-
- backedge = kzalloc_obj(*backedge,
- GFP_KERNEL_ACCOUNT);
- if (!backedge)
- return -ENOMEM;
- err = bpf_copy_verifier_state(&backedge->state, cur);
- backedge->state.equal_state = &sl->state;
- backedge->state.insn_idx = insn_idx;
- err = err ?: add_scc_backedge(env, &sl->state, backedge);
- if (err) {
- bpf_free_verifier_state(&backedge->state, false);
- kfree(backedge);
- return err;
- }
- }
- return 1;
- }
-miss:
- /* when new state is not going to be added do not increase miss count.
- * Otherwise several loop iterations will remove the state
- * recorded earlier. The goal of these heuristics is to have
- * states from some iterations of the loop (some in the beginning
- * and some at the end) to help pruning.
- */
- if (add_new_state)
- sl->miss_cnt++;
- /* heuristic to determine whether this state is beneficial
- * to keep checking from state equivalence point of view.
- * Higher numbers increase max_states_per_insn and verification time,
- * but do not meaningfully decrease insn_processed.
- * 'n' controls how many times state could miss before eviction.
- * Use bigger 'n' for checkpoints because evicting checkpoint states
- * too early would hinder iterator convergence.
- */
- n = (bpf_is_force_checkpoint(env, insn_idx) || loop_checkpoint) &&
- sl->state.branches > 0 ? 64 : 3;
- if (sl->miss_cnt > sl->hit_cnt * n + n) {
- /* the state is unlikely to be useful. Remove it to
- * speed up verification
- */
- sl->in_free_list = true;
- list_del(&sl->node);
- list_add(&sl->node, &env->free_list);
- env->free_list_size++;
- env->explored_states_size--;
- maybe_free_verifier_state(env, sl);
- }
- }
-
- if (env->max_states_per_insn < states_cnt)
- env->max_states_per_insn = states_cnt;
-
- if (!env->bpf_capable && states_cnt > BPF_COMPLEXITY_LIMIT_STATES)
- return 0;
-
- if (!add_new_state)
- return 0;
-
- /* There were no equivalent states, remember the current one.
- * Technically the current state is not proven to be safe yet,
- * but it will either reach outer most bpf_exit (which means it's safe)
- * or it will be rejected. When there are no loops the verifier won't be
- * seeing this tuple (frame[0].callsite, frame[1].callsite, .. insn_idx)
- * again on the way to bpf_exit.
- * When looping the sl->state.branches will be > 0 and this state
- * will not be considered for equivalence until branches == 0.
- */
- return bpf_split_cur_state(env);
-}
-
int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
{
return clean_verifier_state(env, st);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e55ba1db2e1f..2c385a7d459c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -202,8 +202,6 @@ struct bpf_verifier_stack_elem {
u64 diag_log_pos;
};
-#define BPF_COMPLEXITY_LIMIT_STATES 64
-
#define BPF_GLOBAL_PERCPU_MA_MAX_SIZE 512
#define BPF_PRIV_STACK_MIN_SIZE 64
@@ -1788,14 +1786,9 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
dst_state->flow_entry = src->flow_entry;
dst_state->insn_idx = src->insn_idx;
dst_state->curframe = src->curframe;
- dst_state->branches = src->branches;
dst_state->parent = src->parent;
dst_state->first_insn_idx = src->first_insn_idx;
dst_state->last_insn_idx = src->last_insn_idx;
- dst_state->dfs_depth = src->dfs_depth;
- dst_state->callback_unroll_depth = src->callback_unroll_depth;
- dst_state->may_goto_depth = src->may_goto_depth;
- dst_state->equal_state = src->equal_state;
for (i = 0; i <= src->curframe; i++) {
dst = dst_state->frame[i];
if (!dst) {
@@ -1811,45 +1804,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
return 0;
}
-static u32 state_htab_size(struct bpf_verifier_env *env)
-{
- return env->prog->len;
-}
-
-struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx)
-{
- struct bpf_verifier_state *cur = env->cur_state;
- struct bpf_func_state *state = cur->frame[cur->curframe];
-
- return &env->explored_states[(idx ^ state->callsite) % state_htab_size(env)];
-}
-
-static bool same_callsites(struct bpf_verifier_state *a, struct bpf_verifier_state *b)
-{
- int fr;
-
- if (a->curframe != b->curframe)
- return false;
-
- for (fr = a->curframe; fr >= 0; fr--)
- if (a->frame[fr]->callsite != b->frame[fr]->callsite)
- return false;
-
- return true;
-}
-
-void bpf_free_backedges(struct bpf_scc_visit *visit)
-{
- struct bpf_scc_backedge *backedge, *next;
-
- for (backedge = visit->backedges; backedge; backedge = next) {
- bpf_free_verifier_state(&backedge->state, false);
- next = backedge->next;
- kfree(backedge);
- }
- visit->backedges = NULL;
-}
-
static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx,
int *insn_idx, bool pop_log)
{
@@ -1915,18 +1869,6 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
if (err)
return ERR_PTR(-ENOMEM);
elem->st.speculative |= speculative;
- if (elem->st.parent) {
- ++elem->st.parent->branches;
- /* WARN_ON(branches > 2) technically makes sense here,
- * but
- * 1. speculative states will bump 'branches' for non-branch
- * instructions
- * 2. is_state_visited() heuristics may decide not to create
- * a new state for a sequence of branches and all such current
- * and cloned states will be pointing to a single parent state
- * which might have large 'branches' count.
- */
- }
return &elem->st;
}
@@ -2598,7 +2540,6 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
* This is async callback. It starts in a fresh stack.
* Initialize it similar to do_check_common().
*/
- elem->st.branches = 1;
elem->st.in_sleepable = is_sleepable;
frame = kzalloc_obj(*frame, GFP_KERNEL_ACCOUNT);
if (!frame)
@@ -8954,85 +8895,6 @@ static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *
return 0;
}
-/* Look for a previous loop entry at insn_idx: nearest parent state
- * stopped at insn_idx with callsites matching those in cur->frame.
- */
-static struct bpf_verifier_state *find_prev_entry(struct bpf_verifier_env *env,
- struct bpf_verifier_state *cur,
- int insn_idx)
-{
- struct bpf_verifier_state_list *sl;
- struct bpf_verifier_state *st;
- struct list_head *pos, *head;
-
- /* Explored states are pushed in stack order, most recent states come first */
- head = bpf_explored_state(env, insn_idx);
- list_for_each(pos, head) {
- sl = container_of(pos, struct bpf_verifier_state_list, node);
- /* If st->branches != 0 state is a part of current DFS verification path,
- * hence cur & st for a loop.
- */
- st = &sl->state;
- if (st->insn_idx == insn_idx && st->branches && same_callsites(st, cur) &&
- st->dfs_depth < cur->dfs_depth)
- return st;
- }
-
- return NULL;
-}
-
-/*
- * Check if scalar registers are exact for the purpose of not widening.
- * More lenient than regs_exact()
- */
-static bool scalars_exact_for_widen(const struct bpf_reg_state *rold,
- const struct bpf_reg_state *rcur)
-{
- return !memcmp(rold, rcur, offsetof(struct bpf_reg_state, id));
-}
-
-static void maybe_widen_reg(struct bpf_verifier_env *env,
- struct bpf_reg_state *rold, struct bpf_reg_state *rcur)
-{
- if (rold->type != SCALAR_VALUE)
- return;
- if (rold->type != rcur->type)
- return;
- if (rold->precise || rcur->precise || scalars_exact_for_widen(rold, rcur))
- return;
- __mark_reg_unknown(env, rcur);
-}
-
-static int widen_imprecise_scalars(struct bpf_verifier_env *env,
- struct bpf_verifier_state *old,
- struct bpf_verifier_state *cur)
-{
- struct bpf_func_state *fold, *fcur;
- int i, fr, num_slots;
-
- for (fr = old->curframe; fr >= 0; fr--) {
- fold = old->frame[fr];
- fcur = cur->frame[fr];
-
- for (i = 0; i < MAX_BPF_REG; i++)
- maybe_widen_reg(env,
- &fold->regs[i],
- &fcur->regs[i]);
-
- num_slots = min(bpf_stack_nr_slots(fold), bpf_stack_nr_slots(fcur));
- for (i = 0; i < num_slots; i++) {
- struct bpf_stack_state *old_ss = bpf_stack_slot(fold, i);
- struct bpf_stack_state *cur_ss = bpf_stack_slot(fcur, i);
-
- if (!bpf_is_spilled_reg(old_ss) || !bpf_is_spilled_reg(cur_ss))
- continue;
-
- maybe_widen_reg(env, &old_ss->spilled_ptr, &cur_ss->spilled_ptr);
- }
- }
- return 0;
-}
-
static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_st,
struct bpf_call_arg_meta *meta)
{
@@ -11703,7 +11565,6 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins
if (err)
return err;
- callback_state->callback_unroll_depth++;
callback_state->frame[callback_state->curframe - 1]->callback_depth++;
caller->callback_depth = 0;
return 0;
@@ -12676,10 +12537,9 @@ static bool retval_range_within(struct bpf_retval_range range, const struct bpf_
static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
{
- struct bpf_verifier_state *state = env->cur_state, *prev_st;
+ struct bpf_verifier_state *state = env->cur_state;
struct bpf_func_state *caller, *callee;
struct bpf_reg_state *r0;
- bool in_callback_fn;
u32 i, nregs;
int err;
@@ -12767,11 +12627,9 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
return err;
/* for callbacks like bpf_loop or bpf_for_each_map_elem go back to callsite,
- * there function call logic would reschedule callback visit. If iteration
- * converges is_state_visited() would prune that visit eventually.
+ * there function call logic would reschedule callback visit.
*/
- in_callback_fn = callee->in_callback_fn;
- if (in_callback_fn)
+ if (callee->in_callback_fn)
*insn_idx = callee->callsite;
else
*insn_idx = callee->callsite + 1;
@@ -12788,24 +12646,6 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
free_func_state(callee);
state->frame[state->curframe--] = NULL;
invalidate_outgoing_stack_args(env, caller);
-
- /* for callbacks widen imprecise scalars to make programs like below verify:
- *
- * struct ctx { int i; }
- * void cb(int idx, struct ctx *ctx) { ctx->i++; ... }
- * ...
- * struct ctx = { .i = 0; }
- * bpf_loop(100, cb, &ctx, 0);
- *
- * This is similar to what is done in process_iter_next_call() for open
- * coded iterators.
- */
- prev_st = in_callback_fn ? find_prev_entry(env, state, *insn_idx) : NULL;
- if (prev_st) {
- err = widen_imprecise_scalars(env, prev_st, state);
- if (err)
- return err;
- }
return 0;
}
@@ -19690,19 +19530,14 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
}
if (opcode == BPF_JCOND) {
- struct bpf_verifier_state *cur_st = env->cur_state, *queued_st, *prev_st;
+ struct bpf_verifier_state *queued_st;
int idx = *insn_idx;
- prev_st = find_prev_entry(env, cur_st->parent, idx);
-
/* branch out 'fallthrough' insn as a new state to explore */
queued_st = push_stack(env, idx + 1, idx, false);
if (IS_ERR(queued_st))
return PTR_ERR(queued_st);
- queued_st->may_goto_depth++;
- if (prev_st)
- widen_imprecise_scalars(env, prev_st, queued_st);
*insn_idx += insn->off;
return 0;
}
@@ -21175,7 +21010,6 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
case BPF_JMP32: {
u8 opcode = BPF_OP(insn->code);
- env->jmps_processed++;
if (opcode == BPF_CALL) {
if (env->cur_state->active_locks) {
/* similar to static subprog calls callx is allowed under a lock */
@@ -23000,46 +22834,10 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
static void free_states(struct bpf_verifier_env *env)
{
- struct bpf_verifier_state_list *sl;
- struct list_head *head, *pos, *tmp;
- struct bpf_scc_info *info;
- int i, j;
-
bpf_free_verifier_state(env->cur_state, true);
env->cur_state = NULL;
while (!pop_stack(env, NULL, NULL, false));
bpf_flow_reset(env);
-
- list_for_each_safe(pos, tmp, &env->free_list) {
- sl = container_of(pos, struct bpf_verifier_state_list, node);
- bpf_free_verifier_state(&sl->state, false);
- kfree(sl);
- }
- INIT_LIST_HEAD(&env->free_list);
-
- for (i = 0; i < env->scc_cnt; ++i) {
- info = env->scc_info[i];
- if (!info)
- continue;
- for (j = 0; j < info->num_visits; j++)
- bpf_free_backedges(&info->visits[j]);
- kvfree(info);
- env->scc_info[i] = NULL;
- }
-
- if (!env->explored_states)
- return;
-
- for (i = 0; i < state_htab_size(env); i++) {
- head = &env->explored_states[i];
-
- list_for_each_safe(pos, tmp, head) {
- sl = container_of(pos, struct bpf_verifier_state_list, node);
- bpf_free_verifier_state(&sl->state, false);
- kfree(sl);
- }
- INIT_LIST_HEAD(&env->explored_states[i]);
- }
}
static int __do_check_common(struct bpf_verifier_env *env, int subprog,
@@ -23062,7 +22860,6 @@ static int __do_check_common(struct bpf_verifier_env *env, int subprog,
return -ENOMEM;
state->curframe = 0;
state->speculative = false;
- state->branches = 1;
state->in_sleepable = context == BPF_FUNC_CTX_SLEEPABLE;
/* nobody knows what the callers of a global function or of an extension hold */
state->callers_unknown = subprog || env->prog->type == BPF_PROG_TYPE_EXT;
@@ -25250,21 +25047,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (attr->prog_flags & BPF_F_ANY_ALIGNMENT)
env->strict_alignment = false;
- if (is_priv)
- env->test_state_freq = attr->prog_flags & BPF_F_TEST_STATE_FREQ;
env->test_reg_invariants = attr->prog_flags & BPF_F_TEST_REG_INVARIANTS;
- env->explored_states = kvzalloc_objs(struct list_head,
- state_htab_size(env),
- GFP_KERNEL_ACCOUNT);
- ret = -ENOMEM;
- if (!env->explored_states)
- goto skip_full_check;
-
- for (i = 0; i < state_htab_size(env); i++)
- INIT_LIST_HEAD(&env->explored_states[i]);
- INIT_LIST_HEAD(&env->free_list);
-
/* Prepare BTF and func_info needed to discover all subprograms. */
ret = bpf_prepare_btf_info(env, attr, uattr);
if (ret < 0)
@@ -25409,8 +25193,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
ret = bpf_prog_offload_finalize(env);
skip_full_check:
- kvfree(env->explored_states);
-
/* might decrease stack depth, keep it before passes that
* allocate additional slots.
*/
@@ -25565,7 +25347,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
bpf_stack_liveness_free(env);
kvfree(env->cfg.postorder_nums);
kvfree(env->cfg.insn_postorder);
- kvfree(env->scc_info);
kvfree(env->succ);
kvfree(env->gotox_tmp_buf);
kvfree(env->callx_edges);
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 61/81] bpf: Remove precision tracking
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (59 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 60/81] bpf: Remove is_state_visited() Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 62/81] bpf: Remove prune points, levels of state comparison and pop_stack() Alexei Starovoitov
` (21 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Precision tracking is a part of pruning. Before
"bpf: Verify programs as a fixpoint over the control flow graph" the
verifier followed each path of a prog. It pruned a path when a state that
it saw before at the same insn covered the current state. To prune more
paths, the comparison ignored a scalar unless a subsequent check depended
on its value. To find these scalars, the verifier went back from each check
through the history of the path and marked them as precise.
Now the verifier joins the states that meet at the start of a basic block
and differ in a scalar. The result is one state that covers the two.
r1=0 and r1=1 meet pruning join
----------------------- -------------------------- ---------------
a check depends on r1 r1 is precise, verify r1=[0,1]
verify the two states
no check depends on r1 drop the second state verify r1=[0,1]
Each subsequent check sees the full range of r1. Thus the verifier does not
have to know which scalars a check depends on. Nothing uses the marks.
Some arguments of helpers and kfuncs must be known constants, and the join
of two constants is not a constant. For this case flow.c has
flow_compute_need_const(). It finds the registers and stack slots that go
into such arguments, one time, before the main pass. The verifier does not
join states that differ in them.
Remove the 'precise' bit, backtrack.c, jmp_history and the parent links of
states. The verifier log has no "mark_precise:" lines and no 'P' in
"R0=Pscalar()".
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 106 +---
kernel/bpf/Makefile | 2 +-
kernel/bpf/backtrack.c | 1048 ----------------------------------
kernel/bpf/log.c | 2 -
kernel/bpf/scev.c | 49 +-
kernel/bpf/states.c | 9 +-
kernel/bpf/verifier.c | 426 ++------------
7 files changed, 40 insertions(+), 1602 deletions(-)
delete mode 100644 kernel/bpf/backtrack.c
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index d704b3776b68..84deb3974c2e 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -184,9 +184,7 @@ struct bpf_reg_state {
* other registers. Kept outside the metadata union for ID remapping
* during state comparisons.
*/
- u32 map_uid:31;
- /* if (!precise && SCALAR_VALUE) min/max/tnum don't affect safety */
- u32 precise:1;
+ u32 map_uid;
};
static inline s64 reg_smin(const struct bpf_reg_state *reg)
@@ -334,7 +332,6 @@ struct bpf_retval_range {
struct bpf_loop_iters {
u32 min_header_count; /* min number of times header is executed */
u32 max_header_count; /* max number of times header is executed */
- unsigned long precise_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
/* live, changed by the loop, and there is no summary of how */
unsigned long forget_regs[BITS_TO_LONGS(BPF_SCEV_REGS_NUM)];
/* [0, max] at the entry, and the loop only clears bits */
@@ -460,44 +457,6 @@ struct bpf_func_state {
#define MAX_CALL_FRAMES 32
-/* instruction history flags, used in bpf_jmp_history_entry.flags field.
- * Frame number and SPI are stored in dedicated fields of bpf_jmp_history_entry.
- */
-enum {
- INSN_F_STACK_ACCESS = BIT(0),
-
- INSN_F_DST_REG_STACK = BIT(1), /* dst_reg is PTR_TO_STACK */
- INSN_F_SRC_REG_STACK = BIT(2), /* src_reg is PTR_TO_STACK */
-
- INSN_F_STACK_ARG_ACCESS = BIT(3),
-};
-
-/* Registers linked to one jump condition that a history entry can record */
-#define BPF_LINKED_REGS_MAX 5
-
-struct bpf_jmp_history_entry {
- /* insn idx can't be bigger than 1 million */
- u32 idx : 20;
- u32 frame : 5; /* stack access frame number */
- /* special INSN_F_xxx flags */
- u32 flags : 4;
- u32 : 3;
- u32 prev_idx : 20;
- u32 spi : 12; /* stack slot index */
- /*
- * Scalar registers and spilled scalars linked to the condition of
- * this jump, which need precision tracking together when the jump is
- * backtracked. Each is packed as 4 bits of frame number, one bit
- * telling a register from a stack slot and 11 bits of register or
- * slot index, see linked_regs_pack().
- */
- u16 linked_regs[BPF_LINKED_REGS_MAX];
- u8 linked_regs_cnt;
-};
-
-static_assert(MAX_CALL_FRAMES <= (1 << 5));
-static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
-
/* Maximum number of bpf_reg_state objects that can exist at once */
#define MAX_STACK_ARG_SLOTS (MAX_BPF_STATIC_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS)
#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK_SLOTS + MAX_STACK_ARG_SLOTS) * \
@@ -506,7 +465,6 @@ static_assert(MAX_BPF_STACK_SLOTS <= (1 << 12));
struct bpf_verifier_state {
/* call stack tracking */
struct bpf_func_state *frame[MAX_CALL_FRAMES];
- struct bpf_verifier_state *parent;
/* Acquired reference states */
struct bpf_reference_state *refs;
u32 insn_idx;
@@ -534,19 +492,8 @@ struct bpf_verifier_state {
*/
u32 flow_entry;
- /*
- * First and last insn idx of this verifier state.
- * last_insn_idx is -1 if no instructions have been executed yet.
- */
- u32 first_insn_idx;
+ /* The insn that the state comes from, -1 if no instructions have been executed yet. */
u32 last_insn_idx;
- /* jmp history recorded from first to last.
- * backtracking is using it to go from last to first.
- * For most states jmp_history_cnt is [0-3].
- * For loops can go up to ~40.
- */
- struct bpf_jmp_history_entry *jmp_history;
- u32 jmp_history_cnt;
};
/* Number of BPF_REG_SIZE stack slots tracked for the frame so far. */
@@ -1022,14 +969,6 @@ static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub)
struct bpf_diag;
struct bpf_verifier_env;
-struct backtrack_state {
- struct bpf_verifier_env *env;
- u32 frame;
- u32 reg_masks[MAX_CALL_FRAMES];
- unsigned long stack_masks[MAX_CALL_FRAMES][BITS_TO_LONGS(MAX_BPF_STACK_SLOTS)];
- u32 stack_arg_masks[MAX_CALL_FRAMES];
-};
-
struct bpf_id_pair {
u32 old;
u32 cur;
@@ -1143,8 +1082,6 @@ struct bpf_verifier_env {
u32 gotox_edges;
bool subprog_jts_ready;
} cfg;
- struct backtrack_state bt;
- struct bpf_jmp_history_entry *cur_hist_ent;
/* Per-callsite snapshot of the parent's tracked spill slots for cross-frame fills. */
struct spill_snapshot **callsite_at_stack;
u32 pass_cnt; /* number of times do_check() was called */
@@ -1387,7 +1324,6 @@ int bpf_check_ext_prologue(struct bpf_verifier_log *log,
const struct bpf_prog *tgt_prog, u32 btf_id);
void bpf_free_kfunc_btf_tab(struct bpf_kfunc_btf_tab *tab);
-int mark_chain_precision(struct bpf_verifier_env *env, int regno);
int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
@@ -1431,25 +1367,15 @@ bool bpf_insn_leaves_loop(struct bpf_verifier_env *env, int h, int n);
bool bpf_jump_leaves_loop(struct bpf_verifier_env *env, int h, int n);
int bpf_insert_recursion_checks(struct bpf_verifier_env *env);
-void bpf_clear_jmp_history(struct bpf_verifier_state *state);
int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
const struct bpf_verifier_state *src);
void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self);
-int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
- int insn_flags, int spi, int frame, const u16 *linked_regs,
- u8 linked_regs_cnt);
-void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist);
void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
struct bpf_reg_state *reg);
void bpf_mark_reg_known_scalar(struct bpf_reg_state *reg, u64 imm);
-void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg);
+void bpf_mark_reg_unknown(struct bpf_reg_state *reg);
int bpf_grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size);
-void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,
- struct bpf_verifier_state *st);
void bpf_clear_singular_ids(struct bpf_verifier_env *env, struct bpf_verifier_state *st);
-int bpf_mark_chain_precision(struct bpf_verifier_env *env,
- struct bpf_verifier_state *starting_state,
- int regno, bool *changed);
static inline int bpf_get_spi(s32 off)
{
@@ -1526,31 +1452,6 @@ static inline bool bpf_register_is_null(struct bpf_reg_state *reg)
return reg->type == SCALAR_VALUE && tnum_equals_const(reg->var_off, 0);
}
-static inline void bpf_bt_set_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg)
-{
- bt->reg_masks[frame] |= 1 << reg;
-}
-
-static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
- __set_bit(slot, bt->stack_masks[frame]);
-}
-
-static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
- bt->stack_arg_masks[frame] |= 1 << slot;
-}
-
-static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u32 reg)
-{
- return bt->reg_masks[frame] & (1 << reg);
-}
-
-static inline bool bt_is_frame_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)
-{
- return test_bit(slot, bt->stack_masks[frame]);
-}
-
bool bpf_map_is_rdonly(const struct bpf_map *map);
struct bpf_func_ptr *bpf_map_func_ptrs(struct bpf_verifier_env *env,
const struct bpf_map *map, u32 *cnt);
@@ -1750,7 +1651,6 @@ struct bpf_subprog_info *bpf_find_containing_subprog(struct bpf_verifier_env *en
const char *bpf_subprog_name(const struct bpf_verifier_env *env, int subprog);
int bpf_jmp_offset(struct bpf_insn *insn);
struct bpf_iarray *bpf_insn_successors(struct bpf_verifier_env *env, u32 idx);
-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask);
bool bpf_subprog_is_global(const struct bpf_verifier_env *env, int subprog);
/* Kinds of member a by-value struct or union may be composed of. */
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index d5a79e1ea933..d26c62326e58 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -11,7 +11,7 @@ obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
-obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o backtrack.o check_btf.o flow.o join.o
+obj-$(CONFIG_BPF_SYSCALL) += fixups.o cfg.o states.o check_btf.o flow.o join.o
obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
obj-$(CONFIG_BPF_JIT) += trampoline.o
diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c
deleted file mode 100644
index a42b8b830b0b..000000000000
--- a/kernel/bpf/backtrack.c
+++ /dev/null
@@ -1,1048 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
-#include <linux/bpf.h>
-#include <linux/bpf_verifier.h>
-#include <linux/filter.h>
-#include <linux/bitmap.h>
-
-#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
-
-/* for any branch, call, exit record the history of jmps in the given state */
-int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
- int insn_flags, int spi, int frame, const u16 *linked_regs,
- u8 linked_regs_cnt)
-{
- u32 cnt = cur->jmp_history_cnt;
- struct bpf_jmp_history_entry *p;
- size_t alloc_size;
-
- /* FLOW: nothing looks at the history */
- return 0;
-
- /* combine instruction flags if we already recorded this instruction */
- if (env->cur_hist_ent) {
- /* atomic instructions push insn_flags twice, for READ and
- * WRITE sides, but they should agree on stack slot
- */
- verifier_bug_if((env->cur_hist_ent->flags & insn_flags) &&
- (env->cur_hist_ent->flags & insn_flags) != insn_flags,
- env, "insn history: insn_idx %d cur flags %x new flags %x",
- env->insn_idx, env->cur_hist_ent->flags, insn_flags);
- env->cur_hist_ent->flags |= insn_flags;
- env->cur_hist_ent->spi = spi;
- env->cur_hist_ent->frame = frame;
- verifier_bug_if(env->cur_hist_ent->linked_regs_cnt != 0, env,
- "insn history: insn_idx %d has %u linked regs",
- env->insn_idx, env->cur_hist_ent->linked_regs_cnt);
- if (linked_regs_cnt)
- memcpy(env->cur_hist_ent->linked_regs, linked_regs,
- linked_regs_cnt * sizeof(*linked_regs));
- env->cur_hist_ent->linked_regs_cnt = linked_regs_cnt;
- return 0;
- }
-
- cnt++;
- alloc_size = kmalloc_size_roundup(size_mul(cnt, sizeof(*p)));
- p = krealloc(cur->jmp_history, alloc_size, GFP_KERNEL_ACCOUNT);
- if (!p)
- return -ENOMEM;
- cur->jmp_history = p;
-
- p = &cur->jmp_history[cnt - 1];
- p->idx = env->insn_idx;
- p->prev_idx = env->prev_insn_idx;
- p->flags = insn_flags;
- p->spi = spi;
- p->frame = frame;
- if (linked_regs_cnt)
- memcpy(p->linked_regs, linked_regs, linked_regs_cnt * sizeof(*linked_regs));
- p->linked_regs_cnt = linked_regs_cnt;
- cur->jmp_history_cnt = cnt;
- env->cur_hist_ent = p;
-
- return 0;
-}
-
-static bool is_atomic_load_insn(const struct bpf_insn *insn)
-{
- return BPF_CLASS(insn->code) == BPF_STX &&
- BPF_MODE(insn->code) == BPF_ATOMIC &&
- insn->imm == BPF_LOAD_ACQ;
-}
-
-static bool is_atomic_fetch_insn(const struct bpf_insn *insn)
-{
- return BPF_CLASS(insn->code) == BPF_STX &&
- BPF_MODE(insn->code) == BPF_ATOMIC &&
- (insn->imm & BPF_FETCH);
-}
-
-/* Backtrack one insn at a time. If idx is not at the top of recorded
- * history then previous instruction came from straight line execution.
- * Return -ENOENT if we exhausted all instructions within given state.
- *
- * It's legal to have a bit of a looping with the same starting and ending
- * insn index within the same state, e.g.: 3->4->5->3, so just because current
- * instruction index is the same as state's first_idx doesn't mean we are
- * done. If there is still some jump history left, we should keep going. We
- * need to take into account that we might have a jump history between given
- * state's parent and itself, due to checkpointing. In this case, we'll have
- * history entry recording a jump from last instruction of parent state and
- * first instruction of given state.
- */
-static int get_prev_insn_idx(struct bpf_verifier_state *st, int i,
- u32 *history)
-{
- u32 cnt = *history;
-
- if (i == st->first_insn_idx) {
- if (cnt == 0)
- return -ENOENT;
- if (cnt == 1 && st->jmp_history[0].idx == i)
- return -ENOENT;
- }
-
- if (cnt && st->jmp_history[cnt - 1].idx == i) {
- i = st->jmp_history[cnt - 1].prev_idx;
- (*history)--;
- } else {
- i--;
- }
- return i;
-}
-
-static struct bpf_jmp_history_entry *get_jmp_hist_entry(struct bpf_verifier_state *st,
- u32 hist_end, int insn_idx)
-{
- if (hist_end > 0 && st->jmp_history[hist_end - 1].idx == insn_idx)
- return &st->jmp_history[hist_end - 1];
- return NULL;
-}
-
-static inline void bt_init(struct backtrack_state *bt, u32 frame)
-{
- bt->frame = frame;
-}
-
-static inline void bt_reset(struct backtrack_state *bt)
-{
- struct bpf_verifier_env *env = bt->env;
-
- memset(bt, 0, sizeof(*bt));
- bt->env = env;
-}
-
-static inline bool bt_frame_stack_empty(struct backtrack_state *bt, u32 frame)
-{
- return bitmap_empty(bt->stack_masks[frame], MAX_BPF_STACK_SLOTS);
-}
-
-static inline bool bt_stack_empty(struct backtrack_state *bt)
-{
- return bt_frame_stack_empty(bt, bt->frame);
-}
-
-static inline bool bt_empty(struct backtrack_state *bt)
-{
- u32 mask = 0;
- int i;
-
- for (i = 0; i <= bt->frame; i++) {
- mask |= bt->reg_masks[i] | bt->stack_arg_masks[i];
- if (!bt_frame_stack_empty(bt, i))
- return false;
- }
-
- return mask == 0;
-}
-
-static inline void bt_clear_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
- bt->stack_arg_masks[frame] &= ~(1 << slot);
-}
-
-static inline bool bt_is_frame_stack_arg_slot_set(struct backtrack_state *bt, u32 frame, u32 slot)
-{
- return bt->stack_arg_masks[frame] & (1 << slot);
-}
-
-static inline int bt_subprog_enter(struct backtrack_state *bt)
-{
- if (bt->frame == MAX_CALL_FRAMES - 1) {
- verifier_bug(bt->env, "subprog enter from frame %d", bt->frame);
- return -EFAULT;
- }
- bt->frame++;
- return 0;
-}
-
-static inline int bt_subprog_exit(struct backtrack_state *bt)
-{
- if (bt->frame == 0) {
- verifier_bug(bt->env, "subprog exit from frame 0");
- return -EFAULT;
- }
- bt->frame--;
- return 0;
-}
-
-static inline void bt_clear_frame_reg(struct backtrack_state *bt, u32 frame, u32 reg)
-{
- bt->reg_masks[frame] &= ~(1 << reg);
-}
-
-static inline void bt_set_reg(struct backtrack_state *bt, u32 reg)
-{
- bpf_bt_set_frame_reg(bt, bt->frame, reg);
-}
-
-static inline void bt_clear_reg(struct backtrack_state *bt, u32 reg)
-{
- bt_clear_frame_reg(bt, bt->frame, reg);
-}
-
-static inline void bt_clear_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)
-{
- __clear_bit(slot, bt->stack_masks[frame]);
-}
-
-static inline u32 bt_frame_reg_mask(struct backtrack_state *bt, u32 frame)
-{
- return bt->reg_masks[frame];
-}
-
-static inline u32 bt_reg_mask(struct backtrack_state *bt)
-{
- return bt->reg_masks[bt->frame];
-}
-
-static inline unsigned long *bt_frame_stack_mask(struct backtrack_state *bt, u32 frame)
-{
- return bt->stack_masks[frame];
-}
-
-static inline unsigned long *bt_stack_mask(struct backtrack_state *bt)
-{
- return bt_frame_stack_mask(bt, bt->frame);
-}
-
-static inline u32 bt_stack_arg_mask(struct backtrack_state *bt)
-{
- return bt->stack_arg_masks[bt->frame];
-}
-
-static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg)
-{
- return bt->reg_masks[bt->frame] & (1 << reg);
-}
-
-/* format registers bitmask, e.g., "r0,r2,r4" for 0x15 mask */
-static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
-{
- DECLARE_BITMAP(mask, 64);
- bool first = true;
- int i, n;
-
- buf[0] = '\0';
-
- bitmap_from_u64(mask, reg_mask);
- for_each_set_bit(i, mask, 32) {
- n = snprintf(buf, buf_sz, "%sr%d", first ? "" : ",", i);
- first = false;
- buf += n;
- buf_sz -= n;
- if (buf_sz < 0)
- break;
- }
-}
-
-/* format stack slots bitmask, e.g., "-8,-24,-40" for slots 0, 2 and 4 */
-void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask)
-{
- bool first = true;
- int i, n;
-
- buf[0] = '\0';
-
- for_each_set_bit(i, stack_mask, MAX_BPF_STACK_SLOTS) {
- n = snprintf(buf, buf_sz, "%s%d", first ? "" : ",", -(i + 1) * 8);
- first = false;
- buf += n;
- buf_sz -= n;
- if (buf_sz < 0)
- break;
- }
-}
-
-/* For given verifier state backtrack_insn() is called from the last insn to
- * the first insn. Its purpose is to compute a bitmask of registers and
- * stack slots that needs precision in the parent verifier state.
- *
- * @idx is an index of the instruction we are currently processing;
- * @subseq_idx is an index of the subsequent instruction that:
- * - *would be* executed next, if jump history is viewed in forward order;
- * - *was* processed previously during backtracking.
- */
-static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
- struct bpf_jmp_history_entry *hist, struct backtrack_state *bt)
-{
- struct bpf_insn *insn = env->prog->insnsi + idx;
- u8 class = BPF_CLASS(insn->code);
- u8 opcode = BPF_OP(insn->code);
- u8 mode = BPF_MODE(insn->code);
- u32 dreg = insn->dst_reg;
- u32 sreg = insn->src_reg;
- u32 spi, i, fr;
-
- if (insn->code == 0)
- return 0;
- if (env->log.level & BPF_LOG_LEVEL2) {
- fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_reg_mask(bt));
- verbose(env, "mark_precise: frame%d: regs=%s ",
- bt->frame, env->tmp_str_buf);
- bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
- verbose(env, "stack=%s before ", env->tmp_str_buf);
- verbose(env, "%d: ", idx);
- bpf_verbose_insn(env, insn);
- verbose(env, "\n");
- }
-
- /* If there is a history record that some registers gained range at this insn,
- * propagate precision marks to those registers, so that bt_is_reg_set()
- * accounts for these registers.
- */
- bpf_bt_sync_linked_regs(bt, hist);
-
- if (class == BPF_ALU || class == BPF_ALU64) {
- if (!bt_is_reg_set(bt, dreg))
- return 0;
- if (opcode == BPF_END || opcode == BPF_NEG) {
- /* sreg is reserved and unused
- * dreg still need precision before this insn
- */
- return 0;
- } else if (opcode == BPF_MOV) {
- if (BPF_SRC(insn->code) == BPF_X) {
- /* dreg = sreg or dreg = (s8, s16, s32)sreg
- * dreg needs precision after this insn
- * sreg needs precision before this insn
- */
- bt_clear_reg(bt, dreg);
- if (sreg != BPF_REG_FP && sreg != BPF_REG_ARENA_SP)
- bt_set_reg(bt, sreg);
- } else {
- /* dreg = K
- * dreg needs precision after this insn.
- * Corresponding register is already marked
- * as precise=true in this verifier state.
- * No further markings in parent are necessary
- */
- bt_clear_reg(bt, dreg);
- }
- } else {
- if (BPF_SRC(insn->code) == BPF_X) {
- /* dreg += sreg
- * both dreg and sreg need precision
- * before this insn
- */
- if (sreg != BPF_REG_FP)
- bt_set_reg(bt, sreg);
- } /* else dreg += K
- * dreg still needs precision before this insn
- */
- }
- } else if (class == BPF_LDX ||
- is_atomic_load_insn(insn) ||
- is_atomic_fetch_insn(insn)) {
- u32 load_reg = dreg;
-
- /*
- * Atomic fetch operation writes the old value into
- * a register (sreg or r0) and if it was tracked for
- * precision, propagate to the stack slot like we do
- * in regular ldx.
- */
- if (is_atomic_fetch_insn(insn))
- load_reg = insn->imm == BPF_CMPXCHG ?
- BPF_REG_0 : sreg;
-
- if (!bt_is_reg_set(bt, load_reg))
- return 0;
- bt_clear_reg(bt, load_reg);
-
- if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
- spi = hist->spi;
- /*
- * Stack arg read: callee reads from r11+off, but
- * the data lives in the caller's stack_arg_regs.
- * Set the mask in the caller frame so precision
- * is marked in the caller's slot at the callee
- * entry checkpoint.
- */
- bt_set_frame_stack_arg_slot(bt, bt->frame - 1, spi);
- return 0;
- }
-
- /* scalars can only be spilled into stack w/o losing precision.
- * Load from any other memory can be zero extended.
- * The desire to keep that precision is already indicated
- * by 'precise' mark in corresponding register of this state.
- * No further tracking necessary.
- */
- if (!hist || !(hist->flags & INSN_F_STACK_ACCESS))
- return 0;
- /* dreg = *(u64 *)[fp - off] was a fill from the stack.
- * that [fp - off] slot contains scalar that needs to be
- * tracked with precision
- */
- spi = hist->spi;
- fr = hist->frame;
- bpf_bt_set_frame_slot(bt, fr, spi);
- } else if (class == BPF_STX || class == BPF_ST) {
- if (bt_is_reg_set(bt, dreg))
- /* stx & st shouldn't be using _scalar_ dst_reg
- * to access memory. It means backtracking
- * encountered a case of pointer subtraction.
- */
- return -ENOTSUPP;
-
- if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) {
- spi = hist->spi;
- if (!bt_is_frame_stack_arg_slot_set(bt, bt->frame, spi))
- return 0;
- bt_clear_frame_stack_arg_slot(bt, bt->frame, spi);
- if (class == BPF_STX)
- bt_set_reg(bt, sreg);
- return 0;
- }
-
- /* scalars can only be spilled into stack */
- if (!hist || !(hist->flags & INSN_F_STACK_ACCESS))
- return 0;
- spi = hist->spi;
- fr = hist->frame;
- if (!bt_is_frame_slot_set(bt, fr, spi))
- return 0;
- bt_clear_frame_slot(bt, fr, spi);
- if (class == BPF_STX)
- bt_set_reg(bt, sreg);
- } else if (class == BPF_JMP || class == BPF_JMP32) {
- if (bpf_pseudo_call(insn) || bpf_is_callx(insn)) {
- int subprog_insn_idx, subprog = -1;
-
- if (bpf_pseudo_call(insn)) {
- subprog_insn_idx = idx + insn->imm + 1;
- subprog = bpf_find_subprog(env, subprog_insn_idx);
- if (subprog < 0)
- return -EFAULT;
- }
-
- /* callx calls static subprogs only */
- if (subprog >= 0 && bpf_subprog_is_global(env, subprog)) {
- /* check that jump history doesn't have any
- * extra instructions from subprog; the next
- * instruction after call to global subprog
- * should be literally next instruction in
- * caller program
- */
- verifier_bug_if(idx + 1 != subseq_idx, env,
- "extra insn from subprog");
- /* global subprog always sets R0 */
- bt_clear_reg(bt, BPF_REG_0);
- /* and if it does not set R2, main pass would catch it */
- bt_clear_reg(bt, BPF_REG_2);
- /* r1-r5 are invalidated after subprog call,
- * so for global func call it shouldn't be set
- * anymore
- */
- if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
- verifier_bug(env, "global subprog unexpected regs %x",
- bt_reg_mask(bt));
- return -EFAULT;
- }
- return 0;
- } else {
- /* static subprog call instruction, which
- * means that we are exiting current subprog,
- * so only r1-r5 could be still requested as
- * precise, r0 and r6-r10 or any stack slot in
- * the current frame should be zero by now
- */
- if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) {
- verifier_bug(env, "static subprog unexpected regs %x",
- bt_reg_mask(bt));
- return -EFAULT;
- }
- /* we are now tracking register spills correctly,
- * so any instance of leftover slots is a bug
- */
- if (!bt_stack_empty(bt)) {
- bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
- bt_stack_mask(bt));
- verifier_bug(env, "static subprog leftover stack slots %s",
- env->tmp_str_buf);
- return -EFAULT;
- }
- /* propagate r1-r5 to the caller */
- for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
- if (bt_is_reg_set(bt, i)) {
- bt_clear_reg(bt, i);
- bpf_bt_set_frame_reg(bt, bt->frame - 1, i);
- }
- }
- if (bt_stack_arg_mask(bt)) {
- verifier_bug(env,
- "static subprog leftover stack arg slots %x",
- bt_stack_arg_mask(bt));
- return -EFAULT;
- }
- if (bt_subprog_exit(bt))
- return -EFAULT;
- return 0;
- }
- } else if (bpf_is_sync_callback_calling_insn(insn) && idx != subseq_idx - 1) {
- /* exit from callback subprog to callback-calling helper or
- * kfunc call. Use idx/subseq_idx check to discern it from
- * straight line code backtracking.
- * Unlike the subprog call handling above, we shouldn't
- * propagate precision of r1-r5 (if any requested), as they are
- * not actually arguments passed directly to callback subprogs
- */
- if (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) {
- verifier_bug(env, "callback unexpected regs %x",
- bt_reg_mask(bt));
- return -EFAULT;
- }
- if (!bt_stack_empty(bt)) {
- bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
- bt_stack_mask(bt));
- verifier_bug(env, "callback leftover stack slots %s",
- env->tmp_str_buf);
- return -EFAULT;
- }
- /* clear r1-r5 in callback subprog's mask */
- for (i = BPF_REG_1; i <= BPF_REG_5; i++)
- bt_clear_reg(bt, i);
- if (bt_subprog_exit(bt))
- return -EFAULT;
- return 0;
- } else if (opcode == BPF_CALL) {
- /* kfunc with imm==0 is invalid and fixup_kfunc_call will
- * catch this error later. Make backtracking conservative
- * with ENOTSUPP.
- */
- if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL && insn->imm == 0)
- return -ENOTSUPP;
- /* regular helper call sets R0 */
- bt_clear_reg(bt, BPF_REG_0);
- /* kfunc might also set R2 */
- bt_clear_reg(bt, BPF_REG_2);
- if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
- /* if backtracking was looking for registers R1-R5
- * they should have been found already.
- */
- verifier_bug(env, "backtracking call unexpected regs %x",
- bt_reg_mask(bt));
- return -EFAULT;
- }
- if (insn->src_reg == BPF_REG_0 && insn->imm == BPF_FUNC_tail_call
- && subseq_idx - idx != 1) {
- if (bt_subprog_enter(bt))
- return -EFAULT;
- }
- } else if (opcode == BPF_EXIT) {
- bool from_subprog_call, r0_precise, r2_precise;
-
- /* BPF_EXIT in subprog or callback always returns
- * right after the call instruction, so by checking
- * whether the instruction at subseq_idx-1 is subprog
- * call or not we can distinguish actual exit from
- * *subprog* from exit from *callback*. In the former
- * case, we need to propagate the precision of the
- * return registers, if necessary. In the latter we
- * never do that.
- */
- from_subprog_call = subseq_idx - 1 >= 0 &&
- (bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) ||
- bpf_is_callx(&env->prog->insnsi[subseq_idx - 1]));
-
- r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
- r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
-
- /* Backtracking to a nested function call, 'idx' is a part of
- * the inner frame 'subseq_idx' is a part of the outer frame.
- * In case of a regular function call, the callee defines the
- * return registers R0 and R2, so clear them before checking
- * that instructions giving precision to registers R1-R5 have
- * been found already. R2 is also an argument register, hence
- * it has to be cleared before that check.
- * In case of a callback from bpf_loop(), R{1,4} in the calling
- * frame would be set as precise and that is correct, and R2
- * might be a precise helper argument as well, so leave it set.
- */
- bt_clear_reg(bt, BPF_REG_0);
- if (from_subprog_call)
- bt_clear_reg(bt, BPF_REG_2);
- if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) {
- verifier_bug(env, "backtracking exit unexpected regs %x",
- bt_reg_mask(bt));
- return -EFAULT;
- }
-
- if (bt_subprog_enter(bt))
- return -EFAULT;
-
- if (r0_precise)
- bt_set_reg(bt, BPF_REG_0);
- if (r2_precise)
- bt_set_reg(bt, BPF_REG_2);
- /* r6-r9 and stack slots will stay set in caller frame
- * bitmasks until we return back from callee(s)
- */
- return 0;
- } else if (BPF_SRC(insn->code) == BPF_X) {
- if (!bt_is_reg_set(bt, dreg) && !bt_is_reg_set(bt, sreg))
- return 0;
- /* dreg <cond> sreg
- * Both dreg and sreg need precision before
- * this insn. If only sreg was marked precise
- * before it would be equally necessary to
- * propagate it to dreg.
- */
- if (!hist || !(hist->flags & INSN_F_SRC_REG_STACK))
- bt_set_reg(bt, sreg);
- if (!hist || !(hist->flags & INSN_F_DST_REG_STACK))
- bt_set_reg(bt, dreg);
- } else if (BPF_SRC(insn->code) == BPF_K) {
- /* dreg <cond> K
- * Only dreg still needs precision before
- * this insn, so for the K-based conditional
- * there is nothing new to be marked.
- */
- }
- } else if (class == BPF_LD) {
- /* It's ld_imm64 or ld_abs or ld_ind.
- * For ld_imm64 no further tracking of precision
- * into parent is necessary
- */
- if (mode == BPF_IMM) {
- bt_clear_reg(bt, dreg);
- return 0;
- }
- /*
- * BPF_{IND,ABS} are modelled as two branches:
- * - fallthrough;
- * - implicit subprogram exit.
- * It is necessary to switch current frame if
- * implicit subprogram exit branch is backtracked.
- */
- if (mode == BPF_IND || mode == BPF_ABS) {
- if (bt_is_reg_set(bt, dreg))
- return -ENOTSUPP;
- if (subseq_idx != idx + 1)
- if (bt_subprog_enter(bt))
- return -EFAULT;
- return 0;
- }
- }
- /* Propagate precision marks to linked registers, to account for
- * registers marked as precise in this function.
- */
- bpf_bt_sync_linked_regs(bt, hist);
- return 0;
-}
-
-/* the scalar precision tracking algorithm:
- * . at the start all registers have precise=false.
- * . scalar ranges are tracked as normal through alu and jmp insns.
- * . once precise value of the scalar register is used in:
- * . ptr + scalar alu
- * . if (scalar cond K|scalar)
- * . helper_call(.., scalar, ...) where ARG_CONST is expected
- * backtrack through the verifier states and mark all registers and
- * stack slots with spilled constants that these scalar registers
- * should be precise.
- * . during state pruning two registers (or spilled stack slots)
- * are equivalent if both are not precise.
- *
- * Note the verifier cannot simply walk register parentage chain,
- * since many different registers and stack slots could have been
- * used to compute single precise scalar.
- *
- * The approach of starting with precise=true for all registers and then
- * backtrack to mark a register as not precise when the verifier detects
- * that program doesn't care about specific value (e.g., when helper
- * takes register as ARG_ANYTHING parameter) is not safe.
- *
- * It's ok to walk single parentage chain of the verifier states.
- * It's possible that this backtracking will go all the way till 1st insn.
- * All other branches will be explored for needing precision later.
- *
- * The backtracking needs to deal with cases like:
- * R8=map_value(id=0,off=0,ks=4,vs=1952,imm=0) R9_w=map_value(id=0,off=40,ks=4,vs=1952,imm=0)
- * r9 -= r8
- * r5 = r9
- * if r5 > 0x79f goto pc+7
- * R5_w=inv(id=0,umax_value=1951,var_off=(0x0; 0x7ff))
- * r5 += 1
- * ...
- * call bpf_perf_event_output#25
- * where .arg5_type = ARG_MEM_SIZE_OR_ZERO
- *
- * and this case:
- * r6 = 1
- * call foo // uses callee's r6 inside to compute r0
- * r0 += r6
- * if r0 == 0 goto
- *
- * to track above reg_mask/stack_mask needs to be independent for each frame.
- *
- * Also if parent's curframe > frame where backtracking started,
- * the verifier need to mark registers in both frames, otherwise callees
- * may incorrectly prune callers. This is similar to
- * commit 7640ead93924 ("bpf: verifier: make sure callees don't prune with caller differences")
- *
- * For now backtracking falls back into conservative marking.
- */
-void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,
- struct bpf_verifier_state *st)
-{
- struct bpf_func_state *func;
- struct bpf_reg_state *reg;
- int i, j;
-
- if (env->log.level & BPF_LOG_LEVEL2) {
- verbose(env, "mark_precise: frame%d: falling back to forcing all scalars precise\n",
- st->curframe);
- }
-
- /* big hammer: mark all scalars precise in this path.
- * pop_stack may still get !precise scalars.
- * We also skip current state and go straight to first parent state,
- * because precision markings in current non-checkpointed state are
- * not needed. See why in the comment in __mark_chain_precision below.
- */
- for (st = st->parent; st; st = st->parent) {
- for (i = 0; i <= st->curframe; i++) {
- func = st->frame[i];
- for (j = 0; j < BPF_REG_FP; j++) {
- reg = &func->regs[j];
- if (reg->type != SCALAR_VALUE || reg->precise)
- continue;
- reg->precise = true;
- if (env->log.level & BPF_LOG_LEVEL2) {
- verbose(env, "force_precise: frame%d: forcing r%d to be precise\n",
- i, j);
- }
- }
- for (j = 0; j < bpf_stack_nr_slots(func); j++) {
- struct bpf_stack_state *ss = bpf_stack_slot(func, j);
-
- if (!bpf_is_spilled_reg(ss))
- continue;
- reg = &ss->spilled_ptr;
- if (reg->type != SCALAR_VALUE || reg->precise)
- continue;
- reg->precise = true;
- if (env->log.level & BPF_LOG_LEVEL2) {
- verbose(env, "force_precise: frame%d: forcing fp%d to be precise\n",
- i, -(j + 1) * 8);
- }
- }
- }
- }
-}
-
-/*
- * bpf_mark_chain_precision() backtracks BPF program instruction sequence and
- * chain of verifier states making sure that register *regno* (if regno >= 0)
- * and/or stack slot *spi* (if spi >= 0) are marked as precisely tracked
- * SCALARS, as well as any other registers and slots that contribute to
- * a tracked state of given registers/stack slots, depending on specific BPF
- * assembly instructions (see backtrack_insns() for exact instruction handling
- * logic). This backtracking relies on recorded jmp_history and is able to
- * traverse entire chain of parent states. This process ends only when all the
- * necessary registers/slots and their transitive dependencies are marked as
- * precise.
- *
- * One important and subtle aspect is that precise marks *do not matter* in
- * the currently verified state (current state). It is important to understand
- * why this is the case.
- *
- * First, note that current state is the state that is not yet "checkpointed",
- * i.e., it is not yet put into env->explored_states, and it has no children
- * states as well. It's ephemeral, and can end up either a) being discarded if
- * compatible explored state is found at some point or BPF_EXIT instruction is
- * reached or b) checkpointed and put into env->explored_states, branching out
- * into one or more children states.
- *
- * In the former case, precise markings in current state are completely
- * ignored by state comparison code (see regsafe() for details). Only
- * checkpointed ("old") state precise markings are important, and if old
- * state's register/slot is precise, regsafe() assumes current state's
- * register/slot as precise and checks value ranges exactly and precisely. If
- * states turn out to be compatible, current state's necessary precise
- * markings and any required parent states' precise markings are enforced
- * after the fact with propagate_precision() logic, after the fact. But it's
- * important to realize that in this case, even after marking current state
- * registers/slots as precise, we immediately discard current state. So what
- * actually matters is any of the precise markings propagated into current
- * state's parent states, which are always checkpointed (due to b) case above).
- * As such, for scenario a) it doesn't matter if current state has precise
- * markings set or not.
- *
- * Now, for the scenario b), checkpointing and forking into child(ren)
- * state(s). Note that before current state gets to checkpointing step, any
- * processed instruction always assumes precise SCALAR register/slot
- * knowledge: if precise value or range is useful to prune jump branch, BPF
- * verifier takes this opportunity enthusiastically. Similarly, when
- * register's value is used to calculate offset or memory address, exact
- * knowledge of SCALAR range is assumed, checked, and enforced. So, similar to
- * what we mentioned above about state comparison ignoring precise markings
- * during state comparison, BPF verifier ignores and also assumes precise
- * markings *at will* during instruction verification process. But as verifier
- * assumes precision, it also propagates any precision dependencies across
- * parent states, which are not yet finalized, so can be further restricted
- * based on new knowledge gained from restrictions enforced by their children
- * states. This is so that once those parent states are finalized, i.e., when
- * they have no more active children state, state comparison logic in
- * is_state_visited() would enforce strict and precise SCALAR ranges, if
- * required for correctness.
- *
- * To build a bit more intuition, note also that once a state is checkpointed,
- * the path we took to get to that state is not important. This is crucial
- * property for state pruning. When state is checkpointed and finalized at
- * some instruction index, it can be correctly and safely used to "short
- * circuit" any *compatible* state that reaches exactly the same instruction
- * index. I.e., if we jumped to that instruction from a completely different
- * code path than original finalized state was derived from, it doesn't
- * matter, current state can be discarded because from that instruction
- * forward having a compatible state will ensure we will safely reach the
- * exit. States describe preconditions for further exploration, but completely
- * forget the history of how we got here.
- *
- * This also means that even if we needed precise SCALAR range to get to
- * finalized state, but from that point forward *that same* SCALAR register is
- * never used in a precise context (i.e., it's precise value is not needed for
- * correctness), it's correct and safe to mark such register as "imprecise"
- * (i.e., precise marking set to false). This is what we rely on when we do
- * not set precise marking in current state. If no child state requires
- * precision for any given SCALAR register, it's safe to dictate that it can
- * be imprecise. If any child state does require this register to be precise,
- * we'll mark it precise later retroactively during precise markings
- * propagation from child state to parent states.
- *
- * Skipping precise marking setting in current state is a mild version of
- * relying on the above observation. But we can utilize this property even
- * more aggressively by proactively forgetting any precise marking in the
- * current state (which we inherited from the parent state), right before we
- * checkpoint it and branch off into new child state. This is done by
- * mark_all_scalars_imprecise() to hopefully get more permissive and generic
- * finalized states which help in short circuiting more future states.
- */
-int bpf_mark_chain_precision(struct bpf_verifier_env *env,
- struct bpf_verifier_state *starting_state,
- int regno,
- bool *changed)
-{
- struct bpf_verifier_state *st = starting_state;
- struct backtrack_state *bt = &env->bt;
- int first_idx = st->first_insn_idx;
- int last_idx = starting_state->insn_idx;
- int subseq_idx = -1;
- struct bpf_func_state *func;
- bool tmp, skip_first = true;
- struct bpf_stack_state *ss;
-
- /* FLOW: there are no precision marks */
- bt_reset(bt);
- return 0;
- struct bpf_reg_state *reg;
- int i, fr, err;
-
- if (!env->bpf_capable)
- return 0;
-
- changed = changed ?: &tmp;
- /* set frame number from which we are starting to backtrack */
- bt_init(bt, starting_state->curframe);
-
- /* Do sanity checks against current state of register and/or stack
- * slot, but don't set precise flag in current state, as precision
- * tracking in the current state is unnecessary.
- */
- func = st->frame[bt->frame];
- if (regno >= 0) {
- reg = &func->regs[regno];
- if (reg->type != SCALAR_VALUE) {
- verifier_bug(env, "backtracking misuse");
- return -EFAULT;
- }
- bt_set_reg(bt, regno);
- }
-
- if (bt_empty(bt))
- return 0;
-
- for (;;) {
- DECLARE_BITMAP(mask, 64);
- u32 history = st->jmp_history_cnt;
- struct bpf_jmp_history_entry *hist;
-
- if (env->log.level & BPF_LOG_LEVEL2) {
- verbose(env, "mark_precise: frame%d: last_idx %d first_idx %d subseq_idx %d \n",
- bt->frame, last_idx, first_idx, subseq_idx);
- }
-
- if (last_idx < 0) {
- /* Consecutive checkpoints can have no instructions between them. */
- if (st->parent)
- goto parent;
-
- /* we are at the entry into subprog, which
- * is expected for global funcs, but only if
- * requested precise registers are R1-R5
- * (which are global func's input arguments)
- */
- if (st->curframe == 0 &&
- st->frame[0]->subprogno > 0 &&
- st->frame[0]->callsite == BPF_MAIN_FUNC &&
- bt_stack_empty(bt) &&
- (bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) == 0) {
- bitmap_from_u64(mask, bt_reg_mask(bt));
- for_each_set_bit(i, mask, 32) {
- reg = &st->frame[0]->regs[i];
- bt_clear_reg(bt, i);
- if (reg->type == SCALAR_VALUE) {
- reg->precise = true;
- *changed = true;
- }
- }
- return 0;
- }
-
- bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
- verifier_bug(env, "backtracking func entry subprog %d reg_mask %x stack_mask %s",
- st->frame[0]->subprogno, bt_reg_mask(bt), env->tmp_str_buf);
- return -EFAULT;
- }
-
- for (i = last_idx;;) {
- if (skip_first) {
- err = 0;
- skip_first = false;
- } else {
- hist = get_jmp_hist_entry(st, history, i);
- err = backtrack_insn(env, i, subseq_idx, hist, bt);
- }
- if (err == -ENOTSUPP) {
- bpf_mark_all_scalars_precise(env, starting_state);
- bt_reset(bt);
- return 0;
- } else if (err) {
- return err;
- }
- if (bt_empty(bt))
- /* Found assignment(s) into tracked register in this state.
- * Since this state is already marked, just return.
- * Nothing to be tracked further in the parent state.
- */
- return 0;
- subseq_idx = i;
- i = get_prev_insn_idx(st, i, &history);
- if (i == -ENOENT)
- break;
- if (i >= env->prog->len) {
- /* This can happen if backtracking reached insn 0
- * and there are still reg_mask or stack_mask
- * to backtrack.
- * It means the backtracking missed the spot where
- * particular register was initialized with a constant.
- */
- verifier_bug(env, "backtracking idx %d", i);
- return -EFAULT;
- }
- }
-parent:
- st = st->parent;
- if (!st)
- break;
-
- for (fr = bt->frame; fr >= 0; fr--) {
- func = st->frame[fr];
- bitmap_from_u64(mask, bt_frame_reg_mask(bt, fr));
- for_each_set_bit(i, mask, 32) {
- reg = &func->regs[i];
- if (reg->type != SCALAR_VALUE) {
- bt_clear_frame_reg(bt, fr, i);
- continue;
- }
- if (reg->precise) {
- bt_clear_frame_reg(bt, fr, i);
- } else {
- reg->precise = true;
- *changed = true;
- }
- }
-
- for_each_set_bit(i, bt_frame_stack_mask(bt, fr), MAX_BPF_STACK_SLOTS) {
- if (verifier_bug_if(i >= bpf_stack_nr_slots(func),
- env, "stack slot %d, total slots %d",
- i, bpf_stack_nr_slots(func)))
- return -EFAULT;
-
- ss = bpf_stack_slot(func, i);
- if (!bpf_is_spilled_scalar_reg(ss)) {
- bt_clear_frame_slot(bt, fr, i);
- continue;
- }
- reg = &ss->spilled_ptr;
- if (reg->precise) {
- bt_clear_frame_slot(bt, fr, i);
- } else {
- reg->precise = true;
- *changed = true;
- }
- }
- for (i = 0; i < func->out_stack_arg_cnt; i++) {
- if (!bt_is_frame_stack_arg_slot_set(bt, fr, i))
- continue;
- reg = &func->stack_arg_regs[i];
- if (reg->type != SCALAR_VALUE || reg->precise) {
- bt_clear_frame_stack_arg_slot(bt, fr, i);
- } else {
- reg->precise = true;
- *changed = true;
- }
- }
- if (env->log.level & BPF_LOG_LEVEL2) {
- fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
- bt_frame_reg_mask(bt, fr));
- verbose(env, "mark_precise: frame%d: parent state regs=%s ",
- fr, env->tmp_str_buf);
- bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
- bt_frame_stack_mask(bt, fr));
- verbose(env, "stack=%s: ", env->tmp_str_buf);
- print_verifier_state(env, st, fr, true);
- }
- }
-
- if (bt_empty(bt))
- return 0;
-
- subseq_idx = first_idx;
- last_idx = st->last_insn_idx;
- first_idx = st->first_insn_idx;
- }
-
- /* if we still have requested precise regs or slots, we missed
- * something (e.g., stack access through non-r10 register), so
- * fallback to marking all precise
- */
- if (!bt_empty(bt)) {
- bpf_mark_all_scalars_precise(env, starting_state);
- bt_reset(bt);
- }
-
- return 0;
-}
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index 82edff722dea..1c7a13bfee87 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -629,8 +629,6 @@ static void print_reg_state(struct bpf_verifier_env *env,
const char *sep = "";
t = reg->type;
- if (t == SCALAR_VALUE && reg->precise)
- verbose(env, "P");
if (t == SCALAR_VALUE && tnum_is_const(reg->var_off)) {
bpf_verbose_snum(env, reg->var_off.value);
return;
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 853aac03d519..4da7ad48f99e 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1773,7 +1773,7 @@ int bpf_init_scev(struct bpf_verifier_env *env)
return -ENOMEM;
env->scev = scev;
bpf_mark_reg_known_scalar(&scev->zero_reg, 0);
- bpf_mark_reg_unknown_imprecise(&scev->unknown_reg);
+ bpf_mark_reg_unknown(&scev->unknown_reg);
/* A power of two for hash & (exprs_ht_cnt - 1)` indexing. */
scev->exprs_ht_cnt = roundup_pow_of_two(max(256U, DIV_ROUND_UP(env->prog->len, 4)));
scev->exprs_ht = kvcalloc(scev->exprs_ht_cnt, sizeof(*scev->exprs_ht), GFP_KERNEL_ACCOUNT);
@@ -2899,30 +2899,6 @@ static void iv_bounds(struct bounds *bounds, struct cnum64 iv_range)
bounds->r64 = r64;
}
-static int mark_expr_precise(struct bpf_verifier_env *env, struct bpf_func_state *st, u32 id,
- unsigned long *precise_regs)
-{
- struct scev *scev = env->scev;
- struct bpf_reg_state *reg;
- u32 r, order;
-
- scev->stack_sz = 0;
- expr_stack_push(scev, id);
- while (expr_next(scev, &id, &order)) {
- if (order & DEPTH_LIMIT) {
- verifier_bug(env, "scev precision marking exceeds expression depth limit");
- return -EFAULT;
- }
- if (!(order & PRE) || !is_reg(scev, id, &r))
- continue;
- /* Synthetic values have no spilled register to mark precise. */
- reg = scev_reg_for_linear(st, r);
- if (reg && reg->type == SCALAR_VALUE)
- __set_bit(r, precise_regs);
- }
- return 0;
-}
-
/*
* Nothing that @id is computed from is what a register had at the head of the loop, and
* some of it is not known: it's made anew in every iteration.
@@ -3195,17 +3171,6 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
walk = true;
}
- bitmap_zero(iters->precise_regs, REGS_NUM);
- err = mark_expr_precise(env, cur_func, latch.base_expr, iters->precise_regs);
- if (err)
- return err;
- err = mark_expr_precise(env, cur_func, latch.step_expr, iters->precise_regs);
- if (err)
- return err;
- err = mark_expr_precise(env, cur_func, latch.bound_expr, iters->precise_regs);
- if (err)
- return err;
-
/* 2: the count is known, what some of the live registers are is not */
return walk ? 2 : 1;
}
@@ -3218,7 +3183,7 @@ static void scev_forget_reg(struct bpf_verifier_env *env, struct bpf_func_state
if (r < MAX_BPF_REG) {
if (func->regs[r].type == SCALAR_VALUE) {
- bpf_mark_reg_unknown_imprecise(&func->regs[r]);
+ bpf_mark_reg_unknown(&func->regs[r]);
mark_scev_reg_scratched(env, r);
} else if (func->regs[r].type == PTR_TO_ARENA) {
/* NULL or not is all that is known about it */
@@ -3302,16 +3267,6 @@ int bpf_widen_scev_regs(struct bpf_verifier_env *env, struct bpf_verifier_state
int insn_idx = st->insn_idx;
int err;
- for_each_set_bit(r, iters->precise_regs, REGS_NUM) {
- if (r < MAX_BPF_REG)
- bpf_bt_set_frame_reg(&env->bt, st->curframe, r);
- else
- bpf_bt_set_frame_slot(&env->bt, st->curframe, r - MAX_BPF_REG);
- }
- err = bpf_mark_chain_precision(env, st, -1, NULL);
- if (err)
- return err;
-
header_env = find_header_env(scev, insn_idx);
for (r = 0; r < REGS_NUM; r++) {
if (!scev_reg_alive(env, st, r) || is_unbound_scev_reg(env, cur_func, r) ||
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 7cf81ba411eb..d0989d85d2cb 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -178,10 +178,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
if (lo_live && stype == STACK_SPILL) {
if (spill->type != SCALAR_VALUE)
continue;
- /*
- * Can't replace with STACK_ZERO, because
- * that requires bpf_mark_chain_precision().
- */
+ /* keep what is known about the low half */
if (bpf_register_is_null(spill))
continue;
for (j = 0; j < 4; j++) {
@@ -277,8 +274,6 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 &&
check_scalar_ids(rold->id, rcur->id, idmap);
}
- if (!rold->precise && exact == NOT_EXACT)
- return true;
/*
* Linked register tracking uses rold->id to detect relationships.
* When rold->id == 0, the register is independent and any linking
@@ -390,7 +385,7 @@ static struct bpf_reg_state unbound_reg;
static __init int unbound_reg_init(void)
{
- bpf_mark_reg_unknown_imprecise(&unbound_reg);
+ bpf_mark_reg_unknown(&unbound_reg);
return 0;
}
late_initcall(unbound_reg_init);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 2c385a7d459c..245147ff3b1e 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -302,11 +302,6 @@ static int arg_from_argno(argno_t a)
return -1;
}
-static int arg_idx_from_argno(argno_t a)
-{
- return arg_from_argno(a) - 1;
-}
-
/* Normalize helper register numbers and kfunc argument numbers to ABI slots. */
static u32 arg_slot_from_argno(argno_t a)
{
@@ -1717,13 +1712,6 @@ static void free_func_state(struct bpf_func_state *state)
kfree(state);
}
-void bpf_clear_jmp_history(struct bpf_verifier_state *state)
-{
- kfree(state->jmp_history);
- state->jmp_history = NULL;
- state->jmp_history_cnt = 0;
-}
-
void bpf_free_verifier_state(struct bpf_verifier_state *state,
bool free_self)
{
@@ -1734,7 +1722,6 @@ void bpf_free_verifier_state(struct bpf_verifier_state *state,
state->frame[i] = NULL;
}
kfree(state->refs);
- bpf_clear_jmp_history(state);
if (free_self)
kfree(state);
}
@@ -1761,13 +1748,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
struct bpf_func_state *dst;
int i, err;
- dst_state->jmp_history = copy_array(dst_state->jmp_history, src->jmp_history,
- src->jmp_history_cnt, sizeof(*dst_state->jmp_history),
- GFP_KERNEL_ACCOUNT);
- if (!dst_state->jmp_history)
- return -ENOMEM;
- dst_state->jmp_history_cnt = src->jmp_history_cnt;
-
/* if dst has more stack frames then src frame, free them, this is also
* necessary in case of exceptional exits using bpf_throw.
*/
@@ -1786,8 +1766,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
dst_state->flow_entry = src->flow_entry;
dst_state->insn_idx = src->insn_idx;
dst_state->curframe = src->curframe;
- dst_state->parent = src->parent;
- dst_state->first_insn_idx = src->first_insn_idx;
dst_state->last_insn_idx = src->last_insn_idx;
for (i = 0; i <= src->curframe; i++) {
dst = dst_state->frame[i];
@@ -2061,10 +2039,6 @@ static void __mark_reg_const_zero(const struct bpf_verifier_env *env, struct bpf
{
__mark_reg_known(reg, 0);
reg->type = SCALAR_VALUE;
- /* all scalars are assumed imprecise initially (unless unprivileged,
- * in which case everything is forced to be precise)
- */
- reg->precise = !env->bpf_capable;
}
static void mark_reg_known_zero(struct bpf_verifier_env *env,
@@ -2397,7 +2371,7 @@ static int reg_bounds_sanity_check(struct bpf_verifier_env *env,
}
/* Mark a register as having a completely unknown (scalar) value. */
-void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg)
+void bpf_mark_reg_unknown(struct bpf_reg_state *reg)
{
memset(reg, 0, sizeof(*reg));
reg->type = SCALAR_VALUE;
@@ -2406,14 +2380,10 @@ void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg)
reg_step_reset(reg);
}
-/* Mark a register as having a completely unknown (scalar) value,
- * initialize .precise as true when not bpf capable.
- */
static void __mark_reg_unknown(const struct bpf_verifier_env *env,
struct bpf_reg_state *reg)
{
- bpf_mark_reg_unknown_imprecise(reg);
- reg->precise = !env->bpf_capable;
+ bpf_mark_reg_unknown(reg);
}
static void mark_reg_unknown(struct bpf_verifier_env *env,
@@ -3594,19 +3564,7 @@ static void mark_non_stack_access(struct bpf_verifier_env *env, int idx)
env->insn_aux_data[idx].non_stack_access = true;
}
-/* Layout of one packed linked register in the jump history, see linked_regs_pack() */
-#define LR_FRAMENO_BITS 5
-#define LR_INDEX_BITS 10
-#define LR_FRAMENO_MASK ((1u << LR_FRAMENO_BITS) - 1)
-#define LR_IS_REG BIT(LR_FRAMENO_BITS)
-#define LR_INDEX_OFF (LR_FRAMENO_BITS + 1)
-#define LR_INDEX_MASK ((1u << LR_INDEX_BITS) - 1)
-#define LINKED_REGS_MAX BPF_LINKED_REGS_MAX
-
-static_assert(MAX_CALL_FRAMES <= (1 << LR_FRAMENO_BITS));
-static_assert(MAX_BPF_REG <= (1 << LR_INDEX_BITS));
-static_assert(MAX_BPF_STACK_SLOTS <= (1 << LR_INDEX_BITS));
-static_assert(LR_INDEX_OFF + LR_INDEX_BITS <= 16);
+#define LINKED_REGS_MAX 5
struct linked_reg {
u8 frameno;
@@ -3630,38 +3588,6 @@ static struct linked_reg *linked_regs_push(struct linked_regs *s)
return NULL;
}
-/*
- * Pack linked registers for a jump history entry, one u16 each:
- * - 4 bits frameno
- * - 1 bit is_reg
- * - 11 bits register or stack slot index
- */
-static void linked_regs_pack(const struct linked_regs *s, u16 *packed)
-{
- int i;
-
- for (i = 0; i < s->cnt; ++i) {
- const struct linked_reg *e = &s->entries[i];
-
- packed[i] = e->frameno | (e->is_reg ? LR_IS_REG : 0) | (e->spi << LR_INDEX_OFF);
- }
-}
-
-static void linked_regs_unpack(const struct bpf_jmp_history_entry *hist, struct linked_regs *s)
-{
- int i;
-
- s->cnt = hist->linked_regs_cnt;
- for (i = 0; i < s->cnt; ++i) {
- struct linked_reg *e = &s->entries[i];
- u16 packed = hist->linked_regs[i];
-
- e->frameno = packed & LR_FRAMENO_MASK;
- e->is_reg = packed & LR_IS_REG;
- e->spi = (packed >> LR_INDEX_OFF) & LR_INDEX_MASK;
- }
-}
-
const char *bpf_disasm_kfunc_name(void *data, const struct bpf_insn *insn)
{
const struct btf_type *func;
@@ -3691,56 +3617,6 @@ void bpf_verbose_insn(struct bpf_verifier_env *env, struct bpf_insn *insn)
print_bpf_insn(&cbs, insn, env->allow_ptr_leaks);
}
-/* If any register R in hist->linked_regs is marked as precise in bt,
- * do bt_set_frame_{reg,slot}(bt, R) for all registers in hist->linked_regs.
- */
-void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist)
-{
- struct linked_regs linked_regs;
- bool some_precise = false;
- int i;
-
- if (!hist || !hist->linked_regs_cnt)
- return;
-
- linked_regs_unpack(hist, &linked_regs);
- for (i = 0; i < linked_regs.cnt; ++i) {
- struct linked_reg *e = &linked_regs.entries[i];
-
- if ((e->is_reg && bt_is_frame_reg_set(bt, e->frameno, e->regno)) ||
- (!e->is_reg && bt_is_frame_slot_set(bt, e->frameno, e->spi))) {
- some_precise = true;
- break;
- }
- }
-
- if (!some_precise)
- return;
-
- for (i = 0; i < linked_regs.cnt; ++i) {
- struct linked_reg *e = &linked_regs.entries[i];
-
- if (e->is_reg)
- bpf_bt_set_frame_reg(bt, e->frameno, e->regno);
- else
- bpf_bt_set_frame_slot(bt, e->frameno, e->spi);
- }
-}
-
-int mark_chain_precision(struct bpf_verifier_env *env, int regno)
-{
- return bpf_mark_chain_precision(env, env->cur_state, regno, NULL);
-}
-
-/* mark_chain_precision_batch() assumes that env->bt is set in the caller to
- * desired reg and stack masks across all relevant frames
- */
-static int mark_chain_precision_batch(struct bpf_verifier_env *env,
- struct bpf_verifier_state *starting_state)
-{
- return bpf_mark_chain_precision(env, starting_state, -1, NULL);
-}
-
/* check if register is a constant scalar value */
static bool is_reg_const(struct bpf_reg_state *reg, bool subreg32)
{
@@ -3877,8 +3753,6 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
int i, slot = -off - 1, spi = slot / BPF_REG_SIZE, err;
struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
struct bpf_reg_state *reg = NULL;
- int insn_flags = INSN_F_STACK_ACCESS;
- int hist_spi = spi, hist_frame = state->frameno;
struct bpf_stack_state *ss = bpf_stack_slot(state, spi);
/*
@@ -3968,28 +3842,14 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
/* when we zero initialize stack slots mark them as such */
if ((reg && bpf_register_is_null(reg)) ||
- (!reg && is_bpf_st_mem(insn) && insn->imm == 0)) {
- /* STACK_ZERO case happened because register spill
- * wasn't properly aligned at the stack slot boundary,
- * so it's not a register spill anymore; force
- * originating register to be precise to make
- * STACK_ZERO correct for subsequent states
- */
- err = mark_chain_precision(env, value_regno);
- if (err)
- return err;
+ (!reg && is_bpf_st_mem(insn) && insn->imm == 0))
type = STACK_ZERO;
- }
/* Mark slots affected by this stack write. */
for (i = 0; i < size; i++)
ss->slot_type[(slot - i) % BPF_REG_SIZE] = type;
- insn_flags = 0; /* not a register spill */
}
- if (insn_flags)
- return bpf_push_jmp_history(env, env->cur_state, insn_flags,
- hist_spi, hist_frame, NULL, 0);
return 0;
}
@@ -4023,10 +3883,6 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
struct bpf_reg_state *value_reg = NULL;
struct bpf_insn *insn = &env->prog->insnsi[insn_idx];
bool writing_zero = false;
- /* set if the fact that we're writing a zero is used to let any
- * stack slots remain STACK_ZERO
- */
- bool zero_used = false;
cur = env->cur_state->frame[env->cur_state->curframe];
min_off = reg_smin(ptr_reg) + off;
@@ -4077,10 +3933,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
if (writing_zero && *stype == STACK_SPILL && bpf_is_spilled_scalar_reg(ss)) {
struct bpf_reg_state *spill_reg = &ss->spilled_ptr;
- if (tnum_is_const(spill_reg->var_off) && spill_reg->var_off.value == 0) {
- zero_used = true;
+ if (tnum_is_const(spill_reg->var_off) && spill_reg->var_off.value == 0)
continue;
- }
}
/*
@@ -4093,10 +3947,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
/* Update the slot type. */
new_type = STACK_MISC;
- if (writing_zero && *stype == STACK_ZERO) {
+ if (writing_zero && *stype == STACK_ZERO)
new_type = STACK_ZERO;
- zero_used = true;
- }
/* If the slot is STACK_INVALID, we check whether it's OK to
* pretend that it will be initialized by this write. The slot
* might not actually be written to, and so if we mark it as
@@ -4114,12 +3966,6 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
}
*stype = new_type;
}
- if (zero_used) {
- /* backtracking doesn't work for STACK_ZERO yet. */
- err = mark_chain_precision(env, value_regno);
- if (err)
- return err;
- }
bpf_diag_record_scrub_stack(env, state, min_off, max_off,
BPF_DIAG_MOD_VAR_WRITE);
return 0;
@@ -4134,21 +3980,18 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
* read.
*
* STACK_SPILL bytes backed by spilled scalar const zeroes are also considered
- * zero bytes. In that case, mark the contributing stack slots precise so
- * pruning cannot reuse a zero-spill state for a later non-zero spill state.
- *
- * Returns an error if precision backtracking fails.
+ * zero bytes.
*/
-static int mark_reg_stack_read(struct bpf_verifier_env *env,
- /* func where src register points to */
- struct bpf_func_state *ptr_state,
- int min_off, int max_off, int dst_regno)
+static void mark_reg_stack_read(struct bpf_verifier_env *env,
+ /* func where src register points to */
+ struct bpf_func_state *ptr_state,
+ int min_off, int max_off, int dst_regno)
{
struct bpf_verifier_state *vstate = env->cur_state;
struct bpf_func_state *state = vstate->frame[vstate->curframe];
int i, slot, spi;
u8 *stype;
- int zeros = 0, zero_spills = 0;
+ int zeros = 0;
for (i = min_off; i < max_off; i++) {
slot = -i - 1;
@@ -4161,7 +4004,6 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,
}
if (stype[slot % BPF_REG_SIZE] == STACK_SPILL &&
bpf_register_is_null(&bpf_stack_slot(ptr_state, spi)->spilled_ptr)) {
- zero_spills++;
zeros++;
continue;
}
@@ -4172,22 +4014,10 @@ static int mark_reg_stack_read(struct bpf_verifier_env *env,
* so the whole register == const_zero.
*/
__mark_reg_const_zero(env, &state->regs[dst_regno]);
- if (zero_spills) {
- for (i = min_off; i < max_off; i++) {
- slot = -i - 1;
- spi = slot / BPF_REG_SIZE;
- stype = bpf_stack_slot(ptr_state, spi)->slot_type;
- if (stype[slot % BPF_REG_SIZE] == STACK_SPILL)
- bpf_bt_set_frame_slot(&env->bt, ptr_state->frameno, spi);
- }
- return mark_chain_precision_batch(env, env->cur_state);
- }
} else {
/* have read misc data from the stack */
mark_reg_unknown(env, state->regs, dst_regno);
}
-
- return 0;
}
static void bpf_diag_stack_read_invalid(struct bpf_verifier_env *env, int off, int i, int size,
@@ -4251,9 +4081,6 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
int i, slot = -off - 1, spi = slot / BPF_REG_SIZE;
struct bpf_reg_state *reg;
u8 *stype, type;
- int err;
- int insn_flags = INSN_F_STACK_ACCESS;
- int hist_spi = spi, hist_frame = reg_state->frameno;
struct bpf_stack_state *ss = bpf_stack_slot(reg_state, spi);
stype = ss->slot_type;
@@ -4329,17 +4156,12 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
if (spill_cnt == size &&
tnum_is_const(reg->var_off) && reg->var_off.value == 0) {
__mark_reg_const_zero(env, &state->regs[dst_regno]);
- /* this IS register fill, so keep insn_flags */
} else if (zero_cnt == size) {
/* similarly to mark_reg_stack_read(), preserve zeroes */
__mark_reg_const_zero(env, &state->regs[dst_regno]);
- insn_flags = 0; /* not restoring original register state */
} else {
- err = mark_reg_stack_read(env, reg_state, off, off + size,
- dst_regno);
- if (err)
- return err;
- insn_flags = 0; /* not restoring original register state */
+ mark_reg_stack_read(env, reg_state, off, off + size,
+ dst_regno);
}
}
} else if (dst_regno >= 0) {
@@ -4350,10 +4172,6 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
*/
assign_scalar_id_before_mov(env, reg);
state->regs[dst_regno] = *reg;
- /* mark reg as written since spilled pointer state likely
- * has its liveness marks cleared by is_state_visited()
- * which resets stack/reg liveness for state transitions
- */
} else if (__is_pointer_value(env->allow_ptr_leaks, reg)) {
/* If dst_regno==-1, the caller is asking us whether
* it is acceptable to use this value as a SCALAR_VALUE
@@ -4384,16 +4202,9 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
}
return -EACCES;
}
- if (dst_regno >= 0) {
- err = mark_reg_stack_read(env, reg_state, off, off + size, dst_regno);
- if (err)
- return err;
- }
- insn_flags = 0; /* we are not restoring spilled register */
+ if (dst_regno >= 0)
+ mark_reg_stack_read(env, reg_state, off, off + size, dst_regno);
}
- if (insn_flags)
- return bpf_push_jmp_history(env, env->cur_state, insn_flags,
- hist_spi, hist_frame, NULL, 0);
return 0;
}
@@ -4442,10 +4253,7 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, struct bpf_reg
min_off = reg_smin(reg) + off;
max_off = reg_smax(reg) + off;
- err = mark_reg_stack_read(env, ptr_state, min_off, max_off + size,
- dst_regno);
- if (err)
- return err;
+ mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno);
return 0;
}
@@ -4598,8 +4406,7 @@ static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_s
__mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm);
}
bpf_diag_mod_end(env);
- return bpf_push_jmp_history(env, env->cur_state,
- INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
+ return 0;
}
/*
@@ -4634,26 +4441,7 @@ static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_st
bpf_diag_mod_begin(env, &cur->regs[dst_regno], arg, BPF_DIAG_MOD_WRITE);
cur->regs[dst_regno] = *arg;
bpf_diag_mod_end(env);
- return bpf_push_jmp_history(env, env->cur_state,
- INSN_F_STACK_ARG_ACCESS, spi, 0, NULL, 0);
-}
-
-static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx)
-{
- struct bpf_func_state *caller = cur_func(env);
- int spi = arg_idx - MAX_BPF_FUNC_REG_ARGS;
-
- bt_set_frame_stack_arg_slot(&env->bt, caller->frameno, spi);
- return mark_chain_precision_batch(env, env->cur_state);
-}
-
-static int mark_arg_precision(struct bpf_verifier_env *env, argno_t argno)
-{
- int regno = reg_from_argno(argno);
-
- if (regno >= 0)
- return mark_chain_precision(env, regno);
- return mark_stack_arg_precision(env, arg_idx_from_argno(argno));
+ return 0;
}
static int check_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_func_state *caller,
@@ -5118,14 +4906,8 @@ static int check_map_kptr_access(struct bpf_verifier_env *env,
return ret;
} else if (class == BPF_STX) {
val_reg = reg_state(env, value_regno);
- if (bpf_register_is_null(val_reg)) {
- /*
- * This store is valid only because the scalar is known to be
- * zero. Mark it precise so another scalar cannot be pruned
- * against this state.
- */
- return mark_chain_precision(env, value_regno);
- }
+ if (bpf_register_is_null(val_reg))
+ return 0;
if (map_kptr_match_type(env, kptr_field, val_reg, value_regno))
return -EACCES;
} else if (class == BPF_ST) {
@@ -8200,9 +7982,6 @@ static int check_mem_size_reg(struct bpf_verifier_env *env,
if (err && failure)
*failure = BPF_MEM_SIZE_FAIL_MEMORY;
- if (!err)
- err = mark_arg_precision(env, size_argno);
-
return err;
size_error:
@@ -8223,7 +8002,7 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg
int size, err = 0;
if (bpf_register_is_null(reg))
- return mark_arg_precision(env, argno);
+ return 0;
if (known_memory)
*known_memory = true;
@@ -8253,9 +8032,6 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg
static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
argno_t argno, struct ret_mem_desc *ret_mem)
{
- int regno = reg_from_argno(argno);
- int err;
-
if (ret_mem->found) {
verifier_bug(env, "only one allocation size argument permitted");
return -EFAULT;
@@ -8271,13 +8047,6 @@ static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf
return -EINVAL;
}
- if (regno >= 0)
- err = mark_chain_precision(env, regno);
- else
- err = mark_stack_arg_precision(env, arg_idx_from_argno(argno));
- if (err)
- return err;
-
ret_mem->size = reg->var_off.value;
ret_mem->found = true;
@@ -8287,9 +8056,6 @@ static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf
static int process_const_arg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
argno_t argno, struct bpf_call_arg_meta *meta)
{
- int regno = reg_from_argno(argno);
- int err;
-
if (meta->arg_constant.found) {
verifier_bug(env, "only one constant argument permitted");
return -EFAULT;
@@ -8300,13 +8066,6 @@ static int process_const_arg(struct bpf_verifier_env *env, struct bpf_reg_state
return -EINVAL;
}
- if (regno >= 0)
- err = mark_chain_precision(env, regno);
- else
- err = mark_stack_arg_precision(env, arg_idx_from_argno(argno));
- if (err < 0)
- return err;
-
meta->arg_constant.found = true;
meta->arg_constant.value = reg->var_off.value;
@@ -9725,7 +9484,7 @@ static int get_constant_map_key(struct bpf_verifier_env *env,
int spill_size = 0;
int zero_size = 0;
int stack_off;
- int i, err;
+ int i;
u8 *stype;
if (!env->bpf_capable)
@@ -9763,14 +9522,6 @@ static int get_constant_map_key(struct bpf_verifier_env *env,
/* Stack value not statically known */
return -EOPNOTSUPP;
- /* We are relying on a constant value. So mark as precise
- * to prevent pruning on it.
- */
- bpf_bt_set_frame_slot(&env->bt, key->frameno, spi);
- err = mark_chain_precision_batch(env, env->cur_state);
- if (err < 0)
- return err;
-
*value = reg->var_off.value;
return 0;
}
@@ -9888,12 +9639,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, u32 slot, u32 p
if (arg_type_is_raw_mem(arg_type))
meta->arg_raw_mem.mask |= BIT(slot);
- if (bpf_register_is_null(reg) && type_may_be_null(arg_type)) {
- err = mark_arg_precision(env, argno);
- if (err)
- return err;
+ if (bpf_register_is_null(reg) && type_may_be_null(arg_type))
return 0;
- }
err = check_func_arg_nullability(env, reg, arg, argno, arg_type, meta, insn_idx);
if (err)
@@ -12589,11 +12336,6 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
return -EINVAL;
}
- /* we are going to rely on register's precise value */
- err = mark_chain_precision(env, BPF_REG_0);
- if (err)
- return err;
-
/* enforce R0 return value range, and bpf_callback_t returns 64bit */
if (!retval_range_within(callee->callback_ret_range, r0)) {
verbose_invalid_scalar(env, r0, callee->callback_ret_range,
@@ -12758,7 +12500,6 @@ record_func_key(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
struct bpf_reg_state *reg;
struct bpf_map *map = meta->map.ptr;
u64 val, max;
- int err;
if (func_id != BPF_FUNC_tail_call)
return 0;
@@ -12776,9 +12517,6 @@ record_func_key(struct bpf_verifier_env *env, struct bpf_call_arg_meta *meta,
return 0;
}
- err = mark_chain_precision(env, BPF_REG_3);
- if (err)
- return err;
if (bpf_map_key_unseen(aux))
bpf_map_key_store(aux, val);
else if (!bpf_map_key_poisoned(aux) &&
@@ -12921,45 +12659,27 @@ static struct bpf_insn_aux_data *cur_aux(const struct bpf_verifier_env *env)
return &env->insn_aux_data[env->insn_idx];
}
-/* Returns 1 if R4 is a known zero, 0 if it is not, a negative errno on error. */
-static int loop_flag_is_zero(struct bpf_verifier_env *env)
+static bool loop_flag_is_zero(struct bpf_verifier_env *env)
{
- struct bpf_reg_state *reg = reg_state(env, BPF_REG_4);
- int err;
-
- if (!bpf_register_is_null(reg))
- return 0;
-
- err = mark_chain_precision(env, BPF_REG_4);
- if (err)
- return err;
- return 1;
+ return bpf_register_is_null(reg_state(env, BPF_REG_4));
}
-static int update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno)
+static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno)
{
struct bpf_loop_inline_state *state = &cur_aux(env)->loop_inline_state;
- int flag_is_zero;
if (!state->initialized) {
- flag_is_zero = loop_flag_is_zero(env);
- if (flag_is_zero < 0)
- return flag_is_zero;
state->initialized = 1;
- state->fit_for_inline = flag_is_zero;
+ state->fit_for_inline = loop_flag_is_zero(env);
state->callback_subprogno = subprogno;
- return 0;
+ return;
}
if (!state->fit_for_inline)
- return 0;
+ return;
- flag_is_zero = loop_flag_is_zero(env);
- if (flag_is_zero < 0)
- return flag_is_zero;
- state->fit_for_inline = (flag_is_zero &&
+ state->fit_for_inline = (loop_flag_is_zero(env) &&
state->callback_subprogno == subprogno);
- return 0;
}
/* Returns whether or not the given map can potentially elide
@@ -13178,9 +12898,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
verbose(env, "get_local_storage() doesn't support non-zero flags\n");
return -EINVAL;
}
- err = mark_chain_precision(env, BPF_REG_2);
- if (err)
- return err;
break;
case BPF_FUNC_for_each_map_elem:
err = push_callback_call(env, insn, insn_idx, meta.subprogno,
@@ -13198,15 +12915,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
err = check_bpf_snprintf_call(env, regs);
break;
case BPF_FUNC_loop:
- err = update_loop_inline_state(env, meta.subprogno);
- if (err)
- return err;
- /* Verifier relies on R1 value to determine if bpf_loop() iteration
- * is finished, thus mark it precise.
- */
- err = mark_chain_precision(env, BPF_REG_1);
- if (err)
- return err;
+ update_loop_inline_state(env, meta.subprogno);
if (cur_func(env)->callback_depth < reg_umax(®s[BPF_REG_1])) {
err = push_callback_call(env, insn, insn_idx, meta.subprogno,
set_loop_callback_state);
@@ -13255,10 +12964,6 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
bpf_lsm_get_retval_range(env->prog, &range);
}
- err = mark_chain_precision(env, BPF_REG_1);
- if (err)
- return err;
-
if (!retval_range_within(range, r1)) {
verbose_invalid_scalar(env, r1, range, "At bpf_set_retval", "R1");
return -EINVAL;
@@ -18100,24 +17805,13 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env,
* This is legal, but we have to reverse our
* src/dest handling in computing the range
*/
- err = mark_chain_precision(env, insn->dst_reg);
- if (err)
- return err;
off_reg = *dst_reg;
return adjust_ptr_min_max_vals(env, insn, insn->src_reg, src_reg,
&off_reg);
}
} else if (ptr_reg) {
/* pointer += scalar */
- err = mark_chain_precision(env, insn->src_reg);
- if (err)
- return err;
return adjust_ptr_min_max_vals(env, insn, insn->dst_reg, dst_reg, src_reg);
- } else if (dst_reg->precise) {
- /* if dst_reg is precise, src_reg should be precise as well */
- err = mark_chain_precision(env, insn->src_reg);
- if (err)
- return err;
}
} else {
/* Pretend the src is a reg with a known value, since we only
@@ -19501,7 +19195,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
struct bpf_reg_state *eq_branch_regs;
struct linked_regs linked_regs = {};
u8 opcode = BPF_OP(insn->code);
- int insn_flags = 0;
bool is_jmp32;
int pred = -1;
int err;
@@ -19562,24 +19255,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
return -EACCES;
}
- if (src_reg->type == PTR_TO_STACK)
- insn_flags |= INSN_F_SRC_REG_STACK;
- if (dst_reg->type == PTR_TO_STACK)
- insn_flags |= INSN_F_DST_REG_STACK;
} else {
src_reg = &env->fake_reg[0];
memset(src_reg, 0, sizeof(*src_reg));
src_reg->type = SCALAR_VALUE;
__mark_reg_known(src_reg, insn->imm);
-
- if (dst_reg->type == PTR_TO_STACK)
- insn_flags |= INSN_F_DST_REG_STACK;
- }
-
- if (insn_flags) {
- err = bpf_push_jmp_history(env, this_branch, insn_flags, 0, 0, NULL, 0);
- if (err)
- return err;
}
/*
@@ -19598,19 +19278,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
env->true_reg1 = *dst_reg;
env->true_reg2 = *src_reg;
pred = is_branch_taken(env, dst_reg, src_reg, opcode, is_jmp32);
- if (pred >= 0) {
- /* If we get here with a dst_reg pointer type it is because
- * above is_branch_taken() special cased the 0 comparison.
- */
- if (!__is_pointer_value(false, dst_reg))
- err = mark_chain_precision(env, insn->dst_reg);
- if (BPF_SRC(insn->code) == BPF_X && !err &&
- !__is_pointer_value(false, src_reg))
- err = mark_chain_precision(env, insn->src_reg);
- if (err)
- return err;
- }
-
if (pred == 1) {
/* Only follow the goto, ignore fall-through. If needed, push
* the fall-through branch for simulation under speculative
@@ -19641,20 +19308,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
return 0;
}
- /* Push scalar registers sharing same ID to jump history,
- * do this before creating 'other_branch', so that both
- * 'this_branch' and 'other_branch' share this history
- * if parent state is created.
- */
- if (linked_regs.cnt > 1) {
- u16 packed[LINKED_REGS_MAX];
-
- linked_regs_pack(&linked_regs, packed);
- err = bpf_push_jmp_history(env, this_branch, 0, 0, 0, packed, linked_regs.cnt);
- if (err)
- return err;
- }
-
mark_loop_exit_undecided(env, this_branch);
other_branch = push_stack(env, *insn_idx + insn->off + 1, *insn_idx, false);
if (IS_ERR(other_branch))
@@ -19733,15 +19386,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env,
type_may_be_null(dst_reg->type) &&
((BPF_SRC(insn->code) == BPF_K && insn->imm == 0) ||
(BPF_SRC(insn->code) == BPF_X && bpf_register_is_null(src_reg)))) {
- /*
- * For BPF_X the zero is a property of this execution path,
- * hence src_reg has to be precise.
- */
- if (BPF_SRC(insn->code) == BPF_X) {
- err = mark_chain_precision(env, insn->src_reg);
- if (err)
- return err;
- }
/* Mark all identical registers in each branch as either
* safe or unknown depending R == 0 or R != 0 conditional.
*/
@@ -20197,10 +19841,6 @@ static int check_return_code(struct bpf_verifier_env *env, int regno, const char
return -EINVAL;
}
- err = mark_chain_precision(env, regno);
- if (err)
- return err;
-
if (!retval_range_within(range, reg)) {
verbose_invalid_scalar(env, reg, range, exit_ctx, reg_name);
if (prog->expected_attach_type == BPF_LSM_CGROUP &&
@@ -22874,7 +22514,6 @@ static int __do_check_common(struct bpf_verifier_env *env, int subprog,
BPF_MAIN_FUNC /* callsite */,
0 /* frameno */,
subprog);
- state->first_insn_idx = env->subprog_info[subprog].start;
state->last_insn_idx = -1;
regs = state->frame[state->curframe]->regs;
@@ -24969,7 +24608,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (!env)
return -ENOMEM;
- env->bt.env = env;
env->prog = *prog;
env->ops = bpf_verifier_ops[env->prog->type];
env->stack_limit = bpf_prog_stack_limit(env->prog);
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 61/81] bpf: Remove precision tracking
2026-10-26 2:59 ` [PATCH bpf 61/81] bpf: Remove precision tracking Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 6763 bytes --]
> bpf: Remove precision tracking
TL;DR:
- Possible valid programs rejected: need_const seeding may not cover __szk,
rdonly/rdwr_buf_size, stack-passed or map-key args, so joined states may
fail "must be a known constant"
- Nit: dead 'best'/'latch' stores in bpf_compute_loop_iters()
> diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
> index 853aac03d519..4da7ad48f99e 100644
> --- a/kernel/bpf/scev.c
> +++ b/kernel/bpf/scev.c
[ ... ]
> @@ -3195,17 +3171,6 @@ int bpf_compute_loop_iters(struct bpf_verifier_env *env, struct bpf_verifier_sta
> walk = true;
> }
>
> - bitmap_zero(iters->precise_regs, REGS_NUM);
> - err = mark_expr_precise(env, cur_func, latch.base_expr, iters->precise_regs);
> - if (err)
> - return err;
> - err = mark_expr_precise(env, cur_func, latch.step_expr, iters->precise_regs);
> - if (err)
> - return err;
> - err = mark_expr_precise(env, cur_func, latch.bound_expr, iters->precise_regs);
> - if (err)
> - return err;
> -
> /* 2: the count is known, what some of the live registers are is not */
> return walk ? 2 : 1;
> }
this isn't a bug, but is the local 'best' in bpf_compute_loop_iters() still
needed now that these calls are gone?
The only readers of 'latch' after the selection loop were the three
mark_expr_precise() calls removed above:
err = mark_expr_precise(env, cur_func, latch.base_expr, iters->precise_regs);
err = mark_expr_precise(env, cur_func, latch.step_expr, iters->precise_regs);
err = mark_expr_precise(env, cur_func, latch.bound_expr, iters->precise_regs);
That leaves 'struct linear_latch latch, best;', 'best = latch;' in the
selection loop and 'latch = best;' after it as dead stores. What the rest
of the function needs from the chosen latch is already copied into
iters->max_header_count, iv_reg and iv_range inside the loop.
The leftover is still present at the end of the series, after 91e7ff3a122f
("bpf: Remove leftovers of precision tracking and read marks"). Could
'best' and the two assignments be dropped?
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 2c385a7d459c..245147ff3b1e 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -8287,9 +8056,6 @@ static int process_const_alloc_mem_size(struct bpf_verifier_env *env, struct bpf
> static int process_const_arg(struct bpf_verifier_env *env, struct bpf_reg_state *reg,
> argno_t argno, struct bpf_call_arg_meta *meta)
> {
> - int regno = reg_from_argno(argno);
> - int err;
> -
> if (meta->arg_constant.found) {
> verifier_bug(env, "only one constant argument permitted");
> return -EFAULT;
> @@ -8300,13 +8066,6 @@ static int process_const_arg(struct bpf_verifier_env *env, struct bpf_reg_state
> return -EINVAL;
> }
>
> - if (regno >= 0)
> - err = mark_chain_precision(env, regno);
> - else
> - err = mark_stack_arg_precision(env, arg_idx_from_argno(argno));
> - if (err < 0)
> - return err;
> -
> meta->arg_constant.found = true;
> meta->arg_constant.value = reg->var_off.value;
>
The commit message says the precision marks are not needed because
flow_compute_need_const() "finds the registers and stack slots that go into
such arguments" and the verifier does not join states that differ in them.
Is that true for every call site that loses its mark_chain_precision() call
in this commit? flow_compute_need_const() with FLOW_NEED_ARG is seeded
only from these two places:
bpf_kfunc_const_arg_regs():
for (i = 0, slot = 0; i < nargs && slot < MAX_BPF_FUNC_REG_ARGS; i++) {
if (is_kfunc_arg_constant(meta.btf, &args[i])) /* suffix "__k" only */
mask |= BIT(BPF_REG_1 + slot);
bpf_helper_const_arg_regs():
if (fn->arg_type[i] == ARG_CONST_ALLOC_SIZE_OR_ZERO)
Several arguments that process_const_arg() or process_const_alloc_mem_size()
still require to be known constants do not appear to be covered:
1. kfunc args with the __szk suffix. gen_kfunc_arg_proto() maps them to
ARG_CONST_MEM_SIZE and check_func_arg() sends them to process_const_arg().
btf_param_match_suffix("buffer__szk", "__k") is false, so they are not
seeded. bpf_dynptr_slice() and bpf_dynptr_slice_rdwr() take such an
argument.
2. kfunc args named rdonly_buf_size or rdwr_buf_size. They map to
ARG_CONST_ALLOC_SIZE_OR_ZERO and reach process_const_alloc_mem_size(), but
bpf_kfunc_const_arg_regs() only tests for "__k". An in-tree user is
hid_bpf_get_data(ctx, offset, rdwr_buf_size).
3. __k kfunc args passed on the stack (argument 6 and later). The loop
above stops at MAX_BPF_FUNC_REG_ARGS, and struct flow_need has no
stack-arg slots. The removed mark_stack_arg_precision() calls used to cover
these.
4. The map key slot that get_constant_map_key() reads to drop
PTR_MAYBE_NULL from an array map lookup. need_const only adds a slot for
an LDX into a needed register, and the helper reads the key through R2.
As the comment above flow_compute_need_const() describes for
bpf_rdonly_cast(), LLVM can sink two calls into one:
r4 = 20; if r6 == 0 goto L; r4 = 40;
L: call bpf_dynptr_slice /* r4 is buffer__szk */
The two states meet at L with R4=20 and R4=40. In flow_merge(),
flow_needed_consts_differ() finds no need bit for R4. flow_null_differs(),
flow_join_makes_up() (R4 is not tested by a jump) and
flow_callback_depth_differs() do not apply either.
bpf_states_join() -> join_reg() then joins the scalars to [20,40], and
process_const_arg() fails with "R4 must be a known constant".
Before the series, process_const_arg() marked R4 precise, the two states
were not pruned, and the program loaded.
The same happens with an array map lookup whose key is 0 on one path and 1
on the other: R0 stays map_value_or_null, and an access without a NULL
check that was accepted before is now rejected.
No verifier decision changes at this commit itself, since
bpf_mark_chain_precision() was already stubbed to return 0 by 0535bc235aed
("bpf: Verify programs as a fixpoint over the control flow graph"). But
this commit removes the remaining precision handling at these sites and
names flow_compute_need_const() as the replacement.
A search of the commits after this one in the series found none that
extends bpf_kfunc_const_arg_regs(), bpf_helper_const_arg_regs() or
flow_compute_need_const() to cover the cases above.
Should the need_const seeding cover ARG_CONST_MEM_SIZE, the
rdonly_buf_size/rdwr_buf_size args, stack-passed constant args and the
constant map key? Otherwise, should the commit message avoid claiming that
flow_compute_need_const() covers them?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 62/81] bpf: Remove prune points, levels of state comparison and pop_stack()
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (60 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 61/81] bpf: Remove precision tracking Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 63/81] bpf: Print 0 as the limit in the statistics line Alexei Starovoitov
` (20 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Before "bpf: Verify programs as a fixpoint over the control flow graph" the
main pass of the verifier followed each path of a prog depth first. It
compared states at selected insns to prune paths. Some parts of that pass
have no user. Remove them.
removed task in the old pass now
-------------------- -------------------------- ---------------------
prune_point and Select the insns at which States meet at the
other marks of insns the pass compares states. start of each basic
block.
NOT_EXACT, EXACT Two of the three ways to RANGE_WITHIN is the
compare states. only way.
pop_stack() Get the next state when a flow_drain() empties
path ends. the stack after each
insn.
NOT_EXACT ignored scalars without a precision mark. EXACT accepted only
identical states. RANGE_WITHIN is what bpf_state_subsumes() does: the range
of each live scalar in the new state must be within its range in the old
state.
push_stack() stays. The code that verifies an insn uses it to make a second
state, for example for the other branch of a jump. flow_drain() gives these
states to flow.c.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 38 --------------------
kernel/bpf/cfg.c | 56 ++---------------------------
kernel/bpf/diagnostics.c | 7 ----
kernel/bpf/diagnostics.h | 1 -
kernel/bpf/scev.c | 4 ---
kernel/bpf/states.c | 69 ++++++++----------------------------
kernel/bpf/verifier.c | 55 ++++++----------------------
7 files changed, 29 insertions(+), 201 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 84deb3974c2e..9b107553461e 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -718,7 +718,6 @@ struct bpf_insn_aux_data {
u64 needs_zext:1; /* alu op needs to clear upper bits */
u64 prevent_zext:1; /* alu op cannot be zext (already used with 64-bit scalars) */
u64 non_sleepable:1; /* helper/kfunc may be called from non-sleepable context */
- u64 is_iter_next:1; /* bpf_iter_<type>_next() kfunc call */
u64 call_with_percpu_alloc_ptr:1; /* {this,per}_cpu_ptr() with prog percpu alloc */
u64 cast_arena:1; /* addr_space_cast of an address in arena */
u64 cast_mov:1; /* addr_space_cast of a pointer that is not in arena, it's a move */
@@ -750,12 +749,6 @@ struct bpf_insn_aux_data {
u64 non_arena_load:1;
/* below flags are initialized once */
- u64 jmp_point:1;
- u64 prune_point:1;
- /* ensure we check state equivalence and save state checkpoint and
- * this instruction, regardless of any heuristics
- */
- u64 force_checkpoint:1;
/* true if instruction is a call to a helper function that
* accepts callback function as a parameter.
*/
@@ -1009,7 +1002,6 @@ struct bpf_verifier_env {
const struct bpf_verifier_ops *ops;
struct module *attach_btf_mod; /* The owner module of prog->aux->attach_btf */
struct bpf_verifier_stack_elem *head; /* stack of verifier states to be processed */
- int stack_size; /* number of states to be processed */
bool strict_alignment; /* perform strict pointer alignment checks */
bool test_reg_invariants; /* fail verification on register invariants violations */
bool uses_arena_sp; /* the program has insns with BPF_REG_ARENA_SP */
@@ -1222,26 +1214,6 @@ void bpf_verbose_snum(struct bpf_verifier_env *env, s64 num);
bpf_log(&env->log, "verifier bug: " fmt "\n", ##args); \
})
-static inline void mark_prune_point(struct bpf_verifier_env *env, int idx)
-{
- env->insn_aux_data[idx].prune_point = true;
-}
-
-static inline bool bpf_is_prune_point(struct bpf_verifier_env *env, int insn_idx)
-{
- return env->insn_aux_data[insn_idx].prune_point;
-}
-
-static inline void mark_force_checkpoint(struct bpf_verifier_env *env, int idx)
-{
- env->insn_aux_data[idx].force_checkpoint = true;
-}
-
-static inline bool bpf_is_force_checkpoint(struct bpf_verifier_env *env, int insn_idx)
-{
- return env->insn_aux_data[insn_idx].force_checkpoint;
-}
-
static inline void mark_calls_callback(struct bpf_verifier_env *env, int idx)
{
env->insn_aux_data[idx].calls_callback = true;
@@ -1252,11 +1224,6 @@ static inline bool bpf_calls_callback(struct bpf_verifier_env *env, int insn_idx
return env->insn_aux_data[insn_idx].calls_callback;
}
-static inline void mark_jmp_point(struct bpf_verifier_env *env, int idx)
-{
- env->insn_aux_data[idx].jmp_point = true;
-}
-
static inline void mark_jump_target(struct bpf_verifier_env *env, int idx)
{
env->insn_aux_data[idx].jump_target = true;
@@ -1423,11 +1390,6 @@ static inline u32 bpf_frame_insn_idx(struct bpf_verifier_state *st, u32 frame)
: st->frame[frame + 1]->callsite;
}
-static inline bool bpf_is_jmp_point(struct bpf_verifier_env *env, int insn_idx)
-{
- return env->insn_aux_data[insn_idx].jmp_point;
-}
-
static inline bool bpf_is_spilled_reg(const struct bpf_stack_state *stack)
{
return stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL;
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index f5705ea0bcb9..64eccc28703a 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -123,12 +123,8 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env)
return -EINVAL;
}
- if (e == BRANCH) {
- /* mark branch target for state pruning */
- mark_prune_point(env, w);
- mark_jmp_point(env, w);
+ if (e == BRANCH)
mark_jump_target(env, w);
- }
if (insn_state[w] == 0) {
/* tree-edge */
@@ -172,13 +168,8 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,
if (ret)
return ret;
- mark_prune_point(env, t + insn_sz);
- /* when we exit from subprog, we need to record non-linear history */
- mark_jmp_point(env, t + insn_sz);
-
if (visit_callee) {
w = t + insns[t].imm + 1;
- mark_prune_point(env, t);
merge_callee_effects(env, t, w);
ret = push_insn(t, w, BRANCH, env);
}
@@ -439,14 +430,12 @@ static int visit_gotox_insn(int t, struct bpf_verifier_env *env)
return -E2BIG;
}
- mark_prune_point(env, t);
for (i = 0; i < jt->cnt; i++) {
w = jt->items[i];
if (verifier_bug_if(w < 0 || w >= env->prog->len, env,
"indirect jump from insn %d to %d out of range", t, w))
return -EFAULT;
- mark_jmp_point(env, w);
mark_jump_target(env, w);
/* EXPLORED || DISCOVERED */
@@ -521,14 +510,11 @@ static int visit_func_ptrs_insn(int t, struct bpf_verifier_env *env,
}
/* all functions are explored, their effects are known */
- mark_prune_point(env, t);
for (i = 0; i < cnt; i++) {
w = ptrs[i].xlated_off;
merge_callee_effects(env, t, w);
- /* the same marks as push_insn() leaves on a branch target */
- mark_prune_point(env, w);
- mark_jmp_point(env, w);
+ /* the same mark as push_insn() leaves on a branch target */
mark_jump_target(env, w);
}
return DONE_EXPLORING;
@@ -565,28 +551,13 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
return DONE_EXPLORING;
case BPF_CALL:
- if (bpf_is_async_callback_calling_insn(insn))
- /* Mark this call insn as a prune point to trigger
- * is_state_visited() check before call itself is
- * processed by __check_func_call(). Otherwise new
- * async state will be pushed for further exploration.
- */
- mark_prune_point(env, t);
/* For functions that invoke callbacks it is not known how many times
* callback would be called. Verifier models callback calling functions
* by repeatedly visiting callback bodies and returning to origin call
* instruction.
- * In order to stop such iteration verifier needs to identify when a
- * state identical some state from a previous iteration is reached.
- * Check below forces creation of checkpoint before callback calling
- * instruction to allow search for such identical states.
*/
- if (bpf_is_sync_callback_calling_insn(insn)) {
+ if (bpf_is_sync_callback_calling_insn(insn))
mark_calls_callback(env, t);
- mark_force_checkpoint(env, t);
- mark_prune_point(env, t);
- mark_jmp_point(env, t);
- }
if (bpf_helper_call(insn)) {
const struct bpf_func_proto *fp;
@@ -606,21 +577,6 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
struct bpf_call_arg_meta meta;
ret = bpf_fetch_kfunc_arg_meta(env, insn->imm, insn->off, &meta);
- if (ret == 0 && bpf_is_iter_next_kfunc(&meta)) {
- mark_prune_point(env, t);
- /* Checking and saving state checkpoints at iter_next() call
- * is crucial for fast convergence of open-coded iterator loop
- * logic, so we need to force it. If we don't do that,
- * is_state_visited() might skip saving a checkpoint, causing
- * unnecessarily long sequence of not checkpointed
- * instructions and jumps, leading to exhaustion of jump
- * history buffer, and potentially other undesired outcomes.
- * It is expected that with correct open-coded iterators
- * convergence will happen quickly, so we don't run a risk of
- * exhausting memory.
- */
- mark_force_checkpoint(env, t);
- }
/* Same as helpers, if called in a non-sleepable context
* program will be rejected anyway, so we should end up
* with precise sleepable marks on subprogs, except for
@@ -649,18 +605,12 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
if (ret)
return ret;
- mark_prune_point(env, t + off + 1);
- mark_jmp_point(env, t + off + 1);
mark_jump_target(env, t + off + 1);
return ret;
default:
/* conditional jump with two edges */
- mark_prune_point(env, t);
- if (bpf_is_may_goto_insn(insn))
- mark_force_checkpoint(env, t);
-
ret = push_insn(t, t + 1, FALLTHROUGH, env);
if (ret)
return ret;
diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c
index b5204798231d..5acc11788300 100644
--- a/kernel/bpf/diagnostics.c
+++ b/kernel/bpf/diagnostics.c
@@ -392,13 +392,6 @@ static u32 log_pos(const struct bpf_diag_log *log, u32 idx)
return pos < log->cap ? pos : pos - log->cap;
}
-u64 bpf_diag_event_log_save(struct bpf_verifier_env *env)
-{
- struct bpf_diag *diag = env->diag;
-
- return diag ? log_end(&diag->log) : 0;
-}
-
void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos)
{
struct bpf_diag *diag = env->diag;
diff --git a/kernel/bpf/diagnostics.h b/kernel/bpf/diagnostics.h
index a4102fb049ec..55127e045aae 100644
--- a/kernel/bpf/diagnostics.h
+++ b/kernel/bpf/diagnostics.h
@@ -52,7 +52,6 @@ const char *bpf_diag_fmt(struct bpf_verifier_env *env, const char *fmt, ...) __p
const char *bpf_diag_fmt_btf_type(struct bpf_verifier_env *env, const struct btf *btf, u32 type_id);
const char *bpf_diag_reg_type_plain(struct bpf_verifier_env *env, enum bpf_reg_type type);
const char *bpf_diag_arg_type_plain(enum bpf_arg_type type);
-u64 bpf_diag_event_log_save(struct bpf_verifier_env *env);
void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos);
u32 bpf_diag_irq_depth(const struct bpf_verifier_state *state);
void bpf_diag_free(struct bpf_verifier_env *env);
diff --git a/kernel/bpf/scev.c b/kernel/bpf/scev.c
index 4da7ad48f99e..4dbfb4ed27e1 100644
--- a/kernel/bpf/scev.c
+++ b/kernel/bpf/scev.c
@@ -1722,10 +1722,6 @@ int bpf_compute_scev(struct bpf_verifier_env *env)
return err;
}
- for (i = 0; i < env->prog->len; i++)
- if (aux[i].loop)
- aux[i].prune_point = true;
-
if (env->log.level & BPF_LOG_LEVEL2) {
err = log_scevs(env);
if (err)
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index d0989d85d2cb..f1f185f9ce13 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -196,8 +196,7 @@ static void __clean_func_state(struct bpf_verifier_env *env,
}
}
-static int clean_verifier_state(struct bpf_verifier_env *env,
- struct bpf_verifier_state *st)
+int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
{
int i, err;
@@ -223,20 +222,10 @@ static bool regs_exact(const struct bpf_reg_state *rold,
check_ids(rold->map_uid, rcur->map_uid, idmap);
}
-enum exact_level {
- NOT_EXACT,
- EXACT,
- RANGE_WITHIN
-};
-
/* Returns true if (rold safe implies rcur safe) */
static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
- struct bpf_reg_state *rcur, struct bpf_idmap *idmap,
- enum exact_level exact)
+ struct bpf_reg_state *rcur, struct bpf_idmap *idmap)
{
- if (exact == EXACT)
- return regs_exact(rold, rcur, idmap);
-
if (rold->type == NOT_INIT)
/* explored state can't have used this */
return true;
@@ -425,8 +414,7 @@ static struct bpf_reg_state *scalar_reg_for_stack(struct bpf_verifier_env *env,
}
static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
- struct bpf_func_state *cur, struct bpf_idmap *idmap,
- enum exact_level exact)
+ struct bpf_func_state *cur, struct bpf_idmap *idmap)
{
int i, spi;
@@ -445,18 +433,6 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
old_type = old_slot->slot_type[im];
cur_slot = i < cur->allocated_stack ? bpf_stack_slot(cur, spi) : NULL;
- if (exact == EXACT) {
- u8 cur_type = cur_slot ? cur_slot->slot_type[im] : STACK_INVALID;
-
- /* STACK_INVALID and STACK_POISON are equivalent for pruning */
- if (old_type == STACK_POISON)
- old_type = STACK_INVALID;
- if (cur_type == STACK_POISON)
- cur_type = STACK_INVALID;
- if (!cur_slot || old_type != cur_type)
- return false;
- }
-
if (old_type == STACK_INVALID || old_type == STACK_POISON)
continue;
@@ -479,7 +455,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
old_reg = scalar_reg_for_stack(env, old_slot, im);
cur_reg = scalar_reg_for_stack(env, cur_slot, im);
if (old_reg && cur_reg) {
- if (!regsafe(env, old_reg, cur_reg, idmap, exact))
+ if (!regsafe(env, old_reg, cur_reg, idmap))
return false;
i += (im == 0 ? BPF_REG_SIZE - 1 : 3);
continue;
@@ -514,8 +490,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* such verifier states are not equivalent.
* return false to continue verification of this path
*/
- if (!regsafe(env, &old_slot->spilled_ptr, &cur_slot->spilled_ptr,
- idmap, exact))
+ if (!regsafe(env, &old_slot->spilled_ptr, &cur_slot->spilled_ptr, idmap))
return false;
break;
case STACK_DYNPTR:
@@ -569,8 +544,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
* Outgoing stack args are path-local state and must agree for pruning.
*/
static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
- struct bpf_func_state *cur, struct bpf_idmap *idmap,
- enum exact_level exact)
+ struct bpf_func_state *cur, struct bpf_idmap *idmap)
{
int i, nslots;
@@ -583,7 +557,7 @@ static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *
&old->stack_arg_regs[i] : ¬_init;
cur_arg = i < cur->out_stack_arg_cnt ?
&cur->stack_arg_regs[i] : ¬_init;
- if (!regsafe(env, old_arg, cur_arg, idmap, exact))
+ if (!regsafe(env, old_arg, cur_arg, idmap))
return false;
}
@@ -690,7 +664,7 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
* the current state will reach 'bpf_exit' instruction safely
*/
static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_state *old,
- struct bpf_func_state *cur, u32 insn_idx, enum exact_level exact)
+ struct bpf_func_state *cur, u32 insn_idx)
{
u16 live_regs = env->insn_aux_data[insn_idx].live_regs_before;
u16 i;
@@ -706,14 +680,13 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat
for (i = 0; i < MAX_BPF_REG; i++)
if (((1 << i) & live_regs) &&
- !regsafe(env, &old->regs[i], &cur->regs[i],
- &env->idmap_scratch, exact))
+ !regsafe(env, &old->regs[i], &cur->regs[i], &env->idmap_scratch))
return false;
- if (!stacksafe(env, old, cur, &env->idmap_scratch, exact))
+ if (!stacksafe(env, old, cur, &env->idmap_scratch))
return false;
- if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact))
+ if (!stack_arg_safe(env, old, cur, &env->idmap_scratch))
return false;
if (!loop_stack_safe(env, old, cur))
@@ -730,10 +703,9 @@ static void reset_idmap_scratch(struct bpf_verifier_env *env)
idmap->cnt = 0;
}
-static bool states_equal(struct bpf_verifier_env *env,
- struct bpf_verifier_state *old,
- struct bpf_verifier_state *cur,
- enum exact_level exact)
+/* Everything that @cur describes is described by @old. */
+bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
+ struct bpf_verifier_state *cur)
{
u32 insn_idx;
int i;
@@ -768,21 +740,10 @@ static bool states_equal(struct bpf_verifier_env *env,
insn_idx = bpf_frame_insn_idx(old, i);
if (old->frame[i]->callsite != cur->frame[i]->callsite)
return false;
- if (!func_states_equal(env, old->frame[i], cur->frame[i], insn_idx, exact))
+ if (!func_states_equal(env, old->frame[i], cur->frame[i], insn_idx))
return false;
}
return true;
}
-int bpf_clean_verifier_state(struct bpf_verifier_env *env, struct bpf_verifier_state *st)
-{
- return clean_verifier_state(env, st);
-}
-
-/* Everything that @cur describes is described by @old. */
-bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state *old,
- struct bpf_verifier_state *cur)
-{
- return states_equal(env, old, cur, RANGE_WITHIN);
-}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 245147ff3b1e..864009dbac47 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -185,7 +185,10 @@ static bool bpf_global_percpu_ma_set;
* reference remains at the end of the program, the verifier rejects it.
*/
-/* verifier_state + insn_idx are pushed to stack when branch is encountered */
+/*
+ * verifier_state + insn_idx are pushed to stack when branch is encountered.
+ * It's emptied after every insn, see flow_drain().
+ */
struct bpf_verifier_stack_elem {
/* verifier state is 'st'
* before processing instruction 'insn_idx'
@@ -197,9 +200,6 @@ struct bpf_verifier_stack_elem {
/* it goes there only if the guard at prev_insn_idx has to, see check_loop_guard() */
bool parked;
struct bpf_verifier_stack_elem *next;
- /* length of verifier log at the time this state was pushed on stack */
- u32 log_pos;
- u64 diag_log_pos;
};
#define BPF_GLOBAL_PERCPU_MA_MAX_SIZE 512
@@ -1782,36 +1782,6 @@ int bpf_copy_verifier_state(struct bpf_verifier_state *dst_state,
return 0;
}
-static int pop_stack(struct bpf_verifier_env *env, int *prev_insn_idx,
- int *insn_idx, bool pop_log)
-{
- struct bpf_verifier_state *cur = env->cur_state;
- struct bpf_verifier_stack_elem *elem, *head = env->head;
- int err;
-
- if (env->head == NULL)
- return -ENOENT;
-
- if (cur) {
- err = bpf_copy_verifier_state(cur, &head->st);
- if (err)
- return err;
- bpf_diag_event_log_restore(env, head->diag_log_pos);
- }
- if (pop_log)
- bpf_vlog_reset(&env->log, head->log_pos);
- if (insn_idx)
- *insn_idx = head->insn_idx;
- if (prev_insn_idx)
- *prev_insn_idx = head->prev_insn_idx;
- elem = head->next;
- bpf_free_verifier_state(&head->st, false);
- kfree(head);
- env->head = elem;
- env->stack_size--;
- return 0;
-}
-
static bool error_recoverable_with_nospec(int err)
{
/* Should only return true for non-fatal errors that are allowed to
@@ -1839,10 +1809,7 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env,
elem->insn_idx = insn_idx;
elem->prev_insn_idx = prev_insn_idx;
elem->next = env->head;
- elem->log_pos = env->log.end_pos;
- elem->diag_log_pos = bpf_diag_event_log_save(env);
env->head = elem;
- env->stack_size++;
err = bpf_copy_verifier_state(&elem->st, cur);
if (err)
return ERR_PTR(-ENOMEM);
@@ -2501,10 +2468,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env,
elem->insn_idx = insn_idx;
elem->prev_insn_idx = prev_insn_idx;
elem->next = env->head;
- elem->log_pos = env->log.end_pos;
- elem->diag_log_pos = bpf_diag_event_log_save(env);
env->head = elem;
- env->stack_size++;
/* Unlike push_stack() do not bpf_copy_verifier_state().
* The caller state doesn't matter.
* This is async callback. It starts in a fresh stack.
@@ -15208,7 +15172,6 @@ bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *ins
return info;
}
-/* insn_aux_data->is_iter_next is not there before the main pass */
bool bpf_insn_is_iter_next(struct bpf_verifier_env *env, struct bpf_insn *insn)
{
struct bpf_call_arg_meta meta;
@@ -15657,7 +15620,6 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
}
meta.fn = &desc->proto;
- insn_aux->is_iter_next = bpf_is_iter_next_kfunc(&meta);
if (!insn->off &&
(insn->imm == special_kfunc_list[KF_bpf_res_spin_lock] ||
@@ -21206,7 +21168,6 @@ static int flow_drain(struct bpf_verifier_env *env)
while ((elem = env->head)) {
env->head = elem->next;
- env->stack_size--;
if (!err && elem->parked)
err = bpf_flow_park(env, &elem->st, elem->insn_idx, elem->prev_insn_idx);
else if (!err)
@@ -22474,9 +22435,15 @@ static void sanitize_dead_code(struct bpf_verifier_env *env)
static void free_states(struct bpf_verifier_env *env)
{
+ struct bpf_verifier_stack_elem *elem;
+
bpf_free_verifier_state(env->cur_state, true);
env->cur_state = NULL;
- while (!pop_stack(env, NULL, NULL, false));
+ while ((elem = env->head)) {
+ env->head = elem->next;
+ bpf_free_verifier_state(&elem->st, false);
+ kfree(elem);
+ }
bpf_flow_reset(env);
}
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 62/81] bpf: Remove prune points, levels of state comparison and pop_stack()
2026-10-26 2:59 ` [PATCH bpf 62/81] bpf: Remove prune points, levels of state comparison and pop_stack() Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1837 bytes --]
> bpf: Remove prune points, levels of state comparison and pop_stack()
TL;DR: Cleanup nit only: with bpf_diag_event_log_save() gone,
bpf_diag_event_log_restore() is always called with 0, leaving dead branches
and a stale comment; could it become a plain reset?
> diff --git a/kernel/bpf/diagnostics.c b/kernel/bpf/diagnostics.c
> index b5204798231d..5acc11788300 100644
> --- a/kernel/bpf/diagnostics.c
> +++ b/kernel/bpf/diagnostics.c
[ ... ]
> @@ -392,13 +392,6 @@ static u32 log_pos(const struct bpf_diag_log *log, u32 idx)
> return pos < log->cap ? pos : pos - log->cap;
> }
>
> -u64 bpf_diag_event_log_save(struct bpf_verifier_env *env)
> -{
> - struct bpf_diag *diag = env->diag;
> -
> - return diag ? log_end(&diag->log) : 0;
> -}
> -
> void bpf_diag_event_log_restore(struct bpf_verifier_env *env, u64 log_pos)
> {
> struct bpf_diag *diag = env->diag;
This isn't a bug, but now that bpf_diag_event_log_save() is gone, the only
remaining callers of bpf_diag_event_log_restore() are in verifier.c, and both
pass 0:
bpf_diag_event_log_restore(env, 0);
With log_pos == 0, the WARN_ON_ONCE(log_pos > end_seq) clamp cannot fire, the
"if (log_pos <= log->first_seq)" test is always true, and the final
"log->cnt = log_pos - log->first_seq;" is unreachable.
The comment above that test ("A deep abandoned path may have rotated away the
shared prefix") describes the pop_stack() restore that this patch removes.
Could bpf_diag_event_log_restore() become a plain reset without the log_pos
argument, dropping the partial-restore branch and the abandoned-path comment
along with it?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 63/81] bpf: Print 0 as the limit in the statistics line
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (61 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 62/81] bpf: Remove prune points, levels of state comparison and pop_stack() Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:52 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 64/81] bpf: Remove bpf_compute_scc() Alexei Starovoitov
` (19 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The last line of the verifier log shows the number of processed insns and
the limit for it:
processed %d insns (limit %d) ...
The limit was BPF_COMPLEXITY_LIMIT_INSNS. Since
"bpf: Verify programs as a fixpoint over the control flow graph" the
verifier does not limit the number of processed insns. It limits the number
of states, see BPF_FLOW_MAX_STATES. BPF_COMPLEXITY_LIMIT_INSNS is now the
maximum length of a prog. Thus the line shows a limit that does not exist.
veristat and other tools parse this line. Keep the format and print 0.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf.h | 1 +
kernel/bpf/verifier.c | 5 +++--
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index d8cd7c63f157..8cd82f506e0d 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -2636,6 +2636,7 @@ struct bpf_array {
*/
int bpf_array_get_next_key(struct bpf_map *map, void *key, void *next_key);
+/* the longest program. There is no limit on what it takes to verify it. */
#define BPF_COMPLEXITY_LIMIT_INSNS 1000000 /* yes. 1M insns */
#define MAX_TAIL_CALL_CNT 33
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 864009dbac47..3bc0dcbeaa60 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -22783,9 +22783,10 @@ static void print_verification_stats(struct bpf_verifier_env *env)
env->subprog_info[i].stack_depth);
}
}
- verbose(env, "processed %d insns (limit %d) max_states_per_insn %d "
+ /* user space parses this line. There is no limit. */
+ verbose(env, "processed %d insns (limit 0) max_states_per_insn %d "
"total_states %d peak_states %d mark_read %d\n",
- env->insn_processed, BPF_COMPLEXITY_LIMIT_INSNS,
+ env->insn_processed,
env->max_states_per_insn, env->total_states,
env->peak_states, env->longest_mark_read_walk);
}
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 63/81] bpf: Print 0 as the limit in the statistics line
2026-10-26 2:59 ` [PATCH bpf 63/81] bpf: Print 0 as the limit in the statistics line Alexei Starovoitov
@ 2026-10-26 3:52 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:52 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3405 bytes --]
> bpf: Print 0 as the limit in the statistics line
TL;DR: Comments only: "no limit on verification" in bpf.h and verifier.c looks
inaccurate, since flow.c still rejects programs with -E2BIG when states exceed
bounds. Suggests narrower wording. No code issues raised.
> diff --git a/include/linux/bpf.h b/include/linux/bpf.h
> index d8cd7c63f157..8cd82f506e0d 100644
> --- a/include/linux/bpf.h
> +++ b/include/linux/bpf.h
> @@ -2636,6 +2636,7 @@ struct bpf_array {
> */
> int bpf_array_get_next_key(struct bpf_map *map, void *key, void *next_key);
>
> +/* the longest program. There is no limit on what it takes to verify it. */
> #define BPF_COMPLEXITY_LIMIT_INSNS 1000000 /* yes. 1M insns */
> #define MAX_TAIL_CALL_CNT 33
>
Is it accurate to say there is no limit on what it takes to verify the
program? The commit message says the verifier now limits the number of
states, see BPF_FLOW_MAX_STATES.
kernel/bpf/flow.c still rejects a program with -E2BIG in two cases. The
first is when a single insn has too many states that cannot be joined:
if (flow->nr_states[insn_idx] >= BPF_FLOW_MAX_STATES) {
verbose(env, "more than %d states that cannot be joined at insn %d\n",
BPF_FLOW_MAX_STATES, insn_idx);
The second is when the total number of states is too large for the program
size:
verbose(env, "too many states for %d insns: %d, %d at a time\n", ...
with the bound computed as BPF_FLOW_TOTAL_PER_INSN * env->prog->len +
BPF_FLOW_TOTAL_BASE.
So the effort to verify a program is bounded, and a program of
BPF_COMPLEXITY_LIMIT_INSNS insns can still be rejected for complexity. What
is true is narrower: the number of processed insns (env->insn_processed) is
no longer compared with any limit.
The same shorthand appears in the verifier.c comment "user space parses this
line. There is no limit."
Would it be more precise to say the macro is only the maximum program length
(it is still used by the insn_cnt check in bpf_prog_load(), by
BPF_MAX_GOTOX_EDGES in cfg.c and for sizing in liveness.c), and that
complexity is bounded by the number of states? For example:
/* the longest program; verification effort is bounded by the number of
* states, not by the number of processed insns */
[ ... ]
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 864009dbac47..3bc0dcbeaa60 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
> @@ -22783,9 +22783,10 @@ static void print_verification_stats(struct bpf_verifier_env *env)
> env->subprog_info[i].stack_depth);
> }
> }
> - verbose(env, "processed %d insns (limit %d) max_states_per_insn %d "
> + /* user space parses this line. There is no limit. */
> + verbose(env, "processed %d insns (limit 0) max_states_per_insn %d "
> "total_states %d peak_states %d mark_read %d\n",
> - env->insn_processed, BPF_COMPLEXITY_LIMIT_INSNS,
> + env->insn_processed,
> env->max_states_per_insn, env->total_states,
> env->peak_states, env->longest_mark_read_walk);
> }
The same wording applies to this comment. "There is no limit" could be
narrowed to say there is no limit on the number of processed insns.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 64/81] bpf: Remove bpf_compute_scc()
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (62 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 63/81] bpf: Print 0 as the limit in the statistics line Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 65/81] bpf: Update comments about paths, pruning and explored states Alexei Starovoitov
` (18 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
bpf_compute_scc() finds the strongly connected components (SCC) of the
control flow graph and gives each insn the number of its SCC. An SCC is an
outermost loop together with the loops in it. When the verifier followed
each path of a prog, is_state_visited() used the numbers to find the moment
when all paths through a loop were explored. Then it propagated precision
marks over the back edges of the loop.
"bpf: Remove is_state_visited()" removed the user of the numbers, and
"bpf: Remove precision tracking" removed the marks. The main pass finds
loops through loop_header in bpf_insn_aux_data.
Remove the calculation. The "Program dump" of log level 2 loses the scc
column.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 6 --
kernel/bpf/cfg.c | 190 -----------------------------------
kernel/bpf/verifier.c | 10 +-
3 files changed, 1 insertion(+), 205 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 9b107553461e..edac38665f54 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -759,11 +759,6 @@ struct bpf_insn_aux_data {
u64 jump_target:1;
unsigned int orig_idx; /* original instruction index, initialized once */
- /*
- * CFG strongly connected component this instruction belongs to,
- * zero if it is a singleton SCC.
- */
- u32 scc;
/* registers alive before this instruction. */
u16 live_regs_before;
/*
@@ -1632,7 +1627,6 @@ int bpf_prune_dead_branches(struct bpf_verifier_env *env);
int bpf_check_cfg(struct bpf_verifier_env *env);
void bpf_free_subprog_jts(struct bpf_verifier_env *env);
int bpf_compute_postorder(struct bpf_verifier_env *env);
-int bpf_compute_scc(struct bpf_verifier_env *env);
struct bpf_map_desc {
struct bpf_map *ptr;
diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
index 64eccc28703a..8973ec413b8c 100644
--- a/kernel/bpf/cfg.c
+++ b/kernel/bpf/cfg.c
@@ -784,193 +784,3 @@ int bpf_compute_postorder(struct bpf_verifier_env *env)
return 0;
}
-/*
- * Compute strongly connected components (SCCs) on the CFG.
- * Assign an SCC number to each instruction, recorded in env->insn_aux[*].scc.
- * If instruction is a sole member of its SCC and there are no self edges,
- * assign it SCC number of zero.
- * Uses a non-recursive adaptation of Tarjan's algorithm for SCC computation.
- */
-int bpf_compute_scc(struct bpf_verifier_env *env)
-{
- const u32 NOT_ON_STACK = U32_MAX;
-
- struct bpf_insn_aux_data *aux = env->insn_aux_data;
- const u32 insn_cnt = env->prog->len;
- int stack_sz, dfs_sz, err = 0;
- u32 *stack, *pre, *low, *dfs, *dfs_pos;
- u32 i, j, t, w;
- u32 next_preorder_num;
- u32 next_scc_id;
- bool assign_scc;
- struct bpf_iarray *succ;
-
- next_preorder_num = 1;
- next_scc_id = 1;
- /*
- * - 'stack' accumulates vertices in DFS order, see invariant comment below;
- * - 'pre[t] == p' => preorder number of vertex 't' is 'p';
- * - 'low[t] == n' => smallest preorder number of the vertex reachable from 't' is 'n';
- * - 'dfs' DFS traversal stack, used to emulate explicit recursion;
- * - 'dfs_pos[k] == j' => the frame 'dfs[k]' resumes visiting its
- * successors at index 'j'.
- */
- stack = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
- pre = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
- low = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
- dfs = kvcalloc(insn_cnt, sizeof(*dfs), GFP_KERNEL_ACCOUNT);
- dfs_pos = kvcalloc(insn_cnt, sizeof(*dfs_pos), GFP_KERNEL_ACCOUNT);
- if (!stack || !pre || !low || !dfs || !dfs_pos) {
- err = -ENOMEM;
- goto exit;
- }
- /*
- * References:
- * [1] R. Tarjan "Depth-First Search and Linear Graph Algorithms"
- * [2] D. J. Pearce "A Space-Efficient Algorithm for Finding Strongly Connected Components"
- *
- * The algorithm maintains the following invariant:
- * - suppose there is a path 'u' ~> 'v', such that 'pre[v] < pre[u]';
- * - then, vertex 'u' remains on stack while vertex 'v' is on stack.
- *
- * Consequently:
- * - If 'low[v] < pre[v]', there is a path from 'v' to some vertex 'u',
- * such that 'pre[u] == low[v]'; vertex 'u' is currently on the stack,
- * and thus there is an SCC (loop) containing both 'u' and 'v'.
- * - If 'low[v] == pre[v]', loops containing 'v' have been explored,
- * and 'v' can be considered the root of some SCC.
- *
- * Here is a pseudo-code for an explicitly recursive version of the algorithm:
- *
- * NOT_ON_STACK = insn_cnt + 1
- * pre = [0] * insn_cnt
- * low = [0] * insn_cnt
- * scc = [0] * insn_cnt
- * stack = []
- *
- * next_preorder_num = 1
- * next_scc_id = 1
- *
- * def recur(w):
- * nonlocal next_preorder_num
- * nonlocal next_scc_id
- *
- * pre[w] = next_preorder_num
- * low[w] = next_preorder_num
- * next_preorder_num += 1
- * stack.append(w)
- * for s in successors(w):
- * # Note: for classic algorithm the block below should look as:
- * #
- * # if pre[s] == 0:
- * # recur(s)
- * # low[w] = min(low[w], low[s])
- * # elif low[s] != NOT_ON_STACK:
- * # low[w] = min(low[w], pre[s])
- * #
- * # But replacing both 'min' instructions with 'low[w] = min(low[w], low[s])'
- * # does not break the invariant and makes iterative version of the algorithm
- * # simpler. See 'Algorithm #3' from [2].
- *
- * # 's' not yet visited
- * if pre[s] == 0:
- * recur(s)
- * # if 's' is on stack, pick lowest reachable preorder number from it;
- * # if 's' is not on stack 'low[s] == NOT_ON_STACK > low[w]',
- * # so 'min' would be a noop.
- * low[w] = min(low[w], low[s])
- *
- * if low[w] == pre[w]:
- * # 'w' is the root of an SCC, pop all vertices
- * # below 'w' on stack and assign same SCC to them.
- * while True:
- * t = stack.pop()
- * low[t] = NOT_ON_STACK
- * scc[t] = next_scc_id
- * if t == w:
- * break
- * next_scc_id += 1
- *
- * for i in range(0, insn_cnt):
- * if pre[i] == 0:
- * recur(i)
- *
- * Below implementation replaces explicit recursion with array 'dfs'.
- */
- for (i = 0; i < insn_cnt; i++) {
- if (pre[i])
- continue;
- stack_sz = 0;
- dfs_sz = 1;
- dfs[0] = i;
- dfs_pos[0] = 0;
-dfs_continue:
- while (dfs_sz) {
- w = dfs[dfs_sz - 1];
- if (pre[w] == 0) {
- low[w] = next_preorder_num;
- pre[w] = next_preorder_num;
- next_preorder_num++;
- stack[stack_sz++] = w;
- }
- /* Visit remaining 'w' successors */
- succ = bpf_insn_successors(env, w);
- for (j = dfs_pos[dfs_sz - 1]; j < succ->cnt; ++j) {
- if (pre[succ->items[j]]) {
- low[w] = min(low[w], low[succ->items[j]]);
- } else {
- /*
- * Once DFS for succ->items[j] is complete,
- * pre[succ->items[j]] is non-zero, hence
- * resuming at 'j' allows to follow the
- * low[w] = min(...) update branch above.
- */
- dfs_pos[dfs_sz - 1] = j;
- dfs_pos[dfs_sz] = 0;
- dfs[dfs_sz] = succ->items[j];
- dfs_sz++;
- goto dfs_continue;
- }
- }
- /*
- * Preserve the invariant: if some vertex above in the stack
- * is reachable from 'w', keep 'w' on the stack.
- */
- if (low[w] < pre[w]) {
- dfs_sz--;
- goto dfs_continue;
- }
- /*
- * Assign SCC number only if component has two or more elements,
- * or if component has a self reference, or if instruction is a
- * callback calling function (implicit loop).
- */
- assign_scc = stack[stack_sz - 1] != w; /* two or more elements? */
- for (j = 0; j < succ->cnt; ++j) { /* self reference? */
- if (succ->items[j] == w) {
- assign_scc = true;
- break;
- }
- }
- if (bpf_calls_callback(env, w)) /* implicit loop? */
- assign_scc = true;
- /* Pop component elements from stack */
- do {
- t = stack[--stack_sz];
- low[t] = NOT_ON_STACK;
- if (assign_scc)
- aux[t].scc = next_scc_id;
- } while (t != w);
- if (assign_scc)
- next_scc_id++;
- dfs_sz--;
- }
- }
-exit:
- kvfree(stack);
- kvfree(pre);
- kvfree(low);
- kvfree(dfs);
- kvfree(dfs_pos);
- return err;
-}
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 3bc0dcbeaa60..51106df0e76e 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -24525,13 +24525,9 @@ static void log_program(struct bpf_verifier_env *env)
u64 pos, insn_pos;
u32 i, j;
- verbose(env, "Program dump (scc? loop_header? idom insn#: live_regs_before):\n");
+ verbose(env, "Program dump (loop_header? idom insn#: live_regs_before):\n");
for (i = 0; i < insn_cnt; ++i) {
verbose_linfo(env, i, " ; ");
- if (env->insn_aux_data[i].scc)
- verbose(env, "%3d ", env->insn_aux_data[i].scc);
- else
- verbose(env, " ");
if (env->insn_aux_data[i].loop_header >= 0)
verbose(env, "%3d ", env->insn_aux_data[i].loop_header);
else
@@ -24742,10 +24738,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
if (ret < 0)
goto skip_full_check;
- ret = bpf_compute_scc(env);
- if (ret < 0)
- goto skip_full_check;
-
/* must precede the first bpf_ret_reg_pair() user below */
ret = bpf_compute_subprog_ret_regs(env);
if (ret < 0)
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 64/81] bpf: Remove bpf_compute_scc()
2026-10-26 2:59 ` [PATCH bpf 64/81] bpf: Remove bpf_compute_scc() Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]
> bpf: Remove bpf_compute_scc()
TL;DR: Nit only: removing bpf_compute_scc() leaves a stray blank line at the
end of kernel/bpf/cfg.c, which git diff --check flags; it should be dropped.
> diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c
> index 64eccc28703a..8973ec413b8c 100644
> --- a/kernel/bpf/cfg.c
> +++ b/kernel/bpf/cfg.c
> @@ -784,193 +784,3 @@ int bpf_compute_postorder(struct bpf_verifier_env *env)
> return 0;
> }
>
> -/*
> - * Compute strongly connected components (SCCs) on the CFG.
[ ... ]
> - return err;
> -}
bpf_compute_scc() was the last function in kernel/bpf/cfg.c. The hunk
removes it but keeps the blank line that used to separate it from
bpf_compute_postorder(), so the file now ends with:
}
(blank line)
Is this trailing blank line intended? git diff --check on this commit
reports:
kernel/bpf/cfg.c: new blank line at EOF.
The blank line is still present at the end of the series (51224d16c11b).
Could the context line before the removed "/*" be dropped, so that the file
ends right after the closing brace of bpf_compute_postorder()?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 65/81] bpf: Update comments about paths, pruning and explored states
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (63 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 64/81] bpf: Remove bpf_compute_scc() Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 66/81] bpf: Remove iter.depth Alexei Starovoitov
` (17 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Many comments in verifier.c and states.c describe a verifier that explores
all paths of a prog and prunes a path against explored states. Since
"bpf: Verify programs as a fixpoint over the control flow graph" the
verifier does not do that.
Describe what it does. States meet at the start of a basic block. The
verifier drops a state when a state that is already there covers it.
Otherwise it joins the two states or keeps the two.
The largest change is above process_iter_next_call(). The verifier verifies
a loop with an open-coded iterator in the same way as other loops. Thus
remove the description of the special convergence logic.
No functional changes.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 4 +-
kernel/bpf/states.c | 56 ++++++++-------------
kernel/bpf/verifier.c | 97 +++++++-----------------------------
3 files changed, 41 insertions(+), 116 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index edac38665f54..d02f6938c2e1 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -51,7 +51,7 @@ enum bpf_iter_state {
};
struct bpf_reg_state {
- /* Ordering of fields matters. See states_equal() */
+ /* Ordering of fields matters. See regsafe() */
enum bpf_reg_type type;
/*
* Constant delta between "linked" scalars with the same ID.
@@ -1758,7 +1758,7 @@ bool bpf_is_may_goto_insn(struct bpf_insn *insn);
void bpf_verbose_insn(struct bpf_verifier_env *env, struct bpf_insn *insn);
bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call,
struct bpf_call_summary *cs);
-/* Stack effects used for state pruning; must_write implies may_write. */
+/* Stack effects used for liveness; must_write implies may_write. */
struct arg_access_info {
u32 size; /* Maximum extent; U32_MAX if unknown. */
u8 may_read:1; /* Incoming contents or initialization may be needed. */
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index f1f185f9ce13..5487987b2441 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -227,7 +227,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
struct bpf_reg_state *rcur, struct bpf_idmap *idmap)
{
if (rold->type == NOT_INIT)
- /* explored state can't have used this */
+ /* nothing can be done with it */
return true;
/* Enforce that register types have to match exactly, including their
@@ -306,7 +306,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
* ADD_CONST flags must match exactly: BPF_ADD_CONST32 and
* BPF_ADD_CONST64 have different linking semantics in
* sync_linked_regs() (alu32 zero-extends, alu64 does not),
- * so pruning across different flag types is unsafe.
+ * so one doesn't stand for the other.
*/
if (rold->id &&
(rold->id & BPF_ADD_CONST) != (rcur->id & BPF_ADD_CONST))
@@ -418,9 +418,9 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
{
int i, spi;
- /* walk slots of the explored stack and ignore any additional
- * slots in the current stack, since explored(safe) state
- * didn't use them
+ /* walk slots of the old stack and ignore any additional
+ * slots in the current stack, since the old state
+ * can't use them
*/
for (i = 0; i < old->allocated_stack; i++) {
struct bpf_stack_state *old_slot, *cur_slot;
@@ -439,9 +439,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
if (env->allow_uninit_stack && old_type == STACK_MISC)
continue;
- /* explored stack has more populated slots than current stack
- * and these slots were used
- */
+ /* old stack has more populated slots than current stack */
if (!cur_slot)
return false;
@@ -469,10 +467,9 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
if (old_type == STACK_MISC && cur_slot->slot_type[im] == STACK_ZERO)
continue;
if (old_type != cur_slot->slot_type[im])
- /* Ex: old explored (safe) state has STACK_SPILL in
+ /* Ex: old state has STACK_SPILL in
* this stack slot, but current has STACK_MISC ->
- * this verifier states are not equivalent,
- * return false to continue verification of this path
+ * this verifier states are not equivalent
*/
return false;
if (im != BPF_REG_SIZE - 1)
@@ -480,15 +477,14 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
/* Both old and cur are having same slot_type */
switch (old_type) {
case STACK_SPILL:
- /* when explored and current stack slot are both storing
+ /* when old and current stack slot are both storing
* spilled registers, check that stored pointers types
* are the same as well.
- * Ex: explored safe path could have stored
+ * Ex: old state could have stored
* (bpf_reg_state) {.type = PTR_TO_STACK, .off = -8}
- * but current path has stored:
+ * but current state has stored:
* (bpf_reg_state) {.type = PTR_TO_STACK, .off = -16}
* such verifier states are not equivalent.
- * return false to continue verification of this path
*/
if (!regsafe(env, &old_slot->spilled_ptr, &cur_slot->spilled_ptr, idmap))
return false;
@@ -541,7 +537,7 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
/*
* Compare stack arg slots between old and current states.
- * Outgoing stack args are path-local state and must agree for pruning.
+ * Outgoing stack args are a part of the state and must agree.
*/
static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old,
struct bpf_func_state *cur, struct bpf_idmap *idmap)
@@ -639,29 +635,17 @@ static bool loop_stack_safe(struct bpf_verifier_env *env, struct bpf_func_state
/* compare two verifier states
*
- * all states stored in state_list are known to be valid, since
- * verifier reached 'bpf_exit' instruction through them
- *
- * this function is called when verifier exploring different branches of
- * execution popped from the state stack. If it sees an old state that has
- * more strict register state and more strict stack state then this execution
- * branch doesn't need to be explored further, since verifier already
- * concluded that more strict state leads to valid finish.
- *
- * Therefore two states are equivalent if register state is more conservative
- * and explored stack state is more conservative than the current one.
+ * The state that gets to an insn doesn't have to be looked at when there is
+ * a state at that insn that is more conservative in registers and in the
+ * stack: if the program is fine with the old state it's fine with the current
+ * one.
* Example:
- * explored current
+ * old current
* (slot1=INV slot2=MISC) == (slot1=MISC slot2=MISC)
* (slot1=MISC slot2=MISC) != (slot1=INV slot2=MISC)
*
- * In other words if current stack state (one being explored) has more
- * valid slots than old one that already passed validation, it means
- * the verifier can stop exploring and conclude that current state is valid too
- *
- * Similarly with registers. If explored state has register type as invalid
- * whereas register type in current state is meaningful, it means that
- * the current state will reach 'bpf_exit' instruction safely
+ * Similarly with registers: the type in the old state is invalid whereas
+ * the type in the current state is meaningful.
*/
static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_state *old,
struct bpf_func_state *cur, u32 insn_idx)
@@ -716,7 +700,7 @@ bool bpf_state_subsumes(struct bpf_verifier_env *env, struct bpf_verifier_state
reset_idmap_scratch(env);
/* Verification state from speculative execution simulation
- * must never prune a non-speculative execution one.
+ * must never stand for a non-speculative execution one.
*/
if (old->speculative && !cur->speculative)
return false;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 51106df0e76e..2e46287deb08 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -62,19 +62,16 @@ static bool bpf_global_percpu_ma_set;
/* bpf_check() is a static code analyzer that walks eBPF program
* instruction by instruction and updates register/stack state.
- * All paths of conditional branches are analyzed until 'bpf_exit' insn.
*
- * The first pass is depth-first-search to check that the program is a DAG.
+ * The first pass is depth-first-search over the control flow graph.
* It rejects the following programs:
- * - larger than BPF_MAXINSNS insns
- * - if loop is present (detected via back-edge)
+ * - if loop is present (detected via back-edge) and the program is unprivileged
* - unreachable insns exist (shouldn't be a forest. program = one function)
* - out of bounds or malformed jumps
- * The second pass is all possible path descent from the 1st insn.
- * Since it's analyzing all paths through the program, the length of the
- * analysis is limited to 64k insn, which may be hit even if total number of
- * insn is less then 4K, but there are too many branches that change stack/regs.
- * Number of 'branches to be analyzed' is limited to 1k
+ * The second pass goes from basic block to basic block, starting at the 1st
+ * insn. States that meet at the start of a block are joined, so the length of
+ * the analysis follows the size of the program and not the number of paths
+ * through it. See flow.c.
*
* On entry to each instruction, each register has a type, and the instruction
* changes the types of the registers depending on instruction semantics.
@@ -1343,7 +1340,7 @@ static bool is_stack_slot_special(const struct bpf_stack_state *stack)
* - STACK_POISON, which truly forbids access to the slot.
* Regardless of allow_ptr_leaks setting (i.e., privileged or unprivileged
* mode), we won't promote STACK_INVALID to STACK_MISC. In privileged case it is
- * unnecessary as both are considered equivalent when loading data and pruning,
+ * unnecessary as both are considered equivalent when loading data and comparing states,
* in case of unprivileged mode it will be incorrect to allow reads of invalid
* slots.
*/
@@ -8641,69 +8638,13 @@ static struct bpf_reg_state *get_iter_from_state(struct bpf_verifier_state *cur_
* without having to simulate infinite amount of iterations.
*
* In current state, we first assume that iter_next() returned NULL and
- * iterator state is set to DRAINED (BPF_ITER_STATE_DRAINED). In such
- * conditions we should not form an infinite loop and should eventually reach
- * exit.
+ * iterator state is set to DRAINED (BPF_ITER_STATE_DRAINED).
*
- * Besides that, we also fork current state and enqueue it for later
- * verification. In a forked state we keep iterator state as ACTIVE
- * (BPF_ITER_STATE_ACTIVE) and assume non-NULL return from iter_next(). We
- * also bump iteration depth to prevent erroneous infinite loop detection
- * later on (see iter_active_depths_differ() comment for details). In this
- * state we assume that we'll eventually loop back to another iter_next()
- * calls (it could be in exactly same location or in some other instruction,
- * it doesn't matter, we don't make any unnecessary assumptions about this,
- * everything revolves around iterator state in a stack slot, not which
- * instruction is calling iter_next()). When that happens, we either will come
- * to iter_next() with equivalent state and can conclude that next iteration
- * will proceed in exactly the same way as we just verified, so it's safe to
- * assume that loop converges. If not, we'll go on another iteration
- * simulation with a different input state, until all possible starting states
- * are validated or we reach maximum number of instructions limit.
- *
- * This way, we will either exhaustively discover all possible input states
- * that iterator loop can start with and eventually will converge, or we'll
- * effectively regress into bounded loop simulation logic and either reach
- * maximum number of instructions if loop is not provably convergent, or there
- * is some statically known limit on number of iterations (e.g., if there is
- * an explicit `if n > 100 then break;` statement somewhere in the loop).
- *
- * Iteration convergence logic in is_state_visited() relies on exact
- * states comparison, which ignores read and precision marks.
- * This is necessary because read and precision marks are not finalized
- * while in the loop. Exact comparison might preclude convergence for
- * simple programs like below:
- *
- * i = 0;
- * while(iter_next(&it))
- * i++;
- *
- * At each iteration step i++ would produce a new distinct state and
- * eventually instruction processing limit would be reached.
- *
- * To avoid such behavior speculatively forget (widen) range for
- * imprecise scalar registers, if those registers were not precise at the
- * end of the previous iteration and do not match exactly.
- *
- * This is a conservative heuristic that allows to verify wide range of programs,
- * however it precludes verification of programs that conjure an
- * imprecise value on the first loop iteration and use it as precise on a second.
- * For example, the following safe program would fail to verify:
- *
- * struct bpf_num_iter it;
- * int arr[10];
- * int i = 0, a = 0;
- * bpf_iter_num_new(&it, 0, 10);
- * while (bpf_iter_num_next(&it)) {
- * if (a == 0) {
- * a = 1;
- * i = 7; // Because i changed verifier would forget
- * // it's range on second loop entry.
- * } else {
- * arr[i] = 42; // This would fail to verify.
- * }
- * }
- * bpf_iter_num_destroy(&it);
+ * Besides that, we also fork current state. In a forked state we keep iterator
+ * state as ACTIVE (BPF_ITER_STATE_ACTIVE) and assume non-NULL return from
+ * iter_next(). This state goes around the loop like in any other loop: what
+ * comes back to the head of the loop is joined with what is there until
+ * nothing changes, see flow.c.
*/
/*
* The program asks for the next element after it was told that there is none. So it didn't
@@ -12348,7 +12289,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx)
}
account_processed_insns(env, callee, caller);
/* clear everything in the callee. In case of exceptional exits using
- * bpf_throw, this will be done by copy_verifier_state for extra frames. */
+ * bpf_throw, this will be done by do_check_block() for extra frames. */
free_func_state(callee);
state->frame[state->curframe--] = NULL;
invalidate_outgoing_stack_args(env, caller);
@@ -15039,7 +14980,7 @@ int bpf_fetch_kfunc_arg_meta(struct bpf_verifier_env *env,
* Describe a helper's stack access through argument @arg (0-based, R1-R5).
* must_write means the verifier destroys prior state throughout size bytes.
* In other words, must_write follows the verifier's model of the
- * function's behaviour, which is safe to use for is_state_visited() pruning.
+ * function's behaviour, which is safe to use when states are cleaned and compared.
*/
struct arg_access_info
bpf_helper_stack_access_bytes(struct bpf_verifier_env *env, struct bpf_insn *insn,
@@ -16300,7 +16241,7 @@ static int sanitize_ptr_alu(struct bpf_verifier_env *env,
alu_state |= ptr_is_dst_reg ?
BPF_ALU_SANITIZE_SRC : BPF_ALU_SANITIZE_DST;
- /* Limit pruning on unknown scalars to enable deep search for
+ /* Keep states that differ in unknown scalars apart to find
* potential masking differences from other program paths.
*/
if (!off_is_imm)
@@ -18875,7 +18816,7 @@ static void mark_ptr_or_null_reg(struct bpf_func_state *state,
if (is_null) {
/* We don't need id from this point
* onwards anymore, thus we should better reset it,
- * so that state pruning has chances to take effect.
+ * so that states have chances to be equal.
*/
__mark_reg_known_zero(reg);
reg->type = SCALAR_VALUE;
@@ -20335,8 +20276,8 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
/* The side effect of the prepare_func_exit which is
* being skipped is that it frees bpf_func_state.
* Typically, process_bpf_exit will only be hit with
- * outermost exit. copy_verifier_state in pop_stack will
- * handle freeing of any extra bpf_func_state left over
+ * outermost exit. do_check_block() frees the state with
+ * any extra bpf_func_state left over
* from not processing all nested function exits. We
* also skip return code checks as they are not needed
* for exceptional exits.
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 65/81] bpf: Update comments about paths, pruning and explored states
2026-10-26 2:59 ` [PATCH bpf 65/81] bpf: Update comments about paths, pruning and explored states Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 4496 bytes --]
> bpf: Update comments about paths, pruning and explored states
TL;DR: Comment-only patch; the new verifier.c comment may overstate that
joining makes analysis length follow program size, since flow.c keeps states
apart and enforces limits. Suggests rewording.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -62,19 +62,16 @@ static bool bpf_global_percpu_ma_set;
>
> /* bpf_check() is a static code analyzer that walks eBPF program
> * instruction by instruction and updates register/stack state.
> - * All paths of conditional branches are analyzed until 'bpf_exit' insn.
> *
> - * The first pass is depth-first-search to check that the program is a DAG.
> + * The first pass is depth-first-search over the control flow graph.
> * It rejects the following programs:
> [ ... ]
> - * The second pass is all possible path descent from the 1st insn.
> - * Since it's analyzing all paths through the program, the length of the
> - * analysis is limited to 64k insn, which may be hit even if total number of
> - * insn is less then 4K, but there are too many branches that change stack/regs.
> - * Number of 'branches to be analyzed' is limited to 1k
> + * The second pass goes from basic block to basic block, starting at the 1st
> + * insn. States that meet at the start of a block are joined, so the length of
> + * the analysis follows the size of the program and not the number of paths
> + * through it. See flow.c.
> *
> * On entry to each instruction, each register has a type, and the instruction
> * changes the types of the registers depending on instruction semantics.
Is it accurate to say that states meeting at the start of a block are
joined, and that the length of the analysis therefore follows the size of
the program?
The commit message itself says the verifier drops a state that is covered,
"Otherwise it joins the two states or keeps the two", and flow_merge() in
kernel/bpf/flow.c deliberately keeps joinable states apart while there is
room:
/*
* States that have a join are kept apart while there is room: half of
* every limit. After that they are joined, which may lose what the
* prog needs, and does not run into the limit.
*/
if (!widen && flow_has_room(env, flow->nr_states[insn_idx]) &&
flow_null_differs(&t->st, &s->st))
continue;
...
if (!widen && flow_has_room(env, flow->nr_states[insn_idx]) &&
(flow_unset_differs(&t->st, &s->st) ||
flow_join_makes_up(flow, &t->st, &s->st)))
continue;
States of a different flow_kind() are never compared or joined, and when
bpf_states_join() finds no join (for example scalars that differ while
env->explore_alu_limits is set, which the sanitize_ptr_alu() comment in
this same patch describes as keeping states apart), the state is added to
the set with list_add_tail(&s->node, set).
bpf_flow_loop_enter() also walks some loops one iteration at a time, with
each trip a separate kind of state:
/*
* There is no summary of what the loop does to something that is
* live. Go through the iterations one by one if that is not too long.
*/
if (err == 2 && flow_walk_factor(cur) * iters.max_header_count *
loop->nr_insns <= BPF_FLOW_MAX_WALK) {
...
entry->apart = true;
For such a loop the work follows the trip count, up to BPF_FLOW_MAX_WALK
insns, which is not the size of the program.
What bounds the work is the set of limits in flow.c
(BPF_FLOW_TOTAL_PER_INSN / BPF_FLOW_MAX_STATES), and a program over them
fails with -E2BIG ("too many states for %d insns" or "more than %d states
that cannot be joined"). The new text drops every mention of a limit and
gives joining as the reason for the bound.
The header comment in flow.c describes the three outcomes (dropped, joined,
added) correctly. Could this summary match it, for example:
A state that arrives at the start of a block is dropped if a state
there covers it, joined with one if possible, or kept; the number of
states is bounded by the size of the program, see flow.c.
No later commit in be06e70f1a7f..51224d16c11b changes this comment.
[ ... ]
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 66/81] bpf: Remove iter.depth
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (64 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 65/81] bpf: Update comments about paths, pruning and explored states Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 67/81] bpf: Remove leftovers of precision tracking and read marks Alexei Starovoitov
` (16 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The state of an open-coded iterator has a counter, iter.depth. The verifier
increments it at each bpf_iter_*_next() that returns an element. The
counter had one user: the check for infinite loops in is_state_visited().
That check rejected a prog when a path came back to an insn in an identical
state. In a loop with an iterator this is not an error, because the
iterator stops the loop. A different depth told the check that this is a
new iteration.
"bpf: Remove is_state_visited()" removed the check. The comparison of
states and the join of states ignore depth. Remove it. The verifier log
does not print "depth=" for an iterator.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 4 +---
kernel/bpf/join.c | 1 -
kernel/bpf/log.c | 5 ++---
kernel/bpf/states.c | 7 -------
kernel/bpf/verifier.c | 3 ---
5 files changed, 3 insertions(+), 17 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index d02f6938c2e1..f46963591e18 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -104,9 +104,7 @@ struct bpf_reg_state {
*/
struct btf *btf;
u32 btf_id;
- /* packing following two fields to fit iter state into 16 bytes */
- enum bpf_iter_state state:2;
- int depth:30;
+ enum bpf_iter_state state;
} iter;
/* For irq stack slots */
diff --git a/kernel/bpf/join.c b/kernel/bpf/join.c
index dde40dc4ff4f..477fe159811b 100644
--- a/kernel/bpf/join.c
+++ b/kernel/bpf/join.c
@@ -346,7 +346,6 @@ static bool join_special_slot(struct bpf_join_state *js, const struct bpf_stack_
return false;
break;
case STACK_ITER:
- /* iter.depth is not a part of the state, see stacksafe() */
if (ra->type != rb->type || ra->iter.btf != rb->iter.btf ||
ra->iter.btf_id != rb->iter.btf_id || ra->iter.state != rb->iter.state)
return false;
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index 1c7a13bfee87..0013ab01e72d 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -767,11 +767,10 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie
if (!reg->id)
continue;
- verbose(env, " fp%d=iter_%s(id=%d,state=%s,depth=%u)",
+ verbose(env, " fp%d=iter_%s(id=%d,state=%s)",
(-i - 1) * BPF_REG_SIZE,
iter_type_str(reg->iter.btf, reg->iter.btf_id),
- reg->id, iter_state_str(reg->iter.state),
- reg->iter.depth);
+ reg->id, iter_state_str(reg->iter.state));
break;
case STACK_MISC:
case STACK_ZERO:
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 5487987b2441..91c3a23b6e12 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -501,17 +501,10 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old,
case STACK_ITER:
old_reg = &old_slot->spilled_ptr;
cur_reg = &cur_slot->spilled_ptr;
- /* iter.depth is not compared between states as it
- * doesn't matter for correctness and would otherwise
- * prevent convergence; we maintain it only to prevent
- * infinite loop check triggering, see
- * iter_active_depths_differ()
- */
if (old_reg->type != cur_reg->type ||
old_reg->iter.btf != cur_reg->iter.btf ||
old_reg->iter.btf_id != cur_reg->iter.btf_id ||
old_reg->iter.state != cur_reg->iter.state ||
- /* ignore {old_reg,cur_reg}->iter.depth, see above */
!check_ids(old_reg->id, cur_reg->id, idmap))
return false;
break;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 2e46287deb08..17a9d1bf5318 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -1053,7 +1053,6 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env,
st->iter.btf = btf;
st->iter.btf_id = btf_id;
st->iter.state = BPF_ITER_STATE_ACTIVE;
- st->iter.depth = 0;
for (j = 0; j < BPF_REG_SIZE; j++)
slot->slot_type[j] = STACK_ITER;
@@ -8703,13 +8702,11 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
queued_iter = get_iter_from_state(queued_st, meta);
queued_iter->iter.state = BPF_ITER_STATE_ACTIVE;
- queued_iter->iter.depth++;
queued_fr = queued_st->frame[queued_st->curframe];
mark_ptr_not_null_reg(&queued_fr->regs[BPF_REG_0]);
}
- /* switch to DRAINED state, but keep the depth unchanged */
/* mark current iter state as drained and assume returned NULL */
cur_iter->iter.state = BPF_ITER_STATE_DRAINED;
__mark_reg_const_zero(env, &cur_fr->regs[BPF_REG_0]);
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 67/81] bpf: Remove leftovers of precision tracking and read marks
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (65 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 66/81] bpf: Remove iter.depth Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 68/81] bpf: Track a pointer to one of a set of functions Alexei Starovoitov
` (15 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
"bpf: Remove precision tracking" removed the 'precise' bit and backtrack.c.
This code has no purpose after that:
- __mark_reg_unknown() and __mark_reg_const_zero() have the argument env
only to set the bit. Replace the first function with
bpf_mark_reg_unknown(). Remove env from the second function.
- Only backtrack.c used BPF_REGMASK_ARGS.
The verifier has no read marks, and nothing sets longest_mark_read_walk.
Remove it. veristat and other tools parse the statistics line of the
verifier log. Thus the line continues to show "mark_read 0".
No functional changes.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 6 ----
kernel/bpf/states.c | 2 --
kernel/bpf/verifier.c | 53 +++++++++++++++---------------------
3 files changed, 22 insertions(+), 39 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index f46963591e18..0cd7fa277fe5 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -274,10 +274,6 @@ enum bpf_stack_slot_type {
#define BPF_HALF_REG_SIZE 4
#define STACK_SLOT_SZ 4
-#define BPF_REGMASK_ARGS ((1 << BPF_REG_1) | (1 << BPF_REG_2) | \
- (1 << BPF_REG_3) | (1 << BPF_REG_4) | \
- (1 << BPF_REG_5))
-
#define BPF_MAIN_FUNC (-1)
#define BPF_DYNPTR_SIZE sizeof(struct bpf_dynptr_kern)
@@ -1088,8 +1084,6 @@ struct bpf_verifier_env {
* memory consumption during verification
*/
u32 peak_states;
- /* longest register parentage chain walked for liveness marking */
- u32 longest_mark_read_walk;
/*
* The program's fd_array comes in two shapes, told apart by whether
* the caller passed fd_array_cnt. They are mutually exclusive:
diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c
index 91c3a23b6e12..8b7c13be9265 100644
--- a/kernel/bpf/states.c
+++ b/kernel/bpf/states.c
@@ -5,8 +5,6 @@
#include <linux/cnum.h>
#include <linux/filter.h>
-#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
-
/* check %cur's range satisfies %old's */
static bool range_within(const struct bpf_reg_state *old,
const struct bpf_reg_state *cur)
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 17a9d1bf5318..21ea43422133 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -847,15 +847,12 @@ static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_re
: reg->id);
}
-static void __mark_reg_unknown(const struct bpf_verifier_env *env,
- struct bpf_reg_state *reg);
-
static void mark_reg_invalid(const struct bpf_verifier_env *env, struct bpf_reg_state *reg)
{
if (!env->allow_ptr_leaks)
bpf_mark_reg_not_init(env, reg);
else
- __mark_reg_unknown(env, reg);
+ bpf_mark_reg_unknown(reg);
}
static int dynptr_ref_cnt(struct bpf_verifier_env *env, int v_parent_id)
@@ -1998,7 +1995,7 @@ static void __mark_reg_known_zero(struct bpf_reg_state *reg)
__mark_reg_known(reg, 0);
}
-static void __mark_reg_const_zero(const struct bpf_verifier_env *env, struct bpf_reg_state *reg)
+static void __mark_reg_const_zero(struct bpf_reg_state *reg)
{
__mark_reg_known(reg, 0);
reg->type = SCALAR_VALUE;
@@ -2343,16 +2340,10 @@ void bpf_mark_reg_unknown(struct bpf_reg_state *reg)
reg_step_reset(reg);
}
-static void __mark_reg_unknown(const struct bpf_verifier_env *env,
- struct bpf_reg_state *reg)
-{
- bpf_mark_reg_unknown(reg);
-}
-
static void mark_reg_unknown(struct bpf_verifier_env *env,
struct bpf_reg_state *regs, u32 regno)
{
- __mark_reg_unknown(env, regs + regno);
+ bpf_mark_reg_unknown(regs + regno);
}
static int __mark_reg_s32_range(struct bpf_verifier_env *env,
@@ -2378,7 +2369,7 @@ static int __mark_reg_s32_range(struct bpf_verifier_env *env,
void bpf_mark_reg_not_init(const struct bpf_verifier_env *env,
struct bpf_reg_state *reg)
{
- __mark_reg_unknown(env, reg);
+ bpf_mark_reg_unknown(reg);
reg->type = NOT_INIT;
}
@@ -3973,7 +3964,7 @@ static void mark_reg_stack_read(struct bpf_verifier_env *env,
/* Any access_size read into register is zero extended,
* so the whole register == const_zero.
*/
- __mark_reg_const_zero(env, &state->regs[dst_regno]);
+ __mark_reg_const_zero(&state->regs[dst_regno]);
} else {
/* have read misc data from the stack */
mark_reg_unknown(env, state->regs, dst_regno);
@@ -4115,10 +4106,10 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env,
if (spill_cnt == size &&
tnum_is_const(reg->var_off) && reg->var_off.value == 0) {
- __mark_reg_const_zero(env, &state->regs[dst_regno]);
+ __mark_reg_const_zero(&state->regs[dst_regno]);
} else if (zero_cnt == size) {
/* similarly to mark_reg_stack_read(), preserve zeroes */
- __mark_reg_const_zero(env, &state->regs[dst_regno]);
+ __mark_reg_const_zero(&state->regs[dst_regno]);
} else {
mark_reg_stack_read(env, reg_state, off, off + size,
dst_regno);
@@ -7744,7 +7735,7 @@ static int check_stack_range_initialized(
(ss->spilled_ptr.type == SCALAR_VALUE ||
env->allow_ptr_leaks)) {
if (clobber) {
- __mark_reg_unknown(env, &ss->spilled_ptr);
+ bpf_mark_reg_unknown(&ss->spilled_ptr);
for (j = 0; j < BPF_REG_SIZE; j++)
scrub_spilled_slot(&ss->slot_type[j]);
}
@@ -8709,7 +8700,7 @@ static int process_iter_next_call(struct bpf_verifier_env *env, int insn_idx,
/* mark current iter state as drained and assume returned NULL */
cur_iter->iter.state = BPF_ITER_STATE_DRAINED;
- __mark_reg_const_zero(env, &cur_fr->regs[BPF_REG_0]);
+ __mark_reg_const_zero(&cur_fr->regs[BPF_REG_0]);
return 0;
}
@@ -15815,7 +15806,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
ret_nregs = mark_kfunc_ret_regs(env, regs, t->size);
if (meta.btf == btf_vmlinux && (meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock] ||
meta.func_id == special_kfunc_list[KF_bpf_res_spin_lock_irqsave]))
- __mark_reg_const_zero(env, ®s[BPF_REG_0]);
+ __mark_reg_const_zero(®s[BPF_REG_0]);
} else if (btf_type_is_struct(t)) {
struct btf_member_path path = {};
const char *member_note = "";
@@ -16445,14 +16436,14 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
/* Taint dst register if offset had invalid bounds derived from
* e.g. dead branches.
*/
- __mark_reg_unknown(env, dst_reg);
+ bpf_mark_reg_unknown(dst_reg);
return 0;
}
if (BPF_CLASS(insn->code) != BPF_ALU64 && !to_scalar) {
/* 32-bit ALU ops on pointers produce (meaningless) scalars */
if (opcode == BPF_SUB && env->allow_ptr_leaks) {
- __mark_reg_unknown(env, dst_reg);
+ bpf_mark_reg_unknown(dst_reg);
return 0;
}
@@ -16516,7 +16507,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn
}
if (to_scalar) {
- __mark_reg_unknown(env, dst_reg);
+ bpf_mark_reg_unknown(dst_reg);
return 0;
}
@@ -17463,7 +17454,7 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env,
int ret;
if (!is_safe_to_compute_dst_reg_range(insn, &src_reg)) {
- __mark_reg_unknown(env, dst_reg);
+ bpf_mark_reg_unknown(dst_reg);
return 0;
}
@@ -19392,7 +19383,7 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
if (insn->src_reg == BPF_PSEUDO_MAP_VALUE ||
insn->src_reg == BPF_PSEUDO_MAP_IDX_VALUE) {
if (map->map_type == BPF_MAP_TYPE_ARENA) {
- __mark_reg_unknown(env, dst_reg);
+ bpf_mark_reg_unknown(dst_reg);
dst_reg->map_ptr = map;
bpf_diag_mod_end(env);
return 0;
@@ -20487,7 +20478,7 @@ static void null_to_arena(struct bpf_verifier_env *env, int regno)
reg = &cur_regs(env)[regno];
if (reg->type != SCALAR_VALUE || !tnum_equals_const(reg->var_off, 0))
return;
- __mark_reg_const_zero(env, reg);
+ __mark_reg_const_zero(reg);
reg->type = PTR_TO_ARENA;
}
@@ -20983,9 +20974,9 @@ static int push_budget_exit(struct bpf_verifier_env *env)
return PTR_ERR(branch);
branch->unwound = sub->bail_unwinds;
regs = branch->frame[branch->curframe]->regs;
- __mark_reg_const_zero(env, ®s[BPF_REG_0]);
+ __mark_reg_const_zero(®s[BPF_REG_0]);
if (bpf_ret_reg_pair(env, cur_func(env)->subprogno))
- __mark_reg_const_zero(env, ®s[BPF_REG_2]);
+ __mark_reg_const_zero(®s[BPF_REG_2]);
return 0;
}
@@ -21070,7 +21061,7 @@ static int check_loop_guard(struct bpf_verifier_env *env)
aux->guard_zero |= BIT(i);
for (i = BPF_REG_0; i < BPF_REG_FP; i++)
if (aux->guard_zero & BIT(i))
- __mark_reg_const_zero(env, ®s[i]);
+ __mark_reg_const_zero(®s[i]);
func = branch->frame[branch->curframe];
branch->insn_idx = aux->guard_to;
@@ -22721,12 +22712,12 @@ static void print_verification_stats(struct bpf_verifier_env *env)
env->subprog_info[i].stack_depth);
}
}
- /* user space parses this line. There is no limit. */
+ /* user space parses this line. There is no limit and there are no read marks. */
verbose(env, "processed %d insns (limit 0) max_states_per_insn %d "
- "total_states %d peak_states %d mark_read %d\n",
+ "total_states %d peak_states %d mark_read 0\n",
env->insn_processed,
env->max_states_per_insn, env->total_states,
- env->peak_states, env->longest_mark_read_walk);
+ env->peak_states);
}
int bpf_prog_ctx_arg_info_init(struct bpf_prog *prog,
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 68/81] bpf: Track a pointer to one of a set of functions
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (66 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 67/81] bpf: Remove leftovers of precision tracking and read marks Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 69/81] bpf, docs: Rewrite verifier.rst Alexei Starovoitov
` (14 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
'ops[i](x)' in C compiles to a load from a table of functions at a variable
offset and then callx. A call of a method of a trait object in Rust
compiles to the same code. At the load the verifier splits the state into N
states, one for each function that the load can return. The states differ
only in this register, but the verifier verifies the code from the load to
callx N times.
The states also count against the limit of 1024 states at one insn
(BPF_FLOW_MAX_STATES). In a prog of rust-bpf (Rust compiled to BPF) fmt()
of a trait object has 245 implementations. 3 states get to the load, and
each adds 244 states. The result is 732 states at one insn.
Do not split the state at the load. The register holds a pointer to one of
N functions until an insn must know the function.
before this patch
r3 = *(u64 *)(r3 + 8) r3 = *(u64 *)(r3 + 8)
/ | \ |
r3=f1 r3=f2 ... r3=fN r3 = one of f1 .. fN
| | | |
... ... ... ...
| | | |
callx r3 callx r3 callx r3 callx r3
| | | / | \
f1 f2 fN f1 f2 ... fN
callx verifies each function of the set. Thus the verifier verifies the
same calls as before. The verifier log shows the register in this way:
r3 = *(u64 *)(r3 +8) ; R3=func(one_of=245)
The register holds the number of a set (func_set), and equal sets have the
same number. Thus the code that compares states does not change.
insn that must know the function what the verifier does
-------------------------------- ---------------------------------------
callx, func_is_on_its_own() A copy of the state of the caller waits
for the exit states of the function.
callx, other functions Each function gets a state at callx.
call of a helper or a kfunc, Each function gets a state at the call.
the pointer is an argument A callback is one function.
func_is_on_its_own() is true for most functions. The verifier verifies such
a function for the states at its entry, and not again for each caller. The
copy that waits is not a state at an insn. Thus it does not count against
the limit.
Limitations:
- Only the load makes a set. Two pointers to functions that come from two
branches of a prog continue to be two states.
- The verifier does not know that two copies of the pointer are equal.
After 'r6 = r2; callx r2' a subsequent 'callx r6' verifies all N
functions. Code that is dead only when the two calls go to the same
function stays in the prog.
In the prog above the largest number of states at one insn changes from 732
to 159.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
include/linux/bpf_verifier.h | 15 +++-
kernel/bpf/log.c | 2 +
kernel/bpf/verifier.c | 168 +++++++++++++++++++++++++++++------
3 files changed, 158 insertions(+), 27 deletions(-)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 0cd7fa277fe5..3bd2965e531e 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -121,7 +121,11 @@ struct bpf_reg_state {
unsigned long raw2;
} raw;
- u32 subprogno; /* for PTR_TO_FUNC */
+ struct { /* for PTR_TO_FUNC */
+ u32 subprogno;
+ /* if not 0: one of env->func_sets[func_set - 1], subprogno is 0 */
+ u32 func_set;
+ };
};
/* For scalar types (SCALAR_VALUE), this represents our knowledge of
* the actual value.
@@ -1036,6 +1040,9 @@ struct bpf_verifier_env {
*/
struct bpf_func_ptr *func_ptrs;
u32 func_ptr_cnt;
+ /* What a pointer that was read from a table of functions may point to. */
+ struct bpf_func_set **func_sets;
+ u32 func_set_cnt;
bool has_callx;
/*
* Call graph edges created by callx instructions. A bitmap of
@@ -1402,6 +1409,12 @@ static inline bool bpf_register_is_null(struct bpf_reg_state *reg)
}
bool bpf_map_is_rdonly(const struct bpf_map *map);
+/* Sorted, no function is there twice */
+struct bpf_func_set {
+ u16 cnt;
+ u16 subprog[] __counted_by(cnt);
+};
+
struct bpf_func_ptr *bpf_map_func_ptrs(struct bpf_verifier_env *env,
const struct bpf_map *map, u32 *cnt);
struct bpf_func_ptr *bpf_map_range_func_ptrs(struct bpf_verifier_env *env,
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index 0013ab01e72d..3d26b0b7ed46 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -656,6 +656,8 @@ static void print_reg_state(struct bpf_verifier_env *env,
verbose_a("parent_id=%d", reg->parent_id);
if (type_is_non_owning_ref(reg->type))
verbose_a("%s", "non_own_ref");
+ if (t == PTR_TO_FUNC && reg->func_set)
+ verbose_a("one_of=%d", env->func_sets[reg->func_set - 1]->cnt);
if (type_is_map_ptr(t)) {
if (reg->map_ptr->name[0])
verbose_a("map=%s", reg->map_ptr->name);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 21ea43422133..6c6980d74fd6 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -6935,7 +6935,52 @@ static void mark_reg_func_ptr(struct bpf_verifier_env *env, struct bpf_reg_state
regs[regno].subprogno = subprog;
}
-/* a read from a table of functions branches into that many states at most */
+static int cmp_u16(const void *a, const void *b)
+{
+ return *(const u16 *)a - *(const u16 *)b;
+}
+
+/*
+ * Pointers to functions have a join: the set of the functions. The states would differ in
+ * nothing else up to the call, and there every function of the set is verified.
+ * Equal sets have the same number, so registers are compared as they are.
+ */
+static int mark_reg_func_set(struct bpf_verifier_env *env, struct bpf_reg_state *regs, int regno,
+ u16 *subprogs, u32 cnt)
+{
+ struct bpf_func_set **sets, *set;
+ u32 i, n = 0;
+
+ sort(subprogs, cnt, sizeof(*subprogs), cmp_u16, NULL);
+ for (i = 0; i < cnt; i++)
+ if (!n || subprogs[n - 1] != subprogs[i])
+ subprogs[n++] = subprogs[i];
+ mark_reg_func_ptr(env, regs, regno, n == 1 ? subprogs[0] : 0);
+ if (n == 1)
+ return 0;
+
+ for (i = 0; i < env->func_set_cnt; i++) {
+ set = env->func_sets[i];
+ if (set->cnt == n && !memcmp(set->subprog, subprogs, n * sizeof(*subprogs)))
+ break;
+ }
+ if (i == env->func_set_cnt) {
+ sets = krealloc_array(env->func_sets, i + 1, sizeof(*sets), GFP_KERNEL_ACCOUNT);
+ if (!sets)
+ return -ENOMEM;
+ env->func_sets = sets;
+ set = kzalloc_flex(*set, subprog, n, GFP_KERNEL_ACCOUNT);
+ if (!set)
+ return -ENOMEM;
+ set->cnt = n;
+ memcpy(set->subprog, subprogs, n * sizeof(*subprogs));
+ sets[env->func_set_cnt++] = set;
+ }
+ regs[regno].func_set = i + 1;
+ return 0;
+}
+
+/* a read from a table of functions yields one of that many at most */
#define BPF_MAX_FUNC_PTR_TARGETS BPF_MAX_SUBPROGS
/* and that many pointers at most are next to what is read at variable offset */
#define BPF_MAX_FUNC_PTR_RANGE 4096
@@ -6968,7 +7013,7 @@ static u32 count_read_offsets(struct tnum offs, u64 min_off, u64 max_off, u32 li
* is possible at, e.g. var_off is the stride of the elements of an array.
* A read of exactly one pointer yields PTR_TO_FUNC. When the offset is variable
* and only pointers can be read, which is how an element of a table of
- * functions is loaded, the verification continues with each of them. Other
+ * functions is loaded, it is one of them, see mark_reg_func_set(). Other
* reads that overlap with a pointer are rejected, because their result is not
* known until the program is jitted.
*
@@ -6981,7 +7026,6 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
struct tnum offs = tnum_add(reg->var_off, tnum_const(off));
struct bpf_reg_state *regs = cur_regs(env);
struct bpf_map *map = reg->map_ptr;
- struct bpf_verifier_state *branch;
u16 *targets __free(kfree) = NULL;
struct bpf_func_ptr *ptrs;
u32 i, cnt, n = 0;
@@ -7032,16 +7076,7 @@ static int check_func_ptr_read(struct bpf_verifier_env *env, struct bpf_reg_stat
if (err)
return err;
- for (i = 0; i < n - 1; i++) {
- branch = push_stack(env, env->insn_idx + 1, env->insn_idx,
- env->cur_state->speculative);
- if (IS_ERR(branch))
- return PTR_ERR(branch);
- mark_reg_func_ptr(env, branch->frame[branch->curframe]->regs, value_regno,
- targets[i]);
- }
- mark_reg_func_ptr(env, regs, value_regno, targets[n - 1]);
- return 0;
+ return mark_reg_func_set(env, regs, value_regno, targets, n);
overlap:
verbose(env, "read of %d bytes at offset [%llu,%llu] of map '%s' overlaps with a pointer to a function\n",
@@ -11757,13 +11792,92 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
* PTR_TO_FUNC register identifies the callee, so from here on callx is verified
* as a direct call of that static subprog.
*/
+static int check_func_callx_one(struct bpf_verifier_env *env, int subprog, int *insn_idx)
+{
+ int err;
+
+ err = btf_check_subprog_call(env, subprog);
+ if (err == -EFAULT)
+ return err;
+
+ err = record_callx_edge(env, cur_func(env)->subprogno, subprog);
+ if (err)
+ return err;
+
+ return check_static_func_call(env, subprog, insn_idx);
+}
+
+/* The insn needs to know the function: go on with each function of the set. */
+static int split_func_set(struct bpf_verifier_env *env, int regno)
+{
+ struct bpf_reg_state *reg = reg_state(env, regno);
+ struct bpf_verifier_state *branch;
+ struct bpf_func_set *set;
+ u32 i;
+
+ if (reg->type != PTR_TO_FUNC || !reg->func_set)
+ return 0;
+ set = env->func_sets[reg->func_set - 1];
+ reg->func_set = 0;
+ for (i = 0; i < set->cnt; i++) {
+ reg->subprogno = set->subprog[i];
+ if (i == set->cnt - 1)
+ break;
+ branch = push_stack(env, env->insn_idx, env->insn_idx, false);
+ if (IS_ERR(branch))
+ return PTR_ERR(branch);
+ }
+ return 0;
+}
+
+/*
+ * callx of one of the functions of a set. A function that is verified on its own
+ * gets a copy of the state that waits for it. Nothing meets at an insn.
+ */
+static int check_func_callx_set(struct bpf_verifier_env *env, int regno, int *insn_idx)
+{
+ struct bpf_verifier_state *cur = env->cur_state, *copy, *branch;
+ struct bpf_reg_state *reg = reg_state(env, regno);
+ struct bpf_func_set *set = env->func_sets[reg->func_set - 1];
+ int err = 0;
+ u32 i;
+
+ /* once, and not in every copy */
+ account_current_path(env);
+ reg->func_set = 0;
+ for (i = 0; i < set->cnt && !err; i++) {
+ reg->subprogno = set->subprog[i];
+ if (!func_is_on_its_own(env, cur, set->subprog[i])) {
+ branch = push_stack(env, *insn_idx, *insn_idx, false);
+ if (IS_ERR(branch))
+ return PTR_ERR(branch);
+ continue;
+ }
+ copy = kzalloc_obj(*copy, GFP_KERNEL_ACCOUNT);
+ if (!copy)
+ return -ENOMEM;
+ err = bpf_copy_verifier_state(copy, cur);
+ if (!err) {
+ env->cur_state = copy;
+ err = check_func_callx_one(env, set->subprog[i], insn_idx);
+ env->cur_state = cur;
+ }
+ bpf_free_verifier_state(copy, true);
+ if (err == PROCESS_BPF_EXIT)
+ err = 0;
+ else if (verifier_bug_if(!err, env, "callx: func#%d doesn't wait", set->subprog[i]))
+ err = -EFAULT;
+ }
+ /* all that is to be done has a state of its own */
+ return err ?: PROCESS_BPF_EXIT;
+}
+
static int check_func_callx(struct bpf_verifier_env *env, struct bpf_insn *insn,
int *insn_idx)
{
- struct bpf_func_state *caller = cur_func(env);
struct bpf_reg_state *reg;
const char *reason;
- int err, subprog;
+ int err;
err = check_reg_arg(env, insn->dst_reg, SRC_OP);
if (err)
@@ -11796,16 +11910,9 @@ static int check_func_callx(struct bpf_verifier_env *env, struct bpf_insn *insn,
return err;
/* PTR_TO_FUNC is a pointer to a static subprog */
- subprog = reg->subprogno;
- err = btf_check_subprog_call(env, subprog);
- if (err == -EFAULT)
- return err;
-
- err = record_callx_edge(env, caller->subprogno, subprog);
- if (err)
- return err;
-
- return check_static_func_call(env, subprog, insn_idx);
+ if (reg->func_set)
+ return check_func_callx_set(env, insn->dst_reg, insn_idx);
+ return check_func_callx_one(env, reg->subprogno, insn_idx);
}
int map_set_for_each_callback_args(struct bpf_verifier_env *env,
@@ -20484,7 +20591,7 @@ static void null_to_arena(struct bpf_verifier_env *env, int regno)
static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
{
- int err;
+ int err, i;
struct bpf_insn *insn = &env->prog->insnsi[env->insn_idx];
u8 class = BPF_CLASS(insn->code);
@@ -20566,6 +20673,12 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
return check_func_callx(env, insn, &env->insn_idx);
if (insn->src_reg == BPF_PSEUDO_CALL)
return check_func_call(env, insn, &env->insn_idx);
+ /* a callback is one function */
+ for (i = BPF_REG_1; i <= BPF_REG_5; i++) {
+ err = split_func_set(env, i);
+ if (err)
+ return err;
+ }
if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL)
return check_kfunc_call(env, insn, &env->insn_idx);
return check_helper_call(env, insn, &env->insn_idx);
@@ -24878,6 +24991,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr,
kvfree(env->gotox_tmp_buf);
kvfree(env->callx_edges);
kvfree(env->func_ptrs);
+ for (i = 0; i < env->func_set_cnt; i++)
+ kfree(env->func_sets[i]);
+ kfree(env->func_sets);
bpf_diag_free(env);
kvfree(env->idoms);
kvfree(env);
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 68/81] bpf: Track a pointer to one of a set of functions
2026-10-26 2:59 ` [PATCH bpf 68/81] bpf: Track a pointer to one of a set of functions Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 2568 bytes --]
> bpf: Track a pointer to one of a set of functions
TL;DR: Nit, not a bug: the explicit account_current_path() in
check_func_callx_set() seems redundant, since process_bpf_exit accounts the
path anyway; the comment's rationale doesn't match, so drop the call or
explain it.
> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index 21ea43422133..6c6980d74fd6 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c
[ ... ]
> @@ -11757,13 +11792,92 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn,
[ ... ]
> +/*
> + * callx of one of the functions of a set. A function that is verified on its own
> + * gets a copy of the state that waits for it. Nothing meets at an insn.
> + */
> +static int check_func_callx_set(struct bpf_verifier_env *env, int regno, int *insn_idx)
> +{
> + struct bpf_verifier_state *cur = env->cur_state, *copy, *branch;
> + struct bpf_reg_state *reg = reg_state(env, regno);
> + struct bpf_func_set *set = env->func_sets[reg->func_set - 1];
> + int err = 0;
> + u32 i;
> +
> + /* once, and not in every copy */
> + account_current_path(env);
> + reg->func_set = 0;
> + for (i = 0; i < set->cnt && !err; i++) {
[ ... ]
this isn't a bug, but is the explicit account_current_path() call needed in
check_func_callx_set()?
The copies made with bpf_copy_verifier_state() start with insns_subtotal at
0, since copy_func_state() sets dst->insns_subtotal = 0 (its comment says
"Instruction accounting is path-local, not part of verifier state."), so
the account_current_path() that runs on each copy adds nothing, and the
states from push_stack() do not carry the count either.
check_func_callx_set() then returns PROCESS_BPF_EXIT, and do_check_block()
goes to process_bpf_exit:, which calls account_current_path(env) on the same
cur state:
check_func_callx_set()
account_current_path(env);
...
return err ?: PROCESS_BPF_EXIT;
do_check_block()
process_bpf_exit:
account_current_path(env);
That call would account the path exactly once even without the one in
check_func_callx_set(), so the explicit call does not seem to change any
total, and the reason given in the comment ("once, and not in every copy")
does not match how the copies behave.
If the call stays, could the comment say why?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 69/81] bpf, docs: Rewrite verifier.rst
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (67 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 68/81] bpf: Track a pointer to one of a set of functions Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 70/81] selftests/bpf: Adjust tests of precision tracking Alexei Starovoitov
` (13 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
verifier.rst describes a verifier that follows each path of a prog and
prunes. The main pass of the verifier is now a data-flow analysis. It
verifies one basic block at a time, and it joins the states that meet at a
block. Most of the other text is from 2014-2018 and is also incorrect.
verifier.rst says the verifier
------------------------------------------ ---------------------------
It descends all paths and prunes. It joins states.
A DAG check disallows loops. It permits loops.
Signed and unsigned bounds are separate. One range holds the two.
A pointer has a fixed offset and a A pointer has one offset.
variable offset.
PTR_TO_MAP_VALUE_OR_NULL is a type. "Can be NULL" is a flag.
The log shows R0=inv1. The log has a new format.
Rewrite the file.
section contents
------------------------ -----------------------------------------------
The stages from the first checks to the JIT
What a state contains registers, ranges, ids, stack, held objects
The check of one insn the rule for each insn, with its message
Direct packet access the comparison with the end of the packet
Data-flow analysis the main part
Overview, Terms what a join is, one meaning for each term
The algorithm a flow chart: drop, join or add a state
Examples 1 to 4 what the verifier joins, what a join loses
Examples 5 to 8 loops, widening, the added may_goto
Examples 9 to 11 subprogs, tables of functions, recursion
The limits why the analysis ends
Each example shows the control flow graph with the states on its edges, or
the insns with the states adjacent to them. Two lines below most examples
compare the algorithm with a verifier that follows each path. The states,
the messages and the numbers in the examples are the output of the kernel.
The text does not use names of verifier functions. It shows a state in its
own notation, { r6=0 r7=1 }, and not in the format of the log. The names
and the format change frequently.
Remove the section "Understanding eBPF verifier messages". The table of the
checks has the messages adjacent to the rules.
bpf_design_QA.rst lists a limit on the insns that the verifier processes
and a limit on pending branches. These limits do not exist. kfuncs.rst
refers to pruning. Update the two files.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
Documentation/bpf/bpf_design_QA.rst | 7 +-
Documentation/bpf/kfuncs.rst | 2 +-
Documentation/bpf/verifier.rst | 1661 ++++++++++++++++++---------
3 files changed, 1144 insertions(+), 526 deletions(-)
diff --git a/Documentation/bpf/bpf_design_QA.rst b/Documentation/bpf/bpf_design_QA.rst
index 98d70b8a2533..f86b272920ae 100644
--- a/Documentation/bpf/bpf_design_QA.rst
+++ b/Documentation/bpf/bpf_design_QA.rst
@@ -92,11 +92,8 @@ Q: What are the verifier limits?
A: The only limit known to the user space is BPF_MAXINSNS (4096).
It's the maximum number of instructions that the unprivileged bpf
program can have. The verifier has various internal limits.
-Like the maximum number of instructions that can be explored during
-program analysis. Currently, that limit is set to 1 million.
-Which essentially means that the largest program can consist
-of 1 million NOP instructions. There is a limit to the maximum number
-of subsequent branches, a limit to the number of nested bpf-to-bpf
+The largest program can consist of 1 million instructions.
+There is a limit to the number of nested bpf-to-bpf
calls, a limit to the number of the verifier states per instruction,
a limit to the number of maps used by the program.
All these limits can be hit with a sufficiently complex program.
diff --git a/Documentation/bpf/kfuncs.rst b/Documentation/bpf/kfuncs.rst
index 47d3bed73b83..9fc272345138 100644
--- a/Documentation/bpf/kfuncs.rst
+++ b/Documentation/bpf/kfuncs.rst
@@ -155,7 +155,7 @@ An example is given below::
Here, bpf_obj_new uses local_type_id argument to find out the size of that type
ID in program's BTF and return a sized pointer to it. Each type ID will have a
distinct size, hence it is crucial to treat each such call as distinct when
-values don't match during verifier state pruning checks.
+values don't match when the verifier compares and joins states.
Hence, whenever a constant scalar argument is accepted by a kfunc which is not a
size parameter, and the value of the constant matters for program safety, __k
diff --git a/Documentation/bpf/verifier.rst b/Documentation/bpf/verifier.rst
index 510d15bc697b..cc1e6eae5471 100644
--- a/Documentation/bpf/verifier.rst
+++ b/Documentation/bpf/verifier.rst
@@ -3,558 +3,1179 @@
eBPF verifier
=============
-The safety of the eBPF program is determined in two steps.
+The verifier checks each program at load time. The kernel runs only a program that the verifier
+accepts. The verifier accepts a program only when it can show that no run breaks these rules:
-First step does DAG check to disallow loops and other CFG validation.
-In particular it will detect programs that have unreachable instructions.
-(though classic BPF checker allows them)
+- Each load and each store uses a valid pointer and stays in the object of the pointer.
+- Each helper and each kfunc gets the arguments that its prototype requires.
+- At exit the program holds no reference and no lock, and r0 is valid for the program type.
+- Each loop and each recursion ends: the verifier counts the iterations, or the kernel stops it.
-Second step starts from the first insn and descends all possible paths.
-It simulates execution of every insn and observes the state change of
-registers and stack.
+The stages
+==========
-At the start of the program the register R1 contains a pointer to context
-and has type PTR_TO_CTX.
-If verifier sees an insn that does R2=R1, then R2 has now type
-PTR_TO_CTX as well and can be used on the right hand side of expression.
-If R1=PTR_TO_CTX and insn is R2=R1+R1, then R2=SCALAR_VALUE,
-since addition of two valid pointers makes invalid pointer.
-(In 'secure' mode verifier will reject any type of pointer arithmetic to make
-sure that kernel addresses don't leak to unprivileged users)
-
-If register was never written to, it's not readable::
+::
- bpf_mov R0 = R2
- bpf_exit
+ 1. checks Each insn has a valid encoding. Each jump stays in its function. Each insn is
+ v reachable: unreachable insn 2
+ 2. analyses Constants, call graph, loops, live and tested registers, scalar evolution.
+ v A jump that constants decide becomes goto. Stage 3 does not see its dead branch.
+ 3. main pass It executes the program on states and checks each insn against them. See
+ v "Data-flow analysis". It verifies a global subprogram alone, from its prototype.
+ 4. rewrites Add may_goto and recursion checks (examples 8 and 11). Remove dead code. Convert
+ v context accesses. Guard division by 0. Inline some helpers.
+ 5. JIT It compiles each function to machine code.
-will be rejected, since R2 is unreadable at the start of the program.
+What a state contains
+=====================
-After kernel function call, R1-R5 are reset to unreadable and
-R0 has a return type of the function.
+::
-Since R6-R9 are callee saved, their state is preserved across the call.
+ part contents
+ ------------ ----------------------------------------------------------------------------------
+ frame r0-r10 and the stack slots of one function. A state has the frame of the current
+ function. In some cases it also has the frames of the callers. See example 9.
+ register Not set, a scalar, or a pointer to one kind of object: context, stack, map value,
+ packet, kernel object, arena and others. Some pointers can be NULL.
+ range The value of a scalar, or the offset of a pointer in its object. See below.
+ known bits The pair (value; mask) for the same number. A 1 in mask is an unknown bit. A load
+ of a byte gives (0x0; 0xff), |= 0x40 gives (0x40; 0xbf), += 1 gives (0x0; 0x1ff).
+ step The distance between the possible values: r0=[0,7], then r0 *= 12 gives step 12.
+ id Registers and stack slots with the same id change together at a jump or a release.
+ stack slot 8 bytes: a spilled register with its full description, or a mark for each byte
+ (not written, zero, unknown), or part of a dynptr or iterator, or saved irq flags.
+ held objects References, locks, saved irq flags, open rcu and preempt-disable sections.
+
+A range is one arc on the circle of all values. It can wrap around. Thus one range holds signed
+and unsigned bounds. ``[-5,10]`` goes from -5, the unsigned value 2^64 - 5, through 0 to 10. After
+``if r0 > 3`` the branch that is not taken has ``r0=[0,3]``. The other branch needs two arcs,
+``[-5,-1]`` and ``[4,10]``. Thus it keeps ``r0=[-5,10]``. A second range describes the low 32
+bits.
+
+The check of one insn
+=====================
::
- bpf_mov R6 = 1
- bpf_call foo
- bpf_mov R0 = R6
- bpf_exit
-
-is a correct program. If there was R1 instead of R6, it would have
-been rejected.
-
-load/store instructions are allowed only with registers of valid types, which
-are PTR_TO_CTX, PTR_TO_MAP, PTR_TO_STACK. They are bounds and alignment checked.
-For example::
-
- bpf_mov R1 = 1
- bpf_mov R2 = 2
- bpf_xadd *(u32 *)(R1 + 3) += R2
- bpf_exit
-
-will be rejected, since R1 doesn't have a valid pointer type at the time of
-execution of instruction bpf_xadd.
-
-At the start R1 type is PTR_TO_CTX (a pointer to generic ``struct bpf_context``)
-A callback is used to customize verifier to restrict eBPF program access to only
-certain fields within ctx structure with specified size and alignment.
-
-For example, the following insn::
-
- bpf_ld R0 = *(u32 *)(R6 + 8)
-
-intends to load a word from address R6 + 8 and store it into R0
-If R6=PTR_TO_CTX, via is_valid_access() callback the verifier will know
-that offset 8 of size 4 bytes can be accessed for reading, otherwise
-the verifier will reject the program.
-If R6=PTR_TO_STACK, then access should be aligned and be within
-stack bounds, which are [-MAX_BPF_STACK, 0). In this example offset is 8,
-so it will fail verification, since it's out of bounds.
-
-The verifier will allow eBPF program to read data from stack only after
-it wrote into it.
-
-Classic BPF verifier does similar check with M[0-15] memory slots.
-For example::
-
- bpf_ld R0 = *(u32 *)(R10 - 4)
- bpf_exit
-
-is invalid program.
-Though R10 is correct read-only register and has type PTR_TO_STACK
-and R10 - 4 is within stack bounds, there were no stores into that location.
-
-Pointer register spill/fill is tracked as well, since four (R6-R9)
-callee saved registers may not be enough for some programs.
-
-Allowed function calls are customized with bpf_verifier_ops->get_func_proto()
-The eBPF verifier will check that registers match argument constraints.
-After the call register R0 will be set to return type of the function.
-
-Function calls is a main mechanism to extend functionality of eBPF programs.
-Socket filters may let programs to call one set of functions, whereas tracing
-filters may allow completely different set.
-
-If a function made accessible to eBPF program, it needs to be thought through
-from safety point of view. The verifier will guarantee that the function is
-called with valid arguments.
-
-seccomp vs socket filters have different security restrictions for classic BPF.
-Seccomp solves this by two stage verifier: classic BPF verifier is followed
-by seccomp verifier. In case of eBPF one configurable verifier is shared for
-all use cases.
-
-See details of eBPF verifier in kernel/bpf/verifier.c
-
-Register value tracking
-=======================
-
-In order to determine the safety of an eBPF program, the verifier must track
-the range of possible values in each register and also in each stack slot.
-This is done with ``struct bpf_reg_state``, defined in include/linux/
-bpf_verifier.h, which unifies tracking of scalar and pointer values. Each
-register state has a type, which is either NOT_INIT (the register has not been
-written to), SCALAR_VALUE (some value which is not usable as a pointer), or a
-pointer type. The types of pointers describe their base, as follows:
-
-
- PTR_TO_CTX
- Pointer to bpf_context.
- CONST_PTR_TO_MAP
- Pointer to struct bpf_map. "Const" because arithmetic
- on these pointers is forbidden.
- PTR_TO_MAP_VALUE
- Pointer to the value stored in a map element.
- PTR_TO_MAP_VALUE_OR_NULL
- Either a pointer to a map value, or NULL; map accesses
- (see maps.rst) return this type, which becomes a
- PTR_TO_MAP_VALUE when checked != NULL. Arithmetic on
- these pointers is forbidden.
- PTR_TO_STACK
- Frame pointer.
- PTR_TO_PACKET
- skb->data.
- PTR_TO_PACKET_END
- skb->data + headlen; arithmetic forbidden.
- PTR_TO_SOCKET
- Pointer to struct bpf_sock_ops, implicitly refcounted.
- PTR_TO_SOCKET_OR_NULL
- Either a pointer to a socket, or NULL; socket lookup
- returns this type, which becomes a PTR_TO_SOCKET when
- checked != NULL. PTR_TO_SOCKET is reference-counted,
- so programs must release the reference through the
- socket release function before the end of the program.
- Arithmetic on these pointers is forbidden.
-
-However, a pointer may be offset from this base (as a result of pointer
-arithmetic), and this is tracked in two parts: the 'fixed offset' and 'variable
-offset'. The former is used when an exactly-known value (e.g. an immediate
-operand) is added to a pointer, while the latter is used for values which are
-not exactly known. The variable offset is also used in SCALAR_VALUEs, to track
-the range of possible values in the register.
-
-The verifier's knowledge about the variable offset consists of:
-
-* minimum and maximum values as unsigned
-* minimum and maximum values as signed
-
-* knowledge of the values of individual bits, in the form of a 'tnum': a u64
- 'mask' and a u64 'value'. 1s in the mask represent bits whose value is unknown;
- 1s in the value represent bits known to be 1. Bits known to be 0 have 0 in both
- mask and value; no bit should ever be 1 in both. For example, if a byte is read
- into a register from memory, the register's top 56 bits are known zero, while
- the low 8 are unknown - which is represented as the tnum (0x0; 0xff). If we
- then OR this with 0x40, we get (0x40; 0xbf), then if we add 1 we get (0x0;
- 0x1ff), because of potential carries.
-
-Besides arithmetic, the register state can also be updated by conditional
-branches. For instance, if a SCALAR_VALUE is compared > 8, in the 'true' branch
-it will have a umin_value (unsigned minimum value) of 9, whereas in the 'false'
-branch it will have a umax_value of 8. A signed compare (with BPF_JSGT or
-BPF_JSGE) would instead update the signed minimum/maximum values. Information
-from the signed and unsigned bounds can be combined; for instance if a value is
-first tested < 8 and then tested s> 4, the verifier will conclude that the value
-is also > 4 and s< 8, since the bounds prevent crossing the sign boundary.
-
-PTR_TO_PACKETs with a variable offset part have an 'id', which is common to all
-pointers sharing that same variable offset. This is important for packet range
-checks: after adding a variable to a packet pointer register A, if you then copy
-it to another register B and then add a constant 4 to A, both registers will
-share the same 'id' but the A will have a fixed offset of +4. Then if A is
-bounds-checked and found to be less than a PTR_TO_PACKET_END, the register B is
-now known to have a safe range of at least 4 bytes. See 'Direct packet access',
-below, for more on PTR_TO_PACKET ranges.
-
-The 'id' field is also used on PTR_TO_MAP_VALUE_OR_NULL, common to all copies of
-the pointer returned from a map lookup. This means that when one copy is
-checked and found to be non-NULL, all copies can become PTR_TO_MAP_VALUEs.
-As well as range-checking, the tracked information is also used for enforcing
-alignment of pointer accesses. For instance, on most systems the packet pointer
-is 2 bytes after a 4-byte alignment. If a program adds 14 bytes to that to jump
-over the Ethernet header, then reads IHL and adds (IHL * 4), the resulting
-pointer will have a variable offset known to be 4n+2 for some n, so adding the 2
-bytes (NET_IP_ALIGN) gives a 4-byte alignment and so word-sized accesses through
-that pointer are safe.
-The 'id' field is also used on PTR_TO_SOCKET and PTR_TO_SOCKET_OR_NULL, common
-to all copies of the pointer returned from a socket lookup. This has similar
-behaviour to the handling for PTR_TO_MAP_VALUE_OR_NULL->PTR_TO_MAP_VALUE, but
-it also handles reference tracking for the pointer. PTR_TO_SOCKET implicitly
-represents a reference to the corresponding ``struct sock``. To ensure that the
-reference is not leaked, it is imperative to NULL-check the reference and in
-the non-NULL case, and pass the valid reference to the socket release function.
+ insn rule, and the message for a program that breaks it
+ ----------------- -----------------------------------------------------------------------------
+ each insn It reads only registers that are set: R2 !read_ok
+ It does not write r10: frame pointer is read only
+ call The program type permits the helper or kfunc. r1-r5 agree with its prototype.
+ Then r1-r5 are not set, r0 has the result, and r6-r9 do not change.
+ pointer, scalar 64-bit += and -= change the offset. The scalar must have a lower bound. Other
+ ops (*=, &=, <<=) give a scalar. No op is permitted on a pointer that can be
+ NULL, a socket, the end of a packet. No -= on a pointer to the stack.
+ pointer, pointer -= gives a scalar. Other ops: R2 pointer += pointer prohibited
+ pointer to arena Each op is permitted and gives a pointer to arena.
+ conditional jump Each branch gets its ranges: if r0 > 8 gives r0=[9,U64_MAX] and r0=[0,8].
+ Copies of r0 have its id and get the same ranges. A NULL check changes all
+ copies of the pointer: not NULL in one branch, the scalar 0 in the other.
+ acquire, release The acquired pointer has the id of a reference. A release makes all pointers
+ with this id scalars. At exit: Unreleased reference id=2 alloc_insn=12
+ load, store The address is a pointer that cannot be NULL: R1 invalid mem access 'scalar'
+ context A callback of the program type checks the offset. It gives the result type.
+ stack In the frame, and aligned. The load of a spilled register gives it back.
+ map value Smallest offset >= 0, and largest offset + size <= size of the value.
+ packet See "Direct packet access".
+ kernel object BTF gives the type of the member at the offset. A load that faults gives 0.
+ A callback of the program type must permit a store: only read is supported
+ arena No check. The JIT keeps each access in the arena. A load that faults gives 0.
Direct packet access
====================
-In cls_bpf and act_bpf programs the verifier allows direct access to the packet
-data via skb->data and skb->data_end pointers.
-Ex::
-
- 1: r4 = *(u32 *)(r1 +80) /* load skb->data_end */
- 2: r3 = *(u32 *)(r1 +76) /* load skb->data */
- 3: r5 = r3
- 4: r5 += 14
- 5: if r5 > r4 goto pc+16
- R1=ctx R3=pkt(id=0,off=0,r=14) R4=pkt_end R5=pkt(id=0,off=14,r=14) R10=fp
- 6: r0 = *(u16 *)(r3 +12) /* access 12 and 13 bytes of the packet */
-
-this 2byte load from the packet is safe to do, since the program author
-did check ``if (skb->data + 14 > skb->data_end) goto err`` at insn #5 which
-means that in the fall-through case the register R3 (which points to skb->data)
-has at least 14 directly accessible bytes. The verifier marks it
-as R3=pkt(id=0,off=0,r=14).
-id=0 means that no additional variables were added to the register.
-off=0 means that no additional constants were added.
-r=14 is the range of safe access which means that bytes [R3, R3 + 14) are ok.
-Note that R5 is marked as R5=pkt(id=0,off=14,r=14). It also points
-to the packet data, but constant 14 was added to the register, so
-it now points to ``skb->data + 14`` and accessible range is [R5, R5 + 14 - 14)
-which is zero bytes.
-
-More complex packet access may look like::
-
-
- R0=inv1 R1=ctx R3=pkt(id=0,off=0,r=14) R4=pkt_end R5=pkt(id=0,off=14,r=14) R10=fp
- 6: r0 = *(u8 *)(r3 +7) /* load 7th byte from the packet */
- 7: r4 = *(u8 *)(r3 +12)
- 8: r4 *= 14
- 9: r3 = *(u32 *)(r1 +76) /* load skb->data */
- 10: r3 += r4
- 11: r2 = r1
- 12: r2 <<= 48
- 13: r2 >>= 48
- 14: r3 += r2
- 15: r2 = r3
- 16: r2 += 8
- 17: r1 = *(u32 *)(r1 +80) /* load skb->data_end */
- 18: if r2 > r1 goto pc+2
- R0=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R1=pkt_end R2=pkt(id=2,off=8,r=8) R3=pkt(id=2,off=0,r=8) R4=inv(id=0,umax_value=3570,var_off=(0x0; 0xfffe)) R5=pkt(id=0,off=14,r=14) R10=fp
- 19: r1 = *(u8 *)(r3 +4)
-
-The state of the register R3 is R3=pkt(id=2,off=0,r=8)
-id=2 means that two ``r3 += rX`` instructions were seen, so r3 points to some
-offset within a packet and since the program author did
-``if (r3 + 8 > r1) goto err`` at insn #18, the safe range is [R3, R3 + 8).
-The verifier only allows 'add'/'sub' operations on packet registers. Any other
-operation will set the register state to 'SCALAR_VALUE' and it won't be
-available for direct packet access.
-
-Operation ``r3 += rX`` may overflow and become less than original skb->data,
-therefore the verifier has to prevent that. So when it sees ``r3 += rX``
-instruction and rX is more than 16-bit value, any subsequent bounds-check of r3
-against skb->data_end will not give us 'range' information, so attempts to read
-through the pointer will give "invalid access to packet" error.
-
-Ex. after insn ``r4 = *(u8 *)(r3 +12)`` (insn #7 above) the state of r4 is
-R4=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) which means that upper 56 bits
-of the register are guaranteed to be zero, and nothing is known about the lower
-8 bits. After insn ``r4 *= 14`` the state becomes
-R4=inv(id=0,umax_value=3570,var_off=(0x0; 0xfffe)), since multiplying an 8-bit
-value by constant 14 will keep upper 52 bits as zero, also the least significant
-bit will be zero as 14 is even. Similarly ``r2 >>= 48`` will make
-R2=inv(id=0,umax_value=65535,var_off=(0x0; 0xffff)), since the shift is not sign
-extending. This logic is implemented in adjust_reg_min_max_vals() function,
-which calls adjust_ptr_min_max_vals() for adding pointer to scalar (or vice
-versa) and adjust_scalar_min_max_vals() for operations on two scalars.
-
-The end result is that bpf program author can access packet directly
-using normal C code as::
+Programs for tc, XDP and other network hooks access packet data through pointers. The verifier does
+not know the length of a packet. Thus the program compares a pointer with the end of the packet::
void *data = (void *)(long)skb->data;
void *data_end = (void *)(long)skb->data_end;
- struct eth_hdr *eth = data;
- struct iphdr *iph = data + sizeof(*eth);
- struct udphdr *udp = data + sizeof(*eth) + sizeof(*iph);
+ if (data + 14 > data_end) { data=packet, 0 bytes are safe }
+ return 0;
+ return *(__u16 *)(data + 12); { data=packet, 14 bytes are safe }
+
+- The largest offset of the compared pointer gives the number of safe bytes.
+- All pointers to the packet with the same id get this number: copies, and copies plus a constant.
+- An access has the check of a map value, with this number as the size of the value. A load of 2
+ bytes at ``data + 13``: ``invalid access to packet, off=13 size=2, R1(id=0,off=13,r=14)``
+- ``pointer += variable`` makes a new id. The pointer keeps the number only if the variable is >= 0
+ and the largest offset stays below it. If not, the program must compare the new pointer.
+- After a helper that can change the packet (``bpf_skb_pull_data()``) all pointers to it are
+ scalars.
+
+Data-flow analysis
+==================
+
+Overview
+--------
+
+The verifier must show that no run of a program does an unsafe operation. It executes the program
+on states. A state represents all runs that arrive at an insn with the same types in registers
+and stack.
+The sections above describe the contents of a state and the check of one insn against a state.
+This section describes how states move through a program.
+
+The main pass is a forward data-flow analysis over the control flow graph. Where two paths meet,
+the verifier makes one state from the two states. This operation is a join::
+
+ follow each path join
+
+ [if] [if]
+ / \ / \
+ r1=1 r1=5 r1=1 r1=5
+ \ / \ /
+ [next] [next]
+ verified 2 times: verified 1 time:
+ r1=1, then r1=5 r1=[1,5]
+
+The work depends on the size of the program. It does not depend on the number of paths.
+
+Three properties are different from a textbook analysis:
+
+- A basic block has a set of states, not one state. The verifier joins two states only when the
+ join keeps the facts that programs use. See examples 1 to 4.
+- Scalar evolution describes a loop before the verifier iterates the loop. See examples 5 to 7.
+- The verifier does not prove termination. The kernel stops a loop or a recursion at run time.
+ See examples 8 and 11.
+
+Examples 9 to 11 show subprograms. The limits are at the end.
+
+How to read the examples:
+
+- The kernel loaded each program below, and ran it where the text gives a return value. The
+ states, the messages and the numbers are the output of the kernel.
+- ``r7=[0,7]`` is a scalar in this range. ``{ r6=0 r7=1 }`` is a state. The examples omit the
+ registers that are not important.
+- The lines ``paths:`` and ``joins:`` below an example compare two algorithms.
+
+The two algorithms::
+
+ paths: processed 9 insns a verifier that follows each path
+ joins: processed 7 insns the algorithm of this section
+
+A verifier that follows each path goes from the entry to exit, depth first. It stops a path when
+a state that it saw before at the same insn covers the current state. Each branch can double the
+number of paths. Thus such a verifier needs a limit on the number of insns that it processes. The
+limit in the examples is 1 million.
+
+Terms
+-----
+
+The text uses each term with one meaning only::
+
+ term meaning
+ ---------------- ---------------------------------------------------------------------------
+ state It represents the runs that arrive at one insn with the same types. It
+ contains the type of each live register and stack slot, the range and the
+ known bits of each scalar and of each pointer offset, and the references and
+ locks that the program holds.
+ live, dead A register is live at an insn if this insn or a later insn can read its
+ value. If not, it is dead. The same applies to a stack slot.
+ function The main program or a subprogram.
+ block A basic block of the control flow graph.
+ set(B) The set of states of block B.
+ new state A state in a set that the verifier did not use yet.
+ t covers s State t describes each run that state s describes. Thus s cannot arrive at
+ an insn at which t does not arrive.
+ join(s, t) A state that covers s and covers t.
+ no join No useful state covers s and t. The verifier keeps the two states.
+ tested register A register or a stack slot whose value arrives, without a change, at a later
+ jump that compares it with a constant.
+ gap The values between two ranges: -1 and 0 are the gap between -2 and 1.
+ head The first insn of a loop.
+ latch The jump from the end of a loop to the head.
+ counted loop A loop for which scalar evolution calculates the number of iterations.
+ unrolled loop A loop that the verifier verifies iteration by iteration, with the real
+ values. Each iteration has its own state.
+ widening A join at a head that removes the bound that grows.
+ added may_goto A may_goto insn that the verifier adds to a loop.
+ time slice 250 ms. All loops that have may_goto share it in one run of a program.
+ entry state The state at the first insn of a subprogram. It contains the arguments and
+ nothing from the frame of the caller.
+ exit state The state at an exit of a subprogram.
+ landing pad The target of unwind_goto: the cleanup code of a caller.
+ room Less than half of each limit on the number of states is in use.
+
+The algorithm
+-------------
+
+Each block has a set of states. At the start all sets are empty, and the entry state is a new
+state at insn 0.
+
+::
+
+ 1. Take the first block, in the order of the program, that has a new state.
+ 2. Verify the insns of the block with that state.
+ 3. Send the result to each successor of the block.
+ 4. In the successor:
+
+ state S arrives at block B
+ |
+ v
+ +---------------------------+ yes
+ | a state of set(B) |------> drop S
+ | covers S ? |
+ +---------------------------+
+ | no
+ v
+ +---------------------------+ yes
+ | S and a state T of set(B) |------> replace T with join(S, T)
+ | have a join ? | the join is a new state
+ +---------------------------+
+ | no
+ v
+ add S to set(B)
+ S is a new state
+
+ 5. Stop when no block has a new state.
+
+The verifier accepts the program when it stops and no insn failed.
+
+The order of the program is reverse post order, with the insns of a loop kept together:
+
+- The verifier verifies a block after the predecessors of the block. Thus the states from all
+ predecessors meet before the verifier uses them.
+- The verifier completes a loop before it verifies the code after the loop.
+
+A state contains only live registers and live stack slots. Two states that differ only in a dead
+register are equal.
+
+**Live registers and stack slots.** A register is live at an insn if this insn or a later insn can
+read its value. If not, it is dead. The same applies to a stack slot.
+
+::
+
+ insn live before the insn states that arrive
+ 0: r0 = random() -
+ 1: r1 = 0 r0
+ 2: if r0 == 0 goto 4 r0 r1
+ 3: r0 = 1 r1 { r1=0 }
+ 4: r0 = r1 r1 { r1=0 } from 2, { r1=0 } from 3
+ 5: exit r0
+
+A run that arrives at 4 from 2 has 0 in r0. A run that arrives from 3 has 1 in r0. Insn 4 writes r0
+and does not read it, thus r0 is dead at 4. The two states are equal. The verifier drops the second
+state and verifies block 4 one time.
+
+A dead register also cannot prevent a join. See r6 at insn 14 of example 2.
+
+The verifier calculates the live registers and stack slots one time, before the main pass:
+
+- The calculation goes backward through the control flow graph until nothing changes.
+- A call of a static subprogram reads those of r1-r5 that are live at the entry of the subprogram.
+- The calculation follows pointers to the stack through copies, spills and arguments. Thus it knows
+ which stack slots of a caller a subprogram reads and writes.
+
+1. Two paths meet: join
+-----------------------
+
+::
+
+ +--------------------------+
+ | 0: r0 = random() |
+ | 1: r7 = r0 |
+ | 2: r7 &= 7 |
+ | 3: if r0 > 100 goto 5 |
+ +--------------------------+
+ r7=[0,7] | | r7=[0,7]
+ v |
+ +--------------+ |
+ | 4: r7 += 16 | |
+ +--------------+ |
+ r7=[16,23] | |
+ v v
+ +--------------------------+
+ | 5: r0 = r7 | join: r7=[0,23]
+ | 6: exit |
+ +--------------------------+
+
+Block 4 is before block 5 in the order. Thus the state from insn 3 waits in set(5).
+
+::
+
+ event set(5)
+ ------------------------------ ------------------------------
+ r7=[0,7] arrives from 3 { r7=[0,7] } new state
+ the verifier verifies block 4 { r7=[0,7] } new state
+ r7=[16,23] arrives from 4 { r7=[0,23] } the join, new state
+ the verifier verifies block 5 { r7=[0,23] }
+
+The verifier verifies block 5 one time.
+
+::
+
+ paths: processed 9 insns 5 and 6 are verified 2 times
+ joins: processed 7 insns
+
+The next program has 20 such ifs in sequence. Each ``if`` sets a different bit of r7. At the end
+``r7 & 63`` is an index. The program has 75 insns and 2^20 paths. r7 is different in each path,
+and the value of r7 is important.
+
+::
+
+ if (a) r7 |= 1; 2 paths
+ if (b) r7 |= 2; 4 paths
+ ...
+ if (t) r7 |= 1 << 19; 2^20 paths
+ x = arr[r7 & 63];
+
+ paths: BPF program is too large. Processed 1000001 insn
+ joins: processed 74 insns
+
+2. No join: why a block has a set of states
+-------------------------------------------
+
+::
+
+ p = lookup(&map, &key);
+ found = p != NULL;
+ ...
+ if (found)
+ x = *p;
+
+ 6: r0 = lookup(hash map) pointer or NULL
+ 7: r6 = r0
+ 8: r7 = 0
+ 9: if r6 == 0 goto 11
+ 10: r7 = 1
+ 11: r0 = 0 { r6=0 r7=0 } { r6=map_value r7=1 }
+ 12: if r7 == 0 goto 14 taken not taken
+ 13: r0 = *(u8 *)(r6 + 0) { r6=map_value }
+ 14: exit { r0=[0,255] }
+
+Two states arrive at 11. A scalar and a pointer have no join, thus the verifier keeps the two
+states.
+
+::
+
+ 9: if r6 == 0
+ / \
+ { r6=0 r7=0 } { r6=map_value r7=1 }
+ \ /
+ set(11) = { r6=0 r7=0 }, { r6=map_value r7=1 } 2 states, no join
+ | |
+ 12: taken 12: not taken
+ | |
+ | 13: the load uses a pointer
+ \ /
+ set(14) = { r0=[0,255] } r6 and r7 are dead: 1 state
+
+One state for each block is not sufficient here. In one state r6 is "a pointer or a scalar". The
+program can do nothing with such a register. The state also loses the fact that r7 shows which of
+the two r6 is.
+
+::
+
+ paths: processed 17 insns
+ joins: processed 16 insns
+
+The rule: **the verifier joins scalars. States that differ in other properties have no join.**
+
+::
+
+ in state s in state t join
+ ------------------------- -------------------------- -------------
+ scalar 1 scalar 5 scalar [1,5]
+ scalar pointer no join
+ pointer to map A pointer to map B no join
+ pointer to function f pointer to function g no join
+ a reference no reference no join
+ a lock no lock no join
+ iterator is active iterator is drained no join
+ called from insn 2 called from insn 7 no join
+
+This rule keeps the facts that programs use: NULL checks, and error paths that release an object.
+
+**A register that only one state sets.**
+
+::
+
+ ret = get(&p); inlined: get() writes p when ret is 0
+ if (ret < 0)
+ goto drop;
+ return p->x;
+
+This pattern is frequent in C. On the error path p is not NULL. It has no value at all, because
+the compiler has no reason to write r8 there.
+
+::
+
+ 6: r0 = lookup(hash map)
+ 7: r6 = -2 ret = -ENOENT
+ 8: if r0 == 0 goto 11
+ 9: r8 = r0 p
+ 10: r6 = 0 ret = 0
+ 11: w7 = 0 { r6=-2 } { r6=0 r8=map_value }
+ 12: if r6 s< 0 goto 15 taken not taken
+ 13: r0 = *(u32 *)(r8 + 0) { r8=map_value }
+ 14: exit
+ 15: r0 = 0 { }
+ 16: exit
+
+These two states have a join. In a textbook "not set" joins with each value, and the result is
+"not set"::
+
+ { r6=-2 } { r6=0 r8=map_value }
+ \ /
+ join: { r6=[-2,0] } r8 is not set
+ |
+ 12: if r6 s< 0 goto 15 the two directions are possible
+ |
+ 13: r0 = *(u32 *)(r8 + 0) no pointer in r8: the verifier rejects the load
+
+This join is not useful:
+
+- r8 is in the second state because r8 is live. A later insn reads it.
+- The first state does not arrive at that insn. A jump sends it in a different direction.
+- The join cannot select the direction. It arrives at the read without r8.
+
+Thus the verifier keeps the two states. The method that the program uses to separate them is not
+important: ``if (ret < 0)``, ``if (flags & 1)``, or a test 300 insns later.
+
+::
+
+ paths: processed 18 insns
+ joins: processed 18 insns
+
+3. Scalars that the verifier does not join
+------------------------------------------
+
+Two scalars always have a join. In some programs the join is not correct for the program.
+
+::
+
+ p = NULL;
+ ret = find(key, &p); -ENOENT, 1: nothing to do, 0: p is the entry
+ if (ret < 0)
+ goto err;
+ if (ret > 0)
+ goto skip;
+ return p->x;
+
+ 0: r0 = random() key
+ 1: r8 = 0 p = NULL
+ 2: r6 = 1 ret = 1
+ 3: if r0 == 0 goto 14
+ ...
+ 9: r0 = lookup(hash map)
+ 10: r6 = -2 ret = -ENOENT
+ 11: if r0 == 0 goto 14
+ 12: r8 = r0 p
+ 13: r6 = 0 ret = 0
+ 14: if r6 s< 0 goto 18 { r6=1 r8=0 } { r6=-2 r8=0 } { r6=0 r8=map_value }
+ 15: if r6 s> 0 goto 18 { r6=1 r8=0 } { r6=0 r8=map_value }
+ 16: r0 = *(u32 *)(r8 + 0) { r8=map_value }
+ 17: exit
+ 18: r0 = 0 { }
+ 19: exit
+
+The third state has no join with the other two: r8 is a pointer. The first two states differ only
+in a scalar. This is the result if the verifier joins them::
+
+ { r6=1 r8=0 } { r6=-2 r8=0 }
+ \ /
+ join: { r6=[-2,1] r8=0 } the join contains ret = 0
+ |
+ 14: if r6 s< 0 | not taken for 0
+ 15: if r6 s> 0 | not taken for 0
+ v
+ 16: r0 = *(u32 *)(r8 + 0) R8 invalid mem access 'scalar'
+
+The rule: **a join must not contain a value that no input has, if a later jump tests this value.**
+The rule has two parts.
+
+**Part 1: the tested registers.** The verifier calculates them one time, before the analysis. The
+calculation goes backward until nothing changes, like the calculation of live registers.
+
+::
+
+ insn effect
+ ----------------------------- ------------------------------------------------------------
+ if rX < constant rX is tested
+ rX = constant, rX += ... the insn writes rX: rX is not tested before the insn
+ rX = rY if rX is tested after the insn, rY is tested before it
+ *(u64 *)(r10 - 8) = rX if the slot is tested after the insn, rX is tested before it
+ rX = *(u64 *)(r10 - 8) if rX is tested after the insn, the slot is tested before it
+ call helper the call writes r0-r5
+ call static subprogram the registers of r1-r5 that the subprogram tests
+
+Only a value without a change counts: a copy, a spill, a fill. Arithmetic stops the calculation.
+The result for the example::
+
+ 9: r0 = lookup(hash map) -
+ 10: r6 = -2 r0
+ 11: if r0 == 0 goto 14 r0 r6
+ 12: r8 = r0 -
+ 13: r6 = 0 -
+ 14: if r6 s< 0 goto 18 r6
+ 15: if r6 s> 0 goto 18 r6
+ 16: r0 = *(u32 *)(r8 + 0) -
+
+The distance to the jump and the number of jumps are not important.
+
+**Part 2: the decision.** States a and b have a join. The verifier examines the tested registers
+of the block.
+
+::
+
+ the tested registers in a and in b decision
+ ---------------------------------------------------------- --------------------
+ one register: its two ranges have a gap keep the two states
+ two registers: each has two ranges with no common value keep the two states
+ all other cases join
+
+A gap on the number line::
+
+ -2 -1 0 1
+ a . . b the join [-2,1] contains -1 and 0. No input has them.
+
+ 0 1
+ a b the join [0,1] contains only the inputs.
+
+ a gap: -2 and 1 0 and -155 [0,7] and [16,23]
+ no gap: 0 and 1 -1 and 0 -134 and -135 [0,7] and [8,15] [0,7] and [4,9]
+
+**After the decision** no more rules are necessary:
+
+- The two states are new states in the set of the block. The verifier uses each of them.
+- Each jump has one known direction for each state.
+- After the last jump that tests r6, r6 is dead. A dead register is not part of a state.
+- From there the usual steps apply to each state that arrives at a block: drop, join or add.
+
+The result::
+
+ the paths meet the last test of r6
+ | |
+ v v
+ --------+==========================+-----------------> insns
+ 2 states r6 is dead: the states can join
+
+ paths: processed 24 insns
+ joins: processed 22 insns
+
+**Two registers.** This pattern is from a real program::
+
+ ret = 0;
+ state = lookup(&map, &key);
+ if (!state) {
+ err = update(&map, &key, &new);
+ ret = err < 0 ? DROP : 0;
+ ...
+ if (err < 0)
+ goto out;
+ state = lookup(&map, &key);
+ if (!state)
+ return 0;
+ }
+ out:
+ if (ret < 0)
+ goto drop;
+ return state->x;
+
+ 25: r0 = update(hash map) r8=0: state is NULL
+ 26: w9 = 0 ret = 0
+ 27: if w0 s> -1 goto 29
+ 28: w9 = -155 ret = DROP
+ 29: w6 = 0 { w0>=0 w9=0 r8=0 } { w0<0 w9=-155 r8=0 }
+ 30: if w0 s< 0 goto 38 not taken taken
+ ...
+ 35: r0 = lookup(hash map)
+ 36: r8 = r0
+ 37: if r8 == 0 goto 41
+ 38: if w9 s< 0 goto 41 { w9=0 r8=map_value } { w9=-155 r8=0 }
+ 39: r0 = *(u32 *)(r8 + 0) { r8=map_value }
+ 40: exit
+ 41: r0 = 0
+ 42: exit
+
+A join at 29 contains err < 0 together with ret = 0. This state goes from 30 to 38 with NULL in
+r8. The jump at 38 is not taken. At 39 the result is ``R8 invalid mem access 'scalar'`` again.
+
+::
+
+ paths: processed 46 insns
+ joins: processed 43 insns
+
+0 and -155 have a gap, thus the first line of the table is sufficient here. Now change err and ret
+to flags, 0 or 1. No register has a gap. The second line of the table applies::
+
+ ret=0 ret=1
+ err=0 a x a, b: the inputs
+ err=1 x b x: pairs that only the join contains
+
+ paths: processed 53 insns
+ joins: processed 46 insns
+
+More cases::
+
+ the states that meet the later jumps decision
+ ------------------------------- --------------------------- ------------------------
+ ret=-2 and ret=1 ret < 0, then ret > 0 a gap: keep
+ ret=-134 and ret=-135 ret < 0 no gap: join
+ ret=-134 and ret=-160 ret < 0, then ret == -158 a gap: keep
+ (a>=0 b=0) and (a<0 b=-155) a < 0, then b < 0 a gap: keep
+ (a=0 b=0) and (a=1 b=1) a != 0, then b != 0 two registers: keep
+ i=0 and i=1 i < 64 no gap: join
+ the 20 ifs of example 1 no jump tests r7 join
+
+The rule only refuses a join. Thus it cannot make the verifier accept an unsafe program. The rule
+does not apply at a head, where the verifier uses widening (example 7). It also does not apply
+when there is no room, see "Why the analysis ends, and the limits".
+
+The programs of this example, and ``ret = get(&p)`` of example 2, also load when a compiler makes
+them from C.
+
+4. What a join loses
+--------------------
+
+A join loses the relation between scalars.
+
+::
+
+ if (c) { i = 3; n = 4; } else { i = 7; n = 8; }
+ arr[n - i]
+
+ c !c
+ { i=3 n=4 } { i=7 n=8 } n - i is 1 in each input
+ \ /
+ join: { i=[3,7] n=[4,8] } n - i is [-3,5]
+
+ paths: processed 25 insns
+ joins: R8 min value is negative, either use unsigned index or do a if (index >=0) check.
+
+No jump tests i or n, thus the verifier joins them.
+
+The second case is a scalar that guards the range of a different scalar, when its own values have
+no gap::
+
+ ret = idx < 64 ? 0 : -1;
+ ...
+ if (ret < 0)
+ goto drop;
+ return val->data[idx];
+
+ { ret=0 idx=[0,63] } { ret=-1 idx>=64 }
+ \ /
+ join: { ret=[-1,0] idx=scalar } -1 and 0 have no gap, no jump tests idx
+
+ paths: processed 24 insns
+ joins: math between map_value pointer and register with unbounded min value is not allowed
+
+With -EINVAL as a replacement for -1 there is a gap. Then the program loads: 22 insns.
+
+Recommendation: test a scalar at the place where the program uses it.
+
+**All rules that keep two states.** Each line is a pair of states that has a join. The verifier
+does not use the join, because programs depend on the difference.
+
+::
+
+ the states differ in example applies
+ ---------------------------------------- ------------------------------ ---------------------
+ a register that only one state sets example 2 while there is room
+ a tested register, with a gap example 3 while there is room
+ two tested registers example 3 while there is room
+ a scalar that is 0 in one state only if (h->v4) .. else if (h->v6) while there is room
+ a constant that a subprogram returns true or false, an enum while there is room
+ a constant that becomes a stack offset a[n++] = p while there is room
+ a constant that a helper or a kfunc the size argument of always
+ must know bpf_ringbuf_reserve()
+ the iteration of bpf_loop(), for 8 bpf_loop(4, cb, ...) always
+ iterations or less
+ the iteration of an unrolled loop example 6 always
+
+At a head the verifier uses widening. There only the rule for the stack offset and the two rules
+for iterations apply.
+
+About the stack offset: ``if (x) a[n++] = p; else b[m++] = p;`` is one store insn. A store at an
+unknown offset removes the pointers from all slots that it can write.
+
+5. A counted loop
+-----------------
+
+::
+
+ 8: r6 = 0
+ 9: r1 = r0 <- head. r6=[0,63] r0: map value of 64 bytes
+ 10: r1 += r6
+ 11: r2 = 0
+ 12: *(u8 *)(r1 + 0) = r2
+ 13: r6 += 1
+ 14: if r6 < 64 goto 9 r6=[1,63] goes to 9: covered
+ 15: ... r6=64
+
+Scalar evolution finds that r6 is {0,+,1}, and that the latch permits 64 iterations. Thus the loop
+is a counted loop.
+
+::
- if (data + sizeof(*eth) + sizeof(*iph) + sizeof(*udp) > data_end)
- return 0;
- if (eth->h_proto != htons(ETH_P_IP))
- return 0;
- if (iph->protocol != IPPROTO_UDP || iph->ihl != 5)
- return 0;
- if (udp->dest == 53 || udp->source == 9)
- ...;
+ r6=0
+ | the verifier changes the state before it adds the state to set(9):
+ v each induction variable gets its range for all iterations
+ +--> 9: head { r6=[0,63] }
+ | body verified 1 time. The store stays in the 64 bytes.
+ | 13: r6 += 1
+ +-- 14: if r6 < 64 { r6=[1,63] } returns to the head: { r6=[0,63] } covers it
+ |
+ v
+ 15: { r6=64 }
-which makes such programs easier to write comparing to LD_ABS insn
-and significantly faster.
+No block of the loop has a new state after one pass. The loop is complete.
-Pruning
-=======
+6. A loop that is not counted
+-----------------------------
-The verifier does not actually walk all possible paths through the program. For
-each new branch to analyse, the verifier looks at all the states it's previously
-been in when at this instruction. If any of them contain the current state as a
-subset, the branch is 'pruned' - that is, the fact that the previous state was
-accepted implies the current state would be as well. For instance, if in the
-previous state, r1 held a packet-pointer, and in the current state, r1 holds a
-packet-pointer with a range as long or longer and at least as strict an
-alignment, then r1 is safe. Similarly, if r2 was NOT_INIT before then it can't
-have been used by any path from that point, so any value in r2 (including
-another NOT_INIT) is safe. The implementation is in the function regsafe().
-Pruning considers not only the registers but also the stack (and any spilled
-registers it may hold). They must all be safe for the branch to be pruned.
-This is implemented in states_equal().
+::
+
+ 9: r7 = 0
+ 10: r7 += 3 <- head. r7=scalar
+ 11: r1 = *(u64 *)(r6 + 0) memory that a different cpu can change
+ 12: if r1 != 0 goto 10 r7=scalar goes to 10: covered
+ 13: r0 = r7
+ 14: exit
+
+There are two questions, and they have two answers.
+
+::
+
+ question answer
+ ----------------------------- ---------------------------------------------------------------
+ Are all iterations safe? The fixpoint shows it, see below.
+ Does the loop end? The verifier does not prove it. See example 8.
+
+The number of iterations is not known. But scalar evolution knows what the loop changes: r7 is
+{0,+,3}. Without a limit on the iterations r7 can have each value. Thus the verifier makes r7 an
+unknown scalar in the state that enters the loop. After one pass the state that returns to the
+head is covered.
+
+::
+
+ paths: BPF program is too large. Processed 1000001 insn r7 differs in each iteration
+ joins: processed 18 insns
+
+**The verifier unrolls first.** It verifies the first iterations one by one, with the real values:
+``r7=0`` in the first iteration. Many loops exit in the first or second iteration. In other loops
+constants decide all exits. The verifier stops the unroll when a jump that exits the loop can go
+in the two directions. In this example that occurs in the first iteration.
+
+::
+
+ loop method
+ ------------------------------------------------- -------------------------------------------
+ counted, scalar evolution describes all registers ranges at the head, example 5
+ counted, a register has no description unroll, 100000 insns at most
+ not counted unroll while constants decide the exits,
+ 4096 insns at most. Then this example and
+ example 7
+
+Scalar evolution describes these registers and stack slots of a counted loop::
+
+ what the loop does with the register the register at the head
+ ------------------------------------------------- -------------------------------------------
+ nothing no change
+ computes the same constant in each iteration no change
+ adds a constant range from the number of iterations
+ sets it to one of some values that do not change union of the values
+ loads it again in each iteration unknown scalar
+ clears bits of [0, maximum]: &, zero extension no change
+ moves a pointer to arena pointer to arena, it has no offset
+
+7. Widening
+-----------
+
+::
+
+ 9: r7 = 1
+ 10: r7 *= 3 <- head
+ 11: r7 &= 63
+ 12: r1 = *(u64 *)(r6 + 0)
+ 13: if r1 != 0 goto 10
+ 14: r6 += r7 r7=[0,63]
+ 15: r0 = *(u8 *)(r6 + 0) r6: map value of 64 bytes
+
+r7 is not an add recurrence. Thus scalar evolution has no description of it. The verifier iterates
+the loop::
+
+ pass r7 at the head r7 at the latch step
+ ----- ----------------- ------------------ ------------------------------------------------
+ 1 1 3 the verifier unrolls this iteration (example 6)
+ 2 3 9 not covered, the upper bound grows: [3, S32_MAX]
+ 3 [3, S32_MAX] [0,63] not covered, the lower bound grows: [0, S32_MAX]
+ 4 [0, S32_MAX] [0,63] covered: the loop is complete
+
+With a usual join a range can grow by a small step in each pass, and 2^64 passes are possible.
+Widening moves the bound that grows directly to one of a small number of stops::
+
+ lower bound: S64_MIN <-- S32_MIN <-- 0
+ upper bound: 0 --> S32_MAX --> U32_MAX --> S64_MAX
+
+Thus a range can grow only a small number of times.
+
+The edge from 13 to 14 has ``r7=[0,63]``, the value of r7 after ``&= 63``. Thus the access at 15 is
+correct. Widening at the head does not change a value that the body calculates again.
+
+::
+
+ paths: infinite loop detected at insn 10
+ joins: processed 31 insns
+
+8. A loop ends at run time: may_goto
+------------------------------------
+
+It is possible that the loop of example 6 never ends. The verifier adds a may_goto insn before the
+latch.
+
+- may_goto decrements a count.
+- After 65535 iterations it reads the clock.
+- It jumps to its target when the time slice is over.
+
+One run of a program has one time slice and one count. All loops and all subprograms of the run
+share them.
+
+::
+
+ one run of a program on a cpu
+ +---------------------------------------------------------------+
+ | start of the time slice (250 ms) count (65535) |
+ +---------------------------------------------------------------+
+ | entry: the subprogram starts from the count that is left
+ | exit: the subprogram gives back the count that it has left
+ v
+ +----------------+ +----------------+ +----------------+
+ | main program | call | subprogram f | call | subprogram g |
+ | loop, may_goto | ---> | loop, may_goto | ---> | loop, may_goto |
+ +----------------+ +----------------+ +----------------+
+
+Thus a run reads the clock even if each loop in each subprogram is short. Each frame on the stack
+does at most 65535 iterations between two reads of the clock.
+
+This is the program that runs::
+
+ 17: r7 += 3
+ 18: r1 = *(u64 *)(r6 + 0)
+ 19..25: may_goto 30
+ 26: if r1 != 0 goto 17
+ 27: r0 = r7
+ 28: exit
+ ...
+ 30: r0 = 0 after the last insn of the function, for all its added may_goto
+ 31: exit
+
+With 0 in memory it returns 3. With 1 in memory it returns 0 after 250 ms.
+
+The edge of the added may_goto is also an edge for the analysis. The state at the latch goes to
+exit with r0 = 0. The verifier checks it as it checks each exit: the program must hold nothing,
+and 0 must be a valid return value.
+
+The verifier adds may_goto only where it is necessary::
+
+ loop the verifier
+ ----------------------------------------------------------- ---------------------------------
+ counted adds nothing
+ unrolled to its end adds nothing
+ each iteration executes may_goto or bpf_iter_*_next() adds nothing
+ no exit rejects the program:
+ loop at insn N never exits
+ all other loops adds may_goto
+
+No return value is correct for each program type. Thus the kernel informs user space that it
+stopped a run. It sets ``unwound`` in ``struct bpf_prog_info``. It also writes the cause to the
+stderr stream of the program.
+
+**The target of an added may_goto.** An added may_goto has one target, and the verifier verifies
+only this edge. The verifier selects the target at load time::
+
+ does the frame hold a lock, a reference or an rcu section
+ that it did not hold at its entry ?
+ | no | yes
+ v v
+ exit from the function exit from the loop
+ (insn 30 above) (insn 27 above: jump across the latch)
+
+Three programs show that one target for all loops is not possible.
+
+Program 1: exit from the function.
+
+::
+
+ do { p = lookup(hash map); } while (!p);
+ return *p;
+
+ paths: infinite loop detected at insn 1
+ joins: processed 14 insns. With an empty map it returns 0 after 251 ms
+
+The code after the loop uses the fact that the latch found: p is not NULL. A jump across the latch
+sends NULL to that code.
+
+Program 2: exit from the loop.
+
+::
+
+ lock(&v->l);
+ do { v->cnt++; } while (v->flag);
+ unlock(&v->l);
+ return 1;
+
+ paths: infinite loop detected at insn 13
+ joins: processed 24 insns. With flag = 1 it returns its own 1 after 250 ms, the lock is free
+
+The program cannot exit with the lock. The code after the loop releases the lock.
+
+Program 3: no target is possible.
+
+::
+
+ rcu_read_lock();
+ do { p = lookup(hash map); } while (!p);
+ x = *p;
+ rcu_read_unlock();
+
+ paths: infinite loop detected at insn 2
+ joins: R0 invalid mem access 'map_value_or_null'
+
+The program cannot exit in the rcu section, and after the loop p can be NULL. There is no third
+target. Code that releases objects and then exits is a landing pad. Landing pads belong to calls,
+not to loops.
-Some technical details about state pruning implementation could be found below.
+**Exit from the function.**
-Register liveness tracking
---------------------------
+::
+
+ the may_goto is in insns at the target
+ --------------------------------------------------------- -----------------------------------
+ the main program, the callback of a timer r0 = 0; exit
+ a subprogram, and unwind_goto follows each call of it r0 = 0; unwind_exit
+ the callers continue at their
+ landing pads
+ a different subprogram, a callback of bpf_loop() not possible: exit from the loop
+
+Some states at the may_goto can hold an object while other states hold nothing. Then the target
+is the exit from the loop for all states.
+
+**Exit from the loop.**
+
+::
+
+ position of the may_goto target is the edge in the program?
+ ------------------------------- -------------------------- ---------------------------------
+ before a jump that exits the the target of that jump yes. The facts about the loops
+ loop stay correct.
+ at the head, when no such jump an exit of this loop and no. The edge skips the body.
+ is in each iteration of all loops around it
+
+The body can contain the insn that stops a loop around it, for example a call of
+``bpf_iter_num_next()``. Thus a may_goto at the head exits from all loops around it.
+
+9. Subprograms
+--------------
+
+::
+
+ 1: r1 = random()
+ 2: call low3 waits. r1=scalar goes to 10
+ 3: r6 = r0 r0=[0,7]
+ ...
+ 7: call low3 r1=scalar goes to 10: covered. r0=[0,7] is known
+ 8: r0 += r6
+ 9: exit
+ 10: r0 = r1 low3(x) { return x & 7; }
+ 11: r0 &= 7
+ 12: exit r0=[0,7] goes to the calls that wait
+
+The verifier verifies a static subprogram for each entry state, not for each chain of calls.
+
+::
+
+ 2: call low3 ---- { r1=scalar } ----+
+ v
+ +------------------+
+ 7: call low3 -- covered ->| 10: r0 = r1 | verified 1 time
+ | 11: r0 &= 7 |
+ | 12: exit |
+ +------------------+
+ |
+ 3: r6 = r0 <--- { r0=[0,7] } -----+
+ 8: r0 += r6 <--- { r0=[0,7] } -----+
+
+- The entry state arrives at the first block of the subprogram like each state at a block. The
+ verifier drops it, joins it or adds it.
+- The call waits. The verifier saves the state of the caller.
+- Each exit state goes to all calls that wait for this entry state.
+
+The result::
+
+ paths: processed 16 insns
+ joins: processed 13 insns each insn 1 time
+
+The next program has 12 subprograms. Each subprogram calls the next one 2 times. It has 103 insns.
+
+::
+
+ paths: f1 --> f2 f2 --> f3 f3 f3 f3 --> ... 2^11 calls of f12
+ joins: f1 --> f2 --> f3 --> ... each subprogram 1 time
+
+ paths: processed 45052 insns
+ joins: processed 103 insns
+
+In some cases the verifier verifies a subprogram in the context of the caller. Then the frames of
+the callers are part of the state, and states with different chains of calls have no join. The
+most important cases:
-In order to make state pruning effective, liveness state is tracked for each
-register and stack slot. The basic idea is to track which registers and stack
-slots are actually used during subseqeuent execution of the program, until
-program exit is reached. Registers and stack slots that were never used could be
-removed from the cached state thus making more states equivalent to a cached
-state. This could be illustrated by the following program::
+- The subprogram gets a pointer to the stack of the caller, or a dynptr.
+- The subprogram gets a pointer that can be NULL. It does the NULL check for the caller.
+- The call is in an unrolled loop. The iterations must not meet in the subprogram.
+- The subprogram is a callback.
- 0: call bpf_get_prandom_u32()
- 1: r1 = 0
- 2: if r0 == 0 goto +1
- 3: r0 = 1
- --- checkpoint ---
- 4: r0 = r1
- 5: exit
+The verifier verifies a global subprogram alone, after the main program. The entry state has the
+arguments of its BTF prototype. At a call of a global subprogram the verifier checks the arguments
+against the prototype. It does not enter the subprogram. After the call the return value is an
+unknown scalar. The program must not hold a lock at the call.
-Suppose that a state cache entry is created at instruction #4 (such entries are
-also called "checkpoints" in the text below). The verifier could reach the
-instruction with one of two possible register states:
+10. A call through a table of functions
+---------------------------------------
+
+::
+
+ static const op_fn ops[16] = { op0, op1, ... op15 };
+ return ops[kind & 15](3);
+
+ 3: w1 &= 15
+ 4: r1 <<= 3 r1=[0,120] in steps of 8
+ 5: r2 = ops
+ 7: r2 += r1
+ 8: r2 = *(u64 *)(r2 + 0) { r2=one of 16 functions }
+ 9: w1 = 3
+ 10: callx r2 waits for each of the 16. r1=3 goes to all of them
+ 11: exit r0 of each of them arrives here, and the verifier joins them
+
+The load at 8 can give 16 different pointers. The verifier does not make 16 states. It makes one
+state, in which r2 holds one of a set of functions.
+
+::
+
+ +-----+-----+-------+------+
+ | op0 | op1 | ... | op15 | 8: load --> { r2=one of 16 functions } 1 state
+ +-----+-----+-------+------+
+ 10: callx r2
+ | the entry state { r1=3 } goes to each function
+ +------------+---------------+------------+
+ v v v
+ op0 op1 ... op15
+ | | |
+ +------------+---------------+------------+
+ v the exit states arrive at 11: join
+ 11: exit
+
+16 states are not useful here. They differ only in the function, and they are parallel from the
+load to the call. At the call the verifier verifies each function of the set with the arguments
+of the call, as for a direct call.
+
+::
+
+ paths: processed 118 insns
+ joins: processed 73 insns
+
+- Only a load from a table makes a set. Two pointers to different functions that arrive from two
+ branches have no join (example 2).
+- A helper or a kfunc that gets a callback must know the callback. There the verifier makes one
+ state for each function of the set.
+
+11. Recursion
+-------------
+
+::
+
+ depth(n) { return n ? (depth(n - 1) + 1) & 255 : 0; } called with 5
+
+Recursion uses the mechanisms of examples 7 and 9. The verifier handles the entry of a recursive
+subprogram like a head. A call of a subprogram that the verifier verifies at that time waits like
+each other call.
+
+::
+
+ main program: call depth entry state { r1=5 }
+ |
+ v
+ +--> entry of depth() like a head: join and widening
+ | ...
+ +--- call depth(n - 1) entry state { r1=4 }. The call waits.
+ ^
+ |
+ exit of depth() each exit state goes to all calls that wait.
+ The call in depth() is one of them.
+
+Two values must become stable::
+
+ n at the entry step
+ ------------------ ---------------------------------------------------------------------
+ 5 from the main program
+ [0,5] 4 arrives from the call in depth(): join and widening,
+ the lower bound moves to 0
+ [0,5] [0,4] arrives from that call: covered. Stable.
+
+ the return value step
+ ------------------ ---------------------------------------------------------------------
+ 0 n == 0. It goes to the calls that wait.
+ [0,1] (0 + 1) & 255 arrives at the exit
+ [0,S32_MAX] the range grows on a cycle: widening
+ [0,S32_MAX] ([0,S32_MAX] + 1) & 255 = [0,255] arrives at the exit: covered. Stable.
+
+The depth of the recursion is not known, like the number of iterations in example 6. Thus each
+subprogram on a cycle of the call graph gets a check at its entry. The check counts, for each
+program and cpu, the stack that recursive frames use and the calls in this run.
+
+::
+
+ limit for one run of a program value
+ ------------------------------------------ ---------------------------------------------
+ kernel stack of recursive frames 4096 bytes
+ stack in arena of recursive frames the part of this stack that is free
+ calls of recursive subprograms 8M
+
+ kernel memory arena
+ +----------------------+ +---------------------------+
+ | BPF stack <- r10 | | stack in arena <- r12 |
+ | frames without | | frames without |
+ | recursion | | recursion |
+ +----------------------+ +---------------------------+
+ | recursive frames: | | recursive frames: |
+ | 4096 bytes | | the remaining part |
+ +----------------------+ +---------------------------+
+
+When a run hits a limit, the subprogram returns 0. If unwind_goto follows each call of the
+subprogram, it uses unwind_exit as a replacement. The kernel marks the program as in example 8.
+
+::
-* r0 = 1, r1 = 0
-* r0 = 0, r1 = 0
+ paths: recursive call from depth() to depth()
+ joins: processed 19 insns. It returns 5
-However, only the value of register ``r1`` is important to successfully finish
-verification. The goal of the liveness tracking algorithm is to spot this fact
-and figure out that both states are actually equivalent.
+Why the analysis ends, and the limits
+-------------------------------------
-Understanding eBPF verifier messages
-====================================
+- A state in a set can only grow. With widening it grows only a small number of times.
+- Two states stay in a set only if they have no join, or if a rule of example 4 keeps them. Real
+ programs have a small number of such states at an insn, because a state contains only live
+ registers.
+- It is possible to write a program that has many such states. Thus the verifier limits the number
+ of states. The limits depend on the size of the program.
-The following are few examples of invalid eBPF programs and verifier error
-messages as seen in the log:
+For a program of N insns::
-Program with unreachable instructions::
+ states at one insn 1024
+ states at one time 16 * N + 4096
+ states in total 256 * N + 64K
- static struct bpf_insn prog[] = {
- BPF_EXIT_INSN(),
- BPF_EXIT_INSN(),
- };
-
-Error::
+The verifier rejects a program that hits a limit. The messages are
+``more than 1024 states that cannot be joined at insn N`` and ``too many states for N insns``.
- unreachable insn 1
+The rules of example 4 do not cause a program to hit a limit. Most of them apply only while there
+is room::
-Program that reads uninitialized register::
+ 0 half of a limit limit
+ |--------------------------|-------------------------------|
+ room: the rules keep no room: the verifier the verifier
+ two states uses each join that exists rejects the program
- BPF_MOV64_REG(BPF_REG_0, BPF_REG_2),
- BPF_EXIT_INSN(),
-
-Error::
-
- 0: (bf) r0 = r2
- R2 !read_ok
-
-Program that doesn't initialize R0 before exiting::
-
- BPF_MOV64_REG(BPF_REG_2, BPF_REG_1),
- BPF_EXIT_INSN(),
-
-Error::
-
- 0: (bf) r2 = r1
- 1: (95) exit
- R0 !read_ok
-
-Program that accesses stack out of bounds::
-
- BPF_ST_MEM(BPF_DW, BPF_REG_10, 8, 0),
- BPF_EXIT_INSN(),
-
-Error::
-
- 0: (7a) *(u64 *)(r10 +8) = 0
- invalid stack off=8 size=8
-
-Program that doesn't initialize stack before passing its address into function::
-
- BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
- BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
- BPF_LD_MAP_FD(BPF_REG_1, 0),
- BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
- BPF_EXIT_INSN(),
-
-Error::
-
- 0: (bf) r2 = r10
- 1: (07) r2 += -8
- 2: (b7) r1 = 0x0
- 3: (85) call 1
- invalid indirect read from stack off -8+0 size 8
-
-Program that uses invalid map_fd=0 while calling to map_lookup_elem() function::
-
- BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
- BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
- BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
- BPF_LD_MAP_FD(BPF_REG_1, 0),
- BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
- BPF_EXIT_INSN(),
-
-Error::
-
- 0: (7a) *(u64 *)(r10 -8) = 0
- 1: (bf) r2 = r10
- 2: (07) r2 += -8
- 3: (b7) r1 = 0x0
- 4: (85) call 1
- fd 0 is not pointing to valid bpf_map
-
-Program that doesn't check return value of map_lookup_elem() before accessing
-map element::
-
- BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
- BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
- BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
- BPF_LD_MAP_FD(BPF_REG_1, 0),
- BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
- BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 0),
- BPF_EXIT_INSN(),
-
-Error::
-
- 0: (7a) *(u64 *)(r10 -8) = 0
- 1: (bf) r2 = r10
- 2: (07) r2 += -8
- 3: (b7) r1 = 0x0
- 4: (85) call 1
- 5: (7a) *(u64 *)(r0 +0) = 0
- R0 invalid mem access 'map_value_or_null'
-
-Program that correctly checks map_lookup_elem() returned value for NULL, but
-accesses the memory with incorrect alignment::
-
- BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
- BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
- BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
- BPF_LD_MAP_FD(BPF_REG_1, 0),
- BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
- BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 1),
- BPF_ST_MEM(BPF_DW, BPF_REG_0, 4, 0),
- BPF_EXIT_INSN(),
-
-Error::
-
- 0: (7a) *(u64 *)(r10 -8) = 0
- 1: (bf) r2 = r10
- 2: (07) r2 += -8
- 3: (b7) r1 = 1
- 4: (85) call 1
- 5: (15) if r0 == 0x0 goto pc+1
- R0=map_ptr R10=fp
- 6: (7a) *(u64 *)(r0 +4) = 0
- misaligned access off 4 size 8
-
-Program that correctly checks map_lookup_elem() returned value for NULL and
-accesses memory with correct alignment in one side of 'if' branch, but fails
-to do so in the other side of 'if' branch::
-
- BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
- BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
- BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
- BPF_LD_MAP_FD(BPF_REG_1, 0),
- BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
- BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 2),
- BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 0),
- BPF_EXIT_INSN(),
- BPF_ST_MEM(BPF_DW, BPF_REG_0, 0, 1),
- BPF_EXIT_INSN(),
-
-Error::
-
- 0: (7a) *(u64 *)(r10 -8) = 0
- 1: (bf) r2 = r10
- 2: (07) r2 += -8
- 3: (b7) r1 = 1
- 4: (85) call 1
- 5: (15) if r0 == 0x0 goto pc+2
- R0=map_ptr R10=fp
- 6: (7a) *(u64 *)(r0 +0) = 0
- 7: (95) exit
-
- from 5 to 8: R0=imm0 R10=fp
- 8: (7a) *(u64 *)(r0 +0) = 1
- R0 invalid mem access 'imm'
-
-Program that performs a socket lookup then sets the pointer to NULL without
-checking it::
-
- BPF_MOV64_IMM(BPF_REG_2, 0),
- BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_2, -8),
- BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
- BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
- BPF_MOV64_IMM(BPF_REG_3, 4),
- BPF_MOV64_IMM(BPF_REG_4, 0),
- BPF_MOV64_IMM(BPF_REG_5, 0),
- BPF_EMIT_CALL(BPF_FUNC_sk_lookup_tcp),
- BPF_MOV64_IMM(BPF_REG_0, 0),
- BPF_EXIT_INSN(),
-
-Error::
-
- 0: (b7) r2 = 0
- 1: (63) *(u32 *)(r10 -8) = r2
- 2: (bf) r2 = r10
- 3: (07) r2 += -8
- 4: (b7) r3 = 4
- 5: (b7) r4 = 0
- 6: (b7) r5 = 0
- 7: (85) call bpf_sk_lookup_tcp#65
- 8: (b7) r0 = 0
- 9: (95) exit
- Unreleased reference id=1, alloc_insn=7
-
-Program that performs a socket lookup but does not NULL-check the returned
-value::
-
- BPF_MOV64_IMM(BPF_REG_2, 0),
- BPF_STX_MEM(BPF_W, BPF_REG_10, BPF_REG_2, -8),
- BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
- BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
- BPF_MOV64_IMM(BPF_REG_3, 4),
- BPF_MOV64_IMM(BPF_REG_4, 0),
- BPF_MOV64_IMM(BPF_REG_5, 0),
- BPF_EMIT_CALL(BPF_FUNC_sk_lookup_tcp),
- BPF_EXIT_INSN(),
-
-Error::
-
- 0: (b7) r2 = 0
- 1: (63) *(u32 *)(r10 -8) = r2
- 2: (bf) r2 = r10
- 3: (07) r2 += -8
- 4: (b7) r3 = 4
- 5: (b7) r4 = 0
- 6: (b7) r5 = 0
- 7: (85) call bpf_sk_lookup_tcp#65
- 8: (95) exit
- Unreleased reference id=1, alloc_insn=7
+A join without the rules can lose a fact that the program uses. But 20 flags do not make a million
+states.
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 69/81] bpf, docs: Rewrite verifier.rst
2026-10-26 2:59 ` [PATCH bpf 69/81] bpf, docs: Rewrite verifier.rst Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 9120 bytes --]
> bpf, docs: Rewrite verifier.rst
TL;DR:
- Joining verifier may reject programs older kernels accepted, contradicting
the Q&A's compatibility guarantee
- verifier.rst may_goto and loop-unroll docs appear inaccurate for some JITs,
configs and long loops
- Q&A also omits new limits and still says loops are unclear
> diff --git a/Documentation/bpf/bpf_design_QA.rst b/Documentation/bpf/bpf_design_QA.rst
> index 98d70b8a2533..f86b272920ae 100644
> --- a/Documentation/bpf/bpf_design_QA.rst
> +++ b/Documentation/bpf/bpf_design_QA.rst
> @@ -92,11 +92,8 @@ Q: What are the verifier limits?
> A: The only limit known to the user space is BPF_MAXINSNS (4096).
> It's the maximum number of instructions that the unprivileged bpf
> program can have. The verifier has various internal limits.
> -Like the maximum number of instructions that can be explored during
> -program analysis. Currently, that limit is set to 1 million.
> -Which essentially means that the largest program can consist
> -of 1 million NOP instructions. There is a limit to the maximum number
> -of subsequent branches, a limit to the number of nested bpf-to-bpf
> +The largest program can consist of 1 million instructions.
> +There is a limit to the number of nested bpf-to-bpf
> calls, a limit to the number of the verifier states per instruction,
> a limit to the number of maps used by the program.
> All these limits can be hit with a sufficiently complex program.
The answer still ends with the guarantee:
The bpf development process guarantees that the future kernel
versions will accept all bpf programs that were accepted by
the earlier versions.
Does this contradict the verifier.rst that this same patch rewrites?
Example 4 in verifier.rst has programs that the verifier that follows each
path accepts, and that the joining verifier rejects:
if (c) { i = 3; n = 4; } else { i = 7; n = 8; }
arr[n - i]
paths: processed 25 insns
joins: R8 min value is negative, either use unsigned index or do a if (index >=0) check.
The ret = idx < 64 ? 0 : -1 program in the same example likewise goes from
loading to "math between map_value pointer and register with unbounded min
value is not allowed".
verifier.rst also says that a join without the rules can lose a fact that
the program uses, and that most of the rules apply only while there is
room. So the number of states elsewhere in a program can decide whether
the program loads.
Programs that loaded before can now fail, and user space can see that.
Should the guarantee be qualified here, or should the series keep
accepting such programs?
Two smaller points about this answer.
The new text does not mention the per-program limits that replace the 1M
insn limit ("too many states for N insns", 16*N+4096 states at one time and
256*N+64K in total, see kernel/bpf/flow.c).
Also, the Q/A earlier in this file still reads:
Q: Can loops be supported in a safe way?
A: It's not clear yet.
while the new verifier.rst says the verifier permits loops.
[ ... ]
> diff --git a/Documentation/bpf/verifier.rst b/Documentation/bpf/verifier.rst
> index 510d15bc697b..cc1e6eae5471 100644
> --- a/Documentation/bpf/verifier.rst
> +++ b/Documentation/bpf/verifier.rst
> @@ -3,558 +3,1179 @@
[ ... ]
> +
> + loop method
> + ------------------------------------------------- -------------------------------------------
> + counted, scalar evolution describes all registers ranges at the head, example 5
> + counted, a register has no description unroll, 100000 insns at most
> + not counted unroll while constants decide the exits,
> + 4096 insns at most. Then this example and
> + example 7
Is "4096 insns at most" accurate for a loop that is not counted?
bpf_flow_loop_enter() in kernel/bpf/verifier.c raises the unroll budget for
long loops:
/* Two trips for a loop of any length. Many a loop is left in the second one. */
if (patience && flow_walk_factor(cur) == 1)
patience = max_t(u64, patience, 2ULL * loop->nr_insns);
The walk then continues while:
env->insn_processed - entry->walk_start <= entry->walk_patience
A loop with more than 2048 insns that is not inside another unrolled loop
is therefore unrolled for 2 * nr_insns insns, for example 6000 insns for a
loop of 3000 insns.
Should the table say "4096 insns, or two iterations of a longer loop"?
[ ... ]
> +8. A loop ends at run time: may_goto
> +------------------------------------
> +
> +It is possible that the loop of example 6 never ends. The verifier adds a may_goto insn before the
> +latch.
> +
> +- may_goto decrements a count.
> +- After 65535 iterations it reads the clock.
> +- It jumps to its target when the time slice is over.
> +
> +One run of a program has one time slice and one count. All loops and all subprograms of the run
> +share them.
Does may_goto behave like this on every configuration? The description of
the run time behavior depends on the JIT.
First, the clock. bpf_do_misc_fixups() in kernel/bpf/fixups.c emits the
timed sequence (count 0xffff, call to arch_bpf_timed_may_goto, 250 ms in
bpf_check_timed_may_goto()) only when bpf_jit_supports_timed_may_goto() is
true. Otherwise each frame starts with:
BPF_ST_MEM(BPF_DW, BPF_REG_FP, -stack_depth, BPF_MAX_LOOPS)
and only counts down 8M iterations. No clock is read, and
bpf_prog_report_may_goto_violation() is never called, so unwound is not set
and nothing is written to stderr.
That contradicts "After 65535 iterations it reads the clock" and the later
paragraph about "unwound" and the stderr stream of the program.
Second, the shared budget. The per-run budget ("One run of a program has
one time slice and one count. All loops and all subprograms of the run
share them.") is allocated only under:
subprogs[i].stack_extra && bpf_jit_supports_timed_may_goto() &&
bpf_jit_supports_percpu_insn() && bpf_jit_supports_exceptions() &&
prog->jit_requested && prog->type != BPF_PROG_TYPE_EXT
Several cases do not meet this:
- riscv and loongarch support timed may_goto but not exceptions.
- x86-64 without the ORC unwinder and arm64 without the frame-pointer
unwinder report no exceptions support.
- freplace programs (BPF_PROG_TYPE_EXT) are excluded.
In all of these each frame has its own 250 ms slice and its own count.
That is the behavior the comment above bpf_check_timed_may_goto() says the
budget exists to prevent.
So the diagram and "Thus a run reads the clock even if each loop in each
subprogram is short" hold only on x86-64 and arm64 with the required
unwinder. Should the section state these conditions?
[ ... ]
> +The edge of the added may_goto is also an edge for the analysis. The state at the latch goes to
> +exit with r0 = 0. The verifier checks it as it checks each exit: the program must hold nothing,
> +and 0 must be a valid return value.
[ ... ]
> +**The target of an added may_goto.** An added may_goto has one target, and the verifier verifies
> +only this edge. The verifier selects the target at load time::
> +
> + does the frame hold a lock, a reference or an rcu section
> + that it did not hold at its entry ?
> + | no | yes
> + v v
> + exit from the function exit from the loop
> + (insn 30 above) (insn 27 above: jump across the latch)
Is this chart complete? It says the target depends only on whether the
frame holds a lock, a reference or an rcu section, and the earlier
paragraph says the r0 = 0 exit is checked like each exit, so that "0 must
be a valid return value".
budget_can_unwind() in kernel/bpf/verifier.c chooses the target otherwise:
if (st->acquired_refs != frame->entry_refs ||
st->active_rcu_locks != frame->entry_rcu_locks ||
st->active_preempt_locks != frame->entry_preempt_locks)
return false;
...
if (top && !frame->subprogno && !zero_is_valid_retval(env))
return false;
Two things are missing from the chart.
First, a preempt-disable section also sends the may_goto to the exit of the
loop.
Second, when the main program cannot return 0, the verifier does not check
and reject the r0 = 0 exit. It never makes that edge, and selects the exit
of the loop instead (guard_no_unwind in check_loop_guard()).
For program 1 (do { p = lookup(); } while (!p); return *p;) under a program
type whose return range excludes 0, the result is therefore "invalid mem
access 'map_value_or_null'", not a return value error.
Should the chart and the sentence about "0 must be a valid return value"
list these conditions?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 70/81] selftests/bpf: Adjust tests of precision tracking
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (68 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 69/81] bpf, docs: Rewrite verifier.rst Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 71/81] selftests/bpf: Adjust tests of linked registers and spills Alexei Starovoitov
` (12 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Precision tracking went back from a check through the insns of a path. It
found the scalars that the check depends on. At log level 2 it printed each
step as a "mark_precise:" line. These tests match 358 such lines.
"bpf: Remove precision tracking" removed the mechanism and the lines.
Keep the progs. Each prog moves a scalar along a difficult route. Then it
uses the scalar where the verifier must know the value, for example as an
offset.
file the route of the scalar
---------------------------- -------------------------------------------
verifier_precision.c neg, byte swap, atomic insns, load-acquire,
store-release
verifier_subprog_precision.c static and global subprogs, callbacks,
spills into the frame of the caller
stack_arg_precision.c stack arguments of a subprog and of a kfunc
Match the value that the log shows for the scalar.
before mark_precise: frame0: regs=r2 stack= before 1: (87) r2 = -r2
after 1: (87) r2 = -r2 ; R2=-8
The four tests in verifier/precise.c examine only the verdict and the
return value. No expected verdict changes.
The tests in progs/ also drop BPF_F_TEST_STATE_FREQ. The flag made the
verifier save states for pruning more frequently. It has no effect now.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/stack_arg_precision.c | 48 +--
.../selftests/bpf/progs/verifier_precision.c | 166 +++------
.../bpf/progs/verifier_subprog_precision.c | 348 +++---------------
.../testing/selftests/bpf/verifier/precise.c | 69 +---
4 files changed, 112 insertions(+), 519 deletions(-)
diff --git a/tools/testing/selftests/bpf/progs/stack_arg_precision.c b/tools/testing/selftests/bpf/progs/stack_arg_precision.c
index c9564776d733..bf42ca629a01 100644
--- a/tools/testing/selftests/bpf/progs/stack_arg_precision.c
+++ b/tools/testing/selftests/bpf/progs/stack_arg_precision.c
@@ -23,7 +23,7 @@ int __btf_kfunc_gen(struct __sk_buff *ctx)
}
/*
- * Test precision backtracking across bpf-to-bpf call for kfunc stack arg.
+ * Test that the range of a stack arg is known across bpf-to-bpf call for kfunc stack arg.
* subprog_call_mem_kfunc receives a size as incoming stack arg (arg6)
* and forwards it as mem__sz (arg7) to bpf_kfunc_call_stack_arg_mem.
*/
@@ -52,52 +52,12 @@ static long subprog_call_mem_kfunc(long a, long b, long c, long d, long e, long
}
SEC("tc")
-__description("stack_arg: precision backtracking across bpf2bpf call for kfunc")
+__description("stack_arg: range across bpf2bpf call for kfunc")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("from 9 to 11: joined")
+__msg("13: (79) r1 = *(u64 *)(r11 +8) ; R1=scalar(smin=umin=smin32=umin32=4,smax=umax=smax32=umax32=6,var_off=(0x4; 0x2))")
__btf_func_path("btf__stack_arg_precision.bpf.o")
-__msg("mark_precise: frame1: last_idx 26 first_idx 13 subseq_idx -1")
-__msg("mark_precise: frame1: regs= stack= before 25: (b7) r5 = 5")
-__msg("mark_precise: frame1: regs= stack= before 24: (b7) r4 = 4")
-__msg("mark_precise: frame1: regs= stack= before 23: (b7) r3 = 3")
-__msg("mark_precise: frame1: regs= stack= before 22: (b7) r2 = 2")
-__msg("mark_precise: frame1: regs= stack= before 21: (b7) r1 = 1")
-__msg("mark_precise: frame1: regs= stack= before 20: (7b) *(u64 *)(r11 -16) = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 19: (7b) *(u64 *)(r11 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 18: (07) r2 += -8")
-__msg("mark_precise: frame1: regs=r1 stack= before 17: (bf) r2 = r10")
-__msg("mark_precise: frame1: regs=r1 stack= before 16: (7b) *(u64 *)(r10 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 14: (18) r2 = 0x807060504030201")
-__msg("mark_precise: frame1: regs=r1 stack= before 13: (79) r1 = *(u64 *)(r11 +8)")
-__msg("mark_precise: frame1: parent state regs= stack=: frame1: R10=fp0")
-__msg("mark_precise: frame0: parent state regs= stack=: R10=fp0")
-__msg("mark_precise: frame1: last_idx 11 first_idx 11 subseq_idx 13")
-__msg("mark_precise: frame1: regs= stack= before 11: (85) call pc+1")
-__msg("mark_precise: frame0: parent state regs= stack=: R10=fp0")
-__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx 11")
-__msg("mark_precise: frame0: regs= stack= before 9: (05) goto pc+1")
-__msg("mark_precise: frame0: regs= stack= before 8: (7a) *(u64 *)(r11 -8) = 4")
-__msg("mark_precise: frame1: last_idx 26 first_idx 13 subseq_idx -1 ")
-__msg("mark_precise: frame1: regs= stack= before 25: (b7) r5 = 5")
-__msg("mark_precise: frame1: regs= stack= before 24: (b7) r4 = 4")
-__msg("mark_precise: frame1: regs= stack= before 23: (b7) r3 = 3")
-__msg("mark_precise: frame1: regs= stack= before 22: (b7) r2 = 2")
-__msg("mark_precise: frame1: regs= stack= before 21: (b7) r1 = 1")
-__msg("mark_precise: frame1: regs= stack= before 20: (7b) *(u64 *)(r11 -16) = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 19: (7b) *(u64 *)(r11 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 18: (07) r2 += -8")
-__msg("mark_precise: frame1: regs=r1 stack= before 17: (bf) r2 = r10")
-__msg("mark_precise: frame1: regs=r1 stack= before 16: (7b) *(u64 *)(r10 -8) = r2")
-__msg("mark_precise: frame1: regs=r1 stack= before 14: (18) r2 = 0x807060504030201")
-__msg("mark_precise: frame1: regs=r1 stack= before 13: (79) r1 = *(u64 *)(r11 +8)")
-__msg("mark_precise: frame1: parent state regs= stack=: frame1: R10=fp0")
-__msg("mark_precise: frame0: parent state regs= stack=: R10=fp0")
-__msg("mark_precise: frame1: last_idx 11 first_idx 11 subseq_idx 13 ")
-__msg("mark_precise: frame1: regs= stack= before 11: (85) call pc+1")
-__msg("mark_precise: frame0: parent state regs= stack=: R10=fp0")
-__msg("mark_precise: frame0: last_idx 10 first_idx 10 subseq_idx 11 ")
-__msg("mark_precise: frame0: regs= stack= before 10: (7a) *(u64 *)(r11 -8) = 6")
__naked void stack_arg_precision_bpf2bpf(void)
{
asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_precision.c b/tools/testing/selftests/bpf/progs/verifier_precision.c
index 2ec3ee7821c0..aacb75dffb52 100644
--- a/tools/testing/selftests/bpf/progs/verifier_precision.c
+++ b/tools/testing/selftests/bpf/progs/verifier_precision.c
@@ -16,10 +16,7 @@ struct {
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0xfffffff8 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (87) r2 = -r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 8")
+__msg("1: (87) r2 = -r2 ; R2=-8")
__naked int bpf_neg(void)
{
asm volatile (
@@ -36,10 +33,7 @@ __naked int bpf_neg(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0x0 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (d4) r2 = le16 r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 0")
+__msg("1: (d4) r2 = le16 r2 ; R2=0")
__naked int bpf_end_to_le(void)
{
asm volatile (
@@ -57,10 +51,7 @@ __naked int bpf_end_to_le(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0x0 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (dc) r2 = be16 r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 0")
+__msg("1: (dc) r2 = be16 r2 ; R2=0")
__naked int bpf_end_to_be(void)
{
asm volatile (
@@ -82,10 +73,7 @@ __naked int bpf_end_to_be(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (55) if r2 != 0x0 goto pc+2")
-__msg("mark_precise: frame0: regs=r2 stack= before 1: (d7) r2 = bswap16 r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (b7) r2 = 0")
+__msg("1: (d7) r2 = bswap16 r2 ; R2=0")
__naked int bpf_end_bswap(void)
{
asm volatile (
@@ -104,10 +92,8 @@ __naked int bpf_end_bswap(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (db) r2 = load_acquire((u64 *)(r10 -8))")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("2: (db) r2 = load_acquire((u64 *)(r10 -8)){{ *}}; R2=8 R10=fp0 fp-8=8")
+__msg("4: (0f) r3 += r2 ; R2=8 R3=fp8")
__naked int bpf_load_acquire(void)
{
asm volatile (
@@ -115,7 +101,7 @@ __naked int bpf_load_acquire(void)
"*(u64 *)(r10 - 8) = r1;"
".8byte %[load_acquire_insn];" /* r2 = load_acquire((u64 *)(r10 - 8)); */
"r3 = r10;"
- "r3 += r2;" /* mark_precise */
+ "r3 += r2;"
"r0 = 0;"
"exit;"
:
@@ -126,10 +112,8 @@ __naked int bpf_load_acquire(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r2 = r10")
-__msg("mark_precise: frame0: regs=r1 stack= before 2: (79) r1 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (db) store_release((u64 *)(r10 -8), r1)")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("1: (db) store_release((u64 *)(r10 -8), r1){{ *}}; R1=8 R10=fp0 fp-8=8")
+__msg("4: (0f) r2 += r1 ; R1=8 R2=fp8")
__naked int bpf_store_release(void)
{
asm volatile (
@@ -137,7 +121,7 @@ __naked int bpf_store_release(void)
".8byte %[store_release_insn];" /* store_release((u64 *)(r10 - 8), r1); */
"r1 = *(u64 *)(r10 - 8);"
"r2 = r10;"
- "r2 += r1;" /* mark_precise */
+ "r2 += r1;"
"r0 = 0;"
"exit;"
:
@@ -184,10 +168,6 @@ __success __log_level(2)
__msg("8: (bd) if r2 <= r10 goto pc+3")
__msg("9: (35) if r1 >= 0xffe3fff8 goto pc+0")
__msg("10: (b5) if r2 <= 0x8 goto pc+0")
-__msg("mark_precise: frame1: last_idx 10 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame1: regs=r2 stack= before 9: (35) if r1 >= 0xffe3fff8 goto pc+0")
-__msg("mark_precise: frame1: regs=r2 stack= before 8: (bd) if r2 <= r10 goto pc+3")
-__msg("mark_precise: frame1: regs=r2 stack= before 7: (05) goto pc+0")
__naked void bpf_cond_op_r10(void)
{
asm volatile (
@@ -203,9 +183,6 @@ __success __log_level(2)
__msg("3: (bf) r3 = r10")
__msg("4: (bd) if r3 <= r2 goto pc+1")
__msg("5: (b5) if r2 <= 0x8 goto pc+2")
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bd) if r3 <= r2 goto pc+1")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (bf) r3 = r10")
__naked void bpf_cond_op_not_r10(void)
{
asm volatile (
@@ -223,9 +200,6 @@ SEC("lsm.s/socket_connect")
__success __log_level(2)
__msg("0: (b7) r0 = 1 ; R0=1")
__msg("1: (84) w0 = -w0 ; R0=0xffffffff")
-__msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 1: (84) w0 = -w0")
-__msg("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1")
__naked int bpf_neg_2(void)
{
/*
@@ -258,9 +232,6 @@ SEC("lsm.s/socket_connect")
__success __log_level(2)
__msg("0: (b7) r0 = 1 ; R0=1")
__msg("1: (87) r0 = -r0 ; R0=-1")
-__msg("mark_precise: frame0: last_idx 2 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 1: (87) r0 = -r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 0: (b7) r0 = 1")
__naked int bpf_neg_4(void)
{
/*
@@ -291,11 +262,8 @@ __naked int bpf_neg_5(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_add((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_add((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2 ; R2=8 R3=fp8")
__naked int bpf_atomic_fetch_add_precision(void)
{
asm volatile (
@@ -304,7 +272,7 @@ __naked int bpf_atomic_fetch_add_precision(void)
"r2 = 0;"
".8byte %[fetch_add_insn];" /* r2 = atomic_fetch_add(*(u64 *)(r10 - 8), r2) */
"r3 = r10;"
- "r3 += r2;" /* mark_precise */
+ "r3 += r2;"
"r0 = 0;"
"exit;"
:
@@ -315,11 +283,8 @@ __naked int bpf_atomic_fetch_add_precision(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_xchg((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_xchg((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2 ; R2=8 R3=fp8")
__naked int bpf_atomic_xchg_precision(void)
{
asm volatile (
@@ -328,7 +293,7 @@ __naked int bpf_atomic_xchg_precision(void)
"r2 = 0;"
".8byte %[xchg_insn];" /* r2 = atomic_xchg(*(u64 *)(r10 - 8), r2) */
"r3 = r10;"
- "r3 += r2;" /* mark_precise */
+ "r3 += r2;"
"r0 = 0;"
"exit;"
:
@@ -339,11 +304,8 @@ __naked int bpf_atomic_xchg_precision(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_or((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_or((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2 ; R2=8 R3=fp8")
__naked int bpf_atomic_fetch_or_precision(void)
{
asm volatile (
@@ -352,7 +314,7 @@ __naked int bpf_atomic_fetch_or_precision(void)
"r2 = 0;"
".8byte %[fetch_or_insn];" /* r2 = atomic_fetch_or(*(u64 *)(r10 - 8), r2) */
"r3 = r10;"
- "r3 += r2;" /* mark_precise */
+ "r3 += r2;"
"r0 = 0;"
"exit;"
:
@@ -363,11 +325,8 @@ __naked int bpf_atomic_fetch_or_precision(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_and((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_and((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2 ; R2=8 R3=fp8")
__naked int bpf_atomic_fetch_and_precision(void)
{
asm volatile (
@@ -376,7 +335,7 @@ __naked int bpf_atomic_fetch_and_precision(void)
"r2 = 0;"
".8byte %[fetch_and_insn];" /* r2 = atomic_fetch_and(*(u64 *)(r10 - 8), r2) */
"r3 = r10;"
- "r3 += r2;" /* mark_precise */
+ "r3 += r2;"
"r0 = 0;"
"exit;"
:
@@ -387,11 +346,8 @@ __naked int bpf_atomic_fetch_and_precision(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_xor((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("3: (db) r2 = atomic64_fetch_xor((u64 *)(r10 -8), r2){{ *}}; R2=8 R10=fp0 fp-8=mmmmmmmm")
+__msg("5: (0f) r3 += r2 ; R2=8 R3=fp8")
__naked int bpf_atomic_fetch_xor_precision(void)
{
asm volatile (
@@ -400,7 +356,7 @@ __naked int bpf_atomic_fetch_xor_precision(void)
"r2 = 0;"
".8byte %[fetch_xor_insn];" /* r2 = atomic_fetch_xor(*(u64 *)(r10 - 8), r2) */
"r3 = r10;"
- "r3 += r2;" /* mark_precise */
+ "r3 += r2;"
"r0 = 0;"
"exit;"
:
@@ -411,12 +367,8 @@ __naked int bpf_atomic_fetch_xor_precision(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r3 = r10")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (db) r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r0 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("4: (db) r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r2){{ *}}; R0=8 R2=0 R10=fp0 fp-8=mmmmmmmm")
+__msg("6: (0f) r3 += r0 ; R0=8 R3=fp8")
__naked int bpf_atomic_cmpxchg_precision(void)
{
asm volatile (
@@ -426,7 +378,7 @@ __naked int bpf_atomic_cmpxchg_precision(void)
"r2 = 0;"
".8byte %[cmpxchg_insn];" /* r0 = atomic_cmpxchg(*(u64 *)(r10 - 8), r0, r2) */
"r3 = r10;"
- "r3 += r0;" /* mark_precise */
+ "r3 += r0;"
"r0 = 0;"
"exit;"
:
@@ -435,19 +387,14 @@ __naked int bpf_atomic_cmpxchg_precision(void)
: __clobber_all);
}
-/* Regression test for dual precision: Both the fetched value (r2) and
- * a reread of the same stack slot (r3) are tracked for precision. After
- * the atomic operation, the stack slot is STACK_MISC. Thus, the ldx at
- * insn 4 does NOT set INSN_F_STACK_ACCESS. Precision for the stack slot
- * propagates solely through the atomic fetch's load side (insn 3).
+/* Both the fetched value (r2) and a reread of the same stack slot (r3) are used.
+ * The fetched value is known. After the atomic operation, the stack slot is
+ * STACK_MISC, so what is reread is not.
*/
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r2,r3 stack= before 4: (79) r3 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs=r2 stack= before 3: (db) r2 = atomic64_fetch_add((u64 *)(r10 -8), r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("5: (bf) r4 = r2 ; R2=8 R4=8")
+__msg("6: (0f) r4 += r3 ; R3=scalar() R4=scalar()")
__naked int bpf_atomic_fetch_add_dual_precision(void)
{
asm volatile (
@@ -460,7 +407,7 @@ __naked int bpf_atomic_fetch_add_dual_precision(void)
"r4 += r3;"
"r4 &= 7;"
"r5 = r10;"
- "r5 += r4;" /* mark_precise */
+ "r5 += r4;"
"r0 = 0;"
"exit;"
:
@@ -471,12 +418,8 @@ __naked int bpf_atomic_fetch_add_dual_precision(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r0,r3 stack= before 5: (79) r3 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (db) r0 = atomic64_cmpxchg((u64 *)(r10 -8), r0, r2)")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (b7) r2 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 2: (b7) r0 = 8")
-__msg("mark_precise: frame0: regs= stack=-8 before 1: (7b) *(u64 *)(r10 -8) = r1")
-__msg("mark_precise: frame0: regs=r1 stack= before 0: (b7) r1 = 8")
+__msg("6: (bf) r4 = r0 ; R0=8 R4=8")
+__msg("7: (0f) r4 += r3 ; R3=scalar() R4=scalar()")
__naked int bpf_atomic_cmpxchg_dual_precision(void)
{
asm volatile (
@@ -490,7 +433,7 @@ __naked int bpf_atomic_cmpxchg_dual_precision(void)
"r4 += r3;"
"r4 &= 7;"
"r5 = r10;"
- "r5 += r4;" /* mark_precise */
+ "r5 += r4;"
"r0 = 0;"
"exit;"
:
@@ -501,9 +444,7 @@ __naked int bpf_atomic_cmpxchg_dual_precision(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r1 stack= before 10: (57) r1 &= 7")
-__msg("mark_precise: frame0: regs=r1 stack= before 9: (db) r1 = atomic64_fetch_add((u64 *)(r0 +0), r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("9: (db) r1 = atomic64_fetch_add((u64 *)(r0 +0), r1){{ *}}; R0=map_value({{.*}}) R1=scalar()")
__naked int bpf_atomic_fetch_add_map_precision(void)
{
asm volatile (
@@ -518,7 +459,7 @@ __naked int bpf_atomic_fetch_add_map_precision(void)
".8byte %[fetch_add_insn];" /* r1 = atomic_fetch_add(*(u64 *)(r0 + 0), r1) */
"r1 &= 7;"
"r2 = r10;"
- "r2 += r1;" /* mark_precise */
+ "r2 += r1;"
"1: r0 = 0;"
"exit;"
:
@@ -531,9 +472,7 @@ __naked int bpf_atomic_fetch_add_map_precision(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (57) r0 &= 7")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (db) r0 = atomic64_cmpxchg((u64 *)(r6 +0), r0, r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("11: (db) r0 = atomic64_cmpxchg((u64 *)(r6 +0), r0, r1){{ *}}; R0=scalar() R1=0")
__naked int bpf_atomic_cmpxchg_map_precision(void)
{
asm volatile (
@@ -550,7 +489,7 @@ __naked int bpf_atomic_cmpxchg_map_precision(void)
".8byte %[cmpxchg_insn];" /* r0 = atomic_cmpxchg(*(u64 *)(r6 + 0), r0, r1) */
"r0 &= 7;"
"r2 = r10;"
- "r2 += r0;" /* mark_precise */
+ "r2 += r0;"
"1: r0 = 0;"
"exit;"
:
@@ -563,9 +502,7 @@ __naked int bpf_atomic_cmpxchg_map_precision(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r1 stack= before 10: (57) r1 &= 7")
-__msg("mark_precise: frame0: regs=r1 stack= before 9: (c3) r1 = atomic_fetch_add((u32 *)(r0 +0), r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("9: (c3) r1 = atomic_fetch_add((u32 *)(r0 +0), r1){{ *}}; R0=map_value({{.*}}) R1=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))")
__naked int bpf_atomic_fetch_add_32bit_precision(void)
{
asm volatile (
@@ -580,7 +517,7 @@ __naked int bpf_atomic_fetch_add_32bit_precision(void)
".8byte %[fetch_add_insn];" /* r1 = atomic_fetch_add(*(u32 *)(r0 + 0), r1) */
"r1 &= 7;"
"r2 = r10;"
- "r2 += r1;" /* mark_precise */
+ "r2 += r1;"
"1: r0 = 0;"
"exit;"
:
@@ -593,9 +530,7 @@ __naked int bpf_atomic_fetch_add_32bit_precision(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (57) r0 &= 7")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (c3) r0 = atomic_cmpxchg((u32 *)(r6 +0), r0, r1)")
-__not_msg("falling back to forcing all scalars precise")
+__msg("11: (c3) r0 = atomic_cmpxchg((u32 *)(r6 +0), r0, r1){{ *}}; R0=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R1=0")
__naked int bpf_atomic_cmpxchg_32bit_precision(void)
{
asm volatile (
@@ -612,7 +547,7 @@ __naked int bpf_atomic_cmpxchg_32bit_precision(void)
".8byte %[cmpxchg_insn];" /* r0 = atomic_cmpxchg(*(u32 *)(r6 + 0), r0, r1) */
"r0 &= 7;"
"r2 = r10;"
- "r2 += r0;" /* mark_precise */
+ "r2 += r0;"
"1: r0 = 0;"
"exit;"
:
@@ -626,11 +561,9 @@ __naked int bpf_atomic_cmpxchg_32bit_precision(void)
/*
* Verification takes two paths: with r1 being scalar zero on path (1)
* and with r1 being some other scalar on path (2).
- * Check that the verifier does not use checkpoints created
- * on path (1) to prune path (2).
+ * Check that the state of path (1) does not stand for path (2).
*/
SEC("?tc")
-__flag(BPF_F_TEST_STATE_FREQ)
__failure __msg("R1 type=scalar expected=fp")
__naked int null_mem_arg_zero_size(void)
{
@@ -667,11 +600,9 @@ __weak int subprog_mem_arg(int *p)
/*
* Verification takes two paths: with r1 being scalar zero on path (1)
* and with r1 being some other scalar on path (2).
- * Check that the verifier does not use checkpoints created
- * on path (1) to prune path (2).
+ * Check that the state of path (1) does not stand for path (2).
*/
SEC("?raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
__failure __msg("R1 type=scalar expected=fp")
__naked int null_mem_arg_global_subprog(void)
{
@@ -689,9 +620,8 @@ __naked int null_mem_arg_global_subprog(void)
: __clobber_all);
}
-/* Same as above, check that path with r3 == 0 does not prune the path with r3 != 0 */
+/* Same as above, check that path with r3 == 0 does not stand for the path with r3 != 0 */
SEC("?tc")
-__flag(BPF_F_TEST_STATE_FREQ)
__failure __msg("R3 type=scalar expected=fp")
int null_kfunc_arg_dynptr_slice(struct __sk_buff *skb)
{
diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
index dc0c7034c04f..3f9a65ff134e 100644
--- a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
+++ b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
@@ -39,31 +39,20 @@ static unsigned long callback_subprog()
SEC("?raw_tp")
__success __log_level(2)
-__msg("7: (0f) r1 += r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 6: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 10: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 4: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("7: (0f) r1 += r0 ; R0=12 R1=map_value(map=.data.vals,ks=4,vs=16,imm=12)")
__naked int subprog_result_precise(void)
{
asm volatile (
"r6 = 3;"
- /* pass r6 through r1 into subprog to get it back as r0;
- * this whole chain will have to be marked as precise later
- */
+ /* pass r6 through r1 into subprog to get it back as r0 */
"r1 = r6;"
"call identity_subprog;"
/* now use subprog's returned value (which is a
- * r6 -> r1 -> r0 chain), as index into vals array, forcing
- * all of that to be known precisely
+ * r6 -> r1 -> r0 chain), as index into vals array
*/
"r0 *= 4;"
"r1 = %[vals];"
- /* here r0->r1->r6 chain is forced to be precise and has to be
- * propagated back to the beginning, including through the
+ /* here r0 has to be known: the constant went through the
* subprog call
*/
"r1 += r0;"
@@ -98,26 +87,18 @@ static unsigned long sneaky_fp_leaking_subprog()
SEC("?raw_tp")
__success __log_level(2)
-__msg("6: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 6 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (57) r0 &= 3")
-__msg("mark_precise: frame0: regs=r0 stack= before 10: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 9: (bf) r0 = (s8)r10")
-__msg("7: R0=scalar")
+__msg("3: R0=scalar() R6=")
+__msg("6: (0f) r1 += r0 ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
__naked int fp_precise_subprog_result(void)
{
asm volatile (
"call fp_leaking_subprog;"
/* use subprog's returned value (which is derived from r10=fp
- * register), as index into vals array, forcing all of that to
- * be known precisely
+ * register, so it's unknown), as index into vals array
*/
"r0 &= 3;"
"r0 *= 4;"
"r1 = %[vals];"
- /* force precision marking */
"r1 += r0;"
"r0 = *(u32 *)(r1 + 0);"
"exit;"
@@ -129,31 +110,18 @@ __naked int fp_precise_subprog_result(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("6: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 6 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (57) r0 &= 3")
-__msg("mark_precise: frame0: regs=r0 stack= before 11: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 10: (bf) r0 = (s8)r1")
-/* here r1 is marked precise, even though it's fp register, but that's fine
- * because by the time we get out of subprogram it has to be derived from r10
- * anyways, at which point we'll break precision chain
- */
-__msg("mark_precise: frame1: regs=r1 stack= before 9: (bf) r1 = r10")
-__msg("7: R0=scalar")
+__msg("3: R0=scalar() R6=")
+__msg("6: (0f) r1 += r0 ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
__naked int sneaky_fp_precise_subprog_result(void)
{
asm volatile (
"call sneaky_fp_leaking_subprog;"
/* use subprog's returned value (which is derived from r10=fp
- * register), as index into vals array, forcing all of that to
- * be known precisely
+ * register, so it's unknown), as index into vals array
*/
"r0 &= 3;"
"r0 *= 4;"
"r1 = %[vals];"
- /* force precision marking */
"r1 += r0;"
"r0 = *(u32 *)(r1 + 0);"
"exit;"
@@ -165,35 +133,25 @@ __naked int sneaky_fp_precise_subprog_result(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("9: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 9 first_idx 0")
-__msg("mark_precise: frame0: regs=r0 stack= before 8: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r0 stack= before 7: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (a5) if r0 < 0x4 goto pc+1")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (85) call pc+7")
+__msg("9: (0f) r1 += r0 ; R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
__naked int global_subprog_result_precise(void)
{
asm volatile (
"r6 = 3;"
/* pass r6 through r1 into subprog to get it back as r0;
- * given global_identity_subprog is global, precision won't
- * propagate all the way back to r6
+ * given global_identity_subprog is global, nothing is known
+ * about what it returns
*/
"r1 = r6;"
"call global_identity_subprog;"
/* now use subprog's returned value (which is unknown now, so
- * we need to clamp it), as index into vals array, forcing r0
- * to be marked precise (with no effect on r6, though)
+ * we need to clamp it), as index into vals array
*/
"if r0 < %[vals_arr_sz] goto 1f;"
"r0 = %[vals_arr_sz] - 1;"
"1:"
"r0 *= 4;"
"r1 = %[vals];"
- /* here r0 is forced to be precise and has to be
- * propagated back to the global subprog call, but it
- * shouldn't go all the way to mark r6 as precise
- */
"r1 += r0;"
"r0 = *(u32 *)(r1 + 0);"
"exit;"
@@ -213,11 +171,9 @@ static unsigned long loop_callback_bad()
"if r0 s> 1000 goto 1f;"
"r0 = 0;"
"1:"
- "goto +0;" /* checkpoint */
+ "goto +0;"
/* bpf_loop() expects [0, 1] values, so branch above skipping
- * r0 = 0; should lead to a failure, but if exit instruction
- * doesn't enforce r0's precision, this callback will be
- * successfully verified
+ * r0 = 0; should lead to a failure
*/
"exit;"
:
@@ -228,14 +184,8 @@ static unsigned long loop_callback_bad()
SEC("?raw_tp")
__failure __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-/* check that fallthrough code path marks r0 as precise */
-__msg("mark_precise: frame1: regs=r0 stack= before 11: (b7) r0 = 0")
-/* check that we have branch code path doing its own validation */
-__msg("from 10 to 12: frame1: R0=scalar(smin=umin=1001")
-/* check that branch code path marks r0 as precise, before failing */
-__msg("mark_precise: frame1: regs=r0 stack= before 9: (85) call bpf_get_prandom_u32#7")
-__msg("At callback return the register R0 has smin=1001 should have been in [0, 1]")
+/* 0 of the fallthrough code path doesn't stand for what the branch code path has */
+__msg("At callback return the register R0 has {{.*}} should have been in [0, 1]")
__naked int callback_precise_return_fail(void)
{
asm volatile (
@@ -256,41 +206,15 @@ __naked int callback_precise_return_fail(void)
SEC("?raw_tp")
__success __log_level(2)
-/* First simulated path does not include callback body,
- * r1 and r4 are always precise for bpf_loop() calls.
- */
__msg("9: (85) call bpf_loop#181")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: parent state regs=r4 stack=:")
-__msg("mark_precise: frame0: last_idx 8 first_idx 0 subseq_idx 9")
-__msg("mark_precise: frame0: regs=r4 stack= before 8: (b7) r4 = 0")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: parent state regs=r1 stack=:")
-__msg("mark_precise: frame0: last_idx 8 first_idx 0 subseq_idx 9")
-__msg("mark_precise: frame0: regs=r1 stack= before 8: (b7) r4 = 0")
-__msg("mark_precise: frame0: regs=r1 stack= before 7: (b7) r3 = 0")
-__msg("mark_precise: frame0: regs=r1 stack= before 6: (bf) r2 = r8")
-__msg("mark_precise: frame0: regs=r1 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3")
-/* r6 precision propagation */
-__msg("14: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 14 first_idx 9")
-__msg("mark_precise: frame0: regs=r6 stack= before 13: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 12: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (25) if r6 > 0x3 goto pc+4")
-__msg("mark_precise: frame0: regs=r0,r6 stack= before 10: (bf) r6 = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 9: (85) call bpf_loop")
-/* State entering callback body popped from states stack */
-__msg("from 9 to 17: frame1:")
__msg("17: frame1: R10=fp0 cb")
__msg("17: (b7) r0 = 0")
__msg("18: (95) exit")
__msg("returning from callee:")
__msg("to caller at 9:")
-__msg("frame 0: propagating r1,r3,r4")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r3,r4 stack= before 18: (95) exit")
-__msg("from 18 to 9: safe")
+/* the callback changes nothing for the caller */
+__msg("from 18 to 9: within")
+__msg("14: (0f) r1 += r6 ; R1=map_value(map=.data.vals,ks=4,vs=16,smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4) R6=scalar(smin=smin32=0,smax=umax=smax32=umax32=12,var_off=(0x0; 0xc),step=0+4)")
__naked int callback_result_precise(void)
{
asm volatile (
@@ -309,9 +233,7 @@ __naked int callback_result_precise(void)
"if r6 > 3 goto 1f;"
"r6 *= 4;"
"r1 = %[vals];"
- /* here r6 is forced to be precise and has to be propagated
- * back to the bpf_loop() call, but not beyond
- */
+ /* here the range of r6 has to be known */
"r1 += r6;"
"r0 = *(u32 *)(r1 + 0);"
"1:"
@@ -326,15 +248,7 @@ __naked int callback_result_precise(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("7: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 7 first_idx 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 10: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs= stack= before 4: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("7: (0f) r1 += r6 ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
__naked int parent_callee_saved_reg_precise(void)
{
asm volatile (
@@ -348,9 +262,7 @@ __naked int parent_callee_saved_reg_precise(void)
"r6 *= 4;"
"r1 = %[vals];"
- /* here r6 is forced to be precise and has to be propagated
- * back to the beginning, handling (and ignoring) subprog call
- */
+ /* here r6 has to be known: the subprog call doesn't change it */
"r1 += r6;"
"r0 = *(u32 *)(r1 + 0);"
"exit;"
@@ -362,13 +274,7 @@ __naked int parent_callee_saved_reg_precise(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("7: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 7 first_idx 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("7: (0f) r1 += r6 ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
__naked int parent_callee_saved_reg_precise_global(void)
{
asm volatile (
@@ -382,9 +288,7 @@ __naked int parent_callee_saved_reg_precise_global(void)
"r6 *= 4;"
"r1 = %[vals];"
- /* here r6 is forced to be precise and has to be propagated
- * back to the beginning, handling (and ignoring) subprog call
- */
+ /* here r6 has to be known: the subprog call doesn't change it */
"r1 += r6;"
"r0 = *(u32 *)(r1 + 0);"
"exit;"
@@ -396,36 +300,14 @@ __naked int parent_callee_saved_reg_precise_global(void)
SEC("?raw_tp")
__success __log_level(2)
-/* First simulated path does not include callback body */
-__msg("12: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 12 first_idx 9")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 10: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 9: (85) call bpf_loop")
-__msg("mark_precise: frame0: parent state regs=r6 stack=:")
-__msg("mark_precise: frame0: last_idx 8 first_idx 0 subseq_idx 9")
-__msg("mark_precise: frame0: regs=r6 stack= before 8: (b7) r4 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 7: (b7) r3 = 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (bf) r2 = r8")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (b7) r1 = 1")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3")
-/* State entering callback body popped from states stack */
-__msg("from 9 to 15: frame1:")
+__msg("12: (0f) r1 += r6 ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
__msg("15: frame1: R10=fp0 cb")
__msg("15: (b7) r0 = 0")
__msg("16: (95) exit")
__msg("returning from callee:")
__msg("to caller at 9:")
-/* r1, r4 are always precise for bpf_loop(),
- * r6 was marked before backtracking to callback body.
- */
-__msg("frame 0: propagating r1,r3,r4,r6")
-__msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r3,r4,r6 stack= before 16: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 15: (b7) r0 = 0")
-__msg("mark_precise: frame1: regs= stack= before 9: (85) call bpf_loop")
-__msg("mark_precise: frame0: parent state regs= stack=:")
-__msg("from 16 to 9: safe")
+/* r6 is what it was before the callback */
+__msg("from 16 to 9: within")
__naked int parent_callee_saved_reg_precise_with_callback(void)
{
asm volatile (
@@ -442,9 +324,7 @@ __naked int parent_callee_saved_reg_precise_with_callback(void)
"r6 *= 4;"
"r1 = %[vals];"
- /* here r6 is forced to be precise and has to be propagated
- * back to the beginning, handling (and ignoring) callback call
- */
+ /* here r6 has to be known: the callback doesn't change it */
"r1 += r6;"
"r0 = *(u32 *)(r1 + 0);"
"exit;"
@@ -458,19 +338,7 @@ __naked int parent_callee_saved_reg_precise_with_callback(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("9: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 9 first_idx 6")
-__msg("mark_precise: frame0: regs=r6 stack= before 8: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 7: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (79) r6 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: parent state regs= stack=-8:")
-__msg("mark_precise: frame0: last_idx 13 first_idx 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 13: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 12: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs= stack= before 5: (85) call pc+6")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (7b) *(u64 *)(r10 -8) = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("9: (0f) r1 += r6 ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
__naked int parent_stack_slot_precise(void)
{
asm volatile (
@@ -492,9 +360,7 @@ __naked int parent_stack_slot_precise(void)
"r6 *= 4;"
"r1 = %[vals];"
- /* here r6 is forced to be precise and has to be propagated
- * back to the beginning, handling (and ignoring) subprog call
- */
+ /* here r6 has to be known: the subprog call doesn't change it */
"r1 += r6;"
"r0 = *(u32 *)(r1 + 0);"
"exit;"
@@ -506,15 +372,7 @@ __naked int parent_stack_slot_precise(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("9: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 9 first_idx 0")
-__msg("mark_precise: frame0: regs=r6 stack= before 8: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 7: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (79) r6 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (85) call pc+6")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r1 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (7b) *(u64 *)(r10 -8) = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("9: (0f) r1 += r6 ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
__naked int parent_stack_slot_precise_global(void)
{
asm volatile (
@@ -536,9 +394,7 @@ __naked int parent_stack_slot_precise_global(void)
"r6 *= 4;"
"r1 = %[vals];"
- /* here r6 is forced to be precise and has to be propagated
- * back to the beginning, handling (and ignoring) subprog call
- */
+ /* here r6 has to be known: the subprog call doesn't change it */
"r1 += r6;"
"r0 = *(u32 *)(r1 + 0);"
"exit;"
@@ -550,38 +406,14 @@ __naked int parent_stack_slot_precise_global(void)
SEC("?raw_tp")
__success __log_level(2)
-/* First simulated path does not include callback body */
-__msg("14: (0f) r1 += r6")
-__msg("mark_precise: frame0: last_idx 14 first_idx 10")
-__msg("mark_precise: frame0: regs=r6 stack= before 13: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r6 stack= before 12: (27) r6 *= 4")
-__msg("mark_precise: frame0: regs=r6 stack= before 11: (79) r6 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 10: (85) call bpf_loop")
-__msg("mark_precise: frame0: parent state regs= stack=-8:")
-__msg("mark_precise: frame0: last_idx 9 first_idx 0 subseq_idx 10")
-__msg("mark_precise: frame0: regs= stack=-8 before 9: (b7) r4 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 8: (b7) r3 = 0")
-__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r2 = r8")
-__msg("mark_precise: frame0: regs= stack=-8 before 6: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -8) = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (b7) r6 = 3")
-/* State entering callback body popped from states stack */
-__msg("from 10 to 17: frame1:")
+__msg("14: (0f) r1 += r6 ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=12) R6=12")
__msg("17: frame1: R10=fp0 cb")
__msg("17: (b7) r0 = 0")
__msg("18: (95) exit")
__msg("returning from callee:")
__msg("to caller at 10:")
-/* r1, r4 are always precise for bpf_loop(),
- * fp-8 was marked before backtracking to callback body.
- */
-__msg("frame 0: propagating r1,r3,r4,fp-8")
-__msg("mark_precise: frame0: last_idx 10 first_idx 10 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r1,r3,r4 stack=-8 before 18: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 17: (b7) r0 = 0")
-__msg("mark_precise: frame1: regs= stack= before 10: (85) call bpf_loop#181")
-__msg("mark_precise: frame0: parent state regs= stack=:")
-__msg("from 18 to 10: safe")
+/* fp-8 is what it was before the callback */
+__msg("from 18 to 10: within")
__naked int parent_stack_slot_precise_with_callback(void)
{
asm volatile (
@@ -604,9 +436,7 @@ __naked int parent_stack_slot_precise_with_callback(void)
"r6 *= 4;"
"r1 = %[vals];"
- /* here r6 is forced to be precise and has to be propagated
- * back to the beginning, handling (and ignoring) subprog call
- */
+ /* here r6 has to be known: the subprog call doesn't change it */
"r1 += r6;"
"r0 = *(u32 *)(r1 + 0);"
"exit;"
@@ -621,27 +451,18 @@ __naked int parent_stack_slot_precise_with_callback(void)
__noinline __used
static __u64 subprog_with_precise_arg(__u64 x)
{
- return vals[x]; /* x is forced to be precise */
+ return vals[x]; /* x has to be known */
}
SEC("?raw_tp")
__success __log_level(2)
-__msg("8: (0f) r2 += r1")
-__msg("mark_precise: frame1: last_idx 8 first_idx 0")
-__msg("mark_precise: frame1: regs=r1 stack= before 6: (18) r2 = ")
-__msg("mark_precise: frame1: regs=r1 stack= before 5: (67) r1 <<= 2")
-__msg("mark_precise: frame1: regs=r1 stack= before 2: (85) call pc+2")
-__msg("mark_precise: frame0: regs=r1 stack= before 1: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 0: (b7) r6 = 3")
+__msg("8: (0f) r2 += r1 ; R1=12 R2=map_value(map=.data.vals,ks=4,vs=16,imm=12)")
__naked int subprog_arg_precise(void)
{
asm volatile (
"r6 = 3;"
"r1 = r6;"
- /* subprog_with_precise_arg expects its argument to be
- * precise, so r1->r6 will be marked precise from inside the
- * subprog
- */
+ /* subprog_with_precise_arg needs to know its argument */
"call subprog_with_precise_arg;"
"r0 += r6;"
"exit;"
@@ -666,7 +487,7 @@ static __u64 subprog_spill_reg_precise(void)
* verifier log output)
*/
"*(u64 *)(r10 - 16) = r2;"
- /* use both spills as return result to propagete precision everywhere */
+ /* use both spills as return result */
"r0 = *(u64 *)(r10 - 16);"
"r2 = *(u64 *)(r1 + 0);"
"r0 += r2;"
@@ -676,32 +497,14 @@ static __u64 subprog_spill_reg_precise(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("10: (0f) r1 += r7")
-__msg("mark_precise: frame0: last_idx 10 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r7 stack= before 9: (bf) r1 = r8")
-__msg("mark_precise: frame0: regs=r7 stack= before 8: (27) r7 *= 4")
-__msg("mark_precise: frame0: regs=r7 stack= before 7: (79) r7 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: parent state regs= stack=-8: R8=map_value(map=.data.vals,ks=4,vs=16) R10=fp0 fp-8=P1")
-__msg("mark_precise: frame0: last_idx 18 first_idx 0 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-8 before 18: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 17: (0f) r0 += r2")
-__msg("mark_precise: frame1: regs= stack= before 16: (79) r2 = *(u64 *)(r1 +0)")
-__msg("mark_precise: frame1: regs= stack= before 15: (79) r0 = *(u64 *)(r10 -16)")
-__msg("mark_precise: frame1: regs= stack= before 14: (7b) *(u64 *)(r10 -16) = r2")
-__msg("mark_precise: frame1: regs= stack= before 13: (7b) *(u64 *)(r1 +0) = r2")
-__msg("mark_precise: frame1: regs=r2 stack= before 6: (85) call pc+6")
-__msg("mark_precise: frame0: regs=r2 stack= before 5: (bf) r2 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 1")
+__msg("10: (0f) r1 += r7 ; R1=map_value(map=.data.vals,ks=4,vs=16,imm=4) R7=4")
__naked int subprog_spill_into_parent_stack_slot_precise(void)
{
asm volatile (
"r6 = 1;"
/* pass pointer to stack slot and r6 to subprog;
- * r6 will be marked precise and spilled into fp-8 slot, which
- * also should be marked precise
+ * r6 will be spilled into fp-8 slot
*/
"r1 = r10;"
"r1 += -8;"
@@ -716,8 +519,8 @@ __naked int subprog_spill_into_parent_stack_slot_precise(void)
"r7 *= 4;"
"r1 = %[vals];"
- /* here r7 is forced to be precise and has to be propagated
- * back to the beginning, handling subprog call and logic
+ /* here r7 has to be known: it's what the subprog spilled into
+ * the stack slot of its caller
*/
"r1 += r7;"
"r0 = *(u32 *)(r1 + 0);"
@@ -730,32 +533,12 @@ __naked int subprog_spill_into_parent_stack_slot_precise(void)
SEC("?raw_tp")
__success __log_level(2)
-__msg("17: (0f) r1 += r0")
-__msg("mark_precise: frame0: last_idx 17 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r0 stack= before 16: (bf) r1 = r7")
-__msg("mark_precise: frame0: regs=r0 stack= before 15: (27) r0 *= 4")
-__msg("mark_precise: frame0: regs=r0 stack= before 14: (79) r0 = *(u64 *)(r10 -16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 13: (7b) *(u64 *)(r7 -8) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (79) r0 = *(u64 *)(r8 +16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 11: (7b) *(u64 *)(r8 +16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 10: (79) r0 = *(u64 *)(r7 -8)")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 8: (07) r8 += -32")
-__msg("mark_precise: frame0: regs=r0 stack= before 7: (bf) r8 = r10")
-__msg("mark_precise: frame0: regs=r0 stack= before 6: (07) r7 += -8")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r7 = r10")
-__msg("mark_precise: frame0: regs=r0 stack= before 21: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 20: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 4: (85) call pc+15")
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 1")
+__msg("17: (0f) r1 += r0 ; R0=4 R1=fp-4")
__naked int stack_slot_aliases_precision(void)
{
asm volatile (
"r6 = 1;"
- /* pass r6 through r1 into subprog to get it back as r0;
- * this whole chain will have to be marked as precise later
- */
+ /* pass r6 through r1 into subprog to get it back as r0 */
"r1 = r6;"
"call identity_subprog;"
/* let's setup two registers that are aliased to r10 */
@@ -777,11 +560,10 @@ __naked int stack_slot_aliases_precision(void)
"r0 = *(u64 *)(r8 + 16);"
"*(u64 *)(r7 - 8) = r0;"
"r0 = *(u64 *)(r10 - 16);"
- /* get ready to use r0 as an index into array to force precision */
+ /* get ready to use r0 as an index into array */
"r0 *= 4;"
"r1 = %[vals];"
- /* here r0->r1->r6 chain is forced to be precise and has to be
- * propagated back to the beginning, including through the
+ /* here r0 has to be known: the constant went through the
* subprog call and all the stack spills and loads
*/
"r1 += r0;"
@@ -820,15 +602,11 @@ static unsigned long identity_tail_call(void)
SEC("?raw_tp")
__failure __log_level(2)
__msg("13: (85) call bpf_tail_call#12")
-__msg("mark_precise: frame1: last_idx 13 first_idx 0 subseq_idx -1 ")
__msg("returning from callee:")
__msg("frame1: R0=scalar() R6=3 R10=fp0")
__msg("to caller at 4:")
__msg("R0=scalar() R6=map_value(map=.data.vals,ks=4,vs=16) R10=fp0")
__msg("6: (0f) r1 += r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (27) r0 *= 4")
-__msg("mark_precise: frame0: parent state regs=r0 stack=: R0=Pscalar() R6=map_value(map=.data.vals,ks=4,vs=16) R10=fp0")
__msg("math between map_value pointer and register with unbounded min value is not allowed")
__naked int subprog_result_tail_call(void)
{
@@ -860,36 +638,22 @@ static int ld_abs_subprog(void)
}
/*
- * Buggy verifier did not properly backtrack early subprogram exit
- * modelled for BPF_LD | BPF_ABS instruction, causing a segfault.
+ * BPF_LD | BPF_ABS instruction is modelled as an early exit of the subprogram.
+ * r7 is known on both paths.
*/
SEC("socket")
__success
__log_level(2)
-/* early exit path */
-__msg("3: (0f) r1 += r7")
-__msg("mark_precise: frame0: regs=r7 stack= before 2: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r7 stack= before 9: (20) r0 = *(u32 *)skb[0]")
-__msg("mark_precise: frame1: regs= stack= before 8: (bf) r7 = r1")
-__msg("mark_precise: frame1: regs= stack= before 7: (bf) r6 = r1")
-__msg("mark_precise: frame1: regs= stack= before 1: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r7 stack= before 0: (b7) r7 = -8")
-/* fallthrough path */
-__msg("3: (0f) r1 += r7")
-__msg("mark_precise: frame0: regs=r7 stack= before 2: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r7 stack= before 10: (95) exit")
-__msg("mark_precise: frame1: regs= stack= before 9: (20) r0 = *(u32 *)skb[0]")
-__msg("mark_precise: frame1: regs= stack= before 8: (bf) r7 = r1")
-__msg("mark_precise: frame1: regs= stack= before 7: (bf) r6 = r1")
-__msg("mark_precise: frame1: regs= stack= before 1: (85) call pc+5")
-__msg("mark_precise: frame0: regs=r7 stack= before 0: (b7) r7 = -8")
+/* the early exit and the exit of the subprogram give the same state */
+__msg("from 10 to 2: within")
+__msg("3: (0f) r1 += r7 ; R1=fp-8 R7=-8")
__naked int ld_abs_backtrack_both_paths(void)
{
asm volatile (
"r7 = -8;"
"call ld_abs_subprog;"
"r1 = r10;"
- "r1 += r7;" /* mark r7 as precise */
+ "r1 += r7;"
"*(u64 *)(r1 + 0) = 0;"
"r0 = 0;"
"exit;"
diff --git a/tools/testing/selftests/bpf/verifier/precise.c b/tools/testing/selftests/bpf/verifier/precise.c
index a9242103dc47..de9d35ed8a4b 100644
--- a/tools/testing/selftests/bpf/verifier/precise.c
+++ b/tools/testing/selftests/bpf/verifier/precise.c
@@ -36,26 +36,7 @@
},
.prog_type = BPF_PROG_TYPE_TRACEPOINT,
.fixup_map_array_48b = { 1 },
- .result = VERBOSE_ACCEPT,
- .errstr =
- "mark_precise: frame0: last_idx 26 first_idx 20\
- mark_precise: frame0: regs=r2 stack= before 25\
- mark_precise: frame0: regs=r2 stack= before 24\
- mark_precise: frame0: regs=r2 stack= before 23\
- mark_precise: frame0: regs=r2 stack= before 22\
- mark_precise: frame0: regs=r2 stack= before 20\
- mark_precise: frame0: parent state regs=r2 stack=:\
- mark_precise: frame0: last_idx 19 first_idx 10\
- mark_precise: frame0: regs=r2 stack= before 19\
- mark_precise: frame0: regs=r9 stack= before 18\
- mark_precise: frame0: regs=r8,r9 stack= before 17\
- mark_precise: frame0: regs=r0,r9 stack= before 15\
- mark_precise: frame0: regs=r0,r9 stack= before 14\
- mark_precise: frame0: regs=r9 stack= before 13\
- mark_precise: frame0: regs=r9 stack= before 12\
- mark_precise: frame0: regs=r9 stack= before 11\
- mark_precise: frame0: regs=r9 stack= before 10\
- mark_precise: frame0: parent state regs= stack=:",
+ .result = ACCEPT,
},
{
"precise: test 2",
@@ -95,24 +76,8 @@
},
.prog_type = BPF_PROG_TYPE_TRACEPOINT,
.fixup_map_array_48b = { 1 },
- .result = VERBOSE_ACCEPT,
+ .result = ACCEPT,
.flags = BPF_F_TEST_STATE_FREQ,
- .errstr =
- "26: (85) call bpf_probe_read_kernel#113\
- mark_precise: frame0: last_idx 26 first_idx 22\
- mark_precise: frame0: regs=r2 stack= before 25\
- mark_precise: frame0: regs=r2 stack= before 24\
- mark_precise: frame0: regs=r2 stack= before 23\
- mark_precise: frame0: regs=r2 stack= before 22\
- mark_precise: frame0: parent state regs=r2 stack=:\
- mark_precise: frame0: last_idx 20 first_idx 20\
- mark_precise: frame0: regs=r2 stack= before 20\
- mark_precise: frame0: parent state regs=r2 stack=:\
- mark_precise: frame0: last_idx 19 first_idx 17\
- mark_precise: frame0: regs=r2 stack= before 19\
- mark_precise: frame0: regs=r9 stack= before 18\
- mark_precise: frame0: regs=r8,r9 stack= before 17\
- mark_precise: frame0: parent state regs= stack=:",
},
{
"precise: cross frame pruning",
@@ -153,17 +118,7 @@
},
.prog_type = BPF_PROG_TYPE_XDP,
.flags = BPF_F_TEST_STATE_FREQ,
- .errstr = "mark_precise: frame0: last_idx 5 first_idx 5\
- mark_precise: frame0: parent state regs=r4 stack=:\
- mark_precise: frame0: last_idx 4 first_idx 2\
- mark_precise: frame0: regs=r4 stack= before 4\
- mark_precise: frame0: regs=r4 stack= before 3\
- mark_precise: frame0: last_idx 5 first_idx 5\
- mark_precise: frame0: parent state regs=r0 stack=:\
- mark_precise: frame0: last_idx 4 first_idx 2\
- mark_precise: frame0: regs=r0 stack= before 4\
- 5: R0=-1 R4=0",
- .result = VERBOSE_ACCEPT,
+ .result = ACCEPT,
.retval = -1,
},
{
@@ -182,23 +137,7 @@
},
.prog_type = BPF_PROG_TYPE_XDP,
.flags = BPF_F_TEST_STATE_FREQ,
- .errstr = "mark_precise: frame0: last_idx 7 first_idx 7\
- mark_precise: frame0: parent state regs=r4 stack=:\
- mark_precise: frame0: last_idx 6 first_idx 4\
- mark_precise: frame0: regs=r4 stack= before 6: (b7) r0 = -1\
- mark_precise: frame0: regs=r4 stack= before 5: (79) r4 = *(u64 *)(r10 -8)\
- mark_precise: frame0: regs= stack=-8 before 4: (7b) *(u64 *)(r3 -8) = r0\
- mark_precise: frame0: parent state regs=r0 stack=:\
- mark_precise: frame0: last_idx 3 first_idx 3\
- mark_precise: frame0: regs=r0 stack= before 3: (55) if r3 != 0x7b goto pc+0\
- mark_precise: frame0: regs=r0 stack= before 2: (bf) r3 = r10\
- mark_precise: frame0: regs=r0 stack= before 1: (57) r0 &= 255\
- mark_precise: frame0: parent state regs=r0 stack=:\
- mark_precise: frame0: last_idx 0 first_idx 0\
- mark_precise: frame0: regs=r0 stack= before 0: (85) call bpf_get_prandom_u32#7\
- mark_precise: frame0: last_idx 7 first_idx 7\
- mark_precise: frame0: parent state regs= stack=:",
- .result = VERBOSE_ACCEPT,
+ .result = ACCEPT,
.retval = -1,
},
{
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 70/81] selftests/bpf: Adjust tests of precision tracking
2026-10-26 2:59 ` [PATCH bpf 70/81] selftests/bpf: Adjust tests of precision tracking Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 3932 bytes --]
> selftests/bpf: Adjust tests of precision tracking
TL;DR:
- Test issue: stack_slot_aliases_precision may index the stack (R1=fp-4)
instead of vals, as r7/r8 are missing from clobbers
- Nit: comments still describe the removed precision backtracking
> diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
> index dc0c7034c04f..3f9a65ff134e 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c
[ ... ]
> @@ -604,9 +436,7 @@ __naked int parent_stack_slot_precise_with_callback(void)
>
> "r6 *= 4;"
> "r1 = %[vals];"
> - /* here r6 is forced to be precise and has to be propagated
> - * back to the beginning, handling (and ignoring) subprog call
> - */
> + /* here r6 has to be known: the subprog call doesn't change it */
this isn't a bug, but the new comment here says "the subprog call doesn't
change it" in a prog that calls bpf_loop() with a callback, while the
register variant (parent_callee_saved_reg_precise_with_callback) says "the
callback doesn't change it".
The comments next to the rewritten ones still describe the removed
backtracking, for example in this prog:
/* ensure we have callback frame in jump history */
...
/* restore reg from stack; in this case we'll be carrying
* stack mask when going back into subprog through jump
* history
*/
and "complicate jump history" in parent_callee_saved_reg_precise,
parent_callee_saved_reg_precise_global, parent_stack_slot_precise and
parent_stack_slot_precise_global.
Since jump history and stack masks no longer exist, should these be
reworded along with the "here r6 has to be known" comments?
[ ... ]
> @@ -730,32 +533,12 @@ __naked int subprog_spill_into_parent_stack_slot_precise(void)
>
> SEC("?raw_tp")
> __success __log_level(2)
> -__msg("17: (0f) r1 += r0")
> -__msg("mark_precise: frame0: regs=r0 stack= before 16: (bf) r1 = r7")
> [ ... ]
> -__msg("mark_precise: frame0: regs=r0 stack= before 6: (07) r7 += -8")
> -__msg("mark_precise: frame0: regs=r0 stack= before 5: (bf) r7 = r10")
> [ ... ]
> +__msg("17: (0f) r1 += r0 ; R0=4 R1=fp-4")
> __naked int stack_slot_aliases_precision(void)
> {
> asm volatile (
> [ ... ]
> @@ -777,11 +560,10 @@ __naked int stack_slot_aliases_precision(void)
> "r0 = *(u64 *)(r8 + 16);"
> "*(u64 *)(r7 - 8) = r0;"
> "r0 = *(u64 *)(r10 - 16);"
> - /* get ready to use r0 as an index into array to force precision */
> + /* get ready to use r0 as an index into array */
> "r0 *= 4;"
> "r1 = %[vals];"
> - /* here r0->r1->r6 chain is forced to be precise and has to be
> - * propagated back to the beginning, including through the
> + /* here r0 has to be known: the constant went through the
> * subprog call and all the stack spills and loads
> */
> "r1 += r0;"
this isn't a bug, but the new __msg() expects R1=fp-4 when "r1 += r0" runs,
so r1 appears to be fp-8 here and not &vals.
The removed log lines show that "r1 = %[vals]" was compiled as "r1 = r7",
and the asm sets r7 earlier with "r7 = r10; r7 += -8". The clobber list at
the end of the asm only names:
: __clobber_common, "r6"
so the compiler is free to place &vals in r7 or r8, which the asm
overwrites. The prog then reads the unwritten stack slot at fp-4 instead
of vals[1].
Should "r7" and "r8" be added to the clobbers, with R1=map_value(...,imm=4)
as the expected value? Otherwise, should "get ready to use r0 as an index
into array" and "here r0 has to be known" be reworded to say the prog
indexes the stack?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 71/81] selftests/bpf: Adjust tests of linked registers and spills
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (69 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 70/81] selftests/bpf: Adjust tests of precision tracking Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 72/81] selftests/bpf: Adjust tests of pruning Alexei Starovoitov
` (11 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
After 'r1 = r0' the two scalars have the same id: they are linked. When a
jump limits the range of one register, the verifier gives the same range to
the other register. Most tests of linked registers did not match the
ranges. They matched "mark_precise:" lines, the log of precision tracking,
which marked linked registers together. "bpf: Remove precision tracking"
removed these lines.
Match the effect of the id: the range of each linked register after the
jump, and the id after the two branches meet.
r0 &= 0xff
r1 = r0
r2 = r0
if r1 > 7 goto +0 ; r0, r1 and r2 have id=1 and [0,7]
r3 = r10 ; the branches meet: id=1 and [0,255]
In many other tests, for example the tests of spills, remove only the
"mark_precise:" lines.
The tests of stacksafe() and of ids in regsafe() let two states meet at an
insn. The result depends on which state arrives first. The verifier
followed each path, the fall-through path first. Now it verifies basic
blocks in a fixed order, and the target of a conditional jump is before the
fall-through.
if r0 == 0 goto 1f
... ; makes state A
goto 2f
1: ... ; makes state B
2: ... ; A and B meet
arrives at 2 the verifier follows paths basic blocks in order
------------ -------------------------- ---------------------
first A B
second B A
Swap the two branches to keep the sequence of a test.
ignore_unique_scalar_ids_cur and ignore_unique_scalar_ids_old swap their
names. When the first state covers the second state, the log shows
"from M to N: within", not "N: safe".
Two more differences are visible in the expected logs:
- The verifier clears an id that only one live register has. Thus
exceptions_assert.c does not expect "id=1".
- In most cases the verifier verifies a static subprog without the frames
of its callers. Thus a jump in the subprog does not change the linked
registers of a caller. See precision_many_frames.
No expected verdict changes. The changed tests drop BPF_F_TEST_STATE_FREQ,
which has no effect without pruning.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/aggregate_ret_func.c | 15 -
.../selftests/bpf/progs/aggregate_ret_kfunc.c | 7 -
.../selftests/bpf/progs/exceptions_assert.c | 24 +-
.../selftests/bpf/progs/test_global_func15.c | 5 -
.../bpf/progs/verifier_mem_size_reg.c | 8 +-
.../selftests/bpf/progs/verifier_scalar_ids.c | 313 +++++-------------
.../selftests/bpf/progs/verifier_spill_fill.c | 218 ++++--------
.../selftests/bpf/progs/verifier_var_off.c | 3 -
8 files changed, 165 insertions(+), 428 deletions(-)
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
index f0077d951a25..2671323255cd 100644
--- a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
@@ -121,14 +121,6 @@ static __naked __noinline u128 static_agg_precise(void)
SEC("tc")
__load_if_JITed()
__success __retval(0)
-__log_level(2)
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
-__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 12: (95) exit")
-__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
__naked int aggregate_ret_static_precise(void)
{
asm volatile (
@@ -150,13 +142,6 @@ __naked int aggregate_ret_static_precise(void)
SEC("tc")
__load_if_JITed()
__success __retval(0)
-__log_level(2)
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (57) r6 &= 7")
-__msg("mark_precise: frame0: regs=r6 stack= before 1: (bf) r6 = r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 0: (85) call pc+9")
__naked int aggregate_ret_global_precise(void)
{
asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
index 0361a7b42c9b..102156175ba0 100644
--- a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
+++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
@@ -31,13 +31,6 @@ SEC("tc")
__arch_x86_64 __arch_arm64
__load_if_JITed()
__success __retval(0)
-__log_level(2)
-__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r6 stack= before 6: (07) r1 += -8")
-__msg("mark_precise: frame0: regs=r6 stack= before 5: (bf) r1 = r10")
-__msg("mark_precise: frame0: regs=r6 stack= before 4: (57) r6 &= 7")
-__msg("mark_precise: frame0: regs=r6 stack= before 3: (bf) r6 = r2")
-__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
__naked int aggregate_ret_kfunc_precise(void)
{
asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/exceptions_assert.c b/tools/testing/selftests/bpf/progs/exceptions_assert.c
index e4abf4172fca..67b51f728af2 100644
--- a/tools/testing/selftests/bpf/progs/exceptions_assert.c
+++ b/tools/testing/selftests/bpf/progs/exceptions_assert.c
@@ -29,32 +29,32 @@ check_assert(s64, ==, eq_llong_min, LLONG_MIN);
__msg("R{{.}}=0x7fffffffffffffff")
check_assert(s64, ==, eq_llong_max, LLONG_MAX);
-__msg("R{{.}}=scalar(id=1,smax=0x7ffffffe)")
+__msg("R{{.}}=scalar(smax=0x7ffffffe)")
check_assert(s64, <, lt_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smax=-1,umin=0x8000000000000000,var_off=(0x8000000000000000; 0x7fffffffffffffff))")
check_assert(s64, <, lt_zero, 0);
-__msg("R{{.}}=scalar(id=1,smax=0xffffffff7fffffff")
+__msg("R{{.}}=scalar(smax=0xffffffff7fffffff")
check_assert(s64, <, lt_neg, INT_MIN);
-__msg("R{{.}}=scalar(id=1,smax=0x7fffffff)")
+__msg("R{{.}}=scalar(smax=0x7fffffff)")
check_assert(s64, <=, le_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smax=0)")
+__msg("R{{.}}=scalar(smax=0)")
check_assert(s64, <=, le_zero, 0);
-__msg("R{{.}}=scalar(id=1,smax=0xffffffff80000000")
+__msg("R{{.}}=scalar(smax=0xffffffff80000000")
check_assert(s64, <=, le_neg, INT_MIN);
-__msg("R{{.}}=scalar(id=1,smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=umin=0x80000000,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
check_assert(s64, >, gt_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=umin=1,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
check_assert(s64, >, gt_zero, 0);
-__msg("R{{.}}=scalar(id=1,smin=0xffffffff80000001")
+__msg("R{{.}}=scalar(smin=0xffffffff80000001")
check_assert(s64, >, gt_neg, INT_MIN);
-__msg("R{{.}}=scalar(id=1,smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=umin=0x7fffffff,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
check_assert(s64, >=, ge_pos, INT_MAX);
-__msg("R{{.}}=scalar(id=1,smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
+__msg("R{{.}}=scalar(smin=0,umax=0x7fffffffffffffff,var_off=(0x0; 0x7fffffffffffffff))")
check_assert(s64, >=, ge_zero, 0);
-__msg("R{{.}}=scalar(id=1,smin=0xffffffff80000000")
+__msg("R{{.}}=scalar(smin=0xffffffff80000000")
check_assert(s64, >=, ge_neg, INT_MIN);
SEC("?tc")
diff --git a/tools/testing/selftests/bpf/progs/test_global_func15.c b/tools/testing/selftests/bpf/progs/test_global_func15.c
index 201cc000b3f4..497f09dc8876 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func15.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func15.c
@@ -24,12 +24,7 @@ int global_func15(struct __sk_buff *skb)
}
SEC("cgroup_skb/ingress")
-__log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
__failure
-/* check that fallthrough code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before 2: (b7) r0 = 1")
-/* check that branch code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before 0: (85) call bpf_get_prandom_u32#7")
__msg("At program exit the register R0 has ")
__naked int global_func15_tricky_pruning(void)
{
diff --git a/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
index 7e24706a764e..aba9510a1c08 100644
--- a/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_mem_size_reg.c
@@ -7,13 +7,9 @@
char _license[] SEC("license") = "GPL";
-/*
- * The __szk size of a kfunc memory/size pair must be marked precise even when
- * the nullable buffer is passed as NULL.
- */
+/* The __szk size of a kfunc memory/size pair when the nullable buffer is passed as NULL */
SEC("?tc")
-__success __log_level(2)
-__msg("mark_precise: frame0: regs=r4 stack= before")
+__success
int dynptr_slice_null_buf_size_precise(struct __sk_buff *skb)
{
struct bpf_dynptr dptr;
diff --git a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
index 256547048cc4..92331e1a6c63 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
@@ -11,27 +11,21 @@ struct {
__type(value, long long);
} map_hash_8b SEC(".maps");
-/* Check that precision marks propagate through scalar IDs.
- * Registers r{0,1,2} have the same scalar ID.
- * Range information is propagated for scalars sharing same ID.
- * Check that precision mark for r0 causes precision marks for r{1,2}
- * when range information is propagated for 'if <reg> <op> <const>' insn.
+/* scalar in [0, max] that shares id with other registers, as it is in the log */
+#define LINKED(id, max, mask) \
+ "scalar(id=" id ",smin=smin32=0,smax=umax=smax32=umax32=" #max ",var_off=(0x0; " #mask "))"
+#define ALONE(max, mask) \
+ "scalar(smin=smin32=0,smax=umax=smax32=umax32=" #max ",var_off=(0x0; " #mask "))"
+
+/* Registers r{0,1,2} have the same scalar ID.
+ * Range information is propagated for scalars sharing same ID
+ * for 'if <reg> <op> <const>' insn.
+ * The registers share the ID in the join of both branches too.
*/
SEC("socket")
__success __log_level(2)
-/* first 'if' branch */
-__msg("6: (0f) r3 += r0")
-__msg("frame0: regs=r0 stack= before 4: (25) if r1 > 0x7 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2 stack=:")
-__msg("frame0: regs=r0,r1,r2 stack= before 3: (bf) r2 = r0")
-/* second 'if' branch */
-__msg("from 4 to 5: ")
-__msg("6: (0f) r3 += r0")
-__msg("frame0: regs=r0 stack= before 5: (bf) r3 = r10")
-__msg("frame0: regs=r0 stack= before 4: (25) if r1 > 0x7 goto pc+0")
-/* parent state already has r{0,1,2} as precise */
-__msg("frame0: parent state regs= stack=:")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("4: (25) if r1 > 0x7 goto pc+0 ; R0=" LINKED("1", 7, 0x7) " R1=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7))
+__msg("5: R0=" LINKED("1", 255, 0xff) " R1=" LINKED("1", 255, 0xff) " R2=" LINKED("1", 255, 0xff))
__naked void linked_regs_bpf_k(void)
{
asm volatile (
@@ -42,9 +36,6 @@ __naked void linked_regs_bpf_k(void)
"r1 = r0;"
"r2 = r0;"
"if r1 > 7 goto +0;"
- /* force r0 to be precise, this eventually marks r1 and r2 as
- * precise as well because of shared IDs
- */
"r3 = r10;"
"r3 += r0;"
/* Mark r1 and r2 as alive. */
@@ -57,16 +48,13 @@ __naked void linked_regs_bpf_k(void)
: __clobber_all);
}
-/* Registers r{0,1,2} share same ID when 'if r1 > ...' insn is processed,
- * check that verifier marks r{1,2} as precise while backtracking
- * 'if r1 > ...' with r0 already marked.
+/* Registers r{0,1,2} share same ID when 'if r1 > r3' insn is processed,
+ * check that range information is propagated to all of them.
*/
SEC("socket")
__success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("frame0: regs=r0 stack= before 5: (2d) if r1 > r3 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2,r3 stack=:")
-__msg("frame0: regs=r0,r1,r2,r3 stack= before 4: (b7) r3 = 7")
+__msg("5: (2d) if r1 > r3 goto pc+0 ; R0=" LINKED("1", 7, 0x7) " R1=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7))
+__msg("6: R0=" LINKED("1", 255, 0xff) " R1=" LINKED("1", 255, 0xff) " R2=" LINKED("1", 255, 0xff))
__naked void linked_regs_bpf_x_src(void)
{
asm volatile (
@@ -78,9 +66,6 @@ __naked void linked_regs_bpf_x_src(void)
"r2 = r0;"
"r3 = 7;"
"if r1 > r3 goto +0;"
- /* force r0 to be precise, this eventually marks r1 and r2 as
- * precise as well because of shared IDs
- */
"r4 = r10;"
"r4 += r0;"
/* Mark r1 and r2 as alive. */
@@ -93,16 +78,11 @@ __naked void linked_regs_bpf_x_src(void)
: __clobber_all);
}
-/* Registers r{0,1,2} share same ID when 'if r1 > r3' insn is processed,
- * check that verifier marks r{0,1,2} as precise while backtracking
- * 'if r1 > r3' with r3 already marked.
- */
+/* Same as linked_regs_bpf_x_src(), r3 is what is used after the jump. */
SEC("socket")
__success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("frame0: regs=r3 stack= before 5: (2d) if r1 > r3 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2,r3 stack=:")
-__msg("frame0: regs=r0,r1,r2,r3 stack= before 4: (b7) r3 = 7")
+__msg("5: (2d) if r1 > r3 goto pc+0 ; R0=" LINKED("1", 7, 0x7) " R1=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7))
+__msg("7: (0f) r4 += r3 ; R3=7 R4=fp7")
__naked void linked_regs_bpf_x_dst(void)
{
asm volatile (
@@ -114,9 +94,6 @@ __naked void linked_regs_bpf_x_dst(void)
"r2 = r0;"
"r3 = 7;"
"if r1 > r3 goto +0;"
- /* force r0 to be precise, this eventually marks r1 and r2 as
- * precise as well because of shared IDs
- */
"r4 = r10;"
"r4 += r3;"
/* Mark r1 and r2 as alive. */
@@ -135,12 +112,9 @@ __naked void linked_regs_bpf_x_dst(void)
*/
SEC("socket")
__success __log_level(2)
-__msg("7: (0f) r3 += r0")
-__msg("frame0: regs=r0 stack= before 6: (bf) r3 = r10")
-__msg("frame0: parent state regs=r0 stack=:")
-__msg("frame0: regs=r0 stack= before 5: (25) if r0 > 0x7 goto pc+0")
-__msg("frame0: parent state regs=r0,r2 stack=:")
-__flag(BPF_F_TEST_STATE_FREQ)
+/* r1 is not there */
+__msg("5: (25) if r0 > 0x7 goto pc+0 ; R0=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7) "{{$}}")
+__msg("8: (bf) r1 = r1 ; R1=0")
__naked void linked_regs_broken_link(void)
{
asm volatile (
@@ -155,9 +129,6 @@ __naked void linked_regs_broken_link(void)
*/
"r1 = 0;"
"if r0 > 7 goto +0;"
- /* force r0 to be precise,
- * this eventually marks r2 as precise because of shared IDs
- */
"r3 = r10;"
"r3 += r0;"
/* Mark r1 and r2 as alive. */
@@ -170,50 +141,15 @@ __naked void linked_regs_broken_link(void)
: __clobber_all);
}
-/* Check that precision marks propagate through scalar IDs.
- * Use the same scalar ID in multiple stack frames, check that
- * precision information is propagated up the call stack.
+/* Use the same scalar ID in multiple stack frames. A static function is verified
+ * on its own: registers of the callers are not there when r1 is compared, so it has no ID.
+ * They share their IDs after the calls like before, if there is more than one left.
*/
SEC("socket")
__success __log_level(2)
-__msg("17: (0f) r2 += r1")
-/* Current state */
-__msg("frame2: last_idx 17 first_idx 14 subseq_idx -1 ")
-__msg("frame2: regs=r1 stack= before 16: (bf) r2 = r10")
-__msg("frame2: parent state regs=r1 stack=")
-__msg("frame1: parent state regs= stack=")
-__msg("frame0: parent state regs= stack=")
-/* Parent state */
-__msg("frame2: last_idx 13 first_idx 13 subseq_idx 14 ")
-__msg("frame2: regs=r1 stack= before 13: (25) if r1 > 0x7 goto pc+0")
-__msg("frame2: parent state regs=r1 stack=")
-/* frame1.r{6,7} are marked because mark_precise_scalar_ids()
- * looks for all registers with frame2.r1.id in the current state
- */
-__msg("frame1: parent state regs=r6,r7 stack=")
-__msg("frame0: parent state regs=r6 stack=")
-/* Parent state */
-__msg("frame2: last_idx 9 first_idx 9 subseq_idx 13")
-__msg("frame2: regs=r1 stack= before 9: (85) call pc+3")
-/* frame1.r1 is marked because of backtracking of call instruction */
-__msg("frame1: parent state regs=r1,r6,r7 stack=")
-__msg("frame0: parent state regs=r6 stack=")
-/* Parent state */
-__msg("frame1: last_idx 8 first_idx 7 subseq_idx 9")
-__msg("frame1: regs=r1,r6,r7 stack= before 8: (bf) r7 = r1")
-__msg("frame1: regs=r1,r6 stack= before 7: (bf) r6 = r1")
-__msg("frame1: parent state regs=r1 stack=")
-__msg("frame0: parent state regs=r6 stack=")
-/* Parent state */
-__msg("frame1: last_idx 4 first_idx 4 subseq_idx 7")
-__msg("frame1: regs=r1 stack= before 4: (85) call pc+2")
-__msg("frame0: parent state regs=r1,r6 stack=")
-/* Parent state */
-__msg("frame0: last_idx 3 first_idx 1 subseq_idx 4")
-__msg("frame0: regs=r1,r6 stack= before 3: (bf) r6 = r0")
-__msg("frame0: regs=r0,r1 stack= before 2: (bf) r1 = r0")
-__msg("frame0: regs=r0 stack= before 1: (57) r0 &= 255")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("13: (25) if r1 > 0x7 goto pc+0 ; R1=" ALONE(7, 0x7))
+__msg("10: R0=0 R6=scalar(id={{.*}}umax32=255,{{.*}} R7=scalar(id={{.*}}umax32=255,")
+__msg("5: R0=0 R6=" ALONE(255, 0xff))
__naked void precision_many_frames(void)
{
asm volatile (
@@ -254,11 +190,6 @@ void precision_many_frames__bar(void)
"if r1 > 7 goto +0;"
"r6 = 0;" /* mark r6 as live */
"r7 = 0;" /* mark r7 as live */
- /* force r1 to be precise, this eventually marks:
- * - bar frame r1
- * - foo frame r{1,6,7}
- * - main frame r{1,6}
- */
"r2 = r10;"
"r2 += r1;"
"r0 = 0;"
@@ -266,23 +197,12 @@ void precision_many_frames__bar(void)
::: __clobber_all);
}
-/* Check that scalars with the same IDs are marked precise on stack as
+/* Check that range information is propagated to scalars with the same IDs on stack as
* well as in registers.
*/
SEC("socket")
__success __log_level(2)
-__msg("12: (0f) r2 += r1")
-/* foo frame */
-__msg("frame1: regs=r1 stack= before 11: (bf) r2 = r10")
-__msg("frame1: regs=r1 stack= before 10: (25) if r1 > 0x7 goto pc+0")
-__msg("frame1: regs=r1 stack=-8,-16 before 9: (7b) *(u64 *)(r10 -16) = r1")
-__msg("frame1: regs=r1 stack=-8 before 8: (7b) *(u64 *)(r10 -8) = r1")
-__msg("frame1: regs=r1 stack= before 4: (85) call pc+3")
-/* main frame */
-__msg("frame0: regs=r1 stack=-8 before 3: (7b) *(u64 *)(r10 -8) = r1")
-__msg("frame0: regs=r1 stack= before 2: (bf) r1 = r0")
-__msg("frame0: regs=r0 stack= before 1: (57) r0 &= 255")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("10: (25) if r1 > 0x7 goto pc+0 ; R1=scalar({{.*}}umax32=7,{{.*}} fp-8=scalar({{.*}}umax32=7,{{.*}} fp-16=scalar({{.*}}umax32=7,")
__naked void precision_stack(void)
{
asm volatile (
@@ -311,10 +231,6 @@ void precision_stack__foo(void)
"*(u64*)(r10 - 8) = r1;"
"*(u64*)(r10 - 16) = r1;"
"if r1 > 7 goto +0;"
- /* force r1 to be precise, this eventually marks:
- * - foo frame r1,fp{-8,-16}
- * - main frame r1,fp{-8}
- */
"r2 = r10;"
"r2 += r1;"
"r0 = *(u64*)(r10 - 8);"
@@ -329,20 +245,9 @@ void precision_stack__foo(void)
SEC("socket")
__success __log_level(2)
/* r{6,7} */
-__msg("12: (0f) r3 += r7")
-__msg("frame0: regs=r7 stack= before 11: (bf) r3 = r10")
-__msg("frame0: regs=r7 stack= before 9: (25) if r7 > 0x7 goto pc+0")
-/* ... skip some insns ... */
-__msg("frame0: regs=r6,r7 stack= before 3: (bf) r7 = r0")
-__msg("frame0: regs=r0,r6 stack= before 2: (bf) r6 = r0")
+__msg("9: (25) if r7 > 0x7 goto pc+0 ; R6=" LINKED("1", 7, 0x7) " R7=" LINKED("1", 7, 0x7) "{{$}}")
/* r{8,9} */
-__msg("13: (0f) r3 += r9")
-__msg("frame0: regs=r9 stack= before 12: (0f) r3 += r7")
-/* ... skip some insns ... */
-__msg("frame0: regs=r9 stack= before 10: (25) if r9 > 0x7 goto pc+0")
-__msg("frame0: regs=r8,r9 stack= before 7: (bf) r9 = r0")
-__msg("frame0: regs=r0,r8 stack= before 6: (bf) r8 = r0")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("10: (25) if r9 > 0x7 goto pc+0 ; R8=" LINKED("2", 7, 0x7) " R9=" LINKED("2", 7, 0x7) "{{$}}")
__naked void precision_two_ids(void)
{
asm volatile (
@@ -360,13 +265,11 @@ __naked void precision_two_ids(void)
"r9 = r0;"
/* clear r0 id */
"r0 = 0;"
- /* propagate equal scalars precision */
+ /* propagate range information to equal scalars */
"if r7 > 7 goto +0;"
"if r9 > 7 goto +0;"
"r3 = r10;"
- /* force r7 to be precise, this also marks r6 */
"r3 += r7;"
- /* force r9 to be precise, this also marks r8 */
"r3 += r9;"
"r6 = r6;" /* mark r6 as live */
"r8 = r8;" /* mark r8 as live */
@@ -378,7 +281,6 @@ __naked void precision_two_ids(void)
SEC("socket")
__success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
/*
* check that r5 is unlinked after 'if', collect_linked_regs() can't tie
* more than 5 registers for a single insn and the register compared by
@@ -386,9 +288,6 @@ __flag(BPF_F_TEST_STATE_FREQ)
*/
__msg("7: (25) if r5 > 0x7 goto pc+0 ; R5=scalar(smin=")
__msg("12: (bf) r5 = r5 ; R5=scalar(id=2")
-/* check that r{0-4} are marked precise after 'if' */
-__msg("frame0: regs=r0 stack= before 7: (25) if r5 > 0x7 goto pc+0")
-__msg("frame0: parent state regs=r0,r1,r2,r3,r4 stack=:")
__naked void linked_regs_too_many_regs(void)
{
asm volatile (
@@ -410,9 +309,6 @@ __naked void linked_regs_too_many_regs(void)
"r4 = r4;"
/* make r5 appear in the log */
"r5 = r5;"
- /* force r0 to be precise,
- * this would cause r{0-4} to be precise because of shared IDs
- */
"r7 = r10;"
"r7 += r0;"
"r0 = 0;"
@@ -424,10 +320,7 @@ __naked void linked_regs_too_many_regs(void)
SEC("socket")
__failure __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("regs=r7 stack= before 5: (3d) if r8 >= r0")
-__msg("parent state regs=r0,r7,r8")
-__msg("regs=r0,r7,r8 stack= before 4: (25) if r0 > 0x1")
+__msg("5: (3d) if r8 >= r0 goto pc+3 ; R0=scalar(umin=1) R7=scalar(id=1,umax=0xfffffffffffffffe) R8=scalar(id=1,umax=0xfffffffffffffffe)")
__msg("div by zero")
__naked void linked_regs_broken_link_2(void)
{
@@ -438,13 +331,11 @@ __naked void linked_regs_broken_link_2(void)
"call %[bpf_get_prandom_u32];"
"if r0 > 1 goto +0;"
/* r7.id == r8.id,
- * thus r7 precision implies r8 precision,
- * which implies r0 precision because of the conditional below.
+ * thus what is learned about r8 is learned about r7
*/
"if r8 >= r0 goto 1f;"
/* break id relation between r7 and r8 */
"r8 += r8;"
- /* make r7 precise */
"if r7 == 0 goto 1f;"
"r0 /= 0;"
"1:"
@@ -455,13 +346,11 @@ __naked void linked_regs_broken_link_2(void)
: __clobber_all);
}
-/* Check that mark_chain_precision() for one of the conditional jump
- * operands does not trigger equal scalars precision propagation.
- */
+/* A conditional jump that is predicted says nothing new about its operands. */
SEC("socket")
__success __log_level(2)
-__msg("3: (25) if r1 > 0x100 goto pc+0")
-__msg("frame0: regs=r1 stack= before 2: (bf) r1 = r0")
+__msg("3: (25) if r1 > 0x100 goto pc+0 ; R1=" LINKED("1", 255, 0xff) "{{$}}")
+__msg("processed 6 insns")
__naked void cjmp_no_linked_regs_trigger(void)
{
asm volatile (
@@ -470,9 +359,7 @@ __naked void cjmp_no_linked_regs_trigger(void)
"r0 &= 0xff;"
/* tie r0.id == r1.id */
"r1 = r0;"
- /* the jump below would be predicted, thus r1 would be marked precise,
- * this should not imply precision mark for r0
- */
+ /* the jump below is predicted */
"if r1 > 256 goto +0;"
"r0 = 0;"
"exit;"
@@ -497,7 +384,6 @@ __naked void cjmp_no_linked_regs_trigger(void)
*/
SEC("socket")
__failure __msg("register with unbounded min value")
-__flag(BPF_F_TEST_STATE_FREQ)
__naked void check_ids_in_regsafe(void)
{
asm volatile (
@@ -539,12 +425,9 @@ __naked void check_ids_in_regsafe(void)
* (2) r6{.id=B}, r7{.id=A}, r8{.id=B}
*
* Where (2) is not safe, as "r7 > 4" check won't propagate range for it.
- * This example would be considered safe without changes to
- * mark_chain_precision() to track scalar values with equal IDs.
*/
SEC("socket")
__failure __msg("register with unbounded min value")
-__flag(BPF_F_TEST_STATE_FREQ)
__naked void check_ids_in_regsafe_2(void)
{
asm volatile (
@@ -598,15 +481,13 @@ __naked void check_ids_in_regsafe_2(void)
* (1) r1{.id=A}, r2{.id=A}
* (2) r1{.id=C}, r2{.id=C}
*
- * Thus forcing 'if r1 == r2' verification twice.
+ * neither of which is within the other one.
*/
SEC("socket")
__success __log_level(2)
-__msg("14: (1d) if r3 == r4 goto pc+0")
-__msg("frame 0: propagating r3,r4")
-__msg("14: safe")
-__msg("processed 18 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("from 7 to 14: within")
+__msg("14: (1d) if r3 == r4 goto pc+0 ; R3=0 R4=0")
+__msg("processed 17 insns")
__naked void no_scalar_id_for_const(void)
{
asm volatile (
@@ -629,7 +510,7 @@ __naked void no_scalar_id_for_const(void)
"r3 = r1;"
"r4 = r2;"
"l1_%=:"
- /* predictable jump, marks r3 and r4 precise */
+ /* predictable jump */
"if r3 == r4 goto +0;"
"r0 = 0;"
"exit;"
@@ -641,11 +522,9 @@ __naked void no_scalar_id_for_const(void)
/* Same as no_scalar_id_for_const() but for 32-bit values */
SEC("socket")
__success __log_level(2)
-__msg("14: (1e) if w3 == w4 goto pc+0")
-__msg("frame 0: propagating r3,r4")
-__msg("14: safe")
-__msg("processed 18 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("from 7 to 14: within")
+__msg("14: (1e) if w3 == w4 goto pc+0 ; R3=0 R4=0")
+__msg("processed 17 insns")
__naked void no_scalar_id_for_const32(void)
{
asm volatile (
@@ -668,7 +547,7 @@ __naked void no_scalar_id_for_const32(void)
"w3 = w1;"
"w4 = w2;"
"l1_%=:"
- /* predictable jump, marks r3 and r4 precise */
+ /* predictable jump */
"if w3 == w4 goto +0;"
"r0 = 0;"
"exit;"
@@ -677,20 +556,19 @@ __naked void no_scalar_id_for_const32(void)
: __clobber_all);
}
-/* Check that unique scalar IDs are ignored when new verifier state is
- * compared to cached verifier state. For this test:
- * - cached state has no id on r1
- * - new state has a unique id on r1
+/* Check that unique scalar IDs are ignored when a verifier state is
+ * compared to the state that is there. For this test:
+ * - the state that is there had a unique id on r1
+ * - the state that arrives has no id on r1
*/
SEC("socket")
__success __log_level(2)
__msg("6: (25) if r6 > 0x7 goto pc+1")
__msg("7: (57) r1 &= 255")
-__msg("8: (bf) r2 = r10")
-__msg("from 6 to 8: safe")
-__msg("processed 12 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
-__naked void ignore_unique_scalar_ids_cur(void)
+__msg("from 7 to 8: within")
+__msg("8: R0=0 R1=" ALONE(255, 0xff))
+__msg("processed 11 insns")
+__naked void ignore_unique_scalar_ids_old(void)
{
asm volatile (
"call %[bpf_ktime_get_ns];"
@@ -706,8 +584,8 @@ __naked void ignore_unique_scalar_ids_cur(void)
"r1 &= 0xff;"
"l0_%=:"
/* get here in two states:
- * - first: r1 has no id (cached state)
- * - second: r1 has a unique id (should be considered equivalent)
+ * - first: r1 has a unique id
+ * - second: r1 has no id (should be considered to be within the first)
*/
"r2 = r10;"
"r2 += r1;"
@@ -717,20 +595,19 @@ __naked void ignore_unique_scalar_ids_cur(void)
: __clobber_all);
}
-/* Check that unique scalar IDs are ignored when new verifier state is
- * compared to cached verifier state. For this test:
- * - cached state has a unique id on r1
- * - new state has no id on r1
+/* Check that unique scalar IDs are ignored when a verifier state is
+ * compared to the state that is there. For this test:
+ * - the state that is there has no id on r1
+ * - the state that arrives has a unique id on r1
*/
SEC("socket")
__success __log_level(2)
__msg("6: (25) if r6 > 0x7 goto pc+1")
__msg("7: (05) goto pc+1")
-__msg("9: (bf) r2 = r10")
-__msg("9: safe")
-__msg("processed 13 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
-__naked void ignore_unique_scalar_ids_old(void)
+__msg("from 7 to 9: within")
+__msg("9: R0=0 R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff)) R10=fp0")
+__msg("processed 12 insns")
+__naked void ignore_unique_scalar_ids_cur(void)
{
asm volatile (
"call %[bpf_ktime_get_ns];"
@@ -748,8 +625,8 @@ __naked void ignore_unique_scalar_ids_old(void)
"r1 &= 0xff;"
"l0_%=:"
/* get here in two states:
- * - first: r1 has a unique id (cached state)
- * - second: r1 has no id (should be considered equivalent)
+ * - first: r1 has no id
+ * - second: r1 has a unique id (should be considered to be within the first)
*/
"r2 = r10;"
"r2 += r1;"
@@ -759,16 +636,13 @@ __naked void ignore_unique_scalar_ids_old(void)
: __clobber_all);
}
-/* Check that two registers with 0 scalar IDs in a verified state can be mapped
- * to the same scalar ID in current state.
+/* Check that two registers with 0 scalar IDs in the state that is there can be mapped
+ * to the same scalar ID in the state that arrives.
*/
SEC("socket")
__success __log_level(2)
-/* The states should be equivalent on reaching insn 12.
- */
-__msg("12: safe")
-__msg("processed 17 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("from 10 to 11: within")
+__msg("processed 15 insns")
__naked void two_nil_old_ids_one_cur_id(void)
{
asm volatile (
@@ -783,16 +657,13 @@ __naked void two_nil_old_ids_one_cur_id(void)
"r7 *= 1;"
"r0 = 0;"
/* Maybe make r{6,7} IDs identical */
- "if r6 > r7 goto l0_%=;"
- "goto l1_%=;"
-"l0_%=:"
+ "if r6 <= r7 goto l1_%=;"
"r6 = r7;"
"l1_%=:"
- /* Mark r{6,7} precise.
- * Get here in two states:
- * - first: r6{.id=0}, r7{.id=0} (cached state)
+ /* Get here in two states:
+ * - first: r6{.id=0}, r7{.id=0}
* - second: r6{.id=A}, r7{.id=A}
- * Verifier considers such states equivalent.
+ * Verifier considers the second one to be within the first one.
* Thus "exit;" would be verified only once.
*/
"r2 = r10;"
@@ -804,18 +675,13 @@ __naked void two_nil_old_ids_one_cur_id(void)
: __clobber_all);
}
-/* Check that two different scalar IDs in a verified state can't be
- * mapped to the same scalar ID in current state.
+/* Check that two different scalar IDs in the state that is there can't be
+ * mapped to the same scalar ID in the state that arrives.
*/
SEC("socket")
__success __log_level(2)
-/* The exit instruction should be reachable from two states,
- * use two matches and "processed .. insns" to ensure this.
- */
-__msg("16: (95) exit")
-__msg("16: (95) exit")
-__msg("processed 22 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__not_msg("from 10 to 11: within")
+__msg("11: R0=0 R6=" ALONE(255, 0xff) " R7=" ALONE(255, 0xff))
__naked void two_old_ids_one_cur_id(void)
{
asm volatile (
@@ -830,17 +696,14 @@ __naked void two_old_ids_one_cur_id(void)
"r9 = r0;"
"r0 = 0;"
/* Maybe make r{6,7} IDs identical */
- "if r6 > r7 goto l0_%=;"
- "goto l1_%=;"
-"l0_%=:"
+ "if r6 <= r7 goto l1_%=;"
"r6 = r7;"
"l1_%=:"
- /* Mark r{6,7} precise.
- * Get here in two states:
- * - first: r6{.id=A}, r7{.id=B} (cached state)
- * - second: r6{.id=A}, r7{.id=A}
- * Currently we don't want to consider such states equivalent.
- * Thus "exit;" would be verified twice.
+ /* Get here in two states:
+ * - first: r6{.id=A}, r7{.id=B}
+ * - second: r6{.id=B}, r7{.id=B}
+ * Currently we don't want to consider the second one to be within
+ * the first one. Thus they are joined.
*/
"r2 = r10;"
"r2 += r6;"
@@ -878,10 +741,6 @@ __log_level(2)
__msg("4: (77) r1 >>= 32 ; R1=0")
__msg("5: (bf) r0 = r1 ; R0=0 R1=0")
__msg("6: (95) exit")
-__msg("from 3 to 4")
-__msg("4: (77) r1 >>= 32 ; R1=0")
-__msg("5: (bf) r0 = r1 ; R0=0 R1=0")
-__msg("6: (95) exit")
/* Verify that statements to randomize upper half of r1 had not been
* generated.
*/
diff --git a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
index 4d1374885f6d..b41f390df3af 100644
--- a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
+++ b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
@@ -518,15 +518,6 @@ __msg("6: (71) r2 = *(u8 *)(r10 -1) ; R2=0 R10=fp0 fp-8=0")
*/
__msg("10: (71) r2 = *(u8 *)(r10 -9) ; R2=0 R10=fp0 fp-16=0")
__msg("11: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 11 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 10: (71) r2 = *(u8 *)(r10 -9)")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 8: (73) *(u8 *)(r1 +0) = r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 7: (0f) r1 += r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 6: (71) r2 = *(u8 *)(r10 -1)")
-__msg("mark_precise: frame0: regs= stack=-16 before 5: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 4: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 3: (b7) r0 = 0")
__naked void partial_stack_load_preserves_zeros(void)
{
asm volatile (
@@ -600,8 +591,6 @@ __success
__msg("2: (62) *(u32 *)(r10 -4) = 0 ; R10=fp0 fp-8=0000????")
__msg("4: (71) r2 = *(u8 *)(r10 -1) ; R2=0 R10=fp0 fp-8=0000????")
__msg("5: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 4: (71) r2 = *(u8 *)(r10 -1)")
__naked void partial_stack_load_preserves_partial_zeros(void)
{
asm volatile (
@@ -637,7 +626,6 @@ __naked void partial_stack_load_preserves_partial_zeros(void)
SEC("raw_tp")
__log_level(2)
__success
-__msg("mark_precise: frame0: regs= stack=-8")
__msg("R2=0")
__naked void stack_load_preserves_mixed_zero_and_zero_spill(void)
{
@@ -663,50 +651,21 @@ __naked void stack_load_preserves_mixed_zero_and_zero_spill(void)
char two_byte_buf[2] SEC(".data.two_byte_buf");
SEC("raw_tp")
-__log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
+__log_level(2)
__success
-/* make sure fp-8 is IMPRECISE fake register spill */
+/* make sure fp-8 is fake register spill */
__msg("3: (7a) *(u64 *)(r10 -8) = 1 ; R10=fp0 fp-8=1")
-/* and fp-16 is spilled IMPRECISE const reg */
+/* and fp-16 is spilled const reg */
__msg("5: (7b) *(u64 *)(r10 -16) = r0 ; R0=1 R10=fp0 fp-16=1")
/* validate load from fp-8, which was initialized using BPF_ST_MEM */
__msg("8: (79) r2 = *(u64 *)(r10 -8) ; R2=1 R10=fp0 fp-8=1")
-__msg("9: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 8: (79) r2 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
-/* note, fp-8 is precise, fp-16 is not yet precise, we'll get there */
-__msg("mark_precise: frame0: parent state regs= stack=-8: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (7a) *(u64 *)(r10 -8) = 1")
-__msg("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("9: (0f) r1 += r2 ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
/* validate load from fp-16, which was initialized using BPF_STX_MEM */
__msg("12: (79) r2 = *(u64 *)(r10 -16) ; R2=1 R10=fp0 fp-16=1")
-__msg("13: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 12: (79) r2 = *(u64 *)(r10 -16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 11: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 10: (73) *(u8 *)(r1 +0) = r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 8: (79) r2 = *(u64 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
-/* now both fp-8 and fp-16 are precise, very good */
-__msg("mark_precise: frame0: parent state regs= stack=-16: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=P1 fp-16=P1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-16 before 5: (7b) *(u64 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1")
-__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("13: (0f) r1 += r2 ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
__naked void stack_load_preserves_const_precision(void)
{
asm volatile (
- /* establish checkpoint with state that has no stack slots;
- * if we bubble up to this state without finding desired stack
- * slot, then it's a bug and should be caught
- */
"goto +0;"
/* fp-8 is const 1 *fake* register */
@@ -716,7 +675,6 @@ __naked void stack_load_preserves_const_precision(void)
"r0 = 1;"
"*(u64 *)(r10 -16) = r0;"
- /* force checkpoint to check precision marks preserved in parent states */
"goto +0;"
/* load single U64 from aligned FAKE_REG=1 slot */
@@ -740,48 +698,21 @@ __naked void stack_load_preserves_const_precision(void)
}
SEC("raw_tp")
-__log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
+__log_level(2)
__success
/* make sure fp-8 is 32-bit FAKE subregister spill */
__msg("3: (62) *(u32 *)(r10 -8) = 1 ; R10=fp0 fp-8=????1")
-/* but fp-16 is spilled IMPRECISE zero const reg */
+/* and fp-16 is spilled const reg */
__msg("5: (63) *(u32 *)(r10 -16) = r0 ; R0=1 R10=fp0 fp-16=????1")
/* validate load from fp-8, which was initialized using BPF_ST_MEM */
__msg("8: (61) r2 = *(u32 *)(r10 -8) ; R2=1 R10=fp0 fp-8=????1")
-__msg("9: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 8: (61) r2 = *(u32 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-8 before 7: (bf) r1 = r6")
-__msg("mark_precise: frame0: parent state regs= stack=-8: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-8 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-8 before 5: (63) *(u32 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs= stack=-8 before 4: (b7) r0 = 1")
-__msg("mark_precise: frame0: regs= stack=-8 before 3: (62) *(u32 *)(r10 -8) = 1")
-__msg("10: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("9: (0f) r1 += r2 ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
/* validate load from fp-16, which was initialized using BPF_STX_MEM */
__msg("12: (61) r2 = *(u32 *)(r10 -16) ; R2=1 R10=fp0 fp-16=????1")
-__msg("13: (0f) r1 += r2")
-__msg("mark_precise: frame0: last_idx 13 first_idx 7 subseq_idx -1")
-__msg("mark_precise: frame0: regs=r2 stack= before 12: (61) r2 = *(u32 *)(r10 -16)")
-__msg("mark_precise: frame0: regs= stack=-16 before 11: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs= stack=-16 before 10: (73) *(u8 *)(r1 +0) = r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 9: (0f) r1 += r2")
-__msg("mark_precise: frame0: regs= stack=-16 before 8: (61) r2 = *(u32 *)(r10 -8)")
-__msg("mark_precise: frame0: regs= stack=-16 before 7: (bf) r1 = r6")
-__msg("mark_precise: frame0: parent state regs= stack=-16: R6=map_value(map=.data.two_byte_,ks=4,vs=2) R10=fp0 fp-8=????P1 fp-16=????P1")
-__msg("mark_precise: frame0: last_idx 6 first_idx 3 subseq_idx 7")
-__msg("mark_precise: frame0: regs= stack=-16 before 6: (05) goto pc+0")
-__msg("mark_precise: frame0: regs= stack=-16 before 5: (63) *(u32 *)(r10 -16) = r0")
-__msg("mark_precise: frame0: regs=r0 stack= before 4: (b7) r0 = 1")
-__msg("14: R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
+__msg("13: (0f) r1 += r2 ; R1=map_value(map=.data.two_byte_,ks=4,vs=2,imm=1) R2=1")
__naked void stack_load_preserves_const_precision_subreg(void)
{
asm volatile (
- /* establish checkpoint with state that has no stack slots;
- * if we bubble up to this state without finding desired stack
- * slot, then it's a bug and should be caught
- */
"goto +0;"
/* fp-8 is const 1 *fake* SUB-register */
@@ -791,7 +722,6 @@ __naked void stack_load_preserves_const_precision_subreg(void)
"r0 = 1;"
"*(u32 *)(r10 -16) = r0;"
- /* force checkpoint to check precision marks preserved in parent states */
"goto +0;"
/* load single U32 from aligned FAKE_REG=1 slot */
@@ -1118,41 +1048,36 @@ l0_%=: r1 >>= 32; \
: __clobber_all);
}
-/* stacksafe(): check if stack spill of an imprecise scalar in old state
- * is considered equivalent to STACK_{MISC,INVALID} in cur state.
+/* stacksafe(): the state with a stack spill of a scalar is there (label 1 is
+ * looked at first), the state with STACK_{MISC,INVALID} arrives.
+ * The first one is within the second one, which stands for both.
*/
SEC("socket")
__success __log_level(2)
+__msg("8: R0=scalar() R10=fp0 fp-8=??mm??mm")
__msg("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg("8: safe")
-__msg("processed 11 insns")
-/* STACK_INVALID should prevent verifier in unpriv mode from
- * considering states equivalent and force an error on second
- * verification path (entry - label 1 - label 2).
- */
+__msg("processed 10 insns")
+/* STACK_INVALID is an error in unpriv mode */
__failure_unpriv
__msg_unpriv("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg_unpriv("9: (95) exit")
-__msg_unpriv("8: (79) r1 = *(u64 *)(r10 -8)")
__msg_unpriv("invalid read from stack off -8+2 size 8")
-__flag(BPF_F_TEST_STATE_FREQ)
__naked void old_imprecise_scalar_vs_cur_stack_misc(void)
{
asm volatile(
/* get a random value for branching */
"call %[bpf_ktime_get_ns];"
"if r0 == 0 goto 1f;"
- /* conjure scalar at fp-8 */
- "r0 = 42;"
- "*(u64*)(r10 - 8) = r0;"
- "goto 2f;"
-"1:"
/* conjure STACK_{MISC,INVALID} at fp-8 */
"call %[bpf_ktime_get_ns];"
"*(u16*)(r10 - 8) = r0;"
"*(u16*)(r10 - 4) = r0;"
+ "goto 2f;"
+"1:"
+ /* conjure scalar at fp-8 */
+ "r0 = 42;"
+ "*(u64*)(r10 - 8) = r0;"
"2:"
- /* read fp-8, should be considered safe on second visit */
+ /* read fp-8, it's looked at once */
"r1 = *(u64*)(r10 - 8);"
"exit;"
:
@@ -1160,38 +1085,33 @@ __naked void old_imprecise_scalar_vs_cur_stack_misc(void)
: __clobber_all);
}
-/* stacksafe(): check that stack spill of a precise scalar in old state
- * is not considered equivalent to STACK_MISC in cur state.
+/* stacksafe(): check that the state with STACK_MISC is not considered to be
+ * within the state with a stack spill of a known scalar, which is there.
*/
SEC("socket")
__success __log_level(2)
-/* verifier should visit 'if r1 == 0x2a ...' two times:
- * - once for path entry - label 2;
- * - once for path entry - label 1 - label 2.
- */
-__msg("if r1 == 0x2a goto pc+0")
-__msg("if r1 == 0x2a goto pc+0")
-__msg("processed 15 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+/* what is loaded is not known to be 42 */
+__msg("8: (79) r1 = *(u64 *)(r10 -8) ; R1=scalar() R10=fp0 fp-8=mmmmmmmm")
+__msg("9: (15) if r1 == 0x2a goto pc+0 ; R1=scalar()")
+__msg("processed 11 insns")
__naked void old_precise_scalar_vs_cur_stack_misc(void)
{
asm volatile(
/* get a random value for branching */
"call %[bpf_ktime_get_ns];"
"if r0 == 0 goto 1f;"
- /* conjure scalar at fp-8 */
- "r0 = 42;"
- "*(u64*)(r10 - 8) = r0;"
- "goto 2f;"
-"1:"
/* conjure STACK_MISC at fp-8 */
"call %[bpf_ktime_get_ns];"
"*(u64*)(r10 - 8) = r0;"
"*(u32*)(r10 - 4) = r0;"
+ "goto 2f;"
+"1:"
+ /* conjure scalar at fp-8 */
+ "r0 = 42;"
+ "*(u64*)(r10 - 8) = r0;"
"2:"
- /* read fp-8, should not be considered safe on second visit */
"r1 = *(u64*)(r10 - 8);"
- /* use r1 in precise context */
+ /* the value of r1 matters */
"if r1 == 42 goto +0;"
"exit;"
:
@@ -1199,32 +1119,31 @@ __naked void old_precise_scalar_vs_cur_stack_misc(void)
: __clobber_all);
}
-/* stacksafe(): check if STACK_MISC in old state is considered
- * equivalent to stack spill of a scalar in cur state.
+/* stacksafe(): check if the state with a stack spill of a scalar is considered
+ * to be within the state with STACK_MISC, which is there.
*/
SEC("socket")
__success __log_level(2)
+__msg("from 4 to 8: within")
__msg("8: (79) r0 = *(u64 *)(r10 -8)")
-__msg("8: safe")
-__msg("processed 11 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("processed 10 insns")
__naked void old_stack_misc_vs_cur_scalar(void)
{
asm volatile(
/* get a random value for branching */
"call %[bpf_ktime_get_ns];"
"if r0 == 0 goto 1f;"
+ /* conjure scalar at fp-8 */
+ "r0 = 42;"
+ "*(u64*)(r10 - 8) = r0;"
+ "goto 2f;"
+"1:"
/* conjure STACK_{MISC,INVALID} at fp-8 */
"call %[bpf_ktime_get_ns];"
"*(u16*)(r10 - 8) = r0;"
"*(u16*)(r10 - 4) = r0;"
- "goto 2f;"
-"1:"
- /* conjure scalar at fp-8 */
- "r0 = 42;"
- "*(u64*)(r10 - 8) = r0;"
"2:"
- /* read fp-8, should be considered safe on second visit */
+ /* read fp-8, it's looked at once */
"r0 = *(u64*)(r10 - 8);"
"exit;"
:
@@ -1232,21 +1151,15 @@ __naked void old_stack_misc_vs_cur_scalar(void)
: __clobber_all);
}
-/* stacksafe(): check that STACK_MISC in old state is not considered
- * equivalent to stack spill of a non-scalar in cur state.
+/* stacksafe(): the state with STACK_MISC is there, the state with a stack spill
+ * of a non-scalar arrives. A program that may read uninitialized stack may
+ * take a pointer for a number: the second state is within the first one.
*/
SEC("socket")
__success __log_level(2)
-/* verifier should process exit instructions twice:
- * - once for path entry - label 2;
- * - once for path entry - label 1 - label 2.
- */
-__msg("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg("9: (95) exit")
-__msg("from 2 to 7")
-__msg("8: safe")
-__msg("processed 11 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("from 4 to 8: within")
+__msg("8: (79) r1 = *(u64 *)(r10 -8) ; R1=scalar()")
+__msg("processed 10 insns")
__naked void old_stack_misc_vs_cur_ctx_ptr(void)
{
asm volatile(
@@ -1255,16 +1168,16 @@ __naked void old_stack_misc_vs_cur_ctx_ptr(void)
/* get a random value for branching */
"call %[bpf_ktime_get_ns];"
"if r0 == 0 goto 1f;"
+ /* conjure context pointer in fp-8 */
+ "*(u64*)(r10 - 8) = r9;"
+ "goto 2f;"
+"1:"
/* conjure STACK_MISC at fp-8 */
"call %[bpf_ktime_get_ns];"
"*(u64*)(r10 - 8) = r0;"
"*(u32*)(r10 - 4) = r0;"
- "goto 2f;"
-"1:"
- /* conjure context pointer in fp-8 */
- "*(u64*)(r10 - 8) = r9;"
"2:"
- /* read fp-8, should not be considered safe on second visit */
+ /* read fp-8, it's looked at once */
"r1 = *(u64*)(r10 - 8);"
"exit;"
:
@@ -1310,35 +1223,34 @@ __naked void stack_noperfmon_spill_32bit_onto_64bit_slot(void)
}
/*
- * stacksafe(): check if 32-bit scalar spill in old state is considered
- * equivalent to STACK_MISC in cur state.
+ * stacksafe(): the state with a 32-bit scalar spill is there, the state with
+ * STACK_MISC arrives. The first one is within the second one.
* 32-bit scalar spill creates slot[0-3] = STACK_MISC, slot[4-7] = STACK_SPILL.
* Without 32-bit spill support in stacksafe(), the STACK_SPILL vs STACK_MISC
- * mismatch at slot[4] causes pruning to fail.
+ * mismatch at slot[4] keeps the states apart.
*/
SEC("socket")
__success __log_level(2)
+__msg("8: R0=scalar() R10=fp0 fp-8=????mmmm")
__msg("8: (79) r1 = *(u64 *)(r10 -8)")
-__msg("8: safe")
-__msg("processed 11 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("processed 10 insns")
__naked void old_imprecise_scalar32_vs_cur_stack_misc(void)
{
asm volatile(
/* get a random value for branching */
"call %[bpf_ktime_get_ns];"
"if r0 == 0 goto 1f;"
- /* conjure 32-bit scalar spill at fp-8 */
- "r0 = 42;"
- "*(u32*)(r10 - 8) = r0;"
- "goto 2f;"
-"1:"
/* conjure STACK_MISC at fp-8 */
"call %[bpf_ktime_get_ns];"
"*(u16*)(r10 - 8) = r0;"
"*(u16*)(r10 - 6) = r0;"
+ "goto 2f;"
+"1:"
+ /* conjure 32-bit scalar spill at fp-8 */
+ "r0 = 42;"
+ "*(u32*)(r10 - 8) = r0;"
"2:"
- /* read fp-8, should be considered safe on second visit */
+ /* read fp-8, it's looked at once */
"r1 = *(u64*)(r10 - 8);"
"exit;"
:
diff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c
index 399884911ea5..d92215c5b923 100644
--- a/tools/testing/selftests/bpf/progs/verifier_var_off.c
+++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c
@@ -63,7 +63,6 @@ SEC("cgroup/skb")
__description("variable-offset stack read preserves spilled zero")
__success
__log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8")
__msg("R3=0")
__retval(0)
__naked void stack_read_var_off_preserves_spilled_zero(void)
@@ -89,7 +88,6 @@ SEC("cgroup/skb")
__description("variable-offset stack read preserves spilled zero across slots")
__success
__log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8,-16")
__msg("R3=0")
__retval(0)
__naked void stack_read_var_off_preserves_spilled_zero_across_slots(void)
@@ -116,7 +114,6 @@ SEC("cgroup/skb")
__description("variable-offset stack read preserves partial spilled zero")
__success
__log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8")
__msg("R3=0")
__retval(0)
__naked void stack_read_var_off_preserves_partial_spilled_zero(void)
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 71/81] selftests/bpf: Adjust tests of linked registers and spills
2026-10-26 2:59 ` [PATCH bpf 71/81] selftests/bpf: Adjust tests of linked registers and spills Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 19070 bytes --]
> selftests/bpf: Adjust tests of linked registers and spills
TL;DR:
- Test issue: rewritten stacksafe() spill-vs-STACK_MISC selftests may be
decided by r0, not stacksafe(), losing coverage
- Stale or wrong comments mentioning precision tracking
> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
> index f0077d951a25..2671323255cd 100644
> --- a/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_func.c
> @@ -110,64 +110,49 @@ __naked int aggregate_ret_static_uninit_fail(void)
> }
>
> static __naked __noinline u128 static_agg_precise(void)
> {
> asm volatile (
> "r0 = 0;"
> "r2 = 4;" /* second half; its value is made precise below */
> "exit;"
> );
> }
>
> SEC("tc")
> __load_if_JITed()
> __success __retval(0)
> -__log_level(2)
> -__msg("mark_precise: frame0: last_idx 5 first_idx 0 subseq_idx -1")
[ ... ]
> -__msg("mark_precise: frame1: regs=r2 stack= before 11: (b7) r2 = 4")
> __naked int aggregate_ret_static_precise(void)
> {
> asm volatile (
> "call %[static_agg_precise];"
> "r6 = r2;" /* derived from the aggregate's second half */
> "r6 &= 7;" /* keep it in [0, 7] to index the stack */
> "r1 = r10;"
> "r1 += -8;"
> "r1 += r6;" /* ptr += scalar marks r6 (hence R2) precise */
The patch removes the __log_level(2) and all of the mark_precise:
expectations from aggregate_ret_static_precise(), but the comments in the
programs still describe the removed behavior. Is "its value is made precise
below" still accurate?
Nothing marks a register precise any more, and nothing in the test checks it.
The test now only checks that the load succeeds and that the return value is
0.
The comment "ptr += scalar marks r6 (hence R2) precise" is repeated in
aggregate_ret_global_precise() further down in this file, and in
aggregate_ret_kfunc_precise() in aggregate_ret_kfunc.c:
> diff --git a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
> index 0361a7b42c9b..102156175ba0 100644
> --- a/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
> +++ b/tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
> @@ -22,31 +22,24 @@ void __kfunc_btf_root(void)
[ ... ]
> SEC("tc")
> __arch_x86_64 __arch_arm64
> __load_if_JITed()
> __success __retval(0)
> -__log_level(2)
> -__msg("mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1")
[ ... ]
> -__msg("mark_precise: frame0: regs=r2 stack= before 2: (85) call bpf_kfunc_call_test_i128")
> __naked int aggregate_ret_kfunc_precise(void)
> {
> asm volatile (
> "r1 = 1;"
> "r2 = 2;"
> "call %[bpf_kfunc_call_test_i128];"
> "r6 = r2;" /* second return half */
> "r6 &= 7;" /* keep it in [0, 7] to index the stack */
> "r1 = r10;"
> "r1 += -8;"
> "r1 += r6;" /* ptr += scalar marks r6 (hence R2) precise */
The same patch deletes the matching "force r0 to be precise" comments in
verifier_scalar_ids.c. Should these three comments be reworded to say what
the tests check now, for example "ptr += scalar needs the range of r6 (from
R2)", or dropped?
The same question applies to the comment inside
global_func15_tricky_pruning():
> diff --git a/tools/testing/selftests/bpf/progs/test_global_func15.c b/tools/testing/selftests/bpf/progs/test_global_func15.c
> index 201cc000b3f4..497f09dc8876 100644
> --- a/tools/testing/selftests/bpf/progs/test_global_func15.c
> +++ b/tools/testing/selftests/bpf/progs/test_global_func15.c
> @@ -2,55 +2,50 @@
[ ... ]
> SEC("cgroup_skb/ingress")
> -__log_level(2) __flag(BPF_F_TEST_STATE_FREQ)
> __failure
> -/* check that fallthrough code path marks r0 as precise */
> -__msg("mark_precise: frame0: regs=r0 stack= before 2: (b7) r0 = 1")
> -/* check that branch code path marks r0 as precise */
> -__msg("mark_precise: frame0: regs=r0 stack= before 0: (85) call bpf_get_prandom_u32#7")
> __msg("At program exit the register R0 has ")
> __naked int global_func15_tricky_pruning(void)
> {
> asm volatile (
> "call %[bpf_get_prandom_u32];"
> "if r0 s> 1000 goto 1f;"
> "r0 = 1;"
> "1:"
> "goto +0;" /* checkpoint */
> /* cgroup_skb/ingress program is expected to return [0, 1]
> * values, so branch above makes sure that in a fallthrough
> * case we have a valid 1 stored in R0 register, but in
> * a branch case we assign some random value to R0. So if
> * there is something wrong with precision tracking for R0 at
> * program exit, we might erroneously prune branch case,
> * because R0 in fallthrough case is imprecise (and thus any
> * value is valid from POV of verifier is_state_equal() logic)
> */
This comment explains the test in terms of "precision tracking for R0" and
the "is_state_equal() logic", while this hunk removes the annotations that
checked r0 precision. Is it meant to stay as it is?
> diff --git a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
> index 256547048cc4..92331e1a6c63 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c
[ ... ]
> @@ -132,18 +109,15 @@ __naked void linked_regs_bpf_x_dst(void)
>
> /* Same as linked_regs_bpf_k, but break one of the
> * links, note that r1 is absent from regs=... in __msg below.
> */
> SEC("socket")
> __success __log_level(2)
> -__msg("7: (0f) r3 += r0")
> -__msg("frame0: regs=r0 stack= before 6: (bf) r3 = r10")
> -__msg("frame0: parent state regs=r0 stack=:")
> -__msg("frame0: regs=r0 stack= before 5: (25) if r0 > 0x7 goto pc+0")
> -__msg("frame0: parent state regs=r0,r2 stack=:")
> -__flag(BPF_F_TEST_STATE_FREQ)
> +/* r1 is not there */
> +__msg("5: (25) if r0 > 0x7 goto pc+0 ; R0=" LINKED("1", 7, 0x7) " R2=" LINKED("1", 7, 0x7) "{{$}}")
> +__msg("8: (bf) r1 = r1 ; R1=0")
> __naked void linked_regs_broken_link(void)
The header comment still points to "regs=..." in the __msg below. That was
the "frame0: parent state regs=r0,r2 stack=:" line of precision tracking,
which this patch removes, and the new __msg contains no "regs=" text.
The patch also adds "/* r1 is not there */" a few lines further down, so the
test now has two comments saying the same thing, and one of them names a log
format that no longer exists. Should the header say something like "note
that R1 is absent from the registers printed after the jump in __msg below"
and the added comment be dropped?
[ ... ]
> @@ -595,21 +478,19 @@ __naked void check_ids_in_regsafe_2(void)
> * If such IDs *are* generated the 'l1' below would be reached in
> * two states:
> *
> * (1) r1{.id=A}, r2{.id=A}
> * (2) r1{.id=C}, r2{.id=C}
> *
> - * Thus forcing 'if r1 == r2' verification twice.
> + * neither of which is within the other one.
> */
Is "neither of which is within the other one" correct for the two states
listed just above it? (1) r1{.id=A}, r2{.id=A} and (2) r1{.id=C},
r2{.id=C} have the same shape, so check_scalar_ids()/check_ids() map A<->C
one to one and each state is within the other.
The listing was already inaccurate before this patch, but the new sentence
now contradicts it. The states the program would produce at l1 if constants
got ids use r3 and r4:
(1) r3{.id=A}, r4{.id=A} from "r3 = r1; r4 = r1;"
(2) r3{.id=B}, r4{.id=C} from "r3 = r1; r4 = r2;"
For that pair check_ids() fails in both directions, because one id would
have to map to two ids, and so "from 7 to 14: within" would not appear.
Could the listing be updated to those states? no_scalar_id_for_const32()
relies on this comment through "Same as no_scalar_id_for_const()".
[ ... ]
> @@ -743,109 +620,95 @@ __naked void ignore_unique_scalar_ids_old(void)
[ ... ]
> -/* Check that two different scalar IDs in a verified state can't be
> - * mapped to the same scalar ID in current state.
> +/* Check that two different scalar IDs in the state that is there can't be
> + * mapped to the same scalar ID in the state that arrives.
> */
> SEC("socket")
> __success __log_level(2)
> -/* The exit instruction should be reachable from two states,
> - * use two matches and "processed .. insns" to ensure this.
> - */
> -__msg("16: (95) exit")
> -__msg("16: (95) exit")
> -__msg("processed 22 insns")
> -__flag(BPF_F_TEST_STATE_FREQ)
> +__not_msg("from 10 to 11: within")
> +__msg("11: R0=0 R6=" ALONE(255, 0xff) " R7=" ALONE(255, 0xff))
> __naked void two_old_ids_one_cur_id(void)
> {
> asm volatile (
> /* Give unique scalar IDs to r{6,7} */
> "call %[bpf_ktime_get_ns];"
> "r0 &= 0xff;"
> "r6 = r0;"
> "r8 = r0;"
> "call %[bpf_ktime_get_ns];"
> "r0 &= 0xff;"
> "r7 = r0;"
> "r9 = r0;"
> "r0 = 0;"
> /* Maybe make r{6,7} IDs identical */
> - "if r6 > r7 goto l0_%=;"
> - "goto l1_%=;"
> -"l0_%=:"
> + "if r6 <= r7 goto l1_%=;"
> "r6 = r7;"
> "l1_%=:"
[ ... ]
> "r2 = r10;"
> "r2 += r6;"
> "r2 += r7;"
> /*
> * keep r8 and r9 live, otherwise r6->id and r7->id
> * will become singular and reset to zero before if r6 > r7
> */
Is this comment still accurate? The program has no "if r6 > r7" any more,
because the patch replaces it with "if r6 <= r7 goto l1_%=;".
The timing also differs under the new verifier. bpf_clear_singular_ids() is
now called from flow_arrive() when a state arrives at a block, that is at
the two arrivals at l1 after the "if r6 <= r7" jump, and not before the jump
as is_state_visited() used to do. Should this read something like "when the
states arrive at l1"?
> diff --git a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
> index 4d1374885f6d..b41f390df3af 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c
> @@ -515,21 +515,12 @@ __msg("4: (7b) *(u64 *)(r10 -16) = r0 ; R0=0 R10=fp0 fp-16=0")
> __msg("6: (71) r2 = *(u8 *)(r10 -1) ; R2=0 R10=fp0 fp-8=0")
> /* similarly, when R2 is assigned from spilled register, it is initially
> * imprecise, but will be marked precise later once it is used in precise context
> */
> __msg("10: (71) r2 = *(u8 *)(r10 -9) ; R2=0 R10=fp0 fp-16=0")
> __msg("11: (0f) r1 += r2")
> -__msg("mark_precise: frame0: last_idx 11 first_idx 0 subseq_idx -1")
[ ... ]
> -__msg("mark_precise: frame0: regs=r0 stack= before 3: (b7) r0 = 0")
> __naked void partial_stack_load_preserves_zeros(void)
This patch removes the mark_precise: expectations from
partial_stack_load_preserves_zeros(), but the comments above them still
describe precision tracking. They talk about "IMPRECISE" spills, and say
R2 "will be marked precise later once it is used in precise context".
There is no precise field in struct bpf_reg_state any more, and the
annotations that showed the later marking are gone. In the same file the
patch changes the matching wording in stack_load_preserves_const_precision()
and stack_load_preserves_const_precision_subreg() from "IMPRECISE fake
register spill" to "fake register spill" and from "spilled IMPRECISE ... const
reg" to "spilled const reg". Should the comments here be updated the same
way?
[ ... ]
> @@ -1115,159 +1045,142 @@ l0_%=: r1 >>= 32; \
[ ... ]
> -/* stacksafe(): check that stack spill of a precise scalar in old state
> - * is not considered equivalent to STACK_MISC in cur state.
> +/* stacksafe(): check that the state with STACK_MISC is not considered to be
> + * within the state with a stack spill of a known scalar, which is there.
> */
> SEC("socket")
> __success __log_level(2)
> -/* verifier should visit 'if r1 == 0x2a ...' two times:
> - * - once for path entry - label 2;
> - * - once for path entry - label 1 - label 2.
> - */
> -__msg("if r1 == 0x2a goto pc+0")
> -__msg("if r1 == 0x2a goto pc+0")
> -__msg("processed 15 insns")
> -__flag(BPF_F_TEST_STATE_FREQ)
> +/* what is loaded is not known to be 42 */
> +__msg("8: (79) r1 = *(u64 *)(r10 -8) ; R1=scalar() R10=fp0 fp-8=mmmmmmmm")
> +__msg("9: (15) if r1 == 0x2a goto pc+0 ; R1=scalar()")
> +__msg("processed 11 insns")
> __naked void old_precise_scalar_vs_cur_stack_misc(void)
> {
> asm volatile(
> /* get a random value for branching */
> "call %[bpf_ktime_get_ns];"
> "if r0 == 0 goto 1f;"
> /* conjure STACK_MISC at fp-8 */
> "call %[bpf_ktime_get_ns];"
> "*(u64*)(r10 - 8) = r0;"
> "*(u32*)(r10 - 4) = r0;"
> + "goto 2f;"
> +"1:"
> + /* conjure scalar at fp-8 */
> + "r0 = 42;"
> + "*(u64*)(r10 - 8) = r0;"
> "2:"
> - /* read fp-8, should not be considered safe on second visit */
> "r1 = *(u64*)(r10 - 8);"
> - /* use r1 in precise context */
> + /* the value of r1 matters */
> "if r1 == 42 goto +0;"
> "exit;"
After this rewrite, is the outcome decided by fp-8 or by r0? The comment
says the test checks "that the state with STACK_MISC is not considered to be
within the state with a stack spill of a known scalar", but with the new
block order it looks like r0 decides it.
The label 1 state (r0 = 42, fp-8 = spill of 42) reaches insn 8 first. The
fall-through state (r0 = unknown from bpf_ktime_get_ns(), fp-8 = mmmmmmmm)
arrives second. flow_merge() first calls flow_within(t, s), which is
bpf_state_subsumes(old = spill state, cur = MISC state). func_states_equal()
runs regsafe() on the live registers before it calls stacksafe(), and r0 is
live there because exit reads it:
regsafe(old r0 = 42, cur r0 = unknown) fails range_within()
so stacksafe() is never reached in the direction the comment describes. The
second check, flow_within(s, t), succeeds through:
if (env->allow_uninit_stack && old_type == STACK_MISC)
continue;
and that alone gives the expected "R1=scalar()" and "processed 11 insns".
Suppose stacksafe() wrongly accepted MISC in place of a spilled known
scalar, which is the unsound pruning the original test guarded against. Would
the r0 mismatch still reject the "within" and leave this test passing?
Before precision tracking was removed, r0 was imprecise at the checkpoint and
only fp-8 was precise (through "if r1 == 42"), so stacksafe() decided the
outcome. Now that every scalar is compared by range, "r0 = 42" decides it.
old_imprecise_scalar_vs_cur_stack_misc() has the same problem: its
"r0 = 42" label 1 state fails the first check on r0, so the 64-bit scalar
spill vs MISC handling (im == 0) in stacksafe() is never what decides it.
Would making r0 dead at label 2, for example with "r0 = 0;" before exit,
help? Then fp-8 would decide the first check, and a stacksafe() bug would
show up as "from 5 to 8: within" and R1=42.
[ ... ]
> @@ -1307,41 +1220,40 @@ __naked void stack_noperfmon_spill_32bit_onto_64bit_slot(void)
[ ... ]
> /*
> - * stacksafe(): check if 32-bit scalar spill in old state is considered
> - * equivalent to STACK_MISC in cur state.
> + * stacksafe(): the state with a 32-bit scalar spill is there, the state with
> + * STACK_MISC arrives. The first one is within the second one.
> * 32-bit scalar spill creates slot[0-3] = STACK_MISC, slot[4-7] = STACK_SPILL.
> * Without 32-bit spill support in stacksafe(), the STACK_SPILL vs STACK_MISC
> - * mismatch at slot[4] causes pruning to fail.
> + * mismatch at slot[4] keeps the states apart.
> */
> SEC("socket")
> __success __log_level(2)
> +__msg("8: R0=scalar() R10=fp0 fp-8=????mmmm")
> __msg("8: (79) r1 = *(u64 *)(r10 -8)")
> -__msg("8: safe")
> -__msg("processed 11 insns")
> -__flag(BPF_F_TEST_STATE_FREQ)
> +__msg("processed 10 insns")
> __naked void old_imprecise_scalar32_vs_cur_stack_misc(void)
> {
> asm volatile(
> /* get a random value for branching */
> "call %[bpf_ktime_get_ns];"
> "if r0 == 0 goto 1f;"
> - /* conjure 32-bit scalar spill at fp-8 */
> - "r0 = 42;"
> - "*(u32*)(r10 - 8) = r0;"
> - "goto 2f;"
> -"1:"
> /* conjure STACK_MISC at fp-8 */
> "call %[bpf_ktime_get_ns];"
> "*(u16*)(r10 - 8) = r0;"
> "*(u16*)(r10 - 6) = r0;"
> + "goto 2f;"
> +"1:"
> + /* conjure 32-bit scalar spill at fp-8 */
> + "r0 = 42;"
> + "*(u32*)(r10 - 8) = r0;"
> "2:"
Does this test still run the stacksafe() code it was written to guard?
7b4f1a29c704 ("selftests/bpf: Test 32-bit scalar spill pruning in
stacksafe()") added it for the im == 4 handling from 596bef1d718d ("bpf:
Support 32-bit scalar spills in stacksafe()").
With the branches swapped, the 32-bit spill state A (r0 = 42, slot[4-7] =
STACK_SPILL, slot[0-3] = STACK_INVALID) reaches insn 8 first. The
STACK_MISC state B (r0 = scalar(), fp-8 = ????mmmm) arrives from insn 5.
flow_merge() makes two checks.
The first is flow_within(A, B), i.e. bpf_state_subsumes(old = A, cur = B).
func_states_equal() compares the live registers before it calls stacksafe(),
and r0 is live at insn 8 because exit reads it:
regsafe(old r0 = 42, cur r0 = scalar()) fails range_within()
so stacksafe() is never called.
The second is flow_within(B, A), i.e. stacksafe(old = B, cur = A). Every
byte of B is STACK_INVALID or STACK_MISC, and stacksafe() skips both before
it reaches the scalar spill code:
if (old_type == STACK_INVALID || old_type == STACK_POISON)
continue;
if (env->allow_uninit_stack && old_type == STACK_MISC)
continue;
...
if (im == 0 || im == 4) {
old_reg = scalar_reg_for_stack(env, old_slot, im);
allow_uninit_stack must be true in this run, otherwise the 8-byte read of
????mmmm at insn 8 would be rejected and the test expects __success.
So the im == 4 handling is never executed here. If it were removed, A would
still be within B, and the log would still have
"8: R0=scalar() R10=fp0 fp-8=????mmmm" and "processed 10 insns". Is the
edited sentence "the STACK_SPILL vs STACK_MISC mismatch at slot[4] keeps the
states apart" still correct? Without 32-bit spill support the states would
be joined in exactly the same way, so reverting 596bef1d718d would not fail
this test.
The same root cause affects the neighbouring tests.
old_imprecise_scalar_vs_cur_stack_misc() (the 64-bit im == 0 path from
6efbde200bf3) is decided by the same two steps, and
old_precise_scalar_vs_cur_stack_misc() is decided by regsafe() on r0
(42 vs scalar()) rather than by stacksafe().
To get the coverage back, could the two branches leave the same r0, with the
32-bit spill holding an unbounded scalar and the STACK_MISC state reaching
the join after the spill state? Then "from N to 8: within" would be printed
only when stacksafe() accepts the spill. Another option is a
__caps_unpriv(CAP_BPF) variant, like the nearby stack_noperfmon tests, where
allow_uninit_stack is false and the MISC bytes go through
scalar_reg_for_stack(). No later patch in this series changes these tests or
stacksafe().
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 72/81] selftests/bpf: Adjust tests of pruning
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (70 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 71/81] selftests/bpf: Adjust tests of linked registers and spills Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 73/81] selftests/bpf: Adjust tests of loops Alexei Starovoitov
` (10 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier followed each path of a prog. It pruned a path when a state
that it saw before at the same insn covered the current state. regsafe(),
stacksafe() and range_within() made this decision, and these tests examine
them. Now the verifier joins the states that meet at a basic block. The
same functions compare a state that arrives at a block with the states that
are already there.
a state arrives at N the verifier follows the verifier joins
paths states
----------------------- ----------------------- -----------------------
a state at N covers it It stops the path. It drops the state.
"N: safe" "from M to N: within"
no state covers it It verifies the insns It joins the two states
from N again. and verifies the join.
Tests of the first case match the new message. Tests of the second case
match the join.
verifier_bounds_step.c examines the step of a scalar, which is the distance
between its possible values.
test the values that meet result
------------------------------ ----------------------------- -------
step_prune_hit_multiple [8,68] step 4, [4,130] step 2 step 2
step_prune_miss_non_multiple [0,30] step 2, [0,189] step 3 no step
step_prune_hit_const_on_line 6, and 0,3,6,... (step 3) step 3
step_prune_miss_const_off_line 7, and 0,3,6,... (step 3) no step
In the two "miss" tests the bounds of the second value contain the first
value. If range_within() ignores the step, the result keeps step 3 and does
not contain all values.
One verdict changes. short_loop1 has a loop with an unknown number of
iterations. The verifier followed the iterations until
"BPF program is too large". Now one state at the loop head covers all
iterations, and the prog loads.
"precision tracking for u32 spills, u64 fill" has 'w3 /= 0' two times: in a
filler insn, and after the jump that it tests. The verifier rejected the
prog at the filler insn, thus the test examined nothing. Replace the filler
insn.
Other changes in the expected logs:
- "precision tracking for u32 spill/fill" and
ringbuf_missing_release_callback get a different message for the same
problem.
- A state that the verifier drops does not count as a processed insn. Thus
"processed N insns" is lower by one.
- The changed tests drop BPF_F_TEST_STATE_FREQ, which has no effect
without pruning.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../testing/selftests/bpf/progs/dynptr_fail.c | 4 +-
.../bpf/progs/verifier_bounds_step.c | 73 ++++++++++---------
.../selftests/bpf/progs/verifier_live_stack.c | 23 +++---
.../bpf/progs/verifier_search_pruning.c | 30 +++-----
.../selftests/bpf/progs/verifier_spin_lock.c | 4 +-
.../selftests/bpf/progs/verifier_stack_ptr.c | 15 ++--
6 files changed, 68 insertions(+), 81 deletions(-)
diff --git a/tools/testing/selftests/bpf/progs/dynptr_fail.c b/tools/testing/selftests/bpf/progs/dynptr_fail.c
index c2247b38c849..3d895a90df25 100644
--- a/tools/testing/selftests/bpf/progs/dynptr_fail.c
+++ b/tools/testing/selftests/bpf/progs/dynptr_fail.c
@@ -127,7 +127,7 @@ static int missing_release_callback_fn(__u32 index, void *data)
/* A callback cannot return with the last dynptr for a referenced resource. */
SEC("?raw_tp")
-__failure __msg("cannot overwrite referenced dynptr")
+__failure __msg("Unreleased reference id=1")
int ringbuf_missing_release_callback(void *ctx)
{
bpf_loop(10, missing_release_callback_fn, NULL, 0);
@@ -782,7 +782,7 @@ int dynptr_pruning_overwrite(struct __sk_buff *ctx)
}
SEC("?tc")
-__success __msg("12: safe") __log_level(2)
+__success __msg("to 12: within") __log_level(2)
int dynptr_pruning_stacksafe(struct __sk_buff *ctx)
{
asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
index d824a5d48a39..51b703058cf7 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds_step.c
@@ -371,14 +371,17 @@ struct {
__type(value, struct step_val);
} step_map SEC(".maps");
-/* Old register [4..130, step 2] should prune cur register [8..68, step 4]. */
+/*
+ * Register [8..68, step 4] is within register [4..130, step 2]: one state stands for both
+ * and nothing is lost.
+ */
SEC("socket")
__success __log_level(2)
__msg("7: (27) r6 *= 4 ; R6=scalar({{.*}}umin32=8,{{.*}}umax32=68,{{.*}},step=0+4)")
__msg("10: (27) r7 *= 2 ; R7=scalar({{.*}}umin32=4,{{.*}}umax32=130,{{.*}},step=0+2)")
__msg("11: (25) if r0 > 0x2a goto pc+1")
-__msg("from 11 to 13: safe")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("13: R6=scalar({{.*}}umin32=4,{{.*}}umax32=130,{{.*}},step=0+2) R10=fp0")
+__msg("processed 18 insns")
__naked void step_prune_hit_multiple(void)
{
asm volatile (" \
@@ -394,11 +397,11 @@ __naked void step_prune_hit_multiple(void)
r7 += 2; \
r7 *= 2; \
if r0 > 42 goto 1f; /* can't predict */ \
- r6 = r7; /* step=2 explored first, step=4 explored next */ \
+ r6 = r7; /* step=2 here, step=4 on the other path */ \
1: r0 = r10; \
r6 = -r6; \
r0 += r6; \
- *(u8 *)(r0 + 0) = 7; /* force r6 precise */ \
+ *(u8 *)(r0 + 0) = 7; \
exit; \
" :
: __imm(bpf_get_prandom_u32)
@@ -406,17 +409,16 @@ __naked void step_prune_hit_multiple(void)
}
/*
- * Old register [0..189, step 3] should not prune cur register [0..30, step 2].
- * Bounds and tnum containment allow pruning; step divisibility must prevent it.
+ * Register [0..30, step 2] is not within register [0..189, step 3].
+ * Bounds and tnum say that it is; step divisibility must prevent it.
+ * The states are joined and the result has no step.
*/
SEC("socket")
__success __log_level(2)
__msg("6: (27) r6 *= 2 ; R6=scalar({{.*}}smin32=0,{{.*}}umax32=30,{{.*}},step=0+2)")
__msg("8: (27) r7 *= 3 ; R7=scalar({{.*}}smin32=0,{{.*}}umax32=189,{{.*}},step=0+3)")
__msg("9: (25) if r0 > 0x2a goto pc+1")
-__msg("17: (95) exit")
-__msg("from 9 to 11: R6=scalar({{.*}},step=0+2)")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("11: R6=scalar({{.*}}umax32=189,var_off=(0x0; 0xff)) R10=fp0")
__naked void step_prune_miss_non_multiple(void)
{
asm volatile (" \
@@ -430,13 +432,13 @@ __naked void step_prune_miss_non_multiple(void)
r7 &= 0x3f; \
r7 *= 3; \
if r0 > 42 goto 1f; /* can't predict */ \
- r6 = r7; /* step=3 explored first, step=2 explored next */ \
+ r6 = r7; /* step=3 here, step=2 on the other path */ \
1: \
r2 = r10; \
r6 += 1; \
r6 = -r6; \
r2 += r6; \
- *(u8 *)(r2 + 0) = 7; /* force r6 precise */ \
+ *(u8 *)(r2 + 0) = 7; \
r0 = 0; \
exit; \
" :
@@ -445,17 +447,17 @@ __naked void step_prune_miss_non_multiple(void)
}
/*
- * Constant current register lying on the cached line: cached is {0,3,6,...}
- * (step 3, base 0), current is the constant 6. range_within() takes the
- * constant branch: imod(6, 3) == base 0, so cur is on the line and the
- * (precise) state is pruned -> "safe".
+ * Constant register lying on the line of the other state: that is {0,3,6,...}
+ * (step 3, base 0), this is the constant 6. range_within() takes the
+ * constant branch: imod(6, 3) == base 0, so the constant is on the line and
+ * the state with the line stands for both.
*/
SEC("socket")
__success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("14: (27) r1 *= 3") /* cached path: step 3 line */
-__msg("16: (b7) r1 = 6") /* current path: const 6, on the line */
-__msg("17: safe") /* pruned at join: imod(6, 3) == 0 */
+__msg("16: (b7) r1 = 6") /* const 6, on the line */
+__msg("14: (27) r1 *= 3") /* step 3 line */
+__msg("from 15 to 17: joined")
+__msg("17: R1=scalar({{.*}},step=0+3) R7=") /* imod(6, 3) == 0 */
__naked void step_prune_hit_const_on_line(void)
{
asm volatile (" \
@@ -472,13 +474,13 @@ __naked void step_prune_hit_const_on_line(void)
r1 = r6; \
r1 &= 0xff; \
if r6 > 0 goto l_cur_%=; \
- r1 *= 3; /* old: step 3 */ \
+ r1 *= 3; /* step 3 */ \
goto l_join_%=; \
l_cur_%=: \
- r1 = 6; /* cur: const on line */ \
+ r1 = 6; /* const on line */ \
l_join_%=: \
r0 = r7; \
- r0 += r1; /* r1 forced precise */ \
+ r0 += r1; \
r2 = *(u8 *)(r0 + 0); \
l_out_%=: \
r0 = 0; \
@@ -491,22 +493,21 @@ l_out_%=: \
}
/*
- * Constant current register NOT on the cached line: cached is {0,3,6,...}
- * (step 3, base 0), current is the constant 7. imod(7, 3) == 1 != base 0,
- * so range_within() fails and the join is traversed again. A power-of-two
+ * Constant register NOT on the line of the other state: that is {0,3,6,...}
+ * (step 3, base 0), this is the constant 7. imod(7, 3) == 1 != base 0,
+ * so range_within() fails and the states are joined. A power-of-two
* step is avoided on purpose: with step 3 the tnum is loose enough to admit
* 7, so imod() is
* the sole check that rejects it. No failure shape is possible here: 7 is
- * within the line's bounds and tnum, and the cached line path already
- * verifies the whole outro, so an eager prune could not miss an error.
+ * within the line's bounds and tnum.
*/
SEC("socket")
__success __log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("14: (27) r1 *= 3") /* cached path: step 3 line */
-__msg("16: (b7) r1 = 7") /* current path: const 7, off the line */
-/* not pruned: current continues past the join with the constant offset 7 */
-__msg("19: R0=map_value({{.*}}imm=7) R1=7")
+__msg("16: (b7) r1 = 7") /* const 7, off the line */
+__msg("14: (27) r1 *= 3") /* step 3 line */
+/* the result stands for 7 too: it has no step */
+__msg("from 15 to 17: joined")
+__msg("17: R1=scalar({{.*}}var_off=(0x0; 0x3ff)) R7=")
__naked void step_prune_miss_const_off_line(void)
{
asm volatile (" \
@@ -523,13 +524,13 @@ __naked void step_prune_miss_const_off_line(void)
r1 = r6; \
r1 &= 0xff; \
if r6 > 0 goto l_cur_%=; \
- r1 *= 3; /* old: step 3 */ \
+ r1 *= 3; /* step 3 */ \
goto l_join_%=; \
l_cur_%=: \
- r1 = 7; /* cur: const off line */ \
+ r1 = 7; /* const off line */ \
l_join_%=: \
r0 = r7; \
- r0 += r1; /* r1 forced precise */ \
+ r0 += r1; \
r2 = *(u8 *)(r0 + 0); \
l_out_%=: \
r0 = 0; \
diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
index 5f9b11a81260..f70b23abc702 100644
--- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
@@ -380,9 +380,9 @@ __msg("10: (b7) r0 = 0{{$}}")
__msg("11: (05) goto pc+0{{$}}")
__msg("12: (95) exit")
/*
- * Checkpoint at goto +0 fires because fp0-8 is dead → state pruning.
+ * fp0-8 is dead after goto +0, so one state stands for both.
*/
-__msg("12: safe")
+__msg("from 11 to 12: within")
__naked void caller_stack_pruning(void)
{
asm volatile (
@@ -497,7 +497,6 @@ struct {
SEC("socket")
__failure __msg("invalid read from stack R2 off=-4096 size=8")
-__flag(BPF_F_TEST_STATE_FREQ)
__naked unsigned long caller_stack_write_tail_call(void)
{
asm volatile (
@@ -855,7 +854,7 @@ static __used __naked void maybe_clobber_key(void)
/*
* Demonstrate that subprog arg spill/reload breaks arg tracking,
- * inflating caller stack liveness and preventing state pruning.
+ * inflating caller stack liveness and keeping states apart.
*
* modifier2(fp-24) has two paths: one writes a scalar to *(r1+8)
* = caller fp-16, the other leaves it as zero. After modifier2
@@ -871,16 +870,15 @@ static __used __naked void maybe_clobber_key(void)
* the reader never touches it.
*
* Result: at modifier2's return point two states with different
- * fp-16 values cannot be pruned.
+ * fp-16 values are kept apart.
*
- * With correct (or old dynamic) liveness fp-16 is dead at that
- * point and the states prune → "6: safe" appears in the log.
+ * With correct liveness fp-16 is dead at that point and one state is
+ * within the other one.
*/
SEC("socket")
-__flag(BPF_F_TEST_STATE_FREQ)
__log_level(2)
__success
-__msg("6: safe")
+__msg("from 16 to 6: within")
__naked void spill_reload_inflates_stack_liveness(void)
{
asm volatile (
@@ -1372,13 +1370,12 @@ __naked void half_spi_clean_preserves_stack_zero(void)
*
* Static liveness marks fp-8 dead at merge. clean_verifier_state()
* converts path A's STACK_SPILL to STACK_INVALID. Path B has
- * STACK_INVALID. stacksafe() matches -> path B pruned -> "6: safe".
+ * STACK_INVALID. stacksafe() matches -> one state stands for both paths.
*/
SEC("socket")
-__flag(BPF_F_TEST_STATE_FREQ)
__success
__log_level(2)
-__msg("6: safe")
+__msg("from 5 to 6: within")
__naked void dead_spill_at_merge_enables_pruning(void)
{
asm volatile (
@@ -1390,7 +1387,7 @@ __naked void dead_spill_at_merge_enables_pruning(void)
"call %[bpf_get_prandom_u32];"
"r6 = *(u64 *)(r10 - 8);"
"l_skip%=:"
- /* fp-8 dead. Path B pruned here -> "6: safe" */
+ /* fp-8 dead. The states of both paths are the same here */
"r0 = r6;"
"exit;"
:
diff --git a/tools/testing/selftests/bpf/progs/verifier_search_pruning.c b/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
index f40e57251e94..f13ff9380947 100644
--- a/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
+++ b/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
@@ -96,7 +96,7 @@ l1_%=: exit; \
SEC("socket")
__description("varlen_map_value_access pruning")
__failure __msg("R0 unbounded memory access")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 unbounded memory access")
__flag(BPF_F_ANY_ALIGNMENT)
__naked void varlen_map_value_access_pruning(void)
{
@@ -193,7 +193,7 @@ l0_%=: exit; \
SEC("tracepoint")
__description("precision tracking for u32 spill/fill")
-__failure __msg("R0 min value is outside of the allowed memory range")
+__failure __msg("invalid access to map value, value_size=8 off=32 size=4")
__naked void tracking_for_u32_spill_fill(void)
{
asm volatile (" \
@@ -202,7 +202,7 @@ __naked void tracking_for_u32_spill_fill(void)
w6 = 32; \
if r0 == 0 goto l0_%=; \
w6 = 4; \
-l0_%=: /* Additional insns to introduce a pruning point. */\
+l0_%=: /* Additional insns to have one more place where paths meet. */\
call %[bpf_get_prandom_u32]; \
r3 = 0; \
r3 = 0; \
@@ -239,15 +239,11 @@ __naked void for_u32_spills_u64_fill(void)
call %[bpf_get_prandom_u32]; \
r6 = r0; \
w7 = 0xffffffff; \
- /* Additional insns to introduce a pruning point. */\
- r3 = 1; \
- r3 = 1; \
- r3 = 1; \
- r3 = 1; \
+ /* Additional insns to have a place where paths meet. */\
call %[bpf_get_prandom_u32]; \
if r0 == 0 goto l0_%=; \
r3 = 1; \
-l0_%=: w3 /= 0; \
+l0_%=: r3 = 1; \
/* u32 spills, u64 fill */ \
*(u32*)(r10 - 4) = r6; \
*(u32*)(r10 - 8) = r7; \
@@ -255,11 +251,7 @@ l0_%=: w3 /= 0; \
/* if r8 != X goto pc+1 r8 known in fallthrough branch */\
if r8 != 0xffffffff goto l1_%=; \
r3 = 1; \
-l1_%=: /* if r8 == X goto pc+1 condition always true on first\
- * traversal, so starts backtracking to mark r8 as requiring\
- * precision. r7 marked as needing precision. r6 not marked\
- * since it's not tracked. \
- */ \
+l1_%=: /* if r8 == X goto pc+1 */ \
if r8 == 0xffffffff goto l2_%=; \
/* fails if r8 correctly marked unknown after fill. */\
w3 /= 0; \
@@ -272,7 +264,8 @@ l2_%=: r0 = 0; \
SEC("socket")
__description("allocated_stack")
-__success __msg("processed 15 insns")
+/* every insn once: what is written to the stack on one path is dead where the paths meet */
+__success __msg("processed 14 insns")
__success_unpriv __msg_unpriv("") __log_level(1) __retval(0)
__naked void allocated_stack(void)
{
@@ -337,12 +330,9 @@ l0_%=: r1 = 42; \
: __clobber_all);
}
-/* Without checkpoint forcibly inserted at the back-edge a loop this
- * test would take a very long time to verify.
- */
+/* The number of trips is not known */
SEC("kprobe")
-__failure __log_level(4)
-__msg("BPF program is too large.")
+__success
__naked void short_loop1(void)
{
asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_spin_lock.c b/tools/testing/selftests/bpf/progs/verifier_spin_lock.c
index d9d7b05cf6d2..f4e7f47e4cdd 100644
--- a/tools/testing/selftests/bpf/progs/verifier_spin_lock.c
+++ b/tools/testing/selftests/bpf/progs/verifier_spin_lock.c
@@ -469,9 +469,9 @@ l3_%=: r1 = r7; \
SEC("cgroup/skb")
__description("spin_lock: regsafe() check_ids() similar id mappings")
-__success __msg("29: safe")
+__success __msg("to 29: within")
__failure_unpriv __msg_unpriv("")
-__log_level(2) __retval(0) __flag(BPF_F_TEST_STATE_FREQ)
+__log_level(2) __retval(0)
__naked void check_ids_similar_id_mappings(void)
{
asm volatile (" \
diff --git a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
index 7fa7bc207850..4c6e4465092f 100644
--- a/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
+++ b/tools/testing/selftests/bpf/progs/verifier_stack_ptr.c
@@ -587,14 +587,13 @@ __naked void stack_check_size_512_with_may_goto(void)
#endif
/*
- * Verify that old PTR_TO_STACK state is considered a super-set of
- * new PTR_TO_STACK state when new variable range is a sub-range
- * of the old range, e.g. old [-72, -16] vs new [-64, -16].
+ * Verify that a PTR_TO_STACK state is considered a super-set of
+ * another PTR_TO_STACK state when the variable range of that is a sub-range
+ * of its range, e.g. [-72, -16] vs [-64, -16].
*/
SEC("socket")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
/*
* r7 is widened to [-72, -16] at the loop header (insn 3),
* the loop body sees [-64, -8] after 'r7 += 8'
@@ -602,12 +601,12 @@ __flag(BPF_F_TEST_STATE_FREQ)
__msg("loop header at 3, widening r7 to -72..-16 step 8")
__msg("R7=fp(smin=smin32=-64,smax=smax32=-8")
/*
- * back-edge state is clamped to the remaining iterations and pruned
- * at the header, because [-64, -16] is within the widened [-72, -16]
+ * back-edge state is clamped to the remaining iterations and goes no further
+ * than the header, because [-64, -16] is within the widened [-72, -16]
*/
__msg("loop header at 3, clamping r7 to -64..-16 (s32 -64..-8) step 8")
-__msg("from 5 to 3: safe")
-__msg("processed 9 insns")
+__msg("from 5 to 3: within")
+__msg("processed 8 insns")
__naked void stack_ptr_subrange_in_loop(void)
{
asm volatile (" \
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 72/81] selftests/bpf: Adjust tests of pruning
2026-10-26 2:59 ` [PATCH bpf 72/81] selftests/bpf: Adjust tests of pruning Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 5228 bytes --]
> selftests/bpf: Adjust tests of pruning
TL;DR: Test issue: dead_spill_at_merge_enables_pruning and allocated_stack may
no longer test fp-8 liveness at the merge, since the stackless path arrives
first and stacksafe() passes trivially, so regressions could go undetected.
> diff --git a/tools/testing/selftests/bpf/progs/verifier_live_stack.c b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
> index 5f9b11a81260..f70b23abc702 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_live_stack.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_live_stack.c
[ ... ]
> @@ -1372,13 +1370,12 @@ __naked void half_spi_clean_preserves_stack_zero(void)
> *
> * Static liveness marks fp-8 dead at merge. clean_verifier_state()
> * converts path A's STACK_SPILL to STACK_INVALID. Path B has
> - * STACK_INVALID. stacksafe() matches -> path B pruned -> "6: safe".
> + * STACK_INVALID. stacksafe() matches -> one state stands for both paths.
> */
> SEC("socket")
> -__flag(BPF_F_TEST_STATE_FREQ)
> __success
> __log_level(2)
> -__msg("6: safe")
> +__msg("from 5 to 6: within")
> __naked void dead_spill_at_merge_enables_pruning(void)
> {
> asm volatile (
Does this test still check that fp-8 is dead at the merge?
When the jump state of "if r0 != 0 goto l_skip" reaches insn 6 first, path B
is the old state and path A (the one with the spill) is the state being
compared. Path B never touches the stack, so its allocated_stack is 0 and
the loop in stacksafe() has nothing to walk:
for (i = 0; i < old->allocated_stack; i++) {
stacksafe() then returns true whatever fp-8 holds in path A. r6 is the
constant 7 on both paths and r0 is dead, so "from 5 to 6: within" is
printed even if liveness reports fp-8 as live and bpf_clean_verifier_state()
leaves the STACK_SPILL of path A in place.
Before this patch the fallthrough path with the spill was explored first,
and path B's empty stack matched it only if fp-8 was dead ("6: safe"). The
comments updated here still give that reasoning:
clean_verifier_state() converts path A's STACK_SPILL to STACK_INVALID.
Path B has STACK_INVALID. stacksafe() matches
A regression that keeps fp-8 live at the merge (the scx_lavd state growth
this test models) would no longer fail it.
Could the spilling path be the one that arrives first, for example by
putting the spill block on the jump-target side the way
step_prune_hit_const_on_line does? Another option is to have path B store
a different kind of value to fp-8 as well.
I did not see a later patch in the series that changes this file.
> @@ -1390,7 +1387,7 @@ __naked void dead_spill_at_merge_enables_pruning(void)
> "call %[bpf_get_prandom_u32];"
> "r6 = *(u64 *)(r10 - 8);"
> "l_skip%=:"
> - /* fp-8 dead. Path B pruned here -> "6: safe" */
> + /* fp-8 dead. The states of both paths are the same here */
> "r0 = r6;"
> "exit;"
> :
[ ... ]
> diff --git a/tools/testing/selftests/bpf/progs/verifier_search_pruning.c b/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
> index f40e57251e94..f13ff9380947 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_search_pruning.c
[ ... ]
> @@ -272,7 +264,8 @@ l2_%=: r0 = 0; \
>
> SEC("socket")
> __description("allocated_stack")
> -__success __msg("processed 15 insns")
> +/* every insn once: what is written to the stack on one path is dead where the paths meet */
> +__success __msg("processed 14 insns")
> __success_unpriv __msg_unpriv("") __log_level(1) __retval(0)
> __naked void allocated_stack(void)
> {
Does "processed 14 insns" still depend on the stack written by one path
being dead where the paths meet?
The new comment explains the count by the dead stack, but the order in
which the states arrive decides the result. At "if r0 == 0 goto l0_%=" the
jump state reaches l0 before the block that spills to fp-8 and fp-9 has
run. That jump state has written no stack, so allocated_stack is 0.
The state that did the spills then arrives and is the one being compared.
stacksafe() loops over old->allocated_stack, which is 0, and returns true.
The registers that differ (r6, r7) are dead at l0 and r0 is 0 on both
paths, so the second state is "within" and every insn is processed once.
This holds even if fp-8 and fp-9 were treated as live, and even if
stacksafe() went back to rejecting the case where
old->allocated_stack > cur->allocated_stack.
That means the test no longer covers what 19e2dbb7dd97 ("bpf: improve
stacksafe state comparison") added it for, an old state with more allocated
stack than the current one where the extra slots are not live. Before this
patch the fallthrough path with the stack was the explored state, and the
count of 15 depended on those slots being dead.
Could the stack writes be put on the jump-target side of the branch so the
state that wrote the stack reaches l0 first?
I did not see a later patch in the series that changes this file.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 73/81] selftests/bpf: Adjust tests of loops
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (71 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 72/81] selftests/bpf: Adjust tests of pruning Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 74/81] selftests/bpf: Adjust tests of recursion Alexei Starovoitov
` (9 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Scalar evolution calculates how many times a loop runs. When it could not,
the verifier verified the loop one iteration at a time. If a state occurred
a second time, the verifier rejected the prog with
"infinite loop detected".
Now one state at the loop head covers all iterations of such a loop. This
shows that each iteration is safe, but not that the loop stops. Thus the
verifier adds may_goto to the loop, and may_goto stops the loop after
250 ms.
the loop before now
------------------------------ ----------------------- ---------------
has an exit, can spin forever infinite loop detected loads
has no exit infinite loop detected, loop at insn N
program is too large never exits
is unsafe in a later iteration rejected rejected, other
message
The tests run most of the progs of the first row: may_goto stops the loop,
and the prog returns 0. In the last row the verifier now sees the values of
all iterations as one range, for example -24 and -25. Thus it reports the
unsafe access with a different message.
Other changes:
- unsafe_on_2nd_iter also wrote 8 bytes into 'char buf[4]', and the
verifier finds this error first. Make the buffer 8 bytes long.
- bpf_loop_iter_limit_nested compared two sums with a set of values. The
verifier knows only their range. Compare them with the maximum.
- bounded_recursion loads and returns 4, because the kernel limits
recursion at run time.
In the log, the dump of the prog has no scc column, and the state of an
iterator has no depth. BPF_F_TEST_STATE_FREQ has no effect, thus many tests
drop it.
Add tests to verifier_scev.c for loops that scalar evolution can count. The
verifier verifies the body of such a loop one time, with ranges that cover
all iterations, or it verifies each iteration.
in the loop verified because
------------------------------ -------------- -------------------------
'x &= y', 'w7 = w7' one time x stays in [0, max]
'p += 8', p points to arena one time its offset is not tracked
'x >>= 1', 'x /= 10', 'x %= y' each iteration progs need the smaller x
'p += 1', p points to a buffer each iteration needs a constant offset
When the pointer to a buffer is also the counter, 'p < end', scalar
evolution counts the loop. Before, the verifier hit the limit of 1M insns.
'r <<= 32; r >>= 32' is a zero extension. 'r <<= 32; w >>= w2' with 32 in
w2 is not: the cpu shifts by 32 & 31, and the result is 0. The verifier
must reject progs that are unsafe only with this result. A register that is
the same constant in all iterations must stay a constant.
Add two progs to verifier_iterating_callbacks.c that have the call of
bpf_loop() at a loop head. The state that returns from the callback belongs
to the same iteration as the call.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../bpf/prog_tests/bpf_verif_scale.c | 4 +-
tools/testing/selftests/bpf/progs/iters.c | 112 +-
.../selftests/bpf/progs/iters_state_safety.c | 8 +-
.../selftests/bpf/progs/iters_testmod_seq.c | 12 +-
.../selftests/bpf/progs/verifier_cfg.c | 8 +-
.../selftests/bpf/progs/verifier_idoms.c | 5 +-
.../bpf/progs/verifier_iterating_callbacks.c | 110 +-
.../bpf/progs/verifier_loop_hierarchy.c | 125 +-
.../selftests/bpf/progs/verifier_loops1.c | 14 +-
.../selftests/bpf/progs/verifier_movsx.c | 2 +-
.../selftests/bpf/progs/verifier_scev.c | 1103 ++++++++++++++++-
11 files changed, 1277 insertions(+), 226 deletions(-)
diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c b/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c
index 902a70d5afeb..2dc37090d7a4 100644
--- a/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c
+++ b/tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c
@@ -154,9 +154,9 @@ void test_verif_scale_loop2()
scale_test("loop2.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
}
-void test_verif_scale_loop3_fail()
+void test_verif_scale_loop3()
{
- scale_test("loop3.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, true /* fails */);
+ scale_test("loop3.bpf.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
}
void test_verif_scale_loop4()
diff --git a/tools/testing/selftests/bpf/progs/iters.c b/tools/testing/selftests/bpf/progs/iters.c
index ae845ad747bf..fc2a5ab373a7 100644
--- a/tools/testing/selftests/bpf/progs/iters.c
+++ b/tools/testing/selftests/bpf/progs/iters.c
@@ -47,7 +47,7 @@ int iter_err_unsafe_c_loop(const void *ctx)
}
SEC("?raw_tp")
-__failure __msg("unbounded memory access")
+__failure __msg("math between map_value pointer and register with unbounded min value is not allowed")
int iter_err_unsafe_asm_loop(const void *ctx)
{
struct bpf_iter_num it;
@@ -878,19 +878,15 @@ __naked int delayed_read_mark(void)
SEC("?raw_tp")
__failure
-__msg("math between fp pointer and register with unbounded")
+__msg("21: (79) r8 = *(u64 *)(r0 +0)")
+__msg("misaligned stack access off -33+0 size 8")
__naked int delayed_precision_mark(void)
{
/* This is equivalent to C program below.
- * The test is similar to delayed_iter_mark but verifies that incomplete
- * precision don't fool verifier.
+ * The test is similar to delayed_iter_mark.
* The call to bpf_iter_num_next() is reachable with r7 values -16 and -32.
- * State with r7=-16 is visited first and follows r6 != 42 ... continue branch.
- * At this point iterator next() call is reached with r7 that has no read
- * and precision marks.
- * Loop body with r7=-32 would only be visited if verifier would decide to continue
- * with second loop iteration. Absence of precision mark on r7 might affect state
- * equivalent logic used for iterator convergence tracking.
+ * The loop body with r7=-32 is only looked at if the verifier goes on
+ * with the second iteration of the loop.
*
* r8 = 0
* fp[-16] = 0
@@ -957,22 +953,18 @@ __naked int delayed_precision_mark(void)
SEC("?raw_tp")
__failure
-__msg("math between fp pointer and register with unbounded")
-__flag(BPF_F_TEST_STATE_FREQ)
+/* -24 or -25 */
+__msg("misaligned stack access off (0xffffffffffffffe0; 0xf)+0 size 8")
__naked int loop_state_deps1(void)
{
/* This is equivalent to C program below.
*
* The case turns out to be tricky in a sense that:
- * - states with c=-25 are explored only on a second iteration
- * of the outer loop;
- * - states with read+precise mark on c are explored only on
- * second iteration of the inner loop and in a state which
- * is pushed to states stack first.
+ * - c is -25 only on a second iteration of the outer loop;
+ * - c is used only on a second iteration of the inner loop.
*
- * Depending on the details of iterator convergence logic
- * verifier might stop states traversal too early and miss
- * unsafe c=-25 memory access.
+ * The verifier might stop too early and miss unsafe c=-25
+ * memory access.
*
* j = iter_new(); // fp[-16]
* a = 0; // r6
@@ -1082,23 +1074,18 @@ __naked int loop_state_deps1(void)
SEC("?raw_tp")
__failure
-__msg("math between fp pointer and register with unbounded")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("34: (0f) r0 += r8")
+__msg("makes fp pointer be out of bounds")
__naked int loop_state_deps2(void)
{
/* This is equivalent to C program below.
*
* The case turns out to be tricky in a sense that:
- * - states with read+precise mark on c are explored only on a second
- * iteration of the first inner loop and in a state which is pushed to
- * states stack first.
- * - states with c=-25 are explored only on a second iteration of the
- * second inner loop and in a state which is pushed to states stack
- * first.
+ * - c is used only on a second iteration of the first inner loop;
+ * - c is -25 only on a second iteration of the second inner loop.
*
- * Depending on the details of iterator convergence logic
- * verifier might stop states traversal too early and miss
- * unsafe c=-25 memory access.
+ * The verifier might stop too early and miss unsafe c=-25
+ * memory access.
*
* j = iter_new(); // fp[-16]
* a = 0; // r6
@@ -1259,49 +1246,39 @@ __naked int loop_state_deps2(void)
SEC("?raw_tp")
__failure
-__msg("math between fp pointer and register with unbounded")
-__flag(BPF_F_TEST_STATE_FREQ)
+/* -24 or -25 */
+__msg("misaligned stack access off (0xffffffffffffffe0; 0xf)+0 size 8")
__naked int loop_state_deps3(void)
{
/* This is equivalent to a C program below.
*
- * if (random() != 24) { // assume false branch is placed first
+ * if (random() != 24) {
* i = iter_new(); // fp[-8]
* while (iter_next(i));
* iter_destroy(i);
* return;
* }
*
- * for (i = 10; i > 0; i--); // increase dfs_depth for child states
+ * for (i = 10; i > 0; i--);
*
* i = iter_new(); // fp[-8]
* b = -24; // r8
- * for (;;) { // checkpoint (L)
- * if (iter_next(i)) // checkpoint (N)
+ * for (;;) {
+ * if (iter_next(i))
* break;
- * if (random() == 77) { // assume false branch is placed first
+ * if (random() == 77) {
* *(u64 *)(r10 + b) = 7; // this is not safe when b == -25
* iter_destroy(i);
* return;
* }
- * if (random() == 42) { // assume false branch is placed first
+ * if (random() == 42) {
* b = -25;
* }
* }
* iter_destroy(i);
*
- * In case of a buggy verifier first loop might poison
- * env->cur_state->loop_entry with a state having 0 branches
- * and small dfs_depth. This would trigger NOT_EXACT states
- * comparison for some states within second loop.
- * Specifically, checkpoint (L) might be problematic if:
- * - branch with '*(u64 *)(r10 + b) = 7' is not explored yet;
- * - checkpoint (L) is first reached in state {b=-24};
- * - traversal is pruned at checkpoint (N) setting checkpoint's (L)
- * branch count to 0, thus making it eligible for use in pruning;
- * - checkpoint (L) is next reached in state {b=-25},
- * this would cause NOT_EXACT comparison with a state {b=-24}
- * while 'b' is not marked precise yet.
+ * What the verifier learned in the first two loops must not make it
+ * take {b=-25} at the head of the third one for {b=-24}.
*/
asm volatile (
"call %[bpf_get_prandom_u32];"
@@ -1884,7 +1861,6 @@ __naked int absent_mark_in_the_middle_state2(void)
}
SEC("?raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
__failure __msg("misaligned stack access off -31+0 size 8")
__naked int absent_mark_in_the_middle_state3(void)
{
@@ -2014,8 +1990,8 @@ static int loop1_wrapper(void)
* but adapted for use with bpf_loop().
*/
SEC("raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
-__failure __msg("math between fp pointer and register with unbounded min value is not allowed")
+/* -32 or -31 */
+__failure __msg("misaligned stack access off (0xffffffffffffffe0; 0x1)+0 size 8")
__naked void absent_mark_in_the_middle_state4(void)
{
/*
@@ -2159,8 +2135,7 @@ static int loop_cb5(int i, __u64 *ctx)
}
SEC("raw_tp")
-__flag(BPF_F_TEST_STATE_FREQ)
-__failure __msg("memory access is {{.*}} and is outside of the object of size 64")
+__failure __msg("invalid access to map value, value_size=64 off=402000 size=4")
__naked void loop_counter_precision_2nd_iter(void)
{
asm volatile (
@@ -2174,9 +2149,8 @@ __naked void loop_counter_precision_2nd_iter(void)
"r3 += -8;"
"r4 = 0;"
/*
- * Explore with nr_loops=1 on a first path and nr_loops=2 on a second path.
- * Buggy verifier did not propagate r1 precision properly,
- * and thus checkpoints created for nr_loops=1 case matched nr_loops=2 case.
+ * nr_loops is 1 on one path and 2 on the other.
+ * The second call of the callback must not be lost.
*/
"call %[bpf_loop];"
"r0 = 0;"
@@ -2190,12 +2164,12 @@ __naked void loop_counter_precision_2nd_iter(void)
/*
* An iterator that is destroyed and created again inside its loop starts from
- * the beginning and the loop never ends. In the first four progs the first
- * prune point after bpf_iter_num_new() is the call of bpf_iter_num_next().
+ * the beginning and the loop never ends. bpf_iter_num_next() is not what stops such
+ * loop: the verifier adds may_goto. The progs spin until it's out of budget.
*/
SEC("socket")
__description("iter: remake in the loop, jump to next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
__naked void iter_remake_jmp(void)
{
asm volatile (
@@ -2256,7 +2230,7 @@ __used __naked static void iter_remake(void)
SEC("socket")
__description("iter: remake in a callee, jump to next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
__naked void iter_remake_callee_jmp(void)
{
asm volatile (
@@ -2295,7 +2269,7 @@ __naked void iter_remake_callee_jmp(void)
SEC("socket")
__description("iter: remake after next, jump to next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
__naked void iter_remake_after_next_jmp(void)
{
asm volatile (
@@ -2336,7 +2310,7 @@ __naked void iter_remake_after_next_jmp(void)
SEC("socket")
__description("iter: two iterators, remake each after its next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
__naked void iter_remake_two_jmp(void)
{
asm volatile (
@@ -2398,7 +2372,7 @@ __naked void iter_remake_two_jmp(void)
/* Same loops with a prune point in front of bpf_iter_num_next() */
SEC("socket")
__description("iter: remake in the loop")
-__failure __msg("infinite loop detected")
+__success __retval(0)
__flag(BPF_F_TEST_STATE_FREQ)
__naked void iter_remake_inline(void)
{
@@ -2439,7 +2413,7 @@ __naked void iter_remake_inline(void)
SEC("socket")
__description("iter: remake in a callee")
-__failure __msg("infinite loop detected")
+__success __retval(0)
__flag(BPF_F_TEST_STATE_FREQ)
__naked void iter_remake_callee(void)
{
@@ -2475,7 +2449,7 @@ __naked void iter_remake_callee(void)
SEC("socket")
__description("iter: remake after next")
-__failure __msg("infinite loop detected")
+__success __retval(0)
__flag(BPF_F_TEST_STATE_FREQ)
__naked void iter_remake_after_next(void)
{
@@ -2515,7 +2489,7 @@ __naked void iter_remake_after_next(void)
SEC("socket")
__description("iter: remake when drained")
-__failure __msg("infinite loop detected")
+__success __retval(0)
__flag(BPF_F_TEST_STATE_FREQ)
__naked void iter_remake_drained(void)
{
diff --git a/tools/testing/selftests/bpf/progs/iters_state_safety.c b/tools/testing/selftests/bpf/progs/iters_state_safety.c
index b2ba4609a31e..fa9604f0a173 100644
--- a/tools/testing/selftests/bpf/progs/iters_state_safety.c
+++ b/tools/testing/selftests/bpf/progs/iters_state_safety.c
@@ -30,7 +30,7 @@ int force_clang_to_emit_btf_for_externs(void *ctx)
SEC("?raw_tp")
__success __log_level(2)
-__msg("fp-8=iter_num(id=1,state=active,depth=0)")
+__msg("fp-8=iter_num(id=1,state=active)")
int create_and_destroy(void *ctx)
{
struct bpf_iter_num iter;
@@ -207,8 +207,8 @@ static __noinline void subprog_with_iter(void)
SEC("?raw_tp")
__failure
/* ensure there was a call to subprog, which might happen without __noinline */
-__msg("returning from callee:")
-__msg("Unreleased reference id=1")
+__msg("0: (85) call pc+2")
+__msg("Unreleased reference id=2 alloc_insn=7")
int leak_iter_from_subprog_fail(void *ctx)
{
subprog_with_iter();
@@ -218,7 +218,7 @@ int leak_iter_from_subprog_fail(void *ctx)
SEC("?raw_tp")
__success __log_level(2)
-__msg("fp-8=iter_num(id=1,state=active,depth=0)")
+__msg("fp-8=iter_num(id=1,state=active)")
int valid_stack_reuse(void *ctx)
{
struct bpf_iter_num iter;
diff --git a/tools/testing/selftests/bpf/progs/iters_testmod_seq.c b/tools/testing/selftests/bpf/progs/iters_testmod_seq.c
index d00888f6687a..88ce4b7a9c33 100644
--- a/tools/testing/selftests/bpf/progs/iters_testmod_seq.c
+++ b/tools/testing/selftests/bpf/progs/iters_testmod_seq.c
@@ -20,8 +20,8 @@ __s64 res_empty;
SEC("raw_tp/sys_enter")
__success __log_level(2)
-__msg("fp-16=iter_testmod_seq(id=1,state=active,depth=0)")
-__msg("fp-16=iter_testmod_seq(id=1,state=drained,depth=0)")
+__msg("fp-16=iter_testmod_seq(id=1,state=active)")
+__msg("fp-16=iter_testmod_seq(id=1,state=drained)")
__msg("call bpf_iter_testmod_seq_destroy")
int testmod_seq_empty(const void *ctx)
{
@@ -38,8 +38,8 @@ __s64 res_full;
SEC("raw_tp/sys_enter")
__success __log_level(2)
-__msg("fp-16=iter_testmod_seq(id=1,state=active,depth=0)")
-__msg("fp-16=iter_testmod_seq(id=1,state=drained,depth=0)")
+__msg("fp-16=iter_testmod_seq(id=1,state=active)")
+__msg("fp-16=iter_testmod_seq(id=1,state=drained)")
__msg("call bpf_iter_testmod_seq_destroy")
int testmod_seq_full(const void *ctx)
{
@@ -58,8 +58,8 @@ static volatile int zero = 0;
SEC("raw_tp/sys_enter")
__success __log_level(2)
-__msg("fp-16=iter_testmod_seq(id=1,state=active,depth=0)")
-__msg("fp-16=iter_testmod_seq(id=1,state=drained,depth=0)")
+__msg("fp-16=iter_testmod_seq(id=1,state=active)")
+__msg("fp-16=iter_testmod_seq(id=1,state=drained)")
__msg("call bpf_iter_testmod_seq_destroy")
int testmod_seq_truncated(const void *ctx)
{
diff --git a/tools/testing/selftests/bpf/progs/verifier_cfg.c b/tools/testing/selftests/bpf/progs/verifier_cfg.c
index b429fa7a08f5..e97e89ba3d68 100644
--- a/tools/testing/selftests/bpf/progs/verifier_cfg.c
+++ b/tools/testing/selftests/bpf/progs/verifier_cfg.c
@@ -138,8 +138,8 @@ l0_%=: r1 = r0; \
SEC("socket")
__description("conditional loop")
-__failure __msg("infinite loop detected")
-__msg_unpriv("back-edge")
+__success
+__failure_unpriv __msg_unpriv("back-edge")
__naked void conditional_loop(void)
{
asm volatile (" \
@@ -175,7 +175,7 @@ l1_%=: r8 += r4; \
SEC("socket")
__description("unconditional loop after conditional jump")
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn 2 never exits")
__failure_unpriv __msg_unpriv("back-edge from insn 3 to 2")
__naked void uncond_loop_after_cond_jmp(void)
{
@@ -201,7 +201,7 @@ static unsigned long never_ending_subprog()
SEC("socket")
__description("unconditional loop after conditional jump")
/* infinite loop is detected *after* check_cfg() */
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn 6 never exits")
__naked void uncond_loop_in_subprog_after_cond_jmp(void)
{
asm volatile (" \
diff --git a/tools/testing/selftests/bpf/progs/verifier_idoms.c b/tools/testing/selftests/bpf/progs/verifier_idoms.c
index fdee6bced77b..78cc8650923a 100644
--- a/tools/testing/selftests/bpf/progs/verifier_idoms.c
+++ b/tools/testing/selftests/bpf/progs/verifier_idoms.c
@@ -10,7 +10,7 @@
* The immediate dominator is printed by log_program() as the numeric column
* immediately before the "<insn#>:" field of every "Program dump" line:
*
- * Program dump (scc? loop_header? idom insn#: live_regs_before):
+ * Program dump (loop_header? idom insn#: live_regs_before):
* -1 0: ....... (b7) r0 = 0 <- idom(0) = -1 (subprog entry)
* 0 1: ....... (07) r0 += 1 <- idom(1) = 0
* ^^^^^^
@@ -311,13 +311,12 @@ __naked void entry_is_loop_header(void)
* infinite loop, but the dump (and idoms) are printed before that.
*/
SEC("socket")
-__failure
+__success
__log_level(2)
__msg("Program dump")
__msg("{{.*}} -1 0: {{.*}} (85) call bpf_get_prandom_u32#7")
__msg("{{.*}} 0 1: {{.*}} (a5) if r0 < 0xa goto pc-1")
__msg("{{.*}} 1 2: {{.*}} (95) exit")
-__msg("infinite loop detected")
__naked void self_loop(void)
{
asm volatile (" \
diff --git a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
index a76bce7278e7..a48b48fb2e51 100644
--- a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
+++ b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
@@ -45,7 +45,7 @@ SEC("?raw_tp")
__failure __msg("R1 type=scalar expected=fp")
int unsafe_on_2nd_iter(void *unused)
{
- char buf[4];
+ char buf[8];
struct buf_context loop_ctx = { .buf = buf };
bpf_loop(100, unsafe_on_2nd_iter_cb, &loop_ctx, 0);
@@ -94,7 +94,7 @@ static int loop_detection_cb(__u32 idx, struct num_context *ctx)
}
SEC("?raw_tp")
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn 12 never exits")
int loop_detection(void *unused)
{
struct num_context loop_ctx = { .i = 0 };
@@ -246,10 +246,9 @@ int bpf_loop_iter_limit_nested(void *unused)
/* This makes 'c' zero, but neither clang nor verifier know it. */
c /= 10;
/* Make sure that verifier does not visit 'impossible' states:
- * enumerate all possible callback visit masks.
+ * one more call of any of the callbacks gives more than that.
*/
- if (a != 0 && a != 1 && a != 11 && a != 101 && a != 111 &&
- b != 0 && b != 1 && b != 11 && b != 101 && b != 111)
+ if (a > 111 || b > 111)
asm volatile ("r0 /= 0;" ::: "r0");
return 1000 * a + b + c;
}
@@ -291,9 +290,106 @@ static __naked void iter_limit_bug_cb(void)
);
}
+static __naked __noinline __used void set_flag_cb(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "*(u64 *)(r2 + 0) = r1;"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+/*
+ * bpf_loop() is the head of a loop whose iterations are looked at one by one, because of r1.
+ * The callback comes back to it in the same iteration, the last one included.
+ */
+SEC("socket")
+__failure __msg("frame pointer is read only")
+__naked void callback_returns_to_loop_head(void)
+{
+ asm volatile (
+ "r7 = 0;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = 1;"
+ "r2 = %[set_flag_cb] ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "1:"
+ "call %[bpf_loop];"
+ "r1 = *(u64 *)(r10 - 8);"
+ "if r1 == 0 goto 2f;"
+ "if r7 != 3 goto 2f;"
+ "r10 = 0;"
+ "2:"
+ "r7 += 1;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = r7;"
+ "r2 = %[set_flag_cb] ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "if r7 < 4 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(set_flag_cb),
+ __imm(bpf_loop)
+ : __clobber_all
+ );
+}
+
+/* r7 is 4 when the loop is left, and not before */
+SEC("socket")
+__success __log_level(2)
+__msg("r0 = r7 {{.*}}R7=4")
+__naked void callback_returns_to_loop_head_ok(void)
+{
+ asm volatile (
+ "r7 = 0;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = 1;"
+ "r2 = %[set_flag_cb] ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "1:"
+ "call %[bpf_loop];"
+ "r1 = *(u64 *)(r10 - 8);"
+ "if r1 == 0 goto 2f;"
+ "if r7 != 4 goto 2f;"
+ "r10 = 0;"
+ "2:"
+ "r7 += 1;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = r7;"
+ "r2 = %[set_flag_cb] ll;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "if r7 < 4 goto 1b;"
+ "r0 = r7;"
+ "exit;"
+ :
+ : __imm_addr(set_flag_cb),
+ __imm(bpf_loop)
+ : __clobber_all
+ );
+}
+
+/*
+ * may_goto doesn't leave the loop, so it's not what stops it. The address is greater than 10:
+ * the loop spins until may_goto that the verifier adds ends the prog.
+ */
int tmp_var;
SEC("socket")
-__failure __msg("infinite loop detected at insn 2")
+__success __retval(0)
__naked void jgt_imm64_and_may_goto(void)
{
asm volatile (" \
@@ -310,7 +406,7 @@ l0_%=: .byte 0xe5; /* may_goto */ \
}
SEC("socket")
-__failure __msg("infinite loop detected at insn 1")
+__success __retval(0)
__naked void may_goto_self(void)
{
asm volatile (" \
diff --git a/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c b/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c
index 0608e983d7fc..4f4e04d50bc7 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loop_hierarchy.c
@@ -22,10 +22,10 @@ __success
__log_level(2)
__msg("loop at 1{{$}}")
__msg("Program dump")
-__msg(" -1 0: {{.*}} (b7) r0 = 0")
-__msg(" 1 0 1: {{.*}} (07) r0 += 1")
-__msg(" 1 1 1 2: {{.*}} (a5) if r0 < 0xa goto pc-2")
-__msg(" 2 3: {{.*}} (95) exit")
+__msg(" -1 0: {{.*}} (b7) r0 = 0")
+__msg(" 0 1: {{.*}} (07) r0 += 1")
+__msg(" 1 1 2: {{.*}} (a5) if r0 < 0xa goto pc-2")
+__msg(" 2 3: {{.*}} (95) exit")
__naked void loop_single(void)
{
asm volatile (" \
@@ -43,13 +43,13 @@ __log_level(2)
__msg("loop at 1{{$}}")
__msg("loop at 4{{$}}")
__msg("Program dump")
-__msg(" -1 0: {{.*}} (b7) r0 = 0")
-__msg(" 2 0 1: {{.*}} (07) r0 += 1")
-__msg(" 2 1 1 2: {{.*}} (a5) if r0 < 0xa goto pc-2")
-__msg(" 2 3: {{.*}} (b7) r1 = 0")
-__msg(" 1 3 4: {{.*}} (07) r1 += 1")
-__msg(" 1 4 4 5: {{.*}} (a5) if r1 < 0xa goto pc-2")
-__msg(" 5 6: {{.*}} (95) exit")
+__msg(" -1 0: {{.*}} (b7) r0 = 0")
+__msg(" 0 1: {{.*}} (07) r0 += 1")
+__msg(" 1 1 2: {{.*}} (a5) if r0 < 0xa goto pc-2")
+__msg(" 2 3: {{.*}} (b7) r1 = 0")
+__msg(" 3 4: {{.*}} (07) r1 += 1")
+__msg(" 4 4 5: {{.*}} (a5) if r1 < 0xa goto pc-2")
+__msg(" 5 6: {{.*}} (95) exit")
__naked void loop_two_independent(void)
{
asm volatile (" \
@@ -70,13 +70,13 @@ __log_level(2)
__msg("loop at 1{{$}}")
__msg("loop at 3, nested in 1")
__msg("Program dump")
-__msg(" -1 0: {{.*}} (b7) r0 = 0")
-__msg(" 1 0 1: {{.*}} (07) r0 += 1") /* outer loop header */
-__msg(" 1 1 1 2: {{.*}} (b7) r1 = 0") /* outer loop insn */
-__msg(" 1 1 2 3: {{.*}} (07) r1 += 1") /* inner loop header */
-__msg(" 1 3 3 4: {{.*}} (a5) if r1 < 0x5 goto pc-2") /* inner loop insn */
-__msg(" 1 1 4 5: {{.*}} (a5) if r0 < 0xa goto pc-5") /* outer loop insn */
-__msg(" 5 6: {{.*}} (95) exit")
+__msg(" -1 0: {{.*}} (b7) r0 = 0")
+__msg(" 0 1: {{.*}} (07) r0 += 1") /* outer loop header */
+__msg(" 1 1 2: {{.*}} (b7) r1 = 0") /* outer loop insn */
+__msg(" 1 2 3: {{.*}} (07) r1 += 1") /* inner loop header */
+__msg(" 3 3 4: {{.*}} (a5) if r1 < 0x5 goto pc-2") /* inner loop insn */
+__msg(" 1 4 5: {{.*}} (a5) if r0 < 0xa goto pc-5") /* outer loop insn */
+__msg(" 5 6: {{.*}} (95) exit")
__naked void loop_nested(void)
{
asm volatile (" \
@@ -95,12 +95,12 @@ SEC("socket")
__success
__log_level(2)
__msg("Program dump")
-__msg(" -1 0: {{.*}} (85) call bpf_get_prandom_u32")
-__msg(" 0 1: {{.*}} (25) if r0 > 0x0 goto pc+2")
-__msg(" 1 2: {{.*}} (b7) r0 = 2")
-__msg(" 2 3: {{.*}} (05) goto pc+1")
-__msg(" 1 4: {{.*}} (b7) r0 = 3")
-__msg(" 1 5: {{.*}} (95) exit")
+__msg(" -1 0: {{.*}} (85) call bpf_get_prandom_u32")
+__msg(" 0 1: {{.*}} (25) if r0 > 0x0 goto pc+2")
+__msg(" 1 2: {{.*}} (b7) r0 = 2")
+__msg(" 2 3: {{.*}} (05) goto pc+1")
+__msg(" 1 4: {{.*}} (b7) r0 = 3")
+__msg(" 1 5: {{.*}} (95) exit")
__naked void fwd_edges_no_loop(void)
{
asm volatile (" \
@@ -123,16 +123,16 @@ __msg("loop at 1{{$}}")
__msg("loop at 3, nested in 1")
__msg("loop at 6, nested in 1")
__msg("Program dump")
-__msg(" -1 0: {{.*}} (b7) r0 = 0")
-__msg(" 1 0 1: {{.*}} (07) r0 += 1")
-__msg(" 1 1 1 2: {{.*}} (b7) r1 = 0")
-__msg(" 1 1 2 3: {{.*}} (07) r1 += 1")
-__msg(" 1 3 3 4: {{.*}} (a5) if r1 < 0x5 goto pc-2")
-__msg(" 1 1 4 5: {{.*}} (b7) r2 = 0")
-__msg(" 1 1 5 6: {{.*}} (07) r2 += 1")
-__msg(" 1 6 6 7: {{.*}} (a5) if r2 < 0x5 goto pc-2")
-__msg(" 1 1 7 8: {{.*}} (a5) if r0 < 0xa goto pc-8")
-__msg(" 8 9: {{.*}} (95) exit")
+__msg(" -1 0: {{.*}} (b7) r0 = 0")
+__msg(" 0 1: {{.*}} (07) r0 += 1")
+__msg(" 1 1 2: {{.*}} (b7) r1 = 0")
+__msg(" 1 2 3: {{.*}} (07) r1 += 1")
+__msg(" 3 3 4: {{.*}} (a5) if r1 < 0x5 goto pc-2")
+__msg(" 1 4 5: {{.*}} (b7) r2 = 0")
+__msg(" 1 5 6: {{.*}} (07) r2 += 1")
+__msg(" 6 6 7: {{.*}} (a5) if r2 < 0x5 goto pc-2")
+__msg(" 1 7 8: {{.*}} (a5) if r0 < 0xa goto pc-8")
+__msg(" 8 9: {{.*}} (95) exit")
__naked void loop_nested_siblings(void)
{
asm volatile (" \
@@ -157,16 +157,16 @@ __msg("loop at 1{{$}}")
__msg("loop at 3, nested in 1")
__msg("loop at 5, nested in 3")
__msg("Program dump")
-__msg(" -1 0: {{.*}} (b7) r0 = 0")
-__msg(" 1 0 1: {{.*}} (07) r0 += 1")
-__msg(" 1 1 1 2: {{.*}} (b7) r1 = 0")
-__msg(" 1 1 2 3: {{.*}} (07) r1 += 1")
-__msg(" 1 3 3 4: {{.*}} (b7) r2 = 0")
-__msg(" 1 3 4 5: {{.*}} (07) r2 += 1")
-__msg(" 1 5 5 6: {{.*}} (a5) if r2 < 0x3 goto pc-2")
-__msg(" 1 3 6 7: {{.*}} (a5) if r1 < 0x5 goto pc-5")
-__msg(" 1 1 7 8: {{.*}} (a5) if r0 < 0xa goto pc-8")
-__msg(" 8 9: {{.*}} (95) exit")
+__msg(" -1 0: {{.*}} (b7) r0 = 0")
+__msg(" 0 1: {{.*}} (07) r0 += 1")
+__msg(" 1 1 2: {{.*}} (b7) r1 = 0")
+__msg(" 1 2 3: {{.*}} (07) r1 += 1")
+__msg(" 3 3 4: {{.*}} (b7) r2 = 0")
+__msg(" 3 4 5: {{.*}} (07) r2 += 1")
+__msg(" 5 5 6: {{.*}} (a5) if r2 < 0x3 goto pc-2")
+__msg(" 3 6 7: {{.*}} (a5) if r1 < 0x5 goto pc-5")
+__msg(" 1 7 8: {{.*}} (a5) if r0 < 0xa goto pc-8")
+__msg(" 8 9: {{.*}} (95) exit")
__naked void loop_three_levels(void)
{
asm volatile (" \
@@ -189,14 +189,14 @@ __success
__log_level(2)
__msg("loop at 1")
__msg("Program dump")
-__msg(" -1 0: {{.*}} (b7) r0 = 0")
-__msg(" 1 0 1: {{.*}} (07) r0 += 1")
-__msg(" 1 1 1 2: {{.*}} (bf) r1 = r0")
-__msg(" 1 1 2 3: {{.*}} (25) if r1 > 0x5 goto pc+1")
-__msg(" 1 1 3 4: {{.*}} (b7) r1 = 1")
-__msg(" 1 1 3 5: {{.*}} (0f) r0 += r1")
-__msg(" 1 1 5 6: {{.*}} (a5) if r0 < 0x64 goto pc-6")
-__msg(" 6 7: {{.*}} (95) exit")
+__msg(" -1 0: {{.*}} (b7) r0 = 0")
+__msg(" 0 1: {{.*}} (07) r0 += 1")
+__msg(" 1 1 2: {{.*}} (bf) r1 = r0")
+__msg(" 1 2 3: {{.*}} (25) if r1 > 0x5 goto pc+1")
+__msg(" 1 3 4: {{.*}} (b7) r1 = 1")
+__msg(" 1 3 5: {{.*}} (0f) r0 += r1")
+__msg(" 1 5 6: {{.*}} (a5) if r0 < 0x64 goto pc-6")
+__msg(" 6 7: {{.*}} (95) exit")
__naked void loop_with_if_else(void)
{
asm volatile (" \
@@ -217,15 +217,15 @@ __success
__log_level(2)
__msg("loop at 3, irreducible")
__msg("Program dump")
-__msg(" -1 0: {{.*}} (85) call bpf_get_prandom_u32#7")
-__msg(" 0 1: {{.*}} (b7) r1 = 0")
-__msg(" 1 2: {{.*}} (25) if r0 > 0x5 goto pc+2")
-__msg(" 1 2 3: {{.*}} (b7) r1 = 1")
-__msg(" 1 3 3 4: {{.*}} (05) goto pc+1")
-__msg(" 2 5: {{.*}} (b7) r1 = 2")
-__msg(" 1 3 2 6: {{.*}} (0f) r0 += r1")
-__msg(" 1 3 6 7: {{.*}} (a5) if r0 < 0x10 goto pc-5")
-__msg(" 7 8: {{.*}} (95) exit")
+__msg(" -1 0: {{.*}} (85) call bpf_get_prandom_u32#7")
+__msg(" 0 1: {{.*}} (b7) r1 = 0")
+__msg(" 1 2: {{.*}} (25) if r0 > 0x5 goto pc+2")
+__msg(" 2 3: {{.*}} (b7) r1 = 1")
+__msg(" 3 3 4: {{.*}} (05) goto pc+1")
+__msg(" 2 5: {{.*}} (b7) r1 = 2")
+__msg(" 3 2 6: {{.*}} (0f) r0 += r1")
+__msg(" 3 6 7: {{.*}} (a5) if r0 < 0x10 goto pc-5")
+__msg(" 7 8: {{.*}} (95) exit")
__naked void loop_irreducible(void)
{
asm volatile (" \
@@ -244,10 +244,9 @@ __naked void loop_irreducible(void)
}
SEC("socket")
-__failure
+__success
__log_level(2)
__msg("loop at 1")
-__msg("infinite loop detected at insn 1")
__naked void loop_self(void)
{
asm volatile (" \
diff --git a/tools/testing/selftests/bpf/progs/verifier_loops1.c b/tools/testing/selftests/bpf/progs/verifier_loops1.c
index 48a966cda199..1071d29974b7 100644
--- a/tools/testing/selftests/bpf/progs/verifier_loops1.c
+++ b/tools/testing/selftests/bpf/progs/verifier_loops1.c
@@ -123,7 +123,7 @@ l0_%=: exit; \
SEC("tracepoint")
__description("infinite loop after a conditional jump")
-__failure __msg("program is too large")
+__failure __msg("loop at insn 2 never exits")
__naked void loop_after_a_conditional_jump(void)
{
asm volatile (" \
@@ -135,10 +135,9 @@ l0_%=: exit; \
" ::: __clobber_all);
}
-SEC("tracepoint")
+SEC("xdp")
__description("bounded recursion")
-__failure
-__msg("recursive call from")
+__success __retval(4)
__naked void bounded_recursion(void)
{
asm volatile (" \
@@ -163,7 +162,7 @@ l0_%=: call bounded_recursion__1; \
SEC("tracepoint")
__description("infinite loop in two jumps")
-__failure __msg("loop detected")
+__failure __msg("loop at insn 1 never exits")
__naked void infinite_loop_in_two_jumps(void)
{
asm volatile (" \
@@ -174,9 +173,10 @@ l0_%=: if r0 < 4 goto l1_%=; \
" ::: __clobber_all);
}
-SEC("tracepoint")
+/* It never ends on its own. may_goto that the verifier adds ends the prog. */
+SEC("xdp")
__description("infinite loop: three-jump trick")
-__failure __msg("loop detected")
+__success __retval(0)
__naked void infinite_loop_three_jump_trick(void)
{
asm volatile (" \
diff --git a/tools/testing/selftests/bpf/progs/verifier_movsx.c b/tools/testing/selftests/bpf/progs/verifier_movsx.c
index 195b27a51224..280ccee33663 100644
--- a/tools/testing/selftests/bpf/progs/verifier_movsx.c
+++ b/tools/testing/selftests/bpf/progs/verifier_movsx.c
@@ -226,7 +226,7 @@ l0_%=: \
SEC("socket")
__description("MOV32SX, S8, var_off u32_max")
-__failure __msg("infinite loop detected")
+__success
__failure_unpriv __msg_unpriv("back-edge from insn 2 to 0")
__naked void mov64sx_s32_varoff_1(void)
{
diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
index b30802c9e9e3..f4ea854d3212 100644
--- a/tools/testing/selftests/bpf/progs/verifier_scev.c
+++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
@@ -238,7 +238,7 @@ __naked void meet_agrees(void)
}
SEC("xdp")
-__failure
+__success __retval(0)
__log_level(2)
__msg_next("scev at header 1:")
__msg_next(" r6=(any r6 (+ r6 1)) / ?")
@@ -1015,10 +1015,9 @@ POST_COND_TEST32(post32_lt_wrap, "<", -4, 2,
SEC("xdp")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__msg("loop header at 1, header_count is 10 ")
__msg("loop header at 1, widening r0 to 0..9 step 1")
-__msg("processed 5 insns")
+__msg("processed 4 insns")
__naked void post_cond_jlt(void)
{
asm volatile (" \
@@ -1030,8 +1029,8 @@ __naked void post_cond_jlt(void)
}
/*
- * Explore with r7=10 first, r7=4096 next.
- * Should trigger invalid stack read if r7 is marked precise.
+ * r7 is 10 on one path to the loop and 4096 on the other.
+ * Should trigger invalid stack read if the larger one counts.
*
* r0 = 0;
* r6 = 0;
@@ -1046,11 +1045,7 @@ __naked void post_cond_jlt(void)
SEC("xdp")
__failure
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("loop header at 6, header_count is 10 ")
-__msg("mark_precise: frame0: parent state regs=r6,r7 stack=:")
-__msg("loop header at 6, widening r0 to 0..9 step 1")
-__msg("loop header at 6, header_count is 4096 ")
+__msg("loop header at 6, header_count is [0..4096] ")
__msg("loop header at 6, widening r0 to 0..4095 step 1")
__msg("invalid variable-offset read from stack R2")
__naked void mark_bound_precise(void)
@@ -1076,8 +1071,8 @@ __naked void mark_bound_precise(void)
}
/*
- * Explore with r6=4086 first, r6=0 next.
- * Should trigger invalid stack read if r6 is marked precise.
+ * r6 is 4086 on one path to the loop and 0 on the other.
+ * Should trigger invalid stack read if the smaller one counts.
*
* r0 = 0;
* r6 = bpf_get_prandom_u32() == 42 ? 4086 : 0;
@@ -1091,11 +1086,7 @@ __naked void mark_bound_precise(void)
SEC("xdp")
__failure
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("loop header at 5, header_count is 10 ")
-__msg("mark_precise: frame0: parent state regs=r6 stack=:")
-__msg("loop header at 5, widening r0 to 0..9 step 1")
-__msg("loop header at 5, header_count is 4096 ")
+__msg("loop header at 5, header_count is [0..4096] ")
__msg("loop header at 5, widening r0 to 0..4095 step 1")
__msg("invalid variable-offset read from stack R2")
__naked void mark_base_precise(void)
@@ -1309,7 +1300,7 @@ __msg("scev at header 1:")
__msg_next("r6=(+ r6 1) / (linear r6 1)")
__msg_next("*fp-8=(any *fp-8 7) / (any *fp-8 7)")
__msg("loop header at 1, widening r6 to 0..2 step 1")
-__msg("2: (7a) *(u64 *)(r10 -8) = 7 ; R10=fp0 fp-8=7 loop_stack=1")
+__msg("3: (79) r0 = *(u64 *)(r10 -8) ; R0=scalar() R10=fp0 loop_stack=1")
__naked void widen_any_unallocated_slot(void)
{
asm volatile (" \
@@ -1433,6 +1424,278 @@ __naked void latch_tests_a_32bit_copy(void)
::: __clobber_all);
}
+/* The same, and the first value is not known */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop header at 3, header_count is [0..17]")
+__naked void latch_32bit_copy_from_a_range(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "w6 = w0;"
+ "if w6 > 16 goto 2f;"
+"1:"
+ "w1 = w6;"
+ "w6 += 1;"
+ "if w1 < 16 goto 1b;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/* (u32)i the way clang does it for cpu v1 and v2: two 64-bit shifts by 32 */
+SEC("socket")
+__success __retval(16)
+__log_level(2)
+__msg(" r6=(zext32 (+ r6 1)) / (zext32 (linear r6 1))")
+__msg("loop header at 1, header_count is 16")
+__naked void two_shifts_by_32_are_zext32(void)
+{
+ asm volatile (
+ "r6 = 0;"
+"1:"
+ "r6 += 1;"
+ "r6 <<= 32;"
+ "r6 >>= 32;"
+ "if r6 < 16 goto 1b;"
+ "r0 = r6;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+const volatile __u64 rodata_32 = 32;
+
+/*
+ * A 32-bit shift by a register that has 32 in it shifts by 32 & 31 = 0. @shifts make 0 of r7,
+ * so r7 is 100 in the first iteration and 1 in the others. If they are taken for (u32)r7
+ * then r7 is in [100, 107] and the read of r9 looks like dead code.
+ */
+#define NO_ZEXT32_TEST(name, before, shifts) \
+SEC("socket") \
+__failure __msg("R9 !read_ok") \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ before \
+ "r7 = 100;" \
+ "r8 = 0;" \
+"1:" \
+ "if r7 >= 100 goto 2f;" \
+ "r0 = *(u64 *)(r9 + 0);" \
+"2:" \
+ shifts \
+ "w7 += 1;" \
+ "r8 += 1;" \
+ "if r8 < 8 goto 1b;" \
+ "r0 = 0;" \
+ "exit;" \
+ : \
+ : __imm_addr(rodata_32) \
+ : __clobber_all); \
+}
+
+NO_ZEXT32_TEST(rsh32_by_reg_32, "", "r7 <<= 32; r2 = 32; w7 >>= w2;")
+NO_ZEXT32_TEST(rsh32_by_wreg_32, "", "r7 <<= 32; w2 = 32; w7 >>= w2;")
+NO_ZEXT32_TEST(rsh32_by_16_plus_16, "", "r7 <<= 32; r2 = 16; r2 += 16; w7 >>= w2;")
+NO_ZEXT32_TEST(rsh32_by_32_or_32, "",
+ "r7 <<= 32; r2 = 32; if r8 == 5 goto +1; r2 = 32; w7 >>= w2;")
+NO_ZEXT32_TEST(rsh32_by_spilled_32, "",
+ "r7 <<= 32; r2 = 32; *(u64 *)(r10 - 8) = r2; r2 = *(u64 *)(r10 - 8); w7 >>= w2;")
+NO_ZEXT32_TEST(rsh32_by_stored_32, "",
+ "r7 <<= 32; *(u64 *)(r10 - 8) = 32; r2 = *(u32 *)(r10 - 8); w7 >>= w2;")
+NO_ZEXT32_TEST(rsh32_by_32_in_rodata, "",
+ "r7 <<= 32; r2 = %[rodata_32] ll; r2 = *(u64 *)(r2 + 0); w7 >>= w2;")
+NO_ZEXT32_TEST(rsh32_by_32_set_before_the_loop, "r6 = 32;", "r7 <<= 32; w7 >>= w6;")
+NO_ZEXT32_TEST(rsh32_by_96, "", "r7 <<= 32; r2 = 96; w7 >>= w2;")
+NO_ZEXT32_TEST(rsh32_by_32_in_the_low_half, "", "r7 <<= 32; r2 = 0x100000020 ll; w7 >>= w2;")
+NO_ZEXT32_TEST(arsh32_by_reg_32, "", "r7 <<= 32; r2 = 32; w7 s>>= w2;")
+/* here it's the shift to the left that does nothing */
+NO_ZEXT32_TEST(lsh32_by_reg_32, "", "r2 = 32; w7 <<= w2; r7 >>= 32;")
+
+/* 64-bit shifts by 32 do make (u32)r7 */
+#define IS_ZEXT32_TEST(name, msg, shifts) \
+SEC("socket") \
+__success __retval(108) \
+__log_level(2) \
+__msg("loop header at 2, " msg) \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "r7 = 100;" \
+ "r8 = 0;" \
+"1:" \
+ shifts \
+ "w7 += 1;" \
+ "r8 += 1;" \
+ "if r8 < 8 goto 1b;" \
+ "r0 = r7;" \
+ "exit;" \
+ ::: __clobber_all); \
+}
+
+IS_ZEXT32_TEST(shifts64_by_reg_32, "widening r7 to 100..107", "r2 = 32; r7 <<= r2; r7 >>= r2;")
+/* 96 & 63 is 32 as well, but the result of a shift by more than 63 is unknown to the verifier */
+IS_ZEXT32_TEST(rsh64_by_96, "can't widen r7", "r7 <<= 32; r2 = 96; r7 >>= r2;")
+/* and here nothing is shifted */
+IS_ZEXT32_TEST(shifts32_by_reg_32, "can't widen r7", "r2 = 32; w7 <<= w2; w7 >>= w2;")
+
+/*
+ * The count is 32 for the cpu. The verifier doesn't know r0 after a shift by more than 63, so it
+ * goes on after the jump with r7 in [0, 1000]. A summary of r7 that starts at 0xffff and has the
+ * step -3 would have nothing in common with that.
+ */
+SEC("socket")
+__success __retval(0)
+__log_level(2)
+__msg("loop header at 2, can't widen r7")
+__naked void lsh64_by_96_makes_the_step(void)
+{
+ asm volatile (
+ "r7 = 0xffff;"
+ "r9 = 0;"
+"1:"
+ "r0 = 3;"
+ "r2 = 96;"
+ "r0 <<= r2;"
+ "r0 >>= 32;"
+ "w7 -= w0;"
+ "if w7 > 1000 goto 2f;"
+ "r9 += 1;"
+ "if r9 < 8 goto 1b;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* What the prog computes with '&' is left as it is */
+SEC("socket")
+__success __retval(70)
+__log_level(2)
+__msg("loop header at 2, can't widen r7, expr is (+ r7 (& 15 7))")
+__naked void and_of_constants_makes_the_step(void)
+{
+ asm volatile (
+ "r7 = 0;"
+ "r8 = 0;"
+"1:"
+ "r4 = 15;"
+ "r4 &= 7;"
+ "r7 += r4;"
+ "r8 += 1;"
+ "if r8 < 10 goto 1b;"
+ "r0 = r7;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/*
+ * r3 is 0, not 7: r1 stays 0 and it's r9 that ends the loop, after 1000 iterations and not after
+ * 10. r6 is 8000 then.
+ */
+SEC("socket")
+__failure __msg("R8 !read_ok")
+__naked void rsh32_by_reg_32_makes_the_step(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "r6 = 0;"
+ "r9 = 0;"
+"1:"
+ "r6 += 8;"
+ "r3 = 7;"
+ "r3 <<= 32;"
+ "r2 = 32;"
+ "w3 >>= w2;"
+ "r1 += r3;"
+ "r9 += 1;"
+ "if r9 > 999 goto 2f;"
+ "if r1 < 70 goto 1b;"
+"2:"
+ "if r6 <= 80 goto 3f;"
+ "r0 = *(u64 *)(r8 + 0);"
+"3:"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* The same when only the copy of r7 that is compared is shifted: r1 is 0 */
+SEC("socket")
+__failure __msg("R8 !read_ok")
+__naked void rsh32_by_reg_32_in_the_latch(void)
+{
+ asm volatile (
+ "r7 = 0;"
+ "r9 = 0;"
+"1:"
+ "r7 += 1;"
+ "r9 += 1;"
+ "if r9 > 999 goto 2f;"
+ "r1 = r7;"
+ "r1 <<= 32;"
+ "r2 = 32;"
+ "w1 >>= w2;"
+ "if r1 < 8 goto 1b;"
+"2:"
+ "if r7 <= 8 goto 3f;"
+ "r0 = *(u64 *)(r8 + 0);"
+"3:"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* The bound r4 is 0, not 8: the loop ends when r7 is 0 */
+SEC("socket")
+__failure __msg("R8 !read_ok")
+__naked void rsh32_by_reg_32_makes_the_bound(void)
+{
+ asm volatile (
+ "r7 = 20;"
+"1:"
+ "r7 -= 1;"
+ "r4 = 8;"
+ "r4 <<= 32;"
+ "r2 = 32;"
+ "w4 >>= w2;"
+ "if r7 > r4 goto 1b;"
+ "if r7 >= 8 goto 2f;"
+ "r0 = *(u64 *)(r8 + 0);"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/*
+ * r7 is 1 at the jump, always: the loop has no count.
+ * The prog is not run: it never ends when it's loaded without may_goto.
+ */
+SEC("socket")
+__success
+__log_level(2)
+__msg("loop header at 1, can't compute iterations count")
+__not_msg("header_count is")
+__naked void rsh32_by_reg_32_never_ends(void)
+{
+ asm volatile (
+ "r7 = 100;"
+"1:"
+ "r7 <<= 32;"
+ "r2 = 32;"
+ "w7 >>= w2;"
+ "w7 += 1;"
+ "if w7 < 108 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
const volatile __u32 rodata_bound = 10;
/* for (i = 0; i < rodata_bound; i++): the bound is loaded in every iteration. */
@@ -1574,11 +1837,10 @@ __naked void eval_expr_unallocated_stack_bound(void)
SEC("xdp")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__msg("loop header at 3, header_count is 8")
__msg("loop header at 3, widening r0 to 2..9 step 1")
__msg("3: R0=scalar(smin=umin=smin32=umin32=2,smax=umax=smax32=umax32=9,{{.*}})")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
__naked void post_cond_jlt_with_base(void)
{
asm volatile (" \
@@ -1612,11 +1874,10 @@ __naked void latch_base_differs_from_step(void)
SEC("xdp")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__msg("loop header at 1, header_count is 11")
__msg("loop header at 1, widening r0 to 0..10 step 1")
__msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=10,{{.*}})")
-__msg("processed 5 insns")
+__msg("processed 4 insns")
__naked void post_cond_jle(void)
{
asm volatile (" \
@@ -1630,11 +1891,10 @@ __naked void post_cond_jle(void)
SEC("xdp")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__msg("loop header at 1, header_count is 10")
__msg("loop header at 1, widening r0 to 0..9 step 1")
__msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=9,{{.*}})")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
__naked void post_cond_jge(void)
{
asm volatile (" \
@@ -1649,11 +1909,10 @@ __naked void post_cond_jge(void)
SEC("xdp")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__msg("loop header at 1, header_count is 11")
__msg("loop header at 1, widening r0 to 0..10 step 1")
__msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=10,{{.*}})")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
__naked void pre_cond_jge(void)
{
asm volatile (" \
@@ -1668,7 +1927,6 @@ __naked void pre_cond_jge(void)
SEC("xdp")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__msg_next("scev at header 1:")
__msg_next(" r0=(+ r0 1) / (linear r0 1)")
__msg_next(" scev at latch 2:")
@@ -1677,10 +1935,9 @@ __msg("loop header at 1, widening r0")
__msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,{{.*}})")
__msg("1: (07) r0 += 1 ; R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,{{.*}})")
__msg("2: (55) if r0 != 0x3 goto pc-2")
-__msg("3: (95) exit")
__msg("loop header at 1, clamping r0")
-__msg("from 2 to 1: safe")
-__msg("processed 5 insns")
+__msg("3: (95) exit")
+__msg("processed 4 insns")
__naked void post_cond_jne(void)
{
asm volatile (" \
@@ -1694,16 +1951,15 @@ __naked void post_cond_jne(void)
SEC("xdp")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__msg_next("scev at header 1:")
__msg_next(" r0=(+ r0 -1) / (linear r0 -1)")
__msg_next(" scev at latch 2:")
__msg_next(" r0=(+ r0 -1) / (linear (+ r0 -1) -1)")
__msg("loop header at 1, header_count is 3 ")
__msg("loop header at 1, widening r0 to 1..3 step 1")
-__msg("1: R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,var_off=(0x0; 0x3)) loop_stack=1")
+__msg("1: R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,var_off=(0x0; 0x3)) R10=fp0 loop_stack=1")
__msg("loop header at 1, clamping r0 to 1..2 (s32 1..2) step 1")
-__msg("processed 5 insns")
+__msg("processed 4 insns")
__naked void post_cond_jne_neg_step(void)
{
asm volatile (" \
@@ -1717,7 +1973,6 @@ __naked void post_cond_jne_neg_step(void)
SEC("xdp")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__msg_next("scev at header 1:")
__msg_next(" r0=(+ r0 1) / (linear r0 1)")
__msg_next(" scev at latch 1:")
@@ -1728,11 +1983,10 @@ __msg("1: (15) if r0 == 0x3 goto pc+2 ; R0=scalar(smin=smin32=0,smax=umax
__msg("2: (07) r0 += 1 ; R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,{{.*}})")
__msg("3: (05) goto pc-3")
__msg("loop header at 1, clamping r0")
-__msg("1: safe")
__msg("from 1 to 4: R0=3")
__msg("4: R0=3")
__msg("4: (95) exit")
-__msg("processed 6 insns")
+__msg("processed 5 insns")
__naked void pre_cond_je1(void)
{
asm volatile (" \
@@ -1747,7 +2001,6 @@ __naked void pre_cond_je1(void)
SEC("xdp")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__msg_next("scev at header 1:")
__msg_next(" r0=(zext32 (+ r0 1)) / (zext32 (linear r0 1))")
__msg_next(" scev at latch 2:")
@@ -1756,10 +2009,9 @@ __msg("loop header at 1, widening r0")
__msg("1: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,{{.*}})")
__msg("1: (04) w0 += 1 ; R0=scalar(smin=umin=smin32=umin32=1,smax=umax=smax32=umax32=3,{{.*}})")
__msg("2: (56) if w0 != 0x3 goto pc-2")
-__msg("3: (95) exit")
__msg("loop header at 1, clamping r0")
-__msg("from 2 to 1: safe")
-__msg("processed 5 insns")
+__msg("3: (95) exit")
+__msg("processed 4 insns")
__naked void post_cond_jne32(void)
{
asm volatile (" \
@@ -1995,7 +2247,6 @@ __naked void one_backedge_two_exits(void)
SEC("xdp")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__msg_next("scev at header 11:")
__msg_next(" r0=(+ r0 1) / (linear r0 1)")
__msg_next(" r1=(+ r1 2) / (linear r1 2)")
@@ -2005,18 +2256,15 @@ __msg_next(" r1=(+ r1 2) / (linear (+ r1 2) 2)")
__msg("loop header at 11, widening r0")
__msg("loop header at 11, widening r1")
__msg("11: R0=scalar(smin=smin32=0,smax=umax=smax32=umax32=7,var_off=(0x0; 0x7)) R1=scalar(smin=smin32=0,smax=umax=smax32=umax32=14,var_off=(0x0; 0xe),step=0+2)")
-/* loop exit */
__msg("16: (a5) if r0 < 0x8 goto pc-6")
-__msg("exiting loop 11")
-__msg("17: (95) exit")
/* second iteration */
__msg("loop header at 11, clamping r0")
__msg("loop header at 11, clamping r1")
/* iteration convergence */
-__msg("from 16 to 11: safe")
__not_msg("{{^}}11:")
-/* map lookup error path */
-__msg("from 7 to 17: safe")
+/* loop exit */
+__msg("exiting loop 11")
+__msg("17: (95) exit")
__naked void correlated_regs(void)
{
asm volatile (" \
@@ -2050,15 +2298,14 @@ __naked void correlated_regs(void)
* k += 1
* k <<= 1 // make SCEV construction not possible
* k >>= 1
- * map[k] = 1 // make k precise
+ * map[k] = 1 // k has to be known
*/
SEC("xdp")
__success
__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__msg("19: (72) *(u8 *)(r4 +0) = 1 ; R4=map_value(id={{.*}},map=map,ks=4,vs=1024,imm=16)")
__not_msg("19: ")
-__msg("processed 106 insns")
+__msg("processed 105 insns")
__naked void nested_loops_precise_var1(void)
{
asm volatile (" \
@@ -2097,8 +2344,7 @@ __log_level(2)
__msg("loop header at 2, header_count is 100 ")
__msg("loop header at 4, header_count is [0..100] ")
__msg("loop header at 6, header_count is [0..100] ")
-__msg("processed 20 insns")
-__flag(BPF_F_TEST_STATE_FREQ)
+__msg("processed 14 insns")
__naked void nested_loop_with_two_exits(void)
{
asm volatile (" \
@@ -2159,14 +2405,14 @@ __msg("loop at 5")
__msg("loop at 6, nested in 5")
__msg("loop at 11, irreducible")
__msg("loop at 12, nested in 11")
-/* entry via if r0 == 5 goto d_%= false branch */
+/* entry via if r0 == 5 goto d_%= true branch */
__msg("loop header at 12, header_count is 3 ")
__msg("loop header at 12, widening r9 to 0..2 step 1")
-__msg("12: R8=1 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) loop_stack=11,12")
-/* entry via if r0 == 5 goto d_%= true branch */
+/* entry via if r0 == 5 goto d_%= false branch */
__msg("loop header at 12, header_count is 3 ")
__msg("loop header at 12, widening r9 to 0..2 step 1")
-__msg("from 8 to 12: R8=0 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) R10=fp0 loop_stack=11,12")
+__msg("12: R8=0 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) R10=fp0 loop_stack=11,12")
+__msg("12: R8=1 R9=scalar(smin=smin32=0,smax=umax=smax32=umax32=2,var_off=(0x0; 0x3)) R10=fp0 loop_stack=11,12")
__naked void enter_nested_loop_from_side(void)
{
asm volatile (" \
@@ -3459,12 +3705,23 @@ exit_%=: \
* } while (r8 < 3);
* }
* return 0;
+ *
+ * The outer loop gets may_goto in front of the jump that leaves it, the inner loop doesn't.
+ * At run time it's may_goto that ends the prog.
*/
SEC("xdp")
-__failure
-__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
-__msg("infinite loop detected at insn 6")
+__success __retval(0)
+__xlated("call unknown")
+__xlated("...")
+__xlated("if r0 == 0x2a goto pc+4")
+__xlated("r8 = 0")
+__xlated("r8 += 1")
+__xlated("if r8 < 0x3 goto pc-2")
+__xlated("goto pc-{{[0-9]+}}")
+__xlated("r0 = 0")
+__xlated("exit")
+__xlated("r0 = 0")
+__xlated("exit")
__naked void nested_terminating_prune(void)
{
asm volatile (" \
@@ -3561,6 +3818,732 @@ __naked void dominating_exit_bounds_count(void)
" ::: __clobber_all);
}
+/*
+ * 'x &= e' has no linear formula. [0, max] that loses bits is in [0, max]: the state that enters the
+ * loop stands for all iterations.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("header_count is 16")
+__msg("r6 stays in 0..65535, it loses bits only")
+__naked void and_loses_bits(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r6 = r0; \
+ r6 &= 0xffff; \
+ r7 = 0; \
+1: r1 = 1; \
+ r1 <<= r7; \
+ r1 ^= -1; \
+ r6 &= r1; \
+ r7 += 1; \
+ if r7 < 16 goto 1b; \
+ if r6 > 0xffff goto 2f; \
+ r0 = 0; \
+ exit; \
+2: r0 = *(u64 *)(r6 + 0); \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/* r7 enters as [0, max]. After the loop it's at most 'after'. */
+#define LOSES_BITS_TEST(name, init, op, count, max, after) \
+SEC("socket") \
+__success __log_level(2) \
+__msg("header_count is " #count) \
+__msg("r7 stays in 0.." #max ", it loses bits only") \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "call %[bpf_get_prandom_u32];" \
+ "r8 = r0;" \
+ "call %[bpf_get_prandom_u32];" \
+ init \
+ "r6 = 0;" \
+"1:" \
+ op \
+ "r6 += 1;" \
+ "if r6 < " #count " goto 1b;" \
+ "r5 = " #after " ll;" \
+ "if r7 > r5 goto 2f;" \
+ "r0 = 0;" \
+ "exit;" \
+"2:" \
+ "r0 = *(u64 *)(r7 + 0);" \
+ "exit;" \
+ : \
+ : __imm(bpf_get_prandom_u32) \
+ : __clobber_all); \
+}
+
+LOSES_BITS_TEST(and_reg, "r7 = r0; r7 &= 255;", "r7 &= r8;", 8, 255, 255)
+LOSES_BITS_TEST(and_reg32, "r7 = r0; r7 &= 255;", "w7 &= w8;", 8, 255, 255)
+LOSES_BITS_TEST(and_max_100, "r7 = r0; r7 %%= 101;", "r7 &= r8;", 8, 100, 100)
+LOSES_BITS_TEST(zext8, "r7 = r0; r7 &= 0xffff;", "r7 &= 255;", 8, 65535, 255)
+LOSES_BITS_TEST(zext32, "r7 = r0; r7 <<= 32; r7 |= r8; r7 >>= 8;", "w7 = w7;", 8,
+ 72057594037927935, 0xffffffff)
+/* nothing, '&' in 32 bits and a zero extension, a zero extension: three paths */
+LOSES_BITS_TEST(masks_on_three_paths, "r7 = r0; r7 &= 0xffff;",
+ "if r8 == 0 goto +3; if r8 == 1 goto +1; w7 &= w8; r7 &= 0xfff;", 8, 65535, 65535)
+/* too long to unroll */
+LOSES_BITS_TEST(and_reg_long, "r7 = r0; r7 &= 255;", "r7 &= r8;", 1000000, 255, 255)
+
+/* The same in the stack. */
+SEC("socket")
+__success __log_level(2)
+__msg("header_count is 8")
+__msg("*fp-8 stays in 0..255, it loses bits only")
+__naked void and_reg_spilled(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r0 &= 255; \
+ *(u64 *)(r10 - 8) = r0; \
+ r6 = 0; \
+1: call %[bpf_get_prandom_u32]; \
+ r7 = *(u64 *)(r10 - 8); \
+ r7 &= r0; \
+ *(u64 *)(r10 - 8) = r7; \
+ r6 += 1; \
+ if r6 < 8 goto 1b; \
+ r7 = *(u64 *)(r10 - 8); \
+ if r7 > 255 goto 2f; \
+ r0 = 0; \
+ exit; \
+2: r0 = *(u64 *)(r7 + 0); \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+SEC("socket")
+__failure __msg("R7 invalid mem access 'scalar'")
+__naked void and_reg_is_not_below_max(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r7 = r0; \
+ r7 &= 255; \
+ r6 = 0; \
+1: call %[bpf_get_prandom_u32]; \
+ r7 &= r0; \
+ r6 += 1; \
+ if r6 < 8 goto 1b; \
+ if r7 > 254 goto 2f; \
+ r0 = 0; \
+ exit; \
+2: r0 = *(u64 *)(r7 + 0); \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/* r8 is a copy of r7 in the first iteration only. */
+SEC("socket")
+__failure __msg("R7 invalid mem access 'scalar'")
+__naked void and_reg_is_not_its_copy(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r7 = r0; \
+ r7 &= 255; \
+ r8 = r7; \
+ r6 = 0; \
+1: if r8 < 200 goto 3f; \
+ if r7 >= 200 goto 2f; \
+ r0 = *(u64 *)(r7 + 0); \
+2: call %[bpf_get_prandom_u32]; \
+ r7 &= r0; \
+ r6 += 1; \
+ if r6 < 8 goto 1b; \
+3: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/*
+ * Shifts to the right, division and modulo don't make a number greater, but there is no formula
+ * for them: iterations one by one tell more.
+ */
+SEC("socket")
+__success __log_level(2)
+__msg("header_count is 8")
+__msg("can't widen r6, expr is (any")
+__not_msg("it loses bits only")
+__naked void shift_div_mod_are_unrolled(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r6 = r0; \
+ r6 &= 0xfff; \
+ call %[bpf_get_prandom_u32]; \
+ r8 = r0; \
+ r7 = 0; \
+1: if r8 == 1 goto 2f; \
+ if r8 == 2 goto 3f; \
+ if r8 == 3 goto 4f; \
+ r6 >>= 1; \
+ goto 5f; \
+2: w6 /= w8; \
+ goto 5f; \
+3: r6 %%= r8; \
+ goto 5f; \
+4: w6 >>= 2; \
+5: r7 += 1; \
+ if r7 < 8 goto 1b; \
+ if r6 > 0xfff goto 6f; \
+ r0 = 0; \
+ exit; \
+6: r0 = *(u64 *)(r6 + 0); \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/* The count and the divisor are registers, with bits that a 32-bit insn does not look at. */
+SEC("socket")
+__success __retval(0) __log_level(2)
+__msg("header_count is 8")
+__not_msg("it loses bits only")
+__naked void shift_div_mod_by_reg(void)
+{
+ asm volatile (" \
+ r6 = 0x300000064 ll; \
+ r7 = r6; \
+ r8 = r6; \
+ r9 = r6; \
+ r1 = 0; \
+1: r2 = 33; \
+ w6 >>= w2; \
+ r2 = 65; \
+ r7 >>= r2; \
+ r2 = 0x100000000 ll; \
+ w8 /= w2; \
+ w9 %%= w2; \
+ r1 += 1; \
+ if r1 < 8 goto 1b; \
+ r0 = 1; \
+ if r6 != 0 goto 2f; \
+ if r7 != 0x3000000 goto 2f; \
+ if r8 != 0 goto 2f; \
+ if r9 != 0x64 goto 2f; \
+ r0 = 0; \
+2: exit; \
+" ::: __clobber_all);
+}
+
+/* A shift to the left, an arithmetic shift and a signed division can make the number greater. */
+SEC("socket")
+__success __log_level(2)
+__msg("can't widen r6, expr is (<< r6 1)")
+__not_msg("it loses bits only")
+__naked void shift_left_grows(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r6 = r0; \
+ r6 &= 0xf; \
+ r7 = 0; \
+1: r6 <<= 1; \
+ r7 += 1; \
+ if r7 < 8 goto 1b; \
+ r0 = r6; \
+ r0 &= 1; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+SEC("socket")
+__success __log_level(2)
+__msg("can't widen r6, expr is (s>> r6 1)")
+__msg("can't widen r8, expr is (s/ r8 -1)")
+__not_msg("it loses bits only")
+__naked void signed_ops_grow(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r6 = r0; \
+ r6 &= 0xf; \
+ r8 = r6; \
+ r7 = 0; \
+1: r6 s>>= 1; \
+ r8 s/= -1; \
+ r7 += 1; \
+ if r7 < 8 goto 1b; \
+ r0 = r6; \
+ r0 += r8; \
+ r0 &= 1; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/*
+ * The size is never 0: bpf_skb_load_bytes() doesn't take 0. [0, max at the entry] has a 0 in it
+ * that is made up, so the loop is unrolled.
+ */
+#define SIZE_TEST(name, init, op, count, expr) \
+SEC("tc") \
+__success __log_level(2) \
+__msg("header_count is " #count) \
+__msg("can't widen r7, expr is " expr) \
+__not_msg("it loses bits only") \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "r9 = r1;" \
+ "call %[bpf_get_prandom_u32];" \
+ init \
+ "r6 = 0;" \
+"1:" \
+ "r1 = r9;" \
+ "r2 = 0;" \
+ "r3 = r10;" \
+ "r3 += -128;" \
+ "r4 = r7;" \
+ "call %[bpf_skb_load_bytes];" \
+ op \
+ "r6 += 1;" \
+ "if r6 < " #count " goto 1b;" \
+ "r0 = 0;" \
+ "exit;" \
+ : \
+ : __imm(bpf_get_prandom_u32), __imm(bpf_skb_load_bytes) \
+ : __clobber_all); \
+}
+
+/* 8, 4, 2, 1 */
+SIZE_TEST(halve_const, "r7 = 8;", "r7 >>= 1;", 4, "(>> r7 1)")
+SIZE_TEST(halve_const32, "r7 = 64;", "w7 >>= 1;", 6, "(zext32 (>> r7 1))")
+SIZE_TEST(divide_const, "r7 = 100;", "r7 /= 10;", 3, "(/ r7 10)")
+/* [64, 127], [32, 63], [16, 31] */
+SIZE_TEST(halve_range, "r7 = r0; r7 &= 63; r7 |= 64;", "r7 >>= 1;", 3, "(>> r7 1)")
+/* 8 or 9 */
+SIZE_TEST(halve_two_values, "r7 = r0; r7 &= 1; r7 |= 8;", "r7 >>= 1;", 4, "(>> r7 1)")
+/* 8 or 16 */
+SIZE_TEST(halve_one_of_two, "r7 = 8; if r0 == 0 goto +1; r7 = 16;", "r7 >>= 1;", 4, "(>> r7 1)")
+/* it loses bits only, but it doesn't enter as [0, max]. The loop doesn't change it. */
+SIZE_TEST(mask_keeps_range, "r7 = r0; r7 &= 63; r7 |= 64;", "r7 &= 127;", 3, "(& r7 127)")
+SIZE_TEST(mov32_keeps_range, "r7 = r0; r7 &= 63; r7 |= 64;", "w7 = w7;", 3, "(zext32 r7)")
+
+/* The same, the size is in the stack. */
+SEC("tc")
+__success __log_level(2)
+__msg("header_count is 4")
+__msg("can't widen *fp-136, expr is (>> *fp-136 1)")
+__not_msg("it loses bits only")
+__naked void halve_const_spilled(void)
+{
+ asm volatile (" \
+ r9 = r1; \
+ r7 = 8; \
+ *(u64 *)(r10 - 136) = r7; \
+ r6 = 0; \
+1: r1 = r9; \
+ r2 = 0; \
+ r3 = r10; \
+ r3 += -128; \
+ r4 = *(u64 *)(r10 - 136); \
+ call %[bpf_skb_load_bytes]; \
+ r7 = *(u64 *)(r10 - 136); \
+ r7 >>= 1; \
+ *(u64 *)(r10 - 136) = r7; \
+ r6 += 1; \
+ if r6 < 4 goto 1b; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_skb_load_bytes)
+ : __clobber_all);
+}
+
+/*
+ * A number that is divided gets smaller and smaller. Only iterations one by one tell how small it
+ * is after the loop, also when it enters as [0, max].
+ */
+#define DIVIDED_TEST(name, init, op, count, max) \
+SEC("socket") \
+__success __log_level(2) \
+__msg("header_count is " #count) \
+__msg("can't widen r7") \
+__not_msg("it loses bits only") \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "call %[bpf_get_prandom_u32];" \
+ "r8 = r0;" \
+ init \
+ "r6 = 0;" \
+"1:" \
+ op \
+ "r6 += 1;" \
+ "if r6 < " #count " goto 1b;" \
+ "if r7 > " #max " goto 2f;" \
+ "r0 = 0;" \
+ "exit;" \
+"2:" \
+ "r0 = *(u64 *)(r7 + 0);" \
+ "exit;" \
+ : \
+ : __imm(bpf_get_prandom_u32) \
+ : __clobber_all); \
+}
+
+DIVIDED_TEST(shift_out_const, "r7 = 255;", "r7 >>= 1;", 8, 0)
+DIVIDED_TEST(shift_out_range, "r7 = r0; r7 &= 255;", "r7 >>= 1;", 8, 0)
+DIVIDED_TEST(shift_out_range32, "r7 = r0; r7 &= 255;", "w7 >>= 1;", 8, 0)
+DIVIDED_TEST(div10_range, "r7 = r0; r7 &= 0xffff;", "r7 /= 10;", 3, 65)
+DIVIDED_TEST(divided_on_all_paths, "r7 = r0; r7 &= 255;",
+ "if r8 == 0 goto +2; r7 >>= 1; goto +1; w7 /= 3;", 8, 0)
+/* the other operand is a register */
+DIVIDED_TEST(shift_by_const_reg, "r7 = r0; r7 &= 255; r5 = 1;", "r7 >>= r5;", 8, 0)
+DIVIDED_TEST(div_by_const_reg, "r7 = r0; r7 &= 255; r5 = 2;", "r7 /= r5;", 8, 0)
+/* 0 + 1 + 2 + 3 + 4 bits */
+DIVIDED_TEST(shift_by_counter, "r7 = r0; r7 &= 255;", "r7 >>= r6;", 5, 0)
+/* the last one is % 1 */
+DIVIDED_TEST(mod_by_counter, "r7 = r0; r7 &= 255;", "r5 = 8; r5 -= r6; r7 %%= r5;", 8, 0)
+/* together with what loses bits */
+DIVIDED_TEST(shift_then_and, "r7 = r0; r7 &= 255;", "r7 >>= 1; r7 &= r8;", 8, 0)
+DIVIDED_TEST(and_then_shift, "r7 = r0; r7 &= 255;", "r7 &= r8; r7 >>= 1;", 8, 0)
+DIVIDED_TEST(shift_then_zext, "r7 = r0; r7 &= 255;", "r7 >>= 1; w7 = w7;", 8, 0)
+/* divided on one of two paths */
+DIVIDED_TEST(divided_on_one_path, "r7 = r0; r7 &= 255;", "if r8 == 0 goto +1; r7 >>= 1;", 8, 255)
+
+SEC("socket")
+__success __log_level(2)
+__msg("header_count is 3")
+__msg("can't widen *fp-8, expr is (/ *fp-8 10)")
+__not_msg("it loses bits only")
+__naked void div10_range_spilled(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r0 &= 0xffff; \
+ *(u64 *)(r10 - 8) = r0; \
+ r6 = 0; \
+1: r7 = *(u64 *)(r10 - 8); \
+ r7 /= 10; \
+ *(u64 *)(r10 - 8) = r7; \
+ r6 += 1; \
+ if r6 < 3 goto 1b; \
+ r7 = *(u64 *)(r10 - 8); \
+ if r7 > 65 goto 2f; \
+ r0 = 0; \
+ exit; \
+2: r0 = *(u64 *)(r7 + 0); \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/* In the second iteration r7 is at most 9 at the head. */
+SEC("socket")
+__success __log_level(2)
+__msg("header_count is [0..3]")
+__msg("can't widen r7, expr is (% r7 10)")
+__naked void mod_at_the_head(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r7 = r0; \
+ r7 &= 0xffff; \
+ r6 = 0; \
+1: if r6 == 0 goto 2f; \
+ if r7 > 9 goto 3f; \
+2: r7 %%= 10; \
+ r6 += 1; \
+ if r6 < 3 goto 1b; \
+ r0 = 0; \
+ exit; \
+3: r0 = *(u64 *)(r7 + 0); \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/* r7 is 0 or 64, 'r7 &= r0' keeps it so. 1..63 are made up in [0, 64]. */
+SEC("socket")
+__success __log_level(2)
+__msg("header_count is 4")
+__msg("can't widen r7, expr is (& r7 ?)")
+__not_msg("it loses bits only")
+__naked void and_keeps_known_bits(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r7 = r0; \
+ r7 &= 64; \
+ r6 = 0; \
+1: call %[bpf_get_prandom_u32]; \
+ r7 &= r0; \
+ r6 += 1; \
+ if r6 < 4 goto 1b; \
+ r7 &= 63; \
+ if r7 != 0 goto 2f; \
+ r0 = 0; \
+ exit; \
+2: r0 = *(u64 *)(r7 + 0); \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/*
+ * r7 has no formula: the verifier has no bounds for 'x op= unknown'. The loop is unrolled, and
+ * the pointer in r6, which moves in each iteration, keeps its offset.
+ */
+#define PTR_TEST(name, op, add, count) \
+SEC("socket") \
+__success __log_level(2) \
+__msg("header_count is " #count) \
+__not_msg(": widened") \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "r1 = 0;" \
+ "*(u64*)(r10 - 8) = r1;" \
+ "r2 = r10;" \
+ "r2 += -8;" \
+ "r1 = %[map] ll;" \
+ "call %[bpf_map_lookup_elem];" \
+ "if r0 == 0 goto 2f;" \
+ "r6 = r0;" \
+ "r9 = r0;" \
+ "r7 = *(u64 *)(r9 + 512);" \
+ "r7 &= 0xff;" \
+ "r1 = 0;" \
+"1:" \
+ "*(u8 *)(r6 + 0) = r7;" \
+ "r6 += " #add ";" \
+ "r3 = *(u64 *)(r9 + 520);" \
+ op \
+ "r1 += 1;" \
+ "if r1 < " #count " goto 1b;" \
+"2:" \
+ "r0 = 0;" \
+ "exit;" \
+ : \
+ : __imm(bpf_map_lookup_elem), __imm_addr(map) \
+ : __clobber_all); \
+}
+
+PTR_TEST(ptr_rsh_2, "r7 >>= r3;", 8, 2)
+PTR_TEST(ptr_rsh32_2, "w7 >>= w3;", 8, 2)
+PTR_TEST(ptr_div_2, "r7 /= r3;", 8, 2)
+PTR_TEST(ptr_mod_2, "r7 %%= r3;", 8, 2)
+PTR_TEST(ptr_div_2_step1, "r7 /= r3;", 1, 2)
+PTR_TEST(ptr_div_2_step2, "r7 /= r3;", 2, 2)
+PTR_TEST(ptr_div_2_step3, "r7 /= r3;", 3, 2)
+PTR_TEST(ptr_div_3_step3, "r7 /= r3;", 3, 3)
+PTR_TEST(ptr_rsh_10, "r7 >>= r3;", 8, 10)
+/* what loses bits returns within [0, 255] */
+PTR_TEST(ptr_and_2, "r7 &= r3;", 8, 2)
+PTR_TEST(ptr_and32_2, "w7 &= w3;", 8, 2)
+PTR_TEST(ptr_zext8_2, "r7 &= 255;", 8, 2)
+
+/* The pointer is in the stack. */
+SEC("socket")
+__success __log_level(2)
+__msg("header_count is 2")
+__not_msg(": widened")
+__naked void ptr_rsh_2_spilled(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto 2f; \
+ *(u64 *)(r10 - 16) = r0; \
+ r9 = r0; \
+ r7 = *(u64 *)(r9 + 512); \
+ r7 &= 0xff; \
+ r1 = 0; \
+1: r6 = *(u64 *)(r10 - 16); \
+ *(u8 *)(r6 + 0) = r7; \
+ r6 += 8; \
+ *(u64 *)(r10 - 16) = r6; \
+ r3 = *(u64 *)(r9 + 520); \
+ r7 >>= r3; \
+ r1 += 1; \
+ if r1 < 2 goto 1b; \
+2: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem), __imm_addr(map)
+ : __clobber_all);
+}
+
+/*
+ * bpf_compute_const_regs() knows what r7 is in the whole loop, and r8 that is made of it. If r7 gets
+ * a range, or is forgotten because the loop is too long to unroll, then r8 can be 0, the jump finds
+ * that, and do_check() sees a constant that is not the one it has to be. r0 is 0 for the main
+ * pass only.
+ */
+#define CONST_REGS_TEST(name, init, op, probe, count, val) \
+SEC("socket") \
+__success __retval(val) __log_level(2) \
+__msg("header_count is " #count) \
+__not_msg("wid{{(ening|en)}} r7") \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ init \
+ "r6 = 0;" \
+"1:" \
+ op \
+ probe \
+ "call %[bpf_get_prandom_u32];" \
+ "if r0 != 0 goto 2f;" \
+ "if r8 != r0 goto 2f;" \
+ "*(u64 *)(r10 - 8) = r8;" \
+"2:" \
+ "r6 += 1;" \
+ "if r6 < " #count " goto 1b;" \
+ "r0 = r8;" \
+ "exit;" \
+ : \
+ : __imm(bpf_get_prandom_u32) \
+ : __clobber_all); \
+}
+
+CONST_REGS_TEST(const_regs_and, "r7 = 17;", "r7 &= 31;", "r8 = r7;", 10, 17)
+CONST_REGS_TEST(const_regs_mov32, "r7 = 17;", "w7 = w7;", "r8 = r7;", 10, 17)
+CONST_REGS_TEST(const_regs_add_sub32, "r7 = 17;", "w7 += 1; w7 += -1;", "r8 = r7;", 10, 17)
+/* too long to unroll */
+CONST_REGS_TEST(const_regs_and_long, "r7 = 17;", "r7 &= 31;", "r8 = r7;", 1000000, 17)
+CONST_REGS_TEST(const_regs_mov32_long, "r7 = 17;", "w7 = w7;", "r8 = r7;", 1000000, 17)
+CONST_REGS_TEST(const_regs_add_add32_long, "r7 = 17;", "w7 += 0x80000000; w7 += 0x80000000;",
+ "r8 = r7;", 1000000, 17)
+/* r7 doesn't fit in 32 bits, r8 does */
+CONST_REGS_TEST(const_regs_neg_long, "r7 = -16;", "r7 &= -8;", "r8 = r7; r8 &= 255;", 1000000, 240)
+CONST_REGS_TEST(const_regs_wide_long, "r7 = 0xffffffff00000011 ll;", "r7 &= -239;",
+ "r8 = r7; r8 &= 255;", 1000000, 17)
+CONST_REGS_TEST(const_regs_wide_long32, "r7 = 0xffffffff00000011 ll;", "r7 &= -239;",
+ "w8 = w7;", 1000000, 17)
+CONST_REGS_TEST(const_regs_wide_reg_long, "r7 = 0x100000011 ll; r9 = 0x1000000ff ll;",
+ "r7 &= r9;", "r8 = r7; r8 &= 255;", 1000000, 17)
+
+/* A loop without a count around it. */
+SEC("socket")
+__success
+__naked void const_regs_and_nested(void)
+{
+ asm volatile (" \
+ r7 = 17; \
+1: r6 = 0; \
+2: r7 &= 31; \
+ call %[bpf_get_prandom_u32]; \
+ if r0 != 0 goto 3f; \
+ if r7 != r0 goto 3f; \
+ *(u64 *)(r10 - 8) = r7; \
+3: r6 += 1; \
+ if r6 < 10 goto 2b; \
+ call %[bpf_get_prandom_u32]; \
+ if r0 != 0 goto 1b; \
+ r0 = r7; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/* 200 iterations are not too long, 200 x 200 are. */
+SEC("socket")
+__success __retval(240)
+__naked void const_regs_neg_nested(void)
+{
+ asm volatile (" \
+ r7 = -16; \
+ r9 = 0; \
+1: r6 = 0; \
+2: r7 &= -8; \
+ r8 = r7; \
+ r8 &= 255; \
+ call %[bpf_get_prandom_u32]; \
+ if r0 != 0 goto 3f; \
+ if r8 != r0 goto 3f; \
+ *(u64 *)(r10 - 8) = r8; \
+3: r6 += 1; \
+ if r6 < 200 goto 2b; \
+ r9 += 1; \
+ if r9 < 200 goto 1b; \
+ r0 = r8; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+#ifdef __BPF_FEATURE_ADDR_SPACE_CAST
+struct {
+ __uint(type, BPF_MAP_TYPE_ARENA);
+ __uint(map_flags, BPF_F_MMAPABLE);
+ __uint(max_entries, 1);
+} arena SEC(".maps");
+
+/* The offset of a pointer to arena is not tracked. A loop that moves the pointer is not unrolled. */
+SEC("syscall")
+__success __log_level(2)
+__msg("header_count is 100")
+__not_msg("can't widen r6")
+__naked void arena_pointer_moves(void)
+{
+ asm volatile (" \
+ r1 = %[arena] ll; \
+ r6 = 0; \
+ r6 = addr_space_cast(r6, 0, 1); \
+ r7 = 0; \
+1: *(u64 *)(r6 + 0) = r7; \
+ r6 += 8; \
+ r7 += 1; \
+ if r7 < 100 goto 1b; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm_addr(arena)
+ : __clobber_all);
+}
+#endif
+
+/* The pointer is not widened, see ptr_buf. The loop is counted: it is unrolled. */
+SEC("iter/bpf_map_elem")
+__success
+__log_level(2)
+__msg("header_count is 4 ")
+__msg("can't widen r6, expr is (+ r6 1)")
+__naked void ptr_buf_is_the_counter(void)
+{
+ asm volatile (" \
+ r6 = *(u64 *)(r1 + %[value]); \
+ if r6 == 0 goto 2f; \
+ r7 = r6; \
+ r7 += 4; \
+1: r6 += 1; \
+ if r6 < r7 goto 1b; \
+2: r0 = 0; \
+ exit; \
+" :
+ : __imm_const(value, offsetof(struct bpf_iter__bpf_map_elem, value))
+ : __clobber_all);
+}
+
/*
* r6 and r7 count together. At the head of the loop both are 0..9 and nothing says
* that they are equal: r7 = 9 gets through the first jump with r6 = 0..9 and
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 73/81] selftests/bpf: Adjust tests of loops
2026-10-26 2:59 ` [PATCH bpf 73/81] selftests/bpf: Adjust tests of loops Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 7742 bytes --]
> selftests/bpf: Adjust tests of loops
TL;DR: Test issues only: the hard-coded insn number in loop_detection may fail
on cpuv4/gcc builds, and the movsx and scev test changes may no longer catch
the regressions they were meant to.
> diff --git a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
> index a76bce7278e7..a48b48fb2e51 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c
[ ... ]
> @@ -94,7 +94,7 @@ static int loop_detection_cb(__u32 idx, struct num_context *ctx)
> }
>
> SEC("?raw_tp")
> -__failure __msg("infinite loop detected")
> +__failure __msg("loop at insn 12 never exits")
> int loop_detection(void *unused)
> {
> struct num_context loop_ctx = { .i = 0 };
Will this expectation hold for the test_progs-cpuv4 flavor? loop_detection()
is compiled from C, and the message hard-codes the index of the loop in the
callback, which depends on how many insns the compiler emits for the main
prog:
struct num_context loop_ctx = { .i = 0 };
With the default -mcpu=v3, the zero initialization of this 16-byte struct is
'r1 = 0; *(u64 *)(r10 - 8) = r1; *(u64 *)(r10 - 16) = r1', which gives the
12 insns the test expects.
With -mcpu=v4 (which test_progs-cpuv4 builds with), and with bpf-gcc, the
compiler emits BPF_ST instead, so the two zero stores become
'*(u64 *)(r10 - 8) = 0; *(u64 *)(r10 - 16) = 0'. The main prog is then one
insn shorter, the callback loop starts at insn 11, and the __msg() does not
match.
The tree has run into this before. Commit 898f55f50a00 ("selftests/bpf:
relax expected log messages to allow emitting BPF_ST") says:
Expected messages in these tests hard-code instruction numbers for BPF
programs compiled from C. These instruction numbers change when BPF_ST is
allowed because single BPF_ST instruction replaces a pair of
BPF_MOV/BPF_STX instructions
Commit 9786d424a36b ("selftests/bpf: Fix chained_global_func_calls_success()
for cpu v4") fixed a test_progs-cpuv4 failure caused by hard-coded insn
counts in a C prog.
The old expectation, "infinite loop detected", had no insn number. Would
matching with __msg("loop at insn {{[0-9]+}} never exits") keep the test
independent of codegen? The other new 'loop at insn N' expectations in this
series are in naked asm progs, where the index is fixed.
[ ... ]
> diff --git a/tools/testing/selftests/bpf/progs/verifier_movsx.c b/tools/testing/selftests/bpf/progs/verifier_movsx.c
> index 195b27a51224..280ccee33663 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_movsx.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_movsx.c
> @@ -226,7 +226,7 @@ l0_%=: \
>
> SEC("socket")
> __description("MOV32SX, S8, var_off u32_max")
> -__failure __msg("infinite loop detected")
> +__success
> __failure_unpriv __msg_unpriv("back-edge from insn 2 to 0")
> __naked void mov64sx_s32_varoff_1(void)
> {
Does this change leave the privileged run of this test unable to catch the
bug it was added for?
The test came in with a62293c33b05 ("selftests/bpf: Add a few tests to
cover") as the reproducer for 380d5f89a481 ("bpf: Add missed var_off setting
in set_sext32_default_val()"). Before that fix, var_off was left stale after
'w7 = (s8)w3', so the verifier took w7 to be in [0, 127] and decided that
'if w7 >= 0x2533823b goto l0_%=' was never taken. It walked only the
fall-through and the program loaded. Once var_off was fixed, both branches
were possible and the back-edge led to "infinite loop detected". So
'__failure __msg("infinite loop detected")' was what told the fixed verifier
from the broken one.
With this series, a loop that has an exit is accepted and gets may_goto, so
the fixed verifier loads the program. A verifier that has lost the
'reg->var_off = tnum_subreg(tnum_unknown)' line in set_sext32_default_val()
also loads it, because it never reaches the back-edge. Both outcomes match
__success.
The unprivileged expectation does not help here, since the back-edge is
rejected before any range is tracked. No other test in verifier_movsx.c
covers this case: mov64sx_s32_varoff_2 and _3 mask the input to 0xf, and
mov32sx_s8_range gets a fully unknown input, so a stale var_off is harmless
there. No later commit in the series changes this test.
Could the test check something that only a correct verifier produces? For
example, __log_level(2) together with a __msg() for the R7 state after
'w7 = (s8)w3' (var_off=(0x0; 0xffffffff), smin32=-128), or for the loop at
insn 0 being verified or getting its may_goto.
[ ... ]
> diff --git a/tools/testing/selftests/bpf/progs/verifier_scev.c b/tools/testing/selftests/bpf/progs/verifier_scev.c
> index b30802c9e9e3..f4ea854d3212 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_scev.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_scev.c
> @@ -2005,18 +2256,15 @@ __msg_next(" r1=(+ r1 2) / (linear (+ r1 2) 2)")
[ ... ]
> __msg("16: (a5) if r0 < 0x8 goto pc-6")
> -__msg("exiting loop 11")
> -__msg("17: (95) exit")
> /* second iteration */
> __msg("loop header at 11, clamping r0")
> __msg("loop header at 11, clamping r1")
> /* iteration convergence */
> -__msg("from 16 to 11: safe")
> __not_msg("{{^}}11:")
> -/* map lookup error path */
> -__msg("from 7 to 17: safe")
> +/* loop exit */
> +__msg("exiting loop 11")
> +__msg("17: (95) exit")
Can the "iteration convergence" check in correlated_regs() still fail?
Before this change the test asserted convergence with
__msg("from 16 to 11: safe"), and the __not_msg("{{^}}11:") span ran from
there to "from 7 to 17: safe". The new version drops the positive
convergence line, so the __not_msg() now sits between
"loop header at 11, clamping r1" and "exiting loop 11".
match_negative_msgs() in test_loader.c only searches for a negative pattern
between the end of the previous positive match and the start of the next
one. Both neighbouring messages are logged while insn 16
('if r0 < 0x8 goto pc-6') is processed.
check_cond_jmp_op() pushes the taken branch, so flow_drain() calls
bpf_flow_deliver(11) first, and:
bpf_flow_loop_enter()
maybe_clamp_scev_regs()
bpf_clamp_scev_regs()
prints the 'clamping' lines. flow_merge() then prints either
'from 16 to 11: within' or, if the state does not converge,
'from 16 to 11: widened', and makes the state at 11 pending.
Still inside the same do_check_block(), the fall-through goes to:
bpf_flow_deliver(17)
flow_arrive()
bpf_flow_loop_leave()
loop_stack_pop()
which prints 'exiting loop 11'. A pending state at 11 is only verified
again after do_check_block() returns and bpf_flow_next() picks it, so any
'11: (bf) r2 = r6' lines from a second pass come after 'exiting loop 11' and
fall outside the span. The only line that can appear inside the span is
'from 16 to 11: within|widened', and it does not start with '11:'.
Nothing else in the test checks convergence (there is no 'processed N insns'
line), so the test passes even when the first back edge does not converge.
Would __msg("from 16 to 11: within") bring the convergence check back?
Other tests in this series use that kind of message, for example
__msg("from 18 to 9: within") in verifier_subprog_precision.c. Moving the
__not_msg() between "exiting loop 11" and "17: (95) exit" would also work,
since flow_compute_order() finishes the loop before insn 17.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 74/81] selftests/bpf: Adjust tests of recursion
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (72 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 73/81] selftests/bpf: Adjust tests of loops Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 75/81] selftests/bpf: Adjust tests to the order of verification and to the log Alexei Starovoitov
` (8 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier rejected recursion, because it cannot calculate the depth of
the stack at load time. Now a prog with recursion loads, and the kernel
limits the recursion at run time. Each recursive subprog starts with a
check of the stack and of the number of calls. When the check fails, the
subprog returns 0.
test the recursion result
-------------------------------------- --------------------- ---------
callx_unbounded_recursion never stops returns 0
callx_bounded_recursion counts down to 0 returns 0
callx_mutual_recursion two subprogs loads
callx_rodata_recursion pointer from .rodata loads
recursion_via_global_func_and_callback global subprog and loads
bpf_loop() callback
calls: recursive call. test1 a call insn is its rejected
own target
BPF_PSEUDO_FUNC reference to the main the main prog is a rejected
program timer callback
The verifier rejected the first five progs. It continues to reject the last
two, with new messages: "calls itself" and
"recursive call of the main program".
Other changes:
- MAX_CALL_FRAMES is 32. The two chains of calls that must be too deep get
16 more subprogs.
- "calls: conditional call 6" is a loop that can spin forever. The
verifier rejected it with "infinite loop detected". Now the verifier
adds may_goto to such a loop, and the prog loads.
- callx_retval_is_tracked and callx_precision expected "to caller at" and
mark_precise lines. The log does not have these lines. Expect the values
of the registers.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_callx.c | 22 ++--
.../bpf/progs/verifier_callx_rodata.c | 2 +-
.../bpf/progs/verifier_global_subprogs.c | 7 +-
tools/testing/selftests/bpf/verifier/calls.c | 101 +++++++++++++++++-
.../selftests/bpf/verifier/pseudo_func.c | 2 +-
5 files changed, 110 insertions(+), 24 deletions(-)
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx.c b/tools/testing/selftests/bpf/progs/verifier_callx.c
index 7e59188d9fca..5594bce7601c 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx.c
@@ -245,8 +245,7 @@ __naked void callx_scratches_args(void)
/* return value of the callee is tracked */
SEC("socket")
__success __log_level(2)
-__msg("to caller at 4:")
-__msg("R0=6")
+__msg("4: R0=6 R10=fp0")
__naked void callx_retval_is_tracked(void)
{
asm volatile (
@@ -523,9 +522,9 @@ static unsigned long self(void)
);
}
-/* unbounded recursion is caught by the main verification pass */
+/* unbounded recursion is stopped at run time */
SEC("socket")
-__failure __msg("frames is too deep")
+__success __retval(0)
__naked void callx_unbounded_recursion(void)
{
asm volatile (
@@ -551,12 +550,8 @@ static unsigned long countdown(void)
);
}
-/*
- * The depth of the recursion is known to the main verification pass,
- * but recursive calls are not allowed regardless.
- */
SEC("socket")
-__failure __msg("recursive call from countdown() to countdown()")
+__success __retval(0)
__naked void callx_bounded_recursion(void)
{
asm volatile (
@@ -603,7 +598,7 @@ static unsigned long pong(void)
}
SEC("socket")
-__failure __msg("recursive call from")
+__success
__naked void callx_mutual_recursion(void)
{
asm volatile (
@@ -926,11 +921,8 @@ long long vals[] SEC(".data.vals") = {1, 2, 3, 4};
SEC("socket")
__success __log_level(2)
-__msg("mark_precise: frame0: regs=r0 stack= before 12: (95) exit")
-__msg("mark_precise: frame1: regs=r0 stack= before 11: (bf) r0 = r1")
-__msg("mark_precise: frame1: regs=r1 stack= before 4: (8d) callx r2")
-__msg("mark_precise: frame0: regs=r1 stack= before 3: (bf) r1 = r6")
-__msg("mark_precise: frame0: regs=r6 stack= before 2: (b7) r6 = 3")
+__msg("5: R0=3 R10=fp0")
+__msg("8: (0f) r1 += r0 ; R0=24 R1=map_value(map=.data.vals,ks=4,vs=32,imm=24)")
__retval(4)
__naked void callx_precision(void)
{
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
index 6cf58ca9ac5b..85141cd73298 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
@@ -571,7 +571,7 @@ static unsigned long selfcall(void)
}
SEC("socket")
-__failure __msg("recursive call from selfcall() to selfcall()")
+__success
__naked void callx_rodata_recursion(void)
{
asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c b/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c
index 574b26b5a7df..9f5898feb437 100644
--- a/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c
+++ b/tools/testing/selftests/bpf/progs/verifier_global_subprogs.c
@@ -556,12 +556,11 @@ __noinline int global_calls_loop(int x)
/*
* loop_cb_calls_static() -> static_calls_global() -> global_calls_loop() ->
- * bpf_loop() -> loop_cb_calls_static() is an unbounded recursion that the
- * main verification pass can't see, because it doesn't follow calls of global
- * functions. None of the functions use stack.
+ * bpf_loop() -> loop_cb_calls_static() is an unbounded recursion through a global
+ * function and a helper. It's stopped at run time. None of the functions use stack.
*/
SEC("?raw_tp")
-__failure __msg("recursive call from global_calls_loop() to loop_cb_calls_static()")
+__success
int recursion_via_global_func_and_callback(const void *ctx)
{
return loop_cb_calls_static(0, NULL);
diff --git a/tools/testing/selftests/bpf/verifier/calls.c b/tools/testing/selftests/bpf/verifier/calls.c
index 0af237c02ddf..330421e9c164 100644
--- a/tools/testing/selftests/bpf/verifier/calls.c
+++ b/tools/testing/selftests/bpf/verifier/calls.c
@@ -559,8 +559,7 @@
BPF_EXIT_INSN(),
},
.prog_type = BPF_PROG_TYPE_SCHED_CLS,
- .errstr = "infinite loop detected",
- .result = REJECT,
+ .result = ACCEPT,
},
{
"calls: using r0 returned by callee",
@@ -812,7 +811,7 @@
BPF_EXIT_INSN(),
},
.prog_type = BPF_PROG_TYPE_TRACEPOINT,
- .errstr = "recursive call",
+ .errstr = "calls itself",
.result = REJECT,
},
{
@@ -1325,6 +1324,54 @@
BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P */
BPF_EXIT_INSN(),
/* P */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P1 */
+ BPF_EXIT_INSN(),
+ /* P1 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P2 */
+ BPF_EXIT_INSN(),
+ /* P2 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P3 */
+ BPF_EXIT_INSN(),
+ /* P3 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P4 */
+ BPF_EXIT_INSN(),
+ /* P4 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P5 */
+ BPF_EXIT_INSN(),
+ /* P5 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P6 */
+ BPF_EXIT_INSN(),
+ /* P6 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P7 */
+ BPF_EXIT_INSN(),
+ /* P7 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P8 */
+ BPF_EXIT_INSN(),
+ /* P8 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P9 */
+ BPF_EXIT_INSN(),
+ /* P9 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P10 */
+ BPF_EXIT_INSN(),
+ /* P10 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P11 */
+ BPF_EXIT_INSN(),
+ /* P11 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P12 */
+ BPF_EXIT_INSN(),
+ /* P12 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P13 */
+ BPF_EXIT_INSN(),
+ /* P13 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P14 */
+ BPF_EXIT_INSN(),
+ /* P14 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P15 */
+ BPF_EXIT_INSN(),
+ /* P15 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P16 */
+ BPF_EXIT_INSN(),
+ /* P16 */
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
@@ -1387,6 +1434,54 @@
BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P */
BPF_EXIT_INSN(),
/* P */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P1 */
+ BPF_EXIT_INSN(),
+ /* P1 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P2 */
+ BPF_EXIT_INSN(),
+ /* P2 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P3 */
+ BPF_EXIT_INSN(),
+ /* P3 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P4 */
+ BPF_EXIT_INSN(),
+ /* P4 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P5 */
+ BPF_EXIT_INSN(),
+ /* P5 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P6 */
+ BPF_EXIT_INSN(),
+ /* P6 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P7 */
+ BPF_EXIT_INSN(),
+ /* P7 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P8 */
+ BPF_EXIT_INSN(),
+ /* P8 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P9 */
+ BPF_EXIT_INSN(),
+ /* P9 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P10 */
+ BPF_EXIT_INSN(),
+ /* P10 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P11 */
+ BPF_EXIT_INSN(),
+ /* P11 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P12 */
+ BPF_EXIT_INSN(),
+ /* P12 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P13 */
+ BPF_EXIT_INSN(),
+ /* P13 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P14 */
+ BPF_EXIT_INSN(),
+ /* P14 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P15 */
+ BPF_EXIT_INSN(),
+ /* P15 */
+ BPF_RAW_INSN(BPF_JMP|BPF_CALL, 0, 1, 0, 1), /* call P16 */
+ BPF_EXIT_INSN(),
+ /* P16 */
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
diff --git a/tools/testing/selftests/bpf/verifier/pseudo_func.c b/tools/testing/selftests/bpf/verifier/pseudo_func.c
index 63c5c67d51de..44d36b01aa6d 100644
--- a/tools/testing/selftests/bpf/verifier/pseudo_func.c
+++ b/tools/testing/selftests/bpf/verifier/pseudo_func.c
@@ -30,7 +30,7 @@
.prog_type = BPF_PROG_TYPE_TRACEPOINT,
.fixup_map_timer = { 3, 9 },
.result = REJECT,
- .errstr = "callback function cannot be the main program",
+ .errstr = "recursive call of the main program",
.func_info = { { 0, 4 /* main_prog */ } },
.func_info_cnt = 1,
.btf_strings = "\0int\0ctx\0main_prog",
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 75/81] selftests/bpf: Adjust tests to the order of verification and to the log
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (73 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 74/81] selftests/bpf: Adjust tests of recursion Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 76/81] selftests/bpf: Adjust tests of unprivileged progs Alexei Starovoitov
` (7 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier followed one path of a prog to the exit, and it took the fall
through of each conditional jump first. Then it went back to the last jump
and followed the other branch. Now the verifier goes through the basic
blocks in reverse post order: it verifies a block after the blocks that
jump to it. The target of a conditional jump comes before the fall through.
before now
2: if r0 & r0 goto 5
3: r0 = 0 first second
4: exit
5: r0 = 1 second first
6: exit
No verdict of an existing test changes. But the verifier finds the other
error first in a prog that has errors on two paths. The log has the same
states in a different sequence. Each test gets one of these changes:
- Remove the error that the test is not about.
- Expect the other message or the new sequence.
- Swap the two branches to keep the old sequence.
The format of the log changes too:
- A state has only live registers. "0: R1=ctx() R10=fp0" becomes
"0: R10=fp0".
- The log has no mark_precise lines.
- The statistics line has "limit 0", and it counts at least one state.
Add tests of maybe_fork_scalars() to verifier_bounds.c. When a register is
-1 or 0, this function splits the state in two before an 'and' or an 'or'
with a constant. The verifier continues with -1 and verifies a copy of the
state with 0 later.
At some insns the verifier keeps the states that arrive, and it drops a new
state that a kept state covers. [-1, 0] covers 0. Thus the copy must not go
back to such an insn. These insns are the insn where two paths meet, a loop
head, the first insn of a subprog and the insn after a call. The tests put
the 'and' or the 'or' at each of them. Only the copy makes an access out of
bounds, and the verifier must reject the prog.
One more prog has many forks in one basic block. The number of states must
not multiply.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/log_buf.c | 4 +-
.../selftests/bpf/prog_tests/log_fixup.c | 6 +-
.../selftests/bpf/prog_tests/reg_bounds.c | 5 +-
...f_qdisc_fail__invalid_dynptr_cross_frame.c | 4 +-
.../selftests/bpf/progs/cpumask_failure.c | 2 +-
.../bpf/progs/mem_rdonly_untrusted.c | 16 +-
.../selftests/bpf/progs/timer_failure.c | 9 +-
.../selftests/bpf/progs/verifier_align.c | 10 +-
.../selftests/bpf/progs/verifier_and.c | 4 +-
.../selftests/bpf/progs/verifier_bounds.c | 589 +++++++++++++++++-
.../bpf/progs/verifier_helper_value_access.c | 8 +-
.../bpf/progs/verifier_subprog_insn_stats.c | 16 +-
.../selftests/bpf/progs/verifier_unpriv.c | 23 +-
.../bpf/progs/verifier_unpriv_perf.c | 4 +-
14 files changed, 640 insertions(+), 60 deletions(-)
diff --git a/tools/testing/selftests/bpf/prog_tests/log_buf.c b/tools/testing/selftests/bpf/prog_tests/log_buf.c
index d6f14a232002..4f821fdc4a02 100644
--- a/tools/testing/selftests/bpf/prog_tests/log_buf.c
+++ b/tools/testing/selftests/bpf/prog_tests/log_buf.c
@@ -125,7 +125,7 @@ static void obj_load_log_buf(void)
ASSERT_OK_PTR(strstr(libbpf_log_buf, "libbpf: prog 'bad_prog': -- BEGIN PROG LOAD LOG --"),
"libbpf_log_correct");
ASSERT_STREQ(obj_log_buf, "", "obj_log__empty");
- ASSERT_STREQ(good_log_buf, "processed 4 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n",
+ ASSERT_STREQ(good_log_buf, "processed 4 insns (limit 0) max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n",
"good_log_ok");
ASSERT_STREQ(bad_log_buf, "", "bad_log_empty");
@@ -179,7 +179,7 @@ static void bpf_prog_load_log_buf(void)
opts.log_level = 2;
fd = bpf_prog_load(BPF_PROG_TYPE_SOCKET_FILTER, "good_prog", "GPL",
good_prog_insns, good_prog_insn_cnt, &opts);
- ASSERT_OK_PTR(strstr(log_buf, "0: R1=ctx() R10=fp0"), "good_log_2");
+ ASSERT_OK_PTR(strstr(log_buf, "0: R10=fp0"), "good_log_2");
ASSERT_GE(fd, 0, "good_fd2");
if (fd >= 0)
close(fd);
diff --git a/tools/testing/selftests/bpf/prog_tests/log_fixup.c b/tools/testing/selftests/bpf/prog_tests/log_fixup.c
index 90a98e23be61..89a7f5947081 100644
--- a/tools/testing/selftests/bpf/prog_tests/log_fixup.c
+++ b/tools/testing/selftests/bpf/prog_tests/log_fixup.c
@@ -41,7 +41,7 @@ static void bad_core_relo(size_t log_buf_size, enum trunc_type trunc_type)
"struct task_struct___bad.fake_field (0:1 @ offset 4)\n",
"log_buf_part2");
ASSERT_HAS_SUBSTR(log_buf,
- "max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n",
+ "max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n",
"log_buf_end");
break;
case TRUNC_PARTIAL:
@@ -50,7 +50,7 @@ static void bad_core_relo(size_t log_buf_size, enum trunc_type trunc_type)
"struct task_struct___bad.fake_field (0:1 @ offset 4)\n",
"log_buf_part2");
/* we shouldn't get full end of BPF verifier log */
- ASSERT_NULL(strstr(log_buf, "max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n"),
+ ASSERT_NULL(strstr(log_buf, "max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n"),
"log_buf_end");
break;
case TRUNC_FULL:
@@ -58,7 +58,7 @@ static void bad_core_relo(size_t log_buf_size, enum trunc_type trunc_type)
ASSERT_NULL(strstr(log_buf, "struct task_struct___bad.fake_field (0:1 @ offset 4)\n"),
"log_buf_part2");
/* we shouldn't get full end of BPF verifier log */
- ASSERT_NULL(strstr(log_buf, "max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0\n"),
+ ASSERT_NULL(strstr(log_buf, "max_states_per_insn 1 total_states 1 peak_states 1 mark_read 0\n"),
"log_buf_end");
break;
}
diff --git a/tools/testing/selftests/bpf/prog_tests/reg_bounds.c b/tools/testing/selftests/bpf/prog_tests/reg_bounds.c
index 7f170a69d1d8..d9e136720b49 100644
--- a/tools/testing/selftests/bpf/prog_tests/reg_bounds.c
+++ b/tools/testing/selftests/bpf/prog_tests/reg_bounds.c
@@ -1276,7 +1276,7 @@ static int parse_range_cmp_log(const char *log_buf, struct case_spec spec,
{true_pos + 1, 7, "R7=", true2_reg},
};
char buf[32];
- const char *p = log_buf, *q;
+ const char *p, *q;
int i, err;
for (i = 0; i < 4; i++) {
@@ -1291,8 +1291,9 @@ static int parse_range_cmp_log(const char *log_buf, struct case_spec spec,
* Different verifier passes may print
* 18: (bf) r0 = r6
* as well, but never followed by ';'.
+ * The branches are in the log in the order the verifier looked at them.
*/
- q = p;
+ q = log_buf;
while ((q = strstr(q, buf)) != NULL) {
const char *s = q + strlen(buf);
diff --git a/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c b/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c
index 2e23b8593af9..1d1684d2cf3b 100644
--- a/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c
+++ b/tools/testing/selftests/bpf/progs/bpf_qdisc_fail__invalid_dynptr_cross_frame.c
@@ -26,8 +26,10 @@ int BPF_PROG(invalid_dynptr_cross_frame, struct sk_buff *skb, struct Qdisc *sch,
bpf_dynptr_from_skb((struct __sk_buff *)skb, 0, &ptr);
hdr = bpf_dynptr_slice(&ptr, 0, NULL, sizeof(*hdr));
- if (!hdr)
+ if (!hdr) {
+ bpf_kfree_skb(skb);
return NET_XMIT_DROP;
+ }
free_skb(skb);
diff --git a/tools/testing/selftests/bpf/progs/cpumask_failure.c b/tools/testing/selftests/bpf/progs/cpumask_failure.c
index c89c88db39d1..715218d32105 100644
--- a/tools/testing/selftests/bpf/progs/cpumask_failure.c
+++ b/tools/testing/selftests/bpf/progs/cpumask_failure.c
@@ -45,7 +45,7 @@ int BPF_PROG(test_alloc_no_release, struct task_struct *task, u64 clone_flags)
}
SEC("tp_btf/task_newtask")
-__failure __msg("NULL pointer passed to trusted R1")
+__failure __msg("R1 type=scalar expected=ptr_, trusted_ptr_, rcu_ptr_")
int BPF_PROG(test_alloc_double_release, struct task_struct *task, u64 clone_flags)
{
struct bpf_cpumask *cpumask;
diff --git a/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c b/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c
index 23019023511a..6419e7ce1248 100644
--- a/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c
+++ b/tools/testing/selftests/bpf/progs/mem_rdonly_untrusted.c
@@ -203,9 +203,9 @@ int mixed_mem_untrusted_btf_id_type(void *ctx)
* returning 0.
*/
asm volatile (
- "r7 = %[p];"
- "if %[zero] != 0 goto +1;"
"r7 = %[q];"
+ "if %[zero] == 0 goto +1;"
+ "r7 = %[p];"
"r8 = *(u64 *)(r7 + 0);"
"%[v] = r8;"
: [v]"=r"(v)
@@ -247,9 +247,9 @@ int mixed_mem_btf_id_type(void *ctx)
* gotten on its own.
*/
asm volatile (
- "r7 = %[p];"
- "if %[zero] != 0 goto +1;"
"r7 = %[q];"
+ "if %[zero] == 0 goto +1;"
+ "r7 = %[p];"
"r8 = *(u32 *)(r7 + 0);"
"%[v] = r8;"
: [v]"=r"(v)
@@ -289,9 +289,9 @@ int mixed_rdonly_mem_btf_id_type(void *ctx)
* only its PTR_UNTRUSTED variant is.
*/
asm volatile (
- "r7 = %[p];"
- "if %[zero] != 0 goto +1;"
"r7 = %[q];"
+ "if %[zero] == 0 goto +1;"
+ "r7 = %[p];"
"r8 = *(u32 *)(r7 + 0);"
"%[v] = r8;"
: [v]"=r"(v)
@@ -324,9 +324,9 @@ int mixed_mem_mem_type(void *ctx)
* hence without the BPF_PROBE_MEM rewrite the other path needs.
*/
asm volatile (
- "r7 = %[q];"
- "if %[zero] == 0 goto +1;"
"r7 = %[p];"
+ "if %[zero] != 0 goto +1;"
+ "r7 = %[q];"
"r8 = *(u64 *)(r7 + 0);"
"%[v] = r8;"
: [v]"=r"(v)
diff --git a/tools/testing/selftests/bpf/progs/timer_failure.c b/tools/testing/selftests/bpf/progs/timer_failure.c
index 0538269101ca..84ea861575fc 100644
--- a/tools/testing/selftests/bpf/progs/timer_failure.c
+++ b/tools/testing/selftests/bpf/progs/timer_failure.c
@@ -43,14 +43,7 @@ static unsigned long timer_cb_ret_bad()
}
SEC("fentry/bpf_fentry_test1")
-__log_level(2)
-__flag(BPF_F_TEST_STATE_FREQ)
__failure
-/* check that fallthrough code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before")
-__msg(": (85) call bpf_get_prandom_u32#7") /* anchor message */
-/* check that branch code path marks r0 as precise */
-__msg("mark_precise: frame0: regs=r0 stack= before ") __msg(": (85) call bpf_get_prandom_u32#7")
__msg("should have been in [0, 0]")
long BPF_PROG2(test_bad_ret, int, a)
{
@@ -80,7 +73,7 @@ static int timer_cb_infinite_loop(void *map, int *key, struct elem *val)
}
SEC("fentry/bpf_fentry_test1")
-__failure __msg("infinite loop detected")
+__failure __msg("loop at insn {{[0-9]+}} never exits")
long BPF_PROG2(test_infinite_loop_cb, int, a)
{
struct bpf_timer *timer;
diff --git a/tools/testing/selftests/bpf/progs/verifier_align.c b/tools/testing/selftests/bpf/progs/verifier_align.c
index 0083ef8b8ba7..aecc5279dbaf 100644
--- a/tools/testing/selftests/bpf/progs/verifier_align.c
+++ b/tools/testing/selftests/bpf/progs/verifier_align.c
@@ -13,7 +13,7 @@
SEC("tc")
__success __log_level(2)
__flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
__msg("0: {{.*}} R3=2")
__msg("1: {{.*}} R3=4")
__msg("2: {{.*}} R3=8")
@@ -35,7 +35,7 @@ __naked void mov(void)
SEC("tc")
__success __log_level(2)
__flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
__msg("0: {{.*}}R3=1")
__msg("1: {{.*}}R3=2")
__msg("2: {{.*}}R3=4")
@@ -69,7 +69,7 @@ __naked void shift(void)
SEC("tc")
__success __log_level(2)
__flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
__msg("0: {{.*}}R3=4")
__msg("1: {{.*}}R3=8")
__msg("2: {{.*}}R3=10")
@@ -93,7 +93,7 @@ __naked void addsub(void)
SEC("tc")
__success __log_level(2)
__flag(BPF_F_ANY_ALIGNMENT)
-__msg("0: R1=ctx() R10=fp0")
+__msg("0: R10=fp0")
__msg("0: {{.*}}R3=7")
__msg("1: {{.*}}R3=7")
__msg("2: {{.*}}R3=14")
@@ -532,7 +532,7 @@ __msg("9: {{.*}} R6={{[^)]*}}var_off=(0x0; 0x3c)")
/* Adding 14 makes R6 be (4n+2) */
__msg("10: {{.*}} R6={{[^)]*}}var_off=(0x2; 0x7c)")
/* Subtracting from packet pointer overflows ubounds */
-__msg("13: R5={{[^)]*}}var_off=(0xffffffffffffff82; 0x7c)")
+__msg("12: {{.*}} R5={{[^)]*}}var_off=(0xffffffffffffff82; 0x7c)")
/* New unknown value in R7 is (4n), >= 76 */
__msg("14: {{.*}} R7={{[^)]*}}var_off=(0x0; 0x7fc)")
/* Adding it to packet pointer gives nice bounds again */
diff --git a/tools/testing/selftests/bpf/progs/verifier_and.c b/tools/testing/selftests/bpf/progs/verifier_and.c
index 2b4fdca162be..fb64ea9928f1 100644
--- a/tools/testing/selftests/bpf/progs/verifier_and.c
+++ b/tools/testing/selftests/bpf/progs/verifier_and.c
@@ -38,9 +38,9 @@ __naked void invalid_and_of_negative_number(void)
r1 &= -4; \
r1 <<= 2; \
r0 += r1; \
-l0_%=: r1 = %[test_val_foo]; \
+ r1 = %[test_val_foo]; \
*(u64*)(r0 + 0) = r1; \
- exit; \
+l0_%=: exit; \
" :
: __imm(bpf_map_lookup_elem),
__imm_addr(map_hash_48b),
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds.c b/tools/testing/selftests/bpf/progs/verifier_bounds.c
index df8d5309657e..c999d6ae5fc8 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds.c
@@ -1797,8 +1797,8 @@ l1_%=: r0 = 1; \
SEC("socket")
__description("jset on same register, scalar value unknown branch 1")
-__msg("3: (b7) r0 = 0 {{.*}} R0=0")
__msg("5: (b7) r0 = 1 {{.*}} R0=1")
+__msg("3: (b7) r0 = 0 {{.*}} R0=0")
__success __log_level(2)
__naked void jset_on_same_register_3(void *ctx)
{
@@ -1818,8 +1818,8 @@ l1_%=: r0 = 1; \
SEC("socket")
__description("jset on same register, scalar value unknown branch 2")
-__msg("4: (b7) r0 = 0 {{.*}} R0=0")
__msg("6: (b7) r0 = 1 {{.*}} R0=1")
+__msg("4: (b7) r0 = 0 {{.*}} R0=0")
__success __log_level(2)
__naked void jset_on_same_register_4(void *ctx)
{
@@ -1840,8 +1840,8 @@ l1_%=: r0 = 1; \
SEC("socket")
__description("jset on same register, scalar value unknown branch 3")
-__msg("4: (b7) r0 = 0 {{.*}} R0=0")
__msg("6: (b7) r0 = 1 {{.*}} R0=1")
+__msg("4: (b7) r0 = 0 {{.*}} R0=0")
__success __log_level(2)
__naked void jset_on_same_register_5(void *ctx)
{
@@ -2159,6 +2159,589 @@ l0_%=: r0 = 0; \
: __clobber_all);
}
+/*
+ * -1 and 0 meet at the insn that forks. The state with [-1, 0] is kept there. The
+ * pushed state is within it, so it's not there that it is pushed to.
+ */
+SEC("socket")
+__description("maybe_fork_scalars: AND where paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and_scalar_fork_at_join_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r1 = *(u64*)(r9 + 0); \
+ r6 = -1; \
+ if r1 == 0 goto l1_%=; \
+ r6 = 0; \
+l1_%=: r6 &= 8; \
+ /* r6 is 8 (current) or 0 (pushed) */ \
+ r6 ^= 8; \
+ /* pushed path: r6 = 8, OOB for value_size=8 */ \
+ r9 += r6; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+SEC("socket")
+__description("maybe_fork_scalars: OR where paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void or_scalar_fork_at_join_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r1 = *(u64*)(r9 + 0); \
+ r6 = -1; \
+ if r1 == 0 goto l1_%=; \
+ r6 = 0; \
+l1_%=: r6 |= 8; \
+ /* r6 is -1 (current) or 8 (pushed) */ \
+ if r6 s< 0 goto l0_%=; \
+ /* pushed path: r6 = 8, OOB for value_size=8 */ \
+ r9 += r6; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+SEC("socket")
+__description("maybe_fork_scalars: 32-bit AND where paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and32_scalar_fork_at_join_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r1 = *(u64*)(r9 + 0); \
+ r6 = -1; \
+ if r1 == 0 goto l1_%=; \
+ r6 = 0; \
+l1_%=: w6 &= 8; \
+ /* r6 is 8 (current) or 0 (pushed) */ \
+ r6 ^= 8; \
+ /* pushed path: r6 = 8, OOB for value_size=8 */ \
+ r9 += r6; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+SEC("socket")
+__description("maybe_fork_scalars: 32-bit OR where paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void or32_scalar_fork_at_join_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r1 = *(u64*)(r9 + 0); \
+ r6 = -1; \
+ if r1 == 0 goto l1_%=; \
+ r6 = 0; \
+l1_%=: w6 |= 8; \
+ /* r6 is 0xffffffff (current) or 8 (pushed) */ \
+ if w6 s< 0 goto l0_%=; \
+ /* pushed path: r6 = 8, OOB for value_size=8 */ \
+ r9 += r6; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+/* Only the low half of r6 is in [-1, 0]. */
+SEC("socket")
+__description("maybe_fork_scalars: AND of a 32-bit range where paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and_scalar32_fork_at_join_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r1 = *(u64*)(r9 + 0); \
+ w6 = -1; \
+ if r1 == 0 goto l1_%=; \
+ r6 = 0; \
+l1_%=: r6 &= 8; \
+ /* r6 is 8 (current) or 0 (pushed) */ \
+ r6 ^= 8; \
+ /* pushed path: r6 = 8, OOB for value_size=8 */ \
+ r9 += r6; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+SEC("socket")
+__description("maybe_fork_scalars: 32-bit AND of a 32-bit range where paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and32_scalar32_fork_at_join_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r1 = *(u64*)(r9 + 0); \
+ w6 = -1; \
+ if r1 == 0 goto l1_%=; \
+ r6 = 0; \
+l1_%=: w6 &= 8; \
+ /* r6 is 8 (current) or 0 (pushed) */ \
+ r6 ^= 8; \
+ /* pushed path: r6 = 8, OOB for value_size=8 */ \
+ r9 += r6; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+SEC("socket")
+__description("maybe_fork_scalars: 32-bit OR of a 32-bit range where paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void or32_scalar32_fork_at_join_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r1 = *(u64*)(r9 + 0); \
+ w6 = -1; \
+ if r1 == 0 goto l1_%=; \
+ r6 = 0; \
+l1_%=: w6 |= 8; \
+ /* r6 is 0xffffffff (current) or 8 (pushed) */ \
+ if w6 s< 0 goto l0_%=; \
+ /* pushed path: r6 = 8, OOB for value_size=8 */ \
+ r9 += r6; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+/* The high half is not known, and a 64-bit insn doesn't make it 0 in the pushed state. */
+SEC("socket")
+__description("maybe_fork_scalars: AND of a 32-bit range keeps the high half")
+__failure __msg("invalid access to map value")
+__naked void and_scalar32_fork_keeps_high_half(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r6 = *(u64*)(r9 + 0); \
+ if w6 s> 0 goto l0_%=; \
+ if w6 s< -1 goto l0_%=; \
+ r1 = 0x800000008 ll; \
+ if r6 s< 0 goto l1_%=; \
+ r0 = 0; \
+l1_%=: r6 &= r1; \
+ /* the low half is 8 (current) or 0 (pushed) */ \
+ if w6 != 0 goto l0_%=; \
+ /* pushed path: r6 = 8 if bit 35 was set, OOB */\
+ r6 >>= 32; \
+ r9 += r6; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+SEC("socket")
+__description("maybe_fork_scalars: AND after paths meet rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and_scalar_fork_after_join_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r1 = *(u64*)(r9 + 0); \
+ r6 = -1; \
+ if r1 == 0 goto l1_%=; \
+ r6 = 0; \
+l1_%=: r1 = 0; \
+ r6 &= 8; \
+ /* r6 is 8 (current) or 0 (pushed) */ \
+ r6 ^= 8; \
+ /* pushed path: r6 = 8, OOB for value_size=8 */ \
+ r9 += r6; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+SEC("socket")
+__description("maybe_fork_scalars: AND where paths meet allows in-bounds")
+__success __retval(0)
+__log_level(2)
+__msg("r6 &= 4 {{.*}} R6=4")
+__msg("from 13 to 14: R6=0")
+__naked void and_scalar_fork_at_join_allows_inbounds(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r1 = *(u64*)(r9 + 0); \
+ r6 = -1; \
+ if r1 == 0 goto l1_%=; \
+ r6 = 0; \
+l1_%=: r6 &= 4; \
+ /* r6 is 4 (current) or 0 (pushed) */ \
+ r9 += r6; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+/*
+ * The insn that forks is the head of a loop whose iterations are looked at one by one,
+ * because of r8. The pushed state is in the iteration of the state that it comes from.
+ */
+SEC("socket")
+__description("maybe_fork_scalars: AND at the head of a loop rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and_scalar_fork_at_loop_head_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r7 = 0; \
+ r8 = 0; \
+ r6 = *(u64*)(r9 + 0); \
+ r6 s>>= 63; \
+l1_%=: r6 &= 4; \
+ /* r6 is 4 (current) or 0 (pushed) */ \
+ r6 ^= 4; \
+ r6 *= r8; \
+ /* pushed path: r6 = 8 at last, OOB */ \
+ r1 = r9; \
+ r1 += r6; \
+ r0 = *(u8*)(r1 + 0); \
+ r6 = *(u64*)(r9 + 0); \
+ r6 s>>= 63; \
+ r7 += 1; \
+ r8 = r7; \
+ if r7 < 3 goto l1_%=; \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+/*
+ * The other insns that keep their states: the head of a loop that has one state for
+ * all iterations,
+ */
+SEC("socket")
+__description("maybe_fork_scalars: AND at the head of a long loop rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and_scalar_fork_at_long_loop_head_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r7 = 0; \
+ r6 = -1; \
+l1_%=: r6 &= 8; \
+ /* r6 is 8 (current) or 0 (pushed) */ \
+ r6 ^= 8; \
+ /* pushed path: r6 = 8, OOB for value_size=8 */ \
+ r1 = r9; \
+ r1 += r6; \
+ r0 = *(u8*)(r1 + 0); \
+ r1 = *(u64*)(r9 + 0); \
+ r6 = -1; \
+ if r1 == 0 goto l2_%=; \
+ r6 = 0; \
+l2_%=: r7 += 1; \
+ if r7 < 100 goto l1_%=; \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+static __naked __noinline __used
+unsigned long and_8(void)
+{
+ asm volatile (" \
+ r1 &= 8; \
+ r0 = r1; \
+ exit; \
+" ::: __clobber_all);
+}
+
+/* the first insn of a function, */
+SEC("socket")
+__description("maybe_fork_scalars: AND at the entry of a function rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and_scalar_fork_at_func_entry_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r1 = *(u64*)(r9 + 0); \
+ r1 s>>= 63; \
+ call and_8; \
+ /* r0 is 8 (current) or 0 (pushed) */ \
+ r0 ^= 8; \
+ /* pushed path: r0 = 8, OOB for value_size=8 */ \
+ r9 += r0; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+static __naked __noinline __used
+unsigned long sign_of(void)
+{
+ asm volatile (" \
+ r0 = r1; \
+ r0 s>>= 63; \
+ exit; \
+" ::: __clobber_all);
+}
+
+/* and the insn that a function returns to. */
+SEC("socket")
+__description("maybe_fork_scalars: AND after a call rejects OOB")
+__failure __msg("invalid access to map value")
+__naked void and_scalar_fork_after_call_rejects_oob(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u64*)(r10 - 8) = r1; \
+ r2 = r10; \
+ r2 += -8; \
+ r1 = %[map_hash_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l0_%=; \
+ r9 = r0; \
+ r1 = *(u64*)(r9 + 0); \
+ call sign_of; \
+ r0 &= 8; \
+ /* r0 is 8 (current) or 0 (pushed) */ \
+ r0 ^= 8; \
+ /* pushed path: r0 = 8, OOB for value_size=8 */ \
+ r9 += r0; \
+ r0 = *(u8*)(r9 + 0); \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+static __naked __noinline __used
+unsigned long and_7_is_not_3(void)
+{
+ asm volatile (" \
+ r1 &= 7; \
+ if r1 != 3 goto +1; \
+ r10 = 0; \
+ if r2 == 0 goto l0_%=; \
+ r6 = r2; \
+ call %[bpf_get_prandom_u32]; \
+ r1 = r0; \
+ r1 <<= 32; \
+ r1 s>>= 63; \
+ r2 = r6; \
+ r2 += -1; \
+ call and_7_is_not_3; \
+l0_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/*
+ * What a call of a recursive function brings to its entry is widened against the states that
+ * are there. The pushed state is not one of them: [-1, 0] would look like 0 that grew.
+ */
+SEC("socket")
+__description("maybe_fork_scalars: AND at the entry of a recursive function")
+__success
+__naked void and_scalar_fork_at_recursive_func_entry(void)
+{
+ asm volatile (" \
+ call %[bpf_get_prandom_u32]; \
+ r1 = r0; \
+ r1 <<= 32; \
+ r1 s>>= 63; \
+ r2 = 2; \
+ call and_7_is_not_3; \
+ r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+#define FORK \
+ "r6 = *(u64*)(r9 + 0);" \
+ "r6 s>>= 63;" \
+ "r6 &= 4;" \
+ "*(u64*)(r9 + 0) = r6;"
+#define FORK4 FORK FORK FORK FORK
+
+/*
+ * The pushed states of one block that don't differ in what is live are one state: 16 forks
+ * are not 2^16 states.
+ */
+SEC("socket")
+__description("maybe_fork_scalars: 16 forks in a block")
+__success __retval(0)
+__log_level(4)
+__msg("processed {{[0-9][0-9]?[0-9]?}} insns")
+__naked void and_scalar_16_forks_in_a_block(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64*)(r10 - 8) = r1;"
+ "r2 = r10;"
+ "r2 += -8;"
+ "r1 = %[map_hash_8b] ll;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto l0_%=;"
+ "r9 = r0;"
+ FORK4 FORK4 FORK4 FORK4
+"l0_%=:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_map_lookup_elem),
+ __imm_addr(map_hash_8b)
+ : __clobber_all);
+}
+
+#undef FORK4
+#undef FORK
+
/*
* Last jump can be detected as always taken because the intersection of R5 and
* R7 32bit tnums produces a constant that isn't within R7's s32 bounds.
diff --git a/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c b/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c
index c6603a118fdc..1b13d3a8dcfa 100644
--- a/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c
+++ b/tools/testing/selftests/bpf/progs/verifier_helper_value_access.c
@@ -717,12 +717,12 @@ __naked void bounds_check_using_bad_access_1(void)
*(u64*)(r2 + 0) = r1; \
r1 = %[map_hash_48b] ll; \
call %[bpf_map_lookup_elem]; \
- if r0 == 0 goto l0_%=; \
+ if r0 == 0 goto l1_%=; \
r1 = r0; \
r3 = *(u32*)(r0 + 0); \
if r3 < 32 goto l1_%=; \
r1 += r3; \
-l0_%=: r0 = 0; \
+ r0 = 0; \
*(u8*)(r1 + 0) = r0; \
r0 = 0; \
exit; \
@@ -775,12 +775,12 @@ __naked void bounds_check_using_bad_access_2(void)
*(u64*)(r2 + 0) = r1; \
r1 = %[map_hash_48b] ll; \
call %[bpf_map_lookup_elem]; \
- if r0 == 0 goto l0_%=; \
+ if r0 == 0 goto l1_%=; \
r1 = r0; \
r3 = *(u32*)(r0 + 0); \
if r3 <= 32 goto l1_%=; \
r1 += r3; \
-l0_%=: r0 = 0; \
+ r0 = 0; \
*(u8*)(r1 + 0) = r0; \
r0 = 0; \
exit; \
diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c b/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c
index 8f6082fdb5c8..e7636b86cc29 100644
--- a/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c
+++ b/tools/testing/selftests/bpf/progs/verifier_subprog_insn_stats.c
@@ -127,13 +127,13 @@ static __u64 stats_async_schedule(struct bpf_map *map, __u32 *key,
SEC("?raw_tp")
__success __log_level(4)
/*
- * self: 9 + 7 + 2 = 18
- * totals: leaf 2, scheduler 7, main root 18
+ * self: 8 + 7 + 2 = 17
+ * totals: leaf 2, scheduler 7, main root 17
*/
-__msg("subprog 0 (stats_async_direct) main insns_self 9 insns_total 18 stack 0")
+__msg("subprog 0 (stats_async_direct) main insns_self 8 insns_total 17 stack 0")
__msg("subprog {{[0-9]+}} (stats_async_schedule) static insns_self 7 insns_total 7 stack 0")
__msg("subprog {{[0-9]+}} (stats_async_leaf) static insns_self 2 insns_total 2 stack 0")
-__msg("processed 18 insns")
+__msg("processed 17 insns")
__naked int stats_async_direct(void)
{
asm volatile (
@@ -194,14 +194,14 @@ static __u64 stats_async_nested_schedule(struct bpf_map *map, __u32 *key,
SEC("?raw_tp")
__success __log_level(4)
/*
- * self: 9 + 7 + 7 + 2 = 25
- * totals: leaf 2, outer 7, scheduler 7, main root 25
+ * self: 8 + 7 + 7 + 2 = 24
+ * totals: leaf 2, outer 7, scheduler 7, main root 24
*/
-__msg("subprog 0 (stats_async_nested) main insns_self 9 insns_total 25 stack 0")
+__msg("subprog 0 (stats_async_nested) main insns_self 8 insns_total 24 stack 0")
__msg("subprog {{[0-9]+}} (stats_async_nested_schedule) static insns_self 7 insns_total 7 stack 0")
__msg("subprog {{[0-9]+}} (stats_async_outer) static insns_self 7 insns_total 7 stack 0")
__msg("subprog {{[0-9]+}} (stats_async_nested_leaf) static insns_self 2 insns_total 2 stack 0")
-__msg("processed 25 insns")
+__msg("processed 24 insns")
__naked int stats_async_nested(void)
{
asm volatile (
diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv.c b/tools/testing/selftests/bpf/progs/verifier_unpriv.c
index a54b5a58b945..eda7ac8e2145 100644
--- a/tools/testing/selftests/bpf/progs/verifier_unpriv.c
+++ b/tools/testing/selftests/bpf/progs/verifier_unpriv.c
@@ -394,8 +394,8 @@ __naked void fill_of_different_pointers_stx(void)
r2 = r10; \
r2 += -16; \
*(u64*)(r6 + 0) = r2; \
-l0_%=: if r1 != 0 goto l1_%=; \
- *(u64*)(r6 + 0) = r1; \
+ goto l1_%=; \
+l0_%=: *(u64*)(r6 + 0) = r1; \
l1_%=: r1 = *(u64*)(r6 + 0); \
*(u32*)(r1 + %[__sk_buff_mark]) = r3; \
r0 = 0; \
@@ -420,8 +420,8 @@ __naked void fill_of_different_pointers_st(void)
r2 = r10; \
r2 += -16; \
*(u64*)(r6 + 0) = r2; \
-l0_%=: if r1 != 0 goto l1_%=; \
- *(u64*)(r6 + 0) = r1; \
+ goto l1_%=; \
+l0_%=: *(u64*)(r6 + 0) = r1; \
l1_%=: r1 = *(u64*)(r6 + 0); \
.8byte %[st_mem]; \
r0 = 0; \
@@ -435,7 +435,8 @@ l1_%=: r1 = *(u64*)(r6 + 0); \
SEC("tc")
__description("unpriv: spill/fill of different pointers stx - ctx and sock")
-__failure __msg("type=ctx expected=sock")
+/* bpf_sk_release(skb) is an error too: "type=ctx expected=sock". This one is found first. */
+__failure __msg("R1 invalid mem access 'sock_or_null'")
__naked void pointers_stx_ctx_and_sock(void)
{
asm volatile (" \
@@ -449,10 +450,10 @@ __naked void pointers_stx_ctx_and_sock(void)
r6 += -8; \
r1 = r8; \
/* if (skb == NULL) *target = sock; */ \
- if r1 == 0 goto l0_%=; \
+ if r1 != 0 goto l0_%=; \
*(u64*)(r6 + 0) = r2; \
+ goto l1_%=; \
l0_%=: /* else *target = skb; */ \
- if r1 != 0 goto l1_%=; \
*(u64*)(r6 + 0) = r1; \
l1_%=: /* struct __sk_buff *skb = *target; */ \
r1 = *(u64*)(r6 + 0); \
@@ -490,10 +491,10 @@ __naked void different_pointers_stx_leak_sock(void)
r6 += -8; \
r1 = r8; \
/* if (skb == NULL) *target = sock; */ \
- if r1 == 0 goto l0_%=; \
+ if r1 != 0 goto l0_%=; \
*(u64*)(r6 + 0) = r2; \
+ goto l1_%=; \
l0_%=: /* else *target = skb; */ \
- if r1 != 0 goto l1_%=; \
*(u64*)(r6 + 0) = r1; \
l1_%=: /* struct __sk_buff *skb = *target; */ \
r1 = *(u64*)(r6 + 0); \
@@ -526,8 +527,8 @@ __naked void stx_sock_and_ctx_read(void)
/* if (skb) *target = skb */ \
if r1 == 0 goto l0_%=; \
*(u64*)(r6 + 0) = r1; \
+ goto l1_%=; \
l0_%=: /* else *target = sock */ \
- if r1 != 0 goto l1_%=; \
*(u64*)(r6 + 0) = r2; \
l1_%=: /* struct bpf_sock *sk = *target; */ \
r1 = *(u64*)(r6 + 0); \
@@ -565,8 +566,8 @@ __naked void stx_sock_and_ctx_write(void)
/* if (skb) *target = skb */ \
if r1 == 0 goto l0_%=; \
*(u64*)(r6 + 0) = r1; \
+ goto l1_%=; \
l0_%=: /* else *target = sock */ \
- if r1 != 0 goto l1_%=; \
*(u64*)(r6 + 0) = r2; \
l1_%=: /* struct bpf_sock *sk = *target; */ \
r1 = *(u64*)(r6 + 0); \
diff --git a/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c b/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c
index 4d77407a0a79..5363093a4a5c 100644
--- a/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c
+++ b/tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c
@@ -17,8 +17,8 @@ __naked void fill_of_different_pointers_ldx(void)
r2 = r10; \
r2 += %[__imm_0]; \
*(u64*)(r6 + 0) = r2; \
-l0_%=: if r1 != 0 goto l1_%=; \
- *(u64*)(r6 + 0) = r1; \
+ goto l1_%=; \
+l0_%=: *(u64*)(r6 + 0) = r1; \
l1_%=: r1 = *(u64*)(r6 + 0); \
r1 = *(u64*)(r1 + %[sample_period]); \
r0 = 0; \
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 76/81] selftests/bpf: Adjust tests of unprivileged progs
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (74 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 75/81] selftests/bpf: Adjust tests to the order of verification and to the log Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 77/81] selftests/bpf: Test states that are not joined Alexei Starovoitov
` (6 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
For an unprivileged prog the verifier masks 'ptr += scalar' against
speculative execution. Each insn has one limit for the mask. Thus the
verifier rejected a prog in which two paths get to the insn with different
scalars:
r1 = 3
goto 2f
1: r1 = 5
2: r1 += r0 ; r0 is a pointer to a map value
R1 tried to add from different maps, paths or scalars
Now the verifier joins the two states before the insn. r1 is [3, 5], and
the insn gets one limit. The verifier verifies the subsequent code for this
range. 5 progs of verifier_value_ptr_arith.c load.
5 other progs have two errors. The verifier rejects them as before, but the
order of verification is different, and it finds the other error first.
Expect that message.
3 progs of verifier_array_access.c expected mark_precise lines. The log
does not have these lines.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/progs/verifier_array_access.c | 14 +++++---------
.../bpf/progs/verifier_bounds_deduction.c | 2 +-
.../selftests/bpf/progs/verifier_map_ret_val.c | 2 +-
.../selftests/bpf/progs/verifier_value_or_null.c | 2 +-
.../bpf/progs/verifier_value_ptr_arith.c | 15 +++++----------
5 files changed, 13 insertions(+), 22 deletions(-)
diff --git a/tools/testing/selftests/bpf/progs/verifier_array_access.c b/tools/testing/selftests/bpf/progs/verifier_array_access.c
index 0a187ff725cc..d7f16878648a 100644
--- a/tools/testing/selftests/bpf/progs/verifier_array_access.c
+++ b/tools/testing/selftests/bpf/progs/verifier_array_access.c
@@ -248,7 +248,7 @@ l0_%=: exit; \
SEC("socket")
__description("invalid map access into an array with no floor check")
__failure __msg("R0 unbounded memory access")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 unbounded memory access")
__flag(BPF_F_ANY_ALIGNMENT)
__naked void array_with_no_floor_check(void)
{
@@ -280,7 +280,7 @@ l0_%=: exit; \
SEC("socket")
__description("invalid map access into an array with a invalid max check")
__failure __msg("invalid access to map value, value_size=48 off=44 size=8")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 max value is outside of the allowed memory range")
__flag(BPF_F_ANY_ALIGNMENT)
__naked void with_a_invalid_max_check_1(void)
{
@@ -541,8 +541,7 @@ l0_%=: exit; \
SEC("socket")
__description("valid map access into an array using constant without nullness")
-__success __retval(4) __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -8) = {{(1|r[0-9])}}")
+__success __retval(4)
unsigned int an_array_with_a_constant_no_nullness(void)
{
/* Need 8-byte alignment for spill tracking */
@@ -557,9 +556,7 @@ unsigned int an_array_with_a_constant_no_nullness(void)
SEC("socket")
__description("valid multiple map access into an array using constant without nullness")
-__success __retval(8) __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -16) = {{(0|r[0-9])}}")
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -8) = {{(1|r[0-9])}}")
+__success __retval(8)
unsigned int multiple_array_with_a_constant_no_nullness(void)
{
__u32 __attribute__((aligned(8))) key = 1;
@@ -595,8 +592,7 @@ unsigned int an_array_with_a_32bit_constant_0_no_nullness(void)
SEC("socket")
__description("valid map access into a pcpu array using constant without nullness")
-__success __retval(4) __log_level(2)
-__msg("mark_precise: frame0: regs= stack=-8 before {{[0-9]}}: ({{[a-f0-9]+}}) *(u32 *)(r10 -8) = {{(1|r[0-9])}}")
+__success __retval(4)
unsigned int a_pcpu_array_with_a_constant_no_nullness(void)
{
__u32 __attribute__((aligned(8))) key = 1;
diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c b/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c
index 260a6df264e3..eddbb4f906ee 100644
--- a/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c
+++ b/tools/testing/selftests/bpf/progs/verifier_bounds_deduction.c
@@ -102,7 +102,7 @@ l0_%=: r0 -= r1; \
SEC("socket")
__description("check deducing bounds from const, 7")
__failure __msg("dereference of modified ctx ptr")
-__msg_unpriv("R1 has pointer with unsupported alu operation")
+__msg_unpriv("math between ctx pointer and register with unbounded min value is not allowed")
__flag(BPF_F_ANY_ALIGNMENT)
__naked void deducing_bounds_from_const_7(void)
{
diff --git a/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c b/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c
index 1639628b832d..2b65ab508596 100644
--- a/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c
+++ b/tools/testing/selftests/bpf/progs/verifier_map_ret_val.c
@@ -83,7 +83,7 @@ l0_%=: exit; \
SEC("socket")
__description("sometimes access memory with incorrect alignment")
__failure __msg("R0 invalid mem access")
-__msg_unpriv("R0 leaks addr")
+__msg_unpriv("R0 invalid mem access")
__flag(BPF_F_STRICT_ALIGNMENT)
__naked void access_memory_with_incorrect_alignment_2(void)
{
diff --git a/tools/testing/selftests/bpf/progs/verifier_value_or_null.c b/tools/testing/selftests/bpf/progs/verifier_value_or_null.c
index 8ff668a242eb..2a5bdac24925 100644
--- a/tools/testing/selftests/bpf/progs/verifier_value_or_null.c
+++ b/tools/testing/selftests/bpf/progs/verifier_value_or_null.c
@@ -183,7 +183,7 @@ l1_%=: exit; \
SEC("socket")
__description("invalid map access from else condition")
__failure __msg("R0 unbounded memory access")
-__failure_unpriv __msg_unpriv("R0 leaks addr")
+__failure_unpriv __msg_unpriv("R0 unbounded memory access")
__flag(BPF_F_ANY_ALIGNMENT)
__naked void map_access_from_else_condition(void)
{
diff --git a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
index b3b701b44550..2355915766e3 100644
--- a/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
+++ b/tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c
@@ -41,8 +41,7 @@ struct {
SEC("socket")
__description("map access: known scalar += value_ptr unknown vs const")
-__success __failure_unpriv
-__msg_unpriv("R1 tried to add from different maps, paths or scalars")
+__success __success_unpriv
__retval(1)
__naked void value_ptr_unknown_vs_const(void)
{
@@ -79,8 +78,7 @@ l2_%=: r0 = 1; \
SEC("socket")
__description("map access: known scalar += value_ptr const vs unknown")
-__success __failure_unpriv
-__msg_unpriv("R1 tried to add from different maps, paths or scalars")
+__success __success_unpriv
__retval(1)
__naked void value_ptr_const_vs_unknown(void)
{
@@ -117,8 +115,7 @@ l2_%=: r0 = 1; \
SEC("socket")
__description("map access: known scalar += value_ptr const vs const (ne)")
-__success __failure_unpriv
-__msg_unpriv("R1 tried to add from different maps, paths or scalars")
+__success __success_unpriv
__retval(1)
__naked void ptr_const_vs_const_ne(void)
{
@@ -225,8 +222,7 @@ l2_%=: r0 = 1; \
SEC("socket")
__description("map access: known scalar += value_ptr unknown vs unknown (lt)")
-__success __failure_unpriv
-__msg_unpriv("R1 tried to add from different maps, paths or scalars")
+__success __success_unpriv
__retval(1)
__naked void ptr_unknown_vs_unknown_lt(void)
{
@@ -270,8 +266,7 @@ l2_%=: r0 = 1; \
SEC("socket")
__description("map access: known scalar += value_ptr unknown vs unknown (gt)")
-__success __failure_unpriv
-__msg_unpriv("R1 tried to add from different maps, paths or scalars")
+__success __success_unpriv
__retval(1)
__naked void ptr_unknown_vs_unknown_gt(void)
{
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 77/81] selftests/bpf: Test states that are not joined
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (75 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 76/81] selftests/bpf: Adjust tests of unprivileged progs Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 78/81] selftests/bpf: Test progs that run out of budget Alexei Starovoitov
` (5 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
Where two paths of a prog meet, the verifier joins their states into one
state that covers both. For example, ret = -2 and ret = 0 give
ret = [-2, 0]. Then the verifier verifies the subsequent code one time. But
a join loses the relation between registers, and progs depend on such
relations:
ret = lookup(&p); /* ret = -ENOENT and p is not set, or */
if (ret < 0) /* ret = 0 and p is set */
goto drop;
use(p);
When the compiler inlines lookup(), the two states meet before the jump.
Their join gets to use(p) without p. Thus the verifier keeps both states,
and verifies each of them, when the prog can depend on the difference.
the two states differ in example result
----------------------------------- ------------------------- ---------
a register that only one state p is set, p is not set both stay
initializes
a scalar that a later jump compares ret = -2, ret = 0 both stay
with a constant, the ranges leave
a gap
such a scalar, the ranges touch ret = -1, ret = 0 joined
two such scalars err = 1 and ret = 1, both stay
err = 0 and ret = 0
a constant that becomes an offset off = 0, off = 8 both stay
in the stack
The join of two ranges with a gap has values that neither state has. A
later jump can send these values to code that neither state gets to. The
join of the two pairs has err = 1 with ret = 0, which never occurs. Ranges
that touch have no such values. But their join loses what ret says about
other registers:
ret = idx < 8 ? 0 : -1;
if (ret < 0)
goto drop;
use(buf[idx]); /* rejected, the range of idx is lost */
A verifier that follows each path accepts this prog. With -EINVAL in place
of -1 it loads.
verifier_jumps_ahead.c has the first four rows, with patterns from Cilium.
verifier_var_off.c has the last row, with patterns from sched-ext
schedulers. The progs load only if both states stay. The verifier must
reject the unsafe variants.
The last row applies around a loop too. A prog that pushes to and pops from
'u8 stack[10]' gets one state for each value of the index. The compiler
tests a copy of the index and calculates the index again for the access.
When the index is a constant, the verifier knows the result of both. When
it is a range, the test of the copy says nothing about the index.
States that stay multiply: 20 flags give 2^20 combinations. Thus the rules
apply only while less than half of each limit on the number of states is
used. After that the verifier joins the states. Progs with 2^20
combinations must load.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../bpf/progs/verifier_jumps_ahead.c | 864 ++++++++++++++++++
.../selftests/bpf/progs/verifier_var_off.c | 554 +++++++++++
3 files changed, 1420 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 089d8ea3f651..78bce26662c4 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -65,6 +65,7 @@
#include "verifier_iterating_callbacks.skel.h"
#include "verifier_jeq_infer_not_null.skel.h"
#include "verifier_jit_convergence.skel.h"
+#include "verifier_jumps_ahead.skel.h"
#include "verifier_kfunc_packet_access.skel.h"
#include "verifier_kfunc_uninit.skel.h"
#include "verifier_kfunc_uninit_multi.skel.h"
@@ -258,6 +259,7 @@ void test_verifier_int_ptr(void) { RUN(verifier_int_ptr); }
void test_verifier_iterating_callbacks(void) { RUN(verifier_iterating_callbacks); }
void test_verifier_jeq_infer_not_null(void) { RUN(verifier_jeq_infer_not_null); }
void test_verifier_jit_convergence(void) { RUN(verifier_jit_convergence); }
+void test_verifier_jumps_ahead(void) { RUN(verifier_jumps_ahead); }
void test_verifier_kfunc_packet_access(void) { RUN_TESTS(verifier_kfunc_packet_access); }
void test_verifier_kfunc_uninit(void) { RUN_TESTS(verifier_kfunc_uninit); }
void test_verifier_kfunc_uninit_multi(void) { RUN_TESTS(verifier_kfunc_uninit_multi); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c b/tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c
new file mode 100644
index 000000000000..a862f46fb3de
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_jumps_ahead.c
@@ -0,0 +1,864 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+
+/*
+ * A join must not make up what neither state has where a jump ahead can see it:
+ * ret = lookup(&p);
+ * if (ret < 0)
+ * goto drop;
+ * use(p);
+ * When lookup() is inlined (ret = -ENOENT, p is not set) and (ret = 0, p is set) meet in front of
+ * the jump.
+ */
+
+SEC("socket")
+__description("jumps ahead: a register is set when ret is 0")
+__success __retval(0)
+__naked void ret_and_register(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "w6 = -2;"
+ "if r0 == 0 goto 1f;"
+ "r8 = 0;"
+ "w6 = 0;"
+ "1:"
+ "w7 = 0;"
+ "if w6 s< 0 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+ "2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+SEC("socket")
+__description("jumps ahead: a register is not set and ret doesn't tell")
+__failure __msg("R8 !read_ok")
+__naked void ret_does_not_tell(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "w6 = 1;"
+ "if r0 == 0 goto 1f;"
+ "r8 = 0;"
+ "w6 = 0;"
+ "1:"
+ "w7 = 0;"
+ "if w6 s< 0 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+ "2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/*
+ * As in bpf_host.o of cilium:
+ * err = helper();
+ * ret = err < 0 ? -155 : 0;
+ * if (err < 0)
+ * goto out;
+ * ...
+ * out:
+ * if (ret < 0)
+ * goto drop;
+ * The states differ in what two jumps look at, and in nothing else.
+ * (err < 0, ret = 0) is neither of them.
+ */
+SEC("socket")
+__description("jumps ahead: two numbers that go together")
+__success
+__naked void two_numbers(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "w6 = 0;"
+ "if w0 s> -1 goto 1f;"
+ "w6 = -155;"
+ "1:"
+ "w7 = 0;"
+ "if w0 s< 0 goto 2f;"
+ "r0 = 0;"
+ "exit;"
+ "2:"
+ "if w6 s< 0 goto 3f;"
+ "r0 = r8;"
+ "3:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/* -134 and -135 go the same way */
+SEC("socket")
+__description("jumps ahead: states that go the same way are joined")
+__success __log_level(2)
+__msg("from 4 to 7: joined")
+__msg("7: R6=scalar({{.*}}smin32=-135,smax32=-134{{.*}}) R10=fp0")
+__msg("from 6 to 7: R6=0 R8=0 R10=fp0")
+__naked void same_way(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "w6 = -134;"
+ "if r0 == 1 goto 1f;"
+ "w6 = -135;"
+ "if r0 == 2 goto 1f;"
+ "r8 = 0;"
+ "w6 = 0;"
+ "1:"
+ "w7 = 0;"
+ "if w6 s< 0 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+ "2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/*
+ * As in bpf_lxc.o of cilium. -134 and -160 go the same way at both jumps, [-160, -134] does not
+ * at the second one.
+ */
+SEC("socket")
+__description("jumps ahead: the join goes where neither state goes")
+__success __retval(0)
+__naked void two_jumps(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "w6 = -134;"
+ "if r0 == 1 goto 1f;"
+ "w6 = -160;"
+ "if r0 == 2 goto 1f;"
+ "r8 = 0;"
+ "w6 = -158;"
+ "if r0 == 3 goto 1f;"
+ "w6 = 0;"
+ "1:"
+ "w7 = 0;"
+ "if w6 s> -1 goto 3f;"
+ "if w6 != -158 goto 2f;"
+ "3:"
+ "r0 = r8;"
+ "exit;"
+ "2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/* What a helper returned is looked at twice. These are ranges, not constants. */
+SEC("socket")
+__description("jumps ahead: ranges on both sides of the jump")
+__success
+__naked void ranges(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "if w0 s< 0 goto 1f;"
+ "r8 = 0;"
+ "1:"
+ "w7 = 0;"
+ "if w0 s< 0 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+ "2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/* The jump is decided in one state only. The other one goes both ways, with r8. */
+SEC("socket")
+__description("jumps ahead: decided in one of the states")
+__success
+__naked void decided_in_one(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "w6 = 0;"
+ "if r0 == 0 goto 1f;"
+ "call %[bpf_get_prandom_u32];"
+ "w6 = w0;"
+ "r8 = 0;"
+ "1:"
+ "w7 = 0;"
+ "if w6 s> -1 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+ "2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+SEC("socket")
+__description("jumps ahead: ret is in the stack where the states meet")
+__success __retval(0)
+__naked void ret_is_spilled(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "w6 = -2;"
+ "*(u32 *)(r10 - 8) = r6;"
+ "if r0 == 0 goto 1f;"
+ "r8 = 0;"
+ "w6 = 0;"
+ "*(u32 *)(r10 - 8) = r6;"
+ "1:"
+ "w7 = 0;"
+ "r1 = *(u32 *)(r10 - 8);"
+ "w2 = w1;"
+ "if w2 s< 0 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+ "2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/* They go different ways, and that is all. One state does for both. */
+SEC("socket")
+__description("jumps ahead: states that differ in nothing else are joined")
+__success __log_level(2)
+__msg("4: R6=scalar({{.*}}smax=umax=smax32=umax32=1,{{.*}}) R10=fp0")
+__msg("processed 10 insns")
+__naked void nothing_else(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "w6 = 1;"
+ "if r0 == 0 goto 1f;"
+ "w6 = 0;"
+ "1:"
+ "w7 = 0;"
+ "if w6 == 0 goto 2f;"
+ "r0 = 1;"
+ "exit;"
+ "2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/*
+ * ret = idx < 8 ? 0 : -EINVAL;
+ * if (ret < 0)
+ * goto drop;
+ * use(buf[idx]);
+ * 0 and -EINVAL leave a gap, so the states are not joined, and idx keeps its range.
+ */
+SEC("socket")
+__description("jumps ahead: ret vouches for the range of another number")
+__success
+__naked void ret_and_range(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "call %[bpf_get_prandom_u32];"
+ "r7 = r0;"
+ "w6 = 0;"
+ "if r7 < 8 goto 1f;"
+ "w6 = -22;"
+ "1:"
+ "w8 = 0;"
+ "if w6 s< 0 goto 2f;"
+ "r1 = r10;"
+ "r1 += -8;"
+ "r1 += r7;"
+ "r0 = *(u8 *)(r1 + 0);"
+ "exit;"
+ "2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/*
+ * 16 flags, each goes together with a number, all are looked at later: 2^16 ways.
+ * There is a limit to what is kept apart.
+ */
+#define FLAG(flag, num) \
+ "call %[bpf_get_prandom_u32];" \
+ "r1 = 0;" \
+ "r2 = 0;" \
+ "if r0 == 0 goto +2;" \
+ "r1 = 1;" \
+ "r2 = 2;" \
+ "*(u64 *)(r10 - " #flag ") = r1;" \
+ "*(u64 *)(r10 - " #num ") = r2;"
+#define LOOK(flag, num) \
+ "r1 = *(u64 *)(r10 - " #flag ");" \
+ "r2 = *(u64 *)(r10 - " #num ");" \
+ "if r1 == 0 goto +1;" \
+ "r6 += r2;"
+
+SEC("socket")
+__description("jumps ahead: 16 flags")
+__success
+__naked void many_flags(void)
+{
+ asm volatile (
+ "r6 = 0;"
+ FLAG(8, 16) FLAG(24, 32) FLAG(40, 48) FLAG(56, 64)
+ FLAG(72, 80) FLAG(88, 96) FLAG(104, 112) FLAG(120, 128)
+ FLAG(136, 144) FLAG(152, 160) FLAG(168, 176) FLAG(184, 192)
+ FLAG(200, 208) FLAG(216, 224) FLAG(232, 240) FLAG(248, 256)
+ LOOK(8, 16) LOOK(24, 32) LOOK(40, 48) LOOK(56, 64)
+ LOOK(72, 80) LOOK(88, 96) LOOK(104, 112) LOOK(120, 128)
+ LOOK(136, 144) LOOK(152, 160) LOOK(168, 176) LOOK(184, 192)
+ LOOK(200, 208) LOOK(216, 224) LOOK(232, 240) LOOK(248, 256)
+ "r0 = r6;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/*
+ * -2 and 1 are not joined: [-2, 1] has 0, which gets through both jumps, and r8 is set
+ * only in the state that really has 0.
+ */
+SEC("socket")
+__description("jumps ahead: less, greater, or use it")
+__success
+__naked void three_ways(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "r6 = -2;"
+ "if r0 == 1 goto 1f;"
+ "r6 = 1;"
+ "if r0 == 2 goto 1f;"
+ "r8 = 0;"
+ "r6 = 0;"
+"1:"
+ "if r6 s< 0 goto 2f;"
+ "if r6 s> 0 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+SEC("socket")
+__description("jumps ahead: not 0, or use it")
+__success
+__naked void not_zero(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "r6 = -2;"
+ "if r0 == 1 goto 1f;"
+ "r6 = 1;"
+ "if r0 == 2 goto 1f;"
+ "r8 = 0;"
+ "r6 = 0;"
+"1:"
+ "if r6 != 0 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/* ret = -1 and ret = 0 touch. What -1 says about idx is lost. Test idx where it is used. */
+SEC("socket")
+__description("jumps ahead: numbers that touch are joined")
+__failure __msg("math between fp pointer and register with unbounded min value is not allowed")
+__naked void ret_and_range_touch(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "call %[bpf_get_prandom_u32];"
+ "r7 = r0;"
+ "r6 = 0;"
+ "if r7 < 8 goto 1f;"
+ "r6 = -1;"
+ "1:"
+ "w8 = 0;"
+ "if r6 s< 0 goto 2f;"
+ "r1 = r10;"
+ "r1 += -8;"
+ "r1 += r7;"
+ "r0 = *(u8 *)(r1 + 0);"
+ "exit;"
+ "2:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all
+ );
+}
+
+/*
+ * A register that is set in one state and not in the other: these states are not joined,
+ * whatever it is that tells them apart later.
+ */
+#define ADD8 "r0 += 1; r0 += 1; r0 += 1; r0 += 1; r0 += 1; r0 += 1; r0 += 1; r0 += 1;"
+#define ADD64 ADD8 ADD8 ADD8 ADD8 ADD8 ADD8 ADD8 ADD8
+
+SEC("socket")
+__description("set or not: a register is set when a bit of flags is, the bit is tested in a copy")
+__success
+__naked void flag_and(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "r6 = 0;"
+ "if r0 == 0 goto 1f;"
+ "r8 = 0;"
+ "r6 = 1;"
+"1:"
+ "r1 = r6;"
+ "r1 &= 1;"
+ "if r1 == 0 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+SEC("socket")
+__description("set or not: a register is set when ret is 0, ret + 1 is tested")
+__success
+__naked void flag_add(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "r6 = 0;"
+ "if r0 == 0 goto 1f;"
+ "r8 = 0;"
+ "r6 = 1;"
+"1:"
+ "r1 = r6;"
+ "r1 += 4;"
+ "if r1 != 5 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+SEC("socket")
+__description("set or not: a register is set when ret is 0, the jump is 264 insns away")
+__success
+__naked void jump_is_far(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "w6 = -2;"
+ "if r0 == 0 goto 1f;"
+ "r8 = 5;"
+ "w6 = 0;"
+"1:"
+ "r0 = 0;"
+ ADD64 ADD64 ADD64 ADD64 ADD8
+ "if w6 s< 0 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+SEC("socket")
+__description("set or not: a register is set when ret is 0, ret is spilled with 4 and with 8 bytes")
+__success
+__naked void spill32_move(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "w1 = -2;"
+ "*(u32 *)(r10 - 8) = r1;"
+ "if r0 == 0 goto 1f;"
+ "r8 = 0;"
+ "w1 = 0;"
+ "*(u32 *)(r10 - 8) = r1;"
+"1:"
+ "r1 = *(u32 *)(r10 - 8);"
+ "*(u64 *)(r10 - 16) = r1;"
+ "r2 = *(u64 *)(r10 - 16);"
+ "if w2 s< 0 goto 2f;"
+ "r0 = r8;"
+ "exit;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(max_entries, 4);
+ __type(key, int);
+ __type(value, char[64]);
+} h SEC(".maps");
+
+/*
+ * p = NULL;
+ * ret = find(key, &p); -ENOENT, 1: nothing to do, 0: p is the entry
+ * if (ret < 0)
+ * goto err;
+ * if (ret > 0)
+ * goto skip;
+ * return p->x;
+ */
+SEC("socket")
+__description("jumps ahead: less, greater, or load through the pointer")
+__success
+__naked void ptr_three_ways(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "r8 = 0;"
+ "r6 = 1;"
+ "if r0 == 0 goto 1f;"
+ "*(u32 *)(r10 - 4) = r0;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "r1 = %[h] ll;"
+ "call %[bpf_map_lookup_elem];"
+ "r6 = -2;"
+ "if r0 == 0 goto 1f;"
+ "r8 = r0;"
+ "r6 = 0;"
+"1:"
+ "if r6 s< 0 goto 2f;"
+ "if r6 s> 0 goto 2f;"
+ "r0 = *(u32 *)(r8 + 0);"
+ "exit;"
+"2:"
+ "r0 = 0;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32), __imm(bpf_map_lookup_elem), __imm_addr(h) : __clobber_all);
+}
+
+/*
+ * ret = 0;
+ * state = lookup(&h, &key);
+ * if (!state) {
+ * err = update(&h, &key, &new, 0);
+ * ret = err < 0 ? DROP : 0;
+ * ...
+ * if (err < 0)
+ * goto out;
+ * state = lookup(&h, &key);
+ * if (!state)
+ * return 0;
+ * }
+ * out:
+ * if (ret < 0)
+ * goto drop;
+ * return state->x;
+ */
+SEC("socket")
+__description("jumps ahead: err and ret, load through the pointer")
+__success
+__naked void ptr_err_and_ret(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u32 *)(r10 - 4) = r1;"
+ "*(u64 *)(r10 - 72) = r1;"
+ "*(u64 *)(r10 - 64) = r1;"
+ "*(u64 *)(r10 - 56) = r1;"
+ "*(u64 *)(r10 - 48) = r1;"
+ "*(u64 *)(r10 - 40) = r1;"
+ "*(u64 *)(r10 - 32) = r1;"
+ "*(u64 *)(r10 - 24) = r1;"
+ "*(u64 *)(r10 - 16) = r1;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "r1 = %[h] ll;"
+ "call %[bpf_map_lookup_elem];"
+ "r8 = r0;"
+ "w9 = 0;"
+ "if r8 != 0 goto 3f;"
+ "r1 = %[h] ll;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "r3 = r10;"
+ "r3 += -72;"
+ "r4 = 0;"
+ "call %[bpf_map_update_elem];"
+ "w9 = 0;"
+ "if w0 s> -1 goto 1f;"
+ "w9 = -155;"
+"1:"
+ "w6 = 0;"
+ "if w0 s< 0 goto 3f;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "r1 = %[h] ll;"
+ "call %[bpf_map_lookup_elem];"
+ "r8 = r0;"
+ "if r8 == 0 goto 4f;"
+"3:"
+ "if w9 s< 0 goto 4f;"
+ "r0 = *(u32 *)(r8 + 0);"
+ "exit;"
+"4:"
+ "r0 = 0;"
+ "exit;"
+ :: __imm(bpf_map_lookup_elem), __imm(bpf_map_update_elem), __imm_addr(h) : __clobber_all);
+}
+
+#define PAIR(name, desc, is_err, drop, is_drop) \
+SEC("socket") \
+__description(desc) \
+__success \
+__naked void name(void) \
+{ \
+ asm volatile ( \
+ "r1 = 0;" \
+ "*(u32 *)(r10 - 4) = r1;" \
+ "*(u64 *)(r10 - 72) = r1;" \
+ "*(u64 *)(r10 - 64) = r1;" \
+ "*(u64 *)(r10 - 56) = r1;" \
+ "*(u64 *)(r10 - 48) = r1;" \
+ "*(u64 *)(r10 - 40) = r1;" \
+ "*(u64 *)(r10 - 32) = r1;" \
+ "*(u64 *)(r10 - 24) = r1;" \
+ "*(u64 *)(r10 - 16) = r1;" \
+ "r2 = r10;" \
+ "r2 += -4;" \
+ "r1 = %[h] ll;" \
+ "call %[bpf_map_lookup_elem];" \
+ "r8 = r0;" \
+ "w9 = 0;" \
+ "if r8 != 0 goto 3f;" \
+ "r1 = %[h] ll;" \
+ "r2 = r10;" \
+ "r2 += -4;" \
+ "r3 = r10;" \
+ "r3 += -72;" \
+ "r4 = 0;" \
+ "call %[bpf_map_update_elem];" \
+ is_err \
+ "w9 = 0;" \
+ "if w7 == 0 goto 1f;" \
+ "w9 = " drop ";" \
+"1:" \
+ "w6 = 0;" \
+ "if w7 != 0 goto 3f;" \
+ "r2 = r10;" \
+ "r2 += -4;" \
+ "r1 = %[h] ll;" \
+ "call %[bpf_map_lookup_elem];" \
+ "r8 = r0;" \
+ "if r8 == 0 goto 4f;" \
+"3:" \
+ "if w9 " is_drop " goto 4f;" \
+ "r0 = *(u32 *)(r8 + 0);" \
+ "exit;" \
+"4:" \
+ "r0 = 0;" \
+ "exit;" \
+ :: __imm(bpf_map_lookup_elem), __imm(bpf_map_update_elem), \
+ __imm_addr(h) : __clobber_all); \
+}
+
+/* 0 and 1 touch, in both numbers. The join would have err = 1 with ret = 0. */
+PAIR(ptr_two_flags, "jumps ahead: two flags, load through the pointer",
+ "w7 = 1; if w0 s< 0 goto +1; w7 = 0;", "1", "!= 0")
+
+/*
+ * 20 flags, 0 or 5, all are looked at later: 2^20 ways, and 0 and 5 leave a gap.
+ * States are kept apart while there is room. After that they are joined.
+ */
+#define PICK_FLAG(slot) \
+ "call %[bpf_get_prandom_u32];" \
+ "r1 = 0;" \
+ "if r0 == 0 goto +1;" \
+ "r1 = 5;" \
+ "*(u64*)(r10 - " #slot ") = r1;"
+#define USE_FLAG(slot) \
+ "r1 = *(u64*)(r10 - " #slot ");" \
+ "if r1 == 0 goto +1;" \
+ "r6 += 1;"
+
+SEC("socket")
+__description("jumps ahead: 20 flags that leave a gap")
+__success
+__naked void many_flags_gap(void)
+{
+ asm volatile (
+ "r6 = 0;"
+ PICK_FLAG(8) PICK_FLAG(16) PICK_FLAG(24) PICK_FLAG(32) PICK_FLAG(40) PICK_FLAG(48)
+ PICK_FLAG(56) PICK_FLAG(64) PICK_FLAG(72) PICK_FLAG(80) PICK_FLAG(88) PICK_FLAG(96)
+ PICK_FLAG(104) PICK_FLAG(112) PICK_FLAG(120) PICK_FLAG(128) PICK_FLAG(136) PICK_FLAG(144)
+ PICK_FLAG(152) PICK_FLAG(160)
+ USE_FLAG(8) USE_FLAG(16) USE_FLAG(24) USE_FLAG(32) USE_FLAG(40) USE_FLAG(48)
+ USE_FLAG(56) USE_FLAG(64) USE_FLAG(72) USE_FLAG(80) USE_FLAG(88) USE_FLAG(96)
+ USE_FLAG(104) USE_FLAG(112) USE_FLAG(120) USE_FLAG(128) USE_FLAG(136) USE_FLAG(144)
+ USE_FLAG(152) USE_FLAG(160)
+ "r0 = r6;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/* 33 numbers, each with an exit of its own */
+static __naked __noinline __used int ret_exits(void)
+{
+ asm volatile (
+ "call %[bpf_get_prandom_u32];"
+ "r1 = r0;"
+ "if r1 != 0 goto +2; r0 = 0; exit;"
+ "if r1 != 1 goto +2; r0 = 3; exit;"
+ "if r1 != 2 goto +2; r0 = 6; exit;"
+ "if r1 != 3 goto +2; r0 = 9; exit;"
+ "if r1 != 4 goto +2; r0 = 12; exit;"
+ "if r1 != 5 goto +2; r0 = 15; exit;"
+ "if r1 != 6 goto +2; r0 = 18; exit;"
+ "if r1 != 7 goto +2; r0 = 21; exit;"
+ "if r1 != 8 goto +2; r0 = 24; exit;"
+ "if r1 != 9 goto +2; r0 = 27; exit;"
+ "if r1 != 10 goto +2; r0 = 30; exit;"
+ "if r1 != 11 goto +2; r0 = 33; exit;"
+ "if r1 != 12 goto +2; r0 = 36; exit;"
+ "if r1 != 13 goto +2; r0 = 39; exit;"
+ "if r1 != 14 goto +2; r0 = 42; exit;"
+ "if r1 != 15 goto +2; r0 = 45; exit;"
+ "if r1 != 16 goto +2; r0 = 48; exit;"
+ "if r1 != 17 goto +2; r0 = 51; exit;"
+ "if r1 != 18 goto +2; r0 = 54; exit;"
+ "if r1 != 19 goto +2; r0 = 57; exit;"
+ "if r1 != 20 goto +2; r0 = 60; exit;"
+ "if r1 != 21 goto +2; r0 = 63; exit;"
+ "if r1 != 22 goto +2; r0 = 66; exit;"
+ "if r1 != 23 goto +2; r0 = 69; exit;"
+ "if r1 != 24 goto +2; r0 = 72; exit;"
+ "if r1 != 25 goto +2; r0 = 75; exit;"
+ "if r1 != 26 goto +2; r0 = 78; exit;"
+ "if r1 != 27 goto +2; r0 = 81; exit;"
+ "if r1 != 28 goto +2; r0 = 84; exit;"
+ "if r1 != 29 goto +2; r0 = 87; exit;"
+ "if r1 != 30 goto +2; r0 = 90; exit;"
+ "if r1 != 31 goto +2; r0 = 93; exit;"
+ "r0 = 99;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+/* 25 is not one of them, and r8 is not set */
+SEC("socket")
+__description("jumps ahead: one of 33 numbers out of a function")
+__success
+__naked void rets_33(void)
+{
+ asm volatile (
+ "call ret_exits;"
+ "if r0 != 25 goto 1f;"
+ "r0 = r8;"
+ "exit;"
+"1:"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* 33^4 ways: not all of them are looked at */
+SEC("socket")
+__description("jumps ahead: one of 33 numbers out of a function, 4 times")
+__success
+__naked void rets_33_4_times(void)
+{
+ asm volatile (
+ "call ret_exits; r6 = r0;"
+ "call ret_exits; r7 = r0;"
+ "call ret_exits; r8 = r0;"
+ "call ret_exits; r9 = r0;"
+ "r0 = 0;"
+ "if r6 != 3 goto +1; r0 += 1;"
+ "if r7 != 6 goto +1; r0 += 1;"
+ "if r8 != 9 goto +1; r0 += 1;"
+ "if r9 != 12 goto +1; r0 += 1;"
+ "exit;"
+ ::: __clobber_all);
+}
+
+/* 16 numbers, 0 or not, all are looked at later: 2^16 ways */
+#define PICK_NULL(slot) \
+ "call %[bpf_get_prandom_u32];" \
+ "r0 <<= 32;" \
+ "r7 = r0;" \
+ "call %[bpf_get_prandom_u32];" \
+ "r0 |= r7;" \
+ "if r0 == 0 goto +0;" \
+ "*(u64*)(r10 - " #slot ") = r0;"
+#define USE_NULL(slot) \
+ "r1 = *(u64*)(r10 - " #slot ");" \
+ "if r1 == 0 goto +1;" \
+ "r6 += 1;"
+
+SEC("socket")
+__description("jumps ahead: 16 times 0 or not")
+__success
+__naked void many_nulls(void)
+{
+ asm volatile (
+ "r6 = 0;"
+ PICK_NULL(8) PICK_NULL(16) PICK_NULL(24) PICK_NULL(32) PICK_NULL(40) PICK_NULL(48)
+ PICK_NULL(56) PICK_NULL(64) PICK_NULL(72) PICK_NULL(80) PICK_NULL(88) PICK_NULL(96)
+ PICK_NULL(104) PICK_NULL(112) PICK_NULL(120) PICK_NULL(128)
+ USE_NULL(8) USE_NULL(16) USE_NULL(24) USE_NULL(32) USE_NULL(40) USE_NULL(48)
+ USE_NULL(56) USE_NULL(64) USE_NULL(72) USE_NULL(80) USE_NULL(88) USE_NULL(96)
+ USE_NULL(104) USE_NULL(112) USE_NULL(120) USE_NULL(128)
+ "r0 = r6;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c
index d92215c5b923..1891995df973 100644
--- a/tools/testing/selftests/bpf/progs/verifier_var_off.c
+++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c
@@ -12,6 +12,13 @@ struct {
__type(value, long long);
} map_hash_8b SEC(".maps");
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, long long);
+} map_array_8b SEC(".maps");
+
SEC("lwt_in")
__description("variable-offset ctx access")
__failure __msg("variable ctx access var_off=(0x0; 0x4)")
@@ -556,4 +563,551 @@ __naked void variable_offset_stack_access_ok(void)
: __clobber_all);
}
+/*
+ * if (x) a[0] = p; else a[1] = p; with one store. The paths meet in front of it.
+ * The offset is known on each of them.
+ */
+SEC("tc")
+__description("pointer is spilled at one of two offsets")
+__success __retval(0)
+__naked void ptr_at_one_of_two_offsets(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u32*)(r10 - 4) = r1; \
+ r2 = r10; \
+ r2 += -4; \
+ r1 = %[map_array_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l2_%=; \
+ r6 = r0; \
+ r1 = 0; \
+ *(u64*)(r10 - 16) = r1; \
+ *(u64*)(r10 - 24) = r1; \
+ call %[bpf_get_prandom_u32]; \
+ r7 = 0; \
+ if r0 > 100 goto l0_%=; \
+ r7 = 8; \
+l0_%=: r2 = r10; \
+ r2 += -24; \
+ r2 += r7; \
+ *(u64*)(r2 + 0) = r6; \
+ if r7 != 0 goto l1_%=; \
+ r3 = *(u64*)(r10 - 24); \
+ r0 = *(u64*)(r3 + 0); \
+ exit; \
+l1_%=: r3 = *(u64*)(r10 - 16); \
+ r0 = *(u64*)(r3 + 0); \
+ exit; \
+l2_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32),
+ __imm(bpf_map_lookup_elem),
+ __imm_addr(map_array_8b)
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("pointer is spilled at one of two offsets, the offset is spilled too")
+__success __retval(0)
+__naked void ptr_at_one_of_two_spilled_offsets(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u32*)(r10 - 4) = r1; \
+ r2 = r10; \
+ r2 += -4; \
+ r1 = %[map_array_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l2_%=; \
+ r6 = r0; \
+ r1 = 0; \
+ *(u64*)(r10 - 16) = r1; \
+ *(u64*)(r10 - 24) = r1; \
+ *(u64*)(r10 - 32) = r1; \
+ call %[bpf_get_prandom_u32]; \
+ if r0 > 100 goto l0_%=; \
+ r1 = 8; \
+ *(u64*)(r10 - 32) = r1; \
+l0_%=: r7 = *(u64*)(r10 - 32); \
+ r2 = r10; \
+ r2 += -24; \
+ r2 += r7; \
+ *(u64*)(r2 + 0) = r6; \
+ if r7 != 0 goto l1_%=; \
+ r3 = *(u64*)(r10 - 24); \
+ r0 = *(u64*)(r3 + 0); \
+ exit; \
+l1_%=: r3 = *(u64*)(r10 - 16); \
+ r0 = *(u64*)(r3 + 0); \
+ exit; \
+l2_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32),
+ __imm(bpf_map_lookup_elem),
+ __imm_addr(map_array_8b)
+ : __clobber_all);
+}
+
+static __naked __noinline __used void spill_r3_at_r1_plus_r2(void)
+{
+ asm volatile (" \
+ r1 += r2; \
+ *(u64*)(r1 + 0) = r3; \
+ r0 = 0; \
+ exit; \
+" ::: __clobber_all);
+}
+
+SEC("tc")
+__description("pointer is spilled at one of two offsets by a subprog")
+__success __retval(0)
+__naked void ptr_at_one_of_two_offsets_subprog(void)
+{
+ asm volatile (" \
+ r1 = 0; \
+ *(u32*)(r10 - 4) = r1; \
+ r2 = r10; \
+ r2 += -4; \
+ r1 = %[map_array_8b] ll; \
+ call %[bpf_map_lookup_elem]; \
+ if r0 == 0 goto l2_%=; \
+ r6 = r0; \
+ r1 = 0; \
+ *(u64*)(r10 - 16) = r1; \
+ *(u64*)(r10 - 24) = r1; \
+ call %[bpf_get_prandom_u32]; \
+ r7 = 0; \
+ if r0 > 100 goto l0_%=; \
+ r7 = 8; \
+l0_%=: r1 = r10; \
+ r1 += -24; \
+ r2 = r7; \
+ r3 = r6; \
+ call spill_r3_at_r1_plus_r2; \
+ if r7 != 0 goto l1_%=; \
+ r3 = *(u64*)(r10 - 24); \
+ r0 = *(u64*)(r3 + 0); \
+ exit; \
+l1_%=: r3 = *(u64*)(r10 - 16); \
+ r0 = *(u64*)(r3 + 0); \
+ exit; \
+l2_%=: r0 = 0; \
+ exit; \
+" :
+ : __imm(bpf_get_prandom_u32),
+ __imm(bpf_map_lookup_elem),
+ __imm_addr(map_array_8b)
+ : __clobber_all);
+}
+
+#define PICK_OFF(slot) \
+ "call %[bpf_get_prandom_u32];" \
+ "r1 = 0;" \
+ "if r0 > 100 goto +1;" \
+ "r1 = 8;" \
+ "*(u64*)(r10 - " #slot ") = r1;"
+
+#define USE_OFF(slot) \
+ "r1 = *(u64*)(r10 - " #slot ");" \
+ "r2 = r10;" \
+ "r2 += -16;" \
+ "r2 += r1;" \
+ "*(u64*)(r2 + 0) = r0;"
+
+/* 4096 combinations of offsets that are known: not all of them are looked at */
+SEC("tc")
+__description("12 times one of two offsets")
+__success __retval(0)
+__naked void many_offsets(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64*)(r10 - 8) = r1;"
+ "*(u64*)(r10 - 16) = r1;"
+ PICK_OFF(24) PICK_OFF(32) PICK_OFF(40) PICK_OFF(48)
+ PICK_OFF(56) PICK_OFF(64) PICK_OFF(72) PICK_OFF(80)
+ PICK_OFF(88) PICK_OFF(96) PICK_OFF(104) PICK_OFF(112)
+ USE_OFF(24) USE_OFF(32) USE_OFF(40) USE_OFF(48)
+ USE_OFF(56) USE_OFF(64) USE_OFF(72) USE_OFF(80)
+ USE_OFF(88) USE_OFF(96) USE_OFF(104) USE_OFF(112)
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+extern int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end) __ksym;
+extern int *bpf_iter_num_next(struct bpf_iter_num *it) __ksym;
+extern void bpf_iter_num_destroy(struct bpf_iter_num *it) __ksym;
+
+/* The verifier doesn't know what is in there */
+unsigned long zero;
+
+#define ITER_NEW "r1 = r10; r1 += -24; w2 = 0; w3 = 100; call %[bpf_iter_num_new];"
+#define ITER_NEXT "r1 = r10; r1 += -24; call %[bpf_iter_num_next];"
+#define ITER_DESTROY "r1 = r10; r1 += -24; call %[bpf_iter_num_destroy];"
+#define __imm_iter __imm(bpf_iter_num_new), __imm(bpf_iter_num_next), \
+ __imm(bpf_iter_num_destroy)
+
+/*
+ * u8 stack[10], depth = 0;
+ * loop {
+ * if (x) { if (depth >= 10) break; stack[depth++] = 1; }
+ * else { depth--; if (depth >= 10) break; y = stack[depth]; }
+ * }
+ * depth is in a 32-bit register. 'depth & 255' is tested, and computed anew for the index.
+ */
+#define PUSH_OR_POP \
+ "r1 = *(u64 *)(r6 + 0);" \
+ "if r1 == 0 goto 2f;" \
+ "w7 = w8;" \
+ "w7 &= 255;" \
+ "if w7 > 9 goto 9f;" \
+ "r1 = r8;" \
+ "r1 &= 255;" \
+ "r2 = r10;" \
+ "r2 += -10;" \
+ "r2 += r1;" \
+ "*(u8 *)(r2 + 0) = r1;" \
+ "w8 += 1;" \
+ "goto 1b;" \
+ "2:" \
+ "w8 += -1;" \
+ "w2 = w8;" \
+ "w2 &= 255;" \
+ "if w2 > 9 goto 9f;" \
+ "r2 = r8;" \
+ "r2 &= 255;" \
+ "r3 = r10;" \
+ "r3 += -10;" \
+ "r3 += r2;" \
+ "r9 = *(u8 *)(r3 + 0);" \
+ "goto 1b;"
+
+SEC("tc")
+__description("depth of a stack on the stack: a copy is tested")
+__success __retval(0)
+__naked void depth_copy_is_tested(void)
+{
+ asm volatile (
+ "r6 = %[zero] ll;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "*(u64 *)(r10 - 16) = r1;"
+ "w8 = 0;"
+ ITER_NEW
+ "1:"
+ ITER_NEXT
+ "if r0 == 0 goto 9f;"
+ PUSH_OR_POP
+ "9:"
+ ITER_DESTROY
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(zero), __imm_iter
+ : __clobber_all);
+}
+
+/* Nothing of the prog stops the loop */
+SEC("tc")
+__description("depth of a stack on the stack: a copy is tested, no iterator")
+__success __retval(0)
+__naked void depth_copy_is_tested_no_iter(void)
+{
+ asm volatile (
+ "r6 = %[zero] ll;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "*(u64 *)(r10 - 16) = r1;"
+ "w8 = 0;"
+ "1:"
+ PUSH_OR_POP
+ "9:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(zero)
+ : __clobber_all);
+}
+
+/*
+ * u32 stack[16]. The copy that is tested is volatile: it's stored once and read twice.
+ * depth takes 18 values: 0..16 and 255.
+ */
+SEC("tc")
+__description("depth of a stack on the stack: a volatile copy is tested")
+__success __retval(0)
+__naked void depth_volatile_copy_is_tested(void)
+{
+ asm volatile (
+ "r6 = %[zero] ll;"
+ "w8 = 0;"
+ ITER_NEW
+ "1:"
+ ITER_NEXT
+ "if r0 == 0 goto 9f;"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 == 0 goto 2f;"
+ "w2 = w8;"
+ "w2 &= 255;"
+ "if w2 > 15 goto 9f;"
+ "*(u8 *)(r10 - 32) = r8;"
+ "r2 = *(u8 *)(r10 - 32);"
+ "if w2 > 15 goto 9f;"
+ "r2 = *(u8 *)(r10 - 32);"
+ "r2 <<= 2;"
+ "r3 = r10;"
+ "r3 += -96;"
+ "r3 += r2;"
+ "*(u32 *)(r3 + 0) = r1;"
+ "w8 += 1;"
+ "goto 1b;"
+ "2:"
+ "*(u8 *)(r10 - 32) = r8;"
+ "r1 = *(u8 *)(r10 - 32);"
+ "r1 += -1;"
+ "*(u8 *)(r10 - 32) = r1;"
+ "r1 = *(u8 *)(r10 - 32);"
+ "if r1 > 15 goto 9f;"
+ "r1 = *(u8 *)(r10 - 32);"
+ "r1 <<= 2;"
+ "r3 = r10;"
+ "r3 += -96;"
+ "r3 += r1;"
+ "r7 = *(u32 *)(r3 + 0);"
+ "r8 = *(u8 *)(r10 - 32);"
+ "goto 1b;"
+ "9:"
+ ITER_DESTROY
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(zero), __imm_iter
+ : __clobber_all);
+}
+
+SEC("tc")
+__description("depth of a stack on the stack: no bound")
+__failure __msg("invalid variable-offset write to stack R2")
+__naked void depth_without_a_bound(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "*(u64 *)(r10 - 16) = r1;"
+ "w8 = 0;"
+ ITER_NEW
+ "1:"
+ ITER_NEXT
+ "if r0 == 0 goto 9f;"
+ "r1 = r8;"
+ "r1 &= 255;"
+ "r2 = r10;"
+ "r2 += -10;"
+ "r2 += r1;"
+ "*(u8 *)(r2 + 0) = r1;"
+ "w8 += 1;"
+ "goto 1b;"
+ "9:"
+ ITER_DESTROY
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_iter
+ : __clobber_all);
+}
+
+/*
+ * The counters never come to rest, and go up by 1 or by 2, so that there is no summary of them.
+ * There are so many constants, then it's a range.
+ */
+#define UP(r) "call %[bpf_get_prandom_u32]; " #r " += 1; if r0 > 100 goto +1; " #r " += 1;"
+
+SEC("tc")
+__description("counters without a bound are masked for an offset in the stack")
+__success __retval(0)
+__naked void masked_counters(void)
+{
+ asm volatile (
+ "r6 = %[zero] ll;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "*(u64 *)(r10 - 16) = r1;"
+ "w7 = 0;"
+ "w8 = 0;"
+ "w9 = 0;"
+ "1:"
+ UP(w7)
+ "2:"
+ UP(w8)
+ "3:"
+ UP(w9)
+ "r1 = r7;"
+ "r1 += r8;"
+ "r1 += r9;"
+ "r1 &= 15;"
+ "r2 = r10;"
+ "r2 += -16;"
+ "r2 += r1;"
+ "*(u8 *)(r2 + 0) = r1;"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 == 1 goto 3b;"
+ "if r1 == 2 goto 2b;"
+ "if r1 == 3 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(zero), __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/* BTF FUNC records are not generated for kfuncs referenced from inline assembly */
+void __kfunc_btf_root(void)
+{
+ bpf_iter_num_new(0, 0, 0);
+ bpf_iter_num_next(0);
+ bpf_iter_num_destroy(0);
+}
+
+/* u8 stack[40] and u8 stack[200]: depth takes 42 and 202 values */
+SEC("tc")
+__description("depth of a stack on the stack: 40 deep")
+__success __retval(0)
+__naked void depth_40(void)
+{
+ asm volatile (
+ "r6 = %[zero] ll;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "*(u64 *)(r10 - 16) = r1;"
+ "*(u64 *)(r10 - 24) = r1;"
+ "*(u64 *)(r10 - 32) = r1;"
+ "*(u64 *)(r10 - 40) = r1;"
+ "w8 = 0;"
+ "1:"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 == 0 goto 2f;"
+ "w7 = w8;"
+ "w7 &= 255;"
+ "if w7 > 39 goto 9f;"
+ "r1 = r8;"
+ "r1 &= 255;"
+ "r2 = r10;"
+ "r2 += -40;"
+ "r2 += r1;"
+ "*(u8 *)(r2 + 0) = r1;"
+ "w8 += 1;"
+ "goto 1b;"
+ "2:"
+ "w8 += -1;"
+ "w2 = w8;"
+ "w2 &= 255;"
+ "if w2 > 39 goto 9f;"
+ "r2 = r8;"
+ "r2 &= 255;"
+ "r3 = r10;"
+ "r3 += -40;"
+ "r3 += r2;"
+ "r9 = *(u8 *)(r3 + 0);"
+ "goto 1b;"
+ "9:"
+ "r0 = 0;"
+ "exit;"
+ :: __imm_addr(zero) : __clobber_all);
+}
+
+SEC("tc")
+__description("depth of a stack on the stack: 200 deep")
+__success __retval(0)
+__naked void depth_200(void)
+{
+ asm volatile (
+ "r6 = %[zero] ll;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "*(u64 *)(r10 - 16) = r1;"
+ "*(u64 *)(r10 - 24) = r1;"
+ "*(u64 *)(r10 - 32) = r1;"
+ "*(u64 *)(r10 - 40) = r1;"
+ "*(u64 *)(r10 - 48) = r1;"
+ "*(u64 *)(r10 - 56) = r1;"
+ "*(u64 *)(r10 - 64) = r1;"
+ "*(u64 *)(r10 - 72) = r1;"
+ "*(u64 *)(r10 - 80) = r1;"
+ "*(u64 *)(r10 - 88) = r1;"
+ "*(u64 *)(r10 - 96) = r1;"
+ "*(u64 *)(r10 - 104) = r1;"
+ "*(u64 *)(r10 - 112) = r1;"
+ "*(u64 *)(r10 - 120) = r1;"
+ "*(u64 *)(r10 - 128) = r1;"
+ "*(u64 *)(r10 - 136) = r1;"
+ "*(u64 *)(r10 - 144) = r1;"
+ "*(u64 *)(r10 - 152) = r1;"
+ "*(u64 *)(r10 - 160) = r1;"
+ "*(u64 *)(r10 - 168) = r1;"
+ "*(u64 *)(r10 - 176) = r1;"
+ "*(u64 *)(r10 - 184) = r1;"
+ "*(u64 *)(r10 - 192) = r1;"
+ "*(u64 *)(r10 - 200) = r1;"
+ "w8 = 0;"
+ "1:"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 == 0 goto 2f;"
+ "w7 = w8;"
+ "w7 &= 255;"
+ "if w7 > 199 goto 9f;"
+ "r1 = r8;"
+ "r1 &= 255;"
+ "r2 = r10;"
+ "r2 += -200;"
+ "r2 += r1;"
+ "*(u8 *)(r2 + 0) = r1;"
+ "w8 += 1;"
+ "goto 1b;"
+ "2:"
+ "w8 += -1;"
+ "w2 = w8;"
+ "w2 &= 255;"
+ "if w2 > 199 goto 9f;"
+ "r2 = r8;"
+ "r2 &= 255;"
+ "r3 = r10;"
+ "r3 += -200;"
+ "r3 += r2;"
+ "r9 = *(u8 *)(r3 + 0);"
+ "goto 1b;"
+ "9:"
+ "r0 = 0;"
+ "exit;"
+ :: __imm_addr(zero) : __clobber_all);
+}
+
+/* 2^20 combinations */
+SEC("tc")
+__description("20 times one of two offsets")
+__success __retval(0)
+__naked void many_more_offsets(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u64*)(r10 - 8) = r1;"
+ "*(u64*)(r10 - 16) = r1;"
+ PICK_OFF(24) PICK_OFF(32) PICK_OFF(40) PICK_OFF(48) PICK_OFF(56) PICK_OFF(64) PICK_OFF(72)
+ PICK_OFF(80) PICK_OFF(88) PICK_OFF(96) PICK_OFF(104) PICK_OFF(112) PICK_OFF(120)
+ PICK_OFF(128) PICK_OFF(136) PICK_OFF(144) PICK_OFF(152) PICK_OFF(160) PICK_OFF(168)
+ PICK_OFF(176)
+ USE_OFF(24) USE_OFF(32) USE_OFF(40) USE_OFF(48) USE_OFF(56) USE_OFF(64) USE_OFF(72)
+ USE_OFF(80) USE_OFF(88) USE_OFF(96) USE_OFF(104) USE_OFF(112) USE_OFF(120)
+ USE_OFF(128) USE_OFF(136) USE_OFF(144) USE_OFF(152) USE_OFF(160) USE_OFF(168)
+ USE_OFF(176)
+ "r0 = 0;"
+ "exit;"
+ :: __imm(bpf_get_prandom_u32) : __clobber_all);
+}
+
char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 77/81] selftests/bpf: Test states that are not joined
2026-10-26 2:59 ` [PATCH bpf 77/81] selftests/bpf: Test states that are not joined Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1829 bytes --]
> selftests/bpf: Test states that are not joined
TL;DR: Nit only, in the selftests: depth_40() and depth_200() repeat the
PUSH_OR_POP asm sequence with different constants; the macro could take the
stack size as an argument.
> diff --git a/tools/testing/selftests/bpf/progs/verifier_var_off.c b/tools/testing/selftests/bpf/progs/verifier_var_off.c
> index d92215c5b923..1891995df973 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_var_off.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_var_off.c
[ ... ]
> +#define PUSH_OR_POP \
> + "r1 = *(u64 *)(r6 + 0);" \
> + "if r1 == 0 goto 2f;" \
> + "w7 = w8;" \
> + "w7 &= 255;" \
> + "if w7 > 9 goto 9f;" \
> + "r1 = r8;" \
> + "r1 &= 255;" \
> + "r2 = r10;" \
> + "r2 += -10;" \
> + "r2 += r1;" \
> + "*(u8 *)(r2 + 0) = r1;" \
[ ... ]
> +__naked void depth_40(void)
> +{
> + asm volatile (
> + "r6 = %[zero] ll;"
[ ... ]
> + "r1 = *(u64 *)(r6 + 0);"
> + "if r1 == 0 goto 2f;"
> + "w7 = w8;"
> + "w7 &= 255;"
> + "if w7 > 39 goto 9f;"
> + "r1 = r8;"
> + "r1 &= 255;"
> + "r2 = r10;"
> + "r2 += -40;"
> + "r2 += r1;"
> + "*(u8 *)(r2 + 0) = r1;"
> + "w8 += 1;"
> + "goto 1b;"
[ ... ]
This isn't a bug, but depth_40() and depth_200() repeat the PUSH_OR_POP
sequence above instruction for instruction, changing only the bound (9, 39,
199) and the base offset (-10, -40, -200). Could PUSH_OR_POP take the stack
size as an argument, the way PICK_OFF(slot) does, so that depth_40() and
depth_200() reuse it instead of repeating the sequence with different
constants?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 78/81] selftests/bpf: Test progs that run out of budget
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (76 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 77/81] selftests/bpf: Test states that are not joined Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 2:59 ` [PATCH bpf 79/81] selftests/bpf: Test 'unwound' in bpf_prog_info Alexei Starovoitov
` (4 subsequent siblings)
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The verifier does not prove that a prog ends. It adds may_goto to each loop
that it cannot bound, and a check to the entry of each recursive subprog.
Thus a run has a budget of time, stack and calls. A mistake in the added
code does not reject a prog. It locks up a cpu.
Load and run progs with such loops and such recursion. Many of them never
end without the added code. Each run must return, and the return value
shows where the prog continued.
The added may_goto ends the prog. When the prog holds a lock, it exits the
loop instead. That is unsafe when the code after the loop depends on the
condition of the loop, for example p != NULL. A subprog can also unwind:
'call f; unwind_goto pad' continues at pad, the landing pad, when f exits
with unwind_exit.
the prog result
------------------------------------------------ ---------------------
a loop spins, the prog holds nothing the prog ends, r0 = 0
a loop spins under a lock, or with an iterator the loop exits
a lock is held, and to exit the loop is unsafe reject
a loop has no exit reject
a loop or bpf_loop() calls a subprog that spins the run returns
may_goto of the prog does not exit the loop the prog ends, r0 = 0
a jump to itself, a loop with two entries the prog ends, r0 = 0
recursion is too deep, calls have no unwind_goto the run returns
recursion is too deep, calls have unwind_goto the landing pad runs
a landing pad does not release a lock reject
recursion is not deep, but makes 2^60 calls the run returns
recursion through the callback of bpf_loop() the run returns
recursion with an inlined bpf_loop() stack <= 4096 bytes
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/verifier_budget.c | 1159 +++++++++++++++++
2 files changed, 1161 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_budget.c
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 78bce26662c4..017b15fa4b6a 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -29,6 +29,7 @@
#include "verifier_bpf_get_stack.skel.h"
#include "verifier_bpf_trap.skel.h"
#include "verifier_bswap.skel.h"
+#include "verifier_budget.skel.h"
#include "verifier_btf_array_access.skel.h"
#include "verifier_btf_ctx_access.skel.h"
#include "verifier_btf_flex_array.skel.h"
@@ -223,6 +224,7 @@ void test_verifier_bounds_mix_sign_unsign(void) { RUN(verifier_bounds_mix_sign_u
void test_verifier_bpf_get_stack(void) { RUN(verifier_bpf_get_stack); }
void test_verifier_bpf_trap(void) { RUN(verifier_bpf_trap); }
void test_verifier_bswap(void) { RUN(verifier_bswap); }
+void test_verifier_budget(void) { RUN(verifier_budget); }
void test_verifier_btf_array_access(void) { RUN(verifier_btf_array_access); }
void test_verifier_btf_ctx_access(void) { RUN(verifier_btf_ctx_access); }
void test_verifier_btf_flex_array(void) { RUN(verifier_btf_flex_array); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_budget.c b/tools/testing/selftests/bpf/progs/verifier_budget.c
new file mode 100644
index 000000000000..c126ddb6bfd2
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_budget.c
@@ -0,0 +1,1159 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+/*
+ * The verifier adds may_goto to a loop that it cannot count and a check of
+ * the stack to a recursive subprog. What the program does when it is out of budget:
+ * - it unwinds, when unwind_goto follows every call of the subprog and the subprog
+ * holds nothing of its own. The pads of the callers are verified.
+ * - the main prog and an async callback end
+ * - otherwise it leaves the loop, or returns 0 from the subprog
+ *
+ * It takes unwinding, per-cpu insns and timed may_goto in JIT. Some of the progs never end
+ * on their own.
+ */
+#define unwind_goto(off) ".8byte %[insn_unwind_goto_" #off "];"
+#define unwind_exit ".8byte %[insn_unwind_exit];"
+
+#define __imm_unwind_goto(off) \
+ __imm_insn(insn_unwind_goto_##off, \
+ BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, BPF_UNWIND_GOTO, off, 0))
+#define __imm_unwind_exit \
+ __imm_insn(insn_unwind_exit, \
+ BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0))
+
+extern void bpf_rcu_read_lock(void) __ksym;
+extern void bpf_rcu_read_unlock(void) __ksym;
+
+extern int bpf_iter_num_new(struct bpf_iter_num *it, int start, int end) __ksym;
+extern int *bpf_iter_num_next(struct bpf_iter_num *it) __ksym;
+extern void bpf_iter_num_destroy(struct bpf_iter_num *it) __ksym;
+
+/* The verifier doesn't know what is in there */
+unsigned long zero;
+unsigned long one = 1;
+
+struct {
+ __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, long);
+} empty SEC(".maps");
+
+struct locked {
+ struct bpf_spin_lock lock;
+ long val;
+};
+
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, int);
+ __type(value, struct locked);
+} locked SEC(".maps");
+
+SEC("socket")
+__description("budget: a loop in the main prog ends it")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_in_main(void)
+{
+ asm volatile (
+ "r6 = %[zero] ll;"
+ "1:"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(zero)
+ : __clobber_all
+ );
+}
+
+SEC("socket")
+__description("budget: a loop in the main prog that holds something is left")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_in_main_rcu(void)
+{
+ asm volatile (
+ "r6 = %[zero] ll;"
+ "call %[bpf_rcu_read_lock];"
+ "1:"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 1b;"
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(zero),
+ __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock)
+ : __clobber_all
+ );
+}
+
+/* 0 is not what this prog may return */
+SEC("cgroup/recvmsg4")
+__description("budget: a loop in the main prog that cannot return 0 is left")
+__arch_x86_64
+__arch_arm64
+__success
+__naked void loop_in_main_retval(void)
+{
+ asm volatile (
+ "r6 = %[zero] ll;"
+ "1:"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(zero)
+ : __clobber_all
+ );
+}
+
+/*
+ * do { p = lookup(); } while (!p); return *p;
+ * may_goto must not go where the loop goes when it's over: p may be NULL. Nothing is held, so
+ * it doesn't have to.
+ */
+SEC("socket")
+__description("budget: the code behind the loop counts on what the exit of the loop found out")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void exit_of_loop_checks_null(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u32 *)(r10 - 4) = r1;"
+ "1:"
+ "r2 = r10;"
+ "r2 += -4;"
+ "r1 = %[empty] ll;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto 1b;"
+ "r0 = *(u64 *)(r0 + 0);"
+ "exit;"
+ :
+ : __imm_addr(empty),
+ __imm(bpf_map_lookup_elem)
+ : __clobber_all
+ );
+}
+
+/* lock(); do { } while (one); unlock(); may_goto has to go where the lock is released */
+SEC("tc")
+__description("budget: a loop under a spin lock is left")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_under_spin_lock(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u32 *)(r10 - 4) = r1;"
+ "r2 = r10;"
+ "r2 += -4;"
+ "r1 = %[locked] ll;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto 2f;"
+ "r6 = r0;"
+ "r7 = %[one] ll;"
+ "r1 = r6;"
+ "call %[bpf_spin_lock];"
+ "1:"
+ "r1 = *(u64 *)(r7 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r1 = r6;"
+ "call %[bpf_spin_unlock];"
+ "r0 = 1;"
+ "2:"
+ "exit;"
+ :
+ : __imm_addr(locked),
+ __imm_addr(one),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_spin_lock),
+ __imm(bpf_spin_unlock)
+ : __clobber_all
+ );
+}
+
+/* Both: there is no way out */
+SEC("socket")
+__description("budget: something is held and the code behind the loop counts on its exit")
+__arch_x86_64
+__arch_arm64
+__failure __msg("R0 invalid mem access 'map_value_or_null'")
+__naked void exit_of_loop_checks_null_rcu(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "*(u32 *)(r10 - 4) = r1;"
+ "call %[bpf_rcu_read_lock];"
+ "1:"
+ "r2 = r10;"
+ "r2 += -4;"
+ "r1 = %[empty] ll;"
+ "call %[bpf_map_lookup_elem];"
+ "if r0 == 0 goto 1b;"
+ "r6 = *(u64 *)(r0 + 0);"
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = r6;"
+ "exit;"
+ :
+ : __imm_addr(empty),
+ __imm(bpf_map_lookup_elem),
+ __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock)
+ : __clobber_all
+ );
+}
+
+/*
+ * if (x) lock(); do { } while (zero); if (x) unlock();
+ * The state that holds nothing gets to the loop first, and may_goto would leave the prog.
+ * Then comes the state that holds the lock. It must not take a new start for every loop.
+ */
+#define LOOP_MAYBE_LOCKED \
+ "call %[bpf_get_prandom_u32];" \
+ "r7 = r0;" \
+ "if r7 == 0 goto 1f;" \
+ "call %[bpf_rcu_read_lock];" \
+ "1:" \
+ "r1 = *(u64 *)(r6 + 0);" \
+ "if r1 != 0 goto 1b;" \
+ "if r7 == 0 goto 2f;" \
+ "call %[bpf_rcu_read_unlock];" \
+ "2:"
+
+SEC("socket")
+__description("budget: 10 loops that find out late that something is held")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loops_maybe_locked(void)
+{
+ asm volatile (
+ "r6 = %[zero] ll;"
+ LOOP_MAYBE_LOCKED
+ LOOP_MAYBE_LOCKED
+ LOOP_MAYBE_LOCKED
+ LOOP_MAYBE_LOCKED
+ LOOP_MAYBE_LOCKED
+ LOOP_MAYBE_LOCKED
+ LOOP_MAYBE_LOCKED
+ LOOP_MAYBE_LOCKED
+ LOOP_MAYBE_LOCKED
+ LOOP_MAYBE_LOCKED
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(zero),
+ __imm(bpf_get_prandom_u32),
+ __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock)
+ : __clobber_all
+ );
+}
+
+__naked __noinline __used
+static unsigned long spins(void)
+{
+ asm volatile (
+ "r2 = %[zero] ll;"
+ "1:"
+ "r1 = *(u64 *)(r2 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 7;"
+ "exit;"
+ :
+ : __imm_addr(zero)
+ );
+}
+
+/* the same, for the tests that call it another way */
+__naked __noinline __used
+static unsigned long spins_caught(void)
+{
+ asm volatile (
+ "r2 = %[zero] ll;"
+ "1:"
+ "r1 = *(u64 *)(r2 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 7;"
+ "exit;"
+ :
+ : __imm_addr(zero)
+ );
+}
+
+SEC("socket")
+__description("budget: a loop in a subprog is left when there is no unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void loop_in_subprog(void)
+{
+ asm volatile (
+ "call spins;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("socket")
+__description("budget: a loop in a subprog unwinds when there is unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void loop_in_subprog_caught(void)
+{
+ asm volatile (
+ "call spins_caught;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+__naked __noinline __used
+static unsigned long spins_caught2(void)
+{
+ asm volatile (
+ "r2 = %[zero] ll;"
+ "1:"
+ "r1 = *(u64 *)(r2 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 7;"
+ "exit;"
+ :
+ : __imm_addr(zero)
+ );
+}
+
+SEC("socket")
+__description("budget: the pad releases what the caller holds")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void pad_unlocks(void)
+{
+ asm volatile (
+ "call %[bpf_rcu_read_lock];"
+ "call spins_caught2;"
+ unwind_goto(4)
+ "r6 = r0;"
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = r6;"
+ "exit;"
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_unwind_goto(4),
+ __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock)
+ : __clobber_all
+ );
+}
+
+__naked __noinline __used
+static unsigned long spins_caught3(void)
+{
+ asm volatile (
+ "r2 = %[zero] ll;"
+ "1:"
+ "r1 = *(u64 *)(r2 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 7;"
+ "exit;"
+ :
+ : __imm_addr(zero)
+ );
+}
+
+/* Nothing in the subprog unwinds. The pad is looked at because of may_goto that is added. */
+SEC("socket")
+__description("budget: the pad forgets what the caller holds")
+__arch_x86_64
+__arch_arm64
+__failure __msg("cannot be used inside bpf_rcu_read_lock-ed region")
+__naked void pad_forgets(void)
+{
+ asm volatile (
+ "call %[bpf_rcu_read_lock];"
+ "call spins_caught3;"
+ unwind_goto(4)
+ "r6 = r0;"
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = r6;"
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_unwind_goto(4),
+ __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock)
+ : __clobber_all
+ );
+}
+
+/* There is no pad for what the subprog took itself */
+__naked __noinline __used
+static unsigned long spins_locked(void)
+{
+ asm volatile (
+ "call %[bpf_rcu_read_lock];"
+ "r2 = %[zero] ll;"
+ "1:"
+ "r1 = *(u64 *)(r2 + 0);"
+ "if r1 != 0 goto 1b;"
+ "call %[bpf_rcu_read_unlock];"
+ "r0 = 7;"
+ "exit;"
+ :
+ : __imm_addr(zero),
+ __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock)
+ );
+}
+
+SEC("socket")
+__description("budget: a loop in a subprog that holds something is left")
+__arch_x86_64
+__arch_arm64
+__success __retval(7)
+__naked void loop_in_subprog_locked(void)
+{
+ asm volatile (
+ "call spins_locked;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+__naked __noinline __used
+static unsigned long spins_cb(void)
+{
+ asm volatile (
+ "r2 = %[zero] ll;"
+ "1:"
+ "r1 = *(u64 *)(r2 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(zero)
+ );
+}
+
+SEC("socket")
+__description("budget: a loop in a callback is left")
+__arch_x86_64
+__arch_arm64
+__success __retval(3)
+__naked void loop_in_callback(void)
+{
+ asm volatile (
+ "r1 = 3;"
+ "r2 = spins_cb ll;"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "exit;"
+ :
+ : __imm(bpf_loop)
+ : __clobber_all
+ );
+}
+
+/* r1: how deep to go. Returns the sum of 1 .. r1 */
+__naked __noinline __used
+static unsigned long sum_to(void)
+{
+ asm volatile (
+ "if r1 != 0 goto 1f;"
+ "r0 = 0;"
+ "exit;"
+ "1:"
+ "r6 = r1;"
+ "r1 += -1;"
+ "call sum_to;"
+ "r0 += r6;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("socket")
+__description("budget: recursion")
+__arch_x86_64
+__arch_arm64
+__success __retval(55)
+__naked void recursion(void)
+{
+ asm volatile (
+ "r1 = 10;"
+ "call sum_to;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+/* The one that is called too deep returns 0: 1000 + 999 + ... down to where it was */
+SEC("socket")
+__description("budget: recursion that is too deep returns 0 when there is no unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success
+__naked void recursion_too_deep(void)
+{
+ asm volatile (
+ "r1 = 1000;"
+ "call sum_to;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+unsigned long stack_top, stack_bottom;
+
+__naked __noinline __used
+static unsigned long does_nothing_cb(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "exit;"
+ );
+}
+
+/* Inlined bpf_loop() takes 3 slots of the stack and 3 registers that have to be saved. */
+__naked __noinline __used
+static unsigned long sum_to_with_loop(void)
+{
+ asm volatile (
+ "r2 = %[stack_bottom] ll;"
+ "*(u64 *)(r2 + 0) = r10;"
+ "if r1 != 0 goto 1f;"
+ "r0 = 0;"
+ "exit;"
+ "1:"
+ "r6 = r1;"
+ "r1 = 1;"
+ "r2 = does_nothing_cb ll;"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "r1 = r6;"
+ "r1 += -1;"
+ "call sum_to_with_loop;"
+ "r0 += r6;"
+ "exit;"
+ :
+ : __imm_addr(stack_bottom),
+ __imm(bpf_loop)
+ : __clobber_all
+ );
+}
+
+SEC("socket")
+__description("budget: frames of a recursion with inlined bpf_loop() fit in 4096 bytes")
+__arch_x86_64
+__success __retval(1)
+__naked void recursion_with_inlined_loop(void)
+{
+ asm volatile (
+ "r2 = %[stack_top] ll;"
+ "*(u64 *)(r2 + 0) = r10;"
+ "r1 = 1000;"
+ "call sum_to_with_loop;"
+ "r1 = %[stack_top] ll;"
+ "r1 = *(u64 *)(r1 + 0);"
+ "r2 = %[stack_bottom] ll;"
+ "r2 = *(u64 *)(r2 + 0);"
+ "r1 -= r2;"
+ "r0 = 1;"
+ "if r1 <= 4096 goto +1;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(stack_top),
+ __imm_addr(stack_bottom)
+ : __clobber_all
+ );
+}
+
+__naked __noinline __used
+static unsigned long sum_to_caught(void)
+{
+ asm volatile (
+ "if r1 != 0 goto 1f;"
+ "r0 = 0;"
+ "exit;"
+ "1:"
+ "r6 = r1;"
+ "r1 += -1;"
+ "call sum_to_caught;"
+ unwind_goto(2)
+ "r0 += r6;"
+ "exit;"
+ "r0 = 0;"
+ unwind_exit
+ :
+ : __imm_unwind_goto(2),
+ __imm_unwind_exit
+ : __clobber_all
+ );
+}
+
+SEC("socket")
+__description("budget: recursion with unwind_goto")
+__arch_x86_64
+__arch_arm64
+__success __retval(55)
+__naked void recursion_caught(void)
+{
+ asm volatile (
+ "r1 = 10;"
+ "call sum_to_caught;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+SEC("socket")
+__description("budget: recursion that is too deep unwinds")
+__arch_x86_64
+__arch_arm64
+__success __retval(42)
+__naked void recursion_too_deep_caught(void)
+{
+ asm volatile (
+ "r1 = 1000;"
+ "call sum_to_caught;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+/* r1: how deep to go. It calls itself twice: 2^r1 calls */
+__naked __noinline __used
+static unsigned long calls_itself_twice(void)
+{
+ asm volatile (
+ "if r1 != 0 goto 1f;"
+ "r0 = 1;"
+ "exit;"
+ "1:"
+ "r6 = r1;"
+ "r1 += -1;"
+ "call calls_itself_twice;"
+ "r7 = r0;"
+ "r1 = r6;"
+ "r1 += -1;"
+ "call calls_itself_twice;"
+ "r0 += r7;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("socket")
+__description("budget: recursion that is not deep, 2^10 calls")
+__arch_x86_64
+__arch_arm64
+__success __retval(1024)
+__naked void recursion_tree(void)
+{
+ asm volatile (
+ "r1 = 10;"
+ "call calls_itself_twice;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+/* There is room in the stack for 60 frames. The number of calls has to be limited too. */
+SEC("socket")
+__description("budget: recursion that is not deep, 2^60 calls")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void recursion_tree_forever(void)
+{
+ asm volatile (
+ "r1 = 60;"
+ "call calls_itself_twice;"
+ "r0 = 0;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+/* The calls are counted in a run of the prog, not from the call that starts a recursion */
+SEC("socket")
+__description("budget: a loop that calls a subprog that calls itself 2^60 times")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void loop_calls_recursion_tree(void)
+{
+ asm volatile (
+ "r8 = %[one] ll;"
+ "1:"
+ "r1 = 60;"
+ "call calls_itself_twice;"
+ "r1 = *(u64 *)(r8 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(one)
+ : __clobber_all
+ );
+}
+
+/*
+ * Every frame loops forever and calls itself in every trip. When the recursion is out of
+ * budget every call returns at once, 65535 times in every frame that is on the stack.
+ * It has to be cheap: the report is printed once.
+ */
+__naked __noinline __used
+static unsigned long calls_itself_in_a_loop(void)
+{
+ asm volatile (
+ "r6 = %[one] ll;"
+ "1:"
+ "call calls_itself_in_a_loop;"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(one)
+ );
+}
+
+SEC("socket")
+__description("budget: a subprog that calls itself in a loop that never ends")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void recursion_in_a_loop(void)
+{
+ asm volatile (
+ "call calls_itself_in_a_loop;"
+ "r0 = 1;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+/*
+ * calls_loop() -> bpf_loop() -> loop_cb() -> calls_loop(): the frames of the helper are on
+ * the stack too
+ */
+static int loop_cb(__u32 i, void *ctx);
+
+static __noinline int calls_loop(int x)
+{
+ bpf_loop(1, loop_cb, NULL, 0);
+ return 0;
+}
+
+static __noinline int loop_cb(__u32 i, void *ctx)
+{
+ return calls_loop(i);
+}
+
+SEC("socket")
+__description("budget: recursion through a helper that calls a callback")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+int recursion_through_callback(void *ctx)
+{
+ return calls_loop(0);
+}
+
+/*
+ * There is one time slice for a run of the prog, not one for every call of a subprog.
+ * The subprogs below use it up, leave their loops and return. Their callers call them again.
+ */
+__naked __noinline __used
+static unsigned long spins_and_returns(void)
+{
+ asm volatile (
+ "r2 = %[one] ll;"
+ "1:"
+ "r1 = *(u64 *)(r2 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(one)
+ );
+}
+
+SEC("socket")
+__description("budget: a loop that calls a subprog that spins")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void loop_calls_spinning_subprog(void)
+{
+ asm volatile (
+ "r6 = %[one] ll;"
+ "1:"
+ "call spins_and_returns;"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(one)
+ : __clobber_all
+ );
+}
+
+SEC("socket")
+__description("budget: bpf_loop() that calls a callback that spins 8M times")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void bpf_loop_calls_spinning_callback(void)
+{
+ asm volatile (
+ "r1 = 0x800000;"
+ "r2 = spins_and_returns ll;"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm(bpf_loop)
+ : __clobber_all
+ );
+}
+
+/*
+ * may_goto of the program stops the loop that it leaves. This one goes to itself
+ * when it is out of budget. The loop spins until may_goto that is added ends the prog.
+ */
+SEC("socket")
+__description("budget: may_goto that doesn't leave the loop")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void may_goto_stays_in_loop(void)
+{
+ asm volatile (
+ "r6 = %[one] ll;"
+ "1:"
+ ".8byte %[may_goto_self];"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(one),
+ __imm_insn(may_goto_self, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, -1, 0))
+ : __clobber_all
+ );
+}
+
+/* The jump to itself has to go through may_goto that is added in front of it */
+SEC("socket")
+__description("budget: a conditional jump to itself")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void jump_to_itself(void)
+{
+ asm volatile (
+ "r6 = %[one] ll;"
+ "r1 = *(u64 *)(r6 + 0);"
+ "1:"
+ "if r1 != 0 goto 1b;"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(one)
+ : __clobber_all
+ );
+}
+
+/* A cycle with two entries: there is no insn that every way into it goes through */
+SEC("socket")
+__description("budget: a loop with two entries")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void loop_with_two_entries(void)
+{
+ asm volatile (
+ "r6 = %[one] ll;"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 2f;"
+ "1:"
+ "r2 = *(u64 *)(r6 + 0);"
+ "if r2 == 0 goto 3f;"
+ "2:"
+ "r3 = *(u64 *)(r6 + 0);"
+ "if r3 != 0 goto 1b;"
+ "3:"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(one)
+ : __clobber_all
+ );
+}
+
+/*
+ * may_goto in front of a loop with two entries dominates the back edge, like everything on
+ * the way to the loop does when the head of the loop doesn't. It's not in the loop.
+ */
+SEC("socket")
+__description("budget: may_goto in front of a loop with two entries")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void may_goto_before_loop_with_two_entries(void)
+{
+ asm volatile (
+ "r6 = %[one] ll;"
+ ".8byte %[may_goto_0];"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 == 0 goto 2f;"
+ "1:"
+ "r2 = *(u64 *)(r6 + 0);"
+ "2:"
+ "r3 = *(u64 *)(r6 + 0);"
+ "if r3 != 0 goto 1b;"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(one),
+ __imm_insn(may_goto_0, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 0, 0))
+ : __clobber_all
+ );
+}
+
+/* It leaves the inner loop. The outer loop goes on. */
+SEC("socket")
+__description("budget: may_goto that leaves the inner loop only")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void may_goto_leaves_inner_loop(void)
+{
+ asm volatile (
+ "r6 = %[one] ll;"
+ "1:"
+ ".8byte %[may_goto_2];"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(one),
+ __imm_insn(may_goto_2, BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, 0, 2, 0))
+ : __clobber_all
+ );
+}
+
+/*
+ * bpf_iter_num_next() stops the loop that the program leaves when there is nothing left.
+ * This one goes on. The iterator is held, so the loop is left when it is out of budget.
+ */
+SEC("socket")
+__description("budget: a loop that goes on when its iterator is drained")
+__arch_x86_64
+__arch_arm64
+__success __retval(1)
+__naked void loop_after_iter_is_drained(void)
+{
+ asm volatile (
+ "r1 = r10;"
+ "r1 += -8;"
+ "r2 = 0;"
+ "r3 = 10;"
+ "call %[bpf_iter_num_new];"
+ "r6 = %[one] ll;"
+ "1:"
+ "r1 = r10;"
+ "r1 += -8;"
+ "call %[bpf_iter_num_next];"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r1 = r10;"
+ "r1 += -8;"
+ "call %[bpf_iter_num_destroy];"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(one),
+ __imm(bpf_iter_num_new),
+ __imm(bpf_iter_num_next),
+ __imm(bpf_iter_num_destroy)
+ : __clobber_all
+ );
+}
+
+/*
+ * The loop at 3: spins. It has no jump that every trip goes through, so may_goto is at its head.
+ * The first way out of it, from 2: to 1:, is the back edge of the loop at 1:, which
+ * has no may_goto of its own: bpf_iter_num_next() stops it. From the head of 3: to 1: there is
+ * no bpf_iter_num_next().
+ */
+SEC("socket")
+__description("budget: may_goto at the head of a loop leaves the loops around it too")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__naked void guard_at_head_of_nested_loop(void)
+{
+ asm volatile (
+ "r6 = %[one] ll;"
+ "r9 = %[zero] ll;"
+ "r7 = 0;"
+ "r1 = r10;"
+ "r1 += -8;"
+ "r2 = 0;"
+ "r3 = 2;"
+ "call %[bpf_iter_num_new];"
+ "call %[bpf_rcu_read_lock];"
+ "r7 += 1;"
+ "goto 1f;"
+ "2:"
+ "r7 += 1;"
+ "r1 = *(u64 *)(r9 + 0);"
+ "if r1 != 0 goto 4f;"
+ "r1 = *(u64 *)(r9 + 0);"
+ "if r1 != 0 goto 4f;"
+ "1:"
+ "r7 += 1;"
+ "r1 = *(u64 *)(r6 + 0);"
+ "5:"
+ "if r1 == 0 goto 5b;"
+ "3:"
+ "r1 = *(u64 *)(r9 + 0);"
+ "if r1 != 0 goto +0;"
+ "4:"
+ "r7 += 1;"
+ "r1 = *(u64 *)(r9 + 0);"
+ "if r1 == 0 goto 3b;"
+ "r1 = r10;"
+ "r1 += -8;"
+ "call %[bpf_iter_num_next];"
+ "if r0 == 0 goto 6f;"
+ "r1 = *(u64 *)(r9 + 0);"
+ "if r1 == 0 goto 2b;"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 4b;"
+ "6:"
+ "call %[bpf_rcu_read_unlock];"
+ "r1 = r10;"
+ "r1 += -8;"
+ "call %[bpf_iter_num_destroy];"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_addr(one),
+ __imm_addr(zero),
+ __imm(bpf_iter_num_new),
+ __imm(bpf_iter_num_next),
+ __imm(bpf_iter_num_destroy),
+ __imm(bpf_rcu_read_lock),
+ __imm(bpf_rcu_read_unlock)
+ : __clobber_all
+ );
+}
+
+SEC("socket")
+__description("budget: a loop without an exit that is not entered at its head")
+__arch_x86_64
+__arch_arm64
+__failure __msg("never exits")
+__naked void loop_without_exit_two_entries(void)
+{
+ asm volatile (
+ "r6 = %[zero] ll;"
+ "r0 = 0;"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 == 0 goto 2f;"
+ "1:"
+ "r0 += 1;"
+ "2:"
+ "goto 1b;"
+ :
+ : __imm_addr(zero)
+ : __clobber_all
+ );
+}
+
+SEC("socket")
+__description("budget: a loop without an exit")
+__arch_x86_64
+__arch_arm64
+__failure __msg("never exits")
+__naked void loop_without_exit(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "1:"
+ "r0 += 1;"
+ "goto 1b;"
+ ::: __clobber_all
+ );
+}
+
+/*
+ * BTF FUNC records are not generated for kfuncs referenced
+ * from inline assembly. These records are necessary for
+ * libbpf to link the program. The function below is a hack
+ * to ensure that BTF FUNC records are generated.
+ */
+void __kfunc_btf_root(void)
+{
+ bpf_rcu_read_lock();
+ bpf_rcu_read_unlock();
+ bpf_iter_num_new(0, 0, 0);
+ bpf_iter_num_next(0);
+ bpf_iter_num_destroy(0);
+}
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 78/81] selftests/bpf: Test progs that run out of budget
2026-10-26 2:59 ` [PATCH bpf 78/81] selftests/bpf: Test progs that run out of budget Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 2866 bytes --]
> selftests/bpf: Test progs that run out of budget
TL;DR: Test issue: recursion_through_callback's bpf_loop() with constant flags
is likely inlined, so it doesn't test helper-frame (BPF_CALLBACK_OVERHEAD)
recursion as its comment claims; a non-constant flags value may avoid
inlining.
> diff --git a/tools/testing/selftests/bpf/progs/verifier_budget.c b/tools/testing/selftests/bpf/progs/verifier_budget.c
> new file mode 100644
> index 000000000000..c126ddb6bfd2
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/verifier_budget.c
> @@ -0,0 +1,1159 @@
[ ... ]
> +/*
> + * calls_loop() -> bpf_loop() -> loop_cb() -> calls_loop(): the frames of the helper are on
> + * the stack too
> + */
> +static int loop_cb(__u32 i, void *ctx);
> +
> +static __noinline int calls_loop(int x)
> +{
> + bpf_loop(1, loop_cb, NULL, 0);
> + return 0;
> +}
> +
> +static __noinline int loop_cb(__u32 i, void *ctx)
> +{
> + return calls_loop(i);
> +}
> +
> +SEC("socket")
> +__description("budget: recursion through a helper that calls a callback")
> +__arch_x86_64
> +__arch_arm64
> +__success __retval(0)
> +int recursion_through_callback(void *ctx)
> +{
> + return calls_loop(0);
> +}
The comment says the frames of the helper are on the stack, but is that the
case here? The flags argument is the constant 0 and the callback is always
loop_cb, so update_loop_inline_state() leaves fit_for_inline set, and
bpf_optimize_bpf_loop() replaces the call:
kernel/bpf/fixups.c:bpf_optimize_bpf_loop() {
...
if (is_bpf_loop_call(insn) && inline_state->fit_for_inline &&
!subprogs[inline_state->callback_subprogno].arena_stack_entry) {
...
}
bpf_check() runs that pass before bpf_insert_recursion_checks(), so at run
time this looks like plain mutual recursion calls_loop() -> loop_cb() ->
calls_loop() with an inlined loop, with no helper frame between the BPF
frames.
That is also what recursion_with_inlined_loop covers. The commit message
lists "recursion through the callback of bpf_loop()" and "recursion with an
inlined bpf_loop()" as two separate cases, so was this test meant to be the
one that is not inlined?
That is the case BPF_CALLBACK_OVERHEAD exists for, the frames of the helper
or kfunc between a callback and its caller. I could not find a test in this
commit, or later in the series, that runs such a recursion.
recursion_via_global_func_and_callback in verifier_global_subprogs.c is only
loaded.
Would passing a flags value the verifier cannot see, for example
bpf_loop(1, loop_cb, NULL, zero) using the existing "unsigned long zero;"
global, stop the inlining while still being 0 at run time?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* [PATCH bpf 79/81] selftests/bpf: Test 'unwound' in bpf_prog_info
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (77 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 78/81] selftests/bpf: Test progs that run out of budget Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 80/81] selftests/bpf: Test the stack in arena after recursion ran out of budget Alexei Starovoitov
` (3 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
The kernel sets the 'unwound' bit in bpf_prog_info when a run of the prog
does not end in the usual way. It prints the reason to stderr of the prog,
the BPF_STDERR stream. Run four progs one time each. The bit is 0 before
each run. 'call f; unwind_goto pad' continues at pad, the landing pad, when
f exits with unwind_exit.
prog what occurs retval unwound
---------- ----------------------------------------- ------ -------
caught a landing pad catches unwind_exit 42 0
not_caught the main prog exits with unwind_exit 7 1
spins may_goto that the verifier adds times out 0 1
too_deep a subprog calls itself without end 3 1
stderr of caught is empty. The other progs print:
not_caught
ERROR: The program was unwound
spins
ERROR: Timeout detected for may_goto instruction
too_deep
ERROR: No room in the stack for a call of a recursive function
not_caught has 7 in r0 at unwind_exit. The call that reports the event must
not change r0.
The test also attaches to the recursive subprog. fentry loads. The verifier
rejects fexit with
"Only fentry can attach to calls_itself, which is recursive". The
trampoline of fexit stays on the stack at each level of the recursion, and
the check of the stack does not count it.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/unwound.c | 96 +++++++++++++++
tools/testing/selftests/bpf/progs/unwound.c | 113 ++++++++++++++++++
2 files changed, 209 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/unwound.c
create mode 100644 tools/testing/selftests/bpf/progs/unwound.c
diff --git a/tools/testing/selftests/bpf/prog_tests/unwound.c b/tools/testing/selftests/bpf/prog_tests/unwound.c
new file mode 100644
index 000000000000..d5b4454ae011
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/unwound.c
@@ -0,0 +1,96 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <test_progs.h>
+#include "unwound.skel.h"
+
+/* Run the prog once. What it returns, what bpf_prog_info says and what is in its stderr. */
+static void run(struct bpf_program *prog, int retval, bool unwound, const char *msg)
+{
+ char pkt[64] = {}, buf[4096] = {};
+ LIBBPF_OPTS(bpf_test_run_opts, opts, .data_in = pkt, .data_size_in = sizeof(pkt), .repeat = 1);
+ struct bpf_prog_info info = {};
+ __u32 len = sizeof(info);
+ int fd = bpf_program__fd(prog);
+
+ if (!test__start_subtest(bpf_program__name(prog)))
+ return;
+
+ ASSERT_OK(bpf_prog_get_info_by_fd(fd, &info, &len), "info_before");
+ ASSERT_EQ((int)info.unwound, 0, "unwound_before");
+
+ ASSERT_OK(bpf_prog_test_run_opts(fd, &opts), "test_run");
+ ASSERT_EQ(opts.retval, retval, "retval");
+
+ memset(&info, 0, sizeof(info));
+ len = sizeof(info);
+ ASSERT_OK(bpf_prog_get_info_by_fd(fd, &info, &len), "info_after");
+ ASSERT_EQ((int)info.unwound, (int)unwound, "unwound_after");
+
+ bpf_prog_stream_read(fd, BPF_STREAM_STDERR, buf, sizeof(buf) - 1, NULL);
+ if (msg)
+ ASSERT_HAS_SUBSTR(buf, msg, "stderr");
+ else
+ ASSERT_STREQ(buf, "", "stderr");
+}
+
+/*
+ * The trampoline of fexit calls the target and stays on the stack at every level of a recursion.
+ * The check of the stack in the target knows nothing about it.
+ */
+static void attach(struct unwound *tgt, const char *name, const char *msg)
+{
+ static char log_buf[4096];
+ struct bpf_program *prog, *p;
+ struct unwound *skel;
+ int err;
+
+ if (!test__start_subtest(name))
+ return;
+ skel = unwound__open();
+ if (!ASSERT_OK_PTR(skel, "open"))
+ return;
+ prog = bpf_object__find_program_by_name(skel->obj, name);
+ if (!ASSERT_OK_PTR(prog, name))
+ goto out;
+ bpf_object__for_each_program(p, skel->obj)
+ bpf_program__set_autoload(p, p == prog);
+ log_buf[0] = 0;
+ bpf_program__set_log_buf(prog, log_buf, sizeof(log_buf));
+ err = bpf_program__set_attach_target(prog, bpf_program__fd(tgt->progs.too_deep),
+ "calls_itself");
+ if (!ASSERT_OK(err, "set_attach_target"))
+ goto out;
+ err = unwound__load(skel);
+ if (msg) {
+ ASSERT_ERR(err, name);
+ ASSERT_HAS_SUBSTR(log_buf, msg, name);
+ } else {
+ ASSERT_OK(err, name);
+ }
+out:
+ unwound__destroy(skel);
+}
+
+void test_unwound(void)
+{
+ struct unwound *skel;
+
+#if !defined(__x86_64__) && !defined(__aarch64__)
+ /* unwinding, per-cpu insns and timed may_goto in JIT */
+ test__skip();
+ return;
+#endif
+ skel = unwound__open_and_load();
+ if (!ASSERT_OK_PTR(skel, "open_and_load"))
+ return;
+
+ run(skel->progs.caught, 42, false, NULL);
+ run(skel->progs.not_caught, 7, true, "ERROR: The program was unwound");
+ run(skel->progs.spins, 0, true, "ERROR: Timeout detected for may_goto instruction");
+ run(skel->progs.too_deep, 3, true,
+ "ERROR: No room in the stack for a call of a recursive function");
+
+ attach(skel, "on_entry", NULL);
+ attach(skel, "on_exit", "Only fentry can attach to calls_itself, which is recursive");
+
+ unwound__destroy(skel);
+}
diff --git a/tools/testing/selftests/bpf/progs/unwound.c b/tools/testing/selftests/bpf/progs/unwound.c
new file mode 100644
index 000000000000..66ec90908838
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/unwound.c
@@ -0,0 +1,113 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "../../../include/linux/filter.h"
+#include "bpf_misc.h"
+
+#define unwind_goto(off) ".8byte %[insn_unwind_goto_" #off "];"
+#define unwind_exit ".8byte %[insn_unwind_exit];"
+
+#define __imm_unwind_goto(off) \
+ __imm_insn(insn_unwind_goto_##off, \
+ BPF_RAW_INSN(BPF_JMP | BPF_JCOND, 0, BPF_UNWIND_GOTO, off, 0))
+#define __imm_unwind_exit \
+ __imm_insn(insn_unwind_exit, \
+ BPF_RAW_INSN(BPF_JMP | BPF_EXIT, 0, BPF_UNWIND_EXIT, 0, 0))
+
+unsigned long one = 1;
+
+__naked __noinline __used
+static unsigned long unwinds(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ unwind_exit
+ :
+ : __imm_unwind_exit
+ );
+}
+
+SEC("socket")
+__naked void caught(void)
+{
+ asm volatile (
+ "call unwinds;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 42;"
+ "exit;"
+ :
+ : __imm_unwind_goto(1)
+ : __clobber_all
+ );
+}
+
+SEC("socket")
+__naked void not_caught(void)
+{
+ asm volatile (
+ "call unwinds;"
+ unwind_goto(1)
+ "exit;"
+ "r0 = 7;"
+ unwind_exit
+ :
+ : __imm_unwind_goto(1),
+ __imm_unwind_exit
+ : __clobber_all
+ );
+}
+
+SEC("socket")
+__naked void spins(void)
+{
+ asm volatile (
+ "r6 = %[one] ll;"
+ "1:"
+ "r1 = *(u64 *)(r6 + 0);"
+ "if r1 != 0 goto 1b;"
+ "r0 = 1;"
+ "exit;"
+ :
+ : __imm_addr(one)
+ : __clobber_all
+ );
+}
+
+__naked __noinline __used
+static unsigned long calls_itself(void)
+{
+ asm volatile (
+ "r1 += 1;"
+ "call calls_itself;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+SEC("socket")
+__naked void too_deep(void)
+{
+ asm volatile (
+ "r1 = 0;"
+ "call calls_itself;"
+ "r0 = 3;"
+ "exit;"
+ ::: __clobber_all
+ );
+}
+
+/* the test attaches them to calls_itself() */
+SEC("?fentry")
+int on_entry(void *ctx)
+{
+ return 0;
+}
+
+SEC("?fexit")
+int on_exit(void *ctx)
+{
+ return 0;
+}
+
+char _license[] SEC("license") = "GPL";
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 80/81] selftests/bpf: Test the stack in arena after recursion ran out of budget
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (78 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 79/81] selftests/bpf: Test 'unwound' in bpf_prog_info Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 2:59 ` [PATCH bpf 81/81] selftests/bpf: Test callx of one of a set of functions Alexei Starovoitov
` (2 subsequent siblings)
82 siblings, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A prog can have a second stack in a BPF arena, and r12 is its stack
pointer. A subprog takes its frame with 'r12 += N' as its first insn, and
the verifier adds 'r12 += -N' before each exit. The verifier also adds a
check to the entry of a recursive subprog. When the stack or the number of
calls is used up, the check exits the subprog immediately. This exit gives
the frame back too. Thus the check must come after 'r12 += N'.
recurses:
r12 += 32 ; the subprog takes its frame
if there is budget goto 1f ; the check
r0 = 0
r12 += -32
exit
1:
...
r12 += -32
exit
With the check before 'r12 += 32', the call that is stopped gives back a
frame that it did not take. Then r12 of the caller is incorrect.
The test recurses until the check stops it. After the call, r12 of the
caller and the value that the caller stored in its frame must be the same
as before.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../bpf/progs/verifier_arena_stack.c | 48 +++++++++++++++++++
1 file changed, 48 insertions(+)
diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_stack.c b/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
index 79a4552f1a93..9eafdf032a27 100644
--- a/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
+++ b/tools/testing/selftests/bpf/progs/verifier_arena_stack.c
@@ -213,6 +213,54 @@ __naked void helper(void)
: __clobber_all);
}
+static __naked __noinline __used long recurses(long n)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[stx32];"
+ ".8byte %[stx8];"
+ "if r1 == 0 goto 1f;"
+ "r1 += -1;"
+ "call recurses;"
+"1:"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(32)),
+ __imm_insn(stx32, STX(BPF_DW, -32, BPF_REG_1)),
+ __imm_insn(stx8, STX(BPF_DW, -8, BPF_REG_1))
+ : __clobber_all);
+}
+
+SEC("syscall")
+__description("arena stack: r12 after the recursion ran out of budget")
+__success __retval(0)
+__naked void recursion(void)
+{
+ asm volatile (
+ ".8byte %[adj];"
+ ".8byte %[st];" /* *(u64 *)(r12 - 8) = 4660 */
+ ".8byte %[mov6];" /* r6 = r12 */
+ "r1 = 100000;"
+ "call recurses;"
+ ".8byte %[mov7];" /* r7 = r12 */
+ "r0 = 1;"
+ "if r6 != r7 goto 1f;"
+ "r0 = 2;"
+ ".8byte %[ldx];" /* r1 = *(u64 *)(r12 - 8) */
+ "if r1 != 4660 goto 1f;"
+ "r0 = 0;"
+"1:"
+ "exit;"
+ :
+ : __imm_insn(adj, ADJ(16)),
+ __imm_insn(st, ST(BPF_DW, -8, 4660)),
+ __imm_insn(mov6, MOV(BPF_REG_6)),
+ __imm_insn(mov7, MOV(BPF_REG_7)),
+ __imm_insn(ldx, LDX(BPF_DW, BPF_REG_1, -8))
+ : __clobber_all);
+}
+
static __naked __noinline __used long store_through(long addr)
{
asm volatile (
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* [PATCH bpf 81/81] selftests/bpf: Test callx of one of a set of functions
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (79 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 80/81] selftests/bpf: Test the stack in arena after recursion ran out of budget Alexei Starovoitov
@ 2026-10-26 2:59 ` Alexei Starovoitov
2026-10-26 3:53 ` bot+bpf-ci
2026-10-26 16:57 ` [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Ihor Solodrai
2026-10-26 22:28 ` Ihor Solodrai
82 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 2:59 UTC (permalink / raw)
To: bpf; +Cc: daniel, andrii, eddyz87, memxor
From: Alexei Starovoitov <ast@kernel.org>
A load from a table of functions at a variable offset gives one register
that holds a pointer to one of N functions, func(one_of=N). See
"bpf: Track a pointer to one of a set of functions".
test, callx_rodata_... the case result
-------------------------------- ------------------------------ ------
one_of_two a table of two functions loads
same_function_twice the same function two times loads
callback_one_of_two bpf_loop() gets the register loads
one_of_two_at_loop_head callx in a loop, 4 iterations reject
one_of_two_at_loop_head_ok callx in a loop, 3 iterations loads
callback_one_of_two_at_loop_head bpf_loop() in a loop reject
400_functions 3 states, 400 functions loads
- one_of_two expects R2=func(one_of=2) in the log, and callx must call the
function that the index selects. same_function_twice expects R2=func().
- In the tests with a loop, callx or bpf_loop() is the loop head. The prog
is unsafe only in the fourth iteration, and only with one of the two
functions.
- In 400_functions three states that the verifier cannot join get to the
load. One state for each function gives 1200 states at one insn, and the
verifier rejects that with
"more than 1024 states that cannot be joined". With a set there are
3 states.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
---
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/verifier_callx_many.c | 90 +++++++
.../bpf/progs/verifier_callx_rodata.c | 220 ++++++++++++++++++
3 files changed, 312 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_callx_many.c
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index 017b15fa4b6a..3b56cad22156 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -36,6 +36,7 @@
#include "verifier_btf_unreliable_prog.skel.h"
#include "verifier_call_large_imm.skel.h"
#include "verifier_callx.skel.h"
+#include "verifier_callx_many.skel.h"
#include "verifier_callx_rodata.skel.h"
#include "verifier_cfg.skel.h"
#include "verifier_cgroup_inv_retcode.skel.h"
@@ -231,6 +232,7 @@ void test_verifier_btf_flex_array(void) { RUN(verifier_btf_flex_array); }
void test_verifier_btf_unreliable_prog(void) { RUN(verifier_btf_unreliable_prog); }
void test_verifier_call_large_imm(void) { RUN(verifier_call_large_imm); }
void test_verifier_callx(void) { RUN_JITED(verifier_callx); }
+void test_verifier_callx_many(void) { RUN_JITED(verifier_callx_many); }
void test_verifier_callx_rodata(void) { RUN_JITED(verifier_callx_rodata); }
void test_verifier_cfg(void) { RUN(verifier_cfg); }
void test_verifier_cgroup_inv_retcode(void) { RUN(verifier_cgroup_inv_retcode); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_many.c b/tools/testing/selftests/bpf/progs/verifier_callx_many.c
new file mode 100644
index 000000000000..5f8ee8490076
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_many.c
@@ -0,0 +1,90 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * callx through a table of many functions. libbpf appends the functions that read-only
+ * data points to to every prog of the object that has callx, hence a file of its own.
+ */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../../../include/linux/filter.h"
+
+/* gcc doesn't support indirect calls */
+#if defined(__clang__) && (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
+ (defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64))
+
+/*
+ * Read-only data with pointers to functions, where the compiler puts tables
+ * of functions, structures of operations and vtables. libbpf resolves
+ * a pointer to the offset of the function in the program and the kernel
+ * recognizes it by that value.
+ */
+#define DATA(SECTION, NAME, ...) \
+ ".pushsection " SECTION ",@progbits;" \
+ ".balign 8;" \
+ #NAME "_%=:" \
+ __VA_ARGS__ \
+ ".type " #NAME "_%=, @object;" \
+ ".size " #NAME "_%=, .-" #NAME "_%=;" \
+ ".popsection;"
+
+#define RODATA(NAME, ...) DATA(".rodata,\"a\"", NAME, __VA_ARGS__)
+
+#define REP10(M, p) M(p##0) M(p##1) M(p##2) M(p##3) M(p##4) M(p##5) M(p##6) M(p##7) M(p##8) M(p##9)
+#define REP100(M, p) REP10(M, p##0) REP10(M, p##1) REP10(M, p##2) REP10(M, p##3) REP10(M, p##4) \
+ REP10(M, p##5) REP10(M, p##6) REP10(M, p##7) REP10(M, p##8) REP10(M, p##9)
+#define REP400(M) REP100(M, 0) REP100(M, 1) REP100(M, 2) REP100(M, 3)
+
+#define MANY_FUNC(n) \
+__naked __noinline __used static unsigned long many##n(void) \
+{ \
+ asm volatile ("r0 = r1; r0 += 1; exit;"); \
+}
+#define MANY_QUAD(n) ".quad many" #n ";"
+
+REP400(MANY_FUNC)
+
+/* 3 states that have no join get to a table of 400 functions: 3 states, and not 1200 */
+SEC("socket")
+__success __retval(11)
+__naked void callx_rodata_400_functions(void)
+{
+ asm volatile (
+ RODATA(big, REP400(MANY_QUAD))
+ "r7 = r1;"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 == 1 goto 1f;"
+ "r7 = r10;"
+ "if r0 == 2 goto 1f;"
+ "r7 = 0;"
+ "1:"
+ "call %[bpf_get_prandom_u32];"
+ "if r0 > 399 goto 2f;"
+ "r0 <<= 3;"
+ "r2 = big_%= ll;"
+ "r2 += r0;"
+ "r2 = *(u64 *)(r2 + 0);"
+ "r1 = 10;"
+ "callx r2;"
+ "if r7 != 0 goto +0;"
+ "exit;"
+ "2:"
+ "r0 = 11;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+#else
+
+SEC("socket")
+__success
+int dummy(void *ctx)
+{
+ return 0;
+}
+
+#endif
+
+char _license[] SEC("license") = "GPL";
diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
index 85141cd73298..2575313a977a 100644
--- a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
+++ b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
@@ -173,6 +173,226 @@ __naked void callx_rodata_array_of_structs(void)
: __clobber_all);
}
+/* one state with one of two functions, and not two states */
+SEC("socket")
+__success __retval(11)
+__log_level(2)
+__msg("r2 = *(u64 *)(r2 +0) ; R2=func(one_of=2)")
+__naked void callx_rodata_one_of_two(void)
+{
+ asm volatile (
+ FUNC_TABLE2(tbl, add1, add2)
+ "call %[bpf_get_prandom_u32];"
+ "r6 = r0;"
+ "r6 &= 1;"
+ "r3 = r6;"
+ "r3 <<= 3;"
+ "r2 = tbl_%= ll;"
+ "r2 += r3;"
+ "r2 = *(u64 *)(r2 + 0);"
+ "r1 = 10;"
+ "callx r2;"
+ "r0 -= r6;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+SEC("socket")
+__success __retval(11)
+__log_level(2)
+__msg("r2 = *(u64 *)(r2 +0) ; R2=func()")
+__naked void callx_rodata_same_function_twice(void)
+{
+ asm volatile (
+ FUNC_TABLE2(tbl, add1, add1)
+ "call %[bpf_get_prandom_u32];"
+ "r0 &= 1;"
+ "r0 <<= 3;"
+ "r2 = tbl_%= ll;"
+ "r2 += r0;"
+ "r2 = *(u64 *)(r2 + 0);"
+ "r1 = 10;"
+ "callx r2;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+__naked __noinline __used
+static unsigned long loop_cb_stop(void)
+{
+ asm volatile (
+ "r0 = 1;"
+ "exit;"
+ );
+}
+
+__naked __noinline __used
+static unsigned long loop_cb_go_on(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "exit;"
+ );
+}
+
+/* a helper needs to know the callback */
+SEC("socket")
+__success __retval(0)
+__naked void callx_rodata_callback_one_of_two(void)
+{
+ asm volatile (
+ FUNC_TABLE2(cbs, loop_cb_stop, loop_cb_go_on)
+ "call %[bpf_get_prandom_u32];"
+ "r0 &= 1;"
+ "r0 <<= 3;"
+ "r2 = cbs_%= ll;"
+ "r2 += r0;"
+ "r2 = *(u64 *)(r2 + 0);"
+ "r6 = r2;"
+ "r1 = 2;"
+ "r3 = 0;"
+ "r4 = 0;"
+ "call %[bpf_loop];"
+ /* libbpf resolves the pointers for progs that have callx */
+ "callx r6;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32), __imm(bpf_loop)
+ : __clobber_all);
+}
+
+__naked __noinline __used
+static unsigned long not_three(void)
+{
+ asm volatile (
+ "r0 = 0;"
+ "if r1 != 3 goto +1;"
+ "r10 = 0;"
+ "exit;"
+ );
+}
+
+/*
+ * callx is the head of a loop whose iterations are looked at one by one, because of r1.
+ * The states with one function each are in the iteration of the state with both.
+ */
+SEC("socket")
+__failure __msg("frame pointer is read only")
+__naked void callx_rodata_one_of_two_at_loop_head(void)
+{
+ asm volatile (
+ FUNC_TABLE2(tbl, add1, not_three)
+ "call %[bpf_get_prandom_u32];"
+ "r0 &= 1;"
+ "r0 <<= 3;"
+ "r6 = tbl_%= ll;"
+ "r6 += r0;"
+ "r6 = *(u64 *)(r6 + 0);"
+ "r7 = 0;"
+ "r1 = 0;"
+ "l_%=:"
+ "callx r6;"
+ "r7 += 1;"
+ "r1 = r7;"
+ "if r7 < 4 goto l_%=;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+/* and the last iteration leaves the loop. It is not run: a loop that is not left has no end. */
+SEC("socket")
+__success __log_level(2)
+__msg("r0 = r7 {{.*}}R7=3")
+__naked void callx_rodata_one_of_two_at_loop_head_ok(void)
+{
+ asm volatile (
+ FUNC_TABLE2(tbl, add1, not_three)
+ "call %[bpf_get_prandom_u32];"
+ "r0 &= 1;"
+ "r0 <<= 3;"
+ "r6 = tbl_%= ll;"
+ "r6 += r0;"
+ "r6 = *(u64 *)(r6 + 0);"
+ "r7 = 0;"
+ "r1 = 0;"
+ "l_%=:"
+ "callx r6;"
+ "r7 += 1;"
+ "r1 = r7;"
+ "if r7 < 3 goto l_%=;"
+ "r0 = r7;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32)
+ : __clobber_all);
+}
+
+__naked __noinline __used
+static unsigned long loop_cb_set_flag(void)
+{
+ asm volatile (
+ "r1 = 1;"
+ "*(u64 *)(r2 + 0) = r1;"
+ "r0 = 0;"
+ "exit;"
+ );
+}
+
+/* the same for the helper that needs to know the callback, and that the callback comes back to */
+SEC("socket")
+__failure __msg("frame pointer is read only")
+__naked void callx_rodata_callback_one_of_two_at_loop_head(void)
+{
+ asm volatile (
+ FUNC_TABLE2(cbs, loop_cb_set_flag, loop_cb_go_on)
+ "call %[bpf_get_prandom_u32];"
+ "r0 &= 1;"
+ "r0 <<= 3;"
+ "r6 = cbs_%= ll;"
+ "r6 += r0;"
+ "r6 = *(u64 *)(r6 + 0);"
+ "r7 = 0;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = 1;"
+ "r2 = r6;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "l_%=:"
+ "call %[bpf_loop];"
+ "r1 = *(u64 *)(r10 - 8);"
+ "if r1 == 0 goto +2;"
+ "if r7 != 3 goto +1;"
+ "r10 = 0;"
+ "r7 += 1;"
+ "r1 = 0;"
+ "*(u64 *)(r10 - 8) = r1;"
+ "r1 = r7;"
+ "r2 = r6;"
+ "r3 = r10;"
+ "r3 += -8;"
+ "r4 = 0;"
+ "if r7 < 4 goto l_%=;"
+ /* libbpf resolves the pointers for progs that have callx */
+ "r2 = r10;"
+ "r2 += -8;"
+ "callx r6;"
+ "r0 = 0;"
+ "exit;"
+ :
+ : __imm(bpf_get_prandom_u32), __imm(bpf_loop)
+ : __clobber_all);
+}
+
/* and the data of the element that the index selects can be read */
SEC("socket")
__success __retval(16)
--
2.55.0
^ permalink raw reply related [flat|nested] 144+ messages in thread* Re: [PATCH bpf 81/81] selftests/bpf: Test callx of one of a set of functions
2026-10-26 2:59 ` [PATCH bpf 81/81] selftests/bpf: Test callx of one of a set of functions Alexei Starovoitov
@ 2026-10-26 3:53 ` bot+bpf-ci
0 siblings, 0 replies; 144+ messages in thread
From: bot+bpf-ci @ 2026-10-26 3:53 UTC (permalink / raw)
To: alexei.starovoitov, bpf
Cc: daniel, andrii, eddyz87, memxor, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 1949 bytes --]
> selftests/bpf: Test callx of one of a set of functions
TL;DR: Test nit only: callx_rodata_one_of_two() duplicates
callx_rodata_var_index(); could the log-level and R2=func(one_of=2) message
checks go on the existing test instead?
> diff --git a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
> index 85141cd73298..2575313a977a 100644
> --- a/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
> +++ b/tools/testing/selftests/bpf/progs/verifier_callx_rodata.c
[ ... ]
> @@ -173,6 +173,226 @@ __naked void callx_rodata_array_of_structs(void)
> : __clobber_all);
> }
>
> +/* one state with one of two functions, and not two states */
> +SEC("socket")
> +__success __retval(11)
> +__log_level(2)
> +__msg("r2 = *(u64 *)(r2 +0) ; R2=func(one_of=2)")
> +__naked void callx_rodata_one_of_two(void)
> +{
> + asm volatile (
> + FUNC_TABLE2(tbl, add1, add2)
> + "call %[bpf_get_prandom_u32];"
> + "r6 = r0;"
> + "r6 &= 1;"
> + "r3 = r6;"
> + "r3 <<= 3;"
> + "r2 = tbl_%= ll;"
> + "r2 += r3;"
> + "r2 = *(u64 *)(r2 + 0);"
> + "r1 = 10;"
> + "callx r2;"
> + "r0 -= r6;"
> + "exit;"
> + :
> + : __imm(bpf_get_prandom_u32)
> + : __clobber_all);
> +}
This isn't a bug, but callx_rodata_one_of_two() has the same body as the
existing callx_rodata_var_index() in this file: the same
FUNC_TABLE2(tbl, add1, add2) table, the same instructions, and the same
__retval(11).
The only addition is the __log_level(2) and __msg("... R2=func(one_of=2)")
check, so the file now contains two copies of one program.
Could the __log_level(2) and __msg() annotations go on
callx_rodata_var_index() instead of adding a copy?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/40263142572
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (80 preceding siblings ...)
2026-10-26 2:59 ` [PATCH bpf 81/81] selftests/bpf: Test callx of one of a set of functions Alexei Starovoitov
@ 2026-10-26 16:57 ` Ihor Solodrai
2026-10-26 23:53 ` Alexei Starovoitov
2026-10-26 22:28 ` Ihor Solodrai
82 siblings, 1 reply; 144+ messages in thread
From: Ihor Solodrai @ 2026-10-26 16:57 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: daniel, andrii, eddyz87, memxor
On 2026-10-25 7:58 p.m., Alexei Starovoitov wrote:
> From: Alexei Starovoitov <ast@kernel.org>
>
> The goal of this series is to load Rust programs as rustc and LLVM compile
> them. These programs keep data, heap and address-taken locals in arena.
> They use trait objects, recursion, unwinding, and loops that have no known
> bound. The test programs are in https://github.com/4ast/rust-bpf
>
> The series has three parts:
>
> patches 1-15 Fixes and a revert. They do not depend on the other
> parts.
> patches 16-31 A stack in arena.
> patches 32-81 A verifier that joins states.
Hi Alexei,
Just FYI bpf-ci-bot AI review workflow is limited to 50 patches per series.
The limit was set as a guardrail for commit selection logic (to not
accidentally run review for 10k commits). I chose 50 because usually
we ask people to break up long series into smaller reviewable pieces,
and pre-singulariy >10 patches series would already raise questions.
Since the approach to patch review on bpf list has obviously changed
recently, what would you suggest as a new limit?
>
> Only patches 1-15 are for merging now, into the bpf tree. Parts 2 and 3 are
> for review. The series applies to the bpf tree.
>
> Parts 2 and 3 can be reviewed separately. They touch each other only in
> patches 51, 53 and 80.
>
> [...]
>
^ permalink raw reply [flat|nested] 144+ messages in thread* Re: [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states
2026-10-26 16:57 ` [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Ihor Solodrai
@ 2026-10-26 23:53 ` Alexei Starovoitov
2026-10-27 0:04 ` Ihor Solodrai
0 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 23:53 UTC (permalink / raw)
To: Ihor Solodrai, bpf; +Cc: daniel, andrii, eddyz87, memxor
On Mon Oct 26, 2026 at 4:57 PM UTC, Ihor Solodrai wrote:
> On 2026-10-25 7:58 p.m., Alexei Starovoitov wrote:
>> From: Alexei Starovoitov <ast@kernel.org>
>>
>> The goal of this series is to load Rust programs as rustc and LLVM compile
>> them. These programs keep data, heap and address-taken locals in arena.
>> They use trait objects, recursion, unwinding, and loops that have no known
>> bound. The test programs are in https://github.com/4ast/rust-bpf
>>
>> The series has three parts:
>>
>> patches 1-15 Fixes and a revert. They do not depend on the other
>> parts.
>> patches 16-31 A stack in arena.
>> patches 32-81 A verifier that joins states.
>
> Hi Alexei,
>
> Just FYI bpf-ci-bot AI review workflow is limited to 50 patches per series.
>
> The limit was set as a guardrail for commit selection logic (to not
> accidentally run review for 10k commits). I chose 50 because usually
> we ask people to break up long series into smaller reviewable pieces,
> and pre-singulariy >10 patches series would already raise questions.
>
> Since the approach to patch review on bpf list has obviously changed
> recently, what would you suggest as a new limit?
I didn't know that there is such limit. I'm not sure there is a good limit.
Maybe whitelist it by known developers with commits?
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states
2026-10-26 23:53 ` Alexei Starovoitov
@ 2026-10-27 0:04 ` Ihor Solodrai
2026-10-27 0:14 ` Alexei Starovoitov
0 siblings, 1 reply; 144+ messages in thread
From: Ihor Solodrai @ 2026-10-27 0:04 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: daniel, andrii, eddyz87, memxor
On 2026-10-26 4:53 p.m., Alexei Starovoitov wrote:
> On Mon Oct 26, 2026 at 4:57 PM UTC, Ihor Solodrai wrote:
>> On 2026-10-25 7:58 p.m., Alexei Starovoitov wrote:
>>> From: Alexei Starovoitov <ast@kernel.org>
>>>
>>> The goal of this series is to load Rust programs as rustc and LLVM compile
>>> them. These programs keep data, heap and address-taken locals in arena.
>>> They use trait objects, recursion, unwinding, and loops that have no known
>>> bound. The test programs are in https://github.com/4ast/rust-bpf
>>>
>>> The series has three parts:
>>>
>>> patches 1-15 Fixes and a revert. They do not depend on the other
>>> parts.
>>> patches 16-31 A stack in arena.
>>> patches 32-81 A verifier that joins states.
>>
>> Hi Alexei,
>>
>> Just FYI bpf-ci-bot AI review workflow is limited to 50 patches per series.
>>
>> The limit was set as a guardrail for commit selection logic (to not
>> accidentally run review for 10k commits). I chose 50 because usually
>> we ask people to break up long series into smaller reviewable pieces,
>> and pre-singulariy >10 patches series would already raise questions.
>>
>> Since the approach to patch review on bpf list has obviously changed
>> recently, what would you suggest as a new limit?
>
> I didn't know that there is such limit. I'm not sure there is a good limit.
> Maybe whitelist it by known developers with commits?
The purpose of the limit is to guard against unintentional blow up of
compute and token cost. It's not infinite.
Are there plans for 10k-patch series?
I'm sure we can come up with a reasonable limit for a "reviewable"
series.
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states
2026-10-27 0:04 ` Ihor Solodrai
@ 2026-10-27 0:14 ` Alexei Starovoitov
2026-10-27 5:08 ` Ihor Solodrai
0 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-27 0:14 UTC (permalink / raw)
To: Ihor Solodrai, bpf; +Cc: daniel, andrii, eddyz87, memxor
On Tue Oct 27, 2026 at 12:04 AM UTC, Ihor Solodrai wrote:
> On 2026-10-26 4:53 p.m., Alexei Starovoitov wrote:
>> On Mon Oct 26, 2026 at 4:57 PM UTC, Ihor Solodrai wrote:
>>> On 2026-10-25 7:58 p.m., Alexei Starovoitov wrote:
>>>> From: Alexei Starovoitov <ast@kernel.org>
>>>>
>>>> The goal of this series is to load Rust programs as rustc and LLVM compile
>>>> them. These programs keep data, heap and address-taken locals in arena.
>>>> They use trait objects, recursion, unwinding, and loops that have no known
>>>> bound. The test programs are in https://github.com/4ast/rust-bpf
>>>>
>>>> The series has three parts:
>>>>
>>>> patches 1-15 Fixes and a revert. They do not depend on the other
>>>> parts.
>>>> patches 16-31 A stack in arena.
>>>> patches 32-81 A verifier that joins states.
>>>
>>> Hi Alexei,
>>>
>>> Just FYI bpf-ci-bot AI review workflow is limited to 50 patches per series.
>>>
>>> The limit was set as a guardrail for commit selection logic (to not
>>> accidentally run review for 10k commits). I chose 50 because usually
>>> we ask people to break up long series into smaller reviewable pieces,
>>> and pre-singulariy >10 patches series would already raise questions.
>>>
>>> Since the approach to patch review on bpf list has obviously changed
>>> recently, what would you suggest as a new limit?
>>
>> I didn't know that there is such limit. I'm not sure there is a good limit.
>> Maybe whitelist it by known developers with commits?
>
> The purpose of the limit is to guard against unintentional blow up of
> compute and token cost. It's not infinite.
> Are there plans for 10k-patch series?
yes. they already exist.
> I'm sure we can come up with a reasonable limit for a "reviewable"
> series.
There are not reviewable by current infra.
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states
2026-10-27 0:14 ` Alexei Starovoitov
@ 2026-10-27 5:08 ` Ihor Solodrai
0 siblings, 0 replies; 144+ messages in thread
From: Ihor Solodrai @ 2026-10-27 5:08 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: daniel, andrii, eddyz87, memxor
On 2026-10-26 5:14 p.m., Alexei Starovoitov wrote:
> On Tue Oct 27, 2026 at 12:04 AM UTC, Ihor Solodrai wrote:
>> On 2026-10-26 4:53 p.m., Alexei Starovoitov wrote:
>>> On Mon Oct 26, 2026 at 4:57 PM UTC, Ihor Solodrai wrote:
>>>> On 2026-10-25 7:58 p.m., Alexei Starovoitov wrote:
>>>>> From: Alexei Starovoitov <ast@kernel.org>
>>>>>
>>>>> The goal of this series is to load Rust programs as rustc and LLVM compile
>>>>> them. These programs keep data, heap and address-taken locals in arena.
>>>>> They use trait objects, recursion, unwinding, and loops that have no known
>>>>> bound. The test programs are in https://github.com/4ast/rust-bpf
>>>>>
>>>>> The series has three parts:
>>>>>
>>>>> patches 1-15 Fixes and a revert. They do not depend on the other
>>>>> parts.
>>>>> patches 16-31 A stack in arena.
>>>>> patches 32-81 A verifier that joins states.
>>>>
>>>> Hi Alexei,
>>>>
>>>> Just FYI bpf-ci-bot AI review workflow is limited to 50 patches per series.
>>>>
>>>> The limit was set as a guardrail for commit selection logic (to not
>>>> accidentally run review for 10k commits). I chose 50 because usually
>>>> we ask people to break up long series into smaller reviewable pieces,
>>>> and pre-singulariy >10 patches series would already raise questions.
>>>>
>>>> Since the approach to patch review on bpf list has obviously changed
>>>> recently, what would you suggest as a new limit?
>>>
>>> I didn't know that there is such limit. I'm not sure there is a good limit.
>>> Maybe whitelist it by known developers with commits?
>>
>> The purpose of the limit is to guard against unintentional blow up of
>> compute and token cost. It's not infinite.
>> Are there plans for 10k-patch series?
>
> yes. they already exist.
It doesn't look like you are joking, so let's consider it seriously.
I don't want to come off as too skeptical or critical. I'm just trying
to be rational. These are difficult problems and there is a lot of
uncertainty. So please bear with me.
For the sake of argument, let's imagine a 10k-patch series that fixes
multiple important problems which have to be addressed urgently.
What is the logistics of landing it? Is it going to be posted to the
mailing list?
Is it expected for people to read the patches? (my guess is no, it's
too slow)
Is it expected to get independent AI assessments of every patch?
Sub-sequences? Which ones? Only the whole series?
Can today's SOTA AI "agent" systems even do that - analyse entire
10k-patch series in one go without dropping any balls? Let's not
speculate about future better AI, the series has to land *now*, no
time to wait.
And remember, the same AI with same root prompt can produce different
results, depending on many factors. Should we run k-of-n or something?
How long does it run? Does the infra to execute this exist?
How much does it cost? Who pays the bills?
How many independent AI-driven assessments should be done before the
whole chain of (Linux) command is satisfied with the results and
pulls the trigger? Pardon my unample metaphors.
Who and how assesses the assessments?
Is there enough confidence in today's AI capability, that these 10k
patches are landable in the real world and won't create more new
problems vs what they are fixing?
These questions are just from the top of my head.
There is of course an answer that kinda works: don't worry about most
of the above, trust the AI output (the agents that generated patches
*surely* tested them well) and just git push.
But this is a "leap of faith" approach, and it requires for the vast
majority of participants to be onboard, which doesn't seem to be the
case yet if ever.
If we are not doing the leap of faith, then the questions above need
thoughtful answers, before anything approaching 10k-patch series can
be seriously considered.
If we are, well let's be explicit about it and stop pretending to do
proper patch reviews (whether with AI or not).
>
>> I'm sure we can come up with a reasonable limit for a "reviewable"
>> series.
>
> There are not reviewable by current infra.
Correct. Current testing / review infra is not able to support regular
100+ patches, let alone 10k. And that's bpf, which has a pretty good
infra in comparison to other corners of the kernel.
The questions are: how does the new infra look like, who builds it,
and who pays for it? Is it even needed? As of today there are no
answers.
A proof of personhood in text communication hasn't been invented yet,
so I'll do what I can:
Typed-by: Ihor Solodrai <ihor.solodrai@linux.dev>
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states
2026-10-26 2:58 [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Alexei Starovoitov
` (81 preceding siblings ...)
2026-10-26 16:57 ` [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states Ihor Solodrai
@ 2026-10-26 22:28 ` Ihor Solodrai
2026-10-26 23:45 ` Alexei Starovoitov
82 siblings, 1 reply; 144+ messages in thread
From: Ihor Solodrai @ 2026-10-26 22:28 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: daniel, andrii, eddyz87, memxor
On 2026-10-25 7:58 p.m., Alexei Starovoitov wrote:
> From: Alexei Starovoitov <ast@kernel.org>
>
> The goal of this series is to load Rust programs as rustc and LLVM compile
> them. These programs keep data, heap and address-taken locals in arena.
> They use trait objects, recursion, unwinding, and loops that have no known
> bound. The test programs are in https://github.com/4ast/rust-bpf
>
> The series has three parts:
>
> patches 1-15 Fixes and a revert. They do not depend on the other
> parts.
> patches 16-31 A stack in arena.
> patches 32-81 A verifier that joins states.
>
> Only patches 1-15 are for merging now, into the bpf tree. Parts 2 and 3 are
> for review. The series applies to the bpf tree.
>
> Parts 2 and 3 can be reviewed separately. They touch each other only in
> patches 51, 53 and 80.
Alexei, everyone,
This series shouldn't land as is. It's not ready.
It breaks a bunch of production BPF programs, and may have security
issues which Opus 5.5 and GPT 6.1 refuse to investigate.
pw-bot: cr
Alexei, a couple of suggestions that I think will help to iterate
faster while keeping the changes well tested:
* split up the series into independent parts of reasonable size: 80+
patches and 20k+ diff is difficult to process for AI, and
impossible for humans
* get someone with cybersecurity-blessed LLM access involved into
the review / development process: we definitely don't want to ship
new bugs that common AIs miss or refuse to find, even if they are
not "real" security bugs
* ask your development bots to ensure that verification of prod BPF
programs (bots can pull those tested by BPF CI from github
artifacts, just ask) has not regressed, and that there is a path
for updating existing programs to work both with old and new
verifier
Pasting below Meta progs regressions, top 10 issues found and confirmed
by my "agent swarm", and a list of "forbidden" topics.
I can share the full list of issues, code and giant .md reports too,
if anyone is interested let me know.
Thanks Meta for paying for the tokens.
======== PASTE #1: Meta prod BPF programs regressions
LOAD FAILURES REPORTED BY CI
These six production programs change from success to failure.
Independent review reproduced each on fixed objects and identified patch
53 as first bad.
Program Family / object
bpfj_jailer_scan bpfjailer / jailer.bpf.o
jailer_cgroup_attach_task bpfjailer / jailer.bpf.o
bpfj_fs_file_open bpfjailer / fs_enforce.bpf.o
syar_file_open sysarmor / security_file_open.bpf.o
armr_filemod_security_inode_link metarmor / filemod.bpf.o
tcdecapinfo katran / tc_decap_info.o
ADDITIONAL CONFIGURATION-DEPENDENT LOAD FAILURES
Both programs below are present in the same CI results and pass with
CI's default .rodata values. Independent review confirmed
success-to-failure regressions when the relevant non-default .rodata
options are enabled:
- bpfj_base_role_assign (bpfjailer / jailer.bpf.o)
- bpfj_jailer_exec (bpfjailer / jailer.bpf.o)
VERIFICATION COST REGRESSIONS
The following eleven production programs are also present in that CI
run. They continue to load. Cost regressions were independently
confirmed in controlled measurements; CI's comparison records verdicts
and state counts, so the timing figures below come from non-debug
review runs.
bpfjailer / net_enforce.bpf.o:
- enforce_security_socket_recvmsg about 2x slower
- enforce_security_socket_recvmsg_sleepable about 2x slower
- enforce_security_socket_sendmsg about 1.4x slower
- enforce_security_socket_sendmsg_sleepable about 1.5x slower
- enforce_security_socket_connect about 1.5x slower
- enforce_security_socket_connect_sleepable about 1.4x slower
bpfjailer / sock_addr_redirect.bpf.o:
- bpfj_redirect_socket_connect about 1.3x slower
strobelight / strobelight_process_monitor.o:
- tracepoint__syscalls_exit_mmap
- tracepoint__syscalls_enter_exit
- tracepoint__syscalls_enter_exit_group
All three increase from a few hundred processed insns to about 22.8K.
exit_mmap and enter_exit were timed at about 12x slower verification.
bpfjailer / exec_enforce.bpf.o:
- bpfj_exec_bprm_creds_for_exec about 1.1x slower
======== END OF PASTE #1
======== PASTE #2: Top 10 issues
Top 10 issues: "bpf: Stack in arena and a verifier that joins states"
Patch numbers refer to the 2026-10-26, 81-patch posting.
All ten findings were upheld by independent review.
1. Patch 7 overwrites the second return register of aggregate returns.
In kernel/bpf/fixups.c, the subprog-exit may_goto budget update
assumes R1-R3 are dead and loads the frame count into R2. A static
subprog returning a 9-16-byte struct uses R0:R2, so callers receive
a value inconsistent with the verified state. Confirmed outcomes
include wrong results, KASAN out-of-bounds writes and a fault
through a corrupted pointer. From patch 53, automatically guarded
plain C loops trigger this too (clang 23).
2. Part 1 leaves a reachable out-of-bounds bug fixed by patch 46. On
the base and after patch 15, maybe_fork_scalars() pushes an AND/OR
fork to the same instruction;
kernel/bpf/states.c:bpf_is_state_visited() can prune that fork
against an unfinished state using RANGE_WITHIN inside a
SCEV-terminating loop. Both loop-header and body checkpoints
reproduce out-of-bounds map-value writes on the base and after
patch 15. Patch 46 rejects them, but is outside the proposed
merge-now subset. The added test is
and_scalar_fork_at_long_loop_head_rejects_oob. This predates the
series.
3. Patch 18 truncates kernel pointers on JITs requiring explicit zero
extension. kernel/bpf/fixups.c:bpf_lower_cast_mov() lowers rX =
addr_space_cast(rY,0,1) to a 64-bit move while retaining zext_dst.
bpf_opt_subreg_zext_lo32_rnd_hi32() then inserts wX = wX,
discarding the pointer's upper half before dereference. This
affects different-register casts on s390x/riscv64/ppc64. The
instruction sequence appears in real s390x CI; enabling needs_zext
in the verification harness reproduced a crash.
4. Iterator-based termination detection accepts an unguarded infinite
loop. kernel/bpf/flow.c:flow_is_bound() treats a dominating
iter_next as bounding a loop without establishing that the iterator
was created outside it. do { bpf_for(i,0,1000) { ... } } while
(v->x) is rejected before the switch, but accepted with zero
may_goto guards afterwards and causes a soft lockup. The related
flow_iters_are_remade() heuristic also mistakes nine distinct entry
iterator IDs for recreation: eight sequential valid split-state
bpf_for loops exhaust verification restarts and fail to load.
5. Patches 51/53 admit recursion where the shared runtime budget is
unavailable. The recursion admission check in
kernel/bpf/verifier.c requires only bpf_jit_supports_percpu_insn();
budget setup in fixups.c additionally needs timed_may_goto and
exception support. On x86 with CONFIG_UNWINDER_FRAME_POINTER=y, a
loop around recursive calls, with no explicit may_goto, loads but
soft-locks instead of terminating; runs remained active beyond 187
seconds. riscv64 and loongarch have the same capability
mismatch. The documented 8M-call limit is insufficient for this
shape.
6. Newly admitted loops and recursion are unsafe for hardware offload.
From patch 53, offloaded uncounted loops and recursive subprogs
load, while their inserted guards/counters protect only the host
instruction stream. netdevsim confirmed both acceptance
changes. In drivers/net/ethernet/netronome/nfp/bpf/verifier.c,
nfp_bpf_get_stack_usage() assumes recursion was rejected and walks
calls through fixed MAX_CALL_FRAMES arrays. Patch 50's
bpf_insert_loop_guards() also shifts aux indices before offload
finalization; NFP still indexes zext flags by the original
instruction position. Actual NFP execution/crash consequences were
not tested.
7. Patch 53's uncounted-loop fallback rejects previously valid bounded
loops. kernel/bpf/verifier.c:bpf_flow_loop_enter() stops walking
at an undecided exit or the 4096-insn budget; calls/iterators can
also prevent walking. bpf_scev_forget_regs()/widening then discard
bounds needed for accesses. Reproducer shapes include multiple
backedges, duplicated constant exits, flag latches, loop-carried
induction-variable copies and bpf_for with a 32-bit spilled
counter. Seven confirmed load regressions: bpfj_jailer_scan,
jailer_cgroup_attach_task, bpfj_fs_file_open, syar_file_open,
test_select_dst_max, test_select_dst_max_boundary, and
on_perf_event in torchscriptstacks_sample.o. All load before patch
53 and fail afterwards.
8. Joining constant offsets loses facts required by otherwise valid
accesses. kernel/bpf/flow.c's replacement for precision tracking
lacks rules for non-stack pointer addends and constant array-map
keys. tcdecapinfo loads before patch 53 but fails afterwards:
joining two constant packet offsets breaks range propagation in
find_good_pkt_pointers(). Independent plain-C reproducers also
fail: selecting t->pid versus t->tgid through a trusted BTF pointer
(clang 23), and selecting array-map key 0 or 1 before a lookup
whose constant-key proof previously made a NULL check unnecessary.
9. The 16-join heuristic widens states even in acyclic code.
kernel/bpf/flow.c:flow_merge() sets widen_all when s->joins exceeds
BPF_FLOW_MAX_JOINS, bypassing NULL, initialization, constant and
pointer offset distinctions. armr_filemod_security_inode_link is a
confirmed patch-53 load regression. No loop is required: a static
helper called once with v+32 and from 18 sites with v+0 fails with
an unbounded pointer offset; the 17-site control loads. This cutoff
is absent from verifier.rst's description of when states remain
separate.
10. Scanning states makes verification much slower despite fewer
processed insns. kernel/bpf/flow.c:flow_merge()/flow_within()
repeatedly scan resident states through bpf_state_subsumes(). On
non-debug kernels, vip_filter takes about 4.5 times longer while
processing 24% fewer insns. A 1029-insn synthetic with 1024
distinct offset states, 12 merge points and 64 live stack slots
took 40.6 seconds; the base rejected it in 0.67 seconds. Patch 53
removes the processed-insn budget; remaining state-count limits do
not bound the comparisons proportionally. Loads remain
interruptible. KASAN timings obscure the regression.
======== END OF PASTE #2
======== PASTE #3: Topics blocked by LLM guardrails
And the list of issues the bots refused to look into:
1. Run-time recursion checks (patches 51, 53): whether the stack
charged for recursive frames bounds the real kernel and arena
stack use, how the per-run call budget is enforced, and how both
behave when runs nest.
2. Join (patches 47, 66, 68): whether bpf_state_subsumes(), which
decides that a joined state covers both inputs, checks every
register and stack-slot property that matters, including the
function sets of patch 68.
3. Main pass (patches 52/53): whether verifying a static subprogram
once per entry state, rather than per call chain, keeps all the
facts its callers rely on, and how tail calls interact with
5. Stack in arena (patches 16-31): memory-safety properties of r12,
the per-cpu arena stacks, BPF_ATOMIC_LOCAL, the JIT changes and
the new arena kfuncs, including how arena stack frames are
released on early exits.
6. Removals and raised limits (patches 32, 54, 56, 58, 60-67):
whether any remaining safety check depended on the removed
precision tracking, pruning or loop detection, or on the old
frame, stack-argument and subprogram limits.
7. Worst-case verification cost (patches 52/53, 60): CPU time and
memory the verifier can be made to spend under the new state
limits, now that the processed-insn and pending-jump limits are
gone.
8. Function tables and callx (patches 42, 68): whether every
function in a tracked set is verified for each indirect call or
helper/kfunc argument it reaches, with up to 1024 targets per
table.
9. Packet pointer range (patch 40): whether keeping a packet
pointer's range after adding a non-negative scalar is correct in
every case the patch allows.
10. Unprivileged programs (patches 47, 52/53): whether the Spectre
mitigations still hold when states are joined and the main pass
is a fixpoint.
11. Architectures other than x86-64: how the per-run budget,
unwinding and recursion checks (patches 7, 43, 51) and the arm64
r12 support (patch 28) behave on other JITs.
======== END OF PASTE #3
>
> [...]
>
^ permalink raw reply [flat|nested] 144+ messages in thread* Re: [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states
2026-10-26 22:28 ` Ihor Solodrai
@ 2026-10-26 23:45 ` Alexei Starovoitov
2026-10-27 0:01 ` Ihor Solodrai
0 siblings, 1 reply; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-26 23:45 UTC (permalink / raw)
To: Ihor Solodrai, bpf; +Cc: daniel, andrii, eddyz87, memxor
On Mon, Oct 26, 2026 at 03:28 PM Ihor Solodrai <ihor.solodrai@linux.dev> wrote:
> This series shouldn't land as is. It's not ready.
your bot didn't read the cover letter :(
that said that parts 2 and 3 were posted for review only
Only Part 1 is for landing, but parts 2 and 3 depend on it.
> * ask your development bots to ensure that verification of prod BPF
> programs (bots can pull those tested by BPF CI from github
> artifacts, just ask) has not regressed, and that there is a path
> for updating existing programs to work both with old and new
> verifier
your bot is wrong again. meta progs are not public.
So test on meta progs the whole set had to go as a whole.
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states
2026-10-26 23:45 ` Alexei Starovoitov
@ 2026-10-27 0:01 ` Ihor Solodrai
2026-10-27 0:13 ` Ihor Solodrai
2026-10-27 0:16 ` Alexei Starovoitov
0 siblings, 2 replies; 144+ messages in thread
From: Ihor Solodrai @ 2026-10-27 0:01 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: daniel, andrii, eddyz87, memxor
On 2026-10-26 4:45 p.m., Alexei Starovoitov wrote:
> On Mon, Oct 26, 2026 at 03:28 PM Ihor Solodrai <ihor.solodrai@linux.dev> wrote:
>> This series shouldn't land as is. It's not ready.
>
> your bot didn't read the cover letter :(
> that said that parts 2 and 3 were posted for review only
> Only Part 1 is for landing, but parts 2 and 3 depend on it.
Why not post part 1, land it and then post parts 2-3 as RFC?
The review was done for the full series, that's the whole point of
posting the series, no?
>
>> * ask your development bots to ensure that verification of prod BPF
>> programs (bots can pull those tested by BPF CI from github
>> artifacts, just ask) has not regressed, and that there is a path
>> for updating existing programs to work both with old and new
>> verifier
>
> your bot is wrong again. meta progs are not public.
That was my mistake, I thought we publish the objects as artifacts.
I'll see if can do that.
But even if they don't become public, we run veristat on them as part of
upstream BPF CI, and those signals shouldn't be ignored.
Also bpfjailer was open sourced recently:
https://github.com/facebookincubator/bpfjailer
> So test on meta progs the whole set had to go as a whole.
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states
2026-10-27 0:01 ` Ihor Solodrai
@ 2026-10-27 0:13 ` Ihor Solodrai
2026-10-27 0:16 ` Alexei Starovoitov
1 sibling, 0 replies; 144+ messages in thread
From: Ihor Solodrai @ 2026-10-27 0:13 UTC (permalink / raw)
To: Alexei Starovoitov, bpf; +Cc: daniel, andrii, eddyz87, memxor
On 2026-10-26 5:01 p.m., Ihor Solodrai wrote:
> On 2026-10-26 4:45 p.m., Alexei Starovoitov wrote:
>> On Mon, Oct 26, 2026 at 03:28 PM Ihor Solodrai
>> <ihor.solodrai@linux.dev> wrote:
>>> This series shouldn't land as is. It's not ready.
>>
>> your bot didn't read the cover letter :(
>> that said that parts 2 and 3 were posted for review only
>> Only Part 1 is for landing, but parts 2 and 3 depend on it.
>
> Why not post part 1, land it and then post parts 2-3 as RFC?
> The review was done for the full series, that's the whole point of
> posting the series, no?
>
>>
>>> * ask your development bots to ensure that verification of prod BPF
>>> programs (bots can pull those tested by BPF CI from github
>>> artifacts, just ask) has not regressed, and that there is a path
>>> for updating existing programs to work both with old and new
>>> verifier
>>
>> your bot is wrong again. meta progs are not public.
>
> That was my mistake, I thought we publish the objects as artifacts.
> I'll see if can do that.
>
> But even if they don't become public, we run veristat on them as part of
> upstream BPF CI, and those signals shouldn't be ignored.
And by the way, just in case you're not aware: you can trigger a full CI
run with veristat jobs and AI reviews in kernel-patches/bpf github repo
if you have a write access there (I believe you do, lmk if not).
Just push a branch to kernel-patches/bpf repo and open a PR against
bpf_base. This should be enough for the bots to iterate.
>
> Also bpfjailer was open sourced recently:
> https://github.com/facebookincubator/bpfjailer
>> So test on meta progs the whole set had to go as a whole.
>
^ permalink raw reply [flat|nested] 144+ messages in thread
* Re: [PATCH bpf 00/81] bpf: Stack in arena and a verifier that joins states
2026-10-27 0:01 ` Ihor Solodrai
2026-10-27 0:13 ` Ihor Solodrai
@ 2026-10-27 0:16 ` Alexei Starovoitov
1 sibling, 0 replies; 144+ messages in thread
From: Alexei Starovoitov @ 2026-10-27 0:16 UTC (permalink / raw)
To: Ihor Solodrai, bpf; +Cc: daniel, andrii, eddyz87, memxor
On Tue Oct 27, 2026 at 12:01 AM UTC, Ihor Solodrai wrote:
> On 2026-10-26 4:45 p.m., Alexei Starovoitov wrote:
>> On Mon, Oct 26, 2026 at 03:28 PM Ihor Solodrai <ihor.solodrai@linux.dev> wrote:
>>> This series shouldn't land as is. It's not ready.
>>
>> your bot didn't read the cover letter :(
>> that said that parts 2 and 3 were posted for review only
>> Only Part 1 is for landing, but parts 2 and 3 depend on it.
>
> Why not post part 1, land it and then post parts 2-3 as RFC?
> The review was done for the full series, that's the whole point of
> posting the series, no?
To see the whole thing through CI and sashiko lenses.
>>
>>> * ask your development bots to ensure that verification of prod BPF
>>> programs (bots can pull those tested by BPF CI from github
>>> artifacts, just ask) has not regressed, and that there is a path
>>> for updating existing programs to work both with old and new
>>> verifier
>>
>> your bot is wrong again. meta progs are not public.
>
> That was my mistake, I thought we publish the objects as artifacts.
> I'll see if can do that.
>
> But even if they don't become public, we run veristat on them as part of
> upstream BPF CI, and those signals shouldn't be ignored.
No one said they're ignored.
^ permalink raw reply [flat|nested] 144+ messages in thread