* [PATCH v7 bpf-next 00/12] BPF indirect jumps
@ 2025-10-26 19:26 Anton Protopopov
2025-10-26 19:26 ` [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array Anton Protopopov
` (11 more replies)
0 siblings, 12 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:26 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
This patchset implements a new type of map, instruction set, and uses
it to build support for indirect branches in BPF (on x86). (The same
map will be later used to provide support for indirect calls and static
keys.) See [1], [2] for more context.
Short table of contents:
* Patches 1-4 implement the new map of type
BPF_MAP_TYPE_INSN_SET and corresponding selftests. This map can
be used to track the "original -> xlated -> jitted mapping" for
a given program. Patches 1,2 implement the main functionality,
patches 3,4 add support for the "blinded" variant.
* Patches 5-8 implement the support for indirect jumps on x86
* Patches 9-12 add libbpf support for LLVM-compiled programs
containing indirect jumps, and selftests.
Since the v3 the jump table support was merged to LLVM and now can be
enabled with -mcpu=v4. See [3] for the PR which added the support.
The selftests, however, don't use LLVM for now: the verifier_gotox
is implemented in asm, and bpf_gotox is actually turned off, as
CI only runs with LLVM 21, and the indirect jumps are supported
starting from 22.
See individual patches for more details on the implementation details.
v6 -> v7 (this series):
* rebased and dropped already merged commits
* instruction arrays
* use jit_data to find mappings from insn to jit (Alexei)
* alloc `ips` as part of the main allocation (Eduard)
* the `jitted_ip` member wasn't actually used (Eduard)
* remove the bpf_insn_ptr structure, which is not needed for this patch
* indirect jumps, kernel:
* fix a memory leak in `create_jt` (AI)
* use proper reg+8*ereg in `its_static_thunk` (AI)
* some minor cleanups (Eduard)
* indirect jumps, libbpf:
* refactor the `jt_adjust_off()` piece (Edurad)
* move "JUMPTABLES_SEC" into libbpf_internal.h (Eduard)
* remove an unnecessary if (Eduard)
* verifier_gotox: add tests to verify that `gotox rX` works with all registers
v5 -> v6 (https://lore.kernel.org/bpf/20251019202145.3944697-1-a.s.protopopov@gmail.com/T/#u):
* instruction arrays:
* better document `struct bpf_insn_array_value` (Eduard)
* remove a condition in `bpf_insn_array_adjust_after_remove` (Eduard)
* make userspace see original, xlated, and jitted indexes (+original) (Eduard)
* indirect jumps, kernel:
* reject writes to the map
* reject unaligned ops
* add a check what `w` is not outside the program in check_config for `gotox` (Eduard)
* do not introduce unneeded `bpf_find_containing_subprog_idx`
* simplify error processing for `bpf_find_containing_subprog` (Eduard)
* add `insn_state |= DISCOVERED` when it's discovered (Eduard)
* support SUB operations on PTR_TO_INSN (Eduard)
* make `gotox_tmp_buf` a bpf_iarray and use helper to relocate it (Eduard)
* rename fields of `bpf_iarray` to more generic (Eduard)
* re-implement `visit_gotox_insn` in a loop (Eduard)
* some minor cleanups (Eduard)
* libbpf:
* `struct reloc_desc`: add a comment about `union` (Eduard)
* rename parameters of (and one other place in code) `{create,add}_jt_map` to `sym_off` (Eduard)
* `create_jt_map`: check that size/off are 8-byte aligned (Eduard)
* Selftests:
* instruction array selftests:
* only run tests on x86_64
* write a more generic function to test things to reduce code (Eduard)
* errno wasn't used in checks, so don't reset it (Eduard)
* print `i`, `xlated_off` and `map_out[i]` here (Eduard)
* added `verifier_gotox` selftests which do not depend on LLVM:
* disabled `bpf_gotox` tests by default
* other changes:
* remove an extra function in bpf disasm (Eduard)
* some minor cleanups in the insn_successors patch (Eduard)
* update documentation in `Documentation/bpf/linux-notes.html` about jumps, now it is supported :)
v3 -> v4 -> v5 (https://lore.kernel.org/bpf/20250930125111.1269861-1-a.s.protopopov@gmail.com/):
* [v4 -> v5] rebased on top of the last bpf-next/master
* instruction arrays:
* add copyright (Alexei)
* remove mutexes, add frozen back (Alexei)
* setup 1:1 prog-map correspondence using atomic_xchg
* do not copy/paste array_map_get_next_key, add a common helper (Alexei)
* misc minor code cleanups (Alexei)
* indirect jumps, kernel side:
* remove jt_allocated, just check if insn is gotox (Eduard)
* use copy_register_state instead of individual copies (Eduard)
* in push_stack is_speculative should be inherited (Eduard)
* a few cleanups for insn_successors, including omitting error path (Eduard)
* check if reserved fields are used when considering `gotox` instruction (Eduard)
* read size and alignment of read from insn_array should be 8 (Eduard)
* put buffer for sorting in subfun info and realloc to grow as needed (Eduard)
* properly do `jump_point` / `prune_point` from `push_gotox_edge` (Eduard)
* use range_within to check states (Eduard)
* some minor cleanups and fix commit message (Eduard)
* indirect jumps, libbpf side:
* close map_fd in some error paths in create_jt_map (Andrii)
* maps for jump tables are actually not closed at all, fix this (Andrii)
* rename map from `jt` to `.jumptables` (Andrii)
* use `errstr` in an error message (Andrii)
* rephrase error message to look more standard (Andrii)
* misc other minor renames and cleanups (Andrii)
* selftests:
* add the frozen selftest back
* add a selftest for two jumps loading same table
* some other changes:
* rebase and split insn_successor changes into separate patch
* use PTR_ERR_OR_ZERO in the push stack patch (Eduard)
* indirect jumps on x86: properly re-read *pprog (Eduard)
v2 -> v3 (https://lore.kernel.org/bpf/20250918093850.455051-1-a.s.protopopov@gmail.com/):
* fix build failure when CONFIG_BPF_SYSCALL is not set (kbuild-bot)
* reformat bpftool help messages (Quentin)
v1 -> v2 (https://lore.kernel.org/bpf/20250913193922.1910480-1-a.s.protopopov@gmail.com/):
* push_stack changes:
* sanitize_speculative_path should just return int (Eduard)
* return code from sanitize_speculative_path, not EFAULT (Eduard)
* when BPF_COMPLEXITY_LIMIT_JMP_SEQ is reached, return E2BIG (Eduard)
* indirect jumps:
* omit support for .imm=fd in gotox, as we're not using it for now (Eduard)
* struct jt -> struct bpf_iarray (Eduard)
* insn_successors: rewrite the interface to just return a pointer (Eduard)
* remove min_index/max_index, use umin_value/umax_value instead (Alexei, Eduard)
* move emit_indirect_jump args change to the previous patch (Eduard)
* add a comment to map_mem_size() (Eduard)
* use verifier_bug for some error cases in check_indirect_jump (Eduard)
* clear_insn_aux_data: use start,len instead of start,end (Eduard)
* make regs[insn->dst_reg].type = PTR_TO_INSN part of check_mem_access (Eduard)
* constant blinding changes:
* make subprog_start adjustment better readable (Eduard)
* do not set subprog len, it is already set (Eduard)
* libbpf:
* remove check that relocations from .rodata are ok (Anton)
* do not freeze the map, it is not necessary anymore (Anton)
* rename the goto_x -> gotox everywhere (Anton)
* use u64 when parsing LLVM jump tables (Eduard)
* split patch in two due to spaces->tabs change (Eduard)
* split bpftool changes to bpftool patch (Andrii)
* make sym_size it a union with ext_idx (Andrii)
* properly copy/free the jumptables_data section from elf (Andrii)
* a few cosmetic changes around create_jt_map (Andrii)
* fix some comments + rewrite patch description (Andrii)
* inline bpf_prog__append_subprog_offsets (Andrii)
* subprog_sec_offst -> subprog_sec_off (Andrii)
* !strcmp -> strcmp() == 0 (Andrii)
* make some function names more readable (Andrii)
* allocate table of subfunc offsets via libbpf_reallocarray (Andrii)
* selftests:
* squash insn_array* tests together (Anton)
* fixed build warnings (kernel test robot)
RFC -> v1 (https://lore.kernel.org/bpf/20250816180631.952085-1-a.s.protopopov@gmail.com/):
* I've tried to address all the comments provided by Alexei and
Eduard in RFC. Will try to list the most important of them below.
* One big change: move from older LLVM version [5] to newer [4].
Now LLVM generates jump tables as symbols in the new special
section ".jumptables". Another part of this change is that
libbpf now doesn't try to link map load and goto *rX, as
1) this is absolutely not reliable 2) for some use cases this
is impossible (namely, when more than one jump table can be used
in the same gotox instruction).
* Added insn_successors() support (Alexei, Eduard). This includes
getting rid of the ugly bpf_insn_set_iter_xlated_offset()
interface (Eduard).
* Removed hack for the unreachable instruction, as new LLVM thank to
Eduard doesn't generate it.
* Set mem_size for direct map access properly instead of hacking.
Remove off>0 check. (Alexei)
* Do not allocate new memory for min_index/max_index (Alexei, Eduard)
* Information required during check_cfg is now cached to be reused
later (Alexei + general logic for supporting multiple JT per jump)
* Properly compare registers in regsafe (Alexei, Eduard)
* Remove support for JMP32 (Eduard)
* Better checks in adjust_ptr_min_max_vals (Eduard)
* More selftests were added (but still there's room for more) which
directly use gotox (Alexei)
* More checks and verbose messages added
* "unique pointers" are no more in the map
Links:
1. https://lpc.events/event/18/contributions/1941/
2. https://lwn.net/Articles/1017439/
3. https://github.com/llvm/llvm-project/pull/149715
4. https://github.com/llvm/llvm-project/pull/149715#issuecomment-3274833753
6. rfc: https://lore.kernel.org/bpf/20250615085943.3871208-1-a.s.protopopov@gmail.com/
Anton Protopopov (12):
bpf, x86: add new map type: instructions array
selftests/bpf: add selftests for new insn_array map
bpf: support instructions arrays with constants blinding
selftests/bpf: test instructions arrays with blinding
bpf, x86: allow indirect jumps to r8...r15
bpf, x86: add support for indirect jumps
bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X
bpf, docs: do not state that indirect jumps are not supported
libbpf: support llvm-generated indirect jumps
bpftool: Recognize insn_array map type
selftests/bpf: add new verifier_gotox test
selftests/bpf: add C-level selftests for indirect jumps
Documentation/bpf/linux-notes.rst | 8 -
arch/x86/net/bpf_jit_comp.c | 56 +-
include/linux/bpf.h | 20 +
include/linux/bpf_types.h | 1 +
include/linux/bpf_verifier.h | 11 +
include/uapi/linux/bpf.h | 21 +
kernel/bpf/Makefile | 2 +-
kernel/bpf/bpf_insn_array.c | 299 +++++++++++
kernel/bpf/core.c | 21 +
kernel/bpf/disasm.c | 3 +
kernel/bpf/liveness.c | 3 +
kernel/bpf/log.c | 1 +
kernel/bpf/syscall.c | 22 +
kernel/bpf/verifier.c | 418 ++++++++++++++-
.../bpf/bpftool/Documentation/bpftool-map.rst | 3 +-
tools/bpf/bpftool/map.c | 3 +-
tools/include/uapi/linux/bpf.h | 21 +
tools/lib/bpf/libbpf.c | 249 ++++++++-
tools/lib/bpf/libbpf_internal.h | 4 +
tools/lib/bpf/libbpf_probes.c | 4 +
tools/lib/bpf/linker.c | 9 +-
tools/testing/selftests/bpf/Makefile | 4 +-
.../selftests/bpf/prog_tests/bpf_gotox.c | 185 +++++++
.../selftests/bpf/prog_tests/bpf_insn_array.c | 499 ++++++++++++++++++
.../selftests/bpf/prog_tests/verifier.c | 2 +
tools/testing/selftests/bpf/progs/bpf_gotox.c | 402 ++++++++++++++
.../selftests/bpf/progs/verifier_gotox.c | 317 +++++++++++
27 files changed, 2558 insertions(+), 30 deletions(-)
create mode 100644 kernel/bpf/bpf_insn_array.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
create mode 100644 tools/testing/selftests/bpf/progs/bpf_gotox.c
create mode 100644 tools/testing/selftests/bpf/progs/verifier_gotox.c
--
2.34.1
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
@ 2025-10-26 19:26 ` Anton Protopopov
2025-10-26 20:12 ` Anton Protopopov
` (4 more replies)
2025-10-26 19:26 ` [PATCH v7 bpf-next 02/12] selftests/bpf: add selftests for new insn_array map Anton Protopopov
` (10 subsequent siblings)
11 siblings, 5 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:26 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
On bpf(BPF_PROG_LOAD) syscall user-supplied BPF programs are
translated by the verifier into "xlated" BPF programs. During this
process the original instructions offsets might be adjusted and/or
individual instructions might be replaced by new sets of instructions,
or deleted.
Add a new BPF map type which is aimed to keep track of how, for a
given program, the original instructions were relocated during the
verification. Also, besides keeping track of the original -> xlated
mapping, make x86 JIT to build the xlated -> jitted mapping for every
instruction listed in an instruction array. This is required for every
future application of instruction arrays: static keys, indirect jumps
and indirect calls.
A map of the BPF_MAP_TYPE_INSN_ARRAY type must be created with a u32
keys and value of size 8. The values have different semantics for
userspace and for BPF space. For userspace a value consists of two
u32 values – xlated and jitted offsets. For BPF side the value is
a real pointer to a jitted instruction.
On map creation/initialization, before loading the program, each
element of the map should be initialized to point to an instruction
offset within the program. Before the program load such maps should
be made frozen. After the program verification xlated and jitted
offsets can be read via the bpf(2) syscall.
If a tracked instruction is removed by the verifier, then the xlated
offset is set to (u32)-1 which is considered to be too big for a valid
BPF program offset.
One such a map can, obviously, be used to track one and only one BPF
program. If the verification process was unsuccessful, then the same
map can be re-used to verify the program with a different log level.
However, if the program was loaded fine, then such a map, being
frozen in any case, can't be reused by other programs even after the
program release.
Example. Consider the following original and xlated programs:
Original prog: Xlated prog:
0: r1 = 0x0 0: r1 = 0
1: *(u32 *)(r10 - 0x4) = r1 1: *(u32 *)(r10 -4) = r1
2: r2 = r10 2: r2 = r10
3: r2 += -0x4 3: r2 += -4
4: r1 = 0x0 ll 4: r1 = map[id:88]
6: call 0x1 6: r1 += 272
7: r0 = *(u32 *)(r2 +0)
8: if r0 >= 0x1 goto pc+3
9: r0 <<= 3
10: r0 += r1
11: goto pc+1
12: r0 = 0
7: r6 = r0 13: r6 = r0
8: if r6 == 0x0 goto +0x2 14: if r6 == 0x0 goto pc+4
9: call 0x76 15: r0 = 0xffffffff8d2079c0
17: r0 = *(u64 *)(r0 +0)
10: *(u64 *)(r6 + 0x0) = r0 18: *(u64 *)(r6 +0) = r0
11: r0 = 0x0 19: r0 = 0x0
12: exit 20: exit
An instruction array map, containing, e.g., instructions [0,4,7,12]
will be translated by the verifier to [0,4,13,20]. A map with
index 5 (the middle of 16-byte instruction) or indexes greater than 12
(outside the program boundaries) would be rejected.
The functionality provided by this patch will be extended in consequent
patches to implement BPF Static Keys, indirect jumps, and indirect calls.
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
---
arch/x86/net/bpf_jit_comp.c | 23 +++
include/linux/bpf.h | 19 +++
include/linux/bpf_types.h | 1 +
include/linux/bpf_verifier.h | 2 +
include/uapi/linux/bpf.h | 21 +++
kernel/bpf/Makefile | 2 +-
kernel/bpf/bpf_insn_array.c | 284 +++++++++++++++++++++++++++++++++
kernel/bpf/syscall.c | 22 +++
kernel/bpf/verifier.c | 43 +++++
tools/include/uapi/linux/bpf.h | 21 +++
10 files changed, 437 insertions(+), 1 deletion(-)
create mode 100644 kernel/bpf/bpf_insn_array.c
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d4c93d9e73e4..d8ee0c4d9af8 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -3648,6 +3648,22 @@ struct x64_jit_data {
struct jit_context ctx;
};
+struct insn_ptrs_data {
+ int *addrs;
+ u8 *image;
+};
+
+static void update_insn_ptr(void *jit_priv, u32 xlated_off, u32 *jitted_off, long *ip)
+{
+ struct insn_ptrs_data *data = jit_priv;
+
+ if (!data->addrs || !data->image || !jitted_off || !ip)
+ return;
+
+ *jitted_off = data->addrs[xlated_off];
+ *ip = (long)(data->image + *jitted_off);
+}
+
#define MAX_PASSES 20
#define PADDING_PASSES (MAX_PASSES - 5)
@@ -3658,6 +3674,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
struct bpf_prog *tmp, *orig_prog = prog;
void __percpu *priv_stack_ptr = NULL;
struct x64_jit_data *jit_data;
+ struct insn_ptrs_data insn_ptrs_data;
int priv_stack_alloc_sz;
int proglen, oldproglen = 0;
struct jit_context ctx = {};
@@ -3827,6 +3844,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
jit_data->header = header;
jit_data->rw_header = rw_header;
}
+
+ /* jit_data may not contain proper info, copy the required fields */
+ insn_ptrs_data.addrs = addrs;
+ insn_ptrs_data.image = image;
+ bpf_prog_update_insn_ptrs(prog, &insn_ptrs_data, update_insn_ptr);
+
/*
* ctx.prog_offset is used when CFI preambles put code *before*
* the function. See emit_cfi(). For FineIBT specifically this code
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index e53cda0aabb6..b64c2382fbb7 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -3789,4 +3789,23 @@ int bpf_prog_get_file_line(struct bpf_prog *prog, unsigned long ip, const char *
const char **linep, int *nump);
struct bpf_prog *bpf_prog_find_from_stack(void);
+int bpf_insn_array_init(struct bpf_map *map, const struct bpf_prog *prog);
+int bpf_insn_array_ready(struct bpf_map *map);
+void bpf_insn_array_release(struct bpf_map *map);
+void bpf_insn_array_adjust(struct bpf_map *map, u32 off, u32 len);
+void bpf_insn_array_adjust_after_remove(struct bpf_map *map, u32 off, u32 len);
+
+typedef void (*update_insn_ptr_func_t)(void *jit_priv, u32 xlated_off, u32 *jitted_off, long *ip);
+
+#ifdef CONFIG_BPF_SYSCALL
+void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
+ update_insn_ptr_func_t update_insn_ptr);
+#else
+static inline void
+bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
+ update_insn_ptr_func_t update_insn_ptr);
+{
+}
+#endif
+
#endif /* _LINUX_BPF_H */
diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h
index fa78f49d4a9a..b13de31e163f 100644
--- a/include/linux/bpf_types.h
+++ b/include/linux/bpf_types.h
@@ -133,6 +133,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_RINGBUF, ringbuf_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops)
BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops)
+BPF_MAP_TYPE(BPF_MAP_TYPE_INSN_ARRAY, insn_array_map_ops)
BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint)
BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing)
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index c6eb68b6389c..6b820d8d77af 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -754,8 +754,10 @@ struct bpf_verifier_env {
struct list_head free_list; /* list of struct bpf_verifier_state_list */
struct bpf_map *used_maps[MAX_USED_MAPS]; /* array of map's used by eBPF program */
struct btf_mod_pair used_btfs[MAX_USED_BTFS]; /* array of BTF's used by BPF program */
+ struct bpf_map *insn_array_maps[MAX_USED_MAPS]; /* array of INSN_ARRAY map's to be relocated */
u32 used_map_cnt; /* number of used maps */
u32 used_btf_cnt; /* number of used BTF objects */
+ u32 insn_array_map_cnt; /* number of used maps of type BPF_MAP_TYPE_INSN_ARRAY */
u32 id_gen; /* used to generate unique reg IDs */
u32 hidden_subprog_cnt; /* number of hidden subprogs */
int exception_callback_subprog;
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 6829936d33f5..805d441363cd 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -1026,6 +1026,7 @@ enum bpf_map_type {
BPF_MAP_TYPE_USER_RINGBUF,
BPF_MAP_TYPE_CGRP_STORAGE,
BPF_MAP_TYPE_ARENA,
+ BPF_MAP_TYPE_INSN_ARRAY,
__MAX_BPF_MAP_TYPE
};
@@ -7645,4 +7646,24 @@ enum bpf_kfunc_flags {
BPF_F_PAD_ZEROS = (1ULL << 0),
};
+/*
+ * Values of a BPF_MAP_TYPE_INSN_ARRAY entry must be of this type.
+ *
+ * Before the map is used the orig_off field should point to an
+ * instruction inside the program being loaded. The other fields
+ * must be set to 0.
+ *
+ * After the program is loaded, the xlated_off will be adjusted
+ * by the verifier to point to the index of the original instruction
+ * in the xlated program. If the instruction is deleted, it will
+ * be set to (u32)-1. The jitted_off will be set to the corresponding
+ * offset in the jitted image of the program.
+ */
+struct bpf_insn_array_value {
+ __u32 orig_off;
+ __u32 xlated_off;
+ __u32 jitted_off;
+ __u32 :32;
+};
+
#endif /* _UAPI__LINUX_BPF_H__ */
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 7fd0badfacb1..232cbc97434d 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -9,7 +9,7 @@ CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o liveness.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
-obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
+obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o bpf_task_storage.o
obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
obj-$(CONFIG_BPF_SYSCALL) += disasm.o mprog.o
diff --git a/kernel/bpf/bpf_insn_array.c b/kernel/bpf/bpf_insn_array.c
new file mode 100644
index 000000000000..f9f875ee2027
--- /dev/null
+++ b/kernel/bpf/bpf_insn_array.c
@@ -0,0 +1,284 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright (c) 2025 Isovalent */
+
+#include <linux/bpf.h>
+
+#define MAX_INSN_ARRAY_ENTRIES 256
+
+struct bpf_insn_array {
+ struct bpf_map map;
+ atomic_t used;
+ long *ips;
+ DECLARE_FLEX_ARRAY(struct bpf_insn_array_value, values);
+};
+
+#define cast_insn_array(MAP_PTR) \
+ container_of((MAP_PTR), struct bpf_insn_array, map)
+
+#define INSN_DELETED ((u32)-1)
+
+static inline u32 insn_array_alloc_size(u32 max_entries)
+{
+ const u32 base_size = sizeof(struct bpf_insn_array);
+ const u32 entry_size = sizeof(struct bpf_insn_array_value);
+
+ return base_size + max_entries * (entry_size + sizeof(long));
+}
+
+static int insn_array_alloc_check(union bpf_attr *attr)
+{
+ u32 value_size = sizeof(struct bpf_insn_array_value);
+
+ if (attr->max_entries == 0 || attr->key_size != 4 ||
+ attr->value_size != value_size || attr->map_flags != 0)
+ return -EINVAL;
+
+ if (attr->max_entries > MAX_INSN_ARRAY_ENTRIES)
+ return -E2BIG;
+
+ return 0;
+}
+
+static void insn_array_free(struct bpf_map *map)
+{
+ struct bpf_insn_array *insn_array = cast_insn_array(map);
+
+ bpf_map_area_free(insn_array);
+}
+
+static struct bpf_map *insn_array_alloc(union bpf_attr *attr)
+{
+ u64 size = insn_array_alloc_size(attr->max_entries);
+ struct bpf_insn_array *insn_array;
+
+ insn_array = bpf_map_area_alloc(size, NUMA_NO_NODE);
+ if (!insn_array)
+ return ERR_PTR(-ENOMEM);
+
+ /* ips are allocated right after the insn_array->values[] array */
+ insn_array->ips = (void *)&insn_array->values[attr->max_entries];
+
+ bpf_map_init_from_attr(&insn_array->map, attr);
+
+ return &insn_array->map;
+}
+
+static void *insn_array_lookup_elem(struct bpf_map *map, void *key)
+{
+ struct bpf_insn_array *insn_array = cast_insn_array(map);
+ u32 index = *(u32 *)key;
+
+ if (unlikely(index >= insn_array->map.max_entries))
+ return NULL;
+
+ return &insn_array->values[index];
+}
+
+static long insn_array_update_elem(struct bpf_map *map, void *key, void *value, u64 map_flags)
+{
+ struct bpf_insn_array *insn_array = cast_insn_array(map);
+ u32 index = *(u32 *)key;
+ struct bpf_insn_array_value val = {};
+
+ if (unlikely(index >= insn_array->map.max_entries))
+ return -E2BIG;
+
+ if (unlikely(map_flags & BPF_NOEXIST))
+ return -EEXIST;
+
+ copy_map_value(map, &val, value);
+ if (val.jitted_off || val.xlated_off)
+ return -EINVAL;
+
+ insn_array->values[index].orig_off = val.orig_off;
+
+ return 0;
+}
+
+static long insn_array_delete_elem(struct bpf_map *map, void *key)
+{
+ return -EINVAL;
+}
+
+static int insn_array_check_btf(const struct bpf_map *map,
+ const struct btf *btf,
+ const struct btf_type *key_type,
+ const struct btf_type *value_type)
+{
+ if (!btf_type_is_i32(key_type))
+ return -EINVAL;
+
+ if (!btf_type_is_i64(value_type))
+ return -EINVAL;
+
+ return 0;
+}
+
+static u64 insn_array_mem_usage(const struct bpf_map *map)
+{
+ return insn_array_alloc_size(map->max_entries);
+}
+
+BTF_ID_LIST_SINGLE(insn_array_btf_ids, struct, bpf_insn_array)
+
+const struct bpf_map_ops insn_array_map_ops = {
+ .map_alloc_check = insn_array_alloc_check,
+ .map_alloc = insn_array_alloc,
+ .map_free = insn_array_free,
+ .map_get_next_key = bpf_array_get_next_key,
+ .map_lookup_elem = insn_array_lookup_elem,
+ .map_update_elem = insn_array_update_elem,
+ .map_delete_elem = insn_array_delete_elem,
+ .map_check_btf = insn_array_check_btf,
+ .map_mem_usage = insn_array_mem_usage,
+ .map_btf_id = &insn_array_btf_ids[0],
+};
+
+static inline bool is_frozen(struct bpf_map *map)
+{
+ guard(mutex)(&map->freeze_mutex);
+
+ return map->frozen;
+}
+
+static bool is_insn_array(const struct bpf_map *map)
+{
+ return map->map_type == BPF_MAP_TYPE_INSN_ARRAY;
+}
+
+static inline bool valid_offsets(const struct bpf_insn_array *insn_array,
+ const struct bpf_prog *prog)
+{
+ u32 off;
+ int i;
+
+ for (i = 0; i < insn_array->map.max_entries; i++) {
+ off = insn_array->values[i].orig_off;
+
+ if (off >= prog->len)
+ return false;
+
+ if (off > 0) {
+ if (prog->insnsi[off-1].code == (BPF_LD | BPF_DW | BPF_IMM))
+ return false;
+ }
+ }
+
+ return true;
+}
+
+int bpf_insn_array_init(struct bpf_map *map, const struct bpf_prog *prog)
+{
+ struct bpf_insn_array *insn_array = cast_insn_array(map);
+ struct bpf_insn_array_value *values = insn_array->values;
+ int i;
+
+ if (!is_frozen(map))
+ return -EINVAL;
+
+ if (!valid_offsets(insn_array, prog))
+ return -EINVAL;
+
+ /*
+ * There can be only one program using the map
+ */
+ if (atomic_xchg(&insn_array->used, 1))
+ return -EBUSY;
+
+ /*
+ * Reset all the map indexes to the original values. This is needed,
+ * e.g., when a replay of verification with different log level should
+ * be performed.
+ */
+ for (i = 0; i < map->max_entries; i++)
+ values[i].xlated_off = values[i].orig_off;
+
+ return 0;
+}
+
+int bpf_insn_array_ready(struct bpf_map *map)
+{
+ struct bpf_insn_array *insn_array = cast_insn_array(map);
+ int i;
+
+ for (i = 0; i < map->max_entries; i++) {
+ if (insn_array->values[i].xlated_off == INSN_DELETED)
+ continue;
+ if (!insn_array->ips[i])
+ return -EFAULT;
+ }
+
+ return 0;
+}
+
+void bpf_insn_array_release(struct bpf_map *map)
+{
+ struct bpf_insn_array *insn_array = cast_insn_array(map);
+
+ atomic_set(&insn_array->used, 0);
+}
+
+void bpf_insn_array_adjust(struct bpf_map *map, u32 off, u32 len)
+{
+ struct bpf_insn_array *insn_array = cast_insn_array(map);
+ int i;
+
+ if (len <= 1)
+ return;
+
+ for (i = 0; i < map->max_entries; i++) {
+ if (insn_array->values[i].xlated_off <= off)
+ continue;
+ if (insn_array->values[i].xlated_off == INSN_DELETED)
+ continue;
+ insn_array->values[i].xlated_off += len - 1;
+ }
+}
+
+void bpf_insn_array_adjust_after_remove(struct bpf_map *map, u32 off, u32 len)
+{
+ struct bpf_insn_array *insn_array = cast_insn_array(map);
+ int i;
+
+ for (i = 0; i < map->max_entries; i++) {
+ if (insn_array->values[i].xlated_off < off)
+ continue;
+ if (insn_array->values[i].xlated_off == INSN_DELETED)
+ continue;
+ if (insn_array->values[i].xlated_off < off + len)
+ insn_array->values[i].xlated_off = INSN_DELETED;
+ else
+ insn_array->values[i].xlated_off -= len;
+ }
+}
+
+void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
+ update_insn_ptr_func_t update_insn_ptr)
+{
+ struct bpf_insn_array *insn_array;
+ struct bpf_map *map;
+ u32 xlated_off;
+ int i, j;
+
+ for (i = 0; i < prog->aux->used_map_cnt; i++) {
+ map = prog->aux->used_maps[i];
+ if (!is_insn_array(map))
+ continue;
+
+ insn_array = cast_insn_array(map);
+ for (j = 0; j < map->max_entries; j++) {
+ xlated_off = insn_array->values[j].xlated_off;
+ if (xlated_off == INSN_DELETED)
+ continue;
+ if (xlated_off < prog->aux->subprog_start)
+ continue;
+ xlated_off -= prog->aux->subprog_start;
+ if (xlated_off >= prog->len)
+ continue;
+
+ update_insn_ptr(jit_priv, xlated_off,
+ &insn_array->values[j].jitted_off,
+ &insn_array->ips[j]);
+ }
+ }
+}
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 8a129746bd6c..f62d61b6730a 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -1493,6 +1493,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr)
case BPF_MAP_TYPE_STRUCT_OPS:
case BPF_MAP_TYPE_CPUMAP:
case BPF_MAP_TYPE_ARENA:
+ case BPF_MAP_TYPE_INSN_ARRAY:
if (!bpf_token_capable(token, CAP_BPF))
goto put_token;
break;
@@ -2853,6 +2854,23 @@ static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr
return err;
}
+static int bpf_prog_mark_insn_arrays_ready(struct bpf_prog *prog)
+{
+ int err;
+ int i;
+
+ for (i = 0; i < prog->aux->used_map_cnt; i++) {
+ if (prog->aux->used_maps[i]->map_type != BPF_MAP_TYPE_INSN_ARRAY)
+ continue;
+
+ err = bpf_insn_array_ready(prog->aux->used_maps[i]);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
/* last field in 'union bpf_attr' used by this command */
#define BPF_PROG_LOAD_LAST_FIELD keyring_id
@@ -3082,6 +3100,10 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size)
if (err < 0)
goto free_used_maps;
+ err = bpf_prog_mark_insn_arrays_ready(prog);
+ if (err < 0)
+ goto free_used_maps;
+
err = bpf_prog_alloc_id(prog);
if (err)
goto free_used_maps;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 6d175849e57a..5c3f3c4e4f47 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -10083,6 +10083,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,
func_id != BPF_FUNC_map_push_elem)
goto error;
break;
+ case BPF_MAP_TYPE_INSN_ARRAY:
+ goto error;
default:
break;
}
@@ -20541,6 +20543,15 @@ static int __add_used_map(struct bpf_verifier_env *env, struct bpf_map *map)
env->used_maps[env->used_map_cnt++] = map;
+ if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) {
+ err = bpf_insn_array_init(map, env->prog);
+ if (err) {
+ verbose(env, "Failed to properly initialize insn array\n");
+ return err;
+ }
+ env->insn_array_maps[env->insn_array_map_cnt++] = map;
+ }
+
return env->used_map_cnt - 1;
}
@@ -20787,6 +20798,33 @@ static void adjust_subprog_starts(struct bpf_verifier_env *env, u32 off, u32 len
}
}
+static void release_insn_arrays(struct bpf_verifier_env *env)
+{
+ int i;
+
+ for (i = 0; i < env->insn_array_map_cnt; i++)
+ bpf_insn_array_release(env->insn_array_maps[i]);
+}
+
+static void adjust_insn_arrays(struct bpf_verifier_env *env, u32 off, u32 len)
+{
+ int i;
+
+ if (len == 1)
+ return;
+
+ for (i = 0; i < env->insn_array_map_cnt; i++)
+ bpf_insn_array_adjust(env->insn_array_maps[i], off, len);
+}
+
+static void adjust_insn_arrays_after_remove(struct bpf_verifier_env *env, u32 off, u32 len)
+{
+ int i;
+
+ for (i = 0; i < env->insn_array_map_cnt; i++)
+ bpf_insn_array_adjust_after_remove(env->insn_array_maps[i], off, len);
+}
+
static void adjust_poke_descs(struct bpf_prog *prog, u32 off, u32 len)
{
struct bpf_jit_poke_descriptor *tab = prog->aux->poke_tab;
@@ -20828,6 +20866,7 @@ static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of
}
adjust_insn_aux_data(env, new_prog, off, len);
adjust_subprog_starts(env, off, len);
+ adjust_insn_arrays(env, off, len);
adjust_poke_descs(new_prog, off, len);
return new_prog;
}
@@ -21011,6 +21050,8 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
if (err)
return err;
+ adjust_insn_arrays_after_remove(env, off, cnt);
+
memmove(aux_data + off, aux_data + off + cnt,
sizeof(*aux_data) * (orig_prog_len - off - cnt));
@@ -24811,6 +24852,8 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3
adjust_btf_func(env);
err_release_maps:
+ if (ret)
+ release_insn_arrays(env);
if (!env->prog->aux->used_maps)
/* if we didn't copy map pointers into bpf_prog_info, release
* them now. Otherwise free_used_maps() will release them.
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 6829936d33f5..805d441363cd 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -1026,6 +1026,7 @@ enum bpf_map_type {
BPF_MAP_TYPE_USER_RINGBUF,
BPF_MAP_TYPE_CGRP_STORAGE,
BPF_MAP_TYPE_ARENA,
+ BPF_MAP_TYPE_INSN_ARRAY,
__MAX_BPF_MAP_TYPE
};
@@ -7645,4 +7646,24 @@ enum bpf_kfunc_flags {
BPF_F_PAD_ZEROS = (1ULL << 0),
};
+/*
+ * Values of a BPF_MAP_TYPE_INSN_ARRAY entry must be of this type.
+ *
+ * Before the map is used the orig_off field should point to an
+ * instruction inside the program being loaded. The other fields
+ * must be set to 0.
+ *
+ * After the program is loaded, the xlated_off will be adjusted
+ * by the verifier to point to the index of the original instruction
+ * in the xlated program. If the instruction is deleted, it will
+ * be set to (u32)-1. The jitted_off will be set to the corresponding
+ * offset in the jitted image of the program.
+ */
+struct bpf_insn_array_value {
+ __u32 orig_off;
+ __u32 xlated_off;
+ __u32 jitted_off;
+ __u32 :32;
+};
+
#endif /* _UAPI__LINUX_BPF_H__ */
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 02/12] selftests/bpf: add selftests for new insn_array map
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
2025-10-26 19:26 ` [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array Anton Protopopov
@ 2025-10-26 19:26 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 03/12] bpf: support instructions arrays with constants blinding Anton Protopopov
` (9 subsequent siblings)
11 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:26 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
Add the following selftests for new insn_array map:
* Incorrect instruction indexes are rejected
* Two programs can't use the same map
* BPF progs can't operate the map
* no changes to code => map is the same
* expected changes when instructions are added
* expected changes when instructions are deleted
* expected changes when multiple functions are present
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
---
.../selftests/bpf/prog_tests/bpf_insn_array.c | 404 ++++++++++++++++++
1 file changed, 404 insertions(+)
create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
new file mode 100644
index 000000000000..a4304ef5be13
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
@@ -0,0 +1,404 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <bpf/bpf.h>
+#include <test_progs.h>
+
+#ifdef __x86_64__
+static int map_create(__u32 map_type, __u32 max_entries)
+{
+ const char *map_name = "insn_array";
+ __u32 key_size = 4;
+ __u32 value_size = sizeof(struct bpf_insn_array_value);
+
+ return bpf_map_create(map_type, map_name, key_size, value_size, max_entries, NULL);
+}
+
+static int prog_load(struct bpf_insn *insns, __u32 insn_cnt, int *fd_array, __u32 fd_array_cnt)
+{
+ LIBBPF_OPTS(bpf_prog_load_opts, opts);
+
+ opts.fd_array = fd_array;
+ opts.fd_array_cnt = fd_array_cnt;
+
+ return bpf_prog_load(BPF_PROG_TYPE_XDP, NULL, "GPL", insns, insn_cnt, &opts);
+}
+
+static void __check_success(struct bpf_insn *insns, __u32 insn_cnt, __u32 *map_in, __u32 *map_out)
+{
+ struct bpf_insn_array_value val = {};
+ int prog_fd = -1, map_fd, i;
+
+ map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, insn_cnt);
+ if (!ASSERT_GE(map_fd, 0, "map_create"))
+ return;
+
+ for (i = 0; i < insn_cnt; i++) {
+ val.orig_off = map_in[i];
+ if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &i, &val, 0), 0, "bpf_map_update_elem"))
+ goto cleanup;
+ }
+
+ if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+ goto cleanup;
+
+ prog_fd = prog_load(insns, insn_cnt, &map_fd, 1);
+ if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
+ goto cleanup;
+
+ for (i = 0; i < insn_cnt; i++) {
+ char buf[64];
+
+ if (!ASSERT_EQ(bpf_map_lookup_elem(map_fd, &i, &val), 0, "bpf_map_lookup_elem"))
+ goto cleanup;
+
+ snprintf(buf, sizeof(buf), "val.xlated_off should be equal map_out[%d]", i);
+ ASSERT_EQ(val.xlated_off, map_out[i], buf);
+ }
+
+cleanup:
+ close(prog_fd);
+ close(map_fd);
+}
+
+/*
+ * Load a program, which will not be anyhow mangled by the verifier. Add an
+ * insn_array map pointing to every instruction. Check that it hasn't changed
+ * after the program load.
+ */
+static void check_one_to_one_mapping(void)
+{
+ struct bpf_insn insns[] = {
+ BPF_MOV64_IMM(BPF_REG_0, 4),
+ BPF_MOV64_IMM(BPF_REG_0, 3),
+ BPF_MOV64_IMM(BPF_REG_0, 2),
+ BPF_MOV64_IMM(BPF_REG_0, 1),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ };
+ __u32 map_in[] = {0, 1, 2, 3, 4, 5};
+ __u32 map_out[] = {0, 1, 2, 3, 4, 5};
+
+ __check_success(insns, ARRAY_SIZE(insns), map_in, map_out);
+}
+
+/*
+ * Load a program with two patches (get jiffies, for simplicity). Add an
+ * insn_array map pointing to every instruction. Check how it was changed
+ * after the program load.
+ */
+static void check_simple(void)
+{
+ struct bpf_insn insns[] = {
+ BPF_MOV64_IMM(BPF_REG_0, 2),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_jiffies64),
+ BPF_MOV64_IMM(BPF_REG_0, 1),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_jiffies64),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ };
+ __u32 map_in[] = {0, 1, 2, 3, 4, 5};
+ __u32 map_out[] = {0, 1, 4, 5, 8, 9};
+
+ __check_success(insns, ARRAY_SIZE(insns), map_in, map_out);
+}
+
+/*
+ * Verifier can delete code in two cases: nops & dead code. From insn
+ * array's point of view, the two cases are the same, so test using
+ * the simplest method: by loading some nops
+ */
+static void check_deletions(void)
+{
+ struct bpf_insn insns[] = {
+ BPF_MOV64_IMM(BPF_REG_0, 2),
+ BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+ BPF_MOV64_IMM(BPF_REG_0, 1),
+ BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ };
+ __u32 map_in[] = {0, 1, 2, 3, 4, 5};
+ __u32 map_out[] = {0, -1, 1, -1, 2, 3};
+
+ __check_success(insns, ARRAY_SIZE(insns), map_in, map_out);
+}
+
+/*
+ * Same test as check_deletions, but also add code which adds instructions
+ */
+static void check_deletions_with_functions(void)
+{
+ struct bpf_insn insns[] = {
+ BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_jiffies64),
+ BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 1, 0, 2),
+ BPF_MOV64_IMM(BPF_REG_0, 1),
+ BPF_EXIT_INSN(),
+ BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_jiffies64),
+ BPF_JMP_IMM(BPF_JA, 0, 0, 0), /* nop */
+ BPF_MOV64_IMM(BPF_REG_0, 2),
+ BPF_EXIT_INSN(),
+ };
+ __u32 map_in[] = { 0, 1, 2, 3, 4, 5, /* func */ 6, 7, 8, 9, 10};
+ __u32 map_out[] = {-1, 0, -1, 3, 4, 5, /* func */ -1, 6, -1, 9, 10};
+
+ __check_success(insns, ARRAY_SIZE(insns), map_in, map_out);
+}
+
+/*
+ * Try to load a program with a map which points to outside of the program
+ */
+static void check_out_of_bounds_index(void)
+{
+ struct bpf_insn insns[] = {
+ BPF_MOV64_IMM(BPF_REG_0, 4),
+ BPF_MOV64_IMM(BPF_REG_0, 3),
+ BPF_MOV64_IMM(BPF_REG_0, 2),
+ BPF_MOV64_IMM(BPF_REG_0, 1),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ };
+ int prog_fd, map_fd;
+ struct bpf_insn_array_value val = {};
+ int key;
+
+ map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, 1);
+ if (!ASSERT_GE(map_fd, 0, "map_create"))
+ return;
+
+ key = 0;
+ val.orig_off = ARRAY_SIZE(insns); /* too big */
+ if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &key, &val, 0), 0, "bpf_map_update_elem"))
+ goto cleanup;
+
+ if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+ goto cleanup;
+
+ prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+ if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)")) {
+ close(prog_fd);
+ goto cleanup;
+ }
+
+cleanup:
+ close(map_fd);
+}
+
+/*
+ * Try to load a program with a map which points to the middle of 16-bit insn
+ */
+static void check_mid_insn_index(void)
+{
+ struct bpf_insn insns[] = {
+ BPF_LD_IMM64(BPF_REG_0, 0), /* 2 x 8 */
+ BPF_EXIT_INSN(),
+ };
+ int prog_fd, map_fd;
+ struct bpf_insn_array_value val = {};
+ int key;
+
+ map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, 1);
+ if (!ASSERT_GE(map_fd, 0, "map_create"))
+ return;
+
+ key = 0;
+ val.orig_off = 1; /* middle of 16-byte instruction */
+ if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &key, &val, 0), 0, "bpf_map_update_elem"))
+ goto cleanup;
+
+ if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+ goto cleanup;
+
+ prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+ if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)")) {
+ close(prog_fd);
+ goto cleanup;
+ }
+
+cleanup:
+ close(map_fd);
+}
+
+static void check_incorrect_index(void)
+{
+ check_out_of_bounds_index();
+ check_mid_insn_index();
+}
+
+/* Once map was initialized, it should be frozen */
+static void check_load_unfrozen_map(void)
+{
+ struct bpf_insn insns[] = {
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ };
+ int prog_fd = -1, map_fd;
+ struct bpf_insn_array_value val = {};
+ int i;
+
+ map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, ARRAY_SIZE(insns));
+ if (!ASSERT_GE(map_fd, 0, "map_create"))
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(insns); i++) {
+ val.orig_off = i;
+ if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &i, &val, 0), 0, "bpf_map_update_elem"))
+ goto cleanup;
+ }
+
+ prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+ if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)"))
+ goto cleanup;
+
+ /* correctness: now freeze the map, the program should load fine */
+
+ if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+ goto cleanup;
+
+ prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+ if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
+ goto cleanup;
+
+ for (i = 0; i < ARRAY_SIZE(insns); i++) {
+ if (!ASSERT_EQ(bpf_map_lookup_elem(map_fd, &i, &val), 0, "bpf_map_lookup_elem"))
+ goto cleanup;
+
+ ASSERT_EQ(val.xlated_off, i, "val should be equal i");
+ }
+
+cleanup:
+ close(prog_fd);
+ close(map_fd);
+}
+
+/* Map can be used only by one BPF program */
+static void check_no_map_reuse(void)
+{
+ struct bpf_insn insns[] = {
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ };
+ int prog_fd = -1, map_fd, extra_fd = -1;
+ struct bpf_insn_array_value val = {};
+ int i;
+
+ map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, ARRAY_SIZE(insns));
+ if (!ASSERT_GE(map_fd, 0, "map_create"))
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(insns); i++) {
+ val.orig_off = i;
+ if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &i, &val, 0), 0, "bpf_map_update_elem"))
+ goto cleanup;
+ }
+
+ if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+ goto cleanup;
+
+ prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+ if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
+ goto cleanup;
+
+ for (i = 0; i < ARRAY_SIZE(insns); i++) {
+ if (!ASSERT_EQ(bpf_map_lookup_elem(map_fd, &i, &val), 0, "bpf_map_lookup_elem"))
+ goto cleanup;
+
+ ASSERT_EQ(val.xlated_off, i, "val should be equal i");
+ }
+
+ extra_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+ if (!ASSERT_EQ(extra_fd, -EBUSY, "program should have been rejected (extra_fd != -EBUSY)"))
+ goto cleanup;
+
+ /* correctness: check that prog is still loadable without fd_array */
+ extra_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
+ if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD): expected no error"))
+ goto cleanup;
+
+cleanup:
+ close(extra_fd);
+ close(prog_fd);
+ close(map_fd);
+}
+
+static void check_bpf_no_lookup(void)
+{
+ struct bpf_insn insns[] = {
+ BPF_LD_MAP_FD(BPF_REG_1, 0),
+ BPF_ST_MEM(BPF_DW, BPF_REG_10, -8, 0),
+ BPF_MOV64_REG(BPF_REG_2, BPF_REG_10),
+ BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -8),
+ BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0, BPF_FUNC_map_lookup_elem),
+ BPF_EXIT_INSN(),
+ };
+ int prog_fd = -1, map_fd;
+
+ map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, 1);
+ if (!ASSERT_GE(map_fd, 0, "map_create"))
+ return;
+
+ insns[0].imm = map_fd;
+
+ if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+ goto cleanup;
+
+ prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
+ if (!ASSERT_EQ(prog_fd, -EINVAL, "program should have been rejected (prog_fd != -EINVAL)"))
+ goto cleanup;
+
+ /* correctness: check that prog is still loadable with normal map */
+ close(map_fd);
+ map_fd = map_create(BPF_MAP_TYPE_ARRAY, 1);
+ insns[0].imm = map_fd;
+ prog_fd = prog_load(insns, ARRAY_SIZE(insns), NULL, 0);
+ if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
+ goto cleanup;
+
+cleanup:
+ close(prog_fd);
+ close(map_fd);
+}
+
+static void check_bpf_side(void)
+{
+ check_bpf_no_lookup();
+}
+
+void test_bpf_insn_array(void)
+{
+ /* Test if offsets are adjusted properly */
+
+ if (test__start_subtest("one2one"))
+ check_one_to_one_mapping();
+
+ if (test__start_subtest("simple"))
+ check_simple();
+
+ if (test__start_subtest("deletions"))
+ check_deletions();
+
+ if (test__start_subtest("deletions-with-functions"))
+ check_deletions_with_functions();
+
+ /* Check all kinds of operations and related restrictions */
+
+ if (test__start_subtest("incorrect-index"))
+ check_incorrect_index();
+
+ if (test__start_subtest("load-unfrozen-map"))
+ check_load_unfrozen_map();
+
+ if (test__start_subtest("no-map-reuse"))
+ check_no_map_reuse();
+
+ if (test__start_subtest("bpf-side-ops"))
+ check_bpf_side();
+}
+#else
+void test_bpf_insn_array(void)
+{
+
+}
+#endif
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 03/12] bpf: support instructions arrays with constants blinding
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
2025-10-26 19:26 ` [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array Anton Protopopov
2025-10-26 19:26 ` [PATCH v7 bpf-next 02/12] selftests/bpf: add selftests for new insn_array map Anton Protopopov
@ 2025-10-26 19:27 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 04/12] selftests/bpf: test instructions arrays with blinding Anton Protopopov
` (8 subsequent siblings)
11 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:27 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
When bpf_jit_harden is enabled, all constants in the BPF code are
blinded to prevent JIT spraying attacks. This happens during JIT
phase. Adjust all the related instruction arrays accordingly.
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
---
kernel/bpf/core.c | 20 ++++++++++++++++++++
kernel/bpf/verifier.c | 11 ++++++++++-
2 files changed, 30 insertions(+), 1 deletion(-)
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index d595fe512498..4b62a03d6df5 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1450,6 +1450,23 @@ void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_other)
bpf_prog_clone_free(fp_other);
}
+static void adjust_insn_arrays(struct bpf_prog *prog, u32 off, u32 len)
+{
+#ifdef CONFIG_BPF_SYSCALL
+ struct bpf_map *map;
+ int i;
+
+ if (len <= 1)
+ return;
+
+ for (i = 0; i < prog->aux->used_map_cnt; i++) {
+ map = prog->aux->used_maps[i];
+ if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY)
+ bpf_insn_array_adjust(map, off, len);
+ }
+#endif
+}
+
struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)
{
struct bpf_insn insn_buff[16], aux[2];
@@ -1505,6 +1522,9 @@ struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog)
clone = tmp;
insn_delta = rewritten - 1;
+ /* Instructions arrays must be updated using absolute xlated offsets */
+ adjust_insn_arrays(clone, prog->aux->subprog_start + i, rewritten);
+
/* Walk new program and skip insns we just inserted. */
insn = clone->insnsi + i + insn_delta;
insn_cnt += insn_delta;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 5c3f3c4e4f47..ae017c032944 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -21591,6 +21591,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
struct bpf_insn *insn;
void *old_bpf_func;
int err, num_exentries;
+ int old_len, subprog_start_adjustment = 0;
if (env->subprog_cnt <= 1)
return 0;
@@ -21665,7 +21666,7 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->func_idx = i;
/* Below members will be freed only at prog->aux */
func[i]->aux->btf = prog->aux->btf;
- func[i]->aux->subprog_start = subprog_start;
+ func[i]->aux->subprog_start = subprog_start + subprog_start_adjustment;
func[i]->aux->func_info = prog->aux->func_info;
func[i]->aux->func_info_cnt = prog->aux->func_info_cnt;
func[i]->aux->poke_tab = prog->aux->poke_tab;
@@ -21719,7 +21720,15 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->might_sleep = env->subprog_info[i].might_sleep;
if (!i)
func[i]->aux->exception_boundary = env->seen_exception;
+
+ /*
+ * To properly pass the absolute subprog start to jit
+ * all instruction adjustments should be accumulated
+ */
+ old_len = func[i]->len;
func[i] = bpf_int_jit_compile(func[i]);
+ subprog_start_adjustment += func[i]->len - old_len;
+
if (!func[i]->jited) {
err = -ENOTSUPP;
goto out_free;
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 04/12] selftests/bpf: test instructions arrays with blinding
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
` (2 preceding siblings ...)
2025-10-26 19:27 ` [PATCH v7 bpf-next 03/12] bpf: support instructions arrays with constants blinding Anton Protopopov
@ 2025-10-26 19:27 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 05/12] bpf, x86: allow indirect jumps to r8...r15 Anton Protopopov
` (7 subsequent siblings)
11 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:27 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
Add a specific test for instructions arrays with blinding enabled.
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
---
.../selftests/bpf/prog_tests/bpf_insn_array.c | 95 +++++++++++++++++++
1 file changed, 95 insertions(+)
diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
index a4304ef5be13..bab55ae7687e 100644
--- a/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
+++ b/tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c
@@ -227,6 +227,98 @@ static void check_incorrect_index(void)
check_mid_insn_index();
}
+static int set_bpf_jit_harden(char *level)
+{
+ char old_level;
+ int err = -1;
+ int fd = -1;
+
+ fd = open("/proc/sys/net/core/bpf_jit_harden", O_RDWR | O_NONBLOCK);
+ if (fd < 0) {
+ ASSERT_FAIL("open .../bpf_jit_harden returned %d (errno=%d)", fd, errno);
+ return -1;
+ }
+
+ err = read(fd, &old_level, 1);
+ if (err != 1) {
+ ASSERT_FAIL("read from .../bpf_jit_harden returned %d (errno=%d)", err, errno);
+ err = -1;
+ goto end;
+ }
+
+ lseek(fd, 0, SEEK_SET);
+
+ err = write(fd, level, 1);
+ if (err != 1) {
+ ASSERT_FAIL("write to .../bpf_jit_harden returned %d (errno=%d)", err, errno);
+ err = -1;
+ goto end;
+ }
+
+ err = 0;
+ *level = old_level;
+end:
+ if (fd >= 0)
+ close(fd);
+ return err;
+}
+
+static void check_blindness(void)
+{
+ struct bpf_insn insns[] = {
+ BPF_MOV64_IMM(BPF_REG_0, 4),
+ BPF_MOV64_IMM(BPF_REG_0, 3),
+ BPF_MOV64_IMM(BPF_REG_0, 2),
+ BPF_MOV64_IMM(BPF_REG_0, 1),
+ BPF_EXIT_INSN(),
+ };
+ int prog_fd = -1, map_fd;
+ struct bpf_insn_array_value val = {};
+ char bpf_jit_harden = '@'; /* non-exizsting value */
+ int i;
+
+ map_fd = map_create(BPF_MAP_TYPE_INSN_ARRAY, ARRAY_SIZE(insns));
+ if (!ASSERT_GE(map_fd, 0, "map_create"))
+ return;
+
+ for (i = 0; i < ARRAY_SIZE(insns); i++) {
+ val.orig_off = i;
+ if (!ASSERT_EQ(bpf_map_update_elem(map_fd, &i, &val, 0), 0, "bpf_map_update_elem"))
+ goto cleanup;
+ }
+
+ if (!ASSERT_EQ(bpf_map_freeze(map_fd), 0, "bpf_map_freeze"))
+ goto cleanup;
+
+ bpf_jit_harden = '2';
+ if (set_bpf_jit_harden(&bpf_jit_harden)) {
+ bpf_jit_harden = '@'; /* open, read or write failed => no write was done */
+ goto cleanup;
+ }
+
+ prog_fd = prog_load(insns, ARRAY_SIZE(insns), &map_fd, 1);
+ if (!ASSERT_GE(prog_fd, 0, "bpf(BPF_PROG_LOAD)"))
+ goto cleanup;
+
+ for (i = 0; i < ARRAY_SIZE(insns); i++) {
+ char fmt[32];
+
+ if (!ASSERT_EQ(bpf_map_lookup_elem(map_fd, &i, &val), 0, "bpf_map_lookup_elem"))
+ goto cleanup;
+
+ snprintf(fmt, sizeof(fmt), "val should be equal 3*%d", i);
+ ASSERT_EQ(val.xlated_off, i * 3, fmt);
+ }
+
+cleanup:
+ /* restore the old one */
+ if (bpf_jit_harden != '@')
+ set_bpf_jit_harden(&bpf_jit_harden);
+
+ close(prog_fd);
+ close(map_fd);
+}
+
/* Once map was initialized, it should be frozen */
static void check_load_unfrozen_map(void)
{
@@ -382,6 +474,9 @@ void test_bpf_insn_array(void)
if (test__start_subtest("deletions-with-functions"))
check_deletions_with_functions();
+ if (test__start_subtest("blindness"))
+ check_blindness();
+
/* Check all kinds of operations and related restrictions */
if (test__start_subtest("incorrect-index"))
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 05/12] bpf, x86: allow indirect jumps to r8...r15
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
` (3 preceding siblings ...)
2025-10-26 19:27 ` [PATCH v7 bpf-next 04/12] selftests/bpf: test instructions arrays with blinding Anton Protopopov
@ 2025-10-26 19:27 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 06/12] bpf, x86: add support for indirect jumps Anton Protopopov
` (6 subsequent siblings)
11 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:27 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
Currently the emit_indirect_jump() function only accepts one of the
RAX, RCX, ..., RBP registers as the destination. Make it to accept
R8, R9, ..., R15 as well, and make callers to pass BPF registers, not
native registers. This is required to enable indirect jumps support
in eBPF.
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
---
arch/x86/net/bpf_jit_comp.c | 30 ++++++++++++++++++++++--------
1 file changed, 22 insertions(+), 8 deletions(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index d8ee0c4d9af8..3f59e8040447 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -660,24 +660,38 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,
#define EMIT_LFENCE() EMIT3(0x0F, 0xAE, 0xE8)
-static void emit_indirect_jump(u8 **pprog, int reg, u8 *ip)
+static void __emit_indirect_jump(u8 **pprog, int reg, bool ereg)
{
u8 *prog = *pprog;
+ if (ereg)
+ EMIT1(0x41);
+
+ EMIT2(0xFF, 0xE0 + reg);
+
+ *pprog = prog;
+}
+
+static void emit_indirect_jump(u8 **pprog, int bpf_reg, u8 *ip)
+{
+ u8 *prog = *pprog;
+ int reg = reg2hex[bpf_reg];
+ bool ereg = is_ereg(bpf_reg);
+
if (cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) {
OPTIMIZER_HIDE_VAR(reg);
- emit_jump(&prog, its_static_thunk(reg), ip);
+ emit_jump(&prog, its_static_thunk(reg + 8*ereg), ip);
} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) {
EMIT_LFENCE();
- EMIT2(0xFF, 0xE0 + reg);
+ __emit_indirect_jump(&prog, reg, ereg);
} else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) {
OPTIMIZER_HIDE_VAR(reg);
if (cpu_feature_enabled(X86_FEATURE_CALL_DEPTH))
- emit_jump(&prog, &__x86_indirect_jump_thunk_array[reg], ip);
+ emit_jump(&prog, &__x86_indirect_jump_thunk_array[reg + 8*ereg], ip);
else
- emit_jump(&prog, &__x86_indirect_thunk_array[reg], ip);
+ emit_jump(&prog, &__x86_indirect_thunk_array[reg + 8*ereg], ip);
} else {
- EMIT2(0xFF, 0xE0 + reg); /* jmp *%\reg */
+ __emit_indirect_jump(&prog, reg, ereg);
if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) || IS_ENABLED(CONFIG_MITIGATION_SLS))
EMIT1(0xCC); /* int3 */
}
@@ -797,7 +811,7 @@ static void emit_bpf_tail_call_indirect(struct bpf_prog *bpf_prog,
* rdi == ctx (1st arg)
* rcx == prog->bpf_func + X86_TAIL_CALL_OFFSET
*/
- emit_indirect_jump(&prog, 1 /* rcx */, ip + (prog - start));
+ emit_indirect_jump(&prog, BPF_REG_4 /* R4 -> rcx */, ip + (prog - start));
/* out: */
ctx->tail_call_indirect_label = prog - start;
@@ -3543,7 +3557,7 @@ static int emit_bpf_dispatcher(u8 **pprog, int a, int b, s64 *progs, u8 *image,
if (err)
return err;
- emit_indirect_jump(&prog, 2 /* rdx */, image + (prog - buf));
+ emit_indirect_jump(&prog, BPF_REG_3 /* R3 -> rdx */, image + (prog - buf));
*pprog = prog;
return 0;
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 06/12] bpf, x86: add support for indirect jumps
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
` (4 preceding siblings ...)
2025-10-26 19:27 ` [PATCH v7 bpf-next 05/12] bpf, x86: allow indirect jumps to r8...r15 Anton Protopopov
@ 2025-10-26 19:27 ` Anton Protopopov
2025-10-26 20:41 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 07/12] bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X Anton Protopopov
` (5 subsequent siblings)
11 siblings, 1 reply; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:27 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
Add support for a new instruction
BPF_JMP|BPF_X|BPF_JA, SRC=0, DST=Rx, off=0, imm=0
which does an indirect jump to a location stored in Rx. The register
Rx should have type PTR_TO_INSN. This new type assures that the Rx
register contains a value (or a range of values) loaded from a
correct jump table – map of type instruction array.
For example, for a C switch LLVM will generate the following code:
0: r3 = r1 # "switch (r3)"
1: if r3 > 0x13 goto +0x666 # check r3 boundaries
2: r3 <<= 0x3 # adjust to an index in array of addresses
3: r1 = 0xbeef ll # r1 is PTR_TO_MAP_VALUE, r1->map_ptr=M
5: r1 += r3 # r1 inherits boundaries from r3
6: r1 = *(u64 *)(r1 + 0x0) # r1 now has type INSN_TO_PTR
7: gotox r1 # jit will generate proper code
Here the gotox instruction corresponds to one particular map. This is
possible however to have a gotox instruction which can be loaded from
different maps, e.g.
0: r1 &= 0x1
1: r2 <<= 0x3
2: r3 = 0x0 ll # load from map M_1
4: r3 += r2
5: if r1 == 0x0 goto +0x4
6: r1 <<= 0x3
7: r3 = 0x0 ll # load from map M_2
9: r3 += r1
A: r1 = *(u64 *)(r3 + 0x0)
B: gotox r1 # jump to target loaded from M_1 or M_2
During check_cfg stage the verifier will collect all the maps which
point to inside the subprog being verified. When building the config,
the high 16 bytes of the insn_state are used, so this patch
(theoretically) supports jump tables of up to 2^16 slots.
During the later stage, in check_indirect_jump, it is checked that
the register Rx was loaded from a particular instruction array.
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
---
arch/x86/net/bpf_jit_comp.c | 3 +
include/linux/bpf.h | 1 +
include/linux/bpf_verifier.h | 9 +
kernel/bpf/bpf_insn_array.c | 15 ++
kernel/bpf/core.c | 1 +
kernel/bpf/liveness.c | 3 +
kernel/bpf/log.c | 1 +
kernel/bpf/verifier.c | 364 ++++++++++++++++++++++++++++++++++-
8 files changed, 391 insertions(+), 6 deletions(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 3f59e8040447..2035d6d07e6d 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2628,6 +2628,9 @@ st: if (is_imm8(insn->off))
break;
+ case BPF_JMP | BPF_JA | BPF_X:
+ emit_indirect_jump(&prog, insn->dst_reg, image + addrs[i - 1]);
+ break;
case BPF_JMP | BPF_JA:
case BPF_JMP32 | BPF_JA:
if (BPF_CLASS(insn->code) == BPF_JMP) {
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index b64c2382fbb7..74177c156438 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -995,6 +995,7 @@ enum bpf_reg_type {
PTR_TO_ARENA,
PTR_TO_BUF, /* reg points to a read/write buffer */
PTR_TO_FUNC, /* reg points to a bpf program function */
+ PTR_TO_INSN, /* reg points to a bpf program instruction */
CONST_PTR_TO_DYNPTR, /* reg points to a const struct bpf_dynptr */
__BPF_REG_TYPE_MAX,
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 6b820d8d77af..5441341f1ab9 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -527,6 +527,7 @@ struct bpf_insn_aux_data {
struct {
u32 map_index; /* index into used_maps[] */
u32 map_off; /* offset from value base address */
+ struct bpf_iarray *jt; /* jump table for gotox instruction */
};
struct {
enum bpf_reg_type reg_type; /* type of pseudo_btf_id */
@@ -840,6 +841,7 @@ struct bpf_verifier_env {
struct bpf_scc_info **scc_info;
u32 scc_cnt;
struct bpf_iarray *succ;
+ struct bpf_iarray *gotox_tmp_buf;
};
static inline struct bpf_func_info_aux *subprog_aux(struct bpf_verifier_env *env, int subprog)
@@ -1050,6 +1052,13 @@ static inline bool bpf_stack_narrow_access_ok(int off, int fill_size, int spill_
return !(off % BPF_REG_SIZE);
}
+static inline bool insn_is_gotox(struct bpf_insn *insn)
+{
+ return BPF_CLASS(insn->code) == BPF_JMP &&
+ BPF_OP(insn->code) == BPF_JA &&
+ BPF_SRC(insn->code) == BPF_X;
+}
+
const char *reg_type_str(struct bpf_verifier_env *env, enum bpf_reg_type type);
const char *dynptr_type_str(enum bpf_dynptr_type type);
const char *iter_type_str(const struct btf *btf, u32 btf_id);
diff --git a/kernel/bpf/bpf_insn_array.c b/kernel/bpf/bpf_insn_array.c
index f9f875ee2027..b9b1d6609acb 100644
--- a/kernel/bpf/bpf_insn_array.c
+++ b/kernel/bpf/bpf_insn_array.c
@@ -119,6 +119,20 @@ static u64 insn_array_mem_usage(const struct bpf_map *map)
return insn_array_alloc_size(map->max_entries);
}
+static int insn_array_map_direct_value_addr(const struct bpf_map *map, u64 *imm, u32 off)
+{
+ struct bpf_insn_array *insn_array = cast_insn_array(map);
+
+ if ((off % sizeof(long)) != 0 ||
+ (off / sizeof(long)) >= map->max_entries)
+ return -EINVAL;
+
+ /* from BPF's point of view, this map is a jump table */
+ *imm = (unsigned long)insn_array->ips + off;
+
+ return 0;
+}
+
BTF_ID_LIST_SINGLE(insn_array_btf_ids, struct, bpf_insn_array)
const struct bpf_map_ops insn_array_map_ops = {
@@ -131,6 +145,7 @@ const struct bpf_map_ops insn_array_map_ops = {
.map_delete_elem = insn_array_delete_elem,
.map_check_btf = insn_array_check_btf,
.map_mem_usage = insn_array_mem_usage,
+ .map_direct_value_addr = insn_array_map_direct_value_addr,
.map_btf_id = &insn_array_btf_ids[0],
};
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 4b62a03d6df5..ef4448f18aad 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -1708,6 +1708,7 @@ bool bpf_opcode_in_insntable(u8 code)
[BPF_LD | BPF_IND | BPF_B] = true,
[BPF_LD | BPF_IND | BPF_H] = true,
[BPF_LD | BPF_IND | BPF_W] = true,
+ [BPF_JMP | BPF_JA | BPF_X] = true,
[BPF_JMP | BPF_JCOND] = true,
};
#undef BPF_INSN_3_TBL
diff --git a/kernel/bpf/liveness.c b/kernel/bpf/liveness.c
index bffb495bc933..a7240013fd9d 100644
--- a/kernel/bpf/liveness.c
+++ b/kernel/bpf/liveness.c
@@ -485,6 +485,9 @@ bpf_insn_successors(struct bpf_verifier_env *env, u32 idx)
struct bpf_iarray *succ;
int insn_sz;
+ if (unlikely(insn_is_gotox(insn)))
+ return env->insn_aux_data[idx].jt;
+
/* pre-allocated array of size up to 2; reset cnt, as it may have been used already */
succ = env->succ;
succ->cnt = 0;
diff --git a/kernel/bpf/log.c b/kernel/bpf/log.c
index f50533169cc3..3fb56617af36 100644
--- a/kernel/bpf/log.c
+++ b/kernel/bpf/log.c
@@ -461,6 +461,7 @@ const char *reg_type_str(struct bpf_verifier_env *env, enum bpf_reg_type type)
[PTR_TO_ARENA] = "arena",
[PTR_TO_BUF] = "buf",
[PTR_TO_FUNC] = "func",
+ [PTR_TO_INSN] = "insn",
[PTR_TO_MAP_KEY] = "map_key",
[CONST_PTR_TO_DYNPTR] = "dynptr_ptr",
};
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index ae017c032944..c746652074ab 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -6003,6 +6003,18 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno,
return 0;
}
+/*
+ * Return the size of the memory region accessible from a pointer to map value.
+ * For INSN_ARRAY maps whole bpf_insn_array->ips array is accessible.
+ */
+static u32 map_mem_size(const struct bpf_map *map)
+{
+ if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY)
+ return map->max_entries * sizeof(long);
+
+ return map->value_size;
+}
+
/* check read/write into a map element with possible variable offset */
static int check_map_access(struct bpf_verifier_env *env, u32 regno,
int off, int size, bool zero_size_allowed,
@@ -6012,11 +6024,11 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno,
struct bpf_func_state *state = vstate->frame[vstate->curframe];
struct bpf_reg_state *reg = &state->regs[regno];
struct bpf_map *map = reg->map_ptr;
+ u32 mem_size = map_mem_size(map);
struct btf_record *rec;
int err, i;
- err = check_mem_region_access(env, regno, off, size, map->value_size,
- zero_size_allowed);
+ err = check_mem_region_access(env, regno, off, size, mem_size, zero_size_allowed);
if (err)
return err;
@@ -7478,6 +7490,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
{
struct bpf_reg_state *regs = cur_regs(env);
struct bpf_reg_state *reg = regs + regno;
+ bool insn_array = reg->type == PTR_TO_MAP_VALUE &&
+ reg->map_ptr->map_type == BPF_MAP_TYPE_INSN_ARRAY;
int size, err = 0;
size = bpf_size_to_bytes(bpf_size);
@@ -7485,7 +7499,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
return size;
/* alignment checks will add in reg->off themselves */
- err = check_ptr_alignment(env, reg, off, size, strict_alignment_once);
+ err = check_ptr_alignment(env, reg, off, size, strict_alignment_once || insn_array);
if (err)
return err;
@@ -7512,6 +7526,11 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
verbose(env, "R%d leaks addr into map\n", value_regno);
return -EACCES;
}
+ if (t == BPF_WRITE && insn_array) {
+ verbose(env, "writes into insn_array not allowed\n");
+ return -EACCES;
+ }
+
err = check_map_access_type(env, regno, off, size, t);
if (err)
return err;
@@ -7540,6 +7559,14 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
regs[value_regno].type = SCALAR_VALUE;
__mark_reg_known(®s[value_regno], val);
+ } else if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) {
+ if (bpf_size != BPF_DW) {
+ verbose(env, "Invalid read of %d bytes from insn_array\n",
+ size);
+ return -EACCES;
+ }
+ copy_register_state(®s[value_regno], reg);
+ regs[value_regno].type = PTR_TO_INSN;
} else {
mark_reg_unknown(env, regs, value_regno);
}
@@ -17055,7 +17082,8 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn)
}
dst_reg->type = PTR_TO_MAP_VALUE;
dst_reg->off = aux->map_off;
- WARN_ON_ONCE(map->max_entries != 1);
+ WARN_ON_ONCE(map->map_type != BPF_MAP_TYPE_INSN_ARRAY &&
+ map->max_entries != 1);
/* We want reg->id to be same (0) as map_value is not distinct */
} else if (insn->src_reg == BPF_PSEUDO_MAP_FD ||
insn->src_reg == BPF_PSEUDO_MAP_IDX) {
@@ -17823,6 +17851,197 @@ static struct bpf_iarray *iarray_realloc(struct bpf_iarray *old, size_t n_elem)
return new;
}
+static int copy_insn_array(struct bpf_map *map, u32 start, u32 end, u32 *items)
+{
+ struct bpf_insn_array_value *value;
+ u32 i;
+
+ for (i = start; i <= end; i++) {
+ value = map->ops->map_lookup_elem(map, &i);
+ if (!value)
+ return -EINVAL;
+ items[i - start] = value->xlated_off;
+ }
+ return 0;
+}
+
+static int cmp_ptr_to_u32(const void *a, const void *b)
+{
+ return *(u32 *)a - *(u32 *)b;
+}
+
+static int sort_insn_array_uniq(u32 *items, int cnt)
+{
+ int unique = 1;
+ int i;
+
+ sort(items, cnt, sizeof(items[0]), cmp_ptr_to_u32, NULL);
+
+ for (i = 1; i < cnt; i++)
+ if (items[i] != items[unique - 1])
+ items[unique++] = items[i];
+
+ return unique;
+}
+
+/*
+ * sort_unique({map[start], ..., map[end]}) into off
+ */
+static int copy_insn_array_uniq(struct bpf_map *map, u32 start, u32 end, u32 *off)
+{
+ u32 n = end - start + 1;
+ int err;
+
+ err = copy_insn_array(map, start, end, off);
+ if (err)
+ return err;
+
+ return sort_insn_array_uniq(off, n);
+}
+
+/*
+ * Copy all unique offsets from the map
+ */
+static struct bpf_iarray *jt_from_map(struct bpf_map *map)
+{
+ struct bpf_iarray *jt;
+ int n;
+
+ jt = iarray_realloc(NULL, map->max_entries);
+ if (!jt)
+ return ERR_PTR(-ENOMEM);
+
+ n = copy_insn_array_uniq(map, 0, map->max_entries - 1, jt->items);
+ if (n < 0) {
+ kvfree(jt);
+ return ERR_PTR(n);
+ }
+
+ return jt;
+}
+
+/*
+ * Find and collect all maps which fit in the subprog. Return the result as one
+ * combined jump table in jt->items (allocated with kvcalloc)
+ */
+static struct bpf_iarray *jt_from_subprog(struct bpf_verifier_env *env,
+ int subprog_start, int subprog_end)
+{
+ struct bpf_iarray *jt = NULL;
+ struct bpf_map *map;
+ struct bpf_iarray *jt_cur;
+ int i;
+
+ for (i = 0; i < env->insn_array_map_cnt; i++) {
+ /*
+ * TODO (when needed): collect only jump tables, not static keys
+ * or maps for indirect calls
+ */
+ map = env->insn_array_maps[i];
+
+ jt_cur = jt_from_map(map);
+ if (IS_ERR(jt_cur)) {
+ kvfree(jt);
+ return jt_cur;
+ }
+
+ /*
+ * This is enough to check one element. The full table is
+ * checked to fit inside the subprog later in create_jt()
+ */
+ if (jt_cur->items[0] >= subprog_start && jt_cur->items[0] < subprog_end) {
+ u32 old_cnt = jt ? jt->cnt : 0;
+ jt = iarray_realloc(jt, old_cnt + jt_cur->cnt);
+ if (!jt) {
+ kvfree(jt_cur);
+ return ERR_PTR(-ENOMEM);
+ }
+ memcpy(jt->items + old_cnt, jt_cur->items, jt_cur->cnt << 2);
+ }
+
+ kvfree(jt_cur);
+ }
+
+ if (!jt) {
+ verbose(env, "no jump tables found for subprog starting at %u\n", subprog_start);
+ return ERR_PTR(-EINVAL);
+ }
+
+ jt->cnt = sort_insn_array_uniq(jt->items, jt->cnt);
+ return jt;
+}
+
+static struct bpf_iarray *
+create_jt(int t, struct bpf_verifier_env *env)
+{
+ static struct bpf_subprog_info *subprog;
+ int subprog_start, subprog_end;
+ struct bpf_iarray *jt;
+ int i;
+
+ subprog = bpf_find_containing_subprog(env, t);
+ subprog_start = subprog->start;
+ subprog_end = (subprog + 1)->start;
+ jt = jt_from_subprog(env, subprog_start, subprog_end);
+ if (IS_ERR(jt))
+ return jt;
+
+ /* Check that the every element of the jump table fits within the given subprogram */
+ for (i = 0; i < jt->cnt; i++) {
+ if (jt->items[i] < subprog_start || jt->items[i] >= subprog_end) {
+ verbose(env, "jump table for insn %d points outside of the subprog [%u,%u]",
+ t, subprog_start, subprog_end);
+ kvfree(jt);
+ return ERR_PTR(-EINVAL);
+ }
+ }
+
+ return jt;
+}
+
+/* "conditional jump with N edges" */
+static int visit_gotox_insn(int t, struct bpf_verifier_env *env)
+{
+ int *insn_stack = env->cfg.insn_stack;
+ int *insn_state = env->cfg.insn_state;
+ bool keep_exploring = false;
+ struct bpf_iarray *jt;
+ int i, w;
+
+ jt = env->insn_aux_data[t].jt;
+ if (!jt) {
+ jt = create_jt(t, env);
+ if (IS_ERR(jt))
+ return PTR_ERR(jt);
+
+ env->insn_aux_data[t].jt = jt;
+ }
+
+ mark_prune_point(env, t);
+ for (i = 0; i < jt->cnt; i++) {
+ w = jt->items[i];
+ if (w < 0 || w >= env->prog->len) {
+ verbose(env, "indirect jump out of range from insn %d to %d\n", t, w);
+ return -EINVAL;
+ }
+
+ mark_jmp_point(env, w);
+
+ /* EXPLORED || DISCOVERED */
+ if (insn_state[w])
+ continue;
+
+ if (env->cfg.cur_stack >= env->prog->len)
+ return -E2BIG;
+
+ insn_stack[env->cfg.cur_stack++] = w;
+ insn_state[w] |= DISCOVERED;
+ keep_exploring = true;
+ }
+
+ return keep_exploring ? KEEP_EXPLORING : DONE_EXPLORING;
+}
+
/* Visits the instruction at index t and returns one of the following:
* < 0 - an error occurred
* DONE_EXPLORING - the instruction was fully explored
@@ -17915,8 +18134,8 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
return visit_func_call_insn(t, insns, env, insn->src_reg == BPF_PSEUDO_CALL);
case BPF_JA:
- if (BPF_SRC(insn->code) != BPF_K)
- return -EINVAL;
+ if (BPF_SRC(insn->code) == BPF_X)
+ return visit_gotox_insn(t, env);
if (BPF_CLASS(insn->code) == BPF_JMP)
off = insn->off;
@@ -18845,6 +19064,9 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
return regs_exact(rold, rcur, idmap) && rold->frameno == rcur->frameno;
case PTR_TO_ARENA:
return true;
+ case PTR_TO_INSN:
+ return (rold->off == rcur->off && range_within(rold, rcur) &&
+ tnum_in(rold->var_off, rcur->var_off));
default:
return regs_exact(rold, rcur, idmap);
}
@@ -19854,6 +20076,99 @@ static int process_bpf_exit_full(struct bpf_verifier_env *env,
return PROCESS_BPF_EXIT;
}
+static int indirect_jump_min_max_index(struct bpf_verifier_env *env,
+ int regno,
+ struct bpf_map *map,
+ u32 *pmin_index, u32 *pmax_index)
+{
+ struct bpf_reg_state *reg = reg_state(env, regno);
+ u64 min_index, max_index;
+ const u32 size = 8;
+
+ if (check_add_overflow(reg->umin_value, reg->off, &min_index) ||
+ (min_index > (u64) U32_MAX * size)) {
+ verbose(env, "the sum of R%u umin_value %llu and off %u is too big\n",
+ regno, reg->umin_value, reg->off);
+ return -ERANGE;
+ }
+ if (check_add_overflow(reg->umax_value, reg->off, &max_index) ||
+ (max_index > (u64) U32_MAX * size)) {
+ verbose(env, "the sum of R%u umax_value %llu and off %u is too big\n",
+ regno, reg->umax_value, reg->off);
+ return -ERANGE;
+ }
+
+ min_index /= size;
+ max_index /= size;
+
+ if (max_index >= map->max_entries) {
+ verbose(env, "R%u points to outside of jump table: [%llu,%llu] max_entries %u\n",
+ regno, min_index, max_index, map->max_entries);
+ return -EINVAL;
+ }
+
+ *pmin_index = min_index;
+ *pmax_index = max_index;
+ return 0;
+}
+
+/* gotox *dst_reg */
+static int check_indirect_jump(struct bpf_verifier_env *env, struct bpf_insn *insn)
+{
+ struct bpf_verifier_state *other_branch;
+ struct bpf_reg_state *dst_reg;
+ struct bpf_map *map;
+ u32 min_index, max_index;
+ int err = 0;
+ int n;
+ int i;
+
+ dst_reg = reg_state(env, insn->dst_reg);
+ if (dst_reg->type != PTR_TO_INSN) {
+ verbose(env, "R%d has type %s, expected PTR_TO_INSN\n",
+ insn->dst_reg, reg_type_str(env, dst_reg->type));
+ return -EINVAL;
+ }
+
+ map = dst_reg->map_ptr;
+ if (verifier_bug_if(!map, env, "R%d has an empty map pointer", insn->dst_reg))
+ return -EFAULT;
+
+ if (verifier_bug_if(map->map_type != BPF_MAP_TYPE_INSN_ARRAY, env,
+ "R%d has incorrect map type %d", insn->dst_reg, map->map_type))
+ return -EFAULT;
+
+ err = indirect_jump_min_max_index(env, insn->dst_reg, map, &min_index, &max_index);
+ if (err)
+ return err;
+
+ /* Ensure that the buffer is large enough */
+ if (!env->gotox_tmp_buf || env->gotox_tmp_buf->cnt < max_index - min_index + 1) {
+ env->gotox_tmp_buf = iarray_realloc(env->gotox_tmp_buf,
+ max_index - min_index + 1);
+ if (!env->gotox_tmp_buf)
+ return -ENOMEM;
+ }
+
+ n = copy_insn_array_uniq(map, min_index, max_index, env->gotox_tmp_buf->items);
+ if (n < 0)
+ return n;
+ if (n == 0) {
+ verbose(env, "register R%d doesn't point to any offset in map id=%d\n",
+ insn->dst_reg, map->id);
+ return -EINVAL;
+ }
+
+ for (i = 0; i < n - 1; i++) {
+ other_branch = push_stack(env, env->gotox_tmp_buf->items[i],
+ env->insn_idx, env->cur_state->speculative);
+ if (IS_ERR(other_branch))
+ return PTR_ERR(other_branch);
+ }
+ env->insn_idx = env->gotox_tmp_buf->items[n-1];
+ return 0;
+}
+
static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
{
int err;
@@ -19956,6 +20271,15 @@ static int do_check_insn(struct bpf_verifier_env *env, bool *do_print_state)
mark_reg_scratched(env, BPF_REG_0);
} else if (opcode == BPF_JA) {
+ if (BPF_SRC(insn->code) == BPF_X) {
+ if (insn->src_reg != BPF_REG_0 ||
+ insn->imm != 0 || insn->off != 0) {
+ verbose(env, "BPF_JA|BPF_X uses reserved fields\n");
+ return -EINVAL;
+ }
+ return check_indirect_jump(env, insn);
+ }
+
if (BPF_SRC(insn->code) != BPF_K ||
insn->src_reg != BPF_REG_0 ||
insn->dst_reg != BPF_REG_0 ||
@@ -20472,6 +20796,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env,
case BPF_MAP_TYPE_QUEUE:
case BPF_MAP_TYPE_STACK:
case BPF_MAP_TYPE_ARENA:
+ case BPF_MAP_TYPE_INSN_ARRAY:
break;
default:
verbose(env,
@@ -21029,6 +21354,27 @@ static int bpf_adj_linfo_after_remove(struct bpf_verifier_env *env, u32 off,
return 0;
}
+/*
+ * Clean up dynamically allocated fields of aux data for instructions [start, ...]
+ */
+static void clear_insn_aux_data(struct bpf_verifier_env *env, int start, int len)
+{
+ struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
+ struct bpf_insn *insns = env->prog->insnsi;
+ int end = start + len;
+ int i;
+
+ for (i = start; i < end; i++) {
+ if (insn_is_gotox(&insns[i])) {
+ kvfree(aux_data[i].jt);
+ aux_data[i].jt = NULL;
+ }
+
+ if (bpf_is_ldimm64(&insns[i]))
+ i++;
+ }
+}
+
static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
{
struct bpf_insn_aux_data *aux_data = env->insn_aux_data;
@@ -21052,6 +21398,8 @@ static int verifier_remove_insns(struct bpf_verifier_env *env, u32 off, u32 cnt)
adjust_insn_arrays_after_remove(env, off, cnt);
+ clear_insn_aux_data(env, off, cnt);
+
memmove(aux_data + off, aux_data + off + cnt,
sizeof(*aux_data) * (orig_prog_len - off - cnt));
@@ -21696,6 +22044,8 @@ static int jit_subprogs(struct bpf_verifier_env *env)
func[i]->aux->jited_linfo = prog->aux->jited_linfo;
func[i]->aux->linfo_idx = env->subprog_info[i].linfo_idx;
func[i]->aux->arena = prog->aux->arena;
+ func[i]->aux->used_maps = env->used_maps;
+ func[i]->aux->used_map_cnt = env->used_map_cnt;
num_exentries = 0;
insn = func[i]->insnsi;
for (j = 0; j < func[i]->len; j++, insn++) {
@@ -24883,12 +25233,14 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, bpfptr_t uattr, __u3
err_unlock:
if (!is_priv)
mutex_unlock(&bpf_verifier_lock);
+ clear_insn_aux_data(env, 0, env->prog->len);
vfree(env->insn_aux_data);
err_free_env:
bpf_stack_liveness_free(env);
kvfree(env->cfg.insn_postorder);
kvfree(env->scc_info);
kvfree(env->succ);
+ kvfree(env->gotox_tmp_buf);
kvfree(env);
return ret;
}
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 07/12] bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
` (5 preceding siblings ...)
2025-10-26 19:27 ` [PATCH v7 bpf-next 06/12] bpf, x86: add support for indirect jumps Anton Protopopov
@ 2025-10-26 19:27 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 08/12] bpf, docs: do not state that indirect jumps are not supported Anton Protopopov
` (4 subsequent siblings)
11 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:27 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
Add support for indirect jump instruction.
Example output from bpftool:
0: (79) r3 = *(u64 *)(r1 +0)
1: (25) if r3 > 0x4 goto pc+666
2: (67) r3 <<= 3
3: (18) r1 = 0xffffbeefspameggs
5: (0f) r1 += r3
6: (79) r1 = *(u64 *)(r1 +0)
7: (0d) gotox r1
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
---
kernel/bpf/disasm.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c
index 20883c6b1546..f8a3c7eb451e 100644
--- a/kernel/bpf/disasm.c
+++ b/kernel/bpf/disasm.c
@@ -358,6 +358,9 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs,
} else if (insn->code == (BPF_JMP | BPF_JA)) {
verbose(cbs->private_data, "(%02x) goto pc%+d\n",
insn->code, insn->off);
+ } else if (insn->code == (BPF_JMP | BPF_JA | BPF_X)) {
+ verbose(cbs->private_data, "(%02x) gotox r%d\n",
+ insn->code, insn->dst_reg);
} else if (insn->code == (BPF_JMP | BPF_JCOND) &&
insn->src_reg == BPF_MAY_GOTO) {
verbose(cbs->private_data, "(%02x) may_goto pc%+d\n",
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 08/12] bpf, docs: do not state that indirect jumps are not supported
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
` (6 preceding siblings ...)
2025-10-26 19:27 ` [PATCH v7 bpf-next 07/12] bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X Anton Protopopov
@ 2025-10-26 19:27 ` Anton Protopopov
2025-10-27 6:30 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps Anton Protopopov
` (3 subsequent siblings)
11 siblings, 1 reply; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:27 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
The linux-notes.rst states that indirect jump instruction "is not
currently supported by the verifier". Remove this part as outdated.
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
Documentation/bpf/linux-notes.rst | 8 --------
1 file changed, 8 deletions(-)
diff --git a/Documentation/bpf/linux-notes.rst b/Documentation/bpf/linux-notes.rst
index 00d2693de025..64ac146a926f 100644
--- a/Documentation/bpf/linux-notes.rst
+++ b/Documentation/bpf/linux-notes.rst
@@ -12,14 +12,6 @@ Byte swap instructions
``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively.
-Jump instructions
-=================
-
-``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function
-integer would be read from a specified register, is not currently supported
-by the verifier. Any programs with this instruction will fail to load
-until such support is added.
-
Maps
====
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
` (7 preceding siblings ...)
2025-10-26 19:27 ` [PATCH v7 bpf-next 08/12] bpf, docs: do not state that indirect jumps are not supported Anton Protopopov
@ 2025-10-26 19:27 ` Anton Protopopov
2025-10-26 20:15 ` Anton Protopopov
` (2 more replies)
2025-10-26 19:27 ` [PATCH v7 bpf-next 10/12] bpftool: Recognize insn_array map type Anton Protopopov
` (2 subsequent siblings)
11 siblings, 3 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:27 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
For v4 instruction set LLVM is allowed to generate indirect jumps for
switch statements and for 'goto *rX' assembly. Every such a jump will
be accompanied by necessary metadata, e.g. (`llvm-objdump -Sr ...`):
0: r2 = 0x0 ll
0000000000000030: R_BPF_64_64 BPF.JT.0.0
Here BPF.JT.1.0 is a symbol residing in the .jumptables section:
Symbol table:
4: 0000000000000000 240 OBJECT GLOBAL DEFAULT 4 BPF.JT.0.0
The -bpf-min-jump-table-entries llvm option may be used to control the
minimal size of a switch which will be converted to an indirect jumps.
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
tools/lib/bpf/libbpf.c | 249 +++++++++++++++++++++++++++++++-
tools/lib/bpf/libbpf_internal.h | 4 +
tools/lib/bpf/libbpf_probes.c | 4 +
tools/lib/bpf/linker.c | 9 +-
4 files changed, 263 insertions(+), 3 deletions(-)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index b90574f39d1c..2127f11c9ef2 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -190,6 +190,7 @@ static const char * const map_type_name[] = {
[BPF_MAP_TYPE_USER_RINGBUF] = "user_ringbuf",
[BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage",
[BPF_MAP_TYPE_ARENA] = "arena",
+ [BPF_MAP_TYPE_INSN_ARRAY] = "insn_array",
};
static const char * const prog_type_name[] = {
@@ -369,6 +370,7 @@ enum reloc_type {
RELO_EXTERN_CALL,
RELO_SUBPROG_ADDR,
RELO_CORE,
+ RELO_INSN_ARRAY,
};
struct reloc_desc {
@@ -379,7 +381,16 @@ struct reloc_desc {
struct {
int map_idx;
int sym_off;
- int ext_idx;
+ /*
+ * The following two fields can be unionized, as the
+ * ext_idx field is used for extern symbols, and the
+ * sym_size is used for jump tables, which are never
+ * extern
+ */
+ union {
+ int ext_idx;
+ int sym_size;
+ };
};
};
};
@@ -421,6 +432,11 @@ struct bpf_sec_def {
libbpf_prog_attach_fn_t prog_attach_fn;
};
+struct bpf_light_subprog {
+ __u32 sec_insn_off;
+ __u32 sub_insn_off;
+};
+
/*
* bpf_prog should be a better name but it has been used in
* linux/filter.h.
@@ -494,6 +510,9 @@ struct bpf_program {
__u32 line_info_cnt;
__u32 prog_flags;
__u8 hash[SHA256_DIGEST_LENGTH];
+
+ struct bpf_light_subprog *subprogs;
+ __u32 subprog_cnt;
};
struct bpf_struct_ops {
@@ -667,6 +686,7 @@ struct elf_state {
int symbols_shndx;
bool has_st_ops;
int arena_data_shndx;
+ int jumptables_data_shndx;
};
struct usdt_manager;
@@ -738,6 +758,16 @@ struct bpf_object {
void *arena_data;
size_t arena_data_sz;
+ void *jumptables_data;
+ size_t jumptables_data_sz;
+
+ struct {
+ struct bpf_program *prog;
+ int sym_off;
+ int fd;
+ } *jumptable_maps;
+ size_t jumptable_map_cnt;
+
struct kern_feature_cache *feat_cache;
char *token_path;
int token_fd;
@@ -764,6 +794,7 @@ void bpf_program__unload(struct bpf_program *prog)
zfree(&prog->func_info);
zfree(&prog->line_info);
+ zfree(&prog->subprogs);
}
static void bpf_program__exit(struct bpf_program *prog)
@@ -3942,6 +3973,13 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
} else if (strcmp(name, ARENA_SEC) == 0) {
obj->efile.arena_data = data;
obj->efile.arena_data_shndx = idx;
+ } else if (strcmp(name, JUMPTABLES_SEC) == 0) {
+ obj->jumptables_data = malloc(data->d_size);
+ if (!obj->jumptables_data)
+ return -ENOMEM;
+ memcpy(obj->jumptables_data, data->d_buf, data->d_size);
+ obj->jumptables_data_sz = data->d_size;
+ obj->efile.jumptables_data_shndx = idx;
} else {
pr_info("elf: skipping unrecognized data section(%d) %s\n",
idx, name);
@@ -4634,6 +4672,16 @@ static int bpf_program__record_reloc(struct bpf_program *prog,
return 0;
}
+ /* jump table data relocation */
+ if (shdr_idx == obj->efile.jumptables_data_shndx) {
+ reloc_desc->type = RELO_INSN_ARRAY;
+ reloc_desc->insn_idx = insn_idx;
+ reloc_desc->map_idx = -1;
+ reloc_desc->sym_off = sym->st_value;
+ reloc_desc->sym_size = sym->st_size;
+ return 0;
+ }
+
/* generic map reference relocation */
if (type == LIBBPF_MAP_UNSPEC) {
if (!bpf_object__shndx_is_maps(obj, shdr_idx)) {
@@ -6144,6 +6192,157 @@ static void poison_kfunc_call(struct bpf_program *prog, int relo_idx,
insn->imm = POISON_CALL_KFUNC_BASE + ext_idx;
}
+static int find_jt_map(struct bpf_object *obj, struct bpf_program *prog, int sym_off)
+{
+ size_t i;
+
+ for (i = 0; i < obj->jumptable_map_cnt; i++) {
+ /*
+ * This might happen that same offset is used for two different
+ * programs (as jump tables can be the same). However, for
+ * different programs different maps should be created.
+ */
+ if (obj->jumptable_maps[i].sym_off == sym_off &&
+ obj->jumptable_maps[i].prog == prog)
+ return obj->jumptable_maps[i].fd;
+ }
+
+ return -ENOENT;
+}
+
+static int add_jt_map(struct bpf_object *obj, struct bpf_program *prog, int sym_off, int map_fd)
+{
+ size_t new_cnt = obj->jumptable_map_cnt + 1;
+ size_t size = sizeof(obj->jumptable_maps[0]);
+ void *tmp;
+
+ tmp = libbpf_reallocarray(obj->jumptable_maps, new_cnt, size);
+ if (!tmp)
+ return -ENOMEM;
+
+ obj->jumptable_maps = tmp;
+ obj->jumptable_maps[new_cnt - 1].prog = prog;
+ obj->jumptable_maps[new_cnt - 1].sym_off = sym_off;
+ obj->jumptable_maps[new_cnt - 1].fd = map_fd;
+ obj->jumptable_map_cnt = new_cnt;
+
+ return 0;
+}
+
+static int find_subprog_idx(struct bpf_program *prog, int insn_idx)
+{
+ int i;
+
+ if (insn_idx < 0 || insn_idx >= prog->insns_cnt)
+ return -1;
+
+ for (i = prog->subprog_cnt - 1; i >= 0; i--) {
+ if (insn_idx >= prog->subprogs[i].sub_insn_off)
+ return i;
+ }
+
+ return -1;
+}
+
+static int create_jt_map(struct bpf_object *obj, struct bpf_program *prog, struct reloc_desc *relo)
+{
+ const __u32 jt_entry_size = 8;
+ int sym_off = relo->sym_off;
+ int jt_size = relo->sym_size;
+ __u32 max_entries = jt_size / jt_entry_size;
+ __u32 value_size = sizeof(struct bpf_insn_array_value);
+ struct bpf_insn_array_value val = {};
+ int subprog_idx;
+ int map_fd, err;
+ __u64 insn_off;
+ __u64 *jt;
+ __u32 i;
+
+ map_fd = find_jt_map(obj, prog, sym_off);
+ if (map_fd >= 0)
+ return map_fd;
+
+ if (sym_off % jt_entry_size) {
+ pr_warn("jumptable start %d should be multiple of %u\n",
+ sym_off, jt_entry_size);
+ return -EINVAL;
+ }
+
+ if (jt_size % jt_entry_size) {
+ pr_warn("jumptable size %d should be multiple of %u\n",
+ jt_size, jt_entry_size);
+ return -EINVAL;
+ }
+
+ map_fd = bpf_map_create(BPF_MAP_TYPE_INSN_ARRAY, ".jumptables",
+ 4, value_size, max_entries, NULL);
+ if (map_fd < 0)
+ return map_fd;
+
+ if (!obj->jumptables_data) {
+ pr_warn("map '.jumptables': ELF file is missing jump table data\n");
+ err = -EINVAL;
+ goto err_close;
+ }
+ if (sym_off + jt_size > obj->jumptables_data_sz) {
+ pr_warn("jumptables_data size is %zd, trying to access %d\n",
+ obj->jumptables_data_sz, sym_off + jt_size);
+ err = -EINVAL;
+ goto err_close;
+ }
+
+ jt = (__u64 *)(obj->jumptables_data + sym_off);
+ for (i = 0; i < max_entries; i++) {
+ /*
+ * The offset should be made to be relative to the beginning of
+ * the main function, not the subfunction.
+ */
+ insn_off = jt[i]/sizeof(struct bpf_insn);
+ if (!prog->subprogs) {
+ insn_off -= prog->sec_insn_off;
+ } else {
+ subprog_idx = find_subprog_idx(prog, relo->insn_idx);
+ if (subprog_idx < 0) {
+ pr_warn("invalid jump insn idx[%d]: %d, no subprog found\n",
+ i, relo->insn_idx);
+ err = -EINVAL;
+ }
+ insn_off -= prog->subprogs[subprog_idx].sec_insn_off;
+ insn_off += prog->subprogs[subprog_idx].sub_insn_off;
+ }
+
+ /*
+ * LLVM-generated jump tables contain u64 records, however
+ * should contain values that fit in u32.
+ */
+ if (insn_off > UINT32_MAX) {
+ pr_warn("invalid jump table value %llx at offset %d\n",
+ jt[i], sym_off + i);
+ err = -EINVAL;
+ goto err_close;
+ }
+
+ val.orig_off = insn_off;
+ err = bpf_map_update_elem(map_fd, &i, &val, 0);
+ if (err)
+ goto err_close;
+ }
+
+ err = bpf_map_freeze(map_fd);
+ if (err)
+ goto err_close;
+
+ err = add_jt_map(obj, prog, sym_off, map_fd);
+ if (err)
+ goto err_close;
+
+ return map_fd;
+
+err_close:
+ close(map_fd);
+ return err;
+}
+
/* Relocate data references within program code:
* - map references;
* - global variable references;
@@ -6235,6 +6434,20 @@ bpf_object__relocate_data(struct bpf_object *obj, struct bpf_program *prog)
case RELO_CORE:
/* will be handled by bpf_program_record_relos() */
break;
+ case RELO_INSN_ARRAY: {
+ int map_fd;
+
+ map_fd = create_jt_map(obj, prog, relo);
+ if (map_fd < 0) {
+ pr_warn("prog '%s': relo #%d: can't create jump table: sym_off %u\n",
+ prog->name, i, relo->sym_off);
+ return map_fd;
+ }
+ insn[0].src_reg = BPF_PSEUDO_MAP_VALUE;
+ insn->imm = map_fd;
+ insn->off = 0;
+ }
+ break;
default:
pr_warn("prog '%s': relo #%d: bad relo type %d\n",
prog->name, i, relo->type);
@@ -6432,6 +6645,24 @@ static int append_subprog_relos(struct bpf_program *main_prog, struct bpf_progra
return 0;
}
+static int save_subprog_offsets(struct bpf_program *main_prog, struct bpf_program *subprog)
+{
+ size_t size = sizeof(main_prog->subprogs[0]);
+ int new_cnt = main_prog->subprog_cnt + 1;
+ void *tmp;
+
+ tmp = libbpf_reallocarray(main_prog->subprogs, new_cnt, size);
+ if (!tmp)
+ return -ENOMEM;
+
+ main_prog->subprogs = tmp;
+ main_prog->subprogs[new_cnt - 1].sec_insn_off = subprog->sec_insn_off;
+ main_prog->subprogs[new_cnt - 1].sub_insn_off = subprog->sub_insn_off;
+ main_prog->subprog_cnt = new_cnt;
+
+ return 0;
+}
+
static int
bpf_object__append_subprog_code(struct bpf_object *obj, struct bpf_program *main_prog,
struct bpf_program *subprog)
@@ -6461,6 +6692,15 @@ bpf_object__append_subprog_code(struct bpf_object *obj, struct bpf_program *main
err = append_subprog_relos(main_prog, subprog);
if (err)
return err;
+
+ /* Save subprogram offsets */
+ err = save_subprog_offsets(main_prog, subprog);
+ if (err) {
+ pr_warn("prog '%s': failed to add subprog offsets: %s\n",
+ main_prog->name, errstr(err));
+ return err;
+ }
+
return 0;
}
@@ -9228,6 +9468,13 @@ void bpf_object__close(struct bpf_object *obj)
zfree(&obj->arena_data);
+ zfree(&obj->jumptables_data);
+ obj->jumptables_data_sz = 0;
+
+ for (i = 0; i < obj->jumptable_map_cnt; i++)
+ close(obj->jumptable_maps[i].fd);
+ zfree(&obj->jumptable_maps);
+
free(obj);
}
diff --git a/tools/lib/bpf/libbpf_internal.h b/tools/lib/bpf/libbpf_internal.h
index 35b2527bedec..93bc39bd1307 100644
--- a/tools/lib/bpf/libbpf_internal.h
+++ b/tools/lib/bpf/libbpf_internal.h
@@ -74,6 +74,10 @@
#define ELF64_ST_VISIBILITY(o) ((o) & 0x03)
#endif
+#ifndef JUMPTABLES_SEC
+#define JUMPTABLES_SEC ".jumptables"
+#endif
+
#define BTF_INFO_ENC(kind, kind_flag, vlen) \
((!!(kind_flag) << 31) | ((kind) << 24) | ((vlen) & BTF_MAX_VLEN))
#define BTF_TYPE_ENC(name, info, size_or_type) (name), (info), (size_or_type)
diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c
index 9dfbe7750f56..bccf4bb747e1 100644
--- a/tools/lib/bpf/libbpf_probes.c
+++ b/tools/lib/bpf/libbpf_probes.c
@@ -364,6 +364,10 @@ static int probe_map_create(enum bpf_map_type map_type)
case BPF_MAP_TYPE_SOCKHASH:
case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY:
break;
+ case BPF_MAP_TYPE_INSN_ARRAY:
+ key_size = sizeof(__u32);
+ value_size = sizeof(struct bpf_insn_array_value);
+ break;
case BPF_MAP_TYPE_UNSPEC:
default:
return -EOPNOTSUPP;
diff --git a/tools/lib/bpf/linker.c b/tools/lib/bpf/linker.c
index 56ae77047bc3..f6ca3b23b17a 100644
--- a/tools/lib/bpf/linker.c
+++ b/tools/lib/bpf/linker.c
@@ -27,6 +27,7 @@
#include "strset.h"
#define BTF_EXTERN_SEC ".extern"
+#define JUMPTABLES_REL_SEC ".rel.jumptables"
struct src_sec {
const char *sec_name;
@@ -2025,6 +2026,9 @@ static int linker_append_elf_sym(struct bpf_linker *linker, struct src_obj *obj,
obj->sym_map[src_sym_idx] = dst_sec->sec_sym_idx;
return 0;
}
+
+ if (strcmp(src_sec->sec_name, JUMPTABLES_SEC) == 0)
+ goto add_sym;
}
if (sym_bind == STB_LOCAL)
@@ -2271,8 +2275,9 @@ static int linker_append_elf_relos(struct bpf_linker *linker, struct src_obj *ob
insn->imm += sec->dst_off / sizeof(struct bpf_insn);
else
insn->imm += sec->dst_off;
- } else {
- pr_warn("relocation against STT_SECTION in non-exec section is not supported!\n");
+ } else if (strcmp(src_sec->sec_name, JUMPTABLES_REL_SEC) != 0) {
+ pr_warn("relocation against STT_SECTION in section %s is not supported!\n",
+ src_sec->sec_name);
return -EINVAL;
}
}
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 10/12] bpftool: Recognize insn_array map type
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
` (8 preceding siblings ...)
2025-10-26 19:27 ` [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps Anton Protopopov
@ 2025-10-26 19:27 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 11/12] selftests/bpf: add new verifier_gotox test Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 12/12] selftests/bpf: add C-level selftests for indirect jumps Anton Protopopov
11 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:27 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
Teach bpftool to recognize instruction array map type.
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
Acked-by: Quentin Monnet <qmo@kernel.org>
---
tools/bpf/bpftool/Documentation/bpftool-map.rst | 3 ++-
tools/bpf/bpftool/map.c | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/tools/bpf/bpftool/Documentation/bpftool-map.rst b/tools/bpf/bpftool/Documentation/bpftool-map.rst
index 252e4c538edb..1af3305ea2b2 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-map.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-map.rst
@@ -55,7 +55,8 @@ MAP COMMANDS
| | **devmap** | **devmap_hash** | **sockmap** | **cpumap** | **xskmap** | **sockhash**
| | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage**
| | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage**
-| | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena** }
+| | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena**
+| | **insn_array** }
DESCRIPTION
===========
diff --git a/tools/bpf/bpftool/map.c b/tools/bpf/bpftool/map.c
index c9de44a45778..7ebf7dbcfba4 100644
--- a/tools/bpf/bpftool/map.c
+++ b/tools/bpf/bpftool/map.c
@@ -1477,7 +1477,8 @@ static int do_help(int argc, char **argv)
" devmap | devmap_hash | sockmap | cpumap | xskmap | sockhash |\n"
" cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n"
" queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n"
- " task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena }\n"
+ " task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena |\n"
+ " insn_array }\n"
" " HELP_SPEC_OPTIONS " |\n"
" {-f|--bpffs} | {-n|--nomount} }\n"
"",
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 11/12] selftests/bpf: add new verifier_gotox test
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
` (9 preceding siblings ...)
2025-10-26 19:27 ` [PATCH v7 bpf-next 10/12] bpftool: Recognize insn_array map type Anton Protopopov
@ 2025-10-26 19:27 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 12/12] selftests/bpf: add C-level selftests for indirect jumps Anton Protopopov
11 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:27 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
Add a set of tests to validate core gotox functionality
without need to rely on compilers.
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
.../selftests/bpf/prog_tests/verifier.c | 2 +
.../selftests/bpf/progs/verifier_gotox.c | 317 ++++++++++++++++++
2 files changed, 319 insertions(+)
create mode 100644 tools/testing/selftests/bpf/progs/verifier_gotox.c
diff --git a/tools/testing/selftests/bpf/prog_tests/verifier.c b/tools/testing/selftests/bpf/prog_tests/verifier.c
index c0e8ffdaa484..4b4b081b46cc 100644
--- a/tools/testing/selftests/bpf/prog_tests/verifier.c
+++ b/tools/testing/selftests/bpf/prog_tests/verifier.c
@@ -35,6 +35,7 @@
#include "verifier_global_subprogs.skel.h"
#include "verifier_global_ptr_args.skel.h"
#include "verifier_gotol.skel.h"
+#include "verifier_gotox.skel.h"
#include "verifier_helper_access_var_len.skel.h"
#include "verifier_helper_packet_access.skel.h"
#include "verifier_helper_restricted.skel.h"
@@ -173,6 +174,7 @@ void test_verifier_div_overflow(void) { RUN(verifier_div_overflow); }
void test_verifier_global_subprogs(void) { RUN(verifier_global_subprogs); }
void test_verifier_global_ptr_args(void) { RUN(verifier_global_ptr_args); }
void test_verifier_gotol(void) { RUN(verifier_gotol); }
+void test_verifier_gotox(void) { RUN(verifier_gotox); }
void test_verifier_helper_access_var_len(void) { RUN(verifier_helper_access_var_len); }
void test_verifier_helper_packet_access(void) { RUN(verifier_helper_packet_access); }
void test_verifier_helper_restricted(void) { RUN(verifier_helper_restricted); }
diff --git a/tools/testing/selftests/bpf/progs/verifier_gotox.c b/tools/testing/selftests/bpf/progs/verifier_gotox.c
new file mode 100644
index 000000000000..0f1032783904
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/verifier_gotox.c
@@ -0,0 +1,317 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2025 Isovalent */
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+#include "bpf_misc.h"
+#include "../../../include/linux/filter.h"
+
+#ifdef __TARGET_ARCH_x86
+
+#define DEFINE_SIMPLE_JUMP_TABLE_PROG(NAME, SRC_REG, OFF, IMM, OUTCOME) \
+ \
+ SEC("socket") \
+ OUTCOME \
+ __naked void jump_table_ ## NAME(void) \
+ { \
+ asm volatile (" \
+ .pushsection .jumptables,\"\",@progbits; \
+ jt0_%=: \
+ .quad ret0_%=; \
+ .quad ret1_%=; \
+ .size jt0_%=, 16; \
+ .global jt0_%=; \
+ .popsection; \
+ \
+ r0 = jt0_%= ll; \
+ r0 += 8; \
+ r0 = *(u64 *)(r0 + 0); \
+ .8byte %[gotox_r0]; \
+ ret0_%=: \
+ r0 = 0; \
+ exit; \
+ ret1_%=: \
+ r0 = 1; \
+ exit; \
+ " : \
+ : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, (SRC_REG), (OFF) , (IMM))) \
+ : __clobber_all); \
+ }
+
+/*
+ * The first program which doesn't use reserved fields
+ * loads and works properly. The rest fail to load.
+ */
+DEFINE_SIMPLE_JUMP_TABLE_PROG(ok, BPF_REG_0, 0, 0, __success __retval(1))
+DEFINE_SIMPLE_JUMP_TABLE_PROG(reserved_field_src_reg, BPF_REG_1, 0, 0, __failure __msg("BPF_JA|BPF_X uses reserved fields"))
+DEFINE_SIMPLE_JUMP_TABLE_PROG(reserved_field_non_zero_off, BPF_REG_0, 1, 0, __failure __msg("BPF_JA|BPF_X uses reserved fields"))
+DEFINE_SIMPLE_JUMP_TABLE_PROG(reserved_field_non_zero_imm, BPF_REG_0, 0, 1, __failure __msg("BPF_JA|BPF_X uses reserved fields"))
+
+/*
+ * Gotox is forbidden when there is no jump table loaded
+ * which points to the sub-function where the gotox is used
+ */
+SEC("socket")
+__failure __msg("no jump tables found for subprog starting at 0")
+__naked void jump_table_no_jump_table(void)
+{
+ asm volatile (" \
+ .8byte %[gotox_r0]; \
+ r0 = 1; \
+ exit; \
+" : \
+ : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+ : __clobber_all);
+}
+
+/*
+ * Incorrect type of the target register, only PTR_TO_INSN allowed
+ */
+SEC("socket")
+__failure __msg("R1 has type scalar, expected PTR_TO_INSN")
+__naked void jump_table_incorrect_dst_reg_type(void)
+{
+ asm volatile (" \
+ .pushsection .jumptables,\"\",@progbits; \
+jt0_%=: \
+ .quad ret0_%=; \
+ .quad ret1_%=; \
+ .size jt0_%=, 16; \
+ .global jt0_%=; \
+ .popsection; \
+ \
+ r0 = jt0_%= ll; \
+ r0 += 8; \
+ r0 = *(u64 *)(r0 + 0); \
+ r1 = 42; \
+ .8byte %[gotox_r1]; \
+ ret0_%=: \
+ r0 = 0; \
+ exit; \
+ ret1_%=: \
+ r0 = 1; \
+ exit; \
+" : \
+ : __imm_insn(gotox_r1, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_1, 0, 0 , 0))
+ : __clobber_all);
+}
+
+#define DEFINE_INVALID_SIZE_PROG(READ_SIZE, OUTCOME) \
+ \
+ SEC("socket") \
+ OUTCOME \
+ __naked void jump_table_invalid_read_size_ ## READ_SIZE(void) \
+ { \
+ asm volatile (" \
+ .pushsection .jumptables,\"\",@progbits; \
+ jt0_%=: \
+ .quad ret0_%=; \
+ .quad ret1_%=; \
+ .size jt0_%=, 16; \
+ .global jt0_%=; \
+ .popsection; \
+ \
+ r0 = jt0_%= ll; \
+ r0 += 8; \
+ r0 = *(" #READ_SIZE " *)(r0 + 0); \
+ .8byte %[gotox_r0]; \
+ ret0_%=: \
+ r0 = 0; \
+ exit; \
+ ret1_%=: \
+ r0 = 1; \
+ exit; \
+ " : \
+ : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0)) \
+ : __clobber_all); \
+ }
+
+DEFINE_INVALID_SIZE_PROG(u32, __failure __msg("Invalid read of 4 bytes from insn_array"))
+DEFINE_INVALID_SIZE_PROG(u16, __failure __msg("Invalid read of 2 bytes from insn_array"))
+DEFINE_INVALID_SIZE_PROG(u8, __failure __msg("Invalid read of 1 bytes from insn_array"))
+
+SEC("socket")
+__failure __msg("misaligned value access off 0+1+0 size 8")
+__naked void jump_table_misaligned_access(void)
+{
+ asm volatile (" \
+ .pushsection .jumptables,\"\",@progbits; \
+jt0_%=: \
+ .quad ret0_%=; \
+ .quad ret1_%=; \
+ .size jt0_%=, 16; \
+ .global jt0_%=; \
+ .popsection; \
+ \
+ r0 = jt0_%= ll; \
+ r0 += 1; \
+ r0 = *(u64 *)(r0 + 0); \
+ .8byte %[gotox_r0]; \
+ ret0_%=: \
+ r0 = 0; \
+ exit; \
+ ret1_%=: \
+ r0 = 1; \
+ exit; \
+" : \
+ : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+ : __clobber_all);
+}
+
+SEC("socket")
+__failure __msg("invalid access to map value, value_size=16 off=24 size=8")
+__naked void jump_table_invalid_mem_acceess_pos(void)
+{
+ asm volatile (" \
+ .pushsection .jumptables,\"\",@progbits; \
+jt0_%=: \
+ .quad ret0_%=; \
+ .quad ret1_%=; \
+ .size jt0_%=, 16; \
+ .global jt0_%=; \
+ .popsection; \
+ \
+ r0 = jt0_%= ll; \
+ r0 += 24; \
+ r0 = *(u64 *)(r0 + 0); \
+ .8byte %[gotox_r0]; \
+ ret0_%=: \
+ r0 = 0; \
+ exit; \
+ ret1_%=: \
+ r0 = 1; \
+ exit; \
+" : \
+ : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+ : __clobber_all);
+}
+
+SEC("socket")
+__failure __msg("invalid access to map value, value_size=16 off=-24 size=8")
+__naked void jump_table_invalid_mem_acceess_neg(void)
+{
+ asm volatile (" \
+ .pushsection .jumptables,\"\",@progbits; \
+jt0_%=: \
+ .quad ret0_%=; \
+ .quad ret1_%=; \
+ .size jt0_%=, 16; \
+ .global jt0_%=; \
+ .popsection; \
+ \
+ r0 = jt0_%= ll; \
+ r0 -= 24; \
+ r0 = *(u64 *)(r0 + 0); \
+ .8byte %[gotox_r0]; \
+ ret0_%=: \
+ r0 = 0; \
+ exit; \
+ ret1_%=: \
+ r0 = 1; \
+ exit; \
+" : \
+ : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+ : __clobber_all);
+}
+
+SEC("socket")
+__success __retval(1)
+__naked void jump_table_add_sub_ok(void)
+{
+ asm volatile (" \
+ .pushsection .jumptables,\"\",@progbits; \
+jt0_%=: \
+ .quad ret0_%=; \
+ .quad ret1_%=; \
+ .size jt0_%=, 16; \
+ .global jt0_%=; \
+ .popsection; \
+ \
+ r0 = jt0_%= ll; \
+ r0 -= 24; \
+ r0 += 32; \
+ r0 = *(u64 *)(r0 + 0); \
+ .8byte %[gotox_r0]; \
+ ret0_%=: \
+ r0 = 0; \
+ exit; \
+ ret1_%=: \
+ r0 = 1; \
+ exit; \
+" : \
+ : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+ : __clobber_all);
+}
+
+SEC("socket")
+__failure __msg("writes into insn_array not allowed")
+__naked void jump_table_no_writes(void)
+{
+ asm volatile (" \
+ .pushsection .jumptables,\"\",@progbits; \
+jt0_%=: \
+ .quad ret0_%=; \
+ .quad ret1_%=; \
+ .size jt0_%=, 16; \
+ .global jt0_%=; \
+ .popsection; \
+ \
+ r0 = jt0_%= ll; \
+ r0 += 8; \
+ r1 = 0xbeef; \
+ *(u64 *)(r0 + 0) = r1; \
+ .8byte %[gotox_r0]; \
+ ret0_%=: \
+ r0 = 0; \
+ exit; \
+ ret1_%=: \
+ r0 = 1; \
+ exit; \
+" : \
+ : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
+ : __clobber_all);
+}
+
+#define DEFINE_JUMP_TABLE_USE_REG(REG) \
+ SEC("socket") \
+ __success __retval(1) \
+ __naked void jump_table_use_reg_r ## REG(void) \
+ { \
+ asm volatile (" \
+ .pushsection .jumptables,\"\",@progbits; \
+ jt0_%=: \
+ .quad ret0_%=; \
+ .quad ret1_%=; \
+ .size jt0_%=, 16; \
+ .global jt0_%=; \
+ .popsection; \
+ \
+ r0 = jt0_%= ll; \
+ r0 += 8; \
+ r" #REG " = *(u64 *)(r0 + 0); \
+ .8byte %[gotox_rX]; \
+ ret0_%=: \
+ r0 = 0; \
+ exit; \
+ ret1_%=: \
+ r0 = 1; \
+ exit; \
+ " : \
+ : __imm_insn(gotox_rX, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_ ## REG, 0, 0 , 0)) \
+ : __clobber_all); \
+ }
+
+DEFINE_JUMP_TABLE_USE_REG(0)
+DEFINE_JUMP_TABLE_USE_REG(1)
+DEFINE_JUMP_TABLE_USE_REG(2)
+DEFINE_JUMP_TABLE_USE_REG(3)
+DEFINE_JUMP_TABLE_USE_REG(4)
+DEFINE_JUMP_TABLE_USE_REG(5)
+DEFINE_JUMP_TABLE_USE_REG(6)
+DEFINE_JUMP_TABLE_USE_REG(7)
+DEFINE_JUMP_TABLE_USE_REG(8)
+DEFINE_JUMP_TABLE_USE_REG(9)
+
+#endif /* __TARGET_ARCH_x86 */
+
+char _license[] SEC("license") = "GPL";
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* [PATCH v7 bpf-next 12/12] selftests/bpf: add C-level selftests for indirect jumps
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
` (10 preceding siblings ...)
2025-10-26 19:27 ` [PATCH v7 bpf-next 11/12] selftests/bpf: add new verifier_gotox test Anton Protopopov
@ 2025-10-26 19:27 ` Anton Protopopov
2025-10-27 23:25 ` Eduard Zingerman
11 siblings, 1 reply; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 19:27 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
Cc: Anton Protopopov
Add C-level selftests for indirect jumps to validate LLVM and libbpf
functionality. The tests are intentionally disabled, to be run
locally by developers, but will not make the CI red.
Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
---
tools/testing/selftests/bpf/Makefile | 4 +-
.../selftests/bpf/prog_tests/bpf_gotox.c | 185 ++++++++
tools/testing/selftests/bpf/progs/bpf_gotox.c | 402 ++++++++++++++++++
3 files changed, 590 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
create mode 100644 tools/testing/selftests/bpf/progs/bpf_gotox.c
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 7437c325179e..a897cb31fe6d 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -454,7 +454,9 @@ BPF_CFLAGS = -g -Wall -Werror -D__TARGET_ARCH_$(SRCARCH) $(MENDIAN) \
-I$(abspath $(OUTPUT)/../usr/include) \
-std=gnu11 \
-fno-strict-aliasing \
- -Wno-compare-distinct-pointer-types
+ -Wno-compare-distinct-pointer-types \
+ -Wno-initializer-overrides \
+ #
# TODO: enable me -Wsign-compare
CLANG_CFLAGS = $(CLANG_SYS_INCLUDES)
diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c b/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
new file mode 100644
index 000000000000..4394654ac75a
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/bpf_gotox.c
@@ -0,0 +1,185 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <test_progs.h>
+
+#include <linux/if_ether.h>
+#include <linux/in.h>
+#include <linux/ip.h>
+#include <linux/ipv6.h>
+#include <linux/in6.h>
+#include <linux/udp.h>
+#include <linux/tcp.h>
+
+#include <sys/syscall.h>
+#include <bpf/bpf.h>
+
+#include "bpf_gotox.skel.h"
+
+/* Disable tests for now, as CI runs with LLVM-20 */
+#if 0
+static void __test_run(struct bpf_program *prog, void *ctx_in, size_t ctx_size_in)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, topts,
+ .ctx_in = ctx_in,
+ .ctx_size_in = ctx_size_in,
+ );
+ int err, prog_fd;
+
+ prog_fd = bpf_program__fd(prog);
+ err = bpf_prog_test_run_opts(prog_fd, &topts);
+ ASSERT_OK(err, "test_run_opts err");
+}
+
+static void check_simple(struct bpf_gotox *skel,
+ struct bpf_program *prog,
+ __u64 ctx_in,
+ __u64 expected)
+{
+ skel->bss->ret_user = 0;
+
+ __test_run(prog, &ctx_in, sizeof(ctx_in));
+
+ if (!ASSERT_EQ(skel->bss->ret_user, expected, "skel->bss->ret_user"))
+ return;
+}
+
+static void check_simple_fentry(struct bpf_gotox *skel,
+ struct bpf_program *prog,
+ __u64 ctx_in,
+ __u64 expected)
+{
+ skel->bss->in_user = ctx_in;
+ skel->bss->ret_user = 0;
+
+ /* trigger */
+ usleep(1);
+
+ if (!ASSERT_EQ(skel->bss->ret_user, expected, "skel->bss->ret_user"))
+ return;
+}
+
+/* validate that for two loads of the same jump table libbpf generates only one map */
+static void check_one_map_two_jumps(struct bpf_gotox *skel)
+{
+ struct bpf_prog_info prog_info;
+ struct bpf_map_info map_info;
+ __u32 len;
+ __u32 map_ids[16];
+ int prog_fd, map_fd;
+ int ret;
+ int i;
+ bool seen = false;
+
+ memset(&prog_info, 0, sizeof(prog_info));
+ prog_info.map_ids = (long)map_ids;
+ prog_info.nr_map_ids = ARRAY_SIZE(map_ids);
+ prog_fd = bpf_program__fd(skel->progs.one_map_two_jumps);
+ if (!ASSERT_GE(prog_fd, 0, "bpf_program__fd(one_map_two_jumps)"))
+ return;
+
+ len = sizeof(prog_info);
+ ret = bpf_obj_get_info_by_fd(prog_fd, &prog_info, &len);
+ if (!ASSERT_OK(ret, "bpf_obj_get_info_by_fd(prog_fd)"))
+ return;
+
+ for (i = 0; i < prog_info.nr_map_ids; i++) {
+ map_fd = bpf_map_get_fd_by_id(map_ids[i]);
+ if (!ASSERT_GE(map_fd, 0, "bpf_program__fd(one_map_two_jumps)"))
+ return;
+
+ len = sizeof(map_info);
+ memset(&map_info, 0, len);
+ ret = bpf_obj_get_info_by_fd(map_fd, &map_info, &len);
+ if (!ASSERT_OK(ret, "bpf_obj_get_info_by_fd(map_fd)")) {
+ close(map_fd);
+ return;
+ }
+
+ if (map_info.type == BPF_MAP_TYPE_INSN_ARRAY) {
+ if (!ASSERT_EQ(seen, false, "more than one INSN_ARRAY map")) {
+ close(map_fd);
+ return;
+ }
+ seen = true;
+ }
+ close(map_fd);
+ }
+
+ ASSERT_EQ(seen, true, "no INSN_ARRAY map");
+}
+
+static void check_gotox_skel(struct bpf_gotox *skel)
+{
+ int i;
+ __u64 in[] = {0, 1, 2, 3, 4, 5, 77};
+ __u64 out[] = {2, 3, 4, 5, 7, 19, 19};
+ __u64 out2[] = {103, 104, 107, 205, 115, 1019, 1019};
+ __u64 in3[] = {0, 11, 27, 31, 22, 45, 99};
+ __u64 out3[] = {2, 3, 4, 5, 19, 19, 19};
+ __u64 in4[] = {0, 1, 2, 3, 4, 5, 77};
+ __u64 out4[] = {12, 15, 7 , 15, 12, 15, 15};
+
+ for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple(skel, skel->progs.simple_test, in[i], out[i]);
+
+ for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple(skel, skel->progs.simple_test2, in[i], out[i]);
+
+ for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple(skel, skel->progs.two_switches, in[i], out2[i]);
+
+ if (0) for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple(skel, skel->progs.big_jump_table, in3[i], out3[i]);
+
+ if (0) for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple(skel, skel->progs.one_jump_two_maps, in4[i], out4[i]);
+
+ for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple(skel, skel->progs.use_static_global1, in[i], out[i]);
+
+ for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple(skel, skel->progs.use_static_global2, in[i], out[i]);
+
+ for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple(skel, skel->progs.use_nonstatic_global1, in[i], out[i]);
+
+ for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple(skel, skel->progs.use_nonstatic_global2, in[i], out[i]);
+
+ bpf_program__attach(skel->progs.simple_test_other_sec);
+ for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple_fentry(skel, skel->progs.simple_test_other_sec, in[i], out[i]);
+
+ bpf_program__attach(skel->progs.use_static_global_other_sec);
+ for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple_fentry(skel, skel->progs.use_static_global_other_sec, in[i], out[i]);
+
+ bpf_program__attach(skel->progs.use_nonstatic_global_other_sec);
+ for (i = 0; i < ARRAY_SIZE(in); i++)
+ check_simple_fentry(skel, skel->progs.use_nonstatic_global_other_sec, in[i], out[i]);
+
+ if (0) check_one_map_two_jumps(skel);
+}
+
+void test_bpf_gotox(void)
+{
+ struct bpf_gotox *skel;
+ int ret;
+
+ skel = bpf_gotox__open();
+ if (!ASSERT_NEQ(skel, NULL, "bpf_gotox__open"))
+ return;
+
+ ret = bpf_gotox__load(skel);
+ if (!ASSERT_OK(ret, "bpf_gotox__load"))
+ return;
+
+ check_gotox_skel(skel);
+
+ bpf_gotox__destroy(skel);
+}
+#else
+void test_bpf_gotox(void)
+{
+}
+#endif
diff --git a/tools/testing/selftests/bpf/progs/bpf_gotox.c b/tools/testing/selftests/bpf/progs/bpf_gotox.c
new file mode 100644
index 000000000000..3c8ee363bda1
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/bpf_gotox.c
@@ -0,0 +1,402 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include <bpf/bpf_core_read.h>
+#include "bpf_misc.h"
+
+/* Disable tests for now, as CI runs with LLVM-20 */
+#if 0
+__u64 in_user;
+__u64 ret_user;
+
+struct simple_ctx {
+ __u64 x;
+};
+
+__u64 some_var;
+
+/*
+ * This function adds code which will be replaced by a different
+ * number of instructions by the verifier. This adds additional
+ * stress on testing the insn_array maps corresponding to indirect jumps.
+ */
+static __always_inline void adjust_insns(__u64 x)
+{
+ some_var ^= x + bpf_jiffies64();
+}
+
+SEC("syscall")
+int simple_test(struct simple_ctx *ctx)
+{
+ switch (ctx->x) {
+ case 0:
+ adjust_insns(ctx->x + 1);
+ ret_user = 2;
+ break;
+ case 1:
+ adjust_insns(ctx->x + 7);
+ ret_user = 3;
+ break;
+ case 2:
+ adjust_insns(ctx->x + 9);
+ ret_user = 4;
+ break;
+ case 3:
+ adjust_insns(ctx->x + 11);
+ ret_user = 5;
+ break;
+ case 4:
+ adjust_insns(ctx->x + 17);
+ ret_user = 7;
+ break;
+ default:
+ adjust_insns(ctx->x + 177);
+ ret_user = 19;
+ break;
+ }
+
+ return 0;
+}
+
+SEC("syscall")
+int simple_test2(struct simple_ctx *ctx)
+{
+ switch (ctx->x) {
+ case 0:
+ adjust_insns(ctx->x + 1);
+ ret_user = 2;
+ break;
+ case 1:
+ adjust_insns(ctx->x + 7);
+ ret_user = 3;
+ break;
+ case 2:
+ adjust_insns(ctx->x + 9);
+ ret_user = 4;
+ break;
+ case 3:
+ adjust_insns(ctx->x + 11);
+ ret_user = 5;
+ break;
+ case 4:
+ adjust_insns(ctx->x + 17);
+ ret_user = 7;
+ break;
+ default:
+ adjust_insns(ctx->x + 177);
+ ret_user = 19;
+ break;
+ }
+
+ return 0;
+}
+
+SEC("fentry/" SYS_PREFIX "sys_nanosleep")
+int simple_test_other_sec(struct pt_regs *ctx)
+{
+ __u64 x = in_user;
+
+ switch (x) {
+ case 0:
+ adjust_insns(x + 1);
+ ret_user = 2;
+ break;
+ case 1:
+ adjust_insns(x + 7);
+ ret_user = 3;
+ break;
+ case 2:
+ adjust_insns(x + 9);
+ ret_user = 4;
+ break;
+ case 3:
+ adjust_insns(x + 11);
+ ret_user = 5;
+ break;
+ case 4:
+ adjust_insns(x + 17);
+ ret_user = 7;
+ break;
+ default:
+ adjust_insns(x + 177);
+ ret_user = 19;
+ break;
+ }
+
+ return 0;
+}
+
+SEC("syscall")
+int two_switches(struct simple_ctx *ctx)
+{
+ switch (ctx->x) {
+ case 0:
+ adjust_insns(ctx->x + 1);
+ ret_user = 2;
+ break;
+ case 1:
+ adjust_insns(ctx->x + 7);
+ ret_user = 3;
+ break;
+ case 2:
+ adjust_insns(ctx->x + 9);
+ ret_user = 4;
+ break;
+ case 3:
+ adjust_insns(ctx->x + 11);
+ ret_user = 5;
+ break;
+ case 4:
+ adjust_insns(ctx->x + 17);
+ ret_user = 7;
+ break;
+ default:
+ adjust_insns(ctx->x + 177);
+ ret_user = 19;
+ break;
+ }
+
+ switch (ctx->x + !!ret_user) {
+ case 1:
+ adjust_insns(ctx->x + 7);
+ ret_user = 103;
+ break;
+ case 2:
+ adjust_insns(ctx->x + 9);
+ ret_user = 104;
+ break;
+ case 3:
+ adjust_insns(ctx->x + 11);
+ ret_user = 107;
+ break;
+ case 4:
+ adjust_insns(ctx->x + 11);
+ ret_user = 205;
+ break;
+ case 5:
+ adjust_insns(ctx->x + 11);
+ ret_user = 115;
+ break;
+ default:
+ adjust_insns(ctx->x + 177);
+ ret_user = 1019;
+ break;
+ }
+
+ return 0;
+}
+
+SEC("syscall")
+int big_jump_table(struct simple_ctx *ctx __attribute__((unused)))
+{
+ const void *const jt[256] = {
+ [0 ... 255] = &&default_label,
+ [0] = &&l0,
+ [11] = &&l11,
+ [27] = &&l27,
+ [31] = &&l31,
+ };
+
+ goto *jt[ctx->x & 0xff];
+
+l0:
+ adjust_insns(ctx->x + 1);
+ ret_user = 2;
+ return 0;
+
+l11:
+ adjust_insns(ctx->x + 7);
+ ret_user = 3;
+ return 0;
+
+l27:
+ adjust_insns(ctx->x + 9);
+ ret_user = 4;
+ return 0;
+
+l31:
+ adjust_insns(ctx->x + 11);
+ ret_user = 5;
+ return 0;
+
+default_label:
+ adjust_insns(ctx->x + 177);
+ ret_user = 19;
+ return 0;
+}
+
+SEC("syscall")
+int one_jump_two_maps(struct simple_ctx *ctx __attribute__((unused)))
+{
+ __label__ l1, l2, l3, l4;
+ void *jt1[2] = { &&l1, &&l2 };
+ void *jt2[2] = { &&l3, &&l4 };
+ unsigned int a = ctx->x % 2;
+ unsigned int b = (ctx->x / 2) % 2;
+ volatile int ret = 0;
+
+ if (!(a < 2 && b < 2))
+ return 19;
+
+ if (ctx->x % 2)
+ goto *jt1[a];
+ else
+ goto *jt2[b];
+
+ l1: ret += 1;
+ l2: ret += 3;
+ l3: ret += 5;
+ l4: ret += 7;
+
+ ret_user = ret;
+ return ret;
+}
+
+SEC("syscall")
+int one_map_two_jumps(struct simple_ctx *ctx __attribute__((unused)))
+{
+ __label__ l1, l2, l3;
+ void *jt[3] = { &&l1, &&l2, &&l3 };
+ unsigned int a = (ctx->x >> 2) & 1;
+ unsigned int b = (ctx->x >> 3) & 1;
+ volatile int ret = 0;
+
+ if (ctx->x % 2)
+ goto *jt[a];
+
+ if (ctx->x % 3)
+ goto *jt[a + b];
+
+ l1: ret += 3;
+ l2: ret += 5;
+ l3: ret += 7;
+
+ ret_user = ret;
+ return ret;
+}
+
+/* Just to introduce some non-zero offsets in .text */
+static __noinline int f0(volatile struct simple_ctx *ctx __arg_ctx)
+{
+ if (ctx)
+ return 1;
+ else
+ return 13;
+}
+
+SEC("syscall") int f1(struct simple_ctx *ctx)
+{
+ ret_user = 0;
+ return f0(ctx);
+}
+
+static __noinline int __static_global(__u64 x)
+{
+ switch (x) {
+ case 0:
+ adjust_insns(x + 1);
+ ret_user = 2;
+ break;
+ case 1:
+ adjust_insns(x + 7);
+ ret_user = 3;
+ break;
+ case 2:
+ adjust_insns(x + 9);
+ ret_user = 4;
+ break;
+ case 3:
+ adjust_insns(x + 11);
+ ret_user = 5;
+ break;
+ case 4:
+ adjust_insns(x + 17);
+ ret_user = 7;
+ break;
+ default:
+ adjust_insns(x + 177);
+ ret_user = 19;
+ break;
+ }
+
+ return 0;
+}
+
+SEC("syscall")
+int use_static_global1(struct simple_ctx *ctx)
+{
+ ret_user = 0;
+ return __static_global(ctx->x);
+}
+
+SEC("syscall")
+int use_static_global2(struct simple_ctx *ctx)
+{
+ ret_user = 0;
+ adjust_insns(ctx->x + 1);
+ return __static_global(ctx->x);
+}
+
+SEC("fentry/" SYS_PREFIX "sys_nanosleep")
+int use_static_global_other_sec(void *ctx)
+{
+ return __static_global(in_user);
+}
+
+__noinline int __nonstatic_global(__u64 x)
+{
+ switch (x) {
+ case 0:
+ adjust_insns(x + 1);
+ ret_user = 2;
+ break;
+ case 1:
+ adjust_insns(x + 7);
+ ret_user = 3;
+ break;
+ case 2:
+ adjust_insns(x + 9);
+ ret_user = 4;
+ break;
+ case 3:
+ adjust_insns(x + 11);
+ ret_user = 5;
+ break;
+ case 4:
+ adjust_insns(x + 17);
+ ret_user = 7;
+ break;
+ default:
+ adjust_insns(x + 177);
+ ret_user = 19;
+ break;
+ }
+
+ return 0;
+}
+
+SEC("syscall")
+int use_nonstatic_global1(struct simple_ctx *ctx)
+{
+ ret_user = 0;
+ return __nonstatic_global(ctx->x);
+}
+
+SEC("syscall")
+int use_nonstatic_global2(struct simple_ctx *ctx)
+{
+ ret_user = 0;
+ adjust_insns(ctx->x + 1);
+ return __nonstatic_global(ctx->x);
+}
+
+SEC("fentry/" SYS_PREFIX "sys_nanosleep")
+int use_nonstatic_global_other_sec(void *ctx)
+{
+ return __nonstatic_global(in_user);
+}
+#endif
+
+char _license[] SEC("license") = "GPL";
--
2.34.1
^ permalink raw reply related [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array
2025-10-26 19:26 ` [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array Anton Protopopov
@ 2025-10-26 20:12 ` Anton Protopopov
2025-10-26 22:34 ` kernel test robot
` (3 subsequent siblings)
4 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 20:12 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
On 25/10/26 07:26PM, Anton Protopopov wrote:
> [...]
> +#ifdef CONFIG_BPF_SYSCALL
> +void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
> + update_insn_ptr_func_t update_insn_ptr);
> +#else
> +static inline void
> +bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
> + update_insn_ptr_func_t update_insn_ptr);
> +{
> +}
AI found an issue here. The semi-colon was copy-pasted incorrectly.
> +#endif
> [...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps
2025-10-26 19:27 ` [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps Anton Protopopov
@ 2025-10-26 20:15 ` Anton Protopopov
2025-10-27 22:09 ` Eduard Zingerman
2025-10-27 22:38 ` Eduard Zingerman
2 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 20:15 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
On 25/10/26 07:27PM, Anton Protopopov wrote:
> [...]
> + if (!obj->jumptables_data) {
> + pr_warn("map '.jumptables': ELF file is missing jump table data\n");
> + err = -EINVAL;
> + goto err_close;
> + }
> + if (sym_off + jt_size > obj->jumptables_data_sz) {
> + pr_warn("jumptables_data size is %zd, trying to access %d\n",
> + obj->jumptables_data_sz, sym_off + jt_size);
> + err = -EINVAL;
> + goto err_close;
> + }
> +
> + jt = (__u64 *)(obj->jumptables_data + sym_off);
> + for (i = 0; i < max_entries; i++) {
> + /*
> + * The offset should be made to be relative to the beginning of
> + * the main function, not the subfunction.
> + */
> + insn_off = jt[i]/sizeof(struct bpf_insn);
> + if (!prog->subprogs) {
> + insn_off -= prog->sec_insn_off;
> + } else {
> + subprog_idx = find_subprog_idx(prog, relo->insn_idx);
> + if (subprog_idx < 0) {
> + pr_warn("invalid jump insn idx[%d]: %d, no subprog found\n",
> + i, relo->insn_idx);
> + err = -EINVAL;
> + }
And AI found a correct issue here. Should +goto err_close. Fixed it.
> + insn_off -= prog->subprogs[subprog_idx].sec_insn_off;
> + insn_off += prog->subprogs[subprog_idx].sub_insn_off;
> + }
> [...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 06/12] bpf, x86: add support for indirect jumps
2025-10-26 19:27 ` [PATCH v7 bpf-next 06/12] bpf, x86: add support for indirect jumps Anton Protopopov
@ 2025-10-26 20:41 ` Anton Protopopov
0 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-26 20:41 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
On 25/10/26 07:27PM, Anton Protopopov wrote:
> Add support for a new instruction
>
> [...]
>
> @@ -17823,6 +17851,197 @@ static struct bpf_iarray *iarray_realloc(struct bpf_iarray *old, size_t n_elem)
> return new;
> }
>
> +static int copy_insn_array(struct bpf_map *map, u32 start, u32 end, u32 *items)
> +{
> + struct bpf_insn_array_value *value;
> + u32 i;
> +
> + for (i = start; i <= end; i++) {
> + value = map->ops->map_lookup_elem(map, &i);
> + if (!value)
> + return -EINVAL;
> + items[i - start] = value->xlated_off;
> + }
> + return 0;
> +}
AI said that the function here can copy a INSN_DELETED as xlated
offset. This doesn't look correct. This function is executed in
check_cfg first, then in check_indirect_jump. And the INSN_DELETED
may only be introduced after the verification is done in the "remove
dead code" or "remove nops".
> [...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array
2025-10-26 19:26 ` [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array Anton Protopopov
2025-10-26 20:12 ` Anton Protopopov
@ 2025-10-26 22:34 ` kernel test robot
2025-10-26 22:59 ` kernel test robot
` (2 subsequent siblings)
4 siblings, 0 replies; 29+ messages in thread
From: kernel test robot @ 2025-10-26 22:34 UTC (permalink / raw)
To: Anton Protopopov, bpf, Alexei Starovoitov, Andrii Nakryiko,
Anton Protopopov, Daniel Borkmann, Eduard Zingerman,
Quentin Monnet, Yonghong Song
Cc: oe-kbuild-all
Hi Anton,
kernel test robot noticed the following build errors:
[auto build test ERROR on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Anton-Protopopov/bpf-x86-add-new-map-type-instructions-array/20251027-032442
base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link: https://lore.kernel.org/r/20251026192709.1964787-2-a.s.protopopov%40gmail.com
patch subject: [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array
config: x86_64-buildonly-randconfig-003-20251027 (https://download.01.org/0day-ci/archive/20251027/202510270615.wd4tivUA-lkp@intel.com/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251027/202510270615.wd4tivUA-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202510270615.wd4tivUA-lkp@intel.com/
All error/warnings (new ones prefixed by >>):
In file included from include/linux/security.h:35,
from include/linux/perf_event.h:53,
from include/linux/trace_events.h:10,
from include/trace/syscall.h:7,
from include/linux/syscalls.h:95,
from include/linux/syscalls_api.h:1,
from kernel/sched/sched.h:61,
from kernel/sched/rq-offsets.c:5:
>> include/linux/bpf.h:3807:1: error: expected identifier or '(' before '{' token
3807 | {
| ^
>> include/linux/bpf.h:3805:1: warning: 'bpf_prog_update_insn_ptrs' declared 'static' but never defined [-Wunused-function]
3805 | bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
make[3]: *** [scripts/Makefile.build:182: kernel/sched/rq-offsets.s] Error 1 shuffle=2043598271
make[3]: Target 'prepare' not remade because of errors.
make[2]: *** [Makefile:1280: prepare0] Error 2 shuffle=2043598271
make[2]: Target 'prepare' not remade because of errors.
make[1]: *** [Makefile:248: __sub-make] Error 2 shuffle=2043598271
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:248: __sub-make] Error 2 shuffle=2043598271
make: Target 'prepare' not remade because of errors.
vim +3807 include/linux/bpf.h
3799
3800 #ifdef CONFIG_BPF_SYSCALL
3801 void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
3802 update_insn_ptr_func_t update_insn_ptr);
3803 #else
3804 static inline void
> 3805 bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
3806 update_insn_ptr_func_t update_insn_ptr);
> 3807 {
3808 }
3809 #endif
3810
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array
2025-10-26 19:26 ` [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array Anton Protopopov
2025-10-26 20:12 ` Anton Protopopov
2025-10-26 22:34 ` kernel test robot
@ 2025-10-26 22:59 ` kernel test robot
2025-10-27 21:44 ` Eduard Zingerman
2025-10-29 21:54 ` Eduard Zingerman
4 siblings, 0 replies; 29+ messages in thread
From: kernel test robot @ 2025-10-26 22:59 UTC (permalink / raw)
To: Anton Protopopov, bpf, Alexei Starovoitov, Andrii Nakryiko,
Anton Protopopov, Daniel Borkmann, Eduard Zingerman,
Quentin Monnet, Yonghong Song
Cc: llvm, oe-kbuild-all
Hi Anton,
kernel test robot noticed the following build errors:
[auto build test ERROR on bpf-next/master]
url: https://github.com/intel-lab-lkp/linux/commits/Anton-Protopopov/bpf-x86-add-new-map-type-instructions-array/20251027-032442
base: https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git master
patch link: https://lore.kernel.org/r/20251026192709.1964787-2-a.s.protopopov%40gmail.com
patch subject: [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array
config: i386-buildonly-randconfig-005-20251027 (https://download.01.org/0day-ci/archive/20251027/202510270614.av6f72iy-lkp@intel.com/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251027/202510270614.av6f72iy-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202510270614.av6f72iy-lkp@intel.com/
All errors (new ones prefixed by >>):
In file included from kernel/sched/rq-offsets.c:5:
In file included from kernel/sched/sched.h:61:
In file included from include/linux/syscalls_api.h:1:
In file included from include/linux/syscalls.h:95:
In file included from include/trace/syscall.h:7:
In file included from include/linux/trace_events.h:10:
In file included from include/linux/perf_event.h:53:
In file included from include/linux/security.h:35:
>> include/linux/bpf.h:3807:1: error: expected identifier or '('
3807 | {
| ^
In file included from kernel/sched/rq-offsets.c:5:
kernel/sched/sched.h:3743:18: warning: variable 'cpumask' set but not used [-Wunused-but-set-variable]
3743 | struct cpumask *cpumask;
| ^
1 warning and 1 error generated.
make[3]: *** [scripts/Makefile.build:182: kernel/sched/rq-offsets.s] Error 1 shuffle=3282044347
make[3]: Target 'prepare' not remade because of errors.
make[2]: *** [Makefile:1280: prepare0] Error 2 shuffle=3282044347
make[2]: Target 'prepare' not remade because of errors.
make[1]: *** [Makefile:248: __sub-make] Error 2 shuffle=3282044347
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:248: __sub-make] Error 2 shuffle=3282044347
make: Target 'prepare' not remade because of errors.
vim +3807 include/linux/bpf.h
3799
3800 #ifdef CONFIG_BPF_SYSCALL
3801 void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
3802 update_insn_ptr_func_t update_insn_ptr);
3803 #else
3804 static inline void
3805 bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
3806 update_insn_ptr_func_t update_insn_ptr);
> 3807 {
3808 }
3809 #endif
3810
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 08/12] bpf, docs: do not state that indirect jumps are not supported
2025-10-26 19:27 ` [PATCH v7 bpf-next 08/12] bpf, docs: do not state that indirect jumps are not supported Anton Protopopov
@ 2025-10-27 6:30 ` Anton Protopopov
0 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-27 6:30 UTC (permalink / raw)
To: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Eduard Zingerman, Quentin Monnet, Yonghong Song
On 25/10/26 07:27PM, Anton Protopopov wrote:
> The linux-notes.rst states that indirect jump instruction "is not
> currently supported by the verifier". Remove this part as outdated.
>
> Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> ---
> Documentation/bpf/linux-notes.rst | 8 --------
> 1 file changed, 8 deletions(-)
>
> diff --git a/Documentation/bpf/linux-notes.rst b/Documentation/bpf/linux-notes.rst
> index 00d2693de025..64ac146a926f 100644
> --- a/Documentation/bpf/linux-notes.rst
> +++ b/Documentation/bpf/linux-notes.rst
> @@ -12,14 +12,6 @@ Byte swap instructions
>
> ``BPF_FROM_LE`` and ``BPF_FROM_BE`` exist as aliases for ``BPF_TO_LE`` and ``BPF_TO_BE`` respectively.
>
> -Jump instructions
> -=================
> -
> -``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function
> -integer would be read from a specified register, is not currently supported
> -by the verifier. Any programs with this instruction will fail to load
> -until such support is added.
> -
> Maps
> ====
Here the BPF_CALL is mentioned (thanks, AI).
I will drop this commit in the next version,
and follow up on docs later.
> --
> 2.34.1
>
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array
2025-10-26 19:26 ` [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array Anton Protopopov
` (2 preceding siblings ...)
2025-10-26 22:59 ` kernel test robot
@ 2025-10-27 21:44 ` Eduard Zingerman
2025-10-28 10:10 ` Anton Protopopov
2025-10-29 21:54 ` Eduard Zingerman
4 siblings, 1 reply; 29+ messages in thread
From: Eduard Zingerman @ 2025-10-27 21:44 UTC (permalink / raw)
To: Anton Protopopov, bpf, Alexei Starovoitov, Andrii Nakryiko,
Anton Protopopov, Daniel Borkmann, Quentin Monnet, Yonghong Song
On Sun, 2025-10-26 at 19:26 +0000, Anton Protopopov wrote:
[...]
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index d4c93d9e73e4..d8ee0c4d9af8 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -3648,6 +3648,22 @@ struct x64_jit_data {
> struct jit_context ctx;
> };
>
> +struct insn_ptrs_data {
> + int *addrs;
> + u8 *image;
> +};
> +
> +static void update_insn_ptr(void *jit_priv, u32 xlated_off, u32 *jitted_off, long *ip)
> +{
> + struct insn_ptrs_data *data = jit_priv;
> +
> + if (!data->addrs || !data->image || !jitted_off || !ip)
> + return;
> +
> + *jitted_off = data->addrs[xlated_off];
> + *ip = (long)(data->image + *jitted_off);
> +}
> +
> #define MAX_PASSES 20
> #define PADDING_PASSES (MAX_PASSES - 5)
>
> @@ -3658,6 +3674,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> struct bpf_prog *tmp, *orig_prog = prog;
> void __percpu *priv_stack_ptr = NULL;
> struct x64_jit_data *jit_data;
> + struct insn_ptrs_data insn_ptrs_data;
> int priv_stack_alloc_sz;
> int proglen, oldproglen = 0;
> struct jit_context ctx = {};
> @@ -3827,6 +3844,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> jit_data->header = header;
> jit_data->rw_header = rw_header;
> }
> +
> + /* jit_data may not contain proper info, copy the required fields */
> + insn_ptrs_data.addrs = addrs;
> + insn_ptrs_data.image = image;
> + bpf_prog_update_insn_ptrs(prog, &insn_ptrs_data, update_insn_ptr);
> +
> /*
> * ctx.prog_offset is used when CFI preambles put code *before*
> * the function. See emit_cfi(). For FineIBT specifically this code
[...]
> diff --git a/kernel/bpf/bpf_insn_array.c b/kernel/bpf/bpf_insn_array.c
> new file mode 100644
> index 000000000000..f9f875ee2027
> +++ b/kernel/bpf/bpf_insn_array.c
[...]
> +void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
> + update_insn_ptr_func_t update_insn_ptr)
> +{
Nit: I think the control flow becomes a bit convoluted with this
function pointer. Wdyt about changing the signature to:
void bpf_prog_update_insn_ptrs(struct bpf_prog *prog,
u32 *offsets /* maps xlated_off to offset in image */,
void *image)
x86 jit provides this info, it looks like arm64 and riscv jits do too
(arch/arm64/net/bpf_jit_comp.c:jit_ctx->offset field,
arch/riscv/net/bpf_jit.h:rv_jit_context->offset).
So, seem to be a reasonable assumption.
Wdyt?
> + struct bpf_insn_array *insn_array;
> + struct bpf_map *map;
> + u32 xlated_off;
> + int i, j;
> +
> + for (i = 0; i < prog->aux->used_map_cnt; i++) {
> + map = prog->aux->used_maps[i];
> + if (!is_insn_array(map))
> + continue;
> + insn_array = cast_insn_array(map);
> + for (j = 0; j < map->max_entries; j++) {
> + xlated_off = insn_array->values[j].xlated_off;
> + if (xlated_off == INSN_DELETED)
> + continue;
> + if (xlated_off < prog->aux->subprog_start)
> + continue;
> + xlated_off -= prog->aux->subprog_start;
> + if (xlated_off >= prog->len)
> + continue;
> +
> + update_insn_ptr(jit_priv, xlated_off,
> + &insn_array->values[j].jitted_off,
> + &insn_array->ips[j]);
> + }
> + }
> +}
[...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps
2025-10-26 19:27 ` [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps Anton Protopopov
2025-10-26 20:15 ` Anton Protopopov
@ 2025-10-27 22:09 ` Eduard Zingerman
2025-10-27 22:38 ` Eduard Zingerman
2 siblings, 0 replies; 29+ messages in thread
From: Eduard Zingerman @ 2025-10-27 22:09 UTC (permalink / raw)
To: Anton Protopopov, bpf, Alexei Starovoitov, Andrii Nakryiko,
Anton Protopopov, Daniel Borkmann, Quentin Monnet, Yonghong Song
On Sun, 2025-10-26 at 19:27 +0000, Anton Protopopov wrote:
> For v4 instruction set LLVM is allowed to generate indirect jumps for
> switch statements and for 'goto *rX' assembly. Every such a jump will
> be accompanied by necessary metadata, e.g. (`llvm-objdump -Sr ...`):
>
> 0: r2 = 0x0 ll
> 0000000000000030: R_BPF_64_64 BPF.JT.0.0
>
> Here BPF.JT.1.0 is a symbol residing in the .jumptables section:
>
> Symbol table:
> 4: 0000000000000000 240 OBJECT GLOBAL DEFAULT 4 BPF.JT.0.0
>
> The -bpf-min-jump-table-entries llvm option may be used to control the
> minimal size of a switch which will be converted to an indirect jumps.
>
> Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> ---
Aside from a nit spotted by ai, I think this patch looks good.
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
[...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps
2025-10-26 19:27 ` [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps Anton Protopopov
2025-10-26 20:15 ` Anton Protopopov
2025-10-27 22:09 ` Eduard Zingerman
@ 2025-10-27 22:38 ` Eduard Zingerman
2025-10-27 22:59 ` Eduard Zingerman
2 siblings, 1 reply; 29+ messages in thread
From: Eduard Zingerman @ 2025-10-27 22:38 UTC (permalink / raw)
To: Anton Protopopov, bpf, Alexei Starovoitov, Andrii Nakryiko,
Anton Protopopov, Daniel Borkmann, Quentin Monnet, Yonghong Song
[...]
> +static int create_jt_map(struct bpf_object *obj, struct bpf_program *prog, struct reloc_desc *relo)
> +{
> + const __u32 jt_entry_size = 8;
> + int sym_off = relo->sym_off;
> + int jt_size = relo->sym_size;
> + __u32 max_entries = jt_size / jt_entry_size;
> + __u32 value_size = sizeof(struct bpf_insn_array_value);
> + struct bpf_insn_array_value val = {};
> + int subprog_idx;
> + int map_fd, err;
> + __u64 insn_off;
> + __u64 *jt;
> + __u32 i;
> +
> + map_fd = find_jt_map(obj, prog, sym_off);
> + if (map_fd >= 0)
> + return map_fd;
> +
> + if (sym_off % jt_entry_size) {
> + pr_warn("jumptable start %d should be multiple of %u\n",
> + sym_off, jt_entry_size);
> + return -EINVAL;
> + }
> +
> + if (jt_size % jt_entry_size) {
> + pr_warn("jumptable size %d should be multiple of %u\n",
> + jt_size, jt_entry_size);
> + return -EINVAL;
> + }
> +
> + map_fd = bpf_map_create(BPF_MAP_TYPE_INSN_ARRAY, ".jumptables",
> + 4, value_size, max_entries, NULL);
> + if (map_fd < 0)
> + return map_fd;
> +
> + if (!obj->jumptables_data) {
> + pr_warn("map '.jumptables': ELF file is missing jump table data\n");
> + err = -EINVAL;
> + goto err_close;
> + }
> + if (sym_off + jt_size > obj->jumptables_data_sz) {
> + pr_warn("jumptables_data size is %zd, trying to access %d\n",
> + obj->jumptables_data_sz, sym_off + jt_size);
> + err = -EINVAL;
> + goto err_close;
> + }
> +
> + jt = (__u64 *)(obj->jumptables_data + sym_off);
> + for (i = 0; i < max_entries; i++) {
> + /*
> + * The offset should be made to be relative to the beginning of
> + * the main function, not the subfunction.
> + */
> + insn_off = jt[i]/sizeof(struct bpf_insn);
> + if (!prog->subprogs) {
> + insn_off -= prog->sec_insn_off;
> + } else {
> + subprog_idx = find_subprog_idx(prog, relo->insn_idx);
Nit: find_subprog_idx(prog, relo->insn_idx) can be moved outside of the loop, I think.
> + if (subprog_idx < 0) {
> + pr_warn("invalid jump insn idx[%d]: %d, no subprog found\n",
> + i, relo->insn_idx);
> + err = -EINVAL;
> + }
> + insn_off -= prog->subprogs[subprog_idx].sec_insn_off;
> + insn_off += prog->subprogs[subprog_idx].sub_insn_off;
> + }
> +
> + /*
> + * LLVM-generated jump tables contain u64 records, however
> + * should contain values that fit in u32.
> + */
> + if (insn_off > UINT32_MAX) {
> + pr_warn("invalid jump table value %llx at offset %d\n",
> + jt[i], sym_off + i);
> + err = -EINVAL;
> + goto err_close;
> + }
> +
> + val.orig_off = insn_off;
> + err = bpf_map_update_elem(map_fd, &i, &val, 0);
> + if (err)
> + goto err_close;
> + }
[...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps
2025-10-27 22:38 ` Eduard Zingerman
@ 2025-10-27 22:59 ` Eduard Zingerman
2025-10-28 11:36 ` Anton Protopopov
2025-10-28 11:42 ` Anton Protopopov
0 siblings, 2 replies; 29+ messages in thread
From: Eduard Zingerman @ 2025-10-27 22:59 UTC (permalink / raw)
To: Anton Protopopov, bpf, Alexei Starovoitov, Andrii Nakryiko,
Anton Protopopov, Daniel Borkmann, Quentin Monnet, Yonghong Song
On Mon, 2025-10-27 at 15:38 -0700, Eduard Zingerman wrote:
> [...]
>
> > +static int create_jt_map(struct bpf_object *obj, struct bpf_program *prog, struct reloc_desc *relo)
> > +{
> > + const __u32 jt_entry_size = 8;
> > + int sym_off = relo->sym_off;
> > + int jt_size = relo->sym_size;
> > + __u32 max_entries = jt_size / jt_entry_size;
> > + __u32 value_size = sizeof(struct bpf_insn_array_value);
> > + struct bpf_insn_array_value val = {};
> > + int subprog_idx;
> > + int map_fd, err;
> > + __u64 insn_off;
> > + __u64 *jt;
> > + __u32 i;
> > +
> > + map_fd = find_jt_map(obj, prog, sym_off);
> > + if (map_fd >= 0)
> > + return map_fd;
> > +
> > + if (sym_off % jt_entry_size) {
> > + pr_warn("jumptable start %d should be multiple of %u\n",
> > + sym_off, jt_entry_size);
> > + return -EINVAL;
> > + }
> > +
> > + if (jt_size % jt_entry_size) {
> > + pr_warn("jumptable size %d should be multiple of %u\n",
> > + jt_size, jt_entry_size);
> > + return -EINVAL;
> > + }
> > +
> > + map_fd = bpf_map_create(BPF_MAP_TYPE_INSN_ARRAY, ".jumptables",
> > + 4, value_size, max_entries, NULL);
> > + if (map_fd < 0)
> > + return map_fd;
> > +
> > + if (!obj->jumptables_data) {
> > + pr_warn("map '.jumptables': ELF file is missing jump table data\n");
> > + err = -EINVAL;
> > + goto err_close;
> > + }
> > + if (sym_off + jt_size > obj->jumptables_data_sz) {
> > + pr_warn("jumptables_data size is %zd, trying to access %d\n",
> > + obj->jumptables_data_sz, sym_off + jt_size);
> > + err = -EINVAL;
> > + goto err_close;
> > + }
> > +
> > + jt = (__u64 *)(obj->jumptables_data + sym_off);
> > + for (i = 0; i < max_entries; i++) {
> > + /*
> > + * The offset should be made to be relative to the beginning of
> > + * the main function, not the subfunction.
> > + */
> > + insn_off = jt[i]/sizeof(struct bpf_insn);
> > + if (!prog->subprogs) {
> > + insn_off -= prog->sec_insn_off;
> > + } else {
> > + subprog_idx = find_subprog_idx(prog, relo->insn_idx);
>
> Nit: find_subprog_idx(prog, relo->insn_idx) can be moved outside of the loop, I think.
>
> > + if (subprog_idx < 0) {
> > + pr_warn("invalid jump insn idx[%d]: %d, no subprog found\n",
> > + i, relo->insn_idx);
> > + err = -EINVAL;
> > + }
> > + insn_off -= prog->subprogs[subprog_idx].sec_insn_off;
> > + insn_off += prog->subprogs[subprog_idx].sub_insn_off;
> > + }
I think I found a bug, related to this code path.
Consider the following test case:
SEC("socket")
__naked void foo(void)
{
asm volatile (" \
.pushsection .jumptables,\"\",@progbits; \
jt0_%=: \
.quad ret0_%=; \
.quad ret1_%=; \
.size jt0_%=, 16; \
.global jt0_%=; \
.popsection; \
\
r0 = jt0_%= ll; \
r0 += 8; \
r0 = *(u64 *)(r0 + 0); \
.8byte %[gotox_r0]; \
ret0_%=: \
r0 = 0; \
exit; \
ret1_%=: \
r0 = 1; \
call bar; \
exit; \
" : \
: __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
: __clobber_all);
}
__used
static int bar(void)
{
return 0;
}
Note a call instruction referring bar(). It triggers the code path
above (we need a test case with subprograms in verifier_gotox).
The test case fails to load with the following error:
libbpf: invalid jump insn idx[0]: 0, no subprog found
libbpf: prog 'foo': relo #0: can't create jump table: sym_off 368
libbpf: prog 'foo': failed to relocate data references: -EINVAL
If I remove the `call bar;`, test case loads and passes.
> > +
> > + /*
> > + * LLVM-generated jump tables contain u64 records, however
> > + * should contain values that fit in u32.
> > + */
> > + if (insn_off > UINT32_MAX) {
> > + pr_warn("invalid jump table value %llx at offset %d\n",
^^^^
Nit: maybe add 0x prefix here?
> > + jt[i], sym_off + i);
> > + err = -EINVAL;
> > + goto err_close;
> > + }
> > +
> > + val.orig_off = insn_off;
> > + err = bpf_map_update_elem(map_fd, &i, &val, 0);
> > + if (err)
> > + goto err_close;
> > + }
>
> [...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 12/12] selftests/bpf: add C-level selftests for indirect jumps
2025-10-26 19:27 ` [PATCH v7 bpf-next 12/12] selftests/bpf: add C-level selftests for indirect jumps Anton Protopopov
@ 2025-10-27 23:25 ` Eduard Zingerman
2025-10-28 10:59 ` Anton Protopopov
0 siblings, 1 reply; 29+ messages in thread
From: Eduard Zingerman @ 2025-10-27 23:25 UTC (permalink / raw)
To: Yonghong Song, Anton Protopopov
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Quentin Monnet
On Sun, 2025-10-26 at 19:27 +0000, Anton Protopopov wrote:
> Add C-level selftests for indirect jumps to validate LLVM and libbpf
> functionality. The tests are intentionally disabled, to be run
> locally by developers, but will not make the CI red.
>
> Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> ---
[...]
> diff --git a/tools/testing/selftests/bpf/progs/bpf_gotox.c b/tools/testing/selftests/bpf/progs/bpf_gotox.c
> new file mode 100644
> index 000000000000..3c8ee363bda1
> --- /dev/null
> +++ b/tools/testing/selftests/bpf/progs/bpf_gotox.c
> @@ -0,0 +1,402 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +#include "vmlinux.h"
> +#include <bpf/bpf_helpers.h>
> +#include <bpf/bpf_tracing.h>
> +#include <bpf/bpf_core_read.h>
> +#include "bpf_misc.h"
> +
> +/* Disable tests for now, as CI runs with LLVM-20 */
> +#if 0
Yonghong,
I think we need the following thing in LLVM:
diff --git a/clang/lib/Basic/Targets/BPF.cpp b/clang/lib/Basic/Targets/BPF.cpp
index 0411bcca5178..8de1083d758c 100644
--- a/clang/lib/Basic/Targets/BPF.cpp
+++ b/clang/lib/Basic/Targets/BPF.cpp
@@ -75,6 +75,7 @@ void BPFTargetInfo::getTargetDefines(const LangOptions &Opts,
Builder.defineMacro("__BPF_FEATURE_GOTOL");
Builder.defineMacro("__BPF_FEATURE_ST");
Builder.defineMacro("__BPF_FEATURE_LOAD_ACQ_STORE_REL");
+ Builder.defineMacro("__BPF_FEATURE_GOTOX");
}
}
Then, Anton will be able to use it in order to decide if to skip the
tests, wdyt?
> +__u64 in_user;
> +__u64 ret_user;
> +
> +struct simple_ctx {
> + __u64 x;
> +};
> +
> +__u64 some_var;
> +
> +/*
> + * This function adds code which will be replaced by a different
> + * number of instructions by the verifier. This adds additional
> + * stress on testing the insn_array maps corresponding to indirect jumps.
> + */
> +static __always_inline void adjust_insns(__u64 x)
> +{
> + some_var ^= x + bpf_jiffies64();
> +}
> +
> +SEC("syscall")
> +int simple_test(struct simple_ctx *ctx)
> +{
> + switch (ctx->x) {
> + case 0:
> + adjust_insns(ctx->x + 1);
> + ret_user = 2;
> + break;
[...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array
2025-10-27 21:44 ` Eduard Zingerman
@ 2025-10-28 10:10 ` Anton Protopopov
0 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-28 10:10 UTC (permalink / raw)
To: Eduard Zingerman
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Quentin Monnet, Yonghong Song
On 25/10/27 02:44PM, Eduard Zingerman wrote:
> On Sun, 2025-10-26 at 19:26 +0000, Anton Protopopov wrote:
>
> [...]
>
> > diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> > index d4c93d9e73e4..d8ee0c4d9af8 100644
> > --- a/arch/x86/net/bpf_jit_comp.c
> > +++ b/arch/x86/net/bpf_jit_comp.c
> > @@ -3648,6 +3648,22 @@ struct x64_jit_data {
> > struct jit_context ctx;
> > };
> >
> > +struct insn_ptrs_data {
> > + int *addrs;
> > + u8 *image;
> > +};
> > +
> > +static void update_insn_ptr(void *jit_priv, u32 xlated_off, u32 *jitted_off, long *ip)
> > +{
> > + struct insn_ptrs_data *data = jit_priv;
> > +
> > + if (!data->addrs || !data->image || !jitted_off || !ip)
> > + return;
> > +
> > + *jitted_off = data->addrs[xlated_off];
> > + *ip = (long)(data->image + *jitted_off);
> > +}
> > +
> > #define MAX_PASSES 20
> > #define PADDING_PASSES (MAX_PASSES - 5)
> >
> > @@ -3658,6 +3674,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> > struct bpf_prog *tmp, *orig_prog = prog;
> > void __percpu *priv_stack_ptr = NULL;
> > struct x64_jit_data *jit_data;
> > + struct insn_ptrs_data insn_ptrs_data;
> > int priv_stack_alloc_sz;
> > int proglen, oldproglen = 0;
> > struct jit_context ctx = {};
> > @@ -3827,6 +3844,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
> > jit_data->header = header;
> > jit_data->rw_header = rw_header;
> > }
> > +
> > + /* jit_data may not contain proper info, copy the required fields */
> > + insn_ptrs_data.addrs = addrs;
> > + insn_ptrs_data.image = image;
> > + bpf_prog_update_insn_ptrs(prog, &insn_ptrs_data, update_insn_ptr);
> > +
> > /*
> > * ctx.prog_offset is used when CFI preambles put code *before*
> > * the function. See emit_cfi(). For FineIBT specifically this code
>
> [...]
>
> > diff --git a/kernel/bpf/bpf_insn_array.c b/kernel/bpf/bpf_insn_array.c
> > new file mode 100644
> > index 000000000000..f9f875ee2027
> > +++ b/kernel/bpf/bpf_insn_array.c
>
> [...]
>
> > +void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
> > + update_insn_ptr_func_t update_insn_ptr)
> > +{
>
> Nit: I think the control flow becomes a bit convoluted with this
> function pointer. Wdyt about changing the signature to:
>
> void bpf_prog_update_insn_ptrs(struct bpf_prog *prog,
> u32 *offsets /* maps xlated_off to offset in image */,
> void *image)
>
> x86 jit provides this info, it looks like arm64 and riscv jits do too
> (arch/arm64/net/bpf_jit_comp.c:jit_ctx->offset field,
> arch/riscv/net/bpf_jit.h:rv_jit_context->offset).
> So, seem to be a reasonable assumption.
>
> Wdyt?
I did it a bit more abstract variant right away because in future
(read "static keys") there will be more data passed around. I will
switch to your variant, and then, once I follow up with static keys,
it can be generalized, if needed.
> > + struct bpf_insn_array *insn_array;
> > + struct bpf_map *map;
> > + u32 xlated_off;
> > + int i, j;
> > +
> > + for (i = 0; i < prog->aux->used_map_cnt; i++) {
> > + map = prog->aux->used_maps[i];
> > + if (!is_insn_array(map))
> > + continue;
> > + insn_array = cast_insn_array(map);
> > + for (j = 0; j < map->max_entries; j++) {
> > + xlated_off = insn_array->values[j].xlated_off;
> > + if (xlated_off == INSN_DELETED)
> > + continue;
> > + if (xlated_off < prog->aux->subprog_start)
> > + continue;
> > + xlated_off -= prog->aux->subprog_start;
> > + if (xlated_off >= prog->len)
> > + continue;
> > +
> > + update_insn_ptr(jit_priv, xlated_off,
> > + &insn_array->values[j].jitted_off,
> > + &insn_array->ips[j]);
> > + }
> > + }
> > +}
>
> [...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 12/12] selftests/bpf: add C-level selftests for indirect jumps
2025-10-27 23:25 ` Eduard Zingerman
@ 2025-10-28 10:59 ` Anton Protopopov
0 siblings, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-28 10:59 UTC (permalink / raw)
To: Eduard Zingerman
Cc: Yonghong Song, bpf, Alexei Starovoitov, Andrii Nakryiko,
Anton Protopopov, Daniel Borkmann, Quentin Monnet
On 25/10/27 04:25PM, Eduard Zingerman wrote:
> On Sun, 2025-10-26 at 19:27 +0000, Anton Protopopov wrote:
> > Add C-level selftests for indirect jumps to validate LLVM and libbpf
> > functionality. The tests are intentionally disabled, to be run
> > locally by developers, but will not make the CI red.
> >
> > Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> > ---
>
> [...]
>
> > diff --git a/tools/testing/selftests/bpf/progs/bpf_gotox.c b/tools/testing/selftests/bpf/progs/bpf_gotox.c
> > new file mode 100644
> > index 000000000000..3c8ee363bda1
> > --- /dev/null
> > +++ b/tools/testing/selftests/bpf/progs/bpf_gotox.c
> > @@ -0,0 +1,402 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +
> > +#include "vmlinux.h"
> > +#include <bpf/bpf_helpers.h>
> > +#include <bpf/bpf_tracing.h>
> > +#include <bpf/bpf_core_read.h>
> > +#include "bpf_misc.h"
> > +
> > +/* Disable tests for now, as CI runs with LLVM-20 */
> > +#if 0
>
> Yonghong,
>
> I think we need the following thing in LLVM:
>
> diff --git a/clang/lib/Basic/Targets/BPF.cpp b/clang/lib/Basic/Targets/BPF.cpp
> index 0411bcca5178..8de1083d758c 100644
> --- a/clang/lib/Basic/Targets/BPF.cpp
> +++ b/clang/lib/Basic/Targets/BPF.cpp
> @@ -75,6 +75,7 @@ void BPFTargetInfo::getTargetDefines(const LangOptions &Opts,
> Builder.defineMacro("__BPF_FEATURE_GOTOL");
> Builder.defineMacro("__BPF_FEATURE_ST");
> Builder.defineMacro("__BPF_FEATURE_LOAD_ACQ_STORE_REL");
> + Builder.defineMacro("__BPF_FEATURE_GOTOX");
> }
> }
>
> Then, Anton will be able to use it in order to decide if to skip the
> tests, wdyt?
This will definitely be useful for the BPF side.
Where this doesn't apply is the corresponding prog_tests/bpf_gotox.c,
as it can be compiled independently and for sure it will not have
__BPF_FEATURE_GOTOX enabled. So what is the way to tell preprocessor
if gotox was enabled in BPF program? Is there a way to pass/generate
a macro definition from BPF prog to skeleton? If so, then it can be
used in the prog_tests/bpf_gotox.c to enable/disable tests.
> > +__u64 in_user;
> > +__u64 ret_user;
> > +
> > +struct simple_ctx {
> > + __u64 x;
> > +};
> > +
> > +__u64 some_var;
> > +
> > +/*
> > + * This function adds code which will be replaced by a different
> > + * number of instructions by the verifier. This adds additional
> > + * stress on testing the insn_array maps corresponding to indirect jumps.
> > + */
> > +static __always_inline void adjust_insns(__u64 x)
> > +{
> > + some_var ^= x + bpf_jiffies64();
> > +}
> > +
> > +SEC("syscall")
> > +int simple_test(struct simple_ctx *ctx)
> > +{
> > + switch (ctx->x) {
> > + case 0:
> > + adjust_insns(ctx->x + 1);
> > + ret_user = 2;
> > + break;
>
> [...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps
2025-10-27 22:59 ` Eduard Zingerman
@ 2025-10-28 11:36 ` Anton Protopopov
2025-10-28 11:42 ` Anton Protopopov
1 sibling, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-28 11:36 UTC (permalink / raw)
To: Eduard Zingerman
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Quentin Monnet, Yonghong Song
On 25/10/27 03:59PM, Eduard Zingerman wrote:
> On Mon, 2025-10-27 at 15:38 -0700, Eduard Zingerman wrote:
> > [...]
> >
> > > +static int create_jt_map(struct bpf_object *obj, struct bpf_program *prog, struct reloc_desc *relo)
> > > +{
> > > + const __u32 jt_entry_size = 8;
> > > + int sym_off = relo->sym_off;
> > > + int jt_size = relo->sym_size;
> > > + __u32 max_entries = jt_size / jt_entry_size;
> > > + __u32 value_size = sizeof(struct bpf_insn_array_value);
> > > + struct bpf_insn_array_value val = {};
> > > + int subprog_idx;
> > > + int map_fd, err;
> > > + __u64 insn_off;
> > > + __u64 *jt;
> > > + __u32 i;
> > > +
> > > + map_fd = find_jt_map(obj, prog, sym_off);
> > > + if (map_fd >= 0)
> > > + return map_fd;
> > > +
> > > + if (sym_off % jt_entry_size) {
> > > + pr_warn("jumptable start %d should be multiple of %u\n",
> > > + sym_off, jt_entry_size);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + if (jt_size % jt_entry_size) {
> > > + pr_warn("jumptable size %d should be multiple of %u\n",
> > > + jt_size, jt_entry_size);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + map_fd = bpf_map_create(BPF_MAP_TYPE_INSN_ARRAY, ".jumptables",
> > > + 4, value_size, max_entries, NULL);
> > > + if (map_fd < 0)
> > > + return map_fd;
> > > +
> > > + if (!obj->jumptables_data) {
> > > + pr_warn("map '.jumptables': ELF file is missing jump table data\n");
> > > + err = -EINVAL;
> > > + goto err_close;
> > > + }
> > > + if (sym_off + jt_size > obj->jumptables_data_sz) {
> > > + pr_warn("jumptables_data size is %zd, trying to access %d\n",
> > > + obj->jumptables_data_sz, sym_off + jt_size);
> > > + err = -EINVAL;
> > > + goto err_close;
> > > + }
> > > +
> > > + jt = (__u64 *)(obj->jumptables_data + sym_off);
> > > + for (i = 0; i < max_entries; i++) {
> > > + /*
> > > + * The offset should be made to be relative to the beginning of
> > > + * the main function, not the subfunction.
> > > + */
> > > + insn_off = jt[i]/sizeof(struct bpf_insn);
> > > + if (!prog->subprogs) {
> > > + insn_off -= prog->sec_insn_off;
> > > + } else {
> > > + subprog_idx = find_subprog_idx(prog, relo->insn_idx);
> >
> > Nit: find_subprog_idx(prog, relo->insn_idx) can be moved outside of the loop, I think.
> >
> > > + if (subprog_idx < 0) {
> > > + pr_warn("invalid jump insn idx[%d]: %d, no subprog found\n",
> > > + i, relo->insn_idx);
> > > + err = -EINVAL;
> > > + }
> > > + insn_off -= prog->subprogs[subprog_idx].sec_insn_off;
> > > + insn_off += prog->subprogs[subprog_idx].sub_insn_off;
> > > + }
>
> I think I found a bug, related to this code path.
> Consider the following test case:
>
> SEC("socket")
> __naked void foo(void)
> {
> asm volatile (" \
> .pushsection .jumptables,\"\",@progbits; \
> jt0_%=: \
> .quad ret0_%=; \
> .quad ret1_%=; \
> .size jt0_%=, 16; \
> .global jt0_%=; \
> .popsection; \
> \
> r0 = jt0_%= ll; \
> r0 += 8; \
> r0 = *(u64 *)(r0 + 0); \
> .8byte %[gotox_r0]; \
> ret0_%=: \
> r0 = 0; \
> exit; \
> ret1_%=: \
> r0 = 1; \
> call bar; \
> exit; \
> " : \
> : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
> : __clobber_all);
> }
>
> __used
> static int bar(void)
> {
> return 0;
> }
>
> Note a call instruction referring bar(). It triggers the code path
> above (we need a test case with subprograms in verifier_gotox).
> The test case fails to load with the following error:
>
> libbpf: invalid jump insn idx[0]: 0, no subprog found
> libbpf: prog 'foo': relo #0: can't create jump table: sym_off 368
> libbpf: prog 'foo': failed to relocate data references: -EINVAL
>
> If I remove the `call bar;`, test case loads and passes.
Yes, looks like this was introduced with that latest refactor: just checked
that the main program in libbpf is not a subprog[0] (as in verifier). Will
re-re-factor as follows:
subprog_idx = -1; /* main program */
if (relo->insn_idx < 0 || relo->insn_idx >= prog->insns_cnt) {
pr_warn("invalid instruction index %d\n", relo->insn_idx);
err = -EINVAL;
goto err_close;
}
if (prog->subprogs)
subprog_idx = find_subprog_idx(prog, relo->insn_idx);
jt = (__u64 *)(obj->jumptables_data + sym_off);
for (i = 0; i < max_entries; i++) {
/*
* The offset should be made to be relative to the beginning of
* the main function, not the subfunction.
*/
insn_off = jt[i]/sizeof(struct bpf_insn);
if (subprog_idx >= 0) {
insn_off -= prog->subprogs[subprog_idx].sec_insn_off;
insn_off += prog->subprogs[subprog_idx].sub_insn_off;
} else {
insn_off -= prog->sec_insn_off;
}
> > > +
> > > + /*
> > > + * LLVM-generated jump tables contain u64 records, however
> > > + * should contain values that fit in u32.
> > > + */
> > > + if (insn_off > UINT32_MAX) {
> > > + pr_warn("invalid jump table value %llx at offset %d\n",
> ^^^^
> Nit: maybe add 0x prefix here?
>
> > > + jt[i], sym_off + i);
> > > + err = -EINVAL;
> > > + goto err_close;
> > > + }
> > > +
> > > + val.orig_off = insn_off;
> > > + err = bpf_map_update_elem(map_fd, &i, &val, 0);
> > > + if (err)
> > > + goto err_close;
> > > + }
> >
> > [...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps
2025-10-27 22:59 ` Eduard Zingerman
2025-10-28 11:36 ` Anton Protopopov
@ 2025-10-28 11:42 ` Anton Protopopov
1 sibling, 0 replies; 29+ messages in thread
From: Anton Protopopov @ 2025-10-28 11:42 UTC (permalink / raw)
To: Eduard Zingerman
Cc: bpf, Alexei Starovoitov, Andrii Nakryiko, Anton Protopopov,
Daniel Borkmann, Quentin Monnet, Yonghong Song
On 25/10/27 03:59PM, Eduard Zingerman wrote:
> On Mon, 2025-10-27 at 15:38 -0700, Eduard Zingerman wrote:
> > [...]
> >
> > > +static int create_jt_map(struct bpf_object *obj, struct bpf_program *prog, struct reloc_desc *relo)
> > > +{
> > > + const __u32 jt_entry_size = 8;
> > > + int sym_off = relo->sym_off;
> > > + int jt_size = relo->sym_size;
> > > + __u32 max_entries = jt_size / jt_entry_size;
> > > + __u32 value_size = sizeof(struct bpf_insn_array_value);
> > > + struct bpf_insn_array_value val = {};
> > > + int subprog_idx;
> > > + int map_fd, err;
> > > + __u64 insn_off;
> > > + __u64 *jt;
> > > + __u32 i;
> > > +
> > > + map_fd = find_jt_map(obj, prog, sym_off);
> > > + if (map_fd >= 0)
> > > + return map_fd;
> > > +
> > > + if (sym_off % jt_entry_size) {
> > > + pr_warn("jumptable start %d should be multiple of %u\n",
> > > + sym_off, jt_entry_size);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + if (jt_size % jt_entry_size) {
> > > + pr_warn("jumptable size %d should be multiple of %u\n",
> > > + jt_size, jt_entry_size);
> > > + return -EINVAL;
> > > + }
> > > +
> > > + map_fd = bpf_map_create(BPF_MAP_TYPE_INSN_ARRAY, ".jumptables",
> > > + 4, value_size, max_entries, NULL);
> > > + if (map_fd < 0)
> > > + return map_fd;
> > > +
> > > + if (!obj->jumptables_data) {
> > > + pr_warn("map '.jumptables': ELF file is missing jump table data\n");
> > > + err = -EINVAL;
> > > + goto err_close;
> > > + }
> > > + if (sym_off + jt_size > obj->jumptables_data_sz) {
> > > + pr_warn("jumptables_data size is %zd, trying to access %d\n",
> > > + obj->jumptables_data_sz, sym_off + jt_size);
> > > + err = -EINVAL;
> > > + goto err_close;
> > > + }
> > > +
> > > + jt = (__u64 *)(obj->jumptables_data + sym_off);
> > > + for (i = 0; i < max_entries; i++) {
> > > + /*
> > > + * The offset should be made to be relative to the beginning of
> > > + * the main function, not the subfunction.
> > > + */
> > > + insn_off = jt[i]/sizeof(struct bpf_insn);
> > > + if (!prog->subprogs) {
> > > + insn_off -= prog->sec_insn_off;
> > > + } else {
> > > + subprog_idx = find_subprog_idx(prog, relo->insn_idx);
> >
> > Nit: find_subprog_idx(prog, relo->insn_idx) can be moved outside of the loop, I think.
> >
> > > + if (subprog_idx < 0) {
> > > + pr_warn("invalid jump insn idx[%d]: %d, no subprog found\n",
> > > + i, relo->insn_idx);
> > > + err = -EINVAL;
> > > + }
> > > + insn_off -= prog->subprogs[subprog_idx].sec_insn_off;
> > > + insn_off += prog->subprogs[subprog_idx].sub_insn_off;
> > > + }
>
> I think I found a bug, related to this code path.
> Consider the following test case:
>
> SEC("socket")
> __naked void foo(void)
> {
> asm volatile (" \
> .pushsection .jumptables,\"\",@progbits; \
> jt0_%=: \
> .quad ret0_%=; \
> .quad ret1_%=; \
> .size jt0_%=, 16; \
> .global jt0_%=; \
> .popsection; \
> \
> r0 = jt0_%= ll; \
> r0 += 8; \
> r0 = *(u64 *)(r0 + 0); \
> .8byte %[gotox_r0]; \
> ret0_%=: \
> r0 = 0; \
> exit; \
> ret1_%=: \
> r0 = 1; \
> call bar; \
> exit; \
> " : \
> : __imm_insn(gotox_r0, BPF_RAW_INSN(BPF_JMP | BPF_JA | BPF_X, BPF_REG_0, 0, 0 , 0))
> : __clobber_all);
> }
>
> __used
> static int bar(void)
> {
> return 0;
> }
>
> Note a call instruction referring bar(). It triggers the code path
> above (we need a test case with subprograms in verifier_gotox).
> The test case fails to load with the following error:
>
> libbpf: invalid jump insn idx[0]: 0, no subprog found
> libbpf: prog 'foo': relo #0: can't create jump table: sym_off 368
> libbpf: prog 'foo': failed to relocate data references: -EINVAL
>
> If I remove the `call bar;`, test case loads and passes.
>
> > > +
> > > + /*
> > > + * LLVM-generated jump tables contain u64 records, however
> > > + * should contain values that fit in u32.
> > > + */
> > > + if (insn_off > UINT32_MAX) {
> > > + pr_warn("invalid jump table value %llx at offset %d\n",
> ^^^^
> Nit: maybe add 0x prefix here?
Sure, added.
> > > + jt[i], sym_off + i);
> > > + err = -EINVAL;
> > > + goto err_close;
> > > + }
> > > +
> > > + val.orig_off = insn_off;
> > > + err = bpf_map_update_elem(map_fd, &i, &val, 0);
> > > + if (err)
> > > + goto err_close;
> > > + }
> >
> > [...]
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array
2025-10-26 19:26 ` [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array Anton Protopopov
` (3 preceding siblings ...)
2025-10-27 21:44 ` Eduard Zingerman
@ 2025-10-29 21:54 ` Eduard Zingerman
4 siblings, 0 replies; 29+ messages in thread
From: Eduard Zingerman @ 2025-10-29 21:54 UTC (permalink / raw)
To: Anton Protopopov, bpf, Alexei Starovoitov, Andrii Nakryiko,
Anton Protopopov, Daniel Borkmann, Quentin Monnet, Yonghong Song
On Sun, 2025-10-26 at 19:26 +0000, Anton Protopopov wrote:
> On bpf(BPF_PROG_LOAD) syscall user-supplied BPF programs are
> translated by the verifier into "xlated" BPF programs. During this
> process the original instructions offsets might be adjusted and/or
> individual instructions might be replaced by new sets of instructions,
> or deleted.
>
> Add a new BPF map type which is aimed to keep track of how, for a
> given program, the original instructions were relocated during the
> verification. Also, besides keeping track of the original -> xlated
> mapping, make x86 JIT to build the xlated -> jitted mapping for every
> instruction listed in an instruction array. This is required for every
> future application of instruction arrays: static keys, indirect jumps
> and indirect calls.
>
> A map of the BPF_MAP_TYPE_INSN_ARRAY type must be created with a u32
> keys and value of size 8. The values have different semantics for
> userspace and for BPF space. For userspace a value consists of two
> u32 values – xlated and jitted offsets. For BPF side the value is
> a real pointer to a jitted instruction.
>
> On map creation/initialization, before loading the program, each
> element of the map should be initialized to point to an instruction
> offset within the program. Before the program load such maps should
> be made frozen. After the program verification xlated and jitted
> offsets can be read via the bpf(2) syscall.
>
> If a tracked instruction is removed by the verifier, then the xlated
> offset is set to (u32)-1 which is considered to be too big for a valid
> BPF program offset.
>
> One such a map can, obviously, be used to track one and only one BPF
> program. If the verification process was unsuccessful, then the same
> map can be re-used to verify the program with a different log level.
> However, if the program was loaded fine, then such a map, being
> frozen in any case, can't be reused by other programs even after the
> program release.
>
> Example. Consider the following original and xlated programs:
>
> Original prog: Xlated prog:
>
> 0: r1 = 0x0 0: r1 = 0
> 1: *(u32 *)(r10 - 0x4) = r1 1: *(u32 *)(r10 -4) = r1
> 2: r2 = r10 2: r2 = r10
> 3: r2 += -0x4 3: r2 += -4
> 4: r1 = 0x0 ll 4: r1 = map[id:88]
> 6: call 0x1 6: r1 += 272
> 7: r0 = *(u32 *)(r2 +0)
> 8: if r0 >= 0x1 goto pc+3
> 9: r0 <<= 3
> 10: r0 += r1
> 11: goto pc+1
> 12: r0 = 0
> 7: r6 = r0 13: r6 = r0
> 8: if r6 == 0x0 goto +0x2 14: if r6 == 0x0 goto pc+4
> 9: call 0x76 15: r0 = 0xffffffff8d2079c0
> 17: r0 = *(u64 *)(r0 +0)
> 10: *(u64 *)(r6 + 0x0) = r0 18: *(u64 *)(r6 +0) = r0
> 11: r0 = 0x0 19: r0 = 0x0
> 12: exit 20: exit
>
> An instruction array map, containing, e.g., instructions [0,4,7,12]
> will be translated by the verifier to [0,4,13,20]. A map with
> index 5 (the middle of 16-byte instruction) or indexes greater than 12
> (outside the program boundaries) would be rejected.
>
> The functionality provided by this patch will be extended in consequent
> patches to implement BPF Static Keys, indirect jumps, and indirect calls.
>
> Signed-off-by: Anton Protopopov <a.s.protopopov@gmail.com>
> Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
> ---
[...]
> +void bpf_prog_update_insn_ptrs(struct bpf_prog *prog, void *jit_priv,
> + update_insn_ptr_func_t update_insn_ptr)
> +{
> + struct bpf_insn_array *insn_array;
> + struct bpf_map *map;
> + u32 xlated_off;
> + int i, j;
> +
> + for (i = 0; i < prog->aux->used_map_cnt; i++) {
> + map = prog->aux->used_maps[i];
> + if (!is_insn_array(map))
> + continue;
> +
> + insn_array = cast_insn_array(map);
> + for (j = 0; j < map->max_entries; j++) {
> + xlated_off = insn_array->values[j].xlated_off;
> + if (xlated_off == INSN_DELETED)
> + continue;
> + if (xlated_off < prog->aux->subprog_start)
> + continue;
> + xlated_off -= prog->aux->subprog_start;
> + if (xlated_off >= prog->len)
> + continue;
> +
> + update_insn_ptr(jit_priv, xlated_off,
> + &insn_array->values[j].jitted_off,
> + &insn_array->ips[j]);
> + }
> + }
> +}
Thank you for this update, I think it looks a tad simpler now.
[...]
^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2025-10-29 21:54 UTC | newest]
Thread overview: 29+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-10-26 19:26 [PATCH v7 bpf-next 00/12] BPF indirect jumps Anton Protopopov
2025-10-26 19:26 ` [PATCH v7 bpf-next 01/12] bpf, x86: add new map type: instructions array Anton Protopopov
2025-10-26 20:12 ` Anton Protopopov
2025-10-26 22:34 ` kernel test robot
2025-10-26 22:59 ` kernel test robot
2025-10-27 21:44 ` Eduard Zingerman
2025-10-28 10:10 ` Anton Protopopov
2025-10-29 21:54 ` Eduard Zingerman
2025-10-26 19:26 ` [PATCH v7 bpf-next 02/12] selftests/bpf: add selftests for new insn_array map Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 03/12] bpf: support instructions arrays with constants blinding Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 04/12] selftests/bpf: test instructions arrays with blinding Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 05/12] bpf, x86: allow indirect jumps to r8...r15 Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 06/12] bpf, x86: add support for indirect jumps Anton Protopopov
2025-10-26 20:41 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 07/12] bpf: disasm: add support for BPF_JMP|BPF_JA|BPF_X Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 08/12] bpf, docs: do not state that indirect jumps are not supported Anton Protopopov
2025-10-27 6:30 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 09/12] libbpf: support llvm-generated indirect jumps Anton Protopopov
2025-10-26 20:15 ` Anton Protopopov
2025-10-27 22:09 ` Eduard Zingerman
2025-10-27 22:38 ` Eduard Zingerman
2025-10-27 22:59 ` Eduard Zingerman
2025-10-28 11:36 ` Anton Protopopov
2025-10-28 11:42 ` Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 10/12] bpftool: Recognize insn_array map type Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 11/12] selftests/bpf: add new verifier_gotox test Anton Protopopov
2025-10-26 19:27 ` [PATCH v7 bpf-next 12/12] selftests/bpf: add C-level selftests for indirect jumps Anton Protopopov
2025-10-27 23:25 ` Eduard Zingerman
2025-10-28 10:59 ` Anton Protopopov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).