[PATCH bpf-next 1/6] bpf: Reject load-acquire from pointers requiring fault protection

[PATCH bpf-next 1/6] bpf: Reject load-acquire from pointers requiring fault protection

[Daniel Borkmann]

A BPF_LOAD_ACQ is not rewritten to a BPF_PROBE_MEM load by the verifier,
unlike a regular BPF_LDX, so the JIT emits a plain load with no exception
table entry and a fault panics the kernel instead of being handled.

Reject the source pointer types that a BPF_LDX would have had that fault
protection applied to, i.e. the ones bpf_convert_ctx_accesses() turns
into BPF_PROBE_MEM: a bare PTR_TO_BTF_ID, PTR_TO_BTF_ID | PTR_UNTRUSTED,
PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED and PTR_TO_MEM | MEM_RDONLY |
PTR_UNTRUSTED.

This is reachable e.g. by loading ->mm out of a trusted task_struct
yields an untrusted pointer to mm_struct, and it is NULL for a kernel
thread:

  [...]
  SEC("tp_btf/sched_switch")
  int BPF_PROG(demo, bool preempt, struct task_struct *prev,
               struct task_struct *next)
  {
      struct mm_struct *mm = next->mm;  /* untrusted */

      out_ldx = (__u64)mm->pgd;         /* BPF_LDX      */
      out_acq = load_acquire(&mm->pgd); /* BPF_LOAD_ACQ */
      return 0;
  }
  [...]

Both dereference the same pointer, but only the BPF_LDX is protected
(x86-64 JIT, jump targets shown prog-relative):

  [...]
  ; out_ldx = (__u64)mm->pgd;
  17:   movq    $-10485760, %r10
  1e:   movq    %rsi, %r11
  21:   addq    $184, %r11
  28:   subq    %r10, %r11
  2b:   movabsq $140737498841088, %r10
  35:   cmpq    %r10, %r11
  38:   ja      0x3e                 <-- kernel addr?
  3a:   xorl    %edi, %edi           <-- no: dst = 0, skip the load
  3c:   jmp     0x45
  3e:   movq    184(%rsi), %rdi      <-- yes: load + extable entry
  [...]
  ; load_acquire(&mm->pgd)
  53:	movq    %rsi, %rdi
  56:	movq    184(%rdi), %rax       <-- no check, no extable entry
  [...]

Note that BPF_PROBE_MEM is not visible in a bpftool xlated dump, as
bpf_insn_prepare_dump() rewrites it back to BPF_MEM.

A PTR_TRUSTED pointer is deliberately not on the list. Such a load is
not converted either, but it does not need to be, since the pointer is
guaranteed live, so load-acquire from it stays allowed.

The check is gated on BPF_LOAD_ACQ so that atomic RMW and store-release
error messages are unchanged; writes (RMW / store-release) to such
pointers are already rejected elsewhere, so only load-acquire needs this.

Fixes: 880442305a39 ("bpf: Introduce load-acquire and store-release instructions")
Reported-by: STAR Labs SG <info@starlabs.sg>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
---
 kernel/bpf/verifier.c | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index d925197c2e5f..43415ff0863d 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -4921,6 +4921,30 @@ static bool is_arena_reg(struct bpf_verifier_env *env, int regno)
 	return reg->type == PTR_TO_ARENA;
 }
 
+static bool is_load_acq_unsafe(struct bpf_verifier_env *env, int regno,
+			       struct bpf_insn *insn)
+{
+	const struct bpf_reg_state *reg = reg_state(env, regno);
+
+	/*
+	 * A BPF_LOAD_ACQ is not rewritten to a BPF_PROBE_MEM load by the
+	 * verifier, unlike a regular BPF_LDX. The JIT would emit a plain load
+	 * with no exception table entry, so a fault (e.g. NULL deref) crashes
+	 * the kernel instead of being handled.
+	 *
+	 * Reject the source pointer types that a BPF_LDX would have had that
+	 * fault protection applied to, i.e. the ones bpf_convert_ctx_accesses()
+	 * turns into BPF_PROBE_MEM: a bare PTR_TO_BTF_ID and any PTR_UNTRUSTED
+	 * pointer (untrusted btf ids, untrusted MEM_ALLOC, rdonly untrusted
+	 * memory). A PTR_TRUSTED pointer is not among them, is not converted,
+	 * and stays allowed. Same for the other flagged PTR_TO_BTF_ID variants
+	 * (MEM_ALLOC, MEM_RCU, ...), hence the exact match on the base type.
+	 */
+	return insn->imm == BPF_LOAD_ACQ &&
+	       (reg->type == PTR_TO_BTF_ID ||
+		(type_flag(reg->type) & PTR_UNTRUSTED));
+}
+
 /* Return false if @regno contains a pointer whose type isn't supported for
  * atomic instruction @insn.
  */
@@ -4937,7 +4961,8 @@ static bool atomic_ptr_type_ok(struct bpf_verifier_env *env, int regno,
 		return false;
 	if (is_arena_reg(env, regno))
 		return bpf_jit_supports_insn(insn, true);
-
+	if (is_load_acq_unsafe(env, regno, insn))
+		return false;
 	return true;
 }
 
-- 
2.43.0

[PATCH bpf-next 2/6] bpf, riscv: Add and use bpf_atomic_is_load_acq() helper

[Daniel Borkmann]

A load-acquire is the only BPF_STX class instruction that reads from
src_reg into dst_reg, that is, it has the operand roles of a BPF_LDX.
JIT code which tells loads from stores apart by instruction class alone
has to special case it, for example when deciding which register holds
the faulting address and which one to clear from an exception handler.

riscv64 already does so, open coded as a bare insn->imm test. Add a
bpf_atomic_is_load_acq() helper and convert riscv64 over to it, so that
the x86-64 and arm64 JITs can use the same helper in subsequent patches.

Unlike bpf_atomic_is_load_store(), which presumes that its argument is
already known to be a BPF_ATOMIC instruction, the new helper is called
from code which still sees all instruction classes, so it checks class
and mode itself.

Also, move bpf_atomic_is_load_store() to filter.h next to BPF_ATOMIC_OP,
so that both helpers stay together. No functional change intended.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
---
 arch/riscv/net/bpf_jit_comp64.c |  2 +-
 include/linux/bpf.h             | 15 ---------------
 include/linux/filter.h          | 31 +++++++++++++++++++++++++++++++
 3 files changed, 32 insertions(+), 16 deletions(-)

diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c
index 8fe8969fb8a0..6b9972b07c1b 100644
--- a/arch/riscv/net/bpf_jit_comp64.c
+++ b/arch/riscv/net/bpf_jit_comp64.c
@@ -1994,7 +1994,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 		/* ret can be 1 (skip-zext); extable entry still needs to be added */
 		if (ret >= 0)
 			ret = add_exception_handler(insn,
-				insn->imm == BPF_LOAD_ACQ ? rd : REG_DONT_CLEAR_MARKER,
+				bpf_atomic_is_load_acq(insn) ? rd : REG_DONT_CLEAR_MARKER,
 				ctx) ?: ret;
 
 		if (ret)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 73bacfc6444d..d79bf7557ef6 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1132,21 +1132,6 @@ static inline bool bpf_pseudo_func(const struct bpf_insn *insn)
 	return bpf_is_ldimm64(insn) && insn->src_reg == BPF_PSEUDO_FUNC;
 }
 
-/* Given a BPF_ATOMIC instruction @atomic_insn, return true if it is an
- * atomic load or store, and false if it is a read-modify-write instruction.
- */
-static inline bool
-bpf_atomic_is_load_store(const struct bpf_insn *atomic_insn)
-{
-	switch (atomic_insn->imm) {
-	case BPF_LOAD_ACQ:
-	case BPF_STORE_REL:
-		return true;
-	default:
-		return false;
-	}
-}
-
 struct bpf_prog_ops {
 	int (*test_run)(struct bpf_prog *prog, const union bpf_attr *kattr,
 			union bpf_attr __user *uattr);
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 32d5297c557e..41b02d53e222 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -383,6 +383,37 @@ static inline bool insn_is_cast_user(const struct bpf_insn *insn)
 /* Legacy alias */
 #define BPF_STX_XADD(SIZE, DST, SRC, OFF) BPF_ATOMIC_OP(SIZE, BPF_ADD, DST, SRC, OFF)
 
+/*
+ * Given a BPF_ATOMIC instruction @atomic_insn, return true if it is an
+ * atomic load or store, and false if it is a read-modify-write instruction.
+ */
+static inline bool
+bpf_atomic_is_load_store(const struct bpf_insn *atomic_insn)
+{
+	switch (atomic_insn->imm) {
+	case BPF_LOAD_ACQ:
+	case BPF_STORE_REL:
+		return true;
+	default:
+		return false;
+	}
+}
+
+/*
+ * A load-acquire is the only BPF_STX class instruction that reads into
+ * dst_reg from src_reg + off16, i.e. it has the operand roles of a BPF_LDX.
+ * Unlike bpf_atomic_is_load_store(), @insn is not assumed to be a BPF_ATOMIC
+ * instruction here, so that callers which walk all instruction classes can
+ * use this directly.
+ */
+static inline bool bpf_atomic_is_load_acq(const struct bpf_insn *insn)
+{
+	return BPF_CLASS(insn->code) == BPF_STX &&
+	       (BPF_MODE(insn->code) == BPF_ATOMIC ||
+		BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) &&
+	       insn->imm == BPF_LOAD_ACQ;
+}
+
 /* Memory store, *(uint *) (dst_reg + off16) = imm32 */
 
 #define BPF_ST_MEM(SIZE, DST, OFF, IMM)				\
-- 
2.43.0

[PATCH bpf-next 3/6] bpf, x86: Fix exception table metadata for arena load-acquire

[Daniel Borkmann]

A load-acquire from an arena pointer is converted to BPF_PROBE_ATOMIC and
gets an exception table entry, but the entry is filled in as if it were a
store, since populate_extable() decides based on instruction class alone
and a load-acquire is of BPF_STX class:

  if (BPF_CLASS(insn->code) == BPF_LDX) {
      arena_reg = reg2pt_regs[src_reg];
      fixup_reg = reg2pt_regs[dst_reg];
  } else {
      arena_reg = reg2pt_regs[dst_reg];
      fixup_reg = DONT_CLEAR;
  }

For a load-acquire dst_reg holds the loaded value and src_reg holds the
address, so both assignments in the else branch are wrong. On a fault
over an unmapped arena page ex_handler_bpf() then:

  - computes the reported address from the value register instead
    of the address register
  - reports the access as a WRITE, since it derives the direction
    from fixup_reg == DONT_CLEAR
  - leaves dst_reg untouched, so the program continues with a stale
    value instead of the 0 that BPF_PROBE_* loads deliver

The access itself is emitted correctly, emit_atomic_ld_st_index() uses
src_reg as the address, so this is a broken probe contract and a wrong
diagnostic rather than a memory safety issue.

Use bpf_atomic_is_load_acq() helper so a load-acquire takes the load path.

Fixes: 5341c9a4d833 ("bpf, x86: Support load-acquire and store-release instructions")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: Peilin Ye <yepeilin@google.com>
---
 arch/x86/net/bpf_jit_comp.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 01e7ce569c1e..88ed95b2eaa7 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2331,8 +2331,13 @@ st:			insn_off = insn->off;
 				 * BPF_PROBE_ATOMIC) before being used for the memory access. Pass
 				 * the reg holding the unmodified 32-bit address to
 				 * ex_handler_bpf().
+				 *
+				 * A load-acquire is of BPF_STX class, but reads from src_reg
+				 * into dst_reg like a BPF_LDX does, hence it must not be
+				 * treated as a store here.
 				 */
-				if (BPF_CLASS(insn->code) == BPF_LDX) {
+				if (BPF_CLASS(insn->code) == BPF_LDX ||
+				    bpf_atomic_is_load_acq(insn)) {
 					arena_reg = reg2pt_regs[src_reg];
 					fixup_reg = reg2pt_regs[dst_reg];
 				} else {
-- 
2.43.0

[PATCH bpf-next 4/6] bpf, arm64: Fix exception table metadata for arena load-acquire

[Daniel Borkmann]

Same problem as on x86-64: add_exception_handler() decides whether an
instruction is a load by its class, and a load-acquire is of BPF_STX
class even though it reads from src_reg into dst_reg. As a result ...

  if (BPF_CLASS(insn->code) != BPF_LDX)
          dst_reg = DONT_CLEAR;

... drops the register to clear, and ...

  if (BPF_CLASS(insn->code) == BPF_LDX)
          arena_reg = bpf2a64[insn->src_reg];
  else
          arena_reg = bpf2a64[insn->dst_reg];

... hands ex_handler_bpf() the value register instead of the address
register. A load-acquire from an arena pointer that faults on an
unmapped page is therefore reported as a WRITE at a bogus address,
and dst_reg keeps its previous value instead of being cleared to 0.

Note that emit_atomic_ld_st() already picks src_reg as the address
for BPF_LOAD_ACQ, so only the exception table metadata was out of sync
with the emitted access.

Same as on x86-64, use bpf_atomic_is_load_acq() so a load-acquire takes
the load path.

Fixes: 9bb12368d539 ("bpf, arm64: Support load-acquire and store-release instructions")
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: Peilin Ye <yepeilin@google.com>
---
 arch/arm64/net/bpf_jit_comp.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index 4cdc7dfb05ba..d14d297ebb96 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -1178,7 +1178,12 @@ static int add_exception_handler(const struct bpf_insn *insn,
 
 	ex->insn = ins_offset;
 
-	if (BPF_CLASS(insn->code) != BPF_LDX)
+	/*
+	 * A load-acquire is of BPF_STX class, but reads from src_reg into
+	 * dst_reg like a BPF_LDX does, hence it must not be treated as a store
+	 * here.
+	 */
+	if (BPF_CLASS(insn->code) != BPF_LDX && !bpf_atomic_is_load_acq(insn))
 		dst_reg = DONT_CLEAR;
 
 	ex->fixup = FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg);
@@ -1193,7 +1198,7 @@ static int add_exception_handler(const struct bpf_insn *insn,
 		 * memory access. Pass the reg holding the unmodified 32-bit address to
 		 * ex_handler_bpf.
 		 */
-		if (BPF_CLASS(insn->code) == BPF_LDX)
+		if (BPF_CLASS(insn->code) == BPF_LDX || bpf_atomic_is_load_acq(insn))
 			arena_reg = bpf2a64[insn->src_reg];
 		else
 			arena_reg = bpf2a64[insn->dst_reg];
-- 
2.43.0

[PATCH bpf-next 5/6] selftests/bpf: Add arena fault test for load-acquire

[Daniel Borkmann]

Add stream_arena_load_acquire_fault, which performs a load-acquire from an
unmapped arena address, next to the existing read and write fault tests.

The test covers both halves of the JIT bug that treated a load-acquire as
a store when populating its exception table entry:

  - the fault has to be reported as a READ, and at the address held by
    the source register, which __stderr() and test_address() check, and
  - the destination register has to be cleared by the fault handler,
    which the program checks by poisoning it before the load-acquire
    and returning it, so __retval(0) fails if it is left untouched

  # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t stream_arena_fault_address
  [...]
  #462/1   stream_arena_fault_address/read_fault:OK
  #462/2   stream_arena_fault_address/write_fault:OK
  #462/3   stream_arena_fault_address/load_acquire_fault:OK
  #462     stream_arena_fault_address:OK
  Summary: 1/3 PASSED, 0 SKIPPED, 0 FAILED

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
---
 .../testing/selftests/bpf/prog_tests/stream.c |  2 +
 tools/testing/selftests/bpf/progs/stream.c    | 46 +++++++++++++++++++
 2 files changed, 48 insertions(+)

diff --git a/tools/testing/selftests/bpf/prog_tests/stream.c b/tools/testing/selftests/bpf/prog_tests/stream.c
index c3cce5c292bd..15dd3ae2a84b 100644
--- a/tools/testing/selftests/bpf/prog_tests/stream.c
+++ b/tools/testing/selftests/bpf/prog_tests/stream.c
@@ -103,6 +103,8 @@ void test_stream_arena_fault_address(void)
 		test_address(skel->progs.stream_arena_read_fault, &skel->bss->fault_addr);
 	if (test__start_subtest("write_fault"))
 		test_address(skel->progs.stream_arena_write_fault, &skel->bss->fault_addr);
+	if (test__start_subtest("load_acquire_fault"))
+		test_address(skel->progs.stream_arena_load_acquire_fault, &skel->bss->fault_addr);
 
 	stream__destroy(skel);
 }
diff --git a/tools/testing/selftests/bpf/progs/stream.c b/tools/testing/selftests/bpf/progs/stream.c
index 8d8e53d37266..82c855889180 100644
--- a/tools/testing/selftests/bpf/progs/stream.c
+++ b/tools/testing/selftests/bpf/progs/stream.c
@@ -185,6 +185,52 @@ int stream_arena_read_fault(void *ctx)
 	return 0;
 }
 
+SEC("syscall")
+__arch_x86_64
+__arch_arm64
+__success __retval(0)
+__stderr("ERROR: Arena READ access at unmapped address 0x{{.*}}")
+__stderr("CPU: {{[0-9]+}} UID: 0 PID: {{[0-9]+}} Comm: {{.*}}")
+__stderr("Call trace:\n"
+"{{([a-zA-Z_][a-zA-Z0-9_]*\\+0x[0-9a-fA-F]+/0x[0-9a-fA-F]+\n"
+"|[ \t]+[^\n]+\n)*}}")
+int stream_arena_load_acquire_fault(void *ctx)
+{
+	struct bpf_arena *ptr = (void *)&arena;
+	u64 user_vm_start, val;
+
+	/*
+	 * Prevent GCC bounds warning: casting &arena to struct bpf_arena *
+	 * triggers bounds checking since the map definition is smaller than
+	 * struct bpf_arena. barrier_var() makes the pointer opaque to GCC,
+	 * preventing the bounds analysis.
+	 */
+	barrier_var(ptr);
+	user_vm_start = ptr->user_vm_start;
+	fault_addr = user_vm_start + 0x7fff;
+	bpf_addr_space_cast(user_vm_start, 0, 1);
+	/*
+	 * A load-acquire is of BPF_STX class, but reads from src_reg into
+	 * dst_reg. Poison dst_reg up front: the fault has to be reported
+	 * as a READ from the src_reg address, and the exception handler has
+	 * to clear dst_reg, so the returned value must be 0.
+	 *
+	 * The load-acquire is open coded as BPF_ATOMIC_OP(BPF_W, BPF_LOAD_ACQ,
+	 * BPF_REG_0, BPF_REG_1, 0x7fff) since <linux/filter.h> cannot be
+	 * included alongside vmlinux.h.
+	 */
+	asm volatile (
+		"r1 = %[user_vm_start];"
+		"r0 = 1;"
+		".8byte 0x000001007fff10c3;" /* r0 = load_acquire((u32 *)(r1 + 0x7fff)) */
+		"%[val] = r0;"
+		: [val] "=r" (val)
+		: [user_vm_start] "r" (user_vm_start)
+		: "r0", "r1"
+	);
+	return val;
+}
+
 static __noinline void subprog(void)
 {
 	int __arena *addr = (int __arena *)0xdeadbeef;
-- 
2.43.0

[PATCH bpf-next 6/6] selftests/bpf: Add load-acquire test for probe-memory pointer types

[Daniel Borkmann]

Add a verifier test that a BPF_LOAD_ACQ from a rdonly_untrusted_mem pointer
(PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED, obtained via bpf_rdonly_cast())
is rejected. Such a source requires BPF_PROBE_MEM fault protection which
is not applied to atomic loads; without the verifier fix the load is accepted
and would crash the kernel on a fault.

  # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t verifier_load_acquire
  [...]
  #621/1   verifier_load_acquire/load-acquire, 8-bit:OK
  #621/2   verifier_load_acquire/load-acquire, 8-bit @unpriv:OK
  #621/3   verifier_load_acquire/load-acquire, 16-bit:OK
  #621/4   verifier_load_acquire/load-acquire, 16-bit @unpriv:OK
  #621/5   verifier_load_acquire/load-acquire, 32-bit:OK
  #621/6   verifier_load_acquire/load-acquire, 32-bit @unpriv:OK
  #621/7   verifier_load_acquire/load-acquire, 64-bit:OK
  #621/8   verifier_load_acquire/load-acquire, 64-bit @unpriv:OK
  [...]
  #621/19  verifier_load_acquire/load-acquire from rdonly_untrusted_mem pointer:OK
  #621/20  verifier_load_acquire/load-acquire with invalid register R15:OK
  #621/21  verifier_load_acquire/load-acquire with invalid register R15 @unpriv:OK
  #621/22  verifier_load_acquire/load-acquire from pkt pointer:OK
  #621/23  verifier_load_acquire/load-acquire from flow_keys pointer:OK
  #621/24  verifier_load_acquire/load-acquire from sock pointer:OK
  #621     verifier_load_acquire:OK
  Summary: 1/24 PASSED, 0 SKIPPED, 0 FAILED

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
---
 .../bpf/progs/verifier_load_acquire.c         | 28 +++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/tools/testing/selftests/bpf/progs/verifier_load_acquire.c b/tools/testing/selftests/bpf/progs/verifier_load_acquire.c
index ae1dab1b0cbb..d17026d7480d 100644
--- a/tools/testing/selftests/bpf/progs/verifier_load_acquire.c
+++ b/tools/testing/selftests/bpf/progs/verifier_load_acquire.c
@@ -3,6 +3,7 @@
 
 #include <linux/bpf.h>
 #include <bpf/bpf_helpers.h>
+#include <bpf/bpf_core_read.h>
 #include "../../../include/linux/filter.h"
 #include "bpf_misc.h"
 
@@ -221,6 +222,33 @@ __naked void load_acquire_from_sock_pointer(void)
 	: __clobber_all);
 }
 
+SEC("socket")
+__description("load-acquire from rdonly_untrusted_mem pointer")
+__failure __msg("BPF_ATOMIC loads from R{{[0-9]+}} rdonly_untrusted_mem is not allowed")
+int load_acquire_from_rdonly_untrusted_mem(void *ctx)
+{
+	__u64 val = 0;
+	void *p;
+
+	/*
+	 * bpf_rdonly_cast(x, 0) yields PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED.
+	 * A regular BPF_LDX from it is rewritten to BPF_PROBE_MEM, but a
+	 * load-acquire is not, so it must be rejected, otherwise the JIT emits
+	 * a plain load with no exception table entry and a fault would crash
+	 * the kernel.
+	 */
+	p = bpf_rdonly_cast(&val, 0);
+	asm volatile (
+	"r1 = %[p];"
+	".8byte %[load_acquire_insn];" // r0 = load_acquire((u64 *)(r1 + 0));
+	:
+	: [p] "r" (p),
+	  __imm_insn(load_acquire_insn,
+		     BPF_ATOMIC_OP(BPF_DW, BPF_LOAD_ACQ, BPF_REG_0, BPF_REG_1, 0))
+	: "r0", "r1");
+	return 0;
+}
+
 SEC("socket")
 __description("load-acquire with invalid register R15")
 __failure __failure_unpriv __msg("R15 is invalid")
-- 
2.43.0

Re: [PATCH bpf-next 4/6] bpf, arm64: Fix exception table metadata for arena load-acquire

[Puranjay Mohan]

On Thu, Aug 6, 2026 at 3:39 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> Same problem as on x86-64: add_exception_handler() decides whether an
> instruction is a load by its class, and a load-acquire is of BPF_STX
> class even though it reads from src_reg into dst_reg. As a result ...
>
>   if (BPF_CLASS(insn->code) != BPF_LDX)
>           dst_reg = DONT_CLEAR;
>
> ... drops the register to clear, and ...
>
>   if (BPF_CLASS(insn->code) == BPF_LDX)
>           arena_reg = bpf2a64[insn->src_reg];
>   else
>           arena_reg = bpf2a64[insn->dst_reg];
>
> ... hands ex_handler_bpf() the value register instead of the address
> register. A load-acquire from an arena pointer that faults on an
> unmapped page is therefore reported as a WRITE at a bogus address,
> and dst_reg keeps its previous value instead of being cleared to 0.
>
> Note that emit_atomic_ld_st() already picks src_reg as the address
> for BPF_LOAD_ACQ, so only the exception table metadata was out of sync
> with the emitted access.
>
> Same as on x86-64, use bpf_atomic_is_load_acq() so a load-acquire takes
> the load path.
>
> Fixes: 9bb12368d539 ("bpf, arm64: Support load-acquire and store-release instructions")
> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
> Cc: Peilin Ye <yepeilin@google.com>
> ---
>  arch/arm64/net/bpf_jit_comp.c | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
> index 4cdc7dfb05ba..d14d297ebb96 100644
> --- a/arch/arm64/net/bpf_jit_comp.c
> +++ b/arch/arm64/net/bpf_jit_comp.c
> @@ -1178,7 +1178,12 @@ static int add_exception_handler(const struct bpf_insn *insn,
>
>         ex->insn = ins_offset;
>
> -       if (BPF_CLASS(insn->code) != BPF_LDX)
> +       /*
> +        * A load-acquire is of BPF_STX class, but reads from src_reg into
> +        * dst_reg like a BPF_LDX does, hence it must not be treated as a store
> +        * here.
> +        */
> +       if (BPF_CLASS(insn->code) != BPF_LDX && !bpf_atomic_is_load_acq(insn))
>                 dst_reg = DONT_CLEAR;
>
>         ex->fixup = FIELD_PREP(BPF_FIXUP_REG_MASK, dst_reg);
> @@ -1193,7 +1198,7 @@ static int add_exception_handler(const struct bpf_insn *insn,
>                  * memory access. Pass the reg holding the unmodified 32-bit address to
>                  * ex_handler_bpf.
>                  */
> -               if (BPF_CLASS(insn->code) == BPF_LDX)
> +               if (BPF_CLASS(insn->code) == BPF_LDX || bpf_atomic_is_load_acq(insn))
>                         arena_reg = bpf2a64[insn->src_reg];
>                 else
>                         arena_reg = bpf2a64[insn->dst_reg];
> --
> 2.43.0
>

Thanks for fixing this.

Reviewed-by: Puranjay Mohan <puranjay@kernel.org>

Re: [PATCH bpf-next 3/6] bpf, x86: Fix exception table metadata for arena load-acquire

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Incomplete fix: ARM64 load-acquire and all architectures' RMW atomics fail to clear the return register on fault
--

--- Patch [3]: [PATCH bpf-next 3/6] bpf, x86: Fix exception table metadata for arena load-acquire ---
commit 9812c186ec8bce0c6112e5b77175a8708cc4c31e
Author: Daniel Borkmann <daniel@iogearbox.net>
Subject: bpf, x86: Fix exception table metadata for arena load-acquire

A load-acquire from an arena pointer gets an exception table entry, but the
entry is filled in as if it were a store. This patch uses
bpf_atomic_is_load_acq() so a load-acquire correctly takes the load path.

Cc: Peilin Ye <yepeilin@google.com>

> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c

[ ... ]

>  				 * BPF_PROBE_ATOMIC) before being used for the memory access. Pass
>  				 * the reg holding the unmodified 32-bit address to
>  				 * ex_handler_bpf().
> +				 *
> +				 * A load-acquire is of BPF_STX class, but reads from src_reg
> +				 * into dst_reg like a BPF_LDX does, hence it must not be
> +				 * treated as a store here.
>  				 */
> -				if (BPF_CLASS(insn->code) == BPF_LDX) {
> +				if (BPF_CLASS(insn->code) == BPF_LDX ||
> +				    bpf_atomic_is_load_acq(insn)) {
>  					arena_reg = reg2pt_regs[src_reg];
>  					fixup_reg = reg2pt_regs[dst_reg];
>  				} else {
>  					arena_reg = reg2pt_regs[dst_reg];
>  					fixup_reg = DONT_CLEAR;
>  				}

[Severity: High]
This isn't a bug introduced by this patch, but could this fix be missing other
RMW atomics on x86, as well as leaving similar vulnerabilities on ARM64 and
PowerPC?

For x86_64, RMW atomics like BPF_XCHG, BPF_CMPXCHG, and BPF_FETCH ops return a
value that must be cleared, but they seem to fall into the else branch in
populate_extable() and get fixup_reg = DONT_CLEAR. If a fault occurs on an
unmapped arena address, the exception handler skips the instruction without
clearing the return register. Since the verifier treats the result of these
atomic operations as an unconstrained scalar, a BPF program could stash a kernel
pointer in the return register beforehand, then leak it to userspace after the
fault.

Additionally, this issue appears to exist on other architectures.

On ARM64, add_exception_handler() in arch/arm64/net/bpf_jit_comp.c assumes all
BPF_STX instructions are stores, failing to clear dst_reg for load_acq and other
returning atomics:

    if (BPF_CLASS(insn->code) != BPF_LDX)
        dst_reg = DONT_CLEAR;

On PowerPC, bpf_add_extable_entry() in arch/powerpc/net/bpf_jit_comp.c emits a
NOP fixup instead of clearing the destination register for any BPF_STX
instruction:

    fixup[0] = PPC_RAW_LI(dst_reg, 0);
    if (BPF_CLASS(code) == BPF_ST || BPF_CLASS(code) == BPF_STX)
        fixup[0] = PPC_RAW_NOP();

While load_acq might be unsupported on PowerPC arena pointers, value-returning
atomics like xchg and cmpxchg are supported and vulnerable to the same leak.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260806143924.319238-1-daniel@iogearbox.net?part=3

Re: [PATCH bpf-next 3/6] bpf, x86: Fix exception table metadata for arena load-acquire

[Daniel Borkmann]

On 8/6/26 4:58 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> 
> Pre-existing issues:
> - [High] Incomplete fix: ARM64 load-acquire and all architectures' RMW atomics fail to clear the return register on fault
yeap, I have this as follow-up locally

Re: [PATCH bpf-next 1/6] bpf: Reject load-acquire from pointers requiring fault protection

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] The verifier fails to convert BPF_LDX instructions to BPF_PROBE_MEM when paths with PTR_TO_MEM and PTR_TO_BTF_ID | PTR_UNTRUSTED merge, allowing unhandled page faults.
--

--- Patch [1]: [PATCH bpf-next 1/6] bpf: Reject load-acquire from pointers requiring fault protection ---
commit 7bfb4a68d477eb7f566fc7d106ee4703a225c192
Author: Daniel Borkmann <daniel@iogearbox.net>

bpf: Reject load-acquire from pointers requiring fault protection

This patch prevents BPF_LOAD_ACQ instructions from being used with
pointer types that require exception table entries for fault protection.
It rejects untrusted pointers and bare PTR_TO_BTF_ID pointers since the
JIT does not emit exception tables for load-acquires, which could result
in a kernel panic on a page fault.

> diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
> index d952bd95cbb7b..78637faf13508 100644
> --- a/kernel/bpf/verifier.c
> +++ b/kernel/bpf/verifier.c

[ ... ]

> @@ -4921,6 +4921,30 @@ static bool is_arena_reg(struct bpf_verifier_env *env, int regno)
>  	return reg->type == PTR_TO_ARENA;
>  }
>  
> +static bool is_load_acq_unsafe(struct bpf_verifier_env *env, int regno,
> +			       struct bpf_insn *insn)
> +{
> +	const struct bpf_reg_state *reg = reg_state(env, regno);
> +
> +	/*
> +	 * A BPF_LOAD_ACQ is not rewritten to a BPF_PROBE_MEM load by the
> +	 * verifier, unlike a regular BPF_LDX. The JIT would emit a plain load
> +	 * with no exception table entry, so a fault (e.g. NULL deref) crashes
> +	 * the kernel instead of being handled.
> +	 *
> +	 * Reject the source pointer types that a BPF_LDX would have had that
> +	 * fault protection applied to, i.e. the ones bpf_convert_ctx_accesses()
> +	 * turns into BPF_PROBE_MEM: a bare PTR_TO_BTF_ID and any PTR_UNTRUSTED
> +	 * pointer (untrusted btf ids, untrusted MEM_ALLOC, rdonly untrusted
> +	 * memory). A PTR_TRUSTED pointer is not among them, is not converted,
> +	 * and stays allowed. Same for the other flagged PTR_TO_BTF_ID variants
> +	 * (MEM_ALLOC, MEM_RCU, ...), hence the exact match on the base type.
> +	 */
> +	return insn->imm == BPF_LOAD_ACQ &&
> +	       (reg->type == PTR_TO_BTF_ID ||
> +		(type_flag(reg->type) & PTR_UNTRUSTED));

[Severity: High]
This is a pre-existing issue, but since the comment mentions pointer types
that bpf_convert_ctx_accesses() turns into BPF_PROBE_MEM, does it currently
miss handling PTR_TO_MEM | PTR_UNTRUSTED?

When paths with PTR_TO_MEM and PTR_TO_BTF_ID | PTR_UNTRUSTED merge,
save_aux_ptr_type() produces PTR_TO_MEM | PTR_UNTRUSTED without MEM_RDONLY:

kernel/bpf/verifier.c:save_aux_ptr_type() {
    ...
			if (is_ptr_to_mem(type) || is_ptr_to_mem(*prev_type))
				merged_type = PTR_TO_MEM;
			else
				merged_type = PTR_TO_BTF_ID;
			if ((type & PTR_UNTRUSTED) || (*prev_type & PTR_UNTRUSTED))
				merged_type |= PTR_UNTRUSTED;
			if ((type & MEM_RDONLY) || (*prev_type & MEM_RDONLY))
				merged_type |= MEM_RDONLY;
    ...
}

However, in kernel/bpf/fixups.c:bpf_convert_ctx_accesses(), it seems this
exact type combination is not converted to BPF_PROBE_MEM because the switch
only handles the MEM_RDONLY case:

kernel/bpf/fixups.c:bpf_convert_ctx_accesses() {
    ...
		case PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED:
		case PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED:
			if (type == BPF_READ) {
    ...
}

Could this lead to an unhandled page fault and kernel panic if an untrusted
pointer and a valid writable memory pointer (like from a dynptr) reach the
same BPF_LDX instruction, bypassing the fault protection?

> +}
> +

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260806143924.319238-1-daniel@iogearbox.net?part=1

Re: [PATCH bpf-next 1/6] bpf: Reject load-acquire from pointers requiring fault protection

[Daniel Borkmann]

On 8/6/26 5:01 PM, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> 
> Pre-existing issues:
> - [High] The verifier fails to convert BPF_LDX instructions to BPF_PROBE_MEM when paths with PTR_TO_MEM and PTR_TO_BTF_ID | PTR_UNTRUSTED merge, allowing unhandled page faults.
orthogonal issue, I'll look into a separate fix.

Re: [PATCH bpf-next 5/6] selftests/bpf: Add arena fault test for load-acquire

[Daniel Borkmann]

On 8/6/26 4:39 PM, Daniel Borkmann wrote:
> Add stream_arena_load_acquire_fault, which performs a load-acquire from an
> unmapped arena address, next to the existing read and write fault tests.
> 
> The test covers both halves of the JIT bug that treated a load-acquire as
> a store when populating its exception table entry:
> 
>    - the fault has to be reported as a READ, and at the address held by
>      the source register, which __stderr() and test_address() check, and
>    - the destination register has to be cleared by the fault handler,
>      which the program checks by poisoning it before the load-acquire
>      and returning it, so __retval(0) fails if it is left untouched
> 
>    # LDLIBS=-static PKG_CONFIG='pkg-config --static' ./vmtest.sh -- ./test_progs -t stream_arena_fault_address
>    [...]
>    #462/1   stream_arena_fault_address/read_fault:OK
>    #462/2   stream_arena_fault_address/write_fault:OK
>    #462/3   stream_arena_fault_address/load_acquire_fault:OK
>    #462     stream_arena_fault_address:OK
>    Summary: 1/3 PASSED, 0 SKIPPED, 0 FAILED
> 
> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
(will push a v2 given this one fails on s390 due to encoding)