[PATCH bpf-next 0/2] bpf, x86: Support fetching AND/OR/XOR atomics in arena

[Puranjay Mohan <puranjay@kernel.org>]

A fetching AND/OR/XOR against arena memory is rejected on x86-64:

  BPF_ATOMIC stores into R1 arena is not allowed

x86-64 has no single instruction for these, so the JIT lowers them to a
CMPXCHG loop. The loop performs two memory accesses, the load of the old
value and the CMPXCHG itself, and either can fault when the arena page
goes away. The verifier reserves one exception table entry per
instruction, so there was nowhere to record the second one and
bpf_jit_supports_insn() refused the three opcodes instead.

x86-64 is the only architecture that needs this. riscv64 has native
AMOAND/AMOOR/AMOXOR with fetch, s390 has LAN/LAO/LAX, and arm64 with LSE
has LDCLRAL/LDSETAL/LDEORAL, so all three already accept these in an
arena. arm64 without LSE rejects every arena RMW atomic and is unaffected
either way, since the CMPXCHG that such a lowering would need is not
available there in an arena either.

Patch 1 emits the loop with R12-indexed addressing and gives each of the
two accesses its own exception table entry. Both entries resume past the
whole loop rather than past the faulting instruction, with the fetch
destination cleared, so a fault cannot re-enter the loop. The JIT accounts
for the extra entry itself by rescanning the instruction stream in
bpf_int_jit_compile() before the extable is sized; s390 does the same in
bpf_jit_alloc() for its BPF_XCHG lowering.

Patch 2 makes the selftests actually cover this. The existing arena
and/or/xor tests discarded the returned value, so clang emitted the
non-fetching instruction and the fetching one was never exercised. That
was deliberate: commit 2897b1e2a2f4 ("selftests/bpf: Fix arena_atomics
failure due to llvm change") switched them to __c11_atomic_fetch_*() with
memory_order_relaxed specifically to obtain a non-fetching instruction,
because of the limitation patch 1 removes. They go back to plain
__sync_fetch_and_*() and check the old value, x86 is dropped from the
exclusion list of the uaf test that covers the fault path, and a new
fetch_r0 test pins the two register assignments the JIT special-cases.

Puranjay Mohan (2):
  bpf, x86: Support fetching AND/OR/XOR atomics in arena
  selftests/bpf: Test fetching AND/OR/XOR atomics in arena

 arch/x86/net/bpf_jit_comp.c                   | 255 ++++++++++++------
 .../selftests/bpf/prog_tests/arena_atomics.c  |  32 +++
 .../selftests/bpf/progs/arena_atomics.c       | 114 +++++---
 3 files changed, 286 insertions(+), 115 deletions(-)


base-commit: 91f8613d95ad8cd99d8baf094806d1ef98bc6380
-- 
2.53.0-Meta