All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena
@ 2026-10-14 12:50 Puranjay Mohan
  2026-10-14 12:50 ` [PATCH bpf-next v4 1/3] " Puranjay Mohan
                   ` (3 more replies)
  0 siblings, 4 replies; 7+ messages in thread
From: Puranjay Mohan @ 2026-10-14 12:50 UTC (permalink / raw)
  To: bpf
  Cc: Puranjay Mohan, Alexei Starovoitov, Daniel Borkmann,
	Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
	Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Tejun Heo,
	David Vernet, Andrea Righi

Changelog:
v1: https://lore.kernel.org/all/20260923165301.3463007-1-puranjay@kernel.org/
v2: https://lore.kernel.org/all/20260924160354.531101-1-puranjay@kernel.org/
v3: https://lore.kernel.org/all/20260925134828.2012199-2-puranjay@kernel.org/

Changes in v4:
- Patch 1: go back to the v1 lowering, keeping the load of the old value
  and giving it a second exception table entry, per Tejun's review. v2 and
  v3 dropped the load so that the CMPXCHG was the only faulting insn, but
  the first CMPXCHG then almost always loses, so every operation paid for
  an extra locked CMPXCHG and the fetching insn ended up more expensive on
  x86 than the load and CMPXCHG loop it is meant to replace. The extra
  entry is accounted for in bpf_int_jit_compile() again, as in v1. The
  cleanups from v2 stay.
- Patch 1: refuse these ops when dst_reg is BPF_REG_AX. Accesses through a
  number (BPF_F_ARENA_SCALAR, which arena_scalar_access() rewrites to use
  BPF_REG_AX as the address) leave the loop no register to save R0 in, and
  dropping the bpf_jit_supports_insn() rejection would otherwise let them
  through. Such programs fail to load, as they did before. This only became
  reachable after rebasing over BPF_F_ARENA_SCALAR.
- Patch 1: correct the commit message, which claimed to add the
  reg2pt_regs[BPF_REG_AX] entry that BPF_F_ARENA_SCALAR already added, and
  update the extable sizing comment.
- Patch 2: keep each *_noreturn_value in its operation's block and the or
  block's u32s first. v3 grouped them, leaving two 4-byte holes in
  .addr_space.1 that clang-17 fails to build.
- Patch 2: check the register state after the r0 forms in uaf fault: the
  fetched register is cleared and the address register is untouched. Use
  values with high bits set in fetch_r0, and add fetching and/or/xor
  through a number to verifier_arena_scalar (works on arm64, refused on
  x86-64). Correct the comment about which spelling selects the
  non-fetching insn.
- Patch 3: say in the commit message that x86-64 schedulers built from this
  header need a kernel whose JIT accepts these ops, and that already-set
  bits now cost a locked write. Describe the comment that is actually being
  removed.

Changes in v3:
- Patch 2: keep the 32-bit non-fetching and/or/xor as well.
- Patch 2: guard the new uaf asm on __BPF_FEATURE_ADDR_SPACE_CAST, since
  textual addr_space_cast needs LLVM 19.
- Patch 3: use __sync_fetch_and_or/and in cid.bpf.h and delete the cmpxchg
  loop, rather than only rewording the comment that justified it.

Changes in v2:
- Patch 1: drop the initial load in the arena lowering instead of adding a
  second exception table entry (reverted in v4).
- Patch 1: note the dropped bpf_jit_supports_insn() rejection in the
  changelog, rename the shared "mem32 extable bug" message, let the
  non-arena caller pass NULL for the out-params, correct the INSN_LEN
  comment, and drop an unrelated whitespace hunk.
- Patch 2: keep non-fetching coverage, add the R0-aliased forms to uaf,
  and correct the comment about why arm64 stays out of uaf.
- Patch 3: new.

A fetching AND/OR/XOR against arena memory is rejected on x86-64:

  BPF_ATOMIC stores into R1 arena is not allowed

x86-64 has no single instruction for these, so the JIT lowers them to a
CMPXCHG loop. The loop performs two memory accesses, the load of the old
value and the CMPXCHG itself, and either can fault when the arena page
goes away. The verifier reserves one exception table entry per
instruction, so there was nowhere to record the second one and
bpf_jit_supports_insn() refused the three opcodes instead.

x86-64 is the only architecture that needs this. riscv64 has native
AMOAND/AMOOR/AMOXOR with fetch, s390 has LAN/LAO/LAX, and arm64 with LSE
has LDCLRAL/LDSETAL/LDEORAL, so these already work in an
arena, as does powerpc64. arm64 without LSE and loongarch reject every arena
RMW atomic, CMPXCHG included, so nothing can be lowered there either way.

Patch 1 emits the loop with R12-indexed addressing and gives the load and
the CMPXCHG an exception table entry each. Both resume past the whole loop
rather than past the faulting instruction, with the fetch destination
cleared, so a fault cannot re-enter the loop. The JIT accounts for the
extra entry by rescanning the instruction stream in bpf_int_jit_compile()
before the extable is sized; s390 does the same in bpf_jit_alloc() for
its BPF_XCHG lowering. The resulting code is the same load and CMPXCHG
loop as the non-arena lowering, which is what programs hand-roll today,
minus the early exit a hand-rolled loop can take when the value is
already as wanted. Accesses through a number are the exception: they keep
their address in BPF_REG_AX, where the loop would have to save R0, so the
JIT refuses them and such programs fail to load, as before the series.

Patch 2 makes the selftests cover this. The existing arena and/or/xor tests
discarded the returned value, so clang emitted the non-fetching instruction
and the fetching one was never exercised. That was deliberate:
commit 2897b1e2a2f4 ("selftests/bpf: Fix arena_atomics failure due to llvm
change") switched them to __c11_atomic_fetch_*() with memory_order_relaxed
to dodge the limitation patch 1 removes. They now check the old value via
__sync_fetch_and_*(), while keeping a __c11_atomic_fetch_*() of each width
per operation with the result discarded so the non-fetching lowering keeps
its coverage. A new fetch_r0 test pins the two register assignments the
JIT special-cases, and both also go into the uaf test, since the
BPF_REG_AX substitution only reaches the exception table when such an
access faults. x86 is dropped from the uaf exclusion list.

Patch 3 drops the hand-rolled cmpxchg loop in sched_ext's cid.bpf.h, which
only existed because of the x86 restriction patch 1 removes, and uses the
fetching atomics directly. The loop was not a portability fallback either:
arm64 without LSE rejects every arena read-modify-write including CMPXCHG,
and riscv without Zacas rejects only CMPXCHG, so the loop is strictly
worse than a fetching OR there.

Puranjay Mohan (3):
  bpf, x86: Support fetching AND/OR/XOR atomics in arena
  selftests/bpf: Test fetching AND/OR/XOR atomics in arena
  sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop

 arch/x86/net/bpf_jit_comp.c                   | 269 +++++++++++++-----
 tools/sched_ext/include/scx/cid.bpf.h         |  79 +----
 .../selftests/bpf/prog_tests/arena_atomics.c  |  47 +++
 .../selftests/bpf/progs/arena_atomics.c       | 184 +++++++++---
 .../bpf/progs/verifier_arena_scalar.c         |  98 +++++++
 5 files changed, 491 insertions(+), 186 deletions(-)


base-commit: 763187dbbf83764a57d5811168ef9de7b9a60e2d
-- 
2.53.0-Meta


^ permalink raw reply	[flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-10-15 17:06 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-14 12:50 [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 1/3] " Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 2/3] selftests/bpf: Test " Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 3/3] sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop Puranjay Mohan
2026-10-14 13:23   ` bot+bpf-ci
2026-10-14 13:34     ` Puranjay Mohan
2026-10-15 17:06 ` [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena Tejun Heo

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.