* [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena
@ 2026-10-14 12:50 Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 1/3] " Puranjay Mohan
` (3 more replies)
0 siblings, 4 replies; 7+ messages in thread
From: Puranjay Mohan @ 2026-10-14 12:50 UTC (permalink / raw)
To: bpf
Cc: Puranjay Mohan, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Tejun Heo,
David Vernet, Andrea Righi
Changelog:
v1: https://lore.kernel.org/all/20260923165301.3463007-1-puranjay@kernel.org/
v2: https://lore.kernel.org/all/20260924160354.531101-1-puranjay@kernel.org/
v3: https://lore.kernel.org/all/20260925134828.2012199-2-puranjay@kernel.org/
Changes in v4:
- Patch 1: go back to the v1 lowering, keeping the load of the old value
and giving it a second exception table entry, per Tejun's review. v2 and
v3 dropped the load so that the CMPXCHG was the only faulting insn, but
the first CMPXCHG then almost always loses, so every operation paid for
an extra locked CMPXCHG and the fetching insn ended up more expensive on
x86 than the load and CMPXCHG loop it is meant to replace. The extra
entry is accounted for in bpf_int_jit_compile() again, as in v1. The
cleanups from v2 stay.
- Patch 1: refuse these ops when dst_reg is BPF_REG_AX. Accesses through a
number (BPF_F_ARENA_SCALAR, which arena_scalar_access() rewrites to use
BPF_REG_AX as the address) leave the loop no register to save R0 in, and
dropping the bpf_jit_supports_insn() rejection would otherwise let them
through. Such programs fail to load, as they did before. This only became
reachable after rebasing over BPF_F_ARENA_SCALAR.
- Patch 1: correct the commit message, which claimed to add the
reg2pt_regs[BPF_REG_AX] entry that BPF_F_ARENA_SCALAR already added, and
update the extable sizing comment.
- Patch 2: keep each *_noreturn_value in its operation's block and the or
block's u32s first. v3 grouped them, leaving two 4-byte holes in
.addr_space.1 that clang-17 fails to build.
- Patch 2: check the register state after the r0 forms in uaf fault: the
fetched register is cleared and the address register is untouched. Use
values with high bits set in fetch_r0, and add fetching and/or/xor
through a number to verifier_arena_scalar (works on arm64, refused on
x86-64). Correct the comment about which spelling selects the
non-fetching insn.
- Patch 3: say in the commit message that x86-64 schedulers built from this
header need a kernel whose JIT accepts these ops, and that already-set
bits now cost a locked write. Describe the comment that is actually being
removed.
Changes in v3:
- Patch 2: keep the 32-bit non-fetching and/or/xor as well.
- Patch 2: guard the new uaf asm on __BPF_FEATURE_ADDR_SPACE_CAST, since
textual addr_space_cast needs LLVM 19.
- Patch 3: use __sync_fetch_and_or/and in cid.bpf.h and delete the cmpxchg
loop, rather than only rewording the comment that justified it.
Changes in v2:
- Patch 1: drop the initial load in the arena lowering instead of adding a
second exception table entry (reverted in v4).
- Patch 1: note the dropped bpf_jit_supports_insn() rejection in the
changelog, rename the shared "mem32 extable bug" message, let the
non-arena caller pass NULL for the out-params, correct the INSN_LEN
comment, and drop an unrelated whitespace hunk.
- Patch 2: keep non-fetching coverage, add the R0-aliased forms to uaf,
and correct the comment about why arm64 stays out of uaf.
- Patch 3: new.
A fetching AND/OR/XOR against arena memory is rejected on x86-64:
BPF_ATOMIC stores into R1 arena is not allowed
x86-64 has no single instruction for these, so the JIT lowers them to a
CMPXCHG loop. The loop performs two memory accesses, the load of the old
value and the CMPXCHG itself, and either can fault when the arena page
goes away. The verifier reserves one exception table entry per
instruction, so there was nowhere to record the second one and
bpf_jit_supports_insn() refused the three opcodes instead.
x86-64 is the only architecture that needs this. riscv64 has native
AMOAND/AMOOR/AMOXOR with fetch, s390 has LAN/LAO/LAX, and arm64 with LSE
has LDCLRAL/LDSETAL/LDEORAL, so these already work in an
arena, as does powerpc64. arm64 without LSE and loongarch reject every arena
RMW atomic, CMPXCHG included, so nothing can be lowered there either way.
Patch 1 emits the loop with R12-indexed addressing and gives the load and
the CMPXCHG an exception table entry each. Both resume past the whole loop
rather than past the faulting instruction, with the fetch destination
cleared, so a fault cannot re-enter the loop. The JIT accounts for the
extra entry by rescanning the instruction stream in bpf_int_jit_compile()
before the extable is sized; s390 does the same in bpf_jit_alloc() for
its BPF_XCHG lowering. The resulting code is the same load and CMPXCHG
loop as the non-arena lowering, which is what programs hand-roll today,
minus the early exit a hand-rolled loop can take when the value is
already as wanted. Accesses through a number are the exception: they keep
their address in BPF_REG_AX, where the loop would have to save R0, so the
JIT refuses them and such programs fail to load, as before the series.
Patch 2 makes the selftests cover this. The existing arena and/or/xor tests
discarded the returned value, so clang emitted the non-fetching instruction
and the fetching one was never exercised. That was deliberate:
commit 2897b1e2a2f4 ("selftests/bpf: Fix arena_atomics failure due to llvm
change") switched them to __c11_atomic_fetch_*() with memory_order_relaxed
to dodge the limitation patch 1 removes. They now check the old value via
__sync_fetch_and_*(), while keeping a __c11_atomic_fetch_*() of each width
per operation with the result discarded so the non-fetching lowering keeps
its coverage. A new fetch_r0 test pins the two register assignments the
JIT special-cases, and both also go into the uaf test, since the
BPF_REG_AX substitution only reaches the exception table when such an
access faults. x86 is dropped from the uaf exclusion list.
Patch 3 drops the hand-rolled cmpxchg loop in sched_ext's cid.bpf.h, which
only existed because of the x86 restriction patch 1 removes, and uses the
fetching atomics directly. The loop was not a portability fallback either:
arm64 without LSE rejects every arena read-modify-write including CMPXCHG,
and riscv without Zacas rejects only CMPXCHG, so the loop is strictly
worse than a fetching OR there.
Puranjay Mohan (3):
bpf, x86: Support fetching AND/OR/XOR atomics in arena
selftests/bpf: Test fetching AND/OR/XOR atomics in arena
sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop
arch/x86/net/bpf_jit_comp.c | 269 +++++++++++++-----
tools/sched_ext/include/scx/cid.bpf.h | 79 +----
.../selftests/bpf/prog_tests/arena_atomics.c | 47 +++
.../selftests/bpf/progs/arena_atomics.c | 184 +++++++++---
.../bpf/progs/verifier_arena_scalar.c | 98 +++++++
5 files changed, 491 insertions(+), 186 deletions(-)
base-commit: 763187dbbf83764a57d5811168ef9de7b9a60e2d
--
2.53.0-Meta
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH bpf-next v4 1/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena
2026-10-14 12:50 [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena Puranjay Mohan
@ 2026-10-14 12:50 ` Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 2/3] selftests/bpf: Test " Puranjay Mohan
` (2 subsequent siblings)
3 siblings, 0 replies; 7+ messages in thread
From: Puranjay Mohan @ 2026-10-14 12:50 UTC (permalink / raw)
To: bpf
Cc: Puranjay Mohan, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Tejun Heo,
David Vernet, Andrea Righi
x86-64 has no single instruction for a fetching AND/OR/XOR, so the JIT
lowers them to a CMPXCHG loop. That loop could not be used against arena
memory: it contains two memory accesses, the load of the old value and
the CMPXCHG itself, either of which can fault when the arena page goes
away, while the verifier reserves only one exception table entry per
instruction. bpf_jit_supports_insn() therefore rejected the three opcodes
and such programs failed to load with
BPF_ATOMIC stores into R1 arena is not allowed
Emit the same loop with R12-indexed addressing and give each of the two
accesses its own exception table entry. The verifier still reserves one
per instruction, so the JIT accounts for the extra entry itself by
rescanning the instruction stream in bpf_int_jit_compile() before the
extable is sized, as s390 does in bpf_jit_alloc() for its BPF_XCHG
lowering. The rescan sits after the extra-pass early exit, so a program
is counted exactly once. Drop the bpf_jit_supports_insn() rejection.
A loop without the load would need only one entry, since a CMPXCHG that
loses the comparison reloads RAX, but its first CMPXCHG would then almost
always lose. Every operation would pay for a second locked CMPXCHG, which
makes the fetching insn more expensive than the load and CMPXCHG loop
programs write by hand today. The second entry is paid for once, in the
JIT.
Both entries resume past the whole loop rather than past the faulting
instruction alone, with the fetch destination cleared, so a fault cannot
re-enter the loop, and both are reported as writes since the BPF insn is
a read-modify-write. This repurposes the INSN_LEN field of the fixup as a
resume distance rather than an instruction length; a sequence long enough
to overflow its 8 bits would first have to exceed BPF_MAX_INSN_SIZE, which
do_jit() rejects before the image is used.
The loop needs RAX for CMPXCHG and saves R0 in BPF_REG_AX, which it also
substitutes for R0 when either operand is R0. ex_handler_bpf() may then
have to name r10 as the register holding the arena address or the one to
clear on fault; reg2pt_regs[] already has that entry.
Accesses through a number, made by programs loaded with
BPF_F_ARENA_SCALAR, keep their address in BPF_REG_AX themselves (see
arena_scalar_access()), which leaves the loop nowhere to save R0. Refuse
those in the JIT, since bpf_jit_supports_insn() cannot tell them apart
from accesses through an arena pointer. Such programs fail to load, as
they did before.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
arch/x86/net/bpf_jit_comp.c | 269 +++++++++++++++++++++++++-----------
1 file changed, 192 insertions(+), 77 deletions(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 083fcd6cf15b7..5fb1a31f71a26 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -239,6 +239,13 @@ static const int reg2pt_regs[] = {
[BPF_REG_AX] = offsetof(struct pt_regs, r10),
};
+static bool is_atomic_fetch_op(const struct bpf_insn *insn)
+{
+ return insn->imm == (BPF_AND | BPF_FETCH) ||
+ insn->imm == (BPF_OR | BPF_FETCH) ||
+ insn->imm == (BPF_XOR | BPF_FETCH);
+}
+
/*
* is_ereg() == true if BPF register 'reg' maps to x86-64 r8..r15
* which need extra byte of encoding.
@@ -1680,6 +1687,72 @@ static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size,
return 0;
}
+/*
+ * A fetching AND/OR/XOR can't be implemented with a single x86 insn, so do a
+ * CMPXCHG loop. @index_reg is X86_REG_R12 for an arena access or -1 otherwise,
+ * and @dst_reg/@src_reg are already substituted for R0 by the caller.
+ *
+ * Both the load and the CMPXCHG can fault when accessing the arena, so their
+ * addresses are handed back in @fault[0] and @fault[1]. A fault has to resume
+ * at @resume, which is past the loop and past the move that delivers the old
+ * value, but before the R0 restore. Both are NULL for the non-arena case,
+ * which needs no fixup.
+ */
+static int emit_atomic_fetch_rmw(u8 **pprog, struct bpf_insn *insn, u32 dst_reg,
+ u32 src_reg, int index_reg, u8 **fault,
+ u8 **resume)
+{
+ bool is64 = BPF_SIZE(insn->code) == BPF_DW;
+ u8 *branch_target, *prog = *pprog;
+ int err;
+
+ branch_target = prog;
+
+ /* Load old value */
+ if (fault)
+ fault[0] = prog;
+ if (index_reg < 0)
+ emit_ldx(&prog, BPF_SIZE(insn->code), BPF_REG_0, dst_reg, insn->off);
+ else
+ emit_ldx_index(&prog, BPF_SIZE(insn->code), BPF_REG_0, dst_reg,
+ index_reg, insn->off);
+
+ /*
+ * Perform the (commutative) operation locally, put the result in
+ * the AUX_REG.
+ */
+ emit_mov_reg(&prog, is64, AUX_REG, BPF_REG_0);
+ maybe_emit_mod(&prog, AUX_REG, src_reg, is64);
+ EMIT2(simple_alu_opcodes[BPF_OP(insn->imm)],
+ add_2reg(0xC0, AUX_REG, src_reg));
+
+ /* Attempt to swap in new value */
+ if (fault)
+ fault[1] = prog;
+ if (index_reg < 0)
+ err = emit_atomic_rmw(&prog, BPF_CMPXCHG, dst_reg, AUX_REG,
+ insn->off, BPF_SIZE(insn->code));
+ else
+ err = emit_atomic_rmw_index(&prog, BPF_CMPXCHG, BPF_SIZE(insn->code),
+ dst_reg, AUX_REG, index_reg, insn->off);
+ if (WARN_ON(err))
+ return err;
+
+ /* ZF tells us whether we won the race. If it's cleared we need to try again. */
+ EMIT2(X86_JNE, -(prog - branch_target) - 2);
+ /* Return the pre-modification value */
+ emit_mov_reg(&prog, is64, src_reg, BPF_REG_0);
+
+ if (resume)
+ *resume = prog;
+
+ /* Restore R0 after clobbering RAX */
+ emit_mov_reg(&prog, true, BPF_REG_0, BPF_REG_AX);
+
+ *pprog = prog;
+ return 0;
+}
+
/*
* Metadata encoding for exception handling in JITed code.
*
@@ -1693,7 +1766,12 @@ static int emit_atomic_ld_st_index(u8 **pprog, u32 atomic_op, u32 size,
* | ARENA_ACC | ARENA_WRITE | Unused | ARENA_REG | DST_REG | INSN_LEN |
* +-----------+-------------+--------+-----------+---------+----------+
*
- * - INSN_LEN (8 bits): Length of faulting insn (max x86 insn = 15 bytes (fits in 8 bits)).
+ * - INSN_LEN (8 bits): How far past the faulting insn to resume. That is its own length
+ * for a single-insn access, but the distance to the end of the whole
+ * sequence where one BPF insn became several, as for the CMPXCHG loop
+ * of a fetching AND/OR/XOR. A sequence long enough to overflow this
+ * field would first have to exceed BPF_MAX_INSN_SIZE, which do_jit()
+ * rejects with -EFAULT before the image is used.
* - DST_REG (8 bits): Offset of dst_reg from reg2pt_regs[] (max offset = 112 (fits in 8 bits)).
* This is set to DONT_CLEAR if the insn does not read into a register.
* - ARENA_REG (8 bits): Offset of the register that is used to calculate the
@@ -1748,6 +1826,44 @@ bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs)
return true;
}
+/*
+ * Record an arena access that may fault. @fault_ip is the address of the
+ * faulting insn in the RO image, @resume_off how far past it execution has to
+ * resume: for a multi-insn lowering that is the end of the whole sequence, not
+ * the end of the one insn.
+ */
+static int emit_arena_exentry(struct bpf_prog *bpf_prog, u8 *image, u8 *rw_image,
+ int *excnt, u8 *fault_ip, u32 resume_off,
+ u32 fixup_reg, u32 arena_reg, bool is_write, s16 off)
+{
+ struct exception_table_entry *ex;
+ s64 delta;
+
+ if (!bpf_prog->aux->extable)
+ return 0;
+
+ if (*excnt >= bpf_prog->aux->num_exentries) {
+ pr_err("arena extable bug\n");
+ return -EFAULT;
+ }
+ ex = &bpf_prog->aux->extable[(*excnt)++];
+
+ delta = fault_ip - (u8 *)&ex->insn;
+ /* switch ex to rw buffer for writes */
+ ex = (void *)rw_image + ((void *)ex - (void *)image);
+
+ ex->insn = delta;
+ ex->data = EX_TYPE_BPF | FIELD_PREP(DATA_ARENA_OFFSET_MASK, off);
+ ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, resume_off) |
+ FIELD_PREP(FIXUP_ARENA_REG_MASK, arena_reg) |
+ FIELD_PREP(FIXUP_REG_MASK, fixup_reg) |
+ FIXUP_ARENA_ACCESS;
+ if (is_write)
+ ex->fixup |= FIXUP_ARENA_WRITE;
+
+ return 0;
+}
+
static void detect_reg_usage(struct bpf_insn *insn, int insn_cnt,
bool *regs_used)
{
@@ -2625,28 +2741,8 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
}
populate_extable:
{
- struct exception_table_entry *ex;
- u8 *_insn = image + proglen + (start_of_ldx - temp);
u32 arena_reg, fixup_reg;
bool is_write;
- s64 delta;
-
- if (!bpf_prog->aux->extable)
- break;
-
- if (excnt >= bpf_prog->aux->num_exentries) {
- pr_err("mem32 extable bug\n");
- return -EFAULT;
- }
- ex = &bpf_prog->aux->extable[excnt++];
-
- delta = _insn - (u8 *)&ex->insn;
- /* switch ex to rw buffer for writes */
- ex = (void *)rw_image + ((void *)ex - (void *)image);
-
- ex->insn = delta;
-
- ex->data = EX_TYPE_BPF;
/*
* src_reg/dst_reg holds the address in the arena region with upper
@@ -2682,14 +2778,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
is_write = true;
}
- ex->fixup = FIELD_PREP(FIXUP_INSN_LEN_MASK, prog - start_of_ldx) |
- FIELD_PREP(FIXUP_ARENA_REG_MASK, arena_reg) |
- FIELD_PREP(FIXUP_REG_MASK, fixup_reg);
- ex->fixup |= FIXUP_ARENA_ACCESS;
- if (is_write)
- ex->fixup |= FIXUP_ARENA_WRITE;
-
- ex->data |= FIELD_PREP(DATA_ARENA_OFFSET_MASK, insn->off);
+ err = emit_arena_exentry(bpf_prog, image, rw_image, &excnt,
+ image + proglen + (start_of_ldx - temp),
+ prog - start_of_ldx, fixup_reg,
+ arena_reg, is_write, insn->off);
+ if (err)
+ return err;
}
break;
@@ -2841,20 +2935,12 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
fallthrough;
case BPF_STX | BPF_ATOMIC | BPF_W:
case BPF_STX | BPF_ATOMIC | BPF_DW: {
- bool is64 = BPF_SIZE(insn->code) == BPF_DW;
u32 real_src_reg = src_reg;
u32 real_dst_reg = dst_reg;
+ bool is_atomic_fetch = is_atomic_fetch_op(insn);
u8 *old_prog;
- bool is_atomic_fetch =
- (insn->imm == (BPF_AND | BPF_FETCH) ||
- insn->imm == (BPF_OR | BPF_FETCH) ||
- insn->imm == (BPF_XOR | BPF_FETCH));
- if (is_atomic_fetch) {
- /*
- * Can't be implemented with a single x86 insn.
- * Need to do a CMPXCHG loop.
- */
+ if (is_atomic_fetch) {
/* Will need RAX as a CMPXCHG operand so save R0 */
old_prog = prog;
emit_mov_reg(&prog, true, BPF_REG_AX, BPF_REG_0);
@@ -2875,34 +2961,11 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
}
}
if (is_atomic_fetch) {
- u8 *branch_target = prog;
- /* Load old value */
- emit_ldx(&prog, BPF_SIZE(insn->code),
- BPF_REG_0, real_dst_reg, insn->off);
- /*
- * Perform the (commutative) operation locally,
- * put the result in the AUX_REG.
- */
- emit_mov_reg(&prog, is64, AUX_REG, BPF_REG_0);
- maybe_emit_mod(&prog, AUX_REG, real_src_reg, is64);
- EMIT2(simple_alu_opcodes[BPF_OP(insn->imm)],
- add_2reg(0xC0, AUX_REG, real_src_reg));
- /* Attempt to swap in new value */
- err = emit_atomic_rmw(&prog, BPF_CMPXCHG,
- real_dst_reg, AUX_REG,
- insn->off,
- BPF_SIZE(insn->code));
- if (WARN_ON(err))
+ err = emit_atomic_fetch_rmw(&prog, insn, real_dst_reg,
+ real_src_reg, -1, NULL,
+ NULL);
+ if (err)
return err;
- /*
- * ZF tells us whether we won the race. If it's
- * cleared we need to try again.
- */
- EMIT2(X86_JNE, -(prog - branch_target) - 2);
- /* Return the pre-modification value */
- emit_mov_reg(&prog, is64, real_src_reg, BPF_REG_0);
- /* Restore R0 after clobbering RAX */
- emit_mov_reg(&prog, true, BPF_REG_0, BPF_REG_AX);
break;
}
@@ -2926,6 +2989,51 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *
fallthrough;
case BPF_STX | BPF_PROBE_ATOMIC | BPF_W:
case BPF_STX | BPF_PROBE_ATOMIC | BPF_DW:
+ if (is_atomic_fetch_op(insn)) {
+ u32 real_src_reg = src_reg, real_dst_reg = dst_reg;
+ u8 *fault[2], *resume;
+ int j;
+
+ /*
+ * An access through a number has its address in
+ * BPF_REG_AX (see arena_scalar_access()), where the
+ * loop below has to save R0.
+ */
+ if (dst_reg == BPF_REG_AX)
+ return -EOPNOTSUPP;
+
+ /* Will need RAX as a CMPXCHG operand so save R0 */
+ emit_mov_reg(&prog, true, BPF_REG_AX, BPF_REG_0);
+ if (src_reg == BPF_REG_0)
+ real_src_reg = BPF_REG_AX;
+ if (dst_reg == BPF_REG_0)
+ real_dst_reg = BPF_REG_AX;
+
+ err = emit_atomic_fetch_rmw(&prog, insn, real_dst_reg,
+ real_src_reg, X86_REG_R12,
+ fault, &resume);
+ if (err)
+ return err;
+
+ /*
+ * The page may be unmapped between the load and the
+ * CMPXCHG, so both need an entry. Both report a write,
+ * since the BPF insn is a read-modify-write.
+ */
+ for (j = 0; j < 2; j++) {
+ err = emit_arena_exentry(bpf_prog, image, rw_image,
+ &excnt,
+ image + proglen +
+ (fault[j] - temp),
+ resume - fault[j],
+ reg2pt_regs[real_src_reg],
+ reg2pt_regs[real_dst_reg],
+ true, insn->off);
+ if (err)
+ return err;
+ }
+ break;
+ }
start_of_ldx = prog;
if (bpf_atomic_is_load_store(insn))
@@ -4353,6 +4461,21 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
padding = true;
goto skip_init_addrs;
}
+
+ /*
+ * An arena fetching AND/OR/XOR is lowered to a CMPXCHG loop whose load
+ * and CMPXCHG can both fault, one more entry than the verifier reserved.
+ * Only reachable on the first pass, so the count is adjusted once.
+ */
+ for (i = 0; prog->aux->arena && i < prog->len; i++) {
+ const struct bpf_insn *insn = &prog->insnsi[i];
+
+ if (BPF_CLASS(insn->code) == BPF_STX &&
+ BPF_MODE(insn->code) == BPF_PROBE_ATOMIC &&
+ is_atomic_fetch_op(insn))
+ prog->aux->num_exentries++;
+ }
+
addrs = kvmalloc_objs(*addrs, prog->len + 1);
if (!addrs)
goto out_addrs;
@@ -4407,7 +4530,9 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
* The number of entries in extable is the number of BPF_LDX
* insns that access kernel memory via "pointer to BTF type".
* The verifier changed their opcode from LDX|MEM|size
- * to LDX|PROBE_MEM|size to make JITing easier.
+ * to LDX|PROBE_MEM|size to make JITing easier. Arena
+ * accesses add one each, and an arena fetching AND/OR/XOR
+ * one more, see the adjustment of num_exentries above.
*/
u32 align = __alignof__(struct exception_table_entry);
u32 extable_size = prog->aux->num_exentries *
@@ -4681,16 +4806,6 @@ bool bpf_jit_supports_arena_scalar(void)
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena)
{
- if (!in_arena)
- return true;
- switch (insn->code) {
- case BPF_STX | BPF_ATOMIC | BPF_W:
- case BPF_STX | BPF_ATOMIC | BPF_DW:
- if (insn->imm == (BPF_AND | BPF_FETCH) ||
- insn->imm == (BPF_OR | BPF_FETCH) ||
- insn->imm == (BPF_XOR | BPF_FETCH))
- return false;
- }
return true;
}
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH bpf-next v4 2/3] selftests/bpf: Test fetching AND/OR/XOR atomics in arena
2026-10-14 12:50 [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 1/3] " Puranjay Mohan
@ 2026-10-14 12:50 ` Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 3/3] sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop Puranjay Mohan
2026-10-15 17:06 ` [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena Tejun Heo
3 siblings, 0 replies; 7+ messages in thread
From: Puranjay Mohan @ 2026-10-14 12:50 UTC (permalink / raw)
To: bpf
Cc: Puranjay Mohan, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Tejun Heo,
David Vernet, Andrea Righi
The arena and/or/xor tests never exercised the fetching form of the
instruction. Commit 2897b1e2a2f4 ("selftests/bpf: Fix arena_atomics
failure due to llvm change") deliberately switched them to
__c11_atomic_fetch_*() with memory_order_relaxed, which clang lowers to a
non-fetching locked instruction when the result is unused, because
x86-64 could not JIT the fetching one against an arena.
It can now, so check the returned old value with __sync_fetch_and_*().
Keep a __c11_atomic_fetch_*() of each width per operation with the result
discarded: that spelling is what selects the non-fetching insn, which
would otherwise lose its only coverage, on every architecture rather than
just x86. Compiling the prog before and after confirms the split: 6 non-
fetching and no fetching arena and/or/xor before, 6 and 22 after.
The new globals go in their operation's block, and the or block keeps its
u32s first, so .addr_space.1 stays free of the padding holes that clang
before 18 cannot fill when it is built without address space casts.
Add a fetch_r0 test for the two register assignments the x86 JIT has to
special-case, where the operand and where the arena pointer is R0 and
BPF_REG_AX is substituted for it. Clang picks its own registers and will
not reliably produce either, so spell the instructions out. The values
have bits set above 32 so that a truncating move on that path shows up.
Add both forms to the uaf test as well, since the substitution only
reaches the exception table when such an access faults, and check that
the fault cleared the fetched register and left the address alone.
Both spell addr_space_cast textually, which needs LLVM 19, so uaf gains
the __BPF_FEATURE_ADDR_SPACE_CAST guard that every other textual use in
the file already has. It was plain C before and built on older clang.
Drop x86 from the exclusion list of the uaf test, which covers the fault
path. Its page is gone before the first access, so it faults on the load;
the CMPXCHG entry is only reached when the page goes away between the two
accesses, which a test cannot arrange.
Finally cover fetching AND/OR/XOR through a number in
verifier_arena_scalar: they work on arm64, and x86-64 refuses them.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
.../selftests/bpf/prog_tests/arena_atomics.c | 47 +++++
.../selftests/bpf/progs/arena_atomics.c | 184 ++++++++++++++----
.../bpf/progs/verifier_arena_scalar.c | 98 ++++++++++
3 files changed, 291 insertions(+), 38 deletions(-)
diff --git a/tools/testing/selftests/bpf/prog_tests/arena_atomics.c b/tools/testing/selftests/bpf/prog_tests/arena_atomics.c
index 1ad5d03d07adb..81f6e554a8c12 100644
--- a/tools/testing/selftests/bpf/prog_tests/arena_atomics.c
+++ b/tools/testing/selftests/bpf/prog_tests/arena_atomics.c
@@ -68,6 +68,12 @@ static void test_and(struct arena_atomics *skel)
ASSERT_EQ(skel->arena->and64_value, 0x010ull << 32, "and64_value");
ASSERT_EQ(skel->arena->and32_value, 0x010, "and32_value");
+
+ ASSERT_EQ(skel->arena->and64_result, 0x110ull << 32, "and64_result");
+ ASSERT_EQ(skel->arena->and32_result, 0x110, "and32_result");
+
+ ASSERT_EQ(skel->arena->and64_noreturn_value, 0x010ull << 32, "and64_noreturn_value");
+ ASSERT_EQ(skel->arena->and32_noreturn_value, 0x010, "and32_noreturn_value");
}
static void test_or(struct arena_atomics *skel)
@@ -85,6 +91,12 @@ static void test_or(struct arena_atomics *skel)
ASSERT_EQ(skel->arena->or64_value, 0x111ull << 32, "or64_value");
ASSERT_EQ(skel->arena->or32_value, 0x111, "or32_value");
+
+ ASSERT_EQ(skel->arena->or64_result, 0x110ull << 32, "or64_result");
+ ASSERT_EQ(skel->arena->or32_result, 0x110, "or32_result");
+
+ ASSERT_EQ(skel->arena->or64_noreturn_value, 0x111ull << 32, "or64_noreturn_value");
+ ASSERT_EQ(skel->arena->or32_noreturn_value, 0x111, "or32_noreturn_value");
}
static void test_xor(struct arena_atomics *skel)
@@ -102,6 +114,12 @@ static void test_xor(struct arena_atomics *skel)
ASSERT_EQ(skel->arena->xor64_value, 0x101ull << 32, "xor64_value");
ASSERT_EQ(skel->arena->xor32_value, 0x101, "xor32_value");
+
+ ASSERT_EQ(skel->arena->xor64_result, 0x110ull << 32, "xor64_result");
+ ASSERT_EQ(skel->arena->xor32_result, 0x110, "xor32_result");
+
+ ASSERT_EQ(skel->arena->xor64_noreturn_value, 0x101ull << 32, "xor64_noreturn_value");
+ ASSERT_EQ(skel->arena->xor32_noreturn_value, 0x101, "xor32_noreturn_value");
}
static void test_cmpxchg(struct arena_atomics *skel)
@@ -146,6 +164,27 @@ static void test_xchg(struct arena_atomics *skel)
ASSERT_EQ(skel->arena->xchg32_result, 1, "xchg32_result");
}
+static void test_fetch_r0(struct arena_atomics *skel)
+{
+ LIBBPF_OPTS(bpf_test_run_opts, topts);
+ int err, prog_fd;
+
+ /* No need to attach it, just run it directly */
+ prog_fd = bpf_program__fd(skel->progs.fetch_r0);
+ err = bpf_prog_test_run_opts(prog_fd, &topts);
+ if (!ASSERT_OK(err, "test_run_opts err"))
+ return;
+ if (!ASSERT_OK(topts.retval, "test_run_opts retval"))
+ return;
+
+ ASSERT_EQ(skel->arena->fetch_src_r0_value, 0x11100000111ull, "fetch_src_r0_value");
+ ASSERT_EQ(skel->arena->fetch_src_r0_result, 0x11000000110ull, "fetch_src_r0_result");
+
+ ASSERT_EQ(skel->arena->fetch_dst_r0_value, 0x11100000111ull, "fetch_dst_r0_value");
+ ASSERT_EQ(skel->arena->fetch_dst_r0_result, 0x11000000110ull, "fetch_dst_r0_result");
+ ASSERT_EQ(skel->arena->fetch_dst_r0_readback, 0x11100000111ull, "fetch_dst_r0_readback");
+}
+
static void test_uaf(struct arena_atomics *skel)
{
LIBBPF_OPTS(bpf_test_run_opts, topts);
@@ -160,6 +199,12 @@ static void test_uaf(struct arena_atomics *skel)
return;
ASSERT_EQ(skel->arena->uaf_recovery_fails, 0, "uaf_recovery_fails");
+
+ /* A fault clears the fetched register and leaves the address alone */
+ ASSERT_EQ(skel->arena->uaf_r1_after_dst_r0, 0, "uaf_r1_after_dst_r0");
+ ASSERT_EQ(skel->arena->uaf_r0_after_dst_r0, skel->arena->uaf_r0_before_dst_r0,
+ "uaf_r0_after_dst_r0");
+ ASSERT_EQ(skel->arena->uaf_r0_after_src_r0, 0, "uaf_r0_after_src_r0");
}
static void test_load_acquire(struct arena_atomics *skel)
@@ -256,6 +301,8 @@ void serial_test_arena_atomics(void)
test_cmpxchg(skel);
if (test__start_subtest("xchg"))
test_xchg(skel);
+ if (test__start_subtest("fetch_r0"))
+ test_fetch_r0(skel);
if (test__start_subtest("uaf"))
test_uaf(skel);
if (test__start_subtest("load_acquire"))
diff --git a/tools/testing/selftests/bpf/progs/arena_atomics.c b/tools/testing/selftests/bpf/progs/arena_atomics.c
index 73bc2b835f3fe..e89b468fc6226 100644
--- a/tools/testing/selftests/bpf/progs/arena_atomics.c
+++ b/tools/testing/selftests/bpf/progs/arena_atomics.c
@@ -91,13 +91,17 @@ int sub(const void *ctx)
return 0;
}
-#ifdef __BPF_FEATURE_ATOMIC_MEM_ORDERING
-_Atomic __u64 __arena_global and64_value = (0x110ull << 32);
-_Atomic __u32 __arena_global and32_value = 0x110;
-#else
+/*
+ * A relaxed __c11_atomic_fetch_*() whose result is discarded is what selects
+ * the non-fetching insn, so each operation keeps a *_noreturn_value of each
+ * width to cover that lowering as well.
+ */
__u64 __arena_global and64_value = (0x110ull << 32);
+__u64 __arena_global and64_result = 0;
+_Atomic __u64 __arena_global and64_noreturn_value = (0x110ull << 32);
__u32 __arena_global and32_value = 0x110;
-#endif
+__u32 __arena_global and32_result = 0;
+_Atomic __u32 __arena_global and32_noreturn_value = 0x110;
SEC("raw_tp/sys_enter")
int and(const void *ctx)
@@ -105,25 +109,22 @@ int and(const void *ctx)
if (pid != (bpf_get_current_pid_tgid() >> 32))
return 0;
#ifdef ENABLE_ATOMICS_TESTS
-#ifdef __BPF_FEATURE_ATOMIC_MEM_ORDERING
- __c11_atomic_fetch_and(&and64_value, 0x011ull << 32, memory_order_relaxed);
- __c11_atomic_fetch_and(&and32_value, 0x011, memory_order_relaxed);
-#else
- __sync_fetch_and_and(&and64_value, 0x011ull << 32);
- __sync_fetch_and_and(&and32_value, 0x011);
-#endif
+ and64_result = __sync_fetch_and_and(&and64_value, 0x011ull << 32);
+ and32_result = __sync_fetch_and_and(&and32_value, 0x011);
+ __c11_atomic_fetch_and(&and64_noreturn_value, 0x011ull << 32, memory_order_relaxed);
+ __c11_atomic_fetch_and(&and32_noreturn_value, 0x011, memory_order_relaxed);
#endif
return 0;
}
-#ifdef __BPF_FEATURE_ATOMIC_MEM_ORDERING
-_Atomic __u32 __arena_global or32_value = 0x110;
-_Atomic __u64 __arena_global or64_value = (0x110ull << 32);
-#else
+/* u32s first: .addr_space.1 must not have holes, see the clang-17 note below */
__u32 __arena_global or32_value = 0x110;
+__u32 __arena_global or32_result = 0;
+_Atomic __u32 __arena_global or32_noreturn_value = 0x110;
__u64 __arena_global or64_value = (0x110ull << 32);
-#endif
+__u64 __arena_global or64_result = 0;
+_Atomic __u64 __arena_global or64_noreturn_value = (0x110ull << 32);
SEC("raw_tp/sys_enter")
int or(const void *ctx)
@@ -131,25 +132,21 @@ int or(const void *ctx)
if (pid != (bpf_get_current_pid_tgid() >> 32))
return 0;
#ifdef ENABLE_ATOMICS_TESTS
-#ifdef __BPF_FEATURE_ATOMIC_MEM_ORDERING
- __c11_atomic_fetch_or(&or64_value, 0x011ull << 32, memory_order_relaxed);
- __c11_atomic_fetch_or(&or32_value, 0x011, memory_order_relaxed);
-#else
- __sync_fetch_and_or(&or64_value, 0x011ull << 32);
- __sync_fetch_and_or(&or32_value, 0x011);
-#endif
+ or64_result = __sync_fetch_and_or(&or64_value, 0x011ull << 32);
+ or32_result = __sync_fetch_and_or(&or32_value, 0x011);
+ __c11_atomic_fetch_or(&or64_noreturn_value, 0x011ull << 32, memory_order_relaxed);
+ __c11_atomic_fetch_or(&or32_noreturn_value, 0x011, memory_order_relaxed);
#endif
return 0;
}
-#ifdef __BPF_FEATURE_ATOMIC_MEM_ORDERING
-_Atomic __u64 __arena_global xor64_value = (0x110ull << 32);
-_Atomic __u32 __arena_global xor32_value = 0x110;
-#else
__u64 __arena_global xor64_value = (0x110ull << 32);
+__u64 __arena_global xor64_result = 0;
+_Atomic __u64 __arena_global xor64_noreturn_value = (0x110ull << 32);
__u32 __arena_global xor32_value = 0x110;
-#endif
+__u32 __arena_global xor32_result = 0;
+_Atomic __u32 __arena_global xor32_noreturn_value = 0x110;
SEC("raw_tp/sys_enter")
int xor(const void *ctx)
@@ -157,13 +154,10 @@ int xor(const void *ctx)
if (pid != (bpf_get_current_pid_tgid() >> 32))
return 0;
#ifdef ENABLE_ATOMICS_TESTS
-#ifdef __BPF_FEATURE_ATOMIC_MEM_ORDERING
- __c11_atomic_fetch_xor(&xor64_value, 0x011ull << 32, memory_order_relaxed);
- __c11_atomic_fetch_xor(&xor32_value, 0x011, memory_order_relaxed);
-#else
- __sync_fetch_and_xor(&xor64_value, 0x011ull << 32);
- __sync_fetch_and_xor(&xor32_value, 0x011);
-#endif
+ xor64_result = __sync_fetch_and_xor(&xor64_value, 0x011ull << 32);
+ xor32_result = __sync_fetch_and_xor(&xor32_value, 0x011);
+ __c11_atomic_fetch_xor(&xor64_noreturn_value, 0x011ull << 32, memory_order_relaxed);
+ __c11_atomic_fetch_xor(&xor32_noreturn_value, 0x011, memory_order_relaxed);
#endif
return 0;
@@ -213,23 +207,93 @@ int xchg(const void *ctx)
return 0;
}
+__u64 __arena_global fetch_src_r0_value = 0x11000000110ull;
+__u64 __arena_global fetch_src_r0_result = 0;
+__u64 __arena_global fetch_dst_r0_value = 0x11000000110ull;
+__u64 __arena_global fetch_dst_r0_result = 0;
+__u64 __arena_global fetch_dst_r0_readback = 0;
+
+/*
+ * A fetching OR with the operand in r0, and one with the arena pointer in r0.
+ * The x86 JIT needs RAX for its CMPXCHG loop and substitutes BPF_REG_AX for
+ * whichever of the two is r0, so both have to keep working. Hand-written
+ * because clang picks its own registers and will not reliably emit either.
+ */
+SEC("raw_tp/sys_enter")
+int fetch_r0(const void *ctx)
+{
+ if (pid != (bpf_get_current_pid_tgid() >> 32))
+ return 0;
+#if defined(ENABLE_ATOMICS_TESTS) && defined(__BPF_FEATURE_ADDR_SPACE_CAST)
+ asm volatile (
+ "r1 = %[fetch_src_r0_value] ll;"
+ "r1 = addr_space_cast(r1, 0x0, 0x1);"
+ "r0 = 0x1100000011 ll;"
+ ".8byte %[fetch_src_r0_insn];"
+ "r2 = %[fetch_src_r0_result] ll;"
+ "r2 = addr_space_cast(r2, 0x0, 0x1);"
+ "*(u64 *)(r2 + 0) = r0;"
+ :
+ : __imm_addr(fetch_src_r0_value),
+ __imm_insn(fetch_src_r0_insn,
+ BPF_ATOMIC_OP(BPF_DW, BPF_OR | BPF_FETCH, BPF_REG_1, BPF_REG_0, 0)),
+ __imm_addr(fetch_src_r0_result)
+ : __clobber_all);
+
+ asm volatile (
+ "r0 = %[fetch_dst_r0_value] ll;"
+ "r0 = addr_space_cast(r0, 0x0, 0x1);"
+ "r1 = 0x1100000011 ll;"
+ ".8byte %[fetch_dst_r0_insn];"
+ "r2 = %[fetch_dst_r0_result] ll;"
+ "r2 = addr_space_cast(r2, 0x0, 0x1);"
+ "*(u64 *)(r2 + 0) = r1;"
+ /* r0 is only read by the atomic, so it must still be the pointer. */
+ "r3 = *(u64 *)(r0 + 0);"
+ "r2 = %[fetch_dst_r0_readback] ll;"
+ "r2 = addr_space_cast(r2, 0x0, 0x1);"
+ "*(u64 *)(r2 + 0) = r3;"
+ :
+ : __imm_addr(fetch_dst_r0_value),
+ __imm_insn(fetch_dst_r0_insn,
+ BPF_ATOMIC_OP(BPF_DW, BPF_OR | BPF_FETCH, BPF_REG_0, BPF_REG_1, 0)),
+ __imm_addr(fetch_dst_r0_result),
+ __imm_addr(fetch_dst_r0_readback)
+ : __clobber_all);
+#endif
+
+ return 0;
+}
+
__u64 __arena_global uaf_sink;
volatile __u64 __arena_global uaf_recovery_fails;
+/* Register state after the r0 forms below fault, checked by the test */
+__u64 __arena_global uaf_r1_after_dst_r0;
+__u64 __arena_global uaf_r0_before_dst_r0;
+__u64 __arena_global uaf_r0_after_dst_r0;
+__u64 __arena_global uaf_r0_after_src_r0;
SEC("syscall")
int uaf(const void *ctx)
{
if (pid != (bpf_get_current_pid_tgid() >> 32))
return 0;
+/*
+ * __BPF_FEATURE_ADDR_SPACE_CAST is required by the asm below, which spells
+ * addr_space_cast textually. That and the mnemonic both arrived in LLVM 19.
+ */
#if defined(ENABLE_ATOMICS_TESTS) && !defined(__TARGET_ARCH_arm64) && \
- !defined(__TARGET_ARCH_x86)
+ defined(__BPF_FEATURE_ADDR_SPACE_CAST)
__u32 __arena *page32;
__u64 __arena *page64;
void __arena *page;
page = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
bpf_arena_free_pages(&arena, page, 1);
- uaf_recovery_fails = 24;
+ uaf_recovery_fails = 26;
+ uaf_r1_after_dst_r0 = 1;
+ uaf_r0_after_dst_r0 = 1;
+ uaf_r0_after_src_r0 = 1;
page32 = (__u32 __arena *)page;
uaf_sink += __sync_fetch_and_add(page32, 1);
@@ -282,6 +346,50 @@ int uaf(const void *ctx)
uaf_recovery_fails -= 1;
uaf_sink += __sync_lock_test_and_set(page64, 1);
uaf_recovery_fails -= 1;
+
+ /*
+ * The x86 JIT needs RAX for the CMPXCHG loop it lowers a fetching
+ * AND/OR/XOR into, and substitutes BPF_REG_AX for r0 when r0 is either
+ * operand. Only then does the fault fixup have to name that register,
+ * so spell both forms out: clang picks its own and never produces them.
+ */
+ asm volatile (
+ "r0 = %[page];"
+ "r0 = addr_space_cast(r0, 0x0, 0x1);"
+ "r1 = 1;"
+ "r2 = %[uaf_r0_before_dst_r0] ll;"
+ "r2 = addr_space_cast(r2, 0x0, 0x1);"
+ "*(u64 *)(r2 + 0) = r0;"
+ ".8byte %[fetch_dst_r0];"
+ "r2 = %[uaf_r0_after_dst_r0] ll;"
+ "r2 = addr_space_cast(r2, 0x0, 0x1);"
+ "*(u64 *)(r2 + 0) = r0;"
+ "r2 = %[uaf_r1_after_dst_r0] ll;"
+ "r2 = addr_space_cast(r2, 0x0, 0x1);"
+ "*(u64 *)(r2 + 0) = r1;"
+ : : [page]"r"(page64),
+ __imm_insn(fetch_dst_r0,
+ BPF_ATOMIC_OP(BPF_DW, BPF_OR | BPF_FETCH, BPF_REG_0, BPF_REG_1, 0)),
+ __imm_addr(uaf_r0_before_dst_r0),
+ __imm_addr(uaf_r0_after_dst_r0),
+ __imm_addr(uaf_r1_after_dst_r0)
+ : "r0", "r1", "r2", "memory");
+ uaf_recovery_fails -= 1;
+
+ asm volatile (
+ "r1 = %[page];"
+ "r1 = addr_space_cast(r1, 0x0, 0x1);"
+ "r0 = 1;"
+ ".8byte %[fetch_src_r0];"
+ "r2 = %[uaf_r0_after_src_r0] ll;"
+ "r2 = addr_space_cast(r2, 0x0, 0x1);"
+ "*(u64 *)(r2 + 0) = r0;"
+ : : [page]"r"(page64),
+ __imm_insn(fetch_src_r0,
+ BPF_ATOMIC_OP(BPF_DW, BPF_OR | BPF_FETCH, BPF_REG_1, BPF_REG_0, 0)),
+ __imm_addr(uaf_r0_after_src_r0)
+ : "r0", "r1", "r2", "memory");
+ uaf_recovery_fails -= 1;
#endif
return 0;
diff --git a/tools/testing/selftests/bpf/progs/verifier_arena_scalar.c b/tools/testing/selftests/bpf/progs/verifier_arena_scalar.c
index bebc37f501b7e..7a7e06d9596ce 100644
--- a/tools/testing/selftests/bpf/progs/verifier_arena_scalar.c
+++ b/tools/testing/selftests/bpf/progs/verifier_arena_scalar.c
@@ -256,6 +256,104 @@ __naked void atomics(void)
: __clobber_all);
}
+SEC("syscall")
+__flag(BPF_F_ARENA_SCALAR) __arch_arm64
+__description("arena_scalar: fetching and/or/xor")
+__success __retval(0)
+__load_if_JITed()
+__naked void fetch_and_or_xor(void)
+{
+ asm volatile (" \
+ r1 = %[arena] ll; \
+ r2 = 0; \
+ r3 = 1; \
+ r4 = -1; \
+ r5 = 0; \
+ call %[bpf_arena_alloc_pages]; \
+ r6 = r0; \
+ r0 = 100; \
+ if r6 == 0 goto 9f; \
+ *(u64 *)(r6 + 8) = 0x110; \
+ r1 = 0x011; \
+ .8byte %[fetch_or_insn]; /* r1 = atomic_fetch_or((u64 *)(r6 + 8), r1) */ \
+ r0 = 1; \
+ if r1 != 0x110 goto 9f; \
+ r1 = 0x101; \
+ .8byte %[fetch_and_insn]; /* r1 = atomic_fetch_and((u64 *)(r6 + 8), r1) */ \
+ r0 = 2; \
+ if r1 != 0x111 goto 9f; \
+ r1 = 0x011; \
+ .8byte %[fetch_xor_insn]; /* r1 = atomic_fetch_xor((u64 *)(r6 + 8), r1) */ \
+ r0 = 3; \
+ if r1 != 0x101 goto 9f; \
+ r2 = *(u64 *)(r6 + 8); \
+ r0 = 4; \
+ if r2 != 0x110 goto 9f; \
+ r0 = 0; \
+9: exit; \
+" :
+ : __imm_addr(arena),
+ __imm(bpf_arena_alloc_pages),
+ __imm_insn(fetch_or_insn, BPF_ATOMIC_OP(BPF_DW, BPF_OR | BPF_FETCH,
+ BPF_REG_6, BPF_REG_1, 8)),
+ __imm_insn(fetch_and_insn, BPF_ATOMIC_OP(BPF_DW, BPF_AND | BPF_FETCH,
+ BPF_REG_6, BPF_REG_1, 8)),
+ __imm_insn(fetch_xor_insn, BPF_ATOMIC_OP(BPF_DW, BPF_XOR | BPF_FETCH,
+ BPF_REG_6, BPF_REG_1, 8))
+ : __clobber_all);
+}
+
+/*
+ * The x86-64 JIT saves R0 in BPF_REG_AX to lower these, which is where an
+ * access through a number keeps its address, so it refuses them.
+ */
+SEC("syscall")
+__flag(BPF_F_ARENA_SCALAR) __arch_x86_64
+__description("arena_scalar: fetching and/or/xor refused on x86-64")
+__failure
+__load_if_JITed()
+__naked void fetch_and_or_xor_x86(void)
+{
+ asm volatile (" \
+ r1 = %[arena] ll; \
+ r2 = 0; \
+ r3 = 1; \
+ r4 = -1; \
+ r5 = 0; \
+ call %[bpf_arena_alloc_pages]; \
+ r6 = r0; \
+ r0 = 100; \
+ if r6 == 0 goto 9f; \
+ *(u64 *)(r6 + 8) = 0x110; \
+ r1 = 0x011; \
+ .8byte %[fetch_or_insn]; /* r1 = atomic_fetch_or((u64 *)(r6 + 8), r1) */ \
+ r0 = 1; \
+ if r1 != 0x110 goto 9f; \
+ r1 = 0x101; \
+ .8byte %[fetch_and_insn]; /* r1 = atomic_fetch_and((u64 *)(r6 + 8), r1) */ \
+ r0 = 2; \
+ if r1 != 0x111 goto 9f; \
+ r1 = 0x011; \
+ .8byte %[fetch_xor_insn]; /* r1 = atomic_fetch_xor((u64 *)(r6 + 8), r1) */ \
+ r0 = 3; \
+ if r1 != 0x101 goto 9f; \
+ r2 = *(u64 *)(r6 + 8); \
+ r0 = 4; \
+ if r2 != 0x110 goto 9f; \
+ r0 = 0; \
+9: exit; \
+" :
+ : __imm_addr(arena),
+ __imm(bpf_arena_alloc_pages),
+ __imm_insn(fetch_or_insn, BPF_ATOMIC_OP(BPF_DW, BPF_OR | BPF_FETCH,
+ BPF_REG_6, BPF_REG_1, 8)),
+ __imm_insn(fetch_and_insn, BPF_ATOMIC_OP(BPF_DW, BPF_AND | BPF_FETCH,
+ BPF_REG_6, BPF_REG_1, 8)),
+ __imm_insn(fetch_xor_insn, BPF_ATOMIC_OP(BPF_DW, BPF_XOR | BPF_FETCH,
+ BPF_REG_6, BPF_REG_1, 8))
+ : __clobber_all);
+}
+
SEC("syscall")
__arena_scalar
__description("arena_scalar: cmpxchg through r0")
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH bpf-next v4 3/3] sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop
2026-10-14 12:50 [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 1/3] " Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 2/3] selftests/bpf: Test " Puranjay Mohan
@ 2026-10-14 12:50 ` Puranjay Mohan
2026-10-14 13:23 ` bot+bpf-ci
2026-10-15 17:06 ` [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena Tejun Heo
3 siblings, 1 reply; 7+ messages in thread
From: Puranjay Mohan @ 2026-10-14 12:50 UTC (permalink / raw)
To: bpf
Cc: Puranjay Mohan, Alexei Starovoitov, Daniel Borkmann,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Song Liu, Yonghong Song, Tejun Heo,
David Vernet, Andrea Righi
cid.bpf.h implements cmask_set(), cmask_clear() and the test_and_
variants with a bounded cmpxchg loop, because the x86 BPF JIT used to
reject BPF_OR | BPF_FETCH and BPF_AND | BPF_FETCH on arena pointers.
The x86 JIT accepts them now, and the loop was only ever an x86
workaround, not a fallback for the JITs that still reject these. arm64
without LSE rejects every arena read-modify-write, CMPXCHG included, and
riscv without Zacas rejects only CMPXCHG, where a fetching OR or AND is
accepted and the loop is not.
Use the fetching atomics directly, and drop CMASK_CAS_TRIES along with
the error paths that could only be reached from the loop.
Clang emits the fetching form even when the result is unused, so all four
helpers now need an x86-64 kernel whose JIT accepts fetching AND/OR/XOR in
an arena; schedulers built from this header no longer load on older
x86-64 kernels. When the bit is already as wanted, the helpers now also
do a locked write of the unchanged value where the loop returned after a
plain load.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
---
tools/sched_ext/include/scx/cid.bpf.h | 79 +++------------------------
1 file changed, 8 insertions(+), 71 deletions(-)
diff --git a/tools/sched_ext/include/scx/cid.bpf.h b/tools/sched_ext/include/scx/cid.bpf.h
index 69fb4e97bc771..d0d4cea19e29f 100644
--- a/tools/sched_ext/include/scx/cid.bpf.h
+++ b/tools/sched_ext/include/scx/cid.bpf.h
@@ -3,8 +3,8 @@
* BPF-side helpers for cids and cmasks. See kernel/sched/ext/cid.h for the
* authoritative layout and semantics. The BPF-side helpers use the cmask_*
* naming (no scx_ prefix); cmask is the SCX bitmap type so the prefix is
- * redundant in BPF code. Atomics use __sync_val_compare_and_swap and every
- * helper is inline (no .c counterpart).
+ * redundant in BPF code. Atomics use the __sync_* builtins and every helper
+ * is inline (no .c counterpart).
*
* Included by scx/common.bpf.h; don't include directly.
*
@@ -129,101 +129,38 @@ static __always_inline bool cmask_test(u32 cid, const struct scx_cmask __arena *
return *__cmask_word(cid, m) & BIT_U64(cid & 63);
}
-/*
- * x86 BPF JIT rejects BPF_OR | BPF_FETCH and BPF_AND | BPF_FETCH on arena
- * pointers (see bpf_jit_supports_insn() in arch/x86/net/bpf_jit_comp.c). Only
- * BPF_CMPXCHG / BPF_XCHG / BPF_ADD with FETCH are allowed. Implement
- * test_and_{set,clear} and the atomic set/clear via a cmpxchg loop.
- *
- * CMASK_CAS_TRIES is sized so exhausting it means seconds of real spinning
- * on one word - past any plausible contention. Abort hard.
- */
-#define CMASK_CAS_TRIES (1U << 23)
-
static __always_inline void cmask_set(u32 cid, struct scx_cmask __arena *m)
{
- u64 __arena *w;
- u64 bit, old, new;
- u32 i;
-
if (!__cmask_contains(cid, m))
return;
- w = __cmask_word(cid, m);
- bit = BIT_U64(cid & 63);
- bpf_for(i, 0, CMASK_CAS_TRIES) {
- old = *w;
- if (old & bit)
- return;
- new = old | bit;
- if (__sync_val_compare_and_swap(w, old, new) == old)
- return;
- }
- scx_bpf_error("cmask_set CAS exhausted at cid %u", cid);
+ __sync_fetch_and_or(__cmask_word(cid, m), BIT_U64(cid & 63));
}
static __always_inline void cmask_clear(u32 cid, struct scx_cmask __arena *m)
{
- u64 __arena *w;
- u64 bit, old, new;
- u32 i;
-
if (!__cmask_contains(cid, m))
return;
- w = __cmask_word(cid, m);
- bit = BIT_U64(cid & 63);
- bpf_for(i, 0, CMASK_CAS_TRIES) {
- old = *w;
- if (!(old & bit))
- return;
- new = old & ~bit;
- if (__sync_val_compare_and_swap(w, old, new) == old)
- return;
- }
- scx_bpf_error("cmask_clear CAS exhausted at cid %u", cid);
+ __sync_fetch_and_and(__cmask_word(cid, m), ~BIT_U64(cid & 63));
}
static __always_inline bool cmask_test_and_set(u32 cid, struct scx_cmask __arena *m)
{
- u64 __arena *w;
- u64 bit, old, new;
- u32 i;
+ u64 bit;
if (!__cmask_contains(cid, m))
return false;
- w = __cmask_word(cid, m);
bit = BIT_U64(cid & 63);
- bpf_for(i, 0, CMASK_CAS_TRIES) {
- old = *w;
- if (old & bit)
- return true;
- new = old | bit;
- if (__sync_val_compare_and_swap(w, old, new) == old)
- return false;
- }
- scx_bpf_error("cmask_test_and_set CAS exhausted at cid %u", cid);
- return false;
+ return __sync_fetch_and_or(__cmask_word(cid, m), bit) & bit;
}
static __always_inline bool cmask_test_and_clear(u32 cid, struct scx_cmask __arena *m)
{
- u64 __arena *w;
- u64 bit, old, new;
- u32 i;
+ u64 bit;
if (!__cmask_contains(cid, m))
return false;
- w = __cmask_word(cid, m);
bit = BIT_U64(cid & 63);
- bpf_for(i, 0, CMASK_CAS_TRIES) {
- old = *w;
- if (!(old & bit))
- return false;
- new = old & ~bit;
- if (__sync_val_compare_and_swap(w, old, new) == old)
- return true;
- }
- scx_bpf_error("cmask_test_and_clear CAS exhausted at cid %u", cid);
- return false;
+ return __sync_fetch_and_and(__cmask_word(cid, m), ~bit) & bit;
}
static __always_inline void __cmask_set(u32 cid, struct scx_cmask __arena *m)
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH bpf-next v4 3/3] sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop
2026-10-14 12:50 ` [PATCH bpf-next v4 3/3] sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop Puranjay Mohan
@ 2026-10-14 13:23 ` bot+bpf-ci
2026-10-14 13:34 ` Puranjay Mohan
0 siblings, 1 reply; 7+ messages in thread
From: bot+bpf-ci @ 2026-10-14 13:23 UTC (permalink / raw)
To: puranjay, bpf
Cc: puranjay, ast, daniel, andrii, martin.lau, eddyz87, memxor, song,
yonghong.song, tj, void, arighi, ast, andrii, daniel, martin.lau,
eddyz87, yonghong.song, mason, ihor.solodrai
[-- Attachment #1: Type: text/plain, Size: 4184 bytes --]
> sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop
TL;DR: Commit message only: the riscv-without-Zacas claim looks impossible
since arena needs Zacas, and "all four helpers" needing new JIT support holds
only for LLVM >= 20.
> sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop
>
> cid.bpf.h implements cmask_set(), cmask_clear() and the test_and_
> variants with a bounded cmpxchg loop, because the x86 BPF JIT used to
> reject BPF_OR | BPF_FETCH and BPF_AND | BPF_FETCH on arena pointers.
>
> The x86 JIT accepts them now, and the loop was only ever an x86
> workaround, not a fallback for the JITs that still reject these. arm64
> without LSE rejects every arena read-modify-write, CMPXCHG included, and
> riscv without Zacas rejects only CMPXCHG, where a fetching OR or AND is
> accepted and the loop is not.
>
> Use the fetching atomics directly, and drop CMASK_CAS_TRIES along with
> the error paths that could only be reached from the loop.
>
> Clang emits the fetching form even when the result is unused, so all four
> helpers now need an x86-64 kernel whose JIT accepts fetching AND/OR/XOR in
> an arena; schedulers built from this header no longer load on older
> x86-64 kernels. When the bit is already as wanted, the helpers now also
> do a locked write of the unchanged value where the loop returned after a
> plain load.
>
> Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
The changelog says "riscv without Zacas rejects only CMPXCHG, where a
fetching OR or AND is accepted and the loop is not." Can that configuration
exist?
On riscv, bpf_jit_supports_arena() returns system_has_cmpxchg128(), which is
only defined with CONFIG_RISCV_ISA_ZACAS && CONFIG_TOOLCHAIN_HAS_ZACAS and
expands to riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS). Since commit
536b523b4073 ("bpf, riscv: Make arena support depend on ZACAS"), without
Zacas arena_map_alloc() returns -EOPNOTSUPP and the verifier rejects the
program with "JIT doesn't support arena".
Before that commit, arena creation on non-Zacas riscv already failed with
-ENOMEM because of f8c67d8550ee ("bpf: Use kmalloc_nolock() in range tree").
That predates the cmask helpers (a58e6b79b432), so no arena program loads on
riscv without Zacas, and a fetching OR or AND is not accepted there either.
The "if (insn->imm == BPF_CMPXCHG) return rv_ext_enabled(ZACAS);" check in
riscv's bpf_jit_supports_insn() tests the same extension, so it cannot reject
an arena CMPXCHG once arena is available.
The main point, that the loop was never a fallback for other JITs, still
looks right. Should the riscv sentence be dropped, or reworded to say that
riscv needs Zacas for arena at all?
The changelog also says "Clang emits the fetching form even when the result
is unused, so all four helpers now need an x86-64 kernel whose JIT accepts
fetching AND/OR/XOR in an arena." Is that true for every supported clang?
Commit 2897b1e2a2f4 ("selftests/bpf: Fix arena_atomics failure due to llvm
change") records that only LLVM PR 106494 (LLVM 20) made
__sync_fetch_and_{and,or,xor}() emit atomic_fetch_*() when the return value
is unused. Older LLVM, including LLVM 19 which already has the
addr_space_cast support that __arena needs, rewrites unused-result fetches
into the non-fetching BPF_AND/BPF_OR atomics, and tools/sched_ext/README.md
still lists clang >= 16.0.0.
With such a clang, cmask_set() and cmask_clear() would compile to the
non-fetching form:
__sync_fetch_and_or(__cmask_word(cid, m), BIT_U64(cid & 63));
which the x86 JIT already accepted in an arena before this series. Only
the test_and_set and test_and_clear helpers would strictly need the new
JIT support on those toolchains.
The "schedulers no longer load on older x86-64 kernels" conclusion still
holds for scx_qmap, which uses cmask_test_and_clear() in
pick_direct_dispatch_cid(). Would it be more accurate to say "with
LLVM >= 20" before "all four helpers"?
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/38567454417
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH bpf-next v4 3/3] sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop
2026-10-14 13:23 ` bot+bpf-ci
@ 2026-10-14 13:34 ` Puranjay Mohan
0 siblings, 0 replies; 7+ messages in thread
From: Puranjay Mohan @ 2026-10-14 13:34 UTC (permalink / raw)
To: bot+bpf-ci
Cc: bpf, ast, daniel, andrii, martin.lau, eddyz87, memxor, song,
yonghong.song, tj, void, arighi, martin.lau, mason, ihor.solodrai
On Wed, Oct 14, 2026 at 2:23 PM <bot+bpf-ci@kernel.org> wrote:
>
> > sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop
>
> TL;DR: Commit message only: the riscv-without-Zacas claim looks impossible
> since arena needs Zacas, and "all four helpers" needing new JIT support holds
> only for LLVM >= 20.
LLVM >= 20 is expected now.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena
2026-10-14 12:50 [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena Puranjay Mohan
` (2 preceding siblings ...)
2026-10-14 12:50 ` [PATCH bpf-next v4 3/3] sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop Puranjay Mohan
@ 2026-10-15 17:06 ` Tejun Heo
3 siblings, 0 replies; 7+ messages in thread
From: Tejun Heo @ 2026-10-15 17:06 UTC (permalink / raw)
To: Puranjay Mohan
Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Kumar Kartikeya Dwivedi,
Song Liu, Yonghong Song, David Vernet, Andrea Righi, Changwoo Min,
sched-ext, linux-kernel
Hello, Puranjay.
On Wed, Oct 14, 2026 at 05:50:10AM -0700, Puranjay Mohan wrote:
> A fetching AND/OR/XOR against arena memory is rejected on x86-64:
>
> BPF_ATOMIC stores into R1 arena is not allowed
- 1/3: A fetching atomic through a number now fails to load with an
empty verifier log, where it used to get the message above. That
seems problematic. Can that case keep an error message?
- 3/3: sched_ext/for-7.4 already synced a cid.bpf.h from scx which
probes for the fetching atomics and keeps the CAS loop as the
old-kernel fallback, so this conflicts with it. I'll pull
bpf/for-next into sched_ext/for-7.4 and we can do the modifications
on top.
For 1/3 and 2/3,
Acked-by: Tejun Heo <tj@kernel.org>
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-10-15 17:06 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-14 12:50 [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 1/3] " Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 2/3] selftests/bpf: Test " Puranjay Mohan
2026-10-14 12:50 ` [PATCH bpf-next v4 3/3] sched_ext: Use fetching atomics for cmask instead of a cmpxchg loop Puranjay Mohan
2026-10-14 13:23 ` bot+bpf-ci
2026-10-14 13:34 ` Puranjay Mohan
2026-10-15 17:06 ` [PATCH bpf-next v4 0/3] bpf, x86: Support fetching AND/OR/XOR atomics in arena Tejun Heo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox