From: Sasha Levin <sashal@kernel.org>
To: patches@lists.linux.dev, stable@vger.kernel.org
Cc: "Israel Téllez García" <i.tellez@btesa.com>,
"Andrii Nakryiko" <andrii@kernel.org>,
"Sasha Levin" <sashal@kernel.org>,
eddyz87@gmail.com, ast@kernel.org, daniel@iogearbox.net,
memxor@gmail.com, bpf@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: [PATCH AUTOSEL 6.18-5.10] libbpf: Fix ring buffer consumer loop on 32-bit position wrap
Date: Mon, 19 Oct 2026 09:46:07 -0400 [thread overview]
Message-ID: <20261019135441.1420968-76-sashal@kernel.org> (raw)
In-Reply-To: <20261019135441.1420968-1-sashal@kernel.org>
From: Israel Téllez García <i.tellez@btesa.com>
[ Upstream commit fdd4fad0bbbd08501465c5f9b556963093c5d58a ]
ringbuf_process_ring() walks the records between the consumer and the
producer with an ordering comparison:
while (cons_pos < prod_pos) {
cons_pos and prod_pos mirror the kernel's ring positions and are
unsigned long here too, so on 32-bit they wrap at 2^32 bytes of traffic.
When producer_pos has wrapped and consumer_pos has not, prod_pos is the
smaller of the two, the loop body never runs and no record is consumed.
Since consumer_pos only advances inside that loop, it never wraps either
and the consumer stops delivering samples for good, with no error
returned to the caller: ring_buffer__poll() keeps reporting zero
records while the kernel side fills up and starts dropping.
Compare the distance instead. The consumer never runs ahead of the
producer, so prod_pos - cons_pos is the amount of unconsumed data and
stays correct across the wrap.
64-bit hosts are unaffected in practice: the counters would need 16 EiB
to wrap. This is the userspace counterpart of the kernel-side walk fixed
in "bpf: Fix pending_pos walk on 32-bit ring position wrap"; a 32-bit
consumer hits whichever of the two comes first.
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-5-i.tellez@btesa.com
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
LLM Generated explanations, may be completely bogus:
## Phase 1: Commit message forensics
**Step 1.1: PARSE THE SUBJECT LINE**
Record: **[libbpf] [Fix] ring buffer consumer loop on 32-bit position
wrap**
**Step 1.2: PARSE ALL COMMIT MESSAGE TAGS**
Record:
- **Signed-off-by:** Israel Téllez García `<i.tellez@btesa.com>`
(author)
- **Signed-off-by:** Andrii Nakryiko `<andrii@kernel.org>` (libbpf/BPF
maintainer)
- **Link:** `https://lore.kernel.org/bpf/20260814124843.22041-5-
i.tellez@btesa.com`
- No `Reported-by`, `Tested-by`, `Reviewed-by`, `Acked-by`, or `Cc:
stable` on the original patch
- Original submission had **no `Fixes:` tag**. `b4 am` later attached a
lore trailer: `Fixes: bf99c936f947 ("libbpf: Add BPF ring buffer
support")` (DKIM/kernel.org). That SHA is the 2020 introduction of BPF
ringbuf in libbpf, and it is in this tree.
**Step 1.3: ANALYZE THE COMMIT BODY TEXT**
Record:
- **Bug:** `ringbuf_process_ring()` uses `while (cons_pos < prod_pos)`
on `unsigned long` positions. On 32-bit those wrap at 2^32 bytes.
After producer wrap, `prod_pos` is small and `cons_pos` is still
large, so the loop never runs.
- **Symptom:** `ring_buffer__poll()` / consume APIs keep returning 0
with no error; the kernel ring fills and drops samples permanently.
- **Root cause:** ordering comparison is not wrap-safe; consumer_pos
only advances inside that loop, so it never wraps either.
- **Version info:** 64-bit is unaffected in practice (16 EiB to wrap).
Explicitly the userspace counterpart of the kernel `pending_pos` walk.
Field data for that sibling: four armv7 i.MX7 Dual devices on 6.6.52
stalled after exactly 4295491360 bytes (~2^32 + one 512 KiB ring).
- **Author’s fix:** `prod_pos - cons_pos > 0` (unsigned distance).
**Step 1.4: DETECT HIDDEN BUG FIXES**
Record: Not hidden. Subject and body describe a real wrap-induced stall.
Not cleanup/refactor.
---
## Phase 2: Diff analysis
**Step 2.1: INVENTORY THE CHANGES**
Record:
- File: `tools/lib/bpf/ringbuf.c` — **1 line changed** (`+1 / -1`)
- Function: `ringbuf_process_ring()`
- Scope: single-file surgical fix
**Step 2.2: UNDERSTAND THE CODE FLOW CHANGE**
Record:
- **Before:** `while (cons_pos < prod_pos)` — fails after producer wrap
on 32-bit.
- **After:** `while (prod_pos - cons_pos > 0)` — unsigned modular
distance; stays true while unconsumed data exists.
- Path: normal consume path (`ring_buffer__poll` / `__consume` /
`__consume_n` / `ring__consume*`), not an error/init-only path.
**Step 2.3: IDENTIFY THE BUG MECHANISM**
Record:
- **Category:** logic/correctness (32-bit wrap / unsigned comparison)
- **Mechanism:** same class as kernel `pending_pos`. Unsigned `unsigned
long` counters are documented to wrap
(`Documentation/bpf/ringbuf.rst`). Distance arithmetic is already used
in this tree in `ring__avail_data_size()` (`return prod_pos -
cons_pos`) and kernel `ringbuf_avail_data_sz()`.
**Step 2.4: ASSESS THE FIX QUALITY**
Record:
- Obviously correct: matches the kernel-side pattern already in this
tree (`while (prod_pos - pend_pos > 0)` in `kernel/bpf/ringbuf.c`).
- Minimal: one condition.
- Regression risk: low. On 64-bit equivalent until astronomical wrap. If
consumer ever ran ahead of producer, the new check would wrap and keep
looping; the author and kernel invariant say that does not happen.
Same assumption the kernel pending_pos fix already shipped here.
---
## Phase 3: Git history investigation
**Step 3.1: BLAME THE CHANGED LINES**
Record: Buggy `while (cons_pos < prod_pos)` introduced in
**`bf99c936f9478`** (“libbpf: Add BPF ring buffer support”, 2020-06-01)
by Andrii Nakryiko. Present since BPF ringbuf existed. Surrounding `cnt
>= n` bits are later (`13e8125a22763`, 2024) and unrelated.
**Step 3.2: FOLLOW THE FIXES: TAG**
Record: Original mail had no `Fixes:`. Trailer names `bf99c936f947`,
which **is in this tree**. Kernel sibling `19de99ea47e3c` (“bpf: Fix
pending_pos walk on 32-bit ring position wrap”) **is already in this
tree**; its `Fixes:` target `cfa1a2329a691` is also here.
**Step 3.3: CHECK FILE HISTORY FOR RELATED CHANGES**
Record: Recent `tools/lib/bpf/ringbuf.c` commits are consume_n
overflow/return-value fixes, not wrap handling. This is standalone:
series patch 4/4. Cover letter: patches 1–3 are kernel; 4 is libbpf.
Patches 2–3 are overwrite-mode `ringbuf_avail_data_sz()` —
**overwrite_pos does not exist in this 6.12 tree**. Nakryiko dropped
patch 3 anyway.
**Step 3.4: CHECK THE AUTHOR'S OTHER COMMITS**
Record: Israel Téllez García authored the kernel wrap fix already on
this branch (`19de99ea47e3c`). Andrii Nakryiko (libbpf maintainer)
applied both.
**Step 3.5: CHECK FOR DEPENDENT/PREREQUISITE COMMITS**
Record: No code dependency. One-line condition change in a function that
already exists. `git apply --check` on the original mbox succeeded (exit
0). Does not need overwrite-mode patches.
---
## Phase 4: Mailing list and external research
**Step 4.1: FIND THE ORIGINAL PATCH DISCUSSION**
Record:
- Series: **[PATCH v2 bpf 0/4] bpf: Fix ring buffer position wrap
handling on 32-bit**
- Lore: `https://lore.kernel.org/bpf/20260814124843.22041-1-
i.tellez@btesa.com`
- This patch: `...22041-5-i.tellez@btesa.com` (4/4)
- v1: `https://lore.kernel.org/bpf/20260806130214.66028-1-
i.tellez@btesa.com/` (kernel pending_pos only; v2 added sites 2–4)
- v2 change for patch 1: drop `(long)` cast per Nakryiko; unsigned
difference is enough
- Cover: patch 1 was the field-observed stall; patch 4 is the userspace
counterpart. “A 32-bit consumer hits whichever of the two comes
first.”
- No NAK on patch 4. Patch 3 was dropped by both author and Nakryiko
(ordering of overwrite_pos loads).
**Step 4.2: CHECK WHO REVIEWED THE PATCH**
Record (`b4 dig -w`): To/Cc included Andrii Nakryiko, Alexei
Starovoitov, Daniel Borkmann, Eduard Zingerman, Martin KaFai Lau, Song
Liu, Jiri Olsa, `bpf@vger.kernel.org`. Applied by Nakryiko.
**Step 4.3: SEARCH FOR THE BUG REPORT**
Record: No syzbot/bugzilla for the libbpf loop itself. Field evidence is
on the kernel sibling: four armv7 devices, exact 2^32-byte trigger, 8–17
h uptime, one patched device then passed 10 GiB. Cover: “Userspace kept
consuming normally until the producer stopped” — meaning the kernel
stall fired first. Once that kernel fix is in (it is, here), this libbpf
loop is the next wrap failure.
**Step 4.4: CHECK FOR RELATED PATCHES AND SERIES**
Record:
1. Kernel pending_pos — **already in this tree**
2. Overwrite-mode avail-data — **N/A here** (no overwrite_mode)
3. producer_pos vs overwrite_pos load order — dropped; **N/A here**
4. **This libbpf loop** — candidate
- User-ringbuf peek (`u64` locals, `cons_pos >= prod_pos`) was called
out as a separate issue and is **not** this commit. Confirmed still
present at `kernel/bpf/ringbuf.c:687`.
**Step 4.5: CHECK STABLE MAILING LIST HISTORY**
Record: **UNVERIFIED** — lore.kernel.org/stable returned Anubis bot-
challenge (403/HTML). No stable-list discussion retrieved.
---
## Phase 5: Code semantic analysis
**Step 5.1: IDENTIFY KEY FUNCTIONS**
Record: `ringbuf_process_ring()` only. (semcode MCP not available; used
hunk headers + grep.)
**Step 5.2: TRACE CALLERS**
Record: All in-tree callers of `ringbuf_process_ring()`:
- `ring_buffer__consume_n()`
- `ring_buffer__consume()`
- `ring_buffer__poll()`
- `ring__consume_n()` / `ring__consume()`
These are the public libbpf ringbuf consume APIs. Reachable from any
userspace consumer of `BPF_MAP_TYPE_RINGBUF` (tracing, networking,
etc.). `bpftool` in this tree links `tools/lib/bpf`.
**Step 5.3: TRACE CALLEES**
Record: `smp_load_acquire` on positions/len, `roundup_len()`,
`r->sample_cb()`, `smp_store_release` of consumer_pos. No
allocation/locking in the changed condition.
**Step 5.4: FOLLOW THE CALL CHAIN**
Record: Userspace path: app/bpftool → `ring_buffer__poll/consume` →
`ringbuf_process_ring`. Trigger: ~4 GiB cumulative ring traffic on
32-bit. Unprivileged if the process already has the map fd (typical
tracing daemon). Not a kernel oops, but permanent event loss.
**Step 5.5: SEARCH FOR SIMILAR PATTERNS**
Record:
- Same buggy compare in
`tools/testing/selftests/bpf/benchs/bench_ringbufs.c:305` (benchmark
only; not this commit).
- Wrap-safe distance already in `ring__avail_data_size()` and kernel
`ringbuf_avail_data_sz()`.
- Kernel pending_pos already converted to distance in this tree.
---
## Phase 6: Cross-referencing against the local tree
**Step 6.1: DOES THE BUGGY CODE EXIST IN THIS TREE?**
Record: **Yes.** Local tree is **6.12.111** (`git describe`:
`p-6.12-3-gc504cf0baa3d8`, Makefile `VERSION=6 PATCHLEVEL=12
SUBLEVEL=111`).
```247:247:tools/lib/bpf/ringbuf.c
while (cons_pos < prod_pos) {
```
`struct ring` uses `unsigned long *consumer_pos` / `*producer_pos`
(lines 29–30). Bug introduced in v5.8-era libbpf ringbuf
(`bf99c936f9478`), long before 6.12. Positions are still `unsigned long`
in kernel `struct bpf_ringbuf`.
**Step 6.2: CHECK FOR BACKPORT COMPLICATIONS**
Record: **Clean apply.** `git apply --check` on the original patch mbox
exited 0. Function shape matches the hunk (`ringbuf_process_ring(struct
ring *r, size_t n)` with `cons_pos`/`prod_pos` as `unsigned long`).
**Step 6.3: CHECK IF RELATED FIXES ARE ALREADY HERE**
Record: Kernel pending_pos wrap fix **is** here (`19de99ea47e3c`).
Userspace loop still has `cons_pos < prod_pos`. No alternate libbpf wrap
fix in this file.
---
## Phase 7: Subsystem and maintainer context
**Step 7.1: IDENTIFY THE SUBSYSTEM AND ITS CRITICALITY**
Record: **libbpf** (`tools/lib/bpf/`) — userspace helper shipped with
the kernel, used by bpftool and in-tree BPF tools. Criticality:
**IMPORTANT** for BPF users, especially 32-bit embedded collectors. Not
core mm/sched, but this tree already backports libbpf ringbuf fixes
(`087d757fb4736`, `00f0e08f23fc0`).
**Step 7.2: ASSESS SUBSYSTEM ACTIVITY**
Record: Actively maintained; libbpf ringbuf APIs grew through 6.12
(`ring__consume_n`, etc.). The wrap bug is old (2020), not a new-feature
leftover.
---
## Phase 8: Impact and risk assessment
**Step 8.1: DETERMINE WHO IS AFFECTED**
Record: **Config/platform-specific:** 32-bit hosts using in-tree libbpf
to consume `BPF_MAP_TYPE_RINGBUF` (armv7/i.MX and similar LTS/embedded).
64-bit unaffected in practice.
**Step 8.2: DETERMINE THE TRIGGER CONDITIONS**
Record: After ~4 GiB of ring traffic, if producer wraps while consumer
still has unconsumed bytes. Common with epoll/batching. Timing-dependent
at the wrap, then **sticky forever**. Unprivileged if the process
already holds the map. The sibling bug was reproduced at 441–862
records/s over hours — not a theoretical corner.
**Step 8.3: DETERMINE THE FAILURE MODE SEVERITY**
Record: Permanent silent stop of sample delivery; kernel then drops
events; APIs return 0 with no error. **HIGH** data/event loss (not a
kernel oops). Restart of the consumer is required.
**Step 8.4: CALCULATE RISK-BENEFIT RATIO**
Record:
- **Benefit: high for 32-bit BPF consumers.** Kernel producer wrap is
already fixed here, so this is now the remaining stall. Same class as
a field-proven 6.6.52 armv7 outage.
- **Risk: very low.** One comparison, same idiom as kernel code already
on this branch.
- Ratio: strongly favors backport.
---
## Phase 9: Final synthesis
**Step 9.1: COMPILE THE EVIDENCE**
**FOR:**
- Real wrap bug; documented that 32-bit positions wrap
- Permanent silent event loss, no error
- One-line, obviously correct, applies cleanly
- Kernel counterpart already in this 6.12.y tree — without this,
producer keeps going and userspace is the next stall
- Field-proven sibling on armv7 (core LTS/embedded)
- Maintainer SOB (Nakryiko); v2 reviewed
- No new APIs; no overwrite-mode dependency
**AGAINST:**
- Userspace libbpf (some distros use github libbpf, not in-tree)
- 32-bit only in practice
- Original field report hit kernel pending_pos first
- No `Cc: stable` on the original mail (expected for this review queue;
not a negative)
**UNRESOLVED:**
- Could not query lore.kernel.org/stable (bot wall)
- Could not confirm whether standalone libbpf already carries this
(irrelevant: verdict is for in-tree `tools/lib/bpf`)
**Step 9.2: APPLY THE STABLE RULES CHECKLIST**
1. Obviously correct and tested? **PASS** — same unsigned-distance idiom
as kernel code already here; series tested on armv7 for the sibling;
maintainer applied it.
2. Fixes a real bug that affects users? **PASS** — 32-bit ringbuf
consumers stall after wrap.
3. Important issue (crash, security, corruption, deadlock)? **PASS** at
HIGH — permanent silent event loss, not an oops, but equivalent
operational failure for tracing/telemetry.
4. Small and contained? **PASS** — 1 line, 1 file.
5. No new features or APIs? **PASS**
6. Can apply to the local tree? **PASS** — `git apply --check`
succeeded; buggy loop present.
**Step 9.3: CHECK FOR EXCEPTION CATEGORIES**
Record: Not a device-ID/quirk/DT/docs exception. Ordinary small bugfix.
**Step 9.4: DECISION**
This belongs on **this** 6.12.y tree. The buggy `cons_pos < prod_pos`
loop is here; the matching kernel wrap fix is already here; the one-line
change applies cleanly and closes the remaining 32-bit stall for in-tree
libbpf consumers.
### What the commit solves and why it matters here
On 32-bit, ring positions are 32-bit `unsigned long` and are documented
to wrap. After producer wrap, the consumer loop never runs, never
advances `consumer_pos`, and never recovers. `ring_buffer__poll()`
reports 0 forever while the kernel drops samples. 6.12.y already
contains the kernel `pending_pos` wrap fix, so a 32-bit producer can now
wrap; this userspace loop is the next failure. 32-bit ARM collectors are
a real LTS audience (the sibling was seen on i.MX7 Dual).
### Risk vs benefit
Benefit is high for 32-bit BPF ringbuf users; risk is a one-line
comparison already used in-kernel on this branch.
---
### Verification
- **[Phase 1]** Parsed subject/tags/body from the candidate and from
`/tmp/ringbuf-mbox/wrap-p5`; `Fixes:` only as a later lore trailer to
`bf99c936f947`
- **[Phase 2]** Diff is one condition in `ringbuf_process_ring()`;
current tree still has `while (cons_pos < prod_pos)` at line 247
- **[Phase 3]** `git blame` → `bf99c936f9478` (2020); `git merge-base
--is-ancestor` confirms that SHA, `cfa1a2329a691`, and kernel wrap fix
`19de99ea47e3c` are in HEAD
- **[Phase 3]** `git log --oneline -- tools/lib/bpf/ringbuf.c` — no
userspace wrap fix present
- **[Phase 4]** `b4 dig -c 19de99ea47e3c -a/-w` → v2 4-patch series;
this is 4/4
- **[Phase 4]** Cover letter and patches from `/tmp/ringbuf-
series/*.cover` and `.mbx`; Nakryiko dropped patch 3 (`/tmp/ringbuf-
mbox/nakryiko`)
- **[Phase 4]** lore.kernel.org HTTP 403 / Anubis; used `b4` mboxes
instead
- **[Phase 5]** grep callers of `ringbuf_process_ring` in
`tools/lib/bpf/ringbuf.c`; public consume/poll APIs
- **[Phase 6]** `git describe HEAD` → `p-6.12-3-gc504cf0baa3d8`; `make
kernelversion` → `6.12.111`
- **[Phase 6]** `git apply --check /tmp/ringbuf-mbox/wrap-p5` exit 0
- **[Phase 6]** `overwrite_pos` / `overwrite_mode` absent from
`kernel/bpf/ringbuf.c` in this tree
- **[Phase 7/8]** `Documentation/bpf/ringbuf.rst` documents 32-bit wrap;
kernel sibling commit describes four armv7 devices
- **UNVERIFIED:** stable@ lore discussion; whether distro standalone
libbpf already has this
**YES**
tools/lib/bpf/ringbuf.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/lib/bpf/ringbuf.c b/tools/lib/bpf/ringbuf.c
index 00ec4837a06de..c8fe57401a8b5 100644
--- a/tools/lib/bpf/ringbuf.c
+++ b/tools/lib/bpf/ringbuf.c
@@ -244,7 +244,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)
do {
got_new_data = false;
prod_pos = smp_load_acquire(r->producer_pos);
- while (cons_pos < prod_pos) {
+ while (prod_pos - cons_pos > 0) {
len_ptr = r->data + (cons_pos & r->mask);
len = smp_load_acquire(len_ptr);
--
2.53.0
next prev parent reply other threads:[~2026-10-19 13:56 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <20261019135441.1420968-1-sashal@kernel.org>
2026-10-19 13:45 ` [PATCH AUTOSEL 6.18-6.1] libbpf: Skip bpf_object__probe_loading() when BPF token is in use Sasha Levin
2026-10-19 13:46 ` Sasha Levin [this message]
2026-10-19 13:46 ` [PATCH AUTOSEL 6.18-5.10] riscv, bpf: Add support for BPF_SDIV and BPF_SMOD in RV32 JIT Sasha Levin
2026-10-19 13:48 ` [PATCH AUTOSEL 6.18] bpf: Reject callback subprogs invoke tailcall Sasha Levin
2026-10-19 15:01 ` sashiko-bot
2026-10-19 13:50 ` [PATCH AUTOSEL 6.18-6.6] xsk: account ring allocations to memcg Sasha Levin
2026-10-19 13:52 ` [PATCH AUTOSEL 6.18-5.10] riscv, bpf: Add support for BPF_MOVSX in RV32 JIT Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261019135441.1420968-76-sashal@kernel.org \
--to=sashal@kernel.org \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=eddyz87@gmail.com \
--cc=i.tellez@btesa.com \
--cc=linux-kernel@vger.kernel.org \
--cc=memxor@gmail.com \
--cc=patches@lists.linux.dev \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox