From: Tamir Duberstein <tamird@kernel.org>
To: Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>,
Eduard Zingerman <eddyz87@gmail.com>,
Kumar Kartikeya Dwivedi <memxor@gmail.com>,
Martin KaFai Lau <martin.lau@linux.dev>,
Song Liu <song@kernel.org>,
Yonghong Song <yonghong.song@linux.dev>,
Jiri Olsa <jolsa@kernel.org>,
Emil Tsalapatis <emil@etsalapatis.com>,
Shuah Khan <shuah@kernel.org>, Andrea Righi <arighi@nvidia.com>,
Xu Kuohai <xukuohai@huawei.com>,
Andrea Righi <andrea.righi@canonical.com>,
Bing-Jhong Billy Jheng <billy@starlabs.sg>,
David Vernet <void@manifault.com>,
Ihor Solodrai <ihor.solodrai@linux.dev>
Cc: bpf@vger.kernel.org, linux-kselftest@vger.kernel.org,
linux-kernel@vger.kernel.org,
Andrew Werner <awerner32@gmail.com>,
Zvi Effron <zeffron@riotgames.com>,
Andrii Nakryiko <andriin@fb.com>,
Tamir Duberstein <tamird@kernel.org>,
Sashiko <sashiko-bot@kernel.org>
Subject: [PATCH bpf-next v3 3/8] libbpf: ringbuf: Prevent missed wakeups
Date: Fri, 20 Nov 2026 15:33:16 -0500 [thread overview]
Message-ID: <20261120-bpf-ringbuf-fixes-v3-3-ff7ca15d6c37@kernel.org> (raw)
In-Reply-To: <20261120-bpf-ringbuf-fixes-v3-0-ff7ca15d6c37@kernel.org>
After consuming the last visible record, ringbuf_process_ring()
publishes the consumer position and checks the producer position. These
operations lack a full StoreLoad barrier. A producer can therefore
commit a new record but read the old consumer position while the
consumer reads the old producer position. The producer sends no
notification and the consumer waits despite a queued record.
Insert a full barrier between publishing a consumer position and the
next producer position load. When a record bound or callback ends the
current invocation first, execute the barrier before returning so the
load in a later invocation completes the same handshake.
After publication, retry a busy record once behind the barrier before
returning. Otherwise a producer could commit it and skip a wakeup based
on the old consumer position.
Document that bounded consumers and callbacks that terminate consumption
must drain before waiting again.
Fixes: bf99c936f947 ("libbpf: Add BPF ring buffer support")
Reported-by: Andrew Werner <awerner32@gmail.com>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/bpf/20260614015716.945AF1F000E9@smtp.kernel.org/
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Tamir Duberstein <tamird@kernel.org>
---
tools/lib/bpf/libbpf.h | 23 +++++++++++++++++++++++
tools/lib/bpf/ringbuf.c | 18 ++++++++++++++----
2 files changed, 37 insertions(+), 4 deletions(-)
diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
index 49f1120f4021..856650d58578 100644
--- a/tools/lib/bpf/libbpf.h
+++ b/tools/lib/bpf/libbpf.h
@@ -1542,6 +1542,11 @@ struct ring;
struct user_ring_buffer;
/* Callback-based consumption is unsupported for BPF_F_RB_OVERWRITE maps. */
+/*
+ * A negative return stops consumption; non-negative values continue. Stopping
+ * can leave records queued without a new readiness notification. Before
+ * waiting for readiness again, consume until no records remain.
+ */
typedef int (*ring_buffer_sample_fn)(void *ctx, void *data, size_t size);
struct ring_buffer_opts {
@@ -1558,6 +1563,20 @@ LIBBPF_API int ring_buffer__add(struct ring_buffer *rb, int map_fd,
ring_buffer_sample_fn sample_cb, void *ctx);
LIBBPF_API int ring_buffer__poll(struct ring_buffer *rb, int timeout_ms);
LIBBPF_API int ring_buffer__consume(struct ring_buffer *rb);
+
+/**
+ * @brief **ring_buffer__consume_n()** consumes up to a requested number of
+ * records from a ring buffer manager without event polling.
+ *
+ * @param rb A ring buffer manager object.
+ * @param n Maximum number of records to consume.
+ * @return The number of records consumed, or a negative error code on failure.
+ *
+ * Reaching the requested bound does not establish that every ring is empty.
+ * Records can remain queued without a new readiness notification. Before
+ * calling ring_buffer__poll() or waiting on ring_buffer__epoll_fd(), call
+ * ring_buffer__consume() until it returns 0.
+ */
LIBBPF_API int ring_buffer__consume_n(struct ring_buffer *rb, size_t n);
LIBBPF_API int ring_buffer__epoll_fd(const struct ring_buffer *rb);
@@ -1640,6 +1659,10 @@ LIBBPF_API int ring__consume(struct ring *r);
* @param r A ringbuffer object.
* @param n Maximum number of records to consume.
* @return The number of records consumed, or a negative error code on failure.
+ *
+ * Reaching the requested bound does not establish that the ring is empty.
+ * Records can remain queued without a new readiness notification. Before
+ * waiting on ring__map_fd(), call ring__consume() until it returns 0.
*/
LIBBPF_API int ring__consume_n(struct ring *r, size_t n);
diff --git a/tools/lib/bpf/ringbuf.c b/tools/lib/bpf/ringbuf.c
index db67eb98f3e0..7bf92311da68 100644
--- a/tools/lib/bpf/ringbuf.c
+++ b/tools/lib/bpf/ringbuf.c
@@ -271,16 +271,16 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)
return 0;
cons_pos = __atomic_load_n(r->consumer_pos, __ATOMIC_ACQUIRE);
- do {
+ for (;;) {
got_new_data = false;
prod_pos = __atomic_load_n(r->producer_pos, __ATOMIC_ACQUIRE);
while (prod_pos - cons_pos > 0) {
len_ptr = r->data + (cons_pos & r->mask);
len = __atomic_load_n(len_ptr, __ATOMIC_ACQUIRE);
- /* sample not committed yet, bail out for now */
+ /* Retry a busy record once after publishing prior records. */
if (len & BPF_RINGBUF_BUSY_BIT)
- goto done;
+ break;
got_new_data = true;
cons_pos += roundup_len(len);
@@ -291,6 +291,7 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)
if (err < 0) {
/* update consumer pos and bail out */
__atomic_store_n(r->consumer_pos, cons_pos, __ATOMIC_RELEASE);
+ __atomic_thread_fence(__ATOMIC_SEQ_CST);
return err;
}
cnt++;
@@ -301,8 +302,17 @@ static int64_t ringbuf_process_ring(struct ring *r, size_t n)
if (cnt >= n)
goto done;
}
- } while (got_new_data);
+ if (!got_new_data)
+ return cnt;
+
+ /*
+ * Order the published consumer position before the next
+ * producer-position load, whether below or in a later invocation.
+ */
+ __atomic_thread_fence(__ATOMIC_SEQ_CST);
+ }
done:
+ __atomic_thread_fence(__ATOMIC_SEQ_CST);
return cnt;
}
--
2.56.1
next prev parent reply other threads:[~2026-11-20 20:33 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 20:33 [PATCH bpf-next v3 0/8] bpf: Fix ring buffers and add a libbpf iterator Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 1/8] bpf: user_ringbuf: Handle position wrap Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 2/8] libbpf: ringbuf: Use compiler atomics Tamir Duberstein
2026-11-20 21:34 ` bot+bpf-ci
2026-11-20 22:38 ` Tamir Duberstein
2026-11-20 20:33 ` Tamir Duberstein [this message]
2026-11-20 20:33 ` [PATCH bpf-next v3 4/8] libbpf: ringbuf: Add a caller-owned iterator Tamir Duberstein
2026-11-20 21:34 ` bot+bpf-ci
2026-11-20 22:38 ` Tamir Duberstein
2026-11-21 10:01 ` Andrea Righi
2026-11-20 20:33 ` [PATCH bpf-next v3 5/8] libbpf: ringbuf: Simplify iterator implementation Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 6/8] selftests/bpf: Refactor lifetimes and test wakeups Tamir Duberstein
2026-11-20 21:34 ` bot+bpf-ci
2026-11-20 22:38 ` Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 7/8] selftests/bpf: Verify ringbuf callback semantics Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 8/8] selftests/bpf: Test caller-owned ring iterators Tamir Duberstein
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120-bpf-ringbuf-fixes-v3-3-ff7ca15d6c37@kernel.org \
--to=tamird@kernel.org \
--cc=andrea.righi@canonical.com \
--cc=andrii@kernel.org \
--cc=andriin@fb.com \
--cc=arighi@nvidia.com \
--cc=ast@kernel.org \
--cc=awerner32@gmail.com \
--cc=billy@starlabs.sg \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=eddyz87@gmail.com \
--cc=emil@etsalapatis.com \
--cc=ihor.solodrai@linux.dev \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=martin.lau@linux.dev \
--cc=memxor@gmail.com \
--cc=sashiko-bot@kernel.org \
--cc=shuah@kernel.org \
--cc=song@kernel.org \
--cc=void@manifault.com \
--cc=xukuohai@huawei.com \
--cc=yonghong.song@linux.dev \
--cc=zeffron@riotgames.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.