Linux Kernel Selftest development
 help / color / mirror / Atom feed
From: Andrea Righi <arighi@nvidia.com>
To: Tamir Duberstein <tamird@kernel.org>
Cc: Alexei Starovoitov <ast@kernel.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Andrii Nakryiko <andrii@kernel.org>,
	Eduard Zingerman <eddyz87@gmail.com>,
	Kumar Kartikeya Dwivedi <memxor@gmail.com>,
	Martin KaFai Lau <martin.lau@linux.dev>,
	Song Liu <song@kernel.org>,
	Yonghong Song <yonghong.song@linux.dev>,
	Jiri Olsa <jolsa@kernel.org>,
	Emil Tsalapatis <emil@etsalapatis.com>,
	Shuah Khan <shuah@kernel.org>, Xu Kuohai <xukuohai@huawei.com>,
	Bing-Jhong Billy Jheng <billy@starlabs.sg>,
	David Vernet <void@manifault.com>,
	Ihor Solodrai <ihor.solodrai@linux.dev>,
	bpf@vger.kernel.org, linux-kselftest@vger.kernel.org,
	linux-kernel@vger.kernel.org, Andrew Werner <awerner32@gmail.com>,
	Zvi Effron <zeffron@riotgames.com>,
	Andrii Nakryiko <andriin@fb.com>
Subject: Re: [PATCH bpf-next v3 4/8] libbpf: ringbuf: Add a caller-owned iterator
Date: Sat, 21 Nov 2026 11:01:01 +0100	[thread overview]
Message-ID: <awFr3R3DPS_so3Pp@gpd4> (raw)
In-Reply-To: <20261120-bpf-ringbuf-fixes-v3-4-ff7ca15d6c37@kernel.org>

Hi Tamir,

On Fri, Nov 20, 2026 at 03:33:17PM -0500, Tamir Duberstein wrote:
> Ring buffer records are exposed only through callbacks. Callers that
> need to stop at an operation-specific condition have no non-error way to
> end consumption early.
> 
> Add a caller-owned iterator that returns borrowed records one at a time.
> Each next call ends the borrow and can publish consumed records before
> inspecting another one, allowing callers to batch consumer updates.
> Implement callback consumption on the same internal iterator core so
> both paths share the record traversal and publication protocol. Allow
> iterator-only rings to omit the setup callback.
> 
> Preserve callback validation, so callback-based consumption of an
> iterator-only ring still returns -EINVAL. Keep the consumer-publication
> barrier on early destruction, including bounded and callback-error
> returns.
> 
> Keep overwrite maps out of the public iterator; handling a stale
> consumer position requires a separate resynchronization protocol.
> Require iterator users to observe exhaustion before waiting for another
> readiness notification.
> 
> Signed-off-by: Tamir Duberstein <tamird@kernel.org>
> ---
>  tools/lib/bpf/libbpf.h   |  67 ++++++++++++++++++
>  tools/lib/bpf/libbpf.map |   3 +
>  tools/lib/bpf/ringbuf.c  | 172 +++++++++++++++++++++++++++++++++++++----------
>  3 files changed, 207 insertions(+), 35 deletions(-)
> 
> diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
> index 856650d58578..cc9e141ffd0a 100644
> --- a/tools/lib/bpf/libbpf.h
> +++ b/tools/lib/bpf/libbpf.h
> @@ -1541,6 +1541,11 @@ struct ring_buffer;
>  struct ring;
>  struct user_ring_buffer;
>  
> +/* Opaque caller-owned iterator state. */
> +struct ring_buffer_iter {
> +	__u64 __opaque[8];
> +};
> +
>  /* Callback-based consumption is unsupported for BPF_F_RB_OVERWRITE maps. */
>  /*
>   * A negative return stops consumption; non-negative values continue. Stopping
> @@ -1594,6 +1599,68 @@ LIBBPF_API int ring_buffer__epoll_fd(const struct ring_buffer *rb);
>  LIBBPF_API struct ring *ring_buffer__ring(struct ring_buffer *rb,
>  					  unsigned int idx);
>  
> +/**
> + * @brief **ring_buffer_iter_new()** initializes a caller-owned iterator for a
> + * ring buffer map.
> + *
> + * @param it Pointer to the iterator to initialize.
> + * @param r Ring buffer map to iterate. Its ring buffer manager must remain
> + * valid until the iterator is destroyed.
> + * @return 0 on success; a negative error code on failure.
> + *
> + * Every successful call must be paired with ring_buffer_iter_destroy() before
> + * the iterator storage is reused or the ring buffer manager is freed. An
> + * active iterator must not be copied. On failure, the iterator is unchanged.
> + *
> + * A ring consumed only through iterators may be registered with a NULL sample
> + * callback. A sample callback is required to consume records through
> + * callback-based APIs.
> + *
> + * Only one iterator or callback-based consumer may operate on a ring buffer
> + * map at a time. BPF_F_RB_OVERWRITE ring buffer maps are not supported.
> + */
> +LIBBPF_API int ring_buffer_iter_new(struct ring_buffer_iter *it,
> +				    struct ring *r);
> +
> +/**
> + * @brief **ring_buffer_iter_next()** returns the next available record.
> + *
> + * @param it Pointer to an initialized ring buffer iterator.
> + * @param size Optional pointer that receives the record size.
> + * @param sync Publish consumed records before advancing.
> + * @return A read-only pointer to the record data, or NULL if the iterator is
> + * exhausted.
> + *
> + * The returned record's data becomes invalid on the next call to
> + * ring_buffer_iter_next() or ring_buffer_iter_destroy(), regardless of sync.
> + * When sync is true, this call publishes records returned by earlier calls
> + * before inspecting another one. Otherwise publication may be deferred until
> + * the iterator refreshes the producer position, observes a busy record, or
> + * is destroyed. Once this function returns NULL, it continues returning NULL.
> + * Create a new iterator to check the ring again.
> + *
> + * Only a NULL return establishes a notification-safe observation: after
> + * publishing the consumer position, the iterator observed no committed record.
> + * Before waiting for ring buffer readiness, drive an iterator to NULL. If an
> + * iterator was destroyed early, create another iterator and drive it to NULL
> + * before waiting.
> + */
> +LIBBPF_API const void *ring_buffer_iter_next(struct ring_buffer_iter *it,
> +					     size_t *size, bool sync);
> +
> +/**
> + * @brief **ring_buffer_iter_destroy()** destroys a ring buffer iterator.
> + *
> + * @param it Pointer to a zero-initialized iterator or one initialized by
> + * ring_buffer_iter_new().
> + *
> + * Any record returned by the previous ring_buffer_iter_next() call is consumed.
> + * Destroying an iterator before ring_buffer_iter_next() returns NULL publishes
> + * the consumed position, but records can remain queued without a new readiness
> + * notification.
> + */
> +LIBBPF_API void ring_buffer_iter_destroy(struct ring_buffer_iter *it);
> +
>  /**
>   * @brief **ring__consumer_pos()** returns the current consumer position in the
>   * given ringbuffer.
> diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
> index a45666f546fa..5ab5a46fedb1 100644
> --- a/tools/lib/bpf/libbpf.map
> +++ b/tools/lib/bpf/libbpf.map
> @@ -479,4 +479,7 @@ LIBBPF_1.9.0 {
>  		bpf_program__load;
>  		bpf_program__load_strategy;
>  		bpf_program__set_load_strategy;
> +		ring_buffer_iter_destroy;
> +		ring_buffer_iter_new;
> +		ring_buffer_iter_next;
>  } LIBBPF_1.8.0;
> diff --git a/tools/lib/bpf/ringbuf.c b/tools/lib/bpf/ringbuf.c
> index 7bf92311da68..2e281e2fa573 100644
> --- a/tools/lib/bpf/ringbuf.c
> +++ b/tools/lib/bpf/ringbuf.c
> @@ -255,64 +255,166 @@ static int ringbuf_validate_callbacks(const struct ring_buffer *rb)
>  	return 0;
>  }
>  
> -static int64_t ringbuf_process_ring(struct ring *r, size_t n)
> +/*
> + * The private state aliases the public iterator's opaque __u64 storage.
> + * Allow that access without violating strict aliasing rules.
> + */
> +struct ring_buffer_iter_state {
> +	struct ring *ring;
> +	unsigned long cons_pos;
> +	unsigned long prod_pos;
> +	bool consumer_dirty;
> +	bool needs_wakeup;
> +} __attribute__((__may_alias__));
> +
> +_Static_assert(sizeof(struct ring_buffer_iter_state) <=
> +		       sizeof(struct ring_buffer_iter),
> +	       "ring buffer iterator state exceeds opaque storage");
> +_Static_assert(__alignof__(struct ring_buffer_iter_state) <=
> +		       __alignof__(struct ring_buffer_iter),
> +	       "ring buffer iterator is under-aligned");
> +
> +static void ringbuf_iter_publish(struct ring_buffer_iter_state *state)
>  {
> -	int *len_ptr, len, err;
> -	/* 64-bit to avoid overflow in case of extreme application behavior */
> -	int64_t cnt = 0;
> +	if (!state->consumer_dirty)
> +		return;
> +
> +	/* Release consumed records to the producer. */
> +	__atomic_store_n(state->ring->consumer_pos, state->cons_pos,
> +			 __ATOMIC_RELEASE);
> +	state->consumer_dirty = false;
> +	state->needs_wakeup = true;
> +}
> +
> +static void ringbuf_iter_init(struct ring_buffer_iter *it, struct ring *r)
> +{
> +	struct ring_buffer_iter_state *state = (void *)it;
> +
> +	memset(it, 0, sizeof(*it));
> +	state->ring = r;
> +	state->cons_pos = __atomic_load_n(r->consumer_pos, __ATOMIC_ACQUIRE);
> +	state->prod_pos = state->cons_pos;
> +}
> +
> +int ring_buffer_iter_new(struct ring_buffer_iter *it, struct ring *r)
> +{
> +	if (r->overwrite)
> +		return libbpf_err(-EOPNOTSUPP);
> +
> +	ringbuf_iter_init(it, r);
> +	return 0;
> +}
> +
> +static const void *ringbuf_iter_next(struct ring_buffer_iter *it, size_t *size,
> +				     bool sync)
> +{

Can we make this internal helper __always_inline? Otherwise we have an
out-of-line call per record in ringbuf_process_ring(), which could be a
bit expensive. Inlining would also allow specialization of the sync=true
path.

I tried with a local CPU-pinned benchmark using extracted consumer code
and measured ~1014 ns to drain 512 precommitted 64-byte records, versus
~1231 ns with the final series. Inlining this helper, together with the
private destructor (see below), reduced that to ~1029 ns.

These are synthetic traversal measurements, not end-to-end BPF
throughput. It'd be interesting to check rb-libbpf throughput before and
after the callback conversion.

> +	struct ring_buffer_iter_state *state = (void *)it;
> +	struct ring *r = state->ring;
> +	int *len_ptr, len;
>  	unsigned long cons_pos, prod_pos;
>  	bool got_new_data;
>  	void *sample;
>  
> -	err = ringbuf_validate(r);
> -	if (err)
> -		return err;
> -	if (n == 0)
> -		return 0;
> +	if (!r)
> +		return NULL;
> +	if (sync)
> +		ringbuf_iter_publish(state);
> +	cons_pos = state->cons_pos;
> +	prod_pos = state->prod_pos;
>  
> -	cons_pos = __atomic_load_n(r->consumer_pos, __ATOMIC_ACQUIRE);
> -	for (;;) {
> +	do {
>  		got_new_data = false;
> -		prod_pos = __atomic_load_n(r->producer_pos, __ATOMIC_ACQUIRE);
> -		while (prod_pos - cons_pos > 0) {
> +		if (cons_pos == prod_pos) {
> +			ringbuf_iter_publish(state);
> +			if (state->needs_wakeup) {
> +				/*
> +				 * Ensure either this sees a new record or its producer sees
> +				 * the updated consumer position and sends a notification.
> +				 */
> +				__atomic_thread_fence(__ATOMIC_SEQ_CST);
> +				state->needs_wakeup = false;
> +			}
> +			prod_pos = __atomic_load_n(r->producer_pos,
> +						   __ATOMIC_ACQUIRE);
> +			state->prod_pos = prod_pos;
> +		}
> +
> +		while (cons_pos != prod_pos) {
>  			len_ptr = r->data + (cons_pos & r->mask);
>  			len = __atomic_load_n(len_ptr, __ATOMIC_ACQUIRE);
>  
>  			/* Retry a busy record once after publishing prior records. */
> -			if (len & BPF_RINGBUF_BUSY_BIT)
> +			if (len & BPF_RINGBUF_BUSY_BIT) {
> +				ringbuf_iter_publish(state);
> +				got_new_data = state->needs_wakeup;
> +				if (got_new_data) {
> +					/* Order the consumer update before retrying the header. */
> +					__atomic_thread_fence(__ATOMIC_SEQ_CST);
> +					state->needs_wakeup = false;
> +				}
>  				break;
> +			}
>  
>  			got_new_data = true;
>  			cons_pos += roundup_len(len);
> +			state->cons_pos = cons_pos;
> +			state->consumer_dirty = true;
>  
>  			if ((len & BPF_RINGBUF_DISCARD_BIT) == 0) {
>  				sample = (void *)len_ptr + BPF_RINGBUF_HDR_SZ;
> -				err = r->sample_cb(r->ctx, sample, len);
> -				if (err < 0) {
> -					/* update consumer pos and bail out */
> -					__atomic_store_n(r->consumer_pos, cons_pos, __ATOMIC_RELEASE);
> -					__atomic_thread_fence(__ATOMIC_SEQ_CST);
> -					return err;
> -				}
> -				cnt++;
> +				if (size)
> +					*size = len;
> +				return sample;
>  			}
>  
> -			__atomic_store_n(r->consumer_pos, cons_pos, __ATOMIC_RELEASE);
> -
> -			if (cnt >= n)
> -				goto done;
> +			if (sync)
> +				ringbuf_iter_publish(state);
>  		}
> -		if (!got_new_data)
> -			return cnt;
> +	} while (got_new_data);
>  
> -		/*
> -		 * Order the published consumer position before the next
> -		 * producer-position load, whether below or in a later invocation.
> -		 */
> +	state->ring = NULL;
> +	return NULL;
> +}
> +
> +const void *ring_buffer_iter_next(struct ring_buffer_iter *it, size_t *size,
> +				  bool sync)
> +{
> +	return ringbuf_iter_next(it, size, sync);
> +}
> +
> +void ring_buffer_iter_destroy(struct ring_buffer_iter *it)
> +{
> +	struct ring_buffer_iter_state *state = (void *)it;
> +
> +	ringbuf_iter_publish(state);
> +	/* Order publication before a producer load in a later invocation. */
> +	if (state->needs_wakeup)
>  		__atomic_thread_fence(__ATOMIC_SEQ_CST);
> +	memset(it, 0, sizeof(*it));
> +}

Likewise, can we move this body into a new inline helper and keep the
exported destructor as a wrapper? Something like this:

static __always_inline void
ringbuf_iter_destroy(struct ring_buffer_iter *it)
{
	struct ring_buffer_iter_state *state = (void *)it;

	ringbuf_iter_publish(state);
	/* Order publication before a producer load in a later invocation. */
	if (state->needs_wakeup)
		__atomic_thread_fence(__ATOMIC_SEQ_CST);
	memset(it, 0, sizeof(*it));
}

void ring_buffer_iter_destroy(struct ring_buffer_iter *it)
{
	ringbuf_iter_destroy(it);
}

> +
> +static int64_t ringbuf_process_ring(struct ring *r, size_t n)
> +{
> +	struct ring_buffer_iter it
> +		__attribute__((cleanup(ring_buffer_iter_destroy))) = {};

Then use the new private helper here:

	struct ring_buffer_iter it
		__attribute__((cleanup(ringbuf_iter_destroy))) = {};

> +	const void *sample;
> +	size_t size;
> +	int err = ringbuf_validate(r);
> +	/* 64-bit to avoid overflow in case of extreme application behavior */
> +	int64_t cnt = 0;
> +
> +	if (err)
> +		return err;
> +	if (n == 0)
> +		return 0;
> +
> +	ringbuf_iter_init(&it, r);
> +	while (cnt < n && (sample = ringbuf_iter_next(&it, &size, true))) {
> +		err = r->sample_cb(r->ctx, (void *)sample, size);
> +		if (err < 0)
> +			return err;
> +		cnt++;
>  	}
> -done:
> -	__atomic_thread_fence(__ATOMIC_SEQ_CST);
>  	return cnt;
>  }
>  
> @@ -421,7 +523,7 @@ struct ring *ring_buffer__ring(struct ring_buffer *rb, unsigned int idx)
>  
>  unsigned long ring__consumer_pos(const struct ring *r)
>  {
> -	/* Synchronizes with the release store in ringbuf_process_ring(). */
> +	/* Synchronizes with the iterator's release store. */
>  	return __atomic_load_n(r->consumer_pos, __ATOMIC_ACQUIRE);
>  }

Thanks,
-Andrea

  parent reply	other threads:[~2026-11-21 10:01 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 20:33 [PATCH bpf-next v3 0/8] bpf: Fix ring buffers and add a libbpf iterator Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 1/8] bpf: user_ringbuf: Handle position wrap Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 2/8] libbpf: ringbuf: Use compiler atomics Tamir Duberstein
2026-11-20 21:34   ` bot+bpf-ci
2026-11-20 22:38     ` Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 3/8] libbpf: ringbuf: Prevent missed wakeups Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 4/8] libbpf: ringbuf: Add a caller-owned iterator Tamir Duberstein
2026-11-20 21:34   ` bot+bpf-ci
2026-11-20 22:38     ` Tamir Duberstein
2026-11-21 10:01   ` Andrea Righi [this message]
2026-11-20 20:33 ` [PATCH bpf-next v3 5/8] libbpf: ringbuf: Simplify iterator implementation Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 6/8] selftests/bpf: Refactor lifetimes and test wakeups Tamir Duberstein
2026-11-20 21:34   ` bot+bpf-ci
2026-11-20 22:38     ` Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 7/8] selftests/bpf: Verify ringbuf callback semantics Tamir Duberstein
2026-11-20 20:33 ` [PATCH bpf-next v3 8/8] selftests/bpf: Test caller-owned ring iterators Tamir Duberstein

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=awFr3R3DPS_so3Pp@gpd4 \
    --to=arighi@nvidia.com \
    --cc=andrii@kernel.org \
    --cc=andriin@fb.com \
    --cc=ast@kernel.org \
    --cc=awerner32@gmail.com \
    --cc=billy@starlabs.sg \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=eddyz87@gmail.com \
    --cc=emil@etsalapatis.com \
    --cc=ihor.solodrai@linux.dev \
    --cc=jolsa@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=martin.lau@linux.dev \
    --cc=memxor@gmail.com \
    --cc=shuah@kernel.org \
    --cc=song@kernel.org \
    --cc=tamird@kernel.org \
    --cc=void@manifault.com \
    --cc=xukuohai@huawei.com \
    --cc=yonghong.song@linux.dev \
    --cc=zeffron@riotgames.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox