libbpf ringbuf manager starvation

libbpf ringbuf manager starvation

[Gilad Reti]

Hello there,

When playing with the (relatively) new ringbuf api we encountered
something that we believe can be an interesting usecase.
When registering multiple rinbufs to the same ringbuf manager, one of
which is highly active, other ringbufs may starve. Since libbpf
(e)polls on all the managed ringbufs at once and then tries to read
*as many samples as it can* from ready ringbufs, it may get stuck
indefinitely on one of them, not being able to process the other.
We know that the current ringbuf api exposes the epoll_fd so that one
can implement the epoll logic on his own, but this sounds to us like a
not so advanced usecase that may be worth taking care of specifically.
Does allowing to specify a maximum number of samples to consume sounds
like a reasonable addition to the ringbuf api?

Thanks

Re: libbpf ringbuf manager starvation

[Andrii Nakryiko]

On Tue, Jan 19, 2021 at 7:51 AM Gilad Reti <gilad.reti@gmail.com> wrote:
>
> Hello there,
>

Hi,

> When playing with the (relatively) new ringbuf api we encountered
> something that we believe can be an interesting usecase.
> When registering multiple rinbufs to the same ringbuf manager, one of
> which is highly active, other ringbufs may starve. Since libbpf
> (e)polls on all the managed ringbufs at once and then tries to read
> *as many samples as it can* from ready ringbufs, it may get stuck
> indefinitely on one of them, not being able to process the other.
> We know that the current ringbuf api exposes the epoll_fd so that one
> can implement the epoll logic on his own, but this sounds to us like a
> not so advanced usecase that may be worth taking care of specifically.
> Does allowing to specify a maximum number of samples to consume sounds
> like a reasonable addition to the ringbuf api?

Did you actually run into such a situation in practice? If you have a
BPF program producing so much data so fast that user-space can't keep
up, then it sounds like a suboptimal use case for BPF ringbuf.

But nevertheless, my advice for you situation is to use two instances
of libbpf's ring_buffer: one for super-busy ringbuf, and another for
everything else. Or you can even have one for each. It's very
flexible.

As for having this limit, it's not so simple, unfortunately. The
contract between kernel, epoll, and libbpf is that user-space will
always consume all the items until it runs out of more items to
consume. Internally in kernel BPF ringbuf relies on that to skip
unnecessary epoll notifications. If you consume not all items and will
attempt to (e)poll again, you'll never get another notification
(unless you force-notify from your BPF program, that's an advanced use
case).

We could do a round-robin across all registered ringbufs within the
ring_buffer instance in ring_buffer__poll()/ring_buffer__consume(),
but I think it's over-designing for a quite unusual case.


>
> Thanks

Re: libbpf ringbuf manager starvation

[Gilad Reti]

On Thu, Jan 21, 2021 at 9:29 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Tue, Jan 19, 2021 at 7:51 AM Gilad Reti <gilad.reti@gmail.com> wrote:
> >
> > Hello there,
> >
>
> Hi,
>
> > When playing with the (relatively) new ringbuf api we encountered
> > something that we believe can be an interesting usecase.
> > When registering multiple rinbufs to the same ringbuf manager, one of
> > which is highly active, other ringbufs may starve. Since libbpf
> > (e)polls on all the managed ringbufs at once and then tries to read
> > *as many samples as it can* from ready ringbufs, it may get stuck
> > indefinitely on one of them, not being able to process the other.
> > We know that the current ringbuf api exposes the epoll_fd so that one
> > can implement the epoll logic on his own, but this sounds to us like a
> > not so advanced usecase that may be worth taking care of specifically.
> > Does allowing to specify a maximum number of samples to consume sounds
> > like a reasonable addition to the ringbuf api?
>
> Did you actually run into such a situation in practice? If you have a
> BPF program producing so much data so fast that user-space can't keep
> up, then it sounds like a suboptimal use case for BPF ringbuf.

Yes, we have ran into such a situation. Our userspace is far from
performance-optimal, but currently that is the best we have.


>
> But nevertheless, my advice for you situation is to use two instances
> of libbpf's ring_buffer: one for super-busy ringbuf, and another for
> everything else. Or you can even have one for each. It's very
> flexible.
>

Yes, that what we are doing currently as a workaround. thanks.


> As for having this limit, it's not so simple, unfortunately. The
> contract between kernel, epoll, and libbpf is that user-space will
> always consume all the items until it runs out of more items to
> consume. Internally in kernel BPF ringbuf relies on that to skip
> unnecessary epoll notifications. If you consume not all items and will
> attempt to (e)poll again, you'll never get another notification
> (unless you force-notify from your BPF program, that's an advanced use
> case).
>
> We could do a round-robin across all registered ringbufs within the
> ring_buffer instance in ring_buffer__poll()/ring_buffer__consume(),
> but I think it's over-designing for a quite unusual case.
>

Yes, I agree it is not worth redesigning the entire ringbuf processing
implementation for this usecase. but we thought adding another parameter
will be simpler - thanks for clarifying the difficulties.

>
> >
> > Thanks