What library to use ?

What library to use ?

[Eric Leblond]

Hello,

I've just started to work again on eBPF and XDP. My target it to work
on XDP support for Suricata (Daniel if you read me, yes finally ;)
Target is to be able to start Suricata with --xdp eth5 and get
everything setup by Suricata to get a working capture.

I've done one year ago an implementation of eBPF support in Suricata
using the library in tools/lib/bpf. One year later is using this
library the way to go or is there another library ?

Thanks in advance for your answer.

++
-- 
Eric Leblond <eric@regit.org>
Blog: https://home.regit.org/

Re: What library to use ?

[Daniel Borkmann]

On 08/20/2017 03:03 PM, Eric Leblond wrote:
[...]
> I've just started to work again on eBPF and XDP. My target it to work
> on XDP support for Suricata (Daniel if you read me, yes finally ;)
> Target is to be able to start Suricata with --xdp eth5 and get
> everything setup by Suricata to get a working capture.

Great, finally! ;)

> I've done one year ago an implementation of eBPF support in Suricata
> using the library in tools/lib/bpf. One year later is using this
> library the way to go or is there another library ?

Yep, the lib in tools/lib/bpf would be recommended (also used in
tools/testing/selftests/bpf/ for some of the networking selftests
these days, incl. XDP).

Anyway, patches welcome just in case. ;)

Re: What library to use ?

[Jesper Dangaard Brouer]

On Mon, 21 Aug 2017 00:48:24 +0200 Daniel Borkmann <daniel@iogearbox.net> wrote:

> On 08/20/2017 03:03 PM, Eric Leblond wrote:
> [...]
> > I've just started to work again on eBPF and XDP. My target it to work
> > on XDP support for Suricata (Daniel if you read me, yes finally ;)
> > Target is to be able to start Suricata with --xdp eth5 and get
> > everything setup by Suricata to get a working capture.  
> 
> Great, finally! ;)

This is really great to hear! I would very much like to cooperate in
this area.

I assume that the (currently) recommended interface for transferring
raw XDP packets to userspace is the perf ring buffer via
bpf_perf_event_output() interface?

I want to code-up some benchmarks to establish a baseline of
the expected performance that can be achieved via the perf ring buffer
interface.

Can someone point me to some eBPF+perf-ring example code / docs?

I have noticed that samples/bpf/trace_output_*.c [1][2] contains
something... but I'm hoping someone else have some examples?
 [1] https://github.com/torvalds/linux/blob/master/samples/bpf/trace_output_kern.c
 [2] https://github.com/torvalds/linux/blob/master/samples/bpf/trace_output_user.c


> > I've done one year ago an implementation of eBPF support in Suricata
> > using the library in tools/lib/bpf. One year later is using this
> > library the way to go or is there another library ?  
> 
> Yep, the lib in tools/lib/bpf would be recommended (also used in
> tools/testing/selftests/bpf/ for some of the networking selftests
> these days, incl. XDP).
> 
> Anyway, patches welcome just in case. ;)

I've been baseing my examples[3] on samples/bpf/bpf_load.c, but I would
very much like to move away from this approach, and instead use
tools/lib/bpf/.  Maybe we can do a joined effort and bring
tools/lib/bpf/ into shape?

[3] https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: What library to use ?

[Daniel Borkmann]

On 08/21/2017 10:16 AM, Jesper Dangaard Brouer wrote:
> On Mon, 21 Aug 2017 00:48:24 +0200 Daniel Borkmann <daniel@iogearbox.net> wrote:
>> On 08/20/2017 03:03 PM, Eric Leblond wrote:
>> [...]
>>> I've just started to work again on eBPF and XDP. My target it to work
>>> on XDP support for Suricata (Daniel if you read me, yes finally ;)
>>> Target is to be able to start Suricata with --xdp eth5 and get
>>> everything setup by Suricata to get a working capture.
>>
>> Great, finally! ;)
>
> This is really great to hear! I would very much like to cooperate in
> this area.
>
> I assume that the (currently) recommended interface for transferring
> raw XDP packets to userspace is the perf ring buffer via
> bpf_perf_event_output() interface?

Yep, allows for meta data plus partial or full packet, e.g. see
cilium bpf/lib/drop.h +40 as an example. XDP works the same way.

> I want to code-up some benchmarks to establish a baseline of
> the expected performance that can be achieved via the perf ring buffer
> interface.

That would be great, there's likely room for optimization as
well! ;) Note struct perf_event_attr has couple of wakeup watermark
options, see perf_event_open(2). The sample code lets poll time
out to trigger head/tail check btw.

> Can someone point me to some eBPF+perf-ring example code / docs?
>
> I have noticed that samples/bpf/trace_output_*.c [1][2] contains
> something... but I'm hoping someone else have some examples?
>   [1] https://github.com/torvalds/linux/blob/master/samples/bpf/trace_output_kern.c
>   [2] https://github.com/torvalds/linux/blob/master/samples/bpf/trace_output_user.c

Interface from user space side is effectively the same as
trace_output_user.c, you'd need per cpu pmu fds (the example
above is just for cpu 0), and to pin the processing threads
accordingly to the corresponding cpu. fds go into perf event
map with index : cpu mapping, so you can use BPF_F_CURRENT_CPU
flag from helper side.

>>> I've done one year ago an implementation of eBPF support in Suricata
>>> using the library in tools/lib/bpf. One year later is using this
>>> library the way to go or is there another library ?
>>
>> Yep, the lib in tools/lib/bpf would be recommended (also used in
>> tools/testing/selftests/bpf/ for some of the networking selftests
>> these days, incl. XDP).
>>
>> Anyway, patches welcome just in case. ;)
>
> I've been baseing my examples[3] on samples/bpf/bpf_load.c, but I would
> very much like to move away from this approach, and instead use
> tools/lib/bpf/.

+1, they should be migrated to selftests ideally, so they are
run on regular basis.

Re: What library to use ?

[Eric Leblond]

Hi,

On Mon, 2017-08-21 at 23:57 +0200, Daniel Borkmann wrote:
> On 08/21/2017 10:16 AM, Jesper Dangaard Brouer wrote:
> > On Mon, 21 Aug 2017 00:48:24 +0200 Daniel Borkmann <daniel@iogearbo
> > x.net> wrote:
> > > On 08/20/2017 03:03 PM, Eric Leblond wrote:
> > > [...]
> > > > I've just started to work again on eBPF and XDP. My target it
> > > > to work
> > > > on XDP support for Suricata (Daniel if you read me, yes finally
> > > > ;)
> > > > Target is to be able to start Suricata with --xdp eth5 and get
> > > > everything setup by Suricata to get a working capture.
> > > 
> > > Great, finally! ;)
> > 
> > This is really great to hear! I would very much like to cooperate
> > in
> > this area.

I think I can appreciate some help here :)

> > I assume that the (currently) recommended interface for
> > transferring
> > raw XDP packets to userspace is the perf ring buffer via
> > bpf_perf_event_output() interface?
> 
> Yep, allows for meta data plus partial or full packet, e.g. see
> cilium bpf/lib/drop.h +40 as an example. XDP works the same way.
> 
> > I want to code-up some benchmarks to establish a baseline of
> > the expected performance that can be achieved via the perf ring
> > buffer
> > interface.
> 
> That would be great, there's likely room for optimization as
> well! ;) Note struct perf_event_attr has couple of wakeup watermark
> options, see perf_event_open(2). The sample code lets poll time
> out to trigger head/tail check btw.
> 
> > Can someone point me to some eBPF+perf-ring example code / docs?
> > 
> > I have noticed that samples/bpf/trace_output_*.c [1][2] contains
> > something... but I'm hoping someone else have some examples?
> >   [1] https://github.com/torvalds/linux/blob/master/samples/bpf/tra
> > ce_output_kern.c
> >   [2] https://github.com/torvalds/linux/blob/master/samples/bpf/tra
> > ce_output_user.c
> 
> Interface from user space side is effectively the same as
> trace_output_user.c, you'd need per cpu pmu fds (the example
> above is just for cpu 0), and to pin the processing threads
> accordingly to the corresponding cpu. fds go into perf event
> map with index : cpu mapping, so you can use BPF_F_CURRENT_CPU
> flag from helper side.

OK, this looks like what we were already doing in Suricata so it should
be ok. If I get correctly the design, we will have a per CPU load
balancing. The CPU reading the packet will send data to his own ring
buffer via the bpf_perf_event_output that don't take any CPU related
parameters. As we are really early in the processing, this means that
the per-CPU load balancing will be done by the card.
So we will encounter the asymetric flow hash problem on driver like
ixgbe which do not have a symetric load balancing function.

Thus we need another card to do the testing. I had one test bed ready
with an ixgbe. It looks like I will need some other hardware to do the
tests. 

Did I understood correctly ?

++
-- 
Eric Leblond <eric@regit.org>
Blog: https://home.regit.org/

Re: What library to use ?

[Daniel Borkmann]

On 08/25/2017 08:35 PM, Eric Leblond wrote:
[...]
> OK, this looks like what we were already doing in Suricata so it should
> be ok. If I get correctly the design, we will have a per CPU load
> balancing. The CPU reading the packet will send data to his own ring
> buffer via the bpf_perf_event_output that don't take any CPU related
> parameters. As we are really early in the processing, this means that

Yeah, if you look at __bpf_perf_event_output(), it's basically
the event->oncpu != cpu which would otherwise let it bail out,
but needed iiuc to ensure the RB can be written w/o having to
take locks. The CPU related 'parameter' is basically set up by
the 'orchestrator'. You have the perf event map, and given at
which index you place the corresponding perf fd, you can either
use BPF_F_CURRENT_CPU if the mapping is 1:1 (cpu -> perf fd set
up for this cpu) or a custom index if you have a use case where
you need to demux to one of multiple perf RBs for that CPU.

> the per-CPU load balancing will be done by the card.

Right given you need to have the replies steered into the same
perf RB 'channel' for further processing.