libxdp with bpf_stats_enabled question

libxdp with bpf_stats_enabled question

[Brian G. Merrell]

Hi,

I've been playing around with the new xdp-loader to chain xdp programs
with kernel.bpf_stats_enabled set. This is what I see (from bpftool
prog):

22: xdp  name xdp_dispatcher  tag d51e469e988d81da  gpl run_time_ns 105086 run_cnt 24
        loaded_at 2020-12-15T07:52:27+0000  uid 0
        xlated 720B  jited 535B  memlock 4096B  map_ids 5
        btf_id 8
24: ext  name xdp_pass  tag 3b185187f1855c4c  gpl
        loaded_at 2020-12-15T07:52:27+0000  uid 0
        xlated 16B  jited 18B  memlock 4096B
        btf_id 9
26: ext  name xdp_drop  tag 57cd311f2e27366b  gpl
        loaded_at 2020-12-15T07:52:27+0000  uid 0
        xlated 16B  jited 18B  memlock 4096B
        btf_id 10

You'll notice that the run_time_ns and run_cnt stats are only populated
for the xdp_dispatcher. This is basically the same behavior we see today
with bpf_tail_call chaining, so it's not a regression or anything. It
would be really nice, though, if we could see those stats for each
individual xdp program in the chain. Does anyone know what it would take
for that to happen? Would it be more feasible for xdplib to optionally
report separate runtime stats itself for chained programs?

Thanks,
Brian

Re: libxdp with bpf_stats_enabled question

[Toke Høiland-Jørgensen]

"Brian G. Merrell" <brian.g.merrell@gmail.com> writes:

> Hi,
>
> I've been playing around with the new xdp-loader to chain xdp programs
> with kernel.bpf_stats_enabled set. This is what I see (from bpftool
> prog):
>
> 22: xdp  name xdp_dispatcher  tag d51e469e988d81da  gpl run_time_ns 105086 run_cnt 24
>         loaded_at 2020-12-15T07:52:27+0000  uid 0
>         xlated 720B  jited 535B  memlock 4096B  map_ids 5
>         btf_id 8
> 24: ext  name xdp_pass  tag 3b185187f1855c4c  gpl
>         loaded_at 2020-12-15T07:52:27+0000  uid 0
>         xlated 16B  jited 18B  memlock 4096B
>         btf_id 9
> 26: ext  name xdp_drop  tag 57cd311f2e27366b  gpl
>         loaded_at 2020-12-15T07:52:27+0000  uid 0
>         xlated 16B  jited 18B  memlock 4096B
>         btf_id 10
>
> You'll notice that the run_time_ns and run_cnt stats are only populated
> for the xdp_dispatcher. This is basically the same behavior we see today
> with bpf_tail_call chaining, so it's not a regression or anything. It
> would be really nice, though, if we could see those stats for each
> individual xdp program in the chain. Does anyone know what it would take
> for that to happen?

The stats are computed by the bits behind the static_branch_unlikely()
calls here:

https://elixir.bootlin.com/linux/latest/source/kernel/bpf/trampoline.c#L391

The calls to __bpf_prog_enter() and __bpf_prog_exit() is emitted in the
top-level dispatcher code that invokes a BPF program from whatever
kernel hook it runs at. Which is why it only shows up for the
dispatcher: From the kernel PoV that's the top-level BPF program being
run, the other programs are patched in as substitutions for function
calls inside the top-level BPF program.

That being said, the trampoline code that does the function-level
patching could conceivably add the same conditional stats update code
before and after each jump. Not sure how complicated it would be to add
this, but it would mean that stats would automatically appear for each
freplace program.

> Would it be more feasible for xdplib to optionally report separate
> runtime stats itself for chained programs?

The dispatcher program could certainly track this by just calling
bpf_ktime_get_ns() before and after each sub-program and sticking the
results in a map. It wouldn't be reported by the kernel and bpftool, but
libxdp and xdp-tools could expose it. The drawback with this approach is
that it would be tied to a particular dispatcher instance, so either
they would be reset whenever you load an additional program, or the
library has to copy over the old stats, which would probably not be
doable atomically. For stats that may not matter too much, but it's
something to consider.

-Toke