Re: [RFC v2 bpf-next 8/9] bpf: Provide helper to do lookups in kernel FIB table

[David Ahern <dsahern@gmail.com>]

On 5/2/18 5:27 AM, Jesper Dangaard Brouer wrote:
> On Sun, 29 Apr 2018 11:07:51 -0700
> David Ahern <dsahern@gmail.com> wrote:
> 
>> Initial performance numbers collected by Jesper, forwarded packets/sec:
>>
>>        Full stack    XDP FIB lookup    XDP Direct lookup
>> IPv4   1,947,969       7,074,156          7,415,333
>> IPv6   1,728,000       6,165,504          7,262,720
> 
> Do notice these number is single CPU core forwarding performance!
> On a Broadwell E5-1650 v4 @ 3.60GHz.

I'll add that context to the commit message. Thanks,

> 
> Another interesting data point is that xdp_redirect_map performance is
> 13,365,161 pps, which allow us to calculate/isolate the overhead/cost
> of the FIB lookup.
> 
> (1/13365161-1/7074156)*10^9 = -66.5 ns
> (1/13365161-1/7415333)*10^9 = -60.0 ns
> 
> Which is very close to the measured 50 ns cost of the FIB lookup, done
> by Vincent Bernat.
>   See: https://vincent.bernat.im/en/blog/2017-ipv4-route-lookup-linux
> 
> 
> 
> Another way I calculate this is by (ran a new benchmark):
> 
> Performance: 7641593 (7,641,593) <= tx_unicast /sec
>  * Packet-gap: (1/7641593*10^9)  = 130.86 ns
> 
> Find all FIB related lookup functions in perf-report::
> 
>  Samples: 93K of event 'cycles:ppp', Event count (approx.): 88553104553
>    Overhead   Cost      CPU  Command      Symbol
>      20.63 %  26.99 ns  002  ksoftirqd/2  [k] fib_table_lookup
>      12.92 %  16.90 ns  002  ksoftirqd/2  [k] bpf_fib_lookup
>       2.40 %   3.14 ns  002  ksoftirqd/2  [k] fib_select_path
>       0.83 %   1.09 ns  002  ksoftirqd/2  [k] fib_get_table
>       0.40 %   0.52 ns  002  ksoftirqd/2  [k] l3mdev_fib_table_rcu
>  -----------
>  Tot:37.18 % (20.63+12.92+2.40+0.83+0.40)
>  ===========
> 
> Cost of FIB lookup:
>  - 130.86/100*37.18 = 48.65 ns overhead by FIB lookup.
> 
> Again very close to Vincent's IPv4 measurements of ~50 ns.
> 
> 
> 
> Notice that the IPv6 measurements does not match up:
>  https://vincent.bernat.im/en/blog/2017-ipv6-route-lookup-linux
> This is because, we/I'm just testing the IPv6 route cache here...
> 

Vincent's blog is before recent changes -- 4.15 getting the rcu locking,
net-next getting separate fib entries and now this set adding a FIB
lookup without the dst.

To share numbers from recent testing I did using Vincent's modules,
lookup times in nsec (using local_clock) with MULTIPLE_TABLES config
disabled for IPv4 and IPv6

            IPv4    IPv6-dst    IPv6-fib6
baseline     49        126         52

I have other cases with combinations of configs and rules, but this
shows the best possible case.

IPv6 needs some more work to improve speeds with MULTIPLE_TABLES enabled
(separate local and main tables unlike IPv4) and IPV6_SUBTREES enabled.