From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Ahern Subject: Re: [RFC v2 bpf-next 8/9] bpf: Provide helper to do lookups in kernel FIB table Date: Wed, 2 May 2018 09:37:21 -0600 Message-ID: <4a92f04a-6865-ae70-1c54-e4b92d886491@gmail.com> References: <20180429180752.15428-1-dsahern@gmail.com> <20180429180752.15428-9-dsahern@gmail.com> <20180502132736.3560fcac@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Cc: netdev@vger.kernel.org, borkmann@iogearbox.net, ast@kernel.org, davem@davemloft.net, shm@cumulusnetworks.com, roopa@cumulusnetworks.com, toke@toke.dk, john.fastabend@gmail.com, Vincent Bernat To: Jesper Dangaard Brouer Return-path: Received: from mail-pf0-f193.google.com ([209.85.192.193]:42024 "EHLO mail-pf0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751576AbeEBPhY (ORCPT ); Wed, 2 May 2018 11:37:24 -0400 Received: by mail-pf0-f193.google.com with SMTP id p14so726253pfh.9 for ; Wed, 02 May 2018 08:37:23 -0700 (PDT) In-Reply-To: <20180502132736.3560fcac@redhat.com> Content-Language: en-US Sender: netdev-owner@vger.kernel.org List-ID: On 5/2/18 5:27 AM, Jesper Dangaard Brouer wrote: > On Sun, 29 Apr 2018 11:07:51 -0700 > David Ahern wrote: > >> Initial performance numbers collected by Jesper, forwarded packets/sec: >> >> Full stack XDP FIB lookup XDP Direct lookup >> IPv4 1,947,969 7,074,156 7,415,333 >> IPv6 1,728,000 6,165,504 7,262,720 > > Do notice these number is single CPU core forwarding performance! > On a Broadwell E5-1650 v4 @ 3.60GHz. I'll add that context to the commit message. Thanks, > > Another interesting data point is that xdp_redirect_map performance is > 13,365,161 pps, which allow us to calculate/isolate the overhead/cost > of the FIB lookup. > > (1/13365161-1/7074156)*10^9 = -66.5 ns > (1/13365161-1/7415333)*10^9 = -60.0 ns > > Which is very close to the measured 50 ns cost of the FIB lookup, done > by Vincent Bernat. > See: https://vincent.bernat.im/en/blog/2017-ipv4-route-lookup-linux > > > > Another way I calculate this is by (ran a new benchmark): > > Performance: 7641593 (7,641,593) <= tx_unicast /sec > * Packet-gap: (1/7641593*10^9) = 130.86 ns > > Find all FIB related lookup functions in perf-report:: > > Samples: 93K of event 'cycles:ppp', Event count (approx.): 88553104553 > Overhead Cost CPU Command Symbol > 20.63 % 26.99 ns 002 ksoftirqd/2 [k] fib_table_lookup > 12.92 % 16.90 ns 002 ksoftirqd/2 [k] bpf_fib_lookup > 2.40 % 3.14 ns 002 ksoftirqd/2 [k] fib_select_path > 0.83 % 1.09 ns 002 ksoftirqd/2 [k] fib_get_table > 0.40 % 0.52 ns 002 ksoftirqd/2 [k] l3mdev_fib_table_rcu > ----------- > Tot:37.18 % (20.63+12.92+2.40+0.83+0.40) > =========== > > Cost of FIB lookup: > - 130.86/100*37.18 = 48.65 ns overhead by FIB lookup. > > Again very close to Vincent's IPv4 measurements of ~50 ns. > > > > Notice that the IPv6 measurements does not match up: > https://vincent.bernat.im/en/blog/2017-ipv6-route-lookup-linux > This is because, we/I'm just testing the IPv6 route cache here... > Vincent's blog is before recent changes -- 4.15 getting the rcu locking, net-next getting separate fib entries and now this set adding a FIB lookup without the dst. To share numbers from recent testing I did using Vincent's modules, lookup times in nsec (using local_clock) with MULTIPLE_TABLES config disabled for IPv4 and IPv6 IPv4 IPv6-dst IPv6-fib6 baseline 49 126 52 I have other cases with combinations of configs and rules, but this shows the best possible case. IPv6 needs some more work to improve speeds with MULTIPLE_TABLES enabled (separate local and main tables unlike IPv4) and IPV6_SUBTREES enabled.