[PATCH net-next 0/6] bpf: inline bpf_map_lookup_elem()

[Alexei Starovoitov <ast@fb.com>]

bpf_map_lookup_elem() is one of the most frequently used helper functions.
Improve JITed program performance by inlining this helper.

bpf_map_type	before  after
hash		58M	74M
array		174M	280M

The values are number of lookups per second in ideal conditions
measured by micro-benchmark in patch 6.

The 'perf report' for HASH map type:
before:
    54.23%  map_perf_test  [kernel.kallsyms]  [k] __htab_map_lookup_elem
    14.24%  map_perf_test  [kernel.kallsyms]  [k] lookup_elem_raw
     8.84%  map_perf_test  [kernel.kallsyms]  [k] htab_map_lookup_elem
     5.93%  map_perf_test  [kernel.kallsyms]  [k] bpf_map_lookup_elem
     2.30%  map_perf_test  [kernel.kallsyms]  [k] bpf_prog_da4fc6a3f41761a2
     1.49%  map_perf_test  [kernel.kallsyms]  [k] kprobe_ftrace_handler

after:
    60.03%  map_perf_test  [kernel.kallsyms]  [k] __htab_map_lookup_elem
    18.07%  map_perf_test  [kernel.kallsyms]  [k] lookup_elem_raw
     2.91%  map_perf_test  [kernel.kallsyms]  [k] bpf_prog_da4fc6a3f41761a2
     1.94%  map_perf_test  [kernel.kallsyms]  [k] _einittext
     1.90%  map_perf_test  [kernel.kallsyms]  [k] __audit_syscall_exit
     1.72%  map_perf_test  [kernel.kallsyms]  [k] kprobe_ftrace_handler

so the cost of htab_map_lookup_elem() and bpf_map_lookup_elem()
is gone after inlining.

'per-cpu' and 'lru' map types can be optimized similarly in the future.

Note the sparse will complain that bpf is addictive ;)
kernel/bpf/hashtab.c:438:19: sparse: subtraction of functions? Share your drugs
kernel/bpf/verifier.c:3342:38: sparse: subtraction of functions? Share your drugs
it's not a new warning, just in new places.

Alexei Starovoitov (6):
  bpf: move fixup_bpf_calls() function
  bpf: refactor fixup_bpf_calls()
  bpf: adjust insn_aux_data when patching insns
  bpf: add helper inlining infra and optimize map_array lookup
  bpf: inline htab_map_lookup_elem()
  samples/bpf: add map_lookup microbenchmark

 include/linux/bpf.h              |   1 +
 include/linux/bpf_verifier.h     |   5 +-
 include/linux/filter.h           |  10 +++
 kernel/bpf/arraymap.c            |  29 +++++++++
 kernel/bpf/hashtab.c             |  31 +++++++++-
 kernel/bpf/syscall.c             |  56 -----------------
 kernel/bpf/verifier.c            | 129 ++++++++++++++++++++++++++++++++++++---
 samples/bpf/map_perf_test_kern.c |  33 ++++++++++
 samples/bpf/map_perf_test_user.c |  32 ++++++++++
 9 files changed, 261 insertions(+), 65 deletions(-)

-- 
2.8.0