[PATCH bpf-next v1 0/6] Eliminate spill overhead for bpf_throw()

[Kumar Kartikeya Dwivedi <memxor@gmail.com>]

A program that may throw is an exception boundary: its prologue pushes
every callee-saved register the JIT could clobber, five on x86-64 and
ten on arm64, whether the program uses them or not, and the exception
callback pops them all from the boundary frame before it runs. Every
invocation of such a program pays the pushes and pops: 18 bytes of code
on x86-64 for a program that uses none of the registers, five stp/ldp
pairs and 80 bytes of stack on arm64.

This series moves that work to the throw path. Each JITed function
records where its prologue saved the caller's callee-saved registers,
and bpf_throw() enters through a native entry that captures the
callee-saved registers of the throw site. The unwinder walks from the
throw site outwards, starting from the captured values and taking in
each frame the values the frame saved of its caller's; past the main
frame they are the registers the program's caller expects back. The
exception callback receives them and loads them before it reuses the
main frame. The boundary prologue is then an ordinary prologue on x86-64
and arm64.

Architectures without a native entry, s390 and powerpc, keep their
boundary scheme unchanged. arm64 is compile-tested only.

Kumar Kartikeya Dwivedi (6):
  bpf, x86: Account for the r9 push in exception boundary program
  bpf: Record where JITed subprograms keep callee-saved registers
  bpf: Add bpf_throw_unwind() and a hook for bpf_throw() entry
  bpf, x86: Add a native bpf_throw() entry that captures R6 to R9
  bpf, arm64: Add a native bpf_throw() entry that captures R6 to R9
  bpf: Restore callee-saved registers from the frames being unwound

 arch/arm64/net/Makefile                       |   1 +
 arch/arm64/net/bpf_jit_comp.c                 | 151 ++++++++++--------
 arch/arm64/net/bpf_throw.S                    |  30 ++++
 arch/x86/net/Makefile                         |   1 +
 arch/x86/net/bpf_jit_comp.c                   | 134 +++++++++-------
 arch/x86/net/bpf_throw.S                      |  39 +++++
 include/linux/bpf.h                           |  17 +-
 include/linux/bpf_throw.h                     |  49 ++++++
 kernel/bpf/core.c                             |   9 ++
 kernel/bpf/fixups.c                           |   1 -
 kernel/bpf/helpers.c                          |  47 +++++-
 kernel/bpf/verifier.c                         |   5 +
 .../bpf/progs/verifier_private_stack.c        |  12 +-
 13 files changed, 356 insertions(+), 140 deletions(-)
 create mode 100644 arch/arm64/net/bpf_throw.S
 create mode 100644 arch/x86/net/bpf_throw.S
 create mode 100644 include/linux/bpf_throw.h


base-commit: b71326388a8a24674846c1d8d3b6dd831d9843a9
-- 
2.53.0