[PATCH bpf-next 00/12] bpf: Support 16-byte return values in the R0:R2 register pair

[Yonghong Song <yonghong.song@linux.dev>]

Until now a BPF subprogram or kfunc could only return a scalar value of up to 8
bytes, passed back in R0. LLVM 23 added the ability to return a value in
two registers for the BPF target [1][2]: an __int128, or a struct/union
whose size is greater than 8 and up to 16 bytes, is returned in the
R0:R2 register pair, with the low 64 bits in R0 and the high 64 bits in
R2. (Before LLVM 23 the BPF backend could not return such values at all:
a by-value aggregate return was rejected with "aggregate returns are not
supported", and an __int128 return failed in the backend with "unable to
allocate function return #1".)

This series teaches the kernel about that convention so that a BPF-to-BPF
subprogram call and a kfunc call can return up to 16 bytes. A returned
struct or union must be composed only of scalars (recursively); the bytes
are handed back as the raw contents of R0:R2. Otherwise allowing pointer
members would launder a pointer into a scalar and escape the verifier's
provenance and reference tracking. A value larger than 16 bytes remains
unsupported, as does a <=8 byte return, which continues to use R0 alone.

Support is built up incrementally so each step is reviewable on its own,
and the feature is only turned on end to end by the last enabling patch.

  - Patch 1 refactors the global-subprog return check so the per-register
    scalar validation can be reused for R2.
  - Patch 2 to Patch 7 are preparation patches for core verifier handling
    of R0:R2 returns, JIT support, precision backtracking, liveness
    analysis, and limitation on trampoline bpf programs.
  - Patch 8 relaxes btf_distill_func_proto() and btf_validate_return_type()
    to accept <=16 byte aggregates, enabling the feature end to end.
  - Patches 9-11 add selftests including C and inline asm tests.
  - Patch 12 documents the kfunc return-value convention in kfuncs.rst.

Tested on x86 and arm64. The patch set is built on top of the following commit:
    Merge branch 'verify-bpf-signed-loader-at-load-time'

    [1] https://github.com/llvm/llvm-project/pull/190894
    [2] https://github.com/llvm/llvm-project/pull/206876

Yonghong Song (12):
  bpf: Factor check_global_ret_scalar_reg() out of the global return
    check
  bpf: Add verifier support for 16-byte returns in R0:R2
  bpf: Wire up JIT support for 16-byte kfunc returns
  bpf: Track R2 of register-pair returns in precision backtracking
  bpf: Account R2 of register-pair returns in live register analysis
  bpf: Force JIT for programs using the R0:R2 register pair
  bpf: Reject >8 byte return values on return-reading trampoline paths
  bpf: Enable 16-byte aggregate return types
  selftests/bpf: Add C tests for 16-byte returns in R0:R2
  selftests/bpf: Add inline-asm and subprog tests for R0:R2 returns
  selftests/bpf: Cover tracing on >8 and <=16 byte return targets
  Documentation/bpf: Document 16-byte kfunc return values in R0:R2

 Documentation/bpf/kfuncs.rst                  |  36 ++
 arch/arm64/net/bpf_jit_comp.c                 |   5 +
 arch/riscv/net/bpf_jit_comp64.c               |   5 +
 arch/x86/net/bpf_jit_comp.c                   |  19 +
 include/linux/bpf.h                           |   1 +
 include/linux/bpf_verifier.h                  |   4 +
 include/linux/filter.h                        |   1 +
 kernel/bpf/backtrack.c                        |  32 +-
 kernel/bpf/bpf_struct_ops.c                   |  12 +
 kernel/bpf/btf.c                              |  14 +-
 kernel/bpf/core.c                             |   7 +-
 kernel/bpf/liveness.c                         |  13 +-
 kernel/bpf/verifier.c                         | 179 ++++++++-
 .../selftests/bpf/prog_tests/aggregate_ret.c  | 249 ++++++++++++
 .../selftests/bpf/progs/aggregate_ret_func.c  | 361 ++++++++++++++++++
 .../bpf/progs/aggregate_ret_int128_c.c        |  48 +++
 .../selftests/bpf/progs/aggregate_ret_kfunc.c |  81 ++++
 .../selftests/bpf/progs/aggregate_ret_run.c   | 160 ++++++++
 .../bpf/progs/aggregate_ret_struct_c.c        |  98 +++++
 .../selftests/bpf/progs/aggregate_ret_trace.c |  44 +++
 .../bpf/progs/aggregate_ret_union_c.c         |  51 +++
 .../selftests/bpf/progs/exceptions_fail.c     |   2 +-
 .../selftests/bpf/test_kmods/bpf_testmod.c    |  80 ++++
 .../selftests/bpf/test_kmods/bpf_testmod.h    |   5 +
 .../bpf/test_kmods/bpf_testmod_kfunc.h        |  38 ++
 25 files changed, 1517 insertions(+), 28 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/aggregate_ret.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_func.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_int128_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_run.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_struct_c.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_trace.c
 create mode 100644 tools/testing/selftests/bpf/progs/aggregate_ret_union_c.c

-- 
2.53.0-Meta