[PATCH bpf v3 0/2] bpf, x86: fix per-CPU address resolution into an extended register

[Vineet Gupta <vineet.gupta@linux.dev>]

The JIT resolves a per-CPU address with

  add <dst>, gs:[this_cpu_off]

but builds the REX prefix with add_1mod(), which sets REX.B. The
destination is encoded in ModRM.reg, which REX.R extends, and the memory
operand is disp32 with no base, so REX.B does nothing and the high
register bit is dropped. Every extended destination therefore resolves
into whichever register shares the low three bits:

  R5 -> RAX    R7 -> RBP    R8 -> RSI    R9 -> RDI

The address is left unadjusted and an unrelated register is clobbered.
Patch 1 switches to add_2mod() so the bit goes through REX.R.

Clang reloads the address into R1 before each per-CPU access, so the
destination is never an extended register and the bug has been dormant
since v6.10. GCC keeps several per-CPU addresses live at once, which is
how it turned up: test_progs-bpf_gcc panics the kernel in
global_percpu_data/init, with the address of a .percpu variable in R5.

Patch 2 covers every register. A functional test only catches this if
the address happens to land in an extended register, so the test matches
the JITed add instead.

Changes in v3:
- Fold the five per-register programs into one that loads every
  register, and drop the comment explaining the register choice
  (Eduard Zingerman).
- Move the percpu_data declaration inside the arch guard, so other
  targets no longer carry a .percpu section and an unused map (bpf-ci).
- Match the movabsq of each address as well as the add, so the matchers
  stay on consecutive lines and the pair is checked to use the same
  register.
- Restore the Reviewed-by on patch 1, dropped by mistake in v2.

Changes in v2:
- Add the selftest, patch 2/2 (Eduard Zingerman). It uses __jited()
  rather than __xlated(): the xlated stream is identical for every
  register, and the wrong prefix is only visible in the native encoding.
- No functional change to patch 1.

Vineet Gupta (2):
  bpf, x86: Fix per-CPU address resolution into an extended register
  selftests/bpf: Check per-CPU address resolution per register

 arch/x86/net/bpf_jit_comp.c                   |  2 +-
 .../selftests/bpf/prog_tests/verifier.c       |  2 +
 .../bpf/progs/verifier_percpu_addr.c          | 72 +++++++++++++++++++
 3 files changed, 75 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/progs/verifier_percpu_addr.c

-- 
2.53.0-Meta