Re: bpf: Question about odd BPF verifier behaviour

[Matt Bobrowski <mattbobrowski@google.com>]

Hey Eduard!


On Wed, Feb 22, 2023 at 05:28:52PM +0200, Eduard Zingerman wrote:
> On Mon, 2023-02-20 at 22:35 +0000, Matt Bobrowski wrote:
> > Hello!
> > 
> > Whilst in the midst of testing a v5.19 to v6.1 kernel upgrade, we
> > happened to notice that one of our sleepable LSM based eBPF programs
> > was failing to load on the newer v6.1 kernel. Using the below trivial
> > eBPF program as our reproducer:
> > 
> > #include "vmlinux.h"
> > #include <bpf/bpf_helpers.h>
> > #include <bpf/bpf_tracing.h>
> > 
> > char LICENSE[] SEC("license") = "Dual BSD/GPL";
> > 
> > SEC("lsm.s/bprm_committed_creds")
> > int BPF_PROG(dbg, struct linux_binprm *bprm)
> > {
> > 	char buf[64] = {0};
> > 	bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
> > 	return 0;
> > }
> > 
> > The verifier emits the following error message when attempting to load
> > the above eBPF program:
> > 
> > -- BEGIN PROG LOAD LOG --
> > reg type unsupported for arg#0 function dbg#5
> > 0: R1=ctx(off=0,imm=0) R10=fp0
> > ; int BPF_PROG(dbg, struct linux_binprm *bprm)
> > 0: (79) r1 = *(u64 *)(r1 +0)
> > func 'bpf_lsm_bprm_committed_creds' arg0 has btf_id 137293 type STRUCT 'linux_binprm'
> > 1: R1_w=ptr_linux_binprm(off=0,imm=0)
> > 1: (b7) r2 = 0                        ; R2_w=0
> > ; char buf[64] = {0};
> > [...]
> > ; bpf_ima_file_hash(bprm->file, buf, 64);
> > 10: (79) r1 = *(u64 *)(r1 +64)        ; R1_w=ptr_file(off=0,imm=0)
> > 11: (bf) r2 = r10                     ; R2_w=fp0 R10=fp0
> > ; 
> > 12: (07) r2 += -64                    ; R2_w=fp-64
> > ; bpf_ima_file_hash(bprm->file, buf, 64);
> > 13: (b7) r3 = 64                      ; R3_w=64
> > 14: (85) call bpf_ima_file_hash#193
> > cannot access ptr member next with moff 0 in struct llist_node with off 0 size 1
> > R1 is of type file but file is expected
> > processed 15 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
> > -- END PROG LOAD LOG --
> > 
> > What particularly strikes out at me is the following 2 lines returned
> > in the error message:
> > 
> > cannot access ptr member next with moff 0 in struct llist_node with off 0 size 1
> > R1 is of type file but file is expected
> 
> Hi Matt,
> 
> I tried your program as a ./test_progs test using v6.1 kernel and
> don't see any error messages:
>
> VERIFIER LOG:
> =============
> func#0 @0
> reg type unsupported for arg#0 function dbg#5
> 0: R1=ctx(off=0,imm=0) R10=fp0
> ; int BPF_PROG(dbg, struct linux_binprm *bprm)
> 0: (79) r1 = *(u64 *)(r1 +0)
> func 'bpf_lsm_bprm_committed_creds' arg0 has btf_id 3061 type STRUCT 'linux_binprm'
> 1: R1_w=ptr_linux_binprm(off=0,imm=0)
> 1: (b7) r2 = 0                        ; R2_w=0
> ; char buf[64] = {0};
> 2: (7b) *(u64 *)(r10 -8) = r2
> last_idx 2 first_idx 0
> regs=4 stack=0 before 1: (b7) r2 = 0
> 3: R2_w=P0 R10=fp0 fp-8_w=00000000
> 3: (7b) *(u64 *)(r10 -16) = r2        ; R2_w=P0 R10=fp0 fp-16_w=00000000
> 4: (7b) *(u64 *)(r10 -24) = r2        ; R2_w=P0 R10=fp0 fp-24_w=00000000
> 5: (7b) *(u64 *)(r10 -32) = r2        ; R2_w=P0 R10=fp0 fp-32_w=00000000
> 6: (7b) *(u64 *)(r10 -40) = r2        ; R2_w=P0 R10=fp0 fp-40_w=00000000
> 7: (7b) *(u64 *)(r10 -48) = r2        ; R2_w=P0 R10=fp0 fp-48_w=00000000
> 8: (7b) *(u64 *)(r10 -56) = r2        ; R2_w=P0 R10=fp0 fp-56_w=00000000
> 9: (7b) *(u64 *)(r10 -64) = r2        ; R2_w=P0 R10=fp0 fp-64_w=00000000
> ; bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
> 10: (79) r1 = *(u64 *)(r1 +64)        ; R1_w=ptr_file(off=0,imm=0)
> 11: (bf) r2 = r10                     ; R2_w=fp0 R10=fp0
> ; 
> 12: (07) r2 += -64                    ; R2_w=fp-64
> ; bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
> 13: (b4) w3 = 64                      ; R3_w=64
> 14: (85) call bpf_ima_file_hash#193
> last_idx 14 first_idx 0
> regs=8 stack=0 before 13: (b4) w3 = 64
> 15: R0_w=scalar() fp-8_w=mmmmmmmm fp-16_w=mmmmmmmm fp-24_w=mmmmmmmm fp-32_w=mmmmmmmm fp-40_w=mmmmmmmm fp-48_w=mmmmmmmm fp-56_w=mmmmmmmm fp-64_w=mmmmmmmm
> ; int BPF_PROG(dbg, struct linux_binprm *bprm)
> 15: (b4) w0 = 0                       ; R0_w=0
> 16: (95) exit
> 
> I use the following revision: 830b3c68c1fb "Linux 6.1".
> (also works with current bpf-next master).
> 
> Could you please provide some details on how you compile/load the program?

Firstly, thanks for taking a peek at this! Secondly, I do apologies, I
should've provided some more detailed on how I'm reproducing this in
my initial email. Below you can find a transcript of how I'm
conducting my tests:

The source OS which things (kernel and BPF reproducer program) are
being built on:

 $ cat /etc/os-release 
 PRETTY_NAME="Debian GNU/Linux rodete"
 NAME="Debian GNU/Linux"
 VERSION_ID="rodete"
 VERSION="12 (rodete)"
 VERSION_CODENAME=rodete
 ID=debian

Building latest LLVM and Pahole from source:

 $ sudo apt install -y cmake

 $ cmake --version
 cmake version 3.25.1

 $ git clone https://github.com/llvm/llvm-project.git  && cd llvm-project && \
 mkdir build && \
 cd build && \
 cmake -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_BUILD_TYPE=Release -G "Unix Makefiles" ../llvm && \
 make -j $(nproc) && \
 sudo make install

 $ clang --version
 clang version 17.0.0 (https://github.com/llvm/llvm-project.git bc85cf1687435f28fb01b1aa5303317e6118490c)
 Target: x86_64-unknown-linux-gnu
 Thread model: posix
 InstalledDir: /usr/local/bin

 $ sudo apt install -y libdwarf-dev libdw-dev

 $ git clone git://git.kernel.org/pub/scm/devel/pahole/pahole.git && \
 cd pahole && \
 mkdir build && \
 cd build && \
 cmake -DCMAKE_INSTALL_PREFIX=/usr -D__LIB=lib .. && \
 make -j $(nproc) && \
 sudo make install

 $ pahole --version
 v1.25

Building a test kernel:

 $ git clone https://github.com/torvalds/linux.git && cd linux

 $ make defconfig && make kvm_guest.config

 $ scripts/config \
 -e BPF \
 -e BPF_SYSCALL \
 -e BPF_LSM \
 -e BPF_JIT \
 -e BPF_EVENTS \
 -e DEBUG_INFO \
 -e DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \
 -e DEBUG_INFO_BTF \
 -e DEBUG_INFO_BTF_MODULES \
 -e PAHOLE_HAS_SPLIT_BTF \
 -e FTRACE \
 -e DYNAMIC_FTRACE \
 -e FUNCTION_TRACER

 $ make olddefconfig

 $ make -j`nproc`

Building the BPF reproducer program:

 $ git clone https://github.com/libbpf/libbpf-bootstrap.git

 # Both libbpf and bpftool should be the latest versions here.
 $ git submodule update --init --recursive

 $ cd examples/c

 $ cat > fentry.bpf.c<<EOF
 #include "vmlinux.h"
 #include <bpf/bpf_helpers.h>
 #include <bpf/bpf_tracing.h>

 char LICENSE[] SEC("license") = "Dual BSD/GPL";

 SEC("lsm.s/bprm_committed_creds")
 int BPF_PROG(dbg, struct linux_binprm *bprm)
 {
 char buf[64] = {0};
 bpf_ima_file_hash(bprm->file, buf, sizeof(buf));
 return 0;
 }
 EOF

 $ make -j`nproc` fentry


At this point, I basically launch the built kernel using QEMU and push
the built 'fentry' BPF program to the VM and run it. At that point, I
face the BPF verifier issue.

LMK whether you need any more information.

/M