Re: Replicating kfunc_call_test kernel test on standalone bpf program (calling kernel function is not allowed)

[Jiri Olsa <olsajiri@gmail.com>]

On Wed, Sep 28, 2022 at 08:33:24PM -0500, Henrique Fingler wrote:
> Hi all,
> 
> I'm trying to replicate a bpf test in the kernel that calls a function
> defined in the kernel itself.
> Source code is here:
> https://github.com/torvalds/linux/blob/v5.15/tools/testing/selftests/bpf/progs/kfunc_call_test.c
> 
> I think I have all dependencies:
>  Running within a qemu VM (Ubuntu 18.04)
>  Kernel v 5.15 compiled from scratch with configs from
> tools/bpf/bpftool/feature.c
>  pahole v1.22 (1.24 has a reported bug that doesn't allow me to use it)
>  libbpf v1.0
>  Installed bpf tool from 5.15 kernel directory at `tools/bpf`
>  clang and llvm 15
> 
> The goal is to call `bpf_kfunc_call_test1`, which is defined in
> net/bpf/test_run.c.
> I have two BPF programs and neither works. The first one is as is from
> the kernel:
> 
> #include "vmlinux.h"
> #include <bpf/bpf_helpers.h>
> 
> extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
>                   __u32 c, __u64 d) __ksym;
> 
> SEC("classifier")
> int kfunc_call_test1(struct __sk_buff *skb)
> {
>     struct sock *sk = 0;
>     __u64 a;
>     a = bpf_kfunc_call_test1(sk, 1, 2, 3, 4);

hi,
IIUC you are passing 'sk' pointer defined on the stack, while
bpf_kfunc_call_test1 expects kernel pointer

the kernel selftest test takes it from the skb with:

        struct bpf_sock *sk = skb->sk;

>     bpf_printk("bpf_kfunc_call_test1:  %d.\n", a);
>     return a;
> }
> 
> 
> It is compiled with these commands:
> 
> bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
> clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I. -idirafter
> /usr/lib/llvm-15/lib/clang/15.0.2/include -idirafter
> /usr/local/include -idirafter /usr/include/x86_64-linux-gnu -idirafter
> /usr/include -c hello.bpf.c -o hello.bpf.o
> llvm-strip -g hello.bpf.o
> bpftool gen skeleton hello.bpf.o > hello.skel.h
> cc -g -Wall hello.skel.h hello.c /usr/lib64/libbpf.a   -lelf -lz -o hello
> 
> 
> The output is quite large, here is a gist:
> https://gist.github.com/hfingler/dc96af45d87004d0dc412e35be31709c.
> Mainly:
> 
> libbpf: extern (func ksym) 'bpf_kfunc_call_test1': resolved to kernel [104983]
> libbpf: prog 'kfunc_call_test1': BPF program load failed: Invalid argument
> ...
> kernel function bpf_kfunc_call_test1 args#0 expected pointer to STRUCT
> sock but R1 is not a pointer to btf_id
> processed 6 insns (limit 1000000) max_states_per_insn 0 total_states 0
> peak_states 0 mark_read 0
> -- END PROG LOAD LOG --
> libbpf: prog 'kfunc_call_test1': failed to load: -22
> libbpf: failed to load object 'hello_bpf'
> libbpf: failed to load BPF skeleton 'hello_bpf': -22
> Failed to load and verify BPF skeleton
> 
> 
> The other program is based off of minimal.c from libbpf-bootstrap.
> 
> #include "vmlinux.h"
> #include <bpf/bpf_helpers.h>
> 
> extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
>                   __u32 c, __u64 d) __ksym;
> 
> SEC("tp/raw_syscalls/sys_enter")
> int handle_tp(void *ctx)
> {
>     __u64 a;
>     a = bpf_kfunc_call_test1(0, 1, 2, 3, 4);

you can't call bpf_kfunc_call_test1 from tracepoint, it's registered for:

        ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_prog_test_kfunc_set);
        ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_prog_test_kfunc_set);
        ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_prog_test_kfunc_set);

check net/bpf/test_run.c

also you're passing wrong arguments

>     bpf_printk("bpf_kfunc_call_test1:  %d.\n", a);
>     return 0;
> }
> 
> 
> The output is a little different, gist:
> https://gist.github.com/hfingler/ac69e286f9e527dfd678ef2d768e757c
> mainly:
> 
> libbpf: extern (func ksym) 'bpf_kfunc_call_test1': resolved to kernel [104983]
> ...
> calling kernel function bpf_kfunc_call_test1 is not allowed
> libbpf: prog 'handle_tp': failed to load: -13
> libbpf: failed to load object 'hello_bpf'
> libbpf: failed to load BPF skeleton 'hello_bpf': -13
> Failed to load and verify BPF skeleton
> 
> 
> What could be the problem here? I'm mostly interested in the second
> program, so that I can use it on my own tracepoints and other places.
> 
> I'm aware of filtering in net/core/filter.c, but I can't find any
> reference to `bpf_kfunc` functions. In fact, I added this to filter.c,
> where both functions just return true (I'm not concerned about
> security, this is just research):
> 
> const struct bpf_verifier_ops my_verifier_ops = {
>     .check_kfunc_call   = export_the_world,
>     .is_valid_access    = accept_the_world,
> };
> 
> 
> I'm assuming something is not allowing this program to call it, maybe
> it's the section it's put in. The kernel test's SEC is `classifier`,
> which is
> defined at tools/lib/bpf/libbpf.c as  `BPF_PROG_SEC("classifier",
> BPF_PROG_TYPE_SCHED_CLS),`, while `tp/` is BPF_PROG_TYPE_TRACEPOINT.
> Is there a filter somewhere that allows one but not the other? For
> example, in kernel/bpf/syscall.c I see:
> 
> static bool is_net_admin_prog_type(enum bpf_prog_type prog_type)
> {
>     switch (prog_type) {
>     case BPF_PROG_TYPE_SCHED_CLS:
>         .. others
> 
> but BPF_PROG_TYPE_TRACEPOINT is not here.
> 
> There are so many references to these things that I'm totally lost,
> I'd appreciate some help.

Artem recently added simple kfunc call for kexec:
  133790596406 bpf: export crash_kexec() as destructive kfunc

might be easier way into kfuncs

jirka