Replicating kfunc_call_test kernel test on standalone bpf program (calling kernel function is not allowed)

Replicating kfunc_call_test kernel test on standalone bpf program (calling kernel function is not allowed)

[Henrique Fingler]

Hi all,

I'm trying to replicate a bpf test in the kernel that calls a function
defined in the kernel itself.
Source code is here:
https://github.com/torvalds/linux/blob/v5.15/tools/testing/selftests/bpf/progs/kfunc_call_test.c

I think I have all dependencies:
 Running within a qemu VM (Ubuntu 18.04)
 Kernel v 5.15 compiled from scratch with configs from
tools/bpf/bpftool/feature.c
 pahole v1.22 (1.24 has a reported bug that doesn't allow me to use it)
 libbpf v1.0
 Installed bpf tool from 5.15 kernel directory at `tools/bpf`
 clang and llvm 15

The goal is to call `bpf_kfunc_call_test1`, which is defined in
net/bpf/test_run.c.
I have two BPF programs and neither works. The first one is as is from
the kernel:

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
                  __u32 c, __u64 d) __ksym;

SEC("classifier")
int kfunc_call_test1(struct __sk_buff *skb)
{
    struct sock *sk = 0;
    __u64 a;
    a = bpf_kfunc_call_test1(sk, 1, 2, 3, 4);
    bpf_printk("bpf_kfunc_call_test1:  %d.\n", a);
    return a;
}


It is compiled with these commands:

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I. -idirafter
/usr/lib/llvm-15/lib/clang/15.0.2/include -idirafter
/usr/local/include -idirafter /usr/include/x86_64-linux-gnu -idirafter
/usr/include -c hello.bpf.c -o hello.bpf.o
llvm-strip -g hello.bpf.o
bpftool gen skeleton hello.bpf.o > hello.skel.h
cc -g -Wall hello.skel.h hello.c /usr/lib64/libbpf.a   -lelf -lz -o hello


The output is quite large, here is a gist:
https://gist.github.com/hfingler/dc96af45d87004d0dc412e35be31709c.
Mainly:

libbpf: extern (func ksym) 'bpf_kfunc_call_test1': resolved to kernel [104983]
libbpf: prog 'kfunc_call_test1': BPF program load failed: Invalid argument
...
kernel function bpf_kfunc_call_test1 args#0 expected pointer to STRUCT
sock but R1 is not a pointer to btf_id
processed 6 insns (limit 1000000) max_states_per_insn 0 total_states 0
peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'kfunc_call_test1': failed to load: -22
libbpf: failed to load object 'hello_bpf'
libbpf: failed to load BPF skeleton 'hello_bpf': -22
Failed to load and verify BPF skeleton


The other program is based off of minimal.c from libbpf-bootstrap.

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
                  __u32 c, __u64 d) __ksym;

SEC("tp/raw_syscalls/sys_enter")
int handle_tp(void *ctx)
{
    __u64 a;
    a = bpf_kfunc_call_test1(0, 1, 2, 3, 4);
    bpf_printk("bpf_kfunc_call_test1:  %d.\n", a);
    return 0;
}


The output is a little different, gist:
https://gist.github.com/hfingler/ac69e286f9e527dfd678ef2d768e757c
mainly:

libbpf: extern (func ksym) 'bpf_kfunc_call_test1': resolved to kernel [104983]
...
calling kernel function bpf_kfunc_call_test1 is not allowed
libbpf: prog 'handle_tp': failed to load: -13
libbpf: failed to load object 'hello_bpf'
libbpf: failed to load BPF skeleton 'hello_bpf': -13
Failed to load and verify BPF skeleton


What could be the problem here? I'm mostly interested in the second
program, so that I can use it on my own tracepoints and other places.

I'm aware of filtering in net/core/filter.c, but I can't find any
reference to `bpf_kfunc` functions. In fact, I added this to filter.c,
where both functions just return true (I'm not concerned about
security, this is just research):

const struct bpf_verifier_ops my_verifier_ops = {
    .check_kfunc_call   = export_the_world,
    .is_valid_access    = accept_the_world,
};


I'm assuming something is not allowing this program to call it, maybe
it's the section it's put in. The kernel test's SEC is `classifier`,
which is
defined at tools/lib/bpf/libbpf.c as  `BPF_PROG_SEC("classifier",
BPF_PROG_TYPE_SCHED_CLS),`, while `tp/` is BPF_PROG_TYPE_TRACEPOINT.
Is there a filter somewhere that allows one but not the other? For
example, in kernel/bpf/syscall.c I see:

static bool is_net_admin_prog_type(enum bpf_prog_type prog_type)
{
    switch (prog_type) {
    case BPF_PROG_TYPE_SCHED_CLS:
        .. others

but BPF_PROG_TYPE_TRACEPOINT is not here.

There are so many references to these things that I'm totally lost,
I'd appreciate some help.
Thanks!

Re: Replicating kfunc_call_test kernel test on standalone bpf program (calling kernel function is not allowed)

[Jiri Olsa]

On Wed, Sep 28, 2022 at 08:33:24PM -0500, Henrique Fingler wrote:
> Hi all,
> 
> I'm trying to replicate a bpf test in the kernel that calls a function
> defined in the kernel itself.
> Source code is here:
> https://github.com/torvalds/linux/blob/v5.15/tools/testing/selftests/bpf/progs/kfunc_call_test.c
> 
> I think I have all dependencies:
>  Running within a qemu VM (Ubuntu 18.04)
>  Kernel v 5.15 compiled from scratch with configs from
> tools/bpf/bpftool/feature.c
>  pahole v1.22 (1.24 has a reported bug that doesn't allow me to use it)
>  libbpf v1.0
>  Installed bpf tool from 5.15 kernel directory at `tools/bpf`
>  clang and llvm 15
> 
> The goal is to call `bpf_kfunc_call_test1`, which is defined in
> net/bpf/test_run.c.
> I have two BPF programs and neither works. The first one is as is from
> the kernel:
> 
> #include "vmlinux.h"
> #include <bpf/bpf_helpers.h>
> 
> extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
>                   __u32 c, __u64 d) __ksym;
> 
> SEC("classifier")
> int kfunc_call_test1(struct __sk_buff *skb)
> {
>     struct sock *sk = 0;
>     __u64 a;
>     a = bpf_kfunc_call_test1(sk, 1, 2, 3, 4);

hi,
IIUC you are passing 'sk' pointer defined on the stack, while
bpf_kfunc_call_test1 expects kernel pointer

the kernel selftest test takes it from the skb with:

        struct bpf_sock *sk = skb->sk;

>     bpf_printk("bpf_kfunc_call_test1:  %d.\n", a);
>     return a;
> }
> 
> 
> It is compiled with these commands:
> 
> bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
> clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I. -idirafter
> /usr/lib/llvm-15/lib/clang/15.0.2/include -idirafter
> /usr/local/include -idirafter /usr/include/x86_64-linux-gnu -idirafter
> /usr/include -c hello.bpf.c -o hello.bpf.o
> llvm-strip -g hello.bpf.o
> bpftool gen skeleton hello.bpf.o > hello.skel.h
> cc -g -Wall hello.skel.h hello.c /usr/lib64/libbpf.a   -lelf -lz -o hello
> 
> 
> The output is quite large, here is a gist:
> https://gist.github.com/hfingler/dc96af45d87004d0dc412e35be31709c.
> Mainly:
> 
> libbpf: extern (func ksym) 'bpf_kfunc_call_test1': resolved to kernel [104983]
> libbpf: prog 'kfunc_call_test1': BPF program load failed: Invalid argument
> ...
> kernel function bpf_kfunc_call_test1 args#0 expected pointer to STRUCT
> sock but R1 is not a pointer to btf_id
> processed 6 insns (limit 1000000) max_states_per_insn 0 total_states 0
> peak_states 0 mark_read 0
> -- END PROG LOAD LOG --
> libbpf: prog 'kfunc_call_test1': failed to load: -22
> libbpf: failed to load object 'hello_bpf'
> libbpf: failed to load BPF skeleton 'hello_bpf': -22
> Failed to load and verify BPF skeleton
> 
> 
> The other program is based off of minimal.c from libbpf-bootstrap.
> 
> #include "vmlinux.h"
> #include <bpf/bpf_helpers.h>
> 
> extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
>                   __u32 c, __u64 d) __ksym;
> 
> SEC("tp/raw_syscalls/sys_enter")
> int handle_tp(void *ctx)
> {
>     __u64 a;
>     a = bpf_kfunc_call_test1(0, 1, 2, 3, 4);

you can't call bpf_kfunc_call_test1 from tracepoint, it's registered for:

        ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_prog_test_kfunc_set);
        ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_prog_test_kfunc_set);
        ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_prog_test_kfunc_set);

check net/bpf/test_run.c

also you're passing wrong arguments

>     bpf_printk("bpf_kfunc_call_test1:  %d.\n", a);
>     return 0;
> }
> 
> 
> The output is a little different, gist:
> https://gist.github.com/hfingler/ac69e286f9e527dfd678ef2d768e757c
> mainly:
> 
> libbpf: extern (func ksym) 'bpf_kfunc_call_test1': resolved to kernel [104983]
> ...
> calling kernel function bpf_kfunc_call_test1 is not allowed
> libbpf: prog 'handle_tp': failed to load: -13
> libbpf: failed to load object 'hello_bpf'
> libbpf: failed to load BPF skeleton 'hello_bpf': -13
> Failed to load and verify BPF skeleton
> 
> 
> What could be the problem here? I'm mostly interested in the second
> program, so that I can use it on my own tracepoints and other places.
> 
> I'm aware of filtering in net/core/filter.c, but I can't find any
> reference to `bpf_kfunc` functions. In fact, I added this to filter.c,
> where both functions just return true (I'm not concerned about
> security, this is just research):
> 
> const struct bpf_verifier_ops my_verifier_ops = {
>     .check_kfunc_call   = export_the_world,
>     .is_valid_access    = accept_the_world,
> };
> 
> 
> I'm assuming something is not allowing this program to call it, maybe
> it's the section it's put in. The kernel test's SEC is `classifier`,
> which is
> defined at tools/lib/bpf/libbpf.c as  `BPF_PROG_SEC("classifier",
> BPF_PROG_TYPE_SCHED_CLS),`, while `tp/` is BPF_PROG_TYPE_TRACEPOINT.
> Is there a filter somewhere that allows one but not the other? For
> example, in kernel/bpf/syscall.c I see:
> 
> static bool is_net_admin_prog_type(enum bpf_prog_type prog_type)
> {
>     switch (prog_type) {
>     case BPF_PROG_TYPE_SCHED_CLS:
>         .. others
> 
> but BPF_PROG_TYPE_TRACEPOINT is not here.
> 
> There are so many references to these things that I'm totally lost,
> I'd appreciate some help.

Artem recently added simple kfunc call for kexec:
  133790596406 bpf: export crash_kexec() as destructive kfunc

might be easier way into kfuncs

jirka

Re: Replicating kfunc_call_test kernel test on standalone bpf program (calling kernel function is not allowed)

[Henrique Fingler]

> > Hi all,
> >
> > I'm trying to replicate a bpf test in the kernel that calls a function
> > defined in the kernel itself.
> > Source code is here:
> > https://github.com/torvalds/linux/blob/v5.15/tools/testing/selftests/bpf/progs/kfunc_call_test.c
> >
> > I think I have all dependencies:
> >  Running within a qemu VM (Ubuntu 18.04)
> >  Kernel v 5.15 compiled from scratch with configs from
> > tools/bpf/bpftool/feature.c
> >  pahole v1.22 (1.24 has a reported bug that doesn't allow me to use it)
> >  libbpf v1.0
> >  Installed bpf tool from 5.15 kernel directory at `tools/bpf`
> >  clang and llvm 15
> >
> > The goal is to call `bpf_kfunc_call_test1`, which is defined in
> > net/bpf/test_run.c.
> > I have two BPF programs and neither works. The first one is as is from
> > the kernel:
> >
> > #include "vmlinux.h"
> > #include <bpf/bpf_helpers.h>
> >
> > extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
> >                   __u32 c, __u64 d) __ksym;
> >
> > SEC("classifier")
> > int kfunc_call_test1(struct __sk_buff *skb)
> > {
> >     struct sock *sk = 0;
> >     __u64 a;
> >     a = bpf_kfunc_call_test1(sk, 1, 2, 3, 4);
>
> hi,
> IIUC you are passing 'sk' pointer defined on the stack, while
> bpf_kfunc_call_test1 expects kernel pointer
>
> the kernel selftest test takes it from the skb with:
>
>         struct bpf_sock *sk = skb->sk;

I see. So even if the kernel function (bpf_kfunc_call_test1) does not
use the argument, bpf is checking if it's a kernel pointer? Is the bpf
compiler doing this check?
I assumed that passing a constant 0 pointer would work since the other
parameters are just constants, even in the kernel test.
After changing the first program to the original code, the error
changed, so that's progress. Now it says, even when running with root
permissions:
"libbpf: prog 'kfunc_call_test1': BPF program load failed: Permission denied"
Would be interesting to know why, but not necessary since I won't use
it this way.

> >     bpf_printk("bpf_kfunc_call_test1:  %d.\n", a);
> >     return a;
> > }
> >
> >
> > It is compiled with these commands:
> >
> > bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
> > clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I. -idirafter
> > /usr/lib/llvm-15/lib/clang/15.0.2/include -idirafter
> > /usr/local/include -idirafter /usr/include/x86_64-linux-gnu -idirafter
> > /usr/include -c hello.bpf.c -o hello.bpf.o
> > llvm-strip -g hello.bpf.o
> > bpftool gen skeleton hello.bpf.o > hello.skel.h
> > cc -g -Wall hello.skel.h hello.c /usr/lib64/libbpf.a   -lelf -lz -o hello
> >
> >
> > The output is quite large, here is a gist:
> > https://gist.github.com/hfingler/dc96af45d87004d0dc412e35be31709c.
> > Mainly:
> >
> > libbpf: extern (func ksym) 'bpf_kfunc_call_test1': resolved to kernel [104983]
> > libbpf: prog 'kfunc_call_test1': BPF program load failed: Invalid argument
> > ...
> > kernel function bpf_kfunc_call_test1 args#0 expected pointer to STRUCT
> > sock but R1 is not a pointer to btf_id
> > processed 6 insns (limit 1000000) max_states_per_insn 0 total_states 0
> > peak_states 0 mark_read 0
> > -- END PROG LOAD LOG --
> > libbpf: prog 'kfunc_call_test1': failed to load: -22
> > libbpf: failed to load object 'hello_bpf'
> > libbpf: failed to load BPF skeleton 'hello_bpf': -22
> > Failed to load and verify BPF skeleton
> >
> >
> > The other program is based off of minimal.c from libbpf-bootstrap.
> >
> > #include "vmlinux.h"
> > #include <bpf/bpf_helpers.h>
> >
> > extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
> >                   __u32 c, __u64 d) __ksym;
> >
> > SEC("tp/raw_syscalls/sys_enter")
> > int handle_tp(void *ctx)
> > {
> >     __u64 a;
> >     a = bpf_kfunc_call_test1(0, 1, 2, 3, 4);
>
> you can't call bpf_kfunc_call_test1 from tracepoint, it's registered for:
>
>         ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_prog_test_kfunc_set);
>         ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_prog_test_kfunc_set);
>         ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_prog_test_kfunc_set);
>
> check net/bpf/test_run.c
>
> also you're passing wrong arguments

So there is some filtering happening. I'm on version 5.15 so I don't
see that code, but I do see it on master.
I'll try to figure out what is the matching code in 5.15 (or most
likely just go to 5.19) and perhaps add BPF_PROG_TYPE_TRACEPOINT or
whatever other PROG_TYPE I end up using.

I'm assuming the wrong argument here is the same as above, that the
argument must be a pointer in kernel space.
I'll create a new bpf_kfunc_call_test1 that doesn't have a sock*
argument, just ints and see if it works.

> >     bpf_printk("bpf_kfunc_call_test1:  %d.\n", a);
> >     return 0;
> > }
> >
> >
> > The output is a little different, gist:
> > https://gist.github.com/hfingler/ac69e286f9e527dfd678ef2d768e757c
> > mainly:
> >
> > libbpf: extern (func ksym) 'bpf_kfunc_call_test1': resolved to kernel [104983]
> > ...
> > calling kernel function bpf_kfunc_call_test1 is not allowed
> > libbpf: prog 'handle_tp': failed to load: -13
> > libbpf: failed to load object 'hello_bpf'
> > libbpf: failed to load BPF skeleton 'hello_bpf': -13
> > Failed to load and verify BPF skeleton
> >
> >
> > What could be the problem here? I'm mostly interested in the second
> > program, so that I can use it on my own tracepoints and other places.
> >
> > I'm aware of filtering in net/core/filter.c, but I can't find any
> > reference to `bpf_kfunc` functions. In fact, I added this to filter.c,
> > where both functions just return true (I'm not concerned about
> > security, this is just research):
> >
> > const struct bpf_verifier_ops my_verifier_ops = {
> >     .check_kfunc_call   = export_the_world,
> >     .is_valid_access    = accept_the_world,
> > };
> >
> >
> > I'm assuming something is not allowing this program to call it, maybe
> > it's the section it's put in. The kernel test's SEC is `classifier`,
> > which is
> > defined at tools/lib/bpf/libbpf.c as  `BPF_PROG_SEC("classifier",
> > BPF_PROG_TYPE_SCHED_CLS),`, while `tp/` is BPF_PROG_TYPE_TRACEPOINT.
> > Is there a filter somewhere that allows one but not the other? For
> > example, in kernel/bpf/syscall.c I see:
> >
> > static bool is_net_admin_prog_type(enum bpf_prog_type prog_type)
> > {
> >     switch (prog_type) {
> >     case BPF_PROG_TYPE_SCHED_CLS:
> >         .. others
> >
> > but BPF_PROG_TYPE_TRACEPOINT is not here.
> >
> > There are so many references to these things that I'm totally lost,
> > I'd appreciate some help.
>
> Artem recently added simple kfunc call for kexec:
>   133790596406 bpf: export crash_kexec() as destructive kfunc
>
> might be easier way into kfuncs
>
> jirka

Is creating a `struct bpf_verifier_ops` necessary? That patch you
mentioned does not do it, but perhaps it's somewhere else or it's
completely handled already.
I currently have one that just returns true for both
`check_kfunc_call` and `is_valid_access` but I'm not sure if it is
necessary or how the "macro magic" for these work.
The current kfunc documentation mentions we need to use
`register_btf_kfunc_id_set` explicitly, something that isn't in 5.15.
Is just registering the kfunc_id_set enough that I don't have to worry
about the verifier_ops? I think I will use just kprobes and kfuncs.

Thanks for the help!

Re: Replicating kfunc_call_test kernel test on standalone bpf program (calling kernel function is not allowed)

[Henrique Fingler]

> > > Hi all,
> > >
> > > I'm trying to replicate a bpf test in the kernel that calls a function
> > > defined in the kernel itself.
> > > Source code is here:
> > > https://github.com/torvalds/linux/blob/v5.15/tools/testing/selftests/bpf/progs/kfunc_call_test.c
> > >
> > > I think I have all dependencies:
> > >  Running within a qemu VM (Ubuntu 18.04)
> > >  Kernel v 5.15 compiled from scratch with configs from
> > > tools/bpf/bpftool/feature.c
> > >  pahole v1.22 (1.24 has a reported bug that doesn't allow me to use it)
> > >  libbpf v1.0
> > >  Installed bpf tool from 5.15 kernel directory at `tools/bpf`
> > >  clang and llvm 15
> > >
> > > The goal is to call `bpf_kfunc_call_test1`, which is defined in
> > > net/bpf/test_run.c.
> > > I have two BPF programs and neither works. The first one is as is from
> > > the kernel:
> > >
> > > #include "vmlinux.h"
> > > #include <bpf/bpf_helpers.h>
> > >
> > > extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
> > >                   __u32 c, __u64 d) __ksym;
> > >
> > > SEC("classifier")
> > > int kfunc_call_test1(struct __sk_buff *skb)
> > > {
> > >     struct sock *sk = 0;
> > >     __u64 a;
> > >     a = bpf_kfunc_call_test1(sk, 1, 2, 3, 4);
> >
> > hi,
> > IIUC you are passing 'sk' pointer defined on the stack, while
> > bpf_kfunc_call_test1 expects kernel pointer
> >
> > the kernel selftest test takes it from the skb with:
> >
> >         struct bpf_sock *sk = skb->sk;
>
> I see. So even if the kernel function (bpf_kfunc_call_test1) does not
> use the argument, bpf is checking if it's a kernel pointer? Is the bpf
> compiler doing this check?
> I assumed that passing a constant 0 pointer would work since the other
> parameters are just constants, even in the kernel test.
> After changing the first program to the original code, the error
> changed, so that's progress. Now it says, even when running with root
> permissions:
> "libbpf: prog 'kfunc_call_test1': BPF program load failed: Permission denied"
> Would be interesting to know why, but not necessary since I won't use
> it this way.

Following up on this, I have moved to kernel 5.19.12 and I'm trying to
make any kfunc work.
I changed net/bpf/test_run.c to allow for more prog types, like master
branch does, since originally it only had the first line for
BPF_PROG_TYPE_SCHED_CLS.

ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS,
&bpf_prog_test_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING,
&bpf_prog_test_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_KPROBE,
&bpf_prog_test_kfunc_set);
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACEPOINT,
&bpf_prog_test_kfunc_set);
return ret ?: register_btf_id_dtor_kfuncs(bpf_prog_test_dtor_kfunc,
                          ARRAY_SIZE(bpf_prog_test_dtor_kfunc),
                          THIS_MODULE);

I also added my own function to test it out and added it to the SET

u64 noinline bpf_kfunc_call_test4(u32 a, u64 b, u32 c, u64 d)
{
    return a + b + c + d;
}
//this is inside  BTF_SET_START(test_sk_check_kfunc_ids)
BTF_ID(func, bpf_kfunc_call_test4)

Now I'm spraying every BPF program I can find to call either function:

extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
__u32 c, __u64 d) __ksym;
extern __u64 bpf_kfunc_call_test4(__u32 a, __u64 b, __u32 c, __u64 d) __ksym;

Compiling works, and when I run it I get no errors, *except* Permission denied.
gist here: https://gist.github.com/hfingler/5c2c0b713299daa6b0ba07fa92ff29de

libbpf: prog 'kfunc_call_test1': BPF program load failed: Permission denied
...
calling kernel function bpf_kfunc_call_test1 is not allowed
-- END PROG LOAD LOG --
libbpf: prog 'kfunc_call_test1': failed to load: -13
libbpf: failed to load object 'hello_bpf'
libbpf: failed to load BPF skeleton 'hello_bpf': -13

I've tried a few programs and it would be too much to paste here, so
here's a gist with all of them, which I tried each separately, not all
at once:
https://gist.github.com/hfingler/eb544b23cc36d57b8e9723cd36fbf243
Basically, I've tried SEC("tc"),
SEC("tracepoint/syscalls/sys_enter_open"),
SEC("kprobe/__x64_sys_write")

What permission is being denied? I've tried running as root and I get
the same thing, is there bpf permission checking somewhere else in the
kernel? Do I have to have some sort of capability? Am I missing some
kernel config?
These are the configs I'm enabling in the kernel:
https://gist.github.com/hfingler/ed780bd52b751625f52bbb08eb853641

Re: Replicating kfunc_call_test kernel test on standalone bpf program (calling kernel function is not allowed)

[Jiri Olsa]

On Fri, Sep 30, 2022 at 05:03:30PM -0500, Henrique Fingler wrote:
> > > > Hi all,
> > > >
> > > > I'm trying to replicate a bpf test in the kernel that calls a function
> > > > defined in the kernel itself.
> > > > Source code is here:
> > > > https://github.com/torvalds/linux/blob/v5.15/tools/testing/selftests/bpf/progs/kfunc_call_test.c
> > > >
> > > > I think I have all dependencies:
> > > >  Running within a qemu VM (Ubuntu 18.04)
> > > >  Kernel v 5.15 compiled from scratch with configs from
> > > > tools/bpf/bpftool/feature.c
> > > >  pahole v1.22 (1.24 has a reported bug that doesn't allow me to use it)
> > > >  libbpf v1.0
> > > >  Installed bpf tool from 5.15 kernel directory at `tools/bpf`
> > > >  clang and llvm 15
> > > >
> > > > The goal is to call `bpf_kfunc_call_test1`, which is defined in
> > > > net/bpf/test_run.c.
> > > > I have two BPF programs and neither works. The first one is as is from
> > > > the kernel:
> > > >
> > > > #include "vmlinux.h"
> > > > #include <bpf/bpf_helpers.h>
> > > >
> > > > extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
> > > >                   __u32 c, __u64 d) __ksym;
> > > >
> > > > SEC("classifier")
> > > > int kfunc_call_test1(struct __sk_buff *skb)
> > > > {
> > > >     struct sock *sk = 0;
> > > >     __u64 a;
> > > >     a = bpf_kfunc_call_test1(sk, 1, 2, 3, 4);
> > >
> > > hi,
> > > IIUC you are passing 'sk' pointer defined on the stack, while
> > > bpf_kfunc_call_test1 expects kernel pointer
> > >
> > > the kernel selftest test takes it from the skb with:
> > >
> > >         struct bpf_sock *sk = skb->sk;
> >
> > I see. So even if the kernel function (bpf_kfunc_call_test1) does not
> > use the argument, bpf is checking if it's a kernel pointer? Is the bpf
> > compiler doing this check?

it's the verifier check.. that the function is called with proper
argument types/pointers

> > I assumed that passing a constant 0 pointer would work since the other
> > parameters are just constants, even in the kernel test.
> > After changing the first program to the original code, the error
> > changed, so that's progress. Now it says, even when running with root
> > permissions:
> > "libbpf: prog 'kfunc_call_test1': BPF program load failed: Permission denied"
> > Would be interesting to know why, but not necessary since I won't use
> > it this way.

some of the verifier's failure can return -EACCES and has nothing
to do with root permissions

> 
> Following up on this, I have moved to kernel 5.19.12 and I'm trying to
> make any kfunc work.
> I changed net/bpf/test_run.c to allow for more prog types, like master
> branch does, since originally it only had the first line for
> BPF_PROG_TYPE_SCHED_CLS.
> 
> ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS,
> &bpf_prog_test_kfunc_set);
> ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING,
> &bpf_prog_test_kfunc_set);
> ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_KPROBE,
> &bpf_prog_test_kfunc_set);
> ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACEPOINT,
> &bpf_prog_test_kfunc_set);
> return ret ?: register_btf_id_dtor_kfuncs(bpf_prog_test_dtor_kfunc,
>                           ARRAY_SIZE(bpf_prog_test_dtor_kfunc),
>                           THIS_MODULE);
> 
> I also added my own function to test it out and added it to the SET
> 
> u64 noinline bpf_kfunc_call_test4(u32 a, u64 b, u32 c, u64 d)
> {
>     return a + b + c + d;
> }
> //this is inside  BTF_SET_START(test_sk_check_kfunc_ids)
> BTF_ID(func, bpf_kfunc_call_test4)
> 
> Now I'm spraying every BPF program I can find to call either function:
> 
> extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
> __u32 c, __u64 d) __ksym;
> extern __u64 bpf_kfunc_call_test4(__u32 a, __u64 b, __u32 c, __u64 d) __ksym;
> 
> Compiling works, and when I run it I get no errors, *except* Permission denied.
> gist here: https://gist.github.com/hfingler/5c2c0b713299daa6b0ba07fa92ff29de
> 
> libbpf: prog 'kfunc_call_test1': BPF program load failed: Permission denied

seems like the kfunc is not found on the set for the program type, I guess
your change above did not go as expected

not sure what is your goal exactly, but perhaps better than starting from
scratch would be to take prog_tests/kfunc_call.c and progs/kfunc_call_test.c
and change them accordingly?

jirka

> ...
> calling kernel function bpf_kfunc_call_test1 is not allowed
> -- END PROG LOAD LOG --
> libbpf: prog 'kfunc_call_test1': failed to load: -13
> libbpf: failed to load object 'hello_bpf'
> libbpf: failed to load BPF skeleton 'hello_bpf': -13
> 
> I've tried a few programs and it would be too much to paste here, so
> here's a gist with all of them, which I tried each separately, not all
> at once:
> https://gist.github.com/hfingler/eb544b23cc36d57b8e9723cd36fbf243
> Basically, I've tried SEC("tc"),
> SEC("tracepoint/syscalls/sys_enter_open"),
> SEC("kprobe/__x64_sys_write")
> 
> What permission is being denied? I've tried running as root and I get
> the same thing, is there bpf permission checking somewhere else in the
> kernel? Do I have to have some sort of capability? Am I missing some
> kernel config?
> These are the configs I'm enabling in the kernel:
> https://gist.github.com/hfingler/ed780bd52b751625f52bbb08eb853641

Re: Replicating kfunc_call_test kernel test on standalone bpf program (calling kernel function is not allowed)

[Henrique Fingler]

> > Following up on this, I have moved to kernel 5.19.12 and I'm trying to
> > make any kfunc work.
> > I changed net/bpf/test_run.c to allow for more prog types, like master
> > branch does, since originally it only had the first line for
> > BPF_PROG_TYPE_SCHED_CLS.
> >
> > ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS,
> > &bpf_prog_test_kfunc_set);
> > ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING,
> > &bpf_prog_test_kfunc_set);
> > ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_KPROBE,
> > &bpf_prog_test_kfunc_set);
> > ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACEPOINT,
> > &bpf_prog_test_kfunc_set);
> > return ret ?: register_btf_id_dtor_kfuncs(bpf_prog_test_dtor_kfunc,
> >                           ARRAY_SIZE(bpf_prog_test_dtor_kfunc),
> >                           THIS_MODULE);
> >
> > I also added my own function to test it out and added it to the SET
> >
> > u64 noinline bpf_kfunc_call_test4(u32 a, u64 b, u32 c, u64 d)
> > {
> >     return a + b + c + d;
> > }
> > //this is inside  BTF_SET_START(test_sk_check_kfunc_ids)
> > BTF_ID(func, bpf_kfunc_call_test4)
> >
> > Now I'm spraying every BPF program I can find to call either function:
> >
> > extern __u64 bpf_kfunc_call_test1(struct sock *sk, __u32 a, __u64 b,
> > __u32 c, __u64 d) __ksym;
> > extern __u64 bpf_kfunc_call_test4(__u32 a, __u64 b, __u32 c, __u64 d) __ksym;
> >
> > Compiling works, and when I run it I get no errors, *except* Permission denied.
> > gist here: https://gist.github.com/hfingler/5c2c0b713299daa6b0ba07fa92ff29de
> >
> > libbpf: prog 'kfunc_call_test1': BPF program load failed: Permission denied
>
> seems like the kfunc is not found on the set for the program type, I guess
> your change above did not go as expected
>
> not sure what is your goal exactly, but perhaps better than starting from
> scratch would be to take prog_tests/kfunc_call.c and progs/kfunc_call_test.c
> and change them accordingly?
>
> jirka

My goal is to be able to call any kfunc. If I can do that I can add
trampolines to do more interesting things.
Starting from those files is pretty much what I was doing already, but
I started from scratch again.
I was finally able to load the BPF program without the EPERM error. I
think it's working but I'm not sure since I don't know
how to trigger the function so it prints something.

It seems like the error was having multiple calls to
`register_btf_kfunc_id_set` in `net/bpf/test_run.c`. By commenting out
the three
that I added, the function was found on the set and worked correctly.
With the others uncommented, the set find was returning false.
I suppose multiple of these calls do not work on 5.19 but do work on
later versions.
This part of the code on 6.0 (or master) has two calls to
`register_btf_kfunc_id_set` in sequence, but also many changes around
it, like `btf_kfunc_id_set` having only `.set`,
so maybe something was changed on the backend.

    ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS,
&bpf_prog_test_kfunc_set);
    //ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING,
&bpf_prog_test_kfunc_set);
    //ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_KPROBE,
&bpf_prog_test_kfunc_set);
    //ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACEPOINT,
&bpf_prog_test_kfunc_set);
    return ret ?: register_btf_id_dtor_kfuncs(bpf_prog_test_dtor_kfunc,
                          ARRAY_SIZE(bpf_prog_test_dtor_kfunc),
                          THIS_MODULE);

Before I go at it, are there any immediate restrictions on
BPF_PROG_TYPE_KPROBE calling kfuncs? I'll have to figure out how to do
it, but knowing it *can* work does help.

Thank you.