[QUESTION] Check bpf_loop support on kernels < 5.13

[QUESTION] Check bpf_loop support on kernels < 5.13

[andrea terzolo]

Hi folks! I would like to check with you if the verifier failure I'm
facing is expected. The verifier rejects the following eBPF program on
kernel 5.10.232.

```
static long loop_fn(uint32_t index, void *ctx) {
  bpf_printk("handle_exit\n");
  return 0;
}

SEC("tp/raw_syscalls/sys_enter")
int test(void *ctx) {
  if (bpf_core_enum_value_exists(enum bpf_func_id, BPF_FUNC_loop)) {
    bpf_printk("loop\n");
    bpf_loop(12, loop_fn, NULL, 0);
  } else {
    bpf_printk("skip loop\n");
  }
  return 0;
}
```

With this error:

```
libbpf: prog 'test': BPF program load failed: Invalid argument
libbpf: prog 'test': -- BEGIN PROG LOAD LOG --
number of funcs in func_info doesn't match number of subprogs
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0
peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'test': failed to load: -22
```

This sounds like a valid use case. I would like to use bpf_loop if
supported by the running kernel otherwise I can fall back to a simple
loop. This issue goes away on kernel 5.13 with the introduction of
PTR_TO_FUNC [0]. Is there a way I can use CO-RE features to avoid this
issue? I would expect the verifier to prune the dead code inside the
`if` but the error seems to be triggered before the control flow
analysis.

[0]: https://github.com/torvalds/linux/commit/69c087ba6225b574afb6e505b72cb75242a3d844

Re: [QUESTION] Check bpf_loop support on kernels < 5.13

[Eduard Zingerman]

On Fri, 2025-01-03 at 12:03 +0100, andrea terzolo wrote:
> Hi folks! I would like to check with you if the verifier failure I'm
> facing is expected. The verifier rejects the following eBPF program on
> kernel 5.10.232.
> 
> ```
> static long loop_fn(uint32_t index, void *ctx) {
>   bpf_printk("handle_exit\n");
>   return 0;
> }
> 
> SEC("tp/raw_syscalls/sys_enter")
> int test(void *ctx) {
>   if (bpf_core_enum_value_exists(enum bpf_func_id, BPF_FUNC_loop)) {
>     bpf_printk("loop\n");
>     bpf_loop(12, loop_fn, NULL, 0);
>   } else {
>     bpf_printk("skip loop\n");
>   }
>   return 0;
> }
> ```
> 
> With this error:
> 
> ```
> libbpf: prog 'test': BPF program load failed: Invalid argument
> libbpf: prog 'test': -- BEGIN PROG LOAD LOG --
> number of funcs in func_info doesn't match number of subprogs
> processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0
> peak_states 0 mark_read 0
> -- END PROG LOAD LOG --
> libbpf: prog 'test': failed to load: -22
> ```
> 
> This sounds like a valid use case. I would like to use bpf_loop if
> supported by the running kernel otherwise I can fall back to a simple
> loop. This issue goes away on kernel 5.13 with the introduction of
> PTR_TO_FUNC [0]. Is there a way I can use CO-RE features to avoid this
> issue? I would expect the verifier to prune the dead code inside the
> `if` but the error seems to be triggered before the control flow
> analysis.
> 
> [0]: https://github.com/torvalds/linux/commit/69c087ba6225b574afb6e505b72cb75242a3d844

bpf_loop was introduced by commit [1] and released as a part of 5.17.

The error you see is indeed caused by the lack of PTR_TO_FUNC register
type in an old kernel. In your program the call to bpf_loop would look
like below in the assembly:

  ...
  r2 = loop_fn  ;; here function pointer is taken
  ...
  call bpf_loop

Before main verification pass verifier.c:add_subprog_and_kfunc()
discovers subprogram entries by looking at function calls and function
pointer assignments and compares it to function information provided
via bpf_attr->func_info. The kernel that does not know about
PTR_TO_FUNC would not find the loop_fn entry, hence the error message
about mismatch.

Additionally, verifier.c:check_cfg() looks for parts of the program
that can't be reached by jump and call instructions. For this purpose
pointers to functions are treated as function calls. The kernel that
does not know about PTR_TO_FUNC it would seem that loop_fn is unreachable,
this would cause another error message.

Even if you add a dummy call to loop_fn verifier would most likely
reject the program at 'r2 = loop_fn'.

The approach libbpf uses to detect running kernel features is based on
programs accept/reject status [2]. E.g. your program could be simplified to:

  static int loop_fn(int i, void *c) { return 0; }

  SEC("tp/raw_syscalls/sys_enter")
  int test(void *ctx) {
    bpf_loop(1, loop_fn, NULL, 0);
    return 0;
  }

And checked if load is successful.

[1] e6f2dd0f8067 ("bpf: Add bpf_loop helper")
[2] see <kernel>/tools/lib/bpf/features.c

Re: [QUESTION] Check bpf_loop support on kernels < 5.13

[andrea terzolo]

Thank you for the detailed explanation!
One alternative could be to use two different eBPF programs, one with
bpf_loop and the other with a simple loop, but I was looking for
possible alternatives. I tried using a const variable.
eBPF side:
```
volatile const int has_bpf_loop = 0;

static int loop_cb(int i, void *ctx) {
  return 0;
}

SEC("raw_tp")
int test_prog(void *ctx) {
  if (has_bpf_loop == 1) {
    bpf_loop(10, loop_cb, NULL, 0);
  } else {
    for (int i = 0; i < 10; ++i)
      loop_cb(i, NULL);
  }
  return 0;
}
```

Userspace side:
```
...
  if (libbpf_probe_bpf_helper(BPF_PROG_TYPE_RAW_TRACEPOINT,
BPF_FUNC_loop, NULL) == 1) {
    skel->rodata->has_bpf_loop = 1;
  } else {
    skel->rodata->has_bpf_loop = 0;
  }
...
```

But I ended up with the same error as before
```
libbpf: prog 'test_prog': BPF program load failed: Invalid argument
libbpf: prog 'test_prog': -- BEGIN PROG LOAD LOG --
number of funcs in func_info doesn't match number of subprogs
processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0
peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'test_prog': failed to load: -22
```

I will try to dig a little bit more into it but I'm not sure we can
avoid using 2 different ebpf programs. Even if we were able to
overcome this error I suspect that the verifier will reject the
program when we try to load into r2 the pointer to `loop_cp` as you
correctly highlighted.

On Fri, 3 Jan 2025 at 21:31, Eduard Zingerman <eddyz87@gmail.com> wrote:
>
> On Fri, 2025-01-03 at 12:03 +0100, andrea terzolo wrote:
> > Hi folks! I would like to check with you if the verifier failure I'm
> > facing is expected. The verifier rejects the following eBPF program on
> > kernel 5.10.232.
> >
> > ```
> > static long loop_fn(uint32_t index, void *ctx) {
> >   bpf_printk("handle_exit\n");
> >   return 0;
> > }
> >
> > SEC("tp/raw_syscalls/sys_enter")
> > int test(void *ctx) {
> >   if (bpf_core_enum_value_exists(enum bpf_func_id, BPF_FUNC_loop)) {
> >     bpf_printk("loop\n");
> >     bpf_loop(12, loop_fn, NULL, 0);
> >   } else {
> >     bpf_printk("skip loop\n");
> >   }
> >   return 0;
> > }
> > ```
> >
> > With this error:
> >
> > ```
> > libbpf: prog 'test': BPF program load failed: Invalid argument
> > libbpf: prog 'test': -- BEGIN PROG LOAD LOG --
> > number of funcs in func_info doesn't match number of subprogs
> > processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0
> > peak_states 0 mark_read 0
> > -- END PROG LOAD LOG --
> > libbpf: prog 'test': failed to load: -22
> > ```
> >
> > This sounds like a valid use case. I would like to use bpf_loop if
> > supported by the running kernel otherwise I can fall back to a simple
> > loop. This issue goes away on kernel 5.13 with the introduction of
> > PTR_TO_FUNC [0]. Is there a way I can use CO-RE features to avoid this
> > issue? I would expect the verifier to prune the dead code inside the
> > `if` but the error seems to be triggered before the control flow
> > analysis.
> >
> > [0]: https://github.com/torvalds/linux/commit/69c087ba6225b574afb6e505b72cb75242a3d844
>
> bpf_loop was introduced by commit [1] and released as a part of 5.17.
>
> The error you see is indeed caused by the lack of PTR_TO_FUNC register
> type in an old kernel. In your program the call to bpf_loop would look
> like below in the assembly:
>
>   ...
>   r2 = loop_fn  ;; here function pointer is taken
>   ...
>   call bpf_loop
>
> Before main verification pass verifier.c:add_subprog_and_kfunc()
> discovers subprogram entries by looking at function calls and function
> pointer assignments and compares it to function information provided
> via bpf_attr->func_info. The kernel that does not know about
> PTR_TO_FUNC would not find the loop_fn entry, hence the error message
> about mismatch.
>
> Additionally, verifier.c:check_cfg() looks for parts of the program
> that can't be reached by jump and call instructions. For this purpose
> pointers to functions are treated as function calls. The kernel that
> does not know about PTR_TO_FUNC it would seem that loop_fn is unreachable,
> this would cause another error message.
>
> Even if you add a dummy call to loop_fn verifier would most likely
> reject the program at 'r2 = loop_fn'.
>
> The approach libbpf uses to detect running kernel features is based on
> programs accept/reject status [2]. E.g. your program could be simplified to:
>
>   static int loop_fn(int i, void *c) { return 0; }
>
>   SEC("tp/raw_syscalls/sys_enter")
>   int test(void *ctx) {
>     bpf_loop(1, loop_fn, NULL, 0);
>     return 0;
>   }
>
> And checked if load is successful.
>
> [1] e6f2dd0f8067 ("bpf: Add bpf_loop helper")
> [2] see <kernel>/tools/lib/bpf/features.c
>