Dynamic kfunc discovery

Dynamic kfunc discovery

[Daniel Xu]

Hi,

I was chatting w/ Quentin [0] about how bpftool could:

1. Support a "feature dump" of all supported kfuncs on running kernel
2. Generate vmlinux.h with kfunc prototypes

I had another idea this morning so I thought I'd bounce it around
on the list in case others had better ones. 3 vague ideas:

1. Add a BTF type tag annotation in __bpf_kfunc macro. This would
   let bpftool parse BTF to do discovery. It would be fairly clean and
   straightforward, except that I don't think GCC supports these type
   tags. So only clang-built-linux would work.

2. Do the same thing as above, except rather than tagging src code,
   teach pahole about the .BTF_ids section in vmlinux. pahole could then
   construct BTF with the appropriate type tags.

3. Have the kernel export function prototypes (with no forward decls)
   in a pseudo fs file. bpftool could then append the contents of this file
   to the end of the generated vmlinux.h. Unfortunately this would rule
   out minimized type dumps (IOW only dump dependent types for kfuncs
   or something). Unless bpftool wants to embed clang-front-end or
   something like that to parse C.

Would appreciate any thoughts.

Thanks,
Daniel


[0]: https://github.com/libbpf/bpftool/issues/98

Re: Dynamic kfunc discovery

[Alexei Starovoitov]

On Tue, Dec 19, 2023 at 9:29 AM Daniel Xu <dxu@dxuuu.xyz> wrote:
>
> Hi,
>
> I was chatting w/ Quentin [0] about how bpftool could:
>
> 1. Support a "feature dump" of all supported kfuncs on running kernel
> 2. Generate vmlinux.h with kfunc prototypes
>
> I had another idea this morning so I thought I'd bounce it around
> on the list in case others had better ones. 3 vague ideas:
>
> 1. Add a BTF type tag annotation in __bpf_kfunc macro. This would
>    let bpftool parse BTF to do discovery. It would be fairly clean and
>    straightforward, except that I don't think GCC supports these type
>    tags. So only clang-built-linux would work.
>
> 2. Do the same thing as above, except rather than tagging src code,
>    teach pahole about the .BTF_ids section in vmlinux. pahole could then
>    construct BTF with the appropriate type tags.

resolve_btfids knows about all of them already.
The best is to teach bpftool about them as well.
It can look for BTF_SET8_START and there it can find btf_ids
of all kfuncs.
From there it can generate them into vmlinux.h

We wanted kfuncs to appear in vmlinux.h for quite some time,
but no one had cycles to do it.
Still an awesome feature to have.

Re: Dynamic kfunc discovery

[Jiri Olsa]

On Tue, Dec 19, 2023 at 07:15:42PM -0800, Alexei Starovoitov wrote:
> On Tue, Dec 19, 2023 at 9:29 AM Daniel Xu <dxu@dxuuu.xyz> wrote:
> >
> > Hi,
> >
> > I was chatting w/ Quentin [0] about how bpftool could:
> >
> > 1. Support a "feature dump" of all supported kfuncs on running kernel
> > 2. Generate vmlinux.h with kfunc prototypes
> >
> > I had another idea this morning so I thought I'd bounce it around
> > on the list in case others had better ones. 3 vague ideas:
> >
> > 1. Add a BTF type tag annotation in __bpf_kfunc macro. This would
> >    let bpftool parse BTF to do discovery. It would be fairly clean and
> >    straightforward, except that I don't think GCC supports these type
> >    tags. So only clang-built-linux would work.
> >
> > 2. Do the same thing as above, except rather than tagging src code,
> >    teach pahole about the .BTF_ids section in vmlinux. pahole could then
> >    construct BTF with the appropriate type tags.

I thought it'd be nice to have this in BTF, but to generate the .BTF_ids
section we need the BTF data (for BTF IDs), so that might be tricky

> 
> resolve_btfids knows about all of them already.
> The best is to teach bpftool about them as well.
> It can look for BTF_SET8_START and there it can find btf_ids

with the access to vmlinux, bpftool could get the addresses of all
set8s, read all btf ids and generate the header

$ nm vmlinux | grep BTF_ID__set8 
ffffffff843bf044 r __BTF_ID__set8__bpf_kfunc_check_set_skb
ffffffff843bf064 r __BTF_ID__set8__bpf_kfunc_check_set_sock_addr
ffffffff843bf054 r __BTF_ID__set8__bpf_kfunc_check_set_xdp
ffffffff843be940 r __BTF_ID__set8__bpf_map_iter_kfunc_ids
ffffffff843bf22c r __BTF_ID__set8__bpf_mptcp_fmodret_ids
ffffffff843be604 r __BTF_ID__set8__bpf_rstat_kfunc_ids
ffffffff843bf074 r __BTF_ID__set8__bpf_sk_iter_kfunc_ids
ffffffff843bf1c4 r __BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids
ffffffff843bf0bc r __BTF_ID__set8__bpf_test_modify_return_ids
ffffffff843be864 r __BTF_ID__set8__common_btf_ids
ffffffff843be9a8 r __BTF_ID__set8__cpumask_kfunc_btf_ids
ffffffff843bf174 r __BTF_ID__set8__fou_kfunc_set
ffffffff843be678 r __BTF_ID__set8__fs_kfunc_set_ids
ffffffff843be794 r __BTF_ID__set8__generic_btf_ids
ffffffff843be650 r __BTF_ID__set8__key_sig_kfunc_set
ffffffff843bf10c r __BTF_ID__set8__nf_ct_kfunc_set
ffffffff843bf164 r __BTF_ID__set8__nf_nat_kfunc_set
ffffffff843bf18c r __BTF_ID__set8__tcp_cubic_check_kfunc_ids
ffffffff843bf0dc r __BTF_ID__set8__test_sk_check_kfunc_ids
ffffffff843bf084 r __BTF_ID__set8__xdp_metadata_kfunc_ids
ffffffff843bf1f4 r __BTF_ID__set8__xfrm_ifc_kfunc_set
ffffffff843bf20c r __BTF_ID__set8__xfrm_state_kfunc_set

jirka

> of all kfuncs.
> From there it can generate them into vmlinux.h
> 
> We wanted kfuncs to appear in vmlinux.h for quite some time,
> but no one had cycles to do it.
> Still an awesome feature to have.
> 

Re: Dynamic kfunc discovery

[Jiri Olsa]

On Wed, Dec 20, 2023 at 10:07:34AM +0100, Jiri Olsa wrote:
> On Tue, Dec 19, 2023 at 07:15:42PM -0800, Alexei Starovoitov wrote:
> > On Tue, Dec 19, 2023 at 9:29 AM Daniel Xu <dxu@dxuuu.xyz> wrote:
> > >
> > > Hi,
> > >
> > > I was chatting w/ Quentin [0] about how bpftool could:
> > >
> > > 1. Support a "feature dump" of all supported kfuncs on running kernel
> > > 2. Generate vmlinux.h with kfunc prototypes
> > >
> > > I had another idea this morning so I thought I'd bounce it around
> > > on the list in case others had better ones. 3 vague ideas:
> > >
> > > 1. Add a BTF type tag annotation in __bpf_kfunc macro. This would
> > >    let bpftool parse BTF to do discovery. It would be fairly clean and
> > >    straightforward, except that I don't think GCC supports these type
> > >    tags. So only clang-built-linux would work.
> > >
> > > 2. Do the same thing as above, except rather than tagging src code,
> > >    teach pahole about the .BTF_ids section in vmlinux. pahole could then
> > >    construct BTF with the appropriate type tags.
> 
> I thought it'd be nice to have this in BTF, but to generate the .BTF_ids
> section we need the BTF data (for BTF IDs), so that might be tricky

we could also move resolve_btfids logic into pahole and it could
add the kfunc data to BTF directly

> 
> > 
> > resolve_btfids knows about all of them already.
> > The best is to teach bpftool about them as well.
> > It can look for BTF_SET8_START and there it can find btf_ids
> 
> with the access to vmlinux, bpftool could get the addresses of all
> set8s, read all btf ids and generate the header

and maybe we could also read kfunc data directly from /proc/kcore:

  # cat /proc/kallsyms  | grep __BTF_ID__set8__generic_btf_ids
  ffffffff843be898 r __BTF_ID__set8__generic_btf_ids
  # objdump -s --start-address=0xffffffff843be898 --stop-address=0xffffffff843be8a8 /proc/kcore   

  /proc/kcore:     file format elf64-x86-64

  Contents of section load1:
   ffffffff843be898 17000000 00000000 15750100 85000000  .........u......

I think having it in BTF would be easiest from user's POV,
but seems like a lot of work.. reading it from kcore seems
good enough

jirka

> 
> $ nm vmlinux | grep BTF_ID__set8 
> ffffffff843bf044 r __BTF_ID__set8__bpf_kfunc_check_set_skb
> ffffffff843bf064 r __BTF_ID__set8__bpf_kfunc_check_set_sock_addr
> ffffffff843bf054 r __BTF_ID__set8__bpf_kfunc_check_set_xdp
> ffffffff843be940 r __BTF_ID__set8__bpf_map_iter_kfunc_ids
> ffffffff843bf22c r __BTF_ID__set8__bpf_mptcp_fmodret_ids
> ffffffff843be604 r __BTF_ID__set8__bpf_rstat_kfunc_ids
> ffffffff843bf074 r __BTF_ID__set8__bpf_sk_iter_kfunc_ids
> ffffffff843bf1c4 r __BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids
> ffffffff843bf0bc r __BTF_ID__set8__bpf_test_modify_return_ids
> ffffffff843be864 r __BTF_ID__set8__common_btf_ids
> ffffffff843be9a8 r __BTF_ID__set8__cpumask_kfunc_btf_ids
> ffffffff843bf174 r __BTF_ID__set8__fou_kfunc_set
> ffffffff843be678 r __BTF_ID__set8__fs_kfunc_set_ids
> ffffffff843be794 r __BTF_ID__set8__generic_btf_ids
> ffffffff843be650 r __BTF_ID__set8__key_sig_kfunc_set
> ffffffff843bf10c r __BTF_ID__set8__nf_ct_kfunc_set
> ffffffff843bf164 r __BTF_ID__set8__nf_nat_kfunc_set
> ffffffff843bf18c r __BTF_ID__set8__tcp_cubic_check_kfunc_ids
> ffffffff843bf0dc r __BTF_ID__set8__test_sk_check_kfunc_ids
> ffffffff843bf084 r __BTF_ID__set8__xdp_metadata_kfunc_ids
> ffffffff843bf1f4 r __BTF_ID__set8__xfrm_ifc_kfunc_set
> ffffffff843bf20c r __BTF_ID__set8__xfrm_state_kfunc_set
> 
> jirka
> 
> > of all kfuncs.
> > From there it can generate them into vmlinux.h
> > 
> > We wanted kfuncs to appear in vmlinux.h for quite some time,
> > but no one had cycles to do it.
> > Still an awesome feature to have.
> > 

Re: Dynamic kfunc discovery

[Daniel Xu]

Hi Jiri,

On Wed, Dec 20, 2023 at 10:07:34AM +0100, Jiri Olsa wrote:
> On Tue, Dec 19, 2023 at 07:15:42PM -0800, Alexei Starovoitov wrote:
> > On Tue, Dec 19, 2023 at 9:29 AM Daniel Xu <dxu@dxuuu.xyz> wrote:
> > >
> > > Hi,
> > >
> > > I was chatting w/ Quentin [0] about how bpftool could:
> > >
> > > 1. Support a "feature dump" of all supported kfuncs on running kernel
> > > 2. Generate vmlinux.h with kfunc prototypes
> > >
> > > I had another idea this morning so I thought I'd bounce it around
> > > on the list in case others had better ones. 3 vague ideas:
> > >
> > > 1. Add a BTF type tag annotation in __bpf_kfunc macro. This would
> > >    let bpftool parse BTF to do discovery. It would be fairly clean and
> > >    straightforward, except that I don't think GCC supports these type
> > >    tags. So only clang-built-linux would work.
> > >
> > > 2. Do the same thing as above, except rather than tagging src code,
> > >    teach pahole about the .BTF_ids section in vmlinux. pahole could then
> > >    construct BTF with the appropriate type tags.
> 
> I thought it'd be nice to have this in BTF, but to generate the .BTF_ids
> section we need the BTF data (for BTF IDs), so that might be tricky

Isn't .BTF_ids already present in vmlinux before getting to
resolve_btfids? It looks to me like all resolve_btfids does is patch
symbols to the read BTF ID values.

To inject BTF type tags from pahole, I don't think it needs a patched
.BTF_ids section, right? After pahole has generated all the regular
entries, it could walk .BTF_ids and try to match up symbol names with
BTF function entries. And then inject the BTF type tag.

> 
> > 
> > resolve_btfids knows about all of them already.
> > The best is to teach bpftool about them as well.
> > It can look for BTF_SET8_START and there it can find btf_ids
> 
> with the access to vmlinux, bpftool could get the addresses of all
> set8s, read all btf ids and generate the header
> 
> $ nm vmlinux | grep BTF_ID__set8 
> ffffffff843bf044 r __BTF_ID__set8__bpf_kfunc_check_set_skb
> ffffffff843bf064 r __BTF_ID__set8__bpf_kfunc_check_set_sock_addr
> ffffffff843bf054 r __BTF_ID__set8__bpf_kfunc_check_set_xdp
> ffffffff843be940 r __BTF_ID__set8__bpf_map_iter_kfunc_ids
> ffffffff843bf22c r __BTF_ID__set8__bpf_mptcp_fmodret_ids
> ffffffff843be604 r __BTF_ID__set8__bpf_rstat_kfunc_ids
> ffffffff843bf074 r __BTF_ID__set8__bpf_sk_iter_kfunc_ids
> ffffffff843bf1c4 r __BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids
> ffffffff843bf0bc r __BTF_ID__set8__bpf_test_modify_return_ids
> ffffffff843be864 r __BTF_ID__set8__common_btf_ids
> ffffffff843be9a8 r __BTF_ID__set8__cpumask_kfunc_btf_ids
> ffffffff843bf174 r __BTF_ID__set8__fou_kfunc_set
> ffffffff843be678 r __BTF_ID__set8__fs_kfunc_set_ids
> ffffffff843be794 r __BTF_ID__set8__generic_btf_ids
> ffffffff843be650 r __BTF_ID__set8__key_sig_kfunc_set
> ffffffff843bf10c r __BTF_ID__set8__nf_ct_kfunc_set
> ffffffff843bf164 r __BTF_ID__set8__nf_nat_kfunc_set
> ffffffff843bf18c r __BTF_ID__set8__tcp_cubic_check_kfunc_ids
> ffffffff843bf0dc r __BTF_ID__set8__test_sk_check_kfunc_ids
> ffffffff843bf084 r __BTF_ID__set8__xdp_metadata_kfunc_ids
> ffffffff843bf1f4 r __BTF_ID__set8__xfrm_ifc_kfunc_set
> ffffffff843bf20c r __BTF_ID__set8__xfrm_state_kfunc_set
> 
> jirka
> 
> > of all kfuncs.
> > From there it can generate them into vmlinux.h
> > 
> > We wanted kfuncs to appear in vmlinux.h for quite some time,
> > but no one had cycles to do it.
> > Still an awesome feature to have.
> > 

Thanks,
Daniel

Re: Dynamic kfunc discovery

[Jiri Olsa]

On Wed, Dec 20, 2023 at 09:44:10AM -0700, Daniel Xu wrote:
> Hi Jiri,
> 
> On Wed, Dec 20, 2023 at 10:07:34AM +0100, Jiri Olsa wrote:
> > On Tue, Dec 19, 2023 at 07:15:42PM -0800, Alexei Starovoitov wrote:
> > > On Tue, Dec 19, 2023 at 9:29 AM Daniel Xu <dxu@dxuuu.xyz> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I was chatting w/ Quentin [0] about how bpftool could:
> > > >
> > > > 1. Support a "feature dump" of all supported kfuncs on running kernel
> > > > 2. Generate vmlinux.h with kfunc prototypes
> > > >
> > > > I had another idea this morning so I thought I'd bounce it around
> > > > on the list in case others had better ones. 3 vague ideas:
> > > >
> > > > 1. Add a BTF type tag annotation in __bpf_kfunc macro. This would
> > > >    let bpftool parse BTF to do discovery. It would be fairly clean and
> > > >    straightforward, except that I don't think GCC supports these type
> > > >    tags. So only clang-built-linux would work.
> > > >
> > > > 2. Do the same thing as above, except rather than tagging src code,
> > > >    teach pahole about the .BTF_ids section in vmlinux. pahole could then
> > > >    construct BTF with the appropriate type tags.
> > 
> > I thought it'd be nice to have this in BTF, but to generate the .BTF_ids
> > section we need the BTF data (for BTF IDs), so that might be tricky
> 
> Isn't .BTF_ids already present in vmlinux before getting to
> resolve_btfids? It looks to me like all resolve_btfids does is patch
> symbols to the read BTF ID values.

yes, it's there but it's empty.. IDs are zero

> To inject BTF type tags from pahole, I don't think it needs a patched
> .BTF_ids section, right? After pahole has generated all the regular
> entries, it could walk .BTF_ids and try to match up symbol names with
> BTF function entries. And then inject the BTF type tag.

so what resolve_btfids does is to lookup all __BTF_ID__set8__* symbols,
finds their BTF IDs and stores them where the symbol points

there's explanation on the symbol name in tools/bpf/resolve_btfids/main.c
header

pahole could do the same and once it has the IDs it could add the type
tag to them, initially I thought having extra BTF section with kfuncs
BTF ids, but type tag seems like better way to do that

jirka

> 
> > 
> > > 
> > > resolve_btfids knows about all of them already.
> > > The best is to teach bpftool about them as well.
> > > It can look for BTF_SET8_START and there it can find btf_ids
> > 
> > with the access to vmlinux, bpftool could get the addresses of all
> > set8s, read all btf ids and generate the header
> > 
> > $ nm vmlinux | grep BTF_ID__set8 
> > ffffffff843bf044 r __BTF_ID__set8__bpf_kfunc_check_set_skb
> > ffffffff843bf064 r __BTF_ID__set8__bpf_kfunc_check_set_sock_addr
> > ffffffff843bf054 r __BTF_ID__set8__bpf_kfunc_check_set_xdp
> > ffffffff843be940 r __BTF_ID__set8__bpf_map_iter_kfunc_ids
> > ffffffff843bf22c r __BTF_ID__set8__bpf_mptcp_fmodret_ids
> > ffffffff843be604 r __BTF_ID__set8__bpf_rstat_kfunc_ids
> > ffffffff843bf074 r __BTF_ID__set8__bpf_sk_iter_kfunc_ids
> > ffffffff843bf1c4 r __BTF_ID__set8__bpf_tcp_ca_check_kfunc_ids
> > ffffffff843bf0bc r __BTF_ID__set8__bpf_test_modify_return_ids
> > ffffffff843be864 r __BTF_ID__set8__common_btf_ids
> > ffffffff843be9a8 r __BTF_ID__set8__cpumask_kfunc_btf_ids
> > ffffffff843bf174 r __BTF_ID__set8__fou_kfunc_set
> > ffffffff843be678 r __BTF_ID__set8__fs_kfunc_set_ids
> > ffffffff843be794 r __BTF_ID__set8__generic_btf_ids
> > ffffffff843be650 r __BTF_ID__set8__key_sig_kfunc_set
> > ffffffff843bf10c r __BTF_ID__set8__nf_ct_kfunc_set
> > ffffffff843bf164 r __BTF_ID__set8__nf_nat_kfunc_set
> > ffffffff843bf18c r __BTF_ID__set8__tcp_cubic_check_kfunc_ids
> > ffffffff843bf0dc r __BTF_ID__set8__test_sk_check_kfunc_ids
> > ffffffff843bf084 r __BTF_ID__set8__xdp_metadata_kfunc_ids
> > ffffffff843bf1f4 r __BTF_ID__set8__xfrm_ifc_kfunc_set
> > ffffffff843bf20c r __BTF_ID__set8__xfrm_state_kfunc_set
> > 
> > jirka
> > 
> > > of all kfuncs.
> > > From there it can generate them into vmlinux.h
> > > 
> > > We wanted kfuncs to appear in vmlinux.h for quite some time,
> > > but no one had cycles to do it.
> > > Still an awesome feature to have.
> > > 
> 
> Thanks,
> Daniel

Re: Dynamic kfunc discovery

[Daniel Xu]

On Wed, Dec 20, 2023 at 09:44:10AM -0700, Daniel Xu wrote:
> Hi Jiri,
> 
> On Wed, Dec 20, 2023 at 10:07:34AM +0100, Jiri Olsa wrote:
> > On Tue, Dec 19, 2023 at 07:15:42PM -0800, Alexei Starovoitov wrote:
> > > On Tue, Dec 19, 2023 at 9:29 AM Daniel Xu <dxu@dxuuu.xyz> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I was chatting w/ Quentin [0] about how bpftool could:
> > > >
> > > > 1. Support a "feature dump" of all supported kfuncs on running kernel
> > > > 2. Generate vmlinux.h with kfunc prototypes
> > > >
> > > > I had another idea this morning so I thought I'd bounce it around
> > > > on the list in case others had better ones. 3 vague ideas:
> > > >
> > > > 1. Add a BTF type tag annotation in __bpf_kfunc macro. This would
> > > >    let bpftool parse BTF to do discovery. It would be fairly clean and
> > > >    straightforward, except that I don't think GCC supports these type
> > > >    tags. So only clang-built-linux would work.
> > > >
> > > > 2. Do the same thing as above, except rather than tagging src code,
> > > >    teach pahole about the .BTF_ids section in vmlinux. pahole could then
> > > >    construct BTF with the appropriate type tags.
> > 
> > I thought it'd be nice to have this in BTF, but to generate the .BTF_ids
> > section we need the BTF data (for BTF IDs), so that might be tricky
> 
> Isn't .BTF_ids already present in vmlinux before getting to
> resolve_btfids? It looks to me like all resolve_btfids does is patch
> symbols to the read BTF ID values.
> 
> To inject BTF type tags from pahole, I don't think it needs a patched
> .BTF_ids section, right? After pahole has generated all the regular
> entries, it could walk .BTF_ids and try to match up symbol names with
> BTF function entries. And then inject the BTF type tag.

I have a working prototype. Will send out a patch later today.

[...]