BPF program metadata

BPF program metadata

[sdf]

As discussed in
https://docs.google.com/presentation/d/1A9Anx8JPHl_pK1aXy8hlxs3V5pkrKwHHtkPf_-HeYTc
during BPF office hours, we'd like to attach arbitrary auxiliary
metadata to the program, for example, the build timestamp or the commit
hash.

IIRC, the suggestion was to explore BTF and .BTF.ext section in
particular.
We've spent some time looking at the BTF encoding and BTF.ext section
and we don't see how we can put this data into .BTF.ext or even .BTF
without any kernel changes.

The reasoning (at least how we see it):
* .BTF.ext is just a container with func_info/line_info/relocation_info
   and libbpf extracts the data form this section and passes it to
   sys_bpf(BPF_PROG_LOAD); the important note is that it doesn't pass the
   whole container to the kernel, but passes the data that's been
   extracted from the appropriate sections
* .BTF can be used for metadata, but it looks like we'd have to add
   another BTF_INFO_KIND() to make it a less messy (YiFei, feel free to
   correct me)

So the question is: are we missing something? Is there some way to add
key=value metadata to BTF that doesn't involve a lot of kernel changes?

If the restrictions above are correct, should we go back to trying to
put this metadata into .data section (or maybe even the new .metadata
section)? The only missing piece of the puzzle in that case is the
ability to extend BPF_PROG_LOAD with a way to say 'hold this map
unconditionally'.

Re: BPF program metadata

[sdf]

On 07/29, sdf@google.com wrote:
> As discussed in
> https://docs.google.com/presentation/d/1A9Anx8JPHl_pK1aXy8hlxs3V5pkrKwHHtkPf_-HeYTc
> during BPF office hours, we'd like to attach arbitrary auxiliary
> metadata to the program, for example, the build timestamp or the commit
> hash.

> IIRC, the suggestion was to explore BTF and .BTF.ext section in
> particular.
> We've spent some time looking at the BTF encoding and BTF.ext section
> and we don't see how we can put this data into .BTF.ext or even .BTF
> without any kernel changes.

> The reasoning (at least how we see it):
> * .BTF.ext is just a container with func_info/line_info/relocation_info
>    and libbpf extracts the data form this section and passes it to
>    sys_bpf(BPF_PROG_LOAD); the important note is that it doesn't pass the
>    whole container to the kernel, but passes the data that's been
>    extracted from the appropriate sections
> * .BTF can be used for metadata, but it looks like we'd have to add
>    another BTF_INFO_KIND() to make it a less messy (YiFei, feel free to
>    correct me)

> So the question is: are we missing something? Is there some way to add
> key=value metadata to BTF that doesn't involve a lot of kernel changes?

> If the restrictions above are correct, should we go back to trying to
> put this metadata into .data section (or maybe even the new .metadata
> section)? The only missing piece of the puzzle in that case is the
> ability to extend BPF_PROG_LOAD with a way to say 'hold this map
> unconditionally'.
Should we have a short discussion about that this Thu during the office
hours? I don't see this week's sheet page in there yet :-(

Re: BPF program metadata

[Alexei Starovoitov]

On Tue, Aug 4, 2020 at 12:42 PM <sdf@google.com> wrote:
>
> On 07/29, sdf@google.com wrote:
> > As discussed in
> > https://docs.google.com/presentation/d/1A9Anx8JPHl_pK1aXy8hlxs3V5pkrKwHHtkPf_-HeYTc
> > during BPF office hours, we'd like to attach arbitrary auxiliary
> > metadata to the program, for example, the build timestamp or the commit
> > hash.
>
> > IIRC, the suggestion was to explore BTF and .BTF.ext section in
> > particular.
> > We've spent some time looking at the BTF encoding and BTF.ext section
> > and we don't see how we can put this data into .BTF.ext or even .BTF
> > without any kernel changes.
>
> > The reasoning (at least how we see it):
> > * .BTF.ext is just a container with func_info/line_info/relocation_info
> >    and libbpf extracts the data form this section and passes it to
> >    sys_bpf(BPF_PROG_LOAD); the important note is that it doesn't pass the
> >    whole container to the kernel, but passes the data that's been
> >    extracted from the appropriate sections
> > * .BTF can be used for metadata, but it looks like we'd have to add
> >    another BTF_INFO_KIND() to make it a less messy (YiFei, feel free to
> >    correct me)
>
> > So the question is: are we missing something? Is there some way to add
> > key=value metadata to BTF that doesn't involve a lot of kernel changes?
>
> > If the restrictions above are correct, should we go back to trying to
> > put this metadata into .data section (or maybe even the new .metadata
> > section)? The only missing piece of the puzzle in that case is the
> > ability to extend BPF_PROG_LOAD with a way to say 'hold this map
> > unconditionally'.
> Should we have a short discussion about that this Thu during the office
> hours?

Of course. That's what office hours are for.
Since google folks have trouble with zoom I've added google meets link
to the spreadsheet. Let's try it tomorrow.

Re: BPF program metadata

[sdf]

On 08/05, Alexei Starovoitov wrote:
> On Tue, Aug 4, 2020 at 12:42 PM <sdf@google.com> wrote:
> >
> > On 07/29, sdf@google.com wrote:
> > > As discussed in
> > >  
> https://docs.google.com/presentation/d/1A9Anx8JPHl_pK1aXy8hlxs3V5pkrKwHHtkPf_-HeYTc
> > > during BPF office hours, we'd like to attach arbitrary auxiliary
> > > metadata to the program, for example, the build timestamp or the  
> commit
> > > hash.
> >
> > > IIRC, the suggestion was to explore BTF and .BTF.ext section in
> > > particular.
> > > We've spent some time looking at the BTF encoding and BTF.ext section
> > > and we don't see how we can put this data into .BTF.ext or even .BTF
> > > without any kernel changes.
> >
> > > The reasoning (at least how we see it):
> > > * .BTF.ext is just a container with  
> func_info/line_info/relocation_info
> > >    and libbpf extracts the data form this section and passes it to
> > >    sys_bpf(BPF_PROG_LOAD); the important note is that it doesn't pass  
> the
> > >    whole container to the kernel, but passes the data that's been
> > >    extracted from the appropriate sections
> > > * .BTF can be used for metadata, but it looks like we'd have to add
> > >    another BTF_INFO_KIND() to make it a less messy (YiFei, feel free  
> to
> > >    correct me)
> >
> > > So the question is: are we missing something? Is there some way to add
> > > key=value metadata to BTF that doesn't involve a lot of kernel  
> changes?
> >
> > > If the restrictions above are correct, should we go back to trying to
> > > put this metadata into .data section (or maybe even the new .metadata
> > > section)? The only missing piece of the puzzle in that case is the
> > > ability to extend BPF_PROG_LOAD with a way to say 'hold this map
> > > unconditionally'.
> > Should we have a short discussion about that this Thu during the office
> > hours?

> Of course. That's what office hours are for.
> Since google folks have trouble with zoom I've added google meets link
> to the spreadsheet. Let's try it tomorrow.
Ooooh, thank you for the Google Meets link :-)

BPF office hours via google meet. Was: BPF program metadata

[Alexei Starovoitov]

On Wed, Aug 5, 2020 at 4:10 PM <sdf@google.com> wrote:
> > Since google folks have trouble with zoom I've added google meets link
> > to the spreadsheet. Let's try it tomorrow.

Today we held BPF office hours via google meet and it worked well.
So we will be using it for future meetings.
zoom link has been removed from the spreadsheet to avoid confusion.