Re: Establishing /usr/lib/bpf as a convention for eBPF bytecode files?

["Toke Høiland-Jørgensen" <toke@redhat.com>]

Jesper Dangaard Brouer <brouer@redhat.com> writes:

> On Mon, 9 Dec 2019 17:40:19 -0800
> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
>> On Mon, Dec 09, 2019 at 12:29:27PM +0100, Toke Høiland-Jørgensen wrote:
>> > Hi everyone
>> > 
>> > As you have no doubt noticed, we have started thinking about how to
>> > package eBPF-related applications in distributions. As a part of this,
>> > I've been thinking about what to recommend for applications that ship
>> > pre-compiled BPF byte-code files.
>> > 
>> > The obvious place to place those would be somewhere in the system
>> > $LIBDIR (i.e., /usr/lib or /usr/lib64, depending on the distro). But
>> > since BPF byte code is its own binary format, different from regular
>> > executables, I think having a separate path to put those under makes
>> > sense. So I'm proposing to establish a convention that pre-compiled BPF
>> > programs be installed into /usr/lib{,64}/bpf.
>> > 
>> > This would let users discover which BPF programs are shipped on their
>> > system, and it could be used to discover which package loaded a
>> > particular BPF program, by walking the directory to find the file a
>> > loaded program came from. It would not work for dynamically-generated
>> > bytecode, of course, but I think at least some applications will end up
>> > shipping pre-compiled bytecode files (we're doing that for xdp-tools,
>> > for instance).
>> > 
>> > As I said, this would be a convention. We're already using it for
>> > xdp-tools[0], so my plan is to use that as the "first mover", try to get
>> > distributions to establish the path as a part of their filesystem
>> > layout, and then just try to encourage packages to use it. Hopefully it
>> > will catch on.
>> > 
>> > Does anyone have any objections to this? Do you think it is a complete
>> > waste of time, or is it worth giving it a shot? :)  
>> 
>> What will be the name of file/directory ?
>> What is going to be the mechanism to clean it up?
>> What will be stored in there? Just .o files ?
>> 
>> libbcc stores original C and rewritten C in /var/tmp/bcc/bpf_prog_TAG/
>> It was useful for debugging. Since TAG is used as directory name
>> reloading the same bcc script creates the same dir and /var/tmp
>> periodically gets cleaned by reboot.
>> 
>> Installing bpf .o into common location feels useful. Not sure though
>> how you can convince folks to follow such convention.
>
> I imagine the files under /usr/lib{,64}/bpf/ will be pre-compiled
> binaries (fairly static file).  These will be delivered together with
> the distro RPM file. The distro will detect/enforce that two packages
> cannot use the same name for bpf .o files.

Yes, that was my intention. Packages can choose whether to create a
subdirectory, or just dump files in /usr/lib{,64}/bpf (this is similar
to /usr/lib).

> I see three different types of BPF-object files, which belong in
> different places (suggestion in parentheses):
>
>  1. Pre-compiled binaries via RPM. (/usr/lib? [1])
>  2. Application "startup" compiled "cached" BPF-object (/var/cache? [2]).
>  3. Runtime dynamic compiled BPF-objects short lived (/run? [3])
>
> You can follow the links below, to see if match descriptions in
> the Filesystem Hierarchy Standard[4].
>
> I think that filetype 1 + 2 makes sense to store in files. For
> filetype 3 (the highly dynamic runtime re-compiled files) I'm not
> sure it makes sense to store those in any central place.  Applications
> could use /run/application-name/, but it will be a pain to deal with
> filename-clashes. As Alexei brings up cleanup; /run/ is cleared at the
> beginning of the boot process[3].
>
> For fileytpe 2, I suggest /var/cache/bpf/, but with an additional
> application name as a subdir, this is to avoid name clashes (which then
> becomes the applications responsibility with in its own dir).

/var/cache/bpf seems reasonable, let's go with that. My plan is to try
to get the directories established in distribution packaging
('filesystem' on Arch and Fedora, 'base-files' on Debian); this will
mean the directories are already there on people's systems, which
hopefully will encourage developers to use them. And then we can try to
provide a bit more nudging through the distribution packaging.

-Toke