Best way to share maps between multiple files/objects?

Best way to share maps between multiple files/objects?

[Grant Seltzer Richman]

Hi folks,

I want to organize my BPF programs so that I can load them
individually. I want this so that if loading one fails (because of
lack of kernel support for BPF features), I can load a fall-back
replacement program. To do so, I've organized the BPF programs into
their own source code files and compiled them individually. Each BPF
program references what is supposed to be the same ringbuffer. Using
libbpf I open them and attempt to load each in order.

My question is, how am I supposed to share maps such as ringbuffers
between them? If I have identical map definitions in each, they have
their own file descriptors. Is the best way to call
`bpf_map__reuse_fd()` on each handle of the maps in each BPF object?

I'd also take advice on how to better achieve my overall goal of being
able to load programs individually!

Thanks so much for your help,
Grant Seltzer

Re: Best way to share maps between multiple files/objects?

[Dave Marchevsky]

On 11/10/22 7:32 PM, Grant Seltzer Richman wrote:
> Hi folks,
> 
> I want to organize my BPF programs so that I can load them
> individually. I want this so that if loading one fails (because of
> lack of kernel support for BPF features), I can load a fall-back
> replacement program. To do so, I've organized the BPF programs into
> their own source code files and compiled them individually. Each BPF
> program references what is supposed to be the same ringbuffer. Using
> libbpf I open them and attempt to load each in order.
> 
> My question is, how am I supposed to share maps such as ringbuffers
> between them? If I have identical map definitions in each, they have
> their own file descriptors. Is the best way to call
> `bpf_map__reuse_fd()` on each handle of the maps in each BPF object?

Sounds like each of the source files have the exact same map definitions,
including name? And each is compiled into a separate BPF object?

If so, adding __uint(pinning, LIBBPF_PIN_BY_NAME); to
each definition will probably be the easiest way to get the map reuse
behavior you want. The first bpf object in the set that successfully loads
will pin its maps by name in /sys/fs/bpf and future objects which load same
maps will reuse them instead of creating new maps.

selftests/bpf/progs/test_pinning.c demonstrates this behavior.

I'm curious, though: is this a single BPF program with various fallbacks,
with goal of running only one? Or a set of N programs working together using
same maps, each of which might have fallbacks, with goal of running some
version of all N programs?

> I'd also take advice on how to better achieve my overall goal of being
> able to load programs individually!

You can group each program together with its fallbacks in the same
source file / BPF object by disabling autoload for all variants of the
program via SEC("?foobar") syntax. Then in userspace you could turn
autoload on for the first version you'd like to try after opening
the BPF object, try loading the object, try with 2nd variant if that
fails, etc.

selftests/bpf/progs/dynptr_fail.c + verify_fail function in
selftests/bpf/prog_tests/dynptr.c is an example of this pattern.

> Thanks so much for your help,
> Grant Seltzer

Re: Best way to share maps between multiple files/objects?

[Grant Seltzer Richman]

On Fri, Nov 11, 2022 at 2:34 AM Dave Marchevsky <davemarchevsky@meta.com> wrote:
>
> On 11/10/22 7:32 PM, Grant Seltzer Richman wrote:
> > Hi folks,
> >
> > I want to organize my BPF programs so that I can load them
> > individually. I want this so that if loading one fails (because of
> > lack of kernel support for BPF features), I can load a fall-back
> > replacement program. To do so, I've organized the BPF programs into
> > their own source code files and compiled them individually. Each BPF
> > program references what is supposed to be the same ringbuffer. Using
> > libbpf I open them and attempt to load each in order.
> >
> > My question is, how am I supposed to share maps such as ringbuffers
> > between them? If I have identical map definitions in each, they have
> > their own file descriptors. Is the best way to call
> > `bpf_map__reuse_fd()` on each handle of the maps in each BPF object?
>
> Sounds like each of the source files have the exact same map definitions,
> including name? And each is compiled into a separate BPF object?
>
> If so, adding __uint(pinning, LIBBPF_PIN_BY_NAME); to
> each definition will probably be the easiest way to get the map reuse
> behavior you want. The first bpf object in the set that successfully loads
> will pin its maps by name in /sys/fs/bpf and future objects which load same
> maps will reuse them instead of creating new maps.

This worked beautifully, thank you for the suggestion!

>
> selftests/bpf/progs/test_pinning.c demonstrates this behavior.
>
> I'm curious, though: is this a single BPF program with various fallbacks,
> with goal of running only one? Or a set of N programs working together using
> same maps, each of which might have fallbacks, with goal of running some
> version of all N programs?

The latter. We have N programs all sharing M maps. Each program might
have fallbacks but some version should be loaded.

>
> > I'd also take advice on how to better achieve my overall goal of being
> > able to load programs individually!
>
> You can group each program together with its fallbacks in the same
> source file / BPF object by disabling autoload for all variants of the
> program via SEC("?foobar") syntax. Then in userspace you could turn
> autoload on for the first version you'd like to try after opening
> the BPF object, try loading the object, try with 2nd variant if that
> fails, etc.

Thank you for this suggestion as well, but it doesn't seem to work as
I get: `load can't be attempted twice`. Is this a potential bug?
`obj->loaded` is set to true regardless of success in
`bpf_object_load()`

>
> selftests/bpf/progs/dynptr_fail.c + verify_fail function in
> selftests/bpf/prog_tests/dynptr.c is an example of this pattern.
>
> > Thanks so much for your help,
> > Grant Seltzer

Re: Best way to share maps between multiple files/objects?

[Andrii Nakryiko]

On Mon, Nov 14, 2022 at 11:04 AM Grant Seltzer Richman
<grantseltzer@gmail.com> wrote:
>
> On Fri, Nov 11, 2022 at 2:34 AM Dave Marchevsky <davemarchevsky@meta.com> wrote:
> >
> > On 11/10/22 7:32 PM, Grant Seltzer Richman wrote:
> > > Hi folks,
> > >
> > > I want to organize my BPF programs so that I can load them
> > > individually. I want this so that if loading one fails (because of
> > > lack of kernel support for BPF features), I can load a fall-back
> > > replacement program. To do so, I've organized the BPF programs into
> > > their own source code files and compiled them individually. Each BPF
> > > program references what is supposed to be the same ringbuffer. Using
> > > libbpf I open them and attempt to load each in order.
> > >
> > > My question is, how am I supposed to share maps such as ringbuffers
> > > between them? If I have identical map definitions in each, they have
> > > their own file descriptors. Is the best way to call
> > > `bpf_map__reuse_fd()` on each handle of the maps in each BPF object?
> >
> > Sounds like each of the source files have the exact same map definitions,
> > including name? And each is compiled into a separate BPF object?
> >
> > If so, adding __uint(pinning, LIBBPF_PIN_BY_NAME); to
> > each definition will probably be the easiest way to get the map reuse
> > behavior you want. The first bpf object in the set that successfully loads
> > will pin its maps by name in /sys/fs/bpf and future objects which load same
> > maps will reuse them instead of creating new maps.
>
> This worked beautifully, thank you for the suggestion!
>
> >
> > selftests/bpf/progs/test_pinning.c demonstrates this behavior.
> >
> > I'm curious, though: is this a single BPF program with various fallbacks,
> > with goal of running only one? Or a set of N programs working together using
> > same maps, each of which might have fallbacks, with goal of running some
> > version of all N programs?
>
> The latter. We have N programs all sharing M maps. Each program might
> have fallbacks but some version should be loaded.
>
> >
> > > I'd also take advice on how to better achieve my overall goal of being
> > > able to load programs individually!
> >
> > You can group each program together with its fallbacks in the same
> > source file / BPF object by disabling autoload for all variants of the
> > program via SEC("?foobar") syntax. Then in userspace you could turn
> > autoload on for the first version you'd like to try after opening
> > the BPF object, try loading the object, try with 2nd variant if that
> > fails, etc.
>
> Thank you for this suggestion as well, but it doesn't seem to work as
> I get: `load can't be attempted twice`. Is this a potential bug?
> `obj->loaded` is set to true regardless of success in
> `bpf_object_load()`

set_autoload() should be set before bpf_object__load().
bpf_object__load() can't be loaded twice.

I'd do proper detection of what kernel features are available and
set_autoload() correspondingly. Then do bpf_object__load() once. And
completely avoid pinning, if I could.

>
> >
> > selftests/bpf/progs/dynptr_fail.c + verify_fail function in
> > selftests/bpf/prog_tests/dynptr.c is an example of this pattern.
> >
> > > Thanks so much for your help,
> > > Grant Seltzer