BPF maps don't work without CONFIG_TRACING/CONFIG_FTRACE

BPF maps don't work without CONFIG_TRACING/CONFIG_FTRACE

[Michael Zimmermann]

Hi,

I'm using a kernel which has TRACING and FTRACE disabled and it looks
like BPF programs are unable to communicate with usespace.
I've reproduced this on aarch64 and x86_64 with both aya-rs's XDP
sample and bcc's "tc_perf_event.py" sample. bcc's sample uses
BPF_PERF_OUTPUT instead of maps though.

Everything seems to run and work correctly, but there's no data being
send to userspace resulting in no log output.
Is that expected or am I running into a weird bug here?

Thanks
Michael

Re: BPF maps don't work without CONFIG_TRACING/CONFIG_FTRACE

[Andrii Nakryiko]

On Tue, May 3, 2022 at 2:40 AM Michael Zimmermann
<sigmaepsilon92@gmail.com> wrote:
>
> Hi,
>
> I'm using a kernel which has TRACING and FTRACE disabled and it looks
> like BPF programs are unable to communicate with usespace.
> I've reproduced this on aarch64 and x86_64 with both aya-rs's XDP
> sample and bcc's "tc_perf_event.py" sample. bcc's sample uses
> BPF_PERF_OUTPUT instead of maps though.
>
> Everything seems to run and work correctly, but there's no data being
> send to userspace resulting in no log output.
> Is that expected or am I running into a weird bug here?
>

You probably need to provide few more details on what you are trying
to do, what you expect to happen and what's actually happening. As it
is it's hard to provide any useful help.

> Thanks
> Michael

Re: BPF maps don't work without CONFIG_TRACING/CONFIG_FTRACE

[Michael Zimmermann]

Thank you for your answer.
What I'm ultimately trying to do is: Use aya-rs to watch egress on a
network interface and notify userspace through a map (for certain IPs
only).

In my actual use case, the userspace is supposed to do more complex
stuff but for testing I simply logged the receival of a message
through the BPF map on the console. And that is what I expect to
happen and which does happen as long as CONFIG_TRACING/CONFIG_FTRACE
are active. If not, I simply never receive any messages on any map.

I've also tried this using an XDP program which sends a message every
time it sees a packet. And while the program seemed to be
working(since it did block certain traffic), I never saw any data in
the map when those configs were disabled.

Also, I'm giving you two configs(tracing and ftrace) since the other
one seems to get y-selected automatically if one of them is active.

On Tue, May 10, 2022 at 2:00 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Tue, May 3, 2022 at 2:40 AM Michael Zimmermann
> <sigmaepsilon92@gmail.com> wrote:
> >
> > Hi,
> >
> > I'm using a kernel which has TRACING and FTRACE disabled and it looks
> > like BPF programs are unable to communicate with usespace.
> > I've reproduced this on aarch64 and x86_64 with both aya-rs's XDP
> > sample and bcc's "tc_perf_event.py" sample. bcc's sample uses
> > BPF_PERF_OUTPUT instead of maps though.
> >
> > Everything seems to run and work correctly, but there's no data being
> > send to userspace resulting in no log output.
> > Is that expected or am I running into a weird bug here?
> >
>
> You probably need to provide few more details on what you are trying
> to do, what you expect to happen and what's actually happening. As it
> is it's hard to provide any useful help.
>
> > Thanks
> > Michael

Re: BPF maps don't work without CONFIG_TRACING/CONFIG_FTRACE

[Andrii Nakryiko]

On Mon, May 9, 2022 at 10:12 PM Michael Zimmermann
<sigmaepsilon92@gmail.com> wrote:
>
> Thank you for your answer.
> What I'm ultimately trying to do is: Use aya-rs to watch egress on a
> network interface and notify userspace through a map (for certain IPs
> only).
>
> In my actual use case, the userspace is supposed to do more complex
> stuff but for testing I simply logged the receival of a message
> through the BPF map on the console. And that is what I expect to
> happen and which does happen as long as CONFIG_TRACING/CONFIG_FTRACE
> are active. If not, I simply never receive any messages on any map.
>
> I've also tried this using an XDP program which sends a message every
> time it sees a packet. And while the program seemed to be
> working(since it did block certain traffic), I never saw any data in
> the map when those configs were disabled.
>
> Also, I'm giving you two configs(tracing and ftrace) since the other
> one seems to get y-selected automatically if one of them is active.

Please don't top post, reply inline instead.

I don't think we have enough to investigate here, even "receive any
messages on any map" is so ambiguous that it's hard to even guess what
you are really trying to do. BPF maps are not sending/receiving
messages. So please provide some pieces of code and what you are doing
to check. CONFIG_TRACING and CONFIG_FTRACE shouldn't have any effect
on functioning of BPF maps, so it's most probably that you are doing
something besides BPF map update/lookup, but you don't provide enough
information to check anything.

>
> On Tue, May 10, 2022 at 2:00 AM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Tue, May 3, 2022 at 2:40 AM Michael Zimmermann
> > <sigmaepsilon92@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I'm using a kernel which has TRACING and FTRACE disabled and it looks
> > > like BPF programs are unable to communicate with usespace.
> > > I've reproduced this on aarch64 and x86_64 with both aya-rs's XDP
> > > sample and bcc's "tc_perf_event.py" sample. bcc's sample uses
> > > BPF_PERF_OUTPUT instead of maps though.
> > >
> > > Everything seems to run and work correctly, but there's no data being
> > > send to userspace resulting in no log output.
> > > Is that expected or am I running into a weird bug here?
> > >
> >
> > You probably need to provide few more details on what you are trying
> > to do, what you expect to happen and what's actually happening. As it
> > is it's hard to provide any useful help.
> >
> > > Thanks
> > > Michael

Re: BPF maps don't work without CONFIG_TRACING/CONFIG_FTRACE

[Michael Zimmermann]

On Thu, May 12, 2022 at 5:21 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Mon, May 9, 2022 at 10:12 PM Michael Zimmermann
> <sigmaepsilon92@gmail.com> wrote:
> >
> > Thank you for your answer.
> > What I'm ultimately trying to do is: Use aya-rs to watch egress on a
> > network interface and notify userspace through a map (for certain IPs
> > only).
> >
> > In my actual use case, the userspace is supposed to do more complex
> > stuff but for testing I simply logged the receival of a message
> > through the BPF map on the console. And that is what I expect to
> > happen and which does happen as long as CONFIG_TRACING/CONFIG_FTRACE
> > are active. If not, I simply never receive any messages on any map.
> >
> > I've also tried this using an XDP program which sends a message every
> > time it sees a packet. And while the program seemed to be
> > working(since it did block certain traffic), I never saw any data in
> > the map when those configs were disabled.
> >
> > Also, I'm giving you two configs(tracing and ftrace) since the other
> > one seems to get y-selected automatically if one of them is active.
>
> Please don't top post, reply inline instead.
Sorry for that, GMail does that by default and even hides that it's
quoting at all.

>
> I don't think we have enough to investigate here, even "receive any
> messages on any map" is so ambiguous that it's hard to even guess what
> you are really trying to do. BPF maps are not sending/receiving
> messages. So please provide some pieces of code and what you are doing
> to check. CONFIG_TRACING and CONFIG_FTRACE shouldn't have any effect
> on functioning of BPF maps, so it's most probably that you are doing
> something besides BPF map update/lookup, but you don't provide enough
> information to check anything.

An aya project I tested where I don't receive any events:
https://github.com/aya-rs/book/tree/6b52a6fac5fa3e5a1165f98591b2eaff9692048a/examples/myapp-03

A bcc project where I don't receive any ping event:
https://github.com/iovisor/bcc/blob/2df19cd16ff69429faa8b7b86d6630ba35907734/examples/networking/tc_perf_event.py

If that's too abstract and far away from the kernel for you to figure
out what's going on I'd have to dig deeper into how bcc or aya work
internally which is not that easy to do.
>
> >
> > On Tue, May 10, 2022 at 2:00 AM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Tue, May 3, 2022 at 2:40 AM Michael Zimmermann
> > > <sigmaepsilon92@gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'm using a kernel which has TRACING and FTRACE disabled and it looks
> > > > like BPF programs are unable to communicate with usespace.
> > > > I've reproduced this on aarch64 and x86_64 with both aya-rs's XDP
> > > > sample and bcc's "tc_perf_event.py" sample. bcc's sample uses
> > > > BPF_PERF_OUTPUT instead of maps though.
> > > >
> > > > Everything seems to run and work correctly, but there's no data being
> > > > send to userspace resulting in no log output.
> > > > Is that expected or am I running into a weird bug here?
> > > >
> > >
> > > You probably need to provide few more details on what you are trying
> > > to do, what you expect to happen and what's actually happening. As it
> > > is it's hard to provide any useful help.
> > >
> > > > Thanks
> > > > Michael

Re: BPF maps don't work without CONFIG_TRACING/CONFIG_FTRACE

[Toke Høiland-Jørgensen]

Michael Zimmermann <sigmaepsilon92@gmail.com> writes:

> On Thu, May 12, 2022 at 5:21 AM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
>>
>> On Mon, May 9, 2022 at 10:12 PM Michael Zimmermann
>> <sigmaepsilon92@gmail.com> wrote:
>> >
>> > Thank you for your answer.
>> > What I'm ultimately trying to do is: Use aya-rs to watch egress on a
>> > network interface and notify userspace through a map (for certain IPs
>> > only).
>> >
>> > In my actual use case, the userspace is supposed to do more complex
>> > stuff but for testing I simply logged the receival of a message
>> > through the BPF map on the console. And that is what I expect to
>> > happen and which does happen as long as CONFIG_TRACING/CONFIG_FTRACE
>> > are active. If not, I simply never receive any messages on any map.
>> >
>> > I've also tried this using an XDP program which sends a message every
>> > time it sees a packet. And while the program seemed to be
>> > working(since it did block certain traffic), I never saw any data in
>> > the map when those configs were disabled.
>> >
>> > Also, I'm giving you two configs(tracing and ftrace) since the other
>> > one seems to get y-selected automatically if one of them is active.
>>
>> Please don't top post, reply inline instead.
> Sorry for that, GMail does that by default and even hides that it's
> quoting at all.
>
>>
>> I don't think we have enough to investigate here, even "receive any
>> messages on any map" is so ambiguous that it's hard to even guess what
>> you are really trying to do. BPF maps are not sending/receiving
>> messages. So please provide some pieces of code and what you are doing
>> to check. CONFIG_TRACING and CONFIG_FTRACE shouldn't have any effect
>> on functioning of BPF maps, so it's most probably that you are doing
>> something besides BPF map update/lookup, but you don't provide enough
>> information to check anything.
>
> An aya project I tested where I don't receive any events:
> https://github.com/aya-rs/book/tree/6b52a6fac5fa3e5a1165f98591b2eaff9692048a/examples/myapp-03

It's using a PERF_EVENT_ARRAY map to send events to userspace. This
requires CONFIG_BPF_EVENTS to work, which depends on CONFIG_PERF_EVENTS.
Not sure if this depends on CONFIG_TRACING specifically, but maybe you
disabled PERF_EVENTS as well?

-Toke

Re: BPF maps don't work without CONFIG_TRACING/CONFIG_FTRACE

[Michael Zimmermann]

On Thu, May 12, 2022 at 9:48 AM Toke Høiland-Jørgensen <toke@kernel.org> wrote:
>
> Michael Zimmermann <sigmaepsilon92@gmail.com> writes:
>
> > On Thu, May 12, 2022 at 5:21 AM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> >>
> >> On Mon, May 9, 2022 at 10:12 PM Michael Zimmermann
> >> <sigmaepsilon92@gmail.com> wrote:
> >> >
> >> > Thank you for your answer.
> >> > What I'm ultimately trying to do is: Use aya-rs to watch egress on a
> >> > network interface and notify userspace through a map (for certain IPs
> >> > only).
> >> >
> >> > In my actual use case, the userspace is supposed to do more complex
> >> > stuff but for testing I simply logged the receival of a message
> >> > through the BPF map on the console. And that is what I expect to
> >> > happen and which does happen as long as CONFIG_TRACING/CONFIG_FTRACE
> >> > are active. If not, I simply never receive any messages on any map.
> >> >
> >> > I've also tried this using an XDP program which sends a message every
> >> > time it sees a packet. And while the program seemed to be
> >> > working(since it did block certain traffic), I never saw any data in
> >> > the map when those configs were disabled.
> >> >
> >> > Also, I'm giving you two configs(tracing and ftrace) since the other
> >> > one seems to get y-selected automatically if one of them is active.
> >>
> >> Please don't top post, reply inline instead.
> > Sorry for that, GMail does that by default and even hides that it's
> > quoting at all.
> >
> >>
> >> I don't think we have enough to investigate here, even "receive any
> >> messages on any map" is so ambiguous that it's hard to even guess what
> >> you are really trying to do. BPF maps are not sending/receiving
> >> messages. So please provide some pieces of code and what you are doing
> >> to check. CONFIG_TRACING and CONFIG_FTRACE shouldn't have any effect
> >> on functioning of BPF maps, so it's most probably that you are doing
> >> something besides BPF map update/lookup, but you don't provide enough
> >> information to check anything.
> >
> > An aya project I tested where I don't receive any events:
> > https://github.com/aya-rs/book/tree/6b52a6fac5fa3e5a1165f98591b2eaff9692048a/examples/myapp-03
>
> It's using a PERF_EVENT_ARRAY map to send events to userspace. This
> requires CONFIG_BPF_EVENTS to work, which depends on CONFIG_PERF_EVENTS.
> Not sure if this depends on CONFIG_TRACING specifically, but maybe you
> disabled PERF_EVENTS as well?

PERF_EVENTS is enabled in both my working and my broken config.
But both directly and through others, BPF_EVENTS depends on FTRACE
(which then also selects TRACING).

So is this some weird dependency chain or expected behavior?
If it's expected, are there alternatives to achieve similar
functionality or do I have to convince my distro to enable tracing
support?

>
> -Toke

Re: BPF maps don't work without CONFIG_TRACING/CONFIG_FTRACE

[Toke Høiland-Jørgensen]

Michael Zimmermann <sigmaepsilon92@gmail.com> writes:

> On Thu, May 12, 2022 at 9:48 AM Toke Høiland-Jørgensen <toke@kernel.org> wrote:
>>
>> Michael Zimmermann <sigmaepsilon92@gmail.com> writes:
>>
>> > On Thu, May 12, 2022 at 5:21 AM Andrii Nakryiko
>> > <andrii.nakryiko@gmail.com> wrote:
>> >>
>> >> On Mon, May 9, 2022 at 10:12 PM Michael Zimmermann
>> >> <sigmaepsilon92@gmail.com> wrote:
>> >> >
>> >> > Thank you for your answer.
>> >> > What I'm ultimately trying to do is: Use aya-rs to watch egress on a
>> >> > network interface and notify userspace through a map (for certain IPs
>> >> > only).
>> >> >
>> >> > In my actual use case, the userspace is supposed to do more complex
>> >> > stuff but for testing I simply logged the receival of a message
>> >> > through the BPF map on the console. And that is what I expect to
>> >> > happen and which does happen as long as CONFIG_TRACING/CONFIG_FTRACE
>> >> > are active. If not, I simply never receive any messages on any map.
>> >> >
>> >> > I've also tried this using an XDP program which sends a message every
>> >> > time it sees a packet. And while the program seemed to be
>> >> > working(since it did block certain traffic), I never saw any data in
>> >> > the map when those configs were disabled.
>> >> >
>> >> > Also, I'm giving you two configs(tracing and ftrace) since the other
>> >> > one seems to get y-selected automatically if one of them is active.
>> >>
>> >> Please don't top post, reply inline instead.
>> > Sorry for that, GMail does that by default and even hides that it's
>> > quoting at all.
>> >
>> >>
>> >> I don't think we have enough to investigate here, even "receive any
>> >> messages on any map" is so ambiguous that it's hard to even guess what
>> >> you are really trying to do. BPF maps are not sending/receiving
>> >> messages. So please provide some pieces of code and what you are doing
>> >> to check. CONFIG_TRACING and CONFIG_FTRACE shouldn't have any effect
>> >> on functioning of BPF maps, so it's most probably that you are doing
>> >> something besides BPF map update/lookup, but you don't provide enough
>> >> information to check anything.
>> >
>> > An aya project I tested where I don't receive any events:
>> > https://github.com/aya-rs/book/tree/6b52a6fac5fa3e5a1165f98591b2eaff9692048a/examples/myapp-03
>>
>> It's using a PERF_EVENT_ARRAY map to send events to userspace. This
>> requires CONFIG_BPF_EVENTS to work, which depends on CONFIG_PERF_EVENTS.
>> Not sure if this depends on CONFIG_TRACING specifically, but maybe you
>> disabled PERF_EVENTS as well?
>
> PERF_EVENTS is enabled in both my working and my broken config.
> But both directly and through others, BPF_EVENTS depends on FTRACE
> (which then also selects TRACING).
>
> So is this some weird dependency chain or expected behavior?
> If it's expected, are there alternatives to achieve similar
> functionality or do I have to convince my distro to enable tracing
> support?

Andrii can probably answer for sure, but I *think* this is expected.
There's the BPF_MAP_TYPE_RINGBUF which can be used as an alternative;
not sure what the dependencies are for that, but from a cursory glance
it looks like it doesn't depend on anything in TRACING...

-Toke