eBPF CO-RE cross-compilation for 32-bit ARM platforms

eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Jakov Petrina]

Hi everyone,

recently we have begun extensive research into eBPF and related 
technologies. Seeking an easier development process, we have switched 
over to using the eBPF CO-RE [0] approach internally which has enabled 
us to simplify most aspects of eBPF development, especially those 
related to cross-compilation.

However, as part of these efforts we have stumbled upon several problems 
that we feel would benefit from a community discussion where we may 
share our solutions and discuss alternatives moving forward.

As a reference point, we have started researching and modifying several 
eBPF CO-RE samples that have been developed or migrated from existing 
`bcc` tooling. Most notable examples are those present in `bcc`'s 
`libbpf-tools` directory [1]. Some of these samples have just recently 
been converted to respective eBPF CO-RE variants, of which the 
`tcpconnect` tracing sample has proven to be very interesting.

First showstopper for cross-compiling aforementioned example on the ARM 
32-bit platform has been with regards to generation of the required 
`vmlinux.h` kernel header from the BTF information. More specifically, 
our initial approach to have e.g. a compilation target dependency which 
would invoke `bpftool` at configure time was not appropriate due to 
several issues: a) CO-RE requires host kernel to have been compiled in 
such a way to expose BTF information which may not available, and b) the 
generated `vmlinux.h` was actually architecture-specific.

The second point proved interesting because `tcpconnect` makes use of 
the `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs 
*ctx` as the first function parameter. The `pt_regs` structure is 
defined by the kernel and is architecture-specific. Since `libbpf` does 
have architecture-specific conditionals, pairing it with an "invalid" 
`vmlinux.h` resulted in cross-compilation failure as `libbpf` provided 
macros that work with ARM `pt_regs`, and `vmlinux.h` had an x86 
`pt_regs` definition. To resolve this issue, we have resorted to 
including pre-generated `<arch>_vmlinux.h` files in our CO-RE build system.

However, there are certainly drawbacks to this approach: a) (relatively) 
large file size of the generated headers, b) regular maintenance to 
re-generate the header files for various architectures and kernel 
versions, and c) incompatible definitions being generated, to name a 
few. This last point relates to the the fact that our `aarch64`/`arm64` 
kernel generates the following definition using `bpftool`, which has 
resulted in compilation failure:

```
typedef __Poly8_t poly8x16_t[16];
```

AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. 
We have opted to comment out this line as there was no additional 
`poly8x16_t` usage in the header file.

Given various issues we have encountered so far (among which is a kernel 
panic/crash on a specific device), additional input and feedback 
regarding cross-compilation of the eBPF utilities would be greatly 
appreciated.

[0]
https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html
[1] https://github.com/iovisor/bcc/tree/master/libbpf-tools

Best regards,

Sartura eBPF Team

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Jean-Philippe Brucker]

Hi,

[Adding the linux-arm-kernel list on Cc]

On Fri, Aug 07, 2020 at 04:20:58PM +0200, Jakov Petrina wrote:
> Hi everyone,
> 
> recently we have begun extensive research into eBPF and related
> technologies. Seeking an easier development process, we have switched over
> to using the eBPF CO-RE [0] approach internally which has enabled us to
> simplify most aspects of eBPF development, especially those related to
> cross-compilation.
> 
> However, as part of these efforts we have stumbled upon several problems
> that we feel would benefit from a community discussion where we may share
> our solutions and discuss alternatives moving forward.
> 
> As a reference point, we have started researching and modifying several eBPF
> CO-RE samples that have been developed or migrated from existing `bcc`
> tooling. Most notable examples are those present in `bcc`'s `libbpf-tools`
> directory [1]. Some of these samples have just recently been converted to
> respective eBPF CO-RE variants, of which the `tcpconnect` tracing sample has
> proven to be very interesting.
> 
> First showstopper for cross-compiling aforementioned example on the ARM
> 32-bit platform has been with regards to generation of the required
> `vmlinux.h` kernel header from the BTF information. More specifically, our
> initial approach to have e.g. a compilation target dependency which would
> invoke `bpftool` at configure time was not appropriate due to several
> issues: a) CO-RE requires host kernel to have been compiled in such a way to
> expose BTF information which may not available, and b) the generated
> `vmlinux.h` was actually architecture-specific.
> 
> The second point proved interesting because `tcpconnect` makes use of the
> `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs *ctx` as
> the first function parameter. The `pt_regs` structure is defined by the
> kernel and is architecture-specific. Since `libbpf` does have
> architecture-specific conditionals, pairing it with an "invalid" `vmlinux.h`
> resulted in cross-compilation failure as `libbpf` provided macros that work
> with ARM `pt_regs`, and `vmlinux.h` had an x86 `pt_regs` definition. To
> resolve this issue, we have resorted to including pre-generated
> `<arch>_vmlinux.h` files in our CO-RE build system.
> 
> However, there are certainly drawbacks to this approach: a) (relatively)
> large file size of the generated headers, b) regular maintenance to
> re-generate the header files for various architectures and kernel versions,
> and c) incompatible definitions being generated, to name a few. This last
> point relates to the the fact that our `aarch64`/`arm64` kernel generates
> the following definition using `bpftool`, which has resulted in compilation
> failure:
> 
> ```
> typedef __Poly8_t poly8x16_t[16];
> ```
> 
> AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We
> have opted to comment out this line as there was no additional `poly8x16_t`
> usage in the header file.

It looks like this "__Poly8_t" type is internal to GCC (provided in
arm_neon.h) and clang has its own internals. I managed to reproduce this
with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
the moment. Maybe libbpf should generate defines to translate these
intrinsics between clang and gcc? Not very elegant. I'll take another
look next week.

> Given various issues we have encountered so far (among which is a kernel
> panic/crash on a specific device), additional input and feedback regarding
> cross-compilation of the eBPF utilities would be greatly appreciated.

I don't know if there is a room for improvement regarding your a) and b)
points, as I think the added complexity is inherent to cross-building. But
kernel crashes definitely need to be fixed, as well as the above problem.

Thanks,
Jean

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Andrii Nakryiko]

On Fri, Aug 7, 2020 at 10:24 AM Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> Hi,
>
> [Adding the linux-arm-kernel list on Cc]
>
> On Fri, Aug 07, 2020 at 04:20:58PM +0200, Jakov Petrina wrote:
> > Hi everyone,
> >
> > recently we have begun extensive research into eBPF and related
> > technologies. Seeking an easier development process, we have switched over
> > to using the eBPF CO-RE [0] approach internally which has enabled us to
> > simplify most aspects of eBPF development, especially those related to
> > cross-compilation.
> >
> > However, as part of these efforts we have stumbled upon several problems
> > that we feel would benefit from a community discussion where we may share
> > our solutions and discuss alternatives moving forward.
> >
> > As a reference point, we have started researching and modifying several eBPF
> > CO-RE samples that have been developed or migrated from existing `bcc`
> > tooling. Most notable examples are those present in `bcc`'s `libbpf-tools`
> > directory [1]. Some of these samples have just recently been converted to
> > respective eBPF CO-RE variants, of which the `tcpconnect` tracing sample has
> > proven to be very interesting.
> >
> > First showstopper for cross-compiling aforementioned example on the ARM
> > 32-bit platform has been with regards to generation of the required
> > `vmlinux.h` kernel header from the BTF information. More specifically, our
> > initial approach to have e.g. a compilation target dependency which would
> > invoke `bpftool` at configure time was not appropriate due to several
> > issues: a) CO-RE requires host kernel to have been compiled in such a way to
> > expose BTF information which may not available, and b) the generated
> > `vmlinux.h` was actually architecture-specific.
> >
> > The second point proved interesting because `tcpconnect` makes use of the
> > `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs *ctx` as
> > the first function parameter. The `pt_regs` structure is defined by the
> > kernel and is architecture-specific. Since `libbpf` does have
> > architecture-specific conditionals, pairing it with an "invalid" `vmlinux.h`
> > resulted in cross-compilation failure as `libbpf` provided macros that work
> > with ARM `pt_regs`, and `vmlinux.h` had an x86 `pt_regs` definition. To
> > resolve this issue, we have resorted to including pre-generated
> > `<arch>_vmlinux.h` files in our CO-RE build system.
> >
> > However, there are certainly drawbacks to this approach: a) (relatively)
> > large file size of the generated headers, b) regular maintenance to
> > re-generate the header files for various architectures and kernel versions,
> > and c) incompatible definitions being generated, to name a few. This last
> > point relates to the the fact that our `aarch64`/`arm64` kernel generates
> > the following definition using `bpftool`, which has resulted in compilation
> > failure:
> >
> > ```
> > typedef __Poly8_t poly8x16_t[16];
> > ```
> >
> > AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We
> > have opted to comment out this line as there was no additional `poly8x16_t`
> > usage in the header file.
>
> It looks like this "__Poly8_t" type is internal to GCC (provided in
> arm_neon.h) and clang has its own internals. I managed to reproduce this
> with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
> the moment. Maybe libbpf should generate defines to translate these
> intrinsics between clang and gcc? Not very elegant. I'll take another
> look next week.

libbpf is already blacklisting __builtin_va_list for GCC, so we can
just add __Poly8_t to the list. See [0].
Are there any other types like that? If you guys can provide me this,
I'll gladly update libbpf to take those compiler-provided
types/built-ins into account.

  [0] https://github.com/torvalds/linux/blob/master/tools/lib/bpf/btf_dump.c#L585-L598

>
> > Given various issues we have encountered so far (among which is a kernel
> > panic/crash on a specific device), additional input and feedback regarding
> > cross-compilation of the eBPF utilities would be greatly appreciated.
>
> I don't know if there is a room for improvement regarding your a) and b)
> points, as I think the added complexity is inherent to cross-building. But
> kernel crashes definitely need to be fixed, as well as the above problem.
>
> Thanks,
> Jean

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Alexei Starovoitov]

On Fri, Aug 7, 2020 at 11:41 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Fri, Aug 7, 2020 at 10:24 AM Jean-Philippe Brucker
> <jean-philippe@linaro.org> wrote:
> >
> > Hi,
> >
> > [Adding the linux-arm-kernel list on Cc]
> >
> > On Fri, Aug 07, 2020 at 04:20:58PM +0200, Jakov Petrina wrote:
> > > Hi everyone,
> > >
> > > recently we have begun extensive research into eBPF and related
> > > technologies. Seeking an easier development process, we have switched over
> > > to using the eBPF CO-RE [0] approach internally which has enabled us to
> > > simplify most aspects of eBPF development, especially those related to
> > > cross-compilation.
> > >
> > > However, as part of these efforts we have stumbled upon several problems
> > > that we feel would benefit from a community discussion where we may share
> > > our solutions and discuss alternatives moving forward.
> > >
> > > As a reference point, we have started researching and modifying several eBPF
> > > CO-RE samples that have been developed or migrated from existing `bcc`
> > > tooling. Most notable examples are those present in `bcc`'s `libbpf-tools`
> > > directory [1]. Some of these samples have just recently been converted to
> > > respective eBPF CO-RE variants, of which the `tcpconnect` tracing sample has
> > > proven to be very interesting.
> > >
> > > First showstopper for cross-compiling aforementioned example on the ARM
> > > 32-bit platform has been with regards to generation of the required
> > > `vmlinux.h` kernel header from the BTF information. More specifically, our
> > > initial approach to have e.g. a compilation target dependency which would
> > > invoke `bpftool` at configure time was not appropriate due to several
> > > issues: a) CO-RE requires host kernel to have been compiled in such a way to
> > > expose BTF information which may not available, and b) the generated
> > > `vmlinux.h` was actually architecture-specific.
> > >
> > > The second point proved interesting because `tcpconnect` makes use of the
> > > `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs *ctx` as
> > > the first function parameter. The `pt_regs` structure is defined by the
> > > kernel and is architecture-specific. Since `libbpf` does have
> > > architecture-specific conditionals, pairing it with an "invalid" `vmlinux.h`
> > > resulted in cross-compilation failure as `libbpf` provided macros that work
> > > with ARM `pt_regs`, and `vmlinux.h` had an x86 `pt_regs` definition. To
> > > resolve this issue, we have resorted to including pre-generated
> > > `<arch>_vmlinux.h` files in our CO-RE build system.
> > >
> > > However, there are certainly drawbacks to this approach: a) (relatively)
> > > large file size of the generated headers, b) regular maintenance to
> > > re-generate the header files for various architectures and kernel versions,
> > > and c) incompatible definitions being generated, to name a few. This last
> > > point relates to the the fact that our `aarch64`/`arm64` kernel generates
> > > the following definition using `bpftool`, which has resulted in compilation
> > > failure:
> > >
> > > ```
> > > typedef __Poly8_t poly8x16_t[16];
> > > ```
> > >
> > > AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We
> > > have opted to comment out this line as there was no additional `poly8x16_t`
> > > usage in the header file.
> >
> > It looks like this "__Poly8_t" type is internal to GCC (provided in
> > arm_neon.h) and clang has its own internals. I managed to reproduce this
> > with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
> > the moment. Maybe libbpf should generate defines to translate these
> > intrinsics between clang and gcc? Not very elegant. I'll take another
> > look next week.
>
> libbpf is already blacklisting __builtin_va_list for GCC, so we can
> just add __Poly8_t to the list. See [0].
> Are there any other types like that? If you guys can provide me this,
> I'll gladly update libbpf to take those compiler-provided
> types/built-ins into account.

Shouldn't __Int8x16_t and friends cause the same trouble?
There is a bunch more in gcc/config/arm/arm-simd-builtin-types.def.
May be there is a way to detect compiler builtin types by pattern matching
their dwarf/btf shape and skip them automatically?
The simplest, of course, is to only add a few that caused this known
trouble to blocklist.

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Jean-Philippe Brucker]

On Fri, Aug 07, 2020 at 01:54:02PM -0700, Alexei Starovoitov wrote:
[...]
> > > > ```
> > > > typedef __Poly8_t poly8x16_t[16];
> > > > ```
> > > >
> > > > AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We
> > > > have opted to comment out this line as there was no additional `poly8x16_t`
> > > > usage in the header file.
> > >
> > > It looks like this "__Poly8_t" type is internal to GCC (provided in
> > > arm_neon.h) and clang has its own internals. I managed to reproduce this
> > > with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
> > > the moment. Maybe libbpf should generate defines to translate these
> > > intrinsics between clang and gcc? Not very elegant. I'll take another
> > > look next week.
> >
> > libbpf is already blacklisting __builtin_va_list for GCC, so we can
> > just add __Poly8_t to the list. See [0].
> > Are there any other types like that? If you guys can provide me this,
> > I'll gladly update libbpf to take those compiler-provided
> > types/built-ins into account.
> 
> Shouldn't __Int8x16_t and friends cause the same trouble?

I think these do get properly defined, for example in my vmlinux.h:

	typedef signed char int8x16_t[16];

From a cursory reading of the "ARM C Language Extension" doc (IHI0053D) it
looks like only the poly8/16/64/128_t types are unspecified. It's safe to
drop them as long as they're not used in structs or function parameters,
but I sent a more generic fix [1] that copies the clang defintions. When
building the kernel with clang, the polyX_t types do get typedefs.

Thanks,
Jean

[1] https://lore.kernel.org/bpf/20200810122835.2309026-1-jean-philippe@linaro.org/

> There is a bunch more in gcc/config/arm/arm-simd-builtin-types.def.
> May be there is a way to detect compiler builtin types by pattern matching
> their dwarf/btf shape and skip them automatically?
> The simplest, of course, is to only add a few that caused this known
> trouble to blocklist.

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Andrii Nakryiko]

On Mon, Aug 10, 2020 at 5:58 AM Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> On Fri, Aug 07, 2020 at 01:54:02PM -0700, Alexei Starovoitov wrote:
> [...]
> > > > > ```
> > > > > typedef __Poly8_t poly8x16_t[16];
> > > > > ```
> > > > >
> > > > > AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We
> > > > > have opted to comment out this line as there was no additional `poly8x16_t`
> > > > > usage in the header file.
> > > >
> > > > It looks like this "__Poly8_t" type is internal to GCC (provided in
> > > > arm_neon.h) and clang has its own internals. I managed to reproduce this
> > > > with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
> > > > the moment. Maybe libbpf should generate defines to translate these
> > > > intrinsics between clang and gcc? Not very elegant. I'll take another
> > > > look next week.
> > >
> > > libbpf is already blacklisting __builtin_va_list for GCC, so we can
> > > just add __Poly8_t to the list. See [0].
> > > Are there any other types like that? If you guys can provide me this,
> > > I'll gladly update libbpf to take those compiler-provided
> > > types/built-ins into account.
> >
> > Shouldn't __Int8x16_t and friends cause the same trouble?
>
> I think these do get properly defined, for example in my vmlinux.h:
>
>         typedef signed char int8x16_t[16];
>
> From a cursory reading of the "ARM C Language Extension" doc (IHI0053D) it
> looks like only the poly8/16/64/128_t types are unspecified. It's safe to
> drop them as long as they're not used in structs or function parameters,
> but I sent a more generic fix [1] that copies the clang defintions. When
> building the kernel with clang, the polyX_t types do get typedefs.
>
> Thanks,
> Jean
>

Hi Jean,

Would you be so kind to build some simple C repro code that uses those
polyX_t types? Ideally built by both GCC and Clang. And then run
`pahole -J` on them to get .BTF into them as well. If you can share
those two with me, I'd love to look at how DWARF and BTF look like.

I'm, unfortunately, having trouble making something like that to
cross-compile on my x86-64 machine, I've spent a bunch of time already
on this unsuccessfully and it's really frustrating at this point. If
you have an ARM system (or cross-compilation set up properly), it
shouldn't take much time for you, hopefully. Just make sure that those
polyX_t types do make it into DWARF, so, e.g., use them with static
variable or something, e.g.,:

int main() {
    static poly8_t a = 12;
    return a + 10;
}

Or something along those lines. Thanks!

> [1] https://lore.kernel.org/bpf/20200810122835.2309026-1-jean-philippe@linaro.org/
>
> > There is a bunch more in gcc/config/arm/arm-simd-builtin-types.def.
> > May be there is a way to detect compiler builtin types by pattern matching
> > their dwarf/btf shape and skip them automatically?
> > The simplest, of course, is to only add a few that caused this known
> > trouble to blocklist.

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Jean-Philippe Brucker]

On Mon, Aug 10, 2020 at 11:54:54PM -0700, Andrii Nakryiko wrote:
> On Mon, Aug 10, 2020 at 5:58 AM Jean-Philippe Brucker
> <jean-philippe@linaro.org> wrote:
> >
> > On Fri, Aug 07, 2020 at 01:54:02PM -0700, Alexei Starovoitov wrote:
> > [...]
> > > > > > ```
> > > > > > typedef __Poly8_t poly8x16_t[16];
> > > > > > ```
> > > > > >
> > > > > > AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We
> > > > > > have opted to comment out this line as there was no additional `poly8x16_t`
> > > > > > usage in the header file.
> > > > >
> > > > > It looks like this "__Poly8_t" type is internal to GCC (provided in
> > > > > arm_neon.h) and clang has its own internals. I managed to reproduce this
> > > > > with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
> > > > > the moment. Maybe libbpf should generate defines to translate these
> > > > > intrinsics between clang and gcc? Not very elegant. I'll take another
> > > > > look next week.
> > > >
> > > > libbpf is already blacklisting __builtin_va_list for GCC, so we can
> > > > just add __Poly8_t to the list. See [0].
> > > > Are there any other types like that? If you guys can provide me this,
> > > > I'll gladly update libbpf to take those compiler-provided
> > > > types/built-ins into account.
> > >
> > > Shouldn't __Int8x16_t and friends cause the same trouble?
> >
> > I think these do get properly defined, for example in my vmlinux.h:
> >
> >         typedef signed char int8x16_t[16];
> >
> > From a cursory reading of the "ARM C Language Extension" doc (IHI0053D) it
> > looks like only the poly8/16/64/128_t types are unspecified. It's safe to
> > drop them as long as they're not used in structs or function parameters,
> > but I sent a more generic fix [1] that copies the clang defintions. When
> > building the kernel with clang, the polyX_t types do get typedefs.
> >
> > Thanks,
> > Jean
> >
> 
> Hi Jean,
> 
> Would you be so kind to build some simple C repro code that uses those
> polyX_t types? Ideally built by both GCC and Clang. And then run
> `pahole -J` on them to get .BTF into them as well. If you can share
> those two with me, I'd love to look at how DWARF and BTF look like.
> 
> I'm, unfortunately, having trouble making something like that to
> cross-compile on my x86-64 machine, I've spent a bunch of time already
> on this unsuccessfully and it's really frustrating at this point. If
> you have an ARM system (or cross-compilation set up properly), it
> shouldn't take much time for you, hopefully. Just make sure that those
> polyX_t types do make it into DWARF, so, e.g., use them with static
> variable or something, e.g.,:
> 
> int main() {
>     static poly8_t a = 12;
>     return a + 10;
> }
> 
> Or something along those lines. Thanks!

No problem, I put the source and clang+gcc binaries in a tarball here:
https://jpbrucker.net/tmp/test-poly-neon.tar.bz2

These contain all the base types defined by arm_neon.h (minus the new
bfloat16, which I don't think matters at the moment)

Thanks,
Jean

> 
> > [1] https://lore.kernel.org/bpf/20200810122835.2309026-1-jean-philippe@linaro.org/
> >
> > > There is a bunch more in gcc/config/arm/arm-simd-builtin-types.def.
> > > May be there is a way to detect compiler builtin types by pattern matching
> > > their dwarf/btf shape and skip them automatically?
> > > The simplest, of course, is to only add a few that caused this known
> > > trouble to blocklist.

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Andrii Nakryiko]

On Tue, Aug 11, 2020 at 2:54 AM Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> On Mon, Aug 10, 2020 at 11:54:54PM -0700, Andrii Nakryiko wrote:
> > On Mon, Aug 10, 2020 at 5:58 AM Jean-Philippe Brucker
> > <jean-philippe@linaro.org> wrote:
> > >
> > > On Fri, Aug 07, 2020 at 01:54:02PM -0700, Alexei Starovoitov wrote:
> > > [...]
> > > > > > > ```
> > > > > > > typedef __Poly8_t poly8x16_t[16];
> > > > > > > ```
> > > > > > >
> > > > > > > AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We
> > > > > > > have opted to comment out this line as there was no additional `poly8x16_t`
> > > > > > > usage in the header file.
> > > > > >
> > > > > > It looks like this "__Poly8_t" type is internal to GCC (provided in
> > > > > > arm_neon.h) and clang has its own internals. I managed to reproduce this
> > > > > > with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
> > > > > > the moment. Maybe libbpf should generate defines to translate these
> > > > > > intrinsics between clang and gcc? Not very elegant. I'll take another
> > > > > > look next week.
> > > > >
> > > > > libbpf is already blacklisting __builtin_va_list for GCC, so we can
> > > > > just add __Poly8_t to the list. See [0].
> > > > > Are there any other types like that? If you guys can provide me this,
> > > > > I'll gladly update libbpf to take those compiler-provided
> > > > > types/built-ins into account.
> > > >
> > > > Shouldn't __Int8x16_t and friends cause the same trouble?
> > >
> > > I think these do get properly defined, for example in my vmlinux.h:
> > >
> > >         typedef signed char int8x16_t[16];
> > >
> > > From a cursory reading of the "ARM C Language Extension" doc (IHI0053D) it
> > > looks like only the poly8/16/64/128_t types are unspecified. It's safe to
> > > drop them as long as they're not used in structs or function parameters,
> > > but I sent a more generic fix [1] that copies the clang defintions. When
> > > building the kernel with clang, the polyX_t types do get typedefs.
> > >
> > > Thanks,
> > > Jean
> > >
> >
> > Hi Jean,
> >
> > Would you be so kind to build some simple C repro code that uses those
> > polyX_t types? Ideally built by both GCC and Clang. And then run
> > `pahole -J` on them to get .BTF into them as well. If you can share
> > those two with me, I'd love to look at how DWARF and BTF look like.
> >
> > I'm, unfortunately, having trouble making something like that to
> > cross-compile on my x86-64 machine, I've spent a bunch of time already
> > on this unsuccessfully and it's really frustrating at this point. If
> > you have an ARM system (or cross-compilation set up properly), it
> > shouldn't take much time for you, hopefully. Just make sure that those
> > polyX_t types do make it into DWARF, so, e.g., use them with static
> > variable or something, e.g.,:
> >
> > int main() {
> >     static poly8_t a = 12;
> >     return a + 10;
> > }
> >
> > Or something along those lines. Thanks!
>
> No problem, I put the source and clang+gcc binaries in a tarball here:
> https://jpbrucker.net/tmp/test-poly-neon.tar.bz2
>
> These contain all the base types defined by arm_neon.h (minus the new
> bfloat16, which I don't think matters at the moment)
>

Thanks a lot! It was very helpful. I wonder why there was never
poly32_t defined?

> Thanks,
> Jean
>
> >
> > > [1] https://lore.kernel.org/bpf/20200810122835.2309026-1-jean-philippe@linaro.org/
> > >
> > > > There is a bunch more in gcc/config/arm/arm-simd-builtin-types.def.
> > > > May be there is a way to detect compiler builtin types by pattern matching
> > > > their dwarf/btf shape and skip them automatically?
> > > > The simplest, of course, is to only add a few that caused this known
> > > > trouble to blocklist.

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Russell King - ARM Linux admin]

On Fri, Aug 07, 2020 at 07:23:53PM +0200, Jean-Philippe Brucker wrote:
> On Fri, Aug 07, 2020 at 04:20:58PM +0200, Jakov Petrina wrote:
> > Hi everyone,
> > 
> > recently we have begun extensive research into eBPF and related
> > technologies. Seeking an easier development process, we have switched over
> > to using the eBPF CO-RE [0] approach internally which has enabled us to
> > simplify most aspects of eBPF development, especially those related to
> > cross-compilation.

For those of us not familiar with what CO-RE is, this doesn't help.
I assume the [0] was a reference to something that explained it,
but that isn't included.

> > However, as part of these efforts we have stumbled upon several problems
> > that we feel would benefit from a community discussion where we may share
> > our solutions and discuss alternatives moving forward.
> > 
> > As a reference point, we have started researching and modifying several eBPF
> > CO-RE samples that have been developed or migrated from existing `bcc`
> > tooling. Most notable examples are those present in `bcc`'s `libbpf-tools`
> > directory [1]. Some of these samples have just recently been converted to
> > respective eBPF CO-RE variants, of which the `tcpconnect` tracing sample has
> > proven to be very interesting.
> > 
> > First showstopper for cross-compiling aforementioned example on the ARM
> > 32-bit platform has been with regards to generation of the required
> > `vmlinux.h` kernel header from the BTF information. More specifically, our
> > initial approach to have e.g. a compilation target dependency which would
> > invoke `bpftool` at configure time was not appropriate due to several
> > issues: a) CO-RE requires host kernel to have been compiled in such a way to
> > expose BTF information which may not available, and b) the generated

What is "BTF information"?  Google suggests it's something to do with
the British Thyroid Foundation.

Please don't use three letter abbreviations unless they are widely
understood, or if you wish to, please ensure that you explain them.
TLAs otherwise are an exclusion mechanism.

> > `vmlinux.h` was actually architecture-specific.

What is this "vmlinux.h" ?  It isn't something that the kernel provides
afaics.  It doesn't seem to be present on my existing x86 Debian system.
I've seen it on Fedora systems in the dim and distant past.

Where do you think it comes from?  Where are you finding it?
-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Jakov Petrina]

Hi,

On 07/08/2020 21:00, Russell King - ARM Linux admin wrote:
> 
> For those of us not familiar with what CO-RE is, this doesn't help.
> I assume the [0] was a reference to something that explained it,
> but that isn't included.
> 

the reference [0] is link to a blog post which explains the eBPF CO-RE 
concept; I have added this link as a reference below.

> 
> What is "BTF information"?  Google suggests it's something to do with
> the British Thyroid Foundation.
> 
> Please don't use three letter abbreviations unless they are widely
> understood, or if you wish to, please ensure that you explain them.
> TLAs otherwise are an exclusion mechanism.
> >
> What is this "vmlinux.h" ?  It isn't something that the kernel provides
> afaics.  It doesn't seem to be present on my existing x86 Debian system.
> I've seen it on Fedora systems in the dim and distant past.
> 
> Where do you think it comes from?  Where are you finding it?
> 

The blog post [0] provides description and context for the references 
and abbreviations used, but in the future I will be sure to avoid using 
abbreviations unless they are commonly understood.

[0] 
https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html

Regards,
-- 
Jakov Petrina

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Russell King - ARM Linux admin]

On Mon, Aug 10, 2020 at 09:52:17AM +0200, Jakov Petrina wrote:
> Hi,
> 
> On 07/08/2020 21:00, Russell King - ARM Linux admin wrote:
> > 
> > For those of us not familiar with what CO-RE is, this doesn't help.
> > I assume the [0] was a reference to something that explained it,
> > but that isn't included.
> > 
> 
> the reference [0] is link to a blog post which explains the eBPF CO-RE
> concept; I have added this link as a reference below.
> 
> > 
> > What is "BTF information"?  Google suggests it's something to do with
> > the British Thyroid Foundation.
> > 
> > Please don't use three letter abbreviations unless they are widely
> > understood, or if you wish to, please ensure that you explain them.
> > TLAs otherwise are an exclusion mechanism.
> > >
> > What is this "vmlinux.h" ?  It isn't something that the kernel provides
> > afaics.  It doesn't seem to be present on my existing x86 Debian system.
> > I've seen it on Fedora systems in the dim and distant past.
> > 
> > Where do you think it comes from?  Where are you finding it?
> > 
> 
> The blog post [0] provides description and context for the references and
> abbreviations used, but in the future I will be sure to avoid using
> abbreviations unless they are commonly understood.
> 
> [0] https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html

Okay, you've addressed one point I raised, but you have not addressed
any of the questions I raised.  I'll take this thread as just noise on
the mailing list since it seems to contain nothing of any relevance to
the Linux kernel, and no one seems willing to explain why they think it
is relevant.

Thanks.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Jakov Petrina]

Hi,

On 07/08/2020 19:23, Jean-Philippe Brucker wrote:
> Hi,
> 
> [Adding the linux-arm-kernel list on Cc]
> 
> 
> It looks like this "__Poly8_t" type is internal to GCC (provided in
> arm_neon.h) and clang has its own internals. I managed to reproduce this
> with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
> the moment. Maybe libbpf should generate defines to translate these
> intrinsics between clang and gcc? Not very elegant. I'll take another
> look next week.
> 

indeed, this has only been present in our arm64 kernel builds but I 
suppose it may surface for different configurations as well. Per 
Andrii's suggestion, I think blacklisting such types during the dump 
would be a safe bet for now.

> 
> I don't know if there is a room for improvement regarding your a) and b)
> points, as I think the added complexity is inherent to cross-building. But
> kernel crashes definitely need to be fixed, as well as the above problem.
> 

If that is the case, I suppose an additional step would be to configure 
the kernel in such a way so that the generated header files result in 
the smallest possible file size, while retaining all structures that we 
require in our eBPF programs.

The kernel crash seems to elude our attempts to reproduce it; it occurs 
only on a target 32-bit platform device, but a reproducible program does 
not crash the kernel in a QEMU ARM environment. When we investigate this 
further we will definitely share our results.

> Thanks,
> Jean
> 

Thanks,
-- 
Jakov Petrina

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Andrii Nakryiko]

On Fri, Aug 7, 2020 at 7:21 AM Jakov Petrina <jakov.petrina@sartura.hr> wrote:
>
> Hi everyone,
>
> recently we have begun extensive research into eBPF and related
> technologies. Seeking an easier development process, we have switched
> over to using the eBPF CO-RE [0] approach internally which has enabled
> us to simplify most aspects of eBPF development, especially those
> related to cross-compilation.

Great!

>
> However, as part of these efforts we have stumbled upon several problems
> that we feel would benefit from a community discussion where we may
> share our solutions and discuss alternatives moving forward.
>
> As a reference point, we have started researching and modifying several
> eBPF CO-RE samples that have been developed or migrated from existing
> `bcc` tooling. Most notable examples are those present in `bcc`'s
> `libbpf-tools` directory [1]. Some of these samples have just recently
> been converted to respective eBPF CO-RE variants, of which the
> `tcpconnect` tracing sample has proven to be very interesting.
>
> First showstopper for cross-compiling aforementioned example on the ARM
> 32-bit platform has been with regards to generation of the required
> `vmlinux.h` kernel header from the BTF information. More specifically,
> our initial approach to have e.g. a compilation target dependency which
> would invoke `bpftool` at configure time was not appropriate due to
> several issues: a) CO-RE requires host kernel to have been compiled in
> such a way to expose BTF information which may not available, and b) the
> generated `vmlinux.h` was actually architecture-specific.

That's not exactly true, about "CO-RE requires host kernel to have
been compiled...". You can pass any kernel image as a parameter to
bpftool as an input to generate vmlinux.h for that target
architecture. The only limitation right now, I think, is that their
endianness have to match. We'll probably get over this limitation some
time by end of this year, though.

So in your case, I'd recommend to generate per-architecture vmlinux.h
and use the appropriate one when you cross-compile. I don't think we
ever intended to support single CO-RE BPF binary across architectures,
given it's not too bad to compile same code one time for each target
architecture. Compiling once for each kernel version/variant was much
bigger problem, which is what we tackled.

>
> The second point proved interesting because `tcpconnect` makes use of
> the `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs
> *ctx` as the first function parameter. The `pt_regs` structure is
> defined by the kernel and is architecture-specific. Since `libbpf` does
> have architecture-specific conditionals, pairing it with an "invalid"
> `vmlinux.h` resulted in cross-compilation failure as `libbpf` provided
> macros that work with ARM `pt_regs`, and `vmlinux.h` had an x86
> `pt_regs` definition. To resolve this issue, we have resorted to
> including pre-generated `<arch>_vmlinux.h` files in our CO-RE build system.

yep, see above, that's what I'd do as well.

>
> However, there are certainly drawbacks to this approach: a) (relatively)
> large file size of the generated headers, b) regular maintenance to
> re-generate the header files for various architectures and kernel
> versions, and c) incompatible definitions being generated, to name a
> few. This last point relates to the the fact that our `aarch64`/`arm64`
> kernel generates the following definition using `bpftool`, which has
> resulted in compilation failure:
>
> ```
> typedef __Poly8_t poly8x16_t[16];
> ```
>
> AFAICT these are ARM NEON intrinsic definitions which are GCC-specific.
> We have opted to comment out this line as there was no additional
> `poly8x16_t` usage in the header file.

Ok, so for a) why the size of vmlinux.h is a big factor? You use it on
host machine during compilation only, after that you don't have to
distribute it anywhere. I just checked the size of vmlinux.h we use to
write BPF programs for production, it's at 2.5MB. Having even few of
those (if you need x86 + ARM32 + ARM64 + s390x + whatever) isn't a big
deal, IMO, you can just check them in into your source control system?
If the size is a concern, I'd be curious to hear why.

b) Hm.. how often do you intend to re-geneate them? Unless you are
using some bleeding-edge and volatile features of kernel and/or
compiled-in drivers, you shouldn't need to re-generate it all that
often. Maybe once every kernel release, maybe even less frequently. We
update those vmlinux.h only when there is some new set of features
(e.g., bpf_iter) added and we need those types, or when we get a new
major kernel version bump. So far so good. But your constraints might
differ, so I'd like to learn more.

c) I addressed in another reply. BTF dumper in libbpf maintains a list
of types that are compiler-provided and avoid generating types for
them, assuming compiler will have them. So far we've handled it simply
for __builtin_va_list, we can probably do something like that here as
well?

>
> Given various issues we have encountered so far (among which is a kernel
> panic/crash on a specific device), additional input and feedback
> regarding cross-compilation of the eBPF utilities would be greatly
> appreciated.
>

Please report the panic with more details separately. If you are
referring to cross-compiling libbpf-tools in BCC repo, we can play
with that, generate a separate vmlinux.<arch>.h. It's a bit hard for
me to test as I don't have easy access to anything beyond x86-64, so
some help from other folks would be very appreciated.

> [0]
> https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html
> [1] https://github.com/iovisor/bcc/tree/master/libbpf-tools
>
> Best regards,
>
> Sartura eBPF Team

Re: eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Jakov Petrina]

Hi,

On 07/08/2020 21:46, Andrii Nakryiko wrote:
>> First showstopper for cross-compiling aforementioned example on the ARM
>> 32-bit platform has been with regards to generation of the required
>> `vmlinux.h` kernel header from the BTF information. More specifically,
>> our initial approach to have e.g. a compilation target dependency which
>> would invoke `bpftool` at configure time was not appropriate due to
>> several issues: a) CO-RE requires host kernel to have been compiled in
>> such a way to expose BTF information which may not available, and b) the
>> generated `vmlinux.h` was actually architecture-specific.
> 
> That's not exactly true, about "CO-RE requires host kernel to have
> been compiled...". You can pass any kernel image as a parameter to
> bpftool as an input to generate vmlinux.h for that target
> architecture. The only limitation right now, I think, is that their
> endianness have to match. We'll probably get over this limitation some
> time by end of this year, though.
> 

Ah, I was not aware this was possible, thanks; it will certainly cut 
down on the time it takes to generate headers for other arches.

> So in your case, I'd recommend to generate per-architecture vmlinux.h
> and use the appropriate one when you cross-compile. I don't think we
> ever intended to support single CO-RE BPF binary across architectures,
> given it's not too bad to compile same code one time for each target
> architecture. Compiling once for each kernel version/variant was much
> bigger problem, which is what we tackled.
> 

Agreed, kernel compatibility is a bit more crucial here; we are 
comfortable with handling cross-compilation for other arches.

>>
>> However, there are certainly drawbacks to this approach: a) (relatively)
>> large file size of the generated headers, b) regular maintenance to
>> re-generate the header files for various architectures and kernel
>> versions, and c) incompatible definitions being generated, to name a
>> few. This last point relates to the the fact that our `aarch64`/`arm64`
>> kernel generates the following definition using `bpftool`, which has
>> resulted in compilation failure:
>>
>> ```
>> typedef __Poly8_t poly8x16_t[16];
>> ```
>>
>> AFAICT these are ARM NEON intrinsic definitions which are GCC-specific.
>> We have opted to comment out this line as there was no additional
>> `poly8x16_t` usage in the header file.
> 
> Ok, so for a) why the size of vmlinux.h is a big factor? You use it on
> host machine during compilation only, after that you don't have to
> distribute it anywhere. I just checked the size of vmlinux.h we use to
> write BPF programs for production, it's at 2.5MB. Having even few of
> those (if you need x86 + ARM32 + ARM64 + s390x + whatever) isn't a big
> deal, IMO, you can just check them in into your source control system?
> If the size is a concern, I'd be curious to hear why.
> 

Yup, we currently have these files included with our source and it 
hasn't been that bad. However, it struck us as a not the most elegant 
solution given the fact that these are large pre-generated files which 
require manual intervention to update.

However, given that a running kernel is not necessary to create these 
files perhaps we might develop internal tooling to make this process as 
easy as possible.

> b) Hm.. how often do you intend to re-geneate them? Unless you are
> using some bleeding-edge and volatile features of kernel and/or
> compiled-in drivers, you shouldn't need to re-generate it all that
> often. Maybe once every kernel release, maybe even less frequently. We
> update those vmlinux.h only when there is some new set of features
> (e.g., bpf_iter) added and we need those types, or when we get a new
> major kernel version bump. So far so good. But your constraints might
> differ, so I'd like to learn more.
> 

We are currently looking into bleeding-edge features of the kernel, but 
they mostly concern eBPF itself; I suppose that for us, updating these 
headers should be done when new features are introduced to the kernel. 
When we identify applications of eBPF we will most likely have more 
constraints to keep track of.

> c) I addressed in another reply. BTF dumper in libbpf maintains a list
> of types that are compiler-provided and avoid generating types for
> them, assuming compiler will have them. So far we've handled it simply
> for __builtin_va_list, we can probably do something like that here as
> well?
> 

Great, I think that is an acceptable solution.

>>
>> Given various issues we have encountered so far (among which is a kernel
>> panic/crash on a specific device), additional input and feedback
>> regarding cross-compilation of the eBPF utilities would be greatly
>> appreciated.
>>
> 
> Please report the panic with more details separately. If you are
> referring to cross-compiling libbpf-tools in BCC repo, we can play
> with that, generate a separate vmlinux.<arch>.h. It's a bit hard for
> me to test as I don't have easy access to anything beyond x86-64, so
> some help from other folks would be very appreciated.
> 

Thanks, as mentioned in another reply we have been attempting to 
reproduce this issue in a QEMU ARM environment but so far we haven't 
been successful. We will most likely move over to debugging it directly 
on our target hardware and report it when we have more information.

Regards,
-- 
Jakov Petrina