* bpf: LLVM BTF inner map struct type def missing
@ 2025-07-22 18:44 Mahe Tardy
2025-07-22 23:02 ` Yonghong Song
0 siblings, 1 reply; 4+ messages in thread
From: Mahe Tardy @ 2025-07-22 18:44 UTC (permalink / raw)
To: netdev; +Cc: Yonghong Song, Alexei Starovoitov, Daniel Borkmann
Hello,
While writing a BPF prog using map of maps I bumped into this compiler
bug that GitHub user thediveo and Isovalent colleague Timo Beckers
already discussed in the cilium/ebpf discussions [^1].
The issue is that a struct only used in a inner map is not included in
the program BTF, so it needs a dummy declaration elsewhere to work.
For example such program:
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
struct missing_type {
uint64_t foo;
};
// struct missing_type bar; // commented on purpose
struct {
__uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
__type(key, uint32_t);
__type(value, uint32_t);
__uint(max_entries, 16);
__array(
values, struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, uint64_t);
__type(value, struct missing_type);
__uint(max_entries, 32);
});
} outer_map SEC(".maps");
Then do:
bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
clang -target bpf -g -O2 -c prog.c -o prog.o
bpftool btf dump file prog.o
Will result in:
[1] PTR '(anon)' type_id=3
[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[3] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=12
[4] INT '__ARRAY_SIZE_TYPE__' size=4 bits_offset=0 nr_bits=32 encoding=(none)
[5] PTR '(anon)' type_id=6
[6] TYPEDEF 'uint32_t' type_id=7
[7] TYPEDEF 'u32' type_id=8
[8] TYPEDEF '__u32' type_id=9
[9] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none)
[10] PTR '(anon)' type_id=11
[11] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=16
[12] PTR '(anon)' type_id=13
[13] STRUCT '(anon)' size=32 vlen=4
'type' type_id=14 bits_offset=0
'key' type_id=16 bits_offset=64
'value' type_id=21 bits_offset=128
'max_entries' type_id=22 bits_offset=192
[14] PTR '(anon)' type_id=15
[15] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=1
[16] PTR '(anon)' type_id=17
[17] TYPEDEF 'uint64_t' type_id=18
[18] TYPEDEF 'u64' type_id=19
[19] TYPEDEF '__u64' type_id=20
[20] INT 'unsigned long long' size=8 bits_offset=0 nr_bits=64 encoding=(none)
[21] PTR '(anon)' type_id=28
[22] PTR '(anon)' type_id=23
[23] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=32
[24] ARRAY '(anon)' type_id=12 index_type_id=4 nr_elems=0
[25] STRUCT '(anon)' size=32 vlen=5
'type' type_id=1 bits_offset=0
'key' type_id=5 bits_offset=64
'value' type_id=5 bits_offset=128
'max_entries' type_id=10 bits_offset=192
'values' type_id=24 bits_offset=256
[26] VAR 'outer_map' type_id=25, linkage=global
[27] DATASEC '.maps' size=0 vlen=1
type_id=26 offset=0 size=32 (VAR 'outer_map')
[28] FWD 'missing_type' fwd_kind=struct
You can see that the outer map is [25], with values [24] with type to
[12] thus [13] and then the value of [13] is [21] which points to type
[28]. And [28] is a forward declaration. Thus if we try to load this
program (there's no program but the libbpf error msg is explicit):
bpftool prog load prog.o /sys/fs/bpf/prog
Output is
libbpf: map 'outer_map.inner': can't determine value size for type [28]: -22.
Now if you uncomment the commented line in the example (or use this type
in a function as suggested by Timo), the BTF looks like this:
[1] PTR '(anon)' type_id=3
[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[3] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=12
[4] INT '__ARRAY_SIZE_TYPE__' size=4 bits_offset=0 nr_bits=32 encoding=(none)
[5] PTR '(anon)' type_id=6
[6] TYPEDEF 'uint32_t' type_id=7
[7] TYPEDEF 'u32' type_id=8
[8] TYPEDEF '__u32' type_id=9
[9] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none)
[10] PTR '(anon)' type_id=11
[11] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=16
[12] PTR '(anon)' type_id=13
[13] STRUCT '(anon)' size=32 vlen=4
'type' type_id=14 bits_offset=0
'key' type_id=16 bits_offset=64
'value' type_id=21 bits_offset=128
'max_entries' type_id=22 bits_offset=192
[14] PTR '(anon)' type_id=15
[15] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=1
[16] PTR '(anon)' type_id=17
[17] TYPEDEF 'uint64_t' type_id=18
[18] TYPEDEF 'u64' type_id=19
[19] TYPEDEF '__u64' type_id=20
[20] INT 'unsigned long long' size=8 bits_offset=0 nr_bits=64 encoding=(none)
[21] PTR '(anon)' type_id=27
[22] PTR '(anon)' type_id=23
[23] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=32
[24] ARRAY '(anon)' type_id=12 index_type_id=4 nr_elems=0
[25] STRUCT '(anon)' size=32 vlen=5
'type' type_id=1 bits_offset=0
'key' type_id=5 bits_offset=64
'value' type_id=5 bits_offset=128
'max_entries' type_id=10 bits_offset=192
'values' type_id=24 bits_offset=256
[26] VAR 'outer_map' type_id=25, linkage=global
[27] STRUCT 'missing_type' size=8 vlen=1
'foo' type_id=17 bits_offset=0
[28] VAR 'bar' type_id=27, linkage=global
[29] DATASEC '.bss' size=0 vlen=1
type_id=28 offset=0 size=8 (VAR 'bar')
[30] DATASEC '.maps' size=0 vlen=1
type_id=26 offset=0 size=32 (VAR 'outer_map')
And then the type [27] exists, loading can now proceed.
I tested it with latest LLVM-project head when writing this e789f8bdf369
("[libc][math] Add Generic Comparison Operations for floating point
types (#144983)").
If you think it's reasonable to fix, I would be interested looking into
this.
[^1]: https://github.com/cilium/ebpf/discussions/1658#discussioncomment-12491339
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: bpf: LLVM BTF inner map struct type def missing
2025-07-22 18:44 bpf: LLVM BTF inner map struct type def missing Mahe Tardy
@ 2025-07-22 23:02 ` Yonghong Song
2025-07-25 11:51 ` Mahe Tardy
0 siblings, 1 reply; 4+ messages in thread
From: Yonghong Song @ 2025-07-22 23:02 UTC (permalink / raw)
To: Mahe Tardy, netdev@vger.kernel.org; +Cc: Alexei Starovoitov, Daniel Borkmann
> Hello,
>
> While writing a BPF prog using map of maps I bumped into this compiler
> bug that GitHub user thediveo and Isovalent colleague Timo Beckers
> already discussed in the cilium/ebpf discussions [^1].
>
> The issue is that a struct only used in a inner map is not included in
> the program BTF, so it needs a dummy declaration elsewhere to work.
>
> For example such program:
>
> #include "vmlinux.h"
> #include <bpf/bpf_helpers.h>
>
> struct missing_type {
> uint64_t foo;
> };
>
> // struct missing_type bar; // commented on purpose
>
> struct {
> __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
> __type(key, uint32_t);
> __type(value, uint32_t);
> __uint(max_entries, 16);
> __array(
> values, struct {
> __uint(type, BPF_MAP_TYPE_HASH);
> __type(key, uint64_t);
> __type(value, struct missing_type);
> __uint(max_entries, 32);
> });
> } outer_map SEC(".maps");
>
> Then do:
>
> bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
> clang -target bpf -g -O2 -c prog.c -o prog.o
> bpftool btf dump file prog.o
>
> Will result in:
>
> [1] PTR '(anon)' type_id=3
> [2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
> [3] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=12
> [4] INT '__ARRAY_SIZE_TYPE__' size=4 bits_offset=0 nr_bits=32 encoding=(none)
> [5] PTR '(anon)' type_id=6
> [6] TYPEDEF 'uint32_t' type_id=7
> [7] TYPEDEF 'u32' type_id=8
> [8] TYPEDEF '__u32' type_id=9
> [9] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none)
> [10] PTR '(anon)' type_id=11
> [11] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=16
> [12] PTR '(anon)' type_id=13
> [13] STRUCT '(anon)' size=32 vlen=4
> 'type' type_id=14 bits_offset=0
> 'key' type_id=16 bits_offset=64
> 'value' type_id=21 bits_offset=128
> 'max_entries' type_id=22 bits_offset=192
> [14] PTR '(anon)' type_id=15
> [15] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=1
> [16] PTR '(anon)' type_id=17
> [17] TYPEDEF 'uint64_t' type_id=18
> [18] TYPEDEF 'u64' type_id=19
> [19] TYPEDEF '__u64' type_id=20
> [20] INT 'unsigned long long' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> [21] PTR '(anon)' type_id=28
> [22] PTR '(anon)' type_id=23
> [23] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=32
> [24] ARRAY '(anon)' type_id=12 index_type_id=4 nr_elems=0
> [25] STRUCT '(anon)' size=32 vlen=5
> 'type' type_id=1 bits_offset=0
> 'key' type_id=5 bits_offset=64
> 'value' type_id=5 bits_offset=128
> 'max_entries' type_id=10 bits_offset=192
> 'values' type_id=24 bits_offset=256
> [26] VAR 'outer_map' type_id=25, linkage=global
> [27] DATASEC '.maps' size=0 vlen=1
> type_id=26 offset=0 size=32 (VAR 'outer_map')
> [28] FWD 'missing_type' fwd_kind=struct
>
> You can see that the outer map is [25], with values [24] with type to
> [12] thus [13] and then the value of [13] is [21] which points to type
> [28]. And [28] is a forward declaration. Thus if we try to load this
> program (there's no program but the libbpf error msg is explicit):
>
> bpftool prog load prog.o /sys/fs/bpf/prog
>
> Output is
>
> libbpf: map 'outer_map.inner': can't determine value size for type [28]: -22.
>
> Now if you uncomment the commented line in the example (or use this type
> in a function as suggested by Timo), the BTF looks like this:
>
> [1] PTR '(anon)' type_id=3
> [2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
> [3] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=12
> [4] INT '__ARRAY_SIZE_TYPE__' size=4 bits_offset=0 nr_bits=32 encoding=(none)
> [5] PTR '(anon)' type_id=6
> [6] TYPEDEF 'uint32_t' type_id=7
> [7] TYPEDEF 'u32' type_id=8
> [8] TYPEDEF '__u32' type_id=9
> [9] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none)
> [10] PTR '(anon)' type_id=11
> [11] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=16
> [12] PTR '(anon)' type_id=13
> [13] STRUCT '(anon)' size=32 vlen=4
> 'type' type_id=14 bits_offset=0
> 'key' type_id=16 bits_offset=64
> 'value' type_id=21 bits_offset=128
> 'max_entries' type_id=22 bits_offset=192
> [14] PTR '(anon)' type_id=15
> [15] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=1
> [16] PTR '(anon)' type_id=17
> [17] TYPEDEF 'uint64_t' type_id=18
> [18] TYPEDEF 'u64' type_id=19
> [19] TYPEDEF '__u64' type_id=20
> [20] INT 'unsigned long long' size=8 bits_offset=0 nr_bits=64 encoding=(none)
> [21] PTR '(anon)' type_id=27
> [22] PTR '(anon)' type_id=23
> [23] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=32
> [24] ARRAY '(anon)' type_id=12 index_type_id=4 nr_elems=0
> [25] STRUCT '(anon)' size=32 vlen=5
> 'type' type_id=1 bits_offset=0
> 'key' type_id=5 bits_offset=64
> 'value' type_id=5 bits_offset=128
> 'max_entries' type_id=10 bits_offset=192
> 'values' type_id=24 bits_offset=256
> [26] VAR 'outer_map' type_id=25, linkage=global
> [27] STRUCT 'missing_type' size=8 vlen=1
> 'foo' type_id=17 bits_offset=0
> [28] VAR 'bar' type_id=27, linkage=global
> [29] DATASEC '.bss' size=0 vlen=1
> type_id=28 offset=0 size=8 (VAR 'bar')
> [30] DATASEC '.maps' size=0 vlen=1
> type_id=26 offset=0 size=32 (VAR 'outer_map')
>
> And then the type [27] exists, loading can now proceed.
>
> I tested it with latest LLVM-project head when writing this e789f8bdf369
> ("[libc][math] Add Generic Comparison Operations for floating point
> types (#144983)").
>
> If you think it's reasonable to fix, I would be interested looking into
> this.
Looks like this may be something we want to fix as people starts or already uses
nested maps. So yes, please go ahead to look into this.
FYI, the llvm patch https://github.com/llvm/llvm-project/pull/141719/ tried to solve
nested struct definition issue (similar to here). In that case, a type_tag is the
indicator which allows further type generation. The patch listed llvm source
locations you likely need to touch.
>
> [^1]: https://github.com/cilium/ebpf/discussions/1658#discussioncomment-12491339
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: bpf: LLVM BTF inner map struct type def missing
2025-07-22 23:02 ` Yonghong Song
@ 2025-07-25 11:51 ` Mahe Tardy
2025-07-25 17:40 ` Yonghong Song
0 siblings, 1 reply; 4+ messages in thread
From: Mahe Tardy @ 2025-07-25 11:51 UTC (permalink / raw)
To: Yonghong Song; +Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann
On Tue, Jul 22, 2025 at 11:02:58PM +0000, Yonghong Song wrote:
> > If you think it's reasonable to fix, I would be interested looking into
> > this.
>
> Looks like this may be something we want to fix as people starts or already uses
> nested maps. So yes, please go ahead to look into this.
Here is my proposal patch https://github.com/llvm/llvm-project/pull/150608.
The idea is to visit inner maps as we visit the global maps during BTF
generation. Unfortunately, for that, I had to add a boolean arg to some
functions, I hope it's okay.
>
> FYI, the llvm patch https://github.com/llvm/llvm-project/pull/141719/ tried to solve
> nested struct definition issue (similar to here). In that case, a type_tag is the
> indicator which allows further type generation. The patch listed llvm source
> locations you likely need to touch.
Thank you, it was very useful to have this pointer!
>
> >
> > [^1]: https://github.com/cilium/ebpf/discussions/1658#discussioncomment-12491339
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: bpf: LLVM BTF inner map struct type def missing
2025-07-25 11:51 ` Mahe Tardy
@ 2025-07-25 17:40 ` Yonghong Song
0 siblings, 0 replies; 4+ messages in thread
From: Yonghong Song @ 2025-07-25 17:40 UTC (permalink / raw)
To: Mahe Tardy; +Cc: netdev@vger.kernel.org, Alexei Starovoitov, Daniel Borkmann
>> ________________________________________
>> From: Mahe Tardy <mahe.tardy@gmail.com>
>> Sent: Friday, July 25, 2025 4:51 AM
>> To: Yonghong Song
>> Cc: netdev@vger.kernel.org; Alexei Starovoitov; Daniel Borkmann
>> Subject: Re: bpf: LLVM BTF inner map struct type def missing
>>
>> On Tue, Jul 22, 2025 at 11:02:58PM +0000, Yonghong Song wrote:
>> > > If you think it's reasonable to fix, I would be interested looking into
>> > > this.
>> >
>> > Looks like this may be something we want to fix as people starts or already uses
>> > nested maps. So yes, please go ahead to look into this.
>>
>> Here is my proposal patch https://github.com/llvm/llvm-project/pull/150608 .
>> The idea is to visit inner maps as we visit the global maps during BTF
>> generation. Unfortunately, for that, I had to add a boolean arg to some
>> functions, I hope it's okay.
Sure. Will take a look. Thanks!
>>
>> >
>> > FYI, the llvm patch https://github.com/llvm/llvm-project/pull/141719/ tried to solve
>> > nested struct definition issue (similar to here). In that case, a type_tag is the
>> > indicator which allows further type generation. The patch listed llvm source
>> > locations you likely need to touch.
>>
>> Thank you, it was very useful to have this pointer!
>>
>> >
>> > >
>> > > [^1]: https://github.com/cilium/ebpf/discussions/1658#discussioncomment-12491339
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-07-25 17:40 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-07-22 18:44 bpf: LLVM BTF inner map struct type def missing Mahe Tardy
2025-07-22 23:02 ` Yonghong Song
2025-07-25 11:51 ` Mahe Tardy
2025-07-25 17:40 ` Yonghong Song
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox