* objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR:
@ 2022-10-04 2:05 Jakub Kicinski
2022-10-04 4:49 ` WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 Jakub Kicinski
0 siblings, 1 reply; 22+ messages in thread
From: Jakub Kicinski @ 2022-10-04 2:05 UTC (permalink / raw)
To: Jiri Olsa; +Cc: bpf, netdev@vger.kernel.org
Hi Jiri,
I get the following warning after merging up all the trees:
vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
$ gcc --version
gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15)
Is this known?
^ permalink raw reply [flat|nested] 22+ messages in thread
* WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-04 2:05 objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: Jakub Kicinski
@ 2022-10-04 4:49 ` Jakub Kicinski
2022-10-04 6:42 ` Jiri Olsa
2022-10-04 14:46 ` Daniel Xu
0 siblings, 2 replies; 22+ messages in thread
From: Jakub Kicinski @ 2022-10-04 4:49 UTC (permalink / raw)
To: Daniel Xu, Martin KaFai Lau; +Cc: Jiri Olsa, bpf, netdev@vger.kernel.org
On Mon, 3 Oct 2022 19:05:45 -0700 Jakub Kicinski wrote:
> Hi Jiri,
>
> I get the following warning after merging up all the trees:
>
> vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
> vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
>
> $ gcc --version
> gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15)
>
>
> Is this known?
Also hit this:
WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
WARN: multiple IDs found for 'nf_conn': 92168, 121226 - using 92168
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-04 4:49 ` WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 Jakub Kicinski
@ 2022-10-04 6:42 ` Jiri Olsa
2022-10-04 14:25 ` Jakub Kicinski
2022-10-04 14:46 ` Daniel Xu
1 sibling, 1 reply; 22+ messages in thread
From: Jiri Olsa @ 2022-10-04 6:42 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: Daniel Xu, Martin KaFai Lau, bpf, netdev@vger.kernel.org
On Mon, Oct 03, 2022 at 09:49:41PM -0700, Jakub Kicinski wrote:
> On Mon, 3 Oct 2022 19:05:45 -0700 Jakub Kicinski wrote:
> > Hi Jiri,
> >
> > I get the following warning after merging up all the trees:
> >
> > vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
> > vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
> >
> > $ gcc --version
> > gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15)
> >
> >
> > Is this known?
>
> Also hit this:
>
> WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
> WARN: multiple IDs found for 'nf_conn': 92168, 121226 - using 92168
hi,
I did not see that before, could you please share your config?
thanks,
jirka
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-04 6:42 ` Jiri Olsa
@ 2022-10-04 14:25 ` Jakub Kicinski
2022-10-05 9:45 ` Jiri Olsa
0 siblings, 1 reply; 22+ messages in thread
From: Jakub Kicinski @ 2022-10-04 14:25 UTC (permalink / raw)
To: Jiri Olsa; +Cc: Daniel Xu, Martin KaFai Lau, bpf, netdev@vger.kernel.org
On Tue, 4 Oct 2022 08:42:24 +0200 Jiri Olsa wrote:
> I did not see that before, could you please share your config?
allmodconfig
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-04 4:49 ` WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 Jakub Kicinski
2022-10-04 6:42 ` Jiri Olsa
@ 2022-10-04 14:46 ` Daniel Xu
2022-10-04 15:34 ` Jakub Kicinski
2022-10-18 15:11 ` Christian Borntraeger
1 sibling, 2 replies; 22+ messages in thread
From: Daniel Xu @ 2022-10-04 14:46 UTC (permalink / raw)
To: Jakub Kicinski; +Cc: Martin KaFai Lau, Jiri Olsa, bpf, netdev@vger.kernel.org
Hi Jakub,
On Mon, Oct 03, 2022 at 09:49:41PM -0700, Jakub Kicinski wrote:
> On Mon, 3 Oct 2022 19:05:45 -0700 Jakub Kicinski wrote:
> > Hi Jiri,
> >
> > I get the following warning after merging up all the trees:
> >
> > vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
> > vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
> >
> > $ gcc --version
> > gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15)
> >
> >
> > Is this known?
>
> Also hit this:
>
> WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
> WARN: multiple IDs found for 'nf_conn': 92168, 121226 - using 92168
I believe this is now-fixed pahole bug. See:
https://lore.kernel.org/bpf/20220907023559.22juhtyl3qh2gsym@kashmir.localdomain/
That being said, I didn't manage to find a pahole commit that directly
addresses the issue, so maybe upgrading pahole perturbed enough things
for this warning to go away?
If the warning is back on pahole master I can take another look.
Thanks,
Daniel
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-04 14:46 ` Daniel Xu
@ 2022-10-04 15:34 ` Jakub Kicinski
2022-10-18 15:11 ` Christian Borntraeger
1 sibling, 0 replies; 22+ messages in thread
From: Jakub Kicinski @ 2022-10-04 15:34 UTC (permalink / raw)
To: Daniel Xu; +Cc: Martin KaFai Lau, Jiri Olsa, bpf, netdev@vger.kernel.org
On Tue, 4 Oct 2022 08:46:03 -0600 Daniel Xu wrote:
> > WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
> > WARN: multiple IDs found for 'nf_conn': 92168, 121226 - using 92168
>
> I believe this is now-fixed pahole bug. See:
> https://lore.kernel.org/bpf/20220907023559.22juhtyl3qh2gsym@kashmir.localdomain/
>
> That being said, I didn't manage to find a pahole commit that directly
> addresses the issue, so maybe upgrading pahole perturbed enough things
> for this warning to go away?
>
> If the warning is back on pahole master I can take another look.
From-the-source pahole fixes it, thanks!
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-04 14:25 ` Jakub Kicinski
@ 2022-10-05 9:45 ` Jiri Olsa
2022-10-05 15:44 ` Jakub Kicinski
0 siblings, 1 reply; 22+ messages in thread
From: Jiri Olsa @ 2022-10-05 9:45 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Jiri Olsa, Daniel Xu, Martin KaFai Lau, bpf,
netdev@vger.kernel.org
On Tue, Oct 04, 2022 at 07:25:22AM -0700, Jakub Kicinski wrote:
> On Tue, 4 Oct 2022 08:42:24 +0200 Jiri Olsa wrote:
> > I did not see that before, could you please share your config?
>
> allmodconfig
I compiled linux-next with that and can't see that,
any other hint (apart the old gcc) about your setup?
jirka
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-05 9:45 ` Jiri Olsa
@ 2022-10-05 15:44 ` Jakub Kicinski
2022-10-05 16:18 ` Jakub Kicinski
0 siblings, 1 reply; 22+ messages in thread
From: Jakub Kicinski @ 2022-10-05 15:44 UTC (permalink / raw)
To: Jiri Olsa; +Cc: Daniel Xu, Martin KaFai Lau, bpf, netdev@vger.kernel.org
On Wed, 5 Oct 2022 11:45:46 +0200 Jiri Olsa wrote:
> On Tue, Oct 04, 2022 at 07:25:22AM -0700, Jakub Kicinski wrote:
> > On Tue, 4 Oct 2022 08:42:24 +0200 Jiri Olsa wrote:
> > > I did not see that before, could you please share your config?
> >
> > allmodconfig
>
> I compiled linux-next with that and can't see that,
> any other hint (apart the old gcc) about your setup?
Hm, I was compiling Linus's tree not linux-next.
Let me try linux-next right now.
Did you use the 8.5 gcc (which I believe is what comes with
CentOS Stream)? I only see it there.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-05 15:44 ` Jakub Kicinski
@ 2022-10-05 16:18 ` Jakub Kicinski
2022-10-05 20:07 ` Jiri Olsa
0 siblings, 1 reply; 22+ messages in thread
From: Jakub Kicinski @ 2022-10-05 16:18 UTC (permalink / raw)
To: Jiri Olsa; +Cc: Daniel Xu, Martin KaFai Lau, bpf, netdev@vger.kernel.org
On Wed, 5 Oct 2022 08:44:42 -0700 Jakub Kicinski wrote:
> Hm, I was compiling Linus's tree not linux-next.
> Let me try linux-next right now.
>
> Did you use the 8.5 gcc (which I believe is what comes with
> CentOS Stream)? I only see it there.
Yeah, it's there on linux-next, too.
Let me grab a fresh VM and try there. Maybe it's my system. Somehow.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-05 16:18 ` Jakub Kicinski
@ 2022-10-05 20:07 ` Jiri Olsa
2022-10-13 15:05 ` Jakub Kicinski
0 siblings, 1 reply; 22+ messages in thread
From: Jiri Olsa @ 2022-10-05 20:07 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Jiri Olsa, Daniel Xu, Martin KaFai Lau, bpf,
netdev@vger.kernel.org
On Wed, Oct 05, 2022 at 09:18:01AM -0700, Jakub Kicinski wrote:
> On Wed, 5 Oct 2022 08:44:42 -0700 Jakub Kicinski wrote:
> > Hm, I was compiling Linus's tree not linux-next.
> > Let me try linux-next right now.
> >
> > Did you use the 8.5 gcc (which I believe is what comes with
> > CentOS Stream)? I only see it there.
nope.. latest fedora 12
>
> Yeah, it's there on linux-next, too.
>
> Let me grab a fresh VM and try there. Maybe it's my system. Somehow.
ok, I will look around what's the way to install that centos 8 thing
jirka
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-05 20:07 ` Jiri Olsa
@ 2022-10-13 15:05 ` Jakub Kicinski
2022-10-13 21:55 ` Jiri Olsa
2022-10-13 22:12 ` Jiri Olsa
0 siblings, 2 replies; 22+ messages in thread
From: Jakub Kicinski @ 2022-10-13 15:05 UTC (permalink / raw)
To: Jiri Olsa; +Cc: Martin KaFai Lau, bpf, netdev@vger.kernel.org
On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote:
> > Yeah, it's there on linux-next, too.
> >
> > Let me grab a fresh VM and try there. Maybe it's my system. Somehow.
>
> ok, I will look around what's the way to install that centos 8 thing
Any luck?
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-13 15:05 ` Jakub Kicinski
@ 2022-10-13 21:55 ` Jiri Olsa
2022-10-13 22:12 ` Jiri Olsa
1 sibling, 0 replies; 22+ messages in thread
From: Jiri Olsa @ 2022-10-13 21:55 UTC (permalink / raw)
To: Jakub Kicinski, Peter Zijlstra
Cc: Jiri Olsa, Martin KaFai Lau, bpf, netdev@vger.kernel.org
On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote:
> On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote:
> > > Yeah, it's there on linux-next, too.
> > >
> > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow.
> >
> > ok, I will look around what's the way to install that centos 8 thing
>
> Any luck?
yea, sorry for delay, I reproduced that.. first objtool warnings ;-)
[jolsa@centos8 linux-next]$ make
UPD include/generated/compile.h
CALL scripts/checksyscalls.sh
DESCEND objtool
DESCEND bpf/resolve_btfids
CC init/version.o
AR init/built-in.a
AR built-in.a
AR vmlinux.a
LD vmlinux.o
vmlinux.o: warning: objtool: relocate_restore_code+0x3c: relocation to !ENDBR: next_arg+0x8
vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0x20: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
...
Peter,
as for objtool warnings, looks like with gcc we'll get
endbr64 instruction generated after nops
with centos gcc 8.5:
ffffffff818d2e20 <bpf_dispatcher_xdp_func>:
ffffffff818d2e20: 90 nop
ffffffff818d2e21: 90 nop
ffffffff818d2e22: 90 nop
ffffffff818d2e23: 90 nop
ffffffff818d2e24: 90 nop
ffffffff818d2e25: f3 0f 1e fa endbr64
ffffffff818d2e29: e9 92 eb 52 00 jmpq ffffffff81e019c0 <__x86_indirect_thunk_rdx>
ffffffff818d2e2e: 66 90 xchg %ax,%ax
while latest gcc 12 will put it after:
ffffffff82736900 <bpf_dispatcher_xdp_func>:
ffffffff82736900: f3 0f 1e fa endbr64
ffffffff82736904: 90 nop
ffffffff82736905: 90 nop
ffffffff82736906: 90 nop
ffffffff82736907: 90 nop
ffffffff82736908: 90 nop
ffffffff82736909: 41 54 push %r12
ffffffff8273690b: 49 89 f4 mov %rsi,%r12
ffffffff8273690e: 55 push %rbp
ffffffff8273690f: 48 89 fd mov %rdi,%rbp
ffffffff82736912: 53 push %rbx
ffffffff82736913: 48 89 d3 mov %rdx,%rbx
ffffffff82736916: e8 65 f6 cf fe call ffffffff81435f80 <__sanitizer_cov_trace_pc>
ffffffff8273691b: 4c 89 e6 mov %r12,%rsi
ffffffff8273691e: 48 89 ef mov %rbp,%rdi
ffffffff82736921: 48 89 d8 mov %rbx,%rax
ffffffff82736924: 5b pop %rbx
ffffffff82736925: 5d pop %rbp
ffffffff82736926: 41 5c pop %r12
ffffffff82736928: e9 b3 b6 8c 00 jmp ffffffff83001fe0 <__x86_indirect_thunk_array>
any idea where's the problem? bad backport? ;-)
in any case (unrelated), I'll check the bpf dispatcher code,
I'm not sure the nop update code is aware of the endbr64 instruction
jirka
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-13 15:05 ` Jakub Kicinski
2022-10-13 21:55 ` Jiri Olsa
@ 2022-10-13 22:12 ` Jiri Olsa
2022-10-13 22:24 ` Andrii Nakryiko
1 sibling, 1 reply; 22+ messages in thread
From: Jiri Olsa @ 2022-10-13 22:12 UTC (permalink / raw)
To: Jakub Kicinski, Andrii Nakryiko
Cc: Jiri Olsa, Martin KaFai Lau, bpf, netdev@vger.kernel.org
On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote:
> On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote:
> > > Yeah, it's there on linux-next, too.
> > >
> > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow.
> >
> > ok, I will look around what's the way to install that centos 8 thing
>
> Any luck?
now BTFIDS warnings..
I can see following on centos8 with gcc 8.5:
BTFIDS vmlinux
WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300
WARN: multiple IDs found for 'file': 540, 56649 - using 540
WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549
WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953
WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132
WARN: multiple IDs found for 'path': 1164, 56995 - using 1164
WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300
WARN: multiple IDs found for 'file': 540, 61943 - using 540
WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549
WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132
WARN: multiple IDs found for 'path': 1164, 62058 - using 1164
WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190
WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953
WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960
WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876
WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094
WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993
WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191
WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164
WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296
WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292
WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700
WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944
WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299
WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969
WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623
I'll need to check on that..
and I just actually saw the 'nf_conn' warning on linux-next/master with
latest fedora/gcc-12:
BTF [M] net/netfilter/nf_nat.ko
WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518
WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518
WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518
WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518
WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518
WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518
LD [M] net/netfilter/nf_nat_tftp.ko
looks like maybe dedup missed this struct for some reason
nf_conn dump from module:
[120155] PTR '(anon)' type_id=120156
[120156] STRUCT 'nf_conn' size=320 vlen=14
'ct_general' type_id=105882 bits_offset=0
'lock' type_id=180 bits_offset=64
'timeout' type_id=113 bits_offset=640
'zone' type_id=106520 bits_offset=672
'tuplehash' type_id=106533 bits_offset=704
'status' type_id=1 bits_offset=1600
'ct_net' type_id=3215 bits_offset=1664
'nat_bysource' type_id=139 bits_offset=1728
'__nfct_init_offset' type_id=949 bits_offset=1856
'master' type_id=120155 bits_offset=1856
'mark' type_id=106351 bits_offset=1920
'secmark' type_id=106351 bits_offset=1952
'ext' type_id=106536 bits_offset=1984
'proto' type_id=106532 bits_offset=2048
nf_conn dump from vmlinux:
[106517] PTR '(anon)' type_id=106518
[106518] STRUCT 'nf_conn' size=320 vlen=14
'ct_general' type_id=105882 bits_offset=0
'lock' type_id=180 bits_offset=64
'timeout' type_id=113 bits_offset=640
'zone' type_id=106520 bits_offset=672
'tuplehash' type_id=106533 bits_offset=704
'status' type_id=1 bits_offset=1600
'ct_net' type_id=3215 bits_offset=1664
'nat_bysource' type_id=139 bits_offset=1728
'__nfct_init_offset' type_id=949 bits_offset=1856
'master' type_id=106517 bits_offset=1856
'mark' type_id=106351 bits_offset=1920
'secmark' type_id=106351 bits_offset=1952
'ext' type_id=106536 bits_offset=1984
'proto' type_id=106532 bits_offset=2048
look identical.. Andrii, any idea?
thanks,
jirka
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-13 22:12 ` Jiri Olsa
@ 2022-10-13 22:24 ` Andrii Nakryiko
2022-10-14 6:47 ` Jiri Olsa
0 siblings, 1 reply; 22+ messages in thread
From: Andrii Nakryiko @ 2022-10-13 22:24 UTC (permalink / raw)
To: Jiri Olsa; +Cc: Jakub Kicinski, Martin KaFai Lau, bpf, netdev@vger.kernel.org
On Thu, Oct 13, 2022 at 3:12 PM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote:
> > On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote:
> > > > Yeah, it's there on linux-next, too.
> > > >
> > > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow.
> > >
> > > ok, I will look around what's the way to install that centos 8 thing
> >
> > Any luck?
>
> now BTFIDS warnings..
>
> I can see following on centos8 with gcc 8.5:
>
> BTFIDS vmlinux
> WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300
> WARN: multiple IDs found for 'file': 540, 56649 - using 540
> WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549
> WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953
> WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132
> WARN: multiple IDs found for 'path': 1164, 56995 - using 1164
> WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300
> WARN: multiple IDs found for 'file': 540, 61943 - using 540
> WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549
> WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132
> WARN: multiple IDs found for 'path': 1164, 62058 - using 1164
> WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190
> WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953
> WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960
> WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876
> WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094
> WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993
> WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191
> WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164
> WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296
> WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292
> WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700
> WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944
> WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299
> WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969
> WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623
>
> I'll need to check on that..
>
> and I just actually saw the 'nf_conn' warning on linux-next/master with
> latest fedora/gcc-12:
>
> BTF [M] net/netfilter/nf_nat.ko
> WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518
> WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518
> WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518
> WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518
> WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518
> WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518
> LD [M] net/netfilter/nf_nat_tftp.ko
>
> looks like maybe dedup missed this struct for some reason
>
> nf_conn dump from module:
>
> [120155] PTR '(anon)' type_id=120156
> [120156] STRUCT 'nf_conn' size=320 vlen=14
> 'ct_general' type_id=105882 bits_offset=0
> 'lock' type_id=180 bits_offset=64
> 'timeout' type_id=113 bits_offset=640
> 'zone' type_id=106520 bits_offset=672
> 'tuplehash' type_id=106533 bits_offset=704
> 'status' type_id=1 bits_offset=1600
> 'ct_net' type_id=3215 bits_offset=1664
> 'nat_bysource' type_id=139 bits_offset=1728
> '__nfct_init_offset' type_id=949 bits_offset=1856
> 'master' type_id=120155 bits_offset=1856
> 'mark' type_id=106351 bits_offset=1920
> 'secmark' type_id=106351 bits_offset=1952
> 'ext' type_id=106536 bits_offset=1984
> 'proto' type_id=106532 bits_offset=2048
>
> nf_conn dump from vmlinux:
>
> [106517] PTR '(anon)' type_id=106518
> [106518] STRUCT 'nf_conn' size=320 vlen=14
> 'ct_general' type_id=105882 bits_offset=0
> 'lock' type_id=180 bits_offset=64
> 'timeout' type_id=113 bits_offset=640
> 'zone' type_id=106520 bits_offset=672
> 'tuplehash' type_id=106533 bits_offset=704
> 'status' type_id=1 bits_offset=1600
> 'ct_net' type_id=3215 bits_offset=1664
> 'nat_bysource' type_id=139 bits_offset=1728
> '__nfct_init_offset' type_id=949 bits_offset=1856
> 'master' type_id=106517 bits_offset=1856
> 'mark' type_id=106351 bits_offset=1920
> 'secmark' type_id=106351 bits_offset=1952
> 'ext' type_id=106536 bits_offset=1984
> 'proto' type_id=106532 bits_offset=2048
>
> look identical.. Andrii, any idea?
I'm pretty sure they are not identical. There is somewhere a STRUCT vs
FWD difference. We had a similar discussion recently with Alan
Maguire.
> 'master' type_id=120155 bits_offset=1856
vs
> 'master' type_id=106517 bits_offset=1856
we'd need to unwind all these references to find where they start to differ
>
> thanks,
> jirka
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-13 22:24 ` Andrii Nakryiko
@ 2022-10-14 6:47 ` Jiri Olsa
2022-10-14 17:17 ` Alan Maguire
2022-10-21 18:18 ` Andrii Nakryiko
0 siblings, 2 replies; 22+ messages in thread
From: Jiri Olsa @ 2022-10-14 6:47 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Jiri Olsa, Jakub Kicinski, Martin KaFai Lau, bpf,
netdev@vger.kernel.org
On Thu, Oct 13, 2022 at 03:24:59PM -0700, Andrii Nakryiko wrote:
> On Thu, Oct 13, 2022 at 3:12 PM Jiri Olsa <olsajiri@gmail.com> wrote:
> >
> > On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote:
> > > On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote:
> > > > > Yeah, it's there on linux-next, too.
> > > > >
> > > > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow.
> > > >
> > > > ok, I will look around what's the way to install that centos 8 thing
> > >
> > > Any luck?
> >
> > now BTFIDS warnings..
> >
> > I can see following on centos8 with gcc 8.5:
> >
> > BTFIDS vmlinux
> > WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300
> > WARN: multiple IDs found for 'file': 540, 56649 - using 540
> > WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549
> > WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953
> > WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132
> > WARN: multiple IDs found for 'path': 1164, 56995 - using 1164
> > WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300
> > WARN: multiple IDs found for 'file': 540, 61943 - using 540
> > WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549
> > WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132
> > WARN: multiple IDs found for 'path': 1164, 62058 - using 1164
> > WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190
> > WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953
> > WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960
> > WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876
> > WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094
> > WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993
> > WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191
> > WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164
> > WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296
> > WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292
> > WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700
> > WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944
> > WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299
> > WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969
> > WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623
> >
> > I'll need to check on that..
> >
> > and I just actually saw the 'nf_conn' warning on linux-next/master with
> > latest fedora/gcc-12:
> >
> > BTF [M] net/netfilter/nf_nat.ko
> > WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518
> > WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518
> > WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518
> > WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518
> > WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518
> > WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518
> > LD [M] net/netfilter/nf_nat_tftp.ko
> >
> > looks like maybe dedup missed this struct for some reason
> >
> > nf_conn dump from module:
> >
> > [120155] PTR '(anon)' type_id=120156
> > [120156] STRUCT 'nf_conn' size=320 vlen=14
> > 'ct_general' type_id=105882 bits_offset=0
> > 'lock' type_id=180 bits_offset=64
> > 'timeout' type_id=113 bits_offset=640
> > 'zone' type_id=106520 bits_offset=672
> > 'tuplehash' type_id=106533 bits_offset=704
> > 'status' type_id=1 bits_offset=1600
> > 'ct_net' type_id=3215 bits_offset=1664
> > 'nat_bysource' type_id=139 bits_offset=1728
> > '__nfct_init_offset' type_id=949 bits_offset=1856
> > 'master' type_id=120155 bits_offset=1856
> > 'mark' type_id=106351 bits_offset=1920
> > 'secmark' type_id=106351 bits_offset=1952
> > 'ext' type_id=106536 bits_offset=1984
> > 'proto' type_id=106532 bits_offset=2048
> >
> > nf_conn dump from vmlinux:
> >
> > [106517] PTR '(anon)' type_id=106518
> > [106518] STRUCT 'nf_conn' size=320 vlen=14
> > 'ct_general' type_id=105882 bits_offset=0
> > 'lock' type_id=180 bits_offset=64
> > 'timeout' type_id=113 bits_offset=640
> > 'zone' type_id=106520 bits_offset=672
> > 'tuplehash' type_id=106533 bits_offset=704
> > 'status' type_id=1 bits_offset=1600
> > 'ct_net' type_id=3215 bits_offset=1664
> > 'nat_bysource' type_id=139 bits_offset=1728
> > '__nfct_init_offset' type_id=949 bits_offset=1856
> > 'master' type_id=106517 bits_offset=1856
> > 'mark' type_id=106351 bits_offset=1920
> > 'secmark' type_id=106351 bits_offset=1952
> > 'ext' type_id=106536 bits_offset=1984
> > 'proto' type_id=106532 bits_offset=2048
> >
> > look identical.. Andrii, any idea?
>
> I'm pretty sure they are not identical. There is somewhere a STRUCT vs
> FWD difference. We had a similar discussion recently with Alan
> Maguire.
>
> > 'master' type_id=120155 bits_offset=1856
>
> vs
>
> > 'master' type_id=106517 bits_offset=1856
master is pointer to same 'nf_conn' object, and rest of the ids are same
jirka
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-14 6:47 ` Jiri Olsa
@ 2022-10-14 17:17 ` Alan Maguire
2022-10-21 18:18 ` Andrii Nakryiko
1 sibling, 0 replies; 22+ messages in thread
From: Alan Maguire @ 2022-10-14 17:17 UTC (permalink / raw)
To: Jiri Olsa, Andrii Nakryiko
Cc: Jakub Kicinski, Martin KaFai Lau, bpf, netdev@vger.kernel.org
On 14/10/2022 07:47, Jiri Olsa wrote:
> On Thu, Oct 13, 2022 at 03:24:59PM -0700, Andrii Nakryiko wrote:
>> On Thu, Oct 13, 2022 at 3:12 PM Jiri Olsa <olsajiri@gmail.com> wrote:
>>>
>>> On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote:
>>>> On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote:
>>>>>> Yeah, it's there on linux-next, too.
>>>>>>
>>>>>> Let me grab a fresh VM and try there. Maybe it's my system. Somehow.
>>>>>
>>>>> ok, I will look around what's the way to install that centos 8 thing
>>>>
>>>> Any luck?
>>>
>>> now BTFIDS warnings..
>>>
>>> I can see following on centos8 with gcc 8.5:
>>>
>>> BTFIDS vmlinux
>>> WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300
>>> WARN: multiple IDs found for 'file': 540, 56649 - using 540
>>> WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549
>>> WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953
>>> WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132
>>> WARN: multiple IDs found for 'path': 1164, 56995 - using 1164
>>> WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300
>>> WARN: multiple IDs found for 'file': 540, 61943 - using 540
>>> WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549
>>> WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132
>>> WARN: multiple IDs found for 'path': 1164, 62058 - using 1164
>>> WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190
>>> WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953
>>> WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960
>>> WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876
>>> WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094
>>> WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993
>>> WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191
>>> WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164
>>> WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296
>>> WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292
>>> WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700
>>> WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944
>>> WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299
>>> WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969
>>> WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623
>>>
>>> I'll need to check on that..
>>>
>>> and I just actually saw the 'nf_conn' warning on linux-next/master with
>>> latest fedora/gcc-12:
>>>
>>> BTF [M] net/netfilter/nf_nat.ko
>>> WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518
>>> WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518
>>> WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518
>>> WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518
>>> WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518
>>> WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518
>>> LD [M] net/netfilter/nf_nat_tftp.ko
>>>
>>> looks like maybe dedup missed this struct for some reason
>>>
>>> nf_conn dump from module:
>>>
>>> [120155] PTR '(anon)' type_id=120156
>>> [120156] STRUCT 'nf_conn' size=320 vlen=14
>>> 'ct_general' type_id=105882 bits_offset=0
>>> 'lock' type_id=180 bits_offset=64
>>> 'timeout' type_id=113 bits_offset=640
>>> 'zone' type_id=106520 bits_offset=672
>>> 'tuplehash' type_id=106533 bits_offset=704
>>> 'status' type_id=1 bits_offset=1600
>>> 'ct_net' type_id=3215 bits_offset=1664
>>> 'nat_bysource' type_id=139 bits_offset=1728
>>> '__nfct_init_offset' type_id=949 bits_offset=1856
>>> 'master' type_id=120155 bits_offset=1856
>>> 'mark' type_id=106351 bits_offset=1920
>>> 'secmark' type_id=106351 bits_offset=1952
>>> 'ext' type_id=106536 bits_offset=1984
>>> 'proto' type_id=106532 bits_offset=2048
>>>
>>> nf_conn dump from vmlinux:
>>>
>>> [106517] PTR '(anon)' type_id=106518
>>> [106518] STRUCT 'nf_conn' size=320 vlen=14
>>> 'ct_general' type_id=105882 bits_offset=0
>>> 'lock' type_id=180 bits_offset=64
>>> 'timeout' type_id=113 bits_offset=640
>>> 'zone' type_id=106520 bits_offset=672
>>> 'tuplehash' type_id=106533 bits_offset=704
>>> 'status' type_id=1 bits_offset=1600
>>> 'ct_net' type_id=3215 bits_offset=1664
>>> 'nat_bysource' type_id=139 bits_offset=1728
>>> '__nfct_init_offset' type_id=949 bits_offset=1856
>>> 'master' type_id=106517 bits_offset=1856
>>> 'mark' type_id=106351 bits_offset=1920
>>> 'secmark' type_id=106351 bits_offset=1952
>>> 'ext' type_id=106536 bits_offset=1984
>>> 'proto' type_id=106532 bits_offset=2048
>>>
>>> look identical.. Andrii, any idea?
>>
>> I'm pretty sure they are not identical. There is somewhere a STRUCT vs
>> FWD difference. We had a similar discussion recently with Alan
>> Maguire.
>>
>>> 'master' type_id=120155 bits_offset=1856
>>
>> vs
>>
>>> 'master' type_id=106517 bits_offset=1856
>
> master is pointer to same 'nf_conn' object, and rest of the ids are same
>
> jirka
>
I tried digging into this problem a bit - in my case I was seeing
"struct sk_buff" duplicated in kernel/module BTF. Here's what I found..
Consider a situation like this, where one header file defining a struct s1
has a pointer field, pointing at struct s2. But struct s2 is a fwd definition.
$ cat s1.h
#include <stdio.h>
struct s2;
struct s1 {
struct s1 *f1;
struct s2 *f2;
};
$ cat s1.c
#include "s1.h"
int main(int argc, char *argv[])
{
struct s1 s1;
return 0;
}
Now consider a separate program s2, that #includes definitions for both
s1 and s2:
$ cat s2.h
#include <stdio.h>
struct s1;
struct s2 {
struct s1 *f1;
};
$cat s2.c
#include "s2.h"
#include "s1.h"
int main(int argc, char *argv[])
{
struct s1 s1 = {};
struct s2 s2 = {};
return 0;
}
In this case the generated base BTF contains a definition for s1,
and a FWD for s2, but the "module" BTF for s2 contains a full
definition for s2, so the dedup fails:
$ bpftool btf dump file s1
[29] STRUCT 's1' size=16 vlen=2
'f1' type_id=30 bits_offset=0
'f2' type_id=32 bits_offset=64
[30] PTR '(anon)' type_id=29
[31] FWD 's2' fwd_kind=struct
$ bpftool btf dump -B s1 file s2
[36] STRUCT 's2' size=8 vlen=1
'f1' type_id=38 bits_offset=0
[37] STRUCT 's1' size=16 vlen=2
'f1' type_id=38 bits_offset=0
'f2' type_id=39 bits_offset=64
[38] PTR '(anon)' type_id=37
[39] PTR '(anon)' type_id=36
So we had to redefine struct s1 in the "module" because the
FWD wasn't resolved in the base BTF. This is by design as I
understand it; in effect we can't supplement base BTF with
info we've gotten from module BTF about forward resolution
(at least that's my understanding of the reason).
Now does this sort of thing happen in the kernel? It looks like
it; consider struct nf_conn; it contains a possible_net_t:
typedef struct {
struct net * net; /* 0 8 */
/* size: 8, cachelines: 1, members: 1 */
/* last cacheline: 8 bytes */
} possible_net_t;
...and a struct net * contains pointers to structures
that aren't in the vmlinux BTF (because they are
in modules); for example:
struct netns_ipvs * ipvs; /* 3912 8 */
$ pahole netns_ipvs
pahole: type 'netns_ipvs' not found
...and in vmlinux BTF it is:
[2983] FWD 'netns_ipvs' fwd_kind=struct
[2984] PTR '(anon)' type_id=2983
...and in struct net we can see the fwd type is referenced alright:
[2021] STRUCT 'net' size=4288 vlen=52
...
'ipvs' type_id=2984 bits_offset=31808
So we'd expect any ipvs-related modules to not dedup
struct net, since they'll have the full definition
for netns_ipvs. In xt_ipvs.ko we see:
[111924] STRUCT 'netns_ipvs' size=2176 vlen=78
'gen' type_id=21 bits_offset=0
'enable' type_id=21 bits_offset=32
'rs_table' type_id=4044 bits_offset=64
'app_list' type_id=83 bits_offset=1088
...and when we look at 'struct net' we see:
[111786] STRUCT 'net' size=4288 vlen=52
...
'ipvs' type_id=111925 bits_offset=31808
And then if we don't dedup struct net, it seems likely
that structures referencing struct net (like skbs,
nf_conn etc) won't dedup either since they'll point
at "their" version of struct net.
Not sure if that's the root cause here, but it seems
like it is happening in other modules at least.
More subtle effects are also possible I think; if a type
is in a header file is defined but not referenced anywhere
(as might well happen for a module-related type in vmlinux),
it might not always make it into the DWARF description,
and as a result of that might not have a BTF
representation.
Alan
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-04 14:46 ` Daniel Xu
2022-10-04 15:34 ` Jakub Kicinski
@ 2022-10-18 15:11 ` Christian Borntraeger
1 sibling, 0 replies; 22+ messages in thread
From: Christian Borntraeger @ 2022-10-18 15:11 UTC (permalink / raw)
To: dxu; +Cc: bpf, jolsa, kuba, martin.lau, netdev, linux-kernel
chiming in. I also see that on s390x with 6.1-rc.
latest greates pahole does seem to fix this for me on s390 with gcc, but
with clang I still see this.
Christian
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-14 6:47 ` Jiri Olsa
2022-10-14 17:17 ` Alan Maguire
@ 2022-10-21 18:18 ` Andrii Nakryiko
2022-10-21 21:34 ` Jiri Olsa
1 sibling, 1 reply; 22+ messages in thread
From: Andrii Nakryiko @ 2022-10-21 18:18 UTC (permalink / raw)
To: Jiri Olsa; +Cc: Jakub Kicinski, Martin KaFai Lau, bpf, netdev@vger.kernel.org
On Thu, Oct 13, 2022 at 11:47 PM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Thu, Oct 13, 2022 at 03:24:59PM -0700, Andrii Nakryiko wrote:
> > On Thu, Oct 13, 2022 at 3:12 PM Jiri Olsa <olsajiri@gmail.com> wrote:
> > >
> > > On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote:
> > > > On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote:
> > > > > > Yeah, it's there on linux-next, too.
> > > > > >
> > > > > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow.
> > > > >
> > > > > ok, I will look around what's the way to install that centos 8 thing
> > > >
> > > > Any luck?
> > >
> > > now BTFIDS warnings..
> > >
> > > I can see following on centos8 with gcc 8.5:
> > >
> > > BTFIDS vmlinux
> > > WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300
> > > WARN: multiple IDs found for 'file': 540, 56649 - using 540
> > > WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549
> > > WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953
> > > WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132
> > > WARN: multiple IDs found for 'path': 1164, 56995 - using 1164
> > > WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300
> > > WARN: multiple IDs found for 'file': 540, 61943 - using 540
> > > WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549
> > > WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132
> > > WARN: multiple IDs found for 'path': 1164, 62058 - using 1164
> > > WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190
> > > WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953
> > > WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960
> > > WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876
> > > WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094
> > > WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993
> > > WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191
> > > WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164
> > > WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296
> > > WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292
> > > WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700
> > > WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944
> > > WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299
> > > WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969
> > > WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623
> > >
> > > I'll need to check on that..
> > >
> > > and I just actually saw the 'nf_conn' warning on linux-next/master with
> > > latest fedora/gcc-12:
> > >
> > > BTF [M] net/netfilter/nf_nat.ko
> > > WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518
> > > WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518
> > > WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518
> > > WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518
> > > WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518
> > > WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518
> > > LD [M] net/netfilter/nf_nat_tftp.ko
> > >
> > > looks like maybe dedup missed this struct for some reason
> > >
> > > nf_conn dump from module:
> > >
> > > [120155] PTR '(anon)' type_id=120156
> > > [120156] STRUCT 'nf_conn' size=320 vlen=14
> > > 'ct_general' type_id=105882 bits_offset=0
> > > 'lock' type_id=180 bits_offset=64
> > > 'timeout' type_id=113 bits_offset=640
> > > 'zone' type_id=106520 bits_offset=672
> > > 'tuplehash' type_id=106533 bits_offset=704
> > > 'status' type_id=1 bits_offset=1600
> > > 'ct_net' type_id=3215 bits_offset=1664
> > > 'nat_bysource' type_id=139 bits_offset=1728
> > > '__nfct_init_offset' type_id=949 bits_offset=1856
> > > 'master' type_id=120155 bits_offset=1856
> > > 'mark' type_id=106351 bits_offset=1920
> > > 'secmark' type_id=106351 bits_offset=1952
> > > 'ext' type_id=106536 bits_offset=1984
> > > 'proto' type_id=106532 bits_offset=2048
> > >
> > > nf_conn dump from vmlinux:
> > >
> > > [106517] PTR '(anon)' type_id=106518
> > > [106518] STRUCT 'nf_conn' size=320 vlen=14
> > > 'ct_general' type_id=105882 bits_offset=0
> > > 'lock' type_id=180 bits_offset=64
> > > 'timeout' type_id=113 bits_offset=640
> > > 'zone' type_id=106520 bits_offset=672
> > > 'tuplehash' type_id=106533 bits_offset=704
> > > 'status' type_id=1 bits_offset=1600
> > > 'ct_net' type_id=3215 bits_offset=1664
> > > 'nat_bysource' type_id=139 bits_offset=1728
> > > '__nfct_init_offset' type_id=949 bits_offset=1856
> > > 'master' type_id=106517 bits_offset=1856
> > > 'mark' type_id=106351 bits_offset=1920
> > > 'secmark' type_id=106351 bits_offset=1952
> > > 'ext' type_id=106536 bits_offset=1984
> > > 'proto' type_id=106532 bits_offset=2048
> > >
> > > look identical.. Andrii, any idea?
> >
> > I'm pretty sure they are not identical. There is somewhere a STRUCT vs
> > FWD difference. We had a similar discussion recently with Alan
> > Maguire.
> >
> > > 'master' type_id=120155 bits_offset=1856
> >
> > vs
> >
> > > 'master' type_id=106517 bits_offset=1856
>
> master is pointer to same 'nf_conn' object, and rest of the ids are same
>
You are right, they should be identical once PTR is deduplicated
properly. Sorry, was too quick to jump to conclusions. I was thinking
about situations explained by Alan.
So, is this still an issue or this was fixed by [0]?
[0] https://lore.kernel.org/bpf/1666364523-9648-1-git-send-email-alan.maguire@oracle.com/
> jirka
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-21 18:18 ` Andrii Nakryiko
@ 2022-10-21 21:34 ` Jiri Olsa
2022-10-22 5:36 ` Jakub Kicinski
0 siblings, 1 reply; 22+ messages in thread
From: Jiri Olsa @ 2022-10-21 21:34 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Jiri Olsa, Jakub Kicinski, Martin KaFai Lau, bpf,
netdev@vger.kernel.org
On Fri, Oct 21, 2022 at 11:18:39AM -0700, Andrii Nakryiko wrote:
> On Thu, Oct 13, 2022 at 11:47 PM Jiri Olsa <olsajiri@gmail.com> wrote:
> >
> > On Thu, Oct 13, 2022 at 03:24:59PM -0700, Andrii Nakryiko wrote:
> > > On Thu, Oct 13, 2022 at 3:12 PM Jiri Olsa <olsajiri@gmail.com> wrote:
> > > >
> > > > On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote:
> > > > > On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote:
> > > > > > > Yeah, it's there on linux-next, too.
> > > > > > >
> > > > > > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow.
> > > > > >
> > > > > > ok, I will look around what's the way to install that centos 8 thing
> > > > >
> > > > > Any luck?
> > > >
> > > > now BTFIDS warnings..
> > > >
> > > > I can see following on centos8 with gcc 8.5:
> > > >
> > > > BTFIDS vmlinux
> > > > WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300
> > > > WARN: multiple IDs found for 'file': 540, 56649 - using 540
> > > > WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549
> > > > WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953
> > > > WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132
> > > > WARN: multiple IDs found for 'path': 1164, 56995 - using 1164
> > > > WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300
> > > > WARN: multiple IDs found for 'file': 540, 61943 - using 540
> > > > WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549
> > > > WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132
> > > > WARN: multiple IDs found for 'path': 1164, 62058 - using 1164
> > > > WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190
> > > > WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953
> > > > WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960
> > > > WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876
> > > > WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094
> > > > WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993
> > > > WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191
> > > > WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164
> > > > WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296
> > > > WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292
> > > > WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700
> > > > WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944
> > > > WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299
> > > > WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969
> > > > WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623
> > > >
> > > > I'll need to check on that..
> > > >
> > > > and I just actually saw the 'nf_conn' warning on linux-next/master with
> > > > latest fedora/gcc-12:
> > > >
> > > > BTF [M] net/netfilter/nf_nat.ko
> > > > WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518
> > > > WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518
> > > > WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518
> > > > WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518
> > > > WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518
> > > > WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518
> > > > LD [M] net/netfilter/nf_nat_tftp.ko
> > > >
> > > > looks like maybe dedup missed this struct for some reason
> > > >
> > > > nf_conn dump from module:
> > > >
> > > > [120155] PTR '(anon)' type_id=120156
> > > > [120156] STRUCT 'nf_conn' size=320 vlen=14
> > > > 'ct_general' type_id=105882 bits_offset=0
> > > > 'lock' type_id=180 bits_offset=64
> > > > 'timeout' type_id=113 bits_offset=640
> > > > 'zone' type_id=106520 bits_offset=672
> > > > 'tuplehash' type_id=106533 bits_offset=704
> > > > 'status' type_id=1 bits_offset=1600
> > > > 'ct_net' type_id=3215 bits_offset=1664
> > > > 'nat_bysource' type_id=139 bits_offset=1728
> > > > '__nfct_init_offset' type_id=949 bits_offset=1856
> > > > 'master' type_id=120155 bits_offset=1856
> > > > 'mark' type_id=106351 bits_offset=1920
> > > > 'secmark' type_id=106351 bits_offset=1952
> > > > 'ext' type_id=106536 bits_offset=1984
> > > > 'proto' type_id=106532 bits_offset=2048
> > > >
> > > > nf_conn dump from vmlinux:
> > > >
> > > > [106517] PTR '(anon)' type_id=106518
> > > > [106518] STRUCT 'nf_conn' size=320 vlen=14
> > > > 'ct_general' type_id=105882 bits_offset=0
> > > > 'lock' type_id=180 bits_offset=64
> > > > 'timeout' type_id=113 bits_offset=640
> > > > 'zone' type_id=106520 bits_offset=672
> > > > 'tuplehash' type_id=106533 bits_offset=704
> > > > 'status' type_id=1 bits_offset=1600
> > > > 'ct_net' type_id=3215 bits_offset=1664
> > > > 'nat_bysource' type_id=139 bits_offset=1728
> > > > '__nfct_init_offset' type_id=949 bits_offset=1856
> > > > 'master' type_id=106517 bits_offset=1856
> > > > 'mark' type_id=106351 bits_offset=1920
> > > > 'secmark' type_id=106351 bits_offset=1952
> > > > 'ext' type_id=106536 bits_offset=1984
> > > > 'proto' type_id=106532 bits_offset=2048
> > > >
> > > > look identical.. Andrii, any idea?
> > >
> > > I'm pretty sure they are not identical. There is somewhere a STRUCT vs
> > > FWD difference. We had a similar discussion recently with Alan
> > > Maguire.
> > >
> > > > 'master' type_id=120155 bits_offset=1856
> > >
> > > vs
> > >
> > > > 'master' type_id=106517 bits_offset=1856
> >
> > master is pointer to same 'nf_conn' object, and rest of the ids are same
> >
>
> You are right, they should be identical once PTR is deduplicated
> properly. Sorry, was too quick to jump to conclusions. I was thinking
> about situations explained by Alan.
>
> So, is this still an issue or this was fixed by [0]?
>
> [0] https://lore.kernel.org/bpf/1666364523-9648-1-git-send-email-alan.maguire@oracle.com/
yes, it seems to be fixed by that
Jakub,
could you check with pahole fix [1]?
thanks,
jirka
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-21 21:34 ` Jiri Olsa
@ 2022-10-22 5:36 ` Jakub Kicinski
2022-10-23 1:18 ` Alexei Starovoitov
0 siblings, 1 reply; 22+ messages in thread
From: Jakub Kicinski @ 2022-10-22 5:36 UTC (permalink / raw)
To: Jiri Olsa; +Cc: Andrii Nakryiko, Martin KaFai Lau, bpf, netdev@vger.kernel.org
On Fri, 21 Oct 2022 23:34:13 +0200 Jiri Olsa wrote:
> > You are right, they should be identical once PTR is deduplicated
> > properly. Sorry, was too quick to jump to conclusions. I was thinking
> > about situations explained by Alan.
> >
> > So, is this still an issue or this was fixed by [0]?
> >
> > [0] https://lore.kernel.org/bpf/1666364523-9648-1-git-send-email-alan.maguire@oracle.com/
>
> yes, it seems to be fixed by that
>
> Jakub,
> could you check with pahole fix [1]?
If you mean the warning from the subject then those do seem to be gone.
But if I'm completely honest I don't remember how I triggered them in
the first place :S There weren't there on every build for me.
The objtool warning is still here:
$ make PAHOLE=~/pahole O=build_allmodconfig/ -j 60 >/tmp/stdout 2>/tmp/stderr; \
cat /tmp/stderr
vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-22 5:36 ` Jakub Kicinski
@ 2022-10-23 1:18 ` Alexei Starovoitov
2022-10-24 18:21 ` Jakub Kicinski
0 siblings, 1 reply; 22+ messages in thread
From: Alexei Starovoitov @ 2022-10-23 1:18 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Jiri Olsa, Andrii Nakryiko, Martin KaFai Lau, bpf,
netdev@vger.kernel.org
On Fri, Oct 21, 2022 at 10:38 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Fri, 21 Oct 2022 23:34:13 +0200 Jiri Olsa wrote:
> > > You are right, they should be identical once PTR is deduplicated
> > > properly. Sorry, was too quick to jump to conclusions. I was thinking
> > > about situations explained by Alan.
> > >
> > > So, is this still an issue or this was fixed by [0]?
> > >
> > > [0] https://lore.kernel.org/bpf/1666364523-9648-1-git-send-email-alan.maguire@oracle.com/
> >
> > yes, it seems to be fixed by that
> >
> > Jakub,
> > could you check with pahole fix [1]?
>
> If you mean the warning from the subject then those do seem to be gone.
> But if I'm completely honest I don't remember how I triggered them in
> the first place :S There weren't there on every build for me.
>
> The objtool warning is still here:
>
> $ make PAHOLE=~/pahole O=build_allmodconfig/ -j 60 >/tmp/stdout 2>/tmp/stderr; \
> cat /tmp/stderr
>
> vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
> vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
The effect of the compiler bug was addressed by this fix:
https://lore.kernel.org/all/20221018075934.574415-1-jolsa@kernel.org/
It's in the bpf tree, but the warning will stay.
While the compiler is broken the objtool should keep complaining.
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168
2022-10-23 1:18 ` Alexei Starovoitov
@ 2022-10-24 18:21 ` Jakub Kicinski
0 siblings, 0 replies; 22+ messages in thread
From: Jakub Kicinski @ 2022-10-24 18:21 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Jiri Olsa, Andrii Nakryiko, Martin KaFai Lau, bpf,
netdev@vger.kernel.org
On Sat, 22 Oct 2022 18:18:49 -0700 Alexei Starovoitov wrote:
> > If you mean the warning from the subject then those do seem to be gone.
> > But if I'm completely honest I don't remember how I triggered them in
> > the first place :S There weren't there on every build for me.
> >
> > The objtool warning is still here:
> >
> > $ make PAHOLE=~/pahole O=build_allmodconfig/ -j 60 >/tmp/stdout 2>/tmp/stderr; \
> > cat /tmp/stderr
> >
> > vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
> > vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0
>
> The effect of the compiler bug was addressed by this fix:
> https://lore.kernel.org/all/20221018075934.574415-1-jolsa@kernel.org/
>
> It's in the bpf tree, but the warning will stay.
> While the compiler is broken the objtool should keep complaining.
Thanks! I'll stop tracking it
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2022-10-24 19:59 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-10-04 2:05 objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: Jakub Kicinski
2022-10-04 4:49 ` WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 Jakub Kicinski
2022-10-04 6:42 ` Jiri Olsa
2022-10-04 14:25 ` Jakub Kicinski
2022-10-05 9:45 ` Jiri Olsa
2022-10-05 15:44 ` Jakub Kicinski
2022-10-05 16:18 ` Jakub Kicinski
2022-10-05 20:07 ` Jiri Olsa
2022-10-13 15:05 ` Jakub Kicinski
2022-10-13 21:55 ` Jiri Olsa
2022-10-13 22:12 ` Jiri Olsa
2022-10-13 22:24 ` Andrii Nakryiko
2022-10-14 6:47 ` Jiri Olsa
2022-10-14 17:17 ` Alan Maguire
2022-10-21 18:18 ` Andrii Nakryiko
2022-10-21 21:34 ` Jiri Olsa
2022-10-22 5:36 ` Jakub Kicinski
2022-10-23 1:18 ` Alexei Starovoitov
2022-10-24 18:21 ` Jakub Kicinski
2022-10-04 14:46 ` Daniel Xu
2022-10-04 15:34 ` Jakub Kicinski
2022-10-18 15:11 ` Christian Borntraeger
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).