* objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: @ 2022-10-04 2:05 Jakub Kicinski 2022-10-04 4:49 ` WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 Jakub Kicinski 0 siblings, 1 reply; 22+ messages in thread From: Jakub Kicinski @ 2022-10-04 2:05 UTC (permalink / raw) To: Jiri Olsa; +Cc: bpf, netdev@vger.kernel.org Hi Jiri, I get the following warning after merging up all the trees: vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 $ gcc --version gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15) Is this known? ^ permalink raw reply [flat|nested] 22+ messages in thread
* WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-04 2:05 objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: Jakub Kicinski @ 2022-10-04 4:49 ` Jakub Kicinski 2022-10-04 6:42 ` Jiri Olsa 2022-10-04 14:46 ` Daniel Xu 0 siblings, 2 replies; 22+ messages in thread From: Jakub Kicinski @ 2022-10-04 4:49 UTC (permalink / raw) To: Daniel Xu, Martin KaFai Lau; +Cc: Jiri Olsa, bpf, netdev@vger.kernel.org On Mon, 3 Oct 2022 19:05:45 -0700 Jakub Kicinski wrote: > Hi Jiri, > > I get the following warning after merging up all the trees: > > vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 > vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 > > $ gcc --version > gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15) > > > Is this known? Also hit this: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 WARN: multiple IDs found for 'nf_conn': 92168, 121226 - using 92168 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-04 4:49 ` WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 Jakub Kicinski @ 2022-10-04 6:42 ` Jiri Olsa 2022-10-04 14:25 ` Jakub Kicinski 2022-10-04 14:46 ` Daniel Xu 1 sibling, 1 reply; 22+ messages in thread From: Jiri Olsa @ 2022-10-04 6:42 UTC (permalink / raw) To: Jakub Kicinski; +Cc: Daniel Xu, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Mon, Oct 03, 2022 at 09:49:41PM -0700, Jakub Kicinski wrote: > On Mon, 3 Oct 2022 19:05:45 -0700 Jakub Kicinski wrote: > > Hi Jiri, > > > > I get the following warning after merging up all the trees: > > > > vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 > > vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 > > > > $ gcc --version > > gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15) > > > > > > Is this known? > > Also hit this: > > WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 > WARN: multiple IDs found for 'nf_conn': 92168, 121226 - using 92168 hi, I did not see that before, could you please share your config? thanks, jirka ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-04 6:42 ` Jiri Olsa @ 2022-10-04 14:25 ` Jakub Kicinski 2022-10-05 9:45 ` Jiri Olsa 0 siblings, 1 reply; 22+ messages in thread From: Jakub Kicinski @ 2022-10-04 14:25 UTC (permalink / raw) To: Jiri Olsa; +Cc: Daniel Xu, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Tue, 4 Oct 2022 08:42:24 +0200 Jiri Olsa wrote: > I did not see that before, could you please share your config? allmodconfig ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-04 14:25 ` Jakub Kicinski @ 2022-10-05 9:45 ` Jiri Olsa 2022-10-05 15:44 ` Jakub Kicinski 0 siblings, 1 reply; 22+ messages in thread From: Jiri Olsa @ 2022-10-05 9:45 UTC (permalink / raw) To: Jakub Kicinski Cc: Jiri Olsa, Daniel Xu, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Tue, Oct 04, 2022 at 07:25:22AM -0700, Jakub Kicinski wrote: > On Tue, 4 Oct 2022 08:42:24 +0200 Jiri Olsa wrote: > > I did not see that before, could you please share your config? > > allmodconfig I compiled linux-next with that and can't see that, any other hint (apart the old gcc) about your setup? jirka ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-05 9:45 ` Jiri Olsa @ 2022-10-05 15:44 ` Jakub Kicinski 2022-10-05 16:18 ` Jakub Kicinski 0 siblings, 1 reply; 22+ messages in thread From: Jakub Kicinski @ 2022-10-05 15:44 UTC (permalink / raw) To: Jiri Olsa; +Cc: Daniel Xu, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Wed, 5 Oct 2022 11:45:46 +0200 Jiri Olsa wrote: > On Tue, Oct 04, 2022 at 07:25:22AM -0700, Jakub Kicinski wrote: > > On Tue, 4 Oct 2022 08:42:24 +0200 Jiri Olsa wrote: > > > I did not see that before, could you please share your config? > > > > allmodconfig > > I compiled linux-next with that and can't see that, > any other hint (apart the old gcc) about your setup? Hm, I was compiling Linus's tree not linux-next. Let me try linux-next right now. Did you use the 8.5 gcc (which I believe is what comes with CentOS Stream)? I only see it there. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-05 15:44 ` Jakub Kicinski @ 2022-10-05 16:18 ` Jakub Kicinski 2022-10-05 20:07 ` Jiri Olsa 0 siblings, 1 reply; 22+ messages in thread From: Jakub Kicinski @ 2022-10-05 16:18 UTC (permalink / raw) To: Jiri Olsa; +Cc: Daniel Xu, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Wed, 5 Oct 2022 08:44:42 -0700 Jakub Kicinski wrote: > Hm, I was compiling Linus's tree not linux-next. > Let me try linux-next right now. > > Did you use the 8.5 gcc (which I believe is what comes with > CentOS Stream)? I only see it there. Yeah, it's there on linux-next, too. Let me grab a fresh VM and try there. Maybe it's my system. Somehow. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-05 16:18 ` Jakub Kicinski @ 2022-10-05 20:07 ` Jiri Olsa 2022-10-13 15:05 ` Jakub Kicinski 0 siblings, 1 reply; 22+ messages in thread From: Jiri Olsa @ 2022-10-05 20:07 UTC (permalink / raw) To: Jakub Kicinski Cc: Jiri Olsa, Daniel Xu, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Wed, Oct 05, 2022 at 09:18:01AM -0700, Jakub Kicinski wrote: > On Wed, 5 Oct 2022 08:44:42 -0700 Jakub Kicinski wrote: > > Hm, I was compiling Linus's tree not linux-next. > > Let me try linux-next right now. > > > > Did you use the 8.5 gcc (which I believe is what comes with > > CentOS Stream)? I only see it there. nope.. latest fedora 12 > > Yeah, it's there on linux-next, too. > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow. ok, I will look around what's the way to install that centos 8 thing jirka ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-05 20:07 ` Jiri Olsa @ 2022-10-13 15:05 ` Jakub Kicinski 2022-10-13 21:55 ` Jiri Olsa 2022-10-13 22:12 ` Jiri Olsa 0 siblings, 2 replies; 22+ messages in thread From: Jakub Kicinski @ 2022-10-13 15:05 UTC (permalink / raw) To: Jiri Olsa; +Cc: Martin KaFai Lau, bpf, netdev@vger.kernel.org On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote: > > Yeah, it's there on linux-next, too. > > > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow. > > ok, I will look around what's the way to install that centos 8 thing Any luck? ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-13 15:05 ` Jakub Kicinski @ 2022-10-13 21:55 ` Jiri Olsa 2022-10-13 22:12 ` Jiri Olsa 1 sibling, 0 replies; 22+ messages in thread From: Jiri Olsa @ 2022-10-13 21:55 UTC (permalink / raw) To: Jakub Kicinski, Peter Zijlstra Cc: Jiri Olsa, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote: > On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote: > > > Yeah, it's there on linux-next, too. > > > > > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow. > > > > ok, I will look around what's the way to install that centos 8 thing > > Any luck? yea, sorry for delay, I reproduced that.. first objtool warnings ;-) [jolsa@centos8 linux-next]$ make UPD include/generated/compile.h CALL scripts/checksyscalls.sh DESCEND objtool DESCEND bpf/resolve_btfids CC init/version.o AR init/built-in.a AR built-in.a AR vmlinux.a LD vmlinux.o vmlinux.o: warning: objtool: relocate_restore_code+0x3c: relocation to !ENDBR: next_arg+0x8 vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0x20: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 ... Peter, as for objtool warnings, looks like with gcc we'll get endbr64 instruction generated after nops with centos gcc 8.5: ffffffff818d2e20 <bpf_dispatcher_xdp_func>: ffffffff818d2e20: 90 nop ffffffff818d2e21: 90 nop ffffffff818d2e22: 90 nop ffffffff818d2e23: 90 nop ffffffff818d2e24: 90 nop ffffffff818d2e25: f3 0f 1e fa endbr64 ffffffff818d2e29: e9 92 eb 52 00 jmpq ffffffff81e019c0 <__x86_indirect_thunk_rdx> ffffffff818d2e2e: 66 90 xchg %ax,%ax while latest gcc 12 will put it after: ffffffff82736900 <bpf_dispatcher_xdp_func>: ffffffff82736900: f3 0f 1e fa endbr64 ffffffff82736904: 90 nop ffffffff82736905: 90 nop ffffffff82736906: 90 nop ffffffff82736907: 90 nop ffffffff82736908: 90 nop ffffffff82736909: 41 54 push %r12 ffffffff8273690b: 49 89 f4 mov %rsi,%r12 ffffffff8273690e: 55 push %rbp ffffffff8273690f: 48 89 fd mov %rdi,%rbp ffffffff82736912: 53 push %rbx ffffffff82736913: 48 89 d3 mov %rdx,%rbx ffffffff82736916: e8 65 f6 cf fe call ffffffff81435f80 <__sanitizer_cov_trace_pc> ffffffff8273691b: 4c 89 e6 mov %r12,%rsi ffffffff8273691e: 48 89 ef mov %rbp,%rdi ffffffff82736921: 48 89 d8 mov %rbx,%rax ffffffff82736924: 5b pop %rbx ffffffff82736925: 5d pop %rbp ffffffff82736926: 41 5c pop %r12 ffffffff82736928: e9 b3 b6 8c 00 jmp ffffffff83001fe0 <__x86_indirect_thunk_array> any idea where's the problem? bad backport? ;-) in any case (unrelated), I'll check the bpf dispatcher code, I'm not sure the nop update code is aware of the endbr64 instruction jirka ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-13 15:05 ` Jakub Kicinski 2022-10-13 21:55 ` Jiri Olsa @ 2022-10-13 22:12 ` Jiri Olsa 2022-10-13 22:24 ` Andrii Nakryiko 1 sibling, 1 reply; 22+ messages in thread From: Jiri Olsa @ 2022-10-13 22:12 UTC (permalink / raw) To: Jakub Kicinski, Andrii Nakryiko Cc: Jiri Olsa, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote: > On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote: > > > Yeah, it's there on linux-next, too. > > > > > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow. > > > > ok, I will look around what's the way to install that centos 8 thing > > Any luck? now BTFIDS warnings.. I can see following on centos8 with gcc 8.5: BTFIDS vmlinux WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300 WARN: multiple IDs found for 'file': 540, 56649 - using 540 WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549 WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953 WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132 WARN: multiple IDs found for 'path': 1164, 56995 - using 1164 WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300 WARN: multiple IDs found for 'file': 540, 61943 - using 540 WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549 WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132 WARN: multiple IDs found for 'path': 1164, 62058 - using 1164 WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190 WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953 WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960 WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876 WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094 WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993 WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191 WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164 WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296 WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292 WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700 WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944 WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299 WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969 WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623 I'll need to check on that.. and I just actually saw the 'nf_conn' warning on linux-next/master with latest fedora/gcc-12: BTF [M] net/netfilter/nf_nat.ko WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518 WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518 WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518 WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518 WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518 WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518 LD [M] net/netfilter/nf_nat_tftp.ko looks like maybe dedup missed this struct for some reason nf_conn dump from module: [120155] PTR '(anon)' type_id=120156 [120156] STRUCT 'nf_conn' size=320 vlen=14 'ct_general' type_id=105882 bits_offset=0 'lock' type_id=180 bits_offset=64 'timeout' type_id=113 bits_offset=640 'zone' type_id=106520 bits_offset=672 'tuplehash' type_id=106533 bits_offset=704 'status' type_id=1 bits_offset=1600 'ct_net' type_id=3215 bits_offset=1664 'nat_bysource' type_id=139 bits_offset=1728 '__nfct_init_offset' type_id=949 bits_offset=1856 'master' type_id=120155 bits_offset=1856 'mark' type_id=106351 bits_offset=1920 'secmark' type_id=106351 bits_offset=1952 'ext' type_id=106536 bits_offset=1984 'proto' type_id=106532 bits_offset=2048 nf_conn dump from vmlinux: [106517] PTR '(anon)' type_id=106518 [106518] STRUCT 'nf_conn' size=320 vlen=14 'ct_general' type_id=105882 bits_offset=0 'lock' type_id=180 bits_offset=64 'timeout' type_id=113 bits_offset=640 'zone' type_id=106520 bits_offset=672 'tuplehash' type_id=106533 bits_offset=704 'status' type_id=1 bits_offset=1600 'ct_net' type_id=3215 bits_offset=1664 'nat_bysource' type_id=139 bits_offset=1728 '__nfct_init_offset' type_id=949 bits_offset=1856 'master' type_id=106517 bits_offset=1856 'mark' type_id=106351 bits_offset=1920 'secmark' type_id=106351 bits_offset=1952 'ext' type_id=106536 bits_offset=1984 'proto' type_id=106532 bits_offset=2048 look identical.. Andrii, any idea? thanks, jirka ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-13 22:12 ` Jiri Olsa @ 2022-10-13 22:24 ` Andrii Nakryiko 2022-10-14 6:47 ` Jiri Olsa 0 siblings, 1 reply; 22+ messages in thread From: Andrii Nakryiko @ 2022-10-13 22:24 UTC (permalink / raw) To: Jiri Olsa; +Cc: Jakub Kicinski, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Thu, Oct 13, 2022 at 3:12 PM Jiri Olsa <olsajiri@gmail.com> wrote: > > On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote: > > On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote: > > > > Yeah, it's there on linux-next, too. > > > > > > > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow. > > > > > > ok, I will look around what's the way to install that centos 8 thing > > > > Any luck? > > now BTFIDS warnings.. > > I can see following on centos8 with gcc 8.5: > > BTFIDS vmlinux > WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300 > WARN: multiple IDs found for 'file': 540, 56649 - using 540 > WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549 > WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953 > WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132 > WARN: multiple IDs found for 'path': 1164, 56995 - using 1164 > WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300 > WARN: multiple IDs found for 'file': 540, 61943 - using 540 > WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549 > WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132 > WARN: multiple IDs found for 'path': 1164, 62058 - using 1164 > WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190 > WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953 > WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960 > WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876 > WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094 > WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993 > WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191 > WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164 > WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296 > WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292 > WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700 > WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944 > WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299 > WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969 > WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623 > > I'll need to check on that.. > > and I just actually saw the 'nf_conn' warning on linux-next/master with > latest fedora/gcc-12: > > BTF [M] net/netfilter/nf_nat.ko > WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518 > WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518 > WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518 > WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518 > WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518 > WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518 > LD [M] net/netfilter/nf_nat_tftp.ko > > looks like maybe dedup missed this struct for some reason > > nf_conn dump from module: > > [120155] PTR '(anon)' type_id=120156 > [120156] STRUCT 'nf_conn' size=320 vlen=14 > 'ct_general' type_id=105882 bits_offset=0 > 'lock' type_id=180 bits_offset=64 > 'timeout' type_id=113 bits_offset=640 > 'zone' type_id=106520 bits_offset=672 > 'tuplehash' type_id=106533 bits_offset=704 > 'status' type_id=1 bits_offset=1600 > 'ct_net' type_id=3215 bits_offset=1664 > 'nat_bysource' type_id=139 bits_offset=1728 > '__nfct_init_offset' type_id=949 bits_offset=1856 > 'master' type_id=120155 bits_offset=1856 > 'mark' type_id=106351 bits_offset=1920 > 'secmark' type_id=106351 bits_offset=1952 > 'ext' type_id=106536 bits_offset=1984 > 'proto' type_id=106532 bits_offset=2048 > > nf_conn dump from vmlinux: > > [106517] PTR '(anon)' type_id=106518 > [106518] STRUCT 'nf_conn' size=320 vlen=14 > 'ct_general' type_id=105882 bits_offset=0 > 'lock' type_id=180 bits_offset=64 > 'timeout' type_id=113 bits_offset=640 > 'zone' type_id=106520 bits_offset=672 > 'tuplehash' type_id=106533 bits_offset=704 > 'status' type_id=1 bits_offset=1600 > 'ct_net' type_id=3215 bits_offset=1664 > 'nat_bysource' type_id=139 bits_offset=1728 > '__nfct_init_offset' type_id=949 bits_offset=1856 > 'master' type_id=106517 bits_offset=1856 > 'mark' type_id=106351 bits_offset=1920 > 'secmark' type_id=106351 bits_offset=1952 > 'ext' type_id=106536 bits_offset=1984 > 'proto' type_id=106532 bits_offset=2048 > > look identical.. Andrii, any idea? I'm pretty sure they are not identical. There is somewhere a STRUCT vs FWD difference. We had a similar discussion recently with Alan Maguire. > 'master' type_id=120155 bits_offset=1856 vs > 'master' type_id=106517 bits_offset=1856 we'd need to unwind all these references to find where they start to differ > > thanks, > jirka ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-13 22:24 ` Andrii Nakryiko @ 2022-10-14 6:47 ` Jiri Olsa 2022-10-14 17:17 ` Alan Maguire 2022-10-21 18:18 ` Andrii Nakryiko 0 siblings, 2 replies; 22+ messages in thread From: Jiri Olsa @ 2022-10-14 6:47 UTC (permalink / raw) To: Andrii Nakryiko Cc: Jiri Olsa, Jakub Kicinski, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Thu, Oct 13, 2022 at 03:24:59PM -0700, Andrii Nakryiko wrote: > On Thu, Oct 13, 2022 at 3:12 PM Jiri Olsa <olsajiri@gmail.com> wrote: > > > > On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote: > > > On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote: > > > > > Yeah, it's there on linux-next, too. > > > > > > > > > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow. > > > > > > > > ok, I will look around what's the way to install that centos 8 thing > > > > > > Any luck? > > > > now BTFIDS warnings.. > > > > I can see following on centos8 with gcc 8.5: > > > > BTFIDS vmlinux > > WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300 > > WARN: multiple IDs found for 'file': 540, 56649 - using 540 > > WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549 > > WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953 > > WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132 > > WARN: multiple IDs found for 'path': 1164, 56995 - using 1164 > > WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300 > > WARN: multiple IDs found for 'file': 540, 61943 - using 540 > > WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549 > > WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132 > > WARN: multiple IDs found for 'path': 1164, 62058 - using 1164 > > WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190 > > WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953 > > WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960 > > WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876 > > WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094 > > WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993 > > WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191 > > WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164 > > WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296 > > WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292 > > WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700 > > WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944 > > WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299 > > WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969 > > WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623 > > > > I'll need to check on that.. > > > > and I just actually saw the 'nf_conn' warning on linux-next/master with > > latest fedora/gcc-12: > > > > BTF [M] net/netfilter/nf_nat.ko > > WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518 > > WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518 > > WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518 > > WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518 > > WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518 > > WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518 > > LD [M] net/netfilter/nf_nat_tftp.ko > > > > looks like maybe dedup missed this struct for some reason > > > > nf_conn dump from module: > > > > [120155] PTR '(anon)' type_id=120156 > > [120156] STRUCT 'nf_conn' size=320 vlen=14 > > 'ct_general' type_id=105882 bits_offset=0 > > 'lock' type_id=180 bits_offset=64 > > 'timeout' type_id=113 bits_offset=640 > > 'zone' type_id=106520 bits_offset=672 > > 'tuplehash' type_id=106533 bits_offset=704 > > 'status' type_id=1 bits_offset=1600 > > 'ct_net' type_id=3215 bits_offset=1664 > > 'nat_bysource' type_id=139 bits_offset=1728 > > '__nfct_init_offset' type_id=949 bits_offset=1856 > > 'master' type_id=120155 bits_offset=1856 > > 'mark' type_id=106351 bits_offset=1920 > > 'secmark' type_id=106351 bits_offset=1952 > > 'ext' type_id=106536 bits_offset=1984 > > 'proto' type_id=106532 bits_offset=2048 > > > > nf_conn dump from vmlinux: > > > > [106517] PTR '(anon)' type_id=106518 > > [106518] STRUCT 'nf_conn' size=320 vlen=14 > > 'ct_general' type_id=105882 bits_offset=0 > > 'lock' type_id=180 bits_offset=64 > > 'timeout' type_id=113 bits_offset=640 > > 'zone' type_id=106520 bits_offset=672 > > 'tuplehash' type_id=106533 bits_offset=704 > > 'status' type_id=1 bits_offset=1600 > > 'ct_net' type_id=3215 bits_offset=1664 > > 'nat_bysource' type_id=139 bits_offset=1728 > > '__nfct_init_offset' type_id=949 bits_offset=1856 > > 'master' type_id=106517 bits_offset=1856 > > 'mark' type_id=106351 bits_offset=1920 > > 'secmark' type_id=106351 bits_offset=1952 > > 'ext' type_id=106536 bits_offset=1984 > > 'proto' type_id=106532 bits_offset=2048 > > > > look identical.. Andrii, any idea? > > I'm pretty sure they are not identical. There is somewhere a STRUCT vs > FWD difference. We had a similar discussion recently with Alan > Maguire. > > > 'master' type_id=120155 bits_offset=1856 > > vs > > > 'master' type_id=106517 bits_offset=1856 master is pointer to same 'nf_conn' object, and rest of the ids are same jirka ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-14 6:47 ` Jiri Olsa @ 2022-10-14 17:17 ` Alan Maguire 2022-10-21 18:18 ` Andrii Nakryiko 1 sibling, 0 replies; 22+ messages in thread From: Alan Maguire @ 2022-10-14 17:17 UTC (permalink / raw) To: Jiri Olsa, Andrii Nakryiko Cc: Jakub Kicinski, Martin KaFai Lau, bpf, netdev@vger.kernel.org On 14/10/2022 07:47, Jiri Olsa wrote: > On Thu, Oct 13, 2022 at 03:24:59PM -0700, Andrii Nakryiko wrote: >> On Thu, Oct 13, 2022 at 3:12 PM Jiri Olsa <olsajiri@gmail.com> wrote: >>> >>> On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote: >>>> On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote: >>>>>> Yeah, it's there on linux-next, too. >>>>>> >>>>>> Let me grab a fresh VM and try there. Maybe it's my system. Somehow. >>>>> >>>>> ok, I will look around what's the way to install that centos 8 thing >>>> >>>> Any luck? >>> >>> now BTFIDS warnings.. >>> >>> I can see following on centos8 with gcc 8.5: >>> >>> BTFIDS vmlinux >>> WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300 >>> WARN: multiple IDs found for 'file': 540, 56649 - using 540 >>> WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549 >>> WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953 >>> WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132 >>> WARN: multiple IDs found for 'path': 1164, 56995 - using 1164 >>> WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300 >>> WARN: multiple IDs found for 'file': 540, 61943 - using 540 >>> WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549 >>> WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132 >>> WARN: multiple IDs found for 'path': 1164, 62058 - using 1164 >>> WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190 >>> WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953 >>> WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960 >>> WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876 >>> WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094 >>> WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993 >>> WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191 >>> WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164 >>> WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296 >>> WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292 >>> WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700 >>> WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944 >>> WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299 >>> WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969 >>> WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623 >>> >>> I'll need to check on that.. >>> >>> and I just actually saw the 'nf_conn' warning on linux-next/master with >>> latest fedora/gcc-12: >>> >>> BTF [M] net/netfilter/nf_nat.ko >>> WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518 >>> WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518 >>> WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518 >>> WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518 >>> WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518 >>> WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518 >>> LD [M] net/netfilter/nf_nat_tftp.ko >>> >>> looks like maybe dedup missed this struct for some reason >>> >>> nf_conn dump from module: >>> >>> [120155] PTR '(anon)' type_id=120156 >>> [120156] STRUCT 'nf_conn' size=320 vlen=14 >>> 'ct_general' type_id=105882 bits_offset=0 >>> 'lock' type_id=180 bits_offset=64 >>> 'timeout' type_id=113 bits_offset=640 >>> 'zone' type_id=106520 bits_offset=672 >>> 'tuplehash' type_id=106533 bits_offset=704 >>> 'status' type_id=1 bits_offset=1600 >>> 'ct_net' type_id=3215 bits_offset=1664 >>> 'nat_bysource' type_id=139 bits_offset=1728 >>> '__nfct_init_offset' type_id=949 bits_offset=1856 >>> 'master' type_id=120155 bits_offset=1856 >>> 'mark' type_id=106351 bits_offset=1920 >>> 'secmark' type_id=106351 bits_offset=1952 >>> 'ext' type_id=106536 bits_offset=1984 >>> 'proto' type_id=106532 bits_offset=2048 >>> >>> nf_conn dump from vmlinux: >>> >>> [106517] PTR '(anon)' type_id=106518 >>> [106518] STRUCT 'nf_conn' size=320 vlen=14 >>> 'ct_general' type_id=105882 bits_offset=0 >>> 'lock' type_id=180 bits_offset=64 >>> 'timeout' type_id=113 bits_offset=640 >>> 'zone' type_id=106520 bits_offset=672 >>> 'tuplehash' type_id=106533 bits_offset=704 >>> 'status' type_id=1 bits_offset=1600 >>> 'ct_net' type_id=3215 bits_offset=1664 >>> 'nat_bysource' type_id=139 bits_offset=1728 >>> '__nfct_init_offset' type_id=949 bits_offset=1856 >>> 'master' type_id=106517 bits_offset=1856 >>> 'mark' type_id=106351 bits_offset=1920 >>> 'secmark' type_id=106351 bits_offset=1952 >>> 'ext' type_id=106536 bits_offset=1984 >>> 'proto' type_id=106532 bits_offset=2048 >>> >>> look identical.. Andrii, any idea? >> >> I'm pretty sure they are not identical. There is somewhere a STRUCT vs >> FWD difference. We had a similar discussion recently with Alan >> Maguire. >> >>> 'master' type_id=120155 bits_offset=1856 >> >> vs >> >>> 'master' type_id=106517 bits_offset=1856 > > master is pointer to same 'nf_conn' object, and rest of the ids are same > > jirka > I tried digging into this problem a bit - in my case I was seeing "struct sk_buff" duplicated in kernel/module BTF. Here's what I found.. Consider a situation like this, where one header file defining a struct s1 has a pointer field, pointing at struct s2. But struct s2 is a fwd definition. $ cat s1.h #include <stdio.h> struct s2; struct s1 { struct s1 *f1; struct s2 *f2; }; $ cat s1.c #include "s1.h" int main(int argc, char *argv[]) { struct s1 s1; return 0; } Now consider a separate program s2, that #includes definitions for both s1 and s2: $ cat s2.h #include <stdio.h> struct s1; struct s2 { struct s1 *f1; }; $cat s2.c #include "s2.h" #include "s1.h" int main(int argc, char *argv[]) { struct s1 s1 = {}; struct s2 s2 = {}; return 0; } In this case the generated base BTF contains a definition for s1, and a FWD for s2, but the "module" BTF for s2 contains a full definition for s2, so the dedup fails: $ bpftool btf dump file s1 [29] STRUCT 's1' size=16 vlen=2 'f1' type_id=30 bits_offset=0 'f2' type_id=32 bits_offset=64 [30] PTR '(anon)' type_id=29 [31] FWD 's2' fwd_kind=struct $ bpftool btf dump -B s1 file s2 [36] STRUCT 's2' size=8 vlen=1 'f1' type_id=38 bits_offset=0 [37] STRUCT 's1' size=16 vlen=2 'f1' type_id=38 bits_offset=0 'f2' type_id=39 bits_offset=64 [38] PTR '(anon)' type_id=37 [39] PTR '(anon)' type_id=36 So we had to redefine struct s1 in the "module" because the FWD wasn't resolved in the base BTF. This is by design as I understand it; in effect we can't supplement base BTF with info we've gotten from module BTF about forward resolution (at least that's my understanding of the reason). Now does this sort of thing happen in the kernel? It looks like it; consider struct nf_conn; it contains a possible_net_t: typedef struct { struct net * net; /* 0 8 */ /* size: 8, cachelines: 1, members: 1 */ /* last cacheline: 8 bytes */ } possible_net_t; ...and a struct net * contains pointers to structures that aren't in the vmlinux BTF (because they are in modules); for example: struct netns_ipvs * ipvs; /* 3912 8 */ $ pahole netns_ipvs pahole: type 'netns_ipvs' not found ...and in vmlinux BTF it is: [2983] FWD 'netns_ipvs' fwd_kind=struct [2984] PTR '(anon)' type_id=2983 ...and in struct net we can see the fwd type is referenced alright: [2021] STRUCT 'net' size=4288 vlen=52 ... 'ipvs' type_id=2984 bits_offset=31808 So we'd expect any ipvs-related modules to not dedup struct net, since they'll have the full definition for netns_ipvs. In xt_ipvs.ko we see: [111924] STRUCT 'netns_ipvs' size=2176 vlen=78 'gen' type_id=21 bits_offset=0 'enable' type_id=21 bits_offset=32 'rs_table' type_id=4044 bits_offset=64 'app_list' type_id=83 bits_offset=1088 ...and when we look at 'struct net' we see: [111786] STRUCT 'net' size=4288 vlen=52 ... 'ipvs' type_id=111925 bits_offset=31808 And then if we don't dedup struct net, it seems likely that structures referencing struct net (like skbs, nf_conn etc) won't dedup either since they'll point at "their" version of struct net. Not sure if that's the root cause here, but it seems like it is happening in other modules at least. More subtle effects are also possible I think; if a type is in a header file is defined but not referenced anywhere (as might well happen for a module-related type in vmlinux), it might not always make it into the DWARF description, and as a result of that might not have a BTF representation. Alan ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-14 6:47 ` Jiri Olsa 2022-10-14 17:17 ` Alan Maguire @ 2022-10-21 18:18 ` Andrii Nakryiko 2022-10-21 21:34 ` Jiri Olsa 1 sibling, 1 reply; 22+ messages in thread From: Andrii Nakryiko @ 2022-10-21 18:18 UTC (permalink / raw) To: Jiri Olsa; +Cc: Jakub Kicinski, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Thu, Oct 13, 2022 at 11:47 PM Jiri Olsa <olsajiri@gmail.com> wrote: > > On Thu, Oct 13, 2022 at 03:24:59PM -0700, Andrii Nakryiko wrote: > > On Thu, Oct 13, 2022 at 3:12 PM Jiri Olsa <olsajiri@gmail.com> wrote: > > > > > > On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote: > > > > On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote: > > > > > > Yeah, it's there on linux-next, too. > > > > > > > > > > > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow. > > > > > > > > > > ok, I will look around what's the way to install that centos 8 thing > > > > > > > > Any luck? > > > > > > now BTFIDS warnings.. > > > > > > I can see following on centos8 with gcc 8.5: > > > > > > BTFIDS vmlinux > > > WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300 > > > WARN: multiple IDs found for 'file': 540, 56649 - using 540 > > > WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549 > > > WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953 > > > WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132 > > > WARN: multiple IDs found for 'path': 1164, 56995 - using 1164 > > > WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300 > > > WARN: multiple IDs found for 'file': 540, 61943 - using 540 > > > WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549 > > > WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132 > > > WARN: multiple IDs found for 'path': 1164, 62058 - using 1164 > > > WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190 > > > WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953 > > > WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960 > > > WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876 > > > WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094 > > > WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993 > > > WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191 > > > WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164 > > > WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296 > > > WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292 > > > WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700 > > > WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944 > > > WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299 > > > WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969 > > > WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623 > > > > > > I'll need to check on that.. > > > > > > and I just actually saw the 'nf_conn' warning on linux-next/master with > > > latest fedora/gcc-12: > > > > > > BTF [M] net/netfilter/nf_nat.ko > > > WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518 > > > WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518 > > > WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518 > > > WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518 > > > WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518 > > > WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518 > > > LD [M] net/netfilter/nf_nat_tftp.ko > > > > > > looks like maybe dedup missed this struct for some reason > > > > > > nf_conn dump from module: > > > > > > [120155] PTR '(anon)' type_id=120156 > > > [120156] STRUCT 'nf_conn' size=320 vlen=14 > > > 'ct_general' type_id=105882 bits_offset=0 > > > 'lock' type_id=180 bits_offset=64 > > > 'timeout' type_id=113 bits_offset=640 > > > 'zone' type_id=106520 bits_offset=672 > > > 'tuplehash' type_id=106533 bits_offset=704 > > > 'status' type_id=1 bits_offset=1600 > > > 'ct_net' type_id=3215 bits_offset=1664 > > > 'nat_bysource' type_id=139 bits_offset=1728 > > > '__nfct_init_offset' type_id=949 bits_offset=1856 > > > 'master' type_id=120155 bits_offset=1856 > > > 'mark' type_id=106351 bits_offset=1920 > > > 'secmark' type_id=106351 bits_offset=1952 > > > 'ext' type_id=106536 bits_offset=1984 > > > 'proto' type_id=106532 bits_offset=2048 > > > > > > nf_conn dump from vmlinux: > > > > > > [106517] PTR '(anon)' type_id=106518 > > > [106518] STRUCT 'nf_conn' size=320 vlen=14 > > > 'ct_general' type_id=105882 bits_offset=0 > > > 'lock' type_id=180 bits_offset=64 > > > 'timeout' type_id=113 bits_offset=640 > > > 'zone' type_id=106520 bits_offset=672 > > > 'tuplehash' type_id=106533 bits_offset=704 > > > 'status' type_id=1 bits_offset=1600 > > > 'ct_net' type_id=3215 bits_offset=1664 > > > 'nat_bysource' type_id=139 bits_offset=1728 > > > '__nfct_init_offset' type_id=949 bits_offset=1856 > > > 'master' type_id=106517 bits_offset=1856 > > > 'mark' type_id=106351 bits_offset=1920 > > > 'secmark' type_id=106351 bits_offset=1952 > > > 'ext' type_id=106536 bits_offset=1984 > > > 'proto' type_id=106532 bits_offset=2048 > > > > > > look identical.. Andrii, any idea? > > > > I'm pretty sure they are not identical. There is somewhere a STRUCT vs > > FWD difference. We had a similar discussion recently with Alan > > Maguire. > > > > > 'master' type_id=120155 bits_offset=1856 > > > > vs > > > > > 'master' type_id=106517 bits_offset=1856 > > master is pointer to same 'nf_conn' object, and rest of the ids are same > You are right, they should be identical once PTR is deduplicated properly. Sorry, was too quick to jump to conclusions. I was thinking about situations explained by Alan. So, is this still an issue or this was fixed by [0]? [0] https://lore.kernel.org/bpf/1666364523-9648-1-git-send-email-alan.maguire@oracle.com/ > jirka ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-21 18:18 ` Andrii Nakryiko @ 2022-10-21 21:34 ` Jiri Olsa 2022-10-22 5:36 ` Jakub Kicinski 0 siblings, 1 reply; 22+ messages in thread From: Jiri Olsa @ 2022-10-21 21:34 UTC (permalink / raw) To: Andrii Nakryiko Cc: Jiri Olsa, Jakub Kicinski, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Fri, Oct 21, 2022 at 11:18:39AM -0700, Andrii Nakryiko wrote: > On Thu, Oct 13, 2022 at 11:47 PM Jiri Olsa <olsajiri@gmail.com> wrote: > > > > On Thu, Oct 13, 2022 at 03:24:59PM -0700, Andrii Nakryiko wrote: > > > On Thu, Oct 13, 2022 at 3:12 PM Jiri Olsa <olsajiri@gmail.com> wrote: > > > > > > > > On Thu, Oct 13, 2022 at 08:05:17AM -0700, Jakub Kicinski wrote: > > > > > On Wed, 5 Oct 2022 22:07:57 +0200 Jiri Olsa wrote: > > > > > > > Yeah, it's there on linux-next, too. > > > > > > > > > > > > > > Let me grab a fresh VM and try there. Maybe it's my system. Somehow. > > > > > > > > > > > > ok, I will look around what's the way to install that centos 8 thing > > > > > > > > > > Any luck? > > > > > > > > now BTFIDS warnings.. > > > > > > > > I can see following on centos8 with gcc 8.5: > > > > > > > > BTFIDS vmlinux > > > > WARN: multiple IDs found for 'task_struct': 300, 56614 - using 300 > > > > WARN: multiple IDs found for 'file': 540, 56649 - using 540 > > > > WARN: multiple IDs found for 'vm_area_struct': 549, 56652 - using 549 > > > > WARN: multiple IDs found for 'seq_file': 953, 56690 - using 953 > > > > WARN: multiple IDs found for 'inode': 1132, 56966 - using 1132 > > > > WARN: multiple IDs found for 'path': 1164, 56995 - using 1164 > > > > WARN: multiple IDs found for 'task_struct': 300, 61905 - using 300 > > > > WARN: multiple IDs found for 'file': 540, 61943 - using 540 > > > > WARN: multiple IDs found for 'vm_area_struct': 549, 61946 - using 549 > > > > WARN: multiple IDs found for 'inode': 1132, 62029 - using 1132 > > > > WARN: multiple IDs found for 'path': 1164, 62058 - using 1164 > > > > WARN: multiple IDs found for 'cgroup': 1190, 62067 - using 1190 > > > > WARN: multiple IDs found for 'seq_file': 953, 62253 - using 953 > > > > WARN: multiple IDs found for 'sock': 7960, 62374 - using 7960 > > > > WARN: multiple IDs found for 'sk_buff': 1876, 62485 - using 1876 > > > > WARN: multiple IDs found for 'bpf_prog': 6094, 62542 - using 6094 > > > > WARN: multiple IDs found for 'socket': 7993, 62545 - using 7993 > > > > WARN: multiple IDs found for 'xdp_buff': 6191, 62836 - using 6191 > > > > WARN: multiple IDs found for 'sock_common': 8164, 63152 - using 8164 > > > > WARN: multiple IDs found for 'request_sock': 17296, 63204 - using 17296 > > > > WARN: multiple IDs found for 'inet_request_sock': 36292, 63222 - using 36292 > > > > WARN: multiple IDs found for 'inet_sock': 32700, 63225 - using 32700 > > > > WARN: multiple IDs found for 'inet_connection_sock': 33944, 63240 - using 33944 > > > > WARN: multiple IDs found for 'tcp_request_sock': 36299, 63260 - using 36299 > > > > WARN: multiple IDs found for 'tcp_sock': 33969, 63264 - using 33969 > > > > WARN: multiple IDs found for 'bpf_map': 6623, 63343 - using 6623 > > > > > > > > I'll need to check on that.. > > > > > > > > and I just actually saw the 'nf_conn' warning on linux-next/master with > > > > latest fedora/gcc-12: > > > > > > > > BTF [M] net/netfilter/nf_nat.ko > > > > WARN: multiple IDs found for 'nf_conn': 106518, 120156 - using 106518 > > > > WARN: multiple IDs found for 'nf_conn': 106518, 121853 - using 106518 > > > > WARN: multiple IDs found for 'nf_conn': 106518, 123126 - using 106518 > > > > WARN: multiple IDs found for 'nf_conn': 106518, 124537 - using 106518 > > > > WARN: multiple IDs found for 'nf_conn': 106518, 126442 - using 106518 > > > > WARN: multiple IDs found for 'nf_conn': 106518, 128256 - using 106518 > > > > LD [M] net/netfilter/nf_nat_tftp.ko > > > > > > > > looks like maybe dedup missed this struct for some reason > > > > > > > > nf_conn dump from module: > > > > > > > > [120155] PTR '(anon)' type_id=120156 > > > > [120156] STRUCT 'nf_conn' size=320 vlen=14 > > > > 'ct_general' type_id=105882 bits_offset=0 > > > > 'lock' type_id=180 bits_offset=64 > > > > 'timeout' type_id=113 bits_offset=640 > > > > 'zone' type_id=106520 bits_offset=672 > > > > 'tuplehash' type_id=106533 bits_offset=704 > > > > 'status' type_id=1 bits_offset=1600 > > > > 'ct_net' type_id=3215 bits_offset=1664 > > > > 'nat_bysource' type_id=139 bits_offset=1728 > > > > '__nfct_init_offset' type_id=949 bits_offset=1856 > > > > 'master' type_id=120155 bits_offset=1856 > > > > 'mark' type_id=106351 bits_offset=1920 > > > > 'secmark' type_id=106351 bits_offset=1952 > > > > 'ext' type_id=106536 bits_offset=1984 > > > > 'proto' type_id=106532 bits_offset=2048 > > > > > > > > nf_conn dump from vmlinux: > > > > > > > > [106517] PTR '(anon)' type_id=106518 > > > > [106518] STRUCT 'nf_conn' size=320 vlen=14 > > > > 'ct_general' type_id=105882 bits_offset=0 > > > > 'lock' type_id=180 bits_offset=64 > > > > 'timeout' type_id=113 bits_offset=640 > > > > 'zone' type_id=106520 bits_offset=672 > > > > 'tuplehash' type_id=106533 bits_offset=704 > > > > 'status' type_id=1 bits_offset=1600 > > > > 'ct_net' type_id=3215 bits_offset=1664 > > > > 'nat_bysource' type_id=139 bits_offset=1728 > > > > '__nfct_init_offset' type_id=949 bits_offset=1856 > > > > 'master' type_id=106517 bits_offset=1856 > > > > 'mark' type_id=106351 bits_offset=1920 > > > > 'secmark' type_id=106351 bits_offset=1952 > > > > 'ext' type_id=106536 bits_offset=1984 > > > > 'proto' type_id=106532 bits_offset=2048 > > > > > > > > look identical.. Andrii, any idea? > > > > > > I'm pretty sure they are not identical. There is somewhere a STRUCT vs > > > FWD difference. We had a similar discussion recently with Alan > > > Maguire. > > > > > > > 'master' type_id=120155 bits_offset=1856 > > > > > > vs > > > > > > > 'master' type_id=106517 bits_offset=1856 > > > > master is pointer to same 'nf_conn' object, and rest of the ids are same > > > > You are right, they should be identical once PTR is deduplicated > properly. Sorry, was too quick to jump to conclusions. I was thinking > about situations explained by Alan. > > So, is this still an issue or this was fixed by [0]? > > [0] https://lore.kernel.org/bpf/1666364523-9648-1-git-send-email-alan.maguire@oracle.com/ yes, it seems to be fixed by that Jakub, could you check with pahole fix [1]? thanks, jirka ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-21 21:34 ` Jiri Olsa @ 2022-10-22 5:36 ` Jakub Kicinski 2022-10-23 1:18 ` Alexei Starovoitov 0 siblings, 1 reply; 22+ messages in thread From: Jakub Kicinski @ 2022-10-22 5:36 UTC (permalink / raw) To: Jiri Olsa; +Cc: Andrii Nakryiko, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Fri, 21 Oct 2022 23:34:13 +0200 Jiri Olsa wrote: > > You are right, they should be identical once PTR is deduplicated > > properly. Sorry, was too quick to jump to conclusions. I was thinking > > about situations explained by Alan. > > > > So, is this still an issue or this was fixed by [0]? > > > > [0] https://lore.kernel.org/bpf/1666364523-9648-1-git-send-email-alan.maguire@oracle.com/ > > yes, it seems to be fixed by that > > Jakub, > could you check with pahole fix [1]? If you mean the warning from the subject then those do seem to be gone. But if I'm completely honest I don't remember how I triggered them in the first place :S There weren't there on every build for me. The objtool warning is still here: $ make PAHOLE=~/pahole O=build_allmodconfig/ -j 60 >/tmp/stdout 2>/tmp/stderr; \ cat /tmp/stderr vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-22 5:36 ` Jakub Kicinski @ 2022-10-23 1:18 ` Alexei Starovoitov 2022-10-24 18:21 ` Jakub Kicinski 0 siblings, 1 reply; 22+ messages in thread From: Alexei Starovoitov @ 2022-10-23 1:18 UTC (permalink / raw) To: Jakub Kicinski Cc: Jiri Olsa, Andrii Nakryiko, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Fri, Oct 21, 2022 at 10:38 PM Jakub Kicinski <kuba@kernel.org> wrote: > > On Fri, 21 Oct 2022 23:34:13 +0200 Jiri Olsa wrote: > > > You are right, they should be identical once PTR is deduplicated > > > properly. Sorry, was too quick to jump to conclusions. I was thinking > > > about situations explained by Alan. > > > > > > So, is this still an issue or this was fixed by [0]? > > > > > > [0] https://lore.kernel.org/bpf/1666364523-9648-1-git-send-email-alan.maguire@oracle.com/ > > > > yes, it seems to be fixed by that > > > > Jakub, > > could you check with pahole fix [1]? > > If you mean the warning from the subject then those do seem to be gone. > But if I'm completely honest I don't remember how I triggered them in > the first place :S There weren't there on every build for me. > > The objtool warning is still here: > > $ make PAHOLE=~/pahole O=build_allmodconfig/ -j 60 >/tmp/stdout 2>/tmp/stderr; \ > cat /tmp/stderr > > vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 > vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 The effect of the compiler bug was addressed by this fix: https://lore.kernel.org/all/20221018075934.574415-1-jolsa@kernel.org/ It's in the bpf tree, but the warning will stay. While the compiler is broken the objtool should keep complaining. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-23 1:18 ` Alexei Starovoitov @ 2022-10-24 18:21 ` Jakub Kicinski 0 siblings, 0 replies; 22+ messages in thread From: Jakub Kicinski @ 2022-10-24 18:21 UTC (permalink / raw) To: Alexei Starovoitov Cc: Jiri Olsa, Andrii Nakryiko, Martin KaFai Lau, bpf, netdev@vger.kernel.org On Sat, 22 Oct 2022 18:18:49 -0700 Alexei Starovoitov wrote: > > If you mean the warning from the subject then those do seem to be gone. > > But if I'm completely honest I don't remember how I triggered them in > > the first place :S There weren't there on every build for me. > > > > The objtool warning is still here: > > > > $ make PAHOLE=~/pahole O=build_allmodconfig/ -j 60 >/tmp/stdout 2>/tmp/stderr; \ > > cat /tmp/stderr > > > > vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 > > vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 > > The effect of the compiler bug was addressed by this fix: > https://lore.kernel.org/all/20221018075934.574415-1-jolsa@kernel.org/ > > It's in the bpf tree, but the warning will stay. > While the compiler is broken the objtool should keep complaining. Thanks! I'll stop tracking it ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-04 4:49 ` WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 Jakub Kicinski 2022-10-04 6:42 ` Jiri Olsa @ 2022-10-04 14:46 ` Daniel Xu 2022-10-04 15:34 ` Jakub Kicinski 2022-10-18 15:11 ` Christian Borntraeger 1 sibling, 2 replies; 22+ messages in thread From: Daniel Xu @ 2022-10-04 14:46 UTC (permalink / raw) To: Jakub Kicinski; +Cc: Martin KaFai Lau, Jiri Olsa, bpf, netdev@vger.kernel.org Hi Jakub, On Mon, Oct 03, 2022 at 09:49:41PM -0700, Jakub Kicinski wrote: > On Mon, 3 Oct 2022 19:05:45 -0700 Jakub Kicinski wrote: > > Hi Jiri, > > > > I get the following warning after merging up all the trees: > > > > vmlinux.o: warning: objtool: ___ksymtab+bpf_dispatcher_xdp_func+0x0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 > > vmlinux.o: warning: objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: bpf_dispatcher_xdp_func+0x0 > > > > $ gcc --version > > gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15) > > > > > > Is this known? > > Also hit this: > > WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 > WARN: multiple IDs found for 'nf_conn': 92168, 121226 - using 92168 I believe this is now-fixed pahole bug. See: https://lore.kernel.org/bpf/20220907023559.22juhtyl3qh2gsym@kashmir.localdomain/ That being said, I didn't manage to find a pahole commit that directly addresses the issue, so maybe upgrading pahole perturbed enough things for this warning to go away? If the warning is back on pahole master I can take another look. Thanks, Daniel ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-04 14:46 ` Daniel Xu @ 2022-10-04 15:34 ` Jakub Kicinski 2022-10-18 15:11 ` Christian Borntraeger 1 sibling, 0 replies; 22+ messages in thread From: Jakub Kicinski @ 2022-10-04 15:34 UTC (permalink / raw) To: Daniel Xu; +Cc: Martin KaFai Lau, Jiri Olsa, bpf, netdev@vger.kernel.org On Tue, 4 Oct 2022 08:46:03 -0600 Daniel Xu wrote: > > WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 > > WARN: multiple IDs found for 'nf_conn': 92168, 121226 - using 92168 > > I believe this is now-fixed pahole bug. See: > https://lore.kernel.org/bpf/20220907023559.22juhtyl3qh2gsym@kashmir.localdomain/ > > That being said, I didn't manage to find a pahole commit that directly > addresses the issue, so maybe upgrading pahole perturbed enough things > for this warning to go away? > > If the warning is back on pahole master I can take another look. From-the-source pahole fixes it, thanks! ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 2022-10-04 14:46 ` Daniel Xu 2022-10-04 15:34 ` Jakub Kicinski @ 2022-10-18 15:11 ` Christian Borntraeger 1 sibling, 0 replies; 22+ messages in thread From: Christian Borntraeger @ 2022-10-18 15:11 UTC (permalink / raw) To: dxu; +Cc: bpf, jolsa, kuba, martin.lau, netdev, linux-kernel chiming in. I also see that on s390x with 6.1-rc. latest greates pahole does seem to fix this for me on s390 with gcc, but with clang I still see this. Christian ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2022-10-24 19:59 UTC | newest] Thread overview: 22+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-10-04 2:05 objtool: bpf_dispatcher_xdp+0xa0: data relocation to !ENDBR: Jakub Kicinski 2022-10-04 4:49 ` WARN: multiple IDs found for 'nf_conn': 92168, 117897 - using 92168 Jakub Kicinski 2022-10-04 6:42 ` Jiri Olsa 2022-10-04 14:25 ` Jakub Kicinski 2022-10-05 9:45 ` Jiri Olsa 2022-10-05 15:44 ` Jakub Kicinski 2022-10-05 16:18 ` Jakub Kicinski 2022-10-05 20:07 ` Jiri Olsa 2022-10-13 15:05 ` Jakub Kicinski 2022-10-13 21:55 ` Jiri Olsa 2022-10-13 22:12 ` Jiri Olsa 2022-10-13 22:24 ` Andrii Nakryiko 2022-10-14 6:47 ` Jiri Olsa 2022-10-14 17:17 ` Alan Maguire 2022-10-21 18:18 ` Andrii Nakryiko 2022-10-21 21:34 ` Jiri Olsa 2022-10-22 5:36 ` Jakub Kicinski 2022-10-23 1:18 ` Alexei Starovoitov 2022-10-24 18:21 ` Jakub Kicinski 2022-10-04 14:46 ` Daniel Xu 2022-10-04 15:34 ` Jakub Kicinski 2022-10-18 15:11 ` Christian Borntraeger
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).