* perf test for inet_pton probe fails to match stacktrace
@ 2024-11-22 21:35 Guilherme Amadio
2024-11-24 10:50 ` Leo Yan
0 siblings, 1 reply; 8+ messages in thread
From: Guilherme Amadio @ 2024-11-22 21:35 UTC (permalink / raw)
To: linux-perf-users; +Cc: acme, namhyung
Hi Arnaldo, Namhyung,
I've had this test fail for me since a while, so I'm reporting below
in case we can find a good solution to fix it on my system without
breaking other OSs. The test that fails is this:
gentoo ~ # perf test -vF inet_pton
--- start ---
ping 303105 [029] 47754.974977: probe_libc:inet_pton: (7f70ca48feb7)
7f70ca48feb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)"
---- end ----
92: probe libc's inet_pton & backtrace it with ping : FAILED!
gentoo ~ # perf version
perf version 6.12
Here's why it fails. My stack traces has more than the test expects:
gentoo ~ # perf script | tac | sed -e /^ping/q | tac
ping 267392 [010] 42482.886216: probe_libc:inet_pton: (7f168d9b4eb7)
7f168d9b4eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f168d9c7517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
7f168d9c8b73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
55947171a518 [unknown] (/usr/bin/ping)
7f168d8d6a72 __libc_start_call_main+0x7a (/usr/lib64/libc.so.6)
7f168d8d6b29 __libc_start_main@@GLIBC_2.34+0x84 (/usr/lib64/libc.so.6)
55947171ac55 [unknown] (/usr/bin/ping)
Likely this is due to how I compile glibc:
gentoo ~ # cat /var/db/pkg/sys-libs/glibc-2.40-r5/CFLAGS
-Og -march=native -Wl,--build-id -gdwarf-4 -fno-builtin-strlen -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer
If you could suggest a way to fix this, I'd appreciate it.
Best regards,
-Guilherme
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: perf test for inet_pton probe fails to match stacktrace 2024-11-22 21:35 perf test for inet_pton probe fails to match stacktrace Guilherme Amadio @ 2024-11-24 10:50 ` Leo Yan 2024-11-25 13:12 ` Guilherme Amadio 0 siblings, 1 reply; 8+ messages in thread From: Leo Yan @ 2024-11-24 10:50 UTC (permalink / raw) To: Guilherme Amadio; +Cc: linux-perf-users, acme, namhyung Hi Guilherme, On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote: [...] > Hi Arnaldo, Namhyung, > > I've had this test fail for me since a while, so I'm reporting below > in case we can find a good solution to fix it on my system without > breaking other OSs. The test that fails is this: > > gentoo ~ # perf test -vF inet_pton > --- start --- > ping 303105 [029] 47754.974977: probe_libc:inet_pton: (7f70ca48feb7) > 7f70ca48feb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) > 7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6) > FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)" > ---- end ---- > 92: probe libc's inet_pton & backtrace it with ping : FAILED! > gentoo ~ # perf version > perf version 6.12 > > Here's why it fails. My stack traces has more than the test expects: > > gentoo ~ # perf script | tac | sed -e /^ping/q | tac > ping 267392 [010] 42482.886216: probe_libc:inet_pton: (7f168d9b4eb7) > 7f168d9b4eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) > 7f168d9c7517 gaih_inet+0x112 (/usr/lib64/libc.so.6) > 7f168d9c8b73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) > 55947171a518 [unknown] (/usr/bin/ping) > 7f168d8d6a72 __libc_start_call_main+0x7a (/usr/lib64/libc.so.6) > 7f168d8d6b29 __libc_start_main@@GLIBC_2.34+0x84 (/usr/lib64/libc.so.6) > 55947171ac55 [unknown] (/usr/bin/ping) > > Likely this is due to how I compile glibc: > gentoo ~ # cat /var/db/pkg/sys-libs/glibc-2.40-r5/CFLAGS > -Og -march=native -Wl,--build-id -gdwarf-4 -fno-builtin-strlen -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer This is not the first reported failure on this case, see [1] for previous one. I assume you are working on x86_64 machine. Could you confirm if the change below can fix your issue? If yes, I will send a formal patch. Thanks, Leo ---8<-- diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh index 47a26f25db9f..5a2d641bf166 100755 --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh @@ -54,6 +54,15 @@ trace_libc_inet_pton_backtrace() { ;; *) eventattr='max-stack=3' + # Add gaih_inet to expected backtrace only if it is part of libc. + if nm -D $libc | grep -F -q gaih_inet; then + echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi + # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17 + # which is not used. Do strict checking for 'getaddrinfo' without any suffix. + if nm -D $libc | grep -q 'getaddrinfo$'; then + echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; esac [1] https://lore.kernel.org/linux-perf-users/20241025132021.GB359792@e132581.arm.com/ > If you could suggest a way to fix this, I'd appreciate it. > > Best regards, > -Guilherme > > ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: perf test for inet_pton probe fails to match stacktrace 2024-11-24 10:50 ` Leo Yan @ 2024-11-25 13:12 ` Guilherme Amadio 2024-11-26 5:08 ` Namhyung Kim 0 siblings, 1 reply; 8+ messages in thread From: Guilherme Amadio @ 2024-11-25 13:12 UTC (permalink / raw) To: Leo Yan; +Cc: linux-perf-users, acme, namhyung Dear Leo, On Sun, Nov 24, 2024 at 10:50:53AM +0000, Leo Yan wrote: > Hi Guilherme, > > On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote: > > [...] > > [...] > > This is not the first reported failure on this case, see [1] for > previous one. > > I assume you are working on x86_64 machine. Could you confirm if the > change below can fix your issue? If yes, I will send a formal patch. Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed. The patch you sent didn't work out of the box, but with minor modifications I could fix it. I just needed to replace nm -D $libc with nm $libc for it work. The reason is that gaih_inet doesn't show with nm -D for my glibc: $ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo 0000000000116a40 T getaddrinfo@@GLIBC_2.2.5 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 $ nm /usr/lib64/libc.so.6 | grep getaddrinfo 0000000000116a40 t __GI_getaddrinfo 000000000010c140 t __getaddrinfo_a 0000000000115e53 t __libc_getaddrinfo_freemem 0000000000116a40 T getaddrinfo 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5 $ nm -D /usr/lib64/libc.so.6 | grep gaih_inet $ nm /usr/lib64/libc.so.6 | grep gaih_inet 0000000000115405 t gaih_inet 00000000001139b2 t gaih_inet_serv 0000000000177780 r gaih_inet_typeproto Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass: $ sudo ./perf test -vF pton --- start --- ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6) 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) 55680ac46518 [unknown] (/usr/bin/ping) ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : Ok The final patch I had to apply is below: perf $ git diff diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh index 47a26f25db9f..18274e1c315c 100755 --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh @@ -53,7 +53,16 @@ trace_libc_inet_pton_backtrace() { echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; *) - eventattr='max-stack=3' + eventattr='max-stack=4' + # Add gaih_inet to expected backtrace only if it is part of libc. + if nm $libc | grep -F -q gaih_inet; then + echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi + # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17 + # which is not used. Do strict checking for 'getaddrinfo' without any suffix. + if nm $libc | grep -q 'getaddrinfo$'; then + echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected + fi echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; esac Please feel free to add Tested-by: and/or Reviewed-by: Guilherme Amadio <amadio@gentoo.org> to the final patch as you see fit. Thank you! Best regards, -Guilherme > > Thanks, > Leo > > ---8<-- > > diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh > index 47a26f25db9f..5a2d641bf166 100755 > --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh > +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh > @@ -54,6 +54,15 @@ trace_libc_inet_pton_backtrace() { > ;; > *) > eventattr='max-stack=3' > + # Add gaih_inet to expected backtrace only if it is part of libc. > + if nm -D $libc | grep -F -q gaih_inet; then > + echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected > + fi > + # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17 > + # which is not used. Do strict checking for 'getaddrinfo' without any suffix. > + if nm -D $libc | grep -q 'getaddrinfo$'; then > + echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected > + fi > echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected > ;; > esac > > [1] https://lore.kernel.org/linux-perf-users/20241025132021.GB359792@e132581.arm.com/ > > > If you could suggest a way to fix this, I'd appreciate it. > > > > Best regards, > > -Guilherme > > > > ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: perf test for inet_pton probe fails to match stacktrace 2024-11-25 13:12 ` Guilherme Amadio @ 2024-11-26 5:08 ` Namhyung Kim 2024-11-26 10:27 ` Leo Yan 2024-11-26 16:27 ` Arnaldo Carvalho de Melo 0 siblings, 2 replies; 8+ messages in thread From: Namhyung Kim @ 2024-11-26 5:08 UTC (permalink / raw) To: Guilherme Amadio, acme; +Cc: Leo Yan, linux-perf-users Hello guys, On Mon, Nov 25, 2024 at 02:12:41PM +0100, Guilherme Amadio wrote: > Dear Leo, > > On Sun, Nov 24, 2024 at 10:50:53AM +0000, Leo Yan wrote: > > Hi Guilherme, > > > > On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote: > > > > [...] > > > [...] > > > > This is not the first reported failure on this case, see [1] for > > previous one. Right, and I'm also seeing the failure now. > > > > I assume you are working on x86_64 machine. Could you confirm if the > > change below can fix your issue? If yes, I will send a formal patch. > > Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed. > The patch you sent didn't work out of the box, but with minor modifications > I could fix it. I just needed to replace nm -D $libc with nm $libc for it work. > The reason is that gaih_inet doesn't show with nm -D for my glibc: > > $ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo > 0000000000116a40 T getaddrinfo@@GLIBC_2.2.5 > 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5 > 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 > $ nm /usr/lib64/libc.so.6 | grep getaddrinfo > 0000000000116a40 t __GI_getaddrinfo > 000000000010c140 t __getaddrinfo_a > 0000000000115e53 t __libc_getaddrinfo_freemem > 0000000000116a40 T getaddrinfo > 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 > 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5 > > $ nm -D /usr/lib64/libc.so.6 | grep gaih_inet > $ nm /usr/lib64/libc.so.6 | grep gaih_inet > 0000000000115405 t gaih_inet > 00000000001139b2 t gaih_inet_serv > 0000000000177780 r gaih_inet_typeproto > > Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass: > > $ sudo ./perf test -vF pton > --- start --- > ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) > 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) > 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6) > 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) > 55680ac46518 [unknown] (/usr/bin/ping) > ---- end ---- > 86: probe libc's inet_pton & backtrace it with ping : Ok I think it's very fragile as it depends on specific entries in the call stack. As we want to test if perf probe (for uprobes) and perf record working correctly, maybe it can just check the number of samples only. Arnaldo, what do you think? Thanks, Namhyung > > > The final patch I had to apply is below: > > perf $ git diff > diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh > index 47a26f25db9f..18274e1c315c 100755 > --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh > +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh > @@ -53,7 +53,16 @@ trace_libc_inet_pton_backtrace() { > echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected > ;; > *) > - eventattr='max-stack=3' > + eventattr='max-stack=4' > + # Add gaih_inet to expected backtrace only if it is part of libc. > + if nm $libc | grep -F -q gaih_inet; then > + echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected > + fi > + # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17 > + # which is not used. Do strict checking for 'getaddrinfo' without any suffix. > + if nm $libc | grep -q 'getaddrinfo$'; then > + echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected > + fi > echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected > ;; > esac > > Please feel free to add Tested-by: and/or Reviewed-by: Guilherme Amadio <amadio@gentoo.org> > to the final patch as you see fit. Thank you! Thanks for the report and the fix. Namhyung ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: perf test for inet_pton probe fails to match stacktrace 2024-11-26 5:08 ` Namhyung Kim @ 2024-11-26 10:27 ` Leo Yan 2024-11-26 16:39 ` Arnaldo Carvalho de Melo 2024-11-26 16:27 ` Arnaldo Carvalho de Melo 1 sibling, 1 reply; 8+ messages in thread From: Leo Yan @ 2024-11-26 10:27 UTC (permalink / raw) To: Namhyung Kim; +Cc: Guilherme Amadio, acme, linux-perf-users On Mon, Nov 25, 2024 at 09:08:15PM -0800, Namhyung Kim wrote: [...] > > Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass: > > > > $ sudo ./perf test -vF pton > > --- start --- > > ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) > > 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) > > 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6) > > 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) > > 55680ac46518 [unknown] (/usr/bin/ping) > > ---- end ---- > > 86: probe libc's inet_pton & backtrace it with ping : Ok > > I think it's very fragile as it depends on specific entries in the call > stack. As we want to test if perf probe (for uprobes) and perf record > working correctly, maybe it can just check the number of samples only. > > Arnaldo, what do you think? Though this question is asking Arnaldo, I have a different idea for fixing the test ;) The middle entries in the callchain are varied on different archs and distro. E.g. Guilherme reported back the '-D' option is not needed but at my side it is required on Arm64 machine with Ubuntu distro. However, the three entries in the callchain are existed consistently: ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) ... 55680ac46518 [unknown] (/usr/bin/ping) We can change the code to only matching these three entries. The change is as below. We can consolidate the code for all archs (I am not 100% sure for x390x so I leave it). Please let me know if this makes sense for you? Thanks, Leo ---8<--- diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh index 47a26f25db9f..6ef58aedf2d8 100755 --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh @@ -43,17 +43,8 @@ trace_libc_inet_pton_backtrace() { echo "((__GI_)?getaddrinfo|text_to_binary_address)\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected echo "(gaih_inet|main)\+0x[[:xdigit:]]+[[:space:]]\(inlined|.*/bin/ping.*\)$" >> $expected ;; - ppc64|ppc64le) - eventattr='max-stack=4' - # Add gaih_inet to expected backtrace only if it is part of libc. - if nm $libc | grep -F -q gaih_inet.; then - echo "gaih_inet.*\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected - fi - echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected - echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected - ;; *) - eventattr='max-stack=3' + eventattr='max-stack=4' echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected ;; esac @@ -76,14 +67,23 @@ trace_libc_inet_pton_backtrace() { fi perf script -i $perf_data | tac | grep -m1 ^ping -B9 | tac > $perf_script - exec 3<$perf_script exec 4<$expected - while read line <&3 && read -r pattern <&4; do + while read -r pattern <&4; do + echo "pattern: $pattern" [ -z "$pattern" ] && break - echo $line - echo "$line" | grep -E -q "$pattern" - if [ $? -ne 0 ] ; then - printf "FAIL: expected backtrace entry \"%s\" got \"%s\"\n" "$pattern" "$line" + + exec 3<$perf_script + while read line <&3; do + [ -z "$line" ] && break + echo "line: $line" + echo "$line" | grep -E -q "$pattern" + found=$? + echo "found=$found" + [ $found -eq 0 ] && break + done + + if [ $found -ne 0 ] ; then + printf "FAIL: unfound the expected backtrace entry \"%s\"\n" "$pattern" return 1 fi done ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: perf test for inet_pton probe fails to match stacktrace 2024-11-26 10:27 ` Leo Yan @ 2024-11-26 16:39 ` Arnaldo Carvalho de Melo 2024-11-26 18:27 ` Leo Yan 0 siblings, 1 reply; 8+ messages in thread From: Arnaldo Carvalho de Melo @ 2024-11-26 16:39 UTC (permalink / raw) To: Leo Yan; +Cc: Namhyung Kim, Guilherme Amadio, linux-perf-users On Tue, Nov 26, 2024 at 10:27:05AM +0000, Leo Yan wrote: > On Mon, Nov 25, 2024 at 09:08:15PM -0800, Namhyung Kim wrote: > > [...] > > > > Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass: > > > > > > $ sudo ./perf test -vF pton > > > --- start --- > > > ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) > > > 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) > > > 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6) > > > 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) > > > 55680ac46518 [unknown] (/usr/bin/ping) > > > ---- end ---- > > > 86: probe libc's inet_pton & backtrace it with ping : Ok > > > > I think it's very fragile as it depends on specific entries in the call > > stack. As we want to test if perf probe (for uprobes) and perf record > > working correctly, maybe it can just check the number of samples only. > > > > Arnaldo, what do you think? > > Though this question is asking Arnaldo, I have a different idea for > fixing the test ;) > > The middle entries in the callchain are varied on different archs and > distro. E.g. Guilherme reported back the '-D' option is not needed but > at my side it is required on Arm64 machine with Ubuntu distro. > > However, the three entries in the callchain are existed consistently: > > ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) > 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) > ... > 55680ac46518 [unknown] (/usr/bin/ping) > > We can change the code to only matching these three entries. The > change is as below. We can consolidate the code for all archs > (I am not 100% sure for x390x so I leave it). > > Please let me know if this makes sense for you? > > Thanks, > Leo Seems to work: Before, on a: root@number:~# grep -m1 'model name' /proc/cpuinfo model name : Intel(R) Core(TM) i7-14700K root@number:~# root@number:~# perf test -vF pton --- start --- ping 229025 [017] 34027.968348: probe_libc:inet_pton: (7fc4542c10d0) 7fc4542c10d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) 7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6) FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)" ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : FAILED! root@number:~# With your patch: root@number:~# perf test -vF pton --- start --- pattern: ping[][0-9 \.:]+probe_libc:inet_pton: \([[:xdigit:]]+\) line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) found=0 pattern: .*inet_pton\+0x[[:xdigit:]]+[[:space:]]\(/usr/lib64/libc.so.6|inlined\)$ line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) found=1 line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) found=0 pattern: .*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$ line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) found=1 line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) found=1 line: 7f6352993a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6) found=1 line: 55ae71919969 [unknown] (/usr/bin/ping) found=0 ---- end ---- 86: probe libc's inet_pton & backtrace it with ping : Ok root@number:~# Replace that "unfound" with "Didn't find the" And using 'found' as a variable name that when zero means "Ok, found" and when 1 (grep failed to find the regexp) means "Didn't find" looks confusing. General idea seems to make the test resilient to the differences while still testing the backtrace in uprobes events for expected entries. - Arnaldo > ---8<--- > > diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh > index 47a26f25db9f..6ef58aedf2d8 100755 > --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh > +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh > @@ -43,17 +43,8 @@ trace_libc_inet_pton_backtrace() { > echo "((__GI_)?getaddrinfo|text_to_binary_address)\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected > echo "(gaih_inet|main)\+0x[[:xdigit:]]+[[:space:]]\(inlined|.*/bin/ping.*\)$" >> $expected > ;; > - ppc64|ppc64le) > - eventattr='max-stack=4' > - # Add gaih_inet to expected backtrace only if it is part of libc. > - if nm $libc | grep -F -q gaih_inet.; then > - echo "gaih_inet.*\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected > - fi > - echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected > - echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected > - ;; > *) > - eventattr='max-stack=3' > + eventattr='max-stack=4' > echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected > ;; > esac > @@ -76,14 +67,23 @@ trace_libc_inet_pton_backtrace() { > fi > perf script -i $perf_data | tac | grep -m1 ^ping -B9 | tac > $perf_script > > - exec 3<$perf_script > exec 4<$expected > - while read line <&3 && read -r pattern <&4; do > + while read -r pattern <&4; do > + echo "pattern: $pattern" > [ -z "$pattern" ] && break > - echo $line > - echo "$line" | grep -E -q "$pattern" > - if [ $? -ne 0 ] ; then > - printf "FAIL: expected backtrace entry \"%s\" got \"%s\"\n" "$pattern" "$line" > + > + exec 3<$perf_script > + while read line <&3; do > + [ -z "$line" ] && break > + echo "line: $line" > + echo "$line" | grep -E -q "$pattern" > + found=$? > + echo "found=$found" > + [ $found -eq 0 ] && break > + done > + > + if [ $found -ne 0 ] ; then > + printf "FAIL: unfound the expected backtrace entry \"%s\"\n" "$pattern" > return 1 > fi > done ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: perf test for inet_pton probe fails to match stacktrace 2024-11-26 16:39 ` Arnaldo Carvalho de Melo @ 2024-11-26 18:27 ` Leo Yan 0 siblings, 0 replies; 8+ messages in thread From: Leo Yan @ 2024-11-26 18:27 UTC (permalink / raw) To: Arnaldo Carvalho de Melo; +Cc: Namhyung Kim, Guilherme Amadio, linux-perf-users Hi Arnaldo, On Tue, Nov 26, 2024 at 01:39:39PM -0300, Arnaldo Carvalho de Melo wrote: [...] > Seems to work: > > Before, on a: > > root@number:~# grep -m1 'model name' /proc/cpuinfo > model name : Intel(R) Core(TM) i7-14700K > root@number:~# > > root@number:~# perf test -vF pton > --- start --- > ping 229025 [017] 34027.968348: probe_libc:inet_pton: (7fc4542c10d0) > 7fc4542c10d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) > 7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6) > FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)" > ---- end ---- > 86: probe libc's inet_pton & backtrace it with ping : FAILED! > root@number:~# > > With your patch: > > root@number:~# perf test -vF pton > --- start --- > pattern: ping[][0-9 \.:]+probe_libc:inet_pton: \([[:xdigit:]]+\) > line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) > found=0 > pattern: .*inet_pton\+0x[[:xdigit:]]+[[:space:]]\(/usr/lib64/libc.so.6|inlined\)$ > line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) > found=1 > line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) > found=0 > pattern: .*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$ > line: ping 231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0) > found=1 > line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) > found=1 > line: 7f6352993a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6) > found=1 > line: 55ae71919969 [unknown] (/usr/bin/ping) > found=0 > ---- end ---- > 86: probe libc's inet_pton & backtrace it with ping : Ok > root@number:~# > > Replace that "unfound" with "Didn't find the" Will do. > And using 'found' as a variable name that when zero means "Ok, found" > and when 1 (grep failed to find the regexp) means "Didn't find" looks > confusing. Indeed. Will refine this. > General idea seems to make the test resilient to the differences while > still testing the backtrace in uprobes events for expected entries. Will send formal patch. Thanks a lot for the testing, Leo ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: perf test for inet_pton probe fails to match stacktrace 2024-11-26 5:08 ` Namhyung Kim 2024-11-26 10:27 ` Leo Yan @ 2024-11-26 16:27 ` Arnaldo Carvalho de Melo 1 sibling, 0 replies; 8+ messages in thread From: Arnaldo Carvalho de Melo @ 2024-11-26 16:27 UTC (permalink / raw) To: Namhyung Kim; +Cc: Guilherme Amadio, Leo Yan, linux-perf-users On Mon, Nov 25, 2024 at 09:08:15PM -0800, Namhyung Kim wrote: > Hello guys, > > On Mon, Nov 25, 2024 at 02:12:41PM +0100, Guilherme Amadio wrote: > > Dear Leo, > > > > On Sun, Nov 24, 2024 at 10:50:53AM +0000, Leo Yan wrote: > > > Hi Guilherme, > > > > > > On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote: > > > > > > [...] > > > > [...] > > > > > > This is not the first reported failure on this case, see [1] for > > > previous one. > > Right, and I'm also seeing the failure now. > > > > > > > I assume you are working on x86_64 machine. Could you confirm if the > > > change below can fix your issue? If yes, I will send a formal patch. > > > > Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed. > > The patch you sent didn't work out of the box, but with minor modifications > > I could fix it. I just needed to replace nm -D $libc with nm $libc for it work. > > The reason is that gaih_inet doesn't show with nm -D for my glibc: > > > > $ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo > > 0000000000116a40 T getaddrinfo@@GLIBC_2.2.5 > > 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5 > > 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 > > $ nm /usr/lib64/libc.so.6 | grep getaddrinfo > > 0000000000116a40 t __GI_getaddrinfo > > 000000000010c140 t __getaddrinfo_a > > 0000000000115e53 t __libc_getaddrinfo_freemem > > 0000000000116a40 T getaddrinfo > > 000000000010c140 T getaddrinfo_a@@GLIBC_2.34 > > 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5 > > > > $ nm -D /usr/lib64/libc.so.6 | grep gaih_inet > > $ nm /usr/lib64/libc.so.6 | grep gaih_inet > > 0000000000115405 t gaih_inet > > 00000000001139b2 t gaih_inet_serv > > 0000000000177780 r gaih_inet_typeproto > > > > Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass: > > > > $ sudo ./perf test -vF pton > > --- start --- > > ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7) > > 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6) > > 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6) > > 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6) > > 55680ac46518 [unknown] (/usr/bin/ping) > > ---- end ---- > > 86: probe libc's inet_pton & backtrace it with ping : Ok > > I think it's very fragile as it depends on specific entries in the call > stack. As we want to test if perf probe (for uprobes) and perf record > working correctly, maybe it can just check the number of samples only. > > Arnaldo, what do you think? That would reduce the usefulness of the test as it would be testing fewer aspects of perf's functionality, including backtraces from uprobes in addition to backtraces in general. So I think Leo has a point in a followup message to this one where he seems to have identified a subset of functions that are present in the architectures we test? - Arnaldo ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2024-11-26 18:36 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-11-22 21:35 perf test for inet_pton probe fails to match stacktrace Guilherme Amadio 2024-11-24 10:50 ` Leo Yan 2024-11-25 13:12 ` Guilherme Amadio 2024-11-26 5:08 ` Namhyung Kim 2024-11-26 10:27 ` Leo Yan 2024-11-26 16:39 ` Arnaldo Carvalho de Melo 2024-11-26 18:27 ` Leo Yan 2024-11-26 16:27 ` Arnaldo Carvalho de Melo
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.