perf test for inet_pton probe fails to match stacktrace

perf test for inet_pton probe fails to match stacktrace

[Guilherme Amadio]

Hi Arnaldo, Namhyung,

I've had this test fail for me since a while, so I'm reporting below
in case we can find a good solution to fix it on my system without
breaking other OSs. The test that fails is this:

gentoo ~ # perf test -vF inet_pton
--- start ---
ping 303105 [029] 47754.974977: probe_libc:inet_pton: (7f70ca48feb7)
7f70ca48feb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)"
---- end ----
 92: probe libc's inet_pton & backtrace it with ping                 : FAILED!
gentoo ~ # perf version
perf version 6.12

Here's why it fails. My stack traces has more than the test expects:

gentoo ~ # perf script | tac | sed -e /^ping/q | tac
ping  267392 [010] 42482.886216: probe_libc:inet_pton: (7f168d9b4eb7)
            7f168d9b4eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
            7f168d9c7517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
            7f168d9c8b73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
            55947171a518 [unknown] (/usr/bin/ping)
            7f168d8d6a72 __libc_start_call_main+0x7a (/usr/lib64/libc.so.6)
            7f168d8d6b29 __libc_start_main@@GLIBC_2.34+0x84 (/usr/lib64/libc.so.6)
            55947171ac55 [unknown] (/usr/bin/ping)

Likely this is due to how I compile glibc:
gentoo ~ # cat /var/db/pkg/sys-libs/glibc-2.40-r5/CFLAGS
-Og -march=native -Wl,--build-id -gdwarf-4 -fno-builtin-strlen -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer

If you could suggest a way to fix this, I'd appreciate it.

Best regards,
-Guilherme

Re: perf test for inet_pton probe fails to match stacktrace

[Leo Yan]

Hi Guilherme,

On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:

[...]

> Hi Arnaldo, Namhyung,
> 
> I've had this test fail for me since a while, so I'm reporting below
> in case we can find a good solution to fix it on my system without
> breaking other OSs. The test that fails is this:
> 
> gentoo ~ # perf test -vF inet_pton
> --- start ---
> ping 303105 [029] 47754.974977: probe_libc:inet_pton: (7f70ca48feb7)
> 7f70ca48feb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
> 7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
> FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7f70ca4a2517 gaih_inet+0x112 (/usr/lib64/libc.so.6)"
> ---- end ----
>  92: probe libc's inet_pton & backtrace it with ping                 : FAILED!
> gentoo ~ # perf version
> perf version 6.12
> 
> Here's why it fails. My stack traces has more than the test expects:
> 
> gentoo ~ # perf script | tac | sed -e /^ping/q | tac
> ping  267392 [010] 42482.886216: probe_libc:inet_pton: (7f168d9b4eb7)
>             7f168d9b4eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
>             7f168d9c7517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
>             7f168d9c8b73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
>             55947171a518 [unknown] (/usr/bin/ping)
>             7f168d8d6a72 __libc_start_call_main+0x7a (/usr/lib64/libc.so.6)
>             7f168d8d6b29 __libc_start_main@@GLIBC_2.34+0x84 (/usr/lib64/libc.so.6)
>             55947171ac55 [unknown] (/usr/bin/ping)
> 
> Likely this is due to how I compile glibc:
> gentoo ~ # cat /var/db/pkg/sys-libs/glibc-2.40-r5/CFLAGS
> -Og -march=native -Wl,--build-id -gdwarf-4 -fno-builtin-strlen -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer

This is not the first reported failure on this case, see [1] for
previous one.

I assume you are working on x86_64 machine.  Could you confirm if the
change below can fix your issue?  If yes, I will send a formal patch.

Thanks,
Leo

---8<--

diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
index 47a26f25db9f..5a2d641bf166 100755
--- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
+++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
@@ -54,6 +54,15 @@ trace_libc_inet_pton_backtrace() {
                ;;
        *)
                eventattr='max-stack=3'
+               # Add gaih_inet to expected backtrace only if it is part of libc.
+               if nm -D $libc | grep -F -q gaih_inet; then
+                       echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
+               # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17
+               # which is not used. Do strict checking for 'getaddrinfo' without any suffix.
+               if nm -D $libc | grep -q 'getaddrinfo$'; then
+                       echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
                echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
                ;;
        esac

[1] https://lore.kernel.org/linux-perf-users/20241025132021.GB359792@e132581.arm.com/

> If you could suggest a way to fix this, I'd appreciate it.
> 
> Best regards,
> -Guilherme
> 
> 

Re: perf test for inet_pton probe fails to match stacktrace

[Guilherme Amadio]

Dear Leo,

On Sun, Nov 24, 2024 at 10:50:53AM +0000, Leo Yan wrote:
> Hi Guilherme,
> 
> On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:
> 
> [...]
> > [...]
> 
> This is not the first reported failure on this case, see [1] for
> previous one.
> 
> I assume you are working on x86_64 machine.  Could you confirm if the
> change below can fix your issue?  If yes, I will send a formal patch.

Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed.
The patch you sent didn't work out of the box, but with minor modifications
I could fix it. I just needed to replace nm -D $libc with nm $libc for it work.
The reason is that gaih_inet doesn't show with nm -D for my glibc:

$ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo
0000000000116a40 T getaddrinfo@@GLIBC_2.2.5
000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
000000000010c140 T getaddrinfo_a@@GLIBC_2.34
$ nm /usr/lib64/libc.so.6 | grep getaddrinfo
0000000000116a40 t __GI_getaddrinfo
000000000010c140 t __getaddrinfo_a
0000000000115e53 t __libc_getaddrinfo_freemem
0000000000116a40 T getaddrinfo
000000000010c140 T getaddrinfo_a@@GLIBC_2.34
000000000010c140 T getaddrinfo_a@GLIBC_2.2.5

$ nm -D /usr/lib64/libc.so.6 | grep gaih_inet
$ nm /usr/lib64/libc.so.6 | grep gaih_inet
0000000000115405 t gaih_inet
00000000001139b2 t gaih_inet_serv
0000000000177780 r gaih_inet_typeproto

Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:

$ sudo ./perf test -vF pton
--- start ---
ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
55680ac46518 [unknown] (/usr/bin/ping)
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : Ok


The final patch I had to apply is below:

perf $ git diff
diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
index 47a26f25db9f..18274e1c315c 100755
--- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
+++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
@@ -53,7 +53,16 @@ trace_libc_inet_pton_backtrace() {
                echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
                ;;
        *)
-               eventattr='max-stack=3'
+               eventattr='max-stack=4'
+               # Add gaih_inet to expected backtrace only if it is part of libc.
+               if nm $libc | grep -F -q gaih_inet; then
+                       echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
+               # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17
+               # which is not used. Do strict checking for 'getaddrinfo' without any suffix.
+               if nm $libc | grep -q 'getaddrinfo$'; then
+                       echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
+               fi
                echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
                ;;
        esac

Please feel free to add Tested-by: and/or Reviewed-by: Guilherme Amadio <amadio@gentoo.org> 
to the final patch as you see fit. Thank you!

Best regards,
-Guilherme

> 
> Thanks,
> Leo
> 
> ---8<--
> 
> diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> index 47a26f25db9f..5a2d641bf166 100755
> --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> @@ -54,6 +54,15 @@ trace_libc_inet_pton_backtrace() {
>                 ;;
>         *)
>                 eventattr='max-stack=3'
> +               # Add gaih_inet to expected backtrace only if it is part of libc.
> +               if nm -D $libc | grep -F -q gaih_inet; then
> +                       echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
> +               fi
> +               # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17
> +               # which is not used. Do strict checking for 'getaddrinfo' without any suffix.
> +               if nm -D $libc | grep -q 'getaddrinfo$'; then
> +                       echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
> +               fi
>                 echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
>                 ;;
>         esac
> 
> [1] https://lore.kernel.org/linux-perf-users/20241025132021.GB359792@e132581.arm.com/
> 
> > If you could suggest a way to fix this, I'd appreciate it.
> > 
> > Best regards,
> > -Guilherme
> > 
> > 

Re: perf test for inet_pton probe fails to match stacktrace

[Namhyung Kim]

Hello guys,

On Mon, Nov 25, 2024 at 02:12:41PM +0100, Guilherme Amadio wrote:
> Dear Leo,
> 
> On Sun, Nov 24, 2024 at 10:50:53AM +0000, Leo Yan wrote:
> > Hi Guilherme,
> > 
> > On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:
> > 
> > [...]
> > > [...]
> > 
> > This is not the first reported failure on this case, see [1] for
> > previous one.

Right, and I'm also seeing the failure now.

> > 
> > I assume you are working on x86_64 machine.  Could you confirm if the
> > change below can fix your issue?  If yes, I will send a formal patch.
> 
> Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed.
> The patch you sent didn't work out of the box, but with minor modifications
> I could fix it. I just needed to replace nm -D $libc with nm $libc for it work.
> The reason is that gaih_inet doesn't show with nm -D for my glibc:
> 
> $ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo
> 0000000000116a40 T getaddrinfo@@GLIBC_2.2.5
> 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
> 000000000010c140 T getaddrinfo_a@@GLIBC_2.34
> $ nm /usr/lib64/libc.so.6 | grep getaddrinfo
> 0000000000116a40 t __GI_getaddrinfo
> 000000000010c140 t __getaddrinfo_a
> 0000000000115e53 t __libc_getaddrinfo_freemem
> 0000000000116a40 T getaddrinfo
> 000000000010c140 T getaddrinfo_a@@GLIBC_2.34
> 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
> 
> $ nm -D /usr/lib64/libc.so.6 | grep gaih_inet
> $ nm /usr/lib64/libc.so.6 | grep gaih_inet
> 0000000000115405 t gaih_inet
> 00000000001139b2 t gaih_inet_serv
> 0000000000177780 r gaih_inet_typeproto
> 
> Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:
> 
> $ sudo ./perf test -vF pton
> --- start ---
> ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
> 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
> 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
> 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
> 55680ac46518 [unknown] (/usr/bin/ping)
> ---- end ----
>  86: probe libc's inet_pton & backtrace it with ping                 : Ok

I think it's very fragile as it depends on specific entries in the call
stack.  As we want to test if perf probe (for uprobes) and perf record
working correctly, maybe it can just check the number of samples only.

Arnaldo, what do you think?

Thanks,
Namhyung

> 
> 
> The final patch I had to apply is below:
> 
> perf $ git diff
> diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> index 47a26f25db9f..18274e1c315c 100755
> --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> @@ -53,7 +53,16 @@ trace_libc_inet_pton_backtrace() {
>                 echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
>                 ;;
>         *)
> -               eventattr='max-stack=3'
> +               eventattr='max-stack=4'
> +               # Add gaih_inet to expected backtrace only if it is part of libc.
> +               if nm $libc | grep -F -q gaih_inet; then
> +                       echo "gaih_inet\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
> +               fi
> +               # The new version libc (e.g. libc 2.36) contains symbol getaddrinfo@@GLIBC_2.17
> +               # which is not used. Do strict checking for 'getaddrinfo' without any suffix.
> +               if nm $libc | grep -q 'getaddrinfo$'; then
> +                       echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
> +               fi
>                 echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
>                 ;;
>         esac
> 
> Please feel free to add Tested-by: and/or Reviewed-by: Guilherme Amadio <amadio@gentoo.org> 
> to the final patch as you see fit. Thank you!

Thanks for the report and the fix.
Namhyung

Re: perf test for inet_pton probe fails to match stacktrace

[Leo Yan]

On Mon, Nov 25, 2024 at 09:08:15PM -0800, Namhyung Kim wrote:

[...]

> > Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:
> >
> > $ sudo ./perf test -vF pton
> > --- start ---
> > ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
> > 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
> > 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
> > 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
> > 55680ac46518 [unknown] (/usr/bin/ping)
> > ---- end ----
> >  86: probe libc's inet_pton & backtrace it with ping                 : Ok
> 
> I think it's very fragile as it depends on specific entries in the call
> stack.  As we want to test if perf probe (for uprobes) and perf record
> working correctly, maybe it can just check the number of samples only.
> 
> Arnaldo, what do you think?

Though this question is asking Arnaldo, I have a different idea for
fixing the test ;)

The middle entries in the callchain are varied on different archs and
distro.  E.g. Guilherme reported back the '-D' option is not needed but
at my side it is required on Arm64 machine with Ubuntu distro.

However, the three entries in the callchain are existed consistently:

  ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
  7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
  ...
  55680ac46518 [unknown] (/usr/bin/ping)

We can change the code to only matching these three entries.  The
change is as below.  We can consolidate the code for all archs
(I am not 100% sure for x390x so I leave it).

Please let me know if this makes sense for you?

Thanks,
Leo

---8<---

diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
index 47a26f25db9f..6ef58aedf2d8 100755
--- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
+++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
@@ -43,17 +43,8 @@ trace_libc_inet_pton_backtrace() {
 		echo "((__GI_)?getaddrinfo|text_to_binary_address)\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
 		echo "(gaih_inet|main)\+0x[[:xdigit:]]+[[:space:]]\(inlined|.*/bin/ping.*\)$" >> $expected
 		;;
-	ppc64|ppc64le)
-		eventattr='max-stack=4'
-		# Add gaih_inet to expected backtrace only if it is part of libc.
-		if nm $libc | grep -F -q gaih_inet.; then
-			echo "gaih_inet.*\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
-		fi
-		echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
-		echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
-		;;
 	*)
-		eventattr='max-stack=3'
+		eventattr='max-stack=4'
 		echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
 		;;
 	esac
@@ -76,14 +67,23 @@ trace_libc_inet_pton_backtrace() {
 	fi
 	perf script -i $perf_data | tac | grep -m1 ^ping -B9 | tac > $perf_script
 
-	exec 3<$perf_script
 	exec 4<$expected
-	while read line <&3 && read -r pattern <&4; do
+	while read -r pattern <&4; do
+		echo "pattern: $pattern"
 		[ -z "$pattern" ] && break
-		echo $line
-		echo "$line" | grep -E -q "$pattern"
-		if [ $? -ne 0 ] ; then
-			printf "FAIL: expected backtrace entry \"%s\" got \"%s\"\n" "$pattern" "$line"
+
+		exec 3<$perf_script
+		while read line <&3; do
+			[ -z "$line" ] && break
+			echo "line: $line"
+			echo "$line" | grep -E -q "$pattern"
+			found=$?
+			echo "found=$found"
+			[ $found -eq 0 ] && break
+		done
+
+		if [ $found -ne 0 ] ; then
+			printf "FAIL: unfound the expected backtrace entry \"%s\"\n" "$pattern"
 			return 1
 		fi
 	done

Re: perf test for inet_pton probe fails to match stacktrace

[Arnaldo Carvalho de Melo]

On Tue, Nov 26, 2024 at 10:27:05AM +0000, Leo Yan wrote:
> On Mon, Nov 25, 2024 at 09:08:15PM -0800, Namhyung Kim wrote:
> 
> [...]
> 
> > > Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:
> > >
> > > $ sudo ./perf test -vF pton
> > > --- start ---
> > > ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
> > > 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
> > > 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
> > > 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
> > > 55680ac46518 [unknown] (/usr/bin/ping)
> > > ---- end ----
> > >  86: probe libc's inet_pton & backtrace it with ping                 : Ok
> > 
> > I think it's very fragile as it depends on specific entries in the call
> > stack.  As we want to test if perf probe (for uprobes) and perf record
> > working correctly, maybe it can just check the number of samples only.
> > 
> > Arnaldo, what do you think?
> 
> Though this question is asking Arnaldo, I have a different idea for
> fixing the test ;)
> 
> The middle entries in the callchain are varied on different archs and
> distro.  E.g. Guilherme reported back the '-D' option is not needed but
> at my side it is required on Arm64 machine with Ubuntu distro.
> 
> However, the three entries in the callchain are existed consistently:
> 
>   ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
>   7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
>   ...
>   55680ac46518 [unknown] (/usr/bin/ping)
> 
> We can change the code to only matching these three entries.  The
> change is as below.  We can consolidate the code for all archs
> (I am not 100% sure for x390x so I leave it).
> 
> Please let me know if this makes sense for you?
> 
> Thanks,
> Leo

Seems to work:

Before, on a:

root@number:~# grep -m1 'model name' /proc/cpuinfo 
model name	: Intel(R) Core(TM) i7-14700K
root@number:~#

root@number:~# perf test -vF pton
--- start ---
ping 229025 [017] 34027.968348: probe_libc:inet_pton: (7fc4542c10d0)
7fc4542c10d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)
FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)"
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : FAILED!
root@number:~# 

With your patch:

root@number:~# perf test -vF pton
--- start ---
pattern: ping[][0-9 \.:]+probe_libc:inet_pton: \([[:xdigit:]]+\)
line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
found=0
pattern: .*inet_pton\+0x[[:xdigit:]]+[[:space:]]\(/usr/lib64/libc.so.6|inlined\)$
line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
found=1
line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
found=0
pattern: .*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$
line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
found=1
line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
found=1
line: 7f6352993a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)
found=1
line: 55ae71919969 [unknown] (/usr/bin/ping)
found=0
---- end ----
 86: probe libc's inet_pton & backtrace it with ping                 : Ok
root@number:~#

Replace that "unfound" with "Didn't find the"

And using 'found' as a variable name that when zero means "Ok, found"
and when 1 (grep failed to find the regexp) means "Didn't find" looks
confusing.

General idea seems to make the test resilient to the differences while
still testing the backtrace in uprobes events for expected entries.

- Arnaldo
 
> ---8<---
> 
> diff --git a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> index 47a26f25db9f..6ef58aedf2d8 100755
> --- a/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> +++ b/tools/perf/tests/shell/record+probe_libc_inet_pton.sh
> @@ -43,17 +43,8 @@ trace_libc_inet_pton_backtrace() {
>  		echo "((__GI_)?getaddrinfo|text_to_binary_address)\+0x[[:xdigit:]]+[[:space:]]\($libc|inlined\)$" >> $expected
>  		echo "(gaih_inet|main)\+0x[[:xdigit:]]+[[:space:]]\(inlined|.*/bin/ping.*\)$" >> $expected
>  		;;
> -	ppc64|ppc64le)
> -		eventattr='max-stack=4'
> -		# Add gaih_inet to expected backtrace only if it is part of libc.
> -		if nm $libc | grep -F -q gaih_inet.; then
> -			echo "gaih_inet.*\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
> -		fi
> -		echo "getaddrinfo\+0x[[:xdigit:]]+[[:space:]]\($libc\)$" >> $expected
> -		echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
> -		;;
>  	*)
> -		eventattr='max-stack=3'
> +		eventattr='max-stack=4'
>  		echo ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" >> $expected
>  		;;
>  	esac
> @@ -76,14 +67,23 @@ trace_libc_inet_pton_backtrace() {
>  	fi
>  	perf script -i $perf_data | tac | grep -m1 ^ping -B9 | tac > $perf_script
>  
> -	exec 3<$perf_script
>  	exec 4<$expected
> -	while read line <&3 && read -r pattern <&4; do
> +	while read -r pattern <&4; do
> +		echo "pattern: $pattern"
>  		[ -z "$pattern" ] && break
> -		echo $line
> -		echo "$line" | grep -E -q "$pattern"
> -		if [ $? -ne 0 ] ; then
> -			printf "FAIL: expected backtrace entry \"%s\" got \"%s\"\n" "$pattern" "$line"
> +
> +		exec 3<$perf_script
> +		while read line <&3; do
> +			[ -z "$line" ] && break
> +			echo "line: $line"
> +			echo "$line" | grep -E -q "$pattern"
> +			found=$?
> +			echo "found=$found"
> +			[ $found -eq 0 ] && break
> +		done
> +
> +		if [ $found -ne 0 ] ; then
> +			printf "FAIL: unfound the expected backtrace entry \"%s\"\n" "$pattern"
>  			return 1
>  		fi
>  	done

Re: perf test for inet_pton probe fails to match stacktrace

[Leo Yan]

Hi Arnaldo,

On Tue, Nov 26, 2024 at 01:39:39PM -0300, Arnaldo Carvalho de Melo wrote:

[...]

> Seems to work:
> 
> Before, on a:
> 
> root@number:~# grep -m1 'model name' /proc/cpuinfo
> model name      : Intel(R) Core(TM) i7-14700K
> root@number:~#
> 
> root@number:~# perf test -vF pton
> --- start ---
> ping 229025 [017] 34027.968348: probe_libc:inet_pton: (7fc4542c10d0)
> 7fc4542c10d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
> 7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)
> FAIL: expected backtrace entry ".*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$" got "7fc4542d5a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)"
> ---- end ----
>  86: probe libc's inet_pton & backtrace it with ping                 : FAILED!
> root@number:~#
> 
> With your patch:
> 
> root@number:~# perf test -vF pton
> --- start ---
> pattern: ping[][0-9 \.:]+probe_libc:inet_pton: \([[:xdigit:]]+\)
> line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
> found=0
> pattern: .*inet_pton\+0x[[:xdigit:]]+[[:space:]]\(/usr/lib64/libc.so.6|inlined\)$
> line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
> found=1
> line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
> found=0
> pattern: .*(\+0x[[:xdigit:]]+|\[unknown\])[[:space:]]\(.*/bin/ping.*\)$
> line: ping  231118 [023] 34089.948542: probe_libc:inet_pton: (7f635297f0d0)
> found=1
> line: 7f635297f0d0 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
> found=1
> line: 7f6352993a55 getaddrinfo+0xcf5 (/usr/lib64/libc.so.6)
> found=1
> line: 55ae71919969 [unknown] (/usr/bin/ping)
> found=0
> ---- end ----
>  86: probe libc's inet_pton & backtrace it with ping                 : Ok
> root@number:~#
> 
> Replace that "unfound" with "Didn't find the"

Will do.

> And using 'found' as a variable name that when zero means "Ok, found"
> and when 1 (grep failed to find the regexp) means "Didn't find" looks
> confusing.

Indeed.  Will refine this.

> General idea seems to make the test resilient to the differences while
> still testing the backtrace in uprobes events for expected entries.

Will send formal patch.

Thanks a lot for the testing,
Leo

Re: perf test for inet_pton probe fails to match stacktrace

[Arnaldo Carvalho de Melo]

On Mon, Nov 25, 2024 at 09:08:15PM -0800, Namhyung Kim wrote:
> Hello guys,
> 
> On Mon, Nov 25, 2024 at 02:12:41PM +0100, Guilherme Amadio wrote:
> > Dear Leo,
> > 
> > On Sun, Nov 24, 2024 at 10:50:53AM +0000, Leo Yan wrote:
> > > Hi Guilherme,
> > > 
> > > On Fri, Nov 22, 2024 at 10:35:16PM +0100, Guilherme Amadio wrote:
> > > 
> > > [...]
> > > > [...]
> > > 
> > > This is not the first reported failure on this case, see [1] for
> > > previous one.
> 
> Right, and I'm also seeing the failure now.
> 
> > > 
> > > I assume you are working on x86_64 machine.  Could you confirm if the
> > > change below can fix your issue?  If yes, I will send a formal patch.
> > 
> > Indeed, I am working on an AMD 3950X, and have glibc 2.40 installed.
> > The patch you sent didn't work out of the box, but with minor modifications
> > I could fix it. I just needed to replace nm -D $libc with nm $libc for it work.
> > The reason is that gaih_inet doesn't show with nm -D for my glibc:
> > 
> > $ nm -D /usr/lib64/libc.so.6 | grep getaddrinfo
> > 0000000000116a40 T getaddrinfo@@GLIBC_2.2.5
> > 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
> > 000000000010c140 T getaddrinfo_a@@GLIBC_2.34
> > $ nm /usr/lib64/libc.so.6 | grep getaddrinfo
> > 0000000000116a40 t __GI_getaddrinfo
> > 000000000010c140 t __getaddrinfo_a
> > 0000000000115e53 t __libc_getaddrinfo_freemem
> > 0000000000116a40 T getaddrinfo
> > 000000000010c140 T getaddrinfo_a@@GLIBC_2.34
> > 000000000010c140 T getaddrinfo_a@GLIBC_2.2.5
> > 
> > $ nm -D /usr/lib64/libc.so.6 | grep gaih_inet
> > $ nm /usr/lib64/libc.so.6 | grep gaih_inet
> > 0000000000115405 t gaih_inet
> > 00000000001139b2 t gaih_inet_serv
> > 0000000000177780 r gaih_inet_typeproto
> > 
> > Using nm $libc in both cases, and increasing max-stack to 4 allows the test to pass:
> > 
> > $ sudo ./perf test -vF pton
> > --- start ---
> > ping 1237113 [000] 18561.204983: probe_libc:inet_pton: (7f7c34d38eb7)
> > 7f7c34d38eb7 __GI___inet_pton+0x0 (/usr/lib64/libc.so.6)
> > 7f7c34d4b517 gaih_inet+0x112 (/usr/lib64/libc.so.6)
> > 7f7c34d4cb73 getaddrinfo+0x133 (/usr/lib64/libc.so.6)
> > 55680ac46518 [unknown] (/usr/bin/ping)
> > ---- end ----
> >  86: probe libc's inet_pton & backtrace it with ping                 : Ok
> 
> I think it's very fragile as it depends on specific entries in the call
> stack.  As we want to test if perf probe (for uprobes) and perf record
> working correctly, maybe it can just check the number of samples only.
> 
> Arnaldo, what do you think?

That would reduce the usefulness of the test as it would be testing
fewer aspects of perf's functionality, including backtraces from uprobes
in addition to backtraces in general.

So I think Leo has a point in a followup message to this one where he
seems to have identified a subset of functions that are present in the
architectures we test?

- Arnaldo