[lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev <lttng-dev@lists.lttng.org>]

[-- Attachment #1.1: Type: text/plain, Size: 15681 bytes --]

Hi,

I'm getting a weird segfault when I compile in a lttng-ust tracepoint into
my application.

I'm able to run the lttng-ust hello_world example just fine on my platform
which one would think should prove everything out, but when I use the exact
same hello world trace provider and link it into my multi-threadded app and
simply call:

lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23,
            "hi there!");

I get a segfault.

Background.  I'm on aarch64 platform (imx8mm-lpddr4-evk) with sdk (gcc 11.4
glibc 2.35) and rootfs generated from Yocto Kirkstone 4.0.18.  Built
lttng-tools (2.13.9) and lttng-ust (2.13.6) from yocto recipe.  Built
out-of-tree kernel modules for lttng-modules (2.13.9) and I'm on kernel
6.1.38.

If I do a backtrace with gdb on the core this is what I see:

GNU gdb (GDB) 11.2
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-poky-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
   <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./my_app...

warning: core file may not match specified executable file.
[New LWP 757]
[New LWP 759]
[New LWP 758]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `./my_app'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
96      ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or
directory.
[Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))]
(gdb) bt
#0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
#1  0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register () from
/usr/lib/liblttng-ust-tracepoint.so.1
#2  0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at
/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629

#3  0x0000ffffb872b30c in call_init (env=<optimized out>,
argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145
#4  __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1,
argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=<optimized out>) at
../csu/libc-start.c:376
#5  0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81
(gdb)

If I run valgrind, this is what I see:

==627== Memcheck, a memory error detector
==627== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==627== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==627== Command: ./my_app
==627==
LTTng-UST: Error while registering tracepoint probe.
==627==
==627== Process terminating with default action of signal 6 (SIGABRT)
==627==    at 0x6772278: __pthread_kill_implementation (pthread_kill.c:44)
==627==    by 0x672DD4F: raise (raise.c:26)
==627==    by 0x671AEE3: abort (abort.c:79)
==627==    by 0x2C572B: lttng_ust__events_init__hello_world
(ust-tracepoint-event.h:1221)
==627==    by 0x2C5B97: lttng_ust_constructor_hello_world
(ust-tracepoint-event.h:1239)
==627==    by 0x671B30B: call_init (libc-start.c:145)
==627==    by 0x671B30B: __libc_start_main@@GLIBC_2.34 (libc-start.c:376)
==627==    by 0x21122F: (below main) (start.S:81)
==627==
==627== HEAP SUMMARY:
==627==     in use at exit: 151,712 bytes in 90 blocks
==627==   total heap usage: 1,184 allocs, 1,094 frees, 202,570 bytes
allocated
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 1 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x68D50CB: lttng_ust_tracepoint_module_register (in
/usr/lib/liblttng-ust-tracepoint.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 2 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61E8143: lttng_ust_probe_register (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61FD49B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 3 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61E8143: lttng_ust_probe_register (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61DFF4F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 4 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61E8143: lttng_ust_probe_register (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61E026F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 5 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61E8143: lttng_ust_probe_register (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61E058F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 6 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x68D50CB: lttng_ust_tracepoint_module_register (in
/usr/lib/liblttng-ust-tracepoint.so.1.0.0)
==627==    by 0x2C5A17: lttng_ust__tracepoints__ptrs_init
(tracepoint.h:629)
==627==    by 0x671B30B: call_init (libc-start.c:145)
==627==    by 0x671B30B: __libc_start_main@@GLIBC_2.34 (libc-start.c:376)
==627==    by 0x21122F: (below main) (start.S:81)
==627==
==627== 128 bytes in 1 blocks are still reachable in loss record 7 of 17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x68A388B: ??? (in /usr/lib/liblttng-ust-common.so.1.0.0)
==627==    by 0x68A2FBF: lttng_ust_common_ctor (in
/usr/lib/liblttng-ust-common.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 200 bytes in 1 blocks are still reachable in loss record 8 of 17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x5929E37: malloc (rtld-malloc.h:56)
==627==    by 0x5929E37: add_to_global_resize (dl-open.c:152)
==627==    by 0x592AA8F: dl_open_worker_begin (dl-open.c:716)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x5929C77: dl_open_worker (dl-open.c:782)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x592A07B: _dl_open (dl-open.c:884)
==627==    by 0x676C467: dlopen_doit (dlopen.c:56)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
==627==    by 0x676BF4F: _dlerror_run (dlerror.c:138)
==627==    by 0x676C52B: dlopen_implementation (dlopen.c:71)
==627==    by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
==627==
==627== 240 bytes in 2 blocks are still reachable in loss record 9 of 17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x59214C7: malloc (rtld-malloc.h:56)
==627==    by 0x59214C7: _dl_map_object_deps (dl-deps.c:479)
==627==    by 0x592A50B: dl_open_worker_begin (dl-open.c:592)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x5929C77: dl_open_worker (dl-open.c:782)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x592A07B: _dl_open (dl-open.c:884)
==627==    by 0x676C467: dlopen_doit (dlopen.c:56)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
==627==    by 0x676BF4F: _dlerror_run (dlerror.c:138)
==627==    by 0x676C52B: dlopen_implementation (dlopen.c:71)
==627==    by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
==627==
==627== 292 bytes in 15 blocks are possibly lost in loss record 10 of 17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x6783993: strdup (strdup.c:42)
==627==    by 0x61FD92F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
==627==    by 0x61FE653: lttng_ust_dl_update (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 336 bytes in 1 blocks are possibly lost in loss record 11 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x592ECB7: calloc (rtld-malloc.h:44)
==627==    by 0x592ECB7: allocate_dtv (dl-tls.c:376)
==627==    by 0x592F723: _dl_allocate_tls (dl-tls.c:635)
==627==    by 0x677114F: allocate_stack (allocatestack.c:428)
==627==    by 0x677114F: pthread_create@@GLIBC_2.34 (pthread_create.c:647)
==627==    by 0x61DEF03: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 336 bytes in 1 blocks are possibly lost in loss record 12 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x592ECB7: calloc (rtld-malloc.h:44)
==627==    by 0x592ECB7: allocate_dtv (dl-tls.c:376)
==627==    by 0x592F723: _dl_allocate_tls (dl-tls.c:635)
==627==    by 0x677114F: allocate_stack (allocatestack.c:428)
==627==    by 0x677114F: pthread_create@@GLIBC_2.34 (pthread_create.c:647)
==627==    by 0x61DEEB7: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 340 bytes in 17 blocks are possibly lost in loss record 13 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61FD94F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
==627==    by 0x61FE653: lttng_ust_dl_update (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 1,248 bytes in 26 blocks are still reachable in loss record 14 of
17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x68D529F: lttng_ust_tracepoint_module_register (in
/usr/lib/liblttng-ust-tracepoint.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 4,608 bytes in 2 blocks are possibly lost in loss record 15 of 17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x5922FFB: malloc (rtld-malloc.h:56)
==627==    by 0x5922FFB: _dlfo_mappings_segment_allocate
(dl-find_object.c:217)
==627==    by 0x5922FFB: _dl_find_object_update_1 (dl-find_object.c:671)
==627==    by 0x5922FFB: _dl_find_object_update (dl-find_object.c:805)
==627==    by 0x592A803: dl_open_worker_begin (dl-open.c:735)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x5929C77: dl_open_worker (dl-open.c:782)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x592A07B: _dl_open (dl-open.c:884)
==627==    by 0x676C467: dlopen_doit (dlopen.c:56)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
==627==    by 0x676BF4F: _dlerror_run (dlerror.c:138)
==627==    by 0x676C52B: dlopen_implementation (dlopen.c:71)
==627==    by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
==627==
==627== 70,992 bytes in 17 blocks are possibly lost in loss record 16 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61FD91B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
==627==    by 0x61FE653: lttng_ust_dl_update (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 72,704 bytes in 1 blocks are still reachable in loss record 17 of
17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x653E137: ??? (in /usr/lib/libstdc++.so.6.0.29)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== LEAK SUMMARY:
==627==    definitely lost: 0 bytes in 0 blocks
==627==    indirectly lost: 0 bytes in 0 blocks
==627==      possibly lost: 76,904 bytes in 53 blocks
==627==    still reachable: 74,808 bytes in 37 blocks
==627==         suppressed: 0 bytes in 0 blocks
==627==
==627== For lists of detected and suppressed errors, rerun with: -s
==627== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 0 from 0)
Aborted

I'm sure I'm probably doing something wrong but I've tried static linking,
dynamic linking using LD_PRELOAD helpers like
LD_PRELOAD=liblttng-ust-pthread-wrapper.so.1:liblttng-ust-fd.so.1:liblttng-ust-fork.so.1
and I'm out of options that I know to try.

Looking for pointers on what else I can try and what I could be doing wrong.

I had this same segfault issue using Dunfell version of Yocto and earlier
versions of lttng (2.11) but never dug into it because was planning on
upgrading everything soon, so recently stepped up to newer versions of
everything to see if that solved the issue but apparently not.

Regards,

Brian

[-- Attachment #1.2: Type: text/html, Size: 20255 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev