[lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 15681 bytes --]

Hi,

I'm getting a weird segfault when I compile in a lttng-ust tracepoint into
my application.

I'm able to run the lttng-ust hello_world example just fine on my platform
which one would think should prove everything out, but when I use the exact
same hello world trace provider and link it into my multi-threadded app and
simply call:

lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23,
            "hi there!");

I get a segfault.

Background.  I'm on aarch64 platform (imx8mm-lpddr4-evk) with sdk (gcc 11.4
glibc 2.35) and rootfs generated from Yocto Kirkstone 4.0.18.  Built
lttng-tools (2.13.9) and lttng-ust (2.13.6) from yocto recipe.  Built
out-of-tree kernel modules for lttng-modules (2.13.9) and I'm on kernel
6.1.38.

If I do a backtrace with gdb on the core this is what I see:

GNU gdb (GDB) 11.2
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-poky-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
   <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./my_app...

warning: core file may not match specified executable file.
[New LWP 757]
[New LWP 759]
[New LWP 758]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `./my_app'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
96      ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or
directory.
[Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))]
(gdb) bt
#0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
#1  0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register () from
/usr/lib/liblttng-ust-tracepoint.so.1
#2  0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at
/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629

#3  0x0000ffffb872b30c in call_init (env=<optimized out>,
argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145
#4  __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1,
argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=<optimized out>) at
../csu/libc-start.c:376
#5  0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81
(gdb)

If I run valgrind, this is what I see:

==627== Memcheck, a memory error detector
==627== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==627== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==627== Command: ./my_app
==627==
LTTng-UST: Error while registering tracepoint probe.
==627==
==627== Process terminating with default action of signal 6 (SIGABRT)
==627==    at 0x6772278: __pthread_kill_implementation (pthread_kill.c:44)
==627==    by 0x672DD4F: raise (raise.c:26)
==627==    by 0x671AEE3: abort (abort.c:79)
==627==    by 0x2C572B: lttng_ust__events_init__hello_world
(ust-tracepoint-event.h:1221)
==627==    by 0x2C5B97: lttng_ust_constructor_hello_world
(ust-tracepoint-event.h:1239)
==627==    by 0x671B30B: call_init (libc-start.c:145)
==627==    by 0x671B30B: __libc_start_main@@GLIBC_2.34 (libc-start.c:376)
==627==    by 0x21122F: (below main) (start.S:81)
==627==
==627== HEAP SUMMARY:
==627==     in use at exit: 151,712 bytes in 90 blocks
==627==   total heap usage: 1,184 allocs, 1,094 frees, 202,570 bytes
allocated
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 1 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x68D50CB: lttng_ust_tracepoint_module_register (in
/usr/lib/liblttng-ust-tracepoint.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 2 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61E8143: lttng_ust_probe_register (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61FD49B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 3 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61E8143: lttng_ust_probe_register (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61DFF4F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 4 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61E8143: lttng_ust_probe_register (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61E026F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 5 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61E8143: lttng_ust_probe_register (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61E058F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 48 bytes in 1 blocks are still reachable in loss record 6 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x68D50CB: lttng_ust_tracepoint_module_register (in
/usr/lib/liblttng-ust-tracepoint.so.1.0.0)
==627==    by 0x2C5A17: lttng_ust__tracepoints__ptrs_init
(tracepoint.h:629)
==627==    by 0x671B30B: call_init (libc-start.c:145)
==627==    by 0x671B30B: __libc_start_main@@GLIBC_2.34 (libc-start.c:376)
==627==    by 0x21122F: (below main) (start.S:81)
==627==
==627== 128 bytes in 1 blocks are still reachable in loss record 7 of 17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x68A388B: ??? (in /usr/lib/liblttng-ust-common.so.1.0.0)
==627==    by 0x68A2FBF: lttng_ust_common_ctor (in
/usr/lib/liblttng-ust-common.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 200 bytes in 1 blocks are still reachable in loss record 8 of 17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x5929E37: malloc (rtld-malloc.h:56)
==627==    by 0x5929E37: add_to_global_resize (dl-open.c:152)
==627==    by 0x592AA8F: dl_open_worker_begin (dl-open.c:716)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x5929C77: dl_open_worker (dl-open.c:782)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x592A07B: _dl_open (dl-open.c:884)
==627==    by 0x676C467: dlopen_doit (dlopen.c:56)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
==627==    by 0x676BF4F: _dlerror_run (dlerror.c:138)
==627==    by 0x676C52B: dlopen_implementation (dlopen.c:71)
==627==    by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
==627==
==627== 240 bytes in 2 blocks are still reachable in loss record 9 of 17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x59214C7: malloc (rtld-malloc.h:56)
==627==    by 0x59214C7: _dl_map_object_deps (dl-deps.c:479)
==627==    by 0x592A50B: dl_open_worker_begin (dl-open.c:592)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x5929C77: dl_open_worker (dl-open.c:782)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x592A07B: _dl_open (dl-open.c:884)
==627==    by 0x676C467: dlopen_doit (dlopen.c:56)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
==627==    by 0x676BF4F: _dlerror_run (dlerror.c:138)
==627==    by 0x676C52B: dlopen_implementation (dlopen.c:71)
==627==    by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
==627==
==627== 292 bytes in 15 blocks are possibly lost in loss record 10 of 17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x6783993: strdup (strdup.c:42)
==627==    by 0x61FD92F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
==627==    by 0x61FE653: lttng_ust_dl_update (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 336 bytes in 1 blocks are possibly lost in loss record 11 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x592ECB7: calloc (rtld-malloc.h:44)
==627==    by 0x592ECB7: allocate_dtv (dl-tls.c:376)
==627==    by 0x592F723: _dl_allocate_tls (dl-tls.c:635)
==627==    by 0x677114F: allocate_stack (allocatestack.c:428)
==627==    by 0x677114F: pthread_create@@GLIBC_2.34 (pthread_create.c:647)
==627==    by 0x61DEF03: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 336 bytes in 1 blocks are possibly lost in loss record 12 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x592ECB7: calloc (rtld-malloc.h:44)
==627==    by 0x592ECB7: allocate_dtv (dl-tls.c:376)
==627==    by 0x592F723: _dl_allocate_tls (dl-tls.c:635)
==627==    by 0x677114F: allocate_stack (allocatestack.c:428)
==627==    by 0x677114F: pthread_create@@GLIBC_2.34 (pthread_create.c:647)
==627==    by 0x61DEEB7: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 340 bytes in 17 blocks are possibly lost in loss record 13 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61FD94F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
==627==    by 0x61FE653: lttng_ust_dl_update (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 1,248 bytes in 26 blocks are still reachable in loss record 14 of
17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x68D529F: lttng_ust_tracepoint_module_register (in
/usr/lib/liblttng-ust-tracepoint.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 4,608 bytes in 2 blocks are possibly lost in loss record 15 of 17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x5922FFB: malloc (rtld-malloc.h:56)
==627==    by 0x5922FFB: _dlfo_mappings_segment_allocate
(dl-find_object.c:217)
==627==    by 0x5922FFB: _dl_find_object_update_1 (dl-find_object.c:671)
==627==    by 0x5922FFB: _dl_find_object_update (dl-find_object.c:805)
==627==    by 0x592A803: dl_open_worker_begin (dl-open.c:735)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x5929C77: dl_open_worker (dl-open.c:782)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x592A07B: _dl_open (dl-open.c:884)
==627==    by 0x676C467: dlopen_doit (dlopen.c:56)
==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
==627==    by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
==627==    by 0x676BF4F: _dlerror_run (dlerror.c:138)
==627==    by 0x676C52B: dlopen_implementation (dlopen.c:71)
==627==    by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
==627==
==627== 70,992 bytes in 17 blocks are possibly lost in loss record 16 of 17
==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
==627==    by 0x61FD91B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
==627==    by 0x61FE653: lttng_ust_dl_update (in
/usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== 72,704 bytes in 1 blocks are still reachable in loss record 17 of
17
==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
==627==    by 0x653E137: ??? (in /usr/lib/libstdc++.so.6.0.29)
==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
==627==    by 0x5923767: call_init (dl-init.c:29)
==627==    by 0x5923767: _dl_init (dl-init.c:121)
==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
==627==
==627== LEAK SUMMARY:
==627==    definitely lost: 0 bytes in 0 blocks
==627==    indirectly lost: 0 bytes in 0 blocks
==627==      possibly lost: 76,904 bytes in 53 blocks
==627==    still reachable: 74,808 bytes in 37 blocks
==627==         suppressed: 0 bytes in 0 blocks
==627==
==627== For lists of detected and suppressed errors, rerun with: -s
==627== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 0 from 0)
Aborted

I'm sure I'm probably doing something wrong but I've tried static linking,
dynamic linking using LD_PRELOAD helpers like
LD_PRELOAD=liblttng-ust-pthread-wrapper.so.1:liblttng-ust-fd.so.1:liblttng-ust-fork.so.1
and I'm out of options that I know to try.

Looking for pointers on what else I can try and what I could be doing wrong.

I had this same segfault issue using Dunfell version of Yocto and earlier
versions of lttng (2.11) but never dug into it because was planning on
upgrading everything soon, so recently stepped up to newer versions of
everything to see if that solved the issue but apparently not.

Regards,

Brian

[-- Attachment #1.2: Type: text/html, Size: 20255 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 16961 bytes --]

On Wed, Jul 10, 2024 at 9:53 PM Brian Hutchinson <b.hutchman@gmail.com>
wrote:

> Hi,
>
> I'm getting a weird segfault when I compile in a lttng-ust tracepoint into
> my application.
>
> I'm able to run the lttng-ust hello_world example just fine on my platform
> which one would think should prove everything out, but when I use the exact
> same hello world trace provider and link it into my multi-threadded app and
> simply call:
>
> lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23,                         "hi there!");
>
> I get a segfault.
>
> Background.  I'm on aarch64 platform (imx8mm-lpddr4-evk) with sdk (gcc
> 11.4 glibc 2.35) and rootfs generated from Yocto Kirkstone 4.0.18.  Built
> lttng-tools (2.13.9) and lttng-ust (2.13.6) from yocto recipe.  Built
> out-of-tree kernel modules for lttng-modules (2.13.9) and I'm on kernel
> 6.1.38.
>
> If I do a backtrace with gdb on the core this is what I see:
>
> GNU gdb (GDB) 11.2
> Copyright (C) 2022 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "aarch64-poky-linux".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <https://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
>    <http://www.gnu.org/software/gdb/documentation/>.
>
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from ./my_app...
>
> warning: core file may not match specified executable file.
> [New LWP 757]
> [New LWP 759]
> [New LWP 758]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/libthread_db.so.1".
> Core was generated by `./my_app'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> 96      ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or
> directory.
> [Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))]
> (gdb) bt
> #0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> #1  0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register () from
> /usr/lib/liblttng-ust-tracepoint.so.1
> #2  0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at
> /opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629
>
> #3  0x0000ffffb872b30c in call_init (env=<optimized out>,
> argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145
> #4  __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1,
> argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>,
> rtld_fini=<optimized out>, stack_end=<optimized out>) at
> ../csu/libc-start.c:376
> #5  0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81
> (gdb)
>
> If I run valgrind, this is what I see:
>
> ==627== Memcheck, a memory error detector
> ==627== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==627== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
> ==627== Command: ./my_app
> ==627==
> LTTng-UST: Error while registering tracepoint probe.
> ==627==
> ==627== Process terminating with default action of signal 6 (SIGABRT)
> ==627==    at 0x6772278: __pthread_kill_implementation (pthread_kill.c:44)
> ==627==    by 0x672DD4F: raise (raise.c:26)
> ==627==    by 0x671AEE3: abort (abort.c:79)
> ==627==    by 0x2C572B: lttng_ust__events_init__hello_world
> (ust-tracepoint-event.h:1221)
> ==627==    by 0x2C5B97: lttng_ust_constructor_hello_world
> (ust-tracepoint-event.h:1239)
> ==627==    by 0x671B30B: call_init (libc-start.c:145)
> ==627==    by 0x671B30B: __libc_start_main@@GLIBC_2.34 (libc-start.c:376)
> ==627==    by 0x21122F: (below main) (start.S:81)
> ==627==
> ==627== HEAP SUMMARY:
> ==627==     in use at exit: 151,712 bytes in 90 blocks
> ==627==   total heap usage: 1,184 allocs, 1,094 frees, 202,570 bytes
> allocated
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 1 of 17
> ==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627==    by 0x68D50CB: lttng_ust_tracepoint_module_register (in
> /usr/lib/liblttng-ust-tracepoint.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 2 of 17
> ==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627==    by 0x61E8143: lttng_ust_probe_register (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x61FD49B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 3 of 17
> ==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627==    by 0x61E8143: lttng_ust_probe_register (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x61DFF4F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 4 of 17
> ==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627==    by 0x61E8143: lttng_ust_probe_register (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x61E026F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 5 of 17
> ==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627==    by 0x61E8143: lttng_ust_probe_register (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x61E058F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 48 bytes in 1 blocks are still reachable in loss record 6 of 17
> ==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627==    by 0x68D50CB: lttng_ust_tracepoint_module_register (in
> /usr/lib/liblttng-ust-tracepoint.so.1.0.0)
> ==627==    by 0x2C5A17: lttng_ust__tracepoints__ptrs_init
> (tracepoint.h:629)
> ==627==    by 0x671B30B: call_init (libc-start.c:145)
> ==627==    by 0x671B30B: __libc_start_main@@GLIBC_2.34 (libc-start.c:376)
> ==627==    by 0x21122F: (below main) (start.S:81)
> ==627==
> ==627== 128 bytes in 1 blocks are still reachable in loss record 7 of 17
> ==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627==    by 0x68A388B: ??? (in /usr/lib/liblttng-ust-common.so.1.0.0)
> ==627==    by 0x68A2FBF: lttng_ust_common_ctor (in
> /usr/lib/liblttng-ust-common.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 200 bytes in 1 blocks are still reachable in loss record 8 of 17
> ==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627==    by 0x5929E37: malloc (rtld-malloc.h:56)
> ==627==    by 0x5929E37: add_to_global_resize (dl-open.c:152)
> ==627==    by 0x592AA8F: dl_open_worker_begin (dl-open.c:716)
> ==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627==    by 0x5929C77: dl_open_worker (dl-open.c:782)
> ==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627==    by 0x592A07B: _dl_open (dl-open.c:884)
> ==627==    by 0x676C467: dlopen_doit (dlopen.c:56)
> ==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627==    by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
> ==627==    by 0x676BF4F: _dlerror_run (dlerror.c:138)
> ==627==    by 0x676C52B: dlopen_implementation (dlopen.c:71)
> ==627==    by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
> ==627==
> ==627== 240 bytes in 2 blocks are still reachable in loss record 9 of 17
> ==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627==    by 0x59214C7: malloc (rtld-malloc.h:56)
> ==627==    by 0x59214C7: _dl_map_object_deps (dl-deps.c:479)
> ==627==    by 0x592A50B: dl_open_worker_begin (dl-open.c:592)
> ==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627==    by 0x5929C77: dl_open_worker (dl-open.c:782)
> ==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627==    by 0x592A07B: _dl_open (dl-open.c:884)
> ==627==    by 0x676C467: dlopen_doit (dlopen.c:56)
> ==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627==    by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
> ==627==    by 0x676BF4F: _dlerror_run (dlerror.c:138)
> ==627==    by 0x676C52B: dlopen_implementation (dlopen.c:71)
> ==627==    by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
> ==627==
> ==627== 292 bytes in 15 blocks are possibly lost in loss record 10 of 17
> ==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627==    by 0x6783993: strdup (strdup.c:42)
> ==627==    by 0x61FD92F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
> ==627==    by 0x61FE653: lttng_ust_dl_update (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 336 bytes in 1 blocks are possibly lost in loss record 11 of 17
> ==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627==    by 0x592ECB7: calloc (rtld-malloc.h:44)
> ==627==    by 0x592ECB7: allocate_dtv (dl-tls.c:376)
> ==627==    by 0x592F723: _dl_allocate_tls (dl-tls.c:635)
> ==627==    by 0x677114F: allocate_stack (allocatestack.c:428)
> ==627==    by 0x677114F: pthread_create@@GLIBC_2.34
> (pthread_create.c:647)
> ==627==    by 0x61DEF03: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 336 bytes in 1 blocks are possibly lost in loss record 12 of 17
> ==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627==    by 0x592ECB7: calloc (rtld-malloc.h:44)
> ==627==    by 0x592ECB7: allocate_dtv (dl-tls.c:376)
> ==627==    by 0x592F723: _dl_allocate_tls (dl-tls.c:635)
> ==627==    by 0x677114F: allocate_stack (allocatestack.c:428)
> ==627==    by 0x677114F: pthread_create@@GLIBC_2.34
> (pthread_create.c:647)
> ==627==    by 0x61DEEB7: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 340 bytes in 17 blocks are possibly lost in loss record 13 of 17
> ==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627==    by 0x61FD94F: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
> ==627==    by 0x61FE653: lttng_ust_dl_update (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 1,248 bytes in 26 blocks are still reachable in loss record 14 of
> 17
> ==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627==    by 0x68D529F: lttng_ust_tracepoint_module_register (in
> /usr/lib/liblttng-ust-tracepoint.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 4,608 bytes in 2 blocks are possibly lost in loss record 15 of 17
> ==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627==    by 0x5922FFB: malloc (rtld-malloc.h:56)
> ==627==    by 0x5922FFB: _dlfo_mappings_segment_allocate
> (dl-find_object.c:217)
> ==627==    by 0x5922FFB: _dl_find_object_update_1 (dl-find_object.c:671)
> ==627==    by 0x5922FFB: _dl_find_object_update (dl-find_object.c:805)
> ==627==    by 0x592A803: dl_open_worker_begin (dl-open.c:735)
> ==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627==    by 0x5929C77: dl_open_worker (dl-open.c:782)
> ==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627==    by 0x592A07B: _dl_open (dl-open.c:884)
> ==627==    by 0x676C467: dlopen_doit (dlopen.c:56)
> ==627==    by 0x681D877: _dl_catch_exception (dl-error-skeleton.c:208)
> ==627==    by 0x681D92F: _dl_catch_error (dl-error-skeleton.c:227)
> ==627==    by 0x676BF4F: _dlerror_run (dlerror.c:138)
> ==627==    by 0x676C52B: dlopen_implementation (dlopen.c:71)
> ==627==    by 0x676C52B: dlopen@@GLIBC_2.34 (dlopen.c:81)
> ==627==
> ==627== 70,992 bytes in 17 blocks are possibly lost in loss record 16 of
> 17
> ==627==    at 0x618A790: calloc (vg_replace_malloc.c:1328)
> ==627==    by 0x61FD91B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x681DA9F: dl_iterate_phdr (dl-iteratephdr.c:74)
> ==627==    by 0x61FE653: lttng_ust_dl_update (in
> /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x61DE83B: ??? (in /usr/lib/liblttng-ust.so.1.0.0)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== 72,704 bytes in 1 blocks are still reachable in loss record 17 of
> 17
> ==627==    at 0x618551C: malloc (vg_replace_malloc.c:381)
> ==627==    by 0x653E137: ??? (in /usr/lib/libstdc++.so.6.0.29)
> ==627==    by 0x5923653: call_init.part.0 (dl-init.c:74)
> ==627==    by 0x5923767: call_init (dl-init.c:29)
> ==627==    by 0x5923767: _dl_init (dl-init.c:121)
> ==627==    by 0x5935D37: ??? (in /lib/ld-linux-aarch64.so.1)
> ==627==
> ==627== LEAK SUMMARY:
> ==627==    definitely lost: 0 bytes in 0 blocks
> ==627==    indirectly lost: 0 bytes in 0 blocks
> ==627==      possibly lost: 76,904 bytes in 53 blocks
> ==627==    still reachable: 74,808 bytes in 37 blocks
> ==627==         suppressed: 0 bytes in 0 blocks
> ==627==
> ==627== For lists of detected and suppressed errors, rerun with: -s
> ==627== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 0 from 0)
> Aborted
>
> I'm sure I'm probably doing something wrong but I've tried static linking,
> dynamic linking using LD_PRELOAD helpers like LD_PRELOAD=liblttng-ust-pthread-wrapper.so.1:liblttng-ust-fd.so.1:liblttng-ust-fork.so.1
> and I'm out of options that I know to try.
>
> Looking for pointers on what else I can try and what I could be doing
> wrong.
>
> I had this same segfault issue using Dunfell version of Yocto and earlier
> versions of lttng (2.11) but never dug into it because was planning on
> upgrading everything soon, so recently stepped up to newer versions of
> everything to see if that solved the issue but apparently not.
>
>
I'm not sure about this, still trying to comprehend everything, but I think
my issue could be related to all this gcc TLS, heap, trampoline talk here:

https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html

Regards,

Brian

[-- Attachment #1.2: Type: text/html, Size: 20344 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Kienan Stewart via lttng-dev]

Hi Brian,

On 7/11/24 8:33 AM, Brian Hutchinson via lttng-dev wrote:
 > ...
 >
 > I'm not sure about this, still trying to comprehend everything, but I
 > think my issue could be related to all this gcc TLS, heap, trampoline
 > talk here:
 >
 > https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html
 > <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>
 >
 > Regards,
 >
 > Brian
 >

You could try using lttng-ust 2.13.8, which includes a fix for the 
discussion you referenced.

https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836

Please let us know if it doesn't solve it.

thanks,
kienan
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Kienan Stewart via lttng-dev]

Hi Brian,

On 7/11/24 7:20 PM, Brian Hutchinson wrote:
> Hi Kienan
> 
> Do I need to update lttng-tools and modules too?  I did a bbappend and 
> pulled in 2.13.8 recipe from master and get the same issue.  I'm double 
> checking to make sure everything built right.  Had to update sdk, rootfs 
> and rebuild app with new sdk etc.

I don't believe so.

Do you have a minimal reproducer example that you are able to share 
including the build instructions you are using?

Also, which version of urcu is being used in your environment?

thanks,
kienan

> 
> Regards,
> 
> Brian
> 
> On Thu, Jul 11, 2024 at 8:38 AM Kienan Stewart <kstewart@efficios.com 
> <mailto:kstewart@efficios.com>> wrote:
> 
>     Hi Brian,
> 
>     On 7/11/24 8:33 AM, Brian Hutchinson via lttng-dev wrote:
>       > ...
>       >
>       > I'm not sure about this, still trying to comprehend everything,
>     but I
>       > think my issue could be related to all this gcc TLS, heap,
>     trampoline
>       > talk here:
>       >
>       >
>     https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html
>     <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>
>       >
>     <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>>
>       >
>       > Regards,
>       >
>       > Brian
>       >
> 
>     You could try using lttng-ust 2.13.8, which includes a fix for the
>     discussion you referenced.
> 
>     https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836 <https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836>
> 
>     Please let us know if it doesn't solve it.
> 
>     thanks,
>     kienan
> 
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Kienan Stewart via lttng-dev]

Hi Brian,

On 7/12/24 2:56 PM, Brian Hutchinson wrote:
> Hi again Kienan,
> 
> Manifest says librcu is 0.13.2.

thanks

> 
> The app is made up of a bunch of sup projects (about 40 or so) that are 
> built into a bunch of .o objects (I think it's a structure automatically 
> setup by cmake) and finally linked together with the main app .o object 
> so it looks like it's all static linking.
> 
> The part where I put in a call to lttng-ust trace is in one of the sub 
> projects.  Looking at all of your example linking scenarios in the 
> documentation, I think what we are doing fits the same scenario as the 
> hello world build/link.
> 
> All of the sub project .o's are linked with the main app's .o and linked 
> with lttng-ust and ld at the end.
> 
> I've also tried building the trace provider bits linked with lttng-ust 
> and dl as a .so and using LD_PRELOAD to load that before starting the 
> app.  I get same segfault that way too.
> 
> Stupid question.  Do I need to create a lttng session or something 
> before the app with lttng-ust tracepoints even runs?  Reason I ask is, 
> systemd starts up several other apps that setup and do things that the 
> app in question needs before it can run so I was hoping the app could 
> start then I create a session and attach to the pid to do tracing.  I 
> can't really start it like the hello world where is runs and pauses 
> while you wait for a key press and then setup and start all the lttng 
> session and recording bits, then hit a key and the tracepoints are 
> exercised.  From all I've read, an application with lttng-ust 
> tracepoints should run as a normal program and the tracing lines do 
> nothing until a lttng session is started and the tracepoints enabled.
> 

It's not necessary to create a session, or even have the sessiond 
running when a traced application is launched.

When an application that is compiled with user-space tracing, there's 
are early constructors that run prior to the application's main entry 
point. By default this waits for up to 3000ms to perform registration 
with a sessiond before continuing to the program's main entry point. 
This delay is controlled by the environment variable 
LTTNG_UST_REGISTER_TIMEOUT ( see 
https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables ).

The UST application also opens a piece of shared memory at a well known 
location and reads it waiting for changes. The sessiond will write to 
the shared memory at the location when it starts to signal to any UST 
apps that registration can be retried. C.f. 
https://github.com/lttng/lttng-ust/blob/9b9714603cfa96a63a42dc9cb8266f81df42a4f3/src/lib/lttng-ust/lttng-ust-comm.c#L1978

> Thought I'd ask in case my understanding is wrong and how I'm starting 
> things up is the problem.
> 
> I'll work on fixing the 2.13.8 lttng-ust built (as I'm not convinced yet 
> it's right based on what I mentioned before) and let you know how it goes.
> 

If you're not sure re: size, you could always compute a sha512 sum for 
the libraries.


thanks,
kienan

P.S. Could you please keep lttng-dev in CC? thank you.

> Thanks and have a good weekend!
> 
> Regards,
> 
> Brian
> 
> On Fri, Jul 12, 2024 at 10:30 AM Brian Hutchinson <b.hutchman@gmail.com 
> <mailto:b.hutchman@gmail.com>> wrote:
> 
>     No, it's a giant app.   But made what I'm trying to test as simple
>     as possible to demonstrate the error ... its the same trace provider
>     that worked with hello world and my app is just calling a line of that.
> 
>     I don't think I did something right with my 2.13.8 build.  Bitbake
>     made a package that says it's 2.13.8 but I compared the files to
>     2.13.6 and the file sizes were all identical so somethings not
>     right.   So I need to figure out why recipe isn't working.
> 
>     I'm off work today but when I get a chance I'll check manifest and
>     get you the rcu version and try to describe how app is built.
> 
>     Thanks,
> 
>     Brian
> 
>     On Fri, Jul 12, 2024, 10:17 AM Kienan Stewart <kstewart@efficios.com
>     <mailto:kstewart@efficios.com>> wrote:
> 
>         Hi Brian,
> 
>         On 7/11/24 7:20 PM, Brian Hutchinson wrote:
>          > Hi Kienan
>          >
>          > Do I need to update lttng-tools and modules too?  I did a
>         bbappend and
>          > pulled in 2.13.8 recipe from master and get the same issue. 
>         I'm double
>          > checking to make sure everything built right.  Had to update
>         sdk, rootfs
>          > and rebuild app with new sdk etc.
> 
>         I don't believe so.
> 
>         Do you have a minimal reproducer example that you are able to share
>         including the build instructions you are using?
> 
>         Also, which version of urcu is being used in your environment?
> 
>         thanks,
>         kienan
> 
>          >
>          > Regards,
>          >
>          > Brian
>          >
>          > On Thu, Jul 11, 2024 at 8:38 AM Kienan Stewart
>         <kstewart@efficios.com <mailto:kstewart@efficios.com>
>          > <mailto:kstewart@efficios.com
>         <mailto:kstewart@efficios.com>>> wrote:
>          >
>          >     Hi Brian,
>          >
>          >     On 7/11/24 8:33 AM, Brian Hutchinson via lttng-dev wrote:
>          >       > ...
>          >       >
>          >       > I'm not sure about this, still trying to comprehend
>         everything,
>          >     but I
>          >       > think my issue could be related to all this gcc TLS,
>         heap,
>          >     trampoline
>          >       > talk here:
>          >       >
>          >       >
>          >
>         https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>
>          >   
>           <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>>
>          >       >
>          >   
>           <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html> <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>>>
>          >       >
>          >       > Regards,
>          >       >
>          >       > Brian
>          >       >
>          >
>          >     You could try using lttng-ust 2.13.8, which includes a
>         fix for the
>          >     discussion you referenced.
>          >
>          >
>         https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836 <https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836> <https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836 <https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836>>
>          >
>          >     Please let us know if it doesn't solve it.
>          >
>          >     thanks,
>          >     kienan
>          >
> 
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 4244 bytes --]

Hi Kienan,

On Thu, Jul 11, 2024 at 8:38 AM Kienan Stewart <kstewart@efficios.com>
wrote:

> Hi Brian,
>
> On 7/11/24 8:33 AM, Brian Hutchinson via lttng-dev wrote:
>  > ...
>  >
>  > I'm not sure about this, still trying to comprehend everything, but I
>  > think my issue could be related to all this gcc TLS, heap, trampoline
>  > talk here:
>  >
>  > https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html
>  > <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>
>  >
>  > Regards,
>  >
>  > Brian
>  >
>
> You could try using lttng-ust 2.13.8, which includes a fix for the
> discussion you referenced.
>
>
> https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836
>
> Please let us know if it doesn't solve it.
>
> thanks,
> kienan
>

I updated to lttng-ust 2.13.8 as you suggested.  At first, I pulled in the
recipe from master branch of Yocto, but had the same problem as before
after rebuilding.

Next, I downloaded the 2.13.8 lttng-ust tarball from the lttng site and
setup a native build environment on my target and built lttng-ust 2.13.8
and my app as a second independent build method for sanity check.  I can
report that I get the exact same result as before this way too.

In both cases, I verified that I had the commit you referenced above.

I did a backtrace of the three threads captured in the coredump and show
them below in hopes it will shed light on what I need to try next.

(gdb) thread apply all backtrace

Thread 3 (Thread 0xffffbe7de7e0 (LWP 559)):
#0  syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
#1  0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0, timeout=0x0,
val=0, op=0, uaddr=0xffffbefd8000) at ../../../src/lib/lttng-ust/futex.h:56
#2  lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0,
uaddr=0xffffbefd8000) at ../../../src/lib/lttng-ust/futex.h:88
#3  wait_for_sessiond (sock_info=0xffffbef6b638 <global_apps>) at
lttng-ust-comm.c:1761
#4  ust_listener_thread (arg=0xffffbef6b638 <global_apps>) at
lttng-ust-comm.c:1832
#5  0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at
pthread_create.c:442
#6  0x0000ffffbe98801c in thread_start () at
../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 2 (Thread 0xffffbdfce7e0 (LWP 560)):
#0  syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
#1  0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0, timeout=0x0,
val=0, op=0, uaddr=0xffffbefd7000) at ../../../src/lib/lttng-ust/futex.h:56
#2  lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0,
uaddr=0xffffbefd7000) at ../../../src/lib/lttng-ust/futex.h:88
#3  wait_for_sessiond (sock_info=0xffffbef695e0 <local_apps>) at
lttng-ust-comm.c:1761
#4  ust_listener_thread (arg=0xffffbef695e0 <local_apps>) at
lttng-ust-comm.c:1832
#5  0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at
pthread_create.c:442
#6  0x0000ffffbe98801c in thread_start () at
../sysdeps/unix/sysv/linux/aarch64/clone.S:79

Thread 1 (Thread 0xffffbefcf460 (LWP 545)):
#0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
#1  0x0000ffffbe834bd0 in lttng_ust_tp_validate_event_name
(tp=0xaaaab8f14de0
<lttng_ust_tracepoint_hello_world___my_first_tracepoint>, tp=0xaaaab8f14de0
<lttng_ust_tracepoint_hello_world___my_first_tracepoint>) at
tracepoint.c:143
#2  add_callsite (tp=0xaaaab8f14de0
<lttng_ust_tracepoint_hello_world___my_first_tracepoint>,
lib=0xaaaae74fad30) at tracepoint.c:412
#3  lib_register_callsites (lib=0xaaaae74fad30) at tracepoint.c:552
#4  lttng_ust_tracepoint_module_register (tracepoints_start=<optimized
out>, tracepoints_count=<optimized out>) at tracepoint.c:956
#5  0x0000aaaab8758204 in lttng_ust.tracepoints.ptrs_init ()
#6  0x0000ffffbe8cb30c in call_init (env=<optimized out>,
argv=0xffffc1bd8c88, argc=1) at ../csu/libc-start.c:145
#7  __libc_start_main_impl (main=0xaaaab867aabc <main>, argc=1,
argv=0xffffc1bd8c88, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=<optimized out>) at
../csu/libc-start.c:376
#8  0x0000aaaab867a7f0 in _start () at ../sysdeps/aarch64/start.S:81

Regards,

Brian

[-- Attachment #1.2: Type: text/html, Size: 5738 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Kienan Stewart via lttng-dev]

Hi Brian,

thanks! I'll take a look at these and see if I can get any ideas from them.

To go back to your original e-mail, a detail that I hadn't noticed in 
your original backtrace was pointed out to me and I'd like to verify 
with you:

```
Core was generated by `./my_app'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
96      ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or 
directory.
[Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))]
(gdb) bt
#0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
#1  0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register () from 
/usr/lib/liblttng-ust-tracepoint.so.1
#2  0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at 
/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629
#3  0x0000ffffb872b30c in call_init (env=<optimized out>, 
argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145
#4  __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1, 
argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>, 
rtld_fini=<optimized out>, stack_end=<optimized out>) at 
../csu/libc-start.c:376
#5  0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81
```

The library loaded at frame 1 is 
`/usr/lib/liblttng-ust-tracepoint.so.1`; however, in frame 2 the 
reference is from 
`/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h`.

Are the LTTng libraries in `/usr/lib` the exact same version as the 
headers in 
`/opt/poky/4.0.18/sysroots/cortexa53-cryptop-poky-linux/usr/include/lttng`?

If some of your app is compiled using one version of headers and a 
different version of the library is loaded at runtime, there could be an 
ABI mismatch.

You could check what the include paths are during compilation and 
`LD_LIBRARY_PATH` at runtime. Running with the environment variable 
`LD_DEBUG=bindings,libs` (see `man ld.so` for more info).

In any case, I'll take a look over the most recent backtraces and see if 
anything else jumps out.

thanks,
kienan


On 7/16/24 9:27 PM, Brian Hutchinson wrote:
> Hi Kienan,
> 
> On Thu, Jul 11, 2024 at 8:38 AM Kienan Stewart <kstewart@efficios.com 
> <mailto:kstewart@efficios.com>> wrote:
> 
>     Hi Brian,
> 
>     On 7/11/24 8:33 AM, Brian Hutchinson via lttng-dev wrote:
>       > ...
>       >
>       > I'm not sure about this, still trying to comprehend everything,
>     but I
>       > think my issue could be related to all this gcc TLS, heap,
>     trampoline
>       > talk here:
>       >
>       >
>     https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html
>     <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>
>       >
>     <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html <https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>>
>       >
>       > Regards,
>       >
>       > Brian
>       >
> 
>     You could try using lttng-ust 2.13.8, which includes a fix for the
>     discussion you referenced.
> 
>     https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836 <https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836>
> 
>     Please let us know if it doesn't solve it.
> 
>     thanks,
>     kienan
> 
> 
> I updated to lttng-ust 2.13.8 as you suggested.  At first, I pulled in 
> the recipe from master branch of Yocto, but had the same problem as 
> before after rebuilding.
> 
> Next, I downloaded the 2.13.8 lttng-ust tarball from the lttng site and 
> setup a native build environment on my target and built lttng-ust 2.13.8 
> and my app as a second independent build method for sanity check.  I can 
> report that I get the exact same result as before this way too.
> 
> In both cases, I verified that I had the commit you referenced above.
> 
> I did a backtrace of the three threads captured in the coredump and show 
> them below in hopes it will shed light on what I need to try next.
> 
> (gdb) thread apply all backtrace
> 
> Thread 3 (Thread 0xffffbe7de7e0 (LWP 559)):
> #0  syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
> #1  0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0, 
> timeout=0x0, val=0, op=0, uaddr=0xffffbefd8000) at 
> ../../../src/lib/lttng-ust/futex.h:56
> #2  lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0, 
> uaddr=0xffffbefd8000) at ../../../src/lib/lttng-ust/futex.h:88
> #3  wait_for_sessiond (sock_info=0xffffbef6b638 <global_apps>) at 
> lttng-ust-comm.c:1761
> #4  ust_listener_thread (arg=0xffffbef6b638 <global_apps>) at 
> lttng-ust-comm.c:1832
> #5  0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at 
> pthread_create.c:442
> #6  0x0000ffffbe98801c in thread_start () at 
> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
> 
> Thread 2 (Thread 0xffffbdfce7e0 (LWP 560)):
> #0  syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
> #1  0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0, 
> timeout=0x0, val=0, op=0, uaddr=0xffffbefd7000) at 
> ../../../src/lib/lttng-ust/futex.h:56
> #2  lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0, 
> uaddr=0xffffbefd7000) at ../../../src/lib/lttng-ust/futex.h:88
> #3  wait_for_sessiond (sock_info=0xffffbef695e0 <local_apps>) at 
> lttng-ust-comm.c:1761
> #4  ust_listener_thread (arg=0xffffbef695e0 <local_apps>) at 
> lttng-ust-comm.c:1832
> #5  0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at 
> pthread_create.c:442
> #6  0x0000ffffbe98801c in thread_start () at 
> ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
> 
> Thread 1 (Thread 0xffffbefcf460 (LWP 545)):
> #0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> #1  0x0000ffffbe834bd0 in lttng_ust_tp_validate_event_name 
> (tp=0xaaaab8f14de0 
> <lttng_ust_tracepoint_hello_world___my_first_tracepoint>, 
> tp=0xaaaab8f14de0 
> <lttng_ust_tracepoint_hello_world___my_first_tracepoint>) at 
> tracepoint.c:143
> #2  add_callsite (tp=0xaaaab8f14de0 
> <lttng_ust_tracepoint_hello_world___my_first_tracepoint>, 
> lib=0xaaaae74fad30) at tracepoint.c:412
> #3  lib_register_callsites (lib=0xaaaae74fad30) at tracepoint.c:552
> #4  lttng_ust_tracepoint_module_register (tracepoints_start=<optimized 
> out>, tracepoints_count=<optimized out>) at tracepoint.c:956
> #5  0x0000aaaab8758204 in lttng_ust.tracepoints.ptrs_init ()
> #6  0x0000ffffbe8cb30c in call_init (env=<optimized out>, 
> argv=0xffffc1bd8c88, argc=1) at ../csu/libc-start.c:145
> #7  __libc_start_main_impl (main=0xaaaab867aabc <main>, argc=1, 
> argv=0xffffc1bd8c88, init=<optimized out>, fini=<optimized out>, 
> rtld_fini=<optimized out>, stack_end=<optimized out>) at 
> ../csu/libc-start.c:376
> #8  0x0000aaaab867a7f0 in _start () at ../sysdeps/aarch64/start.S:81
> 
> Regards,
> 
> Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 10420 bytes --]

On Wed, Jul 17, 2024 at 10:13 AM Kienan Stewart <kstewart@efficios.com>
wrote:

> Hi Brian,
>
> thanks! I'll take a look at these and see if I can get any ideas from them.
>
> To go back to your original e-mail, a detail that I hadn't noticed in
> your original backtrace was pointed out to me and I'd like to verify
> with you:
>
> ```
> Core was generated by `./my_app'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> 96      ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or
> directory.
> [Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))]
> (gdb) bt
> #0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> #1  0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register () from
> /usr/lib/liblttng-ust-tracepoint.so.1
> #2  0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at
>
> /opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629
> #3  0x0000ffffb872b30c in call_init (env=<optimized out>,
> argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145
> #4  __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1,
> argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>,
> rtld_fini=<optimized out>, stack_end=<optimized out>) at
> ../csu/libc-start.c:376
> #5  0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81
> ```
>
> The library loaded at frame 1 is
> `/usr/lib/liblttng-ust-tracepoint.so.1`; however, in frame 2 the
> reference is from
>
> `/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h`.


> Are the LTTng libraries in `/usr/lib` the exact same version as the
> headers in
> `/opt/poky/4.0.18/sysroots/cortexa53-cryptop-poky-linux/usr/include/lttng`?
>

Yes.  The sdk was created by yocto using populate_sdk which includes the
target image sysroot in the sdk.  Hopefully I explained that right. The
target image drives the contents of the sdk and the generated sdk is what
is used to cross compile the application specifically for the target we are
running on (NXP imx8mm).


> If some of your app is compiled using one version of headers and a
> different version of the library is loaded at runtime, there could be an
> ABI mismatch.
>

> You could check what the include paths are during compilation and
> `LD_LIBRARY_PATH` at runtime. Running with the environment variable
> `LD_DEBUG=bindings,libs` (see `man ld.so` for more info).
>

I've been down that road a few times before with the same line of thinking
(ABI issue) and believe I've proven I don't have those issues.  That's one
of the reasons I hand built lttng-ust in native environment (on the actual
target) to verify it wasn't an issue with a yocto recipe, cross compiling
etc..

I've been thru this process with different OS versions (yocto Dunfell
before), kernel version, toolchain version etc., and now the same with
Kirkstone based components.


> In any case, I'll take a look over the most recent backtraces and see if
> anything else jumps out.
>

Thanks!  I'm running out of things to try.

I don't fully understand the implications of some of this documentation,
but I have learned enough to know LD_PRELOAD means nothing if everything is
static built.  Our application is heavily multi threaded, uses fork, clone
and who knows if it's doing double closes or not ... so I've tried these
LD_PRELOAD "helpers" or whatever they are, before and didn't get a
different result, but now know it's only for shared library object.  So now
I will experiment with making our tpp a shared object and try LD_PRELOAD of
fork, fd and pthread helpers and starting my app that way.

This is what I'm referring to above from the lttng docs:
Use LTTng-UST with daemons
<https://lttng.org/docs/v2.13/#doc-using-lttng-ust-with-daemons>

If your instrumented application calls fork(2)
<https://man7.org/linux/man-pages/man2/fork.2.html>, clone(2)
<https://man7.org/linux/man-pages/man2/clone.2.html>, or BSD’s rfork(2)
<http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>,
without a following exec(3)
<https://man7.org/linux/man-pages/man3/exec.3.html>-family system call, you
must preload the liblttng-ust-fork.so shared object when you start the
application.

LD_PRELOAD=liblttng-ust-fork.so ./my-app

If your tracepoint provider package is a shared library which you also
preload, you must put both shared objects in LD_PRELOAD:

LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app

Use LTTng-UST with applications which close file descriptors that don’t
belong to them <https://lttng.org/docs/v2.13/#doc-liblttng-ust-fd>
Since 2.9

If your instrumented application closes one or more file descriptors which
it did not open itself, you must preload the liblttng-ust-fd.so shared
object when you start the application:

LD_PRELOAD=liblttng-ust-fd.so ./my-app

Typical use cases include closing all the file descriptors after fork(2)
<https://man7.org/linux/man-pages/man2/fork.2.html> or rfork(2)
<http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>
and buggy applications doing “double closes”.


Thanks,

Brian


>
>
> On 7/16/24 9:27 PM, Brian Hutchinson wrote:
> > Hi Kienan,
> >
> > On Thu, Jul 11, 2024 at 8:38 AM Kienan Stewart <kstewart@efficios.com
> > <mailto:kstewart@efficios.com>> wrote:
> >
> >     Hi Brian,
> >
> >     On 7/11/24 8:33 AM, Brian Hutchinson via lttng-dev wrote:
> >       > ...
> >       >
> >       > I'm not sure about this, still trying to comprehend everything,
> >     but I
> >       > think my issue could be related to all this gcc TLS, heap,
> >     trampoline
> >       > talk here:
> >       >
> >       >
> >     https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html
> >     <
> https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>
> >       >
> >     <
> https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html <
> https://www.mail-archive.com/lttng-dev@lists.lttng.org/msg13584.html>>
> >       >
> >       > Regards,
> >       >
> >       > Brian
> >       >
> >
> >     You could try using lttng-ust 2.13.8, which includes a fix for the
> >     discussion you referenced.
> >
> >
> https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836
> <
> https://github.com/lttng/lttng-ust/commit/5db715e800d9b721f48495a987cd26c25965e836
> >
> >
> >     Please let us know if it doesn't solve it.
> >
> >     thanks,
> >     kienan
> >
> >
> > I updated to lttng-ust 2.13.8 as you suggested.  At first, I pulled in
> > the recipe from master branch of Yocto, but had the same problem as
> > before after rebuilding.
> >
> > Next, I downloaded the 2.13.8 lttng-ust tarball from the lttng site and
> > setup a native build environment on my target and built lttng-ust 2.13.8
> > and my app as a second independent build method for sanity check.  I can
> > report that I get the exact same result as before this way too.
> >
> > In both cases, I verified that I had the commit you referenced above.
> >
> > I did a backtrace of the three threads captured in the coredump and show
> > them below in hopes it will shed light on what I need to try next.
> >
> > (gdb) thread apply all backtrace
> >
> > Thread 3 (Thread 0xffffbe7de7e0 (LWP 559)):
> > #0  syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
> > #1  0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0,
> > timeout=0x0, val=0, op=0, uaddr=0xffffbefd8000) at
> > ../../../src/lib/lttng-ust/futex.h:56
> > #2  lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0,
> > uaddr=0xffffbefd8000) at ../../../src/lib/lttng-ust/futex.h:88
> > #3  wait_for_sessiond (sock_info=0xffffbef6b638 <global_apps>) at
> > lttng-ust-comm.c:1761
> > #4  ust_listener_thread (arg=0xffffbef6b638 <global_apps>) at
> > lttng-ust-comm.c:1832
> > #5  0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at
> > pthread_create.c:442
> > #6  0x0000ffffbe98801c in thread_start () at
> > ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
> >
> > Thread 2 (Thread 0xffffbdfce7e0 (LWP 560)):
> > #0  syscall () at ../sysdeps/unix/sysv/linux/aarch64/syscall.S:38
> > #1  0x0000ffffbeef1bb8 in lttng_ust_futex (val3=0, uaddr2=0x0,
> > timeout=0x0, val=0, op=0, uaddr=0xffffbefd7000) at
> > ../../../src/lib/lttng-ust/futex.h:56
> > #2  lttng_ust_futex_async (op=0, val=0, timeout=0x0, uaddr2=0x0, val3=0,
> > uaddr=0xffffbefd7000) at ../../../src/lib/lttng-ust/futex.h:88
> > #3  wait_for_sessiond (sock_info=0xffffbef695e0 <local_apps>) at
> > lttng-ust-comm.c:1761
> > #4  ust_listener_thread (arg=0xffffbef695e0 <local_apps>) at
> > lttng-ust-comm.c:1832
> > #5  0x0000ffffbe9206a8 in start_thread (arg=0xffffc1bd8687) at
> > pthread_create.c:442
> > #6  0x0000ffffbe98801c in thread_start () at
> > ../sysdeps/unix/sysv/linux/aarch64/clone.S:79
> >
> > Thread 1 (Thread 0xffffbefcf460 (LWP 545)):
> > #0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> > #1  0x0000ffffbe834bd0 in lttng_ust_tp_validate_event_name
> > (tp=0xaaaab8f14de0
> > <lttng_ust_tracepoint_hello_world___my_first_tracepoint>,
> > tp=0xaaaab8f14de0
> > <lttng_ust_tracepoint_hello_world___my_first_tracepoint>) at
> > tracepoint.c:143
> > #2  add_callsite (tp=0xaaaab8f14de0
> > <lttng_ust_tracepoint_hello_world___my_first_tracepoint>,
> > lib=0xaaaae74fad30) at tracepoint.c:412
> > #3  lib_register_callsites (lib=0xaaaae74fad30) at tracepoint.c:552
> > #4  lttng_ust_tracepoint_module_register (tracepoints_start=<optimized
> > out>, tracepoints_count=<optimized out>) at tracepoint.c:956
> > #5  0x0000aaaab8758204 in lttng_ust.tracepoints.ptrs_init ()
> > #6  0x0000ffffbe8cb30c in call_init (env=<optimized out>,
> > argv=0xffffc1bd8c88, argc=1) at ../csu/libc-start.c:145
> > #7  __libc_start_main_impl (main=0xaaaab867aabc <main>, argc=1,
> > argv=0xffffc1bd8c88, init=<optimized out>, fini=<optimized out>,
> > rtld_fini=<optimized out>, stack_end=<optimized out>) at
> > ../csu/libc-start.c:376
> > #8  0x0000aaaab867a7f0 in _start () at ../sysdeps/aarch64/start.S:81
> >
> > Regards,
> >
> > Brian
>

[-- Attachment #1.2: Type: text/html, Size: 14111 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Kienan Stewart via lttng-dev]

Hi Brian,


On 7/17/24 12:27 PM, Brian Hutchinson wrote:
> 
> 
> On Wed, Jul 17, 2024 at 10:13 AM Kienan Stewart <kstewart@efficios.com 
> <mailto:kstewart@efficios.com>> wrote:
> 
>     Hi Brian,
> 
>     thanks! I'll take a look at these and see if I can get any ideas
>     from them.
> 
>     To go back to your original e-mail, a detail that I hadn't noticed in
>     your original backtrace was pointed out to me and I'd like to verify
>     with you:
> 
>     ```
>     Core was generated by `./my_app'.
>     Program terminated with signal SIGSEGV, Segmentation fault.
>     #0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
>     96      ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or
>     directory.
>     [Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))]
>     (gdb) bt
>     #0  __strlen_asimd () at ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
>     #1  0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register () from
>     /usr/lib/liblttng-ust-tracepoint.so.1
>     #2  0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at
>     /opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629
>     #3  0x0000ffffb872b30c in call_init (env=<optimized out>,
>     argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145
>     #4  __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1,
>     argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>,
>     rtld_fini=<optimized out>, stack_end=<optimized out>) at
>     ../csu/libc-start.c:376
>     #5  0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81
>     ```
> 
>     The library loaded at frame 1 is
>     `/usr/lib/liblttng-ust-tracepoint.so.1`; however, in frame 2 the
>     reference is from
>     `/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h`.
> 
> 
>     Are the LTTng libraries in `/usr/lib` the exact same version as the
>     headers in
>     `/opt/poky/4.0.18/sysroots/cortexa53-cryptop-poky-linux/usr/include/lttng`?
> 
> 
> Yes.  The sdk was created by yocto using populate_sdk which includes the 
> target image sysroot in the sdk.  Hopefully I explained that right. The 
> target image drives the contents of the sdk and the generated sdk is 
> what is used to cross compile the application specifically for the 
> target we are running on (NXP imx8mm).

I think that makes sense - the debug information has stored the path of 
the build environment which is the sysroot portion.

> 
> 
>     If some of your app is compiled using one version of headers and a
>     different version of the library is loaded at runtime, there could
>     be an
>     ABI mismatch.
> 
> 
>     You could check what the include paths are during compilation and
>     `LD_LIBRARY_PATH` at runtime. Running with the environment variable
>     `LD_DEBUG=bindings,libs` (see `man ld.so` for more info).
> 
> I've been down that road a few times before with the same line of 
> thinking (ABI issue) and believe I've proven I don't have those issues.  
> That's one of the reasons I hand built lttng-ust in native environment 
> (on the actual target) to verify it wasn't an issue with a yocto recipe, 
> cross compiling etc..
> 
> I've been thru this process with different OS versions (yocto Dunfell 
> before), kernel version, toolchain version etc., and now the same with 
> Kirkstone based components.
> 
>     In any case, I'll take a look over the most recent backtraces and
>     see if
>     anything else jumps out.
> 

I don't see anything more that stands out.

I'm going to step back a bit to make sure I have correct understanding 
of your situation.

Based on your previous statements, it sounded like you weren't sure if 
your application is statically linked or not. This is different than 
using a statically linked probe provider.

You can verify by running `ldd /path/to/my_app`. lttng-ust doesn't 
support being statically linked, it is always loaded dynamically. The 
trace probe providers (TPPs) may be statically linked, which is different.

You mentioned earlier that the building scenario you were using 
resembles the hello world example. To confirm my understanding, do you 
mean the hello-static-lib example in lttng-ust/doc/examples? In the 
docs, this is called "The instrumented application is statically linked 
with the tracepoint provider package archive file". Note that the final 
application still uses dynamic linking (`-ldl -llttng-ust`, and the 
absence of `-static -static-libc` when creating the final executable).

Further along in stepping back:

  - Does make check for lttng-ust pass in your environment?
  - Does make check for lttng-tools pass in your environment?
  - Is this reproducible in non-yocto environments or on other 
architectures with the same project?
  - Does running the traced application with `LTTNG_UST_DEBUG=1` yield 
more information?
  - I'd also run lttng-sessiond with the environment variable 
`LTTNG_UST_DEBUG=1` set and `-vvv --verbose-consumer` in the program 
arguments and capture for stdout & stderr into a file for analysis.
  - Verify if the main program is dynamically linked with `ldd`
  - Verify which libraries are loaded at runtime and which calls are 
shimmed with `LD_DEBUG=libs,binding`
  - Review in detail which gcc commands are executed to produce the 
tracepoint provider and link it to the main executable.

My current understanding is that the statedump tracepoints are 
registered and those events communicated before the main program's init 
is run (at least for C programs). If you do the following test, I think 
you should be able to see the statedumps. Could you confirm if you have 
them or not?


```
LTTNG_UST_DEBUG=1 lttng-sessiond -vvv --verbose-consumer &> 
/tmp/sessiond.log &
lttng create
lttng enable-event -u --all
lttng start

unset LTTNG_UST_WITHOUT_BADDR_STATEDUMP
unset LTTNG_UST_WITHOUT_PROCNAME_STATEDUMP
LTTNG_UST_DEBUG=1 LTTNG_UST_REGISTER_TIMEOUT=-1 ./my_app
# Segfault?

lttng stop
lttng view # Here you should have the statedump events
killall lttng-sessiond
```

> 
> Thanks!  I'm running out of things to try.

Ultimately, given the bespoke environment, build steps, and application 
it's tough to diagnose a lot of things that we would go over with a fine 
tooth comb: seeing how the TPPs are built and linked, seeing how the 
application is built and linked, analyzing what's happening at runtime, 
having a coredump that allows us to see the variables, etc.

If you can provide a minimal reproducer, it's possible to dig further 
into it.

Otherwise, to look in more detail at your specific project would be 
covered under a service contract (for which we can sign NDAs, etc. as 
needed). Feel free to reach out the sales@efficios.com to organise that.

> 
> I don't fully understand the implications of some of this documentation, 
> but I have learned enough to know LD_PRELOAD means nothing if everything 
> is static built.  Our application is heavily multi threaded, uses fork, 
> clone and who knows if it's doing double closes or not ... so I've tried 
> these LD_PRELOAD "helpers" or whatever they are, before and didn't get a 
> different result, but now know it's only for shared library object.  So 
> now I will experiment with making our tpp a shared object and try 
> LD_PRELOAD of fork, fd and pthread helpers and starting my app that way.

LD_PRELOAD has an effect if the application you are running is 
dynamically linked (note: the TPP can be statically linked inside an 
dynamically linked executable, and the preloads are still useful and/or 
needed in that case).

The function of the helpers are to ensure that various system calls 
don't clobber things in use by lttng-ust. E.g., close() is shimmed by 
liblttng-ust-fd.so so that FDs that lttng-ust uses aren't suddenly shut 
by the program.

If you want to see it in action, you could try an experiment with a 
slightly modified hello-static-lib application, with instructions in the 
top comment: 
https://gist.github.com/kienanstewart/299eb6a511d92d458569b210e4a418a8

The effect of the preloads such as liblttng-ust-fd.so are independent of 
how the TPP is built and linked to the application or library.

thanks,
kienan

> 
> This is what I'm referring to above from the lttng docs:
> 
> 
>           Use LTTng-UST with daemons
>           <https://lttng.org/docs/v2.13/#doc-using-lttng-ust-with-daemons>
> 
> If your instrumented application calls fork(2) 
> <https://man7.org/linux/man-pages/man2/fork.2.html>, clone(2) 
> <https://man7.org/linux/man-pages/man2/clone.2.html>, or BSD’s rfork(2) 
> <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>, without a following exec(3) <https://man7.org/linux/man-pages/man3/exec.3.html>-family system call, you must preload the |liblttng-ust-fork.so| shared object when you start the application.
> 
> LD_PRELOAD=liblttng-ust-fork.so ./my-app
> 
> If your tracepoint provider package is a shared library which you also 
> preload, you must put both shared objects in |LD_PRELOAD|:
> 
> LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
> 
> 
>           Use LTTng-UST with applications which close file descriptors
>           that don’t belong to them
>           <https://lttng.org/docs/v2.13/#doc-liblttng-ust-fd>
> 
> Since 2.9
> 
> If your instrumented application closes one or more file descriptors 
> which it did not open itself, you must preload the |liblttng-ust-fd.so| 
> shared object when you start the application:
> 
> LD_PRELOAD=liblttng-ust-fd.so ./my-app
> 
> Typical use cases include closing all the file descriptors after fork(2) 
> <https://man7.org/linux/man-pages/man2/fork.2.html> or rfork(2) 
> <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE> and buggy applications doing “double closes”.
> 
> 
> Thanks,
> 
> Brian
> 
> 
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 72213 bytes --]

Hi Kienan,

Took a while to gather your grocery list but I think I have most of it
below ;)

On Thu, Jul 18, 2024 at 10:27 AM Kienan Stewart <kstewart@efficios.com>
wrote:

> Hi Brian,
>
>
> On 7/17/24 12:27 PM, Brian Hutchinson wrote:
> >
> >
> > On Wed, Jul 17, 2024 at 10:13 AM Kienan Stewart <kstewart@efficios.com
> > <mailto:kstewart@efficios.com>> wrote:
> >
> >     Hi Brian,
> >
> >     thanks! I'll take a look at these and see if I can get any ideas
> >     from them.
> >
> >     To go back to your original e-mail, a detail that I hadn't noticed in
> >     your original backtrace was pointed out to me and I'd like to verify
> >     with you:
> >
> >     ```
> >     Core was generated by `./my_app'.
> >     Program terminated with signal SIGSEGV, Segmentation fault.
> >     #0  __strlen_asimd () at
> ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> >     96      ../sysdeps/aarch64/multiarch/strlen_asimd.S: No such file or
> >     directory.
> >     [Current thread is 1 (Thread 0xffffb8e2d040 (LWP 757))]
> >     (gdb) bt
> >     #0  __strlen_asimd () at
> ../sysdeps/aarch64/multiarch/strlen_asimd.S:96
> >     #1  0x0000ffffb86c5330 in lttng_ust_tracepoint_module_register ()
> from
> >     /usr/lib/liblttng-ust-tracepoint.so.1
> >     #2  0x0000aaaab4c8da18 in lttng_ust__tracepoints__ptrs_init () at
> >
>  /opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h:629
> >     #3  0x0000ffffb872b30c in call_init (env=<optimized out>,
> >     argv=0xffffe2d66098, argc=1) at ../csu/libc-start.c:145
> >     #4  __libc_start_main_impl (main=0xaaaab4bd94e0 <main>, argc=1,
> >     argv=0xffffe2d66098, init=<optimized out>, fini=<optimized out>,
> >     rtld_fini=<optimized out>, stack_end=<optimized out>) at
> >     ../csu/libc-start.c:376
> >     #5  0x0000aaaab4bd9230 in _start () at ../sysdeps/aarch64/start.S:81
> >     ```
> >
> >     The library loaded at frame 1 is
> >     `/usr/lib/liblttng-ust-tracepoint.so.1`; however, in frame 2 the
> >     reference is from
> >
>  `/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux/usr/include/lttng/tracepoint.h`.
> >
> >
> >     Are the LTTng libraries in `/usr/lib` the exact same version as the
> >     headers in
> >
>  `/opt/poky/4.0.18/sysroots/cortexa53-cryptop-poky-linux/usr/include/lttng`?
> >
> >
> > Yes.  The sdk was created by yocto using populate_sdk which includes the
> > target image sysroot in the sdk.  Hopefully I explained that right. The
> > target image drives the contents of the sdk and the generated sdk is
> > what is used to cross compile the application specifically for the
> > target we are running on (NXP imx8mm).
>
> I think that makes sense - the debug information has stored the path of
> the build environment which is the sysroot portion.
>
> >
> >
> >     If some of your app is compiled using one version of headers and a
> >     different version of the library is loaded at runtime, there could
> >     be an
> >     ABI mismatch.
> >
> >
> >     You could check what the include paths are during compilation and
> >     `LD_LIBRARY_PATH` at runtime. Running with the environment variable
> >     `LD_DEBUG=bindings,libs` (see `man ld.so` for more info).
> >
> > I've been down that road a few times before with the same line of
> > thinking (ABI issue) and believe I've proven I don't have those issues.
> > That's one of the reasons I hand built lttng-ust in native environment
> > (on the actual target) to verify it wasn't an issue with a yocto recipe,
> > cross compiling etc..
> >
> > I've been thru this process with different OS versions (yocto Dunfell
> > before), kernel version, toolchain version etc., and now the same with
> > Kirkstone based components.
> >
> >     In any case, I'll take a look over the most recent backtraces and
> >     see if
> >     anything else jumps out.
> >
>
> I don't see anything more that stands out.
>
> I'm going to step back a bit to make sure I have correct understanding
> of your situation.
>
> Based on your previous statements, it sounded like you weren't sure if
> your application is statically linked or not. This is different than
> using a statically linked probe provider.
>
> You can verify by running `ldd /path/to/my_app`. lttng-ust doesn't
> support being statically linked, it is always loaded dynamically. The
> trace probe providers (TPPs) may be statically linked, which is different.
>
> You mentioned earlier that the building scenario you were using
> resembles the hello world example. To confirm my understanding, do you
> mean the hello-static-lib example in lttng-ust/doc/examples? In the
> docs, this is called "The instrumented application is statically linked
> with the tracepoint provider package archive file". Note that the final
> application still uses dynamic linking (`-ldl -llttng-ust`, and the
> absence of `-static -static-libc` when creating the final executable).
>

I may have not been clear.  Most of the application components are
statically linked but I think there are some that are built as shared
objects (.so's) so that's what I was referring to.  I know that lttng-ust
is dynamically linked ... I think the lttng-ust docs say this is only
option but also makes reference to the fact static linking was once
possible (in some versions of the documentation) but not supported anymore
(I probably have the docs memorized by now ha, ha ... I've looked at many,
many versions of them).

Just for full disclosure my ldd looks like:

       linux-vdso.so.1 (0x0000ffffab196000)
       libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
       liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1 (0x0000ffffa5750000)
       libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
       librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
       libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
       libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
       libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
       libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
       /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
       liblttng-ust-common.so.1 => /usr/local/lib/liblttng-ust-common.so.1
(0x0000ffffa50e0000)
       liblttng-ust-tracepoint.so.1 =>
/usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
       libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
       libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)


>
> Further along in stepping back:
>
>   - Does make check for lttng-ust pass in your environment?
>

So I'm glad you mentioned this.  I didn't try to run make check because #1
have never seen anything in the documentation about it and what it does and
#2 It said that it depended on Perl and since this is an embedded system, I
really didn't want to pull it and its dependencies in.

Nevertheless, I ran it on my native target build I setup to see what would
happen.  I didn't notice it complaining about perl so then I checked my
manifest and perl is in fact in our image (I guess some other recipe pulled
it in and I never paid attention to that):

Making check in include
make[1]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/include'
make[1]: Nothing to be done for 'check'.
make[1]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/include'
Making check in src
make[1]: Entering directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/src'
Making check in common
make[2]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/common'
make[2]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/common'
Making check in lib
make[2]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib'
Making check in lttng-ust-common
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-common'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-common'
Making check in lttng-ust-tracepoint
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-tracepoint'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-tracepoint'
Making check in lttng-ust
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust'
Making check in lttng-ust-ctl
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-ctl'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-ctl'
Making check in lttng-ust-fd
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-fd'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-fd'
Making check in lttng-ust-fork
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-fork'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-fork'
Making check in lttng-ust-cyg-profile
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-cyg-profile'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-cyg-profile'
Making check in lttng-ust-libc-wrapper
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-libc-wrapper'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-libc-wrapper'
Making check in lttng-ust-pthread-wrapper
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-pthread-wrapper'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-pthread-wrapper'
Making check in lttng-ust-dl
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-dl'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib/lttng-ust-dl'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib'
make[3]: Nothing to be done for 'check-am'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib'
make[2]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/src/lib'
make[2]: Entering directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/src'
make[2]: Nothing to be done for 'check-am'.
make[2]: Leaving directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/src'
make[1]: Leaving directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/src'
Making check in tools
make[1]: Entering directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tools'
make[1]: Nothing to be done for 'check'.
make[1]: Leaving directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tools'
Making check in doc
make[1]: Entering directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc'
Making check in .
make[2]: Entering directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc'
make[2]: Nothing to be done for 'check-am'.
make[2]: Leaving directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc'
Making check in man
make[2]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/man'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/man'
Making check in examples
make[2]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/easy-ust'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/easy-ust'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/demo'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/demo'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/hello-static-lib'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/hello-static-lib'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/demo-tracef'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/demo-tracef'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/clock-override'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/clock-override'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/getcpu-override'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/getcpu-override'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/demo-tracelog'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/demo-tracelog'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/gen-tp'
make[3]: Nothing to be done for 'all'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples/gen-tp'
make[2]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc/examples'
make[1]: Leaving directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/doc'
Making check in tests
make[1]: Entering directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests'
Making check in utils
make[2]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
make  tap-driver.sh tap.sh utils.sh
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
make[3]: Nothing to be done for 'tap-driver.sh'.
make[3]: Nothing to be done for 'tap.sh'.
make[3]: Nothing to be done for 'utils.sh'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
make[2]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
Making check in unit
make[2]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit'
Making check in gcc-weak-hidden
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/gcc-weak-hidden'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/gcc-weak-hidden'
Making check in libcommon
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/libcommon'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/libcommon'
Making check in libmsgpack
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/libmsgpack'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/libmsgpack'
Making check in libringbuffer
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/libringbuffer'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/libringbuffer'
Making check in pthread_name
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/pthread_name'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/pthread_name'
Making check in snprintf
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/snprintf'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/snprintf'
Making check in ust-elf
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/ust-elf'
make  test_ust_elf
make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/ust-elf'
make[4]: Nothing to be done for 'test_ust_elf'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/ust-elf'
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/ust-elf'
Making check in ust-error
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/ust-error'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/ust-error'
Making check in ust-utils
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/ust-utils'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/ust-utils'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit'
make[3]: Nothing to be done for 'check-am'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit'
make[2]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit'
Making check in compile
make[2]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile'
Making check in api0
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0'
Making check in ctf-types
make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0/ctf-types'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0/ctf-types'
Making check in hello
make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0/hello'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0/hello'
Making check in hello-many
make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0/hello-many'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0/hello-many'
Making check in same_line_tracepoint
make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0/same_line_tracepoint'

make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0/same_line_tracepoint'

make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0'
make[4]: Nothing to be done for 'check-am'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0'
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api0'
Making check in api1
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1'
Making check in ust-fields
make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1/ust-fields'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1/ust-fields'
Making check in hello
make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1/hello'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1/hello'
Making check in hello-many
make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1/hello-many'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1/hello-many'
Making check in same_line_tracepoint
make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1/same_line_tracepoint'

make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1/same_line_tracepoint'

Making check in test-app-ctx
make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1/test-app-ctx'
make[4]: Nothing to be done for 'check'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1/test-app-ctx'
make[4]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1'
make[4]: Nothing to be done for 'check-am'.
make[4]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1'
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile/api1'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile'
make[3]: Nothing to be done for 'check-am'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile'
make[2]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/compile'
Making check in benchmark
make[2]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/benchmark'
make[2]: Nothing to be done for 'check'.
make[2]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/benchmark'
Making check in regression
make[2]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/regression'
Making check in abi0-conflict
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/regression/abi0-conflict'
make[3]: Nothing to be done for 'check'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/regression/abi0-conflict'
make[3]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/regression'
make[3]: Nothing to be done for 'check-am'.
make[3]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/regression'
make[2]: Leaving directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/regression'
make[2]: Entering directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests'
make  check-TESTS
make[3]: Entering directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests'
make[4]: Entering directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests'
PASS: unit/libringbuffer/test_shm 1 - Open a POSIX shm fd
PASS: unit/libringbuffer/test_shm 2 - Create a shm object table
PASS: unit/libringbuffer/test_shm 3 - Allocate the shm object table
PASS: unit/libringbuffer/test_shm 4 - Allocate an object in the shm with
sufficient space
PASS: unit/libringbuffer/test_shm 5 - Allocate an object in the shm with
insufficient space
# unit/gcc-weak-hidden/test_gcc_weak_hidden: Address of weak symbol with
hidden visibility match between compile units within same module for main
program (4 bytes integer object)
# unit/gcc-weak-hidden/test_gcc_weak_hidden: Address of weak symbol with
hidden visibility match between compile units within same module for main
program (pointer object)
# unit/gcc-weak-hidden/test_gcc_weak_hidden: Address of weak symbol with
hidden visibility match between compile units within same module for main
program (24 bytes structure object)
# unit/gcc-weak-hidden/test_gcc_weak_hidden: Address of weak symbol with
hidden visibility match between compile units within same module for shared
library (4 bytes integer object)
# unit/gcc-weak-hidden/test_gcc_weak_hidden: Address of weak symbol with
hidden visibility match between compile units within same module for shared
library (pointer object)
# unit/gcc-weak-hidden/test_gcc_weak_hidden: Address of weak symbol with
hidden visibility match between compile units within same module for shared
library (24 bytes structure object)
PASS: unit/gcc-weak-hidden/test_gcc_weak_hidden 1 - Weak-hidden behavior is
the same for 4 bytes integer and pointer objects within main program
PASS: unit/gcc-weak-hidden/test_gcc_weak_hidden 2 - Weak-hidden behavior is
the same for 4 bytes integer and pointer objects within shared library
PASS: unit/libcommon/test_get_cpu_mask_from_sysfs 1 -
test_get_cpu_mask_from_sysfs - without '\n' expected: '', result: ''
PASS: unit/libcommon/test_get_cpu_mask_from_sysfs 2 -
test_get_cpu_mask_from_sysfs - with '\n' expected: '', result: ''
PASS: unit/libcommon/test_get_cpu_mask_from_sysfs 3 -
test_get_cpu_mask_from_sysfs - without '\n' expected: '0', result: '0'
PASS: unit/libcommon/test_get_cpu_mask_from_sysfs 4 -
test_get_cpu_mask_from_sysfs - with '\n' expected: '0', result: '0'
PASS: unit/libcommon/test_get_cpu_mask_from_sysfs 5 -
test_get_cpu_mask_from_sysfs - without '\n' expected: '0-3', result: '0-3'
PASS: unit/libcommon/test_get_cpu_mask_from_sysfs 6 -
test_get_cpu_mask_from_sysfs - with '\n' expected: '0-3', result: '0-3'
PASS: unit/libcommon/test_get_cpu_mask_from_sysfs 7 -
test_get_cpu_mask_from_sysfs - without '\n' expected: '0,3-7,9', result:
'0,3-7,9'
PASS: unit/libcommon/test_get_cpu_mask_from_sysfs 8 -
test_get_cpu_mask_from_sysfs - with '\n' expected: '0,3-7,9', result:
'0,3-7,9'
# unit/libcommon/test_get_max_cpuid_from_mask: Testing smp helpers
PASS: unit/libcommon/test_get_max_cpuid_from_mask 1 -
get_max_cpuid_from_mask '', expected: '-1', result: '-1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 2 -
get_max_cpuid_from_mask 'abc', expected: '-1', result: '-1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 3 -
get_max_cpuid_from_mask ',,,', expected: '-1', result: '-1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 4 -
get_max_cpuid_from_mask '--', expected: '-1', result: '-1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 5 -
get_max_cpuid_from_mask ',', expected: '-1', result: '-1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 6 -
get_max_cpuid_from_mask '-', expected: '-1', result: '-1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 7 -
get_max_cpuid_from_mask '2147483647', expected: '-1', result: '-1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 8 -
get_max_cpuid_from_mask '18446744073709551615', expected: '-1', result:
'-1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 9 -
get_max_cpuid_from_mask '0-2147483647', expected: '-1', result: '-1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 10 -
get_max_cpuid_from_mask '0-18446744073709551615', expected: '-1', result:
'-1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 11 -
get_max_cpuid_from_mask '0', expected: '0', result: '0'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 12 -
get_max_cpuid_from_mask '1', expected: '1', result: '1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 13 -
get_max_cpuid_from_mask '0-1', expected: '1', result: '1'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 14 -
get_max_cpuid_from_mask '1-3', expected: '3', result: '3'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 15 -
get_max_cpuid_from_mask '0,2', expected: '2', result: '2'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 16 -
get_max_cpuid_from_mask '1,2', expected: '2', result: '2'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 17 -
get_max_cpuid_from_mask '0,4-6,127', expected: '127', result: '127'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 18 -
get_max_cpuid_from_mask '0-4095', expected: '4095', result: '4095'
PASS: unit/libcommon/test_get_max_cpuid_from_mask 19 -
get_max_cpuid_from_mask '
PASS: unit/libcommon/test_get_max_cpuid_from_mask 20 -
get_max_cpuid_from_mask 'abc
PASS: unit/libcommon/test_get_max_cpuid_from_mask 21 -
get_max_cpuid_from_mask ',,,
PASS: unit/libcommon/test_get_max_cpuid_from_mask 22 -
get_max_cpuid_from_mask '--
PASS: unit/libcommon/test_get_max_cpuid_from_mask 23 -
get_max_cpuid_from_mask ',
PASS: unit/libcommon/test_get_max_cpuid_from_mask 24 -
get_max_cpuid_from_mask '-
PASS: unit/libcommon/test_get_max_cpuid_from_mask 25 -
get_max_cpuid_from_mask '2147483647
PASS: unit/libcommon/test_get_max_cpuid_from_mask 26 -
get_max_cpuid_from_mask '18446744073709551615
PASS: unit/libcommon/test_get_max_cpuid_from_mask 27 -
get_max_cpuid_from_mask '0-2147483647
PASS: unit/libcommon/test_get_max_cpuid_from_mask 28 -
get_max_cpuid_from_mask '0-18446744073709551615
PASS: unit/libcommon/test_get_max_cpuid_from_mask 29 -
get_max_cpuid_from_mask '0
PASS: unit/libcommon/test_get_max_cpuid_from_mask 30 -
get_max_cpuid_from_mask '1
PASS: unit/libcommon/test_get_max_cpuid_from_mask 31 -
get_max_cpuid_from_mask '0-1
PASS: unit/libcommon/test_get_max_cpuid_from_mask 32 -
get_max_cpuid_from_mask '1-3
PASS: unit/libcommon/test_get_max_cpuid_from_mask 33 -
get_max_cpuid_from_mask '0,2
PASS: unit/libcommon/test_get_max_cpuid_from_mask 34 -
get_max_cpuid_from_mask '1,2
PASS: unit/libcommon/test_get_max_cpuid_from_mask 35 -
get_max_cpuid_from_mask '0,4-6,127
PASS: unit/libcommon/test_get_max_cpuid_from_mask 36 -
get_max_cpuid_from_mask '0-4095
# unit/libcommon/test_get_max_cpuid_from_sysfs: get_max_cpuid_from_sysfs
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 1 -
get_max_cpuid_from_sysfs - cpu set: 'cpu0', expected: '0', result: '0'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 2 -
get_max_cpuid_from_sysfs - cpu set: 'cpu0 cpu1', expected: '1', result: '1'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 3 -
get_max_cpuid_from_sysfs - cpu set: 'cpu1 cpu0', expected: '1', result: '1'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 4 -
get_max_cpuid_from_sysfs - cpu set: 'cpu3', expected: '3', result: '3'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 5 -
get_max_cpuid_from_sysfs - cpu set: 'cpu99', expected: '99', result: '99'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 6 -
get_max_cpuid_from_sysfs - cpu set: 'cpu0 cpu3', expected: '3', result: '3'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 7 -
get_max_cpuid_from_sysfs - cpu set: 'cpufreq cpuidle cpu0 cpu1 cpu2 cpu3',
expected: '3', result: '3'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 8 -
get_max_cpuid_from_sysfs - cpu set: 'cpu cpu0', expected: '0', result: '0'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 9 -
get_max_cpuid_from_sysfs - cpu set: 'cpu cpu5', expected: '5', result: '5'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 10 -
get_max_cpuid_from_sysfs - cpu set: 'toto', expected: '-1', result: '-1'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 11 -
get_max_cpuid_from_sysfs - cpu set: 'cpu', expected: '-1', result: '-1'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 12 -
get_max_cpuid_from_sysfs - cpu set: 'cpua cpud', expected: '-1', result:
'-1'
PASS: unit/libcommon/test_get_max_cpuid_from_sysfs 13 -
get_max_cpuid_from_sysfs - cpu set: 'cpufreq cpuidle', expected: '-1',
result: '-1'
PASS: unit/libcommon/test_get_possible_cpus_array_len 1 -
get_possible_cpus_array_len (4 > 0)
PASS: unit/libcommon/test_get_possible_cpus_array_len 2 -
get_num_possible_cpus_fallback (4 > 0)
# unit/libmsgpack/test_msgpack: Testing msgpack implementation
PASS: unit/libmsgpack/test_msgpack 1 - NIL object
PASS: unit/libmsgpack/test_msgpack 2 - String "bye" object
PASS: unit/libmsgpack/test_msgpack 3 - Unsigned integer "1337" object
PASS: unit/libmsgpack/test_msgpack 4 - Unsigned integer "127" object
PASS: unit/libmsgpack/test_msgpack 5 - Unsigned integer "128" object
PASS: unit/libmsgpack/test_msgpack 6 - Unsigned integer "256" object
PASS: unit/libmsgpack/test_msgpack 7 - Unsigned integer "65536" object
PASS: unit/libmsgpack/test_msgpack 8 - Unsigned integer "65535" object
PASS: unit/libmsgpack/test_msgpack 9 - Unsigned integer "4294967295" object
PASS: unit/libmsgpack/test_msgpack 10 - Unsigned integer "4294967296"
object
PASS: unit/libmsgpack/test_msgpack 11 - Signed integer "-32" object
PASS: unit/libmsgpack/test_msgpack 12 - Signed integer "-33" object
PASS: unit/libmsgpack/test_msgpack 13 - Signed integer "-129" object
PASS: unit/libmsgpack/test_msgpack 14 - Signed integer "-32768" object
PASS: unit/libmsgpack/test_msgpack 15 - Signed integer "-32769" object
PASS: unit/libmsgpack/test_msgpack 16 - Signed integer "-2147483648" object
PASS: unit/libmsgpack/test_msgpack 17 - Signed integer "-2147483649" object
PASS: unit/libmsgpack/test_msgpack 18 - double "0.0" object
PASS: unit/libmsgpack/test_msgpack 19 - double "PI" object
PASS: unit/libmsgpack/test_msgpack 20 - double "-PI" object
PASS: unit/libmsgpack/test_msgpack 21 - Array of double object
PASS: unit/libmsgpack/test_msgpack 22 - Map object
PASS: unit/libmsgpack/test_msgpack 23 - Complete capture object
PASS: unit/pthread_name/test_pthread_name 1 - Get the thread name:
'test_pthread_na'
PASS: unit/pthread_name/test_pthread_name 2 - Set a too long thread name:
'thisnameistoolong'
PASS: unit/pthread_name/test_pthread_name 3 - Get the thread name:
'test_pthread_na'
PASS: unit/pthread_name/test_pthread_name 4 - Compare the initial thread
name: 'test_pthread_na' == 'test_pthread_na'
PASS: unit/pthread_name/test_pthread_name 5 - Set a short thread name:
'labatt50'
PASS: unit/pthread_name/test_pthread_name 6 - Get a short thread name:
'labatt50'
PASS: unit/pthread_name/test_pthread_name 7 - Compare the short thread
name: 'labatt50' == 'labatt50'
PASS: unit/pthread_name/test_pthread_name 8 - Compare the short UST thread
name: 'labatt50-ust' == 'labatt50-ust'
PASS: unit/pthread_name/test_pthread_name 9 - Set a long thread name:
'procrastinating'
PASS: unit/pthread_name/test_pthread_name 10 - Get a long thread name:
'procrastinating'
PASS: unit/pthread_name/test_pthread_name 11 - Compare the long thread
name: 'procrastinating' == 'procrastinating'
PASS: unit/pthread_name/test_pthread_name 12 - Compare the long UST thread
name: 'procrastina-ust' == 'procrastina-ust'
PASS: unit/snprintf/test_snprintf 1 - Got expected output string with
format string "header %d, %s, %03d, '%*d'"
PASS: unit/ust-elf/test_ust_elf 1 - Invoke as:
/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/unit/ust-elf/.libs/lt-ust-elf
<path>
# unit/ust-elf/test_ust_elf: Testing x86 support
PASS: unit/ust-elf/test_ust_elf 2 - lttng_ust_elf_create
PASS: unit/ust-elf/test_ust_elf 3 - lttng_ust_elf_get_memsz returned
successfully
PASS: unit/ust-elf/test_ust_elf 4 - memsz - expected: 5732, got: 5732
PASS: unit/ust-elf/test_ust_elf 5 - lttng_ust_elf_get_build_id returned
successfully
PASS: unit/ust-elf/test_ust_elf 6 - build id marked as found
PASS: unit/ust-elf/test_ust_elf 7 - build_id_len - expected: 20, got: 20
PASS: unit/ust-elf/test_ust_elf 8 - build_id has expected value
PASS: unit/ust-elf/test_ust_elf 9 - lttng_ust_elf_get_debug_link returned
successfully
PASS: unit/ust-elf/test_ust_elf 10 - debug link marked as found
PASS: unit/ust-elf/test_ust_elf 11 - debug link filename - expected:
main.elf.debug, got: main.elf.debug
PASS: unit/ust-elf/test_ust_elf 12 - debug link crc - expected: 0x1531f73c,
got: 0x1531f73c
# unit/ust-elf/test_ust_elf: Testing x86_64 support
PASS: unit/ust-elf/test_ust_elf 13 - lttng_ust_elf_create
PASS: unit/ust-elf/test_ust_elf 14 - lttng_ust_elf_get_memsz returned
successfully
PASS: unit/ust-elf/test_ust_elf 15 - memsz - expected: 2099376, got:
2099376
PASS: unit/ust-elf/test_ust_elf 16 - lttng_ust_elf_get_build_id returned
successfully
PASS: unit/ust-elf/test_ust_elf 17 - build id marked as found
PASS: unit/ust-elf/test_ust_elf 18 - build_id_len - expected: 20, got: 20
PASS: unit/ust-elf/test_ust_elf 19 - build_id has expected value
PASS: unit/ust-elf/test_ust_elf 20 - lttng_ust_elf_get_debug_link returned
successfully
PASS: unit/ust-elf/test_ust_elf 21 - debug link marked as found
PASS: unit/ust-elf/test_ust_elf 22 - debug link filename - expected:
main.elf.debug, got: main.elf.debug
PASS: unit/ust-elf/test_ust_elf 23 - debug link crc - expected: 0xa048a98f,
got: 0xa048a98f
# unit/ust-elf/test_ust_elf: Testing armeb support
PASS: unit/ust-elf/test_ust_elf 24 - lttng_ust_elf_create
PASS: unit/ust-elf/test_ust_elf 25 - lttng_ust_elf_get_memsz returned
successfully
PASS: unit/ust-elf/test_ust_elf 26 - memsz - expected: 34196, got: 34196
PASS: unit/ust-elf/test_ust_elf 27 - lttng_ust_elf_get_build_id returned
successfully
PASS: unit/ust-elf/test_ust_elf 28 - build id marked as found
PASS: unit/ust-elf/test_ust_elf 29 - build_id_len - expected: 20, got: 20
PASS: unit/ust-elf/test_ust_elf 30 - build_id has expected value
PASS: unit/ust-elf/test_ust_elf 31 - lttng_ust_elf_get_debug_link returned
successfully
PASS: unit/ust-elf/test_ust_elf 32 - debug link marked as found
PASS: unit/ust-elf/test_ust_elf 33 - debug link filename - expected:
main.elf.debug, got: main.elf.debug
PASS: unit/ust-elf/test_ust_elf 34 - debug link crc - expected: 0x9d40261b,
got: 0x9d40261b
# unit/ust-elf/test_ust_elf: Testing aarch64_be support
PASS: unit/ust-elf/test_ust_elf 35 - lttng_ust_elf_create
PASS: unit/ust-elf/test_ust_elf 36 - lttng_ust_elf_get_memsz returned
successfully
PASS: unit/ust-elf/test_ust_elf 37 - memsz - expected: 67632, got: 67632
PASS: unit/ust-elf/test_ust_elf 38 - lttng_ust_elf_get_build_id returned
successfully
PASS: unit/ust-elf/test_ust_elf 39 - build id marked as found
PASS: unit/ust-elf/test_ust_elf 40 - build_id_len - expected: 20, got: 20
PASS: unit/ust-elf/test_ust_elf 41 - build_id has expected value
PASS: unit/ust-elf/test_ust_elf 42 - lttng_ust_elf_get_debug_link returned
successfully
PASS: unit/ust-elf/test_ust_elf 43 - debug link marked as found
PASS: unit/ust-elf/test_ust_elf 44 - debug link filename - expected:
main.elf.debug, got: main.elf.debug
PASS: unit/ust-elf/test_ust_elf 45 - debug link crc - expected: 0x2b8cedce,
got: 0x2b8cedce
PASS: unit/ust-elf/test_ust_elf 46 - hello.exec is not PIC
PASS: unit/ust-elf/test_ust_elf 47 - hello.pie is PIC
PASS: unit/ust-elf/test_ust_elf 48 - hello.pic is PIC
PASS: unit/ust-error/test_ust_error 1 - lttng_ust_strerror - Positive
LTTNG_UST_OK returns 'Success' (Success)
PASS: unit/ust-error/test_ust_error 2 - lttng_ust_strerror - Negative
LTTNG_UST_OK returns 'Success' (Success)
PASS: unit/ust-error/test_ust_error 3 - lttng_ust_strerror - Positive large
int returns 'Unknown error' (Unknown error)
PASS: unit/ust-error/test_ust_error 4 - lttng_ust_strerror - Negative large
int returns 'Unknown error' (Unknown error)
PASS: unit/ust-error/test_ust_error 5 - lttng_ust_strerror - Positive
LTTNG_UST_ERR_NR returns 'Unknown error' (Unknown error)
PASS: unit/ust-error/test_ust_error 6 - lttng_ust_strerror - Negative
LTTNG_UST_ERR_NR returns 'Unknown error' (Unknown error)
PASS: unit/ust-error/test_ust_error 7 - lttng_ust_strerror - Positive
LTTNG_UST_ERR_NR + 1 returns 'Unknown error' (Unknown error)
PASS: unit/ust-error/test_ust_error 8 - lttng_ust_strerror - Negative
LTTNG_UST_ERR_NR - 1 returns 'Unknown error' (Unknown error)
PASS: unit/ust-error/test_ust_error 9 - lttng_ust_strerror - Positive
LTTNG_UST_ERR_NOENT returns 'Unknown error' (Unknown error)
PASS: unit/ust-error/test_ust_error 10 - lttng_ust_strerror - Negative
LTTNG_UST_ERR_NOENT returns 'No entry' (No entry)
PASS: unit/ust-error/test_ust_error 11 - lttng_ust_strerror - Positive
LTTNG_UST_ERR_PEERCRED_PID returns 'Unknown error' (Unknown error)
PASS: unit/ust-error/test_ust_error 12 - lttng_ust_strerror - Negative
LTTNG_UST_ERR_PEERCRED_PID returns 'Peer credentials PID is invalid. Socket
appears to belong to a distinct, non-nested pid namespace.' (Peer
credentials PID is invalid. Socket appears to belong to a disti)
PASS: unit/ust-utils/test_ust_utils 1 - lttng_ust_stringify - literal
integer
PASS: unit/ust-utils/test_ust_utils 2 - lttng_ust_stringify - identifier
PASS: unit/ust-utils/test_ust_utils 3 - lttng_ust_is_signed_type - 'signed
char' is signed
PASS: unit/ust-utils/test_ust_utils 4 - lttng_ust_is_signed_type - 'short'
is signed
PASS: unit/ust-utils/test_ust_utils 5 - lttng_ust_is_signed_type - 'int' is
signed
PASS: unit/ust-utils/test_ust_utils 6 - lttng_ust_is_signed_type - 'long'
is signed
PASS: unit/ust-utils/test_ust_utils 7 - lttng_ust_is_signed_type - 'long
long' is signed
PASS: unit/ust-utils/test_ust_utils 8 - lttng_ust_is_signed_type - 'float'
is signed
PASS: unit/ust-utils/test_ust_utils 9 - lttng_ust_is_signed_type - 'double'
is signed
PASS: unit/ust-utils/test_ust_utils 10 - lttng_ust_is_signed_type - 'long
double' is signed
PASS: unit/ust-utils/test_ust_utils 11 - lttng_ust_is_signed_type -
'int8_t' is signed
PASS: unit/ust-utils/test_ust_utils 12 - lttng_ust_is_signed_type -
'int16_t' is signed
PASS: unit/ust-utils/test_ust_utils 13 - lttng_ust_is_signed_type -
'int32_t' is signed
PASS: unit/ust-utils/test_ust_utils 14 - lttng_ust_is_signed_type -
'int64_t' is signed
PASS: unit/ust-utils/test_ust_utils 15 - lttng_ust_is_signed_type -
'intmax_t' is signed
PASS: unit/ust-utils/test_ust_utils 16 - lttng_ust_is_signed_type -
'ssize_t' is signed
PASS: unit/ust-utils/test_ust_utils 17 - lttng_ust_is_signed_type -
'ptrdiff_t' is signed
PASS: unit/ust-utils/test_ust_utils 18 - lttng_ust_is_signed_type -
'intptr_t' is signed
PASS: unit/ust-utils/test_ust_utils 19 - lttng_ust_is_signed_type -
'unsigned char' is unsigned
PASS: unit/ust-utils/test_ust_utils 20 - lttng_ust_is_signed_type -
'unsigned short' is unsigned
PASS: unit/ust-utils/test_ust_utils 21 - lttng_ust_is_signed_type -
'unsigned int' is unsigned
PASS: unit/ust-utils/test_ust_utils 22 - lttng_ust_is_signed_type -
'unsigned long' is unsigned
PASS: unit/ust-utils/test_ust_utils 23 - lttng_ust_is_signed_type -
'unsigned long long' is unsigned
PASS: unit/ust-utils/test_ust_utils 24 - lttng_ust_is_signed_type -
'uint8_t' is unsigned
PASS: unit/ust-utils/test_ust_utils 25 - lttng_ust_is_signed_type -
'uint16_t' is unsigned
PASS: unit/ust-utils/test_ust_utils 26 - lttng_ust_is_signed_type -
'uint32_t' is unsigned
PASS: unit/ust-utils/test_ust_utils 27 - lttng_ust_is_signed_type -
'uint64_t' is unsigned
PASS: unit/ust-utils/test_ust_utils 28 - lttng_ust_is_signed_type -
'uintmax_t' is unsigned
PASS: unit/ust-utils/test_ust_utils 29 - lttng_ust_is_signed_type - '_Bool'
is unsigned
PASS: unit/ust-utils/test_ust_utils 30 - lttng_ust_is_signed_type -
'size_t' is unsigned
PASS: unit/ust-utils/test_ust_utils 31 - lttng_ust_is_signed_type - 'void
*' is unsigned
PASS: unit/ust-utils/test_ust_utils 32 - lttng_ust_is_integer_type - 'char'
is an integer
PASS: unit/ust-utils/test_ust_utils 33 - lttng_ust_is_integer_type -
'short' is an integer
PASS: unit/ust-utils/test_ust_utils 34 - lttng_ust_is_integer_type - 'int'
is an integer
PASS: unit/ust-utils/test_ust_utils 35 - lttng_ust_is_integer_type - 'long'
is an integer
PASS: unit/ust-utils/test_ust_utils 36 - lttng_ust_is_integer_type - 'long
long' is an integer
PASS: unit/ust-utils/test_ust_utils 37 - lttng_ust_is_integer_type -
'signed char' is an integer
PASS: unit/ust-utils/test_ust_utils 38 - lttng_ust_is_integer_type -
'signed short' is an integer
PASS: unit/ust-utils/test_ust_utils 39 - lttng_ust_is_integer_type -
'signed int' is an integer
PASS: unit/ust-utils/test_ust_utils 40 - lttng_ust_is_integer_type -
'signed long' is an integer
PASS: unit/ust-utils/test_ust_utils 41 - lttng_ust_is_integer_type -
'signed long long' is an integer
PASS: unit/ust-utils/test_ust_utils 42 - lttng_ust_is_integer_type -
'unsigned char' is an integer
PASS: unit/ust-utils/test_ust_utils 43 - lttng_ust_is_integer_type -
'unsigned short' is an integer
PASS: unit/ust-utils/test_ust_utils 44 - lttng_ust_is_integer_type -
'unsigned int' is an integer
PASS: unit/ust-utils/test_ust_utils 45 - lttng_ust_is_integer_type -
'unsigned long' is an integer
PASS: unit/ust-utils/test_ust_utils 46 - lttng_ust_is_integer_type -
'unsigned long long' is an integer
PASS: unit/ust-utils/test_ust_utils 47 - lttng_ust_is_integer_type -
'int8_t' is an integer
PASS: unit/ust-utils/test_ust_utils 48 - lttng_ust_is_integer_type -
'int16_t' is an integer
PASS: unit/ust-utils/test_ust_utils 49 - lttng_ust_is_integer_type -
'int32_t' is an integer
PASS: unit/ust-utils/test_ust_utils 50 - lttng_ust_is_integer_type -
'int64_t' is an integer
PASS: unit/ust-utils/test_ust_utils 51 - lttng_ust_is_integer_type -
'intmax_t' is an integer
PASS: unit/ust-utils/test_ust_utils 52 - lttng_ust_is_integer_type -
'uint8_t' is an integer
PASS: unit/ust-utils/test_ust_utils 53 - lttng_ust_is_integer_type -
'uint16_t' is an integer
PASS: unit/ust-utils/test_ust_utils 54 - lttng_ust_is_integer_type -
'uint32_t' is an integer
PASS: unit/ust-utils/test_ust_utils 55 - lttng_ust_is_integer_type -
'uint64_t' is an integer
PASS: unit/ust-utils/test_ust_utils 56 - lttng_ust_is_integer_type -
'uintmax_t' is an integer
PASS: unit/ust-utils/test_ust_utils 57 - lttng_ust_is_integer_type -
'float' is not an integer
PASS: unit/ust-utils/test_ust_utils 58 - lttng_ust_is_integer_type -
'double' is not an integer
PASS: unit/ust-utils/test_ust_utils 59 - lttng_ust_is_integer_type - 'long
double' is not an integer
PASS: unit/ust-utils/test_ust_utils 60 - lttng_ust_is_integer_type - 'void
*' is not an integer
PASS: unit/ust-utils/test_ust_utils 61 - lttng_ust_is_pointer_type - 'char'
is not a pointer
PASS: unit/ust-utils/test_ust_utils 62 - lttng_ust_is_pointer_type -
'short' is not a pointer
PASS: unit/ust-utils/test_ust_utils 63 - lttng_ust_is_pointer_type - 'int'
is not a pointer
PASS: unit/ust-utils/test_ust_utils 64 - lttng_ust_is_pointer_type - 'long'
is not a pointer
PASS: unit/ust-utils/test_ust_utils 65 - lttng_ust_is_pointer_type - 'long
long' is not a pointer
PASS: unit/ust-utils/test_ust_utils 66 - lttng_ust_is_pointer_type -
'signed char' is not a pointer
PASS: unit/ust-utils/test_ust_utils 67 - lttng_ust_is_pointer_type -
'signed short' is not a pointer
PASS: unit/ust-utils/test_ust_utils 68 - lttng_ust_is_pointer_type -
'signed int' is not a pointer
PASS: unit/ust-utils/test_ust_utils 69 - lttng_ust_is_pointer_type -
'signed long' is not a pointer
PASS: unit/ust-utils/test_ust_utils 70 - lttng_ust_is_pointer_type -
'signed long long' is not a pointer
PASS: unit/ust-utils/test_ust_utils 71 - lttng_ust_is_pointer_type -
'unsigned char' is not a pointer
PASS: unit/ust-utils/test_ust_utils 72 - lttng_ust_is_pointer_type -
'unsigned short' is not a pointer
PASS: unit/ust-utils/test_ust_utils 73 - lttng_ust_is_pointer_type -
'unsigned int' is not a pointer
PASS: unit/ust-utils/test_ust_utils 74 - lttng_ust_is_pointer_type -
'unsigned long' is not a pointer
PASS: unit/ust-utils/test_ust_utils 75 - lttng_ust_is_pointer_type -
'unsigned long long' is not a pointer
PASS: unit/ust-utils/test_ust_utils 76 - lttng_ust_is_pointer_type -
'int8_t' is not a pointer
PASS: unit/ust-utils/test_ust_utils 77 - lttng_ust_is_pointer_type -
'int16_t' is not a pointer
PASS: unit/ust-utils/test_ust_utils 78 - lttng_ust_is_pointer_type -
'int32_t' is not a pointer
PASS: unit/ust-utils/test_ust_utils 79 - lttng_ust_is_pointer_type -
'int64_t' is not a pointer
PASS: unit/ust-utils/test_ust_utils 80 - lttng_ust_is_pointer_type -
'intmax_t' is not a pointer
PASS: unit/ust-utils/test_ust_utils 81 - lttng_ust_is_pointer_type -
'uint8_t' is not a pointer
PASS: unit/ust-utils/test_ust_utils 82 - lttng_ust_is_pointer_type -
'uint16_t' is not a pointer
PASS: unit/ust-utils/test_ust_utils 83 - lttng_ust_is_pointer_type -
'uint32_t' is not a pointer
PASS: unit/ust-utils/test_ust_utils 84 - lttng_ust_is_pointer_type -
'uint64_t' is not a pointer
PASS: unit/ust-utils/test_ust_utils 85 - lttng_ust_is_pointer_type -
'uintmax_t' is not a pointer
PASS: unit/ust-utils/test_ust_utils 86 - lttng_ust_is_pointer_type -
'float' is not a pointer
PASS: unit/ust-utils/test_ust_utils 87 - lttng_ust_is_pointer_type -
'double' is not a pointer
PASS: unit/ust-utils/test_ust_utils 88 - lttng_ust_is_pointer_type - 'long
double' is not a pointer
PASS: unit/ust-utils/test_ust_utils 89 - lttng_ust_is_pointer_type - 'void
*' is a pointer
PASS: unit/ust-utils/test_ust_utils 90 - lttng_ust_is_pointer_type - 'void
**' is a pointer
PASS: unit/ust-utils/test_ust_utils 91 - lttng_ust_is_pointer_type -
'struct dummy *' is a pointer
PASS: unit/ust-utils/test_ust_utils 92 - lttng_ust_is_pointer_type - 'int
*' is a pointer
PASS: unit/ust-utils/test_ust_utils 93 - lttng_ust_is_pointer_type - 'float
*' is a pointer
PASS: unit/ust-utils/test_ust_utils 94 - lttng_ust_is_pointer_type -
'double *' is a pointer
# regression/abi0-conflict/test_abi0_conflict: LD_PRELOAD
PASS: regression/abi0-conflict/test_abi0_conflict 1 - LD_PRELOAD: no-ust
app works
PASS: regression/abi0-conflict/test_abi0_conflict 2 - LD_PRELOAD: no-ust
app with abi0 preload succeeds
PASS: regression/abi0-conflict/test_abi0_conflict 3 - LD_PRELOAD: no-ust
app with abi0 and abi1 preload fails
PASS: regression/abi0-conflict/test_abi0_conflict 4 - LD_PRELOAD: no-ust
app with abi1 and abi0 preload fails
PASS: regression/abi0-conflict/test_abi0_conflict 5 - LD_PRELOAD: ust app
works
PASS: regression/abi0-conflict/test_abi0_conflict 6 - LD_PRELOAD: ust app
with abi0 preload fails
PASS: regression/abi0-conflict/test_abi0_conflict 7 - LD_PRELOAD: ust app
with abi0 and abi1 preload fails
PASS: regression/abi0-conflict/test_abi0_conflict 8 - LD_PRELOAD: ust app
with abi1 and abi0 preload fails
# regression/abi0-conflict/test_abi0_conflict: dlopen
PASS: regression/abi0-conflict/test_abi0_conflict 9 - dlopen: no-ust app
works
PASS: regression/abi0-conflict/test_abi0_conflict 10 - dlopen: no-ust app
with abi1 and abi1 succeeds
PASS: regression/abi0-conflict/test_abi0_conflict 11 - dlopen: no-ust app
with abi0 and abi1 fails
PASS: regression/abi0-conflict/test_abi0_conflict 12 - dlopen: no-ust app
with abi1 and abi0 fails
PASS: regression/abi0-conflict/test_abi0_conflict 13 - dlopen: ust app
works
PASS: regression/abi0-conflict/test_abi0_conflict 14 - dlopen: ust app with
abi0 fails
PASS: regression/abi0-conflict/test_abi0_conflict 15 - dlopen: ust app with
abi0 and abi1 fails
PASS: regression/abi0-conflict/test_abi0_conflict 16 - dlopen: ust app with
abi1 and abi0 fails
# regression/abi0-conflict/test_abi0_conflict: Indirect linking
PASS: regression/abi0-conflict/test_abi0_conflict 17 - indirect: no-ust app
with abi0 succeeds
PASS: regression/abi0-conflict/test_abi0_conflict 18 - indirect: no-ust app
with abi1 succeeds
PASS: regression/abi0-conflict/test_abi0_conflict 19 - indirect: no-ust app
with abi0 and abi1 fails
PASS: regression/abi0-conflict/test_abi0_conflict 20 - indirect: ust app
with abi0 fails
PASS: regression/abi0-conflict/test_abi0_conflict 21 - indirect: ust app
with abi1 succeeds
PASS: regression/abi0-conflict/test_abi0_conflict 22 - indirect: ust app
with abi0 and abi1 fails
============================================================================

Testsuite summary for lttng-ust 2.13.8
============================================================================

# TOTAL: 278
# PASS:  278
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

make[4]: Leaving directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests'
make[3]: Leaving directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests'
make[2]: Leaving directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests'
make[1]: Leaving directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests'
Making check in extras
make[1]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/extras'
make[1]: Nothing to be done for 'check'.
make[1]: Leaving directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8/extras'
make[1]: Entering directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8'
make[1]: Nothing to be done for 'check-am'.
make[1]: Leaving directory '/home/iadmin/lttng-ust/lttng-ust-2.13.8'


>   - Does make check for lttng-tools pass in your environment?
>

I'm still using lttng-tools built by yocto so I don't know how to run make
check in that situation.  From devtool maybe???  I don't know, I'd have to
look into that one.   I never upgraded lttng-tools either, only upgraded
lttng-ust and built from source tar to get the fix you mentioned for TLS
stuff.

  - Is this reproducible in non-yocto environments or on other
> architectures with the same project?
>

Hmm, that's a good question.  I don't have any other embedded targets to
try this on that are a different arch.  I think I heard at one point early
on a heavily stubbed out version ran on a PC but I'd have to look into that.


>   - Does running the traced application with `LTTNG_UST_DEBUG=1` yield
> more information?
>

So far I've just been trying to get the app to run without even starting
lttng just to get past segfault issue.  Here is what it looks like with
debug turned on without starting lttng or a lttng-sessiond:

iblttng_ust_tracepoint[599/599]: Your compiler treats weak symbols with
hidden visibility for integer objects as SAME address between compile units
part of the same module. (in check_weak_hidden() at tracepoint.c:1012)
liblttng_ust_tracepoint[599/599]: Your compiler treats weak symbols with
hidden visibility for pointer objects as SAME address between compile units
part of the same module. (in check_weak_hidden() at tracepoint.c:1016)
liblttng_ust_tracepoint[599/599]: Your compiler treats weak symbols with
hidden visibility for 24-byte structure objects as SAME address between
compile units part of the same module. (in check_weak_hidden() at
tracepoint.c:1020)
liblttng_ust_tracepoint[599/599]: just registered a tracepoints section
from 0xffffb91aec00 and having 26 tracepoints (in
lttng_ust_tracepoint_module_register() at tracepoint.c:960)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_statedump:end" (in lttng_ust_tracepoint_module_register() at
tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_statedump:procname" (in lttng_ust_tracepoint_module_register()
at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_statedump:debug_link" (in lttng_ust_tracepoint_module_register()
at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_statedump:build_id" (in lttng_ust_tracepoint_module_register()
at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_statedump:bin_info" (in lttng_ust_tracepoint_module_register()
at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_statedump:start" (in lttng_ust_tracepoint_module_register() at
tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_lib:unload" (in lttng_ust_tracepoint_module_register() at
tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_lib:debug_link" (in lttng_ust_tracepoint_module_register() at
tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_lib:build_id" (in lttng_ust_tracepoint_module_register() at
tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_lib:load" (in lttng_ust_tracepoint_module_register() at
tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracef:event" (in lttng_ust_tracepoint_module_register() at
tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_LINE" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_FUNCTION" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_UNIT" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_MODULE" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROCESS" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_PROGRAM" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_DEBUG_SYSTEM" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_INFO" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_NOTICE" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_WARNING" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ERR" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_CRIT" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_ALERT" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust_tracepoint[599/599]: registered tracepoint:
"lttng_ust_tracelog:LTTNG_UST_TRACEPOINT_LOGLEVEL_EMERG" (in
lttng_ust_tracepoint_module_register() at tracepoint.c:968)
liblttng_ust[599/599]: Provider "lttng_ust_statedump" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_statedump" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_statedump" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_statedump" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_statedump" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_statedump" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_statedump" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: adding probe lttng_ust_statedump containing 6 events
to lazy registration list (in lttng_ust_probe_register() at
lttng-probes.c:254)
liblttng_ust[599/599]: Provider "lttng_ust_lib" accepted, version 3.0 is
compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_lib" accepted, version 3.0 is
compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_lib" accepted, version 3.0 is
compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_lib" accepted, version 3.0 is
compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_lib" accepted, version 3.0 is
compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: adding probe lttng_ust_lib containing 4 events to
lazy registration list (in lttng_ust_probe_register() at
lttng-probes.c:254)
liblttng_ust[599/599]: Provider "lttng_ust_tracef" accepted, version 3.0 is
compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracef" accepted, version 3.0 is
compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: adding probe lttng_ust_tracef containing 1 events to
lazy registration list (in lttng_ust_probe_register() at
lttng-probes.c:254)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: Provider "lttng_ust_tracelog" accepted, version 3.0
is compatible with LTTng UST provider version 3.0. (in
check_provider_version() at lttng-probes.c:50)
liblttng_ust[599/599]: adding probe lttng_ust_tracelog containing 15 events
to lazy registration list (in lttng_ust_probe_register() at
lttng-probes.c:254)
liblttng_ust[599/599]: dlopened liblttng-ust shared library
(liblttng-ust.so.1). (in lttng_ust_ctor() at lttng-ust-comm.c:2217)
libringbuffer-clients[599/599]: LTT : ltt ring buffer client
"relay-metadata-mmap" init
(in lttng_ring_buffer_metadata_client_init() at
../../src/common/ringbuffer-clients/metadata-template.h:364)
libringbuffer-clients[599/599]: LTT : ltt ring buffer client
"relay-overwrite-mmap" init
(in lttng_ring_buffer_client_overwrite_init() at
../../src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[599/599]: LTT : ltt ring buffer client
"relay-overwrite-rt-mmap" init
(in lttng_ring_buffer_client_overwrite_rt_init() at
../../src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[599/599]: LTT : ltt ring buffer client
"relay-discard-mmap" init
(in lttng_ring_buffer_client_discard_init() at
../../src/common/ringbuffer-clients/template.h:826)
libringbuffer-clients[599/599]: LTT : ltt ring buffer client
"relay-discard-rt-mmap" init
(in lttng_ring_buffer_client_discard_rt_init() at
../../src/common/ringbuffer-clients/template.h:826)
liblttng_ust[599/601]: Info: sessiond not accepting connections to local
apps socket (in ust_listener_thread() at lttng-ust-comm.c:1884)
liblttng_ust[599/601]: Waiting for local apps sessiond (in
wait_for_sessiond() at lttng-ust-comm.c:1758)
liblttng_ust[599/600]: Info: sessiond not accepting connections to global
apps socket (in ust_listener_thread() at lttng-ust-comm.c:1884)
Segmentation fault


>   - I'd also run lttng-sessiond with the environment variable
> `LTTNG_UST_DEBUG=1` set and `-vvv --verbose-consumer` in the program
> arguments and capture for stdout & stderr into a file for analysis.
>   - Verify if the main program is dynamically linked with `ldd`
>   - Verify which libraries are loaded at runtime and which calls are
> shimmed with `LD_DEBUG=libs,binding`
>

I provided some of this above.
The output of LD_DEBUG=libs,bindings is attached as file
ld_debug-libs-bindings.txt
in attached tar file (had to use compression as system rejected this post
as it was too large).


>   - Review in detail which gcc commands are executed to produce the
> tracepoint provider and link it to the main executable.
>

Just like the hello example in the documentation


>
> My current understanding is that the statedump tracepoints are
> registered and those events communicated before the main program's init
> is run (at least for C programs). If you do the following test, I think
> you should be able to see the statedumps. Could you confirm if you have
> them or not?
>
>
> ```
> LTTNG_UST_DEBUG=1 lttng-sessiond -vvv --verbose-consumer &>
> /tmp/sessiond.log &
> lttng create
> lttng enable-event -u --all
> lttng start
>
> unset LTTNG_UST_WITHOUT_BADDR_STATEDUMP
> unset LTTNG_UST_WITHOUT_PROCNAME_STATEDUMP
> LTTNG_UST_DEBUG=1 LTTNG_UST_REGISTER_TIMEOUT=-1 ./my_app
> # Segfault?
>

Yes, still segfault.  The sessiond.log is in attached tar file.


>
> lttng stop
> lttng view # Here you should have the statedump events
> killall lttng-sessiond
>

So the embedded target doesn't have babletrace so I archived the session
and transferred it over to my dev PC and used tracecompas to view the
lttng-session.

You can see that in the my_session.csv file in the attached tar.


> ```
>
> >
> > Thanks!  I'm running out of things to try.
>
> Ultimately, given the bespoke environment, build steps, and application
> it's tough to diagnose a lot of things that we would go over with a fine
> tooth comb: seeing how the TPPs are built and linked, seeing how the
> application is built and linked, analyzing what's happening at runtime,
> having a coredump that allows us to see the variables, etc.
>
> If you can provide a minimal reproducer, it's possible to dig further
> into it.
>
> Otherwise, to look in more detail at your specific project would be
> covered under a service contract (for which we can sign NDAs, etc. as
> needed). Feel free to reach out the sales@efficios.com to organise that.
>

I understand but so far schedules & budget/priorities haven't aligned so
once again I've been scheduled to look at this issue again since we last
tried it on older Dunfell OS with older versions of lttng and it looks like
we have the same problem as before.

Thanks for your help!

Regards,

Brian


>
> >
> > I don't fully understand the implications of some of this documentation,
> > but I have learned enough to know LD_PRELOAD means nothing if everything
> > is static built.  Our application is heavily multi threaded, uses fork,
> > clone and who knows if it's doing double closes or not ... so I've tried
> > these LD_PRELOAD "helpers" or whatever they are, before and didn't get a
> > different result, but now know it's only for shared library object.  So
> > now I will experiment with making our tpp a shared object and try
> > LD_PRELOAD of fork, fd and pthread helpers and starting my app that way.
>
> LD_PRELOAD has an effect if the application you are running is
> dynamically linked (note: the TPP can be statically linked inside an
> dynamically linked executable, and the preloads are still useful and/or
> needed in that case).
>
> The function of the helpers are to ensure that various system calls
> don't clobber things in use by lttng-ust. E.g., close() is shimmed by
> liblttng-ust-fd.so so that FDs that lttng-ust uses aren't suddenly shut
> by the program.
>
> If you want to see it in action, you could try an experiment with a
> slightly modified hello-static-lib application, with instructions in the
> top comment:
> https://gist.github.com/kienanstewart/299eb6a511d92d458569b210e4a418a8
>
> The effect of the preloads such as liblttng-ust-fd.so are independent of
> how the TPP is built and linked to the application or library.
>
> thanks,
> kienan
>
> >
> > This is what I'm referring to above from the lttng docs:
> >
> >
> >           Use LTTng-UST with daemons
> >           <
> https://lttng.org/docs/v2.13/#doc-using-lttng-ust-with-daemons>
> >
> > If your instrumented application calls fork(2)
> > <https://man7.org/linux/man-pages/man2/fork.2.html>, clone(2)
> > <https://man7.org/linux/man-pages/man2/clone.2.html>, or BSD’s rfork(2)
> > <
> http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>,
> without a following exec(3) <
> https://man7.org/linux/man-pages/man3/exec.3.html>-family system call,
> you must preload the |liblttng-ust-fork.so| shared object when you start
> the application.
> >
> > LD_PRELOAD=liblttng-ust-fork.so ./my-app
> >
> > If your tracepoint provider package is a shared library which you also
> > preload, you must put both shared objects in |LD_PRELOAD|:
> >
> > LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
> >
> >
> >           Use LTTng-UST with applications which close file descriptors
> >           that don’t belong to them
> >           <https://lttng.org/docs/v2.13/#doc-liblttng-ust-fd>
> >
> > Since 2.9
> >
> > If your instrumented application closes one or more file descriptors
> > which it did not open itself, you must preload the |liblttng-ust-fd.so|
> > shared object when you start the application:
> >
> > LD_PRELOAD=liblttng-ust-fd.so ./my-app
> >
> > Typical use cases include closing all the file descriptors after fork(2)
> > <https://man7.org/linux/man-pages/man2/fork.2.html> or rfork(2)
> > <
> http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>
> and buggy applications doing “double closes”.
> >
> >
> > Thanks,
> >
> > Brian
> >
> >
>

[-- Attachment #1.2: Type: text/html, Size: 84554 bytes --]

[-- Attachment #2: log-files.tar.xz --]
[-- Type: application/x-xz, Size: 61500 bytes --]

[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Kienan Stewart via lttng-dev]

Hi Brian,

On 7/22/24 6:00 PM, Brian Hutchinson wrote:
> Hi Kienan,
> 
> Took a while to gather your grocery list but I think I have most of it 
> below ;)

thanks for all the extra info. Replies inline below, but I'll cut a lot 
of the long output for readability.

tl;dr the environment continues to be weird, but my present suspicion is 
that something in either compilation, the linking of your app (eg. with 
ld when producing the executable), or some post linking stripping might 
be causing issues.

I will stop digging into further hypotheticals on my side as there is no 
reproducer for both the environment and the application. If you ever end 
up with a minimal reproducer that you can share, I'd be more than happy 
to examine it.

> 
> I may have not been clear.  Most of the application components are 
> statically linked but I think there are some that are built as shared 
> objects (.so's) so that's what I was referring to.  I know that 
> lttng-ust is dynamically linked ... I think the lttng-ust docs say this 
> is only option but also makes reference to the fact static linking was 
> once possible (in some versions of the documentation) but not supported 
> anymore (I probably have the docs memorized by now ha, ha ... I've 
> looked at many, many versions of them).
> 
> Just for full disclosure my ldd looks like:
> 
>         linux-vdso.so.1 (0x0000ffffab196000)
>         libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
>         liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1 
> (0x0000ffffa5750000)
>         libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
>         librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
>         libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
>         libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
>         libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
>         libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
>         /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
>         liblttng-ust-common.so.1 => 
> /usr/local/lib/liblttng-ust-common.so.1 (0x0000ffffa50e0000)
>         liblttng-ust-tracepoint.so.1 => 
> /usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
>         libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
>         libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)
> 
> 

I find it very suspicious that `liblttng-ust.so.1` is in `/usr/lib`, 
while the other lttng-ust libraries are being loaded from `/usr/local/lib`.

This information also matches the statedump and the LD_DEBUG info from 
later on.

Could you verify some of the following information:

1. In your build root for lttng-ust, enumerate all the liblttng*so 
files. For each shared object, run `file $libname` and record the value 
of the BuildID hash.

2. In your deployed environment, enumerate all the liblttng*so files in 
/usr/local/lib and record the value of the BuildID hash.

3. In your deployed environment, enumerate all the liblttng*so files in 
/usr/lib and record the value of the BuildID hash.

The BuildID will vary from one object to the next, but the deployed 
environment BuildIDs should match the BuildIDs of the corresponding 
library from the build root.

In particular my interest is the `liblttng-ust.so.1` library  that is 
loaded  from `/usr/lib` in your deployed environment, and if there are 
any in `/usr/local/lib`.

In the working hello example, does it also end up linking against 
`/usr/lib/liblttng-ust.so.1`?

>     Further along in stepping back:
> 
>        - Does make check for lttng-ust pass in your environment?
> 
> 
> So I'm glad you mentioned this.  I didn't try to run make check because 
> #1 have never seen anything in the documentation about it and what it 
> does and #2 It said that it depended on Perl and since this is an 
> embedded system, I really didn't want to pull it and its dependencies in.
> 
> Nevertheless, I ran it on my native target build I setup to see what 
> would happen.  I didn't notice it complaining about perl so then I 
> checked my manifest and perl is in fact in our image (I guess some other 
> recipe pulled it in and I never paid attention to that):
> 
> Making check in include
> make[1]: Entering directory 
> '/home/iadmin/lttng-ust/lttng-ust-2.13.8/include'
> ...
 >
> 
>        - Does make check for lttng-tools pass in your environment?
> 
> I'm still using lttng-tools built by yocto so I don't know how to run 
> make check in that situation.  From devtool maybe???  I don't know, I'd 
> have to look into that one.   I never upgraded lttng-tools either, only 
> upgraded lttng-ust and built from source tar to get the fix you 
> mentioned for TLS stuff.

The poky version of the tools recipe seems to have some stuff relating 
to ptest, which seems to be a yocto tool for package testing. There 
might be a run to run that?

https://git.yoctoproject.org/poky/plain/meta/recipes-kernel/lttng/lttng-tools_2.13.13.bb
https://wiki.yoctoproject.org/wiki/Ptest

Otherwise I would get the lttng-tools source for your version, 
bootstrap, configure, build, and run the tests in tree.

The bulk of regression testing for lttng-ust, -modules, and -tools 
happens during `make check` for lttng-tools.

> 
>        - Is this reproducible in non-yocto environments or on other
>     architectures with the same project?
> 
> Hmm, that's a good question.  I don't have any other embedded targets to 
> try this on that are a different arch.  I think I heard at one point 
> early on a heavily stubbed out version ran on a PC but I'd have to look 
> into that.
> 
>        - Does running the traced application with `LTTNG_UST_DEBUG=1` yield
>     more information?
> 
> 
> So far I've just been trying to get the app to run without even starting 
> lttng just to get past segfault issue.  Here is what it looks like with 
> debug turned on without starting lttng or a lttng-sessiond:
> 
> liblttng_ust_tracepoint[599/599]: Your compiler treats weak symbols with 
> hidden visibility for integer objects as SAME address between compile 
> units part of the same module. (in check_weak_hidden() at 
>
> liblttng_ust[599/601]: Waiting for local apps sessiond (in 
> wait_for_sessiond() at lttng-ust-comm.c:1758)
> liblttng_ust[599/600]: Info: sessiond not accepting connections to 
> global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1884)
> Segmentation fault
> 
>        - I'd also run lttng-sessiond with the environment variable
>     `LTTNG_UST_DEBUG=1` set and `-vvv --verbose-consumer` in the program
>     arguments and capture for stdout & stderr into a file for analysis.
>        - Verify if the main program is dynamically linked with `ldd`
>        - Verify which libraries are loaded at runtime and which calls are
>     shimmed with `LD_DEBUG=libs,binding`
> 
> 
> I provided some of this above.
> The output of LD_DEBUG=libs,bindings is attached as file 
> ld_debug-libs-bindings.txt

Thanks. For your own debugging, I would recommend to run the hello world 
example with the same environment variables in order to have comparison 
point to a working example.

 From my observations in my own environment, for the hello example (and 
presumably your app if it's set up in the same way) I would expect 
something similar to the following

```
   3624895:     initialize program: ./hello
    3624895:
    3624895:     binding file ./hello [0] to 
/lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `dlopen' [GLIBC_2.34]
    3624895:     opening 
file=/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]; 
direct_opencount=3
    3624895:
    3624895:     binding file ./hello [0] to 
/lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `dlsym' [GLIBC_2.34]
    3624895:     binding file 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol 
`lttng_ust_tp_rcu_re
ad_lock'
    3624895:     binding file 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol 
`lttng_ust_tp_rcu_re
ad_unlock'
    3624895:     binding file 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol 
`lttng_ust_tp_rcu_de
reference_sym'
    3624895:     binding file 
x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol 
`lttng_ust_tracepoin
t_module_register'
    3624895:     binding file 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol 
`lttng_ust_tracepoin
t_module_unregister'
    3624895:     binding file 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol 
`lttng_ust_tp_disabl
e_destructors'
    3624895:     binding file 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to 
/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol 
`lttng_ust_tp_get_de
structors_state'
    3624895:     binding file ./hello [0] to 
/x/lttng/master/usr/lib/liblttng-ust.so.1 [0]: normal symbol 
`lttng_ust_probe_register'
    3624895:
    3624895:     transferring control: ./hello
    3624895:
```

The bit that is missing from your program is the binding of 
`lttng_ust_probe_register`. Here, it's not like the symbol isn't 
resolved at all: it's found earlier and if it didn't exist at all the 
error would probably be relatively clear like:

```
./doc/examples/hello/hello: symbol lookup error: 
/x/lttng/master/usr/lib/liblttng-ust.so.1: undefined symbol: 
lttng_ust_probe_register
```

I'd like you to run another test with your app. This should indicate 
that all symbols should be immediately resolved during dynamic loading 
rather than performing lazy symbol resolution. The goal of this test is 
to separate clearly the symbol resolution from the call that happens in 
the program's initialization. You can also run it with the hello demo to 
compare the runs.

```
LTTNG_UST_DEBUG=1 LD_DEBUG=all LD_BIND_NOW=1 ./my_app
```

> 
>        - Review in detail which gcc commands are executed to produce the
>     tracepoint provider and link it to the main executable.
> 
> Exactly like hello world example.

Okay. I see your program also has at least some portions in C++, which 
is different than the hello world example.

Does your build process use particular optimizations or strip parts from 
the resulting object files?

When you're building the hello world example, does it get compiled in 
the same buildroot as your app, or is it done differently?

Do you use a custom linker script(s)?


> 
> 
> 
> So the embedded target doesn't have babletrace so I archived the session 
> and transferred it over to my dev PC and used tracecompas to view the 
> lttng-session.
> 
> You can see that in the attached file my_session.csv

This is good, the statedump is happening properly.

> 
>     ```
> 
>      >
>      > Thanks!  I'm running out of things to try.
> 

For the next steps I would start disassembling "my_app" or checking in 
gdb which bad address is being fed into strnlen.

Sections of interest (`objdump -sD my_app` _and_ `readelf -W -a my_app`):

- lttng_ust_tracepoints_ptrs
- lttng_ust_tracepoints_strings

Symbols of interest (`objdump -sD my_app` _and `nm my_app`):

- lttng_ust_tracepoint_hello_world___my_first_tracepoint
- lttng_ust_tracepoints_ptrs
- __start_lttng_ust_tracepoints_ptrs
- __stop_lttng_ust_tracepoints_ptrs

thanks,
kienan

>     Ultimately, given the bespoke environment, build steps, and application
>     it's tough to diagnose a lot of things that we would go over with a
>     fine
>     tooth comb: seeing how the TPPs are built and linked, seeing how the
>     application is built and linked, analyzing what's happening at runtime,
>     having a coredump that allows us to see the variables, etc.
> 
>     If you can provide a minimal reproducer, it's possible to dig further
>     into it.
> 
>     Otherwise, to look in more detail at your specific project would be
>     covered under a service contract (for which we can sign NDAs, etc. as
>     needed). Feel free to reach out the sales@efficios.com
>     <mailto:sales@efficios.com> to organise that.
> 
> 
> I understand but so far schedules & budget/priorities haven't aligned so 
> once again I've been scheduled to look at this issue again since we last 
> tried it on older Dunfell OS with older versions of lttng and it looks 
> like we have the same problem as before.
> 
> Thanks for your help!
> 
> Regards,
> 
> Brian
> 
> 
>      >
>      > I don't fully understand the implications of some of this
>     documentation,
>      > but I have learned enough to know LD_PRELOAD means nothing if
>     everything
>      > is static built.  Our application is heavily multi threaded, uses
>     fork,
>      > clone and who knows if it's doing double closes or not ... so
>     I've tried
>      > these LD_PRELOAD "helpers" or whatever they are, before and
>     didn't get a
>      > different result, but now know it's only for shared library
>     object.  So
>      > now I will experiment with making our tpp a shared object and try
>      > LD_PRELOAD of fork, fd and pthread helpers and starting my app
>     that way.
> 
>     LD_PRELOAD has an effect if the application you are running is
>     dynamically linked (note: the TPP can be statically linked inside an
>     dynamically linked executable, and the preloads are still useful and/or
>     needed in that case).
> 
>     The function of the helpers are to ensure that various system calls
>     don't clobber things in use by lttng-ust. E.g., close() is shimmed by
>     liblttng-ust-fd.so so that FDs that lttng-ust uses aren't suddenly shut
>     by the program.
> 
>     If you want to see it in action, you could try an experiment with a
>     slightly modified hello-static-lib application, with instructions in
>     the
>     top comment:
>     https://gist.github.com/kienanstewart/299eb6a511d92d458569b210e4a418a8 <https://gist.github.com/kienanstewart/299eb6a511d92d458569b210e4a418a8>
> 
>     The effect of the preloads such as liblttng-ust-fd.so are
>     independent of
>     how the TPP is built and linked to the application or library.
> 
>     thanks,
>     kienan
> 
>      >
>      > This is what I'm referring to above from the lttng docs:
>      >
>      >
>      >           Use LTTng-UST with daemons
>      >         
>       <https://lttng.org/docs/v2.13/#doc-using-lttng-ust-with-daemons
>     <https://lttng.org/docs/v2.13/#doc-using-lttng-ust-with-daemons>>
>      >
>      > If your instrumented application calls fork(2)
>      > <https://man7.org/linux/man-pages/man2/fork.2.html
>     <https://man7.org/linux/man-pages/man2/fork.2.html>>, clone(2)
>      > <https://man7.org/linux/man-pages/man2/clone.2.html
>     <https://man7.org/linux/man-pages/man2/clone.2.html>>, or BSD’s
>     rfork(2)
>      >
>     <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>>, without a following exec(3) <https://man7.org/linux/man-pages/man3/exec.3.html <https://man7.org/linux/man-pages/man3/exec.3.html>>-family system call, you must preload the |liblttng-ust-fork.so| shared object when you start the application.
>      >
>      > LD_PRELOAD=liblttng-ust-fork.so ./my-app
>      >
>      > If your tracepoint provider package is a shared library which you
>     also
>      > preload, you must put both shared objects in |LD_PRELOAD|:
>      >
>      > LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
>      >
>      >
>      >           Use LTTng-UST with applications which close file
>     descriptors
>      >           that don’t belong to them
>      >           <https://lttng.org/docs/v2.13/#doc-liblttng-ust-fd
>     <https://lttng.org/docs/v2.13/#doc-liblttng-ust-fd>>
>      >
>      > Since 2.9
>      >
>      > If your instrumented application closes one or more file descriptors
>      > which it did not open itself, you must preload the
>     |liblttng-ust-fd.so|
>      > shared object when you start the application:
>      >
>      > LD_PRELOAD=liblttng-ust-fd.so ./my-app
>      >
>      > Typical use cases include closing all the file descriptors after
>     fork(2)
>      > <https://man7.org/linux/man-pages/man2/fork.2.html
>     <https://man7.org/linux/man-pages/man2/fork.2.html>> or rfork(2)
>      >
>     <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>> and buggy applications doing “double closes”.
>      >
>      >
>      > Thanks,
>      >
>      > Brian
>      >
>      >
> 
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev]

Hi Kienan,

I'll answer your questions below, but I've got questions on what I saw
building and installing lttng-tools (2.13.13) and lttng-ust (2.13.8).

Based on the struggles I've had trying to get lttng to work with my
app over various Yocto versions (Dunfell & Kirkstone) and lttng
version, I think the problems I'm facing are mostly around C++ and
weak and hidden symbols in Yocto toolchain.

When I started my app with the options you mentioned previously a
while back, Id see things like:

# LTTNG_UST_DEBUG=1 LTTNG_UST_REGISTER_TIMEOUT=-1 /opt/tc/TrafficController
liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
with hidden visibility for integer objects as SAME address between
compile units part of the same module. (in check_weak_hidden() at
tracepoint.c:1012)
liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
with hidden visibility for pointer objects as SAME address between
compile units part of the same module. (in check_weak_hidden() at
tracepoint.c:1016)
liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
with hidden visibility for 24-byte structure objects as SAME address
between compile units part of the same module. (in check_weak_hidden()
at tracepoint.c:1020)

I further researched this whole 'weak symbol' and 'hidden visibility'
topic in the lttng-dev archives and it smells a lot like what I've
been seeing.  You should be able to mix both tracef and tracepoint
calls in souce code ... but I could not.  I could get a tracef call to
work but if I put a tracepoint call in the same code then nothing
would work.  This was with Dunfell 3.1.7 and earlier versions of
lttng.

At one point I could get a tracepoint call to work but I'd have to let
our cmake build system build and link the tpp.c file and then turn
around and use gcc to recompile it and copy it to where all the
objects were to create the huge .a library the app was built against.
That's when I first learned there are issues with C++.  I think g++ is
used to build even .c files that aren't c++.

Then if I tried to put a tracepoint in another sub project, none of
the tracepoints would work and I'd get empty traces.  This is a
symptom of the 'weak symbols with hidden visibility' issue ... and I
finally found others that were having same issue in the archives.  I
don't fully understand the issues here, although I do understand some
of what's going on ... I just don't know what to do about it.

At this point I was being encouraged to keep upgrading to newer
versions of lttng.  Our app never changed, gcc & lttng etc., kept
changing.  Now with newer versions nothing runs, all I get is an
immediate segfault.  Again, I'm building just like I did before a year
or so ago with older versions of Yocto and lttng.  I say all of that
to give perspective and history of what I've seen and experienced.
Now this TLS thing has entered the picture too and so far I've only
changed lttng, I don't know if I should be applying patches to my gcc
for that issue.  Like I said, I'm currently using Yocto Kirkstone
4.0.18 and 6.1.38 kernel.

Now I'll move into the area of things I've seen building/installing
lttng-tools and lttng-ust natively on the target environment I've
setup where I can run 'make check' etc.  These are in the category of
"hey, is this ok, should I be worried about this":

While building lttng-tools I see things like:

*** Warning: Linking the executable userspace-probe-elf-binary against
the loadable module
*** libfoo.so is not portable!

In file included from ../../../src/common/macros.h:15,
                from ../../../include/lttng/health-internal.h:19,
                from lttng-ctl-health.c:19:
In function 'lttng_strnlen',
   inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
   inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
   inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
../../../src/common/compat/string.h:19:16: warning: 'strnlen'
specified bound 4096 may exceed source size 37 [-Wstringop-overread]
  19 |         return strnlen(str, max);
     |                ^~~~~~~~~~~~~~~~~
lttng-ctl-health.c: At top level:
cc1: note: unrecognized command-line option
'-Wno-incomplete-setjmp-declaration' may have been intended to silence
earlier diagnostics

** Warning: Linking the shared library libbar.la against the loadable module
*** libzzz.so is not portable!

*** Warning: Linking the shared library libfoo.la against the loadable module
*** libbar.so is not portable!

While installing lttng-tools I see things like this:

make[4]: Entering directory '/opt/lttng/lttng-tools-2.13.13/src/lib/lttng-ctl'
 CC       lttng-ctl.lo
 CC       snapshot.lo
 CC       lttng-ctl-health.lo
In file included from ../../../src/common/macros.h:15,
                from ../../../include/lttng/health-internal.h:19,
                from lttng-ctl-health.c:19:
In function 'lttng_strnlen',
   inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
   inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
   inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
../../../src/common/compat/string.h:19:16: warning: 'strnlen'
specified bound 4096 may exceed source size 37 [-Wstringop-overread]
  19 |         return strnlen(str, max);
     |                ^~~~~~~~~~~~~~~~~
lttng-ctl-health.c: At top level:
cc1: note: unrecognized command-line option
'-Wno-incomplete-setjmp-declaration' may have been intended to silence
earlier diagnostics

Making install in trigger-condition-event-matches
make[2]: Entering directory
'/opt/lttng/lttng-tools-2.13.13/doc/examples/trigger-condition-event-matches'
 CC       instrumented-app.o
 CC       tracepoint-trigger-example.o
 AR       libtracepoint-trigger-example.a
ar: `u' modifier ignored since `D' is the default (see `U')

While building lttng-ust I see things like:

Making all in utils
make[2]: Entering directory
'/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
 CC       tap.o
 AR       libtap.a
ar: `u' modifier ignored since `D' is the default (see `U')

*** Warning: Linking the shared library libzero.la against the
loadable module
*** libfakeust0.so is not portable!
 CCLD     app_noust_indirect_abi0

*** Warning: Linking the executable app_noust_indirect_abi0 against
the loadable module
*** libzero.so is not portable!
 CC       app_noust_indirect_abi0_abi1-app_noust.o
 CC       libone.lo
 CCLD     libone.la
 CCLD     app_noust_indirect_abi0_abi1

*** Warning: Linking the executable app_noust_indirect_abi0_abi1
against the loadable module
*** libzero.so is not portable!

*** Warning: Linking the executable app_noust_indirect_abi0_abi1
against the loadable module
*** libone.so is not portable!
 CC       app_noust_indirect_abi1-app_noust.o
 CCLD     app_noust_indirect_abi1

*** Warning: Linking the executable app_noust_indirect_abi0_abi1
against the loadable module
*** libone.so is not portable!
 CC       app_noust_indirect_abi1-app_noust.o
 CCLD     app_noust_indirect_abi1

*** Warning: Linking the executable app_noust_indirect_abi1 against
the loadable module
*** libone.so is not portable!
 CC       app_ust.o
 CC       tp.o
 CCLD     app_ust
 CC       app_ust_dlopen.o
 CCLD     app_ust_dlopen
 CC       app_ust_indirect_abi0-app_ust.o
 CC       app_ust_indirect_abi0-tp.o
 CCLD     app_ust_indirect_abi0

*** Warning: Linking the executable app_ust_indirect_abi0 against the
loadable module
*** libzero.so is not portable!
 CC       app_ust_indirect_abi0_abi1-app_ust.o
 CC       app_ust_indirect_abi0_abi1-tp.o
 CCLD     app_ust_indirect_abi0_abi1

*** Warning: Linking the executable app_ust_indirect_abi0_abi1 against
the loadable module
*** libzero.so is not portable!

I don't know if these are ok or if I should be worried about any of that.

... now on to your questions below.



On Wed, Jul 24, 2024 at 12:04 PM Kienan Stewart <kstewart@efficios.com> wrote:
>
> Hi Brian,
>
> On 7/22/24 6:00 PM, Brian Hutchinson wrote:
> > Hi Kienan,
> >
> > Took a while to gather your grocery list but I think I have most of it
> > below ;)
>
> thanks for all the extra info. Replies inline below, but I'll cut a lot
> of the long output for readability.
>
> tl;dr the environment continues to be weird, but my present suspicion is
> that something in either compilation, the linking of your app (eg. with
> ld when producing the executable), or some post linking stripping might
> be causing issues.

I'm not aware of any stripping that's going on.  In fact everything is
being built with debug symbols at the moment and I even turned off
optimization ... even used the debug friendly -O flag to see if that
made a difference.

>
> I will stop digging into further hypotheticals on my side as there is no
> reproducer for both the environment and the application. If you ever end
> up with a minimal reproducer that you can share, I'd be more than happy
> to examine it.

I'm planning on trying to make a small reproducer I can share but not there yet.

>
> >
> > I may have not been clear.  Most of the application components are
> > statically linked but I think there are some that are built as shared
> > objects (.so's) so that's what I was referring to.  I know that
> > lttng-ust is dynamically linked ... I think the lttng-ust docs say this
> > is only option but also makes reference to the fact static linking was
> > once possible (in some versions of the documentation) but not supported
> > anymore (I probably have the docs memorized by now ha, ha ... I've
> > looked at many, many versions of them).
> >
> > Just for full disclosure my ldd looks like:
> >
> >         linux-vdso.so.1 (0x0000ffffab196000)
> >         libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
> >         liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1
> > (0x0000ffffa5750000)
> >         libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
> >         librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
> >         libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
> >         libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
> >         libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
> >         libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
> >         /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
> >         liblttng-ust-common.so.1 =>
> > /usr/local/lib/liblttng-ust-common.so.1 (0x0000ffffa50e0000)
> >         liblttng-ust-tracepoint.so.1 =>
> > /usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
> >         libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
> >         libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)
> >
> >
>
> I find it very suspicious that `liblttng-ust.so.1` is in `/usr/lib`,
> while the other lttng-ust libraries are being loaded from `/usr/local/lib`.

So Yocto puts all of the lttng libs into /usr/lib.  When I sent the
previous info I was using lttng-tools and modules built by Yocto/OE
and I setup a native build environment on the target so I could run
'make check' etc., and that's why there were things in /usr/local/lib
because that's where you guys want stuff to be.  So I actually left
the lttng-ust installables in /usr/local/build but also copied them to
/usr/lib to overwrite old Yocto versions there.

>
> This information also matches the statedump and the LD_DEBUG info from
> later on.
>
> Could you verify some of the following information:
>
> 1. In your build root for lttng-ust, enumerate all the liblttng*so
> files. For each shared object, run `file $libname` and record the value
> of the BuildID hash.

Sorry, I'm not following you here.  The only buildID hash I can think
of is with 'eu-unstrip -n' but that's on core files, not individual
libs.  And looking at the options I have for 'file' on my target, I
don't see anything that looks like what you are asking.

>
> 2. In your deployed environment, enumerate all the liblttng*so files in
> /usr/local/lib and record the value of the BuildID hash.
>
> 3. In your deployed environment, enumerate all the liblttng*so files in
> /usr/lib and record the value of the BuildID hash.
>
> The BuildID will vary from one object to the next, but the deployed
> environment BuildIDs should match the BuildIDs of the corresponding
> library from the build root.
>
> In particular my interest is the `liblttng-ust.so.1` library  that is
> loaded  from `/usr/lib` in your deployed environment, and if there are
> any in `/usr/local/lib`.

As I said above, when I built lttng-ust from source natively on the
target and installed it, it went to /usr/local/lib, your default.  I
then copied everything from /usr/local/lib (which was just lttng-ust
stuff) to /usr/lib to overwrite the old versions Yocto put there.

>
> In the working hello example, does it also end up linking against
> `/usr/lib/liblttng-ust.so.1`?

Yes

>
> >     Further along in stepping back:
> >
> >        - Does make check for lttng-ust pass in your environment?
> >
> >
> > So I'm glad you mentioned this.  I didn't try to run make check because
> > #1 have never seen anything in the documentation about it and what it
> > does and #2 It said that it depended on Perl and since this is an
> > embedded system, I really didn't want to pull it and its dependencies in.
> >
> > Nevertheless, I ran it on my native target build I setup to see what
> > would happen.  I didn't notice it complaining about perl so then I
> > checked my manifest and perl is in fact in our image (I guess some other
> > recipe pulled it in and I never paid attention to that):
> >
> > Making check in include
> > make[1]: Entering directory
> > '/home/iadmin/lttng-ust/lttng-ust-2.13.8/include'
> > ...
>  >
> >
> >        - Does make check for lttng-tools pass in your environment?
> >
> > I'm still using lttng-tools built by yocto so I don't know how to run
> > make check in that situation.  From devtool maybe???  I don't know, I'd
> > have to look into that one.   I never upgraded lttng-tools either, only
> > upgraded lttng-ust and built from source tar to get the fix you
> > mentioned for TLS stuff.
>
> The poky version of the tools recipe seems to have some stuff relating
> to ptest, which seems to be a yocto tool for package testing. There
> might be a run to run that?
>
> https://git.yoctoproject.org/poky/plain/meta/recipes-kernel/lttng/lttng-tools_2.13.13.bb
> https://wiki.yoctoproject.org/wiki/Ptest
>
> Otherwise I would get the lttng-tools source for your version,
> bootstrap, configure, build, and run the tests in tree.
>
> The bulk of regression testing for lttng-ust, -modules, and -tools
> happens during `make check` for lttng-tools.

I did all of that.  First I did it with lttng-ust, building it from
source tarball on target natively to eliminate any Yocto recipe/patch
issues.  Then did the same with lttng-tools.

>
> >
> >        - Is this reproducible in non-yocto environments or on other
> >     architectures with the same project?
> >
> > Hmm, that's a good question.  I don't have any other embedded targets to
> > try this on that are a different arch.  I think I heard at one point
> > early on a heavily stubbed out version ran on a PC but I'd have to look
> > into that.
> >
> >        - Does running the traced application with `LTTNG_UST_DEBUG=1` yield
> >     more information?
> >
> >
> > So far I've just been trying to get the app to run without even starting
> > lttng just to get past segfault issue.  Here is what it looks like with
> > debug turned on without starting lttng or a lttng-sessiond:
> >
> > liblttng_ust_tracepoint[599/599]: Your compiler treats weak symbols with
> > hidden visibility for integer objects as SAME address between compile
> > units part of the same module. (in check_weak_hidden() at
> >
> > liblttng_ust[599/601]: Waiting for local apps sessiond (in
> > wait_for_sessiond() at lttng-ust-comm.c:1758)
> > liblttng_ust[599/600]: Info: sessiond not accepting connections to
> > global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1884)
> > Segmentation fault
> >
> >        - I'd also run lttng-sessiond with the environment variable
> >     `LTTNG_UST_DEBUG=1` set and `-vvv --verbose-consumer` in the program
> >     arguments and capture for stdout & stderr into a file for analysis.
> >        - Verify if the main program is dynamically linked with `ldd`
> >        - Verify which libraries are loaded at runtime and which calls are
> >     shimmed with `LD_DEBUG=libs,binding`
> >
> >
> > I provided some of this above.
> > The output of LD_DEBUG=libs,bindings is attached as file
> > ld_debug-libs-bindings.txt
>
> Thanks. For your own debugging, I would recommend to run the hello world
> example with the same environment variables in order to have comparison
> point to a working example.

I ran the hello world in the example directory
lttng-ust-2.13.8/doc/examples/hello-static-lib and it ran fine.

I can send the sessiond.log etc. if you want to see it but it ran just
fine on target using the native built lttng-ust and lttng-tools.

>
>  From my observations in my own environment, for the hello example (and
> presumably your app if it's set up in the same way) I would expect
> something similar to the following
>
> ```
>    3624895:     initialize program: ./hello
>     3624895:
>     3624895:     binding file ./hello [0] to
> /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `dlopen' [GLIBC_2.34]
>     3624895:     opening
> file=/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0];
> direct_opencount=3
>     3624895:
>     3624895:     binding file ./hello [0] to
> /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `dlsym' [GLIBC_2.34]
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tp_rcu_re
> ad_lock'
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tp_rcu_re
> ad_unlock'
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tp_rcu_de
> reference_sym'
>     3624895:     binding file
> x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tracepoin
> t_module_register'
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tracepoin
> t_module_unregister'
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tp_disabl
> e_destructors'
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tp_get_de
> structors_state'
>     3624895:     binding file ./hello [0] to
> /x/lttng/master/usr/lib/liblttng-ust.so.1 [0]: normal symbol
> `lttng_ust_probe_register'
>     3624895:
>     3624895:     transferring control: ./hello
>     3624895:
> ```
>
> The bit that is missing from your program is the binding of
> `lttng_ust_probe_register`. Here, it's not like the symbol isn't
> resolved at all: it's found earlier and if it didn't exist at all the
> error would probably be relatively clear like:
>
> ```
> ./doc/examples/hello/hello: symbol lookup error:
> /x/lttng/master/usr/lib/liblttng-ust.so.1: undefined symbol:
> lttng_ust_probe_register
> ```
>
> I'd like you to run another test with your app. This should indicate
> that all symbols should be immediately resolved during dynamic loading
> rather than performing lazy symbol resolution. The goal of this test is
> to separate clearly the symbol resolution from the call that happens in
> the program's initialization. You can also run it with the hello demo to
> compare the runs.

At this point I need to quit for a bit so I'll send what I have up to
this point and perform these tests on both my app and the hello
example and report back as soon as I can.

>
> ```
> LTTNG_UST_DEBUG=1 LD_DEBUG=all LD_BIND_NOW=1 ./my_app
> ```
>
> >
> >        - Review in detail which gcc commands are executed to produce the
> >     tracepoint provider and link it to the main executable.
> >
> > Exactly like hello world example.
>
> Okay. I see your program also has at least some portions in C++, which
> is different than the hello world example.
>
> Does your build process use particular optimizations or strip parts from
> the resulting object files?
>
> When you're building the hello world example, does it get compiled in
> the same buildroot as your app, or is it done differently?
>
> Do you use a custom linker script(s)?
>
>
> >
> >
> >
> > So the embedded target doesn't have babletrace so I archived the session
> > and transferred it over to my dev PC and used tracecompas to view the
> > lttng-session.
> >
> > You can see that in the attached file my_session.csv
>
> This is good, the statedump is happening properly.
>
> >
> >     ```
> >
> >      >
> >      > Thanks!  I'm running out of things to try.
> >
>
> For the next steps I would start disassembling "my_app" or checking in
> gdb which bad address is being fed into strnlen.
>
> Sections of interest (`objdump -sD my_app` _and_ `readelf -W -a my_app`):
>
> - lttng_ust_tracepoints_ptrs
> - lttng_ust_tracepoints_strings
>
> Symbols of interest (`objdump -sD my_app` _and `nm my_app`):
>
> - lttng_ust_tracepoint_hello_world___my_first_tracepoint
> - lttng_ust_tracepoints_ptrs
> - __start_lttng_ust_tracepoints_ptrs
> - __stop_lttng_ust_tracepoints_ptrs
>
> thanks,
> kienan
>
> >     Ultimately, given the bespoke environment, build steps, and application
> >     it's tough to diagnose a lot of things that we would go over with a
> >     fine
> >     tooth comb: seeing how the TPPs are built and linked, seeing how the
> >     application is built and linked, analyzing what's happening at runtime,
> >     having a coredump that allows us to see the variables, etc.
> >
> >     If you can provide a minimal reproducer, it's possible to dig further
> >     into it.
> >
> >     Otherwise, to look in more detail at your specific project would be
> >     covered under a service contract (for which we can sign NDAs, etc. as
> >     needed). Feel free to reach out the sales@efficios.com
> >     <mailto:sales@efficios.com> to organise that.
> >
> >
> > I understand but so far schedules & budget/priorities haven't aligned so
> > once again I've been scheduled to look at this issue again since we last
> > tried it on older Dunfell OS with older versions of lttng and it looks
> > like we have the same problem as before.
> >
> > Thanks for your help!
> >
> > Regards,
> >
> > Brian
> >
> >
> >      >
> >      > I don't fully understand the implications of some of this
> >     documentation,
> >      > but I have learned enough to know LD_PRELOAD means nothing if
> >     everything
> >      > is static built.  Our application is heavily multi threaded, uses
> >     fork,
> >      > clone and who knows if it's doing double closes or not ... so
> >     I've tried
> >      > these LD_PRELOAD "helpers" or whatever they are, before and
> >     didn't get a
> >      > different result, but now know it's only for shared library
> >     object.  So
> >      > now I will experiment with making our tpp a shared object and try
> >      > LD_PRELOAD of fork, fd and pthread helpers and starting my app
> >     that way.
> >
> >     LD_PRELOAD has an effect if the application you are running is
> >     dynamically linked (note: the TPP can be statically linked inside an
> >     dynamically linked executable, and the preloads are still useful and/or
> >     needed in that case).
> >
> >     The function of the helpers are to ensure that various system calls
> >     don't clobber things in use by lttng-ust. E.g., close() is shimmed by
> >     liblttng-ust-fd.so so that FDs that lttng-ust uses aren't suddenly shut
> >     by the program.
> >
> >     If you want to see it in action, you could try an experiment with a
> >     slightly modified hello-static-lib application, with instructions in
> >     the
> >     top comment:
> >     https://gist.github.com/kienanstewart/299eb6a511d92d458569b210e4a418a8 <https://gist.github.com/kienanstewart/299eb6a511d92d458569b210e4a418a8>
> >
> >     The effect of the preloads such as liblttng-ust-fd.so are
> >     independent of
> >     how the TPP is built and linked to the application or library.
> >
> >     thanks,
> >     kienan
> >
> >      >
> >      > This is what I'm referring to above from the lttng docs:
> >      >
> >      >
> >      >           Use LTTng-UST with daemons
> >      >
> >       <https://lttng.org/docs/v2.13/#doc-using-lttng-ust-with-daemons
> >     <https://lttng.org/docs/v2.13/#doc-using-lttng-ust-with-daemons>>
> >      >
> >      > If your instrumented application calls fork(2)
> >      > <https://man7.org/linux/man-pages/man2/fork.2.html
> >     <https://man7.org/linux/man-pages/man2/fork.2.html>>, clone(2)
> >      > <https://man7.org/linux/man-pages/man2/clone.2.html
> >     <https://man7.org/linux/man-pages/man2/clone.2.html>>, or BSD’s
> >     rfork(2)
> >      >
> >     <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>>, without a following exec(3) <https://man7.org/linux/man-pages/man3/exec.3.html <https://man7.org/linux/man-pages/man3/exec.3.html>>-family system call, you must preload the |liblttng-ust-fork.so| shared object when you start the application.
> >      >
> >      > LD_PRELOAD=liblttng-ust-fork.so ./my-app
> >      >
> >      > If your tracepoint provider package is a shared library which you
> >     also
> >      > preload, you must put both shared objects in |LD_PRELOAD|:
> >      >
> >      > LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
> >      >
> >      >
> >      >           Use LTTng-UST with applications which close file
> >     descriptors
> >      >           that don’t belong to them
> >      >           <https://lttng.org/docs/v2.13/#doc-liblttng-ust-fd
> >     <https://lttng.org/docs/v2.13/#doc-liblttng-ust-fd>>
> >      >
> >      > Since 2.9
> >      >
> >      > If your instrumented application closes one or more file descriptors
> >      > which it did not open itself, you must preload the
> >     |liblttng-ust-fd.so|
> >      > shared object when you start the application:
> >      >
> >      > LD_PRELOAD=liblttng-ust-fd.so ./my-app
> >      >
> >      > Typical use cases include closing all the file descriptors after
> >     fork(2)
> >      > <https://man7.org/linux/man-pages/man2/fork.2.html
> >     <https://man7.org/linux/man-pages/man2/fork.2.html>> or rfork(2)
> >      >
> >     <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>> and buggy applications doing “double closes”.
> >      >
> >      >
> >      > Thanks,
> >      >
> >      > Brian
> >      >
> >      >
> >
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Kienan Stewart via lttng-dev]

Hi Brian,

On 7/25/24 3:54 PM, Brian Hutchinson wrote:
> Hi Kienan,
> 
> I'll answer your questions below, but I've got questions on what I saw
> building and installing lttng-tools (2.13.13) and lttng-ust (2.13.8).
> 
> Based on the struggles I've had trying to get lttng to work with my
> app over various Yocto versions (Dunfell & Kirkstone) and lttng
> version, I think the problems I'm facing are mostly around C++ and
> weak and hidden symbols in Yocto toolchain.
> 
> When I started my app with the options you mentioned previously a
> while back, Id see things like:
> 
> # LTTNG_UST_DEBUG=1 LTTNG_UST_REGISTER_TIMEOUT=-1 /opt/tc/TrafficController
> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> with hidden visibility for integer objects as SAME address between
> compile units part of the same module. (in check_weak_hidden() at
> tracepoint.c:1012)
> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> with hidden visibility for pointer objects as SAME address between
> compile units part of the same module. (in check_weak_hidden() at
> tracepoint.c:1016)
> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> with hidden visibility for 24-byte structure objects as SAME address
> between compile units part of the same module. (in check_weak_hidden()
> at tracepoint.c:1020)
> 

These messages are extra information for debugging and not indicative of 
a problem in of itself. C.f. 
https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/src/lib/lttng-ust-tracepoint/tracepoint.c#L1010

There is a unit test related to this: 
https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/tests/unit/gcc-weak-hidden/main.c#L76


> I further researched this whole 'weak symbol' and 'hidden visibility'
> topic in the lttng-dev archives and it smells a lot like what I've
> been seeing.  You should be able to mix both tracef and tracepoint
> calls in souce code ... but I could not.  I could get a tracef call to
> work but if I put a tracepoint call in the same code then nothing
> would work.  This was with Dunfell 3.1.7 and earlier versions of
> lttng.
> 
> At one point I could get a tracepoint call to work but I'd have to let
> our cmake build system build and link the tpp.c file and then turn
> around and use gcc to recompile it and copy it to where all the
> objects were to create the huge .a library the app was built against.
> That's when I first learned there are issues with C++.  I think g++ is
> used to build even .c files that aren't c++.
> 
> Then if I tried to put a tracepoint in another sub project, none of
> the tracepoints would work and I'd get empty traces.  This is a
> symptom of the 'weak symbols with hidden visibility' issue ... and I
> finally found others that were having same issue in the archives.  I
> don't fully understand the issues here, although I do understand some
> of what's going on ... I just don't know what to do about it.
> 

You said initially said that you're using `lttng_ust_tracepoint` exactly 
as the hello world from the documentation; however, you have just 
described several attempts at doing different things. Which case are we 
trying to understand here?

> At this point I was being encouraged to keep upgrading to newer
> versions of lttng.  Our app never changed, gcc & lttng etc., kept
> changing.  Now with newer versions nothing runs, all I get is an
> immediate segfault.  Again, I'm building just like I did before a year
> or so ago with older versions of Yocto and lttng.  I say all of that
> to give perspective and history of what I've seen and experienced.
> Now this TLS thing has entered the picture too and so far I've only
> changed lttng, I don't know if I should be applying patches to my gcc
> for that issue.  Like I said, I'm currently using Yocto Kirkstone
> 4.0.18 and 6.1.38 kernel.
> 
> Now I'll move into the area of things I've seen building/installing
> lttng-tools and lttng-ust natively on the target environment I've
> setup where I can run 'make check' etc.  These are in the category of
> "hey, is this ok, should I be worried about this":
> 
> While building lttng-tools I see things like:
> 
> *** Warning: Linking the executable userspace-probe-elf-binary against
> the loadable module
> *** libfoo.so is not portable!
> 

The library is for a test program. My understanding is that the library 
is compiled that way to force a stripped shared object to be produced in 
order to validate that symbol lookups in libraries with no symtab 
function as expected by using the dynsym table.

C.f. 
https://github.com/lttng/lttng-tools/commit/ef3dfe5d31c88fb548189a6441aaf8b2afc0bd4b

> In file included from ../../../src/common/macros.h:15,
>                  from ../../../include/lttng/health-internal.h:19,
>                  from lttng-ctl-health.c:19:
> In function 'lttng_strnlen',
>     inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
>     inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
>     inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> specified bound 4096 may exceed source size 37 [-Wstringop-overread]
>    19 |         return strnlen(str, max);
>       |                ^~~~~~~~~~~~~~~~~
> lttng-ctl-health.c: At top level:
> cc1: note: unrecognized command-line option
> '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> earlier diagnostics

This warning is addressed in 
https://github.com/lttng/lttng-tools/commit/b25a59916106e5055be516f61f183a48f459b0b3

> ** Warning: Linking the shared library libbar.la against the loadable module
> *** libzzz.so is not portable!
> 
> *** Warning: Linking the shared library libfoo.la against the loadable module
> *** libbar.so is not portable!
> 
> While installing lttng-tools I see things like this:
> 
> make[4]: Entering directory '/opt/lttng/lttng-tools-2.13.13/src/lib/lttng-ctl'
>   CC       lttng-ctl.lo
>   CC       snapshot.lo
>   CC       lttng-ctl-health.lo
> In file included from ../../../src/common/macros.h:15,
>                  from ../../../include/lttng/health-internal.h:19,
>                  from lttng-ctl-health.c:19:
> In function 'lttng_strnlen',
>     inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
>     inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
>     inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> specified bound 4096 may exceed source size 37 [-Wstringop-overread]
>    19 |         return strnlen(str, max);
>       |                ^~~~~~~~~~~~~~~~~
> lttng-ctl-health.c: At top level:
> cc1: note: unrecognized command-line option
> '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> earlier diagnostics
> 
> Making install in trigger-condition-event-matches
> make[2]: Entering directory
> '/opt/lttng/lttng-tools-2.13.13/doc/examples/trigger-condition-event-matches'
>   CC       instrumented-app.o
>   CC       tracepoint-trigger-example.o
>   AR       libtracepoint-trigger-example.a
> ar: `u' modifier ignored since `D' is the default (see `U')
> 
> While building lttng-ust I see things like:
> 
> Making all in utils
> make[2]: Entering directory
> '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
>   CC       tap.o
>   AR       libtap.a
> ar: `u' modifier ignored since `D' is the default (see `U')
> 

While libtool now uses `cr` by default, automake still defines the 
default to `cru` which is what ends up getting used in the example. 
Since many distros have changed the configuration of ar such that 'D' is 
the default rather than the previous behaviour 'U', 'u' is redundant.

The behaviour in automake has been changed in automake 1.16.90+.

C.f. 
https://github.com/autotools-mirror/libtool/commit/418129bc63afc312701e84cb8afa5ca413df1ab5

C.f. 
http://git.savannah.gnu.org/cgit/automake.git/commit/?id=8cdbdda5aec652c356fe6dbba96810202176ae75

> *** Warning: Linking the shared library libzero.la against the
> loadable module
> *** libfakeust0.so is not portable!
>   CCLD     app_noust_indirect_abi0
> 
> *** Warning: Linking the executable app_noust_indirect_abi0 against
> the loadable module
> *** libzero.so is not portable!
>   CC       app_noust_indirect_abi0_abi1-app_noust.o
>   CC       libone.lo
>   CCLD     libone.la
>   CCLD     app_noust_indirect_abi0_abi1
> 
> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> against the loadable module
> *** libzero.so is not portable!
> 
> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> against the loadable module
> *** libone.so is not portable!
>   CC       app_noust_indirect_abi1-app_noust.o
>   CCLD     app_noust_indirect_abi1
> 
> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> against the loadable module
> *** libone.so is not portable!
>   CC       app_noust_indirect_abi1-app_noust.o
>   CCLD     app_noust_indirect_abi1
> 
> *** Warning: Linking the executable app_noust_indirect_abi1 against
> the loadable module
> *** libone.so is not portable!
>   CC       app_ust.o
>   CC       tp.o
>   CCLD     app_ust
>   CC       app_ust_dlopen.o
>   CCLD     app_ust_dlopen
>   CC       app_ust_indirect_abi0-app_ust.o
>   CC       app_ust_indirect_abi0-tp.o
>   CCLD     app_ust_indirect_abi0
> 
> *** Warning: Linking the executable app_ust_indirect_abi0 against the
> loadable module
> *** libzero.so is not portable!
>   CC       app_ust_indirect_abi0_abi1-app_ust.o
>   CC       app_ust_indirect_abi0_abi1-tp.o
>   CCLD     app_ust_indirect_abi0_abi1
> 
> *** Warning: Linking the executable app_ust_indirect_abi0_abi1 against
> the loadable module
> *** libzero.so is not portable!
> 
> I don't know if these are ok or if I should be worried about any of that.
> 

These are all for different tests.

> ... now on to your questions below.
> 
> 
> 
> On Wed, Jul 24, 2024 at 12:04 PM Kienan Stewart <kstewart@efficios.com> wrote:
>>
>> Hi Brian,
>>
>> On 7/22/24 6:00 PM, Brian Hutchinson wrote:
>>> Hi Kienan,
>>>
>>> Took a while to gather your grocery list but I think I have most of it
>>> below ;)
>>
>> thanks for all the extra info. Replies inline below, but I'll cut a lot
>> of the long output for readability.
>>
>> tl;dr the environment continues to be weird, but my present suspicion is
>> that something in either compilation, the linking of your app (eg. with
>> ld when producing the executable), or some post linking stripping might
>> be causing issues.
> 
> I'm not aware of any stripping that's going on.  In fact everything is
> being built with debug symbols at the moment and I even turned off
> optimization ... even used the debug friendly -O flag to see if that
> made a difference.
> 
>>
>> I will stop digging into further hypotheticals on my side as there is no
>> reproducer for both the environment and the application. If you ever end
>> up with a minimal reproducer that you can share, I'd be more than happy
>> to examine it.
> 
> I'm planning on trying to make a small reproducer I can share but not there yet.
> 

Great! I appreciate that you're taking the time to do so.

>>
>>>
>>> I may have not been clear.  Most of the application components are
>>> statically linked but I think there are some that are built as shared
>>> objects (.so's) so that's what I was referring to.  I know that
>>> lttng-ust is dynamically linked ... I think the lttng-ust docs say this
>>> is only option but also makes reference to the fact static linking was
>>> once possible (in some versions of the documentation) but not supported
>>> anymore (I probably have the docs memorized by now ha, ha ... I've
>>> looked at many, many versions of them).
>>>
>>> Just for full disclosure my ldd looks like:
>>>
>>>          linux-vdso.so.1 (0x0000ffffab196000)
>>>          libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
>>>          liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1
>>> (0x0000ffffa5750000)
>>>          libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
>>>          librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
>>>          libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
>>>          libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
>>>          libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
>>>          libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
>>>          /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
>>>          liblttng-ust-common.so.1 =>
>>> /usr/local/lib/liblttng-ust-common.so.1 (0x0000ffffa50e0000)
>>>          liblttng-ust-tracepoint.so.1 =>
>>> /usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
>>>          libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
>>>          libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)
>>>
>>>
>>
>> I find it very suspicious that `liblttng-ust.so.1` is in `/usr/lib`,
>> while the other lttng-ust libraries are being loaded from `/usr/local/lib`.
> 
> So Yocto puts all of the lttng libs into /usr/lib.  When I sent the
> previous info I was using lttng-tools and modules built by Yocto/OE
> and I setup a native build environment on the target so I could run
> 'make check' etc., and that's why there were things in /usr/local/lib
> because that's where you guys want stuff to be.  So I actually left
> the lttng-ust installables in /usr/local/build but also copied them to
> /usr/lib to overwrite old Yocto versions there.
> 

It's not so much that it's "where we want it to be". The documentation 
uses `/usr/local/lib` because `/usr/local` is meant for software 
installed by the sysadmin administrator, as is the case when building a 
custom version. `/usr/lib` should be used by packages shipped with the 
system.

C.f. https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s09.html

You're free to do as you see fit, but when you start mixing and matching 
libraries and some are put in /usr/lib by your system packages and some 
you move there manually I find it more difficult to follow what is going on.

>>
>> This information also matches the statedump and the LD_DEBUG info from
>> later on.
>>
>> Could you verify some of the following information:
>>
>> 1. In your build root for lttng-ust, enumerate all the liblttng*so
>> files. For each shared object, run `file $libname` and record the value
>> of the BuildID hash.R5jow
> 
> Sorry, I'm not following you here.  The only buildID hash I can think
> of is with 'eu-unstrip -n' but that's on core files, not individual
> libs.  And looking at the options I have for 'file' on my target, I
> don't see anything that looks like what you are asking.

Perhaps I wasn't clear, the command to run is really just `file`. As a 
fuller example:

```
$ file ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0
./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0: ELF 64-bit 
LSB shared object, x86-64, version 1 (SYSV), dynamically linked, 
BuildID[sha1]=b2b4a0fc449cf317e32c23e0bb57ea1ad702b702, with debug_info, 
not stripped
```

BuildID is added by default when linking with ld >= ~ 2.18

The idea here is to use the BuildID as a way to validate that the 
libraries built and the ones deployed and loaded at runtime.

> 
>>
>> 2. In your deployed environment, enumerate all the liblttng*so files in
>> /usr/local/lib and record the value of the BuildID hash.
>>
>> 3. In your deployed environment, enumerate all the liblttng*so files in
>> /usr/lib and record the value of the BuildID hash.
>>
>> The BuildID will vary from one object to the next, but the deployed
>> environment BuildIDs should match the BuildIDs of the corresponding
>> library from the build root.
>>
>> In particular my interest is the `liblttng-ust.so.1` library  that is
>> loaded  from `/usr/lib` in your deployed environment, and if there are
>> any in `/usr/local/lib`.
> 
> As I said above, when I built lttng-ust from source natively on the
> target and installed it, it went to /usr/local/lib, your default.  I
> then copied everything from /usr/local/lib (which was just lttng-ust
> stuff) to /usr/lib to overwrite the old versions Yocto put there.
> 
>>
>> In the working hello example, does it also end up linking against
>> `/usr/lib/liblttng-ust.so.1`?
> 
> Yes

Okay.

> 
>>
>>>      Further along in stepping back:
>>>
>>>         - Does make check for lttng-ust pass in your environment?
>>>
>>>
>>> So I'm glad you mentioned this.  I didn't try to run make check because
>>> #1 have never seen anything in the documentation about it and what it
>>> does and #2 It said that it depended on Perl and since this is an
>>> embedded system, I really didn't want to pull it and its dependencies in.
>>>
>>> Nevertheless, I ran it on my native target build I setup to see what
>>> would happen.  I didn't notice it complaining about perl so then I
>>> checked my manifest and perl is in fact in our image (I guess some other
>>> recipe pulled it in and I never paid attention to that):
>>>
>>> Making check in include
>>> make[1]: Entering directory
>>> '/home/iadmin/lttng-ust/lttng-ust-2.13.8/include'
>>> ...
>>   >
>>>
>>>         - Does make check for lttng-tools pass in your environment?
>>>
>>> I'm still using lttng-tools built by yocto so I don't know how to run
>>> make check in that situation.  From devtool maybe???  I don't know, I'd
>>> have to look into that one.   I never upgraded lttng-tools either, only
>>> upgraded lttng-ust and built from source tar to get the fix you
>>> mentioned for TLS stuff.
>>
>> The poky version of the tools recipe seems to have some stuff relating
>> to ptest, which seems to be a yocto tool for package testing. There
>> might be a run to run that?
>>
>> https://git.yoctoproject.org/poky/plain/meta/recipes-kernel/lttng/lttng-tools_2.13.13.bb
>> https://wiki.yoctoproject.org/wiki/Ptest
>>
>> Otherwise I would get the lttng-tools source for your version,
>> bootstrap, configure, build, and run the tests in tree.
>>
>> The bulk of regression testing for lttng-ust, -modules, and -tools
>> happens during `make check` for lttng-tools.
> 
> I did all of that.  First I did it with lttng-ust, building it from
> source tarball on target natively to eliminate any Yocto recipe/patch
> issues.  Then did the same with lttng-tools.

Sounds like `make check` for lttng-tools passed then?

> 
>>
>>>
>>>         - Is this reproducible in non-yocto environments or on other
>>>      architectures with the same project?
>>>
>>> Hmm, that's a good question.  I don't have any other embedded targets to
>>> try this on that are a different arch.  I think I heard at one point
>>> early on a heavily stubbed out version ran on a PC but I'd have to look
>>> into that.
>>>
>>>         - Does running the traced application with `LTTNG_UST_DEBUG=1` yield
>>>      more information?
>>>
>>>
>>> So far I've just been trying to get the app to run without even starting
>>> lttng just to get past segfault issue.  Here is what it looks like with
>>> debug turned on without starting lttng or a lttng-sessiond:
>>>
>>> liblttng_ust_tracepoint[599/599]: Your compiler treats weak symbols with
>>> hidden visibility for integer objects as SAME address between compile
>>> units part of the same module. (in check_weak_hidden() at
>>>
>>> liblttng_ust[599/601]: Waiting for local apps sessiond (in
>>> wait_for_sessiond() at lttng-ust-comm.c:1758)
>>> liblttng_ust[599/600]: Info: sessiond not accepting connections to
>>> global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1884)
>>> Segmentation fault
>>>
>>>         - I'd also run lttng-sessiond with the environment variable
>>>      `LTTNG_UST_DEBUG=1` set and `-vvv --verbose-consumer` in the program
>>>      arguments and capture for stdout & stderr into a file for analysis.
>>>         - Verify if the main program is dynamically linked with `ldd`
>>>         - Verify which libraries are loaded at runtime and which calls are
>>>      shimmed with `LD_DEBUG=libs,binding`
>>>
>>>
>>> I provided some of this above.
>>> The output of LD_DEBUG=libs,bindings is attached as file
>>> ld_debug-libs-bindings.txt
>>
>> Thanks. For your own debugging, I would recommend to run the hello world
>> example with the same environment variables in order to have comparison
>> point to a working example.
> 
> I ran the hello world in the example directory
> lttng-ust-2.13.8/doc/examples/hello-static-lib and it ran fine.
> 

Good.

> I can send the sessiond.log etc. if you want to see it but it ran just
> fine on target using the native built lttng-ust and lttng-tools.


My understanding at this point is the unit tests are passing for 
LTTng-UST on your system, as are the unit and regression tests for 
LTTng-tools. The example programs shipped with LTTng-UST work on your 
system, as does the example from the documentation. The statedump 
tracepoints loaded from LTTng-UST are also working fine, as evinced by 
the program logs and the LTTng trace you shared.

Despite my confusion about how exactly you're using the `hello world` 
tracepoint in your application (as you've now described several 
variations), the direction this points to for me are details related to 
how you're using LTTng-UST and/or how are your building and linking your 
application. To be clear, I don't mean to say that there is or is not an 
issue in LTTng-UST, but to point at where to examine next in detail 
including analysis of the produced object files.

> 
>>
>>   From my observations in my own environment, for the hello example (and
>> presumably your app if it's set up in the same way) I would expect
>> something similar to the following
>>
>> ```
>>     3624895:     initialize program: ./hello
>>      3624895:
>>      3624895:     binding file ./hello [0] to
>> /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `dlopen' [GLIBC_2.34]
>>      3624895:     opening
>> file=/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0];
>> direct_opencount=3
>>      3624895:
>>      3624895:     binding file ./hello [0] to
>> /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `dlsym' [GLIBC_2.34]
>>      3624895:     binding file
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
>> `lttng_ust_tp_rcu_re
>> ad_lock'
>>      3624895:     binding file
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
>> `lttng_ust_tp_rcu_re
>> ad_unlock'
>>      3624895:     binding file
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
>> `lttng_ust_tp_rcu_de
>> reference_sym'
>>      3624895:     binding file
>> x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
>> `lttng_ust_tracepoin
>> t_module_register'
>>      3624895:     binding file
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
>> `lttng_ust_tracepoin
>> t_module_unregister'
>>      3624895:     binding file
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
>> `lttng_ust_tp_disabl
>> e_destructors'
>>      3624895:     binding file
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
>> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
>> `lttng_ust_tp_get_de
>> structors_state'
>>      3624895:     binding file ./hello [0] to
>> /x/lttng/master/usr/lib/liblttng-ust.so.1 [0]: normal symbol
>> `lttng_ust_probe_register'
>>      3624895:
>>      3624895:     transferring control: ./hello
>>      3624895:
>> ```
>>
>> The bit that is missing from your program is the binding of
>> `lttng_ust_probe_register`. Here, it's not like the symbol isn't
>> resolved at all: it's found earlier and if it didn't exist at all the
>> error would probably be relatively clear like:
>>
>> ```
>> ./doc/examples/hello/hello: symbol lookup error:
>> /x/lttng/master/usr/lib/liblttng-ust.so.1: undefined symbol:
>> lttng_ust_probe_register
>> ```
>>
>> I'd like you to run another test with your app. This should indicate
>> that all symbols should be immediately resolved during dynamic loading
>> rather than performing lazy symbol resolution. The goal of this test is
>> to separate clearly the symbol resolution from the call that happens in
>> the program's initialization. You can also run it with the hello demo to
>> compare the runs.
> 
> At this point I need to quit for a bit so I'll send what I have up to
> this point and perform these tests on both my app and the hello
> example and report back as soon as I can.
> 
>>
>> ```
>> LTTNG_UST_DEBUG=1 LD_DEBUG=all LD_BIND_NOW=1 ./my_app
>> ```
>>
>>>
>>>         - Review in detail which gcc commands are executed to produce the
>>>      tracepoint provider and link it to the main executable.
>>>
>>> Exactly like hello world example.
>>
>> Okay. I see your program also has at least some portions in C++, which
>> is different than the hello world example.
>>
>> Does your build process use particular optimizations or strip parts from
>> the resulting object files?
>>
>> When you're building the hello world example, does it get compiled in
>> the same buildroot as your app, or is it done differently?
>>
>> Do you use a custom linker script(s)?
>>
>>
>>>
>>>
>>>
>>> So the embedded target doesn't have babletrace so I archived the session
>>> and transferred it over to my dev PC and used tracecompas to view the
>>> lttng-session.
>>>
>>> You can see that in the attached file my_session.csv
>>
>> This is good, the statedump is happening properly.
>>
>>>
>>>      ```
>>>
>>>       >
>>>       > Thanks!  I'm running out of things to try.
>>>
>>
>> For the next steps I would start disassembling "my_app" or checking in
>> gdb which bad address is being fed into strnlen.
>>
>> Sections of interest (`objdump -sD my_app` _and_ `readelf -W -a my_app`):
>>
>> - lttng_ust_tracepoints_ptrs
>> - lttng_ust_tracepoints_strings
>>
>> Symbols of interest (`objdump -sD my_app` _and `nm my_app`):
>>
>> - lttng_ust_tracepoint_hello_world___my_first_tracepoint
>> - lttng_ust_tracepoints_ptrs
>> - __start_lttng_ust_tracepoints_ptrs
>> - __stop_lttng_ust_tracepoints_ptrs
>>
>> thanks,
>> kienan
>>
>>>      Ultimately, given the bespoke environment, build steps, and application
>>>      it's tough to diagnose a lot of things that we would go over with a
>>>      fine
>>>      tooth comb: seeing how the TPPs are built and linked, seeing how the
>>>      application is built and linked, analyzing what's happening at runtime,
>>>      having a coredump that allows us to see the variables, etc.
>>>
>>>      If you can provide a minimal reproducer, it's possible to dig further
>>>      into it.
>>>
>>>      Otherwise, to look in more detail at your specific project would be
>>>      covered under a service contract (for which we can sign NDAs, etc. as
>>>      needed). Feel free to reach out the sales@efficios.com
>>>      <mailto:sales@efficios.com> to organise that.
>>>
>>>
>>> I understand but so far schedules & budget/priorities haven't aligned so
>>> once again I've been scheduled to look at this issue again since we last
>>> tried it on older Dunfell OS with older versions of lttng and it looks
>>> like we have the same problem as before.
>>>
>>> Thanks for your help!
>>>
>>> Regards,
>>>
>>> Brian
>>>
>>>
>>>       >
>>>       > I don't fully understand the implications of some of this
>>>      documentation,
>>>       > but I have learned enough to know LD_PRELOAD means nothing if
>>>      everything
>>>       > is static built.  Our application is heavily multi threaded, uses
>>>      fork,
>>>       > clone and who knows if it's doing double closes or not ... so
>>>      I've tried
>>>       > these LD_PRELOAD "helpers" or whatever they are, before and
>>>      didn't get a
>>>       > different result, but now know it's only for shared library
>>>      object.  So
>>>       > now I will experiment with making our tpp a shared object and try
>>>       > LD_PRELOAD of fork, fd and pthread helpers and starting my app
>>>      that way.
>>>
>>>      LD_PRELOAD has an effect if the application you are running is
>>>      dynamically linked (note: the TPP can be statically linked inside an
>>>      dynamically linked executable, and the preloads are still useful and/or
>>>      needed in that case).
>>>
>>>      The function of the helpers are to ensure that various system calls
>>>      don't clobber things in use by lttng-ust. E.g., close() is shimmed by
>>>      liblttng-ust-fd.so so that FDs that lttng-ust uses aren't suddenly shut
>>>      by the program.
>>>
>>>      If you want to see it in action, you could try an experiment with a
>>>      slightly modified hello-static-lib application, with instructions in
>>>      the
>>>      top comment:
>>>      https://gist.github.com/kienanstewart/299eb6a511d92d458569b210e4a418a8 <https://gist.github.com/kienanstewart/299eb6a511d92d458569b210e4a418a8>
>>>
>>>      The effect of the preloads such as liblttng-ust-fd.so are
>>>      independent of
>>>      how the TPP is built and linked to the application or library.
>>>
>>>      thanks,
>>>      kienan
>>>
>>>       >
>>>       > This is what I'm referring to above from the lttng docs:
>>>       >
>>>       >
>>>       >           Use LTTng-UST with daemons
>>>       >
>>>        <https://lttng.org/docs/v2.13/#doc-using-lttng-ust-with-daemons
>>>      <https://lttng.org/docs/v2.13/#doc-using-lttng-ust-with-daemons>>
>>>       >
>>>       > If your instrumented application calls fork(2)
>>>       > <https://man7.org/linux/man-pages/man2/fork.2.html
>>>      <https://man7.org/linux/man-pages/man2/fork.2.html>>, clone(2)
>>>       > <https://man7.org/linux/man-pages/man2/clone.2.html
>>>      <https://man7.org/linux/man-pages/man2/clone.2.html>>, or BSD’s
>>>      rfork(2)
>>>       >
>>>      <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>>, without a following exec(3) <https://man7.org/linux/man-pages/man3/exec.3.html <https://man7.org/linux/man-pages/man3/exec.3.html>>-family system call, you must preload the |liblttng-ust-fork.so| shared object when you start the application.
>>>       >
>>>       > LD_PRELOAD=liblttng-ust-fork.so ./my-app
>>>       >
>>>       > If your tracepoint provider package is a shared library which you
>>>      also
>>>       > preload, you must put both shared objects in |LD_PRELOAD|:
>>>       >
>>>       > LD_PRELOAD=liblttng-ust-fork.so:/path/to/tp.so ./my-app
>>>       >
>>>       >
>>>       >           Use LTTng-UST with applications which close file
>>>      descriptors
>>>       >           that don’t belong to them
>>>       >           <https://lttng.org/docs/v2.13/#doc-liblttng-ust-fd
>>>      <https://lttng.org/docs/v2.13/#doc-liblttng-ust-fd>>
>>>       >
>>>       > Since 2.9
>>>       >
>>>       > If your instrumented application closes one or more file descriptors
>>>       > which it did not open itself, you must preload the
>>>      |liblttng-ust-fd.so|
>>>       > shared object when you start the application:
>>>       >
>>>       > LD_PRELOAD=liblttng-ust-fd.so ./my-app
>>>       >
>>>       > Typical use cases include closing all the file descriptors after
>>>      fork(2)
>>>       > <https://man7.org/linux/man-pages/man2/fork.2.html
>>>      <https://man7.org/linux/man-pages/man2/fork.2.html>> or rfork(2)
>>>       >
>>>      <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE <http://www.freebsd.org/cgi/man.cgi?query=rfork&sektion=2&manpath=FreeBSD+4.10-RELEASE>> and buggy applications doing “double closes”.
>>>       >
>>>       >
>>>       > Thanks,
>>>       >
>>>       > Brian
>>>       >
>>>       >
>>>

thanks,
kienan
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev]

On Mon, Jul 29, 2024 at 3:03 PM Kienan Stewart <kstewart@efficios.com> wrote:
>
> Hi Brian,
>
> On 7/25/24 3:54 PM, Brian Hutchinson wrote:
> > Hi Kienan,
> >
> > I'll answer your questions below, but I've got questions on what I saw
> > building and installing lttng-tools (2.13.13) and lttng-ust (2.13.8).
> >
> > Based on the struggles I've had trying to get lttng to work with my
> > app over various Yocto versions (Dunfell & Kirkstone) and lttng
> > version, I think the problems I'm facing are mostly around C++ and
> > weak and hidden symbols in Yocto toolchain.
> >
> > When I started my app with the options you mentioned previously a
> > while back, Id see things like:
> >
> > # LTTNG_UST_DEBUG=1 LTTNG_UST_REGISTER_TIMEOUT=-1 /opt/tc/TrafficController
> > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > with hidden visibility for integer objects as SAME address between
> > compile units part of the same module. (in check_weak_hidden() at
> > tracepoint.c:1012)
> > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > with hidden visibility for pointer objects as SAME address between
> > compile units part of the same module. (in check_weak_hidden() at
> > tracepoint.c:1016)
> > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > with hidden visibility for 24-byte structure objects as SAME address
> > between compile units part of the same module. (in check_weak_hidden()
> > at tracepoint.c:1020)
> >
>
> These messages are extra information for debugging and not indicative of
> a problem in of itself. C.f.
> https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/src/lib/lttng-ust-tracepoint/tracepoint.c#L1010
>
> There is a unit test related to this:
> https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/tests/unit/gcc-weak-hidden/main.c#L76
>
>
> > I further researched this whole 'weak symbol' and 'hidden visibility'
> > topic in the lttng-dev archives and it smells a lot like what I've
> > been seeing.  You should be able to mix both tracef and tracepoint
> > calls in souce code ... but I could not.  I could get a tracef call to
> > work but if I put a tracepoint call in the same code then nothing
> > would work.  This was with Dunfell 3.1.7 and earlier versions of
> > lttng.
> >
> > At one point I could get a tracepoint call to work but I'd have to let
> > our cmake build system build and link the tpp.c file and then turn
> > around and use gcc to recompile it and copy it to where all the
> > objects were to create the huge .a library the app was built against.
> > That's when I first learned there are issues with C++.  I think g++ is
> > used to build even .c files that aren't c++.
> >
> > Then if I tried to put a tracepoint in another sub project, none of
> > the tracepoints would work and I'd get empty traces.  This is a
> > symptom of the 'weak symbols with hidden visibility' issue ... and I
> > finally found others that were having same issue in the archives.  I
> > don't fully understand the issues here, although I do understand some
> > of what's going on ... I just don't know what to do about it.
> >
>
> You said initially said that you're using `lttng_ust_tracepoint` exactly
> as the hello world from the documentation; however, you have just
> described several attempts at doing different things. Which case are we
> trying to understand here?

lttng_ust_tracepoint.  I only mentioned prior tests for context to
similar struggles from a year or more ago.


>
> > At this point I was being encouraged to keep upgrading to newer
> > versions of lttng.  Our app never changed, gcc & lttng etc., kept
> > changing.  Now with newer versions nothing runs, all I get is an
> > immediate segfault.  Again, I'm building just like I did before a year
> > or so ago with older versions of Yocto and lttng.  I say all of that
> > to give perspective and history of what I've seen and experienced.
> > Now this TLS thing has entered the picture too and so far I've only
> > changed lttng, I don't know if I should be applying patches to my gcc
> > for that issue.  Like I said, I'm currently using Yocto Kirkstone
> > 4.0.18 and 6.1.38 kernel.
> >
> > Now I'll move into the area of things I've seen building/installing
> > lttng-tools and lttng-ust natively on the target environment I've
> > setup where I can run 'make check' etc.  These are in the category of
> > "hey, is this ok, should I be worried about this":
> >
> > While building lttng-tools I see things like:
> >
> > *** Warning: Linking the executable userspace-probe-elf-binary against
> > the loadable module
> > *** libfoo.so is not portable!
> >
>
> The library is for a test program. My understanding is that the library
> is compiled that way to force a stripped shared object to be produced in
> order to validate that symbol lookups in libraries with no symtab
> function as expected by using the dynsym table.
>
> C.f.
> https://github.com/lttng/lttng-tools/commit/ef3dfe5d31c88fb548189a6441aaf8b2afc0bd4b
>
> > In file included from ../../../src/common/macros.h:15,
> >                  from ../../../include/lttng/health-internal.h:19,
> >                  from lttng-ctl-health.c:19:
> > In function 'lttng_strnlen',
> >     inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
> >     inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
> >     inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> > ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> > specified bound 4096 may exceed source size 37 [-Wstringop-overread]
> >    19 |         return strnlen(str, max);
> >       |                ^~~~~~~~~~~~~~~~~
> > lttng-ctl-health.c: At top level:
> > cc1: note: unrecognized command-line option
> > '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> > earlier diagnostics
>
> This warning is addressed in
> https://github.com/lttng/lttng-tools/commit/b25a59916106e5055be516f61f183a48f459b0b3
>
> > ** Warning: Linking the shared library libbar.la against the loadable module
> > *** libzzz.so is not portable!
> >
> > *** Warning: Linking the shared library libfoo.la against the loadable module
> > *** libbar.so is not portable!
> >
> > While installing lttng-tools I see things like this:
> >
> > make[4]: Entering directory '/opt/lttng/lttng-tools-2.13.13/src/lib/lttng-ctl'
> >   CC       lttng-ctl.lo
> >   CC       snapshot.lo
> >   CC       lttng-ctl-health.lo
> > In file included from ../../../src/common/macros.h:15,
> >                  from ../../../include/lttng/health-internal.h:19,
> >                  from lttng-ctl-health.c:19:
> > In function 'lttng_strnlen',
> >     inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
> >     inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
> >     inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> > ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> > specified bound 4096 may exceed source size 37 [-Wstringop-overread]
> >    19 |         return strnlen(str, max);
> >       |                ^~~~~~~~~~~~~~~~~
> > lttng-ctl-health.c: At top level:
> > cc1: note: unrecognized command-line option
> > '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> > earlier diagnostics
> >
> > Making install in trigger-condition-event-matches
> > make[2]: Entering directory
> > '/opt/lttng/lttng-tools-2.13.13/doc/examples/trigger-condition-event-matches'
> >   CC       instrumented-app.o
> >   CC       tracepoint-trigger-example.o
> >   AR       libtracepoint-trigger-example.a
> > ar: `u' modifier ignored since `D' is the default (see `U')
> >
> > While building lttng-ust I see things like:
> >
> > Making all in utils
> > make[2]: Entering directory
> > '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
> >   CC       tap.o
> >   AR       libtap.a
> > ar: `u' modifier ignored since `D' is the default (see `U')
> >
>
> While libtool now uses `cr` by default, automake still defines the
> default to `cru` which is what ends up getting used in the example.
> Since many distros have changed the configuration of ar such that 'D' is
> the default rather than the previous behaviour 'U', 'u' is redundant.
>
> The behaviour in automake has been changed in automake 1.16.90+.
>
> C.f.
> https://github.com/autotools-mirror/libtool/commit/418129bc63afc312701e84cb8afa5ca413df1ab5
>
> C.f.
> http://git.savannah.gnu.org/cgit/automake.git/commit/?id=8cdbdda5aec652c356fe6dbba96810202176ae75
>
> > *** Warning: Linking the shared library libzero.la against the
> > loadable module
> > *** libfakeust0.so is not portable!
> >   CCLD     app_noust_indirect_abi0
> >
> > *** Warning: Linking the executable app_noust_indirect_abi0 against
> > the loadable module
> > *** libzero.so is not portable!
> >   CC       app_noust_indirect_abi0_abi1-app_noust.o
> >   CC       libone.lo
> >   CCLD     libone.la
> >   CCLD     app_noust_indirect_abi0_abi1
> >
> > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > against the loadable module
> > *** libzero.so is not portable!
> >
> > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > against the loadable module
> > *** libone.so is not portable!
> >   CC       app_noust_indirect_abi1-app_noust.o
> >   CCLD     app_noust_indirect_abi1
> >
> > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > against the loadable module
> > *** libone.so is not portable!
> >   CC       app_noust_indirect_abi1-app_noust.o
> >   CCLD     app_noust_indirect_abi1
> >
> > *** Warning: Linking the executable app_noust_indirect_abi1 against
> > the loadable module
> > *** libone.so is not portable!
> >   CC       app_ust.o
> >   CC       tp.o
> >   CCLD     app_ust
> >   CC       app_ust_dlopen.o
> >   CCLD     app_ust_dlopen
> >   CC       app_ust_indirect_abi0-app_ust.o
> >   CC       app_ust_indirect_abi0-tp.o
> >   CCLD     app_ust_indirect_abi0
> >
> > *** Warning: Linking the executable app_ust_indirect_abi0 against the
> > loadable module
> > *** libzero.so is not portable!
> >   CC       app_ust_indirect_abi0_abi1-app_ust.o
> >   CC       app_ust_indirect_abi0_abi1-tp.o
> >   CCLD     app_ust_indirect_abi0_abi1
> >
> > *** Warning: Linking the executable app_ust_indirect_abi0_abi1 against
> > the loadable module
> > *** libzero.so is not portable!
> >
> > I don't know if these are ok or if I should be worried about any of that.
> >
>
> These are all for different tests.
>
> > ... now on to your questions below.
> >
> >
> >
> > On Wed, Jul 24, 2024 at 12:04 PM Kienan Stewart <kstewart@efficios.com> wrote:
> >>
> >> Hi Brian,
> >>
> >> On 7/22/24 6:00 PM, Brian Hutchinson wrote:
> >>> Hi Kienan,
> >>>
> >>> Took a while to gather your grocery list but I think I have most of it
> >>> below ;)
> >>
> >> thanks for all the extra info. Replies inline below, but I'll cut a lot
> >> of the long output for readability.
> >>
> >> tl;dr the environment continues to be weird, but my present suspicion is
> >> that something in either compilation, the linking of your app (eg. with
> >> ld when producing the executable), or some post linking stripping might
> >> be causing issues.
> >
> > I'm not aware of any stripping that's going on.  In fact everything is
> > being built with debug symbols at the moment and I even turned off
> > optimization ... even used the debug friendly -O flag to see if that
> > made a difference.
> >
> >>
> >> I will stop digging into further hypotheticals on my side as there is no
> >> reproducer for both the environment and the application. If you ever end
> >> up with a minimal reproducer that you can share, I'd be more than happy
> >> to examine it.
> >
> > I'm planning on trying to make a small reproducer I can share but not there yet.
> >
>
> Great! I appreciate that you're taking the time to do so.
>
> >>
> >>>
> >>> I may have not been clear.  Most of the application components are
> >>> statically linked but I think there are some that are built as shared
> >>> objects (.so's) so that's what I was referring to.  I know that
> >>> lttng-ust is dynamically linked ... I think the lttng-ust docs say this
> >>> is only option but also makes reference to the fact static linking was
> >>> once possible (in some versions of the documentation) but not supported
> >>> anymore (I probably have the docs memorized by now ha, ha ... I've
> >>> looked at many, many versions of them).
> >>>
> >>> Just for full disclosure my ldd looks like:
> >>>
> >>>          linux-vdso.so.1 (0x0000ffffab196000)
> >>>          libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
> >>>          liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1
> >>> (0x0000ffffa5750000)
> >>>          libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
> >>>          librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
> >>>          libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
> >>>          libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
> >>>          libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
> >>>          libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
> >>>          /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
> >>>          liblttng-ust-common.so.1 =>
> >>> /usr/local/lib/liblttng-ust-common.so.1 (0x0000ffffa50e0000)
> >>>          liblttng-ust-tracepoint.so.1 =>
> >>> /usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
> >>>          libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
> >>>          libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)
> >>>
> >>>
> >>
> >> I find it very suspicious that `liblttng-ust.so.1` is in `/usr/lib`,
> >> while the other lttng-ust libraries are being loaded from `/usr/local/lib`.
> >
> > So Yocto puts all of the lttng libs into /usr/lib.  When I sent the
> > previous info I was using lttng-tools and modules built by Yocto/OE
> > and I setup a native build environment on the target so I could run
> > 'make check' etc., and that's why there were things in /usr/local/lib
> > because that's where you guys want stuff to be.  So I actually left
> > the lttng-ust installables in /usr/local/build but also copied them to
> > /usr/lib to overwrite old Yocto versions there.
> >
>
> It's not so much that it's "where we want it to be". The documentation
> uses `/usr/local/lib` because `/usr/local` is meant for software
> installed by the sysadmin administrator, as is the case when building a
> custom version. `/usr/lib` should be used by packages shipped with the
> system.
>
> C.f. https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s09.html
>
> You're free to do as you see fit, but when you start mixing and matching
> libraries and some are put in /usr/lib by your system packages and some
> you move there manually I find it more difficult to follow what is going on.
>
> >>
> >> This information also matches the statedump and the LD_DEBUG info from
> >> later on.
> >>
> >> Could you verify some of the following information:
> >>
> >> 1. In your build root for lttng-ust, enumerate all the liblttng*so
> >> files. For each shared object, run `file $libname` and record the value
> >> of the BuildID hash.R5jow
> >
> > Sorry, I'm not following you here.  The only buildID hash I can think
> > of is with 'eu-unstrip -n' but that's on core files, not individual
> > libs.  And looking at the options I have for 'file' on my target, I
> > don't see anything that looks like what you are asking.
>
> Perhaps I wasn't clear, the command to run is really just `file`. As a
> fuller example:
>
> ```
> $ file ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0
> ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0: ELF 64-bit
> LSB shared object, x86-64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=b2b4a0fc449cf317e32c23e0bb57ea1ad702b702, with debug_info,
> not stripped

Ok, feel stupid now.  When I ran the command before, I used short name
and didn't do it on the long name and just got back:

# file /usr/lib/liblttng-ust.so
/usr/lib/liblttng-ust.so: symbolic link to liblttng-ust.so.1.0.0

... and immediately looked at man page to try to figure out what
switch showed BuildID etc., ha, ha.

When I do it on long name here is what I see:

# file /usr/lib/liblttng-ust-common.so.1.0.0
/usr/lib/liblttng-ust-common.so.1.0.0: ELF 64-bit LSB shared object,
ARM aarch64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=a50c9a77163b6b91e1f84e57d167c7b77ae707a3, with
debug_info, not stripped
# file /usr/lib/liblttng-ust-ctl.so.5.0.0
/usr/lib/liblttng-ust-ctl.so.5.0.0: ELF 64-bit LSB shared object, ARM
aarch64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=547cccac08721ed1c9a7f3c7ebf1de84ddba7fba, with
debug_info, not stripped
# file /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0
/usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0: ELF 64-bit LSB shared
object, ARM aarch64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=ad5f71ef5e83ab9a972488976c47db265d3360b9, with
debug_info, not stripped
# file /usr/lib/liblttng-ust-cyg-profile.so.1.0.0
/usr/lib/liblttng-ust-cyg-profile.so.1.0.0: ELF 64-bit LSB shared
object, ARM aarch64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=c2c246c1973bd3241aa4f7229fcbcb27ebe08e82, with
debug_info, not stripped
# file /usr/lib/liblttng-ust-dl.so.1.0.0
/usr/lib/liblttng-ust-dl.so.1.0.0: ELF 64-bit LSB shared object, ARM
aarch64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=6082ee88c9394319bc3adff16b0b3ea9f8d549ec, with
debug_info, not stripped
# file /usr/lib/liblttng-ust-fd.so.1.0.0
/usr/lib/liblttng-ust-fd.so.1.0.0: ELF 64-bit LSB shared object, ARM
aarch64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=a2813245af91abe98615771dfd7d5f19b033a410, with
debug_info, not stripped
# file /usr/lib/liblttng-ust-fork.so.1.0.0
/usr/lib/liblttng-ust-fork.so.1.0.0: ELF 64-bit LSB shared object, ARM
aarch64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=86afa53808502873830c02290f477d4ff8013afb, with
debug_info, not stripped
# file /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0
/usr/lib/liblttng-ust-libc-wrapper.so.1.0.0: ELF 64-bit LSB shared
object, ARM aarch64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=52f391875a378b5f2c46747a58020b86cb7c9a83, with
debug_info, not stripped
# file /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0
/usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0: ELF 64-bit LSB shared
object, ARM aarch64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=7127223a7e8ed67c6697b95ae1f8ac107df7e47e, with
debug_info, not stripped
# file /usr/lib/liblttng-ust-tracepoint.so.1.0.0
/usr/lib/liblttng-ust-tracepoint.so.1.0.0: ELF 64-bit LSB shared
object, ARM aarch64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=5971b4d84ec1efe61c6d47c38e92de20569f0f49, with
debug_info, not stripped
# file /usr/lib/liblttng-ust.so.1.0.0
/usr/lib/liblttng-ust.so.1.0.0: ELF 64-bit LSB shared object, ARM
aarch64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=ce7097ae9bbf42a02dccd386fdfbd37e3224858b, with
debug_info, not stripped


cutting some stuff out cause it's getting long again.

>
> Sounds like `make check` for lttng-tools passed then?

At first no.  But I think this is because I built the new lttng-tools
in my on target native environment and ran make check and forgot to do
make install first, so it was using the older version of lttng-tools.

So then I ran make install of lttng-tools and even did a make clean
and rebuild of lttng-ust and re-installed it and ran make check on
both (and things looked a lot better) ... that's where those warnings
etc., I asked you about came from.

>
> My understanding at this point is the unit tests are passing for
> LTTng-UST on your system, as are the unit and regression tests for
> LTTng-tools. The example programs shipped with LTTng-UST work on your
> system, as does the example from the documentation. The statedump
> tracepoints loaded from LTTng-UST are also working fine, as evinced by
> the program logs and the LTTng trace you shared.
>
> Despite my confusion about how exactly you're using the `hello world`
> tracepoint in your application (as you've now described several
> variations), the direction this points to for me are details related to
> how you're using LTTng-UST and/or how are your building and linking your
> application. To be clear, I don't mean to say that there is or is not an
> issue in LTTng-UST, but to point at where to examine next in detail
> including analysis of the produced object files.

I compared the doc/examples/hello-static-lib to what I picked out of
the LTTng documentation on the web site "Quick start" section and the
traceprovider headder file is including stddef.h and mine isn't and
the doc/examples/hello-static-lib/hello.c code is doing a sighandler
and mine isn't doing any of that either.  I think I've probably posted
it before but will do it again.  Here is what I'm calling my "hello".
It's from the lttng documentation but I cut it down even further just
to make sure I didn't fat finger something.  Like I said before, the
full hello example from the documentation works.  But when I call
pretty much the same code from my app it seg faults.

I don't know if the differences I see between my "hello" and the
"hello-static-lib" matter.

hello-tp.h:

#undef LTTNG_UST_TRACEPOINT_PROVIDER
#define LTTNG_UST_TRACEPOINT_PROVIDER hello_world

#undef LTTNG_UST_TRACEPOINT_INCLUDE
#define LTTNG_UST_TRACEPOINT_INCLUDE "./hello-tp.h"

#if !defined(_HELLO_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
#define _HELLO_TP_H

#include <lttng/tracepoint.h>

LTTNG_UST_TRACEPOINT_EVENT(
   hello_world,
   my_first_tracepoint,
   LTTNG_UST_TP_ARGS(
       int, my_integer_arg
   ),
   LTTNG_UST_TP_FIELDS(
       lttng_ust_field_integer(int, my_integer_field, my_integer_arg)
   )
)

#endif /* _HELLO_TP_H */

#include <lttng/tracepoint-event.h>

hello-tp.c:

#define LTTNG_UST_TRACEPOINT_CREATE_PROBES

#include "hello-tp.h"

From my_app:

#define LTTNG_UST_TRACEPOINT_DEFINE
//#define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
#include "hello-tp.h"

.
.
.
lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

In the above case the tpp is static but I've tried to make it a shared
object too (thus the commented out DYNAMIC_LINKAGE above) but get the
same result.

Again, I think the issue is probably g++ and weak/hidden symbol
related and or TLS but that's based on the totality of what I've
experienced over the past year or so and seeing the
experiences/problems of others in the lttng-dev archives.

Regards,

Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev]

[-- Attachment #1: Type: text/plain, Size: 24766 bytes --]

Hi Kienan,

I noticed looking thru the 38k line LTTNG_UST_DEBUG session this line:

"       810:     /usr/lib/liblttng-ust.so.1: error: symbol lookup
error: undefined symbol: ltt_probe_register (fatal)"

... and it jogged my memory that I did see some abi conflict messages
in lttng-ust "make check" that I don't know if they are "good" or
"bad" but could be related???

Anyway, attaching a tar of lttng-ust and lttng-tools "make check" for
your enjoyment.

Regards,

Brian




On Tue, Jul 30, 2024 at 8:40 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>
> On Mon, Jul 29, 2024 at 3:03 PM Kienan Stewart <kstewart@efficios.com> wrote:
> >
> > Hi Brian,
> >
> > On 7/25/24 3:54 PM, Brian Hutchinson wrote:
> > > Hi Kienan,
> > >
> > > I'll answer your questions below, but I've got questions on what I saw
> > > building and installing lttng-tools (2.13.13) and lttng-ust (2.13.8).
> > >
> > > Based on the struggles I've had trying to get lttng to work with my
> > > app over various Yocto versions (Dunfell & Kirkstone) and lttng
> > > version, I think the problems I'm facing are mostly around C++ and
> > > weak and hidden symbols in Yocto toolchain.
> > >
> > > When I started my app with the options you mentioned previously a
> > > while back, Id see things like:
> > >
> > > # LTTNG_UST_DEBUG=1 LTTNG_UST_REGISTER_TIMEOUT=-1 /opt/tc/TrafficController
> > > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > > with hidden visibility for integer objects as SAME address between
> > > compile units part of the same module. (in check_weak_hidden() at
> > > tracepoint.c:1012)
> > > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > > with hidden visibility for pointer objects as SAME address between
> > > compile units part of the same module. (in check_weak_hidden() at
> > > tracepoint.c:1016)
> > > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > > with hidden visibility for 24-byte structure objects as SAME address
> > > between compile units part of the same module. (in check_weak_hidden()
> > > at tracepoint.c:1020)
> > >
> >
> > These messages are extra information for debugging and not indicative of
> > a problem in of itself. C.f.
> > https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/src/lib/lttng-ust-tracepoint/tracepoint.c#L1010
> >
> > There is a unit test related to this:
> > https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/tests/unit/gcc-weak-hidden/main.c#L76
> >
> >
> > > I further researched this whole 'weak symbol' and 'hidden visibility'
> > > topic in the lttng-dev archives and it smells a lot like what I've
> > > been seeing.  You should be able to mix both tracef and tracepoint
> > > calls in souce code ... but I could not.  I could get a tracef call to
> > > work but if I put a tracepoint call in the same code then nothing
> > > would work.  This was with Dunfell 3.1.7 and earlier versions of
> > > lttng.
> > >
> > > At one point I could get a tracepoint call to work but I'd have to let
> > > our cmake build system build and link the tpp.c file and then turn
> > > around and use gcc to recompile it and copy it to where all the
> > > objects were to create the huge .a library the app was built against.
> > > That's when I first learned there are issues with C++.  I think g++ is
> > > used to build even .c files that aren't c++.
> > >
> > > Then if I tried to put a tracepoint in another sub project, none of
> > > the tracepoints would work and I'd get empty traces.  This is a
> > > symptom of the 'weak symbols with hidden visibility' issue ... and I
> > > finally found others that were having same issue in the archives.  I
> > > don't fully understand the issues here, although I do understand some
> > > of what's going on ... I just don't know what to do about it.
> > >
> >
> > You said initially said that you're using `lttng_ust_tracepoint` exactly
> > as the hello world from the documentation; however, you have just
> > described several attempts at doing different things. Which case are we
> > trying to understand here?
>
> lttng_ust_tracepoint.  I only mentioned prior tests for context to
> similar struggles from a year or more ago.
>
>
> >
> > > At this point I was being encouraged to keep upgrading to newer
> > > versions of lttng.  Our app never changed, gcc & lttng etc., kept
> > > changing.  Now with newer versions nothing runs, all I get is an
> > > immediate segfault.  Again, I'm building just like I did before a year
> > > or so ago with older versions of Yocto and lttng.  I say all of that
> > > to give perspective and history of what I've seen and experienced.
> > > Now this TLS thing has entered the picture too and so far I've only
> > > changed lttng, I don't know if I should be applying patches to my gcc
> > > for that issue.  Like I said, I'm currently using Yocto Kirkstone
> > > 4.0.18 and 6.1.38 kernel.
> > >
> > > Now I'll move into the area of things I've seen building/installing
> > > lttng-tools and lttng-ust natively on the target environment I've
> > > setup where I can run 'make check' etc.  These are in the category of
> > > "hey, is this ok, should I be worried about this":
> > >
> > > While building lttng-tools I see things like:
> > >
> > > *** Warning: Linking the executable userspace-probe-elf-binary against
> > > the loadable module
> > > *** libfoo.so is not portable!
> > >
> >
> > The library is for a test program. My understanding is that the library
> > is compiled that way to force a stripped shared object to be produced in
> > order to validate that symbol lookups in libraries with no symtab
> > function as expected by using the dynsym table.
> >
> > C.f.
> > https://github.com/lttng/lttng-tools/commit/ef3dfe5d31c88fb548189a6441aaf8b2afc0bd4b
> >
> > > In file included from ../../../src/common/macros.h:15,
> > >                  from ../../../include/lttng/health-internal.h:19,
> > >                  from lttng-ctl-health.c:19:
> > > In function 'lttng_strnlen',
> > >     inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
> > >     inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
> > >     inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> > > ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> > > specified bound 4096 may exceed source size 37 [-Wstringop-overread]
> > >    19 |         return strnlen(str, max);
> > >       |                ^~~~~~~~~~~~~~~~~
> > > lttng-ctl-health.c: At top level:
> > > cc1: note: unrecognized command-line option
> > > '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> > > earlier diagnostics
> >
> > This warning is addressed in
> > https://github.com/lttng/lttng-tools/commit/b25a59916106e5055be516f61f183a48f459b0b3
> >
> > > ** Warning: Linking the shared library libbar.la against the loadable module
> > > *** libzzz.so is not portable!
> > >
> > > *** Warning: Linking the shared library libfoo.la against the loadable module
> > > *** libbar.so is not portable!
> > >
> > > While installing lttng-tools I see things like this:
> > >
> > > make[4]: Entering directory '/opt/lttng/lttng-tools-2.13.13/src/lib/lttng-ctl'
> > >   CC       lttng-ctl.lo
> > >   CC       snapshot.lo
> > >   CC       lttng-ctl-health.lo
> > > In file included from ../../../src/common/macros.h:15,
> > >                  from ../../../include/lttng/health-internal.h:19,
> > >                  from lttng-ctl-health.c:19:
> > > In function 'lttng_strnlen',
> > >     inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
> > >     inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
> > >     inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> > > ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> > > specified bound 4096 may exceed source size 37 [-Wstringop-overread]
> > >    19 |         return strnlen(str, max);
> > >       |                ^~~~~~~~~~~~~~~~~
> > > lttng-ctl-health.c: At top level:
> > > cc1: note: unrecognized command-line option
> > > '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> > > earlier diagnostics
> > >
> > > Making install in trigger-condition-event-matches
> > > make[2]: Entering directory
> > > '/opt/lttng/lttng-tools-2.13.13/doc/examples/trigger-condition-event-matches'
> > >   CC       instrumented-app.o
> > >   CC       tracepoint-trigger-example.o
> > >   AR       libtracepoint-trigger-example.a
> > > ar: `u' modifier ignored since `D' is the default (see `U')
> > >
> > > While building lttng-ust I see things like:
> > >
> > > Making all in utils
> > > make[2]: Entering directory
> > > '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
> > >   CC       tap.o
> > >   AR       libtap.a
> > > ar: `u' modifier ignored since `D' is the default (see `U')
> > >
> >
> > While libtool now uses `cr` by default, automake still defines the
> > default to `cru` which is what ends up getting used in the example.
> > Since many distros have changed the configuration of ar such that 'D' is
> > the default rather than the previous behaviour 'U', 'u' is redundant.
> >
> > The behaviour in automake has been changed in automake 1.16.90+.
> >
> > C.f.
> > https://github.com/autotools-mirror/libtool/commit/418129bc63afc312701e84cb8afa5ca413df1ab5
> >
> > C.f.
> > http://git.savannah.gnu.org/cgit/automake.git/commit/?id=8cdbdda5aec652c356fe6dbba96810202176ae75
> >
> > > *** Warning: Linking the shared library libzero.la against the
> > > loadable module
> > > *** libfakeust0.so is not portable!
> > >   CCLD     app_noust_indirect_abi0
> > >
> > > *** Warning: Linking the executable app_noust_indirect_abi0 against
> > > the loadable module
> > > *** libzero.so is not portable!
> > >   CC       app_noust_indirect_abi0_abi1-app_noust.o
> > >   CC       libone.lo
> > >   CCLD     libone.la
> > >   CCLD     app_noust_indirect_abi0_abi1
> > >
> > > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > > against the loadable module
> > > *** libzero.so is not portable!
> > >
> > > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > > against the loadable module
> > > *** libone.so is not portable!
> > >   CC       app_noust_indirect_abi1-app_noust.o
> > >   CCLD     app_noust_indirect_abi1
> > >
> > > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > > against the loadable module
> > > *** libone.so is not portable!
> > >   CC       app_noust_indirect_abi1-app_noust.o
> > >   CCLD     app_noust_indirect_abi1
> > >
> > > *** Warning: Linking the executable app_noust_indirect_abi1 against
> > > the loadable module
> > > *** libone.so is not portable!
> > >   CC       app_ust.o
> > >   CC       tp.o
> > >   CCLD     app_ust
> > >   CC       app_ust_dlopen.o
> > >   CCLD     app_ust_dlopen
> > >   CC       app_ust_indirect_abi0-app_ust.o
> > >   CC       app_ust_indirect_abi0-tp.o
> > >   CCLD     app_ust_indirect_abi0
> > >
> > > *** Warning: Linking the executable app_ust_indirect_abi0 against the
> > > loadable module
> > > *** libzero.so is not portable!
> > >   CC       app_ust_indirect_abi0_abi1-app_ust.o
> > >   CC       app_ust_indirect_abi0_abi1-tp.o
> > >   CCLD     app_ust_indirect_abi0_abi1
> > >
> > > *** Warning: Linking the executable app_ust_indirect_abi0_abi1 against
> > > the loadable module
> > > *** libzero.so is not portable!
> > >
> > > I don't know if these are ok or if I should be worried about any of that.
> > >
> >
> > These are all for different tests.
> >
> > > ... now on to your questions below.
> > >
> > >
> > >
> > > On Wed, Jul 24, 2024 at 12:04 PM Kienan Stewart <kstewart@efficios.com> wrote:
> > >>
> > >> Hi Brian,
> > >>
> > >> On 7/22/24 6:00 PM, Brian Hutchinson wrote:
> > >>> Hi Kienan,
> > >>>
> > >>> Took a while to gather your grocery list but I think I have most of it
> > >>> below ;)
> > >>
> > >> thanks for all the extra info. Replies inline below, but I'll cut a lot
> > >> of the long output for readability.
> > >>
> > >> tl;dr the environment continues to be weird, but my present suspicion is
> > >> that something in either compilation, the linking of your app (eg. with
> > >> ld when producing the executable), or some post linking stripping might
> > >> be causing issues.
> > >
> > > I'm not aware of any stripping that's going on.  In fact everything is
> > > being built with debug symbols at the moment and I even turned off
> > > optimization ... even used the debug friendly -O flag to see if that
> > > made a difference.
> > >
> > >>
> > >> I will stop digging into further hypotheticals on my side as there is no
> > >> reproducer for both the environment and the application. If you ever end
> > >> up with a minimal reproducer that you can share, I'd be more than happy
> > >> to examine it.
> > >
> > > I'm planning on trying to make a small reproducer I can share but not there yet.
> > >
> >
> > Great! I appreciate that you're taking the time to do so.
> >
> > >>
> > >>>
> > >>> I may have not been clear.  Most of the application components are
> > >>> statically linked but I think there are some that are built as shared
> > >>> objects (.so's) so that's what I was referring to.  I know that
> > >>> lttng-ust is dynamically linked ... I think the lttng-ust docs say this
> > >>> is only option but also makes reference to the fact static linking was
> > >>> once possible (in some versions of the documentation) but not supported
> > >>> anymore (I probably have the docs memorized by now ha, ha ... I've
> > >>> looked at many, many versions of them).
> > >>>
> > >>> Just for full disclosure my ldd looks like:
> > >>>
> > >>>          linux-vdso.so.1 (0x0000ffffab196000)
> > >>>          libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
> > >>>          liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1
> > >>> (0x0000ffffa5750000)
> > >>>          libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
> > >>>          librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
> > >>>          libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
> > >>>          libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
> > >>>          libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
> > >>>          libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
> > >>>          /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
> > >>>          liblttng-ust-common.so.1 =>
> > >>> /usr/local/lib/liblttng-ust-common.so.1 (0x0000ffffa50e0000)
> > >>>          liblttng-ust-tracepoint.so.1 =>
> > >>> /usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
> > >>>          libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
> > >>>          libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)
> > >>>
> > >>>
> > >>
> > >> I find it very suspicious that `liblttng-ust.so.1` is in `/usr/lib`,
> > >> while the other lttng-ust libraries are being loaded from `/usr/local/lib`.
> > >
> > > So Yocto puts all of the lttng libs into /usr/lib.  When I sent the
> > > previous info I was using lttng-tools and modules built by Yocto/OE
> > > and I setup a native build environment on the target so I could run
> > > 'make check' etc., and that's why there were things in /usr/local/lib
> > > because that's where you guys want stuff to be.  So I actually left
> > > the lttng-ust installables in /usr/local/build but also copied them to
> > > /usr/lib to overwrite old Yocto versions there.
> > >
> >
> > It's not so much that it's "where we want it to be". The documentation
> > uses `/usr/local/lib` because `/usr/local` is meant for software
> > installed by the sysadmin administrator, as is the case when building a
> > custom version. `/usr/lib` should be used by packages shipped with the
> > system.
> >
> > C.f. https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s09.html
> >
> > You're free to do as you see fit, but when you start mixing and matching
> > libraries and some are put in /usr/lib by your system packages and some
> > you move there manually I find it more difficult to follow what is going on.
> >
> > >>
> > >> This information also matches the statedump and the LD_DEBUG info from
> > >> later on.
> > >>
> > >> Could you verify some of the following information:
> > >>
> > >> 1. In your build root for lttng-ust, enumerate all the liblttng*so
> > >> files. For each shared object, run `file $libname` and record the value
> > >> of the BuildID hash.R5jow
> > >
> > > Sorry, I'm not following you here.  The only buildID hash I can think
> > > of is with 'eu-unstrip -n' but that's on core files, not individual
> > > libs.  And looking at the options I have for 'file' on my target, I
> > > don't see anything that looks like what you are asking.
> >
> > Perhaps I wasn't clear, the command to run is really just `file`. As a
> > fuller example:
> >
> > ```
> > $ file ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0
> > ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0: ELF 64-bit
> > LSB shared object, x86-64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=b2b4a0fc449cf317e32c23e0bb57ea1ad702b702, with debug_info,
> > not stripped
>
> Ok, feel stupid now.  When I ran the command before, I used short name
> and didn't do it on the long name and just got back:
>
> # file /usr/lib/liblttng-ust.so
> /usr/lib/liblttng-ust.so: symbolic link to liblttng-ust.so.1.0.0
>
> ... and immediately looked at man page to try to figure out what
> switch showed BuildID etc., ha, ha.
>
> When I do it on long name here is what I see:
>
> # file /usr/lib/liblttng-ust-common.so.1.0.0
> /usr/lib/liblttng-ust-common.so.1.0.0: ELF 64-bit LSB shared object,
> ARM aarch64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=a50c9a77163b6b91e1f84e57d167c7b77ae707a3, with
> debug_info, not stripped
> # file /usr/lib/liblttng-ust-ctl.so.5.0.0
> /usr/lib/liblttng-ust-ctl.so.5.0.0: ELF 64-bit LSB shared object, ARM
> aarch64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=547cccac08721ed1c9a7f3c7ebf1de84ddba7fba, with
> debug_info, not stripped
> # file /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0
> /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0: ELF 64-bit LSB shared
> object, ARM aarch64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=ad5f71ef5e83ab9a972488976c47db265d3360b9, with
> debug_info, not stripped
> # file /usr/lib/liblttng-ust-cyg-profile.so.1.0.0
> /usr/lib/liblttng-ust-cyg-profile.so.1.0.0: ELF 64-bit LSB shared
> object, ARM aarch64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=c2c246c1973bd3241aa4f7229fcbcb27ebe08e82, with
> debug_info, not stripped
> # file /usr/lib/liblttng-ust-dl.so.1.0.0
> /usr/lib/liblttng-ust-dl.so.1.0.0: ELF 64-bit LSB shared object, ARM
> aarch64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=6082ee88c9394319bc3adff16b0b3ea9f8d549ec, with
> debug_info, not stripped
> # file /usr/lib/liblttng-ust-fd.so.1.0.0
> /usr/lib/liblttng-ust-fd.so.1.0.0: ELF 64-bit LSB shared object, ARM
> aarch64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=a2813245af91abe98615771dfd7d5f19b033a410, with
> debug_info, not stripped
> # file /usr/lib/liblttng-ust-fork.so.1.0.0
> /usr/lib/liblttng-ust-fork.so.1.0.0: ELF 64-bit LSB shared object, ARM
> aarch64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=86afa53808502873830c02290f477d4ff8013afb, with
> debug_info, not stripped
> # file /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0
> /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0: ELF 64-bit LSB shared
> object, ARM aarch64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=52f391875a378b5f2c46747a58020b86cb7c9a83, with
> debug_info, not stripped
> # file /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0
> /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0: ELF 64-bit LSB shared
> object, ARM aarch64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=7127223a7e8ed67c6697b95ae1f8ac107df7e47e, with
> debug_info, not stripped
> # file /usr/lib/liblttng-ust-tracepoint.so.1.0.0
> /usr/lib/liblttng-ust-tracepoint.so.1.0.0: ELF 64-bit LSB shared
> object, ARM aarch64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=5971b4d84ec1efe61c6d47c38e92de20569f0f49, with
> debug_info, not stripped
> # file /usr/lib/liblttng-ust.so.1.0.0
> /usr/lib/liblttng-ust.so.1.0.0: ELF 64-bit LSB shared object, ARM
> aarch64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=ce7097ae9bbf42a02dccd386fdfbd37e3224858b, with
> debug_info, not stripped
>
>
> cutting some stuff out cause it's getting long again.
>
> >
> > Sounds like `make check` for lttng-tools passed then?
>
> At first no.  But I think this is because I built the new lttng-tools
> in my on target native environment and ran make check and forgot to do
> make install first, so it was using the older version of lttng-tools.
>
> So then I ran make install of lttng-tools and even did a make clean
> and rebuild of lttng-ust and re-installed it and ran make check on
> both (and things looked a lot better) ... that's where those warnings
> etc., I asked you about came from.
>
> >
> > My understanding at this point is the unit tests are passing for
> > LTTng-UST on your system, as are the unit and regression tests for
> > LTTng-tools. The example programs shipped with LTTng-UST work on your
> > system, as does the example from the documentation. The statedump
> > tracepoints loaded from LTTng-UST are also working fine, as evinced by
> > the program logs and the LTTng trace you shared.
> >
> > Despite my confusion about how exactly you're using the `hello world`
> > tracepoint in your application (as you've now described several
> > variations), the direction this points to for me are details related to
> > how you're using LTTng-UST and/or how are your building and linking your
> > application. To be clear, I don't mean to say that there is or is not an
> > issue in LTTng-UST, but to point at where to examine next in detail
> > including analysis of the produced object files.
>
> I compared the doc/examples/hello-static-lib to what I picked out of
> the LTTng documentation on the web site "Quick start" section and the
> traceprovider headder file is including stddef.h and mine isn't and
> the doc/examples/hello-static-lib/hello.c code is doing a sighandler
> and mine isn't doing any of that either.  I think I've probably posted
> it before but will do it again.  Here is what I'm calling my "hello".
> It's from the lttng documentation but I cut it down even further just
> to make sure I didn't fat finger something.  Like I said before, the
> full hello example from the documentation works.  But when I call
> pretty much the same code from my app it seg faults.
>
> I don't know if the differences I see between my "hello" and the
> "hello-static-lib" matter.
>
> hello-tp.h:
>
> #undef LTTNG_UST_TRACEPOINT_PROVIDER
> #define LTTNG_UST_TRACEPOINT_PROVIDER hello_world
>
> #undef LTTNG_UST_TRACEPOINT_INCLUDE
> #define LTTNG_UST_TRACEPOINT_INCLUDE "./hello-tp.h"
>
> #if !defined(_HELLO_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
> #define _HELLO_TP_H
>
> #include <lttng/tracepoint.h>
>
> LTTNG_UST_TRACEPOINT_EVENT(
>    hello_world,
>    my_first_tracepoint,
>    LTTNG_UST_TP_ARGS(
>        int, my_integer_arg
>    ),
>    LTTNG_UST_TP_FIELDS(
>        lttng_ust_field_integer(int, my_integer_field, my_integer_arg)
>    )
> )
>
> #endif /* _HELLO_TP_H */
>
> #include <lttng/tracepoint-event.h>
>
> hello-tp.c:
>
> #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
>
> #include "hello-tp.h"
>
> From my_app:
>
> #define LTTNG_UST_TRACEPOINT_DEFINE
> //#define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
> #include "hello-tp.h"
>
> .
> .
> .
> lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
>
> In the above case the tpp is static but I've tried to make it a shared
> object too (thus the commented out DYNAMIC_LINKAGE above) but get the
> same result.
>
> Again, I think the issue is probably g++ and weak/hidden symbol
> related and or TLS but that's based on the totality of what I've
> experienced over the past year or so and seeing the
> experiences/problems of others in the lttng-dev archives.
>
> Regards,
>
> Brian

[-- Attachment #2: make-check.tar.xz --]
[-- Type: application/x-xz, Size: 70348 bytes --]

[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] [yocto] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Khem Raj via lttng-dev]

check if you have CXXFLAGS += "-fvisibility-inlines-hidden" and try to
toggle it and see if that helps.

On Tue, Jul 30, 2024 at 8:54 AM Brian Hutchinson via
lists.yoctoproject.org <b.hutchman=gmail.com@lists.yoctoproject.org>
wrote:
>
> Hi Kienan,
>
> I noticed looking thru the 38k line LTTNG_UST_DEBUG session this line:
>
> "       810:     /usr/lib/liblttng-ust.so.1: error: symbol lookup
> error: undefined symbol: ltt_probe_register (fatal)"
>
> ... and it jogged my memory that I did see some abi conflict messages
> in lttng-ust "make check" that I don't know if they are "good" or
> "bad" but could be related???
>
> Anyway, attaching a tar of lttng-ust and lttng-tools "make check" for
> your enjoyment.
>
> Regards,
>
> Brian
>
>
>
>
> On Tue, Jul 30, 2024 at 8:40 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
> >
> > On Mon, Jul 29, 2024 at 3:03 PM Kienan Stewart <kstewart@efficios.com> wrote:
> > >
> > > Hi Brian,
> > >
> > > On 7/25/24 3:54 PM, Brian Hutchinson wrote:
> > > > Hi Kienan,
> > > >
> > > > I'll answer your questions below, but I've got questions on what I saw
> > > > building and installing lttng-tools (2.13.13) and lttng-ust (2.13.8).
> > > >
> > > > Based on the struggles I've had trying to get lttng to work with my
> > > > app over various Yocto versions (Dunfell & Kirkstone) and lttng
> > > > version, I think the problems I'm facing are mostly around C++ and
> > > > weak and hidden symbols in Yocto toolchain.
> > > >
> > > > When I started my app with the options you mentioned previously a
> > > > while back, Id see things like:
> > > >
> > > > # LTTNG_UST_DEBUG=1 LTTNG_UST_REGISTER_TIMEOUT=-1 /opt/tc/TrafficController
> > > > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > > > with hidden visibility for integer objects as SAME address between
> > > > compile units part of the same module. (in check_weak_hidden() at
> > > > tracepoint.c:1012)
> > > > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > > > with hidden visibility for pointer objects as SAME address between
> > > > compile units part of the same module. (in check_weak_hidden() at
> > > > tracepoint.c:1016)
> > > > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > > > with hidden visibility for 24-byte structure objects as SAME address
> > > > between compile units part of the same module. (in check_weak_hidden()
> > > > at tracepoint.c:1020)
> > > >
> > >
> > > These messages are extra information for debugging and not indicative of
> > > a problem in of itself. C.f.
> > > https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/src/lib/lttng-ust-tracepoint/tracepoint.c#L1010
> > >
> > > There is a unit test related to this:
> > > https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/tests/unit/gcc-weak-hidden/main.c#L76
> > >
> > >
> > > > I further researched this whole 'weak symbol' and 'hidden visibility'
> > > > topic in the lttng-dev archives and it smells a lot like what I've
> > > > been seeing.  You should be able to mix both tracef and tracepoint
> > > > calls in souce code ... but I could not.  I could get a tracef call to
> > > > work but if I put a tracepoint call in the same code then nothing
> > > > would work.  This was with Dunfell 3.1.7 and earlier versions of
> > > > lttng.
> > > >
> > > > At one point I could get a tracepoint call to work but I'd have to let
> > > > our cmake build system build and link the tpp.c file and then turn
> > > > around and use gcc to recompile it and copy it to where all the
> > > > objects were to create the huge .a library the app was built against.
> > > > That's when I first learned there are issues with C++.  I think g++ is
> > > > used to build even .c files that aren't c++.
> > > >
> > > > Then if I tried to put a tracepoint in another sub project, none of
> > > > the tracepoints would work and I'd get empty traces.  This is a
> > > > symptom of the 'weak symbols with hidden visibility' issue ... and I
> > > > finally found others that were having same issue in the archives.  I
> > > > don't fully understand the issues here, although I do understand some
> > > > of what's going on ... I just don't know what to do about it.
> > > >
> > >
> > > You said initially said that you're using `lttng_ust_tracepoint` exactly
> > > as the hello world from the documentation; however, you have just
> > > described several attempts at doing different things. Which case are we
> > > trying to understand here?
> >
> > lttng_ust_tracepoint.  I only mentioned prior tests for context to
> > similar struggles from a year or more ago.
> >
> >
> > >
> > > > At this point I was being encouraged to keep upgrading to newer
> > > > versions of lttng.  Our app never changed, gcc & lttng etc., kept
> > > > changing.  Now with newer versions nothing runs, all I get is an
> > > > immediate segfault.  Again, I'm building just like I did before a year
> > > > or so ago with older versions of Yocto and lttng.  I say all of that
> > > > to give perspective and history of what I've seen and experienced.
> > > > Now this TLS thing has entered the picture too and so far I've only
> > > > changed lttng, I don't know if I should be applying patches to my gcc
> > > > for that issue.  Like I said, I'm currently using Yocto Kirkstone
> > > > 4.0.18 and 6.1.38 kernel.
> > > >
> > > > Now I'll move into the area of things I've seen building/installing
> > > > lttng-tools and lttng-ust natively on the target environment I've
> > > > setup where I can run 'make check' etc.  These are in the category of
> > > > "hey, is this ok, should I be worried about this":
> > > >
> > > > While building lttng-tools I see things like:
> > > >
> > > > *** Warning: Linking the executable userspace-probe-elf-binary against
> > > > the loadable module
> > > > *** libfoo.so is not portable!
> > > >
> > >
> > > The library is for a test program. My understanding is that the library
> > > is compiled that way to force a stripped shared object to be produced in
> > > order to validate that symbol lookups in libraries with no symtab
> > > function as expected by using the dynsym table.
> > >
> > > C.f.
> > > https://github.com/lttng/lttng-tools/commit/ef3dfe5d31c88fb548189a6441aaf8b2afc0bd4b
> > >
> > > > In file included from ../../../src/common/macros.h:15,
> > > >                  from ../../../include/lttng/health-internal.h:19,
> > > >                  from lttng-ctl-health.c:19:
> > > > In function 'lttng_strnlen',
> > > >     inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
> > > >     inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
> > > >     inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> > > > ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> > > > specified bound 4096 may exceed source size 37 [-Wstringop-overread]
> > > >    19 |         return strnlen(str, max);
> > > >       |                ^~~~~~~~~~~~~~~~~
> > > > lttng-ctl-health.c: At top level:
> > > > cc1: note: unrecognized command-line option
> > > > '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> > > > earlier diagnostics
> > >
> > > This warning is addressed in
> > > https://github.com/lttng/lttng-tools/commit/b25a59916106e5055be516f61f183a48f459b0b3
> > >
> > > > ** Warning: Linking the shared library libbar.la against the loadable module
> > > > *** libzzz.so is not portable!
> > > >
> > > > *** Warning: Linking the shared library libfoo.la against the loadable module
> > > > *** libbar.so is not portable!
> > > >
> > > > While installing lttng-tools I see things like this:
> > > >
> > > > make[4]: Entering directory '/opt/lttng/lttng-tools-2.13.13/src/lib/lttng-ctl'
> > > >   CC       lttng-ctl.lo
> > > >   CC       snapshot.lo
> > > >   CC       lttng-ctl-health.lo
> > > > In file included from ../../../src/common/macros.h:15,
> > > >                  from ../../../include/lttng/health-internal.h:19,
> > > >                  from lttng-ctl-health.c:19:
> > > > In function 'lttng_strnlen',
> > > >     inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
> > > >     inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
> > > >     inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> > > > ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> > > > specified bound 4096 may exceed source size 37 [-Wstringop-overread]
> > > >    19 |         return strnlen(str, max);
> > > >       |                ^~~~~~~~~~~~~~~~~
> > > > lttng-ctl-health.c: At top level:
> > > > cc1: note: unrecognized command-line option
> > > > '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> > > > earlier diagnostics
> > > >
> > > > Making install in trigger-condition-event-matches
> > > > make[2]: Entering directory
> > > > '/opt/lttng/lttng-tools-2.13.13/doc/examples/trigger-condition-event-matches'
> > > >   CC       instrumented-app.o
> > > >   CC       tracepoint-trigger-example.o
> > > >   AR       libtracepoint-trigger-example.a
> > > > ar: `u' modifier ignored since `D' is the default (see `U')
> > > >
> > > > While building lttng-ust I see things like:
> > > >
> > > > Making all in utils
> > > > make[2]: Entering directory
> > > > '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
> > > >   CC       tap.o
> > > >   AR       libtap.a
> > > > ar: `u' modifier ignored since `D' is the default (see `U')
> > > >
> > >
> > > While libtool now uses `cr` by default, automake still defines the
> > > default to `cru` which is what ends up getting used in the example.
> > > Since many distros have changed the configuration of ar such that 'D' is
> > > the default rather than the previous behaviour 'U', 'u' is redundant.
> > >
> > > The behaviour in automake has been changed in automake 1.16.90+.
> > >
> > > C.f.
> > > https://github.com/autotools-mirror/libtool/commit/418129bc63afc312701e84cb8afa5ca413df1ab5
> > >
> > > C.f.
> > > http://git.savannah.gnu.org/cgit/automake.git/commit/?id=8cdbdda5aec652c356fe6dbba96810202176ae75
> > >
> > > > *** Warning: Linking the shared library libzero.la against the
> > > > loadable module
> > > > *** libfakeust0.so is not portable!
> > > >   CCLD     app_noust_indirect_abi0
> > > >
> > > > *** Warning: Linking the executable app_noust_indirect_abi0 against
> > > > the loadable module
> > > > *** libzero.so is not portable!
> > > >   CC       app_noust_indirect_abi0_abi1-app_noust.o
> > > >   CC       libone.lo
> > > >   CCLD     libone.la
> > > >   CCLD     app_noust_indirect_abi0_abi1
> > > >
> > > > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > > > against the loadable module
> > > > *** libzero.so is not portable!
> > > >
> > > > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > > > against the loadable module
> > > > *** libone.so is not portable!
> > > >   CC       app_noust_indirect_abi1-app_noust.o
> > > >   CCLD     app_noust_indirect_abi1
> > > >
> > > > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > > > against the loadable module
> > > > *** libone.so is not portable!
> > > >   CC       app_noust_indirect_abi1-app_noust.o
> > > >   CCLD     app_noust_indirect_abi1
> > > >
> > > > *** Warning: Linking the executable app_noust_indirect_abi1 against
> > > > the loadable module
> > > > *** libone.so is not portable!
> > > >   CC       app_ust.o
> > > >   CC       tp.o
> > > >   CCLD     app_ust
> > > >   CC       app_ust_dlopen.o
> > > >   CCLD     app_ust_dlopen
> > > >   CC       app_ust_indirect_abi0-app_ust.o
> > > >   CC       app_ust_indirect_abi0-tp.o
> > > >   CCLD     app_ust_indirect_abi0
> > > >
> > > > *** Warning: Linking the executable app_ust_indirect_abi0 against the
> > > > loadable module
> > > > *** libzero.so is not portable!
> > > >   CC       app_ust_indirect_abi0_abi1-app_ust.o
> > > >   CC       app_ust_indirect_abi0_abi1-tp.o
> > > >   CCLD     app_ust_indirect_abi0_abi1
> > > >
> > > > *** Warning: Linking the executable app_ust_indirect_abi0_abi1 against
> > > > the loadable module
> > > > *** libzero.so is not portable!
> > > >
> > > > I don't know if these are ok or if I should be worried about any of that.
> > > >
> > >
> > > These are all for different tests.
> > >
> > > > ... now on to your questions below.
> > > >
> > > >
> > > >
> > > > On Wed, Jul 24, 2024 at 12:04 PM Kienan Stewart <kstewart@efficios.com> wrote:
> > > >>
> > > >> Hi Brian,
> > > >>
> > > >> On 7/22/24 6:00 PM, Brian Hutchinson wrote:
> > > >>> Hi Kienan,
> > > >>>
> > > >>> Took a while to gather your grocery list but I think I have most of it
> > > >>> below ;)
> > > >>
> > > >> thanks for all the extra info. Replies inline below, but I'll cut a lot
> > > >> of the long output for readability.
> > > >>
> > > >> tl;dr the environment continues to be weird, but my present suspicion is
> > > >> that something in either compilation, the linking of your app (eg. with
> > > >> ld when producing the executable), or some post linking stripping might
> > > >> be causing issues.
> > > >
> > > > I'm not aware of any stripping that's going on.  In fact everything is
> > > > being built with debug symbols at the moment and I even turned off
> > > > optimization ... even used the debug friendly -O flag to see if that
> > > > made a difference.
> > > >
> > > >>
> > > >> I will stop digging into further hypotheticals on my side as there is no
> > > >> reproducer for both the environment and the application. If you ever end
> > > >> up with a minimal reproducer that you can share, I'd be more than happy
> > > >> to examine it.
> > > >
> > > > I'm planning on trying to make a small reproducer I can share but not there yet.
> > > >
> > >
> > > Great! I appreciate that you're taking the time to do so.
> > >
> > > >>
> > > >>>
> > > >>> I may have not been clear.  Most of the application components are
> > > >>> statically linked but I think there are some that are built as shared
> > > >>> objects (.so's) so that's what I was referring to.  I know that
> > > >>> lttng-ust is dynamically linked ... I think the lttng-ust docs say this
> > > >>> is only option but also makes reference to the fact static linking was
> > > >>> once possible (in some versions of the documentation) but not supported
> > > >>> anymore (I probably have the docs memorized by now ha, ha ... I've
> > > >>> looked at many, many versions of them).
> > > >>>
> > > >>> Just for full disclosure my ldd looks like:
> > > >>>
> > > >>>          linux-vdso.so.1 (0x0000ffffab196000)
> > > >>>          libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
> > > >>>          liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1
> > > >>> (0x0000ffffa5750000)
> > > >>>          libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
> > > >>>          librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
> > > >>>          libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
> > > >>>          libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
> > > >>>          libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
> > > >>>          libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
> > > >>>          /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
> > > >>>          liblttng-ust-common.so.1 =>
> > > >>> /usr/local/lib/liblttng-ust-common.so.1 (0x0000ffffa50e0000)
> > > >>>          liblttng-ust-tracepoint.so.1 =>
> > > >>> /usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
> > > >>>          libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
> > > >>>          libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)
> > > >>>
> > > >>>
> > > >>
> > > >> I find it very suspicious that `liblttng-ust.so.1` is in `/usr/lib`,
> > > >> while the other lttng-ust libraries are being loaded from `/usr/local/lib`.
> > > >
> > > > So Yocto puts all of the lttng libs into /usr/lib.  When I sent the
> > > > previous info I was using lttng-tools and modules built by Yocto/OE
> > > > and I setup a native build environment on the target so I could run
> > > > 'make check' etc., and that's why there were things in /usr/local/lib
> > > > because that's where you guys want stuff to be.  So I actually left
> > > > the lttng-ust installables in /usr/local/build but also copied them to
> > > > /usr/lib to overwrite old Yocto versions there.
> > > >
> > >
> > > It's not so much that it's "where we want it to be". The documentation
> > > uses `/usr/local/lib` because `/usr/local` is meant for software
> > > installed by the sysadmin administrator, as is the case when building a
> > > custom version. `/usr/lib` should be used by packages shipped with the
> > > system.
> > >
> > > C.f. https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s09.html
> > >
> > > You're free to do as you see fit, but when you start mixing and matching
> > > libraries and some are put in /usr/lib by your system packages and some
> > > you move there manually I find it more difficult to follow what is going on.
> > >
> > > >>
> > > >> This information also matches the statedump and the LD_DEBUG info from
> > > >> later on.
> > > >>
> > > >> Could you verify some of the following information:
> > > >>
> > > >> 1. In your build root for lttng-ust, enumerate all the liblttng*so
> > > >> files. For each shared object, run `file $libname` and record the value
> > > >> of the BuildID hash.R5jow
> > > >
> > > > Sorry, I'm not following you here.  The only buildID hash I can think
> > > > of is with 'eu-unstrip -n' but that's on core files, not individual
> > > > libs.  And looking at the options I have for 'file' on my target, I
> > > > don't see anything that looks like what you are asking.
> > >
> > > Perhaps I wasn't clear, the command to run is really just `file`. As a
> > > fuller example:
> > >
> > > ```
> > > $ file ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0
> > > ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0: ELF 64-bit
> > > LSB shared object, x86-64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=b2b4a0fc449cf317e32c23e0bb57ea1ad702b702, with debug_info,
> > > not stripped
> >
> > Ok, feel stupid now.  When I ran the command before, I used short name
> > and didn't do it on the long name and just got back:
> >
> > # file /usr/lib/liblttng-ust.so
> > /usr/lib/liblttng-ust.so: symbolic link to liblttng-ust.so.1.0.0
> >
> > ... and immediately looked at man page to try to figure out what
> > switch showed BuildID etc., ha, ha.
> >
> > When I do it on long name here is what I see:
> >
> > # file /usr/lib/liblttng-ust-common.so.1.0.0
> > /usr/lib/liblttng-ust-common.so.1.0.0: ELF 64-bit LSB shared object,
> > ARM aarch64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=a50c9a77163b6b91e1f84e57d167c7b77ae707a3, with
> > debug_info, not stripped
> > # file /usr/lib/liblttng-ust-ctl.so.5.0.0
> > /usr/lib/liblttng-ust-ctl.so.5.0.0: ELF 64-bit LSB shared object, ARM
> > aarch64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=547cccac08721ed1c9a7f3c7ebf1de84ddba7fba, with
> > debug_info, not stripped
> > # file /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0
> > /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0: ELF 64-bit LSB shared
> > object, ARM aarch64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=ad5f71ef5e83ab9a972488976c47db265d3360b9, with
> > debug_info, not stripped
> > # file /usr/lib/liblttng-ust-cyg-profile.so.1.0.0
> > /usr/lib/liblttng-ust-cyg-profile.so.1.0.0: ELF 64-bit LSB shared
> > object, ARM aarch64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=c2c246c1973bd3241aa4f7229fcbcb27ebe08e82, with
> > debug_info, not stripped
> > # file /usr/lib/liblttng-ust-dl.so.1.0.0
> > /usr/lib/liblttng-ust-dl.so.1.0.0: ELF 64-bit LSB shared object, ARM
> > aarch64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=6082ee88c9394319bc3adff16b0b3ea9f8d549ec, with
> > debug_info, not stripped
> > # file /usr/lib/liblttng-ust-fd.so.1.0.0
> > /usr/lib/liblttng-ust-fd.so.1.0.0: ELF 64-bit LSB shared object, ARM
> > aarch64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=a2813245af91abe98615771dfd7d5f19b033a410, with
> > debug_info, not stripped
> > # file /usr/lib/liblttng-ust-fork.so.1.0.0
> > /usr/lib/liblttng-ust-fork.so.1.0.0: ELF 64-bit LSB shared object, ARM
> > aarch64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=86afa53808502873830c02290f477d4ff8013afb, with
> > debug_info, not stripped
> > # file /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0
> > /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0: ELF 64-bit LSB shared
> > object, ARM aarch64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=52f391875a378b5f2c46747a58020b86cb7c9a83, with
> > debug_info, not stripped
> > # file /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0
> > /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0: ELF 64-bit LSB shared
> > object, ARM aarch64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=7127223a7e8ed67c6697b95ae1f8ac107df7e47e, with
> > debug_info, not stripped
> > # file /usr/lib/liblttng-ust-tracepoint.so.1.0.0
> > /usr/lib/liblttng-ust-tracepoint.so.1.0.0: ELF 64-bit LSB shared
> > object, ARM aarch64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=5971b4d84ec1efe61c6d47c38e92de20569f0f49, with
> > debug_info, not stripped
> > # file /usr/lib/liblttng-ust.so.1.0.0
> > /usr/lib/liblttng-ust.so.1.0.0: ELF 64-bit LSB shared object, ARM
> > aarch64, version 1 (SYSV), dynamically linked,
> > BuildID[sha1]=ce7097ae9bbf42a02dccd386fdfbd37e3224858b, with
> > debug_info, not stripped
> >
> >
> > cutting some stuff out cause it's getting long again.
> >
> > >
> > > Sounds like `make check` for lttng-tools passed then?
> >
> > At first no.  But I think this is because I built the new lttng-tools
> > in my on target native environment and ran make check and forgot to do
> > make install first, so it was using the older version of lttng-tools.
> >
> > So then I ran make install of lttng-tools and even did a make clean
> > and rebuild of lttng-ust and re-installed it and ran make check on
> > both (and things looked a lot better) ... that's where those warnings
> > etc., I asked you about came from.
> >
> > >
> > > My understanding at this point is the unit tests are passing for
> > > LTTng-UST on your system, as are the unit and regression tests for
> > > LTTng-tools. The example programs shipped with LTTng-UST work on your
> > > system, as does the example from the documentation. The statedump
> > > tracepoints loaded from LTTng-UST are also working fine, as evinced by
> > > the program logs and the LTTng trace you shared.
> > >
> > > Despite my confusion about how exactly you're using the `hello world`
> > > tracepoint in your application (as you've now described several
> > > variations), the direction this points to for me are details related to
> > > how you're using LTTng-UST and/or how are your building and linking your
> > > application. To be clear, I don't mean to say that there is or is not an
> > > issue in LTTng-UST, but to point at where to examine next in detail
> > > including analysis of the produced object files.
> >
> > I compared the doc/examples/hello-static-lib to what I picked out of
> > the LTTng documentation on the web site "Quick start" section and the
> > traceprovider headder file is including stddef.h and mine isn't and
> > the doc/examples/hello-static-lib/hello.c code is doing a sighandler
> > and mine isn't doing any of that either.  I think I've probably posted
> > it before but will do it again.  Here is what I'm calling my "hello".
> > It's from the lttng documentation but I cut it down even further just
> > to make sure I didn't fat finger something.  Like I said before, the
> > full hello example from the documentation works.  But when I call
> > pretty much the same code from my app it seg faults.
> >
> > I don't know if the differences I see between my "hello" and the
> > "hello-static-lib" matter.
> >
> > hello-tp.h:
> >
> > #undef LTTNG_UST_TRACEPOINT_PROVIDER
> > #define LTTNG_UST_TRACEPOINT_PROVIDER hello_world
> >
> > #undef LTTNG_UST_TRACEPOINT_INCLUDE
> > #define LTTNG_UST_TRACEPOINT_INCLUDE "./hello-tp.h"
> >
> > #if !defined(_HELLO_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
> > #define _HELLO_TP_H
> >
> > #include <lttng/tracepoint.h>
> >
> > LTTNG_UST_TRACEPOINT_EVENT(
> >    hello_world,
> >    my_first_tracepoint,
> >    LTTNG_UST_TP_ARGS(
> >        int, my_integer_arg
> >    ),
> >    LTTNG_UST_TP_FIELDS(
> >        lttng_ust_field_integer(int, my_integer_field, my_integer_arg)
> >    )
> > )
> >
> > #endif /* _HELLO_TP_H */
> >
> > #include <lttng/tracepoint-event.h>
> >
> > hello-tp.c:
> >
> > #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
> >
> > #include "hello-tp.h"
> >
> > From my_app:
> >
> > #define LTTNG_UST_TRACEPOINT_DEFINE
> > //#define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
> > #include "hello-tp.h"
> >
> > .
> > .
> > .
> > lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
> >
> > In the above case the tpp is static but I've tried to make it a shared
> > object too (thus the commented out DYNAMIC_LINKAGE above) but get the
> > same result.
> >
> > Again, I think the issue is probably g++ and weak/hidden symbol
> > related and or TLS but that's based on the totality of what I've
> > experienced over the past year or so and seeing the
> > experiences/problems of others in the lttng-dev archives.
> >
> > Regards,
> >
> > Brian
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> You automatically follow any topics you start or reply to.
> View/Reply Online (#63605): https://lists.yoctoproject.org/g/yocto/message/63605
> Mute This Topic: https://lists.yoctoproject.org/mt/107264247/1997914
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] [yocto] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev]

Hey Khem,

Good to hear from you again.  Hope you're doing well.

No, I don't see anything like that.  The Yocto generated sdk
environment script I use if I'm cross compiling on PC for the target
looks like this:

# Check for LD_LIBRARY_PATH being set, which can break SDK and
generally is a bad practice
# http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html#AEN80
# http://xahlee.info/UnixResource_dir/_/ldpath.html
# Only disable this check if you are absolutely know what you are doing!
if [ ! -z "${LD_LIBRARY_PATH:-}" ]; then
   echo "Your environment is misconfigured, you probably need to
'unset LD_LIBRARY_PATH'"
   echo "but please check why this was set in the first place and that
it's safe to unset."
   echo "The SDK will not operate correctly in most cases when
LD_LIBRARY_PATH is set."
   echo "For more references see:"
   echo "  http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html#AEN80"
   echo "  http://xahlee.info/UnixResource_dir/_/ldpath.html"
   return 1
fi

export SDKTARGETSYSROOT=/opt/poky/4.0.18/sysroots/cortexa53-crypto-poky-linux
export PATH=/opt/poky/4.0.18/sysroots/x86_64-pokysdk-linux/usr/bin:/opt/poky/4.0.18/sysroots/x86_64-pokysdk-linux/usr/sbin:/opt/poky/4.0.18/sysroots/x86_64-pokysdk-linux/bin:/opt/poky/4.0.18/sysroots/x86_64-pokysdk-linux/sbin:/opt/poky/4.0.18/sysroots/x86_64-pokysdk-linux/usr/
bin/../x86_64-pokysdk-linux/bin:/opt/poky/4.0.18/sysroots/x86_64-pokysdk-linux/usr/bin/aarch64-poky-linux:/opt/poky/4.0.18/sysroots/x86_64-pokysdk-linux/usr/bin/aarch64-poky-linux-musl:"$PATH"
export PKG_CONFIG_SYSROOT_DIR=$SDKTARGETSYSROOT
export PKG_CONFIG_PATH=$SDKTARGETSYSROOT/usr/lib/pkgconfig:$SDKTARGETSYSROOT/usr/share/pkgconfig
export CONFIG_SITE=/opt/poky/4.0.18/site-config-cortexa53-crypto-poky-linux
export OECORE_NATIVE_SYSROOT="/opt/poky/4.0.18/sysroots/x86_64-pokysdk-linux"
export OECORE_TARGET_SYSROOT="$SDKTARGETSYSROOT"
export OECORE_ACLOCAL_OPTS="-I
/opt/poky/4.0.18/sysroots/x86_64-pokysdk-linux/usr/share/aclocal"
export OECORE_BASELIB="lib"
export OECORE_TARGET_ARCH="aarch64"
export OECORE_TARGET_OS="linux"
unset command_not_found_handle
export CC="aarch64-poky-linux-gcc  -mcpu=cortex-a53
-march=armv8-a+crc+crypto -mbranch-protection=standard
-fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat
-Wformat-security -Werror=format-security --sysroot=$SDKTARGETSYSROOT"
export CXX="aarch64-poky-linux-g++  -mcpu=cortex-a53
-march=armv8-a+crc+crypto -mbranch-protection=standard
-fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat
-Wformat-security -Werror=format-security --sysroot=$SDKTARGETSYSROOT"
export CPP="aarch64-poky-linux-gcc -E  -mcpu=cortex-a53
-march=armv8-a+crc+crypto -mbranch-protection=standard
-fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat
-Wformat-security -Werror=format-security --sysroot=$SDKTARGETSYSROOT"
export AS="aarch64-poky-linux-as "
export LD="aarch64-poky-linux-ld  --sysroot=$SDKTARGETSYSROOT"
export GDB=aarch64-poky-linux-gdb
export STRIP=aarch64-poky-linux-strip
export RANLIB=aarch64-poky-linux-ranlib
export OBJCOPY=aarch64-poky-linux-objcopy
export OBJDUMP=aarch64-poky-linux-objdump
export READELF=aarch64-poky-linux-readelf
export AR=aarch64-poky-linux-ar
export NM=aarch64-poky-linux-nm
export M4=m4
export TARGET_PREFIX=aarch64-poky-linux-
export CONFIGURE_FLAGS="--target=aarch64-poky-linux
--host=aarch64-poky-linux --build=x86_64-linux
--with-libtool-sysroot=$SDKTARGETSYSROOT"
export CFLAGS=" -O2 -pipe -g -feliminate-unused-debug-types "
export CXXFLAGS=" -O2 -pipe -g -feliminate-unused-debug-types "
export LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed
-Wl,-z,relro,-z,now"
export CPPFLAGS=""
export KCFLAGS="--sysroot=$SDKTARGETSYSROOT"
export OECORE_DISTRO_VERSION="4.0.18"
export OECORE_SDK_VERSION="4.0.18"
export ARCH=arm64
export CROSS_COMPILE=aarch64-poky-linux-
export OECORE_TUNE_CCARGS=" -mcpu=cortex-a53 -march=armv8-a+crc+crypto
-mbranch-protection=standard"

# Append environment subscripts
if [ -d "$OECORE_TARGET_SYSROOT/environment-setup.d" ]; then
   for envfile in $OECORE_TARGET_SYSROOT/environment-setup.d/*.sh; do
           . $envfile
   done
fi
if [ -d "$OECORE_NATIVE_SYSROOT/environment-setup.d" ]; then
   for envfile in $OECORE_NATIVE_SYSROOT/environment-setup.d/*.sh; do
           . $envfile
   done
fi

But I set myself up with a native build environment on the target to
be able to better run some of the lttng tests to verify it's
compatible with the aarch64 platform (NXP imx8mm in this case) I'm on
etc..

Not using the Yocto environment-setup script on the target and relying
on the built-in values.  If I look at some cmake logs, I see these
options:

COLLECT_GCC=/usr/bin/cc
Target: aarch64-poky-linux
Configured with:
../../../../../../work-shared/gcc-11.4.0-r0/gcc-11.4.0/configure
--build=x86_64-linux --host=aarch64-poky-linux
--target=aarch64-poky-linux --prefix=/usr --exec_prefix=/usr
--bindir=/usr/bin --sbindir=/usr/sbin --libexecdir=/usr/libexec
--datadir=/usr/share --
sysconfdir=/etc --sharedstatedir=/com --localstatedir=/var
--libdir=/usr/lib --includedir=/usr/include
--oldincludedir=/usr/include --infodir=/usr/share/info
--mandir=/usr/share/man --disable-silent-rules
--disable-dependency-tracking --with-libtool-sysroot=/ --with-gnu-ld
--e
nable-shared --enable-languages=c,c++ --enable-threads=posix
--enable-multilib --enable-default-pie --enable-c99 --enable-long-long
--enable-symvers=gnu --enable-libstdcxx-pch
--program-prefix=aarch64-poky-linux- --without-local-prefix
--disable-install-libiberty --disable-lib
ssp --enable-libitm --enable-lto --disable-bootstrap
--with-system-zlib --with-linker-hash-style=gnu
--enable-linker-build-id --with-ppl=no --with-cloog=no
--enable-checking=release --enable-cheaders=c_global --without-isl
--with-build-sysroot=/ --disable-static --enable-nls -
-with-glibc-version=2.28 --enable-initfini-array --enable-__cxa_atexit
Thread model: posix

Regards,

Brian

On Tue, Jul 30, 2024 at 11:56 AM Khem Raj <raj.khem@gmail.com> wrote:
>
> check if you have CXXFLAGS += "-fvisibility-inlines-hidden" and try to
> toggle it and see if that helps.
>
> On Tue, Jul 30, 2024 at 8:54 AM Brian Hutchinson via
> lists.yoctoproject.org <b.hutchman=gmail.com@lists.yoctoproject.org>
> wrote:
> >
> > Hi Kienan,
> >
> > I noticed looking thru the 38k line LTTNG_UST_DEBUG session this line:
> >
> > "       810:     /usr/lib/liblttng-ust.so.1: error: symbol lookup
> > error: undefined symbol: ltt_probe_register (fatal)"
> >
> > ... and it jogged my memory that I did see some abi conflict messages
> > in lttng-ust "make check" that I don't know if they are "good" or
> > "bad" but could be related???
> >
> > Anyway, attaching a tar of lttng-ust and lttng-tools "make check" for
> > your enjoyment.
> >
> > Regards,
> >
> > Brian
> >
> >
> >
> >
> > On Tue, Jul 30, 2024 at 8:40 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
> > >
> > > On Mon, Jul 29, 2024 at 3:03 PM Kienan Stewart <kstewart@efficios.com> wrote:
> > > >
> > > > Hi Brian,
> > > >
> > > > On 7/25/24 3:54 PM, Brian Hutchinson wrote:
> > > > > Hi Kienan,
> > > > >
> > > > > I'll answer your questions below, but I've got questions on what I saw
> > > > > building and installing lttng-tools (2.13.13) and lttng-ust (2.13.8).
> > > > >
> > > > > Based on the struggles I've had trying to get lttng to work with my
> > > > > app over various Yocto versions (Dunfell & Kirkstone) and lttng
> > > > > version, I think the problems I'm facing are mostly around C++ and
> > > > > weak and hidden symbols in Yocto toolchain.
> > > > >
> > > > > When I started my app with the options you mentioned previously a
> > > > > while back, Id see things like:
> > > > >
> > > > > # LTTNG_UST_DEBUG=1 LTTNG_UST_REGISTER_TIMEOUT=-1 /opt/tc/TrafficController
> > > > > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > > > > with hidden visibility for integer objects as SAME address between
> > > > > compile units part of the same module. (in check_weak_hidden() at
> > > > > tracepoint.c:1012)
> > > > > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > > > > with hidden visibility for pointer objects as SAME address between
> > > > > compile units part of the same module. (in check_weak_hidden() at
> > > > > tracepoint.c:1016)
> > > > > liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> > > > > with hidden visibility for 24-byte structure objects as SAME address
> > > > > between compile units part of the same module. (in check_weak_hidden()
> > > > > at tracepoint.c:1020)
> > > > >
> > > >
> > > > These messages are extra information for debugging and not indicative of
> > > > a problem in of itself. C.f.
> > > > https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/src/lib/lttng-ust-tracepoint/tracepoint.c#L1010
> > > >
> > > > There is a unit test related to this:
> > > > https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/tests/unit/gcc-weak-hidden/main.c#L76
> > > >
> > > >
> > > > > I further researched this whole 'weak symbol' and 'hidden visibility'
> > > > > topic in the lttng-dev archives and it smells a lot like what I've
> > > > > been seeing.  You should be able to mix both tracef and tracepoint
> > > > > calls in souce code ... but I could not.  I could get a tracef call to
> > > > > work but if I put a tracepoint call in the same code then nothing
> > > > > would work.  This was with Dunfell 3.1.7 and earlier versions of
> > > > > lttng.
> > > > >
> > > > > At one point I could get a tracepoint call to work but I'd have to let
> > > > > our cmake build system build and link the tpp.c file and then turn
> > > > > around and use gcc to recompile it and copy it to where all the
> > > > > objects were to create the huge .a library the app was built against.
> > > > > That's when I first learned there are issues with C++.  I think g++ is
> > > > > used to build even .c files that aren't c++.
> > > > >
> > > > > Then if I tried to put a tracepoint in another sub project, none of
> > > > > the tracepoints would work and I'd get empty traces.  This is a
> > > > > symptom of the 'weak symbols with hidden visibility' issue ... and I
> > > > > finally found others that were having same issue in the archives.  I
> > > > > don't fully understand the issues here, although I do understand some
> > > > > of what's going on ... I just don't know what to do about it.
> > > > >
> > > >
> > > > You said initially said that you're using `lttng_ust_tracepoint` exactly
> > > > as the hello world from the documentation; however, you have just
> > > > described several attempts at doing different things. Which case are we
> > > > trying to understand here?
> > >
> > > lttng_ust_tracepoint.  I only mentioned prior tests for context to
> > > similar struggles from a year or more ago.
> > >
> > >
> > > >
> > > > > At this point I was being encouraged to keep upgrading to newer
> > > > > versions of lttng.  Our app never changed, gcc & lttng etc., kept
> > > > > changing.  Now with newer versions nothing runs, all I get is an
> > > > > immediate segfault.  Again, I'm building just like I did before a year
> > > > > or so ago with older versions of Yocto and lttng.  I say all of that
> > > > > to give perspective and history of what I've seen and experienced.
> > > > > Now this TLS thing has entered the picture too and so far I've only
> > > > > changed lttng, I don't know if I should be applying patches to my gcc
> > > > > for that issue.  Like I said, I'm currently using Yocto Kirkstone
> > > > > 4.0.18 and 6.1.38 kernel.
> > > > >
> > > > > Now I'll move into the area of things I've seen building/installing
> > > > > lttng-tools and lttng-ust natively on the target environment I've
> > > > > setup where I can run 'make check' etc.  These are in the category of
> > > > > "hey, is this ok, should I be worried about this":
> > > > >
> > > > > While building lttng-tools I see things like:
> > > > >
> > > > > *** Warning: Linking the executable userspace-probe-elf-binary against
> > > > > the loadable module
> > > > > *** libfoo.so is not portable!
> > > > >
> > > >
> > > > The library is for a test program. My understanding is that the library
> > > > is compiled that way to force a stripped shared object to be produced in
> > > > order to validate that symbol lookups in libraries with no symtab
> > > > function as expected by using the dynsym table.
> > > >
> > > > C.f.
> > > > https://github.com/lttng/lttng-tools/commit/ef3dfe5d31c88fb548189a6441aaf8b2afc0bd4b
> > > >
> > > > > In file included from ../../../src/common/macros.h:15,
> > > > >                  from ../../../include/lttng/health-internal.h:19,
> > > > >                  from lttng-ctl-health.c:19:
> > > > > In function 'lttng_strnlen',
> > > > >     inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
> > > > >     inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
> > > > >     inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> > > > > ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> > > > > specified bound 4096 may exceed source size 37 [-Wstringop-overread]
> > > > >    19 |         return strnlen(str, max);
> > > > >       |                ^~~~~~~~~~~~~~~~~
> > > > > lttng-ctl-health.c: At top level:
> > > > > cc1: note: unrecognized command-line option
> > > > > '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> > > > > earlier diagnostics
> > > >
> > > > This warning is addressed in
> > > > https://github.com/lttng/lttng-tools/commit/b25a59916106e5055be516f61f183a48f459b0b3
> > > >
> > > > > ** Warning: Linking the shared library libbar.la against the loadable module
> > > > > *** libzzz.so is not portable!
> > > > >
> > > > > *** Warning: Linking the shared library libfoo.la against the loadable module
> > > > > *** libbar.so is not portable!
> > > > >
> > > > > While installing lttng-tools I see things like this:
> > > > >
> > > > > make[4]: Entering directory '/opt/lttng/lttng-tools-2.13.13/src/lib/lttng-ctl'
> > > > >   CC       lttng-ctl.lo
> > > > >   CC       snapshot.lo
> > > > >   CC       lttng-ctl-health.lo
> > > > > In file included from ../../../src/common/macros.h:15,
> > > > >                  from ../../../include/lttng/health-internal.h:19,
> > > > >                  from lttng-ctl-health.c:19:
> > > > > In function 'lttng_strnlen',
> > > > >     inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
> > > > >     inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
> > > > >     inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> > > > > ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> > > > > specified bound 4096 may exceed source size 37 [-Wstringop-overread]
> > > > >    19 |         return strnlen(str, max);
> > > > >       |                ^~~~~~~~~~~~~~~~~
> > > > > lttng-ctl-health.c: At top level:
> > > > > cc1: note: unrecognized command-line option
> > > > > '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> > > > > earlier diagnostics
> > > > >
> > > > > Making install in trigger-condition-event-matches
> > > > > make[2]: Entering directory
> > > > > '/opt/lttng/lttng-tools-2.13.13/doc/examples/trigger-condition-event-matches'
> > > > >   CC       instrumented-app.o
> > > > >   CC       tracepoint-trigger-example.o
> > > > >   AR       libtracepoint-trigger-example.a
> > > > > ar: `u' modifier ignored since `D' is the default (see `U')
> > > > >
> > > > > While building lttng-ust I see things like:
> > > > >
> > > > > Making all in utils
> > > > > make[2]: Entering directory
> > > > > '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
> > > > >   CC       tap.o
> > > > >   AR       libtap.a
> > > > > ar: `u' modifier ignored since `D' is the default (see `U')
> > > > >
> > > >
> > > > While libtool now uses `cr` by default, automake still defines the
> > > > default to `cru` which is what ends up getting used in the example.
> > > > Since many distros have changed the configuration of ar such that 'D' is
> > > > the default rather than the previous behaviour 'U', 'u' is redundant.
> > > >
> > > > The behaviour in automake has been changed in automake 1.16.90+.
> > > >
> > > > C.f.
> > > > https://github.com/autotools-mirror/libtool/commit/418129bc63afc312701e84cb8afa5ca413df1ab5
> > > >
> > > > C.f.
> > > > http://git.savannah.gnu.org/cgit/automake.git/commit/?id=8cdbdda5aec652c356fe6dbba96810202176ae75
> > > >
> > > > > *** Warning: Linking the shared library libzero.la against the
> > > > > loadable module
> > > > > *** libfakeust0.so is not portable!
> > > > >   CCLD     app_noust_indirect_abi0
> > > > >
> > > > > *** Warning: Linking the executable app_noust_indirect_abi0 against
> > > > > the loadable module
> > > > > *** libzero.so is not portable!
> > > > >   CC       app_noust_indirect_abi0_abi1-app_noust.o
> > > > >   CC       libone.lo
> > > > >   CCLD     libone.la
> > > > >   CCLD     app_noust_indirect_abi0_abi1
> > > > >
> > > > > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > > > > against the loadable module
> > > > > *** libzero.so is not portable!
> > > > >
> > > > > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > > > > against the loadable module
> > > > > *** libone.so is not portable!
> > > > >   CC       app_noust_indirect_abi1-app_noust.o
> > > > >   CCLD     app_noust_indirect_abi1
> > > > >
> > > > > *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> > > > > against the loadable module
> > > > > *** libone.so is not portable!
> > > > >   CC       app_noust_indirect_abi1-app_noust.o
> > > > >   CCLD     app_noust_indirect_abi1
> > > > >
> > > > > *** Warning: Linking the executable app_noust_indirect_abi1 against
> > > > > the loadable module
> > > > > *** libone.so is not portable!
> > > > >   CC       app_ust.o
> > > > >   CC       tp.o
> > > > >   CCLD     app_ust
> > > > >   CC       app_ust_dlopen.o
> > > > >   CCLD     app_ust_dlopen
> > > > >   CC       app_ust_indirect_abi0-app_ust.o
> > > > >   CC       app_ust_indirect_abi0-tp.o
> > > > >   CCLD     app_ust_indirect_abi0
> > > > >
> > > > > *** Warning: Linking the executable app_ust_indirect_abi0 against the
> > > > > loadable module
> > > > > *** libzero.so is not portable!
> > > > >   CC       app_ust_indirect_abi0_abi1-app_ust.o
> > > > >   CC       app_ust_indirect_abi0_abi1-tp.o
> > > > >   CCLD     app_ust_indirect_abi0_abi1
> > > > >
> > > > > *** Warning: Linking the executable app_ust_indirect_abi0_abi1 against
> > > > > the loadable module
> > > > > *** libzero.so is not portable!
> > > > >
> > > > > I don't know if these are ok or if I should be worried about any of that.
> > > > >
> > > >
> > > > These are all for different tests.
> > > >
> > > > > ... now on to your questions below.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jul 24, 2024 at 12:04 PM Kienan Stewart <kstewart@efficios.com> wrote:
> > > > >>
> > > > >> Hi Brian,
> > > > >>
> > > > >> On 7/22/24 6:00 PM, Brian Hutchinson wrote:
> > > > >>> Hi Kienan,
> > > > >>>
> > > > >>> Took a while to gather your grocery list but I think I have most of it
> > > > >>> below ;)
> > > > >>
> > > > >> thanks for all the extra info. Replies inline below, but I'll cut a lot
> > > > >> of the long output for readability.
> > > > >>
> > > > >> tl;dr the environment continues to be weird, but my present suspicion is
> > > > >> that something in either compilation, the linking of your app (eg. with
> > > > >> ld when producing the executable), or some post linking stripping might
> > > > >> be causing issues.
> > > > >
> > > > > I'm not aware of any stripping that's going on.  In fact everything is
> > > > > being built with debug symbols at the moment and I even turned off
> > > > > optimization ... even used the debug friendly -O flag to see if that
> > > > > made a difference.
> > > > >
> > > > >>
> > > > >> I will stop digging into further hypotheticals on my side as there is no
> > > > >> reproducer for both the environment and the application. If you ever end
> > > > >> up with a minimal reproducer that you can share, I'd be more than happy
> > > > >> to examine it.
> > > > >
> > > > > I'm planning on trying to make a small reproducer I can share but not there yet.
> > > > >
> > > >
> > > > Great! I appreciate that you're taking the time to do so.
> > > >
> > > > >>
> > > > >>>
> > > > >>> I may have not been clear.  Most of the application components are
> > > > >>> statically linked but I think there are some that are built as shared
> > > > >>> objects (.so's) so that's what I was referring to.  I know that
> > > > >>> lttng-ust is dynamically linked ... I think the lttng-ust docs say this
> > > > >>> is only option but also makes reference to the fact static linking was
> > > > >>> once possible (in some versions of the documentation) but not supported
> > > > >>> anymore (I probably have the docs memorized by now ha, ha ... I've
> > > > >>> looked at many, many versions of them).
> > > > >>>
> > > > >>> Just for full disclosure my ldd looks like:
> > > > >>>
> > > > >>>          linux-vdso.so.1 (0x0000ffffab196000)
> > > > >>>          libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
> > > > >>>          liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1
> > > > >>> (0x0000ffffa5750000)
> > > > >>>          libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
> > > > >>>          librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
> > > > >>>          libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
> > > > >>>          libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
> > > > >>>          libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
> > > > >>>          libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
> > > > >>>          /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
> > > > >>>          liblttng-ust-common.so.1 =>
> > > > >>> /usr/local/lib/liblttng-ust-common.so.1 (0x0000ffffa50e0000)
> > > > >>>          liblttng-ust-tracepoint.so.1 =>
> > > > >>> /usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
> > > > >>>          libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
> > > > >>>          libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)
> > > > >>>
> > > > >>>
> > > > >>
> > > > >> I find it very suspicious that `liblttng-ust.so.1` is in `/usr/lib`,
> > > > >> while the other lttng-ust libraries are being loaded from `/usr/local/lib`.
> > > > >
> > > > > So Yocto puts all of the lttng libs into /usr/lib.  When I sent the
> > > > > previous info I was using lttng-tools and modules built by Yocto/OE
> > > > > and I setup a native build environment on the target so I could run
> > > > > 'make check' etc., and that's why there were things in /usr/local/lib
> > > > > because that's where you guys want stuff to be.  So I actually left
> > > > > the lttng-ust installables in /usr/local/build but also copied them to
> > > > > /usr/lib to overwrite old Yocto versions there.
> > > > >
> > > >
> > > > It's not so much that it's "where we want it to be". The documentation
> > > > uses `/usr/local/lib` because `/usr/local` is meant for software
> > > > installed by the sysadmin administrator, as is the case when building a
> > > > custom version. `/usr/lib` should be used by packages shipped with the
> > > > system.
> > > >
> > > > C.f. https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s09.html
> > > >
> > > > You're free to do as you see fit, but when you start mixing and matching
> > > > libraries and some are put in /usr/lib by your system packages and some
> > > > you move there manually I find it more difficult to follow what is going on.
> > > >
> > > > >>
> > > > >> This information also matches the statedump and the LD_DEBUG info from
> > > > >> later on.
> > > > >>
> > > > >> Could you verify some of the following information:
> > > > >>
> > > > >> 1. In your build root for lttng-ust, enumerate all the liblttng*so
> > > > >> files. For each shared object, run `file $libname` and record the value
> > > > >> of the BuildID hash.R5jow
> > > > >
> > > > > Sorry, I'm not following you here.  The only buildID hash I can think
> > > > > of is with 'eu-unstrip -n' but that's on core files, not individual
> > > > > libs.  And looking at the options I have for 'file' on my target, I
> > > > > don't see anything that looks like what you are asking.
> > > >
> > > > Perhaps I wasn't clear, the command to run is really just `file`. As a
> > > > fuller example:
> > > >
> > > > ```
> > > > $ file ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0
> > > > ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0: ELF 64-bit
> > > > LSB shared object, x86-64, version 1 (SYSV), dynamically linked,
> > > > BuildID[sha1]=b2b4a0fc449cf317e32c23e0bb57ea1ad702b702, with debug_info,
> > > > not stripped
> > >
> > > Ok, feel stupid now.  When I ran the command before, I used short name
> > > and didn't do it on the long name and just got back:
> > >
> > > # file /usr/lib/liblttng-ust.so
> > > /usr/lib/liblttng-ust.so: symbolic link to liblttng-ust.so.1.0.0
> > >
> > > ... and immediately looked at man page to try to figure out what
> > > switch showed BuildID etc., ha, ha.
> > >
> > > When I do it on long name here is what I see:
> > >
> > > # file /usr/lib/liblttng-ust-common.so.1.0.0
> > > /usr/lib/liblttng-ust-common.so.1.0.0: ELF 64-bit LSB shared object,
> > > ARM aarch64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=a50c9a77163b6b91e1f84e57d167c7b77ae707a3, with
> > > debug_info, not stripped
> > > # file /usr/lib/liblttng-ust-ctl.so.5.0.0
> > > /usr/lib/liblttng-ust-ctl.so.5.0.0: ELF 64-bit LSB shared object, ARM
> > > aarch64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=547cccac08721ed1c9a7f3c7ebf1de84ddba7fba, with
> > > debug_info, not stripped
> > > # file /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0
> > > /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0: ELF 64-bit LSB shared
> > > object, ARM aarch64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=ad5f71ef5e83ab9a972488976c47db265d3360b9, with
> > > debug_info, not stripped
> > > # file /usr/lib/liblttng-ust-cyg-profile.so.1.0.0
> > > /usr/lib/liblttng-ust-cyg-profile.so.1.0.0: ELF 64-bit LSB shared
> > > object, ARM aarch64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=c2c246c1973bd3241aa4f7229fcbcb27ebe08e82, with
> > > debug_info, not stripped
> > > # file /usr/lib/liblttng-ust-dl.so.1.0.0
> > > /usr/lib/liblttng-ust-dl.so.1.0.0: ELF 64-bit LSB shared object, ARM
> > > aarch64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=6082ee88c9394319bc3adff16b0b3ea9f8d549ec, with
> > > debug_info, not stripped
> > > # file /usr/lib/liblttng-ust-fd.so.1.0.0
> > > /usr/lib/liblttng-ust-fd.so.1.0.0: ELF 64-bit LSB shared object, ARM
> > > aarch64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=a2813245af91abe98615771dfd7d5f19b033a410, with
> > > debug_info, not stripped
> > > # file /usr/lib/liblttng-ust-fork.so.1.0.0
> > > /usr/lib/liblttng-ust-fork.so.1.0.0: ELF 64-bit LSB shared object, ARM
> > > aarch64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=86afa53808502873830c02290f477d4ff8013afb, with
> > > debug_info, not stripped
> > > # file /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0
> > > /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0: ELF 64-bit LSB shared
> > > object, ARM aarch64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=52f391875a378b5f2c46747a58020b86cb7c9a83, with
> > > debug_info, not stripped
> > > # file /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0
> > > /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0: ELF 64-bit LSB shared
> > > object, ARM aarch64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=7127223a7e8ed67c6697b95ae1f8ac107df7e47e, with
> > > debug_info, not stripped
> > > # file /usr/lib/liblttng-ust-tracepoint.so.1.0.0
> > > /usr/lib/liblttng-ust-tracepoint.so.1.0.0: ELF 64-bit LSB shared
> > > object, ARM aarch64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=5971b4d84ec1efe61c6d47c38e92de20569f0f49, with
> > > debug_info, not stripped
> > > # file /usr/lib/liblttng-ust.so.1.0.0
> > > /usr/lib/liblttng-ust.so.1.0.0: ELF 64-bit LSB shared object, ARM
> > > aarch64, version 1 (SYSV), dynamically linked,
> > > BuildID[sha1]=ce7097ae9bbf42a02dccd386fdfbd37e3224858b, with
> > > debug_info, not stripped
> > >
> > >
> > > cutting some stuff out cause it's getting long again.
> > >
> > > >
> > > > Sounds like `make check` for lttng-tools passed then?
> > >
> > > At first no.  But I think this is because I built the new lttng-tools
> > > in my on target native environment and ran make check and forgot to do
> > > make install first, so it was using the older version of lttng-tools.
> > >
> > > So then I ran make install of lttng-tools and even did a make clean
> > > and rebuild of lttng-ust and re-installed it and ran make check on
> > > both (and things looked a lot better) ... that's where those warnings
> > > etc., I asked you about came from.
> > >
> > > >
> > > > My understanding at this point is the unit tests are passing for
> > > > LTTng-UST on your system, as are the unit and regression tests for
> > > > LTTng-tools. The example programs shipped with LTTng-UST work on your
> > > > system, as does the example from the documentation. The statedump
> > > > tracepoints loaded from LTTng-UST are also working fine, as evinced by
> > > > the program logs and the LTTng trace you shared.
> > > >
> > > > Despite my confusion about how exactly you're using the `hello world`
> > > > tracepoint in your application (as you've now described several
> > > > variations), the direction this points to for me are details related to
> > > > how you're using LTTng-UST and/or how are your building and linking your
> > > > application. To be clear, I don't mean to say that there is or is not an
> > > > issue in LTTng-UST, but to point at where to examine next in detail
> > > > including analysis of the produced object files.
> > >
> > > I compared the doc/examples/hello-static-lib to what I picked out of
> > > the LTTng documentation on the web site "Quick start" section and the
> > > traceprovider headder file is including stddef.h and mine isn't and
> > > the doc/examples/hello-static-lib/hello.c code is doing a sighandler
> > > and mine isn't doing any of that either.  I think I've probably posted
> > > it before but will do it again.  Here is what I'm calling my "hello".
> > > It's from the lttng documentation but I cut it down even further just
> > > to make sure I didn't fat finger something.  Like I said before, the
> > > full hello example from the documentation works.  But when I call
> > > pretty much the same code from my app it seg faults.
> > >
> > > I don't know if the differences I see between my "hello" and the
> > > "hello-static-lib" matter.
> > >
> > > hello-tp.h:
> > >
> > > #undef LTTNG_UST_TRACEPOINT_PROVIDER
> > > #define LTTNG_UST_TRACEPOINT_PROVIDER hello_world
> > >
> > > #undef LTTNG_UST_TRACEPOINT_INCLUDE
> > > #define LTTNG_UST_TRACEPOINT_INCLUDE "./hello-tp.h"
> > >
> > > #if !defined(_HELLO_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
> > > #define _HELLO_TP_H
> > >
> > > #include <lttng/tracepoint.h>
> > >
> > > LTTNG_UST_TRACEPOINT_EVENT(
> > >    hello_world,
> > >    my_first_tracepoint,
> > >    LTTNG_UST_TP_ARGS(
> > >        int, my_integer_arg
> > >    ),
> > >    LTTNG_UST_TP_FIELDS(
> > >        lttng_ust_field_integer(int, my_integer_field, my_integer_arg)
> > >    )
> > > )
> > >
> > > #endif /* _HELLO_TP_H */
> > >
> > > #include <lttng/tracepoint-event.h>
> > >
> > > hello-tp.c:
> > >
> > > #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
> > >
> > > #include "hello-tp.h"
> > >
> > > From my_app:
> > >
> > > #define LTTNG_UST_TRACEPOINT_DEFINE
> > > //#define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
> > > #include "hello-tp.h"
> > >
> > > .
> > > .
> > > .
> > > lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
> > >
> > > In the above case the tpp is static but I've tried to make it a shared
> > > object too (thus the commented out DYNAMIC_LINKAGE above) but get the
> > > same result.
> > >
> > > Again, I think the issue is probably g++ and weak/hidden symbol
> > > related and or TLS but that's based on the totality of what I've
> > > experienced over the past year or so and seeing the
> > > experiences/problems of others in the lttng-dev archives.
> > >
> > > Regards,
> > >
> > > Brian
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > You automatically follow any topics you start or reply to.
> > View/Reply Online (#63605): https://lists.yoctoproject.org/g/yocto/message/63605
> > Mute This Topic: https://lists.yoctoproject.org/mt/107264247/1997914
> > Group Owner: yocto+owner@lists.yoctoproject.org
> > Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [raj.khem@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Kienan Stewart via lttng-dev]

Hi Brian,

On 7/30/24 11:54 AM, Brian Hutchinson wrote:
> Hi Kienan,
> 
> I noticed looking thru the 38k line LTTNG_UST_DEBUG session this line:
> 
> "       810:     /usr/lib/liblttng-ust.so.1: error: symbol lookup
> error: undefined symbol: ltt_probe_register (fatal)"
>  > ... and it jogged my memory that I did see some abi conflict messages
> in lttng-ust "make check" that I don't know if they are "good" or
> "bad" but could be related???
> 

This is how lttng-ust.so.1 checks for lttng-ust.so.0 in the same 
process. It's normal.

C.f. 
https://github.com/lttng/lttng-ust/blob/5db855839d4526cb2b80c45096884b7f6136da9f/src/lib/lttng-ust/lttng-ust-comm.c#L2221

> Anyway, attaching a tar of lttng-ust and lttng-tools "make check" for
> your enjoyment.
> 
> Regards,
> 
> Brian
> 
> 
> 
> 
> On Tue, Jul 30, 2024 at 8:40 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>>
>> On Mon, Jul 29, 2024 at 3:03 PM Kienan Stewart <kstewart@efficios.com> wrote:
>>>
>>> Hi Brian,
>>>
>>> On 7/25/24 3:54 PM, Brian Hutchinson wrote:
>>>> Hi Kienan,
>>>>
>>>> I'll answer your questions below, but I've got questions on what I saw
>>>> building and installing lttng-tools (2.13.13) and lttng-ust (2.13.8).
>>>>
>>>> Based on the struggles I've had trying to get lttng to work with my
>>>> app over various Yocto versions (Dunfell & Kirkstone) and lttng
>>>> version, I think the problems I'm facing are mostly around C++ and
>>>> weak and hidden symbols in Yocto toolchain.
>>>>
>>>> When I started my app with the options you mentioned previously a
>>>> while back, Id see things like:
>>>>
>>>> # LTTNG_UST_DEBUG=1 LTTNG_UST_REGISTER_TIMEOUT=-1 /opt/tc/TrafficController
>>>> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
>>>> with hidden visibility for integer objects as SAME address between
>>>> compile units part of the same module. (in check_weak_hidden() at
>>>> tracepoint.c:1012)
>>>> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
>>>> with hidden visibility for pointer objects as SAME address between
>>>> compile units part of the same module. (in check_weak_hidden() at
>>>> tracepoint.c:1016)
>>>> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
>>>> with hidden visibility for 24-byte structure objects as SAME address
>>>> between compile units part of the same module. (in check_weak_hidden()
>>>> at tracepoint.c:1020)
>>>>
>>>
>>> These messages are extra information for debugging and not indicative of
>>> a problem in of itself. C.f.
>>> https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/src/lib/lttng-ust-tracepoint/tracepoint.c#L1010
>>>
>>> There is a unit test related to this:
>>> https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/tests/unit/gcc-weak-hidden/main.c#L76
>>>
>>>
>>>> I further researched this whole 'weak symbol' and 'hidden visibility'
>>>> topic in the lttng-dev archives and it smells a lot like what I've
>>>> been seeing.  You should be able to mix both tracef and tracepoint
>>>> calls in souce code ... but I could not.  I could get a tracef call to
>>>> work but if I put a tracepoint call in the same code then nothing
>>>> would work.  This was with Dunfell 3.1.7 and earlier versions of
>>>> lttng.
>>>>
>>>> At one point I could get a tracepoint call to work but I'd have to let
>>>> our cmake build system build and link the tpp.c file and then turn
>>>> around and use gcc to recompile it and copy it to where all the
>>>> objects were to create the huge .a library the app was built against.
>>>> That's when I first learned there are issues with C++.  I think g++ is
>>>> used to build even .c files that aren't c++.
>>>>
>>>> Then if I tried to put a tracepoint in another sub project, none of
>>>> the tracepoints would work and I'd get empty traces.  This is a
>>>> symptom of the 'weak symbols with hidden visibility' issue ... and I
>>>> finally found others that were having same issue in the archives.  I
>>>> don't fully understand the issues here, although I do understand some
>>>> of what's going on ... I just don't know what to do about it.
>>>>
>>>
>>> You said initially said that you're using `lttng_ust_tracepoint` exactly
>>> as the hello world from the documentation; however, you have just
>>> described several attempts at doing different things. Which case are we
>>> trying to understand here?
>>
>> lttng_ust_tracepoint.  I only mentioned prior tests for context to
>> similar struggles from a year or more ago.
>>
>>
>>>
>>>> At this point I was being encouraged to keep upgrading to newer
>>>> versions of lttng.  Our app never changed, gcc & lttng etc., kept
>>>> changing.  Now with newer versions nothing runs, all I get is an
>>>> immediate segfault.  Again, I'm building just like I did before a year
>>>> or so ago with older versions of Yocto and lttng.  I say all of that
>>>> to give perspective and history of what I've seen and experienced.
>>>> Now this TLS thing has entered the picture too and so far I've only
>>>> changed lttng, I don't know if I should be applying patches to my gcc
>>>> for that issue.  Like I said, I'm currently using Yocto Kirkstone
>>>> 4.0.18 and 6.1.38 kernel.
>>>>
>>>> Now I'll move into the area of things I've seen building/installing
>>>> lttng-tools and lttng-ust natively on the target environment I've
>>>> setup where I can run 'make check' etc.  These are in the category of
>>>> "hey, is this ok, should I be worried about this":
>>>>
>>>> While building lttng-tools I see things like:
>>>>
>>>> *** Warning: Linking the executable userspace-probe-elf-binary against
>>>> the loadable module
>>>> *** libfoo.so is not portable!
>>>>
>>>
>>> The library is for a test program. My understanding is that the library
>>> is compiled that way to force a stripped shared object to be produced in
>>> order to validate that symbol lookups in libraries with no symtab
>>> function as expected by using the dynsym table.
>>>
>>> C.f.
>>> https://github.com/lttng/lttng-tools/commit/ef3dfe5d31c88fb548189a6441aaf8b2afc0bd4b
>>>
>>>> In file included from ../../../src/common/macros.h:15,
>>>>                   from ../../../include/lttng/health-internal.h:19,
>>>>                   from lttng-ctl-health.c:19:
>>>> In function 'lttng_strnlen',
>>>>      inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
>>>>      inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
>>>>      inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
>>>> ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
>>>> specified bound 4096 may exceed source size 37 [-Wstringop-overread]
>>>>     19 |         return strnlen(str, max);
>>>>        |                ^~~~~~~~~~~~~~~~~
>>>> lttng-ctl-health.c: At top level:
>>>> cc1: note: unrecognized command-line option
>>>> '-Wno-incomplete-setjmp-declaration' may have been intended to silence
>>>> earlier diagnostics
>>>
>>> This warning is addressed in
>>> https://github.com/lttng/lttng-tools/commit/b25a59916106e5055be516f61f183a48f459b0b3
>>>
>>>> ** Warning: Linking the shared library libbar.la against the loadable module
>>>> *** libzzz.so is not portable!
>>>>
>>>> *** Warning: Linking the shared library libfoo.la against the loadable module
>>>> *** libbar.so is not portable!
>>>>
>>>> While installing lttng-tools I see things like this:
>>>>
>>>> make[4]: Entering directory '/opt/lttng/lttng-tools-2.13.13/src/lib/lttng-ctl'
>>>>    CC       lttng-ctl.lo
>>>>    CC       snapshot.lo
>>>>    CC       lttng-ctl-health.lo
>>>> In file included from ../../../src/common/macros.h:15,
>>>>                   from ../../../include/lttng/health-internal.h:19,
>>>>                   from lttng-ctl-health.c:19:
>>>> In function 'lttng_strnlen',
>>>>      inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
>>>>      inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
>>>>      inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
>>>> ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
>>>> specified bound 4096 may exceed source size 37 [-Wstringop-overread]
>>>>     19 |         return strnlen(str, max);
>>>>        |                ^~~~~~~~~~~~~~~~~
>>>> lttng-ctl-health.c: At top level:
>>>> cc1: note: unrecognized command-line option
>>>> '-Wno-incomplete-setjmp-declaration' may have been intended to silence
>>>> earlier diagnostics
>>>>
>>>> Making install in trigger-condition-event-matches
>>>> make[2]: Entering directory
>>>> '/opt/lttng/lttng-tools-2.13.13/doc/examples/trigger-condition-event-matches'
>>>>    CC       instrumented-app.o
>>>>    CC       tracepoint-trigger-example.o
>>>>    AR       libtracepoint-trigger-example.a
>>>> ar: `u' modifier ignored since `D' is the default (see `U')
>>>>
>>>> While building lttng-ust I see things like:
>>>>
>>>> Making all in utils
>>>> make[2]: Entering directory
>>>> '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
>>>>    CC       tap.o
>>>>    AR       libtap.a
>>>> ar: `u' modifier ignored since `D' is the default (see `U')
>>>>
>>>
>>> While libtool now uses `cr` by default, automake still defines the
>>> default to `cru` which is what ends up getting used in the example.
>>> Since many distros have changed the configuration of ar such that 'D' is
>>> the default rather than the previous behaviour 'U', 'u' is redundant.
>>>
>>> The behaviour in automake has been changed in automake 1.16.90+.
>>>
>>> C.f.
>>> https://github.com/autotools-mirror/libtool/commit/418129bc63afc312701e84cb8afa5ca413df1ab5
>>>
>>> C.f.
>>> http://git.savannah.gnu.org/cgit/automake.git/commit/?id=8cdbdda5aec652c356fe6dbba96810202176ae75
>>>
>>>> *** Warning: Linking the shared library libzero.la against the
>>>> loadable module
>>>> *** libfakeust0.so is not portable!
>>>>    CCLD     app_noust_indirect_abi0
>>>>
>>>> *** Warning: Linking the executable app_noust_indirect_abi0 against
>>>> the loadable module
>>>> *** libzero.so is not portable!
>>>>    CC       app_noust_indirect_abi0_abi1-app_noust.o
>>>>    CC       libone.lo
>>>>    CCLD     libone.la
>>>>    CCLD     app_noust_indirect_abi0_abi1
>>>>
>>>> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
>>>> against the loadable module
>>>> *** libzero.so is not portable!
>>>>
>>>> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
>>>> against the loadable module
>>>> *** libone.so is not portable!
>>>>    CC       app_noust_indirect_abi1-app_noust.o
>>>>    CCLD     app_noust_indirect_abi1
>>>>
>>>> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
>>>> against the loadable module
>>>> *** libone.so is not portable!
>>>>    CC       app_noust_indirect_abi1-app_noust.o
>>>>    CCLD     app_noust_indirect_abi1
>>>>
>>>> *** Warning: Linking the executable app_noust_indirect_abi1 against
>>>> the loadable module
>>>> *** libone.so is not portable!
>>>>    CC       app_ust.o
>>>>    CC       tp.o
>>>>    CCLD     app_ust
>>>>    CC       app_ust_dlopen.o
>>>>    CCLD     app_ust_dlopen
>>>>    CC       app_ust_indirect_abi0-app_ust.o
>>>>    CC       app_ust_indirect_abi0-tp.o
>>>>    CCLD     app_ust_indirect_abi0
>>>>
>>>> *** Warning: Linking the executable app_ust_indirect_abi0 against the
>>>> loadable module
>>>> *** libzero.so is not portable!
>>>>    CC       app_ust_indirect_abi0_abi1-app_ust.o
>>>>    CC       app_ust_indirect_abi0_abi1-tp.o
>>>>    CCLD     app_ust_indirect_abi0_abi1
>>>>
>>>> *** Warning: Linking the executable app_ust_indirect_abi0_abi1 against
>>>> the loadable module
>>>> *** libzero.so is not portable!
>>>>
>>>> I don't know if these are ok or if I should be worried about any of that.
>>>>
>>>
>>> These are all for different tests.
>>>
>>>> ... now on to your questions below.
>>>>
>>>>
>>>>
>>>> On Wed, Jul 24, 2024 at 12:04 PM Kienan Stewart <kstewart@efficios.com> wrote:
>>>>>
>>>>> Hi Brian,
>>>>>
>>>>> On 7/22/24 6:00 PM, Brian Hutchinson wrote:
>>>>>> Hi Kienan,
>>>>>>
>>>>>> Took a while to gather your grocery list but I think I have most of it
>>>>>> below ;)
>>>>>
>>>>> thanks for all the extra info. Replies inline below, but I'll cut a lot
>>>>> of the long output for readability.
>>>>>
>>>>> tl;dr the environment continues to be weird, but my present suspicion is
>>>>> that something in either compilation, the linking of your app (eg. with
>>>>> ld when producing the executable), or some post linking stripping might
>>>>> be causing issues.
>>>>
>>>> I'm not aware of any stripping that's going on.  In fact everything is
>>>> being built with debug symbols at the moment and I even turned off
>>>> optimization ... even used the debug friendly -O flag to see if that
>>>> made a difference.
>>>>
>>>>>
>>>>> I will stop digging into further hypotheticals on my side as there is no
>>>>> reproducer for both the environment and the application. If you ever end
>>>>> up with a minimal reproducer that you can share, I'd be more than happy
>>>>> to examine it.
>>>>
>>>> I'm planning on trying to make a small reproducer I can share but not there yet.
>>>>
>>>
>>> Great! I appreciate that you're taking the time to do so.
>>>
>>>>>
>>>>>>
>>>>>> I may have not been clear.  Most of the application components are
>>>>>> statically linked but I think there are some that are built as shared
>>>>>> objects (.so's) so that's what I was referring to.  I know that
>>>>>> lttng-ust is dynamically linked ... I think the lttng-ust docs say this
>>>>>> is only option but also makes reference to the fact static linking was
>>>>>> once possible (in some versions of the documentation) but not supported
>>>>>> anymore (I probably have the docs memorized by now ha, ha ... I've
>>>>>> looked at many, many versions of them).
>>>>>>
>>>>>> Just for full disclosure my ldd looks like:
>>>>>>
>>>>>>           linux-vdso.so.1 (0x0000ffffab196000)
>>>>>>           libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
>>>>>>           liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1
>>>>>> (0x0000ffffa5750000)
>>>>>>           libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
>>>>>>           librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
>>>>>>           libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
>>>>>>           libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
>>>>>>           libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
>>>>>>           libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
>>>>>>           /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
>>>>>>           liblttng-ust-common.so.1 =>
>>>>>> /usr/local/lib/liblttng-ust-common.so.1 (0x0000ffffa50e0000)
>>>>>>           liblttng-ust-tracepoint.so.1 =>
>>>>>> /usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
>>>>>>           libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
>>>>>>           libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)
>>>>>>
>>>>>>
>>>>>
>>>>> I find it very suspicious that `liblttng-ust.so.1` is in `/usr/lib`,
>>>>> while the other lttng-ust libraries are being loaded from `/usr/local/lib`.
>>>>
>>>> So Yocto puts all of the lttng libs into /usr/lib.  When I sent the
>>>> previous info I was using lttng-tools and modules built by Yocto/OE
>>>> and I setup a native build environment on the target so I could run
>>>> 'make check' etc., and that's why there were things in /usr/local/lib
>>>> because that's where you guys want stuff to be.  So I actually left
>>>> the lttng-ust installables in /usr/local/build but also copied them to
>>>> /usr/lib to overwrite old Yocto versions there.
>>>>
>>>
>>> It's not so much that it's "where we want it to be". The documentation
>>> uses `/usr/local/lib` because `/usr/local` is meant for software
>>> installed by the sysadmin administrator, as is the case when building a
>>> custom version. `/usr/lib` should be used by packages shipped with the
>>> system.
>>>
>>> C.f. https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s09.html
>>>
>>> You're free to do as you see fit, but when you start mixing and matching
>>> libraries and some are put in /usr/lib by your system packages and some
>>> you move there manually I find it more difficult to follow what is going on.
>>>
>>>>>
>>>>> This information also matches the statedump and the LD_DEBUG info from
>>>>> later on.
>>>>>
>>>>> Could you verify some of the following information:
>>>>>
>>>>> 1. In your build root for lttng-ust, enumerate all the liblttng*so
>>>>> files. For each shared object, run `file $libname` and record the value
>>>>> of the BuildID hash.R5jow
>>>>
>>>> Sorry, I'm not following you here.  The only buildID hash I can think
>>>> of is with 'eu-unstrip -n' but that's on core files, not individual
>>>> libs.  And looking at the options I have for 'file' on my target, I
>>>> don't see anything that looks like what you are asking.
>>>
>>> Perhaps I wasn't clear, the command to run is really just `file`. As a
>>> fuller example:
>>>
>>> ```
>>> $ file ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0
>>> ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0: ELF 64-bit
>>> LSB shared object, x86-64, version 1 (SYSV), dynamically linked,
>>> BuildID[sha1]=b2b4a0fc449cf317e32c23e0bb57ea1ad702b702, with debug_info,
>>> not stripped
>>
>> Ok, feel stupid now.  When I ran the command before, I used short name
>> and didn't do it on the long name and just got back:
>>
>> # file /usr/lib/liblttng-ust.so
>> /usr/lib/liblttng-ust.so: symbolic link to liblttng-ust.so.1.0.0
>>
>> ... and immediately looked at man page to try to figure out what
>> switch showed BuildID etc., ha, ha.
>>
>> When I do it on long name here is what I see:
>>
>> # file /usr/lib/liblttng-ust-common.so.1.0.0
>> /usr/lib/liblttng-ust-common.so.1.0.0: ELF 64-bit LSB shared object,
>> ARM aarch64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=a50c9a77163b6b91e1f84e57d167c7b77ae707a3, with
>> debug_info, not stripped
>> # file /usr/lib/liblttng-ust-ctl.so.5.0.0
>> /usr/lib/liblttng-ust-ctl.so.5.0.0: ELF 64-bit LSB shared object, ARM
>> aarch64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=547cccac08721ed1c9a7f3c7ebf1de84ddba7fba, with
>> debug_info, not stripped
>> # file /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0
>> /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0: ELF 64-bit LSB shared
>> object, ARM aarch64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=ad5f71ef5e83ab9a972488976c47db265d3360b9, with
>> debug_info, not stripped
>> # file /usr/lib/liblttng-ust-cyg-profile.so.1.0.0
>> /usr/lib/liblttng-ust-cyg-profile.so.1.0.0: ELF 64-bit LSB shared
>> object, ARM aarch64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=c2c246c1973bd3241aa4f7229fcbcb27ebe08e82, with
>> debug_info, not stripped
>> # file /usr/lib/liblttng-ust-dl.so.1.0.0
>> /usr/lib/liblttng-ust-dl.so.1.0.0: ELF 64-bit LSB shared object, ARM
>> aarch64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=6082ee88c9394319bc3adff16b0b3ea9f8d549ec, with
>> debug_info, not stripped
>> # file /usr/lib/liblttng-ust-fd.so.1.0.0
>> /usr/lib/liblttng-ust-fd.so.1.0.0: ELF 64-bit LSB shared object, ARM
>> aarch64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=a2813245af91abe98615771dfd7d5f19b033a410, with
>> debug_info, not stripped
>> # file /usr/lib/liblttng-ust-fork.so.1.0.0
>> /usr/lib/liblttng-ust-fork.so.1.0.0: ELF 64-bit LSB shared object, ARM
>> aarch64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=86afa53808502873830c02290f477d4ff8013afb, with
>> debug_info, not stripped
>> # file /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0
>> /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0: ELF 64-bit LSB shared
>> object, ARM aarch64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=52f391875a378b5f2c46747a58020b86cb7c9a83, with
>> debug_info, not stripped
>> # file /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0
>> /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0: ELF 64-bit LSB shared
>> object, ARM aarch64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=7127223a7e8ed67c6697b95ae1f8ac107df7e47e, with
>> debug_info, not stripped
>> # file /usr/lib/liblttng-ust-tracepoint.so.1.0.0
>> /usr/lib/liblttng-ust-tracepoint.so.1.0.0: ELF 64-bit LSB shared
>> object, ARM aarch64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=5971b4d84ec1efe61c6d47c38e92de20569f0f49, with
>> debug_info, not stripped
>> # file /usr/lib/liblttng-ust.so.1.0.0
>> /usr/lib/liblttng-ust.so.1.0.0: ELF 64-bit LSB shared object, ARM
>> aarch64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=ce7097ae9bbf42a02dccd386fdfbd37e3224858b, with
>> debug_info, not stripped
>>
>>
>> cutting some stuff out cause it's getting long again.
>>
>>>
>>> Sounds like `make check` for lttng-tools passed then?
>>
>> At first no.  But I think this is because I built the new lttng-tools
>> in my on target native environment and ran make check and forgot to do
>> make install first, so it was using the older version of lttng-tools.
>>
>> So then I ran make install of lttng-tools and even did a make clean
>> and rebuild of lttng-ust and re-installed it and ran make check on
>> both (and things looked a lot better) ... that's where those warnings
>> etc., I asked you about came from.
>>
>>>
>>> My understanding at this point is the unit tests are passing for
>>> LTTng-UST on your system, as are the unit and regression tests for
>>> LTTng-tools. The example programs shipped with LTTng-UST work on your
>>> system, as does the example from the documentation. The statedump
>>> tracepoints loaded from LTTng-UST are also working fine, as evinced by
>>> the program logs and the LTTng trace you shared.
>>>
>>> Despite my confusion about how exactly you're using the `hello world`
>>> tracepoint in your application (as you've now described several
>>> variations), the direction this points to for me are details related to
>>> how you're using LTTng-UST and/or how are your building and linking your
>>> application. To be clear, I don't mean to say that there is or is not an
>>> issue in LTTng-UST, but to point at where to examine next in detail
>>> including analysis of the produced object files.
>>
>> I compared the doc/examples/hello-static-lib to what I picked out of
>> the LTTng documentation on the web site "Quick start" section and the
>> traceprovider headder file is including stddef.h and mine isn't and
>> the doc/examples/hello-static-lib/hello.c code is doing a sighandler
>> and mine isn't doing any of that either.  I think I've probably posted
>> it before but will do it again.  Here is what I'm calling my "hello".
>> It's from the lttng documentation but I cut it down even further just
>> to make sure I didn't fat finger something.  Like I said before, the
>> full hello example from the documentation works.  But when I call
>> pretty much the same code from my app it seg faults.
>>
>> I don't know if the differences I see between my "hello" and the
>> "hello-static-lib" matter.
>>
>> hello-tp.h:
>>
>> #undef LTTNG_UST_TRACEPOINT_PROVIDER
>> #define LTTNG_UST_TRACEPOINT_PROVIDER hello_world
>>
>> #undef LTTNG_UST_TRACEPOINT_INCLUDE
>> #define LTTNG_UST_TRACEPOINT_INCLUDE "./hello-tp.h"
>>
>> #if !defined(_HELLO_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
>> #define _HELLO_TP_H
>>
>> #include <lttng/tracepoint.h>
>>
>> LTTNG_UST_TRACEPOINT_EVENT(
>>     hello_world,
>>     my_first_tracepoint,
>>     LTTNG_UST_TP_ARGS(
>>         int, my_integer_arg
>>     ),
>>     LTTNG_UST_TP_FIELDS(
>>         lttng_ust_field_integer(int, my_integer_field, my_integer_arg)
>>     )
>> )
>>
>> #endif /* _HELLO_TP_H */
>>
>> #include <lttng/tracepoint-event.h>
>>
>> hello-tp.c:
>>
>> #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
>>
>> #include "hello-tp.h"
>>
>>  From my_app:
>>
>> #define LTTNG_UST_TRACEPOINT_DEFINE
>> //#define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
>> #include "hello-tp.h"
>>
>> .
>> .
>> .
>> lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
>>
>> In the above case the tpp is static but I've tried to make it a shared
>> object too (thus the commented out DYNAMIC_LINKAGE above) but get the
>> same result.
>>
>> Again, I think the issue is probably g++ and weak/hidden symbol
>> related and or TLS but that's based on the totality of what I've
>> experienced over the past year or so and seeing the
>> experiences/problems of others in the lttng-dev archives.
>>
>> Regards,
>>
>> Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev]

Update.

Finally figured out how to make a demonstrator so you all could
reproduce the segfault problem.

After much going in circles, a colleague found the issue.  Simply add
-fpack-struct=1 to hello example.

gcc -fpack-struct=1 -c -I. hello-tp.c
gcc -fpack-struct=1 -c hello.c
gcc -o hello hello.o hello-tp.o -llttng-ust -ldl

I know ... you're gonna say don't do that, but this is embedded and
millions of lines of code so it is what it is.

Regards,

Brian

On Tue, Jul 30, 2024 at 11:58 AM Kienan Stewart <kstewart@efficios.com> wrote:
>
> Hi Brian,
>
> On 7/30/24 11:54 AM, Brian Hutchinson wrote:
> > Hi Kienan,
> >
> > I noticed looking thru the 38k line LTTNG_UST_DEBUG session this line:
> >
> > "       810:     /usr/lib/liblttng-ust.so.1: error: symbol lookup
> > error: undefined symbol: ltt_probe_register (fatal)"
> >  > ... and it jogged my memory that I did see some abi conflict messages
> > in lttng-ust "make check" that I don't know if they are "good" or
> > "bad" but could be related???
> >
>
> This is how lttng-ust.so.1 checks for lttng-ust.so.0 in the same
> process. It's normal.
>
> C.f.
> https://github.com/lttng/lttng-ust/blob/5db855839d4526cb2b80c45096884b7f6136da9f/src/lib/lttng-ust/lttng-ust-comm.c#L2221
>
> > Anyway, attaching a tar of lttng-ust and lttng-tools "make check" for
> > your enjoyment.
> >
> > Regards,
> >
> > Brian
> >
> >
> >
> >
> > On Tue, Jul 30, 2024 at 8:40 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
> >>
> >> On Mon, Jul 29, 2024 at 3:03 PM Kienan Stewart <kstewart@efficios.com> wrote:
> >>>
> >>> Hi Brian,
> >>>
> >>> On 7/25/24 3:54 PM, Brian Hutchinson wrote:
> >>>> Hi Kienan,
> >>>>
> >>>> I'll answer your questions below, but I've got questions on what I saw
> >>>> building and installing lttng-tools (2.13.13) and lttng-ust (2.13.8).
> >>>>
> >>>> Based on the struggles I've had trying to get lttng to work with my
> >>>> app over various Yocto versions (Dunfell & Kirkstone) and lttng
> >>>> version, I think the problems I'm facing are mostly around C++ and
> >>>> weak and hidden symbols in Yocto toolchain.
> >>>>
> >>>> When I started my app with the options you mentioned previously a
> >>>> while back, Id see things like:
> >>>>
> >>>> # LTTNG_UST_DEBUG=1 LTTNG_UST_REGISTER_TIMEOUT=-1 /opt/tc/TrafficController
> >>>> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> >>>> with hidden visibility for integer objects as SAME address between
> >>>> compile units part of the same module. (in check_weak_hidden() at
> >>>> tracepoint.c:1012)
> >>>> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> >>>> with hidden visibility for pointer objects as SAME address between
> >>>> compile units part of the same module. (in check_weak_hidden() at
> >>>> tracepoint.c:1016)
> >>>> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
> >>>> with hidden visibility for 24-byte structure objects as SAME address
> >>>> between compile units part of the same module. (in check_weak_hidden()
> >>>> at tracepoint.c:1020)
> >>>>
> >>>
> >>> These messages are extra information for debugging and not indicative of
> >>> a problem in of itself. C.f.
> >>> https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/src/lib/lttng-ust-tracepoint/tracepoint.c#L1010
> >>>
> >>> There is a unit test related to this:
> >>> https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/tests/unit/gcc-weak-hidden/main.c#L76
> >>>
> >>>
> >>>> I further researched this whole 'weak symbol' and 'hidden visibility'
> >>>> topic in the lttng-dev archives and it smells a lot like what I've
> >>>> been seeing.  You should be able to mix both tracef and tracepoint
> >>>> calls in souce code ... but I could not.  I could get a tracef call to
> >>>> work but if I put a tracepoint call in the same code then nothing
> >>>> would work.  This was with Dunfell 3.1.7 and earlier versions of
> >>>> lttng.
> >>>>
> >>>> At one point I could get a tracepoint call to work but I'd have to let
> >>>> our cmake build system build and link the tpp.c file and then turn
> >>>> around and use gcc to recompile it and copy it to where all the
> >>>> objects were to create the huge .a library the app was built against.
> >>>> That's when I first learned there are issues with C++.  I think g++ is
> >>>> used to build even .c files that aren't c++.
> >>>>
> >>>> Then if I tried to put a tracepoint in another sub project, none of
> >>>> the tracepoints would work and I'd get empty traces.  This is a
> >>>> symptom of the 'weak symbols with hidden visibility' issue ... and I
> >>>> finally found others that were having same issue in the archives.  I
> >>>> don't fully understand the issues here, although I do understand some
> >>>> of what's going on ... I just don't know what to do about it.
> >>>>
> >>>
> >>> You said initially said that you're using `lttng_ust_tracepoint` exactly
> >>> as the hello world from the documentation; however, you have just
> >>> described several attempts at doing different things. Which case are we
> >>> trying to understand here?
> >>
> >> lttng_ust_tracepoint.  I only mentioned prior tests for context to
> >> similar struggles from a year or more ago.
> >>
> >>
> >>>
> >>>> At this point I was being encouraged to keep upgrading to newer
> >>>> versions of lttng.  Our app never changed, gcc & lttng etc., kept
> >>>> changing.  Now with newer versions nothing runs, all I get is an
> >>>> immediate segfault.  Again, I'm building just like I did before a year
> >>>> or so ago with older versions of Yocto and lttng.  I say all of that
> >>>> to give perspective and history of what I've seen and experienced.
> >>>> Now this TLS thing has entered the picture too and so far I've only
> >>>> changed lttng, I don't know if I should be applying patches to my gcc
> >>>> for that issue.  Like I said, I'm currently using Yocto Kirkstone
> >>>> 4.0.18 and 6.1.38 kernel.
> >>>>
> >>>> Now I'll move into the area of things I've seen building/installing
> >>>> lttng-tools and lttng-ust natively on the target environment I've
> >>>> setup where I can run 'make check' etc.  These are in the category of
> >>>> "hey, is this ok, should I be worried about this":
> >>>>
> >>>> While building lttng-tools I see things like:
> >>>>
> >>>> *** Warning: Linking the executable userspace-probe-elf-binary against
> >>>> the loadable module
> >>>> *** libfoo.so is not portable!
> >>>>
> >>>
> >>> The library is for a test program. My understanding is that the library
> >>> is compiled that way to force a stripped shared object to be produced in
> >>> order to validate that symbol lookups in libraries with no symtab
> >>> function as expected by using the dynsym table.
> >>>
> >>> C.f.
> >>> https://github.com/lttng/lttng-tools/commit/ef3dfe5d31c88fb548189a6441aaf8b2afc0bd4b
> >>>
> >>>> In file included from ../../../src/common/macros.h:15,
> >>>>                   from ../../../include/lttng/health-internal.h:19,
> >>>>                   from lttng-ctl-health.c:19:
> >>>> In function 'lttng_strnlen',
> >>>>      inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
> >>>>      inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
> >>>>      inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> >>>> ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> >>>> specified bound 4096 may exceed source size 37 [-Wstringop-overread]
> >>>>     19 |         return strnlen(str, max);
> >>>>        |                ^~~~~~~~~~~~~~~~~
> >>>> lttng-ctl-health.c: At top level:
> >>>> cc1: note: unrecognized command-line option
> >>>> '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> >>>> earlier diagnostics
> >>>
> >>> This warning is addressed in
> >>> https://github.com/lttng/lttng-tools/commit/b25a59916106e5055be516f61f183a48f459b0b3
> >>>
> >>>> ** Warning: Linking the shared library libbar.la against the loadable module
> >>>> *** libzzz.so is not portable!
> >>>>
> >>>> *** Warning: Linking the shared library libfoo.la against the loadable module
> >>>> *** libbar.so is not portable!
> >>>>
> >>>> While installing lttng-tools I see things like this:
> >>>>
> >>>> make[4]: Entering directory '/opt/lttng/lttng-tools-2.13.13/src/lib/lttng-ctl'
> >>>>    CC       lttng-ctl.lo
> >>>>    CC       snapshot.lo
> >>>>    CC       lttng-ctl-health.lo
> >>>> In file included from ../../../src/common/macros.h:15,
> >>>>                   from ../../../include/lttng/health-internal.h:19,
> >>>>                   from lttng-ctl-health.c:19:
> >>>> In function 'lttng_strnlen',
> >>>>      inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
> >>>>      inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
> >>>>      inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
> >>>> ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
> >>>> specified bound 4096 may exceed source size 37 [-Wstringop-overread]
> >>>>     19 |         return strnlen(str, max);
> >>>>        |                ^~~~~~~~~~~~~~~~~
> >>>> lttng-ctl-health.c: At top level:
> >>>> cc1: note: unrecognized command-line option
> >>>> '-Wno-incomplete-setjmp-declaration' may have been intended to silence
> >>>> earlier diagnostics
> >>>>
> >>>> Making install in trigger-condition-event-matches
> >>>> make[2]: Entering directory
> >>>> '/opt/lttng/lttng-tools-2.13.13/doc/examples/trigger-condition-event-matches'
> >>>>    CC       instrumented-app.o
> >>>>    CC       tracepoint-trigger-example.o
> >>>>    AR       libtracepoint-trigger-example.a
> >>>> ar: `u' modifier ignored since `D' is the default (see `U')
> >>>>
> >>>> While building lttng-ust I see things like:
> >>>>
> >>>> Making all in utils
> >>>> make[2]: Entering directory
> >>>> '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
> >>>>    CC       tap.o
> >>>>    AR       libtap.a
> >>>> ar: `u' modifier ignored since `D' is the default (see `U')
> >>>>
> >>>
> >>> While libtool now uses `cr` by default, automake still defines the
> >>> default to `cru` which is what ends up getting used in the example.
> >>> Since many distros have changed the configuration of ar such that 'D' is
> >>> the default rather than the previous behaviour 'U', 'u' is redundant.
> >>>
> >>> The behaviour in automake has been changed in automake 1.16.90+.
> >>>
> >>> C.f.
> >>> https://github.com/autotools-mirror/libtool/commit/418129bc63afc312701e84cb8afa5ca413df1ab5
> >>>
> >>> C.f.
> >>> http://git.savannah.gnu.org/cgit/automake.git/commit/?id=8cdbdda5aec652c356fe6dbba96810202176ae75
> >>>
> >>>> *** Warning: Linking the shared library libzero.la against the
> >>>> loadable module
> >>>> *** libfakeust0.so is not portable!
> >>>>    CCLD     app_noust_indirect_abi0
> >>>>
> >>>> *** Warning: Linking the executable app_noust_indirect_abi0 against
> >>>> the loadable module
> >>>> *** libzero.so is not portable!
> >>>>    CC       app_noust_indirect_abi0_abi1-app_noust.o
> >>>>    CC       libone.lo
> >>>>    CCLD     libone.la
> >>>>    CCLD     app_noust_indirect_abi0_abi1
> >>>>
> >>>> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> >>>> against the loadable module
> >>>> *** libzero.so is not portable!
> >>>>
> >>>> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> >>>> against the loadable module
> >>>> *** libone.so is not portable!
> >>>>    CC       app_noust_indirect_abi1-app_noust.o
> >>>>    CCLD     app_noust_indirect_abi1
> >>>>
> >>>> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
> >>>> against the loadable module
> >>>> *** libone.so is not portable!
> >>>>    CC       app_noust_indirect_abi1-app_noust.o
> >>>>    CCLD     app_noust_indirect_abi1
> >>>>
> >>>> *** Warning: Linking the executable app_noust_indirect_abi1 against
> >>>> the loadable module
> >>>> *** libone.so is not portable!
> >>>>    CC       app_ust.o
> >>>>    CC       tp.o
> >>>>    CCLD     app_ust
> >>>>    CC       app_ust_dlopen.o
> >>>>    CCLD     app_ust_dlopen
> >>>>    CC       app_ust_indirect_abi0-app_ust.o
> >>>>    CC       app_ust_indirect_abi0-tp.o
> >>>>    CCLD     app_ust_indirect_abi0
> >>>>
> >>>> *** Warning: Linking the executable app_ust_indirect_abi0 against the
> >>>> loadable module
> >>>> *** libzero.so is not portable!
> >>>>    CC       app_ust_indirect_abi0_abi1-app_ust.o
> >>>>    CC       app_ust_indirect_abi0_abi1-tp.o
> >>>>    CCLD     app_ust_indirect_abi0_abi1
> >>>>
> >>>> *** Warning: Linking the executable app_ust_indirect_abi0_abi1 against
> >>>> the loadable module
> >>>> *** libzero.so is not portable!
> >>>>
> >>>> I don't know if these are ok or if I should be worried about any of that.
> >>>>
> >>>
> >>> These are all for different tests.
> >>>
> >>>> ... now on to your questions below.
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Jul 24, 2024 at 12:04 PM Kienan Stewart <kstewart@efficios.com> wrote:
> >>>>>
> >>>>> Hi Brian,
> >>>>>
> >>>>> On 7/22/24 6:00 PM, Brian Hutchinson wrote:
> >>>>>> Hi Kienan,
> >>>>>>
> >>>>>> Took a while to gather your grocery list but I think I have most of it
> >>>>>> below ;)
> >>>>>
> >>>>> thanks for all the extra info. Replies inline below, but I'll cut a lot
> >>>>> of the long output for readability.
> >>>>>
> >>>>> tl;dr the environment continues to be weird, but my present suspicion is
> >>>>> that something in either compilation, the linking of your app (eg. with
> >>>>> ld when producing the executable), or some post linking stripping might
> >>>>> be causing issues.
> >>>>
> >>>> I'm not aware of any stripping that's going on.  In fact everything is
> >>>> being built with debug symbols at the moment and I even turned off
> >>>> optimization ... even used the debug friendly -O flag to see if that
> >>>> made a difference.
> >>>>
> >>>>>
> >>>>> I will stop digging into further hypotheticals on my side as there is no
> >>>>> reproducer for both the environment and the application. If you ever end
> >>>>> up with a minimal reproducer that you can share, I'd be more than happy
> >>>>> to examine it.
> >>>>
> >>>> I'm planning on trying to make a small reproducer I can share but not there yet.
> >>>>
> >>>
> >>> Great! I appreciate that you're taking the time to do so.
> >>>
> >>>>>
> >>>>>>
> >>>>>> I may have not been clear.  Most of the application components are
> >>>>>> statically linked but I think there are some that are built as shared
> >>>>>> objects (.so's) so that's what I was referring to.  I know that
> >>>>>> lttng-ust is dynamically linked ... I think the lttng-ust docs say this
> >>>>>> is only option but also makes reference to the fact static linking was
> >>>>>> once possible (in some versions of the documentation) but not supported
> >>>>>> anymore (I probably have the docs memorized by now ha, ha ... I've
> >>>>>> looked at many, many versions of them).
> >>>>>>
> >>>>>> Just for full disclosure my ldd looks like:
> >>>>>>
> >>>>>>           linux-vdso.so.1 (0x0000ffffab196000)
> >>>>>>           libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
> >>>>>>           liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1
> >>>>>> (0x0000ffffa5750000)
> >>>>>>           libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
> >>>>>>           librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
> >>>>>>           libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
> >>>>>>           libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
> >>>>>>           libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
> >>>>>>           libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
> >>>>>>           /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
> >>>>>>           liblttng-ust-common.so.1 =>
> >>>>>> /usr/local/lib/liblttng-ust-common.so.1 (0x0000ffffa50e0000)
> >>>>>>           liblttng-ust-tracepoint.so.1 =>
> >>>>>> /usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
> >>>>>>           libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
> >>>>>>           libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> I find it very suspicious that `liblttng-ust.so.1` is in `/usr/lib`,
> >>>>> while the other lttng-ust libraries are being loaded from `/usr/local/lib`.
> >>>>
> >>>> So Yocto puts all of the lttng libs into /usr/lib.  When I sent the
> >>>> previous info I was using lttng-tools and modules built by Yocto/OE
> >>>> and I setup a native build environment on the target so I could run
> >>>> 'make check' etc., and that's why there were things in /usr/local/lib
> >>>> because that's where you guys want stuff to be.  So I actually left
> >>>> the lttng-ust installables in /usr/local/build but also copied them to
> >>>> /usr/lib to overwrite old Yocto versions there.
> >>>>
> >>>
> >>> It's not so much that it's "where we want it to be". The documentation
> >>> uses `/usr/local/lib` because `/usr/local` is meant for software
> >>> installed by the sysadmin administrator, as is the case when building a
> >>> custom version. `/usr/lib` should be used by packages shipped with the
> >>> system.
> >>>
> >>> C.f. https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s09.html
> >>>
> >>> You're free to do as you see fit, but when you start mixing and matching
> >>> libraries and some are put in /usr/lib by your system packages and some
> >>> you move there manually I find it more difficult to follow what is going on.
> >>>
> >>>>>
> >>>>> This information also matches the statedump and the LD_DEBUG info from
> >>>>> later on.
> >>>>>
> >>>>> Could you verify some of the following information:
> >>>>>
> >>>>> 1. In your build root for lttng-ust, enumerate all the liblttng*so
> >>>>> files. For each shared object, run `file $libname` and record the value
> >>>>> of the BuildID hash.R5jow
> >>>>
> >>>> Sorry, I'm not following you here.  The only buildID hash I can think
> >>>> of is with 'eu-unstrip -n' but that's on core files, not individual
> >>>> libs.  And looking at the options I have for 'file' on my target, I
> >>>> don't see anything that looks like what you are asking.
> >>>
> >>> Perhaps I wasn't clear, the command to run is really just `file`. As a
> >>> fuller example:
> >>>
> >>> ```
> >>> $ file ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0
> >>> ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0: ELF 64-bit
> >>> LSB shared object, x86-64, version 1 (SYSV), dynamically linked,
> >>> BuildID[sha1]=b2b4a0fc449cf317e32c23e0bb57ea1ad702b702, with debug_info,
> >>> not stripped
> >>
> >> Ok, feel stupid now.  When I ran the command before, I used short name
> >> and didn't do it on the long name and just got back:
> >>
> >> # file /usr/lib/liblttng-ust.so
> >> /usr/lib/liblttng-ust.so: symbolic link to liblttng-ust.so.1.0.0
> >>
> >> ... and immediately looked at man page to try to figure out what
> >> switch showed BuildID etc., ha, ha.
> >>
> >> When I do it on long name here is what I see:
> >>
> >> # file /usr/lib/liblttng-ust-common.so.1.0.0
> >> /usr/lib/liblttng-ust-common.so.1.0.0: ELF 64-bit LSB shared object,
> >> ARM aarch64, version 1 (SYSV), dynamically linked,
> >> BuildID[sha1]=a50c9a77163b6b91e1f84e57d167c7b77ae707a3, with
> >> debug_info, not stripped
> >> # file /usr/lib/liblttng-ust-ctl.so.5.0.0
> >> /usr/lib/liblttng-ust-ctl.so.5.0.0: ELF 64-bit LSB shared object, ARM
> >> aarch64, version 1 (SYSV), dynamically linked,
> >> BuildID[sha1]=547cccac08721ed1c9a7f3c7ebf1de84ddba7fba, with
> >> debug_info, not stripped
> >> # file /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0
> >> /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0: ELF 64-bit LSB shared
> >> object, ARM aarch64, version 1 (SYSV), dynamically linked,
> >> BuildID[sha1]=ad5f71ef5e83ab9a972488976c47db265d3360b9, with
> >> debug_info, not stripped
> >> # file /usr/lib/liblttng-ust-cyg-profile.so.1.0.0
> >> /usr/lib/liblttng-ust-cyg-profile.so.1.0.0: ELF 64-bit LSB shared
> >> object, ARM aarch64, version 1 (SYSV), dynamically linked,
> >> BuildID[sha1]=c2c246c1973bd3241aa4f7229fcbcb27ebe08e82, with
> >> debug_info, not stripped
> >> # file /usr/lib/liblttng-ust-dl.so.1.0.0
> >> /usr/lib/liblttng-ust-dl.so.1.0.0: ELF 64-bit LSB shared object, ARM
> >> aarch64, version 1 (SYSV), dynamically linked,
> >> BuildID[sha1]=6082ee88c9394319bc3adff16b0b3ea9f8d549ec, with
> >> debug_info, not stripped
> >> # file /usr/lib/liblttng-ust-fd.so.1.0.0
> >> /usr/lib/liblttng-ust-fd.so.1.0.0: ELF 64-bit LSB shared object, ARM
> >> aarch64, version 1 (SYSV), dynamically linked,
> >> BuildID[sha1]=a2813245af91abe98615771dfd7d5f19b033a410, with
> >> debug_info, not stripped
> >> # file /usr/lib/liblttng-ust-fork.so.1.0.0
> >> /usr/lib/liblttng-ust-fork.so.1.0.0: ELF 64-bit LSB shared object, ARM
> >> aarch64, version 1 (SYSV), dynamically linked,
> >> BuildID[sha1]=86afa53808502873830c02290f477d4ff8013afb, with
> >> debug_info, not stripped
> >> # file /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0
> >> /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0: ELF 64-bit LSB shared
> >> object, ARM aarch64, version 1 (SYSV), dynamically linked,
> >> BuildID[sha1]=52f391875a378b5f2c46747a58020b86cb7c9a83, with
> >> debug_info, not stripped
> >> # file /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0
> >> /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0: ELF 64-bit LSB shared
> >> object, ARM aarch64, version 1 (SYSV), dynamically linked,
> >> BuildID[sha1]=7127223a7e8ed67c6697b95ae1f8ac107df7e47e, with
> >> debug_info, not stripped
> >> # file /usr/lib/liblttng-ust-tracepoint.so.1.0.0
> >> /usr/lib/liblttng-ust-tracepoint.so.1.0.0: ELF 64-bit LSB shared
> >> object, ARM aarch64, version 1 (SYSV), dynamically linked,
> >> BuildID[sha1]=5971b4d84ec1efe61c6d47c38e92de20569f0f49, with
> >> debug_info, not stripped
> >> # file /usr/lib/liblttng-ust.so.1.0.0
> >> /usr/lib/liblttng-ust.so.1.0.0: ELF 64-bit LSB shared object, ARM
> >> aarch64, version 1 (SYSV), dynamically linked,
> >> BuildID[sha1]=ce7097ae9bbf42a02dccd386fdfbd37e3224858b, with
> >> debug_info, not stripped
> >>
> >>
> >> cutting some stuff out cause it's getting long again.
> >>
> >>>
> >>> Sounds like `make check` for lttng-tools passed then?
> >>
> >> At first no.  But I think this is because I built the new lttng-tools
> >> in my on target native environment and ran make check and forgot to do
> >> make install first, so it was using the older version of lttng-tools.
> >>
> >> So then I ran make install of lttng-tools and even did a make clean
> >> and rebuild of lttng-ust and re-installed it and ran make check on
> >> both (and things looked a lot better) ... that's where those warnings
> >> etc., I asked you about came from.
> >>
> >>>
> >>> My understanding at this point is the unit tests are passing for
> >>> LTTng-UST on your system, as are the unit and regression tests for
> >>> LTTng-tools. The example programs shipped with LTTng-UST work on your
> >>> system, as does the example from the documentation. The statedump
> >>> tracepoints loaded from LTTng-UST are also working fine, as evinced by
> >>> the program logs and the LTTng trace you shared.
> >>>
> >>> Despite my confusion about how exactly you're using the `hello world`
> >>> tracepoint in your application (as you've now described several
> >>> variations), the direction this points to for me are details related to
> >>> how you're using LTTng-UST and/or how are your building and linking your
> >>> application. To be clear, I don't mean to say that there is or is not an
> >>> issue in LTTng-UST, but to point at where to examine next in detail
> >>> including analysis of the produced object files.
> >>
> >> I compared the doc/examples/hello-static-lib to what I picked out of
> >> the LTTng documentation on the web site "Quick start" section and the
> >> traceprovider headder file is including stddef.h and mine isn't and
> >> the doc/examples/hello-static-lib/hello.c code is doing a sighandler
> >> and mine isn't doing any of that either.  I think I've probably posted
> >> it before but will do it again.  Here is what I'm calling my "hello".
> >> It's from the lttng documentation but I cut it down even further just
> >> to make sure I didn't fat finger something.  Like I said before, the
> >> full hello example from the documentation works.  But when I call
> >> pretty much the same code from my app it seg faults.
> >>
> >> I don't know if the differences I see between my "hello" and the
> >> "hello-static-lib" matter.
> >>
> >> hello-tp.h:
> >>
> >> #undef LTTNG_UST_TRACEPOINT_PROVIDER
> >> #define LTTNG_UST_TRACEPOINT_PROVIDER hello_world
> >>
> >> #undef LTTNG_UST_TRACEPOINT_INCLUDE
> >> #define LTTNG_UST_TRACEPOINT_INCLUDE "./hello-tp.h"
> >>
> >> #if !defined(_HELLO_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
> >> #define _HELLO_TP_H
> >>
> >> #include <lttng/tracepoint.h>
> >>
> >> LTTNG_UST_TRACEPOINT_EVENT(
> >>     hello_world,
> >>     my_first_tracepoint,
> >>     LTTNG_UST_TP_ARGS(
> >>         int, my_integer_arg
> >>     ),
> >>     LTTNG_UST_TP_FIELDS(
> >>         lttng_ust_field_integer(int, my_integer_field, my_integer_arg)
> >>     )
> >> )
> >>
> >> #endif /* _HELLO_TP_H */
> >>
> >> #include <lttng/tracepoint-event.h>
> >>
> >> hello-tp.c:
> >>
> >> #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
> >>
> >> #include "hello-tp.h"
> >>
> >>  From my_app:
> >>
> >> #define LTTNG_UST_TRACEPOINT_DEFINE
> >> //#define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
> >> #include "hello-tp.h"
> >>
> >> .
> >> .
> >> .
> >> lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
> >>
> >> In the above case the tpp is static but I've tried to make it a shared
> >> object too (thus the commented out DYNAMIC_LINKAGE above) but get the
> >> same result.
> >>
> >> Again, I think the issue is probably g++ and weak/hidden symbol
> >> related and or TLS but that's based on the totality of what I've
> >> experienced over the past year or so and seeing the
> >> experiences/problems of others in the lttng-dev archives.
> >>
> >> Regards,
> >>
> >> Brian
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Kienan Stewart via lttng-dev]

Hi Brian,

On 8/21/24 8:51 AM, Brian Hutchinson wrote:
> Update.
> 
> Finally figured out how to make a demonstrator so you all could
> reproduce the segfault problem.
> 
> After much going in circles, a colleague found the issue.  Simply add
> -fpack-struct=1 to hello example.
> 

I'm glad to hear you and your colleague found the issue that was causing 
your program to crash.

> gcc -fpack-struct=1 -c -I. hello-tp.c
> gcc -fpack-struct=1 -c hello.c
> gcc -o hello hello.o hello-tp.o -llttng-ust -ldl
> 
> I know ... you're gonna say don't do that, but this is embedded and
> millions of lines of code so it is what it is.
> 

 From https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html, " 
Warning: the -fpack-struct switch causes GCC to generate code that is 
not binary compatible with code generated without that switch. 
Additionally, it makes the code suboptimal. Use it to conform to a 
non-default application binary interface."

I think there are two different approaches to try:

   1) Change you build system to compile your tracepoint providers 
without setting `-fpack-struct=1` for those compilation units
   2) Re-compile lttng-ust with `-fpack-struct=1` as well. I'm not sure 
if other libraries or tools will also require re-compilation.

In the 2nd case I think you are on your own for any issues that may arise.

thanks,
kienan

> Regards,
> 
> Brian
> 
> On Tue, Jul 30, 2024 at 11:58 AM Kienan Stewart <kstewart@efficios.com> wrote:
>>
>> Hi Brian,
>>
>> On 7/30/24 11:54 AM, Brian Hutchinson wrote:
>>> Hi Kienan,
>>>
>>> I noticed looking thru the 38k line LTTNG_UST_DEBUG session this line:
>>>
>>> "       810:     /usr/lib/liblttng-ust.so.1: error: symbol lookup
>>> error: undefined symbol: ltt_probe_register (fatal)"
>>>   > ... and it jogged my memory that I did see some abi conflict messages
>>> in lttng-ust "make check" that I don't know if they are "good" or
>>> "bad" but could be related???
>>>
>>
>> This is how lttng-ust.so.1 checks for lttng-ust.so.0 in the same
>> process. It's normal.
>>
>> C.f.
>> https://github.com/lttng/lttng-ust/blob/5db855839d4526cb2b80c45096884b7f6136da9f/src/lib/lttng-ust/lttng-ust-comm.c#L2221
>>
>>> Anyway, attaching a tar of lttng-ust and lttng-tools "make check" for
>>> your enjoyment.
>>>
>>> Regards,
>>>
>>> Brian
>>>
>>>
>>>
>>>
>>> On Tue, Jul 30, 2024 at 8:40 AM Brian Hutchinson <b.hutchman@gmail.com> wrote:
>>>>
>>>> On Mon, Jul 29, 2024 at 3:03 PM Kienan Stewart <kstewart@efficios.com> wrote:
>>>>>
>>>>> Hi Brian,
>>>>>
>>>>> On 7/25/24 3:54 PM, Brian Hutchinson wrote:
>>>>>> Hi Kienan,
>>>>>>
>>>>>> I'll answer your questions below, but I've got questions on what I saw
>>>>>> building and installing lttng-tools (2.13.13) and lttng-ust (2.13.8).
>>>>>>
>>>>>> Based on the struggles I've had trying to get lttng to work with my
>>>>>> app over various Yocto versions (Dunfell & Kirkstone) and lttng
>>>>>> version, I think the problems I'm facing are mostly around C++ and
>>>>>> weak and hidden symbols in Yocto toolchain.
>>>>>>
>>>>>> When I started my app with the options you mentioned previously a
>>>>>> while back, Id see things like:
>>>>>>
>>>>>> # LTTNG_UST_DEBUG=1 LTTNG_UST_REGISTER_TIMEOUT=-1 /opt/tc/TrafficController
>>>>>> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
>>>>>> with hidden visibility for integer objects as SAME address between
>>>>>> compile units part of the same module. (in check_weak_hidden() at
>>>>>> tracepoint.c:1012)
>>>>>> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
>>>>>> with hidden visibility for pointer objects as SAME address between
>>>>>> compile units part of the same module. (in check_weak_hidden() at
>>>>>> tracepoint.c:1016)
>>>>>> liblttng_ust_tracepoint[4012/4012]: Your compiler treats weak symbols
>>>>>> with hidden visibility for 24-byte structure objects as SAME address
>>>>>> between compile units part of the same module. (in check_weak_hidden()
>>>>>> at tracepoint.c:1020)
>>>>>>
>>>>>
>>>>> These messages are extra information for debugging and not indicative of
>>>>> a problem in of itself. C.f.
>>>>> https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/src/lib/lttng-ust-tracepoint/tracepoint.c#L1010
>>>>>
>>>>> There is a unit test related to this:
>>>>> https://github.com/lttng/lttng-ust/blob/24f7193c9b918bf714a40e9fc908eeb4978ada1c/tests/unit/gcc-weak-hidden/main.c#L76
>>>>>
>>>>>
>>>>>> I further researched this whole 'weak symbol' and 'hidden visibility'
>>>>>> topic in the lttng-dev archives and it smells a lot like what I've
>>>>>> been seeing.  You should be able to mix both tracef and tracepoint
>>>>>> calls in souce code ... but I could not.  I could get a tracef call to
>>>>>> work but if I put a tracepoint call in the same code then nothing
>>>>>> would work.  This was with Dunfell 3.1.7 and earlier versions of
>>>>>> lttng.
>>>>>>
>>>>>> At one point I could get a tracepoint call to work but I'd have to let
>>>>>> our cmake build system build and link the tpp.c file and then turn
>>>>>> around and use gcc to recompile it and copy it to where all the
>>>>>> objects were to create the huge .a library the app was built against.
>>>>>> That's when I first learned there are issues with C++.  I think g++ is
>>>>>> used to build even .c files that aren't c++.
>>>>>>
>>>>>> Then if I tried to put a tracepoint in another sub project, none of
>>>>>> the tracepoints would work and I'd get empty traces.  This is a
>>>>>> symptom of the 'weak symbols with hidden visibility' issue ... and I
>>>>>> finally found others that were having same issue in the archives.  I
>>>>>> don't fully understand the issues here, although I do understand some
>>>>>> of what's going on ... I just don't know what to do about it.
>>>>>>
>>>>>
>>>>> You said initially said that you're using `lttng_ust_tracepoint` exactly
>>>>> as the hello world from the documentation; however, you have just
>>>>> described several attempts at doing different things. Which case are we
>>>>> trying to understand here?
>>>>
>>>> lttng_ust_tracepoint.  I only mentioned prior tests for context to
>>>> similar struggles from a year or more ago.
>>>>
>>>>
>>>>>
>>>>>> At this point I was being encouraged to keep upgrading to newer
>>>>>> versions of lttng.  Our app never changed, gcc & lttng etc., kept
>>>>>> changing.  Now with newer versions nothing runs, all I get is an
>>>>>> immediate segfault.  Again, I'm building just like I did before a year
>>>>>> or so ago with older versions of Yocto and lttng.  I say all of that
>>>>>> to give perspective and history of what I've seen and experienced.
>>>>>> Now this TLS thing has entered the picture too and so far I've only
>>>>>> changed lttng, I don't know if I should be applying patches to my gcc
>>>>>> for that issue.  Like I said, I'm currently using Yocto Kirkstone
>>>>>> 4.0.18 and 6.1.38 kernel.
>>>>>>
>>>>>> Now I'll move into the area of things I've seen building/installing
>>>>>> lttng-tools and lttng-ust natively on the target environment I've
>>>>>> setup where I can run 'make check' etc.  These are in the category of
>>>>>> "hey, is this ok, should I be worried about this":
>>>>>>
>>>>>> While building lttng-tools I see things like:
>>>>>>
>>>>>> *** Warning: Linking the executable userspace-probe-elf-binary against
>>>>>> the loadable module
>>>>>> *** libfoo.so is not portable!
>>>>>>
>>>>>
>>>>> The library is for a test program. My understanding is that the library
>>>>> is compiled that way to force a stripped shared object to be produced in
>>>>> order to validate that symbol lookups in libraries with no symtab
>>>>> function as expected by using the dynsym table.
>>>>>
>>>>> C.f.
>>>>> https://github.com/lttng/lttng-tools/commit/ef3dfe5d31c88fb548189a6441aaf8b2afc0bd4b
>>>>>
>>>>>> In file included from ../../../src/common/macros.h:15,
>>>>>>                    from ../../../include/lttng/health-internal.h:19,
>>>>>>                    from lttng-ctl-health.c:19:
>>>>>> In function 'lttng_strnlen',
>>>>>>       inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
>>>>>>       inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
>>>>>>       inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
>>>>>> ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
>>>>>> specified bound 4096 may exceed source size 37 [-Wstringop-overread]
>>>>>>      19 |         return strnlen(str, max);
>>>>>>         |                ^~~~~~~~~~~~~~~~~
>>>>>> lttng-ctl-health.c: At top level:
>>>>>> cc1: note: unrecognized command-line option
>>>>>> '-Wno-incomplete-setjmp-declaration' may have been intended to silence
>>>>>> earlier diagnostics
>>>>>
>>>>> This warning is addressed in
>>>>> https://github.com/lttng/lttng-tools/commit/b25a59916106e5055be516f61f183a48f459b0b3
>>>>>
>>>>>> ** Warning: Linking the shared library libbar.la against the loadable module
>>>>>> *** libzzz.so is not portable!
>>>>>>
>>>>>> *** Warning: Linking the shared library libfoo.la against the loadable module
>>>>>> *** libbar.so is not portable!
>>>>>>
>>>>>> While installing lttng-tools I see things like this:
>>>>>>
>>>>>> make[4]: Entering directory '/opt/lttng/lttng-tools-2.13.13/src/lib/lttng-ctl'
>>>>>>     CC       lttng-ctl.lo
>>>>>>     CC       snapshot.lo
>>>>>>     CC       lttng-ctl-health.lo
>>>>>> In file included from ../../../src/common/macros.h:15,
>>>>>>                    from ../../../include/lttng/health-internal.h:19,
>>>>>>                    from lttng-ctl-health.c:19:
>>>>>> In function 'lttng_strnlen',
>>>>>>       inlined from 'lttng_strncpy' at ../../../src/common/macros.h:128:6,
>>>>>>       inlined from 'set_health_socket_path' at lttng-ctl-health.c:146:9,
>>>>>>       inlined from 'lttng_health_query' at lttng-ctl-health.c:264:8:
>>>>>> ../../../src/common/compat/string.h:19:16: warning: 'strnlen'
>>>>>> specified bound 4096 may exceed source size 37 [-Wstringop-overread]
>>>>>>      19 |         return strnlen(str, max);
>>>>>>         |                ^~~~~~~~~~~~~~~~~
>>>>>> lttng-ctl-health.c: At top level:
>>>>>> cc1: note: unrecognized command-line option
>>>>>> '-Wno-incomplete-setjmp-declaration' may have been intended to silence
>>>>>> earlier diagnostics
>>>>>>
>>>>>> Making install in trigger-condition-event-matches
>>>>>> make[2]: Entering directory
>>>>>> '/opt/lttng/lttng-tools-2.13.13/doc/examples/trigger-condition-event-matches'
>>>>>>     CC       instrumented-app.o
>>>>>>     CC       tracepoint-trigger-example.o
>>>>>>     AR       libtracepoint-trigger-example.a
>>>>>> ar: `u' modifier ignored since `D' is the default (see `U')
>>>>>>
>>>>>> While building lttng-ust I see things like:
>>>>>>
>>>>>> Making all in utils
>>>>>> make[2]: Entering directory
>>>>>> '/home/iadmin/lttng-ust/lttng-ust-2.13.8/tests/utils'
>>>>>>     CC       tap.o
>>>>>>     AR       libtap.a
>>>>>> ar: `u' modifier ignored since `D' is the default (see `U')
>>>>>>
>>>>>
>>>>> While libtool now uses `cr` by default, automake still defines the
>>>>> default to `cru` which is what ends up getting used in the example.
>>>>> Since many distros have changed the configuration of ar such that 'D' is
>>>>> the default rather than the previous behaviour 'U', 'u' is redundant.
>>>>>
>>>>> The behaviour in automake has been changed in automake 1.16.90+.
>>>>>
>>>>> C.f.
>>>>> https://github.com/autotools-mirror/libtool/commit/418129bc63afc312701e84cb8afa5ca413df1ab5
>>>>>
>>>>> C.f.
>>>>> http://git.savannah.gnu.org/cgit/automake.git/commit/?id=8cdbdda5aec652c356fe6dbba96810202176ae75
>>>>>
>>>>>> *** Warning: Linking the shared library libzero.la against the
>>>>>> loadable module
>>>>>> *** libfakeust0.so is not portable!
>>>>>>     CCLD     app_noust_indirect_abi0
>>>>>>
>>>>>> *** Warning: Linking the executable app_noust_indirect_abi0 against
>>>>>> the loadable module
>>>>>> *** libzero.so is not portable!
>>>>>>     CC       app_noust_indirect_abi0_abi1-app_noust.o
>>>>>>     CC       libone.lo
>>>>>>     CCLD     libone.la
>>>>>>     CCLD     app_noust_indirect_abi0_abi1
>>>>>>
>>>>>> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
>>>>>> against the loadable module
>>>>>> *** libzero.so is not portable!
>>>>>>
>>>>>> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
>>>>>> against the loadable module
>>>>>> *** libone.so is not portable!
>>>>>>     CC       app_noust_indirect_abi1-app_noust.o
>>>>>>     CCLD     app_noust_indirect_abi1
>>>>>>
>>>>>> *** Warning: Linking the executable app_noust_indirect_abi0_abi1
>>>>>> against the loadable module
>>>>>> *** libone.so is not portable!
>>>>>>     CC       app_noust_indirect_abi1-app_noust.o
>>>>>>     CCLD     app_noust_indirect_abi1
>>>>>>
>>>>>> *** Warning: Linking the executable app_noust_indirect_abi1 against
>>>>>> the loadable module
>>>>>> *** libone.so is not portable!
>>>>>>     CC       app_ust.o
>>>>>>     CC       tp.o
>>>>>>     CCLD     app_ust
>>>>>>     CC       app_ust_dlopen.o
>>>>>>     CCLD     app_ust_dlopen
>>>>>>     CC       app_ust_indirect_abi0-app_ust.o
>>>>>>     CC       app_ust_indirect_abi0-tp.o
>>>>>>     CCLD     app_ust_indirect_abi0
>>>>>>
>>>>>> *** Warning: Linking the executable app_ust_indirect_abi0 against the
>>>>>> loadable module
>>>>>> *** libzero.so is not portable!
>>>>>>     CC       app_ust_indirect_abi0_abi1-app_ust.o
>>>>>>     CC       app_ust_indirect_abi0_abi1-tp.o
>>>>>>     CCLD     app_ust_indirect_abi0_abi1
>>>>>>
>>>>>> *** Warning: Linking the executable app_ust_indirect_abi0_abi1 against
>>>>>> the loadable module
>>>>>> *** libzero.so is not portable!
>>>>>>
>>>>>> I don't know if these are ok or if I should be worried about any of that.
>>>>>>
>>>>>
>>>>> These are all for different tests.
>>>>>
>>>>>> ... now on to your questions below.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 24, 2024 at 12:04 PM Kienan Stewart <kstewart@efficios.com> wrote:
>>>>>>>
>>>>>>> Hi Brian,
>>>>>>>
>>>>>>> On 7/22/24 6:00 PM, Brian Hutchinson wrote:
>>>>>>>> Hi Kienan,
>>>>>>>>
>>>>>>>> Took a while to gather your grocery list but I think I have most of it
>>>>>>>> below ;)
>>>>>>>
>>>>>>> thanks for all the extra info. Replies inline below, but I'll cut a lot
>>>>>>> of the long output for readability.
>>>>>>>
>>>>>>> tl;dr the environment continues to be weird, but my present suspicion is
>>>>>>> that something in either compilation, the linking of your app (eg. with
>>>>>>> ld when producing the executable), or some post linking stripping might
>>>>>>> be causing issues.
>>>>>>
>>>>>> I'm not aware of any stripping that's going on.  In fact everything is
>>>>>> being built with debug symbols at the moment and I even turned off
>>>>>> optimization ... even used the debug friendly -O flag to see if that
>>>>>> made a difference.
>>>>>>
>>>>>>>
>>>>>>> I will stop digging into further hypotheticals on my side as there is no
>>>>>>> reproducer for both the environment and the application. If you ever end
>>>>>>> up with a minimal reproducer that you can share, I'd be more than happy
>>>>>>> to examine it.
>>>>>>
>>>>>> I'm planning on trying to make a small reproducer I can share but not there yet.
>>>>>>
>>>>>
>>>>> Great! I appreciate that you're taking the time to do so.
>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I may have not been clear.  Most of the application components are
>>>>>>>> statically linked but I think there are some that are built as shared
>>>>>>>> objects (.so's) so that's what I was referring to.  I know that
>>>>>>>> lttng-ust is dynamically linked ... I think the lttng-ust docs say this
>>>>>>>> is only option but also makes reference to the fact static linking was
>>>>>>>> once possible (in some versions of the documentation) but not supported
>>>>>>>> anymore (I probably have the docs memorized by now ha, ha ... I've
>>>>>>>> looked at many, many versions of them).
>>>>>>>>
>>>>>>>> Just for full disclosure my ldd looks like:
>>>>>>>>
>>>>>>>>            linux-vdso.so.1 (0x0000ffffab196000)
>>>>>>>>            libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
>>>>>>>>            liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1
>>>>>>>> (0x0000ffffa5750000)
>>>>>>>>            libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
>>>>>>>>            librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
>>>>>>>>            libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
>>>>>>>>            libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
>>>>>>>>            libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
>>>>>>>>            libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
>>>>>>>>            /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
>>>>>>>>            liblttng-ust-common.so.1 =>
>>>>>>>> /usr/local/lib/liblttng-ust-common.so.1 (0x0000ffffa50e0000)
>>>>>>>>            liblttng-ust-tracepoint.so.1 =>
>>>>>>>> /usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
>>>>>>>>            libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
>>>>>>>>            libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> I find it very suspicious that `liblttng-ust.so.1` is in `/usr/lib`,
>>>>>>> while the other lttng-ust libraries are being loaded from `/usr/local/lib`.
>>>>>>
>>>>>> So Yocto puts all of the lttng libs into /usr/lib.  When I sent the
>>>>>> previous info I was using lttng-tools and modules built by Yocto/OE
>>>>>> and I setup a native build environment on the target so I could run
>>>>>> 'make check' etc., and that's why there were things in /usr/local/lib
>>>>>> because that's where you guys want stuff to be.  So I actually left
>>>>>> the lttng-ust installables in /usr/local/build but also copied them to
>>>>>> /usr/lib to overwrite old Yocto versions there.
>>>>>>
>>>>>
>>>>> It's not so much that it's "where we want it to be". The documentation
>>>>> uses `/usr/local/lib` because `/usr/local` is meant for software
>>>>> installed by the sysadmin administrator, as is the case when building a
>>>>> custom version. `/usr/lib` should be used by packages shipped with the
>>>>> system.
>>>>>
>>>>> C.f. https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s09.html
>>>>>
>>>>> You're free to do as you see fit, but when you start mixing and matching
>>>>> libraries and some are put in /usr/lib by your system packages and some
>>>>> you move there manually I find it more difficult to follow what is going on.
>>>>>
>>>>>>>
>>>>>>> This information also matches the statedump and the LD_DEBUG info from
>>>>>>> later on.
>>>>>>>
>>>>>>> Could you verify some of the following information:
>>>>>>>
>>>>>>> 1. In your build root for lttng-ust, enumerate all the liblttng*so
>>>>>>> files. For each shared object, run `file $libname` and record the value
>>>>>>> of the BuildID hash.R5jow
>>>>>>
>>>>>> Sorry, I'm not following you here.  The only buildID hash I can think
>>>>>> of is with 'eu-unstrip -n' but that's on core files, not individual
>>>>>> libs.  And looking at the options I have for 'file' on my target, I
>>>>>> don't see anything that looks like what you are asking.
>>>>>
>>>>> Perhaps I wasn't clear, the command to run is really just `file`. As a
>>>>> fuller example:
>>>>>
>>>>> ```
>>>>> $ file ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0
>>>>> ./src/lib/lttng-ust-fork/.libs/liblttng-ust-fork.so.1.0.0: ELF 64-bit
>>>>> LSB shared object, x86-64, version 1 (SYSV), dynamically linked,
>>>>> BuildID[sha1]=b2b4a0fc449cf317e32c23e0bb57ea1ad702b702, with debug_info,
>>>>> not stripped
>>>>
>>>> Ok, feel stupid now.  When I ran the command before, I used short name
>>>> and didn't do it on the long name and just got back:
>>>>
>>>> # file /usr/lib/liblttng-ust.so
>>>> /usr/lib/liblttng-ust.so: symbolic link to liblttng-ust.so.1.0.0
>>>>
>>>> ... and immediately looked at man page to try to figure out what
>>>> switch showed BuildID etc., ha, ha.
>>>>
>>>> When I do it on long name here is what I see:
>>>>
>>>> # file /usr/lib/liblttng-ust-common.so.1.0.0
>>>> /usr/lib/liblttng-ust-common.so.1.0.0: ELF 64-bit LSB shared object,
>>>> ARM aarch64, version 1 (SYSV), dynamically linked,
>>>> BuildID[sha1]=a50c9a77163b6b91e1f84e57d167c7b77ae707a3, with
>>>> debug_info, not stripped
>>>> # file /usr/lib/liblttng-ust-ctl.so.5.0.0
>>>> /usr/lib/liblttng-ust-ctl.so.5.0.0: ELF 64-bit LSB shared object, ARM
>>>> aarch64, version 1 (SYSV), dynamically linked,
>>>> BuildID[sha1]=547cccac08721ed1c9a7f3c7ebf1de84ddba7fba, with
>>>> debug_info, not stripped
>>>> # file /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0
>>>> /usr/lib/liblttng-ust-cyg-profile-fast.so.1.0.0: ELF 64-bit LSB shared
>>>> object, ARM aarch64, version 1 (SYSV), dynamically linked,
>>>> BuildID[sha1]=ad5f71ef5e83ab9a972488976c47db265d3360b9, with
>>>> debug_info, not stripped
>>>> # file /usr/lib/liblttng-ust-cyg-profile.so.1.0.0
>>>> /usr/lib/liblttng-ust-cyg-profile.so.1.0.0: ELF 64-bit LSB shared
>>>> object, ARM aarch64, version 1 (SYSV), dynamically linked,
>>>> BuildID[sha1]=c2c246c1973bd3241aa4f7229fcbcb27ebe08e82, with
>>>> debug_info, not stripped
>>>> # file /usr/lib/liblttng-ust-dl.so.1.0.0
>>>> /usr/lib/liblttng-ust-dl.so.1.0.0: ELF 64-bit LSB shared object, ARM
>>>> aarch64, version 1 (SYSV), dynamically linked,
>>>> BuildID[sha1]=6082ee88c9394319bc3adff16b0b3ea9f8d549ec, with
>>>> debug_info, not stripped
>>>> # file /usr/lib/liblttng-ust-fd.so.1.0.0
>>>> /usr/lib/liblttng-ust-fd.so.1.0.0: ELF 64-bit LSB shared object, ARM
>>>> aarch64, version 1 (SYSV), dynamically linked,
>>>> BuildID[sha1]=a2813245af91abe98615771dfd7d5f19b033a410, with
>>>> debug_info, not stripped
>>>> # file /usr/lib/liblttng-ust-fork.so.1.0.0
>>>> /usr/lib/liblttng-ust-fork.so.1.0.0: ELF 64-bit LSB shared object, ARM
>>>> aarch64, version 1 (SYSV), dynamically linked,
>>>> BuildID[sha1]=86afa53808502873830c02290f477d4ff8013afb, with
>>>> debug_info, not stripped
>>>> # file /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0
>>>> /usr/lib/liblttng-ust-libc-wrapper.so.1.0.0: ELF 64-bit LSB shared
>>>> object, ARM aarch64, version 1 (SYSV), dynamically linked,
>>>> BuildID[sha1]=52f391875a378b5f2c46747a58020b86cb7c9a83, with
>>>> debug_info, not stripped
>>>> # file /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0
>>>> /usr/lib/liblttng-ust-pthread-wrapper.so.1.0.0: ELF 64-bit LSB shared
>>>> object, ARM aarch64, version 1 (SYSV), dynamically linked,
>>>> BuildID[sha1]=7127223a7e8ed67c6697b95ae1f8ac107df7e47e, with
>>>> debug_info, not stripped
>>>> # file /usr/lib/liblttng-ust-tracepoint.so.1.0.0
>>>> /usr/lib/liblttng-ust-tracepoint.so.1.0.0: ELF 64-bit LSB shared
>>>> object, ARM aarch64, version 1 (SYSV), dynamically linked,
>>>> BuildID[sha1]=5971b4d84ec1efe61c6d47c38e92de20569f0f49, with
>>>> debug_info, not stripped
>>>> # file /usr/lib/liblttng-ust.so.1.0.0
>>>> /usr/lib/liblttng-ust.so.1.0.0: ELF 64-bit LSB shared object, ARM
>>>> aarch64, version 1 (SYSV), dynamically linked,
>>>> BuildID[sha1]=ce7097ae9bbf42a02dccd386fdfbd37e3224858b, with
>>>> debug_info, not stripped
>>>>
>>>>
>>>> cutting some stuff out cause it's getting long again.
>>>>
>>>>>
>>>>> Sounds like `make check` for lttng-tools passed then?
>>>>
>>>> At first no.  But I think this is because I built the new lttng-tools
>>>> in my on target native environment and ran make check and forgot to do
>>>> make install first, so it was using the older version of lttng-tools.
>>>>
>>>> So then I ran make install of lttng-tools and even did a make clean
>>>> and rebuild of lttng-ust and re-installed it and ran make check on
>>>> both (and things looked a lot better) ... that's where those warnings
>>>> etc., I asked you about came from.
>>>>
>>>>>
>>>>> My understanding at this point is the unit tests are passing for
>>>>> LTTng-UST on your system, as are the unit and regression tests for
>>>>> LTTng-tools. The example programs shipped with LTTng-UST work on your
>>>>> system, as does the example from the documentation. The statedump
>>>>> tracepoints loaded from LTTng-UST are also working fine, as evinced by
>>>>> the program logs and the LTTng trace you shared.
>>>>>
>>>>> Despite my confusion about how exactly you're using the `hello world`
>>>>> tracepoint in your application (as you've now described several
>>>>> variations), the direction this points to for me are details related to
>>>>> how you're using LTTng-UST and/or how are your building and linking your
>>>>> application. To be clear, I don't mean to say that there is or is not an
>>>>> issue in LTTng-UST, but to point at where to examine next in detail
>>>>> including analysis of the produced object files.
>>>>
>>>> I compared the doc/examples/hello-static-lib to what I picked out of
>>>> the LTTng documentation on the web site "Quick start" section and the
>>>> traceprovider headder file is including stddef.h and mine isn't and
>>>> the doc/examples/hello-static-lib/hello.c code is doing a sighandler
>>>> and mine isn't doing any of that either.  I think I've probably posted
>>>> it before but will do it again.  Here is what I'm calling my "hello".
>>>> It's from the lttng documentation but I cut it down even further just
>>>> to make sure I didn't fat finger something.  Like I said before, the
>>>> full hello example from the documentation works.  But when I call
>>>> pretty much the same code from my app it seg faults.
>>>>
>>>> I don't know if the differences I see between my "hello" and the
>>>> "hello-static-lib" matter.
>>>>
>>>> hello-tp.h:
>>>>
>>>> #undef LTTNG_UST_TRACEPOINT_PROVIDER
>>>> #define LTTNG_UST_TRACEPOINT_PROVIDER hello_world
>>>>
>>>> #undef LTTNG_UST_TRACEPOINT_INCLUDE
>>>> #define LTTNG_UST_TRACEPOINT_INCLUDE "./hello-tp.h"
>>>>
>>>> #if !defined(_HELLO_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
>>>> #define _HELLO_TP_H
>>>>
>>>> #include <lttng/tracepoint.h>
>>>>
>>>> LTTNG_UST_TRACEPOINT_EVENT(
>>>>      hello_world,
>>>>      my_first_tracepoint,
>>>>      LTTNG_UST_TP_ARGS(
>>>>          int, my_integer_arg
>>>>      ),
>>>>      LTTNG_UST_TP_FIELDS(
>>>>          lttng_ust_field_integer(int, my_integer_field, my_integer_arg)
>>>>      )
>>>> )
>>>>
>>>> #endif /* _HELLO_TP_H */
>>>>
>>>> #include <lttng/tracepoint-event.h>
>>>>
>>>> hello-tp.c:
>>>>
>>>> #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
>>>>
>>>> #include "hello-tp.h"
>>>>
>>>>   From my_app:
>>>>
>>>> #define LTTNG_UST_TRACEPOINT_DEFINE
>>>> //#define LTTNG_UST_TRACEPOINT_PROBE_DYNAMIC_LINKAGE
>>>> #include "hello-tp.h"
>>>>
>>>> .
>>>> .
>>>> .
>>>> lttng_ust_tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");
>>>>
>>>> In the above case the tpp is static but I've tried to make it a shared
>>>> object too (thus the commented out DYNAMIC_LINKAGE above) but get the
>>>> same result.
>>>>
>>>> Again, I think the issue is probably g++ and weak/hidden symbol
>>>> related and or TLS but that's based on the totality of what I've
>>>> experienced over the past year or so and seeing the
>>>> experiences/problems of others in the lttng-dev archives.
>>>>
>>>> Regards,
>>>>
>>>> Brian

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Using lttng-ust 2.13.6 from Yocto Kirkstone and getting weird segfault saying strlen_asimd.S can't be found.

[Brian Hutchinson via lttng-dev]

[-- Attachment #1: Type: text/plain, Size: 11119 bytes --]

Hi Kienan,

Just now answering you on the question I owed you regarding the
experiment you requested about resolved symbols.

See below.

On Wed, Jul 24, 2024 at 12:04 PM Kienan Stewart <kstewart@efficios.com> wrote:
>
> Hi Brian,
>
> On 7/22/24 6:00 PM, Brian Hutchinson wrote:
> > Hi Kienan,
> >
> > Took a while to gather your grocery list but I think I have most of it
> > below ;)
>
> thanks for all the extra info. Replies inline below, but I'll cut a lot
> of the long output for readability.
>
> tl;dr the environment continues to be weird, but my present suspicion is
> that something in either compilation, the linking of your app (eg. with
> ld when producing the executable), or some post linking stripping might
> be causing issues.
>
> I will stop digging into further hypotheticals on my side as there is no
> reproducer for both the environment and the application. If you ever end
> up with a minimal reproducer that you can share, I'd be more than happy
> to examine it.
>
> >
> > I may have not been clear.  Most of the application components are
> > statically linked but I think there are some that are built as shared
> > objects (.so's) so that's what I was referring to.  I know that
> > lttng-ust is dynamically linked ... I think the lttng-ust docs say this
> > is only option but also makes reference to the fact static linking was
> > once possible (in some versions of the documentation) but not supported
> > anymore (I probably have the docs memorized by now ha, ha ... I've
> > looked at many, many versions of them).
> >
> > Just for full disclosure my ldd looks like:
> >
> >         linux-vdso.so.1 (0x0000ffffab196000)
> >         libfcgi.so.0 => /usr/lib/libfcgi.so.0 (0x0000ffffa57f0000)
> >         liblttng-ust.so.1 => /usr/lib/liblttng-ust.so.1
> > (0x0000ffffa5750000)
> >         libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffffa55d0000)
> >         librt.so.1 => /lib/librt.so.1 (0x0000ffffa55b0000)
> >         libm.so.6 => /lib/libm.so.6 (0x0000ffffa5510000)
> >         libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffffa52f0000)
> >         libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffffa52c0000)
> >         libc.so.6 => /lib/libc.so.6 (0x0000ffffa5110000)
> >         /lib/ld-linux-aarch64.so.1 (0x0000ffffab15d000)
> >         liblttng-ust-common.so.1 =>
> > /usr/local/lib/liblttng-ust-common.so.1 (0x0000ffffa50e0000)
> >         liblttng-ust-tracepoint.so.1 =>
> > /usr/local/lib/liblttng-ust-tracepoint.so.1 (0x0000ffffa50a0000)
> >         libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffffa5080000)
> >         libz.so.1 => /lib/libz.so.1 (0x0000ffffa5050000)
> >
> >
>
> I find it very suspicious that `liblttng-ust.so.1` is in `/usr/lib`,
> while the other lttng-ust libraries are being loaded from `/usr/local/lib`.
>
> This information also matches the statedump and the LD_DEBUG info from
> later on.
>
> Could you verify some of the following information:
>
> 1. In your build root for lttng-ust, enumerate all the liblttng*so
> files. For each shared object, run `file $libname` and record the value
> of the BuildID hash.
>
> 2. In your deployed environment, enumerate all the liblttng*so files in
> /usr/local/lib and record the value of the BuildID hash.
>
> 3. In your deployed environment, enumerate all the liblttng*so files in
> /usr/lib and record the value of the BuildID hash.
>
> The BuildID will vary from one object to the next, but the deployed
> environment BuildIDs should match the BuildIDs of the corresponding
> library from the build root.
>
> In particular my interest is the `liblttng-ust.so.1` library  that is
> loaded  from `/usr/lib` in your deployed environment, and if there are
> any in `/usr/local/lib`.
>
> In the working hello example, does it also end up linking against
> `/usr/lib/liblttng-ust.so.1`?
>
> >     Further along in stepping back:
> >
> >        - Does make check for lttng-ust pass in your environment?
> >
> >
> > So I'm glad you mentioned this.  I didn't try to run make check because
> > #1 have never seen anything in the documentation about it and what it
> > does and #2 It said that it depended on Perl and since this is an
> > embedded system, I really didn't want to pull it and its dependencies in.
> >
> > Nevertheless, I ran it on my native target build I setup to see what
> > would happen.  I didn't notice it complaining about perl so then I
> > checked my manifest and perl is in fact in our image (I guess some other
> > recipe pulled it in and I never paid attention to that):
> >
> > Making check in include
> > make[1]: Entering directory
> > '/home/iadmin/lttng-ust/lttng-ust-2.13.8/include'
> > ...
>  >
> >
> >        - Does make check for lttng-tools pass in your environment?
> >
> > I'm still using lttng-tools built by yocto so I don't know how to run
> > make check in that situation.  From devtool maybe???  I don't know, I'd
> > have to look into that one.   I never upgraded lttng-tools either, only
> > upgraded lttng-ust and built from source tar to get the fix you
> > mentioned for TLS stuff.
>
> The poky version of the tools recipe seems to have some stuff relating
> to ptest, which seems to be a yocto tool for package testing. There
> might be a run to run that?
>
> https://git.yoctoproject.org/poky/plain/meta/recipes-kernel/lttng/lttng-tools_2.13.13.bb
> https://wiki.yoctoproject.org/wiki/Ptest
>
> Otherwise I would get the lttng-tools source for your version,
> bootstrap, configure, build, and run the tests in tree.
>
> The bulk of regression testing for lttng-ust, -modules, and -tools
> happens during `make check` for lttng-tools.
>
> >
> >        - Is this reproducible in non-yocto environments or on other
> >     architectures with the same project?
> >
> > Hmm, that's a good question.  I don't have any other embedded targets to
> > try this on that are a different arch.  I think I heard at one point
> > early on a heavily stubbed out version ran on a PC but I'd have to look
> > into that.
> >
> >        - Does running the traced application with `LTTNG_UST_DEBUG=1` yield
> >     more information?
> >
> >
> > So far I've just been trying to get the app to run without even starting
> > lttng just to get past segfault issue.  Here is what it looks like with
> > debug turned on without starting lttng or a lttng-sessiond:
> >
> > liblttng_ust_tracepoint[599/599]: Your compiler treats weak symbols with
> > hidden visibility for integer objects as SAME address between compile
> > units part of the same module. (in check_weak_hidden() at
> >
> > liblttng_ust[599/601]: Waiting for local apps sessiond (in
> > wait_for_sessiond() at lttng-ust-comm.c:1758)
> > liblttng_ust[599/600]: Info: sessiond not accepting connections to
> > global apps socket (in ust_listener_thread() at lttng-ust-comm.c:1884)
> > Segmentation fault
> >
> >        - I'd also run lttng-sessiond with the environment variable
> >     `LTTNG_UST_DEBUG=1` set and `-vvv --verbose-consumer` in the program
> >     arguments and capture for stdout & stderr into a file for analysis.
> >        - Verify if the main program is dynamically linked with `ldd`
> >        - Verify which libraries are loaded at runtime and which calls are
> >     shimmed with `LD_DEBUG=libs,binding`
> >
> >
> > I provided some of this above.
> > The output of LD_DEBUG=libs,bindings is attached as file
> > ld_debug-libs-bindings.txt
>
> Thanks. For your own debugging, I would recommend to run the hello world
> example with the same environment variables in order to have comparison
> point to a working example.
>
>  From my observations in my own environment, for the hello example (and
> presumably your app if it's set up in the same way) I would expect
> something similar to the following
>
> ```
>    3624895:     initialize program: ./hello
>     3624895:
>     3624895:     binding file ./hello [0] to
> /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `dlopen' [GLIBC_2.34]
>     3624895:     opening
> file=/x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0];
> direct_opencount=3
>     3624895:
>     3624895:     binding file ./hello [0] to
> /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `dlsym' [GLIBC_2.34]
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tp_rcu_re
> ad_lock'
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tp_rcu_re
> ad_unlock'
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tp_rcu_de
> reference_sym'
>     3624895:     binding file
> x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tracepoin
> t_module_register'
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tracepoin
> t_module_unregister'
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tp_disabl
> e_destructors'
>     3624895:     binding file
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0] to
> /x/lttng/master/usr/lib/liblttng-ust-tracepoint.so.1 [0]: normal symbol
> `lttng_ust_tp_get_de
> structors_state'
>     3624895:     binding file ./hello [0] to
> /x/lttng/master/usr/lib/liblttng-ust.so.1 [0]: normal symbol
> `lttng_ust_probe_register'
>     3624895:
>     3624895:     transferring control: ./hello
>     3624895:
> ```
>
> The bit that is missing from your program is the binding of
> `lttng_ust_probe_register`. Here, it's not like the symbol isn't
> resolved at all: it's found earlier and if it didn't exist at all the
> error would probably be relatively clear like:
>
> ```
> ./doc/examples/hello/hello: symbol lookup error:
> /x/lttng/master/usr/lib/liblttng-ust.so.1: undefined symbol:
> lttng_ust_probe_register
> ```
>
> I'd like you to run another test with your app. This should indicate
> that all symbols should be immediately resolved during dynamic loading
> rather than performing lazy symbol resolution. The goal of this test is
> to separate clearly the symbol resolution from the call that happens in
> the program's initialization. You can also run it with the hello demo to
> compare the runs.
>
> ```
> LTTNG_UST_DEBUG=1 LD_DEBUG=all LD_BIND_NOW=1 ./my_app

Done!  See the attached tar ... it's quite big 38k lines!

I see interesting stuff in there but not quite sure what's going on.

Regards,

Brian

[-- Attachment #2: my_resolved_symbols.tar.xz --]
[-- Type: application/x-xz, Size: 129328 bytes --]

[-- Attachment #3: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev