* [lttng-dev] LTTNG LIB UST Crash
@ 2023-09-05 6:30 Lakshmi Deverkonda via lttng-dev
2023-09-05 15:27 ` Kienan Stewart via lttng-dev
[not found] ` <b16699e8-92de-2088-2912-e956443f4a3e@efficios.com>
0 siblings, 2 replies; 6+ messages in thread
From: Lakshmi Deverkonda via lttng-dev @ 2023-09-05 6:30 UTC (permalink / raw)
To: lttng-dev@lists.lttng.org
[-- Attachment #1.1: Type: text/plain, Size: 3477 bytes --]
Hi All,
I am observing lttng crash while trying to interface LTTNG for one of my python3 application.
I just tried the following things,
Added "import lttngust " in the code.
# lttng create clagd
Session clagd created.
Traces will be written in /root/lttng-traces/clagd-20230905-062210
# lttng enable-event --python clagd
#lttng start
#service clagd start
cumulus-core: Running cl-support for core files "clagd-ust.95410.1693895062.core"
#0 0x00007f134fb938eb in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f134fb7e535 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f134fb7e40f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f134fb8c1a2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f134f1a9677 in lttng_ust_add_fd_to_tracker () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#5 0x00007f134f1bdcf4 in lttng_ust_elf_create () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#6 0x00007f134f1bf8de in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#7 0x00007f134fc8f957 in dl_iterate_phdr () from /lib/x86_64-linux-gnu/libc.so.6
#8 0x00007f134f1bff6b in lttng_ust_dl_update () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#9 0x00007f134f1c061a in do_lttng_ust_statedump () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#10 0x00007f134f1b5ca9 in lttng_handle_pending_statedump () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#11 0x00007f134f1ab6d1 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#12 0x00007f134f1ad7eb in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#13 0x00007f134ff0dfa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007f134fc5506f in clone () from /lib/x86_64-linux-gnu/libc.so.6
When I stop the lttng session, I see another core
#0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7fad9331f700 (LWP 2221103))]
(gdb) bt
#0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fad95183535 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fad9518340f in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fad951911a2 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007fad947e8d9f in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#5 0x00007fad947e98e3 in shm_object_table_destroy () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#6 0x00007fad947e4d9a in channel_destroy () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#7 0x00007fad947ba6b5 in lttng_session_destroy () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#8 0x00007fad947b47c6 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#9 0x00007fad947b4bac in lttng_ust_objd_unref () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#10 0x00007fad947b4bac in lttng_ust_objd_unref () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#11 0x00007fad947b4bac in lttng_ust_objd_unref () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#12 0x00007fad947b5304 in lttng_ust_objd_table_owner_cleanup () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#13 0x00007fad947b2b75 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0
#14 0x00007fad95512fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007fad9525a06f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Can you please help here if I'm missing something. This is a critical task item for us but are currently stuck with multiple lttng crashes.
Regards,
Lakshmi
[-- Attachment #1.2: Type: text/html, Size: 9370 bytes --]
[-- Attachment #2: Type: text/plain, Size: 156 bytes --]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [lttng-dev] LTTNG LIB UST Crash 2023-09-05 6:30 [lttng-dev] LTTNG LIB UST Crash Lakshmi Deverkonda via lttng-dev @ 2023-09-05 15:27 ` Kienan Stewart via lttng-dev [not found] ` <b16699e8-92de-2088-2912-e956443f4a3e@efficios.com> 1 sibling, 0 replies; 6+ messages in thread From: Kienan Stewart via lttng-dev @ 2023-09-05 15:27 UTC (permalink / raw) To: lttng-dev Hi Lakshmi, could you please provide us with the system details and version information for LTTng tools and UST? The bug reporting guidelines which cover the type of information required to respond adequately questions can be found here: https://lttng.org/community/#bug-reporting-guidelines Given that you are instrumenting a user space application, do you have a minimal reproducer of the crash including the details of how the application is invoked that you would be able to share? Some types of user space applications required tracing helpers loaded via LD_PRELOAD. More information can be found here https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers If you're unable to share code or other log files due to company policy, or require responses within a guaranteed time frame, EfficiOS offers commercial support services: https://www.efficios.com/services/ thanks, kienan On 2023-09-05 02:30, Lakshmi Deverkonda via lttng-dev wrote: > Hi All, > > I am observing lttng crash while trying to interface LTTNG for one of my > python3 application. > I just tried the following things, > > Added "import lttngust " in the code. > > # lttng create clagd > Session clagd created. > Traces will be written in /root/lttng-traces/clagd-20230905-062210 > > # lttng enable-event --python clagd > #lttng start > > #service clagd start > * > * > *cumulus-core: Running cl-support for core files > "clagd-ust.95410.1693895062.core"* > > #0 0x00007f134fb938eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007f134fb7e535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 > #2 0x00007f134fb7e40f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #3 0x00007f134fb8c1a2 in __assert_fail () from > /lib/x86_64-linux-gnu/libc.so.6 > #4 0x00007f134f1a9677 in lttng_ust_add_fd_to_tracker () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #5 0x00007f134f1bdcf4 in lttng_ust_elf_create () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #6 0x00007f134f1bf8de in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #7 0x00007f134fc8f957 in dl_iterate_phdr () from > /lib/x86_64-linux-gnu/libc.so.6 > #8 0x00007f134f1bff6b in lttng_ust_dl_update () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #9 0x00007f134f1c061a in do_lttng_ust_statedump () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #10 0x00007f134f1b5ca9 in lttng_handle_pending_statedump () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #11 0x00007f134f1ab6d1 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #12 0x00007f134f1ad7eb in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #13 0x00007f134ff0dfa3 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #14 0x00007f134fc5506f in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > > When I stop the lttng session, I see another core > > #0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 > [Current thread is 1 (Thread 0x7fad9331f700 (LWP 2221103))] > (gdb) bt > #0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007fad95183535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 > #2 0x00007fad9518340f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #3 0x00007fad951911a2 in __assert_fail () from > /lib/x86_64-linux-gnu/libc.so.6 > #4 0x00007fad947e8d9f in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #5 0x00007fad947e98e3 in shm_object_table_destroy () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #6 0x00007fad947e4d9a in channel_destroy () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #7 0x00007fad947ba6b5 in lttng_session_destroy () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #8 0x00007fad947b47c6 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #9 0x00007fad947b4bac in lttng_ust_objd_unref () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #10 0x00007fad947b4bac in lttng_ust_objd_unref () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #11 0x00007fad947b4bac in lttng_ust_objd_unref () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #12 0x00007fad947b5304 in lttng_ust_objd_table_owner_cleanup () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #13 0x00007fad947b2b75 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #14 0x00007fad95512fa3 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #15 0x00007fad9525a06f in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > > Can you please help here if I'm missing something. This is a critical > task item for us but are currently stuck with multiple lttng crashes. > > Regards, > Lakshmi > > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <b16699e8-92de-2088-2912-e956443f4a3e@efficios.com>]
* Re: [lttng-dev] LTTNG LIB UST Crash [not found] ` <b16699e8-92de-2088-2912-e956443f4a3e@efficios.com> @ 2023-09-06 10:02 ` Lakshmi Deverkonda via lttng-dev 2023-09-06 15:31 ` Kienan Stewart via lttng-dev 0 siblings, 1 reply; 6+ messages in thread From: Lakshmi Deverkonda via lttng-dev @ 2023-09-06 10:02 UTC (permalink / raw) To: lttng-dev@lists.lttng.org [-- Attachment #1.1: Type: text/plain, Size: 7065 bytes --] Thanks for the reply. Issue is fixed after loading the tracing helpers. I have one query wrt to logging wrt lttng on python3 application. Is there any way I can avoid the file logging and only trace via lttng ? Regards, Lakshmi ________________________________ From: Kienan Stewart <kstewart@efficios.com> Sent: 05 September 2023 21:20 To: Lakshmi Deverkonda <laksd@nvidia.com> Subject: Re: [lttng-dev] LTTNG LIB UST Crash External email: Use caution opening links or attachments Hi Lakshmi, could you please provide us with the system details and version information for LTTng tools and UST? The bug reporting guidelines which cover the type of information required to respond adequately questions can be found here: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fcommunity%2F%23bug-reporting-guidelines&data=05%7C01%7Claksd%40nvidia.com%7C087f25ce0bc54b0b0cf008dbae27e747%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638295258729072557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZUKJj2JTtxWJNp9Prf8XjarnbPRPbLGXQy1Eou8AvRE%3D&reserved=0<https://lttng.org/community/#bug-reporting-guidelines> Given that you are instrumenting a user space application, do you have a minimal reproducer of the crash including the details of how the application is invoked that you would be able to share? Some types of user space applications required tracing helpers loaded via LD_PRELOAD. More information can be found here https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.13%2F%23doc-prebuilt-ust-helpers&data=05%7C01%7Claksd%40nvidia.com%7C087f25ce0bc54b0b0cf008dbae27e747%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638295258729072557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=g9HSdPQPqYw%2BdTXH9kKKyYjNZDX90NGArrUTW8UHp7s%3D&reserved=0<https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers> If you're unable to share code or other log files due to company policy, or require responses within a guaranteed time frame, EfficiOS offers commercial support services: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.efficios.com%2Fservices%2F&data=05%7C01%7Claksd%40nvidia.com%7C087f25ce0bc54b0b0cf008dbae27e747%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638295258729072557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Yry9Y74Ydo9eHmC5uP0bhcueqYTNoXbXpyiOLIytShc%3D&reserved=0<https://www.efficios.com/services/> thanks, kienan p.s. Sorry for forgetting to CC you in my earlier reply to the list! On 2023-09-05 02:30, Lakshmi Deverkonda via lttng-dev wrote: > Hi All, > > I am observing lttng crash while trying to interface LTTNG for one of my > python3 application. > I just tried the following things, > > Added "import lttngust " in the code. > > # lttng create clagd > Session clagd created. > Traces will be written in /root/lttng-traces/clagd-20230905-062210 > > # lttng enable-event --python clagd > #lttng start > > #service clagd start > * > * > *cumulus-core: Running cl-support for core files > "clagd-ust.95410.1693895062.core"* > > #0 0x00007f134fb938eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007f134fb7e535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 > #2 0x00007f134fb7e40f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #3 0x00007f134fb8c1a2 in __assert_fail () from > /lib/x86_64-linux-gnu/libc.so.6 > #4 0x00007f134f1a9677 in lttng_ust_add_fd_to_tracker () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #5 0x00007f134f1bdcf4 in lttng_ust_elf_create () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #6 0x00007f134f1bf8de in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #7 0x00007f134fc8f957 in dl_iterate_phdr () from > /lib/x86_64-linux-gnu/libc.so.6 > #8 0x00007f134f1bff6b in lttng_ust_dl_update () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #9 0x00007f134f1c061a in do_lttng_ust_statedump () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #10 0x00007f134f1b5ca9 in lttng_handle_pending_statedump () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #11 0x00007f134f1ab6d1 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #12 0x00007f134f1ad7eb in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #13 0x00007f134ff0dfa3 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #14 0x00007f134fc5506f in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > > When I stop the lttng session, I see another core > > #0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 > [Current thread is 1 (Thread 0x7fad9331f700 (LWP 2221103))] > (gdb) bt > #0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007fad95183535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 > #2 0x00007fad9518340f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > #3 0x00007fad951911a2 in __assert_fail () from > /lib/x86_64-linux-gnu/libc.so.6 > #4 0x00007fad947e8d9f in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #5 0x00007fad947e98e3 in shm_object_table_destroy () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #6 0x00007fad947e4d9a in channel_destroy () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #7 0x00007fad947ba6b5 in lttng_session_destroy () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #8 0x00007fad947b47c6 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #9 0x00007fad947b4bac in lttng_ust_objd_unref () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #10 0x00007fad947b4bac in lttng_ust_objd_unref () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #11 0x00007fad947b4bac in lttng_ust_objd_unref () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #12 0x00007fad947b5304 in lttng_ust_objd_table_owner_cleanup () from > /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #13 0x00007fad947b2b75 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 > #14 0x00007fad95512fa3 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #15 0x00007fad9525a06f in clone () from /lib/x86_64-linux-gnu/libc.so.6 > > > Can you please help here if I'm missing something. This is a critical > task item for us but are currently stuck with multiple lttng crashes. > > Regards, > Lakshmi > > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=05%7C01%7Claksd%40nvidia.com%7C087f25ce0bc54b0b0cf008dbae27e747%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638295258729072557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=k3GLHQwri5y6mlLv4nO3XL97f2nuYLM2KHWN3%2FpbK1U%3D&reserved=0<https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev> [-- Attachment #1.2: Type: text/html, Size: 10669 bytes --] [-- Attachment #2: Type: text/plain, Size: 156 bytes --] _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [lttng-dev] LTTNG LIB UST Crash 2023-09-06 10:02 ` Lakshmi Deverkonda via lttng-dev @ 2023-09-06 15:31 ` Kienan Stewart via lttng-dev 2023-09-07 11:46 ` Lakshmi Deverkonda via lttng-dev 0 siblings, 1 reply; 6+ messages in thread From: Kienan Stewart via lttng-dev @ 2023-09-06 15:31 UTC (permalink / raw) To: lttng-dev, Lakshmi Deverkonda; +Cc: Kienan Stewart Hi Lakshmi, On 2023-09-06 06:02, Lakshmi Deverkonda via lttng-dev wrote: > Thanks for the reply. Issue is fixed after loading the tracing helpers. > > I have one query wrt to logging wrt lttng on python3 application. Is > there any way I can avoid the file logging and only trace via lttng ? > In the example python application at https://lttng.org/docs/v2.13/#doc-python-application the log messages are not written to disk or to stderr. I lack the details of your application to give you a more precise answer. Hope this helps, kienan > Regards, > Lakshmi > > > ------------------------------------------------------------------------ > *From:* Kienan Stewart <kstewart@efficios.com> > *Sent:* 05 September 2023 21:20 > *To:* Lakshmi Deverkonda <laksd@nvidia.com> > *Subject:* Re: [lttng-dev] LTTNG LIB UST Crash > External email: Use caution opening links or attachments > > > Hi Lakshmi, > > could you please provide us with the system details and version > information for LTTng tools and UST? > > The bug reporting guidelines which cover the type of information > required to respond adequately questions can be found here: > https://lttng.org/community/#bug-reporting-guidelines <https://lttng.org/community/#bug-reporting-guidelines> > > Given that you are instrumenting a user space application, do you have a > minimal reproducer of the crash including the details of how the > application is invoked that you would be able to share? > > Some types of user space applications required tracing helpers loaded > via LD_PRELOAD. More information can be found here > https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers <https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers> > > If you're unable to share code or other log files due to company policy, > or require responses within a guaranteed time frame, EfficiOS offers > commercial support services: > https://www.efficios.com/services/ <https://www.efficios.com/services/> > > thanks, > kienan > > p.s. Sorry for forgetting to CC you in my earlier reply to the list! > > On 2023-09-05 02:30, Lakshmi Deverkonda via lttng-dev wrote: >> Hi All, >> >> I am observing lttng crash while trying to interface LTTNG for one of my >> python3 application. >> I just tried the following things, >> >> Added "import lttngust " in the code. >> >> # lttng create clagd >> Session clagd created. >> Traces will be written in /root/lttng-traces/clagd-20230905-062210 >> >> # lttng enable-event --python clagd >> #lttng start >> >> #service clagd start >> * >> * >> *cumulus-core: Running cl-support for core files >> "clagd-ust.95410.1693895062.core"* >> >> #0 0x00007f134fb938eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x00007f134fb7e535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 >> #2 0x00007f134fb7e40f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >> #3 0x00007f134fb8c1a2 in __assert_fail () from >> /lib/x86_64-linux-gnu/libc.so.6 >> #4 0x00007f134f1a9677 in lttng_ust_add_fd_to_tracker () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #5 0x00007f134f1bdcf4 in lttng_ust_elf_create () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #6 0x00007f134f1bf8de in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #7 0x00007f134fc8f957 in dl_iterate_phdr () from >> /lib/x86_64-linux-gnu/libc.so.6 >> #8 0x00007f134f1bff6b in lttng_ust_dl_update () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #9 0x00007f134f1c061a in do_lttng_ust_statedump () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #10 0x00007f134f1b5ca9 in lttng_handle_pending_statedump () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #11 0x00007f134f1ab6d1 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #12 0x00007f134f1ad7eb in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #13 0x00007f134ff0dfa3 in start_thread () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> #14 0x00007f134fc5506f in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> >> >> When I stop the lttng session, I see another core >> >> #0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 >> [Current thread is 1 (Thread 0x7fad9331f700 (LWP 2221103))] >> (gdb) bt >> #0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x00007fad95183535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 >> #2 0x00007fad9518340f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >> #3 0x00007fad951911a2 in __assert_fail () from >> /lib/x86_64-linux-gnu/libc.so.6 >> #4 0x00007fad947e8d9f in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #5 0x00007fad947e98e3 in shm_object_table_destroy () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #6 0x00007fad947e4d9a in channel_destroy () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #7 0x00007fad947ba6b5 in lttng_session_destroy () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #8 0x00007fad947b47c6 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #9 0x00007fad947b4bac in lttng_ust_objd_unref () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #10 0x00007fad947b4bac in lttng_ust_objd_unref () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #11 0x00007fad947b4bac in lttng_ust_objd_unref () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #12 0x00007fad947b5304 in lttng_ust_objd_table_owner_cleanup () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #13 0x00007fad947b2b75 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #14 0x00007fad95512fa3 in start_thread () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> #15 0x00007fad9525a06f in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> >> >> Can you please help here if I'm missing something. This is a critical >> task item for us but are currently stuck with multiple lttng crashes. >> >> Regards, >> Lakshmi >> >> _______________________________________________ >> lttng-dev mailing list >> lttng-dev@lists.lttng.org >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev <https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev> > > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [lttng-dev] LTTNG LIB UST Crash 2023-09-06 15:31 ` Kienan Stewart via lttng-dev @ 2023-09-07 11:46 ` Lakshmi Deverkonda via lttng-dev 2023-09-07 16:06 ` Kienan Stewart via lttng-dev 0 siblings, 1 reply; 6+ messages in thread From: Lakshmi Deverkonda via lttng-dev @ 2023-09-07 11:46 UTC (permalink / raw) To: lttng-dev@lists.lttng.org [-- Attachment #1.1: Type: text/plain, Size: 11875 bytes --] Thanks for replying. Basically in our python3 application, we already have a logger which will redirect the logs to a log file. By default, only info logs gets logged unless user explicitly turns on debug logging via cli. For LTTNG Tracing, we would want to log all the events that is both info/debugs. >>Do you think there would be any overhead on the application? >> I cannot use the existing logger which does a file logging so for lttng only I have created just a new logger without any handler. Should this be fine? >>Also, I see the default channel created for the python logging in lttng "lttng_python_channel" . From the documentation, I see that we cannot create another channel for python logging. I would want to modify some of the attribute for the default channel such as making event-loss-mode as "overwrite" and increasing the trace_file_count. How can I do it ? This is one of the necessary requirement for our application. Can you please guide on this ? Regards, Lakshmi ________________________________ From: Kienan Stewart <kstewart@efficios.com> Sent: 06 September 2023 21:01 To: lttng-dev@lists.lttng.org <lttng-dev@lists.lttng.org>; Lakshmi Deverkonda <laksd@nvidia.com> Subject: Re: [lttng-dev] LTTNG LIB UST Crash External email: Use caution opening links or attachments Hi Lakshmi, On 2023-09-06 06:02, Lakshmi Deverkonda via lttng-dev wrote: > Thanks for the reply. Issue is fixed after loading the tracing helpers. > > I have one query wrt to logging wrt lttng on python3 application. Is > there any way I can avoid the file logging and only trace via lttng ? > In the example python application at https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.13%2F%23doc-python-application&data=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638296111141743499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HMryNCihtLBKA49TWfsxp4%2FkzkFJhZrlyocoAsR7w78%3D&reserved=0<https://lttng.org/docs/v2.13/#doc-python-application> the log messages are not written to disk or to stderr. I lack the details of your application to give you a more precise answer. Hope this helps, kienan > Regards, > Lakshmi > > > ------------------------------------------------------------------------ > *From:* Kienan Stewart <kstewart@efficios.com> > *Sent:* 05 September 2023 21:20 > *To:* Lakshmi Deverkonda <laksd@nvidia.com> > *Subject:* Re: [lttng-dev] LTTNG LIB UST Crash > External email: Use caution opening links or attachments > > > Hi Lakshmi, > > could you please provide us with the system details and version > information for LTTng tools and UST? > > The bug reporting guidelines which cover the type of information > required to respond adequately questions can be found here: > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fcommunity%2F%23bug-reporting-guidelines&data=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638296111141743499%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZU4mg%2BDNKMbtxCVA3kRuyCZAaj7yUH5bLz6p4oXpROA%3D&reserved=0<https://lttng.org/community/#bug-reporting-guidelines> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fcommunity%2F%23bug-reporting-guidelines&data=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638296111141899697%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=x0QzE9thuFtUFdNKd2qg0Fo4XSYqeTB4vbEBl88dGxE%3D&reserved=0<https://lttng.org/community/#bug-reporting-guidelines>> > > Given that you are instrumenting a user space application, do you have a > minimal reproducer of the crash including the details of how the > application is invoked that you would be able to share? > > Some types of user space applications required tracing helpers loaded > via LD_PRELOAD. More information can be found here > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.13%2F%23doc-prebuilt-ust-helpers&data=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638296111141899697%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=WpDm%2FpMZMb3EukOw9NF9L2GiMZpWeOAq3It8H8p%2BNSI%3D&reserved=0<https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flttng.org%2Fdocs%2Fv2.13%2F%23doc-prebuilt-ust-helpers&data=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638296111141899697%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=WpDm%2FpMZMb3EukOw9NF9L2GiMZpWeOAq3It8H8p%2BNSI%3D&reserved=0<https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers>> > > If you're unable to share code or other log files due to company policy, > or require responses within a guaranteed time frame, EfficiOS offers > commercial support services: > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.efficios.com%2Fservices%2F&data=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638296111141899697%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=92syXIQdFYIdm7jBG%2BEc%2FNFcSBujENDvSRG%2BaeyPCMs%3D&reserved=0<https://www.efficios.com/services/> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.efficios.com%2Fservices%2F&data=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638296111141899697%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=92syXIQdFYIdm7jBG%2BEc%2FNFcSBujENDvSRG%2BaeyPCMs%3D&reserved=0<https://www.efficios.com/services/>> > > thanks, > kienan > > p.s. Sorry for forgetting to CC you in my earlier reply to the list! > > On 2023-09-05 02:30, Lakshmi Deverkonda via lttng-dev wrote: >> Hi All, >> >> I am observing lttng crash while trying to interface LTTNG for one of my >> python3 application. >> I just tried the following things, >> >> Added "import lttngust " in the code. >> >> # lttng create clagd >> Session clagd created. >> Traces will be written in /root/lttng-traces/clagd-20230905-062210 >> >> # lttng enable-event --python clagd >> #lttng start >> >> #service clagd start >> * >> * >> *cumulus-core: Running cl-support for core files >> "clagd-ust.95410.1693895062.core"* >> >> #0 0x00007f134fb938eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x00007f134fb7e535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 >> #2 0x00007f134fb7e40f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >> #3 0x00007f134fb8c1a2 in __assert_fail () from >> /lib/x86_64-linux-gnu/libc.so.6 >> #4 0x00007f134f1a9677 in lttng_ust_add_fd_to_tracker () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #5 0x00007f134f1bdcf4 in lttng_ust_elf_create () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #6 0x00007f134f1bf8de in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #7 0x00007f134fc8f957 in dl_iterate_phdr () from >> /lib/x86_64-linux-gnu/libc.so.6 >> #8 0x00007f134f1bff6b in lttng_ust_dl_update () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #9 0x00007f134f1c061a in do_lttng_ust_statedump () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #10 0x00007f134f1b5ca9 in lttng_handle_pending_statedump () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #11 0x00007f134f1ab6d1 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #12 0x00007f134f1ad7eb in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #13 0x00007f134ff0dfa3 in start_thread () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> #14 0x00007f134fc5506f in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> >> >> When I stop the lttng session, I see another core >> >> #0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 >> [Current thread is 1 (Thread 0x7fad9331f700 (LWP 2221103))] >> (gdb) bt >> #0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 >> #1 0x00007fad95183535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 >> #2 0x00007fad9518340f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >> #3 0x00007fad951911a2 in __assert_fail () from >> /lib/x86_64-linux-gnu/libc.so.6 >> #4 0x00007fad947e8d9f in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #5 0x00007fad947e98e3 in shm_object_table_destroy () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #6 0x00007fad947e4d9a in channel_destroy () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #7 0x00007fad947ba6b5 in lttng_session_destroy () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #8 0x00007fad947b47c6 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #9 0x00007fad947b4bac in lttng_ust_objd_unref () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #10 0x00007fad947b4bac in lttng_ust_objd_unref () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #11 0x00007fad947b4bac in lttng_ust_objd_unref () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #12 0x00007fad947b5304 in lttng_ust_objd_table_owner_cleanup () from >> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #13 0x00007fad947b2b75 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >> #14 0x00007fad95512fa3 in start_thread () from >> /lib/x86_64-linux-gnu/libpthread.so.0 >> #15 0x00007fad9525a06f in clone () from /lib/x86_64-linux-gnu/libc.so.6 >> >> >> Can you please help here if I'm missing something. This is a critical >> task item for us but are currently stuck with multiple lttng crashes. >> >> Regards, >> Lakshmi >> >> _______________________________________________ >> lttng-dev mailing list >> lttng-dev@lists.lttng.org >> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638296111141899697%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BaJ6W4wzDuPk2BuBxWO6bEptWW1NpA2oWZwEOGIDpRY%3D&reserved=0<https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev> <https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638296111141899697%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BaJ6W4wzDuPk2BuBxWO6bEptWW1NpA2oWZwEOGIDpRY%3D&reserved=0<https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev>> > > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.lttng.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Flttng-dev&data=05%7C01%7Claksd%40nvidia.com%7C5dec50116ea24aef578508dbaeee6437%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638296111141899697%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BaJ6W4wzDuPk2BuBxWO6bEptWW1NpA2oWZwEOGIDpRY%3D&reserved=0<https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev> [-- Attachment #1.2: Type: text/html, Size: 16653 bytes --] [-- Attachment #2: Type: text/plain, Size: 156 bytes --] _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [lttng-dev] LTTNG LIB UST Crash 2023-09-07 11:46 ` Lakshmi Deverkonda via lttng-dev @ 2023-09-07 16:06 ` Kienan Stewart via lttng-dev 0 siblings, 0 replies; 6+ messages in thread From: Kienan Stewart via lttng-dev @ 2023-09-07 16:06 UTC (permalink / raw) To: lttng-dev, Lakshmi Deverkonda; +Cc: Kienan Stewart Hi Lakshmi, On 2023-09-07 07:46, Lakshmi Deverkonda via lttng-dev wrote: > Thanks for replying. > > Basically in our python3 application, we already have a logger which > will redirect the logs to a log file. By default, only info logs gets > logged unless user explicitly turns on debug logging via cli. > > For LTTNG Tracing, we would want to log all the events that is both > info/debugs. > >>Do you think there would be any overhead on the application? Yes, but I'm not in position to quantify its effect on your application. When lttngust is imported, one or two threads are spawned to register and communicate with the lttng-sessiond (see `lttngust.agent`). Some debug information about this process is available when you run the application with `LTTNG_UST_PYTHON_DEBUG=1` set in the environment In terms of logging, the default behaviour is to add an instance of `lttngust.loghandler._Handler` to the root logger with level `logging.NOTSET`. Instances of this handler load the liblttng-ust-python-agent.so dynamic library, and the `emit()` method encodes the record data and calls the library's `py_tracepoint` function. This is a thin wrapper of liblttng-ust, which will ultimately send the encoded event to the running consumerd. One of the goals of the tracing libraries is to have a minimal footprint and impact on the running applications, but it is non-zero. In any case, if you`re interested in the overhead, you can profile your application with and without tracing and measure the effect. > >> I cannot use the existing logger which does a file logging so for > lttng only I have created just a new logger without any handler. Should > this be fine? I think this question is more about how you're writing your python application and not about LTTng / tracing userspace applications, so I don't think I can answer it for you. > >>Also, I see the default channel created for the python logging in > lttng "lttng_python_channel" . From the documentation, I see that we > cannot create another channel for python logging. > I would want to modify some of the attribute for the default channel > such as making event-loss-mode as "overwrite" and increasing the > trace_file_count. How can I do it ? This is one of the necessary > requirement for our application. Can you please guide on this ? You can create the channel explicitly with your desired configuration before running `lttng enable-events --python my-logger` For example: ``` $ lttng create Session auto-20230907-103141 created. $ lttng enable-channel --userspace --overwrite --tracefile-count 10 --tracefile-size 1073741824 lttng_python_channel ust channel lttng_python_channel enabled for session auto-20230907-103141 $ lttng enable-event --python my-logger $ lttng list auto-20230907-103141 ... - lttng_python_channel: [enabled] Attributes; Event-loss mode: overwrite Sub-buffer size: 524288 bytes ... Trace file count: 10 per stream Trace file size: 1073741824 bytes ... ``` thanks, kienan > > > Regards, > Lakshmi > ------------------------------------------------------------------------ > *From:* Kienan Stewart <kstewart@efficios.com> > *Sent:* 06 September 2023 21:01 > *To:* lttng-dev@lists.lttng.org <lttng-dev@lists.lttng.org>; Lakshmi > Deverkonda <laksd@nvidia.com> > *Subject:* Re: [lttng-dev] LTTNG LIB UST Crash > External email: Use caution opening links or attachments > > > Hi Lakshmi, > > On 2023-09-06 06:02, Lakshmi Deverkonda via lttng-dev wrote: >> Thanks for the reply. Issue is fixed after loading the tracing helpers. >> >> I have one query wrt to logging wrt lttng on python3 application. Is >> there any way I can avoid the file logging and only trace via lttng ? >> > > In the example python application at > https://lttng.org/docs/v2.13/#doc-python-application <https://lttng.org/docs/v2.13/#doc-python-application> the log messages > are not written to disk or to stderr. > > I lack the details of your application to give you a more precise answer. > > Hope this helps, > kienan > >> Regards, >> Lakshmi >> >> >> ------------------------------------------------------------------------ >> *From:* Kienan Stewart <kstewart@efficios.com> >> *Sent:* 05 September 2023 21:20 >> *To:* Lakshmi Deverkonda <laksd@nvidia.com> >> *Subject:* Re: [lttng-dev] LTTNG LIB UST Crash >> External email: Use caution opening links or attachments >> >> >> Hi Lakshmi, >> >> could you please provide us with the system details and version >> information for LTTng tools and UST? >> >> The bug reporting guidelines which cover the type of information >> required to respond adequately questions can be found here: >> https://lttng.org/community/#bug-reporting-guidelines <https://lttng.org/community/#bug-reporting-guidelines> <https://lttng.org/community/#bug-reporting-guidelines <https://lttng.org/community/#bug-reporting-guidelines>> >> >> Given that you are instrumenting a user space application, do you have a >> minimal reproducer of the crash including the details of how the >> application is invoked that you would be able to share? >> >> Some types of user space applications required tracing helpers loaded >> via LD_PRELOAD. More information can be found here >> https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers <https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers> <https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers <https://lttng.org/docs/v2.13/#doc-prebuilt-ust-helpers>> >> >> If you're unable to share code or other log files due to company policy, >> or require responses within a guaranteed time frame, EfficiOS offers >> commercial support services: >> https://www.efficios.com/services/ <https://www.efficios.com/services/> <https://www.efficios.com/services/ <https://www.efficios.com/services/>> >> >> thanks, >> kienan >> >> p.s. Sorry for forgetting to CC you in my earlier reply to the list! >> >> On 2023-09-05 02:30, Lakshmi Deverkonda via lttng-dev wrote: >>> Hi All, >>> >>> I am observing lttng crash while trying to interface LTTNG for one of my >>> python3 application. >>> I just tried the following things, >>> >>> Added "import lttngust " in the code. >>> >>> # lttng create clagd >>> Session clagd created. >>> Traces will be written in /root/lttng-traces/clagd-20230905-062210 >>> >>> # lttng enable-event --python clagd >>> #lttng start >>> >>> #service clagd start >>> * >>> * >>> *cumulus-core: Running cl-support for core files >>> "clagd-ust.95410.1693895062.core"* >>> >>> #0 0x00007f134fb938eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 >>> #1 0x00007f134fb7e535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 >>> #2 0x00007f134fb7e40f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >>> #3 0x00007f134fb8c1a2 in __assert_fail () from >>> /lib/x86_64-linux-gnu/libc.so.6 >>> #4 0x00007f134f1a9677 in lttng_ust_add_fd_to_tracker () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #5 0x00007f134f1bdcf4 in lttng_ust_elf_create () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #6 0x00007f134f1bf8de in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #7 0x00007f134fc8f957 in dl_iterate_phdr () from >>> /lib/x86_64-linux-gnu/libc.so.6 >>> #8 0x00007f134f1bff6b in lttng_ust_dl_update () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #9 0x00007f134f1c061a in do_lttng_ust_statedump () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #10 0x00007f134f1b5ca9 in lttng_handle_pending_statedump () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #11 0x00007f134f1ab6d1 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #12 0x00007f134f1ad7eb in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #13 0x00007f134ff0dfa3 in start_thread () from >>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> #14 0x00007f134fc5506f in clone () from /lib/x86_64-linux-gnu/libc.so.6 >>> >>> >>> When I stop the lttng session, I see another core >>> >>> #0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 >>> [Current thread is 1 (Thread 0x7fad9331f700 (LWP 2221103))] >>> (gdb) bt >>> #0 0x00007fad951988eb in raise () from /lib/x86_64-linux-gnu/libc.so.6 >>> #1 0x00007fad95183535 in abort () from /lib/x86_64-linux-gnu/libc.so.6 >>> #2 0x00007fad9518340f in ?? () from /lib/x86_64-linux-gnu/libc.so.6 >>> #3 0x00007fad951911a2 in __assert_fail () from >>> /lib/x86_64-linux-gnu/libc.so.6 >>> #4 0x00007fad947e8d9f in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #5 0x00007fad947e98e3 in shm_object_table_destroy () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #6 0x00007fad947e4d9a in channel_destroy () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #7 0x00007fad947ba6b5 in lttng_session_destroy () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #8 0x00007fad947b47c6 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #9 0x00007fad947b4bac in lttng_ust_objd_unref () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #10 0x00007fad947b4bac in lttng_ust_objd_unref () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #11 0x00007fad947b4bac in lttng_ust_objd_unref () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #12 0x00007fad947b5304 in lttng_ust_objd_table_owner_cleanup () from >>> /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #13 0x00007fad947b2b75 in ?? () from /lib/x86_64-linux-gnu/liblttng-ust.so.0 >>> #14 0x00007fad95512fa3 in start_thread () from >>> /lib/x86_64-linux-gnu/libpthread.so.0 >>> #15 0x00007fad9525a06f in clone () from /lib/x86_64-linux-gnu/libc.so.6 >>> >>> >>> Can you please help here if I'm missing something. This is a critical >>> task item for us but are currently stuck with multiple lttng crashes. >>> >>> Regards, >>> Lakshmi >>> >>> _______________________________________________ >>> lttng-dev mailing list >>> lttng-dev@lists.lttng.org >>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev <https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev> <https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev <https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev>> >> >> _______________________________________________ >> lttng-dev mailing list >> lttng-dev@lists.lttng.org >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev <https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev> > > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-09-07 16:07 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-09-05 6:30 [lttng-dev] LTTNG LIB UST Crash Lakshmi Deverkonda via lttng-dev
2023-09-05 15:27 ` Kienan Stewart via lttng-dev
[not found] ` <b16699e8-92de-2088-2912-e956443f4a3e@efficios.com>
2023-09-06 10:02 ` Lakshmi Deverkonda via lttng-dev
2023-09-06 15:31 ` Kienan Stewart via lttng-dev
2023-09-07 11:46 ` Lakshmi Deverkonda via lttng-dev
2023-09-07 16:06 ` Kienan Stewart via lttng-dev
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).