lttng-dev.lists.lttng.org archive mirror
 help / color / mirror / Atom feed
* [lttng-dev] What is DEFINE_LTTNG_UST_SIGBUS_STATE for and other beginner questions
@ 2023-10-04 16:56 Christopher Harvey via lttng-dev
  2023-10-04 19:04 ` Kienan Stewart via lttng-dev
  0 siblings, 1 reply; 5+ messages in thread
From: Christopher Harvey via lttng-dev @ 2023-10-04 16:56 UTC (permalink / raw)
  To: lttng-dev

Hi all,

I'm just getting started with lttng and have a couple of questions. My target is getting userspace tracepoints to work in a c++ application I control. I don't need kernel or system tracing. The target is running in a docker container I have control over as well. Running with --privileged is possible in my setup. I could run lttng inside the target container, outside on the host, or in another container on the same host. It's all the same to me. I don't know which setup is best, or if it even matters given that I don't need system or kernel traces.

I am using:
lttng-tools-2.13.11/      and       lttng-ust-2.13.6/
with userspace-rcu-0.14.0/

There is a slight version mismatch there, I just downloaded the "-lastest" from the links in the guide.

Here is my tracepoint provider header:
-----
#undef LTTNG_UST_TRACEPOINT_PROVIDER
#define LTTNG_UST_TRACEPOINT_PROVIDER my_provider

#undef LTTNG_UST_TRACEPOINT_INCLUDE
#define LTTNG_UST_TRACEPOINT_INCLUDE "./myLttngUstProvider.hpp"

#if !defined(_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
#define _TP_H

#include "lttng/tracepoint.h"

LTTNG_UST_TRACEPOINT_EVENT(
    /* Tracepoint provider name */
    my_provider,

    /* Tracepoint name */
    test,

    /* Input arguments */
    LTTNG_UST_TP_ARGS(
        int, arg1
    ),

    /* Output event fields */
    LTTNG_UST_TP_FIELDS(
        lttng_ust_field_integer(int, arg1, arg1)
    )
)

#endif /* _TP_H */

#include "lttng/tracepoint-event.h"
--------

and the cpp:
-------
#define LTTNG_UST_TRACEPOINT_CREATE_PROBES
#define LTTNG_UST_TRACEPOINT_DEFINE

#include "myLttngUstProvider.hpp"
#include "lttng/ust-sigbus.h"

DEFINE_LTTNG_UST_SIGBUS_STATE();
----

I'm suspicious of a couple of things in my setup.

First, if I omit the DEFINE_LTTNG_UST_SIGBUS_STATE(); line, my program fails to compile with the following error:
liblttng-ust-ctl.so: undefined reference to `lttng_ust_sigbus_state'

The getting started guide doesn't say anything about needing DEFINE_LTTNG_UST_SIGBUS_STATE so it makes me wonder if I have done something else wrong somewhere else. I ran a find *.so | xargs nm -D | grep lttng_ust_sigbus_state and nothing in lttng-ust provides that symbol, so it looks like the target application has to define it. Is that normal?

If I leave DEFINE_LTTNG_UST_SIGBUS_STATE in then my program compiles. If I run lttng-sessiond in the target container, then run lttng list --userspace I see no tracepoints in the list.

I tried looking at the lttng source code but there is a lot there. I'm looking for some next steps for debugging this setup. What are some things I could check to make sure the lttng daemon is running properly? What is the mechanism for advertising tracepoints to the lttng daemon? Is it possible that docker has somehow interfered with the lttng ipc mechanism? It would be a tiny bit more convenient for me to simply deploy lttng-tools as a separate container, can I consider doing that instead? I'm not running anything as root, just a normal user.

I appreciate any and all guidance. Looking forward to using this neat looking tool.

-Chris
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [lttng-dev] What is DEFINE_LTTNG_UST_SIGBUS_STATE for and other beginner questions
  2023-10-04 16:56 [lttng-dev] What is DEFINE_LTTNG_UST_SIGBUS_STATE for and other beginner questions Christopher Harvey via lttng-dev
@ 2023-10-04 19:04 ` Kienan Stewart via lttng-dev
  2023-10-04 23:12   ` Christopher Harvey via lttng-dev
  0 siblings, 1 reply; 5+ messages in thread
From: Kienan Stewart via lttng-dev @ 2023-10-04 19:04 UTC (permalink / raw)
  To: lttng-dev, chris

Hi Chris,

On 2023-10-04 12:56, Christopher Harvey via lttng-dev wrote:
> Hi all,
> 
> I'm just getting started with lttng and have a couple of questions. My target is getting userspace tracepoints to work in a c++ application I control. I don't need kernel or system tracing. The target is running in a docker container I have control over as well. Running with --privileged is possible in my setup. I could run lttng inside the target container, outside on the host, or in another container on the same host. It's all the same to me. I don't know which setup is best, or if it even matters given that I don't need system or kernel traces.
> 
> I am using:
> lttng-tools-2.13.11/      and       lttng-ust-2.13.6/
> with userspace-rcu-0.14.0/
> 
> There is a slight version mismatch there, I just downloaded the "-lastest" from the links in the guide.
> 
> Here is my tracepoint provider header:
> -----
> #undef LTTNG_UST_TRACEPOINT_PROVIDER
> #define LTTNG_UST_TRACEPOINT_PROVIDER my_provider
> 
> #undef LTTNG_UST_TRACEPOINT_INCLUDE
> #define LTTNG_UST_TRACEPOINT_INCLUDE "./myLttngUstProvider.hpp"
> 
> #if !defined(_TP_H) || defined(LTTNG_UST_TRACEPOINT_HEADER_MULTI_READ)
> #define _TP_H
> 
> #include "lttng/tracepoint.h"
> 
> LTTNG_UST_TRACEPOINT_EVENT(
>      /* Tracepoint provider name */
>      my_provider,
> 
>      /* Tracepoint name */
>      test,
> 
>      /* Input arguments */
>      LTTNG_UST_TP_ARGS(
>          int, arg1
>      ),
> 
>      /* Output event fields */
>      LTTNG_UST_TP_FIELDS(
>          lttng_ust_field_integer(int, arg1, arg1)
>      )
> )
> 
> #endif /* _TP_H */
> 
> #include "lttng/tracepoint-event.h"
> --------
> 
> and the cpp:
> -------
> #define LTTNG_UST_TRACEPOINT_CREATE_PROBES
> #define LTTNG_UST_TRACEPOINT_DEFINE
> 
> #include "myLttngUstProvider.hpp"
> #include "lttng/ust-sigbus.h"
> 
> DEFINE_LTTNG_UST_SIGBUS_STATE();
> ----
> 
> I'm suspicious of a couple of things in my setup.
> 
> First, if I omit the DEFINE_LTTNG_UST_SIGBUS_STATE(); line, my program fails to compile with the following error:
> liblttng-ust-ctl.so: undefined reference to `lttng_ust_sigbus_state'
> 

I wasn't able to reproduce the compilation error you are seeing. Could 
you describe in more detail how you are compiling and linking your 
tracepoint provider and application?

> The getting started guide doesn't say anything about needing DEFINE_LTTNG_UST_SIGBUS_STATE so it makes me wonder if I have done something else wrong somewhere else. I ran a find *.so | xargs nm -D | grep lttng_ust_sigbus_state and nothing in lttng-ust provides that symbol, so it looks like the target application has to define it. Is that normal?
> 

If the application is actually using liblttng-ust-ctl, then it would be 
normal to do the define. The header file explains that this is done to 
avoid unnecessarily pre-allocating space in the thread-local storage.
https://github.com/lttng/lttng-ust/blob/master/include/lttng/ust-sigbus.h#L39

Traced applications typically don't link against liblttng-ust-ctl.


> If I leave DEFINE_LTTNG_UST_SIGBUS_STATE in then my program compiles. If I run lttng-sessiond in the target container, then run lttng list --userspace I see no tracepoints in the list.
> 

Is your application actively running while you run `lttng list --userspace`?

`lttng list --userspace` will only list the userspace tracepoints of 
applications which are currently registered to the running session 
daemon. If an application disconnects, the tracepoints it registered are 
unloaded.

> I tried looking at the lttng source code but there is a lot there. I'm looking for some next steps for debugging this setup. What are some things I could check to make sure the lttng daemon is running properly? What is the mechanism for advertising tracepoints to the lttng daemon? Is it possible that docker has somehow interfered with the lttng ipc mechanism? It would be a tiny bit more convenient for me to simply deploy lttng-tools as a separate container, can I consider doing that instead? I'm not running anything as root, just a normal user.
> 
> I appreciate any and all guidance. Looking forward to using this neat looking tool.
> 
> -Chris

thanks,
kienan

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [lttng-dev] What is DEFINE_LTTNG_UST_SIGBUS_STATE for and other beginner questions
  2023-10-04 19:04 ` Kienan Stewart via lttng-dev
@ 2023-10-04 23:12   ` Christopher Harvey via lttng-dev
  2023-10-05 19:40     ` Christopher Harvey via lttng-dev
  0 siblings, 1 reply; 5+ messages in thread
From: Christopher Harvey via lttng-dev @ 2023-10-04 23:12 UTC (permalink / raw)
  To: Kienan Stewart, lttng-dev

On Wed, Oct 4, 2023, at 3:04 PM, Kienan Stewart wrote:
> Hi Chris,
>
> On 2023-10-04 12:56, Christopher Harvey via lttng-dev wrote:
>> Hi all,
>> 
>> I'm just getting started with lttng and have a couple of questions. My target is getting userspace tracepoints to work in a c++ application I control. I don't need kernel or system tracing. The target is running in a docker container I have control over as well. Running with --privileged is possible in my setup. I could run lttng inside the target container, outside on the host, or in another container on the same host. It's all the same to me. I don't know which setup is best, or if it even matters given that I don't need system or kernel traces.
>> 
>> I am using:
>> lttng-tools-2.13.11/      and       lttng-ust-2.13.6/
>> with userspace-rcu-0.14.0/
>> 
>> There is a slight version mismatch there, I just downloaded the "-lastest" from the links in the guide.
>> 
>> Here is my tracepoint provider header:
>> -----
[SNIP]
>> --------
>> 
>> and the cpp:
>> -------
[SNIP]
>> ----
>> 
>> I'm suspicious of a couple of things in my setup.
>> 
>> First, if I omit the DEFINE_LTTNG_UST_SIGBUS_STATE(); line, my program fails to compile with the following error:
>> liblttng-ust-ctl.so: undefined reference to `lttng_ust_sigbus_state'
>> 
>
> I wasn't able to reproduce the compilation error you are seeing. Could 
> you describe in more detail how you are compiling and linking your 
> tracepoint provider and application?

I just create a .o from the tracepoint provider compilation unit. I'm not making shared libraries or anything like that. More on this below.

>> The getting started guide doesn't say anything about needing DEFINE_LTTNG_UST_SIGBUS_STATE so it makes me wonder if I have done something else wrong somewhere else. I ran a find *.so | xargs nm -D | grep lttng_ust_sigbus_state and nothing in lttng-ust provides that symbol, so it looks like the target application has to define it. Is that normal?
>> 
>
> If the application is actually using liblttng-ust-ctl, then it would be 
> normal to do the define. The header file explains that this is done to 
> avoid unnecessarily pre-allocating space in the thread-local storage.
> https://github.com/lttng/lttng-ust/blob/master/include/lttng/ust-sigbus.h#L39
>
> Traced applications typically don't link against liblttng-ust-ctl.
>

This ended up being one of the main problems. My final linking flags are
-llttng-ust -llttng-ust-common -llttng-ust-tracepoint
I didn't have a particular reason for linking against liblttng-ust-ctl before. It was a mistake.

>> If I leave DEFINE_LTTNG_UST_SIGBUS_STATE in then my program compiles. If I run lttng-sessiond in the target container, then run lttng list --userspace I see no tracepoints in the list.
>> 
>
> Is your application actively running while you run `lttng list --userspace`?

yes

I was reading some lttng-sessiond source code and discovered -vvv as well as LTTNG_UST_DEBUG.

I'm still not seeing the tracepoints show up in lttng list --userspace, but I found more hints as to why.
Here is some output from my target application:
 liblttng_ust[454/467]: Info: sessiond not accepting connections to local apps socket (in ust_listener_thread() at lttng-ust-comm.c:1884)
liblttng_ust[454/467]: Waiting for local apps sessiond (in wait_for_sessiond() at lttng-ust-comm.c:1758

These prints only happen periodically when lttng-sessiond is running. If I stop lttng-sessiond then the prints stop so there is clearly some communication happening. Apparently this gets printed when there is an ECONNREFUSED from lttng-sessiond. I haven't gotten much further than that right now. I have pasted the output from sessiond -vvv here:

https://pastebin.com/raw/ayJJ2J9p

thank you for your time,
Chris
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [lttng-dev] What is DEFINE_LTTNG_UST_SIGBUS_STATE for and other beginner questions
  2023-10-04 23:12   ` Christopher Harvey via lttng-dev
@ 2023-10-05 19:40     ` Christopher Harvey via lttng-dev
  2023-10-05 19:59       ` Kienan Stewart via lttng-dev
  0 siblings, 1 reply; 5+ messages in thread
From: Christopher Harvey via lttng-dev @ 2023-10-05 19:40 UTC (permalink / raw)
  To: Kienan Stewart, lttng-dev

I got this one figured out. I used strace -fp <pid> -trace=connect on the target app to see why the connect was failing and it turns out there was a mismatch between sessiond's apps-socket paths and libust's.

time to review my build environment I guess. It's very possible my docker container's path environment variables are non-standard.

If I had a tiny bit of feedback to provide it would be to print out the errno and paths in the connect/bind commands in the DBG commands. I had to bust out strace to get that info. Small problem, but it could have saved me a few hours and some mailing list messages.

Thank you for making this nice looking software available for free. Looking forward to playing with it some more.

-Chris
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [lttng-dev] What is DEFINE_LTTNG_UST_SIGBUS_STATE for and other beginner questions
  2023-10-05 19:40     ` Christopher Harvey via lttng-dev
@ 2023-10-05 19:59       ` Kienan Stewart via lttng-dev
  0 siblings, 0 replies; 5+ messages in thread
From: Kienan Stewart via lttng-dev @ 2023-10-05 19:59 UTC (permalink / raw)
  To: Christopher Harvey, lttng-dev, chris

Hi Chris,

glad that you got it figured out! I created an issue based on your 
feedback. https://bugs.lttng.org/issues/1393

thanks,
kienan


On 2023-10-05 15:40, Christopher Harvey wrote:
> I got this one figured out. I used strace -fp <pid> -trace=connect on the target app to see why the connect was failing and it turns out there was a mismatch between sessiond's apps-socket paths and libust's.
> 
> time to review my build environment I guess. It's very possible my docker container's path environment variables are non-standard.
> 
> If I had a tiny bit of feedback to provide it would be to print out the errno and paths in the connect/bind commands in the DBG commands. I had to bust out strace to get that info. Small problem, but it could have saved me a few hours and some mailing list messages.
> 
> Thank you for making this nice looking software available for free. Looking forward to playing with it some more.
> 
> -Chris
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2023-10-05 19:59 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-10-04 16:56 [lttng-dev] What is DEFINE_LTTNG_UST_SIGBUS_STATE for and other beginner questions Christopher Harvey via lttng-dev
2023-10-04 19:04 ` Kienan Stewart via lttng-dev
2023-10-04 23:12   ` Christopher Harvey via lttng-dev
2023-10-05 19:40     ` Christopher Harvey via lttng-dev
2023-10-05 19:59       ` Kienan Stewart via lttng-dev

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).