[lttng-dev] delay between trigger and action

[lttng-dev] delay between trigger and action

[Yitschak, Yehuda via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 2137 bytes --]

Hi everyone

I am experimenting with enabling trace for a specific iteration of a loop in my application.
I created 2 trace points at the start and end of the loop which log the iteration number.
the loop itself creates thousands of trace points. It runs for ~5ms.

on lttng side I created 2 triggers to start and stop my session in case the loop tracepoint registers a specific iteration number.
the triggers seem to work since there is another session entry created under the session folder but there are very few tracepoints.

is it possible that the delay between the event happening and the session trace starting causes loss of events ?
If yes, is there a way around this  ?

here is my setup :

#lttng-status
Recording session session: [inactive]
    Trace output: /home/user/lttng-traces/session-20230523-113537

=== Domain: User space ===

Buffering scheme: per-user

Tracked process attributes
  Virtual process IDs:  all
  Virtual user IDs:     all
  Virtual group IDs:    all

Channels:
-------------
- user-channel: [enabled]

    Attributes:
      Event-loss mode:  discard
      Sub-buffer size:  16777216 bytes
      Sub-buffer count: 128
      Switch timer:     inactive
      Read timer:       inactive
      Monitor timer:    1000000 us
      Blocking timeout: 0 us
      Trace file count: 1 per stream
      Trace file size:  unlimited
      Output mode:      mmap

    Statistics:
      Discarded events: 0

    Recording event rules:
      trace_events* (type: tracepoint) [enabled]
      loop* (type: tracepoint) [enabled]

#lttng list-triggers

- name: iter-start
  owner uid: 1000
  condition: event rule matches
    rule: loop:iteration (type: user tracepoint, filter: iter==2 && start == 1)
    errors: none
  actions:
    start session `session`
      errors: none
  errors: none
- name: iter-stop
  owner uid: 1000
  condition: event rule matches
    rule: loop:iteration (type: user tracepoint, filter: iter==2 && start == 0)
    errors: none
  actions:
    stop session `session`
      errors: none
  errors: none


Thanks

Yehuda

[-- Attachment #1.2: Type: text/html, Size: 6525 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] delay between trigger and action

[Jérémie Galarneau via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 3494 bytes --]

Hi Yehuda,

The trigger mechanism is asynchroneous. The applications (or the kernel) won't block until the session daemon has processed the actions of the trigger.

As such, it is likely that you will lose events between the moment the trigger is triggered and when the session is started.

We have been considering implementing synchroneous start/stop actions, but no one has sponsored that work for the moment.

In your case, a work-around you could consider is to:

  *   setup a snapshot session with the same events
  *   set the trigger's actions to record a snapshot on that session, and start your existing session

In that setup, you would most likely see the events that led to the "start" (since they are captured in the in-memory buffers of the snapshot session), and you would capture the follow-up events in the on-disk session.

I hope that helps,
Jérémie

--
Jérémie Galarneau
EfficiOS Inc.
https://www.efficios.com
________________________________
From: lttng-dev <lttng-dev-bounces@lists.lttng.org> on behalf of Yitschak, Yehuda via lttng-dev <lttng-dev@lists.lttng.org>
Sent: May 23, 2023 08:17
To: lttng-dev@lists.lttng.org <lttng-dev@lists.lttng.org>
Subject: [lttng-dev] delay between trigger and action


Hi everyone

I am experimenting with enabling trace for a specific iteration of a loop in my application.
I created 2 trace points at the start and end of the loop which log the iteration number.
the loop itself creates thousands of trace points. It runs for ~5ms.

on lttng side I created 2 triggers to start and stop my session in case the loop tracepoint registers a specific iteration number.
the triggers seem to work since there is another session entry created under the session folder but there are very few tracepoints.

is it possible that the delay between the event happening and the session trace starting causes loss of events ?
If yes, is there a way around this  ?

here is my setup :

#lttng-status
Recording session session: [inactive]

    Trace output: /home/user/lttng-traces/session-20230523-113537



=== Domain: User space ===



Buffering scheme: per-user



Tracked process attributes

  Virtual process IDs:  all

  Virtual user IDs:     all

  Virtual group IDs:    all



Channels:

-------------

- user-channel: [enabled]



    Attributes:

      Event-loss mode:  discard

      Sub-buffer size:  16777216 bytes

      Sub-buffer count: 128

      Switch timer:     inactive

      Read timer:       inactive

      Monitor timer:    1000000 us

      Blocking timeout: 0 us

      Trace file count: 1 per stream

      Trace file size:  unlimited

      Output mode:      mmap



    Statistics:

      Discarded events: 0



    Recording event rules:

      trace_events* (type: tracepoint) [enabled]

      loop* (type: tracepoint) [enabled]

#lttng list-triggers

- name: iter-start

  owner uid: 1000

  condition: event rule matches

    rule: loop:iteration (type: user tracepoint, filter: iter==2 && start == 1)

    errors: none

  actions:

    start session `session`

      errors: none

  errors: none

- name: iter-stop

  owner uid: 1000

  condition: event rule matches

    rule: loop:iteration (type: user tracepoint, filter: iter==2 && start == 0)

    errors: none

  actions:

    stop session `session`

      errors: none

  errors: none


Thanks

Yehuda

[-- Attachment #1.2: Type: text/html, Size: 9212 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev