[lttng-dev] Discarded events

[lttng-dev] Discarded events

[Bala Gundeboina via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 1600 bytes --]

Hi,

I am Using LTTng in my project , I am recording a kernel session for long duration (eg more than 15minutes) I am enabling all the events for recording but some events discarding after stopping the session, if i run more than 5 minutes only some events are discarding. I have tried to change in the .ltttngrc file also to increase buffer-size.

lttng destroy
Destroying session `my-kernel-session`...
Session `my-kernel-session` destroyed
Warning: 94424667 events were discarded, please refer to the documentation on channel configuration

actually when i am running below command i am getting below warning i am thinking this is the issue because sometimes this warning is not coming that i didn't seen discarded events.  how to overcome this problem can you provide some detailed
lttng-sessiond --daemonize
Warning: Failed to produce a random seed using getrandom(), falling back to pseudo-random device seed generation which will block until its pool is initialized: Failed to get true random data using getre

Thanks & Regards
Bala Gundeboina
This message contains information that may be privileged or confidential and is the property of the KPIT Technologies Ltd. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorized to read, print, retain copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message. KPIT Technologies Ltd. does not accept any liability for virus infected mails.

[-- Attachment #1.2: Type: text/html, Size: 3818 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Discarded events

[Erica Bugden via lttng-dev]

On 2023-08-13 10:58, Bala Gundeboina via lttng-dev wrote:
> Hi,
> 
> I am Using LTTng in my project , I am recording a kernel session for 
> long duration (eg more than 15minutes) I am enabling all the events for 
> recording but some events discarding after stopping the session, if i 
> run more than 5 minutes only some events are discarding. I have tried to 
> change in the .ltttngrc file also to increase buffer-size.
> 
> lttng destroy
> Destroying session `my-kernel-session`...
> Session `my-kernel-session` destroyed
> Warning: 94424667 events were discarded, please refer to the 
> documentation on channel configuration
> 

Hello Bala,

Based on my understanding of the scenario, discarded events are to be 
expected. Rather than trying to avoid discarded events, I would 
recommend adjusting how much data is being collected. In most tracing 
use cases, it is not relevant to enable all kernel tracepoints for a 
long period of time (e.g. several minutes) as it can quickly generate an 
enormous amount of noisy data that is nearly impossible to sort through. 
As a reference, a busy 8 core machine could generate 100 MB/s of data.

Typically when trying to understand a problem in detail with tracing, 
you would start with a very small number of tracepoints enabled for a 
longer period of time (with the goal of developing a high-level 
understanding of when/where the problem is happening). Then as you 
iteratively narrow down when/where the problem happens you can gradually 
increase the number of relevant tracepoints without being overwhelmed 
with data.

A general rule of thumb is that if you're storing trace data for a long 
period of time (e.g. more than a minute) then very few tracepoints 
should be enabled (e.g. 2-5). Maximum tracing detail would typically 
only be used to trace for a couple seconds (or ideally less if you can 
automate starting and stopping tracing).

Here are some general references about tracing strategy:
  - Whether tracing is the appropriate approach: 
https://github.com/tuxology/tracevizlab/tree/master/labs/001-what-is-tracing#when-to-trace
  - Iterative investigation, selecting tracepoints: 
https://wiki.linuxfoundation.org/realtime/documentation/howto/debugging/debug-steps 
(Recommended sections: Isolate the source, Trace detail)

Hope this helps!

Best,
Erica

> actually when i am running below command i am getting below warning i am 
> thinking this is the issue because sometimes this warning is not coming 
> that i didn't seen discarded events.  how to overcome this problem can 
> you provide some detailed
> lttng-sessiond --daemonize
> Warning: Failed to produce a random seed using getrandom(), falling back 
> to pseudo-random device seed generation which will block until its pool 
> is initialized: Failed to get true random data using getre
> 

At first glance, this error seems unrelated to whether or not events are 
discarded.

> Thanks & Regards
> Bala Gundeboina
> This message contains information that may be privileged or confidential 
> and is the property of the KPIT Technologies Ltd. It is intended only 
> for the person to whom it is addressed. If you are not the intended 
> recipient, you are not authorized to read, print, retain copy, 
> disseminate, distribute, or use this message or any part thereof. If you 
> receive this message in error, please notify the sender immediately and 
> delete all copies of this message. KPIT Technologies Ltd. does not 
> accept any liability for virus infected mails.
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Discarded events

[Milian Wolff via lttng-dev]

On Sonntag, 13. August 2023 16:58:17 CEST Bala Gundeboina via lttng-dev wrote:
> Hi,
> 
> I am Using LTTng in my project , I am recording a kernel session for long
> duration (eg more than 15minutes) I am enabling all the events for
> recording but some events discarding after stopping the session, if i run
> more than 5 minutes only some events are discarding. I have tried to change
> in the .ltttngrc file also to increase buffer-size.

While not directly related to your question, I strongly encourage you to 
review the list of events and only enable those that are actually required for 
the specific issue you are investigating. Some kernel events are _extremely_ 
high frequency but have often no practical use for many (at least in my 
experience and for my own purposes). Examples for the latter are esp. the 
kmalloc tracepoints.

> lttng destroy
> Destroying session `my-kernel-session`...
> Session `my-kernel-session` destroyed
> Warning: 94424667 events were discarded, please refer to the documentation
> on channel configuration
> 
> actually when i am running below command i am getting below warning i am
> thinking this is the issue because sometimes this warning is not coming
> that i didn't seen discarded events.  how to overcome this problem can you
> provide some detailed lttng-sessiond --daemonize
> Warning: Failed to produce a random seed using getrandom(), falling back to
> pseudo-random device seed generation which will block until its pool is
> initialized: Failed to get true random data using getre
> 
> Thanks & Regards
> Bala Gundeboina
> This message contains information that may be privileged or confidential and
> is the property of the KPIT Technologies Ltd. It is intended only for the
> person to whom it is addressed. If you are not the intended recipient, you
> are not authorized to read, print, retain copy, disseminate, distribute, or
> use this message or any part thereof. If you receive this message in error,
> please notify the sender immediately and delete all copies of this message.
> KPIT Technologies Ltd. does not accept any liability for virus infected
> mails.


-- 
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Discarded events

[Erica Bugden via lttng-dev]

Hello Bala,

If you'd like more information about which event types (tracepoints) are 
practical for investigating which areas, the `ltting-utils` [1] project 
could be useful! The repository contains scripts that record LTTng 
traces. Each script is focused on a different investigation area (e.g. 
network, interrupts, disk, etc).

You could use the scripts themselves and/or read the `*.profile` files 
[2] to see the event types enabled by the different profiles. For 
example, in the disk analysis profile (disk.profile), the following 
event types are enabled:

     - block_rq_complete
     - block_rq_insert
     - block_rq_issue
     - block_bio_frontmerge

As Millian suggested, it is often best to only trace the event types you 
need for the type of issue you are investigating since this increases 
the accuracy of the data collected (less tracing overhead, fewer dropped 
events) and also makes the trace easier to understand (less data volume, 
less data noise).

Best,

Erica

----

LINKS

[1] lttng-utils: https://github.com/tahini/lttng-utils/tree/master
[2] Tracing Profiles: 
https://github.com/tahini/lttng-utils/tree/master/lttngutils/profiles

On 2023-08-15 01:24, Milian Wolff via lttng-dev wrote:
> On Sonntag, 13. August 2023 16:58:17 CEST Bala Gundeboina via lttng-dev wrote:
>> Hi,
>>
>> I am Using LTTng in my project , I am recording a kernel session for long
>> duration (eg more than 15minutes) I am enabling all the events for
>> recording but some events discarding after stopping the session, if i run
>> more than 5 minutes only some events are discarding. I have tried to change
>> in the .ltttngrc file also to increase buffer-size.
> 
> While not directly related to your question, I strongly encourage you to
> review the list of events and only enable those that are actually required for
> the specific issue you are investigating. Some kernel events are _extremely_
> high frequency but have often no practical use for many (at least in my
> experience and for my own purposes). Examples for the latter are esp. the
> kmalloc tracepoints.
> 
>> lttng destroy
>> Destroying session `my-kernel-session`...
>> Session `my-kernel-session` destroyed
>> Warning: 94424667 events were discarded, please refer to the documentation
>> on channel configuration
>>
>> actually when i am running below command i am getting below warning i am
>> thinking this is the issue because sometimes this warning is not coming
>> that i didn't seen discarded events.  how to overcome this problem can you
>> provide some detailed lttng-sessiond --daemonize
>> Warning: Failed to produce a random seed using getrandom(), falling back to
>> pseudo-random device seed generation which will block until its pool is
>> initialized: Failed to get true random data using getre
>>
>> Thanks & Regards
>> Bala Gundeboina
>> This message contains information that may be privileged or confidential and
>> is the property of the KPIT Technologies Ltd. It is intended only for the
>> person to whom it is addressed. If you are not the intended recipient, you
>> are not authorized to read, print, retain copy, disseminate, distribute, or
>> use this message or any part thereof. If you receive this message in error,
>> please notify the sender immediately and delete all copies of this message.
>> KPIT Technologies Ltd. does not accept any liability for virus infected
>> mails.
> 
> 
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev