From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.lttng.org (lists.lttng.org [167.114.26.123]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C02CCEE14D8 for ; Sun, 10 Sep 2023 14:18:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1694355519; bh=pnz3qnOBY69xGNZGz5H0DEzNQkHO0SiBkdX9UzaqpJ8=; h=To:Date:References:In-Reply-To:Subject:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=ROymS+4PIKwk264eZs720hy1b5BZ4jnwgSLMshcTF298QgMt14fAflckYX/pX9Khm 5Nt1AjaN/O79o1n0nwx7M6j7RFIX9Qnqq8LVENUxCJ/TLbpR0SpXBA8H4kw2ERtWcX DdTicwJK+XXy8bSzess4kc8Ndk6EHqTHRdo/5W3/cSn+iHRpQxp2fpP9y+TxI1FleS FVo5okBMzw2UG31OX/q9325nRs5Ho7jiGkuRP44ydWSPpAlxy+wu82RH/knlkoMb7K XogvoqQVHTaiHf5aqDirZjBcyjkvruQBWG3Z5S37sNpS+rSupLfTdjMvy7vCrDp0xq 9Ciwwn3NSWK8A== Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4RkBly4T50z2XS4; Sun, 10 Sep 2023 10:18:38 -0400 (EDT) Received: from smtp-fw-52003.amazon.com (smtp-fw-52003.amazon.com [52.119.213.152]) by lists.lttng.org (Postfix) with ESMTPS id 4RkBlx2VGCz2XS2 for ; Sun, 10 Sep 2023 10:18:37 -0400 (EDT) X-IronPort-AV: E=Sophos;i="6.02,241,1688428800"; d="scan'208,217";a="607047031" Thread-Topic: [lttng-dev] Profiling LTTng tracepoint latency on different arm platforms Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-iad-1a-m6i4x-bbc6e425.us-east-1.amazon.com) ([10.43.8.6]) by smtp-border-fw-52003.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Sep 2023 14:18:35 +0000 Received: from EX19D011EUA003.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-iad-1a-m6i4x-bbc6e425.us-east-1.amazon.com (Postfix) with ESMTPS id AECB48069E; Sun, 10 Sep 2023 14:18:32 +0000 (UTC) Received: from EX19D007EUA001.ant.amazon.com (10.252.50.133) by EX19D011EUA003.ant.amazon.com (10.252.50.178) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.37; Sun, 10 Sep 2023 14:18:32 +0000 Received: from EX19D008EUC001.ant.amazon.com (10.252.51.165) by EX19D007EUA001.ant.amazon.com (10.252.50.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.37; Sun, 10 Sep 2023 14:18:31 +0000 Received: from EX19D008EUC001.ant.amazon.com ([fe80::9611:c62b:a7ba:aee1]) by EX19D008EUC001.ant.amazon.com ([fe80::9611:c62b:a7ba:aee1%3]) with mapi id 15.02.1118.037; Sun, 10 Sep 2023 14:18:31 +0000 To: "Yitschak, Yehuda" , Mathieu Desnoyers , "lttng-dev@lists.lttng.org" Thread-Index: AQHZo2HEplRT2zY9N0ijivZSO5DOOK+TvfwAgAA+SACAAMJ3gIAAiEyAgAAJk4CAf0qKPw== Date: Sun, 10 Sep 2023 14:18:31 +0000 Message-ID: References: <6e06b605ec9f446da5dc8948c8621518@amazon.com> <63288a1f-59b4-80f9-2a26-bc252ce9cd1f@efficios.com> <900fe19a-5642-9124-4d83-0716e0d11a22@efficios.com> , <618d035e059840d3a32b75633aa378f5@amazon.com> In-Reply-To: <618d035e059840d3a32b75633aa378f5@amazon.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.1.213.27] MIME-Version: 1.0 Subject: Re: [lttng-dev] Profiling LTTng tracepoint latency on different arm platforms X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: "Mousa, Anas via lttng-dev" Reply-To: "Mousa, Anas" Content-Type: multipart/mixed; boundary="===============6824250047945973309==" Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" --===============6824250047945973309== Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_e658fc7033fd4a268dc9890093de6807amazoncom_" --_000_e658fc7033fd4a268dc9890093de6807amazoncom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hey Mathieu, We see that upon recording a tracepoint, there are multiple stages of reser= ve-commit-write, where atomics and shared memory accesses take up a big part of the recordin= g time, we're wondering, is there a "light-mode" of recording a tracepoint involvin= g less logic or a mode which can potentially have lower latency? Also, are there any recent docs to share regarding tracepoint latency? Regards, Anas. ________________________________ From: Yitschak, Yehuda Sent: Wednesday, June 21, 2023 5:21:35 PM To: Mathieu Desnoyers; Mousa, Anas; lttng-dev@lists.lttng.org Subject: RE: [EXTERNAL][lttng-dev] Profiling LTTng tracepoint latency on di= fferent arm platforms > On 6/21/23 01:39, Yitschak, Yehuda wrote: > >> On 6/20/23 10:20, Mathieu Desnoyers via lttng-dev wrote: > >>> On 6/20/23 06:27, Mousa, Anas via lttng-dev wrote: > >>>> Hello, > >>> > >>>> > >>>> > >> > Arethereanysuggestionstorootcausethehighlatencyandpotentiallyimprovei > >> to > >> n*platform****1*? > >>>> > >>>> Thanks and best regards, > >>>> > >>>> Anas. > >>>> > >>> > >>> I recommend using "perf" when tracing with the sample program in a > >>> loop to figure out the hot spots. With that information on the "fast" > >>> and "slow" system, we might be able to figure out what differs. > >>> > >>> Also, comparing the kernel configurations of the two systems can help= . > >>> Also comparing the glibc versions of the two systems would be relevan= t. > >>> > >> > >> Also make sure you benchmark the lttng "snapshot" mode [1] to make > >> sure you don't run into a situation where the disk/network I/O > >> throughput cannot cope with the generated event throughput, thus > >> causing the ring buffer to discard events. This would therefore > >> "speed up" tracing from the application perspective because > >> discarding an event is faster than writing it to a ring buffer. > > > > You mean we should avoid the "discard" loss mode and use "overwrite" > loss mode since discard mode can fake fast performance ? > > Yes. In addition to use "overwrite-when-buffer-full" mode, the "snapshot" > session also ensures that no consumer daemon extracts the trace data > (unless an explicit snapshot record is performed), which allows comparing > the ring buffer producer performance with minimal noise. > > If you really want to benchmark the discard-when-buffer-full mode and the > the consumer daemon I/O behavior, then you need to take into account > event discarded counts and the actual trace data size that was written to > disk. Since you mentioned this, is there any "stat" command which lists events su= ch as discards and disk writes, etc ? I looked this up in the past but couldn't find anything > > Thanks, > > Mathieu > > > > >> > >> Thanks, > >> > >> Mathieu > >> > >> [1] https://lttng.org/docs/v2.13/#doc-taking-a-snapshot > >> > >>> Thanks, > >>> > >>> Mathieu > >>> > >>> > >> > >> -- > >> Mathieu Desnoyers > >> EfficiOS Inc. > >> https://www.efficios.com > >> > >> _______________________________________________ > >> lttng-dev mailing list > >> lttng-dev@lists.lttng.org > >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > -- > Mathieu Desnoyers > EfficiOS Inc. > https://www.efficios.com --_000_e658fc7033fd4a268dc9890093de6807amazoncom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hey Mathieu,

We see that upon recording a tracepoint, there are multiple stages of reser= ve-commit-write,

where atomics and shared memory accesses take up a big part of the recordin= g time,

we're wondering, is there a "light-mode" of recording a tracepoin= t involving less logic or

a mode which can potentially have lower latency?


Also, are there any recent docs to share regarding tracepoint lat= ency?


Regards,

Anas.


From: Yitschak, Yehuda Sent: Wednesday, June 21, 2023 5:21:35 PM
To: Mathieu Desnoyers; Mousa, Anas; lttng-dev@lists.lttng.org
Subject: RE: [EXTERNAL][lttng-dev] Profiling LTTng tracepoint latenc= y on different arm platforms
 

> On 6/21/23 01:39, Yitschak, Yehuda wrote:
> >> On 6/20/23 10:20, Mathieu Desnoyers via lttng-dev wrote:
> >>> On 6/20/23 06:27, Mousa, Anas via lttng-dev wrote:
> >>>> Hello,
> >>>
> >>>>
> >>>>
> >>
> Arethereanysuggestionstorootcausethehighlatencyandpotentiallyimprovei<= br> > >> to
> >> n*platform****1*?
> >>>>
> >>>> Thanks and best regards,
> >>>>
> >>>> Anas.
> >>>>
> >>>
> >>> I recommend using "perf" when tracing with the = sample program in a
> >>> loop to figure out the hot spots. With that information o= n the "fast"
> >>> and "slow" system, we might be able to figure o= ut what differs.
> >>>
> >>> Also, comparing the kernel configurations of the two syst= ems can help.
> >>> Also comparing the glibc versions of the two systems woul= d be relevant.
> >>>
> >>
> >> Also make sure you benchmark the lttng "snapshot" m= ode [1] to make
> >> sure you don't run into a situation where the disk/network I/= O
> >> throughput cannot cope with the generated event throughput, t= hus
> >> causing the ring buffer to discard events. This would therefo= re
> >> "speed up" tracing from the application perspective= because
> >> discarding an event is faster than writing it to a ring buffe= r.
> >
> > You mean we should avoid the "discard" loss mode and us= e "overwrite"
> loss mode since discard mode can fake fast performance ?
>
> Yes. In addition to use "overwrite-when-buffer-full" mode, t= he "snapshot"
> session also ensures that no consumer daemon extracts the trace data > (unless an explicit snapshot record is performed), which allows compar= ing
> the ring buffer producer performance with minimal noise.
>
> If you really want to benchmark the discard-when-buffer-full mode and = the
> the consumer daemon I/O behavior, then you need to take into account > event discarded counts and the actual trace data size that was written= to
> disk.

Since you mentioned this, is there any "stat" command which lists= events such as discards and disk writes, etc ?
I looked this up in the past but couldn't find anything

>
> Thanks,
>
> Mathieu
>
> >
> >>
> >> Thanks,
> >>
> >> Mathieu
> >>
> >> [1] https://lttng.org/docs/v2.13/#doc-taking-a-snapshot
> >>
> >>> Thanks,
> >>>
> >>> Mathieu
> >>>
> >>>
> >>
> >> --
> >> Mathieu Desnoyers
> >> EfficiOS Inc.
> >> https://www.efficios.com=
> >>
> >> _______________________________________________
> >> lttng-dev mailing list
> >> lttng-dev@lists.lttng.org
> >> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev >
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com

 --_000_e658fc7033fd4a268dc9890093de6807amazoncom_-- --===============6824250047945973309== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============6824250047945973309==--