From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36B07C63798 for ; Fri, 27 Nov 2020 16:04:36 +0000 (UTC) Received: from lists.lttng.org (lists.lttng.org [167.114.26.123]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 49ED4208D5 for ; Fri, 27 Nov 2020 16:04:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.lttng.org header.i=@lists.lttng.org header.b="nR6dbhz1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 49ED4208D5 Authentication-Results: mail.kernel.org; dmarc=pass (p=none dis=none) header.from=lists.lttng.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lttng-dev-bounces@lists.lttng.org Received: from lists-lttng01.efficios.com (localhost [IPv6:::1]) by lists.lttng.org (Postfix) with ESMTP id 4CjKFY2nHDz1Q2T; Fri, 27 Nov 2020 11:04:33 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=lists.lttng.org; s=default; t=1606493074; bh=4vfGBh2bhbNNdbBF7Y6vOetyhWv8mjBzE6tFMfYV5XM=; h=Date:To:Cc:In-Reply-To:References:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=nR6dbhz1I2APciEw0eiBsIK1R4K51RIJmtDKragPaRQVnYlOeXsMI9If+tiwaDBC6 4MZ52P1WVNhxGPQP0Y+1NT+whFK4bBQpTrjN3DcTJ+bAWerZN5O8WHNsm+Tkit5L69 KLX6l8h0lFCikdByfJn4nHAV4p1BjThwFlougSg+fNZvuIasns3TSNR1kb88DhnO0z vJO6/dLNmsGC41oJ4lspTAMVnkey+0ytS5ziy+uStNGLNHo3GV8FNpJnBUJEZ0FRHy 6yQC1vn49fvfGCO84GvJcNBewvWtq3LfPSRjYbcGtvuFhmMw5Vbo3785J0lAWSo1M9 OP8wMMwJiWQZQ== Received: from mail.efficios.com (mail.efficios.com [167.114.26.124]) by lists.lttng.org (Postfix) with ESMTPS id 4CjKFW4GHWz1Pl3 for ; Fri, 27 Nov 2020 11:04:31 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 415C127C360 for ; Fri, 27 Nov 2020 11:04:25 -0500 (EST) Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id pJB04G2_6XAx; Fri, 27 Nov 2020 11:04:24 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 1B3A827C593; Fri, 27 Nov 2020 11:04:24 -0500 (EST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.efficios.com 1B3A827C593 X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (mail03.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id B7_0ArZQRytn; Fri, 27 Nov 2020 11:04:24 -0500 (EST) Received: from mail03.efficios.com (mail03.efficios.com [167.114.26.124]) by mail.efficios.com (Postfix) with ESMTP id 099C227C0F3; Fri, 27 Nov 2020 11:04:24 -0500 (EST) Date: Fri, 27 Nov 2020 11:04:23 -0500 (EST) To: =?utf-8?B?54aK5q+T5Y2O?= Cc: lttng-dev Message-ID: <828264314.65604.1606493063962.JavaMail.zimbra@efficios.com> In-Reply-To: <25e54ac4.4261.1760a553e88.Coremail.xiongyuhua@zju.edu.cn> References: <40a80cf.2b99.176086d97b1.Coremail.xiongyuhua@zju.edu.cn> <20201127140548.GB3947@joraj-alpa> <25e54ac4.4261.1760a553e88.Coremail.xiongyuhua@zju.edu.cn> MIME-Version: 1.0 X-Originating-IP: [167.114.26.124] X-Mailer: Zimbra 8.8.15_GA_3975 (ZimbraWebClient - GC86 (Linux)/8.8.15_GA_3975) Thread-Topic: Some confusion about cpu usage of the lttng-consumerd process Thread-Index: yp5QYAUpKnzdSMpykx0c2/cDegrSAg== Subject: Re: [lttng-dev] Some confusion about cpu usage of the lttng-consumerd process X-BeenThere: lttng-dev@lists.lttng.org X-Mailman-Version: 2.1.31 Precedence: list List-Id: LTTng development list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Jonathan Rajotte-Julien via lttng-dev Reply-To: Jonathan Rajotte-Julien Content-Type: multipart/mixed; boundary="===============1031602953658582660==" Errors-To: lttng-dev-bounces@lists.lttng.org Sender: "lttng-dev" --===============1031602953658582660== Content-Type: multipart/alternative; boundary="=_969e784f-864a-41e7-9b10-6df4f78292ab" --=_969e784f-864a-41e7-9b10-6df4f78292ab Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable > From: "=E7=86=8A=E6=AF=93=E5=8D=8E" > To: "Jonathan Rajotte-Julien" , > "lttng-dev" > Sent: Friday, November 27, 2020 10:32:07 AM > Subject: Re: Re: [lttng-dev] Some confusion about cpu usage of the > lttng-consumerd process > Hi=EF=BC=8CDear. Side note, you can remove the "Dear" here. ;)=20 > The test script was used to generate some common fileIO,netIO events. Please provide a complete code repository if possible. So that we can at le= ast have a baseline for reproduction.=20 > On all servers, the monitoring strategy I set up when I start lttng is th= e same, > monitoring all fileIO, netIO and some related system calls. > The following table records the amount of events generated by the test sc= ript > per minute, and one babeltrace record represents one event. For some reason the image does not load here. Please provide a text based a= lternative for this figure.=20 > The unit of the number is every ten thousand events per minute. And the n= umber > were read out after parsing by babeltrace. > In addition, the server1 is 1core4G, server2 is 2core8G, server3 is 4core= 16G, > server4 and server5 are 8core16G. > It can be seen that the average amount of data generated per minute on al= l > servers is roughly the same.However, the CPU usage of the lttng-consumerd > process behave differently on server4 and server5, as I mentioned in my l= ast > email. > In addition, the usage of cpu is recorded using the "top" command. > My test concluded that, while the same number of events collected, > lttng-consumerd process need to consume more cpu on the 8-core server. > I want to know why is this and what else information do you need? Well we also want to know why! You will understand that albeit we develop l= ttng we do not always have a quick and easy answer to all problems. Perform= ance related problem are always tricky.=20 And we also have to keep in mind that we do not necessarily optimize for lo= w-cpu usage on the lttng-consumerd side.=20 We have to take a look at what "work" scale with the number of CPU on the l= ttng-consumerd side. One such thing is the live timer which is fired on an = interval (default is 1s (1000000us)).=20 You could test this hypothesis by streaming the trace instead of using the = live feature.=20 lttng create --set-url ....=20 Cheers=20 > Looking forward to your reply. > thanks, > yuhua. > > -----=E5=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6----- > > =E5=8F=91=E4=BB=B6=E4=BA=BA: "Jonathan Rajotte-Julien" < jonathan.rajot= te-julien@efficios.com > > > =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: [ callto:2020-11-27 22 | 2020-11-= 27=C2=A022 ] :05:48 (=E6=98=9F=E6=9C=9F=E4=BA=94) > > =E6=94=B6=E4=BB=B6=E4=BA=BA: "=E7=86=8A=E6=AF=93=E5=8D=8E" < xiongyuhua= @zju.edu.cn > > > =E6=8A=84=E9=80=81: lttng-dev@lists.lttng.org >> =E4=B8=BB=E9=A2=98: Re: [lttng-dev] Some confusion about cpu usage of th= e lttng-consumerd > > process > > Hi, > > On Fri, Nov 27 , 2020 at 02:39:28PM +0800, =E7=86=8A=E6=AF=93=E5=8D=8E = via lttng-dev wrote: > > > Hi=EF=BC=8Cdear. >> > I have been using lttng to monitor my server these days=EF=BC=8Cbut I = found something > > > interesting. > > > The cpu usage of lttng varies with the number of cpu cores of the ser= ver. > > Which is a bit expected since more CPU means more "data" source from th= e point > > of view of lttng hence more "work" overall. >> > On the server, I create a tracing session in live mode, using "lttng c= reate > > > my-session --live". >> > Then,I Start the babeltrace2 and configure it to connect to the relay > > > daemon,using "--input-format=3Dlttng-live" mode. > > > I used 5 cloud servers=EF=BC=8C1core4G 2core8G 4core16G 8core16G 8cor= e16G. > > > And,the same test script was executed above to provide the same workl= oad. > > We would need the test script to have some context here of the workload= . > > > As we all know,lttng has 5 processes, > > > 1.lttng-runas --daemonize >> > 2.lttng-runas -k --consumerd-cmd-sock /var/run/lttng/kconsumerd/comman= d > > > --consumerd-err-sock /var/run/lttng/kconsumerd/error --group tracing > > Based on this you are performing kernel tracing. > > > 3.lttng-sessiond --daemonize > > > 4.lttng-relayd -L tcp://localhost:5344 >> > 5.lttng-consumerd -k --consumerd-cmd-sock /var/run/lttng/kconsumerd/co= mmand > > > --consumerd-err-sock /var/run/lttng/kconsumerd/error --group tracing >> > The CPU usage of the first four processes is below 2% on the 5 servers= ,but the > > > lttng-consumerd process is different. >> > On 1-core=E3=80=812-core=E3=80=814-core servers=EF=BC=8Cthe CPU usage = of the lttng-consumerd process is > > > below 2%. > > How is the cpu usage measured here? >> > But on two 8-core machines, the cpu usage of the lttng-consumerd proce= ss reached > > > 10% or more. > > Consumerd is responsible of "fetching" data from the ring buffers and "= saving" >> it either locally (trace on disk) or remotely (streaming/live session). = CPU > > usage >> should be a bit correlated with the event production rate. Did you have = a look > > at the > > number of events generated for a similar interval? >> > And,the cpu usage of the babeltrace process is not much different,just= the cpu >> > usage of the lttng-consumerd process varies with the number of cpu cor= es of the > > > server. > > > Why it is like this=EF=BC=9FHow should this phenomenon be analyzed? > > > Looking forward to your reply. > > > thanks, > > > yuhua > > > _______________________________________________ > > > lttng-dev mailing list > > > lttng-dev@lists.lttng.org >> > [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev | > > > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ] > > -- > > Jonathan Rajotte-Julien > > EfficiOS --=_969e784f-864a-41e7-9b10-6df4f78292ab Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

From: "=E7=86=8A=E6=AF=93=E5=8D=8E" <= ;xiongyuhua@zju.edu.cn>
To: "Jonathan Rajotte-Julien" <jona= than.rajotte-julien@efficios.com>, "lttng-dev" <lttng-dev@lists.lttng= .org>
Sent: Friday, November 27, 2020 10:32:07 AM
Subjec= t: Re: Re: [lttng-dev] Some confusion about cpu usage of the lttng-cons= umerd process
Hi=EF=BC=8CDe= ar.

Side note, you can remove the "De= ar" here. ;)



The test script was used to generate some common fileIO,netIO events.
=

Please provide a complete code repository = if possible. So that we can at least have a baseline for reproduction.



On all servers, the monitoring strategy I set up when I start lttng is the = same, monitoring all fileIO, netIO and some related system calls. 
The following table records the amount of events generated by the test scri= pt per minute, and one babeltrace record represents one event.

=093D""


= For some reason the image does not load here. Please provide a text based a= lternative for this figure.

=09The unit of the number is every ten thousand events per minute. And the = number were read out after parsing by babeltrace.
In addition, the server1 is 1core4G, server2 is 2core8G, server3 is 4core16= G, server4 and server5 are 8core16G.

It can be seen that the average amount of data generated per minute on all = servers is roughly the same.However, the CPU usage of the lttng-consumerd p= rocess behave differently on server4 and server5, as I mentioned in my last= email.

=09
In addition, the usage of cpu is recorded using the "top" command.

=09

=09My test concluded that, while the same number of events collected, lttng= -consumerd process need to consume more cpu on the 8-core server.

I want to know why is this and what else information do you need?

=

Well we also want to know why! You will un= derstand that albeit we develop lttng we do not always have a quick and eas= y answer to all problems. Performance related problem are always tricky.
And we also have to keep in mind that we do not necessarily optimiz= e for low-cpu usage on the lttng-consumerd side. 

We have to take a look at what "work" scale with = the number of CPU on the lttng-consumerd side. One such thing is the live t= imer which is fired on an interval (default is 1s (1000000us)).
<= br data-mce-bogus=3D"1">
You could test this hypothesis by stream= ing the trace instead of using the live feature.

lttng create --set-url ....

Cheers



Looking forward to your reply.
thanks,
yuhua.


> -----=E5=8E=9F=E5=A7=8B=E9=82=AE=E4=BB=B6-----
> =E5=8F=91=E4=BB=B6=E4=BA=BA: "Jonathan Rajotte-Julien"&= nbsp;<jonathan.rajotte-julien@efficios.com&g= t;
> =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2020-11-27 22:05:48 (=E6=98=9F=E6=9C=9F=E4=BA=94)
> =E6=94=B6=E4=BB=B6=E4=BA=BA: "=E7=86=8A=E6=AF=93=E5=8D=8E"&n= bsp;<xiongyuhua@zju.edu.cn>
> =E6=8A=84=E9=80=81: lttng-dev@lists.lttng.org=
> =E4=B8=BB=E9=A2=98: Re: [lttng-dev] Some conf= usion about cpu usage of the lttng-consumerd&= nbsp;process
> 
> Hi,
> 
> On Fri, Nov 27, 2020 a= t 02:39:28PM +0800, =E7=86=8A=E6=AF=93=E5=8D=8E via&nbs= p;lttng-dev wrote:
> > Hi=EF=BC=8Cdear.
> > 
> > I have been using lttng to&nbs= p;monitor my server these days=EF=BC=8Cbut I = found something interesting.
> > 
> > The cpu usage of lttng varies&= nbsp;with the number of cpu cores of the=  server.
> 
> Which is a bit expected since more&= nbsp;CPU means more "data" source from the&nb= sp;point
> of view of lttng hence more "work"&= nbsp;overall.
> 
> > 
> > On the server, I create a = ;tracing session in live mode, using "lttng&n= bsp;create my-session --live". 
> > 
> > Then,I Start the babeltrace2 and&nb= sp;configure it to connect to the relay = daemon,using "--input-format=3Dlttng-live" mode.
> > 
> > I used 5 cloud servers=EF=BC=8C1cor= e4G 2core8G 4core16G 8core16G 8core16G.
> > 
> > And,the same test script was e= xecuted above to provide the same workload. > 
> We would need the test script to&nb= sp;have some context here of the workload. > 
> > 
> > As we all know,lttng has 5&nbs= p;processes,
> > 
> > 1.lttng-runas    --daemonize
> > 
> > 2.lttng-runas      -k&nbs= p;--consumerd-cmd-sock /var/run/lttng/kconsumerd/command --consum= erd-err-sock /var/run/lttng/kconsumerd/error --group tracing=
> 
> Based on this you are performing ke= rnel tracing.
> 
> > 
> > 3.lttng-sessiond --daemonize
> > 
> > 4.lttng-relayd -L tcp://localhost:5344
> > 
> > 5.lttng-consumerd  -k --consumerd-cmd-so= ck /var/run/lttng/kconsumerd/command --consumerd-err-sock /v= ar/run/lttng/kconsumerd/error --group tracing
> > 
> > 
> > The CPU usage of the first&nbs= p;four processes is below 2% on the 5&nb= sp;servers,but the lttng-consumerd process is diff= erent.
> > 
> > On 1-core=E3=80=812-core=E3=80=814-core serve= rs=EF=BC=8Cthe CPU usage of the lttng-consumerd&nb= sp;process is below 2%.
> 
> How is the cpu usage measured here?=
> 
> > 
> > But on two 8-core machines, th= e cpu usage of the lttng-consumerd process&nb= sp;reached 10% or more.
> 
> Consumerd is responsible of "fetching" d= ata from the ring buffers and "saving"
> it either locally (trace on disk) o= r remotely (streaming/live session). CPU usage
> should be a bit correlated with the=  event production rate. Did you have a&n= bsp;look at the
> number of events generated for a si= milar interval?
> 
> > And,the cpu usage of the babel= trace process is not much different,just the&= nbsp;cpu usage of the lttng-consumerd process = ;varies with the number of cpu cores of&= nbsp;the server.
> > 
> > Why it is like this=EF=BC=9FHow&nbs= p;should this phenomenon be analyzed?
> > 
> > Looking forward to your reply.
> > 
> > thanks,
> > yuhua
> > 
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev@lists.lttng.org > > https://lists.lttng.org/cgi-bin/mailman= /listinfo/lttng-dev
> 
> 
> -- 
> Jonathan Rajotte-Julien
> EfficiOS


--=_969e784f-864a-41e7-9b10-6df4f78292ab-- --===============1031602953658582660== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev --===============1031602953658582660==--