Re: [lttng-dev] Some confusion about cpu usage of the lttng-consumerd process - Mathieu Desnoyers via lttng-dev

From: Mathieu Desnoyers via lttng-dev <lttng-dev@lists.lttng.org>
To: Jonathan Rajotte-Julien <jonathan.rajotte-julien@efficios.com>
Cc: 熊毓华 <xiongyuhua@zju.edu.cn>, lttng-dev <lttng-dev@lists.lttng.org>
Subject: Re: [lttng-dev] Some confusion about cpu usage of the lttng-consumerd process
Date: Fri, 27 Nov 2020 12:11:34 -0500 (EST)	[thread overview]
Message-ID: <556721467.65922.1606497094670.JavaMail.zimbra@efficios.com> (raw)
In-Reply-To: <828264314.65604.1606493063962.JavaMail.zimbra@efficios.com>

----- On Nov 27, 2020, at 11:04 AM, lttng-dev lttng-dev@lists.lttng.org wrote:

> Well we also want to know why! You will understand that albeit we develop lttng
> we do not always have a quick and easy answer to all problems. Performance
> related problem are always tricky.
> And we also have to keep in mind that we do not necessarily optimize for low-cpu
> usage on the lttng-consumerd side.

That being said, we did optimize for low-cpu usage of lttng-consumerd for use-cases
streaming to disk or to the network. However, the "live" mode was originally created
for use-cases where only a few events per second would be emitted, and no such
requirements were placed on performance. We can see today that its use has grown
much beyond the few events per seconds, but then in those use-cases the live mode
may not be the appropriate tool for the job then. We have introduced the "session
rotation" feature as a more efficient alternative to the live mode.

> We have to take a look at what "work" scale with the number of CPU on the
> lttng-consumerd side. One such thing is the live timer which is fired on an
> interval (default is 1s (1000000us)).

> You could test this hypothesis by streaming the trace instead of using the live
> feature.

> lttng create --set-url ....

Yes, I agree with Jonathan's recommendation: you should compare this cpu usage with
that of the streaming mode of lttng by *not* using the "--live" option when creating
the trace session. It will at least help identify whether consumerd also exhibits this
cpu usage increase with number of cores in streaming mode, or if it is an expected
additional overhead of periodically flushing more cpu buffers (because there are more
cores) caused by the live timer.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev