linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Ingo Molnar <mingo@kernel.org>
To: kan.liang@intel.com
Cc: peterz@infradead.org, mingo@redhat.com, acme@kernel.org,
	linux-kernel@vger.kernel.org, alexander.shishkin@linux.intel.com,
	tglx@linutronix.de, namhyung@kernel.org, jolsa@kernel.org,
	adrian.hunter@intel.com, wangnan0@huawei.com,
	mark.rutland@arm.com, andi@firstfloor.org
Subject: Re: [PATCH 00/14] export perf overheads information
Date: Thu, 24 Nov 2016 05:27:02 +0100	[thread overview]
Message-ID: <20161124042702.GA10321@gmail.com> (raw)
In-Reply-To: <1479894292-16277-1-git-send-email-kan.liang@intel.com>


* kan.liang@intel.com <kan.liang@intel.com> wrote:

> From: Kan Liang <kan.liang@intel.com>
> 
> Profiling brings additional overhead. High overhead may impacts the
> behavior of the profiling object, impacts the accuracy of the
> profiling result, and even hang the system.
> Currently, perf has dynamic interrupt throttle mechanism to lower the
> sample rate and overhead. But it has limitations.
>  - The mechanism only focus in the overhead from NMI. However, there
>    are other parts which bring big overhead. E.g, multiplexing.
>  - The hint from the mechanism doesn't work on fixed period.
>  - The system changes which caused by the mechanism are not recorded
>    in the perf.data. Users have no idea about the overhead and its
>    impact.
> Acctually, any passive ways like dynamic interrupt throttle mechanism
> are only palliative. The best way is to export overheads information,
> provide more hints, and help the users design more proper perf command.
> 
> According to our test, there are four parts which can bring big overhead.
> They include NMI handler, multiplexing handler, iterate side-band events,
> and write data in file. Two new perf record type PERF_RECORD_OVERHEAD and
> PERF_RECORD_USER_OVERHEAD are introduced to record the overhead
> information in kernel and user space respectively.
> The overhead information is the system per-CPU overhead, not per-event
> overhead. The implementation takes advantage of the existing event log
> mechanism.
> To reduce the additional overhead from logging overhead information, the
> overhead information only be output when the event is going to be
> disabled or task is scheduling out.
> 
> In perf report, the overhead will be checked automatically. If the
> overhead rate is larger than 10%. A warning will be displayed.
> A new option is also introduced to display detial per-CPU overhead
> information.
> 
> Current implementation only include four overhead sources. There could be
> more in other parts. The new overhead source can be easily added as a
> new type.

Please include sample output of the new instrumentation!

Not even the tooling patches show any of the output, nor is it clear anywhere what 
kind of 'overhead' measurement it is, what the units are, what the metrics are, 
how users can _use_ this information, etc.

This is totally inadequate description.

Thanks,

	Ingo

      parent reply	other threads:[~2016-11-24  4:27 UTC|newest]

Thread overview: 58+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-11-23  9:44 [PATCH 00/14] export perf overheads information kan.liang
2016-11-23  9:44 ` [PATCH 01/14] perf/x86: Introduce PERF_RECORD_OVERHEAD kan.liang
2016-11-23 20:11   ` Peter Zijlstra
2016-11-23 20:13   ` Peter Zijlstra
2016-11-23 23:41   ` Jiri Olsa
2016-11-24 13:45     ` Liang, Kan
2016-11-24 13:50       ` Peter Zijlstra
2016-11-24 13:56         ` Liang, Kan
2016-11-24 14:27           ` Jiri Olsa
2016-11-24 14:39             ` Liang, Kan
2016-11-24 14:47               ` Jiri Olsa
2016-11-24 18:28         ` Andi Kleen
2016-11-24 18:58           ` Peter Zijlstra
2016-11-24 19:02             ` Andi Kleen
2016-11-24 19:08               ` Peter Zijlstra
2016-11-23  9:44 ` [PATCH 02/14] perf/x86: output NMI overhead kan.liang
2016-11-23 20:06   ` Peter Zijlstra
2016-11-24 16:19   ` Mark Rutland
2016-11-24 19:02     ` Peter Zijlstra
2016-11-24 19:40     ` Liang, Kan
2016-11-24 23:26       ` Namhyung Kim
2016-11-23  9:44 ` [PATCH 03/14] perf/x86: output multiplexing overhead kan.liang
2016-11-23 20:06   ` Peter Zijlstra
2016-11-23 20:09     ` Liang, Kan
2016-11-23  9:44 ` [PATCH 04/14] perf/x86: output side-band events overhead kan.liang
2016-11-23 20:06   ` Peter Zijlstra
2016-11-24 16:21   ` Mark Rutland
2016-11-24 19:40     ` Liang, Kan
2016-11-23  9:44 ` [PATCH 05/14] perf tools: handle PERF_RECORD_OVERHEAD record type kan.liang
2016-11-23 22:35   ` Jiri Olsa
2016-11-23 22:58     ` Jiri Olsa
2016-11-23  9:44 ` [PATCH 06/14] perf tools: show NMI overhead kan.liang
2016-11-23 22:51   ` Jiri Olsa
2016-11-24 13:37     ` Liang, Kan
2016-11-24 15:27       ` Jiri Olsa
2016-11-24 23:20         ` Namhyung Kim
2016-11-24 23:45           ` Jiri Olsa
2016-11-25  0:21         ` Andi Kleen
2016-11-23 22:52   ` Jiri Olsa
2016-11-23 22:52   ` Jiri Olsa
2016-11-23  9:44 ` [PATCH 07/14] perf tools: show multiplexing overhead kan.liang
2016-11-23  9:44 ` [PATCH 08/14] perf tools: show side-band events overhead kan.liang
2016-11-23  9:44 ` [PATCH 09/14] perf tools: make get_nsecs visible for buildin files kan.liang
2016-11-23  9:44 ` [PATCH 10/14] perf tools: introduce PERF_RECORD_USER_OVERHEAD kan.liang
2016-11-23  9:44 ` [PATCH 11/14] perf tools: record write data overhead kan.liang
2016-11-23 23:02   ` Jiri Olsa
2016-11-23 23:06   ` Jiri Olsa
2016-11-23  9:44 ` [PATCH 12/14] perf tools: record elapsed time kan.liang
2016-11-23  9:44 ` [PATCH 13/14] perf tools: warn on high overhead kan.liang
2016-11-23 20:25   ` Andi Kleen
2016-11-23 22:03     ` Liang, Kan
2016-11-25 20:42       ` Andi Kleen
2016-11-23  9:44 ` [PATCH 14/14] perf script: show overhead events kan.liang
2016-11-23 23:17   ` Jiri Olsa
2016-11-23 23:18   ` Jiri Olsa
2016-11-23 23:19   ` Jiri Olsa
2016-11-23 23:22   ` Jiri Olsa
2016-11-24  4:27 ` Ingo Molnar [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20161124042702.GA10321@gmail.com \
    --to=mingo@kernel.org \
    --cc=acme@kernel.org \
    --cc=adrian.hunter@intel.com \
    --cc=alexander.shishkin@linux.intel.com \
    --cc=andi@firstfloor.org \
    --cc=jolsa@kernel.org \
    --cc=kan.liang@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mark.rutland@arm.com \
    --cc=mingo@redhat.com \
    --cc=namhyung@kernel.org \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    --cc=wangnan0@huawei.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).