[PATCH v5 0/8] perf report: Add latency and parallelism profiling

Dmitry Vyukov
2025-02-05T16:28:27Z

There are two notions of time: wall-clock time and CPU time.
For a single-threaded program, or a program running on a single-core
machine, these notions are the same. However, for a multi-threaded/
multi-process program running on a multi-core machine, these notions are
significantly different. Each second of wall-clock time we have
number-of-cores seconds of CPU time.

Currently perf only allows to profile CPU time. Perf (and all other
existing profilers to the be best of my knowledge) does not allow to
profile wall-clock time.

Optimizing CPU overhead is useful to improve 'throughput', while
optimizing wall-clock overhead is useful to improve 'latency'.
These profiles are complementary and are not interchangeable.
Examples of where latency profile is needed:
- optimzing build latency
- optimizing server request latency
- optimizing ML training/inference latency
- optimizing running time of any command line program

CPU profile is useless for these use cases at best (if a user understands
the difference), or misleading at worst (if a user tries to use a wrong
profile for a job).

This series add latency and parallelization profiling.
See the added documentation and flags descriptions for details.

Brief outline of the implementation:
- add context switch collection during record
- calculate number of threads running on CPUs (parallelism level)
during report
- divide each sample weight by the parallelism level
This effectively models that we were taking 1 sample per unit of
wall-clock time.

We still default to the CPU profile, so it's up to users to learn
about the second profiling mode and use it when appropriate.

Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Ian Rogers <irogers@google.com>
Cc: linux-perf-users@vger.kernel.org
Cc: linux-kernel@vger.kernel.org

Changes in v5:
- fixed formatting of latency field in --stdout mode
- added description of --latency flag in Documentation flags

Changes in v4:
- added "Shrink struct hist_entry size" commit
- rebased to perf-tools-next HEAD

Changes in v3:
- rebase and split into patches
- rename 'wallclock' to 'latency' everywhere
- don't enable latency profiling by default,
instead add record/report --latency flag

Dmitry Vyukov (8):
perf report: Add machine parallelism
perf report: Add parallelism sort key
perf report: Switch filtered from u8 to u16
perf report: Add parallelism filter
perf report: Add latency output field
perf report: Add --latency flag
perf report: Add latency and parallelism profiling documentation
perf hist: Shrink struct hist_entry size

.../callchain-overhead-calculation.txt | 5 +-
.../cpu-and-latency-overheads.txt | 85 ++++++++++++++
tools/perf/Documentation/perf-record.txt | 4 +
tools/perf/Documentation/perf-report.txt | 54 ++++++---
tools/perf/Documentation/tips.txt | 3 +
tools/perf/builtin-record.c | 20 ++++
tools/perf/builtin-report.c | 39 +++++++
tools/perf/ui/browsers/hists.c | 27 +++--
tools/perf/ui/hist.c | 104 ++++++++++++------
tools/perf/util/addr_location.c | 1 +
tools/perf/util/addr_location.h | 7 +-
tools/perf/util/event.c | 11 ++
tools/perf/util/events_stats.h | 2 +
tools/perf/util/hist.c | 90 ++++++++++++---
tools/perf/util/hist.h | 32 +++++-
tools/perf/util/machine.c | 7 ++
tools/perf/util/machine.h | 6 +
tools/perf/util/sample.h | 2 +-
tools/perf/util/session.c | 12 ++
tools/perf/util/session.h | 1 +
tools/perf/util/sort.c | 69 +++++++++++-
tools/perf/util/sort.h | 3 +-
tools/perf/util/symbol.c | 34 ++++++
tools/perf/util/symbol_conf.h | 8 +-
24 files changed, 531 insertions(+), 95 deletions(-)
create mode 100644 tools/perf/Documentation/cpu-and-latency-overheads.txt

base-commit: 8ce0d2da14d3fb62844dd0e95982c194326b1a5f
--
2.48.1.362.g079036d154-goog