From: Aaron Tomlin <atomlin@atomlin.com>
To: peterz@infradead.org, mingo@redhat.com, acme@kernel.org,
namhyung@kernel.org
Cc: mark.rutland@arm.com, alexander.shishkin@linux.intel.com,
jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com,
james.clark@linaro.org, howardchu95@gmail.com,
atomlin@atomlin.com, neelx@suse.com, chjohnst@mail.com,
sean@ashe.io, steve@abita.co, rishil1999@outlook.com,
linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: [PATCH v5 0/3] perf sched latency: Refine outputs, unit scaling, and histogram support
Date: Thu, 30 Jul 2026 14:54:13 -0400 [thread overview]
Message-ID: <20260730185416.97166-1-atomlin@atomlin.com> (raw)
This patch series improves 'perf sched latency' by fixing misleading empty
table output, introducing dynamic unit auto-scaling for latency and runtime
statistics, and adding latency histogram visualisation along with time-span
filtering.
Patch 1 fixes an issue where 'perf sched latency' would fall through and
return success (0) when perf_session__has_traces() failed due to missing
tracepoint events in perf.data. This resulted in empty header tables and
zeroed summary statistics being printed.
To support pipe mode streams:
- Missing callbacks (.attr, .tracing_data, .build_id, and .feature) are
registered in cmd_sched() so pipe header events are properly parsed
- The handlers array is promoted to file-scope latency_handlers[] and
evlist__set_tracepoints_handlers() is invoked dynamically inside
perf_sched__process_tracepoint_sample() when evsel->handler is NULL
- Trace presence checks are performed post-processing when handling
pipe data, while non-pipe files continue to abort early upfront.
Additionally, map_switch_event() is guarded against potential NULL
pointer dereferences under memory allocation failures
Patch 2 introduces dynamic auto-scaling for latency and runtime display
columns (Runtime, Avg delay, Max delay). Previously, all values were
unconditionally formatted in milliseconds (ms), making microsecond or
second-scale latencies hard to read. Columns are now scaled dynamically to
the most appropriate unit (ns, us, ms, s), column headers are updated, and
format specifiers are aligned character-for-character with the table
headers.
Patch 3 adds three new command-line options to 'perf sched latency':
--histogram (-H):
Displays an ASCII bar chart of CPU wait latencies between
snapshots
--hist-mode:
Configures the bucketing scheme to either logarithmic (log) or
100 us equal-width linear (linear) mode
--time:
Filters event processing to a specified [start,stop] time span
Changes since v4:
- Added the missing .feature callback to sched.tool in cmd_sched()
- Fixed memory leak by reusing uncompleted work atoms when sched_in events
are skipped or lost
- Promoted the handlers array to file-scope (i.e., latency_handlers[]) and
added a dynamic lookup via evlist__set_tracepoints_handlers() inside
perf_sched__process_tracepoint_sample() whenever a sample arrives with
an unattached handler (i.e., evsel->handler == NULL)
- Prevented double-counting wakeups (i.e., ignore sched:sched_wakeup)
in pipe mode and non-pipe modes
- Added missing trailing pipe in output_lat_thread's new auto-scaling
format string
- Stripped redundant prefix strings from each row's format string to
produce a clean, tabular output
- Link to v4: https://lore.kernel.org/lkml/20260729144451.38286-1-atomlin@atomlin.com/
Changes since v3:
- Registered Missing Callbacks. In perf_tool__init configuration inside
cmd_sched(), added the .attr, .tracing_data, and .build_id callbacks.
Without these callbacks, pipe mode drops header attributes entirely,
preventing tracepoints from being populated in session->evlist
- Introduced an explicit post-processing pipe check. This ensures that
pipe mode aborts correctly and does not produce superfluous empty
latency tables when no trace samples are available
- Prevented potential NULL pointer dereference. Added a NULL check for
thread__get_runtime() in map_switch_event() under memory allocation
failures
- Fixed column alignment. Modified format specifiers to align
character-for-character with header column widths across all rows
- Adjusted header label padding. Updated header label padding and column
width delimiters in perf_sched__lat() to match the exact field format
specifiers printed in output_lat_thread()
- Fixed swapper histogram inclusion. Excluded "swapper" (i.e.,
CPU-specific idle thread) from global_hist buckets
- Preserved task state machine across --time bounds. Moved time-window
filtering inside add_sched_in_event() to gate latency statistics
recording without breaking wakeup state tracking across time interval
boundaries
- Link to v3: https://lore.kernel.org/lkml/20260726032533.712462-1-atomlin@atomlin.com/
Changes since v2:
- Ensured vertical pipe separators align across all latency table columns
- Excluded "swapper" (idle thread) latency samples from global_hist
- Preserved task state machine transitions across --time boundaries
- Suppressed empty table headers, total lines, and histogram graphs when
no matching trace samples exist (e.g., when --time, --CPU, or --pids
exclude all samples), outputting "No matching trace samples found."
instead
- Linked to v2: https://lore.kernel.org/lkml/20260725173341.679782-1-atomlin@atomlin.com/
Changes since v1:
- Fixed integer overflow in latency_bucket() (Ian Rogers)
- Linked to v1: https://lore.kernel.org/lkml/20260724142901.634761-1-atomlin@atomlin.com/
Aaron Tomlin (3):
perf sched: Suppress latency table output when trace samples are
missing
perf sched latency: Auto-scale latency and runtime display units
perf sched latency: Add histogram and time interval options
tools/perf/Documentation/perf-sched.txt | 6 +
tools/perf/builtin-sched.c | 318 +++++++++++++++++++++---
2 files changed, 284 insertions(+), 40 deletions(-)
--
2.55.0
next reply other threads:[~2026-07-30 18:54 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-30 18:54 Aaron Tomlin [this message]
2026-07-30 18:54 ` [PATCH v5 1/3] perf sched: Suppress latency table output when trace samples are missing Aaron Tomlin
2026-07-30 18:54 ` [PATCH v5 2/3] perf sched latency: Auto-scale latency and runtime display units Aaron Tomlin
2026-07-30 18:54 ` [PATCH v5 3/3] perf sched latency: Add histogram and time interval options Aaron Tomlin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260730185416.97166-1-atomlin@atomlin.com \
--to=atomlin@atomlin.com \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=chjohnst@mail.com \
--cc=howardchu95@gmail.com \
--cc=irogers@google.com \
--cc=james.clark@linaro.org \
--cc=jolsa@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=neelx@suse.com \
--cc=peterz@infradead.org \
--cc=rishil1999@outlook.com \
--cc=sean@ashe.io \
--cc=steve@abita.co \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox