From: Steven Rostedt <rostedt@kernel.org>
To: linux-kernel@vger.kernel.org
Cc: Tomas Glozar <tglozar@redhat.com>, John Kacur <jkacur@redhat.com>,
Luis Goncalves <lgoncalv@redhat.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Chang Yin <cyin@redhat.com>,
Costa Shulyupin <costa.shul@redhat.com>,
Crystal Wood <crwood@redhat.com>,
Gabriele Monaco <gmonaco@redhat.com>
Subject: [for-next][PATCH 1/9] rtla/timerlat: Introduce enum timerlat_tracing_mode
Date: Mon, 21 Jul 2025 19:12:05 -0400 [thread overview]
Message-ID: <20250721231222.293482311@kernel.org> (raw)
In-Reply-To: 20250721231204.100737734@kernel.org
From: Tomas Glozar <tglozar@redhat.com>
After the introduction of BPF-based sample collection, rtla-timerlat
effectively runs in one of three modes:
- Pure BPF mode, with tracefs only being used to set up the timerlat
tracer. Sample processing and stop on threshold are handled by BPF.
- tracefs mode. BPF is unsupported or kernel is lacking the necessary
trace event (osnoise:timerlat_sample). Stop on theshold is handled by
timerlat tracer stopping tracing in all instances.
- BPF/tracefs mixed mode - BPF is used for sample collection for top or
histogram, tracefs is used for trace output and/or auto-analysis. Stop
on threshold is handled both through BPF program, which stops sample
collection for top/histogram and wakes up rtla, and by timerlat
tracer, which stops tracing for trace output/auto-analysis instances.
Add enum timerlat_tracing_mode, with three values:
- TRACING_MODE_BPF
- TRACING_MODE_TRACEFS
- TRACING_MODE_MIXED
Those represent the modes described above. A field of this type is added
to struct timerlat_params, named "mode", replacing the no_bpf variable.
params->mode is set in timerlat_{top,hist}_parse_args to
TRACING_MODE_BPF or TRACING_MODE_MIXED based on whether trace output
and/or auto-analysis is requested. timerlat_{top,hist}_main then checks
if BPF is not unavailable or disabled, in that case, it sets
params->mode to TRACING_MODE_TRACEFS.
A condition is added to timerlat_apply_config that skips setting
timerlat tracer thresholds if params->mode is TRACING_MODE_BPF (those
are unnecessary, since they only turn off tracing, which is already
turned off in that case, since BPF is used to collect samples).
Cc: John Kacur <jkacur@redhat.com>
Cc: Luis Goncalves <lgoncalv@redhat.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Chang Yin <cyin@redhat.com>
Cc: Costa Shulyupin <costa.shul@redhat.com>
Cc: Crystal Wood <crwood@redhat.com>
Cc: Gabriele Monaco <gmonaco@redhat.com>
Link: https://lore.kernel.org/20250626123405.1496931-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
---
tools/tracing/rtla/src/timerlat.c | 24 +++++++----
tools/tracing/rtla/src/timerlat.h | 18 ++++++++
tools/tracing/rtla/src/timerlat_hist.c | 51 +++++++++++++----------
tools/tracing/rtla/src/timerlat_top.c | 57 +++++++++++++++-----------
4 files changed, 97 insertions(+), 53 deletions(-)
diff --git a/tools/tracing/rtla/src/timerlat.c b/tools/tracing/rtla/src/timerlat.c
index c29e2ba2d7d8..63d6d43eafff 100644
--- a/tools/tracing/rtla/src/timerlat.c
+++ b/tools/tracing/rtla/src/timerlat.c
@@ -40,16 +40,22 @@ timerlat_apply_config(struct osnoise_tool *tool, struct timerlat_params *params)
CPU_SET(i, ¶ms->monitored_cpus);
}
- retval = osnoise_set_stop_us(tool->context, params->stop_us);
- if (retval) {
- err_msg("Failed to set stop us\n");
- goto out_err;
- }
+ if (params->mode != TRACING_MODE_BPF) {
+ /*
+ * In tracefs and mixed mode, timerlat tracer handles stopping
+ * on threshold
+ */
+ retval = osnoise_set_stop_us(tool->context, params->stop_us);
+ if (retval) {
+ err_msg("Failed to set stop us\n");
+ goto out_err;
+ }
- retval = osnoise_set_stop_total_us(tool->context, params->stop_total_us);
- if (retval) {
- err_msg("Failed to set stop total us\n");
- goto out_err;
+ retval = osnoise_set_stop_total_us(tool->context, params->stop_total_us);
+ if (retval) {
+ err_msg("Failed to set stop total us\n");
+ goto out_err;
+ }
}
diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/timerlat.h
index 73045aef23fa..e0a553545d03 100644
--- a/tools/tracing/rtla/src/timerlat.h
+++ b/tools/tracing/rtla/src/timerlat.h
@@ -1,6 +1,23 @@
// SPDX-License-Identifier: GPL-2.0
#include "osnoise.h"
+/*
+ * Define timerlat tracing mode.
+ *
+ * There are three tracing modes:
+ * - tracefs-only, used when BPF is unavailable.
+ * - BPF-only, used when BPF is available and neither trace saving nor
+ * auto-analysis are enabled.
+ * - mixed mode, used when BPF is available and either trace saving or
+ * auto-analysis is enabled (which rely on sample collection through
+ * tracefs).
+ */
+enum timerlat_tracing_mode {
+ TRACING_MODE_BPF,
+ TRACING_MODE_TRACEFS,
+ TRACING_MODE_MIXED,
+};
+
struct timerlat_params {
/* Common params */
char *cpus;
@@ -30,6 +47,7 @@ struct timerlat_params {
cpu_set_t hk_cpu_set;
struct sched_attr sched_param;
struct trace_events *events;
+ enum timerlat_tracing_mode mode;
union {
struct {
/* top only */
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index 36d2294c963d..6cf260e8553b 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -802,6 +802,9 @@ static struct timerlat_params
params->bucket_size = 1;
params->entries = 256;
+ /* default to BPF mode */
+ params->mode = TRACING_MODE_BPF;
+
while (1) {
static struct option long_options[] = {
{"auto", required_argument, 0, 'a'},
@@ -1054,6 +1057,13 @@ static struct timerlat_params
if (params->kernel_workload && params->user_workload)
timerlat_hist_usage("--kernel-threads and --user-threads are mutually exclusive!");
+ /*
+ * If auto-analysis or trace output is enabled, switch from BPF mode to
+ * mixed mode
+ */
+ if (params->mode == TRACING_MODE_BPF && params->trace_output && !params->no_aa)
+ params->mode = TRACING_MODE_MIXED;
+
return params;
}
@@ -1149,7 +1159,6 @@ int timerlat_hist_main(int argc, char *argv[])
pthread_t timerlat_u;
int retval;
int nr_cpus, i;
- bool no_bpf = false;
params = timerlat_hist_parse_args(argc, argv);
if (!params)
@@ -1161,12 +1170,6 @@ int timerlat_hist_main(int argc, char *argv[])
goto out_exit;
}
- retval = timerlat_hist_apply_config(tool, params);
- if (retval) {
- err_msg("Could not apply config\n");
- goto out_free;
- }
-
trace = &tool->trace;
/*
* Save trace instance into global variable so that SIGINT can stop
@@ -1175,24 +1178,30 @@ int timerlat_hist_main(int argc, char *argv[])
*/
hist_inst = trace;
+ /*
+ * Try to enable BPF, unless disabled explicitly.
+ * If BPF enablement fails, fall back to tracefs mode.
+ */
if (getenv("RTLA_NO_BPF") && strncmp(getenv("RTLA_NO_BPF"), "1", 2) == 0) {
debug_msg("RTLA_NO_BPF set, disabling BPF\n");
- no_bpf = true;
- }
-
- if (!no_bpf && !tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sample")) {
+ params->mode = TRACING_MODE_TRACEFS;
+ } else if (!tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sample")) {
debug_msg("osnoise:timerlat_sample missing, disabling BPF\n");
- no_bpf = true;
- }
-
- if (!no_bpf) {
+ params->mode = TRACING_MODE_TRACEFS;
+ } else {
retval = timerlat_bpf_init(params);
if (retval) {
debug_msg("Could not enable BPF\n");
- no_bpf = true;
+ params->mode = TRACING_MODE_TRACEFS;
}
}
+ retval = timerlat_hist_apply_config(tool, params);
+ if (retval) {
+ err_msg("Could not apply config\n");
+ goto out_free;
+ }
+
retval = enable_timerlat(trace);
if (retval) {
err_msg("Failed to enable timerlat tracer\n");
@@ -1320,7 +1329,7 @@ int timerlat_hist_main(int argc, char *argv[])
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
- if (no_bpf) {
+ if (params->mode == TRACING_MODE_TRACEFS) {
trace_instance_start(trace);
} else {
retval = timerlat_bpf_attach();
@@ -1333,7 +1342,7 @@ int timerlat_hist_main(int argc, char *argv[])
tool->start_time = time(NULL);
timerlat_hist_set_signals(params);
- if (no_bpf) {
+ if (params->mode == TRACING_MODE_TRACEFS) {
while (!stop_tracing) {
sleep(params->sleep_time);
@@ -1362,7 +1371,7 @@ int timerlat_hist_main(int argc, char *argv[])
} else
timerlat_bpf_wait(-1);
- if (!no_bpf) {
+ if (params->mode != TRACING_MODE_TRACEFS) {
timerlat_bpf_detach();
retval = timerlat_hist_bpf_pull_data(tool);
if (retval) {
@@ -1409,10 +1418,10 @@ int timerlat_hist_main(int argc, char *argv[])
osnoise_destroy_tool(aa);
osnoise_destroy_tool(record);
osnoise_destroy_tool(tool);
+ if (params->mode != TRACING_MODE_TRACEFS)
+ timerlat_bpf_destroy();
free(params);
free_cpu_idle_disable_states();
- if (!no_bpf)
- timerlat_bpf_destroy();
out_exit:
exit(return_value);
}
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index 7365e08fe986..1644eeb60181 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -559,6 +559,9 @@ static struct timerlat_params
/* display data in microseconds */
params->output_divisor = 1000;
+ /* default to BPF mode */
+ params->mode = TRACING_MODE_BPF;
+
while (1) {
static struct option long_options[] = {
{"auto", required_argument, 0, 'a'},
@@ -790,6 +793,13 @@ static struct timerlat_params
if (params->kernel_workload && params->user_workload)
timerlat_top_usage("--kernel-threads and --user-threads are mutually exclusive!");
+ /*
+ * If auto-analysis or trace output is enabled, switch from BPF mode to
+ * mixed mode
+ */
+ if (params->mode == TRACING_MODE_BPF && params->trace_output && !params->no_aa)
+ params->mode = TRACING_MODE_MIXED;
+
return params;
}
@@ -994,7 +1004,6 @@ int timerlat_top_main(int argc, char *argv[])
char *max_lat;
int retval;
int nr_cpus, i;
- bool no_bpf = false;
params = timerlat_top_parse_args(argc, argv);
if (!params)
@@ -1006,38 +1015,38 @@ int timerlat_top_main(int argc, char *argv[])
goto out_exit;
}
- retval = timerlat_top_apply_config(top, params);
- if (retval) {
- err_msg("Could not apply config\n");
- goto out_free;
- }
-
trace = &top->trace;
/*
- * Save trace instance into global variable so that SIGINT can stop
- * the timerlat tracer.
- * Otherwise, rtla could loop indefinitely when overloaded.
- */
+ * Save trace instance into global variable so that SIGINT can stop
+ * the timerlat tracer.
+ * Otherwise, rtla could loop indefinitely when overloaded.
+ */
top_inst = trace;
+ /*
+ * Try to enable BPF, unless disabled explicitly.
+ * If BPF enablement fails, fall back to tracefs mode.
+ */
if (getenv("RTLA_NO_BPF") && strncmp(getenv("RTLA_NO_BPF"), "1", 2) == 0) {
debug_msg("RTLA_NO_BPF set, disabling BPF\n");
- no_bpf = true;
- }
-
- if (!no_bpf && !tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sample")) {
+ params->mode = TRACING_MODE_TRACEFS;
+ } else if (!tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sample")) {
debug_msg("osnoise:timerlat_sample missing, disabling BPF\n");
- no_bpf = true;
- }
-
- if (!no_bpf) {
+ params->mode = TRACING_MODE_TRACEFS;
+ } else {
retval = timerlat_bpf_init(params);
if (retval) {
debug_msg("Could not enable BPF\n");
- no_bpf = true;
+ params->mode = TRACING_MODE_TRACEFS;
}
}
+ retval = timerlat_top_apply_config(top, params);
+ if (retval) {
+ err_msg("Could not apply config\n");
+ goto out_free;
+ }
+
retval = enable_timerlat(trace);
if (retval) {
err_msg("Failed to enable timerlat tracer\n");
@@ -1166,7 +1175,7 @@ int timerlat_top_main(int argc, char *argv[])
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
- if (no_bpf) {
+ if (params->mode == TRACING_MODE_TRACEFS) {
trace_instance_start(trace);
} else {
retval = timerlat_bpf_attach();
@@ -1179,7 +1188,7 @@ int timerlat_top_main(int argc, char *argv[])
top->start_time = time(NULL);
timerlat_top_set_signals(params);
- if (no_bpf)
+ if (params->mode == TRACING_MODE_TRACEFS)
retval = timerlat_top_main_loop(top, record, params, ¶ms_u);
else
retval = timerlat_top_bpf_main_loop(top, record, params, ¶ms_u);
@@ -1187,7 +1196,7 @@ int timerlat_top_main(int argc, char *argv[])
if (retval)
goto out_top;
- if (!no_bpf)
+ if (params->mode != TRACING_MODE_TRACEFS)
timerlat_bpf_detach();
if (params->user_workload && !params_u.stopped_running) {
@@ -1239,6 +1248,8 @@ int timerlat_top_main(int argc, char *argv[])
osnoise_destroy_tool(aa);
osnoise_destroy_tool(record);
osnoise_destroy_tool(top);
+ if (params->mode != TRACING_MODE_TRACEFS)
+ timerlat_bpf_destroy();
free(params);
free_cpu_idle_disable_states();
out_exit:
--
2.47.2
next prev parent reply other threads:[~2025-07-21 23:11 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-07-21 23:12 [for-next][PATCH 0/9] tracing/tools: Updates for v6.17 Steven Rostedt
2025-07-21 23:12 ` Steven Rostedt [this message]
2025-07-21 23:12 ` [for-next][PATCH 2/9] rtla/timerlat: Add action on threshold feature Steven Rostedt
2025-07-21 23:12 ` [for-next][PATCH 3/9] rtla/timerlat_bpf: Allow resuming tracing Steven Rostedt
2025-07-21 23:12 ` [for-next][PATCH 4/9] rtla/timerlat: Add continue action Steven Rostedt
2025-07-21 23:12 ` [for-next][PATCH 5/9] rtla/timerlat: Add action on end feature Steven Rostedt
2025-07-21 23:12 ` [for-next][PATCH 6/9] rtla/tests: Check rtla output with grep Steven Rostedt
2025-07-21 23:12 ` [for-next][PATCH 7/9] rtla/tests: Add tests for actions Steven Rostedt
2025-07-21 23:12 ` [for-next][PATCH 8/9] rtla/tests: Limit duration to maximum of 10s Steven Rostedt
2025-07-21 23:12 ` [for-next][PATCH 9/9] Documentation/rtla: Add actions feature Steven Rostedt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250721231222.293482311@kernel.org \
--to=rostedt@kernel.org \
--cc=acme@kernel.org \
--cc=costa.shul@redhat.com \
--cc=crwood@redhat.com \
--cc=cyin@redhat.com \
--cc=gmonaco@redhat.com \
--cc=jkacur@redhat.com \
--cc=lgoncalv@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=tglozar@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.