* [PATCH 1/8] rtla/timerlat: Introduce enum timerlat_tracing_mode
2025-06-11 13:56 [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Tomas Glozar
@ 2025-06-11 13:56 ` Tomas Glozar
2025-06-11 13:56 ` [PATCH 2/8] rtla/timerlat: Add action on threshold feature Tomas Glozar
` (7 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Tomas Glozar @ 2025-06-11 13:56 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Arnaldo Carvalho de Melo, Chang Yin, Costa Shulyupin,
Tomas Glozar
After the introduction of BPF-based sample collection, rtla-timerlat
effectively runs in one of three modes:
- Pure BPF mode, with tracefs only being used to set up the timerlat
tracer. Sample processing and stop on threshold are handled by BPF.
- tracefs mode. BPF is unsupported or kernel is lacking the necessary
trace event (osnoise:timerlat_sample). Stop on theshold is handled by
timerlat tracer stopping tracing in all instances.
- BPF/tracefs mixed mode - BPF is used for sample collection for top or
histogram, tracefs is used for trace output and/or auto-analysis. Stop
on threshold is handled both through BPF program, which stops sample
collection for top/histogram and wakes up rtla, and by timerlat
tracer, which stops tracing for trace output/auto-analysis instances.
Add enum timerlat_tracing_mode, with three values:
- TRACING_MODE_BPF
- TRACING_MODE_TRACEFS
- TRACING_MODE_MIXED
Those represent the modes described above. A field of this type is added
to struct timerlat_params, named "mode", replacing the no_bpf variable.
params->mode is set in timerlat_{top,hist}_parse_args to
TRACING_MODE_BPF or TRACING_MODE_MIXED based on whether trace output
and/or auto-analysis is requested. timerlat_{top,hist}_main then checks
if BPF is not unavailable or disabled, in that case, it sets
params->mode to TRACING_MODE_TRACEFS.
A condition is added to timerlat_apply_config that skips setting
timerlat tracer thresholds if params->mode is TRACING_MODE_BPF (those
are unnecessary, since they only turn off tracing, which is already
turned off in that case, since BPF is used to collect samples).
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/src/timerlat.c | 24 +++++++----
tools/tracing/rtla/src/timerlat.h | 18 ++++++++
tools/tracing/rtla/src/timerlat_hist.c | 51 +++++++++++++----------
tools/tracing/rtla/src/timerlat_top.c | 57 +++++++++++++++-----------
4 files changed, 97 insertions(+), 53 deletions(-)
diff --git a/tools/tracing/rtla/src/timerlat.c b/tools/tracing/rtla/src/timerlat.c
index c29e2ba2d7d8..63d6d43eafff 100644
--- a/tools/tracing/rtla/src/timerlat.c
+++ b/tools/tracing/rtla/src/timerlat.c
@@ -40,16 +40,22 @@ timerlat_apply_config(struct osnoise_tool *tool, struct timerlat_params *params)
CPU_SET(i, ¶ms->monitored_cpus);
}
- retval = osnoise_set_stop_us(tool->context, params->stop_us);
- if (retval) {
- err_msg("Failed to set stop us\n");
- goto out_err;
- }
+ if (params->mode != TRACING_MODE_BPF) {
+ /*
+ * In tracefs and mixed mode, timerlat tracer handles stopping
+ * on threshold
+ */
+ retval = osnoise_set_stop_us(tool->context, params->stop_us);
+ if (retval) {
+ err_msg("Failed to set stop us\n");
+ goto out_err;
+ }
- retval = osnoise_set_stop_total_us(tool->context, params->stop_total_us);
- if (retval) {
- err_msg("Failed to set stop total us\n");
- goto out_err;
+ retval = osnoise_set_stop_total_us(tool->context, params->stop_total_us);
+ if (retval) {
+ err_msg("Failed to set stop total us\n");
+ goto out_err;
+ }
}
diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/timerlat.h
index 73045aef23fa..e0a553545d03 100644
--- a/tools/tracing/rtla/src/timerlat.h
+++ b/tools/tracing/rtla/src/timerlat.h
@@ -1,6 +1,23 @@
// SPDX-License-Identifier: GPL-2.0
#include "osnoise.h"
+/*
+ * Define timerlat tracing mode.
+ *
+ * There are three tracing modes:
+ * - tracefs-only, used when BPF is unavailable.
+ * - BPF-only, used when BPF is available and neither trace saving nor
+ * auto-analysis are enabled.
+ * - mixed mode, used when BPF is available and either trace saving or
+ * auto-analysis is enabled (which rely on sample collection through
+ * tracefs).
+ */
+enum timerlat_tracing_mode {
+ TRACING_MODE_BPF,
+ TRACING_MODE_TRACEFS,
+ TRACING_MODE_MIXED,
+};
+
struct timerlat_params {
/* Common params */
char *cpus;
@@ -30,6 +47,7 @@ struct timerlat_params {
cpu_set_t hk_cpu_set;
struct sched_attr sched_param;
struct trace_events *events;
+ enum timerlat_tracing_mode mode;
union {
struct {
/* top only */
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index 36d2294c963d..6cf260e8553b 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -802,6 +802,9 @@ static struct timerlat_params
params->bucket_size = 1;
params->entries = 256;
+ /* default to BPF mode */
+ params->mode = TRACING_MODE_BPF;
+
while (1) {
static struct option long_options[] = {
{"auto", required_argument, 0, 'a'},
@@ -1054,6 +1057,13 @@ static struct timerlat_params
if (params->kernel_workload && params->user_workload)
timerlat_hist_usage("--kernel-threads and --user-threads are mutually exclusive!");
+ /*
+ * If auto-analysis or trace output is enabled, switch from BPF mode to
+ * mixed mode
+ */
+ if (params->mode == TRACING_MODE_BPF && params->trace_output && !params->no_aa)
+ params->mode = TRACING_MODE_MIXED;
+
return params;
}
@@ -1149,7 +1159,6 @@ int timerlat_hist_main(int argc, char *argv[])
pthread_t timerlat_u;
int retval;
int nr_cpus, i;
- bool no_bpf = false;
params = timerlat_hist_parse_args(argc, argv);
if (!params)
@@ -1161,12 +1170,6 @@ int timerlat_hist_main(int argc, char *argv[])
goto out_exit;
}
- retval = timerlat_hist_apply_config(tool, params);
- if (retval) {
- err_msg("Could not apply config\n");
- goto out_free;
- }
-
trace = &tool->trace;
/*
* Save trace instance into global variable so that SIGINT can stop
@@ -1175,24 +1178,30 @@ int timerlat_hist_main(int argc, char *argv[])
*/
hist_inst = trace;
+ /*
+ * Try to enable BPF, unless disabled explicitly.
+ * If BPF enablement fails, fall back to tracefs mode.
+ */
if (getenv("RTLA_NO_BPF") && strncmp(getenv("RTLA_NO_BPF"), "1", 2) == 0) {
debug_msg("RTLA_NO_BPF set, disabling BPF\n");
- no_bpf = true;
- }
-
- if (!no_bpf && !tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sample")) {
+ params->mode = TRACING_MODE_TRACEFS;
+ } else if (!tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sample")) {
debug_msg("osnoise:timerlat_sample missing, disabling BPF\n");
- no_bpf = true;
- }
-
- if (!no_bpf) {
+ params->mode = TRACING_MODE_TRACEFS;
+ } else {
retval = timerlat_bpf_init(params);
if (retval) {
debug_msg("Could not enable BPF\n");
- no_bpf = true;
+ params->mode = TRACING_MODE_TRACEFS;
}
}
+ retval = timerlat_hist_apply_config(tool, params);
+ if (retval) {
+ err_msg("Could not apply config\n");
+ goto out_free;
+ }
+
retval = enable_timerlat(trace);
if (retval) {
err_msg("Failed to enable timerlat tracer\n");
@@ -1320,7 +1329,7 @@ int timerlat_hist_main(int argc, char *argv[])
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
- if (no_bpf) {
+ if (params->mode == TRACING_MODE_TRACEFS) {
trace_instance_start(trace);
} else {
retval = timerlat_bpf_attach();
@@ -1333,7 +1342,7 @@ int timerlat_hist_main(int argc, char *argv[])
tool->start_time = time(NULL);
timerlat_hist_set_signals(params);
- if (no_bpf) {
+ if (params->mode == TRACING_MODE_TRACEFS) {
while (!stop_tracing) {
sleep(params->sleep_time);
@@ -1362,7 +1371,7 @@ int timerlat_hist_main(int argc, char *argv[])
} else
timerlat_bpf_wait(-1);
- if (!no_bpf) {
+ if (params->mode != TRACING_MODE_TRACEFS) {
timerlat_bpf_detach();
retval = timerlat_hist_bpf_pull_data(tool);
if (retval) {
@@ -1409,10 +1418,10 @@ int timerlat_hist_main(int argc, char *argv[])
osnoise_destroy_tool(aa);
osnoise_destroy_tool(record);
osnoise_destroy_tool(tool);
+ if (params->mode != TRACING_MODE_TRACEFS)
+ timerlat_bpf_destroy();
free(params);
free_cpu_idle_disable_states();
- if (!no_bpf)
- timerlat_bpf_destroy();
out_exit:
exit(return_value);
}
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index 7365e08fe986..1644eeb60181 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -559,6 +559,9 @@ static struct timerlat_params
/* display data in microseconds */
params->output_divisor = 1000;
+ /* default to BPF mode */
+ params->mode = TRACING_MODE_BPF;
+
while (1) {
static struct option long_options[] = {
{"auto", required_argument, 0, 'a'},
@@ -790,6 +793,13 @@ static struct timerlat_params
if (params->kernel_workload && params->user_workload)
timerlat_top_usage("--kernel-threads and --user-threads are mutually exclusive!");
+ /*
+ * If auto-analysis or trace output is enabled, switch from BPF mode to
+ * mixed mode
+ */
+ if (params->mode == TRACING_MODE_BPF && params->trace_output && !params->no_aa)
+ params->mode = TRACING_MODE_MIXED;
+
return params;
}
@@ -994,7 +1004,6 @@ int timerlat_top_main(int argc, char *argv[])
char *max_lat;
int retval;
int nr_cpus, i;
- bool no_bpf = false;
params = timerlat_top_parse_args(argc, argv);
if (!params)
@@ -1006,38 +1015,38 @@ int timerlat_top_main(int argc, char *argv[])
goto out_exit;
}
- retval = timerlat_top_apply_config(top, params);
- if (retval) {
- err_msg("Could not apply config\n");
- goto out_free;
- }
-
trace = &top->trace;
/*
- * Save trace instance into global variable so that SIGINT can stop
- * the timerlat tracer.
- * Otherwise, rtla could loop indefinitely when overloaded.
- */
+ * Save trace instance into global variable so that SIGINT can stop
+ * the timerlat tracer.
+ * Otherwise, rtla could loop indefinitely when overloaded.
+ */
top_inst = trace;
+ /*
+ * Try to enable BPF, unless disabled explicitly.
+ * If BPF enablement fails, fall back to tracefs mode.
+ */
if (getenv("RTLA_NO_BPF") && strncmp(getenv("RTLA_NO_BPF"), "1", 2) == 0) {
debug_msg("RTLA_NO_BPF set, disabling BPF\n");
- no_bpf = true;
- }
-
- if (!no_bpf && !tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sample")) {
+ params->mode = TRACING_MODE_TRACEFS;
+ } else if (!tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sample")) {
debug_msg("osnoise:timerlat_sample missing, disabling BPF\n");
- no_bpf = true;
- }
-
- if (!no_bpf) {
+ params->mode = TRACING_MODE_TRACEFS;
+ } else {
retval = timerlat_bpf_init(params);
if (retval) {
debug_msg("Could not enable BPF\n");
- no_bpf = true;
+ params->mode = TRACING_MODE_TRACEFS;
}
}
+ retval = timerlat_top_apply_config(top, params);
+ if (retval) {
+ err_msg("Could not apply config\n");
+ goto out_free;
+ }
+
retval = enable_timerlat(trace);
if (retval) {
err_msg("Failed to enable timerlat tracer\n");
@@ -1166,7 +1175,7 @@ int timerlat_top_main(int argc, char *argv[])
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
- if (no_bpf) {
+ if (params->mode == TRACING_MODE_TRACEFS) {
trace_instance_start(trace);
} else {
retval = timerlat_bpf_attach();
@@ -1179,7 +1188,7 @@ int timerlat_top_main(int argc, char *argv[])
top->start_time = time(NULL);
timerlat_top_set_signals(params);
- if (no_bpf)
+ if (params->mode == TRACING_MODE_TRACEFS)
retval = timerlat_top_main_loop(top, record, params, ¶ms_u);
else
retval = timerlat_top_bpf_main_loop(top, record, params, ¶ms_u);
@@ -1187,7 +1196,7 @@ int timerlat_top_main(int argc, char *argv[])
if (retval)
goto out_top;
- if (!no_bpf)
+ if (params->mode != TRACING_MODE_TRACEFS)
timerlat_bpf_detach();
if (params->user_workload && !params_u.stopped_running) {
@@ -1239,6 +1248,8 @@ int timerlat_top_main(int argc, char *argv[])
osnoise_destroy_tool(aa);
osnoise_destroy_tool(record);
osnoise_destroy_tool(top);
+ if (params->mode != TRACING_MODE_TRACEFS)
+ timerlat_bpf_destroy();
free(params);
free_cpu_idle_disable_states();
out_exit:
--
2.49.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH 2/8] rtla/timerlat: Add action on threshold feature
2025-06-11 13:56 [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Tomas Glozar
2025-06-11 13:56 ` [PATCH 1/8] rtla/timerlat: Introduce enum timerlat_tracing_mode Tomas Glozar
@ 2025-06-11 13:56 ` Tomas Glozar
2025-08-03 11:08 ` Costa Shulyupin
2025-06-11 13:56 ` [PATCH 3/8] rtla/timerlat_bpf: Allow resuming tracing Tomas Glozar
` (6 subsequent siblings)
8 siblings, 1 reply; 13+ messages in thread
From: Tomas Glozar @ 2025-06-11 13:56 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Arnaldo Carvalho de Melo, Chang Yin, Costa Shulyupin,
Tomas Glozar
Extend the functionality provided by the -t/--trace option, which
triggers saving the contents of a tracefs buffer after tracing is
stopped, to support implementing arbitrary actions.
A new option, -A/--on-threshold, is added, taking an argument
that further specifies the action. Actions added in this patch are:
- trace[,file=<filename>]: Saves tracefs buffer, optionally taking a
filename.
- signal,num=<sig>,pid=<pid>: Sends signal to process. "parent" might
be specified instead of number to send signal to parent process.
- shell,command=<command>: Execute shell command.
Multiple actions may be specified and will be executed in order,
including multiple actions of the same type. Trace output requested via
-t and -a now adds a trace action to the end of the list.
If an action fails, the following actions are not executed. For
example, this command:
$ rtla timerlat -T 20 -A trace \
-A shell,command="grep ipi_send timerlat_output.txt" \
-A signal,num=2,pid=parent
will send signal 2 (SIGINT) to parent process, but only if saved trace
contains the text "ipi_send".
This way, the feature can be used for flexible reactions on latency
spikes, and allows combining rtla with other tooling like perf.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/src/Build | 1 +
tools/tracing/rtla/src/actions.c | 235 +++++++++++++++++++++++++
tools/tracing/rtla/src/actions.h | 49 ++++++
tools/tracing/rtla/src/timerlat.h | 3 +-
tools/tracing/rtla/src/timerlat_hist.c | 42 +++--
tools/tracing/rtla/src/timerlat_top.c | 40 +++--
6 files changed, 345 insertions(+), 25 deletions(-)
create mode 100644 tools/tracing/rtla/src/actions.c
create mode 100644 tools/tracing/rtla/src/actions.h
diff --git a/tools/tracing/rtla/src/Build b/tools/tracing/rtla/src/Build
index 7bb7e39e391a..66631280b75b 100644
--- a/tools/tracing/rtla/src/Build
+++ b/tools/tracing/rtla/src/Build
@@ -1,5 +1,6 @@
rtla-y += trace.o
rtla-y += utils.o
+rtla-y += actions.o
rtla-y += osnoise.o
rtla-y += osnoise_top.o
rtla-y += osnoise_hist.o
diff --git a/tools/tracing/rtla/src/actions.c b/tools/tracing/rtla/src/actions.c
new file mode 100644
index 000000000000..63bee5bdabfd
--- /dev/null
+++ b/tools/tracing/rtla/src/actions.c
@@ -0,0 +1,235 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <stdlib.h>
+#include <string.h>
+#include <signal.h>
+#include <unistd.h>
+
+#include "actions.h"
+#include "trace.h"
+#include "utils.h"
+
+/*
+ * actions_init - initialize struct actions
+ */
+void
+actions_init(struct actions *self)
+{
+ self->size = action_default_size;
+ self->list = calloc(self->size, sizeof(struct action));
+ self->len = 0;
+
+ memset(&self->present, 0, sizeof(self->present));
+
+ /* This has to be set by the user */
+ self->trace_output_inst = NULL;
+}
+
+/*
+ * actions_destroy - destroy struct actions
+ */
+void
+actions_destroy(struct actions *self)
+{
+ /* Free any action-specific data */
+ for (struct action *action = self->list; action < self->list + self->len; action++) {
+ if (action->type == ACTION_SHELL)
+ free(action->command);
+ if (action->type == ACTION_TRACE_OUTPUT)
+ free(action->trace_output);
+ }
+
+ /* Free action list */
+ free(self->list);
+}
+
+/*
+ * actions_new - Get pointer to new action
+ */
+static struct action *
+actions_new(struct actions *self)
+{
+ if (self->size >= self->len) {
+ self->size *= 2;
+ self->list = realloc(self->list, self->size * sizeof(struct action));
+ }
+
+ return &self->list[self->len++];
+}
+
+/*
+ * actions_add_trace_output - add an action to output trace
+ */
+int
+actions_add_trace_output(struct actions *self, const char *trace_output)
+{
+ struct action *action = actions_new(self);
+
+ self->present[ACTION_TRACE_OUTPUT] = true;
+ action->type = ACTION_TRACE_OUTPUT;
+ action->trace_output = calloc(strlen(trace_output) + 1, sizeof(char));
+ if (!action->trace_output)
+ return -1;
+ strcpy(action->trace_output, trace_output);
+
+ return 0;
+}
+
+/*
+ * actions_add_trace_output - add an action to send signal to a process
+ */
+int
+actions_add_signal(struct actions *self, int signal, int pid)
+{
+ struct action *action = actions_new(self);
+
+ self->present[ACTION_SIGNAL] = true;
+ action->type = ACTION_SIGNAL;
+ action->signal = signal;
+ action->pid = pid;
+
+ return 0;
+}
+
+/*
+ * actions_add_shell - add an action to execute a shell command
+ */
+int
+actions_add_shell(struct actions *self, const char *command)
+{
+ struct action *action = actions_new(self);
+
+ self->present[ACTION_SHELL] = true;
+ action->type = ACTION_SHELL;
+ action->command = calloc(strlen(command) + 1, sizeof(char));
+ if (!action->command)
+ return -1;
+ strcpy(action->command, command);
+
+ return 0;
+}
+
+/*
+ * actions_parse - add an action based on text specification
+ */
+int
+actions_parse(struct actions *self, const char *trigger)
+{
+ enum action_type type = ACTION_NONE;
+ char *token;
+ char trigger_c[strlen(trigger)];
+
+ /* For ACTION_SIGNAL */
+ int signal = 0, pid = 0;
+
+ /* For ACTION_TRACE_OUTPUT */
+ char *trace_output;
+
+ strcpy(trigger_c, trigger);
+ token = strtok(trigger_c, ",");
+
+ if (strcmp(token, "trace") == 0)
+ type = ACTION_TRACE_OUTPUT;
+ else if (strcmp(token, "signal") == 0)
+ type = ACTION_SIGNAL;
+ else if (strcmp(token, "shell") == 0)
+ type = ACTION_SHELL;
+ else
+ /* Invalid trigger type */
+ return -1;
+
+ token = strtok(NULL, ",");
+
+ switch (type) {
+ case ACTION_TRACE_OUTPUT:
+ /* Takes no argument */
+ if (token == NULL)
+ trace_output = "timerlat_trace.txt";
+ else {
+ if (strlen(token) > 5 && strncmp(token, "file=", 5) == 0) {
+ trace_output = token + 5;
+ } else {
+ /* Invalid argument */
+ return -1;
+ }
+
+ token = strtok(NULL, ",");
+ if (token != NULL)
+ /* Only one argument allowed */
+ return -1;
+ }
+ return actions_add_trace_output(self, trace_output);
+ case ACTION_SIGNAL:
+ /* Takes two arguments, num (signal) and pid */
+ while (token != NULL) {
+ if (strlen(token) > 4 && strncmp(token, "num=", 4) == 0) {
+ signal = atoi(token + 4);
+ } else if (strlen(token) > 4 && strncmp(token, "pid=", 4) == 0) {
+ if (strncmp(token + 4, "parent", 7) == 0)
+ pid = -1;
+ else
+ pid = atoi(token + 4);
+ } else {
+ /* Invalid argument */
+ return -1;
+ }
+
+ token = strtok(NULL, ",");
+ }
+
+ if (!signal || !pid)
+ /* Missing argument */
+ return -1;
+
+ return actions_add_signal(self, signal, pid);
+ case ACTION_SHELL:
+ if (token == NULL)
+ return -1;
+ if (strlen(token) > 8 && strncmp(token, "command=", 8) == 0)
+ return actions_add_shell(self, token + 8);
+ return -1;
+ default:
+ return -1;
+ }
+}
+
+/*
+ * actions_perform - perform all actions
+ */
+int
+actions_perform(const struct actions *self)
+{
+ int pid, retval;
+ const struct action *action;
+
+ for (action = self->list; action < self->list + self->len; action++) {
+ switch (action->type) {
+ case ACTION_TRACE_OUTPUT:
+ retval = save_trace_to_file(self->trace_output_inst, action->trace_output);
+ if (retval) {
+ err_msg("Error saving trace\n");
+ return retval;
+ }
+ break;
+ case ACTION_SIGNAL:
+ if (action->pid == -1)
+ pid = getppid();
+ else
+ pid = action->pid;
+ retval = kill(pid, action->signal);
+ if (retval) {
+ err_msg("Error sending signal\n");
+ return retval;
+ }
+ break;
+ case ACTION_SHELL:
+ retval = system(action->command);
+ if (retval)
+ return retval;
+ break;
+ default:
+ break;
+ }
+ }
+
+ return 0;
+}
diff --git a/tools/tracing/rtla/src/actions.h b/tools/tracing/rtla/src/actions.h
new file mode 100644
index 000000000000..076bbff8519e
--- /dev/null
+++ b/tools/tracing/rtla/src/actions.h
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <tracefs.h>
+#include <stdbool.h>
+
+enum action_type {
+ ACTION_NONE = 0,
+ ACTION_TRACE_OUTPUT,
+ ACTION_SIGNAL,
+ ACTION_SHELL,
+ ACTION_FIELD_N
+};
+
+struct action {
+ enum action_type type;
+ union {
+ struct {
+ /* For ACTION_TRACE_OUTPUT */
+ char *trace_output;
+ };
+ struct {
+ /* For ACTION_SIGNAL */
+ int signal;
+ int pid;
+ };
+ struct {
+ /* For ACTION_SHELL */
+ char *command;
+ };
+ };
+};
+
+static const int action_default_size = 8;
+
+struct actions {
+ struct action *list;
+ int len, size;
+ bool present[ACTION_FIELD_N];
+
+ /* External dependencies */
+ struct tracefs_instance *trace_output_inst;
+};
+
+void actions_init(struct actions *self);
+void actions_destroy(struct actions *self);
+int actions_add_trace_output(struct actions *self, const char *trace_output);
+int actions_add_signal(struct actions *self, int signal, int pid);
+int actions_add_shell(struct actions *self, const char *command);
+int actions_parse(struct actions *self, const char *trigger);
+int actions_perform(const struct actions *self);
diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/timerlat.h
index e0a553545d03..d1fcf9a97621 100644
--- a/tools/tracing/rtla/src/timerlat.h
+++ b/tools/tracing/rtla/src/timerlat.h
@@ -1,4 +1,5 @@
// SPDX-License-Identifier: GPL-2.0
+#include "actions.h"
#include "osnoise.h"
/*
@@ -22,7 +23,6 @@ struct timerlat_params {
/* Common params */
char *cpus;
cpu_set_t monitored_cpus;
- char *trace_output;
char *cgroup_name;
unsigned long long runtime;
long long stop_us;
@@ -48,6 +48,7 @@ struct timerlat_params {
struct sched_attr sched_param;
struct trace_events *events;
enum timerlat_tracing_mode mode;
+ struct actions actions;
union {
struct {
/* top only */
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index 6cf260e8553b..06ee3d925ef9 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -753,6 +753,7 @@ static void timerlat_hist_usage(char *usage)
" in nanoseconds",
" -u/--user-threads: use rtla user-space threads instead of kernel-space timerlat threads",
" -k/--kernel-threads: use timerlat kernel-space threads instead of rtla user-space threads",
+ " -A/--on-threshold <action>: define action to be executed at latency threshold, multiple -A are allowed",
" -U/--user-load: enable timerlat for user-defined user-space workload",
" --warm-up s: let the workload run for s seconds before collecting data",
" --trace-buffer-size kB: set the per-cpu trace buffer size in kB",
@@ -786,11 +787,14 @@ static struct timerlat_params
int auto_thresh;
int retval;
int c;
+ char *trace_output = NULL;
params = calloc(1, sizeof(*params));
if (!params)
exit(1);
+ actions_init(¶ms->actions);
+
/* disabled by default */
params->dma_latency = -1;
@@ -807,6 +811,7 @@ static struct timerlat_params
while (1) {
static struct option long_options[] = {
+ {"on-threshold", required_argument, 0, 'A'},
{"auto", required_argument, 0, 'a'},
{"cpus", required_argument, 0, 'c'},
{"cgroup", optional_argument, 0, 'C'},
@@ -847,8 +852,9 @@ static struct timerlat_params
/* getopt_long stores the option index here. */
int option_index = 0;
- c = getopt_long(argc, argv, "a:c:C::b:d:e:E:DhH:i:knp:P:s:t::T:uU0123456:7:8:9\1\2:\3:",
- long_options, &option_index);
+ c = getopt_long(argc, argv,
+ "A:a:c:C::b:d:e:E:DhH:i:knp:P:s:t::T:uU0123456:7:8:9\1\2:\3:",
+ long_options, &option_index);
/* detect the end of the options. */
if (c == -1)
@@ -866,8 +872,15 @@ static struct timerlat_params
params->print_stack = auto_thresh;
/* set trace */
- params->trace_output = "timerlat_trace.txt";
+ trace_output = "timerlat_trace.txt";
+ break;
+ case 'A':
+ retval = actions_parse(¶ms->actions, optarg);
+ if (retval) {
+ err_msg("Invalid action %s\n", optarg);
+ exit(EXIT_FAILURE);
+ }
break;
case 'c':
retval = parse_cpu_set(optarg, ¶ms->monitored_cpus);
@@ -956,13 +969,13 @@ static struct timerlat_params
case 't':
if (optarg) {
if (optarg[0] == '=')
- params->trace_output = &optarg[1];
+ trace_output = &optarg[1];
else
- params->trace_output = &optarg[0];
+ trace_output = &optarg[0];
} else if (optind < argc && argv[optind][0] != '-')
- params->trace_output = argv[optind];
+ trace_output = argv[optind];
else
- params->trace_output = "timerlat_trace.txt";
+ trace_output = "timerlat_trace.txt";
break;
case 'u':
params->user_workload = 1;
@@ -1037,6 +1050,9 @@ static struct timerlat_params
}
}
+ if (trace_output)
+ actions_add_trace_output(¶ms->actions, trace_output);
+
if (geteuid()) {
err_msg("rtla needs root permission\n");
exit(EXIT_FAILURE);
@@ -1061,7 +1077,8 @@ static struct timerlat_params
* If auto-analysis or trace output is enabled, switch from BPF mode to
* mixed mode
*/
- if (params->mode == TRACING_MODE_BPF && params->trace_output && !params->no_aa)
+ if (params->mode == TRACING_MODE_BPF &&
+ (params->actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa))
params->mode = TRACING_MODE_MIXED;
return params;
@@ -1254,12 +1271,13 @@ int timerlat_hist_main(int argc, char *argv[])
}
}
- if (params->trace_output) {
+ if (params->actions.present[ACTION_TRACE_OUTPUT]) {
record = osnoise_init_trace_tool("timerlat");
if (!record) {
err_msg("Failed to enable the trace instance\n");
goto out_free;
}
+ params->actions.trace_output_inst = record->trace.inst;
if (params->events) {
retval = trace_events_enable(&record->trace, params->events);
@@ -1325,7 +1343,7 @@ int timerlat_hist_main(int argc, char *argv[])
* tracing while enabling other instances. The trace instance is the
* one with most valuable information.
*/
- if (params->trace_output)
+ if (params->actions.present[ACTION_TRACE_OUTPUT])
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
@@ -1395,8 +1413,7 @@ int timerlat_hist_main(int argc, char *argv[])
if (!params->no_aa)
timerlat_auto_analysis(params->stop_us, params->stop_total_us);
- save_trace_to_file(record ? record->trace.inst : NULL,
- params->trace_output);
+ actions_perform(¶ms->actions);
return_value = FAILED;
}
@@ -1418,6 +1435,7 @@ int timerlat_hist_main(int argc, char *argv[])
osnoise_destroy_tool(aa);
osnoise_destroy_tool(record);
osnoise_destroy_tool(tool);
+ actions_destroy(¶ms->actions);
if (params->mode != TRACING_MODE_TRACEFS)
timerlat_bpf_destroy();
free(params);
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index 1644eeb60181..fa1072c0d126 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -513,6 +513,7 @@ static void timerlat_top_usage(char *usage)
" -u/--user-threads: use rtla user-space threads instead of kernel-space timerlat threads",
" -k/--kernel-threads: use timerlat kernel-space threads instead of rtla user-space threads",
" -U/--user-load: enable timerlat for user-defined user-space workload",
+ " -A/--on-threshold <action>: define action to be executed at latency threshold, multiple -A are allowed",
" --warm-up s: let the workload run for s seconds before collecting data",
" --trace-buffer-size kB: set the per-cpu trace buffer size in kB",
" --deepest-idle-state n: only go down to idle state n on cpus used by timerlat to reduce exit from idle latency",
@@ -545,11 +546,14 @@ static struct timerlat_params
long long auto_thresh;
int retval;
int c;
+ char *trace_output = NULL;
params = calloc(1, sizeof(*params));
if (!params)
exit(1);
+ actions_init(¶ms->actions);
+
/* disabled by default */
params->dma_latency = -1;
@@ -564,6 +568,7 @@ static struct timerlat_params
while (1) {
static struct option long_options[] = {
+ {"action-on-overflow", required_argument, 0, 'A'},
{"auto", required_argument, 0, 'a'},
{"cpus", required_argument, 0, 'c'},
{"cgroup", optional_argument, 0, 'C'},
@@ -598,7 +603,7 @@ static struct timerlat_params
/* getopt_long stores the option index here. */
int option_index = 0;
- c = getopt_long(argc, argv, "a:c:C::d:De:hH:i:knp:P:qs:t::T:uU0:1:2:345:6:7:",
+ c = getopt_long(argc, argv, "A:a:c:C::d:De:hH:i:knp:P:qs:t::T:uU0:1:2:345:6:7:",
long_options, &option_index);
/* detect the end of the options. */
@@ -617,7 +622,14 @@ static struct timerlat_params
params->print_stack = auto_thresh;
/* set trace */
- params->trace_output = "timerlat_trace.txt";
+ trace_output = "timerlat_trace.txt";
+ break;
+ case 'A':
+ retval = actions_parse(¶ms->actions, optarg);
+ if (retval) {
+ err_msg("Invalid action %s\n", optarg);
+ exit(EXIT_FAILURE);
+ }
break;
case '5':
/* it is here because it is similar to -a */
@@ -712,14 +724,13 @@ static struct timerlat_params
case 't':
if (optarg) {
if (optarg[0] == '=')
- params->trace_output = &optarg[1];
+ trace_output = &optarg[1];
else
- params->trace_output = &optarg[0];
+ trace_output = &optarg[0];
} else if (optind < argc && argv[optind][0] != '-')
- params->trace_output = argv[optind];
+ trace_output = argv[optind];
else
- params->trace_output = "timerlat_trace.txt";
-
+ trace_output = "timerlat_trace.txt";
break;
case 'u':
params->user_workload = true;
@@ -776,6 +787,9 @@ static struct timerlat_params
}
}
+ if (trace_output)
+ actions_add_trace_output(¶ms->actions, trace_output);
+
if (geteuid()) {
err_msg("rtla needs root permission\n");
exit(EXIT_FAILURE);
@@ -797,7 +811,8 @@ static struct timerlat_params
* If auto-analysis or trace output is enabled, switch from BPF mode to
* mixed mode
*/
- if (params->mode == TRACING_MODE_BPF && params->trace_output && !params->no_aa)
+ if (params->mode == TRACING_MODE_BPF &&
+ (params->actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa))
params->mode = TRACING_MODE_MIXED;
return params;
@@ -1099,12 +1114,13 @@ int timerlat_top_main(int argc, char *argv[])
}
}
- if (params->trace_output) {
+ if (params->actions.present[ACTION_TRACE_OUTPUT]) {
record = osnoise_init_trace_tool("timerlat");
if (!record) {
err_msg("Failed to enable the trace instance\n");
goto out_free;
}
+ params->actions.trace_output_inst = record->trace.inst;
if (params->events) {
retval = trace_events_enable(&record->trace, params->events);
@@ -1171,7 +1187,7 @@ int timerlat_top_main(int argc, char *argv[])
* tracing while enabling other instances. The trace instance is the
* one with most valuable information.
*/
- if (params->trace_output)
+ if (params->actions.present[ACTION_TRACE_OUTPUT])
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
@@ -1214,8 +1230,7 @@ int timerlat_top_main(int argc, char *argv[])
if (!params->no_aa)
timerlat_auto_analysis(params->stop_us, params->stop_total_us);
- save_trace_to_file(record ? record->trace.inst : NULL,
- params->trace_output);
+ actions_perform(¶ms->actions);
return_value = FAILED;
} else if (params->aa_only) {
/*
@@ -1248,6 +1263,7 @@ int timerlat_top_main(int argc, char *argv[])
osnoise_destroy_tool(aa);
osnoise_destroy_tool(record);
osnoise_destroy_tool(top);
+ actions_destroy(¶ms->actions);
if (params->mode != TRACING_MODE_TRACEFS)
timerlat_bpf_destroy();
free(params);
--
2.49.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH 2/8] rtla/timerlat: Add action on threshold feature
2025-06-11 13:56 ` [PATCH 2/8] rtla/timerlat: Add action on threshold feature Tomas Glozar
@ 2025-08-03 11:08 ` Costa Shulyupin
2025-08-04 11:04 ` Tomas Glozar
0 siblings, 1 reply; 13+ messages in thread
From: Costa Shulyupin @ 2025-08-03 11:08 UTC (permalink / raw)
To: Tomas Glozar
Cc: Steven Rostedt, linux-trace-kernel, linux-kernel, John Kacur,
Luis Goncalves, Arnaldo Carvalho de Melo, Chang Yin
On Wed, 11 Jun 2025 at 16:57, Tomas Glozar <tglozar@redhat.com> wrote:
> A new option, -A/--on-threshold, is added, taking an argument
The term "threshold" is ambiguous. The use of the term is inconsistent
across the tools. In osnoise top and hist, it means "the minimum delta
to be considered noise," which conflicts with the semantics of the
`--on-threshold <action>` option. To avoid confusion, I propose
introducing "low" and "high" thresholds and updating the sources and
documentation accordingly.
Thanks
Costa
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/8] rtla/timerlat: Add action on threshold feature
2025-08-03 11:08 ` Costa Shulyupin
@ 2025-08-04 11:04 ` Tomas Glozar
0 siblings, 0 replies; 13+ messages in thread
From: Tomas Glozar @ 2025-08-04 11:04 UTC (permalink / raw)
To: Costa Shulyupin
Cc: Steven Rostedt, linux-trace-kernel, linux-kernel, John Kacur,
Luis Goncalves, Arnaldo Carvalho de Melo, Chang Yin
ne 3. 8. 2025 v 13:08 odesílatel Costa Shulyupin <costa.shul@redhat.com> napsal:
>
> The term "threshold" is ambiguous. The use of the term is inconsistent
> across the tools. In osnoise top and hist, it means "the minimum delta
> to be considered noise," which conflicts with the semantics of the
> `--on-threshold <action>` option. To avoid confusion, I propose
> introducing "low" and "high" thresholds and updating the sources and
> documentation accordingly.
>
There is already existing naming for that: "minimum delta to be
considered noise" is called "sample threshold" while the timerlat one
is called "stop tracing threshold" (well, the source simply calls the
latter "stop tracing" or "stop"); timerlat also has another threshold,
the stack threshold (specified with -s).
The patchset (well, the v2 of it) has been already merged upstream, so
I'd prefer keeping the naming for rtla-timerlat, and if this is
implemented in rtla-osnoise in the future, a different name can be
used (--on-stop/--on-stop-threshold maybe?). The documentation clearly
says the threshold is the one specified by either -i or -T [1]. What
do you think?
Tomas
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/tools/rtla/common_timerlat_options.rst#n62
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 3/8] rtla/timerlat_bpf: Allow resuming tracing
2025-06-11 13:56 [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Tomas Glozar
2025-06-11 13:56 ` [PATCH 1/8] rtla/timerlat: Introduce enum timerlat_tracing_mode Tomas Glozar
2025-06-11 13:56 ` [PATCH 2/8] rtla/timerlat: Add action on threshold feature Tomas Glozar
@ 2025-06-11 13:56 ` Tomas Glozar
2025-06-11 13:56 ` [PATCH 4/8] rtla/timerlat: Add continue action Tomas Glozar
` (5 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Tomas Glozar @ 2025-06-11 13:56 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Arnaldo Carvalho de Melo, Chang Yin, Costa Shulyupin,
Tomas Glozar
Currently, rtla-timerlat BPF program uses a global variable stored in a
.bss section to store whether tracing has been stopped.
Move the information to a separate map, so that it is easily writable
from userspace, and add a function that clears the value, resuming
tracing after it has been stopped.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/src/timerlat.bpf.c | 13 +++++++++----
tools/tracing/rtla/src/timerlat_bpf.c | 13 +++++++++++++
tools/tracing/rtla/src/timerlat_bpf.h | 3 +++
3 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/tools/tracing/rtla/src/timerlat.bpf.c b/tools/tracing/rtla/src/timerlat.bpf.c
index 96196d46e170..084cd10c21fc 100644
--- a/tools/tracing/rtla/src/timerlat.bpf.c
+++ b/tools/tracing/rtla/src/timerlat.bpf.c
@@ -28,6 +28,13 @@ struct {
__type(value, unsigned long long);
} summary_irq SEC(".maps"), summary_thread SEC(".maps"), summary_user SEC(".maps");
+struct {
+ __uint(type, BPF_MAP_TYPE_ARRAY);
+ __uint(max_entries, 1);
+ __type(key, unsigned int);
+ __type(value, unsigned long long);
+} stop_tracing SEC(".maps");
+
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
__uint(max_entries, 1);
@@ -41,8 +48,6 @@ const volatile int irq_threshold;
const volatile int thread_threshold;
const volatile bool aa_only;
-int stop_tracing;
-
nosubprog unsigned long long map_get(void *map,
unsigned int key)
{
@@ -109,7 +114,7 @@ nosubprog void set_stop_tracing(void)
int value = 0;
/* Suppress further sample processing */
- stop_tracing = 1;
+ map_set(&stop_tracing, 0, 1);
/* Signal to userspace */
bpf_ringbuf_output(&signal_stop_tracing, &value, sizeof(value), 0);
@@ -121,7 +126,7 @@ int handle_timerlat_sample(struct trace_event_raw_timerlat_sample *tp_args)
unsigned long long latency, latency_us;
int bucket;
- if (stop_tracing)
+ if (map_get(&stop_tracing, 0))
return 0;
latency = tp_args->timer_latency / output_divisor;
diff --git a/tools/tracing/rtla/src/timerlat_bpf.c b/tools/tracing/rtla/src/timerlat_bpf.c
index 0bc44ce5d69b..1666215dd687 100644
--- a/tools/tracing/rtla/src/timerlat_bpf.c
+++ b/tools/tracing/rtla/src/timerlat_bpf.c
@@ -106,6 +106,19 @@ int timerlat_bpf_wait(int timeout)
return retval;
}
+/*
+ * timerlat_bpf_restart_tracing - restart stopped tracing
+ */
+int timerlat_bpf_restart_tracing(void)
+{
+ unsigned int key = 0;
+ unsigned long long value = 0;
+
+ return bpf_map__update_elem(bpf->maps.stop_tracing,
+ &key, sizeof(key),
+ &value, sizeof(value), BPF_ANY);
+}
+
static int get_value(struct bpf_map *map_irq,
struct bpf_map *map_thread,
struct bpf_map *map_user,
diff --git a/tools/tracing/rtla/src/timerlat_bpf.h b/tools/tracing/rtla/src/timerlat_bpf.h
index f1b54dbddb0e..118487436d30 100644
--- a/tools/tracing/rtla/src/timerlat_bpf.h
+++ b/tools/tracing/rtla/src/timerlat_bpf.h
@@ -18,6 +18,7 @@ int timerlat_bpf_attach(void);
void timerlat_bpf_detach(void);
void timerlat_bpf_destroy(void);
int timerlat_bpf_wait(int timeout);
+int timerlat_bpf_restart_tracing(void);
int timerlat_bpf_get_hist_value(int key,
long long *value_irq,
long long *value_thread,
@@ -28,6 +29,7 @@ int timerlat_bpf_get_summary_value(enum summary_field key,
long long *value_thread,
long long *value_user,
int cpus);
+
static inline int have_libbpf_support(void) { return 1; }
#else
static inline int timerlat_bpf_init(struct timerlat_params *params)
@@ -38,6 +40,7 @@ static inline int timerlat_bpf_attach(void) { return -1; }
static inline void timerlat_bpf_detach(void) { };
static inline void timerlat_bpf_destroy(void) { };
static inline int timerlat_bpf_wait(int timeout) { return -1; }
+static inline int timerlat_bpf_restart_tracing(void) { return -1; };
static inline int timerlat_bpf_get_hist_value(int key,
long long *value_irq,
long long *value_thread,
--
2.49.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH 4/8] rtla/timerlat: Add continue action
2025-06-11 13:56 [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Tomas Glozar
` (2 preceding siblings ...)
2025-06-11 13:56 ` [PATCH 3/8] rtla/timerlat_bpf: Allow resuming tracing Tomas Glozar
@ 2025-06-11 13:56 ` Tomas Glozar
2025-06-11 13:56 ` [PATCH 5/8] rtla/timerlat: Add action on end feature Tomas Glozar
` (4 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Tomas Glozar @ 2025-06-11 13:56 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Arnaldo Carvalho de Melo, Chang Yin, Costa Shulyupin,
Tomas Glozar
Introduce option to resume tracing after a latency threshold overflow.
The option is implemented as an action, specified using "-A continue".
Example:
$ rtla timerlat top -q -T 200 -d 1s -A exec,command="echo Threshold" \
-A continue
Threshold
Threshold
Threshold
Timer Latency
...
The feature is supported for both hist and top. After the continue
action is executed, processing of the list of actions is stopped and
tracing is resumed.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/src/actions.c | 27 +++++++++++-
tools/tracing/rtla/src/actions.h | 5 ++-
tools/tracing/rtla/src/timerlat_hist.c | 40 +++++++++++++++---
tools/tracing/rtla/src/timerlat_top.c | 57 ++++++++++++++++----------
4 files changed, 100 insertions(+), 29 deletions(-)
diff --git a/tools/tracing/rtla/src/actions.c b/tools/tracing/rtla/src/actions.c
index 63bee5bdabfd..aaf0808125d7 100644
--- a/tools/tracing/rtla/src/actions.c
+++ b/tools/tracing/rtla/src/actions.c
@@ -17,6 +17,7 @@ actions_init(struct actions *self)
self->size = action_default_size;
self->list = calloc(self->size, sizeof(struct action));
self->len = 0;
+ self->continue_flag = false;
memset(&self->present, 0, sizeof(self->present));
@@ -108,6 +109,20 @@ actions_add_shell(struct actions *self, const char *command)
return 0;
}
+/*
+ * actions_add_continue - add an action to resume measurement
+ */
+int
+actions_add_continue(struct actions *self)
+{
+ struct action *action = actions_new(self);
+
+ self->present[ACTION_CONTINUE] = true;
+ action->type = ACTION_CONTINUE;
+
+ return 0;
+}
+
/*
* actions_parse - add an action based on text specification
*/
@@ -133,6 +148,8 @@ actions_parse(struct actions *self, const char *trigger)
type = ACTION_SIGNAL;
else if (strcmp(token, "shell") == 0)
type = ACTION_SHELL;
+ else if (strcmp(token, "continue") == 0)
+ type = ACTION_CONTINUE;
else
/* Invalid trigger type */
return -1;
@@ -187,6 +204,11 @@ actions_parse(struct actions *self, const char *trigger)
if (strlen(token) > 8 && strncmp(token, "command=", 8) == 0)
return actions_add_shell(self, token + 8);
return -1;
+ case ACTION_CONTINUE:
+ /* Takes no argument */
+ if (token != NULL)
+ return -1;
+ return actions_add_continue(self);
default:
return -1;
}
@@ -196,7 +218,7 @@ actions_parse(struct actions *self, const char *trigger)
* actions_perform - perform all actions
*/
int
-actions_perform(const struct actions *self)
+actions_perform(struct actions *self)
{
int pid, retval;
const struct action *action;
@@ -226,6 +248,9 @@ actions_perform(const struct actions *self)
if (retval)
return retval;
break;
+ case ACTION_CONTINUE:
+ self->continue_flag = true;
+ return 0;
default:
break;
}
diff --git a/tools/tracing/rtla/src/actions.h b/tools/tracing/rtla/src/actions.h
index 076bbff8519e..b10a19d55c49 100644
--- a/tools/tracing/rtla/src/actions.h
+++ b/tools/tracing/rtla/src/actions.h
@@ -7,6 +7,7 @@ enum action_type {
ACTION_TRACE_OUTPUT,
ACTION_SIGNAL,
ACTION_SHELL,
+ ACTION_CONTINUE,
ACTION_FIELD_N
};
@@ -35,6 +36,7 @@ struct actions {
struct action *list;
int len, size;
bool present[ACTION_FIELD_N];
+ bool continue_flag;
/* External dependencies */
struct tracefs_instance *trace_output_inst;
@@ -45,5 +47,6 @@ void actions_destroy(struct actions *self);
int actions_add_trace_output(struct actions *self, const char *trace_output);
int actions_add_signal(struct actions *self, int signal, int pid);
int actions_add_shell(struct actions *self, const char *command);
+int actions_add_continue(struct actions *self);
int actions_parse(struct actions *self, const char *trigger);
-int actions_perform(const struct actions *self);
+int actions_perform(struct actions *self);
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index 06ee3d925ef9..1feae978344b 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -1375,8 +1375,20 @@ int timerlat_hist_main(int argc, char *argv[])
goto out_hist;
}
- if (osnoise_trace_is_off(tool, record))
- break;
+ if (osnoise_trace_is_off(tool, record)) {
+ actions_perform(¶ms->actions);
+
+ if (!params->actions.continue_flag)
+ /* continue flag not set, break */
+ break;
+
+ /* continue action reached, re-enable tracing */
+ if (params->actions.present[ACTION_TRACE_OUTPUT])
+ trace_instance_start(&record->trace);
+ if (!params->no_aa)
+ trace_instance_start(&aa->trace);
+ trace_instance_start(trace);
+ }
/* is there still any user-threads ? */
if (params->user_workload) {
@@ -1386,8 +1398,27 @@ int timerlat_hist_main(int argc, char *argv[])
}
}
}
- } else
- timerlat_bpf_wait(-1);
+ } else {
+ while (!stop_tracing) {
+ timerlat_bpf_wait(-1);
+
+ if (!stop_tracing) {
+ /* Threshold overflow, perform actions on threshold */
+ actions_perform(¶ms->actions);
+
+ if (!params->actions.continue_flag)
+ /* continue flag not set, break */
+ break;
+
+ /* continue action reached, re-enable tracing */
+ if (params->actions.present[ACTION_TRACE_OUTPUT])
+ trace_instance_start(&record->trace);
+ if (!params->no_aa)
+ trace_instance_start(&aa->trace);
+ timerlat_bpf_restart_tracing();
+ }
+ }
+ }
if (params->mode != TRACING_MODE_TRACEFS) {
timerlat_bpf_detach();
@@ -1413,7 +1444,6 @@ int timerlat_hist_main(int argc, char *argv[])
if (!params->no_aa)
timerlat_auto_analysis(params->stop_us, params->stop_total_us);
- actions_perform(¶ms->actions);
return_value = FAILED;
}
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index fa1072c0d126..ff0e6a7c3a1b 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -906,6 +906,7 @@ timerlat_top_set_signals(struct timerlat_params *params)
static int
timerlat_top_main_loop(struct osnoise_tool *top,
struct osnoise_tool *record,
+ struct osnoise_tool *aa,
struct timerlat_params *params,
struct timerlat_u_params *params_u)
{
@@ -932,8 +933,20 @@ timerlat_top_main_loop(struct osnoise_tool *top,
if (!params->quiet)
timerlat_print_stats(params, top);
- if (osnoise_trace_is_off(top, record))
- break;
+ if (osnoise_trace_is_off(top, record)) {
+ actions_perform(¶ms->actions);
+
+ if (!params->actions.continue_flag)
+ /* continue flag not set, break */
+ break;
+
+ /* continue action reached, re-enable tracing */
+ if (params->actions.present[ACTION_TRACE_OUTPUT])
+ trace_instance_start(&record->trace);
+ if (!params->no_aa)
+ trace_instance_start(&aa->trace);
+ trace_instance_start(trace);
+ }
/* is there still any user-threads ? */
if (params->user_workload) {
@@ -953,6 +966,7 @@ timerlat_top_main_loop(struct osnoise_tool *top,
static int
timerlat_top_bpf_main_loop(struct osnoise_tool *top,
struct osnoise_tool *record,
+ struct osnoise_tool *aa,
struct timerlat_params *params,
struct timerlat_u_params *params_u)
{
@@ -964,22 +978,9 @@ timerlat_top_bpf_main_loop(struct osnoise_tool *top,
return 0;
}
- if (params->quiet) {
- /* Quiet mode: wait for stop and then, print results */
- timerlat_bpf_wait(-1);
-
- retval = timerlat_top_bpf_pull_data(top);
- if (retval) {
- err_msg("Error pulling BPF data\n");
- return retval;
- }
-
- return 0;
- }
-
/* Pull and display data in a loop */
while (!stop_tracing) {
- wait_retval = timerlat_bpf_wait(params->sleep_time);
+ wait_retval = timerlat_bpf_wait(params->quiet ? -1 : params->sleep_time);
retval = timerlat_top_bpf_pull_data(top);
if (retval) {
@@ -987,11 +988,24 @@ timerlat_top_bpf_main_loop(struct osnoise_tool *top,
return retval;
}
- timerlat_print_stats(params, top);
+ if (!params->quiet)
+ timerlat_print_stats(params, top);
- if (wait_retval == 1)
+ if (wait_retval == 1) {
/* Stopping requested by tracer */
- break;
+ actions_perform(¶ms->actions);
+
+ if (!params->actions.continue_flag)
+ /* continue flag not set, break */
+ break;
+
+ /* continue action reached, re-enable tracing */
+ if (params->actions.present[ACTION_TRACE_OUTPUT])
+ trace_instance_start(&record->trace);
+ if (!params->no_aa)
+ trace_instance_start(&aa->trace);
+ timerlat_bpf_restart_tracing();
+ }
/* is there still any user-threads ? */
if (params->user_workload) {
@@ -1205,9 +1219,9 @@ int timerlat_top_main(int argc, char *argv[])
timerlat_top_set_signals(params);
if (params->mode == TRACING_MODE_TRACEFS)
- retval = timerlat_top_main_loop(top, record, params, ¶ms_u);
+ retval = timerlat_top_main_loop(top, record, aa, params, ¶ms_u);
else
- retval = timerlat_top_bpf_main_loop(top, record, params, ¶ms_u);
+ retval = timerlat_top_bpf_main_loop(top, record, aa, params, ¶ms_u);
if (retval)
goto out_top;
@@ -1230,7 +1244,6 @@ int timerlat_top_main(int argc, char *argv[])
if (!params->no_aa)
timerlat_auto_analysis(params->stop_us, params->stop_total_us);
- actions_perform(¶ms->actions);
return_value = FAILED;
} else if (params->aa_only) {
/*
--
2.49.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH 5/8] rtla/timerlat: Add action on end feature
2025-06-11 13:56 [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Tomas Glozar
` (3 preceding siblings ...)
2025-06-11 13:56 ` [PATCH 4/8] rtla/timerlat: Add continue action Tomas Glozar
@ 2025-06-11 13:56 ` Tomas Glozar
2025-06-11 13:56 ` [PATCH 6/8] rtla/tests: Check rtla output with grep Tomas Glozar
` (3 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Tomas Glozar @ 2025-06-11 13:56 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Arnaldo Carvalho de Melo, Chang Yin, Costa Shulyupin,
Tomas Glozar
Implement actions on end next to actions on threshold. A new option,
-N/--on-end is added, parallel to -A/--on-threshold. Instead of being
executed whenever a latency threshold is reached, it is executed at the
end of the measurement.
For example:
$ rtla timerlat hist -d 5s -N trace
will save the trace output at the end.
All actions supported by -A are also supported by -N, except for
continue, which does nothing with -N.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/src/timerlat.h | 5 ++-
tools/tracing/rtla/src/timerlat_hist.c | 46 +++++++++++++++++--------
tools/tracing/rtla/src/timerlat_top.c | 47 ++++++++++++++++++--------
3 files changed, 67 insertions(+), 31 deletions(-)
diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/timerlat.h
index d1fcf9a97621..bc55ed04fc96 100644
--- a/tools/tracing/rtla/src/timerlat.h
+++ b/tools/tracing/rtla/src/timerlat.h
@@ -48,7 +48,10 @@ struct timerlat_params {
struct sched_attr sched_param;
struct trace_events *events;
enum timerlat_tracing_mode mode;
- struct actions actions;
+
+ struct actions threshold_actions;
+ struct actions end_actions;
+
union {
struct {
/* top only */
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index 1feae978344b..1b4d5ddaa676 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -754,6 +754,7 @@ static void timerlat_hist_usage(char *usage)
" -u/--user-threads: use rtla user-space threads instead of kernel-space timerlat threads",
" -k/--kernel-threads: use timerlat kernel-space threads instead of rtla user-space threads",
" -A/--on-threshold <action>: define action to be executed at latency threshold, multiple -A are allowed",
+ " -N/--on-end: define action to be executed at measurement end, multiple -N are allowed"
" -U/--user-load: enable timerlat for user-defined user-space workload",
" --warm-up s: let the workload run for s seconds before collecting data",
" --trace-buffer-size kB: set the per-cpu trace buffer size in kB",
@@ -793,7 +794,8 @@ static struct timerlat_params
if (!params)
exit(1);
- actions_init(¶ms->actions);
+ actions_init(¶ms->threshold_actions);
+ actions_init(¶ms->end_actions);
/* disabled by default */
params->dma_latency = -1;
@@ -822,6 +824,7 @@ static struct timerlat_params
{"house-keeping", required_argument, 0, 'H'},
{"help", no_argument, 0, 'h'},
{"irq", required_argument, 0, 'i'},
+ {"on-end", required_argument, 0, 'N'},
{"nano", no_argument, 0, 'n'},
{"period", required_argument, 0, 'p'},
{"priority", required_argument, 0, 'P'},
@@ -853,7 +856,7 @@ static struct timerlat_params
int option_index = 0;
c = getopt_long(argc, argv,
- "A:a:c:C::b:d:e:E:DhH:i:knp:P:s:t::T:uU0123456:7:8:9\1\2:\3:",
+ "A:a:c:C::b:d:e:E:DhH:i:kN:np:P:s:t::T:uU0123456:7:8:9\1\2:\3:",
long_options, &option_index);
/* detect the end of the options. */
@@ -876,7 +879,7 @@ static struct timerlat_params
break;
case 'A':
- retval = actions_parse(¶ms->actions, optarg);
+ retval = actions_parse(¶ms->threshold_actions, optarg);
if (retval) {
err_msg("Invalid action %s\n", optarg);
exit(EXIT_FAILURE);
@@ -946,6 +949,13 @@ static struct timerlat_params
case 'k':
params->kernel_workload = 1;
break;
+ case 'N':
+ retval = actions_parse(¶ms->end_actions, optarg);
+ if (retval) {
+ err_msg("Invalid action %s\n", optarg);
+ exit(EXIT_FAILURE);
+ }
+ break;
case 'n':
params->output_divisor = 1;
break;
@@ -1051,7 +1061,7 @@ static struct timerlat_params
}
if (trace_output)
- actions_add_trace_output(¶ms->actions, trace_output);
+ actions_add_trace_output(¶ms->threshold_actions, trace_output);
if (geteuid()) {
err_msg("rtla needs root permission\n");
@@ -1078,7 +1088,8 @@ static struct timerlat_params
* mixed mode
*/
if (params->mode == TRACING_MODE_BPF &&
- (params->actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa))
+ (params->threshold_actions.present[ACTION_TRACE_OUTPUT] ||
+ params->end_actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa))
params->mode = TRACING_MODE_MIXED;
return params;
@@ -1271,13 +1282,15 @@ int timerlat_hist_main(int argc, char *argv[])
}
}
- if (params->actions.present[ACTION_TRACE_OUTPUT]) {
+ if (params->threshold_actions.present[ACTION_TRACE_OUTPUT] ||
+ params->end_actions.present[ACTION_TRACE_OUTPUT]) {
record = osnoise_init_trace_tool("timerlat");
if (!record) {
err_msg("Failed to enable the trace instance\n");
goto out_free;
}
- params->actions.trace_output_inst = record->trace.inst;
+ params->threshold_actions.trace_output_inst = record->trace.inst;
+ params->end_actions.trace_output_inst = record->trace.inst;
if (params->events) {
retval = trace_events_enable(&record->trace, params->events);
@@ -1343,7 +1356,7 @@ int timerlat_hist_main(int argc, char *argv[])
* tracing while enabling other instances. The trace instance is the
* one with most valuable information.
*/
- if (params->actions.present[ACTION_TRACE_OUTPUT])
+ if (record)
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
@@ -1376,14 +1389,14 @@ int timerlat_hist_main(int argc, char *argv[])
}
if (osnoise_trace_is_off(tool, record)) {
- actions_perform(¶ms->actions);
+ actions_perform(¶ms->threshold_actions);
- if (!params->actions.continue_flag)
+ if (!params->threshold_actions.continue_flag)
/* continue flag not set, break */
break;
/* continue action reached, re-enable tracing */
- if (params->actions.present[ACTION_TRACE_OUTPUT])
+ if (record)
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
@@ -1404,14 +1417,14 @@ int timerlat_hist_main(int argc, char *argv[])
if (!stop_tracing) {
/* Threshold overflow, perform actions on threshold */
- actions_perform(¶ms->actions);
+ actions_perform(¶ms->threshold_actions);
- if (!params->actions.continue_flag)
+ if (!params->threshold_actions.continue_flag)
/* continue flag not set, break */
break;
/* continue action reached, re-enable tracing */
- if (params->actions.present[ACTION_TRACE_OUTPUT])
+ if (record)
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
@@ -1436,6 +1449,8 @@ int timerlat_hist_main(int argc, char *argv[])
timerlat_print_stats(params, tool);
+ actions_perform(¶ms->end_actions);
+
return_value = PASSED;
if (osnoise_trace_is_off(tool, record) && !stop_tracing) {
@@ -1465,7 +1480,8 @@ int timerlat_hist_main(int argc, char *argv[])
osnoise_destroy_tool(aa);
osnoise_destroy_tool(record);
osnoise_destroy_tool(tool);
- actions_destroy(¶ms->actions);
+ actions_destroy(¶ms->threshold_actions);
+ actions_destroy(¶ms->end_actions);
if (params->mode != TRACING_MODE_TRACEFS)
timerlat_bpf_destroy();
free(params);
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index ff0e6a7c3a1b..f022fbd9435e 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -514,6 +514,7 @@ static void timerlat_top_usage(char *usage)
" -k/--kernel-threads: use timerlat kernel-space threads instead of rtla user-space threads",
" -U/--user-load: enable timerlat for user-defined user-space workload",
" -A/--on-threshold <action>: define action to be executed at latency threshold, multiple -A are allowed",
+ " -N/--on-end: define action to be executed at measurement end, multiple -N are allowed"
" --warm-up s: let the workload run for s seconds before collecting data",
" --trace-buffer-size kB: set the per-cpu trace buffer size in kB",
" --deepest-idle-state n: only go down to idle state n on cpus used by timerlat to reduce exit from idle latency",
@@ -552,7 +553,8 @@ static struct timerlat_params
if (!params)
exit(1);
- actions_init(¶ms->actions);
+ actions_init(¶ms->threshold_actions);
+ actions_init(¶ms->end_actions);
/* disabled by default */
params->dma_latency = -1;
@@ -578,6 +580,7 @@ static struct timerlat_params
{"help", no_argument, 0, 'h'},
{"house-keeping", required_argument, 0, 'H'},
{"irq", required_argument, 0, 'i'},
+ {"on-end", required_argument, 0, 'N'},
{"nano", no_argument, 0, 'n'},
{"period", required_argument, 0, 'p'},
{"priority", required_argument, 0, 'P'},
@@ -603,7 +606,7 @@ static struct timerlat_params
/* getopt_long stores the option index here. */
int option_index = 0;
- c = getopt_long(argc, argv, "A:a:c:C::d:De:hH:i:knp:P:qs:t::T:uU0:1:2:345:6:7:",
+ c = getopt_long(argc, argv, "A:a:c:C::d:De:hH:i:kN:np:P:qs:t::T:uU0:1:2:345:6:7:",
long_options, &option_index);
/* detect the end of the options. */
@@ -623,9 +626,10 @@ static struct timerlat_params
/* set trace */
trace_output = "timerlat_trace.txt";
+
break;
case 'A':
- retval = actions_parse(¶ms->actions, optarg);
+ retval = actions_parse(¶ms->threshold_actions, optarg);
if (retval) {
err_msg("Invalid action %s\n", optarg);
exit(EXIT_FAILURE);
@@ -698,6 +702,13 @@ static struct timerlat_params
case 'k':
params->kernel_workload = true;
break;
+ case 'N':
+ retval = actions_parse(¶ms->end_actions, optarg);
+ if (retval) {
+ err_msg("Invalid action %s\n", optarg);
+ exit(EXIT_FAILURE);
+ }
+ break;
case 'n':
params->output_divisor = 1;
break;
@@ -788,7 +799,7 @@ static struct timerlat_params
}
if (trace_output)
- actions_add_trace_output(¶ms->actions, trace_output);
+ actions_add_trace_output(¶ms->threshold_actions, trace_output);
if (geteuid()) {
err_msg("rtla needs root permission\n");
@@ -812,7 +823,8 @@ static struct timerlat_params
* mixed mode
*/
if (params->mode == TRACING_MODE_BPF &&
- (params->actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa))
+ (params->threshold_actions.present[ACTION_TRACE_OUTPUT] ||
+ params->end_actions.present[ACTION_TRACE_OUTPUT] || !params->no_aa))
params->mode = TRACING_MODE_MIXED;
return params;
@@ -934,14 +946,14 @@ timerlat_top_main_loop(struct osnoise_tool *top,
timerlat_print_stats(params, top);
if (osnoise_trace_is_off(top, record)) {
- actions_perform(¶ms->actions);
+ actions_perform(¶ms->threshold_actions);
- if (!params->actions.continue_flag)
+ if (!params->threshold_actions.continue_flag)
/* continue flag not set, break */
break;
/* continue action reached, re-enable tracing */
- if (params->actions.present[ACTION_TRACE_OUTPUT])
+ if (record)
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
@@ -993,14 +1005,14 @@ timerlat_top_bpf_main_loop(struct osnoise_tool *top,
if (wait_retval == 1) {
/* Stopping requested by tracer */
- actions_perform(¶ms->actions);
+ actions_perform(¶ms->threshold_actions);
- if (!params->actions.continue_flag)
+ if (!params->threshold_actions.continue_flag)
/* continue flag not set, break */
break;
/* continue action reached, re-enable tracing */
- if (params->actions.present[ACTION_TRACE_OUTPUT])
+ if (record)
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
@@ -1128,13 +1140,15 @@ int timerlat_top_main(int argc, char *argv[])
}
}
- if (params->actions.present[ACTION_TRACE_OUTPUT]) {
+ if (params->threshold_actions.present[ACTION_TRACE_OUTPUT] ||
+ params->end_actions.present[ACTION_TRACE_OUTPUT]) {
record = osnoise_init_trace_tool("timerlat");
if (!record) {
err_msg("Failed to enable the trace instance\n");
goto out_free;
}
- params->actions.trace_output_inst = record->trace.inst;
+ params->threshold_actions.trace_output_inst = record->trace.inst;
+ params->end_actions.trace_output_inst = record->trace.inst;
if (params->events) {
retval = trace_events_enable(&record->trace, params->events);
@@ -1201,7 +1215,7 @@ int timerlat_top_main(int argc, char *argv[])
* tracing while enabling other instances. The trace instance is the
* one with most valuable information.
*/
- if (params->actions.present[ACTION_TRACE_OUTPUT])
+ if (record)
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
@@ -1236,6 +1250,8 @@ int timerlat_top_main(int argc, char *argv[])
timerlat_print_stats(params, top);
+ actions_perform(¶ms->end_actions);
+
return_value = PASSED;
if (osnoise_trace_is_off(top, record) && !stop_tracing) {
@@ -1276,7 +1292,8 @@ int timerlat_top_main(int argc, char *argv[])
osnoise_destroy_tool(aa);
osnoise_destroy_tool(record);
osnoise_destroy_tool(top);
- actions_destroy(¶ms->actions);
+ actions_destroy(¶ms->threshold_actions);
+ actions_destroy(¶ms->end_actions);
if (params->mode != TRACING_MODE_TRACEFS)
timerlat_bpf_destroy();
free(params);
--
2.49.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH 6/8] rtla/tests: Check rtla output with grep
2025-06-11 13:56 [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Tomas Glozar
` (4 preceding siblings ...)
2025-06-11 13:56 ` [PATCH 5/8] rtla/timerlat: Add action on end feature Tomas Glozar
@ 2025-06-11 13:56 ` Tomas Glozar
2025-06-11 13:56 ` [PATCH 7/8] rtla/tests: Add tests for actions Tomas Glozar
` (2 subsequent siblings)
8 siblings, 0 replies; 13+ messages in thread
From: Tomas Glozar @ 2025-06-11 13:56 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Arnaldo Carvalho de Melo, Chang Yin, Costa Shulyupin,
Tomas Glozar
Add argument to the check command in the test suite that takes a regular
expression that the output of rtla command is checked against. This
allows testing for specific information in rtla output in addition
to checking the return value.
Two minor improvements are included: running rtla with "eval" so that
arguments with spaces can be passed to it via shell quotations, and
the stdout of pushd and popd is suppressed to clean up the test output.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/tests/engine.sh | 21 +++++++++++++++++----
1 file changed, 17 insertions(+), 4 deletions(-)
diff --git a/tools/tracing/rtla/tests/engine.sh b/tools/tracing/rtla/tests/engine.sh
index f2616a8e4179..64c5be4313de 100644
--- a/tools/tracing/rtla/tests/engine.sh
+++ b/tools/tracing/rtla/tests/engine.sh
@@ -11,7 +11,7 @@ test_begin() {
reset_osnoise() {
# Reset osnoise options to default and remove any dangling instances created
# by improperly exited rtla runs.
- pushd /sys/kernel/tracing || return 1
+ pushd /sys/kernel/tracing >/dev/null || return 1
# Remove dangling instances created by previous rtla run
echo 0 > tracing_thresh
@@ -35,11 +35,14 @@ reset_osnoise() {
echo 0 > stop_tracing_us
echo 1000 > timerlat_period_us
- popd
+ popd >/dev/null
}
check() {
+ test_name=$0
+ tested_command=$1
expected_exitcode=${3:-0}
+ expected_output=$4
# Simple check: run rtla with given arguments and test exit code.
# If TEST_COUNT is set, run the test. Otherwise, just count.
ctr=$(($ctr + 1))
@@ -49,8 +52,16 @@ check() {
[ "$NO_RESET_OSNOISE" == 1 ] || reset_osnoise
# Run rtla; in case of failure, include its output as comment
# in the test results.
- result=$(stdbuf -oL $TIMEOUT "$RTLA" $2 2>&1); exitcode=$?
- if [ $exitcode -eq $expected_exitcode ]
+ result=$(eval stdbuf -oL $TIMEOUT "$RTLA" $2 2>&1); exitcode=$?
+ # Test if the results matches if requested
+ if [ -n "$expected_output" ]
+ then
+ grep -E "$expected_output" <<< "$result" > /dev/null; grep_result=$?
+ else
+ grep_result=0
+ fi
+
+ if [ $exitcode -eq $expected_exitcode ] && [ $grep_result -eq 0 ]
then
echo "ok $ctr - $1"
else
@@ -58,6 +69,8 @@ check() {
# Add rtla output and exit code as comments in case of failure
echo "$result" | col -b | while read line; do echo "# $line"; done
printf "#\n# exit code %s\n" $exitcode
+ [ -n "$expected_output" ] && \
+ printf "# Output match failed: \"%s\"\n" "$expected_output"
fi
fi
}
--
2.49.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH 7/8] rtla/tests: Add tests for actions
2025-06-11 13:56 [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Tomas Glozar
` (5 preceding siblings ...)
2025-06-11 13:56 ` [PATCH 6/8] rtla/tests: Check rtla output with grep Tomas Glozar
@ 2025-06-11 13:56 ` Tomas Glozar
2025-06-11 13:56 ` [PATCH 8/8] rtla/tests: Limit duration to maximum of 10s Tomas Glozar
2025-06-11 14:46 ` [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Arnaldo Carvalho de Melo
8 siblings, 0 replies; 13+ messages in thread
From: Tomas Glozar @ 2025-06-11 13:56 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Arnaldo Carvalho de Melo, Chang Yin, Costa Shulyupin,
Tomas Glozar
Add a bunch of tests covering most of both action on threshold
(-A/--on-threshold) and action on end (-N/--on-end). Parts sensitive to
implementation of hist/top are tested for both.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/tests/timerlat.t | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index 579c12a85e8f..b810b36a04a3 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -18,6 +18,8 @@ fi
for option in $no_bpf_options
do
export RTLA_NO_BPF=$option
+
+# Basic tests
check "verify help page" \
"timerlat --help"
check "verify -s/--stack" \
@@ -36,6 +38,32 @@ check "verify -c/--cpus" \
"timerlat hist -c 0 -d 30s"
check "hist test in nanoseconds" \
"timerlat hist -i 2 -c 0 -n -d 30s" 2
+
+# Actions tests
+check "trace output through -t" \
+ "timerlat hist -T 2 -t" 2 "^ Saving trace to timerlat_trace.txt$"
+check "trace output through -t with custom filename" \
+ "timerlat hist -T 2 -t custom_filename.txt" 2 "^ Saving trace to custom_filename.txt$"
+check "trace output through -A trace" \
+ "timerlat hist -T 2 -A trace" 2 "^ Saving trace to timerlat_trace.txt$"
+check "trace output through -A trace with custom filename" \
+ "timerlat hist -T 2 -A trace,file=custom_filename.txt" 2 "^ Saving trace to custom_filename.txt$"
+check "exec command" \
+ "timerlat hist -T 2 -A shell,command='echo TestOutput'" 2 "^TestOutput$"
+check "multiple actions" \
+ "timerlat hist -T 2 -A shell,command='echo -n 1' -A shell,command='echo 2'" 2 "^12$"
+check "hist stop at failed action" \
+ "timerlat hist -T 2 -A shell,command='echo -n 1; false' -A shell,command='echo -n 2'" 2 "^1# RTLA timerlat histogram$"
+check "top stop at failed action" \
+ "timerlat top -T 2 -A shell,command='echo -n 1; false' -A shell,command='echo -n 2'" 2 "^1ALL"
+check "hist with continue" \
+ "timerlat hist -T 2 -d 1s -A shell,command='echo TestOutput' -A continue" 0 "^TestOutput$"
+check "top with continue" \
+ "timerlat top -q -T 2 -d 1s -A shell,command='echo TestOutput' -A continue" 0 "^TestOutput$"
+check "hist with trace output at end" \
+ "timerlat hist -d 1s -N trace" 0 "^ Saving trace to timerlat_trace.txt$"
+check "top with trace output at end" \
+ "timerlat top -d 1s -N trace" 0 "^ Saving trace to timerlat_trace.txt$"
done
test_end
--
2.49.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH 8/8] rtla/tests: Limit duration to maximum of 10s
2025-06-11 13:56 [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Tomas Glozar
` (6 preceding siblings ...)
2025-06-11 13:56 ` [PATCH 7/8] rtla/tests: Add tests for actions Tomas Glozar
@ 2025-06-11 13:56 ` Tomas Glozar
2025-06-11 14:46 ` [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Arnaldo Carvalho de Melo
8 siblings, 0 replies; 13+ messages in thread
From: Tomas Glozar @ 2025-06-11 13:56 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Arnaldo Carvalho de Melo, Chang Yin, Costa Shulyupin,
Tomas Glozar
Many of the original rtla tests included durations of 1 minute and 30
seconds. Experience has shown this is unnecessary, since 10 seconds as
waiting time for samples to appear.
Change duration of all rtla tests to at most 10 seconds. This speeds up
testing significantly.
Before:
$ make check
All tests successful.
Files=3, Tests=54, 536 wallclock secs
( 0.03 usr 0.00 sys + 20.31 cusr 22.02 csys = 42.36 CPU)
Result: PASS
After:
$ make check
...
All tests successful.
Files=3, Tests=54, 196 wallclock secs
( 0.03 usr 0.01 sys + 20.28 cusr 20.68 csys = 41.00 CPU)
Result: PASS
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/tests/hwnoise.t | 8 ++++----
tools/tracing/rtla/tests/osnoise.t | 4 ++--
tools/tracing/rtla/tests/timerlat.t | 8 ++++----
3 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/tools/tracing/rtla/tests/hwnoise.t b/tools/tracing/rtla/tests/hwnoise.t
index 5f71401a139e..448877564b8d 100644
--- a/tools/tracing/rtla/tests/hwnoise.t
+++ b/tools/tracing/rtla/tests/hwnoise.t
@@ -10,12 +10,12 @@ check "verify help page" \
check "detect noise higher than one microsecond" \
"hwnoise -c 0 -T 1 -d 5s -q"
check "set the automatic trace mode" \
- "hwnoise -a 5 -d 30s" 2
+ "hwnoise -a 5 -d 10s" 2
check "set scheduling param to the osnoise tracer threads" \
- "hwnoise -P F:1 -c 0 -r 900000 -d 1M -q"
+ "hwnoise -P F:1 -c 0 -r 900000 -d 10s -q"
check "stop the trace if a single sample is higher than 1 us" \
- "hwnoise -s 1 -T 1 -t -d 30s" 2
+ "hwnoise -s 1 -T 1 -t -d 10s" 2
check "enable a trace event trigger" \
- "hwnoise -t -e osnoise:irq_noise trigger=\"hist:key=desc,duration:sort=desc,duration:vals=hitcount\" -d 1m"
+ "hwnoise -t -e osnoise:irq_noise trigger=\"hist:key=desc,duration:sort=desc,duration:vals=hitcount\" -d 10s"
test_end
diff --git a/tools/tracing/rtla/tests/osnoise.t b/tools/tracing/rtla/tests/osnoise.t
index 44908fc01abf..6a4dfa31dc55 100644
--- a/tools/tracing/rtla/tests/osnoise.t
+++ b/tools/tracing/rtla/tests/osnoise.t
@@ -8,13 +8,13 @@ set_timeout 2m
check "verify help page" \
"osnoise --help"
check "verify the --priority/-P param" \
- "osnoise top -P F:1 -c 0 -r 900000 -d 1M -q"
+ "osnoise top -P F:1 -c 0 -r 900000 -d 10s -q"
check "verify the --stop/-s param" \
"osnoise top -s 30 -T 1 -t" 2
check "verify the --trace param" \
"osnoise hist -s 30 -T 1 -t" 2
check "verify the --entries/-E param" \
- "osnoise hist -P F:1 -c 0 -r 900000 -d 1M -b 10 -E 25"
+ "osnoise hist -P F:1 -c 0 -r 900000 -d 10s -b 10 -E 25"
# Test setting default period by putting an absurdly high period
# and stopping on threshold.
diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index b810b36a04a3..291d047420c3 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -25,9 +25,9 @@ check "verify help page" \
check "verify -s/--stack" \
"timerlat top -s 3 -T 10 -t" 2
check "verify -P/--priority" \
- "timerlat top -P F:1 -c 0 -d 1M -q"
+ "timerlat top -P F:1 -c 0 -d 10s -q"
check "test in nanoseconds" \
- "timerlat top -i 2 -c 0 -n -d 30s" 2
+ "timerlat top -i 2 -c 0 -n -d 10s" 2
check "set the automatic trace mode" \
"timerlat top -a 5 --dump-tasks" 2
check "print the auto-analysis if hits the stop tracing condition" \
@@ -35,9 +35,9 @@ check "print the auto-analysis if hits the stop tracing condition" \
check "disable auto-analysis" \
"timerlat top -s 3 -T 10 -t --no-aa" 2
check "verify -c/--cpus" \
- "timerlat hist -c 0 -d 30s"
+ "timerlat hist -c 0 -d 10s"
check "hist test in nanoseconds" \
- "timerlat hist -i 2 -c 0 -n -d 30s" 2
+ "timerlat hist -i 2 -c 0 -n -d 10s" 2
# Actions tests
check "trace output through -t" \
--
2.49.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end
2025-06-11 13:56 [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Tomas Glozar
` (7 preceding siblings ...)
2025-06-11 13:56 ` [PATCH 8/8] rtla/tests: Limit duration to maximum of 10s Tomas Glozar
@ 2025-06-11 14:46 ` Arnaldo Carvalho de Melo
2025-06-23 14:15 ` Tomas Glozar
8 siblings, 1 reply; 13+ messages in thread
From: Arnaldo Carvalho de Melo @ 2025-06-11 14:46 UTC (permalink / raw)
To: Tomas Glozar
Cc: Steven Rostedt, linux-trace-kernel, linux-kernel, John Kacur,
Luis Goncalves, Chang Yin, Costa Shulyupin
On Wed, Jun 11, 2025 at 03:56:36PM +0200, Tomas Glozar wrote:
> This series adds a feature that allows to user to specify certain
> kinds of "actions" to be executed at one of two places in the rtla
> measurement: when tracing is stopped on latency threshold, and at the
> end of tracing.
> Two new options are added: -A/--on-threshold, and -N/--on-end, taking
> the action as an argument. For example:
I wouldn't add -A and -N, leaving just the long options, as it documents
scripts (and we should have autocomplete as well), leaving the one
letter options for things that are used super frequently, which could be
these new options, after a while, time will tell :-)
But I see that "A"ction connection, and since you show it is used
multiple times in a single command line, maybe its warranted the
one-letter option.
> $ rtla timerlat hist -T 10 -A shell,command="echo Threshold" \
> -N shell,command="echo Tracing stopped"
> will print "Threshold" if a thread latency higher than 10 microseconds
> is reached, and "Tracing stopped" always at the end.
> The list of possible actions is extensible and is covered in
> the commit messages. Later, a documentation patch series will be sent
> with clear explanation of every action and its syntax.
I think having the documentation together with the new options is
desirable.
> Notably, a special action "continue" resumes tracing. For example:
> $ rtla timerlat hist -T 100 -A shell,command="echo Threshold" \
> -A continue -d 10s
so --on-threshold ends up being a list of things to do when the
threshold is hit?
> will print "Threshold" as many times as tracing is stopped after
> thread latency reaches 100us.
> The feature was inspired by a case where collecting perf data on rtla
> latency overflow was required, which can be done by sending a signal
> to the perf process.
> Example of this with Intel PT aux buffer:
> $ perf record --cpu 0 -e intel_pt// -S -- rtla timerlat top -q -T 100 \
> -c 0 -A signal,pid=parent,num=12 -A continue
> In general, the feature is aiming to allow integration with external
> tooling. To implement even more flexibility, passing context to the
> shell through environmental variables, or even an entire scripting
> language with access to the rtla internals can be implemented if
> needed.
That is an interesting example of cross-tool integration using existing
mechanisms for detecting special events and asking for hardware tracing
snapshots, good stuff!
At some point we need to have this signalling to not involve userspace,
shortcircuiting the snapshot request closer to the event of interest,
inside the kernel.
- Arnaldo
> Tomas Glozar (8):
> rtla/timerlat: Introduce enum timerlat_tracing_mode
> rtla/timerlat: Add action on threshold feature
> rtla/timerlat_bpf: Allow resuming tracing
> rtla/timerlat: Add continue action
> rtla/timerlat: Add action on end feature
> rtla/tests: Check rtla output with grep
> rtla/tests: Add tests for actions
> rtla/tests: Limit duration to maximum of 10s
>
> tools/tracing/rtla/src/Build | 1 +
> tools/tracing/rtla/src/actions.c | 260 +++++++++++++++++++++++++
> tools/tracing/rtla/src/actions.h | 52 +++++
> tools/tracing/rtla/src/timerlat.bpf.c | 13 +-
> tools/tracing/rtla/src/timerlat.c | 24 ++-
> tools/tracing/rtla/src/timerlat.h | 24 ++-
> tools/tracing/rtla/src/timerlat_bpf.c | 13 ++
> tools/tracing/rtla/src/timerlat_bpf.h | 3 +
> tools/tracing/rtla/src/timerlat_hist.c | 145 ++++++++++----
> tools/tracing/rtla/src/timerlat_top.c | 167 ++++++++++------
> tools/tracing/rtla/tests/engine.sh | 21 +-
> tools/tracing/rtla/tests/hwnoise.t | 8 +-
> tools/tracing/rtla/tests/osnoise.t | 4 +-
> tools/tracing/rtla/tests/timerlat.t | 36 +++-
> 14 files changed, 652 insertions(+), 119 deletions(-)
> create mode 100644 tools/tracing/rtla/src/actions.c
> create mode 100644 tools/tracing/rtla/src/actions.h
>
> --
> 2.49.0
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end
2025-06-11 14:46 ` [PATCH 0/8] rtla/timerlat: Support actions on threshold and on end Arnaldo Carvalho de Melo
@ 2025-06-23 14:15 ` Tomas Glozar
0 siblings, 0 replies; 13+ messages in thread
From: Tomas Glozar @ 2025-06-23 14:15 UTC (permalink / raw)
To: Arnaldo Carvalho de Melo
Cc: Steven Rostedt, linux-trace-kernel, linux-kernel, John Kacur,
Luis Goncalves, Chang Yin, Costa Shulyupin
st 11. 6. 2025 v 16:46 odesílatel Arnaldo Carvalho de Melo
<acme@kernel.org> napsal:
>
> I wouldn't add -A and -N, leaving just the long options, as it documents
> scripts (and we should have autocomplete as well), leaving the one
> letter options for things that are used super frequently, which could be
> these new options, after a while, time will tell :-)
>
Hmm, my reasoning for those is that one might have multiple actions,
and the action argument itself is long, so one would get a very long
command, e.g.:
$ rtla timerlat hist -T 10 --on-threshold shell,command="echo
Threshold" --on-end shell,command="echo Tracing stopped"
for the command from the example below. But it's true that this is an
experimental feature, and I don't even precisely know the direction in
which I'm going (which is to be determined based on the use of this in
practice). So your suggestion makes a lot of sense.
>
> I think having the documentation together with the new options is
> desirable.
>
Right, this is a user facing change. I did the documentation
separately before, but that was for a change in implementation (BPF
sample collection). Also, I did not have the documentation at the time
of sending of the patchset ready yet :) I'll add it to the v2.
>
> so --on-threshold ends up being a list of things to do when the
> threshold is hit?
>
Yes, the list is executed in order. Now when I'm looking at the cover
letter, this is not clear, I'm only talking about the "list" of the
supported actions (which I perhaps should more accurately call "set").
>
> That is an interesting example of cross-tool integration using existing
> mechanisms for detecting special events and asking for hardware tracing
> snapshots, good stuff!
>
Thanks!
> At some point we need to have this signalling to not involve userspace,
> shortcircuiting the snapshot request closer to the event of interest,
> inside the kernel.
>
I have a feature in mind for that. We already use a BPF program to
process the samples [note1], which means that BPF tail call [1] can be
used to implement in-kernel actions next to userspace ones. Those can
be built-in BPF programs, or custom BPF programs supplied by the user.
[1] https://docs.ebpf.io/linux/helper-function/bpf_tail_call
[note1] In BPF mode. However, outside BPF mode, actions are not that
useful in the first place, as they are only executed when rtla wakes
up to process samples, incurring up to 1s latency.
Tomas
^ permalink raw reply [flat|nested] 13+ messages in thread