[PATCH 0/4] rtla/timerlat: Add --bpf-action option

[PATCH 0/4] rtla/timerlat: Add --bpf-action option

[Tomas Glozar]

This patchset extends rtla-timerlat's BPF support with the option of
executing a user-supplied BPF program on latency threshold overflow.

See the supplied example and documentation for how to create a program.
bpf_tail_call() is used to chain the program with the built-in BPF
sample collection program, if the threshold is hit.

The feature can be used for both in-kernel data collection and sending
signals to userspace directly from the kernel, if the kernel version
allows it.

Note: The patchset will have to be rebased on top of [1], or vice versa,
since they both touch stop_tracing() ([1] adds one call of it, and this
patchset adds an extra argument to it).

I have contemplated adding this as --on-threshold bpf,... but it does
not fit the existing actions infrastructure very well, since the action
happens in the BPF program, not in RTLA, and only one BPF action is
supported.

[1] https://lore.kernel.org/linux-trace-kernel/20251006143100.137255-1-tglozar@redhat.com/

Tomas Glozar (4):
  rtla/timerlat: Support tail call from BPF program
  rtla/timerlat: Add --bpf-action option
  rtla/timerlat: Add example for BPF action program
  Documentation/rtla: Document --bpf-action option

 .../tools/rtla/common_timerlat_options.rst    | 18 +++++
 .../tracing/rtla/sample/timerlat_bpf_action.c | 16 +++++
 tools/tracing/rtla/src/timerlat.bpf.c         | 23 ++++++-
 tools/tracing/rtla/src/timerlat.c             | 11 +++
 tools/tracing/rtla/src/timerlat.h             |  2 +-
 tools/tracing/rtla/src/timerlat_bpf.c         | 67 +++++++++++++++++++
 tools/tracing/rtla/src/timerlat_bpf.h         |  7 +-
 tools/tracing/rtla/src/timerlat_hist.c        |  5 ++
 tools/tracing/rtla/src/timerlat_top.c         |  5 ++
 9 files changed, 149 insertions(+), 5 deletions(-)
 create mode 100644 tools/tracing/rtla/sample/timerlat_bpf_action.c

-- 
2.51.0

[PATCH 1/4] rtla/timerlat: Support tail call from BPF program

[Tomas Glozar]

Add a map to the rtla-timerlat BPF program that holds a file descriptor
of another BPF program, to be executed on threshold overflow.

timerlat_bpf_set_action() is added as an interface to set the program.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/src/timerlat.bpf.c | 23 ++++++++++++++++++++---
 tools/tracing/rtla/src/timerlat_bpf.c | 13 +++++++++++++
 tools/tracing/rtla/src/timerlat_bpf.h |  1 +
 3 files changed, 34 insertions(+), 3 deletions(-)

diff --git a/tools/tracing/rtla/src/timerlat.bpf.c b/tools/tracing/rtla/src/timerlat.bpf.c
index 084cd10c21fc..19ccd9abf8d4 100644
--- a/tools/tracing/rtla/src/timerlat.bpf.c
+++ b/tools/tracing/rtla/src/timerlat.bpf.c
@@ -40,6 +40,17 @@ struct {
 	__uint(max_entries, 1);
 } signal_stop_tracing SEC(".maps");
 
+struct {
+	__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
+	__uint(key_size, sizeof(unsigned int));
+	__uint(max_entries, 1);
+	__array(values, unsigned int (void *));
+} bpf_action SEC(".maps") = {
+	.values = {
+		[0] = 0
+	},
+};
+
 /* Params to be set by rtla */
 const volatile int bucket_size = 1;
 const volatile int output_divisor = 1000;
@@ -109,7 +120,7 @@ nosubprog void update_summary(void *map,
 	map_set(map, SUMMARY_SUM, map_get(map, SUMMARY_SUM) + latency);
 }
 
-nosubprog void set_stop_tracing(void)
+nosubprog void set_stop_tracing(struct trace_event_raw_timerlat_sample *tp_args)
 {
 	int value = 0;
 
@@ -118,6 +129,12 @@ nosubprog void set_stop_tracing(void)
 
 	/* Signal to userspace */
 	bpf_ringbuf_output(&signal_stop_tracing, &value, sizeof(value), 0);
+
+	/*
+	 * Call into BPF action program, if attached.
+	 * Otherwise, just silently fail.
+	 */
+	bpf_tail_call(tp_args, &bpf_action, 0);
 }
 
 SEC("tp/osnoise/timerlat_sample")
@@ -138,13 +155,13 @@ int handle_timerlat_sample(struct trace_event_raw_timerlat_sample *tp_args)
 		update_summary(&summary_irq, latency, bucket);
 
 		if (irq_threshold != 0 && latency_us >= irq_threshold)
-			set_stop_tracing();
+			set_stop_tracing(tp_args);
 	} else if (tp_args->context == 1) {
 		update_main_hist(&hist_thread, bucket);
 		update_summary(&summary_thread, latency, bucket);
 
 		if (thread_threshold != 0 && latency_us >= thread_threshold)
-			set_stop_tracing();
+			set_stop_tracing(tp_args);
 	} else {
 		update_main_hist(&hist_user, bucket);
 		update_summary(&summary_user, latency, bucket);
diff --git a/tools/tracing/rtla/src/timerlat_bpf.c b/tools/tracing/rtla/src/timerlat_bpf.c
index e97d16646bcd..1d619e502c65 100644
--- a/tools/tracing/rtla/src/timerlat_bpf.c
+++ b/tools/tracing/rtla/src/timerlat_bpf.c
@@ -59,6 +59,19 @@ int timerlat_bpf_init(struct timerlat_params *params)
 	return 0;
 }
 
+/*
+ * timerlat_bpf_set_action - set action on threshold executed on BPF side
+ */
+static int timerlat_bpf_set_action(struct bpf_program *prog)
+{
+	unsigned int key = 0, value = bpf_program__fd(prog);
+
+	return bpf_map__update_elem(bpf->maps.bpf_action,
+				    &key, sizeof(key),
+				    &value, sizeof(value),
+				    BPF_ANY);
+}
+
 /*
  * timerlat_bpf_attach - attach BPF program to collect timerlat data
  */
diff --git a/tools/tracing/rtla/src/timerlat_bpf.h b/tools/tracing/rtla/src/timerlat_bpf.h
index 118487436d30..b5009092c7a3 100644
--- a/tools/tracing/rtla/src/timerlat_bpf.h
+++ b/tools/tracing/rtla/src/timerlat_bpf.h
@@ -12,6 +12,7 @@ enum summary_field {
 };
 
 #ifndef __bpf__
+#include <bpf/libbpf.h>
 #ifdef HAVE_BPF_SKEL
 int timerlat_bpf_init(struct timerlat_params *params);
 int timerlat_bpf_attach(void);
-- 
2.51.0

[PATCH 2/4] rtla/timerlat: Add --bpf-action option

[Tomas Glozar]

Add option --bpf-action that allows the user to attach an external BPF
program that will be executed via BPF tail call on latency threshold
overflow.

Executing additional BPF code on latency threshold overflow allows doing
doing low-latency and in-kernel troubleshooting of the cause of the
overflow.

The option takes an argument, which is a path to a BPF ELF file
expected to contain a function named "action_handler" in a section named
"tp/timerlat_action" (the section is necessary for libbpf to asssign the
correct BPF program type to it).

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/src/timerlat.c      | 11 ++++++
 tools/tracing/rtla/src/timerlat.h      |  2 +-
 tools/tracing/rtla/src/timerlat_bpf.c  | 54 ++++++++++++++++++++++++++
 tools/tracing/rtla/src/timerlat_bpf.h  |  6 ++-
 tools/tracing/rtla/src/timerlat_hist.c |  5 +++
 tools/tracing/rtla/src/timerlat_top.c  |  5 +++
 6 files changed, 81 insertions(+), 2 deletions(-)

diff --git a/tools/tracing/rtla/src/timerlat.c b/tools/tracing/rtla/src/timerlat.c
index b69212874127..6907a323f9ec 100644
--- a/tools/tracing/rtla/src/timerlat.c
+++ b/tools/tracing/rtla/src/timerlat.c
@@ -48,6 +48,17 @@ timerlat_apply_config(struct osnoise_tool *tool, struct timerlat_params *params)
 		}
 	}
 
+	/* Check if BPF action program is requested but BPF is not available */
+	if (params->bpf_action_program) {
+		if (params->mode == TRACING_MODE_TRACEFS) {
+			err_msg("BPF actions are not supported in tracefs-only mode\n");
+			goto out_err;
+		}
+
+		if (timerlat_load_bpf_action_program(params->bpf_action_program))
+			goto out_err;
+	}
+
 	if (params->mode != TRACING_MODE_BPF) {
 		/*
 		 * In tracefs and mixed mode, timerlat tracer handles stopping
diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/timerlat.h
index fd6065f48bb7..8dd5d134ce08 100644
--- a/tools/tracing/rtla/src/timerlat.h
+++ b/tools/tracing/rtla/src/timerlat.h
@@ -27,6 +27,7 @@ struct timerlat_params {
 	int			dump_tasks;
 	int			deepest_idle_state;
 	enum timerlat_tracing_mode mode;
+	const char		*bpf_action_program;
 };
 
 #define to_timerlat_params(ptr) container_of(ptr, struct timerlat_params, common)
@@ -36,4 +37,3 @@ int timerlat_main(int argc, char *argv[]);
 int timerlat_enable(struct osnoise_tool *tool);
 void timerlat_analyze(struct osnoise_tool *tool, bool stopped);
 void timerlat_free(struct osnoise_tool *tool);
-
diff --git a/tools/tracing/rtla/src/timerlat_bpf.c b/tools/tracing/rtla/src/timerlat_bpf.c
index 1d619e502c65..3c63bf7aa607 100644
--- a/tools/tracing/rtla/src/timerlat_bpf.c
+++ b/tools/tracing/rtla/src/timerlat_bpf.c
@@ -7,6 +7,10 @@
 
 static struct timerlat_bpf *bpf;
 
+/* BPF object and program for action program */
+static struct bpf_object *obj;
+static struct bpf_program *prog;
+
 /*
  * timerlat_bpf_init - load and initialize BPF program to collect timerlat data
  */
@@ -56,6 +60,10 @@ int timerlat_bpf_init(struct timerlat_params *params)
 		return err;
 	}
 
+	/* Set BPF action program to NULL */
+	prog = NULL;
+	obj = NULL;
+
 	return 0;
 }
 
@@ -96,6 +104,11 @@ void timerlat_bpf_detach(void)
 void timerlat_bpf_destroy(void)
 {
 	timerlat_bpf__destroy(bpf);
+	bpf = NULL;
+	if (obj)
+		bpf_object__close(obj);
+	obj = NULL;
+	prog = NULL;
 }
 
 static int handle_rb_event(void *ctx, void *data, size_t data_sz)
@@ -190,4 +203,45 @@ int timerlat_bpf_get_summary_value(enum summary_field key,
 			 bpf->maps.summary_user,
 			 key, value_irq, value_thread, value_user, cpus);
 }
+
+/*
+ * timerlat_load_bpf_action_program - load and register a BPF action program
+ */
+int timerlat_load_bpf_action_program(const char *program_path)
+{
+	int err;
+
+	obj = bpf_object__open_file(program_path, NULL);
+	if (!obj) {
+		err_msg("Failed to open BPF action program: %s\n", program_path);
+		return -1;
+	}
+
+	err = bpf_object__load(obj);
+	if (err) {
+		err_msg("Failed to load BPF action program: %s\n", program_path);
+		return -1;
+	}
+
+	prog = bpf_object__find_program_by_name(obj, "action_handler");
+	if (!prog) {
+		err_msg("BPF action program must have 'action_handler' function: %s\n",
+			program_path);
+		bpf_object__close(obj);
+		obj = NULL;
+		return -1;
+	}
+
+	err = timerlat_bpf_set_action(prog);
+	if (err) {
+		err_msg("Failed to register BPF action program: %s\n", program_path);
+		bpf_object__close(obj);
+		obj = NULL;
+		prog = NULL;
+		return -1;
+	}
+
+	return 0;
+}
+
 #endif /* HAVE_BPF_SKEL */
diff --git a/tools/tracing/rtla/src/timerlat_bpf.h b/tools/tracing/rtla/src/timerlat_bpf.h
index b5009092c7a3..169abeaf4363 100644
--- a/tools/tracing/rtla/src/timerlat_bpf.h
+++ b/tools/tracing/rtla/src/timerlat_bpf.h
@@ -30,7 +30,7 @@ int timerlat_bpf_get_summary_value(enum summary_field key,
 				   long long *value_thread,
 				   long long *value_user,
 				   int cpus);
-
+int timerlat_load_bpf_action_program(const char *program_path);
 static inline int have_libbpf_support(void) { return 1; }
 #else
 static inline int timerlat_bpf_init(struct timerlat_params *params)
@@ -58,6 +58,10 @@ static inline int timerlat_bpf_get_summary_value(enum summary_field key,
 {
 	return -1;
 }
+static inline int timerlat_load_bpf_action_program(const char *program_path)
+{
+	return -1;
+}
 static inline int have_libbpf_support(void) { return 0; }
 #endif /* HAVE_BPF_SKEL */
 #endif /* __bpf__ */
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index 606c1688057b..5e639cc34f64 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -763,6 +763,7 @@ static void timerlat_hist_usage(char *usage)
 		"	     --deepest-idle-state n: only go down to idle state n on cpus used by timerlat to reduce exit from idle latency",
 		"	     --on-threshold <action>: define action to be executed at latency threshold, multiple are allowed",
 		"	     --on-end <action>: define action to be executed at measurement end, multiple are allowed",
+		"	     --bpf-action <program>: load and execute BPF program when latency threshold is exceeded",
 		NULL,
 	};
 
@@ -853,6 +854,7 @@ static struct common_params
 			{"deepest-idle-state",	required_argument,	0, '\4'},
 			{"on-threshold",	required_argument,	0, '\5'},
 			{"on-end",		required_argument,	0, '\6'},
+			{"bpf-action",		required_argument,	0, '\7'},
 			{0, 0, 0, 0}
 		};
 
@@ -1062,6 +1064,9 @@ static struct common_params
 				exit(EXIT_FAILURE);
 			}
 			break;
+		case '\7':
+			params->bpf_action_program = optarg;
+			break;
 		default:
 			timerlat_hist_usage("Invalid option");
 		}
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index fc479a0dcb59..da5d5db1bc17 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -521,6 +521,7 @@ static void timerlat_top_usage(char *usage)
 		"	     --deepest-idle-state n: only go down to idle state n on cpus used by timerlat to reduce exit from idle latency",
 		"	     --on-threshold <action>: define action to be executed at latency threshold, multiple are allowed",
 		"	     --on-end: define action to be executed at measurement end, multiple are allowed",
+		"	     --bpf-action <program>: load and execute BPF program when latency threshold is exceeded",
 		NULL,
 	};
 
@@ -603,6 +604,7 @@ static struct common_params
 			{"deepest-idle-state",	required_argument,	0, '8'},
 			{"on-threshold",	required_argument,	0, '9'},
 			{"on-end",		required_argument,	0, '\1'},
+			{"bpf-action",		required_argument,	0, '\2'},
 			{0, 0, 0, 0}
 		};
 
@@ -798,6 +800,9 @@ static struct common_params
 				exit(EXIT_FAILURE);
 			}
 			break;
+		case '\2':
+			params->bpf_action_program = optarg;
+			break;
 		default:
 			timerlat_top_usage("Invalid option");
 		}
-- 
2.51.0

Re: [PATCH 2/4] rtla/timerlat: Add --bpf-action option

[Wander Lairson Costa]

On Fri, Oct 17, 2025 at 04:46:48PM +0200, Tomas Glozar wrote:
> Add option --bpf-action that allows the user to attach an external BPF
> program that will be executed via BPF tail call on latency threshold
> overflow.
> 
> Executing additional BPF code on latency threshold overflow allows doing
> doing low-latency and in-kernel troubleshooting of the cause of the
> overflow.
> 
> The option takes an argument, which is a path to a BPF ELF file
> expected to contain a function named "action_handler" in a section named
> "tp/timerlat_action" (the section is necessary for libbpf to asssign the
> correct BPF program type to it).
> 
> Signed-off-by: Tomas Glozar <tglozar@redhat.com>
> ---
>  tools/tracing/rtla/src/timerlat.c      | 11 ++++++
>  tools/tracing/rtla/src/timerlat.h      |  2 +-
>  tools/tracing/rtla/src/timerlat_bpf.c  | 54 ++++++++++++++++++++++++++
>  tools/tracing/rtla/src/timerlat_bpf.h  |  6 ++-
>  tools/tracing/rtla/src/timerlat_hist.c |  5 +++
>  tools/tracing/rtla/src/timerlat_top.c  |  5 +++
>  6 files changed, 81 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/tracing/rtla/src/timerlat.c b/tools/tracing/rtla/src/timerlat.c
> index b69212874127..6907a323f9ec 100644
> --- a/tools/tracing/rtla/src/timerlat.c
> +++ b/tools/tracing/rtla/src/timerlat.c
> @@ -48,6 +48,17 @@ timerlat_apply_config(struct osnoise_tool *tool, struct timerlat_params *params)
>  		}
>  	}
>  
> +	/* Check if BPF action program is requested but BPF is not available */
> +	if (params->bpf_action_program) {
> +		if (params->mode == TRACING_MODE_TRACEFS) {
> +			err_msg("BPF actions are not supported in tracefs-only mode\n");
> +			goto out_err;
> +		}
> +
> +		if (timerlat_load_bpf_action_program(params->bpf_action_program))
> +			goto out_err;
> +	}
> +
>  	if (params->mode != TRACING_MODE_BPF) {
>  		/*
>  		 * In tracefs and mixed mode, timerlat tracer handles stopping
> diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/timerlat.h
> index fd6065f48bb7..8dd5d134ce08 100644
> --- a/tools/tracing/rtla/src/timerlat.h
> +++ b/tools/tracing/rtla/src/timerlat.h
> @@ -27,6 +27,7 @@ struct timerlat_params {
>  	int			dump_tasks;
>  	int			deepest_idle_state;
>  	enum timerlat_tracing_mode mode;
> +	const char		*bpf_action_program;
>  };
>  
>  #define to_timerlat_params(ptr) container_of(ptr, struct timerlat_params, common)
> @@ -36,4 +37,3 @@ int timerlat_main(int argc, char *argv[]);
>  int timerlat_enable(struct osnoise_tool *tool);
>  void timerlat_analyze(struct osnoise_tool *tool, bool stopped);
>  void timerlat_free(struct osnoise_tool *tool);
> -
> diff --git a/tools/tracing/rtla/src/timerlat_bpf.c b/tools/tracing/rtla/src/timerlat_bpf.c
> index 1d619e502c65..3c63bf7aa607 100644
> --- a/tools/tracing/rtla/src/timerlat_bpf.c
> +++ b/tools/tracing/rtla/src/timerlat_bpf.c
> @@ -7,6 +7,10 @@
>  
>  static struct timerlat_bpf *bpf;
>  
> +/* BPF object and program for action program */
> +static struct bpf_object *obj;
> +static struct bpf_program *prog;
> +
>  /*
>   * timerlat_bpf_init - load and initialize BPF program to collect timerlat data
>   */
> @@ -56,6 +60,10 @@ int timerlat_bpf_init(struct timerlat_params *params)
>  		return err;
>  	}
>  
> +	/* Set BPF action program to NULL */
> +	prog = NULL;
> +	obj = NULL;
> +
>  	return 0;
>  }
>  
> @@ -96,6 +104,11 @@ void timerlat_bpf_detach(void)
>  void timerlat_bpf_destroy(void)
>  {
>  	timerlat_bpf__destroy(bpf);
> +	bpf = NULL;
> +	if (obj)
> +		bpf_object__close(obj);
> +	obj = NULL;
> +	prog = NULL;
>  }
>  
>  static int handle_rb_event(void *ctx, void *data, size_t data_sz)
> @@ -190,4 +203,45 @@ int timerlat_bpf_get_summary_value(enum summary_field key,
>  			 bpf->maps.summary_user,
>  			 key, value_irq, value_thread, value_user, cpus);
>  }
> +
> +/*
> + * timerlat_load_bpf_action_program - load and register a BPF action program
> + */
> +int timerlat_load_bpf_action_program(const char *program_path)
> +{
> +	int err;
> +
> +	obj = bpf_object__open_file(program_path, NULL);
> +	if (!obj) {
> +		err_msg("Failed to open BPF action program: %s\n", program_path);
> +		return -1;
> +	}
> +
> +	err = bpf_object__load(obj);
> +	if (err) {
> +		err_msg("Failed to load BPF action program: %s\n", program_path);

This error path misses the call to to bpf_object__close()

> +		return -1;
> +	}
> +
> +	prog = bpf_object__find_program_by_name(obj, "action_handler");
> +	if (!prog) {
> +		err_msg("BPF action program must have 'action_handler' function: %s\n",
> +			program_path);
> +		bpf_object__close(obj);
> +		obj = NULL;
> +		return -1;
> +	}
> +
> +	err = timerlat_bpf_set_action(prog);
> +	if (err) {
> +		err_msg("Failed to register BPF action program: %s\n", program_path);
> +		bpf_object__close(obj);
> +		obj = NULL;
> +		prog = NULL;
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
>  #endif /* HAVE_BPF_SKEL */
> diff --git a/tools/tracing/rtla/src/timerlat_bpf.h b/tools/tracing/rtla/src/timerlat_bpf.h
> index b5009092c7a3..169abeaf4363 100644
> --- a/tools/tracing/rtla/src/timerlat_bpf.h
> +++ b/tools/tracing/rtla/src/timerlat_bpf.h
> @@ -30,7 +30,7 @@ int timerlat_bpf_get_summary_value(enum summary_field key,
>  				   long long *value_thread,
>  				   long long *value_user,
>  				   int cpus);
> -
> +int timerlat_load_bpf_action_program(const char *program_path);
>  static inline int have_libbpf_support(void) { return 1; }
>  #else
>  static inline int timerlat_bpf_init(struct timerlat_params *params)
> @@ -58,6 +58,10 @@ static inline int timerlat_bpf_get_summary_value(enum summary_field key,
>  {
>  	return -1;
>  }
> +static inline int timerlat_load_bpf_action_program(const char *program_path)
> +{
> +	return -1;
> +}
>  static inline int have_libbpf_support(void) { return 0; }
>  #endif /* HAVE_BPF_SKEL */
>  #endif /* __bpf__ */
> diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
> index 606c1688057b..5e639cc34f64 100644
> --- a/tools/tracing/rtla/src/timerlat_hist.c
> +++ b/tools/tracing/rtla/src/timerlat_hist.c
> @@ -763,6 +763,7 @@ static void timerlat_hist_usage(char *usage)
>  		"	     --deepest-idle-state n: only go down to idle state n on cpus used by timerlat to reduce exit from idle latency",
>  		"	     --on-threshold <action>: define action to be executed at latency threshold, multiple are allowed",
>  		"	     --on-end <action>: define action to be executed at measurement end, multiple are allowed",
> +		"	     --bpf-action <program>: load and execute BPF program when latency threshold is exceeded",
>  		NULL,
>  	};
>  
> @@ -853,6 +854,7 @@ static struct common_params
>  			{"deepest-idle-state",	required_argument,	0, '\4'},
>  			{"on-threshold",	required_argument,	0, '\5'},
>  			{"on-end",		required_argument,	0, '\6'},
> +			{"bpf-action",		required_argument,	0, '\7'},
>  			{0, 0, 0, 0}
>  		};
>  
> @@ -1062,6 +1064,9 @@ static struct common_params
>  				exit(EXIT_FAILURE);
>  			}
>  			break;
> +		case '\7':
> +			params->bpf_action_program = optarg;
> +			break;
>  		default:
>  			timerlat_hist_usage("Invalid option");
>  		}
> diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
> index fc479a0dcb59..da5d5db1bc17 100644
> --- a/tools/tracing/rtla/src/timerlat_top.c
> +++ b/tools/tracing/rtla/src/timerlat_top.c
> @@ -521,6 +521,7 @@ static void timerlat_top_usage(char *usage)
>  		"	     --deepest-idle-state n: only go down to idle state n on cpus used by timerlat to reduce exit from idle latency",
>  		"	     --on-threshold <action>: define action to be executed at latency threshold, multiple are allowed",
>  		"	     --on-end: define action to be executed at measurement end, multiple are allowed",
> +		"	     --bpf-action <program>: load and execute BPF program when latency threshold is exceeded",
>  		NULL,
>  	};
>  
> @@ -603,6 +604,7 @@ static struct common_params
>  			{"deepest-idle-state",	required_argument,	0, '8'},
>  			{"on-threshold",	required_argument,	0, '9'},
>  			{"on-end",		required_argument,	0, '\1'},
> +			{"bpf-action",		required_argument,	0, '\2'},
>  			{0, 0, 0, 0}
>  		};
>  
> @@ -798,6 +800,9 @@ static struct common_params
>  				exit(EXIT_FAILURE);
>  			}
>  			break;
> +		case '\2':
> +			params->bpf_action_program = optarg;
> +			break;
>  		default:
>  			timerlat_top_usage("Invalid option");
>  		}
> -- 
> 2.51.0
> 

Re: [PATCH 2/4] rtla/timerlat: Add --bpf-action option

[Crystal Wood]

On Fri, 2025-10-17 at 16:46 +0200, Tomas Glozar wrote:

> diff --git a/tools/tracing/rtla/src/timerlat_bpf.c b/tools/tracing/rtla/src/timerlat_bpf.c
> index 1d619e502c65..3c63bf7aa607 100644
> --- a/tools/tracing/rtla/src/timerlat_bpf.c
> +++ b/tools/tracing/rtla/src/timerlat_bpf.c
> @@ -7,6 +7,10 @@
>  
>  static struct timerlat_bpf *bpf;
>  
> +/* BPF object and program for action program */
> +static struct bpf_object *obj;
> +static struct bpf_program *prog;
> +
>  /*
>   * timerlat_bpf_init - load and initialize BPF program to collect timerlat data
>   */
> @@ -56,6 +60,10 @@ int timerlat_bpf_init(struct timerlat_params *params)
>  		return err;
>  	}
>  
> +	/* Set BPF action program to NULL */
> +	prog = NULL;
> +	obj = NULL;

They should already be NULL...

-Crystal

[PATCH 3/4] rtla/timerlat: Add example for BPF action program

[Tomas Glozar]

Add an example BPF action program that prints the measured latency to
the tracefs buffer via bpf_printk().

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/sample/timerlat_bpf_action.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
 create mode 100644 tools/tracing/rtla/sample/timerlat_bpf_action.c

diff --git a/tools/tracing/rtla/sample/timerlat_bpf_action.c b/tools/tracing/rtla/sample/timerlat_bpf_action.c
new file mode 100644
index 000000000000..ac1be049a848
--- /dev/null
+++ b/tools/tracing/rtla/sample/timerlat_bpf_action.c
@@ -0,0 +1,16 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_tracing.h>
+
+char LICENSE[] SEC("license") = "GPL";
+
+struct trace_event_raw_timerlat_sample {
+	unsigned long long timer_latency;
+} __attribute__((preserve_access_index));
+
+SEC("tp/timerlat_action")
+int action_handler(struct trace_event_raw_timerlat_sample *tp_args)
+{
+	bpf_printk("Latency: %lld\n", tp_args->timer_latency);
+	return 0;
+}
-- 
2.51.0

Re: [PATCH 3/4] rtla/timerlat: Add example for BPF action program

[Crystal Wood]

On Fri, 2025-10-17 at 16:46 +0200, Tomas Glozar wrote:
> Add an example BPF action program that prints the measured latency to
> the tracefs buffer via bpf_printk().
> 
> Signed-off-by: Tomas Glozar <tglozar@redhat.com>
> ---
>  tools/tracing/rtla/sample/timerlat_bpf_action.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
>  create mode 100644 tools/tracing/rtla/sample/timerlat_bpf_action.c

We should have the makefile build this, and add a test that uses it.

-Crystal

Re: [PATCH 3/4] rtla/timerlat: Add example for BPF action program

[Tomas Glozar]

po 20. 10. 2025 v 19:53 odesílatel Crystal Wood <crwood@redhat.com> napsal:
>
> We should have the makefile build this, and add a test that uses it.
>

I agree. I tried to use the example, but it would be also good to
check if the BPF program was actually executed.

That is hard to do reliably for the current example, as it writes into
the global tracefs instance, which might conflict with another user of
the same instance. I will write another BPF program that will create a
map instead, and a script that will check the map value via
--on-threshold in the test.

Tomas

Re: [PATCH 3/4] rtla/timerlat: Add example for BPF action program

[Crystal Wood]

On Tue, 2025-10-21 at 16:54 +0200, Tomas Glozar wrote:
> po 20. 10. 2025 v 19:53 odesílatel Crystal Wood <crwood@redhat.com> napsal:
> > 
> > We should have the makefile build this, and add a test that uses it.
> > 
> 
> I agree. I tried to use the example, but it would be also good to
> check if the BPF program was actually executed.
> 
> That is hard to do reliably for the current example, as it writes into
> the global tracefs instance, which might conflict with another user of
> the same instance. I will write another BPF program that will create a
> map instead, and a script that will check the map value via
> --on-threshold in the test.

Huh, so I guess BPF is an exception to the "no generic printk to the
global trace instance except for debugging that generates a big boot
splat" rule?

Speaking of which, why doesn't trace_osnoise.c call
trace_array_init_printk() given that it uses trace_array_printk_buf()?

-Crystal

Re: [PATCH 3/4] rtla/timerlat: Add example for BPF action program

[Steven Rostedt]

On Tue, 21 Oct 2025 10:58:06 -0500
Crystal Wood <crwood@redhat.com> wrote:

> Huh, so I guess BPF is an exception to the "no generic printk to the
> global trace instance except for debugging that generates a big boot
> splat" rule?

bpf_printk() is an event and not the generic trace_printk() that would
cause that splat. You can turn it off.

> 
> Speaking of which, why doesn't trace_osnoise.c call
> trace_array_init_printk() given that it uses trace_array_printk_buf()?

Note, trace_array_printk() (which trace_array_init_printk()) only works for
instances and does not print into the top level trace buffer.

-- Steve

Re: [PATCH 3/4] rtla/timerlat: Add example for BPF action program

[Crystal Wood]

On Tue, 2025-10-21 at 13:02 -0400, Steven Rostedt wrote:
> On Tue, 21 Oct 2025 10:58:06 -0500
> Crystal Wood <crwood@redhat.com> wrote:
> 
> > Huh, so I guess BPF is an exception to the "no generic printk to the
> > global trace instance except for debugging that generates a big boot
> > splat" rule?
> 
> bpf_printk() is an event and not the generic trace_printk() that would
> cause that splat. You can turn it off.

Ah, I see.

> > Speaking of which, why doesn't trace_osnoise.c call
> > trace_array_init_printk() given that it uses trace_array_printk_buf()?
> 
> Note, trace_array_printk() (which trace_array_init_printk()) only works for
> instances and does not print into the top level trace buffer.

Yes, it's for instances, such as the ones rtla creates.

-Crystal

Re: [PATCH 3/4] rtla/timerlat: Add example for BPF action program

[Tomas Glozar]

út 21. 10. 2025 v 19:32 odesílatel Crystal Wood <crwood@redhat.com> napsal:
>
> On Tue, 2025-10-21 at 13:02 -0400, Steven Rostedt wrote:
> >
> > Note, trace_array_printk() (which trace_array_init_printk()) only works for
> > instances and does not print into the top level trace buffer.
>
> Yes, it's for instances, such as the ones rtla creates.
>
> -Crystal
>

Right, it actually works with the trace instance, when set up properly:

[root@cs9 rtla]# ./rtla timerlat hist -T 1 -c 0 -e
bpf_trace:bpf_trace_printk --on-end trace --on-end shell,command='grep
bpf_trace_printk timerlat_trace.txt' --bpf-action
example/timerlat_bpf_action.o
# RTLA timerlat histogram
# Time unit is microseconds (us)
# Duration:   0 00:00:01
Index   IRQ-000   Thr-000   Usr-000
6             1         0         0
21            0         1         0
over:         0         0         0
count:        1         1         0
min:          6        21         -
avg:          6        21         -
max:          6        21         -
ALL:        IRQ       Thr       Usr
count:        1         1         0
min:          6        21         -
avg:          6        21         -
max:          6        21         -
 Saving trace to timerlat_trace.txt
    timerlatu/0-47791   [000] ...22 863353.286497: bpf_trace_printk:
Latency: 21282

Tomas

Re: [PATCH 3/4] rtla/timerlat: Add example for BPF action program

[Crystal Wood]

On Thu, 2025-10-23 at 09:27 +0200, Tomas Glozar wrote:
> út 21. 10. 2025 v 19:32 odesílatel Crystal Wood <crwood@redhat.com> napsal:
> > 
> > On Tue, 2025-10-21 at 13:02 -0400, Steven Rostedt wrote:
> > > 
> > > Note, trace_array_printk() (which trace_array_init_printk()) only works for
> > > instances and does not print into the top level trace buffer.
> > 
> > Yes, it's for instances, such as the ones rtla creates.
> > 
> > -Crystal
> > 
> 
> Right, it actually works with the trace instance, when set up properly:
[snip]
>  Saving trace to timerlat_trace.txt
>     timerlatu/0-47791   [000] ...22 863353.286497: bpf_trace_printk:
> Latency: 21282

OK, but as far as I can tell there's no way to get the non-BPF "stop
tracing hit" messages without adding a call to trace_array_init_printk()
into trace_osnoise.c.

-Crystal

Re: [PATCH 3/4] rtla/timerlat: Add example for BPF action program

[Tomas Glozar]

pá 24. 10. 2025 v 3:49 odesílatel Crystal Wood <crwood@redhat.com> napsal:
>
> OK, but as far as I can tell there's no way to get the non-BPF "stop
> tracing hit" messages without adding a call to trace_array_init_printk()
> into trace_osnoise.c.
>

I'm not sure what you mean. The --bpf-action functionality is
exclusive for BPF mode, bpf_trace_printk() / bpf_printk(), as Steven
clarified, is a BPF helper that triggers a trace event, defined in
kernel/trace/bpf_trace.c and kernel/trace/bpf_trace.h.

Either way, it seems that the event is not always recorded in the RTLA
trace instance due to a bug. It might have something to do with the
trace instance recording both osnoise:timerlat_sample and
bpf_trace_printk, and failing to record a trace event generated during
the handling of another trace event. So this is not a reliable way to
test the BPF actions feature, and I'll probably have to use my
original idea.

Tomas

Re: [PATCH 3/4] rtla/timerlat: Add example for BPF action program

[Crystal Wood]

On Mon, 2025-10-27 at 11:34 +0100, Tomas Glozar wrote:
> pá 24. 10. 2025 v 3:49 odesílatel Crystal Wood <crwood@redhat.com> napsal:
> > 
> > OK, but as far as I can tell there's no way to get the non-BPF "stop
> > tracing hit" messages without adding a call to trace_array_init_printk()
> > into trace_osnoise.c.
> > 
> 
> I'm not sure what you mean. The --bpf-action functionality is
> exclusive for BPF mode, bpf_trace_printk() / bpf_printk(), as Steven
> clarified, is a BPF helper that triggers a trace event, defined in
> kernel/trace/bpf_trace.c and kernel/trace/bpf_trace.h.

This was from a tangential comment, not about --bpf-action.


-Crystal

[PATCH 4/4] Documentation/rtla: Document --bpf-action option

[Tomas Glozar]

Add new option --bpf-action into common_timerlat_options.txt, including
the format in which it takes the BPF program, and a reference to an
example.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 .../tools/rtla/common_timerlat_options.rst     | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/Documentation/tools/rtla/common_timerlat_options.rst b/Documentation/tools/rtla/common_timerlat_options.rst
index 1f5d024b53aa..1b7293ebc897 100644
--- a/Documentation/tools/rtla/common_timerlat_options.rst
+++ b/Documentation/tools/rtla/common_timerlat_options.rst
@@ -65,3 +65,21 @@
         Set timerlat to run without workload, waiting for the user to dispatch a per-cpu
         task that waits for a new period on the tracing/osnoise/per_cpu/cpu$ID/timerlat_fd.
         See linux/tools/rtla/sample/timerlat_load.py for an example of user-load code.
+
+**--bpf-action** *bpf-program*
+
+        Loads a BPF program from an ELF file and executes it when a latency threshold is exceeded.
+
+        The BPF program must be a valid ELF file loadable with libbpf. The program must contain
+        a function named ``action_handler``, declared with ``SEC("tp/timerlat_action")`` for libbpf
+        to correctly detect the program type.
+
+        The program receives a ``struct trace_event_raw_timerlat_sample`` parameter
+        containing timerlat sample data.
+
+        A sample BPF action program is provided in ``tools/tracing/rtla/sample/timerlat_bpf_action.c``.
+        This example demonstrates how to create a BPF program that prints latency information when
+        thresholds are exceeded.
+
+        **Note**: BPF actions require BPF support to be available. If BPF is not available or disabled,
+        the tool will fall back to tracefs mode and BPF actions will not be supported.
-- 
2.51.0