[RFC PATCH 0/3] rtla: Synchronize sample collection methods

[RFC PATCH 0/3] rtla: Synchronize sample collection methods

[Tomas Glozar]

This is a proposal to synchronize the start of sample collection throughout
all places where samples are collected in RTLA, using a lockless method of
tracking the count of active tracefs instances set to the timerlat tracer.

There are three different places where timerlat samples are collected in RTLA:
two tracefs instances (auto-analysis and record), and a BPF program attached
to the osnoise:timerlat_sample tracepoint.

The BPF program collects all samples and generates the main statistics from
them in either top or hist mode, while the auto-analysis and record instances
are used as ring buffers with the intention of capturing the tail of both
timerlat samples and additional tracepoints.

In instances where BPF support is not available, the BPF program is replaced
by a third tracefs instance, called the "main instance".

One problem with this approach is each of these sample collectors is
turned on [1] separately, in the present state of RTLA, in the following order:

- record instance
- auto-analysis instance
- BPF program / main instance

[1] By "turned on", I mean toggling tracing on (for tracefs instances) or
attachment (for the BPF program). Note that timerlat starts measurement
on tracer registration, even if tracing is off by default. This might have
been originally unintentional - considering that explicitly turning off
tracing stops the measurement threads - but is now relied upon in RTLA.

This leads to some samples being seen only by tracers that are enabled
earlier, leading to confusing results. For example, auto-analysis might
show spikes that are not seen in the histogram, and trace output (record)
might show samples that are not seen by either the histogram or
auto-analysis.

To enable RTLA to analyze samples consistently, the first patch adds two fields
to the osnoise:timerlat_sample tracepoint: instances_registered and
instances_on. During the recording of a timerlat sample, timerlat counts
how many instances are registered and how many are on, and attaches
the information to the osnoise:timerlat_sample trace event, which is moved
to occur after the samples are recorded.

The second patch makes RTLA count how many tracefs instances are to be enabled
in total, passes this number to the BPF program, and makes it drop any samples
that arrive before the instances are turned on. This ensures all samples
recorded in the main statistics (e.g. histogram) are seen by all active tracefs
instances, that is, auto-analysis and trace output (record).

The third patch then moves the attachment of the BPF program before the enabling
of the tracefs instances, provided that the kernel supports instance counting
introduced in the first patch. The second patch enforces that all samples
recorded by the program are seen by the last enabled tracefs instance.

Synchronization between different tracefs instances, that is, auto-analysis and
trace output, and the main instance in non-BPF mode, is possible by looking at
the osnoise:timerlat_sample record that comes after each sample, but is not yet
implemented in the current version of the patchset.

Tomas Glozar (3):
  tracing/osnoise: Record timerlat instance counts
  rtla/timerlat_bpf: Filter samples unseen by tracer
  rtla/timerlat: Attach BPF program before tracers

 include/trace/events/osnoise.h        | 14 ++++++---
 kernel/trace/trace_osnoise.c          | 10 +++++--
 tools/tracing/rtla/src/timerlat.bpf.c |  9 ++++++
 tools/tracing/rtla/src/timerlat.c     | 43 +++++++++++++++++++++++++--
 tools/tracing/rtla/src/timerlat.h     |  2 ++
 tools/tracing/rtla/src/timerlat_bpf.c |  8 +++++
 6 files changed, 78 insertions(+), 8 deletions(-)

-- 
2.52.0

[RFC PATCH 1/3] tracing/osnoise: Record timerlat instance counts

[Tomas Glozar]

While recording a timerlat sample, track how many timerlat instances are
registered at the moment and how many instances have tracing on.

This enables a user of the timerlat_sample threshold to synchronize with
any users of the timerlat tracer instance buffer.

Note that this also reverses the order in which the trace buffer record
and the tracepoint are processed: the tracepoint now fires after the
trace buffer entries are recorded.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 include/trace/events/osnoise.h | 14 ++++++++++----
 kernel/trace/trace_osnoise.c   | 10 ++++++++--
 2 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/include/trace/events/osnoise.h b/include/trace/events/osnoise.h
index 3f4273623801..50beaf8fbb52 100644
--- a/include/trace/events/osnoise.h
+++ b/include/trace/events/osnoise.h
@@ -79,26 +79,32 @@ TRACE_EVENT(osnoise_sample,
 #ifdef CONFIG_TIMERLAT_TRACER
 TRACE_EVENT(timerlat_sample,
 
-	TP_PROTO(struct timerlat_sample *s),
+	TP_PROTO(struct timerlat_sample *s, int instances_registered, int instances_on),
 
-	TP_ARGS(s),
+	TP_ARGS(s, instances_registered, instances_on),
 
 	TP_STRUCT__entry(
 		__field(	u64,		timer_latency	)
 		__field(	unsigned int,	seqnum		)
 		__field(	int,		context		)
+		__field(	int,		instances_registered	)
+		__field(	int,		instances_on		)
 	),
 
 	TP_fast_assign(
 		__entry->timer_latency = s->timer_latency;
 		__entry->seqnum = s->seqnum;
 		__entry->context = s->context;
+		__entry->instances_registered = instances_registered;
+		__entry->instances_on = instances_on;
 	),
 
-	TP_printk("timer_latency=%llu seqnum=%u context=%d",
+	TP_printk("timer_latency=%llu seqnum=%u context=%d instances_registered=%d instances_on=%d",
 		  __entry->timer_latency,
 		  __entry->seqnum,
-		  __entry->context)
+		  __entry->context,
+		  __entry->instances_registered,
+		  __entry->instances_on)
 );
 #endif // CONFIG_TIMERLAT_TRACER
 
diff --git a/kernel/trace/trace_osnoise.c b/kernel/trace/trace_osnoise.c
index 827104d00bc0..65ddc1f49c19 100644
--- a/kernel/trace/trace_osnoise.c
+++ b/kernel/trace/trace_osnoise.c
@@ -574,15 +574,21 @@ static void record_timerlat_sample(struct timerlat_sample *sample)
 {
 	struct osnoise_instance *inst;
 	struct trace_buffer *buffer;
-
-	trace_timerlat_sample(sample);
+	int instances_registered = 0;
+	int instances_on = 0;
 
 	rcu_read_lock();
 	list_for_each_entry_rcu(inst, &osnoise_instances, list) {
+		if (tracer_tracing_is_on(inst->tr))
+			instances_on++;
+		instances_registered++;
+
 		buffer = inst->tr->array_buffer.buffer;
 		__record_timerlat_sample(sample, buffer);
 	}
 	rcu_read_unlock();
+
+	trace_timerlat_sample(sample, instances_registered, instances_on);
 }
 
 #ifdef CONFIG_STACKTRACE
-- 
2.52.0

[RFC PATCH 2/3] rtla/timerlat_bpf: Filter samples unseen by tracer

[Tomas Glozar]

If both BPF and tracefs sample collection are used (mixed mode), drop
samples seen by BPF program when the count of timerlat instances that
are both registered and have tracing on does not match the expected value.

This ensures that there are no samples seen by BPF that are not included
in auto-analysis or trace output.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/src/timerlat.bpf.c |  9 +++++++++
 tools/tracing/rtla/src/timerlat.c     | 18 +++++++++++++++++-
 tools/tracing/rtla/src/timerlat.h     |  2 ++
 tools/tracing/rtla/src/timerlat_bpf.c |  8 ++++++++
 4 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/tools/tracing/rtla/src/timerlat.bpf.c b/tools/tracing/rtla/src/timerlat.bpf.c
index 549d2d2191d2..2724a9bfea4e 100644
--- a/tools/tracing/rtla/src/timerlat.bpf.c
+++ b/tools/tracing/rtla/src/timerlat.bpf.c
@@ -12,6 +12,8 @@ char LICENSE[] SEC("license") = "GPL";
 struct trace_event_raw_timerlat_sample {
 	unsigned long long timer_latency;
 	int context;
+	int instances_registered;
+	int instances_on;
 } __attribute__((preserve_access_index));
 
 struct {
@@ -58,6 +60,8 @@ const volatile int entries = 256;
 const volatile int irq_threshold;
 const volatile int thread_threshold;
 const volatile bool aa_only;
+const volatile int instances_registered = -1;
+const volatile int instances_on = -1;
 
 nosubprog unsigned long long map_get(void *map,
 				     unsigned int key)
@@ -143,6 +147,11 @@ int handle_timerlat_sample(struct trace_event_raw_timerlat_sample *tp_args)
 	unsigned long long latency, latency_us;
 	int bucket;
 
+	if ((instances_registered != -1 && tp_args->instances_registered != instances_registered)
+		|| (instances_on != -1 && tp_args->instances_on != instances_on))
+		/* This sample is not seen by all trace instances, filter it out */
+		return 0;
+
 	if (map_get(&stop_tracing, 0))
 		return 0;
 
diff --git a/tools/tracing/rtla/src/timerlat.c b/tools/tracing/rtla/src/timerlat.c
index 8f8811f7a13b..069f916100e7 100644
--- a/tools/tracing/rtla/src/timerlat.c
+++ b/tools/tracing/rtla/src/timerlat.c
@@ -28,6 +28,7 @@ int
 timerlat_apply_config(struct osnoise_tool *tool, struct timerlat_params *params)
 {
 	int retval;
+	struct tep_event *event = NULL;
 
 	/*
 	 * Try to enable BPF, unless disabled explicitly.
@@ -36,10 +37,25 @@ timerlat_apply_config(struct osnoise_tool *tool, struct timerlat_params *params)
 	if (getenv("RTLA_NO_BPF") && strncmp(getenv("RTLA_NO_BPF"), "1", 2) == 0) {
 		debug_msg("RTLA_NO_BPF set, disabling BPF\n");
 		params->mode = TRACING_MODE_TRACEFS;
-	} else if (!tep_find_event_by_name(tool->trace.tep, "osnoise", "timerlat_sample")) {
+	} else if (!(event = tep_find_event_by_name(tool->trace.tep, "osnoise", "timerlat_sample"))) {
 		debug_msg("osnoise:timerlat_sample missing, disabling BPF\n");
 		params->mode = TRACING_MODE_TRACEFS;
+	}
+
+	if (event && tep_find_field(event, "instances_registered") &&
+	    tep_find_field(event, "instances_on")) {
+		params->has_instance_count_fields = true;
+
+		if (!params->no_aa)
+			params->instances_on++;
+		if (params->common.threshold_actions.present[ACTION_TRACE_OUTPUT] ||
+		    params->common.end_actions.present[ACTION_TRACE_OUTPUT])
+			params->instances_on++;
 	} else {
+		params->has_instance_count_fields = false;
+	}
+
+	if (params->mode != TRACING_MODE_TRACEFS) {
 		retval = timerlat_bpf_init(params);
 		if (retval) {
 			debug_msg("Could not enable BPF\n");
diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/timerlat.h
index 8dd5d134ce08..ef97e545c7ff 100644
--- a/tools/tracing/rtla/src/timerlat.h
+++ b/tools/tracing/rtla/src/timerlat.h
@@ -28,6 +28,8 @@ struct timerlat_params {
 	int			deepest_idle_state;
 	enum timerlat_tracing_mode mode;
 	const char		*bpf_action_program;
+	bool			has_instance_count_fields;
+	int			instances_on;
 };
 
 #define to_timerlat_params(ptr) container_of(ptr, struct timerlat_params, common)
diff --git a/tools/tracing/rtla/src/timerlat_bpf.c b/tools/tracing/rtla/src/timerlat_bpf.c
index 05adf18303df..80801796bbf0 100644
--- a/tools/tracing/rtla/src/timerlat_bpf.c
+++ b/tools/tracing/rtla/src/timerlat_bpf.c
@@ -53,6 +53,14 @@ int timerlat_bpf_init(struct timerlat_params *params)
 		bpf_map__set_autocreate(bpf->maps.summary_user, false);
 	}
 
+	if (params->mode == TRACING_MODE_MIXED && params->has_instance_count_fields) {
+		bpf->rodata->instances_on = params->instances_on;
+		bpf->rodata->instances_registered = params->instances_on + 1; /* +1 for the main instance */
+	} else {
+		bpf->rodata->instances_registered = -1;
+		bpf->rodata->instances_on = -1;
+	}
+
 	/* Load and verify BPF program */
 	err = timerlat_bpf__load(bpf);
 	if (err) {
-- 
2.52.0

[RFC PATCH 3/3] rtla/timerlat: Attach BPF program before tracers

[Tomas Glozar]

If osnoise:timerlat_sample has instance count fields, attach the BPF program
before starting trace instances in BPF/mixed mode.

This ensures that what is seen by BPF sample collection is exactly what
is seen by the last enabled tracefs instance.

Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
 tools/tracing/rtla/src/timerlat.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/tools/tracing/rtla/src/timerlat.c b/tools/tracing/rtla/src/timerlat.c
index 069f916100e7..da2e2d003ec9 100644
--- a/tools/tracing/rtla/src/timerlat.c
+++ b/tools/tracing/rtla/src/timerlat.c
@@ -170,6 +170,21 @@ int timerlat_enable(struct osnoise_tool *tool)
 			return -1;
 	}
 
+	if (params->mode != TRACING_MODE_TRACEFS && params->has_instance_count_fields) {
+		/*
+		 * If the timerlat tracer has instance count fields, it is safe
+		 * to attach the BPF program before starting the trace instances.
+		 *
+		 * The BPF program will ignore any samples that arrive before
+		 * the trace instances are started.
+		 */
+		retval = timerlat_bpf_attach();
+		if (retval) {
+			err_msg("Error attaching BPF program\n");
+			return retval;
+		}
+	}
+
 	/*
 	 * Start the tracers here, after having set all instances.
 	 *
@@ -183,7 +198,15 @@ int timerlat_enable(struct osnoise_tool *tool)
 		trace_instance_start(&tool->aa->trace);
 	if (params->mode == TRACING_MODE_TRACEFS) {
 		trace_instance_start(&tool->trace);
-	} else {
+	} else if (!params->has_instance_count_fields) {
+		/*
+		 * Without instance count fields, play safe and attach the BPF program
+		 * after starting the trace instances.
+		 *
+		 * This might lead to samples being seen only by trace output and
+		 * auto analysis, but it is better than RTLA reporting results over
+		 * threshold that cannot be analyzed further.
+		 */
 		retval = timerlat_bpf_attach();
 		if (retval) {
 			err_msg("Error attaching BPF program\n");
-- 
2.52.0

Re: [RFC PATCH 0/3] rtla: Synchronize sample collection methods

[Steven Rostedt]

On Fri, 23 Jan 2026 16:25:31 +0100
Tomas Glozar <tglozar@redhat.com> wrote:

> To enable RTLA to analyze samples consistently, the first patch adds two fields
> to the osnoise:timerlat_sample tracepoint: instances_registered and
> instances_on. During the recording of a timerlat sample, timerlat counts
> how many instances are registered and how many are on, and attaches
> the information to the osnoise:timerlat_sample trace event, which is moved
> to occur after the samples are recorded.

Can't RTLA simply write into trace_marker or trace_marker_raw an event
that states "tracing is now active" and ignore anything before that event.

Heck, it will include a timestamp, so you only need to write once and
ignore any event that occurred before that timestamp.

-- Steve