* [PATCH 0/8] rtla: Collect data using BPF program
@ 2025-02-18 14:58 Tomas Glozar
2025-02-18 14:58 ` [PATCH 1/8] rtla/timerlat: Unify params struct Tomas Glozar
` (7 more replies)
0 siblings, 8 replies; 9+ messages in thread
From: Tomas Glozar @ 2025-02-18 14:58 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Gabriele Monaco, Clark Williams, Tomas Glozar
The current implementation of rtla uses libtracefs and libtraceevent to
pull sample events generated by the timerlat tracer from the trace
buffer. rtla then processes the sample by updating the histogram and
summary (current, maximum, minimum, and sum values) as well as checks
if tracing has been stopped due to threshold overflow.
In use cases where a large number of samples is being generated, that
is, with measurements running on many CPUs and with a low interval,
this sample processing design causes a significant CPU load on the rtla
side. Furthermode, with >100 CPUs and 100us interval, rtla was reported
as not being able to keep up with the samples and dropping most of them,
leading to it being unusable.
A timerlat trace change was proposed [1] to implement an alternative
way of processing timerlat samples by the way of a trace event. This
patchset makes use of that by attaching a BPF program to the trace event
using the BPF skeleton feature of bpftool. One BPF program is shared
for both top and hist, operating in three different modes: top, hist,
and auto-analysis only. Data is collected using per-CPU BPF maps to
achieve maximum performance and avoid lock contention. The maps are then
processed in userspace when the data is to be displayed (at the end of
the run for hist and quiet top, once per second for regular top).
During the time of measurement, the new implementation is idle, waiting
for either a signal or threshold overflow. Unlike the current
implementation, the BPF implementation does not check whether tracing is
stopped (in BPF mode, tracing is always off to improve performance), but
waits for a write to a BPF ringbuffer instead. This allows rtla to exit
immediately when a threshold is violated, without waiting for the next
iteration of the while loop.
If the requirements for the BPF implementation are not met, either at
build time or at run time, the current implementation is used as
fallback. Which implementation is being used can be seen when running
rtla timerlat with "-D" option. rtla can be forced to run in non-BPF
mode by setting the RTLA_NO_BPF option to 1, for debugging purposes.
The BPF implementation has the following build requirements:
- libbpf 1.0.0 or later
- bpftool with skeleton support
- clang with BPF CO-RE support
Unlike perf, rtla does not build its own static libbpf and likewise
relies on system bpftool instead of using an in-tree one. In the future,
this might change if modern BPF features not commonly available on
Linux distributions are introduced.
The runtime requirements are as follows:
- BPF support enabled in the kernel
- libbpf library
- osnoise:timerlat_sample trace event present
No performance penalty was seen during testing on the timerlat tracer
side, as the performance of the BPF program is comparable to writing
the sample entry to the tracefs buffer. As rtla is idle during
measurements, except for printing the summary for timerlat-top in
non-quiet mode, the overall CPU usage is reduced significantly, and
the -H option to pin rtla to housekeeping CPUs becomes unnecessary for
most use cases.
Note: The unification of the timerlat_*_params struct was done to
enable the BPF implementation to be fully shared between top and hist,
besides the processing of the data. The plan is to avoid duplicate code
and instead continually merge the implementations of top and hist. top
was developed first, and currently, is essentially hist without
the histogram, and with some old code. Thus, I expect it to be possible
to fully merge it into the hist implementation in the future.
[1] https://lore.kernel.org/linux-trace-kernel/20250203090418.1458923-1-tglozar@redhat.com
Tomas Glozar (8):
rtla/timerlat: Unify params struct
tools/build: Add bpftool-skeletons feature test
rtla: Add optional dependency on BPF tooling
rtla/timerlat: Add BPF skeleton to collect samples
rtla/timerlat_hist: Use BPF to collect samples
rtla/timerlat_top: Move divisor to update
rtla/timerlat_top: Use BPF to collect samples
rtla/timerlat: Test BPF mode
tools/build/Makefile.feature | 3 +-
tools/build/feature/Makefile | 3 +
tools/scripts/Makefile.include | 3 +
tools/tracing/rtla/.gitignore | 1 +
tools/tracing/rtla/Makefile | 20 +-
tools/tracing/rtla/Makefile.config | 42 +++
tools/tracing/rtla/src/Build | 1 +
tools/tracing/rtla/src/osnoise.h | 2 +
tools/tracing/rtla/src/timerlat.bpf.c | 149 ++++++++++
tools/tracing/rtla/src/timerlat.h | 54 ++++
tools/tracing/rtla/src/timerlat_aa.c | 2 -
tools/tracing/rtla/src/timerlat_bpf.c | 166 +++++++++++
tools/tracing/rtla/src/timerlat_bpf.h | 59 ++++
tools/tracing/rtla/src/timerlat_hist.c | 229 +++++++++++-----
tools/tracing/rtla/src/timerlat_top.c | 366 +++++++++++++++++--------
tools/tracing/rtla/tests/timerlat.t | 14 +
16 files changed, 923 insertions(+), 191 deletions(-)
create mode 100644 tools/tracing/rtla/src/timerlat.bpf.c
create mode 100644 tools/tracing/rtla/src/timerlat_bpf.c
create mode 100644 tools/tracing/rtla/src/timerlat_bpf.h
--
2.48.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/8] rtla/timerlat: Unify params struct
2025-02-18 14:58 [PATCH 0/8] rtla: Collect data using BPF program Tomas Glozar
@ 2025-02-18 14:58 ` Tomas Glozar
2025-02-18 14:58 ` [PATCH 2/8] tools/build: Add bpftool-skeletons feature test Tomas Glozar
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Tomas Glozar @ 2025-02-18 14:58 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Gabriele Monaco, Clark Williams, Tomas Glozar
Instead of having separate structs timerlat_top_params and
timerlat_hist_params, use one struct timerlat_params for both.
This allows code using the structs to be shared between timerlat-top and
timerlat-hist.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/src/osnoise.h | 2 +
tools/tracing/rtla/src/timerlat.h | 54 ++++++++++++++++++++++
tools/tracing/rtla/src/timerlat_aa.c | 2 -
tools/tracing/rtla/src/timerlat_hist.c | 62 +++++---------------------
tools/tracing/rtla/src/timerlat_top.c | 57 +++++------------------
5 files changed, 78 insertions(+), 99 deletions(-)
diff --git a/tools/tracing/rtla/src/osnoise.h b/tools/tracing/rtla/src/osnoise.h
index 91835a7d8c2b..056c8b113dee 100644
--- a/tools/tracing/rtla/src/osnoise.h
+++ b/tools/tracing/rtla/src/osnoise.h
@@ -1,4 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
+#pragma once
+
#include "trace.h"
/*
diff --git a/tools/tracing/rtla/src/timerlat.h b/tools/tracing/rtla/src/timerlat.h
index 88561bfd14f3..e452d385cb0f 100644
--- a/tools/tracing/rtla/src/timerlat.h
+++ b/tools/tracing/rtla/src/timerlat.h
@@ -1,4 +1,58 @@
// SPDX-License-Identifier: GPL-2.0
+#include "utils.h"
+#include "osnoise.h"
+
+struct timerlat_params {
+ /* Common params */
+ char *cpus;
+ cpu_set_t monitored_cpus;
+ char *trace_output;
+ char *cgroup_name;
+ unsigned long long runtime;
+ long long stop_us;
+ long long stop_total_us;
+ long long timerlat_period_us;
+ long long print_stack;
+ int sleep_time;
+ int output_divisor;
+ int duration;
+ int quiet;
+ int set_sched;
+ int dma_latency;
+ int no_aa;
+ int aa_only;
+ int dump_tasks;
+ int cgroup;
+ int hk_cpus;
+ int user_workload;
+ int kernel_workload;
+ int pretty_output;
+ int warmup;
+ int buffer_size;
+ int deepest_idle_state;
+ cpu_set_t hk_cpu_set;
+ struct sched_attr sched_param;
+ struct trace_events *events;
+ union {
+ struct {
+ /* top only */
+ int user_top;
+ };
+ struct {
+ /* hist only */
+ int user_hist;
+ char no_irq;
+ char no_thread;
+ char no_header;
+ char no_summary;
+ char no_index;
+ char with_zeros;
+ int bucket_size;
+ int entries;
+ };
+ };
+};
+
int timerlat_hist_main(int argc, char *argv[]);
int timerlat_top_main(int argc, char *argv[]);
int timerlat_main(int argc, char *argv[]);
diff --git a/tools/tracing/rtla/src/timerlat_aa.c b/tools/tracing/rtla/src/timerlat_aa.c
index 7bd80ee2a5b4..31e66ea2b144 100644
--- a/tools/tracing/rtla/src/timerlat_aa.c
+++ b/tools/tracing/rtla/src/timerlat_aa.c
@@ -5,8 +5,6 @@
#include <stdlib.h>
#include <errno.h>
-#include "utils.h"
-#include "osnoise.h"
#include "timerlat.h"
#include <unistd.h>
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index 6d7d0a2d45b4..fe683a20b0e2 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -14,50 +14,10 @@
#include <sched.h>
#include <pthread.h>
-#include "utils.h"
-#include "osnoise.h"
#include "timerlat.h"
#include "timerlat_aa.h"
#include "timerlat_u.h"
-struct timerlat_hist_params {
- char *cpus;
- cpu_set_t monitored_cpus;
- char *trace_output;
- char *cgroup_name;
- unsigned long long runtime;
- long long stop_us;
- long long stop_total_us;
- long long timerlat_period_us;
- long long print_stack;
- int sleep_time;
- int output_divisor;
- int duration;
- int set_sched;
- int dma_latency;
- int cgroup;
- int hk_cpus;
- int no_aa;
- int dump_tasks;
- int user_workload;
- int kernel_workload;
- int user_hist;
- cpu_set_t hk_cpu_set;
- struct sched_attr sched_param;
- struct trace_events *events;
- char no_irq;
- char no_thread;
- char no_header;
- char no_summary;
- char no_index;
- char with_zeros;
- int bucket_size;
- int entries;
- int warmup;
- int buffer_size;
- int deepest_idle_state;
-};
-
struct timerlat_hist_cpu {
int *irq;
int *thread;
@@ -174,7 +134,7 @@ timerlat_hist_update(struct osnoise_tool *tool, int cpu,
unsigned long long context,
unsigned long long latency)
{
- struct timerlat_hist_params *params = tool->params;
+ struct timerlat_params *params = tool->params;
struct timerlat_hist_data *data = tool->data;
int entries = data->entries;
int bucket;
@@ -238,7 +198,7 @@ timerlat_hist_handler(struct trace_seq *s, struct tep_record *record,
*/
static void timerlat_hist_header(struct osnoise_tool *tool)
{
- struct timerlat_hist_params *params = tool->params;
+ struct timerlat_params *params = tool->params;
struct timerlat_hist_data *data = tool->data;
struct trace_seq *s = tool->trace.seq;
char duration[26];
@@ -300,7 +260,7 @@ static void format_summary_value(struct trace_seq *seq,
* timerlat_print_summary - print the summary of the hist data to the output
*/
static void
-timerlat_print_summary(struct timerlat_hist_params *params,
+timerlat_print_summary(struct timerlat_params *params,
struct trace_instance *trace,
struct timerlat_hist_data *data)
{
@@ -427,7 +387,7 @@ timerlat_print_summary(struct timerlat_hist_params *params,
}
static void
-timerlat_print_stats_all(struct timerlat_hist_params *params,
+timerlat_print_stats_all(struct timerlat_params *params,
struct trace_instance *trace,
struct timerlat_hist_data *data)
{
@@ -575,7 +535,7 @@ timerlat_print_stats_all(struct timerlat_hist_params *params,
* timerlat_print_stats - print data for each CPUs
*/
static void
-timerlat_print_stats(struct timerlat_hist_params *params, struct osnoise_tool *tool)
+timerlat_print_stats(struct timerlat_params *params, struct osnoise_tool *tool)
{
struct timerlat_hist_data *data = tool->data;
struct trace_instance *trace = &tool->trace;
@@ -734,10 +694,10 @@ static void timerlat_hist_usage(char *usage)
/*
* timerlat_hist_parse_args - allocs, parse and fill the cmd line parameters
*/
-static struct timerlat_hist_params
+static struct timerlat_params
*timerlat_hist_parse_args(int argc, char *argv[])
{
- struct timerlat_hist_params *params;
+ struct timerlat_params *params;
struct trace_events *tevent;
int auto_thresh;
int retval;
@@ -1017,7 +977,7 @@ static struct timerlat_hist_params
* timerlat_hist_apply_config - apply the hist configs to the initialized tool
*/
static int
-timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_params *params)
+timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_params *params)
{
int retval, i;
@@ -1122,7 +1082,7 @@ timerlat_hist_apply_config(struct osnoise_tool *tool, struct timerlat_hist_param
* timerlat_init_hist - initialize a timerlat hist tool with parameters
*/
static struct osnoise_tool
-*timerlat_init_hist(struct timerlat_hist_params *params)
+*timerlat_init_hist(struct timerlat_params *params)
{
struct osnoise_tool *tool;
int nr_cpus;
@@ -1170,7 +1130,7 @@ static void stop_hist(int sig)
* timerlat_hist_set_signals - handles the signal to stop the tool
*/
static void
-timerlat_hist_set_signals(struct timerlat_hist_params *params)
+timerlat_hist_set_signals(struct timerlat_params *params)
{
signal(SIGINT, stop_hist);
if (params->duration) {
@@ -1181,7 +1141,7 @@ timerlat_hist_set_signals(struct timerlat_hist_params *params)
int timerlat_hist_main(int argc, char *argv[])
{
- struct timerlat_hist_params *params;
+ struct timerlat_params *params;
struct osnoise_tool *record = NULL;
struct timerlat_u_params params_u;
struct osnoise_tool *tool = NULL;
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index 05a9403b01d2..97eead91decc 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -15,45 +15,10 @@
#include <sched.h>
#include <pthread.h>
-#include "utils.h"
-#include "osnoise.h"
#include "timerlat.h"
#include "timerlat_aa.h"
#include "timerlat_u.h"
-struct timerlat_top_params {
- char *cpus;
- cpu_set_t monitored_cpus;
- char *trace_output;
- char *cgroup_name;
- unsigned long long runtime;
- long long stop_us;
- long long stop_total_us;
- long long timerlat_period_us;
- long long print_stack;
- int sleep_time;
- int output_divisor;
- int duration;
- int quiet;
- int set_sched;
- int dma_latency;
- int no_aa;
- int aa_only;
- int dump_tasks;
- int cgroup;
- int hk_cpus;
- int user_top;
- int user_workload;
- int kernel_workload;
- int pretty_output;
- int warmup;
- int buffer_size;
- int deepest_idle_state;
- cpu_set_t hk_cpu_set;
- struct sched_attr sched_param;
- struct trace_events *events;
-};
-
struct timerlat_top_cpu {
unsigned long long irq_count;
unsigned long long thread_count;
@@ -194,7 +159,7 @@ timerlat_top_handler(struct trace_seq *s, struct tep_record *record,
struct tep_event *event, void *context)
{
struct trace_instance *trace = context;
- struct timerlat_top_params *params;
+ struct timerlat_params *params;
unsigned long long latency, thread;
struct osnoise_tool *top;
int cpu = record->cpu;
@@ -215,7 +180,7 @@ timerlat_top_handler(struct trace_seq *s, struct tep_record *record,
/*
* timerlat_top_header - print the header of the tool output
*/
-static void timerlat_top_header(struct timerlat_top_params *params, struct osnoise_tool *top)
+static void timerlat_top_header(struct timerlat_params *params, struct osnoise_tool *top)
{
struct trace_seq *s = top->trace.seq;
char duration[26];
@@ -263,7 +228,7 @@ static const char *no_value = " -";
static void timerlat_top_print(struct osnoise_tool *top, int cpu)
{
- struct timerlat_top_params *params = top->params;
+ struct timerlat_params *params = top->params;
struct timerlat_top_data *data = top->data;
struct timerlat_top_cpu *cpu_data = &data->cpu_data[cpu];
int divisor = params->output_divisor;
@@ -327,7 +292,7 @@ static void
timerlat_top_print_sum(struct osnoise_tool *top, struct timerlat_top_cpu *summary)
{
const char *split = "----------------------------------------";
- struct timerlat_top_params *params = top->params;
+ struct timerlat_params *params = top->params;
unsigned long long count = summary->irq_count;
int divisor = params->output_divisor;
struct trace_seq *s = top->trace.seq;
@@ -404,7 +369,7 @@ static void clear_terminal(struct trace_seq *seq)
* timerlat_print_stats - print data for all cpus
*/
static void
-timerlat_print_stats(struct timerlat_top_params *params, struct osnoise_tool *top)
+timerlat_print_stats(struct timerlat_params *params, struct osnoise_tool *top)
{
struct trace_instance *trace = &top->trace;
struct timerlat_top_cpu summary;
@@ -505,10 +470,10 @@ static void timerlat_top_usage(char *usage)
/*
* timerlat_top_parse_args - allocs, parse and fill the cmd line parameters
*/
-static struct timerlat_top_params
+static struct timerlat_params
*timerlat_top_parse_args(int argc, char **argv)
{
- struct timerlat_top_params *params;
+ struct timerlat_params *params;
struct trace_events *tevent;
long long auto_thresh;
int retval;
@@ -765,7 +730,7 @@ static struct timerlat_top_params
* timerlat_top_apply_config - apply the top configs to the initialized tool
*/
static int
-timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params *params)
+timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_params *params)
{
int retval;
int i;
@@ -876,7 +841,7 @@ timerlat_top_apply_config(struct osnoise_tool *top, struct timerlat_top_params *
* timerlat_init_top - initialize a timerlat top tool with parameters
*/
static struct osnoise_tool
-*timerlat_init_top(struct timerlat_top_params *params)
+*timerlat_init_top(struct timerlat_params *params)
{
struct osnoise_tool *top;
int nr_cpus;
@@ -924,7 +889,7 @@ static void stop_top(int sig)
* timerlat_top_set_signals - handles the signal to stop the tool
*/
static void
-timerlat_top_set_signals(struct timerlat_top_params *params)
+timerlat_top_set_signals(struct timerlat_params *params)
{
signal(SIGINT, stop_top);
if (params->duration) {
@@ -935,7 +900,7 @@ timerlat_top_set_signals(struct timerlat_top_params *params)
int timerlat_top_main(int argc, char *argv[])
{
- struct timerlat_top_params *params;
+ struct timerlat_params *params;
struct osnoise_tool *record = NULL;
struct timerlat_u_params params_u;
struct osnoise_tool *top = NULL;
--
2.48.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 2/8] tools/build: Add bpftool-skeletons feature test
2025-02-18 14:58 [PATCH 0/8] rtla: Collect data using BPF program Tomas Glozar
2025-02-18 14:58 ` [PATCH 1/8] rtla/timerlat: Unify params struct Tomas Glozar
@ 2025-02-18 14:58 ` Tomas Glozar
2025-02-18 14:58 ` [PATCH 3/8] rtla: Add optional dependency on BPF tooling Tomas Glozar
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Tomas Glozar @ 2025-02-18 14:58 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Gabriele Monaco, Clark Williams, Tomas Glozar
Add bpftool-skeletons feature test, testing the presence of a bpftool
capable of generating skeletons.
This is to be used for tools that do not require building their own
bootstrap bpftool from the kernel source tree.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/build/Makefile.feature | 3 ++-
tools/build/feature/Makefile | 3 +++
tools/scripts/Makefile.include | 3 +++
3 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/tools/build/Makefile.feature b/tools/build/Makefile.feature
index 1931b6321314..1f44ca677ad3 100644
--- a/tools/build/Makefile.feature
+++ b/tools/build/Makefile.feature
@@ -135,7 +135,8 @@ FEATURE_TESTS_EXTRA := \
libbpf-bpf_create_map \
libpfm4 \
libdebuginfod \
- clang-bpf-co-re
+ clang-bpf-co-re \
+ bpftool-skeletons
FEATURE_TESTS ?= $(FEATURE_TESTS_BASIC)
diff --git a/tools/build/feature/Makefile b/tools/build/feature/Makefile
index cb1e3e2feedf..4f9c1d950f5d 100644
--- a/tools/build/feature/Makefile
+++ b/tools/build/feature/Makefile
@@ -418,6 +418,9 @@ $(OUTPUT)test-file-handle.bin:
$(OUTPUT)test-libpfm4.bin:
$(BUILD) -lpfm
+$(OUTPUT)test-bpftool-skeletons.bin:
+ $(BPFTOOL) version | grep '^features:.*skeletons' \
+ > $(@:.bin=.make.output) 2>&1
###############################
clean:
diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
index 0aa4005017c7..71bbe52721b3 100644
--- a/tools/scripts/Makefile.include
+++ b/tools/scripts/Makefile.include
@@ -91,6 +91,9 @@ LLVM_CONFIG ?= llvm-config
LLVM_OBJCOPY ?= llvm-objcopy
LLVM_STRIP ?= llvm-strip
+# Some tools require bpftool
+BPFTOOL ?= bpftool
+
ifeq ($(CC_NO_CLANG), 1)
EXTRA_WARNINGS += -Wstrict-aliasing=3
--
2.48.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 3/8] rtla: Add optional dependency on BPF tooling
2025-02-18 14:58 [PATCH 0/8] rtla: Collect data using BPF program Tomas Glozar
2025-02-18 14:58 ` [PATCH 1/8] rtla/timerlat: Unify params struct Tomas Glozar
2025-02-18 14:58 ` [PATCH 2/8] tools/build: Add bpftool-skeletons feature test Tomas Glozar
@ 2025-02-18 14:58 ` Tomas Glozar
2025-02-18 14:58 ` [PATCH 4/8] rtla/timerlat: Add BPF skeleton to collect samples Tomas Glozar
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Tomas Glozar @ 2025-02-18 14:58 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Gabriele Monaco, Clark Williams, Tomas Glozar
If tooling required for building BPF CO-RE skeletons is present (that
is, libbpf, clang with BPF CO-RE support, and bpftool), turn on
HAVE_BPF_SKEL flag.
Those requirements are similar to what perf requires, with the
difference of using system libbpf and bpftool instead of in-tree
versions.
rtla can be forcefully built without BPF skeleton support by setting
BUILD_BPF_SKEL=0 manually; in that case, a warning is displayed.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/Makefile | 6 +++++
tools/tracing/rtla/Makefile.config | 42 ++++++++++++++++++++++++++++++
2 files changed, 48 insertions(+)
diff --git a/tools/tracing/rtla/Makefile b/tools/tracing/rtla/Makefile
index 8b5101457c70..0aa116f7b355 100644
--- a/tools/tracing/rtla/Makefile
+++ b/tools/tracing/rtla/Makefile
@@ -33,9 +33,15 @@ DOCSRC := ../../../Documentation/tools/rtla/
FEATURE_TESTS := libtraceevent
FEATURE_TESTS += libtracefs
FEATURE_TESTS += libcpupower
+FEATURE_TESTS += libbpf
+FEATURE_TESTS += clang-bpf-co-re
+FEATURE_TESTS += bpftool-skeletons
FEATURE_DISPLAY := libtraceevent
FEATURE_DISPLAY += libtracefs
FEATURE_DISPLAY += libcpupower
+FEATURE_DISPLAY += libbpf
+FEATURE_DISPLAY += clang-bpf-co-re
+FEATURE_DISPLAY += bpftool-skeletons
ifeq ($(V),1)
Q =
diff --git a/tools/tracing/rtla/Makefile.config b/tools/tracing/rtla/Makefile.config
index 92a6e12e42d3..5f2231d8d626 100644
--- a/tools/tracing/rtla/Makefile.config
+++ b/tools/tracing/rtla/Makefile.config
@@ -53,6 +53,48 @@ else
$(info Please install libcpupower-dev/kernel-tools-libs-devel)
endif
+ifndef BUILD_BPF_SKEL
+ # BPF skeletons are used to implement improved sample collection, enable
+ # them by default.
+ BUILD_BPF_SKEL := 1
+endif
+
+ifeq ($(BUILD_BPF_SKEL),0)
+ $(info BPF skeleton support disabled, building without BPF skeleton support.)
+endif
+
+$(call feature_check,libbpf)
+ifeq ($(feature-libbpf), 1)
+ $(call detected,CONFIG_LIBBPF)
+else
+ $(info libbpf is missing, building without BPF skeleton support.)
+ $(info Please install libbpf-dev/libbpf-devel)
+ BUILD_BPF_SKEL := 0
+endif
+
+$(call feature_check,clang-bpf-co-re)
+ifeq ($(feature-clang-bpf-co-re), 1)
+ $(call detected,CONFIG_CLANG_BPF_CO_RE)
+else
+ $(info clang is missing or does not support BPF CO-RE, building without BPF skeleton support.)
+ $(info Please install clang)
+ BUILD_BPF_SKEL := 0
+endif
+
+$(call feature_check,bpftool-skeletons)
+ifeq ($(feature-bpftool-skeletons), 1)
+ $(call detected,CONFIG_BPFTOOL_SKELETONS)
+else
+ $(info bpftool is missing or not supporting skeletons, building without BPF skeleton support.)
+ $(info Please install bpftool)
+ BUILD_BPF_SKEL := 0
+endif
+
+ifeq ($(BUILD_BPF_SKEL),1)
+ CFLAGS += -DHAVE_BPF_SKEL
+ EXTLIBS += -lbpf
+endif
+
ifeq ($(STOP_ERROR),1)
$(error Please, check the errors above.)
endif
--
2.48.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 4/8] rtla/timerlat: Add BPF skeleton to collect samples
2025-02-18 14:58 [PATCH 0/8] rtla: Collect data using BPF program Tomas Glozar
` (2 preceding siblings ...)
2025-02-18 14:58 ` [PATCH 3/8] rtla: Add optional dependency on BPF tooling Tomas Glozar
@ 2025-02-18 14:58 ` Tomas Glozar
2025-02-18 14:58 ` [PATCH 5/8] rtla/timerlat_hist: Use BPF " Tomas Glozar
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Tomas Glozar @ 2025-02-18 14:58 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Gabriele Monaco, Clark Williams, Tomas Glozar
Add BPF program that attaches to the osnoise:timerlat_sample tracepoint
and collects both the summary and the histogram (if requested) into BPF
maps (one map of each kind per context).
The program is designed to be used for both timerlat-top and
timerlat-hist. If using with timerlat-top, the "entries" parameter is
set to zero, which prevents the BPF program from recording histogram
entries. In that case, the maps for histograms do not have to be
created, as the BPF verifier will identify the code using them as
unreachable.
An IRQ or thread latency threshold might be supplied to stop recording
if hit, similar to the timerlat tracer threshold, which stops ftrace
tracing if hit. A BPF ringbuffer is used to signal threshold overflow to
userspace. In aa-only mode, this is the only function of the BPF
program.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/.gitignore | 1 +
tools/tracing/rtla/Makefile | 14 ++-
tools/tracing/rtla/src/Build | 1 +
tools/tracing/rtla/src/timerlat.bpf.c | 149 +++++++++++++++++++++++
tools/tracing/rtla/src/timerlat_bpf.c | 166 ++++++++++++++++++++++++++
tools/tracing/rtla/src/timerlat_bpf.h | 59 +++++++++
6 files changed, 389 insertions(+), 1 deletion(-)
create mode 100644 tools/tracing/rtla/src/timerlat.bpf.c
create mode 100644 tools/tracing/rtla/src/timerlat_bpf.c
create mode 100644 tools/tracing/rtla/src/timerlat_bpf.h
diff --git a/tools/tracing/rtla/.gitignore b/tools/tracing/rtla/.gitignore
index 293f0dbb0ca2..1a394ad26cc1 100644
--- a/tools/tracing/rtla/.gitignore
+++ b/tools/tracing/rtla/.gitignore
@@ -4,3 +4,4 @@ rtla-static
fixdep
feature
FEATURE-DUMP
+*.skel.h
diff --git a/tools/tracing/rtla/Makefile b/tools/tracing/rtla/Makefile
index 0aa116f7b355..557af322be61 100644
--- a/tools/tracing/rtla/Makefile
+++ b/tools/tracing/rtla/Makefile
@@ -73,6 +73,17 @@ CFLAGS += $(INCLUDES) $(LIB_INCLUDES)
export CFLAGS OUTPUT srctree
+ifeq ($(BUILD_BPF_SKEL),1)
+src/timerlat.bpf.o: src/timerlat.bpf.c
+ $(QUIET_CLANG)$(CLANG) -g -O2 -target bpf -c $(filter %.c,$^) -o $@
+
+src/timerlat.skel.h: src/timerlat.bpf.o
+ $(QUIET_GENSKEL)$(BPFTOOL) gen skeleton $< > $@
+else
+src/timerlat.skel.h:
+ $(Q)echo '/* BPF skeleton is disabled */' > src/timerlat.skel.h
+endif
+
$(RTLA): $(RTLA_IN)
$(QUIET_LINK)$(CC) $(LDFLAGS) -o $(RTLA) $(RTLA_IN) $(EXTLIBS)
@@ -83,7 +94,7 @@ static: $(RTLA_IN)
rtla.%: fixdep FORCE
make -f $(srctree)/tools/build/Makefile.build dir=. $@
-$(RTLA_IN): fixdep FORCE
+$(RTLA_IN): fixdep FORCE src/timerlat.skel.h
make $(build)=rtla
clean: doc_clean fixdep-clean
@@ -91,6 +102,7 @@ clean: doc_clean fixdep-clean
$(Q)find . -name '*.o' -delete -o -name '\.*.cmd' -delete -o -name '\.*.d' -delete
$(Q)rm -f rtla rtla-static fixdep FEATURE-DUMP rtla-*
$(Q)rm -rf feature
+ $(Q)rm -f src/timerlat.bpf.o src/timerlat.skel.h
check: $(RTLA)
RTLA=$(RTLA) prove -o -f tests/
.PHONY: FORCE clean check
diff --git a/tools/tracing/rtla/src/Build b/tools/tracing/rtla/src/Build
index dbed9e31829b..7bb7e39e391a 100644
--- a/tools/tracing/rtla/src/Build
+++ b/tools/tracing/rtla/src/Build
@@ -8,4 +8,5 @@ rtla-y += timerlat_top.o
rtla-y += timerlat_hist.o
rtla-y += timerlat_u.o
rtla-y += timerlat_aa.o
+rtla-y += timerlat_bpf.o
rtla-y += rtla.o
diff --git a/tools/tracing/rtla/src/timerlat.bpf.c b/tools/tracing/rtla/src/timerlat.bpf.c
new file mode 100644
index 000000000000..96196d46e170
--- /dev/null
+++ b/tools/tracing/rtla/src/timerlat.bpf.c
@@ -0,0 +1,149 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <bpf/bpf_tracing.h>
+#include <stdbool.h>
+#include "timerlat_bpf.h"
+
+#define nosubprog __always_inline
+#define MAX_ENTRIES_DEFAULT 4096
+
+char LICENSE[] SEC("license") = "GPL";
+
+struct trace_event_raw_timerlat_sample {
+ unsigned long long timer_latency;
+ int context;
+} __attribute__((preserve_access_index));
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+ __uint(max_entries, MAX_ENTRIES_DEFAULT);
+ __type(key, unsigned int);
+ __type(value, unsigned long long);
+} hist_irq SEC(".maps"), hist_thread SEC(".maps"), hist_user SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
+ __uint(max_entries, SUMMARY_FIELD_N);
+ __type(key, unsigned int);
+ __type(value, unsigned long long);
+} summary_irq SEC(".maps"), summary_thread SEC(".maps"), summary_user SEC(".maps");
+
+struct {
+ __uint(type, BPF_MAP_TYPE_RINGBUF);
+ __uint(max_entries, 1);
+} signal_stop_tracing SEC(".maps");
+
+/* Params to be set by rtla */
+const volatile int bucket_size = 1;
+const volatile int output_divisor = 1000;
+const volatile int entries = 256;
+const volatile int irq_threshold;
+const volatile int thread_threshold;
+const volatile bool aa_only;
+
+int stop_tracing;
+
+nosubprog unsigned long long map_get(void *map,
+ unsigned int key)
+{
+ unsigned long long *value_ptr;
+
+ value_ptr = bpf_map_lookup_elem(map, &key);
+
+ return !value_ptr ? 0 : *value_ptr;
+}
+
+nosubprog void map_set(void *map,
+ unsigned int key,
+ unsigned long long value)
+{
+ bpf_map_update_elem(map, &key, &value, BPF_ANY);
+}
+
+nosubprog void map_increment(void *map,
+ unsigned int key)
+{
+ map_set(map, key, map_get(map, key) + 1);
+}
+
+nosubprog void update_main_hist(void *map,
+ int bucket)
+{
+ if (entries == 0)
+ /* No histogram */
+ return;
+
+ if (bucket >= entries)
+ /* Overflow */
+ return;
+
+ map_increment(map, bucket);
+}
+
+nosubprog void update_summary(void *map,
+ unsigned long long latency,
+ int bucket)
+{
+ if (aa_only)
+ /* Auto-analysis only, nothing to be done here */
+ return;
+
+ map_set(map, SUMMARY_CURRENT, latency);
+
+ if (bucket >= entries)
+ /* Overflow */
+ map_increment(map, SUMMARY_OVERFLOW);
+
+ if (latency > map_get(map, SUMMARY_MAX))
+ map_set(map, SUMMARY_MAX, latency);
+
+ if (latency < map_get(map, SUMMARY_MIN) || map_get(map, SUMMARY_COUNT) == 0)
+ map_set(map, SUMMARY_MIN, latency);
+
+ map_increment(map, SUMMARY_COUNT);
+ map_set(map, SUMMARY_SUM, map_get(map, SUMMARY_SUM) + latency);
+}
+
+nosubprog void set_stop_tracing(void)
+{
+ int value = 0;
+
+ /* Suppress further sample processing */
+ stop_tracing = 1;
+
+ /* Signal to userspace */
+ bpf_ringbuf_output(&signal_stop_tracing, &value, sizeof(value), 0);
+}
+
+SEC("tp/osnoise/timerlat_sample")
+int handle_timerlat_sample(struct trace_event_raw_timerlat_sample *tp_args)
+{
+ unsigned long long latency, latency_us;
+ int bucket;
+
+ if (stop_tracing)
+ return 0;
+
+ latency = tp_args->timer_latency / output_divisor;
+ latency_us = tp_args->timer_latency / 1000;
+ bucket = latency / bucket_size;
+
+ if (tp_args->context == 0) {
+ update_main_hist(&hist_irq, bucket);
+ update_summary(&summary_irq, latency, bucket);
+
+ if (irq_threshold != 0 && latency_us >= irq_threshold)
+ set_stop_tracing();
+ } else if (tp_args->context == 1) {
+ update_main_hist(&hist_thread, bucket);
+ update_summary(&summary_thread, latency, bucket);
+
+ if (thread_threshold != 0 && latency_us >= thread_threshold)
+ set_stop_tracing();
+ } else {
+ update_main_hist(&hist_user, bucket);
+ update_summary(&summary_user, latency, bucket);
+ }
+
+ return 0;
+}
diff --git a/tools/tracing/rtla/src/timerlat_bpf.c b/tools/tracing/rtla/src/timerlat_bpf.c
new file mode 100644
index 000000000000..5abee884037a
--- /dev/null
+++ b/tools/tracing/rtla/src/timerlat_bpf.c
@@ -0,0 +1,166 @@
+// SPDX-License-Identifier: GPL-2.0
+#ifdef HAVE_BPF_SKEL
+#include "timerlat.h"
+#include "timerlat_bpf.h"
+#include "timerlat.skel.h"
+
+static struct timerlat_bpf *bpf;
+
+/*
+ * timerlat_bpf_init - load and initialize BPF program to collect timerlat data
+ */
+int timerlat_bpf_init(struct timerlat_params *params)
+{
+ int err;
+
+ debug_msg("Loading BPF program\n");
+
+ bpf = timerlat_bpf__open();
+ if (!bpf)
+ return 1;
+
+ /* Pass common options */
+ bpf->rodata->output_divisor = params->output_divisor;
+ bpf->rodata->entries = params->entries;
+ bpf->rodata->irq_threshold = params->stop_us;
+ bpf->rodata->thread_threshold = params->stop_total_us;
+ bpf->rodata->aa_only = params->aa_only;
+
+ if (params->entries != 0) {
+ /* Pass histogram options */
+ bpf->rodata->bucket_size = params->bucket_size;
+
+ /* Set histogram array sizes */
+ bpf_map__set_max_entries(bpf->maps.hist_irq, params->entries);
+ bpf_map__set_max_entries(bpf->maps.hist_thread, params->entries);
+ bpf_map__set_max_entries(bpf->maps.hist_user, params->entries);
+ } else {
+ /* No entries, disable histogram */
+ bpf_map__set_autocreate(bpf->maps.hist_irq, false);
+ bpf_map__set_autocreate(bpf->maps.hist_thread, false);
+ bpf_map__set_autocreate(bpf->maps.hist_user, false);
+ }
+
+ if (params->aa_only) {
+ /* Auto-analysis only, disable summary */
+ bpf_map__set_autocreate(bpf->maps.summary_irq, false);
+ bpf_map__set_autocreate(bpf->maps.summary_thread, false);
+ bpf_map__set_autocreate(bpf->maps.summary_user, false);
+ }
+
+ /* Load and verify BPF program */
+ err = timerlat_bpf__load(bpf);
+ if (err) {
+ timerlat_bpf__destroy(bpf);
+ return err;
+ }
+
+ return 0;
+}
+
+/*
+ * timerlat_bpf_attach - attach BPF program to collect timerlat data
+ */
+int timerlat_bpf_attach(void)
+{
+ debug_msg("Attaching BPF program\n");
+
+ return timerlat_bpf__attach(bpf);
+}
+
+/*
+ * timerlat_bpf_detach - detach BPF program to collect timerlat data
+ */
+void timerlat_bpf_detach(void)
+{
+ timerlat_bpf__detach(bpf);
+}
+
+/*
+ * timerlat_bpf_detach - destroy BPF program to collect timerlat data
+ */
+void timerlat_bpf_destroy(void)
+{
+ timerlat_bpf__destroy(bpf);
+}
+
+static int handle_rb_event(void *ctx, void *data, size_t data_sz)
+{
+ return 0;
+}
+
+/*
+ * timerlat_bpf_wait - wait until tracing is stopped or signal
+ */
+int timerlat_bpf_wait(int timeout)
+{
+ struct ring_buffer *rb;
+ int retval;
+
+ rb = ring_buffer__new(bpf_map__fd(bpf->maps.signal_stop_tracing),
+ handle_rb_event, NULL, NULL);
+ retval = ring_buffer__poll(rb, timeout * 1000);
+ ring_buffer__free(rb);
+
+ return retval;
+}
+
+static int get_value(struct bpf_map *map_irq,
+ struct bpf_map *map_thread,
+ struct bpf_map *map_user,
+ int key,
+ long long *value_irq,
+ long long *value_thread,
+ long long *value_user,
+ int cpus)
+{
+ int err;
+
+ err = bpf_map__lookup_elem(map_irq, &key,
+ sizeof(unsigned int), value_irq,
+ sizeof(long long) * cpus, 0);
+ if (err)
+ return err;
+ err = bpf_map__lookup_elem(map_thread, &key,
+ sizeof(unsigned int), value_thread,
+ sizeof(long long) * cpus, 0);
+ if (err)
+ return err;
+ err = bpf_map__lookup_elem(map_user, &key,
+ sizeof(unsigned int), value_user,
+ sizeof(long long) * cpus, 0);
+ if (err)
+ return err;
+ return 0;
+}
+
+/*
+ * timerlat_bpf_get_hist_value - get value from BPF hist map
+ */
+int timerlat_bpf_get_hist_value(int key,
+ long long *value_irq,
+ long long *value_thread,
+ long long *value_user,
+ int cpus)
+{
+ return get_value(bpf->maps.hist_irq,
+ bpf->maps.hist_thread,
+ bpf->maps.hist_user,
+ key, value_irq, value_thread, value_user, cpus);
+}
+
+/*
+ * timerlat_bpf_get_summary_value - get value from BPF summary map
+ */
+int timerlat_bpf_get_summary_value(enum summary_field key,
+ long long *value_irq,
+ long long *value_thread,
+ long long *value_user,
+ int cpus)
+{
+ return get_value(bpf->maps.summary_irq,
+ bpf->maps.summary_thread,
+ bpf->maps.summary_user,
+ key, value_irq, value_thread, value_user, cpus);
+}
+#endif /* HAVE_BPF_SKEL */
diff --git a/tools/tracing/rtla/src/timerlat_bpf.h b/tools/tracing/rtla/src/timerlat_bpf.h
new file mode 100644
index 000000000000..f1b54dbddb0e
--- /dev/null
+++ b/tools/tracing/rtla/src/timerlat_bpf.h
@@ -0,0 +1,59 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#pragma once
+
+enum summary_field {
+ SUMMARY_CURRENT,
+ SUMMARY_MIN,
+ SUMMARY_MAX,
+ SUMMARY_COUNT,
+ SUMMARY_SUM,
+ SUMMARY_OVERFLOW,
+ SUMMARY_FIELD_N
+};
+
+#ifndef __bpf__
+#ifdef HAVE_BPF_SKEL
+int timerlat_bpf_init(struct timerlat_params *params);
+int timerlat_bpf_attach(void);
+void timerlat_bpf_detach(void);
+void timerlat_bpf_destroy(void);
+int timerlat_bpf_wait(int timeout);
+int timerlat_bpf_get_hist_value(int key,
+ long long *value_irq,
+ long long *value_thread,
+ long long *value_user,
+ int cpus);
+int timerlat_bpf_get_summary_value(enum summary_field key,
+ long long *value_irq,
+ long long *value_thread,
+ long long *value_user,
+ int cpus);
+static inline int have_libbpf_support(void) { return 1; }
+#else
+static inline int timerlat_bpf_init(struct timerlat_params *params)
+{
+ return -1;
+}
+static inline int timerlat_bpf_attach(void) { return -1; }
+static inline void timerlat_bpf_detach(void) { };
+static inline void timerlat_bpf_destroy(void) { };
+static inline int timerlat_bpf_wait(int timeout) { return -1; }
+static inline int timerlat_bpf_get_hist_value(int key,
+ long long *value_irq,
+ long long *value_thread,
+ long long *value_user,
+ int cpus)
+{
+ return -1;
+}
+static inline int timerlat_bpf_get_summary_value(enum summary_field key,
+ long long *value_irq,
+ long long *value_thread,
+ long long *value_user,
+ int cpus)
+{
+ return -1;
+}
+static inline int have_libbpf_support(void) { return 0; }
+#endif /* HAVE_BPF_SKEL */
+#endif /* __bpf__ */
--
2.48.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 5/8] rtla/timerlat_hist: Use BPF to collect samples
2025-02-18 14:58 [PATCH 0/8] rtla: Collect data using BPF program Tomas Glozar
` (3 preceding siblings ...)
2025-02-18 14:58 ` [PATCH 4/8] rtla/timerlat: Add BPF skeleton to collect samples Tomas Glozar
@ 2025-02-18 14:58 ` Tomas Glozar
2025-02-18 14:58 ` [PATCH 6/8] rtla/timerlat_top: Move divisor to update Tomas Glozar
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Tomas Glozar @ 2025-02-18 14:58 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Gabriele Monaco, Clark Williams, Tomas Glozar
Collect samples using BPF program instead of pulling them from tracefs.
If the osnoise:timerlat_sample tracepoint is unavailable or the BPF
program fails to load for whatever reason, rtla falls back to the old
implementation.
The collection of samples using the BPF program is fully self-contained
and requires no activity of the userspace part of rtla during the
measurement. Thus, instead of waking up every second to collect samples,
rtla simply sleeps until woken up by a signal or threshold overflow.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/src/timerlat_hist.c | 167 +++++++++++++++++++++----
1 file changed, 146 insertions(+), 21 deletions(-)
diff --git a/tools/tracing/rtla/src/timerlat_hist.c b/tools/tracing/rtla/src/timerlat_hist.c
index fe683a20b0e2..202d99a598ba 100644
--- a/tools/tracing/rtla/src/timerlat_hist.c
+++ b/tools/tracing/rtla/src/timerlat_hist.c
@@ -17,6 +17,7 @@
#include "timerlat.h"
#include "timerlat_aa.h"
#include "timerlat_u.h"
+#include "timerlat_bpf.h"
struct timerlat_hist_cpu {
int *irq;
@@ -193,6 +194,89 @@ timerlat_hist_handler(struct trace_seq *s, struct tep_record *record,
return 0;
}
+/*
+ * timerlat_hist_bpf_pull_data - copy data from BPF maps into userspace
+ */
+static int timerlat_hist_bpf_pull_data(struct osnoise_tool *tool)
+{
+ struct timerlat_hist_data *data = tool->data;
+ int i, j, err;
+ long long value_irq[data->nr_cpus],
+ value_thread[data->nr_cpus],
+ value_user[data->nr_cpus];
+
+ /* Pull histogram */
+ for (i = 0; i < data->entries; i++) {
+ err = timerlat_bpf_get_hist_value(i, value_irq, value_thread,
+ value_user, data->nr_cpus);
+ if (err)
+ return err;
+ for (j = 0; j < data->nr_cpus; j++) {
+ data->hist[j].irq[i] = value_irq[j];
+ data->hist[j].thread[i] = value_thread[j];
+ data->hist[j].user[i] = value_user[j];
+ }
+ }
+
+ /* Pull summary */
+ err = timerlat_bpf_get_summary_value(SUMMARY_COUNT,
+ value_irq, value_thread, value_user,
+ data->nr_cpus);
+ if (err)
+ return err;
+ for (i = 0; i < data->nr_cpus; i++) {
+ data->hist[i].irq_count = value_irq[i];
+ data->hist[i].thread_count = value_thread[i];
+ data->hist[i].user_count = value_user[i];
+ }
+
+ err = timerlat_bpf_get_summary_value(SUMMARY_MIN,
+ value_irq, value_thread, value_user,
+ data->nr_cpus);
+ if (err)
+ return err;
+ for (i = 0; i < data->nr_cpus; i++) {
+ data->hist[i].min_irq = value_irq[i];
+ data->hist[i].min_thread = value_thread[i];
+ data->hist[i].min_user = value_user[i];
+ }
+
+ err = timerlat_bpf_get_summary_value(SUMMARY_MAX,
+ value_irq, value_thread, value_user,
+ data->nr_cpus);
+ if (err)
+ return err;
+ for (i = 0; i < data->nr_cpus; i++) {
+ data->hist[i].max_irq = value_irq[i];
+ data->hist[i].max_thread = value_thread[i];
+ data->hist[i].max_user = value_user[i];
+ }
+
+ err = timerlat_bpf_get_summary_value(SUMMARY_SUM,
+ value_irq, value_thread, value_user,
+ data->nr_cpus);
+ if (err)
+ return err;
+ for (i = 0; i < data->nr_cpus; i++) {
+ data->hist[i].sum_irq = value_irq[i];
+ data->hist[i].sum_thread = value_thread[i];
+ data->hist[i].sum_user = value_user[i];
+ }
+
+ err = timerlat_bpf_get_summary_value(SUMMARY_OVERFLOW,
+ value_irq, value_thread, value_user,
+ data->nr_cpus);
+ if (err)
+ return err;
+ for (i = 0; i < data->nr_cpus; i++) {
+ data->hist[i].irq[data->entries] = value_irq[i];
+ data->hist[i].thread[data->entries] = value_thread[i];
+ data->hist[i].user[data->entries] = value_user[i];
+ }
+
+ return 0;
+}
+
/*
* timerlat_hist_header - print the header of the tracer to the output
*/
@@ -1152,6 +1236,7 @@ int timerlat_hist_main(int argc, char *argv[])
pthread_t timerlat_u;
int retval;
int nr_cpus, i;
+ bool no_bpf = false;
params = timerlat_hist_parse_args(argc, argv);
if (!params)
@@ -1177,6 +1262,24 @@ int timerlat_hist_main(int argc, char *argv[])
*/
hist_inst = trace;
+ if (getenv("RTLA_NO_BPF") && strncmp(getenv("RTLA_NO_BPF"), "1", 2) == 0) {
+ debug_msg("RTLA_NO_BPF set, disabling BPF\n");
+ no_bpf = true;
+ }
+
+ if (!no_bpf && !tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sample")) {
+ debug_msg("osnoise:timerlat_sample missing, disabling BPF\n");
+ no_bpf = true;
+ }
+
+ if (!no_bpf) {
+ retval = timerlat_bpf_init(params);
+ if (retval) {
+ debug_msg("Could not enable BPF\n");
+ no_bpf = true;
+ }
+ }
+
retval = enable_timerlat(trace);
if (retval) {
err_msg("Failed to enable timerlat tracer\n");
@@ -1304,35 +1407,55 @@ int timerlat_hist_main(int argc, char *argv[])
trace_instance_start(&record->trace);
if (!params->no_aa)
trace_instance_start(&aa->trace);
- trace_instance_start(trace);
+ if (no_bpf) {
+ trace_instance_start(trace);
+ } else {
+ retval = timerlat_bpf_attach();
+ if (retval) {
+ err_msg("Error attaching BPF program\n");
+ goto out_hist;
+ }
+ }
tool->start_time = time(NULL);
timerlat_hist_set_signals(params);
- while (!stop_tracing) {
- sleep(params->sleep_time);
-
- retval = tracefs_iterate_raw_events(trace->tep,
- trace->inst,
- NULL,
- 0,
- collect_registered_events,
- trace);
- if (retval < 0) {
- err_msg("Error iterating on events\n");
- goto out_hist;
- }
-
- if (osnoise_trace_is_off(tool, record))
- break;
+ if (no_bpf) {
+ while (!stop_tracing) {
+ sleep(params->sleep_time);
+
+ retval = tracefs_iterate_raw_events(trace->tep,
+ trace->inst,
+ NULL,
+ 0,
+ collect_registered_events,
+ trace);
+ if (retval < 0) {
+ err_msg("Error iterating on events\n");
+ goto out_hist;
+ }
- /* is there still any user-threads ? */
- if (params->user_workload) {
- if (params_u.stopped_running) {
- debug_msg("timerlat user-space threads stopped!\n");
+ if (osnoise_trace_is_off(tool, record))
break;
+
+ /* is there still any user-threads ? */
+ if (params->user_workload) {
+ if (params_u.stopped_running) {
+ debug_msg("timerlat user-space threads stopped!\n");
+ break;
+ }
}
}
+ } else
+ timerlat_bpf_wait(-1);
+
+ if (!no_bpf) {
+ timerlat_bpf_detach();
+ retval = timerlat_hist_bpf_pull_data(tool);
+ if (retval) {
+ err_msg("Error pulling BPF data\n");
+ goto out_hist;
+ }
}
if (params->user_workload && !params_u.stopped_running) {
@@ -1376,6 +1499,8 @@ int timerlat_hist_main(int argc, char *argv[])
osnoise_destroy_tool(tool);
free(params);
free_cpu_idle_disable_states();
+ if (!no_bpf)
+ timerlat_bpf_destroy();
out_exit:
exit(return_value);
}
--
2.48.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 6/8] rtla/timerlat_top: Move divisor to update
2025-02-18 14:58 [PATCH 0/8] rtla: Collect data using BPF program Tomas Glozar
` (4 preceding siblings ...)
2025-02-18 14:58 ` [PATCH 5/8] rtla/timerlat_hist: Use BPF " Tomas Glozar
@ 2025-02-18 14:58 ` Tomas Glozar
2025-02-18 14:58 ` [PATCH 7/8] rtla/timerlat_top: Use BPF to collect samples Tomas Glozar
2025-02-18 14:58 ` [PATCH 8/8] rtla/timerlat: Test BPF mode Tomas Glozar
7 siblings, 0 replies; 9+ messages in thread
From: Tomas Glozar @ 2025-02-18 14:58 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Gabriele Monaco, Clark Williams, Tomas Glozar
Unlike timerlat-hist, timerlat-top applies the output divisor used to
set ns/us mode when printing results instead of applying it when
collecting the samples.
Move the application of the divisor from timerlat_top_print into
timerlat_top_update to make it consistent with timerlat-hist.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/src/timerlat_top.c | 54 +++++++++++++--------------
1 file changed, 25 insertions(+), 29 deletions(-)
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index 97eead91decc..a3d9e8f67a4f 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -127,9 +127,13 @@ timerlat_top_update(struct osnoise_tool *tool, int cpu,
unsigned long long thread,
unsigned long long latency)
{
+ struct timerlat_params *params = tool->params;
struct timerlat_top_data *data = tool->data;
struct timerlat_top_cpu *cpu_data = &data->cpu_data[cpu];
+ if (params->output_divisor)
+ latency = latency / params->output_divisor;
+
if (!thread) {
cpu_data->irq_count++;
cpu_data->cur_irq = latency;
@@ -231,12 +235,8 @@ static void timerlat_top_print(struct osnoise_tool *top, int cpu)
struct timerlat_params *params = top->params;
struct timerlat_top_data *data = top->data;
struct timerlat_top_cpu *cpu_data = &data->cpu_data[cpu];
- int divisor = params->output_divisor;
struct trace_seq *s = top->trace.seq;
- if (divisor == 0)
- return;
-
/*
* Skip if no data is available: is this cpu offline?
*/
@@ -251,20 +251,20 @@ static void timerlat_top_print(struct osnoise_tool *top, int cpu)
if (!cpu_data->irq_count) {
trace_seq_printf(s, "%s %s %s %s |", no_value, no_value, no_value, no_value);
} else {
- trace_seq_printf(s, "%9llu ", cpu_data->cur_irq / params->output_divisor);
- trace_seq_printf(s, "%9llu ", cpu_data->min_irq / params->output_divisor);
- trace_seq_printf(s, "%9llu ", (cpu_data->sum_irq / cpu_data->irq_count) / divisor);
- trace_seq_printf(s, "%9llu |", cpu_data->max_irq / divisor);
+ trace_seq_printf(s, "%9llu ", cpu_data->cur_irq);
+ trace_seq_printf(s, "%9llu ", cpu_data->min_irq);
+ trace_seq_printf(s, "%9llu ", cpu_data->sum_irq / cpu_data->irq_count);
+ trace_seq_printf(s, "%9llu |", cpu_data->max_irq);
}
if (!cpu_data->thread_count) {
trace_seq_printf(s, "%s %s %s %s", no_value, no_value, no_value, no_value);
} else {
- trace_seq_printf(s, "%9llu ", cpu_data->cur_thread / divisor);
- trace_seq_printf(s, "%9llu ", cpu_data->min_thread / divisor);
+ trace_seq_printf(s, "%9llu ", cpu_data->cur_thread);
+ trace_seq_printf(s, "%9llu ", cpu_data->min_thread);
trace_seq_printf(s, "%9llu ",
- (cpu_data->sum_thread / cpu_data->thread_count) / divisor);
- trace_seq_printf(s, "%9llu", cpu_data->max_thread / divisor);
+ cpu_data->sum_thread / cpu_data->thread_count);
+ trace_seq_printf(s, "%9llu", cpu_data->max_thread);
}
if (!params->user_top) {
@@ -277,11 +277,11 @@ static void timerlat_top_print(struct osnoise_tool *top, int cpu)
if (!cpu_data->user_count) {
trace_seq_printf(s, "%s %s %s %s\n", no_value, no_value, no_value, no_value);
} else {
- trace_seq_printf(s, "%9llu ", cpu_data->cur_user / divisor);
- trace_seq_printf(s, "%9llu ", cpu_data->min_user / divisor);
+ trace_seq_printf(s, "%9llu ", cpu_data->cur_user);
+ trace_seq_printf(s, "%9llu ", cpu_data->min_user);
trace_seq_printf(s, "%9llu ",
- (cpu_data->sum_user / cpu_data->user_count) / divisor);
- trace_seq_printf(s, "%9llu\n", cpu_data->max_user / divisor);
+ cpu_data->sum_user / cpu_data->user_count);
+ trace_seq_printf(s, "%9llu\n", cpu_data->max_user);
}
}
@@ -294,13 +294,9 @@ timerlat_top_print_sum(struct osnoise_tool *top, struct timerlat_top_cpu *summar
const char *split = "----------------------------------------";
struct timerlat_params *params = top->params;
unsigned long long count = summary->irq_count;
- int divisor = params->output_divisor;
struct trace_seq *s = top->trace.seq;
int e = 0;
- if (divisor == 0)
- return;
-
/*
* Skip if no data is available: is this cpu offline?
*/
@@ -323,19 +319,19 @@ timerlat_top_print_sum(struct osnoise_tool *top, struct timerlat_top_cpu *summar
trace_seq_printf(s, " %s %s %s |", no_value, no_value, no_value);
} else {
trace_seq_printf(s, " ");
- trace_seq_printf(s, "%9llu ", summary->min_irq / params->output_divisor);
- trace_seq_printf(s, "%9llu ", (summary->sum_irq / summary->irq_count) / divisor);
- trace_seq_printf(s, "%9llu |", summary->max_irq / divisor);
+ trace_seq_printf(s, "%9llu ", summary->min_irq);
+ trace_seq_printf(s, "%9llu ", summary->sum_irq / summary->irq_count);
+ trace_seq_printf(s, "%9llu |", summary->max_irq);
}
if (!summary->thread_count) {
trace_seq_printf(s, "%s %s %s %s", no_value, no_value, no_value, no_value);
} else {
trace_seq_printf(s, " ");
- trace_seq_printf(s, "%9llu ", summary->min_thread / divisor);
+ trace_seq_printf(s, "%9llu ", summary->min_thread);
trace_seq_printf(s, "%9llu ",
- (summary->sum_thread / summary->thread_count) / divisor);
- trace_seq_printf(s, "%9llu", summary->max_thread / divisor);
+ summary->sum_thread / summary->thread_count);
+ trace_seq_printf(s, "%9llu", summary->max_thread);
}
if (!params->user_top) {
@@ -349,10 +345,10 @@ timerlat_top_print_sum(struct osnoise_tool *top, struct timerlat_top_cpu *summar
trace_seq_printf(s, " %s %s %s |", no_value, no_value, no_value);
} else {
trace_seq_printf(s, " ");
- trace_seq_printf(s, "%9llu ", summary->min_user / divisor);
+ trace_seq_printf(s, "%9llu ", summary->min_user);
trace_seq_printf(s, "%9llu ",
- (summary->sum_user / summary->user_count) / divisor);
- trace_seq_printf(s, "%9llu\n", summary->max_user / divisor);
+ summary->sum_user / summary->user_count);
+ trace_seq_printf(s, "%9llu\n", summary->max_user);
}
}
--
2.48.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 7/8] rtla/timerlat_top: Use BPF to collect samples
2025-02-18 14:58 [PATCH 0/8] rtla: Collect data using BPF program Tomas Glozar
` (5 preceding siblings ...)
2025-02-18 14:58 ` [PATCH 6/8] rtla/timerlat_top: Move divisor to update Tomas Glozar
@ 2025-02-18 14:58 ` Tomas Glozar
2025-02-18 14:58 ` [PATCH 8/8] rtla/timerlat: Test BPF mode Tomas Glozar
7 siblings, 0 replies; 9+ messages in thread
From: Tomas Glozar @ 2025-02-18 14:58 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Gabriele Monaco, Clark Williams, Tomas Glozar
Collect samples using BPF program instead of pulling them from tracefs.
If the osnoise:timerlat_sample tracepoint is unavailable or the BPF
program fails to load for whatever reason, rtla falls back to the old
implementation.
The collection of samples using the BPF program is fully self-contained
and requires no activity of the userspace part of rtla during the
measurement. Thus, rtla only pulls the summary from the BPF map and
displays it every second, improving the performance.
In --aa-only mode, the BPF program does not collect any data and only
signalizes the end of tracing to userspace. An optimization that re-used
the main trace instance for auto-analysis in aa-only mode was dropped, as
rtla no longer turns tracing on in the main trace instance, making it
useless for auto-analysis.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/src/timerlat_top.c | 255 ++++++++++++++++++++++----
1 file changed, 215 insertions(+), 40 deletions(-)
diff --git a/tools/tracing/rtla/src/timerlat_top.c b/tools/tracing/rtla/src/timerlat_top.c
index a3d9e8f67a4f..1b6455b9e093 100644
--- a/tools/tracing/rtla/src/timerlat_top.c
+++ b/tools/tracing/rtla/src/timerlat_top.c
@@ -18,6 +18,7 @@
#include "timerlat.h"
#include "timerlat_aa.h"
#include "timerlat_u.h"
+#include "timerlat_bpf.h"
struct timerlat_top_cpu {
unsigned long long irq_count;
@@ -181,6 +182,76 @@ timerlat_top_handler(struct trace_seq *s, struct tep_record *record,
return 0;
}
+/*
+ * timerlat_top_bpf_pull_data - copy data from BPF maps into userspace
+ */
+static int timerlat_top_bpf_pull_data(struct osnoise_tool *tool)
+{
+ struct timerlat_top_data *data = tool->data;
+ int i, err;
+ long long value_irq[data->nr_cpus],
+ value_thread[data->nr_cpus],
+ value_user[data->nr_cpus];
+
+ /* Pull summary */
+ err = timerlat_bpf_get_summary_value(SUMMARY_CURRENT,
+ value_irq, value_thread, value_user,
+ data->nr_cpus);
+ if (err)
+ return err;
+ for (i = 0; i < data->nr_cpus; i++) {
+ data->cpu_data[i].cur_irq = value_irq[i];
+ data->cpu_data[i].cur_thread = value_thread[i];
+ data->cpu_data[i].cur_user = value_user[i];
+ }
+
+ err = timerlat_bpf_get_summary_value(SUMMARY_COUNT,
+ value_irq, value_thread, value_user,
+ data->nr_cpus);
+ if (err)
+ return err;
+ for (i = 0; i < data->nr_cpus; i++) {
+ data->cpu_data[i].irq_count = value_irq[i];
+ data->cpu_data[i].thread_count = value_thread[i];
+ data->cpu_data[i].user_count = value_user[i];
+ }
+
+ err = timerlat_bpf_get_summary_value(SUMMARY_MIN,
+ value_irq, value_thread, value_user,
+ data->nr_cpus);
+ if (err)
+ return err;
+ for (i = 0; i < data->nr_cpus; i++) {
+ data->cpu_data[i].min_irq = value_irq[i];
+ data->cpu_data[i].min_thread = value_thread[i];
+ data->cpu_data[i].min_user = value_user[i];
+ }
+
+ err = timerlat_bpf_get_summary_value(SUMMARY_MAX,
+ value_irq, value_thread, value_user,
+ data->nr_cpus);
+ if (err)
+ return err;
+ for (i = 0; i < data->nr_cpus; i++) {
+ data->cpu_data[i].max_irq = value_irq[i];
+ data->cpu_data[i].max_thread = value_thread[i];
+ data->cpu_data[i].max_user = value_user[i];
+ }
+
+ err = timerlat_bpf_get_summary_value(SUMMARY_SUM,
+ value_irq, value_thread, value_user,
+ data->nr_cpus);
+ if (err)
+ return err;
+ for (i = 0; i < data->nr_cpus; i++) {
+ data->cpu_data[i].sum_irq = value_irq[i];
+ data->cpu_data[i].sum_thread = value_thread[i];
+ data->cpu_data[i].sum_user = value_user[i];
+ }
+
+ return 0;
+}
+
/*
* timerlat_top_header - print the header of the tool output
*/
@@ -894,6 +965,111 @@ timerlat_top_set_signals(struct timerlat_params *params)
}
}
+/*
+ * timerlat_top_main_loop - main loop to process events
+ */
+static int
+timerlat_top_main_loop(struct osnoise_tool *top,
+ struct osnoise_tool *record,
+ struct timerlat_params *params,
+ struct timerlat_u_params *params_u)
+{
+ struct trace_instance *trace = &top->trace;
+ int retval;
+
+ while (!stop_tracing) {
+ sleep(params->sleep_time);
+
+ if (params->aa_only && !osnoise_trace_is_off(top, record))
+ continue;
+
+ retval = tracefs_iterate_raw_events(trace->tep,
+ trace->inst,
+ NULL,
+ 0,
+ collect_registered_events,
+ trace);
+ if (retval < 0) {
+ err_msg("Error iterating on events\n");
+ return retval;
+ }
+
+ if (!params->quiet)
+ timerlat_print_stats(params, top);
+
+ if (osnoise_trace_is_off(top, record))
+ break;
+
+ /* is there still any user-threads ? */
+ if (params->user_workload) {
+ if (params_u->stopped_running) {
+ debug_msg("timerlat user space threads stopped!\n");
+ break;
+ }
+ }
+ }
+
+ return 0;
+}
+
+/*
+ * timerlat_top_bpf_main_loop - main loop to process events (BPF variant)
+ */
+static int
+timerlat_top_bpf_main_loop(struct osnoise_tool *top,
+ struct osnoise_tool *record,
+ struct timerlat_params *params,
+ struct timerlat_u_params *params_u)
+{
+ int retval, wait_retval;
+
+ if (params->aa_only) {
+ /* Auto-analysis only, just wait for stop tracing */
+ timerlat_bpf_wait(-1);
+ return 0;
+ }
+
+ if (params->quiet) {
+ /* Quiet mode: wait for stop and then, print results */
+ timerlat_bpf_wait(-1);
+
+ retval = timerlat_top_bpf_pull_data(top);
+ if (retval) {
+ err_msg("Error pulling BPF data\n");
+ return retval;
+ }
+
+ return 0;
+ }
+
+ /* Pull and display data in a loop */
+ while (!stop_tracing) {
+ wait_retval = timerlat_bpf_wait(params->sleep_time);
+
+ retval = timerlat_top_bpf_pull_data(top);
+ if (retval) {
+ err_msg("Error pulling BPF data\n");
+ return retval;
+ }
+
+ timerlat_print_stats(params, top);
+
+ if (wait_retval == 1)
+ /* Stopping requested by tracer */
+ break;
+
+ /* is there still any user-threads ? */
+ if (params->user_workload) {
+ if (params_u->stopped_running) {
+ debug_msg("timerlat user space threads stopped!\n");
+ break;
+ }
+ }
+ }
+
+ return 0;
+}
+
int timerlat_top_main(int argc, char *argv[])
{
struct timerlat_params *params;
@@ -908,6 +1084,7 @@ int timerlat_top_main(int argc, char *argv[])
char *max_lat;
int retval;
int nr_cpus, i;
+ bool no_bpf = false;
params = timerlat_top_parse_args(argc, argv);
if (!params)
@@ -933,6 +1110,23 @@ int timerlat_top_main(int argc, char *argv[])
*/
top_inst = trace;
+ if (getenv("RTLA_NO_BPF") && strncmp(getenv("RTLA_NO_BPF"), "1", 2) == 0) {
+ debug_msg("RTLA_NO_BPF set, disabling BPF\n");
+ no_bpf = true;
+ }
+
+ if (!no_bpf && !tep_find_event_by_name(trace->tep, "osnoise", "timerlat_sample")) {
+ debug_msg("osnoise:timerlat_sample missing, disabling BPF\n");
+ no_bpf = true;
+ }
+
+ if (!no_bpf) {
+ retval = timerlat_bpf_init(params);
+ if (retval) {
+ debug_msg("Could not enable BPF\n");
+ no_bpf = true;
+ }
+ }
retval = enable_timerlat(trace);
if (retval) {
@@ -1007,15 +1201,9 @@ int timerlat_top_main(int argc, char *argv[])
}
if (!params->no_aa) {
- if (params->aa_only) {
- /* as top is not used for display, use it for aa */
- aa = top;
- } else {
- /* otherwise, a new instance is needed */
- aa = osnoise_init_tool("timerlat_aa");
- if (!aa)
- goto out_top;
- }
+ aa = osnoise_init_tool("timerlat_aa");
+ if (!aa)
+ goto out_top;
retval = timerlat_aa_init(aa, params->dump_tasks);
if (retval) {
@@ -1066,44 +1254,31 @@ int timerlat_top_main(int argc, char *argv[])
*/
if (params->trace_output)
trace_instance_start(&record->trace);
- if (!params->no_aa && aa != top)
+ if (!params->no_aa)
trace_instance_start(&aa->trace);
- trace_instance_start(trace);
+ if (no_bpf) {
+ trace_instance_start(trace);
+ } else {
+ retval = timerlat_bpf_attach();
+ if (retval) {
+ err_msg("Error attaching BPF program\n");
+ goto out_top;
+ }
+ }
top->start_time = time(NULL);
timerlat_top_set_signals(params);
- while (!stop_tracing) {
- sleep(params->sleep_time);
+ if (no_bpf)
+ retval = timerlat_top_main_loop(top, record, params, ¶ms_u);
+ else
+ retval = timerlat_top_bpf_main_loop(top, record, params, ¶ms_u);
- if (params->aa_only && !osnoise_trace_is_off(top, record))
- continue;
+ if (retval)
+ goto out_top;
- retval = tracefs_iterate_raw_events(trace->tep,
- trace->inst,
- NULL,
- 0,
- collect_registered_events,
- trace);
- if (retval < 0) {
- err_msg("Error iterating on events\n");
- goto out_top;
- }
-
- if (!params->quiet)
- timerlat_print_stats(params, top);
-
- if (osnoise_trace_is_off(top, record))
- break;
-
- /* is there still any user-threads ? */
- if (params->user_workload) {
- if (params_u.stopped_running) {
- debug_msg("timerlat user space threads stopped!\n");
- break;
- }
- }
- }
+ if (!no_bpf)
+ timerlat_bpf_detach();
if (params->user_workload && !params_u.stopped_running) {
params_u.should_run = 0;
--
2.48.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 8/8] rtla/timerlat: Test BPF mode
2025-02-18 14:58 [PATCH 0/8] rtla: Collect data using BPF program Tomas Glozar
` (6 preceding siblings ...)
2025-02-18 14:58 ` [PATCH 7/8] rtla/timerlat_top: Use BPF to collect samples Tomas Glozar
@ 2025-02-18 14:58 ` Tomas Glozar
7 siblings, 0 replies; 9+ messages in thread
From: Tomas Glozar @ 2025-02-18 14:58 UTC (permalink / raw)
To: Steven Rostedt
Cc: linux-trace-kernel, linux-kernel, John Kacur, Luis Goncalves,
Gabriele Monaco, Clark Williams, Tomas Glozar
Using the RTLA_NO_BPF environmental variable, execute rtla-timerlat
tests both with and without BPF support to cover both paths.
If rtla is built without BPF or the osnoise:timerlat_sample trace event
is not available, test only the non-BPF path.
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
---
tools/tracing/rtla/tests/timerlat.t | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/tools/tracing/rtla/tests/timerlat.t b/tools/tracing/rtla/tests/timerlat.t
index e86f40e5749e..e939ff71d6be 100644
--- a/tools/tracing/rtla/tests/timerlat.t
+++ b/tools/tracing/rtla/tests/timerlat.t
@@ -4,7 +4,20 @@ source tests/engine.sh
test_begin
set_timeout 2m
+timerlat_sample_event='/sys/kernel/tracing/events/osnoise/timerlat_sample'
+if ldd $RTLA | grep libbpf >/dev/null && [ -d "$timerlat_sample_event" ]
+then
+ # rtla build with BPF and system supports BPF mode
+ no_bpf_options='0 1'
+else
+ no_bpf_options='1'
+fi
+
+# Do every test with and without BPF
+for option in $no_bpf_options
+do
+export RTLA_NO_BPF=$option
check "verify help page" \
"timerlat --help"
check "verify -s/--stack" \
@@ -23,5 +36,6 @@ check "verify -c/--cpus" \
"timerlat hist -c 0 -d 30s"
check "hist test in nanoseconds" \
"timerlat hist -i 2 -c 0 -n -d 30s"
+done
test_end
--
2.48.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2025-02-18 14:59 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-02-18 14:58 [PATCH 0/8] rtla: Collect data using BPF program Tomas Glozar
2025-02-18 14:58 ` [PATCH 1/8] rtla/timerlat: Unify params struct Tomas Glozar
2025-02-18 14:58 ` [PATCH 2/8] tools/build: Add bpftool-skeletons feature test Tomas Glozar
2025-02-18 14:58 ` [PATCH 3/8] rtla: Add optional dependency on BPF tooling Tomas Glozar
2025-02-18 14:58 ` [PATCH 4/8] rtla/timerlat: Add BPF skeleton to collect samples Tomas Glozar
2025-02-18 14:58 ` [PATCH 5/8] rtla/timerlat_hist: Use BPF " Tomas Glozar
2025-02-18 14:58 ` [PATCH 6/8] rtla/timerlat_top: Move divisor to update Tomas Glozar
2025-02-18 14:58 ` [PATCH 7/8] rtla/timerlat_top: Use BPF to collect samples Tomas Glozar
2025-02-18 14:58 ` [PATCH 8/8] rtla/timerlat: Test BPF mode Tomas Glozar
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).