[RFC 0/3] rv: per-edge dwell-time statistics for per-cpu monitors

[RFC 0/3] rv: per-edge dwell-time statistics for per-cpu monitors

[Tobias Schaffner]

Hi Steven, Gabriele,

I am using the RV system for a downstream project called EVL (aka Xenomai4).
It is a small co-kernel system, small enough that I can model the whole
co-kernel with RV instead of just a fragment of it.

The patches in this series add a per-monitor "stats" file that exposes how
often an edge has been visited and the dwell time in the prior state for
per-cpu monitors.

This allows me to not only check if the model is correct but also to see if
all paths are exercised by my stress tooling and to decompose the latency of
the co-kernel's wakeup path.

I think that some of the upstream monitors like e.g. the sts monitor could also
profit from this.

As an example, here is the sts monitor on a StarFive VisionFive2 running an rt
kernel (isolcpus=2-3), after enabling it and generating some scheduler load.
Only the edges the automaton actually takes are non-zero:

  # cat monitors/sched/sts/stats
  # cpu edge label count max_ns sum_ns
  0 0 can_sched:irq_disable 367732 3998500 14773091250
  0 4 can_sched:schedule_entry 40282 398750 45510750
  0 7 cant_sched:irq_enable 367732 32500 532791750
  0 8 cant_sched:irq_entry 26188 6250 38357250
  0 13 disable_to_switch:irq_enable 4933 12500 12275000
  0 14 disable_to_switch:irq_entry 31 2250 35250
  0 15 disable_to_switch:sched_switch 35373 26750 111615750
  0 18 enable_to_exit:irq_disable 373 3250 649750
  0 19 enable_to_exit:irq_enable 373 20000 1676250
  0 20 enable_to_exit:irq_entry 309 4750 420000
  0 23 enable_to_exit:schedule_exit 40298 3000 37335000
  0 25 in_irq:irq_enable 31 12250 140750
  0 30 scheduling:irq_disable 40337 18000 45449500
  0 37 switching:irq_enable 35373 9250 72471000
  [ cpus 1-2 omitted ]
  3 0 can_sched:irq_disable 567 3983750 300860250
  3 4 can_sched:schedule_entry 150 1500 171000
  3 7 cant_sched:irq_enable 567 21500 1777500
  3 8 cant_sched:irq_entry 76 6500 164750
  3 15 disable_to_switch:sched_switch 150 4250 413750
  3 23 enable_to_exit:schedule_exit 150 1500 147750
  3 30 scheduling:irq_disable 150 1500 166000
  3 37 switching:irq_enable 150 3750 277000

The sts model splits the schedule->switch path into separate states, so the
per-edge dwell decomposes the scheduler's interrupts-off window: on cpu0 the
prep phase (disable_to_switch, irqs off until the switch) tops out at 26.8us
and the switch itself (switching:irq_enable) at 9.3us, over 35k switches. The
counts also line up with the model, e.g. every irq_disable in can_sched has a
matching irq_enable in cant_sched (367732 == 367732). The isolated cpus 2-3
take almost no switches (150 on cpu3, versus 35k on cpu0) and their worst case
is tighter still (~4us), so the same file also makes the effect of cpu
isolation visible per cpu.

The changes are gated by CONFIG_RV_EDGE_STAT and dormant until a monitor is
enabled, so existing setups are unaffected. When enabled, each accepted
transition adds one local_clock() and a few lock-free local64_t updates. With
the config off there is no code on the hot path at all.

I focused on per-cpu monitors as a first step. Per-task and per-object
monitors would have to aggregate entities that share a cpu, which is harder
to get right, so I left them out for now.

What is your opinion on this? Do you think this is worth getting upstreamed?

Thanks for taking a look,
Tobias

Tobias Schaffner (3):
  rv: add per-edge dwell-time statistics primitive
  rv: add per-monitor edge-stat facility and stats file
  rv: collect per-edge dwell time for per-cpu DA/HA monitors

 .../trace/rv/runtime-verification.rst         |  24 ++++
 MAINTAINERS                                   |   1 +
 include/linux/rv.h                            |  18 +++
 include/linux/rv_edge_stat.h                  |  60 +++++++++
 include/rv/da_monitor.h                       |  53 ++++++++
 kernel/trace/rv/Kconfig                       |  11 ++
 kernel/trace/rv/rv.c                          | 116 +++++++++++++++++-
 tools/testing/selftests/verification/config   |   1 +
 .../verification/test.d/rv_edge_stats.tc      |  32 +++++
 9 files changed, 315 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/rv_edge_stat.h
 create mode 100644 tools/testing/selftests/verification/test.d/rv_edge_stats.tc

-- 
2.43.0

[RFC 1/3] rv: add per-edge dwell-time statistics primitive

[Tobias Schaffner]

Add a small primitive that records, per automaton edge, how long the
monitor dwelled before taking it with a count, a sum and a maximum.

The counters are per-CPU and lock-free, so a monitor's hot path can update
them without disabling interrupts and without perturbing the very latency
being measured.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 MAINTAINERS                  |  1 +
 include/linux/rv_edge_stat.h | 45 ++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
 create mode 100644 include/linux/rv_edge_stat.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 8014b9f8253e..04589dda0873 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -23654,6 +23654,7 @@ L:	linux-trace-kernel@vger.kernel.org
 S:	Maintained
 F:	Documentation/trace/rv/
 F:	include/linux/rv.h
+F:	include/linux/rv_edge_stat.h
 F:	include/rv/
 F:	kernel/trace/rv/
 F:	tools/testing/selftests/verification/
diff --git a/include/linux/rv_edge_stat.h b/include/linux/rv_edge_stat.h
new file mode 100644
index 000000000000..751de8074dcc
--- /dev/null
+++ b/include/linux/rv_edge_stat.h
@@ -0,0 +1,45 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Per-edge dwell-time statistics for RV monitors.
+ *
+ * Copyright (C) 2026 Siemens AG
+ * Author: Tobias Schaffner <tobias.schaffner@siemens.com>
+ */
+#ifndef _LINUX_RV_EDGE_STAT_H
+#define _LINUX_RV_EDGE_STAT_H
+
+#include <linux/compiler.h>
+#include <linux/types.h>
+#include <asm/local64.h>
+
+/*
+ * Per-CPU counters kept in local64_t so accounting is safe against interrupt
+ * and NMI nesting on the owning CPU without disabling interrupts -- the same
+ * approach the trace ring buffer uses. Only the owning CPU writes.
+ */
+struct rv_edge_stat {
+	local64_t	count;
+	local64_t	sum_ns;
+	local64_t	max_ns;
+};
+
+static __always_inline
+void rv_edge_stat_account(struct rv_edge_stat *s, u64 dwell_ns)
+{
+	s64 max;
+
+	local64_inc(&s->count);
+	local64_add(dwell_ns, &s->sum_ns);
+
+	/* Keep the largest dwell; retry only if a nested update raced us. */
+	max = local64_read(&s->max_ns);
+	while (dwell_ns > (u64)max) {
+		s64 prev = local64_cmpxchg(&s->max_ns, max, dwell_ns);
+
+		if (prev == max)
+			break;
+		max = prev;
+	}
+}
+
+#endif /* _LINUX_RV_EDGE_STAT_H */
-- 
2.43.0

[RFC 2/3] rv: add per-monitor edge-stat facility and stats file

[Tobias Schaffner]

Add CONFIG_RV_EDGE_STAT, an optional feature that records how long a
monitor's automaton dwells in a state and exposes it per edge through a
per-monitor "stats" tracefs file.

The core allocates a per-CPU buffer on first enable and reports, per edge
and per CPU, the count, maximum and summed dwell time. Only the owning CPU
writes the counters, so a reader snapshots them with local64_read() with no
IPI and no locking on the accounting path.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 .../trace/rv/runtime-verification.rst         |  24 ++++
 include/linux/rv.h                            |  14 +++
 include/linux/rv_edge_stat.h                  |  25 +++-
 kernel/trace/rv/Kconfig                       |  11 ++
 kernel/trace/rv/rv.c                          | 116 +++++++++++++++++-
 5 files changed, 184 insertions(+), 6 deletions(-)

diff --git a/Documentation/trace/rv/runtime-verification.rst b/Documentation/trace/rv/runtime-verification.rst
index c700dde9259c..194b2f9db461 100644
--- a/Documentation/trace/rv/runtime-verification.rst
+++ b/Documentation/trace/rv/runtime-verification.rst
@@ -229,3 +229,27 @@ For example::
    nop
    [panic]
    printk
+
+**monitors/MONITOR/stats**
+
+Present only when the kernel is built with CONFIG_RV_EDGE_STAT=y and *MONITOR*
+is a per-cpu DA/HA (automaton) monitor. It reports how long the automaton
+dwells in each state before leaving it, timed with local_clock() and accounted
+per outgoing edge and per CPU.
+
+- The first line is a header naming the columns.
+- Each following line describes one edge on one CPU::
+
+   cpu edge label count max_ns sum_ns
+
+  *count* is the number of times the edge was taken, *max_ns* and *sum_ns* are
+  the worst and total dwell in nanoseconds, and *label* is "state:event".
+
+The counters are reset each time the monitor is enabled.
+
+For example::
+
+   # cat monitors/wip/stats
+   # cpu edge label count max_ns sum_ns
+   0 0 preemptive:preempt_disable 4210 183200 95501200
+   0 4 non_preemptive:preempt_enable 4208 42600 3812900
diff --git a/include/linux/rv.h b/include/linux/rv.h
index 541ba404926a..7eeecce17e50 100644
--- a/include/linux/rv.h
+++ b/include/linux/rv.h
@@ -136,6 +136,16 @@ struct rv_reactor {
 };
 #endif
 
+/**
+ * struct rv_edge_cfg - per-edge dwell-time statistics for a monitor
+ * @n_edges:	number of automaton edges (STATE_MAX * EVENT_MAX)
+ * @edge_name:	optional, write a human name for @edge into @buf (may be NULL)
+ */
+struct rv_edge_cfg {
+	unsigned int	n_edges;
+	void		(*edge_name)(unsigned int edge, char *buf, size_t len);
+};
+
 struct rv_monitor {
 	const char		*name;
 	const char		*description;
@@ -146,6 +156,10 @@ struct rv_monitor {
 #ifdef CONFIG_RV_REACTORS
 	struct rv_reactor	*reactor;
 	__printf(1, 0) void	(*react)(const char *msg, va_list args);
+#endif
+#ifdef CONFIG_RV_EDGE_STAT
+	const struct rv_edge_cfg	*edge_cfg;
+	void __percpu			*edge_pcpu;
 #endif
 	struct list_head	list;
 	struct rv_monitor	*parent;
diff --git a/include/linux/rv_edge_stat.h b/include/linux/rv_edge_stat.h
index 751de8074dcc..fda30ff728a1 100644
--- a/include/linux/rv_edge_stat.h
+++ b/include/linux/rv_edge_stat.h
@@ -9,14 +9,11 @@
 #define _LINUX_RV_EDGE_STAT_H
 
 #include <linux/compiler.h>
+#include <linux/percpu.h>
+#include <linux/rv.h>
 #include <linux/types.h>
 #include <asm/local64.h>
 
-/*
- * Per-CPU counters kept in local64_t so accounting is safe against interrupt
- * and NMI nesting on the owning CPU without disabling interrupts -- the same
- * approach the trace ring buffer uses. Only the owning CPU writes.
- */
 struct rv_edge_stat {
 	local64_t	count;
 	local64_t	sum_ns;
@@ -42,4 +39,22 @@ void rv_edge_stat_account(struct rv_edge_stat *s, u64 dwell_ns)
 	}
 }
 
+#ifdef CONFIG_RV_EDGE_STAT
+/**
+ * rv_edge_account - record a dwell of @dwell_ns on @edge of monitor @mon
+ *
+ * Cheap and lock-free: the local64_t counters make this safe against interrupt
+ * and NMI nesting on the current CPU without disabling interrupts, so it does
+ * not perturb the latency being measured. The caller only needs to stay on its
+ * CPU for the call (as tracepoint probes already do).
+ */
+static __always_inline void
+rv_edge_account(struct rv_monitor *mon, unsigned int edge, u64 dwell_ns)
+{
+	struct rv_edge_stat *e = this_cpu_ptr(mon->edge_pcpu);
+
+	rv_edge_stat_account(&e[edge], dwell_ns);
+}
+#endif /* CONFIG_RV_EDGE_STAT */
+
 #endif /* _LINUX_RV_EDGE_STAT_H */
diff --git a/kernel/trace/rv/Kconfig b/kernel/trace/rv/Kconfig
index 3884b14df375..9d76dff394ca 100644
--- a/kernel/trace/rv/Kconfig
+++ b/kernel/trace/rv/Kconfig
@@ -59,6 +59,17 @@ config RV_PER_TASK_MONITORS
 	  This option configures the maximum number of per-task RV monitors that can run
 	  simultaneously.
 
+config RV_EDGE_STAT
+	bool "Per-edge dwell-time statistics"
+	depends on RV
+	help
+	  Record per-edge dwell-time statistics for per-cpu DA/HA monitors and
+	  expose them through a per-monitor "stats" tracefs file. This times
+	  each monitored automaton transition with local_clock(), so leave it
+	  off if you do not need the statistics.
+
+	  If unsure, say N.
+
 source "kernel/trace/rv/monitors/wip/Kconfig"
 source "kernel/trace/rv/monitors/wwnr/Kconfig"
 
diff --git a/kernel/trace/rv/rv.c b/kernel/trace/rv/rv.c
index ee4e68102f17..88a0bbaec4d0 100644
--- a/kernel/trace/rv/rv.c
+++ b/kernel/trace/rv/rv.c
@@ -142,6 +142,12 @@
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/slab.h>
+#include <linux/seq_file.h>
+#ifdef CONFIG_RV_EDGE_STAT
+#include <linux/percpu.h>
+#include <linux/rv_edge_stat.h>
+#include <linux/smp.h>
+#endif
 
 #ifdef CONFIG_RV_MON_EVENTS
 #define CREATE_TRACE_POINTS
@@ -278,6 +284,9 @@ static void rv_disable_single(struct rv_monitor *mon)
 	__rv_disable_monitor(mon, true);
 }
 
+static int rv_edge_setup(struct rv_monitor *mon);
+static void rv_edge_reset(struct rv_monitor *mon);
+
 static int rv_enable_single(struct rv_monitor *mon)
 {
 	int retval;
@@ -289,9 +298,15 @@ static int rv_enable_single(struct rv_monitor *mon)
 
 	retval = mon->enable();
 
-	if (!retval)
+	if (!retval) {
 		mon->enabled = 1;
 
+		if (rv_edge_setup(mon))
+			pr_warn("rv: %s: edge statistics unavailable (out of memory)\n",
+				mon->name);
+		rv_edge_reset(mon);
+	}
+
 	return retval;
 }
 
@@ -412,6 +427,101 @@ static const struct file_operations interface_desc_fops = {
 	.read	= monitor_desc_read_data,
 };
 
+#ifdef CONFIG_RV_EDGE_STAT
+static size_t rv_edge_blob_size(const struct rv_monitor *mon)
+{
+	return mon->edge_cfg->n_edges * sizeof(struct rv_edge_stat);
+}
+
+static void rv_edge_reset_ipi(void *info)
+{
+	struct rv_monitor *mon = info;
+
+	memset(this_cpu_ptr(mon->edge_pcpu), 0, rv_edge_blob_size(mon));
+}
+
+/* rv_edge_reset - zero the statistics; call from a monitor reset/enable. */
+static void rv_edge_reset(struct rv_monitor *mon)
+{
+	if (mon->edge_pcpu)
+		on_each_cpu(rv_edge_reset_ipi, mon, 1);
+}
+
+/*
+ * The counters are per-CPU and only the owning CPU writes them, so a reader on
+ * any CPU can snapshot them with local64_read().
+ */
+static int rv_edge_stats_show(struct seq_file *seq, void *v)
+{
+	struct rv_monitor *mon = seq->private;
+	const struct rv_edge_cfg *cfg = mon->edge_cfg;
+	unsigned int e;
+	int cpu;
+
+	seq_puts(seq, "# cpu edge label count max_ns sum_ns\n");
+
+	if (!mon->edge_pcpu)
+		return 0;
+
+	for_each_online_cpu(cpu) {
+		struct rv_edge_stat *s = per_cpu_ptr(mon->edge_pcpu, cpu);
+
+		for (e = 0; e < cfg->n_edges; e++) {
+			char lbl[48] = "";
+
+			if (cfg->edge_name)
+				cfg->edge_name(e, lbl, sizeof(lbl));
+			seq_printf(seq, "%d %u %s %llu %llu %llu\n",
+				   cpu, e, lbl,
+				   (u64)local64_read(&s[e].count),
+				   (u64)local64_read(&s[e].max_ns),
+				   (u64)local64_read(&s[e].sum_ns));
+		}
+	}
+	return 0;
+}
+
+static int rv_edge_stats_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, rv_edge_stats_show, inode->i_private);
+}
+
+static const struct file_operations rv_edge_stats_fops = {
+	.open		= rv_edge_stats_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
+/*
+ * Allocate the per-CPU buffer and expose stats. Done on first enable
+ * rather than at registration because a DA/HA monitor's edge_cfg is bound by
+ * da_monitor_init(), which runs from the monitor's enable path.
+ */
+static int rv_edge_setup(struct rv_monitor *mon)
+{
+	if (!mon->edge_cfg || !mon->edge_cfg->n_edges || mon->edge_pcpu)
+		return 0;
+
+	mon->edge_pcpu = __alloc_percpu(rv_edge_blob_size(mon),
+					__alignof__(struct rv_edge_stat));
+	if (!mon->edge_pcpu)
+		return -ENOMEM;
+
+	if (!rv_create_file("stats", RV_MODE_READ, mon->root_d, mon,
+			    &rv_edge_stats_fops)) {
+		free_percpu(mon->edge_pcpu);
+		mon->edge_pcpu = NULL;
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+#else
+static int rv_edge_setup(struct rv_monitor *mon) { return 0; }
+static void rv_edge_reset(struct rv_monitor *mon) { }
+#endif /* CONFIG_RV_EDGE_STAT */
+
 /*
  * During the registration of a monitor, this function creates
  * the monitor dir, where the specific options of the monitor
@@ -747,6 +857,10 @@ static const struct file_operations monitoring_on_fops = {
 
 static void destroy_monitor_dir(struct rv_monitor *mon)
 {
+#ifdef CONFIG_RV_EDGE_STAT
+	free_percpu(mon->edge_pcpu);
+	mon->edge_pcpu = NULL;
+#endif
 	rv_remove(mon->root_d);
 }
 
-- 
2.43.0

[RFC 3/3] rv: collect per-edge dwell time for per-cpu DA/HA monitors

[Tobias Schaffner]

With the core facility in place, hook it into the DA/HA layer so that any
per-cpu automaton monitor also reports how long it lingers in each state.

Add a selftest that enables a per-cpu monitor, checks the stats file
appears and is populated under load, and skips cleanly otherwise.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 include/linux/rv.h                            |  4 ++
 include/rv/da_monitor.h                       | 53 +++++++++++++++++++
 tools/testing/selftests/verification/config   |  1 +
 .../verification/test.d/rv_edge_stats.tc      | 32 +++++++++++
 4 files changed, 90 insertions(+)
 create mode 100644 tools/testing/selftests/verification/test.d/rv_edge_stats.tc

diff --git a/include/linux/rv.h b/include/linux/rv.h
index 7eeecce17e50..09363f79ca90 100644
--- a/include/linux/rv.h
+++ b/include/linux/rv.h
@@ -27,6 +27,10 @@
 struct da_monitor {
 	bool		monitoring;
 	unsigned int	curr_state;
+#ifdef CONFIG_RV_EDGE_STAT
+	/* local_clock() ns when curr_state was entered; 0 = not yet stamped. */
+	u64		state_ns;
+#endif
 };
 
 #ifdef CONFIG_RV_LTL_MONITOR
diff --git a/include/rv/da_monitor.h b/include/rv/da_monitor.h
index 34b8fba9ecd4..59ca9a286c10 100644
--- a/include/rv/da_monitor.h
+++ b/include/rv/da_monitor.h
@@ -16,6 +16,8 @@
 
 #include <rv/automata.h>
 #include <linux/rv.h>
+#include <linux/rv_edge_stat.h>
+#include <linux/sched/clock.h>
 #include <linux/stringify.h>
 #include <linux/bug.h>
 #include <linux/sched.h>
@@ -30,6 +32,54 @@
 
 static struct rv_monitor rv_this;
 
+/* per-edge dwell statistics, wired up for per-cpu monitors. */
+#if defined(CONFIG_RV_EDGE_STAT) && RV_MON_TYPE == RV_MON_PER_CPU
+static void
+rv_this_edge_name(unsigned int edge, char *buf, size_t len)
+{
+	snprintf(buf, len, "%s:%s", model_get_state_name(edge / EVENT_MAX),
+		 model_get_event_name(edge % EVENT_MAX));
+}
+
+static const struct rv_edge_cfg rv_this_edge_cfg = {
+	.n_edges   = STATE_MAX * EVENT_MAX,
+	.edge_name = rv_this_edge_name,
+};
+
+/* Hand the model's edge descriptor to the core; called from da_monitor_init(). */
+static inline void rv_edge_bind(void)
+{
+	rv_this.edge_cfg = &rv_this_edge_cfg;
+}
+
+/* Stamp the moment a state is entered, so its dwell can be timed on exit. */
+static __always_inline void rv_da_edge_enter(struct da_monitor *da_mon)
+{
+	da_mon->state_ns = local_clock();
+}
+
+/* Account the dwell in @curr, then stamp entry into the next state. */
+static __always_inline void
+rv_da_edge_account(struct da_monitor *da_mon, enum states curr, enum events ev)
+{
+	u64 now = local_clock();
+	u64 prev = da_mon->state_ns;
+
+	da_mon->state_ns = now;
+	/*
+	 * local_clock() is not guaranteed monotonic; drop the sample if it did
+	 * not advance so a backward step cannot underflow into a bogus dwell.
+	 */
+	if (rv_this.edge_pcpu && prev && now > prev)
+		rv_edge_account(&rv_this, curr * EVENT_MAX + ev, now - prev);
+}
+#else
+static inline void rv_edge_bind(void) { }
+static inline void rv_da_edge_enter(struct da_monitor *da_mon) { }
+static inline void
+rv_da_edge_account(struct da_monitor *da_mon, enum states curr, enum events ev) { }
+#endif /* CONFIG_RV_EDGE_STAT && RV_MON_PER_CPU */
+
 /*
  * Hook to allow the implementation of hybrid automata: define it with a
  * function that takes curr_state, event and next_state and returns true if the
@@ -113,6 +163,7 @@ static inline void da_monitor_reset(struct da_monitor *da_mon)
 static inline void da_monitor_start(struct da_monitor *da_mon)
 {
 	da_mon->curr_state = model_get_initial_state();
+	rv_da_edge_enter(da_mon);
 	da_monitor_init_hook(da_mon);
 	/* Pairs with smp_load_acquire in da_monitoring(). */
 	smp_store_release(&da_mon->monitoring, 1);
@@ -275,6 +326,7 @@ static inline void da_monitor_reset_state_all(void)
  */
 static inline int da_monitor_init(void)
 {
+	rv_edge_bind();
 	da_monitor_reset_state_all();
 	return 0;
 }
@@ -696,6 +748,7 @@ static inline bool da_event(struct da_monitor *da_mon, enum events event, da_id_
 		if (likely(try_cmpxchg(&da_mon->curr_state, &curr_state, next_state))) {
 			if (!da_monitor_event_hook(da_mon, curr_state, event, next_state, id))
 				return false;
+			rv_da_edge_account(da_mon, curr_state, event);
 			da_trace_event(da_mon, model_get_state_name(curr_state),
 				       model_get_event_name(event),
 				       model_get_state_name(next_state),
diff --git a/tools/testing/selftests/verification/config b/tools/testing/selftests/verification/config
index 43072c1c38f4..de229777d5c8 100644
--- a/tools/testing/selftests/verification/config
+++ b/tools/testing/selftests/verification/config
@@ -1 +1,2 @@
 CONFIG_RV=y
+CONFIG_RV_EDGE_STAT=y
diff --git a/tools/testing/selftests/verification/test.d/rv_edge_stats.tc b/tools/testing/selftests/verification/test.d/rv_edge_stats.tc
new file mode 100644
index 000000000000..5fb2384d938c
--- /dev/null
+++ b/tools/testing/selftests/verification/test.d/rv_edge_stats.tc
@@ -0,0 +1,32 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# description: Test per-edge dwell-time statistics (stats)
+
+MON=wip
+
+check_requires "$MON:monitor"
+
+# The "stats" file is created on first enable; if it is still missing the
+# kernel was built without CONFIG_RV_EDGE_STAT, so skip.
+echo 1 > "monitors/$MON/enable"
+if [ ! -e "monitors/$MON/stats" ]; then
+	echo 0 > "monitors/$MON/enable"
+	echo "CONFIG_RV_EDGE_STAT is not enabled."
+	exit_unsupported
+fi
+
+# The first line is the column header; the body has one line per (cpu, edge).
+head -n1 "monitors/$MON/stats" | grep -q "^# cpu edge label count max_ns sum_ns"
+[ "$(grep -cvE '^#' "monitors/$MON/stats")" -gt 0 ]
+
+# Drive some scheduler activity so the automaton records transitions.
+for _ in 1 2 3 4 5 6 7 8 9 10; do
+	(true) &
+	wait
+done
+
+# At least one edge must now show a non-zero count, with sum_ns >= max_ns > 0.
+grep -vE '^#' "monitors/$MON/stats" | \
+	awk '$4 > 0 && $5 > 0 && $6 >= $5 { hit = 1 } END { exit !hit }'
+
+echo 0 > "monitors/$MON/enable"
-- 
2.43.0

Re: [RFC 0/3] rv: per-edge dwell-time statistics for per-cpu monitors

[Gabriele Monaco]

On Thu, 2026-08-27 at 09:23 +0200, Tobias Schaffner wrote:
> Hi Steven, Gabriele,
> 
> I am using the RV system for a downstream project called EVL (aka Xenomai4).
> It is a small co-kernel system, small enough that I can model the whole
> co-kernel with RV instead of just a fragment of it.

Hi Tobias,

Haven't heard about Xenomai in a while.. That sounds very interesting.

Since it's something out-of-tree you likely don't want to submit monitors
upstream, but if you can trigger BPF events from your co-kernel you may want to
have a look at RV BPF monitors, which I'm likely going to submit in a couple of
days.

Otherwise we can always get out-of-tree monitors as modules, currently not
supported just out of a missing use-case.

> The patches in this series add a per-monitor "stats" file that exposes how
> often an edge has been visited and the dwell time in the prior state for
> per-cpu monitors.

The idea crossed my mind a few times, good you came up with an implementation!

> This allows me to not only check if the model is correct but also to see if
> all paths are exercised by my stress tooling and to decompose the latency of
> the co-kernel's wakeup path.

Your first need (see if all paths are exercised) doesn't really require the stat
to be per-monitor-instance (i.e. per-cpu) right?
You could have a global accumulator matrix, at least for count and sum where you
can be atomic. This could extend the feature to other types of monitors.

Just brainstorming here though.

> I think that some of the upstream monitors like e.g. the sts monitor could
> also
> profit from this.
> 
> As an example, here is the sts monitor on a StarFive VisionFive2 running an rt
> kernel (isolcpus=2-3), after enabling it and generating some scheduler load.
> Only the edges the automaton actually takes are non-zero:
> 
>   # cat monitors/sched/sts/stats
>   # cpu edge label count max_ns sum_ns
>   0 0 can_sched:irq_disable 367732 3998500 14773091250
>   0 4 can_sched:schedule_entry 40282 398750 45510750
>   0 7 cant_sched:irq_enable 367732 32500 532791750
>   0 8 cant_sched:irq_entry 26188 6250 38357250
>   0 13 disable_to_switch:irq_enable 4933 12500 12275000
>   0 14 disable_to_switch:irq_entry 31 2250 35250
>   0 15 disable_to_switch:sched_switch 35373 26750 111615750
>   0 18 enable_to_exit:irq_disable 373 3250 649750
>   0 19 enable_to_exit:irq_enable 373 20000 1676250
>   0 20 enable_to_exit:irq_entry 309 4750 420000
>   0 23 enable_to_exit:schedule_exit 40298 3000 37335000
>   0 25 in_irq:irq_enable 31 12250 140750
>   0 30 scheduling:irq_disable 40337 18000 45449500
>   0 37 switching:irq_enable 35373 9250 72471000
>   [ cpus 1-2 omitted ]
>   3 0 can_sched:irq_disable 567 3983750 300860250
>   3 4 can_sched:schedule_entry 150 1500 171000
>   3 7 cant_sched:irq_enable 567 21500 1777500
>   3 8 cant_sched:irq_entry 76 6500 164750
>   3 15 disable_to_switch:sched_switch 150 4250 413750
>   3 23 enable_to_exit:schedule_exit 150 1500 147750
>   3 30 scheduling:irq_disable 150 1500 166000
>   3 37 switching:irq_enable 150 3750 277000
> 
> The sts model splits the schedule->switch path into separate states, so the
> per-edge dwell decomposes the scheduler's interrupts-off window: on cpu0 the
> prep phase (disable_to_switch, irqs off until the switch) tops out at 26.8us
> and the switch itself (switching:irq_enable) at 9.3us, over 35k switches. The
> counts also line up with the model, e.g. every irq_disable in can_sched has a
> matching irq_enable in cant_sched (367732 == 367732). The isolated cpus 2-3
> take almost no switches (150 on cpu3, versus 35k on cpu0) and their worst case
> is tighter still (~4us), so the same file also makes the effect of cpu
> isolation visible per cpu.
> 
> The changes are gated by CONFIG_RV_EDGE_STAT and dormant until a monitor is
> enabled, so existing setups are unaffected. When enabled, each accepted
> transition adds one local_clock() and a few lock-free local64_t updates. With
> the config off there is no code on the hot path at all.
> 
> I focused on per-cpu monitors as a first step. Per-task and per-object
> monitors would have to aggregate entities that share a cpu, which is harder
> to get right, so I left them out for now.

As said before, I haven't really played with an implementation but I think
atomic types may help here.

> What is your opinion on this? Do you think this is worth getting upstreamed?

I will have a look at your patches but it's definitely something I'd want.

Thanks,
Gabriele

> 
> Thanks for taking a look,
> Tobias
> 
> Tobias Schaffner (3):
>   rv: add per-edge dwell-time statistics primitive
>   rv: add per-monitor edge-stat facility and stats file
>   rv: collect per-edge dwell time for per-cpu DA/HA monitors
> 
>  .../trace/rv/runtime-verification.rst         |  24 ++++
>  MAINTAINERS                                   |   1 +
>  include/linux/rv.h                            |  18 +++
>  include/linux/rv_edge_stat.h                  |  60 +++++++++
>  include/rv/da_monitor.h                       |  53 ++++++++
>  kernel/trace/rv/Kconfig                       |  11 ++
>  kernel/trace/rv/rv.c                          | 116 +++++++++++++++++-
>  tools/testing/selftests/verification/config   |   1 +
>  .../verification/test.d/rv_edge_stats.tc      |  32 +++++
>  9 files changed, 315 insertions(+), 1 deletion(-)
>  create mode 100644 include/linux/rv_edge_stat.h
>  create mode 100644
> tools/testing/selftests/verification/test.d/rv_edge_stats.tc

Re: [RFC 1/3] rv: add per-edge dwell-time statistics primitive

[Gabriele Monaco]

On Thu, 2026-08-27 at 09:23 +0200, Tobias Schaffner wrote:
> Add a small primitive that records, per automaton edge, how long the
> monitor dwelled before taking it with a count, a sum and a maximum.
> 
> The counters are per-CPU and lock-free, so a monitor's hot path can update
> them without disabling interrupts and without perturbing the very latency
> being measured.
> 
> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> ---
>  MAINTAINERS                  |  1 +
>  include/linux/rv_edge_stat.h | 45 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 46 insertions(+)
>  create mode 100644 include/linux/rv_edge_stat.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 8014b9f8253e..04589dda0873 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -23654,6 +23654,7 @@ L:	linux-trace-kernel@vger.kernel.org
>  S:	Maintained
>  F:	Documentation/trace/rv/
>  F:	include/linux/rv.h
> +F:	include/linux/rv_edge_stat.h
>  F:	include/rv/

Any reason why you don't use include/rv/edge_stat.h ?

You wouldn't need to touch the maintainers file and could just

  #include <rv/edge_stat.h>

Thanks,
Gabriele

Re: [RFC 1/3] rv: add per-edge dwell-time statistics primitive

[Tobias Schaffner]

On 8/27/26 10:22, Gabriele Monaco wrote:
> On Thu, 2026-08-27 at 09:23 +0200, Tobias Schaffner wrote:
>> Add a small primitive that records, per automaton edge, how long the
>> monitor dwelled before taking it with a count, a sum and a maximum.
>>
>> The counters are per-CPU and lock-free, so a monitor's hot path can update
>> them without disabling interrupts and without perturbing the very latency
>> being measured.
>>
>> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
>> ---
>>   MAINTAINERS                  |  1 +
>>   include/linux/rv_edge_stat.h | 45 ++++++++++++++++++++++++++++++++++++
>>   2 files changed, 46 insertions(+)
>>   create mode 100644 include/linux/rv_edge_stat.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 8014b9f8253e..04589dda0873 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -23654,6 +23654,7 @@ L:	linux-trace-kernel@vger.kernel.org
>>   S:	Maintained
>>   F:	Documentation/trace/rv/
>>   F:	include/linux/rv.h
>> +F:	include/linux/rv_edge_stat.h
>>   F:	include/rv/
> 
> Any reason why you don't use include/rv/edge_stat.h ?

No good reason. You're right. Happy to move it in a V2.

Best,
Tobias

> You wouldn't need to touch the maintainers file and could just
> 
>    #include <rv/edge_stat.h>
> 
> Thanks,
> Gabriele
> 

Re: [RFC 0/3] rv: per-edge dwell-time statistics for per-cpu monitors

[Tobias Schaffner]

Hi Gabriele,

thanks for the quick reply and the motivating words.

On 8/27/26 10:18, Gabriele Monaco wrote:
> On Thu, 2026-08-27 at 09:23 +0200, Tobias Schaffner wrote:
>> Hi Steven, Gabriele,
>>
>> I am using the RV system for a downstream project called EVL (aka Xenomai4).
>> It is a small co-kernel system, small enough that I can model the whole
>> co-kernel with RV instead of just a fragment of it.
> 
> Hi Tobias,
> 
> Haven't heard about Xenomai in a while.. That sounds very interesting.
> 
> Since it's something out-of-tree you likely don't want to submit monitors
> upstream, but if you can trigger BPF events from your co-kernel you may want to
> have a look at RV BPF monitors, which I'm likely going to submit in a couple of
> days.

Exactly. The monitors themselves will live downstream in the Dovetail 
and EVL projects.

I will definitely have a look at the RV BPF monitors!

> Otherwise we can always get out-of-tree monitors as modules, currently not
> supported just out of a missing use-case.
> 
>> The patches in this series add a per-monitor "stats" file that exposes how
>> often an edge has been visited and the dwell time in the prior state for
>> per-cpu monitors.
> 
> The idea crossed my mind a few times, good you came up with an implementation!
> 
>> This allows me to not only check if the model is correct but also to see if
>> all paths are exercised by my stress tooling and to decompose the latency of
>> the co-kernel's wakeup path.
> 
> Your first need (see if all paths are exercised) doesn't really require the stat
> to be per-monitor-instance (i.e. per-cpu) right?
> You could have a global accumulator matrix, at least for count and sum where you
> can be atomic. This could extend the feature to other types of monitors.

Yes that's correct.

For the dwell timings, though, keeping it per-cpu matters to make
isolation effects visible. I use sum and count there to derive
the average.

> Just brainstorming here though.
> 
>> I think that some of the upstream monitors like e.g. the sts monitor could
>> also
>> profit from this.
>>
>> As an example, here is the sts monitor on a StarFive VisionFive2 running an rt
>> kernel (isolcpus=2-3), after enabling it and generating some scheduler load.
>> Only the edges the automaton actually takes are non-zero:
>>
>>    # cat monitors/sched/sts/stats
>>    # cpu edge label count max_ns sum_ns
>>    0 0 can_sched:irq_disable 367732 3998500 14773091250
>>    0 4 can_sched:schedule_entry 40282 398750 45510750
>>    0 7 cant_sched:irq_enable 367732 32500 532791750
>>    0 8 cant_sched:irq_entry 26188 6250 38357250
>>    0 13 disable_to_switch:irq_enable 4933 12500 12275000
>>    0 14 disable_to_switch:irq_entry 31 2250 35250
>>    0 15 disable_to_switch:sched_switch 35373 26750 111615750
>>    0 18 enable_to_exit:irq_disable 373 3250 649750
>>    0 19 enable_to_exit:irq_enable 373 20000 1676250
>>    0 20 enable_to_exit:irq_entry 309 4750 420000
>>    0 23 enable_to_exit:schedule_exit 40298 3000 37335000
>>    0 25 in_irq:irq_enable 31 12250 140750
>>    0 30 scheduling:irq_disable 40337 18000 45449500
>>    0 37 switching:irq_enable 35373 9250 72471000
>>    [ cpus 1-2 omitted ]
>>    3 0 can_sched:irq_disable 567 3983750 300860250
>>    3 4 can_sched:schedule_entry 150 1500 171000
>>    3 7 cant_sched:irq_enable 567 21500 1777500
>>    3 8 cant_sched:irq_entry 76 6500 164750
>>    3 15 disable_to_switch:sched_switch 150 4250 413750
>>    3 23 enable_to_exit:schedule_exit 150 1500 147750
>>    3 30 scheduling:irq_disable 150 1500 166000
>>    3 37 switching:irq_enable 150 3750 277000
>>
>> The sts model splits the schedule->switch path into separate states, so the
>> per-edge dwell decomposes the scheduler's interrupts-off window: on cpu0 the
>> prep phase (disable_to_switch, irqs off until the switch) tops out at 26.8us
>> and the switch itself (switching:irq_enable) at 9.3us, over 35k switches. The
>> counts also line up with the model, e.g. every irq_disable in can_sched has a
>> matching irq_enable in cant_sched (367732 == 367732). The isolated cpus 2-3
>> take almost no switches (150 on cpu3, versus 35k on cpu0) and their worst case
>> is tighter still (~4us), so the same file also makes the effect of cpu
>> isolation visible per cpu.
>>
>> The changes are gated by CONFIG_RV_EDGE_STAT and dormant until a monitor is
>> enabled, so existing setups are unaffected. When enabled, each accepted
>> transition adds one local_clock() and a few lock-free local64_t updates. With
>> the config off there is no code on the hot path at all.
>>
>> I focused on per-cpu monitors as a first step. Per-task and per-object
>> monitors would have to aggregate entities that share a cpu, which is harder
>> to get right, so I left them out for now.
> 
> As said before, I haven't really played with an implementation but I think
> atomic types may help here.
> 
>> What is your opinion on this? Do you think this is worth getting upstreamed?
> 
> I will have a look at your patches but it's definitely something I'd want.

Looking forward to your input! I am very open to guidance on how to get
this right.

Best,
Tobias

> Thanks,
> Gabriele
> 
>>
>> Thanks for taking a look,
>> Tobias
>>
>> Tobias Schaffner (3):
>>    rv: add per-edge dwell-time statistics primitive
>>    rv: add per-monitor edge-stat facility and stats file
>>    rv: collect per-edge dwell time for per-cpu DA/HA monitors
>>
>>   .../trace/rv/runtime-verification.rst         |  24 ++++
>>   MAINTAINERS                                   |   1 +
>>   include/linux/rv.h                            |  18 +++
>>   include/linux/rv_edge_stat.h                  |  60 +++++++++
>>   include/rv/da_monitor.h                       |  53 ++++++++
>>   kernel/trace/rv/Kconfig                       |  11 ++
>>   kernel/trace/rv/rv.c                          | 116 +++++++++++++++++-
>>   tools/testing/selftests/verification/config   |   1 +
>>   .../verification/test.d/rv_edge_stats.tc      |  32 +++++
>>   9 files changed, 315 insertions(+), 1 deletion(-)
>>   create mode 100644 include/linux/rv_edge_stat.h
>>   create mode 100644
>> tools/testing/selftests/verification/test.d/rv_edge_stats.tc
>