[RFC 0/3] rv: per-edge dwell-time statistics for per-cpu monitors

[RFC 0/3] rv: per-edge dwell-time statistics for per-cpu monitors

[Tobias Schaffner]

Hi Steven, Gabriele,

I am using the RV system for a downstream project called EVL (aka Xenomai4).
It is a small co-kernel system, small enough that I can model the whole
co-kernel with RV instead of just a fragment of it.

The patches in this series add a per-monitor "stats" file that exposes how
often an edge has been visited and the dwell time in the prior state for
per-cpu monitors.

This allows me to not only check if the model is correct but also to see if
all paths are exercised by my stress tooling and to decompose the latency of
the co-kernel's wakeup path.

I think that some of the upstream monitors like e.g. the sts monitor could also
profit from this.

As an example, here is the sts monitor on a StarFive VisionFive2 running an rt
kernel (isolcpus=2-3), after enabling it and generating some scheduler load.
Only the edges the automaton actually takes are non-zero:

  # cat monitors/sched/sts/stats
  # cpu edge label count max_ns sum_ns
  0 0 can_sched:irq_disable 367732 3998500 14773091250
  0 4 can_sched:schedule_entry 40282 398750 45510750
  0 7 cant_sched:irq_enable 367732 32500 532791750
  0 8 cant_sched:irq_entry 26188 6250 38357250
  0 13 disable_to_switch:irq_enable 4933 12500 12275000
  0 14 disable_to_switch:irq_entry 31 2250 35250
  0 15 disable_to_switch:sched_switch 35373 26750 111615750
  0 18 enable_to_exit:irq_disable 373 3250 649750
  0 19 enable_to_exit:irq_enable 373 20000 1676250
  0 20 enable_to_exit:irq_entry 309 4750 420000
  0 23 enable_to_exit:schedule_exit 40298 3000 37335000
  0 25 in_irq:irq_enable 31 12250 140750
  0 30 scheduling:irq_disable 40337 18000 45449500
  0 37 switching:irq_enable 35373 9250 72471000
  [ cpus 1-2 omitted ]
  3 0 can_sched:irq_disable 567 3983750 300860250
  3 4 can_sched:schedule_entry 150 1500 171000
  3 7 cant_sched:irq_enable 567 21500 1777500
  3 8 cant_sched:irq_entry 76 6500 164750
  3 15 disable_to_switch:sched_switch 150 4250 413750
  3 23 enable_to_exit:schedule_exit 150 1500 147750
  3 30 scheduling:irq_disable 150 1500 166000
  3 37 switching:irq_enable 150 3750 277000

The sts model splits the schedule->switch path into separate states, so the
per-edge dwell decomposes the scheduler's interrupts-off window: on cpu0 the
prep phase (disable_to_switch, irqs off until the switch) tops out at 26.8us
and the switch itself (switching:irq_enable) at 9.3us, over 35k switches. The
counts also line up with the model, e.g. every irq_disable in can_sched has a
matching irq_enable in cant_sched (367732 == 367732). The isolated cpus 2-3
take almost no switches (150 on cpu3, versus 35k on cpu0) and their worst case
is tighter still (~4us), so the same file also makes the effect of cpu
isolation visible per cpu.

The changes are gated by CONFIG_RV_EDGE_STAT and dormant until a monitor is
enabled, so existing setups are unaffected. When enabled, each accepted
transition adds one local_clock() and a few lock-free local64_t updates. With
the config off there is no code on the hot path at all.

I focused on per-cpu monitors as a first step. Per-task and per-object
monitors would have to aggregate entities that share a cpu, which is harder
to get right, so I left them out for now.

What is your opinion on this? Do you think this is worth getting upstreamed?

Thanks for taking a look,
Tobias

Tobias Schaffner (3):
  rv: add per-edge dwell-time statistics primitive
  rv: add per-monitor edge-stat facility and stats file
  rv: collect per-edge dwell time for per-cpu DA/HA monitors

 .../trace/rv/runtime-verification.rst         |  24 ++++
 MAINTAINERS                                   |   1 +
 include/linux/rv.h                            |  18 +++
 include/linux/rv_edge_stat.h                  |  60 +++++++++
 include/rv/da_monitor.h                       |  53 ++++++++
 kernel/trace/rv/Kconfig                       |  11 ++
 kernel/trace/rv/rv.c                          | 116 +++++++++++++++++-
 tools/testing/selftests/verification/config   |   1 +
 .../verification/test.d/rv_edge_stats.tc      |  32 +++++
 9 files changed, 315 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/rv_edge_stat.h
 create mode 100644 tools/testing/selftests/verification/test.d/rv_edge_stats.tc

-- 
2.43.0

[RFC 1/3] rv: add per-edge dwell-time statistics primitive

[Tobias Schaffner]

Add a small primitive that records, per automaton edge, how long the
monitor dwelled before taking it with a count, a sum and a maximum.

The counters are per-CPU and lock-free, so a monitor's hot path can update
them without disabling interrupts and without perturbing the very latency
being measured.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 MAINTAINERS                  |  1 +
 include/linux/rv_edge_stat.h | 45 ++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
 create mode 100644 include/linux/rv_edge_stat.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 8014b9f8253e..04589dda0873 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -23654,6 +23654,7 @@ L:	linux-trace-kernel@vger.kernel.org
 S:	Maintained
 F:	Documentation/trace/rv/
 F:	include/linux/rv.h
+F:	include/linux/rv_edge_stat.h
 F:	include/rv/
 F:	kernel/trace/rv/
 F:	tools/testing/selftests/verification/
diff --git a/include/linux/rv_edge_stat.h b/include/linux/rv_edge_stat.h
new file mode 100644
index 000000000000..751de8074dcc
--- /dev/null
+++ b/include/linux/rv_edge_stat.h
@@ -0,0 +1,45 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Per-edge dwell-time statistics for RV monitors.
+ *
+ * Copyright (C) 2026 Siemens AG
+ * Author: Tobias Schaffner <tobias.schaffner@siemens.com>
+ */
+#ifndef _LINUX_RV_EDGE_STAT_H
+#define _LINUX_RV_EDGE_STAT_H
+
+#include <linux/compiler.h>
+#include <linux/types.h>
+#include <asm/local64.h>
+
+/*
+ * Per-CPU counters kept in local64_t so accounting is safe against interrupt
+ * and NMI nesting on the owning CPU without disabling interrupts -- the same
+ * approach the trace ring buffer uses. Only the owning CPU writes.
+ */
+struct rv_edge_stat {
+	local64_t	count;
+	local64_t	sum_ns;
+	local64_t	max_ns;
+};
+
+static __always_inline
+void rv_edge_stat_account(struct rv_edge_stat *s, u64 dwell_ns)
+{
+	s64 max;
+
+	local64_inc(&s->count);
+	local64_add(dwell_ns, &s->sum_ns);
+
+	/* Keep the largest dwell; retry only if a nested update raced us. */
+	max = local64_read(&s->max_ns);
+	while (dwell_ns > (u64)max) {
+		s64 prev = local64_cmpxchg(&s->max_ns, max, dwell_ns);
+
+		if (prev == max)
+			break;
+		max = prev;
+	}
+}
+
+#endif /* _LINUX_RV_EDGE_STAT_H */
-- 
2.43.0

Re: [RFC 1/3] rv: add per-edge dwell-time statistics primitive

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] NMI Deadlock / Lock Reentrancy on 32-bit architectures
--

commit 29c9d40bef06feb46125ab67ac5762b58ace9c11
Author: Tobias Schaffner <tobias.schaffner@siemens.com>

rv: add per-edge dwell-time statistics primitive

Add a small primitive that records, per automaton edge, how long the
monitor dwelled before taking it with a count, a sum and a maximum.
The counters are kept in local64_t variables so a monitor's hot path
can update them without disabling interrupts.

> diff --git a/include/linux/rv_edge_stat.h b/include/linux/rv_edge_stat.h
> new file mode 100644
> index 0000000000000..751de8074dcc3
> --- /dev/null
> +++ b/include/linux/rv_edge_stat.h

[ ... ]

> +static __always_inline
> +void rv_edge_stat_account(struct rv_edge_stat *s, u64 dwell_ns)
> +{
> +	s64 max;
> +
> +	local64_inc(&s->count);
> +	local64_add(dwell_ns, &s->sum_ns);
> +
> +	/* Keep the largest dwell; retry only if a nested update raced us. */
> +	max = local64_read(&s->max_ns);
> +	while (dwell_ns > (u64)max) {
> +		s64 prev = local64_cmpxchg(&s->max_ns, max, dwell_ns);

[Severity: High]
Could this cause an NMI deadlock on 32-bit architectures?

The commit message notes these counters are lock-free, but on 32-bit platforms
lacking native 64-bit atomics (e.g., ARMv6 or RISC-V 32 with
CONFIG_GENERIC_ATOMIC64=y), local64_t operations fall back to atomic64_t. 

The generic atomic64 implementation uses a hashed array of spinlocks acquired
via local_irq_save(), which disables regular interrupts but not NMIs.

If an NMI interrupts rv_edge_stat_account() while it holds one of these
non-reentrant spinlocks, and the NMI handler triggers an RV monitor event
that hashes to the same lock, would the CPU deadlock attempting to acquire
the already-held spinlock?

Does this implementation need an explicit check to abort if
IS_ENABLED(CONFIG_GENERIC_ATOMIC64) && in_nmi(), similar to the protection
used in the trace ring buffer?

> +
> +		if (prev == max)
> +			break;
> +		max = prev;
> +	}
> +}
> +
> +#endif /* _LINUX_RV_EDGE_STAT_H */

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260827072400.45734-1-tobias.schaffner@siemens.com?part=1

Re: [RFC 1/3] rv: add per-edge dwell-time statistics primitive

[Gabriele Monaco]

On Thu, 2026-08-27 at 09:23 +0200, Tobias Schaffner wrote:
> Add a small primitive that records, per automaton edge, how long the
> monitor dwelled before taking it with a count, a sum and a maximum.
> 
> The counters are per-CPU and lock-free, so a monitor's hot path can update
> them without disabling interrupts and without perturbing the very latency
> being measured.
> 
> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> ---
>  MAINTAINERS                  |  1 +
>  include/linux/rv_edge_stat.h | 45 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 46 insertions(+)
>  create mode 100644 include/linux/rv_edge_stat.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 8014b9f8253e..04589dda0873 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -23654,6 +23654,7 @@ L:	linux-trace-kernel@vger.kernel.org
>  S:	Maintained
>  F:	Documentation/trace/rv/
>  F:	include/linux/rv.h
> +F:	include/linux/rv_edge_stat.h
>  F:	include/rv/

Any reason why you don't use include/rv/edge_stat.h ?

You wouldn't need to touch the maintainers file and could just

  #include <rv/edge_stat.h>

Thanks,
Gabriele

Re: [RFC 1/3] rv: add per-edge dwell-time statistics primitive

[Tobias Schaffner]

On 8/27/26 10:22, Gabriele Monaco wrote:
> On Thu, 2026-08-27 at 09:23 +0200, Tobias Schaffner wrote:
>> Add a small primitive that records, per automaton edge, how long the
>> monitor dwelled before taking it with a count, a sum and a maximum.
>>
>> The counters are per-CPU and lock-free, so a monitor's hot path can update
>> them without disabling interrupts and without perturbing the very latency
>> being measured.
>>
>> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
>> ---
>>   MAINTAINERS                  |  1 +
>>   include/linux/rv_edge_stat.h | 45 ++++++++++++++++++++++++++++++++++++
>>   2 files changed, 46 insertions(+)
>>   create mode 100644 include/linux/rv_edge_stat.h
>>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 8014b9f8253e..04589dda0873 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -23654,6 +23654,7 @@ L:	linux-trace-kernel@vger.kernel.org
>>   S:	Maintained
>>   F:	Documentation/trace/rv/
>>   F:	include/linux/rv.h
>> +F:	include/linux/rv_edge_stat.h
>>   F:	include/rv/
> 
> Any reason why you don't use include/rv/edge_stat.h ?

No good reason. You're right. Happy to move it in a V2.

Best,
Tobias

> You wouldn't need to touch the maintainers file and could just
> 
>    #include <rv/edge_stat.h>
> 
> Thanks,
> Gabriele
> 

[RFC 2/3] rv: add per-monitor edge-stat facility and stats file

[Tobias Schaffner]

Add CONFIG_RV_EDGE_STAT, an optional feature that records how long a
monitor's automaton dwells in a state and exposes it per edge through a
per-monitor "stats" tracefs file.

The core allocates a per-CPU buffer on first enable and reports, per edge
and per CPU, the count, maximum and summed dwell time. Only the owning CPU
writes the counters, so a reader snapshots them with local64_read() with no
IPI and no locking on the accounting path.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 .../trace/rv/runtime-verification.rst         |  24 ++++
 include/linux/rv.h                            |  14 +++
 include/linux/rv_edge_stat.h                  |  25 +++-
 kernel/trace/rv/Kconfig                       |  11 ++
 kernel/trace/rv/rv.c                          | 116 +++++++++++++++++-
 5 files changed, 184 insertions(+), 6 deletions(-)

diff --git a/Documentation/trace/rv/runtime-verification.rst b/Documentation/trace/rv/runtime-verification.rst
index c700dde9259c..194b2f9db461 100644
--- a/Documentation/trace/rv/runtime-verification.rst
+++ b/Documentation/trace/rv/runtime-verification.rst
@@ -229,3 +229,27 @@ For example::
    nop
    [panic]
    printk
+
+**monitors/MONITOR/stats**
+
+Present only when the kernel is built with CONFIG_RV_EDGE_STAT=y and *MONITOR*
+is a per-cpu DA/HA (automaton) monitor. It reports how long the automaton
+dwells in each state before leaving it, timed with local_clock() and accounted
+per outgoing edge and per CPU.
+
+- The first line is a header naming the columns.
+- Each following line describes one edge on one CPU::
+
+   cpu edge label count max_ns sum_ns
+
+  *count* is the number of times the edge was taken, *max_ns* and *sum_ns* are
+  the worst and total dwell in nanoseconds, and *label* is "state:event".
+
+The counters are reset each time the monitor is enabled.
+
+For example::
+
+   # cat monitors/wip/stats
+   # cpu edge label count max_ns sum_ns
+   0 0 preemptive:preempt_disable 4210 183200 95501200
+   0 4 non_preemptive:preempt_enable 4208 42600 3812900
diff --git a/include/linux/rv.h b/include/linux/rv.h
index 541ba404926a..7eeecce17e50 100644
--- a/include/linux/rv.h
+++ b/include/linux/rv.h
@@ -136,6 +136,16 @@ struct rv_reactor {
 };
 #endif
 
+/**
+ * struct rv_edge_cfg - per-edge dwell-time statistics for a monitor
+ * @n_edges:	number of automaton edges (STATE_MAX * EVENT_MAX)
+ * @edge_name:	optional, write a human name for @edge into @buf (may be NULL)
+ */
+struct rv_edge_cfg {
+	unsigned int	n_edges;
+	void		(*edge_name)(unsigned int edge, char *buf, size_t len);
+};
+
 struct rv_monitor {
 	const char		*name;
 	const char		*description;
@@ -146,6 +156,10 @@ struct rv_monitor {
 #ifdef CONFIG_RV_REACTORS
 	struct rv_reactor	*reactor;
 	__printf(1, 0) void	(*react)(const char *msg, va_list args);
+#endif
+#ifdef CONFIG_RV_EDGE_STAT
+	const struct rv_edge_cfg	*edge_cfg;
+	void __percpu			*edge_pcpu;
 #endif
 	struct list_head	list;
 	struct rv_monitor	*parent;
diff --git a/include/linux/rv_edge_stat.h b/include/linux/rv_edge_stat.h
index 751de8074dcc..fda30ff728a1 100644
--- a/include/linux/rv_edge_stat.h
+++ b/include/linux/rv_edge_stat.h
@@ -9,14 +9,11 @@
 #define _LINUX_RV_EDGE_STAT_H
 
 #include <linux/compiler.h>
+#include <linux/percpu.h>
+#include <linux/rv.h>
 #include <linux/types.h>
 #include <asm/local64.h>
 
-/*
- * Per-CPU counters kept in local64_t so accounting is safe against interrupt
- * and NMI nesting on the owning CPU without disabling interrupts -- the same
- * approach the trace ring buffer uses. Only the owning CPU writes.
- */
 struct rv_edge_stat {
 	local64_t	count;
 	local64_t	sum_ns;
@@ -42,4 +39,22 @@ void rv_edge_stat_account(struct rv_edge_stat *s, u64 dwell_ns)
 	}
 }
 
+#ifdef CONFIG_RV_EDGE_STAT
+/**
+ * rv_edge_account - record a dwell of @dwell_ns on @edge of monitor @mon
+ *
+ * Cheap and lock-free: the local64_t counters make this safe against interrupt
+ * and NMI nesting on the current CPU without disabling interrupts, so it does
+ * not perturb the latency being measured. The caller only needs to stay on its
+ * CPU for the call (as tracepoint probes already do).
+ */
+static __always_inline void
+rv_edge_account(struct rv_monitor *mon, unsigned int edge, u64 dwell_ns)
+{
+	struct rv_edge_stat *e = this_cpu_ptr(mon->edge_pcpu);
+
+	rv_edge_stat_account(&e[edge], dwell_ns);
+}
+#endif /* CONFIG_RV_EDGE_STAT */
+
 #endif /* _LINUX_RV_EDGE_STAT_H */
diff --git a/kernel/trace/rv/Kconfig b/kernel/trace/rv/Kconfig
index 3884b14df375..9d76dff394ca 100644
--- a/kernel/trace/rv/Kconfig
+++ b/kernel/trace/rv/Kconfig
@@ -59,6 +59,17 @@ config RV_PER_TASK_MONITORS
 	  This option configures the maximum number of per-task RV monitors that can run
 	  simultaneously.
 
+config RV_EDGE_STAT
+	bool "Per-edge dwell-time statistics"
+	depends on RV
+	help
+	  Record per-edge dwell-time statistics for per-cpu DA/HA monitors and
+	  expose them through a per-monitor "stats" tracefs file. This times
+	  each monitored automaton transition with local_clock(), so leave it
+	  off if you do not need the statistics.
+
+	  If unsure, say N.
+
 source "kernel/trace/rv/monitors/wip/Kconfig"
 source "kernel/trace/rv/monitors/wwnr/Kconfig"
 
diff --git a/kernel/trace/rv/rv.c b/kernel/trace/rv/rv.c
index ee4e68102f17..88a0bbaec4d0 100644
--- a/kernel/trace/rv/rv.c
+++ b/kernel/trace/rv/rv.c
@@ -142,6 +142,12 @@
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/slab.h>
+#include <linux/seq_file.h>
+#ifdef CONFIG_RV_EDGE_STAT
+#include <linux/percpu.h>
+#include <linux/rv_edge_stat.h>
+#include <linux/smp.h>
+#endif
 
 #ifdef CONFIG_RV_MON_EVENTS
 #define CREATE_TRACE_POINTS
@@ -278,6 +284,9 @@ static void rv_disable_single(struct rv_monitor *mon)
 	__rv_disable_monitor(mon, true);
 }
 
+static int rv_edge_setup(struct rv_monitor *mon);
+static void rv_edge_reset(struct rv_monitor *mon);
+
 static int rv_enable_single(struct rv_monitor *mon)
 {
 	int retval;
@@ -289,9 +298,15 @@ static int rv_enable_single(struct rv_monitor *mon)
 
 	retval = mon->enable();
 
-	if (!retval)
+	if (!retval) {
 		mon->enabled = 1;
 
+		if (rv_edge_setup(mon))
+			pr_warn("rv: %s: edge statistics unavailable (out of memory)\n",
+				mon->name);
+		rv_edge_reset(mon);
+	}
+
 	return retval;
 }
 
@@ -412,6 +427,101 @@ static const struct file_operations interface_desc_fops = {
 	.read	= monitor_desc_read_data,
 };
 
+#ifdef CONFIG_RV_EDGE_STAT
+static size_t rv_edge_blob_size(const struct rv_monitor *mon)
+{
+	return mon->edge_cfg->n_edges * sizeof(struct rv_edge_stat);
+}
+
+static void rv_edge_reset_ipi(void *info)
+{
+	struct rv_monitor *mon = info;
+
+	memset(this_cpu_ptr(mon->edge_pcpu), 0, rv_edge_blob_size(mon));
+}
+
+/* rv_edge_reset - zero the statistics; call from a monitor reset/enable. */
+static void rv_edge_reset(struct rv_monitor *mon)
+{
+	if (mon->edge_pcpu)
+		on_each_cpu(rv_edge_reset_ipi, mon, 1);
+}
+
+/*
+ * The counters are per-CPU and only the owning CPU writes them, so a reader on
+ * any CPU can snapshot them with local64_read().
+ */
+static int rv_edge_stats_show(struct seq_file *seq, void *v)
+{
+	struct rv_monitor *mon = seq->private;
+	const struct rv_edge_cfg *cfg = mon->edge_cfg;
+	unsigned int e;
+	int cpu;
+
+	seq_puts(seq, "# cpu edge label count max_ns sum_ns\n");
+
+	if (!mon->edge_pcpu)
+		return 0;
+
+	for_each_online_cpu(cpu) {
+		struct rv_edge_stat *s = per_cpu_ptr(mon->edge_pcpu, cpu);
+
+		for (e = 0; e < cfg->n_edges; e++) {
+			char lbl[48] = "";
+
+			if (cfg->edge_name)
+				cfg->edge_name(e, lbl, sizeof(lbl));
+			seq_printf(seq, "%d %u %s %llu %llu %llu\n",
+				   cpu, e, lbl,
+				   (u64)local64_read(&s[e].count),
+				   (u64)local64_read(&s[e].max_ns),
+				   (u64)local64_read(&s[e].sum_ns));
+		}
+	}
+	return 0;
+}
+
+static int rv_edge_stats_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, rv_edge_stats_show, inode->i_private);
+}
+
+static const struct file_operations rv_edge_stats_fops = {
+	.open		= rv_edge_stats_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
+/*
+ * Allocate the per-CPU buffer and expose stats. Done on first enable
+ * rather than at registration because a DA/HA monitor's edge_cfg is bound by
+ * da_monitor_init(), which runs from the monitor's enable path.
+ */
+static int rv_edge_setup(struct rv_monitor *mon)
+{
+	if (!mon->edge_cfg || !mon->edge_cfg->n_edges || mon->edge_pcpu)
+		return 0;
+
+	mon->edge_pcpu = __alloc_percpu(rv_edge_blob_size(mon),
+					__alignof__(struct rv_edge_stat));
+	if (!mon->edge_pcpu)
+		return -ENOMEM;
+
+	if (!rv_create_file("stats", RV_MODE_READ, mon->root_d, mon,
+			    &rv_edge_stats_fops)) {
+		free_percpu(mon->edge_pcpu);
+		mon->edge_pcpu = NULL;
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+#else
+static int rv_edge_setup(struct rv_monitor *mon) { return 0; }
+static void rv_edge_reset(struct rv_monitor *mon) { }
+#endif /* CONFIG_RV_EDGE_STAT */
+
 /*
  * During the registration of a monitor, this function creates
  * the monitor dir, where the specific options of the monitor
@@ -747,6 +857,10 @@ static const struct file_operations monitoring_on_fops = {
 
 static void destroy_monitor_dir(struct rv_monitor *mon)
 {
+#ifdef CONFIG_RV_EDGE_STAT
+	free_percpu(mon->edge_pcpu);
+	mon->edge_pcpu = NULL;
+#endif
 	rv_remove(mon->root_d);
 }
 
-- 
2.43.0

Re: [RFC 2/3] rv: add per-monitor edge-stat facility and stats file

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] destroy_monitor_dir() frees mon->edge_pcpu before VFS synchronization, allowing concurrent readers to access freed memory via the tracefs stats file.
- [Medium] Non-atomic memset corrupts local64_t counters, racing with concurrent NMI tracepoint updates and lockless tracefs reads.
- [Medium] rv_edge_reset() only resets statistics on currently online CPUs, violating the guarantee that counters are fully cleared on enablement.
--

commit 5d961f88b7dec86f152d8e652fbd68ebc65c3317
Author: Tobias Schaffner <tobias.schaffner@siemens.com>
rv: add per-monitor edge-stat facility and stats file

This commit introduces CONFIG_RV_EDGE_STAT, an optional feature that records
how long a monitor's automaton dwells in a state. It allocates a per-CPU
buffer on first enable and exposes the counters via a per-monitor stats file.

> diff --git a/kernel/trace/rv/rv.c b/kernel/trace/rv/rv.c
> index ee4e68102f175..88a0bbaec4d0a 100644
> --- a/kernel/trace/rv/rv.c
> +++ b/kernel/trace/rv/rv.c

[ ... ]

> @@ -289,9 +298,15 @@ static int rv_enable_single(struct rv_monitor *mon)
>  
>  	retval = mon->enable();
>  
> -	if (!retval)
> +	if (!retval) {
>  		mon->enabled = 1;
>  
> +		if (rv_edge_setup(mon))
> +			pr_warn("rv: %s: edge statistics unavailable (out of memory)\n",
> +				mon->name);
> +		rv_edge_reset(mon);

[Severity: Medium]
Does resetting the statistics with memset() after the monitor is already
enabled risk data corruption?

Since mon->enable() is called before rv_edge_reset(), NMI tracepoint
updates can fire concurrently. Using memset() in rv_edge_reset_ipi() to
clear the array of local64_t counters could race with these
read-modify-write updates, potentially resulting in torn reads or
corrupted atomic statistics when rv_edge_stats_show() accesses them.

> +	}
> +
>  	return retval;
>  }

[ ... ]

> +static void rv_edge_reset_ipi(void *info)
> +{
> +	struct rv_monitor *mon = info;
> +
> +	memset(this_cpu_ptr(mon->edge_pcpu), 0, rv_edge_blob_size(mon));
> +}
> +
> +/* rv_edge_reset - zero the statistics; call from a monitor reset/enable. */
> +static void rv_edge_reset(struct rv_monitor *mon)
> +{
> +	if (mon->edge_pcpu)
> +		on_each_cpu(rv_edge_reset_ipi, mon, 1);
> +}

[Severity: Medium]
Will this skip clearing the statistics on offline CPUs?

Because on_each_cpu() only targets online CPUs, dynamically allocated
per-CPU edge statistics buffers for offline CPUs remain uncleared. If
an offline CPU is later brought online, reading the tracefs stats file
will iterate over it via for_each_online_cpu() and report stale,
un-reset values from previous runs.

[ ... ]

> @@ -747,6 +857,10 @@ static const struct file_operations monitoring_on_fops = {
>  
>  static void destroy_monitor_dir(struct rv_monitor *mon)
>  {
> +#ifdef CONFIG_RV_EDGE_STAT
> +	free_percpu(mon->edge_pcpu);
> +	mon->edge_pcpu = NULL;
> +#endif
>  	rv_remove(mon->root_d);
>  }

[Severity: High]
Could this lead to a use-after-free if the tracefs stats file is read
while the monitor is being unregistered?

The per-cpu buffer is explicitly freed here before the rv_remove()
synchronization point removes the tracefs file. A concurrent lockless
reader using rv_edge_stats_show() could access the freed mon->edge_pcpu
memory:

kernel/trace/rv/rv.c:rv_edge_stats_show() {
    ...
    if (!mon->edge_pcpu)
        return 0;

    for_each_online_cpu(cpu) {
        struct rv_edge_stat *s = per_cpu_ptr(mon->edge_pcpu, cpu);
    ...
}

Should rv_remove() be called before freeing the per-CPU buffer?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260827072400.45734-1-tobias.schaffner@siemens.com?part=2

[RFC 3/3] rv: collect per-edge dwell time for per-cpu DA/HA monitors

[Tobias Schaffner]

With the core facility in place, hook it into the DA/HA layer so that any
per-cpu automaton monitor also reports how long it lingers in each state.

Add a selftest that enables a per-cpu monitor, checks the stats file
appears and is populated under load, and skips cleanly otherwise.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 include/linux/rv.h                            |  4 ++
 include/rv/da_monitor.h                       | 53 +++++++++++++++++++
 tools/testing/selftests/verification/config   |  1 +
 .../verification/test.d/rv_edge_stats.tc      | 32 +++++++++++
 4 files changed, 90 insertions(+)
 create mode 100644 tools/testing/selftests/verification/test.d/rv_edge_stats.tc

diff --git a/include/linux/rv.h b/include/linux/rv.h
index 7eeecce17e50..09363f79ca90 100644
--- a/include/linux/rv.h
+++ b/include/linux/rv.h
@@ -27,6 +27,10 @@
 struct da_monitor {
 	bool		monitoring;
 	unsigned int	curr_state;
+#ifdef CONFIG_RV_EDGE_STAT
+	/* local_clock() ns when curr_state was entered; 0 = not yet stamped. */
+	u64		state_ns;
+#endif
 };
 
 #ifdef CONFIG_RV_LTL_MONITOR
diff --git a/include/rv/da_monitor.h b/include/rv/da_monitor.h
index 34b8fba9ecd4..59ca9a286c10 100644
--- a/include/rv/da_monitor.h
+++ b/include/rv/da_monitor.h
@@ -16,6 +16,8 @@
 
 #include <rv/automata.h>
 #include <linux/rv.h>
+#include <linux/rv_edge_stat.h>
+#include <linux/sched/clock.h>
 #include <linux/stringify.h>
 #include <linux/bug.h>
 #include <linux/sched.h>
@@ -30,6 +32,54 @@
 
 static struct rv_monitor rv_this;
 
+/* per-edge dwell statistics, wired up for per-cpu monitors. */
+#if defined(CONFIG_RV_EDGE_STAT) && RV_MON_TYPE == RV_MON_PER_CPU
+static void
+rv_this_edge_name(unsigned int edge, char *buf, size_t len)
+{
+	snprintf(buf, len, "%s:%s", model_get_state_name(edge / EVENT_MAX),
+		 model_get_event_name(edge % EVENT_MAX));
+}
+
+static const struct rv_edge_cfg rv_this_edge_cfg = {
+	.n_edges   = STATE_MAX * EVENT_MAX,
+	.edge_name = rv_this_edge_name,
+};
+
+/* Hand the model's edge descriptor to the core; called from da_monitor_init(). */
+static inline void rv_edge_bind(void)
+{
+	rv_this.edge_cfg = &rv_this_edge_cfg;
+}
+
+/* Stamp the moment a state is entered, so its dwell can be timed on exit. */
+static __always_inline void rv_da_edge_enter(struct da_monitor *da_mon)
+{
+	da_mon->state_ns = local_clock();
+}
+
+/* Account the dwell in @curr, then stamp entry into the next state. */
+static __always_inline void
+rv_da_edge_account(struct da_monitor *da_mon, enum states curr, enum events ev)
+{
+	u64 now = local_clock();
+	u64 prev = da_mon->state_ns;
+
+	da_mon->state_ns = now;
+	/*
+	 * local_clock() is not guaranteed monotonic; drop the sample if it did
+	 * not advance so a backward step cannot underflow into a bogus dwell.
+	 */
+	if (rv_this.edge_pcpu && prev && now > prev)
+		rv_edge_account(&rv_this, curr * EVENT_MAX + ev, now - prev);
+}
+#else
+static inline void rv_edge_bind(void) { }
+static inline void rv_da_edge_enter(struct da_monitor *da_mon) { }
+static inline void
+rv_da_edge_account(struct da_monitor *da_mon, enum states curr, enum events ev) { }
+#endif /* CONFIG_RV_EDGE_STAT && RV_MON_PER_CPU */
+
 /*
  * Hook to allow the implementation of hybrid automata: define it with a
  * function that takes curr_state, event and next_state and returns true if the
@@ -113,6 +163,7 @@ static inline void da_monitor_reset(struct da_monitor *da_mon)
 static inline void da_monitor_start(struct da_monitor *da_mon)
 {
 	da_mon->curr_state = model_get_initial_state();
+	rv_da_edge_enter(da_mon);
 	da_monitor_init_hook(da_mon);
 	/* Pairs with smp_load_acquire in da_monitoring(). */
 	smp_store_release(&da_mon->monitoring, 1);
@@ -275,6 +326,7 @@ static inline void da_monitor_reset_state_all(void)
  */
 static inline int da_monitor_init(void)
 {
+	rv_edge_bind();
 	da_monitor_reset_state_all();
 	return 0;
 }
@@ -696,6 +748,7 @@ static inline bool da_event(struct da_monitor *da_mon, enum events event, da_id_
 		if (likely(try_cmpxchg(&da_mon->curr_state, &curr_state, next_state))) {
 			if (!da_monitor_event_hook(da_mon, curr_state, event, next_state, id))
 				return false;
+			rv_da_edge_account(da_mon, curr_state, event);
 			da_trace_event(da_mon, model_get_state_name(curr_state),
 				       model_get_event_name(event),
 				       model_get_state_name(next_state),
diff --git a/tools/testing/selftests/verification/config b/tools/testing/selftests/verification/config
index 43072c1c38f4..de229777d5c8 100644
--- a/tools/testing/selftests/verification/config
+++ b/tools/testing/selftests/verification/config
@@ -1 +1,2 @@
 CONFIG_RV=y
+CONFIG_RV_EDGE_STAT=y
diff --git a/tools/testing/selftests/verification/test.d/rv_edge_stats.tc b/tools/testing/selftests/verification/test.d/rv_edge_stats.tc
new file mode 100644
index 000000000000..5fb2384d938c
--- /dev/null
+++ b/tools/testing/selftests/verification/test.d/rv_edge_stats.tc
@@ -0,0 +1,32 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# description: Test per-edge dwell-time statistics (stats)
+
+MON=wip
+
+check_requires "$MON:monitor"
+
+# The "stats" file is created on first enable; if it is still missing the
+# kernel was built without CONFIG_RV_EDGE_STAT, so skip.
+echo 1 > "monitors/$MON/enable"
+if [ ! -e "monitors/$MON/stats" ]; then
+	echo 0 > "monitors/$MON/enable"
+	echo "CONFIG_RV_EDGE_STAT is not enabled."
+	exit_unsupported
+fi
+
+# The first line is the column header; the body has one line per (cpu, edge).
+head -n1 "monitors/$MON/stats" | grep -q "^# cpu edge label count max_ns sum_ns"
+[ "$(grep -cvE '^#' "monitors/$MON/stats")" -gt 0 ]
+
+# Drive some scheduler activity so the automaton records transitions.
+for _ in 1 2 3 4 5 6 7 8 9 10; do
+	(true) &
+	wait
+done
+
+# At least one edge must now show a non-zero count, with sum_ns >= max_ns > 0.
+grep -vE '^#' "monitors/$MON/stats" | \
+	awk '$4 > 0 && $5 > 0 && $6 >= $5 { hit = 1 } END { exit !hit }'
+
+echo 0 > "monitors/$MON/enable"
-- 
2.43.0

Re: [RFC 3/3] rv: collect per-edge dwell time for per-cpu DA/HA monitors

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [Critical] Race condition in rv_edge_setup error path causes Use-After-Free and NULL pointer dereference by concurrent trace events.
- [High] Race condition and non-atomic update of state_ns leads to corrupted per-edge dwell time statistics.

Pre-existing issues:
- [Critical] Use-After-Free and Control-Flow Hijack when reading tracefs files after the monitor module is unloaded.
--

commit 087c3d3fb8881f044270530a0540c2d8d606c74c
Author: Tobias Schaffner <tobias.schaffner@siemens.com>

rv: collect per-edge dwell time for per-cpu DA/HA monitors

With the core facility in place, hook it into the DA/HA layer so that any
per-cpu automaton monitor also reports how long it lingers in each state.

Add a selftest that enables a per-cpu monitor, checks the stats file
appears and is populated under load, and skips cleanly otherwise.

> diff --git a/include/rv/da_monitor.h b/include/rv/da_monitor.h
> index 34b8fba9ecd44..59ca9a286c106 100644
> --- a/include/rv/da_monitor.h
> +++ b/include/rv/da_monitor.h

[ ... ]

> +static void
> +rv_this_edge_name(unsigned int edge, char *buf, size_t len)
> +{
> +	snprintf(buf, len, "%s:%s", model_get_state_name(edge / EVENT_MAX),
> +		 model_get_event_name(edge % EVENT_MAX));
> +}
> +
> +static const struct rv_edge_cfg rv_this_edge_cfg = {
> +	.n_edges   = STATE_MAX * EVENT_MAX,
> +	.edge_name = rv_this_edge_name,
> +};

[Severity: Critical]
This is a pre-existing issue, but is it possible to trigger a use-after-free
and control-flow hijack when reading tracefs files after the monitor module
is unloaded?

The RV tracefs files (like stats, enable, desc) are created by the core RV
module, so their fops.owner does not pin the individual monitor modules.

If a user keeps monitors/<mon>/stats open and unloads the monitor module,
the module's memory is freed.

A subsequent read() on the open file descriptor invokes rv_edge_stats_show()
which dereferences seq->private pointing to the freed struct rv_monitor and
calls cfg->edge_name(), jumping to unmapped or attacker-controlled memory.

[ ... ]

> +/* Account the dwell in @curr, then stamp entry into the next state. */
> +static __always_inline void
> +rv_da_edge_account(struct da_monitor *da_mon, enum states curr, enum events ev)
> +{
> +	u64 now = local_clock();
> +	u64 prev = da_mon->state_ns;
> +
> +	da_mon->state_ns = now;

[Severity: High]
Is this non-atomic update of state_ns detached from the atomic cmpxchg of
curr_state, allowing nested interrupts to read a stale timestamp or torn
value?

In da_event(), try_cmpxchg is used to safely advance curr_state under
concurrent execution on the same CPU.

If an interrupt fires immediately after try_cmpxchg succeeds but before this
rv_da_edge_account() runs, the nested interrupt will execute its own state
transition and read the stale state_ns.

It will attribute the outer state's dwell time to its own transition, and
write a new state_ns.

When the interrupt returns, the outer event resumes and overwrites state_ns
again, swapping timestamps and causing massive artificial spikes or
underflows in the statistics.

> +	/*
> +	 * local_clock() is not guaranteed monotonic; drop the sample if it did
> +	 * not advance so a backward step cannot underflow into a bogus dwell.
> +	 */
> +	if (rv_this.edge_pcpu && prev && now > prev)

[Severity: Critical]
Does reading rv_this.edge_pcpu here without READ_ONCE() or synchronization
make it vulnerable to a race condition with the error path in
rv_edge_setup()?

In rv_enable_single() from kernel/trace/rv/rv.c, mon->enable() is called to
activate trace events globally before rv_edge_setup() allocates edge_pcpu.

If rv_create_file() fails during rv_edge_setup(), the error path calls
free_percpu(mon->edge_pcpu) and sets the pointer to NULL while tracepoints
are already active.

If a concurrent trace event firing on another CPU evaluates
rv_this.edge_pcpu before it is NULLed but uses it after it is freed, a
use-after-free occurs.

Additionally, if the compiler reloads the NULL pointer, this_cpu_ptr(NULL)
resolves to the base address of the CPU's per-cpu region. This causes trace
events to overwrite critical per-cpu data (such as current_task or
irq_stack_ptr) and lead to an immediate kernel crash.

> +		rv_edge_account(&rv_this, curr * EVENT_MAX + ev, now - prev);
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260827072400.45734-1-tobias.schaffner@siemens.com?part=3

Re: [RFC 0/3] rv: per-edge dwell-time statistics for per-cpu monitors

[Gabriele Monaco]

On Thu, 2026-08-27 at 09:23 +0200, Tobias Schaffner wrote:
> Hi Steven, Gabriele,
> 
> I am using the RV system for a downstream project called EVL (aka Xenomai4).
> It is a small co-kernel system, small enough that I can model the whole
> co-kernel with RV instead of just a fragment of it.

Hi Tobias,

Haven't heard about Xenomai in a while.. That sounds very interesting.

Since it's something out-of-tree you likely don't want to submit monitors
upstream, but if you can trigger BPF events from your co-kernel you may want to
have a look at RV BPF monitors, which I'm likely going to submit in a couple of
days.

Otherwise we can always get out-of-tree monitors as modules, currently not
supported just out of a missing use-case.

> The patches in this series add a per-monitor "stats" file that exposes how
> often an edge has been visited and the dwell time in the prior state for
> per-cpu monitors.

The idea crossed my mind a few times, good you came up with an implementation!

> This allows me to not only check if the model is correct but also to see if
> all paths are exercised by my stress tooling and to decompose the latency of
> the co-kernel's wakeup path.

Your first need (see if all paths are exercised) doesn't really require the stat
to be per-monitor-instance (i.e. per-cpu) right?
You could have a global accumulator matrix, at least for count and sum where you
can be atomic. This could extend the feature to other types of monitors.

Just brainstorming here though.

> I think that some of the upstream monitors like e.g. the sts monitor could
> also
> profit from this.
> 
> As an example, here is the sts monitor on a StarFive VisionFive2 running an rt
> kernel (isolcpus=2-3), after enabling it and generating some scheduler load.
> Only the edges the automaton actually takes are non-zero:
> 
>   # cat monitors/sched/sts/stats
>   # cpu edge label count max_ns sum_ns
>   0 0 can_sched:irq_disable 367732 3998500 14773091250
>   0 4 can_sched:schedule_entry 40282 398750 45510750
>   0 7 cant_sched:irq_enable 367732 32500 532791750
>   0 8 cant_sched:irq_entry 26188 6250 38357250
>   0 13 disable_to_switch:irq_enable 4933 12500 12275000
>   0 14 disable_to_switch:irq_entry 31 2250 35250
>   0 15 disable_to_switch:sched_switch 35373 26750 111615750
>   0 18 enable_to_exit:irq_disable 373 3250 649750
>   0 19 enable_to_exit:irq_enable 373 20000 1676250
>   0 20 enable_to_exit:irq_entry 309 4750 420000
>   0 23 enable_to_exit:schedule_exit 40298 3000 37335000
>   0 25 in_irq:irq_enable 31 12250 140750
>   0 30 scheduling:irq_disable 40337 18000 45449500
>   0 37 switching:irq_enable 35373 9250 72471000
>   [ cpus 1-2 omitted ]
>   3 0 can_sched:irq_disable 567 3983750 300860250
>   3 4 can_sched:schedule_entry 150 1500 171000
>   3 7 cant_sched:irq_enable 567 21500 1777500
>   3 8 cant_sched:irq_entry 76 6500 164750
>   3 15 disable_to_switch:sched_switch 150 4250 413750
>   3 23 enable_to_exit:schedule_exit 150 1500 147750
>   3 30 scheduling:irq_disable 150 1500 166000
>   3 37 switching:irq_enable 150 3750 277000
> 
> The sts model splits the schedule->switch path into separate states, so the
> per-edge dwell decomposes the scheduler's interrupts-off window: on cpu0 the
> prep phase (disable_to_switch, irqs off until the switch) tops out at 26.8us
> and the switch itself (switching:irq_enable) at 9.3us, over 35k switches. The
> counts also line up with the model, e.g. every irq_disable in can_sched has a
> matching irq_enable in cant_sched (367732 == 367732). The isolated cpus 2-3
> take almost no switches (150 on cpu3, versus 35k on cpu0) and their worst case
> is tighter still (~4us), so the same file also makes the effect of cpu
> isolation visible per cpu.
> 
> The changes are gated by CONFIG_RV_EDGE_STAT and dormant until a monitor is
> enabled, so existing setups are unaffected. When enabled, each accepted
> transition adds one local_clock() and a few lock-free local64_t updates. With
> the config off there is no code on the hot path at all.
> 
> I focused on per-cpu monitors as a first step. Per-task and per-object
> monitors would have to aggregate entities that share a cpu, which is harder
> to get right, so I left them out for now.

As said before, I haven't really played with an implementation but I think
atomic types may help here.

> What is your opinion on this? Do you think this is worth getting upstreamed?

I will have a look at your patches but it's definitely something I'd want.

Thanks,
Gabriele

> 
> Thanks for taking a look,
> Tobias
> 
> Tobias Schaffner (3):
>   rv: add per-edge dwell-time statistics primitive
>   rv: add per-monitor edge-stat facility and stats file
>   rv: collect per-edge dwell time for per-cpu DA/HA monitors
> 
>  .../trace/rv/runtime-verification.rst         |  24 ++++
>  MAINTAINERS                                   |   1 +
>  include/linux/rv.h                            |  18 +++
>  include/linux/rv_edge_stat.h                  |  60 +++++++++
>  include/rv/da_monitor.h                       |  53 ++++++++
>  kernel/trace/rv/Kconfig                       |  11 ++
>  kernel/trace/rv/rv.c                          | 116 +++++++++++++++++-
>  tools/testing/selftests/verification/config   |   1 +
>  .../verification/test.d/rv_edge_stats.tc      |  32 +++++
>  9 files changed, 315 insertions(+), 1 deletion(-)
>  create mode 100644 include/linux/rv_edge_stat.h
>  create mode 100644
> tools/testing/selftests/verification/test.d/rv_edge_stats.tc

Re: [RFC 0/3] rv: per-edge dwell-time statistics for per-cpu monitors

[Tobias Schaffner]

Hi Gabriele,

thanks for the quick reply and the motivating words.

On 8/27/26 10:18, Gabriele Monaco wrote:
> On Thu, 2026-08-27 at 09:23 +0200, Tobias Schaffner wrote:
>> Hi Steven, Gabriele,
>>
>> I am using the RV system for a downstream project called EVL (aka Xenomai4).
>> It is a small co-kernel system, small enough that I can model the whole
>> co-kernel with RV instead of just a fragment of it.
> 
> Hi Tobias,
> 
> Haven't heard about Xenomai in a while.. That sounds very interesting.
> 
> Since it's something out-of-tree you likely don't want to submit monitors
> upstream, but if you can trigger BPF events from your co-kernel you may want to
> have a look at RV BPF monitors, which I'm likely going to submit in a couple of
> days.

Exactly. The monitors themselves will live downstream in the Dovetail 
and EVL projects.

I will definitely have a look at the RV BPF monitors!

> Otherwise we can always get out-of-tree monitors as modules, currently not
> supported just out of a missing use-case.
> 
>> The patches in this series add a per-monitor "stats" file that exposes how
>> often an edge has been visited and the dwell time in the prior state for
>> per-cpu monitors.
> 
> The idea crossed my mind a few times, good you came up with an implementation!
> 
>> This allows me to not only check if the model is correct but also to see if
>> all paths are exercised by my stress tooling and to decompose the latency of
>> the co-kernel's wakeup path.
> 
> Your first need (see if all paths are exercised) doesn't really require the stat
> to be per-monitor-instance (i.e. per-cpu) right?
> You could have a global accumulator matrix, at least for count and sum where you
> can be atomic. This could extend the feature to other types of monitors.

Yes that's correct.

For the dwell timings, though, keeping it per-cpu matters to make
isolation effects visible. I use sum and count there to derive
the average.

> Just brainstorming here though.
> 
>> I think that some of the upstream monitors like e.g. the sts monitor could
>> also
>> profit from this.
>>
>> As an example, here is the sts monitor on a StarFive VisionFive2 running an rt
>> kernel (isolcpus=2-3), after enabling it and generating some scheduler load.
>> Only the edges the automaton actually takes are non-zero:
>>
>>    # cat monitors/sched/sts/stats
>>    # cpu edge label count max_ns sum_ns
>>    0 0 can_sched:irq_disable 367732 3998500 14773091250
>>    0 4 can_sched:schedule_entry 40282 398750 45510750
>>    0 7 cant_sched:irq_enable 367732 32500 532791750
>>    0 8 cant_sched:irq_entry 26188 6250 38357250
>>    0 13 disable_to_switch:irq_enable 4933 12500 12275000
>>    0 14 disable_to_switch:irq_entry 31 2250 35250
>>    0 15 disable_to_switch:sched_switch 35373 26750 111615750
>>    0 18 enable_to_exit:irq_disable 373 3250 649750
>>    0 19 enable_to_exit:irq_enable 373 20000 1676250
>>    0 20 enable_to_exit:irq_entry 309 4750 420000
>>    0 23 enable_to_exit:schedule_exit 40298 3000 37335000
>>    0 25 in_irq:irq_enable 31 12250 140750
>>    0 30 scheduling:irq_disable 40337 18000 45449500
>>    0 37 switching:irq_enable 35373 9250 72471000
>>    [ cpus 1-2 omitted ]
>>    3 0 can_sched:irq_disable 567 3983750 300860250
>>    3 4 can_sched:schedule_entry 150 1500 171000
>>    3 7 cant_sched:irq_enable 567 21500 1777500
>>    3 8 cant_sched:irq_entry 76 6500 164750
>>    3 15 disable_to_switch:sched_switch 150 4250 413750
>>    3 23 enable_to_exit:schedule_exit 150 1500 147750
>>    3 30 scheduling:irq_disable 150 1500 166000
>>    3 37 switching:irq_enable 150 3750 277000
>>
>> The sts model splits the schedule->switch path into separate states, so the
>> per-edge dwell decomposes the scheduler's interrupts-off window: on cpu0 the
>> prep phase (disable_to_switch, irqs off until the switch) tops out at 26.8us
>> and the switch itself (switching:irq_enable) at 9.3us, over 35k switches. The
>> counts also line up with the model, e.g. every irq_disable in can_sched has a
>> matching irq_enable in cant_sched (367732 == 367732). The isolated cpus 2-3
>> take almost no switches (150 on cpu3, versus 35k on cpu0) and their worst case
>> is tighter still (~4us), so the same file also makes the effect of cpu
>> isolation visible per cpu.
>>
>> The changes are gated by CONFIG_RV_EDGE_STAT and dormant until a monitor is
>> enabled, so existing setups are unaffected. When enabled, each accepted
>> transition adds one local_clock() and a few lock-free local64_t updates. With
>> the config off there is no code on the hot path at all.
>>
>> I focused on per-cpu monitors as a first step. Per-task and per-object
>> monitors would have to aggregate entities that share a cpu, which is harder
>> to get right, so I left them out for now.
> 
> As said before, I haven't really played with an implementation but I think
> atomic types may help here.
> 
>> What is your opinion on this? Do you think this is worth getting upstreamed?
> 
> I will have a look at your patches but it's definitely something I'd want.

Looking forward to your input! I am very open to guidance on how to get
this right.

Best,
Tobias

> Thanks,
> Gabriele
> 
>>
>> Thanks for taking a look,
>> Tobias
>>
>> Tobias Schaffner (3):
>>    rv: add per-edge dwell-time statistics primitive
>>    rv: add per-monitor edge-stat facility and stats file
>>    rv: collect per-edge dwell time for per-cpu DA/HA monitors
>>
>>   .../trace/rv/runtime-verification.rst         |  24 ++++
>>   MAINTAINERS                                   |   1 +
>>   include/linux/rv.h                            |  18 +++
>>   include/linux/rv_edge_stat.h                  |  60 +++++++++
>>   include/rv/da_monitor.h                       |  53 ++++++++
>>   kernel/trace/rv/Kconfig                       |  11 ++
>>   kernel/trace/rv/rv.c                          | 116 +++++++++++++++++-
>>   tools/testing/selftests/verification/config   |   1 +
>>   .../verification/test.d/rv_edge_stats.tc      |  32 +++++
>>   9 files changed, 315 insertions(+), 1 deletion(-)
>>   create mode 100644 include/linux/rv_edge_stat.h
>>   create mode 100644
>> tools/testing/selftests/verification/test.d/rv_edge_stats.tc
>