Re: [PATCH v2 10/15] tools/rv: Add BPF monitors

[Gabriele Monaco <gmonaco@redhat.com>]

On Mon, 2026-10-19 at 15:08 +0206, John Ogness wrote:
> On 2026-10-19, Tomas Glozar <tglozar@redhat.com> wrote:
> > Do I understand it correctly that the idea is to modify existing
> > tracefs instances when enabling an RV monitor to filter our RV-related
> > bpf_trace_printk events?
> 
> Not necessarily. But does the ordering matter? If I run:
> 
> $ sudo trace-cmd record -e bpf_trace:bpf_trace_printk \
>                         -f 'bpf_string ~ "rv: violation: *"'
> 
> it does not matter if the instance is running before or after starting
> the monitor.

I'm not sure if that was a question, but I don't think it would matter either
way. If RV generates it's own events, these exist even before the monitor runs
(just like bpf_printk) so it doesn't really matter when ftrace attaches to them.

> 
> > > > I can imagine an RV-specific trace event for that, however, it would
> > > > require a separate kfunc to generate it, and ideally also distinct
> > > > labeling of a BPF program eligible to call that kfunc. The "rv"
> > > > userspace tool could then iterate over these labels so the user knows
> > > > which violation events can be expected to occur on the system.
> > > 
> > > That'd work too. It's what I initially envisioned but I'm not sure a kfunc
> > > producing a tracepoint is something BPF folks would really like (and
> > > technically
> > > any other unrelated BPF program could call that, unless there's some
> > > access
> > > control mechanism I'm not aware of).
> > > 
> > > Not sure how much it matters in practice against the glob filter, it
> > > probably
> > > really depends on the other bpf_printk users (and a mildly better
> > > usability).
> > 
> > bpf_trace_printk is already a BPF helper producing a tracepoint, this
> > would be equivalent.

By the way, there are other occurrences in the kernel, apparently used only for
testing in include/trace/events/bpf_test_run.h

> > Of course, that does not mean we want it, but it
> > is something we already have in at least one instance. I know of at
> > least one quirk: you can have a BPF program attached to a tracepoint
> > that will call bpf_trace_printk(), and a tracefs instance tracing both
> > bpf_trace_printk() and the original tracepoint. That leads to a new
> > tracepoint triggering while another tracepoint is already in progress.
> > I don't know how it is currently handled; from what I recall when
> > testing RTLA's BPF actions, the bpf_trace_printk() event might get
> > dropped by the tracefs instance, so I had to use a different way to
> > signal to userspace in BPF actions test.
> 
> This is interesting because it is very likely that the RV monitor is
> attaching to and generating the violation from the same tracepoints that
> I may be additionally tracing. For example, the sleep RV monitor
> attaches to sched_waking, which I would also use while investigating my
> broken real-time system.

Yes, that's actually what standard RV has always been doing, I've never noticed
dropped tracepoints.
I thought bpf_printk used some shared buffers or some other things on top of
that but it doesn't seem to have anything but a tracepoint string.

> 
> However, I just ran a test with a simple eBPF program that attaches to a
> tracepoint and calls bpf_trace_printk(). I started trace-cmd to trace
> both the attach tracepoint and also bpf_trace_printk using a glob. After
> starting trace-cmd, I loaded/attached the eBPF program. The running
> trace-cmd was able to capture both events (with about 15us difference in
> timestamp).
> 
> > As I said, to prevent unrelated BPF programs from calling the new
> > tracepoint, you'd need to recognize and track RV BPF programs as a
> > special type of program, similarly to what your original RFC did. I'm
> > not sure if that's what we want, but I see some advantages. In
> > addition to better filtering of reactor events, you could separate
> > reactors from the monitors entirely: you could trigger the tracepoint
> > unconditionally, and attach reactors as tracepoint triggers to the new
> > tracepoint.
> 
> Thank you for making this suggestion. This is the direction I would
> prefer we take.
> 
> > All of that adds complexity, though, and requires separate discussion,
> > so it's probably not something relevant for the initial
> > implementation. I just wanted to mention it so that it doesn't get
> > lost.
> 
> Since relocating RV to userspace requires an enormous amount of effort
> and coordination (mainlining, packaging, distributions, documentation),
> I do not think it makes sense to have an "initial implementation" that
> is already not what we want in the long term.

Fair enough, I feel to be cautious at redefining what reactors are right now,
though. Currently they are nothing but a function call to the set-up action.
It's trivial to verify where the code goes without adding the tracing subsystem
as a dependency in the path (typically important in some safety critical cases).
Sure, handlers are called by the same tracing subsystem nonetheless, so it may
not add much value.

Let's leave reactors aside for a moment. What I am coming to terms with,
instead, is that the current ringbuffer to userspace design isn't what we want.
In kernel monitors communicate their status through tracepoints, perfectly
integrating to tracing tools, BPF monitors use a custom buffer.

Generating a tracepoint here is probably the way to go. bpf_printk does generate
one, but as Tomas pointed out, hijacking that for everything (think of monitor
events) might be a bit much.
Nonetheless, we want RV BPF handlers to generate tracepoints, then attaching
actions on error tracepoints (whether we call that reactors or not) can be done
later.

Summarising, we have 2 potential designs.

1. bpf_printk only:
  bpf_printk("event: monitor: curr_state x event -> next_state")
  bpf_printk("error: monitor: event not expected in curr_state")

2. new kfunc(s) calling a tracepoint: 
  trace_event_da_bpf("monitor", "curr_state", "event", "next_state")
  trace_error_da_bpf("monitor", "curr_state", "event")

1. is simpler and works mostly out of the box, depending on how powerful glob
filters are, it can reach the same functionality as 2. Might risk to overload
the shared bpf_printk event and perhaps have a higher filtering overhead.
2. is closest to the current in-kernel implementation and offers flexibility to
easily filter the type of event (useful for DA, e.g. I want a backtrace only
when going from state A to B). But it requires a series of kfuncs (e.g. error
and event for each monitor class).

In both cases a malicious actor could call the kfunc and make us believe there
was a reaction (they cannot mess with the model though). If we own the kfuncs we
can at least limit the availability (tracing-only and CAP_SYS_BOOT).
This, unless we register BPF monitors in the kernel like in V1.

I'm going to test each design once I get again to work on this.

Thoughts?

Gabriele