Re: [RFC PATCH 00/11] rv: Add scheduler specification monitors

[Gabriele Monaco <gmonaco@redhat.com>]

On Fri, 2025-02-07 at 15:27 +0100, Juri Lelli wrote:
> On 07/02/25 12:36, Gabriele Monaco wrote:
> > 
> > 
> > On Fri, 2025-02-07 at 11:55 +0100, Juri Lelli wrote:
> > > Hi Gabriele,
> > > 
> > > On 06/02/25 09:09, Gabriele Monaco wrote:
> > > > This patchset starts including adapted scheduler specifications
> > > > from
> > > > Daniel's task model [1].
> > > 
> > > Thanks a lot for working on this. Apart from being cool stuff
> > > per-se,
> > > it
> > > means a lot personally to see Daniel's work continuing to be
> > > developed.
> > > 
> > > > As the model is fairly complicated, it is split in several
> > > > generators
> > > > and specifications. The tool used to create the model can
> > > > output a
> > > > unified model, but that would be hardly readable (9k states).
> > > > 
> > > > RV allows monitors to run and react concurrently. Running the
> > > > cumulative
> > > > model is equivalent to running single components using the same
> > > > reactors, with the advantage that it's easier to point out
> > > > which
> > > > specification failed in case of error.
> > > > 
> > > > We allow this by introducing nested monitors, in short, the
> > > > sysfs
> > > > monitor folder will contain a monitor named sched, which is
> > > > nothing
> > > > but
> > > > an empty container for other monitors. Controlling the sched
> > > > monitor
> > > > (enable, disable, set reactors) controls all nested monitors.
> > > > 
> > > > The task model proposed by Daniel includes 12 generators and 33
> > > > specifications. The generators are good for documentation but
> > > > are
> > > > usually implied in some specifications.
> > > > Not all monitors work out of the box, mainly because of those
> > > > reasons:
> > > > * need to distinguish if preempt disable leads to schedule
> > > > * need to distinguish if irq disable comes from an actual irq
> > > > * assumptions not always true on SMP
> > > > 
> > > > The original task model was designed for PREEMPT_RT and this
> > > > patchset is
> > > > only tested on an upstream kernel with full preemption enabled.
> > > 
> > > I played with your additions a bit and I was able to
> > > enable/disable
> > > monitors, switch reactors, etc., w/o noticing any issue.
> > > 
> > 
> > Thanks for trying it out!
> > 
> > > I wonder if you also had ways to test that the monitors actually
> > > react
> > > properly in case of erroneous conditions (so that we can see a
> > > reactor
> > > actually react :).
> > > 
> > 
> > Well, in my understanding, reactors should fire if there is a
> > problem
> > either in the kernel or in the model logic.
> 
> Right. I guess I wonder if we can find a way to inject kernel
> problems
> somehow, so that model(s) can be further tested explicitly thus
> making
> us confident that they will be able to identify real problems when
> they
> occur.
> 

Just for the sake of testing reactors there is already the wwnr monitor
which is intentionally broken exactly for that reason, Daniel described
a bit what scenario can trigger the error (some IRQ, so it may be
harder to see in a VM).

If we want any monitor to react, yeah that could be a bit trickier. We
could perhaps do something like livepatching/kprobes.

I'd assume we'd need some care though, since some of those monitors are
pretty basic and making them fail may cause pretty bad errors in the
kernel.
Another approach could be to just inject the tracepoint/handler call
those are all static functions but we may have some luck there and
wouldn't break the system, just trick the monitor.

But good point, I can have a look.