Re: [PATCH] rteval: Introduce E2E tests with output checking

[John Kacur <jkacur@redhat.com>]

On Fri, 25 Jul 2025, Tomas Glozar wrote:

> Currently, rteval has two kinds of tests:
> - Unit tests, embedded directly in code, and run by
>   unit-tests/unittest.py.
> - End-to-end tests, implemented in Makefile targets runit, load, and
>   sysreport.
> 
> Introduce a new test suite in folder e2e-tests/ (analogically to
> unit-tests) that uses Test::Harness ("prove" command) together with a
> simple test engine adopted from RTLA.
> 
> The test suite runs rteval in a temporary folder for each test case,
> collects its exit value and output, and validates both according to the
> test specification. grep is used to check the output, optionally with
> custom flags. rteval.conf is generated individually based on the test
> specification of each case.
> 
> Three test sets are implemented in the commit:
> - "basic", running base checks without any specific modules or module
>   options. Replaces "make runit".
> - "loads", testing various kinds of loads, including their options.
> - "measurement", testing both cyclictest and timerlat.
> 
> The latter two check in rteval output whether the command run by rteval
> corresponds to the rteval options and to the specified load or
> measurement, if possible.
> 
> "make runit" and "make load" targets are depracated with a warning;
> "make sysreport" is kept until sysreport tests are added to the new test
> suite.
> 

Interesting.
Note the correct spelling is "deprecated"

I'm not sure we want to deprecate runit and loads, they have different 
purposes. I often run in a non-rt environment and without tuned if I'm 
just testing functionality and not performance, and I got this after 
running make check (skipping some of the output before that)

# Output match failed: "Command: rtla timerlat hist -p1100 -P f:95 -u 
-c[0-9|-]+ -E3500 --no-summary -T1 -t=trace.txt"
e2e-tests/measurement.t .. Failed 6/19 subtests 

Test Summary Report
-------------------
e2e-tests/loads.t      (Wstat: 0 Tests: 8 Failed: 1)
  Failed test:  5
e2e-tests/measurement.t (Wstat: 0 Tests: 19 Failed: 6)
  Failed tests:  13-14, 16-19
Files=3, Tests=33, 308 wallclock secs ( 0.06 usr  0.00 sys + 146.20 cusr 
281.79 csys = 428.05 CPU)
Result: FAIL
make: *** [Makefile:22: check] Error 1

How do I identify which test is test number 5?
Am I failing tests because of performance reasons or because the tests 
expect an environment different from mine?

Thanks

John Kacur