[PATCH 00/11] tests: harden functional test suite against flakiness

[Wander Lairson Costa <wander@redhat.com>]

The functional test suite had several sources of intermittent
failure and weak diagnostics that made CI failures hard to
reproduce and debug: readiness detection for both stalld and
starvation_gen relied on fragile heuristics and racy liveness
checks, a test-ordering assumption used across several test
files turned out to be backwards, and the stalld startup
timeout was too short for PREEMPT_RT systems with large CPU
counts, where BPF tracepoint attachment can take well over 15
seconds due to SRCU synchronization. On top of that, many
assertion and startup failures did not dump the relevant log
output, and the test framework's own logging polluted the
system journal that other tests inspect directly.

This series addresses these issues end to end. stalld gains a
single deterministic "monitoring started" log marker emitted
once initialization completes, giving the test helpers a
stable synchronization point instead of matching internal
debug messages that could change across refactors. The
starvation_gen readiness check is reordered to check the log
file before process liveness, so a transient kill -0 miss no
longer aborts an otherwise healthy test. All 15 affected test
sections across five functional test files (test_fifo_boosting.sh,
test_fifo_priority_starvation.sh, test_starvation_detection.sh,
test_starvation_threshold.sh, and test_log_only.sh) are
reordered to start stalld before starvation_gen, since source
analysis showed the previous ordering rested on an unfounded
assumption about stalld's scanning loop and was itself a
source of intermittent timeouts. The stalld initialization
timeout is raised from 15s to 240s to tolerate slow BPF/SRCU
synchronization on large-CPU RT systems, paired with startup
timing instrumentation so the actual latency stays visible.
Diagnostics are improved by dumping log contents on every
assertion and startup failure, and helper functions are
switched to consistently call fail() instead of silently
swallowing errors. The test framework's own logging is
trimmed to stdout only, removing journal writes that had been
contaminating journalctl-based assertions in other tests.
Finally, run_tests.sh gains --verbose and --fail-fast options
for faster local debugging and CI triage.

With the exception of the single log marker added to stalld's
initialization path, every change in this series is confined
to the test framework and functional test scripts; no other
daemon runtime behavior is touched.

Wander Lairson Costa (11):
  stalld: add readiness log marker for tests
  tests: use fail() for error paths in helpers
  tests: add --verbose flag for bash trace output
  tests: drop journal logging from test framework
  tests: fix intermittent starvation_gen readiness failure
  tests: add --fail-fast flag to stop on first failure
  tests: dump log contents on assert_log_contains failure
  tests: dump log on start_stalld_with_log failure
  tests: start stalld before starvation_gen in all tests
  tests: add startup timing to stalld launch helpers
  tests: increase stalld initialization timeout to 240s

 src/stalld.c                                  |  2 +
 tests/functional/test_backend_selection.sh    |  2 +-
 tests/functional/test_fifo_boosting.sh        | 22 ++---
 .../test_fifo_priority_starvation.sh          | 38 ++++----
 tests/functional/test_log_only.sh             | 10 +--
 tests/functional/test_starvation_detection.sh | 18 ++--
 tests/functional/test_starvation_threshold.sh | 23 ++---
 tests/helpers/test_helpers.sh                 | 86 +++++++++----------
 tests/run_tests.sh                            | 23 ++++-
 9 files changed, 109 insertions(+), 115 deletions(-)

-- 
2.55.0