[RFC PATCH 0/4] check: add --loop* family options for repeating tests

[Zorro Lang <zlang@kernel.org>]

This series originates from Ted Ts'o's RFC:

  [PATCH RFC] check: add new option "--loop <n>" which runs each test multiple times
  https://lore.kernel.org/fstests/20260415213248.1795275-1-tytso@mit.edu/

At the time, we didn't reach a consensus, and because the argument parsing in
./check was quite messy, I decided to refactor it before introducing more
options. That refactoring is now complete:

  https://lore.kernel.org/fstests/20260826102857.3224318-1-zlang@kernel.org/

so I think it's time to revisit this RFC and the follow-up changes.

 --loop re-runs each test unconditionally, fixing two problems of "-i
 <n>": per-run artifacts (e.g. NNN.out.bad) were not preserved, and
result.xml was rewritten for every test set, losing cumulative junit
statistics.  That RFC predates the check argument-parsing rewrite to
getopt, so patch 2 was rewritten on top of it, generalizing the -L
rerun state machine to support multiple loop modes.

Per-test looping options (mutually exclusive; repeating the same mode
overrides its count):

  --loop <n>              run each test <n> times.  Artifacts of every
                          run are stashed with .rerunN suffixes, the
                          junit report gets one testcase per run, an
                          aggregate pass/fail line is printed, and the
                          test fails if any of its runs fails.
  -L, --loop-on-fail <n>  rerun a test <n> times after its first
                          failure (<n>+1 runs in total).  Existing -L
                          behaviour, now with a long name.
  --loop-while-successful <n>
                          run each test up to <n> times, stopping at
                          the first failure; handy for bisecting
                          flaky tests.

Whole-list iteration gets long names as well:

  -i, --iterate <n>       iterate the entire test list <n> times
  -I, --iterate-while-successful <n>
                          same, but stop after the first round that
                          contains a failure

The short options -i, -I and -L keep their existing behaviour and
remain available. Maybe we can remove them one day, or keep them.

One open question about the junit report: with --loop each run adds a
testcase to result.xml, but the testsuite tests/failures attributes
still count unique tests, consistent with the "Ran: ... / Failed ..."
summary.  Before this series the two were always equal, so report
consumers that assume tests matches the number of testcase elements
would see them diverge for the first time.  We kept the current
semantics for now; comments on whether the attributes should count
per-run testcases instead (a follow-up common/report change) are
welcome.

Thanks,
Zorro

Theodore Ts'o (1):
  check: add --loop option to run each test multiple times

Zorro Lang (3):
  check: add long aliases --iterate and --iterate-while-successful
  check: add --loop-on-fail long name for -L
  check: add --loop-while-successful option

 check | 153 +++++++++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 120 insertions(+), 33 deletions(-)

-- 
2.56.0