Re: [RFC 2/5] check: Add -q <n> option to support unconditional looping.

[Ritesh Harjani (IBM) <ritesh.list@gmail.com>]

"Theodore Ts'o" <tytso@mit.edu> writes:

> On Fri, Jan 10, 2025 at 09:10:26AM +0000, Nirjhar Roy (IBM) wrote:
>> This patch adds -q <n> option through which one can run a given test <n>
>> times unconditionally. It also prints pass/fail metrics at the end.
>> 
>> The advantage of this over -L <n> and -i/-I <n> is that:
>>     a. -L <n> will not re-run a flakey test if the test passes for the first time.
>>     b. -I/-i <n> sets up devices during each iteration and hence slower.
>> Note -q <n> will override -L <n>.
>
> This is great!  It's something that I've wanted for a while, since at
> the moment I implement {gce,kvm}-xfstests -C 10 is to run check ten
> times, and doing something which does the looping inside check instead
> of outside will be much more efficient.

Yup "-q 10" could be used which can give pass/fail metrics of how many
times the test passed v/s failed by doing unconditional looping within
xfstest's check script itself. 

>
> One other thing that has been on my todo list to update, but which
> perhaps you might be willing to do while you are doing work in this
> area (nudge, nudge :-), is an optional mode which interates but which
> stops once a test fails.  This is essentially the reverse of -L, and
> the reason why it's useful is when trying to bisect a flakey test,
> which sometimes might only be failing 2-5% of the time, require
> running a test 30-50 times.  But the moment the test fails, we don't
> need to run the test any more, so this would speed up bisection tests
> which today I do via:
>
>    gce-xfstests ltm --repo linux.git --bisect-good v6.12 --bisect-bad \
> 	v6.13-rc1 -C 50 -c ext4/inline_data generic/273

There is -I which stops iterating on encountering a failure.
Does that work for this usecase?

>
> Because of this, I wonder if we should have one option to specify the
> number of interations, and then a different option which specifies the
> iteration mode, which might be "unconditional", "until first failure",
> "only if the test initially fails", etc, instead of separate options
> for -q, -L, etc.

I like the idea of iteration mode here. I will let others determine on
how easy it is to kill any option in xfstests today and replace it with
other. However here is a summary of different iteration and looping
options.

We have 3 options in xfstests today:
1. -i <n>              iterate the test list <n> times
2. -I <n>              iterate the test list <n> times, but stops iterating further in case of any test failure
3. -L <n>              loop tests <n> times following a failure, measuring aggregate pass/fail metrics

So we have -i/-I which are iterations and -q/-L which are loops. 
Looping happens when we can just loop over a particular test <n> times
and give pass/fail metrics.
Whereas in case of iterations it goes over from the beginning which will
also source different rc/config files and prepares the device etc. (hence
it's a bit slow too)

Using -q will be faster over using -i or -I similar to how -L is faster. However -L
will only re-run when there is a failure the 1st time.

-q v/s -l: Can we kill current -l option if it is not in use by anyone?
I would prefer -l since it looks a short form for looping. I don't see
we using -l anymore. But I will let others comment. 
    -l                  line mode diff


-ritesh