Re: [PATCH v2 2/3] check: Add -q <n> option to support unconditional looping.

[Dave Chinner <david@fromorbit.com>]

On Tue, Apr 15, 2025 at 01:02:49PM +0530, Nirjhar Roy (IBM) wrote:
> 
> On 4/14/25 03:18, Theodore Ts'o wrote:
> > On Thu, Apr 03, 2025 at 08:58:19AM +0000, Nirjhar Roy (IBM) wrote:
> > > This patch adds -q <n> option through which one can run a given test <n>
> > > times unconditionally. It also prints pass/fail metrics at the end.
> > > 
> > > The advantage of this over -L <n> and -i/-I <n> is that:
> > >      a. -L <n> will not re-run a flakey test if the test passes for the first time.
> > >      b. -I/-i <n> sets up devices during each iteration and hence slower.
> > > Note -q <n> will override -L <n>.
> > I'm wondering if we need to keep the current behavior of -I/-i.  The
> > primary difference between them and how your proposed -q works is that
> > instead of iterating over the section, your proposed option iterates
> > over each test.  So for example, if a section contains generic/001 and
> > generic/002, iterating using -i 3 will do this:
> 
> Yes, the motivation to introduce -q was to:
> 
> 1. Make the re-run faster and not re-format the device. -i re-formats the
> device and hence is slightly slower.

Why does -i reformat the test device on every run in your setup?
i.e. if the FSTYP is not changing from iteration to iteration, then
each iteration should not reformat the test device at all. Unless, of
course, you have told it to do so via the RECREATE_TEST_DEV env
variable....

Hence it seems to me like this is working around some other setup or
section iteration problem here...

> 2. To unconditionally loop a test - useful for scenarios when a flaky test
> doesn't fail for the first time (something that -L) does.

That's what -i does. it will unconditionally loop over the specified
tests N times regardless of success or failure.

OTOH, -I will abort on first failure. i.e. to enable flakey tests
to be run until it eventually fails and leave the corpse behind for
debugging.

> So, are saying that re-formatting a disk on every run, something that -i
> does, doesn't have much value and can be removed?

-i does not imply that the test device should be reformatted on
every loop. If that is happening, that is likely a result of test
config or environment conditions.

Can you tell us why the test device is getting reformatted on every
iteration in your setup?

> > generic/001
> > generic/002
> > generic/001
> > generic/002
> > generic/001
> > generic/002
> > 
> > While generic -q 3 would do this instead:
> > 
> > generic/001
> > generic/001
> > generic/001
> > generic/002
> > generic/002
> > generic/002

There are arguments both for and against the different iteration
orders. However, if there is no overriding reason to change the
existing order of test execution, then we should not change the
order or test execution....

-Dave.
-- 
Dave Chinner
david@fromorbit.com