Re: [PATCH 3/3] iotests: distinguish 'skipped' and 'not run' states

[Peter Maydell <peter.maydell@linaro.org>]

On Thu, 14 Sept 2023 at 13:29, Kevin Wolf <kwolf@redhat.com> wrote:
>
> Am 13.09.2023 um 18:31 hat Eric Blake geschrieben:
> > I guess it boils down to whether there is an actionable response in
> > that message.  If a test is skipped because it is the wrong format
> > (for example, ./check -raw skipping a test that only works with
> > qcow2), there's nothing for me to do.  If a test is skipped because my
> > setup didn't permit running the test, but where I could enhance my
> > environment (install more software, pick a different file system,
> > ...), then having the skip message call that out is useful if I want
> > to take action to get more test coverage.
>
> I'm not sure if there is a clear line to define whether there is an
> actionable response or not, because that completely depends on what the
> user considers an acceptable response.
>
> You have the relatively clear cases like the test being for another
> format, though you could argue that the actionable response is running
> ./check -qcow2 if raw doesn't work - in fact, why don't we allow ./check
> -raw -qcow2 to run both and count it as skipped only if it wasn't run at
> all?
>
> The relatively clear case on the other end of the spectrum is if a tool
> is missing on the host that you could possibly install. Though maybe
> your distro doesn't even package it, so is that still a reasonable
> action to take?

I think for that sort of thing you need to actually collect the
reasons so that at the end you can say "skipped 21 tests because
frobnicator not present on system; skipped 4 tests because
bazulator not present on system". Then when you're running the
tests you can either (a) install the listed tools or (b) know
you can ignore those specific skips as ones you're aware you
can't run. But that would be a lot of pain and effort to implement...

-- PMM