RE: KCIDB: Support one more test status

Bird, Tim
2023-04-19T20:46:04Z

> -----Original Message-----
> From: Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
> Hello everyone involved with, or interested in KCIDB,
>
> I would like to make KCIDB I/O schema accept one more test status string:
> "MISS" (preliminary name), meaning the test was supposed to run, but didn't
> because of a harness/framework/infrastructure/CI system failure.

>
> This would be distinct from the "SKIP" status, meaning the test was supposed
> to run, but didn't (or didn't complete), because it was not applicable.
>
> Here's the PR in question: https://github.com/kernelci/kcidb-io/pull/74
>
> I'll merge it in two weeks, on Wed, May 3, if there are no unresolved
> objections by that time.
>
> Read on for details, and respond either to this message, or in the PR.
>
> At CKI we have both tests that skip themselves for valid reasons, and tests
> that should've executed, but didn't. Because e.g. the previous test has
> crashed the machine, or the tested kernel failed to boot. At the same time, we
> want to make sure all the tests we planned to execute had their chance.
>
> Before testing, we send our plan to our database as a KCIDB dataset with all
> tests listed without the "status" field. According to KCIDB schema/protocol a
> missing "status" field simply means no data, and is canonically interpreted as
> "execution in progress".
>
> After the testing is done, or we gave up trying, we send the same KCIDB test
> objects to the database, but this time only containing whatever results we
> got, including the "status" fields. However, with the current set of status
> strings [1], the only way we can try to express "wanted to run, but couldn't"
> is with "SKIP", which is not supposed to alert anyone, yet this situation
> should be treated as a problem.
Why can't "wanted to run, but couldn't" be expressed with "ERROR"?

>
> We propose to call this new status "MISS" (as in "the test result should be
> there, but isn't"), and think it would be useful to others as well.
>
> We can break down the testing stack into three layers: the tested code, the
> test, and the harness (and everything above it) that runs the test. If we then
> express each existing test status as one trinary outcome per each of those
> layers, we would get this table (in order of descending status priority):
>
> STATUS CODE TEST HARNESS+ LEGEND
>
> FAIL ❌ ✅ ✅ ❌ - failure
> ERROR ➖ ❌ ✅ ✅ - success
> PASS ✅ ✅ ✅ ➖ - no data
> DONE ➖ ✅ ✅
> SKIP ➖ ➖ ✅
> ➖ ➖ ➖
>
> If you look at the above closely, you will notice one possible state missing
> (because we didn't need to express failing harnesses), and that is the status
> we want to introduce:
>
> STATUS CODE TEST HARNESS+ LEGEND
>
> FAIL ❌ ✅ ✅ ❌ - failure
> ERROR ➖ ❌ ✅ ✅ - success
> => MISS ➖ ➖ ❌ <= ➖ - no data
> PASS ✅ ✅ ✅
> DONE ➖ ✅ ✅
> SKIP ➖ ➖ ✅
> ➖ ➖ ➖
>
> Please respond with comments, objections, and (counter-)proposals,
> if you have them.

I don't understand the rationale for distinguishing a test error from a harness
error. In either case the test was not executed properly, and so there is no
useful test result data available. Diagnostic information should enable
the user to determine whether the problem was due to the test code failing
or the test harness failing.

I think I'm missing something. Are you trying to distinguish these so you
can determine whether there is a problem with the test itself, vs. the harness?
Are you automatically re-running a test if the harness is the problem?

Why do you want to distinguish these error cases?
-- Tim

>
> Thank you for your attention!
> Nick (and the CKI team)
>
> [1] The current set of supported status strings
>
> https://github.com/kernelci/kcidb-io/blob/0bb7ffff3fe012ae138dd2f7c1d817034fe2c0ba/kcidb_io/schema/v04_01.py#L126