RE: KCIDB: Support one more test status

Bird, Tim
2023-04-20T22:08:03Z

> -----Original Message-----
> From: Nikolai Kondrashov <Nikolai.Kondrashov@redhat.com>
> Hi Tim,
>
> Thanks a lot for your response!
> I will do some snipping and answering below.
>
> On 4/19/23 22:38, Bird, Tim wrote:
> >> After the testing is done, or we gave up trying, we send the same KCIDB test
> >> objects to the database, but this time only containing whatever results we
> >> got, including the "status" fields. However, with the current set of status
> >> strings [1], the only way we can try to express "wanted to run, but couldn't"
> >> is with "SKIP", which is not supposed to alert anyone, yet this situation
> >> should be treated as a problem.
> >
> > Why can't "wanted to run, but couldn't" be expressed with "ERROR"?
>
> This is a matter of responsibility area and distinguishing who should be
> fixing the problem.
>
> The three layers I listed below correspond to the three distinct parties
> involved in testing, at Red Hat, and other large CI systems: "CODE" is for
> kernel developers/maintainers, "TEST" is for test maintainers, and "HARNESS+"
> is for CI system maintainers.
>
> At Red Hat we have the CKI project, which is responsible for maintaining the
> pipeline, builds, provisioning, reporting, etc - "HARNESS+". Then we have *a
> lot* of test maintainers, both internal (for the tests Red Hat needs), and
> external (for test suites like LTP) - "TEST". Finally, we have the kernel
> developers/maintainers, of course - "CODE".
>
> Naturally, if there was an issue with the test (normally reported as ERROR),
> we don't want to bother kernel developers. A test maintainer would need to
> deal with that (and they would get their notification), although they would be
> interested in just regular PASS/FAIL results too.
>
> Similarly, if the CI system couldn't manage to run the test, we won't want to
> report it as ERROR, because that would alert the test maintainer, while it
> wouldn't be their fault at all, and they shouldn't go and waste their time
> investigating it.
>
> Now, of course, this particular split is not always there, or it's not so
> clear-cut. E.g. kunit tests are normally maintained by kernel developers
> themselves. So they would be interested in both "CODE" and "TEST" layers for
> those, and so wouldn't really need the "ERROR" status - "FAIL" would be enough.
>
> CI system maintainers often take the role of test maintainers as well, and
> they wouldn't really need to distinguish "TEST" and "HARNESS+", for them it
> would be "TEST+", and they wouldn't need the "MISS" status - "ERROR" would be
> enough.
>
> However, this split (and the various statuses) is a good tool for handling all
> the various responsibility combinations the CI systems submitting to KCIDB
> have. It allows precise targeting of notifications and dashboard data, saving
> time and effort in many cases.

Ok - this sounds reasonable.
-- Tim