Re: [RFC i-g-t] kunit: Remove hard-coded test list from xe_live_ktest

[Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com>]

Hi John,

On Saturday, 14 September 2024 02:20:32 GMT+2 John Harrison wrote:
> On 8/29/2024 02:57, Janusz Krzysztofik wrote:
> > Hi John,
> >
> > On Wednesday, 28 August 2024 20:04:06 GMT+2 John Harrison wrote:
> >> On 8/26/2024 06:07, Janusz Krzysztofik wrote:
> >>> Hi John, Kamil,
> >>>
> >>> On Monday, 26 August 2024 13:46:45 GMT+2 Kamil Konieczny wrote:
> >>>> Hi John.C.Harrison,
> >>>> On 2024-08-23 at 11:24:18 -0700, John.C.Harrison@Intel.com wrote:
> >>>>> From: johnharr <johnharr@invalid-email.com>
> >>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> >>>> Again invalid e-mail here.
> >>>>
> >>>>> The list of supported kunit tests is currently hard coded.
> >>> That pattern originates from i915_selftest, where there are 3 stable 
subtests,
> >>> "mock", "live" and "perf", each of them executing a (possibly long) list 
of
> >>> dynamic sub-subtests actually provided by the module for each category.  
Also,
> >>> IIRC there was a separate module for each Xe test suite before, each 
mapped to
> >>> a separate IGT test, later merged into one module with multiple suites 
and one
> >>> test with multiple corresponding subtests.
> >> Not sure if you are just explaining the history of the test or making a
> >> suggestion as to how it should evolve next?
> > Both, I think.  Maybe not the history, but origin of ideas standing behind 
the
> > implementation, and how tests are expected to use it (and maybe evolve if 
now
> > doing that in a different way).
> >
> >>>>> Which means
> >>>>> that adding a new test to the kernel also requires patches to IGT as
> >>>>> well.
> >>>>>
> >>>>> The list of available kunit tests is already exported by the kernel.
> >>>>> So there is no need to bake a list into the IGT source code. So, add
> >>>>> support for querying the test list dynamically.
> >>>>>
> >>>>> NB: Currently, the test list can only be queried by root but the IGT
> >>>>> build system queries all tests at compile time. In theory this should
> >>>>> not be a problem. However, the kunit helper code is all written to run
> >>>>> inside a test and not inside the prep code, which means that when it
> >>>>> fails to open the root only interfaces, it calls 'skip'. And skip is
> >>>>> not allowed outside of running a test. Hence the build fails with:
> >>>>>     skipping is allowed only in fixtures, subtests or igt_simple_main
> >>>> Looks like we should fix it, move out skips from kunit libs.
> >>> I suggest you consider a different approach: for a module, call 
igt_kunit()
> >>> only once, with NULL suite argument.  As a result, you'll get results 
from one
> >>> IGT subtest called "<module_name>" with a bunch of IGT dynamic sub-
subtests
> >>> called "<test_suite>-<test_case>", one for each test case provided by 
the
> >>> module.
> >> I'm not following. This is what my patch does, isn't it?
> > No, your patch introduces a runtime determined list of subtests -- 
something
> > not existent in IGT.
> >
> > An IGT test may consist of one or more statically defined subtests with 
pre-
> > defined names.  The term dynamic subtest is usually used in two meanings.  
It
> > may mean a subtest of type igt_subtest_with_dynamic, still with a pre-
defined
> > name, but with a runtime determined list of sub-subtests, sometimes called
> > dynamic sub-subtests, but often also called just dynamic subtests.  Names 
of
> > dynamic sub-subtests are determined at runtime.
> >
> > My approach tries to address your need for a maintenance-free kunit IGT 
test
> > source file in a different way.  I'm following the IGT standard of 
statically
> > defined list of subtests with pre-defined names: one subtest of type
> > igt_subtest_with_dynamic, named after the kunit test module name which you
> > have to enter into the test code anyway, and providing all test cases from
> > all test suites contained in that module reported as dynamic sub-subtests
> > named <test_suite>-<test_case>.
> Can you please prototype how to do this?  I can read your words but I 
> don't really get your meaning and I have no clue how to implement what 
> you are saying.

The test code may look as simple as:

igt_main
{
        igt_kunit("xe_live_test", NULL, NULL);
}

Then, igt_kunit() will use xe_live_test module and should report results from 
all its KUnit test cases as results from a set of IGT dynamic sub-subtests 
"<test_suite>-<test_case>" under a single IGT subtest "xe_live".

Janusz

> 
> John.
> 
>