Re: [RFC i-g-t] kunit: Remove hard-coded test list from xe_live_ktest

[John Harrison <john.c.harrison@intel.com>]

On 8/29/2024 02:57, Janusz Krzysztofik wrote:
> Hi John,
>
> On Wednesday, 28 August 2024 20:04:06 GMT+2 John Harrison wrote:
>> On 8/26/2024 06:07, Janusz Krzysztofik wrote:
>>> Hi John, Kamil,
>>>
>>> On Monday, 26 August 2024 13:46:45 GMT+2 Kamil Konieczny wrote:
>>>> Hi John.C.Harrison,
>>>> On 2024-08-23 at 11:24:18 -0700, John.C.Harrison@Intel.com wrote:
>>>>> From: johnharr <johnharr@invalid-email.com>
>>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>> Again invalid e-mail here.
>>>>
>>>>> The list of supported kunit tests is currently hard coded.
>>> That pattern originates from i915_selftest, where there are 3 stable subtests,
>>> "mock", "live" and "perf", each of them executing a (possibly long) list of
>>> dynamic sub-subtests actually provided by the module for each category.  Also,
>>> IIRC there was a separate module for each Xe test suite before, each mapped to
>>> a separate IGT test, later merged into one module with multiple suites and one
>>> test with multiple corresponding subtests.
>> Not sure if you are just explaining the history of the test or making a
>> suggestion as to how it should evolve next?
> Both, I think.  Maybe not the history, but origin of ideas standing behind the
> implementation, and how tests are expected to use it (and maybe evolve if now
> doing that in a different way).
>
>>>>> Which means
>>>>> that adding a new test to the kernel also requires patches to IGT as
>>>>> well.
>>>>>
>>>>> The list of available kunit tests is already exported by the kernel.
>>>>> So there is no need to bake a list into the IGT source code. So, add
>>>>> support for querying the test list dynamically.
>>>>>
>>>>> NB: Currently, the test list can only be queried by root but the IGT
>>>>> build system queries all tests at compile time. In theory this should
>>>>> not be a problem. However, the kunit helper code is all written to run
>>>>> inside a test and not inside the prep code, which means that when it
>>>>> fails to open the root only interfaces, it calls 'skip'. And skip is
>>>>> not allowed outside of running a test. Hence the build fails with:
>>>>>     skipping is allowed only in fixtures, subtests or igt_simple_main
>>>> Looks like we should fix it, move out skips from kunit libs.
>>> I suggest you consider a different approach: for a module, call igt_kunit()
>>> only once, with NULL suite argument.  As a result, you'll get results from one
>>> IGT subtest called "<module_name>" with a bunch of IGT dynamic sub-subtests
>>> called "<test_suite>-<test_case>", one for each test case provided by the
>>> module.
>> I'm not following. This is what my patch does, isn't it?
> No, your patch introduces a runtime determined list of subtests -- something
> not existent in IGT.
>
> An IGT test may consist of one or more statically defined subtests with pre-
> defined names.  The term dynamic subtest is usually used in two meanings.  It
> may mean a subtest of type igt_subtest_with_dynamic, still with a pre-defined
> name, but with a runtime determined list of sub-subtests, sometimes called
> dynamic sub-subtests, but often also called just dynamic subtests.  Names of
> dynamic sub-subtests are determined at runtime.
>
> My approach tries to address your need for a maintenance-free kunit IGT test
> source file in a different way.  I'm following the IGT standard of statically
> defined list of subtests with pre-defined names: one subtest of type
> igt_subtest_with_dynamic, named after the kunit test module name which you
> have to enter into the test code anyway, and providing all test cases from
> all test suites contained in that module reported as dynamic sub-subtests
> named <test_suite>-<test_case>.
Can you please prototype how to do this?  I can read your words but I 
don't really get your meaning and I have no clue how to implement what 
you are saying.

John.