Re: Device Tree runtime unit tests: Harmonisation

[Frank Rowand <frowand.list@gmail.com>]

On 2/2/22 2:54 PM, Brendan Higgins wrote:
> On Wed, Feb 2, 2022 at 1:38 PM Frank Rowand <frowand.list@gmail.com> wrote:
>>
>> On 2/2/22 5:31 AM, Naresh Kamboju wrote:
>>> Linaro started doing Linux kernel Functional Validation (LKFT).
>>> As part of LKFT recently we have enabled CONFIG_OF_UNITTEST=y in our
>>> daily test CI.
>>>
>>> The output of the test looks as below. The current problem is that we
>>> have a hard time to see (grep) pass/fail for each individual test. We
>>> only see a summary at the end with x pass and y fails.
>>
>> The FAIL messages are printed at loglevel KERN_ERR.  The pass messages
>> are printed at loglevel KERN_DEBUG.  To see the pass messages, set the
>> loglevel to allow debug output.
>>
>> Unfortunately this can add lots of debug output, unless you use dynamic
>> debug to only enable debug for drivers/of/unittest.o.  There are only
>> a few other pr_debug() messages in unittest.
>>
>> I think a better solution would be to add a config option, something
>> like CONFIG_OF_UNITTEST_VERBOSE, that would print the pass messages
>> at loglevel KERN_ERR.  I'll submit a patch for that and see what the
>> review responses are.
>>
>>> We would like to get your opinion of how hard it would be to include
>>> that in the output per test. Maybe like TAP version 14?
>>> Another question would be how hard do you think it would be to rewrite
>>> this to a kunit test, if even applicable? I have provided the kunit
>>> output links at the end of this email.
>>
>> Devicetree unittests were suggested as a good candidate as a first
>> test to convert to kunit when kunit was implemented.  Brendan tried
>> to convert it, and we quickly saw that it was not a good candidate.
>> Devicetree unittests do not fit the unit test mold; they are a very
>> different creature.  Brendan has a good term for this type of test
>> (Brendan, was it "acceptance" test?).
> 
> I understood that it was either an integration test or end-to-end test
> (probably an integration test): https://lkml.org/lkml/2019/3/21/1124

Yes, thanks.  Those are the terms I was trying to remember.

-Frank

> 
> Standardizing integration tests in the kernel is still something that
> hasn't happened yet, but there are some examples of integration tests
> being written in KUnit (the KASAN KUnit test is probably the most
> notable example). There are definitely some others written in
> kselftest. It's kind of a tough area because integration tests are
> kind of defined by being in between unit tests and end-to-end tests.
> 
>>> Test output:
>>> ------------
>>> [    0.000000] Booting Linux on physical CPU 0x0000000100 [0x410fd033]
>>> [    0.000000] Linux version 5.17.0-rc1-next-20220127
>>> (tuxmake@tuxmake) (aarch64-linux-gnu-gcc (Debian 11.2.0-9) 11.2.0, GNU
>>> ld (GNU Binutils for Debian) 2.37) #1 SMP PREEMPT @1643255563
>>> [    0.000000] Machine model: ARM Juno development board (r2)
>>>
>>> <trimmed output>
>>>
>>> [    3.285226] ### dt-test ### start of unittest - you will see error messages
>>> [    3.293269] ### dt-test ### EXPECT \ : Duplicate name in
>>> testcase-data, renamed to \"duplicate-name#1\"
>>> [    3.293456] Duplicate name in testcase-data, renamed to \"duplicate-name#1\"
>>> [    3.313367] ### dt-test ### EXPECT / : Duplicate name in
>>> testcase-data, renamed to \"duplicate-name#1\"
>>> [    3.314709] ### dt-test ### EXPECT \ : OF:
>>> /testcase-data/phandle-tests/consumer-a: could not get
>>> #phandle-cells-missing for /testcase-data/phandle-tests/provider1
>>> [    3.323968] OF: /testcase-data/phandle-tests/consumer-a: could not
>>> get #phandle-cells-missing for /testcase-data/phandle-tests/provider1
>>>
>>> <trimmed output>
>>>
>>> [    5.118400] ### dt-test ### EXPECT / : OF: overlay: ERROR: multiple
>>> fragments add and/or delete node
>>> /testcase-data-2/substation@100/motor-1/electric
>>> [    5.121358] atkbd serio1: keyboard reset failed on 1c070000.kmi
>>> [    5.134160] ### dt-test ### end of unittest - 257 passed, 0 failed
>>>
>>>
>>> Ref:
>>> Full test output of of-unittest
>>> https://lkft.validation.linaro.org/scheduler/job/4458582#L1019
>>> https://lkft.validation.linaro.org/scheduler/job/4404330#L428
>>>
>>> Kunit example test output that we are running in our daily CI loop.
>>> https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-5.10.y/build/v5.10.70/testrun/5965109/suite/kunit/tests/
>>>
>>> Kunit Full test logs:
>>> https://lkft.validation.linaro.org/scheduler/job/3643324
>>>
>>> https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-5.10.y/build/v5.10.70/testrun/5965109/suite/kunit/test/kunit_log_test/log
>>>
>>>
>>> --
>>> Linaro LKFT
>>> https://lkft.linaro.org
>>>
>>