Testing Quality Call notes - 2024-06-13

Testing Quality Call notes - 2024-06-13

[Laura Nao]

Hello,

KernelCI is hosting a bi-weekly call on Thursday to discuss improvements
to existing upstream tests, the development of new tests to increase 
kernel testing coverage, and the enablement of these tests in KernelCI. 
In recent months, we at Collabora have focused on various kernel areas, 
assessing the tests already available upstream and contributing patches 
to make them easily runnable in CIs.

Below is a list of the tests we've been working on and their latest 
status updates, as discussed in the last meeting held on 2024-06-13:

*ACPI probe kselftest*

- Proposing new kselftest to detect unprobed devices on ACPI platforms:
  https://lore.kernel.org/all/20240308144933.337107-1-laura.nao@collabora.com
- Regression on acpi_fan driver detected and fixed upstream after
  preliminary testing in KernelCI. Sent follow-up on RFCv2. 

*USB/PCI devices kselftest*

- Upstream test to detect unprobed devices on discoverable buses:
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dacf1d7a78bf8a131346c47bfba7fe1f3ff44beb 
- Preparing to enable the test in KernelCI on MediaTek Chromebooks in the
  Collabora LAVA lab - see: https://github.com/kernelci/kernelci-pipeline/issues/579

*Watchdog kselftest*

- Adding KTAP output and limiting the ping loop to run the test 
  individually in CIs:  https://lore.kernel.org/all/20240506111359.224579-1-laura.nao@collabora.com
- Received feedback on series, discussion ongoing

*Watchdog reset test*

- Implementing out-of-tree test in KernelCI to detect device reset after 
  watchdog timeout expiration: 
  https://github.com/kernelci/kernelci-pipeline/issues/608
- Enabled test in KernelCI for all ARM64 and AMD Chromebooks, currently 
  monitoring the results

*Error log test*

- Proposing new kselftest to report device log errors:
  https://lore.kernel.org/all/20240423-dev-err-log-selftest-v1-0-690c1741d68b@collabora.com/
- Standardizing error log format suggested for easier parsing, including 
  for kernel panic/oops/warning messages.

*Suspend/resume in cpufreq kselftest*

- Enabling suspend/resume test within the cpufreq kselftest in KernelCI
- Adding parameter support to kselftest script in KernelCI:
  https://github.com/Linaro/test-definitions/pull/511 
- Planning to add support for rtcwake in the test, to enable automated
  resume 
- Considering measuring latency for suspend and resume processes

*TAP conformance in kselftests*

- Ongoing improvements to KVM selftests, discussion ongoing upstream
  https://lore.kernel.org/all/20240422170842.2073979-1-usama.anjum@collabora.com/
  https://lore.kernel.org/all/20240612104500.425012-1-usama.anjum@collabora.com/
- Enabling more kselftests in KernelCI  (dmabuf-heaps, exec, and iommu): 
  https://github.com/kernelci/kernelci-pipeline/pull/640/

*Boot time test*

- Investigating possibility of adding new test upstream to measure the 
  kernel boot time and detect regressions 
- Need to investigate available interfaces for reporting boot/probe times
- Multiple measurement points may be necessary, need to establish a clear 
  definition of "boot" first
- Influenced by defconfig; testing with various kernel configurations 
  recommended

*Possible improvements to kselftest documentation*

- Document tests excluded from default kselftest run and parameter 
  support

*Test configuration reporting in KernelCI*

- Discussing reporting more details on the test configuration used:
  https://github.com/kernelci/kernelci-pipeline/issues/636

Please reply to this thread if you'd like to join the call or discuss 
any of the topics further. We look forward to collaborating with the 
community to improve upstream tests and expand coverage to more areas 
of interest within the kernel.

Best regards,

Laura Nao

RE: Testing Quality Call notes - 2024-06-13

[Bird, Tim]

> -----Original Message-----
> From: Laura Nao <laura.nao@collabora.com>
> 
> KernelCI is hosting a bi-weekly call on Thursday to discuss improvements
> to existing upstream tests, the development of new tests to increase
> kernel testing coverage, and the enablement of these tests in KernelCI.

This is interesting.  Is it possible for me to join this bi-weekly call?
If so, can you send me info on how to join?

> In recent months, we at Collabora have focused on various kernel areas,
> assessing the tests already available upstream and contributing patches
> to make them easily runnable in CIs.
> 
...

> 
> *Boot time test*
> 
> - Investigating possibility of adding new test upstream to measure the
>   kernel boot time and detect regressions
> - Need to investigate available interfaces for reporting boot/probe times
> - Multiple measurement points may be necessary, need to establish a clear
>   definition of "boot" first
> - Influenced by defconfig; testing with various kernel configurations
>   recommended

I'm extremely interested in this.  I was planning on creating some boot time instrumentation
and a boot time test, this summer (before LPC in September).  I would be
thrilled to talk to other developers about this (either by e-mail, in your bi-weekly
call, or at Plumbers) before I get too far into it.

If others are working on measuring, testing,  or reducing boot time, please
consider joining the "Embedded and IOT" micro-conference at Plumbers in September.

One of the chief topics of that micro-conference will be boot time.  Please contact me
and I can let you know instructions for submitting a topic proposal for the event.
(Or just submit something at: https://lpc.events/event/18/abstracts/)
The instrumentation and testing of boot time would be a great topic for the
micro-conference!!

 -- Tim

RE: Testing Quality Call notes - 2024-06-13

[Laura Nao]

Hi Tim,

On 6/18/24 19:29, Bird, Tim wrote:
>> -----Original Message-----
>> From: Laura Nao <laura.nao@collabora.com>
>>
>> KernelCI is hosting a bi-weekly call on Thursday to discuss improvements
>> to existing upstream tests, the development of new tests to increase
>> kernel testing coverage, and the enablement of these tests in KernelCI.
> 
> This is interesting.  Is it possible for me to join this bi-weekly call?
> If so, can you send me info on how to join?
> 

We have sent an invitation to the meeting, let us know if you 
didn't receive it and we'll resend it. Please note that the meeting is 
currently scheduled for 13:00 UTC, which unfortunately is not very 
PDT-friendly. 

>> In recent months, we at Collabora have focused on various kernel areas,
>> assessing the tests already available upstream and contributing patches
>> to make them easily runnable in CIs.
>>
> ...
> 
>>
>> *Boot time test*
>>
>> - Investigating possibility of adding new test upstream to measure the
>>    kernel boot time and detect regressions
>> - Need to investigate available interfaces for reporting boot/probe times
>> - Multiple measurement points may be necessary, need to establish a clear
>>    definition of "boot" first
>> - Influenced by defconfig; testing with various kernel configurations
>>    recommended
> 
> I'm extremely interested in this.  I was planning on creating some boot time instrumentation
> and a boot time test, this summer (before LPC in September).  I would be
> thrilled to talk to other developers about this (either by e-mail, in your bi-weekly
> call, or at Plumbers) before I get too far into it.
> 

That's great to hear! We're just beginning our investigation by reviewing 
the available interfaces and assessing how to use them in a potential 
in-tree test. We would be happy to discuss this further during the 
testing quality meeting on June 27th, or sooner via email. We'll reach 
out once we have some ideas to propose and discuss.

> If others are working on measuring, testing,  or reducing boot time, please
> consider joining the "Embedded and IOT" micro-conference at Plumbers in September.
> 
> One of the chief topics of that micro-conference will be boot time.  Please contact me
> and I can let you know instructions for submitting a topic proposal for the event.
> (Or just submit something at: https://lpc.events/event/18/abstracts/)
> The instrumentation and testing of boot time would be a great topic for the
> micro-conference!!
> 
>   -- Tim
> 

Thanks for the heads up, we'll keep this in mind!

Best,

Laura