Testing Quality Call notes - 2024-06-13

[Laura Nao <laura.nao@collabora.com>]

Hello,

KernelCI is hosting a bi-weekly call on Thursday to discuss improvements
to existing upstream tests, the development of new tests to increase 
kernel testing coverage, and the enablement of these tests in KernelCI. 
In recent months, we at Collabora have focused on various kernel areas, 
assessing the tests already available upstream and contributing patches 
to make them easily runnable in CIs.

Below is a list of the tests we've been working on and their latest 
status updates, as discussed in the last meeting held on 2024-06-13:

*ACPI probe kselftest*

- Proposing new kselftest to detect unprobed devices on ACPI platforms:
  https://lore.kernel.org/all/20240308144933.337107-1-laura.nao@collabora.com
- Regression on acpi_fan driver detected and fixed upstream after
  preliminary testing in KernelCI. Sent follow-up on RFCv2. 

*USB/PCI devices kselftest*

- Upstream test to detect unprobed devices on discoverable buses:
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dacf1d7a78bf8a131346c47bfba7fe1f3ff44beb 
- Preparing to enable the test in KernelCI on MediaTek Chromebooks in the
  Collabora LAVA lab - see: https://github.com/kernelci/kernelci-pipeline/issues/579

*Watchdog kselftest*

- Adding KTAP output and limiting the ping loop to run the test 
  individually in CIs:  https://lore.kernel.org/all/20240506111359.224579-1-laura.nao@collabora.com
- Received feedback on series, discussion ongoing

*Watchdog reset test*

- Implementing out-of-tree test in KernelCI to detect device reset after 
  watchdog timeout expiration: 
  https://github.com/kernelci/kernelci-pipeline/issues/608
- Enabled test in KernelCI for all ARM64 and AMD Chromebooks, currently 
  monitoring the results

*Error log test*

- Proposing new kselftest to report device log errors:
  https://lore.kernel.org/all/20240423-dev-err-log-selftest-v1-0-690c1741d68b@collabora.com/
- Standardizing error log format suggested for easier parsing, including 
  for kernel panic/oops/warning messages.

*Suspend/resume in cpufreq kselftest*

- Enabling suspend/resume test within the cpufreq kselftest in KernelCI
- Adding parameter support to kselftest script in KernelCI:
  https://github.com/Linaro/test-definitions/pull/511 
- Planning to add support for rtcwake in the test, to enable automated
  resume 
- Considering measuring latency for suspend and resume processes

*TAP conformance in kselftests*

- Ongoing improvements to KVM selftests, discussion ongoing upstream
  https://lore.kernel.org/all/20240422170842.2073979-1-usama.anjum@collabora.com/
  https://lore.kernel.org/all/20240612104500.425012-1-usama.anjum@collabora.com/
- Enabling more kselftests in KernelCI  (dmabuf-heaps, exec, and iommu): 
  https://github.com/kernelci/kernelci-pipeline/pull/640/

*Boot time test*

- Investigating possibility of adding new test upstream to measure the 
  kernel boot time and detect regressions 
- Need to investigate available interfaces for reporting boot/probe times
- Multiple measurement points may be necessary, need to establish a clear 
  definition of "boot" first
- Influenced by defconfig; testing with various kernel configurations 
  recommended

*Possible improvements to kselftest documentation*

- Document tests excluded from default kselftest run and parameter 
  support

*Test configuration reporting in KernelCI*

- Discussing reporting more details on the test configuration used:
  https://github.com/kernelci/kernelci-pipeline/issues/636

Please reply to this thread if you'd like to join the call or discuss 
any of the topics further. We look forward to collaborating with the 
community to improve upstream tests and expand coverage to more areas 
of interest within the kernel.

Best regards,

Laura Nao