Re: LKFT CI: improving Networking selftests results when validating stable kernels

[Matthieu Baerts <matttbe@kernel.org>]

Hi Dan,

Thank you for your reply!

On 13/11/2024 18:08, Dan Carpenter wrote:
> On Fri, Nov 08, 2024 at 07:21:59PM +0100, Matthieu Baerts wrote:
>> KSelftests from the same version
>> --------------------------------
>>
>> According to the doc [2], kselftests should support all previous kernel
>> versions. The LKFT CI is then using the kselftests from the last stable
>> release to validate all stable versions. Even if there are good reasons
>> to do that, we would like to ask for an opt-out for this policy for the
>> networking tests: this is hard to maintain with the increased
>> complexity, hard to validate on all stable kernels before applying
>> patches, and hard to put in place in some situations. As a result, many
>> tests are failing on older kernels, and it looks like it is a lot of
>> work to support older kernels, and to maintain this.
>>
>> Many networking tests are validating the internal behaviour that is not
>> exposed to the userspace. A typical example: some tests look at the raw
>> packets being exchanged during a test, and this behaviour can change
>> without modifying how the userspace is interacting with the kernel. The
>> kernel could expose capabilities, but that's not something that seems
>> natural to put in place for internal behaviours that are not exposed to
>> end users. Maybe workarounds could be used, e.g. looking at kernel
>> symbols, etc. Nut that doesn't always work, increase the complexity, and
>> often "false positive" issue will be noticed only after a patch hits
>> stable, and will cause a bunch of tests to be ignored.
>>
>> Regarding fixes, ideally they will come with a new or modified test that
>> can also be backported. So the coverage can continue to grow in stable
>> versions too.
>>
>> Do you think that from the kernel v6.12 (or before?), the LKFT CI could
>> run the networking kselftests from the version that is being validated,
>> and not from a newer one? So validating the selftests from v6.12.1 on a
>> v6.12.1, and not the ones from a future v6.16.y on a v6.12.42.
>>
> 
> These kinds of decisions are something that Greg and Shuah need to decide on.

Thank you, it makes sense.

> You would still need some way to automatically detect that kselftest is running
> on an old kernel and disable the networking checks.  Otherwise when random
> people on the internet try to run selftests they would run into issues.


Indeed. I guess we can always add a warning when the kernel and
selftests versions are different. I suppose the selftests are built
using the same kernel version, then executed on older versions: we could
then compare the kernel versions at build time and run time, no?


Regarding the other questions from my previous email -- skipped tests
(e.g. I think Netfilter tests are no longer validated), KVM,
notifications -- do you know who at Linaro could eventually look at them?


Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.