Re: #KCIDB engagement report

[Kees Cook <keescook@chromium.org>]

On Tue, Jun 15, 2021 at 11:23:35PM +0100, Guillaume Tucker wrote:
> +alex 
> 
> On 15/06/2021 23:03, Kees Cook wrote:
> > On Fri, Jun 11, 2021 at 05:11:59PM +0100, Guillaume Tucker wrote:
> >> Hi Kees,
> >>
> >> On 01/06/2021 21:26, Kees Cook wrote:
> >>> On Mon, May 24, 2021 at 10:38:22AM -0700, 'Nick Desaulniers' via Clang Built Linux wrote:
> >>>> On Mon, May 24, 2021 at 12:50 AM Nikolai Kondrashov
> >>>> <Nikolai.Kondrashov@redhat.com> wrote:
> >>>>> [...]
> >>>>>      KernelCI native
> >>>>>          Sending (a lot of) production build and test results.
> >>>>>          https://staging.kernelci.org:3000/?var-origin=kernelci
> >>>>> [...]
> >>>
> >>> Apologies for the thread hijack, but does anyone know what's happening
> >>> with kselftest? It seems missing from the listed[1] build artifacts, but
> >>> it is actually present[2] (and I see the logs for generating the tarball
> >>> there too), but I can't find any builds that actually run the tests?
> >>>
> >>> (Or how do I see a top-level list of all tests and search it?)
> >>
> >> The kselftest results are all there on the KernelCI native
> >> dashboard, for example the futex tests:
> >>
> >>   https://linux.kernelci.org/test/job/mainline/branch/master/kernel/v5.13-rc5-74-g06af8679449d/plan/kselftest-futex/
> > 
> > Thanks for looking at this for me! :)
> > 
> > How do I find the other kselftest stuff? I just see "kselftest-futex"
> > and "kselftest-filesystem". I was expecting _all_ of the kselftests, but
> > I can't find them.
> > 
> > (Specifically, I can't find a top-level "list of all test plans")
> 
> That's because kselftest is rather large, and we're only enabling
> subsets of it one at a time.  As more test labs and more devices

Ah-ha! Okay.

> become available, we'll gradually expand coverage.  We might also
> choose to have full coverage only on say, linux-next, mainline
> and LTS branches but not everywhere to not overload the labs.

Doing this for -next, mainline, and LTS would be extremely helpful for
me, though I suppose I mostly only care about the lkdtm, seccomp, and
pstore tests.

> To answer your question about "all the tests", well you can look
> at any kernel revision to see the tests that were run for it
> since it won't be the same for all of them.  Typically,
> linux-next has the highest number of tests so here's an example:
> 
>   https://linux.kernelci.org/test/job/next/branch/master/kernel/next-20210615/

Right, that's helpful, but I need to know which kernel to test. It'd be
nice to have a top-level "all the tests", and for each test, it should
list the kernels that run those tests, etc.

> As you've already found, there are only 3 kselftest subsets
> or "collections" being run there at the moment.  That's by design
> in the KernelCI configuration, but at least we have good enough
> support for running kselftest now which wasn't completely
> trivial to put in place...

Right, totally understood. I spent time tweaking those pieces too. :)

> There are still a few issues to fix, but I would expect kselftest
> coverage to keep growing over the coming weeks.
> 
> If there are kselftest collections you really want to have
> enabled, you can always make a PR to add them to this file:
> 
>   https://github.com/kernelci/kernelci-core/blob/main/config/core/test-configs.yaml#L187
> 
> As long as there's capacity for it at least on some types of
> devices and it runs as expected, we should be able to get this
> deployed in production pretty easily.

Awesome. I will do so immediately. :)

Thanks!

-Kees

-- 
Kees Cook