Some questions about KernelCI

Some questions about KernelCI

[Dmitry Vyukov]

Hello KernelCI-devs,

I have some questions regarding KernelCI to better understand state of
upstream kernel testing. It's more about missing parts rather than
present parts, so sorry if it all sounds a bit negative.

1. Do I understand it correctly that KernelCI is still mostly about
building/booting, it started running tests some time ago but few tests
are currently onboarded? Is there a full list of tests? I looked at
lava-kernel-tests repo, but on the dashboard I also seen "networking"
tests, so I am not sure if lava-kernel-tests is the full list. There
are plans to onboard more tests in the future, right?

2. Are there test coverage reports anywhere? Is it in plans?

3. Does KernelCI use any of KASAN, LOCKDEP, KMEMLEAK, FAULT_INJECTION,
DEBUG_OBJECTS, DEBUG_VM runtime debugging features?

4. Do I understand it correctly that test failures are currently not
reported in any way (other than being shown on the dashboard)? Who
looks at the dashboard? Is there constant monitoring of all results?
Is automatic reporting in plans?

5. There is no pre-commit testing, right?

6. What are the major TODO items for near/mid-term future?

7. What is rough approximation of total engineer-years spent on the
project? What are the major time sinks? Pain points?

Thanks in advance

Re: Some questions about KernelCI

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2890 bytes --]

On Wed, Aug 28, 2019 at 02:03:22PM -0700,  via Groups.Io wrote:

Just responding for me, this hasn't been reviewed by anyone else.

> 1. Do I understand it correctly that KernelCI is still mostly about
> building/booting, it started running tests some time ago but few tests
> are currently onboarded? Is there a full list of tests? I looked at
> lava-kernel-tests repo, but on the dashboard I also seen "networking"
> tests, so I am not sure if lava-kernel-tests is the full list. There
> are plans to onboard more tests in the future, right?

There's also some of the v4l tests running.  The plan is to get the UI
working well and figure out what we're doing in terms of hardware to run
things on before expanding testsuite coverage.  I'll let the people
working on that provide more detail on how that's going.

> 2. Are there test coverage reports anywhere? Is it in plans?

There's some e-mail reports for tests which are sent to developers at
the minute, the web UI is being looked at as well since as you've
noticed it's currently not great.

> 3. Does KernelCI use any of KASAN, LOCKDEP, KMEMLEAK, FAULT_INJECTION,
> DEBUG_OBJECTS, DEBUG_VM runtime debugging features?

Not at the current time except in so far as they appear in defconfigs,
we do talk about it sometimes - some of them would even be useful during
boot testing - but we've had resource issues in the past with adding the
extra configs.

> 4. Do I understand it correctly that test failures are currently not
> reported in any way (other than being shown on the dashboard)? Who
> looks at the dashboard? Is there constant monitoring of all results?
> Is automatic reporting in plans?

The test support is pretty much in development at the moment, until
we've got the infrastructure working well we're not really pushing on
reporting things out too much.

> 5. There is no pre-commit testing, right?

Right.  It's a feature that gets talked about from time to time but
nobody's stepped forward to implement it and we'd need to figure out
resources to run the tests as well.

> 6. What are the major TODO items for near/mid-term future?

There's a bunch of issues in github:

	https://github.com/kernelci

but really a lot of it is about people scratching their own itches, for
example I want to do some work on the e-mail build reports since they're
not very usable right now but I've never got round to filing an issue
for it.

> 7. What is rough approximation of total engineer-years spent on the
> project? What are the major time sinks? Pain points?

Finger in the air I'd guess more than 8, possibly quite a bit - the
project has been going for a bit over 5 years, we've had periods where
there's been multiple people working on it pretty much full time but
also periods where it's been mostly in maintanence mode.

For me the biggest specific pain point right now is front end developers
able to work on the UI.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Some questions about KernelCI

[Dmitry Vyukov]

On Thu, Aug 29, 2019 at 5:14 AM Mark Brown <broonie@kernel.org> wrote:
>
> On Wed, Aug 28, 2019 at 02:03:22PM -0700,  via Groups.Io wrote:
>
> Just responding for me, this hasn't been reviewed by anyone else.

Hi Mark,

Thanks for providing info!
2 comments below.

> > 1. Do I understand it correctly that KernelCI is still mostly about
> > building/booting, it started running tests some time ago but few tests
> > are currently onboarded? Is there a full list of tests? I looked at
> > lava-kernel-tests repo, but on the dashboard I also seen "networking"
> > tests, so I am not sure if lava-kernel-tests is the full list. There
> > are plans to onboard more tests in the future, right?
>
> There's also some of the v4l tests running.  The plan is to get the UI
> working well and figure out what we're doing in terms of hardware to run
> things on before expanding testsuite coverage.  I'll let the people
> working on that provide more detail on how that's going.
>
> > 2. Are there test coverage reports anywhere? Is it in plans?
>
> There's some e-mail reports for tests which are sent to developers at
> the minute, the web UI is being looked at as well since as you've
> noticed it's currently not great.

I meant code coverage reports/stats, which show what's being tested
and what's not tested.


> > 3. Does KernelCI use any of KASAN, LOCKDEP, KMEMLEAK, FAULT_INJECTION,
> > DEBUG_OBJECTS, DEBUG_VM runtime debugging features?
>
> Not at the current time except in so far as they appear in defconfigs,
> we do talk about it sometimes - some of them would even be useful during
> boot testing - but we've had resource issues in the past with adding the
> extra configs.
>
> > 4. Do I understand it correctly that test failures are currently not
> > reported in any way (other than being shown on the dashboard)? Who
> > looks at the dashboard? Is there constant monitoring of all results?
> > Is automatic reporting in plans?
>
> The test support is pretty much in development at the moment, until
> we've got the infrastructure working well we're not really pushing on
> reporting things out too much.

This equally applies to build and boot testing too. If code does not
build, this needs to be reported. How does this happen?

> > 5. There is no pre-commit testing, right?
>
> Right.  It's a feature that gets talked about from time to time but
> nobody's stepped forward to implement it and we'd need to figure out
> resources to run the tests as well.
>
> > 6. What are the major TODO items for near/mid-term future?
>
> There's a bunch of issues in github:
>
>         https://github.com/kernelci
>
> but really a lot of it is about people scratching their own itches, for
> example I want to do some work on the e-mail build reports since they're
> not very usable right now but I've never got round to filing an issue
> for it.
>
> > 7. What is rough approximation of total engineer-years spent on the
> > project? What are the major time sinks? Pain points?
>
> Finger in the air I'd guess more than 8, possibly quite a bit - the
> project has been going for a bit over 5 years, we've had periods where
> there's been multiple people working on it pretty much full time but
> also periods where it's been mostly in maintanence mode.
>
> For me the biggest specific pain point right now is front end developers
> able to work on the UI.

Re: Some questions about KernelCI

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1388 bytes --]

On Fri, Aug 30, 2019 at 12:07:47PM -0700, Dmitry Vyukov wrote:
> On Thu, Aug 29, 2019 at 5:14 AM Mark Brown <broonie@kernel.org> wrote:

> > > 2. Are there test coverage reports anywhere? Is it in plans?

> > There's some e-mail reports for tests which are sent to developers at
> > the minute, the web UI is being looked at as well since as you've
> > noticed it's currently not great.

> I meant code coverage reports/stats, which show what's being tested
> and what's not tested.

No, I don't think that's something that's ever done for the kernel in
any sustained fashion.  

> > > 4. Do I understand it correctly that test failures are currently not
> > > reported in any way (other than being shown on the dashboard)? Who
> > > looks at the dashboard? Is there constant monitoring of all results?
> > > Is automatic reporting in plans?

> > The test support is pretty much in development at the moment, until
> > we've got the infrastructure working well we're not really pushing on
> > reporting things out too much.

> This equally applies to build and boot testing too. If code does not
> build, this needs to be reported. How does this happen?

The build results are visible via the web site but the main thing for
KernelCI and all the other build testers is that people do some basic
triage and forward on the relevant bits of the e-mail reports.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Some questions about KernelCI

[zhengruoqin]

Hello KernelCI-devs:

    I am a rookie to kernelci.
    I have already installed kernelci-docker and LAVA.

    I have investigated the kernelci-wiki and can't got the usage of kernelci-core, and I don't know how to connect lava to kernelci.

    Could anyone give some hints or link?

Thanks

--------------------------------------------------
Zheng Ruoqin
Nanjing Fujitsu Nanda Software Tech. Co., Ltd.(FNST)
ADDR.: No.6 Wenzhu Road, Software Avenue,
       Nanjing, 210012, China
MAIL : zhengrq.fnst@cn.fujistu.com