[Ksummit-discuss] Fwd: Rough notes from testing unconference

[Ksummit-discuss] Fwd: Rough notes from testing unconference

[Grant Likely]

Rough notes from the Testing discussion during the workshop day of
Kernel Summit. Big thanks to Paul McKenney for writing all of this
down.

g.

Testing session (unconference)

o       In-kernel or out of kernel?  Bisection?  Can do either way,
        one approach is to have two clones of the git tree, and another
        is to "git checkout" a specific version of the tools directory.

o       Levels of tests:  1) get a login prompt, 2) kselftest.

o       Should we have specified userspace?  No, quickly gets into
        the distribution-building business.

o       Will Deacon: Can script using busybox, inittab, and so on.
        Can run trinity on such a setup.

o       Ted: Would be nice if other test frameworks built on top of
        kselftest.  Therefore kselftest should not make any asssumptions.

o       Shua: Goal of kselftest isn't to create something new, but
        rather to gather up what we already have.

o       Ted: Much of the code in tools/testing/selftest is unit tests
        for a tiny part of the kernel.  These don't really belong
        in LTP.

o       Olof: Let's not reinvent the test-infrastructure wheel.

o       Aneesh: Autotest.  Others: Lots of such tests, but lots of
        different ones that are relatively difficult to set up.

o       Paul: Don't need rootfs.  rcutorture runs out of initrd.

o       Olof: Often easier just to write a script.

o       Ted: If we are going to lightweight, need to focus on kernel
        unit tests.

o       What kind of tool can we depend on?  Busybox?  Just busybox,
        has everything it needs.  (This means scripting must be in
        dash.)

o       Arnd: klibc has its own set of binaries, cannot support
        busybox.  Need libc.

o       Josh: Only C and standard C library supported.

o       Paul:  Host or guest?

o       Discussion: apparently need both.  Assume full-fledged distro
        on host, minimal on guest/target.  UART, retrieve a file.
        Some targets don't have persistent mass storage.

o       Grant:  Needs to work on separate hardware, on qemu, on
        fast models.

o       Need stdio output, input optional (some tests take input
        from kernel boot parameters.)

o       How to parse success or failure?  How to know what to ignore?
        Ted: What are high-level requirements rather than low-level
        implementation?  Need host to easily parse the information
        from the target on the host.

o       Mauro: Need something for performance tests as well.

o       Ted: Group the tests.  "Quick" group.  Groups associated with
        given function (e.g., suspend-resume).  Have profiles based
        on types of tests needed and time allotted.

o       Josh: Add test types to the MAINTAINERS file.

o       Tim: Requirements around cross-building?  People make their
        own?  We supply some in-tree?  We supply pointers to known
        good cross-build toolchains?  Grant: "Yes".

o       Tim:  The need is to enable people who have never cross-built
        in their life to successfully cross-build and test with
        minimal effort.

Next steps:  Grant to send call for volunteers for various efforts
        to ksummit-discuss, interested people to reply.

o       Josh: Can we pick an existing test output format that already
        has tools, infrastructure, and so on.

        Tim: Hudson!

        Andy: PASS, FAIL, XFAIL, ...

        Shua: Must fit into the primary goal of being a good kernel
        developer unit test.

        Ted: Proposed requirement: Simplest use case must have simple
        tool that parses output.  Must not include a JRE for output
        parsing.

        Tim: Must be human reasable.

        Andy: Must be short boilerplate, so that it can be generated
        and read quickly.

        Paul: Willing to make reasonable adaptations to rcutorture output.
        But it must not require an in-kernel JRE.

        Arnd: Would like self-test entry for drivers.

        Ben: Have seen similar things in other operating systems, would
        be good in Linux.  Mauro: Would like to separate hardware and
        software tests for drivers.   Arnd: Could be separate profile.

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Shuah Khan]

On Fri, Aug 22, 2014 at 8:19 AM, Grant Likely <grant.likely@secretlab.ca> wrote:
> Rough notes from the Testing discussion during the workshop day of
> Kernel Summit. Big thanks to Paul McKenney for writing all of this
> down.
>

Grant,

You beat me to it. Thanks for sending the notes from the discussion and thanks
Paul McKenney for writing these down.

The next steps on my plate are as follows:

Shuah Khan:

- Work on putting a basic framework for tests: Pass, Fail, Skip to start with.
- update existing tests in tools/testing/selftests to use the
framework to serve as an
  example for future tests. As we discussed and decided on output will
continue to go to
  stdio. This will help these tests being used in other scripts that
run tests in cross
  environments.
- Define profiles for short and longer time tests
- Add install target in addition to the existing kselftest target at
the top level Makefile

I will maintain the test framework - will take care of the logistics
of MAINTAINER
file updates for this and creating a git tree. Individual test
maintainers (we have
several in MAINTAINERS file) will continue to own and maintain their tests.

Fixes to known cross-build failures and new tests are welcome. Please
send patches to linux-api mailing list

These are the initial tasks to continue this project in 3.18 and we
can evolve and refine the
steps as we go. Please add and/or correct if I missed any initial
steps and got the details wrong.

thanks,
-- Shuah

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Amit Kucheria]

On Fri, Aug 22, 2014 at 11:29 PM, Shuah Khan <shuahkhan@gmail.com> wrote:
> Shuah Khan:
>
> - Work on putting a basic framework for tests: Pass, Fail, Skip to start with.
> - update existing tests in tools/testing/selftests to use the
> framework to serve as an
>   example for future tests. As we discussed and decided on output will
> continue to go to
>   stdio. This will help these tests being used in other scripts that
> run tests in cross
>   environments.
> - Define profiles for short and longer time tests
> - Add install target in addition to the existing kselftest target at
> the top level Makefile
>
> I will maintain the test framework - will take care of the logistics
> of MAINTAINER
> file updates for this and creating a git tree. Individual test
> maintainers (we have
> several in MAINTAINERS file) will continue to own and maintain their tests.
>
> Fixes to known cross-build failures and new tests are welcome. Please
> send patches to linux-api mailing list
>

Hi Shuah,

At the Linaro power management working group, we've created a simple
repository of pass/fail *functional* tests for various areas of
interest in power management - cpuidle, cpufreq, hotplug. We call it
PM-QA[1]

Some of our members have found PM-QA useful to exercise the various
sysfs interfaces and get rid of the obvious issues (missing Kconfig
options, basic exercising of cpufreq drivers being a couple of
examples).

Let me know if these are useful and in the right format to seed your
library of testcases.

Regards,
Amit
[1] https://git.linaro.org/power/pm-qa.git

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Shuah Khan]

On Fri, Aug 22, 2014 at 12:08 PM, Amit Kucheria
<amit.kucheria@linaro.org> wrote:

> Hi Shuah,
>
> At the Linaro power management working group, we've created a simple
> repository of pass/fail *functional* tests for various areas of
> interest in power management - cpuidle, cpufreq, hotplug. We call it
> PM-QA[1]
>
> Some of our members have found PM-QA useful to exercise the various
> sysfs interfaces and get rid of the obvious issues (missing Kconfig
> options, basic exercising of cpufreq drivers being a couple of
> examples).
>
> Let me know if these are useful and in the right format to seed your
> library of testcases.
>
> Regards,
> Amit
> [1] https://git.linaro.org/power/pm-qa.git

Hi Amit,

Thanks for sharing these. I will take a look and see if these make
good candidates for selftests.

-- Shuah

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Bird, Tim]

On Friday, August 22, 2014 10:59 AM, Shuah Khan  wrote:
>
> On Fri, Aug 22, 2014 at 8:19 AM, Grant Likely <grant.likely@secretlab.ca> wrote:
> > Rough notes from the Testing discussion during the workshop day of
> > Kernel Summit. Big thanks to Paul McKenney for writing all of this
> > down.
> >
>
> Grant,
>
> You beat me to it. Thanks for sending the notes from the discussion and thanks
> Paul McKenney for writing these down.
>
> The next steps on my plate are as follows:
>
> Shuah Khan:
>
> - Work on putting a basic framework for tests: Pass, Fail, Skip to start with.
> - update existing tests in tools/testing/selftests to use the
> framework to serve as an
>   example for future tests. As we discussed and decided on output will
> continue to go to
>   stdio. This will help these tests being used in other scripts that
> run tests in cross
>   environments.
> - Define profiles for short and longer time tests
> - Add install target in addition to the existing kselftest target at
> the top level Makefile
>
> I will maintain the test framework - will take care of the logistics
> of MAINTAINER
> file updates for this and creating a git tree. Individual test
> maintainers (we have
> several in MAINTAINERS file) will continue to own and maintain their tests.
>
> Fixes to known cross-build failures and new tests are welcome. Please
> send patches to linux-api mailing list

I've been working on integration with Aboriginal (Rob Landley's 
minimal cross-compile distro) and hope to have some results
very soon.

Also, I've requested a 'test' wiki on kernel.org, where we can place 
notes, ideas, and lists of things to work on, or things in progress
(like possible output format guidelines).

I'll let everyone know when the wiki is set up.

Thanks,
 -- Tim

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Grant Likely]

On Sat, Aug 23, 2014 at 8:35 AM, Bird, Tim <Tim.Bird@sonymobile.com> wrote:
> On Friday, August 22, 2014 10:59 AM, Shuah Khan  wrote:
>>
>> On Fri, Aug 22, 2014 at 8:19 AM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> > Rough notes from the Testing discussion during the workshop day of
>> > Kernel Summit. Big thanks to Paul McKenney for writing all of this
>> > down.
>> >
>>
>> Grant,
>>
>> You beat me to it. Thanks for sending the notes from the discussion and thanks
>> Paul McKenney for writing these down.
>>
>> The next steps on my plate are as follows:
>>
>> Shuah Khan:
>>
>> - Work on putting a basic framework for tests: Pass, Fail, Skip to start with.
>> - update existing tests in tools/testing/selftests to use the
>> framework to serve as an
>>   example for future tests. As we discussed and decided on output will
>> continue to go to
>>   stdio. This will help these tests being used in other scripts that
>> run tests in cross
>>   environments.
>> - Define profiles for short and longer time tests
>> - Add install target in addition to the existing kselftest target at
>> the top level Makefile
>>
>> I will maintain the test framework - will take care of the logistics
>> of MAINTAINER
>> file updates for this and creating a git tree. Individual test
>> maintainers (we have
>> several in MAINTAINERS file) will continue to own and maintain their tests.
>>
>> Fixes to known cross-build failures and new tests are welcome. Please
>> send patches to linux-api mailing list
>
> I've been working on integration with Aboriginal (Rob Landley's
> minimal cross-compile distro) and hope to have some results
> very soon.

I've also been doing some hacking. I've taken a good look at several
of the tools people are using. Guenter's scripts. Kevin & Olof's
scripts, your ttc, and the stuff I'm using personally, and then
started a new tool for booting and testing boards, pulling in bits and
pieces that look good. I've been able to come up with something that
works really well for me, will boot on qemu out-of-the-box, and is
easily extendible to real hardware like a board on a desk or a
personal board farm. I'll post my patches this coming week. Right now
I'm cleaning up some ugly bits and I need to graft in the logging from
Kevin's scripts.

g.

[Ksummit-discuss] [kselftest] kselftest wiki (was RE: Fwd: Rough notes from testing unconference)

[Bird, Tim]

On Saturday, August 23, 2014 6:35 AM, Bird, Tim wrote:
> Also, I've requested a 'test' wiki on kernel.org, where we can place
> notes, ideas, and lists of things to work on, or things in progress
> (like possible output format guidelines).
> 
> I'll let everyone know when the wiki is set up.

There is now a kselftest wiki on kernel.org at:
https://kselftest.wiki.kernel.org/

I believe anyone with a Linux Foundation account can edit it.
I have put together a main page, along with a few sub-pages for
some of the sub-projects (as I call them) that were discussed at
the kernel summit.

Please feel free to directly edit the wiki, or to let me know if
there's any content you'd like to see placed on the wiki.

 -- Tim

P.S.  I have copied lkml and linux-api on this.  I believe this is the first 
use of linux-api for discussions about kselftest.  If the linux-api maintainers
would prefer we get our own list, please let me know.

[kselftest] kselftest wiki (was RE: [Ksummit-discuss] Fwd: Rough notes from testing unconference)

[Bird, Tim]

On Saturday, August 23, 2014 6:35 AM, Bird, Tim wrote:
> Also, I've requested a 'test' wiki on kernel.org, where we can place
> notes, ideas, and lists of things to work on, or things in progress
> (like possible output format guidelines).
> 
> I'll let everyone know when the wiki is set up.

There is now a kselftest wiki on kernel.org at:
https://kselftest.wiki.kernel.org/

I believe anyone with a Linux Foundation account can edit it.
I have put together a main page, along with a few sub-pages for
some of the sub-projects (as I call them) that were discussed at
the kernel summit.

Please feel free to directly edit the wiki, or to let me know if
there's any content you'd like to see placed on the wiki.

 -- Tim

P.S.  I have copied lkml and linux-api on this.  I believe this is the first 
use of linux-api for discussions about kselftest.  If the linux-api maintainers
would prefer we get our own list, please let me know.

[kselftest] kselftest wiki (was RE: [Ksummit-discuss] Fwd: Rough notes from testing unconference)

[Bird, Tim]

On Saturday, August 23, 2014 6:35 AM, Bird, Tim wrote:
> Also, I've requested a 'test' wiki on kernel.org, where we can place
> notes, ideas, and lists of things to work on, or things in progress
> (like possible output format guidelines).
> 
> I'll let everyone know when the wiki is set up.

There is now a kselftest wiki on kernel.org at:
https://kselftest.wiki.kernel.org/

I believe anyone with a Linux Foundation account can edit it.
I have put together a main page, along with a few sub-pages for
some of the sub-projects (as I call them) that were discussed at
the kernel summit.

Please feel free to directly edit the wiki, or to let me know if
there's any content you'd like to see placed on the wiki.

 -- Tim

P.S.  I have copied lkml and linux-api on this.  I believe this is the first 
use of linux-api for discussions about kselftest.  If the linux-api maintainers
would prefer we get our own list, please let me know.--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Ksummit-discuss] [kselftest] kselftest wiki (was RE: Fwd: Rough notes from testing unconference)

[Shuah Khan]

On Fri, Aug 29, 2014 at 7:09 PM, Bird, Tim <Tim.Bird@sonymobile.com> wrote:
> On Saturday, August 23, 2014 6:35 AM, Bird, Tim wrote:
>> Also, I've requested a 'test' wiki on kernel.org, where we can place
>> notes, ideas, and lists of things to work on, or things in progress
>> (like possible output format guidelines).
>>
>> I'll let everyone know when the wiki is set up.
>
> There is now a kselftest wiki on kernel.org at:
> https://kselftest.wiki.kernel.org/
>
> I believe anyone with a Linux Foundation account can edit it.
> I have put together a main page, along with a few sub-pages for
> some of the sub-projects (as I call them) that were discussed at
> the kernel summit.
>
> Please feel free to directly edit the wiki, or to let me know if
> there's any content you'd like to see placed on the wiki.
>

Thanks for doing this.

>
> P.S.  I have copied lkml and linux-api on this.  I believe this is the first
> use of linux-api for discussions about kselftest.  If the linux-api maintainers
> would prefer we get our own list, please let me know.

I spoke to a few folks and gave a heads up to Michael Kerrisk about using
linux-api for kselftest - so far no objections to using it. I sent in
a patch adding
entry for Kselftest framework to MAINTAINERS file with linux-api as the
mailing list. I have a git for this:

git git://git.kernel.org/pub/scm/shuah/linux-kselftest

I couldn't edit the wiki to add this detail even after logging into my
LF account.
Might have to request for access perhaps.

thanks,
-- Shuah

Re: [kselftest] kselftest wiki (was RE: [Ksummit-discuss] Fwd: Rough notes from testing unconference)

[Shuah Khan]

On Fri, Aug 29, 2014 at 7:09 PM, Bird, Tim <Tim.Bird@sonymobile.com> wrote:
> On Saturday, August 23, 2014 6:35 AM, Bird, Tim wrote:
>> Also, I've requested a 'test' wiki on kernel.org, where we can place
>> notes, ideas, and lists of things to work on, or things in progress
>> (like possible output format guidelines).
>>
>> I'll let everyone know when the wiki is set up.
>
> There is now a kselftest wiki on kernel.org at:
> https://kselftest.wiki.kernel.org/
>
> I believe anyone with a Linux Foundation account can edit it.
> I have put together a main page, along with a few sub-pages for
> some of the sub-projects (as I call them) that were discussed at
> the kernel summit.
>
> Please feel free to directly edit the wiki, or to let me know if
> there's any content you'd like to see placed on the wiki.
>

Thanks for doing this.

>
> P.S.  I have copied lkml and linux-api on this.  I believe this is the first
> use of linux-api for discussions about kselftest.  If the linux-api maintainers
> would prefer we get our own list, please let me know.

I spoke to a few folks and gave a heads up to Michael Kerrisk about using
linux-api for kselftest - so far no objections to using it. I sent in
a patch adding
entry for Kselftest framework to MAINTAINERS file with linux-api as the
mailing list. I have a git for this:

git git://git.kernel.org/pub/scm/shuah/linux-kselftest

I couldn't edit the wiki to add this detail even after logging into my
LF account.
Might have to request for access perhaps.

thanks,
-- Shuah

Re: [kselftest] kselftest wiki (was RE: [Ksummit-discuss] Fwd: Rough notes from testing unconference)

[Shuah Khan]

On Fri, Aug 29, 2014 at 7:09 PM, Bird, Tim <Tim.Bird-/MT0OVThwyLZJqsBc5GL+g@public.gmane.org> wrote:
> On Saturday, August 23, 2014 6:35 AM, Bird, Tim wrote:
>> Also, I've requested a 'test' wiki on kernel.org, where we can place
>> notes, ideas, and lists of things to work on, or things in progress
>> (like possible output format guidelines).
>>
>> I'll let everyone know when the wiki is set up.
>
> There is now a kselftest wiki on kernel.org at:
> https://kselftest.wiki.kernel.org/
>
> I believe anyone with a Linux Foundation account can edit it.
> I have put together a main page, along with a few sub-pages for
> some of the sub-projects (as I call them) that were discussed at
> the kernel summit.
>
> Please feel free to directly edit the wiki, or to let me know if
> there's any content you'd like to see placed on the wiki.
>

Thanks for doing this.

>
> P.S.  I have copied lkml and linux-api on this.  I believe this is the first
> use of linux-api for discussions about kselftest.  If the linux-api maintainers
> would prefer we get our own list, please let me know.

I spoke to a few folks and gave a heads up to Michael Kerrisk about using
linux-api for kselftest - so far no objections to using it. I sent in
a patch adding
entry for Kselftest framework to MAINTAINERS file with linux-api as the
mailing list. I have a git for this:

git git://git.kernel.org/pub/scm/shuah/linux-kselftest

I couldn't edit the wiki to add this detail even after logging into my
LF account.
Might have to request for access perhaps.

thanks,
-- Shuah

Re: [Ksummit-discuss] [kselftest] kselftest wiki (was RE: Fwd: Rough notes from testing unconference)

[Konstantin Ryabitsev]

[-- Attachment #1: Type: text/plain, Size: 300 bytes --]

On 29 Aug 2014 22:55, "Shuah Khan" <shuahkhan@gmail.com> wrote,
> I couldn't edit the wiki to add this detail even after logging into my
> LF account.
> Might have to request for access perhaps.

I've adjusted permissions to allow registered users to edit the wiki.
Please try now.

Best,
Konstantin

[-- Attachment #2: Type: text/html, Size: 448 bytes --]

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Fengguang Wu]

Hi all,

Sorry the conversation went too fast for me to jump in.
Hopefully it's not too late to have the below inputs.

On Fri, Aug 22, 2014 at 09:19:27AM -0500, Grant Likely wrote:
> Rough notes from the Testing discussion during the workshop day of
> Kernel Summit. Big thanks to Paul McKenney for writing all of this
> down.
> 
> g.
> 
> Testing session (unconference)
> 
> o       In-kernel or out of kernel?  Bisection?  Can do either way,
>         one approach is to have two clones of the git tree, and another
>         is to "git checkout" a specific version of the tools directory.

Yes. What I do when bisecting scripts/coccinelle static check errors
is to copy that directory out to some temp dir and use it as the
stable version to run.

> o       Levels of tests:  1) get a login prompt, 2) kselftest.
> 
> o       Should we have specified userspace?  No, quickly gets into
>         the distribution-building business.

FYI, in lkp-tests, "bin/setup-local job.yaml" does all the necessary
download, build, .deb creation and installation that's required for
running the given job file. Its source code is at

https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git

mmtests and autotest has similar installation capabilities.

> o       Will Deacon: Can script using busybox, inittab, and so on.
>         Can run trinity on such a setup.
> 
> o       Ted: Would be nice if other test frameworks built on top of
>         kselftest.  Therefore kselftest should not make any asssumptions.
> 
> o       Shua: Goal of kselftest isn't to create something new, but
>         rather to gather up what we already have.
> 
> o       Ted: Much of the code in tools/testing/selftest is unit tests
>         for a tiny part of the kernel.  These don't really belong
>         in LTP.
> 
> o       Olof: Let's not reinvent the test-infrastructure wheel.
> 
> o       Aneesh: Autotest.  Others: Lots of such tests, but lots of
>         different ones that are relatively difficult to set up.
> 
> o       Paul: Don't need rootfs.  rcutorture runs out of initrd.
> 
> o       Olof: Often easier just to write a script.
> 
> o       Ted: If we are going to lightweight, need to focus on kernel
>         unit tests.
> 
> o       What kind of tool can we depend on?  Busybox?  Just busybox,
>         has everything it needs.  (This means scripting must be in
>         dash.)
> 
> o       Arnd: klibc has its own set of binaries, cannot support
>         busybox.  Need libc.
> 
> o       Josh: Only C and standard C library supported.

lkp-tests currently depends on ruby to turn a job file into execution.
However it could be converted to shell script. Then there comes an
interesting question: will that make lkp-tests eligible for an
out-of-tree kselftest?

> o       Paul:  Host or guest?
> 
> o       Discussion: apparently need both.  Assume full-fledged distro
>         on host, minimal on guest/target.  UART, retrieve a file.
>         Some targets don't have persistent mass storage.
> 
> o       Grant:  Needs to work on separate hardware, on qemu, on
>         fast models.
> 
> o       Need stdio output, input optional (some tests take input
>         from kernel boot parameters.)

Same here: the lkp-tests post-processing scripts work on stdio output
of its test cases.

> o       How to parse success or failure?  How to know what to ignore?
>         Ted: What are high-level requirements rather than low-level
>         implementation?  Need host to easily parse the information
>         from the target on the host.
> 
> o       Mauro: Need something for performance tests as well.

lkp-tests is designed for performance tests.

LKP = Linux Kernel Performance.

In fact it runs every workload as performance/power tests, including
the functionality tests. For example, yesterday LKP reported an ext4
performance improvement when running xfstests:

        Re: [ext4] 71d4f7d0321: -49.6% xfstests.generic.274.seconds
        https://lkml.org/lkml/2014/8/21/726

> o       Ted: Group the tests.  "Quick" group.  Groups associated with
>         given function (e.g., suspend-resume).  Have profiles based
>         on types of tests needed and time allotted.
> 
> o       Josh: Add test types to the MAINTAINERS file.
> 
> o       Tim: Requirements around cross-building?  People make their
>         own?  We supply some in-tree?  We supply pointers to known
>         good cross-build toolchains?  Grant: "Yes".
> 
> o       Tim:  The need is to enable people who have never cross-built
>         in their life to successfully cross-build and test with
>         minimal effort.

Sorry I may missed some background: given the ktest and aiaiai tools
to help with cross-builds, what's the gap that we are discussing here?

> Next steps:  Grant to send call for volunteers for various efforts
>         to ksummit-discuss, interested people to reply.

We have a team behind lkp-tests actively running tests for all kernel
git trees we are aware of. We look forward to cooperate with everyone
to make it better at catching upstream regressions. Any tests you
would like to keep running for your subsystem, please let me know.

Performance/power tests are in particular suitable for being added
into lkp-tests -- it has very descriptive job file for running a workload
with different parameter sets. Together with elaborate statistics
collection and comparison capabilities, it allows people to catch
subtle changes between different kernels/kconfigs/test parameters.

Thanks,
Fengguang

> o       Josh: Can we pick an existing test output format that already
>         has tools, infrastructure, and so on.
> 
>         Tim: Hudson!
> 
>         Andy: PASS, FAIL, XFAIL, ...
> 
>         Shua: Must fit into the primary goal of being a good kernel
>         developer unit test.
> 
>         Ted: Proposed requirement: Simplest use case must have simple
>         tool that parses output.  Must not include a JRE for output
>         parsing.
> 
>         Tim: Must be human reasable.
> 
>         Andy: Must be short boilerplate, so that it can be generated
>         and read quickly.
> 
>         Paul: Willing to make reasonable adaptations to rcutorture output.
>         But it must not require an in-kernel JRE.
> 
>         Arnd: Would like self-test entry for drivers.
> 
>         Ben: Have seen similar things in other operating systems, would
>         be good in Linux.  Mauro: Would like to separate hardware and
>         software tests for drivers.   Arnd: Could be separate profile.
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Grant Likely]

On 23 Aug 2014 09:12, "Fengguang Wu" <fengguang.wu@intel.com> wrote:
>
> Hi all,
>
> Sorry the conversation went too fast for me to jump in.
> Hopefully it's not too late to have the below inputs.

Absolutely! There's a lot of things we talked about and all of the
topics are still up for discussion.

>
> On Fri, Aug 22, 2014 at 09:19:27AM -0500, Grant Likely wrote:
> > Rough notes from the Testing discussion during the workshop day of
> > Kernel Summit. Big thanks to Paul McKenney for writing all of this
> > down.
> >
> > g.
> >
> > Testing session (unconference)
> >
> > o       In-kernel or out of kernel?  Bisection?  Can do either way,
> >         one approach is to have two clones of the git tree, and another
> >         is to "git checkout" a specific version of the tools directory.
>
> Yes. What I do when bisecting scripts/coccinelle static check errors
> is to copy that directory out to some temp dir and use it as the
> stable version to run.
>
> > o       Levels of tests:  1) get a login prompt, 2) kselftest.
> >
> > o       Should we have specified userspace?  No, quickly gets into
> >         the distribution-building business.
>
> FYI, in lkp-tests, "bin/setup-local job.yaml" does all the necessary
> download, build, .deb creation and installation that's required for
> running the given job file. Its source code is at
>
> https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git
>
> mmtests and autotest has similar installation capabilities.

I'll take a look at those, thanks.

>
> > o       Will Deacon: Can script using busybox, inittab, and so on.
> >         Can run trinity on such a setup.
> >
> > o       Ted: Would be nice if other test frameworks built on top of
> >         kselftest.  Therefore kselftest should not make any asssumptions.
> >
> > o       Shua: Goal of kselftest isn't to create something new, but
> >         rather to gather up what we already have.
> >
> > o       Ted: Much of the code in tools/testing/selftest is unit tests
> >         for a tiny part of the kernel.  These don't really belong
> >         in LTP.
> >
> > o       Olof: Let's not reinvent the test-infrastructure wheel.
> >
> > o       Aneesh: Autotest.  Others: Lots of such tests, but lots of
> >         different ones that are relatively difficult to set up.
> >
> > o       Paul: Don't need rootfs.  rcutorture runs out of initrd.
> >
> > o       Olof: Often easier just to write a script.
> >
> > o       Ted: If we are going to lightweight, need to focus on kernel
> >         unit tests.
> >
> > o       What kind of tool can we depend on?  Busybox?  Just busybox,
> >         has everything it needs.  (This means scripting must be in
> >         dash.)
> >
> > o       Arnd: klibc has its own set of binaries, cannot support
> >         busybox.  Need libc.
> >
> > o       Josh: Only C and standard C library supported.
>
> lkp-tests currently depends on ruby to turn a job file into execution.
> However it could be converted to shell script. Then there comes an
> interesting question: will that make lkp-tests eligible for an
> out-of-tree kselftest?

I think we still want kselftest to live in tree. The issue here I
think is what is the simplest possible environment on the target. I
want to be able to cross build a really simple initrd that works for
any architecture that Linux supports. This gets test coverage out to
the less common architectures like microblaze.

I want to avoid c++, ruby, python, etc. Simply because it brings in a
lot of dependencies and things that can break. Shell script isn't a
problem.

For the simple rootfs that I'm trying to build, we can simply exclude
any tests that require extra libraries or interpreters. Anyone needing
something more complete should bring in a proper distro.

>
> > o       Paul:  Host or guest?
> >
> > o       Discussion: apparently need both.  Assume full-fledged distro
> >         on host, minimal on guest/target.  UART, retrieve a file.
> >         Some targets don't have persistent mass storage.
> >
> > o       Grant:  Needs to work on separate hardware, on qemu, on
> >         fast models.
> >
> > o       Need stdio output, input optional (some tests take input
> >         from kernel boot parameters.)
>
> Same here: the lkp-tests post-processing scripts work on stdio output
> of its test cases.
>
> > o       How to parse success or failure?  How to know what to ignore?
> >         Ted: What are high-level requirements rather than low-level
> >         implementation?  Need host to easily parse the information
> >         from the target on the host.
> >
> > o       Mauro: Need something for performance tests as well.
>
> lkp-tests is designed for performance tests.
>
> LKP = Linux Kernel Performance.
>
> In fact it runs every workload as performance/power tests, including
> the functionality tests. For example, yesterday LKP reported an ext4
> performance improvement when running xfstests:
>
>         Re: [ext4] 71d4f7d0321: -49.6% xfstests.generic.274.seconds
>         https://lkml.org/lkml/2014/8/21/726
>
> > o       Ted: Group the tests.  "Quick" group.  Groups associated with
> >         given function (e.g., suspend-resume).  Have profiles based
> >         on types of tests needed and time allotted.
> >
> > o       Josh: Add test types to the MAINTAINERS file.
> >
> > o       Tim: Requirements around cross-building?  People make their
> >         own?  We supply some in-tree?  We supply pointers to known
> >         good cross-build toolchains?  Grant: "Yes".
> >
> > o       Tim:  The need is to enable people who have never cross-built
> >         in their life to successfully cross-build and test with
> >         minimal effort.
>
> Sorry I may missed some background: given the ktest and aiaiai tools
> to help with cross-builds, what's the gap that we are discussing here?

Cross building the kernel isn't the problem. That's easy. Getting a
rootfs and the tests into the rootfs can be hard, depending on the
platform (a problem for some embedded targets).

ktest depends on a rootfs from elsewhere. I'm only mildly familiar
with aiaiai, but it appears to be the same situation. The gap I see is
the ability to cross compile the kselftest test cases and the ability
to create a rootfs including the test cases for any architecture. A
tool to solve that problem should be usable by both ktest and aiaiai.

>
> > Next steps:  Grant to send call for volunteers for various efforts
> >         to ksummit-discuss, interested people to reply.
>
> We have a team behind lkp-tests actively running tests for all kernel
> git trees we are aware of. We look forward to cooperate with everyone
> to make it better at catching upstream regressions. Any tests you
> would like to keep running for your subsystem, please let me know.

That would be great. What architectures are you currently testing on?

> Performance/power tests are in particular suitable for being added
> into lkp-tests -- it has very descriptive job file for running a workload
> with different parameter sets. Together with elaborate statistics
> collection and comparison capabilities, it allows people to catch
> subtle changes between different kernels/kconfigs/test parameters.
>
> Thanks,
> Fengguang

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Guenter Roeck]

On 08/24/2014 10:12 AM, Grant Likely wrote:
> On 23 Aug 2014 09:12, "Fengguang Wu" <fengguang.wu@intel.com> wrote:
>>
>> Hi all,
>>
>> Sorry the conversation went too fast for me to jump in.
>> Hopefully it's not too late to have the below inputs.
>
> Absolutely! There's a lot of things we talked about and all of the
> topics are still up for discussion.
>
>>
>> On Fri, Aug 22, 2014 at 09:19:27AM -0500, Grant Likely wrote:
>>> Rough notes from the Testing discussion during the workshop day of
>>> Kernel Summit. Big thanks to Paul McKenney for writing all of this
>>> down.
>>>
>>> g.
>>>
>>> Testing session (unconference)
>>>
>>> o       In-kernel or out of kernel?  Bisection?  Can do either way,
>>>          one approach is to have two clones of the git tree, and another
>>>          is to "git checkout" a specific version of the tools directory.
>>
>> Yes. What I do when bisecting scripts/coccinelle static check errors
>> is to copy that directory out to some temp dir and use it as the
>> stable version to run.
>>
>>> o       Levels of tests:  1) get a login prompt, 2) kselftest.
>>>
>>> o       Should we have specified userspace?  No, quickly gets into
>>>          the distribution-building business.
>>
>> FYI, in lkp-tests, "bin/setup-local job.yaml" does all the necessary
>> download, build, .deb creation and installation that's required for
>> running the given job file. Its source code is at
>>
>> https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git
>>
>> mmtests and autotest has similar installation capabilities.
>
> I'll take a look at those, thanks.
>
>>
>>> o       Will Deacon: Can script using busybox, inittab, and so on.
>>>          Can run trinity on such a setup.
>>>
>>> o       Ted: Would be nice if other test frameworks built on top of
>>>          kselftest.  Therefore kselftest should not make any asssumptions.
>>>
>>> o       Shua: Goal of kselftest isn't to create something new, but
>>>          rather to gather up what we already have.
>>>
>>> o       Ted: Much of the code in tools/testing/selftest is unit tests
>>>          for a tiny part of the kernel.  These don't really belong
>>>          in LTP.
>>>
>>> o       Olof: Let's not reinvent the test-infrastructure wheel.
>>>
>>> o       Aneesh: Autotest.  Others: Lots of such tests, but lots of
>>>          different ones that are relatively difficult to set up.
>>>
>>> o       Paul: Don't need rootfs.  rcutorture runs out of initrd.
>>>
>>> o       Olof: Often easier just to write a script.
>>>
>>> o       Ted: If we are going to lightweight, need to focus on kernel
>>>          unit tests.
>>>
>>> o       What kind of tool can we depend on?  Busybox?  Just busybox,
>>>          has everything it needs.  (This means scripting must be in
>>>          dash.)
>>>
>>> o       Arnd: klibc has its own set of binaries, cannot support
>>>          busybox.  Need libc.
>>>
>>> o       Josh: Only C and standard C library supported.
>>
>> lkp-tests currently depends on ruby to turn a job file into execution.
>> However it could be converted to shell script. Then there comes an
>> interesting question: will that make lkp-tests eligible for an
>> out-of-tree kselftest?
>
> I think we still want kselftest to live in tree. The issue here I
> think is what is the simplest possible environment on the target. I
> want to be able to cross build a really simple initrd that works for
> any architecture that Linux supports. This gets test coverage out to
> the less common architectures like microblaze.
>
Absolutely agree. That doesn't prevent out-of-tree usage; my plan is
to use whatever we come up with all the way back to 3.2 kernels.
Getting additional test coverage is essential, and the best way to
achieve it is to have the necessary environment available in the tree.

> I want to avoid c++, ruby, python, etc. Simply because it brings in a
> lot of dependencies and things that can break. Shell script isn't a
> problem.
>
> For the simple rootfs that I'm trying to build, we can simply exclude
> any tests that require extra libraries or interpreters. Anyone needing
> something more complete should bring in a proper distro.
>
>>
>>> o       Paul:  Host or guest?
>>>
>>> o       Discussion: apparently need both.  Assume full-fledged distro
>>>          on host, minimal on guest/target.  UART, retrieve a file.
>>>          Some targets don't have persistent mass storage.
>>>
>>> o       Grant:  Needs to work on separate hardware, on qemu, on
>>>          fast models.
>>>
>>> o       Need stdio output, input optional (some tests take input
>>>          from kernel boot parameters.)
>>
>> Same here: the lkp-tests post-processing scripts work on stdio output
>> of its test cases.
>>
>>> o       How to parse success or failure?  How to know what to ignore?
>>>          Ted: What are high-level requirements rather than low-level
>>>          implementation?  Need host to easily parse the information
>>>          from the target on the host.
>>>
>>> o       Mauro: Need something for performance tests as well.
>>
>> lkp-tests is designed for performance tests.
>>
>> LKP = Linux Kernel Performance.
>>
>> In fact it runs every workload as performance/power tests, including
>> the functionality tests. For example, yesterday LKP reported an ext4
>> performance improvement when running xfstests:
>>
>>          Re: [ext4] 71d4f7d0321: -49.6% xfstests.generic.274.seconds
>>          https://lkml.org/lkml/2014/8/21/726
>>
>>> o       Ted: Group the tests.  "Quick" group.  Groups associated with
>>>          given function (e.g., suspend-resume).  Have profiles based
>>>          on types of tests needed and time allotted.
>>>
>>> o       Josh: Add test types to the MAINTAINERS file.
>>>
>>> o       Tim: Requirements around cross-building?  People make their
>>>          own?  We supply some in-tree?  We supply pointers to known
>>>          good cross-build toolchains?  Grant: "Yes".
>>>
>>> o       Tim:  The need is to enable people who have never cross-built
>>>          in their life to successfully cross-build and test with
>>>          minimal effort.
>>
>> Sorry I may missed some background: given the ktest and aiaiai tools
>> to help with cross-builds, what's the gap that we are discussing here?
>
> Cross building the kernel isn't the problem. That's easy. Getting a
> rootfs and the tests into the rootfs can be hard, depending on the
> platform (a problem for some embedded targets).
>
Plus getting a toolchain which is able to not only build the kernel
but also busybox as well as any other required binaries.

> ktest depends on a rootfs from elsewhere. I'm only mildly familiar
> with aiaiai, but it appears to be the same situation. The gap I see is
> the ability to cross compile the kselftest test cases and the ability
> to create a rootfs including the test cases for any architecture. A
> tool to solve that problem should be usable by both ktest and aiaiai.
>
Exactly.

Wonder if it would be possible to publish prebuilt toolchains on kernel.org
which are suitable to create the root file system.

Guenter

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Grant Likely]

On 24 Aug 2014 19:14, "Guenter Roeck" <linux@roeck-us.net> wrote:
>
> On 08/24/2014 10:12 AM, Grant Likely wrote:
>> ktest depends on a rootfs from elsewhere. I'm only mildly familiar
>> with aiaiai, but it appears to be the same situation. The gap I see is
>> the ability to cross compile the kselftest test cases and the ability
>> to create a rootfs including the test cases for any architecture. A
>> tool to solve that problem should be usable by both ktest and aiaiai.
>>
> Exactly.
>
> Wonder if it would be possible to publish prebuilt toolchains on kernel.org
> which are suitable to create the root file system.

I want to experiment with crosstool-ng, and yes I think the binaries
should go on kernel.org. The current toolchains on kernel.org are
built by Tony Breeds using Segher Boessenkool's buildall scripts (gcc
only, no glibc). There are patches to buildall which will build a
libc, but I haven't had much success with them.

g.

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Guenter Roeck]

On 08/25/2014 02:23 AM, Grant Likely wrote:
> On 24 Aug 2014 19:14, "Guenter Roeck" <linux@roeck-us.net> wrote:
>>
>> On 08/24/2014 10:12 AM, Grant Likely wrote:
>>> ktest depends on a rootfs from elsewhere. I'm only mildly familiar
>>> with aiaiai, but it appears to be the same situation. The gap I see is
>>> the ability to cross compile the kselftest test cases and the ability
>>> to create a rootfs including the test cases for any architecture. A
>>> tool to solve that problem should be usable by both ktest and aiaiai.
>>>
>> Exactly.
>>
>> Wonder if it would be possible to publish prebuilt toolchains on kernel.org
>> which are suitable to create the root file system.
>
> I want to experiment with crosstool-ng, and yes I think the binaries
> should go on kernel.org. The current toolchains on kernel.org are
> built by Tony Breeds using Segher Boessenkool's buildall scripts (gcc
> only, no glibc). There are patches to buildall which will build a
> libc, but I haven't had much success with them.
>

Building toolchains with the latest version of buildroot is working
pretty well for me.

Guenter

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Geert Uytterhoeven]

On Sun, Aug 24, 2014 at 7:12 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
> Cross building the kernel isn't the problem. That's easy. Getting a
> rootfs and the tests into the rootfs can be hard, depending on the
> platform (a problem for some embedded targets).

In the past, I have used OpenWRT to build the rootfs for the test
target.

Still there are dependencies, as not all C libraries support all architectures.

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Guenter Roeck]

On 08/24/2014 11:15 AM, Geert Uytterhoeven wrote:
> On Sun, Aug 24, 2014 at 7:12 PM, Grant Likely <grant.likely@secretlab.ca> wrote:
>> Cross building the kernel isn't the problem. That's easy. Getting a
>> rootfs and the tests into the rootfs can be hard, depending on the
>> platform (a problem for some embedded targets).
>
> In the past, I have used OpenWRT to build the rootfs for the test
> target.
>
I very much prefer Grant's direct method - it ensures that the available
test code is automatically integrated into the root file system, and that
there are no architecture / target dependencies.

For my part I have used various means to create root file systems for my
own tests, depending on the architecture. I see it as significant improvement
to have it all together in a single place, in our control, and to have it
as simple as possible.

Guenter

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Olof Johansson]

On Sun, Aug 24, 2014 at 11:37 AM, Guenter Roeck <linux@roeck-us.net> wrote:
> On 08/24/2014 11:15 AM, Geert Uytterhoeven wrote:
>>
>> On Sun, Aug 24, 2014 at 7:12 PM, Grant Likely <grant.likely@secretlab.ca>
>> wrote:
>>>
>>> Cross building the kernel isn't the problem. That's easy. Getting a
>>> rootfs and the tests into the rootfs can be hard, depending on the
>>> platform (a problem for some embedded targets).
>>
>>
>> In the past, I have used OpenWRT to build the rootfs for the test
>> target.
>>
> I very much prefer Grant's direct method - it ensures that the available
> test code is automatically integrated into the root file system, and that
> there are no architecture / target dependencies.
>
> For my part I have used various means to create root file systems for my
> own tests, depending on the architecture. I see it as significant
> improvement
> to have it all together in a single place, in our control, and to have it
> as simple as possible.

Yep. Unfortunately, having a cross-toolchain that includes libc is a
significant complexity improvement over just having one that can build
a kernel. I wouldn't mind having pre-built rootfs:es available just
like we have toolchains today, but that won't help me when it comes to
building the kselftest binaries.

I have traditionally been using Segher's buildall
(http://git.infradead.org/users/segher/buildall.git) to build and
populate toolchains locally, and that will no longer be enough. Unless
someone convinces Segher to improve his scripts, of course. :)


-Olof

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Jason Cooper]

On Sun, Aug 24, 2014 at 11:47:47AM -0700, Olof Johansson wrote:
...
> I have traditionally been using Segher's buildall
> (http://git.infradead.org/users/segher/buildall.git) to build and
> populate toolchains locally, and that will no longer be enough. Unless
> someone convinces Segher to improve his scripts, of course. :)

fwiw, I've been using Mike Frysinger's crossdev utility on gentoo for
many years.  While it's tied closely to portage (can't live without it),
it has the advantage of automatic toolchain updates during regular
system updates.  Currently, I have the following on my build system:

  aarch64-unknown-linux-gnu-

  arm-none-linux-gnueabi-
  arm-none-linux-uclibceabi-

  powerpc-none-linux-gnueabi-

  mips-none-linux-gnueabi-
  mips-pc-linux-gnu-

  s390-none-linux-gnueabi-

  i586-pc-linux-gnu-
  x86_64-pc-linux-gnu-

Admittedly, I use the arm- one the most.

A while ago I looked at adding crossdev to gentoo-prefix (install gentoo
inside another OS, like OS X, or debian) which would then make the
utility much more useful.  But I never found the time to make it a
reality. :(

thx,

Jason.

Re: [Ksummit-discuss] Fwd: Rough notes from testing unconference

[Jason Cooper]

On Sun, Aug 24, 2014 at 03:14:15PM -0400, Jason Cooper wrote:
> On Sun, Aug 24, 2014 at 11:47:47AM -0700, Olof Johansson wrote:
> ...
> > I have traditionally been using Segher's buildall
> > (http://git.infradead.org/users/segher/buildall.git) to build and
> > populate toolchains locally, and that will no longer be enough. Unless
> > someone convinces Segher to improve his scripts, of course. :)
> 
> fwiw, I've been using Mike Frysinger's crossdev utility on gentoo for
> many years.  While it's tied closely to portage (can't live without it),
> it has the advantage of automatic toolchain updates during regular
> system updates.

I should also mention that I've reliably used it for years to make
minimalist rootfs's, either initrds, or rootfs's on flash.  Which is why
I thought it was relevant to the conversation.

> Currently, I have the following on my build system:
> 
>   aarch64-unknown-linux-gnu-
> 
>   arm-none-linux-gnueabi-
>   arm-none-linux-uclibceabi-
> 
>   powerpc-none-linux-gnueabi-
> 
>   mips-none-linux-gnueabi-
>   mips-pc-linux-gnu-
> 
>   s390-none-linux-gnueabi-
> 
>   i586-pc-linux-gnu-
>   x86_64-pc-linux-gnu-
> 
> Admittedly, I use the arm- one the most.
> 
> A while ago I looked at adding crossdev to gentoo-prefix (install gentoo
> inside another OS, like OS X, or debian) which would then make the
> utility much more useful.  But I never found the time to make it a
> reality. :(

thx,

Jason.