[cip-dev] Interesting talks at OSSE/Kernel Summit

[cip-dev] Interesting talks at OSSE/Kernel Summit

[Ben Hutchings]

I attended several talks at OSSE and Kernel Summit in Prague that might
be interesting to CIP members.  These weren't recorded but I made some
notes on them.  I'll send my notes on each talk as a reply to this
message.

Ben.

-- 
Ben Hutchings
Software Developer, Codethink Ltd.

[cip-dev] Automating Open Source License Compliance - Kate Stewart

[Ben Hutchings]

## Automating Open Source License Compliance - Kate Stewart

[Description](https://osseu17.sched.com/event/BxI3/)

A product distribution may need to include copyright statements,
licence statements, disclaimers.

Why is this a problem???Open source projects are changing quickly,
bringing in more copyrights and licenses.??Sometimes a project's
license doesn't actually apply to every source file.

Different sponsoring organisations and distributions have different
policies for which licenses they accept and how they're recorded.
Language specific package repositories also have their own standards
for recording licenses.

Wish lists for automation:

* Developers want a simple concise way to mark source files with
? license, checked at build time
* Open Source Offices want accurate license summary for third party
? software, and their own.??She referred to "Trusted Upstream
? Processes" but didn't expand on that.

SPDX (Software Package Data eXchange) provides standard ways to
identify licenses, to tag source files and to summarise this
information in a manifest.??Common OSS licenses are assigned short
names; custom licenses are also supported.

SPDX license identifiers supported by Debian (DEP-5), and other
distributions and organisations considering adopting them.??U-Boot
uses them, Linux is starting to use them, Eclipse Package Manager(?)
will start soon.

"Open Government Partnership" created a best practices template
that includes use of SPDX license identifiers.

An SPDX v2.1 document has metadata for itself, each package covered,
each source file included in packages, and any "snippets" (parts of
a source file) with a different license.

Various tooling is needed and available.??Open source tools include
FOSSology, ScanCode, SPDXTools; more are listed at
<https://wiki.debian.org/CopyrightReviewTools>.??Proprietary tools
are available from Wind River, Black Duck, others.

How accurate are the scanning tools???All are based partly on
heuristics.??She recomends testing them against a known set of source
files.

She mentioned some other types of tools, but I wasn't clear on what
they do.

The OpenChain project documents the process of license compliance(?),
which is useful for a supply chain.

Missing pieces:

* Trusted SPDX importers (for reviewed documents?)
* CI/CD build tool integration (check for tags at build time?)
* Curated real world examples
* End-to-end case studies using SPDX documents

-- 
Ben Hutchings
Software Developer, Codethink Ltd.

[cip-dev] Detecting Performance Regressions in the Linux Kernel - Jan Kara

[Ben Hutchings]

## Detecting Performance Regressions in the Linux Kernel - Jan Kara

[Description](https://osseu17.sched.com/event/BxIY/)

SUSE runs performance tests on a "grid" of different machines (10 x86,
1 ARM).??The x86 machines have a wide range of CPUs, memory size,
storage performance.??There are two back-to-back connected pairs for
network tests.

Other instances of the same models are available for debugging.

### Software used

"Marvin" is their framework for deploying, scheduling tests, bisecting.

"MMTests" is a framework for benchmarks - parses results and generates
comparisons - <https://github.com/gormanm/mmtests>.

CPU benchmarks: hackbench. libmicro, kernel page alloc benchmark (with
special module), PFT, SPECcpu2016, and others,

IO benchmarks: Iozone, Bonnie, Postmark, Reaim, Dbench4.??These are
run for all supported filesystems (ext3, ext4, xfs, btrfs) and
different RAID and non-RAID configurations.

Network benchmarks: sockperf, netperf, netpipe, siege.??These are run
over loopback and 10 gigabit Ethernet using Unix domain sockets (where
applicable), TCP, and UDP.??siege doesn't scale well so will be
replaced.

Complex benchmarks: kernbench, SPECjvm, Pgebcnh, sqlite insertion,
Postgres & MariaDB OLTP, ...

### How to detect performance changes?

Comparing a single benchmark result from each version is no good -
there is often significant variance in results.??It is necessary to
take multiple measurements, calculate average and s.d.

Caches and other features for increasing performance involve
prediction, which creates strong statistical dependencies.
Some statistical tests assume samples come from a normal
distribution, but performance results often don't.

It is sometimes possible to use Welch's T-test for significance of a
difference, but it is often necessary to plot a graph to understand
how the performance distribution is different - it can be due to
small numbers of outliers.

Some benchmarks take multiple (but not enough) results and average
them internally.??Ideally a benchmark framework will get all the
results and do its own statistical analysis.??For this reason, MMTests
uses modified versions of some benchmarks.

### Reducing variance in benchmarks

Filesystems: create from scratch each time

Scheduling: bind tasks to specific NUMA nodes; disable background
services; reboot before starting

It's generally not possible to control memory layout (which affects
cache performance) or interrupt timing.

### Benchmarks are buggy

* Setup can take most of the time
* Averages are not always calculated correctly
* Output is sometimes not flushed at exit, causing it to be truncated

-- 
Ben Hutchings
Software Developer, Codethink Ltd.

[Fuego] FW: [cip-dev] Detecting Performance Regressions in the Linux Kernel - Jan Kara

[Daniel Sangorrin]

Hi all,

I forward to the Fuego mailing list these notes from Ben that may be informative for us.
There are a few test suites mentioned that are not supported in Fuego yet.

Regards,
Daniel

-----Original Message-----
From: cip-dev-bounces@lists.cip-project.org [mailto:cip-dev-bounces@lists.cip-project.org] On Behalf Of Ben Hutchings
Sent: Wednesday, November 08, 2017 2:42 AM
To: cip-dev@lists.cip-project.org
Subject: [cip-dev] Detecting Performance Regressions in the Linux Kernel - Jan Kara

## Detecting Performance Regressions in the Linux Kernel - Jan Kara

[Description](https://osseu17.sched.com/event/BxIY/)

SUSE runs performance tests on a "grid" of different machines (10 x86,
1 ARM).  The x86 machines have a wide range of CPUs, memory size,
storage performance.  There are two back-to-back connected pairs for
network tests.

Other instances of the same models are available for debugging.

### Software used

"Marvin" is their framework for deploying, scheduling tests, bisecting.

"MMTests" is a framework for benchmarks - parses results and generates
comparisons - <https://github.com/gormanm/mmtests>.

CPU benchmarks: hackbench. libmicro, kernel page alloc benchmark (with
special module), PFT, SPECcpu2016, and others,

IO benchmarks: Iozone, Bonnie, Postmark, Reaim, Dbench4.  These are
run for all supported filesystems (ext3, ext4, xfs, btrfs) and
different RAID and non-RAID configurations.

Network benchmarks: sockperf, netperf, netpipe, siege.  These are run
over loopback and 10 gigabit Ethernet using Unix domain sockets (where
applicable), TCP, and UDP.  siege doesn't scale well so will be
replaced.

Complex benchmarks: kernbench, SPECjvm, Pgebcnh, sqlite insertion,
Postgres & MariaDB OLTP, ...

### How to detect performance changes?

Comparing a single benchmark result from each version is no good -
there is often significant variance in results.  It is necessary to
take multiple measurements, calculate average and s.d.

Caches and other features for increasing performance involve
prediction, which creates strong statistical dependencies.
Some statistical tests assume samples come from a normal
distribution, but performance results often don't.

It is sometimes possible to use Welch's T-test for significance of a
difference, but it is often necessary to plot a graph to understand
how the performance distribution is different - it can be due to
small numbers of outliers.

Some benchmarks take multiple (but not enough) results and average
them internally.  Ideally a benchmark framework will get all the
results and do its own statistical analysis.  For this reason, MMTests
uses modified versions of some benchmarks.

### Reducing variance in benchmarks

Filesystems: create from scratch each time

Scheduling: bind tasks to specific NUMA nodes; disable background
services; reboot before starting

It's generally not possible to control memory layout (which affects
cache performance) or interrupt timing.

### Benchmarks are buggy

* Setup can take most of the time
* Averages are not always calculated correctly
* Output is sometimes not flushed at exit, causing it to be truncated

-- 
Ben Hutchings
Software Developer, Codethink Ltd.

_______________________________________________
cip-dev mailing list
cip-dev@lists.cip-project.org
https://lists.cip-project.org/mailman/listinfo/cip-dev

[cip-dev] Improve Regression Tracking - Thorsten Leemhuis

[Ben Hutchings]

## Improve Regression Tracking - Thorsten Leemhuis

[Description](https://osseu17.sched.com/event/CnFn/)

Thorsten has started tracking regressions in the kernel, in his spare
time.??He is not a programmer but a journalist (for c't) and community
manager.??He started doing this because Linus said he would like to
see someone do the job which Rafael Wysocki used to.

He expected it to be hard and frustrating, and it's even worse than that!
He needs to search through LKML, subsystem MLs, multiple Bugzillas.
It's very time consuming and he's still missing a lot of regressions.

Discussion of a single issue might change forum and it's not obvious
so he doesn't see that.??An issue might get quietly resolved by commit
without any message to a bug tracker.

He requested people to use a regression ID (which he assigns) and put
it in discussions and commit messages.??This hasn't happened (yet).
Someone suggested to make wider use of `[REGRESSION]` for reports
of recent regressions.??(Both of the above should be added to in-tree
documentation.)

Someone suggested to add another mailing list specifically for
regression reports, that may be cc'd(?) along with specific ML.

The upstream bug reporting process differs a lot across subsystems -
frustrating for distribution maintainers forwarding reports.??It is
also hard to see current regression status of the next stable release
when considering whether to update the kernel package.

Regression tracking is also needed for the "long tail" of bugs that
don't get reported so quickly (within 1 or 2 release cycles).??This
will require a team of people, not just one.

There needs to be some kind of database to collect information, if
only references to discussions elsewhere.??Rafael used to create
tracking bugs on <https://bugzilla.kernel.org>.??Thorsten is using
a spreadsheet.


-- 
Ben Hutchings
Software Developer, Codethink Ltd.

[cip-dev] Kselftest use-cases - Shuah Khan

[Ben Hutchings]

## Kselftest use-cases - Shuah Khan

[Description](https://osseu17.sched.com/event/CnFp/)

kselftest is the kernel developer regression test suite.??It is
written by kernel developers and users.??It is used by developers,
users, automated test infrastructure (kernel-ci.org, 0-day test
robot).

How to run tests:

* `make --silent kselftest` - run all default targets
? (`TARGETS` in `tools/testing/selftests/Makefile`).
* `make --silent TARGETS=timers kselftest` - run all
? non-destructive tests in `tools/testing/selftests/timers`
* `make --silent -C tools/testing/selftests/timers run_tests`
? - same

Set `O=dir` for an out-of-tree build.??(But cureently this
may require a `.config` file in the source directory.)

Set `quicktest=1` to exclude time-consuming tests.

kselftest outputs a summary of results (since 4.14) following
TAP (Test Anything Protocol).

The output of individual tests can be found in `/tmp` (currently),
but it should be changed to allow specifying directory.

It is possible to run latest selftests on older kernels, but there
will be some failures due to missing features.??Ideally missing
dependencies result in a "skip" result but some maintainers aren't
happy to support that.??One reason is that if a feature is broken so
badly it isn't detected, tests may be skipped rather than failed.

Some tests apparently check for dependencies in a kernel config file.
(It wasn't clear to me where they look for it.)

Tips and hints:

* Use the `--silent` option to suppress make output
* Some tests need to run as root
* Beware that some tests are disruptive

More information:

* [Documentation/dev-tools/kselftest.rst](https://www.kernel.org/doc/html/latest/dev-tools/kselftest.html)
* [Blog entries](https://blogs.s-osg.org/author/shuahkh/)


-- 
Ben Hutchings
Software Developer, Codethink Ltd.

[cip-dev] Kselftest use-cases - Shuah Khan

[Daniel Sangorrin]

Hi Ben,
# I added the fuego mailing list to Cc

Thanks for the notes!

> -----Original Message-----
> From: cip-dev-bounces at lists.cip-project.org [mailto:cip-dev-bounces at lists.cip-project.org] On Behalf Of Ben Hutchings
> Sent: Wednesday, November 08, 2017 2:44 AM
> To: cip-dev at lists.cip-project.org
> Subject: [cip-dev] Kselftest use-cases - Shuah Khan
> 
> ## Kselftest use-cases - Shuah Khan
> 
> [Description](https://osseu17.sched.com/event/CnFp/)
> 
> kselftest is the kernel developer regression test suite.  It is
> written by kernel developers and users.  It is used by developers,
> users, automated test infrastructure (kernel-ci.org, 0-day test
> robot).

It also works on Fuego now!
# Thanks Tim for integrating my patches.

> How to run tests:
> 
> * `make --silent kselftest` - run all default targets
>   (`TARGETS` in `tools/testing/selftests/Makefile`).
> * `make --silent TARGETS=timers kselftest` - run all
>   non-destructive tests in `tools/testing/selftests/timers`
> * `make --silent -C tools/testing/selftests/timers run_tests`
>   - same

In Fuego we use a different approach. First we cross-compile and install
the tests on a temporary folder. At this stage a script called "run_kselftest.sh"
is generated. Then, we deploy the binaries and the script to the target where
the "run_kselftest.sh" script is invoked.

The good part of this approach is that Fuego does not require the target board
to have a toolchain installed, kernel source on the target, etc.
 
> Set `O=dir` for an out-of-tree build.  (But cureently this
> may require a `.config` file in the source directory.)
> 
> Set `quicktest=1` to exclude time-consuming tests.
> 
> kselftest outputs a summary of results (since 4.14) following
> TAP (Test Anything Protocol).

Actually I think that this was proposed by Tim. 
There is a TAP plugin for Jenkins that can be used for parsing the results in Fuego.
However, currently "run_kselftest.sh" doesn't seem to use the TAP format. humm..
Maybe this is on the TODO list upstream, I need to investigate it further.
 
> The output of individual tests can be found in `/tmp` (currently),
> but it should be changed to allow specifying directory.
> 
> It is possible to run latest selftests on older kernels, but there
> will be some failures due to missing features.  Ideally missing
> dependencies result in a "skip" result but some maintainers aren't
> happy to support that.  One reason is that if a feature is broken so
> badly it isn't detected, tests may be skipped rather than failed.

In Fuego there is now full control for specifying which test cases
are allowed to fail and which not. I will enable that functionality
on Fuego's integration scripts.

> Some tests apparently check for dependencies in a kernel config file.
> (It wasn't clear to me where they look for it.)

On some kselftest folders there is a "config" file that specifies the kernel
configuration options that need to be enabled (or disabled). From what I see
there is not a general script to check that they are configured on
the target kernel.

Fuego does support checking kernel configuration before running the test (using
information from /proc/config.gz or /boot/config-`uname -r`). Maybe it would a good
idea to add support on Fuego for checking individual kselftest's config files
 
Thank,
Daniel

> Tips and hints:
> 
> * Use the `--silent` option to suppress make output
> * Some tests need to run as root
> * Beware that some tests are disruptive
> 
> More information:
> 
> * [Documentation/dev-tools/kselftest.rst](https://www.kernel.org/doc/html/latest/dev-tools/kselftest.html)
> * [Blog entries](https://blogs.s-osg.org/author/shuahkh/)
> 
> 
> --
> Ben Hutchings
> Software Developer, Codethink Ltd.
> 
> _______________________________________________
> cip-dev mailing list
> cip-dev at lists.cip-project.org
> https://lists.cip-project.org/mailman/listinfo/cip-dev

Re: [Fuego] [cip-dev] Kselftest use-cases - Shuah Khan

[Daniel Sangorrin]

Hi Ben,
# I added the fuego mailing list to Cc

Thanks for the notes!

> -----Original Message-----
> From: cip-dev-bounces@lists.cip-project.org [mailto:cip-dev-bounces@lists.cip-project.org] On Behalf Of Ben Hutchings
> Sent: Wednesday, November 08, 2017 2:44 AM
> To: cip-dev@lists.cip-project.org
> Subject: [cip-dev] Kselftest use-cases - Shuah Khan
> 
> ## Kselftest use-cases - Shuah Khan
> 
> [Description](https://osseu17.sched.com/event/CnFp/)
> 
> kselftest is the kernel developer regression test suite.  It is
> written by kernel developers and users.  It is used by developers,
> users, automated test infrastructure (kernel-ci.org, 0-day test
> robot).

It also works on Fuego now!
# Thanks Tim for integrating my patches.

> How to run tests:
> 
> * `make --silent kselftest` - run all default targets
>   (`TARGETS` in `tools/testing/selftests/Makefile`).
> * `make --silent TARGETS=timers kselftest` - run all
>   non-destructive tests in `tools/testing/selftests/timers`
> * `make --silent -C tools/testing/selftests/timers run_tests`
>   - same

In Fuego we use a different approach. First we cross-compile and install
the tests on a temporary folder. At this stage a script called "run_kselftest.sh"
is generated. Then, we deploy the binaries and the script to the target where
the "run_kselftest.sh" script is invoked.

The good part of this approach is that Fuego does not require the target board
to have a toolchain installed, kernel source on the target, etc.
 
> Set `O=dir` for an out-of-tree build.  (But cureently this
> may require a `.config` file in the source directory.)
> 
> Set `quicktest=1` to exclude time-consuming tests.
> 
> kselftest outputs a summary of results (since 4.14) following
> TAP (Test Anything Protocol).

Actually I think that this was proposed by Tim. 
There is a TAP plugin for Jenkins that can be used for parsing the results in Fuego.
However, currently "run_kselftest.sh" doesn't seem to use the TAP format. humm..
Maybe this is on the TODO list upstream, I need to investigate it further.
 
> The output of individual tests can be found in `/tmp` (currently),
> but it should be changed to allow specifying directory.
> 
> It is possible to run latest selftests on older kernels, but there
> will be some failures due to missing features.  Ideally missing
> dependencies result in a "skip" result but some maintainers aren't
> happy to support that.  One reason is that if a feature is broken so
> badly it isn't detected, tests may be skipped rather than failed.

In Fuego there is now full control for specifying which test cases
are allowed to fail and which not. I will enable that functionality
on Fuego's integration scripts.

> Some tests apparently check for dependencies in a kernel config file.
> (It wasn't clear to me where they look for it.)

On some kselftest folders there is a "config" file that specifies the kernel
configuration options that need to be enabled (or disabled). From what I see
there is not a general script to check that they are configured on
the target kernel.

Fuego does support checking kernel configuration before running the test (using
information from /proc/config.gz or /boot/config-`uname -r`). Maybe it would a good
idea to add support on Fuego for checking individual kselftest's config files
 
Thank,
Daniel

> Tips and hints:
> 
> * Use the `--silent` option to suppress make output
> * Some tests need to run as root
> * Beware that some tests are disruptive
> 
> More information:
> 
> * [Documentation/dev-tools/kselftest.rst](https://www.kernel.org/doc/html/latest/dev-tools/kselftest.html)
> * [Blog entries](https://blogs.s-osg.org/author/shuahkh/)
> 
> 
> --
> Ben Hutchings
> Software Developer, Codethink Ltd.
> 
> _______________________________________________
> cip-dev mailing list
> cip-dev@lists.cip-project.org
> https://lists.cip-project.org/mailman/listinfo/cip-dev

[cip-dev] Interesting talks at OSSE/Kernel Summit

[Jan Kiszka]

On 2017-11-07 18:40, Ben Hutchings wrote:
> I attended several talks at OSSE and Kernel Summit in Prague that might
> be interesting to CIP members.  These weren't recorded but I made some
> notes on them.  I'll send my notes on each talk as a reply to this
> message.
> 
> Ben.
> 

Thanks for the valuable summaries, Ben!

[cip-dev] Interesting talks at OSSE/Kernel Summit

[Chris Paterson]

> From: cip-dev-bounces at lists.cip-project.org [mailto:cip-dev-
> bounces at lists.cip-project.org] On Behalf Of Jan Kiszka
> Sent: 08 November 2017 07:33
> 
> On 2017-11-07 18:40, Ben Hutchings wrote:
> > I attended several talks at OSSE and Kernel Summit in Prague that
> > might be interesting to CIP members.  These weren't recorded but I
> > made some notes on them.  I'll send my notes on each talk as a reply
> > to this message.
> >
> > Ben.
> >
> 
> Thanks for the valuable summaries, Ben!

+1