Discussion: Package testing

Discussion: Package testing

[Björn Stenberg]

Hi.

Many source packages include their own package test suites. We are looking at running these tests on target in a structured way, and I would like a discussion about how to best fit this into the Yocto framework.

The actions that need to be performed for a package test are roughly:

1) Build the test suite
2) Make the test suite appear on target
3) Run the test suite
4) Parse the results

Each action can be done in several ways, and there are different considerations for each solution.

1) Build the test suite
-----------------------
Many package tests are simply bash scripts that run the packaged binaries in various ways, but often the test suites also include binary tools that are not part of the normal package build. These tools have to be built for package testing to work.

Additionally, many packages build and run the test in a single command, such as "make check", which is obviously unsuitable for cross-compiled packages.

We can solve this in different ways:

a) Run the test build+run jobs on target. This avoids the need to modify packages, but building code on target can get quite expensive in terms of disk space. This in turn means many tests would require a harddisk or network disk to run.

b) Patch the makefiles to split test building and test running. Patching makefiles mean we get an additional maintenance and/or upstreaming burden, but we should be able to do this in ways that are acceptable to upstream. This is our suggestion.


2) Make the test suite appear on target
---------------------------------------
The test suite and utilities obviously have to be executable by the target system in order to run. There are a few options for this:

a) Copy all test files to the target at test run time, from the build dir, using whatever means available (scp, ftp etc). This limits testing to targets/images with easy automatic file transfer abilities installed.

b) NFS mount the build dir to access the full work dir and hence the test code. this limits testing to targets (and images) with network+nfs support. Plus it blends the build env and runtime env in an unpleasant way.

c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that contain all test files and add those to the image and hence the rootfs. This is our suggestion.


3) Run the test suite
---------------------
Depending on how the test files are presented to the target, the way we run them can take different shapes:

a) (scp) A top-level run-all-tests.sh runs on the build host, copies all test files from build dir to target, logs in and runs each test.

b) (nfs) run-all-tests.sh is executed in the nfs-mounted build dir on target and build-runs each test in its work dir.

c) (-ptest) Install all test files to /opt/ptest/${PN}-${PV} (for example). Make a package "ptest-runner" that has a script /opt/ptest/run-all-tests to iterate over all installed tests and run them. This is our suggestion.


4) Parse the test results
-------------------------
Just running the tests doesn't give us much. We have to be able to look at the results and make them meaningful to us on a system-global level. Packages present their test results in very different ways, and we need to convert that to a generic format:

a) Patch each package test to produce a generic ptest output format. This is likely difficult to get accepted upstream.

b) Patch the test code minimally and instead use a simple per-packet translate script that converts test suite output from the package-specific format to a generic ptest format. This is our suggestion.


Opinions?

Note that this mail is about the test suites for/in/by specific packages. Standalone test suites such as LTP are a slightly different topic, since they are separate packages (${BPN} == ${PN}) rather than package companion suites.

-- 
Björn

Re: Discussion: Package testing

[Frans Meulenbroeks]

[-- Attachment #1: Type: text/plain, Size: 6325 bytes --]

Bjorn, these are some interesting ideas.
Some feedback below.
This is from the perspective of asomeone who develops for embedded systems
boards

2012/6/14 Björn Stenberg <bjst@enea.com>

> Hi.
>
> Many source packages include their own package test suites. We are looking
> at running these tests on target in a structured way, and I would like a
> discussion about how to best fit this into the Yocto framework.
>
> The actions that need to be performed for a package test are roughly:
>
> 1) Build the test suite
> 2) Make the test suite appear on target
> 3) Run the test suite
> 4) Parse the results
>
> Each action can be done in several ways, and there are different
> considerations for each solution.
>
> 1) Build the test suite
> -----------------------
> Many package tests are simply bash scripts that run the packaged binaries
> in various ways, but often the test suites also include binary tools that
> are not part of the normal package build. These tools have to be built for
> package testing to work.
>
> Additionally, many packages build and run the test in a single command,
> such as "make check", which is obviously unsuitable for cross-compiled
> packages.
>
> We can solve this in different ways:
>
> a) Run the test build+run jobs on target. This avoids the need to modify
> packages, but building code on target can get quite expensive in terms of
> disk space. This in turn means many tests would require a harddisk or
> network disk to run.
>
> Not only storage space can be an issue. Some targets also do have limited
RAM and are low on CPU power.


> b) Patch the makefiles to split test building and test running. Patching
> makefiles mean we get an additional maintenance and/or upstreaming burden,
> but we should be able to do this in ways that are acceptable to upstream.
> This is our suggestion.
>
This seems the most plexible approach. I'd say this could generate
additional -test packages

>
>
> 2) Make the test suite appear on target
> ---------------------------------------
> The test suite and utilities obviously have to be executable by the target
> system in order to run. There are a few options for this:
>
> a) Copy all test files to the target at test run time, from the build dir,
> using whatever means available (scp, ftp etc). This limits testing to
> targets/images with easy automatic file transfer abilities installed.
>
It also somewhat couples the building and the copying (or an additional
command or step would be needed.

>
> b) NFS mount the build dir to access the full work dir and hence the test
> code. this limits testing to targets (and images) with network+nfs support.
> Plus it blends the build env and runtime env in an unpleasant way.
>
> c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that
> contain all test files and add those to the image and hence the rootfs.
> This is our suggestion.
>

Agree, then one can decide whether to add them to an image, use a package
feed to install things or whatever means.
In our case some of our systems do not have network support

>
>
> 3) Run the test suite
> ---------------------
> Depending on how the test files are presented to the target, the way we
> run them can take different shapes:
>
> a) (scp) A top-level run-all-tests.sh runs on the build host, copies all
> test files from build dir to target, logs in and runs each test.
>
> b) (nfs) run-all-tests.sh is executed in the nfs-mounted build dir on
> target and build-runs each test in its work dir.
>
> c) (-ptest) Install all test files to /opt/ptest/${PN}-${PV} (for
> example). Make a package "ptest-runner" that has a script
> /opt/ptest/run-all-tests to iterate over all installed tests and run them.
> This is our suggestion.
>

Seems good to me. a run-all-tests.sh script is somewhat problematic as
different systems have different packages installed so different tests.
E.g. t is not too interesting to run C++ tests if your target system does
not have/need C++
That makes creating a run-all-tests.sh script somewhat nastier (of course
it could be conditional e.g. test -f ./testA && ./testA)

>
>
> 4) Parse the test results
> -------------------------
> Just running the tests doesn't give us much. We have to be able to look at
> the results and make them meaningful to us on a system-global level.
> Packages present their test results in very different ways, and we need to
> convert that to a generic format:
>
> a) Patch each package test to produce a generic ptest output format. This
> is likely difficult to get accepted upstream.
>
> b) Patch the test code minimally and instead use a simple per-packet
> translate script that converts test suite output from the package-specific
> format to a generic ptest format. This is our suggestion.
>

I suspect most tests would indicate success/failure by their return code.
Maybe the test-runner could use the return code. Optionally the recipe
could add  a way to specify how to measure success (e.g by means of
specifying a TEST_SUCCESS variable in the recipe that contains a command or
shell script that when executed will return true otherwise false.

I think testers are mainly interested in success (all tests give ok is
good, if a test fails, it would be fine with me if I would have to go to a
package specific place to find log files etc (or it could be written to a
common log file)

>
>
> Opinions?
>

Good proposal!
The only specific point I can think of is cleaning up after a test is run.
If you are low on resources like storage space and a test creates some
larger outputs and does not delete them, the tests might run out of storage.
So (maybe as an option) a test should contain (or be supplemented with)
code to remove all its output (maybe only on success).

Hope this helps.
Frans

>
> Note that this mail is about the test suites for/in/by specific packages.
> Standalone test suites such as LTP are a slightly different topic, since
> they are separate packages (${BPN} == ${PN}) rather than package companion
> suites.
>
> --
> Björn
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 7883 bytes --]

Re: Discussion: Package testing

[Björn Stenberg]

Frans Meulenbroeks wrote:
> a run-all-tests.sh script is somewhat problematic as
> different systems have different packages installed so different tests.

The idea is that the top level run-all script looks in the /opt/ptest directory and runs all package tests that are installed there. It has no built-in knowledge of any tests.

> I think testers are mainly interested in success (all tests give ok is
> good, if a test fails, it would be fine with me if I would have to go to a
> package specific place to find log files etc (or it could be written to a
> common log file)

While failing tests are the most interesting during development, keeping track of both failed and successful test cases is important for regression tracking and test development over time. Therefore we want to produce an output that clearly identifies each test case ran and its result.

> The only specific point I can think of is cleaning up after a test is run.

Good point, I forgot to mention that. It is the responsibility of the per-package "run-ptest" script to clean up after the test.

-- 
Björn

Re: Discussion: Package testing

[Richard Purdie]

On Thu, 2012-06-14 at 15:26 +0200, Björn Stenberg wrote:
> Many source packages include their own package test suites. We are
> looking at running these tests on target in a structured way, and I
> would like a discussion about how to best fit this into the Yocto
> framework.
>
> The actions that need to be performed for a package test are roughly:
>
> 1) Build the test suite
> 2) Make the test suite appear on target
> 3) Run the test suite
> 4) Parse the results
>
> Each action can be done in several ways, and there are different
> considerations for each solution.
>
> 1) Build the test suite
> -----------------------
> Many package tests are simply bash scripts that run the packaged
> binaries in various ways, but often the test suites also include
> binary tools that are not part of the normal package build. These
> tools have to be built for package testing to work.
>
> Additionally, many packages build and run the test in a single
> command, such as "make check", which is obviously unsuitable for
> cross-compiled packages.
>
> We can solve this in different ways:
>
> a) Run the test build+run jobs on target. This avoids the need to
> modify packages, but building code on target can get quite expensive
> in terms of disk space. This in turn means many tests would require a
> harddisk or network disk to run.
>
> b) Patch the makefiles to split test building and test running.
> Patching makefiles mean we get an additional maintenance and/or
> upstreaming burden, but we should be able to do this in ways that are
> acceptable to upstream. This is our suggestion.

There are some good questions here :)

I think in general, we should be aiming for b) since that falls well
into some of the ideas below. Upstreams should be amenable to splitting
these into two targets (assuming "make check" just runs one after the
other) so we shouldn't have to carry patches in most cases long term.


> 2) Make the test suite appear on target
> ---------------------------------------
> The test suite and utilities obviously have to be executable by the
> target system in order to run. There are a few options for this:
>
> a) Copy all test files to the target at test run time, from the build
> dir, using whatever means available (scp, ftp etc). This limits
> testing to targets/images with easy automatic file transfer abilities
> installed.
>
> b) NFS mount the build dir to access the full work dir and hence the
> test code. this limits testing to targets (and images) with network
> +nfs support. Plus it blends the build env and runtime env in an
> unpleasant way.
>
> c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that
> contain all test files and add those to the image and hence the
> rootfs. This is our suggestion.

I like the idea of test packages.

> 
> 3) Run the test suite
> ---------------------
> Depending on how the test files are presented to the target, the way
> we run them can take different shapes:
>
> a) (scp) A top-level run-all-tests.sh runs on the build host, copies
> all test files from build dir to target, logs in and runs each test.
>
> b) (nfs) run-all-tests.sh is executed in the nfs-mounted build dir on
> target and build-runs each test in its work dir.
>
> c) (-ptest) Install all test files to /opt/ptest/${PN}-${PV} (for
> example). Make a package "ptest-runner" that has a
> script /opt/ptest/run-all-tests to iterate over all installed tests
> and run them. This is our suggestion.

Can we suitably extract the tests out of the source code to be able to
do this? I'm worried a little about the maintenance burden but I do like
option c). 

> 4) Parse the test results
> -------------------------
> Just running the tests doesn't give us much. We have to be able to
> look at the results and make them meaningful to us on a system-global
> level. Packages present their test results in very different ways, and
> we need to convert that to a generic format:
>
> a) Patch each package test to produce a generic ptest output format.
> This is likely difficult to get accepted upstream.
>
> b) Patch the test code minimally and instead use a simple per-packet
> translate script that converts test suite output from the
> package-specific format to a generic ptest format. This is our
> suggestion.
>
> Opinions?
> 
I think the approach described is along the right lines. We'll probably
have to try it and see how it goes. Over time, if upstreams see value in
standard formats and so on, we may be able to effect some change but
right now, we need to start somewhere!

I firmly believe working on these areas will add a lot of value to the
project and attract many more users, as well as help Linux in general so
its something we should work on.

Cheers,

Richard

Re: Discussion: Package testing

[Fredrik Hugosson]

On 06/18/2012 01:43 PM, Richard Purdie wrote:
> On Thu, 2012-06-14 at 15:26 +0200, Björn Stenberg wrote:
>> Many source packages include their own package test suites. We are
>> looking at running these tests on target in a structured way, and I
>> would like a discussion about how to best fit this into the Yocto
>> framework.
>>
>> The actions that need to be performed for a package test are roughly:
>>
>> 1) Build the test suite
>> 2) Make the test suite appear on target
>> 3) Run the test suite
>> 4) Parse the results
>>
>> Each action can be done in several ways, and there are different
>> considerations for each solution.
>>
>> 1) Build the test suite
>> -----------------------
>> Many package tests are simply bash scripts that run the packaged
>> binaries in various ways, but often the test suites also include
>> binary tools that are not part of the normal package build. These
>> tools have to be built for package testing to work.
>>
>> Additionally, many packages build and run the test in a single
>> command, such as "make check", which is obviously unsuitable for
>> cross-compiled packages.
>>
>> We can solve this in different ways:
>>
>> a) Run the test build+run jobs on target. This avoids the need to
>> modify packages, but building code on target can get quite expensive
>> in terms of disk space. This in turn means many tests would require a
>> harddisk or network disk to run.
>>
>> b) Patch the makefiles to split test building and test running.
>> Patching makefiles mean we get an additional maintenance and/or
>> upstreaming burden, but we should be able to do this in ways that are
>> acceptable to upstream. This is our suggestion.
>
> There are some good questions here :)

Yes there are!

> I think in general, we should be aiming for b) since that falls well
> into some of the ideas below. Upstreams should be amenable to splitting
> these into two targets (assuming "make check" just runs one after the
> other) so we shouldn't have to carry patches in most cases long term.
>

I think you both might be missing some cases here. The problem is that 
'make check' does not have a standardized meaning.

 From the top of my head, I can think of these different variants:
1) Function test level with testing run on target, package cross-compiled
2) Function test level with testing run on host, package cross-compiled
3) Function test level with testing run on host, package compiled for host
4) Unit test level with testing run on target, package cross-compiled
5) Unit test level with testing run on host, package compiled for host

I think that we need to be able to separate the different uses of 'make 
check' to be able to support testing nicely.

>> 2) Make the test suite appear on target
>> ---------------------------------------
>> The test suite and utilities obviously have to be executable by the
>> target system in order to run. There are a few options for this:
>>
>> a) Copy all test files to the target at test run time, from the build
>> dir, using whatever means available (scp, ftp etc). This limits
>> testing to targets/images with easy automatic file transfer abilities
>> installed.
>>
>> b) NFS mount the build dir to access the full work dir and hence the
>> test code. this limits testing to targets (and images) with network
>> +nfs support. Plus it blends the build env and runtime env in an
>> unpleasant way.
>>
>> c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that
>> contain all test files and add those to the image and hence the
>> rootfs. This is our suggestion.
>
> I like the idea of test packages.

Would be very nice for functional tests, but it will not be so simple 
for unit testing. For unit tests I belive you need some build target 
too, which leads back to the comment above.

It might be a lot of autoconf hacking to get this working, don't know if 
there are any widely accepted standard targets for it.

/Regards
Fredrik Hugosson

Re: Discussion: Package testing

[Björn Stenberg]

Fredrik Hugosson wrote:
> I think you both might be missing some cases here. The problem is
> that 'make check' does not have a standardized meaning.

Even if there was a standardized meaning to "make check", there are lots of packages that use completely differents systems for their building and testing. Hence there is no generic solution, we have to adapt each individual package manually.

> 1) Function test level with testing run on target, package cross-compiled
> 2) Function test level with testing run on host, package cross-compiled
> 3) Function test level with testing run on host, package compiled for host
> 4) Unit test level with testing run on target, package cross-compiled
> 5) Unit test level with testing run on host, package compiled for host

Our proposal is focused on testing the target binaries on the target, in an attempt to limit complexity. Testing -native and -nativesdk packages is therefore outside this scope.

> Would be very nice for functional tests, but it will not be so
> simple for unit testing. For unit tests I belive you need some build
> target too, which leads back to the comment above.

Yes, this proposal only works for tests that run on a single machine. But from what I have seen so far, that covers the vast majority of included package tests. Do you have examples to the contrary?

There will certainly be room and need for other catergories of tests, which is one reason I've used the specific "ptest" moniker for this. This is not intended to be a covers-all-bases test solution. It's just one part.

-- 
Björn

Re: Discussion: Package testing

[Björn Stenberg]

Richard Purdie wrote:
> > c) (-ptest) Install all test files to /opt/ptest/${PN}-${PV} (for
> > example). Make a package "ptest-runner" that has a
> > script /opt/ptest/run-all-tests to iterate over all installed tests
> > and run them. This is our suggestion.
> 
> Can we suitably extract the tests out of the source code to be able to
> do this? I'm worried a little about the maintenance burden but I do like
> option c). 

Typically the tests don't mix production and test code, since the purpose is to test the production code. Rather the tests are isolated in separate files that we relatively easily can build and install separately from the base package.

However, this is still early in the process. We have only processed a limited number of packages yet. We wanted to involve the community early to align our goals and directions.

-- 
Björn

Re: Discussion: Package testing

[Burton, Ross]

On 18 June 2012 16:04, Björn Stenberg <bjst@enea.com> wrote:
> Typically the tests don't mix production and test code, since the purpose is to test the production code. Rather the tests are isolated in separate files that we relatively easily can build and install separately from the base package.

The problem with that is that you can only test the external
interfaces, not the inner state (black box vs white box).  i.e. DBus
has test cases for each C file inside that C file, so it can exercise
both the external API and internal state.

Ross

Re: Discussion: Package testing

[Björn Stenberg]

Burton, Ross wrote:
> The problem with that is that you can only test the external
> interfaces, not the inner state (black box vs white box).  i.e. DBus
> has test cases for each C file inside that C file, so it can exercise
> both the external API and internal state.

I haven't examined dbus in detail yet, but in principle I don't think it should be a problem. As long as the test binaries are compiled on the build host, in the normal work dir, we don't need to know or care what source code they are built from.

-- 
Björn