RFC: oe-selftest script to run automated tests against bitbake tools

RFC: oe-selftest script to run automated tests against bitbake tools

[Mihai Lindner]

Hello all,

The idea of "oe-selftest" would be to have an extra method of testing to
cover what's not addressed by sanity tests, bitbake-selftest or image
run-time testing. More information and some example tests are posted in
the related bug, here:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4740

Before starting any work on this, maybe we can gather some thoughts,
comments, here or in the bugzilla entry, something in the lines of what
kind of tests should it run or not, what would be the usage scenario,
etc.

Thanks,
Mihai

Re: RFC: oe-selftest script to run automated tests against bitbake tools

[Paul Eggleton]

Hi Mihai,

On Monday 08 July 2013 20:33:20 Mihai Lindner wrote:
> The idea of "oe-selftest" would be to have an extra method of testing to
> cover what's not addressed by sanity tests, bitbake-selftest or image
> run-time testing. More information and some example tests are posted in
> the related bug, here:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=4740
> 
> Before starting any work on this, maybe we can gather some thoughts,
> comments, here or in the bugzilla entry, something in the lines of what
> kind of tests should it run or not, what would be the usage scenario,

Some of the kinds of tests I would expect this tool to be able to handle:

* Tests that require configuration changes or addition/modification of testing 
recipes. One issue here is that if the script did modify existing 
configuration/metadata you could potentially break user's environments if they 
run this script without realising what it does. Maybe it should set up its own 
temporary build directory to avoid this, and add additional recipes via a 
temporary layer? This could help keep the test environment sane in any case.

* As an extension to the above, tests that verify failure output. We often 
don't notice when something breaks in the error handling code, and this leads 
to regressions where we get python tracebacks instead of a proper error 
message.

* yocto-bsp/yocto-layer (although these are not in OE-Core - perhaps this is a 
chance to engineer the oe-selftest script so it can be extended by additional 
layers? I think this is something we want to allow for anyway)

* buildhistory (by running builds and then comparing using buildhistory-diff)

* create-recipe

* runqemu?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre