RFC: automated runtime testing on real hardware

[Paul Eggleton <paul.eggleton@linux.intel.com>]

Hi all,

As part of the feature development work for 1.5, we've recently added the 
ability to easily define runtime tests written in python, replacing the old 
shell-based imagetest.bbclass. However, for 1.5 this is limited to running the 
tests within a QEMU environment; the obvious and desirable next step is to be 
able to have these run on real hardware. Clearly this presents some 
challenges, such as how to manage access to the hardware if you have multiple 
builders potentially wanting to use the same board at around the same time.

Automated testing on real hardware is something that a lot of us do already, 
but the likelihood is that we've all been coming up with our own solutions for 
this, so it would be really great if we could have a single working solution 
to cover the generally applicable use cases. As a start though it would be 
interesting to hear from people who have existing test setups or who are 
generally interested in this area. I'd like to hear feedback on the following:

 * What hardware setup do you use for automated testing?

 * Do you use any software to manage the hardware? Does this include any 
existing open source tools?

 * What would you like to see in a common framework to enable this kind of 
testing?

For reference, some discussion on this started in another thread [1], but I'd 
like to throw this out to a more general audience.

Thanks,
Paul

[1] https://lists.yoctoproject.org/pipermail/yocto/2013-August/017687.html

-- 

Paul Eggleton
Intel Open Source Technology Centre