RFC: automated runtime testing on real hardware

RFC: automated runtime testing on real hardware

[Paul Eggleton]

Hi all,

As part of the feature development work for 1.5, we've recently added the 
ability to easily define runtime tests written in python, replacing the old 
shell-based imagetest.bbclass. However, for 1.5 this is limited to running the 
tests within a QEMU environment; the obvious and desirable next step is to be 
able to have these run on real hardware. Clearly this presents some 
challenges, such as how to manage access to the hardware if you have multiple 
builders potentially wanting to use the same board at around the same time.

Automated testing on real hardware is something that a lot of us do already, 
but the likelihood is that we've all been coming up with our own solutions for 
this, so it would be really great if we could have a single working solution 
to cover the generally applicable use cases. As a start though it would be 
interesting to hear from people who have existing test setups or who are 
generally interested in this area. I'd like to hear feedback on the following:

 * What hardware setup do you use for automated testing?

 * Do you use any software to manage the hardware? Does this include any 
existing open source tools?

 * What would you like to see in a common framework to enable this kind of 
testing?

For reference, some discussion on this started in another thread [1], but I'd 
like to throw this out to a more general audience.

Thanks,
Paul

[1] https://lists.yoctoproject.org/pipermail/yocto/2013-August/017687.html

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: RFC: automated runtime testing on real hardware

[George Harris]

[-- Attachment #1: Type: text/plain, Size: 3000 bytes --]

To this point, our testing strategy uses Yocto, but is not contained within
Yocto. As part of a build, we produce two images: (1) device image, and (2)
image containing everything necessary to run tests against a device. The
second part is important to us, as we want a separation between the device
and tests run against the device. Tests against a device are always
performed across the network, between (1) and (2). We want anyone to easily
be able to run or develop tests without needing to setup a workstation or a
workspace. A separate testing image, runnable in QEMU, allows anyone to
pick up a build and start testing with minimal effort. We also build a
toolchain containing QEMU to ensure that everyone uses the same version and
features of QEMU. Finally, we have many tests, some which take longer than
24 hours to run. We want to parallelize these tests and run them across
multiple systems at the same time. Because of this, we take the output from
a build and run tests. Some tests may make sense to run every build, but
other tests are run periodically throughout the day independent of the
build.

Both our build system and test system are based on Jenkins. One job will
build the code, and another will run tests. The test job is configured to
run multiple types of tests in multiple configurations, all concurrently.

As our tests are always run across the network, tests need not know if the
device under test is a simulation or hardware. Jenkins is used to manage
requests for hardware; if a configuration requires the use of a piece of
hardware, that part of the job blocks until the hardware is available. When
the hardware becomes available, Jenkins will allow that part of the job to
start.

As for the arrangement of the hardware, we have a testbed consisting of
several devices. Each device is connected to the network and hooked up to a
power switch that is connected to the network. When a test is run against a
device, that device is reprogrammed, power cycled, then the tests are run.

To answer some of your specific questions:

 * What hardware setup do you use for automated testing?
>
There is a testbed of devices, each connected to the network with a fixed
IP address. Jenkins is used to run tests, and also manages access to
hardware via labels. When a Jenkins test job requires a piece of hardware
and that hardware is available, a configuration file is used to map the
device to use to its IP address. From there, tests are run against the
device.

 * Do you use any software to manage the hardware? Does this include any
> existing open source tools?
>
The only software we used to manage the hardware is Jenkins. We do use
other software to manipulate the hardware. For example, a power switch
accessible over the network, a DHCP server to assign the proper address,
etc.

Likely this will not help you develop a hardware testing solution in Yocto.
Maybe it will give you a data point though.

- George

[-- Attachment #2: Type: text/html, Size: 3460 bytes --]

Re: RFC: automated runtime testing on real hardware

[Darren Hart]

On Tue, 2013-08-13 at 12:00 +0100, Paul Eggleton wrote:
> Hi all,
> 
> As part of the feature development work for 1.5, we've recently added the 
> ability to easily define runtime tests written in python, replacing the old 
> shell-based imagetest.bbclass. However, for 1.5 this is limited to running the 
> tests within a QEMU environment; the obvious and desirable next step is to be 
> able to have these run on real hardware. Clearly this presents some 
> challenges, such as how to manage access to the hardware if you have multiple 
> builders potentially wanting to use the same board at around the same time.
> 
> Automated testing on real hardware is something that a lot of us do already, 
> but the likelihood is that we've all been coming up with our own solutions for 
> this, so it would be really great if we could have a single working solution 
> to cover the generally applicable use cases. As a start though it would be 
> interesting to hear from people who have existing test setups or who are 
> generally interested in this area. I'd like to hear feedback on the following:
> 
>  * What hardware setup do you use for automated testing?

I am starting to put something together using a customized 19" rack
along the lines of those used by OSADL for their real-time farm:

https://www.osadl.org/Test-Rack.test-rack.0.html

This is a poor man's BladeCenter for embedded boards in that it provides
a common interface to each board for the rack infrastructure. The board
specific stuff sits on the trays themselves. The trays are easily
removable to take a board to a desk for hands-on work.

A smart-switch, web-power strip, and serial concentrator feed into a
"gateway server" where services like conmux glue it all together.

> 
>  * Do you use any software to manage the hardware? Does this include any 
> existing open source tools?

conmux
dnsmasq
udev (for static usb uart naming)

The testrack has some early-stage tools to support it, but I'm still
sorting those out and they aren't particularly robust for initial setup
quite yet.

> 
>  * What would you like to see in a common framework to enable this kind of 
> testing?

To simplify the board management you allude to above, I believe a pull
model, rather than a push, is a better architecture.

Your builders can complete builds and inject test records into a
database with a priority. The boards sit in a
boot/check-db/deploy-test-img/reboot-to-test-image/run-tests loop and
push the results back to the database. They select tests based on
priority, and the builder doesn't have to do anything at all to manage
the boards. This also decouples the builder from the test
infrastructure.

> 
> For reference, some discussion on this started in another thread [1], but I'd 
> like to throw this out to a more general audience.
> 
> Thanks,
> Paul
> 
> [1] https://lists.yoctoproject.org/pipermail/yocto/2013-August/017687.html
> 

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: RFC: automated runtime testing on real hardware

[Paul Eggleton]

Hi Darren,

On Friday 23 August 2013 20:52:41 Darren Hart wrote:
> On Tue, 2013-08-13 at 12:00 +0100, Paul Eggleton wrote:
> > Automated testing on real hardware is something that a lot of us do
> > already, but the likelihood is that we've all been coming up with our own
> > solutions for this, so it would be really great if we could have a single
> > working solution to cover the generally applicable use cases. As a start
> > though it would be interesting to hear from people who have existing test
> > setups or who are generally interested in this area. I'd like to hear
> > feedback on the following:
> >  * What hardware setup do you use for automated testing?
> 
> I am starting to put something together using a customized 19" rack
> along the lines of those used by OSADL for their real-time farm:
> 
> https://www.osadl.org/Test-Rack.test-rack.0.html
> 
> This is a poor man's BladeCenter for embedded boards in that it provides
> a common interface to each board for the rack infrastructure. The board
> specific stuff sits on the trays themselves. The trays are easily
> removable to take a board to a desk for hands-on work.
> 
> A smart-switch, web-power strip, and serial concentrator feed into a
> "gateway server" where services like conmux glue it all together.

All makes sense.

> >  * Do you use any software to manage the hardware? Does this include any
> > existing open source tools?
> 
> conmux
> dnsmasq
> udev (for static usb uart naming)
> 
> The testrack has some early-stage tools to support it, but I'm still
> sorting those out and they aren't particularly robust for initial setup
> quite yet.

Having a standard way to set up the software side for this kind of 
infrastructure would certainly be worthwhile I think. One tricky part would 
perhaps be supporting variations of infrastructure hardware; doing some 
searches I haven't found a completely definitive open source project for 
controlling remotely accessible power strips for example, although there are a 
few scattered projects for doing that with varying levels of recent 
maintenance.

> >  * What would you like to see in a common framework to enable this kind of
> > testing?
> 
> To simplify the board management you allude to above, I believe a pull
> model, rather than a push, is a better architecture.
> 
> Your builders can complete builds and inject test records into a
> database with a priority. The boards sit in a
> boot/check-db/deploy-test-img/reboot-to-test-image/run-tests loop and
> push the results back to the database. They select tests based on
> priority, and the builder doesn't have to do anything at all to manage
> the boards. This also decouples the builder from the test
> infrastructure.

Interesting. This would present some challenges with the way the automated 
tests are currently defined; there's nothing unsolvable that precludes them 
being run outside of the build environment but they will have to be run 
somewhere that has python and any required modules installed, presumably an 
intermediate machine rather than the target machine itself (the latter would 
not be impossible; but if we did do it that way you couldn't test an image 
that didn't have python in it.)

We'd also need to come up with a standard definition for the results database 
and tools to generate reports out of it, but that's something that will likely 
be needed regardless of the model used for running tests, especially when it 
comes to recording the results from ptest.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: RFC: automated runtime testing on real hardware

[Darren Hart]

On Tue, 2013-08-27 at 12:11 +0100, Paul Eggleton wrote:
> Hi Darren,
> 
> On Friday 23 August 2013 20:52:41 Darren Hart wrote:
> > On Tue, 2013-08-13 at 12:00 +0100, Paul Eggleton wrote:
> > > Automated testing on real hardware is something that a lot of us do
> > > already, but the likelihood is that we've all been coming up with our own
> > > solutions for this, so it would be really great if we could have a single
> > > working solution to cover the generally applicable use cases. As a start
> > > though it would be interesting to hear from people who have existing test
> > > setups or who are generally interested in this area. I'd like to hear
> > > feedback on the following:
> > >  * What hardware setup do you use for automated testing?
> > 
> > I am starting to put something together using a customized 19" rack
> > along the lines of those used by OSADL for their real-time farm:
> > 
> > https://www.osadl.org/Test-Rack.test-rack.0.html
> > 
> > This is a poor man's BladeCenter for embedded boards in that it provides
> > a common interface to each board for the rack infrastructure. The board
> > specific stuff sits on the trays themselves. The trays are easily
> > removable to take a board to a desk for hands-on work.
> > 
> > A smart-switch, web-power strip, and serial concentrator feed into a
> > "gateway server" where services like conmux glue it all together.
> 
> All makes sense.
> 
> > >  * Do you use any software to manage the hardware? Does this include any
> > > existing open source tools?
> > 
> > conmux
> > dnsmasq
> > udev (for static usb uart naming)
> > 
> > The testrack has some early-stage tools to support it, but I'm still
> > sorting those out and they aren't particularly robust for initial setup
> > quite yet.
> 
> Having a standard way to set up the software side for this kind of 
> infrastructure would certainly be worthwhile I think. One tricky part would 
> perhaps be supporting variations of infrastructure hardware; doing some 
> searches I haven't found a completely definitive open source project for 
> controlling remotely accessible power strips for example, although there are a 
> few scattered projects for doing that with varying levels of recent 
> maintenance.
> 
> > >  * What would you like to see in a common framework to enable this kind of
> > > testing?
> > 
> > To simplify the board management you allude to above, I believe a pull
> > model, rather than a push, is a better architecture.
> > 
> > Your builders can complete builds and inject test records into a
> > database with a priority. The boards sit in a
> > boot/check-db/deploy-test-img/reboot-to-test-image/run-tests loop and
> > push the results back to the database. They select tests based on
> > priority, and the builder doesn't have to do anything at all to manage
> > the boards. This also decouples the builder from the test
> > infrastructure.
> 
> Interesting. This would present some challenges with the way the automated 
> tests are currently defined; there's nothing unsolvable that precludes them 
> being run outside of the build environment but they will have to be run 
> somewhere that has python and any required modules installed, presumably an 
> intermediate machine rather than the target machine itself (the latter would 
> not be impossible; but if we did do it that way you couldn't test an image 
> that didn't have python in it.)

A new test-image will need to be created regardless, I think including
python is reasonable for the general case. For things like tiny, I
suspect we would have a customized test-suite anyway, possibly written
in C.

> 
> We'd also need to come up with a standard definition for the results database 
> and tools to generate reports out of it, but that's something that will likely 
> be needed regardless of the model used for running tests, especially when it 
> comes to recording the results from ptest.

Agreed.

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel