Automating imaging building and testing, what aproach to use!?

Automating imaging building and testing, what aproach to use!?

[Daniel.]

Hey everybody!

While writing software we're used to delivery packages, libraries and
stacks. There are out there a lot of continuos integration solutions
to automaticaly build and test these kinds of software. But when
dealing to images the thing is more tricky.

I can't run the tests at the same machine that build the image because
crosscompilation take place on 99% of the cases. What is aproach that
you guys are using to automate and increase the quality of your
images?

Automating the build is the easy part my concert is about automating
the runtime tests that need the target board to run. In my case I
depend on hardware to fully test the image features. Is there any
reliable way to automate image installation and boot!?

Best regards to everybody!
-- 
"Do or do not. There is no try"
  Yoda Master

Re: Automating imaging building and testing, what aproach to use!?

[Randy MacLeod]

On 2016-08-30 04:18 PM, Daniel. wrote:
> Hey everybody!
>
> While writing software we're used to delivery packages, libraries and
> stacks. There are out there a lot of continuos integration solutions
> to automatically build and test these kinds of software. But when
> dealing to images the thing is more tricky.
>
> I can't run the tests at the same machine that build the image because
> crosscompilation take place on 99% of the cases. What is approach that
> you guys are using to automate and increase the quality of your
> images?

We have a custom automated build system running 100s of configs/day

Some people I work with have started using LAVA [1] to
do automated testing for both Qemu and hardware targets.

Let me know if you are interested and I'll ask them to
summarize what they think of it so far.

../Randy

[1] http://www.linaro.org/initiatives/lava/

>
> Automating the build is the easy part my concert is about automating
> the runtime tests that need the target board to run. In my case I
> depend on hardware to fully test the image features. Is there any
> reliable way to automate image installation and boot!?
>
> Best regards to everybody!
>


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350 | 350 Terry Fox Drive, Suite 200, Ottawa, ON, 
Canada, K2K 2W5

Re: Automating imaging building and testing, what aproach to use!?

[Mike Looijmans]

[-- Attachment #1: Type: text/html, Size: 6558 bytes --]

[-- Attachment #2: imagec5944f.PNG --]
[-- Type: image/png, Size: 9075 bytes --]

[-- Attachment #3: imageab7767.JPG --]
[-- Type: image/jpeg, Size: 1088 bytes --]

[-- Attachment #4: image32a64f.JPG --]
[-- Type: image/jpeg, Size: 1087 bytes --]

[-- Attachment #5: image312119.JPG --]
[-- Type: image/jpeg, Size: 1060 bytes --]

Re: Automating imaging building and testing, what aproach to use!?

[Daniel.]

[-- Attachment #1.1: Type: text/plain, Size: 4088 bytes --]

Thanks Randy and Mark for the replies!

In my case we have a two boards running basically the same distribuition,
diverging only on some peripherals. I was planing to install/wire
everything up and then use https://mender.io/ to automate update process.
After that I can create post installations tests and see they run at every
image update. I'll have to design the tests so that the output of then goes
to some web interface or pdf+email but this is not a problem.

Another option is using buildbot to automate recipe builing and using it to
feed some intranet rpm/debi/ipk repo. Configure my testing targets with
something like "auto-update". After that is the same mater of creating post
install tests. The pro here is that this way I'll be testing by recipe
builds too, and that updates are faster. The con is that mender updates are
safer ...

The major complexity comes from the fact that we're using distribuited
hardware as may be the case of other people here. We have a "master"
hardware that connects to "slave" hardware, where master is the Linux and
slave are some embedded boards. To test everything up I need to wire
everything up before testing. I would like to hear people solutions for
that case since mulitple "setups" are possible testing every possible setup
is not feasible.

By the way, that LAVA seems promissing! I'll take a closer look on it!
Thanks for sharing :)

Regards,


2016-08-31 2:59 GMT-03:00 Mike Looijmans <mike.looijmans@topic.nl>:

> On 30-08-16 22:18, Daniel. wrote:
> > Hey everybody!
> >
> > While writing software we're used to delivery packages, libraries and
> > stacks. There are out there a lot of continuos integration solutions
> > to automaticaly build and test these kinds of software. But when
> > dealing to images the thing is more tricky.
> >
> > I can't run the tests at the same machine that build the image because
> > crosscompilation take place on 99% of the cases. What is aproach that
> > you guys are using to automate and increase the quality of your
> > images?
> >
> > Automating the build is the easy part my concert is about automating
> > the runtime tests that need the target board to run. In my case I
> > depend on hardware to fully test the image features. Is there any
> > reliable way to automate image installation and boot!?
>
> It helps a lot if you take testing into account when designing the
> hardware.
>
> We have a bunch of hardware targets in a corner. The board can be powered
> on/off forcibly by controlling/monitoring the RTS/DTR/CTS/DSR lines of the
> serial debug port (FTDI chip).
>
> A python script on a PC boots a board, puts it into USB "DFU" mode by
> "typing"
> commands into u-boot, and then sends an image over USB into RAM. The board
> then configures its USB OTG port as network, and the Python script
> connects to
> the board via SSH (paramiko) over that USB connection when it sees the
> "network" come up. Using the SSH connection, it can run whatever tests it
> needs doing (send data and even executables, run shell commands, and get
> reliable feedback on process completion).
>
> One test PC can control multiple boards (each board needs two USB
> connectors
> in this setup).
>
> I'd also be interested to know what other projects are doing.
>
>
>
> Kind regards,
>
>
>
> Mike Looijmans
>
> System Expert
>
>
>
>
>
> *TOPIC Products*
>
>
>
>
>
> Materiaalweg 4
>
>
>
>
>
> 5681 RJ Best
>
> T:
>
> +31 (0) 499 33 69 69
>
> Postbus 440
>
> E:
>
> mike.looijmans@topicproducts.com
>
> 5680 AK Best
>
> W:
>
> www.topicproducts.com
> The Netherlands
>
> <https://www.facebook.com/TopicProducts>
> <https://twitter.com/TopicProducts>
> <https://www.linkedin.com/company/topic-embedded-products>
> Please consider the environment before printing this e-mail
>
> Topic zoekt gedreven (embedded) software specialisten!
> <http://topic.nl/vacancy/topic-zoekt-technische-software-engineers/>
>
>


-- 
*"Do or do not. There is no try"*
  *Yoda Master*

[-- Attachment #1.2: Type: text/html, Size: 7833 bytes --]

[-- Attachment #2: imagec5944f.PNG --]
[-- Type: image/png, Size: 9075 bytes --]

[-- Attachment #3: image32a64f.JPG --]
[-- Type: image/jpeg, Size: 734 bytes --]

[-- Attachment #4: image312119.JPG --]
[-- Type: image/jpeg, Size: 723 bytes --]

[-- Attachment #5: imageab7767.JPG --]
[-- Type: image/jpeg, Size: 745 bytes --]

Re: Automating imaging building and testing, what aproach to use!?

[Paul Eggleton]

Hi Daniel,

On Tue, 30 Aug 2016 17:18:44 Daniel. wrote:
> While writing software we're used to delivery packages, libraries and
> stacks. There are out there a lot of continuos integration solutions
> to automaticaly build and test these kinds of software. But when
> dealing to images the thing is more tricky.
> 
> I can't run the tests at the same machine that build the image because
> crosscompilation take place on 99% of the cases. What is aproach that
> you guys are using to automate and increase the quality of your
> images?
> 
> Automating the build is the easy part my concert is about automating
> the runtime tests that need the target board to run. In my case I
> depend on hardware to fully test the image features. Is there any
> reliable way to automate image installation and boot!?

There are some folks here working on automated hardware tests (on CC), perhaps
they can expand on what we're currently doing in that area. At least in the
existing code we do have basic support for running tests on real hardware that
may be worth looking into - at the moment though it's pretty rudimentary when
it comes to interacting directly with the hardware though. You can see what
we've currently got here:

  http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#hardware-image-enabling-tests

We've looked at LAVA several times, and I'm sorry to say the conclusion each
time is that its a mess - both from a usage perspective and looking at the
code. It was disappointing to us because initially it looked like it was going
to solve a lot of our problems. Maybe others have had different experiences -
I'd love to hear details if anyone is prepared to share.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Automating imaging building and testing, what aproach to use!?

[Bruce Ashfield]

On 2016-08-31 6:23 PM, Paul Eggleton wrote:
> Hi Daniel,
>
> On Tue, 30 Aug 2016 17:18:44 Daniel. wrote:
>> While writing software we're used to delivery packages, libraries and
>> stacks. There are out there a lot of continuos integration solutions
>> to automaticaly build and test these kinds of software. But when
>> dealing to images the thing is more tricky.
>>
>> I can't run the tests at the same machine that build the image because
>> crosscompilation take place on 99% of the cases. What is aproach that
>> you guys are using to automate and increase the quality of your
>> images?
>>
>> Automating the build is the easy part my concert is about automating
>> the runtime tests that need the target board to run. In my case I
>> depend on hardware to fully test the image features. Is there any
>> reliable way to automate image installation and boot!?
>
> There are some folks here working on automated hardware tests (on CC), perhaps
> they can expand on what we're currently doing in that area. At least in the
> existing code we do have basic support for running tests on real hardware that
> may be worth looking into - at the moment though it's pretty rudimentary when
> it comes to interacting directly with the hardware though. You can see what
> we've currently got here:
>
>   http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#hardware-image-enabling-tests
>
> We've looked at LAVA several times, and I'm sorry to say the conclusion each
> time is that its a mess - both from a usage perspective and looking at the
> code. It was disappointing to us because initially it looked like it was going
> to solve a lot of our problems. Maybe others have had different experiences -
> I'd love to hear details if anyone is prepared to share.

We've been using LAVA extensively within WRind River, and haven't run
into any major issues with the code and usage. Perhaps it depends on
the type of test cases that are being run ?

LAVA was active, extensible and able to integrate with our wide range
of targets.

That's not to say that we didn't add a lot of our own tests,
infrastructure, etc, but that was expected work with whatever we
chose.

Cheers,

Bruce

>
> Cheers,
> Paul
>

Re: Automating imaging building and testing, what aproach to use!?

[Daniel.]

Hi Paul, Bruce

Thanks for the replies,

Paul,

I take a look at the documentation. It's an amazing feature I'll try
it as soon as got time avaible.
My next question is about that "master image" that is installed at
target hardware so that the image
is deployed automagically. I read that it writes the image to a second
partition... Well.. I'm dealing
with NAND Flash memory here I would like to avoid writing as much as
possible. Even more for
machines that will be reflashing at each new image built. I know how easy
is to setup an NFS + TFTP server and get u-boot booting things from
network. Also, with this kind
of setup booting a fresh new image is just a matter of rebooting
(which makes the master image
very simple). Would be possible to customise the master image and the
autodeploy to the hardware target!?

Regards,

2016-08-31 19:43 GMT-03:00 Bruce Ashfield <bruce.ashfield@windriver.com>:
> On 2016-08-31 6:23 PM, Paul Eggleton wrote:
>>
>> Hi Daniel,
>>
>> On Tue, 30 Aug 2016 17:18:44 Daniel. wrote:
>>>
>>> While writing software we're used to delivery packages, libraries and
>>> stacks. There are out there a lot of continuos integration solutions
>>> to automaticaly build and test these kinds of software. But when
>>> dealing to images the thing is more tricky.
>>>
>>> I can't run the tests at the same machine that build the image because
>>> crosscompilation take place on 99% of the cases. What is aproach that
>>> you guys are using to automate and increase the quality of your
>>> images?
>>>
>>> Automating the build is the easy part my concert is about automating
>>> the runtime tests that need the target board to run. In my case I
>>> depend on hardware to fully test the image features. Is there any
>>> reliable way to automate image installation and boot!?
>>
>>
>> There are some folks here working on automated hardware tests (on CC),
>> perhaps
>> they can expand on what we're currently doing in that area. At least in
>> the
>> existing code we do have basic support for running tests on real hardware
>> that
>> may be worth looking into - at the moment though it's pretty rudimentary
>> when
>> it comes to interacting directly with the hardware though. You can see
>> what
>> we've currently got here:
>>
>>
>> http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#hardware-image-enabling-tests
>>
>> We've looked at LAVA several times, and I'm sorry to say the conclusion
>> each
>> time is that its a mess - both from a usage perspective and looking at the
>> code. It was disappointing to us because initially it looked like it was
>> going
>> to solve a lot of our problems. Maybe others have had different
>> experiences -
>> I'd love to hear details if anyone is prepared to share.
>
>
> We've been using LAVA extensively within WRind River, and haven't run
> into any major issues with the code and usage. Perhaps it depends on
> the type of test cases that are being run ?
>
> LAVA was active, extensible and able to integrate with our wide range
> of targets.
>
> That's not to say that we didn't add a lot of our own tests,
> infrastructure, etc, but that was expected work with whatever we
> chose.
>
> Cheers,
>
> Bruce
>
>>
>> Cheers,
>> Paul
>>
>



-- 
"Do or do not. There is no try"
  Yoda Master

Re: Automating imaging building and testing, what aproach to use!?

[Mark Asselstine]

On September 1, 2016 11:54:30 AM Daniel. wrote:
> Hi Paul, Bruce
> 
> Thanks for the replies,
> 
> Paul,
> 
> I take a look at the documentation. It's an amazing feature I'll try
> it as soon as got time avaible.
> My next question is about that "master image" that is installed at
> target hardware so that the image
> is deployed automagically. I read that it writes the image to a second
> partition... Well.. I'm dealing
> with NAND Flash memory here I would like to avoid writing as much as
> possible. Even more for
> machines that will be reflashing at each new image built. I know how easy
> is to setup an NFS + TFTP server and get u-boot booting things from
> network. Also, with this kind
> of setup booting a fresh new image is just a matter of rebooting
> (which makes the master image
> very simple). Would be possible to customise the master image and the
> autodeploy to the hardware target!?

Just to prefix I have mostly used LAVA v1 and nearly exclusively with PXE boot 
capable targets. We are currently starting to use LAVA v2 and with more ARM 
boards with uBoot and SD/MMC devices.

LAVA has been fairly flexible when looking to implement custom device types 
and how they are booted. For instance we actually added interfaces to LAVA to 
reserve/unreserve targets via our lab management software as well as configure 
the PXE config file. We created our own master image (kernel+initramfs) which 
we PXE boot to then copy and setup the image we want to test. So although I 
don't want to say outright that you should be able to do whatever you want 
with LAVA I can say that it has been flexible enough to handle everything I 
have thrown at it. To add to this the changes have been 100% independent of 
the LAVA codebase, only new files, which I think speaks well to the design 
they have selected.

The one thing I have had to do is attempt to think more like a tester and less 
like a developer (I am definitely not a tester). So at times I have thought 
"why the heck are they booting the target so much", being the developer I have 
a tendency to want to be efficient, but when thinking about the problem from a 
testing perspective it made sense to ensure the test was done properly and 
results would be available, even if things failed horribly.

With all of the above we haven't been completely free of creating our own 
infrastructure. We use our own build farm to create images to test daily 
(developers can use their own builds). We also created scripts to allow for 
easier deployment of test 'collections'.

At time I also get frustrated with the LAVA documentation but I suppose this 
can be a common complaint for many pieces of software out there. This is 
somewhat offset by the fact that the code is well layed out. I can usually 
find what I am looking for easily, it tends to be in the spot that makes the 
most sense for it to be.

Mark


> 
> Regards,
> 
> 2016-08-31 19:43 GMT-03:00 Bruce Ashfield <bruce.ashfield@windriver.com>:
> > On 2016-08-31 6:23 PM, Paul Eggleton wrote:
> >> Hi Daniel,
> >> 
> >> On Tue, 30 Aug 2016 17:18:44 Daniel. wrote:
> >>> While writing software we're used to delivery packages, libraries and
> >>> stacks. There are out there a lot of continuos integration solutions
> >>> to automaticaly build and test these kinds of software. But when
> >>> dealing to images the thing is more tricky.
> >>> 
> >>> I can't run the tests at the same machine that build the image because
> >>> crosscompilation take place on 99% of the cases. What is aproach that
> >>> you guys are using to automate and increase the quality of your
> >>> images?
> >>> 
> >>> Automating the build is the easy part my concert is about automating
> >>> the runtime tests that need the target board to run. In my case I
> >>> depend on hardware to fully test the image features. Is there any
> >>> reliable way to automate image installation and boot!?
> >> 
> >> There are some folks here working on automated hardware tests (on CC),
> >> perhaps
> >> they can expand on what we're currently doing in that area. At least in
> >> the
> >> existing code we do have basic support for running tests on real hardware
> >> that
> >> may be worth looking into - at the moment though it's pretty rudimentary
> >> when
> >> it comes to interacting directly with the hardware though. You can see
> >> what
> >> we've currently got here:
> >> 
> >> 
> >> http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#hardw
> >> are-image-enabling-tests
> >> 
> >> We've looked at LAVA several times, and I'm sorry to say the conclusion
> >> each
> >> time is that its a mess - both from a usage perspective and looking at
> >> the
> >> code. It was disappointing to us because initially it looked like it was
> >> going
> >> to solve a lot of our problems. Maybe others have had different
> >> experiences -
> >> I'd love to hear details if anyone is prepared to share.
> > 
> > We've been using LAVA extensively within WRind River, and haven't run
> > into any major issues with the code and usage. Perhaps it depends on
> > the type of test cases that are being run ?
> > 
> > LAVA was active, extensible and able to integrate with our wide range
> > of targets.
> > 
> > That's not to say that we didn't add a lot of our own tests,
> > infrastructure, etc, but that was expected work with whatever we
> > chose.
> > 
> > Cheers,
> > 
> > Bruce
> > 
> >> Cheers,
> >> Paul
> 
> --
> "Do or do not. There is no try"
>   Yoda Master
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto