[U-Boot] U-Boot Regression Testing

[U-Boot] U-Boot Regression Testing

[Curt Brune]

Hello,

I have some questions about how U-Boot regression testing works.  I am
assuming some regression testing happens during the release period
across some representative sample of boards and architectures.

I know people check for compilation failures with MAKEALL, but I am
wondering about runtime testing.  I also understand that testing
platform specific boot code is rather difficult (or easy depending on
your perspective) - it boots or it does not boot.

To be concrete -- how are core U-Boot commands and features tested?
For example how do you test that FIT image support is not broken or
that the 'env' command and all its options work properly?

Googling did not turn up much on how this is done.

On the social side -- is that something the community helps out with
or something DENX does, or a mix?

Are you using a test framework of some kind, either home grown or open
source?  These things tend to become home grown over time :)

This kind of testing usually takes the form of 'chat' scripts
communicating over serial consoles.  Perhaps you are using expect,
pexpect, python nose?

We have a project, of which U-Boot is a part, that is starting to span
multiple boards and architectures.  We make a few modifications to
U-Boot and I want to start automated regression testing as the number
of boards increases.

If an existing framework exists that folks are happy with I would love
to hear about it.  Equally, I am interested to hear about what did
*not* work for people.  Whatever method our project uses the scripts
will be publicly available.

Cheers,
Curt

[U-Boot] U-Boot Regression Testing

[Simon Glass]

Hi Curt,

On Wed, Oct 9, 2013 at 10:22 AM, Curt Brune <curt@cumulusnetworks.com> wrote:
> Hello,
>
> I have some questions about how U-Boot regression testing works.  I am
> assuming some regression testing happens during the release period
> across some representative sample of boards and architectures.
>
> I know people check for compilation failures with MAKEALL, but I am
> wondering about runtime testing.  I also understand that testing
> platform specific boot code is rather difficult (or easy depending on
> your perspective) - it boots or it does not boot.
>
> To be concrete -- how are core U-Boot commands and features tested?
> For example how do you test that FIT image support is not broken or
> that the 'env' command and all its options work properly?
>
> Googling did not turn up much on how this is done.
>
> On the social side -- is that something the community helps out with
> or something DENX does, or a mix?
>
> Are you using a test framework of some kind, either home grown or open
> source?  These things tend to become home grown over time :)
>
> This kind of testing usually takes the form of 'chat' scripts
> communicating over serial consoles.  Perhaps you are using expect,
> pexpect, python nose?
>
> We have a project, of which U-Boot is a part, that is starting to span
> multiple boards and architectures.  We make a few modifications to
> U-Boot and I want to start automated regression testing as the number
> of boards increases.
>
> If an existing framework exists that folks are happy with I would love
> to hear about it.  Equally, I am interested to hear about what did
> *not* work for people.  Whatever method our project uses the scripts
> will be publicly available.

You can use sandbox, which is a build of U-Boot that runs under Linux.
There are some existing tests in test/ - you mention FIT support and
there is a basic test for that (could be expanded, in fact I may have
a patch somewhere).

Note that there is also buildman which aims to be the new MAKEALL eventually.

Regards,
Simon


>
> Cheers,
> Curt
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

[U-Boot] U-Boot Regression Testing

[Wolfgang Denk]

Dear Curt Brune,

In message <20131009162255.GG31348@cumulusnetworks.com> you wrote:
> 
> To be concrete -- how are core U-Boot commands and features tested?
> For example how do you test that FIT image support is not broken or
> that the 'env' command and all its options work properly?

Have a look at DUTS [1]; the test cases implemented there serve a
double purpose: 1) they help testing (some of) the functionality of
U-Boot (and Linux) on a specific board, and 2) they generate the
needed include files to produce a board-specific version of the DULG
[2]

[1] http://www.denx.de/wiki/DUTS/DUTSDocs
[2] http://www.denx.de/wiki/DULG/BoardSelect

> On the social side -- is that something the community helps out with
> or something DENX does, or a mix?

Yes, of course this is mainly a community effort. We at DENX have
only a limited number of boards in our lab (just 138 at the moment),
so all board maintainers are appealed for testing on their hardware,
and al custodians will test on a set of boards they have access to.

> Are you using a test framework of some kind, either home grown or open
> source?  These things tend to become home grown over time :)

Here at DENX we use DUTS, though not as much as we should do ...

> This kind of testing usually takes the form of 'chat' scripts
> communicating over serial consoles.  Perhaps you are using expect,
> pexpect, python nose?

DUTS is based on expect (and thus tcl, sigh!)


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"Free markets select for winning solutions."        - Eric S. Raymond

[U-Boot] U-Boot Regression Testing

[Curt Brune]

----- Original Message -----
> Have a look at DUTS [1]; the test cases implemented there serve a
> double purpose: 1) they help testing (some of) the functionality of
> U-Boot (and Linux) on a specific board, and 2) they generate the
> needed include files to produce a board-specific version of the DULG
> [2]
> 
> [1] http://www.denx.de/wiki/DUTS/DUTSDocs
> [2] http://www.denx.de/wiki/DULG/BoardSelect

Thanks.  That was the type of info I was curious about.  Very helpful.

Cheers,
Curt

[U-Boot] U-Boot Regression Testing

[Lukasz Majewski]

Hi Curt,

Sorry for late reply.

> Hello,
> 
> I have some questions about how U-Boot regression testing works.  I am
> assuming some regression testing happens during the release period
> across some representative sample of boards and architectures.

I can only present my approach - I test the board (TRATS) often before
the official u-boot release.

I don't have any list of commands to test -> I just test the ones often
used (mm) or the one to which I contribute (pmic, dfu, ums).

> 
> I know people check for compilation failures with MAKEALL, but I am
> wondering about runtime testing.  I also understand that testing
> platform specific boot code is rather difficult (or easy depending on
> your perspective) - it boots or it does not boot.
> 
> To be concrete -- how are core U-Boot commands and features tested?
> For example how do you test that FIT image support is not broken or
> that the 'env' command and all its options work properly?

As with DFU or UMS - I have got an automated test script for HOST PC to
test if data (including corner cases data size) is written and read
correctly from the eMMC medium. However those tests require typing "dfu
mmc 0" or "ums 0" commands on the target.

This posts reminds me that I shall post them....

> 
> Googling did not turn up much on how this is done.
> 
> On the social side -- is that something the community helps out with
> or something DENX does, or a mix?
> 
> Are you using a test framework of some kind, either home grown or open
> source?  These things tend to become home grown over time :)

I think that such scripts shall be added to ./test directory.

> 
> This kind of testing usually takes the form of 'chat' scripts
> communicating over serial consoles.  Perhaps you are using expect,
> pexpect, python nose?

Unfortunately I don't use serial console. I even thought of writing a
special USB gadget (or extent DFU) to provide a convenient way to test
if e.g. eMMC write works:

- Read file from FS via e.g. ext4load
- Calculate CRC (crc)
- Store the file with other name
- Read it 
- Calculate CRC and compare

I'm also curious how other perform their tests. Since I _really_ want
to avoid over-engineered solution. Maybe someone do it better, simpler?

> 
> We have a project, of which U-Boot is a part, that is starting to span
> multiple boards and architectures.  We make a few modifications to
> U-Boot and I want to start automated regression testing as the number
> of boards increases.

As I have written above. The best approach would be to use only board
under test and host PC. 
However it might be necessary to use the uC based "interface" board [1]
connected to HOST PC to perform power cycle (when we cannot use
u-boot's "reset" command).

> 
> If an existing framework exists that folks are happy with I would love
> to hear about it. 

+1 here

> Equally, I am interested to hear about what did
> *not* work for people. 

I would like to avoid using the [1]. And night run tests would be also
very welcome.

> Whatever method our project uses the scripts
> will be publicly available.
> 
> Cheers,
> Curt



-- 
Best regards,

Lukasz Majewski

Samsung R&D Institute Poland (SRPOL) | Linux Platform Group