[U-Boot] U-Boot Regression Testing

[Lukasz Majewski <l.majewski@samsung.com>]

Hi Curt,

Sorry for late reply.

> Hello,
> 
> I have some questions about how U-Boot regression testing works.  I am
> assuming some regression testing happens during the release period
> across some representative sample of boards and architectures.

I can only present my approach - I test the board (TRATS) often before
the official u-boot release.

I don't have any list of commands to test -> I just test the ones often
used (mm) or the one to which I contribute (pmic, dfu, ums).

> 
> I know people check for compilation failures with MAKEALL, but I am
> wondering about runtime testing.  I also understand that testing
> platform specific boot code is rather difficult (or easy depending on
> your perspective) - it boots or it does not boot.
> 
> To be concrete -- how are core U-Boot commands and features tested?
> For example how do you test that FIT image support is not broken or
> that the 'env' command and all its options work properly?

As with DFU or UMS - I have got an automated test script for HOST PC to
test if data (including corner cases data size) is written and read
correctly from the eMMC medium. However those tests require typing "dfu
mmc 0" or "ums 0" commands on the target.

This posts reminds me that I shall post them....

> 
> Googling did not turn up much on how this is done.
> 
> On the social side -- is that something the community helps out with
> or something DENX does, or a mix?
> 
> Are you using a test framework of some kind, either home grown or open
> source?  These things tend to become home grown over time :)

I think that such scripts shall be added to ./test directory.

> 
> This kind of testing usually takes the form of 'chat' scripts
> communicating over serial consoles.  Perhaps you are using expect,
> pexpect, python nose?

Unfortunately I don't use serial console. I even thought of writing a
special USB gadget (or extent DFU) to provide a convenient way to test
if e.g. eMMC write works:

- Read file from FS via e.g. ext4load
- Calculate CRC (crc)
- Store the file with other name
- Read it 
- Calculate CRC and compare

I'm also curious how other perform their tests. Since I _really_ want
to avoid over-engineered solution. Maybe someone do it better, simpler?

> 
> We have a project, of which U-Boot is a part, that is starting to span
> multiple boards and architectures.  We make a few modifications to
> U-Boot and I want to start automated regression testing as the number
> of boards increases.

As I have written above. The best approach would be to use only board
under test and host PC. 
However it might be necessary to use the uC based "interface" board [1]
connected to HOST PC to perform power cycle (when we cannot use
u-boot's "reset" command).

> 
> If an existing framework exists that folks are happy with I would love
> to hear about it. 

+1 here

> Equally, I am interested to hear about what did
> *not* work for people. 

I would like to avoid using the [1]. And night run tests would be also
very welcome.

> Whatever method our project uses the scripts
> will be publicly available.
> 
> Cheers,
> Curt



-- 
Best regards,

Lukasz Majewski

Samsung R&D Institute Poland (SRPOL) | Linux Platform Group