[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

Jerry Van Baren
2007-11-26T13:49:32Z

Wolfgang Denk wrote:
> Hello,
>
> in message <20071124010338.GA27628@cideas.com> you wrote:
>> ...and now, for something completely different.
>>
>> The concept here is to use PyUnit testing in conjunction with python
>> serial I/O (theoretically, ethernet-based command I/O could be handled
>> too).
>>
>> This is a concept probably sufficiently implemented for the "help"
>> command and a good start for the "fdt" command.
>>
>> What does The List think?
>> * Useful to have regression tests for the u-boot commands?
>
> It is not only useful, but IMHO absolutely necessary to do regular
> regression testing on U-Boot.
>
>> * Implementation-wise, am I heading in the right direction?
>
> I wish you had asked that question before spending efforts on this.
> The point is, that there is already a pretty extensive and extandable
> regression test suite for U-Boot (and Linux, and some more). We just
> failed to announce it loudly enough (which clearly shows that DENX is
> not marketing driven - and funny enough, I'm not a bit ashamed about
> that). However, I'm sad about the avoidable duplication of efforts,
> so I have to say I'm sorry.

Oh no apology necessary. It was a learning exercise, useful in its own
right, WRT Python, PyUnit, PySerial, and seeing what I could do with them.

I've used tk/tcl/expect and am not wild about them. The older scripting
languages are so.... 70s. ;-) I haven't used python much, but I've come
to like it a lot, kind of a rational perl. :-/

My experience with expect is that it worked well for simple cases, but
when I started matching more patterns and more complex patterns, it
would get harder and harder to match reliably. Also, its asynchronous
timing-based capture (which causes "capture breaks" in different places
when you least expect it) drove me crazy. Then to debug the pattern
matching (or not matching, or mismatching, as the case usually was) was
difficult.

Having said that, that was a few years ago and maybe I'm smarter now...

> Please have a look at the DUTS, our regression test suite. See the
> git repo at http://www.denx.de/cgi-bin/gitweb.cgi?p=duts.git;a=summary
> and some documentation at http://www.denx.de/wiki/DUTS/DUTSDocs
>
> I'd really appreciate if this was picked up by others as well. [It
> took us a long and winded road to get there - trying some existing
> solutions like dejagnu etc. and finally ending with a do-it-yourself
> solution.]
>
> Note: there is a non-abvious benefit from the DUTS: once you got it
> running on a board, it will generate a set of log files which just
> need to be uploaded to our web server to generate a new version of
> the DULG for this board. The idea is that if you have a new software
> version you just need to run the DUTS to get (1) a confirmation that
> everything is working as expected fromt he regression test suite and
> (2) an updated version of the documentation with eexamples showing
> exacly this software version on exactly that hardware.
>
> Best regards,
> Wolfgang Denk

Thanks for the pointer. I've poked around a bit and will continue to do so.

Best regards,
gvb