[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[Jerry Van Baren <gerald.vanbaren@ge.com>]

Wolfgang Denk wrote:
> Hello,
> 
> in message <20071124010338.GA27628@cideas.com> you wrote:
>> ...and now, for something completely different.
>>
>> The concept here is to use PyUnit testing in conjunction with python
>> serial I/O (theoretically, ethernet-based command I/O could be handled
>> too).
>>
>> This is a concept probably sufficiently implemented for the "help"
>> command and a good start for the "fdt" command.  
>>
>> What does The List think?
>> * Useful to have regression tests for the u-boot commands?
> 
> It is not only useful, but IMHO absolutely necessary to do regular
> regression testing on U-Boot.
> 
>> * Implementation-wise, am I heading in the right direction?
> 
> I wish you had asked that question before spending efforts  on  this.
> The point is, that there is already a pretty extensive and extandable
> regression  test suite for U-Boot (and Linux, and some more). We just
> failed to announce it loudly enough (which clearly shows that DENX is
> not marketing driven - and funny enough, I'm not a bit ashamed  about
> that).  However,  I'm sad about the avoidable duplication of efforts,
> so I have to say I'm sorry.

Oh no apology necessary.  It was a learning exercise, useful in its own 
right, WRT Python, PyUnit, PySerial, and seeing what I could do with them.

I've used tk/tcl/expect and am not wild about them.  The older scripting 
languages are so.... 70s. ;-)  I haven't used python much, but I've come 
to like it a lot, kind of a rational perl. :-/

My experience with expect is that it worked well for simple cases, but 
when I started matching more patterns and more complex patterns, it 
would get harder and harder to match reliably.  Also, its asynchronous 
timing-based capture (which causes "capture breaks" in different places 
when you least expect it) drove me crazy.  Then to debug the pattern 
matching (or not matching, or mismatching, as the case usually was) was 
difficult.

Having said that, that was a few years ago and maybe I'm smarter now...

> Please have a look at the DUTS, our regression test suite. See the
> git repo at http://www.denx.de/cgi-bin/gitweb.cgi?p=duts.git;a=summary
> and some documentation at http://www.denx.de/wiki/DUTS/DUTSDocs
> 
> I'd really appreciate if this was picked up by others  as  well.  [It
> took  us  a  long and winded road to get there - trying some existing
> solutions like dejagnu etc. and finally ending with a  do-it-yourself
> solution.]
> 
> Note: there is a non-abvious benefit from the DUTS: once you  got  it
> running  on  a  board, it will generate a set of log files which just
> need to be uploaded to our web server to generate a  new  version  of
> the  DULG for this board. The idea is that if you have a new software
> version you just need to run the DUTS to get (1) a confirmation  that
> everything  is working as expected fromt he regression test suite and
> (2) an updated version of the documentation  with  eexamples  showing
> exacly this software version on exactly that hardware.
> 
> Best regards,
> Wolfgang Denk

Thanks for the pointer.  I've poked around a bit and will continue to do so.

Best regards,
gvb