[Buildroot] Buildroot runtime test infrastructure prototype

[Andreas Naumann <dev@andin.de>]

Hi Thomas,

Am 28.06.2015 um 11:50 schrieb Thomas Petazzoni:

> the answers was Robot Framework. So I had a quick look, and definitely
> could not understand how it can work: you are apparently not writing
> the tests in some real programming language, but in some sort of weird
> "tabular" format. I really don't understand how you can express
> complicated test scenarios with such a limited language.
>
> The Python unittest stuff just runs Python code, so you can express
> whatever complicated test logic you want.

RFW also runs plain Python code, the only difference is, it does it 
exclusively via its internal, external or user-written libraries. The 
libraries are simply Python classes with function calls like 
open_connection(self, host, port=23, prompt, ...) or 
execute_command(self, command, loglevel=None) .
In the RFWs tabular format this then looks like
Open Connection | 192.168.0.37 | prompt=[root]#
Login           | root         | pw
Execute Command | echo Hello World

which in an HTML table is quite readable. RFW calls it keywords. You can 
also create custom keywords consisting of any other keywords.
So while the libraries are the equivalent of the fixtures that Thomas S. 
was talking about, you can group keywords that form certain steps into 
domain specific resource files, like filesystem, package, qemu...
Using those higher level keywords you then can hide the lengthy 
parameter set from the top level testcase table, which leads e.g. to 
something like:
Buildroot.Add Package To Defconfig | dropbear
Buildroot.Compile
Qemu.Start System
Telnet.Connect And Login
Command Output Should Contain | netstat -ltn 2 | 0.0.0.0:22

There is another resource type, Variables, which allows for 
configuration of keywords and testcases but before explaining any 
longer, I'll try to come up with a working example...

regards,
Andreas