[U-Boot] [PATCH] Implement pytest-based test infrastructure

[Stephen Warren <swarren@wwwdotorg.org>]

On 11/19/2015 07:45 AM, Simon Glass wrote:
> Hi Stephen,
>
> On 14 November 2015 at 23:53, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> This tool aims to test U-Boot by executing U-Boot shell commands using the
>> console interface. A single top-level script exists to execute or attach
>> to the U-Boot console, run the entire script of tests against it, and
>> summarize the results. Advantages of this approach are:
>>
>> - Testing is performed in the same way a user or script would interact
>>    with U-Boot; there can be no disconnect.
>> - There is no need to write or embed test-related code into U-Boot itself.
>>    It is asserted that writing test-related code in Python is simpler and
>>    more flexible that writing it all in C.
>> - It is reasonably simple to interact with U-Boot in this way.
>>
>> A few simple tests are provided as examples. Soon, we should convert as
>> many as possible of the other tests in test/* and test/cmd_ut.c too.
>
> It's great to see this and thank you for putting in the effort!
>
> It looks like a good way of doing functional tests. I still see a role
> for unit tests and things like test/dm. But if we can arrange to call
> all U-Boot tests (unit and functional) from one 'test.py' command that
> would be a win.
>
> I'll look more when I can get it to work - see below.
...
> I get this on my Ubuntu 64-bit machine (14.04.3)
>
> $ ./test/py/test.py --bd sandbox --buildTraceback (most recent call last):
>    File "./test/py/test.py", line 12, in <module>
>      os.execvp("py.test", args)
>    File "/usr/lib/python2.7/os.py", line 344, in execvp
>      _execvpe(file, args)
>    File "/usr/lib/python2.7/os.py", line 380, in _execvpe
>      func(fullname, *argrest)
> OSError: [Errno 2] No such file or directory

"py.test" isn't in your $PATH. Did you install it? See the following in 
test/py/README.md:

> ## Requirements
>
> The test suite is implemented using pytest. Interaction with the U-Boot
> console uses pexpect. Interaction with real hardware uses the tools of your
> choice; you get to implement various "hook" scripts that are called by the
> test suite at the appropriate time.
>
> On Debian or Debian-like distributions, the following packages are required.
> Similar package names should exist in other distributions.
>
> | Package        | Version tested (Ubuntu 14.04) |
> | -------------- | ----------------------------- |
> | python         | 2.7.5-5ubuntu3                |
> | python-pytest  | 2.5.1-1                       |
> | python-pexpect | 3.1-1ubuntu0.1                |

In the main Python code, I trapped at least one exception location and 
made it print a message about checking the docs for missing 
requirements. I can probably patch the top-level test.py to do the same.