[U-Boot] [PATCH V2 1/7] test/py: Implement pytest infrastructure

[Stephen Warren <swarren@wwwdotorg.org>]

On 12/19/2015 03:24 PM, Simon Glass wrote:
> Hi Stephen,
>
> On 2 December 2015 at 15:18, Stephen Warren <swarren@wwwdotorg.org> wrote:
>> This tool aims to test U-Boot by executing U-Boot shell commands using the
>> console interface. A single top-level script exists to execute or attach
>> to the U-Boot console, run the entire script of tests against it, and
>> summarize the results. Advantages of this approach are:
>>
>> - Testing is performed in the same way a user or script would interact
>>    with U-Boot; there can be no disconnect.
>> - There is no need to write or embed test-related code into U-Boot itself.
>>    It is asserted that writing test-related code in Python is simpler and
>>    more flexible that writing it all in C.
>> - It is reasonably simple to interact with U-Boot in this way.
>>
>> A few simple tests are provided as examples. Soon, we should convert as
>> many as possible of the other tests in test/* and test/cmd_ut.c too.
>>
>> In the future, I hope to publish (out-of-tree) the hook scripts, relay
>> control utilities, and udev rules I will use for my own HW setup.
>>
>> See README.md for more details!

> This is a huge step forward for testing in U-Boot. Congratulations on
> putting this together!
>
> Tested on chromebook_link, sandbox
> Tested-by: Simon Glass <sjg@chromium.org>
>
> I've made various comments in the series as I think it needs a little
> tuning. I'm also interested in how we can arrange for the existing
> unit tests to be run (and results supported) by this framework.
>
> One concern I have is about the ease of running and writing tests. It
> is pretty easy at present to run a particular driver model test:
>
> ./u-boot -d test.dtb -c "ut dm uclass"
>
> and we can run this in gdb and figure out where things are going wrong
> (I do this quite a bit). Somehow we need to preserve this ease of use.
> The tests should be accessible. I'm not sure how you intend to make
> that work.

You can select which individual tests to run by passing "-k testname" on 
the command-line. Assuming Python-level tests have the desired 
granularity, that should be enough for test selection.

There's currently no support for running U-Boot under gdb. The acts of 
running U-Boot under gdb, disabling stdout redirection, and removing 
timeouts would be pretty easy to add. However, I'm not sure how you'd be 
able to interact with the gdb console and U-Boot output without 
interfering with the test script's need to capture the shell output in 
order to run the tests and interpret the results. Perhaps simplest would 
be to add some mechanism to pause the test process so that you could 
manually attach a gdb process externally at the appropriate time, 
perhaps along with the test process automatically spawning another 
shell/terminal/... to host the gdb (in which case it could be made to 
automatically attach to the correct process).

>> diff --git a/test/py/.gitignore b/test/py/.gitignore

>> +## Testing sandbox
>> +
>> +To run the testsuite on the sandbox port (U-Boot built as a native user-space
>> +application), simply execute:
>> +
>> +```
>> +./test/py/test.py --bd sandbox --build
>> +```
>> +
>> +The `--bd` option tells the test suite which board type is being tested. This
>> +lets the test suite know which features the board has, and hence exactly what
>> +can be tested.
>
> Can we use -b to fit in with buildman and patman?

pytest reserves all lower-case single-letter options for itself, so we 
cannot use any of those. I suppose we could write a wrapper script to 
translate from -b to --board etc., but that would be a bit messy and 
prevent access to the shadowed pytest options.

>> diff --git a/test/py/conftest.py b/test/py/conftest.py

>> +def pytest_configure(config):
>
> This series should have function comments throughout on non-trivial
> functions - e.g. purpose of the function and a description of the
> parameters and return value.

I can see this makes sense in some cases, but this particular function 
is a standard pytest function and already documented on 
http://pytest.org/. Would you still want additional documentation even 
for such functions?

>> diff --git a/test/py/uboot_console_base.py b/test/py/uboot_console_base.py

>> +class ConsoleBase(object):

>> +    def ensure_spawned(self):
>> +        if self.p:
>> +            return
>> +        try:
>> +            self.at_prompt = False
>> +            self.log.action("Starting U-Boot")
>> +            self.p = self.get_spawn()
>> +            # Real targets can take a long time to scroll large amounts of
>> +            # text if LCD is enabled. This value may need tweaking in the
>> +            # future, possibly per-test to be optimal. This works for "help"
>> +            # on board "seaboard".
>> +            self.p.timeout = 30000
>> +            self.p.logfile_read = self.logstream
>
> Also I have found that tests fail on chromebook_link because it cannot
> keep up with the pace of keyboard input. I'm not sure what the
> solution is - maybe the best thing is to implement buffering in the
> serial uclass, assuming that fixes it. For now I disabled LCD output.
>
> I think it would be worth adding a test that checks for the banner and
> the prompt, so we know that other test failures are not due to this
> problem.

I had the same problem on Seaboard. I solved that by having the "expect" 
code wait for command echo bit-by-bit so that the host's TX side could 
never overflow the target's RX side FIFOs. Perhaps try lowering the 
max_fifo_fill value in test/py/uboot_console_exec_attach.py; does that 
solve the issue?