[U-Boot] [PATCH V2 1/7] test/py: Implement pytest infrastructure

[Stephen Warren <swarren@wwwdotorg.org>]

On 12/18/2015 07:50 AM, Michal Simek wrote:
> Hi Stephen,
>
>>>               Finally, the example scripts support two boards; my
>>>          home/laptop dev
>>>               setup that uses a Numato relay board to control the signals
>>>          to the
>>>               board I use there, and my work desktop dev setup that
>>> uses our
>>>               "PM342" debug board to controll the signals. The latter
>>> works
>>>               logically the same as the numato relay board, except it
>>>          contains
>>>               electronic switches driven by an FTDI chip.
>>>
>>>          I expect this is FTDI chip on the target right?
>>>
>>>
>>>      It's actually a separate common debug board. Most/all of our
>>>      development boards (and perhaps some production boards) have a
>>>      standardized connector into which the common debug board plugs.
>>>
>>>
>>>
>>> ok.
>>> I think my setup is not that far from what you are using and I expect
>>> that others SoCs will be very similar.
>>> Do you have any other testcases which you are running and you haven't
>>> sent?
>>
>> Not at present.
>>
>> As an FYI, I typically publish my local work-in-progress branch at:
>> git://github.com/swarren/u-boot.git tegra_dev
>
> I have looked at your patches and no problem to get it work on
> microblaze and zynq board. I do use kermit without any problem.
> I used cu on Microblaze.

Great!

> - What I do miss is power off functionality because it is not practical
> to keep board always on. On can be solved via reset script.

Yes, I would expect that the flash or reset script would turn the board 
on. It should be easy to add an extra hook script at the end which turns 
the board off. Or, whatever automation system you use to invoke test.py 
could simply do that right after running test.py.

> - Then place tests to separate folder for better separation.

You mean e.g. test/py/tests/ ?

> - I see that output log doesn't handle tabs correctly - output from i2c
> bus for example.

OK. I can easily make the logging code replace a TAB with something 
else, e.g. a chain of  , although it will mean keeping track of the 
output character count since the last newline which will be a bit more 
painful. Let me look into this.

> - Is there any way to handle timeouts or stucks? For example to
> recognize if sleep 60 fails or just takes long. It means setting up
> timeouts will be good.

ubspawn.py:expect() does have a timeout capability, and 
uboot_console_base.py:ensure_spawned() sets this to 30s by default. 
There isn't currently any example of modifying or saving/restoring the 
timeout, or running commands that are expected to have a timeout, 
although either should be pretty easy to add. I expect the result would 
look something like this in a test:

with uboot_console.push_timeout(60000 + some_margin):
     uboot_console.run_command("sleep 60")
# Perhaps the actual time taken should be validated here too

with uboot_console.timeout_is_expected(10000):
     # code that is expected to time out
# Perhaps the following command would be integrated into the
# timeout_is_expected() implementation, since I think it's the only
# way you could recover from this situation?
uboot_console.ctrlc()

... both modelled after the existing uboot_console.disable_check() code.

> I will have more comments when I spend more time with it but it looks
> pretty good for start.

Thanks.