[U-Boot] U-Boot: using tbot for U-Boot tests was: [PATCH V2 1/7] test/py: Implement pytest infrastructure

[Heiko Schocher <hs@denx.de>]

Hello Simon,

Am 19.01.2016 um 04:42 schrieb Simon Glass:
> Hi Heiko,
>
> On 15 January 2016 at 23:29, Heiko Schocher <hs@denx.de> wrote:
>> Hello Simon,
>>
>> Am 15.01.2016 um 00:12 schrieb Simon Glass:
>>>
>>> Hi Heiko,
>>>
>>> On 16 December 2015 at 22:45, Heiko Schocher <hs@denx.de> wrote:
>>>>
>>>> Hello Stephen,
>>>>
>>>>
>>>> Am 16.12.2015 um 17:27 schrieb Stephen Warren:
>>>>>
>>>>>
>>>>> On 12/16/2015 08:11 AM, Michal Simek wrote:
>>>>>>
>>>>>>
>>>>>> On 9.12.2015 17:32, Stephen Warren wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 12/02/2015 03:18 PM, Stephen Warren wrote:
>>
>> [...]
>>>>>
>>>>> In my current setup I leave the board on all the time (or rather,
>>>>> manually
>>>>> turn on the power when
>>>>> I'm about to run the tests). Automating control of the power source is a
>>>>> step I'll take later.
>>>>
>>>>
>>>>
>>>> Maybe you give tbot (I mentioned it in this thread) a chance?
>>>>
>>>> There, this things are automated, also you can do linux (and other
>>>> console
>>>> based)
>>>> tests ... Currently added a testcase [1] which adds patches from
>>>> patchwork,
>>>> which are in a users ToDo list, to a git tree! In this case it is a
>>>> u-boot
>>>> git
>>>> tree ... checks them with checkpatch, compiles it, and tries the new
>>>> image
>>>> on
>>>> the board and calls testcases ... fully automated in a now weekly build
>>>> [2]
>>>> ...
>>>> (only weekly, but thats a setup parameter in buildbot, as I have tbot and
>>>> buildbot
>>>> running on a raspberry pi)
>>>> and don;t forget, I have the board not where I run tbot, the boards are
>>>> ~1000km
>>>> from me .. So, it is possible to add a U-Boot Testsetup on a server,
>>>> and test boards all over the world ...
>>>
>>>
>>> This sounds like a great development.
>>
>>
>> Thanks!
>>
>>> How can we get this so that it can be used by U-Boot people? Do you
>>> think you could add a README to the mainline, or some scripts to aid
>>> setting it up? I would be interested in setting up a few boards that
>>> run continuous testing, and I suspect others would also if it were
>>> easier.
>>
>>
>> Yes, good idea!
>>
>> I think about preparing ASAP a patch for U-Boot, creating:
>> u-boot:/tools/tbot/README -> common infos about tbot
>> u-boot:/tools/tbot/README.install -> HowTo install / using it
>> u-boot:/tools/tbot/README.create_a_testcase
>> u-boot:/tools/tbot/README-ToDo -> my current list of ToDo
>>
>> Is this Ok?
>
> Sounds great.

Ok, sorry, I had no time yet, to making it ...

>> As a motivation for using tbot ;-)  I just created a video from tbot
>> running the smartweb testcase:
>> https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_smartweb.py
>>
>> https://www.youtube.com/watch?v=ZwUA0QNDnP4
>
> OK that helps explain it. It looks like it uses DFU to write new
> images to the boards - is that right?

No, the board gets the new images through tftp ...
I only test U-Boot DFU functionality on this board.

>> Keep in mind, that I run tbot for the video here on my laptop at my home
>> in hungary, the smartweb board is in munich/germany.
>>
>> I use this testcase also in my weekly buildbot setup on my raspberry pi [2].
>> Tbot logs (not only U-Boot also linux tests) can be found there for
>> interested people
>> (Remark: Wolfgang said, the logs are unreadable, because filled with a
>> lot of unnecessary developer output, Yes, he is correct!! I have on my
>> ToDo list, to add a new logging level, where only board input/output is
>> printed ... the loglevel tbot uses is definded in the board config file
>> https://github.com/hsdenx/tbot/blob/master/tbot_smartweb.cfg#L13 )
>
> Yes that's my main comment.

Yes, on my todo list ... BTW: is the console output in the video
helpful?

>> What is done in the smartweb testcase:
>> - rm old u-boot code, checkout current U-Boot master
>>    @ 00:14
>> - set a toolchain
>>    @ 00:18
>> - get all patchwork patches from my patchwork ToDo list
>>    calling testcase:
>>
>> https://github.com/hsdenx/tbot/blob/master/src/tc/tc_workfd_get_patchwork_number_list.py
>>    @ 00:18
>> - adding some patchwork patches, I have in a python list
>>    (This list is setup in the board config file:
>>     https://github.com/hsdenx/tbot/blob/master/tbot_smartweb.cfg#L34
>>    (Heiko speculating: it would be nice if tbot removes from
>>     this list a patchworkpatchnumber, if it detects, that this patch
>>     is already now in mainline ... )
>> - apply local patches (If there are)
>> - apply the patchwork patches
>>
>> https://github.com/hsdenx/tbot/blob/master/src/tc/tc_workfd_apply_patchwork_patches.py
>>    currently not failing, when checkpatch detects errors/warnings
>>    @ 00:59
>> - compile U-Boot
>>    @ 02:35
>> - update SPL on the board
>>    @ 03:26
>> - update U-Boot on the board
>>    @ 03:55
>> - start DFU testcase on the board
>>    @ 04:13
>>    https://github.com/hsdenx/tbot/blob/master/src/tc/tc_ub_dfu.py
>>
>>    This testcase starts the "dfu" U-Boot command, which waits until Ctrl-C
>>    is pressed
>>    Then I start on the lab PC the Userspace tool "dfu-util" which
>> communicates
>>    over USB with the smartweb board ... and do some dfu up and downloads.
>>
>> - at the end save the SPL and U-Boot bins, so I always have the lastest
>>    working bins [1]
>>
>> - power off the board
>>
>> bye,
>> Heiko
>
> Well I'll await your patches to the list and then see if I can get it
> working. What sort of hardware do I need for the power / reset
> control?

Ok.

You can use what you want for powering on/off the board!

If you can power on/off the board through a linux shell command, you
are ready for tbot ...

Also you need a shell command for getting the current power state
(on or off)

And you need to tell tbot, how you get access to the boards console ...

I seperated this tasks in testcases, so you need to write for
this tasks a tbot testcase. If you want to access the serial console
with kermit, this should work already ...

But let me try to find time for writing a patch ...

bye,
Heiko
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany