[U-Boot] U-Boot: using tbot for U-Boot tests was: [PATCH V2 1/7] test/py: Implement pytest infrastructure

[Heiko Schocher <hs@denx.de>]

Hello Simon,

Am 15.01.2016 um 00:12 schrieb Simon Glass:
> Hi Heiko,
>
> On 16 December 2015 at 22:45, Heiko Schocher <hs@denx.de> wrote:
>> Hello Stephen,
>>
>>
>> Am 16.12.2015 um 17:27 schrieb Stephen Warren:
>>>
>>> On 12/16/2015 08:11 AM, Michal Simek wrote:
>>>>
>>>> On 9.12.2015 17:32, Stephen Warren wrote:
>>>>>
>>>>> On 12/02/2015 03:18 PM, Stephen Warren wrote:
[...]
>>> In my current setup I leave the board on all the time (or rather, manually
>>> turn on the power when
>>> I'm about to run the tests). Automating control of the power source is a
>>> step I'll take later.
>>
>>
>> Maybe you give tbot (I mentioned it in this thread) a chance?
>>
>> There, this things are automated, also you can do linux (and other console
>> based)
>> tests ... Currently added a testcase [1] which adds patches from patchwork,
>> which are in a users ToDo list, to a git tree! In this case it is a u-boot
>> git
>> tree ... checks them with checkpatch, compiles it, and tries the new image
>> on
>> the board and calls testcases ... fully automated in a now weekly build [2]
>> ...
>> (only weekly, but thats a setup parameter in buildbot, as I have tbot and
>> buildbot
>> running on a raspberry pi)
>> and don;t forget, I have the board not where I run tbot, the boards are
>> ~1000km
>> from me .. So, it is possible to add a U-Boot Testsetup on a server,
>> and test boards all over the world ...
>
> This sounds like a great development.

Thanks!

> How can we get this so that it can be used by U-Boot people? Do you
> think you could add a README to the mainline, or some scripts to aid
> setting it up? I would be interested in setting up a few boards that
> run continuous testing, and I suspect others would also if it were
> easier.

Yes, good idea!

I think about preparing ASAP a patch for U-Boot, creating:
u-boot:/tools/tbot/README -> common infos about tbot
u-boot:/tools/tbot/README.install -> HowTo install / using it
u-boot:/tools/tbot/README.create_a_testcase
u-boot:/tools/tbot/README-ToDo -> my current list of ToDo

Is this Ok?

As a motivation for using tbot ;-)  I just created a video from tbot
running the smartweb testcase:
https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_smartweb.py

https://www.youtube.com/watch?v=ZwUA0QNDnP4

Keep in mind, that I run tbot for the video here on my laptop at my home
in hungary, the smartweb board is in munich/germany.

I use this testcase also in my weekly buildbot setup on my raspberry pi [2].
Tbot logs (not only U-Boot also linux tests) can be found there for
interested people
(Remark: Wolfgang said, the logs are unreadable, because filled with a
lot of unnecessary developer output, Yes, he is correct!! I have on my
ToDo list, to add a new logging level, where only board input/output is
printed ... the loglevel tbot uses is definded in the board config file
https://github.com/hsdenx/tbot/blob/master/tbot_smartweb.cfg#L13 )

What is done in the smartweb testcase:
- rm old u-boot code, checkout current U-Boot master
   @ 00:14
- set a toolchain
   @ 00:18
- get all patchwork patches from my patchwork ToDo list
   calling testcase:
   https://github.com/hsdenx/tbot/blob/master/src/tc/tc_workfd_get_patchwork_number_list.py
   @ 00:18
- adding some patchwork patches, I have in a python list
   (This list is setup in the board config file:
    https://github.com/hsdenx/tbot/blob/master/tbot_smartweb.cfg#L34
   (Heiko speculating: it would be nice if tbot removes from
    this list a patchworkpatchnumber, if it detects, that this patch
    is already now in mainline ... )
- apply local patches (If there are)
- apply the patchwork patches
   https://github.com/hsdenx/tbot/blob/master/src/tc/tc_workfd_apply_patchwork_patches.py
   currently not failing, when checkpatch detects errors/warnings
   @ 00:59
- compile U-Boot
   @ 02:35
- update SPL on the board
   @ 03:26
- update U-Boot on the board
   @ 03:55
- start DFU testcase on the board
   @ 04:13
   https://github.com/hsdenx/tbot/blob/master/src/tc/tc_ub_dfu.py

   This testcase starts the "dfu" U-Boot command, which waits until Ctrl-C
   is pressed
   Then I start on the lab PC the Userspace tool "dfu-util" which communicates
   over USB with the smartweb board ... and do some dfu up and downloads.

- at the end save the SPL and U-Boot bins, so I always have the lastest
   working bins [1]

- power off the board

bye,
Heiko

[1] saving the latest working bins is interesting, because if a current
   U-Boot does not work on he board, I :
   - can restore the board with a debugger through a testcase, using the
     latest working bins.
   - and/or I can start a testcase which starts "git bisect" testcase:
     https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_git_bisect.py

     This testcase needs 3 variables:
     tb.board_git_bisect_get_source_tc: Name of testcase which get/switches
       into the source tree you want to start a git bisect session
     tb.board_git_bisect_call_tc: Name of testcase which gets started,
       when "git bisect" waits for good or bad ... This testcase must
       find out if current source is good or bad.
     tb.board_git_bisect_good_commit: last working bins (Therefore I save
       the bins -> so I have the commit ;-)

     This testcase is independent from U-Boot ... you can also use it
     in a linux tree or other source code ...

   I used this testcase for example here:
   http://xeidos.ddns.net/buildbot/builders/tqm5200s/builds/3/steps/shell/logs/tbotlog
   at the end it calls "git bisect log" (search for this string), and
   it found the first bad commit, and I did nothing else as starting
   this testcase :-D

[2]http://xeidos.ddns.net/buildbot/tgrid
    http://xeidos.ddns.net/buildbot/builders/smartweb_dfu
    (If you do not see a webpage, reload it ... my DSL upload speed
     is not the fastest, also if my kids play ps4 games, it is busy)
-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany