From mboxrd@z Thu Jan 1 00:00:00 1970 From: Heiko Schocher Date: Sat, 16 Jan 2016 07:29:25 +0100 Subject: [U-Boot] U-Boot: using tbot for U-Boot tests was: [PATCH V2 1/7] test/py: Implement pytest infrastructure In-Reply-To: References: <1449094708-14784-1-git-send-email-swarren@wwwdotorg.org> <56685784.9040907@wwwdotorg.org> <56717F09.8020407@monstr.eu> <567190E2.5010308@wwwdotorg.org> <56724BFB.5070502@denx.de> Message-ID: <5699E345.7000002@denx.de> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: u-boot@lists.denx.de Hello Simon, Am 15.01.2016 um 00:12 schrieb Simon Glass: > Hi Heiko, > > On 16 December 2015 at 22:45, Heiko Schocher wrote: >> Hello Stephen, >> >> >> Am 16.12.2015 um 17:27 schrieb Stephen Warren: >>> >>> On 12/16/2015 08:11 AM, Michal Simek wrote: >>>> >>>> On 9.12.2015 17:32, Stephen Warren wrote: >>>>> >>>>> On 12/02/2015 03:18 PM, Stephen Warren wrote: [...] >>> In my current setup I leave the board on all the time (or rather, manually >>> turn on the power when >>> I'm about to run the tests). Automating control of the power source is a >>> step I'll take later. >> >> >> Maybe you give tbot (I mentioned it in this thread) a chance? >> >> There, this things are automated, also you can do linux (and other console >> based) >> tests ... Currently added a testcase [1] which adds patches from patchwork, >> which are in a users ToDo list, to a git tree! In this case it is a u-boot >> git >> tree ... checks them with checkpatch, compiles it, and tries the new image >> on >> the board and calls testcases ... fully automated in a now weekly build [2] >> ... >> (only weekly, but thats a setup parameter in buildbot, as I have tbot and >> buildbot >> running on a raspberry pi) >> and don;t forget, I have the board not where I run tbot, the boards are >> ~1000km >> from me .. So, it is possible to add a U-Boot Testsetup on a server, >> and test boards all over the world ... > > This sounds like a great development. Thanks! > How can we get this so that it can be used by U-Boot people? Do you > think you could add a README to the mainline, or some scripts to aid > setting it up? I would be interested in setting up a few boards that > run continuous testing, and I suspect others would also if it were > easier. Yes, good idea! I think about preparing ASAP a patch for U-Boot, creating: u-boot:/tools/tbot/README -> common infos about tbot u-boot:/tools/tbot/README.install -> HowTo install / using it u-boot:/tools/tbot/README.create_a_testcase u-boot:/tools/tbot/README-ToDo -> my current list of ToDo Is this Ok? As a motivation for using tbot ;-) I just created a video from tbot running the smartweb testcase: https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_smartweb.py https://www.youtube.com/watch?v=ZwUA0QNDnP4 Keep in mind, that I run tbot for the video here on my laptop at my home in hungary, the smartweb board is in munich/germany. I use this testcase also in my weekly buildbot setup on my raspberry pi [2]. Tbot logs (not only U-Boot also linux tests) can be found there for interested people (Remark: Wolfgang said, the logs are unreadable, because filled with a lot of unnecessary developer output, Yes, he is correct!! I have on my ToDo list, to add a new logging level, where only board input/output is printed ... the loglevel tbot uses is definded in the board config file https://github.com/hsdenx/tbot/blob/master/tbot_smartweb.cfg#L13 ) What is done in the smartweb testcase: - rm old u-boot code, checkout current U-Boot master @ 00:14 - set a toolchain @ 00:18 - get all patchwork patches from my patchwork ToDo list calling testcase: https://github.com/hsdenx/tbot/blob/master/src/tc/tc_workfd_get_patchwork_number_list.py @ 00:18 - adding some patchwork patches, I have in a python list (This list is setup in the board config file: https://github.com/hsdenx/tbot/blob/master/tbot_smartweb.cfg#L34 (Heiko speculating: it would be nice if tbot removes from this list a patchworkpatchnumber, if it detects, that this patch is already now in mainline ... ) - apply local patches (If there are) - apply the patchwork patches https://github.com/hsdenx/tbot/blob/master/src/tc/tc_workfd_apply_patchwork_patches.py currently not failing, when checkpatch detects errors/warnings @ 00:59 - compile U-Boot @ 02:35 - update SPL on the board @ 03:26 - update U-Boot on the board @ 03:55 - start DFU testcase on the board @ 04:13 https://github.com/hsdenx/tbot/blob/master/src/tc/tc_ub_dfu.py This testcase starts the "dfu" U-Boot command, which waits until Ctrl-C is pressed Then I start on the lab PC the Userspace tool "dfu-util" which communicates over USB with the smartweb board ... and do some dfu up and downloads. - at the end save the SPL and U-Boot bins, so I always have the lastest working bins [1] - power off the board bye, Heiko [1] saving the latest working bins is interesting, because if a current U-Boot does not work on he board, I : - can restore the board with a debugger through a testcase, using the latest working bins. - and/or I can start a testcase which starts "git bisect" testcase: https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_git_bisect.py This testcase needs 3 variables: tb.board_git_bisect_get_source_tc: Name of testcase which get/switches into the source tree you want to start a git bisect session tb.board_git_bisect_call_tc: Name of testcase which gets started, when "git bisect" waits for good or bad ... This testcase must find out if current source is good or bad. tb.board_git_bisect_good_commit: last working bins (Therefore I save the bins -> so I have the commit ;-) This testcase is independent from U-Boot ... you can also use it in a linux tree or other source code ... I used this testcase for example here: http://xeidos.ddns.net/buildbot/builders/tqm5200s/builds/3/steps/shell/logs/tbotlog at the end it calls "git bisect log" (search for this string), and it found the first bad commit, and I did nothing else as starting this testcase :-D [2]http://xeidos.ddns.net/buildbot/tgrid http://xeidos.ddns.net/buildbot/builders/smartweb_dfu (If you do not see a webpage, reload it ... my DSL upload speed is not the fastest, also if my kids play ps4 games, it is busy) -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany