From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Jan-Simon =?ISO-8859-1?Q?M=F6ller?= Date: Wed, 01 Feb 2017 11:46:21 +0100 Message-ID: <4759757.Uek3Dlbnj3@elrond> In-Reply-To: <003201d27c25$f68f8240$e3ae86c0$@toshiba.co.jp> References: <000d01d26ba1$94c8a5d0$be59f170$@toshiba.co.jp> <2787487.KsR9eGO5q9@elrond> <003201d27c25$f68f8240$e3ae86c0$@toshiba.co.jp> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="iso-8859-1" Subject: Re: [Fuego] Fuego's version up and other changes List-Id: Mailing list for the Fuego test framework List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Daniel Sangorrin Cc: "'Bird, Timothy'" , fuego@lists.linuxfoundation.org Hi ! Am Mittwoch, 1. Februar 2017, 09:56:03 schrieb Daniel Sangorrin: > Thanks for the detailed example. A few observations: >=20 > - Normally the timeout value depends on the board's performance and t= he test > duration. In my next branch, timeouts are defined for each test in t= he > testplan_board.json file. Here it looks like the timeout needs to be = the > same for all of the tests. I was wondering if it is possible to put a= list > of timeouts (same amount as the number of tests) and then use timeout= > --signal=3D9 {timeout}. Doing this by hand would be too tedious but w= e could > generate the yaml files from the testplans using a script. Oh, that was just /me taking a shortcut. But yes, the template wouldn't= know directly the board/test specific values.=20 One way would be to make these bash env files and source them before ru= nning the script. The 'same' number of timeouts would create a matrix [ tests x timeouts= ], that is not what we're looking for. > - What would we do if we want to execute the same test with several > test_specs? I was thinking about creating separate jobs for each > combination of test-test_spec because it's the simple but it wonder i= f that > can be implemented with jjb. That would then be: - project: name: fuego-tests-smoke testplan: - smoke testspec: - testspec1 - testspec2 machine: - raspberrypi3 testname: - Benchmark.Dhrystone - Benchmark.IOzone - Functional.hello_world - Functional.ipv6connect - Functional.stress jobs: - fuego-{machine}-{testplan}-{testspec}-batch - fuego-{machine}-{testplan}-{testspec}-{testname} > Another approach would be to create a single > job and then have fuego-core take care of reading the tests specs of = each > test from the testplan, and repeat the test for each test spec. Then = a > modified parser would display and compare the results of the test acr= oss > the different test specs. > [Note] I think that AGL's fuego has a similar parser. Having all in one job let's you not execute one specific test easily.=20= But if we simplify the shell call that jenkins has to do, creating jobs= with specific names and parameters/env-variables is trivial. E.g.: builders: - shell: | # source $FUEGO_RO/conf/boards/{machine}.sh source $FUEGO_CORE/engine/env/${{DISTRO}}.sh # DISTRO= set in machine.sh source $FUEGO_CORE/engine/testplans/{testplan}.sh # taken = from jjb=20 # export Reboot=3Dfalse export Rebuild=3Dtrue export Target_Cleanup=3Dtrue export TESTDIR=3D{testname} export TESTNAME=3D$(echo "{testname}" | sed -e "s#.*\.##") timeout --signal=3D9 ${{JOB_TIMEOUT}} /bin/bash $FUEGO_CORE/e= ngine/tests/${{TESTDIR}}/${{TESTNAME}}.sh In this case, ${{JOB_TIMEOUT}} can be defined in the $FUEGO_RO/conf/boa= rds/{machine}.sh env script. > - In my next branch, test specs have the ability to define which link= s > should be available after the test is complete. I'm using the Descrip= tion > Setter plugin for that. Do you think that would be possible with the = jjb > approach?. Links ... let me see http://docs.openstack.org/infra/jenkins-job-builder/publishers.html#pub= lishers.description-setter ? should work For reference: http://docs.openstack.org/infra/jenkins-job-builder/definition.html http://docs.openstack.org/infra/jenkins-job-builder/builders.html http://docs.openstack.org/infra/jenkins-job-builder/publishers.html >=20 > > While running tests I also found: > >=20 > > > > - we need to split the build of the test and the postprocessing to = be run > > separately> > >=20 > > > > -- this is b/c it is not useful to keep to board waiting on us all = the > > time. > I wasn't aware that the board is waiting on us while we do the > post-processing. Could you elaborate a bit more? We access the board very early even before the build (pre_test). The we build, deploy, run, fetch results, analyse. At this point=20 =20 > > - Integration of board up/down could happen in a few ways: > > -- Either ppl should just generate a wrapper around the (blocking) = batch > > job and trigger it> > > with their pre/post to their needs. Done. > > -- Or we allow hooks like the current "TARGET_SETUP_LINK" and amend= it > > with a matching "TARGET_TEARDOWN_LINK"> > > - If a model like the "TARGET_SETUP_LINK" is used, we add a delay h= ere as > > this call must block until the board is up.> > > In this case our predifined timeouts are bogus as they track not ju= st the > > test run, but all processing. > Good point. I wasn't using the reboot feature so I hadn't thought abo= ut it. > I guess we would need to define a timeout for the board to reboot, an= d then > the timeout for each test. The timeout for the board could be called > BOARD_REBOOT_TIMEOUT for example and be defined in the asdf.board fil= e. And > the timeout for each test defined in another variable such as TEST_TI= MEOUT. As I said, there are multiple ways to integrate that.=20 a) The wrapper-job triggering the batch leaves fuego out of the picture= . But we need to block the whole chain to make flow-control possible. b) TARGET_SETUP_LINK gives us a _per job_ way of brining up the board. = Not just for the wrapp'ed batch. Most users might get away and be happy with a) . While a per-job contro= l as in b) might scale better if you think of the case that you could h= ave multiple target boards of the same type (multiple executors!) and could= run jobs 'in parallel' ! > BOARD_REBOOT_TIMEOUT=3D1m > TEST_TIMEOUT=3D15m > /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh >=20 > Then, internally the timeouts would be used separately. > # we could also split timeouts between execution time and post-proces= sing > time Yep ... we probably should split the build, test and postproc into sepa= rate scripts. or instead of $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh in= jenkins, we could call ftc as you would do on the terminal.=20 Also the above should be in separate jobs ... think of it: - the build phase doesn't need to block, can run in parallel,=20 e.g. on master node! - the test batch needs to block to allow flow-control and=20 run only on the specified executor. No concurrency but multiple jobs=20= of the *same* batch could run in parallel, given we can handle multip= le=20 boards of the same type at the same time (which is true with an 'orch= estrator' like lava). - the batch is only deploy/run/fetch result but can trigger postproce= ssing - the postprocessing is another job, can run in parallel, independent o= f the board, e.g. on master JS --=20 -- Jan-Simon M=F6ller dl9pf@gmx.de