Re: [PATCH 0/1] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

[Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>]

[-- Attachment #1: Type: text/plain, Size: 3811 bytes --]



On Thu, Oct 26, 2017 at 1:44 PM, Anibal Limón <limon.anibal@gmail.com> 
wrote:
> Hi Leo,
> 
> The patch looks good at first glance, i have some comments,
> 
> - We need a way to handle the results output because with this you 
> will have N results files, may be extend the shell script
> to consolidate the results file.

As a proof of concept I use 'parallel' which fits nicely to this 
exercise (one job corresponds to one oe-selftest -r) and with this 
tool, logging can be stored in separate files, each corresponding to a 
job's stdout/stderr. But the important here is that the tool prints 
each job's output into the standard output  either 
first-finished-first-print-to-stdout or keeping the same input order, 
so at the end, you get the same output as running oe-selftest as a 
single job.

> - With this if one selftest depends on other there is no way to now 
> it and the execution will fail, i searched into the selftest cases
> (OETestDepends) and now we don't have dependent cases.

 Think about this approach as running oe-selftest in different build 
folders with its own metadata and build folder. Under this scenario, 
the test execution, with the corresponding dependencies will be 
fulfilled and executed correctly, right? the trade off is some extra 
work done on each oe-selftest due to dependencies but this wont hurt 
much in my opinion.

Leo

> 
> Cheers,
> Anibal
> 
> On Thu, Oct 26, 2017 at 12:33 PM, 
> <leonardo.sandoval.gonzalez@linux.intel.com> wrote:
>> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>> 
>> The below is a profiling experiment, running oe-selftest -r (the 
>> proposed
>> implementation, see patch description for more info):
>> 
>> Procedure:
>> 
>> With patch 1/1, multiple oe-selftest jobs can be launched in
>> parallel. One tool that launch jobs in parallel is GNU Parallel [1], 
>> allowing
>> to construct a simpole pipeline to execute all tests with a pool of 
>> four
>> jobs:
>> 
>>     $ echo $ALLTESTS | time parallel --jobs4 oe-selftest -r
>> 
>> where ALLTESTS is a variable containing all tests cases (modules) 
>> found by the
>> the runner (oe-selftest-internal) (i.e. ALLTESTS="$(oe-selftest -m |
>> awk '{ print $NF }' | grep -v ':')"). This is the result obtained 
>> from the
>> above command:
>> 
>>     739.57user 120.48system 45:34.61elapsed 31%CPU 
>> (0avgtext+0avgdata 124600maxresident)k
>>     390908inputs+15984336outputs (291major+20227951minor)pagefaults 
>> 0swaps
>> 
>> 
>> The import point on the above numbers is that isolation the 
>> oe-selftest execution per
>> module and using a parallelization tool, complete oe-selftest runs 
>> takes less than an hour,
>> beating current single-job times observed at main auto-buildes.
>> 
>> Profiling results were obtained on a machine with 88 Intel Xeon with 
>> 88 cores
>> 
>> [1] https://www.gnu.org/software/parallel/
>> 
>> The following changes since commit 
>> 65d23bd7986615fdfb0f1717b615534a2a14ab80:
>> 
>>   README.qemu: qemuppc64 is not supported (2017-10-16 23:54:31 +0100)
>> 
>> are available in the git repository at:
>> 
>>   git://git.yoctoproject.org/poky-contrib 
>> lsandov1/oe-selftest-own-directory
>>   
>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lsandov1/oe-selftest-own-directory
>> 
>> Leonardo Sandoval (1):
>>   scripts/oe-selftest: oe-selftest-internal wrapper scripts that
>>     isolates execution
>> 
>>  scripts/oe-selftest          | 102 
>> +++++++++++++++++++++++--------------------
>>  scripts/oe-selftest-internal |  75 +++++++++++++++++++++++++++++++
>>  2 files changed, 129 insertions(+), 48 deletions(-)
>>  create mode 100755 scripts/oe-selftest-internal
>> 
>> --
>> 2.12.3
>> 
> 

[-- Attachment #2: Type: text/html, Size: 5287 bytes --]