[PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

[PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

The main idea is to isolate the oe-selftest execution so neither the current
build dir nor the configuration data is touch/polluted. This approach uses
a wrapper script (which is the one presented on this commit) which creates
a unique directory and inside it copies all scripts and metadata, re-initializes
the enviroment (re-sources oe-init-build-env) and finally launches
the oe-selftest-internal (which used to be oe-selftest) command, passing command
arguments to the latter.

The new build directory created when re-initializing the enviroment has the
same configuration data (local.conf, auto.conf, site.conf) as the
main build directory. The latter means that one can set DL_DIR and SSTATE_DIR
into <main build dir>/conf/site.conf and the oe-selftest execution will use this.

[YOCTO #11429]

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 scripts/oe-selftest          | 108 ++++++++++++++++++++++++-------------------
 scripts/oe-selftest-internal |  75 ++++++++++++++++++++++++++++++
 2 files changed, 135 insertions(+), 48 deletions(-)
 create mode 100755 scripts/oe-selftest-internal

diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index 1bf860a415..0505f943e4 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -1,5 +1,7 @@
-#!/usr/bin/env python3
+#!/bin/sh
 
+# scripts/oe-selftest: calls oe-selftest-internal in a isolated environment
+#
 # Copyright (c) 2013-2017 Intel Corporation
 #
 # This program is free software; you can redistribute it and/or modify
@@ -14,62 +16,72 @@
 # You should have received a copy of the GNU General Public License along
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+# Before anything else, check oe-core environment
+if [ -z "$BUILDDIR" ]; then
+    echo "Please initialize the OE-Core environment through oe-init-build-env script"
+    exit 1
+fi
+
+# Variable definitions
+ORIGBUILDDIR="$BUILDDIR"
+OESELFTESTSCRIPTDIR="$(which oe-selftest)"
+SCRIPTSDIR="$(dirname $OESELFTESTSCRIPTDIR)"
+OECOREDIR="$(dirname $SCRIPTSDIR)"
 
-# DESCRIPTION
-# This script runs tests defined in meta/lib/oeqa/selftest/
-# It's purpose is to automate the testing of different bitbake tools.
-# To use it you just need to source your build environment setup script and
-# add the meta-selftest layer to your BBLAYERS.
-# Call the script as: "oe-selftest -a" to run all the tests in meta/lib/oeqa/selftest/
-# Call the script as: "oe-selftest -r <module>.<Class>.<method>" to run just a single test
-# E.g: "oe-selftest -r bblayers.BitbakeLayers" will run just the BitbakeLayers class from meta/lib/oeqa/selftest/bblayers.py
+# oe-selftest-$$ related
+OESELFTESTDIR="$ORIGBUILDDIR/oe-selftest-$$"
+OESELFTESTBUILDDIR="$OESELFTESTDIR/build"
 
+# Return immediately in case no test execution
+ADDTESTS="$(echo "$@" | grep -i '\-a')"
+SOMETESTS="$(echo "$@" | grep -i '\-r')"
+if [ -z "$ADDTESTS" -a -z "$SOMETESTS" ]; then
+    if [ -z "$@" ]; then
+	oe-selftest-internal -h
+    else
+	oe-selftest-internal "$@"
+    fi
+    exit 0
+fi
 
+# Create directories (build, build/conf and layers)
+mkdir -p $OESELFTESTDIR $OESELFTESTBUILDDIR/conf $OESELFTESTLAYERSDIR
 
-import os
-import sys
-import argparse
-import logging
+# Populate the new build/conf, bblayers.conf will be touch latter to reflect new layer structure
+cp $ORIGBUILDDIR/conf/* $OESELFTESTBUILDDIR/conf
 
-scripts_path = os.path.dirname(os.path.realpath(__file__))
-lib_path = scripts_path + '/lib'
-sys.path = sys.path + [lib_path]
-import argparse_oe
-import scriptutils
-import scriptpath
-scriptpath.add_oe_lib_path()
-scriptpath.add_bitbake_lib_path()
+# Copy OE-Core meta directories
+cp -a $OECOREDIR/meta-selftest  $OESELFTESTDIR
+cp -a $OECOREDIR/meta-skeleton  $OESELFTESTDIR
+cp -a $OECOREDIR/meta-yocto-bsp $OESELFTESTDIR
 
-from oeqa.utils import load_test_components
-from oeqa.core.exception import OEQAPreRun
+# Copy non-metadata files
+cp -a $OECOREDIR/bitbake        $OESELFTESTDIR
+cp -a $SCRIPTSDIR               $OESELFTESTDIR
+cp $OECOREDIR/oe-init-build-env $OESELFTESTDIR
+cp $OECOREDIR/.templateconf     $OESELFTESTDIR
 
-logger = scriptutils.logger_create('oe-selftest', stream=sys.stdout)
+# Copy all layers define in BBLAYERS and construct the new bblayers.conf at the same time
+BBLAYERS="$(bitbake -e | grep ^BBLAYERS= | sed -e s/BBLAYERS=// -e s/\"//g)"
+for src in $BBLAYERS; do
+    cp -a $src $OESELFTESTDIR
+    # rename the new layer into the new bblayers.conf
+    tgt="$OESELFTESTDIR/$(basename $src)"
+    sed -e "s|$src|$tgt|" -i $OESELFTESTBUILDDIR/conf/bblayers.conf
+done
 
-def main():
-    description = "Script that runs unit tests against bitbake and other Yocto related tools. The goal is to validate tools functionality and metadata integrity. Refer to https://wiki.yoctoproject.org/wiki/Oe-selftest for more information."
-    parser = argparse_oe.ArgumentParser(description=description)
+# Reinitialized environment with new metadata and templateconf environement
+cd $OESELFTESTDIR
+source ./oe-init-build-env $OESELFTESTBUILDDIR
 
-    comp_name, comp = load_test_components(logger, 'oe-selftest').popitem()
-    comp.register_commands(logger, parser)
+# Execute the tests throught oe-selftest-internal
+oe-selftest-internal "$@"
 
-    try:
-        args = parser.parse_args()
-        results = args.func(logger, args)
-        ret = 0 if results.wasSuccessful() else 1
-    except SystemExit as err:
-        if err.code != 0:
-            raise err
-        ret = err.code
-    except OEQAPreRun as pr:
-        ret = 1
+OESELFTESTRC="$?"
 
-    return ret
+# Echo working folder
+echo -e "\nAll work done under $OESELFTESTDIR\n"
 
-if __name__ == '__main__':
-    try:
-        ret = main()
-    except Exception:
-        ret = 1
-        import traceback
-        traceback.print_exc()
-    sys.exit(ret)
+exit $OESELFTESTRC
diff --git a/scripts/oe-selftest-internal b/scripts/oe-selftest-internal
new file mode 100755
index 0000000000..cff2906763
--- /dev/null
+++ b/scripts/oe-selftest-internal
@@ -0,0 +1,75 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2013-2017 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+# DESCRIPTION
+# This script runs tests defined in meta/lib/oeqa/selftest/
+# It's purpose is to automate the testing of different bitbake tools.
+# To use it you just need to source your build environment setup script and
+# add the meta-selftest layer to your BBLAYERS.
+# Call the script as: "oe-selftest -a" to run all the tests in meta/lib/oeqa/selftest/
+# Call the script as: "oe-selftest -r <module>.<Class>.<method>" to run just a single test
+# E.g: "oe-selftest -r bblayers.BitbakeLayers" will run just the BitbakeLayers class from meta/lib/oeqa/selftest/bblayers.py
+
+
+
+import os
+import sys
+import argparse
+import logging
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + '/lib'
+sys.path = sys.path + [lib_path]
+import argparse_oe
+import scriptutils
+import scriptpath
+scriptpath.add_oe_lib_path()
+scriptpath.add_bitbake_lib_path()
+
+from oeqa.utils import load_test_components
+from oeqa.core.exception import OEQAPreRun
+
+logger = scriptutils.logger_create('oe-selftest', stream=sys.stdout)
+
+def main():
+    description = "Script that runs unit tests against bitbake and other Yocto related tools. The goal is to validate tools functionality and metadata integrity. Refer to https://wiki.yoctoproject.org/wiki/Oe-selftest for more information."
+    parser = argparse_oe.ArgumentParser(prog='oe-selftest', description=description)
+
+    comp_name, comp = load_test_components(logger, 'oe-selftest').popitem()
+    comp.register_commands(logger, parser)
+
+    try:
+        args = parser.parse_args()
+        results = args.func(logger, args)
+        ret = 0 if results.wasSuccessful() else 1
+    except SystemExit as err:
+        if err.code != 0:
+            raise err
+        ret = err.code
+    except OEQAPreRun as pr:
+        ret = 1
+
+    return ret
+
+if __name__ == '__main__':
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc()
+    sys.exit(ret)
-- 
2.12.3

[PATCH v2 2/2] parallel-oe-selftest.sh: runs oe-selftest in parallel

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Parallelize oe-selftest execution using GNU/Parallel: for each test
defined, a job will be launched and at any time at maximun of jobs will
be executing (defaults to 4). Extra parameters can be given to
parallel cmd after double dashes ('--'). Some cmd line examples:

1. Run all modules

    parallel-oe-selftest.sh

2. Run certaing modules and print results in order (see parallel man page for
more info)

    echo wic bblayers | parallel-oe-selftest.sh -- --keep-order

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 scripts/contrib/parallel-oe-selftest.sh | 81 +++++++++++++++++++++++++++++++++
 1 file changed, 81 insertions(+)
 create mode 100755 scripts/contrib/parallel-oe-selftest.sh

diff --git a/scripts/contrib/parallel-oe-selftest.sh b/scripts/contrib/parallel-oe-selftest.sh
new file mode 100755
index 0000000000..5fbdff9f95
--- /dev/null
+++ b/scripts/contrib/parallel-oe-selftest.sh
@@ -0,0 +1,81 @@
+#!/bin/sh
+
+# paralell-oe-selftest: executes oe-selftest in 'parallel'. The tests (modules, clases or test methods, same
+# as oe-selftest --run-tests) to be executed can be piped to this script; if this is not the case, all modules
+# are executed
+#
+# Copyright (c) 2013-2017 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+# Before anything else, check oe-core environment
+if [ -z "$BUILDDIR" ]; then
+    echo "Please initialize the OE-Core environment through oe-init-build-env script"
+    exit 1
+fi
+
+usage() {
+CMD=$(basename $0)
+cat <<EOM
+Usage: $CMD [-h] [-j jobs] [-- <parallel options>]
+  -j jobs             Number of jobs to be running concurrently
+  -h                  Display this help message
+
+ Examples:
+   # run all tests, output results in order and show progress
+   $CMD -j 4 -- --keep-order --progress
+
+   # run just wic and bblayers
+   echo wic bblayers | $CMD
+
+EOM
+}
+
+JOBS=4
+# Parse and validate arguments
+while getopts "hj:" OPT; do
+	case $OPT in
+	j)
+	        JOBS="$OPTARG"
+	        ;;
+	h)
+		usage
+		exit 0
+		;;
+	--)
+		shift
+		break
+		;;
+	esac
+done
+
+shift "$((OPTIND - 1))"
+extraopts="$@"
+echo $extraopts
+
+if [ -t 0 ]; then
+    # no stdin, run all modules
+    TESTCASES="$(oe-selftest -m | awk '{ print $NF } ' | grep -v ':')"
+else
+    TESTCASES="$(cat /dev/stdin)"
+fi
+
+# Parallelization is done through GNU/Paralell, so it must be present in host machine
+which parallel 2>&1 >/dev/null || { echo "Please install GNU/Parallel"; exit 1; }
+
+echo "The following tests will be run in parallel with a $JOBS jobs"
+echo ""; for t in $TESTCASES; do echo -e "\t$t"; done; echo ""
+
+echo "$TESTCASES" | parallel --jobs $JOBS $extraopts oe-selftest -r
-- 
2.12.3

Re: [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

[Patrick Ohly]

On Mon, 2017-11-13 at 10:17 -0800,
leonardo.sandoval.gonzalez@linux.intel.com wrote:
> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
> 
> The main idea is to isolate the oe-selftest execution so neither the
> current build dir nor the configuration data is touch/polluted. This
> approach uses a wrapper script (which is the one presented on this
> commit) which creates a unique directory and inside it copies all
> scripts and metadata, re-initializes the enviroment (re-sources oe-
> init-build-env) and finally launches the oe-selftest-internal (which
> used to be oe-selftest) command, passing command arguments to the
> latter.

This mode of operation may or may not be desirable. Can it be made
optional?

In refkit CI testing, several selftests run tests on build artifacts
(primarily the images) produced during the previous build and only
build them if needed, i.e. they run "bitbake some-image" and that is
usually fast because the image already exists. Reusing sstate and
download dir also is important for speed.

Forcing all tests to run in a clean environment would make the overall
CI run slower.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

[Leonardo Sandoval]

On Tue, 14 Nov 2017 09:24:30 +0100
Patrick Ohly <patrick.ohly@intel.com> wrote:

> On Mon, 2017-11-13 at 10:17 -0800,
> leonardo.sandoval.gonzalez@linux.intel.com wrote:
> > From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
> > 
> > The main idea is to isolate the oe-selftest execution so neither the
> > current build dir nor the configuration data is touch/polluted. This
> > approach uses a wrapper script (which is the one presented on this
> > commit) which creates a unique directory and inside it copies all
> > scripts and metadata, re-initializes the enviroment (re-sources oe-
> > init-build-env) and finally launches the oe-selftest-internal (which
> > used to be oe-selftest) command, passing command arguments to the
> > latter.  
> 
> This mode of operation may or may not be desirable. Can it be made
> optional?

good idea. However, you can call the oe-selftest-internal for the this purpose.

> 
> In refkit CI testing, several selftests run tests on build artifacts
> (primarily the images) produced during the previous build and only
> build them if needed, i.e. they run "bitbake some-image" and that is
> usually fast because the image already exists. Reusing sstate and
> download dir also is important for speed.

oe-selftest creates a new build/conf folder, with the same conf files as the ones present in your current build/conf, so this means that you can set there DL_DIR and SSTATE_DIR (for example) on your main local.conf and these will be used by oe-selftest, effectively reusing these folders.

> 
> Forcing all tests to run in a clean environment would make the overall
> CI run slower.

Agree. better to start with a 'hot cache' scenario, and this can be accomplished the way I mentioned before.



> 
> -- 
> Best Regards, Patrick Ohly
> 
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
> 
> 

Re: [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

[Patrick Ohly]

On Tue, 2017-11-14 at 10:09 -0600, Leonardo Sandoval wrote:
> On Tue, 14 Nov 2017 09:24:30 +0100
> Patrick Ohly <patrick.ohly@intel.com> wrote:
> 
> > On Mon, 2017-11-13 at 10:17 -0800,
> > leonardo.sandoval.gonzalez@linux.intel.com wrote:
> > > From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.c
> > > om>
> > > 
> > > The main idea is to isolate the oe-selftest execution so neither
> > > the
> > > current build dir nor the configuration data is touch/polluted.
> > > This
> > > approach uses a wrapper script (which is the one presented on
> > > this
> > > commit) which creates a unique directory and inside it copies all
> > > scripts and metadata, re-initializes the enviroment (re-sources
> > > oe-
> > > init-build-env) and finally launches the oe-selftest-internal
> > > (which
> > > used to be oe-selftest) command, passing command arguments to the
> > > latter.  
> > 
> > This mode of operation may or may not be desirable. Can it be made
> > optional?
> 
> good idea. However, you can call the oe-selftest-internal for the
> this purpose.

While doable, it feels wrong to rename something as "internal" and then
still expect people to call it directly.

A command line parameter for the new oe-selftest which controls this
behavior and just calls oe-selftest-internal directly when requested
feels cleaner and more discoverable.

Is oe-selftest-internal even going to be in the default PATH? It
probably shouldn't be, once it truly becomes an implementation detail.

> > In refkit CI testing, several selftests run tests on build
> > artifacts
> > (primarily the images) produced during the previous build and only
> > build them if needed, i.e. they run "bitbake some-image" and that
> > is
> > usually fast because the image already exists. Reusing sstate and
> > download dir also is important for speed.
> 
> oe-selftest creates a new build/conf folder, with the same conf files
> as the ones present in your current build/conf, so this means that
> you can set there DL_DIR and SSTATE_DIR (for example) on your main
> local.conf and these will be used by oe-selftest, effectively reusing
> these folders.

Instead of leaving it to the developer to figure this out, should the
oe-selftest have --[no-]reuse-download and --[no]-reuse-sstate options?

> > 
-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

[Leonardo Sandoval]

On Tue, 14 Nov 2017 17:09:28 +0100
Patrick Ohly <patrick.ohly@intel.com> wrote:

> On Tue, 2017-11-14 at 10:09 -0600, Leonardo Sandoval wrote:
> > On Tue, 14 Nov 2017 09:24:30 +0100
> > Patrick Ohly <patrick.ohly@intel.com> wrote:
> >   
> > > On Mon, 2017-11-13 at 10:17 -0800,
> > > leonardo.sandoval.gonzalez@linux.intel.com wrote:  
> > > > From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.c  
> > > > om>  
> > > > 
> > > > The main idea is to isolate the oe-selftest execution so neither
> > > > the
> > > > current build dir nor the configuration data is touch/polluted.
> > > > This
> > > > approach uses a wrapper script (which is the one presented on
> > > > this
> > > > commit) which creates a unique directory and inside it copies all
> > > > scripts and metadata, re-initializes the enviroment (re-sources
> > > > oe-
> > > > init-build-env) and finally launches the oe-selftest-internal
> > > > (which
> > > > used to be oe-selftest) command, passing command arguments to the
> > > > latter.    
> > > 
> > > This mode of operation may or may not be desirable. Can it be made
> > > optional?  
> > 
> > good idea. However, you can call the oe-selftest-internal for the
> > this purpose.  
> 
> While doable, it feels wrong to rename something as "internal" and then
> still expect people to call it directly.
> 
> A command line parameter for the new oe-selftest which controls this
> behavior and just calls oe-selftest-internal directly when requested
> feels cleaner and more discoverable.
> 
> Is oe-selftest-internal even going to be in the default PATH? It
> probably shouldn't be, once it truly becomes an implementation detail.

where can we placed it besides the scripts folder?

> 
> > > In refkit CI testing, several selftests run tests on build
> > > artifacts
> > > (primarily the images) produced during the previous build and only
> > > build them if needed, i.e. they run "bitbake some-image" and that
> > > is
> > > usually fast because the image already exists. Reusing sstate and
> > > download dir also is important for speed.  
> > 
> > oe-selftest creates a new build/conf folder, with the same conf files
> > as the ones present in your current build/conf, so this means that
> > you can set there DL_DIR and SSTATE_DIR (for example) on your main
> > local.conf and these will be used by oe-selftest, effectively reusing
> > these folders.  
> 
> Instead of leaving it to the developer to figure this out, should the
> oe-selftest have --[no-]reuse-download and --[no]-reuse-sstate options?

sounds good. By default, we will let reuse download/sstate.

Good input. I will send a v3.




> 
> > >   
> -- 
> Best Regards, Patrick Ohly
> 
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
> 
> 

Re: [PATCH v2 1/2] scripts/oe-selftest: oe-selftest-internal wrapper scripts that isolates execution

[Patrick Ohly]

On Tue, 2017-11-14 at 11:42 -0600, Leonardo Sandoval wrote:
> On Tue, 14 Nov 2017 17:09:28 +0100
> Patrick Ohly <patrick.ohly@intel.com> wrote:
> > Is oe-selftest-internal even going to be in the default PATH? It
> > probably shouldn't be, once it truly becomes an implementation
> > detail.
> 
> where can we placed it besides the scripts folder?

scripts/lib/ perhaps, then call it from oe-selftest with an absolute
path?

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.