[PATCH 0/4] Script and layer for running tests

[PATCH 0/4] Script and layer for running tests

[Stefan Stanacar]

Hello,

This series adds an oe-selftest script, some modules and a new layer meta-selftest.
which are meant to help in writing tests (using python unittest) for various
bitbake tools/scripts as well as simple output checks or do
complete builds with different options (with the emphasis that everything checked
is done outside of bitbake context, just like a human would do.) For more details,
please see YOCTO #4740.

Cheers,
Stefan

Changes since the RFC:
 - more tests added
 - tests are now in meta/lib/oeqa/selftest instead of scripts/lib/selftest/tests
 - small changes to some of the files in the meta-selftest layer
 - more cleanups done by default in the base class


The following changes since commit 32adaac34a614d106e6dd3e9f1130f4e94ff39ae:

  libpng: set reasonable SUMMARY (2013-11-27 11:51:25 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib stefans/selftest
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stefans/selftest

Alexandru Palalau (1):
  lib/oeqa/selftest: buildoptions.py: add simple image build tests

Corneliu Stoicescu (2):
  meta-selftest: create a new test layer to be used by oe-selftest
    script
  lib/oeqa/selftest: add test modules for expected bitbake output and
    bitbake-layers

Stefan Stanacar (1):
  scripts/oe-selftest: script to run builds as unittest against bitbake
    or various scripts

 .gitignore                                         |   1 +
 meta-selftest/COPYING.MIT                          |  17 +++
 meta-selftest/README                               |   3 +
 meta-selftest/classes/test_events.bbclass          |  16 +++
 meta-selftest/conf/layer.conf                      |  10 ++
 .../recipes-test/aspell/aspell_0.0.0.1.bb          |  28 ++++
 .../recipes-test/aspell/aspell_0.60.6.1.bbappend   |   2 +
 meta-selftest/recipes-test/m4/m4_1.4.17.bbappend   |   2 +
 .../recipes-test/man/man/man-1.5h1-make.patch      |  16 +++
 meta-selftest/recipes-test/man/man_1.6g.bbappend   |   2 +
 .../xcursor-transparent-theme_0.1.1.bbappend       |   2 +
 meta/lib/oeqa/selftest/__init__.py                 |   2 +
 meta/lib/oeqa/selftest/base.py                     |  98 ++++++++++++++
 meta/lib/oeqa/selftest/bblayers.py                 |  37 ++++++
 meta/lib/oeqa/selftest/bbtests.py                  |  97 ++++++++++++++
 meta/lib/oeqa/selftest/buildoptions.py             |  86 ++++++++++++
 meta/lib/oeqa/utils/commands.py                    | 137 +++++++++++++++++++
 meta/lib/oeqa/utils/ftools.py                      |  27 ++++
 scripts/oe-selftest                                | 148 +++++++++++++++++++++
 19 files changed, 731 insertions(+)
 create mode 100644 meta-selftest/COPYING.MIT
 create mode 100644 meta-selftest/README
 create mode 100644 meta-selftest/classes/test_events.bbclass
 create mode 100644 meta-selftest/conf/layer.conf
 create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
 create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.60.6.1.bbappend
 create mode 100644 meta-selftest/recipes-test/m4/m4_1.4.17.bbappend
 create mode 100644 meta-selftest/recipes-test/man/man/man-1.5h1-make.patch
 create mode 100644 meta-selftest/recipes-test/man/man_1.6g.bbappend
 create mode 100644 meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend
 create mode 100644 meta/lib/oeqa/selftest/__init__.py
 create mode 100644 meta/lib/oeqa/selftest/base.py
 create mode 100644 meta/lib/oeqa/selftest/bblayers.py
 create mode 100644 meta/lib/oeqa/selftest/bbtests.py
 create mode 100644 meta/lib/oeqa/selftest/buildoptions.py
 create mode 100644 meta/lib/oeqa/utils/commands.py
 create mode 100644 meta/lib/oeqa/utils/ftools.py
 create mode 100755 scripts/oe-selftest

-- 
1.8.3.1

[PATCH 1/4] scripts/oe-selftest: script to run builds as unittest against bitbake or various scripts

[Stefan Stanacar]

The purpose of oe-selftest is to run unittest modules added from meta/lib/oeqa/selftest,
which are tests against bitbake tools.

Right now the script it's useful for simple tests like:
  - "bitbake --someoption, change some metadata, bitbake X, check something" type scenarios (PR service, error output, etc)
  - or "bitbake-layers <...>" type scripts and yocto-bsp tools.

This commit also adds some helper modules that the tests will use and a base class.
Also, most of the tests will have a dependency on a meta-selftest layer
which contains specially modified recipes/bbappends/include files for the purpose of the tests.
The tests themselves will usually write to ".inc" files from the layer or in conf/selftest.inc
(which is added as an include in local.conf at the start and removed at the end)

It's a simple matter or sourcing the enviroment, adding the meta-selftest layer to bblayers.conf
and running: oe-selftest to get some results. It would finish faster if at least a core-image-minimal
was built before.

[ YOCTO #4740 ]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 meta/lib/oeqa/selftest/__init__.py |   2 +
 meta/lib/oeqa/selftest/base.py     |  98 ++++++++++++++++++++++++
 meta/lib/oeqa/utils/commands.py    | 137 ++++++++++++++++++++++++++++++++++
 meta/lib/oeqa/utils/ftools.py      |  27 +++++++
 scripts/oe-selftest                | 148 +++++++++++++++++++++++++++++++++++++
 5 files changed, 412 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/__init__.py
 create mode 100644 meta/lib/oeqa/selftest/base.py
 create mode 100644 meta/lib/oeqa/utils/commands.py
 create mode 100644 meta/lib/oeqa/utils/ftools.py
 create mode 100755 scripts/oe-selftest

diff --git a/meta/lib/oeqa/selftest/__init__.py b/meta/lib/oeqa/selftest/__init__.py
new file mode 100644
index 0000000..3ad9513
--- /dev/null
+++ b/meta/lib/oeqa/selftest/__init__.py
@@ -0,0 +1,2 @@
+from pkgutil import extend_path
+__path__ = extend_path(__path__, __name__)
diff --git a/meta/lib/oeqa/selftest/base.py b/meta/lib/oeqa/selftest/base.py
new file mode 100644
index 0000000..30a71e8
--- /dev/null
+++ b/meta/lib/oeqa/selftest/base.py
@@ -0,0 +1,98 @@
+# Copyright (c) 2013 Intel Corporation
+#
+# Released under the MIT license (see COPYING.MIT)
+
+
+# DESCRIPTION
+# Base class inherited by test classes in meta/lib/selftest
+
+import unittest
+import os
+import sys
+import logging
+import errno
+
+import oeqa.utils.ftools as ftools
+
+
+class oeSelfTest(unittest.TestCase):
+
+    log = logging.getLogger("selftest.base")
+    longMessage = True
+
+    def __init__(self, methodName="runTest"):
+        self.builddir = os.environ.get("BUILDDIR")
+        self.localconf_path = os.path.join(self.builddir, "conf/local.conf")
+        self.testinc_path = os.path.join(self.builddir, "conf/selftest.inc")
+        self.testlayer_path = oeSelfTest.testlayer_path
+        super(oeSelfTest, self).__init__(methodName)
+
+    def setUp(self):
+        os.chdir(self.builddir)
+        # we don't know what the previous test left around in config or inc files
+        # if it failed so we need a fresh start
+        try:
+            os.remove(self.testinc_path)
+        except OSError as e:
+            if e.errno != errno.ENOENT:
+                raise
+        for root, _, files in os.walk(self.testlayer_path):
+            for f in files:
+                if f == 'test_recipe.inc':
+                    os.remove(os.path.join(root, f))
+        # tests might need their own setup
+        # but if they overwrite this one they have to call
+        # super each time, so let's give them an alternative
+        self.setUpLocal()
+
+    def setUpLocal(self):
+        pass
+
+    def tearDown(self):
+        self.tearDownLocal()
+
+    def tearDownLocal(self):
+        pass
+
+    # write to <builddir>/conf/selftest.inc
+    def write_config(self, data):
+        self.log.debug("Writing to: %s\n%s\n" % (self.testinc_path, data))
+        ftools.write_file(self.testinc_path, data)
+
+    # append to <builddir>/conf/selftest.inc
+    def append_config(self, data):
+        self.log.debug("Appending to: %s\n%s\n" % (self.testinc_path, data))
+        ftools.append_file(self.testinc_path, data)
+
+    # remove data from <builddir>/conf/selftest.inc
+    def remove_config(self, data):
+        self.log.debug("Removing from: %s\n\%s\n" % (self.testinc_path, data))
+        ftools.remove_from_file(self.testinc_path, data)
+
+    # write to meta-sefltest/recipes-test/<recipe>/test_recipe.inc
+    def write_recipeinc(self, recipe, data):
+        inc_file = os.path.join(self.testlayer_path, 'recipes-test', recipe, 'test_recipe.inc')
+        self.log.debug("Writing to: %s\n%s\n" % (inc_file, data))
+        ftools.write_file(inc_file, data)
+
+    # append data to meta-sefltest/recipes-test/<recipe>/test_recipe.inc
+    def append_recipeinc(self, recipe, data):
+        inc_file = os.path.join(self.testlayer_path, 'recipes-test', recipe, 'test_recipe.inc')
+        self.log.debug("Appending to: %s\n%s\n" % (inc_file, data))
+        ftools.append_file(inc_file, data)
+
+    # remove data from meta-sefltest/recipes-test/<recipe>/test_recipe.inc
+    def remove_recipeinc(self, recipe, data):
+        inc_file = os.path.join(self.testlayer_path, 'recipes-test', recipe, 'test_recipe.inc')
+        self.log.debug("Removing from: %s\n%s\n" % (inc_file, data))
+        ftools.remove_from_file(inc_file, data)
+
+    # delete meta-sefltest/recipes-test/<recipe>/test_recipe.inc file
+    def delete_recipeinc(self, recipe):
+        inc_file = os.path.join(self.testlayer_path, 'recipes-test', recipe, 'test_recipe.inc')
+        self.log.debug("Deleting file: %s" % inc_file)
+        try:
+            os.remove(self.testinc_path)
+        except OSError as e:
+            if e.errno != errno.ENOENT:
+                raise
diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
new file mode 100644
index 0000000..9b42620
--- /dev/null
+++ b/meta/lib/oeqa/utils/commands.py
@@ -0,0 +1,137 @@
+# Copyright (c) 2013 Intel Corporation
+#
+# Released under the MIT license (see COPYING.MIT)
+
+# DESCRIPTION
+# This module is mainly used by scripts/oe-selftest and modules under meta/oeqa/selftest
+# It provides a class and methods for running commands on the host in a convienent way for tests.
+
+
+
+import os
+import sys
+import signal
+import subprocess
+import threading
+import logging
+
+class Command(object):
+    def __init__(self, command, bg=False, timeout=None, data=None, **options):
+
+        self.defaultopts = {
+            "stdout": subprocess.PIPE,
+            "stderr": subprocess.STDOUT,
+            "stdin": None,
+            "shell": False,
+            "bufsize": -1,
+        }
+
+        self.cmd = command
+        self.bg = bg
+        self.timeout = timeout
+        self.data = data
+
+        self.options = dict(self.defaultopts)
+        if isinstance(self.cmd, basestring):
+            self.options["shell"] = True
+        if self.data:
+            self.options['stdin'] = subprocess.PIPE
+        self.options.update(options)
+
+        self.status = None
+        self.output = None
+        self.error = None
+        self.thread = None
+
+        self.log = logging.getLogger("utils.commands")
+
+    def run(self):
+        self.process = subprocess.Popen(self.cmd, **self.options)
+
+        def commThread():
+            self.output, self.error = self.process.communicate(self.data)
+
+        self.thread = threading.Thread(target=commThread)
+        self.thread.start()
+
+        self.log.debug("Running command '%s'" % self.cmd)
+
+        if not self.bg:
+            self.thread.join(self.timeout)
+            self.stop()
+
+    def stop(self):
+        if self.thread.isAlive():
+            self.process.terminate()
+            # let's give it more time to terminate gracefully before killing it
+            self.thread.join(5)
+            if self.thread.isAlive():
+                self.process.kill()
+                self.thread.join()
+
+        self.output = self.output.rstrip()
+        self.status = self.process.poll()
+
+        self.log.debug("Command '%s' returned %d as exit code." % (self.cmd, self.status))
+        # logging the complete output is insane
+        # bitbake -e output is really big
+        # and makes the log file useless
+        if self.status:
+            lout = "\n".join(self.output.splitlines()[-20:])
+            self.log.debug("Last 20 lines:\n%s" % lout)
+
+
+class Result(object):
+    pass
+
+def runCmd(command, ignore_status=False, timeout=None, **options):
+
+    result = Result()
+
+    cmd = Command(command, timeout=timeout, **options)
+    cmd.run()
+
+    result.command = command
+    result.status = cmd.status
+    result.output = cmd.output
+    result.pid = cmd.process.pid
+
+    if result.status and not ignore_status:
+        raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, result.output))
+
+    return result
+
+
+def bitbake(command, ignore_status=False, timeout=None, **options):
+    if isinstance(command, basestring):
+        cmd = "bitbake " + command
+    else:
+        cmd = [ "bitbake" ] + command
+
+    return runCmd(cmd, ignore_status, timeout, **options)
+
+
+def get_bb_env(target=None):
+    if target:
+        return runCmd("bitbake -e %s" % target).output
+    else:
+        return runCmd("bitbake -e").output
+
+def get_bb_var(var, target=None):
+    val = None
+    bbenv = get_bb_env(target)
+    for line in bbenv.splitlines():
+        if line.startswith(var + "="):
+            val = line.split('=')[1]
+            val = val.replace('\"','')
+            break
+    return val
+
+def get_test_layer():
+    layers = get_bb_var("BBLAYERS").split()
+    testlayer = None
+    for l in layers:
+        if "/meta-selftest" in l and os.path.isdir(l):
+            testlayer = l
+            break
+    return testlayer
diff --git a/meta/lib/oeqa/utils/ftools.py b/meta/lib/oeqa/utils/ftools.py
new file mode 100644
index 0000000..64ebe3d
--- /dev/null
+++ b/meta/lib/oeqa/utils/ftools.py
@@ -0,0 +1,27 @@
+import os
+import re
+
+def write_file(path, data):
+    wdata = data.rstrip() + "\n"
+    with open(path, "w") as f:
+        f.write(wdata)
+
+def append_file(path, data):
+    wdata = data.rstrip() + "\n"
+    with open(path, "a") as f:
+            f.write(wdata)
+
+def read_file(path):
+    data = None
+    with open(path) as f:
+        data = f.read()
+    return data
+
+def remove_from_file(path, data):
+    lines = read_file(path).splitlines()
+    rmdata = data.strip().splitlines()
+    for l in rmdata:
+        for c in range(0, lines.count(l)):
+            i = lines.index(l)
+            del(lines[i])
+    write_file(path, "\n".join(lines))
diff --git a/scripts/oe-selftest b/scripts/oe-selftest
new file mode 100755
index 0000000..db42e73
--- /dev/null
+++ b/scripts/oe-selftest
@@ -0,0 +1,148 @@
+#!/usr/bin/env python
+
+# Copyright (c) 2013 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+# DESCRIPTION
+# This script runs tests defined in meta/lib/selftest/
+# It's purpose is to automate the testing of different bitbake tools.
+# To use it you just need to source your build environment setup script and
+# add the meta-selftest layer to your BBLAYERS.
+# Call the script as: "oe-selftest" to run all the tests in in meta/lib/selftest/
+# Call the script as: "oe-selftest <module>.<Class>.<method>" to run just a single test
+# E.g: "oe-selftest bboutput.BitbakeLayers" will run just the BitbakeLayers class from meta/lib/selftest/bboutput.py
+
+
+import os
+import sys
+import unittest
+import logging
+
+sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'meta/lib')))
+
+import oeqa.selftest
+import oeqa.utils.ftools as ftools
+from oeqa.utils.commands import runCmd, get_bb_var, get_test_layer
+from oeqa.selftest.base import oeSelfTest
+
+def logger_create():
+    log = logging.getLogger("selftest")
+    log.setLevel(logging.DEBUG)
+
+    fh = logging.FileHandler(filename='oe-selftest.log', mode='w')
+    fh.setLevel(logging.DEBUG)
+
+    ch = logging.StreamHandler(sys.stdout)
+    ch.setLevel(logging.INFO)
+
+    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+    fh.setFormatter(formatter)
+    ch.setFormatter(formatter)
+
+    log.addHandler(fh)
+    log.addHandler(ch)
+
+    return log
+
+log = logger_create()
+
+def preflight_check():
+
+    log.info("Checking that everything is in order before running the tests")
+
+    if not os.environ.get("BUILDDIR"):
+        log.error("BUILDDIR isn't set. Did you forget to source your build environment setup script?")
+        return False
+
+    builddir = os.environ.get("BUILDDIR")
+    if os.getcwd() != builddir:
+        log.info("Changing cwd to %s" % builddir)
+        os.chdir(builddir)
+
+    if not "meta-selftest" in get_bb_var("BBLAYERS"):
+        log.error("You don't seem to have the meta-selftest layer in BBLAYERS")
+        return False
+
+    log.info("Running bitbake -p")
+    runCmd("bitbake -p")
+
+    return True
+
+def add_include():
+    builddir = os.environ.get("BUILDDIR")
+    if "#include added by oe-selftest.py" \
+        not in ftools.read_file(os.path.join(builddir, "conf/local.conf")):
+            log.info("Adding: \"include selftest.inc\" in local.conf")
+            ftools.append_file(os.path.join(builddir, "conf/local.conf"), \
+                    "\n#include added by oe-selftest.py\ninclude selftest.inc")
+
+
+def remove_include():
+    builddir = os.environ.get("BUILDDIR")
+    if "#include added by oe-selftest.py" \
+        in ftools.read_file(os.path.join(builddir, "conf/local.conf")):
+            log.info("Removing the include from local.conf")
+            ftools.remove_from_file(os.path.join(builddir, "conf/local.conf"), \
+                    "#include added by oe-selftest.py\ninclude selftest.inc")
+
+def get_tests():
+    testslist = []
+    for x in sys.argv[1:]:
+        testslist.append('oeqa.selftest.' + x)
+    if not testslist:
+        testpath = os.path.abspath(os.path.dirname(oeqa.selftest.__file__))
+        files = sorted([f for f in os.listdir(testpath) if f.endswith('.py') and not f.startswith('_') and f != 'base.py'])
+        for f in files:
+            module = 'oeqa.selftest.' + f[:-3]
+            testslist.append(module)
+
+    return testslist
+
+def main():
+    if not preflight_check():
+        return 1
+
+    testslist = get_tests()
+    suite = unittest.TestSuite()
+    loader = unittest.TestLoader()
+    loader.sortTestMethodsUsing = None
+    runner = unittest.TextTestRunner(verbosity=2)
+    # we need to do this here, otherwise just loading the tests
+    # will take 2 minutes (bitbake -e calls)
+    oeSelfTest.testlayer_path = get_test_layer()
+    for test in testslist:
+        log.info("Loading tests from: %s" % test)
+        try:
+            suite.addTests(loader.loadTestsFromName(test))
+        except AttributeError as e:
+            log.error("Failed to import %s" % test)
+            log.error(e)
+            return 1
+    add_include()
+    result = runner.run(suite)
+    log.info("Finished")
+
+    return 0
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc(5)
+    finally:
+        remove_include()
+    sys.exit(ret)
-- 
1.8.3.1

[PATCH 2/4] meta-selftest: create a new test layer to be used by oe-selftest script

[Stefan Stanacar]

From: Corneliu Stoicescu <corneliux.stoicescu@intel.com>

Everything in this layer is meant to be used by tests called by
scripts/oe-selftest. These are helper recipes/appends to test various bitbake
options or scripts.
Currently most of these files here only have "include test_recipe.inc" which
is the file tests will actually use.

Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 .gitignore                                         |  1 +
 meta-selftest/COPYING.MIT                          | 17 +++++++++++++
 meta-selftest/README                               |  3 +++
 meta-selftest/classes/test_events.bbclass          | 16 +++++++++++++
 meta-selftest/conf/layer.conf                      | 10 ++++++++
 .../recipes-test/aspell/aspell_0.0.0.1.bb          | 28 ++++++++++++++++++++++
 .../recipes-test/aspell/aspell_0.60.6.1.bbappend   |  2 ++
 meta-selftest/recipes-test/m4/m4_1.4.17.bbappend   |  2 ++
 .../recipes-test/man/man/man-1.5h1-make.patch      | 16 +++++++++++++
 meta-selftest/recipes-test/man/man_1.6g.bbappend   |  2 ++
 .../xcursor-transparent-theme_0.1.1.bbappend       |  2 ++
 11 files changed, 99 insertions(+)
 create mode 100644 meta-selftest/COPYING.MIT
 create mode 100644 meta-selftest/README
 create mode 100644 meta-selftest/classes/test_events.bbclass
 create mode 100644 meta-selftest/conf/layer.conf
 create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
 create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.60.6.1.bbappend
 create mode 100644 meta-selftest/recipes-test/m4/m4_1.4.17.bbappend
 create mode 100644 meta-selftest/recipes-test/man/man/man-1.5h1-make.patch
 create mode 100644 meta-selftest/recipes-test/man/man_1.6g.bbappend
 create mode 100644 meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend

diff --git a/.gitignore b/.gitignore
index 0171597..b6755fe 100644
--- a/.gitignore
+++ b/.gitignore
@@ -21,3 +21,4 @@ bitbake/doc/manual/pdf/
 bitbake/doc/manual/txt/
 bitbake/doc/manual/xhtml/
 pull-*/
+!meta-selftest
diff --git a/meta-selftest/COPYING.MIT b/meta-selftest/COPYING.MIT
new file mode 100644
index 0000000..89de354
--- /dev/null
+++ b/meta-selftest/COPYING.MIT
@@ -0,0 +1,17 @@
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
diff --git a/meta-selftest/README b/meta-selftest/README
new file mode 100644
index 0000000..11a6fee
--- /dev/null
+++ b/meta-selftest/README
@@ -0,0 +1,3 @@
+This layer is intended as test layer, used by scripts/oe-selftest
+and it's probably a mistake to include it in your builds (unless you
+want to run the script).
diff --git a/meta-selftest/classes/test_events.bbclass b/meta-selftest/classes/test_events.bbclass
new file mode 100644
index 0000000..35324eb
--- /dev/null
+++ b/meta-selftest/classes/test_events.bbclass
@@ -0,0 +1,16 @@
+python test1_eventhandler() {
+    bb.note("Test for bb.event.BuildStarted")
+}
+python test2_eventhandler() {
+    bb.note("Test for bb.event.BuildCompleted")
+}
+python test3_eventhandler() {
+    bb.note("Test for bb.event.InvalidEvent")
+}
+
+addhandler test1_eventhandler
+test1_eventhandler[eventmask] = "bb.event.BuildStarted"
+addhandler test2_eventhandler
+test2_eventhandler[eventmask] = "bb.event.BuildCompleted"
+addhandler test3_eventhandler
+test3_eventhandler[eventmask] = "bb.event.InvalidEvent"
diff --git a/meta-selftest/conf/layer.conf b/meta-selftest/conf/layer.conf
new file mode 100644
index 0000000..a847b78
--- /dev/null
+++ b/meta-selftest/conf/layer.conf
@@ -0,0 +1,10 @@
+# We have a conf and classes directory, add to BBPATH
+BBPATH .= ":${LAYERDIR}"
+
+# We have recipes-* directories, add to BBFILES
+BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
+	${LAYERDIR}/recipes-*/*/*.bbappend"
+
+BBFILE_COLLECTIONS += "selftest"
+BBFILE_PATTERN_selftest = "^${LAYERDIR}/"
+BBFILE_PRIORITY_selftest = "5"
diff --git a/meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb b/meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
new file mode 100644
index 0000000..427726a
--- /dev/null
+++ b/meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
@@ -0,0 +1,28 @@
+# This recipe is a copy from the oe-core one.
+# It has a lower and invalid version number in order not to be accidentally used by bitbake.
+# It is used for tests that require overlayed recipe files.
+
+DESCRIPTION = "GNU Aspell spell-checker"
+SECTION = "console/utils"
+LICENSE="LGPLv2 | LGPLv2.1"
+LIC_FILES_CHKSUM = "file://COPYING;md5=7fbc338309ac38fefcd64b04bb903e34"
+PR = "r1"
+
+SRC_URI = "${GNU_MIRROR}/aspell/aspell-${PV}.tar.gz"
+SRC_URI[md5sum] = "e66a9c9af6a60dc46134fdacf6ce97d7"
+SRC_URI[sha256sum] = "f52583a83a63633701c5f71db3dc40aab87b7f76b29723aeb27941eff42df6e1"
+
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[curses] = "--enable-curses,--disable-curses,ncurses"
+
+PACKAGES += "libaspell libpspell libpspell-dev aspell-utils"
+
+FILES_${PN}-dbg += "${libdir}/aspell-0.60/.debu*"
+FILES_libaspell = "${libdir}/libaspell.so.* ${libdir}/aspell*"
+FILES_aspell-utils = "${bindir}/word-list-compress ${bindir}/aspell-import ${bindir}/run-with-aspell ${bindir}/pre*"
+FILES_${PN} = "${bindir}/aspell"
+FILES_libpspell = "${libdir}/libpspell.so.*"
+FILES_libpspell-dev = "${libdir}/libpspell* ${bindir}/pspell-config ${includedir}/pspell"
+
+ARM_INSTRUCTION_SET = "arm"
+inherit autotools gettext
diff --git a/meta-selftest/recipes-test/aspell/aspell_0.60.6.1.bbappend b/meta-selftest/recipes-test/aspell/aspell_0.60.6.1.bbappend
new file mode 100644
index 0000000..2057209
--- /dev/null
+++ b/meta-selftest/recipes-test/aspell/aspell_0.60.6.1.bbappend
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
diff --git a/meta-selftest/recipes-test/m4/m4_1.4.17.bbappend b/meta-selftest/recipes-test/m4/m4_1.4.17.bbappend
new file mode 100644
index 0000000..2057209
--- /dev/null
+++ b/meta-selftest/recipes-test/m4/m4_1.4.17.bbappend
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
diff --git a/meta-selftest/recipes-test/man/man/man-1.5h1-make.patch b/meta-selftest/recipes-test/man/man/man-1.5h1-make.patch
new file mode 100644
index 0000000..a0d59b7
--- /dev/null
+++ b/meta-selftest/recipes-test/man/man/man-1.5h1-make.patch
@@ -0,0 +1,16 @@
+Test patch here!
+This is invalid patch used by tests in scripts/lib/selftests.
+
+--- man-1.5g/man/Makefile.in.mike	Fri Apr  9 13:35:54 1999
++++ man-1.5g/man/Makefile.in	Fri Apr  9 13:36:45 1999
+@@ -1,8 +1,8 @@
+ #MAKE THIS PATCH INVALID
+ MAN1 = man whatis apropos
+-MAN5 = man.conf
++MAN5 = man.config
+ MAN8 = makewhatis
+-ALL = man.1 whatis.1 apropos.1 man.conf.5
++ALL = man.1 whatis.1 apropos.1 man.config.5
+ MAYBE8 = makewhatis
+ 
+ .SUFFIXES: .man .1 .5 .8
diff --git a/meta-selftest/recipes-test/man/man_1.6g.bbappend b/meta-selftest/recipes-test/man/man_1.6g.bbappend
new file mode 100644
index 0000000..2057209
--- /dev/null
+++ b/meta-selftest/recipes-test/man/man_1.6g.bbappend
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
diff --git a/meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend b/meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend
new file mode 100644
index 0000000..2057209
--- /dev/null
+++ b/meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
-- 
1.8.3.1

[PATCH 3/4] lib/oeqa/selftest: buildoptions.py: add simple image build tests

[Stefan Stanacar]

From: Alexandru Palalau <alexandrux.palalau@intel.com>

Build images and tests different build options like RM_OLD_IMAGE
and for WARN_QA/ERROR_QA behaviour.

Signed-off-by: Alexandru Palalau <alexandrux.palalau@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 meta/lib/oeqa/selftest/buildoptions.py | 86 ++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/buildoptions.py

diff --git a/meta/lib/oeqa/selftest/buildoptions.py b/meta/lib/oeqa/selftest/buildoptions.py
new file mode 100644
index 0000000..f99dda7
--- /dev/null
+++ b/meta/lib/oeqa/selftest/buildoptions.py
@@ -0,0 +1,86 @@
+import unittest
+import os
+import logging
+import re
+
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var
+import oeqa.utils.ftools as ftools
+
+class ImageOptionsTests(oeSelfTest):
+
+    def test_incremental_image_generation(self):
+        bitbake("-c cleanall core-image-minimal")
+        self.write_config('INC_RPM_IMAGE_GEN = "1"')
+        self.append_config('IMAGE_FEATURES += "ssh-server-openssh"')
+        bitbake("core-image-minimal")
+        res = runCmd("grep 'Installing openssh-sshd' %s" % (os.path.join(get_bb_var("WORKDIR", "core-image-minimal"), "temp/log.do_rootfs")), ignore_status=True)
+        self.remove_config('IMAGE_FEATURES += "ssh-server-openssh"')
+        self.assertEqual(0, res.status, msg="No match for openssh-sshd in log.do_rootfs")
+        bitbake("core-image-minimal")
+        res = runCmd("grep 'Removing openssh-sshd' %s" %(os.path.join(get_bb_var("WORKDIR", "core-image-minimal"), "temp/log.do_rootfs")),ignore_status=True)
+        self.assertEqual(0, res.status, msg="openssh-sshd was not removed from image")
+
+    def test_rm_old_image(self):
+        bitbake("core-image-minimal")
+        deploydir = get_bb_var("DEPLOY_DIR_IMAGE", target="core-image-minimal")
+        imagename = get_bb_var("IMAGE_LINK_NAME", target="core-image-minimal")
+        oldimgpath = os.path.realpath(os.path.join(deploydir, imagename + ".ext3"))
+        self.append_config("RM_OLD_IMAGE = \"1\"")
+        bitbake("core-image-minimal")
+        self.assertFalse(os.path.exists(oldimgpath), msg="Old image path still exists: %s" % oldimgpath)
+
+    def test_ccache_tool(self):
+        bitbake("ccache-native")
+        self.assertTrue(os.path.isfile(os.path.join(get_bb_var('STAGING_BINDIR_NATIVE', 'ccache-native'), "ccache")))
+        self.write_config('INHERIT += "ccache"')
+        bitbake("m4 -c cleansstate")
+        bitbake("m4 -c compile")
+        res = runCmd("grep ccache %s" % (os.path.join(get_bb_var("WORKDIR","m4"),"temp/log.do_compile")), ignore_status=True)
+        self.assertEqual(0, res.status, msg="No match for ccache in m4 log.do_compile")
+        bitbake("ccache-native -ccleansstate")
+
+
+class DiskMonTest(oeSelfTest):
+
+    def test_stoptask_behavior(self):
+        result = runCmd("df -k %s" % os.getcwd())
+        size = result.output.split("\n")[1].split()[3]
+        self.append_config('BB_DISKMON_DIRS = "STOPTASKS,${TMPDIR},%sK,4510K"' % size)
+        res = bitbake("core-image-minimal", ignore_status = True)
+        self.assertTrue('ERROR: No new tasks can be executed since the disk space monitor action is "STOPTASKS"!' in res.output)
+        self.assertEqual(res.status, 1)
+        self.append_config('BB_DISKMON_DIRS = "ABORT,${TMPDIR},%sK,4510K"' % size)
+        res = bitbake("core-image-minimal", ignore_status = True)
+        self.assertTrue('ERROR: Immediately abort since the disk space monitor action is "ABORT"!' in res.output)
+        self.assertEqual(res.status, 1)
+        self.append_config('BB_DISKMON_DIRS = "WARN,${TMPDIR},%sK,4510K"' % size)
+        res = bitbake("core-image-minimal")
+        self.assertTrue('WARNING: The free space' in res.output)
+
+class SanityOptionsTest(oeSelfTest):
+
+    def test_options_warnqa_errorqa_switch(self):
+        bitbake("xcursor-transparent-theme -ccleansstate")
+
+        if "packages-list" not in get_bb_var("ERROR_QA"):
+            self.write_config("ERROR_QA_append = \" packages-list\"")
+
+        self.write_recipeinc('xcursor-transparent-theme', 'PACKAGES += \"${PN}-dbg\"')
+        res = bitbake("xcursor-transparent-theme", ignore_status=True)
+        self.delete_recipeinc('xcursor-transparent-theme')
+        self.assertTrue("ERROR: QA Issue: xcursor-transparent-theme-dbg is listed in PACKAGES multiple times, this leads to packaging errors." in res.output)
+        self.assertEqual(res.status, 1)
+        self.write_recipeinc('xcursor-transparent-theme', 'PACKAGES += \"${PN}-dbg\"')
+        self.append_config('ERROR_QA_remove = "packages-list"')
+        self.append_config('WARN_QA_append = " packages-list"')
+        bitbake("xcursor-transparent-theme")
+        bitbake("xcursor-transparent-theme -ccleansstate")
+        self.delete_recipeinc('xcursor-transparent-theme')
+
+    def test_sanity_userspace_dependency(self):
+        self.append_config('WARN_QA_append = " unsafe-references-in-binaries unsafe-references-in-scripts"')
+        bitbake("-ccleansstate gzip nfs-utils")
+        res = bitbake("gzip nfs-utils")
+        self.assertTrue("WARNING: QA Issue: gzip" in res.output)
+        self.assertTrue("WARNING: QA Issue: nfs-utils" in res.output)
-- 
1.8.3.1

[PATCH 4/4] lib/oeqa/selftest: add test modules for expected bitbake output and bitbake-layers

[Stefan Stanacar]

From: Corneliu Stoicescu <corneliux.stoicescu@intel.com>

Tests for bitbake-layers and expected output for some bitbake options.

Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 meta/lib/oeqa/selftest/bblayers.py | 37 +++++++++++++++
 meta/lib/oeqa/selftest/bbtests.py  | 97 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 134 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/bblayers.py
 create mode 100644 meta/lib/oeqa/selftest/bbtests.py

diff --git a/meta/lib/oeqa/selftest/bblayers.py b/meta/lib/oeqa/selftest/bblayers.py
new file mode 100644
index 0000000..52aa4f8
--- /dev/null
+++ b/meta/lib/oeqa/selftest/bblayers.py
@@ -0,0 +1,37 @@
+import unittest
+import os
+import logging
+import re
+import shutil
+
+import oeqa.utils.ftools as ftools
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import runCmd
+
+class BitbakeLayers(oeSelfTest):
+
+    def test_bitbakelayers_showcrossdepends(self):
+        result = runCmd('bitbake-layers show-cross-depends')
+        self.assertTrue('aspell' in result.output)
+
+    def test_bitbakelayers_showlayers(self):
+        result = runCmd('bitbake-layers show_layers')
+        self.assertTrue('meta-selftest' in result.output)
+
+    def test_bitbakelayers_showappends(self):
+        result = runCmd('bitbake-layers show_appends')
+        self.assertTrue('xcursor-transparent-theme_0.1.1.bbappend' in result.output, msg='xcursor-transparent-theme_0.1.1.bbappend file was not recognised')
+
+    def test_bitbakelayers_showoverlayed(self):
+        result = runCmd('bitbake-layers show_overlayed')
+        self.assertTrue('aspell' in result.output, msg='xcursor-transparent-theme_0.1.1.bbappend file was not recognised')
+
+    def test_bitbakelayers_flatten(self):
+        self.assertFalse(os.path.isdir(os.path.join(self.builddir, 'test')))
+        result = runCmd('bitbake-layers flatten test')
+        bb_file = os.path.join(self.builddir, 'test/recipes-graphics/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bb')
+        self.assertTrue(os.path.isfile(bb_file))
+        contents = ftools.read_file(bb_file)
+        find_in_contents = re.search("##### bbappended from meta-selftest #####\n(.*\n)*include test_recipe.inc", contents)
+        shutil.rmtree(os.path.join(self.builddir, 'test'))
+        self.assertTrue(find_in_contents)
diff --git a/meta/lib/oeqa/selftest/bbtests.py b/meta/lib/oeqa/selftest/bbtests.py
new file mode 100644
index 0000000..01e0099
--- /dev/null
+++ b/meta/lib/oeqa/selftest/bbtests.py
@@ -0,0 +1,97 @@
+import unittest
+import os
+import logging
+import re
+import shutil
+
+import oeqa.utils.ftools as ftools
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var
+
+class BitbakeTests(oeSelfTest):
+
+    def test_run_bitbake_from_dir_1(self):
+        os.chdir(os.path.join(self.builddir, 'conf'))
+        bitbake('-e')
+
+    def test_run_bitbake_from_dir_2(self):
+        my_env = os.environ.copy()
+        my_env['BBPATH'] = my_env['BUILDDIR']
+        os.chdir(os.path.dirname(os.environ['BUILDDIR']))
+        bitbake('-e', env=my_env)
+
+    def test_event_handler(self):
+        self.write_config("INHERIT += \"test_events\"")
+        result = bitbake('m4-native')
+        find_build_started = re.search("NOTE: Test for bb\.event\.BuildStarted(\n.*)*NOTE: Preparing runqueue", result.output)
+        find_build_completed = re.search("Tasks Summary:.*(\n.*)*NOTE: Test for bb\.event\.BuildCompleted", result.output)
+        self.assertTrue(find_build_started, msg = "Match failed in:\n%s"  % result.output)
+        self.assertTrue(find_build_completed, msg = "Match failed in:\n%s" % result.output)
+        self.assertFalse('Test for bb.event.InvalidEvent' in result.output)
+
+    def test_local_sstate(self):
+        bitbake('m4-native -ccleansstate')
+        bitbake('m4-native')
+        bitbake('m4-native -cclean')
+        result = bitbake('m4-native')
+        find_setscene = re.search("m4-native.*do_.*_setscene", result.output)
+        self.assertTrue(find_setscene)
+
+    def test_bitbake_invalid_recipe(self):
+        result = bitbake('-b asdf', ignore_status=True)
+        self.assertTrue("ERROR: Unable to find any recipe file matching 'asdf'" in result.output)
+
+    def test_bitbake_invalid_target(self):
+        result = bitbake('asdf', ignore_status=True)
+        self.assertTrue("ERROR: Nothing PROVIDES 'asdf'" in result.output)
+
+    def test_warnings_errors(self):
+        result = bitbake('-b asdf', ignore_status=True)
+        find_warnings = re.search("Summary: There was [1-9][0-9]* WARNING message shown.", result.output)
+        find_errors = re.search("Summary: There was [1-9][0-9]* ERROR message shown.", result.output)
+        self.assertTrue(find_warnings)
+        self.assertTrue(find_errors)
+
+    def test_invalid_patch(self):
+        self.write_recipeinc('man', 'SRC_URI += "file://man-1.5h1-make.patch"')
+        result = bitbake('man -c patch', ignore_status=True)
+        self.delete_recipeinc('man')
+        bitbake('-cclean man')
+        self.assertTrue("ERROR: Function failed: patch_do_patch" in result.output)
+
+    def test_force_task(self):
+        bitbake('m4-native')
+        result = bitbake('-C compile m4-native')
+        look_for_tasks = ['do_compile', 'do_install', 'do_populate_sysroot']
+        for task in look_for_tasks:
+            find_task = re.search("m4-native.*%s" % task, result.output)
+            self.assertTrue(find_task)
+
+    def test_bitbake_g(self):
+        result = bitbake('-g core-image-basic')
+        self.assertTrue('NOTE: PN build list saved to \'pn-buildlist\'' in result.output)
+        self.assertTrue('openssh' in ftools.read_file(os.path.join(self.builddir, 'pn-buildlist')))
+        for f in ['pn-buildlist', 'pn-depends.dot', 'package-depends.dot', 'task-depends.dot']:
+            os.remove(f)
+
+    def test_invalid_recipe_src_uri(self):
+        data = 'SRC_URI = "file://invalid"'
+        self.write_recipeinc('man', data)
+        bitbake('-ccleanall man')
+        result = bitbake('-c fetch man', ignore_status=True)
+        bitbake('-ccleanall man')
+        self.delete_recipeinc('man')
+        self.assertEqual(result.status, 1, msg='Command succeded when it should have failed')
+        self.assertTrue('ERROR: Fetcher failure: Unable to find file file://invalid anywhere. The paths that were searched were:' in result.output)
+        self.assertTrue('ERROR: Function failed: Fetcher failure for URL: \'file://invalid\'. Unable to fetch URL from any source.' in result.output)
+
+    def test_rename_downloaded_file(self):
+        data = 'SRC_URI_append = ";downloadfilename=test-aspell.tar.gz"'
+        self.write_recipeinc('aspell', data)
+        bitbake('-ccleanall aspell')
+        result = bitbake('-c fetch aspell', ignore_status=True)
+        self.delete_recipeinc('aspell')
+        self.assertEqual(result.status, 0)
+        self.assertTrue(os.path.isfile(os.path.join(get_bb_var("DL_DIR"), 'test-aspell.tar.gz')))
+        self.assertTrue(os.path.isfile(os.path.join(get_bb_var("DL_DIR"), 'test-aspell.tar.gz.done')))
+        bitbake('-ccleanall aspell')
-- 
1.8.3.1

Re: [PATCH 0/4] Script and layer for running tests

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

On Wed, Nov 27, 2013 at 10:08 AM, Stefan Stanacar <
stefanx.stanacar@intel.com> wrote:

> This series adds an oe-selftest script, some modules and a new layer
> meta-selftest.
> which are meant to help in writing tests (using python unittest) for
> various
> bitbake tools/scripts as well as simple output checks or do
> complete builds with different options (with the emphasis that everything
> checked
> is done outside of bitbake context, just like a human would do.) For more
> details,
> please see YOCTO #4740.
>

Will this have its own repository like all the other poky integrated
layers? It really should, in my opinion.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1164 bytes --]

Re: [PATCH 0/4] Script and layer for running tests

[Paul Eggleton]

Hi Chris,

On Wednesday 27 November 2013 11:42:06 Chris Larson wrote:
> On Wed, Nov 27, 2013 at 10:08 AM, Stefan Stanacar <
> stefanx.stanacar@intel.com> wrote:
> > This series adds an oe-selftest script, some modules and a new layer
> > meta-selftest.
> > which are meant to help in writing tests (using python unittest) for
> > various
> > bitbake tools/scripts as well as simple output checks or do
> > complete builds with different options (with the emphasis that everything
> > checked
> > is done outside of bitbake context, just like a human would do.) For more
> > details,
> > please see YOCTO #4740.
> 
> Will this have its own repository like all the other poky integrated
> layers? It really should, in my opinion.

It really needs to stay in the same place as the tests that rely on it; they 
can't work without it. It might be suggested that the tests should go 
elsewhere with them; but I don't think that would make sense - we want people 
to be able to run these tests easily if they make changes to OE-Core to check 
if they have caused any regressions.

If you have a look at the layer, there's not a lot in it, and I wouldn't 
expect a lot to be added in future. One possible change would be to just have 
the layer created on the fly in some temporary directory by the tests 
themselves, rather than having it in the repository; on the other hand it does 
make it a bit more difficult to look into outside of the test environment.

Thoughts?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/4] Script and layer for running tests

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 2145 bytes --]

On Thu, Nov 28, 2013 at 3:17 AM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> On Wednesday 27 November 2013 11:42:06 Chris Larson wrote:
> > On Wed, Nov 27, 2013 at 10:08 AM, Stefan Stanacar <
> > stefanx.stanacar@intel.com> wrote:
> > > This series adds an oe-selftest script, some modules and a new layer
> > > meta-selftest.
> > > which are meant to help in writing tests (using python unittest) for
> > > various
> > > bitbake tools/scripts as well as simple output checks or do
> > > complete builds with different options (with the emphasis that
> everything
> > > checked
> > > is done outside of bitbake context, just like a human would do.) For
> more
> > > details,
> > > please see YOCTO #4740.
> >
> > Will this have its own repository like all the other poky integrated
> > layers? It really should, in my opinion.
>
> It really needs to stay in the same place as the tests that rely on it;
> they
> can't work without it. It might be suggested that the tests should go
> elsewhere with them; but I don't think that would make sense - we want
> people
> to be able to run these tests easily if they make changes to OE-Core to
> check
> if they have caused any regressions.
>
> If you have a look at the layer, there's not a lot in it, and I wouldn't
> expect a lot to be added in future. One possible change would be to just
> have
> the layer created on the fly in some temporary directory by the tests
> themselves, rather than having it in the repository; on the other hand it
> does
> make it a bit more difficult to look into outside of the test environment.


I understand what you’re saying, but if this script is supposed to be
generally useful for OE/Yocto, not just Poky, then the poky repository
shouldn’t be its definitive location. Further, patches against oe-core
should be submitted against the oe-core repository, not the poky
repository, and anything poky specific should go to the poky list.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2725 bytes --]

Re: [PATCH 0/4] Script and layer for running tests

[Paul Eggleton]

On Thursday 28 November 2013 13:29:32 Chris Larson wrote:
> On Thu, Nov 28, 2013 at 3:17 AM, Paul Eggleton <
> paul.eggleton@linux.intel.com> wrote:
> > On Wednesday 27 November 2013 11:42:06 Chris Larson wrote:
> > > Will this have its own repository like all the other poky integrated
> > > layers? It really should, in my opinion.
> > 
> > It really needs to stay in the same place as the tests that rely on it;
> > they can't work without it. It might be suggested that the tests should go
> > elsewhere with them; but I don't think that would make sense - we want
> > people to be able to run these tests easily if they make changes to OE-
> > Core to check if they have caused any regressions.
> > 
> > If you have a look at the layer, there's not a lot in it, and I wouldn't
> > expect a lot to be added in future. One possible change would be to just
> > have the layer created on the fly in some temporary directory by the tests
> > themselves, rather than having it in the repository; on the other hand it
> > does make it a bit more difficult to look into outside of the test
> > environment.
> 
> I understand what you’re saying, but if this script is supposed to be
> generally useful for OE/Yocto, not just Poky, then the poky repository
> shouldn’t be its definitive location.

Poky isn't its definitive location, it has been sent as a pull request to the 
OE-Core mailing list to be applied on top of OE-Core.

> Further, patches against oe-core should be submitted against the oe-core
> repository, not the poky repository, and anything poky specific should go to
> the poky list.

Point taken, but there isn't actually anything poky-specific in this series. 
Can we please concentrate on the content of the patches?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre