[RFC PATCH 0/4] Script and layer for running tests

[RFC PATCH 0/4] Script and layer for running tests

[Stefan Stanacar]

Hello,

This series adds an oe-selftest script, some modules and a new layer meta-selftest.
which are meant to help in writing tests (using python unittest) for various
bitbake tools/scripts as well as simple output checks or do 
complete builds with different options (with the emphasis that everything checked
is done outside of bitbake context, just like a human would do.) For more details,
please see YOCTO #4740.

Cheers,
Stefan


The following changes since commit e15893adf9268b2920b24c52d5c2bb777c6f778e:

  bitbake: serv.py: Give pr-server up to 5 seconds to commit data (2013-11-18 17:19:11 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib stefans/testme4
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=stefans/testme4

Alexandru Palalau (1):
  scripts/lib/selftest: add more simple build tests

Corneliu Stoicescu (2):
  meta-selftest: create a new test layer to be used by oe-selftest
    script
  scripts/lib/selftest: add a new module for bitbake output tests

Stefan Stanacar (1):
  scripts/oe-selftest: script to run builds as unitests against bitbake
    or various scripts

 .gitignore                                         |   1 +
 meta-selftest/COPYING.MIT                          |  17 +++
 meta-selftest/README                               |   2 +
 meta-selftest/classes/test_events.bbclass          |  16 +++
 meta-selftest/conf/layer.conf                      |  10 ++
 .../recipes-test/aspell/aspell_0.0.0.1.bb          |  28 ++++
 .../recipes-test/man/man/man-1.5h1-make.patch      |  16 +++
 meta-selftest/recipes-test/man/man_1.6g.bbappend   |   2 +
 .../xcursor-transparent-theme_0.1.1.bbappend       |   3 +
 scripts/lib/selftest/__init__.py                   |   0
 scripts/lib/selftest/base.py                       |  54 ++++++++
 scripts/lib/selftest/tests/__init__.py             |   2 +
 scripts/lib/selftest/tests/bboptions.py            |  39 ++++++
 scripts/lib/selftest/tests/bboutput.py             |  86 ++++++++++++
 scripts/lib/selftest/utils/__init__.py             |   0
 scripts/lib/selftest/utils/commands.py             | 146 +++++++++++++++++++++
 scripts/lib/selftest/utils/ftools.py               |  27 ++++
 scripts/oe-selftest                                | 130 ++++++++++++++++++
 18 files changed, 579 insertions(+)
 create mode 100644 meta-selftest/COPYING.MIT
 create mode 100644 meta-selftest/README
 create mode 100644 meta-selftest/classes/test_events.bbclass
 create mode 100644 meta-selftest/conf/layer.conf
 create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
 create mode 100644 meta-selftest/recipes-test/man/man/man-1.5h1-make.patch
 create mode 100644 meta-selftest/recipes-test/man/man_1.6g.bbappend
 create mode 100644 meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend
 create mode 100644 scripts/lib/selftest/__init__.py
 create mode 100644 scripts/lib/selftest/base.py
 create mode 100644 scripts/lib/selftest/tests/__init__.py
 create mode 100644 scripts/lib/selftest/tests/bboptions.py
 create mode 100644 scripts/lib/selftest/tests/bboutput.py
 create mode 100644 scripts/lib/selftest/utils/__init__.py
 create mode 100644 scripts/lib/selftest/utils/commands.py
 create mode 100644 scripts/lib/selftest/utils/ftools.py
 create mode 100755 scripts/oe-selftest

-- 
1.8.3.1

[RFC PATCH 1/4] scripts/oe-selftest: script to run builds as unitests against bitbake or various scripts

[Stefan Stanacar]

The purpose of oe-selftest is to run unittest modules added in scripts/lib/selftest/tests,
which are tests against bitbake tools.
The scripts/lib/selftest/tests path for tests is only temporary.
Tests probably will be moved to meta/lib/oeqa/<something> once we've worked out how that can work.
(the reason to have them there is to allow layers to add their own tests).

Right now the script it's useful for simple tests like:
  - "bitbake --someoption, change some metadata, bitbake X, check something" type scenarios (PR service, error output, etc)
  - or "bitbake-layers <...>" type scripts and yocto-bsp tools.
This commit also adds some helper methods that the tests will use (which are currently used by the script itself,
and those also need to move if the tests move).

Also, most of the tests will have a dependency on a meta-selftest layer
which contains specially modified recipes/bbappends/include files for the purpose of the tests.

It's a simple matter or sourcing the enviroment, adding the meta-selftest layer
and running: oe-selftest to get some results.

[ YOCTO #4740 ]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 scripts/lib/selftest/__init__.py       |   0
 scripts/lib/selftest/base.py           |  54 ++++++++++++
 scripts/lib/selftest/tests/__init__.py |   0
 scripts/lib/selftest/utils/__init__.py |   0
 scripts/lib/selftest/utils/commands.py | 146 +++++++++++++++++++++++++++++++++
 scripts/lib/selftest/utils/ftools.py   |  27 ++++++
 scripts/oe-selftest                    | 130 +++++++++++++++++++++++++++++
 7 files changed, 357 insertions(+)
 create mode 100644 scripts/lib/selftest/__init__.py
 create mode 100644 scripts/lib/selftest/base.py
 create mode 100644 scripts/lib/selftest/tests/__init__.py
 create mode 100644 scripts/lib/selftest/utils/__init__.py
 create mode 100644 scripts/lib/selftest/utils/commands.py
 create mode 100644 scripts/lib/selftest/utils/ftools.py
 create mode 100755 scripts/oe-selftest

diff --git a/scripts/lib/selftest/__init__.py b/scripts/lib/selftest/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/scripts/lib/selftest/base.py b/scripts/lib/selftest/base.py
new file mode 100644
index 0000000..4d90828
--- /dev/null
+++ b/scripts/lib/selftest/base.py
@@ -0,0 +1,54 @@
+# Copyright (c) 2013 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+# DESCRIPTION
+# Base class inherited by test classes in scripts/lib/selftest/tests
+
+import unittest
+import os
+import sys
+import logging
+
+import selftest.utils.ftools as ftools
+
+
+class oeSelfTest(unittest.TestCase):
+
+    log = logging.getLogger("selftest.base")
+    longMessage = True
+
+    def __init__(self, methodName="runTest"):
+        self.builddir = os.environ.get("BUILDDIR")
+        self.localconf_path = os.path.join(self.builddir, "conf/local.conf")
+        self.testinc_path = os.path.join(self.builddir, "conf/selftest.inc")
+        super(oeSelfTest, self).__init__(methodName)
+
+    def setUp(self):
+        os.chdir(self.builddir)
+        # we don't know what the previous test set in here, we need a fresh start
+        if os.path.isfile(self.testinc_path):
+            os.remove(self.testinc_path)
+
+    def write_config(self, data):
+        self.log.debug("Writing to: %s\n%s\n" % (self.testinc_path, data))
+        ftools.write_file(self.testinc_path, data)
+
+    def append_config(self, data):
+        self.log.debug("Appending to: %s\n%s\n" % (self.testinc_path, data))
+        ftools.append_file(self.testinc_path, data)
+
+    def remove_config(self, data):
+        self.log.debug("Removing from: %s\n\%s\n" % (self.testinc_path, data))
+        ftools.remove_from_file(self.testinc_path, data)
diff --git a/scripts/lib/selftest/tests/__init__.py b/scripts/lib/selftest/tests/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/scripts/lib/selftest/utils/__init__.py b/scripts/lib/selftest/utils/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/scripts/lib/selftest/utils/commands.py b/scripts/lib/selftest/utils/commands.py
new file mode 100644
index 0000000..c808927
--- /dev/null
+++ b/scripts/lib/selftest/utils/commands.py
@@ -0,0 +1,146 @@
+# Copyright (c) 2013 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+# DESCRIPTION
+# This module is mainly used by scripts/oe-selftest and modules under scripts/lib/selftest
+# It provides a class and methods for running commands on the host in a convienent way for tests.
+
+import os
+import sys
+import signal
+import subprocess
+import threading
+import logging
+
+class Command(object):
+    def __init__(self, command, bg=False, timeout=None, data=None, **options):
+
+        self.defaultopts = {
+            "stdout": subprocess.PIPE,
+            "stderr": subprocess.STDOUT,
+            "stdin": None,
+            "shell": False,
+            "bufsize": -1,
+        }
+
+        self.cmd = command
+        self.bg = bg
+        self.timeout = timeout
+        self.data = data
+
+        self.options = dict(self.defaultopts)
+        if isinstance(self.cmd, basestring):
+            self.options["shell"] = True
+        if self.data:
+            self.options['stdin'] = subprocess.PIPE
+        self.options.update(options)
+
+        self.status = None
+        self.output = None
+        self.error = None
+        self.thread = None
+
+        self.log = logging.getLogger("selftest.commands")
+
+    def run(self):
+        self.process = subprocess.Popen(self.cmd, **self.options)
+
+        def commThread():
+            self.output, self.error = self.process.communicate(self.data)
+
+        self.thread = threading.Thread(target=commThread)
+        self.thread.start()
+
+        self.log.debug("Running command '%s'" % self.cmd)
+
+        if not self.bg:
+            self.thread.join(self.timeout)
+            self.stop()
+
+    def stop(self):
+        if self.thread.isAlive():
+            self.process.terminate()
+            # let's give it more time to terminate gracefully before killing it
+            self.thread.join(5)
+            if self.thread.isAlive():
+                self.process.kill()
+                self.thread.join()
+
+        self.output = self.output.rstrip()
+        self.status = self.process.poll()
+
+        self.log.debug("Command '%s' returned %d as exit code." % (self.cmd, self.status))
+        # logging the complete output is insane
+        # bitbake -e output is really big
+        # and makes the log file useless
+        if self.status:
+            lout = "\n".join(self.output.splitlines()[-20:])
+            self.log.debug("Last 20 lines:\n%s" % lout)
+
+
+class Result(object):
+    pass
+
+def runCmd(command, ignore_status=False, timeout=None, **options):
+
+    result = Result()
+
+    cmd = Command(command, timeout=timeout, **options)
+    cmd.run()
+
+    result.command = command
+    result.status = cmd.status
+    result.output = cmd.output
+    result.pid = cmd.process.pid
+
+    if result.status and not ignore_status:
+        raise Exception("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, result.output))
+
+    return result
+
+
+def bitbake(command, ignore_status=False, timeout=None, **options):
+    if isinstance(command, basestring):
+        cmd = "bitbake " + command
+    else:
+        cmd = [ "bitbake" ] + command
+
+    return runCmd(cmd, ignore_status, timeout, **options)
+
+
+def get_bb_env(target=None):
+    if target:
+        return runCmd("bitbake -e %s" % target).output
+    else:
+        return runCmd("bitbake -e").output
+
+def get_bb_var(var, target=None):
+    val = None
+    bbenv = get_bb_env(target)
+    for line in bbenv.splitlines():
+        if line.startswith(var + "="):
+            val = line.split('=')[1]
+            val = val.replace('\"','')
+            break
+    return val
+
+def get_test_layer():
+    layers = get_bb_var("BBLAYERS").split()
+    testlayer = None
+    for l in layers:
+        if "/meta-selftest" in l and os.path.isdir(l):
+            testlayer = l
+            break
+    return testlayer
diff --git a/scripts/lib/selftest/utils/ftools.py b/scripts/lib/selftest/utils/ftools.py
new file mode 100644
index 0000000..64ebe3d
--- /dev/null
+++ b/scripts/lib/selftest/utils/ftools.py
@@ -0,0 +1,27 @@
+import os
+import re
+
+def write_file(path, data):
+    wdata = data.rstrip() + "\n"
+    with open(path, "w") as f:
+        f.write(wdata)
+
+def append_file(path, data):
+    wdata = data.rstrip() + "\n"
+    with open(path, "a") as f:
+            f.write(wdata)
+
+def read_file(path):
+    data = None
+    with open(path) as f:
+        data = f.read()
+    return data
+
+def remove_from_file(path, data):
+    lines = read_file(path).splitlines()
+    rmdata = data.strip().splitlines()
+    for l in rmdata:
+        for c in range(0, lines.count(l)):
+            i = lines.index(l)
+            del(lines[i])
+    write_file(path, "\n".join(lines))
diff --git a/scripts/oe-selftest b/scripts/oe-selftest
new file mode 100755
index 0000000..2d4d4aa
--- /dev/null
+++ b/scripts/oe-selftest
@@ -0,0 +1,130 @@
+#!/usr/bin/env python
+
+# Copyright (c) 2013 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+# DESCRIPTION
+# This script runs tests defined in scripts/lib/selftest/tests
+# It's purpose is to automate the testing of different bitbake tools.
+# To use it you just need to source your build environment setup script and
+# add the meta-selftest layer to your BBLAYERS.
+# Call the script as: "oe-selftest" to run all the tests in in scripts/lib/selftest/tests.
+# Call the script as: "oe-selftest <module>.<Class>.<method>" to run just a single test
+# E.g: "oe-selftest bboutput.BitbakeLayers" will run just the BitbakeLayers class in scripts/lib/selftest/tests/bboutput.py
+
+
+import os
+import sys
+import unittest
+import logging
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'lib'))
+
+from selftest.utils.commands import runCmd, get_bb_var
+import selftest.utils.ftools as ftools
+
+def logger_create():
+    log = logging.getLogger("selftest")
+    log.setLevel(logging.DEBUG)
+
+    fh = logging.FileHandler(filename='oe-selftest.log', mode='w')
+    fh.setLevel(logging.DEBUG)
+
+    ch = logging.StreamHandler(sys.stdout)
+    ch.setLevel(logging.INFO)
+
+    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
+    fh.setFormatter(formatter)
+    ch.setFormatter(formatter)
+
+    log.addHandler(fh)
+    log.addHandler(ch)
+
+    return log
+
+log = logger_create()
+
+def preflight_check():
+
+    log.info("Checking that everything is in order before running the tests")
+
+    if not os.environ.get("BUILDDIR"):
+        log.error("BUILDDIR isn't set. Did you forget to source your build environment setup script?")
+        return False
+
+    builddir = os.environ.get("BUILDDIR")
+    if os.getcwd() != builddir:
+        log.info("Changing cwd to %s" % builddir)
+        os.chdir(builddir)
+
+    if not "meta-selftest" in get_bb_var("BBLAYERS"):
+        log.error("You don't seem to have the meta-selftest layer in BBLAYERS")
+        return False
+
+    log.info("Running bitbake -p")
+    runCmd("bitbake -p")
+
+    return True
+
+def add_include():
+    builddir = os.environ.get("BUILDDIR")
+    if "#include added by oe-selftest.py" \
+        not in ftools.read_file(os.path.join(builddir, "conf/local.conf")):
+            log.info("Adding: \"include selftest.inc\" in local.conf")
+            ftools.append_file(os.path.join(builddir, "conf/local.conf"), \
+                    "\n#include added by oe-selftest.py\ninclude selftest.inc")
+
+
+def remove_include():
+    builddir = os.environ.get("BUILDDIR")
+    if "#include added by oe-selftest.py" \
+        in ftools.read_file(os.path.join(builddir, "conf/local.conf")):
+            log.info("Removing the include from local.conf")
+            ftools.remove_from_file(os.path.join(builddir, "conf/local.conf"), \
+                    "#include added by oe-selftest.py\ninclude selftest.inc")
+
+def main():
+
+    if not preflight_check():
+        return 1
+
+    tests_list = []
+    for x in sys.argv[1:]:
+        tests_list.append("selftest.tests." + x)
+    if not tests_list:
+        tests_list.append("selftest.tests")
+
+    suite = unittest.TestSuite()
+    loader = unittest.TestLoader()
+    loader.sortTestMethodsUsing = None
+    runner = unittest.TextTestRunner(verbosity=2)
+    log.info("Running tests from: %s" % " ".join(tests_list))
+    suite = loader.loadTestsFromNames(tests_list)
+    add_include()
+    result = runner.run(suite)
+    log.info("Finished")
+
+    return 0
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc(5)
+    finally:
+        remove_include()
+    sys.exit(ret)
-- 
1.8.3.1

[RFC PATCH 2/4] meta-selftest: create a new test layer to be used by oe-selftest script

[Stefan Stanacar]

From: Corneliu Stoicescu <corneliux.stoicescu@intel.com>

Everything in this layer is meant to be used by tests.

Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 .gitignore                                         |  1 +
 meta-selftest/COPYING.MIT                          | 17 +++++++++++++
 meta-selftest/README                               |  2 ++
 meta-selftest/classes/test_events.bbclass          | 16 +++++++++++++
 meta-selftest/conf/layer.conf                      | 10 ++++++++
 .../recipes-test/aspell/aspell_0.0.0.1.bb          | 28 ++++++++++++++++++++++
 .../recipes-test/man/man/man-1.5h1-make.patch      | 16 +++++++++++++
 meta-selftest/recipes-test/man/man_1.6g.bbappend   |  2 ++
 .../xcursor-transparent-theme_0.1.1.bbappend       |  3 +++
 9 files changed, 95 insertions(+)
 create mode 100644 meta-selftest/COPYING.MIT
 create mode 100644 meta-selftest/README
 create mode 100644 meta-selftest/classes/test_events.bbclass
 create mode 100644 meta-selftest/conf/layer.conf
 create mode 100644 meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
 create mode 100644 meta-selftest/recipes-test/man/man/man-1.5h1-make.patch
 create mode 100644 meta-selftest/recipes-test/man/man_1.6g.bbappend
 create mode 100644 meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend

diff --git a/.gitignore b/.gitignore
index 0171597..b6755fe 100644
--- a/.gitignore
+++ b/.gitignore
@@ -21,3 +21,4 @@ bitbake/doc/manual/pdf/
 bitbake/doc/manual/txt/
 bitbake/doc/manual/xhtml/
 pull-*/
+!meta-selftest
diff --git a/meta-selftest/COPYING.MIT b/meta-selftest/COPYING.MIT
new file mode 100644
index 0000000..89de354
--- /dev/null
+++ b/meta-selftest/COPYING.MIT
@@ -0,0 +1,17 @@
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
diff --git a/meta-selftest/README b/meta-selftest/README
new file mode 100644
index 0000000..e0f803d
--- /dev/null
+++ b/meta-selftest/README
@@ -0,0 +1,2 @@
+This layer is intended as test layer, used by scripts/oe-selftest
+and it's probably a mistake to include it in your builds.
diff --git a/meta-selftest/classes/test_events.bbclass b/meta-selftest/classes/test_events.bbclass
new file mode 100644
index 0000000..35324eb
--- /dev/null
+++ b/meta-selftest/classes/test_events.bbclass
@@ -0,0 +1,16 @@
+python test1_eventhandler() {
+    bb.note("Test for bb.event.BuildStarted")
+}
+python test2_eventhandler() {
+    bb.note("Test for bb.event.BuildCompleted")
+}
+python test3_eventhandler() {
+    bb.note("Test for bb.event.InvalidEvent")
+}
+
+addhandler test1_eventhandler
+test1_eventhandler[eventmask] = "bb.event.BuildStarted"
+addhandler test2_eventhandler
+test2_eventhandler[eventmask] = "bb.event.BuildCompleted"
+addhandler test3_eventhandler
+test3_eventhandler[eventmask] = "bb.event.InvalidEvent"
diff --git a/meta-selftest/conf/layer.conf b/meta-selftest/conf/layer.conf
new file mode 100644
index 0000000..e3322f6
--- /dev/null
+++ b/meta-selftest/conf/layer.conf
@@ -0,0 +1,10 @@
+# We have a conf and classes directory, add to BBPATH
+BBPATH .= ":${LAYERDIR}"
+
+# We have recipes-* directories, add to BBFILES
+BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
+	${LAYERDIR}/recipes-*/*/*.bbappend"
+
+BBFILE_COLLECTIONS += "selftest"
+BBFILE_PATTERN_selftest = "^${LAYERDIR}/"
+BBFILE_PRIORITY_selftest = "6"
diff --git a/meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb b/meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
new file mode 100644
index 0000000..79eb06b
--- /dev/null
+++ b/meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
@@ -0,0 +1,28 @@
+# This recipe is a copy from the oe-core one.
+# It has a lower and invalid version number in order not to be accidentally used by bitbake.
+# It is used for tests that require duplicate recipe files.
+
+DESCRIPTION = "GNU Aspell spell-checker"
+SECTION = "console/utils"
+LICENSE="LGPLv2 | LGPLv2.1"
+LIC_FILES_CHKSUM = "file://COPYING;md5=7fbc338309ac38fefcd64b04bb903e34"
+PR = "r1"
+
+SRC_URI = "${GNU_MIRROR}/aspell/aspell-${PV}.tar.gz"
+SRC_URI[md5sum] = "e66a9c9af6a60dc46134fdacf6ce97d7"
+SRC_URI[sha256sum] = "f52583a83a63633701c5f71db3dc40aab87b7f76b29723aeb27941eff42df6e1"
+
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[curses] = "--enable-curses,--disable-curses,ncurses"
+
+PACKAGES += "libaspell libpspell libpspell-dev aspell-utils"
+
+FILES_${PN}-dbg += "${libdir}/aspell-0.60/.debu*"
+FILES_libaspell = "${libdir}/libaspell.so.* ${libdir}/aspell*"
+FILES_aspell-utils = "${bindir}/word-list-compress ${bindir}/aspell-import ${bindir}/run-with-aspell ${bindir}/pre*"
+FILES_${PN} = "${bindir}/aspell"
+FILES_libpspell = "${libdir}/libpspell.so.*"
+FILES_libpspell-dev = "${libdir}/libpspell* ${bindir}/pspell-config ${includedir}/pspell"
+
+ARM_INSTRUCTION_SET = "arm"
+inherit autotools gettext
diff --git a/meta-selftest/recipes-test/man/man/man-1.5h1-make.patch b/meta-selftest/recipes-test/man/man/man-1.5h1-make.patch
new file mode 100644
index 0000000..a0d59b7
--- /dev/null
+++ b/meta-selftest/recipes-test/man/man/man-1.5h1-make.patch
@@ -0,0 +1,16 @@
+Test patch here!
+This is invalid patch used by tests in scripts/lib/selftests.
+
+--- man-1.5g/man/Makefile.in.mike	Fri Apr  9 13:35:54 1999
++++ man-1.5g/man/Makefile.in	Fri Apr  9 13:36:45 1999
+@@ -1,8 +1,8 @@
+ #MAKE THIS PATCH INVALID
+ MAN1 = man whatis apropos
+-MAN5 = man.conf
++MAN5 = man.config
+ MAN8 = makewhatis
+-ALL = man.1 whatis.1 apropos.1 man.conf.5
++ALL = man.1 whatis.1 apropos.1 man.config.5
+ MAYBE8 = makewhatis
+ 
+ .SUFFIXES: .man .1 .5 .8
diff --git a/meta-selftest/recipes-test/man/man_1.6g.bbappend b/meta-selftest/recipes-test/man/man_1.6g.bbappend
new file mode 100644
index 0000000..2057209
--- /dev/null
+++ b/meta-selftest/recipes-test/man/man_1.6g.bbappend
@@ -0,0 +1,2 @@
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
diff --git a/meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend b/meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend
new file mode 100644
index 0000000..29e86f6
--- /dev/null
+++ b/meta-selftest/recipes-test/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bbappend
@@ -0,0 +1,3 @@
+TEST_VAR = ""
+# This bbappend is used to alter the recipe using the test_recipe.inc file created by tests.
+include test_recipe.inc
-- 
1.8.3.1

[RFC PATCH 3/4] scripts/lib/selftest: add a new module for bitbake output tests

[Stefan Stanacar]

From: Corneliu Stoicescu <corneliux.stoicescu@intel.com>

Tests for bitbake-layers and expected output for some bitbake options.

Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 scripts/lib/selftest/tests/__init__.py |  1 +
 scripts/lib/selftest/tests/bboutput.py | 86 ++++++++++++++++++++++++++++++++++
 2 files changed, 87 insertions(+)
 create mode 100644 scripts/lib/selftest/tests/bboutput.py

diff --git a/scripts/lib/selftest/tests/__init__.py b/scripts/lib/selftest/tests/__init__.py
index e69de29..3acc480 100644
--- a/scripts/lib/selftest/tests/__init__.py
+++ b/scripts/lib/selftest/tests/__init__.py
@@ -0,0 +1 @@
+from selftest.tests.bboutput import *
diff --git a/scripts/lib/selftest/tests/bboutput.py b/scripts/lib/selftest/tests/bboutput.py
new file mode 100644
index 0000000..386572f
--- /dev/null
+++ b/scripts/lib/selftest/tests/bboutput.py
@@ -0,0 +1,86 @@
+import unittest
+import os
+import logging
+import re
+import shutil
+from selftest.base import oeSelfTest
+from selftest.utils.commands import runCmd, bitbake, get_bb_var, get_test_layer
+import selftest.utils.ftools as ftools
+
+class BitbakeTests(oeSelfTest):
+
+    def test_run_bitbake_from_1(self):
+        os.chdir(os.path.join(self.builddir, 'conf'))
+        bitbake('-e')
+
+    def test_run_bitbake_from_2(self):
+        my_env = os.environ.copy()
+        my_env['BBPATH'] = my_env['BUILDDIR']
+        os.chdir(os.path.dirname(os.environ['BUILDDIR']))
+        bitbake('-e', env=my_env)
+
+    def test_event_handler(self):
+        self.write_config("INHERIT += \"test_events\"")
+        result = bitbake('m4-native')
+        find_build_started = re.search("NOTE: Test for bb\.event\.BuildStarted(\n.*)*NOTE: Preparing runqueue", result.output)
+        find_build_completed = re.search("Tasks Summary:.*(\n.*)*NOTE: Test for bb\.event\.BuildCompleted", result.output)
+        self.assertTrue(find_build_started, msg = "Match failed in:\n%s"  % result.output)
+        self.assertTrue(find_build_completed, msg = "Match failed in:\n%s" % result.output)
+        self.assertFalse('Test for bb.event.InvalidEvent' in result.output)
+
+    def test_local_sstate(self):
+        bitbake('m4-native -ccleansstate')
+        bitbake('m4-native')
+        bitbake('m4-native -cclean')
+        result = bitbake('m4-native')
+        find_setscene = re.search("m4-native.*do_.*_setscene", result.output)
+        self.assertTrue(find_setscene)
+
+    def test_bitbake_invalid_recipe(self):
+        result = bitbake('-b asdf', ignore_status=True)
+        self.assertTrue("ERROR: Unable to find any recipe file matching 'asdf'" in result.output)
+
+    def test_bitbake_invalid_target(self):
+        result = bitbake('asdf', ignore_status=True)
+        self.assertTrue("ERROR: Nothing PROVIDES 'asdf'" in result.output)
+
+    def test_warnings_errors(self):
+        result = bitbake('-b asdf', ignore_status=True)
+        find_warnings = re.search("Summary: There was [1-9][0-9]* WARNING message shown.", result.output)
+        find_errors = re.search("Summary: There was [1-9][0-9]* ERROR message shown.", result.output)
+        self.assertTrue(find_warnings)
+        self.assertTrue(find_errors)
+
+    def test_invalid_patch(self):
+        inc_file = os.path.join(get_test_layer(), 'recipes-test/man/test_recipe.inc')
+        ftools.write_file(inc_file, 'SRC_URI += "file://man-1.5h1-make.patch"')
+        result = bitbake('man -c patch', ignore_status=True)
+        os.remove(inc_file)
+        bitbake('-cclean man')
+        self.assertTrue("ERROR: Function failed: patch_do_patch" in result.output)
+
+
+class BitbakeLayers(oeSelfTest):
+
+    def test_bitbakelayers_showlayers(self):
+        result = runCmd('bitbake-layers show_layers')
+        self.assertTrue('meta-selftest' in result.output)
+
+    def test_bitbakelayers_showappends(self):
+        result = runCmd('bitbake-layers show_appends')
+        self.assertTrue('xcursor-transparent-theme_0.1.1.bbappend' in result.output, msg='xcursor-transparent-theme_0.1.1.bbappend file was not recognised')
+
+    def test_bitbakelayers_showoverlayed(self):
+        result = runCmd('bitbake-layers show_overlayed')
+        self.assertTrue('aspell' in result.output, msg='xcursor-transparent-theme_0.1.1.bbappend file was not recognised')
+
+    def test_bitbakelayers_flatten(self):
+        if os.path.isdir(os.path.join(self.builddir, 'test')):
+            shutil.rmtree(os.path.join(self.builddir, 'test'))
+        result = runCmd('bitbake-layers flatten test')
+        bb_file = os.path.join(self.builddir, 'test/recipes-graphics/xcursor-transparent-theme/xcursor-transparent-theme_0.1.1.bb')
+        self.assertTrue(os.path.isfile(bb_file))
+        contents = ftools.read_file(bb_file)
+        find_in_contents = re.search("##### bbappended from meta-selftest #####\nTEST_VAR = \"\"", contents)
+        shutil.rmtree(os.path.join(self.builddir, 'test'))
+        self.assertTrue(find_in_contents)
-- 
1.8.3.1

[RFC PATCH 4/4] scripts/lib/selftest: add more simple build tests

[Stefan Stanacar]

From: Alexandru Palalau <alexandrux.palalau@intel.com>

Tests for RM_OLD_IMAGE and for WARN_QA/ERROR_QA behaviour.

Signed-off-by: Alexandru Palalau <alexandrux.palalau@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
---
 scripts/lib/selftest/tests/__init__.py  |  1 +
 scripts/lib/selftest/tests/bboptions.py | 39 +++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
 create mode 100644 scripts/lib/selftest/tests/bboptions.py

diff --git a/scripts/lib/selftest/tests/__init__.py b/scripts/lib/selftest/tests/__init__.py
index 3acc480..7826054 100644
--- a/scripts/lib/selftest/tests/__init__.py
+++ b/scripts/lib/selftest/tests/__init__.py
@@ -1 +1,2 @@
 from selftest.tests.bboutput import *
+from selftest.tests.bboptions import *
diff --git a/scripts/lib/selftest/tests/bboptions.py b/scripts/lib/selftest/tests/bboptions.py
new file mode 100644
index 0000000..ad35386
--- /dev/null
+++ b/scripts/lib/selftest/tests/bboptions.py
@@ -0,0 +1,39 @@
+import unittest
+import os
+import logging
+
+from selftest.base import oeSelfTest
+from selftest.utils.commands import runCmd, bitbake, get_bb_var, get_test_layer
+import selftest.utils.ftools as ftools
+
+class OptionsTests(oeSelfTest):
+
+    def test_options_warnqa_errorqa_switch(self):
+        bitbake("xcursor-transparent-theme -ccleansstate")
+
+        if "packages-list" not in get_bb_var("ERROR_QA"):
+            self.write_config("ERROR_QA_append = \" packages-list\"")
+
+        xcursor_path = os.path.join(get_test_layer(), "recipes-test/xcursor-transparent-theme/test_recipe.inc")
+        ftools.write_file(xcursor_path, 'PACKAGES += \"${PN}-dbg\"')
+
+        res = bitbake("xcursor-transparent-theme", ignore_status=True)
+        self.assertTrue("ERROR: QA Issue" in res.output)
+        self.assertEqual(res.status, 1)
+
+        self.append_config("ERROR_QA_remove = \"packages-list\"")
+        self.append_config("WARN_QA_append = \" packages-list\"")
+        bitbake("xcursor-transparent-theme")
+        bitbake("xcursor-transparent-theme -ccleansstate")
+        os.remove(xcursor_path)
+
+    def test_options_rm_old_image(self):
+        bitbake("core-image-minimal")
+
+        deploydir = get_bb_var("DEPLOY_DIR_IMAGE", target="core-image-minimal")
+        imagename = get_bb_var("IMAGE_LINK_NAME", target="core-image-minimal")
+        oldimgpath = os.path.realpath(os.path.join(deploydir, imagename + ".ext3"))
+
+        self.append_config("RM_OLD_IMAGE = \"1\"")
+        bitbake("core-image-minimal")
+        self.assertFalse(os.path.exists(oldimgpath), msg="Old image path still exists: %s" % oldimgpath)
-- 
1.8.3.1

Re: [RFC PATCH 2/4] meta-selftest: create a new test layer to be used by oe-selftest script

[Paul Eggleton]

Hi Stefan,

On Tuesday 19 November 2013 17:43:35 Stefan Stanacar wrote:
> +++ b/meta-selftest/recipes-test/aspell/aspell_0.0.0.1.bb
> @@ -0,0 +1,28 @@
> +# This recipe is a copy from the oe-core one.
> +# It has a lower and invalid version number in order not to be accidentally
> used by bitbake. 
> +# It is used for tests that require duplicate recipe files.

FYI the term we use for this is "overlayed" rather than "duplicate" (where the 
two or more recipes exist in the same configuration).

Cheers,
Paul


-- 

Paul Eggleton
Intel Open Source Technology Centre