[PATCH 0/4] oe-build-perf-test: remove inter-dependencies of tests

[PATCH 0/4] oe-build-perf-test: remove inter-dependencies of tests

[Markus Lehtonen]

This patchset gets rid of the dependencies between tests. This should make the
results coherent even if some tests fail, and, make it practical to run only a
subset of tests with --run-tests.


The following changes since commit 00027aee12f4bbc9a4ba607c91fcc1e0e8257fa2:

  core-image-kernel-dev.bb: Standardize use of _append and leading space. (2016-08-25 23:03:26 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib marquiz/buildperf/isolate-tests
  http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/buildperf/isolate-tests


Markus Lehtonen (4):
  oeqa.buildperf: fix checking of invalid results
  oeqa.buildperf: make tests independent
  oeqa.buildperf: treat failed measurements as errors
  oe-build-perf-test: update globalres and git even if tests failed

 meta/lib/oeqa/buildperf/base.py       | 45 ++++++++++++++++-------------------
 meta/lib/oeqa/buildperf/test_basic.py | 10 +++++---
 scripts/oe-build-perf-test            | 12 +++++-----
 3 files changed, 34 insertions(+), 33 deletions(-)

-- 
2.6.6

[PATCH 1/4] oeqa.buildperf: fix checking of invalid results

[Markus Lehtonen]

The test status check done when writing globalres log was incorrect.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index 97be58f..e4a7948 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -175,7 +175,7 @@ class BuildPerfTestResult(unittest.TextTestResult):
 
         values = ['0'] * 12
         for status, (test, msg) in self.all_results():
-            if status not in ['SUCCESS', 'FAILURE', 'EXP_SUCCESS']:
+            if status in ['ERROR', 'SKIPPED']:
                 continue
             (t_ind, t_len), (s_ind, s_len) = gr_map[test.name]
             if t_ind is not None:
-- 
2.6.6

[PATCH 2/4] oeqa.buildperf: make tests independent

[Markus Lehtonen]

Add test set-up functionality so that the individual tests do not depend
on each other. This should make sure that a failure in one test does not
affect the results of another test. The patch also makes it reasonable
to run only a subset of the tests by using the --run-tests option.

The increase in total execution time of the full suite - caused by the
additional set-up steps - is insignificant because normally no
additional tasks need to be run. The previous test has already done all
set-up work.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py       |  7 +++++++
 meta/lib/oeqa/buildperf/test_basic.py | 10 +++++++---
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index e4a7948..83439f1 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -257,6 +257,7 @@ class BuildPerfTestCase(unittest.TestCase):
     """Base class for build performance tests"""
     SYSRES = 'sysres'
     DISKUSAGE = 'diskusage'
+    build_target = None
 
     def __init__(self, *args, **kwargs):
         super(BuildPerfTestCase, self).__init__(*args, **kwargs)
@@ -271,6 +272,12 @@ class BuildPerfTestCase(unittest.TestCase):
         self.times = []
         self.sizes = []
 
+    def setUp(self):
+        """Set-up fixture for each test"""
+        if self.build_target:
+            self.log_cmd_output(['bitbake', self.build_target,
+                                 '-c', 'fetchall'])
+
     def run(self, *args, **kwargs):
         """Run test"""
         self.start_time = datetime.now()
diff --git a/meta/lib/oeqa/buildperf/test_basic.py b/meta/lib/oeqa/buildperf/test_basic.py
index 9310f3d..25dbfb0 100644
--- a/meta/lib/oeqa/buildperf/test_basic.py
+++ b/meta/lib/oeqa/buildperf/test_basic.py
@@ -23,7 +23,6 @@ class Test1P1(BuildPerfTestCase):
 
     def test1(self):
         """Measure wall clock of bitbake core-image-sato and size of tmp dir"""
-        self.log_cmd_output("bitbake {} -c fetchall".format(self.build_target))
         self.rm_tmp()
         self.rm_sstate()
         self.rm_cache()
@@ -39,8 +38,10 @@ class Test1P2(BuildPerfTestCase):
 
     def test12(self):
         """Measure bitbake virtual/kernel"""
-        self.log_cmd_output("bitbake {} -c cleansstate".format(
-            self.build_target))
+        # Build and cleans state in order to get all dependencies pre-built
+        self.log_cmd_output(['bitbake', self.build_target])
+        self.log_cmd_output(['bitbake', self.build_target, '-c', 'cleansstate'])
+
         self.sync()
         self.measure_cmd_resources(['bitbake', self.build_target], 'build',
                                    'bitbake ' + self.build_target)
@@ -73,6 +74,9 @@ class Test2(BuildPerfTestCase):
 
     def test2(self):
         """Measure bitbake core-image-sato -c rootfs with sstate"""
+        # Build once in order to populate sstate cache
+        self.log_cmd_output(['bitbake', self.build_target])
+
         self.rm_tmp()
         self.rm_cache()
         self.sync()
-- 
2.6.6

[PATCH 3/4] oeqa.buildperf: treat failed measurements as errors

[Markus Lehtonen]

Now failed measurements correctly cause a test failure (recorded as an
error). There should be no need to continue the test if one step fails,
especially now that the tests don't depend on each other.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py | 36 +++++++++++++-----------------------
 1 file changed, 13 insertions(+), 23 deletions(-)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index 83439f1..dbc534e 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -75,9 +75,7 @@ def time_cmd(cmd, **kwargs):
         if isinstance(cmd, str):
             timecmd = ' '.join(timecmd) + ' '
         timecmd += cmd
-        # TODO: 'ignore_status' could/should be removed when globalres.log is
-        # deprecated. The function would just raise an exception, instead
-        ret = runCmd2(timecmd, ignore_status=True, **kwargs)
+        ret = runCmd2(timecmd, **kwargs)
         timedata = tmpf.file.read()
     return ret, timedata
 
@@ -315,16 +313,15 @@ class BuildPerfTestCase(unittest.TestCase):
         cmd_str = cmd if isinstance(cmd, str) else ' '.join(cmd)
         log.info("Timing command: %s", cmd_str)
         cmd_log = os.path.join(self.out_dir, 'commands.log')
-        with open(cmd_log, 'a') as fobj:
-            ret, timedata = time_cmd(cmd, stdout=fobj)
-        if ret.status:
-            log.error("Time will be reported as 0. Command failed: %s",
-                      ret.status)
-            etime = timedelta(0)
-            self._failed = True
-        else:
-            match = re.search(r'.*wall clock.*: (?P<etime>.*)\n', timedata)
-            etime = str_time_to_timedelta(match.group('etime'))
+        try:
+            with open(cmd_log, 'a') as fobj:
+                ret, timedata = time_cmd(cmd, stdout=fobj)
+        except CommandError:
+            log.error("Command '%s' failed, see %s for more details", cmd_str,
+                      cmd_log)
+            raise
+        match = re.search(r'.*wall clock.*: (?P<etime>.*)\n', timedata)
+        etime = str_time_to_timedelta(match.group('etime'))
 
         measurement = {'type': self.SYSRES,
                        'name': name,
@@ -344,16 +341,9 @@ class BuildPerfTestCase(unittest.TestCase):
 
     def measure_disk_usage(self, path, name, legend):
         """Estimate disk usage of a file or directory"""
-        # TODO: 'ignore_status' could/should be removed when globalres.log is
-        # deprecated. The function would just raise an exception, instead
-        ret = runCmd2(['du', '-s', path], ignore_status=True)
-        if ret.status:
-            log.error("du failed, disk usage will be reported as 0")
-            size = 0
-            self._failed = True
-        else:
-            size = int(ret.output.split()[0])
-            log.debug("Size of %s path is %s", path, size)
+        ret = runCmd2(['du', '-s', path])
+        size = int(ret.output.split()[0])
+        log.debug("Size of %s path is %s", path, size)
         measurement = {'type': self.DISKUSAGE,
                        'name': name,
                        'legend': legend}
-- 
2.6.6

[PATCH 4/4] oe-build-perf-test: update globalres and git even if tests failed

[Markus Lehtonen]

Write globalres log file and commit results to Git even if some tests
failed. Now that tests do not depend on each other there should be no
risk of bogus results caused by test failures.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/oe-build-perf-test | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/scripts/oe-build-perf-test b/scripts/oe-build-perf-test
index 35a4839..3dab070 100755
--- a/scripts/oe-build-perf-test
+++ b/scripts/oe-build-perf-test
@@ -191,13 +191,13 @@ def main(argv=None):
     # Restore logger output to stderr
     log.handlers[0].setLevel(log.level)
 
+    if args.globalres_file:
+        result.update_globalres_file(args.globalres_file)
+    if args.commit_results:
+        result.git_commit_results(args.commit_results,
+                                  args.commit_results_branch,
+                                  args.commit_results_tag)
     if result.wasSuccessful():
-        if args.globalres_file:
-            result.update_globalres_file(args.globalres_file)
-        if args.commit_results:
-            result.git_commit_results(args.commit_results,
-                                      args.commit_results_branch,
-                                      args.commit_results_tag)
         return 0
 
     return 1
-- 
2.6.6