[PATCH 00/19] oe-build-pef-test: support saving results in a Git repo

[PATCH 00/19] oe-build-pef-test: support saving results in a Git repo

[Markus Lehtonen]

This patchset makes it possible to store build performance test results in a
Git repository. The support is written directly in oe-build-perf-test script
which hopefully makes it easier use. The new feature is disabled by default,
and, it must be enabled with --commit-results option. Further command line
options allow customization of the branch and tag names to use for the results.

[YOCTO #9624]


The following changes since commit 25f6af8895d5f5c6dcedde0a21285d63522769c8:

  oeqa/buildiptables: Switch from netfilter.org to yoctoproject.org mirror (2016-08-23 17:59:20 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib marquiz/buildperf/git
  http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/buildperf/git


Markus Lehtonen (19):
  oe-build-perf-test: implement --run-tests option
  oe-build-perf-test: use absolute paths in cmdline args
  oeqa.utils.git: support git commands with updated env
  oeqa.utils.git: introduce GitRepo.rev_parse()
  oeqa.utils.git: implement GitRepo.get_current_branch()
  oeqa.utils.git.GitRepo: new arg to require topdir
  oeqa.buildperf: use term commit instead of revision
  oe-build-perf-test: support committing results data to Git
  oe-build-perf-test: implement --commit-results-branch
  oeqa.utils.git: implement init() method
  oe-build-perf-test: pre-check Git repo when using --commit-results
  oe-build-perf-test: tag results committed to Git
  oe-build-perf-test: new {tag_num} keyword for --commit-results-tag
  oeqa.buildperf: add git commit count to result data
  oe-build-perf-test: add {git_commit_count} keyword for
    --commit-results-tag
  build-perf-test-wrapper.sh: parse args with getopts
  build-perf-test-wrapper.sh: allow saving results in Git
  build-perf-test-wrapper.sh: make archive dir configurable
  build-perf-test-wrapper.sh: make workdir configurable

 meta/lib/oeqa/buildperf/base.py            | 102 +++++++++++++++++++++++------
 meta/lib/oeqa/utils/git.py                 |  38 +++++++++--
 scripts/contrib/build-perf-test-wrapper.sh |  70 +++++++++++++++-----
 scripts/oe-build-perf-test                 |  51 ++++++++++++++-
 4 files changed, 218 insertions(+), 43 deletions(-)

-- 
2.6.6

[PATCH 01/19] oe-build-perf-test: implement --run-tests option

[Markus Lehtonen]

Makes it possible to run only a subset of tests.

NOTE: The tests currently have (unwritten) dependencies on each other so
use this option with care. Mainly for debugging.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/oe-build-perf-test | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/scripts/oe-build-perf-test b/scripts/oe-build-perf-test
index 88af40a..808531e 100755
--- a/scripts/oe-build-perf-test
+++ b/scripts/oe-build-perf-test
@@ -105,6 +105,8 @@ def parse_args(argv):
                         help="Lock file to use")
     parser.add_argument('-o', '--out-dir', default='results-{date}',
                         help="Output directory for test results")
+    parser.add_argument('--run-tests', nargs='+', metavar='TEST',
+                        help="List of tests to run")
 
     return parser.parse_args(argv)
 
@@ -133,7 +135,10 @@ def main(argv=None):
 
     # Load build perf tests
     loader = BuildPerfTestLoader()
-    suite = loader.discover(start_dir=os.path.dirname(oeqa.buildperf.__file__))
+    if args.run_tests:
+        suite = loader.loadTestsFromNames(args.run_tests, oeqa.buildperf)
+    else:
+        suite = loader.loadTestsFromModule(oeqa.buildperf)
 
     archive_build_conf(out_dir)
     runner = BuildPerfTestRunner(out_dir, verbosity=2)
-- 
2.6.6

[PATCH 02/19] oe-build-perf-test: use absolute paths in cmdline args

[Markus Lehtonen]

This is safer as the current working directory may change.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/oe-build-perf-test | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/oe-build-perf-test b/scripts/oe-build-perf-test
index 808531e..21759c6 100755
--- a/scripts/oe-build-perf-test
+++ b/scripts/oe-build-perf-test
@@ -99,11 +99,13 @@ def parse_args(argv):
     parser.add_argument('-D', '--debug', action='store_true',
                         help='Enable debug level logging')
     parser.add_argument('--globalres-file',
+                        type=os.path.abspath,
                         help="Append results to 'globalres' csv file")
     parser.add_argument('--lock-file', default='./oe-build-perf.lock',
-                        metavar='FILENAME',
+                        metavar='FILENAME', type=os.path.abspath,
                         help="Lock file to use")
     parser.add_argument('-o', '--out-dir', default='results-{date}',
+                        type=os.path.abspath,
                         help="Output directory for test results")
     parser.add_argument('--run-tests', nargs='+', metavar='TEST',
                         help="List of tests to run")
-- 
2.6.6

[PATCH 03/19] oeqa.utils.git: support git commands with updated env

[Markus Lehtonen]

Extend GitRepo.run_cmd so that the caller may redefine and/or define
additional environment variables that will be used when the git command
is run.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/utils/git.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/utils/git.py b/meta/lib/oeqa/utils/git.py
index a4c6741..6a2987f 100644
--- a/meta/lib/oeqa/utils/git.py
+++ b/meta/lib/oeqa/utils/git.py
@@ -30,9 +30,13 @@ class GitRepo(object):
                 cmd_str, ret.status, ret.output))
         return ret.output.strip()
 
-    def run_cmd(self, git_args):
+    def run_cmd(self, git_args, env_update=None):
         """Run Git command"""
-        return self._run_git_cmd_at(git_args, self.top_dir)
+        env = None
+        if env_update:
+            env = os.environ.copy()
+            env.update(env_update)
+        return self._run_git_cmd_at(git_args, self.top_dir, env=env)
 
 
 
-- 
2.6.6

[PATCH 04/19] oeqa.utils.git: introduce GitRepo.rev_parse()

[Markus Lehtonen]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py | 2 +-
 meta/lib/oeqa/utils/git.py      | 7 +++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index 30b8e47..6a8d9fe 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -114,7 +114,7 @@ class BuildPerfTestResult(unittest.TextTestResult):
                      "and OE_BUILDPERFTEST_GIT_BRANCH environment variables")
         else:
             if not rev:
-                rev = self.repo.run_cmd(['rev-parse', 'HEAD'])
+                rev = self.repo.rev_parse('HEAD')
             if not branch:
                 try:
                     # Strip 11 chars, i.e. 'refs/heads' from the beginning
diff --git a/meta/lib/oeqa/utils/git.py b/meta/lib/oeqa/utils/git.py
index 6a2987f..6474654 100644
--- a/meta/lib/oeqa/utils/git.py
+++ b/meta/lib/oeqa/utils/git.py
@@ -38,5 +38,12 @@ class GitRepo(object):
             env.update(env_update)
         return self._run_git_cmd_at(git_args, self.top_dir, env=env)
 
+    def rev_parse(self, revision):
+        """Do git rev-parse"""
+        try:
+            return self.run_cmd(['rev-parse', revision])
+        except GitError:
+            # Revision does not exist
+            return None
 
 
-- 
2.6.6

[PATCH 05/19] oeqa.utils.git: implement GitRepo.get_current_branch()

[Markus Lehtonen]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py | 7 ++-----
 meta/lib/oeqa/utils/git.py      | 8 ++++++++
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index 6a8d9fe..adc3da3 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -116,12 +116,9 @@ class BuildPerfTestResult(unittest.TextTestResult):
             if not rev:
                 rev = self.repo.rev_parse('HEAD')
             if not branch:
-                try:
-                    # Strip 11 chars, i.e. 'refs/heads' from the beginning
-                    branch = self.repo.run_cmd(['symbolic-ref', 'HEAD'])[11:]
-                except GitError:
+                branch = self.repo.get_current_branch()
+                if not branch:
                     log.debug('Currently on detached HEAD')
-                    branch = None
         return str(rev), str(branch)
 
     def addSuccess(self, test):
diff --git a/meta/lib/oeqa/utils/git.py b/meta/lib/oeqa/utils/git.py
index 6474654..0fc8112 100644
--- a/meta/lib/oeqa/utils/git.py
+++ b/meta/lib/oeqa/utils/git.py
@@ -46,4 +46,12 @@ class GitRepo(object):
             # Revision does not exist
             return None
 
+    def get_current_branch(self):
+        """Get current branch"""
+        try:
+            # Strip 11 chars, i.e. 'refs/heads' from the beginning
+            return self.run_cmd(['symbolic-ref', 'HEAD'])[11:]
+        except GitError:
+            return None
+
 
-- 
2.6.6

[PATCH 06/19] oeqa.utils.git.GitRepo: new arg to require topdir

[Markus Lehtonen]

Add a new 'is_topdir' argument to the GitRepo init method which
validates that the given path is the top directory of a Git repository.
Without this argument GitRepo also accepts subdirectories of a Git
repository (in which case GitRepo will point to the parent directory
that is the top directory of this repository) which may have undesired
in some cases.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/utils/git.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/utils/git.py b/meta/lib/oeqa/utils/git.py
index 0fc8112..ca84680 100644
--- a/meta/lib/oeqa/utils/git.py
+++ b/meta/lib/oeqa/utils/git.py
@@ -4,6 +4,8 @@
 # Released under the MIT license (see COPYING.MIT)
 #
 """Git repository interactions"""
+import os
+
 from oeqa.utils.commands import runCmd
 
 
@@ -13,9 +15,12 @@ class GitError(Exception):
 
 class GitRepo(object):
     """Class representing a Git repository clone"""
-    def __init__(self, cwd):
+    def __init__(self, path, is_topdir=False):
         self.top_dir = self._run_git_cmd_at(['rev-parse', '--show-toplevel'],
-                                            cwd)
+                                            path)
+        realpath = os.path.realpath(path)
+        if is_topdir and realpath != self.top_dir:
+            raise GitError("{} is not a Git top directory".format(realpath))
 
     @staticmethod
     def _run_git_cmd_at(git_args, cwd, **kwargs):
-- 
2.6.6

[PATCH 07/19] oeqa.buildperf: use term commit instead of revision

[Markus Lehtonen]

This is basically a internal change, at this point. Term 'commit' better
represents the data we actually have. Term 'revision' is more vague and
could be understood to point to a tag object, for example.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index adc3da3..0e77aae 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -96,30 +96,30 @@ class BuildPerfTestResult(unittest.TextTestResult):
             self.repo = GitRepo('.')
         except GitError:
             self.repo = None
-        self.git_revision, self.git_branch = self.get_git_revision()
+        self.git_commit, self.git_branch = self.get_git_revision()
         self.hostname = socket.gethostname()
         self.start_time = self.elapsed_time = None
         self.successes = []
-        log.info("Using Git branch:revision %s:%s", self.git_branch,
-                 self.git_revision)
+        log.info("Using Git branch:commit %s:%s", self.git_branch,
+                 self.git_commit)
 
     def get_git_revision(self):
-        """Get git branch and revision under testing"""
-        rev = os.getenv('OE_BUILDPERFTEST_GIT_REVISION')
+        """Get git branch and commit under testing"""
+        commit = os.getenv('OE_BUILDPERFTEST_GIT_COMMIT')
         branch = os.getenv('OE_BUILDPERFTEST_GIT_BRANCH')
-        if not self.repo and (not rev or not branch):
+        if not self.repo and (not commit or not branch):
             log.info("The current working directory doesn't seem to be a Git "
-                     "repository clone. You can specify branch and revision "
-                     "used in test results with OE_BUILDPERFTEST_GIT_REVISION "
-                     "and OE_BUILDPERFTEST_GIT_BRANCH environment variables")
+                     "repository clone. You can specify branch and commit "
+                     "displayed in test results with OE_BUILDPERFTEST_GIT_BRANCH "
+                     "and OE_BUILDPERFTEST_GIT_COMMIT environment variables")
         else:
-            if not rev:
-                rev = self.repo.rev_parse('HEAD')
+            if not commit:
+                commit = self.repo.rev_parse('HEAD^0')
             if not branch:
                 branch = self.repo.get_current_branch()
                 if not branch:
                     log.debug('Currently on detached HEAD')
-        return str(rev), str(branch)
+        return str(commit), str(branch)
 
     def addSuccess(self, test):
         """Record results from successful tests"""
@@ -165,9 +165,9 @@ class BuildPerfTestResult(unittest.TextTestResult):
                   'test4': ((7, 1), (10, 2))}
 
         if self.repo:
-            git_tag_rev = self.repo.run_cmd(['describe', self.git_revision])
+            git_tag_rev = self.repo.run_cmd(['describe', self.git_commit])
         else:
-            git_tag_rev = self.git_revision
+            git_tag_rev = self.git_commit
 
         values = ['0'] * 12
         for status, (test, msg) in self.all_results():
@@ -183,7 +183,7 @@ class BuildPerfTestResult(unittest.TextTestResult):
         with open(filename, 'a') as fobj:
             fobj.write('{},{}:{},{},'.format(self.hostname,
                                              self.git_branch,
-                                             self.git_revision,
+                                             self.git_commit,
                                              git_tag_rev))
             fobj.write(','.join(values) + '\n')
 
-- 
2.6.6

[PATCH 08/19] oe-build-perf-test: support committing results data to Git

[Markus Lehtonen]

Implement a new command line option '--commit-results' which commits the
test results data into a Git repository. The given path must be an
existing initialized local Git repository.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py | 35 +++++++++++++++++++++++++++++++++++
 scripts/oe-build-perf-test      |  5 +++++
 2 files changed, 40 insertions(+)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index 0e77aae..8f7d88c 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -188,6 +188,41 @@ class BuildPerfTestResult(unittest.TextTestResult):
             fobj.write(','.join(values) + '\n')
 
 
+    def git_commit_results(self, repo_path, branch=None):
+        """Commit results into a Git repository"""
+        repo = GitRepo(repo_path, is_topdir=True)
+        if not branch:
+            branch = self.git_branch
+        log.info("Committing test results into %s %s", repo_path, branch)
+        tmp_index = os.path.join(repo_path, '.git', 'index.oe-build-perf')
+        try:
+            # Create new commit object from the new results
+            env_update = {'GIT_INDEX_FILE': tmp_index,
+                          'GIT_WORK_TREE': self.out_dir}
+            repo.run_cmd('add .', env_update)
+            tree = repo.run_cmd('write-tree', env_update)
+            parent = repo.rev_parse(branch)
+            msg = "Results of {}:{}\n".format(self.git_branch, self.git_commit)
+            git_cmd = ['commit-tree', tree, '-m', msg]
+            if parent:
+                git_cmd += ['-p', parent]
+            commit = repo.run_cmd(git_cmd, env_update)
+
+            # Update branch head
+            git_cmd = ['update-ref', 'refs/heads/' + branch, commit]
+            if parent:
+                git_cmd.append(parent)
+            repo.run_cmd(git_cmd)
+
+            # Update current HEAD, if we're on branch 'branch'
+            if repo.get_current_branch() == branch:
+                log.info("Updating %s HEAD to latest commit", repo_path)
+                repo.run_cmd('reset --hard')
+        finally:
+            if os.path.exists(tmp_index):
+                os.unlink(tmp_index)
+
+
 class BuildPerfTestCase(unittest.TestCase):
     """Base class for build performance tests"""
     SYSRES = 'sysres'
diff --git a/scripts/oe-build-perf-test b/scripts/oe-build-perf-test
index 21759c6..d6ea5ce 100755
--- a/scripts/oe-build-perf-test
+++ b/scripts/oe-build-perf-test
@@ -109,6 +109,9 @@ def parse_args(argv):
                         help="Output directory for test results")
     parser.add_argument('--run-tests', nargs='+', metavar='TEST',
                         help="List of tests to run")
+    parser.add_argument('--commit-results', metavar='GIT_DIR',
+                        type=os.path.abspath,
+                        help="Commit result data to a (local) git repository")
 
     return parser.parse_args(argv)
 
@@ -158,6 +161,8 @@ def main(argv=None):
     if result.wasSuccessful():
         if args.globalres_file:
             result.update_globalres_file(args.globalres_file)
+        if args.commit_results:
+            result.git_commit_results(args.commit_results)
         return 0
 
     return 1
-- 
2.6.6

[PATCH 09/19] oe-build-perf-test: implement --commit-results-branch

[Markus Lehtonen]

A new command line option for defining the branch where results are
commited. The value is actually a format string accepting two field
names:
- {git_branch} expands to the name of the target branch being tested
- {tester_host} expands to the hostname of the tester machine

The option has no effect if --commit-results is not used.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py | 5 +++++
 scripts/oe-build-perf-test      | 6 +++++-
 2 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index 8f7d88c..119e6ed 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -193,6 +193,11 @@ class BuildPerfTestResult(unittest.TextTestResult):
         repo = GitRepo(repo_path, is_topdir=True)
         if not branch:
             branch = self.git_branch
+        else:
+            # Replace keywords
+            branch = branch.format(git_branch=self.git_branch,
+                                   tester_host=self.hostname)
+
         log.info("Committing test results into %s %s", repo_path, branch)
         tmp_index = os.path.join(repo_path, '.git', 'index.oe-build-perf')
         try:
diff --git a/scripts/oe-build-perf-test b/scripts/oe-build-perf-test
index d6ea5ce..390e4c9 100755
--- a/scripts/oe-build-perf-test
+++ b/scripts/oe-build-perf-test
@@ -112,6 +112,9 @@ def parse_args(argv):
     parser.add_argument('--commit-results', metavar='GIT_DIR',
                         type=os.path.abspath,
                         help="Commit result data to a (local) git repository")
+    parser.add_argument('--commit-results-branch', metavar='BRANCH',
+                        default="{git_branch}",
+                        help="Commit results to branch BRANCH.")
 
     return parser.parse_args(argv)
 
@@ -162,7 +165,8 @@ def main(argv=None):
         if args.globalres_file:
             result.update_globalres_file(args.globalres_file)
         if args.commit_results:
-            result.git_commit_results(args.commit_results)
+            result.git_commit_results(args.commit_results,
+                                      args.commit_results_branch)
         return 0
 
     return 1
-- 
2.6.6

[PATCH 10/19] oeqa.utils.git: implement init() method

[Markus Lehtonen]

Method for doing 'git init'.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/utils/git.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/lib/oeqa/utils/git.py b/meta/lib/oeqa/utils/git.py
index ca84680..ae85d27 100644
--- a/meta/lib/oeqa/utils/git.py
+++ b/meta/lib/oeqa/utils/git.py
@@ -35,6 +35,12 @@ class GitRepo(object):
                 cmd_str, ret.status, ret.output))
         return ret.output.strip()
 
+    @staticmethod
+    def init(path):
+        """Initialize a new Git repository"""
+        GitRepo._run_git_cmd_at('init', cwd=path)
+        return GitRepo(path, is_topdir=True)
+
     def run_cmd(self, git_args, env_update=None):
         """Run Git command"""
         env = None
-- 
2.6.6

[PATCH 11/19] oe-build-perf-test: pre-check Git repo when using --commit-results

[Markus Lehtonen]

Do a pre-check on the path that is specified with --commit-results
before running any tests. The script will create and/or initialize a
fresh Git repository if the given directory does not exist or if it is
an empty directory. It fails if it finds a non-empty directory that is
not a Git repository.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/oe-build-perf-test | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/scripts/oe-build-perf-test b/scripts/oe-build-perf-test
index 390e4c9..437611c 100755
--- a/scripts/oe-build-perf-test
+++ b/scripts/oe-build-perf-test
@@ -31,6 +31,7 @@ import oeqa.buildperf
 from oeqa.buildperf import (BuildPerfTestLoader, BuildPerfTestResult,
                             BuildPerfTestRunner, KernelDropCaches)
 from oeqa.utils.commands import runCmd
+from oeqa.utils.git import GitRepo, GitError
 
 
 # Set-up logging
@@ -68,7 +69,30 @@ def pre_run_sanity_check():
     if ret.status:
         log.error("bitbake command not found")
         return False
+    return True
 
+def init_git_repo(path):
+    """Check/create Git repository where to store results"""
+    path = os.path.abspath(path)
+    if os.path.isfile(path):
+        log.error("Invalid Git repo %s: path exists but is not a directory", path)
+        return False
+    if not os.path.isdir(path):
+        try:
+            os.mkdir(path)
+        except (FileNotFoundError, PermissionError) as err:
+            log.error("Failed to mkdir %s: %s", path, err)
+            return False
+    if not os.listdir(path):
+        log.info("Initializing a new Git repo at %s", path)
+        GitRepo.init(path)
+    try:
+        GitRepo(path, is_topdir=True)
+    except GitError:
+        log.error("No Git repository but a non-empty directory found at %s.\n"
+                  "Please specify a Git repository, an empty directory or "
+                  "a non-existing directory", path)
+        return False
     return True
 
 
@@ -137,6 +161,9 @@ def main(argv=None):
 
     if not pre_run_sanity_check():
         return 1
+    if args.commit_results:
+        if not init_git_repo(args.commit_results):
+            return 1
 
     # Check our capability to drop caches and ask pass if needed
     KernelDropCaches.check()
-- 
2.6.6

[PATCH 12/19] oe-build-perf-test: tag results committed to Git

[Markus Lehtonen]

Create a Git tag when committing results to a Git repository. This patch
also implements --commit-results-tag command line option for controlling
the tag name. The value
is a format string where the following fields may be used:
- {git_branch} - target branch being tested
- {git_commit} - target commit being tested
- {tester_host} - hostname of the tester machine

Tagging can be disabled by giving an empty string to
--commit-results-tag. The option has no effect if --commit-results is
not defined.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py | 11 ++++++++++-
 scripts/oe-build-perf-test      |  6 +++++-
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index 119e6ed..a3cd3f3 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -188,7 +188,7 @@ class BuildPerfTestResult(unittest.TextTestResult):
             fobj.write(','.join(values) + '\n')
 
 
-    def git_commit_results(self, repo_path, branch=None):
+    def git_commit_results(self, repo_path, branch=None, tag=None):
         """Commit results into a Git repository"""
         repo = GitRepo(repo_path, is_topdir=True)
         if not branch:
@@ -223,6 +223,15 @@ class BuildPerfTestResult(unittest.TextTestResult):
             if repo.get_current_branch() == branch:
                 log.info("Updating %s HEAD to latest commit", repo_path)
                 repo.run_cmd('reset --hard')
+
+            # Create (annotated) tag
+            if tag:
+                # Replace keywords
+                tag = tag.format(git_branch=self.git_branch,
+                                 git_commit=self.git_commit,
+                                 tester_host=self.hostname)
+                repo.run_cmd(['tag', '-a', '-m', msg, tag, commit])
+
         finally:
             if os.path.exists(tmp_index):
                 os.unlink(tmp_index)
diff --git a/scripts/oe-build-perf-test b/scripts/oe-build-perf-test
index 437611c..1ed5bdb 100755
--- a/scripts/oe-build-perf-test
+++ b/scripts/oe-build-perf-test
@@ -139,6 +139,9 @@ def parse_args(argv):
     parser.add_argument('--commit-results-branch', metavar='BRANCH',
                         default="{git_branch}",
                         help="Commit results to branch BRANCH.")
+    parser.add_argument('--commit-results-tag', metavar='TAG',
+                        default="{git_branch}/{git_commit}",
+                        help="Tag results commit with TAG.")
 
     return parser.parse_args(argv)
 
@@ -193,7 +196,8 @@ def main(argv=None):
             result.update_globalres_file(args.globalres_file)
         if args.commit_results:
             result.git_commit_results(args.commit_results,
-                                      args.commit_results_branch)
+                                      args.commit_results_branch,
+                                      args.commit_results_tag)
         return 0
 
     return 1
-- 
2.6.6

[PATCH 13/19] oe-build-perf-test: new {tag_num} keyword for --commit-results-tag

[Markus Lehtonen]

This makes it possible to create numbered tags, where the "basename" of
the tag is the same and the only difference is an (automatically)
increasing index number. This is useful if you do multiple test runs on
the same commit. For example, using:
--commit-results-tag {tester_host}/{git_commit}/{tag_num}

would give you tags something like:
myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/0
myhost/decb3119dffd3fd38b800bebc1e510f9217a152e/1
...

The default tag format is updated to use this new keyword in order to
prevent unintentional tag name clashes.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py | 19 +++++++++++++++----
 scripts/oe-build-perf-test      |  2 +-
 2 files changed, 16 insertions(+), 5 deletions(-)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index a3cd3f3..faa30c7 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -226,10 +226,21 @@ class BuildPerfTestResult(unittest.TextTestResult):
 
             # Create (annotated) tag
             if tag:
-                # Replace keywords
-                tag = tag.format(git_branch=self.git_branch,
-                                 git_commit=self.git_commit,
-                                 tester_host=self.hostname)
+                # Find tags matching the pattern
+                tag_keywords = dict(git_branch=self.git_branch,
+                                    git_commit=self.git_commit,
+                                    tester_host=self.hostname,
+                                    tag_num='[0-9]{1,5}')
+                tag_re = re.compile(tag.format(**tag_keywords) + '$')
+                tag_keywords['tag_num'] = 0
+                for existing_tag in repo.run_cmd('tag').splitlines():
+                    if tag_re.match(existing_tag):
+                        tag_keywords['tag_num'] += 1
+
+                tag = tag.format(**tag_keywords)
+                msg = "Test run #{} of {}:{}\n".format(tag_keywords['tag_num'],
+                                                       self.git_branch,
+                                                       self.git_commit)
                 repo.run_cmd(['tag', '-a', '-m', msg, tag, commit])
 
         finally:
diff --git a/scripts/oe-build-perf-test b/scripts/oe-build-perf-test
index 1ed5bdb..cd27584 100755
--- a/scripts/oe-build-perf-test
+++ b/scripts/oe-build-perf-test
@@ -140,7 +140,7 @@ def parse_args(argv):
                         default="{git_branch}",
                         help="Commit results to branch BRANCH.")
     parser.add_argument('--commit-results-tag', metavar='TAG',
-                        default="{git_branch}/{git_commit}",
+                        default="{git_branch}/{git_commit}/{tag_num}",
                         help="Tag results commit with TAG.")
 
     return parser.parse_args(argv)
-- 
2.6.6

[PATCH 14/19] oeqa.buildperf: add git commit count to result data

[Markus Lehtonen]

This number represents the number of commits since the beginning of git
history until the tested revision. This helps e.g. in ordering results.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index faa30c7..f29c167 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -96,30 +96,34 @@ class BuildPerfTestResult(unittest.TextTestResult):
             self.repo = GitRepo('.')
         except GitError:
             self.repo = None
-        self.git_commit, self.git_branch = self.get_git_revision()
+        self.git_commit, self.git_commit_count, self.git_branch = \
+                self.get_git_revision()
         self.hostname = socket.gethostname()
         self.start_time = self.elapsed_time = None
         self.successes = []
-        log.info("Using Git branch:commit %s:%s", self.git_branch,
-                 self.git_commit)
+        log.info("Using Git branch:commit %s:%s (%s)", self.git_branch,
+                 self.git_commit, self.git_commit_count)
 
     def get_git_revision(self):
         """Get git branch and commit under testing"""
         commit = os.getenv('OE_BUILDPERFTEST_GIT_COMMIT')
+        commit_cnt = os.getenv('OE_BUILDPERFTEST_GIT_COMMIT_COUNT')
         branch = os.getenv('OE_BUILDPERFTEST_GIT_BRANCH')
-        if not self.repo and (not commit or not branch):
+        if not self.repo and (not commit or not commit_cnt or not branch):
             log.info("The current working directory doesn't seem to be a Git "
                      "repository clone. You can specify branch and commit "
-                     "displayed in test results with OE_BUILDPERFTEST_GIT_BRANCH "
-                     "and OE_BUILDPERFTEST_GIT_COMMIT environment variables")
+                     "displayed in test results with OE_BUILDPERFTEST_GIT_BRANCH, "
+                     "OE_BUILDPERFTEST_GIT_COMMIT and "
+                     "OE_BUILDPERFTEST_GIT_COMMIT_COUNT environment variables")
         else:
             if not commit:
                 commit = self.repo.rev_parse('HEAD^0')
+                commit_cnt = self.repo.run_cmd(['rev-list', '--count', 'HEAD^0'])
             if not branch:
                 branch = self.repo.get_current_branch()
                 if not branch:
                     log.debug('Currently on detached HEAD')
-        return str(commit), str(branch)
+        return str(commit), str(commit_cnt), str(branch)
 
     def addSuccess(self, test):
         """Record results from successful tests"""
-- 
2.6.6

[PATCH 15/19] oe-build-perf-test: add {git_commit_count} keyword for --commit-results-tag

[Markus Lehtonen]

Makes it possible to create easily sortable tags. Also, the default tag
format is updated to use the new keyword.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 meta/lib/oeqa/buildperf/base.py | 1 +
 scripts/oe-build-perf-test      | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index f29c167..97be58f 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -233,6 +233,7 @@ class BuildPerfTestResult(unittest.TextTestResult):
                 # Find tags matching the pattern
                 tag_keywords = dict(git_branch=self.git_branch,
                                     git_commit=self.git_commit,
+                                    git_commit_count=self.git_commit_count,
                                     tester_host=self.hostname,
                                     tag_num='[0-9]{1,5}')
                 tag_re = re.compile(tag.format(**tag_keywords) + '$')
diff --git a/scripts/oe-build-perf-test b/scripts/oe-build-perf-test
index cd27584..35a4839 100755
--- a/scripts/oe-build-perf-test
+++ b/scripts/oe-build-perf-test
@@ -140,7 +140,7 @@ def parse_args(argv):
                         default="{git_branch}",
                         help="Commit results to branch BRANCH.")
     parser.add_argument('--commit-results-tag', metavar='TAG',
-                        default="{git_branch}/{git_commit}/{tag_num}",
+                        default="{git_branch}/{git_commit_count}-g{git_commit}/{tag_num}",
                         help="Tag results commit with TAG.")
 
     return parser.parse_args(argv)
-- 
2.6.6

[PATCH 16/19] build-perf-test-wrapper.sh: parse args with getopts

[Markus Lehtonen]

Use getopts for parsing the command line. This changes the usage so that
if a commit (to-be-tested) is defined it must be given by using '-c',
instead of a positional argument.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/contrib/build-perf-test-wrapper.sh | 30 +++++++++++++++++++++++-------
 1 file changed, 23 insertions(+), 7 deletions(-)

diff --git a/scripts/contrib/build-perf-test-wrapper.sh b/scripts/contrib/build-perf-test-wrapper.sh
index e8e8021..8eb4fdb 100755
--- a/scripts/contrib/build-perf-test-wrapper.sh
+++ b/scripts/contrib/build-perf-test-wrapper.sh
@@ -20,17 +20,33 @@
 
 script=`basename $0`
 usage () {
-    echo "Usage: $script [COMMITISH]"
+cat << EOF
+Usage: $script [-h] [-c COMMITISH] [-C GIT_REPO]
+
+Optional arguments:
+  -h                show this help and exit.
+  -c COMMITISH      test (checkout) this commit
+EOF
 }
 
-if [ $# -gt 1 ]; then
-    usage
-    exit 1
-fi
-commitish=$1
 
-echo "Running on `uname -n`"
+# Parse command line arguments
+commitish=""
+while getopts "hc:" opt; do
+    case $opt in
+        h)  usage
+            exit 0
+            ;;
+        c)  commitish=$OPTARG
+            ;;
+        *)  usage
+            exit 1
+            ;;
+    esac
+done
+
 
+echo "Running on `uname -n`"
 if ! git_topdir=$(git rev-parse --show-toplevel); then
         echo "The current working dir doesn't seem to be a git clone. Please cd there before running `basename $0`"
         exit 1
-- 
2.6.6

[PATCH 17/19] build-perf-test-wrapper.sh: allow saving results in Git

[Markus Lehtonen]

Add new command line argument '-C' that allows saving results in a Git
repository.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/contrib/build-perf-test-wrapper.sh | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/scripts/contrib/build-perf-test-wrapper.sh b/scripts/contrib/build-perf-test-wrapper.sh
index 8eb4fdb..2fb4b11 100755
--- a/scripts/contrib/build-perf-test-wrapper.sh
+++ b/scripts/contrib/build-perf-test-wrapper.sh
@@ -26,19 +26,23 @@ Usage: $script [-h] [-c COMMITISH] [-C GIT_REPO]
 Optional arguments:
   -h                show this help and exit.
   -c COMMITISH      test (checkout) this commit
+  -C GIT_REPO       commit results into Git
 EOF
 }
 
 
 # Parse command line arguments
 commitish=""
-while getopts "hc:" opt; do
+results_repo=""
+while getopts "hc:C:" opt; do
     case $opt in
         h)  usage
             exit 0
             ;;
         c)  commitish=$OPTARG
             ;;
+        C)  results_repo=`realpath "$OPTARG"`
+            ;;
         *)  usage
             exit 1
             ;;
@@ -73,13 +77,14 @@ base_dir="$git_topdir/build-perf-test"
 build_dir="$base_dir/build-$git_rev-$timestamp"
 results_dir="$base_dir/results-$git_rev-$timestamp"
 globalres_log="$base_dir/globalres.log"
+machine="qemux86"
 
 mkdir -p "$base_dir"
 source ./oe-init-build-env $build_dir >/dev/null || exit 1
 
 # Additional config
 auto_conf="$build_dir/conf/auto.conf"
-echo 'MACHINE = "qemux86"' > "$auto_conf"
+echo "MACHINE = \"$machine\"" > "$auto_conf"
 echo 'BB_NUMBER_THREADS = "8"' >> "$auto_conf"
 echo 'PARALLEL_MAKE = "-j 8"' >> "$auto_conf"
 echo "DL_DIR = \"$base_dir/downloads\"" >> "$auto_conf"
@@ -93,7 +98,10 @@ fi
 # Run actual test script
 if ! oe-build-perf-test --out-dir "$results_dir" \
                         --globalres-file "$globalres_log" \
-                        --lock-file "$base_dir/oe-build-perf.lock"; then
+                        --lock-file "$base_dir/oe-build-perf.lock" \
+                        --commit-results "$results_repo" \
+                        --commit-results-branch "{tester_host}/{git_branch}/$machine" \
+                        --commit-results-tag "{tester_host}/{git_branch}/$machine/{git_commit_count}-g{git_commit}/{tag_num}"; then
     echo "oe-build-perf-test script failed!"
     exit 1
 fi
-- 
2.6.6

[PATCH 18/19] build-perf-test-wrapper.sh: make archive dir configurable

[Markus Lehtonen]

Add new command line argument '-a' that can be used to define the
directory where results (tarballs) are archived. Giving an empty string
disables archiving which makes sense if you store results in Git.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/contrib/build-perf-test-wrapper.sh | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/scripts/contrib/build-perf-test-wrapper.sh b/scripts/contrib/build-perf-test-wrapper.sh
index 2fb4b11..ef2a46e 100755
--- a/scripts/contrib/build-perf-test-wrapper.sh
+++ b/scripts/contrib/build-perf-test-wrapper.sh
@@ -25,6 +25,8 @@ Usage: $script [-h] [-c COMMITISH] [-C GIT_REPO]
 
 Optional arguments:
   -h                show this help and exit.
+  -a ARCHIVE_DIR    archive results tarball here, give an empty string to
+                    disable tarball archiving
   -c COMMITISH      test (checkout) this commit
   -C GIT_REPO       commit results into Git
 EOF
@@ -32,13 +34,16 @@ EOF
 
 
 # Parse command line arguments
+archive_dir=~/perf-results/archives
 commitish=""
 results_repo=""
-while getopts "hc:C:" opt; do
+while getopts "ha:c:C:" opt; do
     case $opt in
         h)  usage
             exit 0
             ;;
+        a)  archive_dir=`realpath "$OPTARG"`
+            ;;
         c)  commitish=$OPTARG
             ;;
         C)  results_repo=`realpath "$OPTARG"`
@@ -112,13 +117,14 @@ echo -ne "\n"
 
 cat "$globalres_log"
 
-echo -ne "\n\n-----------------\n"
-echo "Archiving results dir..."
-archive_dir=~/perf-results/archives
-mkdir -p "$archive_dir"
-results_basename=`basename "$results_dir"`
-results_dirname=`dirname "$results_dir"`
-tar -czf "$archive_dir/`uname -n`-${results_basename}.tar.gz" -C "$results_dirname" "$results_basename"
+if [ -n "$archive_dir" ]; then
+    echo -ne "\n\n-----------------\n"
+    echo "Archiving results in $archive_dir"
+    mkdir -p "$archive_dir"
+    results_basename=`basename "$results_dir"`
+    results_dirname=`dirname "$results_dir"`
+    tar -czf "$archive_dir/`uname -n`-${results_basename}.tar.gz" -C "$results_dirname" "$results_basename"
+fi
 
 rm -rf "$build_dir"
 rm -rf "$results_dir"
-- 
2.6.6

[PATCH 19/19] build-perf-test-wrapper.sh: make workdir configurable

[Markus Lehtonen]

New command line argument '-w' may be used to specify work dir other
than the default <GIT_DIR>/build-perf-test.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/contrib/build-perf-test-wrapper.sh | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/scripts/contrib/build-perf-test-wrapper.sh b/scripts/contrib/build-perf-test-wrapper.sh
index ef2a46e..efc2b79 100755
--- a/scripts/contrib/build-perf-test-wrapper.sh
+++ b/scripts/contrib/build-perf-test-wrapper.sh
@@ -29,6 +29,7 @@ Optional arguments:
                     disable tarball archiving
   -c COMMITISH      test (checkout) this commit
   -C GIT_REPO       commit results into Git
+  -w WORK_DIR       work dir for this script
 EOF
 }
 
@@ -37,7 +38,7 @@ EOF
 archive_dir=~/perf-results/archives
 commitish=""
 results_repo=""
-while getopts "ha:c:C:" opt; do
+while getopts "ha:c:C:w:" opt; do
     case $opt in
         h)  usage
             exit 0
@@ -48,13 +49,14 @@ while getopts "ha:c:C:" opt; do
             ;;
         C)  results_repo=`realpath "$OPTARG"`
             ;;
+        w)  base_dir=`realpath "$OPTARG"`
+            ;;
         *)  usage
             exit 1
             ;;
     esac
 done
 
-
 echo "Running on `uname -n`"
 if ! git_topdir=$(git rev-parse --show-toplevel); then
         echo "The current working dir doesn't seem to be a git clone. Please cd there before running `basename $0`"
@@ -76,9 +78,13 @@ if [ -n "$commitish" ]; then
 fi
 
 # Setup build environment
+if [ -z "$base_dir" ]; then
+    base_dir="$git_topdir/build-perf-test"
+fi
+echo "Using working dir $base_dir"
+
 timestamp=`date "+%Y%m%d%H%M%S"`
 git_rev=$(git rev-parse --short HEAD)  || exit 1
-base_dir="$git_topdir/build-perf-test"
 build_dir="$base_dir/build-$git_rev-$timestamp"
 results_dir="$base_dir/results-$git_rev-$timestamp"
 globalres_log="$base_dir/globalres.log"
-- 
2.6.6