[PATCHv2 0/3] Add communication mechanism for sending test results

[PATCHv2 0/3] Add communication mechanism for sending test results

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

These changes add a mechanism for sending test result to a remote
repository for later consumption of the data from other clients.

These patches were tested in our local autobuilders.

Changes in v2:

- Give more priority to local branches, because it would be difficult
  to match a branch name (because it includes the hostname).
- Instead of throwing an execption when a push fails, log the error
  and return 1 to the caller.
- Instead to call 3 times bitbake -e, just call it one time to get
  needed variables.

The following changes since commit 9e63f81c78e284c9b325fe04a1b59e61c7ad8a1a:

  bitbake: ast: remove BBVERSIONS support (2016-11-30 15:48:10 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib mariano/bug9954v2
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug9954v2

Mariano Lopez (3):
  oeqa/utils/commands.py: Make a copy of variables in get_bb_vars
  oeqa/utils/metadata.py: Add metadata library
  oe-selftest: Add option to submit test result to a git repository.

 meta/lib/oeqa/utils/commands.py |   1 +
 meta/lib/oeqa/utils/metadata.py |  83 ++++++++++++++++++++++++++++++++
 scripts/oe-selftest             | 102 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 186 insertions(+)
 create mode 100644 meta/lib/oeqa/utils/metadata.py

-- 
2.7.3

[PATCHv2 1/3] oeqa/utils/commands.py: Make a copy of variables in get_bb_vars

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

The function get_bb_vars will remove items for the list passed
as the function argument, this will leave the caller with an
empty list and the function never says it will consume the items.

This hasn't been found before because only get_bb_var uses this
function.

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/lib/oeqa/utils/commands.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/lib/oeqa/utils/commands.py b/meta/lib/oeqa/utils/commands.py
index 5cd0f74..e00c879 100644
--- a/meta/lib/oeqa/utils/commands.py
+++ b/meta/lib/oeqa/utils/commands.py
@@ -149,6 +149,7 @@ def get_bb_vars(variables=None, target=None, postconfig=None):
     """Get values of multiple bitbake variables"""
     bbenv = get_bb_env(target, postconfig=postconfig)
 
+    variables = variables.copy()
     var_re = re.compile(r'^(export )?(?P<var>\w+)="(?P<value>.*)"$')
     unset_re = re.compile(r'^unset (?P<var>\w+)$')
     lastline = None
-- 
2.7.3

[PATCHv2 2/3] oeqa/utils/metadata.py: Add metadata library

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

Adds functions to get metadata from the host running the tests.

[YOCTO #9954]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/lib/oeqa/utils/metadata.py | 83 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
 create mode 100644 meta/lib/oeqa/utils/metadata.py

diff --git a/meta/lib/oeqa/utils/metadata.py b/meta/lib/oeqa/utils/metadata.py
new file mode 100644
index 0000000..ecbe763
--- /dev/null
+++ b/meta/lib/oeqa/utils/metadata.py
@@ -0,0 +1,83 @@
+# Copyright (C) 2016 Intel Corporation
+#
+# Released under the MIT license (see COPYING.MIT)
+#
+# Functions to get metadata from the testing host used
+# for analytics of test results.
+
+from git import Repo, InvalidGitRepositoryError, NoSuchPathError
+from collections import OrderedDict
+from collections.abc import MutableMapping
+from xml.dom.minidom import parseString
+from xml.etree.ElementTree import Element, tostring
+
+from oe.lsb import distro_identifier
+from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars
+
+metadata_vars = ['MACHINE', 'DISTRO', 'DISTRO_VERSION']
+
+def metadata_from_bb():
+    """ Returns test's metadata as OrderedDict.
+
+        Data will be gathered using bitbake -e thanks to get_bb_vars.
+    """
+
+    info_dict = OrderedDict()
+    hostname = runCmd('hostname')
+    info_dict['hostname'] = hostname.output
+    data_dict = get_bb_vars(metadata_vars)
+    for var in metadata_vars:
+        info_dict[var.lower()] = data_dict[var]
+    host_distro= distro_identifier()
+    host_distro, _, host_distro_release = host_distro.partition('-')
+    info_dict['host_distro'] = host_distro
+    info_dict['host_distro_release'] = host_distro_release
+    info_dict['layers'] = get_layers(get_bb_var('BBLAYERS'))
+    return info_dict
+
+def metadata_from_data_store(d):
+    """ Returns test's metadata as OrderedDict.
+
+        Data will be collected from the provided data store.
+    """
+    # TODO: Getting metadata from the data store would
+    # be useful when running within bitbake.
+    pass
+
+def get_layers(layers):
+    """ Returns layer name, branch, and revision as OrderedDict. """
+
+    layer_dict = OrderedDict()
+    for layer in layers.split():
+        layer_name = os.path.basename(layer)
+        layer_dict[layer_name] = OrderedDict()
+        try:
+            repo = Repo(layer, search_parent_directories=True)
+            revision, branch = repo.head.object.name_rev.split()
+            layer_dict[layer_name]['branch'] = branch
+            layer_dict[layer_name]['revision'] = revision
+        except (InvalidGitRepositoryError, NoSuchPathError):
+            layer_dict[layer_name]['branch'] = 'unknown'
+            layer_dict[layer_name]['revision'] = 'unknown'
+    return layer_dict
+
+def write_metadata_file(file_path, metadata):
+    """ Writes metadata to a XML file in directory. """
+
+    xml = dict_to_XML('metadata', metadata)
+    xml_doc = parseString(tostring(xml).decode('UTF-8'))
+    with open(file_path, 'w') as f:
+        f.write(xml_doc.toprettyxml())
+
+def dict_to_XML(tag, dictionary):
+    """ Return XML element converting dicts recursively. """
+
+    elem = Element(tag)
+    for key, val in dictionary.items():
+        if isinstance(val, MutableMapping):
+            child = (dict_to_XML(key, val))
+        else:
+            child = Element(key)
+            child.text = str(val)
+        elem.append(child)
+    return elem
-- 
2.7.3

Re: [PATCHv2 2/3] oeqa/utils/metadata.py: Add metadata library

[Markus Lehtonen]

On Thu, 2016-12-01 at 09:37 -0600, mariano.lopez@linux.intel.com wrote:
> From: Mariano Lopez <mariano.lopez@linux.intel.com>
> 
> Adds functions to get metadata from the host running the tests.
> 
> [YOCTO #9954]
> 
> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
> ---
>  meta/lib/oeqa/utils/metadata.py | 83
> +++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 83 insertions(+)
>  create mode 100644 meta/lib/oeqa/utils/metadata.py
> 
> diff --git a/meta/lib/oeqa/utils/metadata.py
> b/meta/lib/oeqa/utils/metadata.py
> new file mode 100644
> index 0000000..ecbe763
> --- /dev/null
> +++ b/meta/lib/oeqa/utils/metadata.py
> @@ -0,0 +1,83 @@
> +# Copyright (C) 2016 Intel Corporation
> +#
> +# Released under the MIT license (see COPYING.MIT)
> +#
> +# Functions to get metadata from the testing host used
> +# for analytics of test results.
> +
> +from git import Repo, InvalidGitRepositoryError, NoSuchPathError

We introduce a dependency on GitPython(?) If we do this, it should at least
be mentioned somewhere as a new required dependency. There is also
oeqa.utils.git which could be used (and extended).



> +from collections import OrderedDict
> +from collections.abc import MutableMapping
> +from xml.dom.minidom import parseString
> +from xml.etree.ElementTree import Element, tostring
> +
> +from oe.lsb import distro_identifier
> +from oeqa.utils.commands import runCmd, get_bb_var, get_bb_vars
> +
> +metadata_vars = ['MACHINE', 'DISTRO', 'DISTRO_VERSION']
> +
> +def metadata_from_bb():
> +    """ Returns test's metadata as OrderedDict.
> +
> +        Data will be gathered using bitbake -e thanks to get_bb_vars.
> +    """
> +
> +    info_dict = OrderedDict()
> +    hostname = runCmd('hostname')
> +    info_dict['hostname'] = hostname.output
> +    data_dict = get_bb_vars(metadata_vars)
> +    for var in metadata_vars:
> +        info_dict[var.lower()] = data_dict[var]
> +    host_distro= distro_identifier()
> +    host_distro, _, host_distro_release = host_distro.partition('-')
> +    info_dict['host_distro'] = host_distro
> +    info_dict['host_distro_release'] = host_distro_release
> +    info_dict['layers'] = get_layers(get_bb_var('BBLAYERS'))
> +    return info_dict

I think it would be better to have a distro revision (i.e. the "main
revision" or "revision of the main repository", for example oe-core or
poky) and only list additional layers that are not included in the main
repository.


Thanks,
  Markus

[PATCHv2 3/3] oe-selftest: Add option to submit test result to a git repository.

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This new option allows to commit the result to a git repository,
along with the results it will add a metadata file for information
of the current selftest run, such as: hostname, machine, distro,
distro version, host version, and layers.

This implementation will have a branch per different hostname,
testing branch, and machine.

To use this feature use:

oe-selftest <options> --repository <repository_link>

[YOCTO #9954]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 scripts/oe-selftest | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 102 insertions(+)

diff --git a/scripts/oe-selftest b/scripts/oe-selftest
index deaa432..f4b861f 100755
--- a/scripts/oe-selftest
+++ b/scripts/oe-selftest
@@ -36,6 +36,7 @@ import re
 import fnmatch
 import collections
 import imp
+import git
 
 sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) + '/lib')
 import scriptpath
@@ -46,6 +47,7 @@ import argparse_oe
 import oeqa.selftest
 import oeqa.utils.ftools as ftools
 from oeqa.utils.commands import runCmd, get_bb_var, get_test_layer
+from oeqa.utils.metadata import metadata_from_bb, write_metadata_file
 from oeqa.selftest.base import oeSelfTest, get_available_machines
 
 try:
@@ -106,6 +108,8 @@ def get_args_parser():
                        help='List all tags that have been set to test cases.')
     parser.add_argument('--machine', required=False, dest='machine', choices=['random', 'all'], default=None,
                         help='Run tests on different machines (random/all).')
+    parser.add_argument('--repository', required=False, dest='repository', default='', action='store',
+                        help='Submit test results to a repository')
     return parser
 
 
@@ -572,6 +576,75 @@ def main():
 
         log.info("Finished")
 
+        if args.repository:
+            # Commit tests results to repository
+            metadata = metadata_from_bb()
+            git_dir = os.path.join(os.getcwd(), 'selftest')
+            if not os.path.isdir(git_dir):
+                os.mkdir(git_dir)
+
+            log.debug('Checking for git repository in %s' % git_dir)
+            try:
+                repo = git.Repo(git_dir)
+            except git.exc.InvalidGitRepositoryError:
+                log.debug("Couldn't find git repository %s; "
+                         "cloning from %s" % (git_dir, args.repository))
+                repo = git.Repo.clone_from(args.repository, git_dir)
+
+            r_branches = repo.git.branch(r=True)
+            r_branches = set(r_branches.replace('origin/', '').split())
+            l_branches = {str(branch) for branch in repo.branches}
+            branch = '%s/%s/%s' % (metadata['hostname'],
+                                   metadata['layers']['meta']['branch'],
+                                   metadata['machine'])
+
+            if branch in l_branches:
+                log.debug('Found branch in local repository, checking out')
+                repo.git.checkout(branch)
+            elif branch in r_branches:
+                log.debug('Found branch in remote repository, checking'
+                          ' out and pulling')
+                repo.git.checkout(branch)
+                repo.git.pull()
+            else:
+                log.debug('New branch %s' % branch)
+                repo.git.checkout('master')
+                repo.git.checkout(b=branch)
+
+            cleanResultsDir(repo)
+            xml_dir = os.path.join(os.getcwd(), log_prefix)
+            copyResultFiles(xml_dir, git_dir, repo)
+            metadata_file = os.path.join(git_dir, 'metadata.xml')
+            write_metadata_file(metadata_file, metadata)
+            repo.index.add([metadata_file])
+            repo.index.write()
+
+            # Get information for commit message
+            layer_info = ''
+            for layer, values in metadata['layers'].items():
+                layer_info = '%s%-17s = %s:%s\n' % (layer_info, layer,
+                              values['branch'], values['revision'])
+            msg = 'Selftest for build %s of %s %s for machine %s on %s\n\n%s' % (
+                   log_prefix[12:], metadata['distro'], metadata['distro_version'],
+                   metadata['machine'], metadata['hostname'], layer_info)
+
+            log.debug('Commiting results to local repository')
+            repo.index.commit(msg)
+            if not repo.is_dirty():
+                try:
+                    if branch in r_branches:
+                        log.debug('Pushing changes to remote repository')
+                        repo.git.push()
+                    else:
+                        log.debug('Pushing changes to remote repository '
+                                  'creating new branch')
+                        repo.git.push('-u', 'origin', branch)
+                except GitCommandError:
+                    log.error('Falied to push to remote repository')
+                    return 1
+            else:
+                log.error('Local repository is dirty, not pushing commits')
+
         if result.wasSuccessful():
             return 0
         else:
@@ -655,6 +728,35 @@ def buildResultClass(args):
 
     return StampedResult
 
+def cleanResultsDir(repo):
+    """ Remove result files from directory """
+
+    xml_files = []
+    directory = repo.working_tree_dir
+    for f in os.listdir(directory):
+        path = os.path.join(directory, f)
+        if os.path.isfile(path) and path.endswith('.xml'):
+            xml_files.append(f)
+    repo.index.remove(xml_files, working_tree=True)
+
+def copyResultFiles(src, dst, repo):
+    """ Copy result files from src to dst removing the time stamp. """
+
+    import shutil
+
+    re_time = re.compile("-[0-9]+")
+    file_list = []
+
+    for root, subdirs, files in os.walk(src):
+        tmp_dir = root.replace(src, '').lstrip('/')
+        for s in subdirs:
+            os.mkdir(os.path.join(dst, tmp_dir, s))
+        for f in files:
+            file_name = os.path.join(dst, tmp_dir, re_time.sub("", f))
+            shutil.copy2(os.path.join(root, f), file_name)
+            file_list.append(file_name)
+    repo.index.add(file_list)
+
 class TestRunner(_TestRunner):
     """Test runner class aware of exporting tests."""
     def __init__(self, *args, **kwargs):
-- 
2.7.3

Re: [PATCHv2 3/3] oe-selftest: Add option to submit test result to a git repository.

[Benjamin Esquivel]

Hi Mariano

On Thu, 2016-12-01 at 09:37 -0600, mariano.lopez@linux.intel.com wrote:
> From: Mariano Lopez <mariano.lopez@linux.intel.com>
> 
> This new option allows to commit the result to a git repository,
> along with the results it will add a metadata file for information
> of the current selftest run, such as: hostname, machine, distro,
> distro version, host version, and layers.
> 
> This implementation will have a branch per different hostname,
> testing branch, and machine.
> 
> To use this feature use:
> 
> oe-selftest <options> --repository <repository_link>
> 
> [YOCTO #9954]
I think this version is okay :) have you tested it in an Autobuilder
task? 
> 
> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
> ---
>  scripts/oe-selftest | 102
> ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 102 insertions(+)
> 
> diff --git a/scripts/oe-selftest b/scripts/oe-selftest
> index deaa432..f4b861f 100755
> --- a/scripts/oe-selftest
> +++ b/scripts/oe-selftest
> @@ -36,6 +36,7 @@ import re
>  import fnmatch
>  import collections
>  import imp
> +import git
>  
>  sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)) +
> '/lib')
>  import scriptpath
> @@ -46,6 +47,7 @@ import argparse_oe
>  import oeqa.selftest
>  import oeqa.utils.ftools as ftools
>  from oeqa.utils.commands import runCmd, get_bb_var, get_test_layer
> +from oeqa.utils.metadata import metadata_from_bb,
> write_metadata_file
>  from oeqa.selftest.base import oeSelfTest, get_available_machines
>  
>  try:
> @@ -106,6 +108,8 @@ def get_args_parser():
>                         help='List all tags that have been set to
> test cases.')
>      parser.add_argument('--machine', required=False, dest='machine',
> choices=['random', 'all'], default=None,
>                          help='Run tests on different machines
> (random/all).')
> +    parser.add_argument('--repository', required=False,
> dest='repository', default='', action='store',
> +                        help='Submit test results to a repository')
>      return parser
>  
>  
> @@ -572,6 +576,75 @@ def main():
>  
>          log.info("Finished")
>  
> +        if args.repository:
> +            # Commit tests results to repository
> +            metadata = metadata_from_bb()
> +            git_dir = os.path.join(os.getcwd(), 'selftest')
> +            if not os.path.isdir(git_dir):
> +                os.mkdir(git_dir)
> +
> +            log.debug('Checking for git repository in %s' % git_dir)
> +            try:
> +                repo = git.Repo(git_dir)
> +            except git.exc.InvalidGitRepositoryError:
> +                log.debug("Couldn't find git repository %s; "
> +                         "cloning from %s" % (git_dir,
> args.repository))
> +                repo = git.Repo.clone_from(args.repository, git_dir)
> +
> +            r_branches = repo.git.branch(r=True)
> +            r_branches = set(r_branches.replace('origin/',
> '').split())
> +            l_branches = {str(branch) for branch in repo.branches}
> +            branch = '%s/%s/%s' % (metadata['hostname'],
> +                                   metadata['layers']['meta']['branc
> h'],
> +                                   metadata['machine'])
> +
> +            if branch in l_branches:
> +                log.debug('Found branch in local repository,
> checking out')
> +                repo.git.checkout(branch)
> +            elif branch in r_branches:
> +                log.debug('Found branch in remote repository,
> checking'
> +                          ' out and pulling')
> +                repo.git.checkout(branch)
> +                repo.git.pull()
> +            else:
> +                log.debug('New branch %s' % branch)
> +                repo.git.checkout('master')
> +                repo.git.checkout(b=branch)
> +
> +            cleanResultsDir(repo)
> +            xml_dir = os.path.join(os.getcwd(), log_prefix)
> +            copyResultFiles(xml_dir, git_dir, repo)
> +            metadata_file = os.path.join(git_dir, 'metadata.xml')
> +            write_metadata_file(metadata_file, metadata)
> +            repo.index.add([metadata_file])
> +            repo.index.write()
> +
> +            # Get information for commit message
> +            layer_info = ''
> +            for layer, values in metadata['layers'].items():
> +                layer_info = '%s%-17s = %s:%s\n' % (layer_info,
> layer,
> +                              values['branch'], values['revision'])
> +            msg = 'Selftest for build %s of %s %s for machine %s on
> %s\n\n%s' % (
> +                   log_prefix[12:], metadata['distro'],
> metadata['distro_version'],
> +                   metadata['machine'], metadata['hostname'],
> layer_info)
> +
> +            log.debug('Commiting results to local repository')
> +            repo.index.commit(msg)
> +            if not repo.is_dirty():
> +                try:
> +                    if branch in r_branches:
> +                        log.debug('Pushing changes to remote
> repository')
> +                        repo.git.push()
> +                    else:
> +                        log.debug('Pushing changes to remote
> repository '
> +                                  'creating new branch')
> +                        repo.git.push('-u', 'origin', branch)
> +                except GitCommandError:
> +                    log.error('Falied to push to remote repository')
> +                    return 1
> +            else:
> +                log.error('Local repository is dirty, not pushing
> commits')
> +
>          if result.wasSuccessful():
>              return 0
>          else:
> @@ -655,6 +728,35 @@ def buildResultClass(args):
>  
>      return StampedResult
>  
> +def cleanResultsDir(repo):
> +    """ Remove result files from directory """
> +
> +    xml_files = []
> +    directory = repo.working_tree_dir
> +    for f in os.listdir(directory):
> +        path = os.path.join(directory, f)
> +        if os.path.isfile(path) and path.endswith('.xml'):
> +            xml_files.append(f)
> +    repo.index.remove(xml_files, working_tree=True)
> +
> +def copyResultFiles(src, dst, repo):
> +    """ Copy result files from src to dst removing the time stamp.
> """
> +
> +    import shutil
> +
> +    re_time = re.compile("-[0-9]+")
> +    file_list = []
> +
> +    for root, subdirs, files in os.walk(src):
> +        tmp_dir = root.replace(src, '').lstrip('/')
> +        for s in subdirs:
> +            os.mkdir(os.path.join(dst, tmp_dir, s))
> +        for f in files:
> +            file_name = os.path.join(dst, tmp_dir, re_time.sub("",
> f))
> +            shutil.copy2(os.path.join(root, f), file_name)
> +            file_list.append(file_name)
> +    repo.index.add(file_list)
> +
>  class TestRunner(_TestRunner):
>      """Test runner class aware of exporting tests."""
>      def __init__(self, *args, **kwargs):

Re: [PATCHv2 3/3] oe-selftest: Add option to submit test result to a git repository.

[Mariano Lopez]

On Thursday, December 01, 2016 10:52:40 AM Benjamin Esquivel wrote:
> Hi Mariano
> 
> On Thu, 2016-12-01 at 09:37 -0600, mariano.lopez@linux.intel.com wrote:
> > From: Mariano Lopez <mariano.lopez@linux.intel.com>
> > 
> > This new option allows to commit the result to a git repository,
> > along with the results it will add a metadata file for information
> > of the current selftest run, such as: hostname, machine, distro,
> > distro version, host version, and layers.
> > 
> > This implementation will have a branch per different hostname,
> > testing branch, and machine.
> > 
> > To use this feature use:
> > 
> > oe-selftest <options> --repository <repository_link>
> > 
> > [YOCTO #9954]

> I think this version is okay :) have you tested it in an Autobuilder
> task? 

Indeed, that is why it took me a bit longer to submit the patch. I
know this won't blow up current master.  But Ihaven't used the
option yet, we would need a remote repo and a new autobuilder
buildset.

Re: [PATCHv2 3/3] oe-selftest: Add option to submit test result to a git repository.

[Benjamin Esquivel]

On Thu, 2016-12-01 at 11:00 -0600, Mariano Lopez wrote:
> > I think this version is okay :) have you tested it in an
> Autobuilder
> > task? 
> 
> Indeed, that is why it took me a bit longer to submit the patch. I
> know this won't blow up current master.  But Ihaven't used the
> option yet, we would need a remote repo and a new autobuilder
> buildset.
So, for enabling it you still need the repo, right? is that repo
already requested? it would help to understand the timing of this
feature coming online.