[PATCH v2 0/8] devtool fixes

[PATCH v2 0/8] devtool fixes

[Paul Eggleton]

This incorporates several of the fixes from my previous series and adds
a few others - I've excluded the multi-source changes from the previous
series as they may not be ready yet and it's too late in the release
cycle now. These have all passed through oe-selftest locally.

Changes since v1:
* Rename is_path_under() -> is_path_parent(), and change it to accept
  paths to test as unlimited arguments to the function and return False
  if there are no paths specified, as suggested by Ola x Nilsson.



The following changes since commit b30153a15715a83c0f9a7d7d1883a15404992a19:

  buildhistory-diff: honour report_all flag (2018-02-24 10:33:12 +0000)

are available in the Git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool33-oe
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool33-oe

Paul Eggleton (8):
  lib/oe/path: implement is_path_parent()
  devtool: fix poor handling of upgraded BBCLASSEXTENDed recipes
  devtool: reset: delete bbappend file if _check_preserve() doesn't
  devtool: finish: fix erroneously creating bbappend for relative paths
  devtool: deploy-target: don't specify ssh/scp port unless user does
  lib/oe/recipeutils: add .txz extension to archive list
  devtool: search: also look in recipe cache
  devtool: search: tweak help text

 meta/lib/oe/path.py             |  22 +++++++++
 meta/lib/oe/recipeutils.py      |   2 +-
 scripts/devtool                 |   5 +-
 scripts/lib/devtool/deploy.py   |  12 ++---
 scripts/lib/devtool/search.py   | 100 ++++++++++++++++++++++++++--------------
 scripts/lib/devtool/standard.py |  17 +++++--
 6 files changed, 108 insertions(+), 50 deletions(-)

-- 
2.14.3

[PATCH v2 1/8] lib/oe/path: implement is_path_parent()

[Paul Eggleton]

In a few places we have checks to see path B is the parent of path A, by
adding / to the end of the path B and then seeing if path A starts with
the suffixed path B. Unfortunately there are two potential flaws:
(1) path A needs to be suffixed with / as well or the directory itself
won't match (semantics perhaps, but in a lot of scenarios returning True
is correct); (2) you need to run os.path.abspath() on both paths first
or you will wrongly return False for some relative paths where you
should return True. Let's solve this once and for all by writing a
function that takes care of these and put it in oe.path.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/path.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 1ea03d5d568..76c58fa7601 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -237,3 +237,25 @@ def realpath(file, root, use_physdir = True, loop_cnt = 100, assume_dir = False)
         raise
 
     return file
+
+def is_path_parent(possible_parent, *paths):
+    """
+    Return True if a path is the parent of another, False otherwise.
+    Multiple paths to test can be specified in which case all
+    specified test paths must be under the parent in order to
+    return True.
+    """
+    def abs_path_trailing(pth):
+        pth_abs = os.path.abspath(pth)
+        if not pth_abs.endswith(os.sep):
+            pth_abs += os.sep
+        return pth_abs
+
+    possible_parent_abs = abs_path_trailing(possible_parent)
+    if not paths:
+        return False
+    for path in paths:
+        path_abs = abs_path_trailing(path)
+        if not path_abs.startswith(possible_parent_abs):
+            return False
+    return True
-- 
2.14.3

[PATCH v2 2/8] devtool: fix poor handling of upgraded BBCLASSEXTENDed recipes

[Paul Eggleton]

Fix two aspects of handling BBCLASSEXTENDed targets (e.g.
openssl-native) that have been run through "devtool upgrade":

* Fix recipe name not showing up in "devtool status"
* Fix "devtool reset" not deleting empty directories under the recipe
  directory within the workspace, which may lead to problems if you
  subsequently run "devtool upgrade" on the same target again

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/devtool                 | 5 +++--
 scripts/lib/devtool/standard.py | 6 +++++-
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/scripts/devtool b/scripts/devtool
index a651d8f213a..b4bfbb8329c 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -117,12 +117,13 @@ def read_workspace():
             for line in f:
                 res = externalsrc_re.match(line.rstrip())
                 if res:
-                    pn = res.group(2) or os.path.splitext(os.path.basename(fn))[0].split('_')[0]
+                    recipepn = os.path.splitext(os.path.basename(fn))[0].split('_')[0]
+                    pn = res.group(2) or recipepn
                     # Find the recipe file within the workspace, if any
                     bbfile = os.path.basename(fn).replace('.bbappend', '.bb').replace('%', '*')
                     recipefile = glob.glob(os.path.join(config.workspace_path,
                                                         'recipes',
-                                                        pn,
+                                                        recipepn,
                                                         bbfile))
                     if recipefile:
                         recipefile = recipefile[0]
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 26187a0c413..f29d8cbb689 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1744,6 +1744,7 @@ def status(args, config, basepath, workspace):
 
 def _reset(recipes, no_clean, config, basepath, workspace):
     """Reset one or more recipes"""
+    import oe.path
 
     def clean_preferred_provider(pn, layerconf_path):
         """Remove PREFERRED_PROVIDER from layer.conf'"""
@@ -1802,7 +1803,10 @@ def _reset(recipes, no_clean, config, basepath, workspace):
                         preservedir(os.path.join(root, dn))
                 os.rmdir(origdir)
 
-        preservedir(os.path.join(config.workspace_path, 'recipes', pn))
+        recipefile = workspace[pn]['recipefile']
+        if recipefile and oe.path.is_path_parent(config.workspace_path, recipefile):
+            # This should always be true if recipefile is set, but just in case
+            preservedir(os.path.dirname(recipefile))
         # We don't automatically create this dir next to appends, but the user can
         preservedir(os.path.join(config.workspace_path, 'appends', pn))
 
-- 
2.14.3

[PATCH v2 3/8] devtool: reset: delete bbappend file if _check_preserve() doesn't

[Paul Eggleton]

If the .devtool_md5 file doesn't contain a reference to the bbappend
file (e.g. because devtool was interrupted before it could write that
out) then _check_preserve() won't delete it, so we need to delete it
separately because otherwise the recipe won't actually be reset.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index f29d8cbb689..2dbb1976422 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1791,6 +1791,13 @@ def _reset(recipes, no_clean, config, basepath, workspace):
     for pn in recipes:
         _check_preserve(config, pn)
 
+        appendfile = workspace[pn]['bbappend']
+        if os.path.exists(appendfile):
+            # This shouldn't happen, but is possible if devtool errored out prior to
+            # writing the md5 file. We need to delete this here or the recipe won't
+            # actually be reset
+            os.remove(appendfile)
+
         preservepath = os.path.join(config.workspace_path, 'attic', pn, pn)
         def preservedir(origdir):
             if os.path.exists(origdir):
-- 
2.14.3

[PATCH v2 4/8] devtool: finish: fix erroneously creating bbappend for relative paths

[Paul Eggleton]

After OE-Core rev 5e3fe00a0233d563781849a44f53885b4e924a9c we call
os.path.abspath() on the original layer path, but we later compare that
to the destination layer path. If that layer path isn't absolute but is
effectively the same path, it should be writing to the original recipe
but because we weren't making it absolute we were writing a bbappend
instead. Call os.path.abspath() on the destination path as well to avoid
that.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 2dbb1976422..a962ebe09b4 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1861,9 +1861,7 @@ def _get_layer(layername, d):
         layerdir = layers.get('meta', None)
     else:
         layerdir = layers.get(layername, None)
-    if layerdir:
-        layerdir = os.path.abspath(layerdir)
-    return layerdir or layername
+    return os.path.abspath(layerdir or layername)
 
 def finish(args, config, basepath, workspace):
     """Entry point for the devtool 'finish' subcommand"""
-- 
2.14.3

[PATCH v2 5/8] devtool: deploy-target: don't specify ssh/scp port unless user does

[Paul Eggleton]

If the user doesn't specify a port then we should avoid specifying one
on the ssh/scp command line in case the user has configured one for the
host they are connecting to, which was being overridden unnecessarily.

Fixes [YOCTO #12381].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/deploy.py | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py
index 9cc492788b6..52e261d5600 100644
--- a/scripts/lib/devtool/deploy.py
+++ b/scripts/lib/devtool/deploy.py
@@ -213,9 +213,7 @@ def deploy(args, config, basepath, workspace):
 
         scp_port = ''
         ssh_port = ''
-        if not args.port:
-            raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target")
-        else:
+        if args.port:
             scp_port = "-P %s" % args.port
             ssh_port = "-p %s" % args.port
 
@@ -280,9 +278,7 @@ def undeploy(args, config, basepath, workspace):
 
     scp_port = ''
     ssh_port = ''
-    if not args.port:
-        raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target")
-    else:
+    if args.port:
         scp_port = "-P %s" % args.port
         ssh_port = "-p %s" % args.port
 
@@ -328,7 +324,7 @@ def register_commands(subparsers, context):
     parser_deploy.add_argument('-n', '--dry-run', help='List files to be deployed only', action='store_true')
     parser_deploy.add_argument('-p', '--no-preserve', help='Do not preserve existing files', action='store_true')
     parser_deploy.add_argument('--no-check-space', help='Do not check for available space before deploying', action='store_true')
-    parser_deploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target')
+    parser_deploy.add_argument('-P', '--port', help='Specify port to use for connection to the target')
 
     strip_opts = parser_deploy.add_mutually_exclusive_group(required=False)
     strip_opts.add_argument('-S', '--strip',
@@ -350,5 +346,5 @@ def register_commands(subparsers, context):
     parser_undeploy.add_argument('-s', '--show-status', help='Show progress/status output', action='store_true')
     parser_undeploy.add_argument('-a', '--all', help='Undeploy all recipes deployed on the target', action='store_true')
     parser_undeploy.add_argument('-n', '--dry-run', help='List files to be undeployed only', action='store_true')
-    parser_undeploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target')
+    parser_undeploy.add_argument('-P', '--port', help='Specify port to use for connection to the target')
     parser_undeploy.set_defaults(func=undeploy)
-- 
2.14.3

[PATCH v2 6/8] lib/oe/recipeutils: add .txz extension to archive list

[Paul Eggleton]

Prompted by bitbake commit 2ba8a6b25ccc12e7b543e8450121e5311c7a701d,
add .txz to the list of archives used within get_recipe_local_files()
here as well.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/recipeutils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 4e0859e6d90..aa64553c06b 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -408,7 +408,7 @@ def get_recipe_local_files(d, patches=False, archives=False):
     # fetcher) though note that this only encompasses actual container formats
     # i.e. that can contain multiple files as opposed to those that only
     # contain a compressed stream (i.e. .tar.gz as opposed to just .gz)
-    archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z']
+    archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.txz', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z']
     ret = {}
     for uri in uris:
         if fetch.ud[uri].type == 'file':
-- 
2.14.3

[PATCH v2 7/8] devtool: search: also look in recipe cache

[Paul Eggleton]

If pkgdata isn't present or is incomplete, then you get either a
traceback or you don't see the results you were hoping for. The recipe
cache that bitbake collects during startup contains some useful
information for each recipe that we could search through as well, and
we can access it easily using tinfoil's all_recipes() API function,
so add some code that does that. (We still show a warning if pkgdata
isn't present, as there are certain dynamic packages that are generated
at packaging time that won't show up in the cache).

One side-effect of this is that we will start showing non-target
recipes - that's actually a good thing, since seeing those is useful,
however we exclude nativesdk recipes when in the eSDK to avoid confusion
since nativesdk isn't directly applicable there.

Fixes [YOCTO #12356].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/search.py | 96 ++++++++++++++++++++++++++++---------------
 1 file changed, 63 insertions(+), 33 deletions(-)

diff --git a/scripts/lib/devtool/search.py b/scripts/lib/devtool/search.py
index 054985b85d5..b51a7b80207 100644
--- a/scripts/lib/devtool/search.py
+++ b/scripts/lib/devtool/search.py
@@ -36,44 +36,74 @@ def search(args, config, basepath, workspace):
 
         keyword_rc = re.compile(args.keyword)
 
-        for fn in os.listdir(pkgdata_dir):
-            pfn = os.path.join(pkgdata_dir, fn)
-            if not os.path.isfile(pfn):
+        def print_match(pn):
+            rd = parse_recipe(config, tinfoil, pn, True)
+            if not rd:
+                return
+            summary = rd.getVar('SUMMARY')
+            if summary == rd.expand(defsummary):
+                summary = ''
+            print("%s  %s" % (pn.ljust(20), summary))
+
+
+        matches = []
+        if os.path.exists(pkgdata_dir):
+            for fn in os.listdir(pkgdata_dir):
+                pfn = os.path.join(pkgdata_dir, fn)
+                if not os.path.isfile(pfn):
+                    continue
+
+                packages = []
+                match = False
+                if keyword_rc.search(fn):
+                    match = True
+
+                if not match:
+                    with open(pfn, 'r') as f:
+                        for line in f:
+                            if line.startswith('PACKAGES:'):
+                                packages = line.split(':', 1)[1].strip().split()
+
+                    for pkg in packages:
+                        if keyword_rc.search(pkg):
+                            match = True
+                            break
+                        if os.path.exists(os.path.join(pkgdata_dir, 'runtime', pkg + '.packaged')):
+                            with open(os.path.join(pkgdata_dir, 'runtime', pkg), 'r') as f:
+                                for line in f:
+                                    if ': ' in line:
+                                        splitline = line.split(':', 1)
+                                        key = splitline[0]
+                                        value = splitline[1].strip()
+                                    if key in ['PKG_%s' % pkg, 'DESCRIPTION', 'FILES_INFO'] or key.startswith('FILERPROVIDES_'):
+                                        if keyword_rc.search(value):
+                                            match = True
+                                            break
+                if match:
+                    print_match(fn)
+                    matches.append(fn)
+        else:
+            logger.warning('Package data is not available, results may be limited')
+
+        for recipe in tinfoil.all_recipes():
+            if args.fixed_setup and 'nativesdk' in recipe.inherits():
                 continue
 
-            packages = []
             match = False
-            if keyword_rc.search(fn):
+            if keyword_rc.search(recipe.pn):
                 match = True
-
-            if not match:
-                with open(pfn, 'r') as f:
-                    for line in f:
-                        if line.startswith('PACKAGES:'):
-                            packages = line.split(':', 1)[1].strip().split()
-
-                for pkg in packages:
-                    if keyword_rc.search(pkg):
+            else:
+                for prov in recipe.provides:
+                    if keyword_rc.search(prov):
                         match = True
                         break
-                    if os.path.exists(os.path.join(pkgdata_dir, 'runtime', pkg + '.packaged')):
-                        with open(os.path.join(pkgdata_dir, 'runtime', pkg), 'r') as f:
-                            for line in f:
-                                if ': ' in line:
-                                    splitline = line.split(':', 1)
-                                    key = splitline[0]
-                                    value = splitline[1].strip()
-                                if key in ['PKG_%s' % pkg, 'DESCRIPTION', 'FILES_INFO'] or key.startswith('FILERPROVIDES_'):
-                                    if keyword_rc.search(value):
-                                        match = True
-                                        break
-
-            if match:
-                rd = parse_recipe(config, tinfoil, fn, True)
-                summary = rd.getVar('SUMMARY')
-                if summary == rd.expand(defsummary):
-                    summary = ''
-                print("%s  %s" % (fn.ljust(20), summary))
+                if not match:
+                    for rprov in recipe.rprovides:
+                        if keyword_rc.search(rprov):
+                            match = True
+                            break
+            if match and not recipe.pn in matches:
+                print_match(recipe.pn)
     finally:
         tinfoil.shutdown()
 
@@ -85,4 +115,4 @@ def register_commands(subparsers, context):
                                             description='Searches for available target recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name on match.',
                                             group='info')
     parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed)')
-    parser_search.set_defaults(func=search, no_workspace=True)
+    parser_search.set_defaults(func=search, no_workspace=True, fixed_setup=context.fixed_setup)
-- 
2.14.3

[PATCH v2 8/8] devtool: search: tweak help text

[Paul Eggleton]

* We now match on more than just target recipes, so don't specify that
  only target recipes are searched.
* We're printing the SUMMARY value in addition to the name, so mention
  that so it's clear where that text is coming from.
* Remind users that they should use quotes around the keyword to avoid
  shell expansion when using regular expressions.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/search.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/search.py b/scripts/lib/devtool/search.py
index b51a7b80207..b4f209b7e3a 100644
--- a/scripts/lib/devtool/search.py
+++ b/scripts/lib/devtool/search.py
@@ -112,7 +112,7 @@ def search(args, config, basepath, workspace):
 def register_commands(subparsers, context):
     """Register devtool subcommands from this plugin"""
     parser_search = subparsers.add_parser('search', help='Search available recipes',
-                                            description='Searches for available target recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name on match.',
+                                            description='Searches for available recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name and summary on match.',
                                             group='info')
-    parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed)')
+    parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed, use quotes to avoid shell expansion)')
     parser_search.set_defaults(func=search, no_workspace=True, fixed_setup=context.fixed_setup)
-- 
2.14.3