[PATCH 0/4] devtool fixes

[PATCH 0/4] devtool fixes

[Paul Eggleton]

Three bugfixes for devtool plus one refactoring patch to improve a
related function in the recipeutils module.


The following changes since commit 059db140885bad379534e6ec713f3ceb4e18faea:

  adt-installer: use DEPLOY_DIR in ANT_DEPLOY expansion (2015-08-16 17:28:14 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool-fixes5
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool-fixes5

Paul Eggleton (4):
  devtool: extract: prevent externalsrc from interfering with extraction
  devtool: extract: remove patches subdirectory when S == WORKDIR
  devtool: fix handling of BBCLASSEXTENDed recipes
  lib/oe/recipeutils: avoid parsing in get_var_files()

 meta/lib/oe/recipeutils.py        |  3 +--
 meta/lib/oeqa/selftest/devtool.py | 30 ++++++++++++++++++++++++++++++
 scripts/devtool                   | 12 ++++++------
 scripts/lib/devtool/standard.py   | 29 +++++++++++++++++------------
 4 files changed, 54 insertions(+), 20 deletions(-)

-- 
2.1.0

[PATCH 1/4] devtool: extract: prevent externalsrc from interfering with extraction

[Paul Eggleton]

In case the user has set up externalsrc outside of devtool, force
EXTERNALSRC to blank for the recipe when extracting so that the original
source URI is still in SRC_URI and we're still able to extract it. (This
isn't a problem with devtool itself because the bbappends within the
workspace layer that apply externalsrc are explicitly filtered out when
devtool parses a recipe).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index ea21877..e4ee7f7 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -294,6 +294,8 @@ def _extract_source(srctree, keep_temp, devbranch, d):
 
         task_executor = BbTaskExecutor(crd)
 
+        crd.setVar('EXTERNALSRC_forcevariable', '')
+
         logger.info('Fetching %s...' % pn)
         task_executor.exec_func('do_fetch', False)
         logger.info('Unpacking...')
-- 
2.1.0

[PATCH 2/4] devtool: extract: remove patches subdirectory when S == WORKDIR

[Paul Eggleton]

Ensure that the "patches" subdirectory is removed from the right
location when S == WORKDIR (e.g. devtool extract makedevs).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index e4ee7f7..658076c 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -301,7 +301,6 @@ def _extract_source(srctree, keep_temp, devbranch, d):
         logger.info('Unpacking...')
         task_executor.exec_func('do_unpack', False)
         srcsubdir = crd.getVar('S', True)
-        patchsubdir = srcsubdir
         if srcsubdir == workdir:
             # Find non-patch sources that were "unpacked" to srctree directory
             recipe_patches = [os.path.basename(patch) for patch in
@@ -322,7 +321,7 @@ def _extract_source(srctree, keep_temp, devbranch, d):
 
         scriptutils.git_convert_standalone_clone(srcsubdir)
 
-        patchdir = os.path.join(patchsubdir, 'patches')
+        patchdir = os.path.join(srcsubdir, 'patches')
         haspatches = False
         if os.path.exists(patchdir):
             if os.listdir(patchdir):
-- 
2.1.0

[PATCH 3/4] devtool: fix handling of BBCLASSEXTENDed recipes

[Paul Eggleton]

If a recipe is BBCLASSEXTENDed (e.g. to -native), its PN value and the
name of the bbappend will be different; we were assuming them to be the
same when reading in the workspace, leading to us seeing the base recipe
name everywhere afterwards.

Also add a test so we ensure this doesn't regress in future.

Fixes [YOCTO #8157].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oeqa/selftest/devtool.py | 30 ++++++++++++++++++++++++++++++
 scripts/devtool                   | 12 ++++++------
 scripts/lib/devtool/standard.py   | 20 ++++++++++++--------
 3 files changed, 48 insertions(+), 14 deletions(-)

diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index b59db15..947d8ee 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -367,6 +367,36 @@ class DevtoolTests(DevtoolBase):
             self.assertNotEqual(result.status, 0, 'devtool modify on %s should have failed. devtool output: %s' %  (testrecipe, result.output))
             self.assertIn('ERROR: ', result.output, 'devtool modify on %s should have given an ERROR' % testrecipe)
 
+    def test_devtool_modify_native(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        # Try modifying some recipes
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+
+        bbclassextended = False
+        inheritnative = False
+        testrecipes = 'mtools-native apt-native desktop-file-utils-native'.split()
+        for testrecipe in testrecipes:
+            checkextend = 'native' in (get_bb_var('BBCLASSEXTEND', testrecipe) or '').split()
+            if not bbclassextended:
+                bbclassextended = checkextend
+            if not inheritnative:
+                inheritnative = not checkextend
+            result = runCmd('devtool modify %s -x %s' % (testrecipe, os.path.join(tempdir, testrecipe)))
+            self.assertNotIn('ERROR: ', result.output, 'ERROR in devtool modify output: %s' % result.output)
+            result = runCmd('devtool build %s' % testrecipe)
+            self.assertNotIn('ERROR: ', result.output, 'ERROR in devtool build output: %s' % result.output)
+            result = runCmd('devtool reset %s' % testrecipe)
+            self.assertNotIn('ERROR: ', result.output, 'ERROR in devtool reset output: %s' % result.output)
+
+        self.assertTrue(bbclassextended, 'None of these recipes are BBCLASSEXTENDed to native - need to adjust testrecipes list: %s' % ', '.join(testrecipes))
+        self.assertTrue(inheritnative, 'None of these recipes do "inherit native" - need to adjust testrecipes list: %s' % ', '.join(testrecipes))
+
+
     @testcase(1165)
     def test_devtool_modify_git(self):
         # Check preconditions
diff --git a/scripts/devtool b/scripts/devtool
index 1c22438..b9d3bb9 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -104,15 +104,15 @@ def read_workspace():
         _enable_workspace_layer(config.workspace_path, config, basepath)
 
     logger.debug('Reading workspace in %s' % config.workspace_path)
-    externalsrc_re = re.compile(r'^EXTERNALSRC(_pn-[^ =]+)? =.*$')
+    externalsrc_re = re.compile(r'^EXTERNALSRC(_pn-([^ =]+))? *= *"([^"]*)"$')
     for fn in glob.glob(os.path.join(config.workspace_path, 'appends', '*.bbappend')):
-        pn = os.path.splitext(os.path.basename(fn))[0].split('_')[0]
         with open(fn, 'r') as f:
             for line in f:
-                if externalsrc_re.match(line.rstrip()):
-                    splitval = line.split('=', 2)
-                    workspace[pn] = splitval[1].strip('" \n\r\t')
-                    break
+                res = externalsrc_re.match(line.rstrip())
+                if res:
+                    pn = res.group(2) or os.path.splitext(os.path.basename(fn))[0].split('_')[0]
+                    workspace[pn] = {'srctree': res.group(3),
+                                     'bbappend': fn}
 
 def create_workspace(args, config, basepath, workspace):
     if args.layerpath:
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 658076c..e85e1ad 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -430,6 +430,16 @@ def modify(args, config, basepath, workspace):
     if not rd:
         return 1
     recipefile = rd.getVar('FILE', True)
+    appendname = os.path.splitext(os.path.basename(recipefile))[0]
+    if args.wildcard:
+        appendname = re.sub(r'_.*', '_%', appendname)
+    appendpath = os.path.join(config.workspace_path, 'appends')
+    appendfile = os.path.join(appendpath, appendname + '.bbappend')
+    if os.path.exists(appendfile):
+        raise DevtoolError("Another variant of recipe %s is already in your "
+                           "workspace (only one variant of a recipe can "
+                           "currently be worked on at once)"
+                           % args.recipename)
 
     _check_compatible_recipe(args.recipename, rd)
 
@@ -467,14 +477,8 @@ def modify(args, config, basepath, workspace):
         srcsubdir = os.path.relpath(s, workdir).split(os.sep, 1)[1]
         srctree = os.path.join(srctree, srcsubdir)
 
-    appendpath = os.path.join(config.workspace_path, 'appends')
     if not os.path.exists(appendpath):
         os.makedirs(appendpath)
-
-    appendname = os.path.splitext(os.path.basename(recipefile))[0]
-    if args.wildcard:
-        appendname = re.sub(r'_.*', '_%', appendname)
-    appendfile = os.path.join(appendpath, appendname + '.bbappend')
     with open(appendfile, 'w') as f:
         f.write('FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"\n\n')
         f.write('inherit externalsrc\n')
@@ -777,7 +781,7 @@ def update_recipe(args, config, basepath, workspace):
     else:
         mode = args.mode
 
-    srctree = workspace[args.recipename]
+    srctree = workspace[args.recipename]['srctree']
 
     if mode == 'srcrev':
         _update_recipe_srcrev(args, srctree, rd, tinfoil.config_data)
@@ -793,7 +797,7 @@ def status(args, config, basepath, workspace):
     """Entry point for the devtool 'status' subcommand"""
     if workspace:
         for recipe, value in workspace.iteritems():
-            print("%s: %s" % (recipe, value))
+            print("%s: %s" % (recipe, value['srctree']))
     else:
         logger.info('No recipes currently in your workspace - you can use "devtool modify" to work on an existing recipe or "devtool add" to add a new one')
     return 0
-- 
2.1.0

[PATCH 4/4] lib/oe/recipeutils: avoid parsing in get_var_files()

[Paul Eggleton]

Let's have the caller do this and then the function is a bit more
flexible (e.g. we can choose to parse with bbappends or not); fix up
calls to this function appropriately (of which there are only two, both
within devtool).

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/recipeutils.py      | 3 +--
 scripts/lib/devtool/standard.py | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index cd74213..d4fa726 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -79,10 +79,9 @@ def get_var_files(fn, varlist, d):
     """Find the file in which each of a list of variables is set.
     Note: requires variable history to be enabled when parsing.
     """
-    envdata = parse_recipe(fn, [], d)
     varfiles = {}
     for v in varlist:
-        history = envdata.varhistory.variable(v)
+        history = d.varhistory.variable(v)
         files = []
         for event in history:
             if 'file' in event and not 'flag' in event:
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index e85e1ad..e1c5584 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -630,7 +630,7 @@ def _update_recipe_srcrev(args, srctree, rd, config_data):
                 rd, args.append, None, wildcardver=args.wildcard_version,
                 extralines=patchfields)
     else:
-        oe.recipeutils.patch_recipe(config_data, recipefile, patchfields)
+        oe.recipeutils.patch_recipe(rd, recipefile, patchfields)
 
     if not 'git://' in orig_src_uri:
         logger.info('You will need to update SRC_URI within the recipe to '
@@ -742,7 +742,7 @@ def _update_recipe_patch(args, config, srctree, rd, config_data):
                     updaterecipe = True
             if updaterecipe:
                 logger.info('Updating recipe %s' % os.path.basename(recipefile))
-                oe.recipeutils.patch_recipe(config_data, recipefile,
+                oe.recipeutils.patch_recipe(rd, recipefile,
                                             {'SRC_URI': ' '.join(srcuri)})
             elif not updatepatches:
                 # Neither patches nor recipe were updated
-- 
2.1.0

[PATCH 0/4] devtool fixes

[Paul Eggleton]

A few fixes for devtool, mostly relating to recipes that fetch from git
where S points to a subdirectory of the repo.

Note: I wasn't 100% sure if it was appropriate to be referencing WORKDIR
in patch.py; let me know if I should try to rework that.


The following changes since commit db28cd0e1540e44db963108430205c8c0c817774:

  gperf: Switch to upstream patch (2022-07-09 20:58:31 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool37-oe
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool37-oe

Paul Eggleton (4):
  patch: handle if S points to a subdirectory of a git repo
  devtool: finish: handle patching when S points to subdir of a git repo
  devtool: ignore pn- overrides when determining SRC_URI overrides
  oe-selftest: devtool: test modify git recipe building from a subdir

 meta/lib/oe/patch.py                    |  8 +++---
 meta/lib/oe/recipeutils.py              |  9 ++++--
 meta/lib/oeqa/selftest/cases/devtool.py | 51 +++++++++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py         | 29 ++++++++++++++-----
 4 files changed, 84 insertions(+), 13 deletions(-)

-- 
1.8.3.1