[PATCH v2 0/3] devtool fixes

[PATCH v2 0/3] devtool fixes

[Paul Eggleton]

A couple of patches Markus previously sent rebased on top of master
(also applicable on top of poky master-next) to fix some minor issues in
devtool, plus one minor optimisation from me.


The following changes since commit 2ad7308ee7166641eff99f3b9fe6794de143f6bc:

  oeqa/utils/qemurunner.py: Remove duplicate message on LoggingThread start (2015-09-22 18:13:02 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool-fixes6
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool-fixes6

Markus Lehtonen (2):
  devtool: second fix for running from a different directory
  devtool: upgrade: use shutil.move instead of os.rename

Paul Eggleton (1):
  devtool: runqemu: avoid recipe parse

 scripts/devtool                    |  5 +----
 scripts/lib/devtool/__init__.py    |  6 +++++-
 scripts/lib/devtool/build-image.py |  2 +-
 scripts/lib/devtool/deploy.py      |  2 +-
 scripts/lib/devtool/package.py     |  2 +-
 scripts/lib/devtool/runqemu.py     |  2 +-
 scripts/lib/devtool/search.py      |  2 +-
 scripts/lib/devtool/standard.py    | 10 +++++-----
 scripts/lib/devtool/upgrade.py     |  4 ++--
 9 files changed, 18 insertions(+), 17 deletions(-)

-- 
2.1.0

[PATCH v2 1/3] devtool: second fix for running from a different directory

[Paul Eggleton]

From: Markus Lehtonen <markus.lehtonen@linux.intel.com>

Do not change change current working directory permanently, but, only
for the duration of tinfoil initialization instead. The previous fix
caused very unintuitive behavior where using relative paths were solved
with respect to the builddir instead of the current working directory.
E.g. calling "devtool extract zlib ./zlib" would always create create
srctree in ${TOPDIR}/zlib, independent of the users cwd.

(From OE-Core rev: 4c7f159b0e17a0475a4a4e9dc4dd012e3d2e6a1f)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/devtool                    |  5 +----
 scripts/lib/devtool/__init__.py    |  6 +++++-
 scripts/lib/devtool/build-image.py |  2 +-
 scripts/lib/devtool/deploy.py      |  2 +-
 scripts/lib/devtool/package.py     |  2 +-
 scripts/lib/devtool/runqemu.py     |  2 +-
 scripts/lib/devtool/search.py      |  2 +-
 scripts/lib/devtool/standard.py    | 10 +++++-----
 scripts/lib/devtool/upgrade.py     |  2 +-
 9 files changed, 17 insertions(+), 16 deletions(-)

diff --git a/scripts/devtool b/scripts/devtool
index 87df951..e4d9db3 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -221,9 +221,6 @@ def main():
     if not config.read():
         return -1
 
-    # We need to be in this directory or we won't be able to initialise tinfoil
-    os.chdir(basepath)
-
     bitbake_subdir = config.get('General', 'bitbake_subdir', '')
     if bitbake_subdir:
         # Normally set for use within the SDK
@@ -244,7 +241,7 @@ def main():
     scriptutils.logger_setup_color(logger, global_args.color)
 
     if global_args.bbpath is None:
-        tinfoil = setup_tinfoil(config_only=True)
+        tinfoil = setup_tinfoil(config_only=True, basepath=basepath)
         global_args.bbpath = tinfoil.config_data.getVar('BBPATH', True)
     else:
         tinfoil = None
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index f815ef2..7b1ab11 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -96,9 +96,12 @@ def exec_fakeroot(d, cmd, **kwargs):
             newenv[splitval[0]] = splitval[1]
     return subprocess.call("%s %s" % (fakerootcmd, cmd), env=newenv, **kwargs)
 
-def setup_tinfoil(config_only=False):
+def setup_tinfoil(config_only=False, basepath=None):
     """Initialize tinfoil api from bitbake"""
     import scriptpath
+    orig_cwd = os.path.abspath(os.curdir)
+    if basepath:
+        os.chdir(basepath)
     bitbakepath = scriptpath.add_bitbake_lib_path()
     if not bitbakepath:
         logger.error("Unable to find bitbake by searching parent directory of this script or PATH")
@@ -108,6 +111,7 @@ def setup_tinfoil(config_only=False):
     tinfoil = bb.tinfoil.Tinfoil()
     tinfoil.prepare(config_only)
     tinfoil.logger.setLevel(logger.getEffectiveLevel())
+    os.chdir(orig_cwd)
     return tinfoil
 
 def get_recipe_file(cooker, pn):
diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index 05c1d75..e53239d 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -59,7 +59,7 @@ def build_image(args, config, basepath, workspace):
     if os.path.isfile(appendfile):
         os.unlink(appendfile)
 
-    tinfoil = setup_tinfoil()
+    tinfoil = setup_tinfoil(basepath=basepath)
     rd = parse_recipe(config, tinfoil, image, True)
     if not rd:
         # Error already shown
diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py
index 41b666f..c90c6b1 100644
--- a/scripts/lib/devtool/deploy.py
+++ b/scripts/lib/devtool/deploy.py
@@ -40,7 +40,7 @@ def deploy(args, config, basepath, workspace):
     deploy_dir = os.path.join(basepath, 'target_deploy', args.target)
     deploy_file = os.path.join(deploy_dir, args.recipename + '.list')
 
-    tinfoil = setup_tinfoil()
+    tinfoil = setup_tinfoil(basepath=basepath)
     try:
         rd = oe.recipeutils.parse_recipe_simple(tinfoil.cooker, args.recipename, tinfoil.config_data)
     except Exception as e:
diff --git a/scripts/lib/devtool/package.py b/scripts/lib/devtool/package.py
index 28ecfed..b8d8423 100644
--- a/scripts/lib/devtool/package.py
+++ b/scripts/lib/devtool/package.py
@@ -34,7 +34,7 @@ def package(args, config, basepath, workspace):
 
     image_pkgtype = config.get('Package', 'image_pkgtype', '')
     if not image_pkgtype:
-        tinfoil = setup_tinfoil()
+        tinfoil = setup_tinfoil(basepath=basepath)
         try:
             tinfoil.prepare(config_only=True)
             image_pkgtype = tinfoil.config_data.getVar('IMAGE_PKGTYPE', True)
diff --git a/scripts/lib/devtool/runqemu.py b/scripts/lib/devtool/runqemu.py
index e7f26ab..4d08d8c 100644
--- a/scripts/lib/devtool/runqemu.py
+++ b/scripts/lib/devtool/runqemu.py
@@ -29,7 +29,7 @@ logger = logging.getLogger('devtool')
 def runqemu(args, config, basepath, workspace):
     """Entry point for the devtool 'runqemu' subcommand"""
 
-    tinfoil = setup_tinfoil()
+    tinfoil = setup_tinfoil(basepath=basepath)
     machine = tinfoil.config_data.getVar('MACHINE', True)
     bindir_native = tinfoil.config_data.getVar('STAGING_BINDIR_NATIVE', True)
     tinfoil.shutdown()
diff --git a/scripts/lib/devtool/search.py b/scripts/lib/devtool/search.py
index e6ae922..c2f420c 100644
--- a/scripts/lib/devtool/search.py
+++ b/scripts/lib/devtool/search.py
@@ -29,7 +29,7 @@ logger = logging.getLogger('devtool')
 def search(args, config, basepath, workspace):
     """Entry point for the devtool 'search' subcommand"""
 
-    tinfoil = setup_tinfoil(config_only=True)
+    tinfoil = setup_tinfoil(config_only=True, basepath=basepath)
     pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR', True)
     tinfoil.shutdown()
 
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 96b271c..1dcf7cd 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -111,7 +111,7 @@ def add(args, config, basepath, workspace):
         (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
         initial_rev = stdout.rstrip()
 
-    tinfoil = setup_tinfoil(config_only=True)
+    tinfoil = setup_tinfoil(config_only=True, basepath=basepath)
     rd = oe.recipeutils.parse_recipe(recipefile, None, tinfoil.config_data)
     if not rd:
         return 1
@@ -231,7 +231,7 @@ class BbTaskExecutor(object):
 def _prep_extract_operation(config, basepath, recipename):
     """HACK: Ugly workaround for making sure that requirements are met when
        trying to extract a package. Returns the tinfoil instance to be used."""
-    tinfoil = setup_tinfoil()
+    tinfoil = setup_tinfoil(basepath=basepath)
     rd = parse_recipe(config, tinfoil, recipename, True)
 
     if bb.data.inherits_class('kernel-yocto', rd):
@@ -239,7 +239,7 @@ def _prep_extract_operation(config, basepath, recipename):
         try:
             stdout, _ = exec_build_env_command(config.init_path, basepath,
                                                'bitbake kern-tools-native')
-            tinfoil = setup_tinfoil()
+            tinfoil = setup_tinfoil(basepath=basepath)
         except bb.process.ExecutionError as err:
             raise DevtoolError("Failed to build kern-tools-native:\n%s" %
                                err.stdout)
@@ -443,7 +443,7 @@ def modify(args, config, basepath, workspace):
     if args.extract:
         tinfoil = _prep_extract_operation(config, basepath, args.recipename)
     else:
-        tinfoil = setup_tinfoil()
+        tinfoil = setup_tinfoil(basepath=basepath)
 
     rd = parse_recipe(config, tinfoil, args.recipename, True)
     if not rd:
@@ -797,7 +797,7 @@ def update_recipe(args, config, basepath, workspace):
             raise DevtoolError('conf/layer.conf not found in bbappend '
                                'destination layer "%s"' % args.append)
 
-    tinfoil = setup_tinfoil()
+    tinfoil = setup_tinfoil(basepath=basepath)
 
     rd = parse_recipe(config, tinfoil, args.recipename, True)
     if not rd:
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 6c1dfee..aa53c8e 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -294,7 +294,7 @@ def upgrade(args, config, basepath, workspace):
     if reason:
         raise DevtoolError(reason)
 
-    tinfoil = setup_tinfoil()
+    tinfoil = setup_tinfoil(basepath=basepath)
 
     rd = parse_recipe(config, tinfoil, args.recipename, True)
     if not rd:
-- 
2.1.0

[PATCH v2 2/3] devtool: runqemu: avoid recipe parse

[Paul Eggleton]

We only need the base configuration to get the variable values we want
to get here, there's no need to parse recipes / load the cache.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/runqemu.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/runqemu.py b/scripts/lib/devtool/runqemu.py
index 4d08d8c..5282afb 100644
--- a/scripts/lib/devtool/runqemu.py
+++ b/scripts/lib/devtool/runqemu.py
@@ -29,7 +29,7 @@ logger = logging.getLogger('devtool')
 def runqemu(args, config, basepath, workspace):
     """Entry point for the devtool 'runqemu' subcommand"""
 
-    tinfoil = setup_tinfoil(basepath=basepath)
+    tinfoil = setup_tinfoil(config_only=True, basepath=basepath)
     machine = tinfoil.config_data.getVar('MACHINE', True)
     bindir_native = tinfoil.config_data.getVar('STAGING_BINDIR_NATIVE', True)
     tinfoil.shutdown()
-- 
2.1.0

[PATCH v2 3/3] devtool: upgrade: use shutil.move instead of os.rename

[Paul Eggleton]

From: Markus Lehtonen <markus.lehtonen@linux.intel.com>

Rename fails over filesystem boundaries.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/upgrade.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index aa53c8e..4f850cf 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -53,7 +53,7 @@ def _copy_source_code(orig, dest):
         dest_dir = os.path.join(dest, os.path.dirname(path))
         bb.utils.mkdirhier(dest_dir)
         dest_path = os.path.join(dest, path)
-        os.rename(os.path.join(orig, path), dest_path)
+        shutil.move(os.path.join(orig, path), dest_path)
 
 def _get_checksums(rf):
     import re
-- 
2.1.0

Re: [PATCH v2 1/3] devtool: second fix for running from a different directory

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1129 bytes --]

On Wed, Sep 23, 2015 at 3:05 AM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> -def setup_tinfoil(config_only=False):
> +def setup_tinfoil(config_only=False, basepath=None):
>      """Initialize tinfoil api from bitbake"""
>      import scriptpath
> +    orig_cwd = os.path.abspath(os.curdir)
> +    if basepath:
> +        os.chdir(basepath)
>      bitbakepath = scriptpath.add_bitbake_lib_path()
>      if not bitbakepath:
>          logger.error("Unable to find bitbake by searching parent
> directory of this script or PATH")
> @@ -108,6 +111,7 @@ def setup_tinfoil(config_only=False):
>      tinfoil = bb.tinfoil.Tinfoil()
>      tinfoil.prepare(config_only)
>      tinfoil.logger.setLevel(logger.getEffectiveLevel())
> +    os.chdir(orig_cwd)
>      return tinfoil
>

Just from a correctness standpoint, this will not go back to orig_cwd if an
exception is raised. We should be using a try/finally block for things like
this.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1702 bytes --]

Re: [PATCH v2 1/3] devtool: second fix for running from a different directory

[Paul Eggleton]

On Wednesday 23 September 2015 08:09:37 Christopher Larson wrote:
> On Wed, Sep 23, 2015 at 3:05 AM, Paul Eggleton <
> 
> paul.eggleton@linux.intel.com> wrote:
> > -def setup_tinfoil(config_only=False):
> > 
> > +def setup_tinfoil(config_only=False, basepath=None):
> >      """Initialize tinfoil api from bitbake"""
> >      import scriptpath
> > 
> > +    orig_cwd = os.path.abspath(os.curdir)
> > +    if basepath:
> > +        os.chdir(basepath)
> > 
> >      bitbakepath = scriptpath.add_bitbake_lib_path()
> >      
> >      if not bitbakepath:
> >          logger.error("Unable to find bitbake by searching parent
> > 
> > directory of this script or PATH")
> > 
> > @@ -108,6 +111,7 @@ def setup_tinfoil(config_only=False):
> >      tinfoil = bb.tinfoil.Tinfoil()
> >      tinfoil.prepare(config_only)
> >      tinfoil.logger.setLevel(logger.getEffectiveLevel())
> > 
> > +    os.chdir(orig_cwd)
> > 
> >      return tinfoil
> 
> Just from a correctness standpoint, this will not go back to orig_cwd if an
> exception is raised. We should be using a try/finally block for things like
> this.

That is a good point yes. I can send out a v3.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH v2 0/3] devtool fixes

[Paul Eggleton]

A few fixes for devtool, mostly relating to recipes that fetch from git
where S points to a subdirectory of the repo.

Changes since v1:
* Fixed erroneous addition of a separator when not needed (which broke
  one of the other devtool tests)
* Fixed missing separator before patchdir= when updating original recipe
* Added extra logic to the test to ensure we can update the original
  recipe properly (as well creating a bbappend that was already in the
  test).
* Rebased and dropped pn- overrides fix since that has just been merged.


The following changes since commit db22dbc1f66d2f76dab719597986cfd96cf18d71:

  linux-yocto/5.10: fix build_OID_registry/conmakehash buildpaths warning (2022-07-12 23:56:06 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/devtool37-oe
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool37-oe

Paul Eggleton (3):
  patch: handle if S points to a subdirectory of a git repo
  devtool: finish: handle patching when S points to subdir of a git repo
  oe-selftest: devtool: test modify git recipe building from a subdir

 meta/lib/oe/patch.py                    |   8 +--
 meta/lib/oe/recipeutils.py              |   9 ++-
 meta/lib/oeqa/selftest/cases/devtool.py | 114 +++++++++++++++++++++++++++-----
 scripts/lib/devtool/standard.py         |  25 +++++--
 4 files changed, 127 insertions(+), 29 deletions(-)

-- 
1.8.3.1