[PATCH 00/14] Extensible SDK / devtool / recipetool improvements

[PATCH 00/14] Extensible SDK / devtool / recipetool improvements

[Paul Eggleton]

A bunch of collected fixes for the extensible SDK, devtool and recipetool.


The following changes since commit 1568f4fa1c8f8abd9b365ecd264841cd4f6b66b1:

  bluez5: include the patch only for 5.36 (2015-12-28 13:02:03 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/extsdk-devtool-1
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/extsdk-devtool-1

Paul Eggleton (14):
  scripts/oe-publish-sdk: add missing call to git update-server-info
  classes/populate_sdk_ext: fix cascading from preparation failure
  classes/populate_sdk_ext: disable signature warnings
  devtool: sdk-update: fix not using updateserver config file option
  devtool: sdk-update: fix metadata update step
  devtool: sdk-update: fix error checking
  devtool: sdk-update: add option to skip preparation step
  recipetool: create: lower case name when determining from filename
  recipetool: create: support creating standalone native/nativesdk recipes
  devtool: reset: do clean for multiple recipes at once with -a
  devtool: add: support adding a native variant
  devtool: refactor code for getting local recipe file
  devtool: reset: support recipes with BBCLASSEXTEND
  devtool: build: support using BBCLASSEXTENDed names

 meta/classes/populate_sdk_ext.bbclass |  4 +--
 scripts/devtool                       | 12 ++++++-
 scripts/lib/devtool/__init__.py       | 50 +++++++++++++++++++++++++---
 scripts/lib/devtool/build.py          |  4 +--
 scripts/lib/devtool/sdk.py            | 61 ++++++++++++++++++-----------------
 scripts/lib/devtool/standard.py       | 58 +++++++++++++++++----------------
 scripts/lib/recipetool/create.py      | 34 +++++++++++++++++--
 scripts/oe-publish-sdk                |  4 +--
 8 files changed, 155 insertions(+), 72 deletions(-)

-- 
2.5.0

[PATCH 01/14] scripts/oe-publish-sdk: add missing call to git update-server-info

[Paul Eggleton]

We need to call git update-server-info here on the created repository or
we can't share it over plain http as we need to be able to for the
update process to function as currently implemented.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/oe-publish-sdk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/oe-publish-sdk b/scripts/oe-publish-sdk
index c8c79c2..9e4f1bf 100755
--- a/scripts/oe-publish-sdk
+++ b/scripts/oe-publish-sdk
@@ -102,9 +102,9 @@ def publish(args):
 
     # Setting up the git repo
     if not is_remote:
-        cmd = 'set -e; mkdir -p %s/layers; cd %s/layers; if [ ! -e .git ]; then git init .; mv .git/hooks/post-update.sample .git/hooks/post-update; fi; git add -A .; git commit -q -m "init repo" || true;' % (destination, destination)
+        cmd = 'set -e; mkdir -p %s/layers; cd %s/layers; if [ ! -e .git ]; then git init .; mv .git/hooks/post-update.sample .git/hooks/post-update; fi; git add -A .; git commit -q -m "init repo" || true; git update-server-info' % (destination, destination)
     else:
-        cmd = "ssh %s 'set -e; mkdir-p %s/layers; cd %s/layers; if [ ! -e .git ]; then git init .; mv .git/hooks/post-update.sample .git/hooks/post-update; fi; git add -A .; git commit -q -m \"init repo\" || true;'" % (host, destdir, destdir)
+        cmd = "ssh %s 'set -e; mkdir-p %s/layers; cd %s/layers; if [ ! -e .git ]; then git init .; mv .git/hooks/post-update.sample .git/hooks/post-update; fi; git add -A .; git commit -q -m \"init repo\" || true; git update-server-info'" % (host, destdir, destdir)
     ret = subprocess.call(cmd, shell=True)
     if ret == 0:
         logger.info('SDK published successfully')
-- 
2.5.0

[PATCH 02/14] classes/populate_sdk_ext: fix cascading from preparation failure

[Paul Eggleton]

During extensible SDK installtion, if the build system preparation step
fails we try to put something at the end of the environment setup script
to show an error when it is sourced, in case the user doesn't realise
that the partially-installed SDK is broken. However, an apostrophe in
the message (actually a single quote) appears to terminate the string
and therefore breaks the command. Drop it to avoid that.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index c30181a..be47a77 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -250,7 +250,7 @@ sdk_ext_postinst() {
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && $target_sdk_dir/ext-sdk-prepare.sh $target_sdk_dir '${SDK_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; echo "printf 'ERROR: this SDK wasn't fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && $target_sdk_dir/ext-sdk-prepare.sh $target_sdk_dir '${SDK_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
 	fi
 	rm -f $target_sdk_dir/ext-sdk-prepare.sh
 	echo done
-- 
2.5.0

[PATCH 03/14] classes/populate_sdk_ext: disable signature warnings

[Paul Eggleton]

The user of the extensible SDK doesn't need to see these.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index be47a77..317043d 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -153,7 +153,7 @@ python copy_buildsystem () {
         f.write('CONNECTIVITY_CHECK_URIS = ""\n\n')
 
         # Ensure locked sstate cache objects are re-used without error
-        f.write('SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "warn"\n\n')
+        f.write('SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "none"\n\n')
 
         # Hide the config information from bitbake output (since it's fixed within the SDK)
         f.write('BUILDCFG_HEADER = ""\n')
-- 
2.5.0

[PATCH 04/14] devtool: sdk-update: fix not using updateserver config file option

[Paul Eggleton]

We read the updateserver setting from the config file but we never
actually used that value - the code then went on to use only the value
supplied on the command line.

Fix courtesy of Dmitry Rozhkov <dmitry.rozhkov@intel.com>

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/sdk.py | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 47b9a7e..afe52cf 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -86,7 +86,7 @@ def sdk_update(args, config, basepath, workspace):
         updateserver = config.get('SDK', 'updateserver', '')
     if not updateserver:
         raise DevtoolError("Update server not specified in config file, you must specify it on the command line")
-    logger.debug("updateserver: %s" % args.updateserver)
+    logger.debug("updateserver: %s" % updateserver)
 
     # Make sure we are using sdk-update from within SDK
     logger.debug("basepath = %s" % basepath)
@@ -97,35 +97,35 @@ def sdk_update(args, config, basepath, workspace):
     else:
         logger.debug("Found conf/locked-sigs.inc in %s" % basepath)
 
-    if ':' in args.updateserver:
+    if ':' in updateserver:
         is_remote = True
     else:
         is_remote = False
 
     if not is_remote:
         # devtool sdk-update /local/path/to/latest/sdk
-        new_locked_sig_file_path = os.path.join(args.updateserver, 'conf/locked-sigs.inc')
+        new_locked_sig_file_path = os.path.join(updateserver, 'conf/locked-sigs.inc')
         if not os.path.exists(new_locked_sig_file_path):
-            logger.error("%s doesn't exist or is not an extensible SDK" % args.updateserver)
+            logger.error("%s doesn't exist or is not an extensible SDK" % updateserver)
             return -1
         else:
-            logger.debug("Found conf/locked-sigs.inc in %s" % args.updateserver)
+            logger.debug("Found conf/locked-sigs.inc in %s" % updateserver)
         update_dict = generate_update_dict(new_locked_sig_file_path, old_locked_sig_file_path)
         logger.debug("update_dict = %s" % update_dict)
-        sstate_objects = get_sstate_objects(update_dict, args.updateserver)
+        sstate_objects = get_sstate_objects(update_dict, updateserver)
         logger.debug("sstate_objects = %s" % sstate_objects)
         if len(sstate_objects) == 0:
             logger.info("No need to update.")
             return 0
         logger.info("Installing sstate objects into %s", basepath)
-        install_sstate_objects(sstate_objects, args.updateserver.rstrip('/'), basepath)
+        install_sstate_objects(sstate_objects, updateserver.rstrip('/'), basepath)
         logger.info("Updating configuration files")
-        new_conf_dir = os.path.join(args.updateserver, 'conf')
+        new_conf_dir = os.path.join(updateserver, 'conf')
         old_conf_dir = os.path.join(basepath, 'conf')
         shutil.rmtree(old_conf_dir)
         shutil.copytree(new_conf_dir, old_conf_dir)
         logger.info("Updating layers")
-        new_layers_dir = os.path.join(args.updateserver, 'layers')
+        new_layers_dir = os.path.join(updateserver, 'layers')
         old_layers_dir = os.path.join(basepath, 'layers')
         shutil.rmtree(old_layers_dir)
         ret = subprocess.call("cp -a %s %s" % (new_layers_dir, old_layers_dir), shell=True)
@@ -140,12 +140,12 @@ def sdk_update(args, config, basepath, workspace):
         os.makedirs(tmpsdk_dir)
         os.makedirs(os.path.join(tmpsdk_dir, 'conf'))
         # Fetch locked-sigs.inc from update server
-        ret = subprocess.call("wget -q -O - %s/conf/locked-sigs.inc > %s/locked-sigs.inc" % (args.updateserver, os.path.join(tmpsdk_dir, 'conf')), shell=True)
+        ret = subprocess.call("wget -q -O - %s/conf/locked-sigs.inc > %s/locked-sigs.inc" % (updateserver, os.path.join(tmpsdk_dir, 'conf')), shell=True)
         if ret != 0:
-            logger.error("Fetching conf/locked-sigs.inc from %s to %s/locked-sigs.inc failed" % (args.updateserver, os.path.join(tmpsdk_dir, 'conf')))
+            logger.error("Fetching conf/locked-sigs.inc from %s to %s/locked-sigs.inc failed" % (updateserver, os.path.join(tmpsdk_dir, 'conf')))
             return ret
         else:
-            logger.info("Fetching conf/locked-sigs.inc from %s to %s/locked-sigs.inc succeeded" % (args.updateserver, os.path.join(tmpsdk_dir, 'conf')))
+            logger.info("Fetching conf/locked-sigs.inc from %s to %s/locked-sigs.inc succeeded" % (updateserver, os.path.join(tmpsdk_dir, 'conf')))
         new_locked_sig_file_path = os.path.join(tmpsdk_dir, 'conf/locked-sigs.inc')
         update_dict = generate_update_dict(new_locked_sig_file_path, old_locked_sig_file_path)
         logger.debug("update_dict = %s" % update_dict)
@@ -160,19 +160,19 @@ def sdk_update(args, config, basepath, workspace):
         else:
             ret = -1
         if ret != 0:
-            ret = subprocess.call("rm -rf layers && git clone %s/layers" % args.updateserver, shell=True)
+            ret = subprocess.call("rm -rf layers && git clone %s/layers" % updateserver, shell=True)
         if ret != 0:
             logger.error("Updating meta data via git failed")
             return ret
         logger.debug("Updating conf files ...")
         conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'locked-sigs.inc']
         for conf in conf_files:
-            ret = subprocess.call("wget -q -O - %s/conf/%s > conf/%s" % (args.updateserver, conf, conf), shell=True)
+            ret = subprocess.call("wget -q -O - %s/conf/%s > conf/%s" % (updateserver, conf, conf), shell=True)
             if ret != 0:
                 logger.error("Update %s failed" % conf)
                 return ret
         with open(os.path.join(basepath, 'conf/local.conf'), 'a') as f:
-            f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % args.updateserver)
+            f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % updateserver)
 
     # Run bitbake command for the whole SDK
     sdk_targets = config.get('SDK', 'sdk_targets')
-- 
2.5.0

[PATCH 05/14] devtool: sdk-update: fix metadata update step

[Paul Eggleton]

* Clone the correct path - we need .git on the end
* Pull from the specified path instead of expecting a remote to be set
* up in the repo already (it isn't by default)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/sdk.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index afe52cf..daa3e4e 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -156,11 +156,11 @@ def sdk_update(args, config, basepath, workspace):
         logger.debug("Updating meta data via git ...")
         # Try using 'git pull', if failed, use 'git clone'
         if os.path.exists(os.path.join(basepath, 'layers/.git')):
-            ret = subprocess.call("cd layers && git pull", shell=True)
+            ret = subprocess.call("cd layers && git pull %s/layers/.git" % updateserver, shell=True)
         else:
             ret = -1
         if ret != 0:
-            ret = subprocess.call("rm -rf layers && git clone %s/layers" % updateserver, shell=True)
+            ret = subprocess.call("rm -rf layers && git clone %s/layers/.git" % updateserver, shell=True)
         if ret != 0:
             logger.error("Updating meta data via git failed")
             return ret
-- 
2.5.0

[PATCH 06/14] devtool: sdk-update: fix error checking

[Paul Eggleton]

Running "raise" with no arguments here is invalid, we're not in
exception handling context. Rather than also adding code to catch the
exception I just moved the check out to the parent function from which
we can just exit.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/sdk.py | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index daa3e4e..2d9d911 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -36,14 +36,9 @@ def generate_update_dict(sigfile_new, sigfile_old):
             continue
     return update_dict
 
-def get_sstate_objects(update_dict, newsdk_path):
+def get_sstate_objects(update_dict, sstate_dir):
     """Return a list containing sstate objects which are to be installed"""
     sstate_objects = []
-    # Ensure newsdk_path points to an extensible SDK
-    sstate_dir = os.path.join(newsdk_path, 'sstate-cache')
-    if not os.path.exists(sstate_dir):
-        logger.error("sstate-cache directory not found under %s" % newsdk_path)
-        raise
     for k in update_dict:
         files = set()
         hashval = update_dict[k]
@@ -112,7 +107,11 @@ def sdk_update(args, config, basepath, workspace):
             logger.debug("Found conf/locked-sigs.inc in %s" % updateserver)
         update_dict = generate_update_dict(new_locked_sig_file_path, old_locked_sig_file_path)
         logger.debug("update_dict = %s" % update_dict)
-        sstate_objects = get_sstate_objects(update_dict, updateserver)
+        sstate_dir = os.path.join(newsdk_path, 'sstate-cache')
+        if not os.path.exists(sstate_dir):
+            logger.error("sstate-cache directory not found under %s" % newsdk_path)
+            return 1
+        sstate_objects = get_sstate_objects(update_dict, sstate_dir)
         logger.debug("sstate_objects = %s" % sstate_objects)
         if len(sstate_objects) == 0:
             logger.info("No need to update.")
-- 
2.5.0

[PATCH 07/14] devtool: sdk-update: add option to skip preparation step

[Paul Eggleton]

For debugging purposes it's useful to be able to skip the preparation
step so you can inspect what the state of the build system is first.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/sdk.py | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 2d9d911..7faefab 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -173,14 +173,15 @@ def sdk_update(args, config, basepath, workspace):
         with open(os.path.join(basepath, 'conf/local.conf'), 'a') as f:
             f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % updateserver)
 
-    # Run bitbake command for the whole SDK
-    sdk_targets = config.get('SDK', 'sdk_targets')
-    logger.info("Executing 'bitbake %s' ... (This may take some time.)" % sdk_targets)
-    try:
-        exec_build_env_command(config.init_path, basepath, 'bitbake %s' % sdk_targets)
-    except:
-        logger.error('bitbake %s failed' % sdk_targets)
-        return -1
+    if not args.skip_prepare:
+        # Run bitbake command for the whole SDK
+        sdk_targets = config.get('SDK', 'sdk_targets')
+        logger.info("Preparing build system... (This may take some time.)")
+        try:
+            exec_build_env_command(config.init_path, basepath, 'bitbake %s' % sdk_targets)
+        except:
+            logger.error('bitbake %s failed' % sdk_targets)
+            return -1
     return 0
 
 def register_commands(subparsers, context):
@@ -188,4 +189,5 @@ def register_commands(subparsers, context):
     if context.fixed_setup:
         parser_sdk = subparsers.add_parser('sdk-update', help='Update SDK components from a nominated location')
         parser_sdk.add_argument('updateserver', help='The update server to fetch latest SDK components from', nargs='?')
+        parser_sdk.add_argument('--skip-prepare', action="store_true", help='Skip re-preparing the build system after updating (for debugging only)')
         parser_sdk.set_defaults(func=sdk_update)
-- 
2.5.0

[PATCH 08/14] recipetool: create: lower case name when determining from filename

[Paul Eggleton]

As a matter of general convention we expect recipe names to be lower
case; in fact some of the packaging backends insist upon it. Since in
this part of the code we're auto-determining the name, we should convert
the name to lowercase if it's not already so that we're following
convention and avoiding any validation issues later on.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 873b7dd..f342857 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -90,9 +90,9 @@ def determine_from_filename(srcfile):
     """Determine name and version from a filename"""
     part = ''
     if '.tar.' in srcfile:
-        namepart = srcfile.split('.tar.')[0]
+        namepart = srcfile.split('.tar.')[0].lower()
     else:
-        namepart = os.path.splitext(srcfile)[0]
+        namepart = os.path.splitext(srcfile)[0].lower()
     splitval = namepart.rsplit('_', 1)
     if len(splitval) == 1:
         splitval = namepart.rsplit('-', 1)
-- 
2.5.0

[PATCH 09/14] recipetool: create: support creating standalone native/nativesdk recipes

[Paul Eggleton]

If the recipe name ends with -native then we should inherit native;
likewise if it starts with nativesdk- then inherit nativesdk.

(Note that the recipe name must actually be specified by the user in
order to trigger this - we won't do it based on any name auto-detected
from e.g. the tarball name.)

Since we're doing this based on the name, "devtool add" will also gain
this functionality automatically.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/recipetool/create.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f342857..775be42 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -232,6 +232,8 @@ def create_recipe(args):
     lines_before.append('LIC_FILES_CHKSUM = "%s"' % ' \\\n                    '.join(lic_files_chksum))
     lines_before.append('')
 
+    classes = []
+
     # FIXME This is kind of a hack, we probably ought to be using bitbake to do this
     pn = None
     pv = None
@@ -249,6 +251,10 @@ def create_recipe(args):
 
     if args.name:
         pn = args.name
+        if args.name.endswith('-native'):
+            classes.append('native')
+        elif args.name.startswith('nativesdk-'):
+            classes.append('nativesdk')
 
     if pv and pv not in 'git svn hg'.split():
         realpv = pv
@@ -312,7 +318,6 @@ def create_recipe(args):
     handlers = [item[0] for item in handlers]
 
     # Apply the handlers
-    classes = []
     handled = []
 
     if args.binary:
-- 
2.5.0

[PATCH 10/14] devtool: reset: do clean for multiple recipes at once with -a

[Paul Eggleton]

We need to run the clean for all recipes that are being reset before we
start deleting things from the workspace; if we don't, recipes providing
dependencies may be missing when we come to clean a recipe later (since
we don't and couldn't practically reset them in dependency order). This
also improves performance since we have the startup startup time for the
clean just once rather than for every recipe.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py | 25 ++++++++++++++-----------
 1 file changed, 14 insertions(+), 11 deletions(-)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index fbc49a9..741f0ea 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1196,21 +1196,24 @@ def reset(args, config, basepath, workspace):
         raise DevtoolError("Recipe must be specified, or specify -a/--all to "
                            "reset all recipes")
     if args.all:
-        recipes = workspace
+        recipes = workspace.keys()
     else:
         recipes = [args.recipename]
 
-    for pn in recipes:
-        if not args.no_clean:
-            logger.info('Cleaning sysroot for recipe %s...' % pn)
-            try:
-                exec_build_env_command(config.init_path, basepath, 'bitbake -c clean %s' % pn)
-            except bb.process.ExecutionError as e:
-                raise DevtoolError('Command \'%s\' failed, output:\n%s\nIf you '
-                                   'wish, you may specify -n/--no-clean to '
-                                   'skip running this command when resetting' %
-                                   (e.command, e.stdout))
+    if recipes and not args.no_clean:
+        if len(recipes) == 1:
+            logger.info('Cleaning sysroot for recipe %s...' % recipes[0])
+        else:
+            logger.info('Cleaning sysroot for recipes %s...' % ', '.join(recipes))
+        try:
+            exec_build_env_command(config.init_path, basepath, 'bitbake -c clean %s' % ' '.join(recipes))
+        except bb.process.ExecutionError as e:
+            raise DevtoolError('Command \'%s\' failed, output:\n%s\nIf you '
+                                'wish, you may specify -n/--no-clean to '
+                                'skip running this command when resetting' %
+                                (e.command, e.stdout))
 
+    for pn in recipes:
         _check_preserve(config, pn)
 
         preservepath = os.path.join(config.workspace_path, 'attic', pn)
-- 
2.5.0

[PATCH 11/14] devtool: add: support adding a native variant

[Paul Eggleton]

Sometimes you need to build a variant of a recipe for the build
host as well as for the target (i.e. BBCLASSEXTEND = "native"); add a
 --also-native command line option to "recipetool create" that enables
this and plumb it through from an identical option for "devtool add".

(We could conceivably do the same for nativesdk, but I felt it might be
confusing within the context of the extensible SDK, where nativesdk
isn't really relevant to the user.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/standard.py  |  3 +++
 scripts/lib/recipetool/create.py | 23 +++++++++++++++++++++++
 2 files changed, 26 insertions(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 741f0ea..7ef0ab8 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -139,6 +139,8 @@ def add(args, config, basepath, workspace):
         extracmdopts += ' -V %s' % args.version
     if args.binary:
         extracmdopts += ' -b'
+    if args.also_native:
+        extracmdopts += ' --also-native'
 
     tempdir = tempfile.mkdtemp(prefix='devtool')
     try:
@@ -1307,6 +1309,7 @@ def register_commands(subparsers, context):
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
     parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true")
     parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure). Useful with binary packages e.g. RPMs.', action='store_true')
+    parser_add.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
     parser_add.set_defaults(func=add)
 
     parser_modify = subparsers.add_parser('modify', help='Modify the source for an existing recipe',
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 775be42..4f95d7e 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -252,8 +252,14 @@ def create_recipe(args):
     if args.name:
         pn = args.name
         if args.name.endswith('-native'):
+            if args.also_native:
+                logger.error('--also-native cannot be specified for a recipe named *-native (*-native denotes a recipe that is already only for native) - either remove the -native suffix from the name or drop --also-native')
+                sys.exit(1)
             classes.append('native')
         elif args.name.startswith('nativesdk-'):
+            if args.also_native:
+                logger.error('--also-native cannot be specified for a recipe named nativesdk-* (nativesdk-* denotes a recipe that is already only for nativesdk)')
+                sys.exit(1)
             classes.append('nativesdk')
 
     if pv and pv not in 'git svn hg'.split():
@@ -393,6 +399,22 @@ def create_recipe(args):
                 line = re.sub('"[^+]*\+', '"%s+' % realpv, line)
         lines_before.append(line)
 
+    if args.also_native:
+        lines = lines_after
+        lines_after = []
+        bbclassextend = None
+        for line in lines:
+            if line.startswith('BBCLASSEXTEND ='):
+                splitval = line.split('"')
+                if len(splitval) > 1:
+                    bbclassextend = splitval[1].split()
+                    if not 'native' in bbclassextend:
+                        bbclassextend.insert(0, 'native')
+                line = 'BBCLASSEXTEND = "%s"' % ' '.join(bbclassextend)
+            lines_after.append(line)
+        if not bbclassextend:
+            lines_after.append('BBCLASSEXTEND = "native"')
+
     outlines = []
     outlines.extend(lines_before)
     if classes:
@@ -591,5 +613,6 @@ def register_commands(subparsers):
     parser_create.add_argument('-N', '--name', help='Name to use within recipe (PN)')
     parser_create.add_argument('-V', '--version', help='Version to use within recipe (PV)')
     parser_create.add_argument('-b', '--binary', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true')
+    parser_create.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
     parser_create.set_defaults(func=create_recipe)
 
-- 
2.5.0

[PATCH 12/14] devtool: refactor code for getting local recipe file

[Paul Eggleton]

We're doing this in a couple of places, let's just find the recipe file
if it exists within the workspace (which it will if it's been added
through "devtool add") when we read in the workspace.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/devtool                 | 12 +++++++++++-
 scripts/lib/devtool/standard.py | 19 ++++---------------
 2 files changed, 15 insertions(+), 16 deletions(-)

diff --git a/scripts/devtool b/scripts/devtool
index bda05e1..2d57da0b 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -112,8 +112,18 @@ def read_workspace():
                 res = externalsrc_re.match(line.rstrip())
                 if res:
                     pn = res.group(2) or os.path.splitext(os.path.basename(fn))[0].split('_')[0]
+                    # Find the recipe file within the workspace, if any
+                    bbfile = os.path.basename(fn).replace('.bbappend', '.bb').replace('%', '*')
+                    recipefile = glob.glob(os.path.join(config.workspace_path,
+                                                        'recipes',
+                                                        pn,
+                                                        bbfile))
+                    if recipefile:
+                        recipefile = recipefile[0]
                     workspace[pn] = {'srctree': res.group(3),
-                                     'bbappend': fn}
+                                     'bbappend': fn,
+                                     'recipefile': recipefile}
+                    logger.debug('Found recipe %s' % workspace[pn])
 
 def create_workspace(args, config, basepath, workspace):
     if args.layerpath:
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 7ef0ab8..e26ce5a 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1171,13 +1171,9 @@ def status(args, config, basepath, workspace):
     """Entry point for the devtool 'status' subcommand"""
     if workspace:
         for recipe, value in workspace.iteritems():
-            bbfile = os.path.basename(value['bbappend']).replace('.bbappend', '.bb').replace('%', '*')
-            recipefile = glob.glob(os.path.join(config.workspace_path,
-                                                'recipes',
-                                                recipe,
-                                                bbfile))
+            recipefile = value['recipefile']
             if recipefile:
-                recipestr = ' (%s)' % recipefile[0]
+                recipestr = ' (%s)' % recipefile
             else:
                 recipestr = ''
             print("%s: %s%s" % (recipe, value['srctree'], recipestr))
@@ -1261,15 +1257,8 @@ def edit_recipe(args, config, basepath, workspace):
             tinfoil.shutdown()
     else:
         check_workspace_recipe(workspace, args.recipename)
-        bbappend = workspace[args.recipename]['bbappend']
-        bbfile = os.path.basename(bbappend).replace('.bbappend', '.bb').replace('%', '*')
-        recipefile = glob.glob(os.path.join(config.workspace_path,
-                                            'recipes',
-                                            args.recipename,
-                                            bbfile))
-        if recipefile:
-            recipefile = recipefile[0]
-        else:
+        recipefile = workspace[args.recipename]['recipefile']
+        if not recipefile:
             raise DevtoolError("Recipe file for %s is not under the workspace" %
                                args.recipename)
 
-- 
2.5.0

[PATCH 13/14] devtool: reset: support recipes with BBCLASSEXTEND

[Paul Eggleton]

If the recipe file itself was created in the workspace, and it uses
BBCLASSEXTEND (e.g. through devtool add --also-native), then we need to
clean the other variants as well.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/__init__.py | 24 ++++++++++++++++++++++++
 scripts/lib/devtool/standard.py | 13 +++++++++++--
 2 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 7f16e17..253e4d5 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -214,3 +214,27 @@ def recipe_to_append(recipefile, config, wildcard=False):
     appendpath = os.path.join(config.workspace_path, 'appends')
     appendfile = os.path.join(appendpath, appendname + '.bbappend')
     return appendfile
+
+def get_bbclassextend_targets(recipefile, pn):
+    """
+    Cheap function to get BBCLASSEXTEND and then convert that to the
+    list of targets that would result.
+    """
+    import bb.utils
+
+    values = {}
+    def get_bbclassextend_varfunc(varname, origvalue, op, newlines):
+        values[varname] = origvalue
+        return origvalue, None, 0, True
+    with open(recipefile, 'r') as f:
+        bb.utils.edit_metadata(f, ['BBCLASSEXTEND'], get_bbclassextend_varfunc)
+
+    targets = []
+    bbclassextend = values.get('BBCLASSEXTEND', '').split()
+    if bbclassextend:
+        for variant in bbclassextend:
+            if variant == 'nativesdk':
+                targets.append('%s-%s' % (variant, pn))
+            elif variant in ['native', 'cross', 'crosssdk']:
+                targets.append('%s-%s' % (pn, variant))
+    return targets
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index e26ce5a..5390f51 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -28,7 +28,7 @@ import scriptutils
 import errno
 import glob
 from collections import OrderedDict
-from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, DevtoolError
 from devtool import parse_recipe
 
 logger = logging.getLogger('devtool')
@@ -1203,8 +1203,17 @@ def reset(args, config, basepath, workspace):
             logger.info('Cleaning sysroot for recipe %s...' % recipes[0])
         else:
             logger.info('Cleaning sysroot for recipes %s...' % ', '.join(recipes))
+        # If the recipe file itself was created in the workspace, and
+        # it uses BBCLASSEXTEND, then we need to also clean the other
+        # variants
+        targets = []
+        for recipe in recipes:
+            targets.append(recipe)
+            recipefile = workspace[recipe]['recipefile']
+            if recipefile:
+                targets.extend(get_bbclassextend_targets(recipefile, recipe))
         try:
-            exec_build_env_command(config.init_path, basepath, 'bitbake -c clean %s' % ' '.join(recipes))
+            exec_build_env_command(config.init_path, basepath, 'bitbake -c clean %s' % ' '.join(targets))
         except bb.process.ExecutionError as e:
             raise DevtoolError('Command \'%s\' failed, output:\n%s\nIf you '
                                 'wish, you may specify -n/--no-clean to '
-- 
2.5.0

[PATCH 14/14] devtool: build: support using BBCLASSEXTENDed names

[Paul Eggleton]

It's logical that you would want to build BBCLASSEXTENDed items
separately through devtool build, so simply allow that - we're just
passing the name verbatim to bitbake, so all it means is adjusting the
validation.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/lib/devtool/__init__.py | 26 +++++++++++++++++++++-----
 scripts/lib/devtool/build.py    |  4 ++--
 2 files changed, 23 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 253e4d5..0405d22 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -146,19 +146,35 @@ def parse_recipe(config, tinfoil, pn, appends):
     return oe.recipeutils.parse_recipe(recipefile, append_files,
                                        tinfoil.config_data)
 
-def check_workspace_recipe(workspace, pn, checksrc=True):
+def check_workspace_recipe(workspace, pn, checksrc=True, bbclassextend=False):
     """
     Check that a recipe is in the workspace and (optionally) that source
     is present.
     """
-    if not pn in workspace:
+
+    workspacepn = pn
+
+    for recipe, value in workspace.iteritems():
+        if recipe == pn:
+            break
+        if bbclassextend:
+            recipefile = value['recipefile']
+            if recipefile:
+                targets = get_bbclassextend_targets(recipefile, recipe)
+                if pn in targets:
+                    workspacepn = recipe
+                    break
+    else:
         raise DevtoolError("No recipe named '%s' in your workspace" % pn)
+
     if checksrc:
-        srctree = workspace[pn]['srctree']
+        srctree = workspace[workspacepn]['srctree']
         if not os.path.exists(srctree):
-            raise DevtoolError("Source tree %s for recipe %s does not exist" % (srctree, pn))
+            raise DevtoolError("Source tree %s for recipe %s does not exist" % (srctree, workspacepn))
         if not os.listdir(srctree):
-            raise DevtoolError("Source tree %s for recipe %s is empty" % (srctree, pn))
+            raise DevtoolError("Source tree %s for recipe %s is empty" % (srctree, workspacepn))
+
+    return workspacepn
 
 def use_external_build(same_dir, no_same_dir, d):
     """
diff --git a/scripts/lib/devtool/build.py b/scripts/lib/devtool/build.py
index a9a0778..c4c0c9f 100644
--- a/scripts/lib/devtool/build.py
+++ b/scripts/lib/devtool/build.py
@@ -51,11 +51,11 @@ def _get_build_task(config):
 
 def build(args, config, basepath, workspace):
     """Entry point for the devtool 'build' subcommand"""
-    check_workspace_recipe(workspace, args.recipename)
+    workspacepn = check_workspace_recipe(workspace, args.recipename, bbclassextend=True)
 
     build_task = _get_build_task(config)
 
-    bbappend = workspace[args.recipename]['bbappend']
+    bbappend = workspace[workspacepn]['bbappend']
     if args.disable_parallel_make:
         logger.info("Disabling 'make' parallelism")
         _set_file_values(bbappend, {'PARALLEL_MAKE': ''})
-- 
2.5.0