* [PATCH 01/14] classes/populate_sdk_ext: add option to bring in pkgdata for world
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 11:59 ` [PATCH 02/14] devtool: build: ensure pkgdata is written out Paul Eggleton
` (12 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
Add a variable SDK_INCLUDE_PKGDATA which you can set to "1" to include
pkgdata for all recipes in the world target. There are a couple of uses
for this:
1) If you use "devtool add" to add a recipe that builds something which
depends on anything in world, the dependency can then be correctly
mapped to the recipe providing it and that recipe can be added to
DEPENDS, since we have the pkg-config and shared library dependency
data within pkgdata.
2) You'll be able to search for these recipes and any files they
package for the target with "devtool search" since that also uses
pkgdata
This of course assumes you've tailored world through EXCLUDE_FROM_WORLD
to only include recipes you'd want built in your distro, but I think
that's a reasonable assumption; failing that there is a
WORLD_PKGDATA_EXCLUDE variable that you can set to exclude any recipes
you don't want.
Note that this patch relies on functionality implemented in a recent
BitBake patch and will not work without it.
Implements [YOCTO #8600].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/populate_sdk_ext.bbclass | 24 +++++++++++-
meta/lib/oe/copy_buildsystem.py | 58 +++++++++++++++++++++++++++-
meta/recipes-core/meta/meta-world-pkgdata.bb | 50 ++++++++++++++++++++++++
scripts/lib/devtool/sdk.py | 6 +--
4 files changed, 131 insertions(+), 7 deletions(-)
create mode 100644 meta/recipes-core/meta/meta-world-pkgdata.bb
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index e5e55b3..ebb57bb 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -24,6 +24,7 @@ SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
SDK_UPDATE_URL ?= ""
SDK_TARGETS ?= "${PN}"
+SDK_INSTALL_TARGETS = "${SDK_TARGETS} ${@'meta-world-pkgdata:do_allpackagedata' if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1' else ''}"
OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
# The files from COREBASE that you want preserved in the COREBASE copied
@@ -45,6 +46,7 @@ SDK_TITLE_task-populate-sdk-ext = "${@d.getVar('DISTRO_NAME', True) or d.getVar(
python copy_buildsystem () {
import re
+ import shutil
import oe.copy_buildsystem
oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True)
@@ -91,6 +93,7 @@ python copy_buildsystem () {
config.set('General', 'core_meta_subdir', core_meta_subdir)
config.add_section('SDK')
config.set('SDK', 'sdk_targets', d.getVar('SDK_TARGETS', True))
+ config.set('SDK', 'sdk_update_targets', d.getVar('SDK_INSTALL_TARGETS', True))
updateurl = d.getVar('SDK_UPDATE_URL', True)
if updateurl:
config.set('SDK', 'updateserver', updateurl)
@@ -199,6 +202,22 @@ python copy_buildsystem () {
d.getVar('SSTATE_DIR', True),
sstate_out, d,
fixedlsbstring)
+
+ # Add packagedata if enabled
+ if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1':
+ lockedsigs_base = d.getVar('WORKDIR', True) + '/locked-sigs-base.inc'
+ lockedsigs_copy = d.getVar('WORKDIR', True) + '/locked-sigs-copy.inc'
+ shutil.move(lockedsigs_pruned, lockedsigs_base)
+ oe.copy_buildsystem.merge_lockedsigs(['do_packagedata'],
+ lockedsigs_base,
+ d.getVar('STAGING_DIR_HOST', True) + '/world-pkgdata/locked-sigs-pkgdata.inc',
+ lockedsigs_pruned,
+ lockedsigs_copy)
+ oe.copy_buildsystem.create_locked_sstate_cache(lockedsigs_copy,
+ d.getVar('SSTATE_DIR', True),
+ sstate_out, d,
+ fixedlsbstring)
+
# We don't need sstate do_package files
for root, dirs, files in os.walk(sstate_out):
for name in files:
@@ -268,7 +287,7 @@ sdk_ext_postinst() {
# current working directory when first ran, nor will it set $1 when
# sourcing a script. That is why this has to look so ugly.
LOGFILE="$target_sdk_dir/preparing_build_system.log"
- sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py '${SDK_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
+ sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py '${SDK_INSTALL_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
fi
echo done
}
@@ -314,7 +333,8 @@ def get_sdk_ext_rdepends(d):
do_populate_sdk_ext[dirs] = "${@d.getVarFlag('do_populate_sdk', 'dirs', False)}"
do_populate_sdk_ext[depends] = "${@d.getVarFlag('do_populate_sdk', 'depends', False)} \
- buildtools-tarball:do_populate_sdk uninative-tarball:do_populate_sdk"
+ buildtools-tarball:do_populate_sdk uninative-tarball:do_populate_sdk \
+ ${@'meta-world-pkgdata:do_collect_packagedata' if d.getVar('SDK_INCLUDE_PKGDATA', True) == '1' else ''}"
do_populate_sdk_ext[rdepends] += "${@' '.join([x + ':do_build' for x in d.getVar('SDK_TARGETS', True).split()])}"
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index a5ca3df..6475510 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -93,10 +93,64 @@ def prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, pruned_output
invalue = True
f.write(line)
+def merge_lockedsigs(copy_tasks, lockedsigs_main, lockedsigs_extra, merged_output, copy_output):
+ merged = {}
+ arch_order = []
+ with open(lockedsigs_main, 'r') as f:
+ invalue = None
+ for line in f:
+ if invalue:
+ if line.endswith('\\\n'):
+ merged[invalue].append(line)
+ else:
+ invalue = None
+ elif line.startswith('SIGGEN_LOCKEDSIGS_t-'):
+ invalue = line[18:].split('=', 1)[0].rstrip()
+ merged[invalue] = []
+ arch_order.append(invalue)
+
+ with open(lockedsigs_extra, 'r') as f:
+ invalue = None
+ tocopy = {}
+ for line in f:
+ if invalue:
+ if line.endswith('\\\n'):
+ if not line in merged[invalue]:
+ target, task = line.strip().split(':')[:2]
+ if task in copy_tasks:
+ tocopy[invalue].append(line)
+ merged[invalue].append(line)
+ else:
+ invalue = None
+ elif line.startswith('SIGGEN_LOCKEDSIGS_t-'):
+ invalue = line[18:].split('=', 1)[0].rstrip()
+ if not invalue in merged:
+ merged[invalue] = []
+ arch_order.append(invalue)
+ tocopy[invalue] = []
+
+ def write_sigs_file(fn, types, sigs):
+ fulltypes = []
+ bb.utils.mkdirhier(os.path.dirname(fn))
+ with open(fn, 'w') as f:
+ for typename in types:
+ lines = sigs[typename]
+ if lines:
+ f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % typename)
+ for line in lines:
+ f.write(line)
+ f.write(' "\n')
+ fulltypes.append(typename)
+ f.write('SIGGEN_LOCKEDSIGS_TYPES = "%s"\n' % ' '.join(fulltypes))
+
+ write_sigs_file(copy_output, tocopy.keys(), tocopy)
+ write_sigs_file(merged_output, arch_order, merged)
+
def create_locked_sstate_cache(lockedsigs, input_sstate_cache, output_sstate_cache, d, fixedlsbstring=""):
bb.note('Generating sstate-cache...')
bb.process.run("gen-lockedsig-cache %s %s %s" % (lockedsigs, input_sstate_cache, output_sstate_cache))
if fixedlsbstring:
- os.rename(output_sstate_cache + '/' + d.getVar('NATIVELSBSTRING', True),
- output_sstate_cache + '/' + fixedlsbstring)
+ nativedir = output_sstate_cache + '/' + d.getVar('NATIVELSBSTRING', True)
+ if os.path.isdir(nativedir):
+ os.rename(nativedir, output_sstate_cache + '/' + fixedlsbstring)
diff --git a/meta/recipes-core/meta/meta-world-pkgdata.bb b/meta/recipes-core/meta/meta-world-pkgdata.bb
new file mode 100644
index 0000000..48a9027
--- /dev/null
+++ b/meta/recipes-core/meta/meta-world-pkgdata.bb
@@ -0,0 +1,50 @@
+SUMMARY = "Pulls in pkgdata for world"
+LICENSE = "MIT"
+INHIBIT_DEFAULT_DEPS = "1"
+
+addtask do_allpackagedata before do_build
+do_allpackagedata() {
+ :
+}
+do_allpackagedata[recrdeptask] = "do_packagedata do_allpackagedata"
+do_allpackagedata[noexec] = "1"
+
+WORLD_PKGDATADIR = "${D}/world-pkgdata"
+
+addtask do_collect_packagedata after do_allpackagedata
+SSTATETASKS += "do_collect_packagedata"
+do_collect_packagedata[sstate-inputdirs] = "${WORLD_PKGDATADIR}"
+do_collect_packagedata[sstate-outputdirs] = "${STAGING_DIR_HOST}/world-pkgdata"
+
+python do_collect_packagedata() {
+ import oe.copy_buildsystem
+ outdir = os.path.join(d.getVar('WORLD_PKGDATADIR', True))
+ bb.utils.mkdirhier(outdir)
+ sigfile = os.path.join(outdir, 'locked-sigs-pkgdata.inc')
+ oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
+}
+
+do_fetch[noexec] = "1"
+do_unpack[noexec] = "1"
+do_patch[noexec] = "1"
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_install[noexec] = "1"
+
+do_configure[deptask] = ""
+
+WORLD_PKGDATA_EXCLUDE ?= "adt-installer"
+
+python calculate_extra_depends() {
+ exclude = '${WORLD_PKGDATA_EXCLUDE}'.split()
+ for p in world_target:
+ if p == self_pn:
+ continue
+
+ if p in exclude:
+ continue
+
+ deps.append(p)
+}
+
+PACKAGES = ""
diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 0872df6..68139aa 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -175,12 +175,12 @@ def sdk_update(args, config, basepath, workspace):
if not args.skip_prepare:
# Run bitbake command for the whole SDK
- sdk_targets = config.get('SDK', 'sdk_targets')
+ sdk_update_targets = config.get('SDK', 'sdk_update_targets', config.get('SDK', 'sdk_targets'))
logger.info("Preparing build system... (This may take some time.)")
try:
- exec_build_env_command(config.init_path, basepath, 'bitbake %s --setscene-only' % sdk_targets)
+ exec_build_env_command(config.init_path, basepath, 'bitbake %s --setscene-only' % sdk_update_targets)
except:
- logger.error('bitbake %s failed' % sdk_targets)
+ logger.error('bitbake %s failed' % sdk_update_targets)
return -1
return 0
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 02/14] devtool: build: ensure pkgdata is written out
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
2016-01-22 11:59 ` [PATCH 01/14] classes/populate_sdk_ext: add option to bring in pkgdata for world Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 11:59 ` [PATCH 03/14] devtool: sdk-update: improve temp directory handling Paul Eggleton
` (11 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
When you run devtool build, you need to have the pkgdata written out at
the end, so that if what you're adding is a library and the next thing
you add is something that depends on that library, the necessary
information to map the dependency back to the recipe is present. In
practical terms all this means is we need do_packagedata to run in
addition to do_populate_sysroot.
This does mean that do_package needs to run which wasn't running before,
and that means that the few package QA tests that run within do_package
such as installed-vs-shipped will now be run. This may be a bit
bothersome, and prompted a fix for one of our oe-selftest tests as a
result, but I don't see an easy way around it. Ultimately if you care
about using the recipe in an image you'll need to fix any such errors
anyway.
Fixes [YOCTO #8887].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oeqa/selftest/devtool.py | 7 ++++++-
scripts/lib/devtool/build.py | 16 +++++++++++-----
2 files changed, 17 insertions(+), 6 deletions(-)
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 43c7cda..d95cb08 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -226,8 +226,13 @@ class DevtoolTests(DevtoolBase):
# Clean up anything in the workdir/sysroot/sstate cache (have to do this *after* devtool add since the recipe only exists then)
bitbake('libftdi -c cleansstate')
# libftdi's python/CMakeLists.txt is a bit broken, so let's just disable it
+ # There's also the matter of it installing cmake files to a path we don't
+ # normally cover, which triggers the installed-vs-shipped QA test we have
+ # within do_package
recipefile = '%s/recipes/libftdi/libftdi_%s.bb' % (self.workspacedir, version)
- result = runCmd('recipetool setvar %s EXTRA_OECMAKE -- "-DPYTHON_BINDINGS=OFF"' % recipefile)
+ result = runCmd('recipetool setvar %s EXTRA_OECMAKE -- \'-DPYTHON_BINDINGS=OFF -DLIBFTDI_CMAKE_CONFIG_DIR=${datadir}/cmake/Modules\'' % recipefile)
+ with open(recipefile, 'a') as f:
+ f.write('\nFILES_${PN}-dev += "${datadir}/cmake/Modules"\n')
# Test devtool build
result = runCmd('devtool build libftdi')
staging_libdir = get_bb_var('STAGING_LIBDIR', 'libftdi')
diff --git a/scripts/lib/devtool/build.py b/scripts/lib/devtool/build.py
index c4c0c9f..b10a6a9 100644
--- a/scripts/lib/devtool/build.py
+++ b/scripts/lib/devtool/build.py
@@ -46,21 +46,27 @@ def _set_file_values(fn, values):
f.writelines(newlines)
return updated
-def _get_build_task(config):
- return config.get('Build', 'build_task', 'populate_sysroot')
+def _get_build_tasks(config):
+ tasks = config.get('Build', 'build_task', 'populate_sysroot,packagedata').split(',')
+ return ['do_%s' % task.strip() for task in tasks]
def build(args, config, basepath, workspace):
"""Entry point for the devtool 'build' subcommand"""
workspacepn = check_workspace_recipe(workspace, args.recipename, bbclassextend=True)
- build_task = _get_build_task(config)
+ build_tasks = _get_build_tasks(config)
bbappend = workspace[workspacepn]['bbappend']
if args.disable_parallel_make:
logger.info("Disabling 'make' parallelism")
_set_file_values(bbappend, {'PARALLEL_MAKE': ''})
try:
- exec_build_env_command(config.init_path, basepath, 'bitbake -c %s %s' % (build_task, args.recipename), watch=True)
+ bbargs = []
+ for task in build_tasks:
+ if args.recipename.endswith('-native') and 'package' in task:
+ continue
+ bbargs.append('%s:%s' % (args.recipename, task))
+ exec_build_env_command(config.init_path, basepath, 'bitbake %s' % ' '.join(bbargs), watch=True)
except bb.process.ExecutionError as e:
# We've already seen the output since watch=True, so just ensure we return something to the user
return e.exitcode
@@ -73,7 +79,7 @@ def build(args, config, basepath, workspace):
def register_commands(subparsers, context):
"""Register devtool subcommands from this plugin"""
parser_build = subparsers.add_parser('build', help='Build a recipe',
- description='Builds the specified recipe using bitbake (up to and including do_%s)' % _get_build_task(context.config))
+ description='Builds the specified recipe using bitbake (up to and including %s)' % ', '.join(_get_build_tasks(context.config)))
parser_build.add_argument('recipename', help='Recipe to build')
parser_build.add_argument('-s', '--disable-parallel-make', action="store_true", help='Disable make parallelism')
parser_build.set_defaults(func=build)
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 03/14] devtool: sdk-update: improve temp directory handling
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
2016-01-22 11:59 ` [PATCH 01/14] classes/populate_sdk_ext: add option to bring in pkgdata for world Paul Eggleton
2016-01-22 11:59 ` [PATCH 02/14] devtool: build: ensure pkgdata is written out Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 11:59 ` [PATCH 04/14] devtool: sdk-update: improve SDK update process robustness Paul Eggleton
` (10 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
* Use tempfile.mkdtemp() instead of hardcoding temp dir
* Set a variable early for the temp locked sigs file and use that
everywhere
* Delete the temp dir at the end
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/sdk.py | 75 +++++++++++++++++++++++-----------------------
1 file changed, 38 insertions(+), 37 deletions(-)
diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 68139aa..4fcd36a 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -7,6 +7,7 @@ import glob
import shutil
import errno
import sys
+import tempfile
from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
logger = logging.getLogger('devtool')
@@ -133,45 +134,45 @@ def sdk_update(args, config, basepath, workspace):
return ret
else:
# devtool sdk-update http://myhost/sdk
- tmpsdk_dir = '/tmp/sdk-ext'
- if os.path.exists(tmpsdk_dir):
- shutil.rmtree(tmpsdk_dir)
- os.makedirs(tmpsdk_dir)
- os.makedirs(os.path.join(tmpsdk_dir, 'conf'))
- # Fetch locked-sigs.inc from update server
- ret = subprocess.call("wget -q -O - %s/conf/locked-sigs.inc > %s/locked-sigs.inc" % (updateserver, os.path.join(tmpsdk_dir, 'conf')), shell=True)
- if ret != 0:
- logger.error("Fetching conf/locked-sigs.inc from %s to %s/locked-sigs.inc failed" % (updateserver, os.path.join(tmpsdk_dir, 'conf')))
- return ret
- else:
- logger.info("Fetching conf/locked-sigs.inc from %s to %s/locked-sigs.inc succeeded" % (updateserver, os.path.join(tmpsdk_dir, 'conf')))
- new_locked_sig_file_path = os.path.join(tmpsdk_dir, 'conf/locked-sigs.inc')
- update_dict = generate_update_dict(new_locked_sig_file_path, old_locked_sig_file_path)
- logger.debug("update_dict = %s" % update_dict)
- if len(update_dict) == 0:
- logger.info("No need to update.")
- return 0
- # Update metadata
- logger.debug("Updating meta data via git ...")
- # Try using 'git pull', if failed, use 'git clone'
- if os.path.exists(os.path.join(basepath, 'layers/.git')):
- ret = subprocess.call("cd layers && git pull %s/layers/.git" % updateserver, shell=True)
- else:
- ret = -1
- if ret != 0:
- ret = subprocess.call("rm -rf layers && git clone %s/layers/.git" % updateserver, shell=True)
- if ret != 0:
- logger.error("Updating meta data via git failed")
- return ret
- logger.debug("Updating conf files ...")
- conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'locked-sigs.inc']
- for conf in conf_files:
- ret = subprocess.call("wget -q -O - %s/conf/%s > conf/%s" % (updateserver, conf, conf), shell=True)
+ tmpsdk_dir = tempfile.mkdtemp()
+ try:
+ os.makedirs(os.path.join(tmpsdk_dir, 'conf'))
+ new_locked_sig_file_path = os.path.join(tmpsdk_dir, 'conf', 'locked-sigs.inc')
+ # Fetch locked-sigs.inc from update server
+ ret = subprocess.call("wget -q -O - %s/conf/locked-sigs.inc > %s" % (updateserver, new_locked_sig_file_path), shell=True)
if ret != 0:
- logger.error("Update %s failed" % conf)
+ logger.error("Fetching conf/locked-sigs.inc from %s to %s failed" % (updateserver, new_locked_sig_file_path))
return ret
- with open(os.path.join(basepath, 'conf/local.conf'), 'a') as f:
- f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % updateserver)
+ else:
+ logger.info("Fetching conf/locked-sigs.inc from %s to %s succeeded" % (updateserver, new_locked_sig_file_path))
+ update_dict = generate_update_dict(new_locked_sig_file_path, old_locked_sig_file_path)
+ logger.debug("update_dict = %s" % update_dict)
+ if len(update_dict) == 0:
+ logger.info("No need to update.")
+ return 0
+ # Update metadata
+ logger.debug("Updating meta data via git ...")
+ # Try using 'git pull', if failed, use 'git clone'
+ if os.path.exists(os.path.join(basepath, 'layers/.git')):
+ ret = subprocess.call("cd layers && git pull %s/layers/.git" % updateserver, shell=True)
+ else:
+ ret = -1
+ if ret != 0:
+ ret = subprocess.call("rm -rf layers && git clone %s/layers/.git" % updateserver, shell=True)
+ if ret != 0:
+ logger.error("Updating meta data via git failed")
+ return ret
+ logger.debug("Updating conf files ...")
+ conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'locked-sigs.inc']
+ for conf in conf_files:
+ ret = subprocess.call("wget -q -O - %s/conf/%s > conf/%s" % (updateserver, conf, conf), shell=True)
+ if ret != 0:
+ logger.error("Update %s failed" % conf)
+ return ret
+ with open(os.path.join(basepath, 'conf/local.conf'), 'a') as f:
+ f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % updateserver)
+ finally:
+ shutil.rmtree(tmpsdk_dir)
if not args.skip_prepare:
# Run bitbake command for the whole SDK
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 04/14] devtool: sdk-update: improve SDK update process robustness
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
` (2 preceding siblings ...)
2016-01-22 11:59 ` [PATCH 03/14] devtool: sdk-update: improve temp directory handling Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 11:59 ` [PATCH 05/14] devtool: add sdk-install subcommand Paul Eggleton
` (9 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
Make the following improvements to the SDK update process:
* Use a manifest file with sha256sums to track files other than sstate
and metadata that we need to update - e.g. conf files. This allows us
to handle where files such as auto.conf may or may not be present,
as well as the configuration changing without affecting task signatures
- we still want the config files copied in that case rather than it
saying nothing needs to be done.
* Write the SSTATE_MIRRORS_append to site.conf rather than local.conf
so that local.conf remains static (since we don't want to trigger an
update every time). Also, If there is an SSTATE_MIRRORS value already
set in the configuration we can skip this and assume it contains the
needed packages.
* Allow the update process to be run in any directory, don't assume
we're already at the base of the SDK
* Where practical, fetch remote files into a temporary location and
then move them to the desired location at the end, to avoid a
failed update leaving the SDK in a broken state.
* Update all installed do_populate_sysroot / do_packagedata tasks
instead of using the SDK targets. This ensures any item installed
through dependencies after installation (e.g. when running
"devtool build") won't go stale.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/populate_sdk_ext.bbclass | 16 +++++
scripts/lib/devtool/sdk.py | 118 ++++++++++++++++++++++++----------
2 files changed, 100 insertions(+), 34 deletions(-)
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index ebb57bb..bb8831b 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -47,6 +47,7 @@ SDK_TITLE_task-populate-sdk-ext = "${@d.getVar('DISTRO_NAME', True) or d.getVar(
python copy_buildsystem () {
import re
import shutil
+ import glob
import oe.copy_buildsystem
oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True)
@@ -224,6 +225,21 @@ python copy_buildsystem () {
if name.endswith("_package.tgz"):
f = os.path.join(root, name)
os.remove(f)
+
+ # Write manifest file
+ # Note: at the moment we cannot include the env setup script here to keep
+ # it updated, since it gets modified during SDK installation (see
+ # sdk_ext_postinst() below) thus the checksum we take here would always
+ # be different.
+ manifest_file_list = ['conf/*']
+ manifest_file = os.path.join(baseoutpath, 'conf', 'sdk-conf-manifest')
+ with open(manifest_file, 'w') as f:
+ for item in manifest_file_list:
+ for fn in glob.glob(os.path.join(baseoutpath, item)):
+ if fn == manifest_file:
+ continue
+ chksum = bb.utils.sha256_file(fn)
+ f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
}
def extsdk_get_buildtools_filename(d):
diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 4fcd36a..80ea871 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -75,6 +75,21 @@ def install_sstate_objects(sstate_objects, src_sdk, dest_sdk):
logger.debug("Copying %s to %s" % (sb, dst))
shutil.copy(sb, dst)
+def check_manifest(fn, basepath):
+ import bb.utils
+ changedfiles = []
+ with open(fn, 'r') as f:
+ for line in f:
+ splitline = line.split()
+ if len(splitline) > 1:
+ chksum = splitline[0]
+ fpath = splitline[1]
+ curr_chksum = bb.utils.sha256_file(os.path.join(basepath, fpath))
+ if chksum != curr_chksum:
+ logger.debug('File %s changed: old csum = %s, new = %s' % (os.path.join(basepath, fpath), curr_chksum, chksum))
+ changedfiles.append(fpath)
+ return changedfiles
+
def sdk_update(args, config, basepath, workspace):
# Fetch locked-sigs.inc file from remote/local destination
updateserver = args.updateserver
@@ -98,6 +113,18 @@ def sdk_update(args, config, basepath, workspace):
else:
is_remote = False
+ layers_dir = os.path.join(basepath, 'layers')
+ conf_dir = os.path.join(basepath, 'conf')
+
+ # Grab variable values
+ tinfoil = setup_tinfoil(config_only=True, basepath=basepath)
+ try:
+ stamps_dir = tinfoil.config_data.getVar('STAMPS_DIR', True)
+ sstate_mirrors = tinfoil.config_data.getVar('SSTATE_MIRRORS', True)
+ site_conf_version = tinfoil.config_data.getVar('SITE_CONF_VERSION', True)
+ finally:
+ tinfoil.shutdown()
+
if not is_remote:
# devtool sdk-update /local/path/to/latest/sdk
new_locked_sig_file_path = os.path.join(updateserver, 'conf/locked-sigs.inc')
@@ -121,16 +148,14 @@ def sdk_update(args, config, basepath, workspace):
install_sstate_objects(sstate_objects, updateserver.rstrip('/'), basepath)
logger.info("Updating configuration files")
new_conf_dir = os.path.join(updateserver, 'conf')
- old_conf_dir = os.path.join(basepath, 'conf')
- shutil.rmtree(old_conf_dir)
- shutil.copytree(new_conf_dir, old_conf_dir)
+ shutil.rmtree(conf_dir)
+ shutil.copytree(new_conf_dir, conf_dir)
logger.info("Updating layers")
new_layers_dir = os.path.join(updateserver, 'layers')
- old_layers_dir = os.path.join(basepath, 'layers')
- shutil.rmtree(old_layers_dir)
- ret = subprocess.call("cp -a %s %s" % (new_layers_dir, old_layers_dir), shell=True)
+ shutil.rmtree(layers_dir)
+ ret = subprocess.call("cp -a %s %s" % (new_layers_dir, layers_dir), shell=True)
if ret != 0:
- logger.error("Copying %s to %s failed" % (new_layers_dir, old_layers_dir))
+ logger.error("Copying %s to %s failed" % (new_layers_dir, layers_dir))
return ret
else:
# devtool sdk-update http://myhost/sdk
@@ -138,50 +163,75 @@ def sdk_update(args, config, basepath, workspace):
try:
os.makedirs(os.path.join(tmpsdk_dir, 'conf'))
new_locked_sig_file_path = os.path.join(tmpsdk_dir, 'conf', 'locked-sigs.inc')
- # Fetch locked-sigs.inc from update server
- ret = subprocess.call("wget -q -O - %s/conf/locked-sigs.inc > %s" % (updateserver, new_locked_sig_file_path), shell=True)
- if ret != 0:
- logger.error("Fetching conf/locked-sigs.inc from %s to %s failed" % (updateserver, new_locked_sig_file_path))
- return ret
- else:
- logger.info("Fetching conf/locked-sigs.inc from %s to %s succeeded" % (updateserver, new_locked_sig_file_path))
- update_dict = generate_update_dict(new_locked_sig_file_path, old_locked_sig_file_path)
- logger.debug("update_dict = %s" % update_dict)
- if len(update_dict) == 0:
- logger.info("No need to update.")
+ # Fetch manifest from server
+ tmpmanifest = os.path.join(tmpsdk_dir, 'conf', 'sdk-conf-manifest')
+ ret = subprocess.call("wget -q -O %s %s/conf/sdk-conf-manifest" % (tmpmanifest, updateserver), shell=True)
+ changedfiles = check_manifest(tmpmanifest, basepath)
+ if not changedfiles:
+ logger.info("Already up-to-date")
return 0
# Update metadata
- logger.debug("Updating meta data via git ...")
+ logger.debug("Updating metadata via git ...")
# Try using 'git pull', if failed, use 'git clone'
if os.path.exists(os.path.join(basepath, 'layers/.git')):
- ret = subprocess.call("cd layers && git pull %s/layers/.git" % updateserver, shell=True)
+ ret = subprocess.call("git pull %s/layers/.git" % updateserver, shell=True, cwd=layers_dir)
else:
ret = -1
if ret != 0:
- ret = subprocess.call("rm -rf layers && git clone %s/layers/.git" % updateserver, shell=True)
- if ret != 0:
- logger.error("Updating meta data via git failed")
- return ret
+ ret = subprocess.call("git clone %s/layers/.git" % updateserver, shell=True, cwd=tmpsdk_dir)
+ if ret != 0:
+ logger.error("Updating metadata via git failed")
+ return ret
logger.debug("Updating conf files ...")
- conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'locked-sigs.inc']
- for conf in conf_files:
- ret = subprocess.call("wget -q -O - %s/conf/%s > conf/%s" % (updateserver, conf, conf), shell=True)
+ for changedfile in changedfiles:
+ ret = subprocess.call("wget -q -O %s %s/%s" % (changedfile, updateserver, changedfile), shell=True, cwd=tmpsdk_dir)
if ret != 0:
- logger.error("Update %s failed" % conf)
+ logger.error("Updating %s failed" % changedfile)
return ret
- with open(os.path.join(basepath, 'conf/local.conf'), 'a') as f:
- f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % updateserver)
+
+ # Ok, all is well at this point - move everything over
+ tmplayers_dir = os.path.join(tmpsdk_dir, 'layers')
+ if os.path.exists(tmplayers_dir):
+ shutil.rmtree(layers_dir)
+ shutil.move(tmplayers_dir, layers_dir)
+ for changedfile in changedfiles:
+ destfile = os.path.join(basepath, changedfile)
+ os.remove(destfile)
+ shutil.move(os.path.join(tmpsdk_dir, changedfile), destfile)
+ os.remove(os.path.join(conf_dir, 'sdk-conf-manifest'))
+ shutil.move(tmpmanifest, conf_dir)
+
+ if not sstate_mirrors:
+ with open(os.path.join(conf_dir, 'site.conf'), 'a') as f:
+ f.write('SCONF_VERSION = "%s"\n' % site_conf_version)
+ f.write('SSTATE_MIRRORS_append = " file://.* %s/sstate-cache/PATH \\n "\n' % updateserver)
finally:
shutil.rmtree(tmpsdk_dir)
if not args.skip_prepare:
+ # Find all potentially updateable tasks
+ sdk_update_targets = []
+ tasks = ['do_populate_sysroot', 'do_packagedata']
+ for root, _, files in os.walk(stamps_dir):
+ for fn in files:
+ if not '.sigdata.' in fn:
+ for task in tasks:
+ if '.%s.' % task in fn or '.%s_setscene.' % task in fn:
+ sdk_update_targets.append('%s:%s' % (os.path.basename(root), task))
# Run bitbake command for the whole SDK
- sdk_update_targets = config.get('SDK', 'sdk_update_targets', config.get('SDK', 'sdk_targets'))
logger.info("Preparing build system... (This may take some time.)")
try:
- exec_build_env_command(config.init_path, basepath, 'bitbake %s --setscene-only' % sdk_update_targets)
- except:
- logger.error('bitbake %s failed' % sdk_update_targets)
+ exec_build_env_command(config.init_path, basepath, 'bitbake --setscene-only %s' % ' '.join(sdk_update_targets), stderr=subprocess.STDOUT)
+ output, _ = exec_build_env_command(config.init_path, basepath, 'bitbake -n %s' % ' '.join(sdk_update_targets), stderr=subprocess.STDOUT)
+ runlines = []
+ for line in output.splitlines():
+ if 'Running task ' in line:
+ runlines.append(line)
+ if runlines:
+ logger.error('Unexecuted tasks found in preparation log:\n %s' % '\n '.join(runlines))
+ return -1
+ except bb.process.ExecutionError as e:
+ logger.error('Preparation failed:\n%s' % e.stdout)
return -1
return 0
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 05/14] devtool: add sdk-install subcommand
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
` (3 preceding siblings ...)
2016-01-22 11:59 ` [PATCH 04/14] devtool: sdk-update: improve SDK update process robustness Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 11:59 ` [PATCH 06/14] classes/populate_sdk_ext: drop ext-sdk-prepare.py when installing Paul Eggleton
` (8 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
Add the ability to install additional pre-built items (from shared
state) into the extensible SDK. This can already be done implicitly by
adding something to DEPENDS within a recipe you're working on and then
running "devtool build", but it's useful to be able to explicitly
install things particularly if you're using the extensible SDK as a
traditional toolchain.
Note that for this command to be useful you need to have SSTATE_MIRRORS
set in your SDK configuration, and that mirror needs to be populated
with sstate artifacts for recipes you wish to be able to install.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/sdk.py | 63 +++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 62 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 80ea871..12de942 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -8,7 +8,7 @@ import shutil
import errno
import sys
import tempfile
-from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, parse_recipe, DevtoolError
logger = logging.getLogger('devtool')
@@ -235,6 +235,64 @@ def sdk_update(args, config, basepath, workspace):
return -1
return 0
+def sdk_install(args, config, basepath, workspace):
+ """Entry point for the devtool sdk-install command"""
+
+ import oe.recipeutils
+ import bb.process
+
+ for recipe in args.recipename:
+ if recipe in workspace:
+ raise DevtoolError('recipe %s is a recipe in your workspace' % recipe)
+
+ tasks = ['do_populate_sysroot', 'do_packagedata']
+ stampprefixes = {}
+ def checkstamp(recipe):
+ stampprefix = stampprefixes[recipe]
+ stamps = glob.glob(stampprefix + '*')
+ for stamp in stamps:
+ if '.sigdata.' not in stamp and stamp.startswith((stampprefix + '.', stampprefix + '_setscene.')):
+ return True
+ else:
+ return False
+
+ install_recipes = []
+ tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
+ try:
+ for recipe in args.recipename:
+ rd = parse_recipe(config, tinfoil, recipe, True)
+ if not rd:
+ return 1
+ stampprefixes[recipe] = '%s.%s' % (rd.getVar('STAMP', True), tasks[0])
+ if checkstamp(recipe):
+ logger.info('%s is already installed' % recipe)
+ else:
+ install_recipes.append(recipe)
+ finally:
+ tinfoil.shutdown()
+
+ if install_recipes:
+ logger.info('Installing %s...' % ', '.join(install_recipes))
+ install_tasks = []
+ for recipe in install_recipes:
+ for task in tasks:
+ if recipe.endswith('-native') and 'package' in task:
+ continue
+ install_tasks.append('%s:%s' % (recipe, task))
+ try:
+ exec_build_env_command(config.init_path, basepath, 'bitbake --setscene-only %s' % ' '.join(install_tasks))
+ except bb.process.ExecutionError as e:
+ raise DevtoolError('Failed to install %s:\n%s' % (recipe, str(e)))
+ failed = False
+ for recipe in install_recipes:
+ if checkstamp(recipe):
+ logger.info('Successfully installed %s' % recipe)
+ else:
+ raise DevtoolError('Failed to install %s - unavailable' % recipe)
+ failed = True
+ if failed:
+ return 2
+
def register_commands(subparsers, context):
"""Register devtool subcommands from the sdk plugin"""
if context.fixed_setup:
@@ -242,3 +300,6 @@ def register_commands(subparsers, context):
parser_sdk.add_argument('updateserver', help='The update server to fetch latest SDK components from', nargs='?')
parser_sdk.add_argument('--skip-prepare', action="store_true", help='Skip re-preparing the build system after updating (for debugging only)')
parser_sdk.set_defaults(func=sdk_update)
+ parser_sdk_install = subparsers.add_parser('sdk-install', help='Install additional SDK components', description='Installs additional recipe development files into the SDK. (You can use "devtool search" to find available recipes.)')
+ parser_sdk_install.add_argument('recipename', help='Name of the recipe to install the development artifacts for', nargs='+')
+ parser_sdk_install.set_defaults(func=sdk_install)
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 06/14] classes/populate_sdk_ext: drop ext-sdk-prepare.py when installing
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
` (4 preceding siblings ...)
2016-01-22 11:59 ` [PATCH 05/14] devtool: add sdk-install subcommand Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 11:59 ` [PATCH 07/14] classes/populate_sdk*: add dependencies on script files Paul Eggleton
` (7 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
At the end of the extensible SDK installation, if we've successfully
prepared the build system then we don't need ext-sdk-prepare.py. I had
thought earlier that this would be used when updating, but a different
mechanism was needed there so this script isn't used for that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/populate_sdk_ext.bbclass | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index bb8831b..2944576 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -304,6 +304,7 @@ sdk_ext_postinst() {
# sourcing a script. That is why this has to look so ugly.
LOGFILE="$target_sdk_dir/preparing_build_system.log"
sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py '${SDK_INSTALL_TARGETS}' >> $LOGFILE 2>&1" || { echo "ERROR: SDK preparation failed: see $LOGFILE"; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
+ rm $target_sdk_dir/ext-sdk-prepare.py
fi
echo done
}
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 07/14] classes/populate_sdk*: add dependencies on script files
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
` (5 preceding siblings ...)
2016-01-22 11:59 ` [PATCH 06/14] classes/populate_sdk_ext: drop ext-sdk-prepare.py when installing Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 11:59 ` [PATCH 08/14] toolchain-shar-extract.sh: improve behaviour when xz is not installed Paul Eggleton
` (6 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
If the script files we use to construct the SDK installer change then
that really ought to trigger re-execution of the do_populate_sdk(_ext)
task, so add file-checksums varflags to ensure that happens.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/populate_sdk_base.bbclass | 3 +++
meta/classes/populate_sdk_ext.bbclass | 5 +++++
2 files changed, 8 insertions(+)
diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index 99b64f7..966161f 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -243,6 +243,9 @@ populate_sdk_log_check() {
done
}
+do_populate_sdk[file-checksums] += "${COREBASE}/meta/files/toolchain-shar-relocate.sh:True \
+ ${COREBASE}/meta/files/toolchain-shar-extract.sh:True"
+
do_populate_sdk[dirs] = "${PKGDATA_DIR} ${TOPDIR}"
do_populate_sdk[depends] += "${@' '.join([x + ':do_populate_sysroot' for x in d.getVar('SDK_DEPENDS', True).split()])} ${@d.getVarFlag('do_rootfs', 'depends', False)}"
do_populate_sdk[rdepends] = "${@' '.join([x + ':do_populate_sysroot' for x in d.getVar('SDK_RDEPENDS', True).split()])}"
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 2944576..a115127 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -259,6 +259,7 @@ install_tools() {
install -m 0644 ${COREBASE}/meta/files/ext-sdk-prepare.py ${SDK_OUTPUT}/${SDKPATH}
}
+do_populate_sdk_ext[file-checksums] += "${COREBASE}/meta/files/ext-sdk-prepare.py:True"
# Since bitbake won't run as root it doesn't make sense to try and install
# the extensible sdk as root.
@@ -358,4 +359,8 @@ do_populate_sdk_ext[rdepends] += "${@' '.join([x + ':do_build' for x in d.getVar
# Make sure codes change in copy_buildsystem can result in rebuilt
do_populate_sdk_ext[vardeps] += "copy_buildsystem"
+do_populate_sdk_ext[file-checksums] += "${COREBASE}/meta/files/toolchain-shar-relocate.sh:True \
+ ${COREBASE}/meta/files/toolchain-shar-extract.sh:True \
+ ${COREBASE}/scripts/gen-lockedsig-cache:True"
+
addtask populate_sdk_ext after do_sdk_depends
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 08/14] toolchain-shar-extract.sh: improve behaviour when xz is not installed
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
` (6 preceding siblings ...)
2016-01-22 11:59 ` [PATCH 07/14] classes/populate_sdk*: add dependencies on script files Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 11:59 ` [PATCH 09/14] gen-lockedsig-cache: copy correct native sstate into ext SDK Paul Eggleton
` (5 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
Currently if xz isn't installed when installing the SDK you get an error
from tar and then the SDK installation continues, cascading into a load
of other failures since files are missing. Add a check to see if xz can
be run and error out if it can't, and also exit if tar fails.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
| 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
--git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index b1ef4b7..d844771 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -29,6 +29,11 @@ if [ "$INST_ARCH" != "$SDK_ARCH" ]; then
fi
fi
+if ! xz -V > /dev/null 2>&1; then
+ echo "Error: xz is required for installation of this SDK, please install it first"
+ exit 1
+fi
+
DEFAULT_INSTALL_DIR="@SDKPATH@"
SUDO_EXEC=""
target_sdk_dir=""
@@ -168,7 +173,7 @@ fi
payload_offset=$(($(grep -na -m1 "^MARKER:$" $0|cut -d':' -f1) + 1))
printf "Extracting SDK..."
-tail -n +$payload_offset $0| $SUDO_EXEC tar xJ -C $target_sdk_dir --checkpoint=.2500
+tail -n +$payload_offset $0| $SUDO_EXEC tar xJ -C $target_sdk_dir --checkpoint=.2500 || exit 1
echo "done"
printf "Setting it up..."
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 09/14] gen-lockedsig-cache: copy correct native sstate into ext SDK
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
` (7 preceding siblings ...)
2016-01-22 11:59 ` [PATCH 08/14] toolchain-shar-extract.sh: improve behaviour when xz is not installed Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 11:59 ` [PATCH 10/14] recipetool: create: strip quotes from values extracted from CMakeLists.txt Paul Eggleton
` (4 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
When constructing the sstate-cache directory for the extensible SDK,
we were copying in any matching native sstate packages, and as the
signature doesn't actually change when the distro changes (since
NATIVELSBSTRING is just a path separator for the artifacts and is not
part of the signature) we ended up copying duplicated packages when the
distro changed e.g. upon host distro upgrade. Only search in the
NATIVELSBSTRING-named subdirectory for native packages and the issue
goes away.
Fixes [YOCTO #8885].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/copy_buildsystem.py | 5 +++--
scripts/gen-lockedsig-cache | 6 +++---
2 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index 6475510..fb51b51 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -149,8 +149,9 @@ def merge_lockedsigs(copy_tasks, lockedsigs_main, lockedsigs_extra, merged_outpu
def create_locked_sstate_cache(lockedsigs, input_sstate_cache, output_sstate_cache, d, fixedlsbstring=""):
bb.note('Generating sstate-cache...')
- bb.process.run("gen-lockedsig-cache %s %s %s" % (lockedsigs, input_sstate_cache, output_sstate_cache))
+ nativelsbstring = d.getVar('NATIVELSBSTRING', True)
+ bb.process.run("gen-lockedsig-cache %s %s %s %s" % (lockedsigs, input_sstate_cache, output_sstate_cache, nativelsbstring))
if fixedlsbstring:
- nativedir = output_sstate_cache + '/' + d.getVar('NATIVELSBSTRING', True)
+ nativedir = output_sstate_cache + '/' + nativelsbstring
if os.path.isdir(nativedir):
os.rename(nativedir, output_sstate_cache + '/' + fixedlsbstring)
diff --git a/scripts/gen-lockedsig-cache b/scripts/gen-lockedsig-cache
index 9c16506..a4e9ded 100755
--- a/scripts/gen-lockedsig-cache
+++ b/scripts/gen-lockedsig-cache
@@ -13,9 +13,9 @@ def mkdir(d):
if e.errno != errno.EEXIST:
raise e
-if len(sys.argv) < 3:
+if len(sys.argv) < 5:
print("Incorrect number of arguments specified")
- print("syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir>")
+ print("syntax: gen-lockedsig-cache <locked-sigs.inc> <input-cachedir> <output-cachedir> <nativelsbstring>")
sys.exit(1)
print('Reading %s' % sys.argv[1])
@@ -30,7 +30,7 @@ files = set()
for s in sigs:
p = sys.argv[2] + "/" + s[:2] + "/*" + s + "*"
files |= set(glob.glob(p))
- p = sys.argv[2] + "/*/" + s[:2] + "/*" + s + "*"
+ p = sys.argv[2] + "/%s/" % sys.argv[4] + s[:2] + "/*" + s + "*"
files |= set(glob.glob(p))
print('Processing files')
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 10/14] recipetool: create: strip quotes from values extracted from CMakeLists.txt
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
` (8 preceding siblings ...)
2016-01-22 11:59 ` [PATCH 09/14] gen-lockedsig-cache: copy correct native sstate into ext SDK Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 11:59 ` [PATCH 11/14] devtool / recipetool: support specifying a subdirectory within the fetched source Paul Eggleton
` (3 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
Quoting is optional in CMakeLists.txt and is occasionally used, so strip
out quotes if they are present.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_buildsys.py | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index 6afb5de..ba393a8 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -142,6 +142,9 @@ class CmakeRecipeHandler(RecipeHandler):
subdir_re = re.compile('add_subdirectory\s*\(\s*([^)\s]*)\s*([^)\s]*)\s*\)', re.IGNORECASE)
dep_re = re.compile('([^ ><=]+)( *[<>=]+ *[^ ><=]+)?')
+ def interpret_value(value):
+ return value.strip('"')
+
def parse_cmake_file(fn, paths=None):
searchpaths = (paths or []) + [os.path.dirname(fn)]
logger.debug('Parsing file %s' % fn)
@@ -166,13 +169,13 @@ class CmakeRecipeHandler(RecipeHandler):
continue
res = proj_re.match(line)
if res:
- extravalues['PN'] = res.group(1).split()[0]
+ extravalues['PN'] = interpret_value(res.group(1).split()[0])
continue
res = pkgcm_re.match(line)
if res:
res = dep_re.findall(res.group(2))
if res:
- pcdeps.extend([x[0] for x in res])
+ pcdeps.extend([interpret_value(x[0]) for x in res])
inherits.append('pkgconfig')
continue
res = pkgsm_re.match(line)
@@ -180,7 +183,7 @@ class CmakeRecipeHandler(RecipeHandler):
res = dep_re.findall(res.group(2))
if res:
# Note: appending a tuple here!
- item = tuple((x[0] for x in res))
+ item = tuple((interpret_value(x[0]) for x in res))
if len(item) == 1:
item = item[0]
pcdeps.append(item)
@@ -189,7 +192,7 @@ class CmakeRecipeHandler(RecipeHandler):
res = findpackage_re.match(line)
if res:
origpkg = res.group(1)
- pkg = origpkg.lower()
+ pkg = interpret_value(origpkg.lower())
if pkg == 'gettext':
inherits.append('gettext')
elif pkg == 'perl':
@@ -209,7 +212,7 @@ class CmakeRecipeHandler(RecipeHandler):
continue
res = checklib_re.match(line)
if res:
- lib = res.group(1)
+ lib = interpret_value(res.group(1))
if not lib.startswith('$'):
libdeps.append(lib)
if line.lower().startswith('useswig'):
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 11/14] devtool / recipetool: support specifying a subdirectory within the fetched source
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
` (9 preceding siblings ...)
2016-01-22 11:59 ` [PATCH 10/14] recipetool: create: strip quotes from values extracted from CMakeLists.txt Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 11:59 ` [PATCH 12/14] recipetool: create: extract SRC_URI from local git repositories Paul Eggleton
` (2 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
Sometimes you don't want to build an entire project, just a subdirectory
of it; add a --src-subdir option to make that easier. (We still look for
a single subdirectory in what gets unpacked, e.g. what you might find
within a tarball, so whatever you specify with this option is added onto
the end of that.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 6 ++++++
scripts/lib/recipetool/create.py | 11 +++++++++--
2 files changed, 15 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 5390f51..f19de27 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -141,6 +141,8 @@ def add(args, config, basepath, workspace):
extracmdopts += ' -b'
if args.also_native:
extracmdopts += ' --also-native'
+ if args.src_subdir:
+ extracmdopts += ' --src-subdir "%s"' % args.src_subdir
tempdir = tempfile.mkdtemp(prefix='devtool')
try:
@@ -208,6 +210,9 @@ def add(args, config, basepath, workspace):
if not rd:
return 1
+ if args.src_subdir:
+ srctree = os.path.join(srctree, args.src_subdir)
+
bb.utils.mkdirhier(os.path.dirname(appendfile))
with open(appendfile, 'w') as f:
f.write('inherit externalsrc\n')
@@ -1308,6 +1313,7 @@ def register_commands(subparsers, context):
parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true")
parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure). Useful with binary packages e.g. RPMs.', action='store_true')
parser_add.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
+ parser_add.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
parser_add.set_defaults(func=add)
parser_modify = subparsers.add_parser('modify', help='Modify the source for an existing recipe',
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 43861ee..9c3a63d 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -324,6 +324,12 @@ def create_recipe(args):
srcuri = ''
srctree = args.source
+ if args.src_subdir:
+ srcsubdir = os.path.join(srcsubdir, args.src_subdir)
+ srctree_use = os.path.join(srctree, args.src_subdir)
+ else:
+ srctree_use = srctree
+
if args.outfile and os.path.isdir(args.outfile):
outfile = None
outdir = args.outfile
@@ -343,7 +349,7 @@ def create_recipe(args):
lines_before.append('# (Feel free to remove these comments when editing.)')
lines_before.append('#')
- licvalues = guess_license(srctree)
+ licvalues = guess_license(srctree_use)
lic_files_chksum = []
if licvalues:
licenses = []
@@ -472,7 +478,7 @@ def create_recipe(args):
extravalues = {}
for handler in handlers:
- handler.process(srctree, classes, lines_before, lines_after, handled, extravalues)
+ handler.process(srctree_use, classes, lines_before, lines_after, handled, extravalues)
if not realpv:
realpv = extravalues.get('PV', None)
@@ -759,5 +765,6 @@ def register_commands(subparsers):
parser_create.add_argument('-V', '--version', help='Version to use within recipe (PV)')
parser_create.add_argument('-b', '--binary', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true')
parser_create.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
+ parser_create.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
parser_create.set_defaults(func=create_recipe)
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 12/14] recipetool: create: extract SRC_URI from local git repositories
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
` (10 preceding siblings ...)
2016-01-22 11:59 ` [PATCH 11/14] devtool / recipetool: support specifying a subdirectory within the fetched source Paul Eggleton
@ 2016-01-22 11:59 ` Paul Eggleton
2016-01-22 12:00 ` [PATCH 13/14] recipetool: create: fix extraction of name from URLs ending in / Paul Eggleton
2016-01-22 12:00 ` [PATCH 14/14] recipetool: create: better fix for fetch error handling Paul Eggleton
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 11:59 UTC (permalink / raw)
To: openembedded-core
If you specify a local directory which happens to be a git repository
with an origin remote (and it is in fact remote), we can use that for
SRC_URI rather than leaving it blank in the recipe.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 30 +++++++++++++++++++++++-------
1 file changed, 23 insertions(+), 7 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 9c3a63d..5caf374 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -260,6 +260,14 @@ def supports_srcrev(uri):
return True
return False
+def reformat_git_uri(uri):
+ '''Convert any http[s]://....git URI into git://...;protocol=http[s]'''
+ res = re.match('(https?)://([^;]+\.git)(;.*)?$', uri)
+ if res:
+ # Need to switch the URI around so that the git fetcher is used
+ return 'git://%s;protocol=%s%s' % (res.group(2), res.group(1), res.group(3) or '')
+ return uri
+
def create_recipe(args):
import bb.process
import tempfile
@@ -275,16 +283,11 @@ def create_recipe(args):
srcrev = '${AUTOREV}'
if '://' in args.source:
# Fetch a URL
- fetchuri = urlparse.urldefrag(args.source)[0]
+ fetchuri = reformat_git_uri(urlparse.urldefrag(args.source)[0])
if args.binary:
# Assume the archive contains the directory structure verbatim
# so we need to extract to a subdirectory
fetchuri += ';subdir=%s' % os.path.splitext(os.path.basename(urlparse.urlsplit(fetchuri).path))[0]
- git_re = re.compile('(https?)://([^;]+\.git)(;.*)?$')
- res = git_re.match(fetchuri)
- if res:
- # Need to switch the URI around so that the git fetcher is used
- fetchuri = 'git://%s;protocol=%s%s' % (res.group(2), res.group(1), res.group(3) or '')
srcuri = fetchuri
rev_re = re.compile(';rev=([^;]+)')
res = rev_re.search(srcuri)
@@ -321,8 +324,21 @@ def create_recipe(args):
if not os.path.isdir(args.source):
logger.error('Invalid source directory %s' % args.source)
sys.exit(1)
- srcuri = ''
srctree = args.source
+ srcuri = ''
+ if os.path.exists(os.path.join(srctree, '.git')):
+ # Try to get upstream repo location from origin remote
+ try:
+ stdout, _ = bb.process.run('git remote -v', cwd=srctree, shell=True)
+ except bb.process.ExecutionError as e:
+ stdout = None
+ if stdout:
+ for line in stdout.splitlines():
+ splitline = line.split()
+ if len(splitline) > 1:
+ if splitline[0] == 'origin' and '://' in splitline[1]:
+ srcuri = reformat_git_uri(splitline[1])
+ break
if args.src_subdir:
srcsubdir = os.path.join(srcsubdir, args.src_subdir)
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 13/14] recipetool: create: fix extraction of name from URLs ending in /
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
` (11 preceding siblings ...)
2016-01-22 11:59 ` [PATCH 12/14] recipetool: create: extract SRC_URI from local git repositories Paul Eggleton
@ 2016-01-22 12:00 ` Paul Eggleton
2016-01-22 12:00 ` [PATCH 14/14] recipetool: create: better fix for fetch error handling Paul Eggleton
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 12:00 UTC (permalink / raw)
To: openembedded-core
If the URL ends in a / then we want to strip that off the path we split
out of the URL before calling os.path.basename() on it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 5caf374..f0bb58e 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -432,7 +432,7 @@ def create_recipe(args):
if srcuri and not realpv or not pn:
parseres = urlparse.urlparse(srcuri)
if parseres.path:
- srcfile = os.path.basename(parseres.path)
+ srcfile = os.path.basename(parseres.path.rstrip('/'))
name_pn, name_pv = determine_from_filename(srcfile)
logger.debug('Determined from filename: name = "%s", version = "%s"' % (name_pn, name_pv))
if name_pn and not pn:
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 14/14] recipetool: create: better fix for fetch error handling
2016-01-22 11:59 [PATCH 00/14] Extensible SDK improvements redux Paul Eggleton
` (12 preceding siblings ...)
2016-01-22 12:00 ` [PATCH 13/14] recipetool: create: fix extraction of name from URLs ending in / Paul Eggleton
@ 2016-01-22 12:00 ` Paul Eggleton
13 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-01-22 12:00 UTC (permalink / raw)
To: openembedded-core
I was a little bit hasty in OE-Core revision
c2cc5abe34169eae92067d97ce1e747e7c1413f5 - it turns out BitBake's
fetcher code is not consistent in whether it logs something useful or
not; when fetching from an http URL it does but with a git repository
it doesn't. In advance of any major reworking of fetch error handling in
BitBake, let's just print the text of the exception and then we know we
have shown something to the user.
Additionally, we were only catching FetchException here but there are
several other classes of exception that the fetcher can raise (e.g.
MalformedUrl); catch the parent BBFetchException class instead so we
avoid tracebacks for those other classes as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f0bb58e..5f90b10 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -299,8 +299,8 @@ def create_recipe(args):
logger.info('Fetching %s...' % srcuri)
try:
checksums = scriptutils.fetch_uri(tinfoil.config_data, fetchuri, srctree, srcrev)
- except bb.fetch2.FetchError:
- # Error already printed
+ except bb.fetch2.BBFetchException as e:
+ logger.error(str(e).rstrip())
sys.exit(1)
dirlist = os.listdir(srctree)
if 'git.indirectionsymlink' in dirlist:
--
2.5.0
^ permalink raw reply related [flat|nested] 15+ messages in thread