* [PATCH 01/27] lib/oe/recipeutils: properly split unexpanded variable values
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 23:00 ` Christopher Larson
2015-09-22 16:21 ` [PATCH 02/27] classes/populate_sdk_ext: fix missing environment settings if running installer with sh Paul Eggleton
` (25 subsequent siblings)
26 siblings, 1 reply; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Variables such as SRC_URI which are space-separated may also contain
Python expressions (${@...}) which themselves contain spaces that
shouldn't be split when splitting the value into items. In order to
ensure this we need to use a custom splitting function instead of just
string.split().
This issue could be seen when doing "devtool modify sudo", adding a
commit to the resulting source repository then "devtool update-recipe" -
the Python expression in SRC_URI was being unnecessarily broken onto
multiple lines.
Fixes [YOCTO #8046].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/recipeutils.py | 61 ++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 59 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index d4fa726..190ac3a 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -95,6 +95,63 @@ def get_var_files(fn, varlist, d):
return varfiles
+def split_var_value(value, assignment=True):
+ """
+ Split a space-separated variable's value into a list of items,
+ taking into account that some of the items might be made up of
+ expressions containing spaces that should not be split.
+ Parameters:
+ value:
+ The string value to split
+ assignment:
+ True to assume that the value represents an assignment
+ statement, False otherwise. If True, and an assignment
+ statement is passed in the first item in
+ the returned list will be the part of the assignment
+ statement up to and including the opening quote character,
+ and the last item will be the closing quote.
+ """
+ inexpr = 0
+ lastchar = None
+ out = []
+ buf = ''
+ for char in value:
+ if char == '{':
+ if lastchar == '$':
+ inexpr += 1
+ elif char == '}':
+ inexpr -= 1
+ elif assignment and char in '"\'' and inexpr == 0:
+ if buf:
+ out.append(buf)
+ out.append(char)
+ char = ''
+ buf = ''
+ elif char.isspace() and inexpr == 0:
+ char = ''
+ if buf:
+ out.append(buf)
+ buf = ''
+ buf += char
+ lastchar = char
+ if buf:
+ out.append(buf)
+
+ # Join together assignment statement and opening quote
+ outlist = out
+ if assignment:
+ assigfound = False
+ for idx, item in enumerate(out):
+ if '=' in item:
+ assigfound = True
+ if assigfound:
+ if '"' in item or "'" in item:
+ outlist = [' '.join(out[:idx+1])]
+ outlist.extend(out[idx+1:])
+ break
+ return outlist
+
+
def patch_recipe_file(fn, values, patch=False, relpath=''):
"""Update or insert variable values into a recipe file (assuming you
have already identified the exact file you want to update.)
@@ -112,7 +169,7 @@ def patch_recipe_file(fn, values, patch=False, relpath=''):
if name in nowrap_vars:
tf.write(rawtext)
elif name in list_vars:
- splitvalue = values[name].split()
+ splitvalue = split_var_value(values[name], assignment=False)
if len(splitvalue) > 1:
linesplit = ' \\\n' + (' ' * (len(name) + 4))
tf.write('%s = "%s%s"\n' % (name, linesplit.join(splitvalue), linesplit))
@@ -518,7 +575,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
instfunclines.append(line)
return (instfunclines, None, 4, False)
else:
- splitval = origvalue.split()
+ splitval = split_var_value(origvalue, assignment=False)
changed = False
removevar = varname
if varname in ['SRC_URI', 'SRC_URI_append%s' % appendoverride]:
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* Re: [PATCH 01/27] lib/oe/recipeutils: properly split unexpanded variable values
2015-09-22 16:21 ` [PATCH 01/27] lib/oe/recipeutils: properly split unexpanded variable values Paul Eggleton
@ 2015-09-22 23:00 ` Christopher Larson
0 siblings, 0 replies; 29+ messages in thread
From: Christopher Larson @ 2015-09-22 23:00 UTC (permalink / raw)
To: Paul Eggleton; +Cc: Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 1702 bytes --]
On Tue, Sep 22, 2015 at 9:21 AM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:
> Variables such as SRC_URI which are space-separated may also contain
> Python expressions (${@...}) which themselves contain spaces that
> shouldn't be split when splitting the value into items. In order to
> ensure this we need to use a custom splitting function instead of just
> string.split().
>
> This issue could be seen when doing "devtool modify sudo", adding a
> commit to the resulting source repository then "devtool update-recipe" -
> the Python expression in SRC_URI was being unnecessarily broken onto
> multiple lines.
>
> Fixes [YOCTO #8046].
>
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
>
This is interesting, I recently had a need for a split like that too. I had
written a prototype class which automatically disables all packageconfigs
which introduce optional dependencies, to see how much it would reduce the
dependencies, and wanted to ignore bits dependent on DISTRO_FEATURES, so
wanted to ignore all inline python.
https://gist.github.com/kergoth/2c556d9e34cbcd03cb82#file-packageconfig-backdel-bbclass-L107-L122
is my version of it. Possibly overly complex, since my version produces an
extremely lightweight parse tree (list-based) for nested references, and
then has to flatten it back out. I found it interesting that we both had a
need for this sort of functionality -- perhaps a split function along these
lines would be of use in one of the oe python package modules.
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
[-- Attachment #2: Type: text/html, Size: 2369 bytes --]
^ permalink raw reply [flat|nested] 29+ messages in thread
* [PATCH 02/27] classes/populate_sdk_ext: fix missing environment settings if running installer with sh
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
2015-09-22 16:21 ` [PATCH 01/27] lib/oe/recipeutils: properly split unexpanded variable values Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 03/27] classes/populate_sdk_ext: allow custom configuration for extensible SDK Paul Eggleton
` (24 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
If you ran the extensible SDK installer file with sh (instead of bash),
then the additional call to buildtools environment setup, extension of
PATH to support running devtool, and setting of OE_SKIP_SDK_CHECK
weren't being added to the end of the script. This is because apparently
bash is happy to expand wildcards in the target of a redirection, but
bash running in POSIX sh mode won't (although it apparently does work on
the sh command line rather than within a script run as an argument to
sh). In any case using a wildcard here is a bit of a crutch which we
don't need, so replace it with the proper path to the environment setup
script.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/populate_sdk_ext.bbclass | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 4ef8838..15cae71 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -222,14 +222,15 @@ sdk_ext_postinst() {
# Make sure when the user sets up the environment, they also get
# the buildtools-tarball tools in their path.
- echo ". $target_sdk_dir/buildtools/environment-setup*" >> $target_sdk_dir/environment-setup*
+ env_setup_script="$target_sdk_dir/environment-setup-${REAL_MULTIMACH_TARGET_SYS}"
+ echo ". $target_sdk_dir/buildtools/environment-setup*" >> $env_setup_script
# Allow bitbake environment setup to be ran as part of this sdk.
- echo "export OE_SKIP_SDK_CHECK=1" >> $target_sdk_dir/environment-setup*
+ echo "export OE_SKIP_SDK_CHECK=1" >> $env_setup_script
# A bit of another hack, but we need this in the path only for devtool
# so put it at the end of $PATH.
- echo "export PATH=\$PATH:$target_sdk_dir/sysroots/${SDK_SYS}/${bindir_nativesdk}" >> $target_sdk_dir/environment-setup*
+ echo "export PATH=\$PATH:$target_sdk_dir/sysroots/${SDK_SYS}/${bindir_nativesdk}" >> $env_setup_script
# For now this is where uninative.bbclass expects the tarball
mv *-nativesdk-libc.tar.* $target_sdk_dir/`dirname ${oe_init_build_env_path}`
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 03/27] classes/populate_sdk_ext: allow custom configuration for extensible SDK
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
2015-09-22 16:21 ` [PATCH 01/27] lib/oe/recipeutils: properly split unexpanded variable values Paul Eggleton
2015-09-22 16:21 ` [PATCH 02/27] classes/populate_sdk_ext: fix missing environment settings if running installer with sh Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 04/27] classes/populate_sdk_ext: drop work-config.inc Paul Eggleton
` (23 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Provide the ability to define a function containing extra configuration
values to be added to the local.conf file that goes into the SDK. For
example, this could be used to set up SSTATE_MIRRORS within the SDK.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/populate_sdk_ext.bbclass | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 15cae71..947ec74 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -157,6 +157,13 @@ python copy_buildsystem () {
# Ensure locked sstate cache objects are re-used without error
f.write('SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "warn"\n\n')
+ # If you define a sdk_extraconf() function then it can contain additional config
+ extraconf = (d.getVar('sdk_extraconf', True) or '').strip()
+ if extraconf:
+ # Strip off any leading / trailing spaces
+ for line in extraconf.splitlines():
+ f.write(line.strip() + '\n')
+
f.write('require conf/locked-sigs.inc\n')
f.write('require conf/work-config.inc\n')
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 04/27] classes/populate_sdk_ext: drop work-config.inc
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (2 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 03/27] classes/populate_sdk_ext: allow custom configuration for extensible SDK Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 05/27] toolchain-shar-extract.sh: show progress when extracting SDK Paul Eggleton
` (22 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
This is not actually used for anything - I thought that we would need to
use it within devtool to set global configuration, but we're able to do
everything we need within the bbappends it creates, which also saves on
parse time. If we're not going to use work-config.inc let's just drop it
completely.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/populate_sdk_ext.bbclass | 5 -----
scripts/lib/devtool/sdk.py | 2 +-
2 files changed, 1 insertion(+), 6 deletions(-)
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 947ec74..c244d93 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -165,7 +165,6 @@ python copy_buildsystem () {
f.write(line.strip() + '\n')
f.write('require conf/locked-sigs.inc\n')
- f.write('require conf/work-config.inc\n')
sigfile = d.getVar('WORKDIR', True) + '/locked-sigs.inc'
oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
@@ -185,10 +184,6 @@ python copy_buildsystem () {
d.getVar('SSTATE_DIR', True),
sstate_out, d,
fixedlsbstring)
-
- # Create a dummy config file for additional settings
- with open(baseoutpath + '/conf/work-config.inc', 'w') as f:
- pass
}
def extsdk_get_buildtools_filename(d):
diff --git a/scripts/lib/devtool/sdk.py b/scripts/lib/devtool/sdk.py
index 2f416b3..7a842af 100644
--- a/scripts/lib/devtool/sdk.py
+++ b/scripts/lib/devtool/sdk.py
@@ -170,7 +170,7 @@ def sdk_update(args, config, basepath, workspace):
logger.error("Updating meta data via git failed")
return ret
logger.debug("Updating conf files ...")
- conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'work-config.inc', 'locked-sigs.inc']
+ conf_files = ['local.conf', 'bblayers.conf', 'devtool.conf', 'locked-sigs.inc']
for conf in conf_files:
ret = subprocess.call("wget -q -O - %s/conf/%s > conf/%s" % (args.updateserver, conf, conf), shell=True)
if ret != 0:
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 05/27] toolchain-shar-extract.sh: show progress when extracting SDK
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (3 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 04/27] classes/populate_sdk_ext: drop work-config.inc Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 06/27] classes/externalsrc: scale back warning to a plain note Paul Eggleton
` (21 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Tar has supported a --checkpoint option since version 1.15.91, so it
should be safe to use here to print dots showing that it's still doing
something (technically it's not really progress unless you know how many
dots it's going to print, which even it doesn't know at the start, but
it's better than nothing).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
| 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index cd0a547..6891279 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -163,7 +163,7 @@ fi
payload_offset=$(($(grep -na -m1 "^MARKER:$" $0|cut -d':' -f1) + 1))
printf "Extracting SDK..."
-tail -n +$payload_offset $0| $SUDO_EXEC tar xj -C $target_sdk_dir
+tail -n +$payload_offset $0| $SUDO_EXEC tar xj -C $target_sdk_dir --checkpoint=.2500
echo "done"
printf "Setting it up..."
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 06/27] classes/externalsrc: scale back warning to a plain note
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (4 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 05/27] toolchain-shar-extract.sh: show progress when extracting SDK Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 07/27] classes/externalsrc: fix setting of deps varflag as a string Paul Eggleton
` (20 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
It turns out that there are folks out there who use externalsrc in
normal builds and don't really need to be warned; additionally within
the extensible SDK or when using devtool, it shouldn't be a warning
situation. Thus, scale it back to a note (we can't use bb.note() here
since that wouldn't actually be piped through to the bitbake UI). Also
touch up the message a little bit.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/externalsrc.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 0fa5817..70e6fd7 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -88,5 +88,5 @@ python () {
python externalsrc_compile_prefunc() {
# Make it obvious that this is happening, since forgetting about it could lead to much confusion
- bb.warn('Compiling %s from external source %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True)))
+ bb.plain('NOTE: %s: compiling from external source tree %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True)))
}
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 07/27] classes/externalsrc: fix setting of deps varflag as a string
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (5 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 06/27] classes/externalsrc: scale back warning to a plain note Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 08/27] lib/oe/patch: fix for git am not cleaning up after itself Paul Eggleton
` (19 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
The value of the deps varflag for tasks is (unusually) expected to be a
list object rather than a space-separated string, hence we cannot use
appendVarFlag() here. This fixes a traceback when parsing the gcc recipe
with externalsrc enabled, for example.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/externalsrc.bbclass | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 70e6fd7..f7ed66d 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -73,7 +73,8 @@ python () {
fetch_tasks = ['do_fetch', 'do_unpack']
# If we deltask do_patch, there's no dependency to ensure do_unpack gets run, so add one
- d.appendVarFlag('do_configure', 'deps', ['do_unpack'])
+ # Note that we cannot use d.appendVarFlag() here because deps is expected to be a list object, not a string
+ d.setVarFlag('do_configure', 'deps', (d.getVarFlag('do_configure', 'deps', False) or []) + ['do_unpack'])
for task in d.getVar("SRCTREECOVEREDTASKS", True).split():
if local_srcuri and task in fetch_tasks:
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 08/27] lib/oe/patch: fix for git am not cleaning up after itself
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (6 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 07/27] classes/externalsrc: fix setting of deps varflag as a string Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 09/27] scripts/contrib: add devtool stress tester Paul Eggleton
` (18 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Unfortunately it appears that under certain circumstances, a failed
git am followed by git am --abort won't clean up any changes the patch
might have made - this was seen when running "devtool extract" on the
unzip recipe; unzip-6.0_overflow3.diff has a malformed date as far as
git am is concerned but it triggers this condition. Add a
git reset --hard HEAD followed by git clean -f in order to recover from
this scenario.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/patch.py | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 108bf1d..7441214 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -407,6 +407,13 @@ class GitApplyTree(PatchTree):
runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
except CmdError:
pass
+ # git am won't always clean up after itself, sadly, so...
+ shellcmd = ["git", "--work-tree=%s" % reporoot, "reset", "--hard", "HEAD"]
+ runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
+ # Also need to take care of any stray untracked files
+ shellcmd = ["git", "--work-tree=%s" % reporoot, "clean", "-f"]
+ runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)
+
# Fall back to git apply
shellcmd = ["git", "--git-dir=%s" % reporoot, "apply", "-p%s" % patch['strippath']]
try:
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 09/27] scripts/contrib: add devtool stress tester
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (7 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 08/27] lib/oe/patch: fix for git am not cleaning up after itself Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 10/27] devtool: check that source tree still exists Paul Eggleton
` (17 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Add a script to run "devtool modify" followed by a build on every target
recipe in the environment (with the option to skip/resume from/only
include specific recipes). This takes far too long to run as an
oe-selftest test but is still something that is useful to be able to
run. There's also a slightly quicker mode that just runs "devtool
extract" on each recipe.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/contrib/devtool-stress.py | 241 ++++++++++++++++++++++++++++++++++++++
1 file changed, 241 insertions(+)
create mode 100755 scripts/contrib/devtool-stress.py
diff --git a/scripts/contrib/devtool-stress.py b/scripts/contrib/devtool-stress.py
new file mode 100755
index 0000000..4b35fc9
--- /dev/null
+++ b/scripts/contrib/devtool-stress.py
@@ -0,0 +1,241 @@
+#!/usr/bin/env python
+
+# devtool stress tester
+#
+# Written by: Paul Eggleton <paul.eggleton@linux.intel.com>
+#
+# Copyright 2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+import sys
+import os
+import os.path
+import subprocess
+import re
+import argparse
+import logging
+import tempfile
+import shutil
+import signal
+import fnmatch
+
+scripts_lib_path = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), '..', 'lib'))
+sys.path.insert(0, scripts_lib_path)
+import scriptutils
+logger = scriptutils.logger_create('devtool-stress')
+
+def select_recipes(args):
+ import bb.tinfoil
+ tinfoil = bb.tinfoil.Tinfoil()
+ tinfoil.prepare(False)
+
+ pkg_pn = tinfoil.cooker.recipecache.pkg_pn
+ (latest_versions, preferred_versions) = bb.providers.findProviders(tinfoil.config_data, tinfoil.cooker.recipecache, pkg_pn)
+
+ skip_classes = args.skip_classes.split(',')
+
+ recipelist = []
+ for pn in sorted(pkg_pn):
+ pref = preferred_versions[pn]
+ inherits = [os.path.splitext(os.path.basename(f))[0] for f in tinfoil.cooker.recipecache.inherits[pref[1]]]
+ for cls in skip_classes:
+ if cls in inherits:
+ break
+ else:
+ recipelist.append(pn)
+
+ tinfoil.shutdown()
+
+ resume_from = args.resume_from
+ if resume_from:
+ if not resume_from in recipelist:
+ print('%s is not a testable recipe' % resume_from)
+ return 1
+ if args.only:
+ only = args.only.split(',')
+ for onlyitem in only:
+ for pn in recipelist:
+ if fnmatch.fnmatch(pn, onlyitem):
+ break
+ else:
+ print('%s does not match any testable recipe' % onlyitem)
+ return 1
+ else:
+ only = None
+ if args.skip:
+ skip = args.skip.split(',')
+ else:
+ skip = []
+
+ recipes = []
+ for pn in recipelist:
+ if resume_from:
+ if pn == resume_from:
+ resume_from = None
+ else:
+ continue
+
+ if args.only:
+ for item in only:
+ if fnmatch.fnmatch(pn, item):
+ break
+ else:
+ continue
+
+ skipit = False
+ for item in skip:
+ if fnmatch.fnmatch(pn, item):
+ skipit = True
+ if skipit:
+ continue
+
+ recipes.append(pn)
+
+ return recipes
+
+
+def stress_extract(args):
+ import bb.process
+
+ recipes = select_recipes(args)
+
+ failures = 0
+ tmpdir = tempfile.mkdtemp()
+ os.setpgrp()
+ try:
+ for pn in recipes:
+ sys.stdout.write('Testing %s ' % (pn + ' ').ljust(40, '.'))
+ sys.stdout.flush()
+ failed = False
+
+ srctree = os.path.join(tmpdir, pn)
+ try:
+ bb.process.run('devtool extract %s %s' % (pn, srctree))
+ except bb.process.CmdError as exc:
+ failed = True
+ with open('stress_%s_extract.log' % pn, 'w') as f:
+ f.write(str(exc))
+
+ if os.path.exists(srctree):
+ shutil.rmtree(srctree)
+
+ if failed:
+ print('failed')
+ failures += 1
+ else:
+ print('ok')
+ except KeyboardInterrupt:
+ # We want any child processes killed. This is crude, but effective.
+ os.killpg(0, signal.SIGTERM)
+
+ if failures:
+ return 1
+ else:
+ return 0
+
+
+def stress_modify(args):
+ import bb.process
+
+ recipes = select_recipes(args)
+
+ failures = 0
+ tmpdir = tempfile.mkdtemp()
+ os.setpgrp()
+ try:
+ for pn in recipes:
+ sys.stdout.write('Testing %s ' % (pn + ' ').ljust(40, '.'))
+ sys.stdout.flush()
+ failed = False
+ reset = True
+
+ srctree = os.path.join(tmpdir, pn)
+ try:
+ bb.process.run('devtool modify -x %s %s' % (pn, srctree))
+ except bb.process.CmdError as exc:
+ with open('stress_%s_modify.log' % pn, 'w') as f:
+ f.write(str(exc))
+ failed = 'modify'
+ reset = False
+
+ if not failed:
+ try:
+ bb.process.run('bitbake -c install %s' % pn)
+ except bb.process.CmdError as exc:
+ with open('stress_%s_install.log' % pn, 'w') as f:
+ f.write(str(exc))
+ failed = 'build'
+ if reset:
+ try:
+ bb.process.run('devtool reset %s' % pn)
+ except bb.process.CmdError as exc:
+ print('devtool reset failed: %s' % str(exc))
+ break
+
+ if os.path.exists(srctree):
+ shutil.rmtree(srctree)
+
+ if failed:
+ print('failed (%s)' % failed)
+ failures += 1
+ else:
+ print('ok')
+ except KeyboardInterrupt:
+ # We want any child processes killed. This is crude, but effective.
+ os.killpg(0, signal.SIGTERM)
+
+ if failures:
+ return 1
+ else:
+ return 0
+
+
+def main():
+ parser = argparse.ArgumentParser(description="devtool stress tester",
+ epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
+ parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
+ parser.add_argument('-r', '--resume-from', help='Resume from specified recipe', metavar='PN')
+ parser.add_argument('-o', '--only', help='Only test specified recipes (comma-separated without spaces, wildcards allowed)', metavar='PNLIST')
+ parser.add_argument('-s', '--skip', help='Skip specified recipes (comma-separated without spaces, wildcards allowed)', metavar='PNLIST')
+ parser.add_argument('-c', '--skip-classes', help='Skip recipes inheriting specified classes (comma-separated) - default %(default)s', metavar='CLASSLIST', default='native,nativesdk,cross,cross-canadian,image,populate_sdk,meta,packagegroup')
+ subparsers = parser.add_subparsers(title='subcommands', metavar='<subcommand>')
+
+ parser_modify = subparsers.add_parser('modify',
+ help='Run "devtool modify" followed by a build with bitbake on matching recipes',
+ description='Runs "devtool modify" followed by a build with bitbake on matching recipes')
+ parser_modify.set_defaults(func=stress_modify)
+
+ parser_extract = subparsers.add_parser('extract',
+ help='Run "devtool extract" on matching recipes',
+ description='Runs "devtool extract" on matching recipes')
+ parser_extract.set_defaults(func=stress_extract)
+
+ args = parser.parse_args()
+
+ if args.debug:
+ logger.setLevel(logging.DEBUG)
+
+ import scriptpath
+ bitbakepath = scriptpath.add_bitbake_lib_path()
+ if not bitbakepath:
+ logger.error("Unable to find bitbake by searching parent directory of this script or PATH")
+ return 1
+ logger.debug('Found bitbake path: %s' % bitbakepath)
+
+ ret = args.func(args)
+
+if __name__ == "__main__":
+ main()
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 10/27] devtool: check that source tree still exists
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (8 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 09/27] scripts/contrib: add devtool stress tester Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 11/27] devtool: add: ensure --color=never turns off recipetool colour output Paul Eggleton
` (16 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Sometimes, particularly if you extracted the source to /tmp which is on
tmpfs, the external source tree that is being pointed to may no longer
exist when you come to run "devtool build" or "devtool update-recipe"
etc. Make all of the commands that need to check for a recipe being in
the workspace call a single function and have that function additionally
check the source tree still exists where appropriate.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/__init__.py | 14 ++++++++++++++
scripts/lib/devtool/build.py | 6 ++----
scripts/lib/devtool/deploy.py | 7 +++----
scripts/lib/devtool/package.py | 6 ++----
scripts/lib/devtool/standard.py | 11 ++++-------
5 files changed, 25 insertions(+), 19 deletions(-)
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 404d3e6..fb699b5 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -136,3 +136,17 @@ def parse_recipe(config, tinfoil, pn, appends):
not path.startswith(config.workspace_path)]
return oe.recipeutils.parse_recipe(recipefile, append_files,
tinfoil.config_data)
+
+def check_workspace_recipe(workspace, pn, checksrc=True):
+ """
+ Check that a recipe is in the workspace and (optionally) that source
+ is present.
+ """
+ if not pn in workspace:
+ raise DevtoolError("No recipe named '%s' in your workspace" % pn)
+ if checksrc:
+ srctree = workspace[pn]['srctree']
+ if not os.path.exists(srctree):
+ raise DevtoolError("Source tree %s for recipe %s does not exist" % (srctree, pn))
+ if not os.listdir(srctree):
+ raise DevtoolError("Source tree %s for recipe %s is empty" % (srctree, pn))
diff --git a/scripts/lib/devtool/build.py b/scripts/lib/devtool/build.py
index 335aff5..9b58858 100644
--- a/scripts/lib/devtool/build.py
+++ b/scripts/lib/devtool/build.py
@@ -21,7 +21,7 @@ import bb
import logging
import argparse
import tempfile
-from devtool import exec_build_env_command, DevtoolError
+from devtool import exec_build_env_command, check_workspace_recipe, DevtoolError
logger = logging.getLogger('devtool')
@@ -42,9 +42,7 @@ def _create_conf_file(values, conf_file=None):
def build(args, config, basepath, workspace):
"""Entry point for the devtool 'build' subcommand"""
- if not args.recipename in workspace:
- raise DevtoolError("no recipe named %s in your workspace" %
- args.recipename)
+ check_workspace_recipe(workspace, args.recipename)
build_task = config.get('Build', 'build_task', 'populate_sysroot')
diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py
index fa93adf..41b666f 100644
--- a/scripts/lib/devtool/deploy.py
+++ b/scripts/lib/devtool/deploy.py
@@ -19,7 +19,7 @@
import os
import subprocess
import logging
-from devtool import exec_fakeroot, setup_tinfoil, DevtoolError
+from devtool import exec_fakeroot, setup_tinfoil, check_workspace_recipe, DevtoolError
logger = logging.getLogger('devtool')
@@ -28,9 +28,8 @@ def deploy(args, config, basepath, workspace):
import re
import oe.recipeutils
- if not args.recipename in workspace:
- raise DevtoolError("no recipe named %s in your workspace" %
- args.recipename)
+ check_workspace_recipe(workspace, args.recipename, checksrc=False)
+
try:
host, destdir = args.target.split(':')
except ValueError:
diff --git a/scripts/lib/devtool/package.py b/scripts/lib/devtool/package.py
index 3a7a36b..28ecfed 100644
--- a/scripts/lib/devtool/package.py
+++ b/scripts/lib/devtool/package.py
@@ -20,7 +20,7 @@ import os
import subprocess
import logging
from bb.process import ExecutionError
-from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError
logger = logging.getLogger('devtool')
@@ -30,9 +30,7 @@ def plugin_init(pluginlist):
def package(args, config, basepath, workspace):
"""Entry point for the devtool 'package' subcommand"""
- if not args.recipename in workspace:
- raise DevtoolError("no recipe named %s in your workspace" %
- args.recipename)
+ check_workspace_recipe(workspace, args.recipename)
image_pkgtype = config.get('Package', 'image_pkgtype', '')
if not image_pkgtype:
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index d5900b4..018ef2a 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -25,7 +25,7 @@ import logging
import argparse
import scriptutils
import errno
-from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError
from devtool import parse_recipe
logger = logging.getLogger('devtool')
@@ -776,9 +776,7 @@ def _guess_recipe_update_mode(srctree, rdata):
def update_recipe(args, config, basepath, workspace):
"""Entry point for the devtool 'update-recipe' subcommand"""
- if not args.recipename in workspace:
- raise DevtoolError("no recipe named %s in your workspace" %
- args.recipename)
+ check_workspace_recipe(workspace, args.recipename)
if args.append:
if not os.path.exists(args.append):
@@ -830,9 +828,8 @@ def reset(args, config, basepath, workspace):
if args.recipename:
if args.all:
raise DevtoolError("Recipe cannot be specified if -a/--all is used")
- elif not args.recipename in workspace:
- raise DevtoolError("no recipe named %s in your workspace" %
- args.recipename)
+ else:
+ check_workspace_recipe(workspace, args.recipename, checksrc=False)
elif not args.all:
raise DevtoolError("Recipe must be specified, or specify -a/--all to "
"reset all recipes")
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 11/27] devtool: add: ensure --color=never turns off recipetool colour output
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (9 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 10/27] devtool: check that source tree still exists Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 12/27] devtool / lib/oe/recipeutils: ensure we can parse without bbappends Paul Eggleton
` (15 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 018ef2a..ec21b3c 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -82,7 +82,7 @@ def add(args, config, basepath, workspace):
else:
bp = args.recipename
recipefile = os.path.join(recipedir, "%s.bb" % bp)
- if sys.stdout.isatty():
+ if args.color == 'auto' and sys.stdout.isatty():
color = 'always'
else:
color = args.color
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 12/27] devtool / lib/oe/recipeutils: ensure we can parse without bbappends
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (10 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 11/27] devtool: add: ensure --color=never turns off recipetool colour output Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 13/27] devtool: add: properly handle separate build directory Paul Eggleton
` (14 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
These functions ostensibly allowed parsing a recipe without bbappends
but this clearly hadn't been tested because a variable was unassigned in
both of them in that case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/recipeutils.py | 2 ++
scripts/lib/devtool/__init__.py | 2 ++
2 files changed, 4 insertions(+)
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 190ac3a..35b88d3 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -72,6 +72,8 @@ def parse_recipe_simple(cooker, pn, d, appends=True):
raise bb.providers.NoProvider('Unable to find any recipe file matching %s' % pn)
if appends:
appendfiles = cooker.collection.get_file_appends(recipefile)
+ else:
+ appendfiles = None
return parse_recipe(recipefile, appendfiles, d)
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index fb699b5..3ea3802 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -134,6 +134,8 @@ def parse_recipe(config, tinfoil, pn, appends):
# Filter out appends from the workspace
append_files = [path for path in append_files if
not path.startswith(config.workspace_path)]
+ else:
+ append_files = None
return oe.recipeutils.parse_recipe(recipefile, append_files,
tinfoil.config_data)
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 13/27] devtool: add: properly handle separate build directory
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (11 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 12/27] devtool / lib/oe/recipeutils: ensure we can parse without bbappends Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 14/27] recipetool: create: fix creating empty shell functions Paul Eggleton
` (13 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
When we were adding a recipe for software that would typically be built
in the same directory as the source, we were always using a separate
build directory unless the user explicitly specified not to, leading to
errors for software that doesn't expect to be built that way (such as
Python modules using distutils). Split out the code that makes this
determination automatically from the "devtool modify" and "devtool
upgrade" code and re-use that here so the behaviour is consistent.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/__init__.py | 18 ++++++++++++++++++
scripts/lib/devtool/standard.py | 30 +++++++++++++++---------------
scripts/lib/devtool/upgrade.py | 20 +++-----------------
3 files changed, 36 insertions(+), 32 deletions(-)
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 3ea3802..07a3636 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -152,3 +152,21 @@ def check_workspace_recipe(workspace, pn, checksrc=True):
raise DevtoolError("Source tree %s for recipe %s does not exist" % (srctree, pn))
if not os.listdir(srctree):
raise DevtoolError("Source tree %s for recipe %s is empty" % (srctree, pn))
+
+def use_external_build(same_dir, no_same_dir, d):
+ """
+ Determine if we should use B!=S (separate build and source directories) or not
+ """
+ b_is_s = True
+ if no_same_dir:
+ logger.info('Using separate build directory since --no-same-dir specified')
+ b_is_s = False
+ elif same_dir:
+ logger.info('Using source tree as build directory since --same-dir specified')
+ elif bb.data.inherits_class('autotools-brokensep', d):
+ logger.info('Using source tree as build directory since recipe inherits autotools-brokensep')
+ elif d.getVar('B', True) == os.path.abspath(d.getVar('S', True)):
+ logger.info('Using source tree as build directory since that would be the default for this recipe')
+ else:
+ b_is_s = False
+ return b_is_s
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index ec21b3c..700a56b 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -25,7 +25,7 @@ import logging
import argparse
import scriptutils
import errno
-from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, DevtoolError
from devtool import parse_recipe
logger = logging.getLogger('devtool')
@@ -107,17 +107,26 @@ def add(args, config, basepath, workspace):
(stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
initial_rev = stdout.rstrip()
+ tinfoil = setup_tinfoil(config_only=True)
+ rd = oe.recipeutils.parse_recipe(recipefile, None, tinfoil.config_data)
+ if not rd:
+ return 1
+
appendfile = os.path.join(appendpath, '%s.bbappend' % bp)
with open(appendfile, 'w') as f:
f.write('inherit externalsrc\n')
f.write('EXTERNALSRC = "%s"\n' % srctree)
- if args.same_dir:
+
+ b_is_s = use_external_build(args.same_dir, args.no_same_dir, rd)
+ if b_is_s:
f.write('EXTERNALSRC_BUILD = "%s"\n' % srctree)
if initial_rev:
f.write('\n# initial_rev: %s\n' % initial_rev)
_add_md5(config, args.recipename, appendfile)
+ tinfoil.shutdown()
+
return 0
@@ -483,18 +492,7 @@ def modify(args, config, basepath, workspace):
f.write('# NOTE: We use pn- overrides here to avoid affecting multiple variants in the case where the recipe uses BBCLASSEXTEND\n')
f.write('EXTERNALSRC_pn-%s = "%s"\n' % (args.recipename, srctree))
- b_is_s = True
- if args.no_same_dir:
- logger.info('using separate build directory since --no-same-dir specified')
- b_is_s = False
- elif args.same_dir:
- logger.info('using source tree as build directory since --same-dir specified')
- elif bb.data.inherits_class('autotools-brokensep', rd):
- logger.info('using source tree as build directory since original recipe inherits autotools-brokensep')
- elif rd.getVar('B', True) == s:
- logger.info('using source tree as build directory since that is the default for this recipe')
- else:
- b_is_s = False
+ b_is_s = use_external_build(args.same_dir, args.no_same_dir, rd)
if b_is_s:
f.write('EXTERNALSRC_BUILD_pn-%s = "%s"\n' % (args.recipename, srctree))
@@ -876,7 +874,9 @@ def register_commands(subparsers, context):
description='Adds a new recipe')
parser_add.add_argument('recipename', help='Name for new recipe to add')
parser_add.add_argument('srctree', help='Path to external source tree')
- parser_add.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
+ group = parser_add.add_mutually_exclusive_group()
+ group.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
+ group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree', metavar='URI')
parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
parser_add.set_defaults(func=add)
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 86443b0..6c1dfee 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -29,7 +29,7 @@ import errno
import bb
import oe.recipeutils
from devtool import standard
-from devtool import exec_build_env_command, setup_tinfoil, DevtoolError, parse_recipe
+from devtool import exec_build_env_command, setup_tinfoil, DevtoolError, parse_recipe, use_external_build
logger = logging.getLogger('devtool')
@@ -126,21 +126,6 @@ def _rename_recipe_files(bpn, oldpv, newpv, path):
_rename_recipe_dirs(oldpv, newpv, path)
return _rename_recipe_file(bpn, oldpv, newpv, path)
-def _use_external_build(same_dir, no_same_dir, d):
- b_is_s = True
- if no_same_dir:
- logger.info('using separate build directory since --no-same-dir specified')
- b_is_s = False
- elif same_dir:
- logger.info('using source tree as build directory since --same-dir specified')
- elif bb.data.inherits_class('autotools-brokensep', d):
- logger.info('using source tree as build directory since original recipe inherits autotools-brokensep')
- elif d.getVar('B', True) == os.path.abspath(d.getVar('S', True)):
- logger.info('using source tree as build directory since that is the default for this recipe')
- else:
- b_is_s = False
- return b_is_s
-
def _write_append(rc, srctree, same_dir, no_same_dir, rev, workspace, d):
"""Writes an append file"""
if not os.path.exists(rc):
@@ -161,7 +146,8 @@ def _write_append(rc, srctree, same_dir, no_same_dir, rev, workspace, d):
f.write(('# NOTE: We use pn- overrides here to avoid affecting'
'multiple variants in the case where the recipe uses BBCLASSEXTEND\n'))
f.write('EXTERNALSRC_pn-%s = "%s"\n' % (pn, srctree))
- if _use_external_build(same_dir, no_same_dir, d):
+ b_is_s = use_external_build(same_dir, no_same_dir, d)
+ if b_is_s:
f.write('EXTERNALSRC_BUILD_pn-%s = "%s"\n' % (pn, srctree))
if rev:
f.write('\n# initial_rev: %s\n' % rev)
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 14/27] recipetool: create: fix creating empty shell functions
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (12 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 13/27] devtool: add: properly handle separate build directory Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 15/27] recipetool: create: fix handling of URIs containing # Paul Eggleton
` (12 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
The shell considers empty functions to be a syntax error, so for
template shell functions that contain only comments (or no lines at all)
then add a : to act as a no-op which avoids the syntax error.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 22 +++++++++++++++++++---
1 file changed, 19 insertions(+), 3 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index c4754db..99d9cc8 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -46,10 +46,26 @@ class RecipeHandler():
results.extend(glob.glob(os.path.join(path, spec)))
return results
- def genfunction(self, outlines, funcname, content):
- outlines.append('%s () {' % funcname)
+ def genfunction(self, outlines, funcname, content, python=False, forcespace=False):
+ if python:
+ prefix = 'python '
+ else:
+ prefix = ''
+ outlines.append('%s%s () {' % (prefix, funcname))
+ if python or forcespace:
+ indent = ' '
+ else:
+ indent = '\t'
+ addnoop = not python
for line in content:
- outlines.append('\t%s' % line)
+ outlines.append('%s%s' % (indent, line))
+ if addnoop:
+ strippedline = line.lstrip()
+ if strippedline and not strippedline.startswith('#'):
+ addnoop = False
+ if addnoop:
+ # Without this there'll be a syntax error
+ outlines.append('%s:' % indent)
outlines.append('}')
outlines.append('')
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 15/27] recipetool: create: fix handling of URIs containing #
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (13 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 14/27] recipetool: create: fix creating empty shell functions Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 16/27] devtool: show proper error when extracting source for recipes with disabled unpack task Paul Eggleton
` (11 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
The # character in a URI denotes a fragment; we don't care about this
since it is never supposed to be sent to the server, so remove it from
the URI before actually trying to fetch it or use it in SRC_URI within
the recipe.
(This has come up because download links on pypi.python.org seem to have
a fragment containing the md5sum of the download; without stripping this
off the fetcher will choke on it.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 99d9cc8..409b255 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -23,6 +23,7 @@ import fnmatch
import re
import logging
import scriptutils
+import urlparse
logger = logging.getLogger('recipetool')
@@ -102,7 +103,8 @@ def create_recipe(args):
srcrev = '${AUTOREV}'
if '://' in args.source:
# Fetch a URL
- srcuri = args.source
+ fetchuri = urlparse.urldefrag(args.source)[0]
+ srcuri = fetchuri
rev_re = re.compile(';rev=([^;]+)')
res = rev_re.search(srcuri)
if res:
@@ -111,7 +113,7 @@ def create_recipe(args):
tempsrc = tempfile.mkdtemp(prefix='recipetool-')
srctree = tempsrc
logger.info('Fetching %s...' % srcuri)
- checksums = scriptutils.fetch_uri(tinfoil.config_data, args.source, srctree, srcrev)
+ checksums = scriptutils.fetch_uri(tinfoil.config_data, fetchuri, srctree, srcrev)
dirlist = os.listdir(srctree)
if 'git.indirectionsymlink' in dirlist:
dirlist.remove('git.indirectionsymlink')
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 16/27] devtool: show proper error when extracting source for recipes with disabled unpack task
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (14 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 15/27] recipetool: create: fix handling of URIs containing # Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 17/27] devtool: fix extracting source for work-shared recipes Paul Eggleton
` (10 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
If you try to use "devtool modify -x" or "devtool extract" on a recipe
where do_unpack has been set as noexec (e.g. glibc-locale), then we get
an error because the expected source wasn't ever unpacked. Do a check up
front for noexec being set on do_unpack and error out with a reasonable
message if that's the case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 700a56b..72ec97b 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -261,6 +261,10 @@ def _extract_source(srctree, keep_temp, devbranch, d):
raise DevtoolError("output path %s already exists and is "
"non-empty" % srctree)
+ if 'noexec' in (d.getVarFlags('do_unpack', False) or []):
+ raise DevtoolError("The %s recipe has do_unpack disabled, unable to "
+ "extract source" % pn)
+
# Prepare for shutil.move later on
bb.utils.mkdirhier(srctree)
os.rmdir(srctree)
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 17/27] devtool: fix extracting source for work-shared recipes
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (15 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 16/27] devtool: show proper error when extracting source for recipes with disabled unpack task Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 18/27] devtool: better handling for recipes that don't unpack source Paul Eggleton
` (9 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Recipes that use work-shared (such as libgcc) are capable of unpacking
the source, but it doesn't necessarily unpack to ${WORKDIR}/${BP}. Use
the last part of the actual S value instead which is more likely to
work.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 72ec97b..2dbe372 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -288,7 +288,7 @@ def _extract_source(srctree, keep_temp, devbranch, d):
if bb.data.inherits_class('kernel', d):
crd.setVar('S', '${WORKDIR}/source')
else:
- crd.setVar('S', '${WORKDIR}/${BP}')
+ crd.setVar('S', '${WORKDIR}/%s' % os.path.basename(d.getVar('S', True)))
if bb.data.inherits_class('kernel', d):
# We don't want to move the source to STAGING_KERNEL_DIR here
crd.setVar('STAGING_KERNEL_DIR', '${S}')
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 18/27] devtool: better handling for recipes that don't unpack source
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (16 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 17/27] devtool: fix extracting source for work-shared recipes Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 19/27] devtool: add: set up fetched source as a git repository by default Paul Eggleton
` (8 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Some recipes don't extract any source (for example, opkg-keyrings). We
were producing a traceback in this case because we weren't checking if
the directory existed.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 2dbe372..ff79c05 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -334,9 +334,11 @@ def _extract_source(srctree, keep_temp, devbranch, d):
else:
os.rmdir(patchdir)
- if not os.listdir(srcsubdir):
- raise DevtoolError("no source unpacked to S, perhaps the %s "
- "recipe doesn't use any source?" % pn)
+ if not os.path.exists(srcsubdir) or not os.listdir(srcsubdir):
+ raise DevtoolError("no source unpacked to S, either the %s "
+ "recipe doesn't use any source or the "
+ "correct source directory could not be "
+ "determined" % pn)
if not os.path.exists(os.path.join(srcsubdir, '.git')):
bb.process.run('git init', cwd=srcsubdir)
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 19/27] devtool: add: set up fetched source as a git repository by default
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (17 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 18/27] devtool: better handling for recipes that don't unpack source Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 20/27] devtool: add: move important "recipe created" message to the end Paul Eggleton
` (7 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
If the fetched source isn't already a git repository, initialise it as
one and then branch and tag, just as we do with "devtool modify". This
makes it easier to make changes, commit them and then use the
"devtool update-recipe" command to turn those commits into patches
on the recipe.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/__init__.py | 17 +++++++++++++++++
scripts/lib/devtool/standard.py | 13 ++++++-------
2 files changed, 23 insertions(+), 7 deletions(-)
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 07a3636..f815ef2 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -170,3 +170,20 @@ def use_external_build(same_dir, no_same_dir, d):
else:
b_is_s = False
return b_is_s
+
+def setup_git_repo(repodir, version, devbranch, basetag='devtool-base'):
+ """
+ Set up the git repository for the source tree
+ """
+ import bb.process
+ if not os.path.exists(os.path.join(repodir, '.git')):
+ bb.process.run('git init', cwd=repodir)
+ bb.process.run('git add .', cwd=repodir)
+ if version:
+ commitmsg = "Initial commit from upstream at version %s" % version
+ else:
+ commitmsg = "Initial commit from upstream"
+ bb.process.run('git commit -q -m "%s"' % commitmsg, cwd=repodir)
+
+ bb.process.run('git checkout -b %s' % devbranch, cwd=repodir)
+ bb.process.run('git tag -f %s' % basetag, cwd=repodir)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index ff79c05..4a05c1d 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -25,7 +25,7 @@ import logging
import argparse
import scriptutils
import errno
-from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, DevtoolError
from devtool import parse_recipe
logger = logging.getLogger('devtool')
@@ -102,6 +102,9 @@ def add(args, config, basepath, workspace):
_add_md5(config, args.recipename, recipefile)
+ if args.fetch and not args.no_git:
+ setup_git_repo(srctree, args.version, 'devtool')
+
initial_rev = None
if os.path.exists(os.path.join(srctree, '.git')):
(stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
@@ -340,16 +343,11 @@ def _extract_source(srctree, keep_temp, devbranch, d):
"correct source directory could not be "
"determined" % pn)
- if not os.path.exists(os.path.join(srcsubdir, '.git')):
- bb.process.run('git init', cwd=srcsubdir)
- bb.process.run('git add .', cwd=srcsubdir)
- bb.process.run('git commit -q -m "Initial commit from upstream at version %s"' % crd.getVar('PV', True), cwd=srcsubdir)
+ setup_git_repo(srcsubdir, crd.getVar('PV', True), devbranch)
(stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srcsubdir)
initial_rev = stdout.rstrip()
- bb.process.run('git checkout -b %s' % devbranch, cwd=srcsubdir)
- bb.process.run('git tag -f devtool-base', cwd=srcsubdir)
crd.setVar('PATCHTOOL', 'git')
logger.info('Patching...')
@@ -885,6 +883,7 @@ def register_commands(subparsers, context):
group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree', metavar='URI')
parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
+ parser_add.add_argument('--no-git', '-g', help='If -f/--fetch is specified, do not set up source tree as a git repository', action="store_true")
parser_add.set_defaults(func=add)
parser_modify = subparsers.add_parser('modify', help='Modify the source for an existing recipe',
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 20/27] devtool: add: move important "recipe created" message to the end
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (18 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 19/27] devtool: add: set up fetched source as a git repository by default Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 21/27] devtool: build-image: fix recipe/package terminology Paul Eggleton
` (6 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
If we end up printing a message about the build directory being the same
as the source, we should print that first and then print the message
about the recipe file possibly needing to be edited to the end so that
it has slightly more impact.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 4a05c1d..c254132 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -96,7 +96,6 @@ def add(args, config, basepath, workspace):
extracmdopts += ' -V %s' % args.version
try:
stdout, _ = exec_build_env_command(config.init_path, basepath, 'recipetool --color=%s create -o %s "%s" %s' % (color, recipefile, source, extracmdopts))
- logger.info('Recipe %s has been automatically created; further editing may be required to make it fully functional' % recipefile)
except bb.process.ExecutionError as e:
raise DevtoolError('Command \'%s\' failed:\n%s' % (e.command, e.stdout))
@@ -128,6 +127,8 @@ def add(args, config, basepath, workspace):
_add_md5(config, args.recipename, appendfile)
+ logger.info('Recipe %s has been automatically created; further editing may be required to make it fully functional' % recipefile)
+
tinfoil.shutdown()
return 0
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 21/27] devtool: build-image: fix recipe/package terminology
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (19 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 20/27] devtool: add: move important "recipe created" message to the end Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 22/27] devtool: build-image: tell user where to find output files Paul Eggleton
` (5 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
We build recipes and include packages into the image, adjust the
terminology used in code and messages accordingly. Also fix a few typos.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/build-image.py | 45 ++++++++++++++++++++------------------
1 file changed, 24 insertions(+), 21 deletions(-)
diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index 2c01428..fa6f0d7 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -25,8 +25,8 @@ from devtool import exec_build_env_command, setup_tinfoil, parse_recipe
logger = logging.getLogger('devtool')
-def _get_recipes(workspace, config):
- """Get list of target recipes from the workspace."""
+def _get_packages(workspace, config):
+ """Get list of packages from recipes in the workspace."""
result = []
tinfoil = setup_tinfoil()
for recipe in workspace:
@@ -35,7 +35,7 @@ def _get_recipes(workspace, config):
if recipe in data.getVar('PACKAGES', True):
result.append(recipe)
else:
- logger.warning("Skipping recipe %s as it doesn't produce "
+ logger.warning("Skipping recipe %s as it doesn't produce a "
"package with the same name", recipe)
tinfoil.shutdown()
return result
@@ -46,29 +46,32 @@ def build_image(args, config, basepath, workspace):
appendfile = os.path.join(config.workspace_path, 'appends',
'%s.bbappend' % image)
- # remove <image>.bbapend to make sure setup_tinfoil doesn't
- # breake because of it
+ # remove <image>.bbappend to make sure setup_tinfoil doesn't
+ # break because of it
if os.path.isfile(appendfile):
os.unlink(appendfile)
- recipes = _get_recipes(workspace, config)
- if recipes:
- with open(appendfile, 'w') as afile:
- # include selected recipes into the image
- afile.write('IMAGE_INSTALL_append = " %s"\n' % ' '.join(recipes))
+ if workspace:
+ packages = _get_packages(workspace, config)
+ if packages:
+ with open(appendfile, 'w') as afile:
+ # include packages from workspace recipes into the image
+ afile.write('IMAGE_INSTALL_append = " %s"\n' % ' '.join(packages))
- # Generate notification callback devtool_warn_image_extended
- afile.write('do_rootfs[prefuncs] += "devtool_warn_image_extended"\n\n')
- afile.write("python devtool_warn_image_extended() {\n")
- afile.write(" bb.plain('NOTE: %%s: building with additional '\n"
- " 'packages due to \"devtool build-image\"'"
- " %% d.getVar('PN', True))\n"
- " bb.plain('NOTE: delete %%s to clear this' %% \\\n"
- " '%s')\n" % os.path.relpath(appendfile, basepath))
- afile.write("}\n")
+ # Generate notification callback devtool_warn_image_extended
+ afile.write('do_rootfs[prefuncs] += "devtool_warn_image_extended"\n\n')
+ afile.write("python devtool_warn_image_extended() {\n")
+ afile.write(" bb.plain('NOTE: %%s: building with additional '\n"
+ " 'packages due to \"devtool build-image\"'"
+ " %% d.getVar('PN', True))\n"
+ " bb.plain('NOTE: delete %%s to clear this' %% \\\n"
+ " '%s')\n" % os.path.relpath(appendfile, basepath))
+ afile.write("}\n")
- logger.info('Building image %s with the following '
- 'additional packages: %s', image, ' '.join(recipes))
+ logger.info('Building image %s with the following '
+ 'additional packages: %s', image, ' '.join(packages))
+ else:
+ logger.warning('No packages to add, building image %s unmodified', image)
else:
logger.warning('No recipes in workspace, building image %s unmodified', image)
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 22/27] devtool: build-image: tell user where to find output files
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (20 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 21/27] devtool: build-image: fix recipe/package terminology Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 23/27] devtool: build-image: improve image recipe handling Paul Eggleton
` (4 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
If the user is running "devtool build-image" within the extensible SDK
then they probably won't know where to find the resulting output files,
so we should tell them explicitly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/build-image.py | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index fa6f0d7..5759bec 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -25,10 +25,9 @@ from devtool import exec_build_env_command, setup_tinfoil, parse_recipe
logger = logging.getLogger('devtool')
-def _get_packages(workspace, config):
+def _get_packages(tinfoil, workspace, config):
"""Get list of packages from recipes in the workspace."""
result = []
- tinfoil = setup_tinfoil()
for recipe in workspace:
data = parse_recipe(config, tinfoil, recipe, True)
if 'class-target' in data.getVar('OVERRIDES', True).split(':'):
@@ -37,7 +36,6 @@ def _get_packages(workspace, config):
else:
logger.warning("Skipping recipe %s as it doesn't produce a "
"package with the same name", recipe)
- tinfoil.shutdown()
return result
def build_image(args, config, basepath, workspace):
@@ -51,8 +49,9 @@ def build_image(args, config, basepath, workspace):
if os.path.isfile(appendfile):
os.unlink(appendfile)
+ tinfoil = setup_tinfoil()
if workspace:
- packages = _get_packages(workspace, config)
+ packages = _get_packages(tinfoil, workspace, config)
if packages:
with open(appendfile, 'w') as afile:
# include packages from workspace recipes into the image
@@ -75,6 +74,10 @@ def build_image(args, config, basepath, workspace):
else:
logger.warning('No recipes in workspace, building image %s unmodified', image)
+ deploy_dir_image = tinfoil.config_data.getVar('DEPLOY_DIR_IMAGE', True)
+
+ tinfoil.shutdown()
+
# run bitbake to build image
try:
exec_build_env_command(config.init_path, basepath,
@@ -82,7 +85,8 @@ def build_image(args, config, basepath, workspace):
except ExecutionError as err:
return err.exitcode
- logger.info('Successfully built %s', image)
+ logger.info('Successfully built %s. You can find output files in %s'
+ % (image, deploy_dir_image))
def register_commands(subparsers, context):
"""Register devtool subcommands from the build-image plugin"""
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 23/27] devtool: build-image: improve image recipe handling
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (21 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 22/27] devtool: build-image: tell user where to find output files Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 24/27] devtool: build-image: delete bbappend at end of build Paul Eggleton
` (3 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
* Make image optional for the extensible SDK (auto-determine it based on
the targets the SDK was built for)
* Check that specified recipe is in fact an image
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/build-image.py | 26 +++++++++++++++++++++++---
1 file changed, 23 insertions(+), 3 deletions(-)
diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index 5759bec..a6c7d81 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -21,7 +21,7 @@ import os
import logging
from bb.process import ExecutionError
-from devtool import exec_build_env_command, setup_tinfoil, parse_recipe
+from devtool import exec_build_env_command, setup_tinfoil, parse_recipe, DevtoolError
logger = logging.getLogger('devtool')
@@ -40,7 +40,17 @@ def _get_packages(tinfoil, workspace, config):
def build_image(args, config, basepath, workspace):
"""Entry point for the devtool 'build-image' subcommand."""
- image = args.recipe
+
+ image = args.imagename
+ auto_image = False
+ if not image:
+ sdk_targets = config.get('SDK', 'sdk_targets', '').split()
+ if sdk_targets:
+ image = sdk_targets[0]
+ auto_image = True
+ if not image:
+ raise DevtoolError('Unable to determine image to build, please specify one')
+
appendfile = os.path.join(config.workspace_path, 'appends',
'%s.bbappend' % image)
@@ -50,6 +60,16 @@ def build_image(args, config, basepath, workspace):
os.unlink(appendfile)
tinfoil = setup_tinfoil()
+ rd = parse_recipe(config, tinfoil, image, True)
+ if not rd:
+ # Error already shown
+ return 1
+ if not bb.data.inherits_class('image', rd):
+ if auto_image:
+ raise DevtoolError('Unable to determine image to build, please specify one')
+ else:
+ raise DevtoolError('Specified recipe %s is not an image recipe' % image)
+
if workspace:
packages = _get_packages(tinfoil, workspace, config)
if packages:
@@ -94,5 +114,5 @@ def register_commands(subparsers, context):
help='Build image including workspace recipe packages',
description='Builds an image, extending it to include '
'packages from recipes in the workspace')
- parser.add_argument('recipe', help='Image recipe to build')
+ parser.add_argument('imagename', help='Image recipe to build', nargs='?')
parser.set_defaults(func=build_image)
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 24/27] devtool: build-image: delete bbappend at end of build
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (22 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 23/27] devtool: build-image: improve image recipe handling Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 25/27] devtool / recipetool: add handling for binary-only packages Paul Eggleton
` (2 subsequent siblings)
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Upon further reflection, it seems to me that this bbappend ought to just
be deleted at the end of the build. This keeps things simple; you never
have to remember to delete any files to get back to where you were
before with the image. This means we can also drop the slightly awkward
message reminding the user how to do that. I've also updated the test to
look at the image manifest to determine if the command has worked
instead of looking for the (now deleted) bbappend.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oeqa/selftest/devtool.py | 22 +++++++++++-----
scripts/lib/devtool/build-image.py | 53 +++++++++++++++++---------------------
2 files changed, 38 insertions(+), 37 deletions(-)
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index 6e731d6..3a8168c 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -919,7 +919,8 @@ class DevtoolTests(DevtoolBase):
self.add_command_to_tearDown('bitbake -c clean %s' % image)
bitbake('%s -c clean' % image)
# Add target and native recipes to workspace
- for recipe in ('mdadm', 'parted-native'):
+ recipes = ['mdadm', 'parted-native']
+ for recipe in recipes:
tempdir = tempfile.mkdtemp(prefix='devtoolqa')
self.track_for_cleanup(tempdir)
self.add_command_to_tearDown('bitbake -c clean %s' % recipe)
@@ -927,12 +928,19 @@ class DevtoolTests(DevtoolBase):
# Try to build image
result = runCmd('devtool build-image %s' % image)
self.assertNotEqual(result, 0, 'devtool build-image failed')
- # Check if image.bbappend has required content
- bbappend = os.path.join(workspacedir, 'appends', image+'.bbappend')
- self.assertTrue(os.path.isfile(bbappend), 'bbappend not created %s' % result.output)
- # NOTE: native recipe parted-native should not be in IMAGE_INSTALL_append
- self.assertTrue('IMAGE_INSTALL_append = " mdadm"\n' in open(bbappend).readlines(),
- 'IMAGE_INSTALL_append = " mdadm" not found in %s' % bbappend)
+ # Check if image contains expected packages
+ deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE')
+ image_link_name = get_bb_var('IMAGE_LINK_NAME', image)
+ reqpkgs = [item for item in recipes if not item.endswith('-native')]
+ with open(os.path.join(deploy_dir_image, image_link_name + '.manifest'), 'r') as f:
+ for line in f:
+ splitval = line.split()
+ if splitval:
+ pkg = splitval[0]
+ if pkg in reqpkgs:
+ reqpkgs.remove(pkg)
+ if reqpkgs:
+ self.fail('The following packages were not present in the image as expected: %s' % ', '.join(reqpkgs))
def test_devtool_upgrade(self):
# Check preconditions
diff --git a/scripts/lib/devtool/build-image.py b/scripts/lib/devtool/build-image.py
index a6c7d81..05c1d75 100644
--- a/scripts/lib/devtool/build-image.py
+++ b/scripts/lib/devtool/build-image.py
@@ -70,40 +70,33 @@ def build_image(args, config, basepath, workspace):
else:
raise DevtoolError('Specified recipe %s is not an image recipe' % image)
- if workspace:
- packages = _get_packages(tinfoil, workspace, config)
- if packages:
- with open(appendfile, 'w') as afile:
- # include packages from workspace recipes into the image
- afile.write('IMAGE_INSTALL_append = " %s"\n' % ' '.join(packages))
-
- # Generate notification callback devtool_warn_image_extended
- afile.write('do_rootfs[prefuncs] += "devtool_warn_image_extended"\n\n')
- afile.write("python devtool_warn_image_extended() {\n")
- afile.write(" bb.plain('NOTE: %%s: building with additional '\n"
- " 'packages due to \"devtool build-image\"'"
- " %% d.getVar('PN', True))\n"
- " bb.plain('NOTE: delete %%s to clear this' %% \\\n"
- " '%s')\n" % os.path.relpath(appendfile, basepath))
- afile.write("}\n")
-
- logger.info('Building image %s with the following '
- 'additional packages: %s', image, ' '.join(packages))
+ try:
+ if workspace:
+ packages = _get_packages(tinfoil, workspace, config)
+ if packages:
+ with open(appendfile, 'w') as afile:
+ # include packages from workspace recipes into the image
+ afile.write('IMAGE_INSTALL_append = " %s"\n' % ' '.join(packages))
+ logger.info('Building image %s with the following '
+ 'additional packages: %s', image, ' '.join(packages))
+ else:
+ logger.warning('No packages to add, building image %s unmodified', image)
else:
- logger.warning('No packages to add, building image %s unmodified', image)
- else:
- logger.warning('No recipes in workspace, building image %s unmodified', image)
+ logger.warning('No recipes in workspace, building image %s unmodified', image)
- deploy_dir_image = tinfoil.config_data.getVar('DEPLOY_DIR_IMAGE', True)
+ deploy_dir_image = tinfoil.config_data.getVar('DEPLOY_DIR_IMAGE', True)
- tinfoil.shutdown()
+ tinfoil.shutdown()
- # run bitbake to build image
- try:
- exec_build_env_command(config.init_path, basepath,
- 'bitbake %s' % image, watch=True)
- except ExecutionError as err:
- return err.exitcode
+ # run bitbake to build image
+ try:
+ exec_build_env_command(config.init_path, basepath,
+ 'bitbake %s' % image, watch=True)
+ except ExecutionError as err:
+ return err.exitcode
+ finally:
+ if os.path.isfile(appendfile):
+ os.unlink(appendfile)
logger.info('Successfully built %s. You can find output files in %s'
% (image, deploy_dir_image))
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 25/27] devtool / recipetool: add handling for binary-only packages
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (23 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 24/27] devtool: build-image: delete bbappend at end of build Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 26/27] devtool: add basic means of running runqemu within the extensible SDK Paul Eggleton
2015-09-22 16:21 ` [PATCH 27/27] devtool: add search command Paul Eggleton
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Add a means of creating recipes for package files or archives that
contain a directory structure to be installed verbatim, for example an
rpm file. (We mostly just re-use bin_package here and skip some of the
normal build system checks.) This support is available in "recipetool
create" and "devtool add" which wraps the former.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 9 +++++++++
scripts/lib/recipetool/create.py | 14 ++++++++++++++
2 files changed, 23 insertions(+)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index c254132..96b271c 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -94,6 +94,8 @@ def add(args, config, basepath, workspace):
source = srctree
if args.version:
extracmdopts += ' -V %s' % args.version
+ if args.binary:
+ extracmdopts += ' -b'
try:
stdout, _ = exec_build_env_command(config.init_path, basepath, 'recipetool --color=%s create -o %s "%s" %s' % (color, recipefile, source, extracmdopts))
except bb.process.ExecutionError as e:
@@ -125,6 +127,12 @@ def add(args, config, basepath, workspace):
if initial_rev:
f.write('\n# initial_rev: %s\n' % initial_rev)
+ if args.binary:
+ f.write('do_install_append() {\n')
+ f.write(' rm -rf ${D}/.git\n')
+ f.write(' rm -f ${D}/singletask.lock\n')
+ f.write('}\n')
+
_add_md5(config, args.recipename, appendfile)
logger.info('Recipe %s has been automatically created; further editing may be required to make it fully functional' % recipefile)
@@ -885,6 +893,7 @@ def register_commands(subparsers, context):
parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree', metavar='URI')
parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
parser_add.add_argument('--no-git', '-g', help='If -f/--fetch is specified, do not set up source tree as a git repository', action="store_true")
+ parser_add.add_argument('--binary', '-b', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true')
parser_add.set_defaults(func=add)
parser_modify = subparsers.add_parser('modify', help='Modify the source for an existing recipe',
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 409b255..844073b 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -104,6 +104,10 @@ def create_recipe(args):
if '://' in args.source:
# Fetch a URL
fetchuri = urlparse.urldefrag(args.source)[0]
+ if args.binary:
+ # Assume the archive contains the directory structure verbatim
+ # so we need to extract to a subdirectory
+ fetchuri += ';subdir=%s' % os.path.splitext(os.path.basename(urlparse.urlsplit(fetchuri).path))[0]
srcuri = fetchuri
rev_re = re.compile(';rev=([^;]+)')
res = rev_re.search(srcuri)
@@ -226,6 +230,10 @@ def create_recipe(args):
lines_after.append('PACKAGE_ARCH = "%s"' % pkgarch)
lines_after.append('')
+ if args.binary:
+ lines_after.append('INSANE_SKIP_${PN} += "already-stripped"')
+ lines_after.append('')
+
# Find all plugins that want to register handlers
handlers = []
for plugin in plugins:
@@ -235,6 +243,11 @@ def create_recipe(args):
# Apply the handlers
classes = []
handled = []
+
+ if args.binary:
+ classes.append('bin_package')
+ handled.append('buildsystem')
+
for handler in handlers:
handler.process(srctree, classes, lines_before, lines_after, handled)
@@ -426,5 +439,6 @@ def register_command(subparsers):
parser_create.add_argument('-m', '--machine', help='Make recipe machine-specific as opposed to architecture-specific', action='store_true')
parser_create.add_argument('-x', '--extract-to', metavar='EXTRACTPATH', help='Assuming source is a URL, fetch it and extract it to the directory specified as %(metavar)s')
parser_create.add_argument('-V', '--version', help='Version to use within recipe (PV)')
+ parser_create.add_argument('-b', '--binary', help='Treat the source tree as something that should be installed verbatim (no compilation, same directory structure)', action='store_true')
parser_create.set_defaults(func=create_recipe)
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 26/27] devtool: add basic means of running runqemu within the extensible SDK
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (24 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 25/27] devtool / recipetool: add handling for binary-only packages Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
2015-09-22 16:21 ` [PATCH 27/27] devtool: add search command Paul Eggleton
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
We ship the runqemu script and if we build QEMU itself within the
extensible SDK, then it would be nice to be able to run it. This is a
very thin wrapper around runqemu, supplying the machine and image name
so the user doesn't need to. (This subcommand is only available within
the extensible SDK since it only really makes sense there where it is
otherwise hard to run runqemu directly.)
Implements [YOCTO #6657].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/runqemu.py | 64 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 64 insertions(+)
create mode 100644 scripts/lib/devtool/runqemu.py
diff --git a/scripts/lib/devtool/runqemu.py b/scripts/lib/devtool/runqemu.py
new file mode 100644
index 0000000..e7f26ab
--- /dev/null
+++ b/scripts/lib/devtool/runqemu.py
@@ -0,0 +1,64 @@
+# Development tool - runqemu command plugin
+#
+# Copyright (C) 2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+"""Devtool runqemu plugin"""
+
+import os
+import bb
+import logging
+import argparse
+import glob
+from devtool import exec_build_env_command, setup_tinfoil, DevtoolError
+
+logger = logging.getLogger('devtool')
+
+def runqemu(args, config, basepath, workspace):
+ """Entry point for the devtool 'runqemu' subcommand"""
+
+ tinfoil = setup_tinfoil()
+ machine = tinfoil.config_data.getVar('MACHINE', True)
+ bindir_native = tinfoil.config_data.getVar('STAGING_BINDIR_NATIVE', True)
+ tinfoil.shutdown()
+
+ if not glob.glob(os.path.join(bindir_native, 'qemu-system-*')):
+ raise DevtoolError('QEMU is not available within this SDK')
+
+ imagename = args.imagename
+ if not imagename:
+ sdk_targets = config.get('SDK', 'sdk_targets', '').split()
+ if sdk_targets:
+ imagename = sdk_targets[0]
+ if not imagename:
+ raise DevtoolError('Unable to determine image name to run, please specify one')
+
+ try:
+ exec_build_env_command(config.init_path, basepath, 'runqemu %s %s %s' % (machine, imagename, " ".join(args.args)), watch=True)
+ except bb.process.ExecutionError as e:
+ # We've already seen the output since watch=True, so just ensure we return something to the user
+ return e.exitcode
+
+ return 0
+
+def register_commands(subparsers, context):
+ """Register devtool subcommands from this plugin"""
+ if context.fixed_setup:
+ parser_runqemu = subparsers.add_parser('runqemu', help='Run QEMU on the specified image',
+ description='Runs QEMU to boot the specified image')
+ parser_runqemu.add_argument('imagename', help='Name of built image to boot within QEMU', nargs='?')
+ parser_runqemu.add_argument('args', help='Any remaining arguments are passed to the runqemu script (pass --help after imagename to see what these are)',
+ nargs=argparse.REMAINDER)
+ parser_runqemu.set_defaults(func=runqemu)
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread* [PATCH 27/27] devtool: add search command
2015-09-22 16:21 [PATCH 00/27] Fixes for devtool and the extensible SDK Paul Eggleton
` (25 preceding siblings ...)
2015-09-22 16:21 ` [PATCH 26/27] devtool: add basic means of running runqemu within the extensible SDK Paul Eggleton
@ 2015-09-22 16:21 ` Paul Eggleton
26 siblings, 0 replies; 29+ messages in thread
From: Paul Eggleton @ 2015-09-22 16:21 UTC (permalink / raw)
To: openembedded-core
Adds a subcommand to search to find the target recipe name providing
some file or capability. This is implemented by searching on recipe
name, package name, description, package contents (file names), and
runtime file provides. For example:
$ devtool search libGL
mesa
$ devtool search X11
xextproto
libxxf86vm
xf86driproto
xf86vidmodeproto
libxfixes
xproto
libx11
...
$ devtool search /bin/sed
busybox
sed
This is particularly useful within the extensible SDK but is also made
available in devtool alongside the build system.
Note of course that because this searches pkgdata, useful results depend
upon do_packagedata(_setscene) having executed for the recipe being
searched for.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/search.py | 80 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 80 insertions(+)
create mode 100644 scripts/lib/devtool/search.py
diff --git a/scripts/lib/devtool/search.py b/scripts/lib/devtool/search.py
new file mode 100644
index 0000000..e6ae922
--- /dev/null
+++ b/scripts/lib/devtool/search.py
@@ -0,0 +1,80 @@
+# Development tool - search command plugin
+#
+# Copyright (C) 2015 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+
+"""Devtool search plugin"""
+
+import os
+import bb
+import logging
+import argparse
+import re
+from devtool import setup_tinfoil, DevtoolError
+
+logger = logging.getLogger('devtool')
+
+def search(args, config, basepath, workspace):
+ """Entry point for the devtool 'search' subcommand"""
+
+ tinfoil = setup_tinfoil(config_only=True)
+ pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR', True)
+ tinfoil.shutdown()
+
+ keyword_rc = re.compile(args.keyword)
+
+ for fn in os.listdir(pkgdata_dir):
+ pfn = os.path.join(pkgdata_dir, fn)
+ if not os.path.isfile(pfn):
+ continue
+
+ packages = []
+ match = False
+ if keyword_rc.search(fn):
+ match = True
+
+ if not match:
+ with open(pfn, 'r') as f:
+ for line in f:
+ if line.startswith('PACKAGES:'):
+ packages = line.split(':', 1)[1].strip().split()
+
+ for pkg in packages:
+ if keyword_rc.search(pkg):
+ match = True
+ break
+ if os.path.exists(os.path.join(pkgdata_dir, 'runtime', pkg + '.packaged')):
+ with open(os.path.join(pkgdata_dir, 'runtime', pkg), 'r') as f:
+ for line in f:
+ if ': ' in line:
+ splitline = line.split(':', 1)
+ key = splitline[0]
+ value = splitline[1].strip()
+ if key in ['PKG_%s' % pkg, 'DESCRIPTION', 'FILES_INFO'] or key.startswith('FILERPROVIDES_'):
+ if keyword_rc.search(value):
+ match = True
+ break
+
+ if match:
+ print(fn)
+
+ return 0
+
+def register_commands(subparsers, context):
+ """Register devtool subcommands from this plugin"""
+ parser_search = subparsers.add_parser('search', help='Search available recipes',
+ description='Searches for available target recipes. Matches on recipe name, package name, description and installed files, and prints the recipe name on match.')
+ parser_search.add_argument('keyword', help='Keyword to search for (regular expression syntax allowed)')
+ parser_search.set_defaults(func=search)
--
2.1.0
^ permalink raw reply related [flat|nested] 29+ messages in thread