* [PATCH 00/12] devtool improvements
@ 2017-12-05 1:41 Paul Eggleton
2017-12-05 1:41 ` [PATCH 01/12] devtool: refactor return for _extract_source() Paul Eggleton
` (13 more replies)
0 siblings, 14 replies; 16+ messages in thread
From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw)
To: openembedded-core
A set of improvements for devtool, mostly aimed at improving the
handling of recipes that fetch multiple source trees (i.e. have multiple
tarballs or repositories in SRC_URI), with a couple of other fixes
thrown in.
The following changes since commit a7cd9d1183be603777fc9c8c448281fe01224f7b:
lib/oe/utils: remove param_bool() (2017-12-02 11:24:38 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib paule/devtool32-oe
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/devtool32-oe
Paul Eggleton (12):
devtool: refactor return for _extract_source()
devtool: reset: delete bbappend file if _check_preserve() doesn't
devtool: finish: fix erroneously creating bbappend for relative paths
devtool: deploy-target: don't specify ssh/scp port unless user does
devtool: extract: fix usage with kern-tools-native
devtool: modify: pick up commits from previously created source tree
lib/oe/recipeutils: allow patch_recipe_file() to be re-called
devtool: support extracting multiple source trees
devtool: fix handling of linux-yocto after multisrc changes
oe-selftest: devtool: fix kernel test for multisrc changes
oe-selftest: devtool: add test for multiple source trees
lib/oe/recipeutils: add .txz extension to archive list
.../devtool/devtool-test-multisrc-subdir_1.0.bb | 13 +
.../devtool-test-multisrc/example-files.tar.gz | Bin 0 -> 236 bytes
.../devtool/devtool-test-multisrc/example.patch | 7 +
.../devtool-test-multisrc/mypackage-1.0.tar.gz | Bin 0 -> 903 bytes
.../devtool/devtool-test-multisrc_1.0.bb | 11 +
meta/classes/devtool-source.bbclass | 63 ++-
meta/classes/externalsrc.bbclass | 2 +-
meta/classes/kernel-yocto.bbclass | 7 +-
meta/lib/oe/recipeutils.py | 15 +-
meta/lib/oe/utils.py | 24 ++
meta/lib/oeqa/selftest/cases/devtool.py | 124 +++++-
scripts/devtool | 61 ++-
scripts/lib/devtool/__init__.py | 14 +
scripts/lib/devtool/deploy.py | 12 +-
scripts/lib/devtool/standard.py | 426 ++++++++++++++-------
scripts/lib/devtool/upgrade.py | 4 +-
16 files changed, 601 insertions(+), 182 deletions(-)
create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc-subdir_1.0.bb
create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example-files.tar.gz
create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch
create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/mypackage-1.0.tar.gz
create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc_1.0.bb
--
2.9.5
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH 01/12] devtool: refactor return for _extract_source() 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 1:41 ` [PATCH 02/12] devtool: reset: delete bbappend file if _check_preserve() doesn't Paul Eggleton ` (12 subsequent siblings) 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core Use a namedtuple to return information to the caller, since I've been expanding that information we should avoid having to change all of the calling code each time. Additionally, it turned out that a bunch of the callers were checking for None being returned in the initial_rev value, but that's no longer possible, so tidy up the calling code. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- scripts/lib/devtool/standard.py | 29 ++++++++++------------------- scripts/lib/devtool/upgrade.py | 4 ++-- 2 files changed, 12 insertions(+), 21 deletions(-) diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index 26187a0..b1669b5 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -29,7 +29,7 @@ import scriptutils import errno import glob import filecmp -from collections import OrderedDict +from collections import OrderedDict, namedtuple from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, update_unlockedsigs, check_prerelease_version, check_git_repo_dirty, check_git_repo_op, DevtoolError from devtool import parse_recipe @@ -437,15 +437,11 @@ def extract(args, config, basepath, workspace): return 1 srctree = os.path.abspath(args.srctree) - initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) + _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) logger.info('Source tree extracted to %s' % srctree) - - if initial_rev: - return 0 - else: - return 1 finally: tinfoil.shutdown() + return 0 def sync(args, config, basepath, workspace): """Entry point for the devtool 'sync' subcommand""" @@ -461,17 +457,15 @@ def sync(args, config, basepath, workspace): return 1 srctree = os.path.abspath(args.srctree) - initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, True, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=True) + _extract_source(srctree, args.keep_temp, args.branch, True, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=True) logger.info('Source tree %s synchronized' % srctree) - - if initial_rev: - return 0 - else: - return 1 finally: tinfoil.shutdown() + return 0 +ExtractSourceResult = namedtuple('ExtractSourceResult', ['initial_rev', 'srcsubdir']) + def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, workspace, fixed_setup, d, tinfoil, no_overrides=False): """Extract sources of a recipe""" import oe.recipeutils @@ -512,8 +506,6 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works if extra_overrides: logger.info('SRC_URI contains some conditional appends/prepends - will create branches to represent these') - initial_rev = None - appendexisted = False recipefile = d.getVar('FILE') appendfile = recipe_to_append(recipefile, config) @@ -659,7 +651,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works logger.info('Preserving temporary directory %s' % tempdir) else: shutil.rmtree(tempdir) - return initial_rev, srcsubdir_rel + return ExtractSourceResult(initial_rev, srcsubdir_rel) def _add_md5(config, recipename, filename): """Record checksum of a file (or recursively for a directory) to the md5-file of the workspace""" @@ -759,9 +751,8 @@ def modify(args, config, basepath, workspace): commits = [] check_commits = False if not args.no_extract: - initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) - if not initial_rev: - return 1 + ret = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) + initial_rev = ret.initial_rev logger.info('Source tree extracted to %s' % srctree) # Get list of commits since this revision (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree) diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py index f6141bf..7afdd80 100644 --- a/scripts/lib/devtool/upgrade.py +++ b/scripts/lib/devtool/upgrade.py @@ -474,12 +474,12 @@ def upgrade(args, config, basepath, workspace): rf = None try: logger.info('Extracting current version source...') - rev1, srcsubdir1 = standard._extract_source(srctree, False, 'devtool-orig', False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) + current_ret = standard._extract_source(srctree, False, 'devtool-orig', False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) logger.info('Extracting upgraded version source...') rev2, md5, sha256, srcbranch, srcsubdir2 = _extract_new_source(args.version, srctree, args.no_patch, args.srcrev, args.srcbranch, args.branch, args.keep_temp, tinfoil, rd) - rf, copied = _create_new_recipe(args.version, md5, sha256, args.srcrev, srcbranch, srcsubdir1, srcsubdir2, config.workspace_path, tinfoil, rd) + rf, copied = _create_new_recipe(args.version, md5, sha256, args.srcrev, srcbranch, current_ret.srcsubdir, srcsubdir2, config.workspace_path, tinfoil, rd) except bb.process.CmdError as e: _upgrade_error(e, rf, srctree) except DevtoolError as e: -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 02/12] devtool: reset: delete bbappend file if _check_preserve() doesn't 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton 2017-12-05 1:41 ` [PATCH 01/12] devtool: refactor return for _extract_source() Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 1:41 ` [PATCH 03/12] devtool: finish: fix erroneously creating bbappend for relative paths Paul Eggleton ` (11 subsequent siblings) 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core If the .devtool_md5 file doesn't contain a reference to the bbappend file (e.g. because devtool was interrupted before it could write that out) then _check_preserve() won't delete it, so we need to delete it separately because otherwise the recipe won't actually be reset. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- scripts/lib/devtool/standard.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index b1669b5..f59c6d6 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -1781,6 +1781,13 @@ def _reset(recipes, no_clean, config, basepath, workspace): for pn in recipes: _check_preserve(config, pn) + appendfile = workspace[pn]['bbappend'] + if os.path.exists(appendfile): + # This shouldn't happen, but is possible if devtool errored out prior to + # writing the md5 file. We need to delete this here or the recipe won't + # actually be reset + os.remove(appendfile) + preservepath = os.path.join(config.workspace_path, 'attic', pn, pn) def preservedir(origdir): if os.path.exists(origdir): -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 03/12] devtool: finish: fix erroneously creating bbappend for relative paths 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton 2017-12-05 1:41 ` [PATCH 01/12] devtool: refactor return for _extract_source() Paul Eggleton 2017-12-05 1:41 ` [PATCH 02/12] devtool: reset: delete bbappend file if _check_preserve() doesn't Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 1:41 ` [PATCH 04/12] devtool: deploy-target: don't specify ssh/scp port unless user does Paul Eggleton ` (10 subsequent siblings) 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core After OE-Core rev 5e3fe00a0233d563781849a44f53885b4e924a9c we call os.path.abspath() on the original layer path, but we later compare that to the destination layer path. If that layer path isn't absolute but is effectively the same path, it should be writing to the original recipe but because we weren't making it absolute we were writing a bbappend instead. Call os.path.abspath() on the destination path as well to avoid that. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- scripts/lib/devtool/standard.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index f59c6d6..cdd2346 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -1848,9 +1848,7 @@ def _get_layer(layername, d): layerdir = layers.get('meta', None) else: layerdir = layers.get(layername, None) - if layerdir: - layerdir = os.path.abspath(layerdir) - return layerdir or layername + return os.path.abspath(layerdir or layername) def finish(args, config, basepath, workspace): """Entry point for the devtool 'finish' subcommand""" -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 04/12] devtool: deploy-target: don't specify ssh/scp port unless user does 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton ` (2 preceding siblings ...) 2017-12-05 1:41 ` [PATCH 03/12] devtool: finish: fix erroneously creating bbappend for relative paths Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 1:41 ` [PATCH 05/12] devtool: extract: fix usage with kern-tools-native Paul Eggleton ` (9 subsequent siblings) 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core If the user doesn't specify a port then we should avoid specifying one on the ssh/scp command line in case the user has configured one for the host they are connecting to, which was being overridden unnecessarily. Fixes [YOCTO #12381]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- scripts/lib/devtool/deploy.py | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/scripts/lib/devtool/deploy.py b/scripts/lib/devtool/deploy.py index 9cc4927..52e261d 100644 --- a/scripts/lib/devtool/deploy.py +++ b/scripts/lib/devtool/deploy.py @@ -213,9 +213,7 @@ def deploy(args, config, basepath, workspace): scp_port = '' ssh_port = '' - if not args.port: - raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target") - else: + if args.port: scp_port = "-P %s" % args.port ssh_port = "-p %s" % args.port @@ -280,9 +278,7 @@ def undeploy(args, config, basepath, workspace): scp_port = '' ssh_port = '' - if not args.port: - raise DevtoolError("If you specify -P/--port then you must provide the port to be used to connect to the target") - else: + if args.port: scp_port = "-P %s" % args.port ssh_port = "-p %s" % args.port @@ -328,7 +324,7 @@ def register_commands(subparsers, context): parser_deploy.add_argument('-n', '--dry-run', help='List files to be deployed only', action='store_true') parser_deploy.add_argument('-p', '--no-preserve', help='Do not preserve existing files', action='store_true') parser_deploy.add_argument('--no-check-space', help='Do not check for available space before deploying', action='store_true') - parser_deploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target') + parser_deploy.add_argument('-P', '--port', help='Specify port to use for connection to the target') strip_opts = parser_deploy.add_mutually_exclusive_group(required=False) strip_opts.add_argument('-S', '--strip', @@ -350,5 +346,5 @@ def register_commands(subparsers, context): parser_undeploy.add_argument('-s', '--show-status', help='Show progress/status output', action='store_true') parser_undeploy.add_argument('-a', '--all', help='Undeploy all recipes deployed on the target', action='store_true') parser_undeploy.add_argument('-n', '--dry-run', help='List files to be undeployed only', action='store_true') - parser_undeploy.add_argument('-P', '--port', default='22', help='Port to use for connection to the target') + parser_undeploy.add_argument('-P', '--port', help='Specify port to use for connection to the target') parser_undeploy.set_defaults(func=undeploy) -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 05/12] devtool: extract: fix usage with kern-tools-native 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton ` (3 preceding siblings ...) 2017-12-05 1:41 ` [PATCH 04/12] devtool: deploy-target: don't specify ssh/scp port unless user does Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 1:41 ` [PATCH 06/12] devtool: modify: pick up commits from previously created source tree Paul Eggleton ` (8 subsequent siblings) 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core The kern-tools-native recipe as it currently stands is unusual in that it fetches source from a repository but sets S = "${WORKDIR}" which causes some problems. First you get a failure because we're calling "git commit" unconditionally even if there are no local files, and there aren't any in this case which means the commit fails. After that's fixed, we hit another problem where "recipe-sysroot-native" subdirectory appears in the extracted source tree. We don't want that so exclude it from copying. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- meta/classes/devtool-source.bbclass | 6 ++++++ scripts/lib/devtool/standard.py | 6 +++--- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/meta/classes/devtool-source.bbclass b/meta/classes/devtool-source.bbclass index 56882a4..540a602 100644 --- a/meta/classes/devtool-source.bbclass +++ b/meta/classes/devtool-source.bbclass @@ -118,7 +118,13 @@ python devtool_post_unpack() { os.path.basename(fname) not in recipe_patches] srcsubdir = d.getVar('DEVTOOL_PATCH_SRCDIR') # Move source files to S + excludevars = ['RECIPE_SYSROOT', 'RECIPE_SYSROOT_NATIVE'] + excludepaths = [] + for excludevar in excludevars: + excludepaths.append(os.path.relpath(d.getVar(excludevar), workdir) + os.sep) for path in src_files: + if path.startswith(tuple(excludepaths)): + continue _move_file(os.path.join(workdir, path), os.path.join(srcsubdir, path)) elif os.path.dirname(srcsubdir) != workdir: diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index cdd2346..a6656e4 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -634,9 +634,9 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works addfiles.append(os.path.join(relpth, fn)) if addfiles: bb.process.run('git add %s' % ' '.join(addfiles), cwd=srctree) - useroptions = [] - oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=d) - bb.process.run('git %s commit -a -m "Committing local file symlinks\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree) + useroptions = [] + oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=d) + bb.process.run('git %s commit -a -m "Committing local file symlinks\n\n%s"' % (' '.join(useroptions), oe.patch.GitApplyTree.ignore_commit_prefix), cwd=srctree) if is_kernel_yocto: logger.info('Copying kernel config to srctree') -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 06/12] devtool: modify: pick up commits from previously created source tree 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton ` (4 preceding siblings ...) 2017-12-05 1:41 ` [PATCH 05/12] devtool: extract: fix usage with kern-tools-native Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 1:41 ` [PATCH 07/12] lib/oe/recipeutils: allow patch_recipe_file() to be re-called Paul Eggleton ` (7 subsequent siblings) 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core If you use devtool modify, then devtool reset, keep the source tree and then devtool modify on the same recipe with the -n option to re-use the existing source tree, we should pick up the commit hashes properly from the source tree so that later on devtool finish has these to compare to the commits in the tree at that time. We also need to be careful the second time around that we only get the original commits rather than the current HEAD which may be the result of user changes (hence using "devtool-patched", the tag that was placed at the original HEAD). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- scripts/lib/devtool/standard.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index a6656e4..ae48406 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -750,13 +750,11 @@ def modify(args, config, basepath, workspace): initial_rev = None commits = [] check_commits = False + torev = 'HEAD' if not args.no_extract: ret = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) initial_rev = ret.initial_rev logger.info('Source tree extracted to %s' % srctree) - # Get list of commits since this revision - (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree) - commits = stdout.split() check_commits = True else: if os.path.exists(os.path.join(srctree, '.git')): @@ -767,6 +765,7 @@ def modify(args, config, basepath, workspace): stdout = '' if stdout: check_commits = True + torev = 'devtool-patched' for line in stdout.splitlines(): if line.startswith('*'): (stdout, _) = bb.process.run('git rev-parse devtool-base', cwd=srctree) @@ -776,6 +775,11 @@ def modify(args, config, basepath, workspace): (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree) initial_rev = stdout.rstrip() + if initial_rev: + # Get list of commits since this revision + (stdout, _) = bb.process.run('git rev-list --reverse %s..%s' % (initial_rev, torev), cwd=srctree) + commits = stdout.split() + branch_patches = {} if check_commits: # Check if there are override branches -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 07/12] lib/oe/recipeutils: allow patch_recipe_file() to be re-called 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton ` (5 preceding siblings ...) 2017-12-05 1:41 ` [PATCH 06/12] devtool: modify: pick up commits from previously created source tree Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 1:41 ` [PATCH 08/12] devtool: support extracting multiple source trees Paul Eggleton ` (6 subsequent siblings) 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core If patch_recipe_file() is called with output redirection on the same file twice in succession, then we don't want to wipe out the changes the first call made so we need to be reading in the redirected file if it exists instead of the original one. This is important to enable devtool finish to work with multiple source trees within the same recipe. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- meta/lib/oe/recipeutils.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py index 4e0859e..4928727 100644 --- a/meta/lib/oe/recipeutils.py +++ b/meta/lib/oe/recipeutils.py @@ -258,13 +258,18 @@ def patch_recipe_file(fn, values, patch=False, relpath='', redirect_output=None) since this cannot handle all situations. """ - with open(fn, 'r') as f: + read_fn = fn + if redirect_output: + redirect_fn = os.path.join(redirect_output, os.path.basename(fn)) + if os.path.exists(redirect_fn): + read_fn = redirect_fn + with open(read_fn, 'r') as f: fromlines = f.readlines() _, tolines = patch_recipe_lines(fromlines, values) if redirect_output: - with open(os.path.join(redirect_output, os.path.basename(fn)), 'w') as f: + with open(redirect_fn, 'w') as f: f.writelines(tolines) return None elif patch: -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 08/12] devtool: support extracting multiple source trees 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton ` (6 preceding siblings ...) 2017-12-05 1:41 ` [PATCH 07/12] lib/oe/recipeutils: allow patch_recipe_file() to be re-called Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 1:41 ` [PATCH 09/12] devtool: fix handling of linux-yocto after multisrc changes Paul Eggleton ` (5 subsequent siblings) 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core If you have multiple source trees being extracted to the work directory within a recipe (e.g. you have two tarballs referred to in SRC_URI) and one isn't being extracted into the other, then devtool failed to extract all the sources because it only took the source tree that S pointed into. To fix this, we need to take a look at the work directory after do_unpack and see if there are any additional subdirectories; if so we need to put the main source tree in a subdirectory and put the additional subdirectories next to it. We also ensure that symlinks from the work directory get created at the end of do_unpack to point to these (so that references in the recipe continue to work). In addition to multiple source trees at the work directory level, this patch also handles multiple nested git trees (where you have multiple git URLs in SRC_URI with one or more inside another). These caused a different problem, where changes in sub-repos are not fully captured at the top level - we need to handle each repo separately. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- meta/classes/devtool-source.bbclass | 57 ++++- meta/lib/oe/recipeutils.py | 4 +- meta/lib/oe/utils.py | 24 +++ meta/lib/oeqa/selftest/cases/devtool.py | 2 +- scripts/devtool | 61 ++++-- scripts/lib/devtool/__init__.py | 14 ++ scripts/lib/devtool/standard.py | 370 +++++++++++++++++++++++--------- 7 files changed, 396 insertions(+), 136 deletions(-) diff --git a/meta/classes/devtool-source.bbclass b/meta/classes/devtool-source.bbclass index 540a602..30c9bdb 100644 --- a/meta/classes/devtool-source.bbclass +++ b/meta/classes/devtool-source.bbclass @@ -69,7 +69,7 @@ python devtool_post_unpack() { import shutil sys.path.insert(0, os.path.join(d.getVar('COREBASE'), 'scripts', 'lib')) import scriptutils - from devtool import setup_git_repo + from devtool import setup_git_repo, find_git_repos tempdir = d.getVar('DEVTOOL_TEMPDIR') workdir = d.getVar('WORKDIR') @@ -95,8 +95,26 @@ python devtool_post_unpack() { oe.recipeutils.get_recipe_patches(d)] local_files = oe.recipeutils.get_recipe_local_files(d) - # Ignore local files with subdir={BP} + excludeitems = recipe_patches + list(local_files.keys()) + pthvars = ['RECIPE_SYSROOT', 'RECIPE_SYSROOT_NATIVE', 'S'] + for pthvar in pthvars: + relp = os.path.relpath(d.getVar(pthvar), d.getVar('WORKDIR')) + if not relp.startswith('..'): + excludeitems.append(relp.split(os.sep)[0]) + extradirs = [] srcabspath = os.path.abspath(srcsubdir) + if srcabspath != os.path.abspath(workdir): + for pth in os.listdir(workdir): + if pth in excludeitems: + continue + wpth = os.path.join(workdir, pth) + if os.path.isdir(wpth) and os.listdir(wpth): + extradirs.append(wpth) + + repos = find_git_repos(srcabspath) + extradirs.extend(repos) + + # Ignore local files with subdir={BP} local_files = [fname for fname in local_files if os.path.exists(os.path.join(workdir, fname)) and (srcabspath == workdir or not @@ -145,11 +163,20 @@ python devtool_post_unpack() { (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srcsubdir) initial_rev = stdout.rstrip() + + initial_revs = {} + for extradir in extradirs: + setup_git_repo(extradir, d.getVar('PV'), devbranch, d=d) + (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=extradir) + initial_revs[extradir] = stdout.rstrip() + with open(os.path.join(tempdir, 'initial_rev'), 'w') as f: f.write(initial_rev) with open(os.path.join(tempdir, 'srcsubdir'), 'w') as f: - f.write(srcsubdir) + f.write('%s\n' % srcsubdir) + for extradir in extradirs: + f.write('%s=%s\n' % (extradir, initial_revs[extradir])) } python devtool_pre_patch() { @@ -160,18 +187,25 @@ python devtool_pre_patch() { python devtool_post_patch() { import shutil tempdir = d.getVar('DEVTOOL_TEMPDIR') + + srcdirs = [] with open(os.path.join(tempdir, 'srcsubdir'), 'r') as f: - srcsubdir = f.read() + for line in f: + line = line.rstrip() + if line: + srcdirs.append(line.split('=')[0]) + srcsubdir = srcdirs[0] + with open(os.path.join(tempdir, 'initial_rev'), 'r') as f: initial_rev = f.read() - def rm_patches(): - patches_dir = os.path.join(srcsubdir, 'patches') + def rm_patches(pth): + patches_dir = os.path.join(pth, 'patches') if os.path.exists(patches_dir): shutil.rmtree(patches_dir) # Restore any "patches" directory that was actually part of the source tree try: - bb.process.run('git checkout -- patches', cwd=srcsubdir) + bb.process.run('git checkout -- patches', cwd=pth) except bb.process.ExecutionError: pass @@ -194,7 +228,7 @@ python devtool_post_patch() { localdata = bb.data.createCopy(d) localdata.setVar('OVERRIDES', ':'.join(no_overrides)) bb.build.exec_func('do_patch', localdata) - rm_patches() + rm_patches(srcsubdir) # Now we need to reconcile the dev branch with the no-overrides one # (otherwise we'd likely be left with identical commits that have different hashes) bb.process.run('git checkout %s' % devbranch, cwd=srcsubdir) @@ -212,12 +246,15 @@ python devtool_post_patch() { # Run do_patch function with the override applied localdata.appendVar('OVERRIDES', ':%s' % override) bb.build.exec_func('do_patch', localdata) - rm_patches() + rm_patches(srcsubdir) # Now we need to reconcile the new branch with the no-overrides one # (otherwise we'd likely be left with identical commits that have different hashes) bb.process.run('git rebase devtool-no-overrides', cwd=srcsubdir) bb.process.run('git checkout %s' % devbranch, cwd=srcsubdir) - bb.process.run('git tag -f devtool-patched', cwd=srcsubdir) + for srcdir in srcdirs: + bb.process.run('git tag -f devtool-patched', cwd=srcdir) + if srcdir != srcsubdir: + rm_patches(srcdir) } python devtool_post_configure() { diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py index 4928727..f4a40c2 100644 --- a/meta/lib/oe/recipeutils.py +++ b/meta/lib/oe/recipeutils.py @@ -445,10 +445,10 @@ def get_recipe_patches(d): """Get a list of the patches included in SRC_URI within a recipe.""" import oe.patch patches = oe.patch.src_patches(d, expand=False) - patchfiles = [] + patchfiles = OrderedDict() for patch in patches: _, _, local, _, _, parm = bb.fetch.decodeurl(patch) - patchfiles.append(local) + patchfiles[local] = parm return patchfiles diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py index 1897c5f..9670dd7 100644 --- a/meta/lib/oe/utils.py +++ b/meta/lib/oe/utils.py @@ -368,3 +368,27 @@ class ImageQAFailed(bb.build.FuncFailed): msg = msg + ' (%s)' % self.description return msg + +def is_path_under(path1, possible_parent): + """ + Return True if a path is underneath another, False otherwise. + Multiple paths to test can be specified (as a list or tuple) in + which case all specified test paths must be under the parent to + return True. + """ + def abs_path_trailing(pth): + pth_abs = os.path.abspath(pth) + if not pth_abs.endswith(os.sep): + pth_abs += os.sep + return pth_abs + + possible_parent_abs = abs_path_trailing(possible_parent) + if isinstance(path1, str): + testpaths = [path1] + else: + testpaths = path1 + for path in testpaths: + path_abs = abs_path_trailing(path) + if not path_abs.startswith(possible_parent_abs): + return False + return True diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py index 43280cd..b3063de 100644 --- a/meta/lib/oeqa/selftest/cases/devtool.py +++ b/meta/lib/oeqa/selftest/cases/devtool.py @@ -7,7 +7,7 @@ import fnmatch import oeqa.utils.ftools as ftools from oeqa.selftest.case import OESelftestTestCase -from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer +from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer, CommandError from oeqa.utils.commands import get_bb_vars, runqemu, get_test_layer from oeqa.core.decorator.oeid import OETestID diff --git a/scripts/devtool b/scripts/devtool index a651d8f..bccbb92 100755 --- a/scripts/devtool +++ b/scripts/devtool @@ -25,6 +25,8 @@ import re import configparser import subprocess import logging +import json +from collections import OrderedDict basepath = '' workspace = {} @@ -109,31 +111,58 @@ def read_workspace(): if not context.fixed_setup: _enable_workspace_layer(config.workspace_path, config, basepath) + def process_inline_json(strvalue, in_value): + if strvalue.count('{') == strvalue.count('}') and strvalue.count('[') == strvalue.count(']'): + pnvalues[in_value] = json.loads(strvalue, object_pairs_hook=OrderedDict) + return True + return False + logger.debug('Reading workspace in %s' % config.workspace_path) externalsrc_re = re.compile(r'^EXTERNALSRC(_pn-([^ =]+))? *= *"([^"]*)"$') for fn in glob.glob(os.path.join(config.workspace_path, 'appends', '*.bbappend')): with open(fn, 'r') as f: pnvalues = {} + in_value = None + strvalue = '' for line in f: - res = externalsrc_re.match(line.rstrip()) - if res: - pn = res.group(2) or os.path.splitext(os.path.basename(fn))[0].split('_')[0] - # Find the recipe file within the workspace, if any - bbfile = os.path.basename(fn).replace('.bbappend', '.bb').replace('%', '*') - recipefile = glob.glob(os.path.join(config.workspace_path, - 'recipes', - pn, - bbfile)) - if recipefile: - recipefile = recipefile[0] - pnvalues['srctree'] = res.group(3) - pnvalues['bbappend'] = fn - pnvalues['recipefile'] = recipefile - elif line.startswith('# srctreebase: '): - pnvalues['srctreebase'] = line.split(':', 1)[1].strip() + if in_value: + if not line.startswith('#'): + logger.error('Syntax error in %s - invalid in-line json' % fn) + sys.exit(1) + strvalue += line[1:] + if process_inline_json(strvalue, in_value): + in_value = None + else: + res = externalsrc_re.match(line.rstrip()) + if res: + pn = res.group(2) or os.path.splitext(os.path.basename(fn))[0].split('_')[0] + # Find the recipe file within the workspace, if any + bbfile = os.path.basename(fn).replace('.bbappend', '.bb').replace('%', '*') + recipefile = glob.glob(os.path.join(config.workspace_path, + 'recipes', + pn, + bbfile)) + if recipefile: + recipefile = recipefile[0] + pnvalues['srctree'] = res.group(3) + pnvalues['bbappend'] = fn + pnvalues['recipefile'] = recipefile + elif line.startswith('# srctreebase: '): + pnvalues['srctreebase'] = line.split(':', 1)[1].strip() + elif line.startswith('# srctreetop: '): + pnvalues['srctreetop'] = line.split(':', 1)[1].strip() + elif line.startswith('# extradirs: '): + strvalue = line.split(':', 1)[1].strip() + in_value = 'extradirs' + if process_inline_json(strvalue, in_value): + in_value = None if pnvalues: if not pnvalues.get('srctreebase', None): pnvalues['srctreebase'] = pnvalues['srctree'] + if not 'srctreetop' in pnvalues: + pnvalues['srctreetop'] = pnvalues['srctreebase'] + if not 'extradirs' in pnvalues: + pnvalues['extradirs'] = [] logger.debug('Found recipe %s' % pnvalues) workspace[pn] = pnvalues diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py index 07d774d..5cdf07a 100644 --- a/scripts/lib/devtool/__init__.py +++ b/scripts/lib/devtool/__init__.py @@ -223,6 +223,20 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None): bb.process.run('git checkout -b %s' % devbranch, cwd=repodir) bb.process.run('git tag -f %s' % basetag, cwd=repodir) +def find_git_repos(pth, toplevel=False): + """ + Find git repositories under a path + """ + repos = [] + if toplevel and os.path.isdir(os.path.join(pth, '.git')): + repos.append(pth) + for root, dirs, _ in os.walk(pth): + for dfn in dirs: + dfp = os.path.join(root, dfn) + if os.path.isdir(os.path.join(dfp, '.git')) and dfp not in repos: + repos.append(dfp) + return repos + def recipe_to_append(recipefile, config, wildcard=False): """ Convert a recipe file to a bbappend file path within the workspace. diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index ae48406..cadd038 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -29,8 +29,9 @@ import scriptutils import errno import glob import filecmp +import json from collections import OrderedDict, namedtuple -from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, update_unlockedsigs, check_prerelease_version, check_git_repo_dirty, check_git_repo_op, DevtoolError +from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, update_unlockedsigs, check_prerelease_version, check_git_repo_dirty, check_git_repo_op, find_git_repos, DevtoolError from devtool import parse_recipe logger = logging.getLogger('devtool') @@ -464,7 +465,7 @@ def sync(args, config, basepath, workspace): return 0 -ExtractSourceResult = namedtuple('ExtractSourceResult', ['initial_rev', 'srcsubdir']) +ExtractSourceResult = namedtuple('ExtractSourceResult', ['initial_rev', 'srcsubdir', 'extradirs']) def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, workspace, fixed_setup, d, tinfoil, no_overrides=False): """Extract sources of a recipe""" @@ -576,15 +577,27 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works if not res: raise DevtoolError('Extracting source for %s failed' % pn) + tempworkdir = os.path.join(tempdir, 'workdir') + extra_revs = OrderedDict() + extradirs = [] try: with open(os.path.join(tempdir, 'initial_rev'), 'r') as f: initial_rev = f.read() with open(os.path.join(tempdir, 'srcsubdir'), 'r') as f: - srcsubdir = f.read() + srcsubdir = f.readline().rstrip() + for line in f: + line = line.rstrip() + if '=' in line: + linesplit = line.rstrip().split('=', 1) + extradir = linesplit[0] + extradirs.append(extradir) + extra_revs[os.path.relpath(extradir, tempworkdir)] = linesplit[1] + except FileNotFoundError as e: raise DevtoolError('Something went wrong with source extraction - the devtool-source class was not active or did not function correctly:\n%s' % str(e)) - srcsubdir_rel = os.path.relpath(srcsubdir, os.path.join(tempdir, 'workdir')) + + srcsubdir_rel = os.path.relpath(srcsubdir, tempworkdir) tempdir_localdir = os.path.join(tempdir, 'oe-local-files') srctree_localdir = os.path.join(srctree, 'oe-local-files') @@ -613,6 +626,19 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works logger.info('Adding local source files to srctree...') shutil.move(tempdir_localdir, srcsubdir) + if extra_revs: + logger.warn('This recipe fetches more than one source tree. Each source tree will be tracked in a separate git repository.') + if not oe.utils.is_path_under(extradirs, srcsubdir): + # There's more than one source directory at the top level, we're going to need to create the parent + os.mkdir(srctree) + for extradir in extradirs: + shutil.move(extradir, srctree) + # Write a file to record which one is the main src in case we need to re-modify the same tree later + with open(os.path.join(srctree, '.devtool'), 'w') as f: + data = OrderedDict() + data['srcsubdir'] = os.path.basename(srcsubdir) + data['extradirs'] = extra_revs + json.dump(data, f, indent=' ') shutil.move(srcsubdir, srctree) if os.path.abspath(d.getVar('S')) == os.path.abspath(d.getVar('WORKDIR')): @@ -640,7 +666,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works if is_kernel_yocto: logger.info('Copying kernel config to srctree') - shutil.copy2(os.path.join(tempdir, '.config'), srctree) + shutil.copy2(os.path.join(tempdir, '.config'), os.path.join(srctree, srcsubdir_rel)) finally: if appendbackup: @@ -651,7 +677,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works logger.info('Preserving temporary directory %s' % tempdir) else: shutil.rmtree(tempdir) - return ExtractSourceResult(initial_rev, srcsubdir_rel) + return ExtractSourceResult(initial_rev, srcsubdir_rel, extra_revs) def _add_md5(config, recipename, filename): """Record checksum of a file (or recursively for a directory) to the md5-file of the workspace""" @@ -704,6 +730,27 @@ def _check_preserve(config, recipename): tf.write(line) os.rename(newfile, origfile) +def _get_initial_rev(srctree): + """ + Given a source tree, get the initial revision and determine if it was prepared by devtool + """ + initial_rev = None + try: + (stdout, _) = bb.process.run('git branch --contains devtool-base', cwd=srctree) + except bb.process.ExecutionError: + stdout = '' + if stdout: + was_devtool = True + for line in stdout.splitlines(): + if line.startswith('*'): + (stdout, _) = bb.process.run('git rev-parse devtool-base', cwd=srctree) + initial_rev = stdout.rstrip() + if not initial_rev: + # Otherwise, just grab the head revision + (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree) + initial_rev = stdout.rstrip() + return initial_rev, was_devtool + def modify(args, config, basepath, workspace): """Entry point for the devtool 'modify' subcommand""" import bb @@ -751,29 +798,40 @@ def modify(args, config, basepath, workspace): commits = [] check_commits = False torev = 'HEAD' + extradirs = OrderedDict() + srctreetop = None + extra_revs = {} if not args.no_extract: ret = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides) initial_rev = ret.initial_rev + extradirs = ret.extradirs + if extradirs and not oe.utils.is_path_under(extradirs.keys(), ret.srcsubdir): + # Multiple source trees, we need to be inside the main one + srctreetop = srctree + srctree = os.path.join(srctree, ret.srcsubdir) logger.info('Source tree extracted to %s' % srctree) check_commits = True else: + devtoolfile = os.path.join(srctree, '.devtool') + if os.path.exists(devtoolfile): + with open(devtoolfile, 'r') as f: + cfg = json.load(f, object_pairs_hook=OrderedDict) + srcsubdir = cfg.get('srcsubdir', '') + if srcsubdir: + srctreetop = srctree + srctree = os.path.abspath(os.path.join(srctree, srcsubdir)) + extradirs.update(cfg.get('extradirs', {})) + else: + # Find any extra source trees + extradirpaths = find_git_repos(srctree) + for extradirpath in extradirpaths: + extradirs[os.path.relpath(extradirpath, srctree)], _ = _get_initial_rev(extradirpath) + if os.path.exists(os.path.join(srctree, '.git')): # Check if it's a tree previously extracted by us - try: - (stdout, _) = bb.process.run('git branch --contains devtool-base', cwd=srctree) - except bb.process.ExecutionError: - stdout = '' - if stdout: - check_commits = True + initial_rev, check_commits = _get_initial_rev(srctree) + if check_commits: torev = 'devtool-patched' - for line in stdout.splitlines(): - if line.startswith('*'): - (stdout, _) = bb.process.run('git rev-parse devtool-base', cwd=srctree) - initial_rev = stdout.rstrip() - if not initial_rev: - # Otherwise, just grab the head revision - (stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree) - initial_rev = stdout.rstrip() if initial_rev: # Get list of commits since this revision @@ -807,6 +865,23 @@ def modify(args, config, basepath, workspace): # Need to grab this here in case the source is within a subdirectory srctreebase = srctree + if extradirs: + extradirentries = [] + for extradir, extrarev in extradirs.items(): + extradirentry = OrderedDict() + if srctreetop: + # Several source trees at the top level + extradirentry['path'] = os.path.join(srctreetop, extradir) + else: + # All of the extra source trees are under the main one, so chop off + # the top level subdirectory + extradirentry['path'] = os.path.join(srctree, os.sep.join(extradir.split(os.sep)[1:])) + extradirentry['initial_rev'] = extrarev + # Get list of commits since this revision + (stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % extrarev, cwd=extradirentry['path']) + extradirentry['commits'] = stdout.split() + extradirentries.append(extradirentry) + # Check that recipe isn't using a shared workdir s = os.path.abspath(rd.getVar('S')) workdir = os.path.abspath(rd.getVar('WORKDIR')) @@ -842,6 +917,23 @@ def modify(args, config, basepath, workspace): ' cp ${B}/.config ${S}/.config.baseline\n' ' ln -sfT ${B}/.config ${S}/.config.new\n' '}\n') + + if extradirs: + f.write('\n# srctreetop: %s\n' % srctreetop) + extradir_json = json.dumps(extradirentries, indent=' ') + prefix = '# extradirs:' + for line in extradir_json.splitlines(): + f.write('%s %s\n' % (prefix, line)) + prefix = '#' + f.write('\ndo_unpack[postfuncs] += "devtool_symlink_srctrees"\n') + f.write('devtool_symlink_srctrees () {\n') + # We could have done a loop in the function itself, but given there's + # usually only going to be a small number, there's not much point + for extradiritem in extradirentries: + if not oe.utils.is_path_under(extradiritem['path'], srctree): + f.write(' ln -sf %s ${WORKDIR}/\n' % extradiritem['path']) + f.write('}\n') + if initial_rev: f.write('\n# initial_rev: %s\n' % initial_rev) for commit in commits: @@ -1056,7 +1148,7 @@ def rename(args, config, basepath, workspace): return 0 -def _get_patchset_revs(srctree, recipe_path, initial_rev=None): +def _get_patchset_revs(recipename, workspace, srctree, initial_rev=None): """Get initial and update rev of a recipe. These are the start point of the whole patchset and start point for the patches to be re-generated/updated. """ @@ -1067,10 +1159,15 @@ def _get_patchset_revs(srctree, recipe_path, initial_rev=None): cwd=srctree) branchname = stdout.rstrip() + append = workspace[recipename]['bbappend'] + if not os.path.exists(append): + raise DevtoolError('unable to find workspace bbappend for recipe %s' % + recipename) + # Parse initial rev from recipe if not specified commits = [] patches = [] - with open(recipe_path, 'r') as f: + with open(append, 'r') as f: for line in f: if line.startswith('# initial_rev:'): if not initial_rev: @@ -1080,6 +1177,15 @@ def _get_patchset_revs(srctree, recipe_path, initial_rev=None): elif line.startswith('# patches_%s:' % branchname): patches = line.split(':')[-1].strip().split(',') + if srctree != workspace[recipename]['srctree']: + # Extra dir + for extradirentry in workspace[recipename]['extradirs']: + if srctree == extradirentry['path']: + commits = extradirentry['commits'] + break + else: + raise Exception('Failed to find extradir entry for %s' % srctree) + update_rev = initial_rev changed_revs = None if initial_rev: @@ -1160,7 +1266,7 @@ def _remove_source_files(append, files, destpath, no_report_remove=False, dry_ru raise -def _export_patches(srctree, rd, start_rev, destdir, changed_revs=None): +def _export_patches(srctree, rd, start_rev, destdir, changed_revs=None, origsrcdir=None): """Export patches from srctree to given location. Returns three-tuple of dicts: 1. updated - patches that already exist in SRCURI @@ -1175,9 +1281,21 @@ def _export_patches(srctree, rd, start_rev, destdir, changed_revs=None): added = OrderedDict() seqpatch_re = re.compile('^([0-9]{4}-)?(.+)') + patches = oe.recipeutils.get_recipe_patches(rd) + if origsrcdir: + if not origsrcdir.endswith(os.sep): + origsrcdir = origsrcdir + os.sep + s = rd.getVar('S') + for patch, parms in list(patches.items()): + patchdir = os.path.abspath(os.path.join(s, parms.get('patchdir', '.'))) + if not patchdir.endswith(os.sep): + patchdir = patchdir + os.sep + if not patchdir.startswith(origsrcdir): + del patches[patch] + existing_patches = dict((os.path.basename(path), path) for path in - oe.recipeutils.get_recipe_patches(rd)) - logger.debug('Existing patches: %s' % existing_patches) + patches) + logger.debug('Existing patches (for %s): %s' % (srctree, existing_patches)) # Generate patches from Git, exclude local files directory patch_pathspec = _git_exclude_path(srctree, 'oe-local-files') @@ -1370,6 +1488,11 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi recipedir = os.path.basename(recipefile) logger.info('Updating SRCREV in recipe %s%s' % (recipedir, dry_run_suffix)) + # Work out what the extracted path would have been (so we can use it + # to filter patches when there are extra source directories) + s = rd.getVar('S') + origsrcdir = os.path.abspath(os.path.join(s, os.path.relpath(srctree, workspace[recipename]['srctree']))) + # Get HEAD revision try: stdout, _ = bb.process.run('git rev-parse HEAD', cwd=srctree) @@ -1398,7 +1521,8 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi patches_dir = tempfile.mkdtemp(dir=tempdir) old_srcrev = rd.getVar('SRCREV') or '' upd_p, new_p, del_p = _export_patches(srctree, rd, old_srcrev, - patches_dir) + patches_dir, + origsrcdir=origsrcdir) logger.debug('Patches: update %s, new %s, delete %s' % (dict(upd_p), dict(new_p), dict(del_p))) # Remove deleted local files and "overlapping" patches @@ -1444,6 +1568,10 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi if update_srcuri: patchfields['SRC_URI'] = ' '.join(srcuri) ret = oe.recipeutils.patch_recipe(rd, recipefile, patchfields, redirect_output=dry_run_outdir) + # Set the values into the datastore for the benefit of the next + # call (if extradirs are set) + for var, value in patchfields.items(): + rd.setVar(var, value) finally: shutil.rmtree(tempdir) if not 'git://' in orig_src_uri: @@ -1461,12 +1589,8 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil recipefile = rd.getVar('FILE') recipedir = os.path.dirname(recipefile) - append = workspace[recipename]['bbappend'] - if not os.path.exists(append): - raise DevtoolError('unable to find workspace bbappend for recipe %s' % - recipename) - initial_rev, update_rev, changed_revs, filter_patches = _get_patchset_revs(srctree, append, initial_rev) + initial_rev, update_rev, changed_revs, filter_patches = _get_patchset_revs(recipename, workspace, srctree, initial_rev) if not initial_rev: raise DevtoolError('Unable to find initial revision - please specify ' 'it with --initial-rev') @@ -1476,6 +1600,17 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil if not dl_dir.endswith('/'): dl_dir += '/' + # Work out what the extracted path would have been (so we can use it + # to filter patches when there are extra source directories) + s = rd.getVar('S') + origsrcdir = os.path.abspath(os.path.join(s, os.path.relpath(srctree, workspace[recipename]['srctree']))) + # Determine patchdir for any new patches + patchdir = os.path.relpath(origsrcdir, s) + if patchdir != '.': + patchsuffix = ';patchdir=%s' % patchdir + else: + patchsuffix = '' + dry_run_suffix = ' (dry-run)' if dry_run_outdir else '' tempdir = tempfile.mkdtemp(prefix='devtool') @@ -1494,14 +1629,16 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil # Get all patches from source tree and check if any should be removed all_patches_dir = tempfile.mkdtemp(dir=tempdir) _, _, del_p = _export_patches(srctree, rd, initial_rev, - all_patches_dir) + all_patches_dir, + origsrcdir=origsrcdir) # Remove deleted local files and patches remove_files = list(del_f.values()) + list(del_p.values()) # Get updated patches from source tree patches_dir = tempfile.mkdtemp(dir=tempdir) upd_p, new_p, _ = _export_patches(srctree, rd, update_rev, - patches_dir, changed_revs) + patches_dir, changed_revs, + origsrcdir=origsrcdir) logger.debug('Pre-filtering: update: %s, new: %s' % (dict(upd_p), dict(new_p))) if filter_patches: new_p = {} @@ -1531,7 +1668,7 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil removevalues=removevalues, redirect_output=dry_run_outdir) else: - logger.info('No patches or local source files needed updating') + return False, None, [] else: # Update existing files files_dir = _determine_files_dir(rd) @@ -1553,7 +1690,7 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil # replace the entry in SRC_URI with our local version logger.info('Replacing remote patch %s with updated local version' % basepath) path = os.path.join(files_dir, basepath) - _replace_srcuri_entry(srcuri, basepath, 'file://%s' % basepath) + _replace_srcuri_entry(srcuri, basepath, 'file://%s%s' % (basepath, patchsuffix)) updaterecipe = True else: logger.info('Updating patch %s%s' % (basepath, dry_run_suffix)) @@ -1575,7 +1712,7 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil os.path.join(files_dir, basepath), dry_run_outdir=dry_run_outdir, base_outdir=recipedir) - srcuri.append('file://%s' % basepath) + srcuri.append('file://%s%s' % (basepath, patchsuffix)) updaterecipe = True # Update recipe, if needed if _remove_file_entries(srcuri, remove_files)[0]: @@ -1586,9 +1723,11 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil ret = oe.recipeutils.patch_recipe(rd, recipefile, {'SRC_URI': ' '.join(srcuri)}, redirect_output=dry_run_outdir) + # Set the value into the datastore for the benefit of the next + # call (if extradirs are set) + rd.setVar('SRC_URI', ' '.join(srcuri)) elif not updatefiles: # Neither patches nor recipe were updated - logger.info('No patches or files need updating') return False, None, [] finally: shutil.rmtree(tempdir) @@ -1619,70 +1758,82 @@ def _guess_recipe_update_mode(srctree, rdata): return 'patch' def _update_recipe(recipename, workspace, rd, mode, appendlayerdir, wildcard_version, no_remove, initial_rev, no_report_remove=False, dry_run_outdir=None, no_overrides=False): - srctree = workspace[recipename]['srctree'] - if mode == 'auto': - mode = _guess_recipe_update_mode(srctree, rd) + main_srctree = workspace[recipename]['srctree'] - override_branches = [] - mainbranch = None - startbranch = None - if not no_overrides: - stdout, _ = bb.process.run('git branch', cwd=srctree) - other_branches = [] - for line in stdout.splitlines(): - branchname = line[2:] - if line.startswith('* '): - startbranch = branchname - if branchname.startswith(override_branch_prefix): - override_branches.append(branchname) - else: - other_branches.append(branchname) + appendfile = workspace[recipename]['bbappend'] + srctrees = OrderedDict([(main_srctree, initial_rev)]) + for extradirentry in workspace[recipename]['extradirs']: + srctrees[extradirentry['path']] = extradirentry['initial_rev'] + + logger.debug('Considering source trees: %s' % srctrees) + if mode == 'auto': + mode = _guess_recipe_update_mode(main_srctree, rd) + + crd = bb.data.createCopy(rd) + for srctree, src_initial_rev in srctrees.items(): + override_branches = [] + mainbranch = None + startbranch = None + if not no_overrides: + stdout, _ = bb.process.run('git branch', cwd=srctree) + other_branches = [] + for line in stdout.splitlines(): + branchname = line[2:] + if line.startswith('* '): + startbranch = branchname + if branchname.startswith(override_branch_prefix): + override_branches.append(branchname) + else: + other_branches.append(branchname) + + if override_branches: + logger.debug('_update_recipe: override branches: %s' % override_branches) + logger.debug('_update_recipe: other branches: %s' % other_branches) + if startbranch.startswith(override_branch_prefix): + if len(other_branches) == 1: + mainbranch = other_branches[1] + else: + raise DevtoolError('Unable to determine main branch - please check out the main branch in source tree first') + else: + mainbranch = startbranch + + checkedout = None + anyupdated = False + appendfile = None + allremoved = [] if override_branches: - logger.debug('_update_recipe: override branches: %s' % override_branches) - logger.debug('_update_recipe: other branches: %s' % other_branches) - if startbranch.startswith(override_branch_prefix): - if len(other_branches) == 1: - mainbranch = other_branches[1] + logger.info('Handling main branch (%s)...' % mainbranch) + if startbranch != mainbranch: + bb.process.run('git checkout %s' % mainbranch, cwd=srctree) + checkedout = mainbranch + try: + branchlist = [mainbranch] + override_branches + for branch in branchlist: + if branch != mainbranch: + logger.info('Handling branch %s...' % branch) + override = branch[len(override_branch_prefix):] + crd.setVar('OVERRIDES', '%s:%s' % rd.getVar('OVERRIDES') + override) + bb.process.run('git checkout %s' % branch, cwd=srctree) + checkedout = branch + + if mode == 'srcrev': + updated, appendf, removed = _update_recipe_srcrev(recipename, workspace, srctree, crd, appendlayerdir, wildcard_version, no_remove, no_report_remove, dry_run_outdir) + elif mode == 'patch': + updated, appendf, removed = _update_recipe_patch(recipename, workspace, srctree, crd, appendlayerdir, wildcard_version, no_remove, no_report_remove, src_initial_rev, dry_run_outdir) else: - raise DevtoolError('Unable to determine main branch - please check out the main branch in source tree first') - else: - mainbranch = startbranch + raise DevtoolError('update_recipe: invalid mode %s' % mode) + if updated: + anyupdated = True + if appendf: + appendfile = appendf + allremoved.extend(removed) + finally: + if startbranch and checkedout != startbranch: + bb.process.run('git checkout %s' % startbranch, cwd=srctree) - checkedout = None - anyupdated = False - appendfile = None - allremoved = [] - if override_branches: - logger.info('Handling main branch (%s)...' % mainbranch) - if startbranch != mainbranch: - bb.process.run('git checkout %s' % mainbranch, cwd=srctree) - checkedout = mainbranch - try: - branchlist = [mainbranch] + override_branches - for branch in branchlist: - crd = bb.data.createCopy(rd) - if branch != mainbranch: - logger.info('Handling branch %s...' % branch) - override = branch[len(override_branch_prefix):] - crd.appendVar('OVERRIDES', ':%s' % override) - bb.process.run('git checkout %s' % branch, cwd=srctree) - checkedout = branch - - if mode == 'srcrev': - updated, appendf, removed = _update_recipe_srcrev(recipename, workspace, srctree, crd, appendlayerdir, wildcard_version, no_remove, no_report_remove, dry_run_outdir) - elif mode == 'patch': - updated, appendf, removed = _update_recipe_patch(recipename, workspace, srctree, crd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev, dry_run_outdir) - else: - raise DevtoolError('update_recipe: invalid mode %s' % mode) - if updated: - anyupdated = True - if appendf: - appendfile = appendf - allremoved.extend(removed) - finally: - if startbranch and checkedout != startbranch: - bb.process.run('git checkout %s' % startbranch, cwd=srctree) + if not anyupdated: + logger.info('No patches or files need updating') return anyupdated, appendfile, allremoved @@ -1808,16 +1959,16 @@ def _reset(recipes, no_clean, config, basepath, workspace): # We don't automatically create this dir next to appends, but the user can preservedir(os.path.join(config.workspace_path, 'appends', pn)) - srctreebase = workspace[pn]['srctreebase'] - if os.path.isdir(srctreebase): - if os.listdir(srctreebase): + srctreetop = workspace[pn]['srctreetop'] + if os.path.isdir(srctreetop): + if os.listdir(srctreetop): # We don't want to risk wiping out any work in progress logger.info('Leaving source tree %s as-is; if you no ' 'longer need it then please delete it manually' - % srctreebase) + % srctreetop) else: # This is unlikely, but if it's empty we can just remove it - os.rmdir(srctreebase) + os.rmdir(srctreetop) clean_preferred_provider(pn, config.workspace_path) @@ -1868,12 +2019,17 @@ def finish(args, config, basepath, workspace): srctree = workspace[args.recipename]['srctree'] check_git_repo_op(srctree, [corebasedir]) - dirty = check_git_repo_dirty(srctree) - if dirty: - if args.force: - logger.warning('Source tree is not clean, continuing as requested by -f/--force') - else: - raise DevtoolError('Source tree is not clean:\n\n%s\nEnsure you have committed your changes or use -f/--force if you are sure there\'s nothing that needs to be committed' % dirty) + srctrees = [srctree] + for extradirentry in workspace[args.recipename]['extradirs']: + srctrees.append(extradirentry['path']) + for pth in srctrees: + dirty = check_git_repo_dirty(pth) + if dirty: + if args.force: + logger.warning('Source tree %s is not clean, continuing as requested by -f/--force' % pth) + else: + raise DevtoolError('Source tree %s is not clean:\n\n%s\nEnsure you have committed your changes or use -f/--force if you are sure there\'s nothing that needs to be committed' % (pth, dirty)) + no_clean = False tinfoil = setup_tinfoil(basepath=basepath, tracking=True) -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 09/12] devtool: fix handling of linux-yocto after multisrc changes 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton ` (7 preceding siblings ...) 2017-12-05 1:41 ` [PATCH 08/12] devtool: support extracting multiple source trees Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 1:41 ` [PATCH 10/12] oe-selftest: devtool: fix kernel test for " Paul Eggleton ` (4 subsequent siblings) 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core devtool now handles multiple source trees for any recipe that includes them in SRC_URI, extracting them all side-by-side so that you can make changes in any of them. As a result, when running devtool modify on a linux-yocto kernel recipe under the source path you will get a "source" subdirectory containing the kernel source and a "kernel-meta" subdirectory next to it containing the kernel metadata. (Previously you just got the source tree and the kernel metadata remained in the work directory). We create a symlink automatically at do_unpack from the work directory so that it can still be found there, however kernel_feature_dirs() expects to find the kernel-meta repository and we also now need to make externalsrc remove that so that it doesn't unpack and overwrite the one we've already extracted. Change kernel_feature_dirs() so that if there are no kmeta entries in SRC_URI, it will fall back to a directory named ${KMETA} if it happens to be present in the work directory, ignoring how it got there. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- meta/classes/externalsrc.bbclass | 2 +- meta/classes/kernel-yocto.bbclass | 7 ++++++- scripts/lib/devtool/standard.py | 4 +++- 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass index 65dd13d..1ce53ee 100644 --- a/meta/classes/externalsrc.bbclass +++ b/meta/classes/externalsrc.bbclass @@ -65,7 +65,7 @@ python () { url_data = fetch.ud[url] parm = url_data.parm if (url_data.type == 'file' or - 'type' in parm and parm['type'] == 'kmeta'): + (parm.get('type', None) == 'kmeta' and d.getVar('EXTERNALSRC_KMETA') != "1")): local_srcuri.append(url) d.setVar('SRC_URI', ' '.join(local_srcuri)) diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass index 1d44795..18a7113 100644 --- a/meta/classes/kernel-yocto.bbclass +++ b/meta/classes/kernel-yocto.bbclass @@ -42,7 +42,12 @@ def find_kernel_feature_dirs(d): destdir = parm["destsuffix"] if type == "kmeta": feature_dirs.append(destdir) - + if not feature_dirs: + # If the kernel-meta directory already exists (e.g from externalsrc) + # with EXTERNALSRC_KMETA = "1" then use it + kmetadir = d.getVar('KMETA') + if os.path.isdir(os.path.join(d.getVar('WORKDIR'), kmetadir)): + feature_dirs.append(kmetadir) return feature_dirs # find the master/machine source branch. In the same way that the fetcher proceses diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py index cadd038..fe33c7e 100644 --- a/scripts/lib/devtool/standard.py +++ b/scripts/lib/devtool/standard.py @@ -909,7 +909,9 @@ def modify(args, config, basepath, workspace): if bb.data.inherits_class('kernel', rd): f.write('SRCTREECOVEREDTASKS = "do_validate_branches do_kernel_checkout ' - 'do_fetch do_unpack do_kernel_configme do_kernel_configcheck"\n') + 'do_fetch do_kernel_configme do_kernel_configcheck"\n') + if bb.data.inherits_class('kernel-yocto', rd): + f.write('EXTERNALSRC_KMETA = "1"\n') f.write('\ndo_patch() {\n' ' :\n' '}\n') -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 10/12] oe-selftest: devtool: fix kernel test for multisrc changes 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton ` (8 preceding siblings ...) 2017-12-05 1:41 ` [PATCH 09/12] devtool: fix handling of linux-yocto after multisrc changes Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 1:41 ` [PATCH 11/12] oe-selftest: devtool: add test for multiple source trees Paul Eggleton ` (3 subsequent siblings) 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core Fix the oe-selftest test to understand that the source will be found in a "source" subdirectory if the kernel inherits linux-yocto since there are multiple trees extracted. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- meta/lib/oeqa/selftest/cases/devtool.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py index b3063de..96072be 100644 --- a/meta/lib/oeqa/selftest/cases/devtool.py +++ b/meta/lib/oeqa/selftest/cases/devtool.py @@ -1676,9 +1676,14 @@ class DevtoolTests(DevtoolBase): bitbake('%s -c clean' % kernel_provider) #Step 4.1 runCmd('devtool modify virtual/kernel -x %s' % tempdir) - self.assertExists(os.path.join(tempdir, 'Makefile'), 'Extracted source could not be found') + # If it's linux-yocto, we'll have two "source" trees (one the actual source, + # the other the kernel metadata) + tempsrc = os.path.join(tempdir, 'source') + if not os.path.isdir(tempsrc): + tempsrc = tempdir + self.assertExists(os.path.join(tempsrc, 'Makefile'), 'Extracted source could not be found') #Step 4.2 - configfile = os.path.join(tempdir,'.config') + configfile = os.path.join(tempsrc, '.config') diff = runCmd('diff %s %s' % (tmpconfig, configfile)) self.assertEqual(0,diff.status,'Kernel .config file is not the same using bitbake and devtool') #Step 4.3 @@ -1689,12 +1694,12 @@ class DevtoolTests(DevtoolBase): self.assertExists(kernelfile, 'Kernel was not build correctly') #Modify the kernel source - modfile = os.path.join(tempdir,'arch/x86/boot/header.S') + modfile = os.path.join(tempsrc, 'arch/x86/boot/header.S') modstring = "Use a boot loader. Devtool testing." modapplied = runCmd("sed -i 's/Use a boot loader./%s/' %s" % (modstring, modfile)) self.assertEqual(0,modapplied.status,'Modification to %s on kernel source failed' % modfile) #Modify the configuration - codeconfigfile = os.path.join(tempdir,'.config.new') + codeconfigfile = os.path.join(tempsrc, '.config.new') modconfopt = "CONFIG_SG_POOL=n" modconf = runCmd("sed -i 's/CONFIG_SG_POOL=y/%s/' %s" % (modconfopt, codeconfigfile)) self.assertEqual(0,modconf.status,'Modification to %s failed' % codeconfigfile) -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 11/12] oe-selftest: devtool: add test for multiple source trees 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton ` (9 preceding siblings ...) 2017-12-05 1:41 ` [PATCH 10/12] oe-selftest: devtool: fix kernel test for " Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 1:41 ` [PATCH 12/12] lib/oe/recipeutils: add .txz extension to archive list Paul Eggleton ` (2 subsequent siblings) 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core Add two synthetic tests for devtool modify + devtool finish: first with multiple source trees side-by-side, and second to test with one as a subdirectory of the main source tree. These also test devtool finish's recently added dry-run option and that detects and errors on uncommitted changes without being forced. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- .../devtool/devtool-test-multisrc-subdir_1.0.bb | 13 +++ .../devtool-test-multisrc/example-files.tar.gz | Bin 0 -> 236 bytes .../devtool/devtool-test-multisrc/example.patch | 7 ++ .../devtool-test-multisrc/mypackage-1.0.tar.gz | Bin 0 -> 903 bytes .../devtool/devtool-test-multisrc_1.0.bb | 11 +++ meta/lib/oeqa/selftest/cases/devtool.py | 109 +++++++++++++++++++++ 6 files changed, 140 insertions(+) create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc-subdir_1.0.bb create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example-files.tar.gz create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc/mypackage-1.0.tar.gz create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-multisrc_1.0.bb diff --git a/meta-selftest/recipes-test/devtool/devtool-test-multisrc-subdir_1.0.bb b/meta-selftest/recipes-test/devtool/devtool-test-multisrc-subdir_1.0.bb new file mode 100644 index 0000000..0430382 --- /dev/null +++ b/meta-selftest/recipes-test/devtool/devtool-test-multisrc-subdir_1.0.bb @@ -0,0 +1,13 @@ +SUMMARY = "Recipe to test multiple source trees" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://settings-daemon.c;beginline=1;endline=26;md5=8d77ba1c7a55df48d8d06c5f3d86b49d" + +SRC_URI = "git://git.yoctoproject.org/xsettings-daemon;name=xsettings-daemon \ + git://git.yoctoproject.org/libfakekey;name=libfakekey;destsuffix=git/libfakekey \ +" + +SRCREV_xsettings-daemon = "b2e5da502f8c5ff75e9e6da771372ef8e40fd9a2" +SRCREV_libfakekey = "7ad885912efb2131e80914e964d5e635b0d07b40" + +S = "${WORKDIR}/git" + diff --git a/meta-selftest/recipes-test/devtool/devtool-test-multisrc/example-files.tar.gz b/meta-selftest/recipes-test/devtool/devtool-test-multisrc/example-files.tar.gz new file mode 100644 index 0000000000000000000000000000000000000000..fdab4b7367cda3613468c7813f47acaaac74368c GIT binary patch literal 236 zcmV<I02BWoiwFR^niN_91MSt{3W6{c25_%^ioHOzKOT4OV|2NTMS+><gdV>c2AY-- zQ;ELc&A<)LfwRxqo4J&^-5t-lt@qI)xTiI>tI)*z>r*6@<Rk=1(h(O@`A>|NX=kXZ zov9d$O4H<@&du%rC_9+Hv$a#fYqSGS)8zg9Rr)f2CH-;6gLg9F&*y)zj+x@>PPSs* zU1059h+`kZc<b(9Y2*Bh;5BCRFMI$0L_YGL5c0o<tQf|A>waKe<NV9uHRkhAz5I9Y mKS%!8kWIb)TQ>y&000000000000000U%3O3K98mVC;$MDWO;W0 literal 0 HcmV?d00001 diff --git a/meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch b/meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch new file mode 100644 index 0000000..2e48bf1 --- /dev/null +++ b/meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch @@ -0,0 +1,7 @@ +diff -udNr example-files/test2 example-files2/test2 +--- example-files/test2 2017-11-22 10:29:11.766891385 +1300 ++++ example-files2/test2 2017-11-22 10:30:08.920094705 +1300 +@@ -1 +1,3 @@ + Another example file ++ ++More content added here diff --git a/meta-selftest/recipes-test/devtool/devtool-test-multisrc/mypackage-1.0.tar.gz b/meta-selftest/recipes-test/devtool/devtool-test-multisrc/mypackage-1.0.tar.gz new file mode 100644 index 0000000000000000000000000000000000000000..b513b283e9adece54a2495ad5f9eaf8eb9eebdd0 GIT binary patch literal 903 zcmV;219<!&iwFP}s}x!Q1MOE!Z`(K!_F2DTkkf9Vb!^8Epf|>%V`62=kmP1_lqK5Y zt|bbh+-&~+&M0=XF3{$ZwkXOE0R#?bzIla1x_qkB?4NWm{}}cM*S&9o0b(|r(04eS z4F3N;qc^-64JO0UWIP=82E*ZOHtxaX+k|t8qiNFypjW5I!`Jj)+y6;XUi5z+t{+5j z*I$X`4Si69r_=G5`ac>BZ$9h)&3JOtgTWid@J0OM{a?$bRK_S>fih6YMsA;A-=tM5 za~F1vl%RKz6=}1VF0>lb>IrJu7&Oq^HdU3X_K-rR>nCtfwJngx=w17mHWIaSNR82% zN|6w9ogGVAwP{NZcj_Pw{MHr{uHuu$)$g5`T&4$bR0Xto-h)Tg7W&viBTd_=j7VHi zRdzV$6vcV_ph|UeLFSFx7p(yYX^#ezpio^Xb*^^wm0f1_aeGjva3NR3y*;)lG$hPq zMOK)}wQj)30|tf^3PbPGeTlDAqUdWXYI`b^A<1K*%gbD_O3v=sROnQ8M!Ckabw2-= zSxXYq-|53aKT>v?u5v|5nfH#9;8wcTA7z(KmmyTT#bBRfpeEEWtvzj-BE{&p@>DbQ zixqZAc21}v|9B#*R$&8b-E<B=PqdE#EIGvCBDrS~2NA<M3O@*sdvL{KTwl3xFOp@r zNdPq>79<Z4E`S9O@TUko7x-W6h{rL65jbMC_60|w2<HCA!vO@hXc&YE_yR`}vL+#r z#}k*}F|n<9G+*M9-3nhM4=y+hkp#rLKw1WC7A0c7@mU1xO|%YUjsbYc8;D>Lp&P#9 zLDEOJC<Fch7Z5L*?{~f&w!s9VE<~D(k+@qXunc{VqwtnvZ0y$O&%Q9PxzEJP1&^)R z9q+6{q`~w^?-bzul6Nxn$nbBTh%lf`=3$UTxO6emDEVZ2FJkTji$qK{Swtc7J5*CN z2|E@v4EQq*)f+C_f-1D$#QY^1k24?1VluvHV&D1EqsZb1;PU=2ah}su?q_f5<CXg# zUjUQy{XZU02Xz0>M)v;yF1UUNbeB-$yZad4T|0IX{5rTi{}E^2iodmwSI_@+`04yl zMs)s1Q{1-a|2u#IzIH#-V^98@mNgE5)Bl4@kH&%?p7;_q_%nch8}#jLu&}VOu&}VO du&}VOu&}VOu&}VOu(0?E@E>2ucu@c-008QizRdst literal 0 HcmV?d00001 diff --git a/meta-selftest/recipes-test/devtool/devtool-test-multisrc_1.0.bb b/meta-selftest/recipes-test/devtool/devtool-test-multisrc_1.0.bb new file mode 100644 index 0000000..2b81f83 --- /dev/null +++ b/meta-selftest/recipes-test/devtool/devtool-test-multisrc_1.0.bb @@ -0,0 +1,11 @@ +SUMMARY = "Recipe to test multiple source trees" +LICENSE = "MIT" +LIC_FILES_CHKSUM = "file://COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420" + +SRC_URI = "file://mypackage-${PV}.tar.gz \ + file://example-files.tar.gz \ + file://example.patch;patchdir=../example-files \ +" + +S = "${WORKDIR}/mypackage-${PV}" + diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py index 96072be..b051b7c 100644 --- a/meta/lib/oeqa/selftest/cases/devtool.py +++ b/meta/lib/oeqa/selftest/cases/devtool.py @@ -1568,6 +1568,115 @@ class DevtoolTests(DevtoolBase): if files: self.fail('Unexpected file(s) copied next to bbappend: %s' % ', '.join(files)) + def test_devtool_finish_modify_multisrc(self): + # Check preconditions + self.assertTrue(not os.path.exists(self.workspacedir), 'This test cannot be run with a workspace directory under the build directory') + # Try modifying a recipe + self.track_for_cleanup(self.workspacedir) + recipe = 'devtool-test-multisrc' + recipefile = get_bb_var('FILE', recipe) + recipedir = os.path.dirname(recipefile) + result = runCmd('git status --porcelain .', cwd=recipedir) + if result.output.strip(): + self.fail('Recipe directory for %s contains uncommitted changes' % recipe) + tempdir = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(tempdir) + self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') + result = runCmd('devtool modify %s %s' % (recipe, tempdir)) + mainsrc = os.path.join(tempdir, 'mypackage-1.0') + extrasrc = os.path.join(tempdir, 'example-files') + self.assertExists(os.path.join(mainsrc, 'mainfile.c'), 'Extracted main source tree could not be found') + self.assertExists(os.path.join(extrasrc, 'test2'), 'Extracted extra source tree could not be found') + # Try dry-run finishing without changes + result = runCmd('devtool finish %s meta-selftest -N' % recipe) + self.assertIn('No patches or files need updating', result.output) + # Make some changes to both the main and extra source trees + with open(os.path.join(mainsrc, 'mainfile.c'), 'a') as f: + f.write('\n/* Additional comment */\n') + result = runCmd('git commit -a -m "Add a comment"', cwd=mainsrc) + with open(os.path.join(extrasrc, 'test2'), 'a') as f: + f.write('\nAnother line\n') + result = runCmd('git commit -a -m "Add another line"', cwd=extrasrc) + # Try finishing now with dry-run to see if it reports what we expect + result = runCmd('devtool finish %s meta-selftest -N' % recipe) + self.assertNotIn('No patches or files need updating', result.output) + self.assertIn('+ file://0001-Add-a-comment.patch \\', result.output) + self.assertIn('+ file://0001-Add-another-line.patch;patchdir=../example-files \\', result.output) + self.assertNotIn('Removing', result.output) + # Now really finish + self.add_command_to_tearDown('rm -rf %s ; cd %s ; git checkout %s' % (recipedir, os.path.dirname(recipedir), recipedir)) + result = runCmd('devtool finish %s meta-selftest' % recipe) + expected_status = [(' M', '.*/%s$' % os.path.basename(recipefile)), + ('??', '.*/.*-Add-a-comment.patch$'), + ('??', '.*/.*-Add-another-line.patch$')] + self._check_repo_status(recipedir, expected_status) + + def test_devtool_finish_modify_multisrc_subdir(self): + # Check preconditions + self.assertTrue(not os.path.exists(self.workspacedir), 'This test cannot be run with a workspace directory under the build directory') + # Try modifying a recipe + self.track_for_cleanup(self.workspacedir) + recipe = 'devtool-test-multisrc-subdir' + recipefile = get_bb_var('FILE', recipe) + recipedir = os.path.dirname(recipefile) + result = runCmd('git status --porcelain .', cwd=recipedir) + if result.output.strip(): + self.fail('Recipe directory for %s contains uncommitted changes' % recipe) + tempdir = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(tempdir) + self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') + result = runCmd('devtool modify %s %s' % (recipe, tempdir)) + mainsrc = tempdir + extrasrc = os.path.join(tempdir, 'libfakekey') + self.assertExists(os.path.join(mainsrc, 'xsettings-client.c'), 'Extracted main source tree could not be found') + self.assertExists(os.path.join(extrasrc, 'configure.ac'), 'Extracted extra source tree could not be found') + # Try dry-run finishing without changes + result = runCmd('devtool finish %s meta-selftest -N' % recipe) + self.assertIn('No patches or files need updating', result.output) + # Make some changes to both the main and extra source trees + with open(os.path.join(mainsrc, 'xsettings-client.c'), 'a') as f: + f.write('\n/* Additional comment */\n') + result = runCmd('git commit -a -m "Add a comment"', cwd=mainsrc) + with open(os.path.join(extrasrc, 'README'), 'a') as f: + f.write('\nAnother line\n') + result = runCmd('git commit -a -m "Add another line to README"', cwd=extrasrc) + # Try finishing now with dry-run to see if it reports what we expect + # However, the first time it will fail because git treats sub-repositories + # the same as submodules, so the parent will appear to have uncommitted changes + failed = False + try: + result = runCmd('devtool finish %s meta-selftest -N' % recipe, assert_error=False) + except CommandError as err: + if err.retcode != 1: + self.fail('Unexpected failure from devtool finish:\n%s' % err.output) + lines = [] + collecting = False + for line in err.output.splitlines(): + if collecting: + if line: + lines.append(line) + elif line.startswith('ERROR: Source tree %s is not clean' % mainsrc): + collecting = True + lines = lines[:-1] + self.assertEqual(lines, [' M libfakekey'], 'Unexpected modifications to main source tree') + if not collecting: + self.fail('devtool finish did not report uncommitted changes as expected:\n%s' % err.output) + failed = True + if not failed: + self.fail('devtool finish was expected to fail but did not:\n%s' % result.output) + result = runCmd('devtool finish %s meta-selftest -N -f' % recipe) + self.assertNotIn('No patches or files need updating', result.output) + self.assertIn('+ file://0001-Add-a-comment.patch \\', result.output) + self.assertIn('+ file://0001-Add-another-line-to-README.patch;patchdir=libfakekey \\', result.output) + self.assertNotIn('Removing', result.output) + self.add_command_to_tearDown('rm -rf %s ; cd %s ; git checkout %s' % (recipedir, os.path.dirname(recipedir), recipedir)) + result = runCmd('devtool finish %s -f meta-selftest' % recipe) + expected_status = [(' M', '.*/%s$' % os.path.basename(recipefile)), + ('??', '.*/%s/$' % recipe)] + self._check_repo_status(recipedir, expected_status) + self.assertExists(os.path.join(os.path.dirname(recipefile), recipe, '0001-Add-a-comment.patch')) + self.assertExists(os.path.join(os.path.dirname(recipefile), recipe, '0001-Add-another-line-to-README.patch')) + @OETestID(1626) def test_devtool_rename(self): # Check preconditions -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 12/12] lib/oe/recipeutils: add .txz extension to archive list 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton ` (10 preceding siblings ...) 2017-12-05 1:41 ` [PATCH 11/12] oe-selftest: devtool: add test for multiple source trees Paul Eggleton @ 2017-12-05 1:41 ` Paul Eggleton 2017-12-05 2:07 ` ✗ patchtest: failure for devtool improvements Patchwork 2017-12-13 14:53 ` [PATCH 00/12] " Alexander Kanavin 13 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-05 1:41 UTC (permalink / raw) To: openembedded-core Prompted by bitbake commit 2ba8a6b25ccc12e7b543e8450121e5311c7a701d, add .txz to the list of archives used within get_recipe_local_files() here as well. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> --- meta/lib/oe/recipeutils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py index f4a40c2..ac3b027 100644 --- a/meta/lib/oe/recipeutils.py +++ b/meta/lib/oe/recipeutils.py @@ -413,7 +413,7 @@ def get_recipe_local_files(d, patches=False, archives=False): # fetcher) though note that this only encompasses actual container formats # i.e. that can contain multiple files as opposed to those that only # contain a compressed stream (i.e. .tar.gz as opposed to just .gz) - archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z'] + archive_exts = ['.tar', '.tgz', '.tar.gz', '.tar.Z', '.tbz', '.tbz2', '.tar.bz2', '.txz', '.tar.xz', '.tar.lz', '.zip', '.jar', '.rpm', '.srpm', '.deb', '.ipk', '.tar.7z', '.7z'] ret = {} for uri in uris: if fetch.ud[uri].type == 'file': -- 2.9.5 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* ✗ patchtest: failure for devtool improvements 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton ` (11 preceding siblings ...) 2017-12-05 1:41 ` [PATCH 12/12] lib/oe/recipeutils: add .txz extension to archive list Paul Eggleton @ 2017-12-05 2:07 ` Patchwork 2017-12-13 14:53 ` [PATCH 00/12] " Alexander Kanavin 13 siblings, 0 replies; 16+ messages in thread From: Patchwork @ 2017-12-05 2:07 UTC (permalink / raw) To: Paul Eggleton; +Cc: openembedded-core == Series Details == Series: devtool improvements Revision: 1 URL : https://patchwork.openembedded.org/series/10071/ State : failure == Summary == Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures: * Issue A patch file has been added, but does not have a Signed-off-by tag [test_signed_off_by_presence] Suggested fix Sign off the added patch file (meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch) * Issue Added patch file is missing Upstream-Status in the header [test_upstream_status_presence_format] Suggested fix Add Upstream-Status: <Valid status> to the header of meta-selftest/recipes-test/devtool/devtool-test-multisrc/example.patch Standard format Upstream-Status: <Valid status> Valid status Pending, Accepted, Backport, Denied, Inappropriate [reason], Submitted [where] If you believe any of these test results are incorrect, please reply to the mailing list (openembedded-core@lists.openembedded.org) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...). --- Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 00/12] devtool improvements 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton ` (12 preceding siblings ...) 2017-12-05 2:07 ` ✗ patchtest: failure for devtool improvements Patchwork @ 2017-12-13 14:53 ` Alexander Kanavin 2017-12-13 17:46 ` Paul Eggleton 13 siblings, 1 reply; 16+ messages in thread From: Alexander Kanavin @ 2017-12-13 14:53 UTC (permalink / raw) To: Paul Eggleton, openembedded-core, Burton, Ross, Richard Purdie On 12/05/2017 03:41 AM, Paul Eggleton wrote: > A set of improvements for devtool, mostly aimed at improving the > handling of recipes that fetch multiple source trees (i.e. have multiple > tarballs or repositories in SRC_URI), with a couple of other fixes > thrown in. Unfortunately, this badly breaks 'devtool upgrade': ====== ak@kanavin-desktop:~/development/poky/build$ devtool upgrade openssl -V 1.0.2n ..... WARNING: This recipe fetches more than one source tree. Each source tree will be tracked in a separate git repository. NOTE: Extracting upgraded version source... ERROR: Execution of 'git checkout devtool-base -b devtool-1.0.2n' failed with exit code 128: fatal: 'devtool-base' is not a commit and a branch 'devtool-1.0.2n' cannot be created from it ===== I would strongly request that this patchset not be merged into master until the issue is resolved. Alex ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 00/12] devtool improvements 2017-12-13 14:53 ` [PATCH 00/12] " Alexander Kanavin @ 2017-12-13 17:46 ` Paul Eggleton 0 siblings, 0 replies; 16+ messages in thread From: Paul Eggleton @ 2017-12-13 17:46 UTC (permalink / raw) To: Alexander Kanavin; +Cc: openembedded-core Hi Alex, On Thursday, 14 December 2017 3:53:22 AM NZDT Alexander Kanavin wrote: > On 12/05/2017 03:41 AM, Paul Eggleton wrote: > > A set of improvements for devtool, mostly aimed at improving the > > handling of recipes that fetch multiple source trees (i.e. have multiple > > tarballs or repositories in SRC_URI), with a couple of other fixes > > thrown in. > > Unfortunately, this badly breaks 'devtool upgrade': > > ====== > ak@kanavin-desktop:~/development/poky/build$ devtool upgrade openssl -V > 1.0.2n > ..... > WARNING: This recipe fetches more than one source tree. Each source tree > will be tracked in a separate git repository. > NOTE: Extracting upgraded version source... > ERROR: Execution of 'git checkout devtool-base -b devtool-1.0.2n' failed > with exit code 128: > fatal: 'devtool-base' is not a commit and a branch 'devtool-1.0.2n' > cannot be created from it > ===== > > I would strongly request that this patchset not be merged into master > until the issue is resolved. That's interesting - the devtool upgrade tests apparently passed so clearly we have a hole in our tests. I will look into it. Cheers, Paul -- Paul Eggleton Intel Open Source Technology Centre ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2017-12-13 17:46 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2017-12-05 1:41 [PATCH 00/12] devtool improvements Paul Eggleton 2017-12-05 1:41 ` [PATCH 01/12] devtool: refactor return for _extract_source() Paul Eggleton 2017-12-05 1:41 ` [PATCH 02/12] devtool: reset: delete bbappend file if _check_preserve() doesn't Paul Eggleton 2017-12-05 1:41 ` [PATCH 03/12] devtool: finish: fix erroneously creating bbappend for relative paths Paul Eggleton 2017-12-05 1:41 ` [PATCH 04/12] devtool: deploy-target: don't specify ssh/scp port unless user does Paul Eggleton 2017-12-05 1:41 ` [PATCH 05/12] devtool: extract: fix usage with kern-tools-native Paul Eggleton 2017-12-05 1:41 ` [PATCH 06/12] devtool: modify: pick up commits from previously created source tree Paul Eggleton 2017-12-05 1:41 ` [PATCH 07/12] lib/oe/recipeutils: allow patch_recipe_file() to be re-called Paul Eggleton 2017-12-05 1:41 ` [PATCH 08/12] devtool: support extracting multiple source trees Paul Eggleton 2017-12-05 1:41 ` [PATCH 09/12] devtool: fix handling of linux-yocto after multisrc changes Paul Eggleton 2017-12-05 1:41 ` [PATCH 10/12] oe-selftest: devtool: fix kernel test for " Paul Eggleton 2017-12-05 1:41 ` [PATCH 11/12] oe-selftest: devtool: add test for multiple source trees Paul Eggleton 2017-12-05 1:41 ` [PATCH 12/12] lib/oe/recipeutils: add .txz extension to archive list Paul Eggleton 2017-12-05 2:07 ` ✗ patchtest: failure for devtool improvements Patchwork 2017-12-13 14:53 ` [PATCH 00/12] " Alexander Kanavin 2017-12-13 17:46 ` Paul Eggleton
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox