* [PATCH 01/24] recipetool: pass absolute source tree path to plugins
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 02/24] recipetool: ignore incidental kernel module source Paul Eggleton
` (22 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
We shouldn't be passing a relative path to the plugins if that's what's
been specified on the recipetool command line.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index ca474fc..1532735 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -613,9 +613,9 @@ def create_recipe(args):
if args.src_subdir:
srcsubdir = os.path.join(srcsubdir, args.src_subdir)
- srctree_use = os.path.join(srctree, args.src_subdir)
+ srctree_use = os.path.abspath(os.path.join(srctree, args.src_subdir))
else:
- srctree_use = srctree
+ srctree_use = os.path.abspath(srctree)
if args.outfile and os.path.isdir(args.outfile):
outfile = None
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 02/24] recipetool: ignore incidental kernel module source
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
2017-11-09 1:55 ` [PATCH 01/24] recipetool: pass absolute source tree path to plugins Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 03/24] lib/oe/recipeutils: fix find_layerdir() to return absolute paths Paul Eggleton
` (21 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
If the source tree happens to contain a kernel module as an example, a
test or under a "contrib" directory then we shouldn't be picking it up
and making the determination that the entire thing is a kernel module.
An example that triggered this is zstd, which ships a kernel module
under contrib/linux-kernel:
https://github.com/facebook/zstd
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 6 ++++--
scripts/lib/recipetool/create_kmod.py | 2 +-
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 1532735..055bdef 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -156,10 +156,12 @@ class RecipeHandler(object):
RecipeHandler.recipebinmap[prog] = pn
@staticmethod
- def checkfiles(path, speclist, recursive=False):
+ def checkfiles(path, speclist, recursive=False, excludedirs=None):
results = []
if recursive:
- for root, _, files in os.walk(path):
+ for root, dirs, files in os.walk(path, topdown=True):
+ if excludedirs:
+ dirs[:] = [d for d in dirs if d not in excludedirs]
for fn in files:
for spec in speclist:
if fnmatch.fnmatch(fn, spec):
diff --git a/scripts/lib/recipetool/create_kmod.py b/scripts/lib/recipetool/create_kmod.py
index 7cf188d..4569b53 100644
--- a/scripts/lib/recipetool/create_kmod.py
+++ b/scripts/lib/recipetool/create_kmod.py
@@ -40,7 +40,7 @@ class KernelModuleRecipeHandler(RecipeHandler):
makefiles = []
- files = RecipeHandler.checkfiles(srctree, ['*.c', '*.h'], recursive=True)
+ files = RecipeHandler.checkfiles(srctree, ['*.c', '*.h'], recursive=True, excludedirs=['contrib', 'test', 'examples'])
if files:
for cfile in files:
# Look in same dir or parent for Makefile
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 03/24] lib/oe/recipeutils: fix find_layerdir() to return absolute paths
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
2017-11-09 1:55 ` [PATCH 01/24] recipetool: pass absolute source tree path to plugins Paul Eggleton
2017-11-09 1:55 ` [PATCH 02/24] recipetool: ignore incidental kernel module source Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 04/24] lib/oe/recipeutils: fix line splitting in patch_recipe_* Paul Eggleton
` (20 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
find_layerdir() should really return absolute paths, so make it do so.
This fixes devtool finish not deleting files it should do after devtool
upgrade if the specified path is relative, since the devtool finish code
was assuming that find_layerdir() was returning an absolute path.
Fixes [YOCTO #12318].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/recipeutils.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index c8570ac..2f818bc 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -801,7 +801,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
def find_layerdir(fn):
""" Figure out the path to the base of the layer containing a file (e.g. a recipe)"""
- pth = fn
+ pth = os.path.abspath(fn)
layerdir = ''
while pth:
if os.path.exists(os.path.join(pth, 'conf', 'layer.conf')):
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 04/24] lib/oe/recipeutils: fix line splitting in patch_recipe_*
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (2 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 03/24] lib/oe/recipeutils: fix find_layerdir() to return absolute paths Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 05/24] devtool: upgrade: fix accidentally swapped parameters Paul Eggleton
` (19 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
If a value was split over multiple lines (e.g. as SRC_URI usually is)
then we were inserting the value as one item in the lines list with
newlines between each line. There's nothing wrong with this if you're
writing the list out to a file, but if you want to generate a patch (as
patch_recipe_file() will do if the patch parameter is set to True) then
the diff output looks a bit odd. Split the value before adding it to the
lines list to resolve this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/recipeutils.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index 2f818bc..cab8e40 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -188,6 +188,11 @@ def patch_recipe_lines(fromlines, values, trailing_newline=True):
for wrapline in wrapped[:-1]:
addlines.append('%s \\%s' % (wrapline, newline))
addlines.append('%s%s' % (wrapped[-1], newline))
+
+ # Split on newlines - this isn't strictly necessary if you are only
+ # going to write the output to disk, but if you want to compare it
+ # (as patch_recipe_file() will do if patch=True) then it's important.
+ addlines = [line for l in addlines for line in l.splitlines(True)]
if rewindcomments:
# Ensure we insert the lines before any leading comments
# (that we'd want to ensure remain leading the next value)
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 05/24] devtool: upgrade: fix accidentally swapped parameters
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (3 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 04/24] lib/oe/recipeutils: fix line splitting in patch_recipe_* Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 06/24] devtool: upgrade: fix not committing deleted files with older git versions Paul Eggleton
` (18 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
It appears that when fixing the signature unlocking in OE-Core commit
4e9a0be32fc30fb87d65da7cd1a4015c99533aff I swapped the parameters here
and did not test it within the eSDK (it does nothing outside of the
eSDK) resulting in a TypeError when devtool upgrade was used in the
eSDK. Swap the parameters around to the correct ordering.
Fixes [YOCTO #12285].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/upgrade.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index f1b3ff0..4cfab0c 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -433,7 +433,7 @@ def upgrade(args, config, basepath, workspace):
copied, config.workspace_path, rd)
standard._add_md5(config, pn, af)
- update_unlockedsigs(basepath, workspace, [pn], args.fixed_setup)
+ update_unlockedsigs(basepath, workspace, args.fixed_setup, [pn])
logger.info('Upgraded source extracted to %s' % srctree)
logger.info('New recipe is %s' % rf)
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 06/24] devtool: upgrade: fix not committing deleted files with older git versions
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (4 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 05/24] devtool: upgrade: fix accidentally swapped parameters Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 07/24] devtool: upgrade: improve performance and show progress when adding files Paul Eggleton
` (17 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
With versions of git older than 2.0, "git add" on a deleted file (i.e.
in this case a file that was removed between versions) will not add the
delete to be committed by default, with the result that the rebase of
patches on top of the new branch will fail. We need to use the -A
option in order to force that for older git versions.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/upgrade.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 4cfab0c..073002b 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -252,7 +252,7 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee
(stdout,_) = __run('git ls-files --modified --others --exclude-standard')
for f in stdout.splitlines():
- __run('git add "%s"' % f)
+ __run('git add -A "%s"' % f)
useroptions = []
oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=rd)
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 07/24] devtool: upgrade: improve performance and show progress when adding files
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (5 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 06/24] devtool: upgrade: fix not committing deleted files with older git versions Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 08/24] devtool: fix handling of oe-local-files when source is in a subdirectory Paul Eggleton
` (16 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
When devtool upgrade is upgrading to a new version where the source is
fetched as an archive (e.g. a tarball), we create a single commit in the
git repository that is the upgrade from the old version to the new. We
do this by extracting the old source, committing it, deleting all files,
copying in the new files, running git add on each new/changed/deleted
file, and then committing the result. When a lot of files have changed
in an upgrade (such as QEMU 2.8.1.1 -> 2.10.0) the penultimate step of
running git add it can take quite a long time; in order to reduce this
and show some feedback to the user, run git add with batches of 100
files at once and also show a progress bar. In a local test with the
aforementioned QEMU upgrade it took the time down from over 7 minutes
down to about 13 seconds.
Fixes [YOCTO #11948].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/upgrade.py | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 073002b..441dd35 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -251,8 +251,15 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee
_copy_source_code(tmpsrctree, srctree)
(stdout,_) = __run('git ls-files --modified --others --exclude-standard')
- for f in stdout.splitlines():
- __run('git add -A "%s"' % f)
+ filelist = stdout.splitlines()
+ pbar = bb.ui.knotty.BBProgress('Adding changed files', len(filelist))
+ pbar.start()
+ batchsize = 100
+ for i in range(0, len(filelist), batchsize):
+ batch = filelist[i:i+batchsize]
+ __run('git add -A %s' % ' '.join(['"%s"' % item for item in batch]))
+ pbar.update(i)
+ pbar.finish()
useroptions = []
oe.patch.GitApplyTree.gitCommandUserOptions(useroptions, d=rd)
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 08/24] devtool: fix handling of oe-local-files when source is in a subdirectory
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (6 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 07/24] devtool: upgrade: improve performance and show progress when adding files Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 09/24] devtool: show some warnings for upgrade versions Paul Eggleton
` (15 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
If S points to a subdirectory of the source rather than the "base" of
the source tree then we weren't handling the oe-local-files directory
properly - it got extracted to the base of the tree but devtool
update-recipe and devtool finish assumed it would be under S which would
be the subdirectory, thus it would be missing and devtool would assume
the files had been deleted and remove them from the recipe. Record the
base of the source tree in the bbappend and read it into the in-memory
workspace so we can use that to find out where oe-local-files should be
found.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/devtool | 15 +++++++++++----
scripts/lib/devtool/standard.py | 24 +++++++++++++++---------
2 files changed, 26 insertions(+), 13 deletions(-)
diff --git a/scripts/devtool b/scripts/devtool
index 5292f18..87bb5c8 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -113,6 +113,7 @@ def read_workspace():
externalsrc_re = re.compile(r'^EXTERNALSRC(_pn-([^ =]+))? *= *"([^"]*)"$')
for fn in glob.glob(os.path.join(config.workspace_path, 'appends', '*.bbappend')):
with open(fn, 'r') as f:
+ pnvalues = {}
for line in f:
res = externalsrc_re.match(line.rstrip())
if res:
@@ -125,10 +126,16 @@ def read_workspace():
bbfile))
if recipefile:
recipefile = recipefile[0]
- workspace[pn] = {'srctree': res.group(3),
- 'bbappend': fn,
- 'recipefile': recipefile}
- logger.debug('Found recipe %s' % workspace[pn])
+ pnvalues['srctree'] = res.group(3)
+ pnvalues['bbappend'] = fn
+ pnvalues['recipefile'] = recipefile
+ elif line.startswith('# srctreebase: '):
+ pnvalues['srctreebase'] = line.split(':', 1)[1].strip()
+ if pnvalues:
+ if not pnvalues.get('srctreebase', None):
+ pnvalues['srctreebase'] = pnvalues['srctree']
+ logger.debug('Found recipe %s' % pnvalues)
+ workspace[pn] = pnvalues
def create_workspace(args, config, basepath, workspace):
if args.layerpath:
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index beea0d4c2..8e4c7f7 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -734,6 +734,9 @@ def modify(args, config, basepath, workspace):
(stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
initial_rev = stdout.rstrip()
+ # Need to grab this here in case the source is within a subdirectory
+ srctreebase = srctree
+
# Check that recipe isn't using a shared workdir
s = os.path.abspath(rd.getVar('S'))
workdir = os.path.abspath(rd.getVar('WORKDIR'))
@@ -748,7 +751,8 @@ def modify(args, config, basepath, workspace):
# Local files can be modified/tracked in separate subdir under srctree
# Mostly useful for packages with S != WORKDIR
f.write('FILESPATH_prepend := "%s:"\n' %
- os.path.join(srctree, 'oe-local-files'))
+ os.path.join(srctreebase, 'oe-local-files'))
+ f.write('# srctreebase: %s\n' % srctreebase)
f.write('\ninherit externalsrc\n')
f.write('# NOTE: We use pn- overrides here to avoid affecting multiple variants in the case where the recipe uses BBCLASSEXTEND\n')
@@ -1166,7 +1170,7 @@ def _create_kconfig_diff(srctree, rd, outfile):
return False
-def _export_local_files(srctree, rd, destdir):
+def _export_local_files(srctree, rd, destdir, srctreebase):
"""Copy local files from srctree to given location.
Returns three-tuple of dicts:
1. updated - files that already exist in SRCURI
@@ -1186,7 +1190,7 @@ def _export_local_files(srctree, rd, destdir):
updated = OrderedDict()
added = OrderedDict()
removed = OrderedDict()
- local_files_dir = os.path.join(srctree, 'oe-local-files')
+ local_files_dir = os.path.join(srctreebase, 'oe-local-files')
git_files = _git_ls_tree(srctree)
if 'oe-local-files' in git_files:
# If tracked by Git, take the files from srctree HEAD. First get
@@ -1199,9 +1203,9 @@ def _export_local_files(srctree, rd, destdir):
new_set = list(_git_ls_tree(srctree, tree, True).keys())
elif os.path.isdir(local_files_dir):
# If not tracked by Git, just copy from working copy
- new_set = _ls_tree(os.path.join(srctree, 'oe-local-files'))
+ new_set = _ls_tree(local_files_dir)
bb.process.run(['cp', '-ax',
- os.path.join(srctree, 'oe-local-files', '.'), destdir])
+ os.path.join(local_files_dir, '.'), destdir])
else:
new_set = []
@@ -1266,7 +1270,7 @@ def _determine_files_dir(rd):
return os.path.join(recipedir, rd.getVar('BPN'))
-def _update_recipe_srcrev(srctree, rd, appendlayerdir, wildcard_version, no_remove):
+def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove):
"""Implement the 'srcrev' mode of update-recipe"""
import bb
import oe.recipeutils
@@ -1294,7 +1298,8 @@ def _update_recipe_srcrev(srctree, rd, appendlayerdir, wildcard_version, no_remo
update_srcuri = False
try:
local_files_dir = tempfile.mkdtemp(dir=tempdir)
- upd_f, new_f, del_f = _export_local_files(srctree, rd, local_files_dir)
+ srctreebase = workspace[recipename]['srctreebase']
+ upd_f, new_f, del_f = _export_local_files(srctree, rd, local_files_dir, srctreebase)
if not no_remove:
# Find list of existing patches in recipe file
patches_dir = tempfile.mkdtemp(dir=tempdir)
@@ -1372,7 +1377,8 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
tempdir = tempfile.mkdtemp(prefix='devtool')
try:
local_files_dir = tempfile.mkdtemp(dir=tempdir)
- upd_f, new_f, del_f = _export_local_files(srctree, rd, local_files_dir)
+ srctreebase = workspace[recipename]['srctreebase']
+ upd_f, new_f, del_f = _export_local_files(srctree, rd, local_files_dir, srctreebase)
remove_files = []
if not no_remove:
@@ -1495,7 +1501,7 @@ def _update_recipe(recipename, workspace, rd, mode, appendlayerdir, wildcard_ver
mode = _guess_recipe_update_mode(srctree, rd)
if mode == 'srcrev':
- updated = _update_recipe_srcrev(srctree, rd, appendlayerdir, wildcard_version, no_remove)
+ updated = _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove)
elif mode == 'patch':
updated = _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, initial_rev)
else:
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 09/24] devtool: show some warnings for upgrade versions
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (7 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 08/24] devtool: fix handling of oe-local-files when source is in a subdirectory Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 10/24] devtool: make find-recipe and edit-recipe always work with any recipe Paul Eggleton
` (14 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
* Show a warning in devtool upgrade if the version is less than the
current version suggesting that the user may need to bump PE in the
recipe
* Show a warning in devtool add and devtool upgrade if the version looks
like a pre-release version suggesting using a version number that
won't mess up the progression when you come to upgrade to the final
release version.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/__init__.py | 13 +++++++++++++
scripts/lib/devtool/standard.py | 4 +++-
scripts/lib/devtool/upgrade.py | 9 +++++++--
3 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 94e3d7d..20ab83f 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -337,3 +337,16 @@ def update_unlockedsigs(basepath, workspace, fixed_setup, extra=None):
for pn in newunlocked:
f.write(' ' + pn)
f.write('"')
+
+def check_prerelease_version(ver, operation):
+ if 'pre' in ver or 'rc' in ver:
+ logger.warning('Version "%s" looks like a pre-release version. '
+ 'If that is the case, in order to ensure that the '
+ 'version doesn\'t appear to go backwards when you '
+ 'later upgrade to the final release version, it is '
+ 'recommmended that instead you use '
+ '<current version>+<pre-release version> e.g. if '
+ 'upgrading from 1.9 to 2.0-rc2 use "1.9+2.0-rc2". '
+ 'If you prefer not to reset and re-try, you can change '
+ 'the version after %s succeeds using "devtool rename" '
+ 'with -V/--version.' % (ver, operation))
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 8e4c7f7..b6e532b 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -30,7 +30,7 @@ import errno
import glob
import filecmp
from collections import OrderedDict
-from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, update_unlockedsigs, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, update_unlockedsigs, check_prerelease_version, DevtoolError
from devtool import parse_recipe
logger = logging.getLogger('devtool')
@@ -298,6 +298,8 @@ def add(args, config, basepath, workspace):
_add_md5(config, recipename, appendfile)
+ check_prerelease_version(rd.getVar('PV'), 'devtool add')
+
logger.info('Recipe %s has been automatically created; further editing may be required to make it fully functional' % recipefile)
finally:
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 441dd35..ab7acd1 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -33,7 +33,7 @@ sys.path = sys.path + [devtool_path]
import oe.recipeutils
from devtool import standard
-from devtool import exec_build_env_command, setup_tinfoil, DevtoolError, parse_recipe, use_external_build, update_unlockedsigs
+from devtool import exec_build_env_command, setup_tinfoil, DevtoolError, parse_recipe, use_external_build, update_unlockedsigs, check_prerelease_version
logger = logging.getLogger('devtool')
@@ -420,8 +420,13 @@ def upgrade(args, config, basepath, workspace):
old_srcrev = None
if old_srcrev and not args.srcrev:
raise DevtoolError("Recipe specifies a SRCREV value; you must specify a new one when upgrading")
- if rd.getVar('PV') == args.version and old_srcrev == args.srcrev:
+ old_ver = rd.getVar('PV')
+ if old_ver == args.version and old_srcrev == args.srcrev:
raise DevtoolError("Current and upgrade versions are the same version")
+ if args.version:
+ if bb.utils.vercmp_string(args.version, old_ver) < 0:
+ logger.warning('Upgrade version %s compares as less than the current version %s. If you are using a package feed for on-target upgrades or providing this recipe for general consumption, then you should increment PE in the recipe (or if there is no current PE value set, set it to "1")' % (args.version, old_ver))
+ check_prerelease_version(args.version, 'devtool upgrade')
rf = None
try:
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 10/24] devtool: make find-recipe and edit-recipe always work with any recipe
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (8 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 09/24] devtool: show some warnings for upgrade versions Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 11/24] devtool: reset: print source tree base path Paul Eggleton
` (13 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
After some reconsideration I think it's a bit annoying for users to be
forced to use an option to work with recipes where the file isn't in the
workspace, so let's just have these commands check the workspace first
for the recipe, and if it isn't there then load the cache and get it
that way.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/utilcmds.py | 26 ++++++++++++++------------
1 file changed, 14 insertions(+), 12 deletions(-)
diff --git a/scripts/lib/devtool/utilcmds.py b/scripts/lib/devtool/utilcmds.py
index b745116..7cd139f 100644
--- a/scripts/lib/devtool/utilcmds.py
+++ b/scripts/lib/devtool/utilcmds.py
@@ -32,6 +32,12 @@ logger = logging.getLogger('devtool')
def _find_recipe_path(args, config, basepath, workspace):
if args.any_recipe:
+ logger.warning('-a/--any-recipe option is now always active, and thus the option will be removed in a future release')
+ if args.recipename in workspace:
+ recipefile = workspace[args.recipename]['recipefile']
+ else:
+ recipefile = None
+ if not recipefile:
tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
try:
rd = parse_recipe(config, tinfoil, args.recipename, True)
@@ -40,12 +46,6 @@ def _find_recipe_path(args, config, basepath, workspace):
recipefile = rd.getVar('FILE')
finally:
tinfoil.shutdown()
- else:
- check_workspace_recipe(workspace, args.recipename)
- recipefile = workspace[args.recipename]['recipefile']
- if not recipefile:
- raise DevtoolError("Recipe file for %s is not under the workspace" %
- args.recipename)
return recipefile
@@ -222,19 +222,21 @@ The ./configure %s output for %s follows.
def register_commands(subparsers, context):
"""Register devtool subcommands from this plugin"""
- parser_edit_recipe = subparsers.add_parser('edit-recipe', help='Edit a recipe file in your workspace',
- description='Runs the default editor (as specified by the EDITOR variable) on the specified recipe. Note that the recipe file itself must be in the workspace (i.e. as a result of "devtool add" or "devtool upgrade"); you can override this with the -a/--any-recipe option.',
+ parser_edit_recipe = subparsers.add_parser('edit-recipe', help='Edit a recipe file',
+ description='Runs the default editor (as specified by the EDITOR variable) on the specified recipe. Note that this will be quicker for recipes in the workspace as the cache does not need to be loaded in that case.',
group='working')
parser_edit_recipe.add_argument('recipename', help='Recipe to edit')
- parser_edit_recipe.add_argument('--any-recipe', '-a', action="store_true", help='Edit any recipe, not just where the recipe file itself is in the workspace')
+ # FIXME drop -a at some point in future
+ parser_edit_recipe.add_argument('--any-recipe', '-a', action="store_true", help='Does nothing (exists for backwards-compatibility)')
parser_edit_recipe.set_defaults(func=edit_recipe)
# Find-recipe
- parser_find_recipe = subparsers.add_parser('find-recipe', help='Find a recipe file in your workspace',
- description='By default, this will find a recipe file in your workspace; you can override this with the -a/--any-recipe option.',
+ parser_find_recipe = subparsers.add_parser('find-recipe', help='Find a recipe file',
+ description='Finds a recipe file. Note that this will be quicker for recipes in the workspace as the cache does not need to be loaded in that case.',
group='working')
parser_find_recipe.add_argument('recipename', help='Recipe to find')
- parser_find_recipe.add_argument('--any-recipe', '-a', action="store_true", help='Find any recipe, not just where the recipe file itself is in the workspace')
+ # FIXME drop -a at some point in future
+ parser_find_recipe.add_argument('--any-recipe', '-a', action="store_true", help='Does nothing (exists for backwards-compatibility)')
parser_find_recipe.set_defaults(func=find_recipe)
# NOTE: Needed to override the usage string here since the default
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 11/24] devtool: reset: print source tree base path
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (9 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 10/24] devtool: make find-recipe and edit-recipe always work with any recipe Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 12/24] devtool: finish: ensure repository is clean before proceeding Paul Eggleton
` (12 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
If S points to a subdirectory of the source rather than the "base" of
the source tree then print that rather than the subdirectory path when
telling the user they need to remove the source tree, since that is the
directory that they will need to remove.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index b6e532b..9bd2d86 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1620,16 +1620,16 @@ def _reset(recipes, no_clean, config, basepath, workspace):
# We don't automatically create this dir next to appends, but the user can
preservedir(os.path.join(config.workspace_path, 'appends', pn))
- srctree = workspace[pn]['srctree']
- if os.path.isdir(srctree):
- if os.listdir(srctree):
+ srctreebase = workspace[pn]['srctreebase']
+ if os.path.isdir(srctreebase):
+ if os.listdir(srctreebase):
# We don't want to risk wiping out any work in progress
logger.info('Leaving source tree %s as-is; if you no '
'longer need it then please delete it manually'
- % srctree)
+ % srctreebase)
else:
# This is unlikely, but if it's empty we can just remove it
- os.rmdir(srctree)
+ os.rmdir(srctreebase)
clean_preferred_provider(pn, config.workspace_path)
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 12/24] devtool: finish: ensure repository is clean before proceeding
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (10 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 11/24] devtool: reset: print source tree base path Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 13/24] devtool: finish: fix "layer not in bblayers.conf" warning when path specified Paul Eggleton
` (11 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
If the git repository for a recipe in the workspace has uncommitted
changes in it then it's possible that the user has forgotten to commit
something, so check and exit if there are any. Provide a -f/--force
option to continue in the case where the uncommitted changes aren't
needed.
Separately, if the repository is in the middle of a rebase or git am /
apply then error out (without the opportunity to force) since the user
really needs to sort this out before finishing.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/__init__.py | 17 +++++++++++++++++
scripts/lib/devtool/standard.py | 17 +++++++++++++++--
2 files changed, 32 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 20ab83f..07d774d 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -350,3 +350,20 @@ def check_prerelease_version(ver, operation):
'If you prefer not to reset and re-try, you can change '
'the version after %s succeeds using "devtool rename" '
'with -V/--version.' % (ver, operation))
+
+def check_git_repo_dirty(repodir):
+ """Check if a git repository is clean or not"""
+ stdout, _ = bb.process.run('git status --porcelain', cwd=repodir)
+ return stdout
+
+def check_git_repo_op(srctree, ignoredirs=None):
+ """Check if a git repository is in the middle of a rebase"""
+ stdout, _ = bb.process.run('git rev-parse --show-toplevel', cwd=srctree)
+ topleveldir = stdout.strip()
+ if ignoredirs and topleveldir in ignoredirs:
+ return
+ gitdir = os.path.join(topleveldir, '.git')
+ if os.path.exists(os.path.join(gitdir, 'rebase-merge')):
+ raise DevtoolError("Source tree %s appears to be in the middle of a rebase - please resolve this first" % srctree)
+ if os.path.exists(os.path.join(gitdir, 'rebase-apply')):
+ raise DevtoolError("Source tree %s appears to be in the middle of 'git am' or 'git apply' - please resolve this first" % srctree)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 9bd2d86..7fee304 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -30,7 +30,7 @@ import errno
import glob
import filecmp
from collections import OrderedDict
-from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, update_unlockedsigs, check_prerelease_version, DevtoolError
+from devtool import exec_build_env_command, setup_tinfoil, check_workspace_recipe, use_external_build, setup_git_repo, recipe_to_append, get_bbclassextend_targets, update_unlockedsigs, check_prerelease_version, check_git_repo_dirty, check_git_repo_op, DevtoolError
from devtool import parse_recipe
logger = logging.getLogger('devtool')
@@ -1675,6 +1675,18 @@ def finish(args, config, basepath, workspace):
check_workspace_recipe(workspace, args.recipename)
+ # Grab the equivalent of COREBASE without having to initialise tinfoil
+ corebasedir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', '..'))
+
+ srctree = workspace[args.recipename]['srctree']
+ check_git_repo_op(srctree, [corebasedir])
+ dirty = check_git_repo_dirty(srctree)
+ if dirty:
+ if args.force:
+ logger.warning('Source tree is not clean, continuing as requested by -f/--force')
+ else:
+ raise DevtoolError('Source tree is not clean:\n\n%s\nEnsure you have committed your changes or use -f/--force if you are sure there\'s nothing that needs to be committed' % dirty)
+
no_clean = False
tinfoil = setup_tinfoil(basepath=basepath, tracking=True)
try:
@@ -1867,10 +1879,11 @@ def register_commands(subparsers, context):
parser_reset.set_defaults(func=reset)
parser_finish = subparsers.add_parser('finish', help='Finish working on a recipe in your workspace',
- description='Pushes any committed changes to the specified recipe to the specified layer and removes it from your workspace. Roughly equivalent to an update-recipe followed by reset, except the update-recipe step will do the "right thing" depending on the recipe and the destination layer specified.',
+ description='Pushes any committed changes to the specified recipe to the specified layer and removes it from your workspace. Roughly equivalent to an update-recipe followed by reset, except the update-recipe step will do the "right thing" depending on the recipe and the destination layer specified. Note that your changes must have been committed to the git repository in order to be recognised.',
group='working', order=-100)
parser_finish.add_argument('recipename', help='Recipe to finish')
parser_finish.add_argument('destination', help='Layer/path to put recipe into. Can be the name of a layer configured in your bblayers.conf, the path to the base of a layer, or a partial path inside a layer. %(prog)s will attempt to complete the path based on the layer\'s structure.')
parser_finish.add_argument('--mode', '-m', choices=['patch', 'srcrev', 'auto'], default='auto', help='Update mode (where %(metavar)s is %(choices)s; default is %(default)s)', metavar='MODE')
parser_finish.add_argument('--initial-rev', help='Override starting revision for patches')
+ parser_finish.add_argument('--force', '-f', action="store_true", help='Force continuing even if there are uncommitted changes in the source tree repository')
parser_finish.set_defaults(func=finish)
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 13/24] devtool: finish: fix "layer not in bblayers.conf" warning when path specified
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (11 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 12/24] devtool: finish: ensure repository is clean before proceeding Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 14/24] devtool: upgrade: handle recipes that use named SRC_URI checksums Paul Eggleton
` (10 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
devtool finish will check if the destination layer is part of
bblayers.conf so that we avoid the user getting confused about the
recipe vanishing from their configuration if it isn't. devtool finish
also accepts a path underneath a layer so that you have a bit
more control over where it ends up. However if you used a path
underneath a layer then it wasn't converting this to the base of the
layer before checking it against BBLAYERS, thus the warning was being
shown erroneously in that case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 7fee304..166862f 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1714,6 +1714,8 @@ def finish(args, config, basepath, workspace):
elif line.startswith('# original_files:'):
origfilelist = line.split(':')[1].split()
+ destlayerbasedir = oe.recipeutils.find_layerdir(destlayerdir)
+
if origlayerdir == config.workspace_path:
# Recipe file itself is in workspace, update it there first
appendlayerdir = None
@@ -1727,7 +1729,7 @@ def finish(args, config, basepath, workspace):
raise DevtoolError("Unable to determine destination layer path - check that %s specifies an actual layer and %s/conf/layer.conf specifies BBFILES. You may also need to specify a more complete path." % (args.destination, destlayerdir))
# Warn if the layer isn't in bblayers.conf (the code to create a bbappend will do this in other cases)
layerdirs = [os.path.abspath(layerdir) for layerdir in rd.getVar('BBLAYERS').split()]
- if not os.path.abspath(destlayerdir) in layerdirs:
+ if not os.path.abspath(destlayerbasedir) in layerdirs:
bb.warn('Specified destination layer is not currently enabled in bblayers.conf, so the %s recipe will now be unavailable in your current configuration until you add the layer there' % args.recipename)
elif destlayerdir == origlayerdir:
@@ -1740,7 +1742,7 @@ def finish(args, config, basepath, workspace):
destpath = None
# Remove any old files in the case of an upgrade
- if origpath and origfilelist and oe.recipeutils.find_layerdir(origpath) == oe.recipeutils.find_layerdir(destlayerdir):
+ if origpath and origfilelist and oe.recipeutils.find_layerdir(origpath) == destlayerbasedir:
for fn in origfilelist:
fnp = os.path.join(origpath, fn)
try:
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 14/24] devtool: upgrade: handle recipes that use named SRC_URI checksums
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (12 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 13/24] devtool: finish: fix "layer not in bblayers.conf" warning when path specified Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 15/24] recipetool: create: drop debug print Paul Eggleton
` (9 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
devtool upgrade did not properly handle setting SRC_URI checksums for
recipes that use named SRC_URI entries and also use those names in the
SRC_URI checksums. A further complication was where the name contained
an expression that changed with the version e.g. ${PV} (probably quite
rare, but the dnsmasq recipe in meta-networking is currently one such
recipe.) All of these are now handled properly.
Additionally, drop the _get_checksums() function that wasn't being
called from anywhere in the code.
Note that this now turns nowrap_vars in recipeutils.py to be a list of
regexes, hence things such as [ and ] need to be appropriately escaped.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/recipeutils.py | 13 +++++++++--
scripts/lib/devtool/upgrade.py | 53 +++++++++++++++++++++++++++++++-----------
2 files changed, 51 insertions(+), 15 deletions(-)
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index cab8e40..cab94b1 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -22,7 +22,7 @@ from collections import OrderedDict, defaultdict
# Help us to find places to insert values
recipe_progression = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION', 'LICENSE', 'LICENSE_FLAGS', 'LIC_FILES_CHKSUM', 'PROVIDES', 'DEPENDS', 'PR', 'PV', 'SRCREV', 'SRCPV', 'SRC_URI', 'S', 'do_fetch()', 'do_unpack()', 'do_patch()', 'EXTRA_OECONF', 'EXTRA_OECMAKE', 'EXTRA_OESCONS', 'do_configure()', 'EXTRA_OEMAKE', 'do_compile()', 'do_install()', 'do_populate_sysroot()', 'INITSCRIPT', 'USERADD', 'GROUPADD', 'PACKAGES', 'FILES', 'RDEPENDS', 'RRECOMMENDS', 'RSUGGESTS', 'RPROVIDES', 'RREPLACES', 'RCONFLICTS', 'ALLOW_EMPTY', 'populate_packages()', 'do_package()', 'do_deploy()']
# Variables that sometimes are a bit long but shouldn't be wrapped
-nowrap_vars = ['SUMMARY', 'HOMEPAGE', 'BUGTRACKER', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]']
+nowrap_vars = ['SUMMARY', 'HOMEPAGE', 'BUGTRACKER', 'SRC_URI\[(.+\.)?md5sum\]', 'SRC_URI\[(.+\.)?sha256sum\]']
list_vars = ['SRC_URI', 'LIC_FILES_CHKSUM']
meta_vars = ['SUMMARY', 'DESCRIPTION', 'HOMEPAGE', 'BUGTRACKER', 'SECTION']
@@ -142,6 +142,10 @@ def patch_recipe_lines(fromlines, values, trailing_newline=True):
else:
newline = ''
+ nowrap_vars_res = []
+ for item in nowrap_vars:
+ nowrap_vars_res.append(re.compile('^%s$' % item))
+
recipe_progression_res = []
recipe_progression_restrs = []
for item in recipe_progression:
@@ -174,7 +178,12 @@ def patch_recipe_lines(fromlines, values, trailing_newline=True):
return
rawtext = '%s = "%s"%s' % (name, values[name], newline)
addlines = []
- if name in nowrap_vars:
+ nowrap = False
+ for nowrap_re in nowrap_vars_res:
+ if nowrap_re.match(name):
+ nowrap = True
+ break
+ if nowrap:
addlines.append(rawtext)
elif name in list_vars:
splitvalue = split_var_value(values[name], assignment=False)
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index ab7acd1..6d51958 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -55,17 +55,6 @@ def _copy_source_code(orig, dest):
dest_path = os.path.join(dest, path)
shutil.move(os.path.join(orig, path), dest_path)
-def _get_checksums(rf):
- import re
- checksums = {}
- with open(rf) as f:
- for line in f:
- for cs in ['md5sum', 'sha256sum']:
- m = re.match("^SRC_URI\[%s\].*=.*\"(.*)\"" % cs, line)
- if m:
- checksums[cs] = m.group(1)
- return checksums
-
def _remove_patch_dirs(recipefolder):
for root, dirs, files in os.walk(recipefolder):
for d in dirs:
@@ -353,9 +342,47 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil
newvalues['PR'] = None
+ # Work out which SRC_URI entries have changed in case the entry uses a name
+ crd = rd.createCopy()
+ crd.setVar('PV', newpv)
+ for var, value in newvalues.items():
+ crd.setVar(var, value)
+ old_src_uri = (rd.getVar('SRC_URI') or '').split()
+ new_src_uri = (crd.getVar('SRC_URI') or '').split()
+ newnames = []
+ addnames = []
+ for newentry in new_src_uri:
+ _, _, _, _, _, params = bb.fetch2.decodeurl(newentry)
+ if 'name' in params:
+ newnames.append(params['name'])
+ if newentry not in old_src_uri:
+ addnames.append(params['name'])
+ # Find what's been set in the original recipe
+ oldnames = []
+ noname = False
+ for varflag in rd.getVarFlags('SRC_URI'):
+ if varflag.endswith(('.md5sum', '.sha256sum')):
+ name = varflag.rsplit('.', 1)[0]
+ if name not in oldnames:
+ oldnames.append(name)
+ elif varflag in ['md5sum', 'sha256sum']:
+ noname = True
+ # Even if SRC_URI has named entries it doesn't have to actually use the name
+ if noname and addnames and addnames[0] not in oldnames:
+ addnames = []
+ # Drop any old names (the name actually might include ${PV})
+ for name in oldnames:
+ if name not in newnames:
+ newvalues['SRC_URI[%s.md5sum]' % name] = None
+ newvalues['SRC_URI[%s.sha256sum]' % name] = None
+
if md5 and sha256:
- newvalues['SRC_URI[md5sum]'] = md5
- newvalues['SRC_URI[sha256sum]'] = sha256
+ if addnames:
+ nameprefix = '%s.' % addnames[0]
+ else:
+ nameprefix = ''
+ newvalues['SRC_URI[%smd5sum]' % nameprefix] = md5
+ newvalues['SRC_URI[%ssha256sum]' % nameprefix] = sha256
rd = tinfoil.parse_recipe_file(fullpath, False)
oe.recipeutils.patch_recipe(rd, fullpath, newvalues)
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 15/24] recipetool: create: drop debug print
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (13 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 14/24] devtool: upgrade: handle recipes that use named SRC_URI checksums Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 16/24] devtool: stop always moving workspace to end of BBLAYERS Paul Eggleton
` (8 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
This looks like some debug printing that was left in by accident.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_npm.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index ae53972..fb57e70 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -73,7 +73,6 @@ class NpmRecipeHandler(RecipeHandler):
license = license.replace(' ', '_')
if not license[0] == '(':
license = '(' + license + ')'
- print('LICENSE: {}'.format(license))
else:
license = license.replace('AND', '&')
if license[0] == '(':
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 16/24] devtool: stop always moving workspace to end of BBLAYERS
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (14 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 15/24] recipetool: create: drop debug print Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 17/24] recipetool: create: show a warning for github archive URLs Paul Eggleton
` (7 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
I noticed that using bitbake-layers add-layer followed by a devtool
command resulted in bitbake re-parsing all of the recipes, which is
annoying. Upon closer inspection I could see that devtool was moving the
workspace layer path to the end of BBLAYERS if it happened to be
somewhere in the middle - there's no need for it to be doing this. This
occurred because we were passing the current workspace path to remove
and the "new" path to add even if the path is not being changed, and I
think earlier versions of bb.utils.edit_bblayers_conf() didn't move the
existing entry under these circumstances as it clearly does now. Fix it
so we only pass the path to be removed if we're actually changing the
path.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/devtool | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/scripts/devtool b/scripts/devtool
index 87bb5c8..a651d8f 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -189,7 +189,11 @@ def _enable_workspace_layer(workspacedir, config, basepath):
if not os.path.exists(bblayers_conf):
logger.error('Unable to find bblayers.conf')
return
- _, added = bb.utils.edit_bblayers_conf(bblayers_conf, workspacedir, config.workspace_path)
+ if os.path.abspath(workspacedir) != os.path.abspath(config.workspace_path):
+ removedir = config.workspace_path
+ else:
+ removedir = None
+ _, added = bb.utils.edit_bblayers_conf(bblayers_conf, workspacedir, removedir)
if added:
logger.info('Enabling workspace layer in bblayers.conf')
if config.workspace_path != workspacedir:
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 17/24] recipetool: create: show a warning for github archive URLs
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (15 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 16/24] devtool: stop always moving workspace to end of BBLAYERS Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 18/24] devtool: upgrade: show messages before source extraction steps Paul Eggleton
` (6 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
github archive URLs are not guaranteed to be stable [1] and thus we
should show a warning if a user specifies one to recipetool create (or
devtool add).
[1] http://lists.openembedded.org/pipermail/openembedded-core/2017-September/142519.html
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 055bdef..54e7e85 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -433,6 +433,9 @@ def create_recipe(args):
source = 'file://%s' % os.path.abspath(source)
if scriptutils.is_src_url(source):
+ # Warn about github archive URLs
+ if re.match('https?://github.com/[^/]+/[^/]+/archive/.+(\.tar\..*|\.zip)$', source):
+ logger.warn('github archive files are not guaranteed to be stable and may be re-generated over time. If the latter occurs, the checksums will likely change and the recipe will fail at do_fetch. It is recommended that you point to an actual commit or tag in the repository instead (using the repository URL in conjunction with the -S/--srcrev option).')
# Fetch a URL
fetchuri = reformat_git_uri(urldefrag(source)[0])
if args.binary:
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 18/24] devtool: upgrade: show messages before source extraction steps
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (16 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 17/24] recipetool: create: show a warning for github archive URLs Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 19/24] devtool: upgrade: automatically handle changes to source subdirectory Paul Eggleton
` (5 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
Give the user a little more insight into what's being done.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/upgrade.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 6d51958..e7d47b9 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -457,7 +457,9 @@ def upgrade(args, config, basepath, workspace):
rf = None
try:
+ logger.info('Extracting current version source...')
rev1 = standard._extract_source(srctree, False, 'devtool-orig', False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
+ logger.info('Extracting upgraded version source...')
rev2, md5, sha256, srcbranch = _extract_new_source(args.version, srctree, args.no_patch,
args.srcrev, args.srcbranch, args.branch, args.keep_temp,
tinfoil, rd)
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 19/24] devtool: upgrade: automatically handle changes to source subdirectory
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (17 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 18/24] devtool: upgrade: show messages before source extraction steps Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 20/24] devtool: upgrade: reformat --no-patch warning message Paul Eggleton
` (4 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
If the directory where the source code extracts to changes (for
example, when upgrading iucode-tool from 1.5 to 2.1.1, the subdirectory
in the tarball changed from "iucode_tool-${PV}" to "iucode-tool-${PV}")
then handle this automatically. Also handle when it changes to match the
default S value (i.e. "${WORKDIR}/${BP}") in which case we just drop
setting S in the recipe.
Fixes [YOCTO #10939].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 9 +++++----
scripts/lib/devtool/upgrade.py | 28 +++++++++++++++++++++++-----
2 files changed, 28 insertions(+), 9 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 166862f..5ac678b 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -409,7 +409,7 @@ def extract(args, config, basepath, workspace):
return 1
srctree = os.path.abspath(args.srctree)
- initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
+ initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
logger.info('Source tree extracted to %s' % srctree)
if initial_rev:
@@ -433,7 +433,7 @@ def sync(args, config, basepath, workspace):
return 1
srctree = os.path.abspath(args.srctree)
- initial_rev = _extract_source(srctree, args.keep_temp, args.branch, True, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
+ initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, True, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
logger.info('Source tree %s synchronized' % srctree)
if initial_rev:
@@ -549,6 +549,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
with open(os.path.join(tempdir, 'srcsubdir'), 'r') as f:
srcsubdir = f.read()
+ srcsubdir_rel = os.path.relpath(srcsubdir, os.path.join(tempdir, 'workdir'))
tempdir_localdir = os.path.join(tempdir, 'oe-local-files')
srctree_localdir = os.path.join(srctree, 'oe-local-files')
@@ -615,7 +616,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
logger.info('Preserving temporary directory %s' % tempdir)
else:
shutil.rmtree(tempdir)
- return initial_rev
+ return initial_rev, srcsubdir_rel
def _add_md5(config, recipename, filename):
"""Record checksum of a file (or recursively for a directory) to the md5-file of the workspace"""
@@ -713,7 +714,7 @@ def modify(args, config, basepath, workspace):
initial_rev = None
commits = []
if not args.no_extract:
- initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
+ initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
if not initial_rev:
return 1
logger.info('Source tree extracted to %s' % srctree)
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index e7d47b9..0db2a50 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -191,6 +191,8 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee
__run('git tag -f devtool-base-new')
md5 = None
sha256 = None
+ _, _, _, _, _, params = bb.fetch2.decodeurl(uri)
+ srcsubdir_rel = params.get('destsuffix', 'git')
if not srcbranch:
check_branch, check_branch_err = __run('git branch -r --contains %s' % srcrev)
get_branch = [x.strip() for x in check_branch.splitlines()]
@@ -225,6 +227,7 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee
tmpsrctree = _get_srctree(tmpdir)
srctree = os.path.abspath(srctree)
+ srcsubdir_rel = os.path.relpath(tmpsrctree, tmpdir)
# Delete all sources so we ensure no stray files are left over
for item in os.listdir(srctree):
@@ -288,9 +291,9 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee
else:
shutil.rmtree(tmpsrctree)
- return (rev, md5, sha256, srcbranch)
+ return (rev, md5, sha256, srcbranch, srcsubdir_rel)
-def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil, rd):
+def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, srcsubdir_old, srcsubdir_new, workspace, tinfoil, rd):
"""Creates the new recipe under workspace"""
bpn = rd.getVar('BPN')
@@ -384,6 +387,21 @@ def _create_new_recipe(newpv, md5, sha256, srcrev, srcbranch, workspace, tinfoil
newvalues['SRC_URI[%smd5sum]' % nameprefix] = md5
newvalues['SRC_URI[%ssha256sum]' % nameprefix] = sha256
+ if srcsubdir_new != srcsubdir_old:
+ s_subdir_old = os.path.relpath(os.path.abspath(rd.getVar('S')), rd.getVar('WORKDIR'))
+ s_subdir_new = os.path.relpath(os.path.abspath(crd.getVar('S')), crd.getVar('WORKDIR'))
+ if srcsubdir_old == s_subdir_old and srcsubdir_new != s_subdir_new:
+ # Subdir for old extracted source matches what S points to (it should!)
+ # but subdir for new extracted source doesn't match what S will be
+ newvalues['S'] = '${WORKDIR}/%s' % srcsubdir_new.replace(newpv, '${PV}')
+ if crd.expand(newvalues['S']) == crd.expand('${WORKDIR}/${BP}'):
+ # It's the default, drop it
+ # FIXME what if S is being set in a .inc?
+ newvalues['S'] = None
+ logger.info('Source subdirectory has changed, dropping S value since it now matches the default ("${WORKDIR}/${BP}")')
+ else:
+ logger.info('Source subdirectory has changed, updating S value')
+
rd = tinfoil.parse_recipe_file(fullpath, False)
oe.recipeutils.patch_recipe(rd, fullpath, newvalues)
@@ -458,12 +476,12 @@ def upgrade(args, config, basepath, workspace):
rf = None
try:
logger.info('Extracting current version source...')
- rev1 = standard._extract_source(srctree, False, 'devtool-orig', False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
+ rev1, srcsubdir1 = standard._extract_source(srctree, False, 'devtool-orig', False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
logger.info('Extracting upgraded version source...')
- rev2, md5, sha256, srcbranch = _extract_new_source(args.version, srctree, args.no_patch,
+ rev2, md5, sha256, srcbranch, srcsubdir2 = _extract_new_source(args.version, srctree, args.no_patch,
args.srcrev, args.srcbranch, args.branch, args.keep_temp,
tinfoil, rd)
- rf, copied = _create_new_recipe(args.version, md5, sha256, args.srcrev, srcbranch, config.workspace_path, tinfoil, rd)
+ rf, copied = _create_new_recipe(args.version, md5, sha256, args.srcrev, srcbranch, srcsubdir1, srcsubdir2, config.workspace_path, tinfoil, rd)
except bb.process.CmdError as e:
_upgrade_error(e, rf, srctree)
except DevtoolError as e:
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 20/24] devtool: upgrade: reformat --no-patch warning message
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (18 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 19/24] devtool: upgrade: automatically handle changes to source subdirectory Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 21/24] devtool: show a better error message if meta-files aren't found Paul Eggleton
` (3 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
* Only log one warning message instead of one per line
* Be a bit more verbose
* "if list" is more pythonic than "if len(list)"
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/upgrade.py | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 0db2a50..3cb523c 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -263,10 +263,8 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee
if no_patch:
patches = oe.recipeutils.get_recipe_patches(crd)
- if len(patches):
- logger.warn('By user choice, the following patches will NOT be applied')
- for patch in patches:
- logger.warn("%s" % os.path.basename(patch))
+ if patches:
+ logger.warn('By user choice, the following patches will NOT be applied to the new source tree:\n %s' % '\n '.join([os.path.basename(patch) for patch in patches]))
else:
__run('git checkout devtool-patched -b %s' % branch)
skiptag = False
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 21/24] devtool: show a better error message if meta-files aren't found
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (19 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 20/24] devtool: upgrade: reformat --no-patch warning message Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 22/24] devtool: finish: improve reporting for removed files Paul Eggleton
` (2 subsequent siblings)
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
If the files that the devtool-source class is supposed to create in the
source tree aren't found in the temporary directory then we know that
the class hasn't worked properly - say that explicitly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 5ac678b..1e7d707 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -544,11 +544,14 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
if not res:
raise DevtoolError('Extracting source for %s failed' % pn)
- with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:
- initial_rev = f.read()
+ try:
+ with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:
+ initial_rev = f.read()
- with open(os.path.join(tempdir, 'srcsubdir'), 'r') as f:
- srcsubdir = f.read()
+ with open(os.path.join(tempdir, 'srcsubdir'), 'r') as f:
+ srcsubdir = f.read()
+ except FileNotFoundError as e:
+ raise DevtoolError('Something went wrong with source extraction - the devtool-source class was not active or did not function correctly:\n%s' % str(e))
srcsubdir_rel = os.path.relpath(srcsubdir, os.path.join(tempdir, 'workdir'))
tempdir_localdir = os.path.join(tempdir, 'oe-local-files')
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 22/24] devtool: finish: improve reporting for removed files
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (20 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 21/24] devtool: show a better error message if meta-files aren't found Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 23/24] devtool: finish: add dry-run option Paul Eggleton
2017-11-09 1:55 ` [PATCH 24/24] devtool: implement conditional patch handling Paul Eggleton
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
If a file is going to be effectively removed from the destination by
devtool finish, we should report that rather than just reporting that
we're removing files from the workspace. This is a little tricky because
the way we actually operate when finishing is to:
(1) remove all original files (as recorded by devtool upgrade, if that
was used)
(2) as part of updating the recipe file, remove the files from next to
the new recipe (i.e. in the workspace for an upgrade, real recipe
otherwise) corresponding to commits not in the git tree
(3) copy over remaining files from the workspace to the destination
To report the files removed with respect to what was originally there,
we need to swap steps 1 and 2 so we can see what no longer exists after
the deletion, and suppress the reporting currently done in step 2 -
however, we still want to report removal in step 2 for the non-upgrade
case, so the latter is conditional.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 32 ++++++++++++++++++--------------
1 file changed, 18 insertions(+), 14 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 1e7d707..9d9031c 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1054,7 +1054,7 @@ def _replace_srcuri_entry(srcuri, filename, newentry):
srcuri.insert(i, newentry)
break
-def _remove_source_files(append, files, destpath):
+def _remove_source_files(append, files, destpath, no_report_remove=False):
"""Unlink existing patch files"""
for path in files:
if append:
@@ -1063,7 +1063,8 @@ def _remove_source_files(append, files, destpath):
path = os.path.join(destpath, os.path.basename(path))
if os.path.exists(path):
- logger.info('Removing file %s' % path)
+ if not no_report_remove:
+ logger.info('Removing file %s' % path)
# FIXME "git rm" here would be nice if the file in question is
# tracked
# FIXME there's a chance that this file is referred to by
@@ -1276,7 +1277,7 @@ def _determine_files_dir(rd):
return os.path.join(recipedir, rd.getVar('BPN'))
-def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove):
+def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove):
"""Implement the 'srcrev' mode of update-recipe"""
import bb
import oe.recipeutils
@@ -1357,10 +1358,10 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi
'point to a git repository where you have pushed your '
'changes')
- _remove_source_files(appendlayerdir, remove_files, destpath)
+ _remove_source_files(appendlayerdir, remove_files, destpath, no_report_remove)
return True
-def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, initial_rev):
+def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev):
"""Implement the 'patch' mode of update-recipe"""
import bb
import oe.recipeutils
@@ -1476,7 +1477,7 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
finally:
shutil.rmtree(tempdir)
- _remove_source_files(appendlayerdir, remove_files, destpath)
+ _remove_source_files(appendlayerdir, remove_files, destpath, no_report_remove)
return True
def _guess_recipe_update_mode(srctree, rdata):
@@ -1501,15 +1502,15 @@ def _guess_recipe_update_mode(srctree, rdata):
return 'patch'
-def _update_recipe(recipename, workspace, rd, mode, appendlayerdir, wildcard_version, no_remove, initial_rev):
+def _update_recipe(recipename, workspace, rd, mode, appendlayerdir, wildcard_version, no_remove, initial_rev, no_report_remove=False):
srctree = workspace[recipename]['srctree']
if mode == 'auto':
mode = _guess_recipe_update_mode(srctree, rd)
if mode == 'srcrev':
- updated = _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove)
+ updated = _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove)
elif mode == 'patch':
- updated = _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, initial_rev)
+ updated = _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev)
else:
raise DevtoolError('update_recipe: invalid mode %s' % mode)
return updated
@@ -1745,24 +1746,27 @@ def finish(args, config, basepath, workspace):
appendlayerdir = destlayerdir
destpath = None
+ # Actually update the recipe / bbappend
+ removing_original = (origpath and origfilelist and oe.recipeutils.find_layerdir(origpath) == destlayerbasedir)
+ _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, initial_rev=args.initial_rev, no_report_remove=removing_original)
+
# Remove any old files in the case of an upgrade
- if origpath and origfilelist and oe.recipeutils.find_layerdir(origpath) == destlayerbasedir:
+ recipedir = os.path.dirname(rd.getVar('FILE'))
+ if removing_original:
for fn in origfilelist:
fnp = os.path.join(origpath, fn)
+ if not os.path.exists(os.path.join(recipedir, fn)):
+ logger.info('Removing file %s' % fnp)
try:
os.remove(fnp)
except FileNotFoundError:
pass
- # Actually update the recipe / bbappend
- _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, initial_rev=args.initial_rev)
-
if origlayerdir == config.workspace_path and destpath:
# Recipe file itself is in the workspace - need to move it and any
# associated files to the specified layer
no_clean = True
logger.info('Moving recipe file to %s' % destpath)
- recipedir = os.path.dirname(rd.getVar('FILE'))
for root, _, files in os.walk(recipedir):
for fn in files:
srcpath = os.path.join(root, fn)
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 23/24] devtool: finish: add dry-run option
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (21 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 22/24] devtool: finish: improve reporting for removed files Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
2017-11-09 1:55 ` [PATCH 24/24] devtool: implement conditional patch handling Paul Eggleton
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
If you're not sure what changes devtool finish is going to make, or
you're not sure you're finished with your modifications, it is useful to
be able to see what devtool finish is going to do beforehand, so add
a -N/--dry-run option to make that possible.
(It's also very useful for debugging devtool finish itself.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/recipeutils.py | 48 +++++--
scripts/lib/devtool/standard.py | 270 +++++++++++++++++++++++++++++-----------
2 files changed, 235 insertions(+), 83 deletions(-)
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index cab94b1..a1e191a 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -251,7 +251,7 @@ def patch_recipe_lines(fromlines, values, trailing_newline=True):
return changed, tolines
-def patch_recipe_file(fn, values, patch=False, relpath=''):
+def patch_recipe_file(fn, values, patch=False, relpath='', redirect_output=None):
"""Update or insert variable values into a recipe file (assuming you
have already identified the exact file you want to update.)
Note that some manual inspection/intervention may be required
@@ -263,7 +263,11 @@ def patch_recipe_file(fn, values, patch=False, relpath=''):
_, tolines = patch_recipe_lines(fromlines, values)
- if patch:
+ if redirect_output:
+ with open(os.path.join(redirect_output, os.path.basename(fn)), 'w') as f:
+ f.writelines(tolines)
+ return None
+ elif patch:
relfn = os.path.relpath(fn, relpath)
diff = difflib.unified_diff(fromlines, tolines, 'a/%s' % relfn, 'b/%s' % relfn)
return diff
@@ -313,7 +317,7 @@ def localise_file_vars(fn, varfiles, varlist):
return filevars
-def patch_recipe(d, fn, varvalues, patch=False, relpath=''):
+def patch_recipe(d, fn, varvalues, patch=False, relpath='', redirect_output=None):
"""Modify a list of variable values in the specified recipe. Handles inc files if
used by the recipe.
"""
@@ -323,7 +327,7 @@ def patch_recipe(d, fn, varvalues, patch=False, relpath=''):
patches = []
for f,v in locs.items():
vals = {k: varvalues[k] for k in v}
- patchdata = patch_recipe_file(f, vals, patch, relpath)
+ patchdata = patch_recipe_file(f, vals, patch, relpath, redirect_output)
if patch:
patches.append(patchdata)
@@ -584,7 +588,7 @@ def get_bbappend_path(d, destlayerdir, wildcardver=False):
return (appendpath, pathok)
-def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, machine=None, extralines=None, removevalues=None):
+def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, machine=None, extralines=None, removevalues=None, redirect_output=None):
"""
Writes a bbappend file for a recipe
Parameters:
@@ -611,6 +615,9 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
value pairs, or simply a list of the lines.
removevalues:
Variable values to remove - a dict of names/values.
+ redirect_output:
+ If specified, redirects writing the output file to the
+ specified directory (for dry-run purposes)
"""
if not removevalues:
@@ -625,7 +632,8 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
bb.warn('Unable to determine correct subdirectory path for bbappend file - check that what %s adds to BBFILES also matches .bbappend files. Using %s for now, but until you fix this the bbappend will not be applied.' % (os.path.join(destlayerdir, 'conf', 'layer.conf'), os.path.dirname(appendpath)))
appenddir = os.path.dirname(appendpath)
- bb.utils.mkdirhier(appenddir)
+ if not redirect_output:
+ bb.utils.mkdirhier(appenddir)
# FIXME check if the bbappend doesn't get overridden by a higher priority layer?
@@ -702,9 +710,18 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
if instfunclines:
bbappendlines.append(('do_install_append%s()' % appendoverride, '', instfunclines))
- bb.note('Writing append file %s' % appendpath)
+ if redirect_output:
+ bb.note('Writing append file %s (dry-run)' % appendpath)
+ outfile = os.path.join(redirect_output, os.path.basename(appendpath))
+ # Only take a copy if the file isn't already there (this function may be called
+ # multiple times per operation when we're handling overrides)
+ if os.path.exists(appendpath) and not os.path.exists(outfile):
+ shutil.copy2(appendpath, outfile)
+ else:
+ bb.note('Writing append file %s' % appendpath)
+ outfile = appendpath
- if os.path.exists(appendpath):
+ if os.path.exists(outfile):
# Work around lack of nonlocal in python 2
extvars = {'destsubdir': destsubdir}
@@ -776,7 +793,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
if removevalues:
varnames.extend(list(removevalues.keys()))
- with open(appendpath, 'r') as f:
+ with open(outfile, 'r') as f:
(updated, newlines) = bb.utils.edit_metadata(f, varnames, appendfile_varfunc)
destsubdir = extvars['destsubdir']
@@ -793,20 +810,27 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
updated = True
if updated:
- with open(appendpath, 'w') as f:
+ with open(outfile, 'w') as f:
f.writelines(newlines)
if copyfiles:
if machine:
destsubdir = os.path.join(destsubdir, machine)
+ if redirect_output:
+ outdir = redirect_output
+ else:
+ outdir = appenddir
for newfile, srcfile in copyfiles.items():
- filedest = os.path.join(appenddir, destsubdir, os.path.basename(srcfile))
+ filedest = os.path.join(outdir, destsubdir, os.path.basename(srcfile))
if os.path.abspath(newfile) != os.path.abspath(filedest):
if newfile.startswith(tempfile.gettempdir()):
newfiledisp = os.path.basename(newfile)
else:
newfiledisp = newfile
- bb.note('Copying %s to %s' % (newfiledisp, filedest))
+ if redirect_output:
+ bb.note('Copying %s to %s (dry-run)' % (newfiledisp, os.path.join(appenddir, destsubdir, os.path.basename(srcfile))))
+ else:
+ bb.note('Copying %s to %s' % (newfiledisp, filedest))
bb.utils.mkdirhier(os.path.dirname(filedest))
shutil.copyfile(newfile, filedest)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 9d9031c..d55d504 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -341,19 +341,45 @@ def _check_compatible_recipe(pn, d):
"from working. You will need to disable this "
"first." % pn)
-def _move_file(src, dst):
- """Move a file. Creates all the directory components of destination path."""
+def _dry_run_copy(src, dst, dry_run_outdir, base_outdir):
+ """Common function for copying a file to the dry run output directory"""
+ relpath = os.path.relpath(dst, base_outdir)
+ if relpath.startswith('..'):
+ raise Exception('Incorrect base path %s for path %s' % (base_outdir, dst))
+ dst = os.path.join(dry_run_outdir, relpath)
dst_d = os.path.dirname(dst)
if dst_d:
bb.utils.mkdirhier(dst_d)
- shutil.move(src, dst)
+ # Don't overwrite existing files, otherwise in the case of an upgrade
+ # the dry-run written out recipe will be overwritten with an unmodified
+ # version
+ if not os.path.exists(dst):
+ shutil.copy(src, dst)
+
+def _move_file(src, dst, dry_run_outdir=None, base_outdir=None):
+ """Move a file. Creates all the directory components of destination path."""
+ dry_run_suffix = ' (dry-run)' if dry_run_outdir else ''
+ logger.debug('Moving %s to %s%s' % (src, dst, dry_run_suffix))
+ if dry_run_outdir:
+ # We want to copy here, not move
+ _dry_run_copy(src, dst, dry_run_outdir, base_outdir)
+ else:
+ dst_d = os.path.dirname(dst)
+ if dst_d:
+ bb.utils.mkdirhier(dst_d)
+ shutil.move(src, dst)
-def _copy_file(src, dst):
+def _copy_file(src, dst, dry_run_outdir=None):
"""Copy a file. Creates all the directory components of destination path."""
- dst_d = os.path.dirname(dst)
- if dst_d:
- bb.utils.mkdirhier(dst_d)
- shutil.copy(src, dst)
+ dry_run_suffix = ' (dry-run)' if dry_run_outdir else ''
+ logger.debug('Copying %s to %s%s' % (src, dst, dry_run_suffix))
+ if dry_run_outdir:
+ _dry_run_copy(src, dst, dry_run_outdir, base_outdir)
+ else:
+ dst_d = os.path.dirname(dst)
+ if dst_d:
+ bb.utils.mkdirhier(dst_d)
+ shutil.copy(src, dst)
def _git_ls_tree(repodir, treeish='HEAD', recursive=False):
"""List contents of a git treeish"""
@@ -1054,8 +1080,11 @@ def _replace_srcuri_entry(srcuri, filename, newentry):
srcuri.insert(i, newentry)
break
-def _remove_source_files(append, files, destpath, no_report_remove=False):
+def _remove_source_files(append, files, destpath, no_report_remove=False, dry_run=False):
"""Unlink existing patch files"""
+
+ dry_run_suffix = ' (dry-run)' if dry_run else ''
+
for path in files:
if append:
if not destpath:
@@ -1064,19 +1093,20 @@ def _remove_source_files(append, files, destpath, no_report_remove=False):
if os.path.exists(path):
if not no_report_remove:
- logger.info('Removing file %s' % path)
- # FIXME "git rm" here would be nice if the file in question is
- # tracked
- # FIXME there's a chance that this file is referred to by
- # another recipe, in which case deleting wouldn't be the
- # right thing to do
- os.remove(path)
- # Remove directory if empty
- try:
- os.rmdir(os.path.dirname(path))
- except OSError as ose:
- if ose.errno != errno.ENOTEMPTY:
- raise
+ logger.info('Removing file %s%s' % (path, dry_run_suffix))
+ if not dry_run:
+ # FIXME "git rm" here would be nice if the file in question is
+ # tracked
+ # FIXME there's a chance that this file is referred to by
+ # another recipe, in which case deleting wouldn't be the
+ # right thing to do
+ os.remove(path)
+ # Remove directory if empty
+ try:
+ os.rmdir(os.path.dirname(path))
+ except OSError as ose:
+ if ose.errno != errno.ENOTEMPTY:
+ raise
def _export_patches(srctree, rd, start_rev, destdir, changed_revs=None):
@@ -1277,13 +1307,16 @@ def _determine_files_dir(rd):
return os.path.join(recipedir, rd.getVar('BPN'))
-def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove):
+def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove, dry_run_outdir=None):
"""Implement the 'srcrev' mode of update-recipe"""
import bb
import oe.recipeutils
+ dry_run_suffix = ' (dry-run)' if dry_run_outdir else ''
+
recipefile = rd.getVar('FILE')
- logger.info('Updating SRCREV in recipe %s' % os.path.basename(recipefile))
+ recipedir = os.path.basename(recipefile)
+ logger.info('Updating SRCREV in recipe %s%s' % (recipedir, dry_run_suffix))
# Get HEAD revision
try:
@@ -1303,6 +1336,7 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi
srcuri = orig_src_uri.split()
tempdir = tempfile.mkdtemp(prefix='devtool')
update_srcuri = False
+ appendfile = None
try:
local_files_dir = tempfile.mkdtemp(dir=tempdir)
srctreebase = workspace[recipename]['srctreebase']
@@ -1328,29 +1362,36 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi
if update_srcuri:
removevalues = {'SRC_URI': removedentries}
patchfields['SRC_URI'] = '\\\n '.join(srcuri)
- _, destpath = oe.recipeutils.bbappend_recipe(
- rd, appendlayerdir, files, wildcardver=wildcard_version,
- extralines=patchfields, removevalues=removevalues)
+ if dry_run_outdir:
+ logger.info('Creating bbappend (dry-run)')
+ else:
+ appendfile, destpath = oe.recipeutils.bbappend_recipe(
+ rd, appendlayerdir, files, wildcardver=wildcard_version,
+ extralines=patchfields, removevalues=removevalues,
+ redirect_output=dry_run_outdir)
else:
files_dir = _determine_files_dir(rd)
for basepath, path in upd_f.items():
- logger.info('Updating file %s' % basepath)
+ logger.info('Updating file %s%s' % (basepath, dry_run_suffix))
if os.path.isabs(basepath):
# Original file (probably with subdir pointing inside source tree)
# so we do not want to move it, just copy
- _copy_file(basepath, path)
+ _copy_file(basepath, path, dry_run_outdir=dry_run_outdir, base_outdir=recipedir)
else:
- _move_file(os.path.join(local_files_dir, basepath), path)
+ _move_file(os.path.join(local_files_dir, basepath), path,
+ dry_run_outdir=dry_run_outdir, base_outdir=recipedir)
update_srcuri= True
for basepath, path in new_f.items():
- logger.info('Adding new file %s' % basepath)
+ logger.info('Adding new file %s%s' % (basepath, dry_run_suffix))
_move_file(os.path.join(local_files_dir, basepath),
- os.path.join(files_dir, basepath))
+ os.path.join(files_dir, basepath),
+ dry_run_outdir=dry_run_outdir,
+ base_outdir=recipedir)
srcuri.append('file://%s' % basepath)
update_srcuri = True
if update_srcuri:
patchfields['SRC_URI'] = ' '.join(srcuri)
- oe.recipeutils.patch_recipe(rd, recipefile, patchfields)
+ ret = oe.recipeutils.patch_recipe(rd, recipefile, patchfields, redirect_output=dry_run_outdir)
finally:
shutil.rmtree(tempdir)
if not 'git://' in orig_src_uri:
@@ -1358,15 +1399,16 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi
'point to a git repository where you have pushed your '
'changes')
- _remove_source_files(appendlayerdir, remove_files, destpath, no_report_remove)
- return True
+ _remove_source_files(appendlayerdir, remove_files, destpath, no_report_remove, dry_run=dry_run_outdir)
+ return True, appendfile, remove_files
-def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev):
+def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev, dry_run_outdir=None):
"""Implement the 'patch' mode of update-recipe"""
import bb
import oe.recipeutils
recipefile = rd.getVar('FILE')
+ recipedir = os.path.dirname(recipefile)
append = workspace[recipename]['bbappend']
if not os.path.exists(append):
raise DevtoolError('unable to find workspace bbappend for recipe %s' %
@@ -1377,10 +1419,13 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
raise DevtoolError('Unable to find initial revision - please specify '
'it with --initial-rev')
+ appendfile = None
dl_dir = rd.getVar('DL_DIR')
if not dl_dir.endswith('/'):
dl_dir += '/'
+ dry_run_suffix = ' (dry-run)' if dry_run_outdir else ''
+
tempdir = tempfile.mkdtemp(prefix='devtool')
try:
local_files_dir = tempfile.mkdtemp(dir=tempdir)
@@ -1418,10 +1463,11 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
remaining = ['file://' + os.path.basename(item) for
item in remaining]
removevalues = {'SRC_URI': removedentries + remaining}
- _, destpath = oe.recipeutils.bbappend_recipe(
+ appendfile, destpath = oe.recipeutils.bbappend_recipe(
rd, appendlayerdir, files,
wildcardver=wildcard_version,
- removevalues=removevalues)
+ removevalues=removevalues,
+ redirect_output=dry_run_outdir)
else:
logger.info('No patches or local source files needed updating')
else:
@@ -1432,9 +1478,11 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
if os.path.isabs(basepath):
# Original file (probably with subdir pointing inside source tree)
# so we do not want to move it, just copy
- _copy_file(basepath, path)
+ _copy_file(basepath, path,
+ dry_run_outdir=dry_run_outdir, base_outdir=recipedir)
else:
- _move_file(os.path.join(local_files_dir, basepath), path)
+ _move_file(os.path.join(local_files_dir, basepath), path,
+ dry_run_outdir=dry_run_outdir, base_outdir=recipedir)
updatefiles = True
for basepath, path in upd_p.items():
patchfn = os.path.join(patches_dir, basepath)
@@ -1446,39 +1494,45 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
_replace_srcuri_entry(srcuri, basepath, 'file://%s' % basepath)
updaterecipe = True
else:
- logger.info('Updating patch %s' % basepath)
- logger.debug('Moving new patch %s to %s' % (patchfn, path))
- _move_file(patchfn, path)
+ logger.info('Updating patch %s%s' % (basepath, dry_run_suffix))
+ _move_file(patchfn, path,
+ dry_run_outdir=dry_run_outdir, base_outdir=recipedir)
updatefiles = True
# Add any new files
for basepath, path in new_f.items():
- logger.info('Adding new file %s' % basepath)
+ logger.info('Adding new file %s%s' % (basepath, dry_run_suffix))
_move_file(os.path.join(local_files_dir, basepath),
- os.path.join(files_dir, basepath))
+ os.path.join(files_dir, basepath),
+ dry_run_outdir=dry_run_outdir,
+ base_outdir=recipedir)
srcuri.append('file://%s' % basepath)
updaterecipe = True
for basepath, path in new_p.items():
- logger.info('Adding new patch %s' % basepath)
+ logger.info('Adding new patch %s%s' % (basepath, dry_run_suffix))
_move_file(os.path.join(patches_dir, basepath),
- os.path.join(files_dir, basepath))
+ os.path.join(files_dir, basepath),
+ dry_run_outdir=dry_run_outdir,
+ base_outdir=recipedir)
srcuri.append('file://%s' % basepath)
updaterecipe = True
# Update recipe, if needed
if _remove_file_entries(srcuri, remove_files)[0]:
updaterecipe = True
if updaterecipe:
- logger.info('Updating recipe %s' % os.path.basename(recipefile))
- oe.recipeutils.patch_recipe(rd, recipefile,
- {'SRC_URI': ' '.join(srcuri)})
+ if not dry_run_outdir:
+ logger.info('Updating recipe %s' % os.path.basename(recipefile))
+ ret = oe.recipeutils.patch_recipe(rd, recipefile,
+ {'SRC_URI': ' '.join(srcuri)},
+ redirect_output=dry_run_outdir)
elif not updatefiles:
# Neither patches nor recipe were updated
logger.info('No patches or files need updating')
- return False
+ return False, None, []
finally:
shutil.rmtree(tempdir)
- _remove_source_files(appendlayerdir, remove_files, destpath, no_report_remove)
- return True
+ _remove_source_files(appendlayerdir, remove_files, destpath, no_report_remove, dry_run=dry_run_outdir)
+ return True, appendfile, remove_files
def _guess_recipe_update_mode(srctree, rdata):
"""Guess the recipe update mode to use"""
@@ -1502,18 +1556,18 @@ def _guess_recipe_update_mode(srctree, rdata):
return 'patch'
-def _update_recipe(recipename, workspace, rd, mode, appendlayerdir, wildcard_version, no_remove, initial_rev, no_report_remove=False):
+def _update_recipe(recipename, workspace, rd, mode, appendlayerdir, wildcard_version, no_remove, initial_rev, no_report_remove=False, dry_run_outdir=None):
srctree = workspace[recipename]['srctree']
if mode == 'auto':
mode = _guess_recipe_update_mode(srctree, rd)
if mode == 'srcrev':
- updated = _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove)
+ updated, appendfile, removed = _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove, dry_run_outdir)
elif mode == 'patch':
- updated = _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev)
+ updated, appendfile, removed = _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev, dry_run_outdir)
else:
raise DevtoolError('update_recipe: invalid mode %s' % mode)
- return updated
+ return updated, appendfile, removed
def update_recipe(args, config, basepath, workspace):
"""Entry point for the devtool 'update-recipe' subcommand"""
@@ -1534,7 +1588,12 @@ def update_recipe(args, config, basepath, workspace):
if not rd:
return 1
- updated = _update_recipe(args.recipename, workspace, rd, args.mode, args.append, args.wildcard_version, args.no_remove, args.initial_rev)
+ dry_run_output = None
+ dry_run_outdir = None
+ if args.dry_run:
+ dry_run_output = tempfile.TemporaryDirectory(prefix='devtool')
+ dry_run_outdir = dry_run_output.name
+ updated, _, _ = _update_recipe(args.recipename, workspace, rd, args.mode, args.append, args.wildcard_version, args.no_remove, args.initial_rev, dry_run_outdir=dry_run_outdir)
if updated:
rf = rd.getVar('FILE')
@@ -1680,6 +1739,8 @@ def finish(args, config, basepath, workspace):
check_workspace_recipe(workspace, args.recipename)
+ dry_run_suffix = ' (dry-run)' if args.dry_run else ''
+
# Grab the equivalent of COREBASE without having to initialise tinfoil
corebasedir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', '..'))
@@ -1700,7 +1761,9 @@ def finish(args, config, basepath, workspace):
return 1
destlayerdir = _get_layer(args.destination, tinfoil.config_data)
- origlayerdir = oe.recipeutils.find_layerdir(rd.getVar('FILE'))
+ recipefile = rd.getVar('FILE')
+ recipedir = os.path.dirname(recipefile)
+ origlayerdir = oe.recipeutils.find_layerdir(recipefile)
if not os.path.isdir(destlayerdir):
raise DevtoolError('Unable to find layer or directory matching "%s"' % args.destination)
@@ -1748,38 +1811,101 @@ def finish(args, config, basepath, workspace):
# Actually update the recipe / bbappend
removing_original = (origpath and origfilelist and oe.recipeutils.find_layerdir(origpath) == destlayerbasedir)
- _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, initial_rev=args.initial_rev, no_report_remove=removing_original)
+ dry_run_output = None
+ dry_run_outdir = None
+ if args.dry_run:
+ dry_run_output = tempfile.TemporaryDirectory(prefix='devtool')
+ dry_run_outdir = dry_run_output.name
+ updated, appendfile, removed = _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, no_report_remove=removing_original, initial_rev=args.initial_rev, dry_run_outdir=dry_run_outdir)
+ removed = [os.path.relpath(pth, recipedir) for pth in removed]
# Remove any old files in the case of an upgrade
- recipedir = os.path.dirname(rd.getVar('FILE'))
if removing_original:
for fn in origfilelist:
fnp = os.path.join(origpath, fn)
- if not os.path.exists(os.path.join(recipedir, fn)):
- logger.info('Removing file %s' % fnp)
- try:
- os.remove(fnp)
- except FileNotFoundError:
- pass
+ if fn in removed or not os.path.exists(os.path.join(recipedir, fn)):
+ logger.info('Removing file %s%s' % (fnp, dry_run_suffix))
+ if not args.dry_run:
+ try:
+ os.remove(fnp)
+ except FileNotFoundError:
+ pass
if origlayerdir == config.workspace_path and destpath:
# Recipe file itself is in the workspace - need to move it and any
# associated files to the specified layer
no_clean = True
- logger.info('Moving recipe file to %s' % destpath)
+ logger.info('Moving recipe file to %s%s' % (destpath, dry_run_suffix))
for root, _, files in os.walk(recipedir):
for fn in files:
srcpath = os.path.join(root, fn)
relpth = os.path.relpath(os.path.dirname(srcpath), recipedir)
destdir = os.path.abspath(os.path.join(destpath, relpth))
- bb.utils.mkdirhier(destdir)
- shutil.move(srcpath, os.path.join(destdir, fn))
+ destfp = os.path.join(destdir, fn)
+ _move_file(srcpath, destfp, dry_run_outdir=dry_run_outdir, base_outdir=destpath)
+ if dry_run_outdir:
+ import difflib
+ comparelist = []
+ for root, _, files in os.walk(dry_run_outdir):
+ for fn in files:
+ outf = os.path.join(root, fn)
+ relf = os.path.relpath(outf, dry_run_outdir)
+ logger.debug('dry-run: output file %s' % relf)
+ if fn.endswith('.bb'):
+ if origfilelist and origpath and destpath:
+ # Need to match this up with the pre-upgrade recipe file
+ for origf in origfilelist:
+ if origf.endswith('.bb'):
+ comparelist.append((os.path.abspath(os.path.join(origpath, origf)),
+ outf,
+ os.path.abspath(os.path.join(destpath, relf))))
+ break
+ else:
+ # Compare to the existing recipe
+ comparelist.append((recipefile, outf, recipefile))
+ elif fn.endswith('.bbappend'):
+ if appendfile:
+ if os.path.exists(appendfile):
+ comparelist.append((appendfile, outf, appendfile))
+ else:
+ comparelist.append((None, outf, appendfile))
+ else:
+ if destpath:
+ recipedest = destpath
+ elif appendfile:
+ recipedest = os.path.dirname(appendfile)
+ else:
+ recipedest = os.path.dirname(recipefile)
+ destfp = os.path.join(recipedest, relf)
+ if os.path.exists(destfp):
+ comparelist.append((destfp, outf, destfp))
+ output = ''
+ for oldfile, newfile, newfileshow in comparelist:
+ if oldfile:
+ with open(oldfile, 'r') as f:
+ oldlines = f.readlines()
+ else:
+ oldfile = '/dev/null'
+ oldlines = []
+ with open(newfile, 'r') as f:
+ newlines = f.readlines()
+ if not newfileshow:
+ newfileshow = newfile
+ diff = difflib.unified_diff(oldlines, newlines, oldfile, newfileshow)
+ difflines = list(diff)
+ if difflines:
+ output += ''.join(difflines)
+ if output:
+ logger.info('Diff of changed files:\n%s' % output)
finally:
tinfoil.shutdown()
# Everything else has succeeded, we can now reset
- _reset([args.recipename], no_clean=no_clean, config=config, basepath=basepath, workspace=workspace)
+ if args.dry_run:
+ logger.info('Resetting recipe (dry-run)')
+ else:
+ _reset([args.recipename], no_clean=no_clean, config=config, basepath=basepath, workspace=workspace)
return 0
@@ -1873,6 +1999,7 @@ def register_commands(subparsers, context):
parser_update_recipe.add_argument('--append', '-a', help='Write changes to a bbappend in the specified layer instead of the recipe', metavar='LAYERDIR')
parser_update_recipe.add_argument('--wildcard-version', '-w', help='In conjunction with -a/--append, use a wildcard to make the bbappend apply to any recipe version', action='store_true')
parser_update_recipe.add_argument('--no-remove', '-n', action="store_true", help='Don\'t remove patches, only add or update')
+ parser_update_recipe.add_argument('--dry-run', '-N', action="store_true", help='Dry-run (just report changes instead of writing them)')
parser_update_recipe.set_defaults(func=update_recipe)
parser_status = subparsers.add_parser('status', help='Show workspace status',
@@ -1896,4 +2023,5 @@ def register_commands(subparsers, context):
parser_finish.add_argument('--mode', '-m', choices=['patch', 'srcrev', 'auto'], default='auto', help='Update mode (where %(metavar)s is %(choices)s; default is %(default)s)', metavar='MODE')
parser_finish.add_argument('--initial-rev', help='Override starting revision for patches')
parser_finish.add_argument('--force', '-f', action="store_true", help='Force continuing even if there are uncommitted changes in the source tree repository')
+ parser_finish.add_argument('--dry-run', '-N', action="store_true", help='Dry-run (just report changes instead of writing them)')
parser_finish.set_defaults(func=finish)
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread* [PATCH 24/24] devtool: implement conditional patch handling
2017-11-09 1:55 [PATCH 00/24] devtool / recipetool improvements Paul Eggleton
` (22 preceding siblings ...)
2017-11-09 1:55 ` [PATCH 23/24] devtool: finish: add dry-run option Paul Eggleton
@ 2017-11-09 1:55 ` Paul Eggleton
23 siblings, 0 replies; 25+ messages in thread
From: Paul Eggleton @ 2017-11-09 1:55 UTC (permalink / raw)
To: openembedded-core
If you have a recipe that uses overrides to conditionally extend
SRC_URI to add additional patches, then you will often need to update
those patches if you're making other changes to the source tree (for
example if you're upgrading the underlying source). Make this possible
with devtool by creating devtool-override-* branches for each override
that conditionally appends/prepends SRC_URI, and have devtool
update-recipe / finish check each branch out in turn and update the
corresponding patches.
A current example of a recipe that does this is the quota recipe - it
applies an additional patch if musl is the selected C library (i.e.
libc-musl is in OVERRIDES).
Note that use of this functionality does require some care - in
particular, updates to patches that appear on the main branch (named
"devtool" by default) should be made there and not only on one of the
specific devtool-override-* branches that are created for each override.
The recommended procedure is to make the changes you want to make to the
main branch first, then check out and rebase each devtool-override-*
branch, testing each one by activating the corresponding configuration,
and then finally run devtool finish.
Fixes [YOCTO #11516].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/devtool-source.bbclass | 56 ++++++++++++
scripts/lib/devtool/standard.py | 176 +++++++++++++++++++++++++++++++-----
scripts/lib/devtool/upgrade.py | 3 +-
3 files changed, 209 insertions(+), 26 deletions(-)
diff --git a/meta/classes/devtool-source.bbclass b/meta/classes/devtool-source.bbclass
index 8f5bc86..56882a4 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -152,9 +152,65 @@ python devtool_pre_patch() {
}
python devtool_post_patch() {
+ import shutil
tempdir = d.getVar('DEVTOOL_TEMPDIR')
with open(os.path.join(tempdir, 'srcsubdir'), 'r') as f:
srcsubdir = f.read()
+ with open(os.path.join(tempdir, 'initial_rev'), 'r') as f:
+ initial_rev = f.read()
+
+ def rm_patches():
+ patches_dir = os.path.join(srcsubdir, 'patches')
+ if os.path.exists(patches_dir):
+ shutil.rmtree(patches_dir)
+ # Restore any "patches" directory that was actually part of the source tree
+ try:
+ bb.process.run('git checkout -- patches', cwd=srcsubdir)
+ except bb.process.ExecutionError:
+ pass
+
+ extra_overrides = d.getVar('DEVTOOL_EXTRA_OVERRIDES')
+ if extra_overrides:
+ extra_override_list = extra_overrides.split(':')
+ devbranch = d.getVar('DEVTOOL_DEVBRANCH')
+ default_overrides = d.getVar('OVERRIDES').split(':')
+ no_overrides = []
+ # First, we may have some overrides that are referred to in the recipe set in
+ # our configuration, so we need to make a branch that excludes those
+ for override in default_overrides:
+ if override not in extra_override_list:
+ no_overrides.append(override)
+ if default_overrides != no_overrides:
+ # Some overrides are active in the current configuration, so
+ # we need to create a branch where none of the overrides are active
+ bb.process.run('git checkout %s -b devtool-no-overrides' % initial_rev, cwd=srcsubdir)
+ # Run do_patch function with the override applied
+ localdata = bb.data.createCopy(d)
+ localdata.setVar('OVERRIDES', ':'.join(no_overrides))
+ bb.build.exec_func('do_patch', localdata)
+ rm_patches()
+ # Now we need to reconcile the dev branch with the no-overrides one
+ # (otherwise we'd likely be left with identical commits that have different hashes)
+ bb.process.run('git checkout %s' % devbranch, cwd=srcsubdir)
+ bb.process.run('git rebase devtool-no-overrides', cwd=srcsubdir)
+ else:
+ bb.process.run('git checkout %s -b devtool-no-overrides' % devbranch, cwd=srcsubdir)
+
+ for override in extra_override_list:
+ localdata = bb.data.createCopy(d)
+ if override in default_overrides:
+ bb.process.run('git branch devtool-override-%s %s' % (override, devbranch), cwd=srcsubdir)
+ else:
+ # Reset back to the initial commit on a new branch
+ bb.process.run('git checkout %s -b devtool-override-%s' % (initial_rev, override), cwd=srcsubdir)
+ # Run do_patch function with the override applied
+ localdata.appendVar('OVERRIDES', ':%s' % override)
+ bb.build.exec_func('do_patch', localdata)
+ rm_patches()
+ # Now we need to reconcile the new branch with the no-overrides one
+ # (otherwise we'd likely be left with identical commits that have different hashes)
+ bb.process.run('git rebase devtool-no-overrides', cwd=srcsubdir)
+ bb.process.run('git checkout %s' % devbranch, cwd=srcsubdir)
bb.process.run('git tag -f devtool-patched', cwd=srcsubdir)
}
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index d55d504..26187a0 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -35,6 +35,8 @@ from devtool import parse_recipe
logger = logging.getLogger('devtool')
+override_branch_prefix = 'devtool-override-'
+
def add(args, config, basepath, workspace):
"""Entry point for the devtool 'add' subcommand"""
@@ -425,7 +427,7 @@ def extract(args, config, basepath, workspace):
"""Entry point for the devtool 'extract' subcommand"""
import bb
- tinfoil = setup_tinfoil(basepath=basepath)
+ tinfoil = setup_tinfoil(basepath=basepath, tracking=True)
if not tinfoil:
# Error already shown
return 1
@@ -435,7 +437,7 @@ def extract(args, config, basepath, workspace):
return 1
srctree = os.path.abspath(args.srctree)
- initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
+ initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
logger.info('Source tree extracted to %s' % srctree)
if initial_rev:
@@ -449,7 +451,7 @@ def sync(args, config, basepath, workspace):
"""Entry point for the devtool 'sync' subcommand"""
import bb
- tinfoil = setup_tinfoil(basepath=basepath)
+ tinfoil = setup_tinfoil(basepath=basepath, tracking=True)
if not tinfoil:
# Error already shown
return 1
@@ -459,7 +461,7 @@ def sync(args, config, basepath, workspace):
return 1
srctree = os.path.abspath(args.srctree)
- initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, True, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
+ initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, True, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=True)
logger.info('Source tree %s synchronized' % srctree)
if initial_rev:
@@ -470,7 +472,7 @@ def sync(args, config, basepath, workspace):
tinfoil.shutdown()
-def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, workspace, fixed_setup, d, tinfoil):
+def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, workspace, fixed_setup, d, tinfoil, no_overrides=False):
"""Extract sources of a recipe"""
import oe.recipeutils
import oe.patch
@@ -500,6 +502,16 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
bb.utils.mkdirhier(srctree)
os.rmdir(srctree)
+ extra_overrides = []
+ if not no_overrides:
+ history = d.varhistory.variable('SRC_URI')
+ for event in history:
+ if not 'flag' in event:
+ if event['op'].startswith(('_append[', '_prepend[')):
+ extra_overrides.append(event['op'].split('[')[1].split(']')[0])
+ if extra_overrides:
+ logger.info('SRC_URI contains some conditional appends/prepends - will create branches to represent these')
+
initial_rev = None
appendexisted = False
@@ -542,6 +554,8 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
if not is_kernel_yocto:
f.write('PATCHTOOL = "git"\n')
f.write('PATCH_COMMIT_FUNCTIONS = "1"\n')
+ if extra_overrides:
+ f.write('DEVTOOL_EXTRA_OVERRIDES = "%s"\n' % ':'.join(extra_overrides))
f.write('inherit devtool-source\n')
f.write('###--- _extract_source\n')
@@ -702,12 +716,13 @@ def modify(args, config, basepath, workspace):
"""Entry point for the devtool 'modify' subcommand"""
import bb
import oe.recipeutils
+ import oe.patch
if args.recipename in workspace:
raise DevtoolError("recipe %s is already in your workspace" %
args.recipename)
- tinfoil = setup_tinfoil(basepath=basepath)
+ tinfoil = setup_tinfoil(basepath=basepath, tracking=True)
try:
rd = parse_recipe(config, tinfoil, args.recipename, True)
if not rd:
@@ -742,14 +757,16 @@ def modify(args, config, basepath, workspace):
initial_rev = None
commits = []
+ check_commits = False
if not args.no_extract:
- initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
+ initial_rev, _ = _extract_source(srctree, args.keep_temp, args.branch, False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
if not initial_rev:
return 1
logger.info('Source tree extracted to %s' % srctree)
# Get list of commits since this revision
(stdout, _) = bb.process.run('git rev-list --reverse %s..HEAD' % initial_rev, cwd=srctree)
commits = stdout.split()
+ check_commits = True
else:
if os.path.exists(os.path.join(srctree, '.git')):
# Check if it's a tree previously extracted by us
@@ -757,6 +774,8 @@ def modify(args, config, basepath, workspace):
(stdout, _) = bb.process.run('git branch --contains devtool-base', cwd=srctree)
except bb.process.ExecutionError:
stdout = ''
+ if stdout:
+ check_commits = True
for line in stdout.splitlines():
if line.startswith('*'):
(stdout, _) = bb.process.run('git rev-parse devtool-base', cwd=srctree)
@@ -766,6 +785,30 @@ def modify(args, config, basepath, workspace):
(stdout, _) = bb.process.run('git rev-parse HEAD', cwd=srctree)
initial_rev = stdout.rstrip()
+ branch_patches = {}
+ if check_commits:
+ # Check if there are override branches
+ (stdout, _) = bb.process.run('git branch', cwd=srctree)
+ branches = []
+ for line in stdout.rstrip().splitlines():
+ branchname = line[2:].rstrip()
+ if branchname.startswith(override_branch_prefix):
+ branches.append(branchname)
+ if branches:
+ logger.warn('SRC_URI is conditionally overridden in this recipe, thus several %s* branches have been created, one for each override that makes changes to SRC_URI. It is recommended that you make changes to the %s branch first, then checkout and rebase each %s* branch and update any unique patches there (duplicates on those branches will be ignored by devtool finish/update-recipe)' % (override_branch_prefix, args.branch, override_branch_prefix))
+ branches.insert(0, args.branch)
+ seen_patches = []
+ for branch in branches:
+ branch_patches[branch] = []
+ (stdout, _) = bb.process.run('git log devtool-base..%s' % branch, cwd=srctree)
+ for line in stdout.splitlines():
+ line = line.strip()
+ if line.startswith(oe.patch.GitApplyTree.patch_line_prefix):
+ origpatch = line[len(oe.patch.GitApplyTree.patch_line_prefix):].split(':', 1)[-1].strip()
+ if not origpatch in seen_patches:
+ seen_patches.append(origpatch)
+ branch_patches[branch].append(origpatch)
+
# Need to grab this here in case the source is within a subdirectory
srctreebase = srctree
@@ -808,6 +851,11 @@ def modify(args, config, basepath, workspace):
f.write('\n# initial_rev: %s\n' % initial_rev)
for commit in commits:
f.write('# commit: %s\n' % commit)
+ if branch_patches:
+ for branch in branch_patches:
+ if branch == args.branch:
+ continue
+ f.write('# patches_%s: %s\n' % (branch, ','.join(branch_patches[branch])))
update_unlockedsigs(basepath, workspace, args.fixed_setup, [pn])
@@ -1019,8 +1067,14 @@ def _get_patchset_revs(srctree, recipe_path, initial_rev=None):
"""
import bb
+ # Get current branch
+ stdout, _ = bb.process.run('git rev-parse --abbrev-ref HEAD',
+ cwd=srctree)
+ branchname = stdout.rstrip()
+
# Parse initial rev from recipe if not specified
commits = []
+ patches = []
with open(recipe_path, 'r') as f:
for line in f:
if line.startswith('# initial_rev:'):
@@ -1028,6 +1082,8 @@ def _get_patchset_revs(srctree, recipe_path, initial_rev=None):
initial_rev = line.split(':')[-1].strip()
elif line.startswith('# commit:'):
commits.append(line.split(':')[-1].strip())
+ elif line.startswith('# patches_%s:' % branchname):
+ patches = line.split(':')[-1].strip().split(',')
update_rev = initial_rev
changed_revs = None
@@ -1054,7 +1110,7 @@ def _get_patchset_revs(srctree, recipe_path, initial_rev=None):
if rev in newcommits:
changed_revs.append(rev)
- return initial_rev, update_rev, changed_revs
+ return initial_rev, update_rev, changed_revs, patches
def _remove_file_entries(srcuri, filelist):
"""Remove file:// entries from SRC_URI"""
@@ -1126,6 +1182,7 @@ def _export_patches(srctree, rd, start_rev, destdir, changed_revs=None):
existing_patches = dict((os.path.basename(path), path) for path in
oe.recipeutils.get_recipe_patches(rd))
+ logger.debug('Existing patches: %s' % existing_patches)
# Generate patches from Git, exclude local files directory
patch_pathspec = _git_exclude_path(srctree, 'oe-local-files')
@@ -1414,7 +1471,7 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
raise DevtoolError('unable to find workspace bbappend for recipe %s' %
recipename)
- initial_rev, update_rev, changed_revs = _get_patchset_revs(srctree, append, initial_rev)
+ initial_rev, update_rev, changed_revs, filter_patches = _get_patchset_revs(srctree, append, initial_rev)
if not initial_rev:
raise DevtoolError('Unable to find initial revision - please specify '
'it with --initial-rev')
@@ -1429,22 +1486,32 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
tempdir = tempfile.mkdtemp(prefix='devtool')
try:
local_files_dir = tempfile.mkdtemp(dir=tempdir)
- srctreebase = workspace[recipename]['srctreebase']
- upd_f, new_f, del_f = _export_local_files(srctree, rd, local_files_dir, srctreebase)
+ if filter_patches:
+ upd_f = {}
+ new_f = {}
+ del_f = {}
+ else:
+ srctreebase = workspace[recipename]['srctreebase']
+ upd_f, new_f, del_f = _export_local_files(srctree, rd, local_files_dir, srctreebase)
remove_files = []
if not no_remove:
# Get all patches from source tree and check if any should be removed
all_patches_dir = tempfile.mkdtemp(dir=tempdir)
- upd_p, new_p, del_p = _export_patches(srctree, rd, initial_rev,
- all_patches_dir)
+ _, _, del_p = _export_patches(srctree, rd, initial_rev,
+ all_patches_dir)
# Remove deleted local files and patches
remove_files = list(del_f.values()) + list(del_p.values())
# Get updated patches from source tree
patches_dir = tempfile.mkdtemp(dir=tempdir)
- upd_p, new_p, del_p = _export_patches(srctree, rd, update_rev,
- patches_dir, changed_revs)
+ upd_p, new_p, _ = _export_patches(srctree, rd, update_rev,
+ patches_dir, changed_revs)
+ logger.debug('Pre-filtering: update: %s, new: %s' % (dict(upd_p), dict(new_p)))
+ if filter_patches:
+ new_p = {}
+ upd_p = {k:v for k,v in upd_p.items() if k in filter_patches}
+ remove_files = [f for f in remove_files if f in filter_patches]
updatefiles = False
updaterecipe = False
destpath = None
@@ -1556,18 +1623,73 @@ def _guess_recipe_update_mode(srctree, rdata):
return 'patch'
-def _update_recipe(recipename, workspace, rd, mode, appendlayerdir, wildcard_version, no_remove, initial_rev, no_report_remove=False, dry_run_outdir=None):
+def _update_recipe(recipename, workspace, rd, mode, appendlayerdir, wildcard_version, no_remove, initial_rev, no_report_remove=False, dry_run_outdir=None, no_overrides=False):
srctree = workspace[recipename]['srctree']
if mode == 'auto':
mode = _guess_recipe_update_mode(srctree, rd)
- if mode == 'srcrev':
- updated, appendfile, removed = _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove, dry_run_outdir)
- elif mode == 'patch':
- updated, appendfile, removed = _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev, dry_run_outdir)
- else:
- raise DevtoolError('update_recipe: invalid mode %s' % mode)
- return updated, appendfile, removed
+ override_branches = []
+ mainbranch = None
+ startbranch = None
+ if not no_overrides:
+ stdout, _ = bb.process.run('git branch', cwd=srctree)
+ other_branches = []
+ for line in stdout.splitlines():
+ branchname = line[2:]
+ if line.startswith('* '):
+ startbranch = branchname
+ if branchname.startswith(override_branch_prefix):
+ override_branches.append(branchname)
+ else:
+ other_branches.append(branchname)
+
+ if override_branches:
+ logger.debug('_update_recipe: override branches: %s' % override_branches)
+ logger.debug('_update_recipe: other branches: %s' % other_branches)
+ if startbranch.startswith(override_branch_prefix):
+ if len(other_branches) == 1:
+ mainbranch = other_branches[1]
+ else:
+ raise DevtoolError('Unable to determine main branch - please check out the main branch in source tree first')
+ else:
+ mainbranch = startbranch
+
+ checkedout = None
+ anyupdated = False
+ appendfile = None
+ allremoved = []
+ if override_branches:
+ logger.info('Handling main branch (%s)...' % mainbranch)
+ if startbranch != mainbranch:
+ bb.process.run('git checkout %s' % mainbranch, cwd=srctree)
+ checkedout = mainbranch
+ try:
+ branchlist = [mainbranch] + override_branches
+ for branch in branchlist:
+ crd = bb.data.createCopy(rd)
+ if branch != mainbranch:
+ logger.info('Handling branch %s...' % branch)
+ override = branch[len(override_branch_prefix):]
+ crd.appendVar('OVERRIDES', ':%s' % override)
+ bb.process.run('git checkout %s' % branch, cwd=srctree)
+ checkedout = branch
+
+ if mode == 'srcrev':
+ updated, appendf, removed = _update_recipe_srcrev(recipename, workspace, srctree, crd, appendlayerdir, wildcard_version, no_remove, no_report_remove, dry_run_outdir)
+ elif mode == 'patch':
+ updated, appendf, removed = _update_recipe_patch(recipename, workspace, srctree, crd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev, dry_run_outdir)
+ else:
+ raise DevtoolError('update_recipe: invalid mode %s' % mode)
+ if updated:
+ anyupdated = True
+ if appendf:
+ appendfile = appendf
+ allremoved.extend(removed)
+ finally:
+ if startbranch and checkedout != startbranch:
+ bb.process.run('git checkout %s' % startbranch, cwd=srctree)
+
+ return anyupdated, appendfile, allremoved
def update_recipe(args, config, basepath, workspace):
"""Entry point for the devtool 'update-recipe' subcommand"""
@@ -1593,7 +1715,7 @@ def update_recipe(args, config, basepath, workspace):
if args.dry_run:
dry_run_output = tempfile.TemporaryDirectory(prefix='devtool')
dry_run_outdir = dry_run_output.name
- updated, _, _ = _update_recipe(args.recipename, workspace, rd, args.mode, args.append, args.wildcard_version, args.no_remove, args.initial_rev, dry_run_outdir=dry_run_outdir)
+ updated, _, _ = _update_recipe(args.recipename, workspace, rd, args.mode, args.append, args.wildcard_version, args.no_remove, args.initial_rev, dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides)
if updated:
rf = rd.getVar('FILE')
@@ -1816,7 +1938,7 @@ def finish(args, config, basepath, workspace):
if args.dry_run:
dry_run_output = tempfile.TemporaryDirectory(prefix='devtool')
dry_run_outdir = dry_run_output.name
- updated, appendfile, removed = _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, no_report_remove=removing_original, initial_rev=args.initial_rev, dry_run_outdir=dry_run_outdir)
+ updated, appendfile, removed = _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, no_report_remove=removing_original, initial_rev=args.initial_rev, dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides)
removed = [os.path.relpath(pth, recipedir) for pth in removed]
# Remove any old files in the case of an upgrade
@@ -1959,6 +2081,7 @@ def register_commands(subparsers, context):
group.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
parser_modify.add_argument('--branch', '-b', default="devtool", help='Name for development branch to checkout (when not using -n/--no-extract) (default "%(default)s")')
+ parser_modify.add_argument('--no-overrides', '-O', action="store_true", help='Do not create branches for other override configurations')
parser_modify.add_argument('--keep-temp', help='Keep temporary directory (for debugging)', action="store_true")
parser_modify.set_defaults(func=modify, fixed_setup=context.fixed_setup)
@@ -1968,6 +2091,7 @@ def register_commands(subparsers, context):
parser_extract.add_argument('recipename', help='Name of recipe to extract the source for')
parser_extract.add_argument('srctree', help='Path to where to extract the source tree')
parser_extract.add_argument('--branch', '-b', default="devtool", help='Name for development branch to checkout (default "%(default)s")')
+ parser_extract.add_argument('--no-overrides', '-O', action="store_true", help='Do not create branches for other override configurations')
parser_extract.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
parser_extract.set_defaults(func=extract, fixed_setup=context.fixed_setup)
@@ -1999,6 +2123,7 @@ def register_commands(subparsers, context):
parser_update_recipe.add_argument('--append', '-a', help='Write changes to a bbappend in the specified layer instead of the recipe', metavar='LAYERDIR')
parser_update_recipe.add_argument('--wildcard-version', '-w', help='In conjunction with -a/--append, use a wildcard to make the bbappend apply to any recipe version', action='store_true')
parser_update_recipe.add_argument('--no-remove', '-n', action="store_true", help='Don\'t remove patches, only add or update')
+ parser_update_recipe.add_argument('--no-overrides', '-O', action="store_true", help='Do not handle other override branches (if they exist)')
parser_update_recipe.add_argument('--dry-run', '-N', action="store_true", help='Dry-run (just report changes instead of writing them)')
parser_update_recipe.set_defaults(func=update_recipe)
@@ -2023,5 +2148,6 @@ def register_commands(subparsers, context):
parser_finish.add_argument('--mode', '-m', choices=['patch', 'srcrev', 'auto'], default='auto', help='Update mode (where %(metavar)s is %(choices)s; default is %(default)s)', metavar='MODE')
parser_finish.add_argument('--initial-rev', help='Override starting revision for patches')
parser_finish.add_argument('--force', '-f', action="store_true", help='Force continuing even if there are uncommitted changes in the source tree repository')
+ parser_finish.add_argument('--no-overrides', '-O', action="store_true", help='Do not handle other override branches (if they exist)')
parser_finish.add_argument('--dry-run', '-N', action="store_true", help='Dry-run (just report changes instead of writing them)')
parser_finish.set_defaults(func=finish)
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 3cb523c..f6141bf 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -474,7 +474,7 @@ def upgrade(args, config, basepath, workspace):
rf = None
try:
logger.info('Extracting current version source...')
- rev1, srcsubdir1 = standard._extract_source(srctree, False, 'devtool-orig', False, config, basepath, workspace, args.fixed_setup, rd, tinfoil)
+ rev1, srcsubdir1 = standard._extract_source(srctree, False, 'devtool-orig', False, config, basepath, workspace, args.fixed_setup, rd, tinfoil, no_overrides=args.no_overrides)
logger.info('Extracting upgraded version source...')
rev2, md5, sha256, srcbranch, srcsubdir2 = _extract_new_source(args.version, srctree, args.no_patch,
args.srcrev, args.srcbranch, args.branch, args.keep_temp,
@@ -513,6 +513,7 @@ def register_commands(subparsers, context):
parser_upgrade.add_argument('--srcbranch', '-B', help='Branch in source repository containing the revision to use (if fetching from an SCM such as git)')
parser_upgrade.add_argument('--branch', '-b', default="devtool", help='Name for new development branch to checkout (default "%(default)s")')
parser_upgrade.add_argument('--no-patch', action="store_true", help='Do not apply patches from the recipe to the new source code')
+ parser_upgrade.add_argument('--no-overrides', '-O', action="store_true", help='Do not create branches for other override configurations')
group = parser_upgrade.add_mutually_exclusive_group()
group.add_argument('--same-dir', '-s', help='Build in same directory as source', action="store_true")
group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
--
2.9.5
^ permalink raw reply related [flat|nested] 25+ messages in thread