* [PATCH v2 1/7] recipetool: create: fix handling of github URLs
2016-07-10 23:07 [PATCH v2 0/7] devtool/recipetool fixes Paul Eggleton
@ 2016-07-10 23:07 ` Paul Eggleton
2016-07-10 23:07 ` [PATCH v2 2/7] recipetool: create: support specifying a file as the local source Paul Eggleton
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-07-10 23:07 UTC (permalink / raw)
To: openembedded-core
For a while now, Github hasn't been advertising a specific repository
URL since cloning the web URL with git works. Armed with this knowledge
and fully expecting people to just paste the github URL, we need to
handle this situation specially. If it looks like a github URL to the
root of a repository then treat it as a git repository instead of a
normal https URL to be fetched by the wget fetcher.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 430f5bb..95b16fb 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -324,7 +324,7 @@ def supports_srcrev(uri):
def reformat_git_uri(uri):
'''Convert any http[s]://....git URI into git://...;protocol=http[s]'''
checkuri = uri.split(';', 1)[0]
- if checkuri.endswith('.git') or '/git/' in checkuri:
+ if checkuri.endswith('.git') or '/git/' in checkuri or re.match('https?://github.com/[^/]+/[^/]+/?', checkuri):
res = re.match('(https?)://([^;]+(\.git)?)(;.*)?$', uri)
if res:
# Need to switch the URI around so that the git fetcher is used
--
2.5.5
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v2 2/7] recipetool: create: support specifying a file as the local source
2016-07-10 23:07 [PATCH v2 0/7] devtool/recipetool fixes Paul Eggleton
2016-07-10 23:07 ` [PATCH v2 1/7] recipetool: create: fix handling of github URLs Paul Eggleton
@ 2016-07-10 23:07 ` Paul Eggleton
2016-07-10 23:07 ` [PATCH v2 3/7] lib/oe/patch: handle encoding differences in patch files Paul Eggleton
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-07-10 23:07 UTC (permalink / raw)
To: openembedded-core
It is currently possible to specify a file (e.g. a tarball) on the local
disk as the source, but you have to know to put file:// in front of it.
There's really no need to force users to jump through that hoop if they
really want to do this so check if the specified source is a file and
prefix it with file:// if that's the case.
Also ensure the same works for "devtool add" at the same time.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 4 ++++
scripts/lib/recipetool/create.py | 15 ++++++++++-----
2 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index ed49a93..737ecc1 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -65,6 +65,10 @@ def add(args, config, basepath, workspace):
elif os.path.isdir(args.recipename):
logger.warn('Ambiguous argument %s - assuming you mean it to be the recipe name')
+ if args.srctree and os.path.isfile(args.srctree):
+ args.fetchuri = 'file://' + os.path.abspath(args.srctree)
+ args.srctree = ''
+
if args.fetch:
if args.fetchuri:
raise DevtoolError('URI specified as positional argument as well as -f/--fetch')
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 95b16fb..f246028 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -351,11 +351,16 @@ def create_recipe(args):
extravalues = {}
checksums = (None, None)
tempsrc = ''
+ source = args.source
srcsubdir = ''
srcrev = '${AUTOREV}'
- if '://' in args.source:
+
+ if os.path.isfile(source):
+ source = 'file://%s' % os.path.abspath(source)
+
+ if '://' in source:
# Fetch a URL
- fetchuri = reformat_git_uri(urldefrag(args.source)[0])
+ fetchuri = reformat_git_uri(urldefrag(source)[0])
if args.binary:
# Assume the archive contains the directory structure verbatim
# so we need to extract to a subdirectory
@@ -426,10 +431,10 @@ def create_recipe(args):
if args.extract_to:
logger.error('--extract-to cannot be specified if source is a directory')
sys.exit(1)
- if not os.path.isdir(args.source):
- logger.error('Invalid source directory %s' % args.source)
+ if not os.path.isdir(source):
+ logger.error('Invalid source directory %s' % source)
sys.exit(1)
- srctree = args.source
+ srctree = source
srcuri = ''
if os.path.exists(os.path.join(srctree, '.git')):
# Try to get upstream repo location from origin remote
--
2.5.5
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v2 3/7] lib/oe/patch: handle encoding differences in patch files
2016-07-10 23:07 [PATCH v2 0/7] devtool/recipetool fixes Paul Eggleton
2016-07-10 23:07 ` [PATCH v2 1/7] recipetool: create: fix handling of github URLs Paul Eggleton
2016-07-10 23:07 ` [PATCH v2 2/7] recipetool: create: support specifying a file as the local source Paul Eggleton
@ 2016-07-10 23:07 ` Paul Eggleton
2016-07-10 23:07 ` [PATCH v2 4/7] devtool: update-recipe: fix --initial-rev option Paul Eggleton
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-07-10 23:07 UTC (permalink / raw)
To: openembedded-core
With Python 3, the encoding of a file is significant; several recipes in
OE-Core have patches which are not fully utf-8 decodable e.g. man,
lrzsz, and gstreamer1.0-libav, leading to errors when using devtool's
modify, upgrade or extract subcommands on these recipes. To work around
this, try reading the patch file as utf-8 first and if that fails try
latin-1 before giving up.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/patch.py | 100 +++++++++++++++++++++++++++++----------------------
1 file changed, 57 insertions(+), 43 deletions(-)
diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index 4a0d3f7..af3adec 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -117,43 +117,50 @@ class PatchSet(object):
return None
return os.sep.join(filesplit[striplevel:])
- copiedmode = False
- filelist = []
- with open(patchfile) as f:
- for line in f:
- if line.startswith('--- '):
- patchpth = patchedpath(line)
- if not patchpth:
- break
- if copiedmode:
- addedfile = patchpth
- else:
- removedfile = patchpth
- elif line.startswith('+++ '):
- addedfile = patchedpath(line)
- if not addedfile:
- break
- elif line.startswith('*** '):
- copiedmode = True
- removedfile = patchedpath(line)
- if not removedfile:
- break
- else:
- removedfile = None
- addedfile = None
-
- if addedfile and removedfile:
- if removedfile == '/dev/null':
- mode = 'A'
- elif addedfile == '/dev/null':
- mode = 'D'
- else:
- mode = 'M'
- if srcdir:
- fullpath = os.path.abspath(os.path.join(srcdir, addedfile))
- else:
- fullpath = addedfile
- filelist.append((fullpath, mode))
+ for encoding in ['utf-8', 'latin-1']:
+ try:
+ copiedmode = False
+ filelist = []
+ with open(patchfile) as f:
+ for line in f:
+ if line.startswith('--- '):
+ patchpth = patchedpath(line)
+ if not patchpth:
+ break
+ if copiedmode:
+ addedfile = patchpth
+ else:
+ removedfile = patchpth
+ elif line.startswith('+++ '):
+ addedfile = patchedpath(line)
+ if not addedfile:
+ break
+ elif line.startswith('*** '):
+ copiedmode = True
+ removedfile = patchedpath(line)
+ if not removedfile:
+ break
+ else:
+ removedfile = None
+ addedfile = None
+
+ if addedfile and removedfile:
+ if removedfile == '/dev/null':
+ mode = 'A'
+ elif addedfile == '/dev/null':
+ mode = 'D'
+ else:
+ mode = 'M'
+ if srcdir:
+ fullpath = os.path.abspath(os.path.join(srcdir, addedfile))
+ else:
+ fullpath = addedfile
+ filelist.append((fullpath, mode))
+ except UnicodeDecodeError:
+ continue
+ break
+ else:
+ raise PatchError('Unable to decode %s' % patchfile)
return filelist
@@ -280,12 +287,19 @@ class GitApplyTree(PatchTree):
"""
Extract just the header lines from the top of a patch file
"""
- lines = []
- with open(patchfile, 'r') as f:
- for line in f.readlines():
- if line.startswith('Index: ') or line.startswith('diff -') or line.startswith('---'):
- break
- lines.append(line)
+ for encoding in ['utf-8', 'latin-1']:
+ lines = []
+ try:
+ with open(patchfile, 'r', encoding=encoding) as f:
+ for line in f:
+ if line.startswith('Index: ') or line.startswith('diff -') or line.startswith('---'):
+ break
+ lines.append(line)
+ except UnicodeDecodeError:
+ continue
+ break
+ else:
+ raise PatchError('Unable to find a character encoding to decode %s' % patchfile)
return lines
@staticmethod
--
2.5.5
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v2 4/7] devtool: update-recipe: fix --initial-rev option
2016-07-10 23:07 [PATCH v2 0/7] devtool/recipetool fixes Paul Eggleton
` (2 preceding siblings ...)
2016-07-10 23:07 ` [PATCH v2 3/7] lib/oe/patch: handle encoding differences in patch files Paul Eggleton
@ 2016-07-10 23:07 ` Paul Eggleton
2016-07-10 23:07 ` [PATCH v2 5/7] devtool: reset: allow reset to work if the recipe file has been deleted Paul Eggleton
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-07-10 23:07 UTC (permalink / raw)
To: openembedded-core
In OE-Core revision 7baf57ad896112cf2258b3e2c2a1f8b756fb39bc I changed
the default update-recipe behaviour to only update patches for commits
that were changed; unfortunately I failed to handle the --initial-rev
option which was broken after that point. Rework how the initial
revision is passed in so that it now operates correctly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 13 +++++--------
1 file changed, 5 insertions(+), 8 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 737ecc1..a7e365f 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -815,22 +815,19 @@ def modify(args, config, basepath, workspace):
return 0
-def _get_patchset_revs(args, srctree, recipe_path):
+def _get_patchset_revs(srctree, recipe_path, initial_rev=None):
"""Get initial and update rev of a recipe. These are the start point of the
whole patchset and start point for the patches to be re-generated/updated.
"""
import bb
- if args.initial_rev:
- return args.initial_rev, args.initial_rev
-
- # Parse initial rev from recipe
+ # Parse initial rev from recipe if not specified
commits = []
- initial_rev = None
with open(recipe_path, 'r') as f:
for line in f:
if line.startswith('# initial_rev:'):
- initial_rev = line.split(':')[-1].strip()
+ if not initial_rev:
+ initial_rev = line.split(':')[-1].strip()
elif line.startswith('# commit:'):
commits.append(line.split(':')[-1].strip())
@@ -1129,7 +1126,7 @@ def _update_recipe_patch(args, config, workspace, srctree, rd, config_data):
raise DevtoolError('unable to find workspace bbappend for recipe %s' %
args.recipename)
- initial_rev, update_rev, changed_revs = _get_patchset_revs(args, srctree, append)
+ initial_rev, update_rev, changed_revs = _get_patchset_revs(srctree, append, args.initial_rev)
if not initial_rev:
raise DevtoolError('Unable to find initial revision - please specify '
'it with --initial-rev')
--
2.5.5
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v2 5/7] devtool: reset: allow reset to work if the recipe file has been deleted
2016-07-10 23:07 [PATCH v2 0/7] devtool/recipetool fixes Paul Eggleton
` (3 preceding siblings ...)
2016-07-10 23:07 ` [PATCH v2 4/7] devtool: update-recipe: fix --initial-rev option Paul Eggleton
@ 2016-07-10 23:07 ` Paul Eggleton
2016-07-10 23:07 ` [PATCH v2 6/7] devtool: return specific exit code for incompatible recipes Paul Eggleton
2016-07-10 23:07 ` [PATCH v2 7/7] scripts/contrib/devtool-stress: skip " Paul Eggleton
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-07-10 23:07 UTC (permalink / raw)
To: openembedded-core
We were attempting to open the recipe file unconditionally here - we
need to account for the possibility that the recipe file has been
deleted or moved away by the user.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index a7e365f..8236dd5 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -1327,7 +1327,7 @@ def reset(args, config, basepath, workspace):
for recipe in recipes:
targets.append(recipe)
recipefile = workspace[recipe]['recipefile']
- if recipefile:
+ if recipefile and os.path.exists(recipefile):
targets.extend(get_bbclassextend_targets(recipefile, recipe))
try:
exec_build_env_command(config.init_path, basepath, 'bitbake -c clean %s' % ' '.join(targets))
--
2.5.5
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v2 6/7] devtool: return specific exit code for incompatible recipes
2016-07-10 23:07 [PATCH v2 0/7] devtool/recipetool fixes Paul Eggleton
` (4 preceding siblings ...)
2016-07-10 23:07 ` [PATCH v2 5/7] devtool: reset: allow reset to work if the recipe file has been deleted Paul Eggleton
@ 2016-07-10 23:07 ` Paul Eggleton
2016-07-10 23:07 ` [PATCH v2 7/7] scripts/contrib/devtool-stress: skip " Paul Eggleton
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-07-10 23:07 UTC (permalink / raw)
To: openembedded-core
Certain recipes cannot be used with devtool extract / modify / upgrade -
usually because they don't provide any source. Return a specific exit
code (4) so that scripts such as scripts/contrib/devtool-stress.py know
the difference between this and a genuine failure.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/devtool | 2 +-
scripts/lib/devtool/__init__.py | 5 +++--
scripts/lib/devtool/standard.py | 16 +++++++++-------
3 files changed, 13 insertions(+), 10 deletions(-)
diff --git a/scripts/devtool b/scripts/devtool
index a93a11f..b1274d6 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -334,7 +334,7 @@ def main():
except DevtoolError as err:
if str(err):
logger.error(str(err))
- ret = 1
+ ret = err.exitcode
except argparse_oe.ArgumentUsageError as ae:
parser.error_subcommand(ae.message, ae.subcommand)
diff --git a/scripts/lib/devtool/__init__.py b/scripts/lib/devtool/__init__.py
index 77b1fd9..65eb452 100644
--- a/scripts/lib/devtool/__init__.py
+++ b/scripts/lib/devtool/__init__.py
@@ -26,10 +26,11 @@ import re
logger = logging.getLogger('devtool')
-
class DevtoolError(Exception):
"""Exception for handling devtool errors"""
- pass
+ def __init__(self, message, exitcode=1):
+ super(DevtoolError, self).__init__(message)
+ self.exitcode = exitcode
def exec_build_env_command(init_path, builddir, cmd, watch=False, **options):
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 8236dd5..f2ba699 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -268,28 +268,30 @@ def _check_compatible_recipe(pn, d):
"""Check if the recipe is supported by devtool"""
if pn == 'perf':
raise DevtoolError("The perf recipe does not actually check out "
- "source and thus cannot be supported by this tool")
+ "source and thus cannot be supported by this tool",
+ 4)
if pn in ['kernel-devsrc', 'package-index'] or pn.startswith('gcc-source'):
- raise DevtoolError("The %s recipe is not supported by this tool" % pn)
+ raise DevtoolError("The %s recipe is not supported by this tool" % pn, 4)
if bb.data.inherits_class('image', d):
raise DevtoolError("The %s recipe is an image, and therefore is not "
- "supported by this tool" % pn)
+ "supported by this tool" % pn, 4)
if bb.data.inherits_class('populate_sdk', d):
raise DevtoolError("The %s recipe is an SDK, and therefore is not "
- "supported by this tool" % pn)
+ "supported by this tool" % pn, 4)
if bb.data.inherits_class('packagegroup', d):
raise DevtoolError("The %s recipe is a packagegroup, and therefore is "
- "not supported by this tool" % pn)
+ "not supported by this tool" % pn, 4)
if bb.data.inherits_class('meta', d):
raise DevtoolError("The %s recipe is a meta-recipe, and therefore is "
- "not supported by this tool" % pn)
+ "not supported by this tool" % pn, 4)
if bb.data.inherits_class('externalsrc', d) and d.getVar('EXTERNALSRC', True):
+ # Not an incompatibility error per se, so we don't pass the error code
raise DevtoolError("externalsrc is currently enabled for the %s "
"recipe. This prevents the normal do_patch task "
"from working. You will need to disable this "
@@ -502,7 +504,7 @@ def _extract_source(srctree, keep_temp, devbranch, sync, d):
if 'noexec' in (d.getVarFlags('do_unpack', False) or []):
raise DevtoolError("The %s recipe has do_unpack disabled, unable to "
- "extract source" % pn)
+ "extract source" % pn, 4)
if not sync:
# Prepare for shutil.move later on
--
2.5.5
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v2 7/7] scripts/contrib/devtool-stress: skip incompatible recipes
2016-07-10 23:07 [PATCH v2 0/7] devtool/recipetool fixes Paul Eggleton
` (5 preceding siblings ...)
2016-07-10 23:07 ` [PATCH v2 6/7] devtool: return specific exit code for incompatible recipes Paul Eggleton
@ 2016-07-10 23:07 ` Paul Eggleton
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-07-10 23:07 UTC (permalink / raw)
To: openembedded-core
If devtool returns exit code 4 then record the recipes as "skipped"
rather than "failed" - these are recipes we know cannot work (usually
because they don't provide any source).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/contrib/devtool-stress.py | 57 ++++++++++++++++++++++++---------------
1 file changed, 35 insertions(+), 22 deletions(-)
diff --git a/scripts/contrib/devtool-stress.py b/scripts/contrib/devtool-stress.py
index ab77a2d..5fff257 100755
--- a/scripts/contrib/devtool-stress.py
+++ b/scripts/contrib/devtool-stress.py
@@ -121,14 +121,18 @@ def stress_extract(args):
sys.stdout.write('Testing %s ' % (pn + ' ').ljust(40, '.'))
sys.stdout.flush()
failed = False
+ skipped = None
srctree = os.path.join(tmpdir, pn)
try:
bb.process.run('devtool extract %s %s' % (pn, srctree))
- except bb.process.CmdError as exc:
- failed = True
- with open('stress_%s_extract.log' % pn, 'w') as f:
- f.write(str(exc))
+ except bb.process.ExecutionError as exc:
+ if exc.exitcode == 4:
+ skipped = 'incompatible'
+ else:
+ failed = True
+ with open('stress_%s_extract.log' % pn, 'w') as f:
+ f.write(str(exc))
if os.path.exists(srctree):
shutil.rmtree(srctree)
@@ -136,6 +140,8 @@ def stress_extract(args):
if failed:
print('failed')
failures += 1
+ elif skipped:
+ print('skipped (%s)' % skipped)
else:
print('ok')
except KeyboardInterrupt:
@@ -162,29 +168,34 @@ def stress_modify(args):
sys.stdout.flush()
failed = False
reset = True
+ skipped = None
srctree = os.path.join(tmpdir, pn)
try:
bb.process.run('devtool modify -x %s %s' % (pn, srctree))
- except bb.process.CmdError as exc:
- with open('stress_%s_modify.log' % pn, 'w') as f:
- f.write(str(exc))
- failed = 'modify'
- reset = False
-
- if not failed:
- try:
- bb.process.run('bitbake -c install %s' % pn)
- except bb.process.CmdError as exc:
- with open('stress_%s_install.log' % pn, 'w') as f:
+ except bb.process.ExecutionError as exc:
+ if exc.exitcode == 4:
+ skipped = 'incompatible'
+ else:
+ with open('stress_%s_modify.log' % pn, 'w') as f:
f.write(str(exc))
- failed = 'build'
- if reset:
- try:
- bb.process.run('devtool reset %s' % pn)
- except bb.process.CmdError as exc:
- print('devtool reset failed: %s' % str(exc))
- break
+ failed = 'modify'
+ reset = False
+
+ if not skipped:
+ if not failed:
+ try:
+ bb.process.run('bitbake -c install %s' % pn)
+ except bb.process.CmdError as exc:
+ with open('stress_%s_install.log' % pn, 'w') as f:
+ f.write(str(exc))
+ failed = 'build'
+ if reset:
+ try:
+ bb.process.run('devtool reset %s' % pn)
+ except bb.process.CmdError as exc:
+ print('devtool reset failed: %s' % str(exc))
+ break
if os.path.exists(srctree):
shutil.rmtree(srctree)
@@ -192,6 +203,8 @@ def stress_modify(args):
if failed:
print('failed (%s)' % failed)
failures += 1
+ elif skipped:
+ print('skipped (%s)' % skipped)
else:
print('ok')
except KeyboardInterrupt:
--
2.5.5
^ permalink raw reply related [flat|nested] 8+ messages in thread