* [PATCH 01/10] recipetool: create: fix picking up name from local python source tree
2016-09-18 20:08 [PATCH 00/10] devtool add / recipetool create fixes Paul Eggleton
@ 2016-09-18 20:08 ` Paul Eggleton
2016-09-18 20:08 ` [PATCH 02/10] recipetool: create: improve python recipe license handling Paul Eggleton
` (8 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Paul Eggleton @ 2016-09-18 20:08 UTC (permalink / raw)
To: openembedded-core
Make use of the extravalues dict to send back other variable values from
the python handling plugin, and enable passing back PV and PN. This not
only places variable values in the final recipe a bit more consistently
with other types of source, it also allows the name and version to be
picked up fron a local source tree and not just when the recipe is
fetched from a remote URL that happens to have those in it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_buildsys_python.py | 36 ++----------------------
1 file changed, 2 insertions(+), 34 deletions(-)
diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index fb9806d..354af0c 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -61,8 +61,6 @@ class PythonRecipeHandler(RecipeHandler):
}
# PN/PV are already set by recipetool core & desc can be extremely long
excluded_fields = [
- 'Name',
- 'Version',
'Description',
]
setup_parse_map = {
@@ -237,7 +235,6 @@ class PythonRecipeHandler(RecipeHandler):
# Map PKG-INFO & setup.py fields to bitbake variables
- bbinfo = {}
for field, values in info.items():
if field in self.excluded_fields:
continue
@@ -251,37 +248,8 @@ class PythonRecipeHandler(RecipeHandler):
value = ' '.join(str(v) for v in values if v)
bbvar = self.bbvar_map[field]
- if bbvar not in bbinfo and value:
- bbinfo[bbvar] = value
-
- comment_lic_line = None
- for pos, line in enumerate(list(lines_before)):
- if line.startswith('#') and 'LICENSE' in line:
- comment_lic_line = pos
- elif line.startswith('LICENSE =') and 'LICENSE' in bbinfo:
- if line in ('LICENSE = "Unknown"', 'LICENSE = "CLOSED"'):
- lines_before[pos] = 'LICENSE = "{}"'.format(bbinfo['LICENSE'])
- if line == 'LICENSE = "CLOSED"' and comment_lic_line:
- lines_before[comment_lic_line:pos] = [
- '# WARNING: the following LICENSE value is a best guess - it is your',
- '# responsibility to verify that the value is complete and correct.'
- ]
- del bbinfo['LICENSE']
-
- src_uri_line = None
- for pos, line in enumerate(lines_before):
- if line.startswith('SRC_URI ='):
- src_uri_line = pos
-
- if bbinfo:
- mdinfo = ['']
- for k in sorted(bbinfo):
- v = bbinfo[k]
- mdinfo.append('{} = "{}"'.format(k, v))
- if src_uri_line:
- lines_before[src_uri_line-1:src_uri_line-1] = mdinfo
- else:
- lines_before.extend(mdinfo)
+ if bbvar not in extravalues and value:
+ extravalues[bbvar] = value
mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals)
--
2.5.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 02/10] recipetool: create: improve python recipe license handling
2016-09-18 20:08 [PATCH 00/10] devtool add / recipetool create fixes Paul Eggleton
2016-09-18 20:08 ` [PATCH 01/10] recipetool: create: fix picking up name from local python source tree Paul Eggleton
@ 2016-09-18 20:08 ` Paul Eggleton
2016-09-18 20:08 ` [PATCH 03/10] recipetool: create: fix name/version extraction from filename Paul Eggleton
` (7 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Paul Eggleton @ 2016-09-18 20:08 UTC (permalink / raw)
To: openembedded-core
Try to ensure that for Apache, GPL and LGPL where the values extracted
from the "Classifiers" field may not be version-specific, if there is a
versioned license in the free-form license field then use that instead.
Also insert the free-form license field as a comment in the recipe for
the user's reference.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_buildsys_python.py | 30 ++++++++++++++++++++++--
1 file changed, 28 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index 354af0c..e41d81a 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -86,8 +86,11 @@ class PythonRecipeHandler(RecipeHandler):
]
setuparg_multi_line_values = ['Description']
replacements = [
+ ('License', r' +$', ''),
+ ('License', r'^ +', ''),
('License', r' ', '-'),
- ('License', r'-License$', ''),
+ ('License', r'^GNU-', ''),
+ ('License', r'-[Ll]icen[cs]e(,?-[Vv]ersion)?', ''),
('License', r'^UNKNOWN$', ''),
# Remove currently unhandled version numbers from these variables
@@ -216,6 +219,9 @@ class PythonRecipeHandler(RecipeHandler):
else:
info = self.get_setup_args_info(setupscript)
+ # Grab the license value before applying replacements
+ license_str = info.get('License', '').strip()
+
self.apply_info_replacements(info)
if uses_setuptools:
@@ -223,17 +229,37 @@ class PythonRecipeHandler(RecipeHandler):
else:
classes.append('distutils')
+ if license_str:
+ for i, line in enumerate(lines_before):
+ if line.startswith('LICENSE = '):
+ lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str)
+ break
+
if 'Classifier' in info:
+ existing_licenses = info.get('License', '')
licenses = []
for classifier in info['Classifier']:
if classifier in self.classifier_license_map:
license = self.classifier_license_map[classifier]
+ if license == 'Apache' and 'Apache-2.0' in existing_licenses:
+ license = 'Apache-2.0'
+ elif license == 'GPL':
+ if 'GPL-2.0' in existing_licenses or 'GPLv2' in existing_licenses:
+ license = 'GPL-2.0'
+ elif 'GPL-3.0' in existing_licenses or 'GPLv3' in existing_licenses:
+ license = 'GPL-3.0'
+ elif license == 'LGPL':
+ if 'LGPL-2.1' in existing_licenses or 'LGPLv2.1' in existing_licenses:
+ license = 'LGPL-2.1'
+ elif 'LGPL-2.0' in existing_licenses or 'LGPLv2' in existing_licenses:
+ license = 'LGPL-2.0'
+ elif 'LGPL-3.0' in existing_licenses or 'LGPLv3' in existing_licenses:
+ license = 'LGPL-3.0'
licenses.append(license)
if licenses:
info['License'] = ' & '.join(licenses)
-
# Map PKG-INFO & setup.py fields to bitbake variables
for field, values in info.items():
if field in self.excluded_fields:
--
2.5.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 03/10] recipetool: create: fix name/version extraction from filename
2016-09-18 20:08 [PATCH 00/10] devtool add / recipetool create fixes Paul Eggleton
2016-09-18 20:08 ` [PATCH 01/10] recipetool: create: fix picking up name from local python source tree Paul Eggleton
2016-09-18 20:08 ` [PATCH 02/10] recipetool: create: improve python recipe license handling Paul Eggleton
@ 2016-09-18 20:08 ` Paul Eggleton
2016-09-18 20:08 ` [PATCH 04/10] recipetool: create: fix error with git tree and no network Paul Eggleton
` (6 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Paul Eggleton @ 2016-09-18 20:08 UTC (permalink / raw)
To: openembedded-core
I ran into an example where recipetool was getting the name/version
completely wrong:
https://bitbucket.org/sortsmill/libunicodenames/downloads/libunicodenames-1.1.0_beta1.tar.xz
From this it would create a libunicodenames-1.1.0-beta1_1.1.0-beta1.bb
file (likely because it couldn't split the file name and therefore took
all of it, then got the version from one of the files inside the
tarball). When this happens it's just irritating because you then have
to delete the recipe / run devtool reset and then run recipetool create
/ devtool add again and specify the version manually.
This patch is the result of systematically running the
determine_from_filename() function over the files on the Yocto Project
source mirror and my local downloads directory and fixing as many of the
generic issues as reasonably practical - it now gets the name and
version correct much more often. There are still cases where it won't,
but they are now in the minority.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 50 +++++++++++++++++++++++++++++-----------
1 file changed, 36 insertions(+), 14 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index cd86747..f7b0676 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -256,27 +256,49 @@ def validate_pv(pv):
def determine_from_filename(srcfile):
"""Determine name and version from a filename"""
- part = ''
- if '.tar.' in srcfile:
- namepart = srcfile.split('.tar.')[0].lower()
- else:
- namepart = os.path.splitext(srcfile)[0].lower()
if is_package(srcfile):
# Force getting the value from the package metadata
return None, None
+
+ if '.tar.' in srcfile:
+ namepart = srcfile.split('.tar.')[0]
else:
- splitval = namepart.rsplit('_', 1)
+ namepart = os.path.splitext(srcfile)[0]
+ namepart = namepart.lower().replace('_', '-')
+ if namepart.endswith('.src'):
+ namepart = namepart[:-4]
+ if namepart.endswith('.orig'):
+ namepart = namepart[:-5]
+ splitval = namepart.split('-')
+ logger.debug('determine_from_filename: split name %s into: %s' % (srcfile, splitval))
+
+ ver_re = re.compile('^v?[0-9]')
+
+ pv = None
+ pn = None
if len(splitval) == 1:
- splitval = namepart.rsplit('-', 1)
- pn = splitval[0].replace('_', '-')
- if len(splitval) > 1:
- if splitval[1][0] in '0123456789':
- pv = splitval[1]
+ # Try to split the version out if there is no separator (or a .)
+ res = re.match('^([^0-9]+)([0-9.]+.*)$', namepart)
+ if res:
+ if len(res.group(1)) > 1 and len(res.group(2)) > 1:
+ pn = res.group(1).rstrip('.')
+ pv = res.group(2)
else:
- pn = '-'.join(splitval).replace('_', '-')
- pv = None
+ pn = namepart
else:
- pv = None
+ if splitval[-1] in ['source', 'src']:
+ splitval.pop()
+ if len(splitval) > 2 and re.match('^(alpha|beta|stable|release|rc[0-9]|pre[0-9]|p[0-9]|[0-9]{8})', splitval[-1]) and ver_re.match(splitval[-2]):
+ pv = '-'.join(splitval[-2:])
+ if pv.endswith('-release'):
+ pv = pv[:-8]
+ splitval = splitval[:-2]
+ elif ver_re.match(splitval[-1]):
+ pv = splitval.pop()
+ pn = '-'.join(splitval)
+ if pv and pv.startswith('v'):
+ pv = pv[1:]
+ logger.debug('determine_from_filename: name = "%s" version = "%s"' % (pn, pv))
return (pn, pv)
def determine_from_url(srcuri):
--
2.5.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 04/10] recipetool: create: fix error with git tree and no network
2016-09-18 20:08 [PATCH 00/10] devtool add / recipetool create fixes Paul Eggleton
` (2 preceding siblings ...)
2016-09-18 20:08 ` [PATCH 03/10] recipetool: create: fix name/version extraction from filename Paul Eggleton
@ 2016-09-18 20:08 ` Paul Eggleton
2016-09-18 20:08 ` [PATCH 05/10] recipetool: create: detect python autoconf macros Paul Eggleton
` (5 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Paul Eggleton @ 2016-09-18 20:08 UTC (permalink / raw)
To: openembedded-core
When creating a recipe for an existing local git clone, we attempt to
use the fetcher to determine if it supports the SRCREV variable.
Unfortunately running this code does a network check to get the latest
revision as a direct result of us using '${AUTOREV}' as a default value.
If you don't have a network connection this will of course fail. Rather
than have this block creating the recipe, catch the exception and just
guess from the URL.
Ultimately this should probably be fixed in the fetcher but for now this
will at least resolve the issue on this end.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f7b0676..f8701d4 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -336,10 +336,16 @@ def supports_srcrev(uri):
# odd interactions with the urldata cache which lead to errors
localdata.setVar('SRCREV', '${AUTOREV}')
bb.data.update_data(localdata)
- fetcher = bb.fetch2.Fetch([uri], localdata)
- urldata = fetcher.ud
- for u in urldata:
- if urldata[u].method.supports_srcrev():
+ try:
+ fetcher = bb.fetch2.Fetch([uri], localdata)
+ urldata = fetcher.ud
+ for u in urldata:
+ if urldata[u].method.supports_srcrev():
+ return True
+ except bb.fetch2.FetchError as e:
+ logger.debug('FetchError in supports_srcrev: %s' % str(e))
+ # Fall back to basic check
+ if uri.startswith(('git://', 'gitsm://')):
return True
return False
--
2.5.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 05/10] recipetool: create: detect python autoconf macros
2016-09-18 20:08 [PATCH 00/10] devtool add / recipetool create fixes Paul Eggleton
` (3 preceding siblings ...)
2016-09-18 20:08 ` [PATCH 04/10] recipetool: create: fix error with git tree and no network Paul Eggleton
@ 2016-09-18 20:08 ` Paul Eggleton
2016-09-18 20:08 ` [PATCH 06/10] recipetool: create: pick up AC_PROG_SWIG Paul Eggleton
` (4 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Paul Eggleton @ 2016-09-18 20:08 UTC (permalink / raw)
To: openembedded-core
If python is required then we need to inherit pythonnative (or
python3native) otherwise do_configure will probably fail since it won't
be able to find python.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_buildsys.py | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index 0ad748e..067be18 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -442,6 +442,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
ac_init_re = re.compile('AC_INIT\(\s*([^,]+),\s*([^,]+)[,)].*')
am_init_re = re.compile('AM_INIT_AUTOMAKE\(\s*([^,]+),\s*([^,]+)[,)].*')
define_re = re.compile('\s*(m4_)?define\(\s*([^,]+),\s*([^,]+)\)')
+ version_re = re.compile('([0-9.]+)')
defines = {}
def subst_defines(value):
@@ -488,6 +489,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
for handler in handlers:
if handler.process_macro(srctree, keyword, value, process_value, libdeps, pcdeps, deps, outlines, inherits, values):
return
+ logger.debug('Found keyword %s with value "%s"' % (keyword, value))
if keyword == 'PKG_CHECK_MODULES':
res = pkg_re.search(value)
if res:
@@ -573,6 +575,17 @@ class AutotoolsRecipeHandler(RecipeHandler):
deps.append('swig-native')
elif keyword == 'AX_PROG_XSLTPROC':
deps.append('libxslt-native')
+ elif keyword in ['AC_PYTHON_DEVEL', 'AX_PYTHON_DEVEL', 'AM_PATH_PYTHON']:
+ pythonclass = 'pythonnative'
+ res = version_re.search(value)
+ if res:
+ if res.group(1).startswith('3'):
+ pythonclass = 'python3native'
+ # Avoid replacing python3native with pythonnative
+ if not pythonclass in inherits and not 'python3native' in inherits:
+ if 'pythonnative' in inherits:
+ inherits.remove('pythonnative')
+ inherits.append(pythonclass)
elif keyword == 'AX_WITH_CURSES':
deps.append('ncurses')
elif keyword == 'AX_PATH_BDB':
@@ -639,6 +652,9 @@ class AutotoolsRecipeHandler(RecipeHandler):
'AX_LIB_TAGLIB',
'AX_PKG_SWIG',
'AX_PROG_XSLTPROC',
+ 'AC_PYTHON_DEVEL',
+ 'AX_PYTHON_DEVEL',
+ 'AM_PATH_PYTHON',
'AX_WITH_CURSES',
'AX_PATH_BDB',
'AX_PATH_LIB_PCRE',
--
2.5.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 06/10] recipetool: create: pick up AC_PROG_SWIG
2016-09-18 20:08 [PATCH 00/10] devtool add / recipetool create fixes Paul Eggleton
` (4 preceding siblings ...)
2016-09-18 20:08 ` [PATCH 05/10] recipetool: create: detect python autoconf macros Paul Eggleton
@ 2016-09-18 20:08 ` Paul Eggleton
2016-09-18 20:08 ` [PATCH 07/10] recipetool: create: tweak license crunching Paul Eggleton
` (3 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Paul Eggleton @ 2016-09-18 20:08 UTC (permalink / raw)
To: openembedded-core
AX_PKG_SWIG is not the only commonly-used macro for detecting swig -
there's also AC_PROG_SWIG. As per AX_PKG_SWIG, add swig-native to
DEPENDS if AC_PROG_SWIG is found in configure.ac.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_buildsys.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index 067be18..e914e53 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -571,7 +571,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
deps.append('sqlite3')
elif keyword == 'AX_LIB_TAGLIB':
deps.append('taglib')
- elif keyword == 'AX_PKG_SWIG':
+ elif keyword in ['AX_PKG_SWIG', 'AC_PROG_SWIG']:
deps.append('swig-native')
elif keyword == 'AX_PROG_XSLTPROC':
deps.append('libxslt-native')
@@ -651,6 +651,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
'AX_LIB_SQLITE3',
'AX_LIB_TAGLIB',
'AX_PKG_SWIG',
+ 'AC_PROG_SWIG',
'AX_PROG_XSLTPROC',
'AC_PYTHON_DEVEL',
'AX_PYTHON_DEVEL',
--
2.5.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 07/10] recipetool: create: tweak license crunching
2016-09-18 20:08 [PATCH 00/10] devtool add / recipetool create fixes Paul Eggleton
` (5 preceding siblings ...)
2016-09-18 20:08 ` [PATCH 06/10] recipetool: create: pick up AC_PROG_SWIG Paul Eggleton
@ 2016-09-18 20:08 ` Paul Eggleton
2016-09-18 20:08 ` [PATCH 08/10] recipetool: create: support git short form URLs Paul Eggleton
` (2 subsequent siblings)
9 siblings, 0 replies; 11+ messages in thread
From: Paul Eggleton @ 2016-09-18 20:08 UTC (permalink / raw)
To: openembedded-core
Filter out a plain "Licensed under the XXXX license" statement, as seen
in the capnproto project (and no doubt others).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f8701d4..7787d8e 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -872,7 +872,7 @@ def crunch_license(licfile):
# Note: these are carefully constructed!
license_title_re = re.compile('^\(?(#+ *)?(The )?.{1,10} [Ll]icen[sc]e( \(.{1,10}\))?\)?:?$')
- license_statement_re = re.compile('^This (project|software) is( free software)? released under the .{1,10} [Ll]icen[sc]e:?$')
+ license_statement_re = re.compile('^(This (project|software) is( free software)? (released|licen[sc]ed)|(Released|Licen[cs]ed)) under the .{1,10} [Ll]icen[sc]e:?$')
copyright_re = re.compile('^(#+)? *Copyright .*$')
crunched_md5sums = {}
--
2.5.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 08/10] recipetool: create: support git short form URLs
2016-09-18 20:08 [PATCH 00/10] devtool add / recipetool create fixes Paul Eggleton
` (6 preceding siblings ...)
2016-09-18 20:08 ` [PATCH 07/10] recipetool: create: tweak license crunching Paul Eggleton
@ 2016-09-18 20:08 ` Paul Eggleton
2016-09-18 20:08 ` [PATCH 09/10] lib/oe/recipeutils: fix invalid character detection in validate_pn() Paul Eggleton
2016-09-18 20:08 ` [PATCH 10/10] devtool: add: drop superfluous validation for recipe name Paul Eggleton
9 siblings, 0 replies; 11+ messages in thread
From: Paul Eggleton @ 2016-09-18 20:08 UTC (permalink / raw)
To: openembedded-core
In keeping with making recipetool create / devtool add as easy to use as
possible, users shouldn't have to know how to reformat git short form ssh
URLs for consumption by BitBake's fetcher (for example
user@git.example.com:repo.git should be expressed as
git://user@git.example.com/repo.git;protocol=ssh ) - instead we should
just take care of that automatically. Add some logic in the appropriate
places to do that.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 4 ++--
scripts/lib/recipetool/create.py | 9 ++++++---
scripts/lib/scriptutils.py | 13 +++++++++++++
3 files changed, 21 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index baef23e..abbc0cb 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -47,13 +47,13 @@ def add(args, config, basepath, workspace):
# These are positional arguments, but because we're nice, allow
# specifying e.g. source tree without name, or fetch URI without name or
# source tree (if we can detect that that is what the user meant)
- if '://' in args.recipename:
+ if scriptutils.is_src_url(args.recipename):
if not args.fetchuri:
if args.fetch:
raise DevtoolError('URI specified as positional argument as well as -f/--fetch')
args.fetchuri = args.recipename
args.recipename = ''
- elif args.srctree and '://' in args.srctree:
+ elif scriptutils.is_src_url(args.srctree):
if not args.fetchuri:
if args.fetch:
raise DevtoolError('URI specified as positional argument as well as -f/--fetch')
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 7787d8e..9b31fe9 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -353,10 +353,13 @@ def reformat_git_uri(uri):
'''Convert any http[s]://....git URI into git://...;protocol=http[s]'''
checkuri = uri.split(';', 1)[0]
if checkuri.endswith('.git') or '/git/' in checkuri or re.match('https?://github.com/[^/]+/[^/]+/?$', checkuri):
- res = re.match('(https?)://([^;]+(\.git)?)(;.*)?$', uri)
+ res = re.match('(http|https|ssh)://([^;]+(\.git)?)(;.*)?$', uri)
if res:
# Need to switch the URI around so that the git fetcher is used
return 'git://%s;protocol=%s%s' % (res.group(2), res.group(1), res.group(4) or '')
+ elif '@' in checkuri:
+ # Catch e.g. git@git.example.com:repo.git
+ return 'git://%s;protocol=ssh' % checkuri.replace(':', '/', 1)
return uri
def is_package(url):
@@ -386,7 +389,7 @@ def create_recipe(args):
if os.path.isfile(source):
source = 'file://%s' % os.path.abspath(source)
- if '://' in source:
+ if scriptutils.is_src_url(source):
# Fetch a URL
fetchuri = reformat_git_uri(urldefrag(source)[0])
if args.binary:
@@ -478,7 +481,7 @@ def create_recipe(args):
for line in stdout.splitlines():
splitline = line.split()
if len(splitline) > 1:
- if splitline[0] == 'origin' and '://' in splitline[1]:
+ if splitline[0] == 'origin' and scriptutils.is_src_url(splitline[1]):
srcuri = reformat_git_uri(splitline[1])
srcsubdir = 'git'
break
diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index bd082d8..5ccc027 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -116,3 +116,16 @@ def run_editor(fn):
except OSError as exc:
logger.error("Execution of editor '%s' failed: %s", editor, exc)
return 1
+
+def is_src_url(param):
+ """
+ Check if a parameter is a URL and return True if so
+ NOTE: be careful about changing this as it will influence how devtool/recipetool command line handling works
+ """
+ if not param:
+ return False
+ elif '://' in param:
+ return True
+ elif param.startswith('git@') or ('@' in param and param.endswith('.git')):
+ return True
+ return False
--
2.5.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 09/10] lib/oe/recipeutils: fix invalid character detection in validate_pn()
2016-09-18 20:08 [PATCH 00/10] devtool add / recipetool create fixes Paul Eggleton
` (7 preceding siblings ...)
2016-09-18 20:08 ` [PATCH 08/10] recipetool: create: support git short form URLs Paul Eggleton
@ 2016-09-18 20:08 ` Paul Eggleton
2016-09-18 20:08 ` [PATCH 10/10] devtool: add: drop superfluous validation for recipe name Paul Eggleton
9 siblings, 0 replies; 11+ messages in thread
From: Paul Eggleton @ 2016-09-18 20:08 UTC (permalink / raw)
To: openembedded-core
* validate_pn() is supposed to protect against invalid characters, fix
the function so that it actually does (unanchored regex strikes
again...)
* However, now that the function is enforcing the restrictions, we do
still want to allow + in recipe names (e.g. "gtk+")
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/recipeutils.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index a0d78dd..58e4028 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -449,8 +449,8 @@ def get_recipe_patched_files(d):
def validate_pn(pn):
"""Perform validation on a recipe name (PN) for a new recipe."""
reserved_names = ['forcevariable', 'append', 'prepend', 'remove']
- if not re.match('[0-9a-z-.]+', pn):
- return 'Recipe name "%s" is invalid: only characters 0-9, a-z, - and . are allowed' % pn
+ if not re.match('^[0-9a-z-.+]+$', pn):
+ return 'Recipe name "%s" is invalid: only characters 0-9, a-z, -, + and . are allowed' % pn
elif pn in reserved_names:
return 'Recipe name "%s" is invalid: is a reserved keyword' % pn
elif pn.startswith('pn-'):
--
2.5.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH 10/10] devtool: add: drop superfluous validation for recipe name
2016-09-18 20:08 [PATCH 00/10] devtool add / recipetool create fixes Paul Eggleton
` (8 preceding siblings ...)
2016-09-18 20:08 ` [PATCH 09/10] lib/oe/recipeutils: fix invalid character detection in validate_pn() Paul Eggleton
@ 2016-09-18 20:08 ` Paul Eggleton
9 siblings, 0 replies; 11+ messages in thread
From: Paul Eggleton @ 2016-09-18 20:08 UTC (permalink / raw)
To: openembedded-core
Now that recipeutils.validate_pn() properly validates characters used in
the name, we can drop this bit checking for '/' since that's not
permitted by validate_pn(). (The FIXME comment here - that I myself
apparently wrote - is questionable since that function was clearly never
intended to allow '/', perhaps I was misled because it was broken and
did so).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/devtool/standard.py | 4 ----
1 file changed, 4 deletions(-)
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index abbc0cb..8319145 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -85,10 +85,6 @@ def add(args, config, basepath, workspace):
if reason:
raise DevtoolError(reason)
- # FIXME this ought to be in validate_pn but we're using that in other contexts
- if '/' in args.recipename:
- raise DevtoolError('"/" is not a valid character in recipe names')
-
if args.srctree:
srctree = os.path.abspath(args.srctree)
srctreeparent = None
--
2.5.5
^ permalink raw reply related [flat|nested] 11+ messages in thread