* [PATCH 1/9] lib/oe/patch: handle non-UTF8 encoding when reading patches
2016-09-06 10:03 [PATCH 0/9] devtool / recipetool fixes Paul Eggleton
@ 2016-09-06 10:03 ` Paul Eggleton
2016-09-06 15:50 ` Enrico Scholz
2016-09-06 10:03 ` [PATCH 2/9] devtool: update-recipe: support files with subdir= Paul Eggleton
` (7 subsequent siblings)
8 siblings, 1 reply; 15+ messages in thread
From: Paul Eggleton @ 2016-09-06 10:03 UTC (permalink / raw)
To: openembedded-core
When extracting patches from a git repository with PATCHTOOL = "git" we
cannot assume that all patches will be UTF-8 formatted, so as with other
places in this module, try latin-1 if utf-8 fails.
This fixes UnicodeDecodeError running devtool update-recipe or devtool
finish on the openssl recipe.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/patch.py | 28 ++++++++++++++++++----------
1 file changed, 18 insertions(+), 10 deletions(-)
diff --git a/meta/lib/oe/patch.py b/meta/lib/oe/patch.py
index cad5015..05e0faa 100644
--- a/meta/lib/oe/patch.py
+++ b/meta/lib/oe/patch.py
@@ -415,16 +415,24 @@ class GitApplyTree(PatchTree):
out = runcmd(["sh", "-c", " ".join(shellcmd)], tree)
if out:
for srcfile in out.split():
- patchlines = []
- outfile = None
- with open(srcfile, 'r') as f:
- for line in f:
- if line.startswith(GitApplyTree.patch_line_prefix):
- outfile = line.split()[-1].strip()
- continue
- if line.startswith(GitApplyTree.ignore_commit_prefix):
- continue
- patchlines.append(line)
+ for encoding in ['utf-8', 'latin-1']:
+ patchlines = []
+ outfile = None
+ try:
+ with open(srcfile, 'r', encoding=encoding) as f:
+ for line in f:
+ if line.startswith(GitApplyTree.patch_line_prefix):
+ outfile = line.split()[-1].strip()
+ continue
+ if line.startswith(GitApplyTree.ignore_commit_prefix):
+ continue
+ patchlines.append(line)
+ except UnicodeDecodeError:
+ continue
+ break
+ else:
+ raise PatchError('Unable to find a character encoding to decode %s' % srcfile)
+
if not outfile:
outfile = os.path.basename(srcfile)
with open(os.path.join(outdir, outfile), 'w') as of:
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [PATCH 1/9] lib/oe/patch: handle non-UTF8 encoding when reading patches
2016-09-06 10:03 ` [PATCH 1/9] lib/oe/patch: handle non-UTF8 encoding when reading patches Paul Eggleton
@ 2016-09-06 15:50 ` Enrico Scholz
2016-09-06 20:16 ` Paul Eggleton
0 siblings, 1 reply; 15+ messages in thread
From: Enrico Scholz @ 2016-09-06 15:50 UTC (permalink / raw)
To: openembedded-core
Paul Eggleton <paul.eggleton-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
writes:
> When extracting patches from a git repository with PATCHTOOL = "git" we
> cannot assume that all patches will be UTF-8 formatted, so as with other
> places in this module, try latin-1 if utf-8 fails.
This will probably not work when patch contains a character between 128
and 159 (which is a blackhole in all locales afaik).
I would read the file as a binary ('rb' instead of 'r') and make the
GitApplyTree.* strings a 'bytes' type.
Enrico
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 1/9] lib/oe/patch: handle non-UTF8 encoding when reading patches
2016-09-06 15:50 ` Enrico Scholz
@ 2016-09-06 20:16 ` Paul Eggleton
2016-09-07 10:07 ` Enrico Scholz
0 siblings, 1 reply; 15+ messages in thread
From: Paul Eggleton @ 2016-09-06 20:16 UTC (permalink / raw)
To: Enrico Scholz; +Cc: openembedded-core
Hi Enrico,
On Tue, 06 Sep 2016 17:50:02 Enrico Scholz wrote:
> Paul Eggleton <paul.eggleton-VuQAYsv1563Yd54FQh9/CA@public.gmane.org>
> writes:
> > When extracting patches from a git repository with PATCHTOOL = "git" we
> > cannot assume that all patches will be UTF-8 formatted, so as with other
> > places in this module, try latin-1 if utf-8 fails.
>
> This will probably not work when patch contains a character between 128
> and 159 (which is a blackhole in all locales afaik).
I realise it's by no means perfect - you may even fairly label it a hack,
since it's only handling two encodings out of many. However I was keen to at
least restore the ability to handle the majority of patches we have in the
core, we can always improve it subsequently (even before the release).
> I would read the file as a binary ('rb' instead of 'r') and make the
> GitApplyTree.* strings a 'bytes' type.
The code is not just passing the data through, it is actually processing it.
If we did what you propose wouldn't it make that processing more difficult?
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 1/9] lib/oe/patch: handle non-UTF8 encoding when reading patches
2016-09-06 20:16 ` Paul Eggleton
@ 2016-09-07 10:07 ` Enrico Scholz
0 siblings, 0 replies; 15+ messages in thread
From: Enrico Scholz @ 2016-09-07 10:07 UTC (permalink / raw)
To: Paul Eggleton; +Cc: openembedded-core
Paul Eggleton <paul.eggleton@linux.intel.com> writes:
>> I would read the file as a binary ('rb' instead of 'r') and make the
>> GitApplyTree.* strings a 'bytes' type.
>
> The code is not just passing the data through, it is actually processing
> it. If we did what you propose wouldn't it make that processing more
> difficult?
Should not be very much... You have to write b'foo' instead of 'foo' at
some places, but things like 'file.readline()' or 'line.startswith()'
are still working with 'bytes' resp. binary files.
Only when trying to print them on stdout, things can get nasty. But
this should not be needed for patches.
Enrico
--
SIGMA Chemnitz GmbH Registergericht: Amtsgericht Chemnitz HRB 1750
Am Erlenwald 13 Geschaeftsfuehrer: Grit Freitag, Frank Pyritz
09128 Chemnitz
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH 2/9] devtool: update-recipe: support files with subdir=
2016-09-06 10:03 [PATCH 0/9] devtool / recipetool fixes Paul Eggleton
2016-09-06 10:03 ` [PATCH 1/9] lib/oe/patch: handle non-UTF8 encoding when reading patches Paul Eggleton
@ 2016-09-06 10:03 ` Paul Eggleton
2016-09-06 10:03 ` [PATCH 3/9] recipetool: create: AX_PKG_SWIG should add dependency on swig-native Paul Eggleton
` (6 subsequent siblings)
8 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-09-06 10:03 UTC (permalink / raw)
To: openembedded-core
It's rare but there are recipes that have individual files (as opposed
to archives) in SRC_URI using subdir= to put them under the source tree,
the examples in OE-Core being bzip2 and openssl. This broke devtool
update-recipe (and devtool finish) because the file wasn't unpacked into
the oe-local-files directory and thus when it came time to update the
recipe, the file was assumed to have been deleted by the user and thus
the file was erroneously removed. Add logic to handle these properly so
that this doesn't happen.
(We still have another potential problem in that these files become part
of the initial commit from upstream, which could be confusing because
they didn't come from there - but that's a separate issue and not one
that is trivially solved.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/lib/oe/recipeutils.py | 12 ++++++++++--
scripts/lib/devtool/standard.py | 38 ++++++++++++++++++++++++++++++++++++--
2 files changed, 46 insertions(+), 4 deletions(-)
diff --git a/meta/lib/oe/recipeutils.py b/meta/lib/oe/recipeutils.py
index e7dd8af..a0d78dd 100644
--- a/meta/lib/oe/recipeutils.py
+++ b/meta/lib/oe/recipeutils.py
@@ -400,8 +400,16 @@ def get_recipe_local_files(d, patches=False):
bb.utils.exec_flat_python_func('patch_path', uri, fetch, '')):
continue
# Skip files that are referenced by absolute path
- if not os.path.isabs(fetch.ud[uri].basepath):
- ret[fetch.ud[uri].basepath] = fetch.localpath(uri)
+ fname = fetch.ud[uri].basepath
+ if os.path.isabs(fname):
+ continue
+ # Handle subdir=
+ subdir = fetch.ud[uri].parm.get('subdir', '')
+ if subdir:
+ if os.path.isabs(subdir):
+ continue
+ fname = os.path.join(subdir, fname)
+ ret[fname] = fetch.localpath(uri)
return ret
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index 98451da..baef23e 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -307,6 +307,13 @@ def _move_file(src, dst):
bb.utils.mkdirhier(dst_d)
shutil.move(src, dst)
+def _copy_file(src, dst):
+ """Copy a file. Creates all the directory components of destination path."""
+ dst_d = os.path.dirname(dst)
+ if dst_d:
+ bb.utils.mkdirhier(dst_d)
+ shutil.copy(src, dst)
+
def _git_ls_tree(repodir, treeish='HEAD', recursive=False):
"""List contents of a git treeish"""
import bb
@@ -1050,6 +1057,23 @@ def _export_local_files(srctree, rd, destdir):
elif fname != '.gitignore':
added[fname] = None
+ workdir = rd.getVar('WORKDIR', True)
+ s = rd.getVar('S', True)
+ if not s.endswith(os.sep):
+ s += os.sep
+
+ if workdir != s:
+ # Handle files where subdir= was specified
+ for fname in list(existing_files.keys()):
+ # FIXME handle both subdir starting with BP and not?
+ fworkpath = os.path.join(workdir, fname)
+ if fworkpath.startswith(s):
+ fpath = os.path.join(srctree, os.path.relpath(fworkpath, s))
+ if os.path.exists(fpath):
+ origpath = existing_files.pop(fname)
+ if not filecmp.cmp(origpath, fpath):
+ updated[fpath] = origpath
+
removed = existing_files
return (updated, added, removed)
@@ -1122,7 +1146,12 @@ def _update_recipe_srcrev(srctree, rd, appendlayerdir, wildcard_version, no_remo
files_dir = _determine_files_dir(rd)
for basepath, path in upd_f.items():
logger.info('Updating file %s' % basepath)
- _move_file(os.path.join(local_files_dir, basepath), path)
+ if os.path.isabs(basepath):
+ # Original file (probably with subdir pointing inside source tree)
+ # so we do not want to move it, just copy
+ _copy_file(basepath, path)
+ else:
+ _move_file(os.path.join(local_files_dir, basepath), path)
update_srcuri= True
for basepath, path in new_f.items():
logger.info('Adding new file %s' % basepath)
@@ -1205,7 +1234,12 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
# Update existing files
for basepath, path in upd_f.items():
logger.info('Updating file %s' % basepath)
- _move_file(os.path.join(local_files_dir, basepath), path)
+ if os.path.isabs(basepath):
+ # Original file (probably with subdir pointing inside source tree)
+ # so we do not want to move it, just copy
+ _copy_file(basepath, path)
+ else:
+ _move_file(os.path.join(local_files_dir, basepath), path)
updatefiles = True
for basepath, path in upd_p.items():
patchfn = os.path.join(patches_dir, basepath)
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 3/9] recipetool: create: AX_PKG_SWIG should add dependency on swig-native
2016-09-06 10:03 [PATCH 0/9] devtool / recipetool fixes Paul Eggleton
2016-09-06 10:03 ` [PATCH 1/9] lib/oe/patch: handle non-UTF8 encoding when reading patches Paul Eggleton
2016-09-06 10:03 ` [PATCH 2/9] devtool: update-recipe: support files with subdir= Paul Eggleton
@ 2016-09-06 10:03 ` Paul Eggleton
2016-09-06 10:03 ` [PATCH 4/9] recipetool: create: fix mapping python dependencies to python-dbg package Paul Eggleton
` (5 subsequent siblings)
8 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-09-06 10:03 UTC (permalink / raw)
To: openembedded-core
If AX_PKG_SWIG is found in configure.ac, then what's being looked for is
the swig binary, not swig for the target - so fix the dependency
accordingly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_buildsys.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index f784f94..0ad748e 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -570,7 +570,7 @@ class AutotoolsRecipeHandler(RecipeHandler):
elif keyword == 'AX_LIB_TAGLIB':
deps.append('taglib')
elif keyword == 'AX_PKG_SWIG':
- deps.append('swig')
+ deps.append('swig-native')
elif keyword == 'AX_PROG_XSLTPROC':
deps.append('libxslt-native')
elif keyword == 'AX_WITH_CURSES':
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 4/9] recipetool: create: fix mapping python dependencies to python-dbg package
2016-09-06 10:03 [PATCH 0/9] devtool / recipetool fixes Paul Eggleton
` (2 preceding siblings ...)
2016-09-06 10:03 ` [PATCH 3/9] recipetool: create: AX_PKG_SWIG should add dependency on swig-native Paul Eggleton
@ 2016-09-06 10:03 ` Paul Eggleton
2016-09-06 10:03 ` [PATCH 5/9] recipetool: create: support git submodules Paul Eggleton
` (4 subsequent siblings)
8 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-09-06 10:03 UTC (permalink / raw)
To: openembedded-core
When trying to map python module dependencies to the packages that
provide them, if we're looking for .so files that satisfy
dependencies then we need to exclude files found under the .debug
directory, otherwise the dependency will get mapped to the python-dbg
package which isn't correct.
For example, this fixes creating a recipe for pyserial and not getting
python-fcntl in RDEPENDS_${PN}, leading to errors when trying to use the
serial module on the target.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_buildsys_python.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index aff13cf..fb9806d 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -504,8 +504,10 @@ class PythonRecipeHandler(RecipeHandler):
for dep in scanned_deps:
mapped = provided_packages.get(dep)
if mapped:
+ logger.debug('Mapped %s to %s' % (dep, mapped))
mapped_deps.add(mapped)
else:
+ logger.debug('Could not map %s' % dep)
unmapped_deps.add(dep)
return mapped_deps, unmapped_deps
@@ -566,6 +568,8 @@ class PythonRecipeHandler(RecipeHandler):
continue
if fn.startswith(dynload_dir + os.sep):
+ if '/.debug/' in fn:
+ continue
base = os.path.basename(fn)
provided = base.split('.', 1)[0]
packages[provided] = os.path.basename(pkgdatafile)
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 5/9] recipetool: create: support git submodules
2016-09-06 10:03 [PATCH 0/9] devtool / recipetool fixes Paul Eggleton
` (3 preceding siblings ...)
2016-09-06 10:03 ` [PATCH 4/9] recipetool: create: fix mapping python dependencies to python-dbg package Paul Eggleton
@ 2016-09-06 10:03 ` Paul Eggleton
2016-09-07 6:23 ` Ola x Nilsson
2016-09-06 10:03 ` [PATCH 6/9] recipetool: create: add --keep-temp command line option Paul Eggleton
` (3 subsequent siblings)
8 siblings, 1 reply; 15+ messages in thread
From: Paul Eggleton @ 2016-09-06 10:03 UTC (permalink / raw)
To: openembedded-core
Ensure we fetch submodules and set SRC_URI correctly when pointing to a
git repository that contains submodules.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index a08352e..7cbd614 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -399,6 +399,11 @@ def create_recipe(args):
if '<html' in f.read(100).lower():
logger.error('Fetching "%s" returned a single HTML page - check the URL is correct and functional' % fetchuri)
sys.exit(1)
+ if os.path.exists(os.path.join(srctree, '.gitmodules')) and srcuri.startswith('git://'):
+ srcuri = 'gitsm://' + srcuri[6:]
+ logger.info('Fetching submodules...')
+ bb.process.run('git submodule init', cwd=srctree)
+ bb.process.run('git submodule update', cwd=srctree)
if is_package(fetchuri):
tmpfdir = tempfile.mkdtemp(prefix='recipetool-')
@@ -658,8 +663,11 @@ def create_recipe(args):
# devtool looks for this specific exit code, so don't change it
sys.exit(15)
else:
- if srcuri and srcuri.startswith(('git://', 'hg://', 'svn://')):
- outfile = '%s_%s.bb' % (pn, srcuri.split(':', 1)[0])
+ if srcuri and srcuri.startswith(('gitsm://', 'git://', 'hg://', 'svn://')):
+ suffix = srcuri.split(':', 1)[0]
+ if suffix == 'gitsm':
+ suffix = 'git'
+ outfile = '%s_%s.bb' % (pn, suffix)
elif realpv:
outfile = '%s_%s.bb' % (pn, realpv)
else:
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* Re: [PATCH 5/9] recipetool: create: support git submodules
2016-09-06 10:03 ` [PATCH 5/9] recipetool: create: support git submodules Paul Eggleton
@ 2016-09-07 6:23 ` Ola x Nilsson
2016-09-07 21:07 ` Paul Eggleton
0 siblings, 1 reply; 15+ messages in thread
From: Ola x Nilsson @ 2016-09-07 6:23 UTC (permalink / raw)
To: openembedded-core@lists.openembedded.org
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> Of Paul Eggleton
> Sent: den 6 september 2016 12:03
>
> Ensure we fetch submodules and set SRC_URI correctly when pointing to a
> git repository that contains submodules.
>
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
> scripts/lib/recipetool/create.py | 12 ++++++++++--
> 1 file changed, 10 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
> index a08352e..7cbd614 100644
> --- a/scripts/lib/recipetool/create.py
> +++ b/scripts/lib/recipetool/create.py
> @@ -399,6 +399,11 @@ def create_recipe(args):
> if '<html' in f.read(100).lower():
> logger.error('Fetching "%s" returned a single HTML page - check
> the URL is correct and functional' % fetchuri)
> sys.exit(1)
> + if os.path.exists(os.path.join(srctree, '.gitmodules')) and
> srcuri.startswith('git://'):
> + srcuri = 'gitsm://' + srcuri[6:]
> + logger.info('Fetching submodules...')
> + bb.process.run('git submodule init', cwd=srctree)
> + bb.process.run('git submodule update', cwd=srctree)
>
This is not how the gitsm fetcher does it.
git submodule update --init --recursive
will actually fetch all submodules recursively.
--
Ola Nilsson
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [PATCH 5/9] recipetool: create: support git submodules
2016-09-07 6:23 ` Ola x Nilsson
@ 2016-09-07 21:07 ` Paul Eggleton
0 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-09-07 21:07 UTC (permalink / raw)
To: Ola x Nilsson; +Cc: openembedded-core
Hi Ola,
On Wed, 07 Sep 2016 06:23:15 Ola x Nilsson wrote:
> > From: openembedded-core-bounces@lists.openembedded.org
> > [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> > Of Paul Eggleton
> > Sent: den 6 september 2016 12:03
> >
> > Ensure we fetch submodules and set SRC_URI correctly when pointing to a
> > git repository that contains submodules.
> >
> > Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> > ---
> >
> > scripts/lib/recipetool/create.py | 12 ++++++++++--
> > 1 file changed, 10 insertions(+), 2 deletions(-)
> >
> > diff --git a/scripts/lib/recipetool/create.py
> > b/scripts/lib/recipetool/create.py index a08352e..7cbd614 100644
> > --- a/scripts/lib/recipetool/create.py
> > +++ b/scripts/lib/recipetool/create.py
> >
> > @@ -399,6 +399,11 @@ def create_recipe(args):
> > if '<html' in f.read(100).lower():
> > logger.error('Fetching "%s" returned a single
> > HTML page - check
> >
> > the URL is correct and functional' % fetchuri)
> >
> > sys.exit(1)
> >
> > + if os.path.exists(os.path.join(srctree, '.gitmodules')) and
> > srcuri.startswith('git://'):
> > + srcuri = 'gitsm://' + srcuri[6:]
> > + logger.info('Fetching submodules...')
> > + bb.process.run('git submodule init', cwd=srctree)
> > + bb.process.run('git submodule update', cwd=srctree)
>
> This is not how the gitsm fetcher does it.
>
> git submodule update --init --recursive
>
> will actually fetch all submodules recursively.
Right, we should indeed be doing that. I've pushed a new commit to the branch.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH 6/9] recipetool: create: add --keep-temp command line option
2016-09-06 10:03 [PATCH 0/9] devtool / recipetool fixes Paul Eggleton
` (4 preceding siblings ...)
2016-09-06 10:03 ` [PATCH 5/9] recipetool: create: support git submodules Paul Eggleton
@ 2016-09-06 10:03 ` Paul Eggleton
2016-09-06 10:03 ` [PATCH 7/9] recipetool: create: allow license variable handling to be rerun Paul Eggleton
` (2 subsequent siblings)
8 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-09-06 10:03 UTC (permalink / raw)
To: openembedded-core
For debugging it's useful to be able to tell recipetool to keep the
temporary directory.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 7cbd614..89ab748 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -765,7 +765,10 @@ def create_recipe(args):
logger.info('Recipe %s has been created; further editing may be required to make it fully functional' % outfile)
if tempsrc:
- shutil.rmtree(tempsrc)
+ if args.keep_temp:
+ logger.info('Preserving temporary directory %s' % tempsrc)
+ else:
+ shutil.rmtree(tempsrc)
return 0
@@ -1048,5 +1051,6 @@ def register_commands(subparsers):
parser_create.add_argument('--also-native', help='Also add native variant (i.e. support building recipe for the build host as well as the target machine)', action='store_true')
parser_create.add_argument('--src-subdir', help='Specify subdirectory within source tree to use', metavar='SUBDIR')
parser_create.add_argument('-a', '--autorev', help='When fetching from a git repository, set SRCREV in the recipe to a floating revision instead of fixed', action="store_true")
+ parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
parser_create.set_defaults(func=create_recipe)
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 7/9] recipetool: create: allow license variable handling to be rerun
2016-09-06 10:03 [PATCH 0/9] devtool / recipetool fixes Paul Eggleton
` (5 preceding siblings ...)
2016-09-06 10:03 ` [PATCH 6/9] recipetool: create: add --keep-temp command line option Paul Eggleton
@ 2016-09-06 10:03 ` Paul Eggleton
2016-09-06 10:03 ` [PATCH 8/9] recipetool: create: support node.js code outside of npm Paul Eggleton
2016-09-06 10:03 ` [PATCH 9/9] recipetool: create: avoid extra blank lines in output recipe Paul Eggleton
8 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-09-06 10:03 UTC (permalink / raw)
To: openembedded-core
If you make adjustments to the source tree (as create_npm.py will be)
then you will need to re-run the license variable handling code at the
end so that we get all of the files that should go into
LIC_FILES_CHKSUM if nothing else. Split out the license variable
handling to a separate function in order to allow this.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 103 ++++++++++++++++++++-------------------
1 file changed, 53 insertions(+), 50 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 89ab748..f1ce28c 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -482,51 +482,8 @@ def create_recipe(args):
# We need a blank line here so that patch_recipe_lines can rewind before the LICENSE comments
lines_before.append('')
- licvalues = guess_license(srctree_use)
- lic_files_chksum = []
- lic_unknown = []
- if licvalues:
- licenses = []
- for licvalue in licvalues:
- if not licvalue[0] in licenses:
- licenses.append(licvalue[0])
- lic_files_chksum.append('file://%s;md5=%s' % (licvalue[1], licvalue[2]))
- if licvalue[0] == 'Unknown':
- lic_unknown.append(licvalue[1])
- lines_before.append('# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is')
- lines_before.append('# your responsibility to verify that the values are complete and correct.')
- if len(licvalues) > 1:
- lines_before.append('#')
- lines_before.append('# NOTE: multiple licenses have been detected; if that is correct you should separate')
- lines_before.append('# these in the LICENSE value using & if the multiple licenses all apply, or | if there')
- lines_before.append('# is a choice between the multiple licenses. If in doubt, check the accompanying')
- lines_before.append('# documentation to determine which situation is applicable.')
- if lic_unknown:
- lines_before.append('#')
- lines_before.append('# The following license files were not able to be identified and are')
- lines_before.append('# represented as "Unknown" below, you will need to check them yourself:')
- for licfile in lic_unknown:
- lines_before.append('# %s' % licfile)
- lines_before.append('#')
- else:
- lines_before.append('# Unable to find any files that looked like license statements. Check the accompanying')
- lines_before.append('# documentation and source headers and set LICENSE and LIC_FILES_CHKSUM accordingly.')
- lines_before.append('#')
- lines_before.append('# NOTE: LICENSE is being set to "CLOSED" to allow you to at least start building - if')
- lines_before.append('# this is not accurate with respect to the licensing of the software being built (it')
- lines_before.append('# will not be in most cases) you must specify the correct value before using this')
- lines_before.append('# recipe for anything other than initial testing/development!')
- licenses = ['CLOSED']
- pkg_license = extravalues.pop('LICENSE', None)
- if pkg_license:
- if licenses == ['Unknown']:
- lines_before.append('# NOTE: The following LICENSE value was determined from the original package metadata')
- licenses = [pkg_license]
- else:
- lines_before.append('# NOTE: Original package metadata indicates license is: %s' % pkg_license)
- lines_before.append('LICENSE = "%s"' % ' '.join(licenses))
- lines_before.append('LIC_FILES_CHKSUM = "%s"' % ' \\\n '.join(lic_files_chksum))
- lines_before.append('')
+ handled = []
+ licvalues = handle_license_vars(srctree_use, lines_before, handled, extravalues, tinfoil.config_data)
classes = []
@@ -623,9 +580,6 @@ def create_recipe(args):
handlers = [item[0] for item in handlers]
# Apply the handlers
- handled = []
- handled.append(('license', licvalues))
-
if args.binary:
classes.append('bin_package')
handled.append('buildsystem')
@@ -772,6 +726,55 @@ def create_recipe(args):
return 0
+def handle_license_vars(srctree, lines_before, handled, extravalues, d):
+ licvalues = guess_license(srctree, d)
+ lic_files_chksum = []
+ lic_unknown = []
+ if licvalues:
+ licenses = []
+ for licvalue in licvalues:
+ if not licvalue[0] in licenses:
+ licenses.append(licvalue[0])
+ lic_files_chksum.append('file://%s;md5=%s' % (licvalue[1], licvalue[2]))
+ if licvalue[0] == 'Unknown':
+ lic_unknown.append(licvalue[1])
+ lines_before.append('# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is')
+ lines_before.append('# your responsibility to verify that the values are complete and correct.')
+ if len(licvalues) > 1:
+ lines_before.append('#')
+ lines_before.append('# NOTE: multiple licenses have been detected; if that is correct you should separate')
+ lines_before.append('# these in the LICENSE value using & if the multiple licenses all apply, or | if there')
+ lines_before.append('# is a choice between the multiple licenses. If in doubt, check the accompanying')
+ lines_before.append('# documentation to determine which situation is applicable.')
+ if lic_unknown:
+ lines_before.append('#')
+ lines_before.append('# The following license files were not able to be identified and are')
+ lines_before.append('# represented as "Unknown" below, you will need to check them yourself:')
+ for licfile in lic_unknown:
+ lines_before.append('# %s' % licfile)
+ lines_before.append('#')
+ else:
+ lines_before.append('# Unable to find any files that looked like license statements. Check the accompanying')
+ lines_before.append('# documentation and source headers and set LICENSE and LIC_FILES_CHKSUM accordingly.')
+ lines_before.append('#')
+ lines_before.append('# NOTE: LICENSE is being set to "CLOSED" to allow you to at least start building - if')
+ lines_before.append('# this is not accurate with respect to the licensing of the software being built (it')
+ lines_before.append('# will not be in most cases) you must specify the correct value before using this')
+ lines_before.append('# recipe for anything other than initial testing/development!')
+ licenses = ['CLOSED']
+ pkg_license = extravalues.pop('LICENSE', None)
+ if pkg_license:
+ if licenses == ['Unknown']:
+ lines_before.append('# NOTE: The following LICENSE value was determined from the original package metadata')
+ licenses = [pkg_license]
+ else:
+ lines_before.append('# NOTE: Original package metadata indicates license is: %s' % pkg_license)
+ lines_before.append('LICENSE = "%s"' % ' '.join(licenses))
+ lines_before.append('LIC_FILES_CHKSUM = "%s"' % ' \\\n '.join(lic_files_chksum))
+ lines_before.append('')
+ handled.append(('license', licvalues))
+ return licvalues
+
def get_license_md5sums(d, static_only=False):
import bb.utils
md5sums = {}
@@ -892,9 +895,9 @@ def crunch_license(licfile):
license = crunched_md5sums.get(md5val, None)
return license, md5val, lictext
-def guess_license(srctree):
+def guess_license(srctree, d):
import bb
- md5sums = get_license_md5sums(tinfoil.config_data)
+ md5sums = get_license_md5sums(d)
licenses = []
licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*']
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 8/9] recipetool: create: support node.js code outside of npm
2016-09-06 10:03 [PATCH 0/9] devtool / recipetool fixes Paul Eggleton
` (6 preceding siblings ...)
2016-09-06 10:03 ` [PATCH 7/9] recipetool: create: allow license variable handling to be rerun Paul Eggleton
@ 2016-09-06 10:03 ` Paul Eggleton
2016-09-06 10:03 ` [PATCH 9/9] recipetool: create: avoid extra blank lines in output recipe Paul Eggleton
8 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-09-06 10:03 UTC (permalink / raw)
To: openembedded-core
If you have your own node.js application you may not publish it (or at
least not immediately) in an npm registry - it might just be in a
repository on github or on your local machine. Add support to recipetool
create for creating recipes to build such applications - extract their
dependencies, fetch them, and add corresponding npm:// URLs to SRC_URI,
and ensure that LICENSE / LIC_FILES_CHKSUM are updated to match. For
example, you can now run:
recipetool create https://github.com/diversario/node-ssdp
(I had to borrow some code from bitbake/lib/bb/fetch2/npm.py to
implement this functionality; this should be refactored out but now
isn't the time to do that refactoring.)
Part of the fix for [YOCTO #9537].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/npm.bbclass | 4 ++
scripts/lib/recipetool/create_npm.py | 111 ++++++++++++++++++++++++++++++++++-
2 files changed, 114 insertions(+), 1 deletion(-)
diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index 55c7c3e..fce4c11 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -16,6 +16,10 @@ def npm_oe_arch_map(target_arch, d):
NPM_ARCH ?= "${@npm_oe_arch_map(d.getVar('TARGET_ARCH', True), d)}"
npm_do_compile() {
+ # Copy in any additionally fetched modules
+ if [ -d ${WORKDIR}/node_modules ] ; then
+ cp -a ${WORKDIR}/node_modules ${S}/
+ fi
# changing the home directory to the working directory, the .npmrc will
# be created in this directory
export HOME=${WORKDIR}
diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index e5aaa60..e794614 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -21,7 +21,7 @@ import subprocess
import tempfile
import shutil
import json
-from recipetool.create import RecipeHandler, split_pkg_licenses
+from recipetool.create import RecipeHandler, split_pkg_licenses, handle_license_vars
logger = logging.getLogger('recipetool')
@@ -83,6 +83,66 @@ class NpmRecipeHandler(RecipeHandler):
extravalues['extrafiles']['lockdown.json'] = tmpfile
lines_before.append('NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"')
+ def _handle_dependencies(self, d, deps, lines_before, srctree):
+ import scriptutils
+ # If this isn't a single module we need to get the dependencies
+ # and add them to SRC_URI
+ def varfunc(varname, origvalue, op, newlines):
+ if varname == 'SRC_URI':
+ if not origvalue.startswith('npm://'):
+ src_uri = origvalue.split()
+ changed = False
+ for dep, depdata in deps.items():
+ version = self.get_node_version(dep, depdata, d)
+ if version:
+ url = 'npm://registry.npmjs.org;name=%s;version=%s;subdir=node_modules/%s' % (dep, version, dep)
+ scriptutils.fetch_uri(d, url, srctree)
+ src_uri.append(url)
+ changed = True
+ if changed:
+ return src_uri, None, -1, True
+ return origvalue, None, 0, True
+ updated, newlines = bb.utils.edit_metadata(lines_before, ['SRC_URI'], varfunc)
+ if updated:
+ del lines_before[:]
+ for line in newlines:
+ # Hack to avoid newlines that edit_metadata inserts
+ if line.endswith('\n'):
+ line = line[:-1]
+ lines_before.append(line)
+ return updated
+
+ def _replace_license_vars(self, srctree, lines_before, handled, extravalues, d):
+ for item in handled:
+ if isinstance(item, tuple):
+ if item[0] == 'license':
+ del item
+ break
+
+ calledvars = []
+ def varfunc(varname, origvalue, op, newlines):
+ if varname in ['LICENSE', 'LIC_FILES_CHKSUM']:
+ for i, e in enumerate(reversed(newlines)):
+ if not e.startswith('#'):
+ stop = i
+ while stop > 0:
+ newlines.pop()
+ stop -= 1
+ break
+ calledvars.append(varname)
+ if len(calledvars) > 1:
+ # The second time around, put the new license text in
+ insertpos = len(newlines)
+ handle_license_vars(srctree, newlines, handled, extravalues, d)
+ return None, None, 0, True
+ return origvalue, None, 0, True
+ updated, newlines = bb.utils.edit_metadata(lines_before, ['LICENSE', 'LIC_FILES_CHKSUM'], varfunc)
+ if updated:
+ del lines_before[:]
+ lines_before.extend(newlines)
+ else:
+ raise Exception('Did not find license variables')
+
def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
import bb.utils
import oe
@@ -108,6 +168,12 @@ class NpmRecipeHandler(RecipeHandler):
if 'homepage' in data:
extravalues['HOMEPAGE'] = data['homepage']
+ deps = data.get('dependencies', {})
+ updated = self._handle_dependencies(tinfoil.config_data, deps, lines_before, srctree)
+ if updated:
+ # We need to redo the license stuff
+ self._replace_license_vars(srctree, lines_before, handled, extravalues, tinfoil.config_data)
+
# Shrinkwrap
localfilesdir = tempfile.mkdtemp(prefix='recipetool-npm')
self._shrinkwrap(srctree, localfilesdir, extravalues, lines_before)
@@ -148,9 +214,52 @@ class NpmRecipeHandler(RecipeHandler):
lines_before[i] = 'LICENSE = "%s"' % ' & '.join(all_licenses)
break
+ # Need to move S setting after inherit npm
+ for i, line in enumerate(lines_before):
+ if line.startswith('S ='):
+ lines_before.pop(i)
+ lines_after.insert(0, '# Must be set after inherit npm since that itself sets S')
+ lines_after.insert(1, line)
+ break
+
return True
return False
+ # FIXME this is duplicated from lib/bb/fetch2/npm.py
+ def _parse_view(self, output):
+ '''
+ Parse the output of npm view --json; the last JSON result
+ is assumed to be the one that we're interested in.
+ '''
+ pdata = None
+ outdeps = {}
+ datalines = []
+ bracelevel = 0
+ for line in output.splitlines():
+ if bracelevel:
+ datalines.append(line)
+ elif '{' in line:
+ datalines = []
+ datalines.append(line)
+ bracelevel = bracelevel + line.count('{') - line.count('}')
+ if datalines:
+ pdata = json.loads('\n'.join(datalines))
+ return pdata
+
+ # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
+ # (split out from _getdependencies())
+ def get_node_version(self, pkg, version, d):
+ import bb.fetch2
+ pkgfullname = pkg
+ if version != '*' and not '/' in version:
+ pkgfullname += "@'%s'" % version
+ logger.debug(2, "Calling getdeps on %s" % pkg)
+ runenv = dict(os.environ, PATH=d.getVar('PATH', True))
+ fetchcmd = "npm view %s --json" % pkgfullname
+ output, _ = bb.process.run(fetchcmd, stderr=subprocess.STDOUT, env=runenv, shell=True)
+ data = self._parse_view(output)
+ return data.get('version', None)
+
def register_recipe_handlers(handlers):
handlers.append((NpmRecipeHandler(), 60))
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread* [PATCH 9/9] recipetool: create: avoid extra blank lines in output recipe
2016-09-06 10:03 [PATCH 0/9] devtool / recipetool fixes Paul Eggleton
` (7 preceding siblings ...)
2016-09-06 10:03 ` [PATCH 8/9] recipetool: create: support node.js code outside of npm Paul Eggleton
@ 2016-09-06 10:03 ` Paul Eggleton
8 siblings, 0 replies; 15+ messages in thread
From: Paul Eggleton @ 2016-09-06 10:03 UTC (permalink / raw)
To: openembedded-core
If we output extra blank lines (because of some automated editing) then
it makes the output recipe look a bit untidy. You could argue that we
should simply have the editing code not do that, but sometimes we don't
have enough context there for that to be practical. It's simple enough
to just filter out the extra blank lines when writing the file, so just
do it that way.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f1ce28c..b6de999 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -715,7 +715,13 @@ def create_recipe(args):
sys.stdout.write('\n'.join(outlines) + '\n')
else:
with open(outfile, 'w') as f:
- f.write('\n'.join(outlines) + '\n')
+ lastline = None
+ for line in outlines:
+ if not lastline and not line:
+ # Skip extra blank lines
+ continue
+ f.write('%s\n' % line)
+ lastline = line
logger.info('Recipe %s has been created; further editing may be required to make it fully functional' % outfile)
if tempsrc:
--
2.5.5
^ permalink raw reply related [flat|nested] 15+ messages in thread