* [PATCH 1/6] recipetool: create: be more tolerant of spacing in configure.ac
2016-03-08 5:35 [PATCH 0/6] recipetool create improvements Paul Eggleton
@ 2016-03-08 5:36 ` Paul Eggleton
2016-03-08 5:36 ` [PATCH 2/6] recipetool: create: improve mapping for autotools program macros Paul Eggleton
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-03-08 5:36 UTC (permalink / raw)
To: openembedded-core
Allow for whitespace in appropriate places, and ensure we match all
whitespace chars not just the space character.
(This fixes extracting dependencies from tmux's configure.ac, for
example.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create_buildsys.py | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index 0228269..909743b 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -413,15 +413,15 @@ class AutotoolsRecipeHandler(RecipeHandler):
progclassmap = {'gconftool-2': 'gconf',
'pkg-config': 'pkgconfig'}
- pkg_re = re.compile('PKG_CHECK_MODULES\(\[?[a-zA-Z0-9_]*\]?, *\[?([^,\]]*)\]?[),].*')
- pkgce_re = re.compile('PKG_CHECK_EXISTS\(\[?([^,\]]*)\]?[),].*')
- lib_re = re.compile('AC_CHECK_LIB\(\[?([^,\]]*)\]?,.*')
- libx_re = re.compile('AX_CHECK_LIBRARY\(\[?[^,\]]*\]?, *\[?([^,\]]*)\]?, *\[?([a-zA-Z0-9-]*)\]?,.*')
- progs_re = re.compile('_PROGS?\(\[?[a-zA-Z0-9_]*\]?, \[?([^,\]]*)\]?[),].*')
+ pkg_re = re.compile('PKG_CHECK_MODULES\(\s*\[?[a-zA-Z0-9_]*\]?,\s*\[?([^,\]]*)\]?[),].*')
+ pkgce_re = re.compile('PKG_CHECK_EXISTS\(\s*\[?([^,\]]*)\]?[),].*')
+ lib_re = re.compile('AC_CHECK_LIB\(\s*\[?([^,\]]*)\]?,.*')
+ libx_re = re.compile('AX_CHECK_LIBRARY\(\s*\[?[^,\]]*\]?,\s*\[?([^,\]]*)\]?,\s*\[?([a-zA-Z0-9-]*)\]?,.*')
+ progs_re = re.compile('_PROGS?\(\s*\[?[a-zA-Z0-9_]*\]?,\s*\[?([^,\]]*)\]?[),].*')
dep_re = re.compile('([^ ><=]+)( [<>=]+ [^ ><=]+)?')
- ac_init_re = re.compile('AC_INIT\(([^,]+), *([^,]+)[,)].*')
- am_init_re = re.compile('AM_INIT_AUTOMAKE\(([^,]+), *([^,]+)[,)].*')
- define_re = re.compile(' *(m4_)?define\(([^,]+), *([^,]+)\)')
+ ac_init_re = re.compile('AC_INIT\(\s*([^,]+),\s*([^,]+)[,)].*')
+ am_init_re = re.compile('AM_INIT_AUTOMAKE\(\s*([^,]+),\s*([^,]+)[,)].*')
+ define_re = re.compile('\s*(m4_)?define\(\s*([^,]+),\s*([^,]+)\)')
defines = {}
def subst_defines(value):
--
2.5.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 2/6] recipetool: create: improve mapping for autotools program macros
2016-03-08 5:35 [PATCH 0/6] recipetool create improvements Paul Eggleton
2016-03-08 5:36 ` [PATCH 1/6] recipetool: create: be more tolerant of spacing in configure.ac Paul Eggleton
@ 2016-03-08 5:36 ` Paul Eggleton
2016-03-08 5:36 ` [PATCH 3/6] recipetool: create: match *LICENSE* as a license file Paul Eggleton
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-03-08 5:36 UTC (permalink / raw)
To: openembedded-core
Make the following improvements to mapping items specified in
AC_CHECK_PROG, AC_PATH_PROG and AX_WITH_PROG to recipes/classes:
* Produce a map of native recipe -> binary for all binaries currently in
STAGING_BINDIR_NATIVE and use this when mapping items
* Add some more entries to the class map
* Ignore autotools binaries since they are covered by the inherit of
autotools
* Ignore coreutils-native since that would almost always be a bogus
dependency
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 22 +++++++++++++++++--
scripts/lib/recipetool/create_buildsys.py | 36 +++++++++++++++++++++++++------
2 files changed, 50 insertions(+), 8 deletions(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 7560cdf..a77c191 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1,6 +1,6 @@
# Recipe creation tool - create command plugin
#
-# Copyright (C) 2014-2015 Intel Corporation
+# Copyright (C) 2014-2016 Intel Corporation
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as
@@ -44,6 +44,7 @@ class RecipeHandler(object):
recipelibmap = {}
recipeheadermap = {}
recipecmakefilemap = {}
+ recipebinmap = {}
@staticmethod
def load_libmap(d):
@@ -122,6 +123,23 @@ class RecipeHandler(object):
RecipeHandler.recipecmakefilemap[fn] = pn
@staticmethod
+ def load_binmap(d):
+ '''Build up native binary->recipe mapping'''
+ if RecipeHandler.recipebinmap:
+ return
+ sstate_manifests = d.getVar('SSTATE_MANIFESTS', True)
+ staging_bindir_native = d.getVar('STAGING_BINDIR_NATIVE', True)
+ build_arch = d.getVar('BUILD_ARCH', True)
+ fileprefix = 'manifest-%s-' % build_arch
+ for fn in glob.glob(os.path.join(sstate_manifests, '%s*-native.populate_sysroot' % fileprefix)):
+ with open(fn, 'r') as f:
+ pn = os.path.basename(fn).rsplit('.', 1)[0][len(fileprefix):]
+ for line in f:
+ if line.startswith(staging_bindir_native):
+ prog = os.path.basename(line.rstrip())
+ RecipeHandler.recipebinmap[prog] = pn
+
+ @staticmethod
def checkfiles(path, speclist, recursive=False):
results = []
if recursive:
@@ -143,7 +161,7 @@ class RecipeHandler(object):
RecipeHandler.load_libmap(d)
ignorelibs = ['socket']
- ignoredeps = ['gcc-runtime', 'glibc', 'uclibc', 'musl', 'tar-native', 'binutils-native']
+ ignoredeps = ['gcc-runtime', 'glibc', 'uclibc', 'musl', 'tar-native', 'binutils-native', 'coreutils-native']
unmappedpc = []
pcdeps = list(set(pcdeps))
diff --git a/scripts/lib/recipetool/create_buildsys.py b/scripts/lib/recipetool/create_buildsys.py
index 909743b..f84ec3d 100644
--- a/scripts/lib/recipetool/create_buildsys.py
+++ b/scripts/lib/recipetool/create_buildsys.py
@@ -1,6 +1,6 @@
# Recipe creation tool - create command build system handlers
#
-# Copyright (C) 2014 Intel Corporation
+# Copyright (C) 2014-2016 Intel Corporation
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as
@@ -404,14 +404,34 @@ class AutotoolsRecipeHandler(RecipeHandler):
values = {}
inherits = []
- # FIXME this mapping is very thin
+ # Hardcoded map, we also use a dynamic one based on what's in the sysroot
progmap = {'flex': 'flex-native',
'bison': 'bison-native',
'm4': 'm4-native',
'tar': 'tar-native',
- 'ar': 'binutils-native'}
+ 'ar': 'binutils-native',
+ 'ranlib': 'binutils-native',
+ 'ld': 'binutils-native',
+ 'strip': 'binutils-native',
+ 'libtool': '',
+ 'autoconf': '',
+ 'autoheader': '',
+ 'automake': '',
+ 'uname': '',
+ 'rm': '',
+ 'cp': '',
+ 'mv': '',
+ 'find': '',
+ 'awk': '',
+ 'sed': '',
+ }
progclassmap = {'gconftool-2': 'gconf',
- 'pkg-config': 'pkgconfig'}
+ 'pkg-config': 'pkgconfig',
+ 'python': 'pythonnative',
+ 'python3': 'python3native',
+ 'perl': 'perlnative',
+ 'makeinfo': 'texinfo',
+ }
pkg_re = re.compile('PKG_CHECK_MODULES\(\s*\[?[a-zA-Z0-9_]*\]?,\s*\[?([^,\]]*)\]?[),].*')
pkgce_re = re.compile('PKG_CHECK_EXISTS\(\s*\[?([^,\]]*)\]?[),].*')
@@ -462,6 +482,8 @@ class AutotoolsRecipeHandler(RecipeHandler):
deps = []
unmapped = []
+ RecipeHandler.load_binmap(tinfoil.config_data)
+
def process_macro(keyword, value):
for handler in handlers:
if handler.process_macro(srctree, keyword, value, process_value, libdeps, pcdeps, deps, outlines, inherits, values):
@@ -498,10 +520,12 @@ class AutotoolsRecipeHandler(RecipeHandler):
if progclass:
inherits.append(progclass)
else:
- progdep = progmap.get(prog, None)
+ progdep = RecipeHandler.recipebinmap.get(prog, None)
+ if not progdep:
+ progdep = progmap.get(prog, None)
if progdep:
deps.append(progdep)
- else:
+ elif progdep is None:
if not prog.startswith('$'):
unmapped.append(prog)
elif keyword == 'AC_CHECK_LIB':
--
2.5.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 3/6] recipetool: create: match *LICENSE* as a license file
2016-03-08 5:35 [PATCH 0/6] recipetool create improvements Paul Eggleton
2016-03-08 5:36 ` [PATCH 1/6] recipetool: create: be more tolerant of spacing in configure.ac Paul Eggleton
2016-03-08 5:36 ` [PATCH 2/6] recipetool: create: improve mapping for autotools program macros Paul Eggleton
@ 2016-03-08 5:36 ` Paul Eggleton
2016-03-08 5:36 ` [PATCH 4/6] recipetool: create: add license file crunching Paul Eggleton
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-03-08 5:36 UTC (permalink / raw)
To: openembedded-core
For example, this picks up a file named MIT-LICENSE.txt.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index a77c191..def2eea 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -722,7 +722,7 @@ def guess_license(srctree):
md5sums = get_license_md5sums(tinfoil.config_data)
licenses = []
- licspecs = ['LICENSE*', 'COPYING*', '*[Ll]icense*', 'LICENCE*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*']
+ licspecs = ['*LICEN[CS]E*', 'COPYING*', '*[Ll]icense*', 'LEGAL*', '[Ll]egal*', '*GPL*', 'README.lic*', 'COPYRIGHT*', '[Cc]opyright*']
licfiles = []
for root, dirs, files in os.walk(srctree):
for fn in files:
--
2.5.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 4/6] recipetool: create: add license file crunching
2016-03-08 5:35 [PATCH 0/6] recipetool create improvements Paul Eggleton
` (2 preceding siblings ...)
2016-03-08 5:36 ` [PATCH 3/6] recipetool: create: match *LICENSE* as a license file Paul Eggleton
@ 2016-03-08 5:36 ` Paul Eggleton
2016-03-08 5:36 ` [PATCH 5/6] recipetool: create: split npm module dependencies into packages Paul Eggleton
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-03-08 5:36 UTC (permalink / raw)
To: openembedded-core
Matching license texts directly to md5sums only goes so far. Some
licenses make the copyright statement an intrinsic part of the license
statement (e.g. MIT) which of course varies between projects. Also,
people often seem to take standard license texts such as GPLv2 and
reformat them cosmetically - re-wrapping lines at a different width or
changing quoting styles are seemingly popular examples. In order to
match license files to their actual licenses more effectively, "crunch"
out these elements before comparing to an md5sum. (The existing plain
md5sum matching has been left in since it's a shortcut, and our list of
crunched md5sums isn't a complete replacement for it.)
As always, this code isn't providing any guarantees (legal or otherwise)
that it will always get the license correct - as indicated by the
accompanying comments the LICENSE values it writes out to the recipe are
indicative and you should verify them yourself by looking at the
documentation supplied from upstream for the software being built if you
have any concerns.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 77 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 76 insertions(+), 1 deletion(-)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index def2eea..718f2aa 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -25,6 +25,7 @@ import json
import logging
import scriptutils
import urlparse
+import hashlib
logger = logging.getLogger('recipetool')
@@ -717,6 +718,76 @@ def get_license_md5sums(d, static_only=False):
md5sums['54c7042be62e169199200bc6477f04d1'] = 'BSD-3-Clause'
return md5sums
+def crunch_license(licfile):
+ '''
+ Remove non-material text from a license file and then check
+ its md5sum against a known list. This works well for licenses
+ which contain a copyright statement, but is also a useful way
+ to handle people's insistence upon reformatting the license text
+ slightly (with no material difference to the text of the
+ license).
+ '''
+
+ import oe.utils
+
+ # Note: these are carefully constructed!
+ license_title_re = re.compile('^\(?(#+ *)?(The )?.{1,10} [Ll]icen[sc]e( \(.{1,10}\))?\)?:?$')
+ license_statement_re = re.compile('^This (project|software) is( free software)? released under the .{1,10} [Ll]icen[sc]e:?$')
+ copyright_re = re.compile('^(#+)? *Copyright .*$')
+
+ crunched_md5sums = {}
+ # The following two were gleaned from the "forever" npm package
+ crunched_md5sums['0a97f8e4cbaf889d6fa51f84b89a79f6'] = 'ISC'
+ crunched_md5sums['eecf6429523cbc9693547cf2db790b5c'] = 'MIT'
+ # https://github.com/vasi/pixz/blob/master/LICENSE
+ crunched_md5sums['2f03392b40bbe663597b5bd3cc5ebdb9'] = 'BSD-2-Clause'
+ # https://github.com/waffle-gl/waffle/blob/master/LICENSE.txt
+ crunched_md5sums['e72e5dfef0b1a4ca8a3d26a60587db66'] = 'BSD-2-Clause'
+ # https://github.com/spigwitmer/fakeds1963s/blob/master/LICENSE
+ crunched_md5sums['8be76ac6d191671f347ee4916baa637e'] = 'GPLv2'
+ # https://github.com/datto/dattobd/blob/master/COPYING
+ # http://git.savannah.gnu.org/cgit/freetype/freetype2.git/tree/docs/GPLv2.TXT
+ crunched_md5sums['1d65c5ad4bf6489f85f4812bf08ae73d'] = 'GPLv2'
+ # http://www.gnu.org/licenses/old-licenses/gpl-2.0.txt
+ # http://git.neil.brown.name/?p=mdadm.git;a=blob;f=COPYING;h=d159169d1050894d3ea3b98e1c965c4058208fe1;hb=HEAD
+ crunched_md5sums['fb530f66a7a89ce920f0e912b5b66d4b'] = 'GPLv2'
+ # https://github.com/gkos/nrf24/blob/master/COPYING
+ crunched_md5sums['7b6aaa4daeafdfa6ed5443fd2684581b'] = 'GPLv2'
+ # https://github.com/josch09/resetusb/blob/master/COPYING
+ crunched_md5sums['8b8ac1d631a4d220342e83bcf1a1fbc3'] = 'GPLv3'
+ # https://github.com/FFmpeg/FFmpeg/blob/master/COPYING.LGPLv2.1
+ crunched_md5sums['2ea316ed973ae176e502e2297b574bb3'] = 'LGPLv2.1'
+ # unixODBC-2.3.4 COPYING
+ crunched_md5sums['1daebd9491d1e8426900b4fa5a422814'] = 'LGPLv2.1'
+ # https://github.com/FFmpeg/FFmpeg/blob/master/COPYING.LGPLv3
+ crunched_md5sums['2ebfb3bb49b9a48a075cc1425e7f4129'] = 'LGPLv3'
+ lictext = []
+ with open(licfile, 'r') as f:
+ for line in f:
+ # Drop opening statements
+ if copyright_re.match(line):
+ continue
+ elif license_title_re.match(line):
+ continue
+ elif license_statement_re.match(line):
+ continue
+ # Squash spaces, and replace smart quotes, double quotes
+ # and backticks with single quotes
+ line = oe.utils.squashspaces(line.strip()).decode("utf-8")
+ line = line.replace(u"\u2018", "'").replace(u"\u2019", "'").replace(u"\u201c","'").replace(u"\u201d", "'").replace('"', '\'').replace('`', '\'')
+ if line:
+ lictext.append(line)
+
+ m = hashlib.md5()
+ try:
+ m.update(' '.join(lictext))
+ md5val = m.hexdigest()
+ except UnicodeEncodeError:
+ md5val = None
+ lictext = ''
+ license = crunched_md5sums.get(md5val, None)
+ return license, md5val, lictext
+
def guess_license(srctree):
import bb
md5sums = get_license_md5sums(tinfoil.config_data)
@@ -733,7 +804,11 @@ def guess_license(srctree):
licfiles.append(fullpath)
for licfile in licfiles:
md5value = bb.utils.md5_file(licfile)
- license = md5sums.get(md5value, 'Unknown')
+ license = md5sums.get(md5value, None)
+ if not license:
+ license, crunched_md5, lictext = crunch_license(licfile)
+ if not license:
+ license = 'Unknown'
licenses.append((license, os.path.relpath(licfile, srctree), md5value))
# FIXME should we grab at least one source file with a license header and add that too?
--
2.5.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 5/6] recipetool: create: split npm module dependencies into packages
2016-03-08 5:35 [PATCH 0/6] recipetool create improvements Paul Eggleton
` (3 preceding siblings ...)
2016-03-08 5:36 ` [PATCH 4/6] recipetool: create: add license file crunching Paul Eggleton
@ 2016-03-08 5:36 ` Paul Eggleton
2016-03-08 5:36 ` [PATCH 6/6] recipetool: create: check if npm available if npm:// URL specified Paul Eggleton
2016-03-08 22:01 ` [PATCH 0/6] recipetool create improvements Paul Eggleton
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-03-08 5:36 UTC (permalink / raw)
To: openembedded-core
Rather than rolling all of an npm module's dependencies into the same
package, split them into one module per package. Additionally, mark each
package with the appropriate license using the license scanning we
already do, falling back to the license stated in the package.json file
for the module if unknown. This means that the modules and their
licenses can now show up in the manifests for the image.
Additionally we set the main LICENSE value more concretely once we've
calculated the per-package licenses, since we have more information at
that point.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
meta/classes/npm.bbclass | 12 +++++++++++
meta/lib/oe/package.py | 24 +++++++++++++++++++++
scripts/lib/recipetool/create.py | 28 ++++++++++++++++++++++++
scripts/lib/recipetool/create_npm.py | 42 +++++++++++++++++++++++++++++++++---
4 files changed, 103 insertions(+), 3 deletions(-)
diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index be76056..e7f2212 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -18,6 +18,18 @@ npm_do_install() {
cp -a ${S}/* ${D}${libdir}/node_modules/${PN}/ --no-preserve=ownership
}
+python populate_packages_prepend () {
+ instdir = d.expand('${D}${libdir}/node_modules/${PN}')
+ extrapackages = oe.package.npm_split_package_dirs(instdir)
+ pkgnames = extrapackages.keys()
+ d.prependVar('PACKAGES', '%s ' % ' '.join(pkgnames))
+ for pkgname in pkgnames:
+ pkgpath = '${libdir}/node_modules/${PN}/' + extrapackages[pkgname]
+ expanded_pkgname = d.expand(pkgname)
+ d.setVar('FILES_%s' % expanded_pkgname, pkgpath)
+ d.appendVar('RDEPENDS_%s' % d.getVar('PN', True), ' %s' % ' '.join(pkgnames))
+}
+
FILES_${PN} += " \
${libdir}/node_modules/${PN} \
"
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index f176446..7747c5c 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -123,3 +123,27 @@ def read_shlib_providers(d):
shlib_provider[s[0]] = {}
shlib_provider[s[0]][s[1]] = (dep_pkg, s[2])
return shlib_provider
+
+
+def npm_split_package_dirs(pkgdir):
+ """
+ Work out the packages fetched and unpacked by BitBake's npm fetcher
+ """
+ from collections import OrderedDict
+ packages = {}
+ for root, dirs, files in os.walk(pkgdir):
+ if os.path.basename(root) == 'node_modules':
+ for d in dirs:
+ relpth = os.path.relpath(os.path.join(root, d), pkgdir)
+ pkgitems = ['${PN}']
+ for pathitem in relpth.split('/'):
+ if pathitem == 'node_modules':
+ continue
+ pkgitems.append(pathitem)
+ pkgname = '-'.join(pkgitems)
+ packages[pkgname] = relpth
+ # We want the main package for a module sorted *after* its subpackages
+ # (so that it doesn't otherwise steal the files for the subpackage), so
+ # this is a cheap way to do that whilst still having an otherwise
+ # alphabetical sort
+ return OrderedDict((key, packages[key]) for key in sorted(packages, key=lambda pkg: pkg + '~'))
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 718f2aa..43c0784 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -544,6 +544,7 @@ def create_recipe(args):
# Apply the handlers
handled = []
+ handled.append(('license', licvalues))
if args.binary:
classes.append('bin_package')
@@ -815,6 +816,33 @@ def guess_license(srctree):
return licenses
+def split_pkg_licenses(licvalues, packages, outlines, fallback_licenses=None, pn='${PN}'):
+ """
+ Given a list of (license, path, md5sum) as returned by guess_license(),
+ a dict of package name to path mappings, write out a set of
+ package-specific LICENSE values.
+ """
+ pkglicenses = {pn: []}
+ for license, licpath, _ in licvalues:
+ for pkgname, pkgpath in packages.iteritems():
+ if licpath.startswith(pkgpath + '/'):
+ if pkgname in pkglicenses:
+ pkglicenses[pkgname].append(license)
+ else:
+ pkglicenses[pkgname] = [license]
+ break
+ else:
+ # Accumulate on the main package
+ pkglicenses[pn].append(license)
+ outlicenses = {}
+ for pkgname in packages:
+ license = ' '.join(list(set(pkglicenses.get(pkgname, ['Unknown']))))
+ if license == 'Unknown' and pkgname in fallback_licenses:
+ license = fallback_licenses[pkgname]
+ outlines.append('LICENSE_%s = "%s"' % (pkgname, license))
+ outlicenses[pkgname] = license.split()
+ return outlicenses
+
def read_pkgconfig_provides(d):
pkgdatadir = d.getVar('PKGDATA_DIR', True)
pkgmap = {}
diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 0e33cc9..9e18a22 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -17,20 +17,25 @@
import logging
import json
-from recipetool.create import RecipeHandler
+from recipetool.create import RecipeHandler, split_pkg_licenses
logger = logging.getLogger('recipetool')
class NpmRecipeHandler(RecipeHandler):
def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
+ import oe
+
if 'buildsystem' in handled:
return False
+ def read_package_json(fn):
+ with open(fn, 'r') as f:
+ return json.loads(f.read())
+
files = RecipeHandler.checkfiles(srctree, ['package.json'])
if files:
- with open(files[0], 'r') as f:
- data = json.loads(f.read())
+ data = read_package_json(files[0])
if 'name' in data and 'version' in data:
extravalues['PN'] = data['name']
extravalues['PV'] = data['version']
@@ -40,6 +45,37 @@ class NpmRecipeHandler(RecipeHandler):
lines_before.append('SUMMARY = "%s"' % data['description'])
if 'homepage' in data:
lines_before.append('HOMEPAGE = "%s"' % data['homepage'])
+
+ # Split each npm module out to is own package
+ packages = oe.package.npm_split_package_dirs(srctree)
+ for item in handled:
+ if isinstance(item, tuple):
+ if item[0] == 'license':
+ licvalues = item[1]
+ break
+ if licvalues:
+ # Augment the license list with information we have in the packages
+ licenses = {}
+ if 'license' in data:
+ licenses['${PN}'] = data['license']
+ for pkgname, pkgpath in packages.iteritems():
+ pkgfn = os.path.join(srctree, pkgpath, 'package.json')
+ if os.path.exists(pkgfn):
+ pdata = read_package_json(pkgfn)
+ if 'license' in pdata:
+ licenses[pkgname] = pdata['license']
+ # Now write out the package-specific license values
+ packages['${PN}'] = ''
+ pkglicenses = split_pkg_licenses(licvalues, packages, lines_after, licenses)
+ all_licenses = list(set([item for pkglicense in pkglicenses.values() for item in pkglicense]))
+ # Go back and update the LICENSE value since we have a bit more
+ # information than when that was written out (and we know all apply
+ # vs. there being a choice, so we can join them with &)
+ for i, line in enumerate(lines_before):
+ if line.startswith('LICENSE = '):
+ lines_before[i] = 'LICENSE = "%s"' % ' & '.join(all_licenses)
+ break
+
return True
return False
--
2.5.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 6/6] recipetool: create: check if npm available if npm:// URL specified
2016-03-08 5:35 [PATCH 0/6] recipetool create improvements Paul Eggleton
` (4 preceding siblings ...)
2016-03-08 5:36 ` [PATCH 5/6] recipetool: create: split npm module dependencies into packages Paul Eggleton
@ 2016-03-08 5:36 ` Paul Eggleton
2016-03-08 22:01 ` [PATCH 0/6] recipetool create improvements Paul Eggleton
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-03-08 5:36 UTC (permalink / raw)
To: openembedded-core
If the user specifies an npm:// URL then the fetcher needs npm to be
available to run, so check if it's available early rather than failing
later.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
scripts/lib/recipetool/create.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 43c0784..1d48e36 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -355,6 +355,12 @@ def create_recipe(args):
srcuri = rev_re.sub('', srcuri)
tempsrc = tempfile.mkdtemp(prefix='recipetool-')
srctree = tempsrc
+ if fetchuri.startswith('npm://'):
+ # Check if npm is available
+ npm = bb.utils.which(tinfoil.config_data.getVar('PATH', True), 'npm')
+ if not npm:
+ logger.error('npm:// URL requested but npm is not available - you need to either build nodejs-native or install npm using your package manager')
+ sys.exit(1)
logger.info('Fetching %s...' % srcuri)
try:
checksums = scriptutils.fetch_uri(tinfoil.config_data, fetchuri, srctree, srcrev)
--
2.5.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH 0/6] recipetool create improvements
2016-03-08 5:35 [PATCH 0/6] recipetool create improvements Paul Eggleton
` (5 preceding siblings ...)
2016-03-08 5:36 ` [PATCH 6/6] recipetool: create: check if npm available if npm:// URL specified Paul Eggleton
@ 2016-03-08 22:01 ` Paul Eggleton
6 siblings, 0 replies; 8+ messages in thread
From: Paul Eggleton @ 2016-03-08 22:01 UTC (permalink / raw)
To: openembedded-core
On Tue, 08 Mar 2016 18:35:59 Paul Eggleton wrote:
> Improvements to the following areas of recipetool create:
> * License detection
> * Autotools program mapping and spacing in macro regexes
> * npm module handling
>
>
> The following changes since commit 0d02159c8d66bb136f7da2c10fda7d1a57f40cec:
>
> image.bbclass: fix incomplete .rootfs customization (2016-03-07 22:01:16
> +0000)
>
> are available in the git repository at:
>
> git://git.openembedded.org/openembedded-core-contrib paule/recipetool3
>
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paul
> e/recipetool3
>
> Paul Eggleton (6):
> recipetool: create: be more tolerant of spacing in configure.ac
> recipetool: create: improve mapping for autotools program macros
> recipetool: create: match *LICENSE* as a license file
> recipetool: create: add license file crunching
> recipetool: create: split npm module dependencies into packages
> recipetool: create: check if npm available if npm:// URL specified
Hmm, I discovered a bug and a deficiency with patch 5/6, as well as some
problems with the npm fetcher itself. I'll send a new series for the former
today, as well as some patches to the bitbake list for the latter.
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
^ permalink raw reply [flat|nested] 8+ messages in thread