[PATCH 0/7] create_npm: fixes to allow eg Angular2 based applications

[PATCH 0/7] create_npm: fixes to allow eg Angular2 based applications

[Anders Darander]

After the fixes to bitbake, which have gone in to master, there's still some issues
with having devtool add work correctly with a angular2 based applications.

Some of the issues, license strings gets split incorrectly, AND and OR are used in license 
strings in npm, @ and / in npm package names...

There's also a need to be able to install devDependencies from a package.json file.



The following changes since commit 0a1427bf9aeeda6bee2cc0af8da4ea5fd90aef6f:

  recipes: Make use of the new bb.utils.filter() function (2017-03-01 11:17:22 +0000)

are available in the git repository at:

  git://github.com/darander/openembedded-core npm
  https://github.com/darander/openembedded-core/tree/npm

Anders Darander (7):
  create_npm.py: add devDependencies to depends
  create_npm.py: convert MIT/X11 to MIT
  lib/oe/package.py: remove @ from package name
  classes/npm: allow installation of devDependencies
  scripts/lib/create_npm: rewrite see license in eula
  scripts/lib/create_npm: handle AND and OR in licenses
  scripts/lib/create_npm: handle Public Domain licenses

 meta/classes/npm.bbclass             | 13 ++++++++++--
 meta/lib/oe/package.py               |  1 +
 scripts/lib/recipetool/create_npm.py | 38 +++++++++++++++++++++++++++++++-----
 3 files changed, 45 insertions(+), 7 deletions(-)

-- 
2.10.2

[PATCH 1/7] create_npm.py: add devDependencies to depends

[Anders Darander]

Web applications built using e.g. angular2, usually requires that the
packages in devDependencies are available.

Signed-off-by: Anders Darander <anders@chargestorm.se>
---
 scripts/lib/recipetool/create_npm.py | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 3ba6de0..11b2950 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -83,7 +83,7 @@ class NpmRecipeHandler(RecipeHandler):
         extravalues['extrafiles']['lockdown.json'] = tmpfile
         lines_before.append('NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"')
 
-    def _handle_dependencies(self, d, deps, optdeps, lines_before, srctree):
+    def _handle_dependencies(self, d, deps, optdeps, devdeps, lines_before, srctree):
         import scriptutils
         # If this isn't a single module we need to get the dependencies
         # and add them to SRC_URI
@@ -97,6 +97,10 @@ class NpmRecipeHandler(RecipeHandler):
                         depdata = self.get_npm_data(dep, depver, d)
                         if self.check_npm_optional_dependency(depdata):
                             deplist[dep] = depdata
+                    for dep, depver in devdeps.items():
+                        depdata = self.get_npm_data(dep, depver, d)
+                        if self.check_npm_optional_dependency(depdata):
+                            deplist[dep] = depdata
                     for dep, depver in deps.items():
                         depdata = self.get_npm_data(dep, depver, d)
                         deplist[dep] = depdata
@@ -179,8 +183,8 @@ class NpmRecipeHandler(RecipeHandler):
                 if 'homepage' in data:
                     extravalues['HOMEPAGE'] = data['homepage']
 
-                deps, optdeps = self.get_npm_package_dependencies(data)
-                updated = self._handle_dependencies(tinfoil.config_data, deps, optdeps, lines_before, srctree)
+                deps, optdeps, devdeps = self.get_npm_package_dependencies(data)
+                updated = self._handle_dependencies(tinfoil.config_data, deps, optdeps, devdeps, lines_before, srctree)
                 if updated:
                     # We need to redo the license stuff
                     self._replace_license_vars(srctree, lines_before, handled, extravalues, tinfoil.config_data)
@@ -277,15 +281,20 @@ class NpmRecipeHandler(RecipeHandler):
     def get_npm_package_dependencies(self, pdata):
         dependencies = pdata.get('dependencies', {})
         optionalDependencies = pdata.get('optionalDependencies', {})
+        devDependencies = pdata.get('devDependencies', {})
         dependencies.update(optionalDependencies)
+        dependencies.update(devDependencies)
         depsfound = {}
         optdepsfound = {}
+        devdepsfound = {}
         for dep in dependencies:
             if dep in optionalDependencies:
                 optdepsfound[dep] = dependencies[dep]
+            elif dep in devDependencies:
+                devdepsfound[dep] = dependencies[dep]
             else:
                 depsfound[dep] = dependencies[dep]
-        return depsfound, optdepsfound
+        return depsfound, optdepsfound, devdepsfound
 
     # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
     # (split out from _getdependencies())
-- 
2.10.2

Re: [PATCH 1/7] create_npm.py: add devDependencies to depends

[Paul Eggleton]

Hi Anders,

On Thursday, 2 March 2017 6:19:55 AM NZDT Anders Darander wrote:
> Web applications built using e.g. angular2, usually requires that the
> packages in devDependencies are available.
> 
> Signed-off-by: Anders Darander <anders@chargestorm.se>
> ---
>  scripts/lib/recipetool/create_npm.py | 17 +++++++++++++----
>  1 file changed, 13 insertions(+), 4 deletions(-)
> 
> diff --git a/scripts/lib/recipetool/create_npm.py
> b/scripts/lib/recipetool/create_npm.py index 3ba6de0..11b2950 100644
> --- a/scripts/lib/recipetool/create_npm.py
> +++ b/scripts/lib/recipetool/create_npm.py
> @@ -83,7 +83,7 @@ class NpmRecipeHandler(RecipeHandler):
>          extravalues['extrafiles']['lockdown.json'] = tmpfile
>          lines_before.append('NPM_LOCKDOWN :=
> "${THISDIR}/${PN}/lockdown.json"')
> 
> -    def _handle_dependencies(self, d, deps, optdeps, lines_before,
> srctree): +    def _handle_dependencies(self, d, deps, optdeps, devdeps,
> lines_before, srctree): import scriptutils
>          # If this isn't a single module we need to get the dependencies
>          # and add them to SRC_URI
> @@ -97,6 +97,10 @@ class NpmRecipeHandler(RecipeHandler):
>                          depdata = self.get_npm_data(dep, depver, d)
>                          if self.check_npm_optional_dependency(depdata):
>                              deplist[dep] = depdata
> +                    for dep, depver in devdeps.items():
> +                        depdata = self.get_npm_data(dep, depver, d)
> +                        if self.check_npm_optional_dependency(depdata):
> +                            deplist[dep] = depdata
>                      for dep, depver in deps.items():
>                          depdata = self.get_npm_data(dep, depver, d)
>                          deplist[dep] = depdata
> @@ -179,8 +183,8 @@ class NpmRecipeHandler(RecipeHandler):
>                  if 'homepage' in data:
>                      extravalues['HOMEPAGE'] = data['homepage']
> 
> -                deps, optdeps = self.get_npm_package_dependencies(data)
> -                updated = self._handle_dependencies(tinfoil.config_data,
> deps, optdeps, lines_before, srctree) +                deps, optdeps,
> devdeps = self.get_npm_package_dependencies(data) +                updated
> = self._handle_dependencies(tinfoil.config_data, deps, optdeps, devdeps,
> lines_before, srctree) if updated:
>                      # We need to redo the license stuff
>                      self._replace_license_vars(srctree, lines_before,
> handled, extravalues, tinfoil.config_data) @@ -277,15 +281,20 @@ class
> NpmRecipeHandler(RecipeHandler):
>      def get_npm_package_dependencies(self, pdata):
>          dependencies = pdata.get('dependencies', {})
>          optionalDependencies = pdata.get('optionalDependencies', {})
> +        devDependencies = pdata.get('devDependencies', {})
>          dependencies.update(optionalDependencies)
> +        dependencies.update(devDependencies)
>          depsfound = {}
>          optdepsfound = {}
> +        devdepsfound = {}
>          for dep in dependencies:
>              if dep in optionalDependencies:
>                  optdepsfound[dep] = dependencies[dep]
> +            elif dep in devDependencies:
> +                devdepsfound[dep] = dependencies[dep]
>              else:
>                  depsfound[dep] = dependencies[dep]
> -        return depsfound, optdepsfound
> +        return depsfound, optdepsfound, devdepsfound
> 
>      # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
>      # (split out from _getdependencies())

This worries me slightly - I've no familiarity with angular2 but AIUI in 
general the dev dependencies most often aren't needed, and may be onerous to 
pull in and build - should we really be pulling those in unconditionally?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 1/7] create_npm.py: add devDependencies to depends

[Anders Darander]

Hi,

* Paul Eggleton <paul.eggleton@linux.intel.com> [170302 02:44]:

> On Thursday, 2 March 2017 6:19:55 AM NZDT Anders Darander wrote:
> > NpmRecipeHandler(RecipeHandler):
> >      def get_npm_package_dependencies(self, pdata):
> >          dependencies = pdata.get('dependencies', {})
> >          optionalDependencies = pdata.get('optionalDependencies', {})
> > +        devDependencies = pdata.get('devDependencies', {})
> >          dependencies.update(optionalDependencies)
> > +        dependencies.update(devDependencies)
> >          depsfound = {}
> >          optdepsfound = {}
> > +        devdepsfound = {}
> >          for dep in dependencies:
> >              if dep in optionalDependencies:
> >                  optdepsfound[dep] = dependencies[dep]
> > +            elif dep in devDependencies:
> > +                devdepsfound[dep] = dependencies[dep]
> >              else:
> >                  depsfound[dep] = dependencies[dep]
> > -        return depsfound, optdepsfound
> > +        return depsfound, optdepsfound, devdepsfound

> >      # FIXME this is effectively duplicated from lib/bb/fetch2/npm.py
> >      # (split out from _getdependencies())

> This worries me slightly - I've no familiarity with angular2 but AIUI in 
> general the dev dependencies most often aren't needed, and may be onerous to 
> pull in and build - should we really be pulling those in unconditionally?

I fully understand. However, I found no obvious way of doing this
conditionally. At least, the devDependencies won't be used during npm
install, unless you add NPM_INSTALL_DEV = "1" to the recipe in question.

Well, I guess, it could be possible to add a command line switch to
recipetool; though, in that question, I've got a few questions. Is it
possible to add this only for create_npm? (Ie not for generation of
other recipes?) How to add this to devtool (and have it propagate for
npm-packages only?

Cheers,
Anders

-- 
Anders Darander, Senior System Architect
ChargeStorm AB / eStorm AB

[PATCH 2/7] create_npm.py: convert MIT/X11 to MIT

[Anders Darander]

Quite a few npm packages declare MIT/X11 as their license. This is equal to
a pure MIT license.

Signed-off-by: Anders Darander <anders@chargestorm.se>
---
 scripts/lib/recipetool/create_npm.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 11b2950..2816861 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -45,6 +45,8 @@ class NpmRecipeHandler(RecipeHandler):
             license = data['license']
             if isinstance(license, dict):
                 license = license.get('type', None)
+            if 'MIT/X11' in license:
+                license = 'MIT'
         return license
 
     def _shrinkwrap(self, srctree, localfilesdir, extravalues, lines_before):
-- 
2.10.2

Re: [PATCH 2/7] create_npm.py: convert MIT/X11 to MIT

[Khem Raj]

On Wed, Mar 1, 2017 at 9:19 AM, Anders Darander <anders@chargestorm.se> wrote:
> Quite a few npm packages declare MIT/X11 as their license. This is equal to
> a pure MIT license.

it would be good if you could attach a link where this can be referred to.

>
> Signed-off-by: Anders Darander <anders@chargestorm.se>
> ---
>  scripts/lib/recipetool/create_npm.py | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
> index 11b2950..2816861 100644
> --- a/scripts/lib/recipetool/create_npm.py
> +++ b/scripts/lib/recipetool/create_npm.py
> @@ -45,6 +45,8 @@ class NpmRecipeHandler(RecipeHandler):
>              license = data['license']
>              if isinstance(license, dict):
>                  license = license.get('type', None)
> +            if 'MIT/X11' in license:
> +                license = 'MIT'
>          return license
>
>      def _shrinkwrap(self, srctree, localfilesdir, extravalues, lines_before):
> --
> 2.10.2
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 2/7] create_npm.py: convert MIT/X11 to MIT

[Anders Darander]

* Khem Raj <raj.khem@gmail.com> [170303 02:18]:

> On Wed, Mar 1, 2017 at 9:19 AM, Anders Darander <anders@chargestorm.se> wrote:
> > Quite a few npm packages declare MIT/X11 as their license. This is equal to
> > a pure MIT license.

> it would be good if you could attach a link where this can be referred to.

Sure, I'll try to find a good, authorative one...

Unfortunately, the initial quick searches mostly returns wikipedia
pages, as well as number of company sites stating this. I need to search
a little bit more in order to find a better reference.

In worst case, I link to a page with the X11 license text, to ease
manual inspection.

Cheers,
Anders

-- 
Anders Darander, Senior System Architect
ChargeStorm AB / eStorm AB

Re: [PATCH 2/7] create_npm.py: convert MIT/X11 to MIT

[Josef Holzmayr]

Hi Anders,

from what I can see the problem is rooted a bit deeper, and this patch 
(together with 5, 6, 7 from the series) is basically symptomatic:
the package.jsons license field is mostly expected to be spdx-compliant 
(see https://docs.npmjs.com/files/package.json#license). So I think we 
need a concept here to actually handle the complexity it brings, or we 
end up rewriting a boatload of license strings. And doing that in 
create_npm.py just feels wrong to me.

my $.02
-- 
Josef Holzmayr
Software Developer Embedded Systems

Tel: +49 8444 9204-48
Fax: +49 8444 9204-50

R-S-I Elektrotechnik GmbH & Co. KG
Woelkestrasse 11
D-85301 Schweitenkirchen
www.rsi-elektrotechnik.de
———————————————
Amtsgericht Ingolstadt – GmbH: HRB 191328 – KG: HRA 170393
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
Ust-IdNr: DE 128592548

_____________________________________________________________
Amtsgericht Ingolstadt - GmbH: HRB 191328 - KG: HRA 170363
Geschäftsführer: Dr.-Ing. Michael Sorg, Dipl.-Ing. Franz Sorg
USt-IdNr.: DE 128592548

[PATCH 3/7] lib/oe/package.py: remove @ from package name

[Anders Darander]

@ isn't allowed in package names. Angular2 packages often have
@ in their names.

Signed-off-by: Anders Darander <anders@chargestorm.se>
---
 meta/lib/oe/package.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 7953895..0ca41aa 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -150,6 +150,7 @@ def npm_split_package_dirs(pkgdir):
                         continue
                     pkgitems.append(pathitem)
                 pkgname = '-'.join(pkgitems).replace('_', '-')
+                pkgname = pkgname.replace('@', '')
                 pkgfile = os.path.join(root, dn, 'package.json')
                 data = None
                 if os.path.exists(pkgfile):
-- 
2.10.2

[PATCH 4/7] classes/npm: allow installation of devDependencies

[Anders Darander]

Often, eg when using angular2, there's a need to install also
the devDependencies.

The default is to keep the old behaviour, to not install
devDependencies.

Signed-off-by: Anders Darander <anders@chargestorm.se>
---
 meta/classes/npm.bbclass | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/meta/classes/npm.bbclass b/meta/classes/npm.bbclass
index c538040..a69bedb 100644
--- a/meta/classes/npm.bbclass
+++ b/meta/classes/npm.bbclass
@@ -14,6 +14,7 @@ def npm_oe_arch_map(target_arch, d):
     return target_arch
 
 NPM_ARCH ?= "${@npm_oe_arch_map(d.getVar('TARGET_ARCH'), d)}"
+NPM_INSTALL_DEV = "0"
 
 npm_do_compile() {
 	# Copy in any additionally fetched modules
@@ -23,12 +24,20 @@ npm_do_compile() {
 	# changing the home directory to the working directory, the .npmrc will
 	# be created in this directory
 	export HOME=${WORKDIR}
-	npm config set dev false
+	if [  "${NPM_INSTALL_DEV}" = "1" ]; then
+		npm config set dev true
+	else
+		npm config set dev false
+	fi
 	npm set cache ${WORKDIR}/npm_cache
 	# clear cache before every build
 	npm cache clear
 	# Install pkg into ${S} without going to the registry
-	npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --production --no-registry install
+	if [  "${NPM_INSTALL_DEV}" = "1" ]; then
+		npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --no-registry install
+	else
+		npm --arch=${NPM_ARCH} --target_arch=${NPM_ARCH} --production --no-registry install
+	fi
 }
 
 npm_do_install() {
-- 
2.10.2

[PATCH 5/7] scripts/lib/create_npm: rewrite see license in eula

[Anders Darander]

Rewrite the 'SEE LICENSE IN EULA' to a single string (without
spaces), to avoid splitting the string later on.

(Otherwise, each word gets split, and assumed to be a license
on it's own.

Signed-off-by: Anders Darander <anders@chargestorm.se>
---
 scripts/lib/recipetool/create_npm.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 2816861..6a0f3c2 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -45,8 +45,10 @@ class NpmRecipeHandler(RecipeHandler):
             license = data['license']
             if isinstance(license, dict):
                 license = license.get('type', None)
-            if 'MIT/X11' in license:
-                license = 'MIT'
+            if license:
+                license = license.replace('MIT/X11', 'MIT')
+                license = license.replace('SEE LICENSE IN EULA',
+                                          'SEE-LICENSE-IN-EULA')
         return license
 
     def _shrinkwrap(self, srctree, localfilesdir, extravalues, lines_before):
-- 
2.10.2

[PATCH 6/7] scripts/lib/create_npm: handle AND and OR in licenses

[Anders Darander]

Handle npm packages with multiple licenses (AND and OR).
Prior to this, AND and OR were treated as licensed in their
own.

Signed-off-by: Anders Darander <anders@chargestorm.se>
---
 scripts/lib/recipetool/create_npm.py | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 6a0f3c2..89973d9 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -46,6 +46,19 @@ class NpmRecipeHandler(RecipeHandler):
             if isinstance(license, dict):
                 license = license.get('type', None)
             if license:
+                if 'OR' in license:
+                    license = license.replace('OR', '|')
+                    license = license.replace('AND', '&')
+                    license = license.replace(' ', '_')
+                    if not license[0] == '(':
+                        license = '(' + license + ')'
+                    print('LICENSE: {}'.format(license))
+                else:
+                    license = license.replace('AND', '&')
+                    if license[0] == '(':
+                        license = license[1:]
+                    if license[-1] == ')':
+                        license = license[:-1]
                 license = license.replace('MIT/X11', 'MIT')
                 license = license.replace('SEE LICENSE IN EULA',
                                           'SEE-LICENSE-IN-EULA')
@@ -224,7 +237,8 @@ class NpmRecipeHandler(RecipeHandler):
                     packages = OrderedDict((x,y[0]) for x,y in npmpackages.items())
                     packages['${PN}'] = ''
                     pkglicenses = split_pkg_licenses(licvalues, packages, lines_after, licenses)
-                    all_licenses = list(set([item for pkglicense in pkglicenses.values() for item in pkglicense]))
+                    all_licenses = list(set([item.replace('_', ' ') for pkglicense in pkglicenses.values() for item in pkglicense]))
+                    all_licenses.remove('&')
                     # Go back and update the LICENSE value since we have a bit more
                     # information than when that was written out (and we know all apply
                     # vs. there being a choice, so we can join them with &)
-- 
2.10.2

[PATCH 7/7] scripts/lib/create_npm: handle Public Domain licenses

[Anders Darander]

Rewrite Public Domain as PD, as that's what the place holder in
meta/files/common_licenses is called.

Signed-off-by: Anders Darander <anders@chargestorm.se>
---
 scripts/lib/recipetool/create_npm.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/lib/recipetool/create_npm.py b/scripts/lib/recipetool/create_npm.py
index 89973d9..95e9732 100644
--- a/scripts/lib/recipetool/create_npm.py
+++ b/scripts/lib/recipetool/create_npm.py
@@ -60,6 +60,7 @@ class NpmRecipeHandler(RecipeHandler):
                     if license[-1] == ')':
                         license = license[:-1]
                 license = license.replace('MIT/X11', 'MIT')
+                license = license.replace('Public Domain', 'PD')
                 license = license.replace('SEE LICENSE IN EULA',
                                           'SEE-LICENSE-IN-EULA')
         return license
-- 
2.10.2