[PATCH] meta: license: fix non-SPDX license being removed from INCOMPATIBLE_LICENSE

[PATCH] meta: license: fix non-SPDX license being removed from INCOMPATIBLE_LICENSE

[Quentin Schulz]

A non-SPDX license (which is not an alias to an SPDX license) cannot
currently be marked as incompatible in INCOMPATIBLE_LICENSE.
In the current state, we take all INCOMPATIBLE_LICENSE and pass them
through expand_wildcard_licenses which is only adding SPDX licenses that
match the glob regexp of what is in INCOMPATIBLE_LICENSE (be it a direct
match to an SPDX license or via an alias).

This does not work well with custom licenses.

E.g.:

foo.bb:
LICENSE = "FooLicense"

conf/local.conf:
INCOMPATIBLE_LICENSE = "FooLicense"

`bitbake foo`

Gives no warning, no error, builds and packages successfully, because
INCOMPATIBLE_LICENSE is basically empty since FooLicense is neither in
SPDXLICENSEMAP nor in SRC_DISTRIBUTE_LICENSES.

Let's add the original licenses to the list returned by
expand_wildcard_licenses to be able to handle the aforementioned case.

INCOMPATIBLE_LICENSE = "FooLicense GPLv2 GPLv3+" used to "resolve" to
"GPLv2 GPLv3". It now resolves to "FooLicense GPLv2 GPLv3 GPLv3+" which
fixes the issue with custom licenses not being in SPDXLICENSEMAP or
SRC_DISTRIBUTE_LICENSES and thus being left out of the blacklisted
licenses.

Signed-off-by: Quentin Schulz <quentin.schulz@streamunlimited.com>
---
 meta/classes/license.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index ed91a4b4db..adca881c85 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -268,7 +268,7 @@ def expand_wildcard_licenses(d, wildcard_licenses):
     wildcards from SPDXLICENSEMAP flags and SRC_DISTRIBUTE_LICENSES values.
     """
     import fnmatch
-    licenses = []
+    licenses = wildcard_licenses[:]
     spdxmapkeys = d.getVarFlags('SPDXLICENSEMAP').keys()
     for wld_lic in wildcard_licenses:
         spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
-- 
2.17.1

Re: [PATCH] meta: license: fix non-SPDX license being removed from INCOMPATIBLE_LICENSE

[Adrian Bunk]

This breaks the build for me:

ERROR: test-image-1.0-r0 do_rootfs: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:license_create_manifest(d)
     0003:
File: '/home/bunk/yoctomusl/yocto-test/sources/poky/meta/classes/license_image.bbclass', lineno: 35, function: license_create_manifest
     0031:            pkg_dic[pkg_name]["LICENSE"] = pkg_dic[pkg_name][pkg_lic_name]
     0032:
     0033:    rootfs_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY'),
     0034:                        d.getVar('IMAGE_NAME'), 'license.manifest')
 *** 0035:    write_license_files(d, rootfs_license_manifest, pkg_dic, rootfs=True)
     0036:}
     0037:
     0038:def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     0039:    import re
File: '/home/bunk/yoctomusl/yocto-test/sources/poky/meta/classes/license_image.bbclass', lineno: 44, function: write_license_files
     0040:    import stat
     0041:
     0042:    bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE") or "").split()
     0043:    bad_licenses = map(lambda l: canonical_license(d, l), bad_licenses)
 *** 0044:    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
     0045:
     0046:    with open(license_manifest, "w") as license_file:
     0047:        for pkg in sorted(pkg_dic):
     0048:            if bad_licenses:
File: '/home/bunk/yoctomusl/yocto-test/sources/poky/meta/classes/license.bbclass', lineno: 271, function: expand_wildcard_licenses
     0267:    Return actual spdx format license names if wildcard used. We expand
     0268:    wildcards from SPDXLICENSEMAP flags and SRC_DISTRIBUTE_LICENSES values.
     0269:    """
     0270:    import fnmatch
 *** 0271:    licenses = wildcard_licenses[:]
     0272:    spdxmapkeys = d.getVarFlags('SPDXLICENSEMAP').keys()
     0273:    for wld_lic in wildcard_licenses:
     0274:        spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
     0275:        licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
Exception: TypeError: 'map' object is not subscriptable

ERROR: test-image-1.0-r0 do_rootfs: 'map' object is not subscriptable
ERROR: test-image-1.0-r0 do_rootfs: Function failed: license_create_manifest
ERROR: Logfile of failure stored in: /home/bunk/yoctomusl/yocto-test/build_test/tmp/work/test-test-linux-musl/test-image/1.0-r0/temp/log.do_rootfs.29928
ERROR: Task (/home/bunk/yoctomusl/yocto-test/sources/meta-test/recipes-test/images/test-image.bb:do_rootfs) failed with exit code '1'


cu
Adrian


On Wed, Jun 05, 2019 at 05:23:52PM +0200, Quentin Schulz wrote:
> A non-SPDX license (which is not an alias to an SPDX license) cannot
> currently be marked as incompatible in INCOMPATIBLE_LICENSE.
> In the current state, we take all INCOMPATIBLE_LICENSE and pass them
> through expand_wildcard_licenses which is only adding SPDX licenses that
> match the glob regexp of what is in INCOMPATIBLE_LICENSE (be it a direct
> match to an SPDX license or via an alias).
> 
> This does not work well with custom licenses.
> 
> E.g.:
> 
> foo.bb:
> LICENSE = "FooLicense"
> 
> conf/local.conf:
> INCOMPATIBLE_LICENSE = "FooLicense"
> 
> `bitbake foo`
> 
> Gives no warning, no error, builds and packages successfully, because
> INCOMPATIBLE_LICENSE is basically empty since FooLicense is neither in
> SPDXLICENSEMAP nor in SRC_DISTRIBUTE_LICENSES.
> 
> Let's add the original licenses to the list returned by
> expand_wildcard_licenses to be able to handle the aforementioned case.
> 
> INCOMPATIBLE_LICENSE = "FooLicense GPLv2 GPLv3+" used to "resolve" to
> "GPLv2 GPLv3". It now resolves to "FooLicense GPLv2 GPLv3 GPLv3+" which
> fixes the issue with custom licenses not being in SPDXLICENSEMAP or
> SRC_DISTRIBUTE_LICENSES and thus being left out of the blacklisted
> licenses.
> 
> Signed-off-by: Quentin Schulz <quentin.schulz@streamunlimited.com>
> ---
>  meta/classes/license.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
> index ed91a4b4db..adca881c85 100644
> --- a/meta/classes/license.bbclass
> +++ b/meta/classes/license.bbclass
> @@ -268,7 +268,7 @@ def expand_wildcard_licenses(d, wildcard_licenses):
>      wildcards from SPDXLICENSEMAP flags and SRC_DISTRIBUTE_LICENSES values.
>      """
>      import fnmatch
> -    licenses = []
> +    licenses = wildcard_licenses[:]
>      spdxmapkeys = d.getVarFlags('SPDXLICENSEMAP').keys()
>      for wld_lic in wildcard_licenses:
>          spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
> -- 
> 2.17.1
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] meta: license: fix non-SPDX license being removed from INCOMPATIBLE_LICENSE

[Quentin Schulz]

Hi Adrian,

On Thu, Jun 06, 2019 at 05:52:36PM +0300, Adrian Bunk wrote:
> This breaks the build for me:
> 
> ERROR: test-image-1.0-r0 do_rootfs: Error executing a python function in exec_python_func() autogenerated:
> 
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
>      0001:
>  *** 0002:license_create_manifest(d)
>      0003:
> File: '/home/bunk/yoctomusl/yocto-test/sources/poky/meta/classes/license_image.bbclass', lineno: 35, function: license_create_manifest
>      0031:            pkg_dic[pkg_name]["LICENSE"] = pkg_dic[pkg_name][pkg_lic_name]
>      0032:
>      0033:    rootfs_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY'),
>      0034:                        d.getVar('IMAGE_NAME'), 'license.manifest')
>  *** 0035:    write_license_files(d, rootfs_license_manifest, pkg_dic, rootfs=True)
>      0036:}
>      0037:
>      0038:def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
>      0039:    import re
> File: '/home/bunk/yoctomusl/yocto-test/sources/poky/meta/classes/license_image.bbclass', lineno: 44, function: write_license_files
>      0040:    import stat
>      0041:
>      0042:    bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE") or "").split()
>      0043:    bad_licenses = map(lambda l: canonical_license(d, l), bad_licenses)
>  *** 0044:    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
>      0045:
>      0046:    with open(license_manifest, "w") as license_file:
>      0047:        for pkg in sorted(pkg_dic):
>      0048:            if bad_licenses:
> File: '/home/bunk/yoctomusl/yocto-test/sources/poky/meta/classes/license.bbclass', lineno: 271, function: expand_wildcard_licenses
>      0267:    Return actual spdx format license names if wildcard used. We expand
>      0268:    wildcards from SPDXLICENSEMAP flags and SRC_DISTRIBUTE_LICENSES values.
>      0269:    """
>      0270:    import fnmatch
>  *** 0271:    licenses = wildcard_licenses[:]
>      0272:    spdxmapkeys = d.getVarFlags('SPDXLICENSEMAP').keys()
>      0273:    for wld_lic in wildcard_licenses:
>      0274:        spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
>      0275:        licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
> Exception: TypeError: 'map' object is not subscriptable
> 
> ERROR: test-image-1.0-r0 do_rootfs: 'map' object is not subscriptable
> ERROR: test-image-1.0-r0 do_rootfs: Function failed: license_create_manifest
> ERROR: Logfile of failure stored in: /home/bunk/yoctomusl/yocto-test/build_test/tmp/work/test-test-linux-musl/test-image/1.0-r0/temp/log.do_rootfs.29928
> ERROR: Task (/home/bunk/yoctomusl/yocto-test/sources/meta-test/recipes-test/images/test-image.bb:do_rootfs) failed with exit code '1'
> 

Funnily enough, you sent the log when I was already working on fixing
the issue :)

I only tested the v1 on package builds and not image builds, my bad.

Thanks for the report though, much appreciated.

I've sent a v2 (and you're Cc'ed to it).

Thanks,
Quentin
-- 
StreamUnlimited Engineering GmbH
High Tech Campus Vienna, Gutheil-Schoder-Gasse 10, 1100 Vienna, Austria
quentin.schulz@streamunlimited.com, www.streamunlimited.com

Re: [PATCH] meta: license: fix non-SPDX license being removed from INCOMPATIBLE_LICENSE

[Adrian Bunk]

On Thu, Jun 06, 2019 at 05:11:37PM +0200, Quentin Schulz wrote:
> Hi Adrian,

Hi Quentin,

>...
> I've sent a v2 (and you're Cc'ed to it).

thanks, I've confirmed this fixes my problem.

> Thanks,
> Quentin

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed