[RFC PATCH 1/2] license.bbclass: split incompatible license check into a helper function

[RFC PATCH 1/2] license.bbclass: split incompatible license check into a helper function

[Alexander Kanavin]

This would help with checking incompatible licenses at do_rootfs step
in image creation, where it is beneficial to pass the license string
directly to the function.

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
---
 meta/classes/license.bbclass | 25 ++++++++++++++-----------
 1 file changed, 14 insertions(+), 11 deletions(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index adca881c85b..648a4d78922 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -287,17 +287,7 @@ def incompatible_license_contains(license, truevalue, falsevalue, d):
     bad_licenses = expand_wildcard_licenses(d, bad_licenses)
     return truevalue if license in bad_licenses else falsevalue
 
-def incompatible_license(d, dont_want_licenses, package=None):
-    """
-    This function checks if a recipe has only incompatible licenses. It also
-    take into consideration 'or' operand.  dont_want_licenses should be passed
-    as canonical (SPDX) names.
-    """
-    import oe.license
-    license = d.getVar("LICENSE_%s" % package) if package else None
-    if not license:
-        license = d.getVar('LICENSE')
-
+def incompatible_pkg_license(d, dont_want_licenses, license):
     # Handles an "or" or two license sets provided by
     # flattened_licenses(), pick one that works if possible.
     def choose_lic_set(a, b):
@@ -311,6 +301,19 @@ def incompatible_license(d, dont_want_licenses, package=None):
     return any(not oe.license.license_ok(canonical_license(d, l), \
                dont_want_licenses) for l in licenses)
 
+def incompatible_license(d, dont_want_licenses, package=None):
+    """
+    This function checks if a recipe has only incompatible licenses. It also
+    take into consideration 'or' operand.  dont_want_licenses should be passed
+    as canonical (SPDX) names.
+    """
+    import oe.license
+    license = d.getVar("LICENSE_%s" % package) if package else None
+    if not license:
+        license = d.getVar('LICENSE')
+
+    return incompatible_pkg_license(d, dont_want_licenses, license)
+
 def check_license_flags(d):
     """
     This function checks if a recipe has any LICENSE_FLAGS that
-- 
2.17.1

[RFC PATCH 2/2] license_image.bbclass: check and reject packages which have incompatible licenses

[Alexander Kanavin]

The use case is setting INCOMPATIBLE_LICENSE per image,
rather than as an awkward, and too strict global setting.

This for example would allow building development images with gplv3 tools,
but production images without them, and checking that nothing gpl3-licensed
gets into the latter.

Examples are provided via the selftest: three scenarios are tested:

- bash is added to the image, with a default gpl3 license; this is rejected
- bash is added to the image, with a "gpl3 & other" license; this is also rejected
- bash is added to the image, with a "gpl3 | other" license; this is accepted, but
only 'other' is added to the license manifest (this was already handled correctly
previously).

Eventually, this would allow deprecating the meta-gplv2 layer, while still
enforcing the no-gpl3 rule where possible and needed.

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
---
 meta/classes/license_image.bbclass            |  2 ++
 .../oeqa/selftest/cases/incompatible_lic.py   | 29 +++++++++++++++++++
 2 files changed, 31 insertions(+)

diff --git a/meta/classes/license_image.bbclass b/meta/classes/license_image.bbclass
index 3f102d0fbc3..e5cb1b8c77d 100644
--- a/meta/classes/license_image.bbclass
+++ b/meta/classes/license_image.bbclass
@@ -47,6 +47,8 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
         for pkg in sorted(pkg_dic):
             if bad_licenses:
                 try:
+                    if incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"]):
+                        bb.fatal("Package %s has an incompatible license %s and cannot be installed into the image." %(pkg, pkg_dic[pkg]["LICENSE"]))
                     (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"]) = \
                         oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"],
                         bad_licenses, canonical_license, d)
diff --git a/meta/lib/oeqa/selftest/cases/incompatible_lic.py b/meta/lib/oeqa/selftest/cases/incompatible_lic.py
index 8fb93af8a89..e80c9783122 100644
--- a/meta/lib/oeqa/selftest/cases/incompatible_lic.py
+++ b/meta/lib/oeqa/selftest/cases/incompatible_lic.py
@@ -39,3 +39,32 @@ class IncompatibleLicenseTests(OESelftestTestCase):
     # INCOMPATIBLE_LICENSE contains this license
     def test_incompatible_nonspdx_license(self):
         self.lic_test('incompatible-nonspdx-license', 'FooLicense', 'FooLicense')
+
+class IncompatibleLicensePerImageTests(OESelftestTestCase):
+    def default_config(self):
+        return """
+IMAGE_INSTALL_append = "bash"
+INCOMPATIBLE_LICENSE_pn-core-image-minimal = "GPL-3.0 LGPL-3.0"
+"""
+
+    def test_bash_default(self):
+        self.write_config(self.default_config())
+        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash has an incompatible license GPLv3+ and cannot be installed into the image."
+
+        result = bitbake('core-image-minimal', ignore_status=True)
+        if error_msg not in result.output:
+            raise AssertionError(result.output)
+
+    def test_bash_and_license(self):
+        self.write_config(self.default_config() + '\nLICENSE_append_pn-bash = " & SomeLicense"')
+        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package bash has an incompatible license GPLv3+ & SomeLicense and cannot be installed into the image."
+
+        result = bitbake('core-image-minimal', ignore_status=True)
+        if error_msg not in result.output:
+            raise AssertionError(result.output)
+
+    def test_bash_or_license(self):
+        self.write_config(self.default_config() + '\nLICENSE_append_pn-bash = " | SomeLicense"')
+
+        bitbake('core-image-minimal')
+
-- 
2.17.1

Re: [RFC PATCH 2/2] license_image.bbclass: check and reject packages which have incompatible licenses

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 5089 bytes --]

Does this obey the whitelist? Can I whitelist a gplv3 package to get it
installed, ideally with a warning rather than silent or error? I don't have
the code in front of me at the moment. I have a class in meta-mentor that
does this sort of check, so wanted to see how the implementations compare.
On a related note,
https://github.com/MentorEmbedded/meta-mentor/blob/master/meta-mentor-common/classes/incompatible-recipe-check.bbclass
+
https://github.com/MentorEmbedded/meta-mentor/blob/95d05dcc12651a7b246f91b240120f92d196b0de/meta-mel-support/recipes-core/packagegroups/packagegroup-tools-benchmark.bb#L10
might be of interest to you in your gplv2 work, perhaps? Specifically
handling the whitelisting.

On Wed, Oct 9, 2019 at 8:45 AM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> The use case is setting INCOMPATIBLE_LICENSE per image,
> rather than as an awkward, and too strict global setting.
>
> This for example would allow building development images with gplv3 tools,
> but production images without them, and checking that nothing gpl3-licensed
> gets into the latter.
>
> Examples are provided via the selftest: three scenarios are tested:
>
> - bash is added to the image, with a default gpl3 license; this is rejected
> - bash is added to the image, with a "gpl3 & other" license; this is also
> rejected
> - bash is added to the image, with a "gpl3 | other" license; this is
> accepted, but
> only 'other' is added to the license manifest (this was already handled
> correctly
> previously).
>
> Eventually, this would allow deprecating the meta-gplv2 layer, while still
> enforcing the no-gpl3 rule where possible and needed.
>
> Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
> ---
>  meta/classes/license_image.bbclass            |  2 ++
>  .../oeqa/selftest/cases/incompatible_lic.py   | 29 +++++++++++++++++++
>  2 files changed, 31 insertions(+)
>
> diff --git a/meta/classes/license_image.bbclass
> b/meta/classes/license_image.bbclass
> index 3f102d0fbc3..e5cb1b8c77d 100644
> --- a/meta/classes/license_image.bbclass
> +++ b/meta/classes/license_image.bbclass
> @@ -47,6 +47,8 @@ def write_license_files(d, license_manifest, pkg_dic,
> rootfs=True):
>          for pkg in sorted(pkg_dic):
>              if bad_licenses:
>                  try:
> +                    if incompatible_pkg_license(d, bad_licenses,
> pkg_dic[pkg]["LICENSE"]):
> +                        bb.fatal("Package %s has an incompatible license
> %s and cannot be installed into the image." %(pkg, pkg_dic[pkg]["LICENSE"]))
>                      (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"]) =
> \
>
>  oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"],
>                          bad_licenses, canonical_license, d)
> diff --git a/meta/lib/oeqa/selftest/cases/incompatible_lic.py
> b/meta/lib/oeqa/selftest/cases/incompatible_lic.py
> index 8fb93af8a89..e80c9783122 100644
> --- a/meta/lib/oeqa/selftest/cases/incompatible_lic.py
> +++ b/meta/lib/oeqa/selftest/cases/incompatible_lic.py
> @@ -39,3 +39,32 @@ class IncompatibleLicenseTests(OESelftestTestCase):
>      # INCOMPATIBLE_LICENSE contains this license
>      def test_incompatible_nonspdx_license(self):
>          self.lic_test('incompatible-nonspdx-license', 'FooLicense',
> 'FooLicense')
> +
> +class IncompatibleLicensePerImageTests(OESelftestTestCase):
> +    def default_config(self):
> +        return """
> +IMAGE_INSTALL_append = "bash"
> +INCOMPATIBLE_LICENSE_pn-core-image-minimal = "GPL-3.0 LGPL-3.0"
> +"""
> +
> +    def test_bash_default(self):
> +        self.write_config(self.default_config())
> +        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package
> bash has an incompatible license GPLv3+ and cannot be installed into the
> image."
> +
> +        result = bitbake('core-image-minimal', ignore_status=True)
> +        if error_msg not in result.output:
> +            raise AssertionError(result.output)
> +
> +    def test_bash_and_license(self):
> +        self.write_config(self.default_config() +
> '\nLICENSE_append_pn-bash = " & SomeLicense"')
> +        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package
> bash has an incompatible license GPLv3+ & SomeLicense and cannot be
> installed into the image."
> +
> +        result = bitbake('core-image-minimal', ignore_status=True)
> +        if error_msg not in result.output:
> +            raise AssertionError(result.output)
> +
> +    def test_bash_or_license(self):
> +        self.write_config(self.default_config() +
> '\nLICENSE_append_pn-bash = " | SomeLicense"')
> +
> +        bitbake('core-image-minimal')
> +
> --
> 2.17.1
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>


-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 6624 bytes --]

Re: [RFC PATCH 2/2] license_image.bbclass: check and reject packages which have incompatible licenses

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 5480 bytes --]

It wouldn't be too hard to add a condition that checks the (image-specific)
whitelist, I just wanted to gather a bit of feedback for the overall idea :)

Alex

On Wed, 9 Oct 2019 at 20:15, Christopher Larson <kergoth@gmail.com> wrote:

> Does this obey the whitelist? Can I whitelist a gplv3 package to get it
> installed, ideally with a warning rather than silent or error? I don't have
> the code in front of me at the moment. I have a class in meta-mentor that
> does this sort of check, so wanted to see how the implementations compare.
> On a related note,
> https://github.com/MentorEmbedded/meta-mentor/blob/master/meta-mentor-common/classes/incompatible-recipe-check.bbclass
> +
> https://github.com/MentorEmbedded/meta-mentor/blob/95d05dcc12651a7b246f91b240120f92d196b0de/meta-mel-support/recipes-core/packagegroups/packagegroup-tools-benchmark.bb#L10
> might be of interest to you in your gplv2 work, perhaps? Specifically
> handling the whitelisting.
>
> On Wed, Oct 9, 2019 at 8:45 AM Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
>
>> The use case is setting INCOMPATIBLE_LICENSE per image,
>> rather than as an awkward, and too strict global setting.
>>
>> This for example would allow building development images with gplv3 tools,
>> but production images without them, and checking that nothing
>> gpl3-licensed
>> gets into the latter.
>>
>> Examples are provided via the selftest: three scenarios are tested:
>>
>> - bash is added to the image, with a default gpl3 license; this is
>> rejected
>> - bash is added to the image, with a "gpl3 & other" license; this is also
>> rejected
>> - bash is added to the image, with a "gpl3 | other" license; this is
>> accepted, but
>> only 'other' is added to the license manifest (this was already handled
>> correctly
>> previously).
>>
>> Eventually, this would allow deprecating the meta-gplv2 layer, while still
>> enforcing the no-gpl3 rule where possible and needed.
>>
>> Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
>> ---
>>  meta/classes/license_image.bbclass            |  2 ++
>>  .../oeqa/selftest/cases/incompatible_lic.py   | 29 +++++++++++++++++++
>>  2 files changed, 31 insertions(+)
>>
>> diff --git a/meta/classes/license_image.bbclass
>> b/meta/classes/license_image.bbclass
>> index 3f102d0fbc3..e5cb1b8c77d 100644
>> --- a/meta/classes/license_image.bbclass
>> +++ b/meta/classes/license_image.bbclass
>> @@ -47,6 +47,8 @@ def write_license_files(d, license_manifest, pkg_dic,
>> rootfs=True):
>>          for pkg in sorted(pkg_dic):
>>              if bad_licenses:
>>                  try:
>> +                    if incompatible_pkg_license(d, bad_licenses,
>> pkg_dic[pkg]["LICENSE"]):
>> +                        bb.fatal("Package %s has an incompatible license
>> %s and cannot be installed into the image." %(pkg, pkg_dic[pkg]["LICENSE"]))
>>                      (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"])
>> = \
>>
>>  oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"],
>>                          bad_licenses, canonical_license, d)
>> diff --git a/meta/lib/oeqa/selftest/cases/incompatible_lic.py
>> b/meta/lib/oeqa/selftest/cases/incompatible_lic.py
>> index 8fb93af8a89..e80c9783122 100644
>> --- a/meta/lib/oeqa/selftest/cases/incompatible_lic.py
>> +++ b/meta/lib/oeqa/selftest/cases/incompatible_lic.py
>> @@ -39,3 +39,32 @@ class IncompatibleLicenseTests(OESelftestTestCase):
>>      # INCOMPATIBLE_LICENSE contains this license
>>      def test_incompatible_nonspdx_license(self):
>>          self.lic_test('incompatible-nonspdx-license', 'FooLicense',
>> 'FooLicense')
>> +
>> +class IncompatibleLicensePerImageTests(OESelftestTestCase):
>> +    def default_config(self):
>> +        return """
>> +IMAGE_INSTALL_append = "bash"
>> +INCOMPATIBLE_LICENSE_pn-core-image-minimal = "GPL-3.0 LGPL-3.0"
>> +"""
>> +
>> +    def test_bash_default(self):
>> +        self.write_config(self.default_config())
>> +        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package
>> bash has an incompatible license GPLv3+ and cannot be installed into the
>> image."
>> +
>> +        result = bitbake('core-image-minimal', ignore_status=True)
>> +        if error_msg not in result.output:
>> +            raise AssertionError(result.output)
>> +
>> +    def test_bash_and_license(self):
>> +        self.write_config(self.default_config() +
>> '\nLICENSE_append_pn-bash = " & SomeLicense"')
>> +        error_msg = "ERROR: core-image-minimal-1.0-r0 do_rootfs: Package
>> bash has an incompatible license GPLv3+ & SomeLicense and cannot be
>> installed into the image."
>> +
>> +        result = bitbake('core-image-minimal', ignore_status=True)
>> +        if error_msg not in result.output:
>> +            raise AssertionError(result.output)
>> +
>> +    def test_bash_or_license(self):
>> +        self.write_config(self.default_config() +
>> '\nLICENSE_append_pn-bash = " | SomeLicense"')
>> +
>> +        bitbake('core-image-minimal')
>> +
>> --
>> 2.17.1
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>
>
> --
> Christopher Larson
> kergoth at gmail dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Senior Software Engineer, Mentor Graphics
>

[-- Attachment #2: Type: text/html, Size: 7204 bytes --]

Re: [RFC PATCH 2/2] license_image.bbclass: check and reject packages which have incompatible licenses

[Mikko.Rapeli]

On Wed, Oct 09, 2019 at 09:41:28PM +0200, Alexander Kanavin wrote:
> It wouldn't be too hard to add a condition that checks the (image-specific)
> whitelist, I just wanted to gather a bit of feedback for the overall idea :)

I like the idea. I'm building images with e.g. GPLv3 forbidden but I enable building
lots of GPLv3 components because they are needed in e.g. ptests. Resulting distro
config snippet is large with lots of lines like:

WHITELIST_GPL-3.0 += "autoconf"
PACKAGE_EXCLUDE += "autoconf-dbg autoconf-staticdev autoconf-dev autoconf-doc autoconf-locale autoconf"
WHITELIST_GPL-3.0 += "ccache"
PACKAGE_EXCLUDE += "ccache-sdktests-dbg ccache-sdktests ccache-dbg ccache-staticdev ccache-dev ccache-doc ccache-locale ccache"

In testing then I start with the pure product image without GPLv3 components and extend it
with extra packages which are needed for the test execution. I have separate product
feature and SW component test phases. Only in the latter case target SW is changed before
test execution.

I want to avoid building separate test images with GPLv3, because in the past
test vs. product images resulted in only the test images being actually tested.

But I can see that having separate product and test images from a single build could
be useful.

Thanks for proposing this patch!

-Mikko