[PATCH 0/6] Fixes for INCOMPATIBLE_LICENSE handling

[PATCH 0/6] Fixes for INCOMPATIBLE_LICENSE handling

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This fixes a number of bugs spotted while trying to use INCOMPATIBLE_LICENSE
= "*GPLv3", and also while experimenting with a recipe whose LICENSE includes
GPLv3, but whose packages do not. In addition, it reduces the message spam by
switching the license incompatibility messages to debug.

Note: there's still one remaining bug I noticed, the wildcard support which
was added is not very well supported in concert with the host tools exclusion
variables. It asks for an spdx version of "*GPLv3", gets nothing but itself,
and tries to look up whitelists for that string. This is not addressed by this
patch series, but I mention it here for awareness, and also a reminder to
myself to look in to it.

The following changes since commit 6080d1dc719c38fd97c2abd38e0ad938fbf6f452:

  connman: fixed init script so connman can runs over nfs (2013-01-15 08:09:30 +0000)

are available in the git repository at:

  https://github.com/kergoth/oe-core incompatible-license

for you to fetch changes up to 5cb12f38b2ac74ea118b58888a0c37f63917fbfa:

  base.bbclass: improve the incompatible license logic a bit (2013-01-15 12:48:51 -0700)

----------------------------------------------------------------
Christopher Larson (6):
      license.bbclass: check LICENSE_pkg, not LICENSE_pn-pkg
      license: correct re.search/fnmatch indentation
      license.bbclass: cleanly handle invalid licenses in incompatible_license
      linux-firmware: use LICENSEs which can actually be parsed
      base.bbclass: use debug messages for INCOMPATIBLE_LICENSE
      base.bbclass: improve the incompatible license logic a bit

 meta/classes/base.bbclass                                |  9 ++++-----
 meta/classes/license.bbclass                             | 16 +++++++++-------
 meta/recipes-kernel/linux-firmware/linux-firmware_git.bb | 13 ++++++-------
 3 files changed, 19 insertions(+), 19 deletions(-)

Christopher Larson (6):
  license.bbclass: check LICENSE_pkg, not LICENSE_pn-pkg
  license: correct re.search/fnmatch indentation
  license.bbclass: cleanly handle invalid licenses in
    incompatible_license
  linux-firmware: use LICENSEs which can actually be parsed
  base.bbclass: use debug messages for INCOMPATIBLE_LICENSE
  base.bbclass: improve the incompatible license logic a bit

 meta/classes/base.bbclass                                |  9 ++++-----
 meta/classes/license.bbclass                             | 16 +++++++++-------
 meta/recipes-kernel/linux-firmware/linux-firmware_git.bb | 13 ++++++-------
 3 files changed, 19 insertions(+), 19 deletions(-)

-- 
1.8.0.3

[PATCH 1/6] license.bbclass: check LICENSE_pkg, not LICENSE_pn-pkg

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Recipes define the latter, as that's what goes into the binary packages, not
the former.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/license.bbclass | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 68f45f5..c8a8c63 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -215,8 +215,7 @@ def incompatible_license(d, dont_want_licenses, package=None):
     import re
     import oe.license
     from fnmatch import fnmatchcase as fnmatch
-    pn = d.getVar('PN', True)
-    license = d.getVar("LICENSE_%s-%s" % (pn, package), True) if package else None
+    license = d.getVar("LICENSE_%s" % package, True) if package else None
     if not license:
         license = d.getVar('LICENSE', True)
 
-- 
1.8.0.3

[PATCH 2/6] license: correct re.search/fnmatch indentation

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This was causing it to only obey the last of the elements in
INCOMPATIBLE_LICENSE.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/license.bbclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index c8a8c63..cfb816d 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -226,10 +226,10 @@ def incompatible_license(d, dont_want_licenses, package=None):
             # will exclude a trailing '+' character from LICENSE in
             # case INCOMPATIBLE_LICENSE is not a 'X+' license.
             lic = license
-        if not re.search('\+$', dwl):
-            lic = re.sub('\+', '', license)
-        if fnmatch(lic, dwl):
-            return False
+            if not re.search('\+$', dwl):
+                lic = re.sub('\+', '', license)
+            if fnmatch(lic, dwl):
+                return False
         return True
 
     # Handles an "or" or two license sets provided by
-- 
1.8.0.3

[PATCH 3/6] license.bbclass: cleanly handle invalid licenses in incompatible_license

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This gives us an error message (bb.fatal) rather than a traceback due to the
uncaught LicenseError.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/license.bbclass | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index cfb816d..cd18e19 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -237,7 +237,10 @@ def incompatible_license(d, dont_want_licenses, package=None):
     def choose_lic_set(a, b):
         return a if all(license_ok(lic) for lic in a) else b
 
-    licenses=oe.license.flattened_licenses(license, choose_lic_set)
+    try:
+        licenses = oe.license.flattened_licenses(license, choose_lic_set)
+    except oe.license.LicenseError as exc:
+        bb.fatal('%s: %s' % (d.getVar('P', True), exc))
     return any(not license_ok(l) for l in licenses)
 
 def check_license_flags(d):
-- 
1.8.0.3

[PATCH 4/6] linux-firmware: use LICENSEs which can actually be parsed

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

: is an invalid character for our license format.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/recipes-kernel/linux-firmware/linux-firmware_git.bb | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/meta/recipes-kernel/linux-firmware/linux-firmware_git.bb b/meta/recipes-kernel/linux-firmware/linux-firmware_git.bb
index 98a2d07..8700ca4 100644
--- a/meta/recipes-kernel/linux-firmware/linux-firmware_git.bb
+++ b/meta/recipes-kernel/linux-firmware/linux-firmware_git.bb
@@ -22,7 +22,7 @@ LIC_FILES_CHKSUM = "file://LICENSE.radeon;md5=e56b405656593a0c97e478513051ea0e \
 SRCREV = "0d6729f2b0bc6254d3628eb831bc80a36b5a17db"
 PE = "1"
 PV = "0.0+git${SRCPV}"
-PR = "r6"
+PR = "r7"
 
 SRC_URI = "git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git;protocol=git"
 
@@ -51,31 +51,31 @@ do_install() {
 
 PACKAGES =+ "${PN}-ralink ${PN}-sd8686 ${PN}-rtl8192cu linux-firmware-rtl8192ce linux-firmware-rtl8192su ${PN}-wl12xx"
 
-LICENSE_${PN}-ralink = "Firmware:LICENCE.ralink_firmware.txt"
+LICENSE_${PN}-ralink = "Firmware-ralink"
 FILES_${PN}-ralink = " \
   /lib/firmware/rt*.bin \
   /lib/firmware/LICENSE.ralink_firmware.txt \
 "
 
-LICENSE_${PN}-sd8686 = "Firmware:LICENSE.libertas"
+LICENSE_${PN}-sd8686 = "Firmware-libertas"
 FILES_${PN}-sd8686 = " \
   /lib/firmware/libertas/sd8686_v9* \
   /lib/firmware/sd8686* \
   /lib/firmware/LICENCE.libertas \
 "
 
-LICENSE_${PN}-rtl8192cu = "Firmware:LICENCE.rtlwifi_firmware"
+LICENSE_${PN}-rtl8192cu = "Firmware-rtlwifi"
 FILES_${PN}-rtl8192cu = " \
   /lib/firmware/rtlwifi/rtl8192cufw.bin \
   /lib/firmware/rtlwifi/LICENCE.rtlwifi_firmware.txt \
 "
 
-LICENSE_${PN}-rtl8192ce = "Firmware:LICENCE.rtlwifi_firmware"
+LICENSE_${PN}-rtl8192ce = "Firmware-rtlwifi"
 FILES_${PN}-rtl8192ce = " \
   /lib/firmware/rtlwifi/rtl8192cfw.bin \
 "
 
-LICENSE_${PN}-rtl8192su = "Firmware:LICENCE.rtlwifi_firmware"
+LICENSE_${PN}-rtl8192su = "Firmware-rtlwifi"
 FILES_${PN}-rtl8192su = " \
   /lib/firmware/rtlwifi/rtl8712u.bin \
 "
@@ -87,4 +87,3 @@ FILES_${PN}-wl12xx = " \
 "
 
 FILES_${PN} += "/lib/firmware/*"
-
-- 
1.8.0.3

[PATCH 5/6] base.bbclass: use debug messages for INCOMPATIBLE_LICENSE

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Spamming potentially over a screenful of messages due to their
INCOMPATIBLE_LICENSE selection seems a bit unnecessary. If the distro was
configured that way, the user likely won't care, and if the user set it, they
likely know the implications, so make the messages debug instead.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/base.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index f7b6fb8..627b943 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -556,12 +556,12 @@ python () {
                 all_skipped = skipped_pkgs and not unskipped_pkgs
                 if some_skipped:
                     for pkg in skipped_pkgs:
-                        bb.note("SKIPPING the package " + pkg + " at do_rootfs because it's " + recipe_license)
+                        bb.debug(1, "SKIPPING the package " + pkg + " at do_rootfs because it's " + recipe_license)
                         d.setVar('LICENSE_EXCLUSION-' + pkg, 1)
                     for pkg in unskipped_pkgs:
-                        bb.note("INCLUDING the package " + pkg)
+                        bb.debug(1, "INCLUDING the package " + pkg)
                 elif all_skipped or incompatible_license(d, bad_licenses):
-                    bb.note("SKIPPING recipe %s because it's %s" % (pn, recipe_license))
+                    bb.debug(1, "SKIPPING recipe %s because it's %s" % (pn, recipe_license))
                     raise bb.parse.SkipPackage("incompatible with license %s" % recipe_license)
 
     srcuri = d.getVar('SRC_URI', True)
-- 
1.8.0.3

[PATCH 6/6] base.bbclass: improve the incompatible license logic a bit

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

Handle the potential case where the recipe level LICENSE contains an
incompatible license, but all of its emitted packages are in fact compatible.
Without this adjustment, it would skip such a recipe.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 meta/classes/base.bbclass | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 627b943..b427a5e 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -552,9 +552,8 @@ python () {
                         skipped_pkgs.append(pkg)
                     else:
                         unskipped_pkgs.append(pkg)
-                some_skipped = skipped_pkgs and unskipped_pkgs
                 all_skipped = skipped_pkgs and not unskipped_pkgs
-                if some_skipped:
+                if unskipped_pkgs:
                     for pkg in skipped_pkgs:
                         bb.debug(1, "SKIPPING the package " + pkg + " at do_rootfs because it's " + recipe_license)
                         d.setVar('LICENSE_EXCLUSION-' + pkg, 1)
-- 
1.8.0.3

Re: [PATCH 0/6] Fixes for INCOMPATIBLE_LICENSE handling

[Flanagan, Elizabeth]

On Tue, Jan 15, 2013 at 12:22 PM, Christopher Larson <kergoth@gmail.com> wrote:
> From: Christopher Larson <chris_larson@mentor.com>
>
> This fixes a number of bugs spotted while trying to use INCOMPATIBLE_LICENSE
> = "*GPLv3", and also while experimenting with a recipe whose LICENSE includes
> GPLv3, but whose packages do not. In addition, it reduces the message spam by
> switching the license incompatibility messages to debug.
>
> Note: there's still one remaining bug I noticed, the wildcard support which
> was added is not very well supported in concert with the host tools exclusion
> variables. It asks for an spdx version of "*GPLv3", gets nothing but itself,
> and tries to look up whitelists for that string. This is not addressed by this
> patch series, but I mention it here for awareness, and also a reminder to
> myself to look in to it.

Acking the entire series. Thanks Chris!

Acked-by: Beth Flanagan <elizabeth.flanagan@intel.com>

>
> The following changes since commit 6080d1dc719c38fd97c2abd38e0ad938fbf6f452:
>
>   connman: fixed init script so connman can runs over nfs (2013-01-15 08:09:30 +0000)
>
> are available in the git repository at:
>
>   https://github.com/kergoth/oe-core incompatible-license
>
> for you to fetch changes up to 5cb12f38b2ac74ea118b58888a0c37f63917fbfa:
>
>   base.bbclass: improve the incompatible license logic a bit (2013-01-15 12:48:51 -0700)
>
> ----------------------------------------------------------------
> Christopher Larson (6):
>       license.bbclass: check LICENSE_pkg, not LICENSE_pn-pkg
>       license: correct re.search/fnmatch indentation
>       license.bbclass: cleanly handle invalid licenses in incompatible_license
>       linux-firmware: use LICENSEs which can actually be parsed
>       base.bbclass: use debug messages for INCOMPATIBLE_LICENSE
>       base.bbclass: improve the incompatible license logic a bit
>
>  meta/classes/base.bbclass                                |  9 ++++-----
>  meta/classes/license.bbclass                             | 16 +++++++++-------
>  meta/recipes-kernel/linux-firmware/linux-firmware_git.bb | 13 ++++++-------
>  3 files changed, 19 insertions(+), 19 deletions(-)
>
> Christopher Larson (6):
>   license.bbclass: check LICENSE_pkg, not LICENSE_pn-pkg
>   license: correct re.search/fnmatch indentation
>   license.bbclass: cleanly handle invalid licenses in
>     incompatible_license
>   linux-firmware: use LICENSEs which can actually be parsed
>   base.bbclass: use debug messages for INCOMPATIBLE_LICENSE
>   base.bbclass: improve the incompatible license logic a bit
>
>  meta/classes/base.bbclass                                |  9 ++++-----
>  meta/classes/license.bbclass                             | 16 +++++++++-------
>  meta/recipes-kernel/linux-firmware/linux-firmware_git.bb | 13 ++++++-------
>  3 files changed, 19 insertions(+), 19 deletions(-)
>
> --
> 1.8.0.3
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

-- 
Elizabeth Flanagan
Yocto Project
Build and Release

Re: [PATCH 6/6] base.bbclass: improve the incompatible license logic a bit

[Saul Wold]

On 01/15/2013 12:22 PM, Christopher Larson wrote:
> From: Christopher Larson <chris_larson@mentor.com>
>
> Handle the potential case where the recipe level LICENSE contains an
> incompatible license, but all of its emitted packages are in fact compatible.
> Without this adjustment, it would skip such a recipe.
>

This changed seems to have caused a failure in the non-gplv3 build for 
util-linux, which is the only package I know that sets a package LICENSE 
field:
> NOTE: recipe e2fsprogs-1.42.1-r4: task do_compile: Started
> WARNING: util-linux-lscpu has an incompatible license. Excluding from packaging.
> ERROR: Error executing a python function in /srv/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-non-gpl3/build/meta/recipes-core/util-linux/util-linux_2.22.1.bb:
> AttributeError: 'str' object has no attribute 'remove'
>
> ERROR: The stack trace of python calls that resulted in this exception/failure was:
> ERROR:   File "populate_packages", line 227, in <module>
> ERROR:
> ERROR:   File "populate_packages", line 71, in populate_packages
> ERROR:
> ERROR: The code that was being executed was:
> ERROR:      0223:                bb.note("%s contains dangling symlink to %s" % (pkg, l))
> ERROR:      0224:        d.setVar('RDEPENDS_' + pkg, bb.utils.join_deps(rdepends, commasep=False))
> ERROR:      0225:
> ERROR:      0226:
> ERROR:  *** 0227:populate_packages(d)
> ERROR:      0228:
> ERROR: [From file: 'populate_packages', lineno: 227, function: <module>]
> ERROR:      0067:
> ERROR:      0068:    for pkg in packages.split():
> ERROR:      0069:        if d.getVar('LICENSE_EXCLUSION-' + pkg, True):
> ERROR:      0070:            bb.warn("%s has an incompatible license. Excluding from packaging." % pkg)
> ERROR:  *** 0071:            packages.remove(pkg)
> ERROR:      0072:        else:
> ERROR:      0073:            if pkg in package_list:
> ERROR:      0074:                bb.error("%s is listed in PACKAGES multiple times, this leads to packaging errors." % pkg)
> ERROR:      0075:            else:
> ERROR: [From file: 'populate_packages', lineno: 71, function: populate_packages]
> ERROR: Function failed: populate_packages
> ERROR: Logfile of failure stored in: /srv/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-non-gpl3/build/build/tmp/work/i586-poky-linux/util-linux/2.22.1-r3/temp/log.do_package.28036
> NOTE: recipe util-linux-2.22.1-r3: task do_package: Failed


Can you please review this as we are closing m3 on Sunday Midnight.

Thanks
	Sau!


> Signed-off-by: Christopher Larson <chris_larson@mentor.com>
> ---
>   meta/classes/base.bbclass | 3 +--
>   1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 627b943..b427a5e 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -552,9 +552,8 @@ python () {
>                           skipped_pkgs.append(pkg)
>                       else:
>                           unskipped_pkgs.append(pkg)
> -                some_skipped = skipped_pkgs and unskipped_pkgs
>                   all_skipped = skipped_pkgs and not unskipped_pkgs
> -                if some_skipped:
> +                if unskipped_pkgs:
>                       for pkg in skipped_pkgs:
>                           bb.debug(1, "SKIPPING the package " + pkg + " at do_rootfs because it's " + recipe_license)
>                           d.setVar('LICENSE_EXCLUSION-' + pkg, 1)
>

Re: [PATCH 6/6] base.bbclass: improve the incompatible license logic a bit

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 296 bytes --]

On Fri, Jan 18, 2013 at 10:02 PM, Saul Wold <sgw@linux.intel.com> wrote:

> Can you please review this as we are closing m3 on Sunday Midnight.
>

Heh, looks like nobody ever really exercised the package exclusion code.
I'll take a look sunday morning if need be.
-- 
Christopher Larson

[-- Attachment #2: Type: text/html, Size: 615 bytes --]