[OE-core][PATCH 0/4] Incompatible license handling fixes

[OE-core][PATCH 0/4] Incompatible license handling fixes

[Joshua Watt]

I noticed that INCOMPATIBLE_LICENSE was not applying *-locale-*
packages at packaging time like other packages. This meant that if there
were *-locale-* packages with an incompatible license, they would still
be generated, and then the final image would likely fail when the
packages were added to it.

These patches are more less ordered from least controversial to most
controversial :)

I suspect the first two are probably fine optimizations, the third uses
the new library code to make sure INCOMPATIBLE_LICENCE correctly applies
to *-locale-* packages. This same call would likely also need to be
called anywhere else dynamic package generation happens, unless there is
some smart way to detect that.

Finally, the last patch makes it possible to set the LICENSE field of
the generated *-locale-* packages, by making them match
LICENSE:${PN}-locale, if set. This provides a means for recipe writes to
affect the license of the split locale packages in the event that it
doesn't match the recipe LICENSE. This makes logical sense to me, but
please let me know if I'm missing some important assumption


Joshua Watt (4):
  classes-global/license: Move functions to library code
  lib: license: Move package license skip to library
  lib: package: Check for incompatible licenses in locale packages
  lib: package: Copy locale license

 meta/classes-global/base.bbclass          |  42 +----
 meta/classes-global/license.bbclass       | 165 ------------------
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 203 ++++++++++++++++++++++
 meta/lib/oe/package.py                    |   7 +
 5 files changed, 224 insertions(+), 207 deletions(-)

-- 
2.46.2

[OE-core][PATCH 1/4] classes-global/license: Move functions to library code

[Joshua Watt]

Moves several of the functions in license.bbclass to be library code

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/classes-global/base.bbclass          |  10 +-
 meta/classes-global/license.bbclass       | 165 ----------------------
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 163 +++++++++++++++++++++
 4 files changed, 175 insertions(+), 177 deletions(-)

diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index b6940bbb6ff..88b932fc3f0 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -528,8 +528,8 @@ python () {
         bb.fatal('This recipe does not have the LICENSE field set (%s)' % pn)
 
     if bb.data.inherits_class('license', d):
-        check_license_format(d)
-        unmatched_license_flags = check_license_flags(d)
+        oe.license.check_license_format(d)
+        unmatched_license_flags = oe.license.check_license_flags(d)
         if unmatched_license_flags:
             for unmatched in unmatched_license_flags:
                 message = "Has a restricted license '%s' which is not listed in your LICENSE_FLAGS_ACCEPTED." % unmatched
@@ -583,7 +583,7 @@ python () {
             check_license = False
 
         if check_license and bad_licenses:
-            bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
 
             exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
 
@@ -599,7 +599,7 @@ python () {
             for pkg in pkgs:
                 remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
 
-                incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
                 if incompatible_lic:
                     skipped_pkgs[pkg] = incompatible_lic
                 else:
@@ -612,7 +612,7 @@ python () {
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
-                incompatible_lic = incompatible_license(d, bad_licenses)
+                incompatible_lic = oe.license.incompatible_license(d, bad_licenses)
                 for pkg in skipped_pkgs:
                     incompatible_lic += skipped_pkgs[pkg]
                 incompatible_lic = sorted(list(set(incompatible_lic)))
diff --git a/meta/classes-global/license.bbclass b/meta/classes-global/license.bbclass
index 043715fcc36..94dcc7f331c 100644
--- a/meta/classes-global/license.bbclass
+++ b/meta/classes-global/license.bbclass
@@ -255,171 +255,6 @@ def find_license_files(d):
 
     return lic_files_paths
 
-def return_spdx(d, license):
-    """
-    This function returns the spdx mapping of a license if it exists.
-     """
-    return d.getVarFlag('SPDXLICENSEMAP', license)
-
-def canonical_license(d, license):
-    """
-    Return the canonical (SPDX) form of the license if available (so GPLv3
-    becomes GPL-3.0-only) or the passed license if there is no canonical form.
-    """
-    return d.getVarFlag('SPDXLICENSEMAP', license) or license
-
-def expand_wildcard_licenses(d, wildcard_licenses):
-    """
-    There are some common wildcard values users may want to use. Support them
-    here.
-    """
-    licenses = set(wildcard_licenses)
-    mapping = {
-        "AGPL-3.0*" : ["AGPL-3.0-only", "AGPL-3.0-or-later"],
-        "GPL-3.0*" : ["GPL-3.0-only", "GPL-3.0-or-later"],
-        "LGPL-3.0*" : ["LGPL-3.0-only", "LGPL-3.0-or-later"],
-    }
-    for k in mapping:
-        if k in wildcard_licenses:
-            licenses.remove(k)
-            for item in mapping[k]:
-                licenses.add(item)
-
-    for l in licenses:
-        if l in oe.license.obsolete_license_list():
-            bb.fatal("Error, %s is an obsolete license, please use an SPDX reference in INCOMPATIBLE_LICENSE" % l)
-        if "*" in l:
-            bb.fatal("Error, %s is an invalid license wildcard entry" % l)
-
-    return list(licenses)
-
-def incompatible_license_contains(license, truevalue, falsevalue, d):
-    license = canonical_license(d, license)
-    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
-    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
-    return truevalue if license in bad_licenses else falsevalue
-
-def incompatible_pkg_license(d, dont_want_licenses, license):
-    # Handles an "or" or two license sets provided by
-    # flattened_licenses(), pick one that works if possible.
-    def choose_lic_set(a, b):
-        return a if all(oe.license.license_ok(canonical_license(d, lic),
-                            dont_want_licenses) for lic in a) else b
-
-    try:
-        licenses = oe.license.flattened_licenses(license, choose_lic_set)
-    except oe.license.LicenseError as exc:
-        bb.fatal('%s: %s' % (d.getVar('P'), exc))
-
-    incompatible_lic = []
-    for l in licenses:
-        license = canonical_license(d, l)
-        if not oe.license.license_ok(license, dont_want_licenses):
-            incompatible_lic.append(license)
-
-    return sorted(incompatible_lic)
-
-def incompatible_license(d, dont_want_licenses, package=None):
-    """
-    This function checks if a recipe has only incompatible licenses. It also
-    take into consideration 'or' operand.  dont_want_licenses should be passed
-    as canonical (SPDX) names.
-    """
-    import oe.license
-    license = d.getVar("LICENSE:%s" % package) if package else None
-    if not license:
-        license = d.getVar('LICENSE')
-
-    return incompatible_pkg_license(d, dont_want_licenses, license)
-
-def check_license_flags(d):
-    """
-    This function checks if a recipe has any LICENSE_FLAGS that
-    aren't acceptable.
-
-    If it does, it returns the all LICENSE_FLAGS missing from the list
-    of acceptable license flags, or all of the LICENSE_FLAGS if there
-    is no list of acceptable flags.
-
-    If everything is is acceptable, it returns None.
-    """
-
-    def license_flag_matches(flag, acceptlist, pn):
-        """
-        Return True if flag matches something in acceptlist, None if not.
-
-        Before we test a flag against the acceptlist, we append _${PN}
-        to it.  We then try to match that string against the
-        acceptlist.  This covers the normal case, where we expect
-        LICENSE_FLAGS to be a simple string like 'commercial', which
-        the user typically matches exactly in the acceptlist by
-        explicitly appending the package name e.g 'commercial_foo'.
-        If we fail the match however, we then split the flag across
-        '_' and append each fragment and test until we either match or
-        run out of fragments.
-        """
-        flag_pn = ("%s_%s" % (flag, pn))
-        for candidate in acceptlist:
-            if flag_pn == candidate:
-                    return True
-
-        flag_cur = ""
-        flagments = flag_pn.split("_")
-        flagments.pop() # we've already tested the full string
-        for flagment in flagments:
-            if flag_cur:
-                flag_cur += "_"
-            flag_cur += flagment
-            for candidate in acceptlist:
-                if flag_cur == candidate:
-                    return True
-        return False
-
-    def all_license_flags_match(license_flags, acceptlist):
-        """ Return all unmatched flags, None if all flags match """
-        pn = d.getVar('PN')
-        split_acceptlist = acceptlist.split()
-        flags = []
-        for flag in license_flags.split():
-            if not license_flag_matches(flag, split_acceptlist, pn):
-                flags.append(flag)
-        return flags if flags else None
-
-    license_flags = d.getVar('LICENSE_FLAGS')
-    if license_flags:
-        acceptlist = d.getVar('LICENSE_FLAGS_ACCEPTED')
-        if not acceptlist:
-            return license_flags.split()
-        unmatched_flags = all_license_flags_match(license_flags, acceptlist)
-        if unmatched_flags:
-            return unmatched_flags
-    return None
-
-def check_license_format(d):
-    """
-    This function checks if LICENSE is well defined,
-        Validate operators in LICENSES.
-        No spaces are allowed between LICENSES.
-    """
-    pn = d.getVar('PN')
-    licenses = d.getVar('LICENSE')
-    from oe.license import license_operator, license_operator_chars, license_pattern
-
-    elements = list(filter(lambda x: x.strip(), license_operator.split(licenses)))
-    for pos, element in enumerate(elements):
-        if license_pattern.match(element):
-            if pos > 0 and license_pattern.match(elements[pos - 1]):
-                oe.qa.handle_error('license-format',
-                        '%s: LICENSE value "%s" has an invalid format - license names ' \
-                        'must be separated by the following characters to indicate ' \
-                        'the license selection: %s' %
-                        (pn, licenses, license_operator_chars), d)
-        elif not license_operator.match(element):
-            oe.qa.handle_error('license-format',
-                    '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
-                    'in the valid list of separators (%s)' %
-                    (pn, licenses, element, license_operator_chars), d)
-
 SSTATETASKS += "do_populate_lic"
 do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}"
 do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
diff --git a/meta/classes-recipe/license_image.bbclass b/meta/classes-recipe/license_image.bbclass
index 0e953856a63..d2c5ab902ce 100644
--- a/meta/classes-recipe/license_image.bbclass
+++ b/meta/classes-recipe/license_image.bbclass
@@ -58,7 +58,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     import stat
 
     bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE") or "").split()
-    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+    bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
     pkgarchs = d.getVar("SSTATE_ARCHS").split()
     pkgarchs.reverse()
 
@@ -66,17 +66,17 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     with open(license_manifest, "w") as license_file:
         for pkg in sorted(pkg_dic):
             remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-            incompatible_licenses = incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
+            incompatible_licenses = oe.license.incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
             if incompatible_licenses:
                 bb.fatal("Package %s cannot be installed into the image because it has incompatible license(s): %s" %(pkg, ' '.join(incompatible_licenses)))
             else:
-                incompatible_licenses = incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
+                incompatible_licenses = oe.license.incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
                 if incompatible_licenses:
                     oe.qa.handle_error('license-exception', "Including %s with incompatible license(s) %s into the image, because it has been allowed by exception list." %(pkg, ' '.join(incompatible_licenses)), d)
             try:
                 (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"]) = \
                     oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"],
-                    remaining_bad_licenses, canonical_license, d)
+                    remaining_bad_licenses, oe.license.canonical_license, d)
             except oe.license.LicenseError as exc:
                 bb.fatal('%s: %s' % (d.getVar('P'), exc))
 
@@ -144,7 +144,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                 if not os.path.exists(pkg_license_dir ):
                     bb.fatal("Couldn't find license information for dependency %s" % pkg)
 
-                pkg_manifest_licenses = [canonical_license(d, lic) \
+                pkg_manifest_licenses = [oe.license.canonical_license(d, lic) \
                         for lic in pkg_dic[pkg]["LICENSES"]]
 
                 licenses = os.listdir(pkg_license_dir)
@@ -153,7 +153,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                     pkg_rootfs_license = os.path.join(pkg_rootfs_license_dir, lic)
 
                     if re.match(r"^generic_.*$", lic):
-                        generic_lic = canonical_license(d,
+                        generic_lic = oe.license.canonical_license(d,
                                 re.search(r"^generic_(.*)$", lic).group(1))
 
                         # Do not copy generic license into package if isn't
@@ -176,7 +176,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                         if not os.path.exists(pkg_rootfs_license):
                             os.symlink(os.path.join('..', generic_lic_file), pkg_rootfs_license)
                     else:
-                        if (oe.license.license_ok(canonical_license(d,
+                        if (oe.license.license_ok(oe.license.canonical_license(d,
                                 lic), bad_licenses) == False or
                                 os.path.exists(pkg_rootfs_license)):
                             continue
diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
index d9c8d94da47..7739697c401 100644
--- a/meta/lib/oe/license.py
+++ b/meta/lib/oe/license.py
@@ -259,3 +259,166 @@ def apply_pkg_license_exception(pkg, bad_licenses, exceptions):
     """Return remaining bad licenses after removing any package exceptions"""
 
     return [lic for lic in bad_licenses if pkg + ':' + lic not in exceptions]
+
+def return_spdx(d, license):
+    """
+    This function returns the spdx mapping of a license if it exists.
+     """
+    return d.getVarFlag('SPDXLICENSEMAP', license)
+
+def canonical_license(d, license):
+    """
+    Return the canonical (SPDX) form of the license if available (so GPLv3
+    becomes GPL-3.0-only) or the passed license if there is no canonical form.
+    """
+    return d.getVarFlag('SPDXLICENSEMAP', license) or license
+
+def expand_wildcard_licenses(d, wildcard_licenses):
+    """
+    There are some common wildcard values users may want to use. Support them
+    here.
+    """
+    licenses = set(wildcard_licenses)
+    mapping = {
+        "AGPL-3.0*" : ["AGPL-3.0-only", "AGPL-3.0-or-later"],
+        "GPL-3.0*" : ["GPL-3.0-only", "GPL-3.0-or-later"],
+        "LGPL-3.0*" : ["LGPL-3.0-only", "LGPL-3.0-or-later"],
+    }
+    for k in mapping:
+        if k in wildcard_licenses:
+            licenses.remove(k)
+            for item in mapping[k]:
+                licenses.add(item)
+
+    for l in licenses:
+        if l in obsolete_license_list():
+            bb.fatal("Error, %s is an obsolete license, please use an SPDX reference in INCOMPATIBLE_LICENSE" % l)
+        if "*" in l:
+            bb.fatal("Error, %s is an invalid license wildcard entry" % l)
+
+    return list(licenses)
+
+def incompatible_license_contains(license, truevalue, falsevalue, d):
+    license = canonical_license(d, license)
+    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+    return truevalue if license in bad_licenses else falsevalue
+
+def incompatible_pkg_license(d, dont_want_licenses, license):
+    # Handles an "or" or two license sets provided by
+    # flattened_licenses(), pick one that works if possible.
+    def choose_lic_set(a, b):
+        return a if all(license_ok(canonical_license(d, lic),
+                            dont_want_licenses) for lic in a) else b
+
+    try:
+        licenses = flattened_licenses(license, choose_lic_set)
+    except LicenseError as exc:
+        bb.fatal('%s: %s' % (d.getVar('P'), exc))
+
+    incompatible_lic = []
+    for l in licenses:
+        license = canonical_license(d, l)
+        if not license_ok(license, dont_want_licenses):
+            incompatible_lic.append(license)
+
+    return sorted(incompatible_lic)
+
+def incompatible_license(d, dont_want_licenses, package=None):
+    """
+    This function checks if a recipe has only incompatible licenses. It also
+    take into consideration 'or' operand.  dont_want_licenses should be passed
+    as canonical (SPDX) names.
+    """
+    license = d.getVar("LICENSE:%s" % package) if package else None
+    if not license:
+        license = d.getVar('LICENSE')
+
+    return incompatible_pkg_license(d, dont_want_licenses, license)
+
+def check_license_flags(d):
+    """
+    This function checks if a recipe has any LICENSE_FLAGS that
+    aren't acceptable.
+
+    If it does, it returns the all LICENSE_FLAGS missing from the list
+    of acceptable license flags, or all of the LICENSE_FLAGS if there
+    is no list of acceptable flags.
+
+    If everything is is acceptable, it returns None.
+    """
+
+    def license_flag_matches(flag, acceptlist, pn):
+        """
+        Return True if flag matches something in acceptlist, None if not.
+
+        Before we test a flag against the acceptlist, we append _${PN}
+        to it.  We then try to match that string against the
+        acceptlist.  This covers the normal case, where we expect
+        LICENSE_FLAGS to be a simple string like 'commercial', which
+        the user typically matches exactly in the acceptlist by
+        explicitly appending the package name e.g 'commercial_foo'.
+        If we fail the match however, we then split the flag across
+        '_' and append each fragment and test until we either match or
+        run out of fragments.
+        """
+        flag_pn = ("%s_%s" % (flag, pn))
+        for candidate in acceptlist:
+            if flag_pn == candidate:
+                    return True
+
+        flag_cur = ""
+        flagments = flag_pn.split("_")
+        flagments.pop() # we've already tested the full string
+        for flagment in flagments:
+            if flag_cur:
+                flag_cur += "_"
+            flag_cur += flagment
+            for candidate in acceptlist:
+                if flag_cur == candidate:
+                    return True
+        return False
+
+    def all_license_flags_match(license_flags, acceptlist):
+        """ Return all unmatched flags, None if all flags match """
+        pn = d.getVar('PN')
+        split_acceptlist = acceptlist.split()
+        flags = []
+        for flag in license_flags.split():
+            if not license_flag_matches(flag, split_acceptlist, pn):
+                flags.append(flag)
+        return flags if flags else None
+
+    license_flags = d.getVar('LICENSE_FLAGS')
+    if license_flags:
+        acceptlist = d.getVar('LICENSE_FLAGS_ACCEPTED')
+        if not acceptlist:
+            return license_flags.split()
+        unmatched_flags = all_license_flags_match(license_flags, acceptlist)
+        if unmatched_flags:
+            return unmatched_flags
+    return None
+
+def check_license_format(d):
+    """
+    This function checks if LICENSE is well defined,
+        Validate operators in LICENSES.
+        No spaces are allowed between LICENSES.
+    """
+    pn = d.getVar('PN')
+    licenses = d.getVar('LICENSE')
+
+    elements = list(filter(lambda x: x.strip(), license_operator.split(licenses)))
+    for pos, element in enumerate(elements):
+        if license_pattern.match(element):
+            if pos > 0 and license_pattern.match(elements[pos - 1]):
+                oe.qa.handle_error('license-format',
+                        '%s: LICENSE value "%s" has an invalid format - license names ' \
+                        'must be separated by the following characters to indicate ' \
+                        'the license selection: %s' %
+                        (pn, licenses, license_operator_chars), d)
+        elif not license_operator.match(element):
+            oe.qa.handle_error('license-format',
+                    '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
+                    'in the valid list of separators (%s)' %
+                    (pn, licenses, element, license_operator_chars), d)
-- 
2.46.2

[OE-core][PATCH 2/4] lib: license: Move package license skip to library

[Joshua Watt]

Moves the code that skips packages with incompatible licenses to the
library code so that it can be called in other locations

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/classes-global/base.bbclass | 36 ++++------------------------
 meta/lib/oe/license.py           | 40 ++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 32 deletions(-)

diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 88b932fc3f0..b81e61fdb72 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -573,42 +573,14 @@ python () {
 
         bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
 
-        check_license = False if pn.startswith("nativesdk-") else True
-        for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
-              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
-              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
-            if pn.endswith(d.expand(t)):
-                check_license = False
-        if pn.startswith("gcc-source-"):
-            check_license = False
-
-        if check_license and bad_licenses:
-            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
-
-            exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
-
-            for lic_exception in exceptions:
-                if ":" in lic_exception:
-                    lic_exception = lic_exception.split(":")[1]
-                if lic_exception in oe.license.obsolete_license_list():
-                    bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
-
-            pkgs = d.getVar('PACKAGES').split()
-            skipped_pkgs = {}
-            unskipped_pkgs = []
-            for pkg in pkgs:
-                remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-
-                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
-                if incompatible_lic:
-                    skipped_pkgs[pkg] = incompatible_lic
-                else:
-                    unskipped_pkgs.append(pkg)
+        pkgs = d.getVar('PACKAGES').split()
+        if pkgs:
+            skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs)
+            unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
 
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
                     bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
index 7739697c401..75b1e70ba0e 100644
--- a/meta/lib/oe/license.py
+++ b/meta/lib/oe/license.py
@@ -422,3 +422,43 @@ def check_license_format(d):
                     '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
                     'in the valid list of separators (%s)' %
                     (pn, licenses, element, license_operator_chars), d)
+
+def skip_incompatible_package_licenses(d, pkgs):
+    if not pkgs:
+        return {}
+
+    pn = d.getVar("PN")
+    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+
+    check_license = False if pn.startswith("nativesdk-") else True
+    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
+            "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
+            "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
+        if pn.endswith(d.expand(t)):
+            check_license = False
+    if pn.startswith("gcc-source-"):
+        check_license = False
+
+    if not check_license or not bad_licenses:
+        return {}
+
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
+    exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
+
+    for lic_exception in exceptions:
+        if ":" in lic_exception:
+            lic_exception = lic_exception.split(":")[1]
+        if lic_exception in obsolete_license_list():
+            bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
+
+    skipped_pkgs = {}
+    for pkg in pkgs:
+        remaining_bad_licenses = apply_pkg_license_exception(pkg, bad_licenses, exceptions)
+
+        incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+        if incompatible_lic:
+            skipped_pkgs[pkg] = incompatible_lic
+            d.setVar('_exclude_incompatible-' + pkg, ' '.join(incompatible_lic))
+
+    return skipped_pkgs
-- 
2.46.2

[OE-core][PATCH 3/4] lib: package: Check for incompatible licenses in locale packages

[Joshua Watt]

Checks if incompatible licenses are present in the dynamically generated
locale packages

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/lib/oe/package.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index c213a9a3ca6..37fbfe355cb 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -685,9 +685,11 @@ def split_locales(d):
     description = d.getVar('DESCRIPTION') or ""
     locale_section = d.getVar('LOCALE_SECTION')
     mlprefix = d.getVar('MLPREFIX') or ""
+    locale_pkgs = []
     for l in sorted(locales):
         ln = legitimize_package_name(l)
         pkg = pn + '-locale-' + ln
+        locale_pkgs.append(pkg)
         packages.insert(locale_index, pkg)
         locale_index += 1
         files = []
@@ -702,6 +704,7 @@ def split_locales(d):
             d.setVar('SECTION:' + pkg, locale_section)
 
     d.setVar('PACKAGES', ' '.join(packages))
+    oe.license.skip_incompatible_package_licenses(d, locale_pkgs)
 
     # Disabled by RP 18/06/07
     # Wildcards aren't supported in debian
-- 
2.46.2

[OE-core][PATCH 4/4] lib: package: Copy locale license

[Joshua Watt]

When creating split locales, copy the license from LICENSE:${PN}-locale
if set, otherwise leave it unspecified (which will result in falling
back to LICENSE)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/lib/oe/package.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 37fbfe355cb..be8bffbd37b 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -663,6 +663,8 @@ def split_locales(d):
     except ValueError:
         locale_index = len(packages)
 
+    lic = d.getVar("LICENSE:" + pn + "-locale")
+
     localepaths = []
     locales = set()
     for localepath in (d.getVar('LOCALE_PATHS') or "").split():
@@ -700,6 +702,8 @@ def split_locales(d):
         d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn, mlprefix, ln))
         d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l))
         d.setVar('DESCRIPTION:' + pkg, '%s  This package contains language translation files for the %s locale.' % (description, l))
+        if lic:
+            d.setVar('LICENSE:' + pkg, lic)
         if locale_section:
             d.setVar('SECTION:' + pkg, locale_section)
 
-- 
2.46.2

[OE-core][PATCH v2 0/4] Incompatible Licenses in Dynamic Packages

[Joshua Watt]

I noticed that INCOMPATIBLE_LICENSE was not applying *-locale-*
packages at packaging time like other packages. This meant that if there
were *-locale-* packages with an incompatible license, they would still
be generated, and then the final image would likely fail when the
packages were added to it. This is actually not just a problem with
*-locale-* packages, but in fact any packages generated dynamically.

To fix this for all cases, the code to detect incompatible licenses in
packages is moved to a library function, and instead of using the
'_exclude-incompatible-' variable to transfer this knowledge from
base.bbclass to do_package, the library function is called a second time
in do_package. While slightly less efficient, this means that do_package
is checking the actual final list of packages for license exclusions,
make it accurate even when dynamic packages are added.

Finally, the last patch makes it possible to set the LICENSE field of
the generated *-locale-* packages, by making them match
LICENSE:${PN}-locale, if set. This provides a means for recipe writes to
affect the license of the split locale packages in the event that it
doesn't match the recipe LICENSE. This makes logical sense to me, but
please let me know if I'm missing some important assumption

Joshua Watt (4):
  classes-global/license: Move functions to library code
  lib: license: Move package license skip to library
  lib: package: Check incompatible licenses at packaging time
  lib: package: Copy locale license

 meta/classes-global/base.bbclass          |  42 +----
 meta/classes-global/license.bbclass       | 165 ------------------
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 202 ++++++++++++++++++++++
 meta/lib/oe/package.py                    |  10 +-
 5 files changed, 223 insertions(+), 210 deletions(-)

-- 
2.46.2

[OE-core][PATCH v2 1/4] classes-global/license: Move functions to library code

[Joshua Watt]

Moves several of the functions in license.bbclass to be library code

New function dependencies were manually verified using bitbake-dumpsigs
to ensure that bitbake identified the same dependencies even though they
are now in library code (although the new function names mean that the
task hashes still change)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/classes-global/base.bbclass          |  10 +-
 meta/classes-global/license.bbclass       | 165 ----------------------
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 163 +++++++++++++++++++++
 4 files changed, 175 insertions(+), 177 deletions(-)

diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index b6940bbb6ff..88b932fc3f0 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -528,8 +528,8 @@ python () {
         bb.fatal('This recipe does not have the LICENSE field set (%s)' % pn)
 
     if bb.data.inherits_class('license', d):
-        check_license_format(d)
-        unmatched_license_flags = check_license_flags(d)
+        oe.license.check_license_format(d)
+        unmatched_license_flags = oe.license.check_license_flags(d)
         if unmatched_license_flags:
             for unmatched in unmatched_license_flags:
                 message = "Has a restricted license '%s' which is not listed in your LICENSE_FLAGS_ACCEPTED." % unmatched
@@ -583,7 +583,7 @@ python () {
             check_license = False
 
         if check_license and bad_licenses:
-            bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
 
             exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
 
@@ -599,7 +599,7 @@ python () {
             for pkg in pkgs:
                 remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
 
-                incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
                 if incompatible_lic:
                     skipped_pkgs[pkg] = incompatible_lic
                 else:
@@ -612,7 +612,7 @@ python () {
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
-                incompatible_lic = incompatible_license(d, bad_licenses)
+                incompatible_lic = oe.license.incompatible_license(d, bad_licenses)
                 for pkg in skipped_pkgs:
                     incompatible_lic += skipped_pkgs[pkg]
                 incompatible_lic = sorted(list(set(incompatible_lic)))
diff --git a/meta/classes-global/license.bbclass b/meta/classes-global/license.bbclass
index 043715fcc36..94dcc7f331c 100644
--- a/meta/classes-global/license.bbclass
+++ b/meta/classes-global/license.bbclass
@@ -255,171 +255,6 @@ def find_license_files(d):
 
     return lic_files_paths
 
-def return_spdx(d, license):
-    """
-    This function returns the spdx mapping of a license if it exists.
-     """
-    return d.getVarFlag('SPDXLICENSEMAP', license)
-
-def canonical_license(d, license):
-    """
-    Return the canonical (SPDX) form of the license if available (so GPLv3
-    becomes GPL-3.0-only) or the passed license if there is no canonical form.
-    """
-    return d.getVarFlag('SPDXLICENSEMAP', license) or license
-
-def expand_wildcard_licenses(d, wildcard_licenses):
-    """
-    There are some common wildcard values users may want to use. Support them
-    here.
-    """
-    licenses = set(wildcard_licenses)
-    mapping = {
-        "AGPL-3.0*" : ["AGPL-3.0-only", "AGPL-3.0-or-later"],
-        "GPL-3.0*" : ["GPL-3.0-only", "GPL-3.0-or-later"],
-        "LGPL-3.0*" : ["LGPL-3.0-only", "LGPL-3.0-or-later"],
-    }
-    for k in mapping:
-        if k in wildcard_licenses:
-            licenses.remove(k)
-            for item in mapping[k]:
-                licenses.add(item)
-
-    for l in licenses:
-        if l in oe.license.obsolete_license_list():
-            bb.fatal("Error, %s is an obsolete license, please use an SPDX reference in INCOMPATIBLE_LICENSE" % l)
-        if "*" in l:
-            bb.fatal("Error, %s is an invalid license wildcard entry" % l)
-
-    return list(licenses)
-
-def incompatible_license_contains(license, truevalue, falsevalue, d):
-    license = canonical_license(d, license)
-    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
-    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
-    return truevalue if license in bad_licenses else falsevalue
-
-def incompatible_pkg_license(d, dont_want_licenses, license):
-    # Handles an "or" or two license sets provided by
-    # flattened_licenses(), pick one that works if possible.
-    def choose_lic_set(a, b):
-        return a if all(oe.license.license_ok(canonical_license(d, lic),
-                            dont_want_licenses) for lic in a) else b
-
-    try:
-        licenses = oe.license.flattened_licenses(license, choose_lic_set)
-    except oe.license.LicenseError as exc:
-        bb.fatal('%s: %s' % (d.getVar('P'), exc))
-
-    incompatible_lic = []
-    for l in licenses:
-        license = canonical_license(d, l)
-        if not oe.license.license_ok(license, dont_want_licenses):
-            incompatible_lic.append(license)
-
-    return sorted(incompatible_lic)
-
-def incompatible_license(d, dont_want_licenses, package=None):
-    """
-    This function checks if a recipe has only incompatible licenses. It also
-    take into consideration 'or' operand.  dont_want_licenses should be passed
-    as canonical (SPDX) names.
-    """
-    import oe.license
-    license = d.getVar("LICENSE:%s" % package) if package else None
-    if not license:
-        license = d.getVar('LICENSE')
-
-    return incompatible_pkg_license(d, dont_want_licenses, license)
-
-def check_license_flags(d):
-    """
-    This function checks if a recipe has any LICENSE_FLAGS that
-    aren't acceptable.
-
-    If it does, it returns the all LICENSE_FLAGS missing from the list
-    of acceptable license flags, or all of the LICENSE_FLAGS if there
-    is no list of acceptable flags.
-
-    If everything is is acceptable, it returns None.
-    """
-
-    def license_flag_matches(flag, acceptlist, pn):
-        """
-        Return True if flag matches something in acceptlist, None if not.
-
-        Before we test a flag against the acceptlist, we append _${PN}
-        to it.  We then try to match that string against the
-        acceptlist.  This covers the normal case, where we expect
-        LICENSE_FLAGS to be a simple string like 'commercial', which
-        the user typically matches exactly in the acceptlist by
-        explicitly appending the package name e.g 'commercial_foo'.
-        If we fail the match however, we then split the flag across
-        '_' and append each fragment and test until we either match or
-        run out of fragments.
-        """
-        flag_pn = ("%s_%s" % (flag, pn))
-        for candidate in acceptlist:
-            if flag_pn == candidate:
-                    return True
-
-        flag_cur = ""
-        flagments = flag_pn.split("_")
-        flagments.pop() # we've already tested the full string
-        for flagment in flagments:
-            if flag_cur:
-                flag_cur += "_"
-            flag_cur += flagment
-            for candidate in acceptlist:
-                if flag_cur == candidate:
-                    return True
-        return False
-
-    def all_license_flags_match(license_flags, acceptlist):
-        """ Return all unmatched flags, None if all flags match """
-        pn = d.getVar('PN')
-        split_acceptlist = acceptlist.split()
-        flags = []
-        for flag in license_flags.split():
-            if not license_flag_matches(flag, split_acceptlist, pn):
-                flags.append(flag)
-        return flags if flags else None
-
-    license_flags = d.getVar('LICENSE_FLAGS')
-    if license_flags:
-        acceptlist = d.getVar('LICENSE_FLAGS_ACCEPTED')
-        if not acceptlist:
-            return license_flags.split()
-        unmatched_flags = all_license_flags_match(license_flags, acceptlist)
-        if unmatched_flags:
-            return unmatched_flags
-    return None
-
-def check_license_format(d):
-    """
-    This function checks if LICENSE is well defined,
-        Validate operators in LICENSES.
-        No spaces are allowed between LICENSES.
-    """
-    pn = d.getVar('PN')
-    licenses = d.getVar('LICENSE')
-    from oe.license import license_operator, license_operator_chars, license_pattern
-
-    elements = list(filter(lambda x: x.strip(), license_operator.split(licenses)))
-    for pos, element in enumerate(elements):
-        if license_pattern.match(element):
-            if pos > 0 and license_pattern.match(elements[pos - 1]):
-                oe.qa.handle_error('license-format',
-                        '%s: LICENSE value "%s" has an invalid format - license names ' \
-                        'must be separated by the following characters to indicate ' \
-                        'the license selection: %s' %
-                        (pn, licenses, license_operator_chars), d)
-        elif not license_operator.match(element):
-            oe.qa.handle_error('license-format',
-                    '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
-                    'in the valid list of separators (%s)' %
-                    (pn, licenses, element, license_operator_chars), d)
-
 SSTATETASKS += "do_populate_lic"
 do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}"
 do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
diff --git a/meta/classes-recipe/license_image.bbclass b/meta/classes-recipe/license_image.bbclass
index 0e953856a63..d2c5ab902ce 100644
--- a/meta/classes-recipe/license_image.bbclass
+++ b/meta/classes-recipe/license_image.bbclass
@@ -58,7 +58,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     import stat
 
     bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE") or "").split()
-    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+    bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
     pkgarchs = d.getVar("SSTATE_ARCHS").split()
     pkgarchs.reverse()
 
@@ -66,17 +66,17 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     with open(license_manifest, "w") as license_file:
         for pkg in sorted(pkg_dic):
             remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-            incompatible_licenses = incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
+            incompatible_licenses = oe.license.incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
             if incompatible_licenses:
                 bb.fatal("Package %s cannot be installed into the image because it has incompatible license(s): %s" %(pkg, ' '.join(incompatible_licenses)))
             else:
-                incompatible_licenses = incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
+                incompatible_licenses = oe.license.incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
                 if incompatible_licenses:
                     oe.qa.handle_error('license-exception', "Including %s with incompatible license(s) %s into the image, because it has been allowed by exception list." %(pkg, ' '.join(incompatible_licenses)), d)
             try:
                 (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"]) = \
                     oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"],
-                    remaining_bad_licenses, canonical_license, d)
+                    remaining_bad_licenses, oe.license.canonical_license, d)
             except oe.license.LicenseError as exc:
                 bb.fatal('%s: %s' % (d.getVar('P'), exc))
 
@@ -144,7 +144,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                 if not os.path.exists(pkg_license_dir ):
                     bb.fatal("Couldn't find license information for dependency %s" % pkg)
 
-                pkg_manifest_licenses = [canonical_license(d, lic) \
+                pkg_manifest_licenses = [oe.license.canonical_license(d, lic) \
                         for lic in pkg_dic[pkg]["LICENSES"]]
 
                 licenses = os.listdir(pkg_license_dir)
@@ -153,7 +153,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                     pkg_rootfs_license = os.path.join(pkg_rootfs_license_dir, lic)
 
                     if re.match(r"^generic_.*$", lic):
-                        generic_lic = canonical_license(d,
+                        generic_lic = oe.license.canonical_license(d,
                                 re.search(r"^generic_(.*)$", lic).group(1))
 
                         # Do not copy generic license into package if isn't
@@ -176,7 +176,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                         if not os.path.exists(pkg_rootfs_license):
                             os.symlink(os.path.join('..', generic_lic_file), pkg_rootfs_license)
                     else:
-                        if (oe.license.license_ok(canonical_license(d,
+                        if (oe.license.license_ok(oe.license.canonical_license(d,
                                 lic), bad_licenses) == False or
                                 os.path.exists(pkg_rootfs_license)):
                             continue
diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
index d9c8d94da47..7739697c401 100644
--- a/meta/lib/oe/license.py
+++ b/meta/lib/oe/license.py
@@ -259,3 +259,166 @@ def apply_pkg_license_exception(pkg, bad_licenses, exceptions):
     """Return remaining bad licenses after removing any package exceptions"""
 
     return [lic for lic in bad_licenses if pkg + ':' + lic not in exceptions]
+
+def return_spdx(d, license):
+    """
+    This function returns the spdx mapping of a license if it exists.
+     """
+    return d.getVarFlag('SPDXLICENSEMAP', license)
+
+def canonical_license(d, license):
+    """
+    Return the canonical (SPDX) form of the license if available (so GPLv3
+    becomes GPL-3.0-only) or the passed license if there is no canonical form.
+    """
+    return d.getVarFlag('SPDXLICENSEMAP', license) or license
+
+def expand_wildcard_licenses(d, wildcard_licenses):
+    """
+    There are some common wildcard values users may want to use. Support them
+    here.
+    """
+    licenses = set(wildcard_licenses)
+    mapping = {
+        "AGPL-3.0*" : ["AGPL-3.0-only", "AGPL-3.0-or-later"],
+        "GPL-3.0*" : ["GPL-3.0-only", "GPL-3.0-or-later"],
+        "LGPL-3.0*" : ["LGPL-3.0-only", "LGPL-3.0-or-later"],
+    }
+    for k in mapping:
+        if k in wildcard_licenses:
+            licenses.remove(k)
+            for item in mapping[k]:
+                licenses.add(item)
+
+    for l in licenses:
+        if l in obsolete_license_list():
+            bb.fatal("Error, %s is an obsolete license, please use an SPDX reference in INCOMPATIBLE_LICENSE" % l)
+        if "*" in l:
+            bb.fatal("Error, %s is an invalid license wildcard entry" % l)
+
+    return list(licenses)
+
+def incompatible_license_contains(license, truevalue, falsevalue, d):
+    license = canonical_license(d, license)
+    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+    return truevalue if license in bad_licenses else falsevalue
+
+def incompatible_pkg_license(d, dont_want_licenses, license):
+    # Handles an "or" or two license sets provided by
+    # flattened_licenses(), pick one that works if possible.
+    def choose_lic_set(a, b):
+        return a if all(license_ok(canonical_license(d, lic),
+                            dont_want_licenses) for lic in a) else b
+
+    try:
+        licenses = flattened_licenses(license, choose_lic_set)
+    except LicenseError as exc:
+        bb.fatal('%s: %s' % (d.getVar('P'), exc))
+
+    incompatible_lic = []
+    for l in licenses:
+        license = canonical_license(d, l)
+        if not license_ok(license, dont_want_licenses):
+            incompatible_lic.append(license)
+
+    return sorted(incompatible_lic)
+
+def incompatible_license(d, dont_want_licenses, package=None):
+    """
+    This function checks if a recipe has only incompatible licenses. It also
+    take into consideration 'or' operand.  dont_want_licenses should be passed
+    as canonical (SPDX) names.
+    """
+    license = d.getVar("LICENSE:%s" % package) if package else None
+    if not license:
+        license = d.getVar('LICENSE')
+
+    return incompatible_pkg_license(d, dont_want_licenses, license)
+
+def check_license_flags(d):
+    """
+    This function checks if a recipe has any LICENSE_FLAGS that
+    aren't acceptable.
+
+    If it does, it returns the all LICENSE_FLAGS missing from the list
+    of acceptable license flags, or all of the LICENSE_FLAGS if there
+    is no list of acceptable flags.
+
+    If everything is is acceptable, it returns None.
+    """
+
+    def license_flag_matches(flag, acceptlist, pn):
+        """
+        Return True if flag matches something in acceptlist, None if not.
+
+        Before we test a flag against the acceptlist, we append _${PN}
+        to it.  We then try to match that string against the
+        acceptlist.  This covers the normal case, where we expect
+        LICENSE_FLAGS to be a simple string like 'commercial', which
+        the user typically matches exactly in the acceptlist by
+        explicitly appending the package name e.g 'commercial_foo'.
+        If we fail the match however, we then split the flag across
+        '_' and append each fragment and test until we either match or
+        run out of fragments.
+        """
+        flag_pn = ("%s_%s" % (flag, pn))
+        for candidate in acceptlist:
+            if flag_pn == candidate:
+                    return True
+
+        flag_cur = ""
+        flagments = flag_pn.split("_")
+        flagments.pop() # we've already tested the full string
+        for flagment in flagments:
+            if flag_cur:
+                flag_cur += "_"
+            flag_cur += flagment
+            for candidate in acceptlist:
+                if flag_cur == candidate:
+                    return True
+        return False
+
+    def all_license_flags_match(license_flags, acceptlist):
+        """ Return all unmatched flags, None if all flags match """
+        pn = d.getVar('PN')
+        split_acceptlist = acceptlist.split()
+        flags = []
+        for flag in license_flags.split():
+            if not license_flag_matches(flag, split_acceptlist, pn):
+                flags.append(flag)
+        return flags if flags else None
+
+    license_flags = d.getVar('LICENSE_FLAGS')
+    if license_flags:
+        acceptlist = d.getVar('LICENSE_FLAGS_ACCEPTED')
+        if not acceptlist:
+            return license_flags.split()
+        unmatched_flags = all_license_flags_match(license_flags, acceptlist)
+        if unmatched_flags:
+            return unmatched_flags
+    return None
+
+def check_license_format(d):
+    """
+    This function checks if LICENSE is well defined,
+        Validate operators in LICENSES.
+        No spaces are allowed between LICENSES.
+    """
+    pn = d.getVar('PN')
+    licenses = d.getVar('LICENSE')
+
+    elements = list(filter(lambda x: x.strip(), license_operator.split(licenses)))
+    for pos, element in enumerate(elements):
+        if license_pattern.match(element):
+            if pos > 0 and license_pattern.match(elements[pos - 1]):
+                oe.qa.handle_error('license-format',
+                        '%s: LICENSE value "%s" has an invalid format - license names ' \
+                        'must be separated by the following characters to indicate ' \
+                        'the license selection: %s' %
+                        (pn, licenses, license_operator_chars), d)
+        elif not license_operator.match(element):
+            oe.qa.handle_error('license-format',
+                    '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
+                    'in the valid list of separators (%s)' %
+                    (pn, licenses, element, license_operator_chars), d)
-- 
2.46.2

[OE-core][PATCH v2 2/4] lib: license: Move package license skip to library

[Joshua Watt]

Moves the code that skips packages with incompatible licenses to the
library code so that it can be called in other locations

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/classes-global/base.bbclass | 35 ++++------------------------
 meta/lib/oe/license.py           | 39 ++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 31 deletions(-)

diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 88b932fc3f0..5b8663f454d 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -573,37 +573,10 @@ python () {
 
         bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
 
-        check_license = False if pn.startswith("nativesdk-") else True
-        for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
-              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
-              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
-            if pn.endswith(d.expand(t)):
-                check_license = False
-        if pn.startswith("gcc-source-"):
-            check_license = False
-
-        if check_license and bad_licenses:
-            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
-
-            exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
-
-            for lic_exception in exceptions:
-                if ":" in lic_exception:
-                    lic_exception = lic_exception.split(":")[1]
-                if lic_exception in oe.license.obsolete_license_list():
-                    bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
-
-            pkgs = d.getVar('PACKAGES').split()
-            skipped_pkgs = {}
-            unskipped_pkgs = []
-            for pkg in pkgs:
-                remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-
-                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
-                if incompatible_lic:
-                    skipped_pkgs[pkg] = incompatible_lic
-                else:
-                    unskipped_pkgs.append(pkg)
+        pkgs = d.getVar('PACKAGES').split()
+        if pkgs:
+            skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs)
+            unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
 
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
index 7739697c401..866a876d7f0 100644
--- a/meta/lib/oe/license.py
+++ b/meta/lib/oe/license.py
@@ -422,3 +422,42 @@ def check_license_format(d):
                     '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
                     'in the valid list of separators (%s)' %
                     (pn, licenses, element, license_operator_chars), d)
+
+def skip_incompatible_package_licenses(d, pkgs):
+    if not pkgs:
+        return {}
+
+    pn = d.getVar("PN")
+    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+
+    check_license = False if pn.startswith("nativesdk-") else True
+    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
+            "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
+            "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
+        if pn.endswith(d.expand(t)):
+            check_license = False
+    if pn.startswith("gcc-source-"):
+        check_license = False
+
+    if not check_license or not bad_licenses:
+        return {}
+
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
+    exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
+
+    for lic_exception in exceptions:
+        if ":" in lic_exception:
+            lic_exception = lic_exception.split(":")[1]
+        if lic_exception in obsolete_license_list():
+            bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
+
+    skipped_pkgs = {}
+    for pkg in pkgs:
+        remaining_bad_licenses = apply_pkg_license_exception(pkg, bad_licenses, exceptions)
+
+        incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+        if incompatible_lic:
+            skipped_pkgs[pkg] = incompatible_lic
+
+    return skipped_pkgs
-- 
2.46.2

RE: [OE-core][PATCH v2 2/4] lib: license: Move package license skip to library

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Joshua Watt
> Sent: den 23 oktober 2024 23:15
> To: openembedded-core@lists.openembedded.org
> Cc: Joshua Watt <JPEWhacker@gmail.com>
> Subject: [OE-core][PATCH v2 2/4] lib: license: Move package license skip to library
> 
> Moves the code that skips packages with incompatible licenses to the
> library code so that it can be called in other locations
> 
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
>  meta/classes-global/base.bbclass | 35 ++++------------------------
>  meta/lib/oe/license.py           | 39 ++++++++++++++++++++++++++++++++
>  2 files changed, 43 insertions(+), 31 deletions(-)
> 
> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
> index 88b932fc3f0..5b8663f454d 100644
> --- a/meta/classes-global/base.bbclass
> +++ b/meta/classes-global/base.bbclass
> @@ -573,37 +573,10 @@ python () {
> 
>          bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
> 
> -        check_license = False if pn.startswith("nativesdk-") else True
> -        for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
> -              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
> -              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
> -            if pn.endswith(d.expand(t)):
> -                check_license = False
> -        if pn.startswith("gcc-source-"):
> -            check_license = False
> -
> -        if check_license and bad_licenses:
> -            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
> -
> -            exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
> -
> -            for lic_exception in exceptions:
> -                if ":" in lic_exception:
> -                    lic_exception = lic_exception.split(":")[1]
> -                if lic_exception in oe.license.obsolete_license_list():
> -                    bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
> -
> -            pkgs = d.getVar('PACKAGES').split()
> -            skipped_pkgs = {}
> -            unskipped_pkgs = []
> -            for pkg in pkgs:
> -                remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
> -
> -                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
> -                if incompatible_lic:
> -                    skipped_pkgs[pkg] = incompatible_lic
> -                else:
> -                    unskipped_pkgs.append(pkg)
> +        pkgs = d.getVar('PACKAGES').split()
> +        if pkgs:
> +            skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs)
> +            unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
> 
>              if unskipped_pkgs:
>                  for pkg in skipped_pkgs:
> diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
> index 7739697c401..866a876d7f0 100644
> --- a/meta/lib/oe/license.py
> +++ b/meta/lib/oe/license.py
> @@ -422,3 +422,42 @@ def check_license_format(d):
>                      '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
>                      'in the valid list of separators (%s)' %
>                      (pn, licenses, element, license_operator_chars), d)
> +
> +def skip_incompatible_package_licenses(d, pkgs):
> +    if not pkgs:
> +        return {}
> +
> +    pn = d.getVar("PN")
> +    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
> +
> +    check_license = False if pn.startswith("nativesdk-") else True
> +    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
> +            "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
> +            "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
> +        if pn.endswith(d.expand(t)):
> +            check_license = False
> +    if pn.startswith("gcc-source-"):
> +        check_license = False
> +
> +    if not check_license or not bad_licenses:
> +        return {}

For efficiency, it would be better to return early, i.e.:

    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
    if not bad_licenses:
        return {}

    pn = d.getVar("PN")
    if if pn.startswith("nativesdk-") or pn.startswith("gcc-source-"):
        return {}
    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
        if pn.endswith(d.expand(t)):
            return {}

> +
> +    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
> +
> +    exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
> +
> +    for lic_exception in exceptions:
> +        if ":" in lic_exception:
> +            lic_exception = lic_exception.split(":")[1]
> +        if lic_exception in obsolete_license_list():
> +            bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
> +
> +    skipped_pkgs = {}
> +    for pkg in pkgs:
> +        remaining_bad_licenses = apply_pkg_license_exception(pkg, bad_licenses, exceptions)
> +
> +        incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
> +        if incompatible_lic:
> +            skipped_pkgs[pkg] = incompatible_lic
> +
> +    return skipped_pkgs
> --
> 2.46.2

//Peter

RE: [OE-core][PATCH v2 2/4] lib: license: Move package license skip to library

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Peter Kjellerstedt
> Sent: den 24 oktober 2024 01:39
> To: Joshua Watt <JPEWhacker@gmail.com>; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core][PATCH v2 2/4] lib: license: Move package license
> skip to library
> 
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Joshua Watt
> > Sent: den 23 oktober 2024 23:15
> > To: openembedded-core@lists.openembedded.org
> > Cc: Joshua Watt <JPEWhacker@gmail.com>
> > Subject: [OE-core][PATCH v2 2/4] lib: license: Move package license skip to library
> >
> > Moves the code that skips packages with incompatible licenses to the
> > library code so that it can be called in other locations
> >
> > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > ---
> >  meta/classes-global/base.bbclass | 35 ++++------------------------
> >  meta/lib/oe/license.py           | 39 ++++++++++++++++++++++++++++++++
> >  2 files changed, 43 insertions(+), 31 deletions(-)
> >
> > diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
> > index 88b932fc3f0..5b8663f454d 100644
> > --- a/meta/classes-global/base.bbclass
> > +++ b/meta/classes-global/base.bbclass
> > @@ -573,37 +573,10 @@ python () {
> >
> >          bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()

Missed this in the first review pass: you can move bad_licenses down to 
where it is being used.

> >
> > -        check_license = False if pn.startswith("nativesdk-") else True
> > -        for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
> > -              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
> > -              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
> > -            if pn.endswith(d.expand(t)):
> > -                check_license = False
> > -        if pn.startswith("gcc-source-"):
> > -            check_license = False
> > -
> > -        if check_license and bad_licenses:
> > -            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
> > -
> > -            exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
> > -
> > -            for lic_exception in exceptions:
> > -                if ":" in lic_exception:
> > -                    lic_exception = lic_exception.split(":")[1]
> > -                if lic_exception in oe.license.obsolete_license_list():
> > -                    bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
> > -
> > -            pkgs = d.getVar('PACKAGES').split()
> > -            skipped_pkgs = {}
> > -            unskipped_pkgs = []
> > -            for pkg in pkgs:
> > -                remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
> > -
> > -                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
> > -                if incompatible_lic:
> > -                    skipped_pkgs[pkg] = incompatible_lic
> > -                else:
> > -                    unskipped_pkgs.append(pkg)
> > +        pkgs = d.getVar('PACKAGES').split()
> > +        if pkgs:
> > +            skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs)
> > +            unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
> >
> >              if unskipped_pkgs:
> >                  for pkg in skipped_pkgs:
> > diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
> > index 7739697c401..866a876d7f0 100644
> > --- a/meta/lib/oe/license.py
> > +++ b/meta/lib/oe/license.py
> > @@ -422,3 +422,42 @@ def check_license_format(d):
> >                      '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
> >                      'in the valid list of separators (%s)' %
> >                      (pn, licenses, element, license_operator_chars), d)
> > +
> > +def skip_incompatible_package_licenses(d, pkgs):
> > +    if not pkgs:
> > +        return {}
> > +
> > +    pn = d.getVar("PN")
> > +    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
> > +
> > +    check_license = False if pn.startswith("nativesdk-") else True
> > +    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
> > +            "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
> > +            "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
> > +        if pn.endswith(d.expand(t)):
> > +            check_license = False
> > +    if pn.startswith("gcc-source-"):
> > +        check_license = False
> > +
> > +    if not check_license or not bad_licenses:
> > +        return {}
> 
> For efficiency, it would be better to return early, i.e.:
> 
>     bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
>     if not bad_licenses:
>         return {}
> 
>     pn = d.getVar("PN")
>     if if pn.startswith("nativesdk-") or pn.startswith("gcc-source-"):
>         return {}
>     for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
>               "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
>               "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
>         if pn.endswith(d.expand(t)):
>             return {}
> 
> > +
> > +    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
> > +
> > +    exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
> > +
> > +    for lic_exception in exceptions:
> > +        if ":" in lic_exception:
> > +            lic_exception = lic_exception.split(":")[1]
> > +        if lic_exception in obsolete_license_list():
> > +            bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
> > +
> > +    skipped_pkgs = {}
> > +    for pkg in pkgs:
> > +        remaining_bad_licenses = apply_pkg_license_exception(pkg, bad_licenses, exceptions)
> > +
> > +        incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
> > +        if incompatible_lic:
> > +            skipped_pkgs[pkg] = incompatible_lic
> > +
> > +    return skipped_pkgs
> > --
> > 2.46.2
> 
> //Peter

//Peter

[OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time

[Joshua Watt]

Instead of checking for incompatible licenses in the anonymous python
and setting '_exclude_incompatible-', (re)check all the packages in
populate_packages(). This ensures that all packages are processed, even
dynamically generated ones.

The use of the '_exclude-incompatible-' variable set in base.bbclass has
been the mechanism used for per-packages licenses since its it was added
as a feature (although with different names for the variable throughout
history). However, since this misses dynamic packages, calling
oe.license.skip_incompatible_package_licenses() a second time on the
actual final package set is a better solution.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/classes-global/base.bbclass | 1 -
 meta/lib/oe/package.py           | 6 +++---
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 5b8663f454d..b81e61fdb72 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -581,7 +581,6 @@ python () {
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
                     bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index c213a9a3ca6..480408e41e3 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -1447,10 +1447,10 @@ def populate_packages(d):
 
     # Handle excluding packages with incompatible licenses
     package_list = []
+    skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs):
     for pkg in packages:
-        licenses = d.getVar('_exclude_incompatible-' + pkg)
-        if licenses:
-            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses)
+        if pkg in skipped_pkgs:
+            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, skipped_pkgs[pkg])
             oe.qa.handle_error("incompatible-license", msg, d)
         else:
             package_list.append(pkg)
-- 
2.46.2

RE: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Joshua Watt
> Sent: den 23 oktober 2024 23:15
> To: openembedded-core@lists.openembedded.org
> Cc: Joshua Watt <JPEWhacker@gmail.com>
> Subject: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time
> 
> Instead of checking for incompatible licenses in the anonymous python
> and setting '_exclude_incompatible-', (re)check all the packages in
> populate_packages(). This ensures that all packages are processed, even
> dynamically generated ones.
> 
> The use of the '_exclude-incompatible-' variable set in base.bbclass has
> been the mechanism used for per-packages licenses since its it was added

Typo: its it -> it

> as a feature (although with different names for the variable throughout
> history). However, since this misses dynamic packages, calling
> oe.license.skip_incompatible_package_licenses() a second time on the
> actual final package set is a better solution.
> 
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
>  meta/classes-global/base.bbclass | 1 -
>  meta/lib/oe/package.py           | 6 +++---
>  2 files changed, 3 insertions(+), 4 deletions(-)
> 
> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
> index 5b8663f454d..b81e61fdb72 100644
> --- a/meta/classes-global/base.bbclass
> +++ b/meta/classes-global/base.bbclass
> @@ -581,7 +581,6 @@ python () {
>              if unskipped_pkgs:
>                  for pkg in skipped_pkgs:
>                      bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
> -                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
>                  for pkg in unskipped_pkgs:
>                      bb.debug(1, "Including the package %s" % pkg)
>              else:
> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> index c213a9a3ca6..480408e41e3 100644
> --- a/meta/lib/oe/package.py
> +++ b/meta/lib/oe/package.py
> @@ -1447,10 +1447,10 @@ def populate_packages(d):
> 
>      # Handle excluding packages with incompatible licenses
>      package_list = []
> +    skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs):
>      for pkg in packages:
> -        licenses = d.getVar('_exclude_incompatible-' + pkg)
> -        if licenses:
> -            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses)
> +        if pkg in skipped_pkgs:
> +            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, skipped_pkgs[pkg])
>              oe.qa.handle_error("incompatible-license", msg, d)
>          else:
>              package_list.append(pkg)
> --
> 2.46.2

There is still a reference to _exclude_incompatible in package.bbclass. 
Not sure what to do about it (it has to do with variable dependencies 
to trigger do_package to re-run for recipes that are affected by changes 
to INCOMPATIBLE_LICENSE). 

//Peter

RE: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Joshua Watt
> Sent: den 23 oktober 2024 23:15
> To: openembedded-core@lists.openembedded.org
> Cc: Joshua Watt <JPEWhacker@gmail.com>
> Subject: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time
> 
> Instead of checking for incompatible licenses in the anonymous python
> and setting '_exclude_incompatible-', (re)check all the packages in
> populate_packages(). This ensures that all packages are processed, even
> dynamically generated ones.
> 
> The use of the '_exclude-incompatible-' variable set in base.bbclass has
> been the mechanism used for per-packages licenses since its it was added
> as a feature (although with different names for the variable throughout
> history). However, since this misses dynamic packages, calling
> oe.license.skip_incompatible_package_licenses() a second time on the
> actual final package set is a better solution.
> 
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
>  meta/classes-global/base.bbclass | 1 -
>  meta/lib/oe/package.py           | 6 +++---
>  2 files changed, 3 insertions(+), 4 deletions(-)
> 
> diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
> index 5b8663f454d..b81e61fdb72 100644
> --- a/meta/classes-global/base.bbclass
> +++ b/meta/classes-global/base.bbclass
> @@ -581,7 +581,6 @@ python () {
>              if unskipped_pkgs:
>                  for pkg in skipped_pkgs:
>                      bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
> -                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
>                  for pkg in unskipped_pkgs:
>                      bb.debug(1, "Including the package %s" % pkg)
>              else:
> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> index c213a9a3ca6..480408e41e3 100644
> --- a/meta/lib/oe/package.py
> +++ b/meta/lib/oe/package.py
> @@ -1447,10 +1447,10 @@ def populate_packages(d):
> 
>      # Handle excluding packages with incompatible licenses
>      package_list = []
> +    skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs):

Umm, how well did you test this? ;)
The Python interpreter is not too happy about the trailing colon on the line above...

>      for pkg in packages:
> -        licenses = d.getVar('_exclude_incompatible-' + pkg)
> -        if licenses:
> -            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses)
> +        if pkg in skipped_pkgs:
> +            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, skipped_pkgs[pkg])
>              oe.qa.handle_error("incompatible-license", msg, d)
>          else:
>              package_list.append(pkg)
> --
> 2.46.2

//Peter

Re: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time

[Joshua Watt]

On Thu, Oct 24, 2024 at 4:03 AM Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
>
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Joshua Watt
> > Sent: den 23 oktober 2024 23:15
> > To: openembedded-core@lists.openembedded.org
> > Cc: Joshua Watt <JPEWhacker@gmail.com>
> > Subject: [OE-core][PATCH v2 3/4] lib: package: Check incompatible licenses at packaging time
> >
> > Instead of checking for incompatible licenses in the anonymous python
> > and setting '_exclude_incompatible-', (re)check all the packages in
> > populate_packages(). This ensures that all packages are processed, even
> > dynamically generated ones.
> >
> > The use of the '_exclude-incompatible-' variable set in base.bbclass has
> > been the mechanism used for per-packages licenses since its it was added
> > as a feature (although with different names for the variable throughout
> > history). However, since this misses dynamic packages, calling
> > oe.license.skip_incompatible_package_licenses() a second time on the
> > actual final package set is a better solution.
> >
> > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > ---
> >  meta/classes-global/base.bbclass | 1 -
> >  meta/lib/oe/package.py           | 6 +++---
> >  2 files changed, 3 insertions(+), 4 deletions(-)
> >
> > diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
> > index 5b8663f454d..b81e61fdb72 100644
> > --- a/meta/classes-global/base.bbclass
> > +++ b/meta/classes-global/base.bbclass
> > @@ -581,7 +581,6 @@ python () {
> >              if unskipped_pkgs:
> >                  for pkg in skipped_pkgs:
> >                      bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
> > -                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
> >                  for pkg in unskipped_pkgs:
> >                      bb.debug(1, "Including the package %s" % pkg)
> >              else:
> > diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> > index c213a9a3ca6..480408e41e3 100644
> > --- a/meta/lib/oe/package.py
> > +++ b/meta/lib/oe/package.py
> > @@ -1447,10 +1447,10 @@ def populate_packages(d):
> >
> >      # Handle excluding packages with incompatible licenses
> >      package_list = []
> > +    skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs):
>
> Umm, how well did you test this? ;)
> The Python interpreter is not too happy about the trailing colon on the line above...

Yep, because I found that, fixed it, and forgot to squash the fix in
before I submitted

I'll send a V3

>
> >      for pkg in packages:
> > -        licenses = d.getVar('_exclude_incompatible-' + pkg)
> > -        if licenses:
> > -            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses)
> > +        if pkg in skipped_pkgs:
> > +            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, skipped_pkgs[pkg])
> >              oe.qa.handle_error("incompatible-license", msg, d)
> >          else:
> >              package_list.append(pkg)
> > --
> > 2.46.2
>
> //Peter
>

[OE-core][PATCH v2 4/4] lib: package: Copy locale license

[Joshua Watt]

When creating split locales, copy the license from LICENSE:${PN}-locale
if set, otherwise leave it unspecified (which will result in falling
back to LICENSE)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/lib/oe/package.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 480408e41e3..236a0edbcb2 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -663,6 +663,8 @@ def split_locales(d):
     except ValueError:
         locale_index = len(packages)
 
+    lic = d.getVar("LICENSE:" + pn + "-locale")
+
     localepaths = []
     locales = set()
     for localepath in (d.getVar('LOCALE_PATHS') or "").split():
@@ -698,6 +700,8 @@ def split_locales(d):
         d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn, mlprefix, ln))
         d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l))
         d.setVar('DESCRIPTION:' + pkg, '%s  This package contains language translation files for the %s locale.' % (description, l))
+        if lic:
+            d.setVar('LICENSE:' + pkg, lic)
         if locale_section:
             d.setVar('SECTION:' + pkg, locale_section)
 
-- 
2.46.2

[OE-core][PATCH v3 0/4] Incompatible Licenses in Dynamic Packages

[Joshua Watt]

I noticed that INCOMPATIBLE_LICENSE was not applying *-locale-*
packages at packaging time like other packages. This meant that if there
were *-locale-* packages with an incompatible license, they would still
be generated, and then the final image would likely fail when the
packages were added to it. This is actually not just a problem with
*-locale-* packages, but in fact any packages generated dynamically.

To fix this for all cases, the code to detect incompatible licenses in
packages is moved to a library function, and instead of using the
'_exclude-incompatible-' variable to transfer this knowledge from
base.bbclass to do_package, the library function is called a second time
in do_package. While slightly less efficient, this means that do_package
is checking the actual final list of packages for license exclusions,
make it accurate even when dynamic packages are added.

Finally, the last patch makes it possible to set the LICENSE field of
the generated *-locale-* packages, by making them match
LICENSE:${PN}-locale, if set. This provides a means for recipe writes to
affect the license of the split locale packages in the event that it
doesn't match the recipe LICENSE. This makes logical sense to me, but
please let me know if I'm missing some important assumption

v3: Fixed several bugs that were accidentally omitted from the previous
patch series, and feedback.

Joshua Watt (4):
  classes-global/license: Move functions to library code
  lib: license: Move package license skip to library
  lib: package: Check incompatible licenses at packaging time
  lib: package: Copy locale license

 meta/classes-global/base.bbclass          |  42 +----
 meta/classes-global/license.bbclass       | 165 ------------------
 meta/classes-global/package.bbclass       |   4 -
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 202 ++++++++++++++++++++++
 meta/lib/oe/package.py                    |  10 +-
 6 files changed, 223 insertions(+), 214 deletions(-)

-- 
2.46.2

[OE-core][PATCH v3 1/4] classes-global/license: Move functions to library code

[Joshua Watt]

Moves several of the functions in license.bbclass to be library code

New function dependencies were manually verified using bitbake-dumpsigs
to ensure that bitbake identified the same dependencies even though they
are now in library code (although the new function names mean that the
task hashes still change)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/classes-global/base.bbclass          |  10 +-
 meta/classes-global/license.bbclass       | 165 ----------------------
 meta/classes-recipe/license_image.bbclass |  14 +-
 meta/lib/oe/license.py                    | 163 +++++++++++++++++++++
 4 files changed, 175 insertions(+), 177 deletions(-)

diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index b6940bbb6ff..88b932fc3f0 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -528,8 +528,8 @@ python () {
         bb.fatal('This recipe does not have the LICENSE field set (%s)' % pn)
 
     if bb.data.inherits_class('license', d):
-        check_license_format(d)
-        unmatched_license_flags = check_license_flags(d)
+        oe.license.check_license_format(d)
+        unmatched_license_flags = oe.license.check_license_flags(d)
         if unmatched_license_flags:
             for unmatched in unmatched_license_flags:
                 message = "Has a restricted license '%s' which is not listed in your LICENSE_FLAGS_ACCEPTED." % unmatched
@@ -583,7 +583,7 @@ python () {
             check_license = False
 
         if check_license and bad_licenses:
-            bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
 
             exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
 
@@ -599,7 +599,7 @@ python () {
             for pkg in pkgs:
                 remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
 
-                incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
                 if incompatible_lic:
                     skipped_pkgs[pkg] = incompatible_lic
                 else:
@@ -612,7 +612,7 @@ python () {
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
-                incompatible_lic = incompatible_license(d, bad_licenses)
+                incompatible_lic = oe.license.incompatible_license(d, bad_licenses)
                 for pkg in skipped_pkgs:
                     incompatible_lic += skipped_pkgs[pkg]
                 incompatible_lic = sorted(list(set(incompatible_lic)))
diff --git a/meta/classes-global/license.bbclass b/meta/classes-global/license.bbclass
index 043715fcc36..94dcc7f331c 100644
--- a/meta/classes-global/license.bbclass
+++ b/meta/classes-global/license.bbclass
@@ -255,171 +255,6 @@ def find_license_files(d):
 
     return lic_files_paths
 
-def return_spdx(d, license):
-    """
-    This function returns the spdx mapping of a license if it exists.
-     """
-    return d.getVarFlag('SPDXLICENSEMAP', license)
-
-def canonical_license(d, license):
-    """
-    Return the canonical (SPDX) form of the license if available (so GPLv3
-    becomes GPL-3.0-only) or the passed license if there is no canonical form.
-    """
-    return d.getVarFlag('SPDXLICENSEMAP', license) or license
-
-def expand_wildcard_licenses(d, wildcard_licenses):
-    """
-    There are some common wildcard values users may want to use. Support them
-    here.
-    """
-    licenses = set(wildcard_licenses)
-    mapping = {
-        "AGPL-3.0*" : ["AGPL-3.0-only", "AGPL-3.0-or-later"],
-        "GPL-3.0*" : ["GPL-3.0-only", "GPL-3.0-or-later"],
-        "LGPL-3.0*" : ["LGPL-3.0-only", "LGPL-3.0-or-later"],
-    }
-    for k in mapping:
-        if k in wildcard_licenses:
-            licenses.remove(k)
-            for item in mapping[k]:
-                licenses.add(item)
-
-    for l in licenses:
-        if l in oe.license.obsolete_license_list():
-            bb.fatal("Error, %s is an obsolete license, please use an SPDX reference in INCOMPATIBLE_LICENSE" % l)
-        if "*" in l:
-            bb.fatal("Error, %s is an invalid license wildcard entry" % l)
-
-    return list(licenses)
-
-def incompatible_license_contains(license, truevalue, falsevalue, d):
-    license = canonical_license(d, license)
-    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
-    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
-    return truevalue if license in bad_licenses else falsevalue
-
-def incompatible_pkg_license(d, dont_want_licenses, license):
-    # Handles an "or" or two license sets provided by
-    # flattened_licenses(), pick one that works if possible.
-    def choose_lic_set(a, b):
-        return a if all(oe.license.license_ok(canonical_license(d, lic),
-                            dont_want_licenses) for lic in a) else b
-
-    try:
-        licenses = oe.license.flattened_licenses(license, choose_lic_set)
-    except oe.license.LicenseError as exc:
-        bb.fatal('%s: %s' % (d.getVar('P'), exc))
-
-    incompatible_lic = []
-    for l in licenses:
-        license = canonical_license(d, l)
-        if not oe.license.license_ok(license, dont_want_licenses):
-            incompatible_lic.append(license)
-
-    return sorted(incompatible_lic)
-
-def incompatible_license(d, dont_want_licenses, package=None):
-    """
-    This function checks if a recipe has only incompatible licenses. It also
-    take into consideration 'or' operand.  dont_want_licenses should be passed
-    as canonical (SPDX) names.
-    """
-    import oe.license
-    license = d.getVar("LICENSE:%s" % package) if package else None
-    if not license:
-        license = d.getVar('LICENSE')
-
-    return incompatible_pkg_license(d, dont_want_licenses, license)
-
-def check_license_flags(d):
-    """
-    This function checks if a recipe has any LICENSE_FLAGS that
-    aren't acceptable.
-
-    If it does, it returns the all LICENSE_FLAGS missing from the list
-    of acceptable license flags, or all of the LICENSE_FLAGS if there
-    is no list of acceptable flags.
-
-    If everything is is acceptable, it returns None.
-    """
-
-    def license_flag_matches(flag, acceptlist, pn):
-        """
-        Return True if flag matches something in acceptlist, None if not.
-
-        Before we test a flag against the acceptlist, we append _${PN}
-        to it.  We then try to match that string against the
-        acceptlist.  This covers the normal case, where we expect
-        LICENSE_FLAGS to be a simple string like 'commercial', which
-        the user typically matches exactly in the acceptlist by
-        explicitly appending the package name e.g 'commercial_foo'.
-        If we fail the match however, we then split the flag across
-        '_' and append each fragment and test until we either match or
-        run out of fragments.
-        """
-        flag_pn = ("%s_%s" % (flag, pn))
-        for candidate in acceptlist:
-            if flag_pn == candidate:
-                    return True
-
-        flag_cur = ""
-        flagments = flag_pn.split("_")
-        flagments.pop() # we've already tested the full string
-        for flagment in flagments:
-            if flag_cur:
-                flag_cur += "_"
-            flag_cur += flagment
-            for candidate in acceptlist:
-                if flag_cur == candidate:
-                    return True
-        return False
-
-    def all_license_flags_match(license_flags, acceptlist):
-        """ Return all unmatched flags, None if all flags match """
-        pn = d.getVar('PN')
-        split_acceptlist = acceptlist.split()
-        flags = []
-        for flag in license_flags.split():
-            if not license_flag_matches(flag, split_acceptlist, pn):
-                flags.append(flag)
-        return flags if flags else None
-
-    license_flags = d.getVar('LICENSE_FLAGS')
-    if license_flags:
-        acceptlist = d.getVar('LICENSE_FLAGS_ACCEPTED')
-        if not acceptlist:
-            return license_flags.split()
-        unmatched_flags = all_license_flags_match(license_flags, acceptlist)
-        if unmatched_flags:
-            return unmatched_flags
-    return None
-
-def check_license_format(d):
-    """
-    This function checks if LICENSE is well defined,
-        Validate operators in LICENSES.
-        No spaces are allowed between LICENSES.
-    """
-    pn = d.getVar('PN')
-    licenses = d.getVar('LICENSE')
-    from oe.license import license_operator, license_operator_chars, license_pattern
-
-    elements = list(filter(lambda x: x.strip(), license_operator.split(licenses)))
-    for pos, element in enumerate(elements):
-        if license_pattern.match(element):
-            if pos > 0 and license_pattern.match(elements[pos - 1]):
-                oe.qa.handle_error('license-format',
-                        '%s: LICENSE value "%s" has an invalid format - license names ' \
-                        'must be separated by the following characters to indicate ' \
-                        'the license selection: %s' %
-                        (pn, licenses, license_operator_chars), d)
-        elif not license_operator.match(element):
-            oe.qa.handle_error('license-format',
-                    '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
-                    'in the valid list of separators (%s)' %
-                    (pn, licenses, element, license_operator_chars), d)
-
 SSTATETASKS += "do_populate_lic"
 do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}"
 do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
diff --git a/meta/classes-recipe/license_image.bbclass b/meta/classes-recipe/license_image.bbclass
index 0e953856a63..d2c5ab902ce 100644
--- a/meta/classes-recipe/license_image.bbclass
+++ b/meta/classes-recipe/license_image.bbclass
@@ -58,7 +58,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     import stat
 
     bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE") or "").split()
-    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+    bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
     pkgarchs = d.getVar("SSTATE_ARCHS").split()
     pkgarchs.reverse()
 
@@ -66,17 +66,17 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
     with open(license_manifest, "w") as license_file:
         for pkg in sorted(pkg_dic):
             remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-            incompatible_licenses = incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
+            incompatible_licenses = oe.license.incompatible_pkg_license(d, remaining_bad_licenses, pkg_dic[pkg]["LICENSE"])
             if incompatible_licenses:
                 bb.fatal("Package %s cannot be installed into the image because it has incompatible license(s): %s" %(pkg, ' '.join(incompatible_licenses)))
             else:
-                incompatible_licenses = incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
+                incompatible_licenses = oe.license.incompatible_pkg_license(d, bad_licenses, pkg_dic[pkg]["LICENSE"])
                 if incompatible_licenses:
                     oe.qa.handle_error('license-exception', "Including %s with incompatible license(s) %s into the image, because it has been allowed by exception list." %(pkg, ' '.join(incompatible_licenses)), d)
             try:
                 (pkg_dic[pkg]["LICENSE"], pkg_dic[pkg]["LICENSES"]) = \
                     oe.license.manifest_licenses(pkg_dic[pkg]["LICENSE"],
-                    remaining_bad_licenses, canonical_license, d)
+                    remaining_bad_licenses, oe.license.canonical_license, d)
             except oe.license.LicenseError as exc:
                 bb.fatal('%s: %s' % (d.getVar('P'), exc))
 
@@ -144,7 +144,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                 if not os.path.exists(pkg_license_dir ):
                     bb.fatal("Couldn't find license information for dependency %s" % pkg)
 
-                pkg_manifest_licenses = [canonical_license(d, lic) \
+                pkg_manifest_licenses = [oe.license.canonical_license(d, lic) \
                         for lic in pkg_dic[pkg]["LICENSES"]]
 
                 licenses = os.listdir(pkg_license_dir)
@@ -153,7 +153,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                     pkg_rootfs_license = os.path.join(pkg_rootfs_license_dir, lic)
 
                     if re.match(r"^generic_.*$", lic):
-                        generic_lic = canonical_license(d,
+                        generic_lic = oe.license.canonical_license(d,
                                 re.search(r"^generic_(.*)$", lic).group(1))
 
                         # Do not copy generic license into package if isn't
@@ -176,7 +176,7 @@ def write_license_files(d, license_manifest, pkg_dic, rootfs=True):
                         if not os.path.exists(pkg_rootfs_license):
                             os.symlink(os.path.join('..', generic_lic_file), pkg_rootfs_license)
                     else:
-                        if (oe.license.license_ok(canonical_license(d,
+                        if (oe.license.license_ok(oe.license.canonical_license(d,
                                 lic), bad_licenses) == False or
                                 os.path.exists(pkg_rootfs_license)):
                             continue
diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
index d9c8d94da47..7739697c401 100644
--- a/meta/lib/oe/license.py
+++ b/meta/lib/oe/license.py
@@ -259,3 +259,166 @@ def apply_pkg_license_exception(pkg, bad_licenses, exceptions):
     """Return remaining bad licenses after removing any package exceptions"""
 
     return [lic for lic in bad_licenses if pkg + ':' + lic not in exceptions]
+
+def return_spdx(d, license):
+    """
+    This function returns the spdx mapping of a license if it exists.
+     """
+    return d.getVarFlag('SPDXLICENSEMAP', license)
+
+def canonical_license(d, license):
+    """
+    Return the canonical (SPDX) form of the license if available (so GPLv3
+    becomes GPL-3.0-only) or the passed license if there is no canonical form.
+    """
+    return d.getVarFlag('SPDXLICENSEMAP', license) or license
+
+def expand_wildcard_licenses(d, wildcard_licenses):
+    """
+    There are some common wildcard values users may want to use. Support them
+    here.
+    """
+    licenses = set(wildcard_licenses)
+    mapping = {
+        "AGPL-3.0*" : ["AGPL-3.0-only", "AGPL-3.0-or-later"],
+        "GPL-3.0*" : ["GPL-3.0-only", "GPL-3.0-or-later"],
+        "LGPL-3.0*" : ["LGPL-3.0-only", "LGPL-3.0-or-later"],
+    }
+    for k in mapping:
+        if k in wildcard_licenses:
+            licenses.remove(k)
+            for item in mapping[k]:
+                licenses.add(item)
+
+    for l in licenses:
+        if l in obsolete_license_list():
+            bb.fatal("Error, %s is an obsolete license, please use an SPDX reference in INCOMPATIBLE_LICENSE" % l)
+        if "*" in l:
+            bb.fatal("Error, %s is an invalid license wildcard entry" % l)
+
+    return list(licenses)
+
+def incompatible_license_contains(license, truevalue, falsevalue, d):
+    license = canonical_license(d, license)
+    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+    return truevalue if license in bad_licenses else falsevalue
+
+def incompatible_pkg_license(d, dont_want_licenses, license):
+    # Handles an "or" or two license sets provided by
+    # flattened_licenses(), pick one that works if possible.
+    def choose_lic_set(a, b):
+        return a if all(license_ok(canonical_license(d, lic),
+                            dont_want_licenses) for lic in a) else b
+
+    try:
+        licenses = flattened_licenses(license, choose_lic_set)
+    except LicenseError as exc:
+        bb.fatal('%s: %s' % (d.getVar('P'), exc))
+
+    incompatible_lic = []
+    for l in licenses:
+        license = canonical_license(d, l)
+        if not license_ok(license, dont_want_licenses):
+            incompatible_lic.append(license)
+
+    return sorted(incompatible_lic)
+
+def incompatible_license(d, dont_want_licenses, package=None):
+    """
+    This function checks if a recipe has only incompatible licenses. It also
+    take into consideration 'or' operand.  dont_want_licenses should be passed
+    as canonical (SPDX) names.
+    """
+    license = d.getVar("LICENSE:%s" % package) if package else None
+    if not license:
+        license = d.getVar('LICENSE')
+
+    return incompatible_pkg_license(d, dont_want_licenses, license)
+
+def check_license_flags(d):
+    """
+    This function checks if a recipe has any LICENSE_FLAGS that
+    aren't acceptable.
+
+    If it does, it returns the all LICENSE_FLAGS missing from the list
+    of acceptable license flags, or all of the LICENSE_FLAGS if there
+    is no list of acceptable flags.
+
+    If everything is is acceptable, it returns None.
+    """
+
+    def license_flag_matches(flag, acceptlist, pn):
+        """
+        Return True if flag matches something in acceptlist, None if not.
+
+        Before we test a flag against the acceptlist, we append _${PN}
+        to it.  We then try to match that string against the
+        acceptlist.  This covers the normal case, where we expect
+        LICENSE_FLAGS to be a simple string like 'commercial', which
+        the user typically matches exactly in the acceptlist by
+        explicitly appending the package name e.g 'commercial_foo'.
+        If we fail the match however, we then split the flag across
+        '_' and append each fragment and test until we either match or
+        run out of fragments.
+        """
+        flag_pn = ("%s_%s" % (flag, pn))
+        for candidate in acceptlist:
+            if flag_pn == candidate:
+                    return True
+
+        flag_cur = ""
+        flagments = flag_pn.split("_")
+        flagments.pop() # we've already tested the full string
+        for flagment in flagments:
+            if flag_cur:
+                flag_cur += "_"
+            flag_cur += flagment
+            for candidate in acceptlist:
+                if flag_cur == candidate:
+                    return True
+        return False
+
+    def all_license_flags_match(license_flags, acceptlist):
+        """ Return all unmatched flags, None if all flags match """
+        pn = d.getVar('PN')
+        split_acceptlist = acceptlist.split()
+        flags = []
+        for flag in license_flags.split():
+            if not license_flag_matches(flag, split_acceptlist, pn):
+                flags.append(flag)
+        return flags if flags else None
+
+    license_flags = d.getVar('LICENSE_FLAGS')
+    if license_flags:
+        acceptlist = d.getVar('LICENSE_FLAGS_ACCEPTED')
+        if not acceptlist:
+            return license_flags.split()
+        unmatched_flags = all_license_flags_match(license_flags, acceptlist)
+        if unmatched_flags:
+            return unmatched_flags
+    return None
+
+def check_license_format(d):
+    """
+    This function checks if LICENSE is well defined,
+        Validate operators in LICENSES.
+        No spaces are allowed between LICENSES.
+    """
+    pn = d.getVar('PN')
+    licenses = d.getVar('LICENSE')
+
+    elements = list(filter(lambda x: x.strip(), license_operator.split(licenses)))
+    for pos, element in enumerate(elements):
+        if license_pattern.match(element):
+            if pos > 0 and license_pattern.match(elements[pos - 1]):
+                oe.qa.handle_error('license-format',
+                        '%s: LICENSE value "%s" has an invalid format - license names ' \
+                        'must be separated by the following characters to indicate ' \
+                        'the license selection: %s' %
+                        (pn, licenses, license_operator_chars), d)
+        elif not license_operator.match(element):
+            oe.qa.handle_error('license-format',
+                    '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
+                    'in the valid list of separators (%s)' %
+                    (pn, licenses, element, license_operator_chars), d)
-- 
2.46.2

[OE-core][PATCH v3 2/4] lib: license: Move package license skip to library

[Joshua Watt]

Moves the code that skips packages with incompatible licenses to the
library code so that it can be called in other locations

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/classes-global/base.bbclass | 35 ++++------------------------
 meta/lib/oe/license.py           | 39 ++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 31 deletions(-)

diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 88b932fc3f0..5b8663f454d 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -573,37 +573,10 @@ python () {
 
         bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
 
-        check_license = False if pn.startswith("nativesdk-") else True
-        for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
-              "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
-              "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
-            if pn.endswith(d.expand(t)):
-                check_license = False
-        if pn.startswith("gcc-source-"):
-            check_license = False
-
-        if check_license and bad_licenses:
-            bad_licenses = oe.license.expand_wildcard_licenses(d, bad_licenses)
-
-            exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
-
-            for lic_exception in exceptions:
-                if ":" in lic_exception:
-                    lic_exception = lic_exception.split(":")[1]
-                if lic_exception in oe.license.obsolete_license_list():
-                    bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
-
-            pkgs = d.getVar('PACKAGES').split()
-            skipped_pkgs = {}
-            unskipped_pkgs = []
-            for pkg in pkgs:
-                remaining_bad_licenses = oe.license.apply_pkg_license_exception(pkg, bad_licenses, exceptions)
-
-                incompatible_lic = oe.license.incompatible_license(d, remaining_bad_licenses, pkg)
-                if incompatible_lic:
-                    skipped_pkgs[pkg] = incompatible_lic
-                else:
-                    unskipped_pkgs.append(pkg)
+        pkgs = d.getVar('PACKAGES').split()
+        if pkgs:
+            skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, pkgs)
+            unskipped_pkgs = [p for p in pkgs if p not in skipped_pkgs]
 
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
diff --git a/meta/lib/oe/license.py b/meta/lib/oe/license.py
index 7739697c401..32c77fa204d 100644
--- a/meta/lib/oe/license.py
+++ b/meta/lib/oe/license.py
@@ -422,3 +422,42 @@ def check_license_format(d):
                     '%s: LICENSE value "%s" has an invalid separator "%s" that is not ' \
                     'in the valid list of separators (%s)' %
                     (pn, licenses, element, license_operator_chars), d)
+
+def skip_incompatible_package_licenses(d, pkgs):
+    if not pkgs:
+        return {}
+
+    pn = d.getVar("PN")
+
+    check_license = False if pn.startswith("nativesdk-") else True
+    for t in ["-native", "-cross-${TARGET_ARCH}", "-cross-initial-${TARGET_ARCH}",
+            "-crosssdk-${SDK_SYS}", "-crosssdk-initial-${SDK_SYS}",
+            "-cross-canadian-${TRANSLATED_TARGET_ARCH}"]:
+        if pn.endswith(d.expand(t)):
+            check_license = False
+    if pn.startswith("gcc-source-"):
+        check_license = False
+
+    bad_licenses = (d.getVar('INCOMPATIBLE_LICENSE') or "").split()
+    if not check_license or not bad_licenses:
+        return {}
+
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
+    exceptions = (d.getVar("INCOMPATIBLE_LICENSE_EXCEPTIONS") or "").split()
+
+    for lic_exception in exceptions:
+        if ":" in lic_exception:
+            lic_exception = lic_exception.split(":")[1]
+        if lic_exception in obsolete_license_list():
+            bb.fatal("Obsolete license %s used in INCOMPATIBLE_LICENSE_EXCEPTIONS" % lic_exception)
+
+    skipped_pkgs = {}
+    for pkg in pkgs:
+        remaining_bad_licenses = apply_pkg_license_exception(pkg, bad_licenses, exceptions)
+
+        incompatible_lic = incompatible_license(d, remaining_bad_licenses, pkg)
+        if incompatible_lic:
+            skipped_pkgs[pkg] = incompatible_lic
+
+    return skipped_pkgs
-- 
2.46.2

[OE-core][PATCH v3 3/4] lib: package: Check incompatible licenses at packaging time

[Joshua Watt]

Instead of checking for incompatible licenses in the anonymous python
and setting '_exclude_incompatible-', (re)check all the packages in
populate_packages(). This ensures that all packages are processed, even
dynamically generated ones.

The use of the '_exclude-incompatible-' variable set in base.bbclass has
been the mechanism used for per-packages licenses since it was added as
a feature (although with different names for the variable throughout
history). However, since this misses dynamic packages, calling
oe.license.skip_incompatible_package_licenses() a second time on the
actual final package set is a better solution.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/classes-global/base.bbclass    | 1 -
 meta/classes-global/package.bbclass | 4 ----
 meta/lib/oe/package.py              | 6 +++---
 3 files changed, 3 insertions(+), 8 deletions(-)

diff --git a/meta/classes-global/base.bbclass b/meta/classes-global/base.bbclass
index 5b8663f454d..b81e61fdb72 100644
--- a/meta/classes-global/base.bbclass
+++ b/meta/classes-global/base.bbclass
@@ -581,7 +581,6 @@ python () {
             if unskipped_pkgs:
                 for pkg in skipped_pkgs:
                     bb.debug(1, "Skipping the package %s at do_rootfs because of incompatible license(s): %s" % (pkg, ' '.join(skipped_pkgs[pkg])))
-                    d.setVar('_exclude_incompatible-' + pkg, ' '.join(skipped_pkgs[pkg]))
                 for pkg in unskipped_pkgs:
                     bb.debug(1, "Including the package %s" % pkg)
             else:
diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass
index 6cd8c0140f2..9be1d6a5b17 100644
--- a/meta/classes-global/package.bbclass
+++ b/meta/classes-global/package.bbclass
@@ -447,10 +447,6 @@ def gen_packagevar(d, pkgvars="PACKAGEVARS"):
     for p in pkgs:
         for v in vars:
             ret.append(v + ":" + p)
-
-        # Ensure that changes to INCOMPATIBLE_LICENSE re-run do_package for
-        # affected recipes.
-        ret.append('_exclude_incompatible-%s' % p)
     return " ".join(ret)
 
 
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index c213a9a3ca6..104b880b9e1 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -1447,10 +1447,10 @@ def populate_packages(d):
 
     # Handle excluding packages with incompatible licenses
     package_list = []
+    skipped_pkgs = oe.license.skip_incompatible_package_licenses(d, packages)
     for pkg in packages:
-        licenses = d.getVar('_exclude_incompatible-' + pkg)
-        if licenses:
-            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, licenses)
+        if pkg in skipped_pkgs:
+            msg = "Excluding %s from packaging as it has incompatible license(s): %s" % (pkg, skipped_pkgs[pkg])
             oe.qa.handle_error("incompatible-license", msg, d)
         else:
             package_list.append(pkg)
-- 
2.46.2

[OE-core][PATCH v3 4/4] lib: package: Copy locale license

[Joshua Watt]

When creating split locales, copy the license from LICENSE:${PN}-locale
if set, otherwise leave it unspecified (which will result in falling
back to LICENSE)

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/lib/oe/package.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 104b880b9e1..16359232ecd 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -663,6 +663,8 @@ def split_locales(d):
     except ValueError:
         locale_index = len(packages)
 
+    lic = d.getVar("LICENSE:" + pn + "-locale")
+
     localepaths = []
     locales = set()
     for localepath in (d.getVar('LOCALE_PATHS') or "").split():
@@ -698,6 +700,8 @@ def split_locales(d):
         d.setVar('RPROVIDES:' + pkg, '%s-locale %s%s-translation' % (pn, mlprefix, ln))
         d.setVar('SUMMARY:' + pkg, '%s - %s translations' % (summary, l))
         d.setVar('DESCRIPTION:' + pkg, '%s  This package contains language translation files for the %s locale.' % (description, l))
+        if lic:
+            d.setVar('LICENSE:' + pkg, lic)
         if locale_section:
             d.setVar('SECTION:' + pkg, locale_section)
 
-- 
2.46.2