[zeus 0/5] More patches for review.

[zeus 0/5] More patches for review.

[Armin Kuster]

Here are somemore patches that where also tested and needing review.
Please have feedback by Tuesday.


The following changes since commit f9739e0f58f5d8a5dc01fdb0efaa778af4edd671:

  kernel: Make symbol link to vmlinux.64 in boot directory (2020-01-08 20:33:12 -0800)

are available in the Git repository at:

  git://git.openembedded.org/openembedded-core-contrib stable/zeus-nut
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/zeus-nut

Peter Kjellerstedt (5):
  toaster.bbclass: Correct pkgdatadir path in toaster_package_dumpdata()
  populate_sdk_ext.bbclass: No longer needed to clean away
    conf/sanity_info
  sanity.bbclass: Move sanity_info from conf to cache
  licenses.conf: Remove the SRC_DISTRIBUTE_LICENSES variable
  license.bbclass: Introduce AVAILABLE_LICENSES that lists all licenses

 meta/classes/license.bbclass                  | 27 ++++++++++--
 meta/classes/populate_sdk_ext.bbclass         |  2 +-
 meta/classes/sanity.bbclass                   |  2 +-
 meta/classes/toaster.bbclass                  | 15 +++++--
 meta/conf/documentation.conf                  |  1 +
 meta/conf/licenses.conf                       | 43 -------------------
 meta/lib/oeqa/buildperf/base.py               |  2 +-
 .../oeqa/selftest/cases/incompatible_lic.py   |  6 +--
 8 files changed, 41 insertions(+), 57 deletions(-)

-- 
2.17.1

[zeus 1/5] toaster.bbclass: Correct pkgdatadir path in toaster_package_dumpdata()

[Armin Kuster]

From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>

In commit 692b2046 (package: Fix race between do_package and
do_packagedata), the path used for do_packagedata[sstate-inputdirs]
was changed from "${PKGDESTWORK}" to "${WORKDIR}/pkgdata-pdata-input".
This commit adapts the path used for pkgdatadir in
toaster_package_dumpdata() accordingly to avoid setscene errors like:

  ERROR: libgcc-9.2.0-r0 do_packagedata_setscene: Error executing a
  python function in exec_python_func() autogenerated:

  The stack trace of python calls that resulted in this
  exception/failure was:
  File: 'exec_python_func() autogenerated', lineno: 2, function:
  <module>
       0001:
   *** 0002:toaster_package_dumpdata(d)
       0003:
  File: 'meta/classes/toaster.bbclass', lineno: 130, function:
  toaster_package_dumpdata
       0126:    lpkgdata = {}
       0127:    datadir = os.path.join(pkgdatadir, 'runtime')
       0128:
       0129:    # scan and send data for each generated package
   *** 0130:    for datafile in os.listdir(datadir):
       0131:        if not datafile.endswith('.packaged'):
       0132:            lpkgdata = _toaster_load_pkgdatafile(datadir,
       0133:            # Fire an event containing the pkg data
       0134:            bb.event.fire(bb.event.MetadataEvent(
  Exception: FileNotFoundError: [Errno 2] No such file or directory:
  'tmp/work/mips32r2el-nf-poky-linux/libgcc/9.2.0-r0/pkgdata/runtime'

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5a0f6f631b86f7107aa72453b6d23f32ba39f713)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 meta/classes/toaster.bbclass | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass
index 6cef0b8f6e..6a65ecb957 100644
--- a/meta/classes/toaster.bbclass
+++ b/meta/classes/toaster.bbclass
@@ -113,7 +113,7 @@ def _toaster_load_pkgdatafile(dirpath, filepath):
                 pass    # ignore lines without valid key: value pairs
     return pkgdata
 
-python toaster_package_dumpdata() {
+def _toaster_dumpdata(pkgdatadir, d):
     """
     Dumps the data about the packages created by a recipe
     """
@@ -122,7 +122,6 @@ python toaster_package_dumpdata() {
     if not d.getVar('PACKAGES'):
         return
 
-    pkgdatadir = d.getVar('PKGDESTWORK')
     lpkgdata = {}
     datadir = os.path.join(pkgdatadir, 'runtime')
 
@@ -132,6 +131,14 @@ python toaster_package_dumpdata() {
             lpkgdata = _toaster_load_pkgdatafile(datadir, datafile)
             # Fire an event containing the pkg data
             bb.event.fire(bb.event.MetadataEvent("SinglePackageInfo", lpkgdata), d)
+
+python toaster_package_dumpdata() {
+    _toaster_dumpdata(d.getVar('PKGDESTWORK'), d)
+}
+
+python toaster_packagedata_dumpdata() {
+    # This path needs to match do_packagedata[sstate-inputdirs]
+    _toaster_dumpdata(os.path.join(d.getVar('WORKDIR'), 'pkgdata-pdata-input'), d)
 }
 
 # 2. Dump output image files information
@@ -366,8 +373,8 @@ toaster_buildhistory_dump[eventmask] = "bb.event.BuildCompleted"
 addhandler toaster_artifacts
 toaster_artifacts[eventmask] = "bb.runqueue.runQueueTaskSkipped bb.runqueue.runQueueTaskCompleted"
 
-do_packagedata_setscene[postfuncs] += "toaster_package_dumpdata "
-do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata "
+do_packagedata_setscene[postfuncs] += "toaster_packagedata_dumpdata "
+do_packagedata_setscene[vardepsexclude] += "toaster_packagedata_dumpdata "
 
 do_package[postfuncs] += "toaster_package_dumpdata "
 do_package[vardepsexclude] += "toaster_package_dumpdata "
-- 
2.17.1

[zeus 2/5] populate_sdk_ext.bbclass: No longer needed to clean away conf/sanity_info

[Armin Kuster]

From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>

Since the sanity_info file has moved from the conf directory to the
cache directory, there is no longer any need to clean it away
explicitly in clean_esdk_builddir() since the whole cache directory is
already cleaned away anyway.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 40c30990e1be72130819c040fe471e2bdc0c6e7d)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 9fda1c9e78..5657af4b6a 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -122,7 +122,7 @@ SDK_TITLE_task-populate-sdk-ext = "${@d.getVar('DISTRO_NAME') or d.getVar('DISTR
 def clean_esdk_builddir(d, sdkbasepath):
     """Clean up traces of the fake build for create_filtered_tasklist()"""
     import shutil
-    cleanpaths = 'cache conf/sanity_info tmp'.split()
+    cleanpaths = ['cache', 'tmp']
     for pth in cleanpaths:
         fullpth = os.path.join(sdkbasepath, pth)
         if os.path.isdir(fullpth):
-- 
2.17.1

[zeus 3/5] sanity.bbclass: Move sanity_info from conf to cache

[Armin Kuster]

From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>

Since this file is written during recipe parsing, having it in the
${BUILDDIR}/conf directory, which is covered by an inotify watcher,
will trigger a re-parse the next time bitbake is run and the resident
bitbake server is enabled. This causes the sanity_info file to be
updated again, which triggers a new parse the next time bitbake is run
ad infinitum. Moving it to ${BUILDDIR}/cache should avoid this.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f98103b548aa7dba6b1be6c8e02ef41858a8e85c)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 meta/classes/sanity.bbclass     | 2 +-
 meta/lib/oeqa/buildperf/base.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index 9d0c784032..936fe913b4 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -927,7 +927,7 @@ def check_sanity(sanity_data):
     last_tmpdir = ""
     last_sstate_dir = ""
     last_nativelsbstr = ""
-    sanityverfile = sanity_data.expand("${TOPDIR}/conf/sanity_info")
+    sanityverfile = sanity_data.expand("${TOPDIR}/cache/sanity_info")
     if os.path.exists(sanityverfile):
         with open(sanityverfile, 'r') as f:
             for line in f:
diff --git a/meta/lib/oeqa/buildperf/base.py b/meta/lib/oeqa/buildperf/base.py
index 3b2fed549f..5f1805d86c 100644
--- a/meta/lib/oeqa/buildperf/base.py
+++ b/meta/lib/oeqa/buildperf/base.py
@@ -462,7 +462,7 @@ class BuildPerfTestCase(unittest.TestCase):
     def rm_tmp(self):
         """Cleanup temporary/intermediate files and directories"""
         log.debug("Removing temporary and cache files")
-        for name in ['bitbake.lock', 'conf/sanity_info',
+        for name in ['bitbake.lock', 'cache/sanity_info',
                      self.bb_vars['TMPDIR']]:
             oe.path.remove(name, recurse=True)
 
-- 
2.17.1

[zeus 4/5] licenses.conf: Remove the SRC_DISTRIBUTE_LICENSES variable

[Armin Kuster]

From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>

The SRC_DISTRIBUTE_LICENSES variable and its static list of licenses
has been replaced by AVAILABLE_LICENSES, which automatically contains
all available licenses.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 64daaf29e2c12c8b587bafdebf9409433187ddf7)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 meta/conf/licenses.conf | 43 -----------------------------------------
 1 file changed, 43 deletions(-)

diff --git a/meta/conf/licenses.conf b/meta/conf/licenses.conf
index 7b01c57632..1c586a3bc6 100644
--- a/meta/conf/licenses.conf
+++ b/meta/conf/licenses.conf
@@ -1,44 +1,3 @@
-# These aren't actually used anywhere that I can tell
-# They may be in the future (or are used by someone else
-# For completion sake, I've updated them
-SRC_DISTRIBUTE_LICENSES += "AAL Adobe AFL-1.2 AFL-2.0 AFL-2.1 AFL-3.0"
-SRC_DISTRIBUTE_LICENSES += "AGPL-3.0 ANTLR-PD Apache-1.0 Apache-1.1 Apache-2.0"
-SRC_DISTRIBUTE_LICENSES += "APL-1.0 APSL-1.0 APSL-1.1 APSL-1.2 APSL-2.0"
-SRC_DISTRIBUTE_LICENSES += "Artistic-1.0 Artistic-2.0 BitstreamVera BSD"
-SRC_DISTRIBUTE_LICENSES += "BSD-2-Clause BSD-3-Clause BSD-4-Clause BSL-1.0"
-SRC_DISTRIBUTE_LICENSES += "CATOSL-1.1 CC0-1.0 CC-BY-1.0 CC-BY-2.0 CC-BY-2.5"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-3.0 CC-BY-NC-1.0 CC-BY-NC-2.0 CC-BY-NC-2.5"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-3.0 CC-BY-NC-ND-1.0 CC-BY-NC-ND-2.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-ND-2.5 CC-BY-NC-ND-3.0 CC-BY-NC-SA-1.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-NC-SA-2.0 CC-BY-NC-SA-2.5 CC-BY-NC-SA-3.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-ND-1.0 CC-BY-ND-2.0 CC-BY-ND-2.5 CC-BY-ND-3.0"
-SRC_DISTRIBUTE_LICENSES += "CC-BY-SA-1.0 CC-BY-SA-2.0 CC-BY-SA-2.5 CC-BY-SA-3.0 CC-BY-SA-4.0"
-SRC_DISTRIBUTE_LICENSES += "CDDL-1.0 CECILL-1.0 CECILL-2.0 CECILL-B CECILL-C"
-SRC_DISTRIBUTE_LICENSES += "ClArtistic CPAL-1.0 CPL-1.0 CUA-OPL-1.0 DSSSL"
-SRC_DISTRIBUTE_LICENSES += "ECL-1.0 ECL-2.0 eCos-2.0 EDL-1.0 EFL-1.0 EFL-2.0"
-SRC_DISTRIBUTE_LICENSES += "Entessa EPL-1.0 EPL-2.0 ErlPL-1.1"
-SRC_DISTRIBUTE_LICENSES += "EUDatagrid EUPL-1.0 EUPL-1.1 Fair Frameworx-1.0"
-SRC_DISTRIBUTE_LICENSES += "FreeType GFDL-1.1 GFDL-1.2 GFDL-1.3 GPL-1.0"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0 GPL-2.0-with-autoconf-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-classpath-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-font-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2.0-with-GCC-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-2-with-bison-exception GPL-3.0"
-SRC_DISTRIBUTE_LICENSES += "GPL-3.0-with-autoconf-exception"
-SRC_DISTRIBUTE_LICENSES += "GPL-3.0-with-GCC-exception"
-SRC_DISTRIBUTE_LICENSES += "gSOAP-1 gSOAP-1.3b HPND IPA IPL-1.0 ISC LGPL-2.0"
-SRC_DISTRIBUTE_LICENSES += "LGPL-2.1 LGPL-3.0 Libpng LPL-1.02 LPPL-1.0 LPPL-1.1"
-SRC_DISTRIBUTE_LICENSES += "LPPL-1.2 LPPL-1.3c MirOS MIT Motosoto MPL-1.0"
-SRC_DISTRIBUTE_LICENSES += "MPL-1.1 MS-PL MS-RL Multics NASA-1.3 Nauman NCSA"
-SRC_DISTRIBUTE_LICENSES += "NGPL Nokia NPOSL-3.0 NTP OASIS OCLC-2.0 ODbL-1.0"
-SRC_DISTRIBUTE_LICENSES += "OFL-1.1 OGTSL OLDAP-2.8 OpenSSL OSL-1.0 OSL-2.0"
-SRC_DISTRIBUTE_LICENSES += "OSL-3.0 PD PHP-3.0 PostgreSQL Proprietary"
-SRC_DISTRIBUTE_LICENSES += "Python-2.0 QPL-1.0 RHeCos-1 RHeCos-1.1 RPL-1.5"
-SRC_DISTRIBUTE_LICENSES += "RPSL-1.0 RSCPL Ruby SAX-PD SGI-1 Simple-2.0 Sleepycat"
-SRC_DISTRIBUTE_LICENSES += "SPL-1.0 SugarCRM-1 SugarCRM-1.1.3 UCB VSL-1.0 W3C"
-SRC_DISTRIBUTE_LICENSES += "Watcom-1.0 WXwindows XFree86-1.0 XFree86-1.1 Xnet XSL YPL-1.1"
-SRC_DISTRIBUTE_LICENSES += "Zimbra-1.3 Zlib ZPL-1.1 ZPL-2.0 ZPL-2.1"
-
 # Standards are great! Everyone has their own. In an effort to standardize licensing
 # names, common-licenses will use the SPDX standard license names. In order to not
 # break the non-standardized license names that we find in LICENSE, we'll set
@@ -188,5 +147,3 @@ FOSS_BASE_URL = "http://localhost/repo/?mod=spdx_license_once"
 FOSS_SERVER = "${FOSS_BASE_URL}&fullSPDXFlag=${FOSS_FULL_SPDX}&noCopyright=${FOSS_NO_COPYRIGHT}&recursiveUnpack=${FOSS_RECURSIVE_UNPACK}"
 
 FOSS_WGET_FLAGS = "-qO - --no-check-certificate --timeout=0"
-
-
-- 
2.17.1

[zeus 5/5] license.bbclass: Introduce AVAILABLE_LICENSES that lists all licenses

[Armin Kuster]

From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>

Previously, there was SRC_DISTRIBUTE_LICENSES, an undocumented
variable that contained a static list of licenses. It was used by
expand_wildcard_licenses() to expand any wildcards used in, e.g.,
INCOMPATIBLE_LICENSE. However, since this static list of licenses has
not been kept up-to-date, many licenses were missing, with the result
that if one tried to use any of those licenses with a wildcard, no
licenses would be found, effectively ignoring that they should be
marked as incompatible.

This introduces a new (documented) variable, AVAILABLE_LICENSES, that
is automatically updated to contain all licenses found in any
directories specified by ${COMMON_LICENSE_DIR} and ${LICENSE_PATH},
and uses it instead of SRC_DISTRIBUTE_LICENSES when expanding
wildcards.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8c9ef587fe499c612a878a1ab42092eb79b334ef)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 meta/classes/license.bbclass                  | 27 ++++++++++++++++---
 meta/conf/documentation.conf                  |  1 +
 .../oeqa/selftest/cases/incompatible_lic.py   |  6 ++---
 3 files changed, 27 insertions(+), 7 deletions(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index adca881c85..b0d37b119c 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -252,7 +252,7 @@ def canonical_license(d, license):
     """
     Return the canonical (SPDX) form of the license if available (so GPLv3
     becomes GPL-3.0), for the license named 'X+', return canonical form of
-    'X' if availabel and the tailing '+' (so GPLv3+ becomes GPL-3.0+), 
+    'X' if available and the tailing '+' (so GPLv3+ becomes GPL-3.0+),
     or the passed license if there is no canonical form.
     """
     lic = d.getVarFlag('SPDXLICENSEMAP', license) or ""
@@ -262,10 +262,29 @@ def canonical_license(d, license):
             lic += '+'
     return lic or license
 
+def available_licenses(d):
+    """
+    Return the available licenses by searching the directories specified by
+    COMMON_LICENSE_DIR and LICENSE_PATH.
+    """
+    lic_dirs = ((d.getVar('COMMON_LICENSE_DIR') or '') + ' ' +
+                (d.getVar('LICENSE_PATH') or '')).split()
+
+    licenses = []
+    for lic_dir in lic_dirs:
+        licenses += os.listdir(lic_dir)
+
+    licenses = sorted(licenses)
+    return licenses
+
+# Only determine the list of all available licenses once. This assumes that any
+# additions to LICENSE_PATH have been done before this file is parsed.
+AVAILABLE_LICENSES := "${@' '.join(available_licenses(d))}"
+
 def expand_wildcard_licenses(d, wildcard_licenses):
     """
-    Return actual spdx format license names if wildcard used. We expand
-    wildcards from SPDXLICENSEMAP flags and SRC_DISTRIBUTE_LICENSES values.
+    Return actual spdx format license names if wildcards are used. We expand
+    wildcards from SPDXLICENSEMAP flags and AVAILABLE_LICENSES.
     """
     import fnmatch
     licenses = wildcard_licenses[:]
@@ -274,7 +293,7 @@ def expand_wildcard_licenses(d, wildcard_licenses):
         spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
         licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
 
-    spdx_lics = (d.getVar('SRC_DISTRIBUTE_LICENSES', False) or '').split()
+    spdx_lics = d.getVar('AVAILABLE_LICENSES').split()
     for wld_lic in wildcard_licenses:
         licenses += fnmatch.filter(spdx_lics, wld_lic)
 
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 550df20b0f..ce2a37e0e5 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -69,6 +69,7 @@ ASSUME_SHLIBS[doc] = "List of shlib:package[_version] mappings. Useful for lib p
 AUTHOR[doc] = "Email address used to contact the original author(s) in order to send patches and forward bugs."
 AUTO_SYSLINUXMENU[doc] = "Enables creating an automatic menu for the syslinux bootloader."
 AUTOREV[doc] = "When SRCREV is set to the value of this variable, it specifies to use the latest source revision in the repository."
+AVAILABLE_LICENSES[doc] = "List of licenses found in the directories specified by COMMON_LICENSE_DIR and LICENSE_PATH."
 
 #B
 
diff --git a/meta/lib/oeqa/selftest/cases/incompatible_lic.py b/meta/lib/oeqa/selftest/cases/incompatible_lic.py
index 8fb93af8a8..3cc5bbc35c 100644
--- a/meta/lib/oeqa/selftest/cases/incompatible_lic.py
+++ b/meta/lib/oeqa/selftest/cases/incompatible_lic.py
@@ -12,12 +12,12 @@ class IncompatibleLicenseTests(OESelftestTestCase):
         if error_msg not in result.output:
             raise AssertionError(result.output)
 
-    # Verify that a package with an SPDX license (from SRC_DISTRIBUTE_LICENSES)
+    # Verify that a package with an SPDX license (from AVAILABLE_LICENSES)
     # cannot be built when INCOMPATIBLE_LICENSE contains this SPDX license
     def test_incompatible_spdx_license(self):
         self.lic_test('incompatible-license', 'GPL-3.0', 'GPL-3.0')
 
-    # Verify that a package with an SPDX license (from SRC_DISTRIBUTE_LICENSES)
+    # Verify that a package with an SPDX license (from AVAILABLE_LICENSES)
     # cannot be built when INCOMPATIBLE_LICENSE contains an alias (in
     # SPDXLICENSEMAP) of this SPDX license
     def test_incompatible_alias_spdx_license(self):
@@ -35,7 +35,7 @@ class IncompatibleLicenseTests(OESelftestTestCase):
         self.lic_test('incompatible-license-alias', 'GPLv3', 'GPLv3')
 
     # Verify that a package with a non-SPDX license (neither in
-    # SRC_DISTRIBUTE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
+    # AVAILABLE_LICENSES nor in SPDXLICENSEMAP) cannot be built when
     # INCOMPATIBLE_LICENSE contains this license
     def test_incompatible_nonspdx_license(self):
         self.lic_test('incompatible-nonspdx-license', 'FooLicense', 'FooLicense')
-- 
2.17.1