[PATCH 0/6] Create image manifest

[PATCH 0/6] Create image manifest

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

When building the rootfs the rootfs is created but the recipes
that deployed files in the image doesn't show up. Usually these
recipes include the bootloaders and the kernel.

With these patches a new manifest will be created that includes
the deployed recipes that were in the deployed. The format of the
manifest is different because the recipes doesn't install packages,
instead the recipes deploy files into the image

[YOCTO #6772]

The following changes since commit fc45deac89ef63ca1c44e763c38ced7dfd72cbe1:

  build-appliance-image: Update to jethro head revision (2015-11-03 14:03:03 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib mariano/bug6772
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug6772

Mariano Lopez (6):
  license.bbclass: Split license create manifest
  license.bbclass: Added get_boot_dependencies function
  license.bbclass: Added function get_deployed_dependencies
  license.bbclass: Added get_package_from_deployed
  license.bbclass: Add function get_deployed_files
  icense.bbclass: Create image license manifest

 meta/classes/license.bbclass | 219 +++++++++++++++++++++++++++++++++++++++----
 1 file changed, 200 insertions(+), 19 deletions(-)

-- 
1.8.4.5

[PATCH 1/6] license.bbclass: Split license create manifest

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This changes moves the writing of the licenses to a
separated function that could be called for other packages.

With these change it will be easier to reuse the writing of
the license for the packages deployed but not installed in
the rootfs.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 36 ++++++++++++++++++++++--------------
 1 file changed, 22 insertions(+), 14 deletions(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index c616a20..98f1733 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -26,14 +26,9 @@ python write_package_manifest() {
 }
 
 python license_create_manifest() {
-    import re
     import oe.packagedata
     from oe.rootfs import image_list_installed_packages
 
-    bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE", True) or "").split()
-    bad_licenses = map(lambda l: canonical_license(d, l), bad_licenses)
-    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
-
     build_images_from_feeds = d.getVar('BUILD_IMAGES_FROM_FEEDS', True)
     if build_images_from_feeds == "1":
         return 0
@@ -49,8 +44,18 @@ python license_create_manifest() {
             pkg_lic_name = "LICENSE_" + pkg_name
             pkg_dic[pkg_name]["LICENSE"] = pkg_dic[pkg_name][pkg_lic_name]
 
-    license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True),
+    rootfs_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True),
                         d.getVar('IMAGE_NAME', True), 'license.manifest')
+    write_license_files(d, rootfs_license_manifest, pkg_dic)
+}
+
+def write_license_files(d, license_manifest, pkg_dic):
+    import re
+
+    bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE", True) or "").split()
+    bad_licenses = map(lambda l: canonical_license(d, l), bad_licenses)
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
     with open(license_manifest, "w") as license_file:
         for pkg in sorted(pkg_dic):
             if bad_licenses:
@@ -98,15 +103,16 @@ python license_create_manifest() {
     if copy_lic_manifest == "1":
         rootfs_license_dir = os.path.join(d.getVar('IMAGE_ROOTFS', 'True'), 
                                 'usr', 'share', 'common-licenses')
-        os.makedirs(rootfs_license_dir)
+        bb.utils.mkdirhier(rootfs_license_dir)
         rootfs_license_manifest = os.path.join(rootfs_license_dir,
-                                                'license.manifest')
-        os.link(license_manifest, rootfs_license_manifest)
+                os.path.split(license_manifest)[1])
+        if not os.path.exists(rootfs_license_manifest):
+            os.link(license_manifest, rootfs_license_manifest)
 
         if copy_lic_dirs == "1":
             for pkg in sorted(pkg_dic):
                 pkg_rootfs_license_dir = os.path.join(rootfs_license_dir, pkg)
-                os.makedirs(pkg_rootfs_license_dir)
+                bb.utils.mkdirhier(pkg_rootfs_license_dir)
                 pkg_license_dir = os.path.join(d.getVar('LICENSE_DIRECTORY', True),
                                             pkg_dic[pkg]["PN"]) 
                 licenses = os.listdir(pkg_license_dir)
@@ -124,14 +130,16 @@ python license_create_manifest() {
                         if not os.path.exists(rootfs_license):
                             os.link(pkg_license, rootfs_license)
 
-                        os.symlink(os.path.join('..', lic), pkg_rootfs_license)
+                        if not os.path.exists(pkg_rootfs_license):
+                            os.symlink(os.path.join('..', lic), pkg_rootfs_license)
                     else:
-                        if oe.license.license_ok(canonical_license(d,
-                            lic), bad_licenses) == False:
+                        if (oe.license.license_ok(canonical_license(d,
+                                lic), bad_licenses) == False or
+                                os.path.exists(pkg_rootfs_license)):
                             continue
 
                         os.link(pkg_license, pkg_rootfs_license)
-}
+
 
 python do_populate_lic() {
     """
-- 
1.8.4.5

[PATCH 2/6] license.bbclass: Added get_boot_dependencies function

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This function gets the dependencies from the classes that
create a boot image, this is required because sometimes
the bootloader dependecy is in these classes. The current
classes covered are bootimg and bootdirectdisk because
these are the only clases that add dependencies.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 98f1733..d1bfe61 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -49,6 +49,37 @@ python license_create_manifest() {
     write_license_files(d, rootfs_license_manifest, pkg_dic)
 }
 
+def get_boot_dependencies(d):
+    """ Return the dependencies from boot tasks """
+
+    depends = ""
+    boot_depends_string = ""
+    taskdepdata = d.getVar("BB_TASKDEPDATA", True)
+    # Only bootimg and bootdirectdisk include the depends flag
+    boot_tasks = ["do_bootimg", "do_bootdirectdisk",]
+
+    for task in boot_tasks:
+        boot_depends_string = "%s %s" % (boot_depends_string,
+                d.getVarFlag(task, "depends", True) or "")
+    boot_depends = [dep.split(":")[0] for dep
+                in boot_depends_string.split()
+                if not dep.split(":")[0].endswith("-native")]
+    for dep in boot_depends:
+        pkgs_file = os.path.join(d.getVar('PKGDATA_DIR', True), dep)
+        # If the package and dependency name is the same
+        if os.path.exists(pkgs_file):
+            depends = "%s %s" % (depends, dep)
+        # We need to search for the provider of the dependency
+        else:
+            for taskdep in taskdepdata.itervalues():
+                # The fifth field contains what the task provides
+                if dep in taskdep[4]:
+                    pkgs_file = os.path.join(d.getVar('PKGDATA_DIR', True), taskdep[0])
+                    if os.path.isfile(pkgs_file):
+                        depends = "%s %s" % (depends, taskdep[0])
+                        break
+    return depends
+
 def write_license_files(d, license_manifest, pkg_dic):
     import re
 
-- 
1.8.4.5

[PATCH 3/6] license.bbclass: Added function get_deployed_dependencies

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This change introduce a new function to get the dependencies
that were deployed into the image. It uses BB_TASKDEPDATA
to get all the dependencies of the current task, so it is
possible to get different packages depending at what
point this function is called.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 46 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index d1bfe61..4e00170 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -49,6 +49,52 @@ python license_create_manifest() {
     write_license_files(d, rootfs_license_manifest, pkg_dic)
 }
 
+def get_deployed_dependencies(d):
+    """ Get all the deployed dependencies of an image """
+
+    deploy = {}
+    # Get all the dependencies for the current task (rootfs).
+    # Also get EXTRA_IMAGEDEPENDS because the bootloader is
+    # usually in this var and not listed in rootfs.
+    # At last, get the dependencies from boot classes because
+    # it might contain the bootloader.
+    taskdata = d.getVar("BB_TASKDEPDATA", True)
+    depends = list(set([dep[0] for dep
+                    in taskdata.itervalues()
+                    if not dep[0].endswith("-native")]))
+    extra_depends = d.getVar("EXTRA_IMAGEDEPENDS", True)
+    boot_depends = get_boot_dependencies(d)
+    depends.extend(extra_depends.split())
+    depends.extend(boot_depends.split())
+    depends = list(set(depends))
+
+    # To check what was deployed it check the rootfs dependencies against
+    # the SSTATE_MANIFESTS for "deploy" task.
+    # The manifest file name contains the arch. Because we are not running
+    # in the package context it is necessary to check every arch used.
+    sstate_manifest_dir = d.getVar("SSTATE_MANIFESTS", True)
+    sstate_archs = d.getVar("SSTATE_ARCHS", True)
+    extra_archs = d.getVar("PACKAGE_EXTRA_ARCHS", True)
+    archs = list(set(("%s %s" % (sstate_archs, extra_archs)).split()))
+    for dep in depends:
+        # Some packages have an arch on their own, so we try that first.
+        special_arch = d.getVar("PACKAGE_ARCH_pn-%s" % dep, True)
+        if special_arch:
+            sstate_manifest_file = os.path.join(sstate_manifest_dir,
+                    "manifest-%s-%s.deploy" % (special_arch, dep))
+            if os.path.exists(sstate_manifest_file):
+                deploy[dep] = sstate_manifest_file
+                continue
+
+        for arch in archs:
+            sstate_manifest_file = os.path.join(sstate_manifest_dir,
+                    "manifest-%s-%s.deploy" % (arch, dep))
+            if os.path.exists(sstate_manifest_file):
+                deploy[dep] = sstate_manifest_file
+                break
+
+    return deploy
+
 def get_boot_dependencies(d):
     """ Return the dependencies from boot tasks """
 
-- 
1.8.4.5

[PATCH 4/6] license.bbclass: Added get_package_from_deployed

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This function will verify if the dependency is
a package by itself or if the dependency installed
more packages. This is special useful with the kernels
because the dependency could be "linux-yocto" that in
fact installed other packages. This function will be
useful to get the version of the packages.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 4e00170..d384491 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -95,6 +95,21 @@ def get_deployed_dependencies(d):
 
     return deploy
 
+def get_package_from_deployed(d, dep):
+    """ Get just one package from a dependency """
+
+    import oe.packagedata
+
+    # Only one package is neccesary to get the version
+    data_file = os.path.join(d.getVar("PKGDATA_DIR",True),
+            "runtime", dep)
+    if os.path.exists(data_file):
+        return dep
+    else:
+        pkg_file = os.path.join(d.getVar("PKGDATA_DIR",True), dep)
+        pkgdata = oe.packagedata.read_pkgdatafile(pkg_file)
+        return pkgdata["PACKAGES"].split()[0]
+
 def get_boot_dependencies(d):
     """ Return the dependencies from boot tasks """
 
-- 
1.8.4.5

[PATCH 5/6] license.bbclass: Add function get_deployed_files

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This function will get the files that were deployed in
the image using the sstate-control manifest file. This
will give a better view of what was deployed in the image.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index d384491..463dd54 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -141,6 +141,19 @@ def get_boot_dependencies(d):
                         break
     return depends
 
+def get_deployed_files(d, man_file):
+    """ This will get the files deployed from the sstate manifest """
+
+    dep_files = ""
+    excluded_files = ["README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt"]
+    with open(man_file, "r") as manifest:
+        all_files = manifest.read()
+    for f in all_files.splitlines():
+        if ((not (os.path.islink(f) or os.path.isdir(f))) and
+                not os.path.basename(f) in excluded_files):
+            dep_files = "%s %s" % (dep_files, os.path.basename(f))
+    return dep_files
+
 def write_license_files(d, license_manifest, pkg_dic):
     import re
 
-- 
1.8.4.5

[PATCH 6/6] icense.bbclass: Create image license manifest

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This change adds the license_deployed_manifest function
that will create the manifest for the packages deployed
in the image but not installed in rootfs.

This new function was added to ROOTFS_POSTPROCESS_COMMAND
so it will run after every rootfs task. Because of this
it could run few times for a single build and get different
dependencies. Sometimes this dependencies won't include all
the deployed packages, in order to avoid missing licenses a
tmp file is create during the build and deleted after the
build (LICENSE_TMP_JSON).

This change also modify the write_license_files because
the image manifest is different from the root manifest.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 78 +++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 73 insertions(+), 5 deletions(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 463dd54..fa8807e 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -11,6 +11,7 @@ LICENSE_CREATE_PACKAGE[type] = "boolean"
 LICENSE_CREATE_PACKAGE ??= "0"
 LICENSE_PACKAGE_SUFFIX ??= "-lic"
 LICENSE_FILES_DIRECTORY ??= "${datadir}/licenses/"
+LICENSE_TMP_JSON ?= "${LICENSE_DIRECTORY}/deploy_packages.json"
 
 addtask populate_lic after do_patch before do_build
 do_populate_lic[dirs] = "${LICSSTATEDIR}/${PN}"
@@ -49,6 +50,59 @@ python license_create_manifest() {
     write_license_files(d, rootfs_license_manifest, pkg_dic)
 }
 
+python license_deployed_manifest_task() {
+    license_deployed_manifest(d)
+}
+
+def license_deployed_manifest(d):
+    """ Write the license manifest for the deployed packages.
+        The deployed packages usually includes the bootloader
+        and extra files to boot the target.
+    """
+    import json
+
+    packages = ""
+    dep_dic = {}
+    pkg_dic = {}
+    info_dir = os.path.join(d.getVar("PKGDATA_DIR",True), "runtime")
+
+    # Sometimes the initramfs image is build and it doesn't have
+    # the boot dependencies. In order to overcome this it is
+    # necessary to get save previous dependencies found. This is
+    # why the json_file is used (and deleted once the build is done)
+    json_file = d.getVar("LICENSE_TMP_JSON", True)
+    json_lock = bb.utils.lockfile("%s.lock" % json_file)
+    if os.path.exists(json_file):
+        with open(json_file) as f:
+            pkg_dic = json.loads(f.read())
+
+    dep_dic = get_deployed_dependencies(d)
+    for dep in dep_dic.keys():
+        # At least one package of the deployed dependency is needed
+        # to get the version.
+        pkg = get_package_from_deployed(d, dep)
+        if pkg and pkg not in pkg_dic.keys():
+            data_file = os.path.join(info_dir, pkg)
+            pkg_dic[pkg] = oe.packagedata.read_pkgdatafile(data_file)
+            # It is necessary to mark this will be used for image manifest
+            pkg_dic[pkg]["IMAGE_MANIFEST"] = True
+            pkg_dic[pkg]["FILES"] = \
+                    get_deployed_files(d, dep_dic[dep])
+            if not "LICENSE" in pkg_dic[pkg].keys():
+                pkg_lic = "LICENSE_" + pkg
+                pkg_dic[pkg]["LICENSE"] = pkg_dic[pkg][pkg_lic]
+
+    with open(json_file, "w") as f:
+        json.dump(pkg_dic, f, indent=4)
+    bb.utils.unlockfile(json_lock)
+
+    # Because it might be called several times we lock the license file
+    image_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True),
+            d.getVar('IMAGE_NAME', True), 'image_license.manifest')
+    manifest_lock = bb.utils.lockfile("%s.lock" % image_license_manifest)
+    write_license_files(d, image_license_manifest, pkg_dic)
+    bb.utils.unlockfile(manifest_lock)
+
 def get_deployed_dependencies(d):
     """ Get all the deployed dependencies of an image """
 
@@ -154,6 +208,11 @@ def get_deployed_files(d, man_file):
             dep_files = "%s %s" % (dep_files, os.path.basename(f))
     return dep_files
 
+python license_delete_tmp_files () {
+    json_file = d.getVar("LICENSE_TMP_JSON", True)
+    os.remove(json_file)
+}
+
 def write_license_files(d, license_manifest, pkg_dic):
     import re
 
@@ -175,10 +234,18 @@ def write_license_files(d, license_manifest, pkg_dic):
                 pkg_dic[pkg]["LICENSES"] = re.sub('  *', ' ', pkg_dic[pkg]["LICENSES"])
                 pkg_dic[pkg]["LICENSES"] = pkg_dic[pkg]["LICENSES"].split()
 
-            license_file.write("PACKAGE NAME: %s\n" % pkg)
-            license_file.write("PACKAGE VERSION: %s\n" % pkg_dic[pkg]["PV"])
-            license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"])
-            license_file.write("LICENSE: %s\n\n" % pkg_dic[pkg]["LICENSE"])
+            if not "IMAGE_MANIFEST" in pkg_dic[pkg]:
+                # Rootfs manifest
+                license_file.write("PACKAGE NAME: %s\n" % pkg)
+                license_file.write("PACKAGE VERSION: %s\n" % pkg_dic[pkg]["PV"])
+                license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"])
+                license_file.write("LICENSE: %s\n\n" % pkg_dic[pkg]["LICENSE"])
+            else:
+                # Image manifest
+                license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"])
+                license_file.write("VERSION: %s\n" % pkg_dic[pkg]["PV"])
+                license_file.write("LICENSE: %s\n" % pkg_dic[pkg]["LICENSE"])
+                license_file.write("FILES:%s\n\n" % pkg_dic[pkg]["FILES"])
 
             # If the package doesn't contain any file, that is, its size is 0, the license
             # isn't relevant as far as the final image is concerned. So doing license check
@@ -586,7 +653,8 @@ SSTATETASKS += "do_populate_lic"
 do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}"
 do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
 
-ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest; license_create_manifest; "
+ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest; license_create_manifest; license_deployed_manifest_task; "
+IMAGE_POSTPROCESS_COMMAND_prepend = " license_delete_tmp_files; "
 
 do_populate_lic_setscene[dirs] = "${LICSSTATEDIR}/${PN}"
 do_populate_lic_setscene[cleandirs] = "${LICSSTATEDIR}"
-- 
1.8.4.5

Re: [PATCH 6/6] icense.bbclass: Create image license manifest

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 367 bytes --]

On Mon, Nov 9, 2015 at 3:23 AM, <mariano.lopez@linux.intel.com> wrote:

> From: Mariano Lopez <mariano.lopez@linux.intel.com>
>

Minor, but the commit message subject line has a typo: icense.bbclass.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 785 bytes --]