[PATCHv2 0/6] Create image manifest

[PATCHv2 0/6] Create image manifest

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

When building the rootfs the rootfs is created but the recipes
that deployed files in the image doesn't show up. Usually these
recipes include the bootloaders and the kernel.

With these patches a new manifest will be created that includes
the deployed recipes that were in the deployed. The format of the
manifest is different because the recipes doesn't install packages,
instead the recipes deploy files into the image

[YOCTO #6772]

Changes in v2:
- Correct a typo in title of the last commit

The following changes since commit fc45deac89ef63ca1c44e763c38ced7dfd72cbe1:

  build-appliance-image: Update to jethro head revision (2015-11-03 14:03:03 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib mariano/bug6772
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug6772

Mariano Lopez (6):
  license.bbclass: Split license create manifest
  license.bbclass: Added get_boot_dependencies function
  license.bbclass: Added function get_deployed_dependencies
  license.bbclass: Added get_package_from_deployed
  license.bbclass: Add function get_deployed_files
  license.bbclass: Create image license manifest

 meta/classes/license.bbclass | 219 +++++++++++++++++++++++++++++++++++++++----
 1 file changed, 200 insertions(+), 19 deletions(-)

-- 
1.8.4.5

[PATCHv2 1/6] license.bbclass: Split license create manifest

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This changes moves the writing of the licenses to a
separated function that could be called for other packages.

With these change it will be easier to reuse the writing of
the license for the packages deployed but not installed in
the rootfs.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 36 ++++++++++++++++++++++--------------
 1 file changed, 22 insertions(+), 14 deletions(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index c616a20..98f1733 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -26,14 +26,9 @@ python write_package_manifest() {
 }
 
 python license_create_manifest() {
-    import re
     import oe.packagedata
     from oe.rootfs import image_list_installed_packages
 
-    bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE", True) or "").split()
-    bad_licenses = map(lambda l: canonical_license(d, l), bad_licenses)
-    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
-
     build_images_from_feeds = d.getVar('BUILD_IMAGES_FROM_FEEDS', True)
     if build_images_from_feeds == "1":
         return 0
@@ -49,8 +44,18 @@ python license_create_manifest() {
             pkg_lic_name = "LICENSE_" + pkg_name
             pkg_dic[pkg_name]["LICENSE"] = pkg_dic[pkg_name][pkg_lic_name]
 
-    license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True),
+    rootfs_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True),
                         d.getVar('IMAGE_NAME', True), 'license.manifest')
+    write_license_files(d, rootfs_license_manifest, pkg_dic)
+}
+
+def write_license_files(d, license_manifest, pkg_dic):
+    import re
+
+    bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE", True) or "").split()
+    bad_licenses = map(lambda l: canonical_license(d, l), bad_licenses)
+    bad_licenses = expand_wildcard_licenses(d, bad_licenses)
+
     with open(license_manifest, "w") as license_file:
         for pkg in sorted(pkg_dic):
             if bad_licenses:
@@ -98,15 +103,16 @@ python license_create_manifest() {
     if copy_lic_manifest == "1":
         rootfs_license_dir = os.path.join(d.getVar('IMAGE_ROOTFS', 'True'), 
                                 'usr', 'share', 'common-licenses')
-        os.makedirs(rootfs_license_dir)
+        bb.utils.mkdirhier(rootfs_license_dir)
         rootfs_license_manifest = os.path.join(rootfs_license_dir,
-                                                'license.manifest')
-        os.link(license_manifest, rootfs_license_manifest)
+                os.path.split(license_manifest)[1])
+        if not os.path.exists(rootfs_license_manifest):
+            os.link(license_manifest, rootfs_license_manifest)
 
         if copy_lic_dirs == "1":
             for pkg in sorted(pkg_dic):
                 pkg_rootfs_license_dir = os.path.join(rootfs_license_dir, pkg)
-                os.makedirs(pkg_rootfs_license_dir)
+                bb.utils.mkdirhier(pkg_rootfs_license_dir)
                 pkg_license_dir = os.path.join(d.getVar('LICENSE_DIRECTORY', True),
                                             pkg_dic[pkg]["PN"]) 
                 licenses = os.listdir(pkg_license_dir)
@@ -124,14 +130,16 @@ python license_create_manifest() {
                         if not os.path.exists(rootfs_license):
                             os.link(pkg_license, rootfs_license)
 
-                        os.symlink(os.path.join('..', lic), pkg_rootfs_license)
+                        if not os.path.exists(pkg_rootfs_license):
+                            os.symlink(os.path.join('..', lic), pkg_rootfs_license)
                     else:
-                        if oe.license.license_ok(canonical_license(d,
-                            lic), bad_licenses) == False:
+                        if (oe.license.license_ok(canonical_license(d,
+                                lic), bad_licenses) == False or
+                                os.path.exists(pkg_rootfs_license)):
                             continue
 
                         os.link(pkg_license, pkg_rootfs_license)
-}
+
 
 python do_populate_lic() {
     """
-- 
1.8.4.5

[PATCHv2 2/6] license.bbclass: Added get_boot_dependencies function

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This function gets the dependencies from the classes that
create a boot image, this is required because sometimes
the bootloader dependecy is in these classes. The current
classes covered are bootimg and bootdirectdisk because
these are the only clases that add dependencies.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 98f1733..d1bfe61 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -49,6 +49,37 @@ python license_create_manifest() {
     write_license_files(d, rootfs_license_manifest, pkg_dic)
 }
 
+def get_boot_dependencies(d):
+    """ Return the dependencies from boot tasks """
+
+    depends = ""
+    boot_depends_string = ""
+    taskdepdata = d.getVar("BB_TASKDEPDATA", True)
+    # Only bootimg and bootdirectdisk include the depends flag
+    boot_tasks = ["do_bootimg", "do_bootdirectdisk",]
+
+    for task in boot_tasks:
+        boot_depends_string = "%s %s" % (boot_depends_string,
+                d.getVarFlag(task, "depends", True) or "")
+    boot_depends = [dep.split(":")[0] for dep
+                in boot_depends_string.split()
+                if not dep.split(":")[0].endswith("-native")]
+    for dep in boot_depends:
+        pkgs_file = os.path.join(d.getVar('PKGDATA_DIR', True), dep)
+        # If the package and dependency name is the same
+        if os.path.exists(pkgs_file):
+            depends = "%s %s" % (depends, dep)
+        # We need to search for the provider of the dependency
+        else:
+            for taskdep in taskdepdata.itervalues():
+                # The fifth field contains what the task provides
+                if dep in taskdep[4]:
+                    pkgs_file = os.path.join(d.getVar('PKGDATA_DIR', True), taskdep[0])
+                    if os.path.isfile(pkgs_file):
+                        depends = "%s %s" % (depends, taskdep[0])
+                        break
+    return depends
+
 def write_license_files(d, license_manifest, pkg_dic):
     import re
 
-- 
1.8.4.5

Re: [PATCHv2 2/6] license.bbclass: Added get_boot_dependencies function

[Paul Eggleton]

Hi Mariano,

On Monday 09 November 2015 14:04:39 mariano.lopez@linux.intel.com wrote:
> From: Mariano Lopez <mariano.lopez@linux.intel.com>
> 
> This function gets the dependencies from the classes that
> create a boot image, this is required because sometimes
> the bootloader dependecy is in these classes. The current
> classes covered are bootimg and bootdirectdisk because
> these are the only clases that add dependencies.
> 
> [YOCTO #6772]
> 
> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
> ---
>  meta/classes/license.bbclass | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
> index 98f1733..d1bfe61 100644
> --- a/meta/classes/license.bbclass
> +++ b/meta/classes/license.bbclass
> @@ -49,6 +49,37 @@ python license_create_manifest() {
>      write_license_files(d, rootfs_license_manifest, pkg_dic)
>  }
> 
> +def get_boot_dependencies(d):
> +    """ Return the dependencies from boot tasks """
> +
> +    depends = ""
> +    boot_depends_string = ""
> +    taskdepdata = d.getVar("BB_TASKDEPDATA", True)
> +    # Only bootimg and bootdirectdisk include the depends flag
> +    boot_tasks = ["do_bootimg", "do_bootdirectdisk",]
> +
> +    for task in boot_tasks:
> +        boot_depends_string = "%s %s" % (boot_depends_string,
> +                d.getVarFlag(task, "depends", True) or "")
> +    boot_depends = [dep.split(":")[0] for dep
> +                in boot_depends_string.split()
> +                if not dep.split(":")[0].endswith("-native")]
> +    for dep in boot_depends:
> +        pkgs_file = os.path.join(d.getVar('PKGDATA_DIR', True), dep)
> +        # If the package and dependency name is the same
> +        if os.path.exists(pkgs_file):
> +            depends = "%s %s" % (depends, dep)
> +        # We need to search for the provider of the dependency
> +        else:
> +            for taskdep in taskdepdata.itervalues():
> +                # The fifth field contains what the task provides
> +                if dep in taskdep[4]:
> +                    pkgs_file = os.path.join(d.getVar('PKGDATA_DIR', True),
> taskdep[0]) 
> +                    if os.path.isfile(pkgs_file):
> +                        depends = "%s %s" % (depends, taskdep[0])
> +                        break

Can you explain what you're getting out of looking into PKGDATA_DIR here? I'm 
not sure this is correct.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCHv2 3/6] license.bbclass: Added function get_deployed_dependencies

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This change introduce a new function to get the dependencies
that were deployed into the image. It uses BB_TASKDEPDATA
to get all the dependencies of the current task, so it is
possible to get different packages depending at what
point this function is called.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 46 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index d1bfe61..4e00170 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -49,6 +49,52 @@ python license_create_manifest() {
     write_license_files(d, rootfs_license_manifest, pkg_dic)
 }
 
+def get_deployed_dependencies(d):
+    """ Get all the deployed dependencies of an image """
+
+    deploy = {}
+    # Get all the dependencies for the current task (rootfs).
+    # Also get EXTRA_IMAGEDEPENDS because the bootloader is
+    # usually in this var and not listed in rootfs.
+    # At last, get the dependencies from boot classes because
+    # it might contain the bootloader.
+    taskdata = d.getVar("BB_TASKDEPDATA", True)
+    depends = list(set([dep[0] for dep
+                    in taskdata.itervalues()
+                    if not dep[0].endswith("-native")]))
+    extra_depends = d.getVar("EXTRA_IMAGEDEPENDS", True)
+    boot_depends = get_boot_dependencies(d)
+    depends.extend(extra_depends.split())
+    depends.extend(boot_depends.split())
+    depends = list(set(depends))
+
+    # To check what was deployed it check the rootfs dependencies against
+    # the SSTATE_MANIFESTS for "deploy" task.
+    # The manifest file name contains the arch. Because we are not running
+    # in the package context it is necessary to check every arch used.
+    sstate_manifest_dir = d.getVar("SSTATE_MANIFESTS", True)
+    sstate_archs = d.getVar("SSTATE_ARCHS", True)
+    extra_archs = d.getVar("PACKAGE_EXTRA_ARCHS", True)
+    archs = list(set(("%s %s" % (sstate_archs, extra_archs)).split()))
+    for dep in depends:
+        # Some packages have an arch on their own, so we try that first.
+        special_arch = d.getVar("PACKAGE_ARCH_pn-%s" % dep, True)
+        if special_arch:
+            sstate_manifest_file = os.path.join(sstate_manifest_dir,
+                    "manifest-%s-%s.deploy" % (special_arch, dep))
+            if os.path.exists(sstate_manifest_file):
+                deploy[dep] = sstate_manifest_file
+                continue
+
+        for arch in archs:
+            sstate_manifest_file = os.path.join(sstate_manifest_dir,
+                    "manifest-%s-%s.deploy" % (arch, dep))
+            if os.path.exists(sstate_manifest_file):
+                deploy[dep] = sstate_manifest_file
+                break
+
+    return deploy
+
 def get_boot_dependencies(d):
     """ Return the dependencies from boot tasks """
 
-- 
1.8.4.5

[PATCHv2 4/6] license.bbclass: Added get_package_from_deployed

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This function will verify if the dependency is
a package by itself or if the dependency installed
more packages. This is special useful with the kernels
because the dependency could be "linux-yocto" that in
fact installed other packages. This function will be
useful to get the version of the packages.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 4e00170..d384491 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -95,6 +95,21 @@ def get_deployed_dependencies(d):
 
     return deploy
 
+def get_package_from_deployed(d, dep):
+    """ Get just one package from a dependency """
+
+    import oe.packagedata
+
+    # Only one package is neccesary to get the version
+    data_file = os.path.join(d.getVar("PKGDATA_DIR",True),
+            "runtime", dep)
+    if os.path.exists(data_file):
+        return dep
+    else:
+        pkg_file = os.path.join(d.getVar("PKGDATA_DIR",True), dep)
+        pkgdata = oe.packagedata.read_pkgdatafile(pkg_file)
+        return pkgdata["PACKAGES"].split()[0]
+
 def get_boot_dependencies(d):
     """ Return the dependencies from boot tasks """
 
-- 
1.8.4.5

[PATCHv2 5/6] license.bbclass: Add function get_deployed_files

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This function will get the files that were deployed in
the image using the sstate-control manifest file. This
will give a better view of what was deployed in the image.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index d384491..463dd54 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -141,6 +141,19 @@ def get_boot_dependencies(d):
                         break
     return depends
 
+def get_deployed_files(d, man_file):
+    """ This will get the files deployed from the sstate manifest """
+
+    dep_files = ""
+    excluded_files = ["README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt"]
+    with open(man_file, "r") as manifest:
+        all_files = manifest.read()
+    for f in all_files.splitlines():
+        if ((not (os.path.islink(f) or os.path.isdir(f))) and
+                not os.path.basename(f) in excluded_files):
+            dep_files = "%s %s" % (dep_files, os.path.basename(f))
+    return dep_files
+
 def write_license_files(d, license_manifest, pkg_dic):
     import re
 
-- 
1.8.4.5

Re: [PATCHv2 5/6] license.bbclass: Add function get_deployed_files

[Paul Eggleton]

On Monday 09 November 2015 14:04:42 mariano.lopez@linux.intel.com wrote:
> From: Mariano Lopez <mariano.lopez@linux.intel.com>
> 
> This function will get the files that were deployed in
> the image using the sstate-control manifest file. This
> will give a better view of what was deployed in the image.

Not *in* the image, next to it.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCHv2 6/6] license.bbclass: Create image license manifest

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

This change adds the license_deployed_manifest function
that will create the manifest for the packages deployed
in the image but not installed in rootfs.

This new function was added to ROOTFS_POSTPROCESS_COMMAND
so it will run after every rootfs task. Because of this
it could run few times for a single build and get different
dependencies. Sometimes this dependencies won't include all
the deployed packages, in order to avoid missing licenses a
tmp file is create during the build and deleted after the
build (LICENSE_TMP_JSON).

This change also modify the write_license_files because
the image manifest is different from the root manifest.

[YOCTO #6772]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/license.bbclass | 78 +++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 73 insertions(+), 5 deletions(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 463dd54..fa8807e 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -11,6 +11,7 @@ LICENSE_CREATE_PACKAGE[type] = "boolean"
 LICENSE_CREATE_PACKAGE ??= "0"
 LICENSE_PACKAGE_SUFFIX ??= "-lic"
 LICENSE_FILES_DIRECTORY ??= "${datadir}/licenses/"
+LICENSE_TMP_JSON ?= "${LICENSE_DIRECTORY}/deploy_packages.json"
 
 addtask populate_lic after do_patch before do_build
 do_populate_lic[dirs] = "${LICSSTATEDIR}/${PN}"
@@ -49,6 +50,59 @@ python license_create_manifest() {
     write_license_files(d, rootfs_license_manifest, pkg_dic)
 }
 
+python license_deployed_manifest_task() {
+    license_deployed_manifest(d)
+}
+
+def license_deployed_manifest(d):
+    """ Write the license manifest for the deployed packages.
+        The deployed packages usually includes the bootloader
+        and extra files to boot the target.
+    """
+    import json
+
+    packages = ""
+    dep_dic = {}
+    pkg_dic = {}
+    info_dir = os.path.join(d.getVar("PKGDATA_DIR",True), "runtime")
+
+    # Sometimes the initramfs image is build and it doesn't have
+    # the boot dependencies. In order to overcome this it is
+    # necessary to get save previous dependencies found. This is
+    # why the json_file is used (and deleted once the build is done)
+    json_file = d.getVar("LICENSE_TMP_JSON", True)
+    json_lock = bb.utils.lockfile("%s.lock" % json_file)
+    if os.path.exists(json_file):
+        with open(json_file) as f:
+            pkg_dic = json.loads(f.read())
+
+    dep_dic = get_deployed_dependencies(d)
+    for dep in dep_dic.keys():
+        # At least one package of the deployed dependency is needed
+        # to get the version.
+        pkg = get_package_from_deployed(d, dep)
+        if pkg and pkg not in pkg_dic.keys():
+            data_file = os.path.join(info_dir, pkg)
+            pkg_dic[pkg] = oe.packagedata.read_pkgdatafile(data_file)
+            # It is necessary to mark this will be used for image manifest
+            pkg_dic[pkg]["IMAGE_MANIFEST"] = True
+            pkg_dic[pkg]["FILES"] = \
+                    get_deployed_files(d, dep_dic[dep])
+            if not "LICENSE" in pkg_dic[pkg].keys():
+                pkg_lic = "LICENSE_" + pkg
+                pkg_dic[pkg]["LICENSE"] = pkg_dic[pkg][pkg_lic]
+
+    with open(json_file, "w") as f:
+        json.dump(pkg_dic, f, indent=4)
+    bb.utils.unlockfile(json_lock)
+
+    # Because it might be called several times we lock the license file
+    image_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True),
+            d.getVar('IMAGE_NAME', True), 'image_license.manifest')
+    manifest_lock = bb.utils.lockfile("%s.lock" % image_license_manifest)
+    write_license_files(d, image_license_manifest, pkg_dic)
+    bb.utils.unlockfile(manifest_lock)
+
 def get_deployed_dependencies(d):
     """ Get all the deployed dependencies of an image """
 
@@ -154,6 +208,11 @@ def get_deployed_files(d, man_file):
             dep_files = "%s %s" % (dep_files, os.path.basename(f))
     return dep_files
 
+python license_delete_tmp_files () {
+    json_file = d.getVar("LICENSE_TMP_JSON", True)
+    os.remove(json_file)
+}
+
 def write_license_files(d, license_manifest, pkg_dic):
     import re
 
@@ -175,10 +234,18 @@ def write_license_files(d, license_manifest, pkg_dic):
                 pkg_dic[pkg]["LICENSES"] = re.sub('  *', ' ', pkg_dic[pkg]["LICENSES"])
                 pkg_dic[pkg]["LICENSES"] = pkg_dic[pkg]["LICENSES"].split()
 
-            license_file.write("PACKAGE NAME: %s\n" % pkg)
-            license_file.write("PACKAGE VERSION: %s\n" % pkg_dic[pkg]["PV"])
-            license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"])
-            license_file.write("LICENSE: %s\n\n" % pkg_dic[pkg]["LICENSE"])
+            if not "IMAGE_MANIFEST" in pkg_dic[pkg]:
+                # Rootfs manifest
+                license_file.write("PACKAGE NAME: %s\n" % pkg)
+                license_file.write("PACKAGE VERSION: %s\n" % pkg_dic[pkg]["PV"])
+                license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"])
+                license_file.write("LICENSE: %s\n\n" % pkg_dic[pkg]["LICENSE"])
+            else:
+                # Image manifest
+                license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"])
+                license_file.write("VERSION: %s\n" % pkg_dic[pkg]["PV"])
+                license_file.write("LICENSE: %s\n" % pkg_dic[pkg]["LICENSE"])
+                license_file.write("FILES:%s\n\n" % pkg_dic[pkg]["FILES"])
 
             # If the package doesn't contain any file, that is, its size is 0, the license
             # isn't relevant as far as the final image is concerned. So doing license check
@@ -586,7 +653,8 @@ SSTATETASKS += "do_populate_lic"
 do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}"
 do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
 
-ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest; license_create_manifest; "
+ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest; license_create_manifest; license_deployed_manifest_task; "
+IMAGE_POSTPROCESS_COMMAND_prepend = " license_delete_tmp_files; "
 
 do_populate_lic_setscene[dirs] = "${LICSSTATEDIR}/${PN}"
 do_populate_lic_setscene[cleandirs] = "${LICSSTATEDIR}"
-- 
1.8.4.5

Re: [PATCHv2 6/6] license.bbclass: Create image license manifest

[Paul Eggleton]

On Monday 09 November 2015 14:04:43 mariano.lopez@linux.intel.com wrote:
> From: Mariano Lopez <mariano.lopez@linux.intel.com>
> 
> This change adds the license_deployed_manifest function
> that will create the manifest for the packages deployed
> in the image but not installed in rootfs.

Again, not "in the image" but "next to the image". I'd suggest giving an 
example or two here so that it's clear what's being referred to.

> This new function was added to ROOTFS_POSTPROCESS_COMMAND
> so it will run after every rootfs task. Because of this
> it could run few times for a single build and get different
> dependencies. Sometimes this dependencies won't include all
> the deployed packages, in order to avoid missing licenses a
> tmp file is create during the build and deleted after the
> build (LICENSE_TMP_JSON).
> 
> This change also modify the write_license_files because
> the image manifest is different from the root manifest.
> 
> [YOCTO #6772]
> 
> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
> ---
>  meta/classes/license.bbclass | 78
> +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 73
> insertions(+), 5 deletions(-)
> 
> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
> index 463dd54..fa8807e 100644
> --- a/meta/classes/license.bbclass
> +++ b/meta/classes/license.bbclass
> @@ -11,6 +11,7 @@ LICENSE_CREATE_PACKAGE[type] = "boolean"
>  LICENSE_CREATE_PACKAGE ??= "0"
>  LICENSE_PACKAGE_SUFFIX ??= "-lic"
>  LICENSE_FILES_DIRECTORY ??= "${datadir}/licenses/"
> +LICENSE_TMP_JSON ?= "${LICENSE_DIRECTORY}/deploy_packages.json"
> 
>  addtask populate_lic after do_patch before do_build
>  do_populate_lic[dirs] = "${LICSSTATEDIR}/${PN}"
> @@ -49,6 +50,59 @@ python license_create_manifest() {
>      write_license_files(d, rootfs_license_manifest, pkg_dic)
>  }
> 
> +python license_deployed_manifest_task() {
> +    license_deployed_manifest(d)
> +}
> +
> +def license_deployed_manifest(d):
> +    """ Write the license manifest for the deployed packages.
> +        The deployed packages usually includes the bootloader
> +        and extra files to boot the target.
> +    """
> +    import json
> +
> +    packages = ""
> +    dep_dic = {}
> +    pkg_dic = {}
> +    info_dir = os.path.join(d.getVar("PKGDATA_DIR",True), "runtime")
> +
> +    # Sometimes the initramfs image is build and it doesn't have
> +    # the boot dependencies. In order to overcome this it is
> +    # necessary to get save previous dependencies found. This is
> +    # why the json_file is used (and deleted once the build is done)
> +    json_file = d.getVar("LICENSE_TMP_JSON", True)
> +    json_lock = bb.utils.lockfile("%s.lock" % json_file)
> +    if os.path.exists(json_file):
> +        with open(json_file) as f:
> +            pkg_dic = json.loads(f.read())
> +
> +    dep_dic = get_deployed_dependencies(d)
> +    for dep in dep_dic.keys():
> +        # At least one package of the deployed dependency is needed
> +        # to get the version.
> +        pkg = get_package_from_deployed(d, dep)
> +        if pkg and pkg not in pkg_dic.keys():
> +            data_file = os.path.join(info_dir, pkg)
> +            pkg_dic[pkg] = oe.packagedata.read_pkgdatafile(data_file)
> +            # It is necessary to mark this will be used for image manifest
> +            pkg_dic[pkg]["IMAGE_MANIFEST"] = True
> +            pkg_dic[pkg]["FILES"] = \
> +                    get_deployed_files(d, dep_dic[dep])
> +            if not "LICENSE" in pkg_dic[pkg].keys():
> +                pkg_lic = "LICENSE_" + pkg
> +                pkg_dic[pkg]["LICENSE"] = pkg_dic[pkg][pkg_lic]
> +
> +    with open(json_file, "w") as f:
> +        json.dump(pkg_dic, f, indent=4)
> +    bb.utils.unlockfile(json_lock)
> +
> +    # Because it might be called several times we lock the license file
> +    image_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY',
> True), +            d.getVar('IMAGE_NAME', True), 'image_license.manifest')
> +    manifest_lock = bb.utils.lockfile("%s.lock" % image_license_manifest)
> +    write_license_files(d, image_license_manifest, pkg_dic)
> +    bb.utils.unlockfile(manifest_lock)
> +
>  def get_deployed_dependencies(d):
>      """ Get all the deployed dependencies of an image """
> 
> @@ -154,6 +208,11 @@ def get_deployed_files(d, man_file):
>              dep_files = "%s %s" % (dep_files, os.path.basename(f))
>      return dep_files
> 
> +python license_delete_tmp_files () {
> +    json_file = d.getVar("LICENSE_TMP_JSON", True)
> +    os.remove(json_file)
> +}
> +
>  def write_license_files(d, license_manifest, pkg_dic):
>      import re
> 
> @@ -175,10 +234,18 @@ def write_license_files(d, license_manifest, pkg_dic):
> pkg_dic[pkg]["LICENSES"] = re.sub('  *', ' ', pkg_dic[pkg]["LICENSES"])
> pkg_dic[pkg]["LICENSES"] = pkg_dic[pkg]["LICENSES"].split()
> 
> -            license_file.write("PACKAGE NAME: %s\n" % pkg)
> -            license_file.write("PACKAGE VERSION: %s\n" %
> pkg_dic[pkg]["PV"]) -            license_file.write("RECIPE NAME: %s\n" %
> pkg_dic[pkg]["PN"]) -            license_file.write("LICENSE: %s\n\n" %
> pkg_dic[pkg]["LICENSE"]) +            if not "IMAGE_MANIFEST" in
> pkg_dic[pkg]:
> +                # Rootfs manifest
> +                license_file.write("PACKAGE NAME: %s\n" % pkg)
> +                license_file.write("PACKAGE VERSION: %s\n" %
> pkg_dic[pkg]["PV"]) +                license_file.write("RECIPE NAME: %s\n"
> % pkg_dic[pkg]["PN"]) +                license_file.write("LICENSE: %s\n\n"
> % pkg_dic[pkg]["LICENSE"]) +            else:
> +                # Image manifest
> +                license_file.write("RECIPE NAME: %s\n" %
> pkg_dic[pkg]["PN"]) +                license_file.write("VERSION: %s\n" %
> pkg_dic[pkg]["PV"]) +                license_file.write("LICENSE: %s\n" %
> pkg_dic[pkg]["LICENSE"]) +                license_file.write("FILES:%s\n\n"
> % pkg_dic[pkg]["FILES"])
> 
>              # If the package doesn't contain any file, that is, its size is
> 0, the license # isn't relevant as far as the final image is concerned. So
> doing license check @@ -586,7 +653,8 @@ SSTATETASKS += "do_populate_lic"
>  do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}"
>  do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
> 
> -ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest;
> license_create_manifest; " +ROOTFS_POSTPROCESS_COMMAND_prepend =
> "write_package_manifest; license_create_manifest;
> license_deployed_manifest_task; " +IMAGE_POSTPROCESS_COMMAND_prepend = "
> license_delete_tmp_files; "
> 
>  do_populate_lic_setscene[dirs] = "${LICSSTATEDIR}/${PN}"
>  do_populate_lic_setscene[cleandirs] = "${LICSSTATEDIR}"

A couple of things:

1) Please take care to use the appropriate terminology. "package" has a 
specific meaning; what we are dealing with here is files deployed directly from 
recipes, packages aren't involved (or shouldn't be).

2) You're looking into PKGDATA_DIR, as I mentioned in one of my other replies 
I don't think this is right. The risk is that if a recipe doesn't actually 
write out any packages, the manifest will be incomplete. We need a different 
approach here where pkgdata isn't involved at all. IIRC my earlier suggestion 
was to use what is written out by do_populate_lic, i.e. tmp/deploy/licenses/ - 
if that isn't practical we can have that task write out the LICENSE value to a 
separate file, and that will then work for any recipe.

Cheers,
Paul 

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCHv2 6/6] license.bbclass: Create image license manifest

[Mariano Lopez]

On 11/10/2015 04:42 AM, Paul Eggleton wrote:
> On Monday 09 November 2015 14:04:43 mariano.lopez@linux.intel.com wrote:
>> From: Mariano Lopez <mariano.lopez@linux.intel.com>
>>
>> This change adds the license_deployed_manifest function
>> that will create the manifest for the packages deployed
>> in the image but not installed in rootfs.
> Again, not "in the image" but "next to the image". I'd suggest giving an
> example or two here so that it's clear what's being referred to.
>
>> This new function was added to ROOTFS_POSTPROCESS_COMMAND
>> so it will run after every rootfs task. Because of this
>> it could run few times for a single build and get different
>> dependencies. Sometimes this dependencies won't include all
>> the deployed packages, in order to avoid missing licenses a
>> tmp file is create during the build and deleted after the
>> build (LICENSE_TMP_JSON).
>>
>> This change also modify the write_license_files because
>> the image manifest is different from the root manifest.
>>
>> [YOCTO #6772]
>>
>> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
>> ---
>>   meta/classes/license.bbclass | 78
>> +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 73
>> insertions(+), 5 deletions(-)
>>
>> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
>> index 463dd54..fa8807e 100644
>> --- a/meta/classes/license.bbclass
>> +++ b/meta/classes/license.bbclass
>> @@ -11,6 +11,7 @@ LICENSE_CREATE_PACKAGE[type] = "boolean"
>>   LICENSE_CREATE_PACKAGE ??= "0"
>>   LICENSE_PACKAGE_SUFFIX ??= "-lic"
>>   LICENSE_FILES_DIRECTORY ??= "${datadir}/licenses/"
>> +LICENSE_TMP_JSON ?= "${LICENSE_DIRECTORY}/deploy_packages.json"
>>
>>   addtask populate_lic after do_patch before do_build
>>   do_populate_lic[dirs] = "${LICSSTATEDIR}/${PN}"
>> @@ -49,6 +50,59 @@ python license_create_manifest() {
>>       write_license_files(d, rootfs_license_manifest, pkg_dic)
>>   }
>>
>> +python license_deployed_manifest_task() {
>> +    license_deployed_manifest(d)
>> +}
>> +
>> +def license_deployed_manifest(d):
>> +    """ Write the license manifest for the deployed packages.
>> +        The deployed packages usually includes the bootloader
>> +        and extra files to boot the target.
>> +    """
>> +    import json
>> +
>> +    packages = ""
>> +    dep_dic = {}
>> +    pkg_dic = {}
>> +    info_dir = os.path.join(d.getVar("PKGDATA_DIR",True), "runtime")
>> +
>> +    # Sometimes the initramfs image is build and it doesn't have
>> +    # the boot dependencies. In order to overcome this it is
>> +    # necessary to get save previous dependencies found. This is
>> +    # why the json_file is used (and deleted once the build is done)
>> +    json_file = d.getVar("LICENSE_TMP_JSON", True)
>> +    json_lock = bb.utils.lockfile("%s.lock" % json_file)
>> +    if os.path.exists(json_file):
>> +        with open(json_file) as f:
>> +            pkg_dic = json.loads(f.read())
>> +
>> +    dep_dic = get_deployed_dependencies(d)
>> +    for dep in dep_dic.keys():
>> +        # At least one package of the deployed dependency is needed
>> +        # to get the version.
>> +        pkg = get_package_from_deployed(d, dep)
>> +        if pkg and pkg not in pkg_dic.keys():
>> +            data_file = os.path.join(info_dir, pkg)
>> +            pkg_dic[pkg] = oe.packagedata.read_pkgdatafile(data_file)
>> +            # It is necessary to mark this will be used for image manifest
>> +            pkg_dic[pkg]["IMAGE_MANIFEST"] = True
>> +            pkg_dic[pkg]["FILES"] = \
>> +                    get_deployed_files(d, dep_dic[dep])
>> +            if not "LICENSE" in pkg_dic[pkg].keys():
>> +                pkg_lic = "LICENSE_" + pkg
>> +                pkg_dic[pkg]["LICENSE"] = pkg_dic[pkg][pkg_lic]
>> +
>> +    with open(json_file, "w") as f:
>> +        json.dump(pkg_dic, f, indent=4)
>> +    bb.utils.unlockfile(json_lock)
>> +
>> +    # Because it might be called several times we lock the license file
>> +    image_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY',
>> True), +            d.getVar('IMAGE_NAME', True), 'image_license.manifest')
>> +    manifest_lock = bb.utils.lockfile("%s.lock" % image_license_manifest)
>> +    write_license_files(d, image_license_manifest, pkg_dic)
>> +    bb.utils.unlockfile(manifest_lock)
>> +
>>   def get_deployed_dependencies(d):
>>       """ Get all the deployed dependencies of an image """
>>
>> @@ -154,6 +208,11 @@ def get_deployed_files(d, man_file):
>>               dep_files = "%s %s" % (dep_files, os.path.basename(f))
>>       return dep_files
>>
>> +python license_delete_tmp_files () {
>> +    json_file = d.getVar("LICENSE_TMP_JSON", True)
>> +    os.remove(json_file)
>> +}
>> +
>>   def write_license_files(d, license_manifest, pkg_dic):
>>       import re
>>
>> @@ -175,10 +234,18 @@ def write_license_files(d, license_manifest, pkg_dic):
>> pkg_dic[pkg]["LICENSES"] = re.sub('  *', ' ', pkg_dic[pkg]["LICENSES"])
>> pkg_dic[pkg]["LICENSES"] = pkg_dic[pkg]["LICENSES"].split()
>>
>> -            license_file.write("PACKAGE NAME: %s\n" % pkg)
>> -            license_file.write("PACKAGE VERSION: %s\n" %
>> pkg_dic[pkg]["PV"]) -            license_file.write("RECIPE NAME: %s\n" %
>> pkg_dic[pkg]["PN"]) -            license_file.write("LICENSE: %s\n\n" %
>> pkg_dic[pkg]["LICENSE"]) +            if not "IMAGE_MANIFEST" in
>> pkg_dic[pkg]:
>> +                # Rootfs manifest
>> +                license_file.write("PACKAGE NAME: %s\n" % pkg)
>> +                license_file.write("PACKAGE VERSION: %s\n" %
>> pkg_dic[pkg]["PV"]) +                license_file.write("RECIPE NAME: %s\n"
>> % pkg_dic[pkg]["PN"]) +                license_file.write("LICENSE: %s\n\n"
>> % pkg_dic[pkg]["LICENSE"]) +            else:
>> +                # Image manifest
>> +                license_file.write("RECIPE NAME: %s\n" %
>> pkg_dic[pkg]["PN"]) +                license_file.write("VERSION: %s\n" %
>> pkg_dic[pkg]["PV"]) +                license_file.write("LICENSE: %s\n" %
>> pkg_dic[pkg]["LICENSE"]) +                license_file.write("FILES:%s\n\n"
>> % pkg_dic[pkg]["FILES"])
>>
>>               # If the package doesn't contain any file, that is, its size is
>> 0, the license # isn't relevant as far as the final image is concerned. So
>> doing license check @@ -586,7 +653,8 @@ SSTATETASKS += "do_populate_lic"
>>   do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}"
>>   do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
>>
>> -ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest;
>> license_create_manifest; " +ROOTFS_POSTPROCESS_COMMAND_prepend =
>> "write_package_manifest; license_create_manifest;
>> license_deployed_manifest_task; " +IMAGE_POSTPROCESS_COMMAND_prepend = "
>> license_delete_tmp_files; "
>>
>>   do_populate_lic_setscene[dirs] = "${LICSSTATEDIR}/${PN}"
>>   do_populate_lic_setscene[cleandirs] = "${LICSSTATEDIR}"
> A couple of things:
>
> 1) Please take care to use the appropriate terminology. "package" has a
> specific meaning; what we are dealing with here is files deployed directly from
> recipes, packages aren't involved (or shouldn't be).

The terminology here is used because I reused the code to write rootfs 
manifest. I can change the terminology of this function or I can write a 
function just for the image manifest.

>
> 2) You're looking into PKGDATA_DIR, as I mentioned in one of my other replies
> I don't think this is right. The risk is that if a recipe doesn't actually
> write out any packages, the manifest will be incomplete. We need a different
> approach here where pkgdata isn't involved at all. IIRC my earlier suggestion
> was to use what is written out by do_populate_lic, i.e. tmp/deploy/licenses/ -
> if that isn't practical we can have that task write out the LICENSE value to a
> separate file, and that will then work for any recipe.

The use of PKGDATA_DIR is for the version of the deployed files, getting 
the license from here was just because it was practical. Is there 
another way to get the version without using PKGDATA_DIR?

>
> Cheers,
> Paul
>
Regards,
Mariano Lopez

Re: [PATCHv2 6/6] license.bbclass: Create image license manifest

[Paul Eggleton]

On Tuesday 10 November 2015 08:52:22 Mariano Lopez wrote:
> On 11/10/2015 04:42 AM, Paul Eggleton wrote:
> > On Monday 09 November 2015 14:04:43 mariano.lopez@linux.intel.com wrote:
> >> From: Mariano Lopez <mariano.lopez@linux.intel.com>
> >> 
> >> This change adds the license_deployed_manifest function
> >> that will create the manifest for the packages deployed
> >> in the image but not installed in rootfs.
> > 
> > Again, not "in the image" but "next to the image". I'd suggest giving an
> > example or two here so that it's clear what's being referred to.
> > 
> >> This new function was added to ROOTFS_POSTPROCESS_COMMAND
> >> so it will run after every rootfs task. Because of this
> >> it could run few times for a single build and get different
> >> dependencies. Sometimes this dependencies won't include all
> >> the deployed packages, in order to avoid missing licenses a
> >> tmp file is create during the build and deleted after the
> >> build (LICENSE_TMP_JSON).
> >> 
> >> This change also modify the write_license_files because
> >> the image manifest is different from the root manifest.
> >> 
> >> [YOCTO #6772]
> >> 
> >> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
> >> ---
> >> 
> >>   meta/classes/license.bbclass | 78
> >> 
> >> +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 73
> >> insertions(+), 5 deletions(-)
> >> 
> >> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
> >> index 463dd54..fa8807e 100644
> >> --- a/meta/classes/license.bbclass
> >> +++ b/meta/classes/license.bbclass
> >> @@ -11,6 +11,7 @@ LICENSE_CREATE_PACKAGE[type] = "boolean"
> >> 
> >>   LICENSE_CREATE_PACKAGE ??= "0"
> >>   LICENSE_PACKAGE_SUFFIX ??= "-lic"
> >>   LICENSE_FILES_DIRECTORY ??= "${datadir}/licenses/"
> >> 
> >> +LICENSE_TMP_JSON ?= "${LICENSE_DIRECTORY}/deploy_packages.json"
> >> 
> >>   addtask populate_lic after do_patch before do_build
> >>   do_populate_lic[dirs] = "${LICSSTATEDIR}/${PN}"
> >> 
> >> @@ -49,6 +50,59 @@ python license_create_manifest() {
> >> 
> >>       write_license_files(d, rootfs_license_manifest, pkg_dic)
> >>   
> >>   }
> >> 
> >> +python license_deployed_manifest_task() {
> >> +    license_deployed_manifest(d)
> >> +}
> >> +
> >> +def license_deployed_manifest(d):
> >> +    """ Write the license manifest for the deployed packages.
> >> +        The deployed packages usually includes the bootloader
> >> +        and extra files to boot the target.
> >> +    """
> >> +    import json
> >> +
> >> +    packages = ""
> >> +    dep_dic = {}
> >> +    pkg_dic = {}
> >> +    info_dir = os.path.join(d.getVar("PKGDATA_DIR",True), "runtime")
> >> +
> >> +    # Sometimes the initramfs image is build and it doesn't have
> >> +    # the boot dependencies. In order to overcome this it is
> >> +    # necessary to get save previous dependencies found. This is
> >> +    # why the json_file is used (and deleted once the build is done)
> >> +    json_file = d.getVar("LICENSE_TMP_JSON", True)
> >> +    json_lock = bb.utils.lockfile("%s.lock" % json_file)
> >> +    if os.path.exists(json_file):
> >> +        with open(json_file) as f:
> >> +            pkg_dic = json.loads(f.read())
> >> +
> >> +    dep_dic = get_deployed_dependencies(d)
> >> +    for dep in dep_dic.keys():
> >> +        # At least one package of the deployed dependency is needed
> >> +        # to get the version.
> >> +        pkg = get_package_from_deployed(d, dep)
> >> +        if pkg and pkg not in pkg_dic.keys():
> >> +            data_file = os.path.join(info_dir, pkg)
> >> +            pkg_dic[pkg] = oe.packagedata.read_pkgdatafile(data_file)
> >> +            # It is necessary to mark this will be used for image
> >> manifest
> >> +            pkg_dic[pkg]["IMAGE_MANIFEST"] = True
> >> +            pkg_dic[pkg]["FILES"] = \
> >> +                    get_deployed_files(d, dep_dic[dep])
> >> +            if not "LICENSE" in pkg_dic[pkg].keys():
> >> +                pkg_lic = "LICENSE_" + pkg
> >> +                pkg_dic[pkg]["LICENSE"] = pkg_dic[pkg][pkg_lic]
> >> +
> >> +    with open(json_file, "w") as f:
> >> +        json.dump(pkg_dic, f, indent=4)
> >> +    bb.utils.unlockfile(json_lock)
> >> +
> >> +    # Because it might be called several times we lock the license file
> >> +    image_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY',
> >> True), +            d.getVar('IMAGE_NAME', True),
> >> 'image_license.manifest')
> >> +    manifest_lock = bb.utils.lockfile("%s.lock" %
> >> image_license_manifest)
> >> +    write_license_files(d, image_license_manifest, pkg_dic)
> >> +    bb.utils.unlockfile(manifest_lock)
> >> +
> >> 
> >>   def get_deployed_dependencies(d):
> >>       """ Get all the deployed dependencies of an image """
> >> 
> >> @@ -154,6 +208,11 @@ def get_deployed_files(d, man_file):
> >>               dep_files = "%s %s" % (dep_files, os.path.basename(f))
> >>       
> >>       return dep_files
> >> 
> >> +python license_delete_tmp_files () {
> >> +    json_file = d.getVar("LICENSE_TMP_JSON", True)
> >> +    os.remove(json_file)
> >> +}
> >> +
> >> 
> >>   def write_license_files(d, license_manifest, pkg_dic):
> >>       import re
> >> 
> >> @@ -175,10 +234,18 @@ def write_license_files(d, license_manifest,
> >> pkg_dic): pkg_dic[pkg]["LICENSES"] = re.sub('  *', ' ',
> >> pkg_dic[pkg]["LICENSES"]) pkg_dic[pkg]["LICENSES"] =
> >> pkg_dic[pkg]["LICENSES"].split()
> >> 
> >> -            license_file.write("PACKAGE NAME: %s\n" % pkg)
> >> -            license_file.write("PACKAGE VERSION: %s\n" %
> >> pkg_dic[pkg]["PV"]) -            license_file.write("RECIPE NAME: %s\n" %
> >> pkg_dic[pkg]["PN"]) -            license_file.write("LICENSE: %s\n\n" %
> >> pkg_dic[pkg]["LICENSE"]) +            if not "IMAGE_MANIFEST" in
> >> pkg_dic[pkg]:
> >> +                # Rootfs manifest
> >> +                license_file.write("PACKAGE NAME: %s\n" % pkg)
> >> +                license_file.write("PACKAGE VERSION: %s\n" %
> >> pkg_dic[pkg]["PV"]) +                license_file.write("RECIPE NAME:
> >> %s\n"
> >> % pkg_dic[pkg]["PN"]) +                license_file.write("LICENSE:
> >> %s\n\n"
> >> % pkg_dic[pkg]["LICENSE"]) +            else:
> >> +                # Image manifest
> >> +                license_file.write("RECIPE NAME: %s\n" %
> >> pkg_dic[pkg]["PN"]) +                license_file.write("VERSION: %s\n" %
> >> pkg_dic[pkg]["PV"]) +                license_file.write("LICENSE: %s\n" %
> >> pkg_dic[pkg]["LICENSE"]) +               
> >> license_file.write("FILES:%s\n\n"
> >> % pkg_dic[pkg]["FILES"])
> >> 
> >>               # If the package doesn't contain any file, that is, its
> >>               size is
> >> 
> >> 0, the license # isn't relevant as far as the final image is concerned.
> >> So
> >> doing license check @@ -586,7 +653,8 @@ SSTATETASKS += "do_populate_lic"
> >> 
> >>   do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}"
> >>   do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
> >> 
> >> -ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest;
> >> license_create_manifest; " +ROOTFS_POSTPROCESS_COMMAND_prepend =
> >> "write_package_manifest; license_create_manifest;
> >> license_deployed_manifest_task; " +IMAGE_POSTPROCESS_COMMAND_prepend = "
> >> license_delete_tmp_files; "
> >> 
> >>   do_populate_lic_setscene[dirs] = "${LICSSTATEDIR}/${PN}"
> >>   do_populate_lic_setscene[cleandirs] = "${LICSSTATEDIR}"
> > 
> > A couple of things:
> > 
> > 1) Please take care to use the appropriate terminology. "package" has a
> > specific meaning; what we are dealing with here is files deployed directly
> > from recipes, packages aren't involved (or shouldn't be).
> 
> The terminology here is used because I reused the code to write rootfs
> manifest. I can change the terminology of this function or I can write a
> function just for the image manifest.

Understood, but I was referring more to how you describe things in the commit 
message and elsewhere in the code.

> > 2) You're looking into PKGDATA_DIR, as I mentioned in one of my other
> > replies I don't think this is right. The risk is that if a recipe doesn't
> > actually write out any packages, the manifest will be incomplete. We need
> > a different approach here where pkgdata isn't involved at all. IIRC my
> > earlier suggestion was to use what is written out by do_populate_lic,
> > i.e. tmp/deploy/licenses/ - if that isn't practical we can have that task
> > write out the LICENSE value to a separate file, and that will then work
> > for any recipe.
> 
> The use of PKGDATA_DIR is for the version of the deployed files, getting
> the license from here was just because it was practical. Is there
> another way to get the version without using PKGDATA_DIR?

In the context you're in, I'm not sure there is. However, I'm tempted to say 
if we're writing out LICENSE to a separate file during do_populate_lic, we 
might as well do the same with PV.

Cheers,
Paul


-- 

Paul Eggleton
Intel Open Source Technology Centre