Re: [PATCHv2 6/6] license.bbclass: Create image license manifest

[Mariano Lopez <mariano.lopez@linux.intel.com>]

On 11/10/2015 04:42 AM, Paul Eggleton wrote:
> On Monday 09 November 2015 14:04:43 mariano.lopez@linux.intel.com wrote:
>> From: Mariano Lopez <mariano.lopez@linux.intel.com>
>>
>> This change adds the license_deployed_manifest function
>> that will create the manifest for the packages deployed
>> in the image but not installed in rootfs.
> Again, not "in the image" but "next to the image". I'd suggest giving an
> example or two here so that it's clear what's being referred to.
>
>> This new function was added to ROOTFS_POSTPROCESS_COMMAND
>> so it will run after every rootfs task. Because of this
>> it could run few times for a single build and get different
>> dependencies. Sometimes this dependencies won't include all
>> the deployed packages, in order to avoid missing licenses a
>> tmp file is create during the build and deleted after the
>> build (LICENSE_TMP_JSON).
>>
>> This change also modify the write_license_files because
>> the image manifest is different from the root manifest.
>>
>> [YOCTO #6772]
>>
>> Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
>> ---
>>   meta/classes/license.bbclass | 78
>> +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 73
>> insertions(+), 5 deletions(-)
>>
>> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
>> index 463dd54..fa8807e 100644
>> --- a/meta/classes/license.bbclass
>> +++ b/meta/classes/license.bbclass
>> @@ -11,6 +11,7 @@ LICENSE_CREATE_PACKAGE[type] = "boolean"
>>   LICENSE_CREATE_PACKAGE ??= "0"
>>   LICENSE_PACKAGE_SUFFIX ??= "-lic"
>>   LICENSE_FILES_DIRECTORY ??= "${datadir}/licenses/"
>> +LICENSE_TMP_JSON ?= "${LICENSE_DIRECTORY}/deploy_packages.json"
>>
>>   addtask populate_lic after do_patch before do_build
>>   do_populate_lic[dirs] = "${LICSSTATEDIR}/${PN}"
>> @@ -49,6 +50,59 @@ python license_create_manifest() {
>>       write_license_files(d, rootfs_license_manifest, pkg_dic)
>>   }
>>
>> +python license_deployed_manifest_task() {
>> +    license_deployed_manifest(d)
>> +}
>> +
>> +def license_deployed_manifest(d):
>> +    """ Write the license manifest for the deployed packages.
>> +        The deployed packages usually includes the bootloader
>> +        and extra files to boot the target.
>> +    """
>> +    import json
>> +
>> +    packages = ""
>> +    dep_dic = {}
>> +    pkg_dic = {}
>> +    info_dir = os.path.join(d.getVar("PKGDATA_DIR",True), "runtime")
>> +
>> +    # Sometimes the initramfs image is build and it doesn't have
>> +    # the boot dependencies. In order to overcome this it is
>> +    # necessary to get save previous dependencies found. This is
>> +    # why the json_file is used (and deleted once the build is done)
>> +    json_file = d.getVar("LICENSE_TMP_JSON", True)
>> +    json_lock = bb.utils.lockfile("%s.lock" % json_file)
>> +    if os.path.exists(json_file):
>> +        with open(json_file) as f:
>> +            pkg_dic = json.loads(f.read())
>> +
>> +    dep_dic = get_deployed_dependencies(d)
>> +    for dep in dep_dic.keys():
>> +        # At least one package of the deployed dependency is needed
>> +        # to get the version.
>> +        pkg = get_package_from_deployed(d, dep)
>> +        if pkg and pkg not in pkg_dic.keys():
>> +            data_file = os.path.join(info_dir, pkg)
>> +            pkg_dic[pkg] = oe.packagedata.read_pkgdatafile(data_file)
>> +            # It is necessary to mark this will be used for image manifest
>> +            pkg_dic[pkg]["IMAGE_MANIFEST"] = True
>> +            pkg_dic[pkg]["FILES"] = \
>> +                    get_deployed_files(d, dep_dic[dep])
>> +            if not "LICENSE" in pkg_dic[pkg].keys():
>> +                pkg_lic = "LICENSE_" + pkg
>> +                pkg_dic[pkg]["LICENSE"] = pkg_dic[pkg][pkg_lic]
>> +
>> +    with open(json_file, "w") as f:
>> +        json.dump(pkg_dic, f, indent=4)
>> +    bb.utils.unlockfile(json_lock)
>> +
>> +    # Because it might be called several times we lock the license file
>> +    image_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY',
>> True), +            d.getVar('IMAGE_NAME', True), 'image_license.manifest')
>> +    manifest_lock = bb.utils.lockfile("%s.lock" % image_license_manifest)
>> +    write_license_files(d, image_license_manifest, pkg_dic)
>> +    bb.utils.unlockfile(manifest_lock)
>> +
>>   def get_deployed_dependencies(d):
>>       """ Get all the deployed dependencies of an image """
>>
>> @@ -154,6 +208,11 @@ def get_deployed_files(d, man_file):
>>               dep_files = "%s %s" % (dep_files, os.path.basename(f))
>>       return dep_files
>>
>> +python license_delete_tmp_files () {
>> +    json_file = d.getVar("LICENSE_TMP_JSON", True)
>> +    os.remove(json_file)
>> +}
>> +
>>   def write_license_files(d, license_manifest, pkg_dic):
>>       import re
>>
>> @@ -175,10 +234,18 @@ def write_license_files(d, license_manifest, pkg_dic):
>> pkg_dic[pkg]["LICENSES"] = re.sub('  *', ' ', pkg_dic[pkg]["LICENSES"])
>> pkg_dic[pkg]["LICENSES"] = pkg_dic[pkg]["LICENSES"].split()
>>
>> -            license_file.write("PACKAGE NAME: %s\n" % pkg)
>> -            license_file.write("PACKAGE VERSION: %s\n" %
>> pkg_dic[pkg]["PV"]) -            license_file.write("RECIPE NAME: %s\n" %
>> pkg_dic[pkg]["PN"]) -            license_file.write("LICENSE: %s\n\n" %
>> pkg_dic[pkg]["LICENSE"]) +            if not "IMAGE_MANIFEST" in
>> pkg_dic[pkg]:
>> +                # Rootfs manifest
>> +                license_file.write("PACKAGE NAME: %s\n" % pkg)
>> +                license_file.write("PACKAGE VERSION: %s\n" %
>> pkg_dic[pkg]["PV"]) +                license_file.write("RECIPE NAME: %s\n"
>> % pkg_dic[pkg]["PN"]) +                license_file.write("LICENSE: %s\n\n"
>> % pkg_dic[pkg]["LICENSE"]) +            else:
>> +                # Image manifest
>> +                license_file.write("RECIPE NAME: %s\n" %
>> pkg_dic[pkg]["PN"]) +                license_file.write("VERSION: %s\n" %
>> pkg_dic[pkg]["PV"]) +                license_file.write("LICENSE: %s\n" %
>> pkg_dic[pkg]["LICENSE"]) +                license_file.write("FILES:%s\n\n"
>> % pkg_dic[pkg]["FILES"])
>>
>>               # If the package doesn't contain any file, that is, its size is
>> 0, the license # isn't relevant as far as the final image is concerned. So
>> doing license check @@ -586,7 +653,8 @@ SSTATETASKS += "do_populate_lic"
>>   do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}"
>>   do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
>>
>> -ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest;
>> license_create_manifest; " +ROOTFS_POSTPROCESS_COMMAND_prepend =
>> "write_package_manifest; license_create_manifest;
>> license_deployed_manifest_task; " +IMAGE_POSTPROCESS_COMMAND_prepend = "
>> license_delete_tmp_files; "
>>
>>   do_populate_lic_setscene[dirs] = "${LICSSTATEDIR}/${PN}"
>>   do_populate_lic_setscene[cleandirs] = "${LICSSTATEDIR}"
> A couple of things:
>
> 1) Please take care to use the appropriate terminology. "package" has a
> specific meaning; what we are dealing with here is files deployed directly from
> recipes, packages aren't involved (or shouldn't be).

The terminology here is used because I reused the code to write rootfs 
manifest. I can change the terminology of this function or I can write a 
function just for the image manifest.

>
> 2) You're looking into PKGDATA_DIR, as I mentioned in one of my other replies
> I don't think this is right. The risk is that if a recipe doesn't actually
> write out any packages, the manifest will be incomplete. We need a different
> approach here where pkgdata isn't involved at all. IIRC my earlier suggestion
> was to use what is written out by do_populate_lic, i.e. tmp/deploy/licenses/ -
> if that isn't practical we can have that task write out the LICENSE value to a
> separate file, and that will then work for any recipe.

The use of PKGDATA_DIR is for the version of the deployed files, getting 
the license from here was just because it was practical. Is there 
another way to get the version without using PKGDATA_DIR?

>
> Cheers,
> Paul
>
Regards,
Mariano Lopez