Passing additional data from packages to be used during image creation

Passing additional data from packages to be used during image creation

[Pascal Bach]

I would like to pass some additional data generated during the creation of a package to the image creation step.
The idea is to use this data to generate a more detailed manifest of what is included in the image. In the concrete case
I would like to pass a list of required copyright files that need to be checked for during image creation.

My current approach is to add this list as a new variable COPYRIGHTS to the pkgdata of each package.
Using the image manifest to determine the package this allows me to compile a list of all required copyrights.

The following patch allows to add extra variables to pkgdata and thus allows to pass additional data

- [PATCH] package.bbclass: allow additional variables to be added to

Is this a valid approach or did a miss a simpler way to access any package data during image creation?

Pascal

[PATCH] package.bbclass: allow additional variables to be added to pkgdata

[Pascal Bach]

Adding variable names to the variable PACKAGE_EXTRA_PKGDATA allows including
this variables in the pkgdata file of a package.

For example PACKAGE_EXTRA_PKGDATA = "MACHINE" will add the MACHINE variable to
pkgdata. These fields can be used later in the process by some custom package
processing for example during image creation.

Signed-off-by: Pascal Bach <pascal.bach@siemens.com>
---
 meta/classes/package.bbclass | 8 +++++++-
 meta/conf/documentation.conf | 1 +
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 568b85c..2ed1ecc 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1388,6 +1388,11 @@ python emit_pkgdata() {
             write_if_exists(sf, pkg, 'FILERDEPENDS_' + dfile)
 
         sf.write('%s_%s: %d\n' % ('PKGSIZE', pkg, total_size))
+
+        # Write additional variables listed in PACKAGE_EXTRA_PKGDATA
+        for extra_var in (d.getVar('PACKAGE_EXTRA_PKGDATA') or "").split():
+            write_if_exists(sf, pkg, extra_var)
+
         sf.close()
 
         # Symlinks needed for rprovides lookup
@@ -1986,12 +1991,13 @@ python package_depchains() {
 
 # Since bitbake can't determine which variables are accessed during package
 # iteration, we need to list them here:
-PACKAGEVARS = "FILES RDEPENDS RRECOMMENDS SUMMARY DESCRIPTION RSUGGESTS RPROVIDES RCONFLICTS PKG ALLOW_EMPTY pkg_postinst pkg_postrm INITSCRIPT_NAME INITSCRIPT_PARAMS DEBIAN_NOAUTONAME ALTERNATIVE PKGE PKGV PKGR USERADD_PARAM GROUPADD_PARAM CONFFILES SYSTEMD_SERVICE LICENSE SECTION pkg_preinst pkg_prerm RREPLACES GROUPMEMS_PARAM SYSTEMD_AUTO_ENABLE"
+PACKAGEVARS = "FILES RDEPENDS RRECOMMENDS SUMMARY DESCRIPTION RSUGGESTS RPROVIDES RCONFLICTS PKG ALLOW_EMPTY pkg_postinst pkg_postrm INITSCRIPT_NAME INITSCRIPT_PARAMS DEBIAN_NOAUTONAME ALTERNATIVE PKGE PKGV PKGR USERADD_PARAM GROUPADD_PARAM CONFFILES SYSTEMD_SERVICE LICENSE SECTION pkg_preinst pkg_prerm RREPLACES GROUPMEMS_PARAM SYSTEMD_AUTO_ENABLE PACKAGE_EXTRA_PKGDATA"
 
 def gen_packagevar(d):
     ret = []
     pkgs = (d.getVar("PACKAGES") or "").split()
     vars = (d.getVar("PACKAGEVARS") or "").split()
+    vars.extend((d.getVar("PACKAGE_EXTRA_PKGDATA") or "").split())
     for p in pkgs:
         for v in vars:
             ret.append(v + "_" + p)
diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index 06527cb..e0fb8cf 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -311,6 +311,7 @@ PACKAGE_BEFORE_PN[doc] = "Enables easily adding packages to PACKAGES before ${PN
 PACKAGE_CLASSES[doc] = "This variable specifies the package manager to use when packaging data. It is set in the conf/local.conf file in the Build Directory."
 PACKAGE_EXCLUDE[doc] = "Packages to exclude from the installation. If a listed package is required, an error is generated."
 PACKAGE_EXTRA_ARCHS[doc] = "Specifies the list of architectures compatible with the device CPU. This variable is useful when you build for several different devices that use miscellaneous processors."
+PACKAGE_EXTRA_PKGDATA[doc] = "Additional variables to be included in a packages pkgdata."
 PACKAGE_GROUP[doc] = "Defines one or more packages to include in an image when a specific item is included in IMAGE_FEATURES."
 PACKAGE_INSTALL[doc] = "List of the packages to be installed into the image. The variable is generally not user-defined and uses IMAGE_INSTALL as part of the list."
 PACKAGE_INSTALL_ATTEMPTONLY[doc] = "List of packages attempted to be installed. If a listed package fails to install, the build system does not generate an error. This variable is generally not user-defined."
-- 
2.1.4

Re: Passing additional data from packages to be used during image creation

[Richard Purdie]

On Thu, 2017-01-19 at 10:16 +0100, Pascal Bach wrote:
> I would like to pass some additional data generated during the
> creation of a package to the image creation step.
> The idea is to use this data to generate a more detailed manifest of
> what is included in the image. In the concrete case
> I would like to pass a list of required copyright files that need to
> be checked for during image creation.
> 
> My current approach is to add this list as a new variable COPYRIGHTS
> to the pkgdata of each package.
> Using the image manifest to determine the package this allows me to
> compile a list of all required copyrights.
> 
> The following patch allows to add extra variables to pkgdata and thus
> allows to pass additional data
> 
> - [PATCH] package.bbclass: allow additional variables to be added to
> 
> Is this a valid approach or did a miss a simpler way to access any
> package data during image creation?

I'm curious if adding something to PACKAGEVARS would work or if not,
why not?

Cheers,

Richard

Re: Passing additional data from packages to be used during image creation

[Pascal Bach]

On 21.01.2017 12:44, Richard Purdie wrote:
> On Thu, 2017-01-19 at 10:16 +0100, Pascal Bach wrote:
>> I would like to pass some additional data generated during the
>> creation of a package to the image creation step.
>> The idea is to use this data to generate a more detailed manifest of
>> what is included in the image. In the concrete case
>> I would like to pass a list of required copyright files that need to
>> be checked for during image creation.
>>
>> My current approach is to add this list as a new variable COPYRIGHTS
>> to the pkgdata of each package.
>> Using the image manifest to determine the package this allows me to
>> compile a list of all required copyrights.
>>
>> The following patch allows to add extra variables to pkgdata and thus
>> allows to pass additional data
>>
>> - [PATCH] package.bbclass: allow additional variables to be added to
>>
>> Is this a valid approach or did a miss a simpler way to access any
>> package data during image creation?
> I'm curious if adding something to PACKAGEVARS would work or if not,
> why not?
Just adding it to PACKAGEVARS is not enough as the list of variables written to the pkgdata files is hard coded in emit_pkgdata here: https://github.com/openembedded/openembedded-core/blob/master/meta/classes/package.bbclass#L1365

However I notices an issue with the code as it seams the do_package step is not triggered if a variable in PACKAGE_EXTRA_PKGDATA changes. I'm not sure why as it should be
added as a vardeps to the do_package task. For example if I add "COPYRIGHTS" to PACKAGE_EXTRA_PKGDATA and than change the COPYRIGHTS variable in a recipe the do_package step doesn't get executed.
I'm trying to figure out why but if somebody can spot what I did wrong this would be appreciated.

Pascal