[PATCH 0/4] extend buildhistory

[PATCH 0/4] extend buildhistory

[Patrick Ohly]

The initial implementation of YOCTO #8138
("buildhistory-extra.bbclass: store more information about a build")
had some issues ("kconfig" file not preserved) but more importantly,
additional changes are needed to support also gathering information
about native recipes.

I am proposing to include only the "buildhistory.bbclass: support
extending the content of the build history" patch into OE-core.

The original buildhistory-extra.bbclass and the modifications made to
it are included here merely to provide the context for that change and
to publish the modifications.

The following changes since commit d0f3f3a294d509560bd12b93b26eeec65cfee314:

  package_manager.py: Delete installed_pkgs.txt file (2015-11-24 15:55:33 +0000)

are available in the git repository at:

  git://github.com/pohly/openembedded-core buildhistory-extra
  https://github.com/pohly/openembedded-core/tree/buildhistory-extra

Ismo Puustinen (1):
  buildhistory-extra.bbclass: simplify usage

Patrick Ohly (2):
  buildhistory.bbclass: support extending the content of the build
    history
  buildhistory-extra.bbclass: more complete and reliable data gathering

Paul Eggleton (1):
  buildhistory-extra.bbclass: store more information about a build

 meta/classes/buildhistory-extra.bbclass | 127 ++++++++++++++++++++++++++++++++
 meta/classes/buildhistory.bbclass       |  13 +++-
 2 files changed, 139 insertions(+), 1 deletion(-)
 create mode 100644 meta/classes/buildhistory-extra.bbclass

-- 
2.1.4

[PATCH 1/4] buildhistory-extra.bbclass: store more information about a build

[Patrick Ohly]

From: Paul Eggleton <paul.eggleton@linux.intel.com>

When distributing a distro, it may be useful to also have additional
information about the sources used for creating a build. YOCTO #8138
specifically mentions:
- set of recipes that we used in builds, including information like
  checksum, originated layer(s), .bbappend and other overrides.
- information about source/patch files used in recipes (URLs, layers
  originating from, checksums).
- for images, links to source recipe names/versions that was used in image.

[Code from Paul, commit message from Patrick]

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/buildhistory-extra.bbclass | 86 +++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)
 create mode 100644 meta/classes/buildhistory-extra.bbclass

diff --git a/meta/classes/buildhistory-extra.bbclass b/meta/classes/buildhistory-extra.bbclass
new file mode 100644
index 0000000..e9fea5d
--- /dev/null
+++ b/meta/classes/buildhistory-extra.bbclass
@@ -0,0 +1,86 @@
+# Provides some extra logging into buildhistory of some of the inputs
+# (buildhistory itself mostly concentrates on build output)
+#
+# Copyright (C) 2015 Intel Corporation
+# Licensed under the MIT license
+
+BUILDHISTORY_EXTRA_PKGVARS ?= "PACKAGECONFIG EXTRA_OEMAKE EXTRA_OECONF EXTRA_OECMAKE EXTRA_OESCONS EXTRA_QMAKEVARS_PRE EXTRA_QMAKEVARS_POST OE_FEATURES SUMMARY DESCRIPTION HOMEPAGE LICENSE"
+
+python buildhistory_emit_pkghistory_append() {
+    import codecs
+
+    relpath = os.path.dirname(d.getVar('TOPDIR', True))
+
+    # List sources
+    pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
+    srcsfile = os.path.join(pkghistdir, "sources")
+    with codecs.open(srcsfile, "w", encoding='utf8') as f:
+        urls = (d.getVar('SRC_URI', True) or '').split()
+        for url in urls:
+            localpath = bb.fetch2.localpath(url, d)
+            if os.path.isfile(localpath):
+                sha256sum = bb.utils.sha256_file(localpath)
+            else:
+                sha256sum = 'N/A'
+            if localpath.startswith(relpath):
+                localpath = os.path.relpath(localpath, relpath)
+            f.write('%s %s %s\n' % (url, localpath, sha256sum))
+
+    # List metadata
+    includes = d.getVar('BBINCLUDED', True).split()
+    metafile = os.path.join(pkghistdir, "metadata")
+    with codecs.open(metafile, "w", encoding='utf8') as f:
+        for path in includes:
+            if os.path.exists(path):
+                sha256sum = bb.utils.sha256_file(path)
+                if path.startswith(relpath):
+                    path = os.path.relpath(path, relpath)
+                f.write('%s %s\n' % (path, sha256sum))
+
+    # List some extra var values
+    latestfile = os.path.join(pkghistdir, "latest")
+    vars = (d.getVar('BUILDHISTORY_EXTRA_PKGVARS', True) or '').split()
+    if vars:
+        with codecs.open(latestfile, "a", encoding='utf8') as f:
+            for var in vars:
+                value = oe.utils.squashspaces(d.getVar(var, True) or '')
+                if value:
+                    f.write('%s = %s\n' % (var, value))
+}
+
+python() {
+    if bb.data.inherits_class('kernel', d):
+        d.appendVarFlag('do_compile', 'prefuncs', ' buildhistory_extra_emit_kernelconfig')
+        d.appendVarFlag('do_compile', 'vardepsexclude', ' buildhistory_extra_emit_kernelconfig')
+}
+
+python buildhistory_extra_emit_kernelconfig() {
+    # Copy the final kernel config
+    # Unlike the rest of buildhistory this will only get run when the kernel is actually built
+    # (as opposed to being restored from the sstate cache); this is because do_shared_workdir
+    # operates outside of sstate and that running is the only way you get the config other than
+    # during the actual kernel build.
+    import shutil
+    pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
+    shutil.copyfile(d.expand('${B}/.config'), os.path.join(pkghistdir, 'kconfig'))
+}
+
+buildhistory_get_image_installed_append() {
+	# Create a file mapping installed packages to recipes
+	printf "" > ${BUILDHISTORY_DIR_IMAGE}/installed-package-recipes.txt
+	cat ${IMAGE_MANIFEST} | while read pkg pkgarch version
+	do
+		if [ -n "$pkg" ] ; then
+			recipe=`oe-pkgdata-util -p ${PKGDATA_DIR} lookup-recipe $pkg`
+			pkge=`oe-pkgdata-util -p ${PKGDATA_DIR} read-value PKGE $pkg`
+			if [ "$pkge" != "" ] ; then
+				pkge="$pkge-"
+			fi
+			pkgr=`oe-pkgdata-util -p ${PKGDATA_DIR} read-value PKGR $pkg`
+			if [ "$pkgr" != "" ] ; then
+				pkgr="-$pkgr"
+			fi
+			echo "$pkg $version $pkge$version$pkgr $recipe" >> ${BUILDHISTORY_DIR_IMAGE}/installed-package-recipes.txt
+		fi
+	done
+}
\ No newline at end of file
-- 
2.1.4

[PATCH 2/4] buildhistory.bbclass: support extending the content of the build history

[Patrick Ohly]

The idea behind the implementation of Yocto #8138 was that an
additional class can write additional files in the recipe directories,
for example by hooking into the functions of buildhistory.bbclass or
by implementing its own SSTATEPOSTINSTFUNCS function.

However, when these additional files get created before
buildhistory_emit_pkghistory(), they get removed again by that
function because it contains code which removes everything it does
not know about. The reason for that is that these unknown items
are probably obsolete.

This logic is the reason why the additional "kconfig" file from
buildhistory-extra.bbclass never showed up in the final build history.

To fix this, the hard-coded list of known files in
buildhistory_emit_pkghistory() must be turned into a variable which
derived classes can extend.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/buildhistory.bbclass | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 729dcd4..6023e5d 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -23,6 +23,16 @@ sstate_install[vardepsexclude] += "buildhistory_emit_pkghistory"
 # then the value added to SSTATEPOSTINSTFUNCS:
 SSTATEPOSTINSTFUNCS[vardepvalueexclude] .= "| buildhistory_emit_pkghistory"
 
+# All items excepts those listed here will be removed from a recipe's
+# build history directory by buildhistory_emit_pkghistory(). This is
+# necessary because some of these items (package directories, files that
+# we no longer emit) might be obsolete.
+#
+# When extending build history, derive your class from buildhistory.bbclass
+# and extend this list here with the additional files created by the derived
+# class.
+BUILDHISTORY_PRESERVE = "latest latest_srcrev"
+
 #
 # Write out metadata about this package for comparison when writing future packages
 #
@@ -165,12 +175,13 @@ python buildhistory_emit_pkghistory() {
             raise
 
     packagelist = packages.split()
+    preserve = d.getVar('BUILDHISTORY_PRESERVE', True).split()
     if not os.path.exists(pkghistdir):
         bb.utils.mkdirhier(pkghistdir)
     else:
         # Remove files for packages that no longer exist
         for item in os.listdir(pkghistdir):
-            if item != "latest" and item != "latest_srcrev":
+            if item not in preserve:
                 if item not in packagelist:
                     itempath = os.path.join(pkghistdir, item)
                     if os.path.isdir(itempath):
-- 
2.1.4

Re: [PATCH 2/4] buildhistory.bbclass: support extending the content of the build history

[Paul Eggleton]

On Wednesday 25 November 2015 10:30:30 Patrick Ohly wrote:
> The idea behind the implementation of Yocto #8138 was that an
> additional class can write additional files in the recipe directories,
> for example by hooking into the functions of buildhistory.bbclass or
> by implementing its own SSTATEPOSTINSTFUNCS function.
> 
> However, when these additional files get created before
> buildhistory_emit_pkghistory(), they get removed again by that
> function because it contains code which removes everything it does
> not know about. The reason for that is that these unknown items
> are probably obsolete.
> 
> This logic is the reason why the additional "kconfig" file from
> buildhistory-extra.bbclass never showed up in the final build history.
> 
> To fix this, the hard-coded list of known files in
> buildhistory_emit_pkghistory() must be turned into a variable which
> derived classes can extend.
> 
> Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
> ---
>  meta/classes/buildhistory.bbclass | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/buildhistory.bbclass
> b/meta/classes/buildhistory.bbclass index 729dcd4..6023e5d 100644
> --- a/meta/classes/buildhistory.bbclass
> +++ b/meta/classes/buildhistory.bbclass
> @@ -23,6 +23,16 @@ sstate_install[vardepsexclude] +=
> "buildhistory_emit_pkghistory" # then the value added to
> SSTATEPOSTINSTFUNCS:
>  SSTATEPOSTINSTFUNCS[vardepvalueexclude] .= "| buildhistory_emit_pkghistory"
> 
> +# All items excepts those listed here will be removed from a recipe's
> +# build history directory by buildhistory_emit_pkghistory(). This is
> +# necessary because some of these items (package directories, files that
> +# we no longer emit) might be obsolete.
> +#
> +# When extending build history, derive your class from buildhistory.bbclass
> +# and extend this list here with the additional files created by the
> derived 
> +# class.
> +BUILDHISTORY_PRESERVE = "latest latest_srcrev"
> +
>  #
>  # Write out metadata about this package for comparison when writing future
> packages #
> @@ -165,12 +175,13 @@ python buildhistory_emit_pkghistory() {
>              raise
> 
>      packagelist = packages.split()
> +    preserve = d.getVar('BUILDHISTORY_PRESERVE', True).split()
>      if not os.path.exists(pkghistdir):
>          bb.utils.mkdirhier(pkghistdir)
>      else:
>          # Remove files for packages that no longer exist
>          for item in os.listdir(pkghistdir):
> -            if item != "latest" and item != "latest_srcrev":
> +            if item not in preserve:
>                  if item not in packagelist:
>                      itempath = os.path.join(pkghistdir, item)
>                      if os.path.isdir(itempath):

Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

[PATCH 3/4] buildhistory-extra.bbclass: simplify usage

[Patrick Ohly]

From: Ismo Puustinen <ismo.puustinen@intel.com>

Instead of expecting the user to put InHERIT += "buildhistory
buildhistory-extra" into the local.conf, now only listing
buildhistory-extra is needed because the base class will always
be inherited.

[Change from Ismo, commit message from Patrick]

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/buildhistory-extra.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/buildhistory-extra.bbclass b/meta/classes/buildhistory-extra.bbclass
index e9fea5d..2dd5c5f 100644
--- a/meta/classes/buildhistory-extra.bbclass
+++ b/meta/classes/buildhistory-extra.bbclass
@@ -4,6 +4,8 @@
 # Copyright (C) 2015 Intel Corporation
 # Licensed under the MIT license
 
+inherit buildhistory
+
 BUILDHISTORY_EXTRA_PKGVARS ?= "PACKAGECONFIG EXTRA_OEMAKE EXTRA_OECONF EXTRA_OECMAKE EXTRA_OESCONS EXTRA_QMAKEVARS_PRE EXTRA_QMAKEVARS_POST OE_FEATURES SUMMARY DESCRIPTION HOMEPAGE LICENSE"
 
 python buildhistory_emit_pkghistory_append() {
-- 
2.1.4

[PATCH 4/4] buildhistory-extra.bbclass: more complete and reliable data gathering

[Patrick Ohly]

buildhistory_emit_pkghistory() only runs for recipes with packages and
when gathering information about packages is enabled. Information
about source files is needed in all cases because one cannot assume
that native compilation is based on exactly the same sources as
compilation for the target and/or there might not be the corresponding
information about the target compilation. This was not mentioned in
the feature description of
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 and thus not
covered by the original implementation.

Therefore we need to ensure that the extra code runs independently
from buildhistory_emit_pkghistory(). This is done in a similar way,
except that the code runs earlier when populating the sysroot, which
is done for native and target recipes.

Because the code now runs before buildhistory_emit_pkghistory(), we
may have to create the recipe directory. This was already true for
buildhistory_extra_emit_kernelconfig() which failed when starting with
a build history that had no directory for the kernel. Even if it was
written, that "kconfig" file then later got removed again by the
directory cleaning code in buildhistory_emit_pkghistory(). Now the new
BUILDHISTORY_PRESERVE mechanism is used to prevent that.

Handling the "latest" file is tricky: it mixes values that only exist
for recipes with packages (PACKAGES, obviously) and values that always
exist (like PV and PR), but the base class only writes it for recipes
with packages. Instead of doing intrusive changes to the base class to
fix that, the file now simply gets written by the extra code first and
later gets overwritten with a more complete version by the base
class. Another alternative would be to store PV and PR twice, but some
redundant work during build time seems better than redundancy in the
result.

Allowing a derived class to extend the file would cause even more
complications (like having to write to the file three times, or
introducing some kind of hook mechanism). While doable, it seems easier
to just store extra variables in a separate "variables" file.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 meta/classes/buildhistory-extra.bbclass | 49 +++++++++++++++++++++++++++++----
 1 file changed, 44 insertions(+), 5 deletions(-)

diff --git a/meta/classes/buildhistory-extra.bbclass b/meta/classes/buildhistory-extra.bbclass
index 2dd5c5f..05ba4cf 100644
--- a/meta/classes/buildhistory-extra.bbclass
+++ b/meta/classes/buildhistory-extra.bbclass
@@ -6,15 +6,51 @@
 
 inherit buildhistory
 
+# Variables to record in the "variables" file of each recipe.
+#
+# WARNING: changing this does not cause a rewrite of that file unless
+# something causes do_populate_sysroot[_setscene] to be run again,
+# like removing the tempdir and rebuilding (with or without sstate).
 BUILDHISTORY_EXTRA_PKGVARS ?= "PACKAGECONFIG EXTRA_OEMAKE EXTRA_OECONF EXTRA_OECMAKE EXTRA_OESCONS EXTRA_QMAKEVARS_PRE EXTRA_QMAKEVARS_POST OE_FEATURES SUMMARY DESCRIPTION HOMEPAGE LICENSE"
 
-python buildhistory_emit_pkghistory_append() {
+BUILDHISTORY_PRESERVE += "sources metadata variables kconfig"
+
+SSTATEPOSTINSTFUNCS_append = " buildhistory_extra_emit_pkghistory"
+# We want to avoid influence the signatures of sstate tasks - first the function itself:
+sstate_install[vardepsexclude] += "buildhistory_extra_emit_pkghistory"
+# then the value added to SSTATEPOSTINSTFUNCS:
+SSTATEPOSTINSTFUNCS[vardepvalueexclude] .= "| buildhistory_extra_emit_pkghistory"
+
+python buildhistory_extra_emit_pkghistory() {
+    bb.note('buildhistory_extra_emit_pkghistory %s' % d.getVar('BB_CURRENTTASK', True))
+    if not d.getVar('BB_CURRENTTASK', True) in ['populate_sysroot', 'populate_sysroot_setscene']:
+        return 0
+
     import codecs
 
     relpath = os.path.dirname(d.getVar('TOPDIR', True))
+    pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
+    if not os.path.exists(pkghistdir):
+        bb.utils.mkdirhier(pkghistdir)
+
+    # Record PV in the "latest" file. This duplicates work in
+    # buildhistory_emit_pkghistory(), but we do not know whether
+    # that will run (it gets skipped for recipes which never
+    # reach the packaging state or when "package" is not in
+    # BUILDHISTORY_FEATURES), so we write the file here
+    # and let buildhistory_emit_pkghistory() overwrite it again
+    # with more information later.
+    infofile = os.path.join(pkghistdir, "latest")
+    pe = d.getVar('PE', True) or "0"
+    pv = d.getVar('PV', True)
+    pr = d.getVar('PR', True)
+    with codecs.open(infofile, "w", encoding='utf8') as f:
+        if pe != "0":
+            f.write(u"PE = %s\n" % pe)
+        f.write(u"PV = %s\n" % pv)
+        f.write(u"PR = %s\n" % pr)
 
     # List sources
-    pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
     srcsfile = os.path.join(pkghistdir, "sources")
     with codecs.open(srcsfile, "w", encoding='utf8') as f:
         urls = (d.getVar('SRC_URI', True) or '').split()
@@ -39,15 +75,16 @@ python buildhistory_emit_pkghistory_append() {
                     path = os.path.relpath(path, relpath)
                 f.write('%s %s\n' % (path, sha256sum))
 
-    # List some extra var values
-    latestfile = os.path.join(pkghistdir, "latest")
     vars = (d.getVar('BUILDHISTORY_EXTRA_PKGVARS', True) or '').split()
+    varsfile = os.path.join(pkghistdir, "variables")
     if vars:
-        with codecs.open(latestfile, "a", encoding='utf8') as f:
+        with codecs.open(varsfile, "w", encoding='utf8') as f:
             for var in vars:
                 value = oe.utils.squashspaces(d.getVar(var, True) or '')
                 if value:
                     f.write('%s = %s\n' % (var, value))
+    elif os.path.exists(varsfile):
+        os.unlink(varsfile)
 }
 
 python() {
@@ -64,6 +101,8 @@ python buildhistory_extra_emit_kernelconfig() {
     # during the actual kernel build.
     import shutil
     pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
+    if not os.path.exists(pkghistdir):
+        bb.utils.mkdirhier(pkghistdir)
     shutil.copyfile(d.expand('${B}/.config'), os.path.join(pkghistdir, 'kconfig'))
 }
 
-- 
2.1.4

Re: [PATCH 4/4] buildhistory-extra.bbclass: more complete and reliable data gathering

[Paul Eggleton]

On Wednesday 25 November 2015 10:30:32 Patrick Ohly wrote:
> buildhistory_emit_pkghistory() only runs for recipes with packages and
> when gathering information about packages is enabled. Information
> about source files is needed in all cases because one cannot assume
> that native compilation is based on exactly the same sources as
> compilation for the target and/or there might not be the corresponding
> information about the target compilation. This was not mentioned in
> the feature description of
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 and thus not
> covered by the original implementation.
> 
> Therefore we need to ensure that the extra code runs independently
> from buildhistory_emit_pkghistory(). This is done in a similar way,
> except that the code runs earlier when populating the sysroot, which
> is done for native and target recipes.
> 
> Because the code now runs before buildhistory_emit_pkghistory(), we
> may have to create the recipe directory. This was already true for
> buildhistory_extra_emit_kernelconfig() which failed when starting with
> a build history that had no directory for the kernel. Even if it was
> written, that "kconfig" file then later got removed again by the
> directory cleaning code in buildhistory_emit_pkghistory(). Now the new
> BUILDHISTORY_PRESERVE mechanism is used to prevent that.
> 
> Handling the "latest" file is tricky: it mixes values that only exist
> for recipes with packages (PACKAGES, obviously) and values that always
> exist (like PV and PR), but the base class only writes it for recipes
> with packages. Instead of doing intrusive changes to the base class to
> fix that, the file now simply gets written by the extra code first and
> later gets overwritten with a more complete version by the base
> class. Another alternative would be to store PV and PR twice, but some
> redundant work during build time seems better than redundancy in the
> result.
> 
> Allowing a derived class to extend the file would cause even more
> complications (like having to write to the file three times, or
> introducing some kind of hook mechanism). While doable, it seems easier
> to just store extra variables in a separate "variables" file.
> 
> Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
> ---
>  meta/classes/buildhistory-extra.bbclass | 49
> +++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 5
> deletions(-)
> 
> diff --git a/meta/classes/buildhistory-extra.bbclass
> b/meta/classes/buildhistory-extra.bbclass index 2dd5c5f..05ba4cf 100644
> --- a/meta/classes/buildhistory-extra.bbclass
> +++ b/meta/classes/buildhistory-extra.bbclass
> @@ -6,15 +6,51 @@
> 
>  inherit buildhistory
> 
> +# Variables to record in the "variables" file of each recipe.
> +#
> +# WARNING: changing this does not cause a rewrite of that file unless
> +# something causes do_populate_sysroot[_setscene] to be run again,
> +# like removing the tempdir and rebuilding (with or without sstate).
>  BUILDHISTORY_EXTRA_PKGVARS ?= "PACKAGECONFIG EXTRA_OEMAKE EXTRA_OECONF
> EXTRA_OECMAKE EXTRA_OESCONS EXTRA_QMAKEVARS_PRE EXTRA_QMAKEVARS_POST
> OE_FEATURES SUMMARY DESCRIPTION HOMEPAGE LICENSE"
> 
> -python buildhistory_emit_pkghistory_append() {
> +BUILDHISTORY_PRESERVE += "sources metadata variables kconfig"
> +
> +SSTATEPOSTINSTFUNCS_append = " buildhistory_extra_emit_pkghistory"
> +# We want to avoid influence the signatures of sstate tasks - first the
> function itself: +sstate_install[vardepsexclude] +=
> "buildhistory_extra_emit_pkghistory" +# then the value added to
> SSTATEPOSTINSTFUNCS:
> +SSTATEPOSTINSTFUNCS[vardepvalueexclude] .= "|
> buildhistory_extra_emit_pkghistory" +
> +python buildhistory_extra_emit_pkghistory() {
> +    bb.note('buildhistory_extra_emit_pkghistory %s' %
> d.getVar('BB_CURRENTTASK', True)) +    if not d.getVar('BB_CURRENTTASK',
> True) in ['populate_sysroot', 'populate_sysroot_setscene']: +        return
> 0
> +
>      import codecs
> 
>      relpath = os.path.dirname(d.getVar('TOPDIR', True))
> +    pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
> +    if not os.path.exists(pkghistdir):
> +        bb.utils.mkdirhier(pkghistdir)
> +
> +    # Record PV in the "latest" file. This duplicates work in
> +    # buildhistory_emit_pkghistory(), but we do not know whether
> +    # that will run (it gets skipped for recipes which never
> +    # reach the packaging state or when "package" is not in
> +    # BUILDHISTORY_FEATURES), so we write the file here
> +    # and let buildhistory_emit_pkghistory() overwrite it again
> +    # with more information later.
> +    infofile = os.path.join(pkghistdir, "latest")
> +    pe = d.getVar('PE', True) or "0"
> +    pv = d.getVar('PV', True)
> +    pr = d.getVar('PR', True)
> +    with codecs.open(infofile, "w", encoding='utf8') as f:
> +        if pe != "0":
> +            f.write(u"PE = %s\n" % pe)
> +        f.write(u"PV = %s\n" % pv)
> +        f.write(u"PR = %s\n" % pr)
> 
>      # List sources
> -    pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
>      srcsfile = os.path.join(pkghistdir, "sources")
>      with codecs.open(srcsfile, "w", encoding='utf8') as f:
>          urls = (d.getVar('SRC_URI', True) or '').split()
> @@ -39,15 +75,16 @@ python buildhistory_emit_pkghistory_append() {
>                      path = os.path.relpath(path, relpath)
>                  f.write('%s %s\n' % (path, sha256sum))
> 
> -    # List some extra var values
> -    latestfile = os.path.join(pkghistdir, "latest")
>      vars = (d.getVar('BUILDHISTORY_EXTRA_PKGVARS', True) or '').split()
> +    varsfile = os.path.join(pkghistdir, "variables")
>      if vars:
> -        with codecs.open(latestfile, "a", encoding='utf8') as f:
> +        with codecs.open(varsfile, "w", encoding='utf8') as f:
>              for var in vars:
>                  value = oe.utils.squashspaces(d.getVar(var, True) or '')
>                  if value:
>                      f.write('%s = %s\n' % (var, value))
> +    elif os.path.exists(varsfile):
> +        os.unlink(varsfile)
>  }
> 
>  python() {
> @@ -64,6 +101,8 @@ python buildhistory_extra_emit_kernelconfig() {
>      # during the actual kernel build.
>      import shutil
>      pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
> +    if not os.path.exists(pkghistdir):
> +        bb.utils.mkdirhier(pkghistdir)
>      shutil.copyfile(d.expand('${B}/.config'), os.path.join(pkghistdir,
> 'kconfig')) }

Changes look OK (aside from general comments in my other reply about 
applicability to OE-Core), one minor note though - no need to check if the 
directory exists before calling mkdirhier() since that function will do that 
itself and will not error if it exists.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/4] extend buildhistory

[Paul Eggleton]

On Wednesday 25 November 2015 10:30:28 Patrick Ohly wrote:
> The initial implementation of YOCTO #8138
> ("buildhistory-extra.bbclass: store more information about a build")
> had some issues ("kconfig" file not preserved) but more importantly,
> additional changes are needed to support also gathering information
> about native recipes.
> 
> I am proposing to include only the "buildhistory.bbclass: support
> extending the content of the build history" patch into OE-core.
> 
> The original buildhistory-extra.bbclass and the modifications made to
> it are included here merely to provide the context for that change and
> to publish the modifications.

For others' reference, buildhistory-extra.bbclass was previously attached to 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 - I didn't and wouldn't 
propose introducing it into OE-Core in its current form because it's kind of 
turning buildhistory into something it wasn't designed for, though I don't 
disagree that it's useful. Happy to hear opinions from others on this though.

One aspect that will probably change in the future is how it picks up the 
kernel configuration - for 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5574 (which I hope to get to 
for 2.1, but no promises) we will need to write the kernel configuration to the 
sysroot so that we have somewhere we can look for it; if that's the case 
there'll be no need to copy it specially in buildhistory-extra.bbclass.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/4] extend buildhistory

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2477 bytes --]

On Fri, Nov 27, 2015 at 01:34:27PM +1300, Paul Eggleton wrote:
> On Wednesday 25 November 2015 10:30:28 Patrick Ohly wrote:
> > The initial implementation of YOCTO #8138
> > ("buildhistory-extra.bbclass: store more information about a build")
> > had some issues ("kconfig" file not preserved) but more importantly,
> > additional changes are needed to support also gathering information
> > about native recipes.
> > 
> > I am proposing to include only the "buildhistory.bbclass: support
> > extending the content of the build history" patch into OE-core.
> > 
> > The original buildhistory-extra.bbclass and the modifications made to
> > it are included here merely to provide the context for that change and
> > to publish the modifications.
> 
> For others' reference, buildhistory-extra.bbclass was previously attached to 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8138 - I didn't and wouldn't 
> propose introducing it into OE-Core in its current form because it's kind of 
> turning buildhistory into something it wasn't designed for, though I don't 
> disagree that it's useful. Happy to hear opinions from others on this though.

I agree it's useful.

I wonder if we can abstract some of these functions to be able to run
some of them without the actual build - we want to generate BOM (Bill of
materials) before building the image and e.g. for verification builds we
generate them twice (for base metadata, then with verification changes
applied) and generate diff.

I've implemented this as separate task which traverse the dependency
tree from the image we're going to build and writes all interesting
metadata in JSON.

Regards,

> One aspect that will probably change in the future is how it picks up the 
> kernel configuration - for 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5574 (which I hope to get to 
> for 2.1, but no promises) we will need to write the kernel configuration to the 
> sysroot so that we have somewhere we can look for it; if that's the case 
> there'll be no need to copy it specially in buildhistory-extra.bbclass.
> 
> Cheers,
> Paul
> 
> -- 
> 
> Paul Eggleton
> Intel Open Source Technology Centre
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: [PATCH 0/4] extend buildhistory

[Paul Eggleton]

On Fri, 27 Nov 2015 13:34:27 Paul Eggleton wrote:
> One aspect that will probably change in the future is how it picks up the
> kernel configuration - for
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5574 (which I hope to get
> to for 2.1, but no promises) we will need to write the kernel configuration
> to the sysroot so that we have somewhere we can look for it; if that's the
> case there'll be no need to copy it specially in
> buildhistory-extra.bbclass.

Hmm, I really should have checked kernel.bbclass before saying anything - it 
is already copying the kernel config to ${STAGING_KERNEL_BUILDDIR}/.config, so 
provided you look there at the right time (i.e. after do_shared_workdir of the 
kernel has executed) it should be possible to optimise how that's dealt with 
right now.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre