[PATCH 0/9] buildhistory improvements

[PATCH 0/9] buildhistory improvements

[Paul Eggleton]

A bunch of improvements for buildhistory, implementing some feature
requests and todo items of my own, as well as tidying up some of the
code.


The following changes since commit 404d2d490fc347203e89d274530c17fb5f0aa20f:

  gcc-configure-target: Set native-system-header-dir for target gcc (2012-08-01 23:11:09 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes7
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes7

Paul Eggleton (9):
  classes/buildhistory: remove obsolete flat package history feature
  classes/buildhistory: ensure old package info is removed
  classes/buildhistory: remove unused functions
  classes/buildhistory: save preinst/postinst/prerm/postrm
  classes/buildhistory: record PKG/PKGE/PKGV/PKGR
  scripts: add buildhistory-tag script
  buildhistory_analysis: tidy up duplicate split_version function
  buildhistory_analysis: ignore removal of self-dependencies
  classes/buildhistory: save metadata revisions

 meta/classes/buildhistory.bbclass    |  197 ++++++++++++++++++----------------
 meta/lib/oe/buildhistory_analysis.py |   96 +++++++++++++----
 scripts/buildhistory-tag             |   42 ++++++++
 3 files changed, 221 insertions(+), 114 deletions(-)
 create mode 100755 scripts/buildhistory-tag

-- 
1.7.9.5

[PATCH 1/9] classes/buildhistory: remove obsolete flat package history feature

[Paul Eggleton]

Remove the ability to set BUILDHISTORY_KEEP_VERSIONS = "1" to save the
package history as flat versioned files rather than relying on git to
keep previous versions of the package information. git has proven to
work quite well in this capacity.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass |   43 +++++++------------------------------
 1 file changed, 8 insertions(+), 35 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index ddb76e8..f5494cd 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -156,7 +156,6 @@ python buildhistory_emit_pkghistory() {
     rcpinfo.depends = sortlist(squashspaces(d.getVar('DEPENDS', True) or ""))
     rcpinfo.packages = packages
     write_recipehistory(rcpinfo, d)
-    write_latestlink(None, pe, pv, pr, d)
 
     # Apparently the version can be different on a per-package basis (see Python)
     pkgdest = d.getVar('PKGDEST', True)
@@ -199,8 +198,6 @@ python buildhistory_emit_pkghistory() {
         pkginfo.filelist = " ".join(filelist)
 
         write_pkghistory(pkginfo, d)
-
-        write_latestlink(pkg, pe, pv, pr, d)
 }
 
 
@@ -212,8 +209,8 @@ def write_recipehistory(rcpinfo, d):
     if not os.path.exists(pkghistdir):
         os.makedirs(pkghistdir)
 
-    verfile = os.path.join(pkghistdir, "%s:%s-%s" % (rcpinfo.pe, rcpinfo.pv, rcpinfo.pr))
-    f = open(verfile, "w")
+    infofile = os.path.join(pkghistdir, "latest")
+    f = open(infofile, "w")
     try:
         if rcpinfo.pe != "0":
             f.write("PE = %s\n" %  rcpinfo.pe)
@@ -226,16 +223,16 @@ def write_recipehistory(rcpinfo, d):
 
 
 def write_pkghistory(pkginfo, d):
-    bb.debug(2, "Writing package history")
+    bb.debug(2, "Writing package history for package %s" % pkginfo.name)
 
     pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
 
-    verpath = os.path.join(pkghistdir, pkginfo.name)
-    if not os.path.exists(verpath):
-        os.makedirs(verpath)
+    pkgpath = os.path.join(pkghistdir, pkginfo.name)
+    if not os.path.exists(pkgpath):
+        os.makedirs(pkgpath)
 
-    verfile = os.path.join(verpath, "%s:%s-%s" % (pkginfo.pe, pkginfo.pv, pkginfo.pr))
-    f = open(verfile, "w")
+    infofile = os.path.join(pkgpath, "latest")
+    f = open(infofile, "w")
     try:
         if pkginfo.pe != "0":
             f.write("PE = %s\n" %  pkginfo.pe)
@@ -250,30 +247,6 @@ def write_pkghistory(pkginfo, d):
         f.close()
 
 
-def write_latestlink(pkg, pe, pv, pr, d):
-    import shutil
-
-    pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
-
-    def rm_link(path):
-        try:
-            os.unlink(path)
-        except OSError:
-            return
-
-    if pkg:
-        filedir = os.path.join(pkghistdir, pkg)
-    else:
-        filedir = pkghistdir
-    latest_file = os.path.join(filedir, "latest")
-    ver_file = os.path.join(filedir, "%s:%s-%s" % (pe, pv, pr))
-    rm_link(latest_file)
-    if d.getVar('BUILDHISTORY_KEEP_VERSIONS', True) == '1':
-        shutil.copy(ver_file, latest_file)
-    else:
-        shutil.move(ver_file, latest_file)
-
-
 buildhistory_get_image_installed() {
 	# Anything requiring the use of the packaging system should be done in here
 	# in case the packaging files are going to be removed for this image
-- 
1.7.9.5

[PATCH 2/9] classes/buildhistory: ensure old package info is removed

[Paul Eggleton]

If a package is removed from PACKAGES, ensure that the package info file
and directory are removed from buildhistory so that we don't have stale
data lying around.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass |   18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index f5494cd..c3555da 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -149,6 +149,19 @@ python buildhistory_emit_pkghistory() {
     pr = d.getVar('PR', True)
     packages = squashspaces(d.getVar('PACKAGES', True))
 
+    packagelist = packages.split()
+    if not os.path.exists(pkghistdir):
+        os.makedirs(pkghistdir)
+    else:
+        # Remove files for packages that no longer exist
+        for item in os.listdir(pkghistdir):
+            if item != "latest":
+                if item not in packagelist:
+                    subdir = os.path.join(pkghistdir, item)
+                    for subfile in os.listdir(subdir):
+                        os.unlink(os.path.join(subdir, subfile))
+                    os.rmdir(subdir)
+
     rcpinfo = RecipeInfo(pn)
     rcpinfo.pe = pe
     rcpinfo.pv = pv
@@ -159,7 +172,7 @@ python buildhistory_emit_pkghistory() {
 
     # Apparently the version can be different on a per-package basis (see Python)
     pkgdest = d.getVar('PKGDEST', True)
-    for pkg in packages.split():
+    for pkg in packagelist:
         pe = getpkgvar(pkg, 'PE') or "0"
         pv = getpkgvar(pkg, 'PV')
         pr = getpkgvar(pkg, 'PR')
@@ -206,9 +219,6 @@ def write_recipehistory(rcpinfo, d):
 
     pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
 
-    if not os.path.exists(pkghistdir):
-        os.makedirs(pkghistdir)
-
     infofile = os.path.join(pkghistdir, "latest")
     f = open(infofile, "w")
     try:
-- 
1.7.9.5

[PATCH 3/9] classes/buildhistory: remove unused functions

[Paul Eggleton]

getlastrecipeversion and readRecipeInfo weren't called anywhere.
Remove them so we don't have to keep them up-to-date with future
additions to the recipe info file.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass |   29 -----------------------------
 1 file changed, 29 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index c3555da..510a6df 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -68,28 +68,6 @@ python buildhistory_emit_pkghistory() {
 
         return val
 
-    def readRecipeInfo(pn, histfile):
-        rcpinfo = RecipeInfo(pn)
-        f = open(histfile, "r")
-        try:
-            for line in f:
-                lns = line.split('=')
-                name = lns[0].strip()
-                value = lns[1].strip(" \t\r\n").strip('"')
-                if name == "PE":
-                    rcpinfo.pe = value
-                elif name == "PV":
-                    rcpinfo.pv = value
-                elif name == "PR":
-                    rcpinfo.pr = value
-                elif name == "DEPENDS":
-                    rcpinfo.depends = value
-                elif name == "PACKAGES":
-                    rcpinfo.packages = value
-        finally:
-            f.close()
-        return rcpinfo
-
     def readPackageInfo(pkg, histfile):
         pkginfo = PackageInfo(pkg)
         f = open(histfile, "r")
@@ -118,13 +96,6 @@ python buildhistory_emit_pkghistory() {
             f.close()
         return pkginfo
 
-    def getlastrecipeversion(pn):
-        try:
-            histfile = os.path.join(pkghistdir, "latest")
-            return readRecipeInfo(pn, histfile)
-        except EnvironmentError:
-            return None
-
     def getlastpkgversion(pkg):
         try:
             histfile = os.path.join(pkghistdir, pkg, "latest")
-- 
1.7.9.5

[PATCH 4/9] classes/buildhistory: save preinst/postinst/prerm/postrm

[Paul Eggleton]

Write the value of these package script variables into the packageinfo
so that any changes to them can be tracked (in separate files since they
are multi-line).

Inspired by an earlier patch from Andreas Müller <schnitzeltony@googlemail.com>

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass    |   24 ++++++++++------
 meta/lib/oe/buildhistory_analysis.py |   50 +++++++++++++++++++++++++++++++++-
 2 files changed, 65 insertions(+), 9 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 510a6df..200d031 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -57,6 +57,8 @@ python buildhistory_emit_pkghistory() {
             self.rrecommends = ""
             self.files = ""
             self.filelist = ""
+            # Variables that need to be written to their own separate file
+            self.filevars = dict.fromkeys(['pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm'])
 
     # Should check PACKAGES here to see if anything removed
 
@@ -167,6 +169,8 @@ python buildhistory_emit_pkghistory() {
         pkginfo.rdepends = sortpkglist(squashspaces(getpkgvar(pkg, 'RDEPENDS') or ""))
         pkginfo.rrecommends = sortpkglist(squashspaces(getpkgvar(pkg, 'RRECOMMENDS') or ""))
         pkginfo.files = squashspaces(getpkgvar(pkg, 'FILES') or "")
+        for filevar in pkginfo.filevars:
+            pkginfo.filevars[filevar] = getpkgvar(pkg, filevar)
 
         # Gather information about packaged files
         pkgdestpkg = os.path.join(pkgdest, pkg)
@@ -191,16 +195,13 @@ def write_recipehistory(rcpinfo, d):
     pkghistdir = d.getVar('BUILDHISTORY_DIR_PACKAGE', True)
 
     infofile = os.path.join(pkghistdir, "latest")
-    f = open(infofile, "w")
-    try:
+    with open(infofile, "w") as f:
         if rcpinfo.pe != "0":
             f.write("PE = %s\n" %  rcpinfo.pe)
         f.write("PV = %s\n" %  rcpinfo.pv)
         f.write("PR = %s\n" %  rcpinfo.pr)
         f.write("DEPENDS = %s\n" %  rcpinfo.depends)
         f.write("PACKAGES = %s\n" %  rcpinfo.packages)
-    finally:
-        f.close()
 
 
 def write_pkghistory(pkginfo, d):
@@ -213,8 +214,7 @@ def write_pkghistory(pkginfo, d):
         os.makedirs(pkgpath)
 
     infofile = os.path.join(pkgpath, "latest")
-    f = open(infofile, "w")
-    try:
+    with open(infofile, "w") as f:
         if pkginfo.pe != "0":
             f.write("PE = %s\n" %  pkginfo.pe)
         f.write("PV = %s\n" %  pkginfo.pv)
@@ -224,8 +224,16 @@ def write_pkghistory(pkginfo, d):
         f.write("PKGSIZE = %d\n" %  pkginfo.size)
         f.write("FILES = %s\n" %  pkginfo.files)
         f.write("FILELIST = %s\n" %  pkginfo.filelist)
-    finally:
-        f.close()
+
+    for filevar in pkginfo.filevars:
+        filevarpath = os.path.join(pkgpath, "latest.%s" % filevar)
+        val = pkginfo.filevars[filevar]
+        if val:
+            with open(filevarpath, "w") as f:
+                f.write(val)
+        else:
+            if os.path.exists(filevarpath):
+                os.unlink(filevarpath)
 
 
 buildhistory_get_image_installed() {
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index 29dc4a9..6c6a085 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -90,6 +90,18 @@ class ChangeRecord:
             else:
                 percentchg = 100
             out = '%s changed from %s to %s (%s%d%%)' % (self.fieldname, self.oldvalue or "''", self.newvalue or "''", '+' if percentchg > 0 else '', percentchg)
+        elif self.fieldname in ['pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm']:
+            if self.oldvalue and self.newvalue:
+                out = '%s changed:\n  ' % self.fieldname
+            elif self.newvalue:
+                out = '%s added:\n  ' % self.fieldname
+            elif self.oldvalue:
+                out = '%s cleared:\n  ' % self.fieldname
+            alines = self.oldvalue.splitlines()
+            blines = self.newvalue.splitlines()
+            diff = difflib.unified_diff(alines, blines, self.fieldname, self.fieldname, lineterm='')
+            out += '\n  '.join(list(diff)[2:])
+            out += '\n  --'
         elif self.fieldname in img_monitor_files:
             if outer:
                 prefix = 'Changes to %s ' % self.path
@@ -330,7 +342,12 @@ def process_changes(repopath, revision1, revision2 = 'HEAD', report_all = False)
     for d in diff.iter_change_type('M'):
         path = os.path.dirname(d.a_blob.path)
         if path.startswith('packages/'):
-            changes.extend(compare_dict_blobs(path, d.a_blob, d.b_blob, report_all))
+            filename = os.path.basename(d.a_blob.path)
+            if filename == 'latest':
+                changes.extend(compare_dict_blobs(path, d.a_blob, d.b_blob, report_all))
+            elif filename.startswith('latest.'):
+                chg = ChangeRecord(path, filename, d.a_blob.data_stream.read(), d.b_blob.data_stream.read(), True)
+                changes.append(chg)
         elif path.startswith('images/'):
             filename = os.path.basename(d.a_blob.path)
             if filename in img_monitor_files:
@@ -356,6 +373,37 @@ def process_changes(repopath, revision1, revision2 = 'HEAD', report_all = False)
             elif filename == 'image-info.txt':
                 changes.extend(compare_dict_blobs(path, d.a_blob, d.b_blob, report_all))
 
+    # Look for added preinst/postinst/prerm/postrm
+    # (without reporting newly added recipes)
+    addedpkgs = []
+    addedchanges = []
+    for d in diff.iter_change_type('A'):
+        path = os.path.dirname(d.b_blob.path)
+        if path.startswith('packages/'):
+            filename = os.path.basename(d.b_blob.path)
+            if filename == 'latest':
+                addedpkgs.append(path)
+            elif filename.startswith('latest.'):
+                chg = ChangeRecord(path, filename[7:], '', d.b_blob.data_stream.read(), True)
+                addedchanges.append(chg)
+    for chg in addedchanges:
+        found = False
+        for pkg in addedpkgs:
+            if chg.path.startswith(pkg):
+                found = True
+                break
+        if not found:
+            changes.append(chg)
+
+    # Look for cleared preinst/postinst/prerm/postrm
+    for d in diff.iter_change_type('D'):
+        path = os.path.dirname(d.a_blob.path)
+        if path.startswith('packages/'):
+            filename = os.path.basename(d.a_blob.path)
+            if filename != 'latest' and filename.startswith('latest.'):
+                chg = ChangeRecord(path, filename[7:], d.a_blob.data_stream.read(), '', True)
+                changes.append(chg)
+
     # Link related changes
     for chg in changes:
         if chg.monitored:
-- 
1.7.9.5

[PATCH 5/9] classes/buildhistory: record PKG/PKGE/PKGV/PKGR

[Paul Eggleton]

Save PKG (the actual output package name, which is often different due
to debian renaming), and PKGE/PKGV/PKGR (which may be manipulated in
certain special cases e.g. gitpkgv.bbclass in meta-oe, the
external-sourcery-toolchain recipe, etc.) Note that these are only
written when they are different from the normal package name in the
case of PKG, or PE/PV/PR for the other variables.

Also, use PKGE/PKGV/PKGR instead of PE/PV/PR when comparing package
versions since these actually represent the version that the package
manager sees.

Implements [YOCTO #2787].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass    |   66 +++++++++++++++++++++++++---------
 meta/lib/oe/buildhistory_analysis.py |   22 +++++++++++-
 2 files changed, 71 insertions(+), 17 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 200d031..76648c9 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -51,6 +51,11 @@ python buildhistory_emit_pkghistory() {
             self.pe = "0"
             self.pv = "0"
             self.pr = "r0"
+            # pkg/pkge/pkgv/pkgr should be empty because we want to be able to default them
+            self.pkg = ""
+            self.pkge = ""
+            self.pkgv = ""
+            self.pkgr = ""
             self.size = 0
             self.depends = ""
             self.rdepends = ""
@@ -72,8 +77,7 @@ python buildhistory_emit_pkghistory() {
 
     def readPackageInfo(pkg, histfile):
         pkginfo = PackageInfo(pkg)
-        f = open(histfile, "r")
-        try:
+        with open(histfile, "r") as f:
             for line in f:
                 lns = line.split('=')
                 name = lns[0].strip()
@@ -84,6 +88,14 @@ python buildhistory_emit_pkghistory() {
                     pkginfo.pv = value
                 elif name == "PR":
                     pkginfo.pr = value
+                elif name == "PKG":
+                    pkginfo.pkg = value
+                elif name == "PKGE":
+                    pkginfo.pkge = value
+                elif name == "PKGV":
+                    pkginfo.pkgv = value
+                elif name == "PKGR":
+                    pkginfo.pkgr = value
                 elif name == "RDEPENDS":
                     pkginfo.rdepends = value
                 elif name == "RRECOMMENDS":
@@ -94,8 +106,15 @@ python buildhistory_emit_pkghistory() {
                     pkginfo.files = value
                 elif name == "FILELIST":
                     pkginfo.filelist = value
-        finally:
-            f.close()
+        # Apply defaults
+        if not pkginfo.pkg:
+            pkginfo.pkg = pkginfo.name
+        if not pkginfo.pkge:
+            pkginfo.pkge = pkginfo.pe
+        if not pkginfo.pkgv:
+            pkginfo.pkgv = pkginfo.pv
+        if not pkginfo.pkgr:
+            pkginfo.pkgr = pkginfo.pr
         return pkginfo
 
     def getlastpkgversion(pkg):
@@ -143,29 +162,33 @@ python buildhistory_emit_pkghistory() {
     rcpinfo.packages = packages
     write_recipehistory(rcpinfo, d)
 
-    # Apparently the version can be different on a per-package basis (see Python)
     pkgdest = d.getVar('PKGDEST', True)
     for pkg in packagelist:
-        pe = getpkgvar(pkg, 'PE') or "0"
-        pv = getpkgvar(pkg, 'PV')
-        pr = getpkgvar(pkg, 'PR')
+        pkge = getpkgvar(pkg, 'PKGE') or "0"
+        pkgv = getpkgvar(pkg, 'PKGV')
+        pkgr = getpkgvar(pkg, 'PKGR')
         #
         # Find out what the last version was
         # Make sure the version did not decrease
         #
         lastversion = getlastpkgversion(pkg)
         if lastversion:
-            last_pe = lastversion.pe
-            last_pv = lastversion.pv
-            last_pr = lastversion.pr
-            r = bb.utils.vercmp((pe, pv, pr), (last_pe, last_pv, last_pr))
+            last_pkge = lastversion.pkge
+            last_pkgv = lastversion.pkgv
+            last_pkgr = lastversion.pkgr
+            r = bb.utils.vercmp((pkge, pkgv, pkgr), (last_pkge, last_pkgv, last_pkgr))
             if r < 0:
-                bb.error("Package version for package %s went backwards which would break package feeds from (%s:%s-%s to %s:%s-%s)" % (pkg, last_pe, last_pv, last_pr, pe, pv, pr))
+                bb.error("Package version for package %s went backwards which would break package feeds from (%s:%s-%s to %s:%s-%s)" % (pkg, last_pkge, last_pkgv, last_pkgr, pkge, pkgv, pkgr))
 
         pkginfo = PackageInfo(pkg)
-        pkginfo.pe = pe
-        pkginfo.pv = pv
-        pkginfo.pr = pr
+        # Apparently the version can be different on a per-package basis (see Python)
+        pkginfo.pe = getpkgvar(pkg, 'PE') or "0"
+        pkginfo.pv = getpkgvar(pkg, 'PV')
+        pkginfo.pr = getpkgvar(pkg, 'PR')
+        pkginfo.pkg = getpkgvar(pkg, 'PKG') or pkg
+        pkginfo.pkge = pkge
+        pkginfo.pkgv = pkgv
+        pkginfo.pkgr = pkgr
         pkginfo.rdepends = sortpkglist(squashspaces(getpkgvar(pkg, 'RDEPENDS') or ""))
         pkginfo.rrecommends = sortpkglist(squashspaces(getpkgvar(pkg, 'RRECOMMENDS') or ""))
         pkginfo.files = squashspaces(getpkgvar(pkg, 'FILES') or "")
@@ -219,6 +242,17 @@ def write_pkghistory(pkginfo, d):
             f.write("PE = %s\n" %  pkginfo.pe)
         f.write("PV = %s\n" %  pkginfo.pv)
         f.write("PR = %s\n" %  pkginfo.pr)
+
+        pkgvars = {}
+        pkgvars['PKG'] = pkginfo.pkg if pkginfo.pkg != pkginfo.name else ''
+        pkgvars['PKGE'] = pkginfo.pkge if pkginfo.pkge != pkginfo.pe else ''
+        pkgvars['PKGV'] = pkginfo.pkgv if pkginfo.pkgv != pkginfo.pv else ''
+        pkgvars['PKGR'] = pkginfo.pkgr if pkginfo.pkgr != pkginfo.pr else ''
+        for pkgvar in pkgvars:
+            val = pkgvars[pkgvar]
+            if val:
+                f.write("%s = %s\n" % (pkgvar, val))
+
         f.write("RDEPENDS = %s\n" %  pkginfo.rdepends)
         f.write("RRECOMMENDS = %s\n" %  pkginfo.rrecommends)
         f.write("PKGSIZE = %d\n" %  pkginfo.size)
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index 6c6a085..55bd7b7 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -19,9 +19,10 @@ import bb.utils
 # How to display fields
 list_fields = ['DEPENDS', 'RDEPENDS', 'RRECOMMENDS', 'FILES', 'FILELIST', 'USER_CLASSES', 'IMAGE_CLASSES', 'IMAGE_FEATURES', 'IMAGE_LINGUAS', 'IMAGE_INSTALL', 'BAD_RECOMMENDATIONS']
 list_order_fields = ['PACKAGES']
+defaultval_fields = ['PKG', 'PKGE', 'PKGV', 'PKGR']
 numeric_fields = ['PKGSIZE', 'IMAGESIZE']
 # Fields to monitor
-monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', 'IMAGESIZE']
+monitor_fields = ['RDEPENDS', 'RRECOMMENDS', 'PACKAGES', 'FILELIST', 'PKGSIZE', 'IMAGESIZE', 'PKG', 'PKGE', 'PKGV', 'PKGR']
 # Percentage change to alert for numeric fields
 monitor_numeric_threshold = 20
 # Image files to monitor (note that image-info.txt is handled separately)
@@ -90,6 +91,10 @@ class ChangeRecord:
             else:
                 percentchg = 100
             out = '%s changed from %s to %s (%s%d%%)' % (self.fieldname, self.oldvalue or "''", self.newvalue or "''", '+' if percentchg > 0 else '', percentchg)
+        elif self.fieldname in defaultval_fields:
+            out = '%s changed from %s to %s' % (self.fieldname, self.oldvalue, self.newvalue)
+            if self.fieldname == 'PKG' and '[default]' in self.newvalue:
+                out += ' - may indicate debian renaming failure'
         elif self.fieldname in ['pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm']:
             if self.oldvalue and self.newvalue:
                 out = '%s changed:\n  ' % self.fieldname
@@ -299,6 +304,14 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
     adict = blob_to_dict(ablob)
     bdict = blob_to_dict(bblob)
 
+    defaultvals = {}
+    defaultvals['PKG'] = os.path.basename(path)
+    defaultvals['PKGE'] = adict.get('PE', '0')
+    defaultvals['PKGV'] = adict.get('PV', '')
+    defaultvals['PKGR'] = adict.get('PR', '')
+    for key in defaultvals:
+        defaultvals[key] = '%s [default]' % defaultvals[key]
+
     changes = []
     keys = list(set(adict.keys()) | set(bdict.keys()))
     for key in keys:
@@ -327,6 +340,13 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
                 blist.sort()
                 if ' '.join(alist) == ' '.join(blist):
                     continue
+
+            if key in defaultval_fields:
+                if not astr:
+                    astr = defaultvals[key]
+                elif not bstr:
+                    bstr = defaultvals[key]
+
             chg = ChangeRecord(path, key, astr, bstr, key in monitor_fields)
             changes.append(chg)
     return changes
-- 
1.7.9.5

[PATCH 6/9] scripts: add buildhistory-tag script

[Paul Eggleton]

This trivial script determines BUILDHISTORY_DIR using bitbake and then
passes the command line options to "git tag" on the buildhistory
repository. This is useful for setting a tag before making some changes;
even if the build takes a number of executions of bitbake (resulting in
multiple commits to the buildhistory repository) you can still do the
overall comparison easily.

Suggested by Ross Burton <ross.burton@intel.com>.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/buildhistory-tag |   42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
 create mode 100755 scripts/buildhistory-tag

diff --git a/scripts/buildhistory-tag b/scripts/buildhistory-tag
new file mode 100755
index 0000000..da90bcf
--- /dev/null
+++ b/scripts/buildhistory-tag
@@ -0,0 +1,42 @@
+#!/bin/sh
+
+# buildhistory-tag
+#
+# Copyright (C) 2012 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+
+if [ -z $1 ] ; then
+    echo "Please specify at least the name of the tag to set"
+    exit 1
+fi
+
+BUILDHISTORY_DIR=`bitbake -e | grep '^BUILDHISTORY_DIR=' | sed -e 's/^[A-Z_]*=//' -e 's/"//g'`
+if [ -z "$BUILDHISTORY_DIR" ]; then
+    echo "Unable to determine BUILDHISTORY_DIR"
+    exit 1
+fi
+
+if [ ! -d "$BUILDHISTORY_DIR" ]; then
+    echo "BUILDHISTORY_DIR $BUILDHISTORY_DIR does not exist"
+    exit 1
+fi
+
+if [ ! -d "$BUILDHISTORY_DIR/.git" ]; then
+    echo "BUILDHISTORY_DIR $BUILDHISTORY_DIR exists but is not a git repository"
+    exit 1
+fi
+
+(cd $BUILDHISTORY_DIR; git tag $@)
-- 
1.7.9.5

[PATCH 7/9] buildhistory_analysis: tidy up duplicate split_version function

[Paul Eggleton]

This function is now provided by bb.utils and since we have now bumped
the minimum bitbake version, we can switch to that one instead.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/buildhistory_analysis.py |   20 +-------------------
 1 file changed, 1 insertion(+), 19 deletions(-)

diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index 55bd7b7..a1a0408 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -261,24 +261,6 @@ def compare_lists(alines, blines):
     return filechanges
 
 
-def split_version(s):
-    """Split a version string into its constituent parts (PE, PV, PR)
-    FIXME: this is a duplicate of a new function in bitbake/lib/bb/utils -
-    we should switch to that once we can bump the minimum bitbake version
-    """
-    s = s.strip(" <>=")
-    e = 0
-    if s.count(':'):
-        e = int(s.split(":")[0])
-        s = s.split(":")[1]
-    r = ""
-    if s.count('-'):
-        r = s.rsplit("-", 1)[1]
-        s = s.rsplit("-", 1)[0]
-    v = s
-    return (e, v, r)
-
-
 def compare_pkg_lists(astr, bstr):
     depvera = bb.utils.explode_dep_versions(astr)
     depverb = bb.utils.explode_dep_versions(bstr)
@@ -290,7 +272,7 @@ def compare_pkg_lists(astr, bstr):
             dva = depvera[k]
             dvb = depverb[k]
             if dva and dvb and dva != dvb:
-                if bb.utils.vercmp(split_version(dva), split_version(dvb)) < 0:
+                if bb.utils.vercmp(bb.utils.split_version(dva), bb.utils.split_version(dvb)) < 0:
                     remove.append(k)
 
     for k in remove:
-- 
1.7.9.5

[PATCH 8/9] buildhistory_analysis: ignore removal of self-dependencies

[Paul Eggleton]

The recent removal of self-dependencies for dbg/dev packages produced
a fair amount of noise, so filter it out since we know this is a
reasonable change.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/buildhistory_analysis.py |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index a1a0408..a5da7d9 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -286,8 +286,9 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
     adict = blob_to_dict(ablob)
     bdict = blob_to_dict(bblob)
 
+    pkgname = os.path.basename(path)
     defaultvals = {}
-    defaultvals['PKG'] = os.path.basename(path)
+    defaultvals['PKG'] = pkgname
     defaultvals['PKGE'] = adict.get('PE', '0')
     defaultvals['PKGV'] = adict.get('PV', '')
     defaultvals['PKGR'] = adict.get('PR', '')
@@ -320,6 +321,9 @@ def compare_dict_blobs(path, ablob, bblob, report_all):
                 alist.sort()
                 blist = bstr.split()
                 blist.sort()
+                # We don't care about the removal of self-dependencies
+                if pkgname in alist and not pkgname in blist:
+                    alist.remove(pkgname)
                 if ' '.join(alist) == ' '.join(blist):
                     continue
 
-- 
1.7.9.5

[PATCH 9/9] classes/buildhistory: save metadata revisions

[Paul Eggleton]

Always write the metadata revisions for each layer into a machine-
readable "metadata-revs" file so that you can potentially link changes
in the output back to changes in the metadata. (Unlike the existing
similar build-id file, this is not specific to image changes.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass |   23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 76648c9..e121088 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -354,6 +354,15 @@ def buildhistory_get_layers(d):
     layertext = "Configured metadata layers:\n%s\n" % '\n'.join(get_layers_branch_rev(d))
     return layertext
 
+def buildhistory_get_metadata_revs(d):
+    # We want an easily machine-readable format here, so get_layers_branch_rev isn't quite what we want
+    layers = (d.getVar("BBLAYERS", True) or "").split()
+    medadata_revs = ["%-17s = %s:%s" % (os.path.basename(i), \
+        base_get_metadata_git_branch(i, None).strip(), \
+        base_get_metadata_git_revision(i, None)) \
+            for i in layers]
+    return '\n'.join(medadata_revs)
+
 
 def squashspaces(string):
     import re
@@ -382,19 +391,25 @@ buildhistory_commit() {
 		return
 	fi
 
+	# Create a machine-readable list of metadata revisions for each layer
+	cat > ${BUILDHISTORY_DIR}/metadata-revs <<END
+${@buildhistory_get_metadata_revs(d)}
+END
+
 	( cd ${BUILDHISTORY_DIR}/
 		# Initialise the repo if necessary
 		if [ ! -d .git ] ; then
 			git init -q
 		fi
-		# Ensure there are new/changed files to commit
-		repostatus=`git status --porcelain`
+		# Check if there are new/changed files to commit (other than metadata-revs)
+		repostatus=`git status --porcelain | grep -v " metadata-revs$"`
 		if [ "$repostatus" != "" ] ; then
-			git add ${BUILDHISTORY_DIR}/*
+			git add .
 			HOSTNAME=`hostname 2>/dev/null || echo unknown`
 			# porcelain output looks like "?? packages/foo/bar"
+			# Ensure we commit metadata-revs with the first commit
 			for entry in `echo "$repostatus" | awk '{print $2}' | awk -F/ '{print $1}' | sort | uniq` ; do
-				git commit ${BUILDHISTORY_DIR}/$entry -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
+				git commit $entry metadata-revs -m "$entry: Build ${BUILDNAME} of ${DISTRO} ${DISTRO_VERSION} for machine ${MACHINE} on $HOSTNAME" --author "${BUILDHISTORY_COMMIT_AUTHOR}" > /dev/null
 			done
 			if [ "${BUILDHISTORY_PUSH_REPO}" != "" ] ; then
 				git push -q ${BUILDHISTORY_PUSH_REPO}
-- 
1.7.9.5

Re: [PATCH 6/9] scripts: add buildhistory-tag script

[Burton, Ross]

On 2 August 2012 10:23, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> This trivial script determines BUILDHISTORY_DIR using bitbake and then
> passes the command line options to "git tag" on the buildhistory
> repository. This is useful for setting a tag before making some changes;
> even if the build takes a number of executions of bitbake (resulting in
> multiple commits to the buildhistory repository) you can still do the
> overall comparison easily.
>
> Suggested by Ross Burton <ross.burton@intel.com>.

I've since changed my local script to be "buildhistory" that just runs
git "$@", so "buildhistory log" etc work nicely.

Ross

Re: [PATCH 6/9] scripts: add buildhistory-tag script

[Paul Eggleton]

On Thursday 02 August 2012 11:18:20 Burton, Ross wrote:
> On 2 August 2012 10:23, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> > This trivial script determines BUILDHISTORY_DIR using bitbake and then
> > passes the command line options to "git tag" on the buildhistory
> > repository. This is useful for setting a tag before making some changes;
> > even if the build takes a number of executions of bitbake (resulting in
> > multiple commits to the buildhistory repository) you can still do the
> > overall comparison easily.
> > 
> > Suggested by Ross Burton <ross.burton@intel.com>.
> 
> I've since changed my local script to be "buildhistory" that just runs
> git "$@", so "buildhistory log" etc work nicely.

Interesting - sounds like it would be useful, but I wonder how it would then 
fit in with buildhistory-diff as a separate script (particularly with diff as a 
valid option to "buildhistory")...

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 6/9] scripts: add buildhistory-tag script

[Burton, Ross]

On 2 August 2012 11:49, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> Interesting - sounds like it would be useful, but I wonder how it would then
> fit in with buildhistory-diff as a separate script (particularly with diff as a
> valid option to "buildhistory")...

Rename it to -compare?

Ross

Re: [PATCH 0/9] buildhistory improvements

[Burton, Ross]

On 2 August 2012 10:23, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> A bunch of improvements for buildhistory, implementing some feature
> requests and todo items of my own, as well as tidying up some of the
> code.

I've been using this branch for an hour or so whilst reviewing
changes, the new features are useful and it didn't break.

Acked-by: Ross Burton <ross.burton@intel.com>

Ross

Re: [PATCH 0/9] buildhistory improvements

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

On Thu, Aug 02, 2012 at 12:27:05PM +0100, Burton, Ross wrote:
> On 2 August 2012 10:23, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> > A bunch of improvements for buildhistory, implementing some feature
> > requests and todo items of my own, as well as tidying up some of the
> > code.
> 
> I've been using this branch for an hour or so whilst reviewing
> changes, the new features are useful and it didn't break.
> 
> Acked-by: Ross Burton <ross.burton@intel.com>

My tests also look good, some interesting (expected) changes recorded
here

http://git.shr-project.org/git/?p=buildhistory.git;a=commitdiff;h=2fe33b15dcec9eb69f9075f138cea2405c061b00

Thanks!

Acked-by: Martin Jansa <Martin.Jansa@gmail.com>

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH 0/9] buildhistory improvements

[Andreas Müller]

On Thu, Aug 2, 2012 at 2:58 PM, Martin Jansa <martin.jansa@gmail.com> wrote:
> On Thu, Aug 02, 2012 at 12:27:05PM +0100, Burton, Ross wrote:
>> On 2 August 2012 10:23, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
>> > A bunch of improvements for buildhistory, implementing some feature
>> > requests and todo items of my own, as well as tidying up some of the
>> > code.
>>
>> I've been using this branch for an hour or so whilst reviewing
>> changes, the new features are useful and it didn't break.
>>
>> Acked-by: Ross Burton <ross.burton@intel.com>
>
> My tests also look good, some interesting (expected) changes recorded
> here
>
> http://git.shr-project.org/git/?p=buildhistory.git;a=commitdiff;h=2fe33b15dcec9eb69f9075f138cea2405c061b00
>
> Thanks!
>
> Acked-by: Martin Jansa <Martin.Jansa@gmail.com>
>
> Cheers,
>
> --
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com
Although I have no time for testing at the moment: thanks for doing this

Andreas

Re: [PATCH 0/9] buildhistory improvements

[Richard Purdie]

On Thu, 2012-08-02 at 10:23 +0100, Paul Eggleton wrote:
> A bunch of improvements for buildhistory, implementing some feature
> requests and todo items of my own, as well as tidying up some of the
> code.
> 
> 
> The following changes since commit 404d2d490fc347203e89d274530c17fb5f0aa20f:
> 
>   gcc-configure-target: Set native-system-header-dir for target gcc (2012-08-01 23:11:09 +0100)
> 
> are available in the git repository at:
> 
>   git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes7
>   http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes7
> 
> Paul Eggleton (9):
>   classes/buildhistory: remove obsolete flat package history feature
>   classes/buildhistory: ensure old package info is removed
>   classes/buildhistory: remove unused functions
>   classes/buildhistory: save preinst/postinst/prerm/postrm
>   classes/buildhistory: record PKG/PKGE/PKGV/PKGR
>   buildhistory_analysis: tidy up duplicate split_version function
>   buildhistory_analysis: ignore removal of self-dependencies
>   classes/buildhistory: save metadata revisions

Merged to master apart from:

  scripts: add buildhistory-tag script

which seems to need more discussion/work.

Cheers,

Richard