[PATCH 0/2] Buildhistory improvements

[PATCH 0/2] Buildhistory improvements

[Paul Eggleton]

(No changes from a few minutes ago, just not erroneously marked as RFC this time.)

The following changes since commit 7846f68537a942d340d5931e23a4fceb84b6edcb:

  vte: Fix conflict between FILES_${PN}-dbg and FILES_vte-dbg (2013-03-22 17:05:09 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fix5
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fix5

Paul Eggleton (2):
  classes/buildhistory: implement history collection for SDKs
  classes/buildhistory: trim trailing spaces in file listings

 meta/classes/buildhistory.bbclass      |  130 ++++++++++++++++++++++----------
 meta/classes/package_deb.bbclass       |   28 +++++++
 meta/classes/package_ipk.bbclass       |   24 ++++++
 meta/classes/package_rpm.bbclass       |   17 +++++
 meta/classes/populate_sdk_base.bbclass |    2 +
 meta/classes/populate_sdk_deb.bbclass  |    1 +
 meta/classes/populate_sdk_ipk.bbclass  |    2 +
 meta/classes/populate_sdk_rpm.bbclass  |    1 +
 meta/classes/rootfs_deb.bbclass        |   27 -------
 meta/classes/rootfs_ipk.bbclass        |   22 ------
 meta/classes/rootfs_rpm.bbclass        |   16 ----
 11 files changed, 165 insertions(+), 105 deletions(-)

-- 
1.7.10.4

[PATCH 0/2] buildhistory improvements

[Paul Eggleton]

The following changes since commit 2253e9f12734c6e6aa489942b5e4628eca1fa29d:

  classes/lib: Fix getcmdstatus breakage (2013-05-09 16:05:02 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-7
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-7

Paul Eggleton (2):
  buildhistory-diff: improve bad command-line argument handling
  classes/buildhistory: track contents of selected files in images

 meta/classes/buildhistory.bbclass    | 10 ++++++++++
 meta/lib/oe/buildhistory_analysis.py | 23 +++++++++++++++++------
 scripts/buildhistory-diff            | 22 +++++++++++++++++-----
 3 files changed, 44 insertions(+), 11 deletions(-)

-- 
1.8.1.2

[PATCH 1/2] buildhistory-diff: improve bad command-line argument handling

[Paul Eggleton]

* Check for existence of specified buildhistory directory and show a
  proper error message if it doesn't
* Show an error message instead of a traceback with a mangled revision
  if one of the specified git revisions is invalid
* Show usage information if --help is specified
* Write error messages to stderr

Fixes [YOCTO #4313].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/buildhistory-diff | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/scripts/buildhistory-diff b/scripts/buildhistory-diff
index 8493da4..30c2c08 100755
--- a/scripts/buildhistory-diff
+++ b/scripts/buildhistory-diff
@@ -2,7 +2,7 @@
 
 # Report significant differences in the buildhistory repository since a specific revision
 #
-# Copyright (C) 2012 Intel Corporation
+# Copyright (C) 2013 Intel Corporation
 # Author: Paul Eggleton <paul.eggleton@linux.intel.com>
 
 import sys
@@ -18,10 +18,10 @@ except ImportError:
 
 def main():
     if LooseVersion(git.__version__) < '0.3.1':
-        print("Version of GitPython is too old, please install GitPython (python-git) 0.3.1 or later in order to use this script")
+        sys.stderr.write("Version of GitPython is too old, please install GitPython (python-git) 0.3.1 or later in order to use this script\n")
         sys.exit(1)
 
-    if (len(sys.argv) < 3):
+    if len(sys.argv) < 3 or '--help' in sys.argv:
         print("Report significant differences in the buildhistory repository")
         print("Syntax: %s <buildhistory-path> <since-revision> [to-revision]" % os.path.basename(sys.argv[0]))
         print("If to-revision is not specified, it defaults to HEAD")
@@ -41,17 +41,29 @@ def main():
                 bitbakepath = os.path.abspath(os.path.join(pth, '..'))
                 break
         if not bitbakepath:
-            print("Unable to find bitbake by searching parent directory of this script or PATH")
+            sys.stderr.write("Unable to find bitbake by searching parent directory of this script or PATH\n")
             sys.exit(1)
 
     sys.path[0:0] = [newpath, bitbakepath + '/lib']
     import oe.buildhistory_analysis
 
+    buildhistory_dir = sys.argv[1]
+    if not os.path.exists(buildhistory_dir):
+        sys.stderr.write('Specified buildhistory directory "%s" does not exist\n' % buildhistory_dir)
+        sys.exit(1)
+
     if len(sys.argv) > 3:
         torev = sys.argv[3]
     else:
         torev = 'HEAD'
-    changes = oe.buildhistory_analysis.process_changes(sys.argv[1], sys.argv[2], torev)
+
+    import gitdb
+    try:
+        changes = oe.buildhistory_analysis.process_changes(buildhistory_dir, sys.argv[2], torev)
+    except gitdb.exc.BadObject as e:
+        sys.stderr.write('Specified git revision "%s" is not valid\n' % e.args[0])
+        sys.exit(1)
+
     for chg in changes:
         print('%s' % chg)
 
-- 
1.8.1.2

[PATCH 2/2] classes/buildhistory: track contents of selected files in images

[Paul Eggleton]

Add a BUILDHISTORY_IMAGE_FILES variable specifying a space-separated
list of files within an image to copy into buildhistory, so that changes
to them can be tracked. Typically this would be used for configuration
files, and by default this includes /etc/passwd and /etc/group, but the
user is free to extend this list by setting the variable in local.conf.

Implements [YOCTO #4154].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass    | 10 ++++++++++
 meta/lib/oe/buildhistory_analysis.py | 23 +++++++++++++++++------
 2 files changed, 27 insertions(+), 6 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 8efd41c..3b6ce99 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -12,6 +12,7 @@ BUILDHISTORY_DIR ?= "${TOPDIR}/buildhistory"
 BUILDHISTORY_DIR_IMAGE = "${BUILDHISTORY_DIR}/images/${MACHINE_ARCH}/${TCLIBC}/${IMAGE_BASENAME}"
 BUILDHISTORY_DIR_PACKAGE = "${BUILDHISTORY_DIR}/packages/${MULTIMACH_TARGET_SYS}/${PN}"
 BUILDHISTORY_DIR_SDK = "${BUILDHISTORY_DIR}/sdk/${SDK_NAME}"
+BUILDHISTORY_IMAGE_FILES ?= "/etc/passwd /etc/group"
 BUILDHISTORY_COMMIT ?= "0"
 BUILDHISTORY_COMMIT_AUTHOR ?= "buildhistory <buildhistory@${DISTRO}>"
 BUILDHISTORY_PUSH_REPO ?= ""
@@ -396,6 +397,15 @@ buildhistory_get_imageinfo() {
 
 	buildhistory_list_files ${IMAGE_ROOTFS} ${BUILDHISTORY_DIR_IMAGE}/files-in-image.txt
 
+	# Collect files requested in BUILDHISTORY_IMAGE_FILES
+	rm -rf ${BUILDHISTORY_DIR_IMAGE}/image-files
+	for f in ${BUILDHISTORY_IMAGE_FILES}; do
+		if [ -f ${IMAGE_ROOTFS}/$f ] ; then
+			mkdir -p ${BUILDHISTORY_DIR_IMAGE}/image-files/`dirname $f`
+			cp ${IMAGE_ROOTFS}/$f ${BUILDHISTORY_DIR_IMAGE}/image-files/$f
+		fi
+	done
+
 	# Record some machine-readable meta-information about the image
 	printf ""  > ${BUILDHISTORY_DIR_IMAGE}/image-info.txt
 	cat >> ${BUILDHISTORY_DIR_IMAGE}/image-info.txt <<END
diff --git a/meta/lib/oe/buildhistory_analysis.py b/meta/lib/oe/buildhistory_analysis.py
index 2306bcb..86b5a12 100644
--- a/meta/lib/oe/buildhistory_analysis.py
+++ b/meta/lib/oe/buildhistory_analysis.py
@@ -52,7 +52,10 @@ class ChangeRecord:
 
     def _str_internal(self, outer):
         if outer:
-            prefix = '%s: ' % self.path
+            if '/image-files/' in self.path:
+                prefix = '%s: ' % self.path.split('/image-files/')[0]
+            else:
+                prefix = '%s: ' % self.path
         else:
             prefix = ''
 
@@ -107,16 +110,21 @@ class ChangeRecord:
             diff = difflib.unified_diff(alines, blines, self.fieldname, self.fieldname, lineterm='')
             out += '\n  '.join(list(diff)[2:])
             out += '\n  --'
-        elif self.fieldname in img_monitor_files:
-            if outer:
-                prefix = 'Changes to %s ' % self.path
-            out = '(%s):\n  ' % self.fieldname
+        elif self.fieldname in img_monitor_files or '/image-files/' in self.path:
+            fieldname = self.fieldname
+            if '/image-files/' in self.path:
+                fieldname = os.path.join('/' + self.path.split('/image-files/')[1], self.fieldname)
+                out = 'Changes to %s:\n  ' % fieldname
+            else:
+                if outer:
+                    prefix = 'Changes to %s ' % self.path
+                out = '(%s):\n  ' % self.fieldname
             if self.filechanges:
                 out += '\n  '.join(['%s' % i for i in self.filechanges])
             else:
                 alines = self.oldvalue.splitlines()
                 blines = self.newvalue.splitlines()
-                diff = difflib.unified_diff(alines, blines, self.fieldname, self.fieldname, lineterm='')
+                diff = difflib.unified_diff(alines, blines, fieldname, fieldname, lineterm='')
                 out += '\n  '.join(list(diff))
                 out += '\n  --'
         else:
@@ -393,6 +401,9 @@ def process_changes(repopath, revision1, revision2 = 'HEAD', report_all = False)
                     changes.append(chg)
             elif filename == 'image-info.txt':
                 changes.extend(compare_dict_blobs(path, d.a_blob, d.b_blob, report_all))
+            elif '/image-files/' in path:
+                chg = ChangeRecord(path, filename, d.a_blob.data_stream.read(), d.b_blob.data_stream.read(), True)
+                changes.append(chg)
 
     # Look for added preinst/postinst/prerm/postrm
     # (without reporting newly added recipes)
-- 
1.8.1.2

Re: [PATCH 2/2] classes/buildhistory: track contents of selected files in images

[Otavio Salvador]

On Thu, May 9, 2013 at 12:57 PM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> Add a BUILDHISTORY_IMAGE_FILES variable specifying a space-separated
> list of files within an image to copy into buildhistory, so that changes
> to them can be tracked. Typically this would be used for configuration
> files, and by default this includes /etc/passwd and /etc/group, but the
> user is free to extend this list by setting the variable in local.conf.
>
> Implements [YOCTO #4154].
>
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>

This is awesome :-) Thanks by looking at it.

--
Otavio Salvador                             O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854              http://projetos.ossystems.com.br