[PATCH 0/3] Buildhistory fixes

[PATCH 0/3] Buildhistory fixes

[Paul Eggleton]

An optimisation for image package size list collection, a fix for image
dependency graph filtering, and a minor comment grammar fix.


The following changes since commit d60806e56aed2f62f6a0e030a564f7fdc4a1314d:

  build-appliance-image: Exclude DDATETIME from task signature (2016-04-03 15:51:10 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/buildhistory-fixes

Paul Eggleton (3):
  classes/buildhistory: optimise getting package size list
  classes/buildhistory: fix filtering of depends-nokernel.dot
  classes/buildhistory: fix grammar in comments

 meta/classes/buildhistory.bbclass | 17 +++++------------
 scripts/oe-pkgdata-util           | 26 +++++++++++++++++++++-----
 2 files changed, 26 insertions(+), 17 deletions(-)

-- 
2.5.5

[PATCH 1/3] classes/buildhistory: optimise getting package size list

[Paul Eggleton]

Invoking oe-pkgdata-util in turn for every package in the list was slow
with a large image. Modify oe-pkgdata-util's read-value command to take
an option to read the list of packages from a file, as well as prefix
the value with the package name; we can then use this to get all of the
package sizes at once. This reduces the time to gather this information
from minutes to just a second or two.

Fixes [YOCTO #7339].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 13 +++----------
 scripts/oe-pkgdata-util           | 26 +++++++++++++++++++++-----
 2 files changed, 24 insertions(+), 15 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 108275a..8af36c5 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -3,7 +3,7 @@
 #
 # Based in part on testlab.bbclass and packagehistory.bbclass
 #
-# Copyright (C) 2011-2014 Intel Corporation
+# Copyright (C) 2011-2016 Intel Corporation
 # Copyright (C) 2007-2011 Koen Kooi <koen@openembedded.org>
 #
 
@@ -416,15 +416,8 @@ buildhistory_get_installed() {
 	rm $1/depends.tmp
 
 	# Produce installed package sizes list
-	printf "" > $1/installed-package-sizes.tmp
-	cat $pkgcache | while read pkg pkgfile pkgarch
-	do
-		size=`oe-pkgdata-util -p ${PKGDATA_DIR} read-value "PKGSIZE" ${pkg}_${pkgarch}`
-		if [ "$size" != "" ] ; then
-			echo "$size $pkg" >> $1/installed-package-sizes.tmp
-		fi
-	done
-	cat $1/installed-package-sizes.tmp | sort -n -r | awk '{print $1 "\tKiB " $2}' > $1/installed-package-sizes.txt
+	oe-pkgdata-util -p ${PKGDATA_DIR} read-value "PKGSIZE" -n -f $pkgcache > $1/installed-package-sizes.tmp
+	cat $1/installed-package-sizes.tmp | awk '{print $2 "\tKiB " $1}' | sort -n -r > $1/installed-package-sizes.txt
 	rm $1/installed-package-sizes.tmp
 
 	# We're now done with the cache, delete it
diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util
index 8e22e02..a04e44d 100755
--- a/scripts/oe-pkgdata-util
+++ b/scripts/oe-pkgdata-util
@@ -161,8 +161,18 @@ def glob(args):
 def read_value(args):
     # Handle both multiple arguments and multiple values within an arg (old syntax)
     packages = []
-    for pkgitem in args.pkg:
-        packages.extend(pkgitem.split())
+    if args.file:
+        with open(args.file, 'r') as f:
+            for line in f:
+                splitline = line.split()
+                if splitline:
+                    packages.append(splitline[0])
+    else:
+        for pkgitem in args.pkg:
+            packages.extend(pkgitem.split())
+        if not packages:
+            logger.error("No packages specified")
+            sys.exit(1)
 
     def readvar(pkgdata_file, valuename):
         val = ""
@@ -187,9 +197,13 @@ def read_value(args):
                 qvar = "%s_%s" % (args.valuename, mappedpkg)
                 # PKGSIZE is now in bytes, but we we want it in KB
                 pkgsize = (int(readvar(revlink, qvar)) + 1024 // 2) // 1024
-                print("%d" % pkgsize)
+                value = "%d" % pkgsize
+            else:
+                value = readvar(revlink, qvar)
+            if args.prefix_name:
+                print('%s %s' % (pkg_name, value))
             else:
-                print(readvar(revlink, qvar))
+                print(value)
 
 def lookup_pkglist(pkgs, pkgdata_dir, reverse):
     if reverse:
@@ -465,7 +479,9 @@ def main():
                                           help='Read any pkgdata value for one or more packages',
                                           description='Reads the named value from the pkgdata files for the specified packages')
     parser_read_value.add_argument('valuename', help='Name of the value to look up')
-    parser_read_value.add_argument('pkg', nargs='+', help='Runtime package name to look up')
+    parser_read_value.add_argument('pkg', nargs='*', help='Runtime package name to look up')
+    parser_read_value.add_argument('-f', '--file', help='Read package names from the specified file (one per line, first field only)')
+    parser_read_value.add_argument('-n', '--prefix-name', help='Prefix output with package name', action='store_true')
     parser_read_value.set_defaults(func=read_value)
 
     parser_glob = subparsers.add_parser('glob',
-- 
2.5.5

[PATCH 2/3] classes/buildhistory: fix filtering of depends-nokernel.dot

[Paul Eggleton]

For images we produce a number of filtered dependency .dot files for
readability, the first of which is depends-nokernel.dot which filters
out just the kernel itself (not kernel modules). Unfortunately the
filter specifications hadn't been updated for the dash-to-underscore
removal or the 4.x kernel upgrade, thus the filtering wasn't actually
doing anything.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 8af36c5..60d19e2 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -425,7 +425,7 @@ buildhistory_get_installed() {
 
 	if [ "$2" != "sdk" ] ; then
 		# Produce some cut-down graphs (for readability)
-		grep -v kernel_image $1/depends.dot | grep -v kernel-2 | grep -v kernel-3 > $1/depends-nokernel.dot
+		grep -v kernel-image $1/depends.dot | grep -v kernel-3 | grep -v kernel-4 > $1/depends-nokernel.dot
 		grep -v libc6 $1/depends-nokernel.dot | grep -v libgcc > $1/depends-nokernel-nolibc.dot
 		grep -v update- $1/depends-nokernel-nolibc.dot > $1/depends-nokernel-nolibc-noupdate.dot
 		grep -v kernel-module $1/depends-nokernel-nolibc-noupdate.dot > $1/depends-nokernel-nolibc-noupdate-nomodules.dot
-- 
2.5.5

[PATCH 3/3] classes/buildhistory: fix grammar in comments

[Paul Eggleton]

Fix a minor grammatical error in the comments here.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 60d19e2..0fa9b16 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -42,7 +42,7 @@ BUILDHISTORY_COMMIT_AUTHOR ?= "buildhistory <buildhistory@${DISTRO}>"
 BUILDHISTORY_PUSH_REPO ?= ""
 
 SSTATEPOSTINSTFUNCS_append = " buildhistory_emit_pkghistory"
-# We want to avoid influence the signatures of sstate tasks - first the function itself:
+# We want to avoid influencing the signatures of sstate tasks - first the function itself:
 sstate_install[vardepsexclude] += "buildhistory_emit_pkghistory"
 # then the value added to SSTATEPOSTINSTFUNCS:
 SSTATEPOSTINSTFUNCS[vardepvalueexclude] .= "| buildhistory_emit_pkghistory"
-- 
2.5.5

[PATCH 0/3] Buildhistory fixes

[Paul Eggleton]

Fixes for a couple of buildhistory bugs in YP bugzilla plus one extra
that I found in the process.


The following changes since commit f9324af88a99eca28b160fa31aa4516fd397e44b:

  distutils3: pass build arguments when doing a clean (2018-07-10 11:10:14 +0100)

are available in the Git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/buildhistory-fixes2
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=paule/buildhistory-fixes2

Paul Eggleton (3):
  lib/oe/buildhistory_analysis: drop related field feature
  classes/buildhistory: handle packaged files with names containing
    spaces
  classes/buildhistory: properly process escaped chars from pkgdata

 meta/classes/buildhistory.bbclass    |  5 ++--
 meta/lib/oe/buildhistory_analysis.py | 42 ++++++++--------------------
 2 files changed, 15 insertions(+), 32 deletions(-)

-- 
2.17.1