[PATCH 0/1] Further optimizations to the sstate.bbclass

[PATCH 0/1] Further optimizations to the sstate.bbclass

[Mark Hatle]

A few minor changes to the generation of the fixmepaths file in
the sstate cache file generation can lead to significant performance
improvements in recipes that generate large file sets.

The following changes since commit 9cd30beba77497288eeb2545920bc23f2a77cf16:

  .gitignore: add /bitbake to the ignore list for external distributions (2012-05-11 17:59:53 +0100)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib mhatle/sstate
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/sstate

Mark Hatle (1):
  sstate.bbclass: Optimize the generation and install path fixups

 meta/classes/sstate.bbclass |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

-- 
1.7.3.4

[PATCH 1/1] sstate.bbclass: Optimize the generation and install path fixups

[Mark Hatle]

The fixmepath file that is generated contains a list of all of the files
that need their paths fixed.  In the previous version the fixmepath was
generated to include all of the files that sed may have changed.  In the
new version, we first grep the files to see if they contain a path that
needs to be changed, only then do we perform the sed operation on those files.

This results in a modest performance increate in the creation of the sstate
file.  The following numbers include the do_package and do_populate_sysroot
tasks on the perl recipe.

Before the change:
real    4m23.018s
user    1m57.067s
sys     1m33.327s

After the change:
real    4m13.083s
user    1m54.062s
sys     1m26.064s

However, a more significnt performance gain is felt during the
extraction/install of sstate cache files, as the fixmepaths file now has a
significantly smaller list of files to modify.

Before the change:
real    0m39.798s
user    0m11.158s
sys     0m12.642s

After the change:
real    0m25.511s
user    0m8.408s
sys     0m5.077s

(All numbers above were recorded with a cold filesystem cache on a machine
with 12 GB of ram.)

Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
 meta/classes/sstate.bbclass |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index ad7d121..3fc615d 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -321,10 +321,13 @@ def sstate_hardcode_path(d):
 	sstate_builddir = d.getVar('SSTATE_BUILDDIR', True)
 
 	if bb.data.inherits_class('native', d) or bb.data.inherits_class('nativesdk', d) or bb.data.inherits_class('crosssdk', d) or bb.data.inherits_class('cross-canadian', d):
+		sstate_grep_cmd = "grep -l -e '%s'" % (staging)
 		sstate_sed_cmd = "sed -i -e 's:%s:FIXMESTAGINGDIR:g'" % (staging)
 	elif bb.data.inherits_class('cross', d):
+		sstate_grep_cmd = "grep -l -e '(%s|%s)'" % (staging_target, staging)
 		sstate_sed_cmd = "sed -i -e 's:%s:FIXMESTAGINGDIRTARGET:g; s:%s:FIXMESTAGINGDIR:g'" % (staging_target, staging)
 	else:
+		sstate_grep_cmd = "grep -l -e '%s'" % (staging_host)
 		sstate_sed_cmd = "sed -i -e 's:%s:FIXMESTAGINGDIRHOST:g'" % (staging_host)
 	
 	sstate_scan_cmd = d.getVar('SSTATE_SCAN_CMD', True)
@@ -333,7 +336,9 @@ def sstate_hardcode_path(d):
 	# fixmepath file needs relative paths, drop sstate_builddir prefix
 	sstate_filelist_relative_cmd = "sed -i -e 's:^%s::g' %sfixmepath" % (sstate_builddir, sstate_builddir)
 
-	sstate_hardcode_cmd = "%s | %s | xargs %s" % (sstate_scan_cmd, sstate_filelist_cmd, sstate_sed_cmd)
+	# Limit the fixpaths and sed operations based on the initial grep search
+	# This has the side effect of making sure the vfs cache is hot
+	sstate_hardcode_cmd = "%s | xargs %s | %s | xargs %s" % (sstate_scan_cmd, sstate_grep_cmd, sstate_filelist_cmd, sstate_sed_cmd)
 
 	print "Removing hardcoded paths from sstate package: '%s'" % (sstate_hardcode_cmd)
 	os.system(sstate_hardcode_cmd)
-- 
1.7.3.4

Re: [PATCH 0/1] Further optimizations to the sstate.bbclass

[Saul Wold]

On 05/11/2012 08:17 PM, Mark Hatle wrote:
> A few minor changes to the generation of the fixmepaths file in
> the sstate cache file generation can lead to significant performance
> improvements in recipes that generate large file sets.
>
> The following changes since commit 9cd30beba77497288eeb2545920bc23f2a77cf16:
>
>    .gitignore: add /bitbake to the ignore list for external distributions (2012-05-11 17:59:53 +0100)
>
> are available in the git repository at:
>    git://git.pokylinux.org/poky-contrib mhatle/sstate
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/sstate
>
> Mark Hatle (1):
>    sstate.bbclass: Optimize the generation and install path fixups
>
>   meta/classes/sstate.bbclass |    7 ++++++-
>   1 files changed, 6 insertions(+), 1 deletions(-)
>

Merged into OE-Core

Thanks
	Sau!