* [PATCH 0/1] Performance improvement in install_sstatepkg
@ 2012-05-10 23:13 Mark Hatle
2012-05-10 23:13 ` [PATCH 1/1] sstate.bbclass: Improve sstate_installpkg performance Mark Hatle
2012-05-11 17:39 ` [PATCH 0/1] Performance improvement in install_sstatepkg Saul Wold
0 siblings, 2 replies; 6+ messages in thread
From: Mark Hatle @ 2012-05-10 23:13 UTC (permalink / raw)
To: openembedded-core
Significant performance improvement when large numbers of files are involved
with install_sstatepkg.
To reproduce the problem:
bitbake perl
bitbake -c clean perl
time bitbake perl
The following changes since commit 910452727fc277c1caec7612b36c37b58d845350:
eglibc: Add patch to fix /var installation location (2012-05-09 21:42:37 +0100)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib mhatle/sstate
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/sstate
Mark Hatle (1):
sstate.bbclass: Improve sstate_installpkg performance
meta/classes/sstate.bbclass | 28 +++++++++++++++++++++-------
1 files changed, 21 insertions(+), 7 deletions(-)
--
1.7.3.4
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/1] sstate.bbclass: Improve sstate_installpkg performance
2012-05-10 23:13 [PATCH 0/1] Performance improvement in install_sstatepkg Mark Hatle
@ 2012-05-10 23:13 ` Mark Hatle
2012-05-11 0:03 ` Peter Seebach
2012-05-11 0:17 ` Richard Purdie
2012-05-11 17:39 ` [PATCH 0/1] Performance improvement in install_sstatepkg Saul Wold
1 sibling, 2 replies; 6+ messages in thread
From: Mark Hatle @ 2012-05-10 23:13 UTC (permalink / raw)
To: openembedded-core
In a pathological case, lots of files to process, the sstate_installpkg
performance was very poor. It interated over each file and ran 3
individual sed commands per file. Changing this to keep iterating
but running only a single command took about 1/3 time time.
However, when looking at the corresponding sstate_hardcode_path
function, it was clear we could optimize this further.
Using the same encoding logic to specify only the minimumal sed
operation necessary, and using xargs to avoid the os.system call the
install step was able to be performed in 13% of the original time.
Example timing numbers for perl:
3m7s original code
1m20s single sed, but interating
0m26s using xargs and limited sed
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
meta/classes/sstate.bbclass | 28 +++++++++++++++++++++-------
1 files changed, 21 insertions(+), 7 deletions(-)
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index a8c98e5..ad7d121 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -174,18 +174,29 @@ def sstate_installpkg(ss, d):
bb.build.exec_func('sstate_unpack_package', d)
# Fixup hardcoded paths
+ #
+ # Note: The logic below must match the reverse logic in
+ # sstate_hardcode_path(d)
+
fixmefn = sstateinst + "fixmepath"
if os.path.isfile(fixmefn):
staging = d.getVar('STAGING_DIR', True)
staging_target = d.getVar('STAGING_DIR_TARGET', True)
staging_host = d.getVar('STAGING_DIR_HOST', True)
- fixmefd = open(fixmefn, "r")
- fixmefiles = fixmefd.readlines()
- fixmefd.close()
- for file in fixmefiles:
- os.system("sed -i -e s:FIXMESTAGINGDIRTARGET:%s:g %s" % (staging_target, sstateinst + file))
- os.system("sed -i -e s:FIXMESTAGINGDIRHOST:%s:g %s" % (staging_host, sstateinst + file))
- os.system("sed -i -e s:FIXMESTAGINGDIR:%s:g %s" % (staging, sstateinst + file))
+
+ if bb.data.inherits_class('native', d) or bb.data.inherits_class('nativesdk', d) or bb.data.inherits_class('crosssdk', d) or bb.data.inherits_class('cross-canadian', d):
+ sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIR:%s:g'" % (staging)
+ elif bb.data.inherits_class('cross', d):
+ sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRTARGET:%s:g; s:FIXMESTAGINGDIR:%s:g'" % (staging_target, staging)
+ else:
+ sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRHOST:%s:g'" % (staging_host)
+
+ # Add sstateinst to each filename in fixmepath, use xargs to efficiently call sed
+ sstate_hardcode_cmd = "sed -e 's:^:%s:g' %s | xargs %s" % (sstateinst, fixmefn, sstate_sed_cmd)
+
+ print "Replacing fixme paths in sstate package: %s" % (sstate_hardcode_cmd)
+ os.system(sstate_hardcode_cmd)
+
# Need to remove this or we'd copy it into the target directory and may
# conflict with another writer
os.remove(fixmefn)
@@ -300,6 +311,9 @@ python sstate_cleanall() {
def sstate_hardcode_path(d):
# Need to remove hardcoded paths and fix these when we install the
# staging packages.
+ #
+ # Note: the logic in this function needs to match the reverse logic
+ # in sstate_installpkg(ss, d)
staging = d.getVar('STAGING_DIR', True)
staging_target = d.getVar('STAGING_DIR_TARGET', True)
--
1.7.3.4
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] sstate.bbclass: Improve sstate_installpkg performance
2012-05-10 23:13 ` [PATCH 1/1] sstate.bbclass: Improve sstate_installpkg performance Mark Hatle
@ 2012-05-11 0:03 ` Peter Seebach
2012-05-11 0:18 ` Richard Purdie
2012-05-11 0:17 ` Richard Purdie
1 sibling, 1 reply; 6+ messages in thread
From: Peter Seebach @ 2012-05-11 0:03 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 10 May 2012 18:13:38 -0500
Mark Hatle <mark.hatle@windriver.com> wrote:
> + sstate_hardcode_cmd = "sed -e 's:^:%s:g' %s | xargs %s" %
> (sstateinst, fixmefn, sstate_sed_cmd)
How confident are we that the file names can never have whitespace in
them?
-s
--
Listen, get this. Nobody with a good compiler needs to be justified.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] sstate.bbclass: Improve sstate_installpkg performance
2012-05-10 23:13 ` [PATCH 1/1] sstate.bbclass: Improve sstate_installpkg performance Mark Hatle
2012-05-11 0:03 ` Peter Seebach
@ 2012-05-11 0:17 ` Richard Purdie
1 sibling, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2012-05-11 0:17 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2012-05-10 at 18:13 -0500, Mark Hatle wrote:
> In a pathological case, lots of files to process, the sstate_installpkg
> performance was very poor. It interated over each file and ran 3
> individual sed commands per file. Changing this to keep iterating
> but running only a single command took about 1/3 time time.
>
> However, when looking at the corresponding sstate_hardcode_path
> function, it was clear we could optimize this further.
>
> Using the same encoding logic to specify only the minimumal sed
> operation necessary, and using xargs to avoid the os.system call the
> install step was able to be performed in 13% of the original time.
>
> Example timing numbers for perl:
>
> 3m7s original code
> 1m20s single sed, but interating
> 0m26s using xargs and limited sed
>
> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
> ---
> meta/classes/sstate.bbclass | 28 +++++++++++++++++++++-------
> 1 files changed, 21 insertions(+), 7 deletions(-)
>
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index a8c98e5..ad7d121 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -174,18 +174,29 @@ def sstate_installpkg(ss, d):
> bb.build.exec_func('sstate_unpack_package', d)
>
> # Fixup hardcoded paths
> + #
> + # Note: The logic below must match the reverse logic in
> + # sstate_hardcode_path(d)
> +
> fixmefn = sstateinst + "fixmepath"
> if os.path.isfile(fixmefn):
> staging = d.getVar('STAGING_DIR', True)
> staging_target = d.getVar('STAGING_DIR_TARGET', True)
> staging_host = d.getVar('STAGING_DIR_HOST', True)
> - fixmefd = open(fixmefn, "r")
> - fixmefiles = fixmefd.readlines()
> - fixmefd.close()
> - for file in fixmefiles:
> - os.system("sed -i -e s:FIXMESTAGINGDIRTARGET:%s:g %s" % (staging_target, sstateinst + file))
> - os.system("sed -i -e s:FIXMESTAGINGDIRHOST:%s:g %s" % (staging_host, sstateinst + file))
> - os.system("sed -i -e s:FIXMESTAGINGDIR:%s:g %s" % (staging, sstateinst + file))
> +
> + if bb.data.inherits_class('native', d) or bb.data.inherits_class('nativesdk', d) or bb.data.inherits_class('crosssdk', d) or bb.data.inherits_class('cross-canadian', d):
> + sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIR:%s:g'" % (staging)
> + elif bb.data.inherits_class('cross', d):
> + sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRTARGET:%s:g; s:FIXMESTAGINGDIR:%s:g'" % (staging_target, staging)
> + else:
> + sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRHOST:%s:g'" % (staging_host)
> +
> + # Add sstateinst to each filename in fixmepath, use xargs to efficiently call sed
> + sstate_hardcode_cmd = "sed -e 's:^:%s:g' %s | xargs %s" % (sstateinst, fixmefn, sstate_sed_cmd)
> +
> + print "Replacing fixme paths in sstate package: %s" % (sstate_hardcode_cmd)
> + os.system(sstate_hardcode_cmd)
> +
> # Need to remove this or we'd copy it into the target directory and may
> # conflict with another writer
> os.remove(fixmefn)
> @@ -300,6 +311,9 @@ python sstate_cleanall() {
> def sstate_hardcode_path(d):
> # Need to remove hardcoded paths and fix these when we install the
> # staging packages.
> + #
> + # Note: the logic in this function needs to match the reverse logic
> + # in sstate_installpkg(ss, d)
>
> staging = d.getVar('STAGING_DIR', True)
> staging_target = d.getVar('STAGING_DIR_TARGET', True)
I was thinking this looked familiar, it was the other half of the
problem we originally solved:
http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/sstate.bbclass?id=db94ad4cf32d8ce3e97a8287d1c89a58d008a142
:)
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/1] sstate.bbclass: Improve sstate_installpkg performance
2012-05-11 0:03 ` Peter Seebach
@ 2012-05-11 0:18 ` Richard Purdie
0 siblings, 0 replies; 6+ messages in thread
From: Richard Purdie @ 2012-05-11 0:18 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On Thu, 2012-05-10 at 19:03 -0500, Peter Seebach wrote:
> On Thu, 10 May 2012 18:13:38 -0500
> Mark Hatle <mark.hatle@windriver.com> wrote:
>
> > + sstate_hardcode_cmd = "sed -e 's:^:%s:g' %s | xargs %s" %
> > (sstateinst, fixmefn, sstate_sed_cmd)
>
> How confident are we that the file names can never have whitespace in
> them?
Fairly since there are 101 other places this would have broken first.
The day autotools thinks about supporting that, we might start to think
about it too. Until then there is little point in worrying about it.
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 0/1] Performance improvement in install_sstatepkg
2012-05-10 23:13 [PATCH 0/1] Performance improvement in install_sstatepkg Mark Hatle
2012-05-10 23:13 ` [PATCH 1/1] sstate.bbclass: Improve sstate_installpkg performance Mark Hatle
@ 2012-05-11 17:39 ` Saul Wold
1 sibling, 0 replies; 6+ messages in thread
From: Saul Wold @ 2012-05-11 17:39 UTC (permalink / raw)
To: Patches and discussions about the oe-core layer
On 05/10/2012 04:13 PM, Mark Hatle wrote:
> Significant performance improvement when large numbers of files are involved
> with install_sstatepkg.
>
> To reproduce the problem:
>
> bitbake perl
> bitbake -c clean perl
> time bitbake perl
>
> The following changes since commit 910452727fc277c1caec7612b36c37b58d845350:
>
> eglibc: Add patch to fix /var installation location (2012-05-09 21:42:37 +0100)
>
> are available in the git repository at:
> git://git.pokylinux.org/poky-contrib mhatle/sstate
> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/sstate
>
> Mark Hatle (1):
> sstate.bbclass: Improve sstate_installpkg performance
>
> meta/classes/sstate.bbclass | 28 +++++++++++++++++++++-------
> 1 files changed, 21 insertions(+), 7 deletions(-)
>
Nice catch!
Merged into OE-Core
Thanks
Sau!
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-05-11 17:49 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-05-10 23:13 [PATCH 0/1] Performance improvement in install_sstatepkg Mark Hatle
2012-05-10 23:13 ` [PATCH 1/1] sstate.bbclass: Improve sstate_installpkg performance Mark Hatle
2012-05-11 0:03 ` Peter Seebach
2012-05-11 0:18 ` Richard Purdie
2012-05-11 0:17 ` Richard Purdie
2012-05-11 17:39 ` [PATCH 0/1] Performance improvement in install_sstatepkg Saul Wold
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox