setscene and perl -- pathological case

setscene and perl -- pathological case

[Mark Hatle]

We've discovered that perl seems to be a pretty bad pathological case for setscene.

Easy way to see what I mean:

bitbake perl
bitbake -c clean perl
time bitbake perl

The system isn't doing anything but running the setscene elements to bring perl 
from the sstate-cache into the sysroot and related directories.. but it takes a 
very long time to do this:

real    3m7.430s
user    0m36.316s
sys     2m46.742s

That is on my 8 core w/ 12 GB of ram system..  (standard SATA disks)

Digging into this, I found that the majority of the time is spent in 
meta/classes/sstate.bbclass: sstate_installpkg

If you look for the three os.system("sed ...  calls, almost all of the 3 minutes 
is there.  Perl has so many text files with embedded paths that a significant 
amount of time is spent preparing these files and fixing paths.

I did a test and changed the three sed cases into a single sed operation:

-            os.system("sed -i -e s:FIXMESTAGINGDIRTARGET:%s:g %s" % 
(staging_target, sstateinst + file))
-            os.system("sed -i -e s:FIXMESTAGINGDIRHOST:%s:g %s" % 
(staging_host, sstateinst + file))
-            os.system("sed -i -e s:FIXMESTAGINGDIR:%s:g %s" % (staging, 
sstateinst + file))
+            os.system("sed -i -e s:FIXMESTAGINGDIRTARGET:%s:g -e 
s:FIXMESTAGINGDIRHOST:%s:g -e s:FIXMESTAGINGDIR:%s:g %s " % (staging_target, 
staging_host, staging, sstateinst + file))

The result:

real    1m20.979s
user    0m19.400s
sys     1m4.115s

This is a very significant performance improvement.  I'm not sure if it's 
possible to speed this up even more, but it would be nice if we could get things 
under a minute.

Does anyone see a way we could do this quicker or in a better way to boost 
performance?  (One possibility is figure out how to thread the file processing 
and use available processors...)

--Mark

Re: setscene and perl -- pathological case

[Khem Raj]

On Thu, May 10, 2012 at 3:20 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
>
> This is a very significant performance improvement.  I'm not sure if it's
> possible to speed this up even more, but it would be nice if we could get
> things under a minute.

good job. in theory you could pythonise it even further using re module

Re: setscene and perl -- pathological case

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2369 bytes --]

On Thu, May 10, 2012 at 05:20:31PM -0500, Mark Hatle wrote:
> We've discovered that perl seems to be a pretty bad pathological case for setscene.
> 
> Easy way to see what I mean:
> 
> bitbake perl
> bitbake -c clean perl
> time bitbake perl
> 
> The system isn't doing anything but running the setscene elements to bring perl 
> from the sstate-cache into the sysroot and related directories.. but it takes a 
> very long time to do this:
> 
> real    3m7.430s
> user    0m36.316s
> sys     2m46.742s
> 
> That is on my 8 core w/ 12 GB of ram system..  (standard SATA disks)
> 
> Digging into this, I found that the majority of the time is spent in 
> meta/classes/sstate.bbclass: sstate_installpkg
> 
> If you look for the three os.system("sed ...  calls, almost all of the 3 minutes 
> is there.  Perl has so many text files with embedded paths that a significant 
> amount of time is spent preparing these files and fixing paths.
> 
> I did a test and changed the three sed cases into a single sed operation:
> 
> -            os.system("sed -i -e s:FIXMESTAGINGDIRTARGET:%s:g %s" % 
> (staging_target, sstateinst + file))
> -            os.system("sed -i -e s:FIXMESTAGINGDIRHOST:%s:g %s" % 
> (staging_host, sstateinst + file))
> -            os.system("sed -i -e s:FIXMESTAGINGDIR:%s:g %s" % (staging, 
> sstateinst + file))
> +            os.system("sed -i -e s:FIXMESTAGINGDIRTARGET:%s:g -e 
> s:FIXMESTAGINGDIRHOST:%s:g -e s:FIXMESTAGINGDIR:%s:g %s " % (staging_target, 
> staging_host, staging, sstateinst + file))
> 
> The result:
> 
> real    1m20.979s
> user    0m19.400s
> sys     1m4.115s
> 
> This is a very significant performance improvement.  I'm not sure if it's 
> possible to speed this up even more, but it would be nice if we could get things 
> under a minute.
> 
> Does anyone see a way we could do this quicker or in a better way to boost 
> performance?  (One possibility is figure out how to thread the file processing 
> and use available processors...)

I guess it does process for all files in fixmepath file, so try to run
sed on all of them (or chunks) together with xargs like in:
http://git.openembedded.org/openembedded-core/commit/meta/classes/sstate.bbclass?id=94c52d68fc2ce258bcc5b0978ac73413480a1a93

Cheers,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]