Re: Performance regression in bitbake and exec() vs fork()

[Richard Purdie <richard.purdie@linuxfoundation.org>]

To update on the current status of bitbake exec() vs fork(), we now have
the following branch:

http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rpurdie/newpseudo

which changes Poky to use the new version of pseudo, adds in an
appropriate wrapper script for bitbake and starts enabling/disabling
pseudo as appropriate as well as switching back to fork() instead of
exec().

We've still trying to work out exactly what this means for performance.
Things 'feel' very much faster and I've at least one use case showing a
double in speed showing the increase in task execution speed of fairly
empty tasks. My test was:

rm 'sstate-*qemu-config*'
MACHINE=qemux86 bitbake qemu-config -c clean
time MACHINE=qemux86 bitbake qemu-config

On the current master branch the timings were:

real 1m8.529s
user 0m58.870s
sys 0m4.690s

Whilst with newpseudo:

real 0m34.264s
user 0m26.200s
sys 0m2.340s

This is good as it gets us back to a much snappier feeling bitbake.

Mark has some numbers which don't quiet add up with improvements in read
and sys but an increase in user too, we're still looking to understand
them. I'd like to give the autobuilder a pass over these changes when we
have the opportunity and see what that real world performance looks
like.

Its likely that the speedups will be greatest on machines with small
numbers of cores which are primarily cpu bound. The benefits will
decrease on disk IO bound systems with large number of cores.

Cheers,

Richard