Re: [yocto] What are the key factors for yocto build speed?

["Mikko Rapeli" <mikko.rapeli@bmw.de>]

On Thu, Mar 19, 2020 at 11:04:26AM +0000, Richard Purdie wrote:
> On Thu, 2020-03-19 at 08:05 +0000, Mikko Rapeli wrote:
> > Once this is done, IO still happens when anything calls sync() and
> > fsync() and worst offenders are package management tools. In yocto
> > builds, package manager actions to flush to disk are always useless
> > since rootfs images are going to be compressed and original ones
> > wiped by rm_work anyway.
> > I've tried to hook eatmydata library into the build which makes
> > sync() and fsync() calls no-ops but I've still failed to fix all the
> > tools and processes called during build from python code. For shell
> > based tasks this does it:
> > 
> > $ export LD_LIBRARY_PATH=/usr/lib/libeatmydata
> > $ export LD_PRELOAD=libeatmydata.so
> > $ grep -rn LD_PRELOAD conf/local.conf
> > conf/local.conf:305:BB_HASHBASE_WHITELIST_append = " LD_PRELOAD"
> > conf/local.conf:306:BB_HASHCONFIG_WHITELIST_append = " LD_PRELOAD"
> 
> Doesn't pseudo intercept and stop these sync calls already? Its
> supposed to so if its not, we should fix that.

I will double check, but I'm sure I see IO going to disk when plenty of RAM
is still available in page cache.

> > The effect is clearly visible during build time using Performance Co-
> > Pilot (pcp) or similar tools to monitor CPU, memory, IO and network
> > IO. The usage of RAM as page cache grows until limits are hit and
> > only then writes to disk start, except for the python image
> > classes... Hints to fix this are welcome!
> > 
> > To my knowledge of monitoring our builds, there is a lot of
> > optimization
> > potential to better build times. CPU are under utilized during
> > bitbake recipe parsing
> 
> Recipe parsing should hit 100% CPU, its one of the few places we can do
> that.

I'm not fully aware what bitbake does before starting task execution.
With sumo, there is an initial spike in CPU use and then a long single
thread wait where log shows "Initialising tasks..." and Cooker process
is using a single core. For me this takes at least one minutes for
every build. Same is visible with zeus too.

Example graph from pmchart:

https://mcfrisk.kapsi.fi/temp/bitbake_start_to_task_execution.png

> > , fetch, configure, package and rootfs tasks. 
> 
> Sadly these tasks are much harder.

Yep.

> > Memory is not fully utilized either since IO through sync()/fsync()
> > happens everywhere
> 
> non-pseudo tasks?

I'll try to check this case once more.

-Mikko