Re: [bitbake-devel] regression of 'world' performance?

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Tue, 2024-02-13 at 10:42 +0100, Alexander Kanavin wrote:
> On Tue, 13 Feb 2024 at 09:57, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> 
> > The patches fix low hanging issues but I suspect there is more we
> > could/should do in this code.
> 
> Thanks, I confirm that after replacing the previous 'return True ->
> continue' patch with the latest one, the profile is similar to that
> patch:
> 
> Tue Feb 13 10:29:59 2024    profile-mainloop.log
> 
>          171090640 function calls (170677396 primitive calls) in
> 1000.777 seconds
> 
>    Ordered by: internal time
> 
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>     35784  301.390    0.008  509.820    0.014
> /srv/work/alex/poky/bitbake/lib/bb/runqueue.py:225(next_buildable_task)
>     34092  164.164    0.005  171.500    0.005
> /srv/work/alex/poky/bitbake/lib/bb/runqueue.py:2438(build_taskdepdata)
>     35787  120.062    0.003  120.062    0.003 {method 'difference' of
> 'set' objects}
>     71572   55.117    0.001   55.117    0.001 {method
> 'difference_update' of 'set' objects}
> 17639/15964   49.593    0.003   49.869    0.003
> /srv/work/alex/poky/bitbake/lib/bb/runqueue.py:2686(scenequeue_updatecounters)
>     34096   44.287    0.001   44.287    0.001 {method 'write' of
> '_io.BufferedWriter' objects}
>    298936   43.070    0.000   43.070    0.000 {built-in method _pickle.dumps}
>     52656   39.374    0.001  936.397    0.018
> /srv/work/alex/poky/bitbake/lib/bb/runqueue.py:2150(execute)
>     35787   21.456    0.001   21.456    0.001 {method
> 'intersection_update' of 'set' objects}
>     35783   10.646    0.000   10.646    0.000 {method 'index' of 'list' objects}
>    314739    7.954    0.000    8.176    0.000 {built-in method io.open}
>  90090452    7.660    0.000    7.660    0.000 {method 'append' of
> 'list' objects}
> 
> 
> There's only appending to list which is still done tens of millions of
> times, not sure if that's a concern. The bulk of it is in
> build_taskdepdata().

We can work out how it compares to what we'd expect. If you generate
task-depends.dot and count the number of dependencies between tasks,
I'd expect the number of appends in build_taskdepdata to be be
approximately:

No. Tasks * No. Dependencies * 0.5

The half is assuming that the number of dependencies a task as scales
linearly which it doesn't but should give a rough number. It certainly
shouldn't be more than tasks * dependencies as an upper bound.

> I first noticed the issue when working on python 3.12 upgrade by the
> way: this version has a higher amount of breaking changes that has
> been typical (this was noticeable already in plain oe-core builds), so
> as a courtesy to those taking care of meta-oe I've decided to fix
> everything there as well. Which meant adding all meta-oe layers, and
> running bitbake -k world with that.

Thanks for doing that, it is appreciated. I think it is also good to
flag the poor performance as there is something we can do about this
case.

Cheers,

Richard