Re: [bitbake-devel] regression of 'world' performance?

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Fri, 2024-02-09 at 17:16 +0100, Alexander Kanavin wrote:
> On Fri, 9 Feb 2024 at 17:06, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > Iterating 28k entries isn't an issue for python. Yes, it will be "slow"
> > when you put logging around it but I doubt it is the real problem.
> > 
> > What is more likely is one of the calls like:
> > 
> > self.check_dependencies(xxx)
> > 
> > getting called once per entry and that call being slow.
> > 
> > Can you try the profile -P data please. If it doesn't make sense, share
> > it somewhere and I'll take a look.
> 
> I get:
> 
> Parsing of 2966 .bb files complete (0 cached, 2966 parsed). 4758
> targets, 129 skipped, 0 masked, 0 errors.
> ERROR: Command execution failed: Traceback (most recent call last):
>   File "/srv/work/alex/poky/bitbake/lib/bb/cooker.py", line 2284, in parse_next
>     parsed, mc, result = next(self.results)
>                          ^^^^^^^^^^^^^^^^^^
> StopIteration
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
>   File "/srv/work/alex/poky/bitbake/lib/bb/command.py", line 122, in
> runAsyncCommand
>     self.cooker.updateCache()
>   File "/srv/work/alex/poky/bitbake/lib/bb/cooker.py", line 1635, in updateCache
>     if not self.parser.parse_next():
>            ^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/srv/work/alex/poky/bitbake/lib/bb/cooker.py", line 2293, in parse_next
>     self.shutdown()
>   File "/srv/work/alex/poky/bitbake/lib/bb/cooker.py", line 2244, in shutdown
>     bb.utils.process_profilelog(profiles, pout = pout)
>   File "/srv/work/alex/poky/bitbake/lib/bb/utils.py", line 1159, in
> process_profilelog
>     p = pstats.Stats(*fn, stream=pout)
>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/usr/lib/python3.11/pstats.py", line 116, in __init__
>     self.add(*args)
>   File "/usr/lib/python3.11/pstats.py", line 174, in add
>     item = Stats(item)
>            ^^^^^^^^^^^
>   File "/usr/lib/python3.11/pstats.py", line 115, in __init__
>     self.init(arg)
>   File "/usr/lib/python3.11/pstats.py", line 129, in init
>     self.load_stats(arg)
>   File "/usr/lib/python3.11/pstats.py", line 143, in load_stats
>     self.stats = marshal.load(f)
>                  ^^^^^^^^^^^^^^^
> EOFError: EOF read where object expected


That is very unexpected. I just tested "bitbake bash -P" myself and it
did work out ok. The stack trace doesn't really make sense to me so no
sure what is going on or why this is failing :(

That does look like it might be the post processing code failing (it
saves the results in binary, then loads and processes the logs to be
human readable as it exits).

Cheers,

Richard