Re: [PATCH 1/1] bitbake: optimize file parsing speed

[Qing He <qing.he@intel.com>]

On Fri, 2010-11-19 at 18:27 +0800, Richard Purdie wrote:
> I'm afraid this isn't going to be quite this simple although this does
> prove those lines of code are a big hotspot in parsing.
> 
> Why? You're creating the key and export lists for the base configuration
> data whereas the original code creates these lists for the *total*
> parsed metadata. There will therefore be differences in the values held
> by the two caches :(.

Is it possible to maintain a list during variable changes? say, tag
several method like setVar, setVarFlag, etc., add something like
d["_conanicalkeys"] and d["_exportkeys"] and when this info is required,
recursively get every "_conanicalkeys" from d and d["_data"].

It has like 13% speed improvement in my experiment, but is quite tricky
and ugly, possibly not worth the trouble.

Btw, looks like much of the performance downgrade of the parsing is due
to moving python/shell code parsing/compilation to parse time? That way,
the performance of parse is OK?

Thanks,
Qing