From: Richard Purdie <rpurdie@linux.intel.com>
To: Dongxiao Xu <dongxiao.xu@intel.com>
Cc: poky@yoctoproject.org
Subject: Re: [PATCH 1/1] bitbake: optimize file parsing speed
Date: Fri, 19 Nov 2010 10:27:32 +0000 [thread overview]
Message-ID: <1290162452.1272.12090.camel@rex> (raw)
In-Reply-To: <fa09979bfc38e15dc4b9ab199c11302284583a18.1289969353.git.dongxiao.xu@intel.com>
On Wed, 2010-11-17 at 12:10 +0800, Dongxiao Xu wrote:
> build some data cache for generate_dependencies() on hand, and later
> each time when parsing the bb file, we do not need to build them again
> and again.
>
> This optimization could get about 50% speed gain when parsing all ~800
> bb files.
>
> Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> ---
> bitbake/lib/bb/cooker.py | 2 ++
> bitbake/lib/bb/data.py | 11 ++++++-----
> 2 files changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index 33eb65e..05e6c16 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -76,6 +76,8 @@ class BBCooker:
>
> self.configuration.data = bb.data.init()
>
> + bb.data.init_data_cache(self.configuration.data)
> +
> if not server:
> bb.data.setVar("BB_WORKERCONTEXT", "1", self.configuration.data)
>
> diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
> index fee10cc..a9e539f 100644
> --- a/bitbake/lib/bb/data.py
> +++ b/bitbake/lib/bb/data.py
> @@ -296,17 +296,18 @@ def build_dependencies(key, keys, shelldeps, d):
> #bb.note("Variable %s references %s and calls %s" % (key, str(deps), str(execs)))
> #d.setVarFlag(key, "vardeps", deps)
>
> -def generate_dependencies(d):
> +def init_data_cache(d):
> + bb.data.keylist = set(key for key in d.keys() if not key.startswith("__"))
> + bb.data.shelldeps = set(key for key in bb.data.keylist if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))
>
> - keys = set(key for key in d.keys() if not key.startswith("__"))
> - shelldeps = set(key for key in keys if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))
> +def generate_dependencies(d):
>
> deps = {}
> taskdeps = {}
>
> tasklist = bb.data.getVar('__BBTASKS', d) or []
> for task in tasklist:
> - deps[task] = build_dependencies(task, keys, shelldeps, d)
> + deps[task] = build_dependencies(task, bb.data.keylist, bb.data.shelldeps, d)
>
> newdeps = deps[task]
> seen = set()
> @@ -316,7 +317,7 @@ def generate_dependencies(d):
> newdeps = set()
> for dep in nextdeps:
> if dep not in deps:
> - deps[dep] = build_dependencies(dep, keys, shelldeps, d)
> + deps[dep] = build_dependencies(dep, bb.data.keylist, bb.data.shelldeps, d)
> newdeps |= deps[dep]
> newdeps -= seen
> taskdeps[task] = seen | newdeps
I'm afraid this isn't going to be quite this simple although this does
prove those lines of code are a big hotspot in parsing.
Why? You're creating the key and export lists for the base configuration
data whereas the original code creates these lists for the *total*
parsed metadata. There will therefore be differences in the values held
by the two caches :(.
As an example, if you set:
FOO = "bar" in a .bb file, 'FOO' will not appear in your keywords cache.
Cheers,
Richard
next prev parent reply other threads:[~2010-11-19 10:27 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-11-17 4:49 [PATCH 0/1] [PULL] bitbake: optimize the file parsing speed Dongxiao Xu
2010-11-17 4:10 ` [PATCH 1/1] bitbake: optimize " Dongxiao Xu
2010-11-19 10:27 ` Richard Purdie [this message]
2010-11-19 13:14 ` Xu, Dongxiao
2010-11-26 8:07 ` Qing He
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1290162452.1272.12090.camel@rex \
--to=rpurdie@linux.intel.com \
--cc=dongxiao.xu@intel.com \
--cc=poky@yoctoproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox