From: Dongxiao Xu <dongxiao.xu@intel.com>
To: poky@yoctoproject.org
Subject: [PATCH 0/1] [PULL] bitbake: optimize the file parsing speed
Date: Wed, 17 Nov 2010 12:49:13 +0800 [thread overview]
Message-ID: <cover.1289969353.git.dongxiao.xu@intel.com> (raw)
Hi Richard and Saul,
This patch is to optimize the file parsing speed. Please help to review
and pull. Thanks!
From the profiling result, we found that generate_dependencies()
occupies a lot of time where the following code will be run when
parsing each bb file.
keys = set(key for key in d.keys() if not key.startswith("__"))
shelldeps = set(key for key in keys if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))
To optimize the code, main thought is to build the data cache for
generate_dependencies() on hand, and later each time when parsing the
bb file, we do not need to build them again and again.
Here I used "bitbake -p -P" to measure the file parsing time.
Each time before testing, we will do "touch conf/local.conf" to ensure
that all the 846 files are re-parsed.
Here are the test results:
Before optimization:
Round 1: 74.980s (deprecated)
Round 2: 60.281s
Round 3: 59.824s
Round 4: 60.771s
--------------------
Average: 60.292s
After optimization:
Round 1: 45.003s (deprecated)
Round 2: 33.063s
Round 3: 32.991s
Round 4: 32.043s
--------------------
Average: 32.699s
For both cases, the first time result is a big higher than later, I think
it is due to cold cache, so here I deprecated it.
For the rest three rounds, I calculated the average time, and we can see
the optimization could gain ~50% parsing time.
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: dxu4/perf
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/perf
Thanks,
Dongxiao Xu <dongxiao.xu@intel.com>
---
Dongxiao Xu (1):
bitbake: optimize file parsing speed
bitbake/lib/bb/cooker.py | 2 ++
bitbake/lib/bb/data.py | 11 ++++++-----
2 files changed, 8 insertions(+), 5 deletions(-)
next reply other threads:[~2010-11-17 4:47 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-11-17 4:49 Dongxiao Xu [this message]
2010-11-17 4:10 ` [PATCH 1/1] bitbake: optimize file parsing speed Dongxiao Xu
2010-11-19 10:27 ` Richard Purdie
2010-11-19 13:14 ` Xu, Dongxiao
2010-11-26 8:07 ` Qing He
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1289969353.git.dongxiao.xu@intel.com \
--to=dongxiao.xu@intel.com \
--cc=poky@yoctoproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox