Poky Project Discussions
 help / color / mirror / Atom feed
From: Dongxiao Xu <dongxiao.xu@intel.com>
To: poky@yoctoproject.org
Subject: [PATCH 0/1] [PULL] bitbake: optimize the file parsing speed
Date: Wed, 17 Nov 2010 12:49:13 +0800	[thread overview]
Message-ID: <cover.1289969353.git.dongxiao.xu@intel.com> (raw)

Hi Richard and Saul,

This patch is to optimize the file parsing speed. Please help to review
and pull. Thanks!

From the profiling result, we found that generate_dependencies()
occupies a lot of time where the following code will be run when
parsing each bb file.

keys = set(key for key in d.keys() if not key.startswith("__"))
shelldeps = set(key for key in keys if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))

To optimize the code, main thought is to build the data cache for
generate_dependencies() on hand, and later each time when parsing the
bb file, we do not need to build them again and again.

Here I used "bitbake -p -P" to measure the file parsing time.

Each time before testing, we will do "touch conf/local.conf" to ensure
that all the 846 files are re-parsed.

Here are the test results:

Before optimization: 
Round 1: 74.980s  (deprecated)
Round 2: 60.281s
Round 3: 59.824s
Round 4: 60.771s
--------------------
Average: 60.292s


After optimization:
Round 1: 45.003s  (deprecated)
Round 2: 33.063s
Round 3: 32.991s
Round 4: 32.043s
--------------------
Average: 32.699s

For both cases, the first time result is a big higher than later, I think
it is due to cold cache, so here I deprecated it.

For the rest three rounds, I calculated the average time, and we can see
the optimization could gain ~50% parsing time.


Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: dxu4/perf
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/perf

Thanks,
    Dongxiao Xu <dongxiao.xu@intel.com>
---


Dongxiao Xu (1):
  bitbake: optimize file parsing speed

 bitbake/lib/bb/cooker.py |    2 ++
 bitbake/lib/bb/data.py   |   11 ++++++-----
 2 files changed, 8 insertions(+), 5 deletions(-)



             reply	other threads:[~2010-11-17  4:47 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-11-17  4:49 Dongxiao Xu [this message]
2010-11-17  4:10 ` [PATCH 1/1] bitbake: optimize file parsing speed Dongxiao Xu
2010-11-19 10:27   ` Richard Purdie
2010-11-19 13:14     ` Xu, Dongxiao
2010-11-26  8:07     ` Qing He

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1289969353.git.dongxiao.xu@intel.com \
    --to=dongxiao.xu@intel.com \
    --cc=poky@yoctoproject.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox