* [PATCH 0/6] Improve caching task file dependency checksum caching
@ 2016-01-26 13:34 Markus Lehtonen
2016-01-26 13:34 ` [PATCH 1/6] bb/cache: drop some unused arguments Markus Lehtonen
` (6 more replies)
0 siblings, 7 replies; 9+ messages in thread
From: Markus Lehtonen @ 2016-01-26 13:34 UTC (permalink / raw)
To: bitbake-devel
This patchset improves the caching of task file dependency checksums in two
ways. First, the patchset enables writing of the cache onto disk so that the
checksums can be re-used on subsequent runs of bitbake. Previously the task
signature checksums were calculated and cached in memory but never really
saved.
Second, this patchset makes the task file dependency checksum cache file
configurable. This is intended to be used e.g. with a further patchset for
oe-core that will utilize task file dependencies heavily.
[YOCTO #8853]
Markus Lehtonen (6):
bb/cache: drop some unused arguments
SignatureGenerator: add method for saving the file checksum cache
bb/runqueue: save task file dependency cache onto disk
FileChecksumCache: add get_checksums() method
MultiProcessCache: make cache filename configurable
SignatureGeneratorBasic: make checksum cache file configurable
lib/bb/cache.py | 9 ++++----
lib/bb/checksum.py | 47 ++++++++++++++++++++++++++++++++++++++++
lib/bb/codeparser.py | 8 +++----
lib/bb/cooker.py | 8 +++----
lib/bb/fetch2/__init__.py | 55 ++++++-----------------------------------------
lib/bb/runqueue.py | 1 +
lib/bb/siggen.py | 25 ++++++++++++++++++++-
7 files changed, 91 insertions(+), 62 deletions(-)
--
2.1.4
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 1/6] bb/cache: drop some unused arguments
2016-01-26 13:34 [PATCH 0/6] Improve caching task file dependency checksum caching Markus Lehtonen
@ 2016-01-26 13:34 ` Markus Lehtonen
2016-01-26 13:34 ` [PATCH 2/6] SignatureGenerator: add method for saving the file checksum cache Markus Lehtonen
` (5 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Markus Lehtonen @ 2016-01-26 13:34 UTC (permalink / raw)
To: bitbake-devel
Drop unused 'd' argument from the cache save methods, simplifying the
API.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
lib/bb/cache.py | 4 ++--
lib/bb/codeparser.py | 8 ++++----
lib/bb/cooker.py | 8 ++++----
lib/bb/fetch2/__init__.py | 10 +++++-----
4 files changed, 15 insertions(+), 15 deletions(-)
diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index 8485eb4..55283b0 100644
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -785,7 +785,7 @@ class MultiProcessCache(object):
data = [{}]
return data
- def save_extras(self, d):
+ def save_extras(self):
if not self.cachefile:
return
@@ -815,7 +815,7 @@ class MultiProcessCache(object):
if h not in dest[j]:
dest[j][h] = source[j][h]
- def save_merge(self, d):
+ def save_merge(self):
if not self.cachefile:
return
diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py
index 577b427..bdfa2ca 100644
--- a/lib/bb/codeparser.py
+++ b/lib/bb/codeparser.py
@@ -167,11 +167,11 @@ codeparsercache = CodeParserCache()
def parser_cache_init(d):
codeparsercache.init_cache(d)
-def parser_cache_save(d):
- codeparsercache.save_extras(d)
+def parser_cache_save():
+ codeparsercache.save_extras()
-def parser_cache_savemerge(d):
- codeparsercache.save_merge(d)
+def parser_cache_savemerge():
+ codeparsercache.save_merge()
Logger = logging.getLoggerClass()
class BufferedLogger(Logger):
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 9c58d95..d48724a 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1999,8 +1999,8 @@ class CookerParser(object):
bb.event.fire(bb.event.ParseStarted(self.toparse), self.cfgdata)
def init():
Parser.cfg = self.cfgdata
- multiprocessing.util.Finalize(None, bb.codeparser.parser_cache_save, args=(self.cfgdata,), exitpriority=1)
- multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save, args=(self.cfgdata,), exitpriority=1)
+ multiprocessing.util.Finalize(None, bb.codeparser.parser_cache_save, exitpriority=1)
+ multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save, exitpriority=1)
self.feeder_quit = multiprocessing.Queue(maxsize=1)
self.parser_quit = multiprocessing.Queue(maxsize=self.num_processes)
@@ -2053,8 +2053,8 @@ class CookerParser(object):
sync = threading.Thread(target=self.bb_cache.sync)
sync.start()
multiprocessing.util.Finalize(None, sync.join, exitpriority=-100)
- bb.codeparser.parser_cache_savemerge(self.cooker.data)
- bb.fetch.fetcher_parse_done(self.cooker.data)
+ bb.codeparser.parser_cache_savemerge()
+ bb.fetch.fetcher_parse_done()
if self.cooker.configuration.profile:
profiles = []
for i in self.process_names:
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 5b416ab..7d7bd58 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -515,13 +515,13 @@ def fetcher_init(d):
if hasattr(m, "init"):
m.init(d)
-def fetcher_parse_save(d):
- _checksum_cache.save_extras(d)
+def fetcher_parse_save():
+ _checksum_cache.save_extras()
-def fetcher_parse_done(d):
- _checksum_cache.save_merge(d)
+def fetcher_parse_done():
+ _checksum_cache.save_merge()
-def fetcher_compare_revisions(d):
+def fetcher_compare_revisions():
"""
Compare the revisions in the persistant cache with current values and
return true/false on whether they've changed.
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 2/6] SignatureGenerator: add method for saving the file checksum cache
2016-01-26 13:34 [PATCH 0/6] Improve caching task file dependency checksum caching Markus Lehtonen
2016-01-26 13:34 ` [PATCH 1/6] bb/cache: drop some unused arguments Markus Lehtonen
@ 2016-01-26 13:34 ` Markus Lehtonen
2016-01-26 13:34 ` [PATCH 3/6] bb/runqueue: save task file dependency cache onto disk Markus Lehtonen
` (4 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Markus Lehtonen @ 2016-01-26 13:34 UTC (permalink / raw)
To: bitbake-devel
Extend the API in order to be able to write out the file checksum cache
onto disk. SignatureGeneratorBasic class now implements a method that
update the fetcher local files checksum cache with the task file
dependency checksums.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
lib/bb/siggen.py | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index c104c19..a7916b2 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -44,6 +44,10 @@ class SignatureGenerator(object):
def get_taskhash(self, fn, task, deps, dataCache):
return "0"
+ def writeout_file_checksum_cache(self):
+ """Write/update the file checksum cache onto disk"""
+ return
+
def set_taskdata(self, hashes, deps, checksum):
return
@@ -215,6 +219,11 @@ class SignatureGeneratorBasic(SignatureGenerator):
#d.setVar("BB_TASKHASH_task-%s" % task, taskhash[task])
return h
+ def writeout_file_checksum_cache(self):
+ """Write/update the file checksum cache onto disk"""
+ bb.fetch2.fetcher_parse_save()
+ bb.fetch2.fetcher_parse_done()
+
def dump_sigtask(self, fn, task, stampbase, runtime):
k = fn + "." + task
if runtime == "customfile":
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 3/6] bb/runqueue: save task file dependency cache onto disk
2016-01-26 13:34 [PATCH 0/6] Improve caching task file dependency checksum caching Markus Lehtonen
2016-01-26 13:34 ` [PATCH 1/6] bb/cache: drop some unused arguments Markus Lehtonen
2016-01-26 13:34 ` [PATCH 2/6] SignatureGenerator: add method for saving the file checksum cache Markus Lehtonen
@ 2016-01-26 13:34 ` Markus Lehtonen
2016-01-26 13:34 ` [PATCH 4/6] FileChecksumCache: add get_checksums() method Markus Lehtonen
` (3 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Markus Lehtonen @ 2016-01-26 13:34 UTC (permalink / raw)
To: bitbake-devel
Before this patch the usage of cache was quite useless as the file
checksums were not actually cached on disk but re-calculated every time.
This patch utilises the new writeout_file_checksum_cache() method of the
SignatureGenerator class to do the job.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
lib/bb/runqueue.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index f840ad2..ecbc1ea 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -829,6 +829,7 @@ class RunQueueData:
procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)
+ bb.parse.siggen.writeout_file_checksum_cache()
return len(self.runq_fnid)
def dump_data(self, taskQueue):
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 4/6] FileChecksumCache: add get_checksums() method
2016-01-26 13:34 [PATCH 0/6] Improve caching task file dependency checksum caching Markus Lehtonen
` (2 preceding siblings ...)
2016-01-26 13:34 ` [PATCH 3/6] bb/runqueue: save task file dependency cache onto disk Markus Lehtonen
@ 2016-01-26 13:34 ` Markus Lehtonen
2016-01-26 13:34 ` [PATCH 5/6] MultiProcessCache: make cache filename configurable Markus Lehtonen
` (2 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Markus Lehtonen @ 2016-01-26 13:34 UTC (permalink / raw)
To: bitbake-devel
Move the local file checksum functionality from bb.fetch2 into
bb.checksum module.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
lib/bb/checksum.py | 47 +++++++++++++++++++++++++++++++++++++++++++++++
lib/bb/fetch2/__init__.py | 45 +--------------------------------------------
2 files changed, 48 insertions(+), 44 deletions(-)
diff --git a/lib/bb/checksum.py b/lib/bb/checksum.py
index 514ff0b..7fb46d8 100644
--- a/lib/bb/checksum.py
+++ b/lib/bb/checksum.py
@@ -15,6 +15,8 @@
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+import glob
+import operator
import os
import stat
import bb.utils
@@ -88,3 +90,48 @@ class FileChecksumCache(MultiProcessCache):
dest[0][h] = source[0][h]
else:
dest[0][h] = source[0][h]
+
+ def get_checksums(self, filelist, pn):
+ """Get checksums for a list of files"""
+
+ def checksum_file(f):
+ try:
+ checksum = self.get_checksum(f)
+ except OSError as e:
+ bb.warn("Unable to get checksum for %s SRC_URI entry %s: %s" % (pn, os.path.basename(f), e))
+ return None
+ return checksum
+
+ def checksum_dir(pth):
+ # Handle directories recursively
+ dirchecksums = []
+ for root, dirs, files in os.walk(pth):
+ for name in files:
+ fullpth = os.path.join(root, name)
+ checksum = checksum_file(fullpth)
+ if checksum:
+ dirchecksums.append((fullpth, checksum))
+ return dirchecksums
+
+ checksums = []
+ for pth in filelist.split():
+ exist = pth.split(":")[1]
+ if exist == "False":
+ continue
+ pth = pth.split(":")[0]
+ if '*' in pth:
+ # Handle globs
+ for f in glob.glob(pth):
+ if os.path.isdir(f):
+ checksums.extend(checksum_dir(f))
+ else:
+ checksum = checksum_file(f)
+ checksums.append((f, checksum))
+ elif os.path.isdir(pth):
+ checksums.extend(checksum_dir(pth))
+ else:
+ checksum = checksum_file(pth)
+ checksums.append((pth, checksum))
+
+ checksums.sort(key=operator.itemgetter(1))
+ return checksums
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 7d7bd58..6fca67a 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -29,11 +29,9 @@ from __future__ import absolute_import
from __future__ import print_function
import os, re
import signal
-import glob
import logging
import urllib
import urlparse
-import operator
import bb.persist_data, bb.utils
import bb.checksum
from bb import data
@@ -1118,48 +1116,7 @@ def get_file_checksums(filelist, pn):
it proceeds
"""
-
- def checksum_file(f):
- try:
- checksum = _checksum_cache.get_checksum(f)
- except OSError as e:
- bb.warn("Unable to get checksum for %s SRC_URI entry %s: %s" % (pn, os.path.basename(f), e))
- return None
- return checksum
-
- def checksum_dir(pth):
- # Handle directories recursively
- dirchecksums = []
- for root, dirs, files in os.walk(pth):
- for name in files:
- fullpth = os.path.join(root, name)
- checksum = checksum_file(fullpth)
- if checksum:
- dirchecksums.append((fullpth, checksum))
- return dirchecksums
-
- checksums = []
- for pth in filelist.split():
- exist = pth.split(":")[1]
- if exist == "False":
- continue
- pth = pth.split(":")[0]
- if '*' in pth:
- # Handle globs
- for f in glob.glob(pth):
- if os.path.isdir(f):
- checksums.extend(checksum_dir(f))
- else:
- checksum = checksum_file(f)
- checksums.append((f, checksum))
- elif os.path.isdir(pth):
- checksums.extend(checksum_dir(pth))
- else:
- checksum = checksum_file(pth)
- checksums.append((pth, checksum))
-
- checksums.sort(key=operator.itemgetter(1))
- return checksums
+ return _checksum_cache.get_checksums(filelist, pn)
class FetchData(object):
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 5/6] MultiProcessCache: make cache filename configurable
2016-01-26 13:34 [PATCH 0/6] Improve caching task file dependency checksum caching Markus Lehtonen
` (3 preceding siblings ...)
2016-01-26 13:34 ` [PATCH 4/6] FileChecksumCache: add get_checksums() method Markus Lehtonen
@ 2016-01-26 13:34 ` Markus Lehtonen
2016-01-26 13:34 ` [PATCH 6/6] SignatureGeneratorBasic: make checksum cache file configurable Markus Lehtonen
2016-01-29 17:09 ` [PATCH 0/6] Improve caching task file dependency checksum caching Richard Purdie
6 siblings, 0 replies; 9+ messages in thread
From: Markus Lehtonen @ 2016-01-26 13:34 UTC (permalink / raw)
To: bitbake-devel
If no cache file name is given a default from class variable is used,
like before.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
lib/bb/cache.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index 55283b0..063ab15 100644
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -755,13 +755,14 @@ class MultiProcessCache(object):
self.cachedata = self.create_cachedata()
self.cachedata_extras = self.create_cachedata()
- def init_cache(self, d):
+ def init_cache(self, d, cache_file_name=None):
cachedir = (d.getVar("PERSISTENT_DIR", True) or
d.getVar("CACHE", True))
if cachedir in [None, '']:
return
bb.utils.mkdirhier(cachedir)
- self.cachefile = os.path.join(cachedir, self.__class__.cache_file_name)
+ self.cachefile = os.path.join(cachedir,
+ cache_file_name or self.__class__.cache_file_name)
logger.debug(1, "Using cache in '%s'", self.cachefile)
glf = bb.utils.lockfile(self.cachefile + ".lock")
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH 6/6] SignatureGeneratorBasic: make checksum cache file configurable
2016-01-26 13:34 [PATCH 0/6] Improve caching task file dependency checksum caching Markus Lehtonen
` (4 preceding siblings ...)
2016-01-26 13:34 ` [PATCH 5/6] MultiProcessCache: make cache filename configurable Markus Lehtonen
@ 2016-01-26 13:34 ` Markus Lehtonen
2016-01-29 17:09 ` [PATCH 0/6] Improve caching task file dependency checksum caching Richard Purdie
6 siblings, 0 replies; 9+ messages in thread
From: Markus Lehtonen @ 2016-01-26 13:34 UTC (permalink / raw)
To: bitbake-devel
Define a new bitbake configuration variable BB_HASH_CHECKSUM_CACHE_FILE
that can be used to define the cache file to use for file checksum
cache.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
lib/bb/siggen.py | 20 +++++++++++++++++---
1 file changed, 17 insertions(+), 3 deletions(-)
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index a7916b2..43cf7b6 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -4,6 +4,7 @@ import os
import re
import tempfile
import bb.data
+from bb.checksum import FileChecksumCache
logger = logging.getLogger('BitBake.SigGen')
@@ -91,6 +92,12 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.basewhitelist = set((data.getVar("BB_HASHBASE_WHITELIST", True) or "").split())
self.taskwhitelist = None
self.init_rundepcheck(data)
+ checksum_cache_file = data.getVar("BB_HASH_CHECKSUM_CACHE_FILE", True)
+ if checksum_cache_file:
+ self.checksum_cache = FileChecksumCache()
+ self.checksum_cache.init_cache(data, checksum_cache_file)
+ else:
+ self.checksum_cache = None
def init_rundepcheck(self, data):
self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST", True) or None
@@ -194,7 +201,10 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.runtaskdeps[k].append(dep)
if task in dataCache.file_checksums[fn]:
- checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename)
+ if self.checksum_cache:
+ checksums = self.checksum_cache.get_checksums(dataCache.file_checksums[fn][task], recipename)
+ else:
+ checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename)
for (f,cs) in checksums:
self.file_checksum_values[k][f] = cs
if cs:
@@ -221,8 +231,12 @@ class SignatureGeneratorBasic(SignatureGenerator):
def writeout_file_checksum_cache(self):
"""Write/update the file checksum cache onto disk"""
- bb.fetch2.fetcher_parse_save()
- bb.fetch2.fetcher_parse_done()
+ if self.checksum_cache:
+ self.checksum_cache.save_extras()
+ self.checksum_cache.save_merge()
+ else:
+ bb.fetch2.fetcher_parse_save()
+ bb.fetch2.fetcher_parse_done()
def dump_sigtask(self, fn, task, stampbase, runtime):
k = fn + "." + task
--
2.1.4
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 0/6] Improve caching task file dependency checksum caching
2016-01-26 13:34 [PATCH 0/6] Improve caching task file dependency checksum caching Markus Lehtonen
` (5 preceding siblings ...)
2016-01-26 13:34 ` [PATCH 6/6] SignatureGeneratorBasic: make checksum cache file configurable Markus Lehtonen
@ 2016-01-29 17:09 ` Richard Purdie
2016-02-02 16:23 ` Markus Lehtonen
6 siblings, 1 reply; 9+ messages in thread
From: Richard Purdie @ 2016-01-29 17:09 UTC (permalink / raw)
To: Markus Lehtonen, bitbake-devel
On Tue, 2016-01-26 at 15:34 +0200, Markus Lehtonen wrote:
> This patchset improves the caching of task file dependency checksums in two
> ways. First, the patchset enables writing of the cache onto disk so that the
> checksums can be re-used on subsequent runs of bitbake. Previously the task
> signature checksums were calculated and cached in memory but never really
> saved.
>
> Second, this patchset makes the task file dependency checksum cache file
> configurable. This is intended to be used e.g. with a further patchset for
> oe-core that will utilize task file dependencies heavily.
>
> [YOCTO #8853]
Before I'd take this, I'd like to understand what this code does with a
large codebase, for example a kernel source tree.
I'd really like some performance measurements for the file counts
involved and the time it takes to stat trees like that. What is the
command overhead in that fairly pathological case?
Would we be better off requiring memory resident bitbake and using
inotify watches for example?
Cheers,
Richard
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 0/6] Improve caching task file dependency checksum caching
2016-01-29 17:09 ` [PATCH 0/6] Improve caching task file dependency checksum caching Richard Purdie
@ 2016-02-02 16:23 ` Markus Lehtonen
0 siblings, 0 replies; 9+ messages in thread
From: Markus Lehtonen @ 2016-02-02 16:23 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 6178 bytes --]
Hi Richard,
On Fri, 2016-01-29 at 17:09 +0000, Richard Purdie wrote:
> On Tue, 2016-01-26 at 15:34 +0200, Markus Lehtonen wrote:
> > This patchset improves the caching of task file dependency checksums in two
> > ways. First, the patchset enables writing of the cache onto disk so that the
> > checksums can be re-used on subsequent runs of bitbake. Previously the task
> > signature checksums were calculated and cached in memory but never really
> > saved.
> >
> > Second, this patchset makes the task file dependency checksum cache file
> > configurable. This is intended to be used e.g. with a further patchset for
> > oe-core that will utilize task file dependencies heavily.
> >
> > [YOCTO #8853]
>
> Before I'd take this, I'd like to understand what this code does with a
> large codebase, for example a kernel source tree.
First, this patchset should improve the performance on "normal" builds
(i.e. where my accompaniying externalsrc patch to oe-core is not
enabled). For current oe-core master I get 1211 files that get checked -
currently they are MD5'd every time bitbake gets run. With this patchset
enabled only the mtime of each of the files need to be checked for most
of the time and we get the checksum from the cache.
I think my externalsrc patchset for oe-core is then slightly another
matter:
http://lists.openembedded.org/pipermail/openembedded-core/2016-January/116296.html
This patchset takes heavy use of the task file dependencies and I fully
understand your concern there. I should've provided some data to look
at.
> I'd really like some performance measurements for the file counts
> involved and the time it takes to stat trees like that. What is the
> command overhead in that fairly pathological case?
I did some profiling on my desktop machine (i7-3770K, 16GB of RAM, 2TB
HDD). I basically ran "bitbake core-image-sato" and profiled the end of
bb.runqueue.RunQueueData.prepare() which does the task hashing.
The test cases I used were
- bb master: bitbake master (only profiling code added), oe-core
master
- bb patched: bitbake with my patchset applied, oe-core master
- bb patched + linux-yocto: bitbake with my patchset applied, oe-core
with my other patchset applied and "devtool modify linux-yocto"
enabled where all files of the kernel source are added as deps
First, the results with kernel caches hot which is probably closer to
the usual use case.
# OF FILE CHECKSUM CUMTIME OF
# TESTCASE FILES CACHE STATE get_taskhash
------------------------------------------------------------------------
1 bb master 1211 cold 0.602
2 bb patched 1211 cold 0.599
3 bb patched 1211 warm 0.487
4 bb patched + linux-yocto 53519 cold 15.06
5 bb patched + linux-yocto 53519 warm 1.218
We can see that the improvement in a "normal" build is about 20% (#2 vs.
#3) but in absolute time this is neglible. Tests #1 and #2 are about the
same as they should as all the same files are hashed. In the "kernel
case" the first run takes quite long as the whole kernel source tree
(over 52000 files) is hashed. After this, the hashes are cached on disk
and on subsequent runs the overhead is quite small. The time taken by
get_taskhash drops to about one tenth.
Next, I made some tests with the kernel caches dropped before every test
run which represents certain worst case scenario and should give more
coherent results.
# OF FILE CHECKSUM CUMTIME OF
# TESTCASE FILES CACHE STATE get_taskhash
------------------------------------------------------------------------
6 bb master 1211 cold 2.222
7 bb patched 1211 cold 2.178
8 bb patched 1211 warm 0.576
9 bb patched + linux-yocto 53519 cold 80.039
10 bb patched + linux-yocto 53519 warm 8.496
Here the "kernel case" is quite rude as the whole source tree is read
and hashed from disk. After the on-disk file checksum cache has been
populated the numbers look better. Again, the time taken by get_taskhash
drops to about one tenth.
Loading of the cache file takes some time, too. On my machine loading
(pickle.load()) a "normal" cache with 1211 entries takes about 0.02
seconds. Loading the big linux-yocto polluted cache file (with 53519
entries) takes about 0.4 seconds. So, I think that should be bearable.
Note that all these results are from one run only, and thus, they are
not statistically reliable as such. But, they should give a ballpark
figure. All the cprofile results are found as attachments of this email,
if you want to take a closer look.
With these results I think that the bitbake patchset could be merged.
Normally, you would get slight performance improvement as the cache file
would contain a reasonable amount of entries. As said before, I think
that the "linux-yocto" case is somewhat separate from this patchset: in
that case you will get the same hashing penalty whether you have this
bitbake patchset applied or not. The difference is that with this
patchset applied the hashing penalty drops dramatically after the
checksums have been cached on disk.
> Would we be better off requiring memory resident bitbake and using
> inotify watches for example?
I don't know :) How would you "require" that?
When taking a closer look at the code I realized that the existing
caching code is partly dead. The checksum cache file written to disk
(local_file_checksum_cache.dat) is basically always empty.
FileChecksumCache class is effectively only used by the task hashing
code but the data gets never written onto disk. It doesn't make much
sense reading/writing an empty local_file_checksum_cache.dat every time.
I'll fix this in the next version of this patchset.
Cheers,
Markus
[-- Attachment #2: profile_01.txt --]
[-- Type: text/plain, Size: 5141 bytes --]
827014 function calls (827009 primitive calls) in 0.612 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
4917 0.009 0.000 0.602 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:132(get_taskhash)
4917 0.069 0.000 0.593 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:177(get_taskhash)
416 0.007 0.000 0.206 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/fetch2/__init__.py:1104(get_file_checksums)
23313 0.016 0.000 0.191 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:117(rundep_check)
1211 0.001 0.000 0.188 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/fetch2/__init__.py:1112(checksum_file)
1211 0.003 0.000 0.187 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:69(get_checksum)
1211 0.124 0.000 0.181 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:512(md5_file)
23313 0.054 0.000 0.175 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:3(sstate_rundepfilter)
14820 0.013 0.000 0.061 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:11(isKernel)
4917 0.014 0.000 0.052 0.000 {sorted}
216557 0.052 0.000 0.052 0.000 {method 'update' of '_hashlib.HASH' objects}
29841 0.039 0.000 0.039 0.000 {method 'find' of 'str' objects}
23313 0.020 0.000 0.038 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:316(clean_basepath)
4917 0.011 0.000 0.035 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:168(read_taint)
15732 0.025 0.000 0.033 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:14(isPackageGroup)
6128 0.028 0.000 0.028 0.000 {open}
23313 0.023 0.000 0.023 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
66338 0.019 0.000 0.019 0.000 {method 'rsplit' of 'str' objects}
32340 0.019 0.000 0.019 0.000 {method 'join' of 'str' objects}
16542 0.005 0.000 0.009 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:5(isNative)
39065 0.007 0.000 0.007 0.000 {method 'startswith' of 'str' objects}
6128 0.007 0.000 0.007 0.000 {_hashlib.openssl_md5}
15732 0.004 0.000 0.007 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:9(isNativeSDK)
16623 0.006 0.000 0.006 0.000 {method 'split' of 'str' objects}
105812 0.005 0.000 0.005 0.000 {len}
1214 0.001 0.000 0.005 0.000 /usr/lib64/python2.7/genericpath.py:38(isdir)
2174 0.005 0.000 0.005 0.000 {posix.stat}
46451 0.005 0.000 0.005 0.000 {method 'append' of 'list' objects}
23313 0.005 0.000 0.005 0.000 {method 'group' of '_sre.SRE_Match' objects}
16562 0.004 0.000 0.004 0.000 {method 'endswith' of 'str' objects}
6128 0.004 0.000 0.004 0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
1211 0.001 0.000 0.003 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:39(cached_mtime)
15760 0.002 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:7(isCross)
651 0.001 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:20(isImage)
55 0.002 0.000 0.002 0.000 {method 'copy' of 'set' objects}
936 0.001 0.000 0.001 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:17(isAllArch)
1214 0.000 0.000 0.001 0.000 /usr/lib64/python2.7/stat.py:40(S_ISDIR)
4917 0.001 0.000 0.001 0.000 {method 'add' of 'set' objects}
4917 0.001 0.000 0.001 0.000 {method 'remove' of 'set' objects}
416 0.000 0.000 0.000 0.000 {method 'sort' of 'list' objects}
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/fetch2/__init__.py:1120(checksum_dir)
1211 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
1216 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/stat.py:24(S_IFMT)
9/4 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/os.py:209(walk)
20 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:68(join)
3 0.000 0.000 0.000 0.000 {posix.listdir}
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:139(islink)
2 0.000 0.000 0.000 0.000 {posix.lstat}
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/stat.py:55(S_ISLNK)
1 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
[-- Attachment #3: profile_02.txt --]
[-- Type: text/plain, Size: 7432 bytes --]
827145 function calls (827140 primitive calls) in 0.613 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
4917 0.008 0.000 0.599 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:132(get_taskhash)
4917 0.068 0.000 0.590 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:188(get_taskhash)
416 0.007 0.000 0.203 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:98(get_checksums)
23313 0.016 0.000 0.192 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:117(rundep_check)
1211 0.001 0.000 0.185 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:101(checksum_file)
1211 0.003 0.000 0.184 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:71(get_checksum)
1211 0.122 0.000 0.179 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:512(md5_file)
23313 0.055 0.000 0.176 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:3(sstate_rundepfilter)
14820 0.014 0.000 0.061 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:11(isKernel)
216557 0.051 0.000 0.051 0.000 {method 'update' of '_hashlib.HASH' objects}
4917 0.014 0.000 0.051 0.000 {sorted}
29841 0.038 0.000 0.038 0.000 {method 'find' of 'str' objects}
23313 0.020 0.000 0.037 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:342(clean_basepath)
4917 0.012 0.000 0.036 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:179(read_taint)
15732 0.025 0.000 0.033 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:14(isPackageGroup)
6134 0.029 0.000 0.029 0.000 {open}
23313 0.024 0.000 0.024 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
66338 0.019 0.000 0.019 0.000 {method 'rsplit' of 'str' objects}
32340 0.019 0.000 0.019 0.000 {method 'join' of 'str' objects}
16542 0.005 0.000 0.009 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:5(isNative)
15732 0.004 0.000 0.007 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:9(isNativeSDK)
6128 0.007 0.000 0.007 0.000 {_hashlib.openssl_md5}
39071 0.007 0.000 0.007 0.000 {method 'startswith' of 'str' objects}
16623 0.006 0.000 0.006 0.000 {method 'split' of 'str' objects}
105820 0.005 0.000 0.005 0.000 {len}
1214 0.001 0.000 0.005 0.000 /usr/lib64/python2.7/genericpath.py:38(isdir)
2184 0.005 0.000 0.005 0.000 {posix.stat}
23313 0.005 0.000 0.005 0.000 {method 'group' of '_sre.SRE_Match' objects}
46451 0.004 0.000 0.004 0.000 {method 'append' of 'list' objects}
16563 0.004 0.000 0.004 0.000 {method 'endswith' of 'str' objects}
1 0.000 0.000 0.004 0.004 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:238(writeout_file_checksum_cache)
6128 0.004 0.000 0.004 0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
1211 0.001 0.000 0.003 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:41(cached_mtime)
15760 0.002 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:7(isCross)
1 0.000 0.000 0.002 0.002 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:819(save_merge)
2 0.002 0.001 0.002 0.001 {method 'dump' of 'cPickle.Pickler' objects}
651 0.001 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:20(isImage)
55 0.002 0.000 0.002 0.000 {method 'copy' of 'set' objects}
936 0.001 0.000 0.001 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:17(isAllArch)
1 0.000 0.000 0.001 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:789(save_extras)
1 0.001 0.001 0.001 0.001 {built-in method load}
1214 0.000 0.000 0.001 0.000 /usr/lib64/python2.7/stat.py:40(S_ISDIR)
4917 0.001 0.000 0.001 0.000 {method 'add' of 'set' objects}
4917 0.001 0.000 0.001 0.000 {method 'remove' of 'set' objects}
416 0.000 0.000 0.000 0.000 {method 'sort' of 'list' objects}
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:109(checksum_dir)
1211 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
1216 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/stat.py:24(S_IFMT)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:443(lockfile)
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:88(merge_data)
9/4 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/os.py:209(walk)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:498(unlockfile)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:699(mkdirhier)
4 0.000 0.000 0.000 0.000 {posix.unlink}
4 0.000 0.000 0.000 0.000 {posix.listdir}
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/os.py:136(makedirs)
21 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:68(join)
7 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/genericpath.py:15(exists)
5 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:127(dirname)
3 0.000 0.000 0.000 0.000 {method 'close' of 'file' objects}
9 0.000 0.000 0.000 0.000 {fcntl.flock}
3 0.000 0.000 0.000 0.000 {posix.mkdir}
2 0.000 0.000 0.000 0.000 {cPickle.Pickler}
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:89(split)
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:139(islink)
3 0.000 0.000 0.000 0.000 {posix.access}
8 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
2 0.000 0.000 0.000 0.000 {posix.lstat}
13 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
9 0.000 0.000 0.000 0.000 {method 'fileno' of 'file' objects}
3 0.000 0.000 0.000 0.000 {posix.fstat}
5 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:119(basename)
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/stat.py:55(S_ISLNK)
1 0.000 0.000 0.000 0.000 {cPickle.Unpickler}
1 0.000 0.000 0.000 0.000 {posix.getpid}
1 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
[-- Attachment #4: profile_03.txt --]
[-- Type: text/plain, Size: 7219 bytes --]
605747 function calls (605742 primitive calls) in 0.498 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
4917 0.008 0.000 0.487 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:132(get_taskhash)
4917 0.069 0.000 0.479 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:188(get_taskhash)
23313 0.016 0.000 0.262 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:117(rundep_check)
23313 0.061 0.000 0.246 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:3(sstate_rundepfilter)
66338 0.085 0.000 0.085 0.000 {method 'rsplit' of 'str' objects}
14820 0.013 0.000 0.060 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:11(isKernel)
4917 0.014 0.000 0.052 0.000 {sorted}
23313 0.020 0.000 0.039 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:342(clean_basepath)
29841 0.038 0.000 0.038 0.000 {method 'find' of 'str' objects}
4917 0.011 0.000 0.035 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:179(read_taint)
15732 0.024 0.000 0.033 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:14(isPackageGroup)
4923 0.024 0.000 0.024 0.000 {open}
23313 0.023 0.000 0.023 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
416 0.006 0.000 0.021 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:98(get_checksums)
32340 0.018 0.000 0.018 0.000 {method 'join' of 'str' objects}
16542 0.005 0.000 0.009 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:5(isNative)
15732 0.004 0.000 0.007 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:9(isNativeSDK)
39072 0.007 0.000 0.007 0.000 {method 'startswith' of 'str' objects}
4917 0.006 0.000 0.006 0.000 {_hashlib.openssl_md5}
16623 0.006 0.000 0.006 0.000 {method 'split' of 'str' objects}
105820 0.005 0.000 0.005 0.000 {len}
1211 0.000 0.000 0.005 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:101(checksum_file)
23313 0.005 0.000 0.005 0.000 {method 'group' of '_sre.SRE_Match' objects}
46451 0.004 0.000 0.004 0.000 {method 'append' of 'list' objects}
1211 0.001 0.000 0.004 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:71(get_checksum)
16563 0.004 0.000 0.004 0.000 {method 'endswith' of 'str' objects}
2184 0.004 0.000 0.004 0.000 {posix.stat}
1214 0.001 0.000 0.004 0.000 /usr/lib64/python2.7/genericpath.py:38(isdir)
4917 0.003 0.000 0.003 0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
1211 0.001 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:41(cached_mtime)
15760 0.002 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:7(isCross)
651 0.001 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:20(isImage)
55 0.002 0.000 0.002 0.000 {method 'copy' of 'set' objects}
1 0.000 0.000 0.001 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:238(writeout_file_checksum_cache)
936 0.001 0.000 0.001 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:17(isAllArch)
1 0.000 0.000 0.001 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:819(save_merge)
2 0.001 0.000 0.001 0.000 {method 'dump' of 'cPickle.Pickler' objects}
4917 0.001 0.000 0.001 0.000 {method 'add' of 'set' objects}
1214 0.000 0.000 0.001 0.000 /usr/lib64/python2.7/stat.py:40(S_ISDIR)
4917 0.001 0.000 0.001 0.000 {method 'remove' of 'set' objects}
1211 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects}
416 0.000 0.000 0.000 0.000 {method 'sort' of 'list' objects}
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:789(save_extras)
1216 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/stat.py:24(S_IFMT)
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:109(checksum_dir)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:443(lockfile)
9/4 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/os.py:209(walk)
4 0.000 0.000 0.000 0.000 {posix.listdir}
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:498(unlockfile)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:699(mkdirhier)
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/os.py:136(makedirs)
7 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/genericpath.py:15(exists)
4 0.000 0.000 0.000 0.000 {posix.unlink}
21 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:68(join)
5 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:127(dirname)
3 0.000 0.000 0.000 0.000 {method 'close' of 'file' objects}
9 0.000 0.000 0.000 0.000 {fcntl.flock}
2 0.000 0.000 0.000 0.000 {cPickle.Pickler}
1 0.000 0.000 0.000 0.000 {built-in method load}
3 0.000 0.000 0.000 0.000 {posix.mkdir}
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:139(islink)
3 0.000 0.000 0.000 0.000 {posix.access}
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:89(split)
6 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:119(basename)
2 0.000 0.000 0.000 0.000 {posix.lstat}
8 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
3 0.000 0.000 0.000 0.000 {posix.fstat}
14 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/stat.py:55(S_ISLNK)
1 0.000 0.000 0.000 0.000 {cPickle.Unpickler}
1 0.000 0.000 0.000 0.000 {posix.getpid}
1 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects}
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:88(merge_data)
9 0.000 0.000 0.000 0.000 {method 'fileno' of 'file' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
[-- Attachment #5: profile_04.txt --]
[-- Type: text/plain, Size: 12185 bytes --]
26278622 function calls (26266186 primitive calls) in 15.331 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
4911 0.014 0.000 15.060 0.003 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:132(get_taskhash)
4911 0.094 0.000 15.046 0.003 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:188(get_taskhash)
417 0.008 0.000 14.622 0.035 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:98(get_checksums)
23 0.088 0.004 14.532 0.632 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:109(checksum_dir)
53519 0.032 0.000 13.886 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:101(checksum_file)
53519 0.216 0.000 13.854 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:71(get_checksum)
52308 5.352 0.000 10.975 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:512(md5_file)
20893617 5.259 0.000 5.259 0.000 {method 'update' of '_hashlib.HASH' objects}
52308 0.070 0.000 2.353 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/__init__.py:84(debug)
52308 0.052 0.000 2.244 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/__init__.py:57(bbdebug)
52308 0.094 0.000 2.192 0.000 /usr/lib64/python2.7/logging/__init__.py:1209(log)
52308 0.075 0.000 2.046 0.000 /usr/lib64/python2.7/logging/__init__.py:1260(_log)
52308 0.043 0.000 0.969 0.000 /usr/lib64/python2.7/logging/__init__.py:1281(handle)
52308 0.093 0.000 0.906 0.000 /usr/lib64/python2.7/logging/__init__.py:1313(callHandlers)
52308 0.150 0.000 0.859 0.000 /usr/lib64/python2.7/logging/__init__.py:1247(makeRecord)
104616 0.142 0.000 0.813 0.000 /usr/lib64/python2.7/logging/__init__.py:744(handle)
52308 0.411 0.000 0.709 0.000 /usr/lib64/python2.7/logging/__init__.py:243(__init__)
15955/3519 0.069 0.000 0.467 0.000 /usr/lib64/python2.7/os.py:209(walk)
110299 0.357 0.000 0.357 0.000 {posix.stat}
57225 0.319 0.000 0.319 0.000 {open}
53519 0.078 0.000 0.276 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:41(cached_mtime)
1 0.022 0.022 0.261 0.261 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:238(writeout_file_checksum_cache)
104616 0.054 0.000 0.255 0.000 /usr/lib64/python2.7/logging/__init__.py:701(acquire)
57021 0.038 0.000 0.225 0.000 /usr/lib64/python2.7/genericpath.py:38(isdir)
104616 0.130 0.000 0.202 0.000 /usr/lib64/python2.7/threading.py:146(acquire)
104616 0.045 0.000 0.199 0.000 /usr/lib64/python2.7/logging/__init__.py:708(release)
23297 0.016 0.000 0.192 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:117(rundep_check)
52308 0.025 0.000 0.181 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/event.py:602(emit)
111605 0.120 0.000 0.177 0.000 /usr/lib64/python2.7/posixpath.py:68(join)
23297 0.056 0.000 0.176 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:3(sstate_rundepfilter)
1 0.004 0.004 0.160 0.160 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:819(save_merge)
52308 0.035 0.000 0.157 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/event.py:163(fire)
104616 0.118 0.000 0.154 0.000 /usr/lib64/python2.7/threading.py:186(release)
52308 0.093 0.000 0.144 0.000 /usr/lib64/python2.7/logging/__init__.py:1226(findCaller)
2 0.143 0.072 0.143 0.072 {method 'dump' of 'cPickle.Pickler' objects}
52308 0.026 0.000 0.099 0.000 /usr/lib64/python2.7/posixpath.py:104(splitext)
52308 0.057 0.000 0.093 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/event.py:139(fire_ui_handlers)
3498 0.087 0.000 0.087 0.000 {posix.listdir}
1 0.000 0.000 0.078 0.078 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:789(save_extras)
52308 0.055 0.000 0.073 0.000 /usr/lib64/python2.7/genericpath.py:85(_splitext)
1 0.066 0.066 0.066 0.066 {built-in method load}
14819 0.013 0.000 0.061 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:11(isKernel)
4911 0.014 0.000 0.052 0.000 {sorted}
57219 0.050 0.000 0.050 0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
158135 0.049 0.000 0.049 0.000 {method 'get' of 'dict' objects}
52314 0.030 0.000 0.049 0.000 /usr/lib64/python2.7/posixpath.py:119(basename)
52308 0.030 0.000 0.047 0.000 /usr/lib64/python2.7/logging/__init__.py:1353(isEnabledFor)
417 0.045 0.000 0.045 0.000 {method 'sort' of 'list' objects}
209233 0.045 0.000 0.045 0.000 {isinstance}
52308 0.037 0.000 0.045 0.000 /usr/lib64/python2.7/logging/__init__.py:154(getLevelName)
104616 0.045 0.000 0.045 0.000 {method 'acquire' of 'thread.lock' objects}
150639 0.044 0.000 0.044 0.000 {method 'startswith' of 'str' objects}
4911 0.012 0.000 0.043 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:179(read_taint)
29839 0.038 0.000 0.038 0.000 {method 'find' of 'str' objects}
23297 0.020 0.000 0.038 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:342(clean_basepath)
57219 0.037 0.000 0.037 0.000 {_hashlib.openssl_md5}
52308 0.023 0.000 0.037 0.000 /usr/lib64/python2.7/logging/__init__.py:82(<lambda>)
156939 0.036 0.000 0.036 0.000 {method 'rfind' of 'str' objects}
52308 0.032 0.000 0.036 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/event.py:257(filter)
104616 0.036 0.000 0.036 0.000 /usr/lib64/python2.7/logging/__init__.py:605(filter)
209232 0.034 0.000 0.034 0.000 /usr/lib64/python2.7/threading.py:63(_note)
15731 0.025 0.000 0.033 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:14(isPackageGroup)
52308 0.026 0.000 0.029 0.000 /usr/lib64/python2.7/threading.py:1152(currentThread)
52308 0.022 0.000 0.028 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/event.py:99(fire_class_handlers)
313848 0.028 0.000 0.028 0.000 {thread.get_ident}
84646 0.028 0.000 0.028 0.000 {method 'join' of 'str' objects}
57021 0.018 0.000 0.027 0.000 /usr/lib64/python2.7/stat.py:40(S_ISDIR)
128146 0.024 0.000 0.024 0.000 {method 'endswith' of 'str' objects}
23300 0.024 0.000 0.024 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
1 0.023 0.023 0.023 0.023 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:88(merge_data)
3479 0.004 0.000 0.020 0.000 /usr/lib64/python2.7/posixpath.py:139(islink)
66305 0.018 0.000 0.018 0.000 {method 'rsplit' of 'str' objects}
52308 0.017 0.000 0.017 0.000 /usr/lib64/python2.7/logging/__init__.py:1339(getEffectiveLevel)
104616 0.016 0.000 0.016 0.000 {hasattr}
154535 0.016 0.000 0.016 0.000 {method 'append' of 'list' objects}
52308 0.014 0.000 0.014 0.000 /usr/lib64/python2.7/threading.py:967(name)
52308 0.014 0.000 0.014 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/event.py:613(filter)
52308 0.014 0.000 0.014 0.000 {sys._getframe}
52308 0.013 0.000 0.013 0.000 /usr/lib64/python2.7/multiprocessing/process.py:161(name)
3479 0.013 0.000 0.013 0.000 {posix.lstat}
104616 0.013 0.000 0.013 0.000 {method 'release' of 'thread.lock' objects}
60500 0.010 0.000 0.010 0.000 /usr/lib64/python2.7/stat.py:24(S_IFMT)
52308 0.010 0.000 0.010 0.000 /usr/lib64/python2.7/multiprocessing/process.py:59(current_process)
52308 0.010 0.000 0.010 0.000 {time.time}
16541 0.005 0.000 0.009 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:5(isNative)
15731 0.004 0.000 0.007 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:9(isNativeSDK)
52309 0.006 0.000 0.006 0.000 /usr/lib64/python2.7/posixpath.py:51(normcase)
52308 0.006 0.000 0.006 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/__init__.py:47(emit)
52309 0.005 0.000 0.005 0.000 {posix.getpid}
16626 0.005 0.000 0.005 0.000 {method 'split' of 'str' objects}
105673 0.005 0.000 0.005 0.000 {len}
23297 0.005 0.000 0.005 0.000 {method 'group' of '_sre.SRE_Match' objects}
3479 0.002 0.000 0.003 0.000 /usr/lib64/python2.7/stat.py:55(S_ISLNK)
15759 0.002 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:7(isCross)
651 0.001 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:20(isImage)
55 0.002 0.000 0.002 0.000 {method 'copy' of 'set' objects}
936 0.001 0.000 0.001 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:17(isAllArch)
4 0.001 0.000 0.001 0.000 {posix.unlink}
4911 0.001 0.000 0.001 0.000 {method 'add' of 'set' objects}
23 0.001 0.000 0.001 0.000 {method 'extend' of 'list' objects}
4911 0.001 0.000 0.001 0.000 {method 'remove' of 'set' objects}
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:443(lockfile)
1 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:18(glob)
31 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:29(iglob)
1 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:66(glob1)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:498(unlockfile)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:699(mkdirhier)
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/os.py:136(makedirs)
7 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/genericpath.py:15(exists)
1 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/fnmatch.py:45(filter)
5 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:127(dirname)
9 0.000 0.000 0.000 0.000 {fcntl.flock}
3 0.000 0.000 0.000 0.000 {method 'close' of 'file' objects}
4 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:89(split)
3 0.000 0.000 0.000 0.000 {posix.mkdir}
2 0.000 0.000 0.000 0.000 {cPickle.Pickler}
1 0.000 0.000 0.000 0.000 {filter}
3 0.000 0.000 0.000 0.000 {posix.access}
30 0.000 0.000 0.000 0.000 {method 'match' of '_sre.SRE_Pattern' objects}
38 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:77(<lambda>)
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:94(has_magic)
9 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
3 0.000 0.000 0.000 0.000 {posix.fstat}
9 0.000 0.000 0.000 0.000 {method 'fileno' of 'file' objects}
1 0.000 0.000 0.000 0.000 {cPickle.Unpickler}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
[-- Attachment #6: profile_05.txt --]
[-- Type: text/plain, Size: 8042 bytes --]
1566521 function calls (1554085 primitive calls) in 1.286 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
4911 0.013 0.000 1.218 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:132(get_taskhash)
4911 0.089 0.000 1.204 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:188(get_taskhash)
417 0.007 0.000 0.801 0.002 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:98(get_checksums)
23 0.047 0.002 0.736 0.032 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:109(checksum_dir)
15955/3519 0.056 0.000 0.325 0.000 /usr/lib64/python2.7/os.py:209(walk)
53519 0.018 0.000 0.238 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:101(checksum_file)
53519 0.055 0.000 0.220 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:71(get_checksum)
110299 0.192 0.000 0.192 0.000 {posix.stat}
111605 0.151 0.000 0.191 0.000 /usr/lib64/python2.7/posixpath.py:68(join)
23297 0.015 0.000 0.188 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:117(rundep_check)
23297 0.054 0.000 0.173 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:3(sstate_rundepfilter)
57021 0.032 0.000 0.156 0.000 /usr/lib64/python2.7/genericpath.py:38(isdir)
53519 0.048 0.000 0.139 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:41(cached_mtime)
14819 0.014 0.000 0.061 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:11(isKernel)
1 0.006 0.006 0.059 0.059 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:238(writeout_file_checksum_cache)
1 0.001 0.001 0.053 0.053 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:819(save_merge)
4911 0.013 0.000 0.051 0.000 {sorted}
2 0.051 0.025 0.051 0.025 {method 'dump' of 'cPickle.Pickler' objects}
417 0.043 0.000 0.043 0.000 {method 'sort' of 'list' objects}
3498 0.040 0.000 0.040 0.000 {posix.listdir}
29839 0.038 0.000 0.038 0.000 {method 'find' of 'str' objects}
23297 0.020 0.000 0.038 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:342(clean_basepath)
4911 0.011 0.000 0.035 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:179(read_taint)
15731 0.024 0.000 0.033 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:14(isPackageGroup)
150639 0.030 0.000 0.030 0.000 {method 'startswith' of 'str' objects}
53519 0.026 0.000 0.026 0.000 {method 'get' of 'dict' objects}
4917 0.025 0.000 0.025 0.000 {open}
57021 0.015 0.000 0.024 0.000 /usr/lib64/python2.7/stat.py:40(S_ISDIR)
23300 0.022 0.000 0.022 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
128146 0.020 0.000 0.020 0.000 {method 'endswith' of 'str' objects}
66305 0.018 0.000 0.018 0.000 {method 'rsplit' of 'str' objects}
32338 0.018 0.000 0.018 0.000 {method 'join' of 'str' objects}
154535 0.013 0.000 0.013 0.000 {method 'append' of 'list' objects}
3479 0.003 0.000 0.011 0.000 /usr/lib64/python2.7/posixpath.py:139(islink)
60500 0.009 0.000 0.009 0.000 /usr/lib64/python2.7/stat.py:24(S_IFMT)
16541 0.005 0.000 0.009 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:5(isNative)
4911 0.009 0.000 0.009 0.000 {_hashlib.openssl_md5}
15731 0.004 0.000 0.007 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:9(isNativeSDK)
3479 0.006 0.000 0.006 0.000 {posix.lstat}
16626 0.005 0.000 0.005 0.000 {method 'split' of 'str' objects}
105673 0.005 0.000 0.005 0.000 {len}
23297 0.005 0.000 0.005 0.000 {method 'group' of '_sre.SRE_Match' objects}
4911 0.003 0.000 0.003 0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
15759 0.002 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:7(isCross)
3479 0.001 0.000 0.002 0.000 /usr/lib64/python2.7/stat.py:55(S_ISLNK)
651 0.001 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:20(isImage)
55 0.002 0.000 0.002 0.000 {method 'copy' of 'set' objects}
936 0.001 0.000 0.001 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:17(isAllArch)
4911 0.001 0.000 0.001 0.000 {method 'add' of 'set' objects}
4911 0.001 0.000 0.001 0.000 {method 'remove' of 'set' objects}
23 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects}
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:789(save_extras)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:443(lockfile)
1 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:18(glob)
31 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:29(iglob)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:498(unlockfile)
1 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:66(glob1)
4 0.000 0.000 0.000 0.000 {posix.unlink}
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:699(mkdirhier)
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/os.py:136(makedirs)
7 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/genericpath.py:15(exists)
1 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/fnmatch.py:45(filter)
3 0.000 0.000 0.000 0.000 {method 'close' of 'file' objects}
5 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:127(dirname)
9 0.000 0.000 0.000 0.000 {fcntl.flock}
1 0.000 0.000 0.000 0.000 {filter}
4 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:89(split)
2 0.000 0.000 0.000 0.000 {cPickle.Pickler}
30 0.000 0.000 0.000 0.000 {method 'match' of '_sre.SRE_Pattern' objects}
1 0.000 0.000 0.000 0.000 {built-in method load}
3 0.000 0.000 0.000 0.000 {posix.mkdir}
3 0.000 0.000 0.000 0.000 {posix.access}
38 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:77(<lambda>)
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:94(has_magic)
6 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:119(basename)
15 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
3 0.000 0.000 0.000 0.000 {posix.fstat}
9 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
1 0.000 0.000 0.000 0.000 {cPickle.Unpickler}
1 0.000 0.000 0.000 0.000 {isinstance}
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:88(merge_data)
1 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:51(normcase)
9 0.000 0.000 0.000 0.000 {method 'fileno' of 'file' objects}
1 0.000 0.000 0.000 0.000 {posix.getpid}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
[-- Attachment #7: profile_06.txt --]
[-- Type: text/plain, Size: 5141 bytes --]
827014 function calls (827009 primitive calls) in 2.235 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
4917 0.012 0.000 2.222 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:132(get_taskhash)
4917 0.084 0.000 2.210 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:177(get_taskhash)
416 0.018 0.000 1.749 0.004 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/fetch2/__init__.py:1104(get_file_checksums)
1211 0.002 0.000 1.690 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/fetch2/__init__.py:1112(checksum_file)
1211 0.008 0.000 1.688 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:69(get_checksum)
1211 1.534 0.001 1.672 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:512(md5_file)
23313 0.018 0.000 0.196 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:117(rundep_check)
23313 0.057 0.000 0.178 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:3(sstate_rundepfilter)
216557 0.121 0.000 0.121 0.000 {method 'update' of '_hashlib.HASH' objects}
4917 0.016 0.000 0.075 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:168(read_taint)
6128 0.071 0.000 0.071 0.000 {open}
14820 0.014 0.000 0.062 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:11(isKernel)
4917 0.017 0.000 0.060 0.000 {sorted}
23313 0.023 0.000 0.043 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:316(clean_basepath)
29841 0.039 0.000 0.039 0.000 {method 'find' of 'str' objects}
15732 0.025 0.000 0.034 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:14(isPackageGroup)
23313 0.026 0.000 0.026 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
66338 0.021 0.000 0.021 0.000 {method 'rsplit' of 'str' objects}
32340 0.018 0.000 0.018 0.000 {method 'join' of 'str' objects}
1214 0.003 0.000 0.016 0.000 /usr/lib64/python2.7/genericpath.py:38(isdir)
2174 0.015 0.000 0.015 0.000 {posix.stat}
1 0.000 0.000 0.014 0.014 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/fetch2/__init__.py:1120(checksum_dir)
16623 0.013 0.000 0.013 0.000 {method 'split' of 'str' objects}
9/4 0.000 0.000 0.011 0.003 /usr/lib64/python2.7/os.py:209(walk)
3 0.010 0.003 0.010 0.003 {posix.listdir}
6128 0.010 0.000 0.010 0.000 {_hashlib.openssl_md5}
16542 0.005 0.000 0.009 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:5(isNative)
39065 0.007 0.000 0.007 0.000 {method 'startswith' of 'str' objects}
1211 0.003 0.000 0.007 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:39(cached_mtime)
6128 0.007 0.000 0.007 0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
105812 0.007 0.000 0.007 0.000 {len}
15732 0.004 0.000 0.007 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:9(isNativeSDK)
23313 0.005 0.000 0.005 0.000 {method 'group' of '_sre.SRE_Match' objects}
46451 0.005 0.000 0.005 0.000 {method 'append' of 'list' objects}
16562 0.004 0.000 0.004 0.000 {method 'endswith' of 'str' objects}
15760 0.002 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:7(isCross)
55 0.002 0.000 0.002 0.000 {method 'copy' of 'set' objects}
1214 0.001 0.000 0.002 0.000 /usr/lib64/python2.7/stat.py:40(S_ISDIR)
651 0.001 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:20(isImage)
936 0.001 0.000 0.001 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:17(isAllArch)
416 0.001 0.000 0.001 0.000 {method 'sort' of 'list' objects}
4917 0.001 0.000 0.001 0.000 {method 'add' of 'set' objects}
4917 0.001 0.000 0.001 0.000 {method 'remove' of 'set' objects}
1211 0.001 0.000 0.001 0.000 {method 'get' of 'dict' objects}
1216 0.001 0.000 0.001 0.000 /usr/lib64/python2.7/stat.py:24(S_IFMT)
20 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:68(join)
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:139(islink)
2 0.000 0.000 0.000 0.000 {posix.lstat}
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/stat.py:55(S_ISLNK)
1 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
[-- Attachment #8: profile_07.txt --]
[-- Type: text/plain, Size: 7432 bytes --]
827142 function calls (827137 primitive calls) in 2.194 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
4917 0.012 0.000 2.178 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:132(get_taskhash)
4917 0.084 0.000 2.166 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:188(get_taskhash)
416 0.019 0.000 1.644 0.004 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:98(get_checksums)
1211 0.002 0.000 1.585 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:101(checksum_file)
1211 0.008 0.000 1.583 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:71(get_checksum)
1211 1.423 0.001 1.568 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:512(md5_file)
23313 0.018 0.000 0.262 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:117(rundep_check)
23313 0.124 0.000 0.245 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:3(sstate_rundepfilter)
216557 0.128 0.000 0.128 0.000 {method 'update' of '_hashlib.HASH' objects}
4917 0.016 0.000 0.072 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:179(read_taint)
6134 0.068 0.000 0.068 0.000 {open}
14820 0.013 0.000 0.062 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:11(isKernel)
4917 0.017 0.000 0.058 0.000 {sorted}
23313 0.022 0.000 0.042 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:342(clean_basepath)
29841 0.039 0.000 0.039 0.000 {method 'find' of 'str' objects}
15732 0.025 0.000 0.034 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:14(isPackageGroup)
23313 0.025 0.000 0.025 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
66338 0.020 0.000 0.020 0.000 {method 'rsplit' of 'str' objects}
32340 0.019 0.000 0.019 0.000 {method 'join' of 'str' objects}
1214 0.003 0.000 0.015 0.000 /usr/lib64/python2.7/genericpath.py:38(isdir)
2184 0.014 0.000 0.014 0.000 {posix.stat}
1 0.000 0.000 0.014 0.014 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:109(checksum_dir)
16623 0.013 0.000 0.013 0.000 {method 'split' of 'str' objects}
9/4 0.000 0.000 0.011 0.003 /usr/lib64/python2.7/os.py:209(walk)
4 0.010 0.003 0.010 0.003 {posix.listdir}
6128 0.009 0.000 0.009 0.000 {_hashlib.openssl_md5}
16542 0.005 0.000 0.009 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:5(isNative)
6128 0.007 0.000 0.007 0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
39070 0.007 0.000 0.007 0.000 {method 'startswith' of 'str' objects}
105820 0.007 0.000 0.007 0.000 {len}
1211 0.003 0.000 0.007 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:41(cached_mtime)
15732 0.004 0.000 0.007 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:9(isNativeSDK)
46451 0.005 0.000 0.005 0.000 {method 'append' of 'list' objects}
23313 0.005 0.000 0.005 0.000 {method 'group' of '_sre.SRE_Match' objects}
16563 0.004 0.000 0.004 0.000 {method 'endswith' of 'str' objects}
1 0.000 0.000 0.004 0.004 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:238(writeout_file_checksum_cache)
15760 0.002 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:7(isCross)
1 0.000 0.000 0.002 0.002 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:819(save_merge)
2 0.002 0.001 0.002 0.001 {method 'dump' of 'cPickle.Pickler' objects}
55 0.002 0.000 0.002 0.000 {method 'copy' of 'set' objects}
1214 0.001 0.000 0.002 0.000 /usr/lib64/python2.7/stat.py:40(S_ISDIR)
651 0.001 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:20(isImage)
1 0.000 0.000 0.001 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:789(save_extras)
936 0.001 0.000 0.001 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:17(isAllArch)
416 0.001 0.000 0.001 0.000 {method 'sort' of 'list' objects}
4917 0.001 0.000 0.001 0.000 {method 'add' of 'set' objects}
1 0.001 0.001 0.001 0.001 {built-in method load}
4917 0.001 0.000 0.001 0.000 {method 'remove' of 'set' objects}
1216 0.001 0.000 0.001 0.000 /usr/lib64/python2.7/stat.py:24(S_IFMT)
1211 0.001 0.000 0.001 0.000 {method 'get' of 'dict' objects}
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:443(lockfile)
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:88(merge_data)
21 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:68(join)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:699(mkdirhier)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:498(unlockfile)
4 0.000 0.000 0.000 0.000 {posix.unlink}
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/os.py:136(makedirs)
7 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/genericpath.py:15(exists)
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:139(islink)
5 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:127(dirname)
3 0.000 0.000 0.000 0.000 {method 'close' of 'file' objects}
2 0.000 0.000 0.000 0.000 {posix.lstat}
9 0.000 0.000 0.000 0.000 {fcntl.flock}
3 0.000 0.000 0.000 0.000 {posix.mkdir}
2 0.000 0.000 0.000 0.000 {cPickle.Pickler}
3 0.000 0.000 0.000 0.000 {posix.access}
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:89(split)
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/stat.py:55(S_ISLNK)
3 0.000 0.000 0.000 0.000 {posix.fstat}
12 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
4 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:119(basename)
8 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
1 0.000 0.000 0.000 0.000 {cPickle.Unpickler}
9 0.000 0.000 0.000 0.000 {method 'fileno' of 'file' objects}
1 0.000 0.000 0.000 0.000 {posix.getpid}
1 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
[-- Attachment #9: profile_08.txt --]
[-- Type: text/plain, Size: 7219 bytes --]
605744 function calls (605739 primitive calls) in 0.617 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
4917 0.011 0.000 0.576 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:132(get_taskhash)
4917 0.082 0.000 0.565 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:188(get_taskhash)
23313 0.017 0.000 0.268 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:117(rundep_check)
23313 0.063 0.000 0.250 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:3(sstate_rundepfilter)
66338 0.086 0.000 0.086 0.000 {method 'rsplit' of 'str' objects}
4917 0.015 0.000 0.066 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:179(read_taint)
14820 0.013 0.000 0.061 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:11(isKernel)
4923 0.061 0.000 0.061 0.000 {open}
4917 0.016 0.000 0.059 0.000 {sorted}
416 0.013 0.000 0.044 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:98(get_checksums)
23313 0.023 0.000 0.043 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:342(clean_basepath)
29841 0.038 0.000 0.038 0.000 {method 'find' of 'str' objects}
15732 0.025 0.000 0.033 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:14(isPackageGroup)
1 0.000 0.000 0.029 0.029 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:238(writeout_file_checksum_cache)
1 0.018 0.018 0.029 0.029 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:819(save_merge)
23313 0.025 0.000 0.025 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
32340 0.018 0.000 0.018 0.000 {method 'join' of 'str' objects}
16623 0.011 0.000 0.011 0.000 {method 'split' of 'str' objects}
1211 0.001 0.000 0.010 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:101(checksum_file)
16542 0.005 0.000 0.009 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:5(isNative)
1211 0.003 0.000 0.009 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:71(get_checksum)
4917 0.008 0.000 0.008 0.000 {_hashlib.openssl_md5}
2184 0.008 0.000 0.008 0.000 {posix.stat}
1214 0.002 0.000 0.008 0.000 /usr/lib64/python2.7/genericpath.py:38(isdir)
39071 0.007 0.000 0.007 0.000 {method 'startswith' of 'str' objects}
15732 0.004 0.000 0.007 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:9(isNativeSDK)
105820 0.007 0.000 0.007 0.000 {len}
23313 0.005 0.000 0.005 0.000 {method 'group' of '_sre.SRE_Match' objects}
46451 0.005 0.000 0.005 0.000 {method 'append' of 'list' objects}
1211 0.002 0.000 0.005 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:41(cached_mtime)
16563 0.004 0.000 0.004 0.000 {method 'endswith' of 'str' objects}
4917 0.004 0.000 0.004 0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
15760 0.002 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:7(isCross)
55 0.002 0.000 0.002 0.000 {method 'copy' of 'set' objects}
651 0.001 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:20(isImage)
936 0.001 0.000 0.001 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:17(isAllArch)
1214 0.001 0.000 0.001 0.000 /usr/lib64/python2.7/stat.py:40(S_ISDIR)
1 0.000 0.000 0.001 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:109(checksum_dir)
9/4 0.000 0.000 0.001 0.000 /usr/lib64/python2.7/os.py:209(walk)
1211 0.001 0.000 0.001 0.000 {method 'get' of 'dict' objects}
4917 0.001 0.000 0.001 0.000 {method 'add' of 'set' objects}
2 0.001 0.000 0.001 0.000 {method 'dump' of 'cPickle.Pickler' objects}
416 0.001 0.000 0.001 0.000 {method 'sort' of 'list' objects}
4917 0.001 0.000 0.001 0.000 {method 'remove' of 'set' objects}
4 0.001 0.000 0.001 0.000 {posix.listdir}
1216 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/stat.py:24(S_IFMT)
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:789(save_extras)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:443(lockfile)
21 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:68(join)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:498(unlockfile)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:699(mkdirhier)
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/os.py:136(makedirs)
4 0.000 0.000 0.000 0.000 {posix.unlink}
7 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/genericpath.py:15(exists)
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:139(islink)
3 0.000 0.000 0.000 0.000 {method 'close' of 'file' objects}
2 0.000 0.000 0.000 0.000 {posix.lstat}
5 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:127(dirname)
9 0.000 0.000 0.000 0.000 {fcntl.flock}
3 0.000 0.000 0.000 0.000 {posix.mkdir}
2 0.000 0.000 0.000 0.000 {cPickle.Pickler}
1 0.000 0.000 0.000 0.000 {built-in method load}
3 0.000 0.000 0.000 0.000 {posix.access}
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:89(split)
3 0.000 0.000 0.000 0.000 {posix.fstat}
5 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:119(basename)
2 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/stat.py:55(S_ISLNK)
8 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
13 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
1 0.000 0.000 0.000 0.000 {cPickle.Unpickler}
1 0.000 0.000 0.000 0.000 {posix.getpid}
1 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects}
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:88(merge_data)
9 0.000 0.000 0.000 0.000 {method 'fileno' of 'file' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
[-- Attachment #10: profile_09.txt --]
[-- Type: text/plain, Size: 8258 bytes --]
22669370 function calls (22656934 primitive calls) in 80.332 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
4911 0.016 0.000 80.039 0.016 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:132(get_taskhash)
4911 0.107 0.000 80.023 0.016 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:188(get_taskhash)
417 0.015 0.000 79.538 0.191 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:98(get_checksums)
23 0.229 0.010 79.169 3.442 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:109(checksum_dir)
53519 0.075 0.000 70.268 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:101(checksum_file)
53519 0.338 0.000 70.193 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:71(get_checksum)
52308 57.299 0.001 69.200 0.001 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:512(md5_file)
20893617 11.202 0.000 11.202 0.000 {method 'update' of '_hashlib.HASH' objects}
15955/3519 0.160 0.000 8.463 0.002 /usr/lib64/python2.7/os.py:209(walk)
3498 5.004 0.001 5.004 0.001 {posix.listdir}
110299 3.558 0.000 3.558 0.000 {posix.stat}
57021 0.088 0.000 3.268 0.000 /usr/lib64/python2.7/genericpath.py:38(isdir)
53519 0.175 0.000 0.613 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:41(cached_mtime)
57225 0.577 0.000 0.577 0.000 {open}
111605 0.297 0.000 0.457 0.000 /usr/lib64/python2.7/posixpath.py:68(join)
1 0.028 0.028 0.282 0.282 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:238(writeout_file_checksum_cache)
23297 0.017 0.000 0.201 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:117(rundep_check)
23297 0.059 0.000 0.184 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:3(sstate_rundepfilter)
1 0.004 0.004 0.176 0.176 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:819(save_merge)
2 0.164 0.082 0.164 0.082 {method 'dump' of 'cPickle.Pickler' objects}
57219 0.131 0.000 0.131 0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
150639 0.117 0.000 0.117 0.000 {method 'startswith' of 'str' objects}
1 0.000 0.000 0.113 0.113 /usr/lib64/python2.7/glob.py:18(glob)
31 0.000 0.000 0.113 0.004 /usr/lib64/python2.7/glob.py:29(iglob)
1 0.000 0.000 0.113 0.113 /usr/lib64/python2.7/glob.py:66(glob1)
1 0.000 0.000 0.078 0.078 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:789(save_extras)
4911 0.014 0.000 0.073 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:179(read_taint)
1 0.069 0.069 0.069 0.069 {built-in method load}
57219 0.064 0.000 0.064 0.000 {_hashlib.openssl_md5}
14819 0.014 0.000 0.063 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:11(isKernel)
417 0.061 0.000 0.061 0.000 {method 'sort' of 'list' objects}
57021 0.038 0.000 0.059 0.000 /usr/lib64/python2.7/stat.py:40(S_ISDIR)
4911 0.015 0.000 0.057 0.000 {sorted}
128146 0.056 0.000 0.056 0.000 {method 'endswith' of 'str' objects}
53519 0.043 0.000 0.043 0.000 {method 'get' of 'dict' objects}
3479 0.009 0.000 0.042 0.000 /usr/lib64/python2.7/posixpath.py:139(islink)
23297 0.022 0.000 0.042 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:342(clean_basepath)
29839 0.040 0.000 0.040 0.000 {method 'find' of 'str' objects}
15731 0.025 0.000 0.034 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:14(isPackageGroup)
154535 0.028 0.000 0.028 0.000 {method 'append' of 'list' objects}
3479 0.027 0.000 0.027 0.000 {posix.lstat}
23300 0.025 0.000 0.025 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
60500 0.023 0.000 0.023 0.000 /usr/lib64/python2.7/stat.py:24(S_IFMT)
66305 0.019 0.000 0.019 0.000 {method 'rsplit' of 'str' objects}
32338 0.019 0.000 0.019 0.000 {method 'join' of 'str' objects}
1 0.016 0.016 0.016 0.016 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:88(merge_data)
16541 0.005 0.000 0.010 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:5(isNative)
16626 0.010 0.000 0.010 0.000 {method 'split' of 'str' objects}
15731 0.005 0.000 0.008 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:9(isNativeSDK)
105673 0.006 0.000 0.006 0.000 {len}
3479 0.004 0.000 0.006 0.000 /usr/lib64/python2.7/stat.py:55(S_ISLNK)
23297 0.005 0.000 0.005 0.000 {method 'group' of '_sre.SRE_Match' objects}
15759 0.003 0.000 0.003 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:7(isCross)
651 0.001 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:20(isImage)
55 0.002 0.000 0.002 0.000 {method 'copy' of 'set' objects}
936 0.001 0.000 0.001 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:17(isAllArch)
4 0.001 0.000 0.001 0.000 {posix.unlink}
4911 0.001 0.000 0.001 0.000 {method 'add' of 'set' objects}
23 0.001 0.000 0.001 0.000 {method 'extend' of 'list' objects}
4911 0.001 0.000 0.001 0.000 {method 'remove' of 'set' objects}
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:443(lockfile)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:498(unlockfile)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:699(mkdirhier)
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/os.py:136(makedirs)
7 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/genericpath.py:15(exists)
1 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/fnmatch.py:45(filter)
1 0.000 0.000 0.000 0.000 {filter}
5 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:127(dirname)
3 0.000 0.000 0.000 0.000 {method 'close' of 'file' objects}
38 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:77(<lambda>)
9 0.000 0.000 0.000 0.000 {fcntl.flock}
30 0.000 0.000 0.000 0.000 {method 'match' of '_sre.SRE_Pattern' objects}
3 0.000 0.000 0.000 0.000 {posix.mkdir}
2 0.000 0.000 0.000 0.000 {cPickle.Pickler}
3 0.000 0.000 0.000 0.000 {posix.access}
4 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:89(split)
15 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:94(has_magic)
3 0.000 0.000 0.000 0.000 {posix.fstat}
9 0.000 0.000 0.000 0.000 {method 'fileno' of 'file' objects}
9 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
6 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:119(basename)
1 0.000 0.000 0.000 0.000 {cPickle.Unpickler}
1 0.000 0.000 0.000 0.000 {isinstance}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:51(normcase)
1 0.000 0.000 0.000 0.000 {posix.getpid}
[-- Attachment #11: profile_10.txt --]
[-- Type: text/plain, Size: 8042 bytes --]
1566521 function calls (1554085 primitive calls) in 8.581 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
4911 0.015 0.000 8.496 0.002 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:132(get_taskhash)
4911 0.101 0.000 8.481 0.002 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:188(get_taskhash)
417 0.012 0.000 8.031 0.019 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:98(get_checksums)
23 0.096 0.004 7.838 0.341 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:109(checksum_dir)
15955/3519 0.224 0.000 7.168 0.002 /usr/lib64/python2.7/os.py:209(walk)
3498 3.700 0.001 3.700 0.001 {posix.listdir}
110299 3.213 0.000 3.213 0.000 {posix.stat}
57021 0.077 0.000 3.165 0.000 /usr/lib64/python2.7/genericpath.py:38(isdir)
53519 0.036 0.000 0.455 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:101(checksum_file)
53519 0.105 0.000 0.420 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:71(get_checksum)
53519 0.093 0.000 0.270 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:41(cached_mtime)
111605 0.182 0.000 0.267 0.000 /usr/lib64/python2.7/posixpath.py:68(join)
23297 0.016 0.000 0.192 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:117(rundep_check)
23297 0.055 0.000 0.175 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:3(sstate_rundepfilter)
1 0.006 0.006 0.073 0.073 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:238(writeout_file_checksum_cache)
1 0.001 0.001 0.067 0.067 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:819(save_merge)
14819 0.013 0.000 0.061 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:11(isKernel)
150639 0.061 0.000 0.061 0.000 {method 'startswith' of 'str' objects}
417 0.060 0.000 0.060 0.000 {method 'sort' of 'list' objects}
4917 0.060 0.000 0.060 0.000 {open}
4911 0.014 0.000 0.058 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:179(read_taint)
4911 0.015 0.000 0.055 0.000 {sorted}
57021 0.034 0.000 0.053 0.000 /usr/lib64/python2.7/stat.py:40(S_ISDIR)
2 0.051 0.025 0.051 0.025 {method 'dump' of 'cPickle.Pickler' objects}
53519 0.044 0.000 0.044 0.000 {method 'get' of 'dict' objects}
23297 0.021 0.000 0.040 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/siggen.py:342(clean_basepath)
29839 0.039 0.000 0.039 0.000 {method 'find' of 'str' objects}
128146 0.037 0.000 0.037 0.000 {method 'endswith' of 'str' objects}
15731 0.025 0.000 0.033 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:14(isPackageGroup)
1 0.000 0.000 0.024 0.024 /usr/lib64/python2.7/glob.py:18(glob)
31 0.000 0.000 0.024 0.001 /usr/lib64/python2.7/glob.py:29(iglob)
1 0.000 0.000 0.024 0.024 /usr/lib64/python2.7/glob.py:66(glob1)
23300 0.024 0.000 0.024 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
154535 0.023 0.000 0.023 0.000 {method 'append' of 'list' objects}
3479 0.005 0.000 0.021 0.000 /usr/lib64/python2.7/posixpath.py:139(islink)
60500 0.020 0.000 0.020 0.000 /usr/lib64/python2.7/stat.py:24(S_IFMT)
66305 0.019 0.000 0.019 0.000 {method 'rsplit' of 'str' objects}
32338 0.018 0.000 0.018 0.000 {method 'join' of 'str' objects}
3479 0.012 0.000 0.012 0.000 {posix.lstat}
16626 0.011 0.000 0.011 0.000 {method 'split' of 'str' objects}
4911 0.010 0.000 0.010 0.000 {_hashlib.openssl_md5}
16541 0.005 0.000 0.009 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:5(isNative)
15731 0.004 0.000 0.007 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:9(isNativeSDK)
105673 0.006 0.000 0.006 0.000 {len}
23297 0.005 0.000 0.005 0.000 {method 'group' of '_sre.SRE_Match' objects}
3479 0.002 0.000 0.004 0.000 /usr/lib64/python2.7/stat.py:55(S_ISLNK)
4911 0.003 0.000 0.003 0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
15759 0.002 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:7(isCross)
55 0.002 0.000 0.002 0.000 {method 'copy' of 'set' objects}
651 0.001 0.000 0.002 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:20(isImage)
936 0.001 0.000 0.001 0.000 /home/marquiz/yocto/openembedded-core/meta/lib/oe/sstatesig.py:17(isAllArch)
4911 0.001 0.000 0.001 0.000 {method 'add' of 'set' objects}
4911 0.001 0.000 0.001 0.000 {method 'remove' of 'set' objects}
23 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects}
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/cache.py:789(save_extras)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:443(lockfile)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:498(unlockfile)
3 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/utils.py:699(mkdirhier)
4 0.000 0.000 0.000 0.000 {posix.unlink}
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/os.py:136(makedirs)
1 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/fnmatch.py:45(filter)
7 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/genericpath.py:15(exists)
3 0.000 0.000 0.000 0.000 {method 'close' of 'file' objects}
1 0.000 0.000 0.000 0.000 {filter}
9 0.000 0.000 0.000 0.000 {fcntl.flock}
5 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:127(dirname)
4 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:89(split)
1 0.000 0.000 0.000 0.000 {built-in method load}
3 0.000 0.000 0.000 0.000 {posix.mkdir}
2 0.000 0.000 0.000 0.000 {cPickle.Pickler}
30 0.000 0.000 0.000 0.000 {method 'match' of '_sre.SRE_Pattern' objects}
38 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:77(<lambda>)
3 0.000 0.000 0.000 0.000 {posix.access}
6 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:119(basename)
3 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/glob.py:94(has_magic)
15 0.000 0.000 0.000 0.000 {method 'rfind' of 'str' objects}
9 0.000 0.000 0.000 0.000 {method 'rstrip' of 'str' objects}
3 0.000 0.000 0.000 0.000 {posix.fstat}
9 0.000 0.000 0.000 0.000 {method 'fileno' of 'file' objects}
1 0.000 0.000 0.000 0.000 {cPickle.Unpickler}
1 0.000 0.000 0.000 0.000 {isinstance}
1 0.000 0.000 0.000 0.000 /home/marquiz/yocto/openembedded-core/bitbake/lib/bb/checksum.py:88(merge_data)
1 0.000 0.000 0.000 0.000 /usr/lib64/python2.7/posixpath.py:51(normcase)
1 0.000 0.000 0.000 0.000 {posix.getpid}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-02-02 16:23 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-01-26 13:34 [PATCH 0/6] Improve caching task file dependency checksum caching Markus Lehtonen
2016-01-26 13:34 ` [PATCH 1/6] bb/cache: drop some unused arguments Markus Lehtonen
2016-01-26 13:34 ` [PATCH 2/6] SignatureGenerator: add method for saving the file checksum cache Markus Lehtonen
2016-01-26 13:34 ` [PATCH 3/6] bb/runqueue: save task file dependency cache onto disk Markus Lehtonen
2016-01-26 13:34 ` [PATCH 4/6] FileChecksumCache: add get_checksums() method Markus Lehtonen
2016-01-26 13:34 ` [PATCH 5/6] MultiProcessCache: make cache filename configurable Markus Lehtonen
2016-01-26 13:34 ` [PATCH 6/6] SignatureGeneratorBasic: make checksum cache file configurable Markus Lehtonen
2016-01-29 17:09 ` [PATCH 0/6] Improve caching task file dependency checksum caching Richard Purdie
2016-02-02 16:23 ` Markus Lehtonen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.