* [PATCH 01/25] hashserv: Add support for equivalent hash reporting
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 02/25] runqueue/siggen: Allow handling of equivalent hashes Armin Kuster
` (23 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
The reason for this should be recorded in the commit logs. Imagine
you have a target recipe (e.g. meta-extsdk-toolchain) which depends on
gdb-cross. sstate in OE-Core allows gdb-cross to have the same hash
regardless of whether its built on x86 or arm. The outhash will be
different.
We need hashequiv to be able to adapt to the prescence of sstate artefacts
for meta-extsdk-toolchain and allow the hashes to re-intersect, rather than
trying to force a rebuild of meta-extsdk-toolchain. By this point in the build,
it would have already been installed from sstate so the build needs to adapt.
Equivalent hashes should be reported to the server as a taskhash that
needs to map to an specific unihash. This patch adds API to the hashserv
client/server to allow this.
[Thanks to Joshua Watt for help with this patch]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 674692fd46a7691a1de59ace6af0556cc5dd6a71)
---
lib/hashserv/client.py | 8 ++++++++
lib/hashserv/server.py | 36 ++++++++++++++++++++++++++++++++++++
2 files changed, 44 insertions(+)
diff --git a/lib/hashserv/client.py b/lib/hashserv/client.py
index f6595661..ae0cce9d 100644
--- a/lib/hashserv/client.py
+++ b/lib/hashserv/client.py
@@ -148,6 +148,14 @@ class Client(object):
m['unihash'] = unihash
return self.send_message({'report': m})
+ def report_unihash_equiv(self, taskhash, method, unihash, extra={}):
+ self._set_mode(self.MODE_NORMAL)
+ m = extra.copy()
+ m['taskhash'] = taskhash
+ m['method'] = method
+ m['unihash'] = unihash
+ return self.send_message({'report-equiv': m})
+
def get_stats(self):
self._set_mode(self.MODE_NORMAL)
return self.send_message({'get-stats': None})
diff --git a/lib/hashserv/server.py b/lib/hashserv/server.py
index 0aff7768..cc7e4823 100644
--- a/lib/hashserv/server.py
+++ b/lib/hashserv/server.py
@@ -143,6 +143,7 @@ class ServerClient(object):
handlers = {
'get': self.handle_get,
'report': self.handle_report,
+ 'report-equiv': self.handle_equivreport,
'get-stream': self.handle_get_stream,
'get-stats': self.handle_get_stats,
'reset-stats': self.handle_reset_stats,
@@ -303,6 +304,41 @@ class ServerClient(object):
self.write_message(d)
+ async def handle_equivreport(self, data):
+ with closing(self.db.cursor()) as cursor:
+ insert_data = {
+ 'method': data['method'],
+ 'outhash': "",
+ 'taskhash': data['taskhash'],
+ 'unihash': data['unihash'],
+ 'created': datetime.now()
+ }
+
+ for k in ('owner', 'PN', 'PV', 'PR', 'task', 'outhash_siginfo'):
+ if k in data:
+ insert_data[k] = data[k]
+
+ cursor.execute('''INSERT OR IGNORE INTO tasks_v2 (%s) VALUES (%s)''' % (
+ ', '.join(sorted(insert_data.keys())),
+ ', '.join(':' + k for k in sorted(insert_data.keys()))),
+ insert_data)
+
+ self.db.commit()
+
+ # Fetch the unihash that will be reported for the taskhash. If the
+ # unihash matches, it means this row was inserted (or the mapping
+ # was already valid)
+ row = self.query_equivalent(data['method'], data['taskhash'])
+
+ if row['unihash'] == data['unihash']:
+ logger.info('Adding taskhash equivalence for %s with unihash %s',
+ data['taskhash'], row['unihash'])
+
+ d = {k: row[k] for k in ('taskhash', 'method', 'unihash')}
+
+ self.write_message(d)
+
+
async def handle_get_stats(self, request):
d = {
'requests': self.request_stats.todict(),
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 02/25] runqueue/siggen: Allow handling of equivalent hashes
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
2020-01-04 2:22 ` [PATCH 01/25] hashserv: Add support for equivalent hash reporting Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 03/25] runqueue: Add extra debugging when locked sigs mismatches occur Armin Kuster
` (22 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Based on the hashserv's new ability to accept hash mappings, update runqueue
to use this through a helper function in siggen.
This addresses problems with meta-extsdk-toolchain and its dependency on
gdb-cross which caused errors when building eSDK. See the previous commit
for more details.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 39098b4ba2133f4d9229a0aa4fcf4c3e1291286a)
---
lib/bb/runqueue.py | 31 +++++++++++++++++++------------
lib/bb/siggen.py | 26 ++++++++++++++++++++++++++
2 files changed, 45 insertions(+), 12 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index bd7f03f9..a869ba52 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2283,12 +2283,26 @@ class RunQueueExecute:
for dep in self.rqdata.runtaskentries[tid].depends:
procdep.append(dep)
orighash = self.rqdata.runtaskentries[tid].hash
- self.rqdata.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)])
+ newhash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)])
origuni = self.rqdata.runtaskentries[tid].unihash
- self.rqdata.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid)
- logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, self.rqdata.runtaskentries[tid].hash, origuni, self.rqdata.runtaskentries[tid].unihash))
+ newuni = bb.parse.siggen.get_unihash(tid)
+ # FIXME, need to check it can come from sstate at all for determinism?
+ remapped = False
+ if newuni == origuni:
+ # Nothing to do, we match, skip code below
+ remapped = True
+ elif tid in self.scenequeue_covered or tid in self.sq_live:
+ # Already ran this setscene task or it running. Report the new taskhash
+ remapped = bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
+ logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
+
+ if not remapped:
+ logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, newhash, origuni, newuni))
+ self.rqdata.runtaskentries[tid].hash = newhash
+ self.rqdata.runtaskentries[tid].unihash = newuni
+ changed.add(tid)
+
next |= self.rqdata.runtaskentries[tid].revdeps
- changed.add(tid)
total.remove(tid)
next.intersection_update(total)
@@ -2307,18 +2321,11 @@ class RunQueueExecute:
self.pending_migrations.add(tid)
for tid in self.pending_migrations.copy():
- if tid in self.runq_running:
+ if tid in self.runq_running or tid in self.sq_live:
# Too late, task already running, not much we can do now
self.pending_migrations.remove(tid)
continue
- if tid in self.scenequeue_covered or tid in self.sq_live:
- # Already ran this setscene task or it running
- # Potentially risky, should we report this hash as a match?
- logger.info("Already covered setscene for %s so ignoring rehash" % (tid))
- self.pending_migrations.remove(tid)
- continue
-
valid = True
# Check no tasks this covers are running
for dep in self.sqdata.sq_covered_tasks[tid]:
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index e19812b1..edf10105 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -525,6 +525,32 @@ class SignatureGeneratorUniHashMixIn(object):
except OSError:
pass
+ def report_unihash_equiv(self, tid, taskhash, wanted_unihash, current_unihash, datacaches):
+ try:
+ extra_data = {}
+ data = self.client().report_unihash_equiv(taskhash, self.method, wanted_unihash, extra_data)
+ bb.note('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
+
+ if data is None:
+ bb.warn("Server unable to handle unihash report")
+ return False
+
+ finalunihash = data['unihash']
+
+ if finalunihash == current_unihash:
+ bb.note('Task %s unihash %s unchanged by server' % (tid, finalunihash))
+ elif finalunihash == wanted_unihash:
+ bb.note('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
+ self.set_unihash(tid, finalunihash)
+ return True
+ else:
+ # TODO: What to do here?
+ bb.note('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
+
+ except hashserv.client.HashConnectionError as e:
+ bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
+
+ return False
#
# Dummy class used for bitbake-selftest
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 03/25] runqueue: Add extra debugging when locked sigs mismatches occur
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
2020-01-04 2:22 ` [PATCH 01/25] hashserv: Add support for equivalent hash reporting Armin Kuster
2020-01-04 2:22 ` [PATCH 02/25] runqueue/siggen: Allow handling of equivalent hashes Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 04/25] knotty/uihelper: Switch from pids to tids for Task event management Armin Kuster
` (21 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3aad9978be2a40d4c535a5ae092f374ba2a5f627)
---
lib/bb/runqueue.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index a869ba52..246a9cdb 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2524,6 +2524,8 @@ class RunQueueExecute:
msg = 'Task %s.%s attempted to execute unexpectedly and should have been setscened' % (pn, taskname)
else:
msg = 'Task %s.%s attempted to execute unexpectedly' % (pn, taskname)
+ for t in self.scenequeue_notcovered:
+ msg = msg + "\nTask %s, unihash %s, taskhash %s" % (t, self.rqdata.runtaskentries[t].unihash, self.rqdata.runtaskentries[t].hash)
logger.error(msg + '\nThis is usually due to missing setscene tasks. Those missing in this build were: %s' % pprint.pformat(self.scenequeue_notcovered))
return True
return False
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 04/25] knotty/uihelper: Switch from pids to tids for Task event management
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (2 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 03/25] runqueue: Add extra debugging when locked sigs mismatches occur Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 05/25] siggen: Avoid taskhash mismatch errors for nostamp tasks when dependencies rehash Armin Kuster
` (20 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen cases where a task can execute with a given pid, complete
and a new task can start using the same pid before the UI handler has
had time to adapt.
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/knotty.py", line 484, in main
helper.eventHandler(event)
File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/uihelper.py", line 30, in eventHandler
del self.running_tasks[event.pid]
KeyError: 13490
This means using pids to match up events on the UI side is a bad
idea. Change the code to use task ids instead. There is a small
amount of fuzzy matching for the progress information since there
is no task information there and we don't want the overhead of a task
ID in every event, however since pid reuse is unlikely, we can live
with a progress bar not quite working properly in a corner case like
this.
[YOCTO #13667]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e427eafa1bb04008d12100ccc5c862122bba53e0)
---
lib/bb/build.py | 25 +++++++++++++------------
lib/bb/ui/knotty.py | 12 ++++++------
lib/bb/ui/uihelper.py | 39 ++++++++++++++++++++++++---------------
3 files changed, 43 insertions(+), 33 deletions(-)
diff --git a/lib/bb/build.py b/lib/bb/build.py
index 30a2ba23..3d9cc10c 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -57,8 +57,9 @@ builtins['os'] = os
class TaskBase(event.Event):
"""Base class for task events"""
- def __init__(self, t, logfile, d):
+ def __init__(self, t, fn, logfile, d):
self._task = t
+ self._fn = fn
self._package = d.getVar("PF")
self._mc = d.getVar("BB_CURRENT_MC")
self.taskfile = d.getVar("FILE")
@@ -81,8 +82,8 @@ class TaskBase(event.Event):
class TaskStarted(TaskBase):
"""Task execution started"""
- def __init__(self, t, logfile, taskflags, d):
- super(TaskStarted, self).__init__(t, logfile, d)
+ def __init__(self, t, fn, logfile, taskflags, d):
+ super(TaskStarted, self).__init__(t, fn, logfile, d)
self.taskflags = taskflags
class TaskSucceeded(TaskBase):
@@ -91,9 +92,9 @@ class TaskSucceeded(TaskBase):
class TaskFailed(TaskBase):
"""Task execution failed"""
- def __init__(self, task, logfile, metadata, errprinted = False):
+ def __init__(self, task, fn, logfile, metadata, errprinted = False):
self.errprinted = errprinted
- super(TaskFailed, self).__init__(task, logfile, metadata)
+ super(TaskFailed, self).__init__(task, fn, logfile, metadata)
class TaskFailedSilent(TaskBase):
"""Task execution failed (silently)"""
@@ -103,8 +104,8 @@ class TaskFailedSilent(TaskBase):
class TaskInvalid(TaskBase):
- def __init__(self, task, metadata):
- super(TaskInvalid, self).__init__(task, None, metadata)
+ def __init__(self, task, fn, metadata):
+ super(TaskInvalid, self).__init__(task, fn, None, metadata)
self._message = "No such task '%s'" % task
class TaskProgress(event.Event):
@@ -572,7 +573,7 @@ def _exec_task(fn, task, d, quieterr):
try:
try:
- event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
+ event.fire(TaskStarted(task, fn, logfn, flags, localdata), localdata)
except (bb.BBHandledException, SystemExit):
return 1
@@ -583,15 +584,15 @@ def _exec_task(fn, task, d, quieterr):
for func in (postfuncs or '').split():
exec_func(func, localdata)
except bb.BBHandledException:
- event.fire(TaskFailed(task, logfn, localdata, True), localdata)
+ event.fire(TaskFailed(task, fn, logfn, localdata, True), localdata)
return 1
except Exception as exc:
if quieterr:
- event.fire(TaskFailedSilent(task, logfn, localdata), localdata)
+ event.fire(TaskFailedSilent(task, fn, logfn, localdata), localdata)
else:
errprinted = errchk.triggered
logger.error(str(exc))
- event.fire(TaskFailed(task, logfn, localdata, errprinted), localdata)
+ event.fire(TaskFailed(task, fn, logfn, localdata, errprinted), localdata)
return 1
finally:
sys.stdout.flush()
@@ -614,7 +615,7 @@ def _exec_task(fn, task, d, quieterr):
logger.debug(2, "Zero size logfn %s, removing", logfn)
bb.utils.remove(logfn)
bb.utils.remove(loglink)
- event.fire(TaskSucceeded(task, logfn, localdata), localdata)
+ event.fire(TaskSucceeded(task, fn, logfn, localdata), localdata)
if not localdata.getVarFlag(task, 'nostamp', False) and not localdata.getVarFlag(task, 'selfstamp', False):
make_stamp(task, localdata)
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index 35736ade..bd9911cf 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -255,19 +255,19 @@ class TerminalFilter(object):
start_time = activetasks[t].get("starttime", None)
if not pbar or pbar.bouncing != (progress < 0):
if progress < 0:
- pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], t), 100, widgets=[progressbar.BouncingSlider(), ''], extrapos=2, resize_handler=self.sigwinch_handle)
+ pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[progressbar.BouncingSlider(), ''], extrapos=2, resize_handler=self.sigwinch_handle)
pbar.bouncing = True
else:
- pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], t), 100, widgets=[progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=4, resize_handler=self.sigwinch_handle)
+ pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=4, resize_handler=self.sigwinch_handle)
pbar.bouncing = False
activetasks[t]["progressbar"] = pbar
tasks.append((pbar, progress, rate, start_time))
else:
start_time = activetasks[t].get("starttime", None)
if start_time:
- tasks.append("%s - %s (pid %s)" % (activetasks[t]["title"], self.elapsed(currenttime - start_time), t))
+ tasks.append("%s - %s (pid %s)" % (activetasks[t]["title"], self.elapsed(currenttime - start_time), activetasks[t]["pid"]))
else:
- tasks.append("%s (pid %s)" % (activetasks[t]["title"], t))
+ tasks.append("%s (pid %s)" % (activetasks[t]["title"], activetasks[t]["pid"]))
if self.main.shutdown:
content = "Waiting for %s running tasks to finish:" % len(activetasks)
@@ -517,8 +517,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
continue
# Prefix task messages with recipe/task
- if event.taskpid in helper.running_tasks and event.levelno != format.PLAIN:
- taskinfo = helper.running_tasks[event.taskpid]
+ if event.taskpid in helper.pidmap and event.levelno != format.PLAIN:
+ taskinfo = helper.running_tasks[helper.pidmap[event.taskpid]]
event.msg = taskinfo['title'] + ': ' + event.msg
if hasattr(event, 'fn'):
event.msg = event.fn + ': ' + event.msg
diff --git a/lib/bb/ui/uihelper.py b/lib/bb/ui/uihelper.py
index c8dd7df0..48d808ae 100644
--- a/lib/bb/ui/uihelper.py
+++ b/lib/bb/ui/uihelper.py
@@ -15,39 +15,48 @@ class BBUIHelper:
# Running PIDs preserves the order tasks were executed in
self.running_pids = []
self.failed_tasks = []
+ self.pidmap = {}
self.tasknumber_current = 0
self.tasknumber_total = 0
def eventHandler(self, event):
+ # PIDs are a bad idea as they can be reused before we process all UI events.
+ # We maintain a 'fuzzy' match for TaskProgress since there is no other way to match
+ def removetid(pid, tid):
+ self.running_pids.remove(tid)
+ del self.running_tasks[tid]
+ if self.pidmap[pid] == tid:
+ del self.pidmap[pid]
+ self.needUpdate = True
+
if isinstance(event, bb.build.TaskStarted):
+ tid = event._fn + ":" + event._task
if event._mc != "default":
- self.running_tasks[event.pid] = { 'title' : "mc:%s:%s %s" % (event._mc, event._package, event._task), 'starttime' : time.time() }
+ self.running_tasks[tid] = { 'title' : "mc:%s:%s %s" % (event._mc, event._package, event._task), 'starttime' : time.time(), 'pid' : event.pid }
else:
- self.running_tasks[event.pid] = { 'title' : "%s %s" % (event._package, event._task), 'starttime' : time.time() }
- self.running_pids.append(event.pid)
+ self.running_tasks[tid] = { 'title' : "%s %s" % (event._package, event._task), 'starttime' : time.time(), 'pid' : event.pid }
+ self.running_pids.append(tid)
+ self.pidmap[event.pid] = tid
self.needUpdate = True
elif isinstance(event, bb.build.TaskSucceeded):
- del self.running_tasks[event.pid]
- self.running_pids.remove(event.pid)
- self.needUpdate = True
+ tid = event._fn + ":" + event._task
+ removetid(event.pid, tid)
elif isinstance(event, bb.build.TaskFailedSilent):
- del self.running_tasks[event.pid]
- self.running_pids.remove(event.pid)
+ tid = event._fn + ":" + event._task
+ removetid(event.pid, tid)
# Don't add to the failed tasks list since this is e.g. a setscene task failure
- self.needUpdate = True
elif isinstance(event, bb.build.TaskFailed):
- del self.running_tasks[event.pid]
- self.running_pids.remove(event.pid)
+ tid = event._fn + ":" + event._task
+ removetid(event.pid, tid)
self.failed_tasks.append( { 'title' : "%s %s" % (event._package, event._task)})
- self.needUpdate = True
elif isinstance(event, bb.runqueue.runQueueTaskStarted):
self.tasknumber_current = event.stats.completed + event.stats.active + event.stats.failed + 1
self.tasknumber_total = event.stats.total
self.needUpdate = True
elif isinstance(event, bb.build.TaskProgress):
- if event.pid > 0:
- self.running_tasks[event.pid]['progress'] = event.progress
- self.running_tasks[event.pid]['rate'] = event.rate
+ if event.pid > 0 and event.pid in self.pidmap:
+ self.running_tasks[self.pidmap[event.pid]]['progress'] = event.progress
+ self.running_tasks[self.pidmap[event.pid]]['rate'] = event.rate
self.needUpdate = True
else:
return False
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 05/25] siggen: Avoid taskhash mismatch errors for nostamp tasks when dependencies rehash
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (3 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 04/25] knotty/uihelper: Switch from pids to tids for Task event management Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 06/25] siggen: Ensure new unihash propagates through the system Armin Kuster
` (19 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
An example:
NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Started
ERROR: Taskhash mismatch b074da4334aff8aa06572e7a8725c941fa6b08de4ce714a65a90c0c0b680abea versus 17375278daed609a7129769b74a1336a37bdef14b534ae85189ccc033a9f2db4 for /home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb:do_check
NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Succeeded
Is caused by a rehash in a dependency happening somewhere earlier in the build
and the taint being reset.
Change the code so that nostamp taints are preserved to avoid the issue.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 61624a3fc38e8546e01356d5ce7a09f21e7094ab)
---
lib/bb/siggen.py | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index edf10105..de853268 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -232,10 +232,14 @@ class SignatureGeneratorBasic(SignatureGenerator):
taskdep = dataCache.task_deps[fn]
if 'nostamp' in taskdep and task in taskdep['nostamp']:
# Nostamp tasks need an implicit taint so that they force any dependent tasks to run
- import uuid
- taint = str(uuid.uuid4())
- data = data + taint
- self.taints[tid] = "nostamp:" + taint
+ if tid in self.taints and self.taints[tid].startswith("nostamp:"):
+ # Don't reset taint value upon every call
+ data = data + self.taints[tid][8:]
+ else:
+ import uuid
+ taint = str(uuid.uuid4())
+ data = data + taint
+ self.taints[tid] = "nostamp:" + taint
taint = self.read_taint(fn, task, dataCache.stamp[fn])
if taint:
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 06/25] siggen: Ensure new unihash propagates through the system
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (4 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 05/25] siggen: Avoid taskhash mismatch errors for nostamp tasks when dependencies rehash Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 07/25] runqueue: Batch scenequeue updates Armin Kuster
` (18 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Its possible the new unihash may not exist in sstate. Currently the code
would create an sstate object with the old hash however this updates it to
create the object with the new unihash.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit abcaa1398031fa5338a43859c661e6d4a9ce863d)
---
lib/bb/siggen.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index de853268..dbf51023 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -513,6 +513,7 @@ class SignatureGeneratorUniHashMixIn(object):
bb.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server))
bb.event.fire(bb.runqueue.taskUniHashUpdate(fn + ':do_' + task, new_unihash), d)
self.set_unihash(tid, new_unihash)
+ d.setVar('BB_UNIHASH', new_unihash)
else:
bb.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server))
except hashserv.client.HashConnectionError as e:
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 07/25] runqueue: Batch scenequeue updates
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (5 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 06/25] siggen: Ensure new unihash propagates through the system Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 08/25] siggen: Fix performance issue in get_unihash Armin Kuster
` (17 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Joshua Watt <jpewhacker@gmail.com>
Batch all updates to scenequeue data together in a single invocation
instead of checking each task serially. This allows the checks for
sstate object to happen in parallel, and also makes sure the log
statement only happens once (per set of rehashes).
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit db033a8f8a276d864bdb2e1eef159ab5794a0658)
---
lib/bb/runqueue.py | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 246a9cdb..cb499a1c 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2320,6 +2320,7 @@ class RunQueueExecute:
if tid not in self.pending_migrations:
self.pending_migrations.add(tid)
+ update_tasks = []
for tid in self.pending_migrations.copy():
if tid in self.runq_running or tid in self.sq_live:
# Too late, task already running, not much we can do now
@@ -2379,11 +2380,13 @@ class RunQueueExecute:
if tid in self.build_stamps:
del self.build_stamps[tid]
- origvalid = False
- if tid in self.sqdata.valid:
- origvalid = True
+ update_tasks.append((tid, harddepfail, tid in self.sqdata.valid))
+
+ if update_tasks:
self.sqdone = False
- update_scenequeue_data([tid], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False)
+ update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False)
+
+ for (tid, harddepfail, origvalid) in update_tasks:
if tid in self.sqdata.valid and not origvalid:
logger.info("Setscene task %s became valid" % tid)
if harddepfail:
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 08/25] siggen: Fix performance issue in get_unihash
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (6 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 07/25] runqueue: Batch scenequeue updates Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 09/25] bb.utils.fileslocked: don't leak files if yield throws Armin Kuster
` (16 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a significant performance issue in get_unihash(). The issue turns out
to be the lookups of setscene tasks. We can fix this by using a set() instead of
the current list.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1e561672d039ebfb8cd0e0654a44dcf48513317c)
---
lib/bb/siggen.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index dbf51023..2fec8599 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -44,7 +44,7 @@ class SignatureGenerator(object):
self.file_checksum_values = {}
self.taints = {}
self.unitaskhashes = {}
- self.setscenetasks = {}
+ self.setscenetasks = set()
def finalise(self, fn, d, varient):
return
@@ -110,7 +110,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.taints = {}
self.gendeps = {}
self.lookupcache = {}
- self.setscenetasks = {}
+ self.setscenetasks = set()
self.basewhitelist = set((data.getVar("BB_HASHBASE_WHITELIST") or "").split())
self.taskwhitelist = None
self.init_rundepcheck(data)
@@ -157,7 +157,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
return taskdeps
def set_setscene_tasks(self, setscene_tasks):
- self.setscenetasks = setscene_tasks
+ self.setscenetasks = set(setscene_tasks)
def finalise(self, fn, d, variant):
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 09/25] bb.utils.fileslocked: don't leak files if yield throws
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (7 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 08/25] siggen: Fix performance issue in get_unihash Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 10/25] runqueue: Rework process_possible_migrations() to improve performance Armin Kuster
` (15 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Chris Laplante via bitbake-devel <bitbake-devel@lists.openembedded.org>
Discovered with a recipe under devtool. The ${S}/singletask.lock file (added by
externalsrc.bbclass) was leaked, giving a warning like:
WARNING: <PN>+git999-r0 do_populate_lic: /home/laplante/yocto/sources/poky/bitbake/lib/bb/build.py:582: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/yocto/build/workspace/sources/<PN>/singletask.lock' mode='a+' encoding='UTF-8'>
exec_func(task, localdata)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6beddf6214e22b4002626761031a9e9d34fb04db)
---
lib/bb/utils.py | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index 8d40bcdf..d65265c4 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -428,10 +428,11 @@ def fileslocked(files):
for lockfile in files:
locks.append(bb.utils.lockfile(lockfile))
- yield
-
- for lock in locks:
- bb.utils.unlockfile(lock)
+ try:
+ yield
+ finally:
+ for lock in locks:
+ bb.utils.unlockfile(lock)
@contextmanager
def timeout(seconds):
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 10/25] runqueue: Rework process_possible_migrations() to improve performance
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (8 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 09/25] bb.utils.fileslocked: don't leak files if yield throws Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 11/25] runqueue: Fix task mismatch failures from incorrect logic Armin Kuster
` (14 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
The looping over multiple changed hashes causes many calls to get_taskhash
and get_unihash which are potentially slow and then overwritten.
Instead, batch up all the tasks which have changed unihashes and then
do one big loop over the changed tasks rather than each in turn.
This makes worlds of difference to the performance graphs and should speed
up build where many tasks are being rehashed.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c9c68d898985cf0bec6fc95f54c151cc50255cac)
---
lib/bb/runqueue.py | 103 ++++++++++++++++++++++++---------------------
1 file changed, 56 insertions(+), 47 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index cb499a1c..a45b27ce 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2248,6 +2248,7 @@ class RunQueueExecute:
def process_possible_migrations(self):
changed = set()
+ toprocess = set()
for tid, unihash in self.updated_taskhash_queue.copy():
if tid in self.runq_running and tid not in self.runq_complete:
continue
@@ -2258,53 +2259,61 @@ class RunQueueExecute:
logger.info("Task %s unihash changed to %s" % (tid, unihash))
self.rqdata.runtaskentries[tid].unihash = unihash
bb.parse.siggen.set_unihash(tid, unihash)
-
- # Work out all tasks which depend on this one
- total = set()
- next = set(self.rqdata.runtaskentries[tid].revdeps)
- while next:
- current = next.copy()
- total = total |next
- next = set()
- for ntid in current:
- next |= self.rqdata.runtaskentries[ntid].revdeps
- next.difference_update(total)
-
- # Now iterate those tasks in dependency order to regenerate their taskhash/unihash
- done = set()
- next = set(self.rqdata.runtaskentries[tid].revdeps)
- while next:
- current = next.copy()
- next = set()
- for tid in current:
- if not self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
- continue
- procdep = []
- for dep in self.rqdata.runtaskentries[tid].depends:
- procdep.append(dep)
- orighash = self.rqdata.runtaskentries[tid].hash
- newhash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)])
- origuni = self.rqdata.runtaskentries[tid].unihash
- newuni = bb.parse.siggen.get_unihash(tid)
- # FIXME, need to check it can come from sstate at all for determinism?
- remapped = False
- if newuni == origuni:
- # Nothing to do, we match, skip code below
- remapped = True
- elif tid in self.scenequeue_covered or tid in self.sq_live:
- # Already ran this setscene task or it running. Report the new taskhash
- remapped = bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
- logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
-
- if not remapped:
- logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, newhash, origuni, newuni))
- self.rqdata.runtaskentries[tid].hash = newhash
- self.rqdata.runtaskentries[tid].unihash = newuni
- changed.add(tid)
-
- next |= self.rqdata.runtaskentries[tid].revdeps
- total.remove(tid)
- next.intersection_update(total)
+ toprocess.add(tid)
+
+ # Work out all tasks which depend upon these
+ total = set()
+ for p in toprocess:
+ next = set(self.rqdata.runtaskentries[p].revdeps)
+ while next:
+ current = next.copy()
+ total = total | next
+ next = set()
+ for ntid in current:
+ next |= self.rqdata.runtaskentries[ntid].revdeps
+ next.difference_update(total)
+
+ # Now iterate those tasks in dependency order to regenerate their taskhash/unihash
+ next = set()
+ for p in total:
+ if len(self.rqdata.runtaskentries[p].depends) == 0:
+ next.add(p)
+ elif self.rqdata.runtaskentries[p].depends.isdisjoint(total):
+ next.add(p)
+
+ # When an item doesn't have dependencies in total, we can process it. Drop items from total when handled
+ while next:
+ current = next.copy()
+ next = set()
+ for tid in current:
+ if not self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
+ continue
+ procdep = []
+ for dep in self.rqdata.runtaskentries[tid].depends:
+ procdep.append(dep)
+ orighash = self.rqdata.runtaskentries[tid].hash
+ newhash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)])
+ origuni = self.rqdata.runtaskentries[tid].unihash
+ newuni = bb.parse.siggen.get_unihash(tid)
+ # FIXME, need to check it can come from sstate at all for determinism?
+ remapped = False
+ if newuni == origuni:
+ # Nothing to do, we match, skip code below
+ remapped = True
+ elif tid in self.scenequeue_covered or tid in self.sq_live:
+ # Already ran this setscene task or it running. Report the new taskhash
+ remapped = bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
+ logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
+
+ if not remapped:
+ #logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, newhash, origuni, newuni))
+ self.rqdata.runtaskentries[tid].hash = newhash
+ self.rqdata.runtaskentries[tid].unihash = newuni
+ changed.add(tid)
+
+ next |= self.rqdata.runtaskentries[tid].revdeps
+ total.remove(tid)
+ next.intersection_update(total)
if changed:
for mc in self.rq.worker:
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 11/25] runqueue: Fix task mismatch failures from incorrect logic
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (9 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 10/25] runqueue: Rework process_possible_migrations() to improve performance Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 12/25] siggen: Split get_tashhash for performance Armin Kuster
` (13 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
The "no dependencies" task case was not being correctly considered in this
code and seemed to be the cause of occasionaly task hash mismatch errors
that were being seen as the dependencies were never accounted for properly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 608b9f821539de813bfbd9e65950dbc56a274bc2)
---
lib/bb/runqueue.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index a45b27ce..b3648ddb 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2286,7 +2286,7 @@ class RunQueueExecute:
current = next.copy()
next = set()
for tid in current:
- if not self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
+ if len(self.rqdata.runtaskentries[p].depends) and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
continue
procdep = []
for dep in self.rqdata.runtaskentries[tid].depends:
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 12/25] siggen: Split get_tashhash for performance
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (10 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 11/25] runqueue: Fix task mismatch failures from incorrect logic Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 13/25] runqueue: Fix sstate task iteration performance Armin Kuster
` (12 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
There are two operations happening in get_taskhash, the building of the
underlying data and the calculation of the hash.
Split these into two funtions since the preparation part doesn't need
to rerun when unihash changes, only the calculation does.
This split allows sigificant performance improvements for hashequiv
in builds where many hashes are equivalent and many hashes are changing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6a32af2808d748819f4af55c443578c8a63062b3)
---
lib/bb/runqueue.py | 1 +
lib/bb/siggen.py | 33 ++++++++++++++++++++++++---------
2 files changed, 25 insertions(+), 9 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index b3648ddb..515e9d43 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1185,6 +1185,7 @@ class RunQueueData:
procdep = []
for dep in self.runtaskentries[tid].depends:
procdep.append(dep)
+ bb.parse.siggen.prep_taskhash(tid, procdep, self.dataCaches[mc_from_tid(tid)])
self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, procdep, self.dataCaches[mc_from_tid(tid)])
self.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid)
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index 2fec8599..e484e5e3 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -52,6 +52,9 @@ class SignatureGenerator(object):
def get_unihash(self, tid):
return self.taskhash[tid]
+ def prep_taskhash(self, tid, deps, dataCache):
+ return
+
def get_taskhash(self, tid, deps, dataCache):
self.taskhash[tid] = hashlib.sha256(tid.encode("utf-8")).hexdigest()
return self.taskhash[tid]
@@ -198,12 +201,11 @@ class SignatureGeneratorBasic(SignatureGenerator):
pass
return taint
- def get_taskhash(self, tid, deps, dataCache):
+ def prep_taskhash(self, tid, deps, dataCache):
(mc, _, task, fn) = bb.runqueue.split_tid_mcfn(tid)
- data = dataCache.basetaskhash[tid]
- self.basehash[tid] = data
+ self.basehash[tid] = dataCache.basetaskhash[tid]
self.runtaskdeps[tid] = []
self.file_checksum_values[tid] = []
recipename = dataCache.pkg_fn[fn]
@@ -216,7 +218,6 @@ class SignatureGeneratorBasic(SignatureGenerator):
continue
if dep not in self.taskhash:
bb.fatal("%s is not in taskhash, caller isn't calling in dependency order?" % dep)
- data = data + self.get_unihash(dep)
self.runtaskdeps[tid].append(dep)
if task in dataCache.file_checksums[fn]:
@@ -226,27 +227,41 @@ class SignatureGeneratorBasic(SignatureGenerator):
checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename)
for (f,cs) in checksums:
self.file_checksum_values[tid].append((f,cs))
- if cs:
- data = data + cs
taskdep = dataCache.task_deps[fn]
if 'nostamp' in taskdep and task in taskdep['nostamp']:
# Nostamp tasks need an implicit taint so that they force any dependent tasks to run
if tid in self.taints and self.taints[tid].startswith("nostamp:"):
# Don't reset taint value upon every call
- data = data + self.taints[tid][8:]
+ pass
else:
import uuid
taint = str(uuid.uuid4())
- data = data + taint
self.taints[tid] = "nostamp:" + taint
taint = self.read_taint(fn, task, dataCache.stamp[fn])
if taint:
- data = data + taint
self.taints[tid] = taint
logger.warning("%s is tainted from a forced run" % tid)
+ return
+
+ def get_taskhash(self, tid, deps, dataCache):
+
+ data = self.basehash[tid]
+ for dep in self.runtaskdeps[tid]:
+ data = data + self.get_unihash(dep)
+
+ for (f, cs) in self.file_checksum_values[tid]:
+ if cs:
+ data = data + cs
+
+ if tid in self.taints:
+ if self.taints[tid].startswith("nostamp:"):
+ data = data + self.taints[tid][8:]
+ else:
+ data = data + self.taints[tid]
+
h = hashlib.sha256(data.encode("utf-8")).hexdigest()
self.taskhash[tid] = h
#d.setVar("BB_TASKHASH_task-%s" % task, taskhash[task])
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 13/25] runqueue: Fix sstate task iteration performance
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (11 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 12/25] siggen: Split get_tashhash for performance Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:22 ` [PATCH 14/25] runqueue: Optimise task migration code slightly Armin Kuster
` (11 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Creating a new sorted list of sstate tasks each iteration through runqueue is
extremely ineffecient and was compounded by the recent change from a list to set.
Create one sorted list instead of recreating it each time.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit de18824996841c3f35f54ff5ad12f94f6dc20d88)
---
lib/bb/runqueue.py | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 515e9d43..2ba4557f 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1965,10 +1965,14 @@ class RunQueueExecute:
self.rq.read_workers()
self.process_possible_migrations()
+ if not hasattr(self, "sorted_setscene_tids"):
+ # Don't want to sort this set every execution
+ self.sorted_setscene_tids = sorted(self.rqdata.runq_setscene_tids)
+
task = None
if not self.sqdone and self.can_start_task():
# Find the next setscene to run
- for nexttask in sorted(self.rqdata.runq_setscene_tids):
+ for nexttask in self.sorted_setscene_tids:
if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values():
if nexttask not in self.sqdata.unskippable and len(self.sqdata.sq_revdeps[nexttask]) > 0 and self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and self.check_dependencies(nexttask, self.sqdata.sq_revdeps[nexttask]):
if nexttask not in self.rqdata.target_tids:
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 14/25] runqueue: Optimise task migration code slightly
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (12 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 13/25] runqueue: Fix sstate task iteration performance Armin Kuster
@ 2020-01-04 2:22 ` Armin Kuster
2020-01-04 2:23 ` [PATCH 15/25] runqueue: Optimise out pointless loop iteration Armin Kuster
` (10 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:22 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the calls to difference_update out a code level which improves efficiency
significantly.
Also further combine the outer loop for efficiency too.
These two changes remove a bottleneck from the performance charts.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit e28ec69356f1797de3e4e3fca0fef710bc4564de)
---
lib/bb/runqueue.py | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 2ba4557f..6da612b7 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2268,15 +2268,16 @@ class RunQueueExecute:
# Work out all tasks which depend upon these
total = set()
+ next = set()
for p in toprocess:
- next = set(self.rqdata.runtaskentries[p].revdeps)
- while next:
- current = next.copy()
- total = total | next
- next = set()
- for ntid in current:
- next |= self.rqdata.runtaskentries[ntid].revdeps
- next.difference_update(total)
+ next |= self.rqdata.runtaskentries[p].revdeps
+ while next:
+ current = next.copy()
+ total = total | next
+ next = set()
+ for ntid in current:
+ next |= self.rqdata.runtaskentries[ntid].revdeps
+ next.difference_update(total)
# Now iterate those tasks in dependency order to regenerate their taskhash/unihash
next = set()
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 15/25] runqueue: Optimise out pointless loop iteration
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (13 preceding siblings ...)
2020-01-04 2:22 ` [PATCH 14/25] runqueue: Optimise task migration code slightly Armin Kuster
@ 2020-01-04 2:23 ` Armin Kuster
2020-01-04 2:23 ` [PATCH 16/25] runqueue: Optimise task filtering Armin Kuster
` (9 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:23 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 105d1f0748edde7753a4063e6fdc758ffc8a8a9e)
---
lib/bb/runqueue.py | 12 +++---------
1 file changed, 3 insertions(+), 9 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 6da612b7..73775d97 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1182,11 +1182,8 @@ class RunQueueData:
return len(self.runtaskentries)
def prepare_task_hash(self, tid):
- procdep = []
- for dep in self.runtaskentries[tid].depends:
- procdep.append(dep)
- bb.parse.siggen.prep_taskhash(tid, procdep, self.dataCaches[mc_from_tid(tid)])
- self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, procdep, self.dataCaches[mc_from_tid(tid)])
+ bb.parse.siggen.prep_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches[mc_from_tid(tid)])
+ self.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, self.runtaskentries[tid].depends, self.dataCaches[mc_from_tid(tid)])
self.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid)
def dump_data(self):
@@ -2294,11 +2291,8 @@ class RunQueueExecute:
for tid in current:
if len(self.rqdata.runtaskentries[p].depends) and not self.rqdata.runtaskentries[tid].depends.isdisjoint(total):
continue
- procdep = []
- for dep in self.rqdata.runtaskentries[tid].depends:
- procdep.append(dep)
orighash = self.rqdata.runtaskentries[tid].hash
- newhash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)])
+ newhash = bb.parse.siggen.get_taskhash(tid, self.rqdata.runtaskentries[tid].depends, self.rqdata.dataCaches[mc_from_tid(tid)])
origuni = self.rqdata.runtaskentries[tid].unihash
newuni = bb.parse.siggen.get_unihash(tid)
# FIXME, need to check it can come from sstate at all for determinism?
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 16/25] runqueue: Optimise task filtering
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (14 preceding siblings ...)
2020-01-04 2:23 ` [PATCH 15/25] runqueue: Optimise out pointless loop iteration Armin Kuster
@ 2020-01-04 2:23 ` Armin Kuster
2020-01-04 2:23 ` [PATCH 17/25] runqueue: Only call into the migrations function if migrations active Armin Kuster
` (8 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:23 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
We were seeing this running thousands of times with hashequiv, do
the filtering where it makes more sense and make it persist.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2cfeb9998a8ad5b1dcda0bb4e192c5e4306dab17)
---
lib/bb/runqueue.py | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 73775d97..b90ac875 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -148,8 +148,9 @@ class RunQueueScheduler(object):
"""
Return the id of the first task we find that is buildable
"""
+ # Once tasks are running we don't need to worry about them again
+ self.buildable.difference_update(self.rq.runq_running)
buildable = set(self.buildable)
- buildable.difference_update(self.rq.runq_running)
buildable.difference_update(self.rq.holdoff_tasks)
buildable.intersection_update(self.rq.tasks_covered | self.rq.tasks_notcovered)
if not buildable:
@@ -207,8 +208,6 @@ class RunQueueScheduler(object):
def newbuildable(self, task):
self.buildable.add(task)
- # Once tasks are running we don't need to worry about them again
- self.buildable.difference_update(self.rq.runq_running)
def removebuildable(self, task):
self.buildable.remove(task)
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 17/25] runqueue: Only call into the migrations function if migrations active
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (15 preceding siblings ...)
2020-01-04 2:23 ` [PATCH 16/25] runqueue: Optimise task filtering Armin Kuster
@ 2020-01-04 2:23 ` Armin Kuster
2020-01-04 2:23 ` [PATCH 18/25] lib/bb: Optimise out debug messages from cooker Armin Kuster
` (7 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:23 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
This doesn't save much time but does make the profile counts for the
function more accurate which is in itself useful.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d446fa89d206fbc6d098215163c968ea5a8cf4a9)
---
lib/bb/runqueue.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index b90ac875..729439ef 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1959,7 +1959,8 @@ class RunQueueExecute:
"""
self.rq.read_workers()
- self.process_possible_migrations()
+ if self.updated_taskhash_queue or self.pending_migrations:
+ self.process_possible_migrations()
if not hasattr(self, "sorted_setscene_tids"):
# Don't want to sort this set every execution
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 18/25] lib/bb: Optimise out debug messages from cooker
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (16 preceding siblings ...)
2020-01-04 2:23 ` [PATCH 17/25] runqueue: Only call into the migrations function if migrations active Armin Kuster
@ 2020-01-04 2:23 ` Armin Kuster
2020-01-04 2:23 ` [PATCH 19/25] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories Armin Kuster
` (6 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:23 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
We have bb.debug(2, xxx) messages in cooker which are useful for debugging
but have really bad effects on performance, 640,000 calls on recent profile
graphs taking tens of seconds.
Rather than commenting out debug which can be useful for debugging, don't
create events for debug log messages from cooker which would never be seen.
We already stop the messages hitting the IPC but this avoids the overhead
of creating the log messages too, which has been shown to be signficiant
on the profiles. This allows the code to perform whilst allowing debug
messages to be availble when wanted/enabled.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f04cd931091fb0508badf3e002d70a6952700495)
---
lib/bb/__init__.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/lib/bb/__init__.py b/lib/bb/__init__.py
index c144311b..ce519ba3 100644
--- a/lib/bb/__init__.py
+++ b/lib/bb/__init__.py
@@ -43,6 +43,11 @@ class BBLogger(Logger):
Logger.__init__(self, name)
def bbdebug(self, level, msg, *args, **kwargs):
+ if not bb.event.worker_pid:
+ if self.name in bb.msg.loggerDefaultDomains and level > (bb.msg.loggerDefaultDomains[self.name]):
+ return
+ if level > (bb.msg.loggerDefaultDebugLevel):
+ return
return self.log(logging.DEBUG - level + 1, msg, *args, **kwargs)
def plain(self, msg, *args, **kwargs):
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 19/25] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (17 preceding siblings ...)
2020-01-04 2:23 ` [PATCH 18/25] lib/bb: Optimise out debug messages from cooker Armin Kuster
@ 2020-01-04 2:23 ` Armin Kuster
2020-01-04 2:23 ` [PATCH 20/25] prserv/serv: Use with while reading pidfile Armin Kuster
` (5 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:23 UTC (permalink / raw)
To: bitbake-devel
From: Aníbal Limón <anibal.limon@linaro.org>
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
of directories to exclude when making taskhash, our specific case
is using SRC_URI that points local VCS directory.
Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn"
Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 923aff060d8aba8456979c35b16d300ba7c13ff9)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
lib/bb/checksum.py | 5 +++--
lib/bb/fetch2/__init__.py | 4 ++--
lib/bb/siggen.py | 5 +++--
3 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/lib/bb/checksum.py b/lib/bb/checksum.py
index 5bc8a8fc..677020f4 100644
--- a/lib/bb/checksum.py
+++ b/lib/bb/checksum.py
@@ -74,7 +74,7 @@ class FileChecksumCache(MultiProcessCache):
else:
dest[0][h] = source[0][h]
- def get_checksums(self, filelist, pn):
+ def get_checksums(self, filelist, pn, localdirsexclude):
"""Get checksums for a list of files"""
def checksum_file(f):
@@ -90,7 +90,8 @@ class FileChecksumCache(MultiProcessCache):
if pth == "/":
bb.fatal("Refusing to checksum /")
dirchecksums = []
- for root, dirs, files in os.walk(pth):
+ for root, dirs, files in os.walk(pth, topdown=True):
+ [dirs.remove(d) for d in list(dirs) if d in localdirsexclude]
for name in files:
fullpth = os.path.join(root, name)
checksum = checksum_file(fullpth)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 07de6c26..731c1608 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1197,14 +1197,14 @@ def get_checksum_file_list(d):
return " ".join(filelist)
-def get_file_checksums(filelist, pn):
+def get_file_checksums(filelist, pn, localdirsexclude):
"""Get a list of the checksums for a list of local files
Returns the checksums for a list of local files, caching the results as
it proceeds
"""
- return _checksum_cache.get_checksums(filelist, pn)
+ return _checksum_cache.get_checksums(filelist, pn, localdirsexclude)
class FetchData(object):
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index e484e5e3..f982bf22 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -126,6 +126,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
self.unihash_cache = bb.cache.SimpleCache("1")
self.unitaskhashes = self.unihash_cache.init_cache(data, "bb_unihashes.dat", {})
+ self.localdirsexclude = (data.getVar("BB_SIGNATURE_LOCAL_DIRS_EXCLUDE") or "CVS .bzr .git .hg .osc .p4 .repo .svn").split()
def init_rundepcheck(self, data):
self.taskwhitelist = data.getVar("BB_HASHTASK_WHITELIST") or None
@@ -222,9 +223,9 @@ class SignatureGeneratorBasic(SignatureGenerator):
if task in dataCache.file_checksums[fn]:
if self.checksum_cache:
- checksums = self.checksum_cache.get_checksums(dataCache.file_checksums[fn][task], recipename)
+ checksums = self.checksum_cache.get_checksums(dataCache.file_checksums[fn][task], recipename, self.localdirsexclude)
else:
- checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename)
+ checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename, self.localdirsexclude)
for (f,cs) in checksums:
self.file_checksum_values[tid].append((f,cs))
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 20/25] prserv/serv: Use with while reading pidfile
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (18 preceding siblings ...)
2020-01-04 2:23 ` [PATCH 19/25] lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on directories Armin Kuster
@ 2020-01-04 2:23 ` Armin Kuster
2020-01-04 2:23 ` [PATCH 21/25] runqueue: Fix equiv hash handling build failures Armin Kuster
` (4 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:23 UTC (permalink / raw)
To: bitbake-devel
From: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6fa8a18ea4994031fdd1253fe363c5d8eeeba456)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
lib/prserv/serv.py | 12 +++++-------
1 file changed, 5 insertions(+), 7 deletions(-)
diff --git a/lib/prserv/serv.py b/lib/prserv/serv.py
index 6d8142fc..446e1fe1 100644
--- a/lib/prserv/serv.py
+++ b/lib/prserv/serv.py
@@ -292,10 +292,9 @@ class PRServer(SimpleXMLRPCServer):
logger.addHandler(streamhandler)
# write pidfile
- pid = str(os.getpid())
- pf = open(self.pidfile, 'w')
- pf.write("%s\n" % pid)
- pf.close()
+ pid = str(os.getpid())
+ with open(self.pidfile, 'w') as pf:
+ pf.write("%s\n" % pid)
self.work_forever()
self.delpid()
@@ -353,9 +352,8 @@ def start_daemon(dbfile, host, port, logfile):
ip = socket.gethostbyname(host)
pidfile = PIDPREFIX % (ip, port)
try:
- pf = open(pidfile,'r')
- pid = int(pf.readline().strip())
- pf.close()
+ with open(pidfile) as pf:
+ pid = int(pf.readline().strip())
except IOError:
pid = None
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 21/25] runqueue: Fix equiv hash handling build failures
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (19 preceding siblings ...)
2020-01-04 2:23 ` [PATCH 20/25] prserv/serv: Use with while reading pidfile Armin Kuster
@ 2020-01-04 2:23 ` Armin Kuster
2020-01-04 2:23 ` [PATCH 22/25] runqueue: Ensure task dependencies are run correctly Armin Kuster
` (3 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:23 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Regardless of whether we remapped the hash on the server or not, we need
to have bitbake work as if we did as we need to match how the stamp files
look.
This change resolves build failures where tasks were rerunning when they
shouldn't.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 40928f6991436cf687821015324483b205abfcb1)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
lib/bb/runqueue.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 729439ef..f8279980 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2302,8 +2302,9 @@ class RunQueueExecute:
remapped = True
elif tid in self.scenequeue_covered or tid in self.sq_live:
# Already ran this setscene task or it running. Report the new taskhash
- remapped = bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
+ bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
+ remapped = True
if not remapped:
#logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, newhash, origuni, newuni))
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 22/25] runqueue: Ensure task dependencies are run correctly
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (20 preceding siblings ...)
2020-01-04 2:23 ` [PATCH 21/25] runqueue: Fix equiv hash handling build failures Armin Kuster
@ 2020-01-04 2:23 ` Armin Kuster
2020-01-04 2:23 ` [PATCH 23/25] runqueue: Fix task dependency corner case in sanity test Armin Kuster
` (2 subsequent siblings)
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:23 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
We've seen a number of mystery failures where task B would run despite
task A, its dependency not having run. An example would be do_compile
when do_unpack didn't run.
This has been tracked down to this code block. In theory it shouldn't
trigger however it can and has due to bugs elsewhere. When it does, it
causes significant weird failures and possible build corruption.
Change the code to abort the build. This avoids any chance of corruption
and should ensure the issues get reported, putting an end to the weird
build failures.
There may be some cases where this triggers and it shouldn't, we'll work
through those as they arise and are identified.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7a92b7f58ab187eddfe550bd6fb687240c7b11bb)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
lib/bb/runqueue.py | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index f8279980..56ca2529 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2353,6 +2353,12 @@ class RunQueueExecute:
if tid in self.tasks_scenequeue_done:
self.tasks_scenequeue_done.remove(tid)
for dep in self.sqdata.sq_covered_tasks[tid]:
+ if dep in self.runq_complete:
+ bb.error("Task %s marked as completed but now needing to rerun? Aborting build." % dep)
+ self.failed_tids.append(tid)
+ self.rq.state = runQueueCleanUp
+ return
+
if dep not in self.runq_complete:
if dep in self.tasks_scenequeue_done and dep not in self.sqdata.unskippable:
self.tasks_scenequeue_done.remove(dep)
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 23/25] runqueue: Fix task dependency corner case in sanity test
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (21 preceding siblings ...)
2020-01-04 2:23 ` [PATCH 22/25] runqueue: Ensure task dependencies are run correctly Armin Kuster
@ 2020-01-04 2:23 ` Armin Kuster
2020-01-04 2:23 ` [PATCH 24/25] siggen: Test extra cross/native hashserv method Armin Kuster
2020-01-04 2:23 ` [PATCH 25/25] cache: Lower debug level for wold build messages Armin Kuster
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:23 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
A corner case was identified where tasks with valid stamps from previous
builds need to be accounted for in the new sanity test in the migration
code. Add a variable to track such completed tasks to ensure the sanity
test works correctly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit d517b1ef13ca7ab2fb4d761d3bd3b9fb7c591514)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
lib/bb/runqueue.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 56ca2529..6e3a91b8 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1708,6 +1708,7 @@ class RunQueueExecute:
self.runq_buildable = set()
self.runq_running = set()
self.runq_complete = set()
+ self.runq_tasksrun = set()
self.build_stamps = {}
self.build_stamps2 = []
@@ -1893,6 +1894,7 @@ class RunQueueExecute:
self.stats.taskCompleted()
bb.event.fire(runQueueTaskCompleted(task, self.stats, self.rq), self.cfgData)
self.task_completeoutright(task)
+ self.runq_tasksrun.add(task)
def task_fail(self, task, exitcode):
"""
@@ -2092,6 +2094,7 @@ class RunQueueExecute:
logger.debug(2, "Stamp current task %s", task)
self.task_skip(task, "existing")
+ self.runq_tasksrun.add(task)
return True
taskdep = self.rqdata.dataCaches[mc].task_deps[taskfn]
@@ -2353,7 +2356,7 @@ class RunQueueExecute:
if tid in self.tasks_scenequeue_done:
self.tasks_scenequeue_done.remove(tid)
for dep in self.sqdata.sq_covered_tasks[tid]:
- if dep in self.runq_complete:
+ if dep in self.runq_complete and dep not in self.runq_tasksrun:
bb.error("Task %s marked as completed but now needing to rerun? Aborting build." % dep)
self.failed_tids.append(tid)
self.rq.state = runQueueCleanUp
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 24/25] siggen: Test extra cross/native hashserv method
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (22 preceding siblings ...)
2020-01-04 2:23 ` [PATCH 23/25] runqueue: Fix task dependency corner case in sanity test Armin Kuster
@ 2020-01-04 2:23 ` Armin Kuster
2020-01-04 2:23 ` [PATCH 25/25] cache: Lower debug level for wold build messages Armin Kuster
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:23 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Hack the hashserv to allow extra data to be injected into the hashserv
method. This allows OE-Core to handle cases where there are multiple
sstate objects for the same taskhash, e.g. native/cross objects based
upon BUILD_ARCH or the host distro (when uninative isn't used).
This has been tested and proven to be very effective. We will likely
rework the code to improve how this is handled but for now this
improves automated builds until we can get to that refactoring and
more invasive changes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0a09b0fa03d1afc08037964dc63a18ef7cff9c78)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
lib/bb/siggen.py | 27 +++++++++++++++++++++------
1 file changed, 21 insertions(+), 6 deletions(-)
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index f982bf22..ded1da02 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -391,12 +391,16 @@ class SignatureGeneratorBasicHash(SignatureGeneratorBasic):
bb.build.write_taint(task, d, fn)
class SignatureGeneratorUniHashMixIn(object):
+ def __init__(self, data):
+ self.extramethod = {}
+ super().__init__(data)
+
def get_taskdata(self):
- return (self.server, self.method) + super().get_taskdata()
+ return (self.server, self.method, self.extramethod) + super().get_taskdata()
def set_taskdata(self, data):
- self.server, self.method = data[:2]
- super().set_taskdata(data[2:])
+ self.server, self.method, self.extramethod = data[:3]
+ super().set_taskdata(data[3:])
def client(self):
if getattr(self, '_client', None) is None:
@@ -453,7 +457,10 @@ class SignatureGeneratorUniHashMixIn(object):
unihash = taskhash
try:
- data = self.client().get_unihash(self.method, self.taskhash[tid])
+ method = self.method
+ if tid in self.extramethod:
+ method = method + self.extramethod[tid]
+ data = self.client().get_unihash(method, self.taskhash[tid])
if data:
unihash = data
# A unique hash equal to the taskhash is not very interesting,
@@ -522,7 +529,11 @@ class SignatureGeneratorUniHashMixIn(object):
extra_data['task'] = task
extra_data['outhash_siginfo'] = sigfile.read().decode('utf-8')
- data = self.client().report_unihash(taskhash, self.method, outhash, unihash, extra_data)
+ method = self.method
+ if tid in self.extramethod:
+ method = method + self.extramethod[tid]
+
+ data = self.client().report_unihash(taskhash, method, outhash, unihash, extra_data)
new_unihash = data['unihash']
if new_unihash != unihash:
@@ -549,7 +560,11 @@ class SignatureGeneratorUniHashMixIn(object):
def report_unihash_equiv(self, tid, taskhash, wanted_unihash, current_unihash, datacaches):
try:
extra_data = {}
- data = self.client().report_unihash_equiv(taskhash, self.method, wanted_unihash, extra_data)
+ method = self.method
+ if tid in self.extramethod:
+ method = method + self.extramethod[tid]
+
+ data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data)
bb.note('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
if data is None:
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread* [PATCH 25/25] cache: Lower debug level for wold build messages
2020-01-04 2:22 [PATCH 00/25] 1.44 Patch review Armin Kuster
` (23 preceding siblings ...)
2020-01-04 2:23 ` [PATCH 24/25] siggen: Test extra cross/native hashserv method Armin Kuster
@ 2020-01-04 2:23 ` Armin Kuster
24 siblings, 0 replies; 26+ messages in thread
From: Armin Kuster @ 2020-01-04 2:23 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
These messages spam the logs for no good reason, they were useful for debugging
a particular problem long ago but are distracting noise now. Disable them.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1a9247c468cf09da60e5d396ccb81e950841c99e)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
lib/bb/cache.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index b6f7da59..ead8abc5 100644
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -208,10 +208,10 @@ class CoreRecipeInfo(RecipeInfoCommon):
# Collect files we may need for possible world-dep
# calculations
- if self.not_world:
- logger.debug(1, "EXCLUDE FROM WORLD: %s", fn)
- else:
+ if not self.not_world:
cachedata.possible_world.append(fn)
+ #else:
+ # logger.debug(2, "EXCLUDE FROM WORLD: %s", fn)
# create a collection of all targets for sanity checking
# tasks, such as upstream versions, license, and tools for
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread