[PATCH] siggen: Ensure taskhash mismatches don't override existing data

[PATCH] siggen: Ensure taskhash mismatches don't override existing data

[Richard Purdie]

We recalculate the taskhash to ensure the version we have matches
what we think it should be. When we write out a sigdata file, use
the calculated value so that we don't overwrite any existing file.
This leaves any original taskhash sigdata file intact to allow a
debugging comparison.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index 75a80bf..be09a96 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -284,6 +284,15 @@ class SignatureGeneratorBasic(SignatureGenerator):
             if 'nostamp:' in self.taints[k]:
                 data['taint'] = self.taints[k]
 
+        computed_basehash = calc_basehash(data)
+        if computed_basehash != self.basehash[k]:
+            bb.error("Basehash mismatch %s versus %s for %s" % (computed_basehash, self.basehash[k], k))
+        if runtime and k in self.taskhash:
+            computed_taskhash = calc_taskhash(data)
+            if computed_taskhash != self.taskhash[k]:
+                bb.error("Taskhash mismatch %s versus %s for %s" % (computed_taskhash, self.taskhash[k], k))
+                sigfile = sigfile.replace(self.taskhash[k], computed_taskhash)
+
         fd, tmpfile = tempfile.mkstemp(dir=os.path.dirname(sigfile), prefix="sigtask.")
         try:
             with os.fdopen(fd, "wb") as stream:
@@ -298,14 +307,6 @@ class SignatureGeneratorBasic(SignatureGenerator):
                 pass
             raise err
 
-        computed_basehash = calc_basehash(data)
-        if computed_basehash != self.basehash[k]:
-            bb.error("Basehash mismatch %s versus %s for %s" % (computed_basehash, self.basehash[k], k))
-        if runtime and k in self.taskhash:
-            computed_taskhash = calc_taskhash(data)
-            if computed_taskhash != self.taskhash[k]:
-                bb.error("Taskhash mismatch %s versus %s for %s" % (computed_taskhash, self.taskhash[k], k))
-
     def dump_sigs(self, dataCaches, options):
         for fn in self.taskdeps:
             for task in self.taskdeps[fn]:

Re: [PATCH] siggen: Ensure taskhash mismatches don't override existing data

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

On Wed, Nov 2, 2016 at 8:07 AM, Richard Purdie <richard.purdie@
linuxfoundation.org> wrote:

> We recalculate the taskhash to ensure the version we have matches
> what we think it should be. When we write out a sigdata file, use
> the calculated value so that we don't overwrite any existing file.
> This leaves any original taskhash sigdata file intact to allow a
> debugging comparison.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>

Thanks, I ran into this trying to diagnose a mismatch with our automated
builds a while back. These patches should make those a lot easier to debug.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1436 bytes --]

Re: [PATCH] siggen: Ensure taskhash mismatches don't override existing data

[Richard Purdie]

On Wed, 2016-11-02 at 08:46 -0700, Christopher Larson wrote:
> 
> On Wed, Nov 2, 2016 at 8:07 AM, Richard Purdie <richard.purdie@linuxf
> oundation.org> wrote:
> > We recalculate the taskhash to ensure the version we have matches
> > what we think it should be. When we write out a sigdata file, use
> > the calculated value so that we don't overwrite any existing file.
> > This leaves any original taskhash sigdata file intact to allow a
> > debugging comparison.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > 
> Thanks, I ran into this trying to diagnose a mismatch with our
> automated builds a while back. These patches should make those a lot
> easier to debug.

Otavio gave me a reproducible test case which was nearly impossible to
debug with the codebase the way it was. I've been beating it up until
it actually gave good debug output which could be used to debug the
problem. Hopefully it works better than before, which wouldn't take
much. Still not 100% happy with it but it has to be better than it
was...

I'd love to find a way to always write out the sigbasedata but it uses
far too much time and slows down parsing too much :(.

Cheers,

Richard