[PATCH] siggen.py: Fix diffsigs output for filename comparisions

[PATCH] siggen.py: Fix diffsigs output for filename comparisions

[Richard Purdie]

When comparing sig files, if the recipe locations had changed, the
dependent tasks list would show as changed even if the actual hash
had not changed. This updates the code to only compare the base part
of the pathnames.

It also tweaks some of the output to add newlines to aid comparing
two lists of variables as it makes the location of the difference
clearer.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index 4ccfc7d..217f29b 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -236,6 +236,16 @@ def compare_sigfiles(a, b):
         removed = sb - sa
         return changed, added, removed
 
+    def clean_basepaths(a):
+        b = {}
+        for x in a:
+            if x.startswith("virtual:"):
+                y = x.rsplit(":", 1)[0] + x.rsplit("/", 1)[1]
+            else:
+                y = x.rsplit("/", 1)[1]
+            b[y] = a[x]
+        return b
+
     if 'basewhitelist' in a_data and a_data['basewhitelist'] != b_data['basewhitelist']:
         print "basewhitelist changed from %s to %s" % (a_data['basewhitelist'], b_data['basewhitelist'])
 
@@ -243,7 +253,7 @@ def compare_sigfiles(a, b):
         print "taskwhitelist changed from %s to %s" % (a_data['taskwhitelist'], b_data['taskwhitelist'])
 
     if a_data['taskdeps'] != b_data['taskdeps']:
-        print "Task dependencies changed from %s to %s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps']))
+        print "Task dependencies changed from:\n%s\nto:\n%s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps']))
 
     if a_data['basehash'] != b_data['basehash']:
         print "basehash changed from %s to %s" % (a_data['basehash'], b_data['basehash'])
@@ -266,7 +276,9 @@ def compare_sigfiles(a, b):
             print "Variable %s value changed from %s to %s" % (dep, a_data['varvals'][dep], b_data['varvals'][dep])
 
     if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data:
-        changed, added, removed = dict_diff(a_data['runtaskhashes'], b_data['runtaskhashes'])
+        a = clean_basepaths(a_data['runtaskhashes'])
+        b = clean_basepaths(b_data['runtaskhashes'])
+        changed, added, removed = dict_diff(a, b)
         if added:
             for dep in added:
                 print "Dependency on task %s was added" % (dep)
@@ -275,7 +287,7 @@ def compare_sigfiles(a, b):
                 print "Dependency on task %s was removed" % (dep)
         if changed:
             for dep in changed:
-                print "Hash for dependent task %s changed from %s to %s" % (dep, a_data['runtaskhashes'][dep], b_data['runtaskhashes'][dep])
+                print "Hash for dependent task %s changed from %s to %s" % (dep, a[dep], b[dep])
     elif 'runtaskdeps' in a_data and 'runtaskdeps' in b_data and sorted(a_data['runtaskdeps']) != sorted(b_data['runtaskdeps']):
         print "Tasks this task depends on changed from %s to %s" % (sorted(a_data['runtaskdeps']), sorted(b_data['runtaskdeps']))
 

Re: [PATCH] siggen.py: Fix diffsigs output for filename comparisions

[McClintock Matthew-B29882]

On Thu, Nov 17, 2011 at 8:02 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
> index 4ccfc7d..217f29b 100644
> --- a/bitbake/lib/bb/siggen.py
> +++ b/bitbake/lib/bb/siggen.py
> @@ -236,6 +236,16 @@ def compare_sigfiles(a, b):
>         removed = sb - sa
>         return changed, added, removed
>
> +    def clean_basepaths(a):
> +        b = {}
> +        for x in a:
> +            if x.startswith("virtual:"):
> +                y = x.rsplit(":", 1)[0] + x.rsplit("/", 1)[1]
> +            else:
> +                y = x.rsplit("/", 1)[1]
> +            b[y] = a[x]
> +        return b
> +

So you are stripping off the full path leading up to the recipe name?
I added some stuff to strip off COREBASE at some point. Does this work
with layers in any location? I would think so.

>     if 'basewhitelist' in a_data and a_data['basewhitelist'] != b_data['basewhitelist']:
>         print "basewhitelist changed from %s to %s" % (a_data['basewhitelist'], b_data['basewhitelist'])
>
> @@ -243,7 +253,7 @@ def compare_sigfiles(a, b):
>         print "taskwhitelist changed from %s to %s" % (a_data['taskwhitelist'], b_data['taskwhitelist'])
>
>     if a_data['taskdeps'] != b_data['taskdeps']:
> -        print "Task dependencies changed from %s to %s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps']))
> +        print "Task dependencies changed from:\n%s\nto:\n%s" % (sorted(a_data['taskdeps']), sorted(b_data['taskdeps']))
>
>     if a_data['basehash'] != b_data['basehash']:
>         print "basehash changed from %s to %s" % (a_data['basehash'], b_data['basehash'])
> @@ -266,7 +276,9 @@ def compare_sigfiles(a, b):
>             print "Variable %s value changed from %s to %s" % (dep, a_data['varvals'][dep], b_data['varvals'][dep])
>
>     if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data:
> -        changed, added, removed = dict_diff(a_data['runtaskhashes'], b_data['runtaskhashes'])
> +        a = clean_basepaths(a_data['runtaskhashes'])
> +        b = clean_basepaths(b_data['runtaskhashes'])
> +        changed, added, removed = dict_diff(a, b)
>         if added:
>             for dep in added:
>                 print "Dependency on task %s was added" % (dep)
> @@ -275,7 +287,7 @@ def compare_sigfiles(a, b):
>                 print "Dependency on task %s was removed" % (dep)
>         if changed:
>             for dep in changed:
> -                print "Hash for dependent task %s changed from %s to %s" % (dep, a_data['runtaskhashes'][dep], b_data['runtaskhashes'][dep])
> +                print "Hash for dependent task %s changed from %s to %s" % (dep, a[dep], b[dep])
>     elif 'runtaskdeps' in a_data and 'runtaskdeps' in b_data and sorted(a_data['runtaskdeps']) != sorted(b_data['runtaskdeps']):
>         print "Tasks this task depends on changed from %s to %s" % (sorted(a_data['runtaskdeps']), sorted(b_data['runtaskdeps']))
>

So now we get shown the tashhash is different even when the
paths/filenames are different? This is helpful for sure - took me a
bit to get to this realization.

Some other bitbake-diffsig improvements/comments:

1) For showing the difference between two sets, it would be helpful to
also show the symmetrical_difference to help see what is different
more easily.

2) For taskdeps, if the task depends is part of the same recipe, it
would be helpful to show that. For example if strace.bb.do_package
depends on strace.bb.do_install, can't we just show that instead?

3) Continuing on #2 it would be nice to import all sigs from a folder
and traverse down the task dependencies, and then even show whats
difference between two sets of sstate-cache. For example:

strace.bb.do_package -> strace.bb.do_install -> eglibc.bb.do_install
                                             |-> foo.bb.do_install
                                             |-> bar.bb.do_install

Showing the hash above as well. Then somehow do a diff between the two
trees... but even just showing this tree is a good first step. Maybe
there is a way to dump this without even involving sstate-cache but
just via bitbake?

Maybe there are easy ways to do these things already? Feedback welcome =)

-M