[PATCH] siggen: compare runtaskdeps dictionary even when they have the same size

[PATCH] siggen: compare runtaskdeps dictionary even when they have the same size

[Martin Jansa]

* otherwise it reports lots of changed checksums just because it compares different tasks

  notice linux-libc-headers_3.1.bb.do_package_write:
  Task dependency hash changed from 42acced29debf54d598802474c5e9cbb to f438a54f995df77620d0727d6f4b4ce5 (for linux-libc-headers_3.1.bb.do_package_write and linux-gta04_git.bb.do_deploy)
  Task dependency hash changed from 61f8babe1d10c6e7fb1423112bb04e1e to 1b3e21ff106ecfcb7ddf76a1e29537bb (for linux-nokia900-meego_git.bb.do_deploy and linux-gta04_git.bb.do_package_write)
  Task dependency hash changed from 512f9d6686d760b318d8b11c8b589226 to 42acced29debf54d598802474c5e9cbb (for linux-nokia900-meego_git.bb.do_package_write and linux-libc-headers_3.1.bb.do_package_write)
  Task dependency hash changed from 153e91dfd1d2053fda7b98cc08d4b802 to 92a293bdd8ed234932b87a66025038c5 (for systemd-serialgetty.bb.do_package_write and systemd-serialgetty.bb.do_package_write)
  Task dependency hash changed from 730abebf9954794bb440c2f3239f79fe to 413eaebaff27a2fd16f5cf68c1f4ff17 (for systemd_git.bb.do_package_write and systemd_git.bb.do_package_write)

  with this patch:
  Dependency on task linux-nokia900-meego_git.bb.do_package_write was added
  Dependency on task linux-nokia900-meego_git.bb.do_deploy was added
  Dependency on task linux-gta04_git.bb.do_deploy was removed
  Dependency on task linux-gta04_git.bb.do_package_write was removed
  Hash for dependent task systemd_git.bb.do_package_write changed from 730abebf9954794bb440c2f3239f79fe to 413eaebaff27a2fd16f5cf68c1f4ff17
  Hash for dependent task systemd-serialgetty.bb.do_package_write changed from 153e91dfd1d2053fda7b98cc08d4b802 to 92a293bdd8ed234932b87a66025038c5

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 lib/bb/siggen.py |   33 ++++++++++++---------------------
 1 files changed, 12 insertions(+), 21 deletions(-)

diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index f31d66a..ffe85dd 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -292,27 +292,18 @@ def compare_sigfiles(a, b):
             print "Variable %s value changed from %s to %s" % (dep, a_data['varvals'][dep], b_data['varvals'][dep])
 
     if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data:
-        if len(a_data['runtaskdeps']) != len(b_data['runtaskdeps']):
-            a = clean_basepaths(a_data['runtaskhashes'])
-            b = clean_basepaths(b_data['runtaskhashes'])
-            changed, added, removed = dict_diff(a, b)
-            if added:
-                for dep in added:
-                    print "Dependency on task %s was added" % (dep)
-            if removed:
-                for dep in removed:
-                    print "Dependency on task %s was removed" % (dep)
-            if changed:
-                for dep in changed:
-                    print "Hash for dependent task %s changed from %s to %s" % (dep, a[dep], b[dep])
-        else:
-            for i in range(len(a_data['runtaskdeps'])):
-                aent = a_data['runtaskdeps'][i]
-                bent = b_data['runtaskdeps'][i]
-                aname = clean_basepath(aent)
-                bname = clean_basepath(bent)
-                if a_data['runtaskhashes'][aent] != b_data['runtaskhashes'][bent]:
-                    print "Task dependency hash changed from %s to %s (for %s and %s)" % (a_data['runtaskhashes'][aent], b_data['runtaskhashes'][bent], aname, bname)
+        a = clean_basepaths(a_data['runtaskhashes'])
+        b = clean_basepaths(b_data['runtaskhashes'])
+        changed, added, removed = dict_diff(a, b)
+        if added:
+            for dep in added:
+                print "Dependency on task %s was added" % (dep)
+        if removed:
+            for dep in removed:
+                print "Dependency on task %s was removed" % (dep)
+        if changed:
+            for dep in changed:
+                print "Hash for dependent task %s changed from %s to %s" % (dep, a[dep], b[dep])
 
 def dump_sigfile(a):
     p1 = pickle.Unpickler(file(a, "rb"))
-- 
1.7.8.4

Re: [PATCH] siggen: compare runtaskdeps dictionary even when they have the same size

[Richard Purdie]

On Wed, 2012-02-22 at 14:03 +0100, Martin Jansa wrote:
> * otherwise it reports lots of changed checksums just because it compares different tasks
> 
>   notice linux-libc-headers_3.1.bb.do_package_write:
>   Task dependency hash changed from 42acced29debf54d598802474c5e9cbb to f438a54f995df77620d0727d6f4b4ce5 (for linux-libc-headers_3.1.bb.do_package_write and linux-gta04_git.bb.do_deploy)
>   Task dependency hash changed from 61f8babe1d10c6e7fb1423112bb04e1e to 1b3e21ff106ecfcb7ddf76a1e29537bb (for linux-nokia900-meego_git.bb.do_deploy and linux-gta04_git.bb.do_package_write)
>   Task dependency hash changed from 512f9d6686d760b318d8b11c8b589226 to 42acced29debf54d598802474c5e9cbb (for linux-nokia900-meego_git.bb.do_package_write and linux-libc-headers_3.1.bb.do_package_write)
>   Task dependency hash changed from 153e91dfd1d2053fda7b98cc08d4b802 to 92a293bdd8ed234932b87a66025038c5 (for systemd-serialgetty.bb.do_package_write and systemd-serialgetty.bb.do_package_write)
>   Task dependency hash changed from 730abebf9954794bb440c2f3239f79fe to 413eaebaff27a2fd16f5cf68c1f4ff17 (for systemd_git.bb.do_package_write and systemd_git.bb.do_package_write)
> 
>   with this patch:
>   Dependency on task linux-nokia900-meego_git.bb.do_package_write was added
>   Dependency on task linux-nokia900-meego_git.bb.do_deploy was added
>   Dependency on task linux-gta04_git.bb.do_deploy was removed
>   Dependency on task linux-gta04_git.bb.do_package_write was removed
>   Hash for dependent task systemd_git.bb.do_package_write changed from 730abebf9954794bb440c2f3239f79fe to 413eaebaff27a2fd16f5cf68c1f4ff17
>   Hash for dependent task systemd-serialgetty.bb.do_package_write changed from 153e91dfd1d2053fda7b98cc08d4b802 to 92a293bdd8ed234932b87a66025038c5
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>

This basically reverts
http://git.openembedded.org/bitbake/commit/?id=7da7dff83ed765c9cde1d7f91ee1b65e49520481
although I guess the elif in the original is no longer present (I can't
decide if that is an issue or not).

The problem is bitbake only cares about the hashes themselves, it
doesn't care about the name attached to them. The name only exists to
aid debugging.

We were therefore seeing bitbake-diffsigs report differences between
sstate files when there was no difference as far as bitbake was
concerned, the hashes just had different tasknames associated with them.
I strongly disliked having bitbake-diffsigs report a difference where
bitbake would treat them the as the same thing.

I'd therefore suggest what the code should do is:

a) Discount duplicates based on hash value regardless of whether the 
   tasknames match or not.
b) Print the differences as per your patch on the data after the 
   removal of the above.

So yes, the code needs to be improved to firstly discount duplicates,
then give a better summary but the patch needs tweaking.

Cheers,

Richard

[PATCH] siggen: compare runtaskdeps dictionary even when they have the same size

[Martin Jansa]

* otherwise it reports lots of changed checksums just because it compares different tasks

  notice linux-libc-headers_3.1.bb.do_package_write:
  Task dependency hash changed from 42acced29debf54d598802474c5e9cbb to f438a54f995df77620d0727d6f4b4ce5 (for linux-libc-headers_3.1.bb.do_package_write and linux-gta04_git.bb.do_deploy)
  Task dependency hash changed from 61f8babe1d10c6e7fb1423112bb04e1e to 1b3e21ff106ecfcb7ddf76a1e29537bb (for linux-nokia900-meego_git.bb.do_deploy and linux-gta04_git.bb.do_package_write)
  Task dependency hash changed from 512f9d6686d760b318d8b11c8b589226 to 42acced29debf54d598802474c5e9cbb (for linux-nokia900-meego_git.bb.do_package_write and linux-libc-headers_3.1.bb.do_package_write)
  Task dependency hash changed from 153e91dfd1d2053fda7b98cc08d4b802 to 92a293bdd8ed234932b87a66025038c5 (for systemd-serialgetty.bb.do_package_write and systemd-serialgetty.bb.do_package_write)
  Task dependency hash changed from 730abebf9954794bb440c2f3239f79fe to 413eaebaff27a2fd16f5cf68c1f4ff17 (for systemd_git.bb.do_package_write and systemd_git.bb.do_package_write)

  with this patch:
  Dependency on task linux-nokia900-meego_git.bb.do_package_write was added
  Dependency on task linux-nokia900-meego_git.bb.do_deploy was added
  Dependency on task linux-gta04_git.bb.do_deploy was removed
  Dependency on task linux-gta04_git.bb.do_package_write was removed
  Hash for dependent task systemd_git.bb.do_package_write changed from 730abebf9954794bb440c2f3239f79fe to 413eaebaff27a2fd16f5cf68c1f4ff17
  Hash for dependent task systemd-serialgetty.bb.do_package_write changed from 153e91dfd1d2053fda7b98cc08d4b802 to 92a293bdd8ed234932b87a66025038c5

* added test if there is different task with same hash then we don't
  show it as added/removed dependency, because bitbake doesn't care

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 lib/bb/siggen.py |   47 ++++++++++++++++++++++++++---------------------
 1 files changed, 26 insertions(+), 21 deletions(-)

diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index f31d66a..5621e1b 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -292,27 +292,32 @@ def compare_sigfiles(a, b):
             print "Variable %s value changed from %s to %s" % (dep, a_data['varvals'][dep], b_data['varvals'][dep])
 
     if 'runtaskhashes' in a_data and 'runtaskhashes' in b_data:
-        if len(a_data['runtaskdeps']) != len(b_data['runtaskdeps']):
-            a = clean_basepaths(a_data['runtaskhashes'])
-            b = clean_basepaths(b_data['runtaskhashes'])
-            changed, added, removed = dict_diff(a, b)
-            if added:
-                for dep in added:
-                    print "Dependency on task %s was added" % (dep)
-            if removed:
-                for dep in removed:
-                    print "Dependency on task %s was removed" % (dep)
-            if changed:
-                for dep in changed:
-                    print "Hash for dependent task %s changed from %s to %s" % (dep, a[dep], b[dep])
-        else:
-            for i in range(len(a_data['runtaskdeps'])):
-                aent = a_data['runtaskdeps'][i]
-                bent = b_data['runtaskdeps'][i]
-                aname = clean_basepath(aent)
-                bname = clean_basepath(bent)
-                if a_data['runtaskhashes'][aent] != b_data['runtaskhashes'][bent]:
-                    print "Task dependency hash changed from %s to %s (for %s and %s)" % (a_data['runtaskhashes'][aent], b_data['runtaskhashes'][bent], aname, bname)
+        a = clean_basepaths(a_data['runtaskhashes'])
+        b = clean_basepaths(b_data['runtaskhashes'])
+        changed, added, removed = dict_diff(a, b)
+        if added:
+            for dep in added:
+	        bdep_found = False
+	        if removed:
+		    for bdep in removed:
+		    	if a[dep] == b[bdep]:
+			   #print "Dependency on task %s was replaced by %s with same hash" % (dep, bdep)
+			   bdep_found = True
+		if not bdep_found:
+                    print "Dependency on task %s was added with hash %s" % (dep, a[dep])
+        if removed:
+            for dep in removed:
+	        adep_found = False
+	        if added:
+		    for adep in added:
+		    	if a[adep] == b[dep]:
+			   #print "Dependency on task %s was replaced by %s with same hash" % (adep, dep)
+			   adep_found = True
+		if not adep_found:
+                   print "Dependency on task %s was removed with hash %s" % (dep, b[dep])
+        if changed:
+            for dep in changed:
+                print "Hash for dependent task %s changed from %s to %s" % (dep, a[dep], b[dep])
 
 def dump_sigfile(a):
     p1 = pickle.Unpickler(file(a, "rb"))
-- 
1.7.8.4

Re: [PATCH] siggen: compare runtaskdeps dictionary even when they have the same size

[Richard Purdie]

On Wed, 2012-02-22 at 16:01 +0100, Martin Jansa wrote:
> * otherwise it reports lots of changed checksums just because it compares different tasks
> 
>   notice linux-libc-headers_3.1.bb.do_package_write:
>   Task dependency hash changed from 42acced29debf54d598802474c5e9cbb to f438a54f995df77620d0727d6f4b4ce5 (for linux-libc-headers_3.1.bb.do_package_write and linux-gta04_git.bb.do_deploy)
>   Task dependency hash changed from 61f8babe1d10c6e7fb1423112bb04e1e to 1b3e21ff106ecfcb7ddf76a1e29537bb (for linux-nokia900-meego_git.bb.do_deploy and linux-gta04_git.bb.do_package_write)
>   Task dependency hash changed from 512f9d6686d760b318d8b11c8b589226 to 42acced29debf54d598802474c5e9cbb (for linux-nokia900-meego_git.bb.do_package_write and linux-libc-headers_3.1.bb.do_package_write)
>   Task dependency hash changed from 153e91dfd1d2053fda7b98cc08d4b802 to 92a293bdd8ed234932b87a66025038c5 (for systemd-serialgetty.bb.do_package_write and systemd-serialgetty.bb.do_package_write)
>   Task dependency hash changed from 730abebf9954794bb440c2f3239f79fe to 413eaebaff27a2fd16f5cf68c1f4ff17 (for systemd_git.bb.do_package_write and systemd_git.bb.do_package_write)
> 
>   with this patch:
>   Dependency on task linux-nokia900-meego_git.bb.do_package_write was added
>   Dependency on task linux-nokia900-meego_git.bb.do_deploy was added
>   Dependency on task linux-gta04_git.bb.do_deploy was removed
>   Dependency on task linux-gta04_git.bb.do_package_write was removed
>   Hash for dependent task systemd_git.bb.do_package_write changed from 730abebf9954794bb440c2f3239f79fe to 413eaebaff27a2fd16f5cf68c1f4ff17
>   Hash for dependent task systemd-serialgetty.bb.do_package_write changed from 153e91dfd1d2053fda7b98cc08d4b802 to 92a293bdd8ed234932b87a66025038c5
> 
> * added test if there is different task with same hash then we don't
>   show it as added/removed dependency, because bitbake doesn't care
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  lib/bb/siggen.py |   47 ++++++++++++++++++++++++++---------------------
>  1 files changed, 26 insertions(+), 21 deletions(-)

Merged to master, thanks.

Richard