[PATCH] fetch2: Fix bug in file checksum generation

[PATCH] fetch2: Fix bug in file checksum generation

[Richard Purdie]

For a while its been puzzling me why connman-gnome rebuilds as often as it
does. It turns out you can trigger this with a new checkout of the metadata.

The SRC_URI that is causing the problems is:

SRC_URI = "file://images/*"

and rather oddly the results in checksums for a file "." being added to
the tree, e.g.:

('.', 'ab48a68186f0e0f277c21ef4cb390b4b')

The problem is that when iterating files lists, the checksum variable can
become set yet we don't break the out from the for loop, which leads to
odd (and non-deterministic) entries being added into the file checksum list.
The exact item added probably depends on the order of items on the disk.

Before this change, bitbake-diffsigs on connman-gnome:do_fetch would report:

This task depends on the checksums of files: [
('connman-signal-03.png', 'f6c16aee57b37b73793a2f1dea433ffa'),
('connman-signal-02.png', 'ad0cd22710c097d8174121fc1023c3be'),
('connman-signal-01.png', '8842bd83d2fa9ba56480df34c727c629'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('connman-signal-05.png', '808589e7e8d502b44c7b007e9e68d48c'),
('connman-signal-04.png', 'ab48a68186f0e0f277c21ef4cb390b4b'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('.', 'ab48a68186f0e0f277c21ef4cb390b4b')]

Afterwards:

This task depends on the checksums of files: [
('connman-signal-03.png', 'f6c16aee57b37b73793a2f1dea433ffa'),
('connman-signal-02.png', 'ad0cd22710c097d8174121fc1023c3be'),
('connman-signal-01.png', '8842bd83d2fa9ba56480df34c727c629'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('connman-signal-05.png', '808589e7e8d502b44c7b007e9e68d48c'),
('connman-signal-04.png', 'ab48a68186f0e0f277c21ef4cb390b4b'),
('null_check_for_ipv4_config.patch', 'a23271e41c9fe81551244d875106af10'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85'),
('0001-Removed-icon-from-connman-gnome-about-applet.patch', 'e2d8269357c1e8c84291606da24eea85')]

which is correct and deterministic without the "." entry.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 4335e7a..8e5342f 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -969,6 +969,7 @@ def get_file_checksums(filelist, pn):
                 checksum = checksum_file(f)
                 if checksum:
                     checksums.append((f, checksum))
+            continue
         elif os.path.isdir(pth):
             # Handle directories
             for root, dirs, files in os.walk(pth):
@@ -977,6 +978,7 @@ def get_file_checksums(filelist, pn):
                     checksum = checksum_file(fullpth)
                     if checksum:
                         checksums.append((fullpth, checksum))
+            continue
         else:
             checksum = checksum_file(pth)
 

Re: [PATCH] fetch2: Fix bug in file checksum generation

[Jacob Kroon]

[-- Attachment #1: Type: text/plain, Size: 1547 bytes --]

Hi Richard,

diff --git a/bitbake/lib/bb/fetch2/__init__.py
b/bitbake/lib/bb/fetch2/__init__.py

> index 4335e7a..8e5342f 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -969,6 +969,7 @@ def get_file_checksums(filelist, pn):
>                  checksum = checksum_file(f)
>                  if checksum:
>                      checksums.append((f, checksum))
> +            continue
>          elif os.path.isdir(pth):
>              # Handle directories
>              for root, dirs, files in os.walk(pth):
> @@ -977,6 +978,7 @@ def get_file_checksums(filelist, pn):
>                      checksum = checksum_file(fullpth)
>                      if checksum:
>                          checksums.append((fullpth, checksum))
> +            continue
>          else:
>              checksum = checksum_file(pth)
>

Perhaps it would be cleaner to do

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 4335e7a..03ab234 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -979,9 +979,8 @@ def get_file_checksums(filelist, pn):
                         checksums.append((fullpth, checksum))
         else:
             checksum = checksum_file(pth)
-
-        if checksum:
-            checksums.append((pth, checksum))
+            if checksum:
+                checksums.append((pth, checksum))

     checksums.sort(key=operator.itemgetter(1))
     return checksums

or am I missing something here ?

/Jacob

[-- Attachment #2: Type: text/html, Size: 1964 bytes --]

Re: [PATCH] fetch2: Fix bug in file checksum generation

[Richard Purdie]

On Sun, 2014-04-06 at 12:39 +0200, Jacob Kroon wrote:
> Perhaps it would be cleaner to do 
>
> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index 4335e7a..03ab234 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -979,9 +979,8 @@ def get_file_checksums(filelist, pn):
>                          checksums.append((fullpth, checksum))
>          else:
>              checksum = checksum_file(pth)
> -
> -        if checksum:
> -            checksums.append((pth, checksum))
> +            if checksum:
> +                checksums.append((pth, checksum))
>  
>      checksums.sort(key=operator.itemgetter(1))
>      return checksums
> 
> 
> or am I missing something here ?

The other patch indicates more specifically what the problem was, this
is probably the cleaner code. They amount to the same thing...

Cheers,

Richard