[PATCH 0/1] Fix file checksum failure with broken symlinks

[PATCH 0/1] Fix file checksum failure with broken symlinks

[Paul Eggleton]

The following changes since commit 5fa0b3a19e7d0f40790f737abeddf499d53f1f6a:

  bitbake: implement idle timeout for xmlrpc server (2016-07-21 07:48:35 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib paule/bb-checksum-fix
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/bb-checksum-fix

Paul Eggleton (1):
  lib/bb/checksum: avoid exception on broken symlinks

 lib/bb/checksum.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

-- 
2.5.5

[PATCH 1/1] lib/bb/checksum: avoid exception on broken symlinks

[Paul Eggleton]

If using OE's externalsrc with a source tree that is not tracked by git
and contains broken symlinks, you can receive "TypeError: unorderable
types: NoneType() < str()" within the file checksum code due to:

 checksums.sort(key=operator.itemgetter(1))

Don't add files with no checksum to the checksums list in order to avoid
this.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 lib/bb/checksum.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/bb/checksum.py b/lib/bb/checksum.py
index be4ab68..8428920 100644
--- a/lib/bb/checksum.py
+++ b/lib/bb/checksum.py
@@ -120,13 +120,15 @@ class FileChecksumCache(MultiProcessCache):
                             checksums.extend(checksum_dir(f))
                     else:
                         checksum = checksum_file(f)
-                        checksums.append((f, checksum))
+                        if checksum:
+                            checksums.append((f, checksum))
             elif os.path.isdir(pth):
                 if not os.path.islink(pth):
                     checksums.extend(checksum_dir(pth))
             else:
                 checksum = checksum_file(pth)
-                checksums.append((pth, checksum))
+                if checksum:
+                    checksums.append((pth, checksum))
 
         checksums.sort(key=operator.itemgetter(1))
         return checksums
-- 
2.5.5

Re: [PATCH 1/1] lib/bb/checksum: avoid exception on broken symlinks

[Richard Purdie]

On Tue, 2016-07-26 at 15:36 +1200, Paul Eggleton wrote:
> If using OE's externalsrc with a source tree that is not tracked by
> git
> and contains broken symlinks, you can receive "TypeError: unorderable
> types: NoneType() < str()" within the file checksum code due to:
> 
>  checksums.sort(key=operator.itemgetter(1))
> 
> Don't add files with no checksum to the checksums list in order to
> avoid
> this.

In some of the cache code we have to note whether files were checked
for but didn't exist. This means that if they then do exist in a later
run, we know to invalidate the cache.

Does this change break that?

Cheers,

Richard

> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
>  lib/bb/checksum.py | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/bb/checksum.py b/lib/bb/checksum.py
> index be4ab68..8428920 100644
> --- a/lib/bb/checksum.py
> +++ b/lib/bb/checksum.py
> @@ -120,13 +120,15 @@ class FileChecksumCache(MultiProcessCache):
>                              checksums.extend(checksum_dir(f))
>                      else:
>                          checksum = checksum_file(f)
> -                        checksums.append((f, checksum))
> +                        if checksum:
> +                            checksums.append((f, checksum))
>              elif os.path.isdir(pth):
>                  if not os.path.islink(pth):
>                      checksums.extend(checksum_dir(pth))
>              else:
>                  checksum = checksum_file(pth)
> -                checksums.append((pth, checksum))
> +                if checksum:
> +                    checksums.append((pth, checksum))
>  
>          checksums.sort(key=operator.itemgetter(1))
>          return checksums
> -- 
> 2.5.5
> 

Re: [PATCH 1/1] lib/bb/checksum: avoid exception on broken symlinks

[Paul Eggleton]

On Tue, 26 Jul 2016 08:08:42 Richard Purdie wrote:
> On Tue, 2016-07-26 at 15:36 +1200, Paul Eggleton wrote:
> > If using OE's externalsrc with a source tree that is not tracked by
> > git and contains broken symlinks, you can receive "TypeError: unorderable
> > 
> > types: NoneType() < str()" within the file checksum code due to:
> >  checksums.sort(key=operator.itemgetter(1))
> > 
> > Don't add files with no checksum to the checksums list in order to
> > avoid this.
> 
> In some of the cache code we have to note whether files were checked
> for but didn't exist. This means that if they then do exist in a later
> run, we know to invalidate the cache.
> 
> Does this change break that?

Took me a couple of days to get back to this, but after double-checking both 
the code and the behaviour the answer is no. As you may recall the file 
checksum operation is broken into two parts:

1) Getting the list of files regardless of whether they exist or not, which is 
done at parse time and goes into the cache

2) Actually getting the checksum of each file that does exist, which is done 
when we calculate the task hashes. The hashes are also cached in their own 
separate cache file based on mtime.

#1 is where we handle files that may appear at some time in future. This patch 
changes the code for #2 and doesn't have any effect on #1.

We should definitely have an oe-selftest test for this though.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 1/1] lib/bb/checksum: avoid exception on broken symlinks

[Paul Eggleton]

On Thu, 28 Jul 2016 08:53:33 Paul Eggleton wrote:
> On Tue, 26 Jul 2016 08:08:42 Richard Purdie wrote:
> > On Tue, 2016-07-26 at 15:36 +1200, Paul Eggleton wrote:
> > > If using OE's externalsrc with a source tree that is not tracked by
> > > git and contains broken symlinks, you can receive "TypeError:
> > > unorderable
> > > 
> > > types: NoneType() < str()" within the file checksum code due to:
> > >  checksums.sort(key=operator.itemgetter(1))
> > > 
> > > Don't add files with no checksum to the checksums list in order to
> > > avoid this.
> > 
> > In some of the cache code we have to note whether files were checked
> > for but didn't exist. This means that if they then do exist in a later
> > run, we know to invalidate the cache.
> > 
> > Does this change break that?
> 
> Took me a couple of days to get back to this, but after double-checking both
> the code and the behaviour the answer is no. As you may recall the file
> checksum operation is broken into two parts:
> 
> 1) Getting the list of files regardless of whether they exist or not, which
> is done at parse time and goes into the cache
> 
> 2) Actually getting the checksum of each file that does exist, which is done
> when we calculate the task hashes. The hashes are also cached in their own
> separate cache file based on mtime.

Sorry, for clarity that should have read "The file checksums are also 
cached..."

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre