Re: [PATCH] Checksums for local files now stored using partial recipe path

[Paul Eggleton <paul.eggleton@linux.intel.com>]

Hi Jate,

On Wednesday 19 June 2013 11:08:10 Jate Sujjavanich wrote:
> This allows sstate-cache to be shared between builds in different
> directories.
> 
> Differences in the full path were triggering a false positive when there
> were actually no changes.
> 
> Signed-off-by: Jate Sujjavanich <jate.sujjavanich@myfuelmaster.com>
> ---
>  bitbake/lib/bb/fetch2/__init__.py |   14 +++++++++-----
>  bitbake/lib/bb/siggen.py          |    3 ++-
>  2 files changed, 11 insertions(+), 6 deletions(-)
> 
> diff --git a/bitbake/lib/bb/fetch2/__init__.py
> b/bitbake/lib/bb/fetch2/__init__.py index dd1cc93..7ab44d7 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -900,8 +900,7 @@ def get_checksum_file_list(d):
> 
>      return " ".join(filelist)
> 
> -
> -def get_file_checksums(filelist, pn):
> +def get_file_checksums(filelist, pn, topdir):
>      """Get a list of the checksums for a list of local files
> 
>      Returns the checksums for a list of local files, caching the results as
> @@ -917,7 +916,12 @@ def get_file_checksums(filelist, pn): bb.warn("Unable
> to get checksum for %s SRC_URI entry %s: %s" % (pn, os.path.basename(f),
> e)) return None
>          return checksum
> +
> +    (recipe_root, _) = os.path.split(topdir)
> 
> +    def remove_recipe_parent(data):
> +        return data.replace(recipe_root, '').strip('/')
> +
>      checksums = []
>      for pth in filelist.split():
>          checksum = None
> @@ -927,7 +931,7 @@ def get_file_checksums(filelist, pn):
>              for f in glob.glob(pth):
>                  checksum = checksum_file(f)
>                  if checksum:
> -                    checksums.append((f, checksum))
> +                    checksums.append((remove_recipe_parent(f),
> + checksum))
>          elif os.path.isdir(pth):
>              # Handle directories
>              for root, dirs, files in os.walk(pth):
> @@ -935,12 +939,12 @@ def get_file_checksums(filelist, pn):
>                      fullpth = os.path.join(root, name)
>                      checksum = checksum_file(fullpth)
>                      if checksum:
> -                        checksums.append((fullpth, checksum))
> +
> + checksums.append((remove_recipe_parent(fullpth), checksum))
>          else:
>              checksum = checksum_file(pth)
> 
>          if checksum:
> -            checksums.append((pth, checksum))
> +            checksums.append((remove_recipe_parent(pth), checksum))
> 
>      checksums.sort(key=operator.itemgetter(1))
>      return checksums
> diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index
> 8861337..c64acfe 100644 --- a/bitbake/lib/bb/siggen.py
> +++ b/bitbake/lib/bb/siggen.py
> @@ -74,6 +74,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>          self.pkgnameextract = re.compile("(?P<fn>.*)\..*")
>          self.basewhitelist = set((data.getVar("BB_HASHBASE_WHITELIST",
> True) or "").split()) self.taskwhitelist = None
> +        self.topdir = data.getVar("TOPDIR", True)
>          self.init_rundepcheck(data)
> 
>      def init_rundepcheck(self, data):
> @@ -187,7 +188,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>              self.runtaskdeps[k].append(dep)
> 
>          if task in dataCache.file_checksums[fn]:
> -            checksums =
> bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task],
> recipename) +            checksums =
> + bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task],
> + recipename, self.topdir)
>              for (f,cs) in checksums:
>                  self.file_checksum_values[k][f] = cs
>                  data = data + cs

Good catch! The only thing is, this will not help for files within different 
layers which may not be underneath TOPDIR; I think we'll need a function that 
determines which layer the file is under (longest path match from 
data.getVar('BBLAYERS', True).split()) and then take that path off the 
beginning.

Additionally, this is a patch against bitbake so it will need to go to the 
bitbake-devel@lists.openembedded.org mailing list.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre