[PATCH v2] cache/siggen: Simplify passing basehash data into the cache

[PATCH v2] cache/siggen: Simplify passing basehash data into the cache

[Richard Purdie]

The basehash data is really internal bitbake data and passing an object
directly is more efficient than creating and then extracting variables.

This will match the format of other data we may optionally wish to
store in the cache so more to the more efficient method. Nothing I
can see is using this data today (and nothing should be).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cache.py  | 4 ++--
 lib/bb/siggen.py | 5 ++++-
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index cb6cf21727..e1214d6ad2 100644
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -28,7 +28,7 @@ import shutil
 
 logger = logging.getLogger("BitBake.Cache")
 
-__cache_version__ = "154"
+__cache_version__ = "155"
 
 def getCacheFile(path, filename, mc, data_hash):
     mcspec = ''
@@ -105,7 +105,7 @@ class CoreRecipeInfo(RecipeInfoCommon):
 
         self.tasks = metadata.getVar('__BBTASKS', False)
 
-        self.basetaskhashes = self.taskvar('BB_BASEHASH', self.tasks, metadata)
+        self.basetaskhashes = metadata.getVar('__siggen_basehashes', False)
         self.hashfilename = self.getvar('BB_HASHFILENAME', metadata)
 
         self.task_deps = metadata.getVar('_task_deps', False) or {'tasks': [], 'parents': {}}
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index 72b906c153..34b71d596a 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -247,8 +247,11 @@ class SignatureGeneratorBasic(SignatureGenerator):
         #for task in self.taskdeps[fn]:
         #    self.dump_sigtask(fn, task, d.getVar("STAMP"), False)
 
+        basehashes = {}
         for task in taskdeps:
-            d.setVar("BB_BASEHASH:task-%s" % task, self.basehash[fn + ":" + task])
+            basehashes[task] = self.basehash[fn + ":" + task]
+
+        d.setVar("__siggen_basehashes", basehashes)
 
     def postparsing_clean_cache(self):
         #
-- 
2.34.1

Re: [bitbake-devel] [PATCH v2] cache/siggen: Simplify passing basehash data into the cache

[Alexandre Belloni]

Hello Richard,

I believe this caused the following bitbake-selftest failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4467/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#builders/127/builds/559/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#builders/80/builds/4412/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#builders/87/builds/4493/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4434/steps/11/logs/stdio

On 01/12/2022 22:17:36+0000, Richard Purdie wrote:
> The basehash data is really internal bitbake data and passing an object
> directly is more efficient than creating and then extracting variables.
> 
> This will match the format of other data we may optionally wish to
> store in the cache so more to the more efficient method. Nothing I
> can see is using this data today (and nothing should be).
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  lib/bb/cache.py  | 4 ++--
>  lib/bb/siggen.py | 5 ++++-
>  2 files changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/lib/bb/cache.py b/lib/bb/cache.py
> index cb6cf21727..e1214d6ad2 100644
> --- a/lib/bb/cache.py
> +++ b/lib/bb/cache.py
> @@ -28,7 +28,7 @@ import shutil
>  
>  logger = logging.getLogger("BitBake.Cache")
>  
> -__cache_version__ = "154"
> +__cache_version__ = "155"
>  
>  def getCacheFile(path, filename, mc, data_hash):
>      mcspec = ''
> @@ -105,7 +105,7 @@ class CoreRecipeInfo(RecipeInfoCommon):
>  
>          self.tasks = metadata.getVar('__BBTASKS', False)
>  
> -        self.basetaskhashes = self.taskvar('BB_BASEHASH', self.tasks, metadata)
> +        self.basetaskhashes = metadata.getVar('__siggen_basehashes', False)
>          self.hashfilename = self.getvar('BB_HASHFILENAME', metadata)
>  
>          self.task_deps = metadata.getVar('_task_deps', False) or {'tasks': [], 'parents': {}}
> diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
> index 72b906c153..34b71d596a 100644
> --- a/lib/bb/siggen.py
> +++ b/lib/bb/siggen.py
> @@ -247,8 +247,11 @@ class SignatureGeneratorBasic(SignatureGenerator):
>          #for task in self.taskdeps[fn]:
>          #    self.dump_sigtask(fn, task, d.getVar("STAMP"), False)
>  
> +        basehashes = {}
>          for task in taskdeps:
> -            d.setVar("BB_BASEHASH:task-%s" % task, self.basehash[fn + ":" + task])
> +            basehashes[task] = self.basehash[fn + ":" + task]
> +
> +        d.setVar("__siggen_basehashes", basehashes)
>  
>      def postparsing_clean_cache(self):
>          #
> -- 
> 2.34.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14130): https://lists.openembedded.org/g/bitbake-devel/message/14130
> Mute This Topic: https://lists.openembedded.org/mt/95394525/3617179
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[PATCH v2] cache/siggen: Simplify passing basehash data into the cache

[Richard Purdie]

The basehash data is really internal bitbake data and passing an object
directly is more efficient than creating and then extracting variables.

This will match the format of other data we may optionally wish to
store in the cache so more to the more efficient method. Nothing I
can see is using this data today (and nothing should be).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cache.py  | 4 ++--
 lib/bb/siggen.py | 5 ++++-
 2 files changed, 6 insertions(+), 3 deletions(-)

v2: Add default for the cache entry so selftest works

diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index cb6cf21727..e117fe56cb 100644
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -28,7 +28,7 @@ import shutil
 
 logger = logging.getLogger("BitBake.Cache")
 
-__cache_version__ = "154"
+__cache_version__ = "155"
 
 def getCacheFile(path, filename, mc, data_hash):
     mcspec = ''
@@ -105,7 +105,7 @@ class CoreRecipeInfo(RecipeInfoCommon):
 
         self.tasks = metadata.getVar('__BBTASKS', False)
 
-        self.basetaskhashes = self.taskvar('BB_BASEHASH', self.tasks, metadata)
+        self.basetaskhashes = metadata.getVar('__siggen_basehashes', False) or {}
         self.hashfilename = self.getvar('BB_HASHFILENAME', metadata)
 
         self.task_deps = metadata.getVar('_task_deps', False) or {'tasks': [], 'parents': {}}
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index 72b906c153..34b71d596a 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -247,8 +247,11 @@ class SignatureGeneratorBasic(SignatureGenerator):
         #for task in self.taskdeps[fn]:
         #    self.dump_sigtask(fn, task, d.getVar("STAMP"), False)
 
+        basehashes = {}
         for task in taskdeps:
-            d.setVar("BB_BASEHASH:task-%s" % task, self.basehash[fn + ":" + task])
+            basehashes[task] = self.basehash[fn + ":" + task]
+
+        d.setVar("__siggen_basehashes", basehashes)
 
     def postparsing_clean_cache(self):
         #
-- 
2.34.1

Re: [bitbake-devel] [PATCH v2] cache/siggen: Simplify passing basehash data into the cache

[Joshua Watt]

On 12/5/22 09:18, Richard Purdie wrote:
> The basehash data is really internal bitbake data and passing an object
> directly is more efficient than creating and then extracting variables.
>
> This will match the format of other data we may optionally wish to
> store in the cache so more to the more efficient method. Nothing I
> can see is using this data today (and nothing should be).
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>   lib/bb/cache.py  | 4 ++--
>   lib/bb/siggen.py | 5 ++++-
>   2 files changed, 6 insertions(+), 3 deletions(-)
>
> v2: Add default for the cache entry so selftest works
>
> diff --git a/lib/bb/cache.py b/lib/bb/cache.py
> index cb6cf21727..e117fe56cb 100644
> --- a/lib/bb/cache.py
> +++ b/lib/bb/cache.py
> @@ -28,7 +28,7 @@ import shutil
>   
>   logger = logging.getLogger("BitBake.Cache")
>   
> -__cache_version__ = "154"
> +__cache_version__ = "155"
>   
>   def getCacheFile(path, filename, mc, data_hash):
>       mcspec = ''
> @@ -105,7 +105,7 @@ class CoreRecipeInfo(RecipeInfoCommon):
>   
>           self.tasks = metadata.getVar('__BBTASKS', False)
>   
> -        self.basetaskhashes = self.taskvar('BB_BASEHASH', self.tasks, metadata)
> +        self.basetaskhashes = metadata.getVar('__siggen_basehashes', False) or {}
>           self.hashfilename = self.getvar('BB_HASHFILENAME', metadata)
>   
>           self.task_deps = metadata.getVar('_task_deps', False) or {'tasks': [], 'parents': {}}
> diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
> index 72b906c153..34b71d596a 100644
> --- a/lib/bb/siggen.py
> +++ b/lib/bb/siggen.py
> @@ -247,8 +247,11 @@ class SignatureGeneratorBasic(SignatureGenerator):
>           #for task in self.taskdeps[fn]:
>           #    self.dump_sigtask(fn, task, d.getVar("STAMP"), False)
>   
> +        basehashes = {}
>           for task in taskdeps:
> -            d.setVar("BB_BASEHASH:task-%s" % task, self.basehash[fn + ":" + task])
> +            basehashes[task] = self.basehash[fn + ":" + task]

In my testing, dictionary comprehension is the fastest method to 
populate a dictionary, so if this is a critical path, you could optimize 
it as:

  basehash = {task: self.basehash["%s:%s" % (fn, task)] for task in 
taskdeps}

I can't remember if that level of micro optimization helps here though; 
if not feel free to ignore me :)

> +
> +        d.setVar("__siggen_basehashes", basehashes)
>   
>       def postparsing_clean_cache(self):
>           #
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14134): https://lists.openembedded.org/g/bitbake-devel/message/14134
> Mute This Topic: https://lists.openembedded.org/mt/95394525/3616693
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [JPEWhacker@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>