[bitbake][kirkstone][2.0][PATCH 0/1] Patch review

[bitbake][kirkstone][2.0][PATCH 0/1] Patch review

[Steve Sakoman]

Please review this set of patches for kirkstone/2.0 and have comments back by
end of day Tuesday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5185

The following changes since commit 2802adb572eb73a3eb2725a74a9bbdaafc543fa7:

  fetch/git: Fix local clone url to make it work with repo (2023-03-31 04:28:22 -1000)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.0-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut

Frank de Brabander (1):
  bin/utils: Ensure locale en_US.UTF-8 is available on the system

 bin/bitbake        |  3 +--
 bin/bitbake-server |  5 +++--
 bin/bitbake-worker |  3 +--
 lib/bb/utils.py    | 16 ++++++++++++++++
 4 files changed, 21 insertions(+), 6 deletions(-)

-- 
2.34.1

[bitbake][kirkstone][2.0][PATCH 0/1] Patch review

[Steve Sakoman]

Please review this change for kirkstone/2.0 and have comments back by
end of day Tuesday, June 25

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7065

The following changes since commit 5a90927f31c4f9fccbe5d9d07d08e6e69485baa8:

  parse: Improve/fix cache invalidation via mtime (2024-05-23 08:52:30 -0700)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.0-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut

Steve Sakoman (1):
  tests/fetch: Tweak test to match upstream repo url change Upstream
    changed their urls, update our test to match.

 lib/bb/tests/fetch.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.34.1

[bitbake][kirkstone][2.0][PATCH 0/1] Patch review

[Steve Sakoman]

Please review this set of changes for 2.0/kirkstone and have comments back by
end of day Thursday, August 15

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7236

The following changes since commit 734b0ea3dfe45eb16ee60f0c2c388e22af4040e0:

  tests/fetch: Tweak test to match upstream repo url change Upstream changed their urls, update our test to match. (2024-06-11 11:22:11 -0700)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.0-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut

Robert Yang (1):
  data_smart: Improve performance for VariableHistory

 lib/bb/data_smart.py | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

-- 
2.34.1

[bitbake][kirkstone][2.0][PATCH 0/1] Patch review

[Steve Sakoman]

Please review this change for kirkstone/2.0 and have comments back by
end of day Friday, November 8

Passed a-full on autobuilder:

https://valkyrie.yoctoproject.org/#/builders/29/builds/398

The following changes since commit 0c30e9aadd30fc6f0dcf811eb8340687b52eb00b:

  tests/fetch: Use our own mirror of mobile-broadband-provider to decouple from gnome gitlab (2024-11-02 09:45:13 +0000)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.0-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut

Philip Lorenz (1):
  codeparser: Fix handling of string AST nodes with older Python
    versions

 lib/bb/codeparser.py | 46 +++++++++++++++++++++++++++++++-------------
 1 file changed, 33 insertions(+), 13 deletions(-)

-- 
2.34.1

[bitbake][kirkstone][2.0][PATCH 0/1] Patch review

[Steve Sakoman]

Please review this change for 2.0/kirkstone and have comments back by
end of day Friday, March 14

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/1177

The following changes since commit e71f1ce53cf3b8320caa481ae62d1ce2900c4670:

  tests/fetch: Fix git shallow test failure with git >= 2.48 (2025-01-24 16:17:43 +0000)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.0-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut

Paulo Neves (1):
  siggen.py: Improve taskhash reproducibility

 lib/bb/siggen.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

-- 
2.43.0

[bitbake][kirkstone][2.0][PATCH 1/1] siggen.py: Improve taskhash reproducibility

[Steve Sakoman]

From: Paulo Neves <paulo@myneves.com>

file checksums are part of the data checksummed
to generate the task hash. The list of file checksums
was not ordered.

In this commit we make sure the task hash checksum takes
a list of checksum data that is ordered by unique file name
thus guaranteeing reproducibility.

Signed-off-by: Paulo Neves <paulo@myneves.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/siggen.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index 0a9ce0ede..828729d8b 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -331,7 +331,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
         for dep in self.runtaskdeps[tid]:
             data += self.get_unihash(dep)
 
-        for (f, cs) in self.file_checksum_values[tid]:
+        for (f, cs) in sorted(self.file_checksum_values[tid], key=clean_checksum_file_path):
             if cs:
                 if "/./" in f:
                     data += "./" + f.split("/./")[1]
@@ -393,7 +393,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
         if runtime and tid in self.taskhash:
             data['runtaskdeps'] = self.runtaskdeps[tid]
             data['file_checksum_values'] = []
-            for f,cs in self.file_checksum_values[tid]:
+            for f,cs in sorted(self.file_checksum_values[tid], key=clean_checksum_file_path):
                 if "/./" in f:
                     data['file_checksum_values'].append(("./" + f.split("/./")[1], cs))
                 else:
@@ -720,6 +720,12 @@ class SignatureGeneratorTestMulticonfigDepends(SignatureGeneratorBasicHash):
     name = "TestMulticonfigDepends"
     supports_multiconfig_datacaches = True
 
+def clean_checksum_file_path(file_checksum_tuple):
+    f, cs = file_checksum_tuple
+    if "/./" in f:
+        return "./" + f.split("/./")[1]
+    return f
+
 def dump_this_task(outfile, d):
     import bb.parse
     fn = d.getVar("BB_FILENAME")
-- 
2.43.0

Re: [bitbake-devel] [bitbake][kirkstone][2.0][PATCH 1/1] siggen.py: Improve taskhash reproducibility

[Ernst Persson]

[-- Attachment #1: Type: text/plain, Size: 3093 bytes --]

I tried building in an old kirkstone workspace with this patch applied and
I got a lot of taskhash mismatch errors.
After deleting tmp and cache the build was fine, though it did rebuild
everything from scratch.
Does this need bumping of something like HASHEQUIV_HASH_VERSION or
SSTATE_VERSION... ?

Regards
//Ernst

Den ons 12 mars 2025 kl 20:48 skrev Steve Sakoman via lists.openembedded.org
<steve=sakoman.com@lists.openembedded.org>:

> From: Paulo Neves <paulo@myneves.com>
>
> file checksums are part of the data checksummed
> to generate the task hash. The list of file checksums
> was not ordered.
>
> In this commit we make sure the task hash checksum takes
> a list of checksum data that is ordered by unique file name
> thus guaranteeing reproducibility.
>
> Signed-off-by: Paulo Neves <paulo@myneves.com>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
> Signed-off-by: Steve Sakoman <steve@sakoman.com>
> ---
>  lib/bb/siggen.py | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
> index 0a9ce0ede..828729d8b 100644
> --- a/lib/bb/siggen.py
> +++ b/lib/bb/siggen.py
> @@ -331,7 +331,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>          for dep in self.runtaskdeps[tid]:
>              data += self.get_unihash(dep)
>
> -        for (f, cs) in self.file_checksum_values[tid]:
> +        for (f, cs) in sorted(self.file_checksum_values[tid],
> key=clean_checksum_file_path):
>              if cs:
>                  if "/./" in f:
>                      data += "./" + f.split("/./")[1]
> @@ -393,7 +393,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>          if runtime and tid in self.taskhash:
>              data['runtaskdeps'] = self.runtaskdeps[tid]
>              data['file_checksum_values'] = []
> -            for f,cs in self.file_checksum_values[tid]:
> +            for f,cs in sorted(self.file_checksum_values[tid],
> key=clean_checksum_file_path):
>                  if "/./" in f:
>                      data['file_checksum_values'].append(("./" +
> f.split("/./")[1], cs))
>                  else:
> @@ -720,6 +720,12 @@ class
> SignatureGeneratorTestMulticonfigDepends(SignatureGeneratorBasicHash):
>      name = "TestMulticonfigDepends"
>      supports_multiconfig_datacaches = True
>
> +def clean_checksum_file_path(file_checksum_tuple):
> +    f, cs = file_checksum_tuple
> +    if "/./" in f:
> +        return "./" + f.split("/./")[1]
> +    return f
> +
>  def dump_this_task(outfile, d):
>      import bb.parse
>      fn = d.getVar("BB_FILENAME")
> --
> 2.43.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#17438):
> https://lists.openembedded.org/g/bitbake-devel/message/17438
> Mute This Topic: https://lists.openembedded.org/mt/111666877/4947266
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> ernstp@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 5302 bytes --]

Re: [bitbake-devel] [bitbake][kirkstone][2.0][PATCH 1/1] siggen.py: Improve taskhash reproducibility

[Steve Sakoman]

On Wed, Mar 19, 2025 at 1:05 AM Ernst Persson <ernstp@gmail.com> wrote:
>
> I tried building in an old kirkstone workspace with this patch applied and I got a lot of taskhash mismatch errors.
> After deleting tmp and cache the build was fine, though it did rebuild everything from scratch.
> Does this need bumping of something like HASHEQUIV_HASH_VERSION or SSTATE_VERSION... ?

I didn't see any taskhash mismatch errors in any of my testing, so I'm
not sure what happened in your build!

I don't think bumps to HASHEQUIV_HASH_VERSION or SSTATE_VERSION are
required, but perhaps Richard can confirm.

Steve

> Den ons 12 mars 2025 kl 20:48 skrev Steve Sakoman via lists.openembedded.org <steve=sakoman.com@lists.openembedded.org>:
>>
>> From: Paulo Neves <paulo@myneves.com>
>>
>> file checksums are part of the data checksummed
>> to generate the task hash. The list of file checksums
>> was not ordered.
>>
>> In this commit we make sure the task hash checksum takes
>> a list of checksum data that is ordered by unique file name
>> thus guaranteeing reproducibility.
>>
>> Signed-off-by: Paulo Neves <paulo@myneves.com>
>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>> Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
>> Signed-off-by: Steve Sakoman <steve@sakoman.com>
>> ---
>>  lib/bb/siggen.py | 10 ++++++++--
>>  1 file changed, 8 insertions(+), 2 deletions(-)
>>
>> diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
>> index 0a9ce0ede..828729d8b 100644
>> --- a/lib/bb/siggen.py
>> +++ b/lib/bb/siggen.py
>> @@ -331,7 +331,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>>          for dep in self.runtaskdeps[tid]:
>>              data += self.get_unihash(dep)
>>
>> -        for (f, cs) in self.file_checksum_values[tid]:
>> +        for (f, cs) in sorted(self.file_checksum_values[tid], key=clean_checksum_file_path):
>>              if cs:
>>                  if "/./" in f:
>>                      data += "./" + f.split("/./")[1]
>> @@ -393,7 +393,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
>>          if runtime and tid in self.taskhash:
>>              data['runtaskdeps'] = self.runtaskdeps[tid]
>>              data['file_checksum_values'] = []
>> -            for f,cs in self.file_checksum_values[tid]:
>> +            for f,cs in sorted(self.file_checksum_values[tid], key=clean_checksum_file_path):
>>                  if "/./" in f:
>>                      data['file_checksum_values'].append(("./" + f.split("/./")[1], cs))
>>                  else:
>> @@ -720,6 +720,12 @@ class SignatureGeneratorTestMulticonfigDepends(SignatureGeneratorBasicHash):
>>      name = "TestMulticonfigDepends"
>>      supports_multiconfig_datacaches = True
>>
>> +def clean_checksum_file_path(file_checksum_tuple):
>> +    f, cs = file_checksum_tuple
>> +    if "/./" in f:
>> +        return "./" + f.split("/./")[1]
>> +    return f
>> +
>>  def dump_this_task(outfile, d):
>>      import bb.parse
>>      fn = d.getVar("BB_FILENAME")
>> --
>> 2.43.0
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#17438): https://lists.openembedded.org/g/bitbake-devel/message/17438
>> Mute This Topic: https://lists.openembedded.org/mt/111666877/4947266
>> Group Owner: bitbake-devel+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [ernstp@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>

Re: [bitbake-devel] [bitbake][kirkstone][2.0][PATCH 1/1] siggen.py: Improve taskhash reproducibility

[Richard Purdie]

On Wed, 2025-03-19 at 06:43 -0700, Steve Sakoman wrote:
> On Wed, Mar 19, 2025 at 1:05 AM Ernst Persson <ernstp@gmail.com>
> wrote:
> > 
> > I tried building in an old kirkstone workspace with this patch
> > applied and I got a lot of taskhash mismatch errors.
> > After deleting tmp and cache the build was fine, though it did
> > rebuild everything from scratch.
> > Does this need bumping of something like HASHEQUIV_HASH_VERSION or
> > SSTATE_VERSION... ?
> 
> I didn't see any taskhash mismatch errors in any of my testing, so
> I'm
> not sure what happened in your build!
> 
> I don't think bumps to HASHEQUIV_HASH_VERSION or SSTATE_VERSION are
> required, but perhaps Richard can confirm.

I think it probably needs a cache version bump, something like:

https://git.yoctoproject.org/poky/commit/bitbake/lib/bb/cache.py?id=fced0b8d7c852e52606fa8da321211a98a1dfaea

which would then trigger a reparse. Things should be ok after the
reparse.

Cheers,

Richard

Re: [bitbake-devel] [bitbake][kirkstone][2.0][PATCH 1/1] siggen.py: Improve taskhash reproducibility

[Steve Sakoman]

On Wed, Mar 19, 2025 at 8:03 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Wed, 2025-03-19 at 06:43 -0700, Steve Sakoman wrote:
> > On Wed, Mar 19, 2025 at 1:05 AM Ernst Persson <ernstp@gmail.com>
> > wrote:
> > >
> > > I tried building in an old kirkstone workspace with this patch
> > > applied and I got a lot of taskhash mismatch errors.
> > > After deleting tmp and cache the build was fine, though it did
> > > rebuild everything from scratch.
> > > Does this need bumping of something like HASHEQUIV_HASH_VERSION or
> > > SSTATE_VERSION... ?
> >
> > I didn't see any taskhash mismatch errors in any of my testing, so
> > I'm
> > not sure what happened in your build!
> >
> > I don't think bumps to HASHEQUIV_HASH_VERSION or SSTATE_VERSION are
> > required, but perhaps Richard can confirm.
>
> I think it probably needs a cache version bump, something like:
>
> https://git.yoctoproject.org/poky/commit/bitbake/lib/bb/cache.py?id=fced0b8d7c852e52606fa8da321211a98a1dfaea
>
> which would then trigger a reparse. Things should be ok after the
> reparse.

Ok, thanks!  I'll send a patch.

Steve

[bitbake][kirkstone][2.0][PATCH 0/1] Patch review

[Yoann Congal]

This is a patch added last-minute to handle the shutdown of the git
protocol on YP/OE git repos.
Given the cherry-pick nature of the patch and the kirkstone release
build to do quickly, I plan to send this for merge quickly.

Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3631
(Build was impacted by a known and fixed AB-INT, I backported the fix)

The following changes since commit 8e2d1f8de055549b2101614d85454fcd1d0f94b2:

  test/fetch: Switch u-boot based test to use our own mirror (2025-07-22 15:09:57 +0100)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.0-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut

for you to fetch changes up to 7fd0197fd5fedd23cc885b5e7e816d86a392fdf9:

  tests/fetch: Avoid using git protocol in tests (2026-04-09 13:28:26 +0200)

----------------------------------------------------------------

Richard Purdie (1):
  tests/fetch: Avoid using git protocol in tests

 lib/bb/tests/fetch.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)