[PATCH] checksum: fix unstable sort of checksums

[PATCH] checksum: fix unstable sort of checksums

[yang.xu]

From: Yang Xu <yang.xu@mediatek.com>

Using only the checksum as sorting key can lead to unstable results when
same content files exist in different subfolders, affecting the taskhash
calculation due to its sensitivity to the order of files in checksums.

This commit changes the sorting key to use both checksum and file path,
ensuring stable sorting and enhancing sstate cache hit rates.

Signed-off-by: Yang Xu <yang.xu@mediatek.com>
---
 lib/bb/checksum.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/checksum.py b/lib/bb/checksum.py
index 557793d3..5294ddf1 100644
--- a/lib/bb/checksum.py
+++ b/lib/bb/checksum.py
@@ -140,5 +140,5 @@ class FileChecksumCache(MultiProcessCache):
                 if checksum:
                     checksums.append((pth, checksum))
 
-        checksums.sort(key=operator.itemgetter(1))
+        checksums.sort(key=lambda x:(x[1], x[0]))
         return checksums
-- 
2.25.1

Re: [bitbake-devel] [PATCH] checksum: fix unstable sort of checksums

[Richard Purdie]

On Tue, 2024-05-14 at 10:12 +0000, Yang Xu via lists.openembedded.org wrote:
> From: Yang Xu <yang.xu@mediatek.com>
> 
> Using only the checksum as sorting key can lead to unstable results when
> same content files exist in different subfolders, affecting the taskhash
> calculation due to its sensitivity to the order of files in checksums.
> 
> This commit changes the sorting key to use both checksum and file path,
> ensuring stable sorting and enhancing sstate cache hit rates.
> 
> Signed-off-by: Yang Xu <yang.xu@mediatek.com>
> ---
>  lib/bb/checksum.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/lib/bb/checksum.py b/lib/bb/checksum.py
> index 557793d3..5294ddf1 100644
> --- a/lib/bb/checksum.py
> +++ b/lib/bb/checksum.py
> @@ -140,5 +140,5 @@ class FileChecksumCache(MultiProcessCache):
>                  if checksum:
>                      checksums.append((pth, checksum))
>  
> -        checksums.sort(key=operator.itemgetter(1))
> +        checksums.sort(key=lambda x:(x[1], x[0]))
>          return checksums
> 

Which version of the project are you testing this with?

The checksum data is sorted where it is used in siggen.py so sorting
here shouldn't make much difference?

Cheers,

Richard

Re: [PATCH] checksum: fix unstable sort of checksums

[Yang Xu]

[-- Attachment #1: Type: text/plain, Size: 222 bytes --]

Dear Richard,

Sorry, I use a old version "kirkstone". Yes, I see the unstable sort problem is fixed in siggen.py in newer version.
I think the existed solution is also OK.

Please ignore current patch.

Thank you

[-- Attachment #2: Type: text/html, Size: 250 bytes --]