[OE-core][PATCH] sstatesig: Improve output hash calculation

[OE-core][PATCH] sstatesig: Improve output hash calculation

[mateuszmar2]

From: Mateusz Marciniec <mateuszmar2@gmail.com>

Symbolic links to the files are included during the output hash
calculation but symlinks to the directories are missed.
So if the new symlink to a directory was the only change made,
then the output hash won't change,
and the Hash Equivalence server may change unihash.
In the next run bitbake may use an older package from sstate-cache.

To fix this followlinks=True flag could be set for os.walk
but it can lead to infinite recursion if link points
to a parent directory of itself.
Also, all files from a directory to which symlink points
would be included in depsig file.
Therefore another solution was applied, I added code that will loop
through directories and process those that are symlinks.

Signed-off-by: Mateusz Marciniec <mateuszmar2@gmail.com>
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
---
 meta/lib/oe/sstatesig.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index f0224454c9..a5bc030f58 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -652,6 +652,11 @@ def OEOuthashBasic(path, sigfile, task, d):
                 if f == 'fixmepath':
                     continue
                 process(os.path.join(root, f))
+
+            for d in dirs:
+                if not os.path.islink(os.path.join(root, d)):
+                    continue
+                process(os.path.join(root, d))
     finally:
         os.chdir(prev_dir)
 
-- 
2.39.1

RE: [OE-core][PATCH] sstatesig: Improve output hash calculation

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Mateusz Marciniec
> Sent: den 9 februari 2023 00:50
> To: openembedded-core@lists.openembedded.org
> Cc: Mateusz Marciniec <mateuszmar2@gmail.com>; Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
> Subject: [OE-core][PATCH] sstatesig: Improve output hash calculation
> 
> From: Mateusz Marciniec <mateuszmar2@gmail.com>
> 
> Symbolic links to the files are included during the output hash
> calculation but symlinks to the directories are missed.
> So if the new symlink to a directory was the only change made,
> then the output hash won't change,
> and the Hash Equivalence server may change unihash.
> In the next run bitbake may use an older package from sstate-cache.
> 
> To fix this followlinks=True flag could be set for os.walk
> but it can lead to infinite recursion if link points
> to a parent directory of itself.
> Also, all files from a directory to which symlink points
> would be included in depsig file.
> Therefore another solution was applied, I added code that will loop
> through directories and process those that are symlinks.
> 
> Signed-off-by: Mateusz Marciniec <mateuszmar2@gmail.com>
> Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
> ---
>  meta/lib/oe/sstatesig.py | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> index f0224454c9..a5bc030f58 100644
> --- a/meta/lib/oe/sstatesig.py
> +++ b/meta/lib/oe/sstatesig.py
> @@ -652,6 +652,11 @@ def OEOuthashBasic(path, sigfile, task, d):
>                  if f == 'fixmepath':
>                      continue
>                  process(os.path.join(root, f))
> +
> +            for d in dirs:

Don't use `d` as a local variable for the directory, it is universally 
used as reference to the datastore (also in this function as the function 
declaration above indicates). Use `dir` instead.

> +                if not os.path.islink(os.path.join(root, d)):
> +                    continue
> +                process(os.path.join(root, d))
>      finally:
>          os.chdir(prev_dir)
> 
> --
> 2.39.1

//Peter