Strange behavior in sstate-diff-machines.sh

Strange behavior in sstate-diff-machines.sh

[chris.laplante]

Hi all,

We are on Scarthgap, and I am seeing an issue with sstate-diff-machines.sh wherein the script claims to see task signature differences, but then doesn't report any. For example, I get:

ERROR: base-files different signature for task do_package_write_rpm.sigdata between my-machine-1 and my-machine-2
NOTE: Starting bitbake server...

There should be output from bitbake-diffsigs after the 'Note:' lines. 

I did some poking around in tmp/sstate-diff. The 'list.M' file for my-machine-1 has:

MACHINEpoky-linux/base-files/3.0.14.do_package_write_rpm.sigdata.af0c3fb3b1b8494fcd7a31ba513495df384fdf25a8eced584e7c71d722b26683

But the list.M for my-machine-2 has:

my-machine-1-poky-linux/base-files/3.0.14.do_package_write_rpm.sigdata.af0c3fb3b1b8494fcd7a31ba513495df384fdf25a8eced584e7c71d722b26683 

So interestingly, the hash is the same (af0c3fb3b1b8494fcd7a31ba513495df384fdf25a8eced584e7c71d722b26683), but in the case of 'base-files' for my-machine-2, it ended up using the sstate (I guess?) from my-machine-1. Maybe this has something to do with hash equiv? (I have BB_HASHSERVE set to 'auto')

Can anyone please help explain?

Thanks,
Chris

Re: [OE-core] Strange behavior in sstate-diff-machines.sh

[Richard Purdie]

On Mon, 2025-02-03 at 21:34 +0000, Chris Laplante via lists.openembedded.org wrote:
> We are on Scarthgap, and I am seeing an issue with sstate-diff-
> machines.sh wherein the script claims to see task signature
> differences, but then doesn't report any. For example, I get:
> 
> ERROR: base-files different signature for task
> do_package_write_rpm.sigdata between my-machine-1 and my-machine-2
> NOTE: Starting bitbake server...
> 
> There should be output from bitbake-diffsigs after the 'Note:' lines.
> 
> I did some poking around in tmp/sstate-diff. The 'list.M' file for
> my-machine-1 has:
> 
> MACHINEpoky-linux/base-
> files/3.0.14.do_package_write_rpm.sigdata.af0c3fb3b1b8494fcd7a31ba513
> 495df384fdf25a8eced584e7c71d722b26683
> 
> But the list.M for my-machine-2 has:
> 
> my-machine-1-poky-linux/base-
> files/3.0.14.do_package_write_rpm.sigdata.af0c3fb3b1b8494fcd7a31ba513
> 495df384fdf25a8eced584e7c71d722b26683 
> 
> So interestingly, the hash is the same
> (af0c3fb3b1b8494fcd7a31ba513495df384fdf25a8eced584e7c71d722b26683),
> but in the case of 'base-files' for my-machine-2, it ended up using
> the sstate (I guess?) from my-machine-1. Maybe this has something to
> do with hash equiv? (I have BB_HASHSERVE set to 'auto')
> 
> Can anyone please help explain?

I've never liked that script and personally, I've always wanted to make
bitbake's owm signature "diff" as helpful as we can. There are ton of
different things that conspire against us in doing that and whilst
we've improved, it still isn't ideal.

I'd note that the script predates hash equivalence and that can be a
big factor in things things like this. We've struggled to reconcile
some of the tools with changes like hash equivalence :(

So yes, I can imagine do_package matching between the two machines,
then hashequiv would reuse the later tasks like do_package_write_rpm
from sstate.

Cheers,

Richard

RE: [OE-core] Strange behavior in sstate-diff-machines.sh

[chris.laplante]

Hi Richard,

> I've never liked that script and personally, I've always wanted to make bitbake's
> owm signature "diff" as helpful as we can. There are ton of different things
> that conspire against us in doing that and whilst we've improved, it still isn't
> ideal.
> 
> I'd note that the script predates hash equivalence and that can be a big factor
> in things things like this. We've struggled to reconcile some of the tools with
> changes like hash equivalence :(
> 
> So yes, I can imagine do_package matching between the two machines, then
> hashequiv would reuse the later tasks like do_package_write_rpm from sstate.

Thanks for the info. 

Would this be the kind of thing that is best ported to bitbake-diffsigs, maybe as some kind of recursive mode? (Just asking out of curiosity - I don't yet have a strong enough grasp of all the caching/signature stuff)

For posterity, I hacked up the script with this patch and it fixed my issue (whilst probably introducing other, subtler issues).

Thanks,
Chris


diff --git a/scripts/sstate-diff-machines.sh b/scripts/sstate-diff-machines.sh
index 5ed413b2ee..b536c423d0 100755
--- a/scripts/sstate-diff-machines.sh
+++ b/scripts/sstate-diff-machines.sh
@@ -121,8 +121,11 @@ for M in ${machines}; do
   if ls ${tmpdir}/stamps/* >/dev/null 2>/dev/null ; then
     cp -ra ${tmpdir}/stamps/* ${OUTPUT}/${M}
     find ${OUTPUT}/${M} -name \*sigdata\* | sed "s#${OUTPUT}/${M}/##g" | sort > ${OUTPUT}/${M}/list
-    M_UNDERSCORE=`echo ${M} | sed 's/-/_/g'`
-    sed "s/^${M_UNDERSCORE}-/MACHINE/g" ${OUTPUT}/${M}/list | sort > ${OUTPUT}/${M}/list.M
+    for M2 in ${machines}; do
+      M_UNDERSCORE=`echo ${M2} | sed 's/-/_/g'`
+      printf "M_UNDERSCORE: ${M_UNDERSCORE}\n"
+      sed "s/^${M_UNDERSCORE}-/MACHINE/g" ${OUTPUT}/${M}/list | sort > ${OUTPUT}/${M}/list.M
+    done
     find ${tmpdir}/stamps/ -name \*sigdata\* | xargs rm -f
   else
     printf "ERROR: no sigdata files were generated for MACHINE $M in ${tmpdir}/stamps\n";

Re: [OE-core] Strange behavior in sstate-diff-machines.sh

[Richard Purdie]

On Mon, 2025-02-03 at 22:19 +0000, chris.laplante@agilent.com wrote:
> > I've never liked that script and personally, I've always wanted to
> > make bitbake's
> > owm signature "diff" as helpful as we can. There are ton of
> > different things
> > that conspire against us in doing that and whilst we've improved,
> > it still isn't
> > ideal.
> > 
> > I'd note that the script predates hash equivalence and that can be
> > a big factor
> > in things things like this. We've struggled to reconcile some of
> > the tools with
> > changes like hash equivalence :(
> > 
> > So yes, I can imagine do_package matching between the two machines,
> > then
> > hashequiv would reuse the later tasks like do_package_write_rpm
> > from sstate.
> 
> Thanks for the info. 
> 
> Would this be the kind of thing that is best ported to bitbake-
> diffsigs, maybe as some kind of recursive mode? (Just asking out of
> curiosity - I don't yet have a strong enough grasp of all the
> caching/signature stuff)
> 
> For posterity, I hacked up the script with this patch and it fixed my
> issue (whilst probably introducing other, subtler issues).

I'm not sure what the best thing to do is. The script can make some
assumptions that bitbake can't and does in some ways have more data so
it may not be possible but I've not looked at this area in a while...

Cheers,

Richard

Re: [OE-core] Strange behavior in sstate-diff-machines.sh

[Alexander Kanavin]

On Mon, 3 Feb 2025 at 23:19, Chris Laplante via lists.openembedded.org
<chris.laplante=agilent.com@lists.openembedded.org> wrote:
> Would this be the kind of thing that is best ported to bitbake-diffsigs, maybe as some kind of recursive mode? (Just asking out of curiosity - I don't yet have a strong enough grasp of all the caching/signature stuff)

We do have recursive sstate lookup to help finding out where things
diverged, but it's in 'bitbake -S printdiff' command, not in
bitbake-diffsigs (unfortunately it's not as easy to disover). If you
can experiment with that (from scarthgap onwards), and come up with
enhancements, would be great!

Alex

RE: [OE-core] Strange behavior in sstate-diff-machines.sh

[chris.laplante]

Hi Alex,

> On Mon, 3 Feb 2025 at 23:19, Chris Laplante via lists.openembedded.org
> <chris.laplante=agilent.com@lists.openembedded.org> wrote:
> > Would this be the kind of thing that is best ported to
> > bitbake-diffsigs, maybe as some kind of recursive mode? (Just asking
> > out of curiosity - I don't yet have a strong enough grasp of all the
> > caching/signature stuff)
> 
> We do have recursive sstate lookup to help finding out where things diverged,
> but it's in 'bitbake -S printdiff' command, not in bitbake-diffsigs (unfortunately
> it's not as easy to disover). If you can experiment with that (from scarthgap
> onwards), and come up with enhancements, would be great!

Thanks for the reminder - I'll give it a try :)

Thanks,
Chris