Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Fri, 2023-09-22 at 11:17 +0200, Alexander Kanavin wrote:
> On Thu, 21 Sept 2023 at 16:39, chris.laplante@agilent.com
> <chris.laplante@agilent.com> wrote:
> 
> > That is very impressive and I'd also love to hear about what heuristics it uses.
> 
> It's actually rather simple. It uses glob.glob on stamps in tmp/, then
> on local sstate to find possible matches, then sorts them by mtime and
> takes the most recent. It's what would work most of the time, but we
> could add printdiff-all (print difference with all sstate matches) or
> printdiff-N (N most recent). It also could abstain from dumping
> locked-sigs.inc into cwd with both -S none and -S printdiff, unless
> explicitly asked
> 
> I just discovered there's also scripts/bitbake-whatchanged (that
> hasn't seen activity in years and is neither documented nor tested).
> Unsurprisingly then, it doesn't work in the same scenario:
> 
> ================
> alex@Zen2:/srv/storage/alex/yocto/build-sstate$ bitbake-whatchanged
> libsolv-native
> Figuring out the STAMPS_DIR ...
> Generating the new stamps ... (need several minutes)
> 
> === Summary: (0 changed, 0 unchanged)
> Newly added: 0
> PV changed: 0
> PR changed: 0
> Dependencies changed: 0
> 
> Removing the newly generated stamps dir ...
> ================
> 
> Maybe this is what RP was referring to when he said the tools don't
> work properly?

No, I've believed that should probably be removed. I think there was a
recent change to it.

I think we had a major step change in this functionality working when
this was fixed:

https://git.yoctoproject.org/poky/commit/?id=84a7485025dd4473403b8da36a0c979a3afd5e93

and this test case was added:

https://git.yoctoproject.org/poky/commit/?id=1bdcd76d2968c3cc6ec2815afceba1cf98efd6d5

Things which used to be problematic:

a) changes involving changes to gcc-source since it uses a shared
sources stamps which confused the tools (at least used to). That may
have been before gcc-source became a recipe?
b) changes to a very common component (e.g. autoconf-native's
do_configure) which make it hard to understand where the root cause of
the changes came from
c) changes which affect many recipes at once, e.g. the do_configure
function in base.bbclass

It might be helpful to write test cases for the scenario you showed as
working above and some of the ones I mention above, then we can
document they work and have an easier way to add tests for issues
if/as/when we identify the problematic scenarios in future.

As you mention, it also uses mtime so perhaps issues happen if you run
a different build, then try and go back to the other config? I suspect
once you understand the algorithm the code uses, you can pick holes in
it.

Cheers,

Richard