Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Thu, 2024-02-08 at 14:35 +0100, Alexander Kanavin wrote:
> On Mon, 5 Feb 2024 at 22:11, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > > - export the hash equivalency database into something human readable,
> > > and placed into the bundle, and then import it back when the bundle is
> > > unpacked and set up. As far as I understand currently there's no code
> > > for this. I'd find it useful in general, as effects of hash
> > > equivalency can be tricky to grasp, and being able to look at the
> > > database in a human readable format goes a long way towards
> > > understanding how it all fits together. There may be other use cases
> > > where you'd want to move the content of the database between build
> > > machines. I'd like to get Joshua's take on this, did I misunderstand
> > > this completely?
> > 
> > You don't misunderstand, this is an issue.
> > 
> > The good news is that the hashes needed will be in
> > cache/bb_unihashes.dat. The eSDK code saves and restores that to ensure
> > things all work out ok.
> > 
> > It isn't human readable but could be translated easily enough.
> 
> Right, I didn't realize bb_unihashes.dat is simply a python pickle,
> and can be dumped to stdout with a three-liner script.
> 
> What I don't understand is why for each task, that file seems to
> contain only a single pair of equivalence, and only matching the last
> build. The complete set seems to be in hashserv.db, but I didn't yet
> look into the structure of that or whether it's portable to other
> locations as a file copy.
> 
> So how do the two files interact? Shouldn't we copy over the full
> database rather, and have bitbake re-create the cache on the other
> side from that?

hashserve.db is a full hash serve database. When we run a build without
specifying a hash serve to use, bitbake will start a local one and this
is the file which backs it. In a build with a remote hashserve, it
won't be present.

the unihashes file is bitbake's internal lookup cache. It exists so
that if the hashserve isn't local, we have a local lookup of our "last"
value.

This means the file is good for the current (last) build not much
beyond that.

I'm not saying any of this entirely solves the problem at hand, just
that some of the data is there.

Cheers,

Richard