Figuring out why sstate-cache is no longer used for a package

Figuring out why sstate-cache is no longer used for a package

[Mike Looijmans]

I've been struggling with this for a few days.

We have a build server that build various images overnight. One of the 
packages in that image is "fpga-image", which takes more than an hour to build.

We have been sharing the the build server's sstate-cache via HTTP and this has 
worked excellently up until yesterday.

The current situation is that a client will grab everything from the 
buildserver's HTTP sstate-cache, potentially finishing a build from scratch in 
about five minutes. However, for some reason, the fpga-image does not fall 
into this category, and eache machine insists on re-building it from scratch. 
I've been trying to debug this, but the sstate-cache is on another machine. I 
tried copying part of the build server's sstate-cache onto my machine, but 
that only results in "bitbake-diffsigs -t fpga-image .." yielding "ERROR: No 
sigdata files found matching fpga-image .." so that apparently is a dead end.

How can I determine what is causing the system to think that it needs to 
rebuild this package?

And/Or how do tell the system that this package only depends on the SRCREV and 
MACHINE and nothing else? (I've already set INHIBIT_DEFAULT_DEPS = "1", is 
there more that I can do?)


--
Mike Looijmans - TOPIC Automation

Re: Figuring out why sstate-cache is no longer used for a package

[Burton, Ross]

On 10 December 2013 06:44, Mike Looijmans <mike.looijmans@topic.nl> wrote:
> The current situation is that a client will grab everything from the
> buildserver's HTTP sstate-cache, potentially finishing a build from scratch
> in about five minutes. However, for some reason, the fpga-image does not
> fall into this category, and eache machine insists on re-building it from
> scratch. I've been trying to debug this, but the sstate-cache is on another
> machine. I tried copying part of the build server's sstate-cache onto my
> machine, but that only results in "bitbake-diffsigs -t fpga-image .."
> yielding "ERROR: No sigdata files found matching fpga-image .." so that
> apparently is a dead end.

The trick is to use bitbake -S to write stamp data for a particular
image and copy that around. The script sstate-diff-machines can help
here.

Note that there's been some sstate fallout from the recent
optimisations: it was discovered yesterday that sstate hashes using
the "contains()" optimisation wasn't reliable so could produce
different hashes on different machines.  Try oe-core master from today
and see if that's any better: it certainly is for me.

Ross

Re: Figuring out why sstate-cache is no longer used for a package

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1674 bytes --]

On Tue, Dec 10, 2013 at 07:44:49AM +0100, Mike Looijmans wrote:
> I've been struggling with this for a few days.
> 
> We have a build server that build various images overnight. One of the 
> packages in that image is "fpga-image", which takes more than an hour to build.
> 
> We have been sharing the the build server's sstate-cache via HTTP and this has 
> worked excellently up until yesterday.
> 
> The current situation is that a client will grab everything from the 
> buildserver's HTTP sstate-cache, potentially finishing a build from scratch in 
> about five minutes. However, for some reason, the fpga-image does not fall 
> into this category, and eache machine insists on re-building it from scratch. 
> I've been trying to debug this, but the sstate-cache is on another machine. I 
> tried copying part of the build server's sstate-cache onto my machine, but 
> that only results in "bitbake-diffsigs -t fpga-image .." yielding "ERROR: No 
> sigdata files found matching fpga-image .." so that apparently is a dead end.
> 
> How can I determine what is causing the system to think that it needs to 
> rebuild this package?

I use this script:
openembedded-core/scripts/sstate-diff-machines.sh

to create "backup" of sstate signatures and when some next build
unexpectedly doesn't reuse some sstate packages I use the script again
and compare new and old signatures to see why.

As bonus each build which populates out SSTATE_MIRROR also creates tarball
with these signatures, so I can easily download one tarball and compare
what and why won't be reused from sstate.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: Figuring out why sstate-cache is no longer used for a package

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 2074 bytes --]

On Tue, Dec 10, 2013 at 7:32 AM, Martin Jansa <martin.jansa@gmail.com>wrote:

> On Tue, Dec 10, 2013 at 07:44:49AM +0100, Mike Looijmans wrote:
> > I've been struggling with this for a few days.
> >
> > We have a build server that build various images overnight. One of the
> > packages in that image is "fpga-image", which takes more than an hour to
> build.
> >
> > We have been sharing the the build server's sstate-cache via HTTP and
> this has
> > worked excellently up until yesterday.
> >
> > The current situation is that a client will grab everything from the
> > buildserver's HTTP sstate-cache, potentially finishing a build from
> scratch in
> > about five minutes. However, for some reason, the fpga-image does not
> fall
> > into this category, and eache machine insists on re-building it from
> scratch.
> > I've been trying to debug this, but the sstate-cache is on another
> machine. I
> > tried copying part of the build server's sstate-cache onto my machine,
> but
> > that only results in "bitbake-diffsigs -t fpga-image .." yielding
> "ERROR: No
> > sigdata files found matching fpga-image .." so that apparently is a dead
> end.
> >
> > How can I determine what is causing the system to think that it needs to
> > rebuild this package?
>
> I use this script:
> openembedded-core/scripts/sstate-diff-machines.sh
>
> to create "backup" of sstate signatures and when some next build
> unexpectedly doesn't reuse some sstate packages I use the script again
> and compare new and old signatures to see why.
>
> As bonus each build which populates out SSTATE_MIRROR also creates tarball
> with these signatures, so I can easily download one tarball and compare
> what and why won't be reused from sstate.


You can also directly use bitbake-whatchanged -v <target>, with STAMPS_DIR
set to the automated build’s stamps directory w/ sstates included.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2572 bytes --]