package.tgz files not touched in sstate-cache

package.tgz files not touched in sstate-cache

[Mike Crowe]

We have a shared sstate-cache that is populated by us building nightly
images for various branches using Jenkins. This cache is pruned
periodically (with special care of the siginfo files since we don't have
80b3974081c4a8c604e23982a6db8fb32c616058 yet.)

Recently we went back to one of those branches that hadn't been changed for
a while and were surprised to find that Bitbake wanted to build loads of
packages from source.

It turned out that whilst the populate_sysroot.gz and package_ipk.tgz files
were still in the sstate-cache, the package.tgz tarball was not. This meant
that the whole package was built again in order to satisfy the build
dependencies of other packages that had changed.

I'm suspicious that this behaviour was introduced by the optimisation in
a0f584ac3d5a94dec121b684206ecd40c968f7fc so we hadn't seen it on older
branches.

Is there a way to cause the package.tgz file to be touched during these
nightly builds too? Or, is there a way to get hold of the complete set of
sstate files so they can be touched all in one go?

Thanks.

Mike.

Re: package.tgz files not touched in sstate-cache

[Richard Purdie]

On Fri, 2016-01-29 at 15:47 +0000, Mike Crowe wrote:
> We have a shared sstate-cache that is populated by us building nightly
> images for various branches using Jenkins. This cache is pruned
> periodically (with special care of the siginfo files since we don't have
> 80b3974081c4a8c604e23982a6db8fb32c616058 yet.)
> 
> Recently we went back to one of those branches that hadn't been changed for
> a while and were surprised to find that Bitbake wanted to build loads of
> packages from source.
> 
> It turned out that whilst the populate_sysroot.gz and package_ipk.tgz files
> were still in the sstate-cache, the package.tgz tarball was not. This meant
> that the whole package was built again in order to satisfy the build
> dependencies of other packages that had changed.
> 
> I'm suspicious that this behaviour was introduced by the optimisation in
> a0f584ac3d5a94dec121b684206ecd40c968f7fc so we hadn't seen it on older
> branches.
> 
> Is there a way to cause the package.tgz file to be touched during these
> nightly builds too? Or, is there a way to get hold of the complete set of
> sstate files so they can be touched all in one go?

The "right" answer here is to figure out what in your builds seems to
be needing the package.tgz files.

The only time the system should need to pull the package tgzs from
sstate is if package_qa or package_write_* were going to rerun. This
doesn't happen often, if ever. I've sometimes wondered if we should in
fact generate them. They are useful for switching package backends.

So the fact they're not getting touched doesn't surprise me. The more
concerning thing is that its actually trying to use them.

We have found a number of bugs in this area in sstate and have some
patches merged, I know there are a small number pending too. Whether
we've found all the issues remains unknown.

For debugging this, you usually need the task graph info and the -DDDv
output of a build, then investigate what was depending on the package
sstate file.

An example of where we have tested this is the extensible SDK code. We
can now remove the package sstate objects from it and have builds
function in the SDK without them. So we know this does work, there
could be some bugs remaining though.

Cheers,

Richard

Re: package.tgz files not touched in sstate-cache

[Mike Crowe]

On Friday 29 January 2016 at 17:42:13 +0000, Richard Purdie wrote:
> On Fri, 2016-01-29 at 15:47 +0000, Mike Crowe wrote:
> > We have a shared sstate-cache that is populated by us building nightly
> > images for various branches using Jenkins. This cache is pruned
> > periodically (with special care of the siginfo files since we don't have
> > 80b3974081c4a8c604e23982a6db8fb32c616058 yet.)
> > 
> > Recently we went back to one of those branches that hadn't been changed for
> > a while and were surprised to find that Bitbake wanted to build loads of
> > packages from source.
> > 
> > It turned out that whilst the populate_sysroot.gz and package_ipk.tgz files
> > were still in the sstate-cache, the package.tgz tarball was not. This meant
> > that the whole package was built again in order to satisfy the build
> > dependencies of other packages that had changed.
> > 
> > I'm suspicious that this behaviour was introduced by the optimisation in
> > a0f584ac3d5a94dec121b684206ecd40c968f7fc so we hadn't seen it on older
> > branches.
> > 
> > Is there a way to cause the package.tgz file to be touched during these
> > nightly builds too? Or, is there a way to get hold of the complete set of
> > sstate files so they can be touched all in one go?
> 
> The "right" answer here is to figure out what in your builds seems to
> be needing the package.tgz files.
> 
> The only time the system should need to pull the package tgzs from
> sstate is if package_qa or package_write_* were going to rerun. This
> doesn't happen often, if ever. I've sometimes wondered if we should in
> fact generate them. They are useful for switching package backends.

Indeed. These files are currently taking up over fifty gigabytes in our
shared sstate-cache!

> So the fact they're not getting touched doesn't surprise me. The more
> concerning thing is that its actually trying to use them.
> 
> We have found a number of bugs in this area in sstate and have some
> patches merged, I know there are a small number pending too. Whether
> we've found all the issues remains unknown.

The branch we saw this on was using dizzy from about
e848484989307ae6826ba0f5217f7702322181e3 so it's quite likely that we're
lacking some more recent fixes.

I've tried to reproduce the problem with the current tip of oe-core and
cannot so I'll go looking for patches.

> For debugging this, you usually need the task graph info and the -DDDv
> output of a build, then investigate what was depending on the package
> sstate file.

Thanks for your help.

Mike.

Re: package.tgz files not touched in sstate-cache

[Mike Crowe]

On Monday 01 February 2016 at 11:14:50 +0000, Mike Crowe wrote:
> On Friday 29 January 2016 at 17:42:13 +0000, Richard Purdie wrote:
> > On Fri, 2016-01-29 at 15:47 +0000, Mike Crowe wrote:
> > > It turned out that whilst the populate_sysroot.gz and package_ipk.tgz files
> > > were still in the sstate-cache, the package.tgz tarball was not. This meant
> > > that the whole package was built again in order to satisfy the build
> > > dependencies of other packages that had changed.

> > The "right" answer here is to figure out what in your builds seems to
> > be needing the package.tgz files.
> > 
> > The only time the system should need to pull the package tgzs from
> > sstate is if package_qa or package_write_* were going to rerun. This
> > doesn't happen often, if ever. I've sometimes wondered if we should in
> > fact generate them. They are useful for switching package backends.
> >
> > So the fact they're not getting touched doesn't surprise me. The more
> > concerning thing is that its actually trying to use them.
> > 
> > We have found a number of bugs in this area in sstate and have some
> > patches merged, I know there are a small number pending too. Whether
> > we've found all the issues remains unknown.
> 
> The branch we saw this on was using dizzy from about
> e848484989307ae6826ba0f5217f7702322181e3 so it's quite likely that we're
> lacking some more recent fixes.

For those following along at home, it appears that only four commits are
required to remove the need for the package.tgz files and they cherry-pick
easily:

 80b3974081c4a8c604e23982a6db8fb32c616058
 06c891f0fa6c7712eae233dd18612f1ab1889c45
 76aa4f49db7d32bbd35703c64470fdfc63f403a4
 654f0eec426e882e50f688f6d097d992e34e5b40

Thanks again for your help.

Mike.