Another sstate rebuild issue, badly need help understanding sstate

Another sstate rebuild issue, badly need help understanding sstate

[Vuille, Martin (Martin)]

[-- Attachment #1: Type: text/plain, Size: 836 bytes --]

Poky 1.6.1 / daisy

I have a variable "REV_TOOLS" defined in distro/conf/somename.conf in a
custom distro layer from a git repo.

This variable is assigned to SRCREV in a number of recipes (SRCREV = "${REV_TOOLS}")
for packages included in my images and SDKs.

Pulled an update to the distro layer that changed the value of REV_TOOLS.

Rebuilt all my images and SDKs, but bitbake decided there was nothing to do.

Looking at toaster, all the do_build tasks for the packages affected by this change
say "Prebuilt" and the other tasks say "Covered". If I understand correctly, this
means sstate didn't detect any change.

I am continually running into this problem and am at my wits' end. Why didn't
the affected packages get rebuilt? Clearly there is something I don't understand
and would appreciate some hint.

MV

[-- Attachment #2: Type: text/html, Size: 3242 bytes --]

Re: Another sstate rebuild issue, badly need help understanding sstate

[Paul Eggleton]

On Monday 03 November 2014 12:53:23 Vuille, Martin wrote:
> Poky 1.6.1 / daisy
> 
> I have a variable "REV_TOOLS" defined in distro/conf/somename.conf in a
> custom distro layer from a git repo.
> 
> This variable is assigned to SRCREV in a number of recipes (SRCREV =
> "${REV_TOOLS}") for packages included in my images and SDKs.
> 
> Pulled an update to the distro layer that changed the value of REV_TOOLS.
> 
> Rebuilt all my images and SDKs, but bitbake decided there was nothing to do.
> 
> Looking at toaster, all the do_build tasks for the packages affected by this
> change say "Prebuilt" and the other tasks say "Covered". If I understand
> correctly, this means sstate didn't detect any change.
> 
> I am continually running into this problem and am at my wits' end. Why
> didn't the affected packages get rebuilt? Clearly there is something I
> don't understand and would appreciate some hint.

I honestly don't know how to explain this behaviour either, because it should 
not happen this way. BitBake automatically extracts variable references within 
variable values and adds them to the dependencies of the variable, so a change 
to REV_TOOLS should have resulted in the signatures changing for the do_fetch 
tasks for all of the recipes where SRCREV contained the reference and thus all 
of those recipes and the ones that depend upon them getting rebuilt.

Did you use oe-init-build-env or oe-init-build-env-memres to initialise your 
build environment?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Another sstate rebuild issue, badly need help understanding sstate

[Vuille, Martin (Martin)]

Hi Paul,

> -----Original Message-----
> From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
> Sent: November 03, 2014 9:13 AM
> 
> On Monday 03 November 2014 12:53:23 Vuille, Martin wrote:
> > Poky 1.6.1 / daisy
> >
> > I have a variable "REV_TOOLS" defined in distro/conf/somename.conf in
> > a custom distro layer from a git repo.
> >
> > This variable is assigned to SRCREV in a number of recipes (SRCREV =
> > "${REV_TOOLS}") for packages included in my images and SDKs.
> >
> > Pulled an update to the distro layer that changed the value of REV_TOOLS.
> >
> > Rebuilt all my images and SDKs, but bitbake decided there was nothing to
> do.
> >
> > Looking at toaster, all the do_build tasks for the packages affected
> > by this change say "Prebuilt" and the other tasks say "Covered". If I
> > understand correctly, this means sstate didn't detect any change.
> >
> > I am continually running into this problem and am at my wits' end. Why
> > didn't the affected packages get rebuilt? Clearly there is something I
> > don't understand and would appreciate some hint.
> 
> I honestly don't know how to explain this behaviour either, because it
> should not happen this way. BitBake automatically extracts variable
> references within variable values and adds them to the dependencies of
> the variable, so a change to REV_TOOLS should have resulted in the
> signatures changing for the do_fetch tasks for all of the recipes where
> SRCREV contained the reference and thus all of those recipes and the ones
> that depend upon them getting rebuilt.
> 
> Did you use oe-init-build-env or oe-init-build-env-memres to initialise your
> build environment?
> 

oe-init-build-env, prefixed with TEMPLATECONF to point to our custom
distro layer.

MV

Re: Another sstate rebuild issue, badly need help understanding sstate

[Vuille, Martin (Martin)]

> -----Original Message-----
> From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
> Sent: November 03, 2014 9:13 AM
> 
> I honestly don't know how to explain this behaviour either, because it
> should not happen this way. BitBake automatically extracts variable
> references within variable values and adds them to the dependencies of
> the variable, so a change to REV_TOOLS should have resulted in the
> signatures changing for the do_fetch tasks for all of the recipes where
> SRCREV contained the reference and thus all of those recipes and the ones
> that depend upon them getting rebuilt.
> 
> Did you use oe-init-build-env or oe-init-build-env-memres to initialise your
> build environment?
> 

I just realized that there may be issues with using upper-case letters
in package and recipe names. It certainly is confusing Toaster.

For example: https://lists.yoctoproject.org/pipermail/yocto/2013-July/014941.html

All my packagegroups have names with upper-case letters, as do my image
and SDK recipes. Could that be part of the problem?

I am in the process of renaming everything with upper-cases letters to
lower-case only.

MV

Re: Another sstate rebuild issue, badly need help understanding sstate

[Richard Purdie]

On Mon, 2014-11-03 at 12:53 +0000, Vuille, Martin (Martin) wrote:
> I have a variable “REV_TOOLS” defined in distro/conf/somename.conf in a
> custom distro layer from a git repo.
>  
> This variable is assigned to SRCREV in a number of recipes (SRCREV = “${REV_TOOLS}”)
> for packages included in my images and SDKs.

Can you describe how the value of REV_TOOLS is set? Is it just assigned
some value or done with some code?

Does clearing the bitbake cache make things rebuild as expected (mv
tmp/cache tmp/cache-old) ?

Cheers,

Richard

Re: Another sstate rebuild issue, badly need help understanding sstate

[Vuille, Martin (Martin)]

Hi Richard,

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: November 03, 2014 10:06 AM
> 
> On Mon, 2014-11-03 at 12:53 +0000, Vuille, Martin (Martin) wrote:
> > I have a variable “REV_TOOLS” defined in distro/conf/somename.conf in
> > a custom distro layer from a git repo.
> >
> > This variable is assigned to SRCREV in a number of recipes (SRCREV =
> > “${REV_TOOLS}”) for packages included in my images and SDKs.
> 
> Can you describe how the value of REV_TOOLS is set? Is it just assigned
> some value or done with some code?
> 

REV_TOOLS = "<some git commit hash string>"

> Does clearing the bitbake cache make things rebuild as expected (mv
> tmp/cache tmp/cache-old) ?
> 

I did a cleansstate on the affected packages and they rebuild as expected.

MV

Re: Another sstate rebuild issue, badly need help understanding sstate

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1480 bytes --]

On Mon, Nov 03, 2014 at 03:09:43PM +0000, Vuille, Martin (Martin) wrote:
> Hi Richard,
> 
> > -----Original Message-----
> > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > Sent: November 03, 2014 10:06 AM
> > 
> > On Mon, 2014-11-03 at 12:53 +0000, Vuille, Martin (Martin) wrote:
> > > I have a variable “REV_TOOLS” defined in distro/conf/somename.conf in
> > > a custom distro layer from a git repo.
> > >
> > > This variable is assigned to SRCREV in a number of recipes (SRCREV =
> > > “${REV_TOOLS}”) for packages included in my images and SDKs.
> > 
> > Can you describe how the value of REV_TOOLS is set? Is it just assigned
> > some value or done with some code?
> > 
> 
> REV_TOOLS = "<some git commit hash string>"
> 
> > Does clearing the bitbake cache make things rebuild as expected (mv
> > tmp/cache tmp/cache-old) ?
> > 
> 
> I did a cleansstate on the affected packages and they rebuild as expected.

Do your packages have SRCPV in PV?

SRCREV isn't included in sstate signature without SRCPV in PV or this
patch:

commit a414b17e1d783ad68a2d0f7d5922967449c05797
Author: Ross Burton <ross.burton@intel.com>
Date:   Mon Sep 29 15:40:14 2014 +0100

    base.bbclass: add SRCREV to do_fetch() hash

but even with this patch it doesn't work for recipes with multiple
repositories (you can find an e-mail with more details in oe-core ML).

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: Another sstate rebuild issue, badly need help understanding sstate

[Vuille, Martin (Martin)]

Hi Martin,

> -----Original Message-----
> From: Martin Jansa [mailto:martin.jansa@gmail.com]
> Sent: November 03, 2014 11:05 AM
> 
> On Mon, Nov 03, 2014 at 03:09:43PM +0000, Vuille, Martin (Martin) wrote:
> > Hi Richard,
> >
> > > -----Original Message-----
> > > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > > Sent: November 03, 2014 10:06 AM
> > >
> > > On Mon, 2014-11-03 at 12:53 +0000, Vuille, Martin (Martin) wrote:
> > > > I have a variable “REV_TOOLS” defined in distro/conf/somename.conf
> > > > in a custom distro layer from a git repo.
> > > >
> > > > This variable is assigned to SRCREV in a number of recipes (SRCREV
> > > > =
> > > > “${REV_TOOLS}”) for packages included in my images and SDKs.
> > >
> > > Can you describe how the value of REV_TOOLS is set? Is it just
> > > assigned some value or done with some code?
> > >
> >
> > REV_TOOLS = "<some git commit hash string>"
> >
> > > Does clearing the bitbake cache make things rebuild as expected (mv
> > > tmp/cache tmp/cache-old) ?
> > >
> >
> > I did a cleansstate on the affected packages and they rebuild as expected.
> 
> Do your packages have SRCPV in PV?
> 
> SRCREV isn't included in sstate signature without SRCPV in PV or this
> patch:
> 
> commit a414b17e1d783ad68a2d0f7d5922967449c05797
> Author: Ross Burton <ross.burton@intel.com>
> Date:   Mon Sep 29 15:40:14 2014 +0100
> 
>     base.bbclass: add SRCREV to do_fetch() hash
> 
> but even with this patch it doesn't work for recipes with multiple
> repositories (you can find an e-mail with more details in oe-core ML).

Thanks!

No, the packages do not use SRCPV. I don't think we have any cases
of recipes with multiple repos.

We will review this and either use SRCPV or apply the patch.

MV