How to find out why my do_compile depends on $MACHINE

How to find out why my do_compile depends on $MACHINE

[Mike Looijmans]

I've been puzzling with this for many hours now, but I can't seem to figure 
this out.

I have a recipe that doesn't mention MACHINE anywhere, but still the 
sstate-cache won't be re-used for other machines, because OE somehow insists 
the package does depend on $MACHINE.

The workaround I could find is to put something like this in the recipe:

MACHINE[vardepvalue] = "any"

(Similar stupid hacks like "vardepsexclude" will probably work too, but I'm 
not interested in forcing this particular recipe, but I'm trying to figure out 
what makes bitbake think it matters)

Since it builds for an FPGA type, the package has 
PACKAGE_ARCH="${FPGA_FAMILY}" set, and FGPA_FAMILY is set in local.conf.


When switching machines, "bitbake -S printdiff fpga-image-dyplo-test" reports 
that MACHINE changed and that is what caused sstate not beign reused. But the 
recipe does not use anything related to $MACHINE at all.

Is there a way to make it tell me WHY it thinks that MACHINE is important to 
the recipe?


Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: How to find out why my do_compile depends on $MACHINE

[Richard Purdie]

On Mon, 2015-10-05 at 11:34 +0200, Mike Looijmans wrote:
> I've been puzzling with this for many hours now, but I can't seem to figure 
> this out.
> 
> I have a recipe that doesn't mention MACHINE anywhere, but still the 
> sstate-cache won't be re-used for other machines, because OE somehow insists 
> the package does depend on $MACHINE.
> 
> The workaround I could find is to put something like this in the recipe:
> 
> MACHINE[vardepvalue] = "any"
> 
> (Similar stupid hacks like "vardepsexclude" will probably work too, but I'm 
> not interested in forcing this particular recipe, but I'm trying to figure out 
> what makes bitbake think it matters)
> 
> Since it builds for an FPGA type, the package has 
> PACKAGE_ARCH="${FPGA_FAMILY}" set, and FGPA_FAMILY is set in local.conf.
> 
> 
> When switching machines, "bitbake -S printdiff fpga-image-dyplo-test" reports 
> that MACHINE changed and that is what caused sstate not beign reused. But the 
> recipe does not use anything related to $MACHINE at all.
> 
> Is there a way to make it tell me WHY it thinks that MACHINE is important to 
> the recipe?

Very briefly (sorry travelling), find the task that you want to know
more around and find the sigdata file in the stamps directory for it.
Run "bitbake-diffsigs <sigdatafile>" and it will show you how the
dependencies are calculated. You can compare two sigdata files using the
same command, "bitbake-diffsigs <sigdatafile1> <sigdatafile2>".

Cheers,

Richard

Re: How to find out why my do_compile depends on $MACHINE

[Mike Looijmans]

On 05-10-15 12:18, Richard Purdie wrote:
> On Mon, 2015-10-05 at 11:34 +0200, Mike Looijmans wrote:
>> I've been puzzling with this for many hours now, but I can't seem to figure
>> this out.
>>
>> I have a recipe that doesn't mention MACHINE anywhere, but still the
>> sstate-cache won't be re-used for other machines, because OE somehow insists
>> the package does depend on $MACHINE.
>>
>> The workaround I could find is to put something like this in the recipe:
>>
>> MACHINE[vardepvalue] = "any"
>>
>> (Similar stupid hacks like "vardepsexclude" will probably work too, but I'm
>> not interested in forcing this particular recipe, but I'm trying to figure out
>> what makes bitbake think it matters)
>>
>> Since it builds for an FPGA type, the package has
>> PACKAGE_ARCH="${FPGA_FAMILY}" set, and FGPA_FAMILY is set in local.conf.
>>
>>
>> When switching machines, "bitbake -S printdiff fpga-image-dyplo-test" reports
>> that MACHINE changed and that is what caused sstate not beign reused. But the
>> recipe does not use anything related to $MACHINE at all.
>>
>> Is there a way to make it tell me WHY it thinks that MACHINE is important to
>> the recipe?
>
> Very briefly (sorry travelling), find the task that you want to know
> more around and find the sigdata file in the stamps directory for it.
> Run "bitbake-diffsigs <sigdatafile>" and it will show you how the
> dependencies are calculated. You can compare two sigdata files using the
> same command, "bitbake-diffsigs <sigdatafile1> <sigdatafile2>".

Thanks, that reveiled the issue.

I was smart in local.conf and calculated the FPGA name from the MACHINE name. 
That yields the same word, but somehow the dependency on the MACHINE variable 
is kept.

A simple fix is then to just use a ":=" assignment for FPGA_FAMILY so that the 
link to MACHINE is lost, but that then results in another rebuild:

   List of dependencies for variable FPGA_FAMILY changed from 
'set(['MACHINE'])' to 'set([])'
   Dependency on Variable MACHINE was removed


The FPGA packages take long to build (4 hours for just one is no exception, on 
an i7 machine) so preventing them to be rebuilt is really important here.

But having found the issue, I guess after a round of rebuilds, it'll go away now.


For the bigger picture:

If I have a recipe that says:

X = "x"

And I refactor it a bit to read:

Y = "x"
X = "${Y}"

What is the logic behind having to rebuild the package now? Why does the 
'source' of the contents matter to the hash? It won't generate different 
output, regardless whether X is 'calculated' or just 'constant'.


In particular, I've noticed this happening when switching between AUTOREV and 
a fixed revision in a recipe.


Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: How to find out why my do_compile depends on $MACHINE

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4201 bytes --]

On Mon, Oct 05, 2015 at 01:49:25PM +0200, Mike Looijmans wrote:
> On 05-10-15 12:18, Richard Purdie wrote:
> > On Mon, 2015-10-05 at 11:34 +0200, Mike Looijmans wrote:
> >> I've been puzzling with this for many hours now, but I can't seem to figure
> >> this out.
> >>
> >> I have a recipe that doesn't mention MACHINE anywhere, but still the
> >> sstate-cache won't be re-used for other machines, because OE somehow insists
> >> the package does depend on $MACHINE.
> >>
> >> The workaround I could find is to put something like this in the recipe:
> >>
> >> MACHINE[vardepvalue] = "any"
> >>
> >> (Similar stupid hacks like "vardepsexclude" will probably work too, but I'm
> >> not interested in forcing this particular recipe, but I'm trying to figure out
> >> what makes bitbake think it matters)
> >>
> >> Since it builds for an FPGA type, the package has
> >> PACKAGE_ARCH="${FPGA_FAMILY}" set, and FGPA_FAMILY is set in local.conf.
> >>
> >>
> >> When switching machines, "bitbake -S printdiff fpga-image-dyplo-test" reports
> >> that MACHINE changed and that is what caused sstate not beign reused. But the
> >> recipe does not use anything related to $MACHINE at all.
> >>
> >> Is there a way to make it tell me WHY it thinks that MACHINE is important to
> >> the recipe?
> >
> > Very briefly (sorry travelling), find the task that you want to know
> > more around and find the sigdata file in the stamps directory for it.
> > Run "bitbake-diffsigs <sigdatafile>" and it will show you how the
> > dependencies are calculated. You can compare two sigdata files using the
> > same command, "bitbake-diffsigs <sigdatafile1> <sigdatafile2>".
> 
> Thanks, that reveiled the issue.
> 
> I was smart in local.conf and calculated the FPGA name from the MACHINE name. 
> That yields the same word, but somehow the dependency on the MACHINE variable 
> is kept.
> 
> A simple fix is then to just use a ":=" assignment for FPGA_FAMILY so that the 
> link to MACHINE is lost, but that then results in another rebuild:
> 
>    List of dependencies for variable FPGA_FAMILY changed from 
> 'set(['MACHINE'])' to 'set([])'
>    Dependency on Variable MACHINE was removed
> 
> 
> The FPGA packages take long to build (4 hours for just one is no exception, on 
> an i7 machine) so preventing them to be rebuilt is really important here.
> 
> But having found the issue, I guess after a round of rebuilds, it'll go away now.
> 
> 
> For the bigger picture:
> 
> If I have a recipe that says:
> 
> X = "x"
> 
> And I refactor it a bit to read:
> 
> Y = "x"
> X = "${Y}"
> 
> What is the logic behind having to rebuild the package now? Why does the 
> 'source' of the contents matter to the hash? It won't generate different 
> output, regardless whether X is 'calculated' or just 'constant'.

You can set vardepvalue flag like this:
X[vardepvalue] = "${X}"
if all you care is the value of X not the expression used to construct
it. This can help with your FGPA_FAMILY and is more readable (at least
to me) then immediate assignment.

This cannot be applied by default, because in some cases you have e.g.
TOPDIR variable used in value and then it's better to exclude just
TOPDIR from the variable dependencies and keep the rest of variables
tracked (using just the value would cause different signatures for 2
builds with different TOPDIR).

> In particular, I've noticed this happening when switching between AUTOREV and 
> a fixed revision in a recipe.
> 
> 
> Kind regards,
> 
> Mike Looijmans
> System Expert
> 
> TOPIC Embedded Products
> Eindhovenseweg 32-C, NL-5683 KH Best
> Postbus 440, NL-5680 AK Best
> Telefoon: +31 (0) 499 33 69 79
> Telefax: +31 (0) 499 33 69 70
> E-mail: mike.looijmans@topicproducts.com
> Website: www.topicproducts.com
> 
> Please consider the environment before printing this e-mail
> 
> 
> 
> 
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: How to find out why my do_compile depends on $MACHINE

[Richard Purdie]

On Mon, 2015-10-05 at 13:49 +0200, Mike Looijmans wrote:
> On 05-10-15 12:18, Richard Purdie wrote:
> > On Mon, 2015-10-05 at 11:34 +0200, Mike Looijmans wrote:
> >> I've been puzzling with this for many hours now, but I can't seem to figure
> >> this out.
> >>
> >> I have a recipe that doesn't mention MACHINE anywhere, but still the
> >> sstate-cache won't be re-used for other machines, because OE somehow insists
> >> the package does depend on $MACHINE.
> >>
> >> The workaround I could find is to put something like this in the recipe:
> >>
> >> MACHINE[vardepvalue] = "any"
> >>
> >> (Similar stupid hacks like "vardepsexclude" will probably work too, but I'm
> >> not interested in forcing this particular recipe, but I'm trying to figure out
> >> what makes bitbake think it matters)
> >>
> >> Since it builds for an FPGA type, the package has
> >> PACKAGE_ARCH="${FPGA_FAMILY}" set, and FGPA_FAMILY is set in local.conf.
> >>
> >>
> >> When switching machines, "bitbake -S printdiff fpga-image-dyplo-test" reports
> >> that MACHINE changed and that is what caused sstate not beign reused. But the
> >> recipe does not use anything related to $MACHINE at all.
> >>
> >> Is there a way to make it tell me WHY it thinks that MACHINE is important to
> >> the recipe?
> >
> > Very briefly (sorry travelling), find the task that you want to know
> > more around and find the sigdata file in the stamps directory for it.
> > Run "bitbake-diffsigs <sigdatafile>" and it will show you how the
> > dependencies are calculated. You can compare two sigdata files using the
> > same command, "bitbake-diffsigs <sigdatafile1> <sigdatafile2>".
> 
> Thanks, that reveiled the issue.
> 
> I was smart in local.conf and calculated the FPGA name from the MACHINE name. 
> That yields the same word, but somehow the dependency on the MACHINE variable 
> is kept.
> 
> A simple fix is then to just use a ":=" assignment for FPGA_FAMILY so that the 
> link to MACHINE is lost, but that then results in another rebuild:
> 
>    List of dependencies for variable FPGA_FAMILY changed from 
> 'set(['MACHINE'])' to 'set([])'
>    Dependency on Variable MACHINE was removed

FWIW you can use vardepvalue, e.g.:

FEATURE_INSTALL[vardepvalue] = "${FEATURE_INSTALL}"

which will make it depend on the value of the variable rather than the
dependencies. Yes, a rebuild will result after this but it should be a
one off.

> The FPGA packages take long to build (4 hours for just one is no exception, on 
> an i7 machine) so preventing them to be rebuilt is really important here.
> 
> But having found the issue, I guess after a round of rebuilds, it'll go away now.
>
> For the bigger picture:
> 
> If I have a recipe that says:
> 
> X = "x"
> 
> And I refactor it a bit to read:
> 
> Y = "x"
> X = "${Y}"
> 
> What is the logic behind having to rebuild the package now? Why does the 
> 'source' of the contents matter to the hash? It won't generate different 
> output, regardless whether X is 'calculated' or just 'constant'.
>
> In particular, I've noticed this happening when switching between AUTOREV and 
> a fixed revision in a recipe.

This is simply just the way the system is designed. It checksums the
intermediate steps as well as the endpoints. I guess in the above
example the system might do "export Y" for example.

There are alternative ways to do things but its simply not the way
things were implemented. If you do need to collapse the dependency
chain, you can use vardepvalue which was added for that reason. There
are only a small number of places you really need to do that.

Cheers,

Richard

Re: How to find out why my do_compile depends on $MACHINE

[Mike Looijmans]

On 05-10-15 15:22, Richard Purdie wrote:
> On Mon, 2015-10-05 at 13:49 +0200, Mike Looijmans wrote:
>> On 05-10-15 12:18, Richard Purdie wrote:
>>> On Mon, 2015-10-05 at 11:34 +0200, Mike Looijmans wrote:
...
>> For the bigger picture:
>>
>> If I have a recipe that says:
>>
>> X = "x"
>>
>> And I refactor it a bit to read:
>>
>> Y = "x"
>> X = "${Y}"
>>
>> What is the logic behind having to rebuild the package now? Why does the
>> 'source' of the contents matter to the hash? It won't generate different
>> output, regardless whether X is 'calculated' or just 'constant'.
>>
>> In particular, I've noticed this happening when switching between AUTOREV and
>> a fixed revision in a recipe.
>
> This is simply just the way the system is designed. It checksums the
> intermediate steps as well as the endpoints. I guess in the above
> example the system might do "export Y" for example.

In that case, "Y" would end up in the package's sstate hash, and hence it 
would rebuild when Y changes.

I have a workaround now, multiple workarounds actually, but I'm trying to 
understand why the system would be designed this way. Currently I'm tempted to 
splatter X[vardepvalue]="${X}" all over the place just to get rid of this 
annoying behaviour.

> There are alternative ways to do things but its simply not the way
> things were implemented. If you do need to collapse the dependency
> chain, you can use vardepvalue which was added for that reason. There
> are only a small number of places you really need to do that.

Lets make things more concrete. My distro.conf now reads:

FULL_OPTIMIZATION = "-O2 -pipe ${DEBUG_FLAGS}"

Say I want to build packages with varying optimization levels. I consider 
myself pretty smart, so in this distro.conf file, I change that line to:

OPT_LEVEL ?= "-O2"
FULL_OPTIMIZATION = "${OPT_LEVEL} -pipe ${DEBUG_FLAGS}"

So now I can replace the optimization level of some packages by simply 
changing the OPT_LEVEL variable.

Nice, but this will trigger a rebuild of ALL packages, event though the build 
flags did not change at all!

I can't think of any reason why this would be desirable. Why is it designed 
this way?



Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: How to find out why my do_compile depends on $MACHINE

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2775 bytes --]

On Tue, Oct 6, 2015 at 5:55 AM, Mike Looijmans <mike.looijmans@topic.nl>
wrote:

> On 05-10-15 15:22, Richard Purdie wrote:
>
>> On Mon, 2015-10-05 at 13:49 +0200, Mike Looijmans wrote:
>>
>>> On 05-10-15 12:18, Richard Purdie wrote:
>>>
>>>> On Mon, 2015-10-05 at 11:34 +0200, Mike Looijmans wrote:
>>>>
>>> ...
>
>> For the bigger picture:
>>>
>>> If I have a recipe that says:
>>>
>>> X = "x"
>>>
>>> And I refactor it a bit to read:
>>>
>>> Y = "x"
>>> X = "${Y}"
>>>
>>> What is the logic behind having to rebuild the package now? Why does the
>>> 'source' of the contents matter to the hash? It won't generate different
>>> output, regardless whether X is 'calculated' or just 'constant'.
>>>
>>> In particular, I've noticed this happening when switching between
>>> AUTOREV and
>>> a fixed revision in a recipe.
>>>
>>
>> This is simply just the way the system is designed. It checksums the
>> intermediate steps as well as the endpoints. I guess in the above
>> example the system might do "export Y" for example.
>>
>
> In that case, "Y" would end up in the package's sstate hash, and hence it
> would rebuild when Y changes.
>
> I have a workaround now, multiple workarounds actually, but I'm trying to
> understand why the system would be designed this way. Currently I'm tempted
> to splatter X[vardepvalue]="${X}" all over the place just to get rid of
> this annoying behaviour.
>
> There are alternative ways to do things but its simply not the way
>> things were implemented. If you do need to collapse the dependency
>> chain, you can use vardepvalue which was added for that reason. There
>> are only a small number of places you really need to do that.
>>
>
> Lets make things more concrete. My distro.conf now reads:
>
> FULL_OPTIMIZATION = "-O2 -pipe ${DEBUG_FLAGS}"
>
> Say I want to build packages with varying optimization levels. I consider
> myself pretty smart, so in this distro.conf file, I change that line to:
>
> OPT_LEVEL ?= "-O2"
> FULL_OPTIMIZATION = "${OPT_LEVEL} -pipe ${DEBUG_FLAGS}"
>
> So now I can replace the optimization level of some packages by simply
> changing the OPT_LEVEL variable.
>
> Nice, but this will trigger a rebuild of ALL packages, event though the
> build flags did not change at all!
>
> I can't think of any reason why this would be desirable. Why is it
> designed this way?


This behavior has bugged me for quite a while, I'd love to see this change
in the long term, assuming Richard isn't opposed to it. We should care
about the results, not how we got there, generally.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 4033 bytes --]