How to move a recipe to another directory without invalidating its sstate-cache?

How to move a recipe to another directory without invalidating its sstate-cache?

[Mike Looijmans]

I renamed "recipes-some/foo/bar.bb" to "recipes-some/buzz/bar.bb"

Rebuilding bar and its dependencies will take about 16 hours. So I don't want 
to trigger a rebuild.

running "bitbake -S printdiff bar" only reveils this:

The differences between the current build and any cached tasks start at the 
following tasks:
.../recipes-some/buzz/bar.bb, do_configure
NOTE: Reparsing files to collect dependency data
Writing locked sigs to .../build/locked-sigs.inc

Task bar:do_configure couldn't be used from the cache because:
   We need hash 52209720c011f1f165a0bbb042d9b12b, closest matching task was 
25b3674b7a678faf1ac623bf0d836bb9
   runtaskdeps changed from [..., 'foobar.bb.do_patch'] to 
['buzzbar.bb.do_patch', ...]


So apparently some mysterious variable appears to include the location of the 
recipe. I would not have expected the "foo" and "buzz" words to be present 
anywhere. The recipe itself doesn't use or need a configure step.

But my primary question is now: What variable can I set to prevent a rebuild?


Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: How to move a recipe to another directory without invalidating its sstate-cache?

[Richard Purdie]

On Wed, 2015-12-16 at 10:38 +0100, Mike Looijmans wrote:
> I renamed "recipes-some/foo/bar.bb" to "recipes-some/buzz/bar.bb"
> 
> Rebuilding bar and its dependencies will take about 16 hours. So I
> don't want 
> to trigger a rebuild.
> 
> running "bitbake -S printdiff bar" only reveils this:

I'm not sure I trust the output of -S printdiff, there are some cases
it doesn't seem to "guess" right. I wish I or someone one could fix but
but we can do its work manually. Can you try something like:

set TMPDIR = "x"
bitbake -S bar
rename the recipe
set TMPDIR = "y"
bitbake -S bar

then

"ls tmp-x/stamps/xxxx/bar"
"ls tmp-y/stamps/xxxx/bar"

and see which tasks change signature. Then run:

"bitbake-diffsigs <sig A> <sig B>"

and see if that makes more sense?

Cheers,

Richard

Re: How to move a recipe to another directory without invalidating its sstate-cache?

[Mike Looijmans]

On 16-12-15 13:35, Richard Purdie wrote:
> On Wed, 2015-12-16 at 10:38 +0100, Mike Looijmans wrote:
>> I renamed "recipes-some/foo/bar.bb" to "recipes-some/buzz/bar.bb"
>>
>> Rebuilding bar and its dependencies will take about 16 hours. So I
>> don't want
>> to trigger a rebuild.
>>
>> running "bitbake -S printdiff bar" only reveils this:
>
> I'm not sure I trust the output of -S printdiff, there are some cases
> it doesn't seem to "guess" right. I wish I or someone one could fix but
> but we can do its work manually. Can you try something like:
>
> set TMPDIR = "x"
> bitbake -S bar
> rename the recipe
> set TMPDIR = "y"
> bitbake -S bar

Found out I need an extra "none" in here:
bitbake -S none bar

>
> then
>
> "ls tmp-x/stamps/xxxx/bar"
> "ls tmp-y/stamps/xxxx/bar"
>
> and see which tasks change signature. Then run:
>
> "bitbake-diffsigs <sig A> <sig B>"
>
> and see if that makes more sense?

That gave the same output. Everything after "runtaskdeps" is bogus because its 
value changes from [blahblah, foobar] into [buzbar, blahblah] and now diffsig 
seems to attempt to match the "blahblah" signatures against those of "buzbar".

Which sort of hints that if my new name starts with an "f" the reordering 
won't happen, so I'm gonna try renaming to "fbuz" now,,,




Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: How to move a recipe to another directory without invalidating its sstate-cache?

[Mike Looijmans]

On 16-12-15 14:18, Mike Looijmans wrote:
> On 16-12-15 13:35, Richard Purdie wrote:
>> On Wed, 2015-12-16 at 10:38 +0100, Mike Looijmans wrote:
>>> I renamed "recipes-some/foo/bar.bb" to "recipes-some/buzz/bar.bb"
>>>
>>> Rebuilding bar and its dependencies will take about 16 hours. So I
>>> don't want
>>> to trigger a rebuild.
>>>
>>> running "bitbake -S printdiff bar" only reveils this:
>>
>> I'm not sure I trust the output of -S printdiff, there are some cases
>> it doesn't seem to "guess" right. I wish I or someone one could fix but
>> but we can do its work manually. Can you try something like:
>>
>> set TMPDIR = "x"
>> bitbake -S bar
>> rename the recipe
>> set TMPDIR = "y"
>> bitbake -S bar
>
> Found out I need an extra "none" in here:
> bitbake -S none bar
>
>>
>> then
>>
>> "ls tmp-x/stamps/xxxx/bar"
>> "ls tmp-y/stamps/xxxx/bar"
>>
>> and see which tasks change signature. Then run:
>>
>> "bitbake-diffsigs <sig A> <sig B>"
>>
>> and see if that makes more sense?
>
> That gave the same output. Everything after "runtaskdeps" is bogus because its
> value changes from [blahblah, foobar] into [buzbar, blahblah] and now diffsig
> seems to attempt to match the "blahblah" signatures against those of "buzbar".
>
> Which sort of hints that if my new name starts with an "f" the reordering
> won't happen, so I'm gonna try renaming to "fbuz" now,,,
>

Indeed.... If I rename the directory without affecting the sort order of that 
array, in this situation, "fbuz" instead of "buz",  the signatures will match 
again, and it won't rebuild.

Looks like the problem is the way the "runtaskdeps" are sorted.

If a recipe depends on another recipe say "X", moving the recipe to a 
directory which sorts to the other side of "X" will trigger a rebuild at some 
stage.


M.


Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: How to move a recipe to another directory without invalidating its sstate-cache?

[Richard Purdie]

On Wed, 2015-12-16 at 14:18 +0100, Mike Looijmans wrote:
> On 16-12-15 13:35, Richard Purdie wrote:
> > On Wed, 2015-12-16 at 10:38 +0100, Mike Looijmans wrote:
> > > I renamed "recipes-some/foo/bar.bb" to "recipes
> > > -some/buzz/bar.bb"
> > > 
> > > Rebuilding bar and its dependencies will take about 16 hours. So
> > > I
> > > don't want
> > > to trigger a rebuild.
> > > 
> > > running "bitbake -S printdiff bar" only reveils this:
> > 
> > I'm not sure I trust the output of -S printdiff, there are some
> > cases
> > it doesn't seem to "guess" right. I wish I or someone one could fix
> > but
> > but we can do its work manually. Can you try something like:
> > 
> > set TMPDIR = "x"
> > bitbake -S bar
> > rename the recipe
> > set TMPDIR = "y"
> > bitbake -S bar
> 
> Found out I need an extra "none" in here:
> bitbake -S none bar

Sorry, yes.

> > then
> > 
> > "ls tmp-x/stamps/xxxx/bar"
> > "ls tmp-y/stamps/xxxx/bar"
> > 
> > and see which tasks change signature. Then run:
> > 
> > "bitbake-diffsigs <sig A> <sig B>"
> > 
> > and see if that makes more sense?
> 
> That gave the same output. Everything after "runtaskdeps" is bogus
> because its 
> value changes from [blahblah, foobar] into [buzbar, blahblah] and now
> diffsig 
> seems to attempt to match the "blahblah" signatures against those of
> "buzbar".
> 
> Which sort of hints that if my new name starts with an "f" the
> reordering 
> won't happen, so I'm gonna try renaming to "fbuz" now,,,

I still suspect this might be the way the tool displays the data rather
than the real problem as there is some fuzziness in the data that code
has to work with. Bitbake just knows there are dependencies on hashes
X, Y, Z and it has to try and guess how to match them up. The paths are
included to help reduce fuzz but are only used by the analysis code,
not the checksum creation.

Did the hashes themselves change or just the paths? I'm wondering if
the order that bitbake adds the checksums to the main checksum may have
some bearing on this...

Being able to see some real data would also help. Does this reproduce
with a public recipe I can see the real data for?

Cheers,

Richard

Re: How to move a recipe to another directory without invalidating its sstate-cache?

[Mike Looijmans]

On 16-12-15 14:33, Richard Purdie wrote:
> On Wed, 2015-12-16 at 14:18 +0100, Mike Looijmans wrote:
>> On 16-12-15 13:35, Richard Purdie wrote:
>>> On Wed, 2015-12-16 at 10:38 +0100, Mike Looijmans wrote:
>>>> I renamed "recipes-some/foo/bar.bb" to "recipes
>>>> -some/buzz/bar.bb"
>>>>
>>>> Rebuilding bar and its dependencies will take about 16 hours. So
>>>> I
>>>> don't want
>>>> to trigger a rebuild.
>>>>
>>>> running "bitbake -S printdiff bar" only reveils this:
>>>
>>> I'm not sure I trust the output of -S printdiff, there are some
>>> cases
>>> it doesn't seem to "guess" right. I wish I or someone one could fix
>>> but
>>> but we can do its work manually. Can you try something like:
>>>
>>> set TMPDIR = "x"
>>> bitbake -S bar
>>> rename the recipe
>>> set TMPDIR = "y"
>>> bitbake -S bar
>>
>> Found out I need an extra "none" in here:
>> bitbake -S none bar
>
> Sorry, yes.
>
>>> then
>>>
>>> "ls tmp-x/stamps/xxxx/bar"
>>> "ls tmp-y/stamps/xxxx/bar"
>>>
>>> and see which tasks change signature. Then run:
>>>
>>> "bitbake-diffsigs <sig A> <sig B>"
>>>
>>> and see if that makes more sense?
>>
>> That gave the same output. Everything after "runtaskdeps" is bogus
>> because its
>> value changes from [blahblah, foobar] into [buzbar, blahblah] and now
>> diffsig
>> seems to attempt to match the "blahblah" signatures against those of
>> "buzbar".
>>
>> Which sort of hints that if my new name starts with an "f" the
>> reordering
>> won't happen, so I'm gonna try renaming to "fbuz" now,,,
>
> I still suspect this might be the way the tool displays the data rather
> than the real problem as there is some fuzziness in the data that code
> has to work with. Bitbake just knows there are dependencies on hashes
> X, Y, Z and it has to try and guess how to match them up. The paths are
> included to help reduce fuzz but are only used by the analysis code,
> not the checksum creation.

No, if the sort order doesn't change, the checksum remains the same as well.

So if i rename "foo" to "far" it will not rebuild the package (using plain 
bitbake without -S switches), but if I rename "foo" to "buz" it will rebuild 
because there's a dependent package that sorts before "far" but after "buz".


> Did the hashes themselves change or just the paths? I'm wondering if
> the order that bitbake adds the checksums to the main checksum may have
> some bearing on this...
>
> Being able to see some real data would also help. Does this reproduce
> with a public recipe I can see the real data for?

There should be quite a few recipes in OE-core that might match this pattern:

Recipe X must depend on recipe Y. The directory of Y sorts before that of X. 
Renaming the directory of X to something that sorts before that of Y should 
trigger the problem.

I'll see if I can locate an existing one.

M


Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail

Re: How to move a recipe to another directory without invalidating its sstate-cache?

[Mike Looijmans]

On 17-12-15 07:24, Mike Looijmans wrote:
> On 16-12-15 14:33, Richard Purdie wrote:
>> On Wed, 2015-12-16 at 14:18 +0100, Mike Looijmans wrote:
>>> On 16-12-15 13:35, Richard Purdie wrote:
>>>> On Wed, 2015-12-16 at 10:38 +0100, Mike Looijmans wrote:
>>>>> I renamed "recipes-some/foo/bar.bb" to "recipes
>>>>> -some/buzz/bar.bb"
>>>>>
>>>>> Rebuilding bar and its dependencies will take about 16 hours. So
>>>>> I
>>>>> don't want
>>>>> to trigger a rebuild.
>>>>>
>>>>> running "bitbake -S printdiff bar" only reveils this:
>>>>
>>>> I'm not sure I trust the output of -S printdiff, there are some
>>>> cases
>>>> it doesn't seem to "guess" right. I wish I or someone one could fix
>>>> but
>>>> but we can do its work manually. Can you try something like:
>>>>
>>>> set TMPDIR = "x"
>>>> bitbake -S bar
>>>> rename the recipe
>>>> set TMPDIR = "y"
>>>> bitbake -S bar
>>>
>>> Found out I need an extra "none" in here:
>>> bitbake -S none bar
>>
>> Sorry, yes.
>>
>>>> then
>>>>
>>>> "ls tmp-x/stamps/xxxx/bar"
>>>> "ls tmp-y/stamps/xxxx/bar"
>>>>
>>>> and see which tasks change signature. Then run:
>>>>
>>>> "bitbake-diffsigs <sig A> <sig B>"
>>>>
>>>> and see if that makes more sense?
>>>
>>> That gave the same output. Everything after "runtaskdeps" is bogus
>>> because its
>>> value changes from [blahblah, foobar] into [buzbar, blahblah] and now
>>> diffsig
>>> seems to attempt to match the "blahblah" signatures against those of
>>> "buzbar".
>>>
>>> Which sort of hints that if my new name starts with an "f" the
>>> reordering
>>> won't happen, so I'm gonna try renaming to "fbuz" now,,,
>>
>> I still suspect this might be the way the tool displays the data rather
>> than the real problem as there is some fuzziness in the data that code
>> has to work with. Bitbake just knows there are dependencies on hashes
>> X, Y, Z and it has to try and guess how to match them up. The paths are
>> included to help reduce fuzz but are only used by the analysis code,
>> not the checksum creation.
>
> No, if the sort order doesn't change, the checksum remains the same as well.
>
> So if i rename "foo" to "far" it will not rebuild the package (using plain
> bitbake without -S switches), but if I rename "foo" to "buz" it will rebuild
> because there's a dependent package that sorts before "far" but after "buz".
>
>
>> Did the hashes themselves change or just the paths? I'm wondering if
>> the order that bitbake adds the checksums to the main checksum may have
>> some bearing on this...
>>
>> Being able to see some real data would also help. Does this reproduce
>> with a public recipe I can see the real data for?
>
> There should be quite a few recipes in OE-core that might match this pattern:
>
> Recipe X must depend on recipe Y. The directory of Y sorts before that of X.
> Renaming the directory of X to something that sorts before that of Y should
> trigger the problem.
>
> I'll see if I can locate an existing one.

Found several candidates, but couldn't get them to trigger this... Must be 
something more involved than just renaming the directory...



Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail