dependencies between packages

dependencies between packages

[C Michael Sundius]

Some of our developers have been running into this problem and I'm wondering
how others deal with it.

It seems if we have a recipe, (A) that depends upon header files staged by
another recipe (B) then:

1) if the source for recipe (B) is modified and recompiled with:

    bitbake -f -c compile B
    bitbake B,

I would expect that when recipe A is run:

   bitbake A

It too would be recompiled, however that does not seem to be the case. In
fact even if recipe B is cleaned up:

    bitbake -c clean B

and then we run recipe A

    bitbake A

only recipe B is run and recipe A is NOT rerun! the dependency is satisfied
by rerunning recipe B and then bitbake stops..

is this expected behaviour? Is there away to force rerunning of recipes that
would be "out of date" due to one of its dependent recipes being return and
re-staging (potentially) new headers and libraries?

Any help of suggestions would be greatly appreciated.

Mike

Re: dependencies between packages

[Frans Meulenbroeks]

2010/1/11 C Michael Sundius <msundius@sundius.com>:
> Some of our developers have been running into this problem and I'm wondering
> how others deal with it.
>
> It seems if we have a recipe, (A) that depends upon header files staged by
> another recipe (B) then:
>
> 1) if the source for recipe (B) is modified and recompiled with:
>
>    bitbake -f -c compile B
>    bitbake B,
>
> I would expect that when recipe A is run:
>
>   bitbake A
>
> It too would be recompiled, however that does not seem to be the case. In
> fact even if recipe B is cleaned up:
>
>    bitbake -c clean B
>
> and then we run recipe A
>
>    bitbake A
>
> only recipe B is run and recipe A is NOT rerun! the dependency is satisfied
> by rerunning recipe B and then bitbake stops..
>
> is this expected behaviour? Is there away to force rerunning of recipes that
> would be "out of date" due to one of its dependent recipes being return and
> re-staging (potentially) new headers and libraries?
>
> Any help of suggestions would be greatly appreciated.
>
> Mike

If you change a recipe you should bump PR. in the recipe of B. That
will cause that bitbake A will resuilt in a rebuild.

Re: dependencies between packages

[C Michael Sundius]

>

> . In
> > fact even if recipe B is cleaned up:
> >
> >    bitbake -c clean B
> >
> > and then we run recipe A
> >
> >    bitbake A
> >
> > only recipe B is run and recipe A is NOT rerun! the dependency is
> satisfied
> > by rerunning recipe B and then bitbake stops..
> >
> > is this expected behaviour? Is there away to force rerunning of recipes
> that
> > would be "out of date" due to one of its dependent recipes being return
> and
> > re-staging (potentially) new headers and libraries?
> >
> > Any help of suggestions would be greatly appreciated.
> >
> > Mike
>
> If you change a recipe you should bump PR. in the recipe of B. That
> will cause that bitbake A will resuilt in a rebuild.
>
> Hmmm. nope... I bumped PR of B, then rebuilt A

vi B.bb    # change PR from r1 to r2
bitbake A

only caused B to be rebuilt. even though B may have staged completely
different headers or libraries.

Re: dependencies between packages

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11-01-10 18:59, C Michael Sundius wrote:
> Some of our developers have been running into this problem and I'm wondering
> how others deal with it.
> 
> It seems if we have a recipe, (A) that depends upon header files staged by
> another recipe (B) then:
> 
> 1) if the source for recipe (B) is modified and recompiled with:
> 
>     bitbake -f -c compile B
>     bitbake B,
> 
> I would expect that when recipe A is run:
> 
>    bitbake A
> 
> It too would be recompiled, however that does not seem to be the case. In
> fact even if recipe B is cleaned up:
> 
>     bitbake -c clean B
> 
> and then we run recipe A
> 
>     bitbake A
> 
> only recipe B is run and recipe A is NOT rerun! the dependency is satisfied
> by rerunning recipe B and then bitbake stops..
> 
> is this expected behaviour?

Yes, although it's a bit counter intuitive from a buildsystem POV, but
we currently don't have a foolproof, distributed way of automatically
bumping PR for dependant packages, so we don't enable that rebuild
feature. It's no use rebuilding when the packagemanager won't notice
that the package has been updated.

> Is there away to force rerunning of recipes that
> would be "out of date" due to one of its dependent recipes being return and
> re-staging (potentially) new headers and libraries?

Have a look at BB_STAMP_POLICY, I forgot the details.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFLS3PSMkyGM64RGpERAnpGAJwIthWw2W383J1cW+OCu41oLUSqQgCfRYW/
DUlLsi+a3tcysXsxLnRa5Pg=
=Upuw
-----END PGP SIGNATURE-----

Re: dependencies between packages

[Michael 'Mickey' Lauer]

Am Montag, den 11.01.2010, 09:59 -0800 schrieb C Michael Sundius:
> Some of our developers have been running into this problem and I'm wondering
> how others deal with it.
> 
> It seems if we have a recipe, (A) that depends upon header files staged by
> another recipe (B) then:
> 
> 1) if the source for recipe (B) is modified and recompiled with:
> 
>     bitbake -f -c compile B
>     bitbake B,
> 
> I would expect that when recipe A is run:
> 
>    bitbake A
> 
> It too would be recompiled, however that does not seem to be the case. In
> fact even if recipe B is cleaned up:
> 
>     bitbake -c clean B
> 
> and then we run recipe A
> 
>     bitbake A
> 
> only recipe B is run and recipe A is NOT rerun! the dependency is satisfied
> by rerunning recipe B and then bitbake stops..
> 
> is this expected behaviour? Is there away to force rerunning of recipes that
> would be "out of date" due to one of its dependent recipes being return and
> re-staging (potentially) new headers and libraries?

It is indeed expected behaviour, since there is no code in Bitbake that
would add any dependending packages and add them to your build.

Frans' suggestion is a workaround, however that only works with targets
that already include all the depending packages, usually a 'task',
'feed', or 'image' target.

I would welcome such a mode that automatically rebuilds all depending
packages, however as this could dramatically increase build time it
would need to be optional. BitBake hackers, what do you think?

-- 
:M:

Re: dependencies between packages

[Frans Meulenbroeks]

2010/1/11 Michael 'Mickey' Lauer <mickey@vanille-media.de>:
> Am Montag, den 11.01.2010, 09:59 -0800 schrieb C Michael Sundius:
>> Some of our developers have been running into this problem and I'm wondering
>> how others deal with it.
>>
>> It seems if we have a recipe, (A) that depends upon header files staged by
>> another recipe (B) then:
>>
>> 1) if the source for recipe (B) is modified and recompiled with:
>>
>>     bitbake -f -c compile B
>>     bitbake B,
>>
>> I would expect that when recipe A is run:
>>
>>    bitbake A
>>
>> It too would be recompiled, however that does not seem to be the case. In
>> fact even if recipe B is cleaned up:
>>
>>     bitbake -c clean B
>>
>> and then we run recipe A
>>
>>     bitbake A
>>
>> only recipe B is run and recipe A is NOT rerun! the dependency is satisfied
>> by rerunning recipe B and then bitbake stops..
>>
>> is this expected behaviour? Is there away to force rerunning of recipes that
>> would be "out of date" due to one of its dependent recipes being return and
>> re-staging (potentially) new headers and libraries?
>
> It is indeed expected behaviour, since there is no code in Bitbake that
> would add any dependending packages and add them to your build.
>
> Frans' suggestion is a workaround, however that only works with targets
> that already include all the depending packages, usually a 'task',
> 'feed', or 'image' target.
>
> I would welcome such a mode that automatically rebuilds all depending
> packages, however as this could dramatically increase build time it
> would need to be optional. BitBake hackers, what do you think?
>

Let's first start with a mechanism that lists the users of a package.

Frans

Re: dependencies between packages

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11-01-10 19:58, Michael 'Mickey' Lauer wrote:

> I would welcome such a mode that automatically rebuilds all depending
> packages, however as this could dramatically increase build time it
> would need to be optional. BitBake hackers, what do you think?

Bitbake already has such a mechanism and like I mentioned earlier in
this thread, it's of limited use, since the PRs won't get changed, so
from a DISTRO POV you still need to manually increment each affected PR
or bump DISTRO_PR, which will rebuild everything.
And even if we could bump PR automatically, we need a way to distribute
that too all buildhosts.

And yes, I'm cranky because I need exactly this to deal with the bzip2
SOVERSION fallout :/

regards,

Koen

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFLS40vMkyGM64RGpERAvo9AKCJENnNSjDBNKTZ63ufo/g+qwja/wCgvNIm
FKkq1RuyaSHHXVq+g1H+QN4=
=kv/H
-----END PGP SIGNATURE-----

Re: dependencies between packages

[Frans Meulenbroeks]

2010/1/11 Koen Kooi <k.kooi@student.utwente.nl>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 11-01-10 19:58, Michael 'Mickey' Lauer wrote:
>
>> I would welcome such a mode that automatically rebuilds all depending
>> packages, however as this could dramatically increase build time it
>> would need to be optional. BitBake hackers, what do you think?
>
> Bitbake already has such a mechanism and like I mentioned earlier in
> this thread, it's of limited use, since the PRs won't get changed, so
> from a DISTRO POV you still need to manually increment each affected PR
> or bump DISTRO_PR, which will rebuild everything.
> And even if we could bump PR automatically, we need a way to distribute
> that too all buildhosts.
>
> And yes, I'm cranky because I need exactly this to deal with the bzip2
> SOVERSION fallout :/
>
> regards,
>
> Koen
>

One way to resolve this without bumping all the PR's would be to
record the PRs of all pacakges it depends on.  (direct and indirect).
If you rebuild a package and any of the depends packages has a PR that
is increased, the package is rebuild, even if PR is not changed.
Another way could be to use the timestamps. If the staging timestamp
of a package one depends on is newer than the last build of the
package itself it is rebuild.
And a third way is to compute all packages that depend on another
package and rebuild those when the package is changed. However that
would probably rebuild way too much (everyone depends on glibc so if
that changes this would become a bitbake world).

However, I think the problem is not the rebuilding of packages, but
more the deployment of packages. Currently PV + PR are used to
determine if an upgrade is needed and that kinda gets lost (unless
opkg also has a version or time check; yuk)..

Frans.

Re: dependencies between packages

[C Michael Sundius]

On Mon, Jan 11, 2010 at 12:54 PM, Frans Meulenbroeks <
fransmeulenbroeks@gmail.com> wrote:

> 2010/1/11 Koen Kooi <k.kooi@student.utwente.nl>:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > On 11-01-10 19:58, Michael 'Mickey' Lauer wrote:
> >
> >> I would welcome such a mode that automatically rebuilds all depending
> >> packages, however as this could dramatically increase build time it
> >> would need to be optional. BitBake hackers, what do you think?
> >
> > Bitbake already has such a mechanism and like I mentioned earlier in
> > this thread, it's of limited use, since the PRs won't get changed, so
> > from a DISTRO POV you still need to manually increment each affected PR
> > or bump DISTRO_PR, which will rebuild everything.
> > And even if we could bump PR automatically, we need a way to distribute
> > that too all buildhosts.
> >
> > And yes, I'm cranky because I need exactly this to deal with the bzip2
> > SOVERSION fallout :/
> >
> > regards,
> >
> > Koen
> >
>
> One way to resolve this without bumping all the PR's would be to
> record the PRs of all pacakges it depends on.  (direct and indirect).
> If you rebuild a package and any of the depends packages has a PR that
> is increased, the package is rebuild, even if PR is not changed.
> Another way could be to use the timestamps. If the staging timestamp
> of a package one depends on is newer than the last build of the
> package itself it is rebuild.
> And a third way is to compute all packages that depend on another
> package and rebuild those when the package is changed. However that
> would probably rebuild way too much (everyone depends on glibc so if
> that changes this would become a bitbake world).
>
> However, I think the problem is not the rebuilding of packages, but
> more the deployment of packages. Currently PV + PR are used to
> determine if an upgrade is needed and that kinda gets lost (unless
> opkg also has a version or time check; yuk)..
>
> Frans.
>
>  Really I'm thinking of this in terms of how a developer works on a
particular build of an image.

in the code-compile-debug-code-compile-debug cycle, engineers are modifying
code and then rebuilding an image.

If the change (in the source code) is an internal one, and no object from
that package are lined statically into any other dependent packages, then
we're golden. just remake the image and boot, or reinstall the new package.

But, I have two use cases that my developers are griping about:

1) if an api is changed in say a library or driver (or the kernel itself)
2) if the package contains an object or library that is statically linked
into other objects in other packages.

Then we have to rebuild the dependent packages. While #1 may be rather rare,
#2 could be every time in the case for some of our driver team who bundle
various unrelated drivers together in one kernel module for efficiency
reasons.

Further, it seems as thou you all have been concerned about bumping a
revision number, which from my understanding will cause the entire recipe to
be rerun starting from setscene. Well, in the case of a developer doing a
code-compile-debug cycle, we really just want to rerun the dependent recipes
starting from the compile task.

In general, I'd be looking for two things in particular:
1) a way to closely couple a group of recipes so that if one changed, then
all would be rebuilt (but that might not result in the entire list of
dependent recipes to be re-built).
2) to cause the dependent recipes to be rerun from the compile (or possibly
the configure)... this way the makefiles would pick up timestamps of
(staged) libraries and headers that are out of date, rather than causing a
total rebuilt.

is there not a way to simply run through the dependency list and touch the
patch or configure timestamp of dependent packages of a recently re-compiled
package? If this was an extra step (task) such as

bitbake -c update-compile-deps  mylibrary

Re: dependencies between packages

[C Michael Sundius]

>>
>
> One way to resolve this without bumping all the PR's would be to
> record the PRs of all pacakges it depends on.  (direct and indirect).
> If you rebuild a package and any of the depends packages has a PR that
> is increased, the package is rebuild, even if PR is not changed.
> Another way could be to use the timestamps. If the staging timestamp
> of a package one depends on is newer than the last build of the
> package itself it is rebuild.
> And a third way is to compute all packages that depend on another
> package and rebuild those when the package is changed. However that
> would probably rebuild way too much (everyone depends on glibc so if
> that changes this would become a bitbake world).
>
> However, I think the problem is not the rebuilding of packages, but
> more the deployment of packages. Currently PV + PR are used to
> determine if an upgrade is needed and that kinda gets lost (unless
> opkg also has a version or time check; yuk)..
>
> Frans.
>

Frans,

In your suggestions for solutions to the problem below you described  
that  "if <condition> then the package should be rebuilt".. .it seems  
that this is within the domain of bitbake, since its deciding if the  
tasks should even be run right?

so could you give me a pointer to where I would look to see the part  
of bitbake I would have to modify to add such a check?

I'm not much of a python hacker, though now it looks like I'll have to  
become one. If anyone has a pointer to how bitbake internals are  
organized or advice I'd appreciate it.

My thoughts are that I'll try to work from the 2nd option Frans  
suggested since it is really developers who are wrenching on code and  
recompiling many times in a day... I don't suspect they'll update the  
PR in the recipe each time. (thoughts?)


mike