do_fetchall broken by recent bitbake

do_fetchall broken by recent bitbake

[Enrico Scholz]

Hi,

the 82d73423c57569b984ee0ae3d93e3c3bd5dc5216 "runqueue: Reimplement
recrdepends so it works more correctly" bitbake commit breaks
OpenEmbedded's 'do_fetchall' task.  Previously sources of all dependent
tasks were downloaded.  Now, only a small subset gets fetched.

Before:

| $ ./bitbake my-image -c fetchall
| NOTE: Running task 1 of 202 (ID: 201, ...openssl/ocf-linux_20100325.bb, do_fetch)


After:

| $ ./bitbake my-image -c fetchall
| NOTE: Running task 1 of 11 (ID: 10, ...opkg-collateral.bb, do_fetch)


Any chance to fix bitbake or OE?



Enrico

Re: [bitbake-devel] do_fetchall broken by recent bitbake

[Richard Purdie]

On Sat, 2012-06-30 at 13:47 +0200, Enrico Scholz wrote:
> the 82d73423c57569b984ee0ae3d93e3c3bd5dc5216 "runqueue: Reimplement
> recrdepends so it works more correctly" bitbake commit breaks
> OpenEmbedded's 'do_fetchall' task.  Previously sources of all dependent
> tasks were downloaded.  Now, only a small subset gets fetched.
> 
> Before:
> 
> | $ ./bitbake my-image -c fetchall
> | NOTE: Running task 1 of 202 (ID: 201, ...openssl/ocf-linux_20100325.bb, do_fetch)

> After:
> 
> | $ ./bitbake my-image -c fetchall
> | NOTE: Running task 1 of 11 (ID: 10, ...opkg-collateral.bb, do_fetch)

> Any chance to fix bitbake or OE?

Hmm, that is broken :(.

We have a problem with recrdepends since it means different things in
different contexts. Its turning out to be very hard to unravel the
different issues. I'll try and summarise the different ways it could
work. 

Firstly, let me define some dependencies:

build - a build time dependencies from DEPENDS
runtime - a runtime dependency from RDEPENDS, RRECOMMENDS or similar
intertask - an intertask dependency do_a[depends] = "somerecipe:do_b"

So what does do_a[recrdeptask] = "do_b" in recipe A mean?

"""
Look though every build, runtime and intertask dependency recipe A has
and see if those dependencies have a task do_b which would be added as a
dependency. Whether they have or have not, then look through all the
dependency's build, runtime and intertask dependencies do a task do_b
and so on.
"""

However we have some corner cases to consider.

a) If some dependency of A does not have a do_b task, should we still
consider its dependencies? Should we consider build, runtime and
intertask dependencies?

b) Say recipe A has a dependency on recipe B which has a task do_b. Do
we consider all build, runtime and intertask dependencies recipe B has?
Or should we consider just the dependencies do_b has? Or a hybrid,
consider the build and runtime dependencies overall but only any
intertask dependencies do_b itself has?

c) Imagine some task do_c has some intertask dependencies in recipe A
but do_a and do_b don't depend on do_c. Should we consider those
intertask dependencies?


Changing c) in the commit was intentional. It seems that indirectly, the
behaviour of b) has changed too and we now only consider the
dependencies do_b has. Personally, having spent several days staring at
this code, I think this actually makes a lot more sense.

So how could we fix fetchall?

I'm wondering if we should support the syntax

do_fetchall[recrdeptask] = "do_fetchall do_fetch"

since that would pull in the build and runtime deps of any dependency.
The trouble is right now this will fail with circular dependency errors
(understandably).

Cheers,

Richard