Wacky dependencies

Wacky dependencies

[Gary Thomas]

I wanted to run an experiment with X11 vs GTK-directfb on my platform.
I built a complete image with X11 in DISTRO_FEATURES.
When I changed DISTRO_FEATURES to have gtk-directfb, nearly everything
was rebuilt!  Why on earth do GCC & EGLIBC have to be rebuilt??  This
doesn't seem logical at all to me.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Wacky dependencies

[Paul Eggleton]

On Friday 16 November 2012 07:12:38 Gary Thomas wrote:
> I wanted to run an experiment with X11 vs GTK-directfb on my platform.
> I built a complete image with X11 in DISTRO_FEATURES.
> When I changed DISTRO_FEATURES to have gtk-directfb, nearly everything
> was rebuilt!  Why on earth do GCC & EGLIBC have to be rebuilt??  This
> doesn't seem logical at all to me.

The build system is not able to determine whether a variable within a recipe 
depends on a single value within DISTRO_FEATURES, it only knows that it looks 
at the variable; thus any change to DISTRO_FEATURES will change the hash of 
all recipes that read DISTRO_FEATURES.

I'd like to fix this but given that we check if DISTRO_FEATURES and other list 
variables like it contain values using inline python code i.e. 
${@base_contains(...)} this seems like it would be tricky to implement in a 
reliable way.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Wacky dependencies

[Gary Thomas]

On 2012-11-16 07:24, Paul Eggleton wrote:
> On Friday 16 November 2012 07:12:38 Gary Thomas wrote:
>> I wanted to run an experiment with X11 vs GTK-directfb on my platform.
>> I built a complete image with X11 in DISTRO_FEATURES.
>> When I changed DISTRO_FEATURES to have gtk-directfb, nearly everything
>> was rebuilt!  Why on earth do GCC & EGLIBC have to be rebuilt??  This
>> doesn't seem logical at all to me.
>
> The build system is not able to determine whether a variable within a recipe
> depends on a single value within DISTRO_FEATURES, it only knows that it looks
> at the variable; thus any change to DISTRO_FEATURES will change the hash of
> all recipes that read DISTRO_FEATURES.
>
> I'd like to fix this but given that we check if DISTRO_FEATURES and other list
> variables like it contain values using inline python code i.e.
> ${@base_contains(...)} this seems like it would be tricky to implement in a
> reliable way.

In this particular case, I think the problem is that DISTRO_FEATURES contains
too many controls.  In particular, all of the LIBC settings which *are* important
to GCC & EGLIBC.  Maybe that part should be kept separate and only have DISTRO_FEATURES
contain non-compiler items?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Wacky dependencies

[Tomas Frydrych]

On 16/11/12 14:29, Gary Thomas wrote:
> In this particular case, I think the problem is that DISTRO_FEATURES 
> contains too many controls.  In particular, all of the LIBC settings
> which *are* important to GCC & EGLIBC.  Maybe that part should be
> kept separate and only have DISTRO_FEATURES contain non-compiler
> items?

Changing distro features is not something you do on daily basis, but
when you do, the breakage potential is significant, so this really needs
to be robust more than efficient. If the check can be made smarter the
way Paul suggested, that would have some benefit, but multiplying the
variables that make up distro config will make maintaining distros, and
packages that depend on distro features, harder and more error prone, I
think.

Tomas

-- 
http://sleepfive.com

Re: Wacky dependencies

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 562 bytes --]

On Fri, Nov 16, 2012 at 7:29 AM, Gary Thomas <gary@mlbassoc.com> wrote:

> I'd like to fix this but given that we check if DISTRO_FEATURES and other
>> list
>> variables like it contain values using inline python code i.e.
>> ${@base_contains(...)} this seems like it would be tricky to implement in
>> a
>> reliable way.
>
>
I think you'd probably have to add the appropriate vardepsexclude on a
case-by-case basis, unless you wanted to play with the ast module and try
to determine what's being passed to base_contains :)
-- 
Christopher Larson

[-- Attachment #2: Type: text/html, Size: 955 bytes --]

Re: Wacky dependencies

[Andy Ross]

On 11/16/2012 06:24 AM, Paul Eggleton wrote:
> The build system is not able to determine whether a variable within a recipe
> depends on a single value within DISTRO_FEATURES, it only knows that it looks
> at the variable; thus any change to DISTRO_FEATURES will change the hash of
> all recipes that read DISTRO_FEATURES.
>
> I'd like to fix this but given that we check if DISTRO_FEATURES and other list
> variables like it contain values using inline python code i.e.
> ${@base_contains(...)} this seems like it would be tricky to implement in a
> reliable way.

I've been playing with variable dependency tracking recently and saw
issues like this too.  I don't think it can be done compatibly (in the
sense of preserving the existing list-in-a-string representation) at
all.  The problem is that dependencies are only tracked at the
variable level and the "logical dependency" is on substructure in the
data that isn't visible to the dependency system.

You could do this with the existing data model though, but splitting
things out into distinct variables.  Test for the presence of
"DISTRO_FEATURES_directfb" for example.  Or use flags:
"DISTRO_FEATURES[directfb]".  Flags would require bitbake modification
though, because they aren't part of the taskdeps field in the
siguatures.

DISTRO_FEATURES is interpreted outside bitbake, of course.  But in a
more limited sense this could be done for OVERRIDES, as it's handled
internally and could be special-cased in the dependency handling.

Andy

Re: Wacky dependencies

[Gary Thomas]

On 2012-11-16 11:14, Andy Ross wrote:
> On 11/16/2012 06:24 AM, Paul Eggleton wrote:
>> The build system is not able to determine whether a variable within a recipe
>> depends on a single value within DISTRO_FEATURES, it only knows that it looks
>> at the variable; thus any change to DISTRO_FEATURES will change the hash of
>> all recipes that read DISTRO_FEATURES.
>>
>> I'd like to fix this but given that we check if DISTRO_FEATURES and other list
>> variables like it contain values using inline python code i.e.
>> ${@base_contains(...)} this seems like it would be tricky to implement in a
>> reliable way.
>
> I've been playing with variable dependency tracking recently and saw
> issues like this too.  I don't think it can be done compatibly (in the
> sense of preserving the existing list-in-a-string representation) at
> all.  The problem is that dependencies are only tracked at the
> variable level and the "logical dependency" is on substructure in the
> data that isn't visible to the dependency system.
>
> You could do this with the existing data model though, but splitting
> things out into distinct variables.  Test for the presence of
> "DISTRO_FEATURES_directfb" for example.  Or use flags:
> "DISTRO_FEATURES[directfb]".  Flags would require bitbake modification
> though, because they aren't part of the taskdeps field in the
> siguatures.
>
> DISTRO_FEATURES is interpreted outside bitbake, of course.  But in a
> more limited sense this could be done for OVERRIDES, as it's handled
> internally and could be special-cased in the dependency handling.

Thanks, I can see that this is a truly "sticky widget".

Sadly, it probably doesn't matter as gtk-directfb isn't really ready
for prime time (it does not play nice with my touch screen device),
so I'll be switching back to X11 and fighting my battles there.

... off to rebuild the world [again]

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------