RFC: FOO_subtract, the logical antidote to FOO_append.

RFC: FOO_subtract, the logical antidote to FOO_append.

[Peter Seebach]

There's a few cases where something is a huge list of space-separated
things, and it is desireable to remove one.  The example currently
afflicting me is DISTRO_FEATURES_LIBC_DEFAULT; I want to end up with
the distro features including all but one of the words in it.

It seems to me that a counterpart to _append would make sense.  Here
is my basic idea:

FOO_subtract = "..."

means that, when you expand FOO:

1. Fully expand it.
2. Fully expand FOO_subtract.
3. Remove any words in FOO_subtract from FOO.
4. Yield the result.

The rationale is that the semantics of things where we're using _append
seem to be consistently of the form "this is a space-separated set",
and being able to remove things from a set would be Super Handy.

So I'm proposing the semantics for consideration, and if people like
them, I will go try to implement it in my Copious Free Time.

-s
-- 
Listen, get this.  Nobody with a good compiler needs to be justified.

Re: RFC: FOO_subtract, the logical antidote to FOO_append.

[Manuel Bessler]

Just a few minutes ago I was wondering if such a feature did exist...

I ran into a situation where I wanted to remove something from a .bbappend
that is added to a variable using VARIABLE_append = "this and that"


Manuel
On Tue, May 15, 2012 at 3:01 PM, Peter Seebach
<peter.seebach@windriver.com> wrote:
> There's a few cases where something is a huge list of space-separated
> things, and it is desireable to remove one.  The example currently
> afflicting me is DISTRO_FEATURES_LIBC_DEFAULT; I want to end up with
> the distro features including all but one of the words in it.
>
> It seems to me that a counterpart to _append would make sense.  Here
> is my basic idea:
>
> FOO_subtract = "..."
>
> means that, when you expand FOO:
>
> 1. Fully expand it.
> 2. Fully expand FOO_subtract.
> 3. Remove any words in FOO_subtract from FOO.
> 4. Yield the result.
>
> The rationale is that the semantics of things where we're using _append
> seem to be consistently of the form "this is a space-separated set",
> and being able to remove things from a set would be Super Handy.
>
> So I'm proposing the semantics for consideration, and if people like
> them, I will go try to implement it in my Copious Free Time.
>
> -s
> --
> Listen, get this.  Nobody with a good compiler needs to be justified.
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: RFC: FOO_subtract, the logical antidote to FOO_append.

[Saul Wold]

On 05/15/2012 11:46 PM, Manuel Bessler wrote:
> Just a few minutes ago I was wondering if such a feature did exist...
>
> I ran into a situation where I wanted to remove something from a .bbappend
> that is added to a variable using VARIABLE_append = "this and that"
>
My understanding is that a _subtract is fraught with danger, there all 
sorts of ordering implications.

For what you are trying to do in a .bbappend, can be done by using 
oe_filter_out() from utils.bbclass, it has to be done in anonymous code.

VARIABLE := "${@oe_filter_out('xxx', '${VARIABLE}', d)}"

This might be what you want.

Sau!

>
> Manuel
> On Tue, May 15, 2012 at 3:01 PM, Peter Seebach
> <peter.seebach@windriver.com>  wrote:
>> There's a few cases where something is a huge list of space-separated
>> things, and it is desireable to remove one.  The example currently
>> afflicting me is DISTRO_FEATURES_LIBC_DEFAULT; I want to end up with
>> the distro features including all but one of the words in it.
>>
>> It seems to me that a counterpart to _append would make sense.  Here
>> is my basic idea:
>>
>> FOO_subtract = "..."
>>
>> means that, when you expand FOO:
>>
>> 1. Fully expand it.
>> 2. Fully expand FOO_subtract.
>> 3. Remove any words in FOO_subtract from FOO.
>> 4. Yield the result.
>>
>> The rationale is that the semantics of things where we're using _append
>> seem to be consistently of the form "this is a space-separated set",
>> and being able to remove things from a set would be Super Handy.
>>
>> So I'm proposing the semantics for consideration, and if people like
>> them, I will go try to implement it in my Copious Free Time.
>>
>> -s
>> --
>> Listen, get this.  Nobody with a good compiler needs to be justified.
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>

Re: RFC: FOO_subtract, the logical antidote to FOO_append.

[Mark Hatle]

On 5/15/12 11:35 PM, Saul Wold wrote:
> On 05/15/2012 11:46 PM, Manuel Bessler wrote:
>> Just a few minutes ago I was wondering if such a feature did exist...
>>
>> I ran into a situation where I wanted to remove something from a .bbappend
>> that is added to a variable using VARIABLE_append = "this and that"
>>
> My understanding is that a _subtract is fraught with danger, there all
> sorts of ordering implications.
>
> For what you are trying to do in a .bbappend, can be done by using
> oe_filter_out() from utils.bbclass, it has to be done in anonymous code.
>
> VARIABLE := "${@oe_filter_out('xxx', '${VARIABLE}', d)}"
>
> This might be what you want.

There are two issues I believe Peter is having.. the first is trying to figure 
out how to filter stuff out of _append.  I'm sure sure the above will be able to 
do that.

But the real problem we keep struggling with is .conf load order.. you try to 
set some nice value in local.conf, and some later configuration file adds to it, 
or simply overwrites it.  He'd like a way to strip out things like that.

I suspect the real answer though is fix the various configuration settings to 
better allow them to be overridden, but I'm not sure exactly how.

--Mark

> Sau!
>
>>
>> Manuel
>> On Tue, May 15, 2012 at 3:01 PM, Peter Seebach
>> <peter.seebach@windriver.com>   wrote:
>>> There's a few cases where something is a huge list of space-separated
>>> things, and it is desireable to remove one.  The example currently
>>> afflicting me is DISTRO_FEATURES_LIBC_DEFAULT; I want to end up with
>>> the distro features including all but one of the words in it.
>>>
>>> It seems to me that a counterpart to _append would make sense.  Here
>>> is my basic idea:
>>>
>>> FOO_subtract = "..."
>>>
>>> means that, when you expand FOO:
>>>
>>> 1. Fully expand it.
>>> 2. Fully expand FOO_subtract.
>>> 3. Remove any words in FOO_subtract from FOO.
>>> 4. Yield the result.
>>>
>>> The rationale is that the semantics of things where we're using _append
>>> seem to be consistently of the form "this is a space-separated set",
>>> and being able to remove things from a set would be Super Handy.
>>>
>>> So I'm proposing the semantics for consideration, and if people like
>>> them, I will go try to implement it in my Copious Free Time.
>>>
>>> -s
>>> --
>>> Listen, get this.  Nobody with a good compiler needs to be justified.
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>
>>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: RFC: FOO_subtract, the logical antidote to FOO_append.

[Chris Larson]

On Wed, May 16, 2012 at 8:07 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
> On 5/15/12 11:35 PM, Saul Wold wrote:
>>
>> On 05/15/2012 11:46 PM, Manuel Bessler wrote:
>>>
>>> Just a few minutes ago I was wondering if such a feature did exist...
>>>
>>> I ran into a situation where I wanted to remove something from a
>>> .bbappend
>>> that is added to a variable using VARIABLE_append = "this and that"
>>>
>> My understanding is that a _subtract is fraught with danger, there all
>> sorts of ordering implications.
>>
>> For what you are trying to do in a .bbappend, can be done by using
>> oe_filter_out() from utils.bbclass, it has to be done in anonymous code.
>>
>> VARIABLE := "${@oe_filter_out('xxx', '${VARIABLE}', d)}"
>>
>> This might be what you want.
>
>
> There are two issues I believe Peter is having.. the first is trying to
> figure out how to filter stuff out of _append.  I'm sure sure the above will
> be able to do that.
>
> But the real problem we keep struggling with is .conf load order.. you try
> to set some nice value in local.conf, and some later configuration file adds
> to it, or simply overwrites it.  He'd like a way to strip out things like
> that.
>
> I suspect the real answer though is fix the various configuration settings
> to better allow them to be overridden, but I'm not sure exactly how.

One related thing that won't solve this, but is worth noting -- I
think ??= is a much more sane choice than ?= in almost all cases. We
should think about a janitorial task to switch to ??= after carefully
considering potential implications for each.
-- 
Christopher Larson

Re: RFC: FOO_subtract, the logical antidote to FOO_append.

[Phil Blundell]

On Wed, 2012-05-16 at 10:07 -0500, Mark Hatle wrote:
> But the real problem we keep struggling with is .conf load order.. you try to 
> set some nice value in local.conf, and some later configuration file adds to it, 
> or simply overwrites it.  He'd like a way to strip out things like that.

OVERRIDES .= ":local"
VAR_local = "I rule"

Or you could use the existing "_forcevariable" override although I have
a feeling that was intended for something else.

p.

Re: RFC: FOO_subtract, the logical antidote to FOO_append.

[Peter Seebach]

On Wed, 16 May 2012 07:35:45 +0300
Saul Wold <sgw@linux.intel.com> wrote:

> My understanding is that a _subtract is fraught with danger, there
> all sorts of ordering implications.

Yes.

But consider, if you will, the specific case of
DISTRO_FEATURES_LIBC_DEFAULT, and a libc which is just like eglibc
except that it lacks RPC.

Anything I do that isn't processed at the tail end of everything,
around the point where _appends are processed, will be unable to
cleanly obtain "the value that would have been set by default if
nothing else happened", and then remove a word from it.  I can't set a
value in advance, because if I do the ?= won't fire and none of those
words will get set.  I can't necessarily set a value later.

Overrides won't work either, because overrides also destroy the
existing values.

It seems to me that for a subtraction to work, it *must* be the very
last thing done.

Basically, the purpose of suggesting this as a formal behavior defined
to be The Very Last Thing is to minimize the complexity of the ordering
implications.  You get exactly what you would have gotten otherwise,
with these words removed.

-s
-- 
Listen, get this.  Nobody with a good compiler needs to be justified.

Re: RFC: FOO_subtract, the logical antidote to FOO_append.

[Chris Larson]

On Wed, May 16, 2012 at 9:58 AM, Peter Seebach
<peter.seebach@windriver.com> wrote:
> On Wed, 16 May 2012 07:35:45 +0300
> Saul Wold <sgw@linux.intel.com> wrote:
>
>> My understanding is that a _subtract is fraught with danger, there
>> all sorts of ordering implications.
>
> Yes.
>
> But consider, if you will, the specific case of
> DISTRO_FEATURES_LIBC_DEFAULT, and a libc which is just like eglibc
> except that it lacks RPC.
>
> Anything I do that isn't processed at the tail end of everything,
> around the point where _appends are processed, will be unable to
> cleanly obtain "the value that would have been set by default if
> nothing else happened", and then remove a word from it.  I can't set a
> value in advance, because if I do the ?= won't fire and none of those
> words will get set.  I can't necessarily set a value later.
>
> Overrides won't work either, because overrides also destroy the
> existing values.
>
> It seems to me that for a subtraction to work, it *must* be the very
> last thing done.
>
> Basically, the purpose of suggesting this as a formal behavior defined
> to be The Very Last Thing is to minimize the complexity of the ordering
> implications.  You get exactly what you would have gotten otherwise,
> with these words removed.

I agree. I think this is the only sane option for implementing this
given bitbake's current behavior. Clearly a -= would be extremely
problematic, and would have clear complications with variable
expansion, as we discussed on IRC. I'd be interested in seeing a
prototype implementation of this, to experiment with it and see how
viable it is.
-- 
Christopher Larson

Re: RFC: FOO_subtract, the logical antidote to FOO_append.

[Richard Purdie]

On Thu, 2012-05-17 at 21:18 -0700, Chris Larson wrote:
> On Wed, May 16, 2012 at 9:58 AM, Peter Seebach
> <peter.seebach@windriver.com> wrote:
> > On Wed, 16 May 2012 07:35:45 +0300
> > Saul Wold <sgw@linux.intel.com> wrote:
> >
> >> My understanding is that a _subtract is fraught with danger, there
> >> all sorts of ordering implications.
> >
> > Yes.
> >
> > But consider, if you will, the specific case of
> > DISTRO_FEATURES_LIBC_DEFAULT, and a libc which is just like eglibc
> > except that it lacks RPC.
> >
> > Anything I do that isn't processed at the tail end of everything,
> > around the point where _appends are processed, will be unable to
> > cleanly obtain "the value that would have been set by default if
> > nothing else happened", and then remove a word from it.  I can't set a
> > value in advance, because if I do the ?= won't fire and none of those
> > words will get set.  I can't necessarily set a value later.
> >
> > Overrides won't work either, because overrides also destroy the
> > existing values.
> >
> > It seems to me that for a subtraction to work, it *must* be the very
> > last thing done.
> >
> > Basically, the purpose of suggesting this as a formal behavior defined
> > to be The Very Last Thing is to minimize the complexity of the ordering
> > implications.  You get exactly what you would have gotten otherwise,
> > with these words removed.
> 
> I agree. I think this is the only sane option for implementing this
> given bitbake's current behavior. Clearly a -= would be extremely
> problematic, and would have clear complications with variable
> expansion, as we discussed on IRC. I'd be interested in seeing a
> prototype implementation of this, to experiment with it and see how
> viable it is.

The _subtract or _remove format of this might be a viable way of adding
this functionality. "-=" is the one I'd prefer to avoid as it will end
up confusing people which is why we've all effectively been against it
but the _xxxx syntax does match the behaviour/ordering of _append and
_prepend and hence could be added with less confusion.

So yes, I think this could be the best way to address some of these
cases. It has always bugged me that its near impossible to counteract an
_append other than with an override.

I also agree we should audit the system and change to ??= where it makes
sense which will reduce the need for this.

Cheers,

Richard