Native vs not

Native vs not

[Gary Thomas]

As pointed out in another thread, I'm trying to build a native
package which sends "I need native" ripples throughout much of
the Poky infrastructure.

Does having a native version available for any given package incur a cost?
If not, would patches for [all of] the packages I need be acceptable?

So far, nearly all of the affected packages built fine, just adding native
to BBCLASSEXTEND.  Many already build nativesdk versions already.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Native vs not

[Richard Purdie]

On Fri, 2011-03-11 at 04:34 -0700, Gary Thomas wrote:
> As pointed out in another thread, I'm trying to build a native
> package which sends "I need native" ripples throughout much of
> the Poky infrastructure.
> 
> Does having a native version available for any given package incur a cost?
> If not, would patches for [all of] the packages I need be acceptable?
> 
> So far, nearly all of the affected packages built fine, just adding native
> to BBCLASSEXTEND.  Many already build nativesdk versions already.

There is a cost incurred by doing this since it does increase parse time
and this is something user exposed which we do try and keep under
control.

Having said that, the BBCLASSEXTEND technology has a lot less overhead
than some of the older approaches to native/sdk recipes.

The main reason I've been against native everywhere is that having
native versions available makes it far too easy for people to add native
dependencies which encourage feature creep without thinking through the
huge additional dependency chains, the extra build time and other
implications. Often there are slightly more difficult but worthwhile
ways we can avoid the native dependency.

Cheers,

Richard

Re: Native vs not

[Gary Thomas]

On 03/11/2011 10:13 AM, Richard Purdie wrote:
> On Fri, 2011-03-11 at 04:34 -0700, Gary Thomas wrote:
>> As pointed out in another thread, I'm trying to build a native
>> package which sends "I need native" ripples throughout much of
>> the Poky infrastructure.
>>
>> Does having a native version available for any given package incur a cost?
>> If not, would patches for [all of] the packages I need be acceptable?
>>
>> So far, nearly all of the affected packages built fine, just adding native
>> to BBCLASSEXTEND.  Many already build nativesdk versions already.
>
> There is a cost incurred by doing this since it does increase parse time
> and this is something user exposed which we do try and keep under
> control.
>
> Having said that, the BBCLASSEXTEND technology has a lot less overhead
> than some of the older approaches to native/sdk recipes.

As is, I created a separate layer with a bunch of bbappend files that are
only
   BBCLASSEXTEND += " native "
I suppose if I never needed them, I could just not enable that layer.

>
> The main reason I've been against native everywhere is that having
> native versions available makes it far too easy for people to add native
> dependencies which encourage feature creep without thinking through the
> huge additional dependency chains, the extra build time and other
> implications. Often there are slightly more difficult but worthwhile
> ways we can avoid the native dependency.

Such as?  I started down this path needing a native tool (which admittedly
came from an OE recipe librsvg) which then cascaded into cairo-native and
beyond, totally 22 packages!.  If I knew of a short circuit for this, I'd
certainly entertain it.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Native vs not

[Tom Rini]

On 03/11/2011 10:20 AM, Gary Thomas wrote:
> On 03/11/2011 10:13 AM, Richard Purdie wrote:
>> On Fri, 2011-03-11 at 04:34 -0700, Gary Thomas wrote:
>>> As pointed out in another thread, I'm trying to build a native
>>> package which sends "I need native" ripples throughout much of
>>> the Poky infrastructure.
>>>
>>> Does having a native version available for any given package incur a
>>> cost?
>>> If not, would patches for [all of] the packages I need be acceptable?
>>>
>>> So far, nearly all of the affected packages built fine, just adding
>>> native
>>> to BBCLASSEXTEND. Many already build nativesdk versions already.
>>
>> There is a cost incurred by doing this since it does increase parse time
>> and this is something user exposed which we do try and keep under
>> control.
>>
>> Having said that, the BBCLASSEXTEND technology has a lot less overhead
>> than some of the older approaches to native/sdk recipes.
>
> As is, I created a separate layer with a bunch of bbappend files that are
> only
> BBCLASSEXTEND += " native "
> I suppose if I never needed them, I could just not enable that layer.
>
>>
>> The main reason I've been against native everywhere is that having
>> native versions available makes it far too easy for people to add native
>> dependencies which encourage feature creep without thinking through the
>> huge additional dependency chains, the extra build time and other
>> implications. Often there are slightly more difficult but worthwhile
>> ways we can avoid the native dependency.
>
> Such as? I started down this path needing a native tool (which admittedly
> came from an OE recipe librsvg) which then cascaded into cairo-native and
> beyond, totally 22 packages!. If I knew of a short circuit for this, I'd
> certainly entertain it.

Having just skimmed the OE recipes, yeah, there we just build a ton of 
stuff in librsvg-native.  But I would start by looking at what we can 
pass to configure to disable things as part of the host tool build.

Doing some git grep'ing I see there's just a few things in OE that need 
librsvg-native so perhaps we can build a much more limited native recipe 
instead and still generate what we need to.

-- 
Tom Rini
Mentor Graphics Corporation

Re: Native vs not

[Gary Thomas]

On 03/11/2011 10:25 AM, Tom Rini wrote:
> On 03/11/2011 10:20 AM, Gary Thomas wrote:
>> On 03/11/2011 10:13 AM, Richard Purdie wrote:
>>> On Fri, 2011-03-11 at 04:34 -0700, Gary Thomas wrote:
>>>> As pointed out in another thread, I'm trying to build a native
>>>> package which sends "I need native" ripples throughout much of
>>>> the Poky infrastructure.
>>>>
>>>> Does having a native version available for any given package incur a
>>>> cost?
>>>> If not, would patches for [all of] the packages I need be acceptable?
>>>>
>>>> So far, nearly all of the affected packages built fine, just adding
>>>> native
>>>> to BBCLASSEXTEND. Many already build nativesdk versions already.
>>>
>>> There is a cost incurred by doing this since it does increase parse time
>>> and this is something user exposed which we do try and keep under
>>> control.
>>>
>>> Having said that, the BBCLASSEXTEND technology has a lot less overhead
>>> than some of the older approaches to native/sdk recipes.
>>
>> As is, I created a separate layer with a bunch of bbappend files that are
>> only
>> BBCLASSEXTEND += " native "
>> I suppose if I never needed them, I could just not enable that layer.
>>
>>>
>>> The main reason I've been against native everywhere is that having
>>> native versions available makes it far too easy for people to add native
>>> dependencies which encourage feature creep without thinking through the
>>> huge additional dependency chains, the extra build time and other
>>> implications. Often there are slightly more difficult but worthwhile
>>> ways we can avoid the native dependency.
>>
>> Such as? I started down this path needing a native tool (which admittedly
>> came from an OE recipe librsvg) which then cascaded into cairo-native and
>> beyond, totally 22 packages!. If I knew of a short circuit for this, I'd
>> certainly entertain it.
>
> Having just skimmed the OE recipes, yeah, there we just build a ton of stuff in librsvg-native. But I would start by looking at what we can pass to configure to disable things as
> part of the host tool build.
>
> Doing some git grep'ing I see there's just a few things in OE that need librsvg-native so perhaps we can build a much more limited native recipe instead and still generate what we
> need to.

What I really needed was rsvg-convert which is used during the build of
midori.  In Poky/Yocto, that tool has to come from librsvg-native.  OE
seems to be happy leaning on the host version.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Native vs not

[Khem Raj]

On (11/03/11 10:32), Gary Thomas wrote:
> On 03/11/2011 10:25 AM, Tom Rini wrote:
> >On 03/11/2011 10:20 AM, Gary Thomas wrote:
> >>On 03/11/2011 10:13 AM, Richard Purdie wrote:
> >>>On Fri, 2011-03-11 at 04:34 -0700, Gary Thomas wrote:
> >>>>As pointed out in another thread, I'm trying to build a native
> >>>>package which sends "I need native" ripples throughout much of
> >>>>the Poky infrastructure.
> >>>>
> >>>>Does having a native version available for any given package incur a
> >>>>cost?
> >>>>If not, would patches for [all of] the packages I need be acceptable?
> >>>>
> >>>>So far, nearly all of the affected packages built fine, just adding
> >>>>native
> >>>>to BBCLASSEXTEND. Many already build nativesdk versions already.
> >>>
> >>>There is a cost incurred by doing this since it does increase parse time
> >>>and this is something user exposed which we do try and keep under
> >>>control.
> >>>
> >>>Having said that, the BBCLASSEXTEND technology has a lot less overhead
> >>>than some of the older approaches to native/sdk recipes.
> >>
> >>As is, I created a separate layer with a bunch of bbappend files that are
> >>only
> >>BBCLASSEXTEND += " native "
> >>I suppose if I never needed them, I could just not enable that layer.
> >>
> >>>
> >>>The main reason I've been against native everywhere is that having
> >>>native versions available makes it far too easy for people to add native
> >>>dependencies which encourage feature creep without thinking through the
> >>>huge additional dependency chains, the extra build time and other
> >>>implications. Often there are slightly more difficult but worthwhile
> >>>ways we can avoid the native dependency.
> >>
> >>Such as? I started down this path needing a native tool (which admittedly
> >>came from an OE recipe librsvg) which then cascaded into cairo-native and
> >>beyond, totally 22 packages!. If I knew of a short circuit for this, I'd
> >>certainly entertain it.
> >
> >Having just skimmed the OE recipes, yeah, there we just build a ton of stuff in librsvg-native. But I would start by looking at what we can pass to configure to disable things as
> >part of the host tool build.
> >
> >Doing some git grep'ing I see there's just a few things in OE that need librsvg-native so perhaps we can build a much more limited native recipe instead and still generate what we
> >need to.
> 
> What I really needed was rsvg-convert which is used during the build of
> midori.  In Poky/Yocto, that tool has to come from librsvg-native.  OE
> seems to be happy leaning on the host version.

I think having some buildhost/native packages (based on distro/version) trusted and assume provided or even
installed using the build host package management could be one approach
in modern distros I think we could use lot of host packages as is. or
may be trust all and keep a list of blacklisted packages for a given
distro might be something useful

> 
> -- 
> ------------------------------------------------------------
> Gary Thomas                 |  Consulting for the
> MLB Associates              |    Embedded world
> ------------------------------------------------------------
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

-- 
-Khem

Re: Native vs not

[Tom Rini]

On 03/11/2011 05:10 PM, Khem Raj wrote:

> I think having some buildhost/native packages (based on distro/version) trusted and assume provided or even
> installed using the build host package management could be one approach
> in modern distros I think we could use lot of host packages as is. or
> may be trust all and keep a list of blacklisted packages for a given
> distro might be something useful

I've been thinking about this a bit and at least at the high level there 
is an argument for making it easy to opt out of a big class of host 
tools, on a modern distro and trade reprodicibility / reliability for 
faster initial build times.  And on the flip side, opt out of a bit more 
of our host tool dependencies.

-- 
Tom Rini
Mentor Graphics Corporation

Re: Native vs not

[Richard Purdie]

On Mon, 2011-03-14 at 10:26 -0700, Tom Rini wrote:
> On 03/11/2011 05:10 PM, Khem Raj wrote:
> 
> > I think having some buildhost/native packages (based on distro/version) trusted and assume provided or even
> > installed using the build host package management could be one approach
> > in modern distros I think we could use lot of host packages as is. or
> > may be trust all and keep a list of blacklisted packages for a given
> > distro might be something useful
> 
> I've been thinking about this a bit and at least at the high level there 
> is an argument for making it easy to opt out of a big class of host 
> tools, on a modern distro and trade reprodicibility / reliability for 
> faster initial build times.  And on the flip side, opt out of a bit more 
> of our host tool dependencies.

This is what ASSUME_PROVIDED is there for :)

Cheers,

Richard

Re: Native vs not

[Tom Rini]

On 03/14/2011 10:40 AM, Richard Purdie wrote:
> On Mon, 2011-03-14 at 10:26 -0700, Tom Rini wrote:
>> On 03/11/2011 05:10 PM, Khem Raj wrote:
>>
>>> I think having some buildhost/native packages (based on distro/version) trusted and assume provided or even
>>> installed using the build host package management could be one approach
>>> in modern distros I think we could use lot of host packages as is. or
>>> may be trust all and keep a list of blacklisted packages for a given
>>> distro might be something useful
>>
>> I've been thinking about this a bit and at least at the high level there
>> is an argument for making it easy to opt out of a big class of host
>> tools, on a modern distro and trade reprodicibility / reliability for
>> faster initial build times.  And on the flip side, opt out of a bit more
>> of our host tool dependencies.
>
> This is what ASSUME_PROVIDED is there for :)

Well yes.  But it'd be nice if we had a few helpers so you could say:
ASSUME_PROVIDED += "${SCM_FETCH_TOOLS} ${DOCUMENTATION_GENERATION_TOOLS} 
..."
and so forth.

-- 
Tom Rini
Mentor Graphics Corporation

Re: Native vs not

[Richard Purdie]

On Mon, 2011-03-14 at 10:46 -0700, Tom Rini wrote:
> On 03/14/2011 10:40 AM, Richard Purdie wrote:
> > On Mon, 2011-03-14 at 10:26 -0700, Tom Rini wrote:
> >> On 03/11/2011 05:10 PM, Khem Raj wrote:
> >>
> >>> I think having some buildhost/native packages (based on distro/version) trusted and assume provided or even
> >>> installed using the build host package management could be one approach
> >>> in modern distros I think we could use lot of host packages as is. or
> >>> may be trust all and keep a list of blacklisted packages for a given
> >>> distro might be something useful
> >>
> >> I've been thinking about this a bit and at least at the high level there
> >> is an argument for making it easy to opt out of a big class of host
> >> tools, on a modern distro and trade reprodicibility / reliability for
> >> faster initial build times.  And on the flip side, opt out of a bit more
> >> of our host tool dependencies.
> >
> > This is what ASSUME_PROVIDED is there for :)
> 
> Well yes.  But it'd be nice if we had a few helpers so you could say:
> ASSUME_PROVIDED += "${SCM_FETCH_TOOLS} ${DOCUMENTATION_GENERATION_TOOLS} 
> ..."
> and so forth.

We need to user to be very clearly aware of what they're adding IMO so
I'd prefer maybe just some commented out lines such as:

# Uncomment this if you want to use the system provided scm utilities
# instead of having the system build them
#ASSUME_PROVIDED += "git-native svn-native"

I'm actually thinking about a local.conf.sample.advanced file where we
list a load of things like this. This means we can keep
local.conf.sample simple (the things they probably *need* to look at)
yet still show users some of the other more advanced options.

Cheers,

Richard

Re: Native vs not

[Tom Rini]

On 03/14/2011 10:55 AM, Richard Purdie wrote:
> On Mon, 2011-03-14 at 10:46 -0700, Tom Rini wrote:
>> On 03/14/2011 10:40 AM, Richard Purdie wrote:
>>> On Mon, 2011-03-14 at 10:26 -0700, Tom Rini wrote:
>>>> On 03/11/2011 05:10 PM, Khem Raj wrote:
>>>>
>>>>> I think having some buildhost/native packages (based on distro/version) trusted and assume provided or even
>>>>> installed using the build host package management could be one approach
>>>>> in modern distros I think we could use lot of host packages as is. or
>>>>> may be trust all and keep a list of blacklisted packages for a given
>>>>> distro might be something useful
>>>>
>>>> I've been thinking about this a bit and at least at the high level there
>>>> is an argument for making it easy to opt out of a big class of host
>>>> tools, on a modern distro and trade reprodicibility / reliability for
>>>> faster initial build times.  And on the flip side, opt out of a bit more
>>>> of our host tool dependencies.
>>>
>>> This is what ASSUME_PROVIDED is there for :)
>>
>> Well yes.  But it'd be nice if we had a few helpers so you could say:
>> ASSUME_PROVIDED += "${SCM_FETCH_TOOLS} ${DOCUMENTATION_GENERATION_TOOLS}
>> ..."
>> and so forth.
>
> We need to user to be very clearly aware of what they're adding IMO so
> I'd prefer maybe just some commented out lines such as:
>
> # Uncomment this if you want to use the system provided scm utilities
> # instead of having the system build them
> #ASSUME_PROVIDED += "git-native svn-native"
>
> I'm actually thinking about a local.conf.sample.advanced file where we
> list a load of things like this. This means we can keep
> local.conf.sample simple (the things they probably *need* to look at)
> yet still show users some of the other more advanced options.

I agree.  There's very real risk involved in not assuming things.  But 
there's also very real cases where people are fine taking the risk (and 
in some cases it's not really that risky, eg scm fetching tools, but 
others really are).

-- 
Tom Rini
Mentor Graphics Corporation

Re: Native vs not

[Gary Thomas]

On 03/14/2011 12:03 PM, Tom Rini wrote:
> On 03/14/2011 10:55 AM, Richard Purdie wrote:
>> On Mon, 2011-03-14 at 10:46 -0700, Tom Rini wrote:
>>> On 03/14/2011 10:40 AM, Richard Purdie wrote:
>>>> On Mon, 2011-03-14 at 10:26 -0700, Tom Rini wrote:
>>>>> On 03/11/2011 05:10 PM, Khem Raj wrote:
>>>>>
>>>>>> I think having some buildhost/native packages (based on distro/version) trusted and assume provided or even
>>>>>> installed using the build host package management could be one approach
>>>>>> in modern distros I think we could use lot of host packages as is. or
>>>>>> may be trust all and keep a list of blacklisted packages for a given
>>>>>> distro might be something useful
>>>>>
>>>>> I've been thinking about this a bit and at least at the high level there
>>>>> is an argument for making it easy to opt out of a big class of host
>>>>> tools, on a modern distro and trade reprodicibility / reliability for
>>>>> faster initial build times. And on the flip side, opt out of a bit more
>>>>> of our host tool dependencies.
>>>>
>>>> This is what ASSUME_PROVIDED is there for :)
>>>
>>> Well yes. But it'd be nice if we had a few helpers so you could say:
>>> ASSUME_PROVIDED += "${SCM_FETCH_TOOLS} ${DOCUMENTATION_GENERATION_TOOLS}
>>> ..."
>>> and so forth.
>>
>> We need to user to be very clearly aware of what they're adding IMO so
>> I'd prefer maybe just some commented out lines such as:
>>
>> # Uncomment this if you want to use the system provided scm utilities
>> # instead of having the system build them
>> #ASSUME_PROVIDED += "git-native svn-native"
>>
>> I'm actually thinking about a local.conf.sample.advanced file where we
>> list a load of things like this. This means we can keep
>> local.conf.sample simple (the things they probably *need* to look at)
>> yet still show users some of the other more advanced options.
>
> I agree. There's very real risk involved in not assuming things. But there's also very real cases where people are fine taking the risk (and in some cases it's not really that
> risky, eg scm fetching tools, but others really are).

I fully agree with the basic idea, but I'm not sure it solves everything.
I ended up down this road when a configure step tried to use a host tool,
in this case /usr/bin/rsvg-convert.  I think there still would have been
some grousing even if I had ASSUME_PROVIDED="librsvg-native" which is
the package I ended up needing to build.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------