design question: should layer.conf contain "PREFERRED_VERSION" settings?

design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Robert P. J. Day]

  i'm just adding some content to intro OE/YP wiki pages about what
various OE files should contain, and what they shouldn't contain, and
this is the layer.conf file for the meta-openstack layer:

  # We have a conf and classes directory, append to BBPATH
  BBPATH .= ":${LAYERDIR}"

  # We have a recipes directory, add to BBFILES
  BBFILES += "${LAYERDIR}/recipes*/*/*.bb
  ${LAYERDIR}/recipes*/*/*.bbappend"

  BBFILE_COLLECTIONS += "openstack-layer"
  BBFILE_PATTERN_openstack-layer := "^${LAYERDIR}/"
  BBFILE_PRIORITY_openstack-layer = "5"

  BB_DANGLINGAPPENDS_WARNONLY ?= "true"

  # pre 0.4.0 greenlet versions segfault python
  PREFERRED_VERSION_python-greenlet = "0.4.5"
  PREFERRED_VERSION_python-sqlalchemy = "1.0.8"
  PREFERRED_VERSION_python-pexpect = "3.3"
  PREFERRED_VERSION_python-futures = "3.0.3%"
  PREFERRED_VERSION_python-django = "1.8.6"
  PREFERRED_VERSION_python-netaddr = "0.7.19"
  PREFERRED_VERSION_python-sqlalchemy = "1.0.16"
  PREFERRED_VERSION_python-requests = "2.12.0"
  PREFERRED_VERSION_python-amqp = "1.4.9"
  PREFERRED_VERSION_python-kombu = "3.0.%"

so, just to be clear, the simple act of adding that layer to a
project's bblayers.conf file will cause those preferred versions to
be selected, correct?

  is that considered good design? in a perfect world, i've always
believed that you should be able to load up your bblayers.conf file
with as many extraneous layers as you want and (well, with some
exceptions), if you choose not to take advantage of the contents of a
layer, it shouldn't hurt you.

  so doesn't the above lock you into those preferred versions,
regardless of whether you use anything from the meta-openstack layer
or not? wouldn't those selections be properly placed elsewhere in the
layer, and not in layer.conf?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

On 22 March 2017 at 11:14, Robert P. J. Day <rpjday@crashcourse.ca> wrote:

> so, just to be clear, the simple act of adding that layer to a
> project's bblayers.conf file will cause those preferred versions to
> be selected, correct?
>
>   is that considered good design? in a perfect world, i've always
> believed that you should be able to load up your bblayers.conf file
> with as many extraneous layers as you want and (well, with some
> exceptions), if you choose not to take advantage of the contents of a
> layer, it shouldn't hurt you.
>

Yes.

No.

It's a violation of the layer design rules and should result in the new
layer compat tools failing the layer.

Ross

[-- Attachment #2: Type: text/html, Size: 1256 bytes --]

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]

On Wed, Mar 22, 2017 at 7:19 AM, Burton, Ross <ross.burton@intel.com> wrote:

>
> On 22 March 2017 at 11:14, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>
>> so, just to be clear, the simple act of adding that layer to a
>> project's bblayers.conf file will cause those preferred versions to
>> be selected, correct?
>>
>>   is that considered good design? in a perfect world, i've always
>> believed that you should be able to load up your bblayers.conf file
>> with as many extraneous layers as you want and (well, with some
>> exceptions), if you choose not to take advantage of the contents of a
>> layer, it shouldn't hurt you.
>>
>
> Yes.
>
> No.
>

So where are they supposed to be specified ? Surely not a separate distro
.conf ? and most
certainly not in local.conf ... I've always been unsure of where the right
place to specify
versions like that.

Bruce


>
> It's a violation of the layer design rules and should result in the new
> layer compat tools failing the layer.
>
> Ross
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>


-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end"

[-- Attachment #2: Type: text/html, Size: 2855 bytes --]

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 475 bytes --]

On 22 March 2017 at 11:57, Bruce Ashfield <bruce.ashfield@gmail.com> wrote:

> So where are they supposed to be specified ? Surely not a separate distro
> .conf ? and most
> certainly not in local.conf ... I've always been unsure of where the right
> place to specify
> versions like that.
>

If it were my layer I'd have an .inc file that set all the preferred
versions where the defaults don't work, that the user would have to include
in their distro.

Ross

[-- Attachment #2: Type: text/html, Size: 867 bytes --]

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 1185 bytes --]

On Wed, Mar 22, 2017 at 8:02 AM, Burton, Ross <ross.burton@intel.com> wrote:

>
> On 22 March 2017 at 11:57, Bruce Ashfield <bruce.ashfield@gmail.com>
> wrote:
>
>> So where are they supposed to be specified ? Surely not a separate distro
>> .conf ? and most
>> certainly not in local.conf ... I've always been unsure of where the
>> right place to specify
>> versions like that.
>>
>
> If it were my layer I'd have an .inc file that set all the preferred
> versions where the defaults don't work, that the user would have to include
> in their distro.
>

Aha. Kind of like the old versions file that used to float around.

Seems reasonable (if a bit less automatic than someone like me who wants ..
to use the layer like a black box)

Luckily I have push access to that layer, so I can make a change like that
;)

One final question, is it considered a design option to set those versions
triggered off an IMAGE or DISTRO
feature ? i.e. in anonymous python .. or is that already too late in the
parsing/resolution process ?

Bruce




>
> Ross
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end"

[-- Attachment #2: Type: text/html, Size: 2315 bytes --]

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Robert P. J. Day]

[-- Attachment #1: Type: text/plain, Size: 2090 bytes --]


  proper attributions seem to have been totally lost here ...

On Wed, 22 Mar 2017, Bruce Ashfield wrote:

> On Wed, Mar 22, 2017 at 8:02 AM, Burton, Ross <ross.burton@intel.com> wrote:
>
>       On 22 March 2017 at 11:57, Bruce Ashfield <bruce.ashfield@gmail.com> wrote:
>             So where are they supposed to be specified ? Surely not a separate distro .conf ? and most
> certainly not in local.conf ... I've always been unsure of where the right place to specify 
> versions like that.
>
> If it were my layer I'd have an .inc file that set all the preferred
> versions where the defaults don't work, that the user would have to
> include in their distro.
>
> Aha. Kind of like the old versions file that used to float around.
>
> Seems reasonable (if a bit less automatic than someone like me who
> wants .. to use the layer like a black box)
>
> Luckily I have push access to that layer, so I can make a change
> like that ;)
>
> One final question, is it considered a design option to set those
> versions triggered off an IMAGE or DISTRO feature ? i.e. in
> anonymous python .. or is that already too late in the
> parsing/resolution process ?
>
> Bruce

  sorry, didn't mean to start something this early in the morning ...
ok, i did. :-)

  in any event, can we agree that:

1) it's bad layer design if the simple act of including a layer in
bblayers.conf suddenly causes weird things to happen like the above?

2) regardless of how the developer eventually does it, picking up
those PREFERRED VERSIONS from meta-openstack should require *some*
kind of explicit selection?

  thank you kindly.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 3180 bytes --]

On Wed, Mar 22, 2017 at 8:24 AM, Robert P. J. Day <rpjday@crashcourse.ca>
wrote:

>
>   proper attributions seem to have been totally lost here ...
>
> On Wed, 22 Mar 2017, Bruce Ashfield wrote:
>
> > On Wed, Mar 22, 2017 at 8:02 AM, Burton, Ross <ross.burton@intel.com>
> wrote:
> >
> >       On 22 March 2017 at 11:57, Bruce Ashfield <
> bruce.ashfield@gmail.com> wrote:
> >             So where are they supposed to be specified ? Surely not a
> separate distro .conf ? and most
> > certainly not in local.conf ... I've always been unsure of where the
> right place to specify
> > versions like that.
> >
> > If it were my layer I'd have an .inc file that set all the preferred
> > versions where the defaults don't work, that the user would have to
> > include in their distro.
> >
> > Aha. Kind of like the old versions file that used to float around.
> >
> > Seems reasonable (if a bit less automatic than someone like me who
> > wants .. to use the layer like a black box)
> >
> > Luckily I have push access to that layer, so I can make a change
> > like that ;)
> >
> > One final question, is it considered a design option to set those
> > versions triggered off an IMAGE or DISTRO feature ? i.e. in
> > anonymous python .. or is that already too late in the
> > parsing/resolution process ?
> >
> > Bruce
>
>   sorry, didn't mean to start something this early in the morning ...
> ok, i did. :-)
>
>   in any event, can we agree that:
>
> 1) it's bad layer design if the simple act of including a layer in
> bblayers.conf suddenly causes weird things to happen like the above?
>
> 2) regardless of how the developer eventually does it, picking up
> those PREFERRED VERSIONS from meta-openstack should require *some*
> kind of explicit selection?
>


Honestly .. I disagree on #2.

The packages within the openstack layer *will not work* with other versions
of those
packages.

So if someone actually wants to use the functionality provided by the
layer, they are
not optional. And the errors/issues that you get are extremely obtuse and
hard to
debug. That's the design point of the layer .. it doesn't let that happen.

Hence making it something they must do as an extra step .. really is a bad
idea.

It gets in the way of people picking packages out of that layer if they
don't want to
actually build openstack, that's the crux of the problem. But to answer
that issue,
that's why meta-cloud-services exists, that doesn't' force versions and
holds the
generally useful "cloud" packages.

Cheers,

Bruce


>
>   thank you kindly.
>
> rday
>
> --
>
> ========================================================================
> Robert P. J. Day                                 Ottawa, Ontario, CANADA
>                         http://crashcourse.ca
>
> Twitter:                                       http://twitter.com/rpjday
> LinkedIn:                               http://ca.linkedin.com/in/rpjday
> ========================================================================
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end"

[-- Attachment #2: Type: text/html, Size: 4657 bytes --]

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Robert P. J. Day]

[-- Attachment #1: Type: text/plain, Size: 3376 bytes --]

On Wed, 22 Mar 2017, Bruce Ashfield wrote:

>
>
> On Wed, Mar 22, 2017 at 8:24 AM, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>
>         proper attributions seem to have been totally lost here ...
>
>       On Wed, 22 Mar 2017, Bruce Ashfield wrote:
>
>       > On Wed, Mar 22, 2017 at 8:02 AM, Burton, Ross <ross.burton@intel.com> wrote:
>       >
>       >       On 22 March 2017 at 11:57, Bruce Ashfield <bruce.ashfield@gmail.com> wrote:
>       >             So where are they supposed to be specified ? Surely not a separate distro .conf ? and most
>       > certainly not in local.conf ... I've always been unsure of where the right place to specify 
>       > versions like that.
>       >
>       > If it were my layer I'd have an .inc file that set all the preferred
>       > versions where the defaults don't work, that the user would have to
>       > include in their distro.
>       >
>       > Aha. Kind of like the old versions file that used to float around.
>       >
>       > Seems reasonable (if a bit less automatic than someone like me who
>       > wants .. to use the layer like a black box)
>       >
>       > Luckily I have push access to that layer, so I can make a change
>       > like that ;)
>       >
>       > One final question, is it considered a design option to set those
>       > versions triggered off an IMAGE or DISTRO feature ? i.e. in
>       > anonymous python .. or is that already too late in the
>       > parsing/resolution process ?
>       >
>       > Bruce
>
>         sorry, didn't mean to start something this early in the morning ...
>       ok, i did. :-)
>
>         in any event, can we agree that:
>
>       1) it's bad layer design if the simple act of including a layer in
>       bblayers.conf suddenly causes weird things to happen like the above?
>
>       2) regardless of how the developer eventually does it, picking up
>       those PREFERRED VERSIONS from meta-openstack should require *some*
>       kind of explicit selection?
>
> Honestly .. I disagree on #2.
>
> The packages within the openstack layer *will not work* with other
> versions of those packages.
>
> So if someone actually wants to use the functionality provided by
> the layer, they are not optional. And the errors/issues that you get
> are extremely obtuse and hard to debug. That's the design point of
> the layer .. it doesn't let that happen.
>
> Hence making it something they must do as an extra step .. really is
> a bad idea.
>
> It gets in the way of people picking packages out of that layer if
> they don't want to actually build openstack, that's the crux of the
> problem. But to answer that issue, that's why meta-cloud-services
> exists, that doesn't' force versions and holds the generally useful
> "cloud" packages.

  ok, i see your point ... i will ponder this. it still seems ...
unclean.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Richard Purdie]

On Wed, 2017-03-22 at 08:33 -0400, Bruce Ashfield wrote:
> 
> 
> On Wed, Mar 22, 2017 at 8:24 AM, Robert P. J. Day <rpjday@crashcourse
> .ca> wrote:
> > 
> >   proper attributions seem to have been totally lost here ...
> > 
> > On Wed, 22 Mar 2017, Bruce Ashfield wrote:
> > 
> > > On Wed, Mar 22, 2017 at 8:02 AM, Burton, Ross <ross.burton@intel.
> > com> wrote:
> > >
> > >       On 22 March 2017 at 11:57, Bruce Ashfield <bruce.ashfield@g
> > mail.com> wrote:
> > >             So where are they supposed to be specified ? Surely
> > not a separate distro .conf ? and most
> > > certainly not in local.conf ... I've always been unsure of where
> > the right place to specify 
> > > versions like that.
> > >
> > > If it were my layer I'd have an .inc file that set all the
> > preferred
> > > versions where the defaults don't work, that the user would have
> > to
> > > include in their distro.
> > >
> > > Aha. Kind of like the old versions file that used to float
> > around.
> > >
> > > Seems reasonable (if a bit less automatic than someone like me
> > who
> > > wants .. to use the layer like a black box)
> > >
> > > Luckily I have push access to that layer, so I can make a change
> > > like that ;)
> > >
> > > One final question, is it considered a design option to set those
> > > versions triggered off an IMAGE or DISTRO feature ? i.e. in
> > > anonymous python .. or is that already too late in the
> > > parsing/resolution process ?
> > >
> > > Bruce
> > 
> >   sorry, didn't mean to start something this early in the morning
> > ...
> > ok, i did. :-)
> > 
> >   in any event, can we agree that:
> > 
> > 1) it's bad layer design if the simple act of including a layer in
> > bblayers.conf suddenly causes weird things to happen like the
> > above?
> > 
> > 2) regardless of how the developer eventually does it, picking up
> > those PREFERRED VERSIONS from meta-openstack should require *some*
> > kind of explicit selection?
> 
> Honestly .. I disagree on #2.
> 
> The packages within the openstack layer *will not work* with other
> versions of those
> packages. 
> 
> So if someone actually wants to use the functionality provided by the
> layer, they are
> not optional. And the errors/issues that you get are extremely obtuse
> and hard to
> debug. That's the design point of the layer .. it doesn't let that
> happen.
> 
> Hence making it something they must do as an extra step .. really is
> a bad idea.
> 
> It gets in the way of people picking packages out of that layer if
> they don't want to 
> actually build openstack, that's the crux of the problem. But to
> answer that issue,
> that's why meta-cloud-services exists, that doesn't' force versions
> and holds the
> generally useful "cloud" packages.

Having layers which magically change the "policy" such as selected
versions goes against the spirit and wording of Yocto Project
Compatible (and will go against the new layer checking tools).

What should happen is that the openstack recipe should complain loudly
(error) if anyone tried to build it (or parse it?) with anything except
the versions its known to work with.

Cheers,

Richard

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

On 22 March 2017 at 14:59, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> What should happen is that the openstack recipe should complain loudly
> (error) if anyone tried to build it (or parse it?) with anything except
> the versions its known to work with.
>

Is this a good time to bring up versions in DEPENDS?

It would be awesome to automatically set PREFERRED_VERSION as appropriate
depending on the constrains in DEPENDS, but I'd be really happy with errors
if the versions specified were not selected.

So in this case openstack could do DEPENDS=python-something (>4.0) and
bitbake would error out if python-something v3 is being built.

Ross

[-- Attachment #2: Type: text/html, Size: 1236 bytes --]

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Richard Purdie]

On Wed, 2017-03-22 at 15:44 +0000, Burton, Ross wrote:
> 
> On 22 March 2017 at 14:59, Richard Purdie <richard.purdie@linuxfounda
> tion.org> wrote:
> > What should happen is that the openstack recipe should complain
> > loudly
> > (error) if anyone tried to build it (or parse it?) with anything
> > except
> > the versions its known to work with.
> > 
> Is this a good time to bring up versions in DEPENDS?

Yes. I'd swear there was a bug for this but I can't find it. I know we
made versions able to be added to DEPENDS without parse failures, its
just a question of getting bitbake to parse them.

> It would be awesome to automatically set PREFERRED_VERSION as
> appropriate depending on the constrains in DEPENDS, but I'd be really
> happy with errors if the versions specified were not selected.
> 
> So in this case openstack could do DEPENDS=python-something (>4.0)
> and bitbake would error out if python-something v3 is being built.

I don't think setting PREFERRED_VERSION magically would be good but I
do like the rest of the idea, its always been the intent really...

Someone needs to file a bug...

Cheers,

Richard

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 345 bytes --]

On 22 March 2017 at 16:32, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> Someone needs to file a bug...
>

Found https://bugzilla.yoctoproject.org/show_bug.cgi?id=8745.

Currently a future enhancement, I think we should reschedule this to 2.4.
Simply bailing shouldn't be that much effort and would be useful.

Ross

[-- Attachment #2: Type: text/html, Size: 985 bytes --]

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 1161 bytes --]

On Wed, Mar 22, 2017 at 11:44 AM, Burton, Ross <ross.burton@intel.com>
wrote:

>
> On 22 March 2017 at 14:59, Richard Purdie <richard.purdie@
> linuxfoundation.org> wrote:
>
>> What should happen is that the openstack recipe should complain loudly
>> (error) if anyone tried to build it (or parse it?) with anything except
>> the versions its known to work with.
>>
>
> Is this a good time to bring up versions in DEPENDS?
>
> It would be awesome to automatically set PREFERRED_VERSION as appropriate
> depending on the constrains in DEPENDS, but I'd be really happy with errors
> if the versions specified were not selected.
>
> So in this case openstack could do DEPENDS=python-something (>4.0) and
> bitbake would error out if python-something v3 is being built.
>

That would be ideal, and it was actually something I tried in the past. But
it was causing
issues with some package backends, so most of those specifics had to be
dropped.

The details are fuzzy, but I could data mine my email if needed.

Bruce


>
> Ross
>



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end"

[-- Attachment #2: Type: text/html, Size: 2471 bytes --]

Re: design question: should layer.conf contain "PREFERRED_VERSION" settings?

[Daniel Dickinson]

On Wed, 22 Mar 2017 08:33:17 -0400
Bruce Ashfield <bruce.ashfield@gmail.com> wrote:

> On Wed, Mar 22, 2017 at 8:24 AM, Robert P. J. Day
> <rpjday@crashcourse.ca> wrote:
> 
> >
> >   proper attributions seem to have been totally lost here ...
> >
> > On Wed, 22 Mar 2017, Bruce Ashfield wrote:
> >  
> > > On Wed, Mar 22, 2017 at 8:02 AM, Burton, Ross
> > > <ross.burton@intel.com>  
> > wrote:  
> > >
> > >       On 22 March 2017 at 11:57, Bruce Ashfield <
> > bruce.ashfield@gmail.com> wrote:
> > >             So where are they supposed to be specified ? Surely
> > > not a  
> > separate distro .conf ? and most  
> > > certainly not in local.conf ... I've always been unsure of where
> > > the  
> > right place to specify  
> > > versions like that.
> > >
> > > If it were my layer I'd have an .inc file that set all the
> > > preferred versions where the defaults don't work, that the user
> > > would have to include in their distro.
> > >
> > > Aha. Kind of like the old versions file that used to float around.
> > >
> > > Seems reasonable (if a bit less automatic than someone like me who
> > > wants .. to use the layer like a black box)
> > >
> > > Luckily I have push access to that layer, so I can make a change
> > > like that ;)
> > >
> > > One final question, is it considered a design option to set those
> > > versions triggered off an IMAGE or DISTRO feature ? i.e. in
> > > anonymous python .. or is that already too late in the
> > > parsing/resolution process ?
> > >
> > > Bruce  
> >
> >   sorry, didn't mean to start something this early in the
> > morning ... ok, i did. :-)
> >
> >   in any event, can we agree that:
> >
> > 1) it's bad layer design if the simple act of including a layer in
> > bblayers.conf suddenly causes weird things to happen like the above?
> >
> > 2) regardless of how the developer eventually does it, picking up
> > those PREFERRED VERSIONS from meta-openstack should require *some*
> > kind of explicit selection?
> >  
> 
> 
> Honestly .. I disagree on #2.
> 
> The packages within the openstack layer *will not work* with other
> versions of those
> packages.
> 
> So if someone actually wants to use the functionality provided by the
> layer, they are
> not optional. And the errors/issues that you get are extremely obtuse
> and hard to
> debug. That's the design point of the layer .. it doesn't let that
> happen.
> 
> Hence making it something they must do as an extra step .. really is
> a bad idea.
> 
> It gets in the way of people picking packages out of that layer if
> they don't want to
> actually build openstack, that's the crux of the problem. But to
> answer that issue,
> that's why meta-cloud-services exists, that doesn't' force versions
> and holds the
> generally useful "cloud" packages.
> 

Wouldn't that make the openstack layer more of a distro or distro
fragment then?  Wouldn't it make sense to have a openstack.bbclass and
INHERIT in local.conf or distro.conf (depending on whether create a
distro or a one-off test?) that takes care of the specific
dependencies for that specific use-case?

Specific use-cases seems to be the point of distros and images to me,
whereas recipes should be pick and choose...

Regards,

Daniel