[RFC PATCH v5 0/2] Pending changes for review/discussion

[RFC PATCH v5 0/2] Pending changes for review/discussion

[Otavio Salvador]

I am resending those two patches as they are not fully
discussed/reviewed.

The EXTRA_DISTRO_FEATURES (1/2) one has a mix of people which likes it
and others who hates it. Personally I would like to get a mechanism
like this one supported and it allows for much easier use of
DISTRO_FEATURES during development phase. People argue it can be
abused and this is a real possibility and something that will happen
however this happens with everything so it does not justifies, for me,
the not inclusion of it.

On the other side the UBOOT_CONFIG (2/2) one is not complete in my
opinion. It needs someone to help to find the gabs so I can fix those
and propose a final patch.

Changes in v5:
- Resend

Changes in v4:
- Drop _prepend/_append hack and remove/add DISTRO_FEATURES (thanks to
  RP for the tip).

Changes in v3:
- Drop _prepend/_append flags from DISTRO_FEATURES to avoid the re-add

Changes in v2:
- Use data copy to fix wrong prepend/append expanding.

Otavio Salvador (2):
  base.bbclass: Add support to EXTRA_DISTRO_FEATURES
  uboot-config.bbclass: Allow choose of U-Boot config for machine

 meta/classes/base.bbclass         |  4 ++++
 meta/classes/uboot-config.bbclass | 39 +++++++++++++++++++++++++++++++++++++++
 meta/lib/oe/utils.py              | 24 ++++++++++++++++++++++++
 3 files changed, 67 insertions(+)
 create mode 100644 meta/classes/uboot-config.bbclass

-- 
1.8.4.rc1

[RFC PATCH v5 1/2] base.bbclass: Add support to EXTRA_DISTRO_FEATURES

[Otavio Salvador]

This allow the addition and removal of distro features easily. To add
a feature, use:

 EXTRA_DISTRO_FEATURES += "wayland"

and to remove, use '~' prefix, as:

 EXTRA_DISTRO_FEATURES += "~x11"

This code has been mostly copied from Mentor Graphics public layer but
changed the variable name for a more descriptive name. The original
code can be seen at user_feature.bbclass at:

 http://git.yoctoproject.org/cgit/cgit.cgi/meta-mentor

Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
---
Changes in v5:
- Resend

Changes in v4:
- Drop _prepend/_append hack and remove/add DISTRO_FEATURES (thanks to
  RP for the tip).

Changes in v3:
- Drop _prepend/_append flags from DISTRO_FEATURES to avoid the re-add

Changes in v2:
- Use data copy to fix wrong prepend/append expanding.

 meta/classes/base.bbclass |  4 ++++
 meta/lib/oe/utils.py      | 24 ++++++++++++++++++++++++
 2 files changed, 28 insertions(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 98b823e..f30f4db 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -288,6 +288,9 @@ def buildcfg_neededvars(d):
     if pesteruser:
         bb.fatal('The following variable(s) were not set: %s\nPlease set them directly, or choose a MACHINE or DISTRO that sets them.' % ', '.join(pesteruser))
 
+EXTRA_DISTRO_FEATURES ?= ""
+EXTRA_DISTRO_FEATURES[type] = "list"
+
 addhandler base_eventhandler
 base_eventhandler[eventmask] = "bb.event.ConfigParsed bb.event.BuildStarted"
 python base_eventhandler() {
@@ -297,6 +300,7 @@ python base_eventhandler() {
         preferred_ml_updates(e.data)
         oe.utils.features_backfill("DISTRO_FEATURES", e.data)
         oe.utils.features_backfill("MACHINE_FEATURES", e.data)
+        oe.utils.extra_distro_features(e.data)
 
     if isinstance(e, bb.event.BuildStarted):
         localdata = bb.data.createCopy(e.data)
diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index 82987e8..0c9dde0 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -117,6 +117,30 @@ def features_backfill(var,d):
     if addfeatures:
         d.appendVar(var, " " + " ".join(addfeatures))
 
+def extra_distro_features(d):
+    import oe.data
+
+    l = d.createCopy()
+    l.finalize()
+
+    extra_features = oe.data.typed_value('EXTRA_DISTRO_FEATURES', l)
+    if not extra_features:
+        return
+
+    distro_features = l.getVar('DISTRO_FEATURES', True).split()
+    for feature in extra_features:
+        if feature.startswith('~'):
+            feature = feature[1:]
+            if feature in distro_features:
+                distro_features.remove(feature)
+        else:
+            if feature not in distro_features:
+                distro_features.append(feature)
+
+    # Avoid readding the removed features later
+    d.delVar("DISTRO_FEATURES")
+
+    d.setVar('DISTRO_FEATURES', ' '.join(distro_features))
 
 def packages_filter_out_system(d):
     """
-- 
1.8.4.rc1

Re: [RFC PATCH v5 1/2] base.bbclass: Add support to EXTRA_DISTRO_FEATURES

[Richard Purdie]

On Fri, 2013-08-23 at 11:48 -0300, Otavio Salvador wrote:
> This allow the addition and removal of distro features easily. To add
> a feature, use:
> 
>  EXTRA_DISTRO_FEATURES += "wayland"
> 
> and to remove, use '~' prefix, as:
> 
>  EXTRA_DISTRO_FEATURES += "~x11"
> 
> This code has been mostly copied from Mentor Graphics public layer but
> changed the variable name for a more descriptive name. The original
> code can be seen at user_feature.bbclass at:
> 
>  http://git.yoctoproject.org/cgit/cgit.cgi/meta-mentor
> 
> Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>

I discussed this with Otavio. My personal view is that we should hold
off this right now. The removal issues he ran into with poky have been
addressed in other ways so there is less of an immediate issue.

The better fix for this kind of problem is -=/=- or more likely _remove
override support in bitbake. I did propose a _remove patch a while back
but Chris has some valid concerns about it, I don't remember the details
offhand. I'd much rather we focused on fixing that and getting that
merged than trying to add this functionality to every variable.

Cheers,

Richard



> ---
> Changes in v5:
> - Resend
> 
> Changes in v4:
> - Drop _prepend/_append hack and remove/add DISTRO_FEATURES (thanks to
>   RP for the tip).
> 
> Changes in v3:
> - Drop _prepend/_append flags from DISTRO_FEATURES to avoid the re-add
> 
> Changes in v2:
> - Use data copy to fix wrong prepend/append expanding.
> 
>  meta/classes/base.bbclass |  4 ++++
>  meta/lib/oe/utils.py      | 24 ++++++++++++++++++++++++
>  2 files changed, 28 insertions(+)
> 
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 98b823e..f30f4db 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -288,6 +288,9 @@ def buildcfg_neededvars(d):
>      if pesteruser:
>          bb.fatal('The following variable(s) were not set: %s\nPlease set them directly, or choose a MACHINE or DISTRO that sets them.' % ', '.join(pesteruser))
>  
> +EXTRA_DISTRO_FEATURES ?= ""
> +EXTRA_DISTRO_FEATURES[type] = "list"
> +
>  addhandler base_eventhandler
>  base_eventhandler[eventmask] = "bb.event.ConfigParsed bb.event.BuildStarted"
>  python base_eventhandler() {
> @@ -297,6 +300,7 @@ python base_eventhandler() {
>          preferred_ml_updates(e.data)
>          oe.utils.features_backfill("DISTRO_FEATURES", e.data)
>          oe.utils.features_backfill("MACHINE_FEATURES", e.data)
> +        oe.utils.extra_distro_features(e.data)
>  
>      if isinstance(e, bb.event.BuildStarted):
>          localdata = bb.data.createCopy(e.data)
> diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
> index 82987e8..0c9dde0 100644
> --- a/meta/lib/oe/utils.py
> +++ b/meta/lib/oe/utils.py
> @@ -117,6 +117,30 @@ def features_backfill(var,d):
>      if addfeatures:
>          d.appendVar(var, " " + " ".join(addfeatures))
>  
> +def extra_distro_features(d):
> +    import oe.data
> +
> +    l = d.createCopy()
> +    l.finalize()
> +
> +    extra_features = oe.data.typed_value('EXTRA_DISTRO_FEATURES', l)
> +    if not extra_features:
> +        return
> +
> +    distro_features = l.getVar('DISTRO_FEATURES', True).split()
> +    for feature in extra_features:
> +        if feature.startswith('~'):
> +            feature = feature[1:]
> +            if feature in distro_features:
> +                distro_features.remove(feature)
> +        else:
> +            if feature not in distro_features:
> +                distro_features.append(feature)
> +
> +    # Avoid readding the removed features later
> +    d.delVar("DISTRO_FEATURES")
> +
> +    d.setVar('DISTRO_FEATURES', ' '.join(distro_features))
>  
>  def packages_filter_out_system(d):
>      """

Re: [RFC PATCH v5 1/2] base.bbclass: Add support to EXTRA_DISTRO_FEATURES

[Otavio Salvador]

On Fri, Aug 23, 2013 at 11:59 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Fri, 2013-08-23 at 11:48 -0300, Otavio Salvador wrote:
>> This allow the addition and removal of distro features easily. To add
>> a feature, use:
>>
>>  EXTRA_DISTRO_FEATURES += "wayland"
>>
>> and to remove, use '~' prefix, as:
>>
>>  EXTRA_DISTRO_FEATURES += "~x11"
>>
>> This code has been mostly copied from Mentor Graphics public layer but
>> changed the variable name for a more descriptive name. The original
>> code can be seen at user_feature.bbclass at:
>>
>>  http://git.yoctoproject.org/cgit/cgit.cgi/meta-mentor
>>
>> Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
>
> I discussed this with Otavio. My personal view is that we should hold
> off this right now. The removal issues he ran into with poky have been
> addressed in other ways so there is less of an immediate issue.
>
> The better fix for this kind of problem is -=/=- or more likely _remove
> override support in bitbake. I did propose a _remove patch a while back
> but Chris has some valid concerns about it, I don't remember the details
> offhand. I'd much rather we focused on fixing that and getting that
> merged than trying to add this functionality to every variable.

Chris, do you know the problems Richard's patch has? I can take a look
and try to fix it so we can get this in for 1.5.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [RFC PATCH v5 1/2] base.bbclass: Add support to EXTRA_DISTRO_FEATURES

[Richard Purdie]

On Fri, 2013-08-23 at 12:11 -0300, Otavio Salvador wrote:
> On Fri, Aug 23, 2013 at 11:59 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Fri, 2013-08-23 at 11:48 -0300, Otavio Salvador wrote:
> >> This allow the addition and removal of distro features easily. To add
> >> a feature, use:
> >>
> >>  EXTRA_DISTRO_FEATURES += "wayland"
> >>
> >> and to remove, use '~' prefix, as:
> >>
> >>  EXTRA_DISTRO_FEATURES += "~x11"
> >>
> >> This code has been mostly copied from Mentor Graphics public layer but
> >> changed the variable name for a more descriptive name. The original
> >> code can be seen at user_feature.bbclass at:
> >>
> >>  http://git.yoctoproject.org/cgit/cgit.cgi/meta-mentor
> >>
> >> Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
> >
> > I discussed this with Otavio. My personal view is that we should hold
> > off this right now. The removal issues he ran into with poky have been
> > addressed in other ways so there is less of an immediate issue.
> >
> > The better fix for this kind of problem is -=/=- or more likely _remove
> > override support in bitbake. I did propose a _remove patch a while back
> > but Chris has some valid concerns about it, I don't remember the details
> > offhand. I'd much rather we focused on fixing that and getting that
> > merged than trying to add this functionality to every variable.
> 
> Chris, do you know the problems Richard's patch has? I can take a look
> and try to fix it so we can get this in for 1.5.

http://lists.openembedded.org/pipermail/bitbake-devel/2013-March/003319.html

We don't want to rush into adding type support to bitbake, that is
something which needs a lot of thought. It might be good enough to have
_remove assume space delimited for now and we can then have the type
flag alter the behaviour later.

So the thing that needs fixing is the naive str.replace() :)

(x.startswith(y) or x.endswith(y) or " " + y + " " in x) kind of thing.

Cheers,

Richard

Re: [RFC PATCH v5 1/2] base.bbclass: Add support to EXTRA_DISTRO_FEATURES

[Richard Purdie]

On Fri, 2013-08-23 at 12:11 -0300, Otavio Salvador wrote:
> On Fri, Aug 23, 2013 at 11:59 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Fri, 2013-08-23 at 11:48 -0300, Otavio Salvador wrote:
> >> This allow the addition and removal of distro features easily. To add
> >> a feature, use:
> >>
> >>  EXTRA_DISTRO_FEATURES += "wayland"
> >>
> >> and to remove, use '~' prefix, as:
> >>
> >>  EXTRA_DISTRO_FEATURES += "~x11"
> >>
> >> This code has been mostly copied from Mentor Graphics public layer but
> >> changed the variable name for a more descriptive name. The original
> >> code can be seen at user_feature.bbclass at:
> >>
> >>  http://git.yoctoproject.org/cgit/cgit.cgi/meta-mentor
> >>
> >> Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
> >
> > I discussed this with Otavio. My personal view is that we should hold
> > off this right now. The removal issues he ran into with poky have been
> > addressed in other ways so there is less of an immediate issue.
> >
> > The better fix for this kind of problem is -=/=- or more likely _remove
> > override support in bitbake. I did propose a _remove patch a while back
> > but Chris has some valid concerns about it, I don't remember the details
> > offhand. I'd much rather we focused on fixing that and getting that
> > merged than trying to add this functionality to every variable.
> 
> Chris, do you know the problems Richard's patch has? I can take a look
> and try to fix it so we can get this in for 1.5.

One more thing, we need to grep the existing metadata and change any
existing _remove in variable names. I seem to remember finding one
somewhere.

Cheers,

Richard

[RFC PATCH v5 2/2] uboot-config.bbclass: Allow choose of U-Boot config for machine

[Otavio Salvador]

Some machines provide several possible configurations and until now
there was no easy way for user to override the default setting.

This class provides a system similar to PACKAGECONFIG but for
U-Boot. The format is:

UBOOT_CONFIG ??= <default>
UBOOT_CONFIG[foo] = "config,images"

There are two possible parameters:

 - config: it is used to set UBOOT_MACHINE
 - images: it is used to append onto IMAGE_FSTYPES

Below there's an usage example:

,----[ i.MX6Q SABRE AUTO based example ]
| UBOOT_CONFIG ??= "sd"
| UBOOT_CONFIG[sd] = "mx6qsabreauto_config,sdcard"
| UBOOT_CONFIG[eimnor] = "mx6qsabreauto_eimnor_config"
| UBOOT_CONFIG[nand] = "mx6qsabreauto_nand_config,ubifs"
| UBOOT_CONFIG[spinor] = "mx6qsabreauto_spinor_config"
`----

User can, from local.conf or environment, use UBOOT_CONFIG=nand and
override the default setting, as:

,----[ Override example from command line ]
| MACHINE=imx6qsabreauto UBOOT_CONFIG=nand bitbake core-image-base
`----

Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
---
Changes in v5:
- Resend

Changes in v4: None
Changes in v3: None
Changes in v2: None

 meta/classes/uboot-config.bbclass | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
 create mode 100644 meta/classes/uboot-config.bbclass

diff --git a/meta/classes/uboot-config.bbclass b/meta/classes/uboot-config.bbclass
new file mode 100644
index 0000000..fc37620
--- /dev/null
+++ b/meta/classes/uboot-config.bbclass
@@ -0,0 +1,39 @@
+# Allow easy override of U-Boot config for a machine
+#
+# The format to specify it, in the machine, is:
+#
+# UBOOT_CONFIG ??= <default>
+# UBOOT_CONFIG[foo] = "config,images"
+#
+# Copyright 2013 (C) O.S. Systems Software LTDA.
+
+addhandler uboot_config_eventhandler
+uboot_config_eventhandler[eventmask] = "bb.event.ConfigParsed"
+python uboot_config_eventhandler() {
+    ubootconfigflags = e.data.getVarFlags('UBOOT_CONFIG')
+    if not ubootconfigflags:
+        return
+
+    ubootconfig = (e.data.getVar('UBOOT_CONFIG', True) or "").split()
+    if len(ubootconfig) > 1:
+        raise bb.parse.SkipPackage('You can only have a single default for UBOOT_CONFIG.')
+    elif len(ubootconfig) == 0:
+        raise bb.parse.SkipPackage('You must set a default in UBOOT_CONFIG.')
+    ubootconfig = ubootconfig[0]
+
+    for f, v in ubootconfigflags.items():
+        if f == 'defaultval':
+            continue
+
+        items = v.split(',')
+        if items[0] and len(items) > 2:
+            raise bb.parse.SkipPackage('Only config,images can be specified!')
+
+        if ubootconfig == f:
+            bb.debug(1, "Setting UBOOT_MACHINE to %s." % items[0])
+            e.data.setVar('UBOOT_MACHINE', items[0])
+
+            if items[1]:
+                bb.debug(1, "Appending '%s' to IMAGE_FSTYPES." % items[1])
+                e.data.appendVar('IMAGE_FSTYPES', ' ' + items[1])
+}
-- 
1.8.4.rc1

Re: [RFC PATCH v5 2/2] uboot-config.bbclass: Allow choose of U-Boot config for machine

[Otavio Salvador]

On Fri, Aug 23, 2013 at 11:48 AM, Otavio Salvador
<otavio@ossystems.com.br> wrote:
> Some machines provide several possible configurations and until now
> there was no easy way for user to override the default setting.
>
> This class provides a system similar to PACKAGECONFIG but for
> U-Boot. The format is:
>
> UBOOT_CONFIG ??= <default>
> UBOOT_CONFIG[foo] = "config,images"
>
> There are two possible parameters:
>
>  - config: it is used to set UBOOT_MACHINE
>  - images: it is used to append onto IMAGE_FSTYPES
>
> Below there's an usage example:
>
> ,----[ i.MX6Q SABRE AUTO based example ]
> | UBOOT_CONFIG ??= "sd"
> | UBOOT_CONFIG[sd] = "mx6qsabreauto_config,sdcard"
> | UBOOT_CONFIG[eimnor] = "mx6qsabreauto_eimnor_config"
> | UBOOT_CONFIG[nand] = "mx6qsabreauto_nand_config,ubifs"
> | UBOOT_CONFIG[spinor] = "mx6qsabreauto_spinor_config"
> `----
>
> User can, from local.conf or environment, use UBOOT_CONFIG=nand and
> override the default setting, as:
>
> ,----[ Override example from command line ]
> | MACHINE=imx6qsabreauto UBOOT_CONFIG=nand bitbake core-image-base
> `----
>
> Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>

Could someone provide a feedback on this? I'd like to fix the gabs on
this and propose a new version of it.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750