[PATCH v2 0/1] Add recipe whitelisting class

[PATCH v2 0/1] Add recipe whitelisting class

[Paul Eggleton]

Changes since the RFC version:
* Check BPN in addition to PN for matching, to make it easier in the
  multilib case
* Use PNWHITELIST_LAYERS to specify which layers to apply whitelisting to,
  and consider an empty whitelist value to be no recipes whitelisted (so
  you can set up empty whitelists for layers not yet in your
  configuration if you wish to do so). This also makes setting
  PNWHITELIST without an override practical.
* Touch up the comments

The following changes since commit f07045fcae859c902434062d1725f1348f42d1dd:

  oeqa/oetest.py: add better package search for hasPackage() (2015-08-26 08:26:37 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/whitelist
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/whitelist

Paul Eggleton (1):
  classes/whitelist: add class to allow whitelisting recipes from a
    layer

 meta/classes/whitelist.bbclass | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 meta/classes/whitelist.bbclass

-- 
2.1.0

[PATCH v2 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer

[Paul Eggleton]

Allow restricting recipes brought from a layer to a specified list. This
is similar in operation to blacklist.bbclass, but instead specifies a
per-layer whitelist of recipes (matched by PN or BPN) that are able to
be built from the layer - anything else is skipped. This is potentially
useful where you want to bring in a select set of recipes from a larger
layer e.g. meta-oe.

Implements [YOCTO #8150].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/whitelist.bbclass | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 meta/classes/whitelist.bbclass

diff --git a/meta/classes/whitelist.bbclass b/meta/classes/whitelist.bbclass
new file mode 100644
index 0000000..ab6ac73
--- /dev/null
+++ b/meta/classes/whitelist.bbclass
@@ -0,0 +1,34 @@
+# Class that allows you to restrict the recipes brought from a layer to
+# a specified list. This is similar in operation to blacklist.bbclass
+# but note the difference in how PNWHITELIST is set - we don't use varflags
+# here, the recipe name goes in the value and we use an override for the
+# layer name (although this is not strictly required - you can have one
+# PNWHITELIST value shared by all of the layers specified in
+# PNWHITELIST_LAYERS). The layer name used here is actually the name that
+# gets added to BBFILE_COLLECTIONS in the layer's layer.conf, which may
+# differ from how the layer is otherwise known - e.g. meta-oe uses
+# "openembedded-layer".
+#
+# INHERIT += "whitelist"
+# PNWHITELIST_LAYERS = "layername"
+# PNWHITELIST_layername = "recipe1 recipe2"
+#
+# If you would prefer to set a reason message other than the default, you
+# can do so:
+#
+# PNWHITELIST_REASON_layername = "not supported by ${DISTRO}"
+
+python() {
+    layer = bb.utils.get_file_layer(d.getVar('FILE', True), d)
+    if layer:
+        layers = (d.getVar('PNWHITELIST_LAYERS', True) or '').split()
+        if layer in layers:
+            localdata = bb.data.createCopy(d)
+            localdata.setVar('OVERRIDES', layer)
+            whitelist = (localdata.getVar('PNWHITELIST', True) or '').split()
+            if not (d.getVar('PN', True) in whitelist or d.getVar('BPN', True) in whitelist):
+                reason = localdata.getVar('PNWHITELIST_REASON', True)
+                if not reason:
+                    reason = 'not in PNWHITELIST for layer %s' % layer
+                raise bb.parse.SkipRecipe(reason)
+}
-- 
2.1.0

Re: [PATCH v2 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 3036 bytes --]


> On Aug 27, 2015, at 5:36 AM, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> 
> Allow restricting recipes brought from a layer to a specified list. This
> is similar in operation to blacklist.bbclass, but instead specifies a
> per-layer whitelist of recipes (matched by PN or BPN) that are able to
> be built from the layer - anything else is skipped. This is potentially
> useful where you want to bring in a select set of recipes from a larger
> layer e.g. meta-oe.

For the record, I am not in favor of such feature in OE, Distros can still use BBMASK if they wish to


> 
> Implements [YOCTO #8150].
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
> meta/classes/whitelist.bbclass | 34 ++++++++++++++++++++++++++++++++++
> 1 file changed, 34 insertions(+)
> create mode 100644 meta/classes/whitelist.bbclass
> 
> diff --git a/meta/classes/whitelist.bbclass b/meta/classes/whitelist.bbclass
> new file mode 100644
> index 0000000..ab6ac73
> --- /dev/null
> +++ b/meta/classes/whitelist.bbclass
> @@ -0,0 +1,34 @@
> +# Class that allows you to restrict the recipes brought from a layer to
> +# a specified list. This is similar in operation to blacklist.bbclass
> +# but note the difference in how PNWHITELIST is set - we don't use varflags
> +# here, the recipe name goes in the value and we use an override for the
> +# layer name (although this is not strictly required - you can have one
> +# PNWHITELIST value shared by all of the layers specified in
> +# PNWHITELIST_LAYERS). The layer name used here is actually the name that
> +# gets added to BBFILE_COLLECTIONS in the layer's layer.conf, which may
> +# differ from how the layer is otherwise known - e.g. meta-oe uses
> +# "openembedded-layer".
> +#
> +# INHERIT += "whitelist"
> +# PNWHITELIST_LAYERS = "layername"
> +# PNWHITELIST_layername = "recipe1 recipe2"
> +#
> +# If you would prefer to set a reason message other than the default, you
> +# can do so:
> +#
> +# PNWHITELIST_REASON_layername = "not supported by ${DISTRO}"
> +
> +python() {
> +    layer = bb.utils.get_file_layer(d.getVar('FILE', True), d)
> +    if layer:
> +        layers = (d.getVar('PNWHITELIST_LAYERS', True) or '').split()
> +        if layer in layers:
> +            localdata = bb.data.createCopy(d)
> +            localdata.setVar('OVERRIDES', layer)
> +            whitelist = (localdata.getVar('PNWHITELIST', True) or '').split()
> +            if not (d.getVar('PN', True) in whitelist or d.getVar('BPN', True) in whitelist):
> +                reason = localdata.getVar('PNWHITELIST_REASON', True)
> +                if not reason:
> +                    reason = 'not in PNWHITELIST for layer %s' % layer
> +                raise bb.parse.SkipRecipe(reason)
> +}
> --
> 2.1.0
> 
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH v2 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer

[Paul Eggleton]

On Thursday 27 August 2015 23:07:44 Khem Raj wrote:
> > On Aug 27, 2015, at 5:36 AM, Paul Eggleton <paul.eggleton@linux.intel.com>
> > wrote:
> > 
> > Allow restricting recipes brought from a layer to a specified list. This
> > is similar in operation to blacklist.bbclass, but instead specifies a
> > per-layer whitelist of recipes (matched by PN or BPN) that are able to
> > be built from the layer - anything else is skipped. This is potentially
> > useful where you want to bring in a select set of recipes from a larger
> > layer e.g. meta-oe.
> 
> For the record, I am not in favor of such feature in OE, Distros can still
> use BBMASK if they wish to

This is significantly easier to use than BBMASK though. (Perhaps that's why you 
don't like it?)

I think we need to be realistic here as I said in my most recent reply to the 
other thread - people will do this using whatever mechanism is available 
whether we like it or not. And what are they saying by doing so? They're 
effectively telling us that they can't consume the recipes in the way we are 
currently collecting them together. We know what the proper fix for the problem 
is (increased splitting of large layers such as meta-oe), it may not be easy 
but other than workarounds like this I don't see an alternative.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH v2 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer

[Paul Eggleton]

On Friday 28 August 2015 09:34:18 Paul Eggleton wrote:
> On Thursday 27 August 2015 23:07:44 Khem Raj wrote:
> > > On Aug 27, 2015, at 5:36 AM, Paul Eggleton
> > > <paul.eggleton@linux.intel.com>
> > > wrote:
> > > 
> > > Allow restricting recipes brought from a layer to a specified list. This
> > > is similar in operation to blacklist.bbclass, but instead specifies a
> > > per-layer whitelist of recipes (matched by PN or BPN) that are able to
> > > be built from the layer - anything else is skipped. This is potentially
> > > useful where you want to bring in a select set of recipes from a larger
> > > layer e.g. meta-oe.
> > 
> > For the record, I am not in favor of such feature in OE, Distros can still
> > use BBMASK if they wish to
> 
> This is significantly easier to use than BBMASK though. (Perhaps that's why
> you don't like it?)

I should expand upon this with a couple of important distinctions:

* BBMASK is a list of files to mask out, so it's a blacklist not a whitelist. 
(Sure, you could probably construct a regex that effectively gives you a 
whitelist; it probably won't be pretty though.)

* BBMASK tells bitbake not to even try reading the masked files, which means if 
you then request to build any of them (either directly or via a dependency) 
you'll be told that there's nothing providing that with no explanation as to 
why. With whitelist.bbclass (and blacklist.bbclass, which whitelist.bbclass is 
modelled after) we are using SkipPackage/SkipRecipe so you at least get a 
statement that the recipe was skipped as well as hopefully an explanatory 
reason as to why.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH v2 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1736 bytes --]


> On Aug 28, 2015, at 1:34 AM, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> 
> On Thursday 27 August 2015 23:07:44 Khem Raj wrote:
>>> On Aug 27, 2015, at 5:36 AM, Paul Eggleton <paul.eggleton@linux.intel.com>
>>> wrote:
>>> 
>>> Allow restricting recipes brought from a layer to a specified list. This
>>> is similar in operation to blacklist.bbclass, but instead specifies a
>>> per-layer whitelist of recipes (matched by PN or BPN) that are able to
>>> be built from the layer - anything else is skipped. This is potentially
>>> useful where you want to bring in a select set of recipes from a larger
>>> layer e.g. meta-oe.
>> 
>> For the record, I am not in favor of such feature in OE, Distros can still
>> use BBMASK if they wish to
> 
> This is significantly easier to use than BBMASK though. (Perhaps that's why you
> don't like it?)

I don’t think its about technical merits of the patch. I understand its pretty well writter.
Its if we want to provide more tools
and ways which can be bad for layer health in general which are besides oe-core.

> 
> I think we need to be realistic here as I said in my most recent reply to the
> other thread - people will do this using whatever mechanism is available
> whether we like it or not. And what are they saying by doing so? They're
> effectively telling us that they can't consume the recipes in the way we are
> currently collecting them together. We know what the proper fix for the problem
> is (increased splitting of large layers such as meta-oe), it may not be easy
> but other than workarounds like this I don't see an alternative.
> 
> Cheers,
> Paul
> 
> --
> 
> Paul Eggleton
> Intel Open Source Technology Centre


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH v2 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer

[Otavio Salvador]

On Fri, Aug 28, 2015 at 3:07 AM, Khem Raj <raj.khem@gmail.com> wrote:
>
>> On Aug 27, 2015, at 5:36 AM, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
>>
>> Allow restricting recipes brought from a layer to a specified list. This
>> is similar in operation to blacklist.bbclass, but instead specifies a
>> per-layer whitelist of recipes (matched by PN or BPN) that are able to
>> be built from the layer - anything else is skipped. This is potentially
>> useful where you want to bring in a select set of recipes from a larger
>> layer e.g. meta-oe.
>
> For the record, I am not in favor of such feature in OE, Distros can still use BBMASK if they wish to

I am also not in favor of the merge of this.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH v2 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer

[Paul Eggleton]

On Friday 28 August 2015 10:55:41 Otavio Salvador wrote:
> On Fri, Aug 28, 2015 at 3:07 AM, Khem Raj <raj.khem@gmail.com> wrote:
> >> On Aug 27, 2015, at 5:36 AM, Paul Eggleton
> >> <paul.eggleton@linux.intel.com> wrote:
> >> 
> >> Allow restricting recipes brought from a layer to a specified list. This
> >> is similar in operation to blacklist.bbclass, but instead specifies a
> >> per-layer whitelist of recipes (matched by PN or BPN) that are able to
> >> be built from the layer - anything else is skipped. This is potentially
> >> useful where you want to bring in a select set of recipes from a larger
> >> layer e.g. meta-oe.
> > 
> > For the record, I am not in favor of such feature in OE, Distros can still
> > use BBMASK if they wish to
>
> I am also not in favor of the merge of this.

Do you have a practical alternative that has less downsides?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH v2 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer

[Otavio Salvador]

On Fri, Aug 28, 2015 at 11:00 AM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> On Friday 28 August 2015 10:55:41 Otavio Salvador wrote:
>> On Fri, Aug 28, 2015 at 3:07 AM, Khem Raj <raj.khem@gmail.com> wrote:
>> >> On Aug 27, 2015, at 5:36 AM, Paul Eggleton
>> >> <paul.eggleton@linux.intel.com> wrote:
>> >>
>> >> Allow restricting recipes brought from a layer to a specified list. This
>> >> is similar in operation to blacklist.bbclass, but instead specifies a
>> >> per-layer whitelist of recipes (matched by PN or BPN) that are able to
>> >> be built from the layer - anything else is skipped. This is potentially
>> >> useful where you want to bring in a select set of recipes from a larger
>> >> layer e.g. meta-oe.
>> >
>> > For the record, I am not in favor of such feature in OE, Distros can still
>> > use BBMASK if they wish to
>>
>> I am also not in favor of the merge of this.
>
> Do you have a practical alternative that has less downsides?

I do want people using full layers and offer tools to make fancy
combinations does not enhance our quality in long term.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH v2 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer

[Paul Eggleton]

On Friday 28 August 2015 11:09:19 Otavio Salvador wrote:
> On Fri, Aug 28, 2015 at 11:00 AM, Paul Eggleton
> 
> <paul.eggleton@linux.intel.com> wrote:
> > On Friday 28 August 2015 10:55:41 Otavio Salvador wrote:
> >> On Fri, Aug 28, 2015 at 3:07 AM, Khem Raj <raj.khem@gmail.com> wrote:
> >> >> On Aug 27, 2015, at 5:36 AM, Paul Eggleton
> >> >> <paul.eggleton@linux.intel.com> wrote:
> >> >> 
> >> >> Allow restricting recipes brought from a layer to a specified list.
> >> >> This
> >> >> is similar in operation to blacklist.bbclass, but instead specifies a
> >> >> per-layer whitelist of recipes (matched by PN or BPN) that are able to
> >> >> be built from the layer - anything else is skipped. This is
> >> >> potentially
> >> >> useful where you want to bring in a select set of recipes from a
> >> >> larger
> >> >> layer e.g. meta-oe.
> >> > 
> >> > For the record, I am not in favor of such feature in OE, Distros can
> >> > still
> >> > use BBMASK if they wish to
> >> 
> >> I am also not in favor of the merge of this.
> > 
> > Do you have a practical alternative that has less downsides?
> 
> I do want people using full layers and offer tools to make fancy
> combinations does not enhance our quality in long term.

OK, sure - but you do appreciate that we do have an underlying problem here 
irrespective of whether this class gets taken into OE-Core or not, right?

Look, if people are really opposed to this then fine, I withdraw it. I just 
hope that it's understood that this class or other equivalent methods *will* 
get used until we fix the problem properly - all it means is the class will get 
copied around instead of being in one place.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH v2 1/1] classes/whitelist: add class to allow whitelisting recipes from a layer

[Otavio Salvador]

On Fri, Aug 28, 2015 at 11:19 AM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> On Friday 28 August 2015 11:09:19 Otavio Salvador wrote:
>> On Fri, Aug 28, 2015 at 11:00 AM, Paul Eggleton
>>
>> <paul.eggleton@linux.intel.com> wrote:
>> > On Friday 28 August 2015 10:55:41 Otavio Salvador wrote:
>> >> On Fri, Aug 28, 2015 at 3:07 AM, Khem Raj <raj.khem@gmail.com> wrote:
>> >> >> On Aug 27, 2015, at 5:36 AM, Paul Eggleton
>> >> >> <paul.eggleton@linux.intel.com> wrote:
>> >> >>
>> >> >> Allow restricting recipes brought from a layer to a specified list.
>> >> >> This
>> >> >> is similar in operation to blacklist.bbclass, but instead specifies a
>> >> >> per-layer whitelist of recipes (matched by PN or BPN) that are able to
>> >> >> be built from the layer - anything else is skipped. This is
>> >> >> potentially
>> >> >> useful where you want to bring in a select set of recipes from a
>> >> >> larger
>> >> >> layer e.g. meta-oe.
>> >> >
>> >> > For the record, I am not in favor of such feature in OE, Distros can
>> >> > still
>> >> > use BBMASK if they wish to
>> >>
>> >> I am also not in favor of the merge of this.
>> >
>> > Do you have a practical alternative that has less downsides?
>>
>> I do want people using full layers and offer tools to make fancy
>> combinations does not enhance our quality in long term.
>
> OK, sure - but you do appreciate that we do have an underlying problem here
> irrespective of whether this class gets taken into OE-Core or not, right?
>
> Look, if people are really opposed to this then fine, I withdraw it. I just
> hope that it's understood that this class or other equivalent methods *will*
> get used until we fix the problem properly - all it means is the class will get
> copied around instead of being in one place.

I don't think we (as OE) should help people to do bad things; people
are good in inventing ways of doing bad things and we should try to
provide them guidance to avoid those, not promote tools to help them
in doing bad development practices.

OE-Core shouldn't handle and maintain this infrastructure, in my view.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH v2 0/1] Add recipe whitelisting class

[Patrick Ohly]

On Thu, 2015-08-27 at 13:36 +0100, Paul Eggleton wrote:
> Changes since the RFC version:
> * Check BPN in addition to PN for matching, to make it easier in the
>   multilib case
> * Use PNWHITELIST_LAYERS to specify which layers to apply whitelisting to,
>   and consider an empty whitelist value to be no recipes whitelisted (so
>   you can set up empty whitelists for layers not yet in your
>   configuration if you wish to do so). This also makes setting
>   PNWHITELIST without an override practical.
> * Touch up the comments

I've tried various configuration variations and it works fine now, so +1
from me.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.