INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

[Saul Wold]

Folks,

I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used 
and processed to possibly include a COMPATIBLE_LICENSES variable as 
well, see PeterK's email [0]

I am trying to determine the usage of WHITELIST_<license> which would be 
used to override a license that might be listed in INCOMPATIBLE_LICENSES 
variable.

Randy and I have done a quick and dirty survey of a 100 or so layers 
(thanks Randy) and could not find any real usage other than what's 
currently in OE-Core for WHITELIST_GPL-3.0.

If you are using WHITELIST_<license>, please let me reply with your usage.


[0] https://lists.openembedded.org/g/openembedded-devel/message/95166
-- 
Sau!

Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

[Andre McCurdy]

On Thu, Feb 17, 2022 at 3:01 PM Saul Wold <Saul.Wold@windriver.com> wrote:
>
> Folks,
>
> I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used
> and processed to possibly include a COMPATIBLE_LICENSES variable as
> well, see PeterK's email [0]
>
> I am trying to determine the usage of WHITELIST_<license> which would be
> used to override a license that might be listed in INCOMPATIBLE_LICENSES
> variable.
>
> Randy and I have done a quick and dirty survey of a 100 or so layers
> (thanks Randy) and could not find any real usage other than what's
> currently in OE-Core for WHITELIST_GPL-3.0.
>
> If you are using WHITELIST_<license>, please let me reply with your usage.

Common case seems to be allowing gdb in debug builds of distros which
are otherwise free of GPLv3 etc, e.g.

  https://github.com/MentorEmbedded/meta-mentor/blob/master/meta-mel/conf/distro/include/gdbserver-gplv3.inc
  https://github.com/lgirdk/meta-mng/blob/ofw-2112.6/conf/distro/mng.conf

Re: [Openembedded-architecture] [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

[Mikko.Rapeli]

Hi,

On Thu, Feb 17, 2022 at 04:20:01PM -0800, Andre McCurdy wrote:
> On Thu, Feb 17, 2022 at 3:01 PM Saul Wold <Saul.Wold@windriver.com> wrote:
> >
> > Folks,
> >
> > I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used
> > and processed to possibly include a COMPATIBLE_LICENSES variable as
> > well, see PeterK's email [0]
> >
> > I am trying to determine the usage of WHITELIST_<license> which would be
> > used to override a license that might be listed in INCOMPATIBLE_LICENSES
> > variable.
> >
> > Randy and I have done a quick and dirty survey of a 100 or so layers
> > (thanks Randy) and could not find any real usage other than what's
> > currently in OE-Core for WHITELIST_GPL-3.0.
> >
> > If you are using WHITELIST_<license>, please let me reply with your usage.
> 
> Common case seems to be allowing gdb in debug builds of distros which
> are otherwise free of GPLv3 etc, e.g.
> 
>   https://github.com/MentorEmbedded/meta-mentor/blob/master/meta-mel/conf/distro/include/gdbserver-gplv3.inc
>   https://github.com/lgirdk/meta-mng/blob/ofw-2112.6/conf/distro/mng.conf

Exactly, I'm using this in a lot of projects.

Basically distro config has:

INCOMPATIBLE_LICENSE_append += " GPLv3 GPLv3+ LGPLv3 LGPLv3+"

and then various GPLv3 tools are allowed in build but not allowed to be
part of images:

WHITELIST_GPL-3.0 += "binutils"
PACKAGE_EXCLUDE += "binutils-dbg binutils-staticdev binutils-dev binutils-doc binutils-locale libbfd binutils"

WHITELIST_GPL-3.0 += "gdb"
PACKAGE_EXCLUDE += "gdb-sdktests-dbg gdb-sdktests gdbserver gdb-dbg gdb-staticdev gdb-dev gdb-doc gdb-locale gdb"

WHITELIST_GPL-3.0 += "rsync"
PACKAGE_EXCLUDE += "rsync-dbg rsync-staticdev rsync-dev rsync-doc rsync-locale rsync"

etc etc.

Use case is that developers can install additional GPLv3 licensed tooling if they
need to debug something, but product images don't include them by default.

Cheers,

-Mikko

Re: [Openembedded-architecture] [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

[Alexander Kanavin]

You should really try per-image INCOMPATIBLE_LICENSE :) Maintaining
those whitelists/excludes is awkward, as developers constantly want
more of them.

Alex

On Fri, 18 Feb 2022 at 09:00, Mikko Rapeli <mikko.rapeli@bmw.de> wrote:
>
> Hi,
>
> On Thu, Feb 17, 2022 at 04:20:01PM -0800, Andre McCurdy wrote:
> > On Thu, Feb 17, 2022 at 3:01 PM Saul Wold <Saul.Wold@windriver.com> wrote:
> > >
> > > Folks,
> > >
> > > I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used
> > > and processed to possibly include a COMPATIBLE_LICENSES variable as
> > > well, see PeterK's email [0]
> > >
> > > I am trying to determine the usage of WHITELIST_<license> which would be
> > > used to override a license that might be listed in INCOMPATIBLE_LICENSES
> > > variable.
> > >
> > > Randy and I have done a quick and dirty survey of a 100 or so layers
> > > (thanks Randy) and could not find any real usage other than what's
> > > currently in OE-Core for WHITELIST_GPL-3.0.
> > >
> > > If you are using WHITELIST_<license>, please let me reply with your usage.
> >
> > Common case seems to be allowing gdb in debug builds of distros which
> > are otherwise free of GPLv3 etc, e.g.
> >
> >   https://github.com/MentorEmbedded/meta-mentor/blob/master/meta-mel/conf/distro/include/gdbserver-gplv3.inc
> >   https://github.com/lgirdk/meta-mng/blob/ofw-2112.6/conf/distro/mng.conf
>
> Exactly, I'm using this in a lot of projects.
>
> Basically distro config has:
>
> INCOMPATIBLE_LICENSE_append += " GPLv3 GPLv3+ LGPLv3 LGPLv3+"
>
> and then various GPLv3 tools are allowed in build but not allowed to be
> part of images:
>
> WHITELIST_GPL-3.0 += "binutils"
> PACKAGE_EXCLUDE += "binutils-dbg binutils-staticdev binutils-dev binutils-doc binutils-locale libbfd binutils"
>
> WHITELIST_GPL-3.0 += "gdb"
> PACKAGE_EXCLUDE += "gdb-sdktests-dbg gdb-sdktests gdbserver gdb-dbg gdb-staticdev gdb-dev gdb-doc gdb-locale gdb"
>
> WHITELIST_GPL-3.0 += "rsync"
> PACKAGE_EXCLUDE += "rsync-dbg rsync-staticdev rsync-dev rsync-doc rsync-locale rsync"
>
> etc etc.
>
> Use case is that developers can install additional GPLv3 licensed tooling if they
> need to debug something, but product images don't include them by default.
>
> Cheers,
>
> -Mikko
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#1460): https://lists.openembedded.org/g/openembedded-architecture/message/1460
> Mute This Topic: https://lists.openembedded.org/mt/89223137/1686489
> Group Owner: openembedded-architecture+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [Openembedded-architecture] [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

[Mikko.Rapeli]

Hi,

On Fri, Feb 18, 2022 at 09:03:53AM +0100, Alexander Kanavin wrote:
> You should really try per-image INCOMPATIBLE_LICENSE :) Maintaining
> those whitelists/excludes is awkward, as developers constantly want
> more of them.

Hmm, this rings a bell. Likely yes then :)

But on the other hand this way we're quite explicit about what GPLv3 things
we allow to compile. Compiling too much also hurts with compilation times too and for
many use cases developers are better off working with Debian or Ubuntu tooling than
compiling every bell and whistle with yocto.

My list of GPLv3 recipes enabled for compilation is quite stable at:

autoconf
bash
bc
binutils
bison
cairo
ccache
coreutils
diffutils
elfutils
findutils
gawk
gcc
gdb
gdbm
gettext
gnutls
grep
libevent
libmpc
libpipeline
libunistring
m4
make
mpfr
python-rfc3987
python-strict-rfc3339
readline
rsync
sed
tcpreplay
which

Cheers,

-Mikko

Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

[Richard Purdie]

On Thu, 2022-02-17 at 15:01 -0800, Saul Wold wrote:
> I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used 
> and processed to possibly include a COMPATIBLE_LICENSES variable as 
> well, see PeterK's email [0]
> 
> I am trying to determine the usage of WHITELIST_<license> which would be 
> used to override a license that might be listed in INCOMPATIBLE_LICENSES 
> variable.
> 
> Randy and I have done a quick and dirty survey of a 100 or so layers 
> (thanks Randy) and could not find any real usage other than what's 
> currently in OE-Core for WHITELIST_GPL-3.0.
> 
> If you are using WHITELIST_<license>, please let me reply with your usage.
> 
> 
> [0] https://lists.openembedded.org/g/openembedded-devel/message/95166

We need to be mindful that we need to resolve this to unblock the other language
changes and feature creep here is potentially problematic. I do think it is
worth trying to improve things rather than blindly allowing the horrible syntax
in this variable to continue though.

The test case we have for this currently is:

WHITELIST_GPL-3.0:pn-core-image-minimal = "bash"

so I'd wondered about an alternative of:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash:GPL-3.0"

which matches the current functionality, removes the issue that the name of the
variable is unknown without iterating every possible license name and makes it
clear where it is applying to.

I don't really like INCOMPATIBLE_LICENSE_ALLOWED_RECIPES since:

a) it is long
b) it refers to recipes when it works against packages

(INCOMPATIBLE_LICENSE_PACKAGE_EXCEPTIONS is more correct but still long)

I do like it mentioning INCOMPATIBLE_LICENSE in full since it works in
conjunction with that variable and that is definitely not clear from the current
WHITELIST_XXX until you look at the code.

I'm still of the opinion the AVAILABLE_LICENSES variable is something we should
be aiming to remove ultimately too, it has horrible issues with layers changing
hashes for all recipes.

Cheers,

Richard

RE: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

[Peter Kjellerstedt]

Warning: wall of text ahead. Sorry about that, but I believe it is 
important that we get this right if we are redesigning it.

TL;DR: see the end for the new suggested licensing variables.

> -----Original Message-----
> From: openembedded-devel@lists.openembedded.org <openembedded-
> devel@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 18 februari 2022 15:14
> To: Saul Wold <Saul.Wold@windriver.com>; openembedded-
> architecture@lists.openembedded.org; OE-core <openembedded-
> core@lists.openembedded.org>; OpenEmbedded Devel List <openembedded-
> devel@lists.openembedded.org>
> Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage
> 
> On Thu, 2022-02-17 at 15:01 -0800, Saul Wold wrote:
> > I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used
> > and processed to possibly include a COMPATIBLE_LICENSES variable as
> > well, see PeterK's email [0]
> >
> > I am trying to determine the usage of WHITELIST_<license> which would be
> > used to override a license that might be listed in INCOMPATIBLE_LICENSES
> > variable.
> >
> > Randy and I have done a quick and dirty survey of a 100 or so layers
> > (thanks Randy) and could not find any real usage other than what's
> > currently in OE-Core for WHITELIST_GPL-3.0.
> >
> > If you are using WHITELIST_<license>, please let me reply with your
> > usage.
> >
> >
> > [0] https://lists.openembedded.org/g/openembedded-devel/message/95166
> 
> We need to be mindful that we need to resolve this to unblock the other
> language changes and feature creep here is potentially problematic. I do 
> think it is worth trying to improve things rather than blindly allowing 
> the horrible syntax in this variable to continue though.
> 
> The test case we have for this currently is:
> 
> WHITELIST_GPL-3.0:pn-core-image-minimal = "bash"
> 
> so I'd wondered about an alternative of:
> 
> INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash:GPL-3.0"

You do not really need the license here (more than possibly as 
information to anyone reading the code). Specifying that a package is 
allowed in an image regardless of its licenses can just as well be 
done by allowing the variable to take a list of packages:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash readline"

An alternative (and in my opinion better) variable name would be:

IMAGE_ALLOW_PACKAGES:pn-core-image-minimal = "bash readline"

focusing on that this is a list of _packages_ allowed for a given 
_image_. This does not explicitly mention licenses, but I do not 
believe that is needed. After all there could be multiple reasons a 
package is not allowed in an image and this variable would allow to 
ignore them all because that is typically what you want to do: 
specify that you really want that package in that image. I guess 
this is in some sense the opposite of PACKAGE_EXCLUDE. And I guess 
like :append vs :remove, if someone for some reason specifies a 
package in both IMAGE_ALLOW_PACKAGES and PACKAGE_EXCLUDE, then the 
later should win (to err on the side of caution).

For the case where you want to allow a recipe to be built despite 
it using licenses that are otherwise not allowed you can simply use 

INCOMPATIBLE_LICENSE:pn-foo = ""

and don't really need a separate variable for it.

> which matches the current functionality, removes the issue that the name
> of the variable is unknown without iterating every possible license name 
> and makes it clear where it is applying to.
> 
> I don't really like INCOMPATIBLE_LICENSE_ALLOWED_RECIPES since:
> 
> a) it is long
> b) it refers to recipes when it works against packages
> 
> (INCOMPATIBLE_LICENSE_PACKAGE_EXCEPTIONS is more correct but still long)
> 
> I do like it mentioning INCOMPATIBLE_LICENSE in full since it works in
> conjunction with that variable and that is definitely not clear from the
> current WHITELIST_XXX until you look at the code.
> 
> I'm still of the opinion the AVAILABLE_LICENSES variable is something we
> should be aiming to remove ultimately too, it has horrible issues with 
> layers changing hashes for all recipes.

Since I was the one to introduce it, I will answer to that. It was 
introduced so that it is possible to specify compatible licenses 
instead of incompatible licenses by basically calculating the 
incompatible licenses by taking the set of available licenses minus 
the set of compatible licenses. If a mechanism to do that is 
introduced, e.g., by adding support for a COMPATIBLE_LICENSES in 
addition to INCOMPATIBLE_LICENSE, then I am of the opinion that we 
can remove AVAILABLE_LICENSES again.

We also need this mechanism in the code for handling allowed vs 
disallowed licenses because the current code really cannot handled a 
list of many hundreds of incompatible licenses, which is what we got 
after all SPDX licenses were added to OE-Core. The code is extremely 
inefficient and evaluates the list of incompatible licenses over and 
over again. In our case that meant the recipe parsing time tripled.

> Cheers,
> 
> Richard

That said, we really need two sets of variables. One for blocking 
the building of recipes because of its license(s), and one for 
preventing packages with disallowed licenses to be included in a 
given image. These are very different use cases and they take 
totally different lists of items (recipes vs packages). The current 
mess where the same variables are used for both cases needs to be 
resolved.

So why do we need both cases? The first case where recipes are 
prevented from being built makes it possible to, e.g., prevent a 
newer versioned recipe that uses GPL-3.0-or-later to be built and 
instead build an older version that uses GPL-2.0-or-later (i.e., 
what meta-gplv2 can be used for today). 

The second case allows to build code that uses disallowed licenses, 
as long as the packages are not added to the image. Why is this 
useful? Because many recipes are built for some packages that are 
never used in the given image, and it is then much easier to allow 
them to be built as long as their output is not used. E.g., very 
many recipes depend on bash which is GPL-3.0-or-later, making it 
near impossible to avoid having to build it. However, it is 
perfectly possible to build production images that do not need bash 
to be installed.

I believe for this second case we should have two variables, 
IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. And to 
make the naming clear for the first case, I would suggest calling 
those variables RECIPE_COMPATIBLE_LICENSES and 
RECIPE_INCOMPATIBLE_LICENSES.

Also note that the use of RECIPE_COMPATIBLE_LICENSES and 
RECIPE_INCOMPATIBLE_LICENSES is mutually exclusive, as is the use 
of IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. I.e., 
you will have to choose whether to specify what licenses to allow 
or what licenses to disallow. You cannot do both. This is because 
specifying a list of compatible licenses means that all other 
licenses are by definition incompatible, and vice versa. However, 
this also means that it makes perfect sense to be able to, e.g., 
specify a few RECIPE_INCOMPATIBLE_LICENSES together with a list of 
IMAGE_COMPATIBLE_LICENSES.

Oh, and another thing I would like to take the opportunity to raise 
is whether we should continue to support patterns in these list, or 
if we should change it so that the lists of licenses need to 
explicitly specify all licenses. The latter would greatly simplify 
the code, especially when combined with only allowing SPDX names.
If we decide to remove support for patterns, and based on the 
assumption that the pattern is typically used to specify "*GPL-3.0*", 
we could make available a variable or two that contain the typical 
sets of GPL licenses. This would also allow us to remove the code 
that handles how an or-later licenses specified as '<license>+' 
should be treated in combination with patterns.

So, to reiterate, these are the variables that I suggest we adopt 
to be able to handle the various use cases regarding licenses:

RECIPE_COMPATIBLE_LICENSES   - list of compatible licenses for 
                               recipes
RECIPE_INCOMPATIBLE_LICENSES - list of incompatible licenses for 
                               recipes
IMAGE_COMPATIBLE_LICENSES    - list of compatible licenses for 
                               packages in images
IMAGE_INCOMPATIBLE_LICENSES  - list of incompatible licenses for 
                               packages in images
IMAGE_ALLOW_PACKAGES         - list of packages allowed in the 
                               image regardless of licenses and 
                               other restrictions

This also means that the old WHITELIST_*, INCOMPATIBLE_LICENSE 
and AVAILABLE_LICENSES variables are removed. I also suggest we 
remove the support for patterns in the new variables.

//Peter

Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

[Richard Purdie]

On Fri, 2022-02-18 at 23:34 +0000, Peter Kjellerstedt wrote:
> Warning: wall of text ahead. Sorry about that, but I believe it is 
> important that we get this right if we are redesigning it.
> 
> TL;DR: see the end for the new suggested licensing variables.
> 
> > -----Original Message-----
> > From: openembedded-devel@lists.openembedded.org <openembedded-
> > devel@lists.openembedded.org> On Behalf Of Richard Purdie
> > Sent: den 18 februari 2022 15:14
> > To: Saul Wold <Saul.Wold@windriver.com>; openembedded-
> > architecture@lists.openembedded.org; OE-core <openembedded-
> > core@lists.openembedded.org>; OpenEmbedded Devel List <openembedded-
> > devel@lists.openembedded.org>
> > Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage
> > 
> > On Thu, 2022-02-17 at 15:01 -0800, Saul Wold wrote:
> > > I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used
> > > and processed to possibly include a COMPATIBLE_LICENSES variable as
> > > well, see PeterK's email [0]
> > > 
> > > I am trying to determine the usage of WHITELIST_<license> which would be
> > > used to override a license that might be listed in INCOMPATIBLE_LICENSES
> > > variable.
> > > 
> > > Randy and I have done a quick and dirty survey of a 100 or so layers
> > > (thanks Randy) and could not find any real usage other than what's
> > > currently in OE-Core for WHITELIST_GPL-3.0.
> > > 
> > > If you are using WHITELIST_<license>, please let me reply with your
> > > usage.
> > > 
> > > 
> > > [0] https://lists.openembedded.org/g/openembedded-devel/message/95166
> > 
> > We need to be mindful that we need to resolve this to unblock the other
> > language changes and feature creep here is potentially problematic. I do 
> > think it is worth trying to improve things rather than blindly allowing 
> > the horrible syntax in this variable to continue though.
> > 
> > The test case we have for this currently is:
> > 
> > WHITELIST_GPL-3.0:pn-core-image-minimal = "bash"
> > 
> > so I'd wondered about an alternative of:
> > 
> > INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash:GPL-3.0"
> 
> You do not really need the license here (more than possibly as 
> information to anyone reading the code). Specifying that a package is 
> allowed in an image regardless of its licenses can just as well be 
> done by allowing the variable to take a list of packages:
> 
> INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash readline"

I'm not sure I agree with that. I get the feeling some people want to allow
these packages as long as they're with license X. Licenses can change over time
and this allows a safety net that if it changes, the exception has to be updated
too.

> An alternative (and in my opinion better) variable name would be:
> 
> IMAGE_ALLOW_PACKAGES:pn-core-image-minimal = "bash readline"
> 
> focusing on that this is a list of _packages_ allowed for a given 
> _image_. This does not explicitly mention licenses, but I do not 
> believe that is needed.

If you assume this isn't being done on license reasons, sure. Except see above,
I think this does need to account for licenses, at least the way many use it.

> After all there could be multiple reasons a package is not allowed in 
> an image and this variable would allow to 
> ignore them all because that is typically what you want to do: 
> specify that you really want that package in that image. I guess 
> this is in some sense the opposite of PACKAGE_EXCLUDE. And I guess 
> like :append vs :remove, if someone for some reason specifies a 
> package in both IMAGE_ALLOW_PACKAGES and PACKAGE_EXCLUDE, then the 
> later should win (to err on the side of caution).

How would IMAGE_ALLOW_PACKAGES work and be different to IMAGE_INSTALL? This
interface would likely confuse more users than it would help.

> For the case where you want to allow a recipe to be built despite 
> it using licenses that are otherwise not allowed you can simply use 
> 
> INCOMPATIBLE_LICENSE:pn-foo = ""
> 
> and don't really need a separate variable for it.

That is a good point and perhaps should influence how an
INCOMPATIBLE_LICENSE_EXCEPTIONS should be package based rather than recipe. I
doubt the pn- override existed when that variable was originally added.

> > I'm still of the opinion the AVAILABLE_LICENSES variable is something we
> > should be aiming to remove ultimately too, it has horrible issues with 
> > layers changing hashes for all recipes.
> 
> Since I was the one to introduce it, I will answer to that. It was 
> introduced so that it is possible to specify compatible licenses 
> instead of incompatible licenses by basically calculating the 
> incompatible licenses by taking the set of available licenses minus 
> the set of compatible licenses. If a mechanism to do that is 
> introduced, e.g., by adding support for a COMPATIBLE_LICENSES in 
> addition to INCOMPATIBLE_LICENSE, then I am of the opinion that we 
> can remove AVAILABLE_LICENSES again.
> 
> We also need this mechanism in the code for handling allowed vs 
> disallowed licenses because the current code really cannot handled a 
> list of many hundreds of incompatible licenses, which is what we got 
> after all SPDX licenses were added to OE-Core. The code is extremely 
> inefficient and evaluates the list of incompatible licenses over and 
> over again. In our case that meant the recipe IMAGE_ALLOW_PACKAGESparsing time tripled.

Ironically, part of the reason I want to change the design of WHITELIST_XXX is
that it forces code that doesn't perform well. You may be surprised to find that
this change actually improves that 3x issue you've seen. If not, I think it
would lead us in a direction where it can certainly help.

> That said, we really need two sets of variables. One for blocking 
> the building of recipes because of its license(s), and one for 
> preventing packages with disallowed licenses to be included in a 
> given image. These are very different use cases and they take 
> totally different lists of items (recipes vs packages). The current 
> mess where the same variables are used for both cases needs to be 
> resolved.
> 
> So why do we need both cases? The first case where recipes are 
> prevented from being built makes it possible to, e.g., prevent a 
> newer versioned recipe that uses GPL-3.0-or-later to be built and 
> instead build an older version that uses GPL-2.0-or-later (i.e., 
> what meta-gplv2 can be used for today). 
> 
> The second case allows to build code that uses disallowed licenses, 
> as long as the packages are not added to the image. Why is this 
> useful? Because many recipes are built for some packages that are 
> never used in the given image, and it is then much easier to allow 
> them to be built as long as their output is not used. E.g., very 
> many recipes depend on bash which is GPL-3.0-or-later, making it 
> near impossible to avoid having to build it. However, it is 
> perfectly possible to build production images that do not need bash 
> to be installed.
> 
> I believe for this second case we should have two variables, 
> IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. And to 
> make the naming clear for the first case, I would suggest calling 
> those variables RECIPE_COMPATIBLE_LICENSES and 
> RECIPE_INCOMPATIBLE_LICENSES.
> 
> Also note that the use of RECIPE_COMPATIBLE_LICENSES and 
> RECIPE_INCOMPATIBLE_LICENSES is mutually exclusive, as is the use 
> of IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. I.e., 
> you will have to choose whether to specify what licenses to allow 
> or what licenses to disallow. You cannot do both. This is because 
> specifying a list of compatible licenses means that all other 
> licenses are by definition incompatible, and vice versa. However, 
> this also means that it makes perfect sense to be able to, e.g., 
> specify a few RECIPE_INCOMPATIBLE_LICENSES together with a list of 
> IMAGE_COMPATIBLE_LICENSES.

I suspect there will be people who will want configurations where they specify
that they are happy with X and definitely don't want Y to work. The unspecified
licenses would effectively be compatible but not be explicitly specified. It
seems like an odd setup but I can imagine people configuring it and wanting it
to work.

I am very worried about the performance implications in this "every license" in
a compatible list. Obviously for the case your legal department cares about, you
have to do it but I'm not sure we want to force it onto everyone (and
AVAILABLE_LICENSES already heads in that direction).

> Oh, and another thing I would like to take the opportunity to raise 
> is whether we should continue to support patterns in these list, or 
> if we should change it so that the lists of licenses need to 
> explicitly specify all licenses. The latter would greatly simplify 
> the code, especially when combined with only allowing SPDX names.
> If we decide to remove support for patterns, and based on the 
> assumption that the pattern is typically used to specify "*GPL-3.0*", 
> we could make available a variable or two that contain the typical 
> sets of GPL licenses. This would also allow us to remove the code 
> that handles how an or-later licenses specified as '<license>+' 
> should be treated in combination with patterns.

I'd love to remove it but it is something which people want and now expect from
the code. You might not like it, others do. How do we please everyone? I don't
think we'd be able to remove that, only perhaps limit it a little more.

> So, to reiterate, these are the variables that I suggest we adopt 
> to be able to handle the various use cases regarding licenses:
> 
> RECIPE_COMPATIBLE_LICENSES   - list of compatible licenses for 
>                                recipes
> RECIPE_INCOMPATIBLE_LICENSES - list of incompatible licenses for 
>                                recipes
> IMAGE_COMPATIBLE_LICENSES    - list of compatible licenses for 
>                                packages in images
> IMAGE_INCOMPATIBLE_LICENSES  - list of incompatible licenses for 
>                                packages in images
> IMAGE_ALLOW_PACKAGES         - list of packages allowed in the 
>                                image regardless of licenses and 
>                                other restrictions
> 
> This also means that the old WHITELIST_*, INCOMPATIBLE_LICENSE 
> and AVAILABLE_LICENSES variables are removed. I also suggest we 
> remove the support for patterns in the new variables.

Whilst I don't just want to map WHITELIST_* to something renamed, I think the
above is a bit too radical to get into this late in the release cycle so I
suspect we'll have to compromise. This discussion needed to happen early in the
cycle with people actively working on it.

Just to be really clear, in particular I detest IMAGE_ALLOW_PACKAGES as a
variable name. I think you're trying to be too wide in scope with "other
restrictions" and it will just confuse people.

Cheers,

Richard

RE: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

[Peter Kjellerstedt]

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: den 19 februari 2022 01:45
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; Saul Wold
> <Saul.Wold@windriver.com>; openembedded-
> architecture@lists.openembedded.org; OE-core <openembedded-
> core@lists.openembedded.org>; OpenEmbedded Devel List <openembedded-
> devel@lists.openembedded.org>
> Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage
> 
> On Fri, 2022-02-18 at 23:34 +0000, Peter Kjellerstedt wrote:
> > Warning: wall of text ahead. Sorry about that, but I believe it is
> > important that we get this right if we are redesigning it.
> >
> > TL;DR: see the end for the new suggested licensing variables.
> >
> > > -----Original Message-----
> > > From: openembedded-devel@lists.openembedded.org <openembedded-
> > > devel@lists.openembedded.org> On Behalf Of Richard Purdie
> > > Sent: den 18 februari 2022 15:14
> > > To: Saul Wold <Saul.Wold@windriver.com>; openembedded-
> > > architecture@lists.openembedded.org; OE-core <openembedded-
> > > core@lists.openembedded.org>; OpenEmbedded Devel List <openembedded-
> > > devel@lists.openembedded.org>
> > > Subject: Re: [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage
> > >
> > > On Thu, 2022-02-17 at 15:01 -0800, Saul Wold wrote:
> > > > I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is
> > > > used and processed to possibly include a COMPATIBLE_LICENSES variable
> > > > as well, see PeterK's email [0]
> > > >
> > > > I am trying to determine the usage of WHITELIST_<license> which
> > > > would be used to override a license that might be listed in
> > > > INCOMPATIBLE_LICENSES variable.
> > > >
> > > > Randy and I have done a quick and dirty survey of a 100 or so layers
> > > > (thanks Randy) and could not find any real usage other than what's
> > > > currently in OE-Core for WHITELIST_GPL-3.0.
> > > >
> > > > If you are using WHITELIST_<license>, please let me reply with your
> > > > usage.
> > > >
> > > > [0] https://lists.openembedded.org/g/openembedded-devel/message/95166
> > >
> > > We need to be mindful that we need to resolve this to unblock the
> > > other language changes and feature creep here is potentially 
> > > problematic. I do think it is worth trying to improve things rather 
> > > than blindly allowing the horrible syntax in this variable to 
> > > continue though.
> > >
> > > The test case we have for this currently is:
> > >
> > > WHITELIST_GPL-3.0:pn-core-image-minimal = "bash"
> > >
> > > so I'd wondered about an alternative of:
> > >
> > > INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash:GPL-3.0"
> >
> > You do not really need the license here (more than possibly as
> > information to anyone reading the code). Specifying that a package is
> > allowed in an image regardless of its licenses can just as well be
> > done by allowing the variable to take a list of packages:
> >
> > INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash readline"
> 
> I'm not sure I agree with that. I get the feeling some people want to
> allow these packages as long as they're with license X. Licenses can 
> change over time and this allows a safety net that if it changes, the 
> exception has to be updated too.

Well, to complicate matters then, if you really want to be explicit 
with the license exceptions you make for a recipe or a package, I will 
argue that you need to do it for all incompatible licenses a recipe 
uses. With today's code, as soon as you allow one incompatible license 
for a recipe, the recipe is allowed to be built, even if it has 
specified multiple incompatible licenses. E.g., if we take the gnupg 
recipe, which specifies LICENSE = "GPLv3 & LGPLv3", and we have an 
INCOMPATIBLE_LICENSE = "GPL-3.0-only LGPL-3.0-only", with your example 
I would then expect the following:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "gnupg:GPL-3.0-only gnupg:LGPL-3.0-only"

to allow the gnupg package to be installed in the image. However, I 
guess you could combine our suggestions by making the license part 
of the value optional, i.e., the following would then also allow the 
package to be included under the same conditions:

INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "gnupg"

> > An alternative (and in my opinion better) variable name would be:
> >
> > IMAGE_ALLOW_PACKAGES:pn-core-image-minimal = "bash readline"
> >
> > focusing on that this is a list of _packages_ allowed for a given
> > _image_. This does not explicitly mention licenses, but I do not
> > believe that is needed.
> 
> If you assume this isn't being done on license reasons, sure. Except 
> see above, I think this does need to account for licenses, at least 
> the way many use it.
> 
> > After all there could be multiple reasons a package is not allowed in
> > an image and this variable would allow to
> > ignore them all because that is typically what you want to do:
> > specify that you really want that package in that image. I guess
> > this is in some sense the opposite of PACKAGE_EXCLUDE. And I guess
> > like :append vs :remove, if someone for some reason specifies a
> > package in both IMAGE_ALLOW_PACKAGES and PACKAGE_EXCLUDE, then the
> > later should win (to err on the side of caution).
> 
> How would IMAGE_ALLOW_PACKAGES work and be different to IMAGE_INSTALL?

Well, IMAGE_INSTALL determines what packages you want in your image, 
while IMAGE_ALLOW_PACKAGES allows packages to be installed that would 
otherwise render an error if you tried. To me it seems perfectly 
natural, but I am probably a bit biased. ;)

> This interface would likely confuse more users than it would help.

Well, it is really based on that only package names need to be specified 
and not the licenses. If both are needed (as you believe), then this name 
makes less sense.

> > For the case where you want to allow a recipe to be built despite
> > it using licenses that are otherwise not allowed you can simply use
> >
> > INCOMPATIBLE_LICENSE:pn-foo = ""
> >
> > and don't really need a separate variable for it.
> 
> That is a good point and perhaps should influence how an
> INCOMPATIBLE_LICENSE_EXCEPTIONS should be package based rather than
> recipe. I doubt the pn- override existed when that variable was 
> originally added.

I definitely believe that we need to separate the case for building 
a recipe from the case for installing a package into an image. For the 
recipe case, I believe that manipulating RECIPE_INCOMPATIBLE_LICENSES 
(or whatever the variable ends up being called) using normal variable 
manipulations should be enough, without having to introduce yet 
another variable. I.e., my example above would correspond to giving 
a blanket exception for all licenses for the given recipe, while a 
case where a specific license is excepted would be something like 
RECIPE_INCOMPATIBLE_LICENSES:remove:pn-foo = "GPL-3.0-only"

For the installing a package into an image case, I don't see a way 
to avoid an extra variable as discussed above. However, the name 
INCOMPATIBLE_LICENSE_EXCEPTIONS gives no indications that it is 
image specific, which I think is bad. If we introduce the other 
names I suggest below, then this would instead map to 
IMAGE_INCOMPATIBLE_LICENSES_EXCEPTIONS (though I dislike that name 
because it is so long, but maybe it is inevitable).

> > > I'm still of the opinion the AVAILABLE_LICENSES variable is something
> > > we should be aiming to remove ultimately too, it has horrible issues
> > > with layers changing hashes for all recipes.
> >
> > Since I was the one to introduce it, I will answer to that. It was
> > introduced so that it is possible to specify compatible licenses
> > instead of incompatible licenses by basically calculating the
> > incompatible licenses by taking the set of available licenses minus
> > the set of compatible licenses. If a mechanism to do that is
> > introduced, e.g., by adding support for a COMPATIBLE_LICENSES in
> > addition to INCOMPATIBLE_LICENSE, then I am of the opinion that we
> > can remove AVAILABLE_LICENSES again.
> >
> > We also need this mechanism in the code for handling allowed vs
> > disallowed licenses because the current code really cannot handled a
> > list of many hundreds of incompatible licenses, which is what we got
> > after all SPDX licenses were added to OE-Core. The code is extremely
> > inefficient and evaluates the list of incompatible licenses over and
> > over again. In our case that meant the recipe parsing time tripled.
> 
> Ironically, part of the reason I want to change the design of
> WHITELIST_XXX is that it forces code that doesn't perform well. You 
> may be surprised to find that this change actually improves that 3x 
> issue you've seen. If not, I think it would lead us in a direction 
> where it can certainly help.
> 
> > That said, we really need two sets of variables. One for blocking
> > the building of recipes because of its license(s), and one for
> > preventing packages with disallowed licenses to be included in a
> > given image. These are very different use cases and they take
> > totally different lists of items (recipes vs packages). The current
> > mess where the same variables are used for both cases needs to be
> > resolved.
> >
> > So why do we need both cases? The first case where recipes are
> > prevented from being built makes it possible to, e.g., prevent a
> > newer versioned recipe that uses GPL-3.0-or-later to be built and
> > instead build an older version that uses GPL-2.0-or-later (i.e.,
> > what meta-gplv2 can be used for today).
> >
> > The second case allows to build code that uses disallowed licenses,
> > as long as the packages are not added to the image. Why is this
> > useful? Because many recipes are built for some packages that are
> > never used in the given image, and it is then much easier to allow
> > them to be built as long as their output is not used. E.g., very
> > many recipes depend on bash which is GPL-3.0-or-later, making it
> > near impossible to avoid having to build it. However, it is
> > perfectly possible to build production images that do not need bash
> > to be installed.
> >
> > I believe for this second case we should have two variables,
> > IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. And to
> > make the naming clear for the first case, I would suggest calling
> > those variables RECIPE_COMPATIBLE_LICENSES and
> > RECIPE_INCOMPATIBLE_LICENSES.
> >
> > Also note that the use of RECIPE_COMPATIBLE_LICENSES and
> > RECIPE_INCOMPATIBLE_LICENSES is mutually exclusive, as is the use
> > of IMAGE_COMPATIBLE_LICENSES and IMAGE_INCOMPATIBLE_LICENSES. I.e.,
> > you will have to choose whether to specify what licenses to allow
> > or what licenses to disallow. You cannot do both. This is because
> > specifying a list of compatible licenses means that all other
> > licenses are by definition incompatible, and vice versa. However,
> > this also means that it makes perfect sense to be able to, e.g.,
> > specify a few RECIPE_INCOMPATIBLE_LICENSES together with a list of
> > IMAGE_COMPATIBLE_LICENSES.
> 
> I suspect there will be people who will want configurations where they
> specify that they are happy with X and definitely don't want Y to work. 
> The unspecified licenses would effectively be compatible but not be 
> explicitly specified. It seems like an odd setup but I can imagine 
> people configuring it and wanting it to work.
> 
> I am very worried about the performance implications in this "every
> license" in a compatible list. Obviously for the case your legal 
> department cares about, you have to do it but I'm not sure we want 
> to force it onto everyone (and AVAILABLE_LICENSES already heads in 
> that direction).

By splitting the list of licenses into two, to handle the opposite 
use cases of allowed vs disallowed licenses, we avoid the need to know 
the full list of potential licenses as we only need to know what is
allowed or disallowed and can exclude anything else.

Thus the expectation is that the lists in either *_COMPATIBLE_LICENSES 
or *_INCOMPATIBLE_LICENSES will be relatively short, compared to lists 
based on all licenses. In our case, we have 80 licenses in our list 
of allowed licenses. That is something that I think even the current 
code can handle reasonably well. And without pattern matching and 
mappings of old license names, the code would basically have to do 
something like:

    compatible_licenses = (d.getVar('COMPATIBLE_LICENSES') or '').split()
    incompatible_licenses = (d.getVar('INCOMPATIBLE_LICENSES') or '').split()
    disallowed_licenses = set()
    ...
    if compatible_licenses and license not in compatible_licenses or \
       incompatible_licenses and license in incompatible_licenses:
        disallowed_licenses.add(license)

Also, based on the assumption that in most cases you want to be able to 
build as much as possible, I would guess that you would typically use 
RECIPES_INCOMPATIBLE_LICENSES to exclude only a select few licenses that 
cannot be used at all (e.g., GPL-3.0), and then use 
IMAGE_COMPATIBLE_LICENSES or IMAGE_INCOMPATIBLE_LICENSES to limit what 
goes into the images. Thus the longer lists would only affect the image 
recipes compared to all recipes as it is today.

> > Oh, and another thing I would like to take the opportunity to raise
> > is whether we should continue to support patterns in these list, or
> > if we should change it so that the lists of licenses need to
> > explicitly specify all licenses. The latter would greatly simplify
> > the code, especially when combined with only allowing SPDX names.
> > If we decide to remove support for patterns, and based on the
> > assumption that the pattern is typically used to specify "*GPL-3.0*",
> > we could make available a variable or two that contain the typical
> > sets of GPL licenses. This would also allow us to remove the code
> > that handles how an or-later licenses specified as '<license>+'
> > should be treated in combination with patterns.
> 
> I'd love to remove it but it is something which people want and now 
> expect from the code. You might not like it, others do. How do we 
> please everyone? I don't think we'd be able to remove that, only 
> perhaps limit it a little more.

But if we redesign the license handling anyway, can't we change their 
expectations at the same time? There really aren't that many groups 
of SPDX licenses where it is useful to use pattern matching anyway 
and that you are likely to put in these variables. Basically I think 
it comes down to some set of GPL licenses. I.e., replacing "*GPL-3.0*" 
with the corresponding licenses really doesn't yield a very long list. 
And as I suggested above, we could even provide a variable or two with 
the expanded lists of the most typical use cases. E.g., if we added a 
GPL3_LICENSES variable, the INCOMPATIBLE_LICENSES = "*GPL-3.0*" would 
become INCOMPATIBLE_LICENSES = "${GPL3_LICENSES}". Yes, it is a little 
less flexible, but it would do wonders for the code...

Anyway, this is a minor gripe compared to the rest, so if we can't 
get rid of it, so be it.

> > So, to reiterate, these are the variables that I suggest we adopt
> > to be able to handle the various use cases regarding licenses:
> >
> > RECIPE_COMPATIBLE_LICENSES   - list of compatible licenses for
> >                                recipes
> > RECIPE_INCOMPATIBLE_LICENSES - list of incompatible licenses for
> >                                recipes
> > IMAGE_COMPATIBLE_LICENSES    - list of compatible licenses for
> >                                packages in images
> > IMAGE_INCOMPATIBLE_LICENSES  - list of incompatible licenses for
> >                                packages in images
> > IMAGE_ALLOW_PACKAGES         - list of packages allowed in the
> >                                image regardless of licenses and
> >                                other restrictions
> >
> > This also means that the old WHITELIST_*, INCOMPATIBLE_LICENSE
> > and AVAILABLE_LICENSES variables are removed. I also suggest we
> > remove the support for patterns in the new variables.
> 
> Whilst I don't just want to map WHITELIST_* to something renamed, I think
> the above is a bit too radical to get into this late in the release cycle 
> so I suspect we'll have to compromise. This discussion needed to happen 
> early in the cycle with people actively working on it.

And I really, really wish I had had the time then to do exactly that. :( 
I know I even said I would after I noticed the problem with the prolonged 
recipe parsing times due to the huge increase in available licenses, but 
alas time wasn't on my side. So I did what anyone would do and made a 
workaround and went on to handle more pressing problems. I am very sorry 
about that.

> Just to be really clear, in particular I detest IMAGE_ALLOW_PACKAGES as a
> variable name. I think you're trying to be too wide in scope with "other
> restrictions" and it will just confuse people.

Ouch, and I thought it was pretty ok. :( Especially compared to 
INCOMPATIBLE_LICENSE_EXCEPTIONS, which I don't like very much (though that 
is mostly based on its length). Anyway, this has been discussed at length
above, so I will not delve on it more here.

> Cheers,
> 
> Richard

//Peter

Re: [OE-core] [oe] INCOMPATIBLE_LICENSES and WHITELIST_<license> usage

[Richard Purdie]

On Fri, 2022-02-18 at 14:13 +0000, Richard Purdie via lists.openembedded.org
wrote:
> On Thu, 2022-02-17 at 15:01 -0800, Saul Wold wrote:
> > I am working on a proposal to re-write how INCOMPATIBLE_LICENSES is used 
> > and processed to possibly include a COMPATIBLE_LICENSES variable as 
> > well, see PeterK's email [0]
> > 
> > I am trying to determine the usage of WHITELIST_<license> which would be 
> > used to override a license that might be listed in INCOMPATIBLE_LICENSES 
> > variable.
> > 
> > Randy and I have done a quick and dirty survey of a 100 or so layers 
> > (thanks Randy) and could not find any real usage other than what's 
> > currently in OE-Core for WHITELIST_GPL-3.0.
> > 
> > If you are using WHITELIST_<license>, please let me reply with your usage.
> > 
> > 
> > [0] https://lists.openembedded.org/g/openembedded-devel/message/95166
> 
> We need to be mindful that we need to resolve this to unblock the other language
> changes and feature creep here is potentially problematic. I do think it is
> worth trying to improve things rather than blindly allowing the horrible syntax
> in this variable to continue though.
> 
> The test case we have for this currently is:
> 
> WHITELIST_GPL-3.0:pn-core-image-minimal = "bash"
> 
> so I'd wondered about an alternative of:
> 
> INCOMPATIBLE_LICENSE_EXCEPTIONS:pn-core-image-minimal = "bash:GPL-3.0"
> 
> which matches the current functionality, removes the issue that the name of the
> variable is unknown without iterating every possible license name and makes it
> clear where it is applying to.
> 
> I don't really like INCOMPATIBLE_LICENSE_ALLOWED_RECIPES since:
> 
> a) it is long
> b) it refers to recipes when it works against packages

Just as a warning, the code is actually confused. The base.bbclass code assumes
it is a recipe name, the license_image code assumes it is a package name.

All the more reason to come up with a standard and resolve this. My proposal
stands but we need to decide if it is a recipe or package name that is listed
(or both?).

Cheers,

Richard