RFC: Package exclusion

RFC: Package exclusion

[Paul Eggleton]

Hi all,

As part of the 1.1 feature list I suggested a review of the 
INCOMPATIBLE_LICENSE and COMMERCIAL* package exclusion mechanisms we have 
within Poky. Below I've outlined my ideas and would appreciate any 
comments/additions/corrections.

==== Aims ==== 

 * Make error messages clear when user/dependencies have asked for something 
to be built that can't be due to restrictions

 * Ensure that exclusion system is reliable

====  Proposed implementation ==== 

 1) Ensure all documentation of LICENSE field value syntax is clear, concise 
and up-to-date (wiki and manual)

 2) Go through and audit all recipes LICENSE field values to ensure that they 
all conform to the specifications. This includes making sure that | (package 
may be used under one of a selection of licences) and & (recipe has mixed 
licences that apply to the code base, so conditions of all must be observed) 
are used correctly.

 3) bitbake/core changes:

 * LICENSE field checking must fully parse the field and understand the 
difference between | and &, and must not e.g. mark Qt as being GPLv3 only.

 * Make the LICENSE validity checking more strict (given recipes have been 
audited and rules are clear after above)

 * Don't exclude any recipes at parse time - simply record all excluded 
recipes and their runtime provides in a blacklist which also includes flags 
indicating the reason for blacklisting

 * Ensure all excluded licences in INCOMPATIBLE_LICENSE are valid (e.g. catch 
GPL3 as apposed to GPLv3) - if not, error out

 * Check when calculating dependencies if anything is scheduled to be built 
that is on the blacklist - if any are, gather all of them up and then stop and 
list them in an error message along with reason and depchain for each one

 * Check when constructing the rootfs if anything in the runtime provides 
blacklist is going to be included - if so, error out


Some further possible extensions:

 * Possibly apply similar logic to COMPATIBLE_MACHINE?

 * Replace COMMERCIAL* with some more generic exclusion mechanism that allows 
the reason to be defined as part of the exclusion list?

 * As a helper for non-en_US users, fail on parse if user defines any of the 
*LICENSE* variables as *LICENCE*? (we definitely don't want the build to 
continue and just ignore this as the user might not realise what has happened)


Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: RFC: Package exclusion

[Elizabeth Flanagan]

On 05/03/2011 09:00 AM, Paul Eggleton wrote:
> Hi all,
>
> As part of the 1.1 feature list I suggested a review of the
> INCOMPATIBLE_LICENSE and COMMERCIAL* package exclusion mechanisms we have
> within Poky. Below I've outlined my ideas and would appreciate any
> comments/additions/corrections.
>
> ==== Aims ====
>
>   * Make error messages clear when user/dependencies have asked for something
> to be built that can't be due to restrictions
>
>   * Ensure that exclusion system is reliable
>
> ====  Proposed implementation ====
>
>   1) Ensure all documentation of LICENSE field value syntax is clear, concise
> and up-to-date (wiki and manual)
>
>   2) Go through and audit all recipes LICENSE field values to ensure that they
> all conform to the specifications. This includes making sure that | (package
> may be used under one of a selection of licences) and&  (recipe has mixed
> licences that apply to the code base, so conditions of all must be observed)
> are used correctly.
>
>   3) bitbake/core changes:
>
>   * LICENSE field checking must fully parse the field and understand the
> difference between | and&, and must not e.g. mark Qt as being GPLv3 only.
>
>   * Make the LICENSE validity checking more strict (given recipes have been
> audited and rules are clear after above)
>
>   * Don't exclude any recipes at parse time - simply record all excluded
> recipes and their runtime provides in a blacklist which also includes flags
> indicating the reason for blacklisting
>
>   * Ensure all excluded licences in INCOMPATIBLE_LICENSE are valid (e.g. catch
> GPL3 as apposed to GPLv3) - if not, error out
>
>   * Check when calculating dependencies if anything is scheduled to be built
> that is on the blacklist - if any are, gather all of them up and then stop and
> list them in an error message along with reason and depchain for each one
>
>   * Check when constructing the rootfs if anything in the runtime provides
> blacklist is going to be included - if so, error out
>

Most of the issues surrounding the license field is currently on my plate for M1.D

License tracking	Build a parser to do license tracking more gracefully and make sure all recipes are correct. (takes 
~2 weeks)	1	Accept	Beth	Beth	M1, Sprint D

There is currently no specification on the LICENSE field values and how they should be parsed. I'm going to be starting 
that in the next week or so. We should sync up with each other if this work is going to overlap. Schedule some time for 
this week?

-b


> Cheers,
> Paul
>

-- 
---------------
Elizabeth Flanagan
Yocto Project
Release Engineer

Re: RFC: Package exclusion

[Paul Eggleton]

Hi Beth,

On Wednesday 04 May 2011 00:08:59 Elizabeth Flanagan wrote:
> Most of the issues surrounding the license field is currently on my plate
> for M1.D
> 
> License tracking	Build a parser to do license tracking more gracefully and
> make sure all recipes are correct. (takes ~2 weeks)	1	Accept	Beth	Beth	
M1,
> Sprint D
>
> There is currently no specification on the LICENSE field values and how
> they should be parsed. I'm going to be starting that in the next week or
> so. We should sync up with each other if this work is going to overlap.
> Schedule some time for this week?

Yes there's definitely quite an overlap here and we should coordinate efforts. 
Will catch you later on IRC to organise some time.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: RFC: Package exclusion

[Joshua Lock]

Apologies for the tardy reply. I like where you're going, usually I have
to disagree to reply swiftly. Sorry.

On Tue, 2011-05-03 at 17:00 +0100, Paul Eggleton wrote:
> Hi all,
> 
> As part of the 1.1 feature list I suggested a review of the 
> INCOMPATIBLE_LICENSE and COMMERCIAL* package exclusion mechanisms we have 
> within Poky. Below I've outlined my ideas and would appreciate any 
> comments/additions/corrections.
> 
> ==== Aims ==== 
> 
>  * Make error messages clear when user/dependencies have asked for something 
> to be built that can't be due to restrictions
> 
>  * Ensure that exclusion system is reliable

Solid.

> 
> ====  Proposed implementation ==== 
> 
>  1) Ensure all documentation of LICENSE field value syntax is clear, concise 
> and up-to-date (wiki and manual)
> 
>  2) Go through and audit all recipes LICENSE field values to ensure that they 
> all conform to the specifications. This includes making sure that | (package 
> may be used under one of a selection of licences) and & (recipe has mixed 
> licences that apply to the code base, so conditions of all must be observed) 
> are used correctly.
> 
>  3) bitbake/core changes:
> 
>  * LICENSE field checking must fully parse the field and understand the 
> difference between | and &, and must not e.g. mark Qt as being GPLv3 only.
> 
>  * Make the LICENSE validity checking more strict (given recipes have been 
> audited and rules are clear after above)
> 
>  * Don't exclude any recipes at parse time - simply record all excluded 
> recipes and their runtime provides in a blacklist which also includes flags 
> indicating the reason for blacklisting
> 
>  * Ensure all excluded licences in INCOMPATIBLE_LICENSE are valid (e.g. catch 
> GPL3 as apposed to GPLv3) - if not, error out
> 
>  * Check when calculating dependencies if anything is scheduled to be built 
> that is on the blacklist - if any are, gather all of them up and then stop and 
> list them in an error message along with reason and depchain for each one

This sounds like overlap with a task Scott Garman is set to work on
(Error handling in bitbake), let's make sure you guys are collaborating
here.
http://bugzilla.yoctoproject.org/show_bug.cgi?id=542


> 
>  * Check when constructing the rootfs if anything in the runtime provides 
> blacklist is going to be included - if so, error out
> 
> 
> Some further possible extensions:
> 
>  * Possibly apply similar logic to COMPATIBLE_MACHINE?
> 
>  * Replace COMMERCIAL* with some more generic exclusion mechanism that allows 
> the reason to be defined as part of the exclusion list?

Sounds reasonable, care to elaborate? Something like:

COMMERCIAL_LICENSE = "naughty packages from copyright hell"
PERL_PACKAGES = "packages I don't want to include because I dislike
Perl"
EXCLUDED_PATTERNS = "PERL_PACKAGES COMMERCIAL_LICENSE"

??

> 
>  * As a helper for non-en_US users, fail on parse if user defines any of the 
> *LICENSE* variables as *LICENCE*? (we definitely don't want the build to 
> continue and just ignore this as the user might not realise what has happened)

Darn tootin!

Looks like a good start, thanks Paul!

Cheers,
Joshua
-- 
Joshua Lock
        Yocto Build System Monkey
        Intel Open Source Technology Centre

Re: RFC: Package exclusion

[Paul Eggleton]

On Wednesday 11 May 2011 00:20:14 Joshua Lock wrote:
> >  * Check when calculating dependencies if anything is scheduled to be
> >  built  that is on the blacklist - if any are, gather all of them up and
> >  then  stop and list them in an error message along with reason and
> >  depchain for each one
> 
> This sounds like overlap with a task Scott Garman is set to work on
> (Error handling in bitbake), let's make sure you guys are collaborating
> here.
> http://bugzilla.yoctoproject.org/show_bug.cgi?id=542

I hadn't considered this - you may well be right. It depends on how Scott 
proposes to attack that particular problem, I will check with him.
 
> >  * Replace COMMERCIAL* with some more generic exclusion mechanism that
> >  allows the reason to be defined as part of the exclusion list?
> 
> Sounds reasonable, care to elaborate? Something like:
> 
> COMMERCIAL_LICENSE = "naughty packages from copyright hell"
> PERL_PACKAGES = "packages I don't want to include because I dislike
> Perl"
> EXCLUDED_PATTERNS = "PERL_PACKAGES COMMERCIAL_LICENSE"

More like:

EXCLUDED_PACKAGES += "broken1,broken2:These packages are broken by design and 
not supported;"
EXCLUDED_PACKAGES += "licenserequired,clickthru,special1:These packages 
require a special license agreement;"

The example syntax is horrible but you get the idea.

> >  * As a helper for non-en_US users, fail on parse if user defines any of
> >  the *LICENSE* variables as *LICENCE*? (we definitely don't want the build
> >  to continue and just ignore this as the user might not realise what has
> > happened)
> 
> Darn tootin!

I think you meant to say "hear hear!" or maybe even "good show!"... ;)
 
Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: RFC: Package exclusion

[Joshua Lock]

On Wed, 2011-05-11 at 10:06 +0100, Paul Eggleton wrote:
> On Wednesday 11 May 2011 00:20:14 Joshua Lock wrote:
> > >  * Check when calculating dependencies if anything is scheduled to be
> > >  built  that is on the blacklist - if any are, gather all of them up and
> > >  then  stop and list them in an error message along with reason and
> > >  depchain for each one
> > 
> > This sounds like overlap with a task Scott Garman is set to work on
> > (Error handling in bitbake), let's make sure you guys are collaborating
> > here.
> > http://bugzilla.yoctoproject.org/show_bug.cgi?id=542
> 
> I hadn't considered this - you may well be right. It depends on how Scott 
> proposes to attack that particular problem, I will check with him.

Cool. If they don't overlap I'd be surprised.

>  
> > >  * Replace COMMERCIAL* with some more generic exclusion mechanism that
> > >  allows the reason to be defined as part of the exclusion list?
> > 
> > Sounds reasonable, care to elaborate? Something like:
> > 
> > COMMERCIAL_LICENSE = "naughty packages from copyright hell"
> > PERL_PACKAGES = "packages I don't want to include because I dislike
> > Perl"
> > EXCLUDED_PATTERNS = "PERL_PACKAGES COMMERCIAL_LICENSE"
> 
> More like:
> 
> EXCLUDED_PACKAGES += "broken1,broken2:These packages are broken by design and 
> not supported;"
> EXCLUDED_PACKAGES += "licenserequired,clickthru,special1:These packages 
> require a special license agreement;"
> 
> The example syntax is horrible but you get the idea.

The idea is sound, yeah.

> 
> > >  * As a helper for non-en_US users, fail on parse if user defines any of
> > >  the *LICENSE* variables as *LICENCE*? (we definitely don't want the build
> > >  to continue and just ignore this as the user might not realise what has
> > > happened)
> > 
> > Darn tootin!
> 
> I think you meant to say "hear hear!" or maybe even "good show!"... ;)
>  

Bravo!

Joshua
-- 
Joshua Lock
        Yocto Build System Monkey
        Intel Open Source Technology Centre