[RFC] Layers, PRINC and bbappends

[RFC] Layers, PRINC and bbappends

[Koen Kooi]

Hi,

For the past 2 weeks I've been trying to fix the angstrom feeds manually due to a combination of PRSERV deciding that going backwards is OK and multiple layers getting updates that make PR go backwards as well.

Getting a simple reduction for PRSERV going backwards is nigh impossible[1], so I can't complain too much about that, but I can complain about what people do :)

What triggered PR going backwards this time was a BSP removing its netbase bbappend because it wasn't needed anymore. That's great, I hate netbase bbappends since they tend to be ifupdown specific and don't work as intended with connman/NM/netctl/etc. Long story short:

	ERROR: Package version for package netbase-dbg went backwards which would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2)
	ERROR: Package version for package netbase-staticdev went backwards which would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2)
	ERROR: Package version for package netbase-dev went backwards which would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2)
	ERROR: Package version for package netbase-doc went backwards which would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2)
	ERROR: Package version for package netbase-locale went backwards which would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2)
	ERROR: Package version for package netbase went backwards which would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2)

To avoid things like that in the future I have a few recommendations I'd like to get feedback on:

1) For a given BSP split it in multiple layers:
	a) One 'core' BSP layer with only:
		o Bootloader
		o Kernel
		o Tooling to build bootloader/kernel/image
	b) One BSP layer with bbappends for shadow/netbase/xorg-conf/etc
	c) One or more layers with bbappends for libav/clutter/etc[2]


2) *Never* use PRINC in type b) bbappends

3) Avoid PRINC in general

4) make buildhistory.bbclass mandatory in OE-core

There's another use case that suffers from PRINC problems and that is removing layers when they break parsing and the maintainer is slow to push patches. With the current situation I have to choose between "being able to do builds" and "having working feeds".

During the last TSC meeting we discussed that there is currently no way to force maintainers to behave. I think the most we can do is have clear guidelines on the OE side and enforce those during the "Yocto Project Compatible" process. 
And have the layer index orange/red flag layers that do stupid things. Come to think of it, that would actually be the most visible way to go, having wikipedia style warning banners on the offending layers.

Thanks,

Koen

[1] It always happens after editing bblayers.conf and dragging in major update to layers. Since I tend to do both at the same time I can't say which action triggers it or if it's the combination.
[2] I spent 3 days trying to figure out why mesa was so *$(*$(@ broken before I finally located the bbappends in meta-ti that were causing this breakage

Re: [RFC] Layers, PRINC and bbappends

[Paul Eggleton]

Hi Koen,

On Tuesday 14 May 2013 09:36:02 Koen Kooi wrote:
> What triggered PR going backwards this time was a BSP removing its netbase
> bbappend because it wasn't needed anymore. That's great, I hate netbase
> bbappends since they tend to be ifupdown specific and don't work as
> intended with connman/NM/netctl/etc. Long story short:
> 
> 	ERROR: Package version for package netbase-dbg went backwards which would
> break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package version
> for package netbase-staticdev went backwards which would break package
> feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package version for package
> netbase-dev went backwards which would break package feeds from (0:5.0-r3.1
> to 0:5.0-r1.2) ERROR: Package version for package netbase-doc went
> backwards which would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2)
> ERROR: Package version for package netbase-locale went backwards which
> would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package
> version for package netbase went backwards which would break package feeds
> from (0:5.0-r3.1 to 0:5.0-r1.2)
> 
> To avoid things like that in the future I have a few recommendations I'd
> like to get feedback on:
> 
> 1) For a given BSP split it in multiple layers:
> 	a) One 'core' BSP layer with only:
> 		o Bootloader
> 		o Kernel
> 		o Tooling to build bootloader/kernel/image
> 	b) One BSP layer with bbappends for shadow/netbase/xorg-conf/etc
> 	c) One or more layers with bbappends for libav/clutter/etc[2]
 
I don't like this. This will mean effectively 3x the number of BSP layers, with 
independent testing effectively expected for the different combinations; I don't 
think this will be practical. It's fine for distros with pre-supplied lists of 
layers but lots of users are assembling their own layer stacks and this is 
just going to add pain for them.

It's probably time we looked seriously at more effective analysis tools for 
examining what a layer is actually doing. We now have the variable history in 
bitbake which means that pinpointing a change to a variable down to the 
specific line in the file that made the change is now possible; we should be 
able to build upon this to provide tools that determine when e.g. BSP layers 
are doing things that affect builds for other machines.

Of course, getting maintainers to deal with these issues is another problem, 
but identifying them in the first place would help a lot.

> 2) *Never* use PRINC in type b) bbappends
> 
> 3) Avoid PRINC in general

Do we even need PRINC at all any more? I realise we have PRINC values around 
in bbappends that we can't easily drop without causing the problem you 
mentioned.

> 4) make buildhistory.bbclass mandatory in OE-core

Buildhistory does have some overhead as you know, which is why it's not on by 
default. I wonder if we could move the version going backwards check to 
package.bbclass or package QA, instead making it read the previous values from 
pkgdata. I'm not sure if sstate would get in the way in that the previous set 
of files might already be removed by the time it got to do_package; this would 
need to be investigated. Even if that is the case it might be that we can work 
around it.

> During the last TSC meeting we discussed that there is currently no way to
> force maintainers to behave. I think the most we can do is have clear
> guidelines on the OE side and enforce those during the "Yocto Project
> Compatible" process. 

Being more specific in the Yocto Project Compatible process requirements might 
help; I wasn't involved in developing that process so I can't really comment 
more than that. That wouldn't help with other layers in the OE community that 
aren't put through that process however.

I think we need to improve the documentation we have about what is and what is 
not appropriate in a BSP layer. Most of the time it's not that maintainers are 
too lazy or deliberately setting out to force values, it's that they don't 
know they're doing anything wrong because we haven't documented all of the 
best practices anywhere. There was recently a question on this very mailing 
list about what constitute distro policy settings in the context of what 
shouldn't go into a BSP layer; the response has been a resounding "".

> And have the layer index orange/red flag layers that
> do stupid things. Come to think of it, that would actually be the most
> visible way to go, having wikipedia style warning banners on the offending
> layers.

I think that's reasonable as long as maintainers can understand the problem. 
We'd need to be notifying maintainers directly about specifically what they are 
doing wrong and how to fix it; just putting a warning in the index isn't enough 
IMO. The layer index does record maintainer information and regularly reads 
layer metadata, so we have the infrastructure to do this now, just not the 
specific tools.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [RFC] Layers, PRINC and bbappends

[Koen Kooi]

Op 15 mei 2013, om 16:44 heeft Paul Eggleton <paul.eggleton@linux.intel.com> het volgende geschreven:

> Hi Koen,
> 
> On Tuesday 14 May 2013 09:36:02 Koen Kooi wrote:
>> What triggered PR going backwards this time was a BSP removing its netbase
>> bbappend because it wasn't needed anymore. That's great, I hate netbase
>> bbappends since they tend to be ifupdown specific and don't work as
>> intended with connman/NM/netctl/etc. Long story short:
>> 
>> 	ERROR: Package version for package netbase-dbg went backwards which would
>> break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package version
>> for package netbase-staticdev went backwards which would break package
>> feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package version for package
>> netbase-dev went backwards which would break package feeds from (0:5.0-r3.1
>> to 0:5.0-r1.2) ERROR: Package version for package netbase-doc went
>> backwards which would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2)
>> ERROR: Package version for package netbase-locale went backwards which
>> would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package
>> version for package netbase went backwards which would break package feeds
>> from (0:5.0-r3.1 to 0:5.0-r1.2)
>> 
>> To avoid things like that in the future I have a few recommendations I'd
>> like to get feedback on:
>> 
>> 1) For a given BSP split it in multiple layers:
>> 	a) One 'core' BSP layer with only:
>> 		o Bootloader
>> 		o Kernel
>> 		o Tooling to build bootloader/kernel/image
>> 	b) One BSP layer with bbappends for shadow/netbase/xorg-conf/etc
>> 	c) One or more layers with bbappends for libav/clutter/etc[2]
> 
> I don't like this. This will mean effectively 3x the number of BSP layers, with 
> independent testing effectively expected for the different combinations; I don't 
> think this will be practical. It's fine for distros with pre-supplied lists of 
> layers but lots of users are assembling their own layer stacks and this is 
> just going to add pain for them.

The flip side is that adding BSP layers becomes a lot safer. I don't mind adding layers. Adding layers is easy. Removing layers is pretty much impossible, though.

> It's probably time we looked seriously at more effective analysis tools for 
> examining what a layer is actually doing. We now have the variable history in 
> bitbake which means that pinpointing a change to a variable down to the 
> specific line in the file that made the change is now possible; we should be 
> able to build upon this to provide tools that determine when e.g. BSP layers 
> are doing things that affect builds for other machines.
> 
> Of course, getting maintainers to deal with these issues is another problem, 
> but identifying them in the first place would help a lot.

Yes, layer maintainers are proving to be the weak link is the layer system.

>> 2) *Never* use PRINC in type b) bbappends
>> 
>> 3) Avoid PRINC in general
> 
> Do we even need PRINC at all any more? I realise we have PRINC values around 
> in bbappends that we can't easily drop without causing the problem you 
> mentioned.

If PRSERV would work reliably we wouldn't need it. But even with the current flaky PRSERV PRINC is causing more problems than it's supposed to solve.

>> 4) make buildhistory.bbclass mandatory in OE-core
> 
> Buildhistory does have some overhead as you know, which is why it's not on by 
> default. I wonder if we could move the version going backwards check to 
> package.bbclass or package QA, instead making it read the previous values from 
> pkgdata. I'm not sure if sstate would get in the way in that the previous set 
> of files might already be removed by the time it got to do_package; this would 
> need to be investigated. Even if that is the case it might be that we can work 
> around it.

I personally think the overhead is worth it, but as you say, moving bits to package.bbclass (and maybe using a sqlite db) would be very good as well.

>> During the last TSC meeting we discussed that there is currently no way to
>> force maintainers to behave. I think the most we can do is have clear
>> guidelines on the OE side and enforce those during the "Yocto Project
>> Compatible" process. 
> 
> Being more specific in the Yocto Project Compatible process requirements might 
> help; I wasn't involved in developing that process so I can't really comment 
> more than that. That wouldn't help with other layers in the OE community that 
> aren't put through that process however.
> 
> I think we need to improve the documentation we have about what is and what is 
> not appropriate in a BSP layer. Most of the time it's not that maintainers are 
> too lazy or deliberately setting out to force values, it's that they don't 
> know they're doing anything wrong because we haven't documented all of the 
> best practices anywhere. There was recently a question on this very mailing 
> list about what constitute distro policy settings in the context of what 
> shouldn't go into a BSP layer; the response has been a resounding "".
> 
>> And have the layer index orange/red flag layers that
>> do stupid things. Come to think of it, that would actually be the most
>> visible way to go, having wikipedia style warning banners on the offending
>> layers.
> 
> I think that's reasonable as long as maintainers can understand the problem. 
> We'd need to be notifying maintainers directly about specifically what they are 
> doing wrong and how to fix it; just putting a warning in the index isn't enough 
> IMO. The layer index does record maintainer information and regularly reads 
> layer metadata, so we have the infrastructure to do this now, just not the 
> specific tools.

My thinking was that adding such a banner would email the maintainers of that layer. That still assumes that the layer maintainer actually reads email, which I don't think is true for a lot of layers :/

regards,

Koen

Re: [RFC] Layers, PRINC and bbappends

[Paul Eggleton]

On Thursday 16 May 2013 08:31:33 Koen Kooi wrote:
> Op 15 mei 2013, om 16:44 heeft Paul Eggleton <paul.eggleton@linux.intel.com>
> het volgende geschreven:
> > On Tuesday 14 May 2013 09:36:02 Koen Kooi wrote:
> >> What triggered PR going backwards this time was a BSP removing its
> >> netbase
> >> bbappend because it wasn't needed anymore. That's great, I hate netbase
> >> bbappends since they tend to be ifupdown specific and don't work as
> >> 
> >> intended with connman/NM/netctl/etc. Long story short:
> >> 	ERROR: Package version for package netbase-dbg went backwards which
> >> 	would
> >> 
> >> break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package
> >> version
> >> for package netbase-staticdev went backwards which would break package
> >> feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package version for package
> >> netbase-dev went backwards which would break package feeds from
> >> (0:5.0-r3.1
> >> to 0:5.0-r1.2) ERROR: Package version for package netbase-doc went
> >> backwards which would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2)
> >> ERROR: Package version for package netbase-locale went backwards which
> >> would break package feeds from (0:5.0-r3.1 to 0:5.0-r1.2) ERROR: Package
> >> version for package netbase went backwards which would break package
> >> feeds
> >> from (0:5.0-r3.1 to 0:5.0-r1.2)
> >> 
> >> To avoid things like that in the future I have a few recommendations I'd
> >> like to get feedback on:
> >> 
> >> 1) For a given BSP split it in multiple layers:
> >> 	a) One 'core' BSP layer with only:
> >> 		o Bootloader
> >> 		o Kernel
> >> 		o Tooling to build bootloader/kernel/image
> >> 	
> >> 	b) One BSP layer with bbappends for shadow/netbase/xorg-conf/etc
> >> 	c) One or more layers with bbappends for libav/clutter/etc[2]
> > 
> > I don't like this. This will mean effectively 3x the number of BSP layers,
> > with independent testing effectively expected for the different
> > combinations; I don't think this will be practical. It's fine for distros
> > with pre-supplied lists of layers but lots of users are assembling their
> > own layer stacks and this is just going to add pain for them.
> 
> The flip side is that adding BSP layers becomes a lot safer. I don't mind
> adding layers. Adding layers is easy.

For distros like Angstrom it is, yes, as I said above.

> Removing layers is pretty much impossible, though.

With PRINC gone would there be any other issue that would make removing layers 
a problem?

> >> 2) *Never* use PRINC in type b) bbappends
> >> 
> >> 3) Avoid PRINC in general
> > 
> > Do we even need PRINC at all any more? I realise we have PRINC values
> > around in bbappends that we can't easily drop without causing the problem
> > you mentioned.
> 
> If PRSERV would work reliably we wouldn't need it. But even with the current
> flaky PRSERV PRINC is causing more problems than it's supposed to solve.

Can you help us to find out what's wrong with the PR server, assuming it's more 
than just PRINC-related? I suspect you'd need to be preserving the database 
prior to each build; maybe dumping the database into a file within buildhistory 
at the end would do the trick and then we'd have the repo revisions to go with 
that in the same place.

> >> During the last TSC meeting we discussed that there is currently no way
> >> to force maintainers to behave. I think the most we can do is have clear
> >> guidelines on the OE side and enforce those during the "Yocto Project
> >> Compatible" process.
> > 
> > Being more specific in the Yocto Project Compatible process requirements
> > might help; I wasn't involved in developing that process so I can't
> > really comment more than that. That wouldn't help with other layers in
> > the OE community that aren't put through that process however.
> > 
> > I think we need to improve the documentation we have about what is and
> > what is not appropriate in a BSP layer. Most of the time it's not that
> > maintainers are too lazy or deliberately setting out to force values,
> > it's that they don't know they're doing anything wrong because we haven't
> > documented all of the best practices anywhere. There was recently a
> > question on this very mailing list about what constitute distro policy
> > settings in the context of what shouldn't go into a BSP layer; the
> > response has been a resounding "".> 

You didn't respond to this; I think this is the biggest issue we have. 
Contributors cannot be expected to just know the right way to do things unless 
you've made it easy for them to learn in the first place.

> >> And have the layer index orange/red flag layers that
> >> do stupid things. Come to think of it, that would actually be the most
> >> visible way to go, having wikipedia style warning banners on the
> >> offending
> >> layers.
> > 
> > I think that's reasonable as long as maintainers can understand the
> > problem. We'd need to be notifying maintainers directly about
> > specifically what they are doing wrong and how to fix it; just putting a
> > warning in the index isn't enough IMO. The layer index does record
> > maintainer information and regularly reads layer metadata, so we have the
> > infrastructure to do this now, just not the specific tools.
> 
> My thinking was that adding such a banner would email the maintainers of
> that layer. That still assumes that the layer maintainer actually reads
> email, which I don't think is true for a lot of layers :/

It's not just adding the banner but what the banner contains; in most cases 
the banner is going to be much shorter than the necessary explanation. In the 
most recent issue which we've been alluding to, the maintainer was very 
responsive when contacted directly and once they understood the nature of the 
problem, it was solved rapidly.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [OE-core] [RFC] Layers, PRINC and bbappends

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

On Thu, May 16, 2013 at 09:29:48AM +0100, Paul Eggleton wrote:
> On Thursday 16 May 2013 08:31:33 Koen Kooi wrote:
> > The flip side is that adding BSP layers becomes a lot safer. I don't mind
> > adding layers. Adding layers is easy.
> 
> For distros like Angstrom it is, yes, as I said above.
> 
> > Removing layers is pretty much impossible, though.
> 
> With PRINC gone would there be any other issue that would make removing layers 
> a problem?

Does PR service really fix removing layers? I don't think so, if you add
layer and checksum of foo is changed AUTOPR is incremented, but then you
remove the layer and the checksum is unfortunately the same as what you
had before AUTOPR will go backwards.

PRINCs were doing this at least consistently and in reproducible way,
removing layer with PR service will cause only some AUTOPRs going
backwards (if there were other changes in between).

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [OE-core] [RFC] Layers, PRINC and bbappends

[Paul Eggleton]

On Thursday 16 May 2013 10:59:54 Martin Jansa wrote:
> On Thu, May 16, 2013 at 09:29:48AM +0100, Paul Eggleton wrote:
> > On Thursday 16 May 2013 08:31:33 Koen Kooi wrote:
> > > Removing layers is pretty much impossible, though.
> > 
> > With PRINC gone would there be any other issue that would make removing
> > layers a problem?
> 
> Does PR service really fix removing layers? I don't think so, if you add
> layer and checksum of foo is changed AUTOPR is incremented, but then you
> remove the layer and the checksum is unfortunately the same as what you
> had before AUTOPR will go backwards.

Then I would have to say if that's not the behaviour you want we should change 
it or at least make it optional to force it to always increment even if the 
checksum is the same as a previous value.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [OE-core] [RFC] Layers, PRINC and bbappends

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1145 bytes --]

On Thu, May 16, 2013 at 10:10:45AM +0100, Paul Eggleton wrote:
> On Thursday 16 May 2013 10:59:54 Martin Jansa wrote:
> > On Thu, May 16, 2013 at 09:29:48AM +0100, Paul Eggleton wrote:
> > > On Thursday 16 May 2013 08:31:33 Koen Kooi wrote:
> > > > Removing layers is pretty much impossible, though.
> > > 
> > > With PRINC gone would there be any other issue that would make removing
> > > layers a problem?
> > 
> > Does PR service really fix removing layers? I don't think so, if you add
> > layer and checksum of foo is changed AUTOPR is incremented, but then you
> > remove the layer and the checksum is unfortunately the same as what you
> > had before AUTOPR will go backwards.
> 
> Then I would have to say if that's not the behaviour you want we should change 
> it or at least make it optional to force it to always increment even if the 
> checksum is the same as a previous value.

That won't work with multiple builders using the same PR server (I don't
wont other builders to always increment PRAUTOs used in builds which
populate official feed).

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [OE-core] [RFC] Layers, PRINC and bbappends

[Richard Purdie]

On Thu, 2013-05-16 at 11:29 +0200, Martin Jansa wrote:
> On Thu, May 16, 2013 at 10:10:45AM +0100, Paul Eggleton wrote:
> > On Thursday 16 May 2013 10:59:54 Martin Jansa wrote:
> > > On Thu, May 16, 2013 at 09:29:48AM +0100, Paul Eggleton wrote:
> > > > On Thursday 16 May 2013 08:31:33 Koen Kooi wrote:
> > > > > Removing layers is pretty much impossible, though.
> > > > 
> > > > With PRINC gone would there be any other issue that would make removing
> > > > layers a problem?
> > > 
> > > Does PR service really fix removing layers? I don't think so, if you add
> > > layer and checksum of foo is changed AUTOPR is incremented, but then you
> > > remove the layer and the checksum is unfortunately the same as what you
> > > had before AUTOPR will go backwards.
> > 
> > Then I would have to say if that's not the behaviour you want we should change 
> > it or at least make it optional to force it to always increment even if the 
> > checksum is the same as a previous value.
> 
> That won't work with multiple builders using the same PR server (I don't
> wont other builders to always increment PRAUTOs used in builds which
> populate official feed).

I just went and looked at the code and as far as I can tell PRServ
defaults to always incrementing upon a change. To get the other
behaviour you'd need nohist=False to PRData() in lib/prserv/db.py and I
can't see where that is set other than to the default :/. I'm tired and
perhaps missing something though.

Cheers,

Richard

Re: [RFC] Layers, PRINC and bbappends

[Richard Purdie]

On Tue, 2013-05-14 at 09:36 +0200, Koen Kooi wrote:
> To avoid things like that in the future I have a few recommendations
> I'd like to get feedback on:

In future can we please discuss this on the oe-core list or at least
give a heads up there as this is pretty key to the core of the project
and we've implicitly agreed to discuss architecture there? I appreciate
this affects oe-devel layer maintainers too.

> 1) For a given BSP split it in multiple layers:
> 	a) One 'core' BSP layer with only:
> 		o Bootloader
> 		o Kernel
> 		o Tooling to build bootloader/kernel/image
> 	b) One BSP layer with bbappends for shadow/netbase/xorg-conf/etc
> 	c) One or more layers with bbappends for libav/clutter/etc[2]

You're missing d) which would presumably be everything not in a-c) e.g.
binary graphics libs which aren't bbappends.

I have a feeling this will break more than it solves so I'm not keen on
this and I think we need to dig deeper into the problem and find a
better solution.
> 
> 2) *Never* use PRINC in type b) bbappends
> 3) Avoid PRINC in general

Could we undergo some mass purge of PRINC from master branches and then
change the PR values once and for all in the core recipes? We could make
PRINC throw a bb.fatal() from that point onwards.

> 4) make buildhistory.bbclass mandatory in OE-core

and force enable the PR backwards check? Again, I think the people
you're trying to target to fix these issues are also the people who'd
probably not have the history around to see the issues that most bother
you.

I agree there are some issues here but I think we're going to need a
more creative fix.

My personal view is we need to reduce the number of bbappends. One way
to do this is to make a general/central bsp-files type recipe which
generates all the standard packages (tscalibration data, xorg config and
all the other single machine specific config files).

Yes, this would be a disruptive change but it would significantly clean
up the BSP layers and hopefully make maintenance easier. It is on the
Yocto project proposed features list although we've only got the idea,
we haven't discussed with the community etc. Now would see as good a
time as any.

> There's another use case that suffers from PRINC problems and that is
> removing layers when they break parsing and the maintainer is slow to
> push patches. With the current situation I have to choose between
> "being able to do builds" and "having working feeds".

We have talked about a version wildcard for bbappend. You are still at
the mercy of what the bbappend does though (e.g. when I include
meta-raspberrypi, it changes the psplash logo, regardless of whether I'm
building for the pi).

In many ways the bbappend is too easy and encourages anti-social
behaviour. I would recommend bouncing meta-rpi as being non yocto
project compatible due to the psplash issue if it were to apply which it
has not.

> During the last TSC meeting we discussed that there is currently no
> way to force maintainers to behave. I think the most we can do is have
> clear guidelines on the OE side and enforce those during the "Yocto
> Project Compatible" process. 

How many layers apply for that status though?

> And have the layer index orange/red flag layers that do stupid things.
> Come to think of it, that would actually be the most visible way to
> go, having wikipedia style warning banners on the offending layers.

I think Paul had good feedback here.

> [1] It always happens after editing bblayers.conf and dragging in
> major update to layers. Since I tend to do both at the same time I
> can't say which action triggers it or if it's the combination.
> [2] I spent 3 days trying to figure out why mesa was so *$(*$(@ broken
> before I finally located the bbappends in meta-ti that were causing
> this breakage

I do appreciate understanding where some of the problem arises from,
thanks!

Cheers,

Richard

Re: [RFC] Layers, PRINC and bbappends

[Paul Barker]

On 15 May 2013 21:33, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:

>
> (e.g. when I include
> meta-raspberrypi, it changes the psplash logo, regardless of whether I'm
> building for the pi).
>
> In many ways the bbappend is too easy and encourages anti-social
> behaviour. I would recommend bouncing meta-rpi as being non yocto
> project compatible due to the psplash issue if it were to apply which it
> has not.
>

I noticed the psplash logo when building a qemu image with
meta-raspberrypi included, will look into fixing it so it only goes
into MACHINE="raspberrypi" builds. Copying in yocto@ and Andrei as
this is raspberrypi related.

--
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

Re: [RFC] Layers, PRINC and bbappends

[Paul Barker]

On 16 May 2013 00:02, Paul Barker <paul@paulbarker.me.uk> wrote:
> On 15 May 2013 21:33, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
>
>>
>> (e.g. when I include
>> meta-raspberrypi, it changes the psplash logo, regardless of whether I'm
>> building for the pi).
>>
>> In many ways the bbappend is too easy and encourages anti-social
>> behaviour. I would recommend bouncing meta-rpi as being non yocto
>> project compatible due to the psplash issue if it were to apply which it
>> has not.
>>
>
> I noticed the psplash logo when building a qemu image with
> meta-raspberrypi included, will look into fixing it so it only goes
> into MACHINE="raspberrypi" builds. Copying in yocto@ and Andrei as
> this is raspberrypi related.
>

I've just sent a patch to the yocto@ list to fix this but it's brought
up two things:

1) In openembedded-core/meta/classes/image.bbclass SPLASH is set with
'?='. I'm overriding this with '=' in
meta-raspberrypi/conf/machine/raspberrypi.conf which means it can't be
overridden further in local.conf. Is this worth changing to '??=' in
image.bbclass, '?=' in the machine conf so that it can be overridden
with '=' in local.conf? Is my understanding of the overrides correct
here?

2) If you look at the patch I've basically added the raspberrypi logo
as an extra psplash image and overridden SPLASH in the machine conf to
make this the default for raspberrypi. Is this the intended approach
for customising psplash images? There's a lot of interesting logic in
the psplash recipe and it took me a while to figure out. If I've got
it right it may be worth adding some documentation comments to explain
this method. If I've got it wrong could someone correct me.

Both of these (assignment change and adding documentation) would be
patches to openembedded-core.

Thanks.

--
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

Re: [OE-core] [RFC] Layers, PRINC and bbappends

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1586 bytes --]

On Wed, May 22, 2013 at 09:47:39AM +0100, Paul Barker wrote:
> On 16 May 2013 00:02, Paul Barker <paul@paulbarker.me.uk> wrote:
> > On 15 May 2013 21:33, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> >
> >>
> >> (e.g. when I include
> >> meta-raspberrypi, it changes the psplash logo, regardless of whether I'm
> >> building for the pi).
> >>
> >> In many ways the bbappend is too easy and encourages anti-social
> >> behaviour. I would recommend bouncing meta-rpi as being non yocto
> >> project compatible due to the psplash issue if it were to apply which it
> >> has not.
> >>
> >
> > I noticed the psplash logo when building a qemu image with
> > meta-raspberrypi included, will look into fixing it so it only goes
> > into MACHINE="raspberrypi" builds. Copying in yocto@ and Andrei as
> > this is raspberrypi related.
> >
> 
> I've just sent a patch to the yocto@ list to fix this but it's brought
> up two things:
> 
> 1) In openembedded-core/meta/classes/image.bbclass SPLASH is set with
> '?='. I'm overriding this with '=' in
> meta-raspberrypi/conf/machine/raspberrypi.conf which means it can't be
> overridden further in local.conf. Is this worth changing to '??=' in
> image.bbclass, '?=' in the machine conf so that it can be overridden
> with '=' in local.conf? Is my understanding of the overrides correct
> here?

You can still use:
SPLASH_forcevariable = "foo"
SPLASH_your-distro = "foo"
SPLASH_your-machine = "foo"
or any other OVERRIDE in local.conf.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [OE-core] [RFC] Layers, PRINC and bbappends

[Paul Eggleton]

On Wednesday 22 May 2013 11:17:38 Martin Jansa wrote:
> On Wed, May 22, 2013 at 09:47:39AM +0100, Paul Barker wrote:
> > I've just sent a patch to the yocto@ list to fix this but it's brought
> > up two things:
> > 
> > 1) In openembedded-core/meta/classes/image.bbclass SPLASH is set with
> > '?='. I'm overriding this with '=' in
> > meta-raspberrypi/conf/machine/raspberrypi.conf which means it can't be
> > overridden further in local.conf. Is this worth changing to '??=' in
> > image.bbclass, '?=' in the machine conf so that it can be overridden
> > with '=' in local.conf? Is my understanding of the overrides correct
> > here?
> 
> You can still use:
> SPLASH_forcevariable = "foo"
> SPLASH_your-distro = "foo"
> SPLASH_your-machine = "foo"
> or any other OVERRIDE in local.conf.

This is true; FWIW as the original author of that line in image.bbclass I
wouldn't object to changing it to ??= though.

Having said that, I would note that people are best advised not to try to set
too much from local.conf, especially when it's stuff that affects what is in the
final image. Anyone doing anything moderately custom should really have their
own distro config. FYI, we now have a section in the Yocto Project development
manual documenting this:

http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#creating-your-own-distribution

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [OE-core] [RFC] Layers, PRINC and bbappends

[Chris Larson]

On Wed, May 22, 2013 at 3:14 AM, Phil Blundell <pb@pbcl.net> wrote:

> On Wed, 2013-05-22 at 09:47 +0100, Paul Barker wrote:
> > I've just sent a patch to the yocto@ list to fix this but it's brought
> > up two things:
> >
> > 1) In openembedded-core/meta/classes/image.bbclass SPLASH is set with
> > '?='. I'm overriding this with '=' in
> > meta-raspberrypi/conf/machine/raspberrypi.conf which means it can't be
> > overridden further in local.conf. Is this worth changing to '??=' in
> > image.bbclass, '?=' in the machine conf so that it can be overridden
> > with '=' in local.conf? Is my understanding of the overrides correct
> > here?
>
> You can always override it from local.conf using a _forcevariable
> override (which used to be named "_local", in fact, but that terminology
> turned out to be a bit confusing), or a _MACHINE override of course.  So
> I don't think there is any major problem here.
>
> But, that said, it probably is true that the semantics of ??= are less
> surprising for users (since it isn't position-dependent in the way
> that ?= is) which might make it a better choice for image.bbclass.


??= is great in config files, but not ideal in classes that aren't globally
inherited. If bitbake.conf sets a ??=, and a distro .conf updates the
default by using ??= again, and then the class uses ??=, it'll replace the
latest default set by the config. So, I think for default values, classes
should use ?=, config files should use ??=.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: [OE-core] [RFC] Layers, PRINC and bbappends

[Paul Barker]

On 23 May 2013 03:50, Chris Larson <clarson@kergoth.com> wrote:
> On Wed, May 22, 2013 at 3:14 AM, Phil Blundell <pb@pbcl.net> wrote:
>>
>> On Wed, 2013-05-22 at 09:47 +0100, Paul Barker wrote:
>> > I've just sent a patch to the yocto@ list to fix this but it's brought
>> > up two things:
>> >
>> > 1) In openembedded-core/meta/classes/image.bbclass SPLASH is set with
>> > '?='. I'm overriding this with '=' in
>> > meta-raspberrypi/conf/machine/raspberrypi.conf which means it can't be
>> > overridden further in local.conf. Is this worth changing to '??=' in
>> > image.bbclass, '?=' in the machine conf so that it can be overridden
>> > with '=' in local.conf? Is my understanding of the overrides correct
>> > here?
>>
>> You can always override it from local.conf using a _forcevariable
>> override (which used to be named "_local", in fact, but that terminology
>> turned out to be a bit confusing), or a _MACHINE override of course.  So
>> I don't think there is any major problem here.
>>
>> But, that said, it probably is true that the semantics of ??= are less
>> surprising for users (since it isn't position-dependent in the way
>> that ?= is) which might make it a better choice for image.bbclass.
>
>
> ??= is great in config files, but not ideal in classes that aren't globally
> inherited. If bitbake.conf sets a ??=, and a distro .conf updates the
> default by using ??= again, and then the class uses ??=, it'll replace the
> latest default set by the config. So, I think for default values, classes
> should use ?=, config files should use ??=.

Lets leave this as it is then.

I've had no responses to the second point I made or feedback on my
patch to meta-raspberrypi. I really think it's worth adding a
documentation comment to the psplash recipe to explain the intended
way to customise the splash image as the logic there isn't
straightforward. My approach was to append the image to SPLASH_IMAGES
in a bbappend and then set this as the default via the SPLASH variable
in the machine config file for raspberrypi. I imagine it'd be a
similar approach for setting a distro default. It then ensures that
just including a machine or distro layer with a bbappend to psplash
doesn't force it's image on the user unless they're building for that
machine or distro. Is that the correct approach? If so I'm happy to
submit a patch adding something to this effect as a documentation
comment to the psplash recipe.

--
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

Re: [OE-core] [RFC] Layers, PRINC and bbappends

[Koen Kooi]

Op 15 mei 2013, om 22:33 heeft Richard Purdie <richard.purdie@linuxfoundation.org> het volgende geschreven:

> On Tue, 2013-05-14 at 09:36 +0200, Koen Kooi wrote:
>> To avoid things like that in the future I have a few recommendations
>> I'd like to get feedback on:
> 
> In future can we please discuss this on the oe-core list or at least
> give a heads up there as this is pretty key to the core of the project
> and we've implicitly agreed to discuss architecture there? I appreciate
> this affects oe-devel layer maintainers too.

I meant to send it to oe-core, but I didn't check autocomplete. Anyway, seems I reached the people I wanted to reach :)

> 
>> 1) For a given BSP split it in multiple layers:
>> 	a) One 'core' BSP layer with only:
>> 		o Bootloader
>> 		o Kernel
>> 		o Tooling to build bootloader/kernel/image
>> 	b) One BSP layer with bbappends for shadow/netbase/xorg-conf/etc
>> 	c) One or more layers with bbappends for libav/clutter/etc[2]
> 
> You're missing d) which would presumably be everything not in a-c) e.g.
> binary graphics libs which aren't bbappends.

I'd group those with c)

> I have a feeling this will break more than it solves so I'm not keen on
> this and I think we need to dig deeper into the problem and find a
> better solution.

>> 
>> 2) *Never* use PRINC in type b) bbappends
>> 3) Avoid PRINC in general
> 
> Could we undergo some mass purge of PRINC from master branches and then
> change the PR values once and for all in the core recipes? We could make
> PRINC throw a bb.fatal() from that point onwards.

That would be nice.

> 
>> 4) make buildhistory.bbclass mandatory in OE-core
> 
> and force enable the PR backwards check?

Yes

> Again, I think the people
> you're trying to target to fix these issues are also the people who'd
> probably not have the history around to see the issues that most bother
> you.

True, but they'd have less plausible deniability :)

> I agree there are some issues here but I think we're going to need a
> more creative fix.
> 
> My personal view is we need to reduce the number of bbappends. One way
> to do this is to make a general/central bsp-files type recipe which
> generates all the standard packages (tscalibration data, xorg config and
> all the other single machine specific config files).
> 
> Yes, this would be a disruptive change but it would significantly clean
> up the BSP layers and hopefully make maintenance easier. It is on the
> Yocto project proposed features list although we've only got the idea,
> we haven't discussed with the community etc. Now would see as good a
> time as any.
> 
>> There's another use case that suffers from PRINC problems and that is
>> removing layers when they break parsing and the maintainer is slow to
>> push patches. With the current situation I have to choose between
>> "being able to do builds" and "having working feeds".
> 
> We have talked about a version wildcard for bbappend. You are still at
> the mercy of what the bbappend does though (e.g. when I include
> meta-raspberrypi, it changes the psplash logo, regardless of whether I'm
> building for the pi).
> 
> In many ways the bbappend is too easy and encourages anti-social
> behaviour.

The funny thing is that I've run into bb(appends) trying to be social and breaking things, e.g.

linux-yocto.bbappend:

SRC_URI = "git://foo"
SRCREV_mymachine = "1849032784092359023"
ANOTHER_VAR_mymachine = "meta"

And that breaks parsing for every machine that isn't named "mymachine". 

> I would recommend bouncing meta-rpi as being non yocto
> project compatible due to the psplash issue if it were to apply which it
> has not.
> 
>> During the last TSC meeting we discussed that there is currently no
>> way to force maintainers to behave. I think the most we can do is have
>> clear guidelines on the OE side and enforce those during the "Yocto
>> Project Compatible" process. 
> 
> How many layers apply for that status though?

Ideally all of them.

regards,

Koen

>> And have the layer index orange/red flag layers that do stupid things.
>> Come to think of it, that would actually be the most visible way to
>> go, having wikipedia style warning banners on the offending layers.
> 
> I think Paul had good feedback here.
> 
>> [1] It always happens after editing bblayers.conf and dragging in
>> major update to layers. Since I tend to do both at the same time I
>> can't say which action triggers it or if it's the combination.
>> [2] I spent 3 days trying to figure out why mesa was so *$(*$(@ broken
>> before I finally located the bbappends in meta-ti that were causing
>> this breakage
> 
> I do appreciate understanding where some of the problem arises from,
> thanks!
> 
> Cheers,
> 
> Richard
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: [OE-core] [RFC] Layers, PRINC and bbappends

[Paul Eggleton]

On Thursday 16 May 2013 08:23:35 Koen Kooi wrote:
> Op 15 mei 2013, om 22:33 heeft Richard Purdie
> <richard.purdie@linuxfoundation.org> het volgende geschreven:
> > On Tue, 2013-05-14 at 09:36 +0200, Koen Kooi wrote:
> >> 2) *Never* use PRINC in type b) bbappends
> >> 3) Avoid PRINC in general
> > 
> > Could we undergo some mass purge of PRINC from master branches and then
> > change the PR values once and for all in the core recipes? We could make
> > PRINC throw a bb.fatal() from that point onwards.
> 
> That would be nice.

Maybe we could add some logic into the PR server that if it sees PRINC go 
backwards, it could just print a warning and increment the stored PR value to 
compensate? That might mean storing the previous value of PRINC if we're not 
already doing that of course.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [OE-core] [RFC] Layers, PRINC and bbappends

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1611 bytes --]

On Thu, May 16, 2013 at 09:37:23AM +0100, Paul Eggleton wrote:
> On Thursday 16 May 2013 08:23:35 Koen Kooi wrote:
> > Op 15 mei 2013, om 22:33 heeft Richard Purdie
> > <richard.purdie@linuxfoundation.org> het volgende geschreven:
> > > On Tue, 2013-05-14 at 09:36 +0200, Koen Kooi wrote:
> > >> 2) *Never* use PRINC in type b) bbappends
> > >> 3) Avoid PRINC in general
> > > 
> > > Could we undergo some mass purge of PRINC from master branches and then
> > > change the PR values once and for all in the core recipes? We could make
> > > PRINC throw a bb.fatal() from that point onwards.
> > 
> > That would be nice.
> 
> Maybe we could add some logic into the PR server that if it sees PRINC go 
> backwards, it could just print a warning and increment the stored PR value to 
> compensate? That might mean storing the previous value of PRINC if we're not 
> already doing that of course.

PRINC is applied to INC_PR/PR
PR server handles only PRAUTO AFAIK
PKGR ?= "${PR}${EXTENDPRAUTO}"

so this wont work

BTW: some files are using AUTOPR as reference to PRAUTO
meta/classes/prexport.bbclass:            bb.warn("prexport_handler: No AUTOPR values found for %s" % ver)
meta/classes/primport.bbclass:        #import all exported AUTOPR values
scripts/bitbake-prserv-tool:    echo -e "\texport <file>: export and lock down the AUTOPR values from the PR service into a file for release."
scripts/bitbake-prserv-tool:    echo -e "\timport <file>: import the AUTOPR values from the exported file into the PR service."

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]