BB_STAMP_POLICY

BB_STAMP_POLICY

[Koen Kooi]

FYI:

11:37:33 Koen: is BB_STAMP_POLICY documented somewhere?
11:37:47 Richard Purdie: Not really
11:38:03 Richard Purdie: Takes the values "file" "whitelist" or "full"
11:38:35 Richard Purdie: "file" is the traditional behaviour with stamps 
just checked within a given recipe
11:38:46 Richard Purdie: "full" checks all the dependencies are consistent
11:39:12 Richard Purdie: "whitelist" allows some packages to be excluded 
from the stamp checking basically so packages staging can work

Re: BB_STAMP_POLICY

[Detlef Vollmann]

Koen Kooi wrote:
> 11:37:33 Koen: is BB_STAMP_POLICY documented somewhere?
> 11:37:47 Richard Purdie: Not really
> 11:38:03 Richard Purdie: Takes the values "file" "whitelist" or "full"
> 11:38:35 Richard Purdie: "file" is the traditional behaviour with stamps 
> just checked within a given recipe
> 11:38:46 Richard Purdie: "full" checks all the dependencies are consistent
> 11:39:12 Richard Purdie: "whitelist" allows some packages to be excluded 
> from the stamp checking basically so packages staging can work
Just for info: I've tested "whitelist" pretty extensivly over the past
few weeks and I'm pretty confident that all the bugs are fixed now.

But I've run into a fundamental problem with dependency checking:
for one of our projects we're changing frequently between NPTL
and linuxthreads for glibc (we'd like to use NPTL for the features,
but run again and again into problems with NPTL).
Normally, this change is just a flip of GLIBC_ADDONS in the distro
config file.  But of course the value of this specific variable
is not tracked by the dependency checking (which is IMHO correct).
So what I did is to split the glibc recipes into a linuxthreads
one and an NPTL one (which was quite some exercise).
Now I face the problem to separate glibc between "--with-tls"
and "--without-tls", which would be a change for
EXTRA_OECONF_glibc-linuxthreads in my distro conf file, but
again this defeats dependency tracking.  So I would have to
split glibc(-linuxthreads) again, but I'm a bit reluctant
to do so...

One solution for that would be to introduce something like
use flags, which could be included into the stamp information
and used for dependency checking.  But there were a number
of discussions about use flags on this list and there was
quite some opposition.
IIRC, this opposition was mainly against the cross-package
nature of use flags, but we don't need this for dependency
checking.  What we need is a naming scheme that the dependency
checker knows about so that it can include specific variables
into the stamp information, something like
USE_glibc_ADDONS = "nptl"
USE_glibc_EXTRA_OECONF = "--with-tls --with-__thread"

What do people think about this, or is there a completely
different way to solve the underlying problem?

   Detlef

Re: BB_STAMP_POLICY

[Richard Purdie]

On Mon, 2008-06-02 at 12:35 +0200, Detlef Vollmann wrote:
> But I've run into a fundamental problem with dependency checking:
> for one of our projects we're changing frequently between NPTL
> and linuxthreads for glibc (we'd like to use NPTL for the features,
> but run again and again into problems with NPTL).
> Normally, this change is just a flip of GLIBC_ADDONS in the distro
> config file.  But of course the value of this specific variable
> is not tracked by the dependency checking (which is IMHO correct).
> So what I did is to split the glibc recipes into a linuxthreads
> one and an NPTL one (which was quite some exercise).
> Now I face the problem to separate glibc between "--with-tls"
> and "--without-tls", which would be a change for
> EXTRA_OECONF_glibc-linuxthreads in my distro conf file, but
> again this defeats dependency tracking.  So I would have to
> split glibc(-linuxthreads) again, but I'm a bit reluctant
> to do so...
> 
> One solution for that would be to introduce something like
> use flags, which could be included into the stamp information
> and used for dependency checking.  But there were a number
> of discussions about use flags on this list and there was
> quite some opposition.
> IIRC, this opposition was mainly against the cross-package
> nature of use flags, but we don't need this for dependency
> checking.  What we need is a naming scheme that the dependency
> checker knows about so that it can include specific variables
> into the stamp information, something like
> USE_glibc_ADDONS = "nptl"
> USE_glibc_EXTRA_OECONF = "--with-tls --with-__thread"
> 
> What do people think about this, or is there a completely
> different way to solve the underlying problem?

The correct solution from a bitbake/OE perspective is to have multiple
providers, one for set of options combination and to switch between them
using PREFERRED_PROVIDER.

The biggest challenge you find when doing this is lots of recipes use
${PN} internally and by creating a new package you then need to hardcode
that variable.

Since thats the main barrier to doing this I'd propose trying to remove
that obstacle. We could do this by having:

bitbake.conf:

PKGPN ?= "${PN}"

and then using PKGPN in recipes instead of PN. If you then create a
glibc-some-extra-option you can just add a line which says:

PKGPN = "glibc"

and it should all work. This would also be useful for -sdk and -native
recipes for example which currently have nastier workarounds for this
problem.

Would this solve the problem? Can anyone see a better name than PKGPN?

Regards,

Richard

Re: BB_STAMP_POLICY

[Detlef Vollmann]

Richard Purdie wrote:
> The correct solution from a bitbake/OE perspective is to have multiple
> providers, one for set of options combination and to switch between them
> using PREFERRED_PROVIDER.
I understand, and that's why I started that way.
But it starts to explode, and I have some uncomfortable feeling about
breaking changes going unnoticed...

> The biggest challenge you find when doing this is lots of recipes use
> ${PN} internally and by creating a new package you then need to hardcode
> that variable.
Another problem is the hardcoded name of packages inside recipes
(glibc-package.bbclass is a prominent example for that).

> Since thats the main barrier to doing this
I wouldn't say that's the main barrier.  It's annoying, but once
you know what to look for it's just some search and replace.

The biggest problem I see are silent breaks, because of changes in
some configuration files that are not tracked by dependency checking.

 > I'd propose trying to remove
> that obstacle. We could do this by having:
> 
> bitbake.conf:
> 
> PKGPN ?= "${PN}"
> 
> and then using PKGPN in recipes instead of PN. If you then create a
> glibc-some-extra-option you can just add a line which says:
> 
> PKGPN = "glibc"
> 
> and it should all work. This would also be useful for -sdk and -native
> recipes for example which currently have nastier workarounds for this
> problem.
> 
> Would this solve the problem?
It would make the the setup of the additional recipes easier, but
it doesn't solve the main problem of breaking changes.
For that maybe something in insane.bbclass could check and complain
about settings like EXTRA_OECONF_xyz in a conig file when
BB_STAMP_POLICY is set.

   Detlef

Re: BB_STAMP_POLICY

[Richard Purdie]

On Tue, 2008-06-03 at 00:09 +0200, Detlef Vollmann wrote:
> The biggest problem I see are silent breaks, because of changes in
> some configuration files that are not tracked by dependency checking.

I not 100% sure what you're getting at here but I'll try and follow this
to its logical conclusion.

If you change something in some recipe, say EXTRA_OECONF you then need
to force a rebuild of the recipe either by bumping PR, using the --force
option etc. to make the change take effect. This means package A would
rebuild and the way BB_STAMP_POLICY full/whitelist works means that
since A rebuilt, anything depending on A will also rebuild. So this case
is covered.

Perhaps you mean that you change some variable in say bitbake.conf and
you then don't know which recipes it may have an effect on. You
therefore don't know which to rebuild?

Bitbake can't know whether a given change in some conf file changes a
given package. What it can in theory tell is whether the some
conf/class/bb/inc file has changed or not since a given package was
built. We already have this cache logic in place but it currently has no
influence on stamp validity decisions. It would be possible to wire that
into the stamp logic so if you touch bitbake.conf anything which could
be influenced by that file (everything) would rebuild. If you touch a
class file like kernel.bbclass, just the kernel would rebuild (and
anything depending on it). Any change to local.conf would rebuild
everything.

For comparison note that your USE variables just cover the EXTRA_OECONF
variable. There are hundreds of other variables which can influence the
build and trying to build a definitive list of them would be difficult.
What makes EXTRA_OECONF special?

Cheers,

Richard

Re: BB_STAMP_POLICY

[Detlef Vollmann]

Richard Purdie wrote:
> On Tue, 2008-06-03 at 00:09 +0200, Detlef Vollmann wrote:
>> The biggest problem I see are silent breaks, because of changes in
>> some configuration files that are not tracked by dependency checking.
> 
> I not 100% sure what you're getting at here but I'll try and follow this
> to its logical conclusion.
> 
> If you change something in some recipe, say EXTRA_OECONF you then need
> to force a rebuild of the recipe either by bumping PR, using the --force
> option etc. to make the change take effect. This means package A would
> rebuild and the way BB_STAMP_POLICY full/whitelist works means that
> since A rebuilt, anything depending on A will also rebuild. So this case
> is covered.
Correct, this is the way as it should be.
But this way it's something manual, and that will get forgotten at
some time.  In our environment we sometimes build different projects
at the same time (using different chroot environments for that).
So if you want to change something for a project, you do the change
and start the build, and then you go to some other work and come
back after 20 minutes to do a short test and then publish the release
for the testers.  In such an environment, things like a forced build
easily get forgotten.
In our current build system, for switches between NPTL and linuxthreads
we have to exchange toolchain manually, and we had more than one case
where the image didn't contain the version that we expected, leading
to nasty, hard to track down errors.

> Perhaps you mean that you change some variable in say bitbake.conf and
> you then don't know which recipes it may have an effect on. You
> therefore don't know which to rebuild?
No, that's not the real problem.

> Bitbake can't know whether a given change in some conf file changes a
> given package. What it can in theory tell is whether the some
> conf/class/bb/inc file has changed or not since a given package was
> built. We already have this cache logic in place but it currently has no
> influence on stamp validity decisions. It would be possible to wire that
> into the stamp logic so if you touch bitbake.conf anything which could
> be influenced by that file (everything) would rebuild. If you touch a
> class file like kernel.bbclass, just the kernel would rebuild (and
> anything depending on it). Any change to local.conf would rebuild
> everything.
That would be the same as rebuilding a whole project because the
Makefile has changed.  I know people who do that, but I don't like this
approach.

> For comparison note that your USE variables just cover the EXTRA_OECONF
> variable. There are hundreds of other variables which can influence the
> build and trying to build a definitive list of them would be difficult.
> What makes EXTRA_OECONF special?
Sorry, but it seems that I didn't make myself very clear.
What I proposed is a naming scheme like:
USE_package_VARIABLE
where 'USE' is the marker for bitbake that this is something to track,
'package' is the package for which this needs to be tracked, and
'VARIABLE' is the name of the variable as it is used in the package's
recipe.

   Detlef