[RFD] increased dependency checking

[RFD] increased dependency checking

[Frans Meulenbroeks]

Currently if a package in the DEPENDS list is modified, a package that
depends on it does not get updated even if explicitly build
E.g. when I just updated libexif and after that did a bitbake
mythplugins (which has a DEPENDS on libexif) libexif got build but
mythplugins did not get rebuild.

This is not really proper. E.g. if the lib changed a .h file the build
of the using package could fail, but it could easily get unnoticed as
one will only become aware of this when forcefully rebuilding that
using package.

I think we can do better by better exploiting the timestamp.
E.g. what about doing it this way:

If you build a package and the timestamp of e.g. do-install  (or
do-stage or another pass) of one of your DEPENDS packages is newer
than your do-install (or another task) then your dependency is most
likely newer, so you need a rebuild. (preceded with a clean to get rid
of the old stuff). This should be added to the task list.

Of course is is not as trivial as I sketch here.
This whole game needs to be played recursively bottom up.
So if A depends on B and B depends on C and C gets changed for
whatever reason, then bitbake A should lead to rebuilding B first.

A similar thing might also be needed for RDEPENDS.
If you run time depend on a package that is newer than you are, it
might also be that there are inconsistencies.

How does this sound? I assume it will increase the workload of
bitbake, but I feel this will lead to more stable packages.
Your feedback is appreciated!

Frans.

Re: [RFD] increased dependency checking

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

As usual you're completely ignoring the impact of this when using
package management, which is probably why you still are clueless why
your openssl fiasco is so much work to fix for distro maintainers.

On 28-08-10 10:23, Frans Meulenbroeks wrote:
> Currently if a package in the DEPENDS list is modified, a package that
> depends on it does not get updated even if explicitly build
> E.g. when I just updated libexif and after that did a bitbake
> mythplugins (which has a DEPENDS on libexif) libexif got build but
> mythplugins did not get rebuild.
> 
> This is not really proper. E.g. if the lib changed a .h file the build
> of the using package could fail, but it could easily get unnoticed as
> one will only become aware of this when forcefully rebuilding that
> using package.
> 
> I think we can do better by better exploiting the timestamp.
> E.g. what about doing it this way:
> 
> If you build a package and the timestamp of e.g. do-install  (or
> do-stage or another pass) of one of your DEPENDS packages is newer
> than your do-install (or another task) then your dependency is most
> likely newer, so you need a rebuild. (preceded with a clean to get rid
> of the old stuff). This should be added to the task list.
> 
> Of course is is not as trivial as I sketch here.
> This whole game needs to be played recursively bottom up.
> So if A depends on B and B depends on C and C gets changed for
> whatever reason, then bitbake A should lead to rebuilding B first.
> 
> A similar thing might also be needed for RDEPENDS.
> If you run time depend on a package that is newer than you are, it
> might also be that there are inconsistencies.
> 
> How does this sound? I assume it will increase the workload of
> bitbake, but I feel this will lead to more stable packages.
> Your feedback is appreciated!
> 
> Frans.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFMeSIVMkyGM64RGpERAk1lAKClt2Qr+fffyiGo7tRdRh99KCjLWACfbZDP
2TlBdc6DaXkJaKKT2ZZu+YY=
=2b/V
-----END PGP SIGNATURE-----

Re: [RFD] increased dependency checking

[Chris Larson]

On Sat, Aug 28, 2010 at 7:49 AM, Koen Kooi <k.kooi@student.utwente.nl>wrote:

> As usual you're completely ignoring the impact of this when using
> package management, which is probably why you still are clueless why
> your openssl fiasco is so much work to fix for distro maintainers.


If someone is missing something, then explain it to them.  Whining and
calling people clueless does absolutely nothing for the project but alienate
developers.  If you need to take out your frustration, buy a punching bag,
don't do it on the mailing list, please.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: [RFD] increased dependency checking

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 28-08-10 16:55, Chris Larson wrote:
> On Sat, Aug 28, 2010 at 7:49 AM, Koen Kooi <k.kooi@student.utwente.nl>wrote:
> 
>> As usual you're completely ignoring the impact of this when using
>> package management, which is probably why you still are clueless why
>> your openssl fiasco is so much work to fix for distro maintainers.
> 
> 
> If someone is missing something, then explain it to them. 

Like I did here
http://lists.linuxtogo.org/pipermail/openembedded-devel/2010-August/022867.html
? How many times do I need to explain the same thing to the same person?
Graeme also pointed it out on multiple occasions this month....
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFMeSXCMkyGM64RGpERAvq+AKCO/3M2XqAwaoVxHeJyM3GS7itOBgCfX1QP
a42RepdtB5WnE7LoA/+oaHo=
=bFcT
-----END PGP SIGNATURE-----

Re: [RFD] increased dependency checking

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 28-08-10 16:55, Chris Larson wrote:
> On Sat, Aug 28, 2010 at 7:49 AM, Koen Kooi <k.kooi@student.utwente.nl>wrote:
> 
>> As usual you're completely ignoring the impact of this when using
>> package management, which is probably why you still are clueless why
>> your openssl fiasco is so much work to fix for distro maintainers.
> 
> 
> If someone is missing something, then explain it to them.  Whining and
> calling people clueless does absolutely nothing for the project but alienate
> developers.  If you need to take out your frustration, buy a punching bag,
> don't do it on the mailing list, please.

And Chris, you don't have much right to say anything about this, since I
raised my frustration with all this to the TSC almost two weeks ago, and
I have only heard from Holger something and like "I haven't followed OE
the past months".
I haven't heard anything else, so keep getting more and more frustrated
by frans moving goal posts ("ack or nak", "nak with explanation" "only
naks from flawless maintainers") with the TSC doing seemingly nothing.

I do *not* have the time to read oe-devel daily, nor to do proper test
builds within a day. Especially not with the current mega patches posted
for review. I must admit that them being posted for review is an
improvement, but by the looks of it the respective maintainers don't get
a say since they aren't doing a perfect job.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFMeSoyMkyGM64RGpERAoVtAKCL+pF8wx34pS6hSi/+HtA3vXwNSQCeMhgi
dR0qTSGHVO3iokRIYAd1FnU=
=YA8h
-----END PGP SIGNATURE-----

Re: [RFD] increased dependency checking

[Chris Larson]

On Sat, Aug 28, 2010 at 8:24 AM, Koen Kooi <k.kooi@student.utwente.nl>wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 28-08-10 16:55, Chris Larson wrote:
> > On Sat, Aug 28, 2010 at 7:49 AM, Koen Kooi <k.kooi@student.utwente.nl
> >wrote:
> >
> >> As usual you're completely ignoring the impact of this when using
> >> package management, which is probably why you still are clueless why
> >> your openssl fiasco is so much work to fix for distro maintainers.
> >
> >
> > If someone is missing something, then explain it to them.  Whining and
> > calling people clueless does absolutely nothing for the project but
> alienate
> > developers.  If you need to take out your frustration, buy a punching
> bag,
> > don't do it on the mailing list, please.
>
> And Chris, you don't have much right to say anything about this, since I
> raised my frustration with all this to the TSC almost two weeks ago, and
> I have only heard from Holger something and like "I haven't followed OE
> the past months".


As a member of the project who cares about its image, I have plenty of room
to say anything I like.  The TSC hasn't met to discuss this, but the issue
has been raised with frans, by mickey if I recall correctly.


> I haven't heard anything else, so keep getting more and more frustrated
> by frans moving goal posts ("ack or nak", "nak with explanation" "only
> naks from flawless maintainers") with the TSC doing seemingly nothing.


I think it's perfectly reasonable to request an explanation with a NAK, and
I'm not the only one.


> I do *not* have the time to read oe-devel daily, nor to do proper test
> builds within a day. Especially not with the current mega patches posted
> for review. I must admit that them being posted for review is an
> improvement, but by the looks of it the respective maintainers don't get
> a say since they aren't doing a perfect job.


A say is fine, childish behavior and insults are not.  This isn't the first
time we've had to deal with this sort of behavior from you, and I'm sure it
won't be the last.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: [RFD] increased dependency checking

[Frans Meulenbroeks]

I will not go into the accusations and incorrectnesses in the posts above.
The only thing I'll say about it on this nice sunday morning is:

Let him who is without sin cast the first stone (Gospel of John
chapter 8 verse 7)

There is another thread on the removal of recipes. Please take your
gripes there and do not hijack all kind of topics by bringing this up.

And back to the original topic:
Actually despite what some people may think I do understand the openssl issue.

The reason I raised it was to actually get to rebuild of the recipes
that are affected by a change within a recipe that would break
requiring packages.
This can be done by a PR bump of all users, but our depends are far
from complete and (apart from grep xyz recipes/*/*) there is not
really a way to find out who uses a recipe.
(and that grep is quite cumbersome if e.g. you want to find out who
depends on popular or short names).

I was unaware of the BB_STAMP_POLICY (and haven't found docs about it
yet). That seems to resolve the openssl issue (at least in a local
build). An alternative could be a flag in a bb file that indicates it
as vulnerable so we could have a different policy behaviour for those
recipes.

I am quite aware that packet management brings additional complications.
If you rebuild the package then at least the new package depends on
the new lib so everyone getting the new version gets the new lib.
And the package installer could just keep the old lib adjacent to the new lib.
That is not so strange. On the system I am typing this I have e.g.
/lib/libreadline.so.5
/lib/libreadline.so.5.2
/lib/libreadline.so.6
/lib/libreadline.so.6.0
That way I feel a migration from one lib to another could go quite smoothly.

An alternative solution could be to not only look at the version but
also at the time a package is created (so if a version shows up with a
newer timestamp it is upgraded). With the above rebuild strategy that
could also work.
(and yes: I know this also has some con's).
Or we could explictly give recipes that produce libraries with a
different SONAME in a differently named recipe (e.g. with the soname
in the PN part and not in PV. (and yes, that most likely has some
package manager impact too).

I have no idea on what the best solution is. That is also why i
started this thread (to deal with the first step in it).
The only thing I am sure of is that the current solution has some
weaknesses (as clearly indicated by the openssl issue), and we should
try to improve that.
One way to achieve that is to have an open and constructive discussion
on it. In such a discussion there will always be bright ideas and
less-than-bright ideas, but at least such a discussion may separate
the wheat from the chaff and lead to improvement.

Have a nice day!
Frans.

Re: [RFD] increased dependency checking

[Frans Meulenbroeks]

2010/8/29 Frans Meulenbroeks <fransmeulenbroeks@gmail.com>:

>
> I am quite aware that packet management brings additional complications.
> If you rebuild the package then at least the new package depends on
> the new lib so everyone getting the new version gets the new lib.
> And the package installer could just keep the old lib adjacent to the new lib.
> That is not so strange. On the system I am typing this I have e.g.
> /lib/libreadline.so.5
> /lib/libreadline.so.5.2
> /lib/libreadline.so.6
> /lib/libreadline.so.6.0
> That way I feel a migration from one lib to another could go quite smoothly.
>
>
Actually for openssl this is not possible as the libraries cannot coexist.
In case lib versions cannot coexist, I feel putting them in different
recipes and providing a virtual would be the best solution
For openssl that would probably mean having an openssl-0.9 and
openssl-1.0 recipe that both provide openssl, where the distro can
choose.
Different recipe names make it much clearer that it are not alternatives.

Frans

Re: [RFD] increased dependency checking

[Vitus Jensen]

Hello Frans,

On Sat, 28 Aug 2010, Frans Meulenbroeks wrote:

> Currently if a package in the DEPENDS list is modified, a package that
> depends on it does not get updated even if explicitly build
> E.g. when I just updated libexif and after that did a bitbake
> mythplugins (which has a DEPENDS on libexif) libexif got build but
> mythplugins did not get rebuild.

First: bitbake isn't make.  So doing it half-automatic is OK.

> This is not really proper. E.g. if the lib changed a .h file the build
> of the using package could fail, but it could easily get unnoticed as
> one will only become aware of this when forcefully rebuilding that
> using package.

In my opinion a change of a header is a change of interface, and a change 
of the package the lib is packed into.  This always requires a bump of PR 
of that package.

> I think we can do better by better exploiting the timestamp.
> E.g. what about doing it this way:

So you don't need to exploit the timestamps, you "just" have to keep the 
dependency chain at the moment a package was built.  On the next run you 
compare the dependency chain and rebuild if a package got a new PV-PR.

This information could be kept in the bitbake cache.  When someone changes
  distro / machine / local conf he has to do the rebuild himself.  after 
all, bitbake isn't make.

Best regards,
    Vitus

-- 
Vitus Jensen, Hannover, Germany, Universe (current)
pgp public key available from keyservers

Re: [RFD] increased dependency checking

[C Michael Sundius]

I have been using a addition that I made to the stamp policy a while ago.
I sent it to the OE list and the bitbake list but I don't think that it was
ever adopted,
eventhough I *think* that I got one or two positive replies to it:

http://article.gmane.org/gmane.comp.handhelds.openembedded/29858/

I would love to see this patch added to bitbake so I don't have to keep it
around
in my repository.

do any of you have any ferther comments?

thanks.


On Mon, Aug 30, 2010 at 9:59 AM, Vitus Jensen <vjensen@gmx.de> wrote:

> Hello Frans,
>
>
> On Sat, 28 Aug 2010, Frans Meulenbroeks wrote:
>
>  Currently if a package in the DEPENDS list is modified, a package that
>> depends on it does not get updated even if explicitly build
>> E.g. when I just updated libexif and after that did a bitbake
>> mythplugins (which has a DEPENDS on libexif) libexif got build but
>> mythplugins did not get rebuild.
>>
>
> First: bitbake isn't make.  So doing it half-automatic is OK.
>
>
>  This is not really proper. E.g. if the lib changed a .h file the build
>> of the using package could fail, but it could easily get unnoticed as
>> one will only become aware of this when forcefully rebuilding that
>> using package.
>>
>
> In my opinion a change of a header is a change of interface, and a change
> of the package the lib is packed into.  This always requires a bump of PR of
> that package.
>
>
>  I think we can do better by better exploiting the timestamp.
>> E.g. what about doing it this way:
>>
>
> So you don't need to exploit the timestamps, you "just" have to keep the
> dependency chain at the moment a package was built.  On the next run you
> compare the dependency chain and rebuild if a package got a new PV-PR.
>
> This information could be kept in the bitbake cache.  When someone changes
>  distro / machine / local conf he has to do the rebuild himself.  after
> all, bitbake isn't make.
>
> Best regards,
>   Vitus
>
> --
> Vitus Jensen, Hannover, Germany, Universe (current)
> pgp public key available from keyservers
>
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: [RFD] increased dependency checking

[Chris Larson]

On Mon, Aug 30, 2010 at 10:28 AM, C Michael Sundius <msundius@sundius.com>wrote:

> I have been using a addition that I made to the stamp policy a while ago.
> I sent it to the OE list and the bitbake list but I don't think that it was
> ever adopted,
> eventhough I *think* that I got one or two positive replies to it:
>
> http://article.gmane.org/gmane.comp.handhelds.openembedded/29858/
>
> I would love to see this patch added to bitbake so I don't have to keep it
> around
> in my repository.
>
> do any of you have any ferther comments?
>

Hmm, should it always be used, the way the patch does currently, or should
it be more the way the whitelist is done, where it only affects for
fulldeptree?  I'm not opposed to this going into bitbake, personally.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: [RFD] increased dependency checking

[Frans Meulenbroeks]

2010/9/4 Chris Larson <clarson@kergoth.com>:
> On Mon, Aug 30, 2010 at 10:28 AM, C Michael Sundius <msundius@sundius.com>wrote:
>
>> I have been using a addition that I made to the stamp policy a while ago.
>> I sent it to the OE list and the bitbake list but I don't think that it was
>> ever adopted,
>> eventhough I *think* that I got one or two positive replies to it:
>>
>> http://article.gmane.org/gmane.comp.handhelds.openembedded/29858/
>>
>> I would love to see this patch added to bitbake so I don't have to keep it
>> around
>> in my repository.
>>
>> do any of you have any ferther comments?
>>
>
> Hmm, should it always be used, the way the patch does currently, or should
> it be more the way the whitelist is done, where it only affects for
> fulldeptree?  I'm not opposed to this going into bitbake, personally.
>
I can't judge on the remarks from Chris on the implementation.
The patch from Michael does look sound to me, but I have never looked
at the whitelist code.
The only thing missing in Michaels patch is some documentation on
these nwe vars.

Frans