Last minute changes - Review Request

Last minute changes - Review Request

[Richard Purdie]

Coming up to release there are a few things that the extended testing
has shown up which we have fixes for and which we should consider
including in the release. I also finally got around to doing the final
sstate stress testing and found several problematic issues. Given that
sstate and checksums are a significant feature of this release, I'd
really like them to work as well as we can make them. Prior to this I
had stress tested the backend up not the use of the packages. These
changes don't change any sstate packages themselves, just the use of
them.

Since we already have the release images prepared and tested and these
are not going to change, the criteria for potential changes:

a) We can unit test the changes and be confident they don't 
   break/regress things.
b) They fix important bugs that the user can easily run into
   or that make the project look bad.
c) The changes are small, well documented and are obviously correct 
   looking at the code/patch.
d) The don't change the generated images.

I'm proposing the following, for each I've provided a rationale:

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=d5504a4275d94868e28b00c272411e82f4999d95

Printing "fatal:" to new users is worrying

[Reported by Dirk]

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=2a69c58046a86d0f783acebd8a77e9419b43139a

Users can easily hit the sanity warning about missing 32 bit libs. We
don't need this functionality at this time so we might as well turn it
off by default

[Reported by Dirk]

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=0068e55d8f64ae13a1049c37164e8b14dc33f53f

Doesn't change the build output but fixes a build issue people can
easily run into in from scratch builds due to a missing dependency.

[Reported by Dexuan]

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=6e277cb014a53aef66ae931b5142495f8a02404f

Removes the "WARNING: Function do_build doesn't exist" message which
could worry users and looks bad

[Reported by Richard]

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=fd4457199ef604dc4d5f8346c8b2a09dc3939129

Several complaints have been received about the inability to easily
delete sstate and DL_DIR contents to recover from failures. This adds a
cleanall task which does this. Downside is that this is undocumented.

[Reported by Darren and others, run into by Dirk]

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=f806c499c031fe0c4da001d41bce635088a90c52

Fixes an sstate bug where the do_package sstate packages don't install
correctly. A user could hit this if cleaning and rebuilding a package
under certain circumstances.

[Reported by Richard]

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=13f116b1ad6a955b07d4cbaba85879913c30e1ee

Fixes an issue with sstate where rebuilding a package using partially
prebuilt state would break rpm generation and silently remove all the
rpm packages.

[Reported by Richard]

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=00a96a20995cefacc52e10559029de32941ecf6e

Fixes a typo spotted in the debian packaging backend. We don't use this
by default but it should be fixed.

[Reported by Richard]

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=36f1ae42fe13dae174b7fb5eb85dc49d7d7b516b

User testing keeps showing up issues with the pseudo directory creation
failing to happen. This patch solves the problem in a brute force
fashion, once and forall.

[Reported by Mark and others]

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=3f599b3f6a47286277cdaa8503f8a8da024eadd4

Fixes an sstate issue seen on the Poky mailing list where file:// sstate
mirrors wouldn't work.

[Reported by Gary Thomas (poky list)]

http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=535a77a9b681423e2f10744aca54858c25a03cb0

Just changes a log level to make the output from sstate slightly more
readable.

[Reported by Richard]


I'm not happy about being in this position and I know Dave will be very
nervous about these late changes. To mitigate this I'd like to propose
that a selection of people (Josh, Mark, Saul?) review these changes and
report back on whether they feel these are appropriate and also give the
build some testing with these applied.

Cheers,

Richard

Re: Last minute changes - Review Request

[Stewart, David C]

>From: yocto-bounces@yoctoproject.org [mailto:yocto-
>bounces@yoctoproject.org] On Behalf Of Richard Purdie
>Sent: Friday, October 22, 2010 7:24 AM
>
>Coming up to release there are a few things that the extended testing
>has shown up which we have fixes for and which we should consider
>including in the release. I also finally got around to doing the final
>sstate stress testing and found several problematic issues. Given that
>sstate and checksums are a significant feature of this release, I'd
>really like them to work as well as we can make them. Prior to this I
>had stress tested the backend up not the use of the packages. These
>changes don't change any sstate packages themselves, just the use of
>them.
>
>Since we already have the release images prepared and tested and these
>are not going to change, the criteria for potential changes:
>
>a) We can unit test the changes and be confident they don't
>   break/regress things.
>b) They fix important bugs that the user can easily run into
>   or that make the project look bad.
>c) The changes are small, well documented and are obviously correct
>   looking at the code/patch.
>d) The don't change the generated images.
>
>I'm proposing the following, for each I've provided a rationale:
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=d5504a4275d94868e
>28b00c272411e82f4999d95
>
>Printing "fatal:" to new users is worrying
>
>[Reported by Dirk]
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=2a69c58046a86d0f7
>83acebd8a77e9419b43139a
>
>Users can easily hit the sanity warning about missing 32 bit libs. We
>don't need this functionality at this time so we might as well turn it
>off by default
>
>[Reported by Dirk]
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=0068e55d8f64ae13a
>1049c37164e8b14dc33f53f
>
>Doesn't change the build output but fixes a build issue people can
>easily run into in from scratch builds due to a missing dependency.
>
>[Reported by Dexuan]
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=6e277cb014a53aef6
>6ae931b5142495f8a02404f
>
>Removes the "WARNING: Function do_build doesn't exist" message which
>could worry users and looks bad
>
>[Reported by Richard]
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=fd4457199ef604dc4
>d5f8346c8b2a09dc3939129
>
>Several complaints have been received about the inability to easily
>delete sstate and DL_DIR contents to recover from failures. This adds a
>cleanall task which does this. Downside is that this is undocumented.
>
>[Reported by Darren and others, run into by Dirk]
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=f806c499c031fe0c4
>da001d41bce635088a90c52
>
>Fixes an sstate bug where the do_package sstate packages don't install
>correctly. A user could hit this if cleaning and rebuilding a package
>under certain circumstances.
>
>[Reported by Richard]
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=13f116b1ad6a955b0
>7d4cbaba85879913c30e1ee
>
>Fixes an issue with sstate where rebuilding a package using partially
>prebuilt state would break rpm generation and silently remove all the
>rpm packages.
>
>[Reported by Richard]
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=00a96a20995cefacc
>52e10559029de32941ecf6e
>
>Fixes a typo spotted in the debian packaging backend. We don't use this
>by default but it should be fixed.
>
>[Reported by Richard]
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=36f1ae42fe13dae17
>4b7fb5eb85dc49d7d7b516b
>
>User testing keeps showing up issues with the pseudo directory creation
>failing to happen. This patch solves the problem in a brute force
>fashion, once and forall.
>
>[Reported by Mark and others]
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=3f599b3f6a4728627
>7cdaa8503f8a8da024eadd4
>
>Fixes an sstate issue seen on the Poky mailing list where file:// sstate
>mirrors wouldn't work.
>
>[Reported by Gary Thomas (poky list)]
>
>http://git.pokylinux.org/cgit/cgit.cgi/poky/commit/?id=535a77a9b681423e2
>f10744aca54858c25a03cb0
>
>Just changes a log level to make the output from sstate slightly more
>readable.
>
>[Reported by Richard]
>
>
>I'm not happy about being in this position and I know Dave will be very
>nervous about these late changes. To mitigate this I'd like to propose
>that a selection of people (Josh, Mark, Saul?) review these changes and
>report back on whether they feel these are appropriate and also give the
>build some testing with these applied.

I'm so predictable... :-) Yes, I'm nervous. I looked at all of the patches and with the exception of one or two, they mostly seem like good ones. I will accept these if Josh/Mark/Saul give us a +1 on their review & testing.

>Cheers,
>
>Richard
>
>
>
>
>
>
>
>
>
>_______________________________________________
>yocto mailing list
>yocto@yoctoproject.org
>https://lists.pokylinux.org/listinfo/yocto

Re: Last minute changes - Review Request

[Joshua Lock]

On Fri, 2010-10-22 at 15:23 +0100, Richard Purdie wrote:
> 
> I'm not happy about being in this position and I know Dave will be very
> nervous about these late changes. To mitigate this I'd like to propose
> that a selection of people (Josh, Mark, Saul?) review these changes and
> report back on whether they feel these are appropriate and also give the
> build some testing with these applied.

I've looked over the changes and spent this afternoon testing them, I
believe they are appropriate for the 0.9 release.

Cheers,
Joshua
-- 
Joshua Lock
        Intel Open Source Technology Centre

Re: Last minute changes - Review Request

[Saul G. Wold]

On 10/22/2010 09:32 AM, Stewart, David C wrote:
>> From: yocto-bounces@yoctoproject.org [mailto:yocto-
>> bounces@yoctoproject.org] On Behalf Of Richard Purdie
>> Sent: Friday, October 22, 2010 7:24 AM
>>
>> Coming up to release there are a few things that the extended testing
>> has shown up which we have fixes for and which we should consider
>> including in the release. I also finally got around to doing the final
>> sstate stress testing and found several problematic issues. Given that
>> sstate and checksums are a significant feature of this release, I'd
>> really like them to work as well as we can make them. Prior to this I
>> had stress tested the backend up not the use of the packages. These
>> changes don't change any sstate packages themselves, just the use of
>> them.
>>
>> Since we already have the release images prepared and tested and these
>> are not going to change, the criteria for potential changes:
>>
>> a) We can unit test the changes and be confident they don't
>>    break/regress things.

For the Future: Besides doing a basic build, we need to have some real 
unit tests for bitbake and the poky infrastructure, I guess I need to 
turn this into a Testing feature request for 1.0 (look for it soon).

>> b) They fix important bugs that the user can easily run into
>>    or that make the project look bad.
After reviewing the changes I agree, don't get me wrong, I am still very 
nervous about these changes.

>> c) The changes are small, well documented and are obviously correct
>>    looking at the code/patch.
Some times we over look the obvious changes, been caught by that myself 
too many time.

>> d) The don't change the generated images.

<SNIP>

>> I'm not happy about being in this position and I know Dave will be very
>> nervous about these late changes. To mitigate this I'd like to propose
>> that a selection of people (Josh, Mark, Saul?) review these changes and
>> report back on whether they feel these are appropriate and also give the
>> build some testing with these applied.
>
> I'm so predictable... :-) Yes, I'm nervous. I looked at all of the patches and with the exception of one or two, they mostly seem like good ones. I will accept these if Josh/Mark/Saul give us a +1 on their review&  testing.
>

If there was 1 or 2 changes, I would be much happier, but there are 
almost a dozen changes, yes mostly individually they are OK, I am still 
reviewing them all, and have not started any testing with them yet.

I agree with Dave that there are a couple that I am more nervous about 
the pseudo/fakeroot as we have had so much trouble in the past, yes I 
know this will make things better, but what else will crop up?


>> Cheers,
>>
>> Richard
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.pokylinux.org/listinfo/yocto
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.pokylinux.org/listinfo/yocto

Re: Last minute changes - Review Request

[Mark Hatle]

Add a +1 to reviewed, worried, but accepting column.  They each seem reasonable, 
low-enough risk..

--Mark

On 10/22/10 12:23 PM, Saul G. Wold wrote:
> On 10/22/2010 09:32 AM, Stewart, David C wrote:
>>> From: yocto-bounces@yoctoproject.org [mailto:yocto-
>>> bounces@yoctoproject.org] On Behalf Of Richard Purdie
>>> Sent: Friday, October 22, 2010 7:24 AM
>>>
>>> Coming up to release there are a few things that the extended testing
>>> has shown up which we have fixes for and which we should consider
>>> including in the release. I also finally got around to doing the final
>>> sstate stress testing and found several problematic issues. Given that
>>> sstate and checksums are a significant feature of this release, I'd
>>> really like them to work as well as we can make them. Prior to this I
>>> had stress tested the backend up not the use of the packages. These
>>> changes don't change any sstate packages themselves, just the use of
>>> them.
>>>
>>> Since we already have the release images prepared and tested and these
>>> are not going to change, the criteria for potential changes:
>>>
>>> a) We can unit test the changes and be confident they don't
>>>     break/regress things.
>
> For the Future: Besides doing a basic build, we need to have some real
> unit tests for bitbake and the poky infrastructure, I guess I need to
> turn this into a Testing feature request for 1.0 (look for it soon).
>
>>> b) They fix important bugs that the user can easily run into
>>>     or that make the project look bad.
> After reviewing the changes I agree, don't get me wrong, I am still very
> nervous about these changes.
>
>>> c) The changes are small, well documented and are obviously correct
>>>     looking at the code/patch.
> Some times we over look the obvious changes, been caught by that myself
> too many time.
>
>>> d) The don't change the generated images.
>
> <SNIP>
>
>>> I'm not happy about being in this position and I know Dave will be very
>>> nervous about these late changes. To mitigate this I'd like to propose
>>> that a selection of people (Josh, Mark, Saul?) review these changes and
>>> report back on whether they feel these are appropriate and also give the
>>> build some testing with these applied.
>>
>> I'm so predictable... :-) Yes, I'm nervous. I looked at all of the patches and with the exception of one or two, they mostly seem like good ones. I will accept these if Josh/Mark/Saul give us a +1 on their review&   testing.
>>
>
> If there was 1 or 2 changes, I would be much happier, but there are
> almost a dozen changes, yes mostly individually they are OK, I am still
> reviewing them all, and have not started any testing with them yet.
>
> I agree with Dave that there are a couple that I am more nervous about
> the pseudo/fakeroot as we have had so much trouble in the past, yes I
> know this will make things better, but what else will crop up?
>
>
>>> Cheers,
>>>
>>> Richard
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> yocto mailing list
>>> yocto@yoctoproject.org
>>> https://lists.pokylinux.org/listinfo/yocto
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.pokylinux.org/listinfo/yocto
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.pokylinux.org/listinfo/yocto

Re: Last minute changes - Review Request

[Dirk Hohndel]

On Fri, 22 Oct 2010 12:24:40 -0500, Mark Hatle <mark.hatle@windriver.com> wrote:
> Add a +1 to reviewed, worried, but accepting column.  They each seem reasonable, 
> low-enough risk..

same here 

We won't have a bug free release. No one ever does. And any change
increases the risk.

So the question is "are they worth the added risk?". I believe the
proposed changes address issues that people WILL run into as they start
playing with things, so I think the risk is worth the reward.

/D


> On 10/22/10 12:23 PM, Saul G. Wold wrote:
> > On 10/22/2010 09:32 AM, Stewart, David C wrote:
> >>> From: yocto-bounces@yoctoproject.org [mailto:yocto-
> >>> bounces@yoctoproject.org] On Behalf Of Richard Purdie
> >>> Sent: Friday, October 22, 2010 7:24 AM
> >>>
> >>> Coming up to release there are a few things that the extended testing
> >>> has shown up which we have fixes for and which we should consider
> >>> including in the release. I also finally got around to doing the final
> >>> sstate stress testing and found several problematic issues. Given that
> >>> sstate and checksums are a significant feature of this release, I'd
> >>> really like them to work as well as we can make them. Prior to this I
> >>> had stress tested the backend up not the use of the packages. These
> >>> changes don't change any sstate packages themselves, just the use of
> >>> them.
> >>>
> >>> Since we already have the release images prepared and tested and these
> >>> are not going to change, the criteria for potential changes:
> >>>
> >>> a) We can unit test the changes and be confident they don't
> >>>     break/regress things.
> >
> > For the Future: Besides doing a basic build, we need to have some real
> > unit tests for bitbake and the poky infrastructure, I guess I need to
> > turn this into a Testing feature request for 1.0 (look for it soon).
> >
> >>> b) They fix important bugs that the user can easily run into
> >>>     or that make the project look bad.
> > After reviewing the changes I agree, don't get me wrong, I am still very
> > nervous about these changes.
> >
> >>> c) The changes are small, well documented and are obviously correct
> >>>     looking at the code/patch.
> > Some times we over look the obvious changes, been caught by that myself
> > too many time.
> >
> >>> d) The don't change the generated images.
> >
> > <SNIP>
> >
> >>> I'm not happy about being in this position and I know Dave will be very
> >>> nervous about these late changes. To mitigate this I'd like to propose
> >>> that a selection of people (Josh, Mark, Saul?) review these changes and
> >>> report back on whether they feel these are appropriate and also give the
> >>> build some testing with these applied.
> >>
> >> I'm so predictable... :-) Yes, I'm nervous. I looked at all of the patches and with the exception of one or two, they mostly seem like good ones. I will accept these if Josh/Mark/Saul give us a +1 on their review&   testing.
> >>
> >
> > If there was 1 or 2 changes, I would be much happier, but there are
> > almost a dozen changes, yes mostly individually they are OK, I am still
> > reviewing them all, and have not started any testing with them yet.
> >
> > I agree with Dave that there are a couple that I am more nervous about
> > the pseudo/fakeroot as we have had so much trouble in the past, yes I
> > know this will make things better, but what else will crop up?
> >
> >
> >>> Cheers,
> >>>
> >>> Richard
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> yocto mailing list
> >>> yocto@yoctoproject.org
> >>> https://lists.pokylinux.org/listinfo/yocto
> >> _______________________________________________
> >> yocto mailing list
> >> yocto@yoctoproject.org
> >> https://lists.pokylinux.org/listinfo/yocto
> > _______________________________________________
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.pokylinux.org/listinfo/yocto
> 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.pokylinux.org/listinfo/yocto

-- 
Dirk Hohndel
Intel Open Source Technology Center

Re: Last minute changes - Review Request

[Stewart, David C]

>From: yocto-bounces@yoctoproject.org [mailto:yocto-
>bounces@yoctoproject.org] On Behalf Of Dirk Hohndel
>Sent: Friday, October 22, 2010 10:48 AM
>
>On Fri, 22 Oct 2010 12:24:40 -0500, Mark Hatle
><mark.hatle@windriver.com> wrote:
>> Add a +1 to reviewed, worried, but accepting column.  They each seem
>reasonable,
>> low-enough risk..
>
>same here
>
>We won't have a bug free release. No one ever does. And any change
>increases the risk.
>
>So the question is "are they worth the added risk?". I believe the
>proposed changes address issues that people WILL run into as they start
>playing with things, so I think the risk is worth the reward.
>
>/D

Saul and Richard and I put a plan together for this, will create a new thread for it.

>
>> On 10/22/10 12:23 PM, Saul G. Wold wrote:
>> > On 10/22/2010 09:32 AM, Stewart, David C wrote:
>> >>> From: yocto-bounces@yoctoproject.org [mailto:yocto-
>> >>> bounces@yoctoproject.org] On Behalf Of Richard Purdie
>> >>> Sent: Friday, October 22, 2010 7:24 AM
>> >>>
>> >>> Coming up to release there are a few things that the extended
>testing
>> >>> has shown up which we have fixes for and which we should consider
>> >>> including in the release. I also finally got around to doing the
>final
>> >>> sstate stress testing and found several problematic issues. Given
>that
>> >>> sstate and checksums are a significant feature of this release,
>I'd
>> >>> really like them to work as well as we can make them. Prior to
>this I
>> >>> had stress tested the backend up not the use of the packages.
>These
>> >>> changes don't change any sstate packages themselves, just the use
>of
>> >>> them.
>> >>>
>> >>> Since we already have the release images prepared and tested and
>these
>> >>> are not going to change, the criteria for potential changes:
>> >>>
>> >>> a) We can unit test the changes and be confident they don't
>> >>>     break/regress things.
>> >
>> > For the Future: Besides doing a basic build, we need to have some
>real
>> > unit tests for bitbake and the poky infrastructure, I guess I need
>to
>> > turn this into a Testing feature request for 1.0 (look for it soon).
>> >
>> >>> b) They fix important bugs that the user can easily run into
>> >>>     or that make the project look bad.
>> > After reviewing the changes I agree, don't get me wrong, I am still
>very
>> > nervous about these changes.
>> >
>> >>> c) The changes are small, well documented and are obviously
>correct
>> >>>     looking at the code/patch.
>> > Some times we over look the obvious changes, been caught by that
>myself
>> > too many time.
>> >
>> >>> d) The don't change the generated images.
>> >
>> > <SNIP>
>> >
>> >>> I'm not happy about being in this position and I know Dave will be
>very
>> >>> nervous about these late changes. To mitigate this I'd like to
>propose
>> >>> that a selection of people (Josh, Mark, Saul?) review these
>changes and
>> >>> report back on whether they feel these are appropriate and also
>give the
>> >>> build some testing with these applied.
>> >>
>> >> I'm so predictable... :-) Yes, I'm nervous. I looked at all of the
>patches and with the exception of one or two, they mostly seem like good
>ones. I will accept these if Josh/Mark/Saul give us a +1 on their
>review&   testing.
>> >>
>> >
>> > If there was 1 or 2 changes, I would be much happier, but there are
>> > almost a dozen changes, yes mostly individually they are OK, I am
>still
>> > reviewing them all, and have not started any testing with them yet.
>> >
>> > I agree with Dave that there are a couple that I am more nervous
>about
>> > the pseudo/fakeroot as we have had so much trouble in the past, yes
>I
>> > know this will make things better, but what else will crop up?
>> >
>> >
>> >>> Cheers,
>> >>>
>> >>> Richard
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> yocto mailing list
>> >>> yocto@yoctoproject.org
>> >>> https://lists.pokylinux.org/listinfo/yocto
>> >> _______________________________________________
>> >> yocto mailing list
>> >> yocto@yoctoproject.org
>> >> https://lists.pokylinux.org/listinfo/yocto
>> > _______________________________________________
>> > yocto mailing list
>> > yocto@yoctoproject.org
>> > https://lists.pokylinux.org/listinfo/yocto
>>
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.pokylinux.org/listinfo/yocto
>
>--
>Dirk Hohndel
>Intel Open Source Technology Center
>_______________________________________________
>yocto mailing list
>yocto@yoctoproject.org
>https://lists.pokylinux.org/listinfo/yocto