Master Stability

Master Stability

[Saul Wold]

Folks,

We had a rash of changes that have broken master, I am trying to resolve 
the last few breakages, which mostly are in meta-toolchain-sdk build. I 
realized that this is not in our normal test build path.

I have fixes for most of the break (I hope), there are still a couple of 
items that we need to address.

There is a failure for libatomics on mips, there may be a configuration 
issue.

Please be sure to verify your builds before makeing pull requests.


Thanks
	Sau!

Re: Master Stability

[Elizabeth Flanagan]

Acked-by: Beth Flanagan

(Stands on build engineer soap box)

If at all possible, when you try to verify a fix, do a full clean build
for more than just one arch using BB_NUMBER_THREADS and PARALLEL_MAKE.
With the number of pull requests coming through, it becomes more and
more vital to verify patches prior to sending them out.

Preferably, I'd like to start seeing people add something like a
Verified-with: <build target> tag to pull requests, so we can at least
identify pulls that need more peer review than normal.

-b


On 12/06/2010 05:58 PM, Saul Wold wrote:
> 
> Folks,
> 
> We had a rash of changes that have broken master, I am trying to resolve 
> the last few breakages, which mostly are in meta-toolchain-sdk build. I 
> realized that this is not in our normal test build path.
> 
> I have fixes for most of the break (I hope), there are still a couple of 
> items that we need to address.
> 
> There is a failure for libatomics on mips, there may be a configuration 
> issue.
> 
> Please be sure to verify your builds before makeing pull requests.
> 
> 
> Thanks
> 	Sau!
> 
> 
> 
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: Master Stability

[Joshua Lock]

On Tue, 2010-12-07 at 09:44 -0800, Elizabeth Flanagan wrote:
> Acked-by: Beth Flanagan
> 
> (Stands on build engineer soap box)
> 
> If at all possible, when you try to verify a fix, do a full clean build
> for more than just one arch using BB_NUMBER_THREADS and PARALLEL_MAKE.
> With the number of pull requests coming through, it becomes more and
> more vital to verify patches prior to sending them out.
> 
> Preferably, I'd like to start seeing people add something like a
> Verified-with: <build target> tag to pull requests, so we can at least
> identify pulls that need more peer review than normal.

Personally I'd like to see links to an autobuilder report showing it has
succeeded and if such a tag is *not* on a pull request it's
automatically pushed to an autobuilder instance with the success of the
build sent back to the list as a reply to the patch thread.

I think hardware constraints make this a bit of a pipe dream though?

Cheers,
Joshua
-- 
Joshua Lock
        Intel Open Source Technology Centre

Re: Master Stability

[Scott Garman]

On 12/07/2010 10:43 AM, Joshua Lock wrote:
> On Tue, 2010-12-07 at 09:44 -0800, Elizabeth Flanagan wrote:
>> Acked-by: Beth Flanagan
>>
>> (Stands on build engineer soap box)
>>
>> If at all possible, when you try to verify a fix, do a full clean build
>> for more than just one arch using BB_NUMBER_THREADS and PARALLEL_MAKE.
>> With the number of pull requests coming through, it becomes more and
>> more vital to verify patches prior to sending them out.
>>
>> Preferably, I'd like to start seeing people add something like a
>> Verified-with:<build target>  tag to pull requests, so we can at least
>> identify pulls that need more peer review than normal.
>
> Personally I'd like to see links to an autobuilder report showing it has
> succeeded and if such a tag is *not* on a pull request it's
> automatically pushed to an autobuilder instance with the success of the
> build sent back to the list as a reply to the patch thread.
>
> I think hardware constraints make this a bit of a pipe dream though?

We're definitely hardware constrained for it. I want a pony also. ;)

Beth has started working on customizing Buildbot's web interface to 
allow finer-grained control over what gets built. A longer-range goal of 
this is eventually be able to pick which targets and architectures get 
built from a custom autobuilder buildset. That way someone with a commit 
to quilt (which has very few dependencies) wouldn't have to build all of 
poky-image-sato to demonstrate their build works. With something like 
that in place, we can start to consider having individual developers 
test their changes on our autobuilder infrastructure. Which would kick ass.

Scott

-- 
Scott Garman
Embedded Linux Distro Engineer - Yocto Project

Re: Master Stability

[deVries, Alex]

On 2010-12-07, at 1:54 PM, Scott Garman wrote:

> We're definitely hardware constrained for it. I want a pony also. ;)

Ponies are pretty cheap these days. 

Does Intel have a plan to expand the amount of hardware used for automated builds? It would give us a lot more build coverage and help with overall build quality.


- A

Re: Master Stability

[Stewart, David C]

> From: poky-bounces@yoctoproject.org [mailto:poky-
> bounces@yoctoproject.org] On Behalf Of deVries, Alex
> Sent: Tuesday, December 07, 2010 1:27 PM
>
> Does Intel have a plan to expand the amount of hardware used for
> automated builds? It would give us a lot more build coverage and help
> with overall build quality.

New server has arrived, waiting to be integrated into the hosted data center.

I'm trying to get a bead on whether we need more, and can try to budget this for next quarter if the new one isn't enough.

Re: Master Stability

[Saul Wold]

On 12/07/2010 05:06 PM, Stewart, David C wrote:
>> From: poky-bounces@yoctoproject.org [mailto:poky-
>> bounces@yoctoproject.org] On Behalf Of deVries, Alex
>> Sent: Tuesday, December 07, 2010 1:27 PM
>>
>> Does Intel have a plan to expand the amount of hardware used for
>> automated builds? It would give us a lot more build coverage and help
>> with overall build quality.
>
> New server has arrived, waiting to be integrated into the hosted data center.
>
> I'm trying to get a bead on whether we need more, and can try to budget this for next quarter if the new one isn't enough.
>
We will need new hardware next qtr, maybe even soon, as we start to 
enable more targets and add support for swabber on a semi-regular basis 
our cycles are going to be greatly diminished.

We should really look into getting a large multi-core machine to help.

Sau!


  _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>

Re: Master Stability

[Joshua Lock]

On Tue, 2010-12-07 at 10:54 -0800, Scott Garman wrote:
> On 12/07/2010 10:43 AM, Joshua Lock wrote:
> > On Tue, 2010-12-07 at 09:44 -0800, Elizabeth Flanagan wrote:
> >> Acked-by: Beth Flanagan
> >>
> >> (Stands on build engineer soap box)
> >>
> >> If at all possible, when you try to verify a fix, do a full clean build
> >> for more than just one arch using BB_NUMBER_THREADS and PARALLEL_MAKE.
> >> With the number of pull requests coming through, it becomes more and
> >> more vital to verify patches prior to sending them out.
> >>
> >> Preferably, I'd like to start seeing people add something like a
> >> Verified-with:<build target>  tag to pull requests, so we can at least
> >> identify pulls that need more peer review than normal.
> >
> > Personally I'd like to see links to an autobuilder report showing it has
> > succeeded and if such a tag is *not* on a pull request it's
> > automatically pushed to an autobuilder instance with the success of the
> > build sent back to the list as a reply to the patch thread.
> >
> > I think hardware constraints make this a bit of a pipe dream though?
> 
> We're definitely hardware constrained for it. I want a pony also. ;)

I've become increasingly jealous because some colleagues have a unicorn!
It's called git test-sequence:
http://dustin.github.com/2010/03/28/git-test-sequence.html

> 
> Beth has started working on customizing Buildbot's web interface to 
> allow finer-grained control over what gets built. A longer-range goal of 
> this is eventually be able to pick which targets and architectures get 
> built from a custom autobuilder buildset. That way someone with a commit 
> to quilt (which has very few dependencies) wouldn't have to build all of 
> poky-image-sato to demonstrate their build works. With something like 
> that in place, we can start to consider having individual developers 
> test their changes on our autobuilder infrastructure. Which would kick ass.

No kidding, can't wait!

Hopefully we can tie something like git test-sequence into this such
that we can do these things from our lovely shell environments :-)

Cheers,
Joshua
-- 
Joshua Lock
        Intel Open Source Technology Centre

Re: Master Stability

[Tian, Kevin]

>From: Saul Wold
>Sent: Tuesday, December 07, 2010 9:59 AM
>
>
>Folks,
>
>We had a rash of changes that have broken master, I am trying to resolve
>the last few breakages, which mostly are in meta-toolchain-sdk build. I
>realized that this is not in our normal test build path.
>
>I have fixes for most of the break (I hope), there are still a couple of
>items that we need to address.
>
>There is a failure for libatomics on mips, there may be a configuration
>issue.
>
>Please be sure to verify your builds before makeing pull requests.
>
>

In our 0.9 cycle, I think the rule was to have people develop in an incremental
way (cover all archs), and then do a full build on one target arch completely different 
from the build arch before the pull request. 

Above should expose most problems without posing too much test burdens on
developers given the long cycle of a nightly test.

Of course this still leaves some potential issues such as parallel make issues, or
occasional bugs exposed in full build on a non-verified arch.

The reality is that current build cycle is too long. I'd suggest to pull back the concept 
of distro/stage, and then only push our bash requests to master after verifying it in 
distro/stage. This gives us an intermediate filter on master quality.

Once sstate is in good shape, we can then ask people do full build on all archs
before sending pull requests to further ensure the quality.

Thanks
Kevin