testing branch 2010-08-23

testing branch 2010-08-23

[Cliff Brake]

Hello,

As detailed in the following page, we are trying to get a testing
branch going where we try to verify OE builds for a number of
combinations of distro/machine/target/

http://wiki.openembedded.net/index.php/Testing

I have branched dev.oe.org to the testing-testing branch, and have
verified it builds the following.

beagleboard 	angstrom-2008.1 	beagleboard-linuxtag2010-demo-image
	Ubuntu 10.04 64-bit 	User:Cbrake
beagleboard 	angstrom-2008.1 	console-image 	Ubuntu 10.04 64-bit 	User:Cbrake 	

Next Monday, we will merge testing-testing to the testing branch, and
tag it with the testing_2010-08-23 tag, and note the combinations that
build in the above wiki page.

We are looking for people who are willing to test clean builds of
other combinations on a regular basis, or other ideas on how to make
this "testing" concept work.  My initial thought is to work on a
weekly cycle, but of course not every combination will be tested every
week.

If this gains enough traction, perhaps we will want to make the
testing branch the default OE meta-data branch so that new users by
default get something that will probably build.  I think its also very
useful to give users a known starting point.  With the myriad of
combinations available, its useful to know what is a good baseline.

Thanks,
Cliff

-- 
=================
http://bec-systems.com

Re: testing branch 2010-08-23

[Stefan Schmidt]

Hello.

On Mon, 2010-08-23 at 19:49, Cliff Brake wrote:
> http://wiki.openembedded.net/index.php/Testing

Added myself with bug20 as machine and openjdk-6 as target. (That is until we
have a real bug image in dev)

> I have branched dev.oe.org to the testing-testing branch, and have
> verified it builds the following.
> 
> beagleboard 	angstrom-2008.1 	beagleboard-linuxtag2010-demo-image
> 	Ubuntu 10.04 64-bit 	User:Cbrake
> beagleboard 	angstrom-2008.1 	console-image 	Ubuntu 10.04 64-bit 	User:Cbrake 	

My build was also working fine:
bug20 	angstrom-2008.1 	openjdk-6 	Debian Unstable 64-bit 	Stefan testing_2010-08-23 

regards
Stefan Schmidt

Re: testing branch 2010-08-23

[Chris Larson]

On Mon, Aug 23, 2010 at 4:49 PM, Cliff Brake <cliff.brake@gmail.com> wrote:

> Hello,
>
> As detailed in the following page, we are trying to get a testing
> branch going where we try to verify OE builds for a number of
> combinations of distro/machine/target/
>
> http://wiki.openembedded.net/index.php/Testing
>
> I have branched dev.oe.org to the testing-testing branch, and have
> verified it builds the following.
>
> beagleboard     angstrom-2008.1         beagleboard-linuxtag2010-demo-image
>        Ubuntu 10.04 64-bit     User:Cbrake
> beagleboard     angstrom-2008.1         console-image   Ubuntu 10.04 64-bit
>     User:Cbrake
>
> Next Monday, we will merge testing-testing to the testing branch, and
> tag it with the testing_2010-08-23 tag, and note the combinations that
> build in the above wiki page.
>
> We are looking for people who are willing to test clean builds of
> other combinations on a regular basis, or other ideas on how to make
> this "testing" concept work.  My initial thought is to work on a
> weekly cycle, but of course not every combination will be tested every
> week.
>
> If this gains enough traction, perhaps we will want to make the
> testing branch the default OE meta-data branch so that new users by
> default get something that will probably build.  I think its also very
> useful to give users a known starting point.  With the myriad of
> combinations available, its useful to know what is a good baseline.


I like the concept.  It seems not unlike the 'master' vs 'next' or linux vs
linux-test, or indeed, as you imply, unstable vs testing in debian.  I do
want to point out that you could, if you chose to, include the tested
baselines in the message attached to the tag (if annotated).  The only
downside to that being you couldn't add to it after the fact the way you
could with a wiki.  At a minimum, I'd link to the wiki page for that tag
from a tag message.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: testing branch 2010-08-23

[Cliff Brake]

On Tue, Aug 24, 2010 at 10:58 AM, Chris Larson <clarson@kergoth.com> wrote:

> I like the concept.  It seems not unlike the 'master' vs 'next' or linux vs
> linux-test, or indeed, as you imply, unstable vs testing in debian.  I do
> want to point out that you could, if you chose to, include the tested
> baselines in the message attached to the tag (if annotated).  The only
> downside to that being you couldn't add to it after the fact the way you
> could with a wiki.  At a minimum, I'd link to the wiki page for that tag
> from a tag message.

Thanks for the ideas -- I have updated the process some, and I think
we could apply the tag on the following Monday after the testing is
complete -- this would allow us to add the annotations.

I'm also thinking on the next cycle to use the name testing-next
(instead of testing-testing) for branch we create every week for
testing.

Thanks,
Cliff

-- 
=================
http://bec-systems.com

Re: testing branch 2010-08-23

[Chris Larson]

On Fri, Aug 27, 2010 at 8:04 AM, Cliff Brake <cliff.brake@gmail.com> wrote:

> On Tue, Aug 24, 2010 at 10:58 AM, Chris Larson <clarson@kergoth.com>
> wrote:
>
> > I like the concept.  It seems not unlike the 'master' vs 'next' or linux
> vs
> > linux-test, or indeed, as you imply, unstable vs testing in debian.  I do
> > want to point out that you could, if you chose to, include the tested
> > baselines in the message attached to the tag (if annotated).  The only
> > downside to that being you couldn't add to it after the fact the way you
> > could with a wiki.  At a minimum, I'd link to the wiki page for that tag
> > from a tag message.
>
> Thanks for the ideas -- I have updated the process some, and I think
> we could apply the tag on the following Monday after the testing is
> complete -- this would allow us to add the annotations.
>
> I'm also thinking on the next cycle to use the name testing-next
> (instead of testing-testing) for branch we create every week for
> testing.


Another similar idea would be to use git-notes to mark up commits/tags with
the builds known to work at that point, and git log would show that
information.  Unfortunately git-notes fetching/pushing/merging isn't quite
up to par yet, but something to think about.  The advantage of that approach
is that the notes are supplementary information to the commits / tags, you
can manipulate them after the fact without changing the commit hash :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: testing branch 2010-08-23

[Gary Thomas]

On 08/23/2010 05:49 PM, Cliff Brake wrote:
> Hello,
>
> As detailed in the following page, we are trying to get a testing
> branch going where we try to verify OE builds for a number of
> combinations of distro/machine/target/
>
> http://wiki.openembedded.net/index.php/Testing
>
> I have branched dev.oe.org to the testing-testing branch, and have
> verified it builds the following.
>
> beagleboard 	angstrom-2008.1 	beagleboard-linuxtag2010-demo-image
> 	Ubuntu 10.04 64-bit 	User:Cbrake
> beagleboard 	angstrom-2008.1 	console-image 	Ubuntu 10.04 64-bit 	User:Cbrake 	

To be clear, a line with an entry in the 'last tested' column indicates a
successful build of that combination?

> Next Monday, we will merge testing-testing to the testing branch, and
> tag it with the testing_2010-08-23 tag, and note the combinations that
> build in the above wiki page.
>
> We are looking for people who are willing to test clean builds of
> other combinations on a regular basis, or other ideas on how to make
> this "testing" concept work.  My initial thought is to work on a
> weekly cycle, but of course not every combination will be tested every
> week.
>
> If this gains enough traction, perhaps we will want to make the
> testing branch the default OE meta-data branch so that new users by
> default get something that will probably build.  I think its also very
> useful to give users a known starting point.  With the myriad of
> combinations available, its useful to know what is a good baseline.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: testing branch 2010-08-23

[Cliff Brake]

On Thu, Aug 26, 2010 at 3:50 PM, Gary Thomas <gary@mlbassoc.com> wrote:
> On 08/23/2010 05:49 PM, Cliff Brake wrote:
>>
>> Hello,
>>
>> As detailed in the following page, we are trying to get a testing
>> branch going where we try to verify OE builds for a number of
>> combinations of distro/machine/target/
>>
>> http://wiki.openembedded.net/index.php/Testing
>>
>> I have branched dev.oe.org to the testing-testing branch, and have
>> verified it builds the following.
>>
>> beagleboard     angstrom-2008.1
>> beagleboard-linuxtag2010-demo-image
>>        Ubuntu 10.04 64-bit     User:Cbrake
>> beagleboard     angstrom-2008.1         console-image   Ubuntu 10.04
>> 64-bit     User:Cbrake
>
> To be clear, a line with an entry in the 'last tested' column indicates a
> successful build of that combination?

Yes, that is my current idea.  I changed the name of the column to
"last successful build" to be a little more clear.

Thanks,
Cliff

-- 
=================
http://bec-systems.com

Re: testing branch 2010-08-23

[Gary Thomas]

On 08/26/2010 02:10 PM, Cliff Brake wrote:
> On Thu, Aug 26, 2010 at 3:50 PM, Gary Thomas<gary@mlbassoc.com>  wrote:
>> On 08/23/2010 05:49 PM, Cliff Brake wrote:
>>>
>>> Hello,
>>>
>>> As detailed in the following page, we are trying to get a testing
>>> branch going where we try to verify OE builds for a number of
>>> combinations of distro/machine/target/
>>>
>>> http://wiki.openembedded.net/index.php/Testing
>>>
>>> I have branched dev.oe.org to the testing-testing branch, and have
>>> verified it builds the following.
>>>
>>> beagleboard     angstrom-2008.1
>>> beagleboard-linuxtag2010-demo-image
>>>         Ubuntu 10.04 64-bit     User:Cbrake
>>> beagleboard     angstrom-2008.1         console-image   Ubuntu 10.04
>>> 64-bit     User:Cbrake
>>
>> To be clear, a line with an entry in the 'last tested' column indicates a
>> successful build of that combination?
>
> Yes, that is my current idea.  I changed the name of the column to
> "last successful build" to be a little more clear.

OK, I've added a couple of builds I'm trying.  It's interesting to
note that all the other builds are using 64-bit hosts and I've had
0% luck with that myself :-(

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: testing branch 2010-08-23

[Graham Gower]

"""if a build fails, then issues are reported, and the testing-testing
branch is moved ahead to capture fixes made in the dev branch. Once
the issue is fixed, all combinations are retested."""

Can I suggest that the testing-testing branch not be moved ahead? Its
not really fair on testers to have to retest, especially if the branch
is moved ahead a couple of times. Obviously breakages should be fixed
and committed, but can't it just wait for the following week's testing
branch to show up as a successful build?

-Graham

Re: testing branch 2010-08-23

[Khem Raj]

On Thu, Aug 26, 2010 at 4:20 PM, Graham Gower <graham.gower@gmail.com> wrote:
> """if a build fails, then issues are reported, and the testing-testing
> branch is moved ahead to capture fixes made in the dev branch. Once
> the issue is fixed, all combinations are retested."""
>
> Can I suggest that the testing-testing branch not be moved ahead?


I think yes the fixes should be committed to master and then pulled
into the testing-testing branch
IOW testing-testing moved ahead to new head on each monday and I think
if you have successful build
on prior monday and dont want to test it the tag should still be
there. I think continuous build is the only
way we can make sure that what we are committing is not breaking
stuff. ideally testing branch should move
every time a fix is committed upstream which was reported in
testing-testing branch but Monday to Monday is
ok too.

 Its
> not really fair on testers to have to retest, especially if the branch
> is moved ahead a couple of times. Obviously breakages should be fixed
> and committed, but can't it just wait for the following week's testing
> branch to show up as a successful build?
>
> -Graham
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: testing branch 2010-08-23

[Graham Gower]

On 27 August 2010 13:15, Khem Raj <raj.khem@gmail.com> wrote:
> On Thu, Aug 26, 2010 at 4:20 PM, Graham Gower <graham.gower@gmail.com> wrote:
>> """if a build fails, then issues are reported, and the testing-testing
>> branch is moved ahead to capture fixes made in the dev branch. Once
>> the issue is fixed, all combinations are retested."""
>>
>> Can I suggest that the testing-testing branch not be moved ahead?
>
>
> I think yes the fixes should be committed to master and then pulled
> into the testing-testing branch
> IOW testing-testing moved ahead to new head on each monday and I think
> if you have successful build
> on prior monday and dont want to test it the tag should still be
> there. I think continuous build is the only
> way we can make sure that what we are committing is not breaking
> stuff. ideally testing branch should move
> every time a fix is committed upstream which was reported in
> testing-testing branch but Monday to Monday is
> ok too.

I think maybe I was just confused about how it will work in practise.
I'll just wait and watch it for a couple of weeks :)

-Graham

Re: testing branch 2010-08-23

[Frans Meulenbroeks]

I'll add myself in a week or so. Need to see how I can get this organized.

Wrt the wiki page
I'd day remove the last succesful build column (or add a new one).
The new column should mention the last build. In yet another column we
can then add the reason.

E.g. currently khem has a ppc qemu thing where it says " 	fails to
build sed ". Next tuesday it will be unclear if this is from last
monday's tree or this monday's tree.

Also i would suggest that everyone who participates builds with
tinderbox enabled, and (at least in case of failure) adds a link to
the tinderbox log.

And one more question:
The wiki pages says: "All volunteers start clean builds for the
combinations they test,"
I have some platforms that are quite identical (eg sheevaplug and
openrd; both marvell/kirkwood based). I have no problems testing for
both, but as they are both armv5 and use the same soc, compiler etc, I
kinda feel this as a waste of time. I think I would prefer it to do
e.g. a bake for sheeva, followed by one for openrd (without a clean
inbetween) to verify that the openrd specifics (probably only u-boot
and kernel) work.
I can imagine for the various omap3530 variants (beagle, overo, ...)
this is also the case.

(thinking of it, if cpu cycles/time an issue we could decide to pull
some packages (e.g. some of the native ones) from pstage)

Frans

Re: testing branch 2010-08-23

[Cliff Brake]

On Fri, Aug 27, 2010 at 2:50 AM, Frans Meulenbroeks
<fransmeulenbroeks@gmail.com> wrote:

> Wrt the wiki page
> I'd day remove the last succesful build column (or add a new one).
> The new column should mention the last build. In yet another column we
> can then add the reason.

> E.g. currently khem has a ppc qemu thing where it says "        fails to
> build sed ". Next tuesday it will be unclear if this is from last
> monday's tree or this monday's tree.

Done.

> Also i would suggest that everyone who participates builds with
> tinderbox enabled, and (at least in case of failure) adds a link to
> the tinderbox log.

Good point -- I added that suggestion to the wiki page.

> And one more question:
> The wiki pages says: "All volunteers start clean builds for the
> combinations they test,"
> I have some platforms that are quite identical (eg sheevaplug and
> openrd; both marvell/kirkwood based). I have no problems testing for
> both, but as they are both armv5 and use the same soc, compiler etc, I
> kinda feel this as a waste of time. I think I would prefer it to do
> e.g. a bake for sheeva, followed by one for openrd (without a clean
> inbetween) to verify that the openrd specifics (probably only u-boot
> and kernel) work.
> I can imagine for the various omap3530 variants (beagle, overo, ...)
> this is also the case.
>
> (thinking of it, if cpu cycles/time an issue we could decide to pull
> some packages (e.g. some of the native ones) from pstage)

Agreed, I clarified the wiki page to say:

"All volunteers start a clean build and build the combinations they
test, update the above chart, and report status/issues as replies to
the above email."

So the clean part is singular in that we want people to start clean,
and then build as many targets as they test.

Thanks,
Cliff

-- 
=================
http://bec-systems.com

Re: testing branch 2010-08-23

[Cliff Brake]

On Thu, Aug 26, 2010 at 7:20 PM, Graham Gower <graham.gower@gmail.com> wrote:
> """if a build fails, then issues are reported, and the testing-testing
> branch is moved ahead to capture fixes made in the dev branch. Once
> the issue is fixed, all combinations are retested."""
>
> Can I suggest that the testing-testing branch not be moved ahead? Its
> not really fair on testers to have to retest, especially if the branch
> is moved ahead a couple of times. Obviously breakages should be fixed
> and committed, but can't it just wait for the following week's testing
> branch to show up as a successful build?

I think this is a good suggestion -- testers including myself may not
have time or CPU bandwidth to be running multiple builds every week.
My original thought was that the testing branch does not get moved
until everything builds, but that is probably not reasonable.  If we
list last good build for each combination, and move the testing branch
when most of the combinations build, that is probably the best we can
do.

I have updated http://wiki.openembedded.net/index.php/Testing to
reflect these thoughts.

Thanks,
Cliff

-- 
=================
http://bec-systems.com

Re: testing branch 2010-08-23

[Steffen Sledz]

Am 24.08.2010 01:49, schrieb Cliff Brake:
> As detailed in the following page, we are trying to get a testing
> branch going where we try to verify OE builds for a number of
> combinations of distro/machine/target/
> 
> http://wiki.openembedded.net/index.php/Testing
> ...

Conforming to the testing procedure i'd expected a message in this list which announces a new testing cycle yesterday. Was a new testing branch created?

Which branch is the right one? I see testing-testing and testing-next.

Steffen