why we don't setup a buildbot for openembedded QA?

why we don't setup a buildbot for openembedded QA?

[Guo Hongruan]

Hi guys,
   I think we had better set up a continuous integration tool buildbot to  
automate the compile/test cycle required to validate changes to the  
project code base.

It can help openembedded user to save time on failed building.

Without buildbot, any volunteers can provide their available machine as  
buildslave and the continuous integration environment will be really  
scalable.

Thanks a lot.

-- 
Guo Hongruan, Embedded Linux Consultant
Skype: camelguo
Twitter: camelguo
http://www.gulessoft.com

Re: why we don't setup a buildbot for openembedded QA?

[Holger Hans Peter Freyther]

On Monday 04 January 2010 02:42:42 Guo Hongruan wrote:
> Hi guys,
>    I think we had better set up a continuous integration tool buildbot to
> automate the compile/test cycle required to validate changes to the
> project code base.
> 
> It can help openembedded user to save time on failed building.

You make one assumption. You assume that the user config is the same as the 
build slaves (this includes different host distros, host machine, target 
machine, target distro and packages to be built).


> Without buildbot, any volunteers can provide their available machine as
> buildslave and the continuous integration environment will be really
> scalable.

This is not different from the tinderbox setup. On top of the buildbot we do 
get much more information about the build, including the log files of each and 
every task.


The basic problem is there are so many different possible configurations that 
they can not be tested with each commit. What can be done and is done is that 
the paths that are walked frequently will work better than the paths that are 
not walked as frequently. E.g. creating a new distribution will be a lot 
harder than building Angstrom for the Beagleboard.

Now what one should do if one is interested in a specific path is to user the 
tinderclient and regularily build, test (and fix) the path one is interested 
in. For me this includes the meta-toolchain-qte target.


The other is the tinderbox setup will soon gain a waterfall view and then it 
becomes a s social problem to fix the (selected) builds whenever they break.


regards
	holger

Re: why we don't setup a buildbot for openembedded QA?

[Guo Hongruan]

在 Mon, 04 Jan 2010 10:57:33 +0800，Holger Hans Peter Freyther  
<holger+oe@freyther.de> 写道:

> On Monday 04 January 2010 02:42:42 Guo Hongruan wrote:
>> Hi guys,
>>    I think we had better set up a continuous integration tool buildbot  
>> to
>> automate the compile/test cycle required to validate changes to the
>> project code base.
>>
>> It can help openembedded user to save time on failed building.
>
> You make one assumption. You assume that the user config is the same as  
> the
> build slaves (this includes different host distros, host machine, target
> machine, target distro and packages to be built).

Yes, you are right. it is impossible to cover every combinations.
But I think it is better that nothing. We can classify the varioubles  
which affect the building and make a test plan. Of course, we can not make  
it perfect at the first time. But we can make it better and better.

>
>
>> Without buildbot, any volunteers can provide their available machine as
>> buildslave and the continuous integration environment will be really
>> scalable.
>
> This is not different from the tinderbox setup. On top of the buildbot  
> we do
> get much more information about the build, including the log files of  
> each and
> every task.

With buildbot, we can still use tinderbox to record every task and their  
log files. buildbot and tinderbot can work together smoothly.

>
>
> The basic problem is there are so many different possible configurations  
> that
> they can not be tested with each commit. What can be done and is done is  
> that
> the paths that are walked frequently will work better than the paths  
> that are
> not walked as frequently. E.g. creating a new distribution will be a lot
> harder than building Angstrom for the Beagleboard.
>

If there are enough buildslavers, we can test most of different possible  
configurations with each commit. After all, it is distributed and can  
scale dynamically.

> Now what one should do if one is interested in a specific path is to  
> user the
> tinderclient and regularily build, test (and fix) the path one is  
> interested
> in. For me this includes the meta-toolchain-qte target.
>
>
> The other is the tinderbox setup will soon gain a waterfall view and  
> then it
> becomes a s social problem to fix the (selected) builds whenever they  
> break.
>

builtbot can also show a waterfail view and through periodic building, we  
can find the bug as early as possible. everyone can access the buildbot  
website to get to know which building failed and to get to know which how  
to reproduce the failed building. Throught tinderbox, it is difficutl to  
do that, for some developers build openembedded at their own branches, and  
tinderbox can not record the complete local modification and setting.

>
> regards
> 	holger
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel


-- 
Guo Hongruan, Embedded Linux Consultant
Skype: camelguo
Twitter: camelguo
http://www.gulessoft.com

Re: why we don't setup a buildbot for openembedded QA?

[Mike Westerhof]

Guo Hongruan wrote:
> 在 Mon, 04 Jan 2010 10:57:33 +0800，Holger Hans Peter Freyther
> <holger+oe@freyther.de> 写道:
> 
>> On Monday 04 January 2010 02:42:42 Guo Hongruan wrote:
>>> Hi guys,
>>>    I think we had better set up a continuous integration tool
>>> buildbot to
>>> automate the compile/test cycle required to validate changes to the
>>> project code base.
>>>
>>> It can help openembedded user to save time on failed building.
>>
>> You make one assumption. You assume that the user config is the same
>> as the
>> build slaves (this includes different host distros, host machine, target
>> machine, target distro and packages to be built).
> 
> Yes, you are right. it is impossible to cover every combinations.
> But I think it is better that nothing. We can classify the varioubles
> which affect the building and make a test plan. Of course, we can not
> make it perfect at the first time. But we can make it better and better.

To what end, though?

I suggest that there are only a small set of configurations that matter,
and for each of those, the maintainers should be running their own
buildbots, on hosts that they intend to support.  To test any other
combinations simply makes no sense; nobody is using them.

>>> Without buildbot, any volunteers can provide their available machine as
>>> buildslave and the continuous integration environment will be really
>>> scalable.
>>
>> This is not different from the tinderbox setup. On top of the buildbot
>> we do
>> get much more information about the build, including the log files of
>> each and
>> every task.
> 
> With buildbot, we can still use tinderbox to record every task and their
> log files. buildbot and tinderbot can work together smoothly.
> 
>>
>>
>> The basic problem is there are so many different possible
>> configurations that
>> they can not be tested with each commit. What can be done and is done
>> is that
>> the paths that are walked frequently will work better than the paths
>> that are
>> not walked as frequently. E.g. creating a new distribution will be a lot
>> harder than building Angstrom for the Beagleboard.
>>
> 
> If there are enough buildslavers, we can test most of different possible
> configurations with each commit. After all, it is distributed and can
> scale dynamically.
> 
>> Now what one should do if one is interested in a specific path is to
>> user the
>> tinderclient and regularily build, test (and fix) the path one is
>> interested
>> in. For me this includes the meta-toolchain-qte target.
>>
>>
>> The other is the tinderbox setup will soon gain a waterfall view and
>> then it
>> becomes a s social problem to fix the (selected) builds whenever they
>> break.

But, as we've seen already, people don't fix builds that nobody cares
about.  Everyone has enough to do of their own -- from my perspective I
see a small group of developers active with OE day-to-day who fix
problems and bugs for a small set of distros that they know are actively
being used.  To add more broken builds to the list will only result in
the active distros being drowned out, and the result will be that each
distro owner will end up tracking and fixing issues independently again.

> builtbot can also show a waterfail view and through periodic building,
> we can find the bug as early as possible. everyone can access the
> buildbot website to get to know which building failed and to get to know
> which how to reproduce the failed building. Throught tinderbox, it is
> difficutl to do that, for some developers build openembedded at their
> own branches, and tinderbox can not record the complete local
> modification and setting.

The tools are not the problem.  Time and motivation is the issue here -
people have time and motivation only to solve issues and problems that
are causing huge pain, or where a resolution will provide much benefit.
 Neither is true for stale distros, or for combinations of build hosts,
distros, and machines that nobody uses.

The best way, clearly, is for each person who values a particular
combination to set up a buildbot for that combination.  VMs and disk are
cheap.

-Mike (mwester)

Re: why we don't setup a buildbot for openembedded QA?

[Rolf Leggewie]

Holger Hans Peter Freyther wrote:
> The other is the tinderbox setup will soon gain a waterfall view[...]

I thought I updated the necessary code about a month ago, no? I'm not
sure how the feature is supposed to be used, but isn't
http://tinderbox.openembedded.net/waterfall/ what you are talking about?

Re: why we don't setup a buildbot for openembedded QA?

[Holger Hans Peter Freyther]

On Monday 04 January 2010 16:07:55 Rolf Leggewie wrote:
> Holger Hans Peter Freyther wrote:
> > The other is the tinderbox setup will soon gain a waterfall view[...]
> 
> I thought I updated the necessary code about a month ago, no? I'm not
> sure how the feature is supposed to be used, but isn't
> http://tinderbox.openembedded.net/waterfall/ what you are talking about?

Oh My God. Thanks so much! The last time we talked you had some issues with 
the update and said it would not work properly. I went so far to setup a 
Hardy, back django and everything needed, packaged the tinderbox and was about 
to try it locally and then...


again, thanks a lot for the update!

	z.

Re: why we don't setup a buildbot for openembedded QA?

[Rolf Leggewie]

Holger Hans Peter Freyther wrote:
> Oh My God. Thanks so much! The last time we talked you had some issues with 
> the update and said it would not work properly.

I think the credit goes to Jeremy for resolving the issue upstream. I
only reported it to him and did the packaging.  I was sure I informed
you about the update but it looks like I didn't.  I'm sorry about
creating unnecessary double-work.

Re: why we don't setup a buildbot for openembedded QA?

[Guo Hongruan]

No, I am not talking about tinderbox but buildbot. I want to setup a  
public buildbot and validate every possible combinations of openembedded.  
In another words, I want to do some QA jobs of openembedded.

Of course, I need help and support from all of you. I want a public server  
which runs python and twist and I also want any volunteer can provide  
their machine as build slaver when it is available.

Thanks a lot!

在 Mon, 04 Jan 2010 23:07:55 +0800，Rolf Leggewie  
<no2spam@nospam.arcornews.de> 写道:

> Holger Hans Peter Freyther wrote:
>> The other is the tinderbox setup will soon gain a waterfall view[...]
>
> I thought I updated the necessary code about a month ago, no? I'm not
> sure how the feature is supposed to be used, but isn't
> http://tinderbox.openembedded.net/waterfall/ what you are talking about?
>
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel


-- 
Guo Hongruan, Embedded Linux Consultant
Skype: camelguo
Twitter: camelguo
http://www.gulessoft.com

Re: why we don't setup a buildbot for openembedded QA?

[Holger Hans Peter Freyther]

On Tuesday 05 January 2010 03:06:47 Guo Hongruan wrote:
> No, I am not talking about tinderbox but buildbot. I want to setup a
> public buildbot and validate every possible combinations of openembedded.
> In another words, I want to do some QA jobs of openembedded.

This is a nice goal, but it is probably not the biggest concern as well. E.g. 
there is little benefit of making every version of binutils work with glibc and 
gcc. It is certainly cool to be able to pick any GCC, BINUTILS, GLIBC and 
guarnatee that it will work on any host distro. But besides being totally cool 
it is also totally wasting your time (unless your goal is to learn how GCC, 
BINUTILS and GLIBC interact). And the next step would be to have make world 
build on top of that...


I have been at your point a couple of years back (seppuku, a tinderbox setup, 
autobuild scripts are the result). The QA that is working best is to have 
people sign up for "maintaining" certain areas.

E.g.: I'm feeling responsible for:

	- QtE, meta-toolchain-qte, qmake and such
	- I'm also testing mipsel from time to time

Other people like pb_ feel responsible for:
	- DISTRO micro (I think i got it wrong)

Or Khem is doing kick ass work on the toolchain bits..


> Of course, I need help and support from all of you. I want a public server
> which runs python and twist and I also want any volunteer can provide
> their machine as build slaver when it is available.

Technology is not the problem here. What use is this system if no one is fixing 
build problems because of a too new or too old version of GCC? What use is it 
to the user to have thousands of tested configurations when he can not navigate 
through the list of packages?


let me write another mail about QA topics.

	z.

Re: why we don't setup a buildbot for openembedded QA?

[Guo Hongruan]

在 Wed, 06 Jan 2010 11:59:45 +0800，Holger Hans Peter Freyther  
<holger+oe@freyther.de> 写道:

> On Tuesday 05 January 2010 03:06:47 Guo Hongruan wrote:
>> No, I am not talking about tinderbox but buildbot. I want to setup a
>> public buildbot and validate every possible combinations of  
>> openembedded.
>> In another words, I want to do some QA jobs of openembedded.
>
> This is a nice goal, but it is probably not the biggest concern as well.  
> E.g.
> there is little benefit of making every version of binutils work with  
> glibc and
> gcc. It is certainly cool to be able to pick any GCC, BINUTILS, GLIBC and
> guarnatee that it will work on any host distro. But besides being  
> totally cool
> it is also totally wasting your time (unless your goal is to learn how  
> GCC,
> BINUTILS and GLIBC interact). And the next step would be to have make  
> world
> build on top of that...

Yes, I agree with you. It is not worth to make sure every combinations of  
GCC, GLIBC and BINUTILS work perfectly. For example, some versions of  
binutils is too old to work with newest glibc/gcc and there are little  
projects using these combinations.

But, My idea is that we can maintain a table which records which version  
of package A can not work with which version of package B.
(for example, glibc-2.6.1 can not work with binutils-2.20)
So that, when the user chooses these combinations, the  
openembedded/bitbake can give them a warning or error to indicate that  
this combination will not work.

>
>
> I have been at your point a couple of years back (seppuku, a tinderbox  
> setup,
> autobuild scripts are the result). The QA that is working best is to have
> people sign up for "maintaining" certain areas.
>
> E.g.: I'm feeling responsible for:
>
> 	- QtE, meta-toolchain-qte, qmake and such
> 	- I'm also testing mipsel from time to time
>
> Other people like pb_ feel responsible for:
> 	- DISTRO micro (I think i got it wrong)
>
> Or Khem is doing kick ass work on the toolchain bits..
>
>
>> Of course, I need help and support from all of you. I want a public  
>> server
>> which runs python and twist and I also want any volunteer can provide
>> their machine as build slaver when it is available.
>
> Technology is not the problem here. What use is this system if no one is  
> fixing
> build problems because of a too new or too old version of GCC? What use  
> is it
> to the user to have thousands of tested configurations when he can not  
> navigate
> through the list of packages?
>
>
> let me write another mail about QA topics.
>
> 	z.
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel


-- 
Guo Hongruan, Embedded Linux Consultant
Skype: camelguo
Twitter: camelguo
http://www.gulessoft.com

Re: why we don't setup a buildbot for openembedded QA?

[Holger Hans Peter Freyther]

On Thursday 07 January 2010 03:02:05 Guo Hongruan wrote:
> 在 Wed, 06 Jan 2010 11:59:45 +0800，Holger Hans Peter Freyther

> 
> Yes, I agree with you. It is not worth to make sure every combinations of
> GCC, GLIBC and BINUTILS work perfectly. For example, some versions of
> binutils is too old to work with newest glibc/gcc and there are little
> projects using these combinations.
> 
> But, My idea is that we can maintain a table which records which version
> of package A can not work with which version of package B.
> (for example, glibc-2.6.1 can not work with binutils-2.20)
> So that, when the user chooses these combinations, the
> openembedded/bitbake can give them a warning or error to indicate that
> this combination will not work.

Well, the Angstrom, Minimal, Micro are these documentations. People sat down 
and tested the configurations and then used it. I believe it is more valuable 
to document what is working and in the mainstream than to document which 
exotic configurations do not work (think of the cross-tool picture where 
everything is red)

OpenEmbedded is a tool for power users, if they really need to use a specific 
version of glibc they should be able to deal with the errors and resolve them.


I pointed out some areas where OE is really lacking in QA and it would be cool 
if you would be interested in that as well.