[Buildroot] Call for autobuild fixing for 2013.11

[Buildroot] Call for autobuild fixing for 2013.11

[Thomas De Schampheleire]

All,

Now that 2013.11-rc1 is out, we have until the end of the month to fix
remaining problems until 2013.11 is released.

The autobuilders are still finding a lot of problems.
Several developers are already looking at and fixing these problems,
which is great. However, we can do much better.

Therefore, I would like to request all developers to temporarily stop
creating new features, adding new packages, etc. during the rest of
this month, and focus on fixing autobuild problems (or other fixes)
instead. With a joint effort I am sure we can fix many problems.

The autobuild website is at http://autobuild.buildroot.org/, the
statistics are at http://autobuild.buildroot.org/stats.php
[currently this site seems to be down]
There is also a daily mail sent to the list with the overview of the
succeeding/failing builds.

To fix a problem, you would start by downloading the defconfig from
the autobuild site, apply it in a fresh repository, probably set a
good value for BR2_DL_DIR, and run 'make' to see if you can reproduce
the problem. Once you can reproduce it, you can analyze in more detail
and create a fix.

Depending on the problem, you can attempt some shortcuts. For example
instead of building everything, you can request to build one
particular package (foo) by running 'make toolchain foo'. If the
problem is caused only by the overall configuration and foo itself
(not by a missing dependency) this faster build would also show the
problem. Remember: it's crucial that you can first reproduce the
problem (with or without shortcut). Otherwise you may 'fix' something
that is not broken.

If you want to fix a particular problem but get stuck, don't hesitate
in sending your findings to the list. Other developers can try to help
you.

Thanks a lot,

Thomas

[Buildroot] autobuild.b.o down

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Thu, 14 Nov 2013 12:24:17 +0100, Thomas De Schampheleire wrote:

> The autobuild website is at http://autobuild.buildroot.org/, the
> statistics are at http://autobuild.buildroot.org/stats.php
> [currently this site seems to be down]

The unfortunate thing is obviously that the site is down. For some
unknown reason, the virtual machine that hosts autobuild.b.o is no
longer responding since yesterday evening. I don't have the control
over the host machine, and I've already mailed the persons that do have
control about this problem. At this point, I cannot give an estimate
for the time at which the service will be back up.

However, it is worth mentioning that I am planning on moving this
service from this VM on which I don't have much control over to a
dedicated server that I own completely, so that such problems should
hopefully be handled in a more timely fashion in the future.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Call for autobuild fixing for 2013.11

[Thomas De Schampheleire]

All,

On Thu, Nov 14, 2013 at 12:24 PM, Thomas De Schampheleire
<patrickdepinguin@gmail.com> wrote:
> All,
>
> Now that 2013.11-rc1 is out, we have until the end of the month to fix
> remaining problems until 2013.11 is released.
>
> The autobuilders are still finding a lot of problems.
> Several developers are already looking at and fixing these problems,
> which is great. However, we can do much better.
>
> Therefore, I would like to request all developers to temporarily stop
> creating new features, adding new packages, etc. during the rest of
> this month, and focus on fixing autobuild problems (or other fixes)
> instead. With a joint effort I am sure we can fix many problems.
>
> The autobuild website is at http://autobuild.buildroot.org/, the
> statistics are at http://autobuild.buildroot.org/stats.php
> [currently this site seems to be down]
> There is also a daily mail sent to the list with the overview of the
> succeeding/failing builds.
>
> To fix a problem, you would start by downloading the defconfig from
> the autobuild site, apply it in a fresh repository, probably set a
> good value for BR2_DL_DIR, and run 'make' to see if you can reproduce
> the problem. Once you can reproduce it, you can analyze in more detail
> and create a fix.
>
> Depending on the problem, you can attempt some shortcuts. For example
> instead of building everything, you can request to build one
> particular package (foo) by running 'make toolchain foo'. If the
> problem is caused only by the overall configuration and foo itself
> (not by a missing dependency) this faster build would also show the
> problem. Remember: it's crucial that you can first reproduce the
> problem (with or without shortcut). Otherwise you may 'fix' something
> that is not broken.
>
> If you want to fix a particular problem but get stuck, don't hesitate
> in sending your findings to the list. Other developers can try to help
> you.

The autobuilders site is back online, for the time being at the
following location:
http://autobuild.humanoidz.org

Therefore, I would like to reiterate my request for looking at and
fixing these failures, to make 2013.11 more stable.

Thanks a lot for your contribution,
Thomas

[Buildroot] Call for autobuild fixing for 2013.11

[Thomas De Schampheleire]

All,

On Thu, Nov 14, 2013 at 12:24 PM, Thomas De Schampheleire
<patrickdepinguin@gmail.com> wrote:
> All,
>
> Now that 2013.11-rc1 is out, we have until the end of the month to fix
> remaining problems until 2013.11 is released.
>
> The autobuilders are still finding a lot of problems.
> Several developers are already looking at and fixing these problems,
> which is great. However, we can do much better.
>
> Therefore, I would like to request all developers to temporarily stop
> creating new features, adding new packages, etc. during the rest of
> this month, and focus on fixing autobuild problems (or other fixes)
> instead. With a joint effort I am sure we can fix many problems.
>
> The autobuild website is at http://autobuild.buildroot.org/, the
> statistics are at http://autobuild.buildroot.org/stats.php
> [currently this site seems to be down]
> There is also a daily mail sent to the list with the overview of the
> succeeding/failing builds.
>
> To fix a problem, you would start by downloading the defconfig from
> the autobuild site, apply it in a fresh repository, probably set a
> good value for BR2_DL_DIR, and run 'make' to see if you can reproduce
> the problem. Once you can reproduce it, you can analyze in more detail
> and create a fix.
>
> Depending on the problem, you can attempt some shortcuts. For example
> instead of building everything, you can request to build one
> particular package (foo) by running 'make toolchain foo'. If the
> problem is caused only by the overall configuration and foo itself
> (not by a missing dependency) this faster build would also show the
> problem. Remember: it's crucial that you can first reproduce the
> problem (with or without shortcut). Otherwise you may 'fix' something
> that is not broken.
>
> If you want to fix a particular problem but get stuck, don't hesitate
> in sending your findings to the list. Other developers can try to help
> you.

It's great to see several people sending patches that fix autobuild failures!
Let's keep up this work until the end of the month, when 2013.11 is released...

Thanks,
Thomas

[Buildroot] Call for autobuild fixing for 2013.11

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Sat, 23 Nov 2013 13:56:28 +0100, Thomas De Schampheleire wrote:

> > If you want to fix a particular problem but get stuck, don't
> > hesitate in sending your findings to the list. Other developers can
> > try to help you.
> 
> It's great to see several people sending patches that fix autobuild
> failures! Let's keep up this work until the end of the month, when
> 2013.11 is released...

Yes, we've made really good progress here, and it's indeed great to see
that more people have decided to the play the "let's fix autobuilder
failures" game :-)

However, I would not say that this effort should stop at the end of the
month, when 2013.11 is released. When 2013.11 is released, next will be
merged in master, and we'll have a nice set of new autobuilder failures
to look at, not to mention the existing ones that haven't been fixed
yet!

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Call for autobuild fixing for 2013.11

[Thomas De Schampheleire]

Hi Thomas, all,

On Tue, Nov 26, 2013 at 2:26 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Thomas De Schampheleire,
>
> On Sat, 23 Nov 2013 13:56:28 +0100, Thomas De Schampheleire wrote:
>
>> > If you want to fix a particular problem but get stuck, don't
>> > hesitate in sending your findings to the list. Other developers can
>> > try to help you.
>>
>> It's great to see several people sending patches that fix autobuild
>> failures! Let's keep up this work until the end of the month, when
>> 2013.11 is released...
>
> Yes, we've made really good progress here, and it's indeed great to see
> that more people have decided to the play the "let's fix autobuilder
> failures" game :-)
>
> However, I would not say that this effort should stop at the end of the
> month, when 2013.11 is released. When 2013.11 is released, next will be
> merged in master, and we'll have a nice set of new autobuilder failures
> to look at, not to mention the existing ones that haven't been fixed
> yet!

Yes, obviously I did not mean to stop fixing autobuild issues when
2013.11 is released.
In fact, the amount of remaining autobuild issues at the release of
2013.11 was relatively small that it is imaginable that we could close
them all... It would be great if we could reach 0 failures at some
point for a few days. Then any new failure could be attributed to a
recent commit, and should be fixed at once (or the commit reverted).

Best regards,
Thomas

[Buildroot] Call for autobuild fixing for 2013.11

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Mon, 2 Dec 2013 12:11:02 +0100, Thomas De Schampheleire wrote:

> Yes, obviously I did not mean to stop fixing autobuild issues when
> 2013.11 is released.
> In fact, the amount of remaining autobuild issues at the release of
> 2013.11 was relatively small that it is imaginable that we could close
> them all... It would be great if we could reach 0 failures at some
> point for a few days. Then any new failure could be attributed to a
> recent commit, and should be fixed at once (or the commit reverted).

Yes, I must say I was impressed by how much we managed to reduce the
number of build failures.

However, it's hard to be 100% sure a failure showing is a new failure,
even if we had several days in a row with 0 failures. The number of
possible combinations is so huge that I don't think it's possible to
test all of them within a reasonable amount of time. So a "new failure"
may just be something that did exist since quite some time was that we
couldn't trigger (like was indeed by another failure, for example).

Also, another effect of having more success in the builds is that we do
less builds (statistically, successful builds take more time than
failed builds, since all successful builds run completely to the end).
Therefore, the number of builds we do each day is now below 100, which
is not enough. I think we should look at adding more autobuilders in
the future. I'll try to clean up my script so that other people can set
up autobuilders.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Call for autobuild fixing for 2013.11

[Thomas De Schampheleire]

Hi Thomas,

On Mon, Dec 2, 2013 at 1:32 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Thomas De Schampheleire,
>
> On Mon, 2 Dec 2013 12:11:02 +0100, Thomas De Schampheleire wrote:
>
>> Yes, obviously I did not mean to stop fixing autobuild issues when
>> 2013.11 is released.
>> In fact, the amount of remaining autobuild issues at the release of
>> 2013.11 was relatively small that it is imaginable that we could close
>> them all... It would be great if we could reach 0 failures at some
>> point for a few days. Then any new failure could be attributed to a
>> recent commit, and should be fixed at once (or the commit reverted).
>
> Yes, I must say I was impressed by how much we managed to reduce the
> number of build failures.

A wild idea I have is to keep count of how many autobuild problems
were fixed by each contributor, and add this ranking to the buildroot
release e-mail, similar to how commits are counted. This would give
some extra incentive to developers in fixing such problems.

>
> However, it's hard to be 100% sure a failure showing is a new failure,
> even if we had several days in a row with 0 failures. The number of
> possible combinations is so huge that I don't think it's possible to
> test all of them within a reasonable amount of time. So a "new failure"
> may just be something that did exist since quite some time was that we
> couldn't trigger (like was indeed by another failure, for example).

True. But, imagine we had zero failures for a few days, and then there
is a new failure. The logical thing to do is check which package
fails, in which way, and have a look at the recent commits. In many
cases, it would be obvious whether the new failure is or could be
caused by such a recent commit, or if it is an 'old' issue that wasn't
seen for a while.
So, while we cannot automatically blame the last set of commits,
having close-to-zero failures would certainly help a lot in the
analysis.

>
> Also, another effect of having more success in the builds is that we do
> less builds (statistically, successful builds take more time than
> failed builds, since all successful builds run completely to the end).
> Therefore, the number of builds we do each day is now below 100, which
> is not enough. I think we should look at adding more autobuilders in
> the future. I'll try to clean up my script so that other people can set
> up autobuilders.

That would be very useful, indeed. Thanks in advance...

Best regards,
Thomas

[Buildroot] Call for autobuild fixing for 2013.11

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Mon, 2 Dec 2013 13:53:53 +0100, Thomas De Schampheleire wrote:

> > Yes, I must say I was impressed by how much we managed to reduce the
> > number of build failures.
> 
> A wild idea I have is to keep count of how many autobuild problems
> were fixed by each contributor, and add this ranking to the buildroot
> release e-mail, similar to how commits are counted. This would give
> some extra incentive to developers in fixing such problems.

In practice, there is usually one commit to fix each autobuilder issue,
so people already get credited by the number of commits.

However, if you want, we can run some statistics based on which commit
logs contained a reference to an autobuild URL, and extract these.
Should not be too complicated to achieve.

... a little bit of scripting later ...

for i in $(git log --format=%H 2013.08..2013.11); do git show --quiet $i | grep -q http://autobuild && git show --quiet --format="%an" $i ; done | sort | uniq -c | sort -rn -k1

gives the following scores for the last release:

     39 Gustavo Zacarias
     21 Peter Korsgaard
     17 Thomas Petazzoni
     14 Simon Dawson
      6 Thomas De Schampheleire
      6 Romain Naour
      5 Vicente Olivert Riera
      5 Samuel Martin
      4 Fatih A??c?
      4 Arnout Vandecappelle
      3 Ryan Barnett
      3 Axel Lin
      2 Phil Eichinger
      2 Matt Weber
      2 Chris Zankel
      1 Michael Rommel
      1 Luca Ceresoli
      1 Jerzy Grzegorek
      1 Clayton Shotwell
      1 Baruch Siach
      1 Alexander Lukichev

Of course, this assumes that any commit that contains a reference to a
http://autobuild<something> URL is fixing an autobuilder failure. Which
I guess is good enough indicator (even though possibly not perfect).

> > However, it's hard to be 100% sure a failure showing is a new failure,
> > even if we had several days in a row with 0 failures. The number of
> > possible combinations is so huge that I don't think it's possible to
> > test all of them within a reasonable amount of time. So a "new failure"
> > may just be something that did exist since quite some time was that we
> > couldn't trigger (like was indeed by another failure, for example).
> 
> True. But, imagine we had zero failures for a few days, and then there
> is a new failure. The logical thing to do is check which package
> fails, in which way, and have a look at the recent commits. In many
> cases, it would be obvious whether the new failure is or could be
> caused by such a recent commit, or if it is an 'old' issue that wasn't
> seen for a while.
> So, while we cannot automatically blame the last set of commits,
> having close-to-zero failures would certainly help a lot in the
> analysis.

True. But in practice, that's something we're already able to do. As I
follow autobuilder failures quite regularly, I'm quite easily able to
determine whether a build failure is something new that is possibly
related to a recent commit, or something that has been around for quite
some time.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Call for autobuild fixing for 2013.11

[Thomas De Schampheleire]

On Mon, Dec 2, 2013 at 2:10 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Thomas De Schampheleire,
>
> On Mon, 2 Dec 2013 13:53:53 +0100, Thomas De Schampheleire wrote:
>
>> > Yes, I must say I was impressed by how much we managed to reduce the
>> > number of build failures.
>>
>> A wild idea I have is to keep count of how many autobuild problems
>> were fixed by each contributor, and add this ranking to the buildroot
>> release e-mail, similar to how commits are counted. This would give
>> some extra incentive to developers in fixing such problems.
>
> In practice, there is usually one commit to fix each autobuilder issue,
> so people already get credited by the number of commits.
>
> However, if you want, we can run some statistics based on which commit
> logs contained a reference to an autobuild URL, and extract these.
> Should not be too complicated to achieve.
>
> ... a little bit of scripting later ...
>
> for i in $(git log --format=%H 2013.08..2013.11); do git show --quiet $i | grep -q http://autobuild && git show --quiet --format="%an" $i ; done | sort | uniq -c | sort -rn -k1
>
> gives the following scores for the last release:
>
>      39 Gustavo Zacarias
>      21 Peter Korsgaard
>      17 Thomas Petazzoni
>      14 Simon Dawson
>       6 Thomas De Schampheleire
>       6 Romain Naour
>       5 Vicente Olivert Riera
>       5 Samuel Martin
>       4 Fatih A??c?
>       4 Arnout Vandecappelle
>       3 Ryan Barnett
>       3 Axel Lin
>       2 Phil Eichinger
>       2 Matt Weber
>       2 Chris Zankel
>       1 Michael Rommel
>       1 Luca Ceresoli
>       1 Jerzy Grzegorek
>       1 Clayton Shotwell
>       1 Baruch Siach
>       1 Alexander Lukichev
>
> Of course, this assumes that any commit that contains a reference to a
> http://autobuild<something> URL is fixing an autobuilder failure. Which
> I guess is good enough indicator (even though possibly not perfect).

Nice!
The results for previous releases do indeed show an increased interest
in fixing autobuild failures this release (results below). While the
number of contributors fixing autobuild problems (and marking it as
such) ranged between 9 and 15 since 2012.08, this release there were
no less than 21 unique contributors!
I assume this 'call for autobuild fixing' has had some part in it, and
therefore I certainly think we should repeat this 'extra attention
mail' next release (February).

2012.08...2012.11
     45 Thomas Petazzoni
     27 Peter Korsgaard
      8 Gustavo Zacarias
      7 Simon Dawson
      5 Arnout Vandecappelle
      2 Maxime Ripard
      2 Markos Chandras
      1 Danomi Manchego
      1 Baruch Siach

2012.11...2013.02
     45 Gustavo Zacarias
     38 Peter Korsgaard
     32 Thomas Petazzoni
      8 Maxime Ripard
      6 Yann E. MORIN
      3 gilles.talis at gmail.com
      2 Simon Dawson
      1 Stephan Hoffmann
      1 Samuel Martin
      1 Markos Chandras
      1 Luca Ceresoli
      1 Gilles Talis
      1 Carsten Schoenert
      1 Baruch Siach
      1 Arnout Vandecappelle (Essensium/Mind)

2013.02...2013.05
     47 Gustavo Zacarias
     32 Thomas Petazzoni
     14 gilles.talis at gmail.com
     13 Peter Korsgaard
      3 Samuel Martin
      2 Simon Dawson
      2 Baruch Siach
      2 Arnout Vandecappelle (Essensium/Mind)
      1 Yann E. MORIN
      1 Thomas De Schampheleire
      1 Olivier Schonken
      1 Markos Chandras
      1 Carsten Schoenert

2013.05...2013.08
     39 Gustavo Zacarias
     16 Thomas Petazzoni
     13 Peter Korsgaard
      4 gilles.talis at gmail.com
      3 Thomas De Schampheleire
      2 Simon Dawson
      2 Samuel Martin
      2 Markos Chandras
      2 J?r?me Pouiller
      1 Yann E. MORIN
      1 Spenser Gilliland
      1 Gilles Talis

2013.08...2013.11
     39 Gustavo Zacarias
     21 Peter Korsgaard
     17 Thomas Petazzoni
     14 Simon Dawson
      6 Thomas De Schampheleire
      6 Romain Naour
      5 Vicente Olivert Riera
      5 Samuel Martin
      4 Fatih A??c?
      4 Arnout Vandecappelle
      3 Ryan Barnett
      3 Axel Lin
      2 Phil Eichinger
      2 Matt Weber
      2 Chris Zankel
      1 Michael Rommel
      1 Luca Ceresoli
      1 Jerzy Grzegorek
      1 Clayton Shotwell
      1 Baruch Siach
      1 Alexander Lukichev


>
>> > However, it's hard to be 100% sure a failure showing is a new failure,
>> > even if we had several days in a row with 0 failures. The number of
>> > possible combinations is so huge that I don't think it's possible to
>> > test all of them within a reasonable amount of time. So a "new failure"
>> > may just be something that did exist since quite some time was that we
>> > couldn't trigger (like was indeed by another failure, for example).
>>
>> True. But, imagine we had zero failures for a few days, and then there
>> is a new failure. The logical thing to do is check which package
>> fails, in which way, and have a look at the recent commits. In many
>> cases, it would be obvious whether the new failure is or could be
>> caused by such a recent commit, or if it is an 'old' issue that wasn't
>> seen for a while.
>> So, while we cannot automatically blame the last set of commits,
>> having close-to-zero failures would certainly help a lot in the
>> analysis.
>
> True. But in practice, that's something we're already able to do. As I
> follow autobuilder failures quite regularly, I'm quite easily able to
> determine whether a build failure is something new that is possibly
> related to a recent commit, or something that has been around for quite
> some time.

This works for you because, as you say, you follow everything very
well. But a 'normal contributor' (no offence :-) ) that does not look
at every commit and every autobuild failure will have a harder time in
making that analysis.

Best regards,
Thomas

[Buildroot] Call for autobuild fixing for 2013.11

[Ryan Barnett]

Thomas(es), All,

Thomas De Schampheleire <patrickdepinguin@gmail.com> wrote on 12/02/2013 
08:15:17 AM:

> On Mon, Dec 2, 2013 at 2:10 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:

[...]

> Nice!
> The results for previous releases do indeed show an increased interest
> in fixing autobuild failures this release (results below). While the
> number of contributors fixing autobuild problems (and marking it as
> such) ranged between 9 and 15 since 2012.08, this release there were
> no less than 21 unique contributors!
> I assume this 'call for autobuild fixing' has had some part in it, and
> therefore I certainly think we should repeat this 'extra attention
> mail' next release (February).

I also believe it analyzes provided by Thomas P, Thomas D and others 
helped contribute to fixing the problems. By providing the relevant logs 
and point people in the right direction of where the problem may lie 
allowed other, less experience developers like myself try to help quickly 
fix autobuilder errors.

[...]

Thanks,
-Ryan