[Buildroot] Availability of old build results

[Buildroot] Availability of old build results

[Thomas Petazzoni]

Hello,

Due to the autobuild.buildroot.org server crash, I had to quickly set
up a new instance, which didn't include the build results available
from the old server.

I'm happy to announce that I have now reimported the old build results
in the new server. This means that:

 1/ All previous autobuild.buildroot.org URLs of build results should
    now work again.

 2/ The graph of success vs. failure rate is a lot more useful, see
    http://autobuild.buildroot.org/stats.php.

 3/ We can be proud of having made more than 79000 build tests since we
    started this effort! :-)

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Availability of old build results

[Yann E. MORIN]

Thomas, All,

On 2013-12-06 00:25 +0100, Thomas Petazzoni spake thusly:
> Due to the autobuild.buildroot.org server crash, I had to quickly set
> up a new instance, which didn't include the build results available
> from the old server.
> 
> I'm happy to announce that I have now reimported the old build results
> in the new server. This means that:
> 
>  1/ All previous autobuild.buildroot.org URLs of build results should
>     now work again.
> 
>  2/ The graph of success vs. failure rate is a lot more useful, see
>     http://autobuild.buildroot.org/stats.php.

There is something striking about this graph: the month before releases,
we clearly see a trend that the ratio success/failure increases in
favour of sucess, and suddenly there is a dramatic reversal the moment
the release is made, ie. when next is merged back into master.

That means we're doing quite a good job at hunting the failures during
the feature-freeze: this cycle, the ratio was ~90% success / ~10% failures
at the time of release. This is pretty good. :-)

As for the sudden drop at the time next is merged, I don't know what to
think of it.

There are occasional spikes where the tendency goes toward failures, but
those are quickly fixed.

It would be nice if the X-axis was labelled with Sundays rather than
every 11 days. It looks like there is a 7-day period in the cycle of
success - failure-spikes.

Thanks for providing this data! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Availability of old build results

[Thomas De Schampheleire]

Hi,

On Fri, Dec 6, 2013 at 12:41 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> Thomas, All,
>
> On 2013-12-06 00:25 +0100, Thomas Petazzoni spake thusly:
>> Due to the autobuild.buildroot.org server crash, I had to quickly set
>> up a new instance, which didn't include the build results available
>> from the old server.
>>
>> I'm happy to announce that I have now reimported the old build results
>> in the new server. This means that:
>>
>>  1/ All previous autobuild.buildroot.org URLs of build results should
>>     now work again.
>>
>>  2/ The graph of success vs. failure rate is a lot more useful, see
>>     http://autobuild.buildroot.org/stats.php.
>
> There is something striking about this graph: the month before releases,
> we clearly see a trend that the ratio success/failure increases in
> favour of sucess, and suddenly there is a dramatic reversal the moment
> the release is made, ie. when next is merged back into master.
>
> That means we're doing quite a good job at hunting the failures during
> the feature-freeze: this cycle, the ratio was ~90% success / ~10% failures
> at the time of release. This is pretty good. :-)
>
> As for the sudden drop at the time next is merged, I don't know what to
> think of it.

I am thinking of two ways to help reduce this problem:

1. During the time -next is opened (until release) we also make test
builds on that branch (but keep the results separate from the master
results, of course).
Pros:
  - better visibility on the quality of -next, before it is being
merged, and a chance to fix the problem beforehand
Cons:
  - attention of developers is diverted away from stabilizing the
upcoming release
  - autobuild computing capacity is diverted away too


2. Similar, in a way: when patches are posted to the list (not only
during the stabilization month), run them through some autobuild
configurations 'automatically' to try catching common problems (for
example thread support, mmu support, uclibc problems, ...) and post
the results somewhere. This generates a kind of 'staging moment' for
patches before they get applied, to check their quality.
Whether this should be done for all patches (even at their initial
send) or only makes sense for patches that have been reviewed first
(to make sure the computing power is used usefully) is something that
can be discussed. In the latter case, we'd need to have a trigger to
request the test builds.
The results would ideally be added to patchwork, next to the requested
'Acked-by' indication, giving the maintainer a better idea of the
quality of a patch, before merging it.
This is very much related to the continuous integration server idea of
Jerome, except that it should be done on patches not-yet-merged into
git.
Pros:
  - quality of master (as measured by the normal autobuilders) should
go up because many problems are fixed beforehand
Cons:
  - need computing power for these extra tests, but this is not unsurmountable


Best regards,
Thomas

[Buildroot] Availability of old build results

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:

Hi,

 > 1. During the time -next is opened (until release) we also make test
 > builds on that branch (but keep the results separate from the master
 > results, of course).
 > Pros:
 >   - better visibility on the quality of -next, before it is being
 > merged, and a chance to fix the problem beforehand
 > Cons:
 >   - attention of developers is diverted away from stabilizing the
 > upcoming release
 >   - autobuild computing capacity is diverted away too

Yeah, I'm not quite sure if this is a good idea.


 > 2. Similar, in a way: when patches are posted to the list (not only
 > during the stabilization month), run them through some autobuild
 > configurations 'automatically' to try catching common problems (for
 > example thread support, mmu support, uclibc problems, ...) and post
 > the results somewhere. This generates a kind of 'staging moment' for
 > patches before they get applied, to check their quality.
 > Whether this should be done for all patches (even at their initial
 > send) or only makes sense for patches that have been reviewed first
 > (to make sure the computing power is used usefully) is something that
 > can be discussed. In the latter case, we'd need to have a trigger to
 > request the test builds.

This one I like! Seems like a nice little weekend task to
implement. I'll try to find time for it (but others are certainly
welcome to work on it as well)

-- 
Bye, Peter Korsgaard

[Buildroot] Availability of old build results

[Thomas Petazzoni]

Dear Peter Korsgaard,

On Fri, 06 Dec 2013 09:47:02 +0100, Peter Korsgaard wrote:

>  > 1. During the time -next is opened (until release) we also make test
>  > builds on that branch (but keep the results separate from the master
>  > results, of course).
>  > Pros:
>  >   - better visibility on the quality of -next, before it is being
>  > merged, and a chance to fix the problem beforehand
>  > Cons:
>  >   - attention of developers is diverted away from stabilizing the
>  > upcoming release
>  >   - autobuild computing capacity is diverted away too
> 
> Yeah, I'm not quite sure if this is a good idea.

It would however be the easiest thing to do.

>  > 2. Similar, in a way: when patches are posted to the list (not only
>  > during the stabilization month), run them through some autobuild
>  > configurations 'automatically' to try catching common problems (for
>  > example thread support, mmu support, uclibc problems, ...) and post
>  > the results somewhere. This generates a kind of 'staging moment' for
>  > patches before they get applied, to check their quality.
>  > Whether this should be done for all patches (even at their initial
>  > send) or only makes sense for patches that have been reviewed first
>  > (to make sure the computing power is used usefully) is something that
>  > can be discussed. In the latter case, we'd need to have a trigger to
>  > request the test builds.
> 
> This one I like! Seems like a nice little weekend task to
> implement. I'll try to find time for it (but others are certainly
> welcome to work on it as well)

I'm not sure to understand how this will work. Which patches will be
run through this testing? Who will decide which patches will go? You?
The patch submitter?

On my side, I'm really skeptical about that one: I think we should
rather merge patches faster, so that we simply rely on the existing
autobuilder infrastructure, which works well.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Availability of old build results

[Thomas De Schampheleire]

Hi Thomas, Peter,

On Fri, Dec 6, 2013 at 9:50 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Peter Korsgaard,
>
> On Fri, 06 Dec 2013 09:47:02 +0100, Peter Korsgaard wrote:
>
>>  > 1. During the time -next is opened (until release) we also make test
>>  > builds on that branch (but keep the results separate from the master
>>  > results, of course).
>>  > Pros:
>>  >   - better visibility on the quality of -next, before it is being
>>  > merged, and a chance to fix the problem beforehand
>>  > Cons:
>>  >   - attention of developers is diverted away from stabilizing the
>>  > upcoming release
>>  >   - autobuild computing capacity is diverted away too
>>
>> Yeah, I'm not quite sure if this is a good idea.
>
> It would however be the easiest thing to do.
>
>>  > 2. Similar, in a way: when patches are posted to the list (not only
>>  > during the stabilization month), run them through some autobuild
>>  > configurations 'automatically' to try catching common problems (for
>>  > example thread support, mmu support, uclibc problems, ...) and post
>>  > the results somewhere. This generates a kind of 'staging moment' for
>>  > patches before they get applied, to check their quality.
>>  > Whether this should be done for all patches (even at their initial
>>  > send) or only makes sense for patches that have been reviewed first
>>  > (to make sure the computing power is used usefully) is something that
>>  > can be discussed. In the latter case, we'd need to have a trigger to
>>  > request the test builds.
>>
>> This one I like! Seems like a nice little weekend task to
>> implement. I'll try to find time for it (but others are certainly
>> welcome to work on it as well)
>
> I'm not sure to understand how this will work. Which patches will be
> run through this testing? Who will decide which patches will go? You?
> The patch submitter?

This is something to be decided (if we go this route).
The simplest one is to automatically take every patch that appears in
patchwork, and run it through the test system. The disadvantage is
that you may be testing crap patches that would easily be spotted
during review, and thus you are investing the limited build capacity
in the wrong builds. This may be ok though, if we can add some extra
servers to the build capacity, and this also greatly depends on the
amount of tests that we run.

Another way is to only test 'requested' patches. One could envision a
way to request a test of a given patch, but this does not need to be
limited to one person. Either you can have a group of admins (for
example the current patchwork admins) or you can keep it open and have
a fair-use-policy that avoids requesting builds too early.


What to test: it doesn't need to be every imaginable configuration,
but it would be nice to have one or more standard builds, a blackfin
(no-mmu) build, a uclibc configuration, a full versus basic
configuration, ... We could have a set of, say, 15 combinations, and
we pick, say, 5 of them for each patch to test. For example, you have:
powerpc, buildroot basic uclibc toolchain
powerpc, buildroot basic glibc toolchain
powerpc, buildroot full uclibc toolchain
powerpc, buildroot full glibc toolchain
powerpc, external sourcery (full) toolchain
(more or less the same for the other archs)

and you pick from all this one buildroot basic toolchain (any arch),
one buildroot full toolchain, one external sourcery toolchain, one
explicit blackfin build, one explicit uclibc build (full or basic).
The rest of the configuration can be random just as in the normal
autobuilders.

The above is mostly relevant for new packages.
If the patch is fixing an autobuild problem, we really should be
repeating that autobuild configuration instead (but that could be done
manually by someone).
If the patch is adding/changing an init script, it doesn't make sense
to build it.

>
> On my side, I'm really skeptical about that one: I think we should
> rather merge patches faster, so that we simply rely on the existing
> autobuilder infrastructure, which works well.

I'm not saying we should keep patches in this test queue for a long
time. I'm also not saying that Peter is not free to apply a patch even
if it was not tested in this system.
However, we are having quite a number of failures on basic things like
thread support, mmu support, ... Ideally this would have been checked
by the submitter (and we could help this by providing a list of
reference configurations that people should test on), but as a
fallback we could also implement an automatic test system.

The idea of providing a list of reference configurations that
developers should test their new packages on may be sufficient too,
and then the more complicated test infrastructure as described above
is not needed. As reviewer, we can ask if the submitter did these
tests and trust that answer. The autobuilders can then catch the other
errors.
Providing this list of configurations is not that hard, we already
have a bunch of toolchains on the autobuilders. A script to run the
selected configurations in turn would be nice.

Best regards,
Thomas

[Buildroot] Availability of old build results

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Fri, 6 Dec 2013 10:57:42 +0100, Thomas De Schampheleire wrote:

> This is something to be decided (if we go this route).
> The simplest one is to automatically take every patch that appears in
> patchwork, and run it through the test system.

How do you know on which commit a given patch does apply?

> The disadvantage is
> that you may be testing crap patches that would easily be spotted
> during review, and thus you are investing the limited build capacity
> in the wrong builds. This may be ok though, if we can add some extra
> servers to the build capacity, and this also greatly depends on the
> amount of tests that we run.

Another problem with this is that someone will propose a patch that
does:

+define FOO_BUILD_CMDS
+	rm -rf /
+endef

Even if you don't run your autobuilds as root, it's going to cause
quite some troubles to the build infrastructure itself, unless you
start each build in a separate container or something like that. It's
possible, but I'm just showing that things are not as easy as one might
think.

> What to test: it doesn't need to be every imaginable configuration,
> but it would be nice to have one or more standard builds, a blackfin
> (no-mmu) build, a uclibc configuration, a full versus basic
> configuration, ... We could have a set of, say, 15 combinations, and
> we pick, say, 5 of them for each patch to test. For example, you have:
> powerpc, buildroot basic uclibc toolchain
> powerpc, buildroot basic glibc toolchain
> powerpc, buildroot full uclibc toolchain
> powerpc, buildroot full glibc toolchain
> powerpc, external sourcery (full) toolchain
> (more or less the same for the other archs)

And who would provide the appropriate configuration to test a package?
For example, I've just tested the ModemManager package, which is only
available if udev is used as the /dev management method.

What I mean is that automating all the choices that a human being is
doing when testing patches is going to be really tricky.

> > On my side, I'm really skeptical about that one: I think we should
> > rather merge patches faster, so that we simply rely on the existing
> > autobuilder infrastructure, which works well.
> 
> I'm not saying we should keep patches in this test queue for a long
> time. I'm also not saying that Peter is not free to apply a patch even
> if it was not tested in this system.
> However, we are having quite a number of failures on basic things like
> thread support, mmu support, ...

Does it matter? We simply fix them, and that's it.

I really think we're trying to avoid a problem that doesn't exist. The
autobuilders have been put in place precisely to test all those things,
so why do you want to put in place additional barriers to get patches
merged, while we already have the testing infrastructure once things
are committed to catch those problems?

You have as goal to have fully green autobuilder results 100% of the
time. I think this goal is wrong, because the autobuilders are
precisely here to lower the barrier to merge patches, by having this
"safety net". I do agree we should aim at 100% green results between
-rc1 and the final release, but not during the development cycle.

We should be looking at _lowering_ the barrier for merging patches, not
adding additional barriers.

> The idea of providing a list of reference configurations that
> developers should test their new packages on may be sufficient too,

Yes, this seems like a *much* better idea. Provide a list of
configurations that are interesting to test, with pre-built toolchains,
so that submitters can very quickly run a test build on various
architectures and in various situations. Don't make it an absolute
prerequisite, but document it in the manual, and explains what each
toolchain/architecture configuration is exercising.

> Providing this list of configurations is not that hard, we already
> have a bunch of toolchains on the autobuilders. A script to run the
> selected configurations in turn would be nice.

A script will not work easily, IMO. If you've already added depends on
BR2_USE_MMU on your package, does it make sense to test Blackfin
toolchains? No. If you've already added the thread dependency on your
package, does it make sense to test the no-thread toolchains? No.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Availability of old build results

[Arnout Vandecappelle]

On 06/12/13 10:57, Thomas De Schampheleire wrote:
> If the patch is fixing an autobuild problem, we really should be
> repeating that autobuild configuration instead (but that could be done
> manually by someone).

  I think it would be nice if a failing autobuild configuration would 
automatically be added to jenkins, and that it would be automatically 
removed from there once it succeeds. Of course, that is only feasible if 
the number of failures is low (maximum 1 or 2 per day) and they get fixed 
quickly, otherwise jenkins will blow up. So it's not something we can 
consider right away.

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Availability of old build results

[Thomas De Schampheleire]

Hi Thomas, all,

On Fri, Dec 6, 2013 at 11:11 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Thomas De Schampheleire,
>
> On Fri, 6 Dec 2013 10:57:42 +0100, Thomas De Schampheleire wrote:
>
>> This is something to be decided (if we go this route).
>> The simplest one is to automatically take every patch that appears in
>> patchwork, and run it through the test system.
>
> How do you know on which commit a given patch does apply?

One would need to assume that it still applies on the current master
(which is often true, but not always). Obviously the test could not
continue if there is a conflict applying the patch.

>
>> The disadvantage is
>> that you may be testing crap patches that would easily be spotted
>> during review, and thus you are investing the limited build capacity
>> in the wrong builds. This may be ok though, if we can add some extra
>> servers to the build capacity, and this also greatly depends on the
>> amount of tests that we run.
>
> Another problem with this is that someone will propose a patch that
> does:
>
> +define FOO_BUILD_CMDS
> +       rm -rf /
> +endef
>
> Even if you don't run your autobuilds as root, it's going to cause
> quite some troubles to the build infrastructure itself, unless you
> start each build in a separate container or something like that. It's
> possible, but I'm just showing that things are not as easy as one might
> think.

Point taken.

>
>> What to test: it doesn't need to be every imaginable configuration,
>> but it would be nice to have one or more standard builds, a blackfin
>> (no-mmu) build, a uclibc configuration, a full versus basic
>> configuration, ... We could have a set of, say, 15 combinations, and
>> we pick, say, 5 of them for each patch to test. For example, you have:
>> powerpc, buildroot basic uclibc toolchain
>> powerpc, buildroot basic glibc toolchain
>> powerpc, buildroot full uclibc toolchain
>> powerpc, buildroot full glibc toolchain
>> powerpc, external sourcery (full) toolchain
>> (more or less the same for the other archs)
>
> And who would provide the appropriate configuration to test a package?
> For example, I've just tested the ModemManager package, which is only
> available if udev is used as the /dev management method.
>
> What I mean is that automating all the choices that a human being is
> doing when testing patches is going to be really tricky.

Point taken.

>
>> > On my side, I'm really skeptical about that one: I think we should
>> > rather merge patches faster, so that we simply rely on the existing
>> > autobuilder infrastructure, which works well.
>>
>> I'm not saying we should keep patches in this test queue for a long
>> time. I'm also not saying that Peter is not free to apply a patch even
>> if it was not tested in this system.
>> However, we are having quite a number of failures on basic things like
>> thread support, mmu support, ...
>
> Does it matter? We simply fix them, and that's it.
>
> I really think we're trying to avoid a problem that doesn't exist. The
> autobuilders have been put in place precisely to test all those things,
> so why do you want to put in place additional barriers to get patches
> merged, while we already have the testing infrastructure once things
> are committed to catch those problems?
>
> You have as goal to have fully green autobuilder results 100% of the
> time. I think this goal is wrong, because the autobuilders are
> precisely here to lower the barrier to merge patches, by having this
> "safety net". I do agree we should aim at 100% green results between
> -rc1 and the final release, but not during the development cycle.
>
> We should be looking at _lowering_ the barrier for merging patches, not
> adding additional barriers.

My goal is indeed to have autobuilder results being 'green' as much as
possible, all the time. Obviously it's unrealistic to have 100%
success all the time: there are many tricky issues one can not
reasonably expect to be right the first time a patch is submitted.

However, many autobuild failures are trivial: thread support required,
mmu support required, missing required dependency, not building with
uClibc toolchains, ... This is the type of failures one could expect
to detect beforehand, not after.

One of the reasons I'd prefer close to 100% green autobuilds is that a
failure will be noticed much more. Attention will be drawn to the fact
that there is a problem. Until now, the amount of autobuild failures
was relatively high, and the amount of patches actually fixing these
problems was relatively low (compared to the total amount of patch
submissions). Each day there'd be a mail with the build results, and
one would see yet another 20 something packages being broken. To dive
in and look at this is not very appealing for the average developer;
fixing one package seems like a drop on a hot plate.
In november, we were able to reach much better figures in autobuild,
thanks to the increased attention and the collaborative efforts of
analysing the problems. Gradually, the amount of remaining failures
dropped and this was very visible too. Developers are then even more
attracted to fix the problems.

Note that I agree that the barrier to submit patches should be low. At
this moment I think it is low: the community is very open and
welcoming towards new developers, patches are generally reviewed well
(certainly for the last few releases) and the documentation about the
expected patch and package format is in good shape. Above all this, it
looks like many new developers do actually read this documentation (!)
and many patches from 'newcomers' are very qualitative already.
In my opinion, having a test system as sketched above does not raise
the barrier for patch submitters. They still have to submit their
patch as before. But, between submission and acceptance some tests are
run (semi-)automatically. If this fails, the author is requested to
look into this problem and fix it, just as other feedback is given
based on code review alone.
In fact, strictly speaking, the alternative proposal below where
developers are asked to run some tests themselves (even though
optional) adds more barrier than the tests between submission and
acceptance.

Regardless, you raised some valid points about the difficulty in
implementing this from a practical perspective, so I agree that the
alternative proposal below may be more worth looking into...

>
>> The idea of providing a list of reference configurations that
>> developers should test their new packages on may be sufficient too,
>
> Yes, this seems like a *much* better idea. Provide a list of
> configurations that are interesting to test, with pre-built toolchains,
> so that submitters can very quickly run a test build on various
> architectures and in various situations. Don't make it an absolute
> prerequisite, but document it in the manual, and explains what each
> toolchain/architecture configuration is exercising.
>
>> Providing this list of configurations is not that hard, we already
>> have a bunch of toolchains on the autobuilders. A script to run the
>> selected configurations in turn would be nice.
>
> A script will not work easily, IMO. If you've already added depends on
> BR2_USE_MMU on your package, does it make sense to test Blackfin
> toolchains? No. If you've already added the thread dependency on your
> package, does it make sense to test the no-thread toolchains? No.

Say you have 5 reference configurations. A developer wishing to submit
a patch would need to either clone 5 repositories with his patch and
run a build in each one of them, or handle one configuration at a time
and clean the repository each time. The amount of commands to run (and
the time to do this) is sufficiently large to talk about a barrier
IMO.
I agree that not all reference configurations may be applicable for
each package, and again it may be difficult to automatically set a
valid, sane configuration for the package to test, but if in any way
it is possible to have some automation, I think it would be
appreciated by developers.

That said, we could already start with the pure manual way and see how
this works out, rather than trying to do too much from the start.

Best regards,
Thomas