[Buildroot] [RFC] Continuous integration

[Buildroot] [RFC] Continuous integration

[Jérôme Pouiller]

Hello,

Unfortunately, I am really not sure to be able to follow Buildroot 
Developpers' Day (even with Google Hangout). However, I would like to 
expose one my ideas about patch integration scalability problem.

IMHO, current autobuilder is great to find corner cases, but for simple 
case, I think we can do better. I suggest to build ALL packages with a 
selection of representative reference configurations (uclibc, glibc, 
static, w/o IPv6, w/o MMU...). To make sense, this autobuilder should 
pool git changes and compile in priority new packages, modified package 
and packages which depends (directly or indirectly) from modified 
packages. 

Ideally, it should be able to give test results for branches before they 
would be merged.

It should also detect regression and send necessary alert. It would be 
better if it may identify commit and author of a regression.

I begin to wrote an autobuilder that would looks like that. You can sees
results there : http://sysmic.org/~jezz/autobuilder/ (at beginning, it 
was mainly for my personal needs)

It use a set of reference configurations which normally include all 
packages (at least as much as possible). It compute list of packages and 
their directories, list of targets for each configuration and 
dependencies of of each packages for each configuration. It is able to 
compute reverse dependencies and recursive dependencies.

Next, it ask to git modification time for each package directory. It is 
able to detect couple of packages/configuration which build time is 
older than package directory modification time.

It compute list of packages/configuration couple and sort it : never 
built first, modified next, a dependency modified after and finally 
other packages, ordered by last build time. Job queue is available 
there: http://sysmic.org/~jezz/autobuilder/jobqueue.html (/!\ 4Mo).

It build elements of job list until change is detected on git repository 
(in this case, it rebuild job queue).

For performance reasons, I don't run make clean between each package. I 
know it may be problematic for reproducibility. However, script will run 
make clean when it is about to rebuild a package that was not modified 
since last build and output directory had not been cleaned since last 
build (= same package with same output directory).

Finally, it dump result. It is able to detect when a package has 
compiled correctly, or it has failed or if a dependency has failed (and 
in this case, it shows problematic package).

Code is available there: 
   https://github.com/jerome-pouiller/br-continuous




-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [RFC] Continuous integration

[Arnout Vandecappelle]

On 25/10/13 14:46, J?r?me Pouiller wrote:
> Hello,
>
> Unfortunately, I am really not sure to be able to follow Buildroot
> Developpers' Day (even with Google Hangout). However, I would like to
> expose one my ideas about patch integration scalability problem.
>
> IMHO, current autobuilder is great to find corner cases, but for simple
> case, I think we can do better. I suggest to build ALL packages with a
> selection of representative reference configurations (uclibc, glibc,
> static, w/o IPv6, w/o MMU...). To make sense, this autobuilder should
> pool git changes and compile in priority new packages, modified package
> and packages which depends (directly or indirectly) from modified
> packages.


  Hi J?r?me,

  This idea sounds good. Unfortunately, it looks like nobody read your 
mail before the developer days so we didn't discuss this subject...

>
> Ideally, it should be able to give test results for branches before they
> would be merged.

  We'd like to keep the current workflow of committing each patch 
individually - it gives cleaner history than with merges. But that's not 
really relevant, because these test builds could run for each patch 
individually and report success, which indicates that it can be applied 
by Peter.

>
> It should also detect regression and send necessary alert. It would be
> better if it may identify commit and author of a regression.
>
> I begin to wrote an autobuilder that would looks like that. You can sees
> results there : http://sysmic.org/~jezz/autobuilder/ (at beginning, it
> was mainly for my personal needs)
>
> It use a set of reference configurations which normally include all
> packages (at least as much as possible). It compute list of packages and
> their directories, list of targets for each configuration and
> dependencies of of each packages for each configuration. It is able to
> compute reverse dependencies and recursive dependencies.

  Why all packages? If it's about testing new packages, then it should be 
tested with only the new package selected (+ any dependencies). If it's 
about testing infrastructural changes, then it should be done on a clean 
tree (which you mention below that you don't do).

>
> Next, it ask to git modification time for each package directory. It is
> able to detect couple of packages/configuration which build time is
> older than package directory modification time.
>
> It compute list of packages/configuration couple and sort it : never
> built first, modified next, a dependency modified after and finally
> other packages, ordered by last build time. Job queue is available
> there: http://sysmic.org/~jezz/autobuilder/jobqueue.html (/!\ 4Mo).
>
> It build elements of job list until change is detected on git repository
> (in this case, it rebuild job queue).
>
> For performance reasons, I don't run make clean between each package. I
> know it may be problematic for reproducibility. However, script will run
> make clean when it is about to rebuild a package that was not modified
> since last build and output directory had not been cleaned since last
> build (= same package with same output directory).
>
> Finally, it dump result. It is able to detect when a package has
> compiled correctly, or it has failed or if a dependency has failed (and
> in this case, it shows problematic package).
>
> Code is available there:
>     https://github.com/jerome-pouiller/br-continuous


  Do you also have a server on which to run this? Could it send out 
success/failure reports to the list (or maybe first to the author and a 
few key developers, until we're sure it works well)?

  Regards,
  Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC] Continuous integration

[Jérôme Pouiller]

On Monday 28 October 2013 07:50:31 Arnout Vandecappelle wrote:
> On 25/10/13 14:46, J?r?me Pouiller wrote:
[...]
>   This idea sounds good. Unfortunately, it looks like nobody read your
> mail before the developer days so we didn't discuss this subject...
It my fault, I was late.

> > Ideally, it should be able to give test results for branches before
> > they would be merged.
> 
>   We'd like to keep the current workflow of committing each patch
> individually - it gives cleaner history than with merges. 
I mean before they would be committed. We are agree.

> But that's
> not really relevant, because these test builds could run for each
> patch individually and report success, which indicates that it can be
> applied by Peter.
Current implementation run tests when a commit is done. You suggest to 
run test as soon as patch is sent to ML. 

It is a good idea but very different than current implementation. I will 
need a little more time to implement this.


> > It should also detect regression and send necessary alert. It would
> > be better if it may identify commit and author of a regression.
> > 
> > I begin to wrote an autobuilder that would looks like that. You can
> > sees results there : http://sysmic.org/~jezz/autobuilder/ (at
> > beginning, it was mainly for my personal needs)
> > 
> > It use a set of reference configurations which normally include all
> > packages (at least as much as possible). It compute list of packages
> > and their directories, list of targets for each configuration and
> > dependencies of of each packages for each configuration. It is able
> > to compute reverse dependencies and recursive dependencies.
> 
>   Why all packages? If it's about testing new packages, then it should
> be tested with only the new package selected (+ any dependencies). If
> it's about testing infrastructural changes, then it should be done on
> a clean tree (which you mention below that you don't do).
Idea was to rebuild in priority new/modified packages. When there is 
nothing more to do. Rebuild a package that was not modified.


> > Next, it ask to git modification time for each package directory. It
> > is able to detect couple of packages/configuration which build time
> > is older than package directory modification time.
> > 
> > It compute list of packages/configuration couple and sort it : never
> > built first, modified next, a dependency modified after and finally
> > other packages, ordered by last build time. Job queue is available
> > there: http://sysmic.org/~jezz/autobuilder/jobqueue.html (/!\ 4Mo).
> > 
> > It build elements of job list until change is detected on git
> > repository (in this case, it rebuild job queue).
> > 
> > For performance reasons, I don't run make clean between each
> > package. I know it may be problematic for reproducibility. However,
> > script will run make clean when it is about to rebuild a package
> > that was not modified since last build and output directory had not
> > been cleaned since last build (= same package with same output
> > directory).
> > 
> > Finally, it dump result. It is able to detect when a package has
> > compiled correctly, or it has failed or if a dependency has failed
> > (and in this case, it shows problematic package).
> > 
> > Code is available there:
> >     https://github.com/jerome-pouiller/br-continuous
> 
>   Do you also have a server on which to run this? Could it send out
> success/failure reports to the list (or maybe first to the author and
> a few key developers, until we're sure it works well)?
Normally, http://sysmic.org/~jezz/autobuilder/ should work, isn't? You 
may also look at http://sysmic.org/~jezz/autobuilder/jobqueue.html for 
job list . 



-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [RFC] Continuous integration

[rjbarnet at rockwellcollins.com]

J?r?me, Thomas P, All,

I don't know what Jerome's plans are for his autobuilder, but I like the 
way that results are displayed on the links you provided below. It has 
colors and has nice features for sorting the results which make it easy 
for me to see the information I'm interested in rather quickly.

Thomas P - in your email entitled - Architecture build statistics - it 
would be nice to see a page where these statistics could displayed 
continuously. One idea that comes to be mind is that it would be nice if 
this autobuild site could aggregate the results from the current build 
infrastructure (maybe it already does). The website would then be able to 
provide these architecture statistic and package statistic break.

J?r?me Pouiller <jezz@sysmic.org> wrote on 10/25/2013 07:46:49 AM:

> I begin to wrote an autobuilder that would looks like that. You can sees
> results there : http://sysmic.org/~jezz/autobuilder/ (at beginning, it 
> was mainly for my personal needs)
> 
> It use a set of reference configurations which normally include all 
> packages (at least as much as possible). It compute list of packages and 

> their directories, list of targets for each configuration and 
> dependencies of of each packages for each configuration. It is able to 
> compute reverse dependencies and recursive dependencies.
> 
> Next, it ask to git modification time for each package directory. It is 
> able to detect couple of packages/configuration which build time is 
> older than package directory modification time.
> 
> It compute list of packages/configuration couple and sort it : never 
> built first, modified next, a dependency modified after and finally 
> other packages, ordered by last build time. Job queue is available 
> there: http://sysmic.org/~jezz/autobuilder/jobqueue.html (/!\ 4Mo).

I like the website and I know some of my colleagues are intrigued by this 
as this might provide an way to continuously test our changes across 
different architectures. Hopefully we have some time soon to investigate 
this.

Thanks,
-Ryan

[Buildroot] [RFC] Continuous integration

[Matt Weber]

On Fri, Oct 25, 2013 at 7:46 AM, J?r?me Pouiller <jezz@sysmic.org> wrote:
> Hello,
>

<snip>

>
> Code is available there:
>    https://github.com/jerome-pouiller/br-continuous
>

We've been giving your script a try and made some updates.  So I went
ahead and created a (github)fork (should have a few updates checked in
tonight).
I fixed a couple items that may have been bugs (or possibly just how I
used the script).  I current have been running into an issue where
after git updates are detected, previously built pkgs try to rebuild
without doing a clean.  I see where the conditional is that should be
executing the clean but haven't tracked down why I'm not hitting that
path yet.
I also noticed that once the script processes all jobs it gets into a
loop, I added a fix for that that prevents running the jobs unless
there are still jobs to execute in the job list.

I noticed there isn't a good way to restart the script after you stop
it, is my understanding correct?  I (temporarily) added a cleanup
sequence before the mainloop executes to remove history and each cfgs
build before proceeding.

I think there might be a bug with detecting when a package fails to
build, I'll see if I can resolve this and let you know.  I haven't had
a package actually error yet, but have had some that never built
because of host dependency error or download error

I was wondering what your thoughts were for the report?  I'm guessing
what's currently committed was the start of something based on this
thread of discussion.  I could see gathering a list of package err,
git id, and list to build output (similarly to the autobuilder
results).

Thanks,
Matt Weber
mlweber1 at rockwellcollins.com


>
>
>
> --
> J?r?me Pouiller, Sysmic
> Embedded Linux specialist
> http://www.sysmic.fr
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [RFC] Continuous integration

[Jérôme Pouiller]

On Tuesday 05 November 2013 10:29:52 Matt Weber wrote:
> On Fri, Oct 25, 2013 at 7:46 AM, J?r?me Pouiller <jezz@sysmic.org> 
wrote:
> > Hello,
> 
> <snip>
> 
> > Code is available there:
> >    https://github.com/jerome-pouiller/br-continuous
> 
> We've been giving your script a try and made some updates.  So I went
> ahead and created a (github)fork (should have a few updates checked in
> tonight).
Thanks for your interest.

> I fixed a couple items that may have been bugs (or possibly just how I
> used the script).  I current have been running into an issue where
> after git updates are detected, previously built pkgs try to rebuild
> without doing a clean. I see where the conditional is that should be
> executing the clean but haven't tracked down why I'm not hitting that
> path yet.
By the way, it's difficult to find the right place to do a clean. 

 - Between each package, it take to much time.

 - After any changes on git. But, for packages with many dependencies 
(like qt5webkit), it means we will have to wait at least 3h per 
configuration to have results. However, this solution is simple and 
guarantee reproducibility of builds.

 - Currently, I check if a clean happens since last build (in reality, 
not any kind of build) of package. If it not the case, I launch make 
clean. 


> I also noticed that once the script processes all jobs it gets into a
> loop, I added a fix for that that prevents running the jobs unless
> there are still jobs to execute in the job list.
Good point. I will merge that.

> I noticed there isn't a good way to restart the script after you stop
> it, is my understanding correct?  I (temporarily) added a cleanup
> sequence before the mainloop executes to remove history and each cfgs
> build before proceeding.
Normally, current implementation should work. It should populate 
database with previous results. It is just long to launch (20min on my 
server !). It would be better to change format of results to improve 
launch time.

> I think there might be a bug with detecting when a package fails to
> build, I'll see if I can resolve this and let you know.  I haven't had
> a package actually error yet, but have had some that never built
> because of host dependency error or download error
I did not noticed that.

> I was wondering what your thoughts were for the report?  I'm guessing
> what's currently committed was the start of something based on this
> thread of discussion.  I could see gathering a list of package err,
> git id, and list to build output (similarly to the autobuilder
> results).
List of all package errors would be too long. I think report should only 
include package status changes (OK -> KO, N/A -> KO, but also KO -> OK). 

Report should also report configuration changes after "yes | make 
oldconfig".

Currently my main problem is execution of "yes | make oldconfig". In 
case "y" is not a valid answer, this command will loop infinitely (and 
"yes '' | make oldconfig" is not a good solution since i want to build 
new packages).



-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [RFC] Continuous integration

[Matthew Weber]

J?r?me Pouiller <j.pouiller@expemb.com> wrote on 11/07/2013 07:37:21 AM:

> From: J?r?me Pouiller <jezz@sysmic.org>
> To: Matt Weber <mlweber1@rockwellcollins.com>
> Cc: buildroot at busybox.net
> Date: 11/07/2013 07:37 AM
> Subject: Re: [Buildroot] [RFC] Continuous integration
> Sent by: J?r?me Pouiller <j.pouiller@expemb.com>
> 
<snip>

> > I fixed a couple items that may have been bugs (or possibly just how I
> > used the script).  I current have been running into an issue where
> > after git updates are detected, previously built pkgs try to rebuild
> > without doing a clean. I see where the conditional is that should be
> > executing the clean but haven't tracked down why I'm not hitting that
> > path yet.
> By the way, it's difficult to find the right place to do a clean. 
> 
>  - Between each package, it take to much time.
> 
>  - After any changes on git. But, for packages with many dependencies 
> (like qt5webkit), it means we will have to wait at least 3h per 
> configuration to have results. However, this solution is simple and 
> guarantee reproducibility of builds.
> 
>  - Currently, I check if a clean happens since last build (in reality, 
> not any kind of build) of package. If it not the case, I launch make 
> clean. 
> 
We might have a fourth semi-continuous integration scenario that prevents 
server thrashing at each check-in. 
- We run the script every night processing the current GIT tip at that 
time. 
- We clean each of the full cfg builds and let the script do a complete 
re-build.  (We do this because the individual pkg  cleans aren't 
guaranteed 
to cleanup the pkg installs)
- Once all jobs complete we terminate the script and eventually will send
the report with status.
(I'll clean-up what I changed and get it committed.  Would you be up for 
optional runtime config options?  One might be to enable continuous 
or a one-stop script execution.)

<snip>

> 
> > I noticed there isn't a good way to restart the script after you stop
> > it, is my understanding correct?  I (temporarily) added a cleanup
> > sequence before the mainloop executes to remove history and each cfgs
> > build before proceeding.
> Normally, current implementation should work. It should populate 
> database with previous results. It is just long to launch (20min on my 
> server !). It would be better to change format of results to improve 
> launch time.
I had a chance to look at this a bit more and I removed some of the 
clean-up
and made it so that the script could be used to do a single "one-shot" 
build
of the current GIT.  (I'll push this once I get it cleaned up)

> 
> > I think there might be a bug with detecting when a package fails to
> > build, I'll see if I can resolve this and let you know.  I haven't had
> > a package actually error yet, but have had some that never built
> > because of host dependency error or download error
> I did not noticed that.
I figured out what was going on.  It was a failure of a Dependent package. 

So in my case the busybox was failing, however the busybox package never 
actually
failed itself.  I think this was because it was built as a Dep for other 
packages and never having it's own build attempted.  I was going to look
at this a bit more, as I think it's doing this because you flag a build 
failure for the current package build, not for a failure of a Dep.

> 
> > I was wondering what your thoughts were for the report?  I'm guessing
> > what's currently committed was the start of something based on this
> > thread of discussion.  I could see gathering a list of package err,
> > git id, and list to build output (similarly to the autobuilder
> > results).
> List of all package errors would be too long. I think report should only 

> include package status changes (OK -> KO, N/A -> KO, but also KO -> OK). 

Yeah, I agree, something with a simple status is preferred.  Maybe with 
a cfg summary at the top and a drill down with hyper links to the package 
status (similarly to the webpage).

> 
> Report should also report configuration changes after "yes | make 
> oldconfig".
Ok, so you're thinking additions/removals of buildroot configuration 
options?

> 
> Currently my main problem is execution of "yes | make oldconfig". In 
> case "y" is not a valid answer, this command will loop infinitely (and 
> "yes '' | make oldconfig" is not a good solution since i want to build 
> new packages).
Yeah, already had some fun with that :-)

Thanks,
Matt Weber
mlweber1 at rockwellcollins.com

[Buildroot] [RFC] Continuous integration

[Maxime Ripard]

Hi Matthew,

On Fri, Nov 08, 2013 at 01:35:46PM -0600, Matthew Weber wrote:
> We might have a fourth semi-continuous integration scenario that prevents 
> server thrashing at each check-in. 
> - We run the script every night processing the current GIT tip at that 
> time. 
> - We clean each of the full cfg builds and let the script do a complete 
> re-build.  (We do this because the individual pkg  cleans aren't 
> guaranteed 
> to cleanup the pkg installs)
> - Once all jobs complete we terminate the script and eventually will send
> the report with status.
> (I'll clean-up what I changed and get it committed.  Would you be up for 
> optional runtime config options?  One might be to enable continuous 
> or a one-stop script execution.)

It's pretty much what I've setup here already:
http://jenkins.free-electrons.com/job/buildroot/

It runs every night and start a clean build of all the configuration
we have if a new commit has been merged.

The only thing missing here is that the emails get only send to Peter,
Thomas P. and I because I didn't want to spam the list, but this can
easily be changed.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131110/6a991dc4/attachment.asc>

[Buildroot] [RFC] Continuous integration

[Matthew Weber]

Maxime Ripard <maxime.ripard@free-electrons.com> wrote on 11/10/2013 
05:18:11 AM:

> From: Maxime Ripard <maxime.ripard@free-electrons.com>
> To: Matthew Weber <mlweber1@rockwellcollins.com>
> Cc: J?r?me Pouiller <jezz@sysmic.org>, buildroot at busybox.net
> Date: 11/10/2013 05:20 AM
> Subject: Re: [Buildroot] [RFC] Continuous integration
> 
> Hi Matthew,
> 
> On Fri, Nov 08, 2013 at 01:35:46PM -0600, Matthew Weber wrote:
> > We might have a fourth semi-continuous integration scenario that 
prevents 
> > server thrashing at each check-in. 
> > - We run the script every night processing the current GIT tip at that 

> > time. 
> > - We clean each of the full cfg builds and let the script do a 
complete 
> > re-build.  (We do this because the individual pkg  cleans aren't 
> > guaranteed 
> > to cleanup the pkg installs)
> > - Once all jobs complete we terminate the script and eventually will 
send
> > the report with status.
> > (I'll clean-up what I changed and get it committed.  Would you be up 
for 
> > optional runtime config options?  One might be to enable continuous 
> > or a one-stop script execution.)
> 
> It's pretty much what I've setup here already:
> http://jenkins.free-electrons.com/job/buildroot/
> 
> It runs every night and start a clean build of all the configuration
> we have if a new commit has been merged.
> 
> The only thing missing here is that the emails get only send to Peter,
> Thomas P. and I because I didn't want to spam the list, but this can
> easily be changed.
> 
Cool, good to know that the default set of defconfigs is being validated 
against
the current tip.
I think the scenario we're trying to target is validating offline/internal 

board targets for different custom products.  We don't necessarily want 
the builds
to catch every commit (limited on build server resources), but just give 
us granular
enough feedback to know roughly which commit caused a new issue. 
We evaluated buildbot and Jenkins and had been debating about just doing a 

simple script like what J?r?me has done.  So when we saw what he had 
started it 
looked like a good opportunity to give it a try.  Currently we have it 
setup for
a nightly validation of a few defconfigs. 
For a continuous integration, I do agree Jenkins is a very good option.

Thanks,
Matt Weber
mlweber1 at rockwellcollins.com

> Maxime
> 
> -- 
> Maxime Ripard, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
> [attachment "signature.asc" deleted by Matthew L Weber/CedarRapids/
> RockwellCollins] 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131111/89080467/attachment.html>