[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Thomas Petazzoni]

Hello,

Most likely the majority of the Buildroot developers are aware of the
Autobuilder mechanism we have, for which the results are available at
http://autobuild.buildroot.org/ and posted daily on the mailing list.

Until now, both Peter and I were using ad-hoc scripts that were so ugly
that they were not published, so it was not easy for others to join the
autobuilder effort.

I'm therefore happy to announce that I've written a completely new
version of a Python script that allows to run an autobuilder instance
on a given machine. It's a single script, and its installation is very
easy (though one must of course pay attention to have the Buildroot
pre-requisites installed). So it means that from now on, it's a lot
easier for others than Peter and I to set up an autobuilder instance.
If you have some spare cycles on a server, or on your workstation, even
for just a few hours per day, this might be of interest to you!

The script is available at:

  http://git.buildroot.net/buildroot-test/plain/scripts/autobuild-run

(NOTE: I am *not* a Python developer, there are very possibly many
possible improvements to do in the script. Error handling and logging
is probably not at its best. Patches welcome!).

Getting started
===============


To get started, you can simply run it with:

  ./autobuild-run --ninstances 2 --njobs 3

This will run 2 parallel independent Buildroot builds, each using 3
parallel jobs (i.e BR2_JLEVEL=3). In this configuration, autobuild-run
will store the results in local tarballs (one for each build), they are
not submitted to the http://autobuild.buildroot.org website.

This mode is a good way to test if your autobuilder instance is working
as expected. It is also a way for people to help improve the script
without necessarily running a full-blown autobuilder instance, and also
to verify that the changes are 

What's going on
===============

Each instance runs in its own instance-<number>/ directory, with inside:

 - dl/, the download directory for this instance. In the current
   Buildroot, we cannot share the same download directory between
   parallel instances. This is being fixed by Yann.

 - buildroot/, the Buildroot source code.

 - output/, the build output.

To see what's going on while builds are running, there are two level of
log files:

 - instance-<number>/instance.log, which persists across builds and
   gives some high-level information on what's going on (build
   starting, build ended with success or not, etc.)

 - instance-<number>/output/logfile, which is valid only for the
   duration of the current build, and gives the usual Buildroot output.

Getting results submitted
=========================

If you want the results of your autobuilder instance to be submitted to
http://autobuild.buildroot.org, you should first contact me to get some
credentials to submit those results. Once this is done, you need to
write a configuration file that looks like this:

[main]
ninstances = 2
njobs = 3
http-login = <the-login>
http-password = <the-password>
submitter = Your Name (name of machine)

And start your autobuilder instance using:

  ./autobuild-run -c <configuration-file-path>

And that's it.

Testing coverage
================

 * The architecture/toolchain combinations are chosen randomly amongst
   a set of pre-defined configurations, listed at
   http://autobuild.buildroot.org/toolchains/configs/ (see the .csv
   file).

 * The packages are chosen randomly using 'randpackageconfig', and
   there are some exceptions encoded in the script to avoid known
   problematic configurations.

Current deployment
==================

The script is currently deployed on two machines:

 - The Free Electrons build server, on which it replaces the old crap I
   was using to do the autobuilds.

 - Another private server of mine, which is the one actually hosting
   the http://autobuild.buildroot.org. I'm not sure I will let builds
   run forever on this machine, but it was useful to have two machines
   using the script to test in different conditions.

It is worth mentioning that due to this deployment, there are a few
bogus results on http://autobuild.buildroot.org due to missing packages
on the new build server, issues in the script or for other reasons.
This should hopefully settle down in the next few days.

Hopefully, Peter will join the dance and convert gcc10, gcc20 and
gcc110 to use the script as well.

Contribute
==========

Feel free to test the script, help improve it, and send patches. It is
available in the Git repository at:
http://git.buildroot.net/buildroot-test/.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Thomas De Schampheleire]

Hi Thomas,

On Sat, Jun 14, 2014 at 9:23 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> Most likely the majority of the Buildroot developers are aware of the
> Autobuilder mechanism we have, for which the results are available at
> http://autobuild.buildroot.org/ and posted daily on the mailing list.
>
> Until now, both Peter and I were using ad-hoc scripts that were so ugly
> that they were not published, so it was not easy for others to join the
> autobuilder effort.
>
> I'm therefore happy to announce that I've written a completely new
> version of a Python script that allows to run an autobuilder instance
> on a given machine. It's a single script, and its installation is very
> easy (though one must of course pay attention to have the Buildroot
> pre-requisites installed). So it means that from now on, it's a lot
> easier for others than Peter and I to set up an autobuilder instance.
> If you have some spare cycles on a server, or on your workstation, even
> for just a few hours per day, this might be of interest to you!
>
> The script is available at:
>
>   http://git.buildroot.net/buildroot-test/plain/scripts/autobuild-run
>
> (NOTE: I am *not* a Python developer, there are very possibly many
> possible improvements to do in the script. Error handling and logging
> is probably not at its best. Patches welcome!).
>
> Getting started
> ===============
>
>
> To get started, you can simply run it with:
>
>   ./autobuild-run --ninstances 2 --njobs 3
>
> This will run 2 parallel independent Buildroot builds, each using 3
> parallel jobs (i.e BR2_JLEVEL=3). In this configuration, autobuild-run
> will store the results in local tarballs (one for each build), they are
> not submitted to the http://autobuild.buildroot.org website.
>
> This mode is a good way to test if your autobuilder instance is working
> as expected. It is also a way for people to help improve the script
> without necessarily running a full-blown autobuilder instance, and also
> to verify that the changes are
>
> What's going on
> ===============
>
> Each instance runs in its own instance-<number>/ directory, with inside:
>
>  - dl/, the download directory for this instance. In the current
>    Buildroot, we cannot share the same download directory between
>    parallel instances. This is being fixed by Yann.
>
>  - buildroot/, the Buildroot source code.
>
>  - output/, the build output.
>
> To see what's going on while builds are running, there are two level of
> log files:
>
>  - instance-<number>/instance.log, which persists across builds and
>    gives some high-level information on what's going on (build
>    starting, build ended with success or not, etc.)
>
>  - instance-<number>/output/logfile, which is valid only for the
>    duration of the current build, and gives the usual Buildroot output.
>
> Getting results submitted
> =========================
>
> If you want the results of your autobuilder instance to be submitted to
> http://autobuild.buildroot.org, you should first contact me to get some
> credentials to submit those results. Once this is done, you need to
> write a configuration file that looks like this:
>
> [main]
> ninstances = 2
> njobs = 3
> http-login = <the-login>
> http-password = <the-password>
> submitter = Your Name (name of machine)
>
> And start your autobuilder instance using:
>
>   ./autobuild-run -c <configuration-file-path>
>
> And that's it.
>
> Testing coverage
> ================
>
>  * The architecture/toolchain combinations are chosen randomly amongst
>    a set of pre-defined configurations, listed at
>    http://autobuild.buildroot.org/toolchains/configs/ (see the .csv
>    file).
>
>  * The packages are chosen randomly using 'randpackageconfig', and
>    there are some exceptions encoded in the script to avoid known
>    problematic configurations.
>
> Current deployment
> ==================
>
> The script is currently deployed on two machines:
>
>  - The Free Electrons build server, on which it replaces the old crap I
>    was using to do the autobuilds.
>
>  - Another private server of mine, which is the one actually hosting
>    the http://autobuild.buildroot.org. I'm not sure I will let builds
>    run forever on this machine, but it was useful to have two machines
>    using the script to test in different conditions.
>
> It is worth mentioning that due to this deployment, there are a few
> bogus results on http://autobuild.buildroot.org due to missing packages
> on the new build server, issues in the script or for other reasons.
> This should hopefully settle down in the next few days.
>
> Hopefully, Peter will join the dance and convert gcc10, gcc20 and
> gcc110 to use the script as well.
>
> Contribute
> ==========
>
> Feel free to test the script, help improve it, and send patches. It is
> available in the Git repository at:
> http://git.buildroot.net/buildroot-test/.


First of all thanks a lot for making this available!
I am currently testing the script, and hope to get permission to run
it during the night on one or more work machines.
I had to make one modification currently: since we are behind a
firewall, running 'git' directly is not possible, but I need to use a
wrapper script. I added this:

+try:
+    gitcmd = os.environ['GIT']
+except KeyError:
+    gitcmd = 'git'

at the top of the script, and replaced all direct occurrences of 'git'
with gitcmd.
This allows me to run 'GIT=git-wrapper autobuild-run'.

I used an environment variable because it seemed superfluous to add a
configuration variable for it, but if you prefer I can do that too.

Is there an easy way to get a web interface like that on
autobuild.buildroot.org, for the locally created builds? This would
make it easy to inspect the results while testing (for example to see
that some host package like 'bzr' needed to be installed, ...).

Best regards,
Thomas

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Tue, 17 Jun 2014 12:24:53 +0200, Thomas De Schampheleire wrote:

> First of all thanks a lot for making this available!

And thanks a lot to you for being apparently the first person to test
the script. Again, please be tolerant with my horrible Python skills
exposed throughout the script :)

> I am currently testing the script, and hope to get permission to run
> it during the night on one or more work machines.
> I had to make one modification currently: since we are behind a
> firewall, running 'git' directly is not possible, but I need to use a
> wrapper script. I added this:
> 
> +try:
> +    gitcmd = os.environ['GIT']
> +except KeyError:
> +    gitcmd = 'git'
> 
> at the top of the script, and replaced all direct occurrences of 'git'
> with gitcmd.
> This allows me to run 'GIT=git-wrapper autobuild-run'.
> 
> I used an environment variable because it seemed superfluous to add a
> configuration variable for it, but if you prefer I can do that too.

I don't have a very strong preference here. Perhaps having it defined
in the configuration file would be better, as it is more similar to
what we do for the other configuration parameters?

Do not hesitate to send me patches, the Git repository is at
git://git.busybox.net/buildroot-test.

> Is there an easy way to get a web interface like that on
> autobuild.buildroot.org, for the locally created builds? This would
> make it easy to inspect the results while testing (for example to see
> that some host package like 'bzr' needed to be installed, ...).

Well, the autobuild.buildroot.org interface is itself available in
git://git.busybox.net/buildroot-test. However, setting it up will
require a bit of configuration (PHP, MySQL + one database). You could
then submit your builds to your own autobuilder instance for testing,
and then decide to switch to the main autobuilder instance at
autobuild.buildroot.org once you believe things are stable enough.

Thanks again for your feedback!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Thomas De Schampheleire]

Hi Thomas,

On Tue, Jun 17, 2014 at 1:00 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
[..]
>>
>> I used an environment variable because it seemed superfluous to add a
>> configuration variable for it, but if you prefer I can do that too.
>
> I don't have a very strong preference here. Perhaps having it defined
> in the configuration file would be better, as it is more similar to
> what we do for the other configuration parameters?
>
> Do not hesitate to send me patches, the Git repository is at
> git://git.busybox.net/buildroot-test.

Sure, once I tested the script some more I will send you patches.

>
>> Is there an easy way to get a web interface like that on
>> autobuild.buildroot.org, for the locally created builds? This would
>> make it easy to inspect the results while testing (for example to see
>> that some host package like 'bzr' needed to be installed, ...).
>
> Well, the autobuild.buildroot.org interface is itself available in
> git://git.busybox.net/buildroot-test. However, setting it up will
> require a bit of configuration (PHP, MySQL + one database). You could
> then submit your builds to your own autobuilder instance for testing,
> and then decide to switch to the main autobuilder instance at
> autobuild.buildroot.org once you believe things are stable enough.

For my current needs it's a bit overkill to set it all up.
With the following simple script I can get a quick overview of status:

for i in *.tar.bz2; do echo -n "$i: "; tar xf $i results/status -O ; echo ; done

And following script shows the build-end.log files for each run:

for i in *.tar.bz2; do echo "$i:"; tar xf $i results/build-end.log -O
; echo ; done


I have been running one instance between 11am yesterday and 9am today,
and here are some observations:

0. A total of 25 result tarballs have been generated during this time
(22 hours). There was one timeout (4 hours) due to a hanging git
download (see below) and one failure to save a defconfig.
Many builds are under the 30 min range, some lasted for about 2 hours.
This machine was running just one instance with jlevel 4 (just testing).
The machine has 8 Intel i7 cores @ 3.4GHz. I have another such machine
at my disposal.
I also have 2 machines available with each 4 Intel i7 cores @ 2.8 GHz.
Note that I cannot make any promises yet.


1. One run showed the following error, any ideas?

[Tue, 17 Jun 2014 21:39:33] INFO: preparing a new build
[Tue, 17 Jun 2014 21:39:33] INFO: removing cmake-2.8.12.2.tar.gz from downloads
[Tue, 17 Jun 2014 21:39:33] INFO: removing sam-ba_2.12_patch5.gz from downloads
[Tue, 17 Jun 2014 21:39:33] INFO: removing m4-1.4.17.tar.xz from downloads
[Tue, 17 Jun 2014 21:39:33] INFO: removing libcdio-0.92.tar.gz from downloads
[Tue, 17 Jun 2014 21:39:33] INFO: removing wsapi-1.6.1-1.src.rock from downloads
Already up-to-date.
[Tue, 17 Jun 2014 21:42:42] INFO: generate the configuration
[Tue, 17 Jun 2014 21:42:45] ERROR: cannot savedefconfig


2. As this machine is behind a git-blocking firewall, I will also have
to set the 'gitp' wrapper script as BR2_GIT. I think the best way is
to allow to set arbitrary settings from the autobuild config file. Of
course these shouldn't be used to change the build itself, as it could
render the problem irreproducible for others, but if we worry about
such abuse we could add this info into the results tarball.
I will add such an option (for the first part that is) and see if it works well.

In fact, there was one run that timed out, while it was doing a shallow clone.


3. If I get permission to run this officially, I will run it outside
working hours only. This means that I want to set a start time and end
time of the script. Ideally, the script is aware of this, so that I do
not have to interrupt a running build (and lose the results) when
stopping. I imagine a check when starting a new build whether we are
still within the defined time interval and stop if needed.
The starting of the script could be done with a simple cron job and
does not need to be taken into account by the script I think. It could
of course, in which case the script would sleep until it can start
again.
If you have a preference, let me know.


Best regards,
Thomas

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Wed, 18 Jun 2014 09:31:07 +0200, Thomas De Schampheleire wrote:

> > Do not hesitate to send me patches, the Git repository is at
> > git://git.busybox.net/buildroot-test.
> 
> Sure, once I tested the script some more I will send you patches.

Great. And thanks a lot again for working on this, very useful!

> For my current needs it's a bit overkill to set it all up.

Yeah, I guessed so.

> With the following simple script I can get a quick overview of status:
> 
> for i in *.tar.bz2; do echo -n "$i: "; tar xf $i results/status -O ; echo ; done
> 
> And following script shows the build-end.log files for each run:
> 
> for i in *.tar.bz2; do echo "$i:"; tar xf $i results/build-end.log -O
> ; echo ; done

Maybe we could have an option of autobuild-run that is not used to
start the builds, but dump the results of the available local build
results? Maybe autobuild-run --show-results or something similar.

> I have been running one instance between 11am yesterday and 9am today,
> and here are some observations:
> 
> 0. A total of 25 result tarballs have been generated during this time
> (22 hours). There was one timeout (4 hours) due to a hanging git
> download (see below) and one failure to save a defconfig.

Ok.

> Many builds are under the 30 min range, some lasted for about 2 hours.

There is a randomization of the KCONFIG_PROBABILITY, which means that
builds can have between 1% and 30% of the packages enabled. So a wide
variation in the build duration is not surprising.

> This machine was running just one instance with jlevel 4 (just testing).
> The machine has 8 Intel i7 cores @ 3.4GHz. I have another such machine
> at my disposal.
> I also have 2 machines available with each 4 Intel i7 cores @ 2.8 GHz.
> Note that I cannot make any promises yet.

Sure, but if that happens, it would be awesome :)

> 1. One run showed the following error, any ideas?
> 
> [Tue, 17 Jun 2014 21:39:33] INFO: preparing a new build
> [Tue, 17 Jun 2014 21:39:33] INFO: removing cmake-2.8.12.2.tar.gz from downloads
> [Tue, 17 Jun 2014 21:39:33] INFO: removing sam-ba_2.12_patch5.gz from downloads
> [Tue, 17 Jun 2014 21:39:33] INFO: removing m4-1.4.17.tar.xz from downloads
> [Tue, 17 Jun 2014 21:39:33] INFO: removing libcdio-0.92.tar.gz from downloads
> [Tue, 17 Jun 2014 21:39:33] INFO: removing wsapi-1.6.1-1.src.rock from downloads
> Already up-to-date.
> [Tue, 17 Jun 2014 21:42:42] INFO: generate the configuration
> [Tue, 17 Jun 2014 21:42:45] ERROR: cannot savedefconfig

No, not at this point. I guess I should redirect more of the output to
the logfile instead of /dev/null. You can try the following patch to
store in the logfile the output of the savedefconfig:

diff --git a/scripts/autobuild-run b/scripts/autobuild-run
index 06fb430..7dba760 100755
--- a/scripts/autobuild-run
+++ b/scripts/autobuild-run
@@ -302,7 +302,7 @@ def gen_config(instance, njobs, log):
         return -1
 
     ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"],
-                          stdout=devnull, stderr=devnull)
+                          stdout=log, stderr=log)
     if ret != 0:
         log_write(log, "ERROR: cannot savedefconfig")
         return -1

> 2. As this machine is behind a git-blocking firewall, I will also have
> to set the 'gitp' wrapper script as BR2_GIT. I think the best way is
> to allow to set arbitrary settings from the autobuild config file. Of
> course these shouldn't be used to change the build itself, as it could
> render the problem irreproducible for others, but if we worry about
> such abuse we could add this info into the results tarball.
> I will add such an option (for the first part that is) and see if it works well.

I'd prefer to not have the custom BR2_GIT defined inside the
configuration file, because it means that people downloading that
configuration file and trying to reproduce the build with not be able
to do so. This problem already exists today for BR2_JLEVEL and
BR2_DL_DIR which are both defined inside the configuration file, and
it's part of my TODO list (visible at the beginning of the script) to
pass them through the environment instead of through the .config file.

I believe the same should be done for your BR2_GIT. We have to see how
to allow such configuration to be done through the configuration file.
Maybe:

[environment-variables]
BR2_GIT = ...
BR2_FOO = ...

and then the script makes sure to pass all these variables in the
environment when calling make on Buildroot.

> 3. If I get permission to run this officially, I will run it outside
> working hours only. This means that I want to set a start time and end
> time of the script. Ideally, the script is aware of this, so that I do
> not have to interrupt a running build (and lose the results) when
> stopping. I imagine a check when starting a new build whether we are
> still within the defined time interval and stop if needed.
> The starting of the script could be done with a simple cron job and
> does not need to be taken into account by the script I think. It could
> of course, in which case the script would sleep until it can start
> again.
> If you have a preference, let me know.

Wouldn't it be simpler to simply kill the build process when the time
happens? So you have a cronjob that starts autobuild-run at 7 PM or 8
PM in the evening, and another one that kills autobuild-run at 7 AM or
8 AM. I've tried to handle killing only the main autobuild-run script
and make sure it kills the building sub-processes, but it doesn't work
completely well so far. It needs to be investigated.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Thomas De Schampheleire]

Hi Thomas,

On Wed, Jun 18, 2014 at 10:25 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
[..]
>
>> With the following simple script I can get a quick overview of status:
>>
>> for i in *.tar.bz2; do echo -n "$i: "; tar xf $i results/status -O ; echo ; done
>>
>> And following script shows the build-end.log files for each run:
>>
>> for i in *.tar.bz2; do echo "$i:"; tar xf $i results/build-end.log -O
>> ; echo ; done
>
> Maybe we could have an option of autobuild-run that is not used to
> start the builds, but dump the results of the available local build
> results? Maybe autobuild-run --show-results or something similar.

Yes, that's a good idea.

[..]
>
>> 1. One run showed the following error, any ideas?
>>
>> [Tue, 17 Jun 2014 21:39:33] INFO: preparing a new build
>> [Tue, 17 Jun 2014 21:39:33] INFO: removing cmake-2.8.12.2.tar.gz from downloads
>> [Tue, 17 Jun 2014 21:39:33] INFO: removing sam-ba_2.12_patch5.gz from downloads
>> [Tue, 17 Jun 2014 21:39:33] INFO: removing m4-1.4.17.tar.xz from downloads
>> [Tue, 17 Jun 2014 21:39:33] INFO: removing libcdio-0.92.tar.gz from downloads
>> [Tue, 17 Jun 2014 21:39:33] INFO: removing wsapi-1.6.1-1.src.rock from downloads
>> Already up-to-date.
>> [Tue, 17 Jun 2014 21:42:42] INFO: generate the configuration
>> [Tue, 17 Jun 2014 21:42:45] ERROR: cannot savedefconfig
>
> No, not at this point. I guess I should redirect more of the output to
> the logfile instead of /dev/null. You can try the following patch to
> store in the logfile the output of the savedefconfig:
>
> diff --git a/scripts/autobuild-run b/scripts/autobuild-run
> index 06fb430..7dba760 100755
> --- a/scripts/autobuild-run
> +++ b/scripts/autobuild-run
> @@ -302,7 +302,7 @@ def gen_config(instance, njobs, log):
>          return -1
>
>      ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"],
> -                          stdout=devnull, stderr=devnull)
> +                          stdout=log, stderr=log)
>      if ret != 0:
>          log_write(log, "ERROR: cannot savedefconfig")
>          return -1
>

Ok, I will add that and monitor this type of failures.

>> 2. As this machine is behind a git-blocking firewall, I will also have
>> to set the 'gitp' wrapper script as BR2_GIT. I think the best way is
>> to allow to set arbitrary settings from the autobuild config file. Of
>> course these shouldn't be used to change the build itself, as it could
>> render the problem irreproducible for others, but if we worry about
>> such abuse we could add this info into the results tarball.
>> I will add such an option (for the first part that is) and see if it works well.
>
> I'd prefer to not have the custom BR2_GIT defined inside the
> configuration file, because it means that people downloading that
> configuration file and trying to reproduce the build with not be able
> to do so. This problem already exists today for BR2_JLEVEL and
> BR2_DL_DIR which are both defined inside the configuration file, and
> it's part of my TODO list (visible at the beginning of the script) to
> pass them through the environment instead of through the .config file.
>
> I believe the same should be done for your BR2_GIT. We have to see how
> to allow such configuration to be done through the configuration file.
> Maybe:
>
> [environment-variables]
> BR2_GIT = ...
> BR2_FOO = ...
>
> and then the script makes sure to pass all these variables in the
> environment when calling make on Buildroot.

Ok, this is indeed better.
It's probably easier (for parsing) to let the user specify one
'environment' string that contains
"BR2_GIT=xxx BR2_FOO=yyy"
than having separate config settings for possibly unknown options.

>
>> 3. If I get permission to run this officially, I will run it outside
>> working hours only. This means that I want to set a start time and end
>> time of the script. Ideally, the script is aware of this, so that I do
>> not have to interrupt a running build (and lose the results) when
>> stopping. I imagine a check when starting a new build whether we are
>> still within the defined time interval and stop if needed.
>> The starting of the script could be done with a simple cron job and
>> does not need to be taken into account by the script I think. It could
>> of course, in which case the script would sleep until it can start
>> again.
>> If you have a preference, let me know.
>
> Wouldn't it be simpler to simply kill the build process when the time
> happens? So you have a cronjob that starts autobuild-run at 7 PM or 8
> PM in the evening, and another one that kills autobuild-run at 7 AM or
> 8 AM. I've tried to handle killing only the main autobuild-run script
> and make sure it kills the building sub-processes, but it doesn't work
> completely well so far. It needs to be investigated.

It's easier, but it has the downside that the ongoing build at the
time of the kill is always lost (no results available). In the case it
happens to be a long build of two hours, it's a pity to lose these
results if the 'stop-time' is not very strict.

Best regards,
Thomas

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Wed, 18 Jun 2014 10:47:48 +0200, Thomas De Schampheleire wrote:


> > [environment-variables]
> > BR2_GIT = ...
> > BR2_FOO = ...
> >
> > and then the script makes sure to pass all these variables in the
> > environment when calling make on Buildroot.
> 
> Ok, this is indeed better.
> It's probably easier (for parsing) to let the user specify one
> 'environment' string that contains
> "BR2_GIT=xxx BR2_FOO=yyy"
> than having separate config settings for possibly unknown options.

Right, even better.

> > Wouldn't it be simpler to simply kill the build process when the time
> > happens? So you have a cronjob that starts autobuild-run at 7 PM or 8
> > PM in the evening, and another one that kills autobuild-run at 7 AM or
> > 8 AM. I've tried to handle killing only the main autobuild-run script
> > and make sure it kills the building sub-processes, but it doesn't work
> > completely well so far. It needs to be investigated.
> 
> It's easier, but it has the downside that the ongoing build at the
> time of the kill is always lost (no results available). In the case it
> happens to be a long build of two hours, it's a pity to lose these
> results if the 'stop-time' is not very strict.

That's true, but I find this thing really use-case specific. For
example, maybe on week-ends you will be able to let the builds run even
during day time? So we need to make the script more complex to handle
this possibility, while cron is the perfect tool to do this kind of
thing.

*However* we could add something like a signal sent to the script that
tells it to stop the builds when they are finished. Either a Unix
signal like SIGUSR1, or the creation of a specific file in a specific
location, or something like that. When the script receives this signal,
it knows it should stop itself at the end of the current build. But the
mechanism to generate this signal (cronjob or whatever) remains
external to the script.

What do you think?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Thomas De Schampheleire]

Hi Thomas,

On Wed, Jun 18, 2014 at 10:51 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
[..]
>> > Wouldn't it be simpler to simply kill the build process when the time
>> > happens? So you have a cronjob that starts autobuild-run at 7 PM or 8
>> > PM in the evening, and another one that kills autobuild-run at 7 AM or
>> > 8 AM. I've tried to handle killing only the main autobuild-run script
>> > and make sure it kills the building sub-processes, but it doesn't work
>> > completely well so far. It needs to be investigated.
>>
>> It's easier, but it has the downside that the ongoing build at the
>> time of the kill is always lost (no results available). In the case it
>> happens to be a long build of two hours, it's a pity to lose these
>> results if the 'stop-time' is not very strict.
>
> That's true, but I find this thing really use-case specific. For
> example, maybe on week-ends you will be able to let the builds run even
> during day time? So we need to make the script more complex to handle
> this possibility, while cron is the perfect tool to do this kind of
> thing.
>
> *However* we could add something like a signal sent to the script that
> tells it to stop the builds when they are finished. Either a Unix
> signal like SIGUSR1, or the creation of a specific file in a specific
> location, or something like that. When the script receives this signal,
> it knows it should stop itself at the end of the current build. But the
> mechanism to generate this signal (cronjob or whatever) remains
> external to the script.
>
> What do you think?

Yes, that's a nice and clean solution.
A signal like SIGUSR1 seems easy enough to implement in Python, and
avoids the problem of removing the magic file afterwards.
I'll look at that when I have time.

Thanks,
Thomas

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Thomas De Schampheleire]

Hi Thomas,

On Wed, Jun 18, 2014 at 10:25 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
[..]
>
>> 1. One run showed the following error, any ideas?
>>
>> [Tue, 17 Jun 2014 21:39:33] INFO: preparing a new build
>> [Tue, 17 Jun 2014 21:39:33] INFO: removing cmake-2.8.12.2.tar.gz from downloads
>> [Tue, 17 Jun 2014 21:39:33] INFO: removing sam-ba_2.12_patch5.gz from downloads
>> [Tue, 17 Jun 2014 21:39:33] INFO: removing m4-1.4.17.tar.xz from downloads
>> [Tue, 17 Jun 2014 21:39:33] INFO: removing libcdio-0.92.tar.gz from downloads
>> [Tue, 17 Jun 2014 21:39:33] INFO: removing wsapi-1.6.1-1.src.rock from downloads
>> Already up-to-date.
>> [Tue, 17 Jun 2014 21:42:42] INFO: generate the configuration
>> [Tue, 17 Jun 2014 21:42:45] ERROR: cannot savedefconfig
>
> No, not at this point. I guess I should redirect more of the output to
> the logfile instead of /dev/null. You can try the following patch to
> store in the logfile the output of the savedefconfig:
>
> diff --git a/scripts/autobuild-run b/scripts/autobuild-run
> index 06fb430..7dba760 100755
> --- a/scripts/autobuild-run
> +++ b/scripts/autobuild-run
> @@ -302,7 +302,7 @@ def gen_config(instance, njobs, log):
>          return -1
>
>      ret = subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "savedefconfig"],
> -                          stdout=devnull, stderr=devnull)
> +                          stdout=log, stderr=log)
>      if ret != 0:
>          log_write(log, "ERROR: cannot savedefconfig")
>          return -1
>

In the mean time another such issue popped up, here is the output:

make: Entering directory `/repo/tdescham/autobuild/workdir/instance-0/buildroot'
package/luajit/luajit.mk:67: *** Configuration error: both "luajit"
and "lua" are selected as providers for virtual package
"luainterpreter". Only one provider can be selected at a time. Please
fix your configuration.  Stop.
make: Leaving directory `/repo/tdescham/autobuild/workdir/instance-0/buildroot'
[Wed, 18 Jun 2014 13:42:43] ERROR: cannot savedefconfig


We already talked about such problems before, did you already have a
solution implemented in the original autobuild instance?

Best regards,
Thomas

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Thomas Petazzoni]

Dear Thomas De Schampheleire,

On Wed, 18 Jun 2014 13:51:27 +0200, Thomas De Schampheleire wrote:

> In the mean time another such issue popped up, here is the output:
> 
> make: Entering directory `/repo/tdescham/autobuild/workdir/instance-0/buildroot'
> package/luajit/luajit.mk:67: *** Configuration error: both "luajit"
> and "lua" are selected as providers for virtual package
> "luainterpreter". Only one provider can be selected at a time. Please
> fix your configuration.  Stop.
> make: Leaving directory `/repo/tdescham/autobuild/workdir/instance-0/buildroot'
> [Wed, 18 Jun 2014 13:42:43] ERROR: cannot savedefconfig

Weird, why is this error happening at "savedefconfig" time ?

> We already talked about such problems before, did you already have a
> solution implemented in the original autobuild instance?

No, I did not implement anything like that in the original autobuild
instance, so I'm wondering why I don't see the problem. Maybe it gets
ignored by improper error checking or something.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Matthew Weber]

Thomas, wanted to thank you for making this available.  Here's some
initial feedback.

On Sat, Jun 14, 2014 at 2:23 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
<snip>
>
> Getting started
> ===============
>
>
> To get started, you can simply run it with:
>
>   ./autobuild-run --ninstances 2 --njobs 3
>
I seem to get the following output.  It looks like the build is right
before the spot where it is doing the initial "make
O=./instance-#/output -C ./buildroot".  If I manually go into the
instance#/output folder and do a "make", things seem to look like
they're working as expected (building packages/downloading archives,
etc).  So my theory is something in my shell environment or the fact
I'm using python2.6 with that subprocess.....  One thing I noticed
when I got this error was that the resulting logfile in the output
folder is empty.  I added a test write to this file before line 327
and I saw what I wrote.  So it looks like the subprocess isn't getting
called successfully.  Since I'm going to run the script on a newer
server that has python2.7, this email is probably just a FYI and I'll
retest once I'm setup to test on that server.

Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "./autobuild-run", line 437, in run_instance
    ret = do_build(instance, instance_log)
  File "./autobuild-run", line 327, in do_build
    stdout=f, stderr=f)
  File "/usr/lib/python2.6/subprocess.py", line 480, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
    errread, errwrite)
  File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

>
>
<snip>

Thanks!
-- 
Matthew L Weber / Sr Software Engineer / Platform Software
MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA
Phone: 319-295-7349
matthew.weber at rockwellcollins.com
www.rockwellcollins.com

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Thomas Petazzoni]

Dear Matthew Weber,

Thanks for your report!

On Fri, 20 Jun 2014 16:03:34 -0500, Matthew Weber wrote:

> I seem to get the following output.  It looks like the build is right
> before the spot where it is doing the initial "make
> O=./instance-#/output -C ./buildroot".  If I manually go into the
> instance#/output folder and do a "make", things seem to look like
> they're working as expected (building packages/downloading archives,
> etc).  So my theory is something in my shell environment or the fact
> I'm using python2.6 with that subprocess.....  One thing I noticed
> when I got this error was that the resulting logfile in the output
> folder is empty.  I added a test write to this file before line 327
> and I saw what I wrote.  So it looks like the subprocess isn't getting
> called successfully.  Since I'm going to run the script on a newer
> server that has python2.7, this email is probably just a FYI and I'll
> retest once I'm setup to test on that server.

The builds on the Free Electrons build server are running inside an old
Debian chroot which also uses Python 2.6 (2.6.6 to be exact), and it's
working fine.

> Process Process-1:
> Traceback (most recent call last):
>   File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
>     self.run()
>   File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
>     self._target(*self._args, **self._kwargs)
>   File "./autobuild-run", line 437, in run_instance
>     ret = do_build(instance, instance_log)
>   File "./autobuild-run", line 327, in do_build
>     stdout=f, stderr=f)

Are you sure you're using the latest version of the script? The code at
lines 327 and 437 do not match what your exception stack is showing.
But I'm a bit surprised, because I don't think I've made any changes to
the script since I made the public announcement on the Buildroot
mailing list.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [ANNOUNCE] Autobuilder script available, join the autobuilder effort!

[Matthew Weber]

Dear Thomas,

On Jun 21, 2014 12:21 PM, "Thomas Petazzoni" <
thomas.petazzoni@free-electrons.com> wrote:
>
> Dear Matthew Weber,
>
> Thanks for your report!
>
> On Fri, 20 Jun 2014 16:03:34 -0500, Matthew Weber wrote:
>
> > I seem to get the following output.  It looks like the build is right
> > before the spot where it is doing the initial "make
> > O=./instance-#/output -C ./buildroot".  If I manually go into the
> > instance#/output folder and do a "make", things seem to look like
> > they're working as expected (building packages/downloading archives,
> > etc).  So my theory is something in my shell environment or the fact
> > I'm using python2.6 with that subprocess.....  One thing I noticemo
> > when I got this error was that the resulting logfile in the output
> > folder is empty.  I added a test write to this file before line 327
> > and I saw what I wrote.  So it looks like the subprocess isn't getting
> > called successfully.  Since I'm going to run the script on a newer
> > server that has python2.7, this email is probably just a FYI and I'll
> > retest once I'm setup to test on that server.
>
> The builds on the Free Electrons build server are running inside an old
> Debian chroot which also uses Python 2.6 (2.6.6 to be exact), and it's
> working fine.
Good to know, I'll take another look at my environment.  I'm running sick
ubuntu10.04.

>
> > Process Process-1:
> > Traceback (most recent call last):
> >   File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
> >     self.run()
> >   File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
> >     self._target(*self._args, **self._kwargs)
> >   File "./autobuild-run", line 437, in run_instance
> >     ret = do_build(instance, instance_log)
> >   File "./autobuild-run", line 327, in do_build
> >     stdout=f, stderr=f)
>
> Are you sure you're using the latest version of the script? The code at
> lines 327 and 437 do not match what your exception stack is showing.
> But I'm a bit surprised, because I don't think I've made any changes to
> the script since I made the public announcement on the Buildroot
> mailing list.
Sorry about that, had to adjust the url file retrieval lines because my
proxy was blocking the autobuilder site.  I left two commented out lines.
I put in a request so that I can use the script as is without any proxy
work around.  I'll respond once I re-test .

Thanks!
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140622/87dbac89/attachment.html>