Re: nightly-release takes more than 24 hours to build.

[Richard Purdie <rpurdie@linux.intel.com>]

On Mon, 2010-11-01 at 02:38 -0700, Scott Garman wrote:
> Leading up to our 0.9 release, our autobuilder has been building an 
> increasing number of targets for our nightly-release buildset. We've now 
> reached the point that the nightly build takes more than 24 hours to run 
> (> 26 hours, in fact) - which is clearly a problem on a build that we'd 
> like to generate on a daily basis.
> 
> The following is a list of everything which is built within nightly-release:
> 
> The following targets are built for qemux86, qemux86-64, qemuarm, 
> qemumips, and qemuppc:
> 
> * poky-image-minimal
> * poky-image-sato
> * poky-image-lsb
> * poky-image-sdk
> * meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)
> 
> For emenlow and atom-pc, we build:
> 
> * poky-image-minimal-live
> * poky-image-sato-live
> * poky-image-sdk-live
> * meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)
> 
> Finally, we also build the Eclipse plugin, and copy the shared state 
> prebuilds and RPM output at the end of the build.
> 
> I was going to post build times for some of these targets for reference, 
> but it would be misleading as we build the targets in succession (e.g, 
> we start with poky-image-sdk which takes the bulk of the time, and then 
> the other targets can largely rely on the shared state builds).
> 
> Ideally I think our nightly build should take much less than 24 hours to 
> build. The question is what we can move out of the nightly build and do 
> on perhaps a weekly basis instead?
> 
> Our buildserver hardware is a dual quad-core Xeon server with 12 GB of 
> RAM. Throwing hardware at the problem is another solution, but not an 
> inexpensive one (we'd be looking at a 4-socket machine filled with 
> quad-cores and 32 GB of RAM).

There doesn't just have to be one build machine, we are going to end up
needing multiple machines and we can split the load between them quite
easily. I think there is going to be a second machine needed alongside
the existing one regardless of what other optimisations we make.

The changes in development at the moment will have a mixed effect on the
autobuilder's load. On the plus side we know there are performance
regressions with 0.9 which we're about to investigate. I can think of
one change I have in mind which on its own should get the builds back
under the 24 hour time.

On the down side there are going to be changes that need increased
horsepower from the build machines such as the runtime testing we're
close to enabling or enabling builds of world.

Cheers,

Richard