nightly-release takes more than 24 hours to build.

nightly-release takes more than 24 hours to build.

[Scott Garman]

Hello,

Leading up to our 0.9 release, our autobuilder has been building an 
increasing number of targets for our nightly-release buildset. We've now 
reached the point that the nightly build takes more than 24 hours to run 
(> 26 hours, in fact) - which is clearly a problem on a build that we'd 
like to generate on a daily basis.

The following is a list of everything which is built within nightly-release:

The following targets are built for qemux86, qemux86-64, qemuarm, 
qemumips, and qemuppc:

* poky-image-minimal
* poky-image-sato
* poky-image-lsb
* poky-image-sdk
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

For emenlow and atom-pc, we build:

* poky-image-minimal-live
* poky-image-sato-live
* poky-image-sdk-live
* meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)

Finally, we also build the Eclipse plugin, and copy the shared state 
prebuilds and RPM output at the end of the build.

I was going to post build times for some of these targets for reference, 
but it would be misleading as we build the targets in succession (e.g, 
we start with poky-image-sdk which takes the bulk of the time, and then 
the other targets can largely rely on the shared state builds).

Ideally I think our nightly build should take much less than 24 hours to 
build. The question is what we can move out of the nightly build and do 
on perhaps a weekly basis instead?

Our buildserver hardware is a dual quad-core Xeon server with 12 GB of 
RAM. Throwing hardware at the problem is another solution, but not an 
inexpensive one (we'd be looking at a 4-socket machine filled with 
quad-cores and 32 GB of RAM).

I'm open to ideas on how to address this issue. QA will be driving a lot 
of the requirements and I'm especially interested to hear your thoughts.

Scott

Re: nightly-release takes more than 24 hours to build.

[Richard Purdie]

On Mon, 2010-11-01 at 02:38 -0700, Scott Garman wrote:
> Leading up to our 0.9 release, our autobuilder has been building an 
> increasing number of targets for our nightly-release buildset. We've now 
> reached the point that the nightly build takes more than 24 hours to run 
> (> 26 hours, in fact) - which is clearly a problem on a build that we'd 
> like to generate on a daily basis.
> 
> The following is a list of everything which is built within nightly-release:
> 
> The following targets are built for qemux86, qemux86-64, qemuarm, 
> qemumips, and qemuppc:
> 
> * poky-image-minimal
> * poky-image-sato
> * poky-image-lsb
> * poky-image-sdk
> * meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)
> 
> For emenlow and atom-pc, we build:
> 
> * poky-image-minimal-live
> * poky-image-sato-live
> * poky-image-sdk-live
> * meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)
> 
> Finally, we also build the Eclipse plugin, and copy the shared state 
> prebuilds and RPM output at the end of the build.
> 
> I was going to post build times for some of these targets for reference, 
> but it would be misleading as we build the targets in succession (e.g, 
> we start with poky-image-sdk which takes the bulk of the time, and then 
> the other targets can largely rely on the shared state builds).
> 
> Ideally I think our nightly build should take much less than 24 hours to 
> build. The question is what we can move out of the nightly build and do 
> on perhaps a weekly basis instead?
> 
> Our buildserver hardware is a dual quad-core Xeon server with 12 GB of 
> RAM. Throwing hardware at the problem is another solution, but not an 
> inexpensive one (we'd be looking at a 4-socket machine filled with 
> quad-cores and 32 GB of RAM).

There doesn't just have to be one build machine, we are going to end up
needing multiple machines and we can split the load between them quite
easily. I think there is going to be a second machine needed alongside
the existing one regardless of what other optimisations we make.

The changes in development at the moment will have a mixed effect on the
autobuilder's load. On the plus side we know there are performance
regressions with 0.9 which we're about to investigate. I can think of
one change I have in mind which on its own should get the builds back
under the 24 hour time.

On the down side there are going to be changes that need increased
horsepower from the build machines such as the runtime testing we're
close to enabling or enabling builds of world.

Cheers,

Richard

Re: nightly-release takes more than 24 hours to build.

[Stewart, David C]

>From: yocto-bounces@yoctoproject.org [mailto:yocto-
>bounces@yoctoproject.org] On Behalf Of Richard Purdie
>Sent: Monday, November 01, 2010 4:03 AM
>
>On Mon, 2010-11-01 at 02:38 -0700, Scott Garman wrote:
>>
>> Our buildserver hardware is a dual quad-core Xeon server with 12 GB of
>> RAM. Throwing hardware at the problem is another solution, but not an
>> inexpensive one (we'd be looking at a 4-socket machine filled with
>> quad-cores and 32 GB of RAM).
>
>There doesn't just have to be one build machine, we are going to end up
>needing multiple machines and we can split the load between them quite
>easily. I think there is going to be a second machine needed alongside
>the existing one regardless of what other optimisations we make.

I can buy another server and contribute it to the build effort. I had intended to buy one this quarter to begin hosting yoctoproject.org and a source mirror, but we could offload part of the builds to it as well.

As you know, I want us to get to the root of the problem, which is efficiency of the build process. Since RP has already committed to addressing this, I'm fine with a short-term fix if it will help. :-)

Scott - please put in an order, will tell you the max amount today when we're f2f.

Dave

Re: nightly-release takes more than 24 hours to build.

[Xu, Jiajun]

> Hello,
> 
> Leading up to our 0.9 release, our autobuilder has been building an
> increasing number of targets for our nightly-release buildset. We've
> now reached the point that the nightly build takes more than 24 hours
> to run (> 26 hours, in fact)
> - which is clearly a problem on a build that we'd like to generate on
> a daily basis.
> 
> The following is a list of everything which is built within nightly-release:
> 
> The following targets are built for qemux86, qemux86-64, qemuarm,
> qemumips, and qemuppc:
> 
> * poky-image-minimal
> * poky-image-sato
> * poky-image-lsb
> * poky-image-sdk
> * meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)
> 
> For emenlow and atom-pc, we build:
> 
> * poky-image-minimal-live
> * poky-image-sato-live
> * poky-image-sdk-live
> * meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)
> 
> Finally, we also build the Eclipse plugin, and copy the shared state
> prebuilds and RPM output at the end of the build.
> 
> I was going to post build times for some of these targets for
> reference, but it would be misleading as we build the targets in
> succession (e.g, we start with poky-image-sdk which takes the bulk of
> the time, and then the other targets can largely rely on the shared state builds).
> 
> Ideally I think our nightly build should take much less than 24 hours
> to build. The question is what we can move out of the nightly build
> and do on perhaps a weekly basis instead?
> 

How about customizing the nightly build to build part of targets for every day? For example, x86/x86_64 for Monday, atom-pc/emenlow for Tuesday..... etc.
And we can schedule a new build task for weekly, which is only triggered on Wednesday(or maybe Tuesday night) for QA weekly testing and with all targets built out.

> Our buildserver hardware is a dual quad-core Xeon server with 12 GB of RAM.
> Throwing hardware at the problem is another solution, but not an
> inexpensive one (we'd be looking at a 4-socket machine filled with
> quad-cores and 32 GB of RAM).
> 
> I'm open to ideas on how to address this issue. QA will be driving a
> lot of the requirements and I'm especially interested to hear your thoughts.
> 
> Scott
> 
> 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.pokylinux.org/listinfo/yocto

Best Regards,
Jiajun

Re: nightly-release takes more than 24 hours to build.

[Tian, Kevin]

[-- Attachment #1: Type: text/plain, Size: 2962 bytes --]

>From: Richard Purdie
>Sent: Monday, November 01, 2010 7:03 PM
>
>On Mon, 2010-11-01 at 02:38 -0700, Scott Garman wrote:
>> Leading up to our 0.9 release, our autobuilder has been building an
>> increasing number of targets for our nightly-release buildset. We've now
>> reached the point that the nightly build takes more than 24 hours to run
>> (> 26 hours, in fact) - which is clearly a problem on a build that we'd
>> like to generate on a daily basis.
>>
>> The following is a list of everything which is built within nightly-release:
>>
>> The following targets are built for qemux86, qemux86-64, qemuarm,
>> qemumips, and qemuppc:
>>
>> * poky-image-minimal
>> * poky-image-sato
>> * poky-image-lsb
>> * poky-image-sdk
>> * meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)
>>
>> For emenlow and atom-pc, we build:
>>
>> * poky-image-minimal-live
>> * poky-image-sato-live
>> * poky-image-sdk-live
>> * meta-toolchain-sdk (SDKMACHINE=i586 and also x86_64)
>>
>> Finally, we also build the Eclipse plugin, and copy the shared state
>> prebuilds and RPM output at the end of the build.
>>
>> I was going to post build times for some of these targets for reference,
>> but it would be misleading as we build the targets in succession (e.g,
>> we start with poky-image-sdk which takes the bulk of the time, and then
>> the other targets can largely rely on the shared state builds).
>>
>> Ideally I think our nightly build should take much less than 24 hours to
>> build. The question is what we can move out of the nightly build and do
>> on perhaps a weekly basis instead?
>>
>> Our buildserver hardware is a dual quad-core Xeon server with 12 GB of
>> RAM. Throwing hardware at the problem is another solution, but not an
>> inexpensive one (we'd be looking at a 4-socket machine filled with
>> quad-cores and 32 GB of RAM).
>
>There doesn't just have to be one build machine, we are going to end up
>needing multiple machines and we can split the load between them quite
>easily. I think there is going to be a second machine needed alongside
>the existing one regardless of what other optimisations we make.
>
>The changes in development at the moment will have a mixed effect on the
>autobuilder's load. On the plus side we know there are performance
>regressions with 0.9 which we're about to investigate. I can think of
>one change I have in mind which on its own should get the builds back
>under the 24 hour time.

In case you didn't note Qing's investigation last week, attach his mail for
your reference.

Thanks
Kevin

>
>On the down side there are going to be changes that need increased
>horsepower from the build machines such as the runtime testing we're
>close to enabling or enabling builds of world.
>
>Cheers,
>
>Richard
>
>_______________________________________________
>yocto mailing list
>yocto@yoctoproject.org
>https://lists.pokylinux.org/listinfo/yocto

[-- Attachment #2: Type: message/rfc822, Size: 5635 bytes --]

From: "He, Qing" <qing.he@intel.com>
To: "rpurdie@linux.intel.com" <rpurdie@linux.intel.com>
Cc: "yocto@yoctoproject.org" <yocto@yoctoproject.org>
Subject: [yocto] Some data collection and analysis on poky performance
Date: Wed, 27 Oct 2010 17:23:08 +0800
Message-ID: <20101027092308.GA18748@qhe2-db>

As we know, many of us have experienced slow builds of recent poky,
and it also takes larger disk space. This affects user exprience thus
is one of our directions in 1.0.

To find the problems leading to performance issues, I tried some
profiling on poky builds, below is a very brief summary of the data.
I profiled poky-image-minimal of both the current master branch and
green release, with similar parameters (4 CPUs) on NHM. Note that
both rpm and ipk packages are built for current master branch, while
oonly ipk packages are built for green release


I. some stats
1. recipes (including -native)
  green release:
    recipes built:  76
    tasks run:  998
  master:
    recipes built: 133
    tasks run:  1600

2. time
  green release:
    real:  28m7s
    user:  57m45s
    sys:    9m41s
    user+sys: 67m26s
  master:
    real:  66m39s
    user: 152m17s
    sys:   27m37s
    user+sys: 179m54s

3. space (haven't analyze though)
  green release:   7.8G
  master:         26.6G


II. profiling
I tried a brief profile by collecting the time used for every task, so
we can scrutinize the result from a microscopic point of view. I'm still
looking into the full result, but there's something of immediate
attention.

In master, hardly any task consume less than 1.3s, this is quite
surprising, since many tasks like do_patch virtually do nothing, while
in the green release, these tasks may simply consume 0.1~0.2s. A deeper
investigation shows that this large overhead goes to bitbake-runtask,
the bitbake config and cache mechanism is executed for every task,
considerably increased the time. The overhead introduced solely by
this is around 1600*1.3=2080s, approximately 35 minutes (user+sys).

Also, we should count the additional rpm packaging system,
do_package_write_rpm costs around 1400s, (excluding 1.3s per task,
btw, this is about 50% slower than do_package_write_ipk, in average),
that's around 23 minutes.

Roughly considering that the build time is proportional to recipes
count, we can try to estimate master build time from green release:
    67 * (76 / 133) + 35 + 23 = 175
very close to the real time consuming (although somewhat too closed...),
so possibly the above two are the most significant time consumers
in the slowness of current poky

Thanks,
Qing




_______________________________________________
yocto mailing list
yocto@yoctoproject.org
https://lists.pokylinux.org/listinfo/yocto

Re: nightly-release takes more than 24 hours to build.

[Scott Garman]

On 11/01/2010 06:35 AM, Stewart, David C wrote:
> I can buy another server and contribute it to the build effort. I had
> intended to buy one this quarter to begin hosting yoctoproject.org
> and a source mirror, but we could offload part of the builds to it as
> well.

Great, thanks Dave! :)

I have spec'ed out and initiated an order for our newest server. In case 
anyone likes to geek out on hardware specs, here they are:

Dell PowerEdge R710
2x Xeon X5680 @ 3.33Ghz
24 GB RAM
6x 1.5 TB hard drives for use in a RAID5 array

While it's not a 4-processor system, the additional resources should be 
sufficient for us to tackle our resource problems my distributing the 
workload more evenly.

The system will likely arrive and be set up around the end of the month.

Scott