[RFC] Performance Issue: Build time increases

[RFC] Performance Issue: Build time increases

[Robert Yang]

Hi folks,

The build time of core-image-sato increases about 5 ~ 10 minutes than
the following commit:

commit 5af197b55a4b779f1ec93186f0723026949ba2b5
Author: Liping Ke <liping.ke@intel.com>
Date:   Fri Jun 3 08:22:40 2011 +0800

     cache: Implement multiple extra cache fields request support

On my host,
1) the build time of 5af197b55a4b779f1ec93186f0723026949ba2b5 is:

real    208m26.133s
user    241m29.280s
sys     47m0.630s

2) and: 068839698fe192d8846c0ed4db65861448e8e524 is:

real    217m39.687s
user    255m34.150s
sys     48m21.510s


I use the bisect build method to find out which patch causes the time
increases, but the build time is not stable on my host(Ubuntu 11.04 64bit),
e.g., the build time of 1) is 208m at the first build, then "git co 
other_commit" and build it, after about 10 builds, then go back to build
5af197b55a4b779f1ec, the build time will increases about 8 minutes, I have
stopped the X and cron, at. This may have relationship with the linux
distribution and disk.

So I have to restart the build, reboot the machine by two days, and go on the
build. It would be better if anyone has other good method.

I think that for the next release(e.g., yocto 1.2), we can find a clean and
stable machine(for the distribution, maybe RHEL is more stable than Ubuntu)
to check the build time weekly, so that we can notice the performance issue
early.

-- 
Thanks

Robert

Re: [RFC] Performance Issue: Build time increases

[Lu, Lianhao]

Robert Yang wrote on 2011-08-12:
> 
> Hi folks,
> 
> The build time of core-image-sato increases about 5 ~ 10 minutes than
> the following commit:
> 
> commit 5af197b55a4b779f1ec93186f0723026949ba2b5
> Author: Liping Ke <liping.ke@intel.com>
> Date:   Fri Jun 3 08:22:40 2011 +0800
> 
>      cache: Implement multiple extra cache fields request support

I think this patch only stores extra information in an extra cache. It only affects the parsing time, and I don't think there will be 5 min difference. How about comparing the "bitbake -e" time with and without the patch(by deleting all the cache files before running bitbake).

Best Regards,
-Lianhao

> On my host,
> 1) the build time of 5af197b55a4b779f1ec93186f0723026949ba2b5 is:
> 
> real    208m26.133s
> user    241m29.280s
> sys     47m0.630s
> 
> 2) and: 068839698fe192d8846c0ed4db65861448e8e524 is:
> 
> real    217m39.687s
> user    255m34.150s
> sys     48m21.510s
> 
> 
> I use the bisect build method to find out which patch causes the time
> increases, but the build time is not stable on my host(Ubuntu 11.04
> 64bit), e.g., the build time of 1) is 208m at the first build, then
> "git co other_commit" and build it, after about 10 builds, then go
> back to build 5af197b55a4b779f1ec, the build time will increases about
> 8 minutes, I have stopped the X and cron, at. This may have relationship with the linux distribution and disk.
> 
> So I have to restart the build, reboot the machine by two days, and go
> on the build. It would be better if anyone has other good method.
> 
> I think that for the next release(e.g., yocto 1.2), we can find a
> clean and stable machine(for the distribution, maybe RHEL is more
> stable than Ubuntu) to check the build time weekly, so that we can notice the performance issue early.
>

Re: [RFC] Performance Issue: Build time increases

[Richard Purdie]

On Fri, 2011-08-12 at 19:06 +0800, Robert Yang wrote:
> Hi folks,
> 
> The build time of core-image-sato increases about 5 ~ 10 minutes than
> the following commit:
> 
> commit 5af197b55a4b779f1ec93186f0723026949ba2b5
> Author: Liping Ke <liping.ke@intel.com>
> Date:   Fri Jun 3 08:22:40 2011 +0800
> 
>      cache: Implement multiple extra cache fields request support
> 
> On my host,
> 1) the build time of 5af197b55a4b779f1ec93186f0723026949ba2b5 is:
> 
> real    208m26.133s
> user    241m29.280s
> sys     47m0.630s
> 
> 2) and: 068839698fe192d8846c0ed4db65861448e8e524 is:
> 
> real    217m39.687s
> user    255m34.150s
> sys     48m21.510s
> 
> 
> I use the bisect build method to find out which patch causes the time
> increases, but the build time is not stable on my host(Ubuntu 11.04 64bit),
> e.g., the build time of 1) is 208m at the first build, then "git co 
> other_commit" and build it, after about 10 builds, then go back to build
> 5af197b55a4b779f1ec, the build time will increases about 8 minutes, I have
> stopped the X and cron, at. This may have relationship with the linux
> distribution and disk.
> 
> So I have to restart the build, reboot the machine by two days, and go on the
> build. It would be better if anyone has other good method.
> 
> I think that for the next release(e.g., yocto 1.2), we can find a clean and
> stable machine(for the distribution, maybe RHEL is more stable than Ubuntu)
> to check the build time weekly, so that we can notice the performance issue
> early.

I think we really need to get to the bottom of why the build times are
so variable. Do we need to start doing these on a clean distro install?
is a fresh boot good enough? is there a way we can clear out the VM
caches and get reproducible times? reformat the build partition?

If that really is just because of the nature of a chaotic system, can we
get some better representation of build time to use as a benchmark? We
could really do with finding a faster test (which would hopefully still
be representative)...

Cheers,

Richard

Re: [RFC] Performance Issue: Build time increases

[Robert Yang]

On 08/12/2011 08:19 PM, Lu, Lianhao wrote:
> Robert Yang wrote on 2011-08-12:
>>
>> Hi folks,
>>
>> The build time of core-image-sato increases about 5 ~ 10 minutes than
>> the following commit:
>>
>> commit 5af197b55a4b779f1ec93186f0723026949ba2b5
>> Author: Liping Ke<liping.ke@intel.com>
>> Date:   Fri Jun 3 08:22:40 2011 +0800
>>
>>       cache: Implement multiple extra cache fields request support
>
> I think this patch only stores extra information in an extra cache. It only affects the parsing time, and I don't think there will be 5 min difference. How about comparing the "bitbake -e" time with and without the patch(by deleting all the cache files before running bitbake).
>

I'm sorry that I didn't describe it clearly, what I meant was that: This patch
worked correctly, it didn't increase the build time, but other patches after
this one made the build time increased, we just use this commit as a reference
point.

// Robert

> Best Regards,
> -Lianhao
>
>> On my host,
>> 1) the build time of 5af197b55a4b779f1ec93186f0723026949ba2b5 is:
>>
>> real    208m26.133s
>> user    241m29.280s
>> sys     47m0.630s
>>
>> 2) and: 068839698fe192d8846c0ed4db65861448e8e524 is:
>>
>> real    217m39.687s
>> user    255m34.150s
>> sys     48m21.510s
>>
>>
>> I use the bisect build method to find out which patch causes the time
>> increases, but the build time is not stable on my host(Ubuntu 11.04
>> 64bit), e.g., the build time of 1) is 208m at the first build, then
>> "git co other_commit" and build it, after about 10 builds, then go
>> back to build 5af197b55a4b779f1ec, the build time will increases about
>> 8 minutes, I have stopped the X and cron, at. This may have relationship with the linux distribution and disk.
>>
>> So I have to restart the build, reboot the machine by two days, and go
>> on the build. It would be better if anyone has other good method.
>>
>> I think that for the next release(e.g., yocto 1.2), we can find a
>> clean and stable machine(for the distribution, maybe RHEL is more
>> stable than Ubuntu) to check the build time weekly, so that we can notice the performance issue early.
>>
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [RFC] Performance Issue: Build time increases

[Robert Yang]

On 08/15/2011 10:31 PM, Richard Purdie wrote:
> On Fri, 2011-08-12 at 19:06 +0800, Robert Yang wrote:
>> Hi folks,
>>
>> The build time of core-image-sato increases about 5 ~ 10 minutes than
>> the following commit:
>>
>> commit 5af197b55a4b779f1ec93186f0723026949ba2b5
>> Author: Liping Ke<liping.ke@intel.com>
>> Date:   Fri Jun 3 08:22:40 2011 +0800
>>
>>       cache: Implement multiple extra cache fields request support
>>
>> On my host,
>> 1) the build time of 5af197b55a4b779f1ec93186f0723026949ba2b5 is:
>>
>> real    208m26.133s
>> user    241m29.280s
>> sys     47m0.630s
>>
>> 2) and: 068839698fe192d8846c0ed4db65861448e8e524 is:
>>
>> real    217m39.687s
>> user    255m34.150s
>> sys     48m21.510s
>>
>>
>> I use the bisect build method to find out which patch causes the time
>> increases, but the build time is not stable on my host(Ubuntu 11.04 64bit),
>> e.g., the build time of 1) is 208m at the first build, then "git co
>> other_commit" and build it, after about 10 builds, then go back to build
>> 5af197b55a4b779f1ec, the build time will increases about 8 minutes, I have
>> stopped the X and cron, at. This may have relationship with the linux
>> distribution and disk.
>>
>> So I have to restart the build, reboot the machine by two days, and go on the
>> build. It would be better if anyone has other good method.
>>
>> I think that for the next release(e.g., yocto 1.2), we can find a clean and
>> stable machine(for the distribution, maybe RHEL is more stable than Ubuntu)
>> to check the build time weekly, so that we can notice the performance issue
>> early.
>
> I think we really need to get to the bottom of why the build times are
> so variable. Do we need to start doing these on a clean distro install?
> is a fresh boot good enough? is there a way we can clear out the VM

Hi Richard,

Thanks for you reply, yes, one build one restart(bitbake core-image-sato,
then restart the computer, and bitbake again) makes the build time stable.
I'm doing this now.

// Robert

> caches and get reproducible times? reformat the build partition?
>
> If that really is just because of the nature of a chaotic system, can we
> get some better representation of build time to use as a benchmark? We
> could really do with finding a faster test (which would hopefully still
> be representative)...
>
> Cheers,
>
> Richard
>
>
>
>
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [RFC] Performance Issue: Build time increases

[Paul Eggleton]

On Tuesday 16 August 2011 09:40:14 Robert Yang wrote:
> Thanks for you reply, yes, one build one restart(bitbake core-image-sato,
> then restart the computer, and bitbake again) makes the build time stable.
> I'm doing this now.

Are you using my scripts (scripts/contrib/test_build_time.sh) ? Much of the 
initial part of the script is ensuring that caches get cleared out; thus 
rebooting should not be necessary in order to get stable results.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [RFC] Performance Issue: Build time increases

[Robert Yang]

On 08/16/2011 04:56 PM, Paul Eggleton wrote:
> On Tuesday 16 August 2011 09:40:14 Robert Yang wrote:
>> Thanks for you reply, yes, one build one restart(bitbake core-image-sato,
>> then restart the computer, and bitbake again) makes the build time stable.
>> I'm doing this now.
>
> Are you using my scripts (scripts/contrib/test_build_time.sh) ? Much of the
> initial part of the script is ensuring that caches get cleared out; thus
> rebooting should not be necessary in order to get stable results.

I'm sorry to say that I have tried it, but I don't know its principle,
and it made my git repo have too many untracked or modified files.
So I built it manually again:-)

// Robert

>
> Cheers,
> Paul
>

Re: [RFC] Performance Issue: Build time increases

[Paul Eggleton]

On Tuesday 16 August 2011 11:37:24 Robert Yang wrote:
> I'm sorry to say that I have tried it, but I don't know its principle,
> and it made my git repo have too many untracked or modified files.

Is this when you specified a list of revisions to be applied to the repository 
prior to building? If so, I'm wondering if the script needs to do these in a 
different way - perhaps using a branch?

I'd like to know how you ran the script when you tried it. If you have any 
questions I'm happy to answer them, and I'd also be happy to spend some more 
time improving the scripts. Personally I think it would be useful to do this 
rather than using manual methods that are difficult to reproduce. The major 
benefit of the scripts is that you can start a bisect operation and "walk 
away", rather than having to monitor it constantly.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre