[Buildroot] How to avoid rebuilding host-cmake

[Buildroot] How to avoid rebuilding host-cmake

[David Kessler]

Hi all,

I am trying to reduce the amount of time it takes to rebuild the linux kernel and its dependencies.  I am already using the external toolchain option (toolchain was originally built using buildroot, but in an external directory) so I?m not rebuilding the toolchain unnecessarily. However, for some reason after doing ?make clean?, host-cmake has to be rebuilt. For my configuration host-cmake is the most time consuming part of the build.

Is there a way that I can avoid having to rebuilding host-cmake after doing a make clean?

Thanks,
David Kessler?

[Buildroot] How to avoid rebuilding host-cmake

[Ryan Barnett]

David,

On Tue, Feb 17, 2015 at 4:38 PM, David Kessler <djkessler@utexas.edu> wrote:
>
> I am trying to reduce the amount of time it takes to rebuild the
> linux kernel and its dependencies.  I am already using the external
> toolchain option (toolchain was originally built using buildroot, but
> in an external directory) so I?m not rebuilding the toolchain
> unnecessarily. However, for some reason after doing ?make clean?,
> host-cmake has to be rebuilt. For my configuration host-cmake is
> the most time consuming part of the build.

A 'make clean' in the concept of Buildroot delete all build products
(including build directories, host, staging and target trees, the
images and the toolchain). Therefore, an understanding of when a full
rebuild is necessary. Please see the following section in the manual:

http://nightly.buildroot.org/manual.html#full-rebuild

A per package rebuild functionality is availalbe and is described here:

http://nightly.buildroot.org/manual.html#rebuild-pkg

Is it fair to assume that you are trying to do kernel development?

If so, then you should have a look at utilizing the
<pkg>_OVERRIDE_SRCDIR functionality that Buildroot has which allows
you to development of a package outside of Buildroot. For more
detailed information on how to use this feature see Section 8.11.6 in
the manual:

http://nightly.buildroot.org/manual.html#_advanced_usage

Lastly, you should take a look at utilizing ccache to speed up your
builds, see Section 8.11.3 in the manual:

http://nightly.buildroot.org/manual.html#_advanced_usage

Thanks,
-Ryan

-- 
Ryan Barnett / Sr Software Engineer
Airborne Information Systems / Security Systems and Software
MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA
ryan.barnett at rockwellcollins.com
www.rockwellcollins.com

[Buildroot] How to avoid rebuilding host-cmake

[David Kessler]

Hi Ryan,

Thanks for the reply. Before posting, I looked over the documentation 
that discusses when a full rebuild is required.  Your assumption is 
mostly correct, I have been doing some light kernel development. And I 
have been utilizing buildroot's <pkg>_OVERRIDE_SRCDIR option. However, I 
have also been experimenting with many different configuration options, 
options provided by buildroot and by the linux kernel.  According to the 
buildroot documentation, a full rebuild is required when removing a 
package, and I haven't had much success with adding packages without 
doing a full rebuild.

So, I ask about rebuilding host-cmake because it takes quite a long 
time, and it doesn't seem like it needs to be rebuilt when I'm merely 
adding or removing a package.  I suppose that is a bit of an assumption 
on my part.... maybe host-cmake does indeed need to be rebuilt, 
depending on the package I've added or removed?

In any case, thanks for your suggestions.

Best Regards,
David Kessler--


On 02/17/2015 05:02 PM, Ryan Barnett wrote:
> David,
>
> On Tue, Feb 17, 2015 at 4:38 PM, David Kessler <djkessler@utexas.edu> wrote:
>> I am trying to reduce the amount of time it takes to rebuild the
>> linux kernel and its dependencies.  I am already using the external
>> toolchain option (toolchain was originally built using buildroot, but
>> in an external directory) so I?m not rebuilding the toolchain
>> unnecessarily. However, for some reason after doing ?make clean?,
>> host-cmake has to be rebuilt. For my configuration host-cmake is
>> the most time consuming part of the build.
> A 'make clean' in the concept of Buildroot delete all build products
> (including build directories, host, staging and target trees, the
> images and the toolchain). Therefore, an understanding of when a full
> rebuild is necessary. Please see the following section in the manual:
>
> http://nightly.buildroot.org/manual.html#full-rebuild
>
> A per package rebuild functionality is availalbe and is described here:
>
> http://nightly.buildroot.org/manual.html#rebuild-pkg
>
> Is it fair to assume that you are trying to do kernel development?
>
> If so, then you should have a look at utilizing the
> <pkg>_OVERRIDE_SRCDIR functionality that Buildroot has which allows
> you to development of a package outside of Buildroot. For more
> detailed information on how to use this feature see Section 8.11.6 in
> the manual:
>
> http://nightly.buildroot.org/manual.html#_advanced_usage
>
> Lastly, you should take a look at utilizing ccache to speed up your
> builds, see Section 8.11.3 in the manual:
>
> http://nightly.buildroot.org/manual.html#_advanced_usage
>
> Thanks,
> -Ryan
>

[Buildroot] How to avoid rebuilding host-cmake

[Ryan Barnett]

David,

For future replies to the mailing list, please don't top post but
instead reply inline.

On Wed, Feb 18, 2015 at 9:28 AM, David Kessler <djkessler@utexas.edu> wrote:

[...]

> So, I ask about rebuilding host-cmake because it takes quite a long time,
> and it doesn't seem like it needs to be rebuilt when I'm merely adding or
> removing a package.  I suppose that is a bit of an assumption on my part....
> maybe host-cmake does indeed need to be rebuilt, depending on the package
> I've added or removed?

I think the best thing that you could do to help speed up your builds
is utilize ccache other than that you are going have to live with the
build time.

Thanks,
-Ryan

-- 
Ryan Barnett / Sr Software Engineer
Airborne Information Systems / Security Systems and Software
MS 131-100, C Ave NE, Cedar Rapids, IA, 52498, USA
ryan.barnett at rockwellcollins.com
www.rockwellcollins.com

[Buildroot] How to avoid rebuilding host-cmake

[Thomas Petazzoni]

Dear David Kessler,

On Wed, 18 Feb 2015 09:28:52 -0600, David Kessler wrote:

> Thanks for the reply. Before posting, I looked over the documentation 
> that discusses when a full rebuild is required.  Your assumption is 
> mostly correct, I have been doing some light kernel development. And I 
> have been utilizing buildroot's <pkg>_OVERRIDE_SRCDIR option. However, I 
> have also been experimenting with many different configuration options, 
> options provided by buildroot and by the linux kernel.  According to the 
> buildroot documentation, a full rebuild is required when removing a 
> package,

It is only necessary when you want the package removal to take effect
immediately. Generally, what I end up doing for such things is that I
do the configuration changes, and then only restart a full build to
make things everything is OK when leaving in the evening, or during
lunch time.

Basically, try to avoid the situation where you do one change in the
config, one full rebuild. Try to batch the configuration changes. You
rarely need to remove a package immediately.

> and I haven't had much success with adding packages without 
> doing a full rebuild.

This however works just fine, as long of course as this package is not
used as an optional dependency of other packages that you have
selected. For example, if you build "bind" initially with the OpenSSL
package disabled, bind will have OpenSSL support disabled. Then if you
enable OpenSSL in the configuration and do just "make", it will build
OpenSSL, but will not automatically rebuild "bind" to make it gain
OpenSSL support. This will only happen either if you do a full rebuild,
or if you rebuild bind explicitly ("make bind-dirclean bind").

> So, I ask about rebuilding host-cmake because it takes quite a long 
> time, and it doesn't seem like it needs to be rebuilt when I'm merely 
> adding or removing a package.  I suppose that is a bit of an assumption 
> on my part.... maybe host-cmake does indeed need to be rebuilt, 
> depending on the package I've added or removed?

Rebuilding host-cmake is indeed not needed, the way it is built doesn't
change depending on packages you've added or removed. However, as Ryan
explained, Buildroot has no concept of keeping a cache of previous
build results in order to re-use them for later builds. Doing so is
very complicated, as we need to be able to capture the entire
configuration with which a certain package has been built, in order to
find out whether a cached pre-built version still matches the current
configuration. This is quite complicated to do in a reliable manner,
and Buildroot simplicity principle tells us to not do so.

Regarding host-cmake specifically, you may be interested in testing
http://patchwork.ozlabs.org/patch/432613/, which reduces the host-cmake
build time by 25%.

Unfortunately, ccache does not speed up the build of host-cmake today,
because the host-cmake build system does not support a compiler with
spaces in it such as "output/host/usr/bin/ccache /usr/bin/gcc".

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com