[Buildroot] Reducing buildroot compile times by reusing toolchain?

[Buildroot] Reducing buildroot compile times by reusing toolchain?

[Paassen, Hiram van]

Hi,

We are using buildroot to build two images for the same target platform. We noticed that most of the build time is in building the host tools. I'd like to reduce our build time and noticed the option to use an external toolchain.  However that comes with a lot of options which I did not have the time yet to explore. However that led me to some feature requests for buildroot:

Would it be possible to add an option to the external toolchain menu to use a buildroot toolchain as external toolchain? This is probably possible right now but then I have to know exactly what to fill in. I imagine that buildroot would be able to infer all those options from another buildroot installation/out-of-tree-build.
I can imagine that such an option would simplify the reuse of the toolchain for our other build.

Secondly, we run "make clean" often since buildroot cannot automatically remove stuff from the target directory however running "make clean" also removes the toolchain. would it be possible to make a "make clean-target" that would delete the target and possible the staging directory and would rebuild everything except the host/toolchain stuff. Or did I miss something and is such a target already possible?

I can imagine that I can simulate the last feature by making an build using a minimal image and then reusing the toolchain from that build for my actual image. So when i run make clean for my actual image, only the target stuff needs to get rebuild.

I would implement  this myself when I have the time but i'm not really comfortable enough in the innards of buildroot to do this yet

Best regards,

Hiram van Paassen

________________________________

Power Products, LLC Email Notice

This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and/or CONFIDENTIAL.
This email is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this email is not an intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.
If you have received this email in error, please notify the sender immediately by return mail and permanently delete the copy you received.

Thank you.

[Buildroot] Reducing buildroot compile times by reusing toolchain?

[Thomas Petazzoni]

Hello,

On Tue, 30 Sep 2014 08:02:34 +0000, Paassen, Hiram van wrote:

> We are using buildroot to build two images for the same target
> platform. We noticed that most of the build time is in building the
> host tools. I'd like to reduce our build time and noticed the option
> to use an external toolchain. 

Absolutely.

> Would it be possible to add an option to the external toolchain menu
> to use a buildroot toolchain as external toolchain? This is probably
> possible right now but then I have to know exactly what to fill in. I
> imagine that buildroot would be able to infer all those options from
> another buildroot installation/out-of-tree-build. I can imagine that
> such an option would simplify the reuse of the toolchain for our
> other build.

I don't really see how this would work. The main issue here is that
we need to know the characteristics of the toolchain (C library being
used, toolchain options, etc.) at the kconfig level (i.e the
configuration menu). And this is done before Buildroot gets the chance
to start the build to find out by itself the characteristics of the
toolchain.

However, what Buildroot does is that it verifies that the
characteristics of the toolchain declared in menuconfig actually match
the external toolchain being provided. This means that there is
basically zero chance for you to use an invalid configuration.

Also:

 * Beyond Buildroot toolchains, there are also other toolchain
   providers, such as Sourcery CodeBench or Linaro. Depending on the
   architecture you're targeting, this may be interesting to explore.

 * We also provide pre-built Buildroot toolchains as part of our
   autobuild infrastructure. If you look at
   http://autobuild.buildroot.org/toolchains/configs/, all the files
   starting in "br-" are minimal configurations using pre-built
   Buildroot toolchains. For example,
   http://autobuild.buildroot.org/toolchains/configs/br-arm-full.config
   is using a Buildroot uClibc ARM toolchain, targeted at ARM926.

> Secondly, we run "make clean" often since buildroot cannot
> automatically remove stuff from the target directory however running
> "make clean" also removes the toolchain. would it be possible to make
> a "make clean-target" that would delete the target and possible the
> staging directory and would rebuild everything except the
> host/toolchain stuff. Or did I miss something and is such a target
> already possible?

We do not provide this possibility because we cannot implement this in
a safe way.

Scenario:

 1. You enable the lftp package, and also the openssl package in
    menuconfig.

 2. You start the build. Since lftp optionally depends on openssl,
    openssl will be built first, and then lftp. lftp is therefore build
    with openssl support, and relies on the openssl libraries to be
    installed in the target.

 3. The build is finished, you now have both lftp and openssl installed
    in your target filesystem. Everything is fine.

 4. Now, you remove openssl from menuconfig. Of course, lftp is still
    enabled.

 5. You do your new "cleanup target and reinstall everything" thing. It
    will install the already built lftp in the target. But it will not
    install openssl, because the package is no longer enabled.

 6. You know have a broken root filesystem: lftp is linked with the
    openssl libraries, but those libraries were not installed in the
    target.

Since Buildroot is very simple and does not track which package
installs what, and which package should be rebuilt when other packages
are enabled/disabled, we cannot provide the feature you request in a
safe way.

We could of course provide it, but then you would see lots and lots of
weird situations like the one exposed above. And it would be very
complicated for us to provide support to people falling into those
situations.

Some final recommendations:

 * Use an external toolchain.
 * Enable ccache if you do repeated builds with the same toolchain.
 * Buy a faster machine. And if you use a virtual machine like VMWare
   or VirtualBox, throw it away, and do your build on a real physical
   machine.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Reducing buildroot compile times by reusing toolchain?

[Richard Braun]

On Tue, Sep 30, 2014 at 11:30:24AM +0200, Thomas Petazzoni wrote:
>  * Beyond Buildroot toolchains, there are also other toolchain
>    providers, such as Sourcery CodeBench or Linaro. Depending on the
>    architecture you're targeting, this may be interesting to explore.

There is also crosstool-ng, if you want precise control over your
toolchain, but you need to know what you're doing, and there may be
bugs that will require you to patch the toolchain. In other words,
it's a very nice solution if you can afford to validate the result.

-- 
Richard Braun

[Buildroot] Reducing buildroot compile times by reusing toolchain?

[Thomas De Schampheleire]

Hi Hiram,

On Tue, Sep 30, 2014 at 10:02 AM, Paassen, Hiram van
<Hiram.van.Paassen@mastervolt.com> wrote:
> Hi,
>
> We are using buildroot to build two images for the same target platform. We noticed that most of the build time is in building the host tools. I'd like to reduce our build time and noticed the option to use an external toolchain.  However that comes with a lot of options which I did not have the time yet to explore. However that led me to some feature requests for buildroot:
>
> Would it be possible to add an option to the external toolchain menu to use a buildroot toolchain as external toolchain? This is probably possible right now but then I have to know exactly what to fill in. I imagine that buildroot would be able to infer all those options from another buildroot installation/out-of-tree-build.
> I can imagine that such an option would simplify the reuse of the toolchain for our other build.

On top of Thomas Petazzoni's answer, I would like to add that using a
Buildroot toolchain as external toolchain is not hard at all.
Basically you have to create a tarball from the output/host/usr
directory, then specify that tarball as external toolchain and set
some options.
It seems this info is not yet in our manual, but we should add that.

Best regards,
Thomas

[Buildroot] Reducing buildroot compile times by reusing toolchain?

[Károly Kasza]

Hi all,


> On top of Thomas Petazzoni's answer, I would like to add that using a
> Buildroot toolchain as external toolchain is not hard at all.
> Basically you have to create a tarball from the output/host/usr
> directory, then specify that tarball as external toolchain and set
> some options.
> It seems this info is not yet in our manual, but we should add that.
>
>
I use the very same method for recompiling BR with the same internal
toolchain.
I have a .config file for preparing the toolchain with a custom
BR2_HOST_DIR. After running the
"make toolchain" target, I tar the custom host dir and clean up buildroot
with a simple "make clean". This
of course delete the custom host (toolchain) directory too.
The second .config can simply use the re-extracted toolchain
as BR2_TOOLCHAIN_EXTERNAL_PATH.

-- 

?dv,
KK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140930/8934c263/attachment.html>

[Buildroot] Reducing buildroot compile times by reusing toolchain?

[Paassen, Hiram van]

Forgot to cc the list


On di, 2014-09-30 at 11:30 +0200, Thomas Petazzoni wrote:
> Hello,
>
> On Tue, 30 Sep 2014 08:02:34 +0000, Paassen, Hiram van wrote:
>
> > We are using buildroot to build two images for the same target
> > platform. We noticed that most of the build time is in building the
> > host tools. I'd like to reduce our build time and noticed the option
> > to use an external toolchain.
>
> Absolutely.
>
> > Would it be possible to add an option to the external toolchain menu
> > to use a buildroot toolchain as external toolchain? This is probably
> > possible right now but then I have to know exactly what to fill in. I
> > imagine that buildroot would be able to infer all those options from
> > another buildroot installation/out-of-tree-build. I can imagine that
> > such an option would simplify the reuse of the toolchain for our
> > other build.
>
> I don't really see how this would work. The main issue here is that
> we need to know the characteristics of the toolchain (C library being
> used, toolchain options, etc.) at the kconfig level (i.e the
> configuration menu). And this is done before Buildroot gets the chance
> to start the build to find out by itself the characteristics of the
> toolchain.
Sure but if you are using another buildroot folder, would it not be
possible to distill that either from that buildroots kconfig file or let
that buildroot create a special file on build which describes the
created toolchain and include that? Similar to whats in
"br-arm-full.config"
>
> However, what Buildroot does is that it verifies that the
> characteristics of the toolchain declared in menuconfig actually match
> the external toolchain being provided. This means that there is
> basically zero chance for you to use an invalid configuration.
>
> Also:
>
>  * Beyond Buildroot toolchains, there are also other toolchain
>    providers, such as Sourcery CodeBench or Linaro. Depending on the
>    architecture you're targeting, this may be interesting to explore.
>
>  * We also provide pre-built Buildroot toolchains as part of our
>    autobuild infrastructure. If you look at
>    http://autobuild.buildroot.org/toolchains/configs/, all the files
>    starting in "br-" are minimal configurations using pre-built
>    Buildroot toolchains. For example,
>    http://autobuild.buildroot.org/toolchains/configs/br-arm-full.config
>    is using a Buildroot uClibc ARM toolchain, targeted at ARM926.
>
Well I suppose I should spend some time to understand that part of
buildroot. Currently were always using the "buildroot-toolchain" but it
seems that is not a common case given the amount of replies which
suggest using an external toolchain.

> > Secondly, we run "make clean" often since buildroot cannot
> > automatically remove stuff from the target directory however running
> > "make clean" also removes the toolchain. would it be possible to make
> > a "make clean-target" that would delete the target and possible the
> > staging directory and would rebuild everything except the
> > host/toolchain stuff. Or did I miss something and is such a target
> > already possible?
>
> We do not provide this possibility because we cannot implement this in
> a safe way.
>
> Scenario:
>
>  1. You enable the lftp package, and also the openssl package in
>     menuconfig.
>
>  2. You start the build. Since lftp optionally depends on openssl,
>     openssl will be built first, and then lftp. lftp is therefore build
>     with openssl support, and relies on the openssl libraries to be
>     installed in the target.
>
>  3. The build is finished, you now have both lftp and openssl installed
>     in your target filesystem. Everything is fine.
>
>  4. Now, you remove openssl from menuconfig. Of course, lftp is still
>     enabled.
>
>  5. You do your new "cleanup target and reinstall everything" thing. It
>     will install the already built lftp in the target. But it will not
>     install openssl, because the package is no longer enabled.
>
>  6. You know have a broken root filesystem: lftp is linked with the
>     openssl libraries, but those libraries were not installed in the
>     target.
>
> Since Buildroot is very simple and does not track which package
> installs what, and which package should be rebuilt when other packages
> are enabled/disabled, we cannot provide the feature you request in a
> safe way.
>
> We could of course provide it, but then you would see lots and lots of
> weird situations like the one exposed above. And it would be very
> complicated for us to provide support to people falling into those
> situations.
>
I understand that it is not possible to do a partial rebuild of target.
But in this case I expect that lftp is also rebuild since it needs to be
installed in $(TARGET_DIR). But would this not work if say
"make-clean-target" removes $(TARGET_DIR) and all package build dirs
which need to be installed in $(TARGET_DIR) (and possibly
$(STAGING_DIR)) and leaves everything else.

I expect that everything that needs does not install something in
$(TARGET_DIR) is the set of packages named "host-*" so if I can "make
clean" without rebuilding all that is already a win for us.

Of course when using an external toolchain there is not much use for
this target since the toolchain/"host-*" building part is much smaller
in that case

Well in some sense you could implement this as treating the compiled
toolchain as being a preconfigured preextracted external toolchain
except for when it is not yet compiled?

> Some final recommendations:
>
>  * Use an external toolchain.
I think I am going to make my own buildroot-toolchain tarball for me and
my colleagues for now. :-)

>  * Enable ccache if you do repeated builds with the same toolchain.
Already on
>  * Buy a faster machine. And if you use a virtual machine like VMWare
>    or VirtualBox, throw it away, and do your build on a real physical
>    machine.
>
Our build using eglibc, systemd and python takes (including toolchain)
45 min on my i7 quad-core laptop. At least half of that seems to be the
toolchain and other "host" tools. I prefer a faster machine but I
suspect my boss won't give that to me given that I already have the
fastest machine in the office.
> Best regards,
>
> Thomas




________________________________

Power Products, LLC Email Notice

This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and/or CONFIDENTIAL.
This email is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this email is not an intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.
If you have received this email in error, please notify the sender immediately by return mail and permanently delete the copy you received.

Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140930/a9dd4123/attachment.html>

[Buildroot] Reducing buildroot compile times by reusing toolchain?

[Paassen, Hiram van]

On di, 2014-09-30 at 11:41 +0200, Thomas De Schampheleire wrote:
> Hi Hiram,
>
> On Tue, Sep 30, 2014 at 10:02 AM, Paassen, Hiram van
> <Hiram.van.Paassen@mastervolt.com> wrote:
> > Hi,
> >
> > We are using buildroot to build two images for the same target platform. We noticed that most of the build time is in building the host tools. I'd like to reduce our build time and noticed the option to use an external toolchain.  However that comes with a lot of options which I did not have the time yet to explore. However that led me to some feature requests for buildroot:
> >
> > Would it be possible to add an option to the external toolchain menu to use a buildroot toolchain as external toolchain? This is probably possible right now but then I have to know exactly what to fill in. I imagine that buildroot would be able to infer all those options from another buildroot installation/out-of-tree-build.
> > I can imagine that such an option would simplify the reuse of the toolchain for our other build.
>
> On top of Thomas Petazzoni's answer, I would like to add that using a
> Buildroot toolchain as external toolchain is not hard at all.
> Basically you have to create a tarball from the output/host/usr
> directory, then specify that tarball as external toolchain and set
> some options.
> It seems this info is not yet in our manual, but we should add that.
>
> Best regards,
> Thomas
Yes this seems a sensible idea. To bad that it takes some manual work to
do while just using the default toolchain does not except checking out
our project and buildroot and doing:

make BR2_EXTERNAL=... ..._defconfig; make

Of course I can maintain a tarball on a server but that is one thing
more that can get out of sync.

For now I think this is the preferred path. May be a good thing to
document in the manual. Especially that typically, using buildroot means
using this method to prevent long build times.

Best regards,

Hiram


________________________________

Power Products, LLC Email Notice

This message is intended only for the use of the Addressee and may contain information that is PRIVILEGED and/or CONFIDENTIAL.
This email is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this email is not an intended recipient, you have received this email in error and any review, dissemination, distribution or copying is strictly prohibited.
If you have received this email in error, please notify the sender immediately by return mail and permanently delete the copy you received.

Thank you.

[Buildroot] Reducing buildroot compile times by reusing toolchain?

[Mike Zick]

On Tue, 30 Sep 2014 15:04:56 +0000
"Paassen, Hiram van" <Hiram.van.Paassen@mastervolt.com> wrote:

> Our build using eglibc, systemd and python takes (including toolchain)
> 45 min on my i7 quad-core laptop. At least half of that seems to be
> the toolchain and other "host" tools. I prefer a faster machine but I
> suspect my boss won't give that to me given that I already have the
> fastest machine in the office.
>

That might be I/O limited -
Get Boss to buy you a server-grade, fast, SSD for the laptop.

[Buildroot] Reducing buildroot compile times by reusing toolchain?

[Thomas Petazzoni]

Hello,

On Tue, 30 Sep 2014 15:04:56 +0000, Paassen, Hiram van wrote:

> Sure but if you are using another buildroot folder, would it not be
> possible to distill that either from that buildroots kconfig file or let
> that buildroot create a special file on build which describes the
> created toolchain and include that? Similar to whats in
> "br-arm-full.config"

We could have a tool that "scans" a given toolchain and generates a
minimal Buildroot config file from that.

But again, have you actually tried what you're talking about? It's
_very_ easy to use an existing toolchain as an external toolchain in
Buildroot. If you're using glibc, there's basically 2 or 3
configuration options to be filled in. If you're using uClibc, there's
a little more, maybe 8 or 10, but not more. And all of those values are
checked, so there's hardly a way for a user to make a mistake at this
level.

> >  * Beyond Buildroot toolchains, there are also other toolchain
> >    providers, such as Sourcery CodeBench or Linaro. Depending on the
> >    architecture you're targeting, this may be interesting to explore.
> >
> >  * We also provide pre-built Buildroot toolchains as part of our
> >    autobuild infrastructure. If you look at
> >    http://autobuild.buildroot.org/toolchains/configs/, all the files
> >    starting in "br-" are minimal configurations using pre-built
> >    Buildroot toolchains. For example,
> >    http://autobuild.buildroot.org/toolchains/configs/br-arm-full.config
> >    is using a Buildroot uClibc ARM toolchain, targeted at ARM926.
> >
> Well I suppose I should spend some time to understand that part of
> buildroot. Currently were always using the "buildroot-toolchain" but it
> seems that is not a common case given the amount of replies which
> suggest using an external toolchain.

Using the "internal backend" (where Buildroot builds the toolchain) is
quite common. It really depends on the use cases. But since I'm amongst
the people who have put a lot of effort in the external toolchain
support, I certainly believe it's a very useful way of using Buildroot.

> I understand that it is not possible to do a partial rebuild of target.
> But in this case I expect that lftp is also rebuild since it needs to be
> installed in $(TARGET_DIR). But would this not work if say
> "make-clean-target" removes $(TARGET_DIR) and all package build dirs
> which need to be installed in $(TARGET_DIR) (and possibly
> $(STAGING_DIR)) and leaves everything else.

But then we would have to cleanup the $(STAGING_DIR) entirely as well.

> I expect that everything that needs does not install something in
> $(TARGET_DIR) is the set of packages named "host-*" so if I can "make
> clean" without rebuilding all that is already a win for us.
> 
> Of course when using an external toolchain there is not much use for
> this target since the toolchain/"host-*" building part is much smaller
> in that case

Right. There are still a bunch of host tools besides the toolchain
itself, that can take a bit of time to build.

However, the general feeling of the Buildroot developers is that such
techniques usually have a lot of corner cases: they work in many cases,
but not in all cases. And since we want a tool that is simple to use
and understand, we dislike quite a lot those techniques that are not
working properly 100% of the time.

> Well in some sense you could implement this as treating the compiled
> toolchain as being a preconfigured preextracted external toolchain
> except for when it is not yet compiled?

Sorry, I did not understand your suggestion here.

> >  * Use an external toolchain.
> I think I am going to make my own buildroot-toolchain tarball for me and
> my colleagues for now. :-)

Yes, if you have a team, it's generally the good thing to do.

> >  * Enable ccache if you do repeated builds with the same toolchain.
> Already on
> >  * Buy a faster machine. And if you use a virtual machine like VMWare
> >    or VirtualBox, throw it away, and do your build on a real physical
> >    machine.
> >
> Our build using eglibc, systemd and python takes (including toolchain)
> 45 min on my i7 quad-core laptop. At least half of that seems to be the
> toolchain and other "host" tools. I prefer a faster machine but I
> suspect my boss won't give that to me given that I already have the
> fastest machine in the office.

Ok, your machine looks good. If you don't have a SSD, switch to a SSD,
it's going to speed up the build nicely.

Also, do a completely clean build:

	make clean all

And then, generate a graph of the build time:

	make graph-build

Once this is done, look in $(O)/graphs/, and look at the PDF files that
were generated. It will give you some hints as to where the build time
is spent.

Best regards,

Thomas Petazzoni
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com