[Buildroot] how to repopulate /output/target/lib?

[Buildroot] how to repopulate /output/target/lib?

[Aras Vaichas]

Hello,

I'm currently moving a project to Buildroot. So I'm configuring
Buildroot to create an identical fs tree to my existing tree.

My workflow is:

* run make
* compare output/target against my original fs (using xxdiff)
* adjust Buildroot settings. e.g. skeleton, post build script, etc
* clear out output/target
* repeat

I have a helper script that does some of the dirty work:

#!/bin/sh
find -name .stamp_target_installed | xargs rm
rm -rf output/target
rm output/build/.root
echo now run make

My problem is that "make" no longer populates output/target/lib or
output/target/usr/lib at this stage of the build and I don't know why.

I understand that output/staging holds all the libraries, but how do
the correct libraries get copied to output/target? How can I trigger
this action without having to do a "make clean; make"?

Thanks in advance,

Aras Vaichas

[Buildroot] how to repopulate /output/target/lib?

[Thomas Petazzoni]

Hello Aras,

Le Mon, 25 Jun 2012 12:46:50 +0100,
Aras Vaichas <aras.vaichas@gmail.com> a ?crit :

> I understand that output/staging holds all the libraries, but how do
> the correct libraries get copied to output/target? How can I trigger
> this action without having to do a "make clean; make"?

So far, we have no supported mechanism other than "make clean; make" to
rebuild the target root filesystem. That's part of the Buildroot
simplicity: we don't try to support partial rebuilds, because it is
very complicated to get 100% right, and we don't want to support
something that is right 90% of the time, but is known to have an
incorrect behavior 10% of the time.

To answer your question, the libraries are installed in
output/target/{usr/,}lib by:

 * The toolchain installation procedure. This one depends on whether
   you're using the internal, external or crosstool-ng backend for your
   toolchain. This part installs the C library and related basic
   libraries (thread, math, rt, etc.).

 * The package installation steps, which should be replayed if you
   remove .stamp_target_installed files, as you did.

Basically, my suggestions to do what you're trying to do is:

 * Don't use a custom skeleton, do your customization in a post-build
   script, because it gets re-run at every build, while the skeleton
   copy is only done once at the beginning of a clean build;

 * When you make a change to a package, just remove this package build
   directory (rm -rf output/build/<pkg>-<version>). This is generally
   enough to check that the new installation results are correct.

 * Use an external toolchain so that doing "make clean; make" is not
   too painful.

 * Ensure you have a fast enough build machine (i.e, not a slow Windows
   laptop that runs Linux inside a VMWare)

I'm applying all those suggestions, and I'm a fairly happy Buildroot
user :)

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] how to repopulate /output/target/lib?

[Thomas Petazzoni]

Hello Aras,

Please keep the list Cc'ed for replies.

Le Mon, 25 Jun 2012 13:55:03 +0100,
Aras Vaichas <aras.vaichas@gmail.com> a ?crit :

> > ?* The toolchain installation procedure. This one depends on whether
> > ? you're using the internal, external or crosstool-ng backend for your
> > ? toolchain. This part installs the C library and related basic
> > ? libraries (thread, math, rt, etc.).
> 
> Thank you for this hint. I found the Makefile responsible and I am now
> able to repopulate the entire output/target directory. I had to remove
> output/stamps/ct-ng-toolchain-installed in order to trigger the
> library copying code. I wasn't sure if removing this file would
> retrigger the entire toolchain build so I was wary of removing it.
> 
> To summarise, this is what I do to recreate output/target (reinstall
> all packages) from scratch:
> 
> # find -name .stamp_target_installed | xargs rm
> # rm -rf output/target
> # rm output/build/.root
> # rm output/stamps/ct-ng-toolchain-installed
> # make
> 
> As you mentioned before, Buildroot doesn't support partial rebuilds so
> I don't guarantee what I did will work in 100% of cases, but it
> certainly does make life a little easier. :)

Sure. The thing is that such a workflow might generate an inconsistent
target root filesystem: you reinstall into the target only the
libraries that are enabled now, but if some libraries previously
selected are installed in the staging/ directory, but have been removed
from menuconfig in the mean time, they are no longer installed in the
target, giving you an inconsistent target root filesystem.
> 
> > ?* Ensure you have a fast enough build machine (i.e, not a slow Windows
> > ? laptop that runs Linux inside a VMWare)
> 
> Unfortunately that is what I have been given for my development, so
> building the toolchain is a day lost.

You should request a more powerful machine.

> I have seen some previous posts about building two copies of Buildroot -
> one for the toolchain, the other for development. I think I will re-read
> them and implement an external toolchain.

If you use the ct-ng backend, then just use Crosstool-NG outside of
Buildroot to generate your toolchain, and import it into Buildroot
using the external toolchain backend.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] how to repopulate /output/target/lib?

[Luca Ceresoli]

Thomas Petazzoni wrote:
> Hello Aras,
>
> Le Mon, 25 Jun 2012 12:46:50 +0100,
> Aras Vaichas <aras.vaichas@gmail.com> a ?crit :
>
>> I understand that output/staging holds all the libraries, but how do
>> the correct libraries get copied to output/target? How can I trigger
>> this action without having to do a "make clean; make"?
>
> So far, we have no supported mechanism other than "make clean; make" to
> rebuild the target root filesystem. That's part of the Buildroot
> simplicity: we don't try to support partial rebuilds, because it is
> very complicated to get 100% right, and we don't want to support
> something that is right 90% of the time, but is known to have an
> incorrect behavior 10% of the time.
>
> To answer your question, the libraries are installed in
> output/target/{usr/,}lib by:
>
>   * The toolchain installation procedure. This one depends on whether
>     you're using the internal, external or crosstool-ng backend for your
>     toolchain. This part installs the C library and related basic
>     libraries (thread, math, rt, etc.).
>
>   * The package installation steps, which should be replayed if you
>     remove .stamp_target_installed files, as you did.
>
> Basically, my suggestions to do what you're trying to do is:
>
>   * Don't use a custom skeleton, do your customization in a post-build
>     script, because it gets re-run at every build, while the skeleton
>     copy is only done once at the beginning of a clean build;
>
>   * When you make a change to a package, just remove this package build
>     directory (rm -rf output/build/<pkg>-<version>). This is generally
>     enough to check that the new installation results are correct.

Instead of `rm -rf output/build/<pkg>-<version>`, you can just do
`make <pkg>-dirclean`, which is equivalent but a bit cleaner IMHO.

>
>   * Use an external toolchain so that doing "make clean; make" is not
>     too painful.
>
>   * Ensure you have a fast enough build machine (i.e, not a slow Windows
>     laptop that runs Linux inside a VMWare)

To speed up recompilations, also enable ccache ("Enable compiler cache" in
the "Build options" menu).

Luca