[Buildroot] [RFCv2] Per-package out-of-tree build

[Buildroot] [RFCv2] Per-package out-of-tree build

[Thomas Petazzoni]

Hello,

Here comes the second version of the per-package out of tree
prototype. The first version was posted as RFCv1 on January, 21st
2013. I am not posting the patches, because the series contains 43
patches and this is only a RFC.

The patches should be considered as a prototype, I haven't tested all
packages (of course), and there are quite certainly some corner case
issues that haven't been solved. However, I think it's a good time to
get more testing in order to find, and solve those issues.

It should also not be considered yet that we want this feature in
Buildroot. It is again only a prototype, which aims at solving the
remaining problems, and see if we think it is a good idea to have this
in Buildroot or not.

For those who want to test, the 'out-of-tree-v2' branch is available at:

  git://git.free-electrons.com/users/thomas-petazzoni/buildroot.git

Here is a quick summary of what this patch set does:

 * The tarballs are now extracted in $(O)/src/<pkg>-<version>

 * Each tarball is extracted and patched only once, even if there is
   both a target build and a host build. Once a package has been
   extracted and patched, its source code is changed to not have write
   permissions, in order to catch misbehaving packages in terms of out
   of tree build.

 * The builds continue to be done in $(O)/build/<pkg>-<version> for
   target packages and $(O)/build/host-<pkg>-<version> for host
   packages.

 * A <pkg>_SUPPORTS_OUT_OF_TREE variable defines whether the package
   supports out of tree build or not. By default, it's set to NO, in
   which case the package source code is rsynced from its source
   directory to the build directory at the beginning of the configure
   step. It is set to YES by both the autotools and cmake package
   infrastructures (which have been modified to do out of tree build),
   and by some packages that use the generic infrastructure but
   nonetheless support out of tree build.

 * The override source directory mechanism is updated to take
   advantage of this out of tree support. Now, if you override a
   package that supports out of tree build, the source code is no
   longer rsynced to the build directory. This makes the override
   source directory mechanism a lot more usable for big packages such
   as the Linux kernel.

We can roughly divide the patch set in the following sets of patches:

 * Patches 1 to 23 have the goal of getting rid of the <pkg>_DIR and
   replace it by <pkg>_BUILDDIR and <pkg>_SRCDIR. Throughout all those
   patches, <pkg>_SRCDIR continues to be equal to <pkg>_BUILDDIR:
   out-of-tree build is not introduced. But this preparation work is
   needed to get out of tree support.

 * Patches 24 to 27 fix some other misc issues that would prevent some
   packages to build in out-of-tree builds, especially autotools-based
   packages that are known to have issues.

 * Patch 28 is where all the magic happens: from this point on, we use
   out of tree build for all autotools and cmake packages, and for the
   generic packages that explicitly declare their wish to use out of
   tree build.

 * Patches 29 to 42 modify various generic packages to have them
   implement the support for out of tree build.

 * Patch 43 is a minor cleanup.

Comments, ideas, suggestions are welcome.

Best regards,

Thomas

----------------------------------------------------------------
Thomas Petazzoni (43):
      packages: Use @D where possible
      package: fix <pkg>_SRCDIR and <pkg>_BUILDDIR values
      Introduce a global SRC_DIR variable
      package: prepare removal of <pkg>_DIR
      iw: fix to use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      owl-linux: fix to use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      linux-fusion: fix to use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      hostapd: fix to use <pkg>_BUILDDIR instead of <pkg>_DIR
      gst-fsl-plugins: use <pkg>_SRCDIR instead of <pkg>_DIR
      wpa_supplicant: use <pkg>_BUILDDIR instead of <pkg>_DIR
      busybox: use <pkg>_BUILDDIR instead of <pkg>_DIR
      python3:use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      rp-pppoe:use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      python: use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      dbus: use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      rtai: use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      perf: use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      lttng-modules: use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      icu: use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      imx-lib: use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      igh-ethercat: use <pkg>_BUILDDIR and <pkg>_SRCDIR instead of <pkg>_DIR
      urg: fix to use <pkg>_SRCDIR instead of <pkg>_DIR
      package: kill the <pkg>_DIR variable
      autoconf: fix out-of-tree build
      pkgconf: doesn't support out of tree build
      python: fix out of tree build issues
      busybox: don't copy .config file in a post-extract hook
      package: switch to out-of-tree build
      busybox: support out of tree build
      mtd: support out of tree build
      barebox: support out of tree build
      uboot: support out of tree build
      cmake: support out of tree build
      makedevs: support out of tree build
      qt: add out of tree build support
      pkg-stats: generate stats for out of tree support
      jquery: support out of tree
      jquery-validation: support out of tree
      jquery-sparkline: support out of tree
      liberation: support out of tree
      linux-firmware: support out of tree
      linux: support out of tree build
      speex: remove useless BUILD_CMDS
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFCv2] Per-package out-of-tree build

[Diego Iastrubni]

On Sat, Apr 13, 2013 at 9:54 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
>  * Each tarball is extracted and patched only once, even if there is
>    both a target build and a host build. Once a package has been
>    extracted and patched, its source code is changed to not have write
>    permissions, in order to catch misbehaving packages in terms of out
>    of tree build.
Nice

>  * A <pkg>_SUPPORTS_OUT_OF_TREE variable defines whether the package
>    supports out of tree build or not. By default, it's set to NO, in
>    which case the package source code is rsynced from its source
>    directory to the build directory at the beginning of the configure
>    step. It is set to YES by both the autotools and cmake package
>    infrastructures (which have been modified to do out of tree build),
>    and by some packages that use the generic infrastructure but
>    nonetheless support out of tree build.
Does buildroot support also qmake? qmake has also out of source build support.

[Buildroot] [RFCv2] Per-package out-of-tree build

[Thomas Petazzoni]

Dear Diego Iastrubni,

On Sun, 14 Apr 2013 08:55:01 +0300, Diego Iastrubni wrote:

> >  * A <pkg>_SUPPORTS_OUT_OF_TREE variable defines whether the package
> >    supports out of tree build or not. By default, it's set to NO, in
> >    which case the package source code is rsynced from its source
> >    directory to the build directory at the beginning of the
> > configure step. It is set to YES by both the autotools and cmake
> > package infrastructures (which have been modified to do out of tree
> > build), and by some packages that use the generic infrastructure but
> >    nonetheless support out of tree build.
> Does buildroot support also qmake? qmake has also out of source build
> support.

We don't have a specific qmake infrastructure, so for now the packages
that rely on the qmake build system use the 'generic-package' Buildroot
package infrastructure. It is therefore up to them to support
out-of-tree build.

If qmake packages become more widespread, it might be useful to add a
specific infrastructure for them. For now, I feel their number is a bit
limited to do so.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFCv2] Per-package out-of-tree build

[Cam Hutchison]

Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

>      pkgconf: doesn't support out of tree build

Just a minor nit, but this description does not say what the patch does.

[Buildroot] [RFCv2] Per-package out-of-tree build

[Thomas Petazzoni]

Dear Cam Hutchison,

On Tue, 16 Apr 2013 23:03:18 -0000, Cam Hutchison wrote:
> Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> 
> >      pkgconf: doesn't support out of tree build
> 
> Just a minor nit, but this description does not say what the patch does.

So something like:

pkgconf: mark as not supporting out of tree build

would be better?

Thanks for looking at the patches!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFCv2] Per-package out-of-tree build

[Arnout Vandecappelle]

On 13/04/13 20:54, Thomas Petazzoni wrote:
>   * A <pkg>_SUPPORTS_OUT_OF_TREE variable defines whether the package
>     supports out of tree build or not. By default, it's set to NO, in
>     which case the package source code is rsynced from its source
>     directory to the build directory at the beginning of the configure
>     step. It is set to YES by both the autotools and cmake package
>     infrastructures (which have been modified to do out of tree build),
>     and by some packages that use the generic infrastructure but
>     nonetheless support out of tree build.

  It may have been discussed before, but wouldn't it be more convenient 
to insert an extra rsync step explicitly? Similar to how it's currently 
done for OVERRIDE_SRCDIR.


  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFCv2] Per-package out-of-tree build

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Wed, 17 Apr 2013 18:33:54 +0200, Arnout Vandecappelle wrote:
> On 13/04/13 20:54, Thomas Petazzoni wrote:
> >   * A <pkg>_SUPPORTS_OUT_OF_TREE variable defines whether the package
> >     supports out of tree build or not. By default, it's set to NO, in
> >     which case the package source code is rsynced from its source
> >     directory to the build directory at the beginning of the configure
> >     step. It is set to YES by both the autotools and cmake package
> >     infrastructures (which have been modified to do out of tree build),
> >     and by some packages that use the generic infrastructure but
> >     nonetheless support out of tree build.
> 
>   It may have been discussed before, but wouldn't it be more convenient 
> to insert an extra rsync step explicitly? Similar to how it's currently 
> done for OVERRIDE_SRCDIR.

Yes, I think it would make sense. I'll try to implement this (I have a
11 hours flight tomorrow, so plenty of time for BR hacking, don't
hesitate to give more feedback on this out of tree stuff).

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com