[Buildroot] [RFC v1] Prototype implementation of per-package out of tree build

[Buildroot] [RFC v1] Prototype implementation of per-package out of tree build

[Thomas Petazzoni]

Hello,

There has recently been some interest in improving the support for
overriden packages by doing out of tree build of all packages. Since I
had the beginning of a prototype working, I cleaned it up, made
numerous fixes to make it work apparently properly, and share it with
the community.

The following patches should be considered as a prototype, I haven't
tested all packages (of course), and there are quite certainly some
corner case issues that haven't been solved. However, I think it's a
good time to get more testing in order to find, and solve those
issues.

It should also not be considered yet that we want this feature in
Buildroot. It is again only a prototype, which aims at solving the
remaining problems, and see if we think it is a good idea to have this
in Buildroot or not.

For those who want to test, the 'out-of-tree' branch is available at:

  git://git.free-electrons.com/users/thomas-petazzoni/buildroot.git

Here is a quick summary of what this patch set does:

 * The tarballs are now extracted in $(O)/src/<pkg>-<version>

 * Each tarball is extracted and patched only once, even if there is
   both a target build and a host build. Once a package has been
   extracted and patched, its source code is changed to not have write
   permissions, in order to catch misbehaving packages in terms of out
   of tree build.

 * The builds continue to be done in $(O)/build/<pkg>-<version> for
   target packages and $(O)/build/host-<pkg>-<version> for host
   packages.

 * A <pkg>_SUPPORTS_OUT_OF_TREE variable defines whether the package
   supports out of tree build or not. By default, it's set to NO, in
   which case the package source code is rsynced from its source
   directory to the build directory at the beginning of the configure
   step. It is set to YES by both the autotools and cmake package
   infrastructures (which have been modified to do out of tree build),
   and by some packages that use the generic infrastructure but
   nonetheless support out of tree build.

 * The override source directory mechanism is updated to take
   advantage of this out of tree support. Now, if you override a
   package that supports out of tree build, the source code is no
   longer rsynced to the build directory. This makes the override
   source directory mechanism a lot more usable for big packages such
   as the Linux kernel.

Amongst the known remaining issues to solve are:

 * The semantic of the 'make <pkg>-dirclean' command. Should it remove
   the build directory and the source directory? Only the build
   directory? If it removes both (which the proposed prototype
   currently does), it is strange because you remove one build variant
   (host or target) and the source directory, but the other build
   variant (target or host) remains.

 * I am not sure on how we want to handle <pkg>-reconfigure and
   <pkg>-rebuild. I guess we want to tell the user to do his changes
   in the package source directory, output/src/<pkg>-<version>. But
   all files in this source directory are marked read-only, so it's
   not very practical.

   In addition to this, for packages that don't support out of tree
   build, we do a rsync at the beginning of the configure step. So if
   the user does "make <pkg>-reconfigure", then the source code gets
   rsynced again from the source directory to the build directory, but
   not if the user does "make <pkg>-rebuild".

Comments, ideas, suggestions are welcome.

Best regards,

Thomas

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Thomas Petazzoni]

This new directory will be used to extract the source directory of the
different packages, as part of the out of tree support.

Note that we need to explicitly re-add write permissions on the
contents of this directory, because write permissions are removed from
the source code of packages in order to ensure that the packages are
not incorrectly modifying their source directory during an out of tree
build. We also re-add write permissions to the build directory since
some packages copy files from their source directory to the build
directory, causing rm to fail due to lack of permissions.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 6f8ed0e..15ce4e4 100644
--- a/Makefile
+++ b/Makefile
@@ -260,6 +260,7 @@ GENERATE_LOCALE=$(call qstrip,$(BR2_GENERATE_LOCALE))
 STAMP_DIR:=$(BASE_DIR)/stamps
 
 BINARIES_DIR:=$(BASE_DIR)/images
+SRC_DIR:=$(BASE_DIR)/src
 TARGET_DIR:=$(BASE_DIR)/target
 TOOLCHAIN_DIR=$(BASE_DIR)/toolchain
 TARGET_SKELETON=$(TOPDIR)/system/skeleton
@@ -383,7 +384,7 @@ TARGETS_LEGAL_INFO:=$(patsubst %,%-legal-info,\
 $(TARGETS_ALL): __real_tgt_%: $(BASE_TARGETS) %
 
 dirs: $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
-	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
+	$(HOST_DIR) $(BINARIES_DIR) $(SRC_DIR) $(STAMP_DIR)
 
 $(BASE_TARGETS): dirs $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
 
@@ -409,7 +410,7 @@ world: toolchain $(TARGETS_ALL)
 # dependencies anywhere else
 #
 #############################################################
-$(TOOLCHAIN_DIR) $(BUILD_DIR) $(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR):
+$(TOOLCHAIN_DIR) $(BUILD_DIR) $(HOST_DIR) $(BINARIES_DIR) $(SRC_DIR) $(STAMP_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR):
 	@mkdir -p $@
 
 $(STAGING_DIR):
@@ -704,9 +705,10 @@ ifeq ($(NEED_WRAPPER),y)
 endif
 
 clean:
+	-chmod u+w -R $(SRC_DIR)/* $(BUILD_DIR)/*
 	rm -rf $(STAGING_DIR) $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
 		$(STAMP_DIR) $(BUILD_DIR) $(TOOLCHAIN_DIR) $(BASE_DIR)/staging \
-		$(LEGAL_INFO_DIR)
+		$(LEGAL_INFO_DIR) $(SRC_DIR)
 
 distclean: clean
 ifeq ($(DL_DIR),$(TOPDIR)/dl)
-- 
1.7.9.5

[Buildroot] [RFC v1 02/14] package infrastructure: move subdir support to autotools

[Thomas Petazzoni]

The <pkg>_SUBDIR variable was currently somewhat handled at the
generic package infrastructure level, with the $(PKG)_SRCDIR variable
taking into account this SUBDIR. But in fact, this <pkg>_SUBDIR
variable only makes sense in the context of the autotools
infrastructure, to indicate that the configure script is not located
at the root of the source tree, but instead in a sub-directory. In the
case of generic packages, the package .mk file already has enough
freedom to handle bizarre situations where the Makefile or other parts
of the build system may not be located at the root of the package
source tree.

In order to simplify the introduction of out of tree support, we
therefore move the SUBDIR support as an autotools infrastructure
specific feature.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/pkg-autotools.mk |   30 ++++++++++++++++++++----------
 package/pkg-generic.mk   |   11 +----------
 2 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
index 890506b..843239d 100644
--- a/package/pkg-autotools.mk
+++ b/package/pkg-autotools.mk
@@ -70,6 +70,16 @@ ifndef $(2)_MAKE
  endif
 endif
 
+ifndef $(2)_SUBDIR
+ ifdef $(3)_SUBDIR
+  $(2)_SUBDIR = $$($(3)_SUBDIR)
+ else
+  $(2)_SUBDIR ?=
+ endif
+endif
+
+$(2)_SRCSUBDIR = $$($(2)_SRCDIR)/$($(2)_SUBDIR)
+
 $(2)_CONF_ENV			?=
 $(2)_CONF_OPT			?=
 $(2)_MAKE_ENV			?=
@@ -93,7 +103,7 @@ ifeq ($(5),target)
 
 # Configure package for target
 define $(2)_CONFIGURE_CMDS
-	(cd $$($$(PKG)_SRCDIR) && rm -rf config.cache && \
+	(cd $$($$(PKG)_SRCSUBDIR) && rm -rf config.cache && \
 	$$(TARGET_CONFIGURE_OPTS) \
 	$$(TARGET_CONFIGURE_ARGS) \
 	$$($$(PKG)_CONF_ENV) \
@@ -120,7 +130,7 @@ else
 # because it often relies on host tools which may or may not be
 # installed.
 define $(2)_CONFIGURE_CMDS
-	(cd $$($$(PKG)_SRCDIR) && rm -rf config.cache; \
+	(cd $$($$(PKG)_SRCSUBDIR) && rm -rf config.cache; \
 	        $$(HOST_CONFIGURE_OPTS) \
 		CFLAGS="$$(HOST_CFLAGS)" \
 		LDFLAGS="$$(HOST_LDFLAGS)" \
@@ -216,11 +226,11 @@ endif
 ifndef $(2)_BUILD_CMDS
 ifeq ($(5),target)
 define $(2)_BUILD_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 else
 define $(2)_BUILD_CMDS
-	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 endif
@@ -231,7 +241,7 @@ endif
 #
 ifndef $(2)_INSTALL_CMDS
 define $(2)_INSTALL_CMDS
-	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -C $$($$(PKG)_SRCDIR) install
+	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -C $$($$(PKG)_SRCSUBDIR) install
 endef
 endif
 
@@ -241,7 +251,7 @@ endif
 #
 ifndef $(2)_INSTALL_STAGING_CMDS
 define $(2)_INSTALL_STAGING_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_STAGING_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_STAGING_OPT) -C $$($$(PKG)_SRCSUBDIR)
 	for i in $$$$(find $(STAGING_DIR)/usr/lib* -name "*.la"); do \
 		cp -f $$$$i $$$$i~; \
 		$$(SED) "s:\(['= ]\)/usr:\\1$(STAGING_DIR)/usr:g" $$$$i; \
@@ -255,7 +265,7 @@ endif
 #
 ifndef $(2)_INSTALL_TARGET_CMDS
 define $(2)_INSTALL_TARGET_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_TARGET_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_TARGET_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 
@@ -265,7 +275,7 @@ endif
 #
 ifndef $(2)_CLEAN_CMDS
 define $(2)_CLEAN_CMDS
-	-$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE)  $$($$(PKG)_CLEAN_OPT) -C $$($$(PKG)_SRCDIR)
+	-$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE)  $$($$(PKG)_CLEAN_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 
@@ -275,7 +285,7 @@ endif
 #
 ifndef $(2)_UNINSTALL_STAGING_CMDS
 define $(2)_UNINSTALL_STAGING_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_UNINSTALL_STAGING_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_UNINSTALL_STAGING_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 
@@ -288,7 +298,7 @@ endif
 #
 ifndef $(2)_UNINSTALL_TARGET_CMDS
 define $(2)_UNINSTALL_TARGET_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -k $$($$(PKG)_UNINSTALL_TARGET_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -k $$($$(PKG)_UNINSTALL_TARGET_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 59de0f0..eeaa4fe 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -218,16 +218,7 @@ endif
 $(2)_BASE_NAME	=  $(1)-$$($(2)_VERSION)
 $(2)_DL_DIR	=  $$(DL_DIR)/$$($(2)_BASE_NAME)
 $(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
-
-ifndef $(2)_SUBDIR
- ifdef $(3)_SUBDIR
-  $(2)_SUBDIR = $$($(3)_SUBDIR)
- else
-  $(2)_SUBDIR ?=
- endif
-endif
-
-$(2)_SRCDIR		       = $$($(2)_DIR)/$$($(2)_SUBDIR)
+$(2)_SRCDIR		       = $$($(2)_DIR)
 $(2)_BUILDDIR		       ?= $$($(2)_SRCDIR)
 
 ifneq ($$($(2)_OVERRIDE_SRCDIR),)
-- 
1.7.9.5

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Thomas Petazzoni]

This new directory will be used to extract the source directory of the
different packages, as part of the out of tree support.

Note that we need to explicitly re-add write permissions on the
contents of this directory, because write permissions are removed from
the source code of packages in order to ensure that the packages are
not incorrectly modifying their source directory during an out of tree
build. We also re-add write permissions to the build directory since
some packages copy files from their source directory to the build
directory, causing rm to fail due to lack of permissions.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 6f8ed0e..15ce4e4 100644
--- a/Makefile
+++ b/Makefile
@@ -260,6 +260,7 @@ GENERATE_LOCALE=$(call qstrip,$(BR2_GENERATE_LOCALE))
 STAMP_DIR:=$(BASE_DIR)/stamps
 
 BINARIES_DIR:=$(BASE_DIR)/images
+SRC_DIR:=$(BASE_DIR)/src
 TARGET_DIR:=$(BASE_DIR)/target
 TOOLCHAIN_DIR=$(BASE_DIR)/toolchain
 TARGET_SKELETON=$(TOPDIR)/system/skeleton
@@ -383,7 +384,7 @@ TARGETS_LEGAL_INFO:=$(patsubst %,%-legal-info,\
 $(TARGETS_ALL): __real_tgt_%: $(BASE_TARGETS) %
 
 dirs: $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
-	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
+	$(HOST_DIR) $(BINARIES_DIR) $(SRC_DIR) $(STAMP_DIR)
 
 $(BASE_TARGETS): dirs $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
 
@@ -409,7 +410,7 @@ world: toolchain $(TARGETS_ALL)
 # dependencies anywhere else
 #
 #############################################################
-$(TOOLCHAIN_DIR) $(BUILD_DIR) $(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR):
+$(TOOLCHAIN_DIR) $(BUILD_DIR) $(HOST_DIR) $(BINARIES_DIR) $(SRC_DIR) $(STAMP_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR):
 	@mkdir -p $@
 
 $(STAGING_DIR):
@@ -704,9 +705,10 @@ ifeq ($(NEED_WRAPPER),y)
 endif
 
 clean:
+	-chmod u+w -R $(SRC_DIR)/* $(BUILD_DIR)/*
 	rm -rf $(STAGING_DIR) $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
 		$(STAMP_DIR) $(BUILD_DIR) $(TOOLCHAIN_DIR) $(BASE_DIR)/staging \
-		$(LEGAL_INFO_DIR)
+		$(LEGAL_INFO_DIR) $(SRC_DIR)
 
 distclean: clean
 ifeq ($(DL_DIR),$(TOPDIR)/dl)
-- 
1.7.9.5

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Jérôme Pouiller]

On Sunday 20 January 2013 21:25:25 Thomas Petazzoni wrote:
> This new directory will be used to extract the source directory of the
> different packages, as part of the out of tree support.
> 
> Note that we need to explicitly re-add write permissions on the
> contents of this directory, because write permissions are removed from
> the source code of packages in order to ensure that the packages are
> not incorrectly modifying their source directory during an out of tree
> build. We also re-add write permissions to the build directory since
> some packages copy files from their source directory to the build
> directory, causing rm to fail due to lack of permissions.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  Makefile |    8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 6f8ed0e..15ce4e4 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -260,6 +260,7 @@ GENERATE_LOCALE=$(call qstrip,$(BR2_GENERATE_LOCALE))
>  STAMP_DIR:=$(BASE_DIR)/stamps
> 
>  BINARIES_DIR:=$(BASE_DIR)/images
> +SRC_DIR:=$(BASE_DIR)/src

IMHO, $(SRC_DIR) should be in $(TOPDIR) to be shared between different 
configurations. In add $(SRC_DIR) have to customizable in case I would like to 
clearly separate two projects.

I think, my workspace would look like this:
  br/srcs-projet1
  br/srcs-projet2
  br/output-project1-configA
  br/output-project1-configB
  br/output-project2-configA
  br/output-project2-configB

Finally SRC_DIR should be handled like DL_DIR.

-- 
J?r?me Pouiller

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Jérôme Pouiller]

Hello Thomas,

(I repost my remarks there in order to keep all discussion in same thread)

On Monday 21 January 2013 00:52:10 Thomas Petazzoni wrote:
> This new directory will be used to extract the source directory of the
> different packages, as part of the out of tree support.
> 
> Note that we need to explicitly re-add write permissions on the
> contents of this directory, because write permissions are removed from
> the source code of packages in order to ensure that the packages are
> not incorrectly modifying their source directory during an out of tree
> build. We also re-add write permissions to the build directory since
> some packages copy files from their source directory to the build
> directory, causing rm to fail due to lack of permissions.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  Makefile |    8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 6f8ed0e..15ce4e4 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -260,6 +260,7 @@ GENERATE_LOCALE=$(call qstrip,$(BR2_GENERATE_LOCALE))
>  STAMP_DIR:=$(BASE_DIR)/stamps
> 
>  BINARIES_DIR:=$(BASE_DIR)/images
> +SRC_DIR:=$(BASE_DIR)/src
IMHO, $(SRC_DIR) should be in $(TOPDIR) to be shared between different 
configurations. In add $(SRC_DIR) have to customizable in case I would like to 
clearly separate two projects.

I think, my workspace would look like this:
  br/srcs-projet1
  br/srcs-projet2
  br/output-project1-configA
  br/output-project1-configB
  br/output-project2-configA
  br/output-project2-configB

Finally SRC_DIR should be handled like DL_DIR.

[...]

-- 
J?r?me Pouiller

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Thomas Petazzoni]

Dear J?r?me Pouiller,

On Mon, 21 Jan 2013 14:47:22 +0100, J?r?me Pouiller wrote:

> IMHO, $(SRC_DIR) should be in $(TOPDIR) to be shared between
> different configurations. In add $(SRC_DIR) have to customizable in
> case I would like to clearly separate two projects.

I have indeed thought about making the source code shareable across
rebuilds and across projects, but since we apply patches to the source
code, it seems a bit complicated to do in a reliable way.

What happens when you bump your Buildroot version and the patches for a
given package change? You would have to manually clean up the source
directories. Not sure we want to do that, at least for now, but the
opinion of others would be interesting here.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Jérôme Pouiller]

On Monday 21 January 2013 18:16:40 Thomas Petazzoni wrote:
> Dear J?r?me Pouiller,
> 
> On Mon, 21 Jan 2013 14:47:22 +0100, J?r?me Pouiller wrote:
> > IMHO, $(SRC_DIR) should be in $(TOPDIR) to be shared between
> > different configurations. In add $(SRC_DIR) have to customizable in
> > case I would like to clearly separate two projects.
> 
> I have indeed thought about making the source code shareable across
> rebuilds and across projects, but since we apply patches to the source
> code, it seems a bit complicated to do in a reliable way.
Hmm... If source is not shared across builds, what is advantage of build out 
of sources? 

I see two advantages:
  - No duplication of source between host and target. But finaly, only a few 
(and small) package are concerned. Shall we care about this?
  - To make build of a package outside of Buildroot easier. In this case, did 
you consider to build in a subdirectory of source? It does not allow to share 
source between host and target but may be sufficient and may solve a bunch of 
questions.

> What happens when you bump your Buildroot version and the patches for a
> given package change? You would have to manually clean up the source
> directories. Not sure we want to do that, at least for now, but the
> opinion of others would be interesting here.
Currently, when you bump your version of Buildroot, you have to clean all. 
Buildroot may provide a rule to also cleaning source directory. 



Regards,

-- 
J?r?me Pouiller

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Thomas Petazzoni]

Dear J?r?me Pouiller,

On Tue, 22 Jan 2013 11:15:11 +0100, J?r?me Pouiller wrote:

> Hmm... If source is not shared across builds, what is advantage of build out 
> of sources? 
> 
> I see two advantages:
>   - No duplication of source between host and target. But finaly, only a few 
> (and small) package are concerned. Shall we care about this?
>   - To make build of a package outside of Buildroot easier. In this case, did 
> you consider to build in a subdirectory of source? It does not allow to share 
> source between host and target but may be sufficient and may solve a bunch of 
> questions.

This last point is the main reason. For now, the LINUX_OVERRIDE_SRCDIR
mechanism is barely usable because it does a complete rsync of the
Linux source tree, which is very annoying. So, switching all packages
to out of tree build sounds like an interesting solution.

> > What happens when you bump your Buildroot version and the patches for a
> > given package change? You would have to manually clean up the source
> > directories. Not sure we want to do that, at least for now, but the
> > opinion of others would be interesting here.
> Currently, when you bump your version of Buildroot, you have to clean all. 
> Buildroot may provide a rule to also cleaning source directory. 

In my opinion, it makes the whole thing a lot more complicated to
understand. At the moment:

	make clean = removes the output directory

That's explained in just one easy line.

If we start putting the source trees outside of the output directory,
things get a lot more complicated for new users. They have to
understand there is one tree for the build results, one tree for the
sources, understand when and why to clean which of the trees and so on.
To me, it's not ok.

Remember, simplicity of use is a *key* point of Buildroot. We are very
attentive about this.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Jérôme Pouiller]

Hello Thomas,

On Tuesday 22 January 2013 17:49:02 Thomas Petazzoni wrote:
> Dear J?r?me Pouiller,
> 
> On Tue, 22 Jan 2013 11:15:11 +0100, J?r?me Pouiller wrote:
> > Hmm... If source is not shared across builds, what is advantage of build
> > out of sources?
> > 
> > I see two advantages:
> >   - No duplication of source between host and target. But finaly, only a
> >   few
> > 
> > (and small) package are concerned. Shall we care about this?
> > 
> >   - To make build of a package outside of Buildroot easier. In this case,
> >   did
> > 
> > you consider to build in a subdirectory of source? It does not allow to
> > share source between host and target but may be sufficient and may solve
> > a bunch of questions.
> 
> This last point is the main reason. For now, the LINUX_OVERRIDE_SRCDIR
> mechanism is barely usable because it does a complete rsync of the
> Linux source tree, which is very annoying. So, switching all packages
> to out of tree build sounds like an interesting solution.
And what about building in a subdirectory of source directory ? 

[...]
> Remember, simplicity of use is a *key* point of Buildroot. We are very
> attentive about this.
Ok, I share your point of view about simplicity. 

Regards,

-- 
J?r?me Pouiller

[Buildroot] [RFC v1] Prototype implementation of per-package out of tree build

[Avery Pennarun]

On Sun, Jan 20, 2013 at 3:25 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> There has recently been some interest in improving the support for
> overriden packages by doing out of tree build of all packages. Since I
> had the beginning of a prototype working, I cleaned it up, made
> numerous fixes to make it work apparently properly, and share it with
> the community.

I'm a little nervous about the out-of-tree stuff in general: my
experience is that out-of-tree builds don't work with quite a lot of
packages, so while it would work with the key stuff (particularly the
kernel and the toolchain, probably) it would always end up a little
flakey.

On the other hand, I've been thinking about a totally different
approach that people might find interesting.  What if we made a local
git repo out of every tree?  Basically:

 - extract the tarball
 - git init
 - git add .
 - git commit -m 'initial tarball $filename'
 - git am /path/to/relevant/patches  # apply the patches and commit them in git

Then if you want to share files between, say, host-* and target-*, you
can use the git_new_worktree script to make a second checkout from the
same git repo.

If someone is working in one of the trees and makes a change, it's
then easy to 'git checkout .' in the other tree and get the same
changes.  So your builds can be identical, but there's no need for the
hackery around read-only files, etc.  And there's a 'git clean' to
guaranteed wipe out all the non-source files (to rebuild without
having to erase and re-extract/re-patch the directory, so it's
faster), 'git diff' to tell you if any of your source files have been
changed, etc.

Using git for the patch queue also makes it easy to update or add
patches using 'git rebase'.  For example, sometimes I've patched a
particular package and notice a bug in my patch, so I would like to
patch the patch.  The way I do this is to 'git am' the patch sequence,
then modify the code, then 'git rebase -i' to merge and rearrange the
patches, then 'git format-patch' to re-extract the set of patches and
drop them back into buildroot.

If buildroot did all this explicitly and automatically as part of the
build process, it would grow a dependency on git.  But on the other
hand, it could automate a lot of neat stuff, like extracting patches.
And "out of tree" builds could just be "in tree" builds with a
different worktree pointing at the same local git repo.

This is just an idea, but it has a lot of potential I think.  I
already do it manually when working on packages inside buildroot, so I
know it works in essence.  And git is so fast that there's basically
no overhead.

Have fun,

Avery

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Thomas Petazzoni]

Dear J?r?me Pouiller,

On Tue, 22 Jan 2013 18:12:49 +0100, J?r?me Pouiller wrote:

> > This last point is the main reason. For now, the
> > LINUX_OVERRIDE_SRCDIR mechanism is barely usable because it does a
> > complete rsync of the Linux source tree, which is very annoying.
> > So, switching all packages to out of tree build sounds like an
> > interesting solution.
> And what about building in a subdirectory of source directory ? 

I am not sure why you think it would solve any problem, but it is
anyway not possible for packages for which <pkg>_OVERRIDE_SRCDIR is
used: we certainly do *not* want to build in a subdirectory of the
source directory.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Jérôme Pouiller]

Hello Thomas,

On Wednesday 23 January 2013 16:45:36 Thomas Petazzoni wrote:
> Dear J?r?me Pouiller,
> 
> On Tue, 22 Jan 2013 18:12:49 +0100, J?r?me Pouiller wrote:
> > > This last point is the main reason. For now, the
> > > LINUX_OVERRIDE_SRCDIR mechanism is barely usable because it does a
> > > complete rsync of the Linux source tree, which is very annoying.
> > > So, switching all packages to out of tree build sounds like an
> > > interesting solution.
> > 
> > And what about building in a subdirectory of source directory ?
> 
> I am not sure why you think it would solve any problem, but it is
> anyway not possible for packages for which <pkg>_OVERRIDE_SRCDIR is
> used: we certainly do *not* want to build in a subdirectory of the
> source directory.
Mainly because packages not compiling out-of-source won't need to be rsynced 
between SRCDIR ans BUILDDIR.

Next it may solve questions about of dirclean behaviour.

Finally because I like idea that a directory contains all files related to a 
package.

Sure, overrided packages have be to handled differently. 

But it is only an idea, not sure it is a good idea :-) .

-- 
J?r?me Pouiller

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Arnout Vandecappelle]

On 01/21/13 06:38, J?r?me Pouiller wrote:
>> >  +SRC_DIR:=$(BASE_DIR)/src
> IMHO, $(SRC_DIR) should be in $(TOPDIR) to be shared between different
> configurations. In add $(SRC_DIR) have to customizable in case I would like to
> clearly separate two projects.
>
> I think, my workspace would look like this:
>    br/srcs-projet1
>    br/srcs-projet2
>    br/output-project1-configA
>    br/output-project1-configB
>    br/output-project2-configA
>    br/output-project2-configB
>
> Finally SRC_DIR should be handled like DL_DIR.

  I agree that that would be a good idea.

  However, what if the patches are different for two configurations? They 
can be different because:

- there is a configuration option for custom patches, like 
BR2_LINUX_KERNEL_PATCH

- there are patches applied depending on configuration options, like 
BR2_LINUX_KERNEL_EXT_XENOMAI_ADEOS_PATCH

- there are manual modifications applied depending on some configuration 
option, like BZIP2_NOLARGEFILE_FIX_MAKEFILE

- override.mk appends to the FOO_PATCH variable

- BR2_EXTRA_PACKAGE_PATCH_DIRS (not committed yet, 
http://patchwork.ozlabs.org/patch/204216/ )

  So in practice, I'm afraid it's not feasible.

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F