[Buildroot] [RFC v1] Prototype implementation of per-package out of tree build

[Buildroot] [RFC v1] Prototype implementation of per-package out of tree build

[Thomas Petazzoni]

Hello,

(Sorry for the previous posting that only had the first two patches. I
was in the train in the middle of the mountains, and the 3G connection
was interrupted for a too long time.)

There has recently been some interest in improving the support for
overriden packages by doing out of tree build of all packages. Since I
had the beginning of a prototype working, I cleaned it up, made
numerous fixes to make it work apparently properly, and share it with
the community.

The following patches should be considered as a prototype, I haven't
tested all packages (of course), and there are quite certainly some
corner case issues that haven't been solved. However, I think it's a
good time to get more testing in order to find, and solve those
issues.

It should also not be considered yet that we want this feature in
Buildroot. It is again only a prototype, which aims at solving the
remaining problems, and see if we think it is a good idea to have this
in Buildroot or not.

For those who want to test, the 'out-of-tree' branch is available at:

  git://git.free-electrons.com/users/thomas-petazzoni/buildroot.git

Here is a quick summary of what this patch set does:

 * The tarballs are now extracted in $(O)/src/<pkg>-<version>

 * Each tarball is extracted and patched only once, even if there is
   both a target build and a host build. Once a package has been
   extracted and patched, its source code is changed to not have write
   permissions, in order to catch misbehaving packages in terms of out
   of tree build.

 * The builds continue to be done in $(O)/build/<pkg>-<version> for
   target packages and $(O)/build/host-<pkg>-<version> for host
   packages.

 * A <pkg>_SUPPORTS_OUT_OF_TREE variable defines whether the package
   supports out of tree build or not. By default, it's set to NO, in
   which case the package source code is rsynced from its source
   directory to the build directory at the beginning of the configure
   step. It is set to YES by both the autotools and cmake package
   infrastructures (which have been modified to do out of tree build),
   and by some packages that use the generic infrastructure but
   nonetheless support out of tree build.

 * The override source directory mechanism is updated to take
   advantage of this out of tree support. Now, if you override a
   package that supports out of tree build, the source code is no
   longer rsynced to the build directory. This makes the override
   source directory mechanism a lot more usable for big packages such
   as the Linux kernel.

Amongst the known remaining issues to solve are:

 * The semantic of the 'make <pkg>-dirclean' command. Should it remove
   the build directory and the source directory? Only the build
   directory? If it removes both (which the proposed prototype
   currently does), it is strange because you remove one build variant
   (host or target) and the source directory, but the other build
   variant (target or host) remains.

 * I am not sure on how we want to handle <pkg>-reconfigure and
   <pkg>-rebuild. I guess we want to tell the user to do his changes
   in the package source directory, output/src/<pkg>-<version>. But
   all files in this source directory are marked read-only, so it's
   not very practical.

   In addition to this, for packages that don't support out of tree
   build, we do a rsync at the beginning of the configure step. So if
   the user does "make <pkg>-reconfigure", then the source code gets
   rsynced again from the source directory to the build directory, but
   not if the user does "make <pkg>-rebuild".

Comments, ideas, suggestions are welcome.

Best regards,

Thomas

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Thomas Petazzoni]

This new directory will be used to extract the source directory of the
different packages, as part of the out of tree support.

Note that we need to explicitly re-add write permissions on the
contents of this directory, because write permissions are removed from
the source code of packages in order to ensure that the packages are
not incorrectly modifying their source directory during an out of tree
build. We also re-add write permissions to the build directory since
some packages copy files from their source directory to the build
directory, causing rm to fail due to lack of permissions.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 6f8ed0e..15ce4e4 100644
--- a/Makefile
+++ b/Makefile
@@ -260,6 +260,7 @@ GENERATE_LOCALE=$(call qstrip,$(BR2_GENERATE_LOCALE))
 STAMP_DIR:=$(BASE_DIR)/stamps
 
 BINARIES_DIR:=$(BASE_DIR)/images
+SRC_DIR:=$(BASE_DIR)/src
 TARGET_DIR:=$(BASE_DIR)/target
 TOOLCHAIN_DIR=$(BASE_DIR)/toolchain
 TARGET_SKELETON=$(TOPDIR)/system/skeleton
@@ -383,7 +384,7 @@ TARGETS_LEGAL_INFO:=$(patsubst %,%-legal-info,\
 $(TARGETS_ALL): __real_tgt_%: $(BASE_TARGETS) %
 
 dirs: $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
-	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
+	$(HOST_DIR) $(BINARIES_DIR) $(SRC_DIR) $(STAMP_DIR)
 
 $(BASE_TARGETS): dirs $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
 
@@ -409,7 +410,7 @@ world: toolchain $(TARGETS_ALL)
 # dependencies anywhere else
 #
 #############################################################
-$(TOOLCHAIN_DIR) $(BUILD_DIR) $(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR):
+$(TOOLCHAIN_DIR) $(BUILD_DIR) $(HOST_DIR) $(BINARIES_DIR) $(SRC_DIR) $(STAMP_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR):
 	@mkdir -p $@
 
 $(STAGING_DIR):
@@ -704,9 +705,10 @@ ifeq ($(NEED_WRAPPER),y)
 endif
 
 clean:
+	-chmod u+w -R $(SRC_DIR)/* $(BUILD_DIR)/*
 	rm -rf $(STAGING_DIR) $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
 		$(STAMP_DIR) $(BUILD_DIR) $(TOOLCHAIN_DIR) $(BASE_DIR)/staging \
-		$(LEGAL_INFO_DIR)
+		$(LEGAL_INFO_DIR) $(SRC_DIR)
 
 distclean: clean
 ifeq ($(DL_DIR),$(TOPDIR)/dl)
-- 
1.7.9.5

[Buildroot] [RFC v1 02/14] package infrastructure: move subdir support to autotools

[Thomas Petazzoni]

The <pkg>_SUBDIR variable was currently somewhat handled at the
generic package infrastructure level, with the $(PKG)_SRCDIR variable
taking into account this SUBDIR. But in fact, this <pkg>_SUBDIR
variable only makes sense in the context of the autotools
infrastructure, to indicate that the configure script is not located
at the root of the source tree, but instead in a sub-directory. In the
case of generic packages, the package .mk file already has enough
freedom to handle bizarre situations where the Makefile or other parts
of the build system may not be located at the root of the package
source tree.

In order to simplify the introduction of out of tree support, we
therefore move the SUBDIR support as an autotools infrastructure
specific feature.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/pkg-autotools.mk |   30 ++++++++++++++++++++----------
 package/pkg-generic.mk   |   11 +----------
 2 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
index 890506b..843239d 100644
--- a/package/pkg-autotools.mk
+++ b/package/pkg-autotools.mk
@@ -70,6 +70,16 @@ ifndef $(2)_MAKE
  endif
 endif
 
+ifndef $(2)_SUBDIR
+ ifdef $(3)_SUBDIR
+  $(2)_SUBDIR = $$($(3)_SUBDIR)
+ else
+  $(2)_SUBDIR ?=
+ endif
+endif
+
+$(2)_SRCSUBDIR = $$($(2)_SRCDIR)/$($(2)_SUBDIR)
+
 $(2)_CONF_ENV			?=
 $(2)_CONF_OPT			?=
 $(2)_MAKE_ENV			?=
@@ -93,7 +103,7 @@ ifeq ($(5),target)
 
 # Configure package for target
 define $(2)_CONFIGURE_CMDS
-	(cd $$($$(PKG)_SRCDIR) && rm -rf config.cache && \
+	(cd $$($$(PKG)_SRCSUBDIR) && rm -rf config.cache && \
 	$$(TARGET_CONFIGURE_OPTS) \
 	$$(TARGET_CONFIGURE_ARGS) \
 	$$($$(PKG)_CONF_ENV) \
@@ -120,7 +130,7 @@ else
 # because it often relies on host tools which may or may not be
 # installed.
 define $(2)_CONFIGURE_CMDS
-	(cd $$($$(PKG)_SRCDIR) && rm -rf config.cache; \
+	(cd $$($$(PKG)_SRCSUBDIR) && rm -rf config.cache; \
 	        $$(HOST_CONFIGURE_OPTS) \
 		CFLAGS="$$(HOST_CFLAGS)" \
 		LDFLAGS="$$(HOST_LDFLAGS)" \
@@ -216,11 +226,11 @@ endif
 ifndef $(2)_BUILD_CMDS
 ifeq ($(5),target)
 define $(2)_BUILD_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 else
 define $(2)_BUILD_CMDS
-	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 endif
@@ -231,7 +241,7 @@ endif
 #
 ifndef $(2)_INSTALL_CMDS
 define $(2)_INSTALL_CMDS
-	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -C $$($$(PKG)_SRCDIR) install
+	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -C $$($$(PKG)_SRCSUBDIR) install
 endef
 endif
 
@@ -241,7 +251,7 @@ endif
 #
 ifndef $(2)_INSTALL_STAGING_CMDS
 define $(2)_INSTALL_STAGING_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_STAGING_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_STAGING_OPT) -C $$($$(PKG)_SRCSUBDIR)
 	for i in $$$$(find $(STAGING_DIR)/usr/lib* -name "*.la"); do \
 		cp -f $$$$i $$$$i~; \
 		$$(SED) "s:\(['= ]\)/usr:\\1$(STAGING_DIR)/usr:g" $$$$i; \
@@ -255,7 +265,7 @@ endif
 #
 ifndef $(2)_INSTALL_TARGET_CMDS
 define $(2)_INSTALL_TARGET_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_TARGET_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_TARGET_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 
@@ -265,7 +275,7 @@ endif
 #
 ifndef $(2)_CLEAN_CMDS
 define $(2)_CLEAN_CMDS
-	-$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE)  $$($$(PKG)_CLEAN_OPT) -C $$($$(PKG)_SRCDIR)
+	-$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE)  $$($$(PKG)_CLEAN_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 
@@ -275,7 +285,7 @@ endif
 #
 ifndef $(2)_UNINSTALL_STAGING_CMDS
 define $(2)_UNINSTALL_STAGING_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_UNINSTALL_STAGING_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_UNINSTALL_STAGING_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 
@@ -288,7 +298,7 @@ endif
 #
 ifndef $(2)_UNINSTALL_TARGET_CMDS
 define $(2)_UNINSTALL_TARGET_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -k $$($$(PKG)_UNINSTALL_TARGET_OPT) -C $$($$(PKG)_SRCDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -k $$($$(PKG)_UNINSTALL_TARGET_OPT) -C $$($$(PKG)_SRCSUBDIR)
 endef
 endif
 
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 59de0f0..eeaa4fe 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -218,16 +218,7 @@ endif
 $(2)_BASE_NAME	=  $(1)-$$($(2)_VERSION)
 $(2)_DL_DIR	=  $$(DL_DIR)/$$($(2)_BASE_NAME)
 $(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
-
-ifndef $(2)_SUBDIR
- ifdef $(3)_SUBDIR
-  $(2)_SUBDIR = $$($(3)_SUBDIR)
- else
-  $(2)_SUBDIR ?=
- endif
-endif
-
-$(2)_SRCDIR		       = $$($(2)_DIR)/$$($(2)_SUBDIR)
+$(2)_SRCDIR		       = $$($(2)_DIR)
 $(2)_BUILDDIR		       ?= $$($(2)_SRCDIR)
 
 ifneq ($$($(2)_OVERRIDE_SRCDIR),)
-- 
1.7.9.5

[Buildroot] [RFC v1 03/14] generic infrastructure: out of tree build support

[Thomas Petazzoni]

This commit adds the out of tree support to the core package
infrastructure.

In details:

 * The stamp_downloaded, stamp_extracted and stamp_patched are now
   created in the source directory, since those steps are relative to
   the source itself, and not to the build process.

 * The rsynced and rsync_overrive_sourced steps are removed, as
   overriden packages are now naturally handled by the package
   infrastructure.

 * In the patch step, at the end, we mark all files in the package
   source directory as non-writable. This is done to ensure that the
   out-of-tree build process really doesn't mess up with the source
   code (which is now shared by the target and host builds).

 * In the configure step, if the package doesn't support out of tree
   build, we rsync its source code into the build directory.

 * We now define the $(2)_RAW_BASE_NAME variable in addition to
   $(2)_BASE_NAME. The former is always "<pkg>-<version>" regarding of
   whether we're building a target or a host package, while the latter
   is "<pkg>-<version>" for target packages, and
   "host-<pkg>-<version>" for host packages. The $(2)_RAW_BASE_NAME
   variable will be used to name the source directory of packages,
   since we have only one, shared for both the target and host build.

 * $(2)_BUILDDIR is no longer equal to $(2)_SRCDIR, and is now defined
   to $(BUILD_DIR)/$(2)_BASE_NAME, i.e
   output/build/(host-)?<pkg>-<version>. $(2)_SRCDIR, in turn, is
   equal to output/src/<pkg>-<version> if the package has not been
   overriden, otherwise it is set to the overriden location.

 * By default, $(2)_SUPPORTS_OUT_OF_TREE is set to NO, so that the
   copy of the sources to the build directory is done. It can be
   overriden on a per-package basis (for example, the Linux, Barebox
   or U-Boot generic packages support out of tree build) or in some
   specialized package infrastructure.

 * The location of the stamp files is now defined relatively to either
   the build directory or the source directory. We consider that the
   download, extract and patch steps are related to the sources, and
   therefore the stamp file is created in the source directory of the
   package. The other steps are consider to be build related.

 * The default extract command is changed to extract to the source
   directory.

 * The sequencing of steps is slightly modified for various reasons:

   - As the autoreconf step will be moved as a post patch hook, we
     need to have a special handling of the host-autoconf,
     host-automake and host-libtool dependencies. Instead of depending
     on them in the $(1)-depends step, we depend on them in the
     $(1)-patch step. We can't move all dependencies to the $(1)-patch
     step, otherwise we would break the RTAI packaging. See commit
     a473a616d27e46951a23d90249a31b08006098bf.

   - dirclean is now implemented differently depending on whether the
     package has an overriden source directory or not. When the
     package is normal (not overriden), both the source directory and
     build directory are removed. When the package is overriden, only
     the build directory is removed.

   - the rsync step for overriden packages no longer exists. It is now
     part of the natural handling of packaging that don't support out
     of tree build.

 * The legal info infrastructure is changed to force copy license
   files. Since the files in the source directories are read-only, if
   we run twice the 'make legal-info' command, it fails at copying the
   license files. Forcing the copy makes it work properly.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/pkg-generic.mk |  128 ++++++++++++++++++++++++++----------------------
 package/pkg-utils.mk   |    2 +-
 2 files changed, 71 insertions(+), 59 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index eeaa4fe..d6ac4ef 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -25,7 +25,7 @@
 ################################################################################
 
 # Retrieve the archive
-$(BUILD_DIR)/%/.stamp_downloaded:
+$(SRC_DIR)/%/.stamp_downloaded:
 ifeq ($(DL_MODE),DOWNLOAD)
 # Only show the download message if it isn't already downloaded
 	$(Q)if test ! -e $(DL_DIR)/$($(PKG)_SOURCE); then \
@@ -48,7 +48,7 @@ ifeq ($(DL_MODE),DOWNLOAD)
 endif
 
 # Unpack the archive
-$(BUILD_DIR)/%/.stamp_extracted:
+$(SRC_DIR)/%/.stamp_extracted:
 	@$(call MESSAGE,"Extracting")
 	$(Q)mkdir -p $(@D)
 	$($(PKG)_EXTRACT_CMDS)
@@ -57,32 +57,13 @@ $(BUILD_DIR)/%/.stamp_extracted:
 	$(foreach hook,$($(PKG)_POST_EXTRACT_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
-# Rsync the source directory if the <pkg>_OVERRIDE_SRCDIR feature is
-# used.
-$(BUILD_DIR)/%/.stamp_rsynced:
-	@$(call MESSAGE,"Syncing from source dir $(SRCDIR)")
-	@test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1)
-	rsync -au $(SRCDIR)/ $(@D)
-	$(Q)touch $@
-
-# Handle the SOURCE_CHECK and SHOW_EXTERNAL_DEPS cases for rsynced
-# packages
-$(BUILD_DIR)/%/.stamp_rsync_sourced:
-ifeq ($(DL_MODE),SOURCE_CHECK)
-	test -d $(SRCDIR)
-else ifeq ($(DL_MODE),SHOW_EXTERNAL_DEPS)
-	echo "file://$(SRCDIR)"
-else
-	@true # Nothing to do to source a local package
-endif
-
 # Patch
 #
 # The RAWNAME variable is the lowercased package name, which allows to
 # find the package directory (typically package/<pkgname>) and the
 # prefix of the patches
-$(BUILD_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION)
-$(BUILD_DIR)/%/.stamp_patched:
+$(SRC_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION)
+$(SRC_DIR)/%/.stamp_patched:
 	@$(call MESSAGE,"Patching $($(PKG)_DIR_PREFIX)/$(RAWNAME)")
 	$(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(foreach p,$($(PKG)_PATCH),support/scripts/apply-patches.sh $(@D) $(DL_DIR) $(p)$(sep))
@@ -100,9 +81,15 @@ $(BUILD_DIR)/%/.stamp_patched:
 	)
 	$(foreach hook,$($(PKG)_POST_PATCH_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
+	chmod a-w -R $(@D)
 
 # Configure
 $(BUILD_DIR)/%/.stamp_configured:
+	@mkdir -p $(@D)
+	@if test "$($(PKG)_SUPPORTS_OUT_OF_TREE)" != "YES"; then \
+		rsync -au $($(PKG)_SRCDIR)/* $(@D)/ ; \
+		chmod u+w -R $(@D)/ ; \
+	fi
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	@$(call MESSAGE,"Configuring")
 	$($(PKG)_CONFIGURE_CMDS)
@@ -167,10 +154,26 @@ $(BUILD_DIR)/%/.stamp_uninstalled:
 	$(if $(BR2_INIT_SYSV)$(BR2_INIT_BUSYBOX),\
 		$($(PKG)_UNINSTALL_INIT_SYSV))
 
-# Remove package sources
+# Remove package build directory
 $(BUILD_DIR)/%/.stamp_dircleaned:
 	rm -Rf $(@D)
 
+# Remove package source directory
+$(SRC_DIR)/%/.stamp_dircleaned:
+	-chmod u+w -R $(@D)
+	rm -Rf $(@D)
+
+# Handle the SOURCE_CHECK and SHOW_EXTERNAL_DEPS cases for overriden
+# packages
+$(BUILD_DIR)/%/.stamp_override_sourced:
+ifeq ($(DL_MODE),SOURCE_CHECK)
+	test -d $($(PKG)_SRCDIR)
+else ifeq ($(DL_MODE),SHOW_EXTERNAL_DEPS)
+	echo "file://$($(PKG)_SRCDIR)"
+else
+	@true # Nothing to do to source a local package
+endif
+
 ################################################################################
 # inner-generic-package -- generates the make targets needed to build a
 # generic package
@@ -215,11 +218,10 @@ else
   $(2)_VERSION = $(subst /,_,$($(2)_VERSION))
 endif
 
-$(2)_BASE_NAME	=  $(1)-$$($(2)_VERSION)
+$(2)_BASE_NAME	   =  $$($(2)_NAME)-$$($(2)_VERSION)
+$(2)_RAW_BASE_NAME = $$($(2)_RAWNAME)-$$($(2)_VERSION)
 $(2)_DL_DIR	=  $$(DL_DIR)/$$($(2)_BASE_NAME)
-$(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
-$(2)_SRCDIR		       = $$($(2)_DIR)
-$(2)_BUILDDIR		       ?= $$($(2)_SRCDIR)
+$(2)_BUILDDIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
 
 ifneq ($$($(2)_OVERRIDE_SRCDIR),)
 $(2)_VERSION = custom
@@ -260,6 +262,12 @@ $(2)_OVERRIDE_SRCDIR = $($(2)_SITE)
 endif
 endif
 
+ifeq ($$($(2)_OVERRIDE_SRCDIR),)
+$(2)_SRCDIR     = $$(SRC_DIR)/$$($(2)_RAW_BASE_NAME)
+else
+$(2)_SRCDIR     = $$($(2)_OVERRIDE_SRCDIR)
+endif
+
 ifndef $(2)_LICENSE
  ifdef $(3)_LICENSE
   $(2)_LICENSE = $($(3)_LICENSE)
@@ -282,6 +290,13 @@ endif
 
 $(2)_REDISTRIBUTE		?= YES
 
+ifndef $(2)_SUPPORTS_OUT_OF_TREE
+ ifdef $(3)_SUPPORTS_OUT_OF_TREE
+  $(2)_SUPPORTS_OUT_OF_TREE = $($(3)_SUPPORTS_OUT_OF_TREE)
+ endif
+endif
+
+$(2)_SUPPORTS_OUT_OF_TREE       ?= NO
 
 $(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
 
@@ -291,25 +306,25 @@ $(2)_INSTALL_TARGET		?= YES
 $(2)_DIR_PREFIX			= $(if $(4),$(4),$(TOP_SRCDIR)/package)
 
 # define sub-target stamps
-$(2)_TARGET_INSTALL_TARGET =	$$($(2)_DIR)/.stamp_target_installed
-$(2)_TARGET_INSTALL_STAGING =	$$($(2)_DIR)/.stamp_staging_installed
-$(2)_TARGET_INSTALL_IMAGES =	$$($(2)_DIR)/.stamp_images_installed
-$(2)_TARGET_INSTALL_HOST =      $$($(2)_DIR)/.stamp_host_installed
-$(2)_TARGET_BUILD =		$$($(2)_DIR)/.stamp_built
-$(2)_TARGET_CONFIGURE =		$$($(2)_DIR)/.stamp_configured
-$(2)_TARGET_RSYNC =	        $$($(2)_DIR)/.stamp_rsynced
-$(2)_TARGET_RSYNC_SOURCE =      $$($(2)_DIR)/.stamp_rsync_sourced
-$(2)_TARGET_PATCH =		$$($(2)_DIR)/.stamp_patched
-$(2)_TARGET_EXTRACT =		$$($(2)_DIR)/.stamp_extracted
-$(2)_TARGET_SOURCE =		$$($(2)_DIR)/.stamp_downloaded
-$(2)_TARGET_UNINSTALL =		$$($(2)_DIR)/.stamp_uninstalled
-$(2)_TARGET_CLEAN =		$$($(2)_DIR)/.stamp_cleaned
-$(2)_TARGET_DIRCLEAN =		$$($(2)_DIR)/.stamp_dircleaned
+$(2)_TARGET_INSTALL_TARGET =	$$($(2)_BUILDDIR)/.stamp_target_installed
+$(2)_TARGET_INSTALL_STAGING =	$$($(2)_BUILDDIR)/.stamp_staging_installed
+$(2)_TARGET_INSTALL_IMAGES =	$$($(2)_BUILDDIR)/.stamp_images_installed
+$(2)_TARGET_INSTALL_HOST =      $$($(2)_BUILDDIR)/.stamp_host_installed
+$(2)_TARGET_BUILD =		$$($(2)_BUILDDIR)/.stamp_built
+$(2)_TARGET_CONFIGURE =		$$($(2)_BUILDDIR)/.stamp_configured
+$(2)_TARGET_PATCH =		$$($(2)_SRCDIR)/.stamp_patched
+$(2)_TARGET_EXTRACT =		$$($(2)_SRCDIR)/.stamp_extracted
+$(2)_TARGET_SOURCE =		$$($(2)_SRCDIR)/.stamp_downloaded
+$(2)_TARGET_OVERRIDE_SOURCE =	$$($(2)_BUILDDIR)/.stamp_override_sourced
+$(2)_TARGET_UNINSTALL =		$$($(2)_BUILDDIR)/.stamp_uninstalled
+$(2)_TARGET_CLEAN =		$$($(2)_BUILDDIR)/.stamp_cleaned
+$(2)_TARGET_SRCDIRCLEAN =	$$($(2)_SRCDIR)/.stamp_dircleaned
+$(2)_TARGET_BUILDDIRCLEAN =	$$($(2)_BUILDDIR)/.stamp_dircleaned
 
 # default extract command
 $(2)_EXTRACT_CMDS ?= \
 	$$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $(DL_DIR)/$$($(2)_SOURCE) | \
-	$(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
+	$(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_SRCDIR) $(TAR_OPTIONS) -)
 
 # post-steps hooks
 $(2)_POST_DOWNLOAD_HOOKS        ?=
@@ -370,7 +385,8 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 $(1)-configure:		$(1)-patch $(1)-depends \
 			$$($(2)_TARGET_CONFIGURE)
 
-$(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
+$(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH) \
+			$(filter host-automake host-autoconf host-libtool,$$($(2)_DEPENDENCIES))
 
 $(1)-extract:		$(1)-source \
 			$$($(2)_TARGET_EXTRACT)
@@ -378,22 +394,24 @@ $(1)-extract:		$(1)-source \
 $(1)-depends:		$$($(2)_DEPENDENCIES)
 
 $(1)-source:		$$($(2)_TARGET_SOURCE)
+
+$(1)-dirclean:		$$($(2)_TARGET_SRCDIRCLEAN) \
+			$$($(2)_TARGET_BUILDDIRCLEAN)
 else
 # In the package override case, the sequence of steps
-#  source, by rsyncing
 #  depends
 #  configure
 $(1)-configure:		$(1)-depends \
 			$$($(2)_TARGET_CONFIGURE)
 
-$(1)-depends:		$(1)-rsync $$($(2)_DEPENDENCIES)
+$(1)-depends:		$$($(2)_DEPENDENCIES)
 
-$(1)-patch:		$(1)-rsync
-$(1)-extract:		$(1)-rsync
+$(1)-patch:
+$(1)-extract:
 
-$(1)-rsync:		$$($(2)_TARGET_RSYNC)
+$(1)-source:		$$($(2)_TARGET_OVERRIDE_SOURCE)
 
-$(1)-source:		$$($(2)_TARGET_RSYNC_SOURCE)
+$(1)-dirclean:		$$($(2)_TARGET_BUILDDIRCLEAN)
 endif
 
 $(1)-show-depends:
@@ -404,8 +422,6 @@ $(1)-uninstall:		$(1)-configure $$($(2)_TARGET_UNINSTALL)
 $(1)-clean:		$(1)-uninstall \
 			$$($(2)_TARGET_CLEAN)
 
-$(1)-dirclean:		$$($(2)_TARGET_DIRCLEAN)
-
 $(1)-clean-for-rebuild:
 ifneq ($$($(2)_OVERRIDE_SRCDIR),)
 			rm -f $$($(2)_TARGET_RSYNC)
@@ -431,17 +447,13 @@ $$($(2)_TARGET_INSTALL_IMAGES):		PKG=$(2)
 $$($(2)_TARGET_INSTALL_HOST):           PKG=$(2)
 $$($(2)_TARGET_BUILD):			PKG=$(2)
 $$($(2)_TARGET_CONFIGURE):		PKG=$(2)
-$$($(2)_TARGET_RSYNC):                  SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
-$$($(2)_TARGET_RSYNC):                  PKG=$(2)
-$$($(2)_TARGET_RSYNC_SOURCE):		SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
-$$($(2)_TARGET_RSYNC_SOURCE):		PKG=$(2)
 $$($(2)_TARGET_PATCH):			PKG=$(2)
 $$($(2)_TARGET_PATCH):			RAWNAME=$(patsubst host-%,%,$(1))
 $$($(2)_TARGET_EXTRACT):		PKG=$(2)
 $$($(2)_TARGET_SOURCE):			PKG=$(2)
+$$($(2)_TARGET_OVERRIDE_SOURCE):	PKG=$(2)
 $$($(2)_TARGET_UNINSTALL):		PKG=$(2)
 $$($(2)_TARGET_CLEAN):			PKG=$(2)
-$$($(2)_TARGET_DIRCLEAN):		PKG=$(2)
 
 # Compute the name of the Kconfig option that correspond to the
 # package being enabled. We handle three cases: the special Linux
@@ -488,7 +500,7 @@ ifeq ($(call qstrip,$$($(2)_LICENSE_FILES)),)
 	@$(call legal-warning-pkg,$$($(2)_RAWNAME),cannot save license ($(2)_LICENSE_FILES not defined))
 else
 	@for F in $$($(2)_LICENSE_FILES); do \
-		$(call legal-license-file,$$($(2)_RAWNAME),$$$${F},$$($(2)_DIR)/$$$${F}); \
+		$(call legal-license-file,$$($(2)_RAWNAME),$$$${F},$$($(2)_SRCDIR)/$$$${F}); \
 		done
 endif
 ifeq ($$($(2)_REDISTRIBUTE),YES)
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 835c588..4eb9df3 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -107,5 +107,5 @@ define legal-license-file # pkg, filename, file-fullpath
 	cat $(3) >>$(LEGAL_LICENSES_TXT) && \
 	echo >>$(LEGAL_LICENSES_TXT) && \
 	mkdir -p $(LICENSE_FILES_DIR)/$(1)/ && \
-	cp $(3) $(LICENSE_FILES_DIR)/$(1)/
+	cp -f $(3) $(LICENSE_FILES_DIR)/$(1)/
 endef
-- 
1.7.9.5

[Buildroot] [RFC v1 04/14] autotools infrastructure: support out of tree build

[Thomas Petazzoni]

All autotools packages are supposed to support out of tree build, so
we mark this at the package infrastructure level, and adjust the
infrastructure to execute the configure script from the build
directory.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/pkg-autotools.mk |   27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)

diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
index 843239d..bd87253 100644
--- a/package/pkg-autotools.mk
+++ b/package/pkg-autotools.mk
@@ -78,8 +78,6 @@ ifndef $(2)_SUBDIR
  endif
 endif
 
-$(2)_SRCSUBDIR = $$($(2)_SRCDIR)/$($(2)_SUBDIR)
-
 $(2)_CONF_ENV			?=
 $(2)_CONF_OPT			?=
 $(2)_MAKE_ENV			?=
@@ -91,6 +89,7 @@ $(2)_INSTALL_TARGET_OPT		?= DESTDIR=$$(TARGET_DIR)  install
 $(2)_CLEAN_OPT			?= clean
 $(2)_UNINSTALL_STAGING_OPT	?= DESTDIR=$$(STAGING_DIR) uninstall
 $(2)_UNINSTALL_TARGET_OPT	?= DESTDIR=$$(TARGET_DIR)  uninstall
+$(2)_SUPPORTS_OUT_OF_TREE        = YES
 
 
 #
@@ -103,11 +102,11 @@ ifeq ($(5),target)
 
 # Configure package for target
 define $(2)_CONFIGURE_CMDS
-	(cd $$($$(PKG)_SRCSUBDIR) && rm -rf config.cache && \
+	(cd $$($$(PKG)_BUILDDIR) && rm -rf config.cache && \
 	$$(TARGET_CONFIGURE_OPTS) \
 	$$(TARGET_CONFIGURE_ARGS) \
 	$$($$(PKG)_CONF_ENV) \
-	./configure \
+	$$($$(PKG)_SRCDIR)/$$($$(PKG)_SUBDIR)/configure \
 		--target=$$(GNU_TARGET_NAME) \
 		--host=$$(GNU_TARGET_NAME) \
 		--build=$$(GNU_HOST_NAME) \
@@ -130,12 +129,12 @@ else
 # because it often relies on host tools which may or may not be
 # installed.
 define $(2)_CONFIGURE_CMDS
-	(cd $$($$(PKG)_SRCSUBDIR) && rm -rf config.cache; \
+	(cd $$($$(PKG)_BUILDDIR) && rm -rf config.cache; \
 	        $$(HOST_CONFIGURE_OPTS) \
 		CFLAGS="$$(HOST_CFLAGS)" \
 		LDFLAGS="$$(HOST_LDFLAGS)" \
                 $$($$(PKG)_CONF_ENV) \
-		./configure \
+		$$($$(PKG)_SRCDIR)/$$($$(PKG)_SUBDIR)/configure \
 		--prefix="$$(HOST_DIR)/usr" \
 		--sysconfdir="$$(HOST_DIR)/etc" \
 		--enable-shared --disable-static \
@@ -226,11 +225,11 @@ endif
 ifndef $(2)_BUILD_CMDS
 ifeq ($(5),target)
 define $(2)_BUILD_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCSUBDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_BUILDDIR)
 endef
 else
 define $(2)_BUILD_CMDS
-	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_SRCSUBDIR)
+	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_MAKE_OPT) -C $$($$(PKG)_BUILDDIR)
 endef
 endif
 endif
@@ -241,7 +240,7 @@ endif
 #
 ifndef $(2)_INSTALL_CMDS
 define $(2)_INSTALL_CMDS
-	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -C $$($$(PKG)_SRCSUBDIR) install
+	$$(HOST_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -C $$($$(PKG)_BUILDDIR) install
 endef
 endif
 
@@ -251,7 +250,7 @@ endif
 #
 ifndef $(2)_INSTALL_STAGING_CMDS
 define $(2)_INSTALL_STAGING_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_STAGING_OPT) -C $$($$(PKG)_SRCSUBDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_STAGING_OPT) -C $$($$(PKG)_BUILDDIR)
 	for i in $$$$(find $(STAGING_DIR)/usr/lib* -name "*.la"); do \
 		cp -f $$$$i $$$$i~; \
 		$$(SED) "s:\(['= ]\)/usr:\\1$(STAGING_DIR)/usr:g" $$$$i; \
@@ -265,7 +264,7 @@ endif
 #
 ifndef $(2)_INSTALL_TARGET_CMDS
 define $(2)_INSTALL_TARGET_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_TARGET_OPT) -C $$($$(PKG)_SRCSUBDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_INSTALL_TARGET_OPT) -C $$($$(PKG)_BUILDDIR)
 endef
 endif
 
@@ -275,7 +274,7 @@ endif
 #
 ifndef $(2)_CLEAN_CMDS
 define $(2)_CLEAN_CMDS
-	-$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE)  $$($$(PKG)_CLEAN_OPT) -C $$($$(PKG)_SRCSUBDIR)
+	-$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE)  $$($$(PKG)_CLEAN_OPT) -C $$($$(PKG)_BUILDDIR)
 endef
 endif
 
@@ -285,7 +284,7 @@ endif
 #
 ifndef $(2)_UNINSTALL_STAGING_CMDS
 define $(2)_UNINSTALL_STAGING_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_UNINSTALL_STAGING_OPT) -C $$($$(PKG)_SRCSUBDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) $$($$(PKG)_UNINSTALL_STAGING_OPT) -C $$($$(PKG)_BUILDDIR)
 endef
 endif
 
@@ -298,7 +297,7 @@ endif
 #
 ifndef $(2)_UNINSTALL_TARGET_CMDS
 define $(2)_UNINSTALL_TARGET_CMDS
-	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -k $$($$(PKG)_UNINSTALL_TARGET_OPT) -C $$($$(PKG)_SRCSUBDIR)
+	$$(TARGET_MAKE_ENV) $$($$(PKG)_MAKE_ENV) $$($$(PKG)_MAKE) -k $$($$(PKG)_UNINSTALL_TARGET_OPT) -C $$($$(PKG)_BUILDDIR)
 endef
 endif
 
-- 
1.7.9.5

[Buildroot] [RFC v1 05/14] autotools infrastructure: do the autoreconf as a post patch step

[Thomas Petazzoni]

Doing the autoreconf step as a pre-configure hook doesn't work
properly, because the source directory is read-only during the
configure step. And in fact, the autoreconf process modifies the
source code, so it is quite logical to do it as part of the patching
process rather than the configuration process.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/pkg-autotools.mk |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
index bd87253..ec394fd 100644
--- a/package/pkg-autotools.mk
+++ b/package/pkg-autotools.mk
@@ -214,7 +214,7 @@ $(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),
 
 
 ifeq ($$($(2)_AUTORECONF),YES)
-$(2)_PRE_CONFIGURE_HOOKS += AUTORECONF_HOOK
+$(2)_POST_PATCH_HOOKS += AUTORECONF_HOOK
 $(2)_DEPENDENCIES += host-automake host-autoconf host-libtool
 endif
 
-- 
1.7.9.5

[Buildroot] [RFC v1 06/14] cmake infrastructure: support out of tree build

[Thomas Petazzoni]

The cmake infrastructure almost naturally supports out of tree
build. The only thing is that the <pkg>_BUILDDIR and <pkg>_SRCDIR
variables are now defined by pkg-generic.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/pkg-cmake.mk |    4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/package/pkg-cmake.mk b/package/pkg-cmake.mk
index 626a0b0..6653d3f 100644
--- a/package/pkg-cmake.mk
+++ b/package/pkg-cmake.mk
@@ -46,9 +46,7 @@ $(2)_INSTALL_HOST_OPT		?= install
 $(2)_INSTALL_STAGING_OPT	?= DESTDIR=$$(STAGING_DIR) install
 $(2)_INSTALL_TARGET_OPT		?= DESTDIR=$$(TARGET_DIR) install
 $(2)_CLEAN_OPT			?= clean
-
-$(2)_SRCDIR			= $$($(2)_DIR)/$($(2)_SUBDIR)
-$(2)_BUILDDIR			= $$($(2)_SRCDIR)
+$(2)_SUPPORTS_OUT_OF_TREE       = YES
 
 #
 # Configure step. Only define it if not already defined by the package
-- 
1.7.9.5

[Buildroot] [RFC v1 07/14] busybox: support out of tree build

[Thomas Petazzoni]

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/busybox/busybox.mk |   26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
index 9154c08..36b0903 100644
--- a/package/busybox/busybox.mk
+++ b/package/busybox/busybox.mk
@@ -15,6 +15,8 @@ BUSYBOX_SOURCE = busybox-$(BUSYBOX_VERSION).tar.bz2
 BUSYBOX_LICENSE = GPLv2
 BUSYBOX_LICENSE_FILES = LICENSE
 
+BUSYBOX_SUPPORTS_OUT_OF_TREE = YES
+
 BUSYBOX_CFLAGS = \
 	$(TARGET_CFLAGS) \
 	-I$(LINUX_HEADERS_DIR)/include
@@ -32,7 +34,7 @@ BUSYBOX_CFLAGS += -I$(STAGING_DIR)/usr/include/tirpc/
 BUSYBOX_CFLAGS_busybox += -ltirpc
 endif
 
-BUSYBOX_BUILD_CONFIG = $(BUSYBOX_DIR)/.config
+BUSYBOX_BUILD_CONFIG = $(BUSYBOX_BUILDDIR)/.config
 # Allows the build system to tweak CFLAGS
 BUSYBOX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
@@ -173,10 +175,8 @@ define BUSYBOX_INSTALL_WATCHDOG_SCRIPT
 endef
 endif
 
-# We do this here to avoid busting a modified .config in configure
-BUSYBOX_POST_EXTRACT_HOOKS += BUSYBOX_COPY_CONFIG
-
 define BUSYBOX_CONFIGURE_CMDS
+	$(BUSYBOX_COPY_CONFIG)
 	$(BUSYBOX_SET_LARGEFILE)
 	$(BUSYBOX_SET_IPV6)
 	$(BUSYBOX_PREFER_STATIC)
@@ -188,15 +188,15 @@ define BUSYBOX_CONFIGURE_CMDS
 	$(BUSYBOX_SET_INIT)
 	$(BUSYBOX_SET_WATCHDOG)
 	@yes "" | $(MAKE) ARCH=$(KERNEL_ARCH) CROSS_COMPILE="$(TARGET_CROSS)" \
-		-C $(@D) oldconfig
+		-C $(BUSYBOX_SRCDIR) O=$(@D) oldconfig
 endef
 
 define BUSYBOX_BUILD_CMDS
-	$(BUSYBOX_MAKE_ENV) $(MAKE) $(BUSYBOX_MAKE_OPTS) -C $(@D)
+	$(BUSYBOX_MAKE_ENV) $(MAKE) $(BUSYBOX_MAKE_OPTS) -C $(BUSYBOX_SRCDIR) O=$(@D)
 endef
 
 define BUSYBOX_INSTALL_TARGET_CMDS
-	$(BUSYBOX_MAKE_ENV) $(MAKE) $(BUSYBOX_MAKE_OPTS) -C $(@D) install
+	$(BUSYBOX_MAKE_ENV) $(MAKE) $(BUSYBOX_MAKE_OPTS) -C $(BUSYBOX_SRCDIR) O=$(@D) install
 	if [ ! -f $(TARGET_DIR)/usr/share/udhcpc/default.script ]; then \
 		$(INSTALL) -m 0755 -D package/busybox/udhcpc.script \
 			$(TARGET_DIR)/usr/share/udhcpc/default.script; \
@@ -208,20 +208,20 @@ define BUSYBOX_INSTALL_TARGET_CMDS
 endef
 
 define BUSYBOX_UNINSTALL_TARGET_CMDS
-	$(BUSYBOX_MAKE_ENV) $(MAKE) $(BUSYBOX_MAKE_OPTS) -C $(@D) uninstall
+	$(BUSYBOX_MAKE_ENV) $(MAKE) $(BUSYBOX_MAKE_OPTS) -C $(BUSYBOX_SRCDIR) O=$(@D) uninstall
 endef
 
 define BUSYBOX_CLEAN_CMDS
-	$(BUSYBOX_MAKE_ENV) $(MAKE) $(BUSYBOX_MAKE_OPTS) -C $(@D) clean
+	$(BUSYBOX_MAKE_ENV) $(MAKE) $(BUSYBOX_MAKE_OPTS) -C $(BUSYBOX_SRCDIR) O=$(@D) clean
 endef
 
 $(eval $(generic-package))
 
 busybox-menuconfig busybox-xconfig busybox-gconfig: busybox-patch
-	$(BUSYBOX_MAKE_ENV) $(MAKE) $(BUSYBOX_MAKE_OPTS) -C $(BUSYBOX_DIR) \
-		$(subst busybox-,,$@)
-	rm -f $(BUSYBOX_DIR)/.stamp_built
-	rm -f $(BUSYBOX_DIR)/.stamp_target_installed
+	$(BUSYBOX_MAKE_ENV) $(MAKE) $(BUSYBOX_MAKE_OPTS) -C $(BUSYBOX_SRCDIR) \
+		O=$(BUSYBOX_BUILDDIR) $(subst busybox-,,$@)
+	rm -f $(BUSYBOX_BUILDDIR)/.stamp_built
+	rm -f $(BUSYBOX_BUILDDIR)/.stamp_target_installed
 
 busybox-update-config:
 	cp -f $(BUSYBOX_BUILD_CONFIG) $(BUSYBOX_CONFIG_FILE)
-- 
1.7.9.5

[Buildroot] [RFC v1 08/14] mtd: support out of tree build

[Thomas Petazzoni]

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/mtd/mtd.mk |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/package/mtd/mtd.mk b/package/mtd/mtd.mk
index eafce45..3123581 100644
--- a/package/mtd/mtd.mk
+++ b/package/mtd/mtd.mk
@@ -9,6 +9,8 @@ MTD_SITE = ftp://ftp.infradead.org/pub/mtd-utils
 MTD_LICENSE = GPLv2
 MTD_LICENSE_FILES = COPYING
 
+MTD_SUPPORTS_OUT_OF_TREE = YES
+
 ifeq ($(BR2_PACKAGE_MTD_MKFSJFFS2),y)
 MTD_DEPENDENCIES = zlib lzo
 endif
@@ -21,11 +23,11 @@ HOST_MTD_DEPENDENCIES = host-zlib host-lzo host-e2fsprogs
 
 define HOST_MTD_BUILD_CMDS
 	$(HOST_CONFIGURE_OPTS) $(MAKE1) \
-		CROSS= BUILDDIR=$(@D) WITHOUT_XATTR=1 -C $(@D)
+		CROSS= BUILDDIR=$(@D) WITHOUT_XATTR=1 -C $(MTD_SRCDIR)
 endef
 
 define HOST_MTD_INSTALL_CMDS
-	$(MAKE1) BUILDDIR=$(@D) DESTDIR=$(HOST_DIR) -C $(@D) install
+	$(MAKE1) BUILDDIR=$(@D) DESTDIR=$(HOST_DIR) -C $(MTD_SRCDIR) install
 endef
 
 MKFS_JFFS2 = $(HOST_DIR)/usr/sbin/mkfs.jffs2
@@ -77,7 +79,7 @@ ifneq ($(MTD_TARGETS_y),)
 
 define MTD_BUILD_CMDS
 	$(TARGET_CONFIGURE_OPTS) $(MAKE1) CROSS=$(TARGET_CROSS) \
-		BUILDDIR=$(@D) WITHOUT_XATTR=1 WITHOUT_LARGEFILE=1 -C $(@D) \
+		BUILDDIR=$(@D) WITHOUT_XATTR=1 WITHOUT_LARGEFILE=1 -C $(MTD_SRCDIR) \
 		$(addprefix $(@D)/,$(MTD_TARGETS_y))
 endef
 
-- 
1.7.9.5

[Buildroot] [RFC v1 09/14] barebox: support out of tree build

[Thomas Petazzoni]

For Barebox, a minor patch is needed to make out-of-tree build
possible when the source directory is read-only. This patch can
probably be merged upstream.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 ...-build-when-source-directory-is-read-only.patch |   42 ++++++++++++++++++++
 boot/barebox/barebox.mk                            |    7 ++--
 2 files changed, 46 insertions(+), 3 deletions(-)
 create mode 100644 boot/barebox/barebox-genenv-fix-build-when-source-directory-is-read-only.patch

diff --git a/boot/barebox/barebox-genenv-fix-build-when-source-directory-is-read-only.patch b/boot/barebox/barebox-genenv-fix-build-when-source-directory-is-read-only.patch
new file mode 100644
index 0000000..29cb496
--- /dev/null
+++ b/boot/barebox/barebox-genenv-fix-build-when-source-directory-is-read-only.patch
@@ -0,0 +1,42 @@
+From e4bc9ba958c2aba5b2830724457f3a70f014cfc5 Mon Sep 17 00:00:00 2001
+From: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+Date: Sun, 20 Jan 2013 17:25:55 +0100
+Subject: [PATCH] genenv: fix build when source directory is read-only
+
+When doing an out-of-tree build where the source directory of Barebox
+is read-only (i.e all files have no write permission), the build of
+Barebox halts in scripts/genenv, because it removes files for which it
+doesn't have write permissions. Those files are being removed from the
+temporary directory .barebox_default_env/ once the environnement is
+generated. However, since the files have been copied from the source
+directory, they carry the permissions without write access. This leads
+a regular rm to wait interactively for the user to confirm the
+suppression, even those removal is possible since we have write access
+to the containing directory.
+
+To solve this, we explicitly re-add write permissions on the copied
+files and directories into this temporary directory.
+
+Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+---
+ scripts/genenv |    4 ++++
+ 1 file changed, 4 insertions(+)
+
+diff --git a/scripts/genenv b/scripts/genenv
+index 374db6d..3d79c63 100755
+--- a/scripts/genenv
++++ b/scripts/genenv
+@@ -24,6 +24,10 @@ for i in $*; do
+ done
+ )
+ 
++# Ensure we have write permissions, in case the source directory is
++# read-only.
++chmod +w -R $tempdir
++
+ find $tempdir -name '.svn' -o -name '*~' | xargs --no-run-if-empty rm -r
+ 
+ $objtree/scripts/bareboxenv -s $tempdir $target
+-- 
+1.7.9.5
+
diff --git a/boot/barebox/barebox.mk b/boot/barebox/barebox.mk
index 143338e..97acac5 100644
--- a/boot/barebox/barebox.mk
+++ b/boot/barebox/barebox.mk
@@ -23,6 +23,7 @@ endif
 BAREBOX_DEPENDENCIES = host-lzop
 BAREBOX_LICENSE = GPLv2 with exceptions
 BAREBOX_LICENSE_FILES = COPYING
+BAREBOX_SUPPORTS_OUT_OF_TREE = YES
 
 ifneq ($(call qstrip,$(BR2_TARGET_BAREBOX_CUSTOM_PATCH_DIR)),)
 define BAREBOX_APPLY_CUSTOM_PATCHES
@@ -50,14 +51,14 @@ BAREBOX_MAKE_FLAGS = ARCH=$(BAREBOX_ARCH) CROSS_COMPILE="$(CCACHE) $(TARGET_CROS
 
 
 ifeq ($(BR2_TARGET_BAREBOX_USE_DEFCONFIG),y)
-BAREBOX_SOURCE_CONFIG = $(@D)/arch/$(BAREBOX_ARCH)/configs/$(call qstrip,$(BR2_TARGET_BAREBOX_BOARD_DEFCONFIG))_defconfig
+BAREBOX_SOURCE_CONFIG = $(BAREBOX_SRCDIR)/arch/$(BAREBOX_ARCH)/configs/$(call qstrip,$(BR2_TARGET_BAREBOX_BOARD_DEFCONFIG))_defconfig
 else ifeq ($(BR2_TARGET_BAREBOX_USE_CUSTOM_CONFIG),y)
 BAREBOX_SOURCE_CONFIG = $(BR2_TARGET_BAREBOX_CUSTOM_CONFIG_FILE)
 endif
 
 define BAREBOX_CONFIGURE_CMDS
-	cp $(BAREBOX_SOURCE_CONFIG) $(@D)/arch/$(BAREBOX_ARCH)/configs/buildroot_defconfig
-	$(TARGET_MAKE_ENV) $(MAKE) $(BAREBOX_MAKE_FLAGS) -C $(@D) buildroot_defconfig
+	cp $(BAREBOX_SOURCE_CONFIG) $(@D)/.config
+	yes '' | $(TARGET_MAKE_ENV) $(MAKE) $(BAREBOX_MAKE_FLAGS) -C $(BAREBOX_SRCDIR) O=$(@D) oldconfig
 endef
 
 ifeq ($(BR2_TARGET_BAREBOX_BAREBOXENV),y)
-- 
1.7.9.5

[Buildroot] [RFC v1 10/14] uboot: support out of tree build

[Thomas Petazzoni]

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 boot/uboot/uboot.mk |    7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/boot/uboot/uboot.mk b/boot/uboot/uboot.mk
index ea77259..1a4f251 100644
--- a/boot/uboot/uboot.mk
+++ b/boot/uboot/uboot.mk
@@ -10,6 +10,7 @@ UBOOT_LICENSE = GPLv2+
 UBOOT_LICENSE_FILES = COPYING
 
 UBOOT_INSTALL_IMAGES = YES
+UBOOT_SUPPORTS_OUT_OF_TREE = YES
 
 ifeq ($(UBOOT_VERSION),custom)
 # Handle custom U-Boot tarballs as specified by the configuration
@@ -71,8 +72,8 @@ UBOOT_POST_PATCH_HOOKS += UBOOT_APPLY_CUSTOM_PATCHES
 endif
 
 define UBOOT_CONFIGURE_CMDS
-	$(TARGET_CONFIGURE_OPTS) $(UBOOT_CONFIGURE_OPTS) 	\
-		$(MAKE) -C $(@D) $(UBOOT_MAKE_OPTS)		\
+	$(TARGET_CONFIGURE_OPTS) $(UBOOT_CONFIGURE_OPTS) 		\
+		$(MAKE) -C $(UBOOT_SRCDIR) O=$(@D) $(UBOOT_MAKE_OPTS)	\
 		$(UBOOT_BOARD_NAME)_config
 	@echo >> $(@D)/include/config.h
 	@echo "/* Add a wrapper around the values Buildroot sets. */" >> $(@D)/include/config.h
@@ -91,7 +92,7 @@ endef
 
 define UBOOT_BUILD_CMDS
 	$(TARGET_CONFIGURE_OPTS) $(UBOOT_CONFIGURE_OPTS) 	\
-		$(MAKE) -C $(@D) $(UBOOT_MAKE_OPTS) 		\
+		$(MAKE) -C $(UBOOT_SRCDIR) O=$(@D) $(UBOOT_MAKE_OPTS) 		\
 		$(UBOOT_MAKE_TARGET)
 endef
 
-- 
1.7.9.5

[Buildroot] [RFC v1 11/14] linux: support out of tree build

[Thomas Petazzoni]

Supporting out of tree build in the Linux package requires a few
changes:

 * A clarification of KERNEL_ARCH_PATH, which used to point to
   $(O)/build/linux-<version>/arch/<architecture>/. Instead, we now
   have KERNEL_ARCH_SRCDIR (pointing to the architecture directory in
   the sources) and KERNEL_ARCH_BUILDDIR (pointing to the architectire
   directory in the build directory)

 * The custom Device Tree source file (if any) is now copied through a
   post-patch hook, rather than during the build step. This is needed
   because copying a Device Tree source file in the source tree
   requires the source tree to be writable, which is no longer the
   case after the patch step. And copying something into the source
   tree *is* a patching action, so doing this in the patch step looks
   like the right thing to do. The only consequence of this is that
   this action will no longer be done if LINUX_OVERRIDE_SRCDIR is
   defined, because the extract/patch steps are not done on packages
   for which the source directory is overriden. But it doesn't seem
   like a big issue since, if the user has overriden the kernel source
   directory, we assume that (s)he can store the Device Tree source
   file in the kernel source directory itself.

 * We no longer use the trick of creating a buildroot_defconfig, since
   this requires creating a buildroot_defconfig file in the source
   tree during the configure step. And it doesn't work with
   out-of-tree because the source tree is read only during the
   configure step (and onwards). So instead, we copy the defconfig to
   .config, and run a silent oldconfig on it.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 linux/linux.mk |   47 ++++++++++++++++++++++++++++-------------------
 1 file changed, 28 insertions(+), 19 deletions(-)

diff --git a/linux/linux.mk b/linux/linux.mk
index 91a9f50..f7adffd 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -35,6 +35,7 @@ LINUX_PATCHES = $(call qstrip,$(BR2_LINUX_KERNEL_PATCH))
 
 LINUX_INSTALL_IMAGES = YES
 LINUX_DEPENDENCIES  += host-module-init-tools
+LINUX_SUPPORTS_OUT_OF_TREE = YES
 
 ifeq ($(BR2_LINUX_KERNEL_UBOOT_IMAGE),y)
 	LINUX_DEPENDENCIES += host-uboot-tools
@@ -106,22 +107,25 @@ endif
 # for bzImage, arch/i386 and arch/x86_64 do not exist when copying the
 # defconfig file.
 ifeq ($(KERNEL_ARCH),i386)
-KERNEL_ARCH_PATH=$(LINUX_DIR)/arch/x86
+KERNEL_ARCH_SUBDIR=arch/x86
 else ifeq ($(KERNEL_ARCH),x86_64)
-KERNEL_ARCH_PATH=$(LINUX_DIR)/arch/x86
+KERNEL_ARCH_SUBDIR=arch/x86
 else
-KERNEL_ARCH_PATH=$(LINUX_DIR)/arch/$(KERNEL_ARCH)
+KERNEL_ARCH_SUBDIR=arch/$(KERNEL_ARCH)
 endif
 
+KERNEL_ARCH_SRCDIR=$(LINUX_SRCDIR)/$(KERNEL_ARCH_SUBDIR)
+KERNEL_ARCH_BUILDDIR=$(LINUX_BUILDDIR)/$(KERNEL_ARCH_SUBDIR)
+
 ifeq ($(BR2_LINUX_KERNEL_VMLINUX),y)
-LINUX_IMAGE_PATH=$(LINUX_DIR)/$(LINUX_IMAGE_NAME)
+LINUX_IMAGE_PATH=$(LINUX_BUILDDIR)/$(LINUX_IMAGE_NAME)
 else ifeq ($(BR2_LINUX_KERNEL_VMLINUZ),y)
-LINUX_IMAGE_PATH=$(LINUX_DIR)/$(LINUX_IMAGE_NAME)
+LINUX_IMAGE_PATH=$(LINUX_BUILDDIR)/$(LINUX_IMAGE_NAME)
 else
 ifeq ($(KERNEL_ARCH),avr32)
-LINUX_IMAGE_PATH=$(KERNEL_ARCH_PATH)/boot/images/$(LINUX_IMAGE_NAME)
+LINUX_IMAGE_PATH=$(KERNEL_ARCH_BUILDDIR)/boot/images/$(LINUX_IMAGE_NAME)
 else
-LINUX_IMAGE_PATH=$(KERNEL_ARCH_PATH)/boot/$(LINUX_IMAGE_NAME)
+LINUX_IMAGE_PATH=$(KERNEL_ARCH_BUILDDIR)/boot/$(LINUX_IMAGE_NAME)
 endif
 endif # BR2_LINUX_KERNEL_VMLINUX
 
@@ -150,15 +154,22 @@ LINUX_POST_PATCH_HOOKS += LINUX_APPLY_PATCHES
 
 
 ifeq ($(BR2_LINUX_KERNEL_USE_DEFCONFIG),y)
-KERNEL_SOURCE_CONFIG = $(KERNEL_ARCH_PATH)/configs/$(call qstrip,$(BR2_LINUX_KERNEL_DEFCONFIG))_defconfig
+KERNEL_SOURCE_CONFIG = $(KERNEL_ARCH_SRCDIR)/configs/$(call qstrip,$(BR2_LINUX_KERNEL_DEFCONFIG))_defconfig
 else ifeq ($(BR2_LINUX_KERNEL_USE_CUSTOM_CONFIG),y)
 KERNEL_SOURCE_CONFIG = $(BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE)
 endif
 
+define LINUX_COPY_DTS
+	cp $(BR2_LINUX_KERNEL_CUSTOM_DTS_PATH) $(KERNEL_ARCH_SRCDIR)/boot/dts/)
+endef
+
+ifneq ($(BR2_LINUX_KERNEL_USE_CUSTOM_DTS),)
+LINUX_POST_PATCH_HOOKS += LINUX_COPY_DTS
+endif
+
 define LINUX_CONFIGURE_CMDS
-	cp $(KERNEL_SOURCE_CONFIG) $(KERNEL_ARCH_PATH)/configs/buildroot_defconfig
-	$(TARGET_MAKE_ENV) $(MAKE1) $(LINUX_MAKE_FLAGS) -C $(@D) buildroot_defconfig
-	rm $(KERNEL_ARCH_PATH)/configs/buildroot_defconfig
+	cp $(KERNEL_SOURCE_CONFIG) $(@D)/.config
+	yes '' | $(TARGET_MAKE_ENV) $(MAKE1) $(LINUX_MAKE_FLAGS) -C $(LINUX_SRCDIR) O=$(@D) oldconfig
 	$(if $(BR2_ARM_EABI),
 		$(call KCONFIG_ENABLE_OPT,CONFIG_AEABI,$(@D)/.config),
 		$(call KCONFIG_DISABLE_OPT,CONFIG_AEABI,$(@D)/.config))
@@ -194,8 +205,8 @@ endef
 define LINUX_INSTALL_DTB
 	# dtbs moved from arch/$ARCH/boot to arch/$ARCH/boot/dts since 3.8-rc1
 	cp $(addprefix \
-		$(KERNEL_ARCH_PATH)/boot/$(if $(wildcard \
-		$(addprefix $(KERNEL_ARCH_PATH)/boot/dts/,$(KERNEL_DTBS))),dts/),$(KERNEL_DTBS)) \
+		$(KERNEL_ARCH_BUILDDIR)/boot/$(if $(wildcard \
+		$(addprefix $(KERNEL_ARCH_BUILDDIR)/boot/dts/,$(KERNEL_DTBS))),dts/),$(KERNEL_DTBS)) \
 		$(BINARIES_DIR)/
 endef
 endif
@@ -204,11 +215,11 @@ endif
 ifeq ($(BR2_LINUX_KERNEL_APPENDED_DTB),y)
 # dtbs moved from arch/$ARCH/boot to arch/$ARCH/boot/dts since 3.8-rc1
 define LINUX_APPEND_DTB
-	if [ -e $(KERNEL_ARCH_PATH)/boot/$(KERNEL_DTS_NAME).dtb ]; then \
-		cat $(KERNEL_ARCH_PATH)/boot/$(KERNEL_DTS_NAME).dtb; \
+	if [ -e $(KERNEL_ARCH_BUILDDIR)/boot/$(KERNEL_DTS_NAME).dtb ]; then \
+		cat $(KERNEL_ARCH_BUILDDIR)/boot/$(KERNEL_DTS_NAME).dtb; \
 	else \
-		cat $(KERNEL_ARCH_PATH)/boot/dts/$(KERNEL_DTS_NAME).dtb; \
-	fi >> $(KERNEL_ARCH_PATH)/boot/zImage
+		cat $(KERNEL_ARCH_BUILDDIR)/boot/dts/$(KERNEL_DTS_NAME).dtb; \
+	fi >> $(KERNEL_ARCH_BUILDDIR)/boot/zImage
 endef
 ifeq ($(BR2_LINUX_KERNEL_APPENDED_UIMAGE),y)
 # We need to generate the uImage here after that so that the uImage is
@@ -220,8 +231,6 @@ endif
 # Compilation. We make sure the kernel gets rebuilt when the
 # configuration has changed.
 define LINUX_BUILD_CMDS
-	$(if $(BR2_LINUX_KERNEL_USE_CUSTOM_DTS),
-		cp $(BR2_LINUX_KERNEL_CUSTOM_DTS_PATH) $(KERNEL_ARCH_PATH)/boot/dts/)
 	$(TARGET_MAKE_ENV) $(MAKE) $(LINUX_MAKE_FLAGS) -C $(@D) $(LINUX_IMAGE_TARGET)
 	@if grep -q "CONFIG_MODULES=y" $(@D)/.config; then 	\
 		$(TARGET_MAKE_ENV) $(MAKE) $(LINUX_MAKE_FLAGS) -C $(@D) modules ;	\
-- 
1.7.9.5

[Buildroot] [RFC v1 12/14] autoconf: fix out-of-tree build

[Thomas Petazzoni]

Even though one might think that autoconf should have been the best
example of a clean and nice example of a correct out of tree build
implementation, this is not the case. The autoconf build process makes
modification to the source tree, for some obscure updating process of
manual pages.

For now, we workaround this problem by re-adding write permissions on
the man/ subdirectory of autoconf sources, but on the long term, a
proper, upstreamable solution would be better.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/autoconf/autoconf.mk |   10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/package/autoconf/autoconf.mk b/package/autoconf/autoconf.mk
index 8807c10..e77bb54 100644
--- a/package/autoconf/autoconf.mk
+++ b/package/autoconf/autoconf.mk
@@ -20,6 +20,16 @@ HOST_AUTOCONF_CONF_ENV = ac_cv_path_M4=$(HOST_DIR)/usr/bin/m4 \
 
 HOST_AUTOCONF_DEPENDENCIES = host-m4 host-libtool
 
+# The autoconf build process writes to its source directory, which is
+# very stupid, but forces us to give write permissions to one
+# directory.
+define AUTOCONF_FIXUP_PERMISSIONS
+	chmod u+w -R $(AUTOCONF_SRCDIR)/man
+endef
+
+AUTOCONF_PRE_CONFIGURE_HOOKS += AUTOCONF_FIXUP_PERMISSIONS
+HOST_AUTOCONF_PRE_CONFIGURE_HOOKS += AUTOCONF_FIXUP_PERMISSIONS
+
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
-- 
1.7.9.5

[Buildroot] [RFC v1 13/14] cmake: support out of tree build

[Thomas Petazzoni]

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/cmake/cmake.mk |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/package/cmake/cmake.mk b/package/cmake/cmake.mk
index e2bc31f..ff586bf 100644
--- a/package/cmake/cmake.mk
+++ b/package/cmake/cmake.mk
@@ -1,11 +1,13 @@
 CMAKE_VERSION = 2.8.9
 CMAKE_SITE = http://www.cmake.org/files/v2.8/
 
+CMAKE_SUPPORTS_OUT_OF_TREE = YES
+
 define HOST_CMAKE_CONFIGURE_CMDS
  (cd $(@D); \
 	LDFLAGS="$(HOST_LDFLAGS)" \
 	CFLAGS="$(HOST_CFLAGS)" \
-	./bootstrap --prefix=$(HOST_DIR)/usr --parallel=$(PARALLEL_JOBS) \
+	$(CMAKE_SRCDIR)/bootstrap --prefix=$(HOST_DIR)/usr --parallel=$(PARALLEL_JOBS) \
  )
 endef
 
-- 
1.7.9.5

[Buildroot] [RFC v1 14/14] makedevs: support out of tree build

[Thomas Petazzoni]

Nothing needs to be done, the source is already directly taken from
package/makedevs/, so it just works.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/makedevs/makedevs.mk |    2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/makedevs/makedevs.mk b/package/makedevs/makedevs.mk
index 5e984c6..1e28ff7 100644
--- a/package/makedevs/makedevs.mk
+++ b/package/makedevs/makedevs.mk
@@ -8,6 +8,8 @@
 MAKEDEVS_SOURCE =
 HOST_MAKEDEVS_SOURCE =
 
+MAKEDEVS_SUPPORTS_OUT_OF_TREE = YES
+
 define MAKEDEVS_BUILD_CMDS
 	$(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS) \
 		package/makedevs/makedevs.c -o $(@D)/makedevs
-- 
1.7.9.5

[Buildroot] [RFC v1] Prototype implementation of per-package out oftree build

[Przemyslaw Wrzos]

> -----Original Message-----
> From: buildroot-bounces at busybox.net [mailto:buildroot-
> bounces at busybox.net] On Behalf Of Thomas Petazzoni
> Sent: Monday, 21 January 2013 10:52
> To: buildroot at uclibc.org
> Subject: [Buildroot] [RFC v1] Prototype implementation of per-
> package out oftree build
> 
> Hello,
> 
> (Sorry for the previous posting that only had the first two patches. I
> was in the train in the middle of the mountains, and the 3G
> connection was interrupted for a too long time.)
> 
> There has recently been some interest in improving the support for
> overriden packages by doing out of tree build of all packages. Since I
> had the beginning of a prototype working, I cleaned it up, made
> numerous fixes to make it work apparently properly, and share it with
> the community.
> 
> The following patches should be considered as a prototype, I haven't
> tested all packages (of course), and there are quite certainly some
> corner case issues that haven't been solved. However, I think it's a
> good time to get more testing in order to find, and solve those
issues.
> 
> It should also not be considered yet that we want this feature in
> Buildroot. It is again only a prototype, which aims at solving the
> remaining problems, and see if we think it is a good idea to have this
> in Buildroot or not.
> 
> For those who want to test, the 'out-of-tree' branch is available at:
> 
>   git://git.free-electrons.com/users/thomas-petazzoni/buildroot.git
> 
> Here is a quick summary of what this patch set does:
> 
>  * The tarballs are now extracted in $(O)/src/<pkg>-<version>
> 
>  * Each tarball is extracted and patched only once, even if there is
>    both a target build and a host build. Once a package has been
>    extracted and patched, its source code is changed to not have write
>    permissions, in order to catch misbehaving packages in terms of out
>    of tree build.
> 
>  * The builds continue to be done in $(O)/build/<pkg>-<version> for
>    target packages and $(O)/build/host-<pkg>-<version> for host
>    packages.
> 
>  * A <pkg>_SUPPORTS_OUT_OF_TREE variable defines whether the
> package
>    supports out of tree build or not. By default, it's set to NO, in
>    which case the package source code is rsynced from its source
>    directory to the build directory at the beginning of the configure
>    step. It is set to YES by both the autotools and cmake package
>    infrastructures (which have been modified to do out of tree build),
>    and by some packages that use the generic infrastructure but
>    nonetheless support out of tree build.
> 
>  * The override source directory mechanism is updated to take
>    advantage of this out of tree support. Now, if you override a
>    package that supports out of tree build, the source code is no
>    longer rsynced to the build directory. This makes the override
>    source directory mechanism a lot more usable for big packages such
>    as the Linux kernel.
> 
> Amongst the known remaining issues to solve are:
> 
>  * The semantic of the 'make <pkg>-dirclean' command. Should it
> remove
>    the build directory and the source directory? Only the build
>    directory? If it removes both (which the proposed prototype
>    currently does), it is strange because you remove one build variant
>    (host or target) and the source directory, but the other build
>    variant (target or host) remains.
> 
>  * I am not sure on how we want to handle <pkg>-reconfigure and
>    <pkg>-rebuild. I guess we want to tell the user to do his changes
>    in the package source directory, output/src/<pkg>-<version>. But
>    all files in this source directory are marked read-only, so it's
>    not very practical.
> 
>    In addition to this, for packages that don't support out of tree
>    build, we do a rsync at the beginning of the configure step. So if
>    the user does "make <pkg>-reconfigure", then the source code gets
>    rsynced again from the source directory to the build directory, but
>    not if the user does "make <pkg>-rebuild".
> 
> Comments, ideas, suggestions are welcome.
> 
> Best regards,
> 
> Thomas


Hi Thomas,

This looks fairly similar to what I was proposing so I'm personally all
for it. Here are a few of my thoughts about it:

* There's one more feature which I think would complete the Linux
out-of-tree build support for me, and that is the ability to specify the
<PKG>_OVERRIDE_SRCDIR value via the configuration. I've attached a patch
for how this might look.
* I like the idea making the source tree read-only to keep the build
honest but I think this removes one of the major benefits of having a
source tree in the first place, that is being able to modify and play
around with it. This leads to the <pkg>-rebuild not making sense as you
point out. I think it might be best to simply not enforce this
restriction (even if it means some packages could abuse that).
* Do we need a source directory when a package doesn't support
out-of-tree building? The downside is additional disk space though that
might not be that serious of a concern and it might just come down to a
choice of aesthetics I suppose.
* As far as <pkg>-dirclean goes, I would say remove both build and
source directories. If we only remove the build directory it will in
effect not be all that different to <pkg>-clean. The only other real
option would be to have a separate source directory for each target
variant.

Cheers,
Przemyslaw Wrzos
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 001-linux-external-source-dir.patch
Type: application/octet-stream
Size: 2748 bytes
Desc: 001-linux-external-source-dir.patch
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20130121/9c9d6a30/attachment.obj>

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Jérôme Pouiller]

Hello Thomas,

(I repost my remarks there in order to keep all discussion in same thread)

On Monday 21 January 2013 00:52:10 Thomas Petazzoni wrote:
> This new directory will be used to extract the source directory of the
> different packages, as part of the out of tree support.
> 
> Note that we need to explicitly re-add write permissions on the
> contents of this directory, because write permissions are removed from
> the source code of packages in order to ensure that the packages are
> not incorrectly modifying their source directory during an out of tree
> build. We also re-add write permissions to the build directory since
> some packages copy files from their source directory to the build
> directory, causing rm to fail due to lack of permissions.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  Makefile |    8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 6f8ed0e..15ce4e4 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -260,6 +260,7 @@ GENERATE_LOCALE=$(call qstrip,$(BR2_GENERATE_LOCALE))
>  STAMP_DIR:=$(BASE_DIR)/stamps
> 
>  BINARIES_DIR:=$(BASE_DIR)/images
> +SRC_DIR:=$(BASE_DIR)/src
IMHO, $(SRC_DIR) should be in $(TOPDIR) to be shared between different 
configurations. In add $(SRC_DIR) have to customizable in case I would like to 
clearly separate two projects.

I think, my workspace would look like this:
  br/srcs-projet1
  br/srcs-projet2
  br/output-project1-configA
  br/output-project1-configB
  br/output-project2-configA
  br/output-project2-configB

Finally SRC_DIR should be handled like DL_DIR.

[...]

-- 
J?r?me Pouiller

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Thomas Petazzoni]

Dear J?r?me Pouiller,

On Mon, 21 Jan 2013 14:47:22 +0100, J?r?me Pouiller wrote:

> IMHO, $(SRC_DIR) should be in $(TOPDIR) to be shared between
> different configurations. In add $(SRC_DIR) have to customizable in
> case I would like to clearly separate two projects.

I have indeed thought about making the source code shareable across
rebuilds and across projects, but since we apply patches to the source
code, it seems a bit complicated to do in a reliable way.

What happens when you bump your Buildroot version and the patches for a
given package change? You would have to manually clean up the source
directories. Not sure we want to do that, at least for now, but the
opinion of others would be interesting here.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFC v1] Prototype implementation of per-package out oftree build

[Thomas Petazzoni]

Dear Przemyslaw Wrzos,

On Mon, 21 Jan 2013 15:24:32 +1100, Przemyslaw Wrzos wrote:

> This looks fairly similar to what I was proposing so I'm personally
> all for it. Here are a few of my thoughts about it:

Yes, indeed it looks similar, but I had already started working on this
prototype some time ago, and I had the feeling that it was already a
bit more complete (support in the autotools and cmake infrastructure
for out of tree build, etc.).

> * There's one more feature which I think would complete the Linux
> out-of-tree build support for me, and that is the ability to specify
> the <PKG>_OVERRIDE_SRCDIR value via the configuration. I've attached
> a patch for how this might look.

In general, I am not really favorable. You want it for Linux, but then
the next guy will want it for Qt, the other guy for X.org server, the
other guy for OpenSSH, etc. And then we have gazillions of
configuration options to customize the source directory... while we
have the much more generic OVERRIDE_SRCDIR mechanism. I really would
like to avoid the profusion of such configuration options.

That said, for packages like Linux, U-Boot or Barebox, we already give
the possibility of building from a custom Git tree or a custom tarball
location, so we could decide that those packages are "special", and we
could therefore add the "custom source directory location" option to
them.

But I really want to put a limit to the type of packages on which we
add such options, because I don't want to see this being added more or
less randomly to each and every package in Buildroot.

> * I like the idea making the source tree read-only to keep the build
> honest but I think this removes one of the major benefits of having a
> source tree in the first place, that is being able to modify and play
> around with it. This leads to the <pkg>-rebuild not making sense as
> you point out. I think it might be best to simply not enforce this
> restriction (even if it means some packages could abuse that).

Indeed, as I pointed in my introduction e-mail, having the source
directory read-only has some drawbacks. However, since the source
directory is shared between the host build and target build of a given
package, ensuring that the configuration/build steps of the package do
not mess with the source code is quite important. Otherwise, we could
see cases of pollution of the host build into the target build or vice
versa.

> * Do we need a source directory when a package doesn't support
> out-of-tree building? The downside is additional disk space though
> that might not be that serious of a concern and it might just come
> down to a choice of aesthetics I suppose.

The idea with this out-of-tree support is to make the "overriden
packages" more "normal". So if you add another special case for
packages that don't support out of tree build, the entire goal is a bit
lost.

For example, will you tell users that if they want to do a quick
and dirty modification of a package source code, they should go in
$(O)/src/foo-1.0 if foo supports out of tree build or
$(O)/build/foo-1.0 if foo doesn't? Seems really not logical.

In general, I have the feeling that the biggest packages use a
reasonable build system, and therefore are capable of doing out of tree
build.

> * As far as <pkg>-dirclean goes, I would say remove both build and
> source directories. If we only remove the build directory it will in
> effect not be all that different to <pkg>-clean. The only other real
> option would be to have a separate source directory for each target
> variant.

The problem is that if you have a package that gets build for the host
and the target, you have:

	output/src/foo-1.0
	output/build/foo-1.0
	output/build/host-foo-1.0

Then, you do:

	make foo-dirclean

So, output/src/foo-1.0 and output/build/foo-1.0 are removed, but
output/build/host-foo-1.0 remains. So we have the build directory for a
given package but no longer the corresponding source directory. Doesn't
seem to be really nice. No?

Thanks a lot for your very interesting feedback on this, good to see
that the discussion is starting!

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFC v1] Prototype implementation of per-package out oftree build

[Stephan Hoffmann]

Am 21.01.2013 18:26, schrieb Thomas Petazzoni:
> Dear Przemyslaw Wrzos,
>
> On Mon, 21 Jan 2013 15:24:32 +1100, Przemyslaw Wrzos wrote:
>
>> This looks fairly similar to what I was proposing so I'm personally
>> all for it. Here are a few of my thoughts about it:
> Yes, indeed it looks similar, but I had already started working on this
> prototype some time ago, and I had the feeling that it was already a
> bit more complete (support in the autotools and cmake infrastructure
> for out of tree build, etc.).
>
>> * There's one more feature which I think would complete the Linux
>> out-of-tree build support for me, and that is the ability to specify
>> the <PKG>_OVERRIDE_SRCDIR value via the configuration. I've attached
>> a patch for how this might look.
> In general, I am not really favorable. You want it for Linux, but then
> the next guy will want it for Qt, the other guy for X.org server, the
> other guy for OpenSSH, etc. And then we have gazillions of
> configuration options to customize the source directory... while we
> have the much more generic OVERRIDE_SRCDIR mechanism. I really would
> like to avoid the profusion of such configuration options.
>
> That said, for packages like Linux, U-Boot or Barebox, we already give
> the possibility of building from a custom Git tree or a custom tarball
> location, so we could decide that those packages are "special", and we
> could therefore add the "custom source directory location" option to
> them.
>
> But I really want to put a limit to the type of packages on which we
> add such options, because I don't want to see this being added more or
> less randomly to each and every package in Buildroot.
Once upon a time I thought of such a feature, but I don't think it can
be done in a good way. And since we have the OVERRIDE, I do not see any
need for it. Maybe even some of the special handlings for Linux, U-Boot
and Barebox could be deprecated, but I did not think about this in detail.

Maybe it's worth thinking about extending the OVERRIDE mechanism to
override SOURCE and SITE, too.
>
>> * I like the idea making the source tree read-only to keep the build
>> honest but I think this removes one of the major benefits of having a
>> source tree in the first place, that is being able to modify and play
>> around with it. This leads to the <pkg>-rebuild not making sense as
>> you point out. I think it might be best to simply not enforce this
>> restriction (even if it means some packages could abuse that).
> Indeed, as I pointed in my introduction e-mail, having the source
> directory read-only has some drawbacks. However, since the source
> directory is shared between the host build and target build of a given
> package, ensuring that the configuration/build steps of the package do
> not mess with the source code is quite important. Otherwise, we could
> see cases of pollution of the host build into the target build or vice
> versa.
>
>> * Do we need a source directory when a package doesn't support
>> out-of-tree building? The downside is additional disk space though
>> that might not be that serious of a concern and it might just come
>> down to a choice of aesthetics I suppose.
> The idea with this out-of-tree support is to make the "overriden
> packages" more "normal". So if you add another special case for
> packages that don't support out of tree build, the entire goal is a bit
> lost.
>
> For example, will you tell users that if they want to do a quick
> and dirty modification of a package source code, they should go in
> $(O)/src/foo-1.0 if foo supports out of tree build or
> $(O)/build/foo-1.0 if foo doesn't? Seems really not logical.
>
> In general, I have the feeling that the biggest packages use a
> reasonable build system, and therefore are capable of doing out of tree
> build.
>
>> * As far as <pkg>-dirclean goes, I would say remove both build and
>> source directories. If we only remove the build directory it will in
>> effect not be all that different to <pkg>-clean. The only other real
>> option would be to have a separate source directory for each target
>> variant.
> The problem is that if you have a package that gets build for the host
> and the target, you have:
>
> 	output/src/foo-1.0
> 	output/build/foo-1.0
> 	output/build/host-foo-1.0
>
> Then, you do:
>
> 	make foo-dirclean
>
> So, output/src/foo-1.0 and output/build/foo-1.0 are removed, but
> output/build/host-foo-1.0 remains. So we have the build directory for a
> given package but no longer the corresponding source directory. Doesn't
> seem to be really nice. No?
To me it would seem most natural if output/build/host-foo-1.0 would be
removed, too.

Similar issues might occur with foo-reconfigure and friends. If they
share one source they would both have to be updated in such a case.

Regards

Stephan
>
> Thanks a lot for your very interesting feedback on this, good to see
> that the discussion is starting!
>
> Thanks,
>
> Thomas


-- 
reLinux     -    Stephan Hoffmann
Am Schmidtgrund 124    50765 K?ln
Tel. +49.221.95595-19    Fax: -64
www.reLinux.de     sho at reLinux.de

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Jérôme Pouiller]

On Monday 21 January 2013 18:16:40 Thomas Petazzoni wrote:
> Dear J?r?me Pouiller,
> 
> On Mon, 21 Jan 2013 14:47:22 +0100, J?r?me Pouiller wrote:
> > IMHO, $(SRC_DIR) should be in $(TOPDIR) to be shared between
> > different configurations. In add $(SRC_DIR) have to customizable in
> > case I would like to clearly separate two projects.
> 
> I have indeed thought about making the source code shareable across
> rebuilds and across projects, but since we apply patches to the source
> code, it seems a bit complicated to do in a reliable way.
Hmm... If source is not shared across builds, what is advantage of build out 
of sources? 

I see two advantages:
  - No duplication of source between host and target. But finaly, only a few 
(and small) package are concerned. Shall we care about this?
  - To make build of a package outside of Buildroot easier. In this case, did 
you consider to build in a subdirectory of source? It does not allow to share 
source between host and target but may be sufficient and may solve a bunch of 
questions.

> What happens when you bump your Buildroot version and the patches for a
> given package change? You would have to manually clean up the source
> directories. Not sure we want to do that, at least for now, but the
> opinion of others would be interesting here.
Currently, when you bump your version of Buildroot, you have to clean all. 
Buildroot may provide a rule to also cleaning source directory. 



Regards,

-- 
J?r?me Pouiller

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Thomas Petazzoni]

Dear J?r?me Pouiller,

On Tue, 22 Jan 2013 11:15:11 +0100, J?r?me Pouiller wrote:

> Hmm... If source is not shared across builds, what is advantage of build out 
> of sources? 
> 
> I see two advantages:
>   - No duplication of source between host and target. But finaly, only a few 
> (and small) package are concerned. Shall we care about this?
>   - To make build of a package outside of Buildroot easier. In this case, did 
> you consider to build in a subdirectory of source? It does not allow to share 
> source between host and target but may be sufficient and may solve a bunch of 
> questions.

This last point is the main reason. For now, the LINUX_OVERRIDE_SRCDIR
mechanism is barely usable because it does a complete rsync of the
Linux source tree, which is very annoying. So, switching all packages
to out of tree build sounds like an interesting solution.

> > What happens when you bump your Buildroot version and the patches for a
> > given package change? You would have to manually clean up the source
> > directories. Not sure we want to do that, at least for now, but the
> > opinion of others would be interesting here.
> Currently, when you bump your version of Buildroot, you have to clean all. 
> Buildroot may provide a rule to also cleaning source directory. 

In my opinion, it makes the whole thing a lot more complicated to
understand. At the moment:

	make clean = removes the output directory

That's explained in just one easy line.

If we start putting the source trees outside of the output directory,
things get a lot more complicated for new users. They have to
understand there is one tree for the build results, one tree for the
sources, understand when and why to clean which of the trees and so on.
To me, it's not ok.

Remember, simplicity of use is a *key* point of Buildroot. We are very
attentive about this.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Jérôme Pouiller]

Hello Thomas,

On Tuesday 22 January 2013 17:49:02 Thomas Petazzoni wrote:
> Dear J?r?me Pouiller,
> 
> On Tue, 22 Jan 2013 11:15:11 +0100, J?r?me Pouiller wrote:
> > Hmm... If source is not shared across builds, what is advantage of build
> > out of sources?
> > 
> > I see two advantages:
> >   - No duplication of source between host and target. But finaly, only a
> >   few
> > 
> > (and small) package are concerned. Shall we care about this?
> > 
> >   - To make build of a package outside of Buildroot easier. In this case,
> >   did
> > 
> > you consider to build in a subdirectory of source? It does not allow to
> > share source between host and target but may be sufficient and may solve
> > a bunch of questions.
> 
> This last point is the main reason. For now, the LINUX_OVERRIDE_SRCDIR
> mechanism is barely usable because it does a complete rsync of the
> Linux source tree, which is very annoying. So, switching all packages
> to out of tree build sounds like an interesting solution.
And what about building in a subdirectory of source directory ? 

[...]
> Remember, simplicity of use is a *key* point of Buildroot. We are very
> attentive about this.
Ok, I share your point of view about simplicity. 

Regards,

-- 
J?r?me Pouiller

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Thomas Petazzoni]

Dear J?r?me Pouiller,

On Tue, 22 Jan 2013 18:12:49 +0100, J?r?me Pouiller wrote:

> > This last point is the main reason. For now, the
> > LINUX_OVERRIDE_SRCDIR mechanism is barely usable because it does a
> > complete rsync of the Linux source tree, which is very annoying.
> > So, switching all packages to out of tree build sounds like an
> > interesting solution.
> And what about building in a subdirectory of source directory ? 

I am not sure why you think it would solve any problem, but it is
anyway not possible for packages for which <pkg>_OVERRIDE_SRCDIR is
used: we certainly do *not* want to build in a subdirectory of the
source directory.

Best regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFC v1 01/14] Add a new "src" directory in the output directory

[Jérôme Pouiller]

Hello Thomas,

On Wednesday 23 January 2013 16:45:36 Thomas Petazzoni wrote:
> Dear J?r?me Pouiller,
> 
> On Tue, 22 Jan 2013 18:12:49 +0100, J?r?me Pouiller wrote:
> > > This last point is the main reason. For now, the
> > > LINUX_OVERRIDE_SRCDIR mechanism is barely usable because it does a
> > > complete rsync of the Linux source tree, which is very annoying.
> > > So, switching all packages to out of tree build sounds like an
> > > interesting solution.
> > 
> > And what about building in a subdirectory of source directory ?
> 
> I am not sure why you think it would solve any problem, but it is
> anyway not possible for packages for which <pkg>_OVERRIDE_SRCDIR is
> used: we certainly do *not* want to build in a subdirectory of the
> source directory.
Mainly because packages not compiling out-of-source won't need to be rsynced 
between SRCDIR ans BUILDDIR.

Next it may solve questions about of dirclean behaviour.

Finally because I like idea that a directory contains all files related to a 
package.

Sure, overrided packages have be to handled differently. 

But it is only an idea, not sure it is a good idea :-) .

-- 
J?r?me Pouiller

[Buildroot] [RFC v1] Prototype implementation of per-package out of tree build

[Arnout Vandecappelle]

On 01/21/13 00:52, Thomas Petazzoni wrote:
[snip]
>   * The semantic of the 'make <pkg>-dirclean' command. Should it remove
>     the build directory and the source directory? Only the build
>     directory? If it removes both (which the proposed prototype
>     currently does), it is strange because you remove one build variant
>     (host or target) and the source directory, but the other build
>     variant (target or host) remains.

  I think the typical usecase for <pkg>-dirclean is when the patches have 
changed. So clearly the source should be removed as well. As mentioned by 
Stephan, Ideally the host-<pkg> build dir should be removed as well, but 
I think this is not that important because the newly patched files will 
have a more recent timestamp so the corresponding host-binaries should be 
rebuilt automatically.


>   * I am not sure on how we want to handle <pkg>-reconfigure and
>     <pkg>-rebuild. I guess we want to tell the user to do his changes
>     in the package source directory, output/src/<pkg>-<version>. But
>     all files in this source directory are marked read-only, so it's
>     not very practical.

  First of all, <pkg>-re{configure,build} is often used with 
OVERRIDE_SRCDIR so there you don't have a problem.

  If people do edit directly in the output/src directory, I think it is 
_very_ good that they have to 'chmod +w' first, as a reminder that 
editing in output/src is a bit risky. And once you've chmod'ed it, it 
stays writable because the chmod a-w is done at the end of the patch step.

  So I don't think there's an issue for <pkg>-re{configure,build}.


>     In addition to this, for packages that don't support out of tree
>     build, we do a rsync at the beginning of the configure step. So if
>     the user does "make <pkg>-reconfigure", then the source code gets
>     rsynced again from the source directory to the build directory, but
>     not if the user does "make <pkg>-rebuild".

  That's not so good. It could be solved by using the same mechanism that 
is currently used by OVERRIDE_SRCDIR: add an intermediate <pkg>-rsync 
target and stamp file, and remove the stamp file in 
<pkg>-clean-for-re{configure,build}.


  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC v1 03/14] generic infrastructure: out of tree build support

[Arnout Vandecappelle]

On 01/21/13 00:52, Thomas Petazzoni wrote:
[snip]
>   # Patch
>   #
>   # The RAWNAME variable is the lowercased package name, which allows to
>   # find the package directory (typically package/<pkgname>) and the
>   # prefix of the patches
> -$(BUILD_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION)
> -$(BUILD_DIR)/%/.stamp_patched:
> +$(SRC_DIR)/%/.stamp_patched: NAMEVER = $(RAWNAME)-$($(PKG)_VERSION)
> +$(SRC_DIR)/%/.stamp_patched:
>   	@$(call MESSAGE,"Patching $($(PKG)_DIR_PREFIX)/$(RAWNAME)")
>   	$(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep))
>   	$(foreach p,$($(PKG)_PATCH),support/scripts/apply-patches.sh $(@D) $(DL_DIR) $(p)$(sep))
> @@ -100,9 +81,15 @@ $(BUILD_DIR)/%/.stamp_patched:
>   	)
>   	$(foreach hook,$($(PKG)_POST_PATCH_HOOKS),$(call $(hook))$(sep))

  Ouch, we forgot something: there may be a HOST_FOO_PRE_PATCH_HOOKS, 
HOST_FOO_PATCH or HOST_FOO_POST_PATCH_HOOKS that is different from its 
non-host equivalent... Perhaps there should be a check that they are the 
same (or the host version is not defined).


>   	$(Q)touch $@
> +	chmod a-w -R $(@D)

  I would do this before touching the stamp file.

[snip]
> +# Handle the SOURCE_CHECK and SHOW_EXTERNAL_DEPS cases for overriden
> +# packages
> +$(BUILD_DIR)/%/.stamp_override_sourced:
> +ifeq ($(DL_MODE),SOURCE_CHECK)
> +	test -d $($(PKG)_SRCDIR)
> +else ifeq ($(DL_MODE),SHOW_EXTERNAL_DEPS)
> +	echo "file://$($(PKG)_SRCDIR)"
> +else
> +	@true # Nothing to do to source a local package
> +endif

  Why did you move this down? It was better when it was close to the rest 
of the source stuff.

> +
>   ################################################################################
>   # inner-generic-package -- generates the make targets needed to build a
>   # generic package
> @@ -215,11 +218,10 @@ else
>     $(2)_VERSION = $(subst /,_,$($(2)_VERSION))
>   endif
>
> -$(2)_BASE_NAME	=  $(1)-$$($(2)_VERSION)
> +$(2)_BASE_NAME	   =  $$($(2)_NAME)-$$($(2)_VERSION)
> +$(2)_RAW_BASE_NAME = $$($(2)_RAWNAME)-$$($(2)_VERSION)

  Whitespace seems to be a bit messed up here. Or it could just be the 
mixing of tabs and spaces.

>   $(2)_DL_DIR	=  $$(DL_DIR)/$$($(2)_BASE_NAME)
> -$(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)

  Err, $(PKG)_DIR is used in a lot of places: barebox, uboot, linux, 
ocf-linux, rtai, xenomai, bash, busybox, host-dbus, dhcp, dmalloc, 
dnsmasq, fbv, file, hostapd, ... OK I'm tired of typing all this, just do
git grep '[A-Z0-9]_DIR\b' -- \*.mk | sed 
's/\(TARGET\|HOST\|STAGING\|DL\)_DIR//g' | grep '[A-Z0-9]_DIR\b'

  I think these should be fixed up first in separate patches (as far as 
possible). It's not always clear whether they should be changed into the 
SRCDIR or BUILDDIR.


  Regards,
  Arnout

> -$(2)_SRCDIR		       = $$($(2)_DIR)
> -$(2)_BUILDDIR		       ?= $$($(2)_SRCDIR)
> +$(2)_BUILDDIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
>
>   ifneq ($$($(2)_OVERRIDE_SRCDIR),)
>   $(2)_VERSION = custom
[snip]



-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC v1 04/14] autotools infrastructure: support out of tree build

[Arnout Vandecappelle]

On 01/21/13 00:52, Thomas Petazzoni wrote:
> All autotools packages are supposed to support out of tree build, so
> we mark this at the package infrastructure level, and adjust the
> infrastructure to execute the configure script from the build
> directory.
>
> Signed-off-by: Thomas Petazzoni<thomas.petazzoni@free-electrons.com>
> ---
>   package/pkg-autotools.mk |   27 +++++++++++++--------------
>   1 file changed, 13 insertions(+), 14 deletions(-)
>
> diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
> index 843239d..bd87253 100644
> --- a/package/pkg-autotools.mk
> +++ b/package/pkg-autotools.mk
> @@ -78,8 +78,6 @@ ifndef $(2)_SUBDIR
>    endif
>   endif
>
> -$(2)_SRCSUBDIR = $$($(2)_SRCDIR)/$($(2)_SUBDIR)
> -
>   $(2)_CONF_ENV			?=
>   $(2)_CONF_OPT			?=
>   $(2)_MAKE_ENV			?=
> @@ -91,6 +89,7 @@ $(2)_INSTALL_TARGET_OPT		?= DESTDIR=$$(TARGET_DIR)  install
>   $(2)_CLEAN_OPT			?= clean
>   $(2)_UNINSTALL_STAGING_OPT	?= DESTDIR=$$(STAGING_DIR) uninstall
>   $(2)_UNINSTALL_TARGET_OPT	?= DESTDIR=$$(TARGET_DIR)  uninstall
> +$(2)_SUPPORTS_OUT_OF_TREE        = YES

  Out of tree build may be broken for some packages, so I would make this 
?= YES.

>
>
>   #
> @@ -103,11 +102,11 @@ ifeq ($(5),target)
>
>   # Configure package for target
>   define $(2)_CONFIGURE_CMDS
> -	(cd $$($$(PKG)_SRCSUBDIR)&&  rm -rf config.cache&&  \
> +	(cd $$($$(PKG)_BUILDDIR)&&  rm -rf config.cache&&  \
>   	$$(TARGET_CONFIGURE_OPTS) \
>   	$$(TARGET_CONFIGURE_ARGS) \
>   	$$($$(PKG)_CONF_ENV) \
> -	./configure \
> +	$$($$(PKG)_SRCDIR)/$$($$(PKG)_SUBDIR)/configure \
>   		--target=$$(GNU_TARGET_NAME) \
>   		--host=$$(GNU_TARGET_NAME) \
>   		--build=$$(GNU_HOST_NAME) \

  Hm, not sure I like this. The build system may refer to something 
outside the subdir and may get confused by the changed path. So I'd do an 
mkdir -p of $($(PKG)_BUILDDIR)/$($(PKG)_SUBDIR) and build there.



  Regards,
  Arnout

[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC v1 05/14] autotools infrastructure: do the autoreconf as a post patch step

[Arnout Vandecappelle]

On 01/21/13 00:52, Thomas Petazzoni wrote:
> Doing the autoreconf step as a pre-configure hook doesn't work
> properly, because the source directory is read-only during the
> configure step. And in fact, the autoreconf process modifies the
> source code, so it is quite logical to do it as part of the patching
> process rather than the configuration process.

  It was in fact moved from POST_PATCH to PRE_CONFIGURE in commit 
89d1ad91fe4b1b65f0e902f94aba99a6cefbf631, and the PRE_CONFIGURE_HOOKS 
were added specifically for this purpose. Unfortunately, the commit gives 
no explanation as to why it was needed.


  Regards,
  Arnout

>
> Signed-off-by: Thomas Petazzoni<thomas.petazzoni@free-electrons.com>
> ---
>   package/pkg-autotools.mk |    2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
> index bd87253..ec394fd 100644
> --- a/package/pkg-autotools.mk
> +++ b/package/pkg-autotools.mk
> @@ -214,7 +214,7 @@ $(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),
>
>
>   ifeq ($$($(2)_AUTORECONF),YES)
> -$(2)_PRE_CONFIGURE_HOOKS += AUTORECONF_HOOK
> +$(2)_POST_PATCH_HOOKS += AUTORECONF_HOOK
>   $(2)_DEPENDENCIES += host-automake host-autoconf host-libtool
>   endif
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC v1 05/14] autotools infrastructure: do the autoreconf as a post patch step

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Sat, 26 Jan 2013 23:03:21 +0100, Arnout Vandecappelle wrote:
> On 01/21/13 00:52, Thomas Petazzoni wrote:
> > Doing the autoreconf step as a pre-configure hook doesn't work
> > properly, because the source directory is read-only during the
> > configure step. And in fact, the autoreconf process modifies the
> > source code, so it is quite logical to do it as part of the patching
> > process rather than the configuration process.
> 
>   It was in fact moved from POST_PATCH to PRE_CONFIGURE in commit 
> 89d1ad91fe4b1b65f0e902f94aba99a6cefbf631, and the PRE_CONFIGURE_HOOKS 
> were added specifically for this purpose. Unfortunately, the commit
> gives no explanation as to why it was needed.

Hum, right. Interesting. I've looked at the commits around this one,
and also the e-mails around the one through which this patch was
submitted, but it didn't give any clue. The other patches around are
version bumps and other, seemingly unrelated things.

It's even more disappointing that I was amongst the people giving a
Acked-by on this patch...

One reason that might explain this is the lack of ordering guarantees
on post patch hooks. For example, if you have a post patch hook that
applies Debian patches, and a post patch hook to do the autoreconf, you
quite certainly want the Debian patches hook to be executed before the
autoreconf hook. But I have absolutely no idea if this is the problem
that we were trying to fix here.

That said, I still believe that the autoreconf thing belongs to the
patch step. It is really a modification of the source code itself, and
it is common to both the target build and host build. So maybe the
autotools infrastructure needs a special hook in the generic
infrastructure (rather than a normal post patch hook), to ensure that
the autoreconf step gets executed after all post patch hooks?

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [RFC v1 05/14] autotools infrastructure: do the autoreconf as a post patch step

[Arnout Vandecappelle]

On 27/01/13 17:27, Thomas Petazzoni wrote:
> One reason that might explain this is the lack of ordering guarantees
> on post patch hooks. For example, if you have a post patch hook that
> applies Debian patches, and a post patch hook to do the autoreconf, you
> quite certainly want the Debian patches hook to be executed before the
> autoreconf hook. But I have absolutely no idea if this is the problem
> that we were trying to fix here.

  That sounds like a likely explanation...


> That said, I still believe that the autoreconf thing belongs to the
> patch step. It is really a modification of the source code itself, and
> it is common to both the target build and host build.

  I completely agree.


> So maybe the
> autotools infrastructure needs a special hook in the generic
> infrastructure (rather than a normal post patch hook), to ensure that
> the autoreconf step gets executed after all post patch hooks?

  Or maybe (just thinking out loud here) there could be the possibility 
for the package infrastructure to add additional steps in the dependency 
chain. Something like

ifeq ($$($(2)_AUTORECONF),YES)
$(2)_TARGET_AUTORECONFIGURE = $$($(3)_SRCDIR)/.stamp_autoreconfigured
$(1)-autoreconfigure: $(1)-patch $$($(2)_TARGET_AUTORECONFIGURE)
$(1)-configure: $(1)-autoreconfigure
endif


  Regards,
  Arnout
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC v1 05/14] autotools infrastructure: do the autoreconf as a post patch step

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Sun, 27 Jan 2013 23:18:39 +0100, Arnout Vandecappelle wrote:

> > So maybe the
> > autotools infrastructure needs a special hook in the generic
> > infrastructure (rather than a normal post patch hook), to ensure
> > that the autoreconf step gets executed after all post patch hooks?
> 
>   Or maybe (just thinking out loud here) there could be the
> possibility for the package infrastructure to add additional steps in
> the dependency chain. Something like
> 
> ifeq ($$($(2)_AUTORECONF),YES)
> $(2)_TARGET_AUTORECONFIGURE = $$($(3)_SRCDIR)/.stamp_autoreconfigured
> $(1)-autoreconfigure: $(1)-patch $$($(2)_TARGET_AUTORECONFIGURE)
> $(1)-configure: $(1)-autoreconfigure
> endif

Sounds interesting. I'll try to experiment around this idea.

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com