[Buildroot] [PATCHv4 0/5] Keeping customizations outside the Buildroot tree with BR2_EXTERNAL

[Buildroot] [PATCHv4 0/5] Keeping customizations outside the Buildroot tree with BR2_EXTERNAL

[Thomas Petazzoni]

Hello,

Here is the fourth version of the BR2_EXTERNAL patch series. Its main
purpose is to take into account the feedback received recently on the
list.

People interested in testing this can find a branch named
things-outside-br-v4 at
http://git.free-electrons.com/users/thomas-petazzoni/buildroot/log/?h=things-outside-br-v4.

Changes since v3
================

 * Fix the issue reported by Yann, where not having a
   output/.br-external file would make the build fail completely. I've
   fixed it in a slightly different way than the patch posted by Yann,
   though.

 * Revert back to the idea used in v2: include only a top-level
   BR2_EXTERNAL/Config.in file and a top-level
   BR2_EXTERNAL/external.mk. This way, the user of BR2_EXTERNAL is
   free to use it for target packages, host packages, bootloaders, and
   more. People have raised concern that enforcing the inclusion of
   BR2_EXTERNAL/package/Config.in was problematic, because they also
   wanted to add host tools in BR2_EXTERNAL/package/Config.in, and so
   on. So since people have differerent use cases, let's give some
   more flexibility.

Changes since v2
================

 * Remove the mechanism that was storing the BR2_EXTERNAL value in the
   external wrapper.

 * Add logic that stores the BR2_EXTERNAL value in a file called
   $(O)/.br-external. Whenever the user passes a BR2_EXTERNAL=<value>,
   it gets stored in this file, and Buildroot re-uses for all
   subsequent make invocations. The user can later override it again
   by passing another BR2_EXTERNAL=<value>. He can also remove it
   entirely by passing just BR2_EXTERNAL= (empty value).

 * Enforce usage of a package/ directory in the BR2_EXTERNAL
   directory. Buildroot only includes $(BR2_EXTERNAL)/Config.in and
   shows it under the "Target packages" menu, and Buildroot will only
   include $(BR2_EXTERNAL)/package/*/*.mk.

 * Clearly separate the Buildroot built-in defconfigs from the
   user-defined external defconfigs in the 'make help' output.

 * Include a fix for building the documentation, provided by Samuel
   Martin.

 * Update the documentation accordingly.

Changes since v1
================

 * Store BR2_EXTERNAL in the Makefile generated in the output
   directory, so that BR2_EXTERNAL doesn't have to be passed over and
   over again. Suggested by Ryan Barnett.

 * Patch set splitted in more fine-grained changes, as suggested by
   Arnout.

 * Instead of having the top-level Config.in file generated at
   run-time, use instead a trick proposed by Arnout: make BR2_EXTERNAL
   point to support/dummy-external/, which contains an empty
   Config.in. This way, regardless of whether BR2_EXTERNAL is
   specified by the user or not, kconfig is happy as 'source
   "$BR2_EXTERNAL/Config.in"' will always point to an existing
   file. If BR2_EXTERNAL is not used, it will point to the dummy file,
   if BR2_EXTERNAL is used, it will point to Config.in file in the
   provided BR2_EXTERNAL directory.

   With this, BR2_EXTERNAL always has a value, so testing whether
   BR2_EXTERNAL is empty no longer tells us if the user has provided a
   value or not. Therefore, we have another variable,
   BR2_EXTERNAL_USED, which tells if the mechanism is used by the user
   or not. This variable avoids trying to use defconfigs from the
   dummy directory, or to encode the BR2_EXTERNAL value in the
   Makefile wrapper.

 * As a consequence of the previous change, it is no longer needed to
   refactor the *config dependencies using a variable, so I've dropped
   what was previously the first patch of the series.

 * BR2_EXTERNAL is turned into an absolute path, in order to avoid any
   problem. Suggested by Arnout.

 * Fixed the usage of <some board> vs. <boardname> in the
   documentation, as noted by Arnout.

Thomas

Samuel Martin (1):
  manual: fix manual generation with BR2_EXTERNAL support

Thomas Petazzoni (4):
  core: introduce the BR2_EXTERNAL variable
  core: allow external Config.in/makefile code to be integrated
  core: allow external defconfigs to be used
  docs/manual: add explanations about BR2_EXTERNAL

 Config.in                            |   6 ++
 Makefile                             |  68 +++++++++++++++++-
 docs/manual/customize-outside-br.txt | 132 +++++++++++++++++++++++++++++++++++
 docs/manual/customize.txt            |   2 +
 docs/manual/manual.mk                |   1 +
 support/dummy-external/Config.in     |   0
 support/scripts/kconfiglib.py        |   2 +-
 7 files changed, 209 insertions(+), 2 deletions(-)
 create mode 100644 docs/manual/customize-outside-br.txt
 create mode 100644 support/dummy-external/Config.in

-- 
1.8.1.2

[Buildroot] [PATCHv4 1/5] core: introduce the BR2_EXTERNAL variable

[Thomas Petazzoni]

This commit introduces the BR2_EXTERNAL environment variable, which
will allow to keep Buildroot customization (board-specific
configuration files or root filesystem overlays, package Config.in and
makefiles, as well as defconfigs) outside of the Buildroot tree.

This commit only introduces the variable itself, and ensures that it
is available within Config.in options, so that string options used to
specify paths to directories or files can use $BR2_EXTERNAL as a
reference. For example, one can use
$BR2_EXTERNAL/board/<someboard>/kernel.config as the
BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE value.

Following patches extend the usage of BR2_EXTERNAL to other areas
(packages and defconfigs).

In details, this commit:

 * Introduces the BR2_EXTERNAL Kconfig option. This option has no
   prompt, and is therefore not visible to the user and also not
   stored in the .config file. It is automatically set to the value of
   the BR2_EXTERNAL environment variable. The only purpose of this
   BR2_EXTERNAL Kconfig option is to allow $BR2_EXTERNAL to be
   properly expanded when used inside Kconfig option values.

 * Calculates the BR2_EXTERNAL value to use. If passed on the command
   line, then this value is taken in priority, and saved to a
   .br-external hidden file in the output directory. If not passed on
   the command line, then we read the .br-external file from the
   output directory. This allows the user to not pass the BR2_EXTERNAL
   value at each make invocation. If no BR2_EXTERNAL value is passed,
   we define it to support/dummy-external/, so that the kconfig code
   finds an existing $(BR2_EXTERNAL)/package/Config.in file to
   include.

 * Passes the BR2_EXTERNAL into the *config environment, so that its
   value is found when parsing/evaluating Config.in files and .config
   values.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Config.in |  4 ++++
 Makefile  | 49 ++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 52 insertions(+), 1 deletion(-)

diff --git a/Config.in b/Config.in
index d87e0f0..98726ab 100644
--- a/Config.in
+++ b/Config.in
@@ -14,6 +14,10 @@ config BR2_HOSTARCH
 	string
 	option env="HOSTARCH"
 
+config BR2_EXTERNAL
+	string
+	option env="BR2_EXTERNAL"
+
 # Hidden boolean selected by pre-built packages for x86, when they
 # need to run on x86-64 machines (example: pre-built external
 # toolchains, binary tools like SAM-BA, etc.).
diff --git a/Makefile b/Makefile
index b5368a3..5878d44 100644
--- a/Makefile
+++ b/Makefile
@@ -99,6 +99,52 @@ export CDPATH:=
 BASE_DIR := $(shell mkdir -p $(O) && cd $(O) >/dev/null && pwd)
 $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 
+
+# Handling of BR2_EXTERNAL. We are handling four cases here:
+#
+#  (Case 1) BR2_EXTERNAL is defined in the command line, but has an
+#           empty value. That's an indication that the user wants to
+#           remove the BR2_EXTERNAL value. So we use the
+#           dummy-external directory as BR2_EXTERNAL and remove the
+#           .br-external file.
+#  (Case 2) BR2_EXTERNAL is defined in the command line, and has a
+#           non-empty value. That's an indication that the user wants
+#           to use the provided location as the BR2_EXTERNAL. We
+#           verify that the location exists, and if it's the case,
+#           store it in .br-external.
+#  (Case 3) BR2_EXTERNAL isn't defined in the command line, and there
+#           is no .br-external file, so like in (Case 1), we use the
+#           dummy-external directory as BR2_EXTERNAL.
+#  (Case 4) BR2_EXTERNAL isn't defined in the command line, but there
+#           is a .br-external file. We load the value from this file,
+#           verify that it's an existing location, and use it.
+
+BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
+
+ifeq ($(origin BR2_EXTERNAL),command line)
+ifeq ($(BR2_EXTERNAL),) # Case 1
+override BR2_EXTERNAL := $(TOPDIR)/support/dummy-external/
+$(shell rm -f $(BR2_EXTERNAL_FILE))
+else # Case 2
+ifeq ($(wildcard $(BR2_EXTERNAL)),)
+$(error "The specified BR2_EXTERNAL '$(BR2_EXTERNAL)' location doesn't exist")
+endif
+override BR2_EXTERNAL := $(realpath $(BR2_EXTERNAL))
+BR2_EXTERNAL_USED = y
+$(shell echo $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
+endif
+else
+ifeq ($(wildcard $(BR2_EXTERNAL_FILE)),) # Case 3
+override BR2_EXTERNAL := $(TOPDIR)/support/dummy-external/
+else # Case 4
+override BR2_EXTERNAL := $(shell cat $(BR2_EXTERNAL_FILE))
+ifeq ($(wildcard $(BR2_EXTERNAL)),)
+$(error "The specified BR2_EXTERNAL '$(BR2_EXTERNAL)' location doesn't exist")
+endif
+BR2_EXTERNAL_USED = y
+endif
+endif
+
 BUILD_DIR:=$(BASE_DIR)/build
 STAMP_DIR:=$(BASE_DIR)/stamps
 BINARIES_DIR:=$(BASE_DIR)/images
@@ -619,7 +665,8 @@ COMMON_CONFIG_ENV = \
 	KCONFIG_AUTOCONFIG=$(BUILD_DIR)/buildroot-config/auto.conf \
 	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
 	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
-	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG)
+	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG) \
+	BR2_EXTERNAL=$(BR2_EXTERNAL)
 
 xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
 	@mkdir -p $(BUILD_DIR)/buildroot-config
-- 
1.8.1.2

[Buildroot] [PATCHv4 2/5] core: allow external Config.in/makefile code to be integrated

[Thomas Petazzoni]

This commit allows the BR2_EXTERNAL directory to contain Config.in and
Makefile code, which gets integrated into the Buildroot build logic:

 - Buildroot automatically includes the $BR2_EXTERNAL/Config.in in the
   top-level configuration menu.

 - Buildroot automatically includes the BR2_EXTERNAL/external.mk in
   the build logic, so it can for example be used to include other .mk
   files that define package recipes.

This is typically intended to be used to create target packages in the
BR2_EXTERNAL directory, but can also be used for bootloaders, host
packages, or other custom make logic.

We also add a dummy Config.in file in support/dummy-external/ to
ensure that the source "$BR2_EXTERNAL/Config.in" line will point to an
existing file even when BR2_EXTERNAL is not used by the user.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Config.in                        | 2 ++
 Makefile                         | 4 ++++
 support/dummy-external/Config.in | 0
 3 files changed, 6 insertions(+)
 create mode 100644 support/dummy-external/Config.in

diff --git a/Config.in b/Config.in
index 98726ab..9b10693 100644
--- a/Config.in
+++ b/Config.in
@@ -486,3 +486,5 @@ source "boot/Config.in"
 source "package/Config.in.host"
 
 source "Config.in.legacy"
+
+source "$BR2_EXTERNAL/Config.in"
diff --git a/Makefile b/Makefile
index 5878d44..b5a9828 100644
--- a/Makefile
+++ b/Makefile
@@ -375,6 +375,10 @@ include boot/common.mk
 include linux/linux.mk
 include system/system.mk
 
+ifeq ($(BR2_EXTERNAL_USED),y)
+include $(BR2_EXTERNAL)/external.mk
+endif
+
 TARGETS+=target-finalize
 
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
diff --git a/support/dummy-external/Config.in b/support/dummy-external/Config.in
new file mode 100644
index 0000000..e69de29
-- 
1.8.1.2

[Buildroot] [PATCHv4 3/5] core: allow external defconfigs to be used

[Thomas Petazzoni]

This commit allows the user to store defconfigs in
$BR2_EXTERNAL/configs/. To achieve this:

 * It adds a new %_defconfig that looks in $BR2_EXTERNAL/configs/ for
   the corresponding defconfig file.

 * Updates the help target to also list external defconfigs.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/Makefile b/Makefile
index b5a9828..eb547a2 100644
--- a/Makefile
+++ b/Makefile
@@ -754,6 +754,12 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@mkdir -p $(BUILD_DIR)/buildroot-config
 	@$(COMMON_CONFIG_ENV) $< --defconfig=$(TOPDIR)/configs/$@ $(CONFIG_CONFIG_IN)
 
+ifeq ($(BR2_EXTERNAL_USED),y)
+%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig outputmakefile
+	@mkdir -p $(BUILD_DIR)/buildroot-config
+	@$(COMMON_CONFIG_ENV) $< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
+endif
+
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@mkdir -p $(BUILD_DIR)/buildroot-config
 	@$(COMMON_CONFIG_ENV) $< \
@@ -862,8 +868,17 @@ endif
 	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
 	@echo '  make O=dir             - Locate all output files in "dir", including .config'
 	@echo
+	@echo 'Built-in configs'
+	@echo
 	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
+ifeq ($(BR2_EXTERNAL_USED),y)
+	@echo
+	@echo 'User-provided configs'
+	@echo
+	@$(foreach b, $(sort $(notdir $(wildcard $(BR2_EXTERNAL)/configs/*_defconfig))), \
+	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
+endif
 	@echo
 	@echo 'See docs/README, or generate the Buildroot manual for further details'
 	@echo
-- 
1.8.1.2

[Buildroot] [PATCHv4 4/5] docs/manual: add explanations about BR2_EXTERNAL

[Thomas Petazzoni]

This commit updates the manual to add details on how to use the
BR2_EXTERNAL feature.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 docs/manual/customize-outside-br.txt | 132 +++++++++++++++++++++++++++++++++++
 docs/manual/customize.txt            |   2 +
 2 files changed, 134 insertions(+)
 create mode 100644 docs/manual/customize-outside-br.txt

diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
new file mode 100644
index 0000000..623a421
--- /dev/null
+++ b/docs/manual/customize-outside-br.txt
@@ -0,0 +1,132 @@
+// -*- mode:doc -*- ;
+
+[[outside-br-custom]]
+Keeping customization outside Buildroot
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+While the Buildroot community recommends and encourages upstreaming to
+the official Buildroot version the packages and boards support that
+are written by developers, it is sometimes not possible or desirable,
+due to these packages or boards being highly specific or proprietary.
+
+In this case, Buildroot users are offered two choices:
+
+ * They can add their packages, board support and configuration files
+   directly within the Buildroot tree, and maintain them by using
+   branches in a version control system.
+
+ * They can use the +BR2_EXTERNAL+ mechanism, which allows to keep
+   package recipes, board support and configuration files outside of
+   the Buildroot tree, while still having them nicely integrated in
+   the build logic. The following paragraphs give details on how to
+   use +BR2_EXTERNAL+.
+
++BR2_EXTERNAL+ is an environment variable that one can use to point to
+a directory that contains Buildroot customizations. It can be passed
+to any Buildroot +make+ invocation, and when it is passed. It is
+automatically saved in the hidden +.br-external+ file in the output
+directory, so that it is not needed to pass +BR2_EXTERNAL+ at every
++make+ invocation. It can however be changed at any time by specifying
+a new value, and can be removed by passing an empty value. The
++BR2_EXTERNAL+ path can be either an absolute or a relative path, but
+if it's passed as a relative path, it is important to note that it is
+interpreted relatively to the main Buildroot source directory, not the
+Buildroot output directory.
+
+Some examples:
+
+-----
+ buildroot/ $ make BR2_EXTERNAL=../foobar menuconfig
+-----
+
+Starting from now on, external definitions fromt he +../company+
+directory will be used:
+
+-----
+ buildroot/ $ make
+ buildroot/ $ make legal-info
+-----
+
+We can switch to another external definitions directory at any time:
+
+-----
+ buildroot/ $ make BR2_EXTERNAL=../barfoo xconfig
+-----
+
+Or disable the usage of external definitions:
+
+-----
+ buildroot/ $ make BR2_EXTERNAL= xconfig
+-----
+
+This +BR2_EXTERNAL+ then allows three different things:
+
+ * One can store all the board-specific configuration files here, such
+   as the kernel configuration, the root filesystem overlay, or any
+   other configuration file for which Buildroot allows to set its
+   location. The +BR2_EXTERNAL+ value is available within the
+   Buildroot configuration using +$(BR2_EXTERNAL)+. As an example, one
+   could set the +BR2_ROOTFS_OVERLAY+ Buildroot option to
+   +$(BR2_EXTERNAL)/board/<boardname>/overlay/+ (to specify a root
+   filesystem overlay), or the +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
+   Buildroot option to +$(BR2_EXTERNAL)/board/<boardname>/kernel.config+
+   (to specify the location of the kernel configuration file).  + To
+   achieve this, it is recommended but not mandatory, to store those
+   details in directories called +board/<boardname>/+ under
+   +BR2_EXTERNAL+.
+
+ * One can store package recipes (i.e +Config.in+ and
+   +<packagename>.mk+), or even custom configuration options and make
+   logic. Buildroot automatically includes +BR2_EXTERNAL/Config.in+ to
+   make it appear in the top-level configuration menu, and includes
+   +BR2_EXTERNAL/external.mk+ with the rest of the makefile logic.
++
+The main usage of this is to store package recipes. The recommended
+   way to do this is to write a +BR2_EXTERNAL/Config.in+ that looks
+   like:
++
+------
+menu "<somecompany> packages"
+
+source "$BR2_EXTERNAL/package/package1/Config.in"
+source "$BR2_EXTERNAL/package/package2/Config.in"
+
+endmenu
+------
++
+Then, have a +BR2_EXTERNAL/external.mk' file that looks like:
++
+------
+include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))
+------
++
+And then in +BR2_EXTERNAL/package/package1+ and
+   +BR2_EXTERNAL/package/package2+ create normal Buildroot package
+   recipes, as explained in xref:adding-packages[].
+
+ * One can store Buildroot defconfigs in the +configs+ subdirectory of
+   +BR2_EXTERNAL+. Buildroot will automatically show them in the
+   output of +make help+ and allow them to be loaded with the normal
+   +make <name>_defconfig+ command.
+
+In the end, a typical +BR2_EXTERNAL+ directory organization would
+generally be:
+
+-----
+??? Config.in
+??? external.mk
+??? board/
+?   ??? <boardname>/
+?       ??? overlay/
+?           ??? etc/
+?               ??? <some file>
+??? configs/
+?   ??? <boardname>_defconfig
+??? package/
+    ??? package1/
+        ??? Config.in
+        ??? package1.mk
+    ??? package2/
+        ??? Config.in
+        ??? package2.mk
+------
diff --git a/docs/manual/customize.txt b/docs/manual/customize.txt
index 0456ef1..7e46fd8 100644
--- a/docs/manual/customize.txt
+++ b/docs/manual/customize.txt
@@ -17,3 +17,5 @@ include::customize-toolchain.txt[]
 include::customize-store.txt[]
 
 include::customize-packages.txt[]
+
+include::customize-outside-br.txt[]
-- 
1.8.1.2

[Buildroot] [PATCHv4 5/5] manual: fix manual generation with BR2_EXTERNAL support

[Thomas Petazzoni]

From: Samuel Martin <s.martin49@gmail.com>

Reported-by: Ryan Barnett <rjbarnet@rockwellcollins.com>
Signed-off-by: Samuel Martin <s.martin49@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Ryan Barnett <rjbarnet@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 docs/manual/manual.mk         | 1 +
 support/scripts/kconfiglib.py | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
index aeafd10..570318f 100644
--- a/docs/manual/manual.mk
+++ b/docs/manual/manual.mk
@@ -1,6 +1,7 @@
 manual-update-lists: manual-check-dependencies-lists
 	$(Q)$(call MESSAGE,"Updating the manual lists...")
 	$(Q)BR2_DEFCONFIG="" TOPDIR=$(TOPDIR) O=$(O)/docs/manual/.build \
+		BR2_EXTERNAL=$(TOPDIR)/support/dummy-external \
 		$(TOPDIR)/support/scripts/gen-manual-lists.py
 
 # we can't use suitable-host-package here because that's not available in
diff --git a/support/scripts/kconfiglib.py b/support/scripts/kconfiglib.py
index 0704cc0..e40947c 100644
--- a/support/scripts/kconfiglib.py
+++ b/support/scripts/kconfiglib.py
@@ -2074,7 +2074,7 @@ set_re   = re.compile(r"CONFIG_(\w+)=(.*)")
 unset_re = re.compile(r"# CONFIG_(\w+) is not set")
 
 # Regular expression for finding $-references to symbols in strings
-sym_ref_re = re.compile(r"\$[A-Za-z_]+")
+sym_ref_re = re.compile(r"\$[A-Za-z_][0-9A-Za-z_]*")
 
 # Integers representing symbol types
 UNKNOWN, BOOL, TRISTATE, STRING, HEX, INT = range(0, 6)
-- 
1.8.1.2

[Buildroot] [PATCHv4 1/5] core: introduce the BR2_EXTERNAL variable

[Yann E. MORIN]

Thomas, All,

On 2013-11-29 20:00 +0100, Thomas Petazzoni spake thusly:
> This commit introduces the BR2_EXTERNAL environment variable, which
> will allow to keep Buildroot customization (board-specific
> configuration files or root filesystem overlays, package Config.in and
> makefiles, as well as defconfigs) outside of the Buildroot tree.
[--SNIP--]
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCHv4 2/5] core: allow external Config.in/makefile code to be integrated

[Yann E. MORIN]

Thomas, All,

On 2013-11-29 20:00 +0100, Thomas Petazzoni spake thusly:
> This commit allows the BR2_EXTERNAL directory to contain Config.in and
> Makefile code, which gets integrated into the Buildroot build logic:
[--SNIP--]
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCHv4 3/5] core: allow external defconfigs to be used

[Yann E. MORIN]

Thomas, All,

On 2013-11-29 20:00 +0100, Thomas Petazzoni spake thusly:
> This commit allows the user to store defconfigs in
> $BR2_EXTERNAL/configs/. To achieve this:
> 
>  * It adds a new %_defconfig that looks in $BR2_EXTERNAL/configs/ for
>    the corresponding defconfig file.
> 
>  * Updates the help target to also list external defconfigs.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  Makefile | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index b5a9828..eb547a2 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -754,6 +754,12 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  	@mkdir -p $(BUILD_DIR)/buildroot-config
>  	@$(COMMON_CONFIG_ENV) $< --defconfig=$(TOPDIR)/configs/$@ $(CONFIG_CONFIG_IN)
>  
> +ifeq ($(BR2_EXTERNAL_USED),y)
> +%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig outputmakefile
> +	@mkdir -p $(BUILD_DIR)/buildroot-config
> +	@$(COMMON_CONFIG_ENV) $< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
> +endif
> +
>  savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  	@mkdir -p $(BUILD_DIR)/buildroot-config
>  	@$(COMMON_CONFIG_ENV) $< \
> @@ -862,8 +868,17 @@ endif
>  	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
>  	@echo '  make O=dir             - Locate all output files in "dir", including .config'
>  	@echo
> +	@echo 'Built-in configs'
> +	@echo

There is no empty line after other help section titles, and the titles
end with a colon, eg.:

---8<--
Miscellaneous:
  source                 - download all sources needed for offline-build
  source-check           - check selected packages for valid download URLs
---8<--

>  	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
>  	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
> +ifeq ($(BR2_EXTERNAL_USED),y)
> +	@echo
> +	@echo 'User-provided configs'
> +	@echo

Ditto.

> +	@$(foreach b, $(sort $(notdir $(wildcard $(BR2_EXTERNAL)/configs/*_defconfig))), \
> +	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
> +endif
>  	@echo
>  	@echo 'See docs/README, or generate the Buildroot manual for further details'
>  	@echo

Otherwise, you get my:
    Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
    Tested-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCHv4 4/5] docs/manual: add explanations about BR2_EXTERNAL

[Yann E. MORIN]

Thomas, All,

On 2013-11-29 20:00 +0100, Thomas Petazzoni spake thusly:
> This commit updates the manual to add details on how to use the
> BR2_EXTERNAL feature.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  docs/manual/customize-outside-br.txt | 132 +++++++++++++++++++++++++++++++++++
>  docs/manual/customize.txt            |   2 +
>  2 files changed, 134 insertions(+)
>  create mode 100644 docs/manual/customize-outside-br.txt
> 
> diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
> new file mode 100644
> index 0000000..623a421
> --- /dev/null
> +++ b/docs/manual/customize-outside-br.txt
> @@ -0,0 +1,132 @@
> +// -*- mode:doc -*- ;
> +
> +[[outside-br-custom]]
> +Keeping customization outside Buildroot
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +While the Buildroot community recommends and encourages upstreaming to
> +the official Buildroot version the packages and boards support that
> +are written by developers, it is sometimes not possible or desirable,
> +due to these packages or boards being highly specific or proprietary.

s/due to/because/

> +
> +In this case, Buildroot users are offered two choices:
> +
> + * They can add their packages, board support and configuration files
> +   directly within the Buildroot tree, and maintain them by using
> +   branches in a version control system.
> +
> + * They can use the +BR2_EXTERNAL+ mechanism, which allows to keep
> +   package recipes, board support and configuration files outside of
> +   the Buildroot tree, while still having them nicely integrated in
> +   the build logic. The following paragraphs give details on how to
> +   use +BR2_EXTERNAL+.
> +
> ++BR2_EXTERNAL+ is an environment variable that one can use to point to
> +a directory that contains Buildroot customizations. It can be passed
> +to any Buildroot +make+ invocation, and when it is passed. It is
                                           ^^^^^^^^^^^^^^^^^^^^^^^^  

Uh? That sentence does not make sense. What about:

It can be passed to any Buildroot +make+ invocation, and is
automatically saved in [...]

> +automatically saved in the hidden +.br-external+ file in the output
> +directory, so that it is not needed to pass +BR2_EXTERNAL+ at every
> ++make+ invocation. It can however be changed at any time by specifying
> +a new value, and can be removed by passing an empty value.

I'd start a new paragraph here, sionce we've switched to another topic.

> The
> ++BR2_EXTERNAL+ path can be either an absolute or a relative path, but
> +if it's passed as a relative path, it is important to note that it is
> +interpreted relatively to the main Buildroot source directory, not the
> +Buildroot output directory.
> +
> +Some examples:
> +
> +-----
> + buildroot/ $ make BR2_EXTERNAL=../foobar menuconfig
> +-----
> +
> +Starting from now on, external definitions fromt he +../company+
                                                 ^^^^
Typo

> +directory will be used:
> +
> +-----
> + buildroot/ $ make
> + buildroot/ $ make legal-info
> +-----
> +
> +We can switch to another external definitions directory at any time:
> +
> +-----
> + buildroot/ $ make BR2_EXTERNAL=../barfoo xconfig
> +-----
> +
> +Or disable the usage of external definitions:
> +
> +-----
> + buildroot/ $ make BR2_EXTERNAL= xconfig
> +-----
> +
> +This +BR2_EXTERNAL+ then allows three different things:
> +
> + * One can store all the board-specific configuration files here, such
                                                               ^^^^
there

> +   as the kernel configuration, the root filesystem overlay, or any
> +   other configuration file for which Buildroot allows to set its
> +   location. The +BR2_EXTERNAL+ value is available within the
> +   Buildroot configuration using +$(BR2_EXTERNAL)+. As an example, one
> +   could set the +BR2_ROOTFS_OVERLAY+ Buildroot option to
> +   +$(BR2_EXTERNAL)/board/<boardname>/overlay/+ (to specify a root
> +   filesystem overlay), or the +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
> +   Buildroot option to +$(BR2_EXTERNAL)/board/<boardname>/kernel.config+
> +   (to specify the location of the kernel configuration file).  + To
                                                                  ^^^
Typo

> +   achieve this, it is recommended but not mandatory, to store those
> +   details in directories called +board/<boardname>/+ under
> +   +BR2_EXTERNAL+.
> +
> + * One can store package recipes (i.e +Config.in+ and
> +   +<packagename>.mk+), or even custom configuration options and make
> +   logic. Buildroot automatically includes +BR2_EXTERNAL/Config.in+ to
> +   make it appear in the top-level configuration menu, and includes
> +   +BR2_EXTERNAL/external.mk+ with the rest of the makefile logic.
> ++
> +The main usage of this is to store package recipes. The recommended
> +   way to do this is to write a +BR2_EXTERNAL/Config.in+ that looks
> +   like:
> ++
> +------
> +menu "<somecompany> packages"
> +
> +source "$BR2_EXTERNAL/package/package1/Config.in"
> +source "$BR2_EXTERNAL/package/package2/Config.in"
> +
> +endmenu
> +------
> ++
> +Then, have a +BR2_EXTERNAL/external.mk' file that looks like:
> ++
> +------
> +include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))
> +------
> ++
> +And then in +BR2_EXTERNAL/package/package1+ and
> +   +BR2_EXTERNAL/package/package2+ create normal Buildroot package
> +   recipes, as explained in xref:adding-packages[].
> +
> + * One can store Buildroot defconfigs in the +configs+ subdirectory of
> +   +BR2_EXTERNAL+. Buildroot will automatically show them in the
> +   output of +make help+ and allow them to be loaded with the normal
> +   +make <name>_defconfig+ command.
> +
> +In the end, a typical +BR2_EXTERNAL+ directory organization would
> +generally be:
> +
> +-----
> +??? Config.in
> +??? external.mk
> +??? board/
> +?   ??? <boardname>/
> +?       ??? overlay/
> +?           ??? etc/
> +?               ??? <some file>
> +??? configs/
> +?   ??? <boardname>_defconfig
> +??? package/
> +    ??? package1/
> +        ??? Config.in
> +        ??? package1.mk
> +    ??? package2/

Missing symbols in the tree structure.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCHv4 5/5] manual: fix manual generation with BR2_EXTERNAL support

[Yann E. MORIN]

Samuel, All,

On 2013-11-29 20:00 +0100, Thomas Petazzoni spake thusly:
> From: Samuel Martin <s.martin49@gmail.com>
> 
> Reported-by: Ryan Barnett <rjbarnet@rockwellcollins.com>
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Signed-off-by: Ryan Barnett <rjbarnet@rockwellcollins.com>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  docs/manual/manual.mk         | 1 +
>  support/scripts/kconfiglib.py | 2 +-
>  2 files changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
> index aeafd10..570318f 100644
> --- a/docs/manual/manual.mk
> +++ b/docs/manual/manual.mk
> @@ -1,6 +1,7 @@
>  manual-update-lists: manual-check-dependencies-lists
>  	$(Q)$(call MESSAGE,"Updating the manual lists...")
>  	$(Q)BR2_DEFCONFIG="" TOPDIR=$(TOPDIR) O=$(O)/docs/manual/.build \
> +		BR2_EXTERNAL=$(TOPDIR)/support/dummy-external \

I think I understand that change: we do not want to generate the manual
with the external packages. Please extend the commit log.

>  		$(TOPDIR)/support/scripts/gen-manual-lists.py
>  
>  # we can't use suitable-host-package here because that's not available in
> diff --git a/support/scripts/kconfiglib.py b/support/scripts/kconfiglib.py
> index 0704cc0..e40947c 100644
> --- a/support/scripts/kconfiglib.py
> +++ b/support/scripts/kconfiglib.py
> @@ -2074,7 +2074,7 @@ set_re   = re.compile(r"CONFIG_(\w+)=(.*)")
>  unset_re = re.compile(r"# CONFIG_(\w+) is not set")
>  
>  # Regular expression for finding $-references to symbols in strings
> -sym_ref_re = re.compile(r"\$[A-Za-z_]+")
> +sym_ref_re = re.compile(r"\$[A-Za-z_][0-9A-Za-z_]*")

I fail to see how this change is related to BR2_EXTERNAL. Care to
explain (and extend the commit log at the same time), please?

>  # Integers representing symbol types
>  UNKNOWN, BOOL, TRISTATE, STRING, HEX, INT = range(0, 6)
> -- 
> 1.8.1.2
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCHv4 1/5] core: introduce the BR2_EXTERNAL variable

[Arnout Vandecappelle]

  Lots of feedback on this one :-)

On 29/11/13 20:00, Thomas Petazzoni wrote:
> This commit introduces the BR2_EXTERNAL environment variable, which
> will allow to keep Buildroot customization (board-specific
> configuration files or root filesystem overlays, package Config.in and
> makefiles, as well as defconfigs) outside of the Buildroot tree.
>
> This commit only introduces the variable itself, and ensures that it
> is available within Config.in options, so that string options used to
> specify paths to directories or files can use $BR2_EXTERNAL as a
> reference. For example, one can use
> $BR2_EXTERNAL/board/<someboard>/kernel.config as the
> BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE value.

  Actually, that's not the reason you need it in Config.in. You only need 
it to be able to source it. Expansions in string values are done by make 
- that's why you need () around them.


> Following patches extend the usage of BR2_EXTERNAL to other areas
> (packages and defconfigs).
>
> In details, this commit:
>
>   * Introduces the BR2_EXTERNAL Kconfig option. This option has no
>     prompt, and is therefore not visible to the user and also not
>     stored in the .config file. It is automatically set to the value of
>     the BR2_EXTERNAL environment variable. The only purpose of this
>     BR2_EXTERNAL Kconfig option is to allow $BR2_EXTERNAL to be
>     properly expanded when used inside Kconfig option values.

  Inside source statements you mean :-)

>
>   * Calculates the BR2_EXTERNAL value to use. If passed on the command
>     line, then this value is taken in priority, and saved to a
>     .br-external hidden file in the output directory. If not passed on
>     the command line, then we read the .br-external file from the
>     output directory. This allows the user to not pass the BR2_EXTERNAL
>     value at each make invocation. If no BR2_EXTERNAL value is passed,
>     we define it to support/dummy-external/, so that the kconfig code
>     finds an existing $(BR2_EXTERNAL)/package/Config.in file to
>     include.
>
>   * Passes the BR2_EXTERNAL into the *config environment, so that its
>     value is found when parsing/evaluating Config.in files and .config
>     values.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>   Config.in |  4 ++++
>   Makefile  | 49 ++++++++++++++++++++++++++++++++++++++++++++++++-
>   2 files changed, 52 insertions(+), 1 deletion(-)
>
> diff --git a/Config.in b/Config.in
> index d87e0f0..98726ab 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -14,6 +14,10 @@ config BR2_HOSTARCH
>   	string
>   	option env="HOSTARCH"
>
> +config BR2_EXTERNAL
> +	string
> +	option env="BR2_EXTERNAL"
> +
>   # Hidden boolean selected by pre-built packages for x86, when they
>   # need to run on x86-64 machines (example: pre-built external
>   # toolchains, binary tools like SAM-BA, etc.).
> diff --git a/Makefile b/Makefile
> index b5368a3..5878d44 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -99,6 +99,52 @@ export CDPATH:=
>   BASE_DIR := $(shell mkdir -p $(O) && cd $(O) >/dev/null && pwd)
>   $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
>
> +
> +# Handling of BR2_EXTERNAL. We are handling four cases here:
> +#
> +#  (Case 1) BR2_EXTERNAL is defined in the command line, but has an
> +#           empty value. That's an indication that the user wants to
> +#           remove the BR2_EXTERNAL value. So we use the
> +#           dummy-external directory as BR2_EXTERNAL and remove the
> +#           .br-external file.
> +#  (Case 2) BR2_EXTERNAL is defined in the command line, and has a
> +#           non-empty value. That's an indication that the user wants
> +#           to use the provided location as the BR2_EXTERNAL. We
> +#           verify that the location exists, and if it's the case,
> +#           store it in .br-external.
> +#  (Case 3) BR2_EXTERNAL isn't defined in the command line, and there
> +#           is no .br-external file, so like in (Case 1), we use the
> +#           dummy-external directory as BR2_EXTERNAL.
> +#  (Case 4) BR2_EXTERNAL isn't defined in the command line, but there
> +#           is a .br-external file. We load the value from this file,
> +#           verify that it's an existing location, and use it.

  This is overly complex IMHO. You can just include .br-external before 
the conditions, then you have only two cases: BR2_EXTERNAL is set or it 
is not set. In the first case, you remove .br-external, in the other case 
you create it. It does mean that it is re-created on every invocation of 
make, but I don't think that that's a problem.

> +
> +BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
> +
> +ifeq ($(origin BR2_EXTERNAL),command line)
> +ifeq ($(BR2_EXTERNAL),) # Case 1
> +override BR2_EXTERNAL := $(TOPDIR)/support/dummy-external/

  There is no reason to use :=, $(TOPDIR) or a trailing /.

+$(shell rm -f $(BR2_EXTERNAL_FILE))
> +else # Case 2
> +ifeq ($(wildcard $(BR2_EXTERNAL)),)
> +$(error "The specified BR2_EXTERNAL '$(BR2_EXTERNAL)' location doesn't exist")

  This error is kind of pointless: it checks if the directory exists, but 
not if external.mk or Config.in exists. And those two will cause separate 
errors anyway. So I'd just remove this one.

> +endif
> +override BR2_EXTERNAL := $(realpath $(BR2_EXTERNAL))

  No reason to use realpath here.

> +BR2_EXTERNAL_USED = y
> +$(shell echo $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))

  If you use an assignment, you can -include the file directly.


  Since I may not have explained what I mean very well, I've just 
implemented all of it myself and it'll come as a reply to this message. 
Feel free to reuse any of the ideas that you like and abandon the others.

  Regards,
  Arnout

> +endif
> +else
> +ifeq ($(wildcard $(BR2_EXTERNAL_FILE)),) # Case 3
> +override BR2_EXTERNAL := $(TOPDIR)/support/dummy-external/
> +else # Case 4
> +override BR2_EXTERNAL := $(shell cat $(BR2_EXTERNAL_FILE))
> +ifeq ($(wildcard $(BR2_EXTERNAL)),)
> +$(error "The specified BR2_EXTERNAL '$(BR2_EXTERNAL)' location doesn't exist")
> +endif
> +BR2_EXTERNAL_USED = y
> +endif
> +endif
> +
>   BUILD_DIR:=$(BASE_DIR)/build
>   STAMP_DIR:=$(BASE_DIR)/stamps
>   BINARIES_DIR:=$(BASE_DIR)/images
> @@ -619,7 +665,8 @@ COMMON_CONFIG_ENV = \
>   	KCONFIG_AUTOCONFIG=$(BUILD_DIR)/buildroot-config/auto.conf \
>   	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
>   	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
> -	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG)
> +	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG) \
> +	BR2_EXTERNAL=$(BR2_EXTERNAL)
>
>   xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
>   	@mkdir -p $(BUILD_DIR)/buildroot-config
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v5] core: introduce the BR2_EXTERNAL variable

[Arnout Vandecappelle]

From: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

This commit introduces the BR2_EXTERNAL environment variable, which
will allow to keep Buildroot customization (board-specific
configuration files or root filesystem overlays, package Config.in and
makefiles, as well as defconfigs) outside of the Buildroot tree.

This commit only introduces the variable itself, and ensures that it
is available within Config.in options. This allows us to use
$BR2_EXTERNAL in a 'source' statement in Config.in.

Following patches extend the usage of BR2_EXTERNAL to other areas
(packages and defconfigs).

In details, this commit:

 * Introduces the BR2_EXTERNAL Kconfig option. This option has no
   prompt, and is therefore not visible to the user and also not
   stored in the .config file. It is automatically set to the value of
   the BR2_EXTERNAL environment variable. The only purpose of this
   BR2_EXTERNAL Kconfig option is to allow $BR2_EXTERNAL to be
   properly expanded when used inside Kconfig source statements.

 * Calculates the BR2_EXTERNAL value to use. If passed on the command
   line, then this value is taken in priority, and saved to a
   .br-external hidden file in the output directory. If not passed on
   the command line, then we read the .br-external file from the
   output directory. This allows the user to not pass the BR2_EXTERNAL
   value at each make invocation. If no BR2_EXTERNAL value is passed,
   we define it to support/dummy-external, so that the kconfig code
   finds an existing $(BR2_EXTERNAL)/package/Config.in file to
   include.

 * Passes the BR2_EXTERNAL into the *config environment, so that its
   value is found when parsing/evaluating Config.in files and .config
   values.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 Config.in |  4 ++++
 Makefile  | 27 ++++++++++++++++++++++++++-
 2 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/Config.in b/Config.in
index d87e0f0..98726ab 100644
--- a/Config.in
+++ b/Config.in
@@ -14,6 +14,10 @@ config BR2_HOSTARCH
 	string
 	option env="HOSTARCH"
 
+config BR2_EXTERNAL
+	string
+	option env="BR2_EXTERNAL"
+
 # Hidden boolean selected by pre-built packages for x86, when they
 # need to run on x86-64 machines (example: pre-built external
 # toolchains, binary tools like SAM-BA, etc.).
diff --git a/Makefile b/Makefile
index 0480a42..ae0bc10 100644
--- a/Makefile
+++ b/Makefile
@@ -99,6 +99,30 @@ export CDPATH:=
 BASE_DIR := $(shell mkdir -p $(O) && cd $(O) >/dev/null && pwd)
 $(if $(BASE_DIR),, $(error output directory "$(O)" does not exist))
 
+
+# Handling of BR2_EXTERNAL.
+#
+# The value of BR2_EXTERNAL is stored in .br-external in the output directory.
+# On subsequent invocations of make, it is read in. It can still be overridden
+# on the command line, therefore the file is re-created every time make is run.
+#
+# When BR2_EXTERNAL is not set, the .br-external file is removed and we point 
+# to support/dummy-external. This makes sure we can unconditionally include the
+# Config.in and external.mk from the BR2_EXTERNAL directory. In this case,
+# override is necessary so the user can clear BR2_EXTERNAL from the command 
+# line, but the dummy path is still used internally.
+
+BR2_EXTERNAL_FILE = $(BASE_DIR)/.br-external
+-include $(BR2_EXTERNAL_FILE)
+ifeq ($(BR2_EXTERNAL),)
+  override BR2_EXTERNAL = support/dummy-external
+  $(shell rm -f $(BR2_EXTERNAL_FILE))
+else
+  BR2_EXTERNAL_USED = y
+  $(shell echo BR2_EXTERNAL ?= $(BR2_EXTERNAL) > $(BR2_EXTERNAL_FILE))
+endif
+
+
 BUILD_DIR:=$(BASE_DIR)/build
 STAMP_DIR:=$(BASE_DIR)/stamps
 BINARIES_DIR:=$(BASE_DIR)/images
@@ -619,7 +643,8 @@ COMMON_CONFIG_ENV = \
 	KCONFIG_AUTOCONFIG=$(BUILD_DIR)/buildroot-config/auto.conf \
 	KCONFIG_AUTOHEADER=$(BUILD_DIR)/buildroot-config/autoconf.h \
 	KCONFIG_TRISTATE=$(BUILD_DIR)/buildroot-config/tristate.config \
-	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG)
+	BUILDROOT_CONFIG=$(BUILDROOT_CONFIG) \
+	BR2_EXTERNAL=$(BR2_EXTERNAL)
 
 xconfig: $(BUILD_DIR)/buildroot-config/qconf outputmakefile
 	@mkdir -p $(BUILD_DIR)/buildroot-config
-- 
1.8.4.4

[Buildroot] [PATCHv4 2/5] core: allow external Config.in/makefile code to be integrated

[Arnout Vandecappelle]

On 29/11/13 20:00, Thomas Petazzoni wrote:
> This commit allows the BR2_EXTERNAL directory to contain Config.in and
> Makefile code, which gets integrated into the Buildroot build logic:
>
>   - Buildroot automatically includes the $BR2_EXTERNAL/Config.in in the
>     top-level configuration menu.
>
>   - Buildroot automatically includes the BR2_EXTERNAL/external.mk in
>     the build logic, so it can for example be used to include other .mk
>     files that define package recipes.

  The advantage of enforcing the package dir would have been that we 
could directly include $(BR2_EXTERNAL)/package/*/*.mk, instead of forcing 
the user to create an external.mk that contains either

include $(BR2_EXTERNAL)/package/*/*.mk

or

include $(BR2_EXTERNAL)/*/*.mk

depending on how he structures things.


  Maybe a compromise is that we source $BR2_EXTERNAL/package/Config.in 
but include it at the end of the top-level Config.in instead of at the 
beginning of the package/Config.in. I guess that's what Samuel meant 
before...

>
> This is typically intended to be used to create target packages in the
> BR2_EXTERNAL directory, but can also be used for bootloaders, host
> packages, or other custom make logic.
>
> We also add a dummy Config.in file in support/dummy-external/ to
> ensure that the source "$BR2_EXTERNAL/Config.in" line will point to an
> existing file even when BR2_EXTERNAL is not used by the user.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>   Config.in                        | 2 ++
>   Makefile                         | 4 ++++
>   support/dummy-external/Config.in | 0
>   3 files changed, 6 insertions(+)
>   create mode 100644 support/dummy-external/Config.in
>
> diff --git a/Config.in b/Config.in
> index 98726ab..9b10693 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -486,3 +486,5 @@ source "boot/Config.in"
>   source "package/Config.in.host"
>
>   source "Config.in.legacy"
> +
> +source "$BR2_EXTERNAL/Config.in"
> diff --git a/Makefile b/Makefile
> index 5878d44..b5a9828 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -375,6 +375,10 @@ include boot/common.mk
>   include linux/linux.mk
>   include system/system.mk
>
> +ifeq ($(BR2_EXTERNAL_USED),y)
> +include $(BR2_EXTERNAL)/external.mk
> +endif

  Wouldn't it be simpler to put a dummy external.mk file in
dummy-external as well? Then BR2_EXTERNAL_USED can be removed (the
following patch doesn't need it: the pattern rule will just never match
because support/dummy-external/configs doesn't exist).


  Regards,
  Arnout

> +
>   TARGETS+=target-finalize
>
>   ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
> diff --git a/support/dummy-external/Config.in b/support/dummy-external/Config.in
> new file mode 100644
> index 0000000..e69de29
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCHv4 3/5] core: allow external defconfigs to be used

[Arnout Vandecappelle]

On 29/11/13 20:00, Thomas Petazzoni wrote:
> This commit allows the user to store defconfigs in
> $BR2_EXTERNAL/configs/. To achieve this:
>
>   * It adds a new %_defconfig that looks in $BR2_EXTERNAL/configs/ for
>     the corresponding defconfig file.
>
>   * Updates the help target to also list external defconfigs.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>   Makefile | 15 +++++++++++++++
>   1 file changed, 15 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index b5a9828..eb547a2 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -754,6 +754,12 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>   	@mkdir -p $(BUILD_DIR)/buildroot-config
>   	@$(COMMON_CONFIG_ENV) $< --defconfig=$(TOPDIR)/configs/$@ $(CONFIG_CONFIG_IN)
>
> +ifeq ($(BR2_EXTERNAL_USED),y)
> +%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig outputmakefile
> +	@mkdir -p $(BUILD_DIR)/buildroot-config
> +	@$(COMMON_CONFIG_ENV) $< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
> +endif
> +
>   savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>   	@mkdir -p $(BUILD_DIR)/buildroot-config
>   	@$(COMMON_CONFIG_ENV) $< \
> @@ -862,8 +868,17 @@ endif
>   	@echo '  make V=0|1             - 0 => quiet build (default), 1 => verbose build'
>   	@echo '  make O=dir             - Locate all output files in "dir", including .config'
>   	@echo
> +	@echo 'Built-in configs'
> +	@echo
>   	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
>   	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
> +ifeq ($(BR2_EXTERNAL_USED),y)

  It's a minor nit, but I'd make this conditional on
ifneq ($(wildcard $(BR2_EXTERNAL)/configs/*_defconfig),)

  Regards,
  Arnout

> +	@echo
> +	@echo 'User-provided configs'
> +	@echo
> +	@$(foreach b, $(sort $(notdir $(wildcard $(BR2_EXTERNAL)/configs/*_defconfig))), \
> +	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
> +endif
>   	@echo
>   	@echo 'See docs/README, or generate the Buildroot manual for further details'
>   	@echo
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCHv4 4/5] docs/manual: add explanations about BR2_EXTERNAL

[Arnout Vandecappelle]

On 29/11/13 20:00, Thomas Petazzoni wrote:
> This commit updates the manual to add details on how to use the
> BR2_EXTERNAL feature.
>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>   docs/manual/customize-outside-br.txt | 132 +++++++++++++++++++++++++++++++++++
>   docs/manual/customize.txt            |   2 +
>   2 files changed, 134 insertions(+)
>   create mode 100644 docs/manual/customize-outside-br.txt
>
> diff --git a/docs/manual/customize-outside-br.txt b/docs/manual/customize-outside-br.txt
> new file mode 100644
> index 0000000..623a421
> --- /dev/null
> +++ b/docs/manual/customize-outside-br.txt
> @@ -0,0 +1,132 @@
> +// -*- mode:doc -*- ;
> +
> +[[outside-br-custom]]
> +Keeping customization outside Buildroot
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +While the Buildroot community recommends and encourages upstreaming to
> +the official Buildroot version the packages and boards support that
> +are written by developers, it is sometimes not possible or desirable,
> +due to these packages or boards being highly specific or proprietary.

While the Buildroot community recommends and encourages upstreaming the
packages and board support that are written by developers to the official
Buildroot version, this is sometimes not possible or desirable, because
these packages or boards are highly specific or proprietary.

> +
> +In this case, Buildroot users are offered two choices:
> +
> + * They can add their packages, board support and configuration files
> +   directly within the Buildroot tree, and maintain them by using
> +   branches in a version control system.
> +
> + * They can use the +BR2_EXTERNAL+ mechanism, which allows to keep
> +   package recipes, board support and configuration files outside of
> +   the Buildroot tree, while still having them nicely integrated in
> +   the build logic. The following paragraphs give details on how to
> +   use +BR2_EXTERNAL+.
> +
> ++BR2_EXTERNAL+ is an environment variable that one can use to point to
> +a directory that contains Buildroot customizations. It can be passed
> +to any Buildroot +make+ invocation, and when it is passed. It is
> +automatically saved in the hidden +.br-external+ file in the output
> +directory, so that it is not needed to pass +BR2_EXTERNAL+ at every
> ++make+ invocation. It can however be changed at any time by specifying
> +a new value, and can be removed by passing an empty value. The
> ++BR2_EXTERNAL+ path can be either an absolute or a relative path, but
> +if it's passed as a relative path, it is important to note that it is
> +interpreted relatively to the main Buildroot source directory, not the
> +Buildroot output directory.
> +
> +Some examples:
> +
> +-----
> + buildroot/ $ make BR2_EXTERNAL=../foobar menuconfig
> +-----
> +
> +Starting from now on, external definitions fromt he +../company+

  ../foobar, not ../company, otherwise people will get confuser :-)

> +directory will be used:
> +
> +-----
> + buildroot/ $ make
> + buildroot/ $ make legal-info
> +-----
> +
> +We can switch to another external definitions directory at any time:
> +
> +-----
> + buildroot/ $ make BR2_EXTERNAL=../barfoo xconfig
> +-----
> +
> +Or disable the usage of external definitions:
> +
> +-----
> + buildroot/ $ make BR2_EXTERNAL= xconfig
> +-----
> +
> +This +BR2_EXTERNAL+ then allows three different things:
> +
> + * One can store all the board-specific configuration files here, such
> +   as the kernel configuration, the root filesystem overlay, or any
> +   other configuration file for which Buildroot allows to set its
> +   location. The +BR2_EXTERNAL+ value is available within the
> +   Buildroot configuration using +$(BR2_EXTERNAL)+. As an example, one
> +   could set the +BR2_ROOTFS_OVERLAY+ Buildroot option to
> +   +$(BR2_EXTERNAL)/board/<boardname>/overlay/+ (to specify a root
> +   filesystem overlay), or the +BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE+
> +   Buildroot option to +$(BR2_EXTERNAL)/board/<boardname>/kernel.config+
> +   (to specify the location of the kernel configuration file).  + To
> +   achieve this, it is recommended but not mandatory, to store those
> +   details in directories called +board/<boardname>/+ under
> +   +BR2_EXTERNAL+.

  Maybe add "This matches the directory structure used within Buildroot."

> +
> + * One can store package recipes (i.e +Config.in+ and
> +   +<packagename>.mk+), or even custom configuration options and make
> +   logic. Buildroot automatically includes +BR2_EXTERNAL/Config.in+ to
> +   make it appear in the top-level configuration menu, and includes
> +   +BR2_EXTERNAL/external.mk+ with the rest of the makefile logic.
> ++
> +The main usage of this is to store package recipes. The recommended
> +   way to do this is to write a +BR2_EXTERNAL/Config.in+ that looks
> +   like:
> ++
> +------
> +menu "<somecompany> packages"
> +
> +source "$BR2_EXTERNAL/package/package1/Config.in"
> +source "$BR2_EXTERNAL/package/package2/Config.in"
> +
> +endmenu
> +------
> ++
> +Then, have a +BR2_EXTERNAL/external.mk' file that looks like:
> ++
> +------
> +include $(sort $(wildcard $(BR2_EXTERNAL)/package/*/*.mk))
> +------
> ++
> +And then in +BR2_EXTERNAL/package/package1+ and
> +   +BR2_EXTERNAL/package/package2+ create normal Buildroot package
> +   recipes, as explained in xref:adding-packages[].
> +
> + * One can store Buildroot defconfigs in the +configs+ subdirectory of
> +   +BR2_EXTERNAL+. Buildroot will automatically show them in the
> +   output of +make help+ and allow them to be loaded with the normal
> +   +make <name>_defconfig+ command.
> +
> +In the end, a typical +BR2_EXTERNAL+ directory organization would
> +generally be:
> +
> +-----
> +??? Config.in
> +??? external.mk
> +??? board/
> +?   ??? <boardname>/

  For completeness, maybe add a linux.config here.


  Regards,
  Arnout


> +?       ??? overlay/
> +?           ??? etc/
> +?               ??? <some file>
> +??? configs/
> +?   ??? <boardname>_defconfig
> +??? package/
> +    ??? package1/
> +        ??? Config.in
> +        ??? package1.mk
> +    ??? package2/
> +        ??? Config.in
> +        ??? package2.mk
> +------
> diff --git a/docs/manual/customize.txt b/docs/manual/customize.txt
> index 0456ef1..7e46fd8 100644
> --- a/docs/manual/customize.txt
> +++ b/docs/manual/customize.txt
> @@ -17,3 +17,5 @@ include::customize-toolchain.txt[]
>   include::customize-store.txt[]
>
>   include::customize-packages.txt[]
> +
> +include::customize-outside-br.txt[]
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCHv4 5/5] manual: fix manual generation with BR2_EXTERNAL support

[Arnout Vandecappelle]

On 29/11/13 20:00, Thomas Petazzoni wrote:
> From: Samuel Martin <s.martin49@gmail.com>
>
> Reported-by: Ryan Barnett <rjbarnet@rockwellcollins.com>
> Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Signed-off-by: Ryan Barnett <rjbarnet@rockwellcollins.com>
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>   docs/manual/manual.mk         | 1 +
>   support/scripts/kconfiglib.py | 2 +-
>   2 files changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
> index aeafd10..570318f 100644
> --- a/docs/manual/manual.mk
> +++ b/docs/manual/manual.mk
> @@ -1,6 +1,7 @@
>   manual-update-lists: manual-check-dependencies-lists
>   	$(Q)$(call MESSAGE,"Updating the manual lists...")
>   	$(Q)BR2_DEFCONFIG="" TOPDIR=$(TOPDIR) O=$(O)/docs/manual/.build \
> +		BR2_EXTERNAL=$(TOPDIR)/support/dummy-external \

  Maybe a comment should be added here, like:

# Packages included in BR2_EXTERNAL are not part of buildroot, so they
# should not be included in the manual.


  Regards,
  Arnout

>   		$(TOPDIR)/support/scripts/gen-manual-lists.py
>
>   # we can't use suitable-host-package here because that's not available in
> diff --git a/support/scripts/kconfiglib.py b/support/scripts/kconfiglib.py
> index 0704cc0..e40947c 100644
> --- a/support/scripts/kconfiglib.py
> +++ b/support/scripts/kconfiglib.py
> @@ -2074,7 +2074,7 @@ set_re   = re.compile(r"CONFIG_(\w+)=(.*)")
>   unset_re = re.compile(r"# CONFIG_(\w+) is not set")
>
>   # Regular expression for finding $-references to symbols in strings
> -sym_ref_re = re.compile(r"\$[A-Za-z_]+")
> +sym_ref_re = re.compile(r"\$[A-Za-z_][0-9A-Za-z_]*")
>
>   # Integers representing symbol types
>   UNKNOWN, BOOL, TRISTATE, STRING, HEX, INT = range(0, 6)
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCHv4 5/5] manual: fix manual generation with BR2_EXTERNAL support

[Samuel Martin]

Yann, all


2013/12/1 Yann E. MORIN <yann.morin.1998@free.fr>

> Samuel, All,
>
> On 2013-11-29 20:00 +0100, Thomas Petazzoni spake thusly:
> > From: Samuel Martin <s.martin49@gmail.com>
> >
> > Reported-by: Ryan Barnett <rjbarnet@rockwellcollins.com>
> > Signed-off-by: Samuel Martin <s.martin49@gmail.com>
> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > Signed-off-by: Ryan Barnett <rjbarnet@rockwellcollins.com>
> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > ---
> >  docs/manual/manual.mk         | 1 +
> >  support/scripts/kconfiglib.py | 2 +-
> >  2 files changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/docs/manual/manual.mk b/docs/manual/manual.mk
> > index aeafd10..570318f 100644
> > --- a/docs/manual/manual.mk
> > +++ b/docs/manual/manual.mk
> > @@ -1,6 +1,7 @@
> >  manual-update-lists: manual-check-dependencies-lists
> >       $(Q)$(call MESSAGE,"Updating the manual lists...")
> >       $(Q)BR2_DEFCONFIG="" TOPDIR=$(TOPDIR) O=$(O)/docs/manual/.build \
> > +             BR2_EXTERNAL=$(TOPDIR)/support/dummy-external \
>
> I think I understand that change: we do not want to generate the manual
> with the external packages. Please extend the commit log.
>
Indeed. This patch aimed to fix the series beforehand, I didn't think it
would be integrated as is.
I'll repost it with an explanatory commit log shortly.


> >               $(TOPDIR)/support/scripts/gen-manual-lists.py
> >
> >  # we can't use suitable-host-package here because that's not available
> in
> > diff --git a/support/scripts/kconfiglib.py
> b/support/scripts/kconfiglib.py
> > index 0704cc0..e40947c 100644
> > --- a/support/scripts/kconfiglib.py
> > +++ b/support/scripts/kconfiglib.py
> > @@ -2074,7 +2074,7 @@ set_re   = re.compile(r"CONFIG_(\w+)=(.*)")
> >  unset_re = re.compile(r"# CONFIG_(\w+) is not set")
> >
> >  # Regular expression for finding $-references to symbols in strings
> > -sym_ref_re = re.compile(r"\$[A-Za-z_]+")
> > +sym_ref_re = re.compile(r"\$[A-Za-z_][0-9A-Za-z_]*")
>
> I fail to see how this change is related to BR2_EXTERNAL. Care to
> explain (and extend the commit log at the same time), please?
>
I'll explain it in the new version of the patch.

Regards,

-- 
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131201/dce87488/attachment.html>