[Buildroot] [PATCH v3 0/6] Makefile: Support merged and nested defconfigs.

[Buildroot] [PATCH v3 0/6] Makefile: Support merged and nested defconfigs.

[Sam Bobroff]

This is a combination and cleanup of the previous efforts by
Sam Bobroff and Elizabeth Liner to introduce merged defconfigs
and nested defconfig directories respectively.

Merged defconfigs allow us to create defconfig snippets for aspects
that are common across multiple machines and create a combined defconfig
from the snippets.  The support scripts for merged kernel defconfigs is reused.

Nested defconfig directories allow us to organize the defconfig
directory into sub-directories.  A likely use for this is to place
defconfigs into architecture-specific subdirectories.


Changes v2 -> v3:

* Added missing Signed-off-bys.
* Split documentation into separate patch.

Patch 4/6: Makefile: Add nested config dirs to list-defconfigs

* Removed unnecessary "/" ("wildcard" adds a trailing "/" to directories).


Changes v1 -> v2:

This is a complete rework of the config merging system, discarding the .merge
files and instead using Makefile fragments included from the configuration
directories that invoke a Make function. This allows much better error handling
as well as a cleaner looking system (somewhat similar to the package .mk
files).

I've also added some documentation to explain what is supported and an example
to show how it's used.  I haven't touched the documentation before and I'm not
familiar with the formatting so I hope I've done it the right way and that it's
useful (feedback would be great). "make manual-html" seems to work at least.


Elizabeth Liner (1):
  Makefile: Support nested config directories

Patrick Williams (3):
  Makefile: Generate %_defconfig recipes from macro.
  Makefile: Add nested config dirs to list-defconfigs
  Makefile: Add merged defconfigs to list-defconfigs.

Sam Bobroff (2):
  Add support for merged defconfigs
  Add documentation for merged defconfigs

 Makefile                             | 58 +++++++++++++++++++++++++++++-----
 docs/manual/adding-board-support.txt |  3 ++
 docs/manual/appendix.txt             |  1 +
 docs/manual/merged-defconfigs.txt    | 60 ++++++++++++++++++++++++++++++++++++
 4 files changed, 114 insertions(+), 8 deletions(-)
 create mode 100644 docs/manual/merged-defconfigs.txt

-- 
2.1.0

[Buildroot] [PATCH v3 1/6] Add support for merged defconfigs

[Sam Bobroff]

Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
---
v3:

* Documentation moved to separate patch.
* Fixed "make savedefconfig".

 Makefile | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/Makefile b/Makefile
index 78b44c5..9f3fc92 100644
--- a/Makefile
+++ b/Makefile
@@ -858,6 +858,44 @@ savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 
 .PHONY: defconfig savedefconfig
 
+# Find the path of a config file that may be either in TOPDIR or BR2_EXTERNAL.
+# Missing or duplicate matches must be checked for elsewhere.
+define FIND_CONFIG_FILE =
+$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(wildcard $(d)/$(1)))
+endef
+
+define CONFIG_FILE_LIST =
+$(foreach f,$1,$(call FIND_CONFIG_FILE,$f))
+endef
+
+# Check that each config file is found once and only once.
+define CHECK_CONFIG_FILE_LIST =
+$(foreach f,$(1),
+ifeq "$(words $(call FIND_CONFIG_FILE,$(f)))" "0"
+$$(error ERROR: Missing input file: $(f))
+else
+ifneq "$(words $(call FIND_CONFIG_FILE,$(f)))" "1"
+$$(error ERROR: Duplicate input file: $(f))
+endif
+endif
+)
+endef
+
+# To be called by configuration fragments (*.mk) to set up defconfigs built
+# by merge_config.sh, via the include that follows:
+define merge_config =
+$(call CHECK_CONFIG_FILE_LIST,$2 $3)
+$(1): $$(BUILD_DIR)/buildroot-config/conf $(call CONFIG_FILE_LIST,$2 $3) outputmakefile
+	@mkdir $$(CONFIG_DIR)/.merge_config
+	@$$(TOPDIR)/support/kconfig/merge_config.sh -m -O $$(CONFIG_DIR)/.merge_config \
+		$(call CONFIG_FILE_LIST,$2 $3)
+	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$$(CONFIG_DIR)/.merge_config/.config \
+		$$< --defconfig=$$(CONFIG_DIR)/.merge_config/.config $$(CONFIG_CONFIG_IN)
+	@rm -rf $$(CONFIG_DIR)/.merge_config
+endef
+
+$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(eval -include $(d)/*.mk))
+
 ################################################################################
 #
 # Cleanup and misc junk
-- 
2.1.0

[Buildroot] [PATCH v3 2/6] Add documentation for merged defconfigs

[Sam Bobroff]

Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
---
 docs/manual/adding-board-support.txt |  3 ++
 docs/manual/appendix.txt             |  1 +
 docs/manual/merged-defconfigs.txt    | 60 ++++++++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+)
 create mode 100644 docs/manual/merged-defconfigs.txt

diff --git a/docs/manual/adding-board-support.txt b/docs/manual/adding-board-support.txt
index f6d74ae..086a67e 100644
--- a/docs/manual/adding-board-support.txt
+++ b/docs/manual/adding-board-support.txt
@@ -24,6 +24,9 @@ savedefconfig+. This will generate a minimal +defconfig+ file at the
 root of the Buildroot source tree. Move this file into the +configs/+
 directory, and rename it +<boardname>_defconfig+.
 
+It is also possible to create configurations by specifying differences from
+an existing configuration. See xref:merged-defconfigs[].
+
 It is recommended to use as much as possible upstream versions of the
 Linux kernel and bootloaders, and to use as much as possible default
 kernel and bootloader configurations. If they are incorrect for your
diff --git a/docs/manual/appendix.txt b/docs/manual/appendix.txt
index 87a20bd..9abc37e 100644
--- a/docs/manual/appendix.txt
+++ b/docs/manual/appendix.txt
@@ -3,6 +3,7 @@
 
 include::makedev-syntax.txt[]
 include::makeusers-syntax.txt[]
+include::merged-defconfigs.txt[]
 
 
 // Automatically generated lists:
diff --git a/docs/manual/merged-defconfigs.txt b/docs/manual/merged-defconfigs.txt
new file mode 100644
index 0000000..e1f50fd
--- /dev/null
+++ b/docs/manual/merged-defconfigs.txt
@@ -0,0 +1,60 @@
+// -*- mode:doc -*- ;
+// vim: set syntax=asciidoc:
+
+[[merged-defconfigs]]
+== Merged configurations
+
+Buildroot supports the use of the Linux kernel's +merge_config.sh+ script to
+create configurations by merging a set of partial configurations (or overlays)
+on top of a base configuration. This may be useful when there are several
+configurations that are specializations of a single generic configuration.
+
+Files in the +configs/+ directory with the extension
++.mk+ will be read by the top level Makefile and the Make function
++merge_config+ can be used within them to define new merged configurations.
+
+The merge_config function takes three parameters: the first is the name of the
+new configuration (which should end with +_defconfig+), the second is the name
+of the base configuration file and the third is a space separated list of the
+partial configuration files to merge from the +configs/+ directory. All files
+should be specified relative to the +configs/+ directory and the name of the
+Make fragment file should match the name of the configuration it creates.
+
+The new configuration can be selected as if it were a normal configuration, i.e. with
++make foo_defconfig+.
+
+Partial configurations have the same format as full configuration
+files except that they contain only a subset of values. They can be created by
+starting with a normal configuration file and deleting lines as appropriate.
+
+External configuration files from the +$(BR2_EXTERNAL)/configs+ directory are fully
+supported, and merge configurations may freely refer to files from either
+location.
+
+Example:
+
+If configs/foo_defconfig contains:
+-----
+BR2_TARGET_GENERIC_HOSTNAME="buildroot"
+-----
+
+And configs/bar_defconfig.mk contains:
+-----
+$(eval $(call merge_config,bar_defconfig,foo_defconfig,b.config))
+-----
+
+And configs/b.config contains:
+-----
+BR2_TARGET_GENERIC_ISSUE="Welcome to BAR"
+-----
+
+After running the command:
+-----
+make bar_defconfig
+-----
+
+The resulting .config file would contain:
+-----
+BR2_TARGET_GENERIC_HOSTNAME="buildroot"
+BR2_TARGET_GENERIC_ISSUE="Welcome to BAR"
+-----
-- 
2.1.0

[Buildroot] [PATCH v3 3/6] Makefile: Generate %_defconfig recipes from macro.

[Sam Bobroff]

From: Patrick Williams <patrick@stwcx.xyz>

To reduce duplication in the %_defconfig recipes with $(TOPDIR) and
$(BR2_EXTERNAL) versions, generate these from a macro.  The macro is
now called on a list of directories containing the appropriate ones.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
---
 Makefile | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/Makefile b/Makefile
index 9f3fc92..e78b9e2 100644
--- a/Makefile
+++ b/Makefile
@@ -842,13 +842,14 @@ defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< --defconfig$(if $(DEFCONFIG),=$(DEFCONFIG)) $(CONFIG_CONFIG_IN)
 
 # Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig
-%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(TOPDIR)/configs/%_defconfig outputmakefile
-	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(TOPDIR)/configs/$@ \
-		$< --defconfig=$(TOPDIR)/configs/$@ $(CONFIG_CONFIG_IN)
+define CREATE_DEFCONFIG_RECIPES
+%_defconfig: $$(BUILD_DIR)/buildroot-config/conf $1/%_defconfig outputmakefile
+	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$$(TOPDIR)/configs/$$@ \
+		$$< --defconfig=$1/$$@ $$(CONFIG_CONFIG_IN)
+endef
 
-%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(BR2_EXTERNAL)/configs/%_defconfig outputmakefile
-	@$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(BR2_EXTERNAL)/configs/$@ \
-		$< --defconfig=$(BR2_EXTERNAL)/configs/$@ $(CONFIG_CONFIG_IN)
+BR2_DEFCONFIG_PATHS=$(TOPDIR)/configs $(BR2_EXTERNAL)/configs
+$(foreach path,$(BR2_DEFCONFIG_PATHS),$(eval $(call CREATE_DEFCONFIG_RECIPES,$(path))))
 
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
 	@$(COMMON_CONFIG_ENV) $< \
@@ -894,7 +895,7 @@ $(1): $$(BUILD_DIR)/buildroot-config/conf $(call CONFIG_FILE_LIST,$2 $3) outputm
 	@rm -rf $$(CONFIG_DIR)/.merge_config
 endef
 
-$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(eval -include $(d)/*.mk))
+$(foreach d,$(BR2_DEFCONFIG_PATHS),$(eval -include $(d)/*.mk))
 
 ################################################################################
 #
-- 
2.1.0

[Buildroot] [PATCH v3 4/6] Makefile: Support nested config directories

[Sam Bobroff]

From: Elizabeth Liner <eliner@us.ibm.com>

Extend the BR2_DEFCONFIG_PATH to support nested config directories.
This allows us to place config files in both .../config and
.../config/<arch>.

Both $(TOPDIR) and $(BR2_EXTERNAL) are supported.

Signed-off-by: Elizabeth Liner <eliner@us.ibm.com>
Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
---
 Makefile | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index e78b9e2..fd07f8c 100644
--- a/Makefile
+++ b/Makefile
@@ -848,7 +848,8 @@ define CREATE_DEFCONFIG_RECIPES
 		$$< --defconfig=$1/$$@ $$(CONFIG_CONFIG_IN)
 endef
 
-BR2_DEFCONFIG_PATHS=$(TOPDIR)/configs $(BR2_EXTERNAL)/configs
+BR2_DEFCONFIG_PATHS=$(sort $(dir $(wildcard $(TOPDIR)/configs/*/))) \
+		$(sort $(dir $(wildcard $(BR2_EXTERNAL)/configs/*/)))
 $(foreach path,$(BR2_DEFCONFIG_PATHS),$(eval $(call CREATE_DEFCONFIG_RECIPES,$(path))))
 
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
-- 
2.1.0

[Buildroot] [PATCH v3 5/6] Makefile: Add nested config dirs to list-defconfigs

[Sam Bobroff]

From: Patrick Williams <patrick@stwcx.xyz>

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
---
v3:

* Removed unnecessary "/" ("wildcard" adds a trailing "/" to directories).

 Makefile | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index fd07f8c..da6008e 100644
--- a/Makefile
+++ b/Makefile
@@ -206,6 +206,10 @@ LEGAL_LICENSES_TXT_HOST = $(LEGAL_INFO_DIR)/host-licenses.txt
 LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings
 LEGAL_REPORT = $(LEGAL_INFO_DIR)/README
 
+BR2_DEFCONFIG_PATHS_LOCAL=$(sort $(dir $(wildcard $(TOPDIR)/configs/*)))
+BR2_DEFCONFIG_PATHS_USER=$(sort $(dir $(wildcard $(BR2_EXTERNAL)/configs/*)))
+BR2_DEFCONFIG_PATHS=$(BR2_DEFCONFIG_PATHS_LOCAL) $(BR2_DEFCONFIG_PATHS_USER)
+
 BR2_CONFIG = $(CONFIG_DIR)/.config
 
 # Pull in the user's configuration file
@@ -848,8 +852,6 @@ define CREATE_DEFCONFIG_RECIPES
 		$$< --defconfig=$1/$$@ $$(CONFIG_CONFIG_IN)
 endef
 
-BR2_DEFCONFIG_PATHS=$(sort $(dir $(wildcard $(TOPDIR)/configs/*/))) \
-		$(sort $(dir $(wildcard $(BR2_EXTERNAL)/configs/*/)))
 $(foreach path,$(BR2_DEFCONFIG_PATHS),$(eval $(call CREATE_DEFCONFIG_RECIPES,$(path))))
 
 savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
@@ -1009,12 +1011,12 @@ help:
 
 list-defconfigs:
 	@echo 'Built-in configs:'
-	@$(foreach b, $(sort $(notdir $(wildcard $(TOPDIR)/configs/*_defconfig))), \
+	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_LOCAL),$(wildcard $(path)/*_defconfig)))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 ifneq ($(wildcard $(BR2_EXTERNAL)/configs/*_defconfig),)
 	@echo
 	@echo 'User-provided configs:'
-	@$(foreach b, $(sort $(notdir $(wildcard $(BR2_EXTERNAL)/configs/*_defconfig))), \
+	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_USER),$(wildcard $(path)/*_defconfig)))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 endif
 	@echo
-- 
2.1.0

[Buildroot] [PATCH v3 6/6] Makefile: Add merged defconfigs to list-defconfigs.

[Sam Bobroff]

From: Patrick Williams <patrick@stwcx.xyz>

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
---
 Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index da6008e..329db79 100644
--- a/Makefile
+++ b/Makefile
@@ -1011,12 +1011,12 @@ help:
 
 list-defconfigs:
 	@echo 'Built-in configs:'
-	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_LOCAL),$(wildcard $(path)/*_defconfig)))), \
+	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_LOCAL),$(basename $(wildcard $(path)/*_defconfig $(path)/*_defconfig.mk))))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 ifneq ($(wildcard $(BR2_EXTERNAL)/configs/*_defconfig),)
 	@echo
 	@echo 'User-provided configs:'
-	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_USER),$(wildcard $(path)/*_defconfig)))), \
+	@$(foreach b, $(sort $(notdir $(foreach path,$(BR2_DEFCONFIG_PATHS_USER),$(basename $(wildcard $(path)/*_defconfig $(path)/*_defconfig.mk))))), \
 	  printf "  %-35s - Build for %s\\n" $(b) $(b:_defconfig=);)
 endif
 	@echo
-- 
2.1.0

[Buildroot] [PATCH v3 1/6] Add support for merged defconfigs

[Romain Naour]

Hi Sam,

Le 29/06/2016 ? 07:06, Sam Bobroff a ?crit :
> Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
> ---
> v3:
> 
> * Documentation moved to separate patch.
> * Fixed "make savedefconfig".

Sorry, I don't see how "make savedefconfig" has been fixed in this patch.
The patch seems the same as v2.

With the series applied I get the same error.

Best regards,
Romain

> 
>  Makefile | 38 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 38 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index 78b44c5..9f3fc92 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -858,6 +858,44 @@ savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
>  
>  .PHONY: defconfig savedefconfig
>  
> +# Find the path of a config file that may be either in TOPDIR or BR2_EXTERNAL.
> +# Missing or duplicate matches must be checked for elsewhere.
> +define FIND_CONFIG_FILE =
> +$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(wildcard $(d)/$(1)))
> +endef
> +
> +define CONFIG_FILE_LIST =
> +$(foreach f,$1,$(call FIND_CONFIG_FILE,$f))
> +endef
> +
> +# Check that each config file is found once and only once.
> +define CHECK_CONFIG_FILE_LIST =
> +$(foreach f,$(1),
> +ifeq "$(words $(call FIND_CONFIG_FILE,$(f)))" "0"
> +$$(error ERROR: Missing input file: $(f))
> +else
> +ifneq "$(words $(call FIND_CONFIG_FILE,$(f)))" "1"
> +$$(error ERROR: Duplicate input file: $(f))
> +endif
> +endif
> +)
> +endef
> +
> +# To be called by configuration fragments (*.mk) to set up defconfigs built
> +# by merge_config.sh, via the include that follows:
> +define merge_config =
> +$(call CHECK_CONFIG_FILE_LIST,$2 $3)
> +$(1): $$(BUILD_DIR)/buildroot-config/conf $(call CONFIG_FILE_LIST,$2 $3) outputmakefile
> +	@mkdir $$(CONFIG_DIR)/.merge_config
> +	@$$(TOPDIR)/support/kconfig/merge_config.sh -m -O $$(CONFIG_DIR)/.merge_config \
> +		$(call CONFIG_FILE_LIST,$2 $3)
> +	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$$(CONFIG_DIR)/.merge_config/.config \
> +		$$< --defconfig=$$(CONFIG_DIR)/.merge_config/.config $$(CONFIG_CONFIG_IN)
> +	@rm -rf $$(CONFIG_DIR)/.merge_config
> +endef
> +
> +$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(eval -include $(d)/*.mk))
> +
>  ################################################################################
>  #
>  # Cleanup and misc junk
> 

[Buildroot] [PATCH v3 2/6] Add documentation for merged defconfigs

[Romain Naour]

Hi Sam,

Le 29/06/2016 ? 07:06, Sam Bobroff a ?crit :
> Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
> ---
>  docs/manual/adding-board-support.txt |  3 ++
>  docs/manual/appendix.txt             |  1 +
>  docs/manual/merged-defconfigs.txt    | 60 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 64 insertions(+)
>  create mode 100644 docs/manual/merged-defconfigs.txt
> 
> diff --git a/docs/manual/adding-board-support.txt b/docs/manual/adding-board-support.txt
> index f6d74ae..086a67e 100644
> --- a/docs/manual/adding-board-support.txt
> +++ b/docs/manual/adding-board-support.txt
> @@ -24,6 +24,9 @@ savedefconfig+. This will generate a minimal +defconfig+ file at the
>  root of the Buildroot source tree. Move this file into the +configs/+
>  directory, and rename it +<boardname>_defconfig+.
>  
> +It is also possible to create configurations by specifying differences from
> +an existing configuration. See xref:merged-defconfigs[].
> +
>  It is recommended to use as much as possible upstream versions of the
>  Linux kernel and bootloaders, and to use as much as possible default
>  kernel and bootloader configurations. If they are incorrect for your
> diff --git a/docs/manual/appendix.txt b/docs/manual/appendix.txt
> index 87a20bd..9abc37e 100644
> --- a/docs/manual/appendix.txt
> +++ b/docs/manual/appendix.txt
> @@ -3,6 +3,7 @@
>  
>  include::makedev-syntax.txt[]
>  include::makeusers-syntax.txt[]
> +include::merged-defconfigs.txt[]
>  
>  
>  // Automatically generated lists:
> diff --git a/docs/manual/merged-defconfigs.txt b/docs/manual/merged-defconfigs.txt
> new file mode 100644
> index 0000000..e1f50fd
> --- /dev/null
> +++ b/docs/manual/merged-defconfigs.txt
> @@ -0,0 +1,60 @@
> +// -*- mode:doc -*- ;
> +// vim: set syntax=asciidoc:
> +
> +[[merged-defconfigs]]
> +== Merged configurations
> +
> +Buildroot supports the use of the Linux kernel's +merge_config.sh+ script to
> +create configurations by merging a set of partial configurations (or overlays)
> +on top of a base configuration. This may be useful when there are several
> +configurations that are specializations of a single generic configuration.
> +
> +Files in the +configs/+ directory with the extension
> ++.mk+ will be read by the top level Makefile and the Make function
> ++merge_config+ can be used within them to define new merged configurations.
> +
> +The merge_config function takes three parameters: the first is the name of the
> +new configuration (which should end with +_defconfig+), the second is the name
> +of the base configuration file and the third is a space separated list of the
> +partial configuration files to merge from the +configs/+ directory. All files
> +should be specified relative to the +configs/+ directory and the name of the
> +Make fragment file should match the name of the configuration it creates.

This series has been discussed with other developers and we concluded that this
feature is not the responsibility of Buildroot to generate a defconfig from
partial configuration files.

Instead we recommend to use merge_config.sh from outside of Buildroot to
generate the merged defconfig before calling make. Maybe we should add some
documentation in the manual to explain how to use it.

Best regards,
Romain

> +
> +The new configuration can be selected as if it were a normal configuration, i.e. with
> ++make foo_defconfig+.
> +
> +Partial configurations have the same format as full configuration
> +files except that they contain only a subset of values. They can be created by
> +starting with a normal configuration file and deleting lines as appropriate.
> +
> +External configuration files from the +$(BR2_EXTERNAL)/configs+ directory are fully
> +supported, and merge configurations may freely refer to files from either
> +location.
> +
> +Example:
> +
> +If configs/foo_defconfig contains:
> +-----
> +BR2_TARGET_GENERIC_HOSTNAME="buildroot"
> +-----
> +
> +And configs/bar_defconfig.mk contains:
> +-----
> +$(eval $(call merge_config,bar_defconfig,foo_defconfig,b.config))
> +-----
> +
> +And configs/b.config contains:
> +-----
> +BR2_TARGET_GENERIC_ISSUE="Welcome to BAR"
> +-----
> +
> +After running the command:
> +-----
> +make bar_defconfig
> +-----
> +
> +The resulting .config file would contain:
> +-----
> +BR2_TARGET_GENERIC_HOSTNAME="buildroot"
> +BR2_TARGET_GENERIC_ISSUE="Welcome to BAR"
> +-----
> 

[Buildroot] [PATCH v3 1/6] Add support for merged defconfigs

[Sam Bobroff]

On Fri, Jul 01, 2016 at 10:47:30AM +0200, Romain Naour wrote:
> Hi Sam,
> 
> Le 29/06/2016 ? 07:06, Sam Bobroff a ?crit :
> > Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
> > ---
> > v3:
> > 
> > * Documentation moved to separate patch.
> > * Fixed "make savedefconfig".
> 
> Sorry, I don't see how "make savedefconfig" has been fixed in this patch.
> The patch seems the same as v2.
> 
> With the series applied I get the same error.

Ah, sorry, I managed to lose that change somehow, this part:

+	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$$(CONFIG_DIR)/.merge_config/.config \
+		$$< --defconfig=$$(CONFIG_DIR)/.merge_config/.config $$(CONFIG_CONFIG_IN)

Should have been:

+	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(dir $(call FIND_CONFIG_FILE,$2))$1 \
+		$$< --defconfig=$$(CONFIG_DIR)/.merge_config/.config $$(CONFIG_CONFIG_IN)

Which would place the saved defconfig in a file with the name of your merged
defconfig in which ever directory contained the "base" part of the config. I'm
not entirely sure what "make safedefconfig" should really mean for a merged
config but this seemed OK.

But, given that you've decided not to take the set, I won't post a new version. ;-)

Cheers,
Sam.

> Best regards,
> Romain
> 
> > 
> >  Makefile | 38 ++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 38 insertions(+)
> > 
> > diff --git a/Makefile b/Makefile
> > index 78b44c5..9f3fc92 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -858,6 +858,44 @@ savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile
> >  
> >  .PHONY: defconfig savedefconfig
> >  
> > +# Find the path of a config file that may be either in TOPDIR or BR2_EXTERNAL.
> > +# Missing or duplicate matches must be checked for elsewhere.
> > +define FIND_CONFIG_FILE =
> > +$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(wildcard $(d)/$(1)))
> > +endef
> > +
> > +define CONFIG_FILE_LIST =
> > +$(foreach f,$1,$(call FIND_CONFIG_FILE,$f))
> > +endef
> > +
> > +# Check that each config file is found once and only once.
> > +define CHECK_CONFIG_FILE_LIST =
> > +$(foreach f,$(1),
> > +ifeq "$(words $(call FIND_CONFIG_FILE,$(f)))" "0"
> > +$$(error ERROR: Missing input file: $(f))
> > +else
> > +ifneq "$(words $(call FIND_CONFIG_FILE,$(f)))" "1"
> > +$$(error ERROR: Duplicate input file: $(f))
> > +endif
> > +endif
> > +)
> > +endef
> > +
> > +# To be called by configuration fragments (*.mk) to set up defconfigs built
> > +# by merge_config.sh, via the include that follows:
> > +define merge_config =
> > +$(call CHECK_CONFIG_FILE_LIST,$2 $3)
> > +$(1): $$(BUILD_DIR)/buildroot-config/conf $(call CONFIG_FILE_LIST,$2 $3) outputmakefile
> > +	@mkdir $$(CONFIG_DIR)/.merge_config
> > +	@$$(TOPDIR)/support/kconfig/merge_config.sh -m -O $$(CONFIG_DIR)/.merge_config \
> > +		$(call CONFIG_FILE_LIST,$2 $3)
> > +	@$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$$(CONFIG_DIR)/.merge_config/.config \
> > +		$$< --defconfig=$$(CONFIG_DIR)/.merge_config/.config $$(CONFIG_CONFIG_IN)
> > +	@rm -rf $$(CONFIG_DIR)/.merge_config
> > +endef
> > +
> > +$(foreach d,$(TOPDIR)/configs $(BR2_EXTERNAL)/configs,$(eval -include $(d)/*.mk))
> > +
> >  ################################################################################
> >  #
> >  # Cleanup and misc junk
> > 

[Buildroot] [PATCH v3 2/6] Add documentation for merged defconfigs

[Sam Bobroff]

On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
> Hi Sam,
> 
[snip]
> 
> This series has been discussed with other developers and we concluded that this
> feature is not the responsibility of Buildroot to generate a defconfig from
> partial configuration files.

Fair enough.

> Instead we recommend to use merge_config.sh from outside of Buildroot to
> generate the merged defconfig before calling make. Maybe we should add some
> documentation in the manual to explain how to use it.

That seems fine, it might even be possible to add something to
$(BR2_EXTERNAL)/external.mk to hook it directly into the Makefiles.

Could you elaborate a bit on what you mean by using merge_config.sh outside of
buildroot? Would the fragment files still be checked into buildroot somewhere
(but just not hooked into the build system) or would they be maintained outside
it?

> Best regards,
> Romain

Cheers,
Sam.

[Buildroot] [PATCH v3 2/6] Add documentation for merged defconfigs

[Thomas Petazzoni]

Hello Sam,

On Tue, 5 Jul 2016 16:06:25 +1000, Sam Bobroff wrote:

> > Instead we recommend to use merge_config.sh from outside of Buildroot to
> > generate the merged defconfig before calling make. Maybe we should add some
> > documentation in the manual to explain how to use it.  
> 
> That seems fine, it might even be possible to add something to
> $(BR2_EXTERNAL)/external.mk to hook it directly into the Makefiles.
> 
> Could you elaborate a bit on what you mean by using merge_config.sh outside of
> buildroot? Would the fragment files still be checked into buildroot somewhere
> (but just not hooked into the build system) or would they be maintained outside
> it?

The idea is that Buildroot just takes as input a configuration file.
How this configuration file is provided/generated is not Buildroot's
responsibility.

For example, you could have in your BR2_EXTERNAL tree a bunch of
defconfig fragments, and then a "genconfig" shell script to generate the
final configuration you want to build using the defconfig fragments.
This "genconfig" script relies on merge_config.sh to actually merge the
relevant fragments together.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v3 2/6] Add documentation for merged defconfigs

[Patrick Williams]

On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
> Hi Sam,
> 
> This series has been discussed with other developers and we concluded that this
> feature is not the responsibility of Buildroot to generate a defconfig from
> partial configuration files.
> 
> Instead we recommend to use merge_config.sh from outside of Buildroot to
> generate the merged defconfig before calling make. Maybe we should add some
> documentation in the manual to explain how to use it.
> 
> Best regards,
> Romain
> 

Earlier we had two independent patch sets that we were asked to merge.

   1) Merged defconfigs (now this patch set).
   2) Subdirs for defconfigs.

If #1 has been rejected as undesireable, should we now prepare an
indepenent patch set for #2 or is that also not wanted now?

There is currently no way for us to manage this behavior in a
BR2_EXTERNAL layer alone because there is no Makefile from the
BR2_EXTERNAL path sourced in the defconfig case.  Is there any issue
with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?

I realize we could have a new script, but we already have users that are
use to 'make <system>_defconfig && make' so we would like to be able to
preserve that behavior even though we are moving to merged defconfig
fragments.

-- 
Patrick Williams
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160705/3485a4b1/attachment.asc>

[Buildroot] [PATCH v3 2/6] Add documentation for merged defconfigs

[Samuel Mendoza-Jonas]

On Tue, 2016-07-05 at 14:32 -0500, Patrick Williams wrote:
> On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
> > Hi Sam,
> > 
> > This series has been discussed with other developers and we concluded that this
> > feature is not the responsibility of Buildroot to generate a defconfig from
> > partial configuration files.
> > 
> > Instead we recommend to use merge_config.sh from outside of Buildroot to
> > generate the merged defconfig before calling make. Maybe we should add some
> > documentation in the manual to explain how to use it.
> > 
> > Best regards,
> > Romain
> > 
> 
> Earlier we had two independent patch sets that we were asked to merge.
> 
>    1) Merged defconfigs (now this patch set).
>    2) Subdirs for defconfigs.
> 
> If #1 has been rejected as undesireable, should we now prepare an
> indepenent patch set for #2 or is that also not wanted now?
> 
> There is currently no way for us to manage this behavior in a
> BR2_EXTERNAL layer alone because there is no Makefile from the
> BR2_EXTERNAL path sourced in the defconfig case.  Is there any issue
> with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?

Actually there might be - Sam was telling me yesterday that the main
Makefile includes?$(BR2_EXTERNAL)/external.mk, which is something we may
use to get mergeconfigs in op-build. That might work for nested configs
as well.

> 
> I realize we could have a new script, but we already have users that are
> use to 'make <system>_defconfig && make' so we would like to be able to
> preserve that behavior even though we are moving to merged defconfig
> fragments.
> 

[Buildroot] [PATCH v3 2/6] Add documentation for merged defconfigs

[Patrick Williams]

On Wed, Jul 06, 2016 at 08:13:43AM +1000, Samuel Mendoza-Jonas wrote:
> On Tue, 2016-07-05 at 14:32 -0500, Patrick Williams wrote:
> > On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
> > > Hi Sam,
> > > 
> > > This series has been discussed with other developers and we concluded that this
> > > feature is not the responsibility of Buildroot to generate a defconfig from
> > > partial configuration files.
> > > 
> > > Instead we recommend to use merge_config.sh from outside of Buildroot to
> > > generate the merged defconfig before calling make. Maybe we should add some
> > > documentation in the manual to explain how to use it.
> > > 
> > > Best regards,
> > > Romain
> > > 
> > 
> > Earlier we had two independent patch sets that we were asked to merge.
> > 
> >    1) Merged defconfigs (now this patch set).
> >    2) Subdirs for defconfigs.
> > 
> > If #1 has been rejected as undesireable, should we now prepare an
> > indepenent patch set for #2 or is that also not wanted now?
> > 
> > There is currently no way for us to manage this behavior in a
> > BR2_EXTERNAL layer alone because there is no Makefile from the
> > BR2_EXTERNAL path sourced in the defconfig case.  Is there any issue
> > with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?
> 
> Actually there might be - Sam was telling me yesterday that the main
> Makefile includes?$(BR2_EXTERNAL)/external.mk, which is something we may
> use to get mergeconfigs in op-build. That might work for nested configs
> as well.
> 

I've already been down this route.  The 'include
$(BR2_EXTERNAL)/external.mk' is inside a big 'ifeq
($(BR2_HAVE_DOT_CONFIG),y)' on line 336.  That effectively means if you
don't already have a .config (which comes about from running a
'defconfig' recipe) then you don't get a source to external.mk.  That is
why I'm asking about a new 'external-defconfig.mk'.

> > 
> > I realize we could have a new script, but we already have users that are
> > use to 'make <system>_defconfig && make' so we would like to be able to
> > preserve that behavior even though we are moving to merged defconfig
> > fragments.
> > 
> 

-- 
Patrick Williams
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20160705/d32d6c33/attachment.asc>

[Buildroot] [PATCH v3 2/6] Add documentation for merged defconfigs

[Sam Bobroff]

On Tue, Jul 05, 2016 at 08:16:44AM +0200, Thomas Petazzoni wrote:
> Hello Sam,
> 
> On Tue, 5 Jul 2016 16:06:25 +1000, Sam Bobroff wrote:
> 
> > > Instead we recommend to use merge_config.sh from outside of Buildroot to
> > > generate the merged defconfig before calling make. Maybe we should add some
> > > documentation in the manual to explain how to use it.  
> > 
> > That seems fine, it might even be possible to add something to
> > $(BR2_EXTERNAL)/external.mk to hook it directly into the Makefiles.
> > 
> > Could you elaborate a bit on what you mean by using merge_config.sh outside of
> > buildroot? Would the fragment files still be checked into buildroot somewhere
> > (but just not hooked into the build system) or would they be maintained outside
> > it?
> 
> The idea is that Buildroot just takes as input a configuration file.
> How this configuration file is provided/generated is not Buildroot's
> responsibility.
> 
> For example, you could have in your BR2_EXTERNAL tree a bunch of
> defconfig fragments, and then a "genconfig" shell script to generate the
> final configuration you want to build using the defconfig fragments.
> This "genconfig" script relies on merge_config.sh to actually merge the
> relevant fragments together.
> 

Ah right, yes I'm sure something like that will work for work that stays
external to buildroot, but the case I'm concerned about is where BR2_EXTERNAL
isn't being used. We want to get our defconfigs out there for everyone to use
and contribute to :-) It looks like existing defconfigs could make use of this
too: from a bit of diff'ing in the configs directory there are already configs
that share a lot of values and would be easier to maintain using merging.

So how would you suggest handling merged configs that should be contributed
back to the community? Would you expect that we maintain our own merge-config
system and fragments and then contribute the full configs back to buildroot?

That seems like the simplest way to do it but I can't see a good way to handle
patches to them: users who have downloaded buildroot would have the defconfigs
but not the input fragments that were used to create them, so it would be
difficult to handle changes they need to make.

Would an alternative approach be to include the fragments and genconfig system
in buildroot but make it a separate component, so it wouldn't add complexity to
the build system? (e.g. you "make configs" to update your defconfigs.)

Cheers,
Sam.

[Buildroot] [PATCH v3 2/6] Add documentation for merged defconfigs

[Romain Naour]

Hi Patrick, Samuel, All,

Le 06/07/2016 ? 00:41, Patrick Williams a ?crit :
> On Wed, Jul 06, 2016 at 08:13:43AM +1000, Samuel Mendoza-Jonas wrote:
>> On Tue, 2016-07-05 at 14:32 -0500, Patrick Williams wrote:
>>> On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
>>>> Hi Sam,
>>>>
>>>> This series has been discussed with other developers and we concluded that this
>>>> feature is not the responsibility of Buildroot to generate a defconfig from
>>>> partial configuration files.
>>>>
>>>> Instead we recommend to use merge_config.sh from outside of Buildroot to
>>>> generate the merged defconfig before calling make. Maybe we should add some
>>>> documentation in the manual to explain how to use it.
>>>>
>>>> Best regards,
>>>> Romain
>>>>
>>>
>>> Earlier we had two independent patch sets that we were asked to merge.
>>>
>>>    1) Merged defconfigs (now this patch set).
>>>    2) Subdirs for defconfigs.
>>>
>>> If #1 has been rejected as undesireable, should we now prepare an
>>> indepenent patch set for #2 or is that also not wanted now?
>>>
>>> There is currently no way for us to manage this behavior in a
>>> BR2_EXTERNAL layer alone because there is no Makefile from the
>>> BR2_EXTERNAL path sourced in the defconfig case.  Is there any issue
>>> with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?
>>
>> Actually there might be - Sam was telling me yesterday that the main
>> Makefile includes $(BR2_EXTERNAL)/external.mk, which is something we may
>> use to get mergeconfigs in op-build. That might work for nested configs
>> as well.
>>
> 
> I've already been down this route.  The 'include
> $(BR2_EXTERNAL)/external.mk' is inside a big 'ifeq
> ($(BR2_HAVE_DOT_CONFIG),y)' on line 336.  That effectively means if you
> don't already have a .config (which comes about from running a
> 'defconfig' recipe) then you don't get a source to external.mk.  That is
> why I'm asking about a new 'external-defconfig.mk'.

Patrick, you're right. It's currently no way to include an external makefile
when BR2_HAVE_DOT_CONFIG is not set. Neither from local.mk or external.mk.

Indeed, adding $(BR2_EXTERNAL)/external-defconfig.mk should work in this case.

I tried with this patch [1] and used Samuel's patch [2] applied to
external-defconfig.mk instead of the Buildroot Makefile.

Lets see what other Buidroot developers think about this, especially Yann which
is currently working on multi-BR2_EXTERNAL.

About the nested configs directory, I don't see any particular problem...

Best regards,
Romain


[1]
https://github.com/RomainNaour/buildroot/commit/396744b032567b0d7fcc6e3d56c37143efde368e
[2] http://patchwork.ozlabs.org/patch/641838/

> 
>>>
>>> I realize we could have a new script, but we already have users that are
>>> use to 'make <system>_defconfig && make' so we would like to be able to
>>> preserve that behavior even though we are moving to merged defconfig
>>> fragments.
>>>
>>
> 
> 
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 

[Buildroot] [PATCH v3 2/6] Add documentation for merged defconfigs

[Yann E. MORIN]

Romain, Samuel, Patrick, All,

On 2016-07-06 21:08 +0200, Romain Naour spake thusly:
> Le 06/07/2016 ? 00:41, Patrick Williams a ?crit :
> > On Wed, Jul 06, 2016 at 08:13:43AM +1000, Samuel Mendoza-Jonas wrote:
> >> On Tue, 2016-07-05 at 14:32 -0500, Patrick Williams wrote:
> >>> On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
> >>>> Hi Sam,
> >>>>
> >>>> This series has been discussed with other developers and we concluded that this
> >>>> feature is not the responsibility of Buildroot to generate a defconfig from
> >>>> partial configuration files.
> >>>>
> >>>> Instead we recommend to use merge_config.sh from outside of Buildroot to
> >>>> generate the merged defconfig before calling make. Maybe we should add some
> >>>> documentation in the manual to explain how to use it.
> >>>>
> >>>> Best regards,
> >>>> Romain
> >>>>
> >>>
> >>> Earlier we had two independent patch sets that we were asked to merge.
> >>>
> >>>    1) Merged defconfigs (now this patch set).
> >>>    2) Subdirs for defconfigs.
> >>>
> >>> If #1 has been rejected as undesireable, should we now prepare an
> >>> indepenent patch set for #2 or is that also not wanted now?
> >>>
> >>> There is currently no way for us to manage this behavior in a
> >>> BR2_EXTERNAL layer alone because there is no Makefile from the
> >>> BR2_EXTERNAL path sourced in the defconfig case.  Is there any issue
> >>> with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?

I can understand the frustration. I am sorry that we came to that
conclusion and that you feel disapointed.

We did reject this because, as Romain explained, we believe it is much
more flexible to handle such a case outside of Buildroot, as Romain
explained.

> >> Actually there might be - Sam was telling me yesterday that the main
> >> Makefile includes $(BR2_EXTERNAL)/external.mk, which is something we may
> >> use to get mergeconfigs in op-build. That might work for nested configs
> >> as well.
> >>
> > 
> > I've already been down this route.  The 'include
> > $(BR2_EXTERNAL)/external.mk' is inside a big 'ifeq
> > ($(BR2_HAVE_DOT_CONFIG),y)' on line 336.  That effectively means if you
> > don't already have a .config (which comes about from running a
> > 'defconfig' recipe) then you don't get a source to external.mk.  That is
> > why I'm asking about a new 'external-defconfig.mk'.
> 
> Patrick, you're right. It's currently no way to include an external makefile
> when BR2_HAVE_DOT_CONFIG is not set. Neither from local.mk or external.mk.
> 
> Indeed, adding $(BR2_EXTERNAL)/external-defconfig.mk should work in this case.
> 
> I tried with this patch [1] and used Samuel's patch [2] applied to
> external-defconfig.mk instead of the Buildroot Makefile.
[--SNIP--]
> [1] https://github.com/RomainNaour/buildroot/commit/396744b032567b0d7fcc6e3d56c37143efde368e

No, we already rejected a similar proposal (by me!) to source custom
help from br2-external:
    http://lists.busybox.net/pipermail/buildroot/2016-March/thread.html#156705
    http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#158710

The correct solution would be that we source external.mk unconditionally.
The patch is actually trivial (I had it locally some time ago, when
working on the aforementioned custom help).

For this, we should ensure it is safe to do so, but in my (very little)
epxerimentation, it seemed OK to do so. However, we must be really
carefull in doing so.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'