[PATCH] kconfig: Add merge_config.sh script

[PATCH] kconfig: Add merge_config.sh script

[John Stultz]

After noticing almost every distro has their own method of managing
config fragments, I went looking at some best practices, and wanted
to try to consolidate some of the different approaches so this fairly
simple infrastructure can be shared (and new distros/build systems
don't have to implement yet another config fragment merge script).

This script is most influenced by the Windriver tools used in
the Yocto Project, reusing some portions found there.

This script merges multiple config fragments, warning on any
overrided values. It then sets any unspecified values to their
default, then finally checks to make sure no specified value was
dropped due to unsatisfied dependencies.

I'm sure this implementation won't work for everyone, and I expect
it will need to evolve to adapt for various use cases. But I think
its a reasonable starting point.

v2:
  * Reworked to use alldefconfig instead of the proposed
olddefconfig as suggested by Sam Ravnborg.

v3:
  * Script improvements from Dmitri.
  * allnoconfig option from Darren
  * pre-make exit option from Darren
  * lots of other fixes/cleanups from Darren.
  * Fix final check to not compain about config values in comments

Please let me know if you have any comments or thoughts!

CC: Sam Ravnborg <sam@ravnborg.org>
CC: gthelen@google.com
CC: tartler@cs.fau.de
CC: Dmitry Fink <Dmitry.Fink@palm.com>
CC: Darren Hart <dvhart@linux.intel.com>
CC: Eric B Munson <ebmunson@us.ibm.com>
CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
CC: Michal Marek <mmarek@suse.cz>
CC: linux-kbuild@vger.kernel.org
Signed-off-by: John Stultz <john.stultz@linaro.org>
---
 scripts/kconfig/merge_config.sh |  117 +++++++++++++++++++++++++++++++++++++++
 1 files changed, 117 insertions(+), 0 deletions(-)
 create mode 100755 scripts/kconfig/merge_config.sh

diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
new file mode 100755
index 0000000..890276b
--- /dev/null
+++ b/scripts/kconfig/merge_config.sh
@@ -0,0 +1,117 @@
+#!/bin/sh
+#  merge_config.sh - Takes a list of config fragment values, and merges
+#  them one by one. Provides warnings on overridden values, and specified
+#  values that did not make it to the resulting .config file (due to missed
+#  dependencies or config symbol removal).
+#
+#  Portions reused from kconf_check and generate_cfg:
+#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/kconf_check
+#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/generate_cfg
+#
+#  Copyright (c) 2009-2010 Wind River Systems, Inc.
+#  Copyright 2011 Linaro
+#
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License version 2 as
+#  published by the Free Software Foundation.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#  See the GNU General Public License for more details.
+
+clean_up() {
+       rm -f $TMP_FILE
+       exit
+}
+trap clean_up SIGHUP SIGINT SIGTERM
+
+usage() {
+	echo "Usage: $0 [OPTIONS] [CONFIG [...]]"
+	echo "  -h    display this help text"
+	echo "  -m    only merge the fragments, do not execute the make command"
+	echo "  -n    use allnoconfig instead of alldefconfig"
+}
+
+MAKE=true
+ALLTARGET=alldefconfig
+
+while true; do
+	case $1 in
+	"-n")
+		ALLTARGET=allnoconfig
+		shift
+		continue
+		;;
+	"-m")
+		MAKE=false
+		shift
+		continue
+		;;
+	"-h")
+		usage
+		exit
+		;;
+	*)
+		break
+		;;
+	esac
+done
+
+
+
+MERGE_LIST=$*
+SED_CONFIG_EXP="s/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/p"
+TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
+
+# Merge files, printing warnings on overrided values
+for MERGE_FILE in $MERGE_LIST ; do
+	echo "Merging $MERGE_FILE"
+	CFG_LIST=$(sed -n "$SED_CONFIG_EXP" $MERGE_FILE)
+
+	for CFG in $CFG_LIST ; do
+		grep -q -w $CFG $TMP_FILE
+		if [ $? -eq 0 ] ; then
+			PREV_VAL=$(grep -w $CFG $TMP_FILE)
+			NEW_VAL=$(grep -w $CFG $MERGE_FILE)
+			if [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then
+			echo Value of $CFG is redefined by fragment $MERGE_FILE:
+			echo Previous  value: $PREV_VAL
+			echo New value:       $NEW_VAL
+			echo
+			fi
+			sed -i "/$CFG[ =]/d" $TMP_FILE
+		fi
+	done
+	cat $MERGE_FILE >> $TMP_FILE
+done
+
+if [ "$MAKE" = "false" ]; then
+	cp $TMP_FILE .config
+	echo "#"
+	echo "# merged configuration written to .config (needs make)"
+	echo "#"
+	clean_up
+	exit
+fi
+
+# Use the merged file as the starting point for:
+# alldefconfig: Fills in any missing symbols with Kconfig default
+# allnoconfig: Fills in any missing symbols with # CONFIG_* is not set
+make KCONFIG_ALLCONFIG=$TMP_FILE $ALLTARGET
+
+
+# Check all specified config values took (might have missed-dependency issues)
+for CFG in $(sed -n "$SED_CONFIG_EXP" $TMP_FILE); do
+
+	REQUESTED_VAL=$(sed -n "$SED_CONFIG_EXP" $TMP_FILE | grep -w -e "$CFG")
+	ACTUAL_VAL=$(sed -n "$SED_CONFIG_EXP" .config | grep -w -e "$CFG")
+	if [ "x$REQUESTED_VAL" != "x$ACTUAL_VAL" ] ; then
+		echo "Value requested for $CFG not in final .config"
+		echo "Requested value:  $REQUESTED_VAL"
+		echo "Actual value:     $ACTUAL_VAL"
+		echo ""
+	fi
+done
+
+clean_up
-- 
1.7.3.2.146.gca209

[PATCH] kconfig: Add merge_config.sh script

[John Stultz]

No changes here from the last time, but since I didn't get any
feedback last time, I wanted to send this out for possible inclusion.

After noticing almost every distro has their own method of managing
config fragments, I went looking at some best practices, and wanted
to try to consolidate some of the different approaches so this fairly
simple infrastructure can be shared (and new distros/build systems
don't have to implement yet another config fragment merge script).

This script is most influenced by the Windriver tools used in
the Yocto Project, reusing some portions found there.

This script merges multiple config fragments, warning on any
overrided values. It then sets any unspecified values to their
default, then finally checks to make sure no specified value was
dropped due to unsatisfied dependencies.

I'm sure this implementation won't work for everyone, and I expect
it will need to evolve to adapt for various use cases. But I think
its a reasonable starting point.

v2:
  * Reworked to use alldefconfig instead of the proposed
olddefconfig as suggested by Sam Ravnborg.

v3:
  * Script improvements from Dmitri.
  * allnoconfig option from Darren
  * pre-make exit option from Darren
  * lots of other fixes/cleanups from Darren.
  * Fix final check to not compain about config values in comments

Please let me know if you have any comments or thoughts!

CC: Sam Ravnborg <sam@ravnborg.org>
CC: gthelen@google.com
CC: tartler@cs.fau.de
CC: Dmitry Fink <Dmitry.Fink@palm.com>
CC: Darren Hart <dvhart@linux.intel.com>
CC: Eric B Munson <ebmunson@us.ibm.com>
CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
CC: Michal Marek <mmarek@suse.cz>
CC: linux-kbuild@vger.kernel.org
Signed-off-by: John Stultz <john.stultz@linaro.org>
---
 scripts/kconfig/merge_config.sh |  117 +++++++++++++++++++++++++++++++++++++++
 1 files changed, 117 insertions(+), 0 deletions(-)
 create mode 100755 scripts/kconfig/merge_config.sh

diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
new file mode 100755
index 0000000..890276b
--- /dev/null
+++ b/scripts/kconfig/merge_config.sh
@@ -0,0 +1,117 @@
+#!/bin/sh
+#  merge_config.sh - Takes a list of config fragment values, and merges
+#  them one by one. Provides warnings on overridden values, and specified
+#  values that did not make it to the resulting .config file (due to missed
+#  dependencies or config symbol removal).
+#
+#  Portions reused from kconf_check and generate_cfg:
+#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/kconf_check
+#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/generate_cfg
+#
+#  Copyright (c) 2009-2010 Wind River Systems, Inc.
+#  Copyright 2011 Linaro
+#
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License version 2 as
+#  published by the Free Software Foundation.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#  See the GNU General Public License for more details.
+
+clean_up() {
+       rm -f $TMP_FILE
+       exit
+}
+trap clean_up SIGHUP SIGINT SIGTERM
+
+usage() {
+	echo "Usage: $0 [OPTIONS] [CONFIG [...]]"
+	echo "  -h    display this help text"
+	echo "  -m    only merge the fragments, do not execute the make command"
+	echo "  -n    use allnoconfig instead of alldefconfig"
+}
+
+MAKE=true
+ALLTARGET=alldefconfig
+
+while true; do
+	case $1 in
+	"-n")
+		ALLTARGET=allnoconfig
+		shift
+		continue
+		;;
+	"-m")
+		MAKE=false
+		shift
+		continue
+		;;
+	"-h")
+		usage
+		exit
+		;;
+	*)
+		break
+		;;
+	esac
+done
+
+
+
+MERGE_LIST=$*
+SED_CONFIG_EXP="s/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/p"
+TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
+
+# Merge files, printing warnings on overrided values
+for MERGE_FILE in $MERGE_LIST ; do
+	echo "Merging $MERGE_FILE"
+	CFG_LIST=$(sed -n "$SED_CONFIG_EXP" $MERGE_FILE)
+
+	for CFG in $CFG_LIST ; do
+		grep -q -w $CFG $TMP_FILE
+		if [ $? -eq 0 ] ; then
+			PREV_VAL=$(grep -w $CFG $TMP_FILE)
+			NEW_VAL=$(grep -w $CFG $MERGE_FILE)
+			if [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then
+			echo Value of $CFG is redefined by fragment $MERGE_FILE:
+			echo Previous  value: $PREV_VAL
+			echo New value:       $NEW_VAL
+			echo
+			fi
+			sed -i "/$CFG[ =]/d" $TMP_FILE
+		fi
+	done
+	cat $MERGE_FILE >> $TMP_FILE
+done
+
+if [ "$MAKE" = "false" ]; then
+	cp $TMP_FILE .config
+	echo "#"
+	echo "# merged configuration written to .config (needs make)"
+	echo "#"
+	clean_up
+	exit
+fi
+
+# Use the merged file as the starting point for:
+# alldefconfig: Fills in any missing symbols with Kconfig default
+# allnoconfig: Fills in any missing symbols with # CONFIG_* is not set
+make KCONFIG_ALLCONFIG=$TMP_FILE $ALLTARGET
+
+
+# Check all specified config values took (might have missed-dependency issues)
+for CFG in $(sed -n "$SED_CONFIG_EXP" $TMP_FILE); do
+
+	REQUESTED_VAL=$(sed -n "$SED_CONFIG_EXP" $TMP_FILE | grep -w -e "$CFG")
+	ACTUAL_VAL=$(sed -n "$SED_CONFIG_EXP" .config | grep -w -e "$CFG")
+	if [ "x$REQUESTED_VAL" != "x$ACTUAL_VAL" ] ; then
+		echo "Value requested for $CFG not in final .config"
+		echo "Requested value:  $REQUESTED_VAL"
+		echo "Actual value:     $ACTUAL_VAL"
+		echo ""
+	fi
+done
+
+clean_up
-- 
1.7.3.2.146.gca209

Re: [PATCH] kconfig: Add merge_config.sh script

[Darren Hart]

On 10/25/2011 12:48 AM, John Stultz wrote:
> No changes here from the last time, but since I didn't get any
> feedback last time, I wanted to send this out for possible inclusion.
> 
> After noticing almost every distro has their own method of managing
> config fragments, I went looking at some best practices, and wanted
> to try to consolidate some of the different approaches so this fairly
> simple infrastructure can be shared (and new distros/build systems
> don't have to implement yet another config fragment merge script).
>
> This script is most influenced by the Windriver tools used in
> the Yocto Project, reusing some portions found there.

This adds some additional features to what we currently do in the Yocto
Project now and we would like to just reuse what is upstream rather than
rolling our own. This would be good to have upstream from our perspective.

--
Darren

> 
> This script merges multiple config fragments, warning on any
> overrided values. It then sets any unspecified values to their
> default, then finally checks to make sure no specified value was
> dropped due to unsatisfied dependencies.
> 
> I'm sure this implementation won't work for everyone, and I expect
> it will need to evolve to adapt for various use cases. But I think
> its a reasonable starting point.
> 
> v2:
>   * Reworked to use alldefconfig instead of the proposed
> olddefconfig as suggested by Sam Ravnborg.
> 
> v3:
>   * Script improvements from Dmitri.
>   * allnoconfig option from Darren
>   * pre-make exit option from Darren
>   * lots of other fixes/cleanups from Darren.
>   * Fix final check to not compain about config values in comments
> 
> Please let me know if you have any comments or thoughts!
> 
> CC: Sam Ravnborg <sam@ravnborg.org>
> CC: gthelen@google.com
> CC: tartler@cs.fau.de
> CC: Dmitry Fink <Dmitry.Fink@palm.com>
> CC: Darren Hart <dvhart@linux.intel.com>
> CC: Eric B Munson <ebmunson@us.ibm.com>
> CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
> CC: Michal Marek <mmarek@suse.cz>
> CC: linux-kbuild@vger.kernel.org
> Signed-off-by: John Stultz <john.stultz@linaro.org>
> ---
>  scripts/kconfig/merge_config.sh |  117 +++++++++++++++++++++++++++++++++++++++
>  1 files changed, 117 insertions(+), 0 deletions(-)
>  create mode 100755 scripts/kconfig/merge_config.sh
> 
> diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
> new file mode 100755
> index 0000000..890276b
> --- /dev/null
> +++ b/scripts/kconfig/merge_config.sh
> @@ -0,0 +1,117 @@
> +#!/bin/sh
> +#  merge_config.sh - Takes a list of config fragment values, and merges
> +#  them one by one. Provides warnings on overridden values, and specified
> +#  values that did not make it to the resulting .config file (due to missed
> +#  dependencies or config symbol removal).
> +#
> +#  Portions reused from kconf_check and generate_cfg:
> +#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/kconf_check
> +#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/generate_cfg
> +#
> +#  Copyright (c) 2009-2010 Wind River Systems, Inc.
> +#  Copyright 2011 Linaro
> +#
> +#  This program is free software; you can redistribute it and/or modify
> +#  it under the terms of the GNU General Public License version 2 as
> +#  published by the Free Software Foundation.
> +#
> +#  This program is distributed in the hope that it will be useful,
> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> +#  See the GNU General Public License for more details.
> +
> +clean_up() {
> +       rm -f $TMP_FILE
> +       exit
> +}
> +trap clean_up SIGHUP SIGINT SIGTERM
> +
> +usage() {
> +	echo "Usage: $0 [OPTIONS] [CONFIG [...]]"
> +	echo "  -h    display this help text"
> +	echo "  -m    only merge the fragments, do not execute the make command"
> +	echo "  -n    use allnoconfig instead of alldefconfig"
> +}
> +
> +MAKE=true
> +ALLTARGET=alldefconfig
> +
> +while true; do
> +	case $1 in
> +	"-n")
> +		ALLTARGET=allnoconfig
> +		shift
> +		continue
> +		;;
> +	"-m")
> +		MAKE=false
> +		shift
> +		continue
> +		;;
> +	"-h")
> +		usage
> +		exit
> +		;;
> +	*)
> +		break
> +		;;
> +	esac
> +done
> +
> +
> +
> +MERGE_LIST=$*
> +SED_CONFIG_EXP="s/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/p"
> +TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
> +
> +# Merge files, printing warnings on overrided values
> +for MERGE_FILE in $MERGE_LIST ; do
> +	echo "Merging $MERGE_FILE"
> +	CFG_LIST=$(sed -n "$SED_CONFIG_EXP" $MERGE_FILE)
> +
> +	for CFG in $CFG_LIST ; do
> +		grep -q -w $CFG $TMP_FILE
> +		if [ $? -eq 0 ] ; then
> +			PREV_VAL=$(grep -w $CFG $TMP_FILE)
> +			NEW_VAL=$(grep -w $CFG $MERGE_FILE)
> +			if [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then
> +			echo Value of $CFG is redefined by fragment $MERGE_FILE:
> +			echo Previous  value: $PREV_VAL
> +			echo New value:       $NEW_VAL
> +			echo
> +			fi
> +			sed -i "/$CFG[ =]/d" $TMP_FILE
> +		fi
> +	done
> +	cat $MERGE_FILE >> $TMP_FILE
> +done
> +
> +if [ "$MAKE" = "false" ]; then
> +	cp $TMP_FILE .config
> +	echo "#"
> +	echo "# merged configuration written to .config (needs make)"
> +	echo "#"
> +	clean_up
> +	exit
> +fi
> +
> +# Use the merged file as the starting point for:
> +# alldefconfig: Fills in any missing symbols with Kconfig default
> +# allnoconfig: Fills in any missing symbols with # CONFIG_* is not set
> +make KCONFIG_ALLCONFIG=$TMP_FILE $ALLTARGET
> +
> +
> +# Check all specified config values took (might have missed-dependency issues)
> +for CFG in $(sed -n "$SED_CONFIG_EXP" $TMP_FILE); do
> +
> +	REQUESTED_VAL=$(sed -n "$SED_CONFIG_EXP" $TMP_FILE | grep -w -e "$CFG")
> +	ACTUAL_VAL=$(sed -n "$SED_CONFIG_EXP" .config | grep -w -e "$CFG")
> +	if [ "x$REQUESTED_VAL" != "x$ACTUAL_VAL" ] ; then
> +		echo "Value requested for $CFG not in final .config"
> +		echo "Requested value:  $REQUESTED_VAL"
> +		echo "Actual value:     $ACTUAL_VAL"
> +		echo ""
> +	fi
> +done
> +
> +clean_up

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

[PATCH] kconfig: Add merge_config.sh script

[John Stultz]

After noticing almost every distro has their own method of managing
config fragments, I went looking at some best practices, and wanted
to try to consolidate some of the different approaches so this fairly
simple infrastructure can be shared (and new distros/build systems
don't have to implement yet another config fragment merge script).

This script is most influenced by the Windriver tools used in
the Yocto Project, reusing some portions found there.

This script merges multiple config fragments, warning on any
overrided values. It then sets any unspecified values to their
default, then finally checks to make sure no specified value was
dropped due to unsatisfied dependencies.

I'm sure I'm sure this implementation won't work for everyone, and
I expect it will need to evolve to adapt for various use cases.
But I think its a reasonable starting point.

v2: Reworked to use alldefconfig instead of the proposed
olddefconfig as suggested by Sam Ravnborg.

Thanks to Darren Hart for early review and feedback!

Please let me know if you have any comments or thoughts!

CC: Sam Ravnborg <sam@ravnborg.org>
CC: gthelen@google.com
CC: tartler@cs.fau.de
CC: Dmitry Fink <Dmitry.Fink@palm.com>
CC: Darren Hart <dvhart@linux.intel.com>
CC: Eric B Munson <ebmunson@us.ibm.com>
CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
CC: Michal Marek <mmarek@suse.cz>
CC: linux-kbuild@vger.kernel.org
Signed-off-by: John Stultz <john.stultz@linaro.org>
---
 scripts/kconfig/merge_config.sh |   71 +++++++++++++++++++++++++++++++++++++++
 1 files changed, 71 insertions(+), 0 deletions(-)
 create mode 100755 scripts/kconfig/merge_config.sh

diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
new file mode 100755
index 0000000..fda0139
--- /dev/null
+++ b/scripts/kconfig/merge_config.sh
@@ -0,0 +1,71 @@
+#!/bin/sh
+#  merge_config.sh - Takes a list of config fragment values, and merges
+#  them one by one. Provides warnings on overridden values, and specified
+#  values that did not make it to the resulting .config file (due to missed
+#  dependencies or config symbol removal).
+#
+#  Portions reused from kconf_check and generate_cfg:
+#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/kconf_check
+#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/generate_cfg
+#
+#  Copyright (c) 2009-2010 Wind River Systems, Inc.
+#  Copyright 2011 Linaro
+#
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License version 2 as
+#  published by the Free Software Foundation.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#  See the GNU General Public License for more details.
+
+MERGE_LIST=$*
+
+
+TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
+
+# Merge files, printing warnings on overrided values
+for MERGE_FILE in $MERGE_LIST ; do
+	echo "Merging $MERGE_FILE"
+	CFG_LIST=`cat $MERGE_FILE | \
+	  sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
+	for CFG in $CFG_LIST ; do
+		grep -q -w $CFG $TMP_FILE
+		if [ $? == 0 ] ; then
+			PREV_VAL=`grep -w $CFG $TMP_FILE`
+			NEW_VAL=`grep -w $CFG $MERGE_FILE`
+			if [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then
+			echo Value of $CFG is redefined by fragment $MERGE_FILE:
+			echo Previous  value: `grep -w $CFG $TMP_FILE`
+			echo New value:       `grep -w $CFG $MERGE_FILE`
+			echo
+			fi
+			sed -i "/$CFG[ =]/d" $TMP_FILE
+		fi
+	done
+	cat $MERGE_FILE >> $TMP_FILE
+done
+
+
+# Use the merged file as the starting point for alldefconfig
+# (Fills in any missing symbols with Kconfig default)
+make KCONFIG_ALLCONFIG=$TMP_FILE alldefconfig
+
+# Check all specified config values took (might have missed-dependency issues)
+cat $TMP_FILE | while read line; do
+	CFG=`echo $line | \
+		sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
+
+	REQUESTED_VAL=`grep -w -e "$CFG" $TMP_FILE`
+	ACTUAL_VAL=`grep -w -e "$CFG" .config`
+	if [ "x$REQUESTED_VAL" != "x$ACTUAL_VAL" ] ; then
+		echo "Value requested for $CFG not in final .config"
+		echo "Requested value:  $REQUESTED_VAL"
+		echo "Actual value:     $ACTUAL_VAL"
+		echo ""
+	fi
+done
+
+# Cleanup
+rm $TMP_FILE
-- 
1.7.3.2.146.gca209

Re: [PATCH] kconfig: Add merge_config.sh script

[Richard Cochran]

On Tue, Sep 20, 2011 at 10:22:41PM -0700, John Stultz wrote:
> After noticing almost every distro has their own method of managing
> config fragments, I went looking at some best practices, and wanted
> to try to consolidate some of the different approaches so this fairly
> simple infrastructure can be shared (and new distros/build systems
> don't have to implement yet another config fragment merge script).

John,

I have been wanting to make or find a way to start with an allnoconfig
and enable a short list of config options (and the options that are
needed to allow them). I don't want the other defaults at all.

Can I use your script for this? If not, is there another way to go
about it?

Thanks,
Richard

Re: [PATCH] kconfig: Add merge_config.sh script

[John Stultz]

On Wed, 2011-09-21 at 08:44 +0200, Richard Cochran wrote:
> On Tue, Sep 20, 2011 at 10:22:41PM -0700, John Stultz wrote:
> > After noticing almost every distro has their own method of managing
> > config fragments, I went looking at some best practices, and wanted
> > to try to consolidate some of the different approaches so this fairly
> > simple infrastructure can be shared (and new distros/build systems
> > don't have to implement yet another config fragment merge script).
> 
> John,
> 
> I have been wanting to make or find a way to start with an allnoconfig
> and enable a short list of config options (and the options that are
> needed to allow them). I don't want the other defaults at all.

"make oldnoconfig" provides something close to this, I think. 

> Can I use your script for this? If not, is there another way to go
> about it?

I guess my script could take an option to use oldnoconfig instead of the
alldefconfig method.

thanks
-john

Re: [PATCH] kconfig: Add merge_config.sh script

[Darren Hart]

On 09/21/2011 08:18 AM, John Stultz wrote:
> On Wed, 2011-09-21 at 08:44 +0200, Richard Cochran wrote:
>> On Tue, Sep 20, 2011 at 10:22:41PM -0700, John Stultz wrote:
>>> After noticing almost every distro has their own method of managing
>>> config fragments, I went looking at some best practices, and wanted
>>> to try to consolidate some of the different approaches so this fairly
>>> simple infrastructure can be shared (and new distros/build systems
>>> don't have to implement yet another config fragment merge script).
>>
>> John,
>>
>> I have been wanting to make or find a way to start with an allnoconfig
>> and enable a short list of config options (and the options that are
>> needed to allow them). I don't want the other defaults at all.
> 
> "make oldnoconfig" provides something close to this, I think. 
> 
>> Can I use your script for this? If not, is there another way to go
>> about it?
> 
> I guess my script could take an option to use oldnoconfig instead of the
> alldefconfig method.

I had considered suggesting this as well. It would be useful for
assembling a bare minimum configuration.

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: [PATCH] kconfig: Add merge_config.sh script

[Sam Ravnborg]

On Wed, Sep 21, 2011 at 08:44:23AM +0200, Richard Cochran wrote:
> On Tue, Sep 20, 2011 at 10:22:41PM -0700, John Stultz wrote:
> > After noticing almost every distro has their own method of managing
> > config fragments, I went looking at some best practices, and wanted
> > to try to consolidate some of the different approaches so this fairly
> > simple infrastructure can be shared (and new distros/build systems
> > don't have to implement yet another config fragment merge script).
> 
> John,
> 
> I have been wanting to make or find a way to start with an allnoconfig
> and enable a short list of config options (and the options that are
> needed to allow them). I don't want the other defaults at all.

>From Documentation/kbuild/kconfig.txt:

===================================================================
This 'KCONFIG_ALLCONFIG' file is a config file which contains
(usually a subset of all) preset config symbols.  These variable
settings are still subject to normal dependency checks.

Examples:
        KCONFIG_ALLCONFIG=custom-notebook.config make allnoconfig
or
        KCONFIG_ALLCONFIG=mini.config make allnoconfig
or
        make KCONFIG_ALLCONFIG=mini.config allnoconfig

These examples will disable most options (allnoconfig) but enable or
disable the options that are explicitly listed in the specified
mini-config files.
===================================================================

I think this should do what you want.

	Sam

Re: [PATCH] kconfig: Add merge_config.sh script

[Michal Marek]

On 21.9.2011 07:22, John Stultz wrote:
> After noticing almost every distro has their own method of managing
> config fragments, I went looking at some best practices, and wanted
> to try to consolidate some of the different approaches so this fairly
> simple infrastructure can be shared (and new distros/build systems
> don't have to implement yet another config fragment merge script).
> 
> This script is most influenced by the Windriver tools used in
> the Yocto Project, reusing some portions found there.
> 
> This script merges multiple config fragments, warning on any
> overrided values. It then sets any unspecified values to their
> default, then finally checks to make sure no specified value was
> dropped due to unsatisfied dependencies.
> 
> I'm sure I'm sure this implementation won't work for everyone, and
> I expect it will need to evolve to adapt for various use cases.
> But I think its a reasonable starting point.
> 
> v2: Reworked to use alldefconfig instead of the proposed
> olddefconfig as suggested by Sam Ravnborg.
> 
> Thanks to Darren Hart for early review and feedback!
> 
> Please let me know if you have any comments or thoughts!
> 
> CC: Sam Ravnborg <sam@ravnborg.org>
> CC: gthelen@google.com
> CC: tartler@cs.fau.de
> CC: Dmitry Fink <Dmitry.Fink@palm.com>
> CC: Darren Hart <dvhart@linux.intel.com>
> CC: Eric B Munson <ebmunson@us.ibm.com>
> CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
> CC: Michal Marek <mmarek@suse.cz>
> CC: linux-kbuild@vger.kernel.org
> Signed-off-by: John Stultz <john.stultz@linaro.org>
> ---
>  scripts/kconfig/merge_config.sh |   71 +++++++++++++++++++++++++++++++++++++++
>  1 files changed, 71 insertions(+), 0 deletions(-)
>  create mode 100755 scripts/kconfig/merge_config.sh
> 
> diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
> new file mode 100755
> index 0000000..fda0139
> --- /dev/null
> +++ b/scripts/kconfig/merge_config.sh
> @@ -0,0 +1,71 @@
> +#!/bin/sh
> +#  merge_config.sh - Takes a list of config fragment values, and merges
> +#  them one by one. Provides warnings on overridden values, and specified
> +#  values that did not make it to the resulting .config file (due to missed
> +#  dependencies or config symbol removal).
> +#
> +#  Portions reused from kconf_check and generate_cfg:
> +#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/kconf_check
> +#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/generate_cfg
> +#
> +#  Copyright (c) 2009-2010 Wind River Systems, Inc.
> +#  Copyright 2011 Linaro
> +#
> +#  This program is free software; you can redistribute it and/or modify
> +#  it under the terms of the GNU General Public License version 2 as
> +#  published by the Free Software Foundation.
> +#
> +#  This program is distributed in the hope that it will be useful,
> +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
> +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> +#  See the GNU General Public License for more details.
> +
> +MERGE_LIST=$*
> +
> +
> +TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
> +
> +# Merge files, printing warnings on overrided values
> +for MERGE_FILE in $MERGE_LIST ; do
> +	echo "Merging $MERGE_FILE"
> +	CFG_LIST=`cat $MERGE_FILE | \
> +	  sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
> +	for CFG in $CFG_LIST ; do
> +		grep -q -w $CFG $TMP_FILE
> +		if [ $? == 0 ] ; then
> +			PREV_VAL=`grep -w $CFG $TMP_FILE`
> +			NEW_VAL=`grep -w $CFG $MERGE_FILE`
> +			if [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then
> +			echo Value of $CFG is redefined by fragment $MERGE_FILE:
> +			echo Previous  value: `grep -w $CFG $TMP_FILE`
> +			echo New value:       `grep -w $CFG $MERGE_FILE`

kconfig already warns when a symbol appears multiple times in .config.
If really wanted, a verbose mode that show the  previous and new value
could be added as well.


> +			echo
> +			fi
> +			sed -i "/$CFG[ =]/d" $TMP_FILE
> +		fi
> +	done
> +	cat $MERGE_FILE >> $TMP_FILE
> +done
> +
> +
> +# Use the merged file as the starting point for alldefconfig
> +# (Fills in any missing symbols with Kconfig default)
> +make KCONFIG_ALLCONFIG=$TMP_FILE alldefconfig
> +
> +# Check all specified config values took (might have missed-dependency issues)
> +cat $TMP_FILE | while read line; do
> +	CFG=`echo $line | \
> +		sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
> +
> +	REQUESTED_VAL=`grep -w -e "$CFG" $TMP_FILE`
> +	ACTUAL_VAL=`grep -w -e "$CFG" .config`
> +	if [ "x$REQUESTED_VAL" != "x$ACTUAL_VAL" ] ; then
> +		echo "Value requested for $CFG not in final .config"
> +		echo "Requested value:  $REQUESTED_VAL"
> +		echo "Actual value:     $ACTUAL_VAL"
> +		echo ""

This requested vs. actual diff would be nice to have in kconfig I think.
The the merge script could be reduced to a simple cat(1) invocation.

Michal

Re: [PATCH] kconfig: Add merge_config.sh script

[John Stultz]

On Wed, 2011-09-21 at 14:10 +0200, Michal Marek wrote:
> On 21.9.2011 07:22, John Stultz wrote:
> > After noticing almost every distro has their own method of managing
> > config fragments, I went looking at some best practices, and wanted
> > to try to consolidate some of the different approaches so this fairly
> > simple infrastructure can be shared (and new distros/build systems
> > don't have to implement yet another config fragment merge script).
> > 
> > This script is most influenced by the Windriver tools used in
> > the Yocto Project, reusing some portions found there.
> > 
> > This script merges multiple config fragments, warning on any
> > overrided values. It then sets any unspecified values to their
> > default, then finally checks to make sure no specified value was
> > dropped due to unsatisfied dependencies.
> > 
> > I'm sure I'm sure this implementation won't work for everyone, and
> > I expect it will need to evolve to adapt for various use cases.
> > But I think its a reasonable starting point.
> > 
> > v2: Reworked to use alldefconfig instead of the proposed
> > olddefconfig as suggested by Sam Ravnborg.
> > 
> > Thanks to Darren Hart for early review and feedback!
> > 
> > Please let me know if you have any comments or thoughts!
> > 
> > CC: Sam Ravnborg <sam@ravnborg.org>
> > CC: gthelen@google.com
> > CC: tartler@cs.fau.de
> > CC: Dmitry Fink <Dmitry.Fink@palm.com>
> > CC: Darren Hart <dvhart@linux.intel.com>
> > CC: Eric B Munson <ebmunson@us.ibm.com>
> > CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
> > CC: Michal Marek <mmarek@suse.cz>
> > CC: linux-kbuild@vger.kernel.org
> > Signed-off-by: John Stultz <john.stultz@linaro.org>
> > ---
> >  scripts/kconfig/merge_config.sh |   71 +++++++++++++++++++++++++++++++++++++++
> >  1 files changed, 71 insertions(+), 0 deletions(-)
> >  create mode 100755 scripts/kconfig/merge_config.sh
> > 
> > diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
> > new file mode 100755
> > index 0000000..fda0139
> > --- /dev/null
> > +++ b/scripts/kconfig/merge_config.sh
> > @@ -0,0 +1,71 @@
> > +#!/bin/sh
> > +#  merge_config.sh - Takes a list of config fragment values, and merges
> > +#  them one by one. Provides warnings on overridden values, and specified
> > +#  values that did not make it to the resulting .config file (due to missed
> > +#  dependencies or config symbol removal).
> > +#
> > +#  Portions reused from kconf_check and generate_cfg:
> > +#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/kconf_check
> > +#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/generate_cfg
> > +#
> > +#  Copyright (c) 2009-2010 Wind River Systems, Inc.
> > +#  Copyright 2011 Linaro
> > +#
> > +#  This program is free software; you can redistribute it and/or modify
> > +#  it under the terms of the GNU General Public License version 2 as
> > +#  published by the Free Software Foundation.
> > +#
> > +#  This program is distributed in the hope that it will be useful,
> > +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> > +#  See the GNU General Public License for more details.
> > +
> > +MERGE_LIST=$*
> > +
> > +
> > +TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
> > +
> > +# Merge files, printing warnings on overrided values
> > +for MERGE_FILE in $MERGE_LIST ; do
> > +	echo "Merging $MERGE_FILE"
> > +	CFG_LIST=`cat $MERGE_FILE | \
> > +	  sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
> > +	for CFG in $CFG_LIST ; do
> > +		grep -q -w $CFG $TMP_FILE
> > +		if [ $? == 0 ] ; then
> > +			PREV_VAL=`grep -w $CFG $TMP_FILE`
> > +			NEW_VAL=`grep -w $CFG $MERGE_FILE`
> > +			if [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then
> > +			echo Value of $CFG is redefined by fragment $MERGE_FILE:
> > +			echo Previous  value: `grep -w $CFG $TMP_FILE`
> > +			echo New value:       `grep -w $CFG $MERGE_FILE`
> 
> kconfig already warns when a symbol appears multiple times in .config.
> If really wanted, a verbose mode that show the  previous and new value
> could be added as well.

Indeed. Unfortunately the override messages at evaluation time becomes
less informative since it cannot detect from which of the fragments the
duplication/override came from.


> > +# Use the merged file as the starting point for alldefconfig
> > +# (Fills in any missing symbols with Kconfig default)
> > +make KCONFIG_ALLCONFIG=$TMP_FILE alldefconfig
> > +
> > +# Check all specified config values took (might have missed-dependency issues)
> > +cat $TMP_FILE | while read line; do
> > +	CFG=`echo $line | \
> > +		sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
> > +
> > +	REQUESTED_VAL=`grep -w -e "$CFG" $TMP_FILE`
> > +	ACTUAL_VAL=`grep -w -e "$CFG" .config`
> > +	if [ "x$REQUESTED_VAL" != "x$ACTUAL_VAL" ] ; then
> > +		echo "Value requested for $CFG not in final .config"
> > +		echo "Requested value:  $REQUESTED_VAL"
> > +		echo "Actual value:     $ACTUAL_VAL"
> > +		echo ""
> 
> This requested vs. actual diff would be nice to have in kconfig I think.
> The the merge script could be reduced to a simple cat(1) invocation.

I actually started with a simple cat call, but that has limitations:
Specifically when checking what was requested and what took in
the .config, we don't want to flag explicitly overrided values (for
instance in file A: CONFIG_FOO=y, in file B:# CONFIG_FOO is not set, we
don't want to complain that CONFIG_FOO=y isn't set.) That's why we do
the de-duplication first, and then compare the deduplicated input with
the output.

thanks
-john

Re: [PATCH] kconfig: Add merge_config.sh script

[Dmitry Fink (Palm GBU)]

Hi John,

I think you may want to apply the following fix:
--- a/scripts/kconfig/merge_config.sh
+++ b/scripts/kconfig/merge_config.sh
@@ -32,7 +32,7 @@ for MERGE_FILE in $MERGE_LIST ; do
          sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
        for CFG in $CFG_LIST ; do
                grep -q -w $CFG $TMP_FILE
-               if [ $? == 0 ] ; then
+               if [ $? -eq 0 ] ; then

or just change it single "=" sign, "==" seems to be a bashism and failed
on ubuntu's /bin/sh.


You also have a minor typo in the commit msg:)
"I'm sure I'm sure this implementation won't work for everyone, and"
 ^        ^

Regards,
Dmitry

On 9/20/11 10:22 PM, "John Stultz" <john.stultz@linaro.org> wrote:

>After noticing almost every distro has their own method of managing
>config fragments, I went looking at some best practices, and wanted
>to try to consolidate some of the different approaches so this fairly
>simple infrastructure can be shared (and new distros/build systems
>don't have to implement yet another config fragment merge script).
>
>This script is most influenced by the Windriver tools used in
>the Yocto Project, reusing some portions found there.
>
>This script merges multiple config fragments, warning on any
>overrided values. It then sets any unspecified values to their
>default, then finally checks to make sure no specified value was
>dropped due to unsatisfied dependencies.
>
>I'm sure I'm sure this implementation won't work for everyone, and
>I expect it will need to evolve to adapt for various use cases.
>But I think its a reasonable starting point.
>
>v2: Reworked to use alldefconfig instead of the proposed
>olddefconfig as suggested by Sam Ravnborg.
>
>Thanks to Darren Hart for early review and feedback!
>
>Please let me know if you have any comments or thoughts!
>
>CC: Sam Ravnborg <sam@ravnborg.org>
>CC: gthelen@google.com
>CC: tartler@cs.fau.de
>CC: Dmitry Fink <Dmitry.Fink@palm.com>
>CC: Darren Hart <dvhart@linux.intel.com>
>CC: Eric B Munson <ebmunson@us.ibm.com>
>CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
>CC: Michal Marek <mmarek@suse.cz>
>CC: linux-kbuild@vger.kernel.org
>Signed-off-by: John Stultz <john.stultz@linaro.org>
>---
> scripts/kconfig/merge_config.sh |   71
>+++++++++++++++++++++++++++++++++++++++
> 1 files changed, 71 insertions(+), 0 deletions(-)
> create mode 100755 scripts/kconfig/merge_config.sh
>
>diff --git a/scripts/kconfig/merge_config.sh
>b/scripts/kconfig/merge_config.sh
>new file mode 100755
>index 0000000..fda0139
>--- /dev/null
>+++ b/scripts/kconfig/merge_config.sh
>@@ -0,0 +1,71 @@
>+#!/bin/sh
>+#  merge_config.sh - Takes a list of config fragment values, and merges
>+#  them one by one. Provides warnings on overridden values, and specified
>+#  values that did not make it to the resulting .config file (due to
>missed
>+#  dependencies or config symbol removal).
>+#
>+#  Portions reused from kconf_check and generate_cfg:
>+#  
>http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/kc
>onf_check
>+#  
>http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/ge
>nerate_cfg
>+#
>+#  Copyright (c) 2009-2010 Wind River Systems, Inc.
>+#  Copyright 2011 Linaro
>+#
>+#  This program is free software; you can redistribute it and/or modify
>+#  it under the terms of the GNU General Public License version 2 as
>+#  published by the Free Software Foundation.
>+#
>+#  This program is distributed in the hope that it will be useful,
>+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
>+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>+#  See the GNU General Public License for more details.
>+
>+MERGE_LIST=$*
>+
>+
>+TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
>+
>+# Merge files, printing warnings on overrided values
>+for MERGE_FILE in $MERGE_LIST ; do
>+	echo "Merging $MERGE_FILE"
>+	CFG_LIST=`cat $MERGE_FILE | \
>+	  sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
>+	for CFG in $CFG_LIST ; do
>+		grep -q -w $CFG $TMP_FILE
>+		if [ $? == 0 ] ; then
>+			PREV_VAL=`grep -w $CFG $TMP_FILE`
>+			NEW_VAL=`grep -w $CFG $MERGE_FILE`
>+			if [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then
>+			echo Value of $CFG is redefined by fragment $MERGE_FILE:
>+			echo Previous  value: `grep -w $CFG $TMP_FILE`
>+			echo New value:       `grep -w $CFG $MERGE_FILE`
>+			echo
>+			fi
>+			sed -i "/$CFG[ =]/d" $TMP_FILE
>+		fi
>+	done
>+	cat $MERGE_FILE >> $TMP_FILE
>+done
>+
>+
>+# Use the merged file as the starting point for alldefconfig
>+# (Fills in any missing symbols with Kconfig default)
>+make KCONFIG_ALLCONFIG=$TMP_FILE alldefconfig
>+
>+# Check all specified config values took (might have missed-dependency
>issues)
>+cat $TMP_FILE | while read line; do
>+	CFG=`echo $line | \
>+		sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
>+
>+	REQUESTED_VAL=`grep -w -e "$CFG" $TMP_FILE`
>+	ACTUAL_VAL=`grep -w -e "$CFG" .config`
>+	if [ "x$REQUESTED_VAL" != "x$ACTUAL_VAL" ] ; then
>+		echo "Value requested for $CFG not in final .config"
>+		echo "Requested value:  $REQUESTED_VAL"
>+		echo "Actual value:     $ACTUAL_VAL"
>+		echo ""
>+	fi
>+done
>+
>+# Cleanup
>+rm $TMP_FILE
>-- 
>1.7.3.2.146.gca209
>

Re: [PATCH] kconfig: Add merge_config.sh script

[Dmitry Fink (Palm GBU)]

Hi, John
Here is another suggestion:

--- a/scripts/kconfig/merge_config.sh
+++ b/scripts/kconfig/merge_config.sh
@@ -28,11 +28,11 @@ TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
 # Merge files, printing warnings on overrided values
 for MERGE_FILE in $MERGE_LIST ; do
 	echo "Merging $MERGE_FILE"
-	CFG_LIST=`cat $MERGE_FILE | \
+	CFG_LIST=`cat $MERGE_FILE | grep -e "CONFIG_[a-zA-Z0-9_]*" | \
 	  sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
 	for CFG in $CFG_LIST ; do
 		grep -q -w $CFG $TMP_FILE
@@ -53,7 +53,7 @@ done
 make KCONFIG_ALLCONFIG=$TMP_FILE alldefconfig
 
 # Check all specified config values took (might have missed-dependency
issues)
-cat $TMP_FILE | while read line; do
+cat $TMP_FILE | grep -e "CONFIG_[a-zA-Z0-9_]*" | while read line; do
 	CFG=`echo $line | \
 		sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`

The original script chokes on fragments that include empty lines or section
comments (and think we do want them in fragments). There might be a nicer
way to
handle  it, this is the quickest fix I found, feel free to consider a
different
approach.

Dmitry


On 9/20/11 10:22 PM, "John Stultz" <john.stultz@linaro.org> wrote:

>After noticing almost every distro has their own method of managing
>config fragments, I went looking at some best practices, and wanted
>to try to consolidate some of the different approaches so this fairly
>simple infrastructure can be shared (and new distros/build systems
>don't have to implement yet another config fragment merge script).
>
>This script is most influenced by the Windriver tools used in
>the Yocto Project, reusing some portions found there.
>
>This script merges multiple config fragments, warning on any
>overrided values. It then sets any unspecified values to their
>default, then finally checks to make sure no specified value was
>dropped due to unsatisfied dependencies.
>
>I'm sure I'm sure this implementation won't work for everyone, and
>I expect it will need to evolve to adapt for various use cases.
>But I think its a reasonable starting point.
>
>v2: Reworked to use alldefconfig instead of the proposed
>olddefconfig as suggested by Sam Ravnborg.
>
>Thanks to Darren Hart for early review and feedback!
>
>Please let me know if you have any comments or thoughts!
>
>CC: Sam Ravnborg <sam@ravnborg.org>
>CC: gthelen@google.com
>CC: tartler@cs.fau.de
>CC: Dmitry Fink <Dmitry.Fink@palm.com>
>CC: Darren Hart <dvhart@linux.intel.com>
>CC: Eric B Munson <ebmunson@us.ibm.com>
>CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
>CC: Michal Marek <mmarek@suse.cz>
>CC: linux-kbuild@vger.kernel.org
>Signed-off-by: John Stultz <john.stultz@linaro.org>
>---
> scripts/kconfig/merge_config.sh |   71
>+++++++++++++++++++++++++++++++++++++++
> 1 files changed, 71 insertions(+), 0 deletions(-)
> create mode 100755 scripts/kconfig/merge_config.sh
>
>diff --git a/scripts/kconfig/merge_config.sh
>b/scripts/kconfig/merge_config.sh
>new file mode 100755
>index 0000000..fda0139
>--- /dev/null
>+++ b/scripts/kconfig/merge_config.sh
>@@ -0,0 +1,71 @@
>+#!/bin/sh
>+#  merge_config.sh - Takes a list of config fragment values, and merges
>+#  them one by one. Provides warnings on overridden values, and specified
>+#  values that did not make it to the resulting .config file (due to
>missed
>+#  dependencies or config symbol removal).
>+#
>+#  Portions reused from kconf_check and generate_cfg:
>+#  
>http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/kc
>onf_check
>+#  
>http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/ge
>nerate_cfg
>+#
>+#  Copyright (c) 2009-2010 Wind River Systems, Inc.
>+#  Copyright 2011 Linaro
>+#
>+#  This program is free software; you can redistribute it and/or modify
>+#  it under the terms of the GNU General Public License version 2 as
>+#  published by the Free Software Foundation.
>+#
>+#  This program is distributed in the hope that it will be useful,
>+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
>+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>+#  See the GNU General Public License for more details.
>+
>+MERGE_LIST=$*
>+
>+
>+TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
>+
>+# Merge files, printing warnings on overrided values
>+for MERGE_FILE in $MERGE_LIST ; do
>+	echo "Merging $MERGE_FILE"
>+	CFG_LIST=`cat $MERGE_FILE | \
>+	  sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
>+	for CFG in $CFG_LIST ; do
>+		grep -q -w $CFG $TMP_FILE
>+		if [ $? == 0 ] ; then
>+			PREV_VAL=`grep -w $CFG $TMP_FILE`
>+			NEW_VAL=`grep -w $CFG $MERGE_FILE`
>+			if [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then
>+			echo Value of $CFG is redefined by fragment $MERGE_FILE:
>+			echo Previous  value: `grep -w $CFG $TMP_FILE`
>+			echo New value:       `grep -w $CFG $MERGE_FILE`
>+			echo
>+			fi
>+			sed -i "/$CFG[ =]/d" $TMP_FILE
>+		fi
>+	done
>+	cat $MERGE_FILE >> $TMP_FILE
>+done
>+
>+
>+# Use the merged file as the starting point for alldefconfig
>+# (Fills in any missing symbols with Kconfig default)
>+make KCONFIG_ALLCONFIG=$TMP_FILE alldefconfig
>+
>+# Check all specified config values took (might have missed-dependency
>issues)
>+cat $TMP_FILE | while read line; do
>+	CFG=`echo $line | \
>+		sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
>+
>+	REQUESTED_VAL=`grep -w -e "$CFG" $TMP_FILE`
>+	ACTUAL_VAL=`grep -w -e "$CFG" .config`
>+	if [ "x$REQUESTED_VAL" != "x$ACTUAL_VAL" ] ; then
>+		echo "Value requested for $CFG not in final .config"
>+		echo "Requested value:  $REQUESTED_VAL"
>+		echo "Actual value:     $ACTUAL_VAL"
>+		echo ""
>+	fi
>+done
>+
>+# Cleanup
>+rm $TMP_FILE
>-- 
>1.7.3.2.146.gca209

Re: [PATCH] kconfig: Add merge_config.sh script

[Arnaud Lacombe]

Hi,

On Wed, Sep 21, 2011 at 9:20 PM, Dmitry Fink (Palm GBU)
<Dmitry.Fink@palm.com> wrote:
> Hi, John
> Here is another suggestion:
>
> --- a/scripts/kconfig/merge_config.sh
> +++ b/scripts/kconfig/merge_config.sh
> @@ -28,11 +28,11 @@ TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
>  # Merge files, printing warnings on overrided values
>  for MERGE_FILE in $MERGE_LIST ; do
>        echo "Merging $MERGE_FILE"
> -       CFG_LIST=`cat $MERGE_FILE | \
> +       CFG_LIST=`cat $MERGE_FILE | grep -e "CONFIG_[a-zA-Z0-9_]*" | \
>          sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
3 process, when only 1 is really needed, the following:

CFG_LIST=`sed '/CONFIG_[a-zA-Z0-9_]*/!d; s/^\(#
\)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/' $MERGE_FILE`

should do the job; however, it's completely untested.

 - Arnaud

>        for CFG in $CFG_LIST ; do
>                grep -q -w $CFG $TMP_FILE
> @@ -53,7 +53,7 @@ done
>  make KCONFIG_ALLCONFIG=$TMP_FILE alldefconfig
>
>  # Check all specified config values took (might have missed-dependency
> issues)
> -cat $TMP_FILE | while read line; do
> +cat $TMP_FILE | grep -e "CONFIG_[a-zA-Z0-9_]*" | while read line; do
>        CFG=`echo $line | \
>                sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
>
> The original script chokes on fragments that include empty lines or section
> comments (and think we do want them in fragments). There might be a nicer
> way to
> handle  it, this is the quickest fix I found, feel free to consider a
> different
> approach.
>
> Dmitry
>
>
> On 9/20/11 10:22 PM, "John Stultz" <john.stultz@linaro.org> wrote:
>
>>After noticing almost every distro has their own method of managing
>>config fragments, I went looking at some best practices, and wanted
>>to try to consolidate some of the different approaches so this fairly
>>simple infrastructure can be shared (and new distros/build systems
>>don't have to implement yet another config fragment merge script).
>>
>>This script is most influenced by the Windriver tools used in
>>the Yocto Project, reusing some portions found there.
>>
>>This script merges multiple config fragments, warning on any
>>overrided values. It then sets any unspecified values to their
>>default, then finally checks to make sure no specified value was
>>dropped due to unsatisfied dependencies.
>>
>>I'm sure I'm sure this implementation won't work for everyone, and
>>I expect it will need to evolve to adapt for various use cases.
>>But I think its a reasonable starting point.
>>
>>v2: Reworked to use alldefconfig instead of the proposed
>>olddefconfig as suggested by Sam Ravnborg.
>>
>>Thanks to Darren Hart for early review and feedback!
>>
>>Please let me know if you have any comments or thoughts!
>>
>>CC: Sam Ravnborg <sam@ravnborg.org>
>>CC: gthelen@google.com
>>CC: tartler@cs.fau.de
>>CC: Dmitry Fink <Dmitry.Fink@palm.com>
>>CC: Darren Hart <dvhart@linux.intel.com>
>>CC: Eric B Munson <ebmunson@us.ibm.com>
>>CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
>>CC: Michal Marek <mmarek@suse.cz>
>>CC: linux-kbuild@vger.kernel.org
>>Signed-off-by: John Stultz <john.stultz@linaro.org>
>>---
>> scripts/kconfig/merge_config.sh |   71
>>+++++++++++++++++++++++++++++++++++++++
>> 1 files changed, 71 insertions(+), 0 deletions(-)
>> create mode 100755 scripts/kconfig/merge_config.sh
>>
>>diff --git a/scripts/kconfig/merge_config.sh
>>b/scripts/kconfig/merge_config.sh
>>new file mode 100755
>>index 0000000..fda0139
>>--- /dev/null
>>+++ b/scripts/kconfig/merge_config.sh
>>@@ -0,0 +1,71 @@
>>+#!/bin/sh
>>+#  merge_config.sh - Takes a list of config fragment values, and merges
>>+#  them one by one. Provides warnings on overridden values, and specified
>>+#  values that did not make it to the resulting .config file (due to
>>missed
>>+#  dependencies or config symbol removal).
>>+#
>>+#  Portions reused from kconf_check and generate_cfg:
>>+#
>>http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/kc
>>onf_check
>>+#
>>http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/ge
>>nerate_cfg
>>+#
>>+#  Copyright (c) 2009-2010 Wind River Systems, Inc.
>>+#  Copyright 2011 Linaro
>>+#
>>+#  This program is free software; you can redistribute it and/or modify
>>+#  it under the terms of the GNU General Public License version 2 as
>>+#  published by the Free Software Foundation.
>>+#
>>+#  This program is distributed in the hope that it will be useful,
>>+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
>>+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>+#  See the GNU General Public License for more details.
>>+
>>+MERGE_LIST=$*
>>+
>>+
>>+TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
>>+
>>+# Merge files, printing warnings on overrided values
>>+for MERGE_FILE in $MERGE_LIST ; do
>>+      echo "Merging $MERGE_FILE"
>>+      CFG_LIST=`cat $MERGE_FILE | \
>>+        sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
>>+      for CFG in $CFG_LIST ; do
>>+              grep -q -w $CFG $TMP_FILE
>>+              if [ $? == 0 ] ; then
>>+                      PREV_VAL=`grep -w $CFG $TMP_FILE`
>>+                      NEW_VAL=`grep -w $CFG $MERGE_FILE`
>>+                      if [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then
>>+                      echo Value of $CFG is redefined by fragment $MERGE_FILE:
>>+                      echo Previous  value: `grep -w $CFG $TMP_FILE`
>>+                      echo New value:       `grep -w $CFG $MERGE_FILE`
>>+                      echo
>>+                      fi
>>+                      sed -i "/$CFG[ =]/d" $TMP_FILE
>>+              fi
>>+      done
>>+      cat $MERGE_FILE >> $TMP_FILE
>>+done
>>+
>>+
>>+# Use the merged file as the starting point for alldefconfig
>>+# (Fills in any missing symbols with Kconfig default)
>>+make KCONFIG_ALLCONFIG=$TMP_FILE alldefconfig
>>+
>>+# Check all specified config values took (might have missed-dependency
>>issues)
>>+cat $TMP_FILE | while read line; do
>>+      CFG=`echo $line | \
>>+              sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
>>+
>>+      REQUESTED_VAL=`grep -w -e "$CFG" $TMP_FILE`
>>+      ACTUAL_VAL=`grep -w -e "$CFG" .config`
>>+      if [ "x$REQUESTED_VAL" != "x$ACTUAL_VAL" ] ; then
>>+              echo "Value requested for $CFG not in final .config"
>>+              echo "Requested value:  $REQUESTED_VAL"
>>+              echo "Actual value:     $ACTUAL_VAL"
>>+              echo ""
>>+      fi
>>+done
>>+
>>+# Cleanup
>>+rm $TMP_FILE
>>--
>>1.7.3.2.146.gca209
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>