[PATCH 0/2] Config fragment tools

[PATCH 0/2] Config fragment tools

[John Stultz]

Last week at Plumbers, Darren Hart and I hosted a BOF on config fragments.
I wanted to say thanks to all the hungry attendees (8pm on the evening where
dinner was fend-for-yourself made it especially hard to draw a crowd) for
their thoughts and discussion. 

There I promised to send out some patches to try to get some common tooling
pushed upstream. I realize this isn't difficult code, but after seeing
almost every distro and build system re-implement the same basic
functionality over and over, it seems like there should be some tooling for
this common use case.

I'm sure this tooling won't work for everyone, and I expect it will need to
evolve to adapt for various use cases, but I wanted to push it out so that I
could make concrete what I was describing in the talk, as well as get some
wider feedback.

So let me know if you have any comments or thoughts!

thanks
-john


CC: gthelen@google.com
CC: tartler@cs.fau.de
CC: Dmitry Fink <Dmitry.Fink@palm.com>
CC: Darren Hart <dvhart@linux.intel.com>
CC: Eric B Munson <ebmunson@us.ibm.com>
CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
CC: Michal Marek <mmarek@suse.cz>
CC: linux-kbuild@vger.kernel.org

John Stultz (2):
  kconfig: Add make olddefconfig
  kconfig: Add merge_config.sh script

 scripts/kconfig/Makefile        |    2 +-
 scripts/kconfig/conf.c          |    4 ++
 scripts/kconfig/merge_config.sh |   72 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 77 insertions(+), 1 deletions(-)
 create mode 100755 scripts/kconfig/merge_config.sh

-- 
1.7.3.2.146.gca209

[PATCH 1/2] kconfig: Add make olddefconfig

[John Stultz]

This adds an "olddefconfig" build target, which updates a .config file
filling any any absent symbols with the Kconfig default value (note:
not the architecture defconfig value).

Similar to oldnoconfig, this option is useful for build systems
when dealing with config fragments.

CC: gthelen@google.com
CC: tartler@cs.fau.de
CC: Dmitry Fink <Dmitry.Fink@palm.com>
CC: Darren Hart <dvhart@linux.intel.com>
CC: Eric B Munson <ebmunson@us.ibm.com>
CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
CC: Michal Marek <mmarek@suse.cz>
CC: linux-kbuild@vger.kernel.org
Signed-off-by: John Stultz <john.stultz@linaro.org>
---
 scripts/kconfig/Makefile |    2 +-
 scripts/kconfig/conf.c   |    4 ++++
 2 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/scripts/kconfig/Makefile b/scripts/kconfig/Makefile
index 82d2eb2..c238f3e 100644
--- a/scripts/kconfig/Makefile
+++ b/scripts/kconfig/Makefile
@@ -103,7 +103,7 @@ allnoconfig allyesconfig allmodconfig alldefconfig randconfig: $(obj)/conf
 
 PHONY += listnewconfig oldnoconfig savedefconfig defconfig
 
-listnewconfig oldnoconfig: $(obj)/conf
+listnewconfig oldnoconfig olddefconfig: $(obj)/conf
 	$< --$@ $(Kconfig)
 
 savedefconfig: $(obj)/conf
diff --git a/scripts/kconfig/conf.c b/scripts/kconfig/conf.c
index f208f90..ab7bc68 100644
--- a/scripts/kconfig/conf.c
+++ b/scripts/kconfig/conf.c
@@ -33,6 +33,7 @@ enum input_mode {
 	savedefconfig,
 	listnewconfig,
 	oldnoconfig,
+	olddefconfig,
 } input_mode = oldaskconfig;
 
 static int indent = 1;
@@ -455,6 +456,7 @@ static struct option long_opts[] = {
 	{"randconfig",      no_argument,       NULL, randconfig},
 	{"listnewconfig",   no_argument,       NULL, listnewconfig},
 	{"oldnoconfig",     no_argument,       NULL, oldnoconfig},
+	{"olddefconfig",    no_argument,       NULL, olddefconfig},
 	{NULL, 0, NULL, 0}
 };
 
@@ -566,6 +568,7 @@ int main(int ac, char **av)
 	case oldconfig:
 	case listnewconfig:
 	case oldnoconfig:
+	case olddefconfig:
 		conf_read(NULL);
 		break;
 	case allnoconfig:
@@ -624,6 +627,7 @@ int main(int ac, char **av)
 		conf_set_all_new_symbols(def_random);
 		break;
 	case defconfig:
+	case olddefconfig:
 		conf_set_all_new_symbols(def_default);
 		break;
 	case savedefconfig:
-- 
1.7.3.2.146.gca209

[PATCH 2/2] kconfig: Add merge_config.sh script

[John Stultz]

After noticing almost every distro has their own method of managing
config fragments, I went looking at some best practices, and wanted
to try to consolidate some of the different approaches so this fairly
simple infrastructure can be shared (and new distros/build systems
don't have to implement yet another config fragment merge script).

This script is most influenced by the Windriver tools used in
the Yocto Project, reusing some portions found there.

This script merges multiple config fragments, warning on any
overrided values. It utilizes olddefconfig to set any unspecified values
to their default, then finally checks to make sure no specified
value was dropped due to unsatisfied dependencies.

Thanks to Darren Hart for early review and feedback!

Please let me know if you have any comments or thoughts!

CC: gthelen@google.com
CC: tartler@cs.fau.de
CC: Dmitry Fink <Dmitry.Fink@palm.com>
CC: Darren Hart <dvhart@linux.intel.com>
CC: Eric B Munson <ebmunson@us.ibm.com>
CC: Bruce Ashfield <Bruce.Ashfield@windriver.com>
CC: Michal Marek <mmarek@suse.cz>
CC: linux-kbuild@vger.kernel.org
Signed-off-by: John Stultz <john.stultz@linaro.org>
---
 scripts/kconfig/merge_config.sh |   72 +++++++++++++++++++++++++++++++++++++++
 1 files changed, 72 insertions(+), 0 deletions(-)
 create mode 100755 scripts/kconfig/merge_config.sh

diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
new file mode 100755
index 0000000..fa90e49
--- /dev/null
+++ b/scripts/kconfig/merge_config.sh
@@ -0,0 +1,72 @@
+#!/bin/sh
+#  merge_config.sh - Takes a list of config fragment values, and merges
+#  them one by one. Provides warnings on overridden values, and specified
+#  values that did not make it to the resulting .config file (due to missed
+#  dependencies or config symbol removal).
+#
+#  Portions reused from kconf_check and generate_cfg:
+#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/kconf_check
+#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/generate_cfg
+#
+#  Copyright (c) 2009-2010 Wind River Systems, Inc.
+#  Copyright 2011 Linaro
+#
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License version 2 as
+#  published by the Free Software Foundation.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#  See the GNU General Public License for more details.
+
+MERGE_LIST=$*
+
+
+TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
+
+# Merge files, printing warnings on overrided values
+for MERGE_FILE in $MERGE_LIST ; do
+	echo "Merging $MERGE_FILE"
+	CFG_LIST=`cat $MERGE_FILE | \
+	  sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
+	for CFG in $CFG_LIST ; do
+		grep -q -w $CFG $TMP_FILE
+		if [ $? == 0 ] ; then
+			PREV_VAL=`grep -w $CFG $TMP_FILE`
+			NEW_VAL=`grep -w $CFG $MERGE_FILE`
+			if [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then
+			echo Value of $CFG is redefined by fragment $MERGE_FILE:
+			echo Previous  value: `grep -w $CFG $TMP_FILE`
+			echo New value:       `grep -w $CFG $MERGE_FILE`
+			echo
+			fi
+			sed -i "/$CFG[ =]/d" $TMP_FILE
+		fi
+	done
+	cat $MERGE_FILE >> $TMP_FILE
+done
+
+
+# Copy the merged file to .config and run olddefconfig
+cp $TMP_FILE .config
+make olddefconfig > /dev/null
+
+
+# Check all specified config values took (might have missed-dependency issues)
+cat $TMP_FILE | while read line; do
+	CFG=`echo $line | \
+		sed 's/^\(# \)\{0,1\}\(CONFIG_[a-zA-Z0-9_]*\)[= ].*/\2/'`
+
+	REQUESTED_VAL=`grep -w -e "$CFG" $TMP_FILE`
+	ACTUAL_VAL=`grep -w -e "$CFG" .config`
+	if [ "x$REQUESTED_VAL" != "x$ACTUAL_VAL" ] ; then
+		echo "Value requested for $CFG not in final .config"
+		echo "Requested value:  $REQUESTED_VAL"
+		echo "Actual value:     $ACTUAL_VAL"
+		echo ""
+	fi
+done
+
+# Cleanup
+rm $TMP_FILE
-- 
1.7.3.2.146.gca209

Re: [PATCH 1/2] kconfig: Add make olddefconfig

[Sam Ravnborg]

On Fri, Sep 16, 2011 at 06:49:18PM -0700, John Stultz wrote:
> This adds an "olddefconfig" build target, which updates a .config file
> filling any any absent symbols with the Kconfig default value (note:
> not the architecture defconfig value).

This looks like a shorthand for: make KCONFIG_ALLCONFIG=.config alldefconfig
Maybe we need the more obvious target - but the functionality is at least now new.

	Sam

Re: [PATCH 1/2] kconfig: Add make olddefconfig

[John Stultz]

On Sat, 2011-09-17 at 08:06 +0200, Sam Ravnborg wrote:
> On Fri, Sep 16, 2011 at 06:49:18PM -0700, John Stultz wrote:
> > This adds an "olddefconfig" build target, which updates a .config file
> > filling any any absent symbols with the Kconfig default value (note:
> > not the architecture defconfig value).
> 
> This looks like a shorthand for: make KCONFIG_ALLCONFIG=.config alldefconfig
> Maybe we need the more obvious target - but the functionality is at least now new.

Thanks for pointing that out! 

Unless anyone feels differently, I'll go ahead and drop this patch and
will rework my merge_config.sh script to utilize the above.

thanks
-john