[PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

I brushed up the implementation in this version.

In the previous RFC, CC_HAS_ was described by using 'option shell=',
like this:

config CC_HAS_STACKPROTECTOR
        bool
        option shell="$CC -Werror -fstack-protector -c -x c /dev/null"

After I thought a bit more, the following syntax is more grammatical,
and flexible.

config CC_HAS_STACKPROTECTOR
        bool
        default $(shell $CC -Werror -fstack-protector -c -x c /dev/null)

This version supports cc-option, so it can be written as:

config CC_HAS_STACKPROTECTOR
        bool
        default $(cc-option -fstack-protector)

To support this in a clean way, I introduced a new concept 'function'
like we see in Makefiles.

$(shell ...) is a built-in function.  $(cc-option ...) is implemented
as macro (user-defined function).

I also try cleaning of stack-protector, gcc-plugins since the Makefile
is so dirty.

Current limitations:

Dependency on outside scripts.
  For example, scripts/gcc-x86_64-has-stack-protecter.sh is run from
  Kconfig.  When the shell script is updated, should Kconfig be re-run
  automatically?

Inter-option dependency:
  $(call cc-option,...) in Makefile accumulates added options to
  KBUILD_CFLAGS, but it is difficult to do it in Kconfig.
  If a compiler option check is dependent on another option,
  this is a difficult case.  Let's see how significant it is.

Functions are evaluated statically:
  Functions are only expanded when parsing the Kconfig.  So, it can
  not refelect user configuration.  If this is required, $(shell )
  must be dynamically re-calculated depending on other symbols.
  But, this is difficult, and may cause performance issue.



Masahiro Yamada (22):
  kbuild: remove kbuild cache
  kbuild: remove CONFIG_CROSS_COMPILE support
  kconfig: add xstrdup() helper
  kconfig: set SYMBOL_AUTO to the symbol marked with defconfig_list
  kconfig: move and rename sym_expand_string_value()
  kconfig: reference environments directly and remove 'option env='
    syntax
  kconfig: add function support and implement 'shell' function
  kconfig: add 'macro' keyword to support user-defined function
  kconfig: add 'cc-option' macro
  stack-protector: test compiler capability in Kconfig and drop AUTO
    mode
  kconfig: add 'shell-stdout' function
  kconfig: replace $UNAME_RELEASE with function call
  kconfig: expand environments/functions in (main)menu, comment, prompt
  kconfig: show compiler version text in the top comment
  kconfig: add CC_IS_GCC and GCC_VERSION
  kconfig: add CC_IS_CLANG and CLANG_VERSION
  gcov: remove CONFIG_GCOV_FORMAT_AUTODETECT
  kcov: imply GCC_PLUGINS and GCC_PLUGIN_SANCOV instead of select'ing
    them
  gcc-plugins: always build plugins with C++
  gcc-plugins: move GCC version check for PowerPC to Kconfig
  gcc-plugins: test GCC plugin support in Kconfig
  gcc-plugins: enable GCC_PLUGINS for COMPILE_TEST

Sami Tolvanen (1):
  kbuild: add clang-version.sh

 Documentation/kbuild/kconfig-language.txt |   8 -
 Kconfig                                   |   4 +-
 Makefile                                  | 103 ++----------
 arch/Kconfig                              |  43 +++--
 arch/powerpc/Kconfig                      |   2 +-
 arch/sh/Kconfig                           |   4 +-
 arch/sparc/Kconfig                        |   4 +-
 arch/tile/Kconfig                         |   2 +-
 arch/um/Kconfig.common                    |   4 -
 arch/x86/Kconfig                          |  12 +-
 arch/x86/um/Kconfig                       |   4 +-
 init/Kconfig                              |  44 +++---
 kernel/gcov/Kconfig                       |  18 +--
 kernel/gcov/Makefile                      |   2 -
 lib/Kconfig.debug                         |   7 +-
 scripts/Kbuild.include                    | 101 ++----------
 scripts/Makefile.gcc-plugins              |  95 ++++-------
 scripts/clang-version.sh                  |  31 ++++
 scripts/gcc-plugin.sh                     |  37 +----
 scripts/gcc-plugins/Makefile              |  15 +-
 scripts/gcc-x86_32-has-stack-protector.sh |   7 +-
 scripts/gcc-x86_64-has-stack-protector.sh |   5 -
 scripts/kconfig/confdata.c                |  33 +---
 scripts/kconfig/function.c                | 251 ++++++++++++++++++++++++++++++
 scripts/kconfig/kconf_id.c                |   2 +-
 scripts/kconfig/kxgettext.c               |   2 +-
 scripts/kconfig/lkc.h                     |   6 +-
 scripts/kconfig/lkc_proto.h               |   7 +-
 scripts/kconfig/menu.c                    |   6 +-
 scripts/kconfig/symbol.c                  | 139 +++--------------
 scripts/kconfig/util.c                    | 186 ++++++++++++++++++++--
 scripts/kconfig/zconf.l                   |  40 ++++-
 scripts/kconfig/zconf.y                   |  48 +++---
 33 files changed, 687 insertions(+), 585 deletions(-)
 create mode 100755 scripts/clang-version.sh
 create mode 100644 scripts/kconfig/function.c

-- 
2.7.4

[PATCH 06/23] kconfig: reference environments directly and remove 'option env=' syntax

[Masahiro Yamada]

To get an environment value, Kconfig needs to define a symbol using
"option env=" syntax.  It is tedious to add a config entry for each
environment given that we need more environments such as 'CC', 'AS',
'srctree' etc. to evaluate the compiler capability in Kconfig.

Adding '$' to symbols is weird.  Kconfig can reference symbols directly
like this:

  config FOO
          string
          default BAR

So, I want to use the following syntax to get environment 'BAR' from
the system:

  config FOO
          string
          default $BAR

Looking at the code, the symbols prefixed with 'S' are expanded by:
 - conf_expand_value()
   This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
 - expand_string_value()
   This is used to expand strings in 'source' and 'mainmenu'

All of them are independent of user configuration, i.e. fixed values.
So, this kind of syntax should be moved to simply take the environment.

This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.

sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
should be be given from the environment.

ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
by 'default ARCH_DEFCONFIG'.

An environment can appear anywhere a symbol reference can appear.
(It is expanded by sym_lookup().)  If an expression (which is derived
from symbols) is a string, environments in the string are also expanded.

For example, the following code works.

Example code:

  config TOOLCHAIN_LIST
          string
          default "My tools: CC=$CC, AS=$AS, CPP=$CPP"

Result:

  $ make -s alldefconfig && tail -n 1 .config
  CONFIG_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
---

I tested all 'make *config' for arch architectures.
I confirmed this commit still produced the same result.


 Documentation/kbuild/kconfig-language.txt |   8 --
 Kconfig                                   |   4 -
 Makefile                                  |   3 +-
 arch/sh/Kconfig                           |   4 +-
 arch/sparc/Kconfig                        |   4 +-
 arch/tile/Kconfig                         |   2 +-
 arch/um/Kconfig.common                    |   4 -
 arch/x86/Kconfig                          |   4 +-
 arch/x86/um/Kconfig                       |   4 +-
 init/Kconfig                              |  10 +-
 scripts/kconfig/confdata.c                |  31 +-----
 scripts/kconfig/kconf_id.c                |   1 -
 scripts/kconfig/lkc.h                     |   4 -
 scripts/kconfig/menu.c                    |   3 -
 scripts/kconfig/symbol.c                  |  84 ++++------------
 scripts/kconfig/util.c                    | 156 +++++++++++++++++++++---------
 scripts/kconfig/zconf.l                   |   2 +-
 scripts/kconfig/zconf.y                   |   1 -
 18 files changed, 144 insertions(+), 185 deletions(-)

diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
index f5b9493..0e966e8 100644
--- a/Documentation/kbuild/kconfig-language.txt
+++ b/Documentation/kbuild/kconfig-language.txt
@@ -198,14 +198,6 @@ applicable everywhere (see syntax).
     enables the third modular state for all config symbols.
     At most one symbol may have the "modules" option set.
 
-  - "env"=<value>
-    This imports the environment variable into Kconfig. It behaves like
-    a default, except that the value comes from the environment, this
-    also means that the behaviour when mixing it with normal defaults is
-    undefined at this point. The symbol is currently not exported back
-    to the build environment (if this is desired, it can be done via
-    another symbol).
-
   - "allnoconfig_y"
     This declares the symbol as one that should have the value y when
     using "allnoconfig". Used for symbols that hide other symbols.
diff --git a/Kconfig b/Kconfig
index 8c4c1cb..e6ece5b 100644
--- a/Kconfig
+++ b/Kconfig
@@ -5,8 +5,4 @@
 #
 mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
 
-config SRCARCH
-	string
-	option env="SRCARCH"
-
 source "arch/$SRCARCH/Kconfig"
diff --git a/Makefile b/Makefile
index 94a957e..9a8c689 100644
--- a/Makefile
+++ b/Makefile
@@ -275,7 +275,8 @@ include scripts/Kbuild.include
 # Read KERNELRELEASE from include/config/kernel.release (if it exists)
 KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
 KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
-export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
+UNAME_RELEASE := $(shell uname --release)
+export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
 
 # SUBARCH tells the usermode build what the underlying arch is.  That is set
 # first, and if a usermode build is happening, the "ARCH=um" on the command
diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
index 97fe293..89fc2f6 100644
--- a/arch/sh/Kconfig
+++ b/arch/sh/Kconfig
@@ -57,7 +57,7 @@ config SUPERH
 	  <http://www.linux-sh.org/>.
 
 config SUPERH32
-	def_bool ARCH = "sh"
+	def_bool $ARCH = "sh"
 	select HAVE_KPROBES
 	select HAVE_KRETPROBES
 	select HAVE_IOREMAP_PROT if MMU && !X2TLB
@@ -76,7 +76,7 @@ config SUPERH32
 	select HAVE_CC_STACKPROTECTOR
 
 config SUPERH64
-	def_bool ARCH = "sh64"
+	def_bool $ARCH = "sh64"
 	select HAVE_EXIT_THREAD
 	select KALLSYMS
 
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 6bf594a..fed3d82 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -1,6 +1,6 @@
 config 64BIT
-	bool "64-bit kernel" if ARCH = "sparc"
-	default ARCH = "sparc64"
+	bool "64-bit kernel" if $ARCH = "sparc"
+	default $ARCH = "sparc64"
 	help
 	  SPARC is a family of RISC microprocessors designed and marketed by
 	  Sun Microsystems, incorporated.  They are very widely found in Sun
diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
index ef9d403..fed372e 100644
--- a/arch/tile/Kconfig
+++ b/arch/tile/Kconfig
@@ -119,7 +119,7 @@ config HVC_TILE
 # Building with ARCH=tilegx (or ARCH=tile) implies using the
 # 64-bit TILE-Gx toolchain, so force CONFIG_TILEGX on.
 config TILEGX
-	def_bool ARCH != "tilepro"
+	def_bool $ARCH != "tilepro"
 	select ARCH_SUPPORTS_ATOMIC_RMW
 	select GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
 	select HAVE_ARCH_JUMP_LABEL
diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
index c68add8..07f84c8 100644
--- a/arch/um/Kconfig.common
+++ b/arch/um/Kconfig.common
@@ -54,10 +54,6 @@ config HZ
 	int
 	default 100
 
-config SUBARCH
-	string
-	option env="SUBARCH"
-
 config NR_CPUS
 	int
 	range 1 1
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index a528c14..54d943a 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1,8 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
 # Select 32 or 64 bit
 config 64BIT
-	bool "64-bit kernel" if ARCH = "x86"
-	default ARCH != "i386"
+	bool "64-bit kernel" if $ARCH = "x86"
+	default $ARCH != "i386"
 	---help---
 	  Say yes to build a 64-bit kernel - formerly known as x86_64
 	  Say no to build a 32-bit kernel - formerly known as i386
diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
index 13ed827..d355413 100644
--- a/arch/x86/um/Kconfig
+++ b/arch/x86/um/Kconfig
@@ -16,8 +16,8 @@ config UML_X86
 	select GENERIC_FIND_FIRST_BIT
 
 config 64BIT
-	bool "64-bit kernel" if SUBARCH = "x86"
-	default SUBARCH != "i386"
+	bool "64-bit kernel" if $SUBARCH = "x86"
+	default $SUBARCH != "i386"
 
 config X86_32
 	def_bool !64BIT
diff --git a/init/Kconfig b/init/Kconfig
index df18492..b4814e6 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1,11 +1,3 @@
-config ARCH
-	string
-	option env="ARCH"
-
-config KERNELVERSION
-	string
-	option env="KERNELVERSION"
-
 config DEFCONFIG_LIST
 	string
 	depends on !UML
@@ -13,7 +5,7 @@ config DEFCONFIG_LIST
 	default "/lib/modules/$UNAME_RELEASE/.config"
 	default "/etc/kernel-config"
 	default "/boot/config-$UNAME_RELEASE"
-	default "$ARCH_DEFCONFIG"
+	default ARCH_DEFCONFIG
 	default "arch/$ARCH/defconfig"
 
 config CONSTRUCTORS
diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
index df26c7b..98c2014 100644
--- a/scripts/kconfig/confdata.c
+++ b/scripts/kconfig/confdata.c
@@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
 	return name ? name : "include/config/auto.conf";
 }
 
-static char *conf_expand_value(const char *in)
-{
-	struct symbol *sym;
-	const char *src;
-	static char res_value[SYMBOL_MAXLENGTH];
-	char *dst, name[SYMBOL_MAXLENGTH];
-
-	res_value[0] = 0;
-	dst = name;
-	while ((src = strchr(in, '$'))) {
-		strncat(res_value, in, src - in);
-		src++;
-		dst = name;
-		while (isalnum(*src) || *src == '_')
-			*dst++ = *src++;
-		*dst = 0;
-		sym = sym_lookup(name, 0);
-		sym_calc_value(sym);
-		strcat(res_value, sym_get_string_value(sym));
-		in = src;
-	}
-	strcat(res_value, in);
-
-	return res_value;
-}
-
 char *conf_get_default_confname(void)
 {
 	struct stat buf;
 	static char fullname[PATH_MAX+1];
 	char *env, *name;
 
-	name = conf_expand_value(conf_defname);
+	name = expand_string_value(conf_defname);
 	env = getenv(SRCTREE);
 	if (env) {
 		sprintf(fullname, "%s/%s", env, name);
@@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
 			if (expr_calc_value(prop->visible.expr) == no ||
 			    prop->expr->type != E_SYMBOL)
 				continue;
-			name = conf_expand_value(prop->expr->left.sym->name);
+			sym_calc_value(prop->expr->left.sym);
+			name = sym_get_string_value(prop->expr->left.sym);
 			in = zconf_fopen(name);
 			if (in) {
 				conf_message(_("using defaults found in %s"),
diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
index 3ea9c5f..b3e0ea0 100644
--- a/scripts/kconfig/kconf_id.c
+++ b/scripts/kconfig/kconf_id.c
@@ -32,7 +32,6 @@ static struct kconf_id kconf_id_array[] = {
 	{ "on",			T_ON,			TF_PARAM },
 	{ "modules",		T_OPT_MODULES,		TF_OPTION },
 	{ "defconfig_list",	T_OPT_DEFCONFIG_LIST,	TF_OPTION },
-	{ "env",		T_OPT_ENV,		TF_OPTION },
 	{ "allnoconfig_y",	T_OPT_ALLNOCONFIG_Y,	TF_OPTION },
 };
 
diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
index fb2503a..0ff3256 100644
--- a/scripts/kconfig/lkc.h
+++ b/scripts/kconfig/lkc.h
@@ -58,7 +58,6 @@ enum conf_def_mode {
 
 #define T_OPT_MODULES		1
 #define T_OPT_DEFCONFIG_LIST	2
-#define T_OPT_ENV		3
 #define T_OPT_ALLNOCONFIG_Y	4
 
 struct kconf_id {
@@ -134,9 +133,6 @@ void str_printf(struct gstr *gs, const char *fmt, ...);
 const char *str_get(struct gstr *gs);
 
 /* symbol.c */
-extern struct expr *sym_env_list;
-
-void sym_init(void);
 void sym_clear_all_valid(void);
 struct symbol *sym_choice_default(struct symbol *sym);
 const char *sym_get_string_default(struct symbol *sym);
diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
index 36cd3e1..a9d0ccc 100644
--- a/scripts/kconfig/menu.c
+++ b/scripts/kconfig/menu.c
@@ -214,9 +214,6 @@ void menu_add_option(int token, char *arg)
 			zconf_error("trying to redefine defconfig symbol");
 		sym_defconfig_list->flags |= SYMBOL_AUTO;
 		break;
-	case T_OPT_ENV:
-		prop_add_env(arg);
-		break;
 	case T_OPT_ALLNOCONFIG_Y:
 		current_entry->sym->flags |= SYMBOL_ALLNOCONFIG_Y;
 		break;
diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
index e4ccf56..96ea8a9 100644
--- a/scripts/kconfig/symbol.c
+++ b/scripts/kconfig/symbol.c
@@ -33,33 +33,6 @@ struct symbol *sym_defconfig_list;
 struct symbol *modules_sym;
 tristate modules_val;
 
-struct expr *sym_env_list;
-
-static void sym_add_default(struct symbol *sym, const char *def)
-{
-	struct property *prop = prop_alloc(P_DEFAULT, sym);
-
-	prop->expr = expr_alloc_symbol(sym_lookup(def, SYMBOL_CONST));
-}
-
-void sym_init(void)
-{
-	struct symbol *sym;
-	struct utsname uts;
-	static bool inited = false;
-
-	if (inited)
-		return;
-	inited = true;
-
-	uname(&uts);
-
-	sym = sym_lookup("UNAME_RELEASE", 0);
-	sym->type = S_STRING;
-	sym->flags |= SYMBOL_AUTO;
-	sym_add_default(sym, uts.release);
-}
-
 enum symbol_type sym_get_type(struct symbol *sym)
 {
 	enum symbol_type type = sym->type;
@@ -828,28 +801,38 @@ static unsigned strhash(const char *s)
 
 struct symbol *sym_lookup(const char *name, int flags)
 {
-	struct symbol *symbol;
+	struct symbol *symbol = NULL;
 	char *new_name;
 	int hash;
 
 	if (name) {
-		if (name[0] && !name[1]) {
-			switch (name[0]) {
-			case 'y': return &symbol_yes;
-			case 'm': return &symbol_mod;
-			case 'n': return &symbol_no;
+		new_name = expand_string_value(name);
+		if (new_name[0] && !new_name[1]) {
+			switch (new_name[0]) {
+			case 'y':
+				symbol = &symbol_yes;
+				break;
+			case 'm':
+				symbol = &symbol_mod;
+				break;
+			case 'n':
+				symbol = &symbol_no;
+				break;
+			}
+			if (symbol) {
+				free(new_name);
+				return symbol;
 			}
 		}
-		hash = strhash(name) % SYMBOL_HASHSIZE;
+		hash = strhash(new_name) % SYMBOL_HASHSIZE;
 
 		for (symbol = symbol_hash[hash]; symbol; symbol = symbol->next) {
 			if (symbol->name &&
-			    !strcmp(symbol->name, name) &&
+			    !strcmp(symbol->name, new_name) &&
 			    (flags ? symbol->flags & flags
 				   : !(symbol->flags & (SYMBOL_CONST|SYMBOL_CHOICE))))
 				return symbol;
 		}
-		new_name = xstrdup(name);
 	} else {
 		new_name = NULL;
 		hash = 0;
@@ -1336,32 +1319,3 @@ const char *prop_get_type_name(enum prop_type type)
 	}
 	return "unknown";
 }
-
-static void prop_add_env(const char *env)
-{
-	struct symbol *sym, *sym2;
-	struct property *prop;
-	char *p;
-
-	sym = current_entry->sym;
-	sym->flags |= SYMBOL_AUTO;
-	for_all_properties(sym, prop, P_ENV) {
-		sym2 = prop_get_symbol(prop);
-		if (strcmp(sym2->name, env))
-			menu_warn(current_entry, "redefining environment symbol from %s",
-				  sym2->name);
-		return;
-	}
-
-	prop = prop_alloc(P_ENV, sym);
-	prop->expr = expr_alloc_symbol(sym_lookup(env, SYMBOL_CONST));
-
-	sym_env_list = expr_alloc_one(E_LIST, sym_env_list);
-	sym_env_list->right.sym = sym;
-
-	p = getenv(env);
-	if (p)
-		sym_add_default(sym, p);
-	else
-		menu_warn(current_entry, "environment variable %s undefined", env);
-}
diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
index 22201a4..dddf85b 100644
--- a/scripts/kconfig/util.c
+++ b/scripts/kconfig/util.c
@@ -8,16 +8,98 @@
 #include <stdarg.h>
 #include <stdlib.h>
 #include <string.h>
+
+#include "list.h"
 #include "lkc.h"
 
+static LIST_HEAD(env_list);
+
+struct env {
+	char *name;
+	char *value;
+	struct list_head node;
+};
+
+static struct env *env_list_lookup(const char *name)
+{
+	struct env *e;
+
+	list_for_each_entry(e, &env_list, node) {
+		if (!strcmp(name, e->name))
+			return e;
+	}
+
+	return NULL;
+}
+
+static void env_list_add(const char *name, const char *value)
+{
+	struct env *e;
+
+	e = xmalloc(sizeof(*e));
+	e->name = xstrdup(name);
+	e->value = xstrdup(value);
+
+	list_add_tail(&e->node, &env_list);
+}
+
+static void env_list_del(struct env *e)
+{
+	list_del(&e->node);
+	free(e->name);
+	free(e->value);
+	free(e);
+}
+
+/* the returned pointer must be freed when done */
+static char *env_expand(const char *name)
+{
+	struct env *e;
+	const char *value;
+
+	e = env_list_lookup(name);
+	if (e)
+		return xstrdup(e->value);
+
+	value = getenv(name);
+	if (!value) {
+		fprintf(stderr, "environment variable \"%s\" undefined\n", name);
+		value = "";
+	}
+
+	/*
+	 * we need to remember all referenced environments.
+	 * They will be written out to include/config/auto.conf.cmd
+	 */
+	env_list_add(name, value);
+
+	return xstrdup(value);
+}
+
+/* works like env_expand, but 'name' does not need to be null-terminated */
+static char *env_expand_n(const char *name, size_t n)
+{
+	char *tmp, *res;
+
+	tmp = xmalloc(n + 1);
+	memcpy(tmp, name, n);
+	*(tmp + n) = '\0';
+
+	res = env_expand(tmp);
+
+	free(tmp);
+
+	return res;
+}
+
 /*
- * Expand symbol's names embedded in the string given in argument. Symbols'
- * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
+ * Expand environments embedded in the string given in argument. Environments
+ * to be expanded shall be prefixed by a '$'. Unknown environment expands to
  * the empty string.
  */
 char *expand_string_value(const char *in)
 {
-	const char *src;
+	const char *p, *q;
 	char *res;
 	size_t reslen;
 
@@ -25,39 +107,28 @@ char *expand_string_value(const char *in)
 	 * Note: 'in' might come from a token that's about to be
 	 * freed, so make sure to always allocate a new string
 	 */
-	reslen = strlen(in) + 1;
-	res = xmalloc(reslen);
-	res[0] = '\0';
-
-	while ((src = strchr(in, '$'))) {
-		char *p, name[SYMBOL_MAXLENGTH];
-		const char *symval = "";
-		struct symbol *sym;
-		size_t newlen;
-
-		strncat(res, in, src - in);
-		src++;
-
-		p = name;
-		while (isalnum(*src) || *src == '_')
-			*p++ = *src++;
-		*p = '\0';
-
-		sym = sym_find(name);
-		if (sym != NULL) {
-			sym_calc_value(sym);
-			symval = sym_get_string_value(sym);
-		}
+	res = xmalloc(1);
+	*res = '\0';
 
-		newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
-		if (newlen > reslen) {
-			reslen = newlen;
-			res = xrealloc(res, reslen);
-		}
+	while ((p = strchr(in, '$'))) {
+		char *new;
+
+		q = p + 1;
+		while (isalnum(*q) || *q == '_')
+			q++;
 
-		strcat(res, symval);
-		in = src;
+		new = env_expand_n(p + 1, q - p - 1);
+
+		reslen = strlen(res) + (p - in) + strlen(new) + 1;
+		res = xrealloc(res, reslen);
+		strncat(res, in, p - in);
+		strcat(res, new);
+		free(new);
+		in = q;
 	}
+
+	reslen = strlen(res) + strlen(in) + 1;
+	res = xrealloc(res, reslen);
 	strcat(res, in);
 
 	return res;
@@ -87,8 +158,7 @@ struct file *file_lookup(const char *name)
 /* write a dependency file as used by kbuild to track dependencies */
 int file_write_dep(const char *name)
 {
-	struct symbol *sym, *env_sym;
-	struct expr *e;
+	struct env *env, *tmp;
 	struct file *file;
 	FILE *out;
 
@@ -107,20 +177,12 @@ int file_write_dep(const char *name)
 	fprintf(out, "\n%s: \\\n"
 		     "\t$(deps_config)\n\n", conf_get_autoconfig_name());
 
-	expr_list_for_each_sym(sym_env_list, e, sym) {
-		struct property *prop;
-		const char *value;
-
-		prop = sym_get_env_prop(sym);
-		env_sym = prop_get_symbol(prop);
-		if (!env_sym)
-			continue;
-		value = getenv(env_sym->name);
-		if (!value)
-			value = "";
-		fprintf(out, "ifneq \"$(%s)\" \"%s\"\n", env_sym->name, value);
+	list_for_each_entry_safe(env, tmp, &env_list, node) {
+		fprintf(out, "ifneq \"$(%s)\" \"%s\"\n",
+			env->name, getenv(env->name) ?: "");
 		fprintf(out, "%s: FORCE\n", conf_get_autoconfig_name());
 		fprintf(out, "endif\n");
+		env_list_del(env);
 	}
 
 	fprintf(out, "\n$(deps_config): ;\n");
diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
index 02de6fe..0d89ea6 100644
--- a/scripts/kconfig/zconf.l
+++ b/scripts/kconfig/zconf.l
@@ -75,7 +75,7 @@ static void warn_ignored_character(char chr)
 }
 %}
 
-n	[A-Za-z0-9_-]
+n	[$A-Za-z0-9_-]
 
 %%
 	int str = 0;
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index 262c464..784083d 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -534,7 +534,6 @@ void conf_parse(const char *name)
 
 	zconf_initscan(name);
 
-	sym_init();
 	_menu_init();
 
 	if (getenv("ZCONF_DEBUG"))
-- 
2.7.4

Re: [PATCH 06/23] kconfig: reference environments directly and remove 'option env=' syntax

[Ulf Magnusson]

On Sat, Feb 17, 2018 at 03:38:34AM +0900, Masahiro Yamada wrote:
> To get an environment value, Kconfig needs to define a symbol using
> "option env=" syntax.  It is tedious to add a config entry for each
> environment given that we need more environments such as 'CC', 'AS',
> 'srctree' etc. to evaluate the compiler capability in Kconfig.
> 
> Adding '$' to symbols is weird.  Kconfig can reference symbols directly
> like this:
> 
>   config FOO
>           string
>           default BAR
> 
> So, I want to use the following syntax to get environment 'BAR' from
> the system:
> 
>   config FOO
>           string
>           default $BAR

I like this idea. These bounce symbols always seemed kinda pointless to
me too (and I've seen some other Kconfig dialects that got rid of them).

Getting rid of those symbols also gets rid of the only case where
Kconfig would previously evaluate Kconfig symbols during parsing (for
'source "$ENV"'), so it makes the parsing/evaluation separation a bit
cleaner.


I wonder if we could just make the quotes mandatory though, and treat
$BAR simply as string interpolation, similar to the idea in
https://lkml.org/lkml/2018/2/17/175.

That would make the behavior clearer I think, and you wouldn't have to
do special parsing for symbols starting with $ (but see the note at the
end).

> 
> Looking at the code, the symbols prefixed with 'S' are expanded by:
>  - conf_expand_value()
>    This is used to expand 'arch/$ARCH/defconfig' and 'defconfig_list'
>  - expand_string_value()
>    This is used to expand strings in 'source' and 'mainmenu'
> 
> All of them are independent of user configuration, i.e. fixed values.
> So, this kind of syntax should be moved to simply take the environment.
> 
> This change makes the code much cleaner.  The bounce symbols 'SRCARCH',
> 'ARCH', 'SUBARCH', 'KERNELVERSION' are gone.
> 
> sym_init() hard-coding 'UNAME_RELEASE' is also gone.  'UNAME_RELEASE'
> should be be given from the environment.
> 
> ARCH_DEFCONFIG is a normal symbol, so it should be simply referenced
> by 'default ARCH_DEFCONFIG'.
> 
> An environment can appear anywhere a symbol reference can appear.
> (It is expanded by sym_lookup().)  If an expression (which is derived
> from symbols) is a string, environments in the string are also expanded.
> 
> For example, the following code works.
> 
> Example code:
> 
>   config TOOLCHAIN_LIST
>           string
>           default "My tools: CC=$CC, AS=$AS, CPP=$CPP"
> 
> Result:
> 
>   $ make -s alldefconfig && tail -n 1 .config
>   CONFIG_TOOLCHAIN_LIST="My tools: CC=gcc, AS=as, CPP=gcc -E"
> 
> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
> ---
> 
> I tested all 'make *config' for arch architectures.
> I confirmed this commit still produced the same result.
> 
> 
>  Documentation/kbuild/kconfig-language.txt |   8 --
>  Kconfig                                   |   4 -
>  Makefile                                  |   3 +-
>  arch/sh/Kconfig                           |   4 +-
>  arch/sparc/Kconfig                        |   4 +-
>  arch/tile/Kconfig                         |   2 +-
>  arch/um/Kconfig.common                    |   4 -
>  arch/x86/Kconfig                          |   4 +-
>  arch/x86/um/Kconfig                       |   4 +-
>  init/Kconfig                              |  10 +-
>  scripts/kconfig/confdata.c                |  31 +-----
>  scripts/kconfig/kconf_id.c                |   1 -
>  scripts/kconfig/lkc.h                     |   4 -
>  scripts/kconfig/menu.c                    |   3 -
>  scripts/kconfig/symbol.c                  |  84 ++++------------
>  scripts/kconfig/util.c                    | 156 +++++++++++++++++++++---------
>  scripts/kconfig/zconf.l                   |   2 +-
>  scripts/kconfig/zconf.y                   |   1 -
>  18 files changed, 144 insertions(+), 185 deletions(-)
> 
> diff --git a/Documentation/kbuild/kconfig-language.txt b/Documentation/kbuild/kconfig-language.txt
> index f5b9493..0e966e8 100644
> --- a/Documentation/kbuild/kconfig-language.txt
> +++ b/Documentation/kbuild/kconfig-language.txt
> @@ -198,14 +198,6 @@ applicable everywhere (see syntax).
>      enables the third modular state for all config symbols.
>      At most one symbol may have the "modules" option set.
>  
> -  - "env"=<value>
> -    This imports the environment variable into Kconfig. It behaves like
> -    a default, except that the value comes from the environment, this
> -    also means that the behaviour when mixing it with normal defaults is
> -    undefined at this point. The symbol is currently not exported back
> -    to the build environment (if this is desired, it can be done via
> -    another symbol).
> -
>    - "allnoconfig_y"
>      This declares the symbol as one that should have the value y when
>      using "allnoconfig". Used for symbols that hide other symbols.
> diff --git a/Kconfig b/Kconfig
> index 8c4c1cb..e6ece5b 100644
> --- a/Kconfig
> +++ b/Kconfig
> @@ -5,8 +5,4 @@
>  #
>  mainmenu "Linux/$ARCH $KERNELVERSION Kernel Configuration"
>  
> -config SRCARCH
> -	string
> -	option env="SRCARCH"
> -
>  source "arch/$SRCARCH/Kconfig"
> diff --git a/Makefile b/Makefile
> index 94a957e..9a8c689 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -275,7 +275,8 @@ include scripts/Kbuild.include
>  # Read KERNELRELEASE from include/config/kernel.release (if it exists)
>  KERNELRELEASE = $(shell cat include/config/kernel.release 2> /dev/null)
>  KERNELVERSION = $(VERSION)$(if $(PATCHLEVEL),.$(PATCHLEVEL)$(if $(SUBLEVEL),.$(SUBLEVEL)))$(EXTRAVERSION)
> -export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
> +UNAME_RELEASE := $(shell uname --release)
> +export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION UNAME_RELEASE
>  
>  # SUBARCH tells the usermode build what the underlying arch is.  That is set
>  # first, and if a usermode build is happening, the "ARCH=um" on the command
> diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig
> index 97fe293..89fc2f6 100644
> --- a/arch/sh/Kconfig
> +++ b/arch/sh/Kconfig
> @@ -57,7 +57,7 @@ config SUPERH
>  	  <http://www.linux-sh.org/>.
>  
>  config SUPERH32
> -	def_bool ARCH = "sh"
> +	def_bool $ARCH = "sh"

With the idea above, this would just be

	def_bool "$ARCH" = "sh"

instead.

>  	select HAVE_KPROBES
>  	select HAVE_KRETPROBES
>  	select HAVE_IOREMAP_PROT if MMU && !X2TLB
> @@ -76,7 +76,7 @@ config SUPERH32
>  	select HAVE_CC_STACKPROTECTOR
>  
>  config SUPERH64
> -	def_bool ARCH = "sh64"
> +	def_bool $ARCH = "sh64"
>  	select HAVE_EXIT_THREAD
>  	select KALLSYMS
>  
> diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
> index 6bf594a..fed3d82 100644
> --- a/arch/sparc/Kconfig
> +++ b/arch/sparc/Kconfig
> @@ -1,6 +1,6 @@
>  config 64BIT
> -	bool "64-bit kernel" if ARCH = "sparc"
> -	default ARCH = "sparc64"
> +	bool "64-bit kernel" if $ARCH = "sparc"
> +	default $ARCH = "sparc64"
>  	help
>  	  SPARC is a family of RISC microprocessors designed and marketed by
>  	  Sun Microsystems, incorporated.  They are very widely found in Sun
> diff --git a/arch/tile/Kconfig b/arch/tile/Kconfig
> index ef9d403..fed372e 100644
> --- a/arch/tile/Kconfig
> +++ b/arch/tile/Kconfig
> @@ -119,7 +119,7 @@ config HVC_TILE
>  # Building with ARCH=tilegx (or ARCH=tile) implies using the
>  # 64-bit TILE-Gx toolchain, so force CONFIG_TILEGX on.
>  config TILEGX
> -	def_bool ARCH != "tilepro"
> +	def_bool $ARCH != "tilepro"
>  	select ARCH_SUPPORTS_ATOMIC_RMW
>  	select GENERIC_IRQ_LEGACY_ALLOC_HWIRQ
>  	select HAVE_ARCH_JUMP_LABEL
> diff --git a/arch/um/Kconfig.common b/arch/um/Kconfig.common
> index c68add8..07f84c8 100644
> --- a/arch/um/Kconfig.common
> +++ b/arch/um/Kconfig.common
> @@ -54,10 +54,6 @@ config HZ
>  	int
>  	default 100
>  
> -config SUBARCH
> -	string
> -	option env="SUBARCH"
> -
>  config NR_CPUS
>  	int
>  	range 1 1
> diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
> index a528c14..54d943a 100644
> --- a/arch/x86/Kconfig
> +++ b/arch/x86/Kconfig
> @@ -1,8 +1,8 @@
>  # SPDX-License-Identifier: GPL-2.0
>  # Select 32 or 64 bit
>  config 64BIT
> -	bool "64-bit kernel" if ARCH = "x86"
> -	default ARCH != "i386"
> +	bool "64-bit kernel" if $ARCH = "x86"
> +	default $ARCH != "i386"
>  	---help---
>  	  Say yes to build a 64-bit kernel - formerly known as x86_64
>  	  Say no to build a 32-bit kernel - formerly known as i386
> diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
> index 13ed827..d355413 100644
> --- a/arch/x86/um/Kconfig
> +++ b/arch/x86/um/Kconfig
> @@ -16,8 +16,8 @@ config UML_X86
>  	select GENERIC_FIND_FIRST_BIT
>  
>  config 64BIT
> -	bool "64-bit kernel" if SUBARCH = "x86"
> -	default SUBARCH != "i386"
> +	bool "64-bit kernel" if $SUBARCH = "x86"
> +	default $SUBARCH != "i386"
>  
>  config X86_32
>  	def_bool !64BIT
> diff --git a/init/Kconfig b/init/Kconfig
> index df18492..b4814e6 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1,11 +1,3 @@
> -config ARCH
> -	string
> -	option env="ARCH"
> -
> -config KERNELVERSION
> -	string
> -	option env="KERNELVERSION"
> -
>  config DEFCONFIG_LIST
>  	string
>  	depends on !UML
> @@ -13,7 +5,7 @@ config DEFCONFIG_LIST
>  	default "/lib/modules/$UNAME_RELEASE/.config"
>  	default "/etc/kernel-config"
>  	default "/boot/config-$UNAME_RELEASE"
> -	default "$ARCH_DEFCONFIG"
> +	default ARCH_DEFCONFIG
>  	default "arch/$ARCH/defconfig"
>  
>  config CONSTRUCTORS
> diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
> index df26c7b..98c2014 100644
> --- a/scripts/kconfig/confdata.c
> +++ b/scripts/kconfig/confdata.c
> @@ -81,39 +81,13 @@ const char *conf_get_autoconfig_name(void)
>  	return name ? name : "include/config/auto.conf";
>  }
>  
> -static char *conf_expand_value(const char *in)
> -{
> -	struct symbol *sym;
> -	const char *src;
> -	static char res_value[SYMBOL_MAXLENGTH];
> -	char *dst, name[SYMBOL_MAXLENGTH];
> -
> -	res_value[0] = 0;
> -	dst = name;
> -	while ((src = strchr(in, '$'))) {
> -		strncat(res_value, in, src - in);
> -		src++;
> -		dst = name;
> -		while (isalnum(*src) || *src == '_')
> -			*dst++ = *src++;
> -		*dst = 0;
> -		sym = sym_lookup(name, 0);
> -		sym_calc_value(sym);
> -		strcat(res_value, sym_get_string_value(sym));
> -		in = src;
> -	}
> -	strcat(res_value, in);
> -
> -	return res_value;
> -}
> -
>  char *conf_get_default_confname(void)
>  {
>  	struct stat buf;
>  	static char fullname[PATH_MAX+1];
>  	char *env, *name;
>  
> -	name = conf_expand_value(conf_defname);
> +	name = expand_string_value(conf_defname);
>  	env = getenv(SRCTREE);
>  	if (env) {
>  		sprintf(fullname, "%s/%s", env, name);
> @@ -274,7 +248,8 @@ int conf_read_simple(const char *name, int def)
>  			if (expr_calc_value(prop->visible.expr) == no ||
>  			    prop->expr->type != E_SYMBOL)
>  				continue;
> -			name = conf_expand_value(prop->expr->left.sym->name);
> +			sym_calc_value(prop->expr->left.sym);
> +			name = sym_get_string_value(prop->expr->left.sym);
>  			in = zconf_fopen(name);
>  			if (in) {
>  				conf_message(_("using defaults found in %s"),
> diff --git a/scripts/kconfig/kconf_id.c b/scripts/kconfig/kconf_id.c
> index 3ea9c5f..b3e0ea0 100644
> --- a/scripts/kconfig/kconf_id.c
> +++ b/scripts/kconfig/kconf_id.c
> @@ -32,7 +32,6 @@ static struct kconf_id kconf_id_array[] = {
>  	{ "on",			T_ON,			TF_PARAM },
>  	{ "modules",		T_OPT_MODULES,		TF_OPTION },
>  	{ "defconfig_list",	T_OPT_DEFCONFIG_LIST,	TF_OPTION },
> -	{ "env",		T_OPT_ENV,		TF_OPTION },
>  	{ "allnoconfig_y",	T_OPT_ALLNOCONFIG_Y,	TF_OPTION },
>  };
>  
> diff --git a/scripts/kconfig/lkc.h b/scripts/kconfig/lkc.h
> index fb2503a..0ff3256 100644
> --- a/scripts/kconfig/lkc.h
> +++ b/scripts/kconfig/lkc.h
> @@ -58,7 +58,6 @@ enum conf_def_mode {
>  
>  #define T_OPT_MODULES		1
>  #define T_OPT_DEFCONFIG_LIST	2
> -#define T_OPT_ENV		3
>  #define T_OPT_ALLNOCONFIG_Y	4

Decrease T_OPT_ALLNOCONFIG_Y to 3 as well? Looks like a bitmask to me
otherwise.

>  
>  struct kconf_id {
> @@ -134,9 +133,6 @@ void str_printf(struct gstr *gs, const char *fmt, ...);
>  const char *str_get(struct gstr *gs);
>  
>  /* symbol.c */
> -extern struct expr *sym_env_list;
> -
> -void sym_init(void);
>  void sym_clear_all_valid(void);
>  struct symbol *sym_choice_default(struct symbol *sym);
>  const char *sym_get_string_default(struct symbol *sym);
> diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
> index 36cd3e1..a9d0ccc 100644
> --- a/scripts/kconfig/menu.c
> +++ b/scripts/kconfig/menu.c
> @@ -214,9 +214,6 @@ void menu_add_option(int token, char *arg)
>  			zconf_error("trying to redefine defconfig symbol");
>  		sym_defconfig_list->flags |= SYMBOL_AUTO;
>  		break;
> -	case T_OPT_ENV:
> -		prop_add_env(arg);
> -		break;
>  	case T_OPT_ALLNOCONFIG_Y:
>  		current_entry->sym->flags |= SYMBOL_ALLNOCONFIG_Y;
>  		break;
> diff --git a/scripts/kconfig/symbol.c b/scripts/kconfig/symbol.c
> index e4ccf56..96ea8a9 100644
> --- a/scripts/kconfig/symbol.c
> +++ b/scripts/kconfig/symbol.c
> @@ -33,33 +33,6 @@ struct symbol *sym_defconfig_list;
>  struct symbol *modules_sym;
>  tristate modules_val;
>  
> -struct expr *sym_env_list;
> -
> -static void sym_add_default(struct symbol *sym, const char *def)
> -{
> -	struct property *prop = prop_alloc(P_DEFAULT, sym);
> -
> -	prop->expr = expr_alloc_symbol(sym_lookup(def, SYMBOL_CONST));
> -}
> -
> -void sym_init(void)
> -{
> -	struct symbol *sym;
> -	struct utsname uts;
> -	static bool inited = false;
> -
> -	if (inited)
> -		return;
> -	inited = true;
> -
> -	uname(&uts);
> -
> -	sym = sym_lookup("UNAME_RELEASE", 0);
> -	sym->type = S_STRING;
> -	sym->flags |= SYMBOL_AUTO;
> -	sym_add_default(sym, uts.release);
> -}
> -
>  enum symbol_type sym_get_type(struct symbol *sym)
>  {
>  	enum symbol_type type = sym->type;
> @@ -828,28 +801,38 @@ static unsigned strhash(const char *s)
>  
>  struct symbol *sym_lookup(const char *name, int flags)
>  {
> -	struct symbol *symbol;
> +	struct symbol *symbol = NULL;
>  	char *new_name;
>  	int hash;
>  
>  	if (name) {
> -		if (name[0] && !name[1]) {
> -			switch (name[0]) {
> -			case 'y': return &symbol_yes;
> -			case 'm': return &symbol_mod;
> -			case 'n': return &symbol_no;
> +		new_name = expand_string_value(name);
> +		if (new_name[0] && !new_name[1]) {
> +			switch (new_name[0]) {
> +			case 'y':
> +				symbol = &symbol_yes;
> +				break;
> +			case 'm':
> +				symbol = &symbol_mod;
> +				break;
> +			case 'n':
> +				symbol = &symbol_no;
> +				break;
> +			}
> +			if (symbol) {
> +				free(new_name);
> +				return symbol;
>  			}
>  		}
> -		hash = strhash(name) % SYMBOL_HASHSIZE;
> +		hash = strhash(new_name) % SYMBOL_HASHSIZE;
>  
>  		for (symbol = symbol_hash[hash]; symbol; symbol = symbol->next) {
>  			if (symbol->name &&
> -			    !strcmp(symbol->name, name) &&
> +			    !strcmp(symbol->name, new_name) &&
>  			    (flags ? symbol->flags & flags
>  				   : !(symbol->flags & (SYMBOL_CONST|SYMBOL_CHOICE))))
>  				return symbol;
>  		}
> -		new_name = xstrdup(name);
>  	} else {
>  		new_name = NULL;
>  		hash = 0;
> @@ -1336,32 +1319,3 @@ const char *prop_get_type_name(enum prop_type type)
>  	}
>  	return "unknown";
>  }
> -
> -static void prop_add_env(const char *env)
> -{
> -	struct symbol *sym, *sym2;
> -	struct property *prop;
> -	char *p;
> -
> -	sym = current_entry->sym;
> -	sym->flags |= SYMBOL_AUTO;
> -	for_all_properties(sym, prop, P_ENV) {
> -		sym2 = prop_get_symbol(prop);
> -		if (strcmp(sym2->name, env))
> -			menu_warn(current_entry, "redefining environment symbol from %s",
> -				  sym2->name);
> -		return;
> -	}
> -
> -	prop = prop_alloc(P_ENV, sym);
> -	prop->expr = expr_alloc_symbol(sym_lookup(env, SYMBOL_CONST));
> -
> -	sym_env_list = expr_alloc_one(E_LIST, sym_env_list);
> -	sym_env_list->right.sym = sym;
> -
> -	p = getenv(env);
> -	if (p)
> -		sym_add_default(sym, p);
> -	else
> -		menu_warn(current_entry, "environment variable %s undefined", env);
> -}
> diff --git a/scripts/kconfig/util.c b/scripts/kconfig/util.c
> index 22201a4..dddf85b 100644
> --- a/scripts/kconfig/util.c
> +++ b/scripts/kconfig/util.c
> @@ -8,16 +8,98 @@
>  #include <stdarg.h>
>  #include <stdlib.h>
>  #include <string.h>
> +
> +#include "list.h"
>  #include "lkc.h"
>  
> +static LIST_HEAD(env_list);
> +
> +struct env {
> +	char *name;
> +	char *value;
> +	struct list_head node;
> +};
> +
> +static struct env *env_list_lookup(const char *name)
> +{
> +	struct env *e;
> +
> +	list_for_each_entry(e, &env_list, node) {
> +		if (!strcmp(name, e->name))
> +			return e;
> +	}
> +
> +	return NULL;
> +}
> +
> +static void env_list_add(const char *name, const char *value)
> +{
> +	struct env *e;
> +
> +	e = xmalloc(sizeof(*e));
> +	e->name = xstrdup(name);
> +	e->value = xstrdup(value);
> +
> +	list_add_tail(&e->node, &env_list);
> +}
> +
> +static void env_list_del(struct env *e)
> +{
> +	list_del(&e->node);
> +	free(e->name);
> +	free(e->value);
> +	free(e);
> +}
> +
> +/* the returned pointer must be freed when done */
> +static char *env_expand(const char *name)
> +{
> +	struct env *e;
> +	const char *value;
> +
> +	e = env_list_lookup(name);
> +	if (e)
> +		return xstrdup(e->value);
> +
> +	value = getenv(name);
> +	if (!value) {
> +		fprintf(stderr, "environment variable \"%s\" undefined\n", name);
> +		value = "";
> +	}
> +
> +	/*
> +	 * we need to remember all referenced environments.
> +	 * They will be written out to include/config/auto.conf.cmd
> +	 */
> +	env_list_add(name, value);
> +
> +	return xstrdup(value);
> +}
> +
> +/* works like env_expand, but 'name' does not need to be null-terminated */
> +static char *env_expand_n(const char *name, size_t n)
> +{
> +	char *tmp, *res;
> +
> +	tmp = xmalloc(n + 1);
> +	memcpy(tmp, name, n);
> +	*(tmp + n) = '\0';
> +
> +	res = env_expand(tmp);
> +
> +	free(tmp);
> +
> +	return res;
> +}
> +
>  /*
> - * Expand symbol's names embedded in the string given in argument. Symbols'
> - * name to be expanded shall be prefixed by a '$'. Unknown symbol expands to
> + * Expand environments embedded in the string given in argument. Environments
> + * to be expanded shall be prefixed by a '$'. Unknown environment expands to
>   * the empty string.
>   */
>  char *expand_string_value(const char *in)
>  {
> -	const char *src;
> +	const char *p, *q;
>  	char *res;
>  	size_t reslen;
>  
> @@ -25,39 +107,28 @@ char *expand_string_value(const char *in)
>  	 * Note: 'in' might come from a token that's about to be
>  	 * freed, so make sure to always allocate a new string
>  	 */
> -	reslen = strlen(in) + 1;
> -	res = xmalloc(reslen);
> -	res[0] = '\0';
> -
> -	while ((src = strchr(in, '$'))) {
> -		char *p, name[SYMBOL_MAXLENGTH];
> -		const char *symval = "";
> -		struct symbol *sym;
> -		size_t newlen;
> -
> -		strncat(res, in, src - in);
> -		src++;
> -
> -		p = name;
> -		while (isalnum(*src) || *src == '_')
> -			*p++ = *src++;
> -		*p = '\0';
> -
> -		sym = sym_find(name);
> -		if (sym != NULL) {
> -			sym_calc_value(sym);
> -			symval = sym_get_string_value(sym);
> -		}
> +	res = xmalloc(1);
> +	*res = '\0';
>  
> -		newlen = strlen(res) + strlen(symval) + strlen(src) + 1;
> -		if (newlen > reslen) {
> -			reslen = newlen;
> -			res = xrealloc(res, reslen);
> -		}
> +	while ((p = strchr(in, '$'))) {
> +		char *new;
> +
> +		q = p + 1;
> +		while (isalnum(*q) || *q == '_')
> +			q++;
>  
> -		strcat(res, symval);
> -		in = src;
> +		new = env_expand_n(p + 1, q - p - 1);
> +
> +		reslen = strlen(res) + (p - in) + strlen(new) + 1;
> +		res = xrealloc(res, reslen);
> +		strncat(res, in, p - in);
> +		strcat(res, new);
> +		free(new);
> +		in = q;
>  	}
> +
> +	reslen = strlen(res) + strlen(in) + 1;
> +	res = xrealloc(res, reslen);
>  	strcat(res, in);
>  
>  	return res;
> @@ -87,8 +158,7 @@ struct file *file_lookup(const char *name)
>  /* write a dependency file as used by kbuild to track dependencies */
>  int file_write_dep(const char *name)
>  {
> -	struct symbol *sym, *env_sym;
> -	struct expr *e;
> +	struct env *env, *tmp;
>  	struct file *file;
>  	FILE *out;
>  
> @@ -107,20 +177,12 @@ int file_write_dep(const char *name)
>  	fprintf(out, "\n%s: \\\n"
>  		     "\t$(deps_config)\n\n", conf_get_autoconfig_name());
>  
> -	expr_list_for_each_sym(sym_env_list, e, sym) {
> -		struct property *prop;
> -		const char *value;
> -
> -		prop = sym_get_env_prop(sym);
> -		env_sym = prop_get_symbol(prop);
> -		if (!env_sym)
> -			continue;
> -		value = getenv(env_sym->name);
> -		if (!value)
> -			value = "";
> -		fprintf(out, "ifneq \"$(%s)\" \"%s\"\n", env_sym->name, value);
> +	list_for_each_entry_safe(env, tmp, &env_list, node) {
> +		fprintf(out, "ifneq \"$(%s)\" \"%s\"\n",
> +			env->name, getenv(env->name) ?: "");
>  		fprintf(out, "%s: FORCE\n", conf_get_autoconfig_name());
>  		fprintf(out, "endif\n");
> +		env_list_del(env);
>  	}
>  
>  	fprintf(out, "\n$(deps_config): ;\n");
> diff --git a/scripts/kconfig/zconf.l b/scripts/kconfig/zconf.l
> index 02de6fe..0d89ea6 100644
> --- a/scripts/kconfig/zconf.l
> +++ b/scripts/kconfig/zconf.l
> @@ -75,7 +75,7 @@ static void warn_ignored_character(char chr)
>  }
>  %}
>  
> -n	[A-Za-z0-9_-]
> +n	[$A-Za-z0-9_-]
>  
>  %%
>  	int str = 0;
> diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
> index 262c464..784083d 100644
> --- a/scripts/kconfig/zconf.y
> +++ b/scripts/kconfig/zconf.y
> @@ -534,7 +534,6 @@ void conf_parse(const char *name)
>  
>  	zconf_initscan(name);
>  
> -	sym_init();
>  	_menu_init();
>  
>  	if (getenv("ZCONF_DEBUG"))
> -- 
> 2.7.4
> 

Another option if we want to allow $ENV outside of quotes (even though I
prefer mandatory quotes) would be to transform it early into "$ENV" by
returning a T_WORD_QUOTE token for it.

I think you could then get expand_string_value() to handle both the
quoted and unquoted cases.

Cheers,
Ulf

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Sam Ravnborg]

Hi Masahiro.

On Sat, Feb 17, 2018 at 03:38:28AM +0900, Masahiro Yamada wrote:
> I brushed up the implementation in this version.
> 
> In the previous RFC, CC_HAS_ was described by using 'option shell=',
> like this:
> 
> config CC_HAS_STACKPROTECTOR
>         bool
>         option shell="$CC -Werror -fstack-protector -c -x c /dev/null"
> 
> After I thought a bit more, the following syntax is more grammatical,
> and flexible.
> 
> config CC_HAS_STACKPROTECTOR
>         bool
>         default $(shell $CC -Werror -fstack-protector -c -x c /dev/null)

Looks good - but maybe we should go one step further.

So we in the syntax explicit handles:
- shell commands
- other commands, defined as strings
- environment variables
- config variables

Each case is explicit - so the reader is not confused what is used when.

$(shell foo)	- output of the shell command foo. Uses $SHELL as the shell.
		  May include optional paramters.
		  foo may be a config variable referenced using ${} or a config variable prefixed with $
		Example:

		config BUILD_DIR
			string
			default $(shell cd ${objtree}; pwd)

$(call bar)	- output of the bar command that may take optional parameters.
		  bar may be a text string, a config variable or an environment variable
		   The definition of bar may reference the parameters using $(1), $(2)
		   In this context a config variable needs to be prefixed with $

		Example:

		config reverse
			string
			default $(2) $(1)

		config NEW_ORDER
			string
			$(call $reverse, A, B)	# Will assign REVERSE the value "B A"


		Example2:

		config CC_OPTION
			string
			default $(shell ${srctree}/scripts/cc-option ${CC} $(1) $(2))

		config CC_OPTIMIZE
			string
			$(call $CC_OPTION, -Oz, -Os)


${FOO}		- environment variable

The above is inspired by how make implement similar functionality.

I'm not happy that we in one context can reference CONFIG variables
directly, but inside the $(call ...) and $(shell ...) needs the $ prefix.
But I could not come up with something un-ambigious where this could be avoided.

The above proposal include the functionality of the macro stuff proposed in this patch-set.
But with a simpler syntax and we keep all the other kconfig logic (depends on etc) - so
users will not be limited in their creativity.

> Current limitations:
> 
> Dependency on outside scripts.
> Inter-option dependency:
> Functions are evaluated statically:

Same limitations exists with the syntax suggested above.

	Sam

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Ulf Magnusson]

Hello,

On Sun, Feb 18, 2018 at 11:13 PM, Sam Ravnborg <sam@ravnborg.org> wrote:
> Hi Masahiro.
>
> On Sat, Feb 17, 2018 at 03:38:28AM +0900, Masahiro Yamada wrote:
>> I brushed up the implementation in this version.
>>
>> In the previous RFC, CC_HAS_ was described by using 'option shell=',
>> like this:
>>
>> config CC_HAS_STACKPROTECTOR
>>         bool
>>         option shell="$CC -Werror -fstack-protector -c -x c /dev/null"
>>
>> After I thought a bit more, the following syntax is more grammatical,
>> and flexible.
>>
>> config CC_HAS_STACKPROTECTOR
>>         bool
>>         default $(shell $CC -Werror -fstack-protector -c -x c /dev/null)
>
> Looks good - but maybe we should go one step further.
>
> So we in the syntax explicit handles:
> - shell commands
> - other commands, defined as strings
> - environment variables
> - config variables
>
> Each case is explicit - so the reader is not confused what is used when.
>
> $(shell foo)    - output of the shell command foo. Uses $SHELL as the shell.
>                   May include optional paramters.
>                   foo may be a config variable referenced using ${} or a config variable prefixed with $
>                 Example:
>
>                 config BUILD_DIR
>                         string
>                         default $(shell cd ${objtree}; pwd)
>
> $(call bar)     - output of the bar command that may take optional parameters.
>                   bar may be a text string, a config variable or an environment variable
>                    The definition of bar may reference the parameters using $(1), $(2)
>                    In this context a config variable needs to be prefixed with $
>
>                 Example:
>
>                 config reverse
>                         string
>                         default $(2) $(1)
>
>                 config NEW_ORDER
>                         string
>                         $(call $reverse, A, B)  # Will assign REVERSE the value "B A"
>
>
>                 Example2:
>
>                 config CC_OPTION
>                         string
>                         default $(shell ${srctree}/scripts/cc-option ${CC} $(1) $(2))
>
>                 config CC_OPTIMIZE
>                         string
>                         $(call $CC_OPTION, -Oz, -Os)
>
>
> ${FOO}          - environment variable
>
> The above is inspired by how make implement similar functionality.
>
> I'm not happy that we in one context can reference CONFIG variables
> directly, but inside the $(call ...) and $(shell ...) needs the $ prefix.
> But I could not come up with something un-ambigious where this could be avoided.

I think we should be careful about allowing references to config
symbols. It mixes up the parsing and evaluation phases, since $() is
expanded during parsing (which I consider a feature and think is
needed to retain sanity).

Patch 06/23 removes the last existing instance of symbol references in
strings by getting rid of 'option env'. That's an improvement to me.
We shouldn't add it back.

>
> The above proposal include the functionality of the macro stuff proposed in this patch-set.
> But with a simpler syntax and we keep all the other kconfig logic (depends on etc) - so
> users will not be limited in their creativity.
>
>> Current limitations:
>>
>> Dependency on outside scripts.
>> Inter-option dependency:
>> Functions are evaluated statically:
>
> Same limitations exists with the syntax suggested above.
>
>         Sam

Cheers,
Ulf

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:

>>
>> I'm not happy that we in one context can reference CONFIG variables
>> directly, but inside the $(call ...) and $(shell ...) needs the $ prefix.
>> But I could not come up with something un-ambigious where this could be avoided.
>
> I think we should be careful about allowing references to config
> symbols. It mixes up the parsing and evaluation phases, since $() is
> expanded during parsing (which I consider a feature and think is
> needed to retain sanity).
>
> Patch 06/23 removes the last existing instance of symbol references in
> strings by getting rid of 'option env'. That's an improvement to me.
> We shouldn't add it back.


This is really important design decision,
so I'd like to hear a little more from experts.


For example, x86 allows users to choose sub-arch, either 'i386' or 'x86_64'.

https://github.com/torvalds/linux/blob/v4.16-rc2/arch/x86/Kconfig#L4



If the user toggles CONFIG_64BIT,
the bi-arch compiler will work in a slightly different mode
(at least, back-end parts)

So, my question is, is there a case,

$(cc-option, -m32 -foo) is y, but
$(cc-option, -m64 -foo) is n  ?
(or vice versa)


If the answer is yes, $(cc-option -foo) would have to be re-calculated
every time CONFIG_64BIT is toggled.

This is what I'd like to avoid, though.



-- 
Best Regards
Masahiro Yamada

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Arnd Bergmann]

On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>
>>>
>>> I'm not happy that we in one context can reference CONFIG variables
>>> directly, but inside the $(call ...) and $(shell ...) needs the $ prefix.
>>> But I could not come up with something un-ambigious where this could be avoided.
>>
>> I think we should be careful about allowing references to config
>> symbols. It mixes up the parsing and evaluation phases, since $() is
>> expanded during parsing (which I consider a feature and think is
>> needed to retain sanity).
>>
>> Patch 06/23 removes the last existing instance of symbol references in
>> strings by getting rid of 'option env'. That's an improvement to me.
>> We shouldn't add it back.
>
>
> This is really important design decision,
> so I'd like to hear a little more from experts.
>
>
> For example, x86 allows users to choose sub-arch, either 'i386' or 'x86_64'.
>
> https://github.com/torvalds/linux/blob/v4.16-rc2/arch/x86/Kconfig#L4
>
>
>
> If the user toggles CONFIG_64BIT,
> the bi-arch compiler will work in a slightly different mode
> (at least, back-end parts)
>
> So, my question is, is there a case,
>
> $(cc-option, -m32 -foo) is y, but
> $(cc-option, -m64 -foo) is n  ?
> (or vice versa)
>
>
> If the answer is yes, $(cc-option -foo) would have to be re-calculated
> every time CONFIG_64BIT is toggled.
>
> This is what I'd like to avoid, though.

The -m32/-m64 trick (and -mbig-endian/-mlittle-endian on other architectures
as well as a couple of other flags) only works if the compiler is configured to
support it. In other cases (e.g. big-endian xtensa), the kernel always
detects what the compiler does and silently configures itself to match
using Makefile logic.

On x86, compilers are usually built as bi-arch, but you can build one that
only allows one of them.

I can see two reasonable ways out:

- we don't use  $(cc-option -foo) in a case like this, and instead require the
  user to have a matching toolchain.
- we could make the 32/64 selection on x86 a 'choice' statement where
  each option depends on both the ARCH= variable and the
  $(cc-option, -m32)/ $(cc-option, -m64) output.

       Arnd

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>>
>>>>
>>>> I'm not happy that we in one context can reference CONFIG variables
>>>> directly, but inside the $(call ...) and $(shell ...) needs the $ prefix.
>>>> But I could not come up with something un-ambigious where this could be avoided.
>>>
>>> I think we should be careful about allowing references to config
>>> symbols. It mixes up the parsing and evaluation phases, since $() is
>>> expanded during parsing (which I consider a feature and think is
>>> needed to retain sanity).
>>>
>>> Patch 06/23 removes the last existing instance of symbol references in
>>> strings by getting rid of 'option env'. That's an improvement to me.
>>> We shouldn't add it back.
>>
>>
>> This is really important design decision,
>> so I'd like to hear a little more from experts.
>>
>>
>> For example, x86 allows users to choose sub-arch, either 'i386' or 'x86_64'.
>>
>> https://github.com/torvalds/linux/blob/v4.16-rc2/arch/x86/Kconfig#L4
>>
>>
>>
>> If the user toggles CONFIG_64BIT,
>> the bi-arch compiler will work in a slightly different mode
>> (at least, back-end parts)
>>
>> So, my question is, is there a case,
>>
>> $(cc-option, -m32 -foo) is y, but
>> $(cc-option, -m64 -foo) is n  ?
>> (or vice versa)
>>
>>
>> If the answer is yes, $(cc-option -foo) would have to be re-calculated
>> every time CONFIG_64BIT is toggled.
>>
>> This is what I'd like to avoid, though.
>
> The -m32/-m64 trick (and -mbig-endian/-mlittle-endian on other architectures
> as well as a couple of other flags) only works if the compiler is configured to
> support it. In other cases (e.g. big-endian xtensa), the kernel always
> detects what the compiler does and silently configures itself to match
> using Makefile logic.
>
> On x86, compilers are usually built as bi-arch, but you can build one that
> only allows one of them.
>
> I can see two reasonable ways out:
>
> - we don't use  $(cc-option -foo) in a case like this, and instead require the
>   user to have a matching toolchain.
> - we could make the 32/64 selection on x86 a 'choice' statement where
>   each option depends on both the ARCH= variable and the
>   $(cc-option, -m32)/ $(cc-option, -m64) output.
>
>        Arnd



Let me clarify my concern.

When we test the compiler flag, is there a case
where a particular flag depends on -m{32,64} ?

For example, is there a compiler that supports -fstack-protector
for 64bit mode, but unsupports it for 32bit mode?

  $(cc-option -m32)                     ->  y
  $(cc-option -m64)                     ->  y
  $(cc-option -fstack-protector)        ->  y
  $(cc-option -m32 -fstack-protector)   ->  n
  $(cc-option -m64 -fstack-protector)   ->  y

I guess this is unlikely to happen,
but I am not whether it is zero possibility.

If this could happen,
$(cc-option ) must be evaluated together with
correct bi-arch option (either -m32 or -m64).


Currently, -m32/-m64 is specified in Makefile,
but we are moving compiler tests to Kconfig
and, CONFIG_64BIT can be dynamically toggled in Kconfig.




-- 
Best Regards
Masahiro Yamada

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Arnd Bergmann]

On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
>> <yamada.masahiro@socionext.com> wrote:
>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>
> Let me clarify my concern.
>
> When we test the compiler flag, is there a case
> where a particular flag depends on -m{32,64} ?
>
> For example, is there a compiler that supports -fstack-protector
> for 64bit mode, but unsupports it for 32bit mode?
>
>   $(cc-option -m32)                     ->  y
>   $(cc-option -m64)                     ->  y
>   $(cc-option -fstack-protector)        ->  y
>   $(cc-option -m32 -fstack-protector)   ->  n
>   $(cc-option -m64 -fstack-protector)   ->  y
>
> I guess this is unlikely to happen,
> but I am not whether it is zero possibility.
>
> If this could happen,
> $(cc-option ) must be evaluated together with
> correct bi-arch option (either -m32 or -m64).
>
>
> Currently, -m32/-m64 is specified in Makefile,
> but we are moving compiler tests to Kconfig
> and, CONFIG_64BIT can be dynamically toggled in Kconfig.

I don't think it can happen for this particular combination (stack protector
and word size), but I'm sure we'll eventually run into options that
need to be tested in combination. For the current CFLAGS_KERNEL
setting, we definitely have the case of needing the variables to be
evaluated in a specific order.

      Arnd

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
> <yamada.masahiro@socionext.com> wrote:
>> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
>>> <yamada.masahiro@socionext.com> wrote:
>>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>>
>> Let me clarify my concern.
>>
>> When we test the compiler flag, is there a case
>> where a particular flag depends on -m{32,64} ?
>>
>> For example, is there a compiler that supports -fstack-protector
>> for 64bit mode, but unsupports it for 32bit mode?
>>
>>   $(cc-option -m32)                     ->  y
>>   $(cc-option -m64)                     ->  y
>>   $(cc-option -fstack-protector)        ->  y
>>   $(cc-option -m32 -fstack-protector)   ->  n
>>   $(cc-option -m64 -fstack-protector)   ->  y
>>
>> I guess this is unlikely to happen,
>> but I am not whether it is zero possibility.
>>
>> If this could happen,
>> $(cc-option ) must be evaluated together with
>> correct bi-arch option (either -m32 or -m64).
>>
>>
>> Currently, -m32/-m64 is specified in Makefile,
>> but we are moving compiler tests to Kconfig
>> and, CONFIG_64BIT can be dynamically toggled in Kconfig.
>
> I don't think it can happen for this particular combination (stack protector
> and word size), but I'm sure we'll eventually run into options that
> need to be tested in combination. For the current CFLAGS_KERNEL
> setting, we definitely have the case of needing the variables to be
> evaluated in a specific order.
>




I was thinking of how we can handle complex cases
in the current approach.



(Case 1)

Compiler flag -foo and -bar interacts, so
we also need to check the combination of the two.


config CC_HAS_FOO
        def_bool $(cc-option -foo)

config CC_HAS_BAR
        def_bool $(cc-option -bar)

config CC_HAS_FOO_WITH_BAR
        def_bool $(cc-option -foo -bar)



(Case 2)
Compiler flag -foo is sensitive to word-size.
So, we need to test this option together with -m32/-m64.
User can toggle CONFIG_64BIT, like i386/x86_64.


config CC_NEEDS_M64
          def_bool $(cc-option -m64) && 64BIT

config CC_NEEDS_M32
          def_bool $(cc-option -m32) && !64BIT

config CC_HAS_FOO
         bool
         default $(cc-option -m64 -foo) if CC_NEEDS_M64
         default $(cc-option -m32 -foo) if CC_NEEDS_M32
         default $(cc-option -foo)



(Case 3)
Compiler flag -foo is sensitive to endian-ness.


config CC_NEEDS_BIG_ENDIAN
          def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN

config CC_NEEDS_LITTLE_ENDIAN
          def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN

config CC_HAS_FOO
         bool
         default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
         default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
         default $(cc-option -foo)




Hmm, I think I can implement those somehow.
But, I hope we do not have many instances like this...


If you know more naive cases, please share your knowledge.

Thanks!


-- 
Best Regards
Masahiro Yamada

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Arnd Bergmann]

On Wed, Feb 21, 2018 at 1:57 PM, Masahiro Yamada
<yamada.masahiro@socionext.com> wrote:
> 2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>> On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
>> <yamada.masahiro@socionext.com> wrote:
>>> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>>>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
>>>> <yamada.masahiro@socionext.com> wrote:
>>>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>
> Hmm, I think I can implement those somehow.
> But, I hope we do not have many instances like this...
>
>
> If you know more naive cases, please share your knowledge.
>

One case that comes to mind would be architecture level selection on 32-bit
ARM, which is roughly this (I probably have some details wrong, but you
get the idea):

- older compilers don't support the latest architecture setting (-march=armv8
  or -march=armv7ve)
- newer compilers no longer support really old architectures (-march=armv4)
- setting -mthumb requires setting one of  -march=armv7-a, armv7ve, armv7-m or
  armv8 if the compiler doesn't default to those
- on a compiler that defaults to -marm, setting -march=armv7-m requires
  setting -mthumb (IIRC)
- really old compilers only support OABI, but not EABI
- newer compilers no longer support OABI
- mthumb requires EABI
- armv6 and higher are subtly broken with OABI, but only when using
  certain inline assembly with 64-bit arguments in register pairs.

I think we just shouldn't try to capture all of the above correctly in Kconfig
conditionals.

       Arnd

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Ulf Magnusson]

On Wed, Feb 21, 2018 at 09:57:03PM +0900, Masahiro Yamada wrote:
> 2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> > On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
> > <yamada.masahiro@socionext.com> wrote:
> >> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> >>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
> >>> <yamada.masahiro@socionext.com> wrote:
> >>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> >>
> >> Let me clarify my concern.
> >>
> >> When we test the compiler flag, is there a case
> >> where a particular flag depends on -m{32,64} ?
> >>
> >> For example, is there a compiler that supports -fstack-protector
> >> for 64bit mode, but unsupports it for 32bit mode?
> >>
> >>   $(cc-option -m32)                     ->  y
> >>   $(cc-option -m64)                     ->  y
> >>   $(cc-option -fstack-protector)        ->  y
> >>   $(cc-option -m32 -fstack-protector)   ->  n
> >>   $(cc-option -m64 -fstack-protector)   ->  y
> >>
> >> I guess this is unlikely to happen,
> >> but I am not whether it is zero possibility.
> >>
> >> If this could happen,
> >> $(cc-option ) must be evaluated together with
> >> correct bi-arch option (either -m32 or -m64).
> >>
> >>
> >> Currently, -m32/-m64 is specified in Makefile,
> >> but we are moving compiler tests to Kconfig
> >> and, CONFIG_64BIT can be dynamically toggled in Kconfig.
> >
> > I don't think it can happen for this particular combination (stack protector
> > and word size), but I'm sure we'll eventually run into options that
> > need to be tested in combination. For the current CFLAGS_KERNEL
> > setting, we definitely have the case of needing the variables to be
> > evaluated in a specific order.
> >
> 
> 
> 
> 
> I was thinking of how we can handle complex cases
> in the current approach.
> 
> 
> 
> (Case 1)
> 
> Compiler flag -foo and -bar interacts, so
> we also need to check the combination of the two.
> 
> 
> config CC_HAS_FOO
>         def_bool $(cc-option -foo)
> 
> config CC_HAS_BAR
>         def_bool $(cc-option -bar)
> 
> config CC_HAS_FOO_WITH_BAR
>         def_bool $(cc-option -foo -bar)
> 
> 
> 
> (Case 2)
> Compiler flag -foo is sensitive to word-size.
> So, we need to test this option together with -m32/-m64.
> User can toggle CONFIG_64BIT, like i386/x86_64.
> 
> 
> config CC_NEEDS_M64
>           def_bool $(cc-option -m64) && 64BIT
> 
> config CC_NEEDS_M32
>           def_bool $(cc-option -m32) && !64BIT
> 
> config CC_HAS_FOO
>          bool
>          default $(cc-option -m64 -foo) if CC_NEEDS_M64
>          default $(cc-option -m32 -foo) if CC_NEEDS_M32
>          default $(cc-option -foo)
> 
> 
> 
> (Case 3)
> Compiler flag -foo is sensitive to endian-ness.
> 
> 
> config CC_NEEDS_BIG_ENDIAN
>           def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN
> 
> config CC_NEEDS_LITTLE_ENDIAN
>           def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN
> 
> config CC_HAS_FOO
>          bool
>          default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
>          default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
>          default $(cc-option -foo)
> 
> 
> 
> 
> Hmm, I think I can implement those somehow.
> But, I hope we do not have many instances like this...
> 
> 
> If you know more naive cases, please share your knowledge.
> 
> Thanks!
> 
> 
> -- 
> Best Regards
> Masahiro Yamada

Would get pretty bad if a test needs to consider multiple symbols.
Exponential explosion there...


I thought some more about the implementation of dynamic (post-parsing)
functions to see how bad it would get with the current implementation.

Some background on how things work now:

  1. All expression operands in Kconfig are symbols.

  2. Returning '$ENV' or '$(fn foo)' as a T_WORD during parsing gets
     you symbols with those strings as names and S_UNKNOWN type (because
     they act like references to undefined symbols).

  3. For "foo-$(fn foo)", you also get a symbol with that string as its
     name and S_UNKNOWN type (stored among the SYMBOL_CONST symbols)

  4. Symbols with S_UNKNOWN type get their name as their string value,
     and the tristate value n.

So, if you do string expansion on the names of symbols with S_UNKNOWN
type in sym_calc_value(), you're almost there with the current
implementation, except for the tristate case.

Maybe you could set the tristate value of S_UNKNOWN symbols depending on
the string value you end up with. Things are getting pretty confusing at
that point.

Could have something like S_DYNAMIC as well. More Kconfig complexity...

Then there's other complications:

  1. SYMBOL_CONST is no longer constant.

  2. Dependency loop detection needs to consider symbol references
     within strings.

  3. Dependency loop detection relies on static knowledge of what
     symbols a symbol depends on. That might get messy for certain
     expansions, though it might be things you wouldn't do in practice.

  4. Symbols still need to be properly invalidated. It looks like at
     least menuconfig just does a dumb invalidate-everything whenever
     the value of a symbol is changed though, so it might not require
     extra work. (Bit messier in Kconfiglib, which does minimal
     invalidation to keep scripts fast, but just need to extract a few
     extra deps there.)


It looks like dynamic functions could get quite messy, but might be
doable if absolutely required. There's probably more devils in the
details though.

I don't think the static function model precludes switching models later
btw, when people have more experience.

Cheers,
Ulf

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Michael Ellerman]

Masahiro Yamada <yamada.masahiro@socionext.com> writes:
>
<snip>
>
> (Case 3)
> Compiler flag -foo is sensitive to endian-ness.
>
>
> config CC_NEEDS_BIG_ENDIAN
>           def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN
>
> config CC_NEEDS_LITTLE_ENDIAN
>           def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN
>
> config CC_HAS_FOO
>          bool
>          default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
>          default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
>          default $(cc-option -foo)

We may do something like this on powerpc, where we have 32/64-bit and
big/little endian (on 64-bit) and then some ABI options that we
set/unset depending on endian.

The above looks like it could work though.

cheers

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Masahiro Yamada]

2018-02-22 6:39 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> On Wed, Feb 21, 2018 at 09:57:03PM +0900, Masahiro Yamada wrote:
>> 2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>> > On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
>> > <yamada.masahiro@socionext.com> wrote:
>> >> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
>> >>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
>> >>> <yamada.masahiro@socionext.com> wrote:
>> >>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
>> >>
>> >> Let me clarify my concern.
>> >>
>> >> When we test the compiler flag, is there a case
>> >> where a particular flag depends on -m{32,64} ?
>> >>
>> >> For example, is there a compiler that supports -fstack-protector
>> >> for 64bit mode, but unsupports it for 32bit mode?
>> >>
>> >>   $(cc-option -m32)                     ->  y
>> >>   $(cc-option -m64)                     ->  y
>> >>   $(cc-option -fstack-protector)        ->  y
>> >>   $(cc-option -m32 -fstack-protector)   ->  n
>> >>   $(cc-option -m64 -fstack-protector)   ->  y
>> >>
>> >> I guess this is unlikely to happen,
>> >> but I am not whether it is zero possibility.
>> >>
>> >> If this could happen,
>> >> $(cc-option ) must be evaluated together with
>> >> correct bi-arch option (either -m32 or -m64).
>> >>
>> >>
>> >> Currently, -m32/-m64 is specified in Makefile,
>> >> but we are moving compiler tests to Kconfig
>> >> and, CONFIG_64BIT can be dynamically toggled in Kconfig.
>> >
>> > I don't think it can happen for this particular combination (stack protector
>> > and word size), but I'm sure we'll eventually run into options that
>> > need to be tested in combination. For the current CFLAGS_KERNEL
>> > setting, we definitely have the case of needing the variables to be
>> > evaluated in a specific order.
>> >
>>
>>
>>
>>
>> I was thinking of how we can handle complex cases
>> in the current approach.
>>
>>
>>
>> (Case 1)
>>
>> Compiler flag -foo and -bar interacts, so
>> we also need to check the combination of the two.
>>
>>
>> config CC_HAS_FOO
>>         def_bool $(cc-option -foo)
>>
>> config CC_HAS_BAR
>>         def_bool $(cc-option -bar)
>>
>> config CC_HAS_FOO_WITH_BAR
>>         def_bool $(cc-option -foo -bar)
>>
>>
>>
>> (Case 2)
>> Compiler flag -foo is sensitive to word-size.
>> So, we need to test this option together with -m32/-m64.
>> User can toggle CONFIG_64BIT, like i386/x86_64.
>>
>>
>> config CC_NEEDS_M64
>>           def_bool $(cc-option -m64) && 64BIT
>>
>> config CC_NEEDS_M32
>>           def_bool $(cc-option -m32) && !64BIT
>>
>> config CC_HAS_FOO
>>          bool
>>          default $(cc-option -m64 -foo) if CC_NEEDS_M64
>>          default $(cc-option -m32 -foo) if CC_NEEDS_M32
>>          default $(cc-option -foo)
>>
>>
>>
>> (Case 3)
>> Compiler flag -foo is sensitive to endian-ness.
>>
>>
>> config CC_NEEDS_BIG_ENDIAN
>>           def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN
>>
>> config CC_NEEDS_LITTLE_ENDIAN
>>           def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN
>>
>> config CC_HAS_FOO
>>          bool
>>          default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
>>          default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
>>          default $(cc-option -foo)
>>
>>
>>
>>
>> Hmm, I think I can implement those somehow.
>> But, I hope we do not have many instances like this...
>>
>>
>> If you know more naive cases, please share your knowledge.
>>
>> Thanks!
>>
>>
>> --
>> Best Regards
>> Masahiro Yamada
>
> Would get pretty bad if a test needs to consider multiple symbols.
> Exponential explosion there...
>
>
> I thought some more about the implementation of dynamic (post-parsing)
> functions to see how bad it would get with the current implementation.
>
> Some background on how things work now:
>
>   1. All expression operands in Kconfig are symbols.
>
>   2. Returning '$ENV' or '$(fn foo)' as a T_WORD during parsing gets
>      you symbols with those strings as names and S_UNKNOWN type (because
>      they act like references to undefined symbols).
>
>   3. For "foo-$(fn foo)", you also get a symbol with that string as its
>      name and S_UNKNOWN type (stored among the SYMBOL_CONST symbols)
>
>   4. Symbols with S_UNKNOWN type get their name as their string value,
>      and the tristate value n.
>
> So, if you do string expansion on the names of symbols with S_UNKNOWN
> type in sym_calc_value(), you're almost there with the current
> implementation, except for the tristate case.
>
> Maybe you could set the tristate value of S_UNKNOWN symbols depending on
> the string value you end up with. Things are getting pretty confusing at
> that point.
>
> Could have something like S_DYNAMIC as well. More Kconfig complexity...
>
> Then there's other complications:
>
>   1. SYMBOL_CONST is no longer constant.
>
>   2. Dependency loop detection needs to consider symbol references
>      within strings.
>
>   3. Dependency loop detection relies on static knowledge of what
>      symbols a symbol depends on. That might get messy for certain
>      expansions, though it might be things you wouldn't do in practice.
>
>   4. Symbols still need to be properly invalidated. It looks like at
>      least menuconfig just does a dumb invalidate-everything whenever
>      the value of a symbol is changed though, so it might not require
>      extra work. (Bit messier in Kconfiglib, which does minimal
>      invalidation to keep scripts fast, but just need to extract a few
>      extra deps there.)
>
>
> It looks like dynamic functions could get quite messy, but might be
> doable if absolutely required. There's probably more devils in the
> details though.
>
> I don't think the static function model precludes switching models later
> btw, when people have more experience.



I really want to start with the static function model
and see if we need the dynamic function implementation.

Here is an idea for the migration path in case
we need to do that in the future.



Currently, every time user input is given,
sym_clear_all_valid() is called.

It is not efficient to blindly re-evaluate expensive $(shell ...)


So, have a list of symbols the function depends on
in its arguments.

For example,

config CC_HAS_SANE_STACKPROTECTOR
        def_bool $(shell $srctree/scripts/gcc-has-stack-protector.sh
$CC $(1), CFLAGS_BASE)


Here the first argument
  $srctree/scripts/gcc-x86-has-stack-protector.sh $CC $(1)

is the shell command.
$(1), $(2), ... will be replaced with the values of symbols (or expressions)
that follow when running the shell command.


The second argument
CFLAGS_BASE
is the dependency symbol (or expression).


CFLAGS_BASE can be dynamically changed like

config CFLAGS_BASE
        string
        default "-m64" if 64BIT
        default "-m32"


When and only when CFLAGS_BASE is updated, the function should be re-calculated.
(This will require efforts to minimize the amount of re-evaluation.)




cc-option will be implemented like follows:

macro cc-option $(shell $CC -Werror $$(1) $(1) -c -x c /dev/null -o
/dev/null, CFLAGS_BASE)



Please notice the difference between $$(1) and $(1).

$(1) is immediately expanded by cc-option macro.

$$(1) is escaped since we want to expand it by $(shell ...), not by
$(cc-option ...)



For example,

  $(cc-option -fstack-protector)

will be expanded to

  $(shell gcc -Werror $(1) -fstack-protector -c -x c /dev/null -o
/dev/null, CFLAGS_BASE)

Since macros are just textual shorthand, so this expansion happens
during the parse phase.



Then, the evaluation phase does the following every time CFLAGS_BASE is updated.

gcc -Werror [value of CFLAGS_BASE] -fstack-protector -c -x c /dev/null
-o /dev/null


This is a new form of expression, so it will be managed in AST tree
with a flag E_SHELL (or E_FUNC) etc.


Not implemented at all.  Just a rough sketch.


-- 
Best Regards
Masahiro Yamada

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Ulf Magnusson]

On Fri, Mar 02, 2018 at 02:50:39PM +0900, Masahiro Yamada wrote:
> 2018-02-22 6:39 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> > On Wed, Feb 21, 2018 at 09:57:03PM +0900, Masahiro Yamada wrote:
> >> 2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> >> > On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
> >> > <yamada.masahiro@socionext.com> wrote:
> >> >> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> >> >>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
> >> >>> <yamada.masahiro@socionext.com> wrote:
> >> >>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> >> >>
> >> >> Let me clarify my concern.
> >> >>
> >> >> When we test the compiler flag, is there a case
> >> >> where a particular flag depends on -m{32,64} ?
> >> >>
> >> >> For example, is there a compiler that supports -fstack-protector
> >> >> for 64bit mode, but unsupports it for 32bit mode?
> >> >>
> >> >>   $(cc-option -m32)                     ->  y
> >> >>   $(cc-option -m64)                     ->  y
> >> >>   $(cc-option -fstack-protector)        ->  y
> >> >>   $(cc-option -m32 -fstack-protector)   ->  n
> >> >>   $(cc-option -m64 -fstack-protector)   ->  y
> >> >>
> >> >> I guess this is unlikely to happen,
> >> >> but I am not whether it is zero possibility.
> >> >>
> >> >> If this could happen,
> >> >> $(cc-option ) must be evaluated together with
> >> >> correct bi-arch option (either -m32 or -m64).
> >> >>
> >> >>
> >> >> Currently, -m32/-m64 is specified in Makefile,
> >> >> but we are moving compiler tests to Kconfig
> >> >> and, CONFIG_64BIT can be dynamically toggled in Kconfig.
> >> >
> >> > I don't think it can happen for this particular combination (stack protector
> >> > and word size), but I'm sure we'll eventually run into options that
> >> > need to be tested in combination. For the current CFLAGS_KERNEL
> >> > setting, we definitely have the case of needing the variables to be
> >> > evaluated in a specific order.
> >> >
> >>
> >>
> >>
> >>
> >> I was thinking of how we can handle complex cases
> >> in the current approach.
> >>
> >>
> >>
> >> (Case 1)
> >>
> >> Compiler flag -foo and -bar interacts, so
> >> we also need to check the combination of the two.
> >>
> >>
> >> config CC_HAS_FOO
> >>         def_bool $(cc-option -foo)
> >>
> >> config CC_HAS_BAR
> >>         def_bool $(cc-option -bar)
> >>
> >> config CC_HAS_FOO_WITH_BAR
> >>         def_bool $(cc-option -foo -bar)
> >>
> >>
> >>
> >> (Case 2)
> >> Compiler flag -foo is sensitive to word-size.
> >> So, we need to test this option together with -m32/-m64.
> >> User can toggle CONFIG_64BIT, like i386/x86_64.
> >>
> >>
> >> config CC_NEEDS_M64
> >>           def_bool $(cc-option -m64) && 64BIT
> >>
> >> config CC_NEEDS_M32
> >>           def_bool $(cc-option -m32) && !64BIT
> >>
> >> config CC_HAS_FOO
> >>          bool
> >>          default $(cc-option -m64 -foo) if CC_NEEDS_M64
> >>          default $(cc-option -m32 -foo) if CC_NEEDS_M32
> >>          default $(cc-option -foo)
> >>
> >>
> >>
> >> (Case 3)
> >> Compiler flag -foo is sensitive to endian-ness.
> >>
> >>
> >> config CC_NEEDS_BIG_ENDIAN
> >>           def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN
> >>
> >> config CC_NEEDS_LITTLE_ENDIAN
> >>           def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN
> >>
> >> config CC_HAS_FOO
> >>          bool
> >>          default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
> >>          default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
> >>          default $(cc-option -foo)
> >>
> >>
> >>
> >>
> >> Hmm, I think I can implement those somehow.
> >> But, I hope we do not have many instances like this...
> >>
> >>
> >> If you know more naive cases, please share your knowledge.
> >>
> >> Thanks!
> >>
> >>
> >> --
> >> Best Regards
> >> Masahiro Yamada
> >
> > Would get pretty bad if a test needs to consider multiple symbols.
> > Exponential explosion there...
> >
> >
> > I thought some more about the implementation of dynamic (post-parsing)
> > functions to see how bad it would get with the current implementation.
> >
> > Some background on how things work now:
> >
> >   1. All expression operands in Kconfig are symbols.
> >
> >   2. Returning '$ENV' or '$(fn foo)' as a T_WORD during parsing gets
> >      you symbols with those strings as names and S_UNKNOWN type (because
> >      they act like references to undefined symbols).
> >
> >   3. For "foo-$(fn foo)", you also get a symbol with that string as its
> >      name and S_UNKNOWN type (stored among the SYMBOL_CONST symbols)
> >
> >   4. Symbols with S_UNKNOWN type get their name as their string value,
> >      and the tristate value n.
> >
> > So, if you do string expansion on the names of symbols with S_UNKNOWN
> > type in sym_calc_value(), you're almost there with the current
> > implementation, except for the tristate case.
> >
> > Maybe you could set the tristate value of S_UNKNOWN symbols depending on
> > the string value you end up with. Things are getting pretty confusing at
> > that point.
> >
> > Could have something like S_DYNAMIC as well. More Kconfig complexity...
> >
> > Then there's other complications:
> >
> >   1. SYMBOL_CONST is no longer constant.
> >
> >   2. Dependency loop detection needs to consider symbol references
> >      within strings.
> >
> >   3. Dependency loop detection relies on static knowledge of what
> >      symbols a symbol depends on. That might get messy for certain
> >      expansions, though it might be things you wouldn't do in practice.
> >
> >   4. Symbols still need to be properly invalidated. It looks like at
> >      least menuconfig just does a dumb invalidate-everything whenever
> >      the value of a symbol is changed though, so it might not require
> >      extra work. (Bit messier in Kconfiglib, which does minimal
> >      invalidation to keep scripts fast, but just need to extract a few
> >      extra deps there.)
> >
> >
> > It looks like dynamic functions could get quite messy, but might be
> > doable if absolutely required. There's probably more devils in the
> > details though.
> >
> > I don't think the static function model precludes switching models later
> > btw, when people have more experience.
> 
> 
> 
> I really want to start with the static function model
> and see if we need the dynamic function implementation.

Yeah, let's start with static functions, IMO.

Either we'll learn that they're powerful enough in practice, and save
ourselves some work, or we'll gain experience for later. Converting from
static to dynamic functions should be painless, if needed.

My plan would be something like:

  1. Implement static functions

  2. Convert as many simple cases over to them as possible

  3. See how bad the bad cases get. If they get really bad, then decide
     what to do next (extend Kconfig, handle them in the Makefiles,
     etc.)

> 
> Here is an idea for the migration path in case
> we need to do that in the future.
> 
> 
> 
> Currently, every time user input is given,
> sym_clear_all_valid() is called.
> 
> It is not efficient to blindly re-evaluate expensive $(shell ...)

I think menuconfig only reevalutes the symbols in the menu that's
currently shown in the interface (along with their dependencies).

Maybe that'd be bad enough though.

> 
> 
> So, have a list of symbols the function depends on
> in its arguments.
> 
> For example,
> 
> config CC_HAS_SANE_STACKPROTECTOR
>         def_bool $(shell $srctree/scripts/gcc-has-stack-protector.sh
> $CC $(1), CFLAGS_BASE)
> 
> 
> Here the first argument
>   $srctree/scripts/gcc-x86-has-stack-protector.sh $CC $(1)
> 
> is the shell command.
> $(1), $(2), ... will be replaced with the values of symbols (or expressions)
> that follow when running the shell command.
> 
> 
> The second argument
> CFLAGS_BASE
> is the dependency symbol (or expression).
> 
> 
> CFLAGS_BASE can be dynamically changed like
> 
> config CFLAGS_BASE
>         string
>         default "-m64" if 64BIT
>         default "-m32"
> 
> 
> When and only when CFLAGS_BASE is updated, the function should be re-calculated.
> (This will require efforts to minimize the amount of re-evaluation.)
> 
> 
> 
> 
> cc-option will be implemented like follows:
> 
> macro cc-option $(shell $CC -Werror $$(1) $(1) -c -x c /dev/null -o
> /dev/null, CFLAGS_BASE)
> 
> 
> 
> Please notice the difference between $$(1) and $(1).
> 
> $(1) is immediately expanded by cc-option macro.
> 
> $$(1) is escaped since we want to expand it by $(shell ...), not by
> $(cc-option ...)
> 
> 
> 
> For example,
> 
>   $(cc-option -fstack-protector)
> 
> will be expanded to
> 
>   $(shell gcc -Werror $(1) -fstack-protector -c -x c /dev/null -o
> /dev/null, CFLAGS_BASE)
> 
> Since macros are just textual shorthand, so this expansion happens
> during the parse phase.
> 
> 
> 
> Then, the evaluation phase does the following every time CFLAGS_BASE is updated.
> 
> gcc -Werror [value of CFLAGS_BASE] -fstack-protector -c -x c /dev/null
> -o /dev/null
> 
> 
> This is a new form of expression, so it will be managed in AST tree
> with a flag E_SHELL (or E_FUNC) etc.
> 
> 
> Not implemented at all.  Just a rough sketch.

A simpler syntax like

	$(shell $CC -Werror {CFLAGS_BASE} -c -x c /dev/null -o /dev/null)

might work as well. Then you wouldn't have to any double escaping, and
having both $(1) and the value for it appear in the same place seems a
bit redundant.

I wonder if dependency management might get messy...

Anyway, let's just go with static functions...

Cheers,
Ulf

Re: [PATCH 00/23] kconfig: move compiler capability tests to Kconfig

[Ulf Magnusson]

On Fri, Mar 02, 2018 at 10:03:26AM +0100, Ulf Magnusson wrote:
> On Fri, Mar 02, 2018 at 02:50:39PM +0900, Masahiro Yamada wrote:
> > 2018-02-22 6:39 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> > > On Wed, Feb 21, 2018 at 09:57:03PM +0900, Masahiro Yamada wrote:
> > >> 2018-02-21 19:52 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> > >> > On Wed, Feb 21, 2018 at 11:20 AM, Masahiro Yamada
> > >> > <yamada.masahiro@socionext.com> wrote:
> > >> >> 2018-02-21 18:56 GMT+09:00 Arnd Bergmann <arnd@arndb.de>:
> > >> >>> On Wed, Feb 21, 2018 at 8:38 AM, Masahiro Yamada
> > >> >>> <yamada.masahiro@socionext.com> wrote:
> > >> >>>> 2018-02-20 0:18 GMT+09:00 Ulf Magnusson <ulfalizer@gmail.com>:
> > >> >>
> > >> >> Let me clarify my concern.
> > >> >>
> > >> >> When we test the compiler flag, is there a case
> > >> >> where a particular flag depends on -m{32,64} ?
> > >> >>
> > >> >> For example, is there a compiler that supports -fstack-protector
> > >> >> for 64bit mode, but unsupports it for 32bit mode?
> > >> >>
> > >> >>   $(cc-option -m32)                     ->  y
> > >> >>   $(cc-option -m64)                     ->  y
> > >> >>   $(cc-option -fstack-protector)        ->  y
> > >> >>   $(cc-option -m32 -fstack-protector)   ->  n
> > >> >>   $(cc-option -m64 -fstack-protector)   ->  y
> > >> >>
> > >> >> I guess this is unlikely to happen,
> > >> >> but I am not whether it is zero possibility.
> > >> >>
> > >> >> If this could happen,
> > >> >> $(cc-option ) must be evaluated together with
> > >> >> correct bi-arch option (either -m32 or -m64).
> > >> >>
> > >> >>
> > >> >> Currently, -m32/-m64 is specified in Makefile,
> > >> >> but we are moving compiler tests to Kconfig
> > >> >> and, CONFIG_64BIT can be dynamically toggled in Kconfig.
> > >> >
> > >> > I don't think it can happen for this particular combination (stack protector
> > >> > and word size), but I'm sure we'll eventually run into options that
> > >> > need to be tested in combination. For the current CFLAGS_KERNEL
> > >> > setting, we definitely have the case of needing the variables to be
> > >> > evaluated in a specific order.
> > >> >
> > >>
> > >>
> > >>
> > >>
> > >> I was thinking of how we can handle complex cases
> > >> in the current approach.
> > >>
> > >>
> > >>
> > >> (Case 1)
> > >>
> > >> Compiler flag -foo and -bar interacts, so
> > >> we also need to check the combination of the two.
> > >>
> > >>
> > >> config CC_HAS_FOO
> > >>         def_bool $(cc-option -foo)
> > >>
> > >> config CC_HAS_BAR
> > >>         def_bool $(cc-option -bar)
> > >>
> > >> config CC_HAS_FOO_WITH_BAR
> > >>         def_bool $(cc-option -foo -bar)
> > >>
> > >>
> > >>
> > >> (Case 2)
> > >> Compiler flag -foo is sensitive to word-size.
> > >> So, we need to test this option together with -m32/-m64.
> > >> User can toggle CONFIG_64BIT, like i386/x86_64.
> > >>
> > >>
> > >> config CC_NEEDS_M64
> > >>           def_bool $(cc-option -m64) && 64BIT
> > >>
> > >> config CC_NEEDS_M32
> > >>           def_bool $(cc-option -m32) && !64BIT
> > >>
> > >> config CC_HAS_FOO
> > >>          bool
> > >>          default $(cc-option -m64 -foo) if CC_NEEDS_M64
> > >>          default $(cc-option -m32 -foo) if CC_NEEDS_M32
> > >>          default $(cc-option -foo)
> > >>
> > >>
> > >>
> > >> (Case 3)
> > >> Compiler flag -foo is sensitive to endian-ness.
> > >>
> > >>
> > >> config CC_NEEDS_BIG_ENDIAN
> > >>           def_bool $(cc-option -mbig-endian) && CPU_BIG_ENDIAN
> > >>
> > >> config CC_NEEDS_LITTLE_ENDIAN
> > >>           def_bool $(cc-option -mlittle-endian) && CPU_LITTLE_ENDIAN
> > >>
> > >> config CC_HAS_FOO
> > >>          bool
> > >>          default $(cc-option -mbig-endian -foo) if CC_NEEDS_BIG_ENDIAN
> > >>          default $(cc-option -mlittle-endian -foo) if CC_NEEDS_LITTLE_ENDIAN
> > >>          default $(cc-option -foo)
> > >>
> > >>
> > >>
> > >>
> > >> Hmm, I think I can implement those somehow.
> > >> But, I hope we do not have many instances like this...
> > >>
> > >>
> > >> If you know more naive cases, please share your knowledge.
> > >>
> > >> Thanks!
> > >>
> > >>
> > >> --
> > >> Best Regards
> > >> Masahiro Yamada
> > >
> > > Would get pretty bad if a test needs to consider multiple symbols.
> > > Exponential explosion there...
> > >
> > >
> > > I thought some more about the implementation of dynamic (post-parsing)
> > > functions to see how bad it would get with the current implementation.
> > >
> > > Some background on how things work now:
> > >
> > >   1. All expression operands in Kconfig are symbols.
> > >
> > >   2. Returning '$ENV' or '$(fn foo)' as a T_WORD during parsing gets
> > >      you symbols with those strings as names and S_UNKNOWN type (because
> > >      they act like references to undefined symbols).
> > >
> > >   3. For "foo-$(fn foo)", you also get a symbol with that string as its
> > >      name and S_UNKNOWN type (stored among the SYMBOL_CONST symbols)
> > >
> > >   4. Symbols with S_UNKNOWN type get their name as their string value,
> > >      and the tristate value n.
> > >
> > > So, if you do string expansion on the names of symbols with S_UNKNOWN
> > > type in sym_calc_value(), you're almost there with the current
> > > implementation, except for the tristate case.
> > >
> > > Maybe you could set the tristate value of S_UNKNOWN symbols depending on
> > > the string value you end up with. Things are getting pretty confusing at
> > > that point.
> > >
> > > Could have something like S_DYNAMIC as well. More Kconfig complexity...
> > >
> > > Then there's other complications:
> > >
> > >   1. SYMBOL_CONST is no longer constant.
> > >
> > >   2. Dependency loop detection needs to consider symbol references
> > >      within strings.
> > >
> > >   3. Dependency loop detection relies on static knowledge of what
> > >      symbols a symbol depends on. That might get messy for certain
> > >      expansions, though it might be things you wouldn't do in practice.
> > >
> > >   4. Symbols still need to be properly invalidated. It looks like at
> > >      least menuconfig just does a dumb invalidate-everything whenever
> > >      the value of a symbol is changed though, so it might not require
> > >      extra work. (Bit messier in Kconfiglib, which does minimal
> > >      invalidation to keep scripts fast, but just need to extract a few
> > >      extra deps there.)
> > >
> > >
> > > It looks like dynamic functions could get quite messy, but might be
> > > doable if absolutely required. There's probably more devils in the
> > > details though.
> > >
> > > I don't think the static function model precludes switching models later
> > > btw, when people have more experience.
> > 
> > 
> > 
> > I really want to start with the static function model
> > and see if we need the dynamic function implementation.
> 
> Yeah, let's start with static functions, IMO.
> 
> Either we'll learn that they're powerful enough in practice, and save
> ourselves some work, or we'll gain experience for later. Converting from
> static to dynamic functions should be painless, if needed.
> 
> My plan would be something like:
> 
>   1. Implement static functions
> 
>   2. Convert as many simple cases over to them as possible
> 
>   3. See how bad the bad cases get. If they get really bad, then decide
>      what to do next (extend Kconfig, handle them in the Makefiles,
>      etc.)
> 
> > 
> > Here is an idea for the migration path in case
> > we need to do that in the future.
> > 
> > 
> > 
> > Currently, every time user input is given,
> > sym_clear_all_valid() is called.
> > 
> > It is not efficient to blindly re-evaluate expensive $(shell ...)
> 
> I think menuconfig only reevalutes the symbols in the menu that's
> currently shown in the interface (along with their dependencies).
> 
> Maybe that'd be bad enough though.
> 
> > 
> > 
> > So, have a list of symbols the function depends on
> > in its arguments.
> > 
> > For example,
> > 
> > config CC_HAS_SANE_STACKPROTECTOR
> >         def_bool $(shell $srctree/scripts/gcc-has-stack-protector.sh
> > $CC $(1), CFLAGS_BASE)
> > 
> > 
> > Here the first argument
> >   $srctree/scripts/gcc-x86-has-stack-protector.sh $CC $(1)
> > 
> > is the shell command.
> > $(1), $(2), ... will be replaced with the values of symbols (or expressions)
> > that follow when running the shell command.
> > 
> > 
> > The second argument
> > CFLAGS_BASE
> > is the dependency symbol (or expression).
> > 
> > 
> > CFLAGS_BASE can be dynamically changed like
> > 
> > config CFLAGS_BASE
> >         string
> >         default "-m64" if 64BIT
> >         default "-m32"
> > 
> > 
> > When and only when CFLAGS_BASE is updated, the function should be re-calculated.
> > (This will require efforts to minimize the amount of re-evaluation.)
> > 
> > 
> > 
> > 
> > cc-option will be implemented like follows:
> > 
> > macro cc-option $(shell $CC -Werror $$(1) $(1) -c -x c /dev/null -o
> > /dev/null, CFLAGS_BASE)
> > 
> > 
> > 
> > Please notice the difference between $$(1) and $(1).
> > 
> > $(1) is immediately expanded by cc-option macro.
> > 
> > $$(1) is escaped since we want to expand it by $(shell ...), not by
> > $(cc-option ...)
> > 
> > 
> > 
> > For example,
> > 
> >   $(cc-option -fstack-protector)
> > 
> > will be expanded to
> > 
> >   $(shell gcc -Werror $(1) -fstack-protector -c -x c /dev/null -o
> > /dev/null, CFLAGS_BASE)
> > 
> > Since macros are just textual shorthand, so this expansion happens
> > during the parse phase.
> > 
> > 
> > 
> > Then, the evaluation phase does the following every time CFLAGS_BASE is updated.
> > 
> > gcc -Werror [value of CFLAGS_BASE] -fstack-protector -c -x c /dev/null
> > -o /dev/null
> > 
> > 
> > This is a new form of expression, so it will be managed in AST tree
> > with a flag E_SHELL (or E_FUNC) etc.
> > 
> > 
> > Not implemented at all.  Just a rough sketch.
> 
> A simpler syntax like
> 
> 	$(shell $CC -Werror {CFLAGS_BASE} -c -x c /dev/null -o /dev/null)

	*$(shell $CC -Werror {CFLAGS_BASE} -fstack-protector -c -x c /dev/null -o /dev/null)