public inbox for kdevops@lists.linux.dev
 help / color / mirror / Atom feed
* [PATCH RFC 0/6] Generate Kconfig environment in Ansible
@ 2025-03-28 20:56 Daniel Gomez
  2025-03-28 20:56 ` [PATCH RFC 1/6] merge_config: add fragment support from kernel Daniel Gomez
                   ` (5 more replies)
  0 siblings, 6 replies; 16+ messages in thread
From: Daniel Gomez @ 2025-03-28 20:56 UTC (permalink / raw)
  To: Luis Chamberlain; +Cc: kdevops, Daniel Gomez, Daniel Gomez

This is an effort to replace the scripts/* we use from Kconfig to
retrieve any possible Kconfig value (y/n/string) with Ansible. I believe
this has been discussed a few times on the mailing list, perhaps as a
potential idea. This aims to explore whether this approach makes sense
and what are the options.

The new workflow is:
* Run make menuconfig
* menuconfig runs kconfig-env target
* kconfig-env target calls Ansible kconfig.yml playbook
* kconfig.yml playbook generates a Kconfig fragment with the environment
(.env.config)
* .env.config is merged with .config
* make olddefconfig is run to generate a valid .config for menuconfig
* menuconfig is shown with the environment generated variables

While this adds new targets, it also removes the $shell calls in
Kconfig. However, when merging fragments, it appears that hidden symbols
cannot be merged and updated in .config, unless these are visible (and
therefore editable). So for this reason, a new Environment menu has
been created to list all these new vars. Any ideas/suggestion to keep
them invisible?

Note that only CONFIG_TOPDIR_PATH_SHA256SUM= is generated with Ansible
and added/merged to the Kconfig environment menu vars.

Thoughts?

Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
---
Daniel Gomez (6):
      merge_config: add fragment support from kernel
      scripts/kconfig/kconfig.Makefile: extend simple-targets
      kconfig-fragments: add docs and fragments folder
      kconfig-env: generate kconfig environment
      kconfig: env: add support
      playbooks: add kconfig support

 .gitignore                             |   2 +
 Kconfig                                |   9 +-
 Makefile                               |   1 +
 configs/callback-debug.config          |   2 +
 configs/callback-dense.config          |   2 +
 docs/kconfig-fragments.md              |  19 +++
 kconfigs/Kconfig.env                   |   5 +
 playbooks/kconfig.yml                  |   6 +
 playbooks/roles/kconfig/tasks/main.yml |  15 +++
 scripts/kconfig-env.Makefile           |  14 +++
 scripts/kconfig/Makefile               |   3 +
 scripts/kconfig/kconfig.Makefile       |   6 +-
 scripts/kconfig/merge_config.sh        | 213 +++++++++++++++++++++++++++++++++
 13 files changed, 289 insertions(+), 8 deletions(-)
---
base-commit: f83ea7595cc531e1c5413ffdf7f82714fc8ab7f0
change-id: 20250328-kconfig-env-with-ansible-33e3c3b3ae97

Best regards,
-- 
Daniel Gomez <da.gomez@samsung.com>


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [PATCH RFC 1/6] merge_config: add fragment support from kernel
  2025-03-28 20:56 [PATCH RFC 0/6] Generate Kconfig environment in Ansible Daniel Gomez
@ 2025-03-28 20:56 ` Daniel Gomez
  2025-03-28 20:56 ` [PATCH RFC 2/6] scripts/kconfig/kconfig.Makefile: extend simple-targets Daniel Gomez
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 16+ messages in thread
From: Daniel Gomez @ 2025-03-28 20:56 UTC (permalink / raw)
  To: Luis Chamberlain; +Cc: kdevops, Daniel Gomez, Daniel Gomez

From: Daniel Gomez <da.gomez@samsung.com>

Add merge_config.sh script support from upstream Linux kernel.

This allows to use fragments in kdevops.

Example: A fragment to enable fstests is called fstests.config. To
enable it while keeping the same config, we can simply do:

./scripts/kconfig/merge_config.sh -m .config fstests.config
make olddefconfig

Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
---
 scripts/kconfig/merge_config.sh | 213 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 213 insertions(+)

diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
new file mode 100755
index 0000000000000000000000000000000000000000..ad35a60de350ae1c5b60d39bf752115d27276f52
--- /dev/null
+++ b/scripts/kconfig/merge_config.sh
@@ -0,0 +1,213 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+#
+#  merge_config.sh - Takes a list of config fragment values, and merges
+#  them one by one. Provides warnings on overridden values, and specified
+#  values that did not make it to the resulting .config file (due to missed
+#  dependencies or config symbol removal).
+#
+#  Portions reused from kconf_check and generate_cfg:
+#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/kconf_check
+#  http://git.yoctoproject.org/cgit/cgit.cgi/yocto-kernel-tools/tree/tools/generate_cfg
+#
+#  Copyright (c) 2009-2010 Wind River Systems, Inc.
+#  Copyright 2011 Linaro
+
+set -e
+
+clean_up() {
+	rm -f $TMP_FILE
+	rm -f $MERGE_FILE
+}
+
+usage() {
+	echo "Usage: $0 [OPTIONS] [CONFIG [...]]"
+	echo "  -h    display this help text"
+	echo "  -m    only merge the fragments, do not execute the make command"
+	echo "  -n    use allnoconfig instead of alldefconfig"
+	echo "  -r    list redundant entries when merging fragments"
+	echo "  -y    make builtin have precedence over modules"
+	echo "  -O    dir to put generated output files.  Consider setting \$KCONFIG_CONFIG instead."
+	echo "  -s    strict mode. Fail if the fragment redefines any value."
+	echo "  -Q    disable warning messages for overridden options."
+	echo
+	echo "Used prefix: '$CONFIG_PREFIX'. You can redefine it with \$CONFIG_ environment variable."
+}
+
+RUNMAKE=true
+ALLTARGET=alldefconfig
+WARNREDUN=false
+BUILTIN=false
+OUTPUT=.
+STRICT=false
+CONFIG_PREFIX=${CONFIG_-CONFIG_}
+WARNOVERRIDE=echo
+
+while true; do
+	case $1 in
+	"-n")
+		ALLTARGET=allnoconfig
+		shift
+		continue
+		;;
+	"-m")
+		RUNMAKE=false
+		shift
+		continue
+		;;
+	"-h")
+		usage
+		exit
+		;;
+	"-r")
+		WARNREDUN=true
+		shift
+		continue
+		;;
+	"-y")
+		BUILTIN=true
+		shift
+		continue
+		;;
+	"-O")
+		if [ -d $2 ];then
+			OUTPUT=$(echo $2 | sed 's/\/*$//')
+		else
+			echo "output directory $2 does not exist" 1>&2
+			exit 1
+		fi
+		shift 2
+		continue
+		;;
+	"-s")
+		STRICT=true
+		shift
+		continue
+		;;
+	"-Q")
+		WARNOVERRIDE=true
+		shift
+		continue
+		;;
+	*)
+		break
+		;;
+	esac
+done
+
+if [ "$#" -lt 1 ] ; then
+	usage
+	exit
+fi
+
+if [ -z "$KCONFIG_CONFIG" ]; then
+	if [ "$OUTPUT" != . ]; then
+		KCONFIG_CONFIG=$(readlink -m -- "$OUTPUT/.config")
+	else
+		KCONFIG_CONFIG=.config
+	fi
+fi
+
+INITFILE=$1
+shift;
+
+if [ ! -r "$INITFILE" ]; then
+	echo "The base file '$INITFILE' does not exist. Creating one..." >&2
+	touch $INITFILE
+fi
+
+MERGE_LIST=$*
+SED_CONFIG_EXP1="s/^\(${CONFIG_PREFIX}[a-zA-Z0-9_]*\)=.*/\1/p"
+SED_CONFIG_EXP2="s/^# \(${CONFIG_PREFIX}[a-zA-Z0-9_]*\) is not set$/\1/p"
+
+TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
+MERGE_FILE=$(mktemp ./.merge_tmp.config.XXXXXXXXXX)
+
+echo "Using $INITFILE as base"
+
+trap clean_up EXIT
+
+cat $INITFILE > $TMP_FILE
+
+# Merge files, printing warnings on overridden values
+for ORIG_MERGE_FILE in $MERGE_LIST ; do
+	echo "Merging $ORIG_MERGE_FILE"
+	if [ ! -r "$ORIG_MERGE_FILE" ]; then
+		echo "The merge file '$ORIG_MERGE_FILE' does not exist.  Exit." >&2
+		exit 1
+	fi
+	cat $ORIG_MERGE_FILE > $MERGE_FILE
+	CFG_LIST=$(sed -n -e "$SED_CONFIG_EXP1" -e "$SED_CONFIG_EXP2" $MERGE_FILE)
+
+	for CFG in $CFG_LIST ; do
+		grep -q -w $CFG $TMP_FILE || continue
+		PREV_VAL=$(grep -w $CFG $TMP_FILE)
+		NEW_VAL=$(grep -w $CFG $MERGE_FILE)
+		BUILTIN_FLAG=false
+		if [ "$BUILTIN" = "true" ] && [ "${NEW_VAL#CONFIG_*=}" = "m" ] && [ "${PREV_VAL#CONFIG_*=}" = "y" ]; then
+			${WARNOVERRIDE} Previous  value: $PREV_VAL
+			${WARNOVERRIDE} New value:       $NEW_VAL
+			${WARNOVERRIDE} -y passed, will not demote y to m
+			${WARNOVERRIDE}
+			BUILTIN_FLAG=true
+		elif [ "x$PREV_VAL" != "x$NEW_VAL" ] ; then
+			${WARNOVERRIDE} Value of $CFG is redefined by fragment $ORIG_MERGE_FILE:
+			${WARNOVERRIDE} Previous  value: $PREV_VAL
+			${WARNOVERRIDE} New value:       $NEW_VAL
+			${WARNOVERRIDE}
+			if [ "$STRICT" = "true" ]; then
+				STRICT_MODE_VIOLATED=true
+			fi
+		elif [ "$WARNREDUN" = "true" ]; then
+			${WARNOVERRIDE} Value of $CFG is redundant by fragment $ORIG_MERGE_FILE:
+		fi
+		if [ "$BUILTIN_FLAG" = "false" ]; then
+			sed -i "/$CFG[ =]/d" $TMP_FILE
+		else
+			sed -i "/$CFG[ =]/d" $MERGE_FILE
+		fi
+	done
+	# In case the previous file lacks a new line at the end
+	echo >> $TMP_FILE
+	cat $MERGE_FILE >> $TMP_FILE
+done
+
+if [ "$STRICT_MODE_VIOLATED" = "true" ]; then
+	echo "The fragment redefined a value and strict mode had been passed."
+	exit 1
+fi
+
+if [ "$RUNMAKE" = "false" ]; then
+	cp -T -- "$TMP_FILE" "$KCONFIG_CONFIG"
+	echo "#"
+	echo "# merged configuration written to $KCONFIG_CONFIG (needs make)"
+	echo "#"
+	exit
+fi
+
+# If we have an output dir, setup the O= argument, otherwise leave
+# it blank, since O=. will create an unnecessary ./source softlink
+OUTPUT_ARG=""
+if [ "$OUTPUT" != "." ] ; then
+	OUTPUT_ARG="O=$OUTPUT"
+fi
+
+
+# Use the merged file as the starting point for:
+# alldefconfig: Fills in any missing symbols with Kconfig default
+# allnoconfig: Fills in any missing symbols with # CONFIG_* is not set
+make KCONFIG_ALLCONFIG=$TMP_FILE $OUTPUT_ARG $ALLTARGET
+
+
+# Check all specified config values took (might have missed-dependency issues)
+for CFG in $(sed -n -e "$SED_CONFIG_EXP1" -e "$SED_CONFIG_EXP2" $TMP_FILE); do
+
+	REQUESTED_VAL=$(grep -w -e "$CFG" $TMP_FILE)
+	ACTUAL_VAL=$(grep -w -e "$CFG" "$KCONFIG_CONFIG" || true)
+	if [ "x$REQUESTED_VAL" != "x$ACTUAL_VAL" ] ; then
+		echo "Value requested for $CFG not in final .config"
+		echo "Requested value:  $REQUESTED_VAL"
+		echo "Actual value:     $ACTUAL_VAL"
+		echo ""
+	fi
+done

-- 
2.47.2


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [PATCH RFC 2/6] scripts/kconfig/kconfig.Makefile: extend simple-targets
  2025-03-28 20:56 [PATCH RFC 0/6] Generate Kconfig environment in Ansible Daniel Gomez
  2025-03-28 20:56 ` [PATCH RFC 1/6] merge_config: add fragment support from kernel Daniel Gomez
@ 2025-03-28 20:56 ` Daniel Gomez
  2025-03-28 20:56 ` [PATCH RFC 3/6] kconfig-fragments: add docs and fragments folder Daniel Gomez
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 16+ messages in thread
From: Daniel Gomez @ 2025-03-28 20:56 UTC (permalink / raw)
  To: Luis Chamberlain; +Cc: kdevops, Daniel Gomez, Daniel Gomez

From: Daniel Gomez <da.gomez@samsung.com>

Add oldconfig and olddefconfig targets.

Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
---
 scripts/kconfig/Makefile         | 3 +++
 scripts/kconfig/kconfig.Makefile | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/kconfig/Makefile b/scripts/kconfig/Makefile
index e3694365bf98281a00fb56d76d3400d236776476..7cef3498e4c9e751c05190fb3da7507587d9ad84 100644
--- a/scripts/kconfig/Makefile
+++ b/scripts/kconfig/Makefile
@@ -69,6 +69,9 @@ help:
 	@echo "allyesconfig       - enables all bells and whistles"
 	@echo "allnoconfig        - disables all bells and whistles"
 	@echo "randconfig         - random configuration"
+	@echo "oldconfig          - update a configuration using provided .config as base"
+	@echo "olddefconfig       - Same as oldconfig but sets new symbosl to their default"
+
 	@echo "defconfig-*        - If you have files in the defconfig directory use default config from there"
 
 .PHONY: clean
diff --git a/scripts/kconfig/kconfig.Makefile b/scripts/kconfig/kconfig.Makefile
index e227ea4233a7e790e90f38c1817c745a0c805132..cc24c7ff3e1c23487aec767075c69fdf3876b39d 100644
--- a/scripts/kconfig/kconfig.Makefile
+++ b/scripts/kconfig/kconfig.Makefile
@@ -51,7 +51,7 @@ $(KCONFIG_DIR)/conf:
 
 # More are supported, however we only list the ones tested on this top
 # level Makefile.
-simple-targets := allnoconfig allyesconfig alldefconfig randconfig
+simple-targets := allnoconfig allyesconfig alldefconfig randconfig oldconfig olddefconfig
 PHONY += $(simple-targets)
 
 $(simple-targets): $(KCONFIG_DIR)/conf Kconfig

-- 
2.47.2


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [PATCH RFC 3/6] kconfig-fragments: add docs and fragments folder
  2025-03-28 20:56 [PATCH RFC 0/6] Generate Kconfig environment in Ansible Daniel Gomez
  2025-03-28 20:56 ` [PATCH RFC 1/6] merge_config: add fragment support from kernel Daniel Gomez
  2025-03-28 20:56 ` [PATCH RFC 2/6] scripts/kconfig/kconfig.Makefile: extend simple-targets Daniel Gomez
@ 2025-03-28 20:56 ` Daniel Gomez
  2025-03-28 20:56 ` [PATCH RFC 4/6] kconfig-env: generate kconfig environment Daniel Gomez
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 16+ messages in thread
From: Daniel Gomez @ 2025-03-28 20:56 UTC (permalink / raw)
  To: Luis Chamberlain; +Cc: kdevops, Daniel Gomez, Daniel Gomez

From: Daniel Gomez <da.gomez@samsung.com>

Add configs/ directory to hold kdevops fragments.
Add docs/kconfig-fragments explaining the fragment workflow.

Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
---
 configs/callback-debug.config |  2 ++
 configs/callback-dense.config |  2 ++
 docs/kconfig-fragments.md     | 19 +++++++++++++++++++
 3 files changed, 23 insertions(+)

diff --git a/configs/callback-debug.config b/configs/callback-debug.config
new file mode 100644
index 0000000000000000000000000000000000000000..880da5fcb4ec60a7ee04490f21ea2a7efa240926
--- /dev/null
+++ b/configs/callback-debug.config
@@ -0,0 +1,2 @@
+CONFIG_ANSIBLE_CFG_CALLBACK_PLUGIN_DENSE=n
+CONFIG_ANSIBLE_CFG_CALLBACK_PLUGIN_DEBUG=y
diff --git a/configs/callback-dense.config b/configs/callback-dense.config
new file mode 100644
index 0000000000000000000000000000000000000000..c59925c4bd30634633be322c9e3b24bb91d2d25b
--- /dev/null
+++ b/configs/callback-dense.config
@@ -0,0 +1,2 @@
+CONFIG_ANSIBLE_CFG_CALLBACK_PLUGIN_DENSE=y
+CONFIG_ANSIBLE_CFG_CALLBACK_PLUGIN_DEBUG=n
diff --git a/docs/kconfig-fragments.md b/docs/kconfig-fragments.md
new file mode 100644
index 0000000000000000000000000000000000000000..969a7289b8be10fbb9eb2fb96d3a28be7b5db39d
--- /dev/null
+++ b/docs/kconfig-fragments.md
@@ -0,0 +1,19 @@
+# Kconfig Fragments
+
+kdevops fragments are supported and can be used with the `scripts/kconfig/merge_config.sh` script.
+
+Workflow:
+
+```sh
+./scripts/kconfig/merge_config.sh \
+-m <INITCONF> \
+<FRAGMENT_LIST>
+```
+
+Example:
+
+```sh
+./scripts/kconfig/merge_config.sh \
+-m .config \
+callback-dense.config
+```

-- 
2.47.2


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [PATCH RFC 4/6] kconfig-env: generate kconfig environment
  2025-03-28 20:56 [PATCH RFC 0/6] Generate Kconfig environment in Ansible Daniel Gomez
                   ` (2 preceding siblings ...)
  2025-03-28 20:56 ` [PATCH RFC 3/6] kconfig-fragments: add docs and fragments folder Daniel Gomez
@ 2025-03-28 20:56 ` Daniel Gomez
  2025-03-28 20:56 ` [PATCH RFC 5/6] kconfig: env: add support Daniel Gomez
  2025-03-28 20:56 ` [PATCH RFC 6/6] playbooks: add kconfig support Daniel Gomez
  5 siblings, 0 replies; 16+ messages in thread
From: Daniel Gomez @ 2025-03-28 20:56 UTC (permalink / raw)
  To: Luis Chamberlain; +Cc: kdevops, Daniel Gomez, Daniel Gomez

From: Daniel Gomez <da.gomez@samsung.com>

When menuconfig target is run, generate first the kconfig environment
and merge with current .config.

Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
---
 Makefile                         |  1 +
 scripts/kconfig-env.Makefile     | 14 ++++++++++++++
 scripts/kconfig/kconfig.Makefile |  4 ++--
 3 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index 5ee7db7d988b9271fed7cb470f807bb9e9ce3cb6..1f939c52284f218a23430a7c5350d7827cb1b905 100644
--- a/Makefile
+++ b/Makefile
@@ -21,6 +21,7 @@ export KDEVOPS_VAGRANT :=
 export PYTHONUNBUFFERED=1
 export TOPDIR=./
 export TOPDIR_PATH = $(shell readlink -f $(TOPDIR))
+include scripts/kconfig-env.Makefile
 include scripts/refs.Makefile
 
 KDEVOPS_NODES_ROLE_TEMPLATE_DIR :=		$(KDEVOPS_PLAYBOOKS_DIR)/roles/gen_nodes/templates
diff --git a/scripts/kconfig-env.Makefile b/scripts/kconfig-env.Makefile
new file mode 100644
index 0000000000000000000000000000000000000000..dc2b9f2a65001ba8698a61bfb20b43b366e3d92a
--- /dev/null
+++ b/scripts/kconfig-env.Makefile
@@ -0,0 +1,14 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+PHONY += kconfig-env
+kconfig-env:
+	$(Q)ANSIBLE_STDOUT_CALLBACK=null ansible-playbook $(ANSIBLE_VERBOSE) --connection=local \
+		--inventory localhost, \
+		playbooks/kconfig.yml \
+		-e 'ansible_python_interpreter=/usr/bin/python3' \
+		--extra-vars "topdir_path=$(TOPDIR_PATH)"
+	$(Q)$(TOPDIR_PATH)/scripts/kconfig/merge_config.sh -m .config \
+	$(TOPDIR_PATH)/.env.config
+	$(Q)$(MAKE) -C $(TOPDIR_PATH) olddefconfig
+
+.PHONY = $(PHONY)
diff --git a/scripts/kconfig/kconfig.Makefile b/scripts/kconfig/kconfig.Makefile
index cc24c7ff3e1c23487aec767075c69fdf3876b39d..19ed56b01db5817dc1e8b7a4db6fa726d87d8065 100644
--- a/scripts/kconfig/kconfig.Makefile
+++ b/scripts/kconfig/kconfig.Makefile
@@ -36,8 +36,8 @@ $(KCONFIG_DIR)/mconf:
 	$(MAKE) -C $(KCONFIG_DIR)/ mconf
 
 PHONY += menuconfig
-menuconfig: $(KCONFIG_DIR)/mconf include/config/project.release Kconfig
-	@$< Kconfig
+menuconfig: $(KCONFIG_DIR)/mconf include/config/project.release Kconfig kconfig-env
+	$< Kconfig
 
 $(KCONFIG_DIR)/nconf:
 	$(MAKE) -C $(KCONFIG_DIR)/ nconf

-- 
2.47.2


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [PATCH RFC 5/6] kconfig: env: add support
  2025-03-28 20:56 [PATCH RFC 0/6] Generate Kconfig environment in Ansible Daniel Gomez
                   ` (3 preceding siblings ...)
  2025-03-28 20:56 ` [PATCH RFC 4/6] kconfig-env: generate kconfig environment Daniel Gomez
@ 2025-03-28 20:56 ` Daniel Gomez
  2025-03-28 20:56 ` [PATCH RFC 6/6] playbooks: add kconfig support Daniel Gomez
  5 siblings, 0 replies; 16+ messages in thread
From: Daniel Gomez @ 2025-03-28 20:56 UTC (permalink / raw)
  To: Luis Chamberlain; +Cc: kdevops, Daniel Gomez, Daniel Gomez

From: Daniel Gomez <da.gomez@samsung.com>

Updating Kconfig with a fragment requires symbols to be visible. Add an
environment submenu to display all generated variables.

Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
---
 Kconfig              | 9 ++++-----
 kconfigs/Kconfig.env | 5 +++++
 2 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/Kconfig b/Kconfig
index 357341be108de3dab79883f9a6babd29547d62de..6877b7aebd5057a925b371d8c6afd9ad568faa6e 100644
--- a/Kconfig
+++ b/Kconfig
@@ -15,11 +15,6 @@ config TOPDIR_PATH_HAS_SHA256SUM
 	output yaml
 	default y
 
-config TOPDIR_PATH_SHA256SUM
-	string
-	output yaml
-	default $(shell, ./scripts/compute_sha256sum.sh $(TOPDIR_PATH))
-
 config HAVE_KDEVOPS_CUSTOM_DEFAULTS
 	bool
 	default n
@@ -81,3 +76,7 @@ endmenu
 menu "Kdevops configuration"
 source "kconfigs/Kconfig.kdevops"
 endmenu
+
+menu "Environment"
+source "kconfigs/Kconfig.env"
+endmenu
diff --git a/kconfigs/Kconfig.env b/kconfigs/Kconfig.env
new file mode 100644
index 0000000000000000000000000000000000000000..400643f9a304556f27a63792239a383be51866c3
--- /dev/null
+++ b/kconfigs/Kconfig.env
@@ -0,0 +1,5 @@
+config TOPDIR_PATH_SHA256SUM
+	string "TOPDIR_PATH checksum"
+	output yaml
+	help
+	  TOPDIR_PATH checksum

-- 
2.47.2


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [PATCH RFC 6/6] playbooks: add kconfig support
  2025-03-28 20:56 [PATCH RFC 0/6] Generate Kconfig environment in Ansible Daniel Gomez
                   ` (4 preceding siblings ...)
  2025-03-28 20:56 ` [PATCH RFC 5/6] kconfig: env: add support Daniel Gomez
@ 2025-03-28 20:56 ` Daniel Gomez
  2025-03-28 22:58   ` Luis Chamberlain
  5 siblings, 1 reply; 16+ messages in thread
From: Daniel Gomez @ 2025-03-28 20:56 UTC (permalink / raw)
  To: Luis Chamberlain; +Cc: kdevops, Daniel Gomez, Daniel Gomez

From: Daniel Gomez <da.gomez@samsung.com>

Add kconfig playbook to generate Kconfig environment with Ansible.

The output will be a Kconfig fragment (.env.config) that will be merged
with existing configuration file.

Add .env.config to the list of git-ignored files.

Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
---
 .gitignore                             |  2 ++
 playbooks/kconfig.yml                  |  6 ++++++
 playbooks/roles/kconfig/tasks/main.yml | 15 +++++++++++++++
 3 files changed, 23 insertions(+)

diff --git a/.gitignore b/.gitignore
index 5ddfcc45e544ddb09ac81807e84fe88b329aec88..2fd102276301a2ce159e1c475f88ad464e2dc735 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,6 +2,8 @@
 *.o
 *.retry
 
+.env.config
+
 .kdevops\.depcheck
 .provisioned_once*
 
diff --git a/playbooks/kconfig.yml b/playbooks/kconfig.yml
new file mode 100644
index 0000000000000000000000000000000000000000..4fa725c9046282a32d9a57c72c637194f3ad2335
--- /dev/null
+++ b/playbooks/kconfig.yml
@@ -0,0 +1,6 @@
+---
+- name: Kconfig
+  hosts: localhost
+  gather_facts: no
+  roles:
+    - role: kconfig
diff --git a/playbooks/roles/kconfig/tasks/main.yml b/playbooks/roles/kconfig/tasks/main.yml
new file mode 100644
index 0000000000000000000000000000000000000000..36e92076c3a63d617f8df6b66dae9d2925800208
--- /dev/null
+++ b/playbooks/roles/kconfig/tasks/main.yml
@@ -0,0 +1,15 @@
+---
+- name: Checksum
+  ansible.builtin.shell: |
+    set -o pipefail
+    echo "{{ topdir_path }}" | sha256sum | awk '{print $1}'
+  register: kconfig_checksum
+  changed_when: false
+
+- name: Create env.config
+  ansible.builtin.copy:
+    dest: "{{ topdir_path }}/.env.config"
+    mode: "0755"
+    content: |
+      CONFIG_TOPDIR_PATH_SHA256SUM="{{ kconfig_checksum.stdout }}"
+  changed_when: true

-- 
2.47.2


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: [PATCH RFC 6/6] playbooks: add kconfig support
  2025-03-28 20:56 ` [PATCH RFC 6/6] playbooks: add kconfig support Daniel Gomez
@ 2025-03-28 22:58   ` Luis Chamberlain
  2025-03-31  6:54     ` Daniel Gomez
  0 siblings, 1 reply; 16+ messages in thread
From: Luis Chamberlain @ 2025-03-28 22:58 UTC (permalink / raw)
  To: Daniel Gomez, Masahiro Yamada; +Cc: kdevops, Daniel Gomez

On Fri, Mar 28, 2025 at 08:56:19PM +0000, Daniel Gomez wrote:
> From: Daniel Gomez <da.gomez@samsung.com>
> 
> Add kconfig playbook to generate Kconfig environment with Ansible.
> 
> The output will be a Kconfig fragment (.env.config) that will be merged
> with existing configuration file.
> 
> Add .env.config to the list of git-ignored files.
> 
> Signed-off-by: Daniel Gomez <da.gomez@samsung.com>

I think I get what this is doing and this seems pretty awesome!
Essentially we replace the kdevops Kconfigs which rely on scripts
with ansible calls which can rely on jinja2 template parsing on the
fly. This works especially well for use given we alreayd process
config symbols to yaml entries and always read them for ansible
playbooks.

While we could explore adapting this for complex scripts we use
like the ones which try to check your existing libvirt pools,
a simple jinja2 processing trick might be nicer to help us scale
with more symbols, otherwise kconfig will be rather slow. So,
any chance we might be able to leverage some regular python jinja2
thing which perhaps upstream Linux can also embrace?

Also I think it makes sense to combine this and patch #5, and also
you forgot to remove the script this is replacing?

  Luis

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH RFC 6/6] playbooks: add kconfig support
  2025-03-28 22:58   ` Luis Chamberlain
@ 2025-03-31  6:54     ` Daniel Gomez
  2025-03-31  7:32       ` Luis Chamberlain
  0 siblings, 1 reply; 16+ messages in thread
From: Daniel Gomez @ 2025-03-31  6:54 UTC (permalink / raw)
  To: Luis Chamberlain; +Cc: Masahiro Yamada, kdevops, Daniel Gomez

On Fri, Mar 28, 2025 at 03:58:50PM +0100, Luis Chamberlain wrote:
> On Fri, Mar 28, 2025 at 08:56:19PM +0000, Daniel Gomez wrote:
> > From: Daniel Gomez <da.gomez@samsung.com>
> > 
> > Add kconfig playbook to generate Kconfig environment with Ansible.
> > 
> > The output will be a Kconfig fragment (.env.config) that will be merged
> > with existing configuration file.
> > 
> > Add .env.config to the list of git-ignored files.
> > 
> > Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
> 
> I think I get what this is doing and this seems pretty awesome!
> Essentially we replace the kdevops Kconfigs which rely on scripts
> with ansible calls

Exactly.

> which can rely on jinja2 template parsing on the
> fly.

The patch still does require templates as it's just mapping
'CONFIG_VARIABLE={{ ansible_generated_variable }}' in the last task ("Create
env.config").

> This works especially well for use given we alreayd process
> config symbols to yaml entries and always read them for ansible
> playbooks.

You're right!

Also, generating CONFIG_TOPDIR_PATH_SHA256SUM in a Kconfig fragment doesn't make
sense since we don't use it in Kconfig. So, we likely need two output files:

1. An Ansible YAML file for extra-vars
2. A Kconfig fragment for Kconfig dependencies (or user modifications)

Some environment variables (like those in Kconfig.distro) will go into the
Kconfig fragment, while others can be used directly in Ansible. I'm just not
sure yet about the ratio.

I think merging all generated variables into Kconfig makes this a bit easier as
we don't have to deal with another Ansible YAML file. Ideally, we would merge
this new YAML with the Kconfig YAML output. Can this be done? Makes sense?

> 
> While we could explore adapting this for complex scripts we use
> like the ones which try to check your existing libvirt pools,
> a simple jinja2 processing trick might be nicer to help us scale
> with more symbols, otherwise kconfig will be rather slow. So,

I agree. However, we may need to expose some symbols to Kconfig anyway, if they
are used as dependency symbols or the user wants to change them like the libvirt
pools.

> any chance we might be able to leverage some regular python jinja2
> thing which perhaps upstream Linux can also embrace?

Do you mean upstream (Linux) Kconfig?

I'll look into the libvirt pool scripts and see how can they be integrated in
Ansible.

> 
> Also I think it makes sense to combine this and patch #5, and also
> you forgot to remove the script this is replacing?

That's right. I'll do this in the next iteration.

> 
>   Luis

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH RFC 6/6] playbooks: add kconfig support
  2025-03-31  6:54     ` Daniel Gomez
@ 2025-03-31  7:32       ` Luis Chamberlain
  2025-03-31 18:31         ` Daniel Gomez
  0 siblings, 1 reply; 16+ messages in thread
From: Luis Chamberlain @ 2025-03-31  7:32 UTC (permalink / raw)
  To: Daniel Gomez; +Cc: Masahiro Yamada, kdevops, Daniel Gomez

On Mon, Mar 31, 2025 at 08:54:59AM +0200, Daniel Gomez wrote:
> On Fri, Mar 28, 2025 at 03:58:50PM +0100, Luis Chamberlain wrote:
> > On Fri, Mar 28, 2025 at 08:56:19PM +0000, Daniel Gomez wrote:
> > > From: Daniel Gomez <da.gomez@samsung.com>
> > > 
> > > Add kconfig playbook to generate Kconfig environment with Ansible.
> > > 
> > > The output will be a Kconfig fragment (.env.config) that will be merged
> > > with existing configuration file.
> > > 
> > > Add .env.config to the list of git-ignored files.
> > > 
> > > Signed-off-by: Daniel Gomez <da.gomez@samsung.com>
> > 
> > I think I get what this is doing and this seems pretty awesome!
> > Essentially we replace the kdevops Kconfigs which rely on scripts
> > with ansible calls
> 
> Exactly.
> 
> > which can rely on jinja2 template parsing on the
> > fly.
> 
> The patch still does require templates as it's just mapping
> 'CONFIG_VARIABLE={{ ansible_generated_variable }}' in the last task ("Create
> env.config").
> 
> > This works especially well for use given we alreayd process
> > config symbols to yaml entries and always read them for ansible
> > playbooks.
> 
> You're right!
> 
> Also, generating CONFIG_TOPDIR_PATH_SHA256SUM in a Kconfig fragment doesn't make
> sense since we don't use it in Kconfig. So, we likely need two output files:
> 
> 1. An Ansible YAML file for extra-vars

Oh I see what you're saying, so Kconfig simply invokes the *need* for this,
and so we don't need to actually get Kconfig to really call anything fo
us. So we want to instruct essentially Kconfig we don't need this for
.config but want this as part of a secondary effort to just ammend our
yaml file, somehow.

> 2. A Kconfig fragment for Kconfig dependencies (or user modifications)

Makes sense, these can however easily go out of sync so we'll need a
sanity checker pre-commit or something later.

> Some environment variables (like those in Kconfig.distro) will go into the
> Kconfig fragment, while others can be used directly in Ansible. I'm just not
> sure yet about the ratio.

No rush, was with optional output yaml, we can start with just 1 and scale
later.

> I think merging all generated variables into Kconfig makes this a bit easier as
> we don't have to deal with another Ansible YAML file. Ideally, we would merge
> this new YAML with the Kconfig YAML output. Can this be done? Makes sense?

Yes. We have:

Kconfig -->
.config --> optional output yaml --> 1st.yaml
        |
	|
	v------- what we need -----> 2nd.yaml
	|
	|
	v--------------------------> extra_vars.yaml
	

Is that right?

> > While we could explore adapting this for complex scripts we use
> > like the ones which try to check your existing libvirt pools,
> > a simple jinja2 processing trick might be nicer to help us scale
> > with more symbols, otherwise kconfig will be rather slow. So,
> 
> I agree. However, we may need to expose some symbols to Kconfig anyway, if they
> are used as dependency symbols or the user wants to change them like the libvirt
> pools.

Well, we already support dynconfig for dynamic Kconfig generation for
really slow automatic Kconfig file writing which is just optional, but
indeed, we have many things in the middle, things we *need* to run on Kconfig,
inferences. So yeah I do agree, there is a middle ground.

> > any chance we might be able to leverage some regular python jinja2
> > thing which perhaps upstream Linux can also embrace?
> 
> Do you mean upstream (Linux) Kconfig?

Yup.

For instance, the challenge there is how can we get to say support
different compiler options so that if you support different set you
get to pick the one you want, and we use generic language to express
this. This allows us to use a .config to *use* a specific target
compiler with a .config to reproduce a build with without having to use
variables on the command line.

I think we've been pushing Kconfig to its limit on kdevops for fancy
stuff so it may make sense for us to leverage what we can to see if
we can help with this endeavor upstream.

  Luis

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH RFC 6/6] playbooks: add kconfig support
  2025-03-31  7:32       ` Luis Chamberlain
@ 2025-03-31 18:31         ` Daniel Gomez
  2025-04-01 21:02           ` Luis Chamberlain
  0 siblings, 1 reply; 16+ messages in thread
From: Daniel Gomez @ 2025-03-31 18:31 UTC (permalink / raw)
  To: Luis Chamberlain; +Cc: Masahiro Yamada, kdevops, Daniel Gomez

On Mon, Mar 31, 2025 at 12:32:29AM +0100, Luis Chamberlain wrote:
> On Mon, Mar 31, 2025 at 08:54:59AM +0200, Daniel Gomez wrote:
> > On Fri, Mar 28, 2025 at 03:58:50PM +0100, Luis Chamberlain wrote:
> > > On Fri, Mar 28, 2025 at 08:56:19PM +0000, Daniel Gomez wrote:
> > > > From: Daniel Gomez <da.gomez@samsung.com>

> > I think merging all generated variables into Kconfig makes this a bit easier as
> > we don't have to deal with another Ansible YAML file. Ideally, we would merge
> > this new YAML with the Kconfig YAML output. Can this be done? Makes sense?
> 
> Yes. We have:
> 
> Kconfig -->
> .config --> optional output yaml --> 1st.yaml
>         |
> 	|
> 	v------- what we need -----> 2nd.yaml
> 	|
> 	|
> 	v--------------------------> extra_vars.yaml
> 	
> 
> Is that right?

I was thinking this instead:

What we have (please, correct if it's wrong):

Kconfig --> .config
        --> .extra_vars_auto.yaml --> Makefile --> extra_vars.yaml --> Ansible

What I'm suggesting above:

Makefile --> Ansible --> env.yaml                                                                                             |
                     --> env.config (Kconfig fragment) --> Kconfig --> .config                                                v
                                                                   --> .extra_vars_auto.yaml --> Makefile --> extra_vars.yaml --> Ansible

But looking at this, do you see a way to simplify this?

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH RFC 6/6] playbooks: add kconfig support
  2025-03-31 18:31         ` Daniel Gomez
@ 2025-04-01 21:02           ` Luis Chamberlain
  2025-04-04 20:33             ` Daniel Gomez
  0 siblings, 1 reply; 16+ messages in thread
From: Luis Chamberlain @ 2025-04-01 21:02 UTC (permalink / raw)
  To: Daniel Gomez; +Cc: Masahiro Yamada, kdevops, Daniel Gomez

On Mon, Mar 31, 2025 at 08:31:55PM +0200, Daniel Gomez wrote:
> On Mon, Mar 31, 2025 at 12:32:29AM +0100, Luis Chamberlain wrote:
> > On Mon, Mar 31, 2025 at 08:54:59AM +0200, Daniel Gomez wrote:
> > > On Fri, Mar 28, 2025 at 03:58:50PM +0100, Luis Chamberlain wrote:
> > > > On Fri, Mar 28, 2025 at 08:56:19PM +0000, Daniel Gomez wrote:
> > > > > From: Daniel Gomez <da.gomez@samsung.com>
> 
> > > I think merging all generated variables into Kconfig makes this a bit easier as
> > > we don't have to deal with another Ansible YAML file. Ideally, we would merge
> > > this new YAML with the Kconfig YAML output. Can this be done? Makes sense?
> > 
> > Yes. We have:
> > 
> > Kconfig -->
> > .config --> optional output yaml --> 1st.yaml
> >         |
> > 	|
> > 	v------- what we need -----> 2nd.yaml
> > 	|
> > 	|
> > 	v--------------------------> extra_vars.yaml
> > 	
> > 
> > Is that right?
> 
> I was thinking this instead:
> 
> What we have (please, correct if it's wrong):
> 
> Kconfig --> .config
>         --> .extra_vars_auto.yaml --> Makefile --> extra_vars.yaml --> Ansible
> 
> What I'm suggesting above:
> 
> Makefile --> Ansible --> env.yaml                                                                                             |
>                      --> env.config (Kconfig fragment) --> Kconfig --> .config                                                v
>                                                                    --> .extra_vars_auto.yaml --> Makefile --> extra_vars.yaml --> Ansible
> 
> But looking at this, do you see a way to simplify this?

Ah yes.

The 'make dynconfig' support already does this but it we currently only
use it it to scrape your PCI tree to support PCI passthrough through
Kconfig support.

It was built to grow, so for example in the future I've hinted before we
could use cloud tools to scrape your cloud options and so to generate
Kconfig files automatically for us, instead of us having to write each
entry.

I think this usecase begs for a more basic form of 'make dynconfig', one
which we always call.

If we're going to do that then we can kill two birds with one stone by
having info-dynconfig and system-info-dynconfig, where the info-dynconfig
consists of targets which won't require root or sudo. This is the part
that might be useful to Masahiro, things like your compiler. Then the
system-info-dynconfig target can be targets which require sudo or root.
In that way, if a user does not want to have local playbooks run sudo
they can simply describe their environment with a yaml file as input.

Nix users can also then just describe their system info and and be happy
with calling qemu in a simple way, if they don't want libvirt to manage
crap for you.

Thoughts?

  Luis

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH RFC 6/6] playbooks: add kconfig support
  2025-04-01 21:02           ` Luis Chamberlain
@ 2025-04-04 20:33             ` Daniel Gomez
  2025-04-07  4:03               ` Luis Chamberlain
  0 siblings, 1 reply; 16+ messages in thread
From: Daniel Gomez @ 2025-04-04 20:33 UTC (permalink / raw)
  To: Luis Chamberlain; +Cc: Masahiro Yamada, kdevops, Daniel Gomez

On Tue, Apr 01, 2025 at 02:02:23PM +0100, Luis Chamberlain wrote:
> On Mon, Mar 31, 2025 at 08:31:55PM +0200, Daniel Gomez wrote:
> > On Mon, Mar 31, 2025 at 12:32:29AM +0100, Luis Chamberlain wrote:
> > > On Mon, Mar 31, 2025 at 08:54:59AM +0200, Daniel Gomez wrote:
> > > > On Fri, Mar 28, 2025 at 03:58:50PM +0100, Luis Chamberlain wrote:
> > > > > On Fri, Mar 28, 2025 at 08:56:19PM +0000, Daniel Gomez wrote:
> > > > > > From: Daniel Gomez <da.gomez@samsung.com>
> > 
> > > > I think merging all generated variables into Kconfig makes this a bit easier as
> > > > we don't have to deal with another Ansible YAML file. Ideally, we would merge
> > > > this new YAML with the Kconfig YAML output. Can this be done? Makes sense?
> > > 
> > > Yes. We have:
> > > 
> > > Kconfig -->
> > > .config --> optional output yaml --> 1st.yaml
> > >         |
> > > 	|
> > > 	v------- what we need -----> 2nd.yaml
> > > 	|
> > > 	|
> > > 	v--------------------------> extra_vars.yaml
> > > 	
> > > 
> > > Is that right?
> > 
> > I was thinking this instead:
> > 
> > What we have (please, correct if it's wrong):
> > 
> > Kconfig --> .config
> >         --> .extra_vars_auto.yaml --> Makefile --> extra_vars.yaml --> Ansible
> > 
> > What I'm suggesting above:
> > 
> > Makefile --> Ansible --> env.yaml                                                                                             |
> >                      --> env.config (Kconfig fragment) --> Kconfig --> .config                                                v
> >                                                                    --> .extra_vars_auto.yaml --> Makefile --> extra_vars.yaml --> Ansible
> > 
> > But looking at this, do you see a way to simplify this?
> 
> Ah yes.
> 
> The 'make dynconfig' support already does this but it we currently only
> use it it to scrape your PCI tree to support PCI passthrough through
> Kconfig support.
> 
> It was built to grow, so for example in the future I've hinted before we
> could use cloud tools to scrape your cloud options and so to generate
> Kconfig files automatically for us, instead of us having to write each
> entry.
> 
> I think this usecase begs for a more basic form of 'make dynconfig', one
> which we always call.

The only difference I see is that dynconfig generates a full Kconfig file, while
this one creates a fragment meant to be merged with an existing .config using
predefined Kconfig options.

If I understood dynconfig correctly, then I agree with you. There's no reason
to declare these in the current Kconfig, unless we want to avoid generating them
every time. Then we'd need the fragment generation -> merge. Also, the fragment
generation is much simpler to generate than an entire Kconfig file.

> 
> If we're going to do that then we can kill two birds with one stone by
> having info-dynconfig and system-info-dynconfig, where the info-dynconfig
> consists of targets which won't require root or sudo. This is the part
> that might be useful to Masahiro, things like your compiler. Then the
> system-info-dynconfig target can be targets which require sudo or root.
> In that way, if a user does not want to have local playbooks run sudo
> they can simply describe their environment with a yaml file as input.

This works with either option, since both are Ansible playbooks anyway.

> 
> Nix users can also then just describe their system info and and be happy
> with calling qemu in a simple way, if they don't want libvirt to manage
> crap for you.

I'm not sure about this. Can you ellaborate?

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH RFC 6/6] playbooks: add kconfig support
  2025-04-04 20:33             ` Daniel Gomez
@ 2025-04-07  4:03               ` Luis Chamberlain
  2025-04-14 10:58                 ` Daniel Gomez
  0 siblings, 1 reply; 16+ messages in thread
From: Luis Chamberlain @ 2025-04-07  4:03 UTC (permalink / raw)
  To: Daniel Gomez; +Cc: Masahiro Yamada, kdevops, Daniel Gomez

On Fri, Apr 04, 2025 at 10:33:07PM +0200, Daniel Gomez wrote:
> On Tue, Apr 01, 2025 at 02:02:23PM +0100, Luis Chamberlain wrote:
> > I think this usecase begs for a more basic form of 'make dynconfig', one
> > which we always call.
> 
> The only difference I see is that dynconfig generates a full Kconfig file, while
> this one creates a fragment meant to be merged with an existing .config using
> predefined Kconfig options.

The relationship is a requirement to be generated before existing
Kconfig files. The current kconfig fork maintained by linux-kdevops/kconfig [0]
is maintained as a git subtree inside kdevops so to allow us to not require
those odd git submodule strategy, ie, we maintain all history inside git
and for users they just see the git tree one, not two.

And so the common goal is to be able to try to avoid having to modify
things which can make kconfig independent from Linux's version of kconfig.

[0] https://github.com/linux-kdevops/kconfig

> If I understood dynconfig correctly, then I agree with you. There's no reason
> to declare these in the current Kconfig, unless we want to avoid generating them
> every time. Then we'd need the fragment generation -> merge. Also, the fragment
> generation is much simpler to generate than an entire Kconfig file.

Indeed,

> > If we're going to do that then we can kill two birds with one stone by
> > having info-dynconfig and system-info-dynconfig, where the info-dynconfig
> > consists of targets which won't require root or sudo. This is the part
> > that might be useful to Masahiro, things like your compiler. Then the
> > system-info-dynconfig target can be targets which require sudo or root.
> > In that way, if a user does not want to have local playbooks run sudo
> > they can simply describe their environment with a yaml file as input.
> 
> This works with either option, since both are Ansible playbooks anyway.

Sure.

> > Nix users can also then just describe their system info and and be happy
> > with calling qemu in a simple way, if they don't want libvirt to manage
> > crap for you.
> 
> I'm not sure about this. Can you ellaborate?

OK so for example kdevops today aims to try infer tons of information
from a user's system for libvirt uses. For debian based systems we go so
far as having to use sudo to query the libvirt pools that exist. To many
this is a security push they wish they could avoid. So an alternative to
this is to instruct users by defaul to write down their systems's
information about storage pool information. For nix users the same might
apply so we do not repeat the same history. Instead of "requiring" to
run sudo to check a system's virsh pool, we just go the other way with
nix, and just have the user write down on a file manually where their
virtpool stuff is. Only if they user a special command do we do the
inference hoop which may require sudo. This sort of informaiton is
needed before kconfig starts processign Kconfig files, and so we have
two parts to this, a) new featuers b) existing features. The b) featuers
are like the libvirt pool stuff. While an example of a) is nix OS
suport.

 Luis

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH RFC 6/6] playbooks: add kconfig support
  2025-04-07  4:03               ` Luis Chamberlain
@ 2025-04-14 10:58                 ` Daniel Gomez
  2025-04-23  6:01                   ` Luis Chamberlain
  0 siblings, 1 reply; 16+ messages in thread
From: Daniel Gomez @ 2025-04-14 10:58 UTC (permalink / raw)
  To: Luis Chamberlain; +Cc: Masahiro Yamada, kdevops, Daniel Gomez

On Sun, Apr 06, 2025 at 09:03:38PM +0100, Luis Chamberlain wrote:
> On Fri, Apr 04, 2025 at 10:33:07PM +0200, Daniel Gomez wrote:
> > On Tue, Apr 01, 2025 at 02:02:23PM +0100, Luis Chamberlain wrote:
> > > I think this usecase begs for a more basic form of 'make dynconfig', one
> > > which we always call.
> > 
> > The only difference I see is that dynconfig generates a full Kconfig file, while
> > this one creates a fragment meant to be merged with an existing .config using
> > predefined Kconfig options.
> 
> The relationship is a requirement to be generated before existing
> Kconfig files. The current kconfig fork maintained by linux-kdevops/kconfig [0]
> is maintained as a git subtree inside kdevops so to allow us to not require
> those odd git submodule strategy, ie, we maintain all history inside git
> and for users they just see the git tree one, not two.
> 
> And so the common goal is to be able to try to avoid having to modify
> things which can make kconfig independent from Linux's version of kconfig.
> 
> [0] https://github.com/linux-kdevops/kconfig
> 
> > If I understood dynconfig correctly, then I agree with you. There's no reason
> > to declare these in the current Kconfig, unless we want to avoid generating them
> > every time. Then we'd need the fragment generation -> merge. Also, the fragment
> > generation is much simpler to generate than an entire Kconfig file.

I was thinking about this recently. We could put these Kconfig files in a
template and just replace the y/n/string options very easy with Ansible as in
this patch series. Maybe this was your earlier suggestion in this thread?

	"Essentially we replace the kdevops Kconfigs which rely on scripts
	with ansible calls which can rely on jinja2 template parsing on the
	fly."

Then, we don't need to merge anything. That way, we'd only regenerate the values
each time, not the whole Kconfig.

	"a simple jinja2 processing trick might be nicer to help us scale with more
	symbols, otherwise kconfig will be rather slow."

But here, your suggestion goes more into replacing Kconfig with jinja2. I'm
confused. Can you clarify this too?

> 
> Indeed,
> 
> > > If we're going to do that then we can kill two birds with one stone by
> > > having info-dynconfig and system-info-dynconfig, where the info-dynconfig
> > > consists of targets which won't require root or sudo. This is the part
> > > that might be useful to Masahiro, things like your compiler. Then the
> > > system-info-dynconfig target can be targets which require sudo or root.
> > > In that way, if a user does not want to have local playbooks run sudo
> > > they can simply describe their environment with a yaml file as input.
> > 
> > This works with either option, since both are Ansible playbooks anyway.
> 
> Sure.
> 
> > > Nix users can also then just describe their system info and and be happy
> > > with calling qemu in a simple way, if they don't want libvirt to manage
> > > crap for you.
> > 
> > I'm not sure about this. Can you ellaborate?
> 
> OK so for example kdevops today aims to try infer tons of information
> from a user's system for libvirt uses. For debian based systems we go so
> far as having to use sudo to query the libvirt pools that exist. To many
> this is a security push they wish they could avoid. So an alternative to
> this is to instruct users by defaul to write down their systems's
> information about storage pool information. For nix users the same might

This means, we need an input yaml with variables, right?

> apply so we do not repeat the same history. Instead of "requiring" to
> run sudo to check a system's virsh pool, we just go the other way with
> nix, and just have the user write down on a file manually where their
> virtpool stuff is. Only if they user a special command do we do the
> inference hoop which may require sudo. This sort of informaiton is
> needed before kconfig starts processign Kconfig files, and so we have
> two parts to this, a) new featuers b) existing features. The b) featuers
> are like the libvirt pool stuff. While an example of a) is nix OS
> suport.

I see. Thanks for clarifying.

With all the Kconfig, YAML, Ansible inputs, merges, and templates, it feels like
we’re overcomplicating something that should be relatively simple.

> 
>  Luis

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH RFC 6/6] playbooks: add kconfig support
  2025-04-14 10:58                 ` Daniel Gomez
@ 2025-04-23  6:01                   ` Luis Chamberlain
  0 siblings, 0 replies; 16+ messages in thread
From: Luis Chamberlain @ 2025-04-23  6:01 UTC (permalink / raw)
  To: Daniel Gomez; +Cc: Masahiro Yamada, kdevops, Daniel Gomez

On Mon, Apr 14, 2025 at 12:58:01PM +0200, Daniel Gomez wrote:
> On Sun, Apr 06, 2025 at 09:03:38PM +0100, Luis Chamberlain wrote:
> > On Fri, Apr 04, 2025 at 10:33:07PM +0200, Daniel Gomez wrote:
> > > On Tue, Apr 01, 2025 at 02:02:23PM +0100, Luis Chamberlain wrote:
> > > > I think this usecase begs for a more basic form of 'make dynconfig', one
> > > > which we always call.
> > > 
> > > The only difference I see is that dynconfig generates a full Kconfig file, while
> > > this one creates a fragment meant to be merged with an existing .config using
> > > predefined Kconfig options.
> > 
> > The relationship is a requirement to be generated before existing
> > Kconfig files. The current kconfig fork maintained by linux-kdevops/kconfig [0]
> > is maintained as a git subtree inside kdevops so to allow us to not require
> > those odd git submodule strategy, ie, we maintain all history inside git
> > and for users they just see the git tree one, not two.
> > 
> > And so the common goal is to be able to try to avoid having to modify
> > things which can make kconfig independent from Linux's version of kconfig.
> > 
> > [0] https://github.com/linux-kdevops/kconfig
> > 
> > > If I understood dynconfig correctly, then I agree with you. There's no reason
> > > to declare these in the current Kconfig, unless we want to avoid generating them
> > > every time. Then we'd need the fragment generation -> merge. Also, the fragment
> > > generation is much simpler to generate than an entire Kconfig file.
> 
> I was thinking about this recently. We could put these Kconfig files in a
> template and just replace the y/n/string options very easy with Ansible as in
> this patch series. Maybe this was your earlier suggestion in this thread?

There are two things being discussed, the fragment stuff, and the
enhancing reliability of scripts used in Kconfig. So while you had
one goal, the first, I was thinking more about the second, with the
hope we can pave the way for both. I am not sure now -- so disregard this
for now with time we can evaluate that other aspect. The reason I say
this is that runs of ansible are typically not used for generating
stdout, whereas scripts in Kconfig want simple bool, tristate, strings
in stdout. I just tried to see how far I can take ansible to the
simplest thing for stdout and I'm not happy about it so I think we can
just stick to python for more robust script calls.

> 	"Essentially we replace the kdevops Kconfigs which rely on scripts
> 	with ansible calls which can rely on jinja2 template parsing on the
> 	fly."
> 
> Then, we don't need to merge anything. That way, we'd only regenerate the values
> each time, not the whole Kconfig.
> 
> 	"a simple jinja2 processing trick might be nicer to help us scale with more
> 	symbols, otherwise kconfig will be rather slow."
> 
> But here, your suggestion goes more into replacing Kconfig with jinja2. I'm
> confused. Can you clarify this too?

I mean more about the usage of scripts in kconfig symbols. A complex
example was the inference for the storage pool path used by users.

> > OK so for example kdevops today aims to try infer tons of information
> > from a user's system for libvirt uses. For debian based systems we go so
> > far as having to use sudo to query the libvirt pools that exist. To many
> > this is a security push they wish they could avoid. So an alternative to
> > this is to instruct users by defaul to write down their systems's
> > information about storage pool information. For nix users the same might
> 
> This means, we need an input yaml with variables, right?

Yes.

> > apply so we do not repeat the same history. Instead of "requiring" to
> > run sudo to check a system's virsh pool, we just go the other way with
> > nix, and just have the user write down on a file manually where their
> > virtpool stuff is. Only if they user a special command do we do the
> > inference hoop which may require sudo. This sort of informaiton is
> > needed before kconfig starts processign Kconfig files, and so we have
> > two parts to this, a) new featuers b) existing features. The b) featuers
> > are like the libvirt pool stuff. While an example of a) is nix OS
> > suport.
> 
> I see. Thanks for clarifying.
> 
> With all the Kconfig, YAML, Ansible inputs, merges, and templates, it feels like
> we’re overcomplicating something that should be relatively simple.

Yeah agreed.

  Luis

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2025-04-23  6:01 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-03-28 20:56 [PATCH RFC 0/6] Generate Kconfig environment in Ansible Daniel Gomez
2025-03-28 20:56 ` [PATCH RFC 1/6] merge_config: add fragment support from kernel Daniel Gomez
2025-03-28 20:56 ` [PATCH RFC 2/6] scripts/kconfig/kconfig.Makefile: extend simple-targets Daniel Gomez
2025-03-28 20:56 ` [PATCH RFC 3/6] kconfig-fragments: add docs and fragments folder Daniel Gomez
2025-03-28 20:56 ` [PATCH RFC 4/6] kconfig-env: generate kconfig environment Daniel Gomez
2025-03-28 20:56 ` [PATCH RFC 5/6] kconfig: env: add support Daniel Gomez
2025-03-28 20:56 ` [PATCH RFC 6/6] playbooks: add kconfig support Daniel Gomez
2025-03-28 22:58   ` Luis Chamberlain
2025-03-31  6:54     ` Daniel Gomez
2025-03-31  7:32       ` Luis Chamberlain
2025-03-31 18:31         ` Daniel Gomez
2025-04-01 21:02           ` Luis Chamberlain
2025-04-04 20:33             ` Daniel Gomez
2025-04-07  4:03               ` Luis Chamberlain
2025-04-14 10:58                 ` Daniel Gomez
2025-04-23  6:01                   ` Luis Chamberlain

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox