[PATCH 0/3] kdevops: update to latest kconfig

[PATCH 0/3] kdevops: update to latest kconfig

[Luis Chamberlain]

This is the first part of the kconfig selective yaml config support.
kdevops has embraced git subtrees (note, not git submodule, those are
gross), so to allow us to adapt kconfig from an external git tree which
allows random projects to easily upgrade their versions of kconfig.

When we decided that the kdevops results archive was too large and to
start our git tree from scratch to keep it slim, the initial import
did not have the kconfig git tree incantations. So what we do is use the
git subtree split feature so to split out the work on scripts/kconfig
up to the latest known kconfig git tree git commit we sync'd to, which is
dce7653757b4 ("kconfig: fix parallel builds for menuconfig") and use
the current kdevops git commit 4b42b104d5ecd  ("linux-mirror: complete
ansible conversion"). And so I just ran:

I just ran:

    git subtree split --prefix=scripts/kconfig/ --onto \
            dce7653757b4f26945c09744b131d2daabe6257d \
            4b42b104d5ecd15b8fc3af0a71944ebb626dad03 \
            --rejoin https://github.com/linux-kdevops/kconfig.git
    
This will allow us to later update kconfig as if we had started
the git subtree on scripts/kconfig with the following:
    
git remote add kconfig https://github.com/linux-kdevops/kconfig.git
git fetch kconfig
git subtree add --prefix=scripts/kconfig kconfig master

Note that the above is not well represented in git with git format-patch
as you'll just see the merge commit in git history, git format-patch will
send the entire history and so I'm just including the commands I run here
to get us back to using a git subtree for scripts/kconfig/. The patches
in this series allow us to later update kconfig using git subtree afterwards,
so the patches in this series apply after the above command.

Then after the patches in this series I just run the following command
to get us to upgrade to the kconfig master branch which gets kdevops kconfig
in sync with the kconfig linux-next tag of next-20240830:

git subtree --squash pull --prefix=scripts/kconfig/  kconfig master

At this point we don't yet have selective yamlconfig support. To merge that
support and use it, will be addressed in the next patch series. But
note.. since the latest kconfig git subtree has nconfig support fixed
we also get to use it, so try running 'make nconfig' on this branch:

https://github.com/mcgrof/kdevops/tree/nconfig

The first two patches in this series are what we need to update the latest
kconfig and support nconfig, the last patch "scripts/gen-refs-default.Makefile:
support direct make requests" is just a cleanup drive-through, and so
not required but we might as well add it.

If this seems agreeable please let me know and I'll just merge this, nothing
controversial at this point other than just getting us up to speed again to
be able to track kconfig as a git subtree and updates us to kconfig linux-next
tag next-20240830.

Luis Chamberlain (3):
  scripts/kconfig/kconfig.Makefile: move _refs-default
  kdevops: remove duplicate symbols and use
    scripts/append-makefile-vars.sh
  scripts/gen-refs-default.Makefile: support direct make requests

 Makefile                              |  2 +-
 kconfigs/Kconfig.ansible_provisioning | 24 ++++++-----------------
 scripts/dynamic-kconfig.Makefile      |  2 +-
 scripts/gen-refs-default.Makefile     | 28 ++++++++++-----------------
 scripts/kconfig/kconfig.Makefile      |  2 +-
 terraform/aws/Kconfig                 | 14 --------------
 vagrant/Kconfig                       |  3 ++-
 vagrant/Kconfig.opensuse              | 22 ---------------------
 8 files changed, 21 insertions(+), 76 deletions(-)

-- 
2.43.0

[PATCH 1/3] scripts/kconfig/kconfig.Makefile: move _refs-default

[Luis Chamberlain]

We should avoid editing scripts/kconfig as much as possible unless
you are aware of the git subtree management for kconfig. So move
the old change to depend on _refs-default over to the Kconfig
dependency we had added to allow dynamic Kconfig. In the old
case we only enabled dynamic kconfigs when someone ran 'make dynconfig'
but Dan's patches which added ref tag support lets us now use dynamic
kconfig always but with minimal things: kernel branches for static
references defined in-tree on yaml files. That is we don't talk online
to fetch information.

Move the _refs-default dependency to the Kconfig dependency to make
this more explicit and to keep scripts/kconfig clean so to allow
easier updates from git subtrees.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 scripts/dynamic-kconfig.Makefile | 2 +-
 scripts/kconfig/kconfig.Makefile | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/dynamic-kconfig.Makefile b/scripts/dynamic-kconfig.Makefile
index bdeb499783b0..b6c0e43d6697 100644
--- a/scripts/dynamic-kconfig.Makefile
+++ b/scripts/dynamic-kconfig.Makefile
@@ -9,7 +9,7 @@ include $(TOPDIR)/scripts/dynamic-pci-kconfig.Makefile
 
 ANSIBLE_EXTRA_ARGS += $(DYNAMIC_KCONFIG_PCIE_ARGS)
 
-Kconfig: $(DYNAMIC_KCONFIG)
+Kconfig: _refs-default $(DYNAMIC_KCONFIG)
 
 dynamic-kconfig-help:
 	@echo "Dynamic kconfig targets:"
diff --git a/scripts/kconfig/kconfig.Makefile b/scripts/kconfig/kconfig.Makefile
index c537bb069264..68af73d2c0df 100644
--- a/scripts/kconfig/kconfig.Makefile
+++ b/scripts/kconfig/kconfig.Makefile
@@ -37,7 +37,7 @@ $(KCONFIG_DIR)/mconf:
 	$(MAKE) -C $(KCONFIG_DIR) mconf
 
 PHONY += menuconfig
-menuconfig: $(KCONFIG_DIR)/mconf include/config/project.release Kconfig _refs-default
+menuconfig: $(KCONFIG_DIR)/mconf include/config/project.release Kconfig
 	@$< Kconfig
 
 $(KCONFIG_DIR)/nconf:
-- 
2.43.0

[PATCH 2/3] kdevops: remove duplicate symbols and use scripts/append-makefile-vars.sh

[Luis Chamberlain]

A few kdevops kconfig duplicate symbols exist, remove them so to ensure
we can work with the latest linux-next version of kconfig. Additionally
we need to ensure to use scripts/append-makefile-vars.sh for when we
want to work with Makefile inherited variables. The KDEVOPS_PLAYBOOK_DIR
can also just be simplified, while we do provide for variability in
practice we never use this so just this a silent default for now.

Without this we end up in a hung state if we run 'make menuconfig'
with the latest kconfig. Fix this.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 kconfigs/Kconfig.ansible_provisioning | 24 ++++++------------------
 terraform/aws/Kconfig                 | 14 --------------
 vagrant/Kconfig                       |  3 ++-
 vagrant/Kconfig.opensuse              | 22 ----------------------
 4 files changed, 8 insertions(+), 55 deletions(-)

diff --git a/kconfigs/Kconfig.ansible_provisioning b/kconfigs/Kconfig.ansible_provisioning
index 82a6f822c3f5..63f04306e7b2 100644
--- a/kconfigs/Kconfig.ansible_provisioning
+++ b/kconfigs/Kconfig.ansible_provisioning
@@ -1,12 +1,6 @@
-if !KDEVOPS_ANSIBLE_PROVISION_ENABLE
-
 config KDEVOPS_PLAYBOOK_DIR
 	string
-	default $(shell, echo $KDEVOPS_PLAYBOOKS_DIR)
-	help
-	  The directory where ansible playbooks are found.
-
-endif # KDEVOPS_ANSIBLE_PROVISION_ENABLE
+	default $(shell, scripts/append-makefile-vars.sh $(KDEVOPS_PLAYBOOKS_DIR))
 
 config HAVE_DISTRO_PREFERS_DISABLE_ANSIBLE
 	bool
@@ -41,7 +35,7 @@ config KDEVOPS_DEVCONFIG_ENABLE_CONSOLE
 	bool "Enable console setup"
 	default y
 	depends on KDEVOPS_DEVCONFIG_ENABLE
-	  help
+	help
 	  Enable this so that we configure and enable console serial settings
           for you on /etc/default/grub.
 
@@ -50,7 +44,7 @@ if KDEVOPS_DEVCONFIG_ENABLE_CONSOLE
 config KDEVOPS_DEVCONFIG_KERNEL_CONSOLE_SETTINGS
 	string "Kernel command line console kernel parameters to use"
 	default "console=tty0 console=tty1 console=ttyS0,115200n8"
-	  help
+	help
           Linux kernel console kernel parameters to use. This will be appended
 	  to the GRUB configuration file /etc/default/grub on the variable
           GRUB_CMDLINE_LINUX_DEFAULT, if and only if you did not have any
@@ -59,7 +53,7 @@ config KDEVOPS_DEVCONFIG_KERNEL_CONSOLE_SETTINGS
 config KDEVOPS_DEVCONFIG_GRUB_SERIAL_COMMAND
 	string "GRUB specific serial settings"
 	default "serial --speed=115200 --unit=0 --parity=no --stop=1"
-	  help
+	help
           GRUB console settings to use. This will be appended to the GRUB
           configuration file /etc/default/grub on the variable
           GRUB_SERIAL_COMMAND, if and only if we did change the your console
@@ -68,7 +62,7 @@ config KDEVOPS_DEVCONFIG_GRUB_SERIAL_COMMAND
 config KDEVOPS_GRUB_TIMEOUT
 	int "GRUB timeout"
 	default 2
-	  help
+	help
           GRUB timeout to wait for user interaction to let you pick another
           kernel or boot options.
 
@@ -104,15 +98,9 @@ config KDEVOPS_DEVCONFIG_SYSTEMD_WATCHDOG_TIMEOUT_KEXEC
 
 endif # KDEVOPS_DEVCONFIG_ENABLE_SYSTEMD_WATCHDOG
 
-config KDEVOPS_PLAYBOOK_DIR
-	string "The directory for the ansible playbooks to use"
-	default $(shell, echo $KDEVOPS_PLAYBOOKS_DIR)
-	help
-	  The directory where ansible playbooks are found.
-
 config KDEVOPS_ANSIBLE_INVENTORY_FILE
 	string "The ansible inventory file to use"
-	default $(shell, echo $KDEVOPS_HOSTFILE)
+	default $(shell, scripts/append-makefile-vars.sh $(KDEVOPS_HOSTFILE))
 	help
 	  The file to use for the ansible inventory.
 
diff --git a/terraform/aws/Kconfig b/terraform/aws/Kconfig
index dafacaeada15..a94045afcfee 100644
--- a/terraform/aws/Kconfig
+++ b/terraform/aws/Kconfig
@@ -280,13 +280,6 @@ config TERRAFORM_AWS_INSTANCE_IS4GEN_MEDIUM
 	  6 GiB RAM, 1vCPU, 1 x 937 GiB NVMe SSD, 25 Gbps Net, 9.6 Gbps EBS.
 	  https://aws.amazon.com/ec2/instance-types/i4g/
 
-config TERRAFORM_AWS_INSTANCE_IS4GEN_8XLARGE
-	bool "is4gen.8xlarge"
-	depends on TARGET_ARCH_ARM64
-	help
-	  192 GiB RAM, 32 vCPUs, 4 x 7500 GiB NVMe SSD, 50 Gbps Net, 19 Gbps EBS.
-	  https://aws.amazon.com/ec2/instance-types/i4g/
-
 config TERRAFORM_AWS_INSTANCE_IS4GEN_8XLARGE
 	bool "is4gen.8xlarge"
 	depends on TARGET_ARCH_ARM64
@@ -308,13 +301,6 @@ config TERRAFORM_AWS_INSTANCE_IM4GN_4XLARGE
 	  64 GiB RAM, 16 vcpus, 1 x 7500 NVMe SSD, 25 Gbps Net, 9.5 Gbps EBS.
 	  https://aws.amazon.com/ec2/instance-types/i4g/
 
-config TERRAFORM_AWS_INSTANCE_IM4GN_4XLARGE
-	bool "im4gn.4xlarge"
-	depends on TARGET_ARCH_ARM64
-	help
-	  64 GiB RAM, 16 vcpus, 1 x 7500 NVMe SSD, 25 Gbps Net, 9.5 Gbps EBS.
-	  https://aws.amazon.com/ec2/instance-types/i4g/
-
 config TERRAFORM_AWS_INSTANCE_I4I_LARGE
 	bool "i4i.large"
 	depends on TARGET_ARCH_X86_64
diff --git a/vagrant/Kconfig b/vagrant/Kconfig
index 93480274885e..ecf7bcafe1ad 100644
--- a/vagrant/Kconfig
+++ b/vagrant/Kconfig
@@ -1,4 +1,5 @@
 if VAGRANT
+
 choice
 	prompt "Vagrant virtualization technology to use"
 	default VAGRANT_LIBVIRT_SELECT
@@ -89,7 +90,7 @@ config VIRTUALBOX_EXTRA_DRIVE_FORMAT
 	default "vmdk" if VIRTUALBOX_EXTRA_DRIVE_VMDK
 	default "vhd" if VIRTUALBOX_EXTRA_DRIVE_VHD
 
-endif
+endif # VAGRANT_VIRTUALBOX
 
 config HAVE_SUSE_VAGRANT
 	bool
diff --git a/vagrant/Kconfig.opensuse b/vagrant/Kconfig.opensuse
index 2ffb1f31c02b..6da402755c26 100644
--- a/vagrant/Kconfig.opensuse
+++ b/vagrant/Kconfig.opensuse
@@ -56,28 +56,6 @@ config VAGRANT_OPENSUSE_X86_64_LEAP_15_3
 	help
 	  This enables the OpenSUSE Leap 15.3 release.
 
-config VAGRANT_OPENSUSE_X86_64_LEAP_15_4
-	bool "Leap-15.4.x86_64"
-	depends on TARGET_ARCH_X86_64
-	select HAVE_VAGRANT_BOX_URL
-	select HAVE_DISTRO_BTRFS_DISABLES_RAID56  if FSTESTS_BTRFS
-	select HAVE_DISTRO_XFS_SUPPORTS_CRC if FSTESTS_XFS
-	select HAVE_DISTRO_XFS_SUPPORTS_EXTERNAL_LOG if FSTESTS_XFS
-	select HAVE_DISTRO_XFS_SUPPORTS_REFLINKS if FSTESTS_XFS
-	select HAVE_DISTRO_XFS_SUPPORTS_BIGBLOCKS if FSTESTS_XFS
-	select HAVE_DISTRO_PREFERS_FSTESTS_WATCHDOG
-	select HAVE_DISTRO_PREFERS_FSTESTS_WATCHDOG_KILL
-	select HAVE_DISTRO_BLKTESTS_TEST_BLOCK if KDEVOPS_WORKFLOW_ENABLE_BLKTESTS
-	select HAVE_DISTRO_BLKTESTS_TEST_LOOP if KDEVOPS_WORKFLOW_ENABLE_BLKTESTS
-	select HAVE_DISTRO_BLKTESTS_TEST_NBD if KDEVOPS_WORKFLOW_ENABLE_BLKTESTS
-	select HAVE_DISTRO_PREFERS_BLKTESTS_WATCHDOG if KDEVOPS_WORKFLOW_ENABLE_BLKTESTS
-	select HAVE_DISTRO_PREFERS_BLKTESTS_WATCHDOG_KILL if KDEVOPS_WORKFLOW_ENABLE_BLKTESTS
-	select HAVE_DISTRO_PREFERS_REFRESH
-	select HAVE_DISTRO_PREFERS_UPDATE
-	select HAVE_DISTRO_PREFERS_INSTALL_KDEVTOOLS
-	help
-	  This enables the OpenSUSE Leap 15.4 release.
-
 endchoice
 
 if VAGRANT_OPENSUSE_X86_64_TW
-- 
2.43.0

[PATCH 3/3] scripts/gen-refs-default.Makefile: support direct make requests

[Luis Chamberlain]

Today we can't use:

rm -f workflows/linux/refs/default/Kconfig.linus
make workflows/linux/refs/default/Kconfig.linus

Add support for this and simplify the definitions.

Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
 Makefile                          |  2 +-
 scripts/gen-refs-default.Makefile | 28 ++++++++++------------------
 2 files changed, 11 insertions(+), 19 deletions(-)

diff --git a/Makefile b/Makefile
index b961e44f4aa6..9d4e954811ab 100644
--- a/Makefile
+++ b/Makefile
@@ -20,6 +20,7 @@ export KDEVOPS_VAGRANT :=
 export PYTHONUNBUFFERED=1
 export TOPDIR=./
 export TOPDIR_PATH = $(shell readlink -f $(TOPDIR))
+include scripts/refs.Makefile
 
 KDEVOPS_NODES_ROLE_TEMPLATE_DIR :=		$(KDEVOPS_PLAYBOOKS_DIR)/roles/gen_nodes/templates
 export KDEVOPS_NODES_TEMPLATE :=
@@ -174,7 +175,6 @@ endif
 
 include scripts/gen-hosts.Makefile
 include scripts/gen-nodes.Makefile
-include scripts/refs.Makefile
 
 # disable built-in rules for this
 .SUFFIXES:
diff --git a/scripts/gen-refs-default.Makefile b/scripts/gen-refs-default.Makefile
index b04542283211..c5d12e152cb4 100644
--- a/scripts/gen-refs-default.Makefile
+++ b/scripts/gen-refs-default.Makefile
@@ -1,14 +1,12 @@
 # SPDX-License-Identifier: copyleft-next-0.3.1
 
-REFS_TARGET_LNS := gen_default_refs_linus
-REFS_TARGET_LNS += gen_default_refs_next
-REFS_TARGET_LNS += gen_default_refs_stable
+REF_DEF_OBJS := $(addprefix $(TOPDIR)/workflows/linux/refs/default/, Kconfig.linus Kconfig.next Kconfig.stable)
+REF_DEF_SRC  := $(addprefix $(TOPDIR)/workflows/linux/refs/static/,  linus.yaml next.yaml stable.yaml)
 
 KRELEASES_FORCE := $(if $(filter --force,$(KRELEASES_FORCE)),--force,)
 
-.PHONY += gen_default_refs_linus
-gen_default_refs_linus:
-	$(Q)$(E) "Generating refs/default/Kconfig.$(patsubst gen_default_refs_%,%,$@)..."
+$(TOPDIR)/workflows/linux/refs/default/Kconfig.linus: $(TOPDIR)/workflows/linux/refs/static/linus.yaml
+	$(Q)$(E) "Generating $@..."
 	$(Q)./scripts/generate_refs.py \
 		--prefix BOOTLINUX_TREE_LINUS \
 		--output workflows/linux/refs/default/Kconfig.linus \
@@ -17,9 +15,8 @@ gen_default_refs_linus:
 		kreleases \
 		--moniker mainline
 
-.PHONY += gen_default_refs_next
-gen_default_refs_next:
-	$(Q)$(E) "Generating refs/default/Kconfig.$(patsubst gen_default_refs_%,%,$@)..."
+$(TOPDIR)/workflows/linux/refs/default/Kconfig.next: $(TOPDIR)/workflows/linux/refs/static/next.yaml
+	$(Q)$(E) "Generating $@..."
 	$(Q)./scripts/generate_refs.py \
 		--prefix BOOTLINUX_TREE_NEXT \
 		--output workflows/linux/refs/default/Kconfig.next \
@@ -28,9 +25,8 @@ gen_default_refs_next:
 		kreleases \
 		--moniker linux-next
 
-.PHONY += gen_default_refs_stable
-gen_default_refs_stable:
-	$(Q)$(E) "Generating refs/default/Kconfig.$(patsubst gen_default_refs_%,%,$@)..."
+$(TOPDIR)/workflows/linux/refs/default/Kconfig.stable: $(TOPDIR)/workflows/linux/refs/static/stable.yaml
+	$(Q)$(E) "Generating $@..."
 	$(Q)./scripts/generate_refs.py \
 		--prefix BOOTLINUX_TREE_STABLE \
 		--output workflows/linux/refs/default/Kconfig.stable \
@@ -39,14 +35,10 @@ gen_default_refs_stable:
 		kreleases \
 		--moniker stable
 
-_gen-default-refs-lns:
-	$(Q)$(E) "Generating refs/default/Kconfig.{linus,next,stable} files..."
-	$(Q)$(MAKE) $(REFS_TARGET_LNS) KRELEASES_FORCE="--force"
-
 PHONY += refs-default
-refs-default: _gen-default-refs-lns _gen-default-refs-development refs-user-clean
+refs-default: $(REF_DEF_OBJS) _gen-default-refs-development refs-user-clean
 
 PHONY += _refs-default
-_refs-default: _gen-default-refs-lns
+_refs-default: $(REF_DEF_OBJS)
 
 .PHONY: $(PHONY)
-- 
2.43.0