* [PATCH 1/6] mirror: add a smart git check
2024-01-25 20:35 [PATCH 0/6] Fix ordering for systemd remote journal support Luis Chamberlain
@ 2024-01-25 20:35 ` Luis Chamberlain
2024-01-25 20:35 ` [PATCH 2/6] bringup: split provisioning into 2 steps Luis Chamberlain
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-01-25 20:35 UTC (permalink / raw)
To: kdevops, da.gomez, p.raghav; +Cc: Luis Chamberlain
If you are on a firewall chances are you will fail miserably with the
default mirroring options and this is a terrible experience as the
git protocol will fail. The git protocol is nice and efficient so
we don't want to change the default to https.
So add a smart git check on make menuconfig which will test if the git
protocol will work, otherwise disable all git options. This incurs
a penalty on checking if the git protocol works once on make menuconfig
with a timeout of 3 seconds. This hack only works on systems which have
netcat installed, otherwise we assume git will work too. Ie, if you want
this smart heuristic install netcat.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
kconfigs/Kconfig.mirror | 24 +++++++++++++++++++-----
scripts/test_git_firewall.sh | 17 +++++++++++++++++
2 files changed, 36 insertions(+), 5 deletions(-)
create mode 100755 scripts/test_git_firewall.sh
diff --git a/kconfigs/Kconfig.mirror b/kconfigs/Kconfig.mirror
index 1b237c7c32c5..3f0b15859319 100644
--- a/kconfigs/Kconfig.mirror
+++ b/kconfigs/Kconfig.mirror
@@ -47,9 +47,14 @@ config INSTALL_LOCAL_LINUX_MIRROR
first linux-mirror setup and just use USE_LOCAL_LINUX_MIRROR so to
make use of the mirror.
+config MIRROR_GIT_WORKS
+ bool
+ default $(shell, ./scripts/test_git_firewall.sh)
+
choice
prompt "Linux mirror protocol"
- default MIRROR_TORVALDS_GIT
+ default MIRROR_TORVALDS_GIT if MIRROR_GIT_WORKS
+ default MIRROR_TORVALDS_HTTPS if !MIRROR_GIT_WORKS
depends on INSTALL_LOCAL_LINUX_MIRROR
config MIRROR_TORVALDS_HTTPS
@@ -70,6 +75,7 @@ config MIRROR_TORVALDS_HTTPS_GOOGLE
config MIRROR_TORVALDS_GIT
bool "Git"
+ depends on MIRROR_GIT_WORKS
help
If you enable this option then git:// protocol will be used as the
source of the mirror. The URL is:
@@ -105,7 +111,8 @@ config MIRROR_KDEVOPS_LINUS_URL
choice
prompt "Linux Next mirror protocol/source"
- default MIRROR_NEXT_GIT
+ default MIRROR_NEXT_GIT if MIRROR_GIT_WORKS
+ default MIRROR_NEXT_HTTPS if !MIRROR_GIT_WORKS
depends on INSTALL_LOCAL_LINUX_MIRROR
config MIRROR_NEXT_HTTPS
@@ -126,6 +133,7 @@ config MIRROR_NEXT_HTTPS_GOOGLE
config MIRROR_NEXT_GIT
bool "Git"
+ depends on MIRROR_GIT_WORKS
help
If you enable this option then Git protocol will be used as the
source of the mirror. The URL is:
@@ -142,7 +150,8 @@ config MIRROR_NEXT_URL
choice
prompt "Linux mcgrof's linux-next fork protocol/source"
- default MIRROR_MCGROF_GIT
+ default MIRROR_MCGROF_GIT if MIRROR_GIT_WORKS
+ default MIRROR_MCGROF_HTTPS if !MIRROR_GIT_WORKS
depends on INSTALL_LOCAL_LINUX_MIRROR
config MIRROR_MCGROF_HTTPS
@@ -163,6 +172,7 @@ config MIRROR_MCGROF_HTTPS_GOOGLE
config MIRROR_MCGROF_GIT
bool "Git"
+ depends on MIRROR_GIT_WORKS
help
If you enable this option then Git protocol will be used as the
source of the mirror. The URL is:
@@ -179,7 +189,8 @@ config MIRROR_MCGROF_URL
choice
prompt "Linux mcgrof's linux fork protocol/source"
- default MIRROR_MCGROF_LINUS_GIT
+ default MIRROR_MCGROF_LINUS_GIT if MIRROR_GIT_WORKS
+ default MIRROR_MCGROF_LINUS_HTTPS if !MIRROR_GIT_WORKS
depends on INSTALL_LOCAL_LINUX_MIRROR
config MIRROR_MCGROF_LINUS_HTTPS
@@ -200,6 +211,7 @@ config MIRROR_MCGROF_LINUS_HTTPS_GOOGLE
config MIRROR_MCGROF_LINUS_GIT
bool "Git"
+ depends on MIRROR_GIT_WORKS
help
If you enable this option then Git protocol will be used as the
source of the mirror. The URL is:
@@ -216,7 +228,8 @@ config MIRROR_MCGROF_LINUS_URL
choice
prompt "Linux stable mirror protocol/source"
- default MIRROR_STABLE_GIT
+ default MIRROR_STABLE_GIT if MIRROR_GIT_WORKS
+ default MIRROR_STABLE_HTTPS if !MIRROR_GIT_WORKS
depends on INSTALL_LOCAL_LINUX_MIRROR
config MIRROR_STABLE_HTTPS
@@ -237,6 +250,7 @@ config MIRROR_STABLE_HTTPS_GOOGLE
config MIRROR_STABLE_GIT
bool "Git"
+ depends on MIRROR_GIT_WORKS
help
If you enable this option then Git protocol will be used as the
source of the mirror. The URL is:
diff --git a/scripts/test_git_firewall.sh b/scripts/test_git_firewall.sh
new file mode 100755
index 000000000000..019151d5ab9d
--- /dev/null
+++ b/scripts/test_git_firewall.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+CUR_VAL=$1
+which nc &> /dev/null
+if [[ $? -ne 0 ]]; then
+ echo y
+ exit
+fi
+
+# change to port 9419 to verify this will fail if your firewall does not
+# allow git access.
+nc -v -z -w 3 git.kernel.org 9418 &> /dev/null
+if [[ $? -eq 0 ]]; then
+ echo y
+else
+ echo n
+fi
--
2.42.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 2/6] bringup: split provisioning into 2 steps
2024-01-25 20:35 [PATCH 0/6] Fix ordering for systemd remote journal support Luis Chamberlain
2024-01-25 20:35 ` [PATCH 1/6] mirror: add a smart git check Luis Chamberlain
@ 2024-01-25 20:35 ` Luis Chamberlain
2024-01-25 20:35 ` [PATCH 3/6] bringup: share bringup method targe and agument by two steps Luis Chamberlain
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-01-25 20:35 UTC (permalink / raw)
To: kdevops, da.gomez, p.raghav; +Cc: Luis Chamberlain
Right now we deal with provisioning as a post bringup work item and it
does two things:
1) Update ssh config file
2) Runs the devconfig playbook, which optionally installs some deps
other than do basic things like setup console, kdump, etc.
The fact that deps are optional on the devconfig playbook makes it
hard to add dependencies we want to install without incurring a penalty
to everyone as a default option to install all standard generic kernel
development tools we define under CONFIG_KDEVOPS_TRY_INSTALL_KDEV_TOOLS.
The reason we disable CONFIG_KDEVOPS_TRY_INSTALL_KDEV_TOOLS by default
is it delays bringup.
However the devconfig playbook is also used to place random oddball
generic tools may want to install on bringup. Some of these tools we
may want to try to install early on, even before we run the devconfig
playbook for generic stuff like setting up the console or kdump.
To help with this, and in order to allow us to keep relying on the
devconfig playbook for random packages we may want to allow users
to install, we split the above two provisioning tasks into two
separate tasks. This will allow us to add another middle set of
targets which get executed after 1) but before 2). We will add
support for this in the next subsequent commit.
For now this should be a no-op for both vagrant and libguest. For
terraform we actually enhance the experience by adding a guard to
not run the devconfig playbook twice if a user by mistake tried to
run 'make bringup' twice. This guard was not present before for
terraform.
Maybe later on we should share these snippets between all 3 bringup
methods. Note we have no generic task for bare metal yet, but we
should -- we should just add the devconfig playbook. That would
cases where bare metal systems which are already brought up and
you have ssh configuration going for already working.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
.gitignore | 4 +++-
Makefile | 5 +++--
scripts/guestfs.Makefile | 24 ++++++++++++++++++------
scripts/terraform.Makefile | 18 ++++++++++++++++++
scripts/vagrant.Makefile | 22 +++++++++++++++++-----
5 files changed, 59 insertions(+), 14 deletions(-)
diff --git a/.gitignore b/.gitignore
index 738d073e889a..1d2653c5b26c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -29,9 +29,11 @@ scripts/workflows/lib/__pycache__/
vagrant/kdevops_nodes.yaml
vagrant/.Vagrantfile.generated
vagrant/Vagrantfile
-vagrant/.provisioned_once
+vagrant/.provisioned_once*
.vagrant/
+terraform/.provisioned_once*
+
include/
# You can override role specific stuff on these
diff --git a/Makefile b/Makefile
index ad34743e3e53..8ee02f2a9b6c 100644
--- a/Makefile
+++ b/Makefile
@@ -86,9 +86,10 @@ endif # CONFIG_NEEDS_LOCAL_DEVELOPMENT_PATH
ANSIBLE_EXTRA_ARGS += $(LOCAL_DEVELOPMENT_ARGS)
# These should be set as non-empty if you want any generic bring up
-# targets to come up. We support 2 bring up methods:
+# targets to come up. We support 3 bring up methods:
#
-# - vagrant: for kvm/virtualbox
+# - vagrant: for kvm/virtualbox - will eventually be deprecated
+# - libguestfs: for kvm
# - terraform: for any cloud provider
#
# If you are using bare metal, you don't do bring up, or you'd
diff --git a/scripts/guestfs.Makefile b/scripts/guestfs.Makefile
index 211ac9ba688e..17c8cbd354c1 100644
--- a/scripts/guestfs.Makefile
+++ b/scripts/guestfs.Makefile
@@ -5,9 +5,11 @@ GUESTFS_ARGS :=
KDEVOPS_NODES_TEMPLATE := $(KDEVOPS_NODES_ROLE_TEMPLATE_DIR)/guestfs_nodes.j2
KDEVOPS_NODES := guestfs/kdevops_nodes.yaml
-export KDEVOPS_GUESTFS_PROVISIONED := guestfs/.provisioned_once
+export KDEVOPS_GUESTFS_PROVISIONED_SSH := guestfs/.provisioned_once_ssh
+export KDEVOPS_GUESTFS_PROVISIONED_DEVCONFIG := guestfs/.provisioned_once_devconfig
-KDEVOPS_MRPROPER += $(KDEVOPS_GUESTFS_PROVISIONED)
+KDEVOPS_MRPROPER += $(KDEVOPS_GUESTFS_PROVISIONED_SSH)
+KDEVOPS_MRPROPER += $(KDEVOPS_GUESTFS_PROVISIONED_DEVCONFIG)
GUESTFS_ARGS += kdevops_enable_guestfs=True
GUESTFS_ARGS += guestfs_path='$(TOPDIR_PATH)/guestfs'
@@ -51,11 +53,18 @@ GUESTFS_BRINGUP_DEPS :=
GUESTFS_BRINGUP_DEPS += $(9P_HOST_CLONE)
GUESTFS_BRINGUP_DEPS += $(LIBVIRT_PCIE_PASSTHROUGH)
-KDEVOPS_BRING_UP_DEPS := bringup_guestfs
+ KDEVOPS_BRING_UP_DEPS := bringup_guestfs
KDEVOPS_DESTROY_DEPS := destroy_guestfs
# Provisioning goes last
-KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_GUESTFS_PROVISIONED)
+#
+# Provisioning split into 2 steps:
+# 1) Ensuring we can use ansible with ssh
+# 2) Generic devconfig final configuration (which may include extra tools)
+#
+# Anything deps after this is dealt with on each respective workflow.
+KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_GUESTFS_PROVISIONED_SSH)
+KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_GUESTFS_PROVISIONED_DEVCONFIG)
9p_linux_clone:
$(Q)make linux-clone
@@ -66,15 +75,18 @@ libvirt_pcie_passthrough_permissions:
playbooks/libvirt_pcie_passthrough.yml \
-e 'ansible_python_interpreter=/usr/bin/python3'
-$(KDEVOPS_GUESTFS_PROVISIONED):
+$(KDEVOPS_GUESTFS_PROVISIONED_SSH):
$(Q)if [[ "$(CONFIG_KDEVOPS_SSH_CONFIG_UPDATE)" == "y" ]]; then \
LIBVIRT_DEFAULT_URI=$(CONFIG_LIBVIRT_URI) $(TOPDIR)/scripts/update_ssh_config_guestfs.py; \
fi
+ $(Q)touch $(KDEVOPS_GUESTFS_PROVISIONED_SSH)
+
+$(KDEVOPS_GUESTFS_PROVISIONED_DEVCONFIG):
$(Q)if [[ "$(CONFIG_KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK)" != "" ]]; then \
ansible-playbook $(ANSIBLE_VERBOSE) -i \
$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/$(KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK) ; \
fi
- $(Q)touch $(KDEVOPS_GUESTFS_PROVISIONED)
+ $(Q)touch $(KDEVOPS_GUESTFS_PROVISIONED_DEVCONFIG)
bringup_guestfs: $(GUESTFS_BRINGUP_DEPS)
$(Q)$(TOPDIR)/scripts/bringup_guestfs.sh
diff --git a/scripts/terraform.Makefile b/scripts/terraform.Makefile
index d4caa4dbb6d6..8e95e8cd822b 100644
--- a/scripts/terraform.Makefile
+++ b/scripts/terraform.Makefile
@@ -2,7 +2,22 @@
TERRAFORM_EXTRA_VARS :=
+export KDEVOPS_TERRAFORM_PROVISIONED_DEVCONFIG := terraform/.provisioned_once_devconfig
+KDEVOPS_MRPROPER += $(KDEVOPS_TERRAFORM_PROVISIONED_DEVCONFIG)
+
KDEVOPS_BRING_UP_DEPS := bringup_terraform
+# Provisioning goes last
+#
+# Provisioning split into 2 steps:
+# 1) Ensuring we can use ansible with ssh - this is implied by all
+# terraform providers. That is, our terraform providers all have a last
+# provisioning element task which does this update for us as part of
+# the above bringup_terraform.
+# 2) Generic devconfig final configuration (which may include extra tools)
+#
+# Anything deps after this is dealt with on each respective workflow.
+KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_TERRAFORM_PROVISIONED_DEVCONFIG)
+
KDEVOPS_DESTROY_DEPS := destroy_terraform
KDEVOPS_NODES_TEMPLATE := $(KDEVOPS_NODES_ROLE_TEMPLATE_DIR)/terraform_nodes.tf.j2
@@ -163,10 +178,13 @@ ANSIBLE_EXTRA_ARGS += $(TERRAFORM_EXTRA_VARS)
bringup_terraform:
$(Q)$(TOPDIR)/scripts/bringup_terraform.sh
+
+$(KDEVOPS_TERRAFORM_PROVISIONED_DEVCONFIG):
$(Q)if [[ "$(CONFIG_KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK)" != "" ]]; then \
ansible-playbook $(ANSIBLE_VERBOSE) -i \
$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/$(KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK) ;\
fi
+ $(Q)touch $(KDEVOPS_TERRAFORM_PROVISIONED_DEVCONFIG)
destroy_terraform:
$(Q)$(TOPDIR)/scripts/destroy_terraform.sh
diff --git a/scripts/vagrant.Makefile b/scripts/vagrant.Makefile
index 0633f90da719..825cfd210c6b 100644
--- a/scripts/vagrant.Makefile
+++ b/scripts/vagrant.Makefile
@@ -8,11 +8,13 @@ KDEVOPS_NODES := vagrant/kdevops_nodes.yaml
KDEVOPS_VAGRANT_TEMPLATE := $(KDEVOPS_NODES_ROLE_TEMPLATE_DIR)/Vagrantfile.j2
KDEVOPS_VAGRANT_GENERATED:= vagrant/.Vagrantfile.generated
KDEVOPS_VAGRANT := vagrant/Vagrantfile
-export KDEVOPS_VAGRANT_PROVISIONED := vagrant/.provisioned_once
+export KDEVOPS_VAGRANT_PROVISIONED_SSH := vagrant/.provisioned_once_ssh
+export KDEVOPS_VAGRANT_PROVISIONED_DEVCONFIG := vagrant/.provisioned_once_devconfig
KDEVOPS_MRPROPER += $(KDEVOPS_VAGRANT_GENERATED)
KDEVOPS_MRPROPER += $(KDEVOPS_VAGRANT)
-KDEVOPS_MRPROPER += $(KDEVOPS_VAGRANT_PROVISIONED)
+KDEVOPS_MRPROPER += $(KDEVOPS_VAGRANT_PROVISIONED_SSH)
+KDEVOPS_MRPROPER += $(KDEVOPS_VAGRANT_PROVISIONED_DEVCONFIG)
VAGRANT_ARGS += kdevops_vagrant_template_full_path='$(TOPDIR_PATH)/$(KDEVOPS_VAGRANT_TEMPLATE)'
@@ -75,7 +77,14 @@ VAGRANT_BRINGUP_DEPS += $(LIBVIRT_PCIE_PASSTHROUGH)
KDEVOPS_BRING_UP_DEPS := bringup_vagrant
# Provisioning goes last
-KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_VAGRANT_PROVISIONED)
+#
+# Provisioning split into 2 steps:
+# 1) Ensuring we can use ansible with ssh
+# 2) Generic devconfig final configuration (which may include extra tools)
+#
+# Anything deps after this is dealt with on each respective workflow.
+KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_VAGRANT_PROVISIONED_SSH)
+KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_VAGRANT_PROVISIONED_DEVCONFIG)
KDEVOPS_DESTROY_DEPS := destroy_vagrant
@@ -99,18 +108,21 @@ libvirt_pcie_passthrough_permissions:
playbooks/libvirt_pcie_passthrough.yml \
-e 'ansible_python_interpreter=/usr/bin/python3'
-$(KDEVOPS_VAGRANT_PROVISIONED):
+$(KDEVOPS_VAGRANT_PROVISIONED_SSH):
$(Q)if [[ "$(CONFIG_KDEVOPS_SSH_CONFIG_UPDATE)" == "y" ]]; then \
ansible-playbook $(ANSIBLE_VERBOSE) --connection=local \
--inventory localhost, \
playbooks/update_ssh_config_vagrant.yml \
-e 'ansible_python_interpreter=/usr/bin/python3' ;\
fi
+ $(Q)touch $(KDEVOPS_VAGRANT_PROVISIONED_SSH)
+
+$(KDEVOPS_VAGRANT_PROVISIONED_DEVCONFIG):
$(Q)if [[ "$(CONFIG_KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK)" != "" ]]; then \
ansible-playbook $(ANSIBLE_VERBOSE) -i \
$(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/$(KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK) ;\
fi
- $(Q)touch $(KDEVOPS_VAGRANT_PROVISIONED)
+ $(Q)touch $(KDEVOPS_VAGRANT_PROVISIONED_DEVCONFIG)
bringup_vagrant: $(VAGRANT_BRINGUP_DEPS)
$(Q)$(TOPDIR)/scripts/bringup_vagrant.sh
--
2.42.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 3/6] bringup: share bringup method targe and agument by two steps
2024-01-25 20:35 [PATCH 0/6] Fix ordering for systemd remote journal support Luis Chamberlain
2024-01-25 20:35 ` [PATCH 1/6] mirror: add a smart git check Luis Chamberlain
2024-01-25 20:35 ` [PATCH 2/6] bringup: split provisioning into 2 steps Luis Chamberlain
@ 2024-01-25 20:35 ` Luis Chamberlain
2024-01-25 20:35 ` [PATCH 4/6] provision: move all provisioning things to its own Makefile Luis Chamberlain
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-01-25 20:35 UTC (permalink / raw)
To: kdevops, da.gomez, p.raghav; +Cc: Luis Chamberlain
First we formalize now a kdevops provisioner as for one which can
bring up nodes for us, can optionally setup ssh for us, and handles
destroying nodes.
Prior to this patch we defined provisioning as a two step process:
1) ssh update
2) devconfig playbook is run
However we implicitly were issuing each provisioning script. So
let us augment this first by defining the provisionin method first.
The devconfig playbook target is also common, so no need to
keep that separate. By sharing the devconfig playbook run as a shared
bringup goal we also implicitly fix bare metal bringup by ensuring we
run the devconfig playbook once at bringup.
We also enhance this with a new step:
KDEVOPS_BRING_UP_DEPS_EARLY
This variable can be augmented by new features for so to be able
to install things using the devconfig playbook before the general
devconfig playbook is run.
The diff stat reveals sharing is best. It will allow us later
to say, for example add ktest support or xfstests-bld, if we want to in a
much easier smoother way, as yet another provisioner.
This should pretty much be a no-op for most deployments, but enhances
bare metal setup so to ensure we run the devconfig playbook. It also
allows makes it easier to add new tools which may be optional which
we want installed early.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
.gitignore | 4 +--
Makefile | 67 ++++++++++++++++++++++++++++++--------
scripts/guestfs.Makefile | 31 +++---------------
scripts/terraform.Makefile | 28 +++-------------
scripts/vagrant.Makefile | 31 ++++--------------
5 files changed, 70 insertions(+), 91 deletions(-)
diff --git a/.gitignore b/.gitignore
index 1d2653c5b26c..d0499ddb2348 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,6 +3,7 @@
*.retry
.kdevops\.depcheck
+.provisioned_once*
guestfs/
@@ -29,11 +30,8 @@ scripts/workflows/lib/__pycache__/
vagrant/kdevops_nodes.yaml
vagrant/.Vagrantfile.generated
vagrant/Vagrantfile
-vagrant/.provisioned_once*
.vagrant/
-terraform/.provisioned_once*
-
include/
# You can override role specific stuff on these
diff --git a/Makefile b/Makefile
index 8ee02f2a9b6c..be686b279e2c 100644
--- a/Makefile
+++ b/Makefile
@@ -85,19 +85,34 @@ endif # CONFIG_NEEDS_LOCAL_DEVELOPMENT_PATH
ANSIBLE_EXTRA_ARGS += $(LOCAL_DEVELOPMENT_ARGS)
-# These should be set as non-empty if you want any generic bring up
-# targets to come up. We support 3 bring up methods:
-#
-# - vagrant: for kvm/virtualbox - will eventually be deprecated
-# - libguestfs: for kvm
-# - terraform: for any cloud provider
-#
-# If you are using bare metal, you don't do bring up, or you'd
-# likely do this yourself. What you *might* need if working
-# with bare metal is provisioning, but our workflows targets
-# provide that. The devconfig ansible role can be also augmented
-# to support many different custom provisioning preferences outside
-# of the scope of workflows. With things like kdump, etc.
+# Provisioning methods should set this to their target which will ensure
+# systems will be up after this.
+KDEVOPS_PROVISION_METHOD :=
+
+# Provisioning methods should set this to their target which will ensure
+# the systems will be removed after this
+KDEVOPS_PROVISION_DESTROY_METHOD :=
+
+# The default guard for ssh provisioning. Provisioning methods can set the
+# KDEVOPS_PROVISIONED_SSH to this if they are OK with the default guard.
+KDEVOPS_PROVISIONED_SSH_DEFAULT_GUARD := .provisioned_once_ssh
+
+# This is empty on purpose so to support terraform which does this for us
+# on the terraform provider and bare metal where it is assumed you already
+# have ssh setup. Later we can grow this for other bare metal setups or to
+# consume / gather existing topologies. To be clear, terraform providers are
+# in charge of figuring out ssh configuration updates for you in kdevops.
+KDEVOPS_PROVISIONED_SSH :=
+
+# You should augment this with targets which should be after ssh is
+# ready to go to each node, and before devconfig playbook is run.
+# The devconfig playbook can be used with tags to ensure deps / setup
+# for early deps are run.
+KDEVOPS_BRING_UP_DEPS_EARLY :=
+
+# This is shared task.
+KDEVOPS_PROVISIONED_DEVCONFIG := .provisioned_once_devconfig
+
KDEVOPS_BRING_UP_DEPS :=
KDEVOPS_DESTROY_DEPS :=
@@ -115,6 +130,32 @@ ifeq (y,$(CONFIG_GUESTFS))
include scripts/guestfs.Makefile
endif
+KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_SSH)
+KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_DEVCONFIG)
+
+$(KDEVOPS_PROVISIONED_DEVCONFIG):
+ $(Q)if [[ "$(CONFIG_KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK)" != "" ]]; then \
+ ansible-playbook $(ANSIBLE_VERBOSE) -i \
+ $(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/$(KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK) ;\
+ fi
+ $(Q)touch $(KDEVOPS_PROVISIONED_DEVCONFIG)
+
+# Provisioning split into 4 steps:
+#
+# 1) Provisioning method, if one is defined
+# 2) Ensuring we can use ansible with ssh, if required
+# 3) Installing early dependencies, if any, which independent of any workflow
+# 4) Optionally run the devconfig playbook for configuration (which may include
+# extra tools)
+#
+# Anything deps after this is dealt with on each respective workflow.
+KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_PROVISION_METHOD)
+KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_PROVISIONED_SSH)
+KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_BRING_UP_DEPS_EARLY)
+KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_PROVISIONED_DEVCONFIG)
+
+KDEVOPS_DESTROY_DEPS += $(KDEVOPS_PROVISION_DESTROY_METHOD)
+
ifeq (y,$(CONFIG_WORKFLOWS))
include workflows/Makefile
endif # CONFIG_WORKFLOWS
diff --git a/scripts/guestfs.Makefile b/scripts/guestfs.Makefile
index 17c8cbd354c1..054ce7a7a3ea 100644
--- a/scripts/guestfs.Makefile
+++ b/scripts/guestfs.Makefile
@@ -5,11 +5,7 @@ GUESTFS_ARGS :=
KDEVOPS_NODES_TEMPLATE := $(KDEVOPS_NODES_ROLE_TEMPLATE_DIR)/guestfs_nodes.j2
KDEVOPS_NODES := guestfs/kdevops_nodes.yaml
-export KDEVOPS_GUESTFS_PROVISIONED_SSH := guestfs/.provisioned_once_ssh
-export KDEVOPS_GUESTFS_PROVISIONED_DEVCONFIG := guestfs/.provisioned_once_devconfig
-
-KDEVOPS_MRPROPER += $(KDEVOPS_GUESTFS_PROVISIONED_SSH)
-KDEVOPS_MRPROPER += $(KDEVOPS_GUESTFS_PROVISIONED_DEVCONFIG)
+export KDEVOPS_PROVISIONED_SSH := $(KDEVOPS_PROVISIONED_SSH_DEFAULT_GUARD)
GUESTFS_ARGS += kdevops_enable_guestfs=True
GUESTFS_ARGS += guestfs_path='$(TOPDIR_PATH)/guestfs'
@@ -53,18 +49,8 @@ GUESTFS_BRINGUP_DEPS :=
GUESTFS_BRINGUP_DEPS += $(9P_HOST_CLONE)
GUESTFS_BRINGUP_DEPS += $(LIBVIRT_PCIE_PASSTHROUGH)
- KDEVOPS_BRING_UP_DEPS := bringup_guestfs
-KDEVOPS_DESTROY_DEPS := destroy_guestfs
-
-# Provisioning goes last
-#
-# Provisioning split into 2 steps:
-# 1) Ensuring we can use ansible with ssh
-# 2) Generic devconfig final configuration (which may include extra tools)
-#
-# Anything deps after this is dealt with on each respective workflow.
-KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_GUESTFS_PROVISIONED_SSH)
-KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_GUESTFS_PROVISIONED_DEVCONFIG)
+KDEVOPS_PROVISION_METHOD := bringup_guestfs
+KDEVOPS_PROVISION_DESTROY_METHOD := destroy_guestfs
9p_linux_clone:
$(Q)make linux-clone
@@ -75,18 +61,11 @@ libvirt_pcie_passthrough_permissions:
playbooks/libvirt_pcie_passthrough.yml \
-e 'ansible_python_interpreter=/usr/bin/python3'
-$(KDEVOPS_GUESTFS_PROVISIONED_SSH):
+$(KDEVOPS_PROVISIONED_SSH):
$(Q)if [[ "$(CONFIG_KDEVOPS_SSH_CONFIG_UPDATE)" == "y" ]]; then \
LIBVIRT_DEFAULT_URI=$(CONFIG_LIBVIRT_URI) $(TOPDIR)/scripts/update_ssh_config_guestfs.py; \
fi
- $(Q)touch $(KDEVOPS_GUESTFS_PROVISIONED_SSH)
-
-$(KDEVOPS_GUESTFS_PROVISIONED_DEVCONFIG):
- $(Q)if [[ "$(CONFIG_KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK)" != "" ]]; then \
- ansible-playbook $(ANSIBLE_VERBOSE) -i \
- $(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/$(KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK) ; \
- fi
- $(Q)touch $(KDEVOPS_GUESTFS_PROVISIONED_DEVCONFIG)
+ $(Q)touch $(KDEVOPS_PROVISIONED_SSH)
bringup_guestfs: $(GUESTFS_BRINGUP_DEPS)
$(Q)$(TOPDIR)/scripts/bringup_guestfs.sh
diff --git a/scripts/terraform.Makefile b/scripts/terraform.Makefile
index 8e95e8cd822b..9c04a89d5a83 100644
--- a/scripts/terraform.Makefile
+++ b/scripts/terraform.Makefile
@@ -2,23 +2,10 @@
TERRAFORM_EXTRA_VARS :=
-export KDEVOPS_TERRAFORM_PROVISIONED_DEVCONFIG := terraform/.provisioned_once_devconfig
-KDEVOPS_MRPROPER += $(KDEVOPS_TERRAFORM_PROVISIONED_DEVCONFIG)
-
-KDEVOPS_BRING_UP_DEPS := bringup_terraform
-# Provisioning goes last
-#
-# Provisioning split into 2 steps:
-# 1) Ensuring we can use ansible with ssh - this is implied by all
-# terraform providers. That is, our terraform providers all have a last
-# provisioning element task which does this update for us as part of
-# the above bringup_terraform.
-# 2) Generic devconfig final configuration (which may include extra tools)
-#
-# Anything deps after this is dealt with on each respective workflow.
-KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_TERRAFORM_PROVISIONED_DEVCONFIG)
-
-KDEVOPS_DESTROY_DEPS := destroy_terraform
+export KDEVOPS_PROVISIONED_SSH := $(KDEVOPS_PROVISIONED_SSH_DEFAULT_GUARD)
+
+KDEVOPS_PROVISION_METHOD := bringup_terraform
+KDEVOPS_PROVISION_DESTROY_METHOD := destroy_terraform
KDEVOPS_NODES_TEMPLATE := $(KDEVOPS_NODES_ROLE_TEMPLATE_DIR)/terraform_nodes.tf.j2
KDEVOPS_NODES := terraform/nodes.tf
@@ -179,13 +166,6 @@ ANSIBLE_EXTRA_ARGS += $(TERRAFORM_EXTRA_VARS)
bringup_terraform:
$(Q)$(TOPDIR)/scripts/bringup_terraform.sh
-$(KDEVOPS_TERRAFORM_PROVISIONED_DEVCONFIG):
- $(Q)if [[ "$(CONFIG_KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK)" != "" ]]; then \
- ansible-playbook $(ANSIBLE_VERBOSE) -i \
- $(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/$(KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK) ;\
- fi
- $(Q)touch $(KDEVOPS_TERRAFORM_PROVISIONED_DEVCONFIG)
-
destroy_terraform:
$(Q)$(TOPDIR)/scripts/destroy_terraform.sh
diff --git a/scripts/vagrant.Makefile b/scripts/vagrant.Makefile
index 825cfd210c6b..85e1c57356de 100644
--- a/scripts/vagrant.Makefile
+++ b/scripts/vagrant.Makefile
@@ -8,13 +8,11 @@ KDEVOPS_NODES := vagrant/kdevops_nodes.yaml
KDEVOPS_VAGRANT_TEMPLATE := $(KDEVOPS_NODES_ROLE_TEMPLATE_DIR)/Vagrantfile.j2
KDEVOPS_VAGRANT_GENERATED:= vagrant/.Vagrantfile.generated
KDEVOPS_VAGRANT := vagrant/Vagrantfile
-export KDEVOPS_VAGRANT_PROVISIONED_SSH := vagrant/.provisioned_once_ssh
-export KDEVOPS_VAGRANT_PROVISIONED_DEVCONFIG := vagrant/.provisioned_once_devconfig
+
+export KDEVOPS_PROVISIONED_SSH := $(KDEVOPS_PROVISIONED_SSH_DEFAULT_GUARD)
KDEVOPS_MRPROPER += $(KDEVOPS_VAGRANT_GENERATED)
KDEVOPS_MRPROPER += $(KDEVOPS_VAGRANT)
-KDEVOPS_MRPROPER += $(KDEVOPS_VAGRANT_PROVISIONED_SSH)
-KDEVOPS_MRPROPER += $(KDEVOPS_VAGRANT_PROVISIONED_DEVCONFIG)
VAGRANT_ARGS += kdevops_vagrant_template_full_path='$(TOPDIR_PATH)/$(KDEVOPS_VAGRANT_TEMPLATE)'
@@ -75,18 +73,8 @@ VAGRANT_BRINGUP_DEPS += $(VAGRANT_PRIVATE_BOX_DEPS)
VAGRANT_BRINGUP_DEPS += $(VAGRANT_9P_HOST_CLONE)
VAGRANT_BRINGUP_DEPS += $(LIBVIRT_PCIE_PASSTHROUGH)
-KDEVOPS_BRING_UP_DEPS := bringup_vagrant
-# Provisioning goes last
-#
-# Provisioning split into 2 steps:
-# 1) Ensuring we can use ansible with ssh
-# 2) Generic devconfig final configuration (which may include extra tools)
-#
-# Anything deps after this is dealt with on each respective workflow.
-KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_VAGRANT_PROVISIONED_SSH)
-KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_VAGRANT_PROVISIONED_DEVCONFIG)
-
-KDEVOPS_DESTROY_DEPS := destroy_vagrant
+KDEVOPS_PROVISION_METHOD := bringup_vagrant
+KDEVOPS_PROVISION_DESTROY_METHOD := destroy_vagrant
extend-extra-args-vagrant:
@if [[ "$(CONFIG_HAVE_VAGRANT_BOX_URL)" == "y" ]]; then \
@@ -108,21 +96,14 @@ libvirt_pcie_passthrough_permissions:
playbooks/libvirt_pcie_passthrough.yml \
-e 'ansible_python_interpreter=/usr/bin/python3'
-$(KDEVOPS_VAGRANT_PROVISIONED_SSH):
+$(KDEVOPS_PROVISIONED_SSH):
$(Q)if [[ "$(CONFIG_KDEVOPS_SSH_CONFIG_UPDATE)" == "y" ]]; then \
ansible-playbook $(ANSIBLE_VERBOSE) --connection=local \
--inventory localhost, \
playbooks/update_ssh_config_vagrant.yml \
-e 'ansible_python_interpreter=/usr/bin/python3' ;\
fi
- $(Q)touch $(KDEVOPS_VAGRANT_PROVISIONED_SSH)
-
-$(KDEVOPS_VAGRANT_PROVISIONED_DEVCONFIG):
- $(Q)if [[ "$(CONFIG_KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK)" != "" ]]; then \
- ansible-playbook $(ANSIBLE_VERBOSE) -i \
- $(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/$(KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK) ;\
- fi
- $(Q)touch $(KDEVOPS_VAGRANT_PROVISIONED_DEVCONFIG)
+ $(Q)touch $(KDEVOPS_PROVISIONED_SSH)
bringup_vagrant: $(VAGRANT_BRINGUP_DEPS)
$(Q)$(TOPDIR)/scripts/bringup_vagrant.sh
--
2.42.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 4/6] provision: move all provisioning things to its own Makefile
2024-01-25 20:35 [PATCH 0/6] Fix ordering for systemd remote journal support Luis Chamberlain
` (2 preceding siblings ...)
2024-01-25 20:35 ` [PATCH 3/6] bringup: share bringup method targe and agument by two steps Luis Chamberlain
@ 2024-01-25 20:35 ` Luis Chamberlain
2024-01-25 20:35 ` [PATCH 5/6] bringup: move journal-server setup early Luis Chamberlain
2024-01-25 20:35 ` [PATCH 6/6] journal-server: fix by adjusting ordering Luis Chamberlain
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-01-25 20:35 UTC (permalink / raw)
To: kdevops, da.gomez, p.raghav; +Cc: Luis Chamberlain
The provisioning aspect of kdevops is creating clutter at the
top level Makefile. Move it all into its own file so to be able
to easily identify related tasks and to easily extend it.
We keep augmenting KDEVOPS_BRING_UP_DEPS with KDEVOPS_BRING_UP_DEPS_EARLY
in the top level Makefile so we can in between add makefiles for tools
which we want to include in between and make it clear these requirements
will be met before KDEVOPS_PROVISIONED_DEVCONFIG, ie running the devconfig
playbook for general setup.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Makefile | 69 +----------------------------------
scripts/provision.Makefile | 74 ++++++++++++++++++++++++++++++++++++++
2 files changed, 75 insertions(+), 68 deletions(-)
create mode 100644 scripts/provision.Makefile
diff --git a/Makefile b/Makefile
index be686b279e2c..67e13edbf6ff 100644
--- a/Makefile
+++ b/Makefile
@@ -85,77 +85,10 @@ endif # CONFIG_NEEDS_LOCAL_DEVELOPMENT_PATH
ANSIBLE_EXTRA_ARGS += $(LOCAL_DEVELOPMENT_ARGS)
-# Provisioning methods should set this to their target which will ensure
-# systems will be up after this.
-KDEVOPS_PROVISION_METHOD :=
-
-# Provisioning methods should set this to their target which will ensure
-# the systems will be removed after this
-KDEVOPS_PROVISION_DESTROY_METHOD :=
-
-# The default guard for ssh provisioning. Provisioning methods can set the
-# KDEVOPS_PROVISIONED_SSH to this if they are OK with the default guard.
-KDEVOPS_PROVISIONED_SSH_DEFAULT_GUARD := .provisioned_once_ssh
-
-# This is empty on purpose so to support terraform which does this for us
-# on the terraform provider and bare metal where it is assumed you already
-# have ssh setup. Later we can grow this for other bare metal setups or to
-# consume / gather existing topologies. To be clear, terraform providers are
-# in charge of figuring out ssh configuration updates for you in kdevops.
-KDEVOPS_PROVISIONED_SSH :=
-
-# You should augment this with targets which should be after ssh is
-# ready to go to each node, and before devconfig playbook is run.
-# The devconfig playbook can be used with tags to ensure deps / setup
-# for early deps are run.
-KDEVOPS_BRING_UP_DEPS_EARLY :=
-
-# This is shared task.
-KDEVOPS_PROVISIONED_DEVCONFIG := .provisioned_once_devconfig
-
-KDEVOPS_BRING_UP_DEPS :=
-KDEVOPS_DESTROY_DEPS :=
-
-include scripts/dynamic-kconfig.Makefile
-
-ifeq (y,$(CONFIG_TERRAFORM))
-include scripts/terraform.Makefile
-endif # CONFIG_TERRAFORM
-
-ifeq (y,$(CONFIG_VAGRANT))
-include scripts/vagrant.Makefile
-endif
-
-ifeq (y,$(CONFIG_GUESTFS))
-include scripts/guestfs.Makefile
-endif
-
-KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_SSH)
-KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_DEVCONFIG)
-
-$(KDEVOPS_PROVISIONED_DEVCONFIG):
- $(Q)if [[ "$(CONFIG_KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK)" != "" ]]; then \
- ansible-playbook $(ANSIBLE_VERBOSE) -i \
- $(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/$(KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK) ;\
- fi
- $(Q)touch $(KDEVOPS_PROVISIONED_DEVCONFIG)
-
-# Provisioning split into 4 steps:
-#
-# 1) Provisioning method, if one is defined
-# 2) Ensuring we can use ansible with ssh, if required
-# 3) Installing early dependencies, if any, which independent of any workflow
-# 4) Optionally run the devconfig playbook for configuration (which may include
-# extra tools)
-#
-# Anything deps after this is dealt with on each respective workflow.
-KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_PROVISION_METHOD)
-KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_PROVISIONED_SSH)
+include scripts/provision.Makefile
KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_BRING_UP_DEPS_EARLY)
KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_PROVISIONED_DEVCONFIG)
-KDEVOPS_DESTROY_DEPS += $(KDEVOPS_PROVISION_DESTROY_METHOD)
-
ifeq (y,$(CONFIG_WORKFLOWS))
include workflows/Makefile
endif # CONFIG_WORKFLOWS
diff --git a/scripts/provision.Makefile b/scripts/provision.Makefile
new file mode 100644
index 000000000000..3767e3332e44
--- /dev/null
+++ b/scripts/provision.Makefile
@@ -0,0 +1,74 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+# Provisioning methods should set this to their target which will ensure
+# systems will be up after this.
+KDEVOPS_PROVISION_METHOD :=
+
+# Provisioning methods should set this to their target which will ensure
+# the systems will be removed after this
+KDEVOPS_PROVISION_DESTROY_METHOD :=
+
+# The default guard for ssh provisioning. Provisioning methods can set the
+# KDEVOPS_PROVISIONED_SSH to this if they are OK with the default guard.
+KDEVOPS_PROVISIONED_SSH_DEFAULT_GUARD := .provisioned_once_ssh
+
+# This is empty on purpose so to support terraform which does this for us
+# on the terraform provider and bare metal where it is assumed you already
+# have ssh setup. Later we can grow this for other bare metal setups or to
+# consume / gather existing topologies. To be clear, terraform providers are
+# in charge of figuring out ssh configuration updates for you in kdevops.
+KDEVOPS_PROVISIONED_SSH :=
+
+# You should augment this with targets which should be after ssh is
+# ready to go to each node, and before devconfig playbook is run.
+# The devconfig playbook can be used with tags to ensure deps / setup
+# for early deps are run.
+KDEVOPS_BRING_UP_DEPS_EARLY :=
+
+# This is shared task.
+KDEVOPS_PROVISIONED_DEVCONFIG := .provisioned_once_devconfig
+
+KDEVOPS_BRING_UP_DEPS :=
+KDEVOPS_DESTROY_DEPS :=
+
+include scripts/dynamic-kconfig.Makefile
+
+ifeq (y,$(CONFIG_TERRAFORM))
+include scripts/terraform.Makefile
+endif # CONFIG_TERRAFORM
+
+ifeq (y,$(CONFIG_VAGRANT))
+include scripts/vagrant.Makefile
+endif
+
+ifeq (y,$(CONFIG_GUESTFS))
+include scripts/guestfs.Makefile
+endif
+
+KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_SSH)
+KDEVOPS_MRPROPER += $(KDEVOPS_PROVISIONED_DEVCONFIG)
+
+$(KDEVOPS_PROVISIONED_DEVCONFIG):
+ $(Q)if [[ "$(CONFIG_KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK)" != "" ]]; then \
+ ansible-playbook $(ANSIBLE_VERBOSE) -i \
+ $(KDEVOPS_HOSTFILE) $(KDEVOPS_PLAYBOOKS_DIR)/$(KDEVOPS_ANSIBLE_PROVISION_PLAYBOOK) ;\
+ fi
+ $(Q)touch $(KDEVOPS_PROVISIONED_DEVCONFIG)
+
+# Provisioning split into 4 steps:
+#
+# 1) Provisioning method, if one is defined
+# 2) Ensuring we can use ansible with ssh, if required
+# 3) Installing early dependencies, if any, which independent of any workflow
+# 4) Optionally run the devconfig playbook for configuration (which may include
+# extra tools)
+#
+# The last two are dealt with in the top level Makefile so to allow us to
+# define early dependencies from the top level Makefile and make it clear
+# that the devconfig playbook runs last.
+#
+# Anything deps after this is dealt with on each respective workflow.
+KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_PROVISION_METHOD)
+KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_PROVISIONED_SSH)
+
+KDEVOPS_DESTROY_DEPS += $(KDEVOPS_PROVISION_DESTROY_METHOD)
--
2.42.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 5/6] bringup: move journal-server setup early
2024-01-25 20:35 [PATCH 0/6] Fix ordering for systemd remote journal support Luis Chamberlain
` (3 preceding siblings ...)
2024-01-25 20:35 ` [PATCH 4/6] provision: move all provisioning things to its own Makefile Luis Chamberlain
@ 2024-01-25 20:35 ` Luis Chamberlain
2024-01-25 20:35 ` [PATCH 6/6] journal-server: fix by adjusting ordering Luis Chamberlain
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-01-25 20:35 UTC (permalink / raw)
To: kdevops, da.gomez, p.raghav; +Cc: Luis Chamberlain
Now that we have done the cleaning up of the provisioners and have
defined an early target task which we can piggy the back on top before
the devconfig playbook is run, move the journal server setup early.
This does not make any functional changes to the file to make it easier
to review the functional changes, those will be done next.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Makefile | 2 ++
scripts/bringup.Makefile | 56 ---------------------------------
scripts/journal-server.Makefile | 55 ++++++++++++++++++++++++++++++++
3 files changed, 57 insertions(+), 56 deletions(-)
create mode 100644 scripts/journal-server.Makefile
diff --git a/Makefile b/Makefile
index 67e13edbf6ff..4ab805ae7ff5 100644
--- a/Makefile
+++ b/Makefile
@@ -86,6 +86,8 @@ endif # CONFIG_NEEDS_LOCAL_DEVELOPMENT_PATH
ANSIBLE_EXTRA_ARGS += $(LOCAL_DEVELOPMENT_ARGS)
include scripts/provision.Makefile
+include scripts/journal-server.Makefile
+
KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_BRING_UP_DEPS_EARLY)
KDEVOPS_BRING_UP_DEPS += $(KDEVOPS_PROVISIONED_DEVCONFIG)
diff --git a/scripts/bringup.Makefile b/scripts/bringup.Makefile
index 6e64b6c12172..5a4778473eab 100644
--- a/scripts/bringup.Makefile
+++ b/scripts/bringup.Makefile
@@ -33,50 +33,6 @@ ifeq (y,$(CONFIG_KDEVOPS_SETUP_SIW))
KDEVOPS_BRING_UP_DEPS += siw
endif # KDEVOPS_SETUP_SIW
-ifeq (y,$(CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE))
-
-JOURNAL_REMOTE:=$(subst ",,$(CONFIG_DEVCONFIG_SYSTEMD_JOURNAL_REMOTE_URL))
-ANSIBLE_EXTRA_ARGS += devconfig_systemd_journal_remote_url=$(JOURNAL_REMOTE)
-ANSIBLE_EXTRA_ARGS += devconfig_enable_systemd_journal_remote='True'
-
-journal-client:
- @$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -l baseline,dev \
- -f 30 -i hosts \
- --extra-vars '{ kdevops_cli_install: True }' \
- --tags vars_simple,journal \
- $(KDEVOPS_PLAYBOOKS_DIR)/devconfig.yml
-
-journal-server:
- @$(Q)ansible-playbook $(ANSIBLE_VERBOSE) --connection=local \
- --inventory localhost, \
- $(KDEVOPS_PLAYBOOKS_DIR)/install_systemd_journal_remote.yml \
- -e 'ansible_python_interpreter=/usr/bin/python3'
-
-journal-restart:
- @$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -l baseline,dev \
- -f 30 -i hosts \
- --tags vars_extra,journal-upload-restart \
- $(KDEVOPS_PLAYBOOKS_DIR)/devconfig.yml
-
-journal-status:
- @$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -l baseline,dev \
- -f 30 -i hosts \
- --tags vars_extra,journal-status \
- $(KDEVOPS_PLAYBOOKS_DIR)/devconfig.yml
-
-journal-ls:
- @$(Q)./workflows/kdevops/scripts/jounal-ls.sh /var/log/journal/remote/
-
-journal-ln:
- @$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -l baseline,dev \
- -f 30 -i hosts \
- --tags vars_extra,journal_ln \
- $(KDEVOPS_PLAYBOOKS_DIR)/devconfig.yml
-
-KDEVOPS_BRING_UP_DEPS += journal-server
-
-endif
-
update_etc_hosts:
$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
-f 30 -i hosts playbooks/update_etc_hosts.yml
@@ -100,18 +56,6 @@ bringup-setup-help-menu:
HELP_TARGETS += bringup-setup-help-menu
-ifeq (y,$(CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE))
-journal-help:
- @echo "journal-server - Setup systemd-journal-remote on localhost"
- @echo "journal-client - Setup systemd-journal-upload on clients"
- @echo "journal-restart - Restart client upload service"
- @echo "journal-status - Ensure systemd-journal-remote works"
- @echo "journal-ls - List journals available and sizes"
- @echo "journal-ln - Add symlinks with hostnames"
-
-HELP_TARGETS += journal-help
-endif
-
bringup-setup-help-end:
@echo ""
diff --git a/scripts/journal-server.Makefile b/scripts/journal-server.Makefile
new file mode 100644
index 000000000000..3c32d5df982e
--- /dev/null
+++ b/scripts/journal-server.Makefile
@@ -0,0 +1,55 @@
+# SPDX-License-Identifier: copyleft-next-0.3.1
+
+ifeq (y,$(CONFIG_DEVCONFIG_ENABLE_SYSTEMD_JOURNAL_REMOTE))
+
+JOURNAL_REMOTE:=$(subst ",,$(CONFIG_DEVCONFIG_SYSTEMD_JOURNAL_REMOTE_URL))
+ANSIBLE_EXTRA_ARGS += devconfig_systemd_journal_remote_url=$(JOURNAL_REMOTE)
+ANSIBLE_EXTRA_ARGS += devconfig_enable_systemd_journal_remote='True'
+
+journal-client:
+ @$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -l baseline,dev \
+ -f 30 -i hosts \
+ --extra-vars '{ kdevops_cli_install: True }' \
+ --tags vars_simple,journal \
+ $(KDEVOPS_PLAYBOOKS_DIR)/devconfig.yml
+
+journal-server:
+ @$(Q)ansible-playbook $(ANSIBLE_VERBOSE) --connection=local \
+ --inventory localhost, \
+ $(KDEVOPS_PLAYBOOKS_DIR)/install_systemd_journal_remote.yml \
+ -e 'ansible_python_interpreter=/usr/bin/python3'
+
+journal-restart:
+ @$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -l baseline,dev \
+ -f 30 -i hosts \
+ --tags vars_extra,journal-upload-restart \
+ $(KDEVOPS_PLAYBOOKS_DIR)/devconfig.yml
+
+journal-status:
+ @$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -l baseline,dev \
+ -f 30 -i hosts \
+ --tags vars_extra,journal-status \
+ $(KDEVOPS_PLAYBOOKS_DIR)/devconfig.yml
+
+journal-ls:
+ @$(Q)./workflows/kdevops/scripts/jounal-ls.sh /var/log/journal/remote/
+
+journal-ln:
+ @$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -l baseline,dev \
+ -f 30 -i hosts \
+ --tags vars_extra,journal_ln \
+ $(KDEVOPS_PLAYBOOKS_DIR)/devconfig.yml
+
+KDEVOPS_BRING_UP_DEPS += journal-server
+
+journal-help:
+ @echo "journal-server - Setup systemd-journal-remote on localhost"
+ @echo "journal-client - Setup systemd-journal-upload on clients"
+ @echo "journal-restart - Restart client upload service"
+ @echo "journal-status - Ensure systemd-journal-remote works"
+ @echo "journal-ls - List journals available and sizes"
+ @echo "journal-ln - Add symlinks with hostnames"
+
+HELP_TARGETS += journal-help
+
+endif
--
2.42.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [PATCH 6/6] journal-server: fix by adjusting ordering
2024-01-25 20:35 [PATCH 0/6] Fix ordering for systemd remote journal support Luis Chamberlain
` (4 preceding siblings ...)
2024-01-25 20:35 ` [PATCH 5/6] bringup: move journal-server setup early Luis Chamberlain
@ 2024-01-25 20:35 ` Luis Chamberlain
5 siblings, 0 replies; 7+ messages in thread
From: Luis Chamberlain @ 2024-01-25 20:35 UTC (permalink / raw)
To: kdevops, da.gomez, p.raghav; +Cc: Luis Chamberlain
Now that we have done all the legwork to address ordering for bringup
and allow for items to be setup / installed with the devconfig playbook
for specific features, prior to running all generic devconfig tasks,
move the systemd remote journal server and client setup to run early.
Fix also ensuring we only try to setup the server once.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
playbooks/roles/devconfig/tasks/main.yml | 1 +
scripts/journal-server.Makefile | 3 ++-
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/playbooks/roles/devconfig/tasks/main.yml b/playbooks/roles/devconfig/tasks/main.yml
index 061d05acdc80..3e38d8f49cdf 100644
--- a/playbooks/roles/devconfig/tasks/main.yml
+++ b/playbooks/roles/devconfig/tasks/main.yml
@@ -485,6 +485,7 @@
ansible.builtin.systemd_service:
name: systemd-journal-remote.service
state: started
+ run_once: true
when:
- devconfig_enable_systemd_journal_remote|bool
diff --git a/scripts/journal-server.Makefile b/scripts/journal-server.Makefile
index 3c32d5df982e..23a149bda87c 100644
--- a/scripts/journal-server.Makefile
+++ b/scripts/journal-server.Makefile
@@ -40,7 +40,8 @@ journal-ln:
--tags vars_extra,journal_ln \
$(KDEVOPS_PLAYBOOKS_DIR)/devconfig.yml
-KDEVOPS_BRING_UP_DEPS += journal-server
+KDEVOPS_BRING_UP_DEPS_EARLY += journal-server
+KDEVOPS_BRING_UP_DEPS_EARLY += journal-client
journal-help:
@echo "journal-server - Setup systemd-journal-remote on localhost"
--
2.42.0
^ permalink raw reply related [flat|nested] 7+ messages in thread