Re: [PATCH 1/4] workflows: Workflow stub for Jorge Mora's nfstest suite

[Scott Mayhew <smayhew@redhat.com>]

On Wed, 10 Apr 2024, cel@kernel.org wrote:

> From: Chuck Lever <chuck.lever@oracle.com>
> 
> The nfstest suite is a detailed test of NFS functionality for NFS
> versions 3 and newer, including delegation, attribute cache
> consistency, sparse files, and server-to-server copy offload.
> 
> When configured in dedicated mode, each nfstest suite is run in a
> separate target node to achieve good parallelism. You can select
> specific test groups to run via Kconfig options.

I like the idea of running the tests in parallel, but would it be
possible to also add an option to spin up a single node and run the
tests sequentially?  I'm running kdevops via buildbot.  I have buildbot
set up to monitor various upstream trees and trigger kdevops whenever
new commits are pushed.  In addition, I have some nightly jobs that run
against RHEL 8 & RHEL 9 kernels.  I'd like to throw nfstest into the
mix, but I'm concerned that if each one of those is spinning up 8+ VMs
my hypervisor will be under water.  Just a thought.

-Scott
> 
> This commit introduces the workflow's documentation and Kconfig
> apparatus to show how this workflow can be configured. Following
> commits will bring in the kdevops components to get the workflow
> working.
> 
> The careful reader will note that not every nfstest group is
> available here. Three groups require the use of a second client,
> and I haven't figured out how that needs to work. For now, those
> groups are not included in this series.
> 
> Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> ---
>  .gitignore                                    |  1 +
>  README.md                                     | 12 ++-
>  docs/nfstest.md                               | 21 ++++
>  kconfigs/workflows/Kconfig                    | 25 +++++
>  scripts/workflows/nfstest/run_kernel_ci.sh    |  1 +
>  .../workflows/nfstest/run_kernel_ci_kotd.sh   |  1 +
>  scripts/workflows/nfstest/run_loop.sh         | 58 +++++++++++
>  workflows/Makefile                            |  5 +
>  workflows/nfstest/Kconfig                     | 87 ++++++++++++++++
>  workflows/nfstest/Makefile                    | 99 +++++++++++++++++++
>  10 files changed, 309 insertions(+), 1 deletion(-)
>  create mode 100644 docs/nfstest.md
>  create mode 120000 scripts/workflows/nfstest/run_kernel_ci.sh
>  create mode 120000 scripts/workflows/nfstest/run_kernel_ci_kotd.sh
>  create mode 100755 scripts/workflows/nfstest/run_loop.sh
>  create mode 100644 workflows/nfstest/Kconfig
>  create mode 100644 workflows/nfstest/Makefile
> 
> diff --git a/.gitignore b/.gitignore
> index 5cb41682dd45..3a1ecd48d84f 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -62,6 +62,7 @@ workflows/demos/reboot-limit/results/
>  
>  workflows/gitr/results/
>  workflows/ltp/results/
> +workflows/nfstest/results/
>  
>  playbooks/roles/linux-mirror/linux-mirror-systemd/mirrors.yaml
>  # We should strive to ignore all and port them to the new mirrors.yaml
> diff --git a/README.md b/README.md
> index 96e048bc8405..ea6b41004767 100644
> --- a/README.md
> +++ b/README.md
> @@ -135,7 +135,17 @@ To test a kernel using the ltp suite, enable the ltp workflow and then run:
>    * `make ltp`
>    * `make ltp-baseline`
>  
> -For more details see [kdevops gitr docs](docs/gitr.md)
> +For more details see [kdevops ltp docs](docs/ltp.md)
> +
> +### Start running the nfstest suite in 2 commands
> +
> +To test a kernel using the nfstest suite, enable the nfstest workflow and
> +then run:
> +
> +  * `make nfstest`
> +  * `make nfstest-baseline`
> +
> +For more details see [kdevops nfstest docs](docs/nfstest.md)
>  
>  ### Runs some kernel selftests in a parallel manner
>  
> diff --git a/docs/nfstest.md b/docs/nfstest.md
> new file mode 100644
> index 000000000000..b358389c2571
> --- /dev/null
> +++ b/docs/nfstest.md
> @@ -0,0 +1,21 @@
> +# kdevops nfstest suite
> +
> +kdevops can run Jorge Mora's nfstest suite against an NFS server.
> +
> +Run `make menuconfig` and select:
> +
> +  Target workflows -> Dedicated target Linux test workflow ->nfstest
> +
> +Then configure the test parameters by going to:
> +
> +  Target workflows -> Configure and run the nfstest suite
> +
> +Choose the location of the repo that contains the version of nfstest
> +you want to use for the test.
> +
> +Then, run:
> +
> +  * `make`
> +  * `make bringup`
> +  * `make nfstest`
> +  * `make nfstest-baseline`
> diff --git a/kconfigs/workflows/Kconfig b/kconfigs/workflows/Kconfig
> index ff6372b7e98f..2fd5bf31347f 100644
> --- a/kconfigs/workflows/Kconfig
> +++ b/kconfigs/workflows/Kconfig
> @@ -166,6 +166,13 @@ config KDEVOPS_WORKFLOW_DEDICATE_LTP
>  	  This will dedicate your configuration to running only the
>  	  ltp workflow in separate target nodes per testing group.
>  
> +config KDEVOPS_WORKFLOW_DEDICATE_NFSTEST
> +	bool "nfstest"
> +	select KDEVOPS_WORKFLOW_ENABLE_NFSTEST
> +	help
> +	  This will dedicate your configuration to running only the
> +	  nfstest workflow in separate target nodes per testing group.
> +
>  endchoice
>  
>  endif
> @@ -239,6 +246,14 @@ config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_LTP
>  	  Select this option if you want to provision ltp on a single
>  	  target node for by-hand testing.
>  
> +config KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_NFSTEST
> +	bool "nfstest"
> +	select KDEVOPS_WORKFLOW_ENABLE_NFSTEST
> +	depends on LIBVIRT || TERRAFORM_PRIVATE_NET
> +	help
> +	  Select this option if you want to provision nfstest on a
> +	  single target node for by-hand testing.
> +
>  endif # !WORKFLOWS_DEDICATED_WORKFLOW
>  
>  config KDEVOPS_WORKFLOW_ENABLE_FSTESTS
> @@ -312,6 +327,16 @@ source "workflows/ltp/Kconfig"
>  endmenu
>  endif # KDEVOPS_WORKFLOW_ENABLE_GITR
>  
> +config KDEVOPS_WORKFLOW_ENABLE_NFSTEST
> +	bool
> +	default y if KDEVOPS_WORKFLOW_NOT_DEDICATED_ENABLE_NFSTEST || KDEVOPS_WORKFLOW_DEDICATE_NFSTEST
> +
> +if KDEVOPS_WORKFLOW_ENABLE_NFSTEST
> +menu "Configure and run the nfstest suite"
> +source "workflows/nfstest/Kconfig"
> +endmenu
> +endif # KDEVOPS_WORKFLOW_ENABLE_NFSTEST
> +
>  config KDEVOPS_WORKFLOW_GIT_CLONES_KDEVOPS_GIT
>  	bool
>  	default y if KDEVOPS_WORKFLOW_ENABLE_FSTESTS || KDEVOPS_WORKFLOW_ENABLE_BLKTESTS
> diff --git a/scripts/workflows/nfstest/run_kernel_ci.sh b/scripts/workflows/nfstest/run_kernel_ci.sh
> new file mode 120000
> index 000000000000..4fd5dc5e8e13
> --- /dev/null
> +++ b/scripts/workflows/nfstest/run_kernel_ci.sh
> @@ -0,0 +1 @@
> +../generic/run_kernel_ci.sh
> \ No newline at end of file
> diff --git a/scripts/workflows/nfstest/run_kernel_ci_kotd.sh b/scripts/workflows/nfstest/run_kernel_ci_kotd.sh
> new file mode 120000
> index 000000000000..8f94d6ba4f52
> --- /dev/null
> +++ b/scripts/workflows/nfstest/run_kernel_ci_kotd.sh
> @@ -0,0 +1 @@
> +../kotd/run_kernel_ci_kotd.sh
> \ No newline at end of file
> diff --git a/scripts/workflows/nfstest/run_loop.sh b/scripts/workflows/nfstest/run_loop.sh
> new file mode 100755
> index 000000000000..22f80ca4e04a
> --- /dev/null
> +++ b/scripts/workflows/nfstest/run_loop.sh
> @@ -0,0 +1,58 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: copyleft-next-0.3.1
> +
> +# Part of kdevops kernel-ci, this is the script which will run the test workflow
> +# as many times as indicated up to CONFIG_KERNEL_CI_STEADY_STATE_GOAL
> +
> +source ${TOPDIR}/.config
> +source ${TOPDIR}/scripts/lib.sh
> +
> +WORKFLOWDIR=${TOPDIR}/workflows/nfstest
> +
> +COUNT=1
> +
> +run_loop()
> +{
> +	while true; do
> +		echo "== kernel-ci nfstest test loop $COUNT start: $(date)" > $KERNEL_CI_FAIL_LOG
> +		echo "/usr/bin/time -f %E make nfstest-baseline" >> $KERNEL_CI_FAIL_LOG
> +		/usr/bin/time -p -o $KERNEL_CI_LOGTIME make nfstest-baseline >> $KERNEL_CI_FAIL_LOG
> +		ANSIBLE_CALL_RET=$?
> +		echo "End   $COUNT: $(date)" >> $KERNEL_CI_FAIL_LOG
> +		cat $KERNEL_CI_LOGTIME >> $KERNEL_CI_FAIL_LOG
> +		echo "git status:" >> $KERNEL_CI_FAIL_LOG
> +		git status >> $KERNEL_CI_FAIL_LOG
> +		echo "Results:" >> $KERNEL_CI_FAIL_LOG
> +
> +		rm -f $KERNEL_CI_DIFF_LOG
> +
> +		if [[ "$ANSIBLE_CALL_RET" -ne 0 ]]; then
> +			echo "Test  $COUNT: FAILED!" >> $KERNEL_CI_DIFF_LOG
> +			echo "== Test loop count $COUNT" >> $KERNEL_CI_DIFF_LOG
> +			echo "$(git describe)" >> $KERNEL_CI_DIFF_LOG
> +			git diff >> $KERNEL_CI_DIFF_LOG
> +			cat $KERNEL_CI_DIFF_LOG >> $KERNEL_CI_FAIL_LOG
> +			cat $KERNEL_CI_FAIL_LOG >> $KERNEL_CI_FULL_LOG
> +			echo $COUNT > $KERNEL_CI_FAIL_FILE
> +			exit 1
> +		else
> +			echo "Test  $COUNT: OK!" >> $KERNEL_CI_FAIL_LOG
> +			echo "----------------------------------------------------------------" >> $KERNEL_CI_FAIL_LOG
> +			cat $KERNEL_CI_FAIL_LOG >> $KERNEL_CI_FULL_LOG
> +		fi
> +
> +		# This let's us keep track of which loop count was last successful
> +		echo $COUNT > $KERNEL_CI_OK_FILE
> +
> +		let COUNT=$COUNT+1
> +		if [[ "$CONFIG_KERNEL_CI_ENABLE_STEADY_STATE" == "y" &&
> +		      "$COUNT" -gt "$CONFIG_KERNEL_CI_STEADY_STATE_GOAL" ]]; then
> +			exit 0
> +		fi
> +		sleep 1
> +	done
> +}
> +
> +rm -f $KERNEL_CI_FAIL_FILE $KERNEL_CI_OK_FILE
> +echo "= kernel-ci full log" > $KERNEL_CI_FULL_LOG
> +run_loop
> diff --git a/workflows/Makefile b/workflows/Makefile
> index 7e3e5e351a49..e305f2629abf 100644
> --- a/workflows/Makefile
> +++ b/workflows/Makefile
> @@ -53,6 +53,11 @@ WORKFLOW_ARGS += kdevops_workflow_enable_ltp='True'
>  include workflows/ltp/Makefile
>  endif # CONFIG_KDEVOPS_WORKFLOW_ENABLE_LTP == y
>  
> +ifeq (y,$(CONFIG_KDEVOPS_WORKFLOW_ENABLE_NFSTEST))
> +WORKFLOW_ARGS += kdevops_workflow_enable_nfstest='True'
> +include workflows/nfstest/Makefile
> +endif # CONFIG_KDEVOPS_WORKFLOW_ENABLE_NFSTEST == y
> +
>  ANSIBLE_EXTRA_ARGS += $(WORKFLOW_ARGS)
>  ANSIBLE_EXTRA_ARGS_SEPARATED += $(WORKFLOW_ARGS_SEPARATED)
>  ANSIBLE_EXTRA_ARGS_DIRECT += $(WORKFLOW_ARGS_DIRECT)
> diff --git a/workflows/nfstest/Kconfig b/workflows/nfstest/Kconfig
> new file mode 100644
> index 000000000000..921ce5fe0a83
> --- /dev/null
> +++ b/workflows/nfstest/Kconfig
> @@ -0,0 +1,87 @@
> +if KDEVOPS_WORKFLOW_ENABLE_NFSTEST
> +
> +config NFSTEST_USE_KDEVOPS_NFSD
> +	bool "Provision and use the kdevops NFS server"
> +	select KDEVOPS_SETUP_NFSD
> +	default y
> +	help
> +	  Select this option to have kdevops provision an additional
> +	  target node to be used as the NFS server for testing.
> +
> +	  Say N if you have a reachable NFS server to test against, and
> +	  specify that server's hostname using the
> +	  NFSTEST_NFS_SERVER_HOST option.
> +
> +config NFSTEST_NFS_SERVER_HOST
> +	string "Test NFS Server hostname"
> +	default ""
> +	depends on !NFSTEST_USE_KDEVOPS_NFSD
> +	help
> +	  The NFS server hostname or address to test against. Tests
> +	  expect that an appropriate share is already available for
> +	  them to mount and use.
> +
> +config NFSTEST_MNT
> +	string "Where clients mount the file system under test"
> +	default "/mnt/t"
> +	help
> +	  The directory over which to mount the file system under test.
> +
> +config NFSTEST_REPO
> +	string "The nfstest repository"
> +	default "git://git.linux-nfs.org/projects/mora/nfstest.git"
> +	help
> +	  The nfstest repository to clone.
> +
> +config NFSTEST_REPO_COMMIT
> +	string "The version of nfstest to check out"
> +	default "v3.2"
> +	help
> +	  The version of nfstest to be used for the test.
> +
> +if KDEVOPS_WORKFLOW_DEDICATE_NFSTEST
> +
> +config NFSTEST_TEST_GROUP_ALLOC
> +	bool "Enable the nfstest_alloc testing section"
> +	default n
> +	help
> +	  Provision a target node to test NFSv4.2 space reservation.
> +
> +config NFSTEST_TEST_GROUP_DIO
> +	bool "Enable the nfstest_dio testing section"
> +	default n
> +	help
> +	  Provision target nodes to run direct I/O tests.
> +
> +config NFSTEST_TEST_GROUP_INTEROP
> +	bool "Enable the nfstest_interop testing section"
> +	default y
> +	help
> +	  Provision target nodes to run NFS interoperability tests.
> +
> +config NFSTEST_TEST_GROUP_LOCK
> +	bool "Enable the nfstest_lock testing section"
> +	default n
> +	help
> +	  Provision target nodes to test NFS file locking.
> +
> +config NFSTEST_TEST_GROUP_POSIX
> +	bool "Enable the nfstest_posix testing section"
> +	default n
> +	help
> +	  Provision a target node to run POSIX file access tests.
> +
> +config NFSTEST_TEST_GROUP_SPARSE
> +	bool "Enable the nfstest_sparse testing section"
> +	help
> +	  Provision a target node to test support for sparse files.
> +
> +config NFSTEST_TEST_GROUP_SSC
> +	bool "Enable the nfstest_ssc testing section"
> +	help
> +	  Provision target nodes to test the operation of server-side
> +	  copy.
> +
> +endif # KDEVOPS_WORKFLOW_DEDICATE_NFSTEST
> +
> +endif # KDEVOPS_WORKFLOW_ENABLE_GITR
> diff --git a/workflows/nfstest/Makefile b/workflows/nfstest/Makefile
> new file mode 100644
> index 000000000000..638ec1fc83dd
> --- /dev/null
> +++ b/workflows/nfstest/Makefile
> @@ -0,0 +1,99 @@
> +ifeq (y,$(CONFIG_WORKFLOWS_DEDICATED_WORKFLOW))
> +export KDEVOPS_HOSTS_TEMPLATE := nfstest.j2
> +endif # CONFIG_WORKFLOWS_DEDICATED_WORKFLOW
> +
> +ifeq (y,$(CONFIG_NFSTEST_USE_KDEVOPS_NFSD))
> +NFSTEST_ARGS += nfstest_nfs_server_host='$(subst ",,$(CONFIG_KDEVOPS_HOSTS_PREFIX))-nfsd'
> +NFSTEST_ARGS += nfstest_nfs_use_kdevops_nfsd='true'
> +else # CONFIG_NFSTEST_USE_KDEVOPS_NFSD
> +NFSTEST_ARGS += nfstest_nfs_server_host='$(subst ",,$(CONFIG_NFSTEST_NFS_SERVER_HOSTNAME))'
> +NFSTEST_ARGS += nfstest_nfs_server_export='$(subst ",,$(CONFIG_NFSTEST_NFS_SERVER_EXPORT))'
> +NFSTEST_ARGS += nfstest_nfs_use_kdevops_nfsd='false'
> +endif # CONFIG_NFSTEST_USE_KDEVOPS_NFSD
> +
> +NFSTEST_MNT:=$(subst ",,$(CONFIG_NFSTEST_MNT))
> +NFSTEST_ARGS += nfstest_mnt=$(NFSTEST_MNT)
> +
> +NFSTEST_REPO:=$(subst ",,$(CONFIG_NFSTEST_REPO))
> +NFSTEST_ARGS += nfstest_repo=$(NFSTEST_REPO)
> +
> +NFSTEST_REPO_COMMIT:=$(subst ",,$(CONFIG_NFSTEST_REPO_COMMIT))
> +NFSTEST_ARGS += nfstest_repo_commit=$(NFSTEST_REPO_COMMIT)
> +
> +ifeq (y,$(CONFIG_NFSTEST_TEST_GROUP_ALLOC))
> +NFSTEST_ENABLED_TEST_GROUPS += alloc
> +endif
> +ifeq (y,$(CONFIG_NFSTEST_TEST_GROUP_DIO))
> +NFSTEST_ENABLED_TEST_GROUPS += dio
> +endif
> +ifeq (y,$(CONFIG_NFSTEST_TEST_GROUP_INTEROP))
> +NFSTEST_ENABLED_TEST_GROUPS += interop
> +endif
> +ifeq (y,$(CONFIG_NFSTEST_TEST_GROUP_LOCK))
> +NFSTEST_ENABLED_TEST_GROUPS += lock
> +endif
> +ifeq (y,$(CONFIG_NFSTEST_TEST_GROUP_POSIX))
> +NFSTEST_ENABLED_TEST_GROUPS += posix
> +endif
> +ifeq (y,$(CONFIG_NFSTEST_TEST_GROUP_SPARSE))
> +NFSTEST_ENABLED_TEST_GROUPS += sparse
> +endif
> +ifeq (y,$(CONFIG_NFSTEST_TEST_GROUP_SSC))
> +NFSTEST_ENABLED_TEST_GROUPS += ssc
> +endif
> +
> +WORKFLOW_ARGS += $(NFSTEST_ARGS)
> +WORKFLOW_ARGS_SEPARATED += nfstest_enabled_test_groups='$(subst $(space),+,$(NFSTEST_ENABLED_TEST_GROUPS))'
> +
> +NFSTEST_KERNEL_CI_LOOP := false
> +NFSTEST_KERNEL_CI_LOOP_KOTD := false
> +
> +ifeq (y,$(CONFIG_KERNEL_CI))
> +NFSTEST_KERNEL_CI_LOOP      := scripts/workflows/nfstest/run_kernel_ci.sh
> +NFSTEST_KERNEL_CI_LOOP_KOTD := scripts/workflows/nfstest/run_kernel_ci_kotd.sh
> +endif # CONFIG_KERNEL_CI
> +
> +nfstest:
> +	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) -l baseline,dev \
> +		-f 30 -i hosts playbooks/nfstest.yml \
> +		--skip-tags run_tests,copy_results
> +
> +nfstest-baseline:
> +	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
> +		-f 30 -i hosts -l baseline playbooks/nfstest.yml \
> +		--tags vars,run_tests,copy_results \
> +		--extra-vars=@./extra_vars.yaml
> +
> +nfstest-baseline-loop:
> +	$(Q)$(NFSTEST_KERNEL_CI_LOOP) baseline
> +
> +nfstest-baseline-kernelci:
> +	$(Q)$(NFSTEST_KERNEL_CI_LOOP_KOTD) baseline
> +
> +nfstest-dev-baseline:
> +	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
> +		-f 30 -i hosts -l dev playbooks/nfstest.yml \
> +		--tags vars,run_tests,copy_results \
> +		--extra-vars=@./extra_vars.yaml
> +
> +nfstest-dev-loop:
> +	$(Q)$(NFSTEST_KERNEL_CI_LOOP) dev
> +
> +nfstest-dev-kernelci:
> +	$(Q)$(NFSTEST_KERNEL_CI_LOOP_KOTD) dev
> +
> +nfstest-dev-reset:
> +	$(Q)ansible-playbook $(ANSIBLE_VERBOSE) \
> +		-f 30 -i hosts -l dev playbooks/nfstest.yml \
> +		--tags vars,reset \
> +		--extra-vars=@./extra_vars.yaml
> +
> +nfstest-help-menu:
> +	@echo "nfstest options:"
> +	@echo "nfstest                              - Git clone nfstest and install it"
> +	@echo "nfstest-{baseline,dev}               - Run selected nfstests on baseline or dev hosts and collect results"
> +	@echo "nfstest-{baseline,dev}-loop"         - Run nfstest in a loop until error or steady state
> +	@echo "nfstest-{baseline,dev}-kernelci      - Run nfstest kernel-ci loop"
> +	@echo ""
> +
> +HELP_TARGETS += nfstest-help-menu
> -- 
> 2.44.0
> 
>