Re: [PATCH kdevops] fstests: provide kconfig guidance for SOAK_DURATION

["Darrick J. Wong" <djwong@kernel.org>]

On Thu, Jan 25, 2024 at 02:10:37PM -0800, Luis Chamberlain wrote:
> The kdevops test runner has supported a custom SOAK_DURATION for
> fstests, however we were not providing any guidance. This means folks
> likely disable this. Throw a bone and provide some basic guidance and
> use 2.5 hours as the default value. There are about 46 tests today
> which use soak duration, this means if you are testing serially it
> increase total test time by about 5 days than the previously known
> total test time.
> 
> Note that if you are using kernel-ci and using a max loop goal of 100
> that means 500 days extra, so about 1.3 years extra total test time.
> If enabling soak duration you may want to then re-evaluate your loop
> target goal for kernel-ci for kdevops.

Yikes, I wouldn't combine multiple runs with large SOAK_DURATION. ;)

You all might consider kicking all the soak tests over to a separate VM
or VMs so that the long soak test do not hold up the rest of the run.

> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
> ---
> 
> Chandan, Amir, lemme know what you think of a default 2.5 hours default
> if soak duration is enabled. The only thing is the math indicates that
> if you are going to enable kernel-ci we won't finish this year.
> 
> To be clear, we've picked up testing with soak duration seriously for
> our LBS testing. It is why we've been able to find pretty hard to
> reproduce issues even on the page cache for the baseline [0], ie, without
> LBS. While folks have seemed to have found value in adopting 2.5 hours
> and of the results we have found, it obviously means a scaling issue
> to consider to decide when we're done with testing our baseline.
> 
> At first I wrote this patch just to provide basic guidance for kdevops,
> but after doing a bit of the math on how it also extends total test
> time, *with* our kernel-ci effort, it reveals clearly we should probably
> reconsider lowering the kernel-ci threshold a bit if adopting soak
> duration.
> 
> CC'ing a bit wider audience so to get a bit better idea of what folks
> might consider a sensible value for your own testing too. From what
> we've been observing, SOAK_DURATION allows us to catch bugs faster than
> just increasing the kernel-ci count, however, using both let's us catch
> even more bugs too.
> 
> To help *reduce* the amount of time to test we've deployed many kdevops
> XFS clusters to help test the baseline. This is why our count time on
> kernel-ci no is about 50-60 with a soak duration of about 2.5 hours.
> 
> Also please not that the reported bugs so far are the ones with crashes,
> there are other failures too, but we just haven't had the time to disect
> and report failures which are non-fatal (crashes) as crashes have been
> our priority.
> 
> [0] https://github.com/linux-kdevops/kdevops/blob/master/docs/xfs-bugs.md
> 
>  playbooks/roles/fstests/defaults/main.yml |  3 +
>  workflows/fstests/Kconfig                 | 89 ++++++++++++++++++++---
>  workflows/fstests/Makefile.sparsefiles    |  4 +
>  3 files changed, 87 insertions(+), 9 deletions(-)
> 
> diff --git a/playbooks/roles/fstests/defaults/main.yml b/playbooks/roles/fstests/defaults/main.yml
> index 2f70f9549cde..4a1f5dec5827 100644
> --- a/playbooks/roles/fstests/defaults/main.yml
> +++ b/playbooks/roles/fstests/defaults/main.yml
> @@ -30,6 +30,9 @@ fstests_test_logdev_mkfs_opts: "/dev/null"
>  fstests_test_dev_zns: "/dev/null"
>  fstests_zns_enabled: False
>  
> +fstests_soak_duration_enable: False
> +fstests_soak_duration: 0
> +
>  fstests_uses_no_devices: False
>  fstests_generate_simple_config_enable: False
>  fstests_generate_nvme_live_config_enable: False
> diff --git a/workflows/fstests/Kconfig b/workflows/fstests/Kconfig
> index 985a7847b6c7..bbd8927b3cd3 100644
> --- a/workflows/fstests/Kconfig
> +++ b/workflows/fstests/Kconfig
> @@ -760,15 +760,23 @@ config FSTESTS_RUN_LARGE_DISK_TESTS
>  	  to run. The "large disk" requirement is test dependent, but
>  	  typically, it means a disk with capacity of at several 10G.
>  
> -config FSTESTS_SOAK_DURATION
> -	int "Custom Soak duration to be used"
> -	default 0
> +config FSTESTS_ENABLE_SOAK_DURATION
> +	bool "Enable custom soak duration time"
>  	help
> -	  Custom Soak duration to be used during test execution. If you set this
> -	  to a non-zero value then fstests will increase the amount of time it
> -	  takes to run certain tests which are time based and support using
> -	  SOAK_DURATION. A moderate high value setting for this is 9900 which is
> -	  2.5 hours.
> +	  Enable soak duration to be used during test execution. If you are not
> +	  interested in extending your testing then leave this disabled.
> +
> +	  Using a custom soak duration to a non-zero value then fstests will
> +	  increase the amount of time it takes to run certain tests which are
> +	  time based and support using SOAK_DURATION. A moderate high value
> +	  setting for this is 9900 which is 2.5 hours.

"A moderately high setting for this is "2.5h" for 2.5 hours."

FWIW, the ./check parser translates floating point numbers with suffixes
to integer seconds; see soak_duration.awk.

The part I don't know is if kdevops merely passes through the value as a
string; or actually treats this as an integer.  If the latter, then
please ignore my comment.

> +
> +	  Note that we have 46 tests today which will be able to use soak
> +	  duration if set. This means your test time will increase by the
> +	  soak duration * these number of tests. When soak duration is
> +	  enabled the test specific watchdog fstests_watchdog.py will be
> +	  aware of tests which require soak duration and consider before
> +	  reporting a possible hang.
>  
>  	  As of 2023-10-31 that consists of the following tests which use either
>  	  fsstress or fsx or fio. Tests either use SOAK_DURATION directly or they
> @@ -786,7 +794,7 @@ config FSTESTS_SOAK_DURATION
>  	  - generic/648 - fsstress + disk failures on loopback
>  	  - generic/650 - fsstress - multithreaded write + CPU hotplug
>  
> -	  The tests below use _scratch_xfs_stress_scrub() to stress
> +	  All the tests below use _scratch_xfs_stress_scrub() to stress
>  	  test an with fsstress with scrub or an alternate xfs_db operation.
>  
>  	  - xfs/285
> @@ -825,4 +833,67 @@ config FSTESTS_SOAK_DURATION
>  	  - xfs/729
>  	  - xfs/800
>  
> +if FSTESTS_ENABLE_SOAK_DURATION
> +
> +choice
> +	prompt "Soak duration value to use"
> +	default FSTESTS_SOAK_DURATION_HIGH
> +
> +config FSTESTS_SOAK_DURATION_CUSTOM
> +	bool "Custom"
> +	help
> +	  You want to specify the value yourself.
> +
> +config FSTESTS_SOAK_DURATION_PATHALOGICAL

"PATHOLOGICAL", and yes that high a setting is pathological. ;)

(Unless you're allocating one soak-fstest per VM in which case "1w"
might be appropriate.)

> +	bool "High (48 hours)"
> +	help
> +	  Use 48 hours for soak duration.
> +
> +	  Using this with 46 tests known to use soak duration means your test
> +	  time will increase by about 92 days, or a bit over 3 months if run
> +	  serially.
> +
> +config FSTESTS_SOAK_DURATION_HIGH
> +	bool "High (2.5 hours)"
> +	help
> +	  Use 2.5 hours for soak duration.
> +
> +	  Using this with 46 tests known to use soak duration means your test
> +	  time will increase by about 5 days if run serially.
> +
> +config FSTESTS_SOAK_DURATION_MID
> +	bool "Mid (1 hour)"
> +	help
> +	  Use 1 hour for soak duration.
> +
> +	  Using this with 46 tests known to use soak duration means your test
> +	  time will increase by about 2 days if run serially.

I wonder, is there any way to scan the number of soak test to generate
these figures automatically at configure time?  I'd guess no, since
kdevops kconfig comes before pulling and compiling, right?

--D

> +
> +config FSTESTS_SOAK_DURATION_LOW
> +	bool "Low (30 minutes)
> +	help
> +	  Use 30 minutes for soak duration.
> +
> +	  Using this with 46 tests known to use soak duration means your test
> +	  time will increase by about 1 day if run serially.
> +
> +endchoice
> +
> +config FSTESTS_SOAK_DURATION_CUSTOM_VAL
> +	int "Custom soak duration value (seconds)"
> +	default 0
> +	depends on FSTESTS_SOAK_DURATION_CUSTOM
> +	help
> +	  Enter your custom soak duration value in seconds.
> +
> +endif # FSTESTS_ENABLE_SOAK_DURATION
> +
> +config FSTESTS_SOAK_DURATION
> +	default 0 if !FSTESTS_ENABLE_SOAK_DURATION
> +	default FSTESTS_SOAK_DURATION_CUSTOM_VAL if FSTESTS_SOAK_DURATION_CUSTOM
> +	default 1800 if FSTESTS_SOAK_DURATION_LOW
> +	default 3600 if FSTESTS_SOAK_DURATION_MID
> +	default 9900 if FSTESTS_SOAK_DURATION_HIGH
> +	default 172800 if FSTESTS_SOAK_DURATION_PATHALOGICAL
> +
>  endif # KDEVOPS_WORKFLOW_ENABLE_FSTESTS
> diff --git a/workflows/fstests/Makefile.sparsefiles b/workflows/fstests/Makefile.sparsefiles
> index c5ca20a9c462..7dd129c4f9cc 100644
> --- a/workflows/fstests/Makefile.sparsefiles
> +++ b/workflows/fstests/Makefile.sparsefiles
> @@ -44,6 +44,10 @@ FSTESTS_ARGS += run_large_disk_tests='$(FSTESTS_RUN_LARGE_DISK_TESTS)'
>  FSTESTS_ARGS += run_auto_group_tests='$(FSTESTS_RUN_AUTO_GROUP_TESTS)'
>  FSTESTS_ARGS += run_custom_group_tests='$(FSTESTS_RUN_CUSTOM_GROUP_TESTS)'
>  FSTESTS_ARGS += exclude_test_groups='$(CONFIG_FSTESTS_EXCLUDE_TEST_GROUPS)'
> +
> +ifeq (y,$(CONFIG_FSTESTS_ENABLE_SOAK_DURATION))
> +FSTESTS_ARGS += fstests_soak_duration_enable='True'
> +endif
>  FSTESTS_ARGS += fstests_soak_duration='$(CONFIG_FSTESTS_SOAK_DURATION)'
>  
>  ifeq (y,$(CONFIG_FSTESTS_ENABLE_RUN_CUSTOM_TESTS))
> -- 
> 2.42.0
> 
>