Re: [PATCH] selftests: allow runners to override the timeout

[Muhammad Usama Anjum <usama.anjum@collabora.com>]

On 4/15/23 12:38 AM, Luis Chamberlain wrote:
> The default timeout for selftests tests is 45 seconds. Although
> we already have 13 settings for tests of about 96 sefltests which
> use a timeout greater than this, we want to try to avoid encouraging
> more tests to forcing a higher test timeout as selftests strives to
> run all tests quickly. Selftests also uses the timeout as a non-fatal
> error. Only tests runners which have control over a system would know
> if to treat a timeout as fatal or not.
> 
> To help with all this:
> 
>   o Enhance documentation to avoid future increases of insane timeouts
>   o Add the option to allow overriding the default timeout with test
>     runners with a command line option
> 
> Suggested-by: Shuah Khan <skhan@linuxfoundation.org>
> Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Reviewed-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
Tested-by:Muhammad Usama Anjum <usama.anjum@collabora.com>

> ---
>  Documentation/dev-tools/kselftest.rst       | 22 +++++++++++++++++++++
>  tools/testing/selftests/kselftest/runner.sh | 11 ++++++++++-
>  tools/testing/selftests/run_kselftest.sh    |  5 +++++
>  3 files changed, 37 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
> index 12b575b76b20..dd214af7b7ff 100644
> --- a/Documentation/dev-tools/kselftest.rst
> +++ b/Documentation/dev-tools/kselftest.rst
> @@ -168,6 +168,28 @@ the `-t` option for specific single tests. Either can be used multiple times::
>  
>  For other features see the script usage output, seen with the `-h` option.
>  
> +Timeout for selftests
> +=====================
> +
> +Selftests are designed to be quick and so a default timeout is used of 45
> +seconds for each test. Tests can override the default timeout by adding
> +a settings file in their directory and set a timeout variable there to the
> +configured a desired upper timeout for the test. Only a few tests override
> +the timeout with a value higher than 45 seconds, selftests strives to keep
> +it that way. Timeouts in selftests are not considered fatal because the
> +system under which a test runs may change and this can also modify the
> +expected time it takes to run a test. If you have control over the systems
> +which will run the tests you can configure a test runner on those systems to
> +use a greater or lower timeout on the command line as with the `-o` or
> +the `--override-timeout` argument. For example to use 165 seconds instead
> +one would use:
> +
> +   $ ./run_kselftest.sh --override-timeout 165
> +
> +You can look at the TAP output to see if you ran into the timeout. Test
> +runners which know a test must run under a specific time can then optionally
> +treat these timeouts then as fatal.
> +
>  Packaging selftests
>  ===================
>  
> diff --git a/tools/testing/selftests/kselftest/runner.sh b/tools/testing/selftests/kselftest/runner.sh
> index 294619ade49f..1c952d1401d4 100644
> --- a/tools/testing/selftests/kselftest/runner.sh
> +++ b/tools/testing/selftests/kselftest/runner.sh
> @@ -8,7 +8,8 @@ export logfile=/dev/stdout
>  export per_test_logging=
>  
>  # Defaults for "settings" file fields:
> -# "timeout" how many seconds to let each test run before failing.
> +# "timeout" how many seconds to let each test run before running
> +# over our soft timeout limit.
>  export kselftest_default_timeout=45
>  
>  # There isn't a shell-agnostic way to find the path of a sourced file,
> @@ -90,6 +91,14 @@ run_one()
>  		done < "$settings"
>  	fi
>  
> +	# Command line timeout overrides the settings file
> +	if [ -n "$kselftest_override_timeout" ]; then
> +		kselftest_timeout="$kselftest_override_timeout"
> +		echo "# overriding timeout to $kselftest_timeout" >> "$logfile"
> +	else
> +		echo "# timeout set to $kselftest_timeout" >> "$logfile"
> +	fi
> +
>  	TEST_HDR_MSG="selftests: $DIR: $BASENAME_TEST"
>  	echo "# $TEST_HDR_MSG"
>  	if [ ! -e "$TEST" ]; then
> diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
> index 97165a83df63..9a981b36bd7f 100755
> --- a/tools/testing/selftests/run_kselftest.sh
> +++ b/tools/testing/selftests/run_kselftest.sh
> @@ -26,6 +26,7 @@ Usage: $0 [OPTIONS]
>    -l | --list			List the available collection:test entries
>    -d | --dry-run		Don't actually run any tests
>    -h | --help			Show this usage info
> +  -o | --override-timeout	Number of seconds after which we timeout
>  EOF
>  	exit $1
>  }
> @@ -33,6 +34,7 @@ EOF
>  COLLECTIONS=""
>  TESTS=""
>  dryrun=""
> +kselftest_override_timeout=""
>  while true; do
>  	case "$1" in
>  		-s | --summary)
> @@ -51,6 +53,9 @@ while true; do
>  		-d | --dry-run)
>  			dryrun="echo"
>  			shift ;;
> +		-o | --override-timeout)
> +			kselftest_override_timeout="$2"
> +			shift 2 ;;
>  		-h | --help)
>  			usage 0 ;;
>  		"")

-- 
BR,
Muhammad Usama Anjum