[PATCH libevl v2 0/4] evl-test: stress load and latmus integration

[PATCH libevl v2 0/4] evl-test: stress load and latmus integration

[Tobias Schaffner]

Extend evl-test to exercise the system while the tests run and to
measure latency at the end of a session.

The motivation is to allow running the suite under stress in CI to
surface bugs that only show up under load.
latmus run under that same load and with a latency threshold will allow
the pipeline to spot latency regressions.

Example:

  # run latmus under CPU/memory stress, fail above 200us,
  # keep the latmus report
  evl test --stress-cmd "stress-ng --cpu 4 --vm 2 --vm-bytes 1G" \
           latmus -- -T 60 -A 200

Changes since v1:
 * Introduce a small shell script wrapper in the test dir to add latmus
   to the testsuite instead of calling it in the end after the other tests.

Tobias Schaffner (4):
  evl-test: add stress command support for background load testing
  evl-test: improve help message with detailed usage information
  latmus: send setup messages to stdout instead of stderr
  tests: Add latmus wrapper script for evl-test integration

 latmus/latmus.c   |  4 ++--
 tests/latmus      | 16 +++++++++++++
 tests/meson.build | 11 +++++++++
 utils/evl-test    | 57 ++++++++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 85 insertions(+), 3 deletions(-)
 create mode 100755 tests/latmus

-- 
2.43.0

[PATCH libevl v2 1/4] evl-test: add stress command support for background load testing

[Tobias Schaffner]

Add the ability to run a stress command (e.g. stress-ng) in the
background during test execution via the new -s option. This allows
testing EVL functionality under system load.

The stress command runs in its own process group (via setsid) to ensure
clean termination of all child processes, and is automatically stopped
when tests complete or the script exits normally or via signals.

The stress command output is dropped by default but can be redirected
using the --stress-out option.

Usage example:
  evl-test --stress-cmd "stress-ng --vm 2 --vm-bytes 1G --cpu 4" -k

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 utils/evl-test | 32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)

diff --git a/utils/evl-test b/utils/evl-test
index 5139d09..f6bd82f 100644
--- a/utils/evl-test
+++ b/utils/evl-test
@@ -7,7 +7,7 @@ if test \! -d $EVL_TESTDIR; then
 fi
 
 usage() {
-   echo >&2 "usage: $(basename $1) [-l][-L][-k][-r][-h] [test-list]"
+   echo >&2 "usage: $(basename $1) [-l][-L][-k][-r][-h][--stress-cmd cmd][--stress-out file] [test-list]"
 }
 
 help=false
@@ -18,6 +18,9 @@ full_path=false
 test_args=false
 test_arg_list=
 test_list=
+stress_cmd=
+stress_pid=
+stress_out=/dev/null
 
 while test $# -gt 0
 do
@@ -37,6 +40,10 @@ do
 	    -L) full_path=true;
 		do_list=true;
 		shift;;
+	    --stress-cmd) stress_cmd="$2";
+		shift 2;;
+	    --stress-out) stress_out="$2";
+		shift 2;;
 	    --) test_args=true
 		shift;;
 	    -@) echo "run the EVL tests"
@@ -49,6 +56,27 @@ do
     fi
 done
 
+stop_stress() {
+    trap - EXIT INT TERM
+    if test -n "$stress_pid"; then
+        kill -TERM -$stress_pid 2>/dev/null
+        # Give the process up to one second to terminate gracefully
+        for i in $(seq 10); do kill -0 -$stress_pid 2>/dev/null || break; sleep 0.1; done
+        kill -KILL -$stress_pid 2>/dev/null
+        wait $stress_pid 2>/dev/null
+    fi
+    exit
+}
+
+start_stress() {
+    if test -n "$stress_cmd"; then
+        trap 'stop_stress' EXIT INT TERM
+        setsid $stress_cmd </dev/null >"$stress_out" &
+        stress_pid=$!
+        sleep 1 # Give stress command time to start
+    fi
+}
+
 if test x$help = xtrue; then
    usage $0
    exit 0
@@ -78,6 +106,8 @@ if test x$do_list = xtrue; then
    exit 0
 fi
 
+start_stress
+
 result=0
 
 while : ; do
-- 
2.43.0

[PATCH libevl v2 2/4] evl-test: improve help message with detailed usage information

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 utils/evl-test | 27 ++++++++++++++++++++++++++-
 1 file changed, 26 insertions(+), 1 deletion(-)

diff --git a/utils/evl-test b/utils/evl-test
index f6bd82f..aa8e314 100644
--- a/utils/evl-test
+++ b/utils/evl-test
@@ -7,7 +7,32 @@ if test \! -d $EVL_TESTDIR; then
 fi
 
 usage() {
-   echo >&2 "usage: $(basename $1) [-l][-L][-k][-r][-h][--stress-cmd cmd][--stress-out file] [test-list]"
+   cat >&2 <<EOF
+usage: $(basename $1) [options] [test-list]
+
+Run EVL test suite or specific tests from the test directory.
+
+Options:
+  -l                    list available tests
+  -L                    list available tests with full paths
+  -k                    keep going after test failures
+  -r                    repeat tests continuously until interrupted
+  --stress-cmd <cmd>    run stress command during the test session (quote if multiple arguments)
+  --stress-out <file>   redirect stress command output (default: /dev/null)
+  -h                    display this help message
+  --                    pass remaining arguments to test programs
+
+Arguments:
+  test-list             one or more test names to run (default: all tests)
+
+Environment Variables:
+  EVL_TESTDIR           directory containing test binaries
+
+Examples:
+  $(basename $1)                                   # run all tests
+  $(basename $1) --stress-cmd "stress-ng --cpu 4"  # run tests under stress-ng CPU stress
+  $(basename $1) -- --verbose                      # pass --verbose to all tests
+EOF
 }
 
 help=false
-- 
2.43.0

[PATCH libevl v2 3/4] latmus: send setup messages to stdout instead of stderr

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 latmus/latmus.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/latmus/latmus.c b/latmus/latmus.c
index 9b3ed1e..772cd9f 100644
--- a/latmus/latmus.c
+++ b/latmus/latmus.c
@@ -369,9 +369,9 @@ static void do_measurement(size_t histogram_cells, bool no_check)
 		cpu_s = " (not isolated)";
 
 	if (verbosity > 0)
-		fprintf(stderr, "warming up on CPU%d%s...\n", responder_cpu, cpu_s);
+		fprintf(stdout, "warming up on CPU%d%s...\n", responder_cpu, cpu_s);
 	else
-		fprintf(stderr, "running quietly for %ld seconds on CPU%d%s\n",
+		fprintf(stdout, "running quietly for %ld seconds on CPU%d%s\n",
 			(long)timeout, responder_cpu, cpu_s);
 
 	switch (context_type) {
-- 
2.43.0

[PATCH libevl v2 4/4] tests: Add latmus wrapper script for evl-test integration

[Tobias Schaffner]

Introduce a shell script wrapper that allows to invoke latmus as a
test binary like any other test:

The wrapper defaults to a 10-second timeout and no stdout if called
without arguments. If called with arguments latmus behaves like
directly called.

  evl test latmus            # runs with -T 10
  evl test latmus -- -A 150  # passes custom latmus arguments

This also allows to run latmus with the --stress-cmd option provided
by evl test:

  evl test --stress-cmd "stress-ng --cpu 1" latmus -- -T 60 -A 150

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 tests/latmus      | 16 ++++++++++++++++
 tests/meson.build | 11 +++++++++++
 2 files changed, 27 insertions(+)
 create mode 100755 tests/latmus

diff --git a/tests/latmus b/tests/latmus
new file mode 100755
index 0000000..c9cf135
--- /dev/null
+++ b/tests/latmus
@@ -0,0 +1,16 @@
+#!/bin/sh
+# SPDX-License-Identifier: MIT
+#
+# Wrapper for latmus test integration with evl-test.
+# Allows invoking latmus through the test suite with:
+#   evl test latmus -- -A 150
+#   evl test latmus -- -T 60
+#   evl test latmus
+#
+# Defaults to a 10-second dropping stdout if run if no arguments are provided.
+
+if [ $# -eq 0 ]; then
+    exec latmus -T 10 >/dev/null
+else
+    exec latmus "$@"
+fi
diff --git a/tests/meson.build b/tests/meson.build
index da2cfe9..d90cbdd 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -119,4 +119,15 @@ foreach t : test_programs_with_atomic
 )
 endforeach
 
+test_scripts = [
+    'latmus',
+]
+
+foreach t : test_scripts
+    install_data(t,
+        install_dir : libexec_evl + '/tests',
+        install_mode : 'rwxr-xr-x'
+    )
+endforeach
+
 subdir('compile-tests')
-- 
2.43.0

Re: [PATCH libevl v2 0/4] evl-test: stress load and latmus integration

[Philippe Gerum]

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

> Extend evl-test to exercise the system while the tests run and to
> measure latency at the end of a session.
>
> The motivation is to allow running the suite under stress in CI to
> surface bugs that only show up under load.
> latmus run under that same load and with a latency threshold will allow
> the pipeline to spot latency regressions.
>
> Example:
>
>   # run latmus under CPU/memory stress, fail above 200us,
>   # keep the latmus report
>   evl test --stress-cmd "stress-ng --cpu 4 --vm 2 --vm-bytes 1G" \
>            latmus -- -T 60 -A 200
>
> Changes since v1:
>  * Introduce a small shell script wrapper in the test dir to add latmus
>    to the testsuite instead of calling it in the end after the other tests.
>
> Tobias Schaffner (4):
>   evl-test: add stress command support for background load testing
>   evl-test: improve help message with detailed usage information
>   latmus: send setup messages to stdout instead of stderr
>   tests: Add latmus wrapper script for evl-test integration
>
>  latmus/latmus.c   |  4 ++--
>  tests/latmus      | 16 +++++++++++++
>  tests/meson.build | 11 +++++++++
>  utils/evl-test    | 57 ++++++++++++++++++++++++++++++++++++++++++++++-
>  4 files changed, 85 insertions(+), 3 deletions(-)
>  create mode 100755 tests/latmus

Series merged, thanks.

-- 
Philippe.