[libevl][PATCH 0/4] evl-test: align with xeno-test

[libevl][PATCH 0/4] evl-test: align with xeno-test

[Tobias Schaffner]

Hi,

at the moment evl-test supports running the available unittests. This
series introduces load generation and latency measurement as present in
xeno-test. This will allow more standardized testing and a basic
comparison between Xenomai 3 and EVL.

To list the available tests --list must be used now, in favor of using
-l for the load command as in xeno-test.

Load generation with dohell and latency measurement was chosen as the
default when running evl test. The combination -u -l "" can be used to
run evl test with the old behaviour.

Rel: https://lore.kernel.org/xenomai/cf01659d-e800-4e97-8246-a4418cf6f4e0@siemens.com/

Tobias Schaffner (4):
  Copy dohell from Xenomai 3
  evl-test: Measure worst case latencies
  evl-test: Start evl-test with load by default
  evl-test: Add hectic for real-time stress

 utils/dohell      |  97 ++++++++++++++++++++++++++++++++++++
 utils/evl-test    | 124 ++++++++++++++++++++++++++++++++++------------
 utils/meson.build |   1 +
 3 files changed, 190 insertions(+), 32 deletions(-)
 create mode 100755 utils/dohell

-- 
2.40.1

[libevl][PATCH 1/4] Copy dohell from Xenomai 3

[Tobias Schaffner]

Dohell is used as the default load generator in Xenomai 3. Add it to
libevl to be able to stress the system while testing evl.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 utils/dohell      | 97 +++++++++++++++++++++++++++++++++++++++++++++++
 utils/meson.build |  1 +
 2 files changed, 98 insertions(+)
 create mode 100755 utils/dohell

diff --git a/utils/dohell b/utils/dohell
new file mode 100755
index 0000000..ea84d4c
--- /dev/null
+++ b/utils/dohell
@@ -0,0 +1,97 @@
+#! /bin/sh
+
+usage() {
+    cat <<EOF
+$0 [ -b path ] [ -s server ] [ -p port ] [ -m mntpoint ] [ -l path | seconds ]
+
+Generate load, using an assorted set of commands and optionnaly:
+- hackbench if the path to the hackbench binary is specified with -b;
+- nc to send TCP data to "server" port "port" if -s is specified (if -p
+is not specified, the port 9, aka discard is used);
+- dd to write data under "mntpoint" if -m is specified.
+
+during the runtime of the LTP test if the path to the LTP installation
+directory is specifed with -l or during "seconds" seconds.
+EOF
+}
+
+port=9
+while [ $# -gt 0 ]; do
+    case $1 in
+	-h|--help) usage
+	    exit 0;;
+	-b) shift; hackbench="$1"; shift
+	    ;;
+	-s) shift; server="$1"; shift
+	    ;;
+	-p) shift; port="$1"; shift
+	    ;;
+	-m) shift; mntpoint="$1"; shift
+	    ;;
+	-l) shift; ltpdir="$1"; shift
+	    ;;
+	-*) usage
+	    exit 1;;
+	*) break;;
+    esac
+done
+
+if [ -z "$ltpdir" -a $# -ne 1 ]; then
+    usage
+    exit 1
+fi
+
+pids=""
+
+if [ -n "$server" ]; then
+    if type nc > /dev/null 2>&1; then
+	nc=nc
+    elif type netcat > /dev/null 2>&1; then
+	nc=netcat
+    else
+	echo netcat or nc not found
+	exit 1
+    fi
+
+    seq 1 399999 > /tmp/netcat.data
+    ( while :; do cat /tmp/netcat.data; sleep 15; done | $nc $server $port ) &
+    pids="$!"
+fi
+
+if [ -n "$mntpoint" ]; then
+    while :; do dd if=/dev/zero of=$mntpoint/bigfile bs=1024000 count=100; sync; done &
+    pids="$pids $!"
+fi
+
+if [ -n "$hackbench" ]; then
+    while :; do $hackbench 1; done &
+    pids="$pids $!"
+fi
+
+while :; do cat /proc/interrupts; done > /dev/null 2>&1 &
+pids="$pids $!"
+
+while :; do ps w; done > /dev/null 2>&1 &
+pids="$pids $!"
+
+dd if=/dev/zero of=/dev/null &
+pids="$pids $!"
+
+while :; do ls -lR / > /dev/null 2>&1; done &
+pids="$pids $!"
+
+test -e /proc/sys/kernel/hung_task_timeout_secs && \
+echo 0 > /proc/sys/kernel/hung_task_timeout_secs
+
+if [ -n "$ltpdir" ]; then
+    cd "$ltpdir" && ./runalltests.sh
+    cd "$ltpdir" && ./runalltests.sh
+else
+    sleep $1
+fi
+
+kill $pids > /dev/null 2>&1
+sleep 5
+killall -KILL cat $nc dd hackbench ls ps > /dev/null 2>&1
+killall -KILL `basename $0` sleep > /dev/null 2>&1
+
diff --git a/utils/meson.build b/utils/meson.build
index 59d818a..17d91cc 100644
--- a/utils/meson.build
+++ b/utils/meson.build
@@ -21,6 +21,7 @@ executable('evl-check',
 )
 
 helper_scripts = [
+	'dohell',
 	'evl-gdb',
 	'evl-help',
 	'evl-start',
-- 
2.40.1

[libevl][PATCH 2/4] evl-test: Measure worst case latencies

[Tobias Schaffner]

Run latmus for 30 seconds after running the unittests to measure the
worst case latencies.

Allow to run with -u to solely run unittests.

Drop getopt to simplify argument parsing in the process.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 utils/evl-test | 55 ++++++++++++++++++++++++--------------------------
 1 file changed, 26 insertions(+), 29 deletions(-)

diff --git a/utils/evl-test b/utils/evl-test
index bc7440f..5c1c3da 100644
--- a/utils/evl-test
+++ b/utils/evl-test
@@ -7,44 +7,37 @@ if test \! -d $EVL_TESTDIR; then
 fi
 
 usage() {
-   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h] [test-list]"
+   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h][-u] [test-list]"
 }
 
-args=$(getopt -n $(basename $0) 'h@klL' "$@")
-if [ $? -ne 0 ]; then
-   usage $0
-   exit 1
-fi
-
-help=false
 keep_going=false
 do_list=false
 full_path=false
+unittests_only=false
 
-set -- $args
-for opt
-do
-case "$opt" in
-   -k) keep_going=true;
-       shift;;
-   -h) help=true;
-       shift;;
-   -l) do_list=true;
-       shift;;
-   -L) full_path=true;
-       do_list=true;
-       shift;;
-   -@) echo "run the EVL tests"
-       exit 0;;
-   --) shift; break;;
+while :; do
+   case "$1" in
+      -k) keep_going=true
+          shift;;
+      -l) do_list=true
+          shift;;
+      -L) full_path=true
+          do_list=true
+          shift;;
+      -u) unittests_only=true
+          shift;;
+      -@) echo "run the EVL tests"
+          exit 0;;
+      --) shift
+          break;;
+      "") break;;
+      -h) usage $0
+          exit 0;;
+      *)  usage $0
+          exit 1;;
    esac
 done
 
-if test x$help = xtrue; then
-   usage $0
-   exit 0
-fi
-
 test_list=
 for t in "$@"; do
     test_list="$test_list $(eval echo $EVL_TESTDIR/$t)"
@@ -89,4 +82,8 @@ for t in $test_list; do
     fi
 done
 
+if test x$unittests_only = xfalse; then
+    latmus -T 30
+fi
+
 exit 0
-- 
2.40.1

[libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Tobias Schaffner]

Stress evl with a load command while running the tests. The load command
may be set to an empty string to run tests without stressing the system.

To align with xeno-test the -l command line argument was used for the load
command. Listing of the unittests can now be done with --list and --List.

The load command is moved to its own process group to allow a clean
teardown of the load process with all its subprocesses.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 utils/evl-test | 71 +++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 61 insertions(+), 10 deletions(-)

diff --git a/utils/evl-test b/utils/evl-test
index 5c1c3da..fb85f69 100644
--- a/utils/evl-test
+++ b/utils/evl-test
@@ -6,33 +6,79 @@ if test \! -d $EVL_TESTDIR; then
    exit 2
 fi
 
-usage() {
-   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h][-u] [test-list]"
-}
-
 keep_going=false
 do_list=false
 full_path=false
 unittests_only=false
+load_cmd="dohell 900"
+load_cmd_pgid_file="/tmp/evl_test_load_cmd_$$"
+
+usage() {
+   cat >&2 <<EOF
+evl test -h
+
+This help text.
+
+evl test [ -l "load command" ] [ -k ] [ -u ] [ --list ] [ --List ]
+
+Run a basic test/benchmark of evl on your platform, by first starting a
+few unit tests, then running the latency test under the load generated by
+"load-command".
+
+By default, the load command is "dohell 900", which will generate load during
+15 minutes. To generate a more realistic load see dohell help.
+
+This script accepts the -k option to tell the unit test loop to keep
+going upon a failing test. Otherwise evl test stops immediately.
+
+Use -u if you want to skip the latency test and only run the unittests.
+
+Use --list to output all the tests available. --List will print all the tests
+with their full path.
+
+Example:
+evl test -l "dohell -s 192.168.0.5 -m /mnt -l /ltp"
+
+Will generate load including network load using the server at IP address
+192.168.0.5, some I/O under the moint point /mnt, and the LTP testsuite
+installed under the /ltp directory, and use the latency test by measuring the
+timer irq latency.
+EOF
+}
+
+cleanup() {
+   err="$?"
+   if test -r "${load_cmd_pgid_file}"; then
+      kill -TERM "-$(cat ${load_cmd_pgid_file})"
+      rm ${load_cmd_pgid_file}
+   fi
+   if test ! -z "${rt_load_pid}"; then
+      kill -TERM "${rt_load_pid}"
+   fi
+   trap '' EXIT
+   exit "$err"
+}
 
 while :; do
    case "$1" in
+      -h) usage $0
+          exit 0;;
       -k) keep_going=true
           shift;;
-      -l) do_list=true
+      -l) load_cmd="$2"
+          shift 2;;
+      -u) unittests_only=true
+          shift;;
+      --list) do_list=true
           shift;;
-      -L) full_path=true
+      --List) full_path=true
           do_list=true
           shift;;
-      -u) unittests_only=true
-          shift;;
       -@) echo "run the EVL tests"
           exit 0;;
       --) shift
           break;;
       "") break;;
-      -h) usage $0
-          exit 0;;
       *)  usage $0
           exit 1;;
    esac
@@ -62,6 +108,11 @@ if test x$do_list = xtrue; then
    exit 0
 fi
 
+if test ! -z "${load_cmd}"; then
+   trap cleanup EXIT INT
+   setsid /bin/sh -c "${load_cmd} & echo \$\$ > ${load_cmd_pgid_file}"
+fi
+
 for t in $test_list; do
     test \! -x $t && echo "$(basename $t): no such test" && exit 2
     # Swap stdout<->stderr for the test, capturing stderr into $log.
-- 
2.40.1

[libevl][PATCH 4/4] evl-test: Add hectic for real-time stress

[Tobias Schaffner]

The -r option allows to add realtime-stress by running hectic in the
background while running the tests.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 utils/evl-test | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/utils/evl-test b/utils/evl-test
index fb85f69..9a12d58 100644
--- a/utils/evl-test
+++ b/utils/evl-test
@@ -12,6 +12,7 @@ full_path=false
 unittests_only=false
 load_cmd="dohell 900"
 load_cmd_pgid_file="/tmp/evl_test_load_cmd_$$"
+rt_load_pid=""
 
 usage() {
    cat >&2 <<EOF
@@ -19,7 +20,7 @@ evl test -h
 
 This help text.
 
-evl test [ -l "load command" ] [ -k ] [ -u ] [ --list ] [ --List ]
+evl test [ -l "load command" ] [ -k ] [ -r ] [ -u ] [ --list ] [ --List ]
 
 Run a basic test/benchmark of evl on your platform, by first starting a
 few unit tests, then running the latency test under the load generated by
@@ -31,6 +32,10 @@ By default, the load command is "dohell 900", which will generate load during
 This script accepts the -k option to tell the unit test loop to keep
 going upon a failing test. Otherwise evl test stops immediately.
 
+If the script is passed the -r option, real-time stress is added to the test,
+with the help of the "hectic" test. But beware: the latency test figures are
+then no longer meaningful.
+
 Use -u if you want to skip the latency test and only run the unittests.
 
 Use --list to output all the tests available. --List will print all the tests
@@ -67,6 +72,8 @@ while :; do
           shift;;
       -l) load_cmd="$2"
           shift 2;;
+      -r) rt_load=true;
+          shift;;
       -u) unittests_only=true
           shift;;
       --list) do_list=true
@@ -113,6 +120,11 @@ if test ! -z "${load_cmd}"; then
    setsid /bin/sh -c "${load_cmd} & echo \$\$ > ${load_cmd_pgid_file}"
 fi
 
+if test x$rt_load = xtrue; then
+   hectic > /dev/null 2>&1 &
+   rt_load_pid="$!"
+fi
+
 for t in $test_list; do
     test \! -x $t && echo "$(basename $t): no such test" && exit 2
     # Swap stdout<->stderr for the test, capturing stderr into $log.
-- 
2.40.1

Re: [libevl][PATCH 2/4] evl-test: Measure worst case latencies

[Philippe Gerum]

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

> Run latmus for 30 seconds after running the unittests to measure the
> worst case latencies.
>
> Allow to run with -u to solely run unittests.
>
> Drop getopt to simplify argument parsing in the process.

Reading the code, I'm not sure this simplifies anything. What is the
actual upside of dropping getopt?

>
> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> ---
>  utils/evl-test | 55 ++++++++++++++++++++++++--------------------------
>  1 file changed, 26 insertions(+), 29 deletions(-)
>
> diff --git a/utils/evl-test b/utils/evl-test
> index bc7440f..5c1c3da 100644
> --- a/utils/evl-test
> +++ b/utils/evl-test
> @@ -7,44 +7,37 @@ if test \! -d $EVL_TESTDIR; then
>  fi
>  
>  usage() {
> -   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h] [test-list]"
> +   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h][-u] [test-list]"
>  }
>  
> -args=$(getopt -n $(basename $0) 'h@klL' "$@")
> -if [ $? -ne 0 ]; then
> -   usage $0
> -   exit 1
> -fi
> -
> -help=false
>  keep_going=false
>  do_list=false
>  full_path=false
> +unittests_only=false

Unit testing should be on by default, with a full-mode switch including
latmus, because this is the original way for running unattended
tests. Since latmus won't trigger any error on unacceptable latency,
there is no upside to have it on unless we really want to attend the
test, looking at its output.

>  
> -set -- $args
> -for opt
> -do
> -case "$opt" in
> -   -k) keep_going=true;
> -       shift;;
> -   -h) help=true;
> -       shift;;
> -   -l) do_list=true;
> -       shift;;
> -   -L) full_path=true;
> -       do_list=true;
> -       shift;;
> -   -@) echo "run the EVL tests"
> -       exit 0;;
> -   --) shift; break;;
> +while :; do
> +   case "$1" in
> +      -k) keep_going=true
> +          shift;;
> +      -l) do_list=true
> +          shift;;
> +      -L) full_path=true
> +          do_list=true
> +          shift;;
> +      -u) unittests_only=true
> +          shift;;
> +      -@) echo "run the EVL tests"
> +          exit 0;;
> +      --) shift
> +          break;;
> +      "") break;;
> +      -h) usage $0
> +          exit 0;;
> +      *)  usage $0
> +          exit 1;;
>     esac
>  done
>  
> -if test x$help = xtrue; then
> -   usage $0
> -   exit 0
> -fi
> -
>  test_list=
>  for t in "$@"; do
>      test_list="$test_list $(eval echo $EVL_TESTDIR/$t)"
> @@ -89,4 +82,8 @@ for t in $test_list; do
>      fi
>  done
>  
> +if test x$unittests_only = xfalse; then
> +    latmus -T 30
> +fi
> +
>  exit 0


-- 
Philippe.

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Philippe Gerum]

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

> Stress evl with a load command while running the tests. The load command
> may be set to an empty string to run tests without stressing the system.
>
> To align with xeno-test the -l command line argument was used for the load
> command. Listing of the unittests can now be done with --list and --List.
>

Nak. Compat with xeno-test is definitely not a requirement if this means
breaking the existing evl command line usage. Besides, --List looks
strange as an option. Let's pick a different option for the load command
instead.

> The load command is moved to its own process group to allow a clean
> teardown of the load process with all its subprocesses.
>
> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> ---
>  utils/evl-test | 71 +++++++++++++++++++++++++++++++++++++++++++-------
>  1 file changed, 61 insertions(+), 10 deletions(-)
>
> diff --git a/utils/evl-test b/utils/evl-test
> index 5c1c3da..fb85f69 100644
> --- a/utils/evl-test
> +++ b/utils/evl-test
> @@ -6,33 +6,79 @@ if test \! -d $EVL_TESTDIR; then
>     exit 2
>  fi
>  
> -usage() {
> -   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h][-u] [test-list]"
> -}
> -
>  keep_going=false
>  do_list=false
>  full_path=false
>  unittests_only=false
> +load_cmd="dohell 900"
> +load_cmd_pgid_file="/tmp/evl_test_load_cmd_$$"
> +
> +usage() {
> +   cat >&2 <<EOF
> +evl test -h
> +
> +This help text.
> +
> +evl test [ -l "load command" ] [ -k ] [ -u ] [ --list ] [ --List ]
> +
> +Run a basic test/benchmark of evl on your platform, by first starting a
> +few unit tests, then running the latency test under the load generated by
> +"load-command".
> +
> +By default, the load command is "dohell 900", which will generate load during
> +15 minutes. To generate a more realistic load see dohell help.
> +
> +This script accepts the -k option to tell the unit test loop to keep
> +going upon a failing test. Otherwise evl test stops immediately.
> +

To be fixed wrt the comment on the previous patch.

> +Use -u if you want to skip the latency test and only run the unittests.
> +
> +Use --list to output all the tests available. --List will print all the tests
> +with their full path.
> +
> +Example:
> +evl test -l "dohell -s 192.168.0.5 -m /mnt -l /ltp"
> +
> +Will generate load including network load using the server at IP address
> +192.168.0.5, some I/O under the moint point /mnt, and the LTP testsuite
> +installed under the /ltp directory, and use the latency test by measuring the
> +timer irq latency.

We should use stress-ng or something alike, instead of the obsolete
dohell script.

-- 
Philippe.

Re: [libevl][PATCH 1/4] Copy dohell from Xenomai 3

[Philippe Gerum]

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

> Dohell is used as the default load generator in Xenomai 3. Add it to
> libevl to be able to stress the system while testing evl.
>

dohell is obsolete. Let's move to stress-ng or something alike for x4.

-- 
Philippe.

Re: [libevl][PATCH 4/4] evl-test: Add hectic for real-time stress

[Philippe Gerum]

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

> The -r option allows to add realtime-stress by running hectic in the
> background while running the tests.
>
> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> ---
>  utils/evl-test | 14 +++++++++++++-
>  1 file changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/utils/evl-test b/utils/evl-test
> index fb85f69..9a12d58 100644
> --- a/utils/evl-test
> +++ b/utils/evl-test
> @@ -12,6 +12,7 @@ full_path=false
>  unittests_only=false
>  load_cmd="dohell 900"
>  load_cmd_pgid_file="/tmp/evl_test_load_cmd_$$"
> +rt_load_pid=""
>  
>  usage() {
>     cat >&2 <<EOF
> @@ -19,7 +20,7 @@ evl test -h
>  
>  This help text.
>  
> -evl test [ -l "load command" ] [ -k ] [ -u ] [ --list ] [ --List ]
> +evl test [ -l "load command" ] [ -k ] [ -r ] [ -u ] [ --list ] [ --List ]
>  
>  Run a basic test/benchmark of evl on your platform, by first starting a
>  few unit tests, then running the latency test under the load generated by
> @@ -31,6 +32,10 @@ By default, the load command is "dohell 900", which will generate load during
>  This script accepts the -k option to tell the unit test loop to keep
>  going upon a failing test. Otherwise evl test stops immediately.
>  
> +If the script is passed the -r option, real-time stress is added to the test,
> +with the help of the "hectic" test. But beware: the latency test figures are
> +then no longer meaningful.
> +
>  Use -u if you want to skip the latency test and only run the unittests.
>  
>  Use --list to output all the tests available. --List will print all the tests
> @@ -67,6 +72,8 @@ while :; do
>            shift;;
>        -l) load_cmd="$2"
>            shift 2;;
> +      -r) rt_load=true;
> +          shift;;
>        -u) unittests_only=true
>            shift;;
>        --list) do_list=true
> @@ -113,6 +120,11 @@ if test ! -z "${load_cmd}"; then
>     setsid /bin/sh -c "${load_cmd} & echo \$\$ > ${load_cmd_pgid_file}"
>  fi
>  
> +if test x$rt_load = xtrue; then
> +   hectic > /dev/null 2>&1 &

hectic -s 200 is even more stressful.

-- 
Philippe.

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Jan Kiszka]

On 13.06.24 16:16, Philippe Gerum wrote:
> 
> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
> 
>> Stress evl with a load command while running the tests. The load command
>> may be set to an empty string to run tests without stressing the system.
>>
>> To align with xeno-test the -l command line argument was used for the load
>> command. Listing of the unittests can now be done with --list and --List.
>>
> 
> Nak. Compat with xeno-test is definitely not a requirement if this means
> breaking the existing evl command line usage. Besides, --List looks
> strange as an option. Let's pick a different option for the load command
> instead.
> 
>> The load command is moved to its own process group to allow a clean
>> teardown of the load process with all its subprocesses.
>>
>> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
>> ---
>>  utils/evl-test | 71 +++++++++++++++++++++++++++++++++++++++++++-------
>>  1 file changed, 61 insertions(+), 10 deletions(-)
>>
>> diff --git a/utils/evl-test b/utils/evl-test
>> index 5c1c3da..fb85f69 100644
>> --- a/utils/evl-test
>> +++ b/utils/evl-test
>> @@ -6,33 +6,79 @@ if test \! -d $EVL_TESTDIR; then
>>     exit 2
>>  fi
>>  
>> -usage() {
>> -   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h][-u] [test-list]"
>> -}
>> -
>>  keep_going=false
>>  do_list=false
>>  full_path=false
>>  unittests_only=false
>> +load_cmd="dohell 900"
>> +load_cmd_pgid_file="/tmp/evl_test_load_cmd_$$"
>> +
>> +usage() {
>> +   cat >&2 <<EOF
>> +evl test -h
>> +
>> +This help text.
>> +
>> +evl test [ -l "load command" ] [ -k ] [ -u ] [ --list ] [ --List ]
>> +
>> +Run a basic test/benchmark of evl on your platform, by first starting a
>> +few unit tests, then running the latency test under the load generated by
>> +"load-command".
>> +
>> +By default, the load command is "dohell 900", which will generate load during
>> +15 minutes. To generate a more realistic load see dohell help.
>> +
>> +This script accepts the -k option to tell the unit test loop to keep
>> +going upon a failing test. Otherwise evl test stops immediately.
>> +
> 
> To be fixed wrt the comment on the previous patch.
> 
>> +Use -u if you want to skip the latency test and only run the unittests.
>> +
>> +Use --list to output all the tests available. --List will print all the tests
>> +with their full path.
>> +
>> +Example:
>> +evl test -l "dohell -s 192.168.0.5 -m /mnt -l /ltp"
>> +
>> +Will generate load including network load using the server at IP address
>> +192.168.0.5, some I/O under the moint point /mnt, and the LTP testsuite
>> +installed under the /ltp directory, and use the latency test by measuring the
>> +timer irq latency.
> 
> We should use stress-ng or something alike, instead of the obsolete
> dohell script.

Fact is that dohell does better than stress-ng, at least in the configs
of the latter that I've seen before.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [libevl][PATCH 1/4] Copy dohell from Xenomai 3

[Jan Kiszka]

On 13.06.24 16:23, Philippe Gerum wrote:
> 
> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
> 
>> Dohell is used as the default load generator in Xenomai 3. Add it to
>> libevl to be able to stress the system while testing evl.
>>
> 
> dohell is obsolete. Let's move to stress-ng or something alike for x4.
> 

See my comment on patch 3: If you can name a configuration of stress-ng
that stresses a co-kernel more than what dohell does, we can take it.
Otherwise, I would strongly recommend to not do less hell over here.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [libevl][PATCH 1/4] Copy dohell from Xenomai 3

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 13.06.24 16:23, Philippe Gerum wrote:
>> 
>> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
>> 
>>> Dohell is used as the default load generator in Xenomai 3. Add it to
>>> libevl to be able to stress the system while testing evl.
>>>
>> 
>> dohell is obsolete. Let's move to stress-ng or something alike for x4.
>> 
>
> See my comment on patch 3: If you can name a configuration of stress-ng
> that stresses a co-kernel more than what dohell does, we can take it.
> Otherwise, I would strongly recommend to not do less hell over here.
>

The most important stress dohell applies is basically hackbench and a dd
loop trashing the D-cache. The latter can be obtained with a combo of
VM+I/O tests from stress-ng. So basically, the whole issue boils down to
having hackbench on board, provided we cannot generate the same load
profile with stress-ng with a CPU bound stressor. Besides, I never saw
any added value in running ltp tests compared to a combo of stress-ng
stressors when it comes to latency testing.

We obviously don't want to throw less hell, but throwing more obsolete
stuff is not an option either. So that's a NAK for v4 ATM, I'll gather
more information about how we could leverage stress-ng in order to
compare to dohell. This is worth a shot.

-- 
Philippe.

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 13.06.24 16:16, Philippe Gerum wrote:
>> 
>> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
>> 
>>> Stress evl with a load command while running the tests. The load command
>>> may be set to an empty string to run tests without stressing the system.
>>>
>>> To align with xeno-test the -l command line argument was used for the load
>>> command. Listing of the unittests can now be done with --list and --List.
>>>
>> 
>> Nak. Compat with xeno-test is definitely not a requirement if this means
>> breaking the existing evl command line usage. Besides, --List looks
>> strange as an option. Let's pick a different option for the load command
>> instead.
>> 
>>> The load command is moved to its own process group to allow a clean
>>> teardown of the load process with all its subprocesses.
>>>
>>> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
>>> ---
>>>  utils/evl-test | 71 +++++++++++++++++++++++++++++++++++++++++++-------
>>>  1 file changed, 61 insertions(+), 10 deletions(-)
>>>
>>> diff --git a/utils/evl-test b/utils/evl-test
>>> index 5c1c3da..fb85f69 100644
>>> --- a/utils/evl-test
>>> +++ b/utils/evl-test
>>> @@ -6,33 +6,79 @@ if test \! -d $EVL_TESTDIR; then
>>>     exit 2
>>>  fi
>>>  
>>> -usage() {
>>> -   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h][-u] [test-list]"
>>> -}
>>> -
>>>  keep_going=false
>>>  do_list=false
>>>  full_path=false
>>>  unittests_only=false
>>> +load_cmd="dohell 900"
>>> +load_cmd_pgid_file="/tmp/evl_test_load_cmd_$$"
>>> +
>>> +usage() {
>>> +   cat >&2 <<EOF
>>> +evl test -h
>>> +
>>> +This help text.
>>> +
>>> +evl test [ -l "load command" ] [ -k ] [ -u ] [ --list ] [ --List ]
>>> +
>>> +Run a basic test/benchmark of evl on your platform, by first starting a
>>> +few unit tests, then running the latency test under the load generated by
>>> +"load-command".
>>> +
>>> +By default, the load command is "dohell 900", which will generate load during
>>> +15 minutes. To generate a more realistic load see dohell help.
>>> +
>>> +This script accepts the -k option to tell the unit test loop to keep
>>> +going upon a failing test. Otherwise evl test stops immediately.
>>> +
>> 
>> To be fixed wrt the comment on the previous patch.
>> 
>>> +Use -u if you want to skip the latency test and only run the unittests.
>>> +
>>> +Use --list to output all the tests available. --List will print all the tests
>>> +with their full path.
>>> +
>>> +Example:
>>> +evl test -l "dohell -s 192.168.0.5 -m /mnt -l /ltp"
>>> +
>>> +Will generate load including network load using the server at IP address
>>> +192.168.0.5, some I/O under the moint point /mnt, and the LTP testsuite
>>> +installed under the /ltp directory, and use the latency test by measuring the
>>> +timer irq latency.
>> 
>> We should use stress-ng or something alike, instead of the obsolete
>> dohell script.
>
> Fact is that dohell does better than stress-ng, at least in the configs
> of the latter that I've seen before.
>

See my comment on the dohell patch.

-- 
Philippe.

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Philippe Gerum]

Philippe Gerum <rpm@xenomai.org> writes:

> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
>
>> Stress evl with a load command while running the tests. The load command
>> may be set to an empty string to run tests without stressing the system.
>>
>> To align with xeno-test the -l command line argument was used for the load
>> command. Listing of the unittests can now be done with --list and --List.
>>
>
> Nak. Compat with xeno-test is definitely not a requirement if this means
> breaking the existing evl command line usage. Besides, --List looks
> strange as an option. Let's pick a different option for the load command
> instead.
>

We may want to implement the stress utility as a separate evl-stress
command, keeping evl-test for unit testing which has a different
purpose. evl-stress as a script could generate the load (dohell) and
possibly check the latency figures reported by latmus (using the -K and
-A switches to detect weird behavior).

PS: Jorge is going to look into leveraging stress-ng for load
generation.

-- 
Philippe.

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Jorge Ramirez-Ortiz, Gmail]

On 13/06/24 19:27:10, Philippe Gerum wrote:
> 
> Philippe Gerum <rpm@xenomai.org> writes:
> 
> > Tobias Schaffner <tobias.schaffner@siemens.com> writes:
> >
> >> Stress evl with a load command while running the tests. The load command
> >> may be set to an empty string to run tests without stressing the system.
> >>
> >> To align with xeno-test the -l command line argument was used for the load
> >> command. Listing of the unittests can now be done with --list and --List.
> >>
> >
> > Nak. Compat with xeno-test is definitely not a requirement if this means
> > breaking the existing evl command line usage. Besides, --List looks
> > strange as an option. Let's pick a different option for the load command
> > instead.
> >
> 
> We may want to implement the stress utility as a separate evl-stress
> command, keeping evl-test for unit testing which has a different
> purpose. evl-stress as a script could generate the load (dohell) and
> possibly check the latency figures reported by latmus (using the -K and
> -A switches to detect weird behavior).
> 
> PS: Jorge is going to look into leveraging stress-ng for load
> generation.

yes I am going to start looking into it. I think that maintaining Linux
stressors should not be Xenomai's business. If we can't replicate the
load with todays stress-ng perhaps we could just usptream what is
needed (stress-ng is well maintained)

> 
> -- 
> Philippe.

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Tobias Schaffner]

Hey Jorge,

On 14.06.24 09:29, Jorge Ramirez-Ortiz, Gmail wrote:
> On 13/06/24 19:27:10, Philippe Gerum wrote:
>>
>> Philippe Gerum <rpm@xenomai.org> writes:
>>
>>> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
>>>
>>>> Stress evl with a load command while running the tests. The load command
>>>> may be set to an empty string to run tests without stressing the system.
>>>>
>>>> To align with xeno-test the -l command line argument was used for the load
>>>> command. Listing of the unittests can now be done with --list and --List.
>>>>
>>>
>>> Nak. Compat with xeno-test is definitely not a requirement if this means
>>> breaking the existing evl command line usage. Besides, --List looks
>>> strange as an option. Let's pick a different option for the load command
>>> instead.
>>>
>>
>> We may want to implement the stress utility as a separate evl-stress
>> command, keeping evl-test for unit testing which has a different
>> purpose. evl-stress as a script could generate the load (dohell) and
>> possibly check the latency figures reported by latmus (using the -K and
>> -A switches to detect weird behavior).
>>
>> PS: Jorge is going to look into leveraging stress-ng for load
>> generation.
> 
> yes I am going to start looking into it. I think that maintaining Linux
> stressors should not be Xenomai's business. If we can't replicate the
> load with todays stress-ng perhaps we could just usptream what is
> needed (stress-ng is well maintained)

Do you already have an update on this? I would like to resolve this 
story to be able to integrate standardized long term tests for EVL into 
the CI.

>>
>> -- 
>> Philippe.

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Schaffner, Tobias]

Hi Philippe, hi Jorge,

On Wed, 2024-07-10 at 09:18 +0200, Tobias Schaffner wrote:
> Hey Jorge,
> 
> On 14.06.24 09:29, Jorge Ramirez-Ortiz, Gmail wrote:
> > On 13/06/24 19:27:10, Philippe Gerum wrote:
> > > 
> > > Philippe Gerum <rpm@xenomai.org> writes:
> > > 
> > > > Tobias Schaffner <tobias.schaffner@siemens.com> writes:
> > > > 
> > > > > Stress evl with a load command while running the tests. The
> > > > > load command
> > > > > may be set to an empty string to run tests without stressing
> > > > > the system.
> > > > > 
> > > > > To align with xeno-test the -l command line argument was used
> > > > > for the load
> > > > > command. Listing of the unittests can now be done with --list
> > > > > and --List.
> > > > > 
> > > > 
> > > > Nak. Compat with xeno-test is definitely not a requirement if
> > > > this means
> > > > breaking the existing evl command line usage. Besides, --List
> > > > looks
> > > > strange as an option. Let's pick a different option for the
> > > > load command
> > > > instead.
> > > > 
> > > 
> > > We may want to implement the stress utility as a separate evl-
> > > stress
> > > command, keeping evl-test for unit testing which has a different
> > > purpose. evl-stress as a script could generate the load (dohell)
> > > and
> > > possibly check the latency figures reported by latmus (using the
> > > -K and
> > > -A switches to detect weird behavior).
> > > 
> > > PS: Jorge is going to look into leveraging stress-ng for load
> > > generation.
> > 
> > yes I am going to start looking into it. I think that maintaining
> > Linux
> > stressors should not be Xenomai's business. If we can't replicate
> > the
> > load with todays stress-ng perhaps we could just usptream what is
> > needed (stress-ng is well maintained)
> 

may I bring this up again? Any new developments or opinions on this?
I would like to see a solution that allows to compare xenomai 3 and
EVL. Reliable numbers are necessary to make an informed decision if EVL
is on par or even better than xenomai 3. Imo we are comparing apples
and oranges as long as we are not using the same tooling on both
platforms. 

Best,
Tobias

> Do you already have an update on this? I would like to resolve this 
> story to be able to integrate standardized long term tests for EVL
> into 
> the CI.
> 
> > > 
> > > -- 
> > > Philippe.
> 

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Jorge Ramirez-Ortiz, Gmail]

On 24/10/24 20:00:12, Schaffner, Tobias wrote:
> Hi Philippe, hi Jorge,
> 
> On Wed, 2024-07-10 at 09:18 +0200, Tobias Schaffner wrote:
> > Hey Jorge,
> > 
> > On 14.06.24 09:29, Jorge Ramirez-Ortiz, Gmail wrote:
> > > On 13/06/24 19:27:10, Philippe Gerum wrote:
> > > > 
> > > > Philippe Gerum <rpm@xenomai.org> writes:
> > > > 
> > > > > Tobias Schaffner <tobias.schaffner@siemens.com> writes:
> > > > > 
> > > > > > Stress evl with a load command while running the tests. The
> > > > > > load command
> > > > > > may be set to an empty string to run tests without stressing
> > > > > > the system.
> > > > > > 
> > > > > > To align with xeno-test the -l command line argument was used
> > > > > > for the load
> > > > > > command. Listing of the unittests can now be done with --list
> > > > > > and --List.
> > > > > > 
> > > > > 
> > > > > Nak. Compat with xeno-test is definitely not a requirement if
> > > > > this means
> > > > > breaking the existing evl command line usage. Besides, --List
> > > > > looks
> > > > > strange as an option. Let's pick a different option for the
> > > > > load command
> > > > > instead.
> > > > > 
> > > > 
> > > > We may want to implement the stress utility as a separate evl-
> > > > stress
> > > > command, keeping evl-test for unit testing which has a different
> > > > purpose. evl-stress as a script could generate the load (dohell)
> > > > and
> > > > possibly check the latency figures reported by latmus (using the
> > > > -K and
> > > > -A switches to detect weird behavior).
> > > > 
> > > > PS: Jorge is going to look into leveraging stress-ng for load
> > > > generation.
> > > 
> > > yes I am going to start looking into it. I think that maintaining
> > > Linux
> > > stressors should not be Xenomai's business. If we can't replicate
> > > the
> > > load with todays stress-ng perhaps we could just usptream what is
> > > needed (stress-ng is well maintained)
> > 
> 
> may I bring this up again? Any new developments or opinions on this?
> I would like to see a solution that allows to compare xenomai 3 and
> EVL. Reliable numbers are necessary to make an informed decision if EVL
> is on par or even better than xenomai 3. Imo we are comparing apples
> and oranges as long as we are not using the same tooling on both
> platforms.

Sorry about this, it is being a bit of a crazy year for me at a personal
levely. I am start on it this week.

My setup uses some Zephyr code executing on frdmk64 I wrote years ago to
benchmark evl

https://github.com/ldts/libevl/commit/905d1946ca117e3889c53d1150a1334cc91033a8

I will load my evl system using stress-ng and measure latency figures
and then replicate with dohell loading tools. I should have something to
share by next friday.

feel free to ping me anytime. In IRC you can find me in LiberaChat
#u-boot (nick ldts) and since recently in OFTC #edk2

> 
> Best,
> Tobias
> 
> > Do you already have an update on this? I would like to resolve this 
> > story to be able to integrate standardized long term tests for EVL
> > into 
> > the CI.
> > 
> > > > 
> > > > -- 
> > > > Philippe.
> > 
> 

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Jorge Ramirez-Ortiz, Gmail]

On 27/10/24 20:13:08, Jorge Ramirez-Ortiz, Gmail wrote:
> On 24/10/24 20:00:12, Schaffner, Tobias wrote:
> > Hi Philippe, hi Jorge,
> > 
> > On Wed, 2024-07-10 at 09:18 +0200, Tobias Schaffner wrote:
> > > Hey Jorge,
> > > 
> > > On 14.06.24 09:29, Jorge Ramirez-Ortiz, Gmail wrote:
> > > > On 13/06/24 19:27:10, Philippe Gerum wrote:
> > > > > 
> > > > > Philippe Gerum <rpm@xenomai.org> writes:
> > > > > 
> > > > > > Tobias Schaffner <tobias.schaffner@siemens.com> writes:
> > > > > > 
> > > > > > > Stress evl with a load command while running the tests. The
> > > > > > > load command
> > > > > > > may be set to an empty string to run tests without stressing
> > > > > > > the system.
> > > > > > > 
> > > > > > > To align with xeno-test the -l command line argument was used
> > > > > > > for the load
> > > > > > > command. Listing of the unittests can now be done with --list
> > > > > > > and --List.
> > > > > > > 
> > > > > > 
> > > > > > Nak. Compat with xeno-test is definitely not a requirement if
> > > > > > this means
> > > > > > breaking the existing evl command line usage. Besides, --List
> > > > > > looks
> > > > > > strange as an option. Let's pick a different option for the
> > > > > > load command
> > > > > > instead.
> > > > > > 
> > > > > 
> > > > > We may want to implement the stress utility as a separate evl-
> > > > > stress
> > > > > command, keeping evl-test for unit testing which has a different
> > > > > purpose. evl-stress as a script could generate the load (dohell)
> > > > > and
> > > > > possibly check the latency figures reported by latmus (using the
> > > > > -K and
> > > > > -A switches to detect weird behavior).
> > > > > 
> > > > > PS: Jorge is going to look into leveraging stress-ng for load
> > > > > generation.
> > > > 
> > > > yes I am going to start looking into it. I think that maintaining
> > > > Linux
> > > > stressors should not be Xenomai's business. If we can't replicate
> > > > the
> > > > load with todays stress-ng perhaps we could just usptream what is
> > > > needed (stress-ng is well maintained)
> > > 
> > 
> > may I bring this up again? Any new developments or opinions on this?
> > I would like to see a solution that allows to compare xenomai 3 and
> > EVL. Reliable numbers are necessary to make an informed decision if EVL
> > is on par or even better than xenomai 3. Imo we are comparing apples
> > and oranges as long as we are not using the same tooling on both
> > platforms.
> 
> Sorry about this, it is being a bit of a crazy year for me at a personal
> levely. I am start on it this week.
> 
> My setup uses some Zephyr code executing on frdmk64 I wrote years ago to
> benchmark evl
> 
> https://github.com/ldts/libevl/commit/905d1946ca117e3889c53d1150a1334cc91033a8
> 
> I will load my evl system using stress-ng and measure latency figures
> and then replicate with dohell loading tools. I should have something to
> share by next friday.
> 
> feel free to ping me anytime. In IRC you can find me in LiberaChat
> #u-boot (nick ldts) and since recently in OFTC #edk2

I was finally able to get some numbers on imx8mm (an Arduino Pro with
the Portenta breakout board recently contributed to my lab by Wim
Taymans for audio testing of the Xenomai4 Pipewire-plugin[0] for
realtime audio)

I am using dohell/rt-tools/ltp and stress-ng on EVL - havent tested
Xenomai3.

My test bench runs with the help of an external FRDMK64F running Zephyr
very similar to what is described in the latmus/latmon combo
documentation [1]. Given how cheap this board is perhaps it should be
more widely used (?).

[RFC]
I would like to update latmon[2] to the tip of Zephyr - it has fallen
behind - I'd like then to propose it upstream (if accepted I would like
to propose that it is removed from EVL). Anyone disagrees?

IMO We should also take latmus to its own separate tree dedicated to RT
benchmarking and extend it to support scheduling latency metrics based
on GPIO data.
[\]

So:

* stress-ng --vm 2 --vm-bytes 1G --mmap 2 --mmap-bytes 1G --page-in \
  --matrix 0 --matrix-size 64 --tz -t 60

This simple command (VM and cpu load for a 2GB DDR system) hits the
system very quick and my externally measured GPIO IRQ latency peaks
within seconds.

stress-ng has FPU, VM, CPU all sort of stressors but I didnt need to go
into those just yet.

I found some info that RH [3] is providing interesting - still the
stress-ng code is simple enough to work with if needed (really good
encapsulation and code base)

* dohell -l /opt/ltp -b /bin/hackbench -m /home/root

After five minutes of run the latency is still contained (far from
peaking, 50% below the peak) but I assume this is because LTP takes a
while to run all tests. Without going into more details, this difference
in the invested time seems enough for me to switch to stress-ng.


Jorge

[0] https://gitlab.freedesktop.org/pipewire/pipewire/-/commit/ffaa365beff0e7f1d1f395800c14b1be26d8d007
[1] https://evlproject.org/core/benchmarks/#latmus-gpio-response-time
[2] https://source.denx.de/Xenomai/xenomai4/libevl/-/tree/master/benchmarks/zephyr/latmon/ 
[3] https://docs.redhat.com/en/documentation/red_hat_enterprise_linux_for_real_time/9/html/optimizing_rhel_9_for_real_time_for_low_latency_operation/assembly_stress-testing-real-time-systems-with-stress-ng_optimizing-rhel9-for-real-time-for-low-latency-operation#proc_testing-cpu-with-multiple-stress-mechanisms_assembly_stress-testing-real-time-systems-with-stress-ng

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Jorge Ramirez-Ortiz, Gmail]

On 13/06/24 17:30:51, Jan Kiszka wrote:
> On 13.06.24 16:16, Philippe Gerum wrote:
> > 
> > Tobias Schaffner <tobias.schaffner@siemens.com> writes:
> > 
> >> Stress evl with a load command while running the tests. The load command
> >> may be set to an empty string to run tests without stressing the system.
> >>
> >> To align with xeno-test the -l command line argument was used for the load
> >> command. Listing of the unittests can now be done with --list and --List.
> >>
> > 
> > Nak. Compat with xeno-test is definitely not a requirement if this means
> > breaking the existing evl command line usage. Besides, --List looks
> > strange as an option. Let's pick a different option for the load command
> > instead.
> > 
> >> The load command is moved to its own process group to allow a clean
> >> teardown of the load process with all its subprocesses.
> >>
> >> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> >> ---
> >>  utils/evl-test | 71 +++++++++++++++++++++++++++++++++++++++++++-------
> >>  1 file changed, 61 insertions(+), 10 deletions(-)
> >>
> >> diff --git a/utils/evl-test b/utils/evl-test
> >> index 5c1c3da..fb85f69 100644
> >> --- a/utils/evl-test
> >> +++ b/utils/evl-test
> >> @@ -6,33 +6,79 @@ if test \! -d $EVL_TESTDIR; then
> >>     exit 2
> >>  fi
> >>  
> >> -usage() {
> >> -   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h][-u] [test-list]"
> >> -}
> >> -
> >>  keep_going=false
> >>  do_list=false
> >>  full_path=false
> >>  unittests_only=false
> >> +load_cmd="dohell 900"
> >> +load_cmd_pgid_file="/tmp/evl_test_load_cmd_$$"
> >> +
> >> +usage() {
> >> +   cat >&2 <<EOF
> >> +evl test -h
> >> +
> >> +This help text.
> >> +
> >> +evl test [ -l "load command" ] [ -k ] [ -u ] [ --list ] [ --List ]
> >> +
> >> +Run a basic test/benchmark of evl on your platform, by first starting a
> >> +few unit tests, then running the latency test under the load generated by
> >> +"load-command".
> >> +
> >> +By default, the load command is "dohell 900", which will generate load during
> >> +15 minutes. To generate a more realistic load see dohell help.
> >> +
> >> +This script accepts the -k option to tell the unit test loop to keep
> >> +going upon a failing test. Otherwise evl test stops immediately.
> >> +
> > 
> > To be fixed wrt the comment on the previous patch.
> > 
> >> +Use -u if you want to skip the latency test and only run the unittests.
> >> +
> >> +Use --list to output all the tests available. --List will print all the tests
> >> +with their full path.
> >> +
> >> +Example:
> >> +evl test -l "dohell -s 192.168.0.5 -m /mnt -l /ltp"
> >> +
> >> +Will generate load including network load using the server at IP address
> >> +192.168.0.5, some I/O under the moint point /mnt, and the LTP testsuite
> >> +installed under the /ltp directory, and use the latency test by measuring the
> >> +timer irq latency.
> > 
> > We should use stress-ng or something alike, instead of the obsolete
> > dohell script.
> 
> Fact is that dohell does better than stress-ng, at least in the configs
> of the latter that I've seen before.

just "dohell -s 192.168.0.5 -m /mnt -l /ltp" ? do you know after how
long (ie, did it take a few hours or just minutes?)

> 
> Jan
> 
> -- 
> Siemens AG, Technology
> Linux Expert Center
> 
> 

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Philippe Gerum]

"Jorge Ramirez-Ortiz, Gmail" <jorge.ramirez.ortiz@gmail.com> writes:

>
> [RFC]
> I would like to update latmon[2] to the tip of Zephyr - it has fallen
> behind - I'd like then to propose it upstream (if accepted I would like
> to propose that it is removed from EVL). Anyone disagrees?
>

Fine by me.

> IMO We should also take latmus to its own separate tree dedicated to RT
> benchmarking and extend it to support scheduling latency metrics based
> on GPIO data.
> [\]
>

We would need to make it evl-agnostic then, assuming you want the latmus
<-> latmon (Zephyr) combo to be broadly available off-the-shelf to
others.

-- 
Philippe.

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Jorge Ramirez-Ortiz, Gmail]

On 01/11/24 18:15:25, Philippe Gerum wrote:
> "Jorge Ramirez-Ortiz, Gmail" <jorge.ramirez.ortiz@gmail.com> writes:
>
> >
> > [RFC]
> > I would like to update latmon[2] to the tip of Zephyr - it has fallen
> > behind - I'd like then to propose it upstream (if accepted I would like
> > to propose that it is removed from EVL). Anyone disagrees?
> >
>
> Fine by me.
>
> > IMO We should also take latmus to its own separate tree dedicated to RT
> > benchmarking and extend it to support scheduling latency metrics based
> > on GPIO data.
> > [\]
> >
>
> We would need to make it evl-agnostic then, assuming you want the latmus
> <-> latmon (Zephyr) combo to be broadly available off-the-shelf to
> others.

yes I think that would make sense. Xenomai3/EVL/PREEMPT_RT or any other linux-rt
could be easily compared with an off the shelf el-cheapo benchmarking tool.


>
> --
> Philippe.

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Jan Kiszka]

[sorry, missed your reply]

On 01.11.24 18:00, Jorge Ramirez-Ortiz, Gmail wrote:
> On 13/06/24 17:30:51, Jan Kiszka wrote:
>> On 13.06.24 16:16, Philippe Gerum wrote:
>>>
>>> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
>>>
>>>> Stress evl with a load command while running the tests. The load command
>>>> may be set to an empty string to run tests without stressing the system.
>>>>
>>>> To align with xeno-test the -l command line argument was used for the load
>>>> command. Listing of the unittests can now be done with --list and --List.
>>>>
>>>
>>> Nak. Compat with xeno-test is definitely not a requirement if this means
>>> breaking the existing evl command line usage. Besides, --List looks
>>> strange as an option. Let's pick a different option for the load command
>>> instead.
>>>
>>>> The load command is moved to its own process group to allow a clean
>>>> teardown of the load process with all its subprocesses.
>>>>
>>>> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
>>>> ---
>>>>  utils/evl-test | 71 +++++++++++++++++++++++++++++++++++++++++++-------
>>>>  1 file changed, 61 insertions(+), 10 deletions(-)
>>>>
>>>> diff --git a/utils/evl-test b/utils/evl-test
>>>> index 5c1c3da..fb85f69 100644
>>>> --- a/utils/evl-test
>>>> +++ b/utils/evl-test
>>>> @@ -6,33 +6,79 @@ if test \! -d $EVL_TESTDIR; then
>>>>     exit 2
>>>>  fi
>>>>  
>>>> -usage() {
>>>> -   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h][-u] [test-list]"
>>>> -}
>>>> -
>>>>  keep_going=false
>>>>  do_list=false
>>>>  full_path=false
>>>>  unittests_only=false
>>>> +load_cmd="dohell 900"
>>>> +load_cmd_pgid_file="/tmp/evl_test_load_cmd_$$"
>>>> +
>>>> +usage() {
>>>> +   cat >&2 <<EOF
>>>> +evl test -h
>>>> +
>>>> +This help text.
>>>> +
>>>> +evl test [ -l "load command" ] [ -k ] [ -u ] [ --list ] [ --List ]
>>>> +
>>>> +Run a basic test/benchmark of evl on your platform, by first starting a
>>>> +few unit tests, then running the latency test under the load generated by
>>>> +"load-command".
>>>> +
>>>> +By default, the load command is "dohell 900", which will generate load during
>>>> +15 minutes. To generate a more realistic load see dohell help.
>>>> +
>>>> +This script accepts the -k option to tell the unit test loop to keep
>>>> +going upon a failing test. Otherwise evl test stops immediately.
>>>> +
>>>
>>> To be fixed wrt the comment on the previous patch.
>>>
>>>> +Use -u if you want to skip the latency test and only run the unittests.
>>>> +
>>>> +Use --list to output all the tests available. --List will print all the tests
>>>> +with their full path.
>>>> +
>>>> +Example:
>>>> +evl test -l "dohell -s 192.168.0.5 -m /mnt -l /ltp"
>>>> +
>>>> +Will generate load including network load using the server at IP address
>>>> +192.168.0.5, some I/O under the moint point /mnt, and the LTP testsuite
>>>> +installed under the /ltp directory, and use the latency test by measuring the
>>>> +timer irq latency.
>>>
>>> We should use stress-ng or something alike, instead of the obsolete
>>> dohell script.
>>
>> Fact is that dohell does better than stress-ng, at least in the configs
>> of the latter that I've seen before.
> 
> just "dohell -s 192.168.0.5 -m /mnt -l /ltp" ? do you know after how
> long (ie, did it take a few hours or just minutes?)
> 

Yes, I ran what Xenomai 3 also runs, for better comparison. That was for
some hours (don't recall exactly anymore). Not just 10 min but also not
for a day or longer.

My theory on this: With RT Preempt, it's more effective to stress the
common code (and locking) paths like stress-ng is triggering. For the
more decoupled co-kernel architecture with way less common locking, and
that rather short and not nested, cache load is more effective.

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> [sorry, missed your reply]
>
> On 01.11.24 18:00, Jorge Ramirez-Ortiz, Gmail wrote:
>> On 13/06/24 17:30:51, Jan Kiszka wrote:
>>> On 13.06.24 16:16, Philippe Gerum wrote:
>>>>
>>>> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
>>>>
>>>>> Stress evl with a load command while running the tests. The load command
>>>>> may be set to an empty string to run tests without stressing the system.
>>>>>
>>>>> To align with xeno-test the -l command line argument was used for the load
>>>>> command. Listing of the unittests can now be done with --list and --List.
>>>>>
>>>>
>>>> Nak. Compat with xeno-test is definitely not a requirement if this means
>>>> breaking the existing evl command line usage. Besides, --List looks
>>>> strange as an option. Let's pick a different option for the load command
>>>> instead.
>>>>
>>>>> The load command is moved to its own process group to allow a clean
>>>>> teardown of the load process with all its subprocesses.
>>>>>
>>>>> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
>>>>> ---
>>>>>  utils/evl-test | 71 +++++++++++++++++++++++++++++++++++++++++++-------
>>>>>  1 file changed, 61 insertions(+), 10 deletions(-)
>>>>>
>>>>> diff --git a/utils/evl-test b/utils/evl-test
>>>>> index 5c1c3da..fb85f69 100644
>>>>> --- a/utils/evl-test
>>>>> +++ b/utils/evl-test
>>>>> @@ -6,33 +6,79 @@ if test \! -d $EVL_TESTDIR; then
>>>>>     exit 2
>>>>>  fi
>>>>>  
>>>>> -usage() {
>>>>> -   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h][-u] [test-list]"
>>>>> -}
>>>>> -
>>>>>  keep_going=false
>>>>>  do_list=false
>>>>>  full_path=false
>>>>>  unittests_only=false
>>>>> +load_cmd="dohell 900"
>>>>> +load_cmd_pgid_file="/tmp/evl_test_load_cmd_$$"
>>>>> +
>>>>> +usage() {
>>>>> +   cat >&2 <<EOF
>>>>> +evl test -h
>>>>> +
>>>>> +This help text.
>>>>> +
>>>>> +evl test [ -l "load command" ] [ -k ] [ -u ] [ --list ] [ --List ]
>>>>> +
>>>>> +Run a basic test/benchmark of evl on your platform, by first starting a
>>>>> +few unit tests, then running the latency test under the load generated by
>>>>> +"load-command".
>>>>> +
>>>>> +By default, the load command is "dohell 900", which will generate load during
>>>>> +15 minutes. To generate a more realistic load see dohell help.
>>>>> +
>>>>> +This script accepts the -k option to tell the unit test loop to keep
>>>>> +going upon a failing test. Otherwise evl test stops immediately.
>>>>> +
>>>>
>>>> To be fixed wrt the comment on the previous patch.
>>>>
>>>>> +Use -u if you want to skip the latency test and only run the unittests.
>>>>> +
>>>>> +Use --list to output all the tests available. --List will print all the tests
>>>>> +with their full path.
>>>>> +
>>>>> +Example:
>>>>> +evl test -l "dohell -s 192.168.0.5 -m /mnt -l /ltp"
>>>>> +
>>>>> +Will generate load including network load using the server at IP address
>>>>> +192.168.0.5, some I/O under the moint point /mnt, and the LTP testsuite
>>>>> +installed under the /ltp directory, and use the latency test by measuring the
>>>>> +timer irq latency.
>>>>
>>>> We should use stress-ng or something alike, instead of the obsolete
>>>> dohell script.
>>>
>>> Fact is that dohell does better than stress-ng, at least in the configs
>>> of the latter that I've seen before.
>> 
>> just "dohell -s 192.168.0.5 -m /mnt -l /ltp" ? do you know after how
>> long (ie, did it take a few hours or just minutes?)
>> 
>
> Yes, I ran what Xenomai 3 also runs, for better comparison. That was for
> some hours (don't recall exactly anymore). Not just 10 min but also not
> for a day or longer.
>
> My theory on this: With RT Preempt, it's more effective to stress the
> common code (and locking) paths like stress-ng is triggering. For the
> more decoupled co-kernel architecture with way less common locking, and
> that rather short and not nested, cache load is more effective.
>

Same conclusion here, this is where the vm stressors may come into
play. They tend to pound the hardware pretty aggressively, in addition
with stressors causing a high rate of context switches (L1 flush +
mandatory hard irqs off section on mmswitch). If we could generate such
load by using the proper stress-ng stressors, we would only depend on a
broadly available tool suite which implements state-of-the-art testing
techniques (Gilles wrote xeno-test 14y ago based on my testing habits
since 2006 or so, which clearly does not qualify anymore).

-- 
Philippe.

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Tobias Schaffner]

Hi Jorge, hi Philippe,

On 18.11.24 09:26, Philippe Gerum wrote:
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> [sorry, missed your reply]
>>
>> On 01.11.24 18:00, Jorge Ramirez-Ortiz, Gmail wrote:
>>> On 13/06/24 17:30:51, Jan Kiszka wrote:
>>>> On 13.06.24 16:16, Philippe Gerum wrote:
>>>>>
>>>>> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
>>>>>
>>>>>> Stress evl with a load command while running the tests. The load command
>>>>>> may be set to an empty string to run tests without stressing the system.
>>>>>>
>>>>>> To align with xeno-test the -l command line argument was used for the load
>>>>>> command. Listing of the unittests can now be done with --list and --List.
>>>>>>
>>>>>
>>>>> Nak. Compat with xeno-test is definitely not a requirement if this means
>>>>> breaking the existing evl command line usage. Besides, --List looks
>>>>> strange as an option. Let's pick a different option for the load command
>>>>> instead.
>>>>>
>>>>>> The load command is moved to its own process group to allow a clean
>>>>>> teardown of the load process with all its subprocesses.
>>>>>>
>>>>>> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
>>>>>> ---
>>>>>>   utils/evl-test | 71 +++++++++++++++++++++++++++++++++++++++++++-------
>>>>>>   1 file changed, 61 insertions(+), 10 deletions(-)
>>>>>>
>>>>>> diff --git a/utils/evl-test b/utils/evl-test
>>>>>> index 5c1c3da..fb85f69 100644
>>>>>> --- a/utils/evl-test
>>>>>> +++ b/utils/evl-test
>>>>>> @@ -6,33 +6,79 @@ if test \! -d $EVL_TESTDIR; then
>>>>>>      exit 2
>>>>>>   fi
>>>>>>   
>>>>>> -usage() {
>>>>>> -   echo >&2 "usage: $(basename $1) [-l][-L][-k][-h][-u] [test-list]"
>>>>>> -}
>>>>>> -
>>>>>>   keep_going=false
>>>>>>   do_list=false
>>>>>>   full_path=false
>>>>>>   unittests_only=false
>>>>>> +load_cmd="dohell 900"
>>>>>> +load_cmd_pgid_file="/tmp/evl_test_load_cmd_$$"
>>>>>> +
>>>>>> +usage() {
>>>>>> +   cat >&2 <<EOF
>>>>>> +evl test -h
>>>>>> +
>>>>>> +This help text.
>>>>>> +
>>>>>> +evl test [ -l "load command" ] [ -k ] [ -u ] [ --list ] [ --List ]
>>>>>> +
>>>>>> +Run a basic test/benchmark of evl on your platform, by first starting a
>>>>>> +few unit tests, then running the latency test under the load generated by
>>>>>> +"load-command".
>>>>>> +
>>>>>> +By default, the load command is "dohell 900", which will generate load during
>>>>>> +15 minutes. To generate a more realistic load see dohell help.
>>>>>> +
>>>>>> +This script accepts the -k option to tell the unit test loop to keep
>>>>>> +going upon a failing test. Otherwise evl test stops immediately.
>>>>>> +
>>>>>
>>>>> To be fixed wrt the comment on the previous patch.
>>>>>
>>>>>> +Use -u if you want to skip the latency test and only run the unittests.
>>>>>> +
>>>>>> +Use --list to output all the tests available. --List will print all the tests
>>>>>> +with their full path.
>>>>>> +
>>>>>> +Example:
>>>>>> +evl test -l "dohell -s 192.168.0.5 -m /mnt -l /ltp"
>>>>>> +
>>>>>> +Will generate load including network load using the server at IP address
>>>>>> +192.168.0.5, some I/O under the moint point /mnt, and the LTP testsuite
>>>>>> +installed under the /ltp directory, and use the latency test by measuring the
>>>>>> +timer irq latency.
>>>>>
>>>>> We should use stress-ng or something alike, instead of the obsolete
>>>>> dohell script.
>>>>
>>>> Fact is that dohell does better than stress-ng, at least in the configs
>>>> of the latter that I've seen before.
>>>
>>> just "dohell -s 192.168.0.5 -m /mnt -l /ltp" ? do you know after how
>>> long (ie, did it take a few hours or just minutes?)
>>>
>>
>> Yes, I ran what Xenomai 3 also runs, for better comparison. That was for
>> some hours (don't recall exactly anymore). Not just 10 min but also not
>> for a day or longer.
>>
>> My theory on this: With RT Preempt, it's more effective to stress the
>> common code (and locking) paths like stress-ng is triggering. For the
>> more decoupled co-kernel architecture with way less common locking, and
>> that rather short and not nested, cache load is more effective.
>>
> 
> Same conclusion here, this is where the vm stressors may come into
> play. They tend to pound the hardware pretty aggressively, in addition
> with stressors causing a high rate of context switches (L1 flush +
> mandatory hard irqs off section on mmswitch). If we could generate such
> load by using the proper stress-ng stressors, we would only depend on a
> broadly available tool suite which implements state-of-the-art testing
> techniques (Gilles wrote xeno-test 14y ago based on my testing habits
> since 2006 or so, which clearly does not qualify anymore).


I’d like to bring this up again.

Thanks, Jorge, for sharing your first results in the other subthread. 
 From my understanding, it’s still unclear whether the stress-ng vm 
stressors load the system as heavily as dohell does. Have you been able 
to confirm this in the meantime? Do you plan to continue this work, or 
are you expecting someone else to pick it up?

Best,
Tobias

Re: [libevl][PATCH 1/4] Copy dohell from Xenomai 3

[Jorge Ramirez-Ortiz, Gmail]

On 13/06/24 17:55:02, Philippe Gerum wrote:
> 
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
> > On 13.06.24 16:23, Philippe Gerum wrote:
> >> 
> >> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
> >> 
> >>> Dohell is used as the default load generator in Xenomai 3. Add it to
> >>> libevl to be able to stress the system while testing evl.
> >>>
> >> 
> >> dohell is obsolete. Let's move to stress-ng or something alike for x4.
> >> 
> >
> > See my comment on patch 3: If you can name a configuration of stress-ng
> > that stresses a co-kernel more than what dohell does, we can take it.
> > Otherwise, I would strongly recommend to not do less hell over here.
> >
> 
> The most important stress dohell applies is basically hackbench and a dd
> loop trashing the D-cache. The latter can be obtained with a combo of
> VM+I/O tests from stress-ng. So basically, the whole issue boils down to
> having hackbench on board, provided we cannot generate the same load
> profile with stress-ng with a CPU bound stressor. Besides, I never saw
> any added value in running ltp tests compared to a combo of stress-ng
> stressors when it comes to latency testing.
> 
> We obviously don't want to throw less hell, but throwing more obsolete
> stuff is not an option either. So that's a NAK for v4 ATM, I'll gather
> more information about how we could leverage stress-ng in order to
> compare to dohell. This is worth a shot.

I should start by apologizing for not having delivered a proper,
stress-ng case study yet. In my defense (if that counts), I got
sidetracked by day-to-day work and the long process of getting latmon gpio
support merged into Zephyr[1] — which ended up taking > six months of
back-and-forth.

BTW If you have access to one of these devices [2] (they’re inexpensive),
plus a couple of GPIOs on your Xenomai-enabled board and a network
interface, they make it quite practical to collect long-term traces in
an automated way.

So comming back to the discussion, I still believe that the better path
is to validate and if necessary extend stress-ng. Just as monitoring
tools like latmon dont belong in the Xenomai repository (the Zephyr work
above mentioned is a step on that direction), I think stress tools
should be kept out as well.

IMO this separation of concerns will also help those trying to decide
between PREEMPT_RT and Xenomai for their use cases.

[1] https://docs.zephyrproject.org/latest/samples/net/latmon/README.html
[2] https://docs.zephyrproject.org/latest/boards/nxp/frdm_k64f/doc/index.html 

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Jan Kiszka]

On 01.11.24 17:55, Jorge Ramirez-Ortiz, Gmail wrote:
> On 27/10/24 20:13:08, Jorge Ramirez-Ortiz, Gmail wrote:
>> On 24/10/24 20:00:12, Schaffner, Tobias wrote:
>>> Hi Philippe, hi Jorge,
>>>
>>> On Wed, 2024-07-10 at 09:18 +0200, Tobias Schaffner wrote:
>>>> Hey Jorge,
>>>>
>>>> On 14.06.24 09:29, Jorge Ramirez-Ortiz, Gmail wrote:
>>>>> On 13/06/24 19:27:10, Philippe Gerum wrote:
>>>>>>
>>>>>> Philippe Gerum <rpm@xenomai.org> writes:
>>>>>>
>>>>>>> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
>>>>>>>
>>>>>>>> Stress evl with a load command while running the tests. The
>>>>>>>> load command
>>>>>>>> may be set to an empty string to run tests without stressing
>>>>>>>> the system.
>>>>>>>>
>>>>>>>> To align with xeno-test the -l command line argument was used
>>>>>>>> for the load
>>>>>>>> command. Listing of the unittests can now be done with --list
>>>>>>>> and --List.
>>>>>>>>
>>>>>>>
>>>>>>> Nak. Compat with xeno-test is definitely not a requirement if
>>>>>>> this means
>>>>>>> breaking the existing evl command line usage. Besides, --List
>>>>>>> looks
>>>>>>> strange as an option. Let's pick a different option for the
>>>>>>> load command
>>>>>>> instead.
>>>>>>>
>>>>>>
>>>>>> We may want to implement the stress utility as a separate evl-
>>>>>> stress
>>>>>> command, keeping evl-test for unit testing which has a different
>>>>>> purpose. evl-stress as a script could generate the load (dohell)
>>>>>> and
>>>>>> possibly check the latency figures reported by latmus (using the
>>>>>> -K and
>>>>>> -A switches to detect weird behavior).
>>>>>>
>>>>>> PS: Jorge is going to look into leveraging stress-ng for load
>>>>>> generation.
>>>>>
>>>>> yes I am going to start looking into it. I think that maintaining
>>>>> Linux
>>>>> stressors should not be Xenomai's business. If we can't replicate
>>>>> the
>>>>> load with todays stress-ng perhaps we could just usptream what is
>>>>> needed (stress-ng is well maintained)
>>>>
>>>
>>> may I bring this up again? Any new developments or opinions on this?
>>> I would like to see a solution that allows to compare xenomai 3 and
>>> EVL. Reliable numbers are necessary to make an informed decision if EVL
>>> is on par or even better than xenomai 3. Imo we are comparing apples
>>> and oranges as long as we are not using the same tooling on both
>>> platforms.
>>
>> Sorry about this, it is being a bit of a crazy year for me at a personal
>> levely. I am start on it this week.
>>
>> My setup uses some Zephyr code executing on frdmk64 I wrote years ago to
>> benchmark evl
>>
>> https://github.com/ldts/libevl/commit/905d1946ca117e3889c53d1150a1334cc91033a8
>>
>> I will load my evl system using stress-ng and measure latency figures
>> and then replicate with dohell loading tools. I should have something to
>> share by next friday.
>>
>> feel free to ping me anytime. In IRC you can find me in LiberaChat
>> #u-boot (nick ldts) and since recently in OFTC #edk2
> 
> I was finally able to get some numbers on imx8mm (an Arduino Pro with
> the Portenta breakout board recently contributed to my lab by Wim
> Taymans for audio testing of the Xenomai4 Pipewire-plugin[0] for
> realtime audio)
> 
> I am using dohell/rt-tools/ltp and stress-ng on EVL - havent tested
> Xenomai3.
> 
> My test bench runs with the help of an external FRDMK64F running Zephyr
> very similar to what is described in the latmus/latmon combo
> documentation [1]. Given how cheap this board is perhaps it should be
> more widely used (?).
> 
> [RFC]
> I would like to update latmon[2] to the tip of Zephyr - it has fallen
> behind - I'd like then to propose it upstream (if accepted I would like
> to propose that it is removed from EVL). Anyone disagrees?
> 
> IMO We should also take latmus to its own separate tree dedicated to RT
> benchmarking and extend it to support scheduling latency metrics based
> on GPIO data.
> [\]
> 
> So:
> 
> * stress-ng --vm 2 --vm-bytes 1G --mmap 2 --mmap-bytes 1G --page-in \
>   --matrix 0 --matrix-size 64 --tz -t 60
> 
> This simple command (VM and cpu load for a 2GB DDR system) hits the
> system very quick and my externally measured GPIO IRQ latency peaks
> within seconds.
> 
> stress-ng has FPU, VM, CPU all sort of stressors but I didnt need to go
> into those just yet.
> 
> I found some info that RH [3] is providing interesting - still the
> stress-ng code is simple enough to work with if needed (really good
> encapsulation and code base)
> 
> * dohell -l /opt/ltp -b /bin/hackbench -m /home/root
> 
> After five minutes of run the latency is still contained (far from
> peaking, 50% below the peak) but I assume this is because LTP takes a
> while to run all tests. Without going into more details, this difference
> in the invested time seems enough for me to switch to stress-ng.
> 

I think we should split these activities into two streams:
 - short-term: setup of a well-defined but not necessarily 100%
   finalized test environment for EVL so that we can run that
   automatically in CI
 - long-term: evaluation of background load profiles and tests to
   trigger maximal latencies, with EVL, Xenomai 3 and may even
   PREEMPT_RT

With the first item solved, the second could even be automated and may
not need some poor intern to press buttons and record numbers on a piece
of paper ;).

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Jorge Ramirez-Ortiz, Gmail]

On 03/09/25 07:29:07, Jan Kiszka wrote:
> On 01.11.24 17:55, Jorge Ramirez-Ortiz, Gmail wrote:
> > On 27/10/24 20:13:08, Jorge Ramirez-Ortiz, Gmail wrote:
> >> On 24/10/24 20:00:12, Schaffner, Tobias wrote:
> >>> Hi Philippe, hi Jorge,
> >>>
> >>> On Wed, 2024-07-10 at 09:18 +0200, Tobias Schaffner wrote:
> >>>> Hey Jorge,
> >>>>
> >>>> On 14.06.24 09:29, Jorge Ramirez-Ortiz, Gmail wrote:
> >>>>> On 13/06/24 19:27:10, Philippe Gerum wrote:
> >>>>>>
> >>>>>> Philippe Gerum <rpm@xenomai.org> writes:
> >>>>>>
> >>>>>>> Tobias Schaffner <tobias.schaffner@siemens.com> writes:
> >>>>>>>
> >>>>>>>> Stress evl with a load command while running the tests. The
> >>>>>>>> load command
> >>>>>>>> may be set to an empty string to run tests without stressing
> >>>>>>>> the system.
> >>>>>>>>
> >>>>>>>> To align with xeno-test the -l command line argument was used
> >>>>>>>> for the load
> >>>>>>>> command. Listing of the unittests can now be done with --list
> >>>>>>>> and --List.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Nak. Compat with xeno-test is definitely not a requirement if
> >>>>>>> this means
> >>>>>>> breaking the existing evl command line usage. Besides, --List
> >>>>>>> looks
> >>>>>>> strange as an option. Let's pick a different option for the
> >>>>>>> load command
> >>>>>>> instead.
> >>>>>>>
> >>>>>>
> >>>>>> We may want to implement the stress utility as a separate evl-
> >>>>>> stress
> >>>>>> command, keeping evl-test for unit testing which has a different
> >>>>>> purpose. evl-stress as a script could generate the load (dohell)
> >>>>>> and
> >>>>>> possibly check the latency figures reported by latmus (using the
> >>>>>> -K and
> >>>>>> -A switches to detect weird behavior).
> >>>>>>
> >>>>>> PS: Jorge is going to look into leveraging stress-ng for load
> >>>>>> generation.
> >>>>>
> >>>>> yes I am going to start looking into it. I think that maintaining
> >>>>> Linux
> >>>>> stressors should not be Xenomai's business. If we can't replicate
> >>>>> the
> >>>>> load with todays stress-ng perhaps we could just usptream what is
> >>>>> needed (stress-ng is well maintained)
> >>>>
> >>>
> >>> may I bring this up again? Any new developments or opinions on this?
> >>> I would like to see a solution that allows to compare xenomai 3 and
> >>> EVL. Reliable numbers are necessary to make an informed decision if EVL
> >>> is on par or even better than xenomai 3. Imo we are comparing apples
> >>> and oranges as long as we are not using the same tooling on both
> >>> platforms.
> >>
> >> Sorry about this, it is being a bit of a crazy year for me at a personal
> >> levely. I am start on it this week.
> >>
> >> My setup uses some Zephyr code executing on frdmk64 I wrote years ago to
> >> benchmark evl
> >>
> >> https://github.com/ldts/libevl/commit/905d1946ca117e3889c53d1150a1334cc91033a8
> >>
> >> I will load my evl system using stress-ng and measure latency figures
> >> and then replicate with dohell loading tools. I should have something to
> >> share by next friday.
> >>
> >> feel free to ping me anytime. In IRC you can find me in LiberaChat
> >> #u-boot (nick ldts) and since recently in OFTC #edk2
> > 
> > I was finally able to get some numbers on imx8mm (an Arduino Pro with
> > the Portenta breakout board recently contributed to my lab by Wim
> > Taymans for audio testing of the Xenomai4 Pipewire-plugin[0] for
> > realtime audio)
> > 
> > I am using dohell/rt-tools/ltp and stress-ng on EVL - havent tested
> > Xenomai3.
> > 
> > My test bench runs with the help of an external FRDMK64F running Zephyr
> > very similar to what is described in the latmus/latmon combo
> > documentation [1]. Given how cheap this board is perhaps it should be
> > more widely used (?).
> > 
> > [RFC]
> > I would like to update latmon[2] to the tip of Zephyr - it has fallen
> > behind - I'd like then to propose it upstream (if accepted I would like
> > to propose that it is removed from EVL). Anyone disagrees?
> > 
> > IMO We should also take latmus to its own separate tree dedicated to RT
> > benchmarking and extend it to support scheduling latency metrics based
> > on GPIO data.
> > [\]
> > 
> > So:
> > 
> > * stress-ng --vm 2 --vm-bytes 1G --mmap 2 --mmap-bytes 1G --page-in \
> >   --matrix 0 --matrix-size 64 --tz -t 60
> > 
> > This simple command (VM and cpu load for a 2GB DDR system) hits the
> > system very quick and my externally measured GPIO IRQ latency peaks
> > within seconds.
> > 
> > stress-ng has FPU, VM, CPU all sort of stressors but I didnt need to go
> > into those just yet.
> > 
> > I found some info that RH [3] is providing interesting - still the
> > stress-ng code is simple enough to work with if needed (really good
> > encapsulation and code base)
> > 
> > * dohell -l /opt/ltp -b /bin/hackbench -m /home/root
> > 
> > After five minutes of run the latency is still contained (far from
> > peaking, 50% below the peak) but I assume this is because LTP takes a
> > while to run all tests. Without going into more details, this difference
> > in the invested time seems enough for me to switch to stress-ng.
> > 
> 
> I think we should split these activities into two streams:
>  - short-term: setup of a well-defined but not necessarily 100%
>    finalized test environment for EVL so that we can run that
>    automatically in CI

100%. Do you think it would be at all possible to setup a simple LAVA
lab with some reference hardware - unless Xenomai has one already?

I am not sure how it would be funded - do you know ? - but it could then
also be connected to KernelCI so that Xenomai enabled systems avoid
regressions.

https://docs.kernelci.org/intro/platform-testing/

Is this something at all possible you think?


>  - long-term: evaluation of background load profiles and tests to
>    trigger maximal latencies, with EVL, Xenomai 3 and may even
>    PREEMPT_RT
> 
> With the first item solved, the second could even be automated and may
> not need some poor intern to press buttons and record numbers on a piece
> of paper ;).
> 
> Jan
> 
> -- 
> Siemens AG, Foundational Technologies
> Linux Expert Center

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Florian Bezdeka]

[snip]

On Wed, 2025-09-03 at 10:12 +0200, Jorge Ramirez-Ortiz, Gmail wrote:
> > 
> > I think we should split these activities into two streams:
> >  - short-term: setup of a well-defined but not necessarily 100%
> >    finalized test environment for EVL so that we can run that
> >    automatically in CI
> 
> 100%. Do you think it would be at all possible to setup a simple LAVA
> lab with some reference hardware - unless Xenomai has one already?
> 
> I am not sure how it would be funded - do you know ? - but it could then
> also be connected to KernelCI so that Xenomai enabled systems avoid
> regressions.
> 
> https://docs.kernelci.org/intro/platform-testing/
> 
> Is this something at all possible you think?

See lava.xenomai.org.

We had two labs in the past, one operated by Siemens, the other WAS
operated by Intel but has been decommissioned recently. (Hardware)
Contributions / further labs highly appreciated.

Revisiting the kernelci topic is on my TODO list for dovetail
maintenance reasons. No timeline yet and unclear how/if we would
integrate that.

> 
> 
> >  - long-term: evaluation of background load profiles and tests to
> >    trigger maximal latencies, with EVL, Xenomai 3 and may even
> >    PREEMPT_RT
> > 
> > With the first item solved, the second could even be automated and may
> > not need some poor intern to press buttons and record numbers on a piece
> > of paper ;).
> > 
> > Jan
> > 

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Jorge Ramirez-Ortiz, Gmail]

On 03/09/25 10:20:01, Florian Bezdeka wrote:
> [snip]
> 
> On Wed, 2025-09-03 at 10:12 +0200, Jorge Ramirez-Ortiz, Gmail wrote:
> > > 
> > > I think we should split these activities into two streams:
> > >  - short-term: setup of a well-defined but not necessarily 100%
> > >    finalized test environment for EVL so that we can run that
> > >    automatically in CI
> > 
> > 100%. Do you think it would be at all possible to setup a simple LAVA
> > lab with some reference hardware - unless Xenomai has one already?
> > 
> > I am not sure how it would be funded - do you know ? - but it could then
> > also be connected to KernelCI so that Xenomai enabled systems avoid
> > regressions.
> > 
> > https://docs.kernelci.org/intro/platform-testing/
> > 
> > Is this something at all possible you think?
> 
> See lava.xenomai.org.

nice, is there some document that I can read about this lava instance?
boards, tests and so on.

since it is maintained by Siemens, who can get an account and access to
the test definitions? 

> 
> We had two labs in the past, one operated by Siemens, the other WAS
> operated by Intel but has been decommissioned recently. (Hardware)
> Contributions / further labs highly appreciated.

ouch, ok. Does Xenomai get any funding ? maybe we could setup something
with Liberapay to get donations for a company independent LAVA lab.

sorry for all the questions, just trying to understand where the project
is since I think there could be value in having an open lab - or maybe it
is in the works already?

> 
> Revisiting the kernelci topic is on my TODO list for dovetail
> maintenance reasons. No timeline yet and unclear how/if we would
> integrate that.

ok.

> 
> > 
> > 
> > >  - long-term: evaluation of background load profiles and tests to
> > >    trigger maximal latencies, with EVL, Xenomai 3 and may even
> > >    PREEMPT_RT
> > > 
> > > With the first item solved, the second could even be automated and may
> > > not need some poor intern to press buttons and record numbers on a piece
> > > of paper ;).
> > > 
> > > Jan
> > > 

Re: [libevl][PATCH 3/4] evl-test: Start evl-test with load by default

[Florian Bezdeka]

On Wed, 2025-09-03 at 10:45 +0200, Jorge Ramirez-Ortiz, Gmail wrote:
> On 03/09/25 10:20:01, Florian Bezdeka wrote:
> > [snip]
> > 
> > On Wed, 2025-09-03 at 10:12 +0200, Jorge Ramirez-Ortiz, Gmail wrote:
> > > > 
> > > > I think we should split these activities into two streams:
> > > >  - short-term: setup of a well-defined but not necessarily 100%
> > > >    finalized test environment for EVL so that we can run that
> > > >    automatically in CI
> > > 
> > > 100%. Do you think it would be at all possible to setup a simple LAVA
> > > lab with some reference hardware - unless Xenomai has one already?
> > > 
> > > I am not sure how it would be funded - do you know ? - but it could then
> > > also be connected to KernelCI so that Xenomai enabled systems avoid
> > > regressions.
> > > 
> > > https://docs.kernelci.org/intro/platform-testing/
> > > 
> > > Is this something at all possible you think?
> > 
> > See lava.xenomai.org.
> 
> nice, is there some document that I can read about this lava instance?
> boards, tests and so on.
> 
> since it is maintained by Siemens, who can get an account and access to
> the test definitions? 

The complete setup is a bit different for this project, so that
accounts to the LAVA instance itself should not be necessary.

We test through the public available xenomai-images [1] project, so
generating images and deploy them to the boards. The xenomai [2]
project is linked to the CI pipelines in xenomai-images, so that
updates to the xenomai repository trigger CI runs of xenomai-images.

The list of boards (real and virtual ones) can be found in [1]. The
test definitions reside inside the tests/jobs directory.

To use this infrastructure you must be allowed to push something to the
affected repositories. Main reason is that we have to care about the
build resources / costs. There is no process defined yet how to get
those permissions. Good community reputation / regular contributions
might be a good starting point. 

> 
> > 
> > We had two labs in the past, one operated by Siemens, the other WAS
> > operated by Intel but has been decommissioned recently. (Hardware)
> > Contributions / further labs highly appreciated.
> 
> ouch, ok. Does Xenomai get any funding ? maybe we could setup something
> with Liberapay to get donations for a company independent LAVA lab.

Well, at the end it needs someone to be responsible for the lab and
regularly care about issues like board hangs / network issues / secure
token exchange ...

It's not just the funding. We would need some kind of "stable
agreement" to operate the lab. Even if funding might get stuck at some
point.

> 
> sorry for all the questions, just trying to understand where the project
> is since I think there could be value in having an open lab - or maybe it
> is in the works already?

Well the lab is somehow "half-open". It just needs 

- the patches you want to test => paste them to the mailing list
- someone triggering a CI run
  If you're not allowed to do so, just ask someone with the necessary
  permissions to do so.
  This is the normal workflow anyways. After patch review your patches
  will land in the xenomai [2] next branch, where CI runs will be
  triggered automatically by Jan.

> 
> > 
> > Revisiting the kernelci topic is on my TODO list for dovetail
> > maintenance reasons. No timeline yet and unclear how/if we would
> > integrate that.
> 
> ok.
> 
> > 
> > > 
> > > 
> > > >  - long-term: evaluation of background load profiles and tests to
> > > >    trigger maximal latencies, with EVL, Xenomai 3 and may even
> > > >    PREEMPT_RT
> > > > 
> > > > With the first item solved, the second could even be automated and may
> > > > not need some poor intern to press buttons and record numbers on a piece
> > > > of paper ;).
> > > > 

[1] https://gitlab.com/Xenomai/xenomai-images
[2] https://gitlab.com/Xenomai/xenomai3/xenomai