[PATCH xenomai-images v2 0/3] longterm EVL testing under load

[PATCH xenomai-images v2 0/3] longterm EVL testing under load

[Tobias Schaffner]

Hi all,

This series introduces EVL test runs with additional stress applied via
stress-ng, as well as long-term testing. Along the way, I also updated the
long-term kernel under test from 6.12 to 6.18.

Please note:
 * The clock-timer-periodic test currently fails as soon as stress is
   applied. I proposed "tests: skip clock-timer-periodic on virtual targets"
   to address this.
 * On the Raspberry Pi 4, even the 3-minute test exceeds the latency
   threshold with debugging disabled:
   latency threshold is exceeded (150000 >= 163.951), aborting.
   https://gitlab.com/Xenomai/xenomai-images/-/jobs/15353275174

Best,
Tobias

Changes since v1:
 * Make stress-ng a dependency of the libevl-test debian package

Tobias Schaffner (3):
  ci: use stress-ng to stress EVL tests
  ci: Update for longterm test runs from 6.12 to 6.18
  ci: add long-term EVL CI test pipeline

 .gitlab-ci.yml                                |  8 +++
 ci/lib/gitlab-ci-base.yml                     |  2 +
 ci/lib/kernel/6_12.yml                        | 45 ++++++++++++++--
 ci/lib/kernel/6_12_physical.yml               | 52 -------------------
 ci/lib/kernel/6_18.yml                        | 42 ++-------------
 ci/lib/kernel/6_18_physical.yml               | 52 +++++++++++++++++++
 ci/xenomai_3_next_long_term.yml               |  4 +-
 ci/xenomai_4_evl.yml                          |  2 +-
 ci/xenomai_4_next.yml                         |  2 +-
 ci/xenomai_4_next_long_term.yml               | 24 +++++++++
 .../libevl/files/debian/control.tmpl          |  2 +-
 11 files changed, 136 insertions(+), 99 deletions(-)
 delete mode 100644 ci/lib/kernel/6_12_physical.yml
 create mode 100644 ci/lib/kernel/6_18_physical.yml
 create mode 100644 ci/xenomai_4_next_long_term.yml

-- 
2.43.0

[PATCH xenomai-images v2 1/3] ci: use stress-ng to stress EVL tests

[Tobias Schaffner]

Use stress-ng to apply CPU, virtual memory, FPU register, and procfs load
during EVL tests, mimicking the stressors used in Xenomai 3.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 ci/lib/gitlab-ci-base.yml                        | 2 ++
 ci/xenomai_4_evl.yml                             | 2 +-
 ci/xenomai_4_next.yml                            | 2 +-
 recipes-xenomai/libevl/files/debian/control.tmpl | 2 +-
 4 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/ci/lib/gitlab-ci-base.yml b/ci/lib/gitlab-ci-base.yml
index 0540628..a3359bb 100644
--- a/ci/lib/gitlab-ci-base.yml
+++ b/ci/lib/gitlab-ci-base.yml
@@ -25,6 +25,8 @@ variables:
   ADDITIONAL_TEST_TIMEOUT_MINUTES: 10
   TEST_MAXIMUM_LATENCY: 0
   TEST_COMMAND: "xeno-test -k -l 'dohell -m /tmp ${TEST_RUNTIME_SECONDS}' ${ADDITIONAL_XENO_TEST_ARGS}"
+  EVL_STRESS_COMMAND: stress-ng --cpu 1 --vm 1 --vm-bytes 64M --fp 1 --procfs 1
+  EVL_TEST_COMMAND: evl test --stress-cmd "${EVL_STRESS_COMMAND}" -k $(evl test -l | grep -Ev "${EVL_TESTS_SKIP_PATTERN}")
   SSTATE_LOCATION: "s3://xenomai-images-artifacts/sstate"
   SSTATE_MIRRORS: "file://.* ${SSTATE_LOCATION}/PATH;downloadfilename=PATH"
   SSTATE_MAX_AGE: 14d
diff --git a/ci/xenomai_4_evl.yml b/ci/xenomai_4_evl.yml
index 283a1fd..c6a92b8 100644
--- a/ci/xenomai_4_evl.yml
+++ b/ci/xenomai_4_evl.yml
@@ -29,4 +29,4 @@ variables:
   XENOMAI_VERSION: "4-next"
   BUILD_IDENTIFIER: "evl-${XENOMAI_VERSION}_kernel-${KERNEL_VERSION}"
   CACHE_KERNEL: evl
-  TEST_COMMAND: "evl test -k $(evl test -l | grep -Ev \"${EVL_TESTS_SKIP_PATTERN}\")"
+  TEST_COMMAND: "${EVL_TEST_COMMAND}"
diff --git a/ci/xenomai_4_next.yml b/ci/xenomai_4_next.yml
index 6ef4b6a..7ecfbb5 100644
--- a/ci/xenomai_4_next.yml
+++ b/ci/xenomai_4_next.yml
@@ -21,4 +21,4 @@ variables:
   KERNEL_VERSION_WITH_QEMU: "head"
   BUILD_IDENTIFIER: "evl-${XENOMAI_VERSION}_kernel-${KERNEL_VERSION}"
   CACHE_KERNEL: evl
-  TEST_COMMAND: "evl test -k $(evl test -l | grep -Ev \"${EVL_TESTS_SKIP_PATTERN}\")"
+  TEST_COMMAND: "${EVL_TEST_COMMAND}"
diff --git a/recipes-xenomai/libevl/files/debian/control.tmpl b/recipes-xenomai/libevl/files/debian/control.tmpl
index 93caffe..2c90442 100644
--- a/recipes-xenomai/libevl/files/debian/control.tmpl
+++ b/recipes-xenomai/libevl/files/debian/control.tmpl
@@ -13,4 +13,4 @@ Depends: ${shlibs:Depends}
 Package: libevl-test
 Architecture: ${DISTRO_ARCH}
 Description: ${DESCRIPTION} - test suite
-Depends: libevl, gdb, ${shlibs:Depends}
+Depends: libevl, gdb, stress-ng, ${shlibs:Depends}
-- 
2.43.0

[PATCH xenomai-images v2 2/3] ci: Update for longterm test runs from 6.12 to 6.18

[Tobias Schaffner]

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 ci/lib/kernel/6_12.yml          | 45 ++++++++++++++++++++++++++--
 ci/lib/kernel/6_12_physical.yml | 52 ---------------------------------
 ci/lib/kernel/6_18.yml          | 42 ++------------------------
 ci/lib/kernel/6_18_physical.yml | 52 +++++++++++++++++++++++++++++++++
 ci/xenomai_3_next_long_term.yml |  4 +--
 5 files changed, 99 insertions(+), 96 deletions(-)
 delete mode 100644 ci/lib/kernel/6_12_physical.yml
 create mode 100644 ci/lib/kernel/6_18_physical.yml

diff --git a/ci/lib/kernel/6_12.yml b/ci/lib/kernel/6_12.yml
index a9bed87..529d24d 100644
--- a/ci/lib/kernel/6_12.yml
+++ b/ci/lib/kernel/6_12.yml
@@ -9,9 +9,6 @@
 # SPDX-License-Identifier: MIT
 #
 
-include:
-  - local: '/ci/lib/kernel/6_12_physical.yml'
-
 build-6.12:qemu-amd64:
   extends: .build:qemu-amd64
   variables:
@@ -50,3 +47,45 @@ lava-test-6.12:qemu-arm64:
   extends: .lava-test:qemu-arm64
   variables:
     KERNEL_VERSION: "6.12"
+
+build-6.12:rpi4:
+  extends: .build:rpi4
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
+    KERNEL_VERSION: "6.12"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.12:rpi4:
+  needs: [ "build-6.12:rpi4" ]
+  extends: .lava-test:rpi4
+  variables:
+    KERNEL_VERSION: "6.12"
+    TEST_MAXIMUM_LATENCY: 150
+
+build-6.12:beagle-bone-black:
+  extends: .build:beagle-bone-black
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
+    KERNEL_VERSION: "6.12"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.12:beagle-bone-black:
+  needs: [ "build-6.12:beagle-bone-black" ]
+  extends: .lava-test:beagle-bone-black
+  variables:
+    KERNEL_VERSION: "6.12"
+    TEST_MAXIMUM_LATENCY: 150
+
+build-6.12:x86-64-efi:
+  extends: .build:x86-64-efi
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
+    KERNEL_VERSION: "6.12"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.12:x86-64-efi:
+  needs: [ "build-6.12:x86-64-efi" ]
+  extends: .lava-test:x86-64-efi
+  variables:
+    KERNEL_VERSION: "6.12"
+    TEST_MAXIMUM_LATENCY: 100
diff --git a/ci/lib/kernel/6_12_physical.yml b/ci/lib/kernel/6_12_physical.yml
deleted file mode 100644
index 86cd23c..0000000
--- a/ci/lib/kernel/6_12_physical.yml
+++ /dev/null
@@ -1,52 +0,0 @@
-#
-# Xenomai Real-Time System
-#
-# Copyright (c) Siemens AG, 2024-2025
-#
-# Authors:
-#  Jan Kiszka <jan.kiszka@siemens.com>
-#
-# SPDX-License-Identifier: MIT
-#
-
-build-6.12:rpi4:
-  extends: .build:rpi4
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
-    KERNEL_VERSION: "6.12"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.12:rpi4:
-  needs: [ "build-6.12:rpi4" ]
-  extends: .lava-test:rpi4
-  variables:
-    KERNEL_VERSION: "6.12"
-    TEST_MAXIMUM_LATENCY: 150
-
-build-6.12:beagle-bone-black:
-  extends: .build:beagle-bone-black
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
-    KERNEL_VERSION: "6.12"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.12:beagle-bone-black:
-  needs: [ "build-6.12:beagle-bone-black" ]
-  extends: .lava-test:beagle-bone-black
-  variables:
-    KERNEL_VERSION: "6.12"
-    TEST_MAXIMUM_LATENCY: 150
-
-build-6.12:x86-64-efi:
-  extends: .build:x86-64-efi
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
-    KERNEL_VERSION: "6.12"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.12:x86-64-efi:
-  needs: [ "build-6.12:x86-64-efi" ]
-  extends: .lava-test:x86-64-efi
-  variables:
-    KERNEL_VERSION: "6.12"
-    TEST_MAXIMUM_LATENCY: 100
diff --git a/ci/lib/kernel/6_18.yml b/ci/lib/kernel/6_18.yml
index 88e3915..5d7e409 100644
--- a/ci/lib/kernel/6_18.yml
+++ b/ci/lib/kernel/6_18.yml
@@ -9,6 +9,9 @@
 # SPDX-License-Identifier: MIT
 #
 
+include:
+  - local: '/ci/lib/kernel/6_18_physical.yml'
+
 build-6.18:qemu-amd64:
   extends: .build:qemu-amd64
   variables:
@@ -47,42 +50,3 @@ lava-test-6.18:qemu-arm64:
   extends: .lava-test:qemu-arm64
   variables:
     KERNEL_VERSION: "6.18"
-
-build-6.18:rpi4:
-  extends: .build:rpi4
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
-    KERNEL_VERSION: "6.18"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.18:rpi4:
-  needs: [ "build-6.18:rpi4" ]
-  extends: .lava-test:rpi4
-  variables:
-    KERNEL_VERSION: "6.18"
-
-build-6.18:beagle-bone-black:
-  extends: .build:beagle-bone-black
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
-    KERNEL_VERSION: "6.18"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.18:beagle-bone-black:
-  needs: [ "build-6.18:beagle-bone-black" ]
-  extends: .lava-test:beagle-bone-black
-  variables:
-    KERNEL_VERSION: "6.18"
-
-build-6.18:x86-64-efi:
-  extends: .build:x86-64-efi
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
-    KERNEL_VERSION: "6.18"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.18:x86-64-efi:
-  needs: [ "build-6.18:x86-64-efi" ]
-  extends: .lava-test:x86-64-efi
-  variables:
-    KERNEL_VERSION: "6.18"
diff --git a/ci/lib/kernel/6_18_physical.yml b/ci/lib/kernel/6_18_physical.yml
new file mode 100644
index 0000000..68b7892
--- /dev/null
+++ b/ci/lib/kernel/6_18_physical.yml
@@ -0,0 +1,52 @@
+#
+# Xenomai Real-Time System
+#
+# Copyright (c) Siemens AG, 2024-2026
+#
+# Authors:
+#  Jan Kiszka <jan.kiszka@siemens.com>
+#
+# SPDX-License-Identifier: MIT
+#
+
+build-6.18:rpi4:
+  extends: .build:rpi4
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
+    KERNEL_VERSION: "6.18"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.18:rpi4:
+  needs: [ "build-6.18:rpi4" ]
+  extends: .lava-test:rpi4
+  variables:
+    KERNEL_VERSION: "6.18"
+    TEST_MAXIMUM_LATENCY: 150
+
+build-6.18:beagle-bone-black:
+  extends: .build:beagle-bone-black
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
+    KERNEL_VERSION: "6.18"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.18:beagle-bone-black:
+  needs: [ "build-6.18:beagle-bone-black" ]
+  extends: .lava-test:beagle-bone-black
+  variables:
+    KERNEL_VERSION: "6.18"
+    TEST_MAXIMUM_LATENCY: 150
+
+build-6.18:x86-64-efi:
+  extends: .build:x86-64-efi
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
+    KERNEL_VERSION: "6.18"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.18:x86-64-efi:
+  needs: [ "build-6.18:x86-64-efi" ]
+  extends: .lava-test:x86-64-efi
+  variables:
+    KERNEL_VERSION: "6.18"
+    TEST_MAXIMUM_LATENCY: 100
diff --git a/ci/xenomai_3_next_long_term.yml b/ci/xenomai_3_next_long_term.yml
index da8749e..c22374c 100644
--- a/ci/xenomai_3_next_long_term.yml
+++ b/ci/xenomai_3_next_long_term.yml
@@ -11,11 +11,11 @@
 
 include:
   - local: '/ci/lib/gitlab-ci-base.yml'
-  - local: '/ci/lib/kernel/6_12_physical.yml'
+  - local: '/ci/lib/kernel/6_18_physical.yml'
 
 variables:
   XENOMAI_BUILD_OPTION: ":kas/opt/xenomai-next.yaml"
   XENOMAI_VERSION: "3-next"
-  KERNEL_VERSION_WITH_QEMU: "6.12"
+  KERNEL_VERSION_WITH_QEMU: "6.18"
   TEST_RUNTIME_SECONDS: 3600
   TEST_COMMAND: "xeno-test -k -l 'dohell -m /tmp ${TEST_RUNTIME_SECONDS}' ${ADDITIONAL_XENO_TEST_ARGS} -- -m ${TEST_MAXIMUM_LATENCY}"
-- 
2.43.0

[PATCH xenomai-images v2 3/3] ci: add long-term EVL CI test pipeline

[Tobias Schaffner]

Add a scheduled Xenomai 4-next long-term CI job with a dedicated
pipeline configuration running extended EVL stress and latency tests.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 .gitlab-ci.yml                  |  8 ++++++++
 ci/xenomai_4_next_long_term.yml | 24 ++++++++++++++++++++++++
 2 files changed, 32 insertions(+)
 create mode 100644 ci/xenomai_4_next_long_term.yml

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index b3d025e..589db8e 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -91,6 +91,14 @@ xenomai-4/next:
     include:
       - local: '/ci/xenomai_4_next.yml'
 
+xenomai-4/next-long-term:
+  extends: .parent
+  trigger:
+    include:
+      - local: '/ci/xenomai_4_next_long_term.yml'
+  rules:
+    - if: '$CI_PIPELINE_SOURCE == "schedule" && $ONLY_RUN == $CI_JOB_NAME'
+
 xenomai-4/evl:
   extends: .parent
   trigger:
diff --git a/ci/xenomai_4_next_long_term.yml b/ci/xenomai_4_next_long_term.yml
new file mode 100644
index 0000000..906f23b
--- /dev/null
+++ b/ci/xenomai_4_next_long_term.yml
@@ -0,0 +1,24 @@
+#
+# Xenomai Real-Time System
+#
+# Copyright (c) Siemens AG, 2026
+#
+# Authors:
+#  Tobias Schaffner <tobias.schaffner@siemens.com>
+#
+# SPDX-License-Identifier: MIT
+#
+
+include:
+  - local: '/ci/lib/gitlab-ci-base.yml'
+  - local: '/ci/lib/kernel/6_18_physical.yml'
+
+variables:
+  XENOMAI_BUILD_OPTION: ":kas/opt/libevl-next.yaml"
+  XENOMAI_VERSION: "4-next"
+  BUILD_IDENTIFIER: "evl-${XENOMAI_VERSION}_kernel-${KERNEL_VERSION}"
+  CACHE_KERNEL: evl
+  TEST_RUNTIME_SECONDS: 3600
+  TEST_COMMAND: >-
+    evl test --stress-cmd "${EVL_STRESS_COMMAND}" -k $(evl test -l | grep -Ev "${EVL_TESTS_SKIP_PATTERN}") &&
+    evl test --stress-cmd "${EVL_STRESS_COMMAND}" latmus -- -T ${TEST_RUNTIME_SECONDS} -A ${TEST_MAXIMUM_LATENCY}
-- 
2.43.0

Re: [PATCH xenomai-images v2 0/3] longterm EVL testing under load

[Jan Kiszka]

On 28.07.26 16:39, Tobias Schaffner wrote:
> Hi all,
> 
> This series introduces EVL test runs with additional stress applied via
> stress-ng, as well as long-term testing. Along the way, I also updated the
> long-term kernel under test from 6.12 to 6.18.
> 
> Please note:
>  * The clock-timer-periodic test currently fails as soon as stress is
>    applied. I proposed "tests: skip clock-timer-periodic on virtual targets"
>    to address this.
>  * On the Raspberry Pi 4, even the 3-minute test exceeds the latency
>    threshold with debugging disabled:
>    latency threshold is exceeded (150000 >= 163.951), aborting.
>    https://gitlab.com/Xenomai/xenomai-images/-/jobs/15353275174

This second issue is basically a "regression" of adding stress-ng,
right? I thought there were some improvements on the kernel side
Philippe recently talked about to address this? Aren't those merged yet,
at least not to 6.18? Do you see similar issues with latest 7.1?

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: [PATCH xenomai-images v2 0/3] longterm EVL testing under load

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 28.07.26 16:39, Tobias Schaffner wrote:
>> Hi all,
>> 
>> This series introduces EVL test runs with additional stress applied via
>> stress-ng, as well as long-term testing. Along the way, I also updated the
>> long-term kernel under test from 6.12 to 6.18.
>> 
>> Please note:
>>  * The clock-timer-periodic test currently fails as soon as stress is
>>    applied. I proposed "tests: skip clock-timer-periodic on virtual targets"
>>    to address this.
>>  * On the Raspberry Pi 4, even the 3-minute test exceeds the latency
>>    threshold with debugging disabled:
>>    latency threshold is exceeded (150000 >= 163.951), aborting.
>>    https://gitlab.com/Xenomai/xenomai-images/-/jobs/15353275174
>
> This second issue is basically a "regression" of adding stress-ng,
> right? I thought there were some improvements on the kernel side
> Philippe recently talked about to address this? Aren't those merged yet,
> at least not to 6.18? Do you see similar issues with latest 7.1?

I'm still to find cycles to work on this, good news is that I could grab
a spare rpi4 for testing at least.

-- 
Philippe.