All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH xenomai-images 0/3] longterm EVL testing under load
@ 2026-07-15  9:28 Tobias Schaffner
  2026-07-15  9:28 ` [PATCH xenomai-images 1/3] ci: use stress-ng to stress EVL tests Tobias Schaffner
                   ` (5 more replies)
  0 siblings, 6 replies; 9+ messages in thread
From: Tobias Schaffner @ 2026-07-15  9:28 UTC (permalink / raw)
  To: xenomai; +Cc: rpm, jan.kiszka, Tobias Schaffner

Hi all,

This series introduces EVL test runs with additional stress applied via
stress-ng, as well as long-term testing. Along the way, I also updated the
long-term kernel under test from 6.12 to 6.18.

Please note:
 * The clock-timer-periodic test currently fails as soon as stress is
   applied. I proposed "tests: skip clock-timer-periodic on virtual targets"
   to address this.
 * On the Raspberry Pi 4, even the 3-minute test exceeds the latency
   threshold with debugging disabled:
   latency threshold is exceeded (150000 >= 163.951), aborting.
   https://gitlab.com/Xenomai/xenomai-images/-/jobs/15353275174

Best,
Tobias

Tobias Schaffner (3):
  ci: use stress-ng to stress EVL tests
  ci: Update for longterm test runs from 6.12 to 6.18
  ci: add long-term EVL CI test pipeline

 .gitlab-ci.yml                  |  8 +++++
 ci/lib/gitlab-ci-base.yml       |  2 ++
 ci/lib/kernel/6_12.yml          | 45 ++++++++++++++++++++++++++--
 ci/lib/kernel/6_12_physical.yml | 52 ---------------------------------
 ci/lib/kernel/6_18.yml          | 42 ++------------------------
 ci/lib/kernel/6_18_physical.yml | 52 +++++++++++++++++++++++++++++++++
 ci/xenomai_3_next_long_term.yml |  4 +--
 ci/xenomai_4_evl.yml            |  2 +-
 ci/xenomai_4_next.yml           |  2 +-
 ci/xenomai_4_next_long_term.yml | 24 +++++++++++++++
 kas/opt/ci.yaml                 |  1 +
 11 files changed, 136 insertions(+), 98 deletions(-)
 delete mode 100644 ci/lib/kernel/6_12_physical.yml
 create mode 100644 ci/lib/kernel/6_18_physical.yml
 create mode 100644 ci/xenomai_4_next_long_term.yml

-- 
2.43.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH xenomai-images 1/3] ci: use stress-ng to stress EVL tests
  2026-07-15  9:28 [PATCH xenomai-images 0/3] longterm EVL testing under load Tobias Schaffner
@ 2026-07-15  9:28 ` Tobias Schaffner
  2026-07-15 13:00   ` Florian Bezdeka
  2026-07-15  9:28 ` [PATCH xenomai-images 2/3] ci: Update for longterm test runs from 6.12 to 6.18 Tobias Schaffner
                   ` (4 subsequent siblings)
  5 siblings, 1 reply; 9+ messages in thread
From: Tobias Schaffner @ 2026-07-15  9:28 UTC (permalink / raw)
  To: xenomai; +Cc: rpm, jan.kiszka, Tobias Schaffner

Use stress-ng to apply CPU, virtual memory, FPU register, and procfs load
during EVL tests, mimicking the stressors used in Xenomai 3.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 ci/lib/gitlab-ci-base.yml | 2 ++
 ci/xenomai_4_evl.yml      | 2 +-
 ci/xenomai_4_next.yml     | 2 +-
 kas/opt/ci.yaml           | 1 +
 4 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/ci/lib/gitlab-ci-base.yml b/ci/lib/gitlab-ci-base.yml
index 0540628..a3359bb 100644
--- a/ci/lib/gitlab-ci-base.yml
+++ b/ci/lib/gitlab-ci-base.yml
@@ -25,6 +25,8 @@ variables:
   ADDITIONAL_TEST_TIMEOUT_MINUTES: 10
   TEST_MAXIMUM_LATENCY: 0
   TEST_COMMAND: "xeno-test -k -l 'dohell -m /tmp ${TEST_RUNTIME_SECONDS}' ${ADDITIONAL_XENO_TEST_ARGS}"
+  EVL_STRESS_COMMAND: stress-ng --cpu 1 --vm 1 --vm-bytes 64M --fp 1 --procfs 1
+  EVL_TEST_COMMAND: evl test --stress-cmd "${EVL_STRESS_COMMAND}" -k $(evl test -l | grep -Ev "${EVL_TESTS_SKIP_PATTERN}")
   SSTATE_LOCATION: "s3://xenomai-images-artifacts/sstate"
   SSTATE_MIRRORS: "file://.* ${SSTATE_LOCATION}/PATH;downloadfilename=PATH"
   SSTATE_MAX_AGE: 14d
diff --git a/ci/xenomai_4_evl.yml b/ci/xenomai_4_evl.yml
index 283a1fd..c6a92b8 100644
--- a/ci/xenomai_4_evl.yml
+++ b/ci/xenomai_4_evl.yml
@@ -29,4 +29,4 @@ variables:
   XENOMAI_VERSION: "4-next"
   BUILD_IDENTIFIER: "evl-${XENOMAI_VERSION}_kernel-${KERNEL_VERSION}"
   CACHE_KERNEL: evl
-  TEST_COMMAND: "evl test -k $(evl test -l | grep -Ev \"${EVL_TESTS_SKIP_PATTERN}\")"
+  TEST_COMMAND: "${EVL_TEST_COMMAND}"
diff --git a/ci/xenomai_4_next.yml b/ci/xenomai_4_next.yml
index 6ef4b6a..7ecfbb5 100644
--- a/ci/xenomai_4_next.yml
+++ b/ci/xenomai_4_next.yml
@@ -21,4 +21,4 @@ variables:
   KERNEL_VERSION_WITH_QEMU: "head"
   BUILD_IDENTIFIER: "evl-${XENOMAI_VERSION}_kernel-${KERNEL_VERSION}"
   CACHE_KERNEL: evl
-  TEST_COMMAND: "evl test -k $(evl test -l | grep -Ev \"${EVL_TESTS_SKIP_PATTERN}\")"
+  TEST_COMMAND: "${EVL_TEST_COMMAND}"
diff --git a/kas/opt/ci.yaml b/kas/opt/ci.yaml
index e1466a0..f9e4bb7 100644
--- a/kas/opt/ci.yaml
+++ b/kas/opt/ci.yaml
@@ -20,6 +20,7 @@ local_conf_header:
   xenomai-ci: |
     ROOTFS_POSTPROCESS_COMMAND:remove = "image_postprocess_sshd_key_regen"
     IMAGE_INSTALL += "customizations-ci"
+    IMAGE_PREINSTALL += "stress-ng"
     IMAGE_INSTALL:remove = "sshd-regen-keys"
     IMAGE_INSTALL:remove = "expand-on-first-boot"
     IMAGE_FSTYPES = "${@ 'ext4.gz' if d.getVar('MACHINE', True).startswith('qemu') else 'tar.gz' }"
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH xenomai-images 2/3] ci: Update for longterm test runs from 6.12 to 6.18
  2026-07-15  9:28 [PATCH xenomai-images 0/3] longterm EVL testing under load Tobias Schaffner
  2026-07-15  9:28 ` [PATCH xenomai-images 1/3] ci: use stress-ng to stress EVL tests Tobias Schaffner
@ 2026-07-15  9:28 ` Tobias Schaffner
  2026-07-15  9:28 ` [PATCH xenomai-images 3/3] ci: add long-term EVL CI test pipeline Tobias Schaffner
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 9+ messages in thread
From: Tobias Schaffner @ 2026-07-15  9:28 UTC (permalink / raw)
  To: xenomai; +Cc: rpm, jan.kiszka, Tobias Schaffner

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 ci/lib/kernel/6_12.yml          | 45 ++++++++++++++++++++++++++--
 ci/lib/kernel/6_12_physical.yml | 52 ---------------------------------
 ci/lib/kernel/6_18.yml          | 42 ++------------------------
 ci/lib/kernel/6_18_physical.yml | 52 +++++++++++++++++++++++++++++++++
 ci/xenomai_3_next_long_term.yml |  4 +--
 5 files changed, 99 insertions(+), 96 deletions(-)
 delete mode 100644 ci/lib/kernel/6_12_physical.yml
 create mode 100644 ci/lib/kernel/6_18_physical.yml

diff --git a/ci/lib/kernel/6_12.yml b/ci/lib/kernel/6_12.yml
index a9bed87..529d24d 100644
--- a/ci/lib/kernel/6_12.yml
+++ b/ci/lib/kernel/6_12.yml
@@ -9,9 +9,6 @@
 # SPDX-License-Identifier: MIT
 #
 
-include:
-  - local: '/ci/lib/kernel/6_12_physical.yml'
-
 build-6.12:qemu-amd64:
   extends: .build:qemu-amd64
   variables:
@@ -50,3 +47,45 @@ lava-test-6.12:qemu-arm64:
   extends: .lava-test:qemu-arm64
   variables:
     KERNEL_VERSION: "6.12"
+
+build-6.12:rpi4:
+  extends: .build:rpi4
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
+    KERNEL_VERSION: "6.12"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.12:rpi4:
+  needs: [ "build-6.12:rpi4" ]
+  extends: .lava-test:rpi4
+  variables:
+    KERNEL_VERSION: "6.12"
+    TEST_MAXIMUM_LATENCY: 150
+
+build-6.12:beagle-bone-black:
+  extends: .build:beagle-bone-black
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
+    KERNEL_VERSION: "6.12"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.12:beagle-bone-black:
+  needs: [ "build-6.12:beagle-bone-black" ]
+  extends: .lava-test:beagle-bone-black
+  variables:
+    KERNEL_VERSION: "6.12"
+    TEST_MAXIMUM_LATENCY: 150
+
+build-6.12:x86-64-efi:
+  extends: .build:x86-64-efi
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
+    KERNEL_VERSION: "6.12"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.12:x86-64-efi:
+  needs: [ "build-6.12:x86-64-efi" ]
+  extends: .lava-test:x86-64-efi
+  variables:
+    KERNEL_VERSION: "6.12"
+    TEST_MAXIMUM_LATENCY: 100
diff --git a/ci/lib/kernel/6_12_physical.yml b/ci/lib/kernel/6_12_physical.yml
deleted file mode 100644
index 86cd23c..0000000
--- a/ci/lib/kernel/6_12_physical.yml
+++ /dev/null
@@ -1,52 +0,0 @@
-#
-# Xenomai Real-Time System
-#
-# Copyright (c) Siemens AG, 2024-2025
-#
-# Authors:
-#  Jan Kiszka <jan.kiszka@siemens.com>
-#
-# SPDX-License-Identifier: MIT
-#
-
-build-6.12:rpi4:
-  extends: .build:rpi4
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
-    KERNEL_VERSION: "6.12"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.12:rpi4:
-  needs: [ "build-6.12:rpi4" ]
-  extends: .lava-test:rpi4
-  variables:
-    KERNEL_VERSION: "6.12"
-    TEST_MAXIMUM_LATENCY: 150
-
-build-6.12:beagle-bone-black:
-  extends: .build:beagle-bone-black
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
-    KERNEL_VERSION: "6.12"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.12:beagle-bone-black:
-  needs: [ "build-6.12:beagle-bone-black" ]
-  extends: .lava-test:beagle-bone-black
-  variables:
-    KERNEL_VERSION: "6.12"
-    TEST_MAXIMUM_LATENCY: 150
-
-build-6.12:x86-64-efi:
-  extends: .build:x86-64-efi
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.12-head.yaml"
-    KERNEL_VERSION: "6.12"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.12:x86-64-efi:
-  needs: [ "build-6.12:x86-64-efi" ]
-  extends: .lava-test:x86-64-efi
-  variables:
-    KERNEL_VERSION: "6.12"
-    TEST_MAXIMUM_LATENCY: 100
diff --git a/ci/lib/kernel/6_18.yml b/ci/lib/kernel/6_18.yml
index 88e3915..5d7e409 100644
--- a/ci/lib/kernel/6_18.yml
+++ b/ci/lib/kernel/6_18.yml
@@ -9,6 +9,9 @@
 # SPDX-License-Identifier: MIT
 #
 
+include:
+  - local: '/ci/lib/kernel/6_18_physical.yml'
+
 build-6.18:qemu-amd64:
   extends: .build:qemu-amd64
   variables:
@@ -47,42 +50,3 @@ lava-test-6.18:qemu-arm64:
   extends: .lava-test:qemu-arm64
   variables:
     KERNEL_VERSION: "6.18"
-
-build-6.18:rpi4:
-  extends: .build:rpi4
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
-    KERNEL_VERSION: "6.18"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.18:rpi4:
-  needs: [ "build-6.18:rpi4" ]
-  extends: .lava-test:rpi4
-  variables:
-    KERNEL_VERSION: "6.18"
-
-build-6.18:beagle-bone-black:
-  extends: .build:beagle-bone-black
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
-    KERNEL_VERSION: "6.18"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.18:beagle-bone-black:
-  needs: [ "build-6.18:beagle-bone-black" ]
-  extends: .lava-test:beagle-bone-black
-  variables:
-    KERNEL_VERSION: "6.18"
-
-build-6.18:x86-64-efi:
-  extends: .build:x86-64-efi
-  variables:
-    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
-    KERNEL_VERSION: "6.18"
-    CACHE_KERNEL: dovetail
-
-lava-test-6.18:x86-64-efi:
-  needs: [ "build-6.18:x86-64-efi" ]
-  extends: .lava-test:x86-64-efi
-  variables:
-    KERNEL_VERSION: "6.18"
diff --git a/ci/lib/kernel/6_18_physical.yml b/ci/lib/kernel/6_18_physical.yml
new file mode 100644
index 0000000..68b7892
--- /dev/null
+++ b/ci/lib/kernel/6_18_physical.yml
@@ -0,0 +1,52 @@
+#
+# Xenomai Real-Time System
+#
+# Copyright (c) Siemens AG, 2024-2026
+#
+# Authors:
+#  Jan Kiszka <jan.kiszka@siemens.com>
+#
+# SPDX-License-Identifier: MIT
+#
+
+build-6.18:rpi4:
+  extends: .build:rpi4
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
+    KERNEL_VERSION: "6.18"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.18:rpi4:
+  needs: [ "build-6.18:rpi4" ]
+  extends: .lava-test:rpi4
+  variables:
+    KERNEL_VERSION: "6.18"
+    TEST_MAXIMUM_LATENCY: 150
+
+build-6.18:beagle-bone-black:
+  extends: .build:beagle-bone-black
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
+    KERNEL_VERSION: "6.18"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.18:beagle-bone-black:
+  needs: [ "build-6.18:beagle-bone-black" ]
+  extends: .lava-test:beagle-bone-black
+  variables:
+    KERNEL_VERSION: "6.18"
+    TEST_MAXIMUM_LATENCY: 150
+
+build-6.18:x86-64-efi:
+  extends: .build:x86-64-efi
+  variables:
+    LINUX_BUILD_OPTION: ":kas/opt/linux-6.18-head.yaml"
+    KERNEL_VERSION: "6.18"
+    CACHE_KERNEL: dovetail
+
+lava-test-6.18:x86-64-efi:
+  needs: [ "build-6.18:x86-64-efi" ]
+  extends: .lava-test:x86-64-efi
+  variables:
+    KERNEL_VERSION: "6.18"
+    TEST_MAXIMUM_LATENCY: 100
diff --git a/ci/xenomai_3_next_long_term.yml b/ci/xenomai_3_next_long_term.yml
index da8749e..c22374c 100644
--- a/ci/xenomai_3_next_long_term.yml
+++ b/ci/xenomai_3_next_long_term.yml
@@ -11,11 +11,11 @@
 
 include:
   - local: '/ci/lib/gitlab-ci-base.yml'
-  - local: '/ci/lib/kernel/6_12_physical.yml'
+  - local: '/ci/lib/kernel/6_18_physical.yml'
 
 variables:
   XENOMAI_BUILD_OPTION: ":kas/opt/xenomai-next.yaml"
   XENOMAI_VERSION: "3-next"
-  KERNEL_VERSION_WITH_QEMU: "6.12"
+  KERNEL_VERSION_WITH_QEMU: "6.18"
   TEST_RUNTIME_SECONDS: 3600
   TEST_COMMAND: "xeno-test -k -l 'dohell -m /tmp ${TEST_RUNTIME_SECONDS}' ${ADDITIONAL_XENO_TEST_ARGS} -- -m ${TEST_MAXIMUM_LATENCY}"
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH xenomai-images 3/3] ci: add long-term EVL CI test pipeline
  2026-07-15  9:28 [PATCH xenomai-images 0/3] longterm EVL testing under load Tobias Schaffner
  2026-07-15  9:28 ` [PATCH xenomai-images 1/3] ci: use stress-ng to stress EVL tests Tobias Schaffner
  2026-07-15  9:28 ` [PATCH xenomai-images 2/3] ci: Update for longterm test runs from 6.12 to 6.18 Tobias Schaffner
@ 2026-07-15  9:28 ` Tobias Schaffner
  2026-07-15 10:19 ` [PATCH xenomai-images 0/3] longterm EVL testing under load Jan Kiszka
                   ` (2 subsequent siblings)
  5 siblings, 0 replies; 9+ messages in thread
From: Tobias Schaffner @ 2026-07-15  9:28 UTC (permalink / raw)
  To: xenomai; +Cc: rpm, jan.kiszka, Tobias Schaffner

Add a scheduled Xenomai 4-next long-term CI job with a dedicated
pipeline configuration running extended EVL stress and latency tests.

Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
---
 .gitlab-ci.yml                  |  8 ++++++++
 ci/xenomai_4_next_long_term.yml | 24 ++++++++++++++++++++++++
 2 files changed, 32 insertions(+)
 create mode 100644 ci/xenomai_4_next_long_term.yml

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index b3d025e..589db8e 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -91,6 +91,14 @@ xenomai-4/next:
     include:
       - local: '/ci/xenomai_4_next.yml'
 
+xenomai-4/next-long-term:
+  extends: .parent
+  trigger:
+    include:
+      - local: '/ci/xenomai_4_next_long_term.yml'
+  rules:
+    - if: '$CI_PIPELINE_SOURCE == "schedule" && $ONLY_RUN == $CI_JOB_NAME'
+
 xenomai-4/evl:
   extends: .parent
   trigger:
diff --git a/ci/xenomai_4_next_long_term.yml b/ci/xenomai_4_next_long_term.yml
new file mode 100644
index 0000000..906f23b
--- /dev/null
+++ b/ci/xenomai_4_next_long_term.yml
@@ -0,0 +1,24 @@
+#
+# Xenomai Real-Time System
+#
+# Copyright (c) Siemens AG, 2026
+#
+# Authors:
+#  Tobias Schaffner <tobias.schaffner@siemens.com>
+#
+# SPDX-License-Identifier: MIT
+#
+
+include:
+  - local: '/ci/lib/gitlab-ci-base.yml'
+  - local: '/ci/lib/kernel/6_18_physical.yml'
+
+variables:
+  XENOMAI_BUILD_OPTION: ":kas/opt/libevl-next.yaml"
+  XENOMAI_VERSION: "4-next"
+  BUILD_IDENTIFIER: "evl-${XENOMAI_VERSION}_kernel-${KERNEL_VERSION}"
+  CACHE_KERNEL: evl
+  TEST_RUNTIME_SECONDS: 3600
+  TEST_COMMAND: >-
+    evl test --stress-cmd "${EVL_STRESS_COMMAND}" -k $(evl test -l | grep -Ev "${EVL_TESTS_SKIP_PATTERN}") &&
+    evl test --stress-cmd "${EVL_STRESS_COMMAND}" latmus -- -T ${TEST_RUNTIME_SECONDS} -A ${TEST_MAXIMUM_LATENCY}
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH xenomai-images 0/3] longterm EVL testing under load
  2026-07-15  9:28 [PATCH xenomai-images 0/3] longterm EVL testing under load Tobias Schaffner
                   ` (2 preceding siblings ...)
  2026-07-15  9:28 ` [PATCH xenomai-images 3/3] ci: add long-term EVL CI test pipeline Tobias Schaffner
@ 2026-07-15 10:19 ` Jan Kiszka
  2026-07-15 10:46   ` Tobias Schaffner
  2026-07-15 10:23 ` Philippe Gerum
  2026-07-15 12:38 ` Philippe Gerum
  5 siblings, 1 reply; 9+ messages in thread
From: Jan Kiszka @ 2026-07-15 10:19 UTC (permalink / raw)
  To: Tobias Schaffner, xenomai; +Cc: rpm

On 15.07.26 11:28, Tobias Schaffner wrote:
> Hi all,
> 
> This series introduces EVL test runs with additional stress applied via
> stress-ng, as well as long-term testing. Along the way, I also updated the
> long-term kernel under test from 6.12 to 6.18.
> 
> Please note:
>  * The clock-timer-periodic test currently fails as soon as stress is
>    applied. I proposed "tests: skip clock-timer-periodic on virtual targets"
>    to address this.
>  * On the Raspberry Pi 4, even the 3-minute test exceeds the latency
>    threshold with debugging disabled:
>    latency threshold is exceeded (150000 >= 163.951), aborting.
>    https://gitlab.com/Xenomai/xenomai-images/-/jobs/15353275174

That is 150 ms?! Sounds like a real bug - unless we enforce some thermal
throttling on the hardware. But we haven't seen that with xenomai 3 so far.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH xenomai-images 0/3] longterm EVL testing under load
  2026-07-15  9:28 [PATCH xenomai-images 0/3] longterm EVL testing under load Tobias Schaffner
                   ` (3 preceding siblings ...)
  2026-07-15 10:19 ` [PATCH xenomai-images 0/3] longterm EVL testing under load Jan Kiszka
@ 2026-07-15 10:23 ` Philippe Gerum
  2026-07-15 12:38 ` Philippe Gerum
  5 siblings, 0 replies; 9+ messages in thread
From: Philippe Gerum @ 2026-07-15 10:23 UTC (permalink / raw)
  To: Tobias Schaffner; +Cc: xenomai, jan.kiszka

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

> Hi all,
>
> This series introduces EVL test runs with additional stress applied via
> stress-ng, as well as long-term testing. Along the way, I also updated the
> long-term kernel under test from 6.12 to 6.18.
>
> Please note:
>  * The clock-timer-periodic test currently fails as soon as stress is
>    applied. I proposed "tests: skip clock-timer-periodic on virtual targets"
>    to address this.
>  * On the Raspberry Pi 4, even the 3-minute test exceeds the latency
>    threshold with debugging disabled:
>    latency threshold is exceeded (150000 >= 163.951), aborting.
>    https://gitlab.com/Xenomai/xenomai-images/-/jobs/15353275174

We should not exceed 80 µs or so on pi4.

-- 
Philippe.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH xenomai-images 0/3] longterm EVL testing under load
  2026-07-15 10:19 ` [PATCH xenomai-images 0/3] longterm EVL testing under load Jan Kiszka
@ 2026-07-15 10:46   ` Tobias Schaffner
  0 siblings, 0 replies; 9+ messages in thread
From: Tobias Schaffner @ 2026-07-15 10:46 UTC (permalink / raw)
  To: Jan Kiszka, xenomai; +Cc: rpm



On 7/15/26 12:19, Jan Kiszka wrote:
> On 15.07.26 11:28, Tobias Schaffner wrote:
>> Hi all,
>>
>> This series introduces EVL test runs with additional stress applied via
>> stress-ng, as well as long-term testing. Along the way, I also updated the
>> long-term kernel under test from 6.12 to 6.18.
>>
>> Please note:
>>   * The clock-timer-periodic test currently fails as soon as stress is
>>     applied. I proposed "tests: skip clock-timer-periodic on virtual targets"
>>     to address this.
>>   * On the Raspberry Pi 4, even the 3-minute test exceeds the latency
>>     threshold with debugging disabled:
>>     latency threshold is exceeded (150000 >= 163.951), aborting.
>>     https://gitlab.com/Xenomai/xenomai-images/-/jobs/15353275174
> 
> That is 150 ms?! Sounds like a real bug - unless we enforce some thermal
> throttling on the hardware. But we haven't seen that with xenomai 3 so far.

Also looks bad to me.

We seem to be a bit worse than the Xenomai 3 results. But the results 
are not great on Xenomai 3 either.

Xenomai3 Kernel 6.18:
https://gitlab.com/Xenomai/xenomai-images/-/jobs/15351736635
WCL: 123.148

Xenomai3 Kernel 6.12 (last weekly running long term run):
https://gitlab.com/Xenomai/xenomai-images/-/jobs/15308563317
WCL 137.195

The oldest Pipeline I was able to find is 11 Month old:
https://gitlab.com/Xenomai/xenomai-images/-/jobs/11042799343
WCL: 124.430

Best,
Tobias

> Jan
> 


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH xenomai-images 0/3] longterm EVL testing under load
  2026-07-15  9:28 [PATCH xenomai-images 0/3] longterm EVL testing under load Tobias Schaffner
                   ` (4 preceding siblings ...)
  2026-07-15 10:23 ` Philippe Gerum
@ 2026-07-15 12:38 ` Philippe Gerum
  5 siblings, 0 replies; 9+ messages in thread
From: Philippe Gerum @ 2026-07-15 12:38 UTC (permalink / raw)
  To: Tobias Schaffner; +Cc: xenomai, jan.kiszka

Tobias Schaffner <tobias.schaffner@siemens.com> writes:

>    latency threshold is exceeded (150000 >= 163.951), aborting.

Even the printout looks wrong.. Fix pushed upstream.

-- 
Philippe.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH xenomai-images 1/3] ci: use stress-ng to stress EVL tests
  2026-07-15  9:28 ` [PATCH xenomai-images 1/3] ci: use stress-ng to stress EVL tests Tobias Schaffner
@ 2026-07-15 13:00   ` Florian Bezdeka
  0 siblings, 0 replies; 9+ messages in thread
From: Florian Bezdeka @ 2026-07-15 13:00 UTC (permalink / raw)
  To: Tobias Schaffner, xenomai; +Cc: rpm, jan.kiszka

On Wed, 2026-07-15 at 11:28 +0200, Tobias Schaffner wrote:
> Use stress-ng to apply CPU, virtual memory, FPU register, and procfs load
> during EVL tests, mimicking the stressors used in Xenomai 3.
> 
> Signed-off-by: Tobias Schaffner <tobias.schaffner@siemens.com>
> ---
>  ci/lib/gitlab-ci-base.yml | 2 ++
>  ci/xenomai_4_evl.yml      | 2 +-
>  ci/xenomai_4_next.yml     | 2 +-
>  kas/opt/ci.yaml           | 1 +
>  4 files changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/ci/lib/gitlab-ci-base.yml b/ci/lib/gitlab-ci-base.yml
> index 0540628..a3359bb 100644
> --- a/ci/lib/gitlab-ci-base.yml
> +++ b/ci/lib/gitlab-ci-base.yml
> @@ -25,6 +25,8 @@ variables:
>    ADDITIONAL_TEST_TIMEOUT_MINUTES: 10
>    TEST_MAXIMUM_LATENCY: 0
>    TEST_COMMAND: "xeno-test -k -l 'dohell -m /tmp ${TEST_RUNTIME_SECONDS}' ${ADDITIONAL_XENO_TEST_ARGS}"
> +  EVL_STRESS_COMMAND: stress-ng --cpu 1 --vm 1 --vm-bytes 64M --fp 1 --procfs 1
> +  EVL_TEST_COMMAND: evl test --stress-cmd "${EVL_STRESS_COMMAND}" -k $(evl test -l | grep -Ev "${EVL_TESTS_SKIP_PATTERN}")
>    SSTATE_LOCATION: "s3://xenomai-images-artifacts/sstate"
>    SSTATE_MIRRORS: "file://.* ${SSTATE_LOCATION}/PATH;downloadfilename=PATH"
>    SSTATE_MAX_AGE: 14d
> diff --git a/ci/xenomai_4_evl.yml b/ci/xenomai_4_evl.yml
> index 283a1fd..c6a92b8 100644
> --- a/ci/xenomai_4_evl.yml
> +++ b/ci/xenomai_4_evl.yml
> @@ -29,4 +29,4 @@ variables:
>    XENOMAI_VERSION: "4-next"
>    BUILD_IDENTIFIER: "evl-${XENOMAI_VERSION}_kernel-${KERNEL_VERSION}"
>    CACHE_KERNEL: evl
> -  TEST_COMMAND: "evl test -k $(evl test -l | grep -Ev \"${EVL_TESTS_SKIP_PATTERN}\")"
> +  TEST_COMMAND: "${EVL_TEST_COMMAND}"
> diff --git a/ci/xenomai_4_next.yml b/ci/xenomai_4_next.yml
> index 6ef4b6a..7ecfbb5 100644
> --- a/ci/xenomai_4_next.yml
> +++ b/ci/xenomai_4_next.yml
> @@ -21,4 +21,4 @@ variables:
>    KERNEL_VERSION_WITH_QEMU: "head"
>    BUILD_IDENTIFIER: "evl-${XENOMAI_VERSION}_kernel-${KERNEL_VERSION}"
>    CACHE_KERNEL: evl
> -  TEST_COMMAND: "evl test -k $(evl test -l | grep -Ev \"${EVL_TESTS_SKIP_PATTERN}\")"
> +  TEST_COMMAND: "${EVL_TEST_COMMAND}"
> diff --git a/kas/opt/ci.yaml b/kas/opt/ci.yaml
> index e1466a0..f9e4bb7 100644
> --- a/kas/opt/ci.yaml
> +++ b/kas/opt/ci.yaml
> @@ -20,6 +20,7 @@ local_conf_header:
>    xenomai-ci: |
>      ROOTFS_POSTPROCESS_COMMAND:remove = "image_postprocess_sshd_key_regen"
>      IMAGE_INSTALL += "customizations-ci"
> +    IMAGE_PREINSTALL += "stress-ng"

Hm... As the CI configuration is now requesting stress-ng it seems right
that it is pre-installed by the CI image.

OTOH I hate those loose dependencies. Maybe the (lib)evl test package
should pull it in instead? So that the dependency is handled at package
level. Someone else might like to install the test package and get all
dependencies accordingly.

To solve that properly it might make sense to make stress-ng the default
stressor. That would allow the packaging to properly depend on it,
instead of only "suggest" stress-ng.

>      IMAGE_INSTALL:remove = "sshd-regen-keys"
>      IMAGE_INSTALL:remove = "expand-on-first-boot"
>      IMAGE_FSTYPES = "${@ 'ext4.gz' if d.getVar('MACHINE', True).startswith('qemu') else 'tar.gz' }"
> -- 
> 2.43.0

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2026-07-15 13:01 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-15  9:28 [PATCH xenomai-images 0/3] longterm EVL testing under load Tobias Schaffner
2026-07-15  9:28 ` [PATCH xenomai-images 1/3] ci: use stress-ng to stress EVL tests Tobias Schaffner
2026-07-15 13:00   ` Florian Bezdeka
2026-07-15  9:28 ` [PATCH xenomai-images 2/3] ci: Update for longterm test runs from 6.12 to 6.18 Tobias Schaffner
2026-07-15  9:28 ` [PATCH xenomai-images 3/3] ci: add long-term EVL CI test pipeline Tobias Schaffner
2026-07-15 10:19 ` [PATCH xenomai-images 0/3] longterm EVL testing under load Jan Kiszka
2026-07-15 10:46   ` Tobias Schaffner
2026-07-15 10:23 ` Philippe Gerum
2026-07-15 12:38 ` Philippe Gerum

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.