Possible Xenomai 3 latency regression on arm64

Possible Xenomai 3 latency regression on arm64

[Jan Kiszka]

Hi all,

it seems we got some >10% worst-case latency increase over the last
month with Xenomai head and 6.12 - that is at least the setup we use in
our long-running weekly test.

Tonight, the RPi4 test triggered by exceeding the configured limit of
150 µs [1]. About half a year ago, we were in 130 µs range with it [2].
x86 and beaglebone are apparently not seeing this change.

Those are preliminary numbers obtained by manually clicking around. Some
visualization of the final worst-case numbers over time would have been
more useful for this. Maybe someone has an idea how to transfer them
into a database or just a plain file from were we could render a gitlab
page or so. A trend analysis which already warns on constant increases
would be the next-but-one logical step.

But for now the question is if this is a real regression or just a shift
in how likely higher latencies became. Or was there some change in our
physical setup, possibly causing more thermal issues or something like
this now?

If it's a real software issue, what is causing this, the 6.12 kernel or
Xenomai 3 head? Anyone willing to dig into some more analysis, probably
first by retrying older runs?

Jan

[1] https://gitlab.com/Xenomai/xenomai-images/-/jobs/14848873648
[2] https://gitlab.com/Xenomai/xenomai-images/-/jobs/12189033381

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: Possible Xenomai 3 latency regression on arm64

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> Hi all,
>
> it seems we got some >10% worst-case latency increase over the last
> month with Xenomai head and 6.12 - that is at least the setup we use in
> our long-running weekly test.
>
> Tonight, the RPi4 test triggered by exceeding the configured limit of
> 150 µs [1]. About half a year ago, we were in 130 µs range with it [2].
> x86 and beaglebone are apparently not seeing this change.
>

My rpi4 fried. Is the issue visible with x4 as well?

-- 
Philippe.

Re: Possible Xenomai 3 latency regression on arm64

[Jan Kiszka]

On 15.06.26 09:35, Philippe Gerum wrote:
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> Hi all,
>>
>> it seems we got some >10% worst-case latency increase over the last
>> month with Xenomai head and 6.12 - that is at least the setup we use in
>> our long-running weekly test.
>>
>> Tonight, the RPi4 test triggered by exceeding the configured limit of
>> 150 µs [1]. About half a year ago, we were in 130 µs range with it [2].
>> x86 and beaglebone are apparently not seeing this change.
>>
> 
> My rpi4 fried. Is the issue visible with x4 as well?
> 

We do not have long-running latency tests for x4 yet, do we, Tobias?

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: Possible Xenomai 3 latency regression on arm64

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 15.06.26 09:35, Philippe Gerum wrote:
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>> 
>>> Hi all,
>>>
>>> it seems we got some >10% worst-case latency increase over the last
>>> month with Xenomai head and 6.12 - that is at least the setup we use in
>>> our long-running weekly test.
>>>
>>> Tonight, the RPi4 test triggered by exceeding the configured limit of
>>> 150 µs [1]. About half a year ago, we were in 130 µs range with it [2].
>>> x86 and beaglebone are apparently not seeing this change.
>>>
>> 
>> My rpi4 fried. Is the issue visible with x4 as well?
>> 
>
> We do not have long-running latency tests for x4 yet, do we, Tobias?
>

Ok, I can try this on an odroid c4. Did anyone succeeded in booting a
mainline kernel on pi5? Last time I checked, root-over-nfs was not
possible yet.

-- 
Philippe.

Re: Possible Xenomai 3 latency regression on arm64

[Tobias Schaffner]

On 6/15/26 10:52, Philippe Gerum wrote:
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> On 15.06.26 09:35, Philippe Gerum wrote:
>>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>>
>>>> Hi all,
>>>>
>>>> it seems we got some >10% worst-case latency increase over the last
>>>> month with Xenomai head and 6.12 - that is at least the setup we use in
>>>> our long-running weekly test.
>>>>
>>>> Tonight, the RPi4 test triggered by exceeding the configured limit of
>>>> 150 µs [1]. About half a year ago, we were in 130 µs range with it [2].
>>>> x86 and beaglebone are apparently not seeing this change.
>>>>
>>>
>>> My rpi4 fried. Is the issue visible with x4 as well?
>>>
>>
>> We do not have long-running latency tests for x4 yet, do we, Tobias?
>>
> 
> Ok, I can try this on an odroid c4. Did anyone succeeded in booting a
> mainline kernel on pi5? Last time I checked, root-over-nfs was not
> possible yet.
> 

No technical input but adding 384434518@qq.com who also asked for this
SOC about a week ago.

Cite:
> For your reference, mainline Linux has added support for the BCM2712 
> SoC starting from version 6.12, and later versions such as 6.18, 
> 6.19, and 7.0 continue to support it.
> 
> Is the BCM2712 / Raspberry Pi 5 currently supported by Xenomai 3
> over the Cobalt core (including the required Dovetail interrupt
> pipeline support)? If not, is there any plan to add support in the
> near future?

Re: Possible Xenomai 3 latency regression on arm64

[Tobias Schaffner]

Hi Jan,

On 6/15/26 10:33, Jan Kiszka wrote:
> On 15.06.26 09:35, Philippe Gerum wrote:
>> Jan Kiszka <jan.kiszka@siemens.com> writes:
>>
>>> Hi all,
>>>
>>> it seems we got some >10% worst-case latency increase over the last
>>> month with Xenomai head and 6.12 - that is at least the setup we use in
>>> our long-running weekly test.
>>>
>>> Tonight, the RPi4 test triggered by exceeding the configured limit of
>>> 150 µs [1]. About half a year ago, we were in 130 µs range with it [2].
>>> x86 and beaglebone are apparently not seeing this change.
>>>
>>
>> My rpi4 fried. Is the issue visible with x4 as well?
>>
> 
> We do not have long-running latency tests for x4 yet, do we, Tobias?

no automated tests in the CI yet, but I already discussed this with
Philippe.

Plan is to move latmus into 'evl test' as already done with hectic.
Combined with the backround stress changes this would allow similar
latency tests as in x3.

I will try to jump back to this topic as soon as I finished polishing of
the riscv patches.

Best,
Tobias

> Jan
>