Re: The state of Xenomai 4 on Xilinx Zynq-7000 (or ARM32 in general)

[Philippe Gerum <rpm@xenomai.org>]

Gerte Hoogewerf <ghoogewerf@lmi3d.com> writes:

> Hi Xenomai team,
>
> At LMI, we are in the machine vision industry, where some applications
> require predictable and bounded ISR latency.   Although we are
> gradually migrating our installations to Xilinx Zynq UltraScale+, most
> field installations still run on the older Xilinx Zynq-7000 SoC.
>
> We've been running Xenomai on our customized Zynq-7000 boards since
> 2021 or so.  Our products have been shipping with Xenomai 3, ipipe,
> kernel 4.14 all this time.  Although it still runs and performs well,
> the age of the stack is starting to affect security certification.
>
> On multiple occasions, I've tried newer versions of Xenomai.   The
> most recent version that still functions properly was kernel 5.4 (i.e.
> the last version to support ipipe + Xenomai 3).  I was never able to
> successfully boot anything newer than that version.
>
> It typically hangs partway through boot, like this:
>
> [    0.000000] Booting Linux on physical CPU 0x0
> [    0.000000] Linux version 6.18.29-01121-g5d9e6ccea5cc
> (ghoogewerf@NLKRLTF0FQR04) (arm-none-linux-gnueabihf-gcc (LMI3D G++
> 11.4.1-p1) 11.4.1 20240321, GNU ld (LMI3D G++ 11.4.1-p1) 2.40) #2 SMP
> PREEMPT EVL Mon Jul 13 12:06:58 CEST 2026
> [    0.000000] CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=18c5387d
> [.....]
> [    0.815404] io scheduler kyber registered
> [    0.820358] zynq-pinctrl 700.pinctrl: zynq pinctrl initialized
> [    0.831589] dma-pl330 f8003000.dma-controller: Loaded driver for
> PL330 DMAC-241330
> [    0.839180] dma-pl330 f8003000.dma-controller:
> DBUFF-128x8bytes Num_Chans-8 Num_Peri-4 Num_Events-16
> [    0.857446] brd: module loaded
> [    0.865454] loop: module loaded
> [    1.260102] Freeing initrd memory: 5620K
> <======    HANG    ======>
>
> The peculiar thing is that either "CONFIG_EVL=n" or "CONFIG_SMP=n"
> works around this problem.  But that's not a good solution for obvious
> reasons.
>
> I am curious where Xenomai 4 stands regarding support for Zynq-7000
> (or ARM32 in general).  Are you aware of problems with these older
> boards?  Is there anything we can do ourselves to debug this further?
> We are happy to share the full boot log, kernel config, and device
> tree if that would help.  Any guidance or support you can offer would
> be appreciated.
>

To put it simply, every architecture Xenomai4 supports is considered
"tier 1", i.e. is actively maintained and must work
flawlessly. Therefore, although arm32 is aging, it is still fully
supported, not to speak of the fact that we do have quite a few projects
depending on it. However, we lost access to many platforms some time
ago, including Zynq hardware, so we can't test them routinely anymore.

Regarding the issue you are observing, it suspect this is
Dovetail-related. If so, xenomai 3.3 which is using Dovetail won't boot
on the Zynq either.

Such hang at boot is often the symptom of an issue with the proxy timer
Dovetail provides and xenomai relies on, i.e. the piece of code which
hands control over the system hardware timer to the real-time core.

To start debugging this, I would run the basic tests Dovetail provides
which cover the proxy tick management by configuring as follows:

CONFIG_DEBUG_IRQ_PIPELINE=y
CONFIG_IRQ_PIPELINE_TORTURE_TEST=y
CONFIG_DEBUG_DOVETAIL=y
CONFIG_EVL=n
CONFIG_SMP=y

These tests are executed during the boot sequence, a successful run
should look like this:

[    0.654959] Starting IRQ pipeline tests...
[    0.654981] IRQ pipeline: high-priority torture stage added.
[    0.654989] irq_pipeline-torture: CPU0 initiates stop_machine()
[    0.659603] irq_pipeline-torture: CPU3 responds to stop_machine()
[    0.659603] irq_pipeline-torture: CPU2 responds to stop_machine()
[    0.659603] irq_pipeline-torture: CPU1 responds to stop_machine()
[    0.659649] irq_pipeline-torture: CPU0 calls function on remote(s)
[    0.659666] irq_pipeline-torture: CPU1 handles remote function call
[    0.659677] irq_pipeline-torture: CPU2 handles remote function call
[    0.659689] irq_pipeline-torture: CPU3 handles remote function call
[    0.659856] CPU0: proxy tick device registered (396.00MHz)
[    0.659861] CPU2: proxy tick device registered (396.00MHz)
[    0.659867] CPU1: proxy tick device registered (396.00MHz)
[    0.666350] CPU3: proxy tick device registered (396.00MHz)
[    0.666440] irq_pipeline-torture: CPU0: irq_work handled
[    0.666451] irq_pipeline-torture: CPU0: in-band->in-band irq_work trigger works
[    0.666457] irq_pipeline-torture: CPU0: stage escalation request works
[    0.666465] irq_pipeline-torture: CPU0: irq_work handled
[    0.666475] irq_pipeline-torture: CPU0: oob->in-band irq_work trigger works
[    1.681508] CPU0: proxy tick device unregistered
[    1.681515] CPU2: proxy tick device unregistered
[    1.681521] CPU1: proxy tick device unregistered
[    1.688222] CPU3: proxy tick device unregistered
[    1.688345] IRQ pipeline: torture stage removed.
[    1.688354] IRQ pipeline tests OK.

Hopefully, some debug assertion this test suite enables is going to trip
instead.

-- 
Philippe.