From: Jan Kiszka <jan.kiszka@siemens.com>
To: JerryJun <jerryjun123@163.com>,
"xenomai@lists.linux.dev" <xenomai@lists.linux.dev>
Subject: Re: schedule delay 15ms
Date: Wed, 12 Nov 2025 09:23:03 +0100 [thread overview]
Message-ID: <d14c720e-021e-4fc8-8aba-340903db0fdf@siemens.com> (raw)
In-Reply-To: <4c6fa8a7.4dce.19a76af1422.Coremail.jerryjun123@163.com>
On 12.11.25 07:09, JerryJun wrote:
>
> hi：
> We are encountering some scheduling issues. Within a 15ms time interval (99058.151147 -> 99058.167775), no real-time threads were scheduled,
> including the swapper idle thread. We have a 1ms-polling-rt-task running on CPU1, but it was also not scheduled during this period.
> Even the RTDM interrupts from peripherals were not responded to or processed. What could be the reason for this?
>
> Architecture： x86_64
> linux kernel：5.10.199-dovetail1
This version is 2 years old. Did you also check if the latest release
(v5.10.245-cip65-dovetail1) has this issue?
> cobalt：3.2.4
> kernel cmdline: xenomai.supported_cpus=0xf idle=halt nosmap
>
> Thread-name CPU PRIO
> main-thread 0 26
> rt-task1 1 86
> rt-task3 1 91
> rt-task4 1 85
> 1ms-polling-rt-task 1 94
> rt-task2 2 85
> rt-task5 2 92
> rt-task6 2 92
> rt-task7 2 85
> rt-task8 2 85
> rt-task13 2 90
> rt-task15 2 85
> kernel-drv-task11 2 98
> kernel-drv-task10 3 94
> kernel-drv-task12 3 98
> kernel-drv-task14 3 98
>
> perf record -e cobalt_core:cobalt_schedule
>
> swapper 0 [001] 99058.149903: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.149905: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.149930: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.149932: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150029: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.150031: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150034: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150035: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150036: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task4 56486 [001] 99058.150037: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150052: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.150055: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150073: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task5 56510 [002] 99058.150075: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150078: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task6 56507 [002] 99058.150079: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150127: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task7 56499 [002] 99058.150131: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task7 56499 [002] 99058.150147: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task7 56499 [002] 99058.150148: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.150149: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task8 56496 [002] 99058.150150: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150184: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150185: cobalt_core:cobalt_schedule: status=0x10000000
> t-task3 56485 [001] 99058.150186: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task4 56486 [001] 99058.150187: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150202: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.150205: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150246: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.150248: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150334: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150335: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150336: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task4 56486 [001] 99058.150337: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150345: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.150347: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150352: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.150355: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [003] 99058.150366: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150372: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task5 56510 [002] 99058.150374: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150376: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task6 56507 [002] 99058.150378: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150381: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task7 56499 [002] 99058.150383: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task10 24070 [003] 99058.150387: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150444: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.150446: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150447: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task8 56496 [002] 99058.150448: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150484: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150485: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150486: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task4 56486 [001] 99058.150487: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150502: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.150504: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [003] 99058.150505: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.150505: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task10 24070 [003] 99058.150517: cobalt_core:cobalt_schedule: status=0x10000000
> main-thread 24485 [002] 99058.150543: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.150544: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150564: cobalt_core:cobalt_schedule: status=0x10000000
> 1ms-polling-rt-task 56488 [001] 99058.150567: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150611: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.150613: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150630: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task7 56499 [002] 99058.150632: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150634: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150635: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150636: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task4 56486 [001] 99058.150637: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150642: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.150643: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150671: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task5 56510 [002] 99058.150673: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150675: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task6 56507 [002] 99058.150677: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150740: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.150742: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150745: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task8 56496 [002] 99058.150747: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150761: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.150763: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150783: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150785: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150786: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task4 56486 [001] 99058.150787: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150839: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.150841: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150879: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task7 56499 [002] 99058.150881: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150911: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.150913: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.150933: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150934: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.150935: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task4 56486 [001] 99058.150936: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150938: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.150940: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150970: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task5 56510 [002] 99058.150971: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.150974: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task6 56507 [002] 99058.150975: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.151037: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.151039: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.151044: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task8 56496 [002] 99058.151046: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.151060: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.151063: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.151082: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.151084: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.151085: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task4 56486 [001] 99058.151086: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.151138: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task7 56499 [002] 99058.151143: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.151147: cobalt_core:cobalt_schedule: status=0x10000000
> (---schedule delay 15ms---)
> swapper 0 [003] 99058.167775: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.167776: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.167777: cobalt_core:cobalt_schedule: status=0x10000000
> 1ms-polling-rt-task 56488 [001] 99058.167792: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task11 24077 [002] 99058.167794: cobalt_core:cobalt_schedule: status=0x10000000
> 1ms-polling-rt-task 56488 [001] 99058.167795: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task12 24088 [003] 99058.167797: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.167798: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task11 24077 [002] 99058.167801: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task5 56510 [002] 99058.167803: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.167804: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task6 56507 [002] 99058.167805: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task4 56486 [001] 99058.167807: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task13 56506 [002] 99058.167808: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task10 24070 [003] 99058.167809: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.167810: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task10 24070 [003] 99058.167811: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task13 56506 [002] 99058.167816: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [003] 99058.167817: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task13 56506 [002] 99058.167824: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task13 56506 [002] 99058.167826: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task10 24070 [003] 99058.167827: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task8 56496 [002] 99058.167828: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task10 24070 [003] 99058.167829: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task13 56506 [002] 99058.167833: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [003] 99058.167834: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task10 24070 [003] 99058.167843: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.167893: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task13 56506 [002] 99058.167894: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task7 56499 [002] 99058.167896: cobalt_core:cobalt_schedule: status=0x10000000
> 1ms-polling-rt-task 56488 [001] 99058.167897: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task15 56598 [002] 99058.167898: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task13 56506 [002] 99058.167906: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task13 56506 [002] 99058.167921: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task15 56598 [002] 99058.167933: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task2 56508 [002] 99058.167934: cobalt_core:cobalt_schedule: status=0x10000000
> main-thread 24483 [001] 99058.167945: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.167948: cobalt_core:cobalt_schedule: status=0x10000000
> main-thread 56538 [003] 99058.167950: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.167950: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.167952: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.167953: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.167954: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.167955: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task10 24070 [003] 99058.167955: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task3 56485 [001] 99058.167956: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task1 56487 [001] 99058.167960: cobalt_core:cobalt_schedule: status=0x10000000
> rt-task4 56486 [001] 99058.167962: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task11 24077 [002] 99058.167968: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task14 24083 [003] 99058.167976: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task14 24083 [003] 99058.167977: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.167977: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task12 24088 [003] 99058.167989: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task11 24077 [002] 99058.167990: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [002] 99058.167993: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task10 24070 [003] 99058.167996: cobalt_core:cobalt_schedule: status=0x10000000
> swapper 0 [001] 99058.167997: cobalt_core:cobalt_schedule: status=0x10000000
> kernel-drv-task14 24083 [003] 99058.168000: cobalt_core:cobalt_schedule: status=0x10000000
>
For a deeper analysis, it would be good to trace all cobalt* and all
irq* events. Those may tell us if we are not getting irqs at all or if
something is blocking their processing.
Jan
--
Siemens AG, Foundational Technologies
Linux Expert Center
next prev parent reply other threads:[~2025-11-12 8:23 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-11-12 6:09 schedule delay 15ms JerryJun
2025-11-12 8:23 ` Jan Kiszka [this message]
2025-11-13 4:00 ` JerryJun
2025-11-13 5:31 ` Jan Kiszka
2025-11-14 2:48 ` JerryJun
2025-11-14 5:38 ` Jan Kiszka
2025-11-18 15:31 ` JerryJun
2025-11-18 16:00 ` Jan Kiszka
2025-11-20 8:10 ` JerryJun
2025-11-20 18:53 ` Jan Kiszka
2025-11-26 3:20 ` JerryJun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=d14c720e-021e-4fc8-8aba-340903db0fdf@siemens.com \
--to=jan.kiszka@siemens.com \
--cc=jerryjun123@163.com \
--cc=xenomai@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.