Re:Re: schedule delay 15ms

[JerryJun <jerryjun123@163.com>]

 We switched to another Intel platform and observed a gap of over 100ms(7574.499872->7574.631649). Could this be related to the ipipe->dovetail changes?
          swapper    0 [002]  7574.499869:             irq:irq_pipeline_entry: irq=33027
         swapper     0 [002]  7574.499870:              irq:irq_handler_entry: irq=33027 name=Out-of-band LAPIC timer interrupt
         swapper     0 [002]  7574.499870:    cobalt_core:cobalt_timer_expire: timer=0xffffc900006deef0
         swapper     0 [002]  7574.499870:       cobalt_core:cobalt_trace_pid: pid=2338, prio=85
         swapper     0 [002]  7574.499871:      cobalt_core:cobalt_timer_stop: timer=0xffffc900006deef0
         swapper     0 [002]  7574.499871:               irq:irq_handler_exit: irq=33027 ret=handled
         swapper     0 [002]  7574.499871:              irq:irq_pipeline_exit: irq=33027
         swapper     0 [002]  7574.499872:        cobalt_core:cobalt_schedule: status=0x10000000
         swapper     0 [002]  7574.499872:       cobalt_core:cobalt_trace_pid: pid=0, prio=-1
         swapper     0 [002]  7574.499872:     cobalt_core:cobalt_timer_start: timer=0xffff888880136828([watchdog]) value=1000000000 interval=0 mode=0x0
         swapper     0 [003]  7574.631649:             irq:irq_pipeline_entry: irq=33027
	rt-task1  2338 [002]  7574.631649:       cobalt_core:cobalt_trace_pid: pid=2338, prio=85
         swapper     0 [000]  7574.631649:             irq:irq_pipeline_entry: irq=33027
         swapper     0 [001]  7574.631650:             irq:irq_pipeline_entry: irq=33027
         swapper     0 [003]  7574.631650:              irq:irq_handler_entry: irq=33027 name=Out-of-band LAPIC timer interrupt
	rt-task1  2338 [002]  7574.631650:             irq:irq_pipeline_entry: irq=33027
         swapper     0 [000]  7574.631650:              irq:irq_handler_entry: irq=33027 name=Out-of-band LAPIC timer interrupt
         swapper     0 [001]  7574.631650:              irq:irq_handler_entry: irq=33027 name=Out-of-band LAPIC timer interrupt
         swapper     0 [003]  7574.631650:    cobalt_core:cobalt_timer_expire: timer=0xffff8888801b6298At 2025-11-21 02:53:35, "Jan Kiszka" <jan.kiszka@siemens.com> wrote:
>On 20.11.25 09:10, JerryJun wrote:
>> 
>> Which configurations do you think could cause this delay? During this process, the hardware remained unchanged. We used the following configurations and attempted to disable certain features in the kernel boot parameters, but it didn't work: xenomai.smi=disabled intel_idle.max_cstate=0 intel_pstate=disable nox2apic
>> 
>
>I cannot name a config option yet. But rather than comparing those, try
>to diff the structures of /sys/devices.
>
>Another shot into the dark: Boot the kernel with efi=noruntime.
>
>BTW, x2apic is harmless, actually better than [x]apic, but the latter
>should also not cause such issues.
>
>Jan
>
>> At 2025-11-19 00:00:12, "Jan Kiszka" <jan.kiszka@siemens.com> wrote:
>>> On 18.11.25 16:31, JerryJun wrote:
>>>>
>>>> Strangely enough, this delay wasn't present on the 4.19.89-ipip kernel - it only emerged after upgrading to 5.10.199-dovetail, despite no changes being made to the power management policy or thermal throttling settings during the update. Could this be related to potential instability in the HPET or TSC clock sources? And does upgrading to a new kernel version also necessitate a BIOS update?
>>>>
>>>
>>> Generally spoken, a newer kernel could always have changed something
>>> regarding how power tuning is done. Simple scenario: a kernel config
>>> switch was renaming or gained a new dependency that is not fulfill ->
>>> tuning feature is silently lost.
>>>
>>> Another reason for different behavior could be the enabling (maybe even
>>> by default) of a new feature that has unexpected side effects. You may
>>> check if there are new drivers active, new devices available.
>>>
>>> Jan
>>>
>>>> At 2025-11-14 13:38:36, "Jan Kiszka" <jan.kiszka@siemens.com> wrote:
>>>>> On 14.11.25 03:48, JerryJun wrote:
>>>>>>
>>>>>>   We added some trace event information, but from this gap point (6208.695948-> 6208.708726), it appears that the Cobalt kernel was frozen and no interrupt handling occurred.
>>>>>>    include trace point events :
>>>>>>     irq_vectors:call_function_entry                    
>>>>>>     irq_vectors:call_function_exit                     
>>>>>>     irq_vectors:call_function_single_entry             
>>>>>>     irq_vectors:call_function_single_exit   
>>>>>>
>>>>>
>>>>> If there are no interrupt in that phase while, e.g., some timer
>>>>> interrupt should have occurred, then you should definitely check for
>>>>> hardware/firmware related delays that the OS cannot influence. Or for
>>>>> power-saving / thermal-throttling causes.
>>>>>
>>>>> Jan
>>>>>
>>>>>> 在 2025-11-13 13:31:54，"Jan Kiszka" <jan.kiszka@siemens.com> 写道：
>>>>>>> On 13.11.25 05:00, JerryJun wrote:
>>>>>>>>
>>>>>>>> We conducted tests on the kernel version v5.10.245-cip65-dovetail1. The test results are attached. During this period, we also performed latency tests, which showed a 10ms delay at 7174.313414 -> 7174.323086.  irq 114 is net driver interrupt 。
>>>>>>>>
>>>>>>>
>>>>>>>         swapper     0 [003]  7174.313414:                     irq:irq_pipeline_entry: irq=114
>>>>>>>         swapper     0 [003]  7174.323086:                     irq:irq_pipeline_entry: irq=33027
>>>>>>>
>>>>>>> So, that is the gap. Did you also turn on irq_pipeline_exit? I'm not 
>>>>>>> seeing that event in the trace. Furthermore, I always for get about 
>>>>>>> that, it is useful to have irq_vectors events because those include 
>>>>>>> timer irqs. But it is unlikely that we missed any in that gap because 
>>>>>>> they tend to trigger activities like cobalt_core:cobalt_timer_expire.
>>>>>>>
>>>>>>> What is more likely here is that your hardware or firmware is not 
>>>>>>> letting IRQs through. We could confirm that if the issue is reproducible 
>>>>>>> while have function tracing enabled. That comes with high overhead, but 
>>>>>>> the detailed result can tell us what the kernel did while there were no 
>>>>>>> interrupts.
>>>>>>>
>>>>>>> There is also native tooling by the kernel to detect "hardware noise",
>>>>>>> see https://docs.kernel.org/tools/rtla/rtla-hwnoise.html. Due to its 
>>>>>>> nature, this noise would affect Xenomai equally.
>>>>>>>
>>>>>>> Jan
>>>>>>>
>>>>>>> -- 
>>>>>>> Siemens AG, Foundational Technologies
>>>>>>> Linux Expert Center
>>>>>
>>>>>
>>>>> -- 
>>>>> Siemens AG, Foundational Technologies
>>>>> Linux Expert Center
>>>
>>> -- 
>>> Siemens AG, Foundational Technologies
>>> Linux Expert Center
>
>
>-- 
>Siemens AG, Foundational Technologies
>Linux Expert Center