Re: i2c jitter is worse in PREEMPT_RT kernel than stock Raspberry Pi kernel

[Michael Franklin <mfranklin@comfiletech.com>]

Michael Franklin
Software Engineer
COMFILE Technology
82-2-711-2592 ext 510
Skype: MFranklinAtComfile
On 12/13/2023 12:33 PM, Michael Franklin wrote:
> On 12/12/2023 9:42 PM, Mike Galbraith wrote:
>> On Tue, 2023-12-12 at 11:53 +0900, Michael Franklin wrote:
>>
>>> Interestingly, in the stock kernel, `htop` shows that most CPU activity
>>> is concentrated on core 3 (which is what I expected and preferred),
>>> while in the realtime kernel, the CPU activity is distributed across all
>>> cores, despite booting with `isolcpus=3` and running the test program
>>> with `task set -cp 3 $(pidof i2ctest)` in both kernels.
>>>
>>> Q1:  Why is i2c communication is more jittery in the realtime kernel
>>> than the stock kernel?
>>
>> I'd speculate it's primarily due to threaded IRQ handling being both
>> more expensive and preemptible.
>>
>>> Q2:  Why is activity distributed across all cores in the realtime
>>> kernel, but more concentrated on core 3 in the stock kernel?
>>
>> I don't see that.  Using isolcpus or not, the test proggy wakes only on
>> CPU3 (as it had damn well better), and box wide affinity i2c IRQ thread
>> wakes on CPU0.
>>
>> Booting the non-rt kernel with 'threadirqs' behaves the same, and I
>> suspect will jitter about the same should you try it.  You're isolating
>> the test proggy, but for the rt kernel the IRQ thread is left dangling
>> in the breeze to be perturbed by other IRQ threads or whatnot.
>>
>>     -Mike
> 
> Thanks.
> 
> I tested the stock kernel with `threadirqs` and indeed the jitter was worse, but the RT kernel still 
> seemed to be more jittery.
> 
> However, based on what you mentioned, I decided to look further interrupts/IRQ behavior, and found 
> this in /proc/interrupts:
> 
> PREEMPT_RT kernel:
>             CPU0       CPU1       CPU2       CPU3
> 109:   22625815          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
> IPI0:     19640   12018430    6870398    5548908       Rescheduling interrupts
> IPI1:       713      26295      18104        367       Function call interrupts
> 
> Stock Kernel:
>             CPU0       CPU1       CPU2       CPU3
> 109:   11129247          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
> IPI0:       582        620        572        694       Rescheduling interrupts
> IPI1:     40061      12360     108946    5475437       Function call interrupts
> 
> Stock Kernel with `threadirqs`:
>             CPU0       CPU1       CPU2       CPU3
> 109:   21774128          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
> IPI0:       674        657        687        617       Rescheduling interrupts
> IPI1:     58655     127527   21331238    5780380       Function call interrupts
> 
> There you can see that, in the PREEMPT_RT kernel, there are a very large number of 'Rescheduling 
> interrupts' and only a few 'Function call interrupts'.  However, in the stock kernel it is the 
> opposite -- a large number of 'Function call interrupts' and a few 'Rescheduling interrupts'.
> 
> Adding `threadirqs` to the stock kenel seemed to just distribute many of the Function call 
> interrupts over more cores.
> 
> Is there anything that can be done about the large number of Rescheduling interrupts in the 
> PREEMPT_RT kernel?

I was able to reduce the number of Rescheduling interrupts on the isolated core by `echo -1 > 
/proc/sys/kernel/sched_rt_runtime_us` but there are still too many Rescheduling interrupts relative 
to Function call interrupts on the other 3 cores and the jitter is still pretty bad.