Re: i2c jitter is worse in PREEMPT_RT kernel than stock Raspberry Pi kernel

[Michael Franklin <mfranklin@comfiletech.com>]

On 12/12/2023 9:42 PM, Mike Galbraith wrote:
> On Tue, 2023-12-12 at 11:53 +0900, Michael Franklin wrote:
>
>> Interestingly, in the stock kernel, `htop` shows that most CPU activity
>> is concentrated on core 3 (which is what I expected and preferred),
>> while in the realtime kernel, the CPU activity is distributed across all
>> cores, despite booting with `isolcpus=3` and running the test program
>> with `task set -cp 3 $(pidof i2ctest)` in both kernels.
>>
>> Q1:  Why is i2c communication is more jittery in the realtime kernel
>> than the stock kernel?
> 
> I'd speculate it's primarily due to threaded IRQ handling being both
> more expensive and preemptible.
> 
>> Q2:  Why is activity distributed across all cores in the realtime
>> kernel, but more concentrated on core 3 in the stock kernel?
> 
> I don't see that.  Using isolcpus or not, the test proggy wakes only on
> CPU3 (as it had damn well better), and box wide affinity i2c IRQ thread
> wakes on CPU0.
> 
> Booting the non-rt kernel with 'threadirqs' behaves the same, and I
> suspect will jitter about the same should you try it.  You're isolating
> the test proggy, but for the rt kernel the IRQ thread is left dangling
> in the breeze to be perturbed by other IRQ threads or whatnot.
> 
> 	-Mike

Thanks.

I tested the stock kernel with `threadirqs` and indeed the jitter was worse, but the RT kernel still 
seemed to be more jittery.

However, based on what you mentioned, I decided to look further interrupts/IRQ behavior, and found 
this in /proc/interrupts:

PREEMPT_RT kernel:
            CPU0       CPU1       CPU2       CPU3
109:   22625815          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
IPI0:     19640   12018430    6870398    5548908       Rescheduling interrupts
IPI1:       713      26295      18104        367       Function call interrupts

Stock Kernel:
            CPU0       CPU1       CPU2       CPU3
109:   11129247          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
IPI0:       582        620        572        694       Rescheduling interrupts
IPI1:     40061      12360     108946    5475437       Function call interrupts

Stock Kernel with `threadirqs`:
            CPU0       CPU1       CPU2       CPU3
109:   21774128          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
IPI0:       674        657        687        617       Rescheduling interrupts
IPI1:     58655     127527   21331238    5780380       Function call interrupts

There you can see that, in the PREEMPT_RT kernel, there are a very large number of 'Rescheduling 
interrupts' and only a few 'Function call interrupts'.  However, in the stock kernel it is the 
opposite -- a large number of 'Function call interrupts' and a few 'Rescheduling interrupts'.

Adding `threadirqs` to the stock kenel seemed to just distribute many of the Function call 
interrupts over more cores.

Is there anything that can be done about the large number of Rescheduling interrupts in the 
PREEMPT_RT kernel?