i2c jitter is worse in PREEMPT_RT kernel than stock Raspberry Pi kernel

i2c jitter is worse in PREEMPT_RT kernel than stock Raspberry Pi kernel

[Michael Franklin]

Hello,

I'm experimenting with the realtime PREEMPT_RT patches on a Raspberry Pi 
4 and Raspberry Pi 5.  I'm using 6.1.66-rt19, but have also tested with 
the latest 6.7 kernel and RT patches.

In most of my experiments the realtime kernel improves jitter over the 
stock kernel, but I've discovered that when using i2c, the jitter is 
worse in the realtime kernel than the stock kernel.

It's a little difficult to describe, but can be seen quite clearly in 
this annotated video: 
https://comfiletechdownloads.z12.web.core.windows.net/RT_i2c_jitter.mp4

The only difference between the stock kernel and the realtime kernel is 
that full preemption is enabled in `menuconfig` for the realtime 
kernel.  The kernel is booted with `isolcpus=3` and the program is moved 
to core 3 with `task set -cp 3 $(pidof i2ctest)`.  Also the program is 
scheduled as SCHED_FIFO.

The i2c test program is very simple.  It is written in C/C++, and simply 
sends 2 bytes to an MCP23017 IO expander.  It is basically the following 
C++ pseudocode:

struct sched_param param;
param.sched_priority = 99;
if (sched_setscheduler(0, SCHED_FIFO, &param) != 0)
{
     perror("sched_setscheduler");
     return 1;
}

int fd = open("/dev/i2c-1");
unsigned char data[2];

while(1)
{
     struct i2c_msg messages[] =
     {
         {
             .addr = 0x20,
             .len = 2,
             .buf = data,
         },
     };

     struct i2c_rdwr_ioctl_data payload =
     {
         .msgs = messages,
         .nmsgs = sizeof(messages) / sizeof(messages[0]),
     };

     ioctl(fd, I2C_RDWR, &payload);

     // to avoid requiring `echo -1 > /proc/sys/kernel/sched_rt_runtime_us`
     std::this_thread::sleep_for(1us);
}

The communication works fine, but it's just too jittery in the reatltime 
kernel.

Interestingly, in the stock kernel, `htop` shows that most CPU activity 
is concentrated on core 3 (which is what I expected and preferred), 
while in the realtime kernel, the CPU activity is distributed across all 
cores, despite booting with `isolcpus=3` and running the test program 
with `task set -cp 3 $(pidof i2ctest)` in both kernels.

Q1:  Why is i2c communication is more jittery in the realtime kernel 
than the stock kernel?

Q2:  Why is activity distributed across all cores in the realtime 
kernel, but more concentrated on core 3 in the stock kernel?

Q3:  Is there anything that can be done, either via kernel 
configuration, boot parameters, or something else, that improve the 
jitter in the realtime kernel for this specific use case?

Thank you for your time and consideration,

Mike

Re: i2c jitter is worse in PREEMPT_RT kernel than stock Raspberry Pi kernel

[Mike Galbraith]

On Tue, 2023-12-12 at 11:53 +0900, Michael Franklin wrote:
> Hello,

Greetings,


<snip proggy etc>


> Interestingly, in the stock kernel, `htop` shows that most CPU activity
> is concentrated on core 3 (which is what I expected and preferred),
> while in the realtime kernel, the CPU activity is distributed across all
> cores, despite booting with `isolcpus=3` and running the test program
> with `task set -cp 3 $(pidof i2ctest)` in both kernels.
>
> Q1:  Why is i2c communication is more jittery in the realtime kernel
> than the stock kernel?

I'd speculate it's primarily due to threaded IRQ handling being both
more expensive and preemptible.

> Q2:  Why is activity distributed across all cores in the realtime
> kernel, but more concentrated on core 3 in the stock kernel?

I don't see that.  Using isolcpus or not, the test proggy wakes only on
CPU3 (as it had damn well better), and box wide affinity i2c IRQ thread
wakes on CPU0.

Booting the non-rt kernel with 'threadirqs' behaves the same, and I
suspect will jitter about the same should you try it.  You're isolating
the test proggy, but for the rt kernel the IRQ thread is left dangling
in the breeze to be perturbed by other IRQ threads or whatnot.

	-Mike

Re: i2c jitter is worse in PREEMPT_RT kernel than stock Raspberry Pi kernel

[Michael Franklin]

On 12/12/2023 9:42 PM, Mike Galbraith wrote:
> On Tue, 2023-12-12 at 11:53 +0900, Michael Franklin wrote:
>
>> Interestingly, in the stock kernel, `htop` shows that most CPU activity
>> is concentrated on core 3 (which is what I expected and preferred),
>> while in the realtime kernel, the CPU activity is distributed across all
>> cores, despite booting with `isolcpus=3` and running the test program
>> with `task set -cp 3 $(pidof i2ctest)` in both kernels.
>>
>> Q1:  Why is i2c communication is more jittery in the realtime kernel
>> than the stock kernel?
> 
> I'd speculate it's primarily due to threaded IRQ handling being both
> more expensive and preemptible.
> 
>> Q2:  Why is activity distributed across all cores in the realtime
>> kernel, but more concentrated on core 3 in the stock kernel?
> 
> I don't see that.  Using isolcpus or not, the test proggy wakes only on
> CPU3 (as it had damn well better), and box wide affinity i2c IRQ thread
> wakes on CPU0.
> 
> Booting the non-rt kernel with 'threadirqs' behaves the same, and I
> suspect will jitter about the same should you try it.  You're isolating
> the test proggy, but for the rt kernel the IRQ thread is left dangling
> in the breeze to be perturbed by other IRQ threads or whatnot.
> 
> 	-Mike

Thanks.

I tested the stock kernel with `threadirqs` and indeed the jitter was worse, but the RT kernel still 
seemed to be more jittery.

However, based on what you mentioned, I decided to look further interrupts/IRQ behavior, and found 
this in /proc/interrupts:

PREEMPT_RT kernel:
            CPU0       CPU1       CPU2       CPU3
109:   22625815          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
IPI0:     19640   12018430    6870398    5548908       Rescheduling interrupts
IPI1:       713      26295      18104        367       Function call interrupts

Stock Kernel:
            CPU0       CPU1       CPU2       CPU3
109:   11129247          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
IPI0:       582        620        572        694       Rescheduling interrupts
IPI1:     40061      12360     108946    5475437       Function call interrupts

Stock Kernel with `threadirqs`:
            CPU0       CPU1       CPU2       CPU3
109:   21774128          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
IPI0:       674        657        687        617       Rescheduling interrupts
IPI1:     58655     127527   21331238    5780380       Function call interrupts

There you can see that, in the PREEMPT_RT kernel, there are a very large number of 'Rescheduling 
interrupts' and only a few 'Function call interrupts'.  However, in the stock kernel it is the 
opposite -- a large number of 'Function call interrupts' and a few 'Rescheduling interrupts'.

Adding `threadirqs` to the stock kenel seemed to just distribute many of the Function call 
interrupts over more cores.

Is there anything that can be done about the large number of Rescheduling interrupts in the 
PREEMPT_RT kernel?

Re: i2c jitter is worse in PREEMPT_RT kernel than stock Raspberry Pi kernel

[Michael Franklin]

Michael Franklin
Software Engineer
COMFILE Technology
82-2-711-2592 ext 510
Skype: MFranklinAtComfile
On 12/13/2023 12:33 PM, Michael Franklin wrote:
> On 12/12/2023 9:42 PM, Mike Galbraith wrote:
>> On Tue, 2023-12-12 at 11:53 +0900, Michael Franklin wrote:
>>
>>> Interestingly, in the stock kernel, `htop` shows that most CPU activity
>>> is concentrated on core 3 (which is what I expected and preferred),
>>> while in the realtime kernel, the CPU activity is distributed across all
>>> cores, despite booting with `isolcpus=3` and running the test program
>>> with `task set -cp 3 $(pidof i2ctest)` in both kernels.
>>>
>>> Q1:  Why is i2c communication is more jittery in the realtime kernel
>>> than the stock kernel?
>>
>> I'd speculate it's primarily due to threaded IRQ handling being both
>> more expensive and preemptible.
>>
>>> Q2:  Why is activity distributed across all cores in the realtime
>>> kernel, but more concentrated on core 3 in the stock kernel?
>>
>> I don't see that.  Using isolcpus or not, the test proggy wakes only on
>> CPU3 (as it had damn well better), and box wide affinity i2c IRQ thread
>> wakes on CPU0.
>>
>> Booting the non-rt kernel with 'threadirqs' behaves the same, and I
>> suspect will jitter about the same should you try it.  You're isolating
>> the test proggy, but for the rt kernel the IRQ thread is left dangling
>> in the breeze to be perturbed by other IRQ threads or whatnot.
>>
>>     -Mike
> 
> Thanks.
> 
> I tested the stock kernel with `threadirqs` and indeed the jitter was worse, but the RT kernel still 
> seemed to be more jittery.
> 
> However, based on what you mentioned, I decided to look further interrupts/IRQ behavior, and found 
> this in /proc/interrupts:
> 
> PREEMPT_RT kernel:
>             CPU0       CPU1       CPU2       CPU3
> 109:   22625815          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
> IPI0:     19640   12018430    6870398    5548908       Rescheduling interrupts
> IPI1:       713      26295      18104        367       Function call interrupts
> 
> Stock Kernel:
>             CPU0       CPU1       CPU2       CPU3
> 109:   11129247          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
> IPI0:       582        620        572        694       Rescheduling interrupts
> IPI1:     40061      12360     108946    5475437       Function call interrupts
> 
> Stock Kernel with `threadirqs`:
>             CPU0       CPU1       CPU2       CPU3
> 109:   21774128          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
> IPI0:       674        657        687        617       Rescheduling interrupts
> IPI1:     58655     127527   21331238    5780380       Function call interrupts
> 
> There you can see that, in the PREEMPT_RT kernel, there are a very large number of 'Rescheduling 
> interrupts' and only a few 'Function call interrupts'.  However, in the stock kernel it is the 
> opposite -- a large number of 'Function call interrupts' and a few 'Rescheduling interrupts'.
> 
> Adding `threadirqs` to the stock kenel seemed to just distribute many of the Function call 
> interrupts over more cores.
> 
> Is there anything that can be done about the large number of Rescheduling interrupts in the 
> PREEMPT_RT kernel?

I was able to reduce the number of Rescheduling interrupts on the isolated core by `echo -1 > 
/proc/sys/kernel/sched_rt_runtime_us` but there are still too many Rescheduling interrupts relative 
to Function call interrupts on the other 3 cores and the jitter is still pretty bad.

Re: i2c jitter is worse in PREEMPT_RT kernel than stock Raspberry Pi kernel

[Mike Galbraith]

On Wed, 2023-12-13 at 12:33 +0900, Michael Franklin wrote:
> 
> I tested the stock kernel with `threadirqs` and indeed the jitter was worse, but the RT kernel still 
> seemed to be more jittery.
> 
> However, based on what you mentioned, I decided to look further interrupts/IRQ behavior, and found 
> this in /proc/interrupts:
> 
> PREEMPT_RT kernel:
>             CPU0       CPU1       CPU2       CPU3
> 109:   22625815          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
> IPI0:     19640   12018430    6870398    5548908       Rescheduling interrupts
> IPI1:       713      26295      18104        367       Function call interrupts
> 
> Stock Kernel:
>             CPU0       CPU1       CPU2       CPU3
> 109:   11129247          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
> IPI0:       582        620        572        694       Rescheduling interrupts
> IPI1:     40061      12360     108946    5475437       Function call interrupts
> 
> Stock Kernel with `threadirqs`:
>             CPU0       CPU1       CPU2       CPU3
> 109:   21774128          0          0          0  rp1_irq_chip   8 Level     1f00074000.i2c
> IPI0:       674        657        687        617       Rescheduling interrupts
> IPI1:     58655     127527   21331238    5780380       Function call interrupts
> 
> There you can see that, in the PREEMPT_RT kernel, there are a very large number of 'Rescheduling 
> interrupts' and only a few 'Function call interrupts'.  However, in the stock kernel it is the 
> opposite -- a large number of 'Function call interrupts' and a few 'Rescheduling interrupts'.

Part of that is mitigation ala 539fbb5be0, and part is RT being its
naturally twitchy self.


	-Mike