Re: Question for AMD/Xilinx Zynq PREEMP_RT configuration check, CAN latency measuremet and FOSDEM 2025

[Pavel Pisa <pisa@fel.cvut.cz>]

Hello Sebastian,

On Wednesday 29 of January 2025 11:17:09 Sebastian Andrzej Siewior wrote:
> On 2025-01-28 16:29:27 [+0100], Pavel Pisa wrote:
> > Please check if you find some problematic choices.
>
> I didn't find anything obviously wrong. Assuming your CPU is busy in
> general you could remove NO_HZ in favour of PERIODIC. This is however
> not to cause spikes you describe below.

Great, thanks much for review by expert.

> > The cyclic test worked well, and we have even delivered two systems
> > to OSADL QA real-time farm
> >
> >   https://www.osadl.org/?id=4109
>
> It shows "IRQ work interrupts". Not sure what causes them.

I am not sure either. That list is from old kernel
in long term testing setup at OSADL.

The actual one show none IRQ work interrupts
after last reboot and overnigh test

Linux mzapo 6.13.0-rc6-rt3-dut #1 SMP PREEMPT_RT
Wed Jan 29 04:46:40 CET 2025 armv7l GNU/Linux

           CPU0       CPU1
 24:          0          0 GIC-0  27 Edge      gt
 25:     700822     327164 GIC-0  29 Edge      twd
 26:        300          0 GIC-0  59 Level     xuartps
 29:          0          0 GIC-0  45 Level     f8003000.dmac
 30:          0          0 GIC-0  46 Level     f8003000.dmac
 31:          0          0 GIC-0  47 Level     f8003000.dmac
 32:          0          0 GIC-0  48 Level     f8003000.dmac
 33:          0          0 GIC-0  49 Level     f8003000.dmac
 34:          0          0 GIC-0  72 Level     f8003000.dmac
 35:          0          0 GIC-0  73 Level     f8003000.dmac
 36:          0          0 GIC-0  74 Level     f8003000.dmac
 37:          0          0 GIC-0  75 Level     f8003000.dmac
 40:     460330          0 GIC-0  54 Level     end0
 41:          0          0 GIC-0  53 Level     e0002000.usb
 42:        356          0 GIC-0  56 Level     mmc0
 43:          0          0 GIC-0  43 Level     ttc_clockevent
 44:         25          0 GIC-0  39 Level     f8007100.adc
 45:          0          0 GIC-0  37 Level     arm-pmu
 46:          0          0 GIC-0  38 Level     arm-pmu
 47:        128          0 GIC-0  40 Level     f8007000.devcfg
 48:     314697          0 GIC-0  61 Level     can2
 49:     314597          0 GIC-0  62 Level     can3
 50:     314759          0 GIC-0  63 Level     can4
 51:     311516          0 GIC-0  64 Level     can5
IPI0:          0          0  CPU wakeup interrupts
IPI1:          0          0  Timer broadcast interrupts
IPI2:      17849     292126  Rescheduling interrupts
IPI3:       5923      11315  Function call interrupts
IPI4:          0          0  CPU stop interrupts
IPI5:     271078      74040  IRQ work interrupts
IPI6:          0          0  completion interrupts
Err:          0

So this seems as no cause.

> > However, the CAN/CAN FD communication latency measured on the CTU CAN FD
> > IP core is far from optimal. Some runs under load with
> > 10 msec latency. Our own CAN FD stack for RTEMS keeps with no exception
> > under 60 usec on the same hardware.
> >
> > I understand that the Linux socket layer and networking
> > stack are complex, and many optimizations are ahead.
> > We will be happy to contribute where we can and find time
> > and even some resources to engage more students etc...
> >
> > But I would like to be sure that the bad results are not
> > caused by our mistakes in configuration.
>
> You have CAN and "regular networking". My guess would be that regular
> networking blocks blocks BH and so your CAN. You could try to have all
> interrupts serviced on CPU0 and move CAN to CPU1. If so this should 
> improve then. Other than that, I would suggest to get some tracing to
> see what delays your CAN interrupts and/ or handling in general.

Yes, I think that design mixing regular networking packet
processing with CAN is the problem. We test even with setup where
CAN interrupts priority is boosted to 90

    echo "-> Rise CAN irq priorities"
    PIDS=$(ps -e | grep -E irq/[0-9]+-can[3-4] | tr -s ' ' | cut -d ' ' -f2)
    TXPID=$(ps -e | grep -E irq/[0-9]+-can2 | tr -s ' ' | cut -d ' ' -f2)
    chrt -f --pid 80 $TXPID
    for pid in $PIDS ; do
        chrt -f --pid 85 $pid
    done

ps Hxa --sort rtprio -o pid,policy,rtprio,state,tname,time,command

...
   70 FF      50 S ?        00:00:00 [irq/37-f8003000.dmac]
   71 FF      50 S ?        00:00:38 [irq/40-eth%d]
...
  405 FF      50 S ?        00:00:00 [irq/26-xuartps]
  355 FF      90 S ?        00:00:06 [irq/48-can2]
  361 FF      90 S ?        00:00:13 [irq/49-can3]
  366 FF      90 S ?        00:00:07 [irq/50-can4]
  371 FF      90 S ?        00:00:06 [irq/51-can5]
   22 FF      99 S ?        00:00:00 [migration/0]
   27 FF      99 S ?        00:00:00 [migration/1]

Even this setup is problematic under load.
The situation with CAN IRQ priority 50 and 90 can be compared
by clicking on "RT priority set" option

https://canbus.pages.fel.cvut.cz/can-latester/inspect.html?kernel=rt&prio=1&load=1&flood=1&fd=1

The switch between in kernel CAN gateway and userpace one
is controlled by "Kernel GW".

User CAN gateway is run with priority 80

  chrt -r 80 ugw -f can3 can2

I spot interesting trend after

  run-250103-045322-hist+6.13.0-rc1-rt1-g5374fecd2695+flood-prio-fd-load.json

that user gateway case, simple copy of frames from can3 to can2
has never exceed 1.4 ms almost for one month.
It could be interesting to corelate that with kernel changes.

We use branch

  for-kbuild-bot/current-stable

from

  git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git

to run daily testing. We can consider even something different,
but this choice has been given by interest in something
functional for each day and ahead of mainline merges to
catch some problems in advance.

It is interesting than in kernel gateway is significantly worse
now. It does not overhead of switching to userspace. But I am not
sure if it is not invoked in some kernel worker which
has lower or same real time priority than Ethenet networking.

In general, I think that the problem is that incommin
packets (CAN and Ethernet) load the same per CPU
worker. There are even backlog_napi threads per CPU

   46 TS       - S ?        00:00:00 [backlog_napi/0]
   47 TS       - S ?        00:00:00 [backlog_napi/1]

It has even TS priority. If I remember well, there has been
added option to allocate separate RX packets processing
therad (instead for default per CPU one) for given interface.
But I have no experience with such configuration.

Do you have or somebody else have idea how to achieve
that and if it is legal to boost such kernel therad
priority. It could help, because my general experience
with PREEMPT_RT even on this target is very positive
for tasks mapping HW directly and doing RT control.
Same for latency tester. No spikes under load over
250 usec or less.

> > I will be happy to meet you and discuss Linux and other
> > control and real-time areas at FOSDEM 2025.
>
> I should be able to make it.

Great, I would be happy to meet at FOSDEM or discuss
these topic later at some event.

> > Slides in English which I want to update/correct for FOSDEM
> >
> >  
> > https://talks.openalt.cz/media/openalt-2024/submissions/3XTMDF/resources/
> >openalt24_linux_for_rt-reduced_FbZPuS0.pdf
>
> looks good. If you want additional history points, I have some at
> 	https://files.speakerdeck.com/presentations/0620b5b3a00b42fc91fba6cc4092d2
>78/KR_2024_PREEMPT_RT_over_the_years.pdf Slide 11 - 21.

Thanks much for the input

> However you have most of the pieces so.
>

Best wishes,

                Pavel
-- 
                Pavel Pisa
    phone:      +420 603531357
    e-mail:     pisa@cmp.felk.cvut.cz
    Department of Control Engineering FEE CVUT
    Karlovo namesti 13, 121 35, Prague 2
    university: http://control.fel.cvut.cz/
    personal:   http://cmp.felk.cvut.cz/~pisa
    social:     https://social.kernel.org/ppisa
    projects:   https://www.openhub.net/accounts/ppisa
    CAN related:http://canbus.pages.fel.cvut.cz/
    RISC-V education: https://comparch.edu.cvut.cz/
    Open Technologies Research Education and Exchange Services
    https://gitlab.fel.cvut.cz/otrees/org/-/wikis/home