CAN latency measuremet on AMD/Xilinx Zynq with PREEMP_RT - added threaded NAPI configuration

[Pavel Pisa <pisa@fel.cvut.cz>]

[-- Attachment #1: Type: text/plain, Size: 6629 bytes --]

Hello Marc and Sebastian,

thanks for suggestions and discussion at FOSDEM 2025.

We are slow, sorry, I have teaching, we have worked
on new NuttX boot loader, pysmCoder and RTEMS, etc..

But there is some partial progress for our Linux
CAN/CAN FD latency benchmarking

  https://canbus.pages.fel.cvut.cz/#can-bus-channels-mutual-latency-testing

On Saturday 01 of February 2025 14:56:49 you wrote:
> On 27.01.2025 00:27:02, Pavel Pisa wrote:
> > May it be that there is even some problem in our RT kernel
> > configuration. But I have not found which option could
> > be problematic. I would send config files if you want to look
> > at them.

Thanks to Sebastian for review which did not reveal
something suspicious.

> Try switching the CAN interfaces to threaded NAPI:
>
>     echo 1 | sudo tee /sys/class/net/canX/threaded
>
> and configure the priorities of the can interface NAPI thread. You might
> also switch the Ethernet interfaces to threaded NAPI and give them a
> different/lower prio.

Pavel Hronek has found some time on his Erasmus stay in Italy
and added Threaded NAPI into options of our CAN testing matrix.

The scripts to set paramaters for CAN interfaces on the device
under the test (DUT) are located in the directory device-scripts
of our latency automation repository
 
  https://gitlab.fel.cvut.cz/canbus/can-benchmark/can-latester-automation/-/tree/master/device-scripts

The scripts are attached as well for easier review.

The set-can-threaded.sh sets

  echo $1 > /sys/class/net/canX/threaded

and I have added increase of the napi/canX-Y
treads priority to it on this Monday (March 24)

  chrt -f --pid 80 $pid

This setup logs are selected by option "Threaded NAPI"

The classical boost of CAN interfaces IRQ threads
is under switch "RT priority set", which locates
irq/[0-9]+-can[0-9] and sets

  chrt -f --pid 90 $pid

The final state of the system in the threaded config show

  ps Hxa --sort rtprio -o pid,policy,rtprio,state,tname,time,command

   16 FF       1 S ?        00:00:13 [ktimers/0]
   17 FF       1 I ?        00:00:01 [rcu_preempt]
   18 FF       1 S ?        00:00:00 [rcub/0]
   20 FF       1 S ?        00:00:02 [rcuc/0]
   23 FF       1 S ?        00:00:04 [irq_work/0]
   26 FF       1 S ?        00:00:01 [irq_work/1]
   28 FF       1 S ?        00:00:02 [rcuc/1]
   29 FF       1 S ?        00:00:05 [ktimers/1]
   48 FF      50 S ?        00:00:00 [watchdogd]
   63 FF      50 S ?        00:00:00 [irq/29-f8003000.dmac]
   ...
   72 FF      50 S ?        00:01:19 [irq/40-eth%d]
   75 FF      50 S ?        00:00:00 [irq/41-e0002000.usb]
   78 FF      50 S ?        00:00:00 [irq/44-f8007100.adc]
   80 FF      50 S ?        00:00:00 [irq/42-mmc0]
   81 FF      50 S ?        00:00:00 [irq/42-s-mmc0]
   84 FF      50 S ?        00:00:00 [irq/47-f8007000.devcfg]
  410 FF      50 S ?        00:00:00 [irq/26-xuartps]
 1907 FF      80 S ?        00:00:00 [napi/can2-19]
 1908 FF      80 S ?        00:00:13 [napi/can3-20]
 1909 FF      80 S ?        00:00:09 [napi/can4-21]
 1910 FF      80 S ?        00:00:08 [napi/can5-22]
  355 FF      90 S ?        00:00:12 [irq/48-can2]
  364 FF      90 S ?        00:00:20 [irq/49-can3]
  369 FF      90 S ?        00:00:14 [irq/50-can4]
  376 FF      90 S ?        00:00:14 [irq/51-can5]
   22 FF      99 S ?        00:00:00 [migration/0]
   27 FF      99 S ?        00:00:00 [migration/1]

There is recorded significant change when the respective
NAPI threads priority has been increased and policy
changed from default TS to FF. We are starting to get
maximal times under one millisecond in the most cases

CAN FD messages, flood, system under network load,
CAN gateway (retransmit all messages to the secondary
intreface on DUT) kernel

  https://canbus.pages.fel.cvut.cz/can-latester/inspect.html?kernel=rt&load=1&prio=1&flood=1&fd=1&thrd=1&kern=1

For standard CAN messages

  https://canbus.pages.fel.cvut.cz/can-latester/inspect.html?kernel=rt&load=1&prio=1&flood=1&fd=0&thrd=1&kern=1

You can switch on and of other options to see how the latency
profiles vary. Suspicious is the series from run

  run-250326-05****-hist+6.14.0-rc1-rt1-gecdc0d0bb42d+***.json

Which should be after priority boost but maximal latency
are high still for more combinations. But it could be some
interference. Our daily mainline and RT kernel data starts from
April 2023 (Archives available under Data range), so we will
be more certain what is the result of the current setup
after some months.

> MY coworker Lucas mentioned another option would be to stay with
> traditional soft IRQ based NAPI but reduce the NAPI weight for the
> "unimportant" interfaces to 1.

OK, we will keep that on our list of the experiments to try.
I hope that our tests could help to enhance CAN users situation
on GNU/Linux when they use it for control. The result can
be even manual how to setup priorities and interfaces for
reliable operation. We already provide toolong to obtain
some confidence in the given setup because all our tooling
can be run by everybody on their setup.

May it be we can catch something which can be enhanced in the
mainline kernel code as well...

For sure, deeper ftrace based analysis is on our list as well,
but it could be art of some studnet thesis or some joint debug
session with somebody from you. I probably do not find time
to study that deeper without guidance soon.

> > RTEMS behavior on same HW experiences maximal latency about 60 usec
> > even with networking and other load, sources, documentation, reports
> >
> >  
> > https://canbus.pages.fel.cvut.cz/#cancan-fd-subsystem-and-drivers-for-rtems
> >
> > The CiA article is already available to public and referenced from our
> > page same as the Michal Lenc's Master's thesis.

Some update to above mentioned project of our complete new CAN/CAN FD stack
for RTEMS. It has been mainlined in January and documentation propagated
to the online site in March

  https://ftp.rtems.org/pub/rtems/people/amar/docs1/bsp-howto/can.html

Best wishes,

                Pavel

                Pavel Pisa
    phone:      +420 603531357
    e-mail:     pisa@cmp.felk.cvut.cz
    Department of Control Engineering FEE CVUT
    Karlovo namesti 13, 121 35, Prague 2
    university: http://control.fel.cvut.cz/
    personal:   http://cmp.felk.cvut.cz/~pisa
    social:     https://social.kernel.org/ppisa
    projects:   https://www.openhub.net/accounts/ppisa
    CAN related:http://canbus.pages.fel.cvut.cz/
    RISC-V education: https://comparch.edu.cvut.cz/
    Open Technologies Research Education and Exchange Services
    https://gitlab.fel.cvut.cz/otrees/org/-/wikis/home

[-- Attachment #2: set-can-threaded.sh --]
[-- Type: text/x-shellscript, Size: 408 bytes --]

#!/bin/bash

NAPI_PRIO=80

if (( $# != 1 )); then
  echo "Need 1 argument (value 0 or 1)" >&2
  exit 1
fi

for ifc in can2 can3 can4 can5 ; do
  echo $1 > /sys/class/net/$ifc/threaded
done

if [ $1 -eq 1 ] ; then
  PIDS=$(ps -e | grep -E napi/can[0-9]+-[0-9] | tr -s ' ' | cut -d ' ' -f2)
  for pid in $PIDS ; do
    echo "Setting RT priority $NAPI_PRIO for $pid"
    chrt -f --pid $NAPI_PRIO $pid
  done
fi

[-- Attachment #3: set-irq-prio.sh --]
[-- Type: text/x-shellscript, Size: 228 bytes --]

#!/bin/bash
if [ ! -z $1 ] ; then
PRIO=$1
else
PRIO=90
fi

PIDS=$(ps -e | grep -E irq/[0-9]+-can[0-9] | tr -s ' ' | cut -d ' ' -f2)
for pid in $PIDS ; do
	echo "Setting RT priority $PRIO for $pid"
	chrt -f --pid $PRIO $pid
done