high prioritized threaded interrupt is delayed due to low priority softirq job

[yosi yarchi <yosi.yarchi@gmail.com>]

  Hi

  my platform is:
    SAM9X35 (8 CAN mailboxes),
    Linux mscb 5.4.41-linux4sam-2020.04-rt24 #1 PREEMPT_RT Wed Nov 9 
06:12:28 UTC 2022 armv5tejl GNU/Linux
    CAN threaded interrupt (irq/30-can0-346) is configured to be highest 
priority on
      system, and it is scheduled very quickly after CAN irq_handler expires
      (irq_handler_entry: irq=30 name=can0), doing its job (at91_irq).

  my problem is that from time to time I get CAN RX overflow. problem 
analysis is as the following (see trace below):
    after CAN threaded interrupt completes its job (at91_irq_r), 
scheduler force it to
      run some other low priority softirq jobs (softirq_entry: vec=1 
[action=TIMER],
      softirq_entry: vec=3 [action=NET_RX]). during the processing of 
those jobs, new HW
      CAN irq_handler expires (irq_handler_entry: irq=30 name=can0), but 
scheduler
      schedule the at91_irq job at the end of job queue, which is too 
late, and I get CAN
      rx overflow.

  I've tried some solutions, no one worked out of the box:
    1. disable all relevant irqs (timer, network) at CAN irq_handler 
(irq_handler_entry:
      irq=30 name=can0), and enable them at the end of CAN threaded 
interrupt job
      processing (at91_irq_r)
    2. configure CAN job (at91_irq) to run inside the CAN irq_handler 
(irq_handler_entry:
      irq=30 name=can0) by adding IRQF_NO_THREAD flag to request_irq.

  is there built-in solution for this problem at linux-rt?
  if not, what could be good 'hacked' solutions for the problem?

  with best regards
  Yosi Yarchi

  kernel trace:
  -------------

         <idle>-0     [000] d..h1..  6020.481859: irq_handler_entry: 
irq=30 name=can0
         <idle>-0     [000] d..h1..  6020.481875: irq_handler_exit: 
irq=30 ret=handled
    irq/30-can0-346   [000] d......  6020.481972: at91_irq: 
(at91_irq+0x0/0xab0)
    irq/30-can0-346   [000] d...1..  6020.482090: at91_irq_r: 
(irq_forced_thread_fn+0x3c/0x8c <- at91_irq)
         <idle>-0     [000] d..h1..  6020.482330: irq_handler_entry: 
irq=30 name=can0
         <idle>-0     [000] d..h1..  6020.482347: irq_handler_exit: 
irq=30 ret=handled
    irq/30-can0-346   [000] d......  6020.482438: at91_irq: 
(at91_irq+0x0/0xab0)
    irq/30-can0-346   [000] d...1..  6020.482553: at91_irq_r: 
(irq_forced_thread_fn+0x3c/0x8c <- at91_irq)
         <idle>-0     [000] d..h1..  6020.482812: irq_handler_entry: 
irq=30 name=can0
         <idle>-0     [000] d..h1..  6020.482829: irq_handler_exit: 
irq=30 ret=handled
    irq/30-can0-346   [000] d......  6020.482923: at91_irq: 
(at91_irq+0x0/0xab0)
    irq/30-can0-346   [000] d...1..  6020.483040: at91_irq_r: 
(irq_forced_thread_fn+0x3c/0x8c <- at91_irq)
         <idle>-0     [000] d..h1..  6020.483290: irq_handler_entry: 
irq=30 name=can0
         <idle>-0     [000] d..h1..  6020.483308: irq_handler_exit: 
irq=30 ret=handled
    irq/30-can0-346   [000] d......  6020.483400: at91_irq: 
(at91_irq+0x0/0xab0)
    irq/30-can0-346   [000] d...1..  6020.483514: at91_irq_r: 
(irq_forced_thread_fn+0x3c/0x8c <- at91_irq)
         <idle>-0     [000] d..h1..  6020.483781: irq_handler_entry: 
irq=30 name=can0
         <idle>-0     [000] d..h1..  6020.483799: irq_handler_exit: 
irq=30 ret=handled
    irq/30-can0-346   [000] d..h...  6020.483891: irq_handler_entry: 
irq=16 name=tc_clkevt
    irq/30-can0-346   [000] d..h...  6020.483956: softirq_raise: vec=1 
[action=TIMER]
    irq/30-can0-346   [000] d..h...  6020.484006: irq_handler_exit: 
irq=16 ret=handled
    irq/30-can0-346   [000] d......  6020.484048: at91_irq: 
(at91_irq+0x0/0xab0)
    irq/30-can0-346   [000] d...1..  6020.484169: at91_irq_r: 
(irq_forced_thread_fn+0x3c/0x8c <- at91_irq)
    irq/30-can0-346   [000] .......  6020.484207: softirq_entry: vec=1 
[action=TIMER]
    irq/30-can0-346   [000] d..h1..  6020.484254: irq_handler_entry: 
irq=30 name=can0
    irq/30-can0-346   [000] d..h1..  6020.484267: irq_handler_exit: 
irq=30 ret=handled
    irq/30-can0-346   [000] d......  6020.484493: softirq_raise: vec=3 
[action=NET_RX]
    irq/30-can0-346   [000] .......  6020.485282: softirq_exit: vec=1 
[action=TIMER]
    irq/30-can0-346   [000] .......  6020.485306: softirq_entry: vec=3 
[action=NET_RX]
    irq/30-can0-346   [000] d..h...  6020.485607: irq_handler_entry: 
irq=16 name=tc_clkevt
    irq/30-can0-346   [000] d..h...  6020.485673: irq_handler_entry: 
irq=17 name=at91_rtc
    irq/30-can0-346   [000] d..h...  6020.485685: irq_handler_exit: 
irq=17 ret=handled
    irq/30-can0-346   [000] d..h...  6020.485706: irq_handler_entry: 
irq=17 name=atmel_usart_serial.0.auto
    irq/30-can0-346   [000] d..h...  6020.485713: irq_handler_exit: 
irq=17 ret=handled
    irq/30-can0-346   [000] d..h...  6020.485754: irq_handler_exit: 
irq=16 ret=handled
    irq/30-can0-346   [000] d..h...  6020.486008: irq_handler_entry: 
irq=29 name=eth0
    irq/30-can0-346   [000] d..h...  6020.486027: irq_handler_exit: 
irq=29 ret=handled
    irq/30-can0-346   [000] .......  6020.486571: softirq_exit: vec=3 
[action=NET_RX]
    irq/30-can0-346   [000] d......  6020.486626: at91_irq: 
(at91_irq+0x0/0xab0)
    irq/30-can0-346   [000] d......  6020.486788: at91_rx_overflow_err: 
(at91_irq+0x200/0xab0)