Priority of serial driver

Priority of serial driver

[Cliff Brake]

I have a system where I am doing the following on a PXA270 ARM system:

An applications is sending and receiving a packet on a serial port
every 40ms.  With 2.6.20, I simply enabled kernel preemption and set
the priority of my application to real-time, and it worked great, no
matter what the system load.

With 2.6.23-rc6, I have enabled CONFIG_HIGH_RES_TIMERS &
CONFIG_PREEMPT.  Scheduling in the real-time application is rock solid
and looking at timing with a scope, and instrumenting the loop in the
application indicates the 40ms is very solid.  However, it seems the
receive data is getting stuck in the serial driver for relatively long
periods of time (>= 40ms), which is a problem in this system.

Is there anything that would have changed between 2.6.20 and
2.6.23-rc6 that would explain this?  What priority do pieces of the
serial driver run at?  Is there any way to make the serial driver run
at high priority so that data is processed in a timely manner?

Thanks,
Cliff

-- 
=======================
Cliff Brake
http://bec-systems.com

Re: Priority of serial driver

[Arnaldo Carvalho de Melo]

Em Sat, Sep 22, 2007 at 02:22:00PM -0400, Cliff Brake escreveu:
> I have a system where I am doing the following on a PXA270 ARM system:
> 
> An applications is sending and receiving a packet on a serial port
> every 40ms.  With 2.6.20, I simply enabled kernel preemption and set
> the priority of my application to real-time, and it worked great, no
> matter what the system load.

You didn't made it clear what exactly is the kernel version you are
using. You mention 2.6.20, but is this with or without the PREEMPT_RT
patch?
 
> With 2.6.23-rc6, I have enabled CONFIG_HIGH_RES_TIMERS &
> CONFIG_PREEMPT.  Scheduling in the real-time application is rock solid
> and looking at timing with a scope, and instrumenting the loop in the
> application indicates the 40ms is very solid.  However, it seems the
> receive data is getting stuck in the serial driver for relatively long
> periods of time (>= 40ms), which is a problem in this system.
> 
> Is there anything that would have changed between 2.6.20 and
> 2.6.23-rc6 that would explain this?  What priority do pieces of the
> serial driver run at?  Is there any way to make the serial driver run
> at high priority so that data is processed in a timely manner?

Look at the IRQ-<N> kernel thread that services the serial port hardware
interrupt, use chrt to bump its priority and see if it helps.

To see which is the interrupt:

[root@tonchinha ~]# dmesg | grep ttyS0
Kernel command line: root=/dev/hda1 console=ttyS0,115200 console=tty0
ignore_loglevel
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:02: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A

Now check what is the default rt priority for its hard interrupt kernel
thread handler, "IRQ-4":

[root@tonchinha ~]# ps -C IRQ-4 -To pid,rtprio,cmd
PID   RTPRIO CMD
1721  50     [IRQ-4]
[root@tonchinha ~]# 

Use the `chrt' utility to bump the priority to, say, 75:

[root@tonchinha ~]# chrt -p 75 1721

Check again:

[root@tonchinha ~]# ps -C IRQ-4 -To pid,rtprio,cmd
PID   RTPRIO CMD
1721  75     [IRQ-4]

- Arnaldo

Re: Priority of serial driver

[Cliff Brake]

On 9/22/07, Arnaldo Carvalho de Melo <acme@ghostprotocols.net> wrote:
> Em Sat, Sep 22, 2007 at 02:22:00PM -0400, Cliff Brake escreveu:
> > I have a system where I am doing the following on a PXA270 ARM system:
> >
> > An applications is sending and receiving a packet on a serial port
> > every 40ms.  With 2.6.20, I simply enabled kernel preemption and set
> > the priority of my application to real-time, and it worked great, no
> > matter what the system load.
>
> You didn't made it clear what exactly is the kernel version you are
> using. You mention 2.6.20, but is this with or without the PREEMPT_RT
> patch?

without the PREEMPT_RT patch -- just using the mainline PREEMPT support.

> > With 2.6.23-rc6, I have enabled CONFIG_HIGH_RES_TIMERS &
> > CONFIG_PREEMPT.  Scheduling in the real-time application is rock solid
> > and looking at timing with a scope, and instrumenting the loop in the
> > application indicates the 40ms is very solid.  However, it seems the
> > receive data is getting stuck in the serial driver for relatively long
> > periods of time (>= 40ms), which is a problem in this system.
> >
> > Is there anything that would have changed between 2.6.20 and
> > 2.6.23-rc6 that would explain this?  What priority do pieces of the
> > serial driver run at?  Is there any way to make the serial driver run
> > at high priority so that data is processed in a timely manner?
>
> Look at the IRQ-<N> kernel thread that services the serial port hardware
> interrupt, use chrt to bump its priority and see if it helps.

I don't see any IRQ-<N> processes on my system -- does this require
the RT patch?  Perhaps this is different on ARM/x86?

Thanks,
Cliff

root@compulab-pxa270:~$ ps -A
  PID TTY          TIME CMD
    1 ?        00:00:02 init
    2 ?        00:00:00 kthreadd
    3 ?        00:00:00 ksoftirqd/0
    4 ?        00:00:03 events/0
    5 ?        00:00:00 khelper
   46 ?        00:00:00 kblockd/0
   56 ?        00:00:00 khubd
   59 ?        00:00:00 kseriod
   82 ?        00:00:00 pdflush
   83 ?        00:00:00 pdflush
   84 ?        00:00:00 kswapd0
   85 ?        00:00:00 aio/0
  194 ?        00:00:00 mtdblockd
  240 ?        00:00:00 kpsmoused
  397 ?        00:00:00 udevd
 1221 ?        00:00:00 power
 1238 ?        00:00:15 jffs2_gcd_mtd4
 1271 ?        00:00:00 kmmcd
 1278 ?        00:00:00 pegasus
 1288 ?        00:00:01 mmcqd
 1433 ?        00:00:00 kjournald
 1497 ?        00:00:00 portmap
 1551 ?        00:00:00 dropbear
 1565 ?        00:00:00 dbus-daemon
 1571 ?        00:00:00 syslogd
 1573 ?        00:00:00 klogd
 1582 ?        00:00:00 avahi-daemon
 1630 tty2     00:02:55 Xfbdev
 1633 ttyS0    00:11:23 svs
 1649 tty1     00:00:00 sh
 1668 ttyS0    00:00:00 svs
 1711 ?        00:00:00 udhcpc
 1714 ?        00:00:00 dropbear
 1715 pts/0    00:00:00 sh
 1725 pts/0    00:00:00 ps
root@compulab-pxa270:~$

Re: Priority of serial driver

[Cliff Brake]

On 9/24/07, Cliff Brake <cliff.brake@gmail.com> wrote:
> On 9/22/07, Arnaldo Carvalho de Melo <acme@ghostprotocols.net> wrote:
> > Look at the IRQ-<N> kernel thread that services the serial port hardware
> > interrupt, use chrt to bump its priority and see if it helps.
>
> I don't see any IRQ-<N> processes on my system -- does this require
> the RT patch?  Perhaps this is different on ARM/x86?

I've applied the RT patch which gives me the IRQ-N processes -- trying now ...

Cliff

Re: Priority of serial driver

[Cliff Brake]

On 9/24/07, Cliff Brake <cliff.brake@gmail.com> wrote:
> On 9/24/07, Cliff Brake <cliff.brake@gmail.com> wrote:
> > On 9/22/07, Arnaldo Carvalho de Melo <acme@ghostprotocols.net> wrote:
> > > Look at the IRQ-<N> kernel thread that services the serial port hardware
> > > interrupt, use chrt to bump its priority and see if it helps.
> >
> > I don't see any IRQ-<N> processes on my system -- does this require
> > the RT patch?  Perhaps this is different on ARM/x86?
>
> I've applied the RT patch which gives me the IRQ-N processes -- trying now ...

RT patch (2.6.23-rc4-rt1) fixed the problem without any additional
priority adjustments, so that is obviously the solution.  I guess I
should know by now that RT with Linux is still hit and miss without
the RT patch.

Thanks,
Cliff

-- 
=======================
Cliff Brake
http://bec-systems.com

Re: Priority of serial driver

[Esben Nielsen]

On Mon, 24 Sep 2007, Cliff Brake wrote:

> On 9/24/07, Cliff Brake <cliff.brake@gmail.com> wrote:
>> On 9/24/07, Cliff Brake <cliff.brake@gmail.com> wrote:
>>> On 9/22/07, Arnaldo Carvalho de Melo <acme@ghostprotocols.net> wrote:
>>>> Look at the IRQ-<N> kernel thread that services the serial port hardware
>>>> interrupt, use chrt to bump its priority and see if it helps.
>>>
>>> I don't see any IRQ-<N> processes on my system -- does this require
>>> the RT patch?  Perhaps this is different on ARM/x86?
>>
>> I've applied the RT patch which gives me the IRQ-N processes -- trying now ...
>
> RT patch (2.6.23-rc4-rt1) fixed the problem without any additional
> priority adjustments, so that is obviously the solution.  I guess I
> should know by now that RT with Linux is still hit and miss without
> the RT patch.

Yes, that is the point of the -RT patch. Without the patch there are 
unbounded code paths running with preemption off. Your maximum jitter thus 
highly depends on the complete workload of your system. With the -RT patch 
applied there are (almost) no unbounded code paths running with preemption 
off. The maximum jitter of a given task will thus only depend on the 
workload of the tasks running at the same or higher priority. That is much more 
controlable.

Esben

>
> Thanks,
> Cliff
>
> -- 
> =======================
> Cliff Brake
> http://bec-systems.com
> -
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>