Re: Need help to measure and tune the latency in Linux RT

Re: Need help to measure and tune the latency in Linux RT

[Jason Cooper]

Ashoka,

linux-embedded is pretty quiet.  I'm not even sure why I'm still
subscribed to it.  You'll have better luck with your question by asking
on linux-arm-kernel (added to the cc).  I believe there is also a
linux-rt-users mailinglist [1].  A few comments below:

On Tue, Apr 30, 2013 at 01:57:21PM +0530, Ashoka K wrote:
> Hi,
> 
> I have an embedded system running 2.6.33 cross compiled for ARM OMAP
> with 1 GHZ uni-processor system.

Wow, this is ancient.  Is this a vanilla kernel?  If it's a vendor
kernel, you'll need to ask the vendor.

> The application has many threads and processes (around 15 total). In
> that 1 thread is time critical and must run every interval confiurable
> as 2, 4 or 8 milli sec etc. There is another thread which transfers
> Images to a FTP or to HMI etc, whoever requested the image.
> With this image transfer enabled my critical process misses the
> deadline and timesout. This critical thread is at RR priority 50. I
> used a HR timer thread with RR pri 55 to wakeup the critical thread
> every 1 milli sec to do the job.
> If image transfer is enabled i see that real time thread misses deadlines.
> 
> All traffic is flowing on common ethernet connection. Does it affect
> in any way ?
> 
> How to find out where the kernel is waiting or delaying to schedule
> the critical thread even though it is at higher RR priority. How to
> measure the latency ?
> 
> I am trying to use Oprofile tool, but got some error in cross
> compiling. Doesn't these tools add their own latency to the original
> problem ?
> 
> Any tools ? which one is better. Please provide your input on this.

I'm not the guy to answer this ;-)  I'll just forward to the appropriate
lists.

hth,

Jason.

[1] http://vger.kernel.org/vger-lists.html#linux-rt-users
--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Need help to measure and tune the latency in Linux RT

[Stanislav Meduna]

>> This critical thread is at RR priority 50.

Try to put it higher than the irq thread handling your HD hardware.
I've even encountered hardware where I had to put the ATA thread
in the SHED_OTHER domain - see [1] for the thread.

>> I used a HR timer thread with RR pri 55 to wakeup the critical thread
>> every 1 milli sec to do the job.

Which prio is the ksoftirqd running at? The HRTIMER interrupts are
routed through it so it also has to be high enough.

I got burned with this myself [2]

Regards
                                     Stano

[1] http://www.spinics.net/lists/linux-rt-users/msg08711.html
[2] http://www.spinics.net/lists/linux-rt-users/msg08745.html

Re: Need help to measure and tune the latency in Linux RT

[Ashoka K]

Thanks for all the replies.
I am stuck with this issue for more tan a month. Explained this in
more detail at:

http://stackoverflow.com/questions/15805231/need-to-improve-the-linux-performance-for-embedded-system

But with no response from anyone.

I use PREEMPT patch for 2.6.33 vanilla kernel.

I have pasted all the processes running on my system:

  PID USER       VSZ STAT COMMAND
    1 root      3192 S    init
    2 root         0 SW   [kthreadd]
    3 root         0 SW   [ksoftirqd/0]
    4 root         0 SW   [events/0]
    5 root         0 SW   [khelper]
    8 root         0 SW   [async/mgr]
   84 root         0 SW   [sync_supers]
   86 root         0 SW   [bdi-default]
   88 root         0 SW   [kblockd/0]
   91 root         0 SW   [omap2_mcspi]
  116 root         0 SW   [rpciod/0]
  123 root         0 SW   [kswapd0]
  124 root         0 SW   [aio/0]
  125 root         0 SW   [nfsiod]
  126 root         0 SW<  [kslowd001]
  127 root         0 SW<  [kslowd000]
  128 root         0 SW   [crypto/0]
  267 root         0 SW   [mtdblockd]
  351 root         0 SWN  [jffs2_gcd_mtd8]
  359 root      3192 S    /sbin/syslogd
  361 root      3192 S    /usr/sbin/telnetd
  364 root      1996 S    vsftpd
  410 root     43844 S   my_appl
  416 root     56412 S <  my_appl
  417 root     84804 S    my_appl
  418 root     43844 S    my_appl
  419 root     84824 S    my_appl
  430 root     46552 S <  my_appl

The process my_appl is my application processes.

the ksoftirq is running with SCHED_OTHER.

$ chrt -p 3
pid 3's current scheduling policy: SCHED_OTHER
pid 3's current scheduling priority: 0

I will test by changing ksoftirq priority to see how it works.

Regards
Ashoka.  K


On Tue, Apr 30, 2013 at 5:43 PM, Stanislav Meduna <stano@meduna.org> wrote:
>>> This critical thread is at RR priority 50.
>
> Try to put it higher than the irq thread handling your HD hardware.
> I've even encountered hardware where I had to put the ATA thread
> in the SHED_OTHER domain - see [1] for the thread.
>
>>> I used a HR timer thread with RR pri 55 to wakeup the critical thread
>>> every 1 milli sec to do the job.
>
> Which prio is the ksoftirqd running at? The HRTIMER interrupts are
> routed through it so it also has to be high enough.
>
> I got burned with this myself [2]
>
> Regards
>                                      Stano
>
> [1] http://www.spinics.net/lists/linux-rt-users/msg08711.html
> [2] http://www.spinics.net/lists/linux-rt-users/msg08745.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs