Need help to measure and tune the latency in Linux RT

Need help to measure and tune the latency in Linux RT

[Ashoka K]

Hi,

I have an embedded system running 2.6.33 cross compiled for ARM OMAP
with 1 GHZ uni-processor system.

The application has many threads and processes (around 15 total). In
that 1 thread is time critical and must run every interval confiurable
as 2, 4 or 8 milli sec etc. There is another thread which transfers
Images to a FTP or to HMI etc, whoever requested the image.
With this image transfer enabled my critical process misses the
deadline and timesout. This critical thread is at RR priority 50. I
used a HR timer thread with RR pri 55 to wakeup the critical thread
every 1 milli sec to do the job.
If image transfer is enabled i see that real time thread misses deadlines.

All traffic is flowing on common ethernet connection. Does it affect
in any way ?

How to find out where the kernel is waiting or delaying to schedule
the critical thread even though it is at higher RR priority. How to
measure the latency ?

I am trying to use Oprofile tool, but got some error in cross
compiling. Doesn't these tools add their own latency to the original
problem ?

Any tools ? which one is better. Please provide your input on this.


Regards
Ashoka.  K
--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Need help to measure and tune the latency in Linux RT

[Jason Cooper]

Ashoka,

linux-embedded is pretty quiet.  I'm not even sure why I'm still
subscribed to it.  You'll have better luck with your question by asking
on linux-arm-kernel (added to the cc).  I believe there is also a
linux-rt-users mailinglist [1].  A few comments below:

On Tue, Apr 30, 2013 at 01:57:21PM +0530, Ashoka K wrote:
> Hi,
> 
> I have an embedded system running 2.6.33 cross compiled for ARM OMAP
> with 1 GHZ uni-processor system.

Wow, this is ancient.  Is this a vanilla kernel?  If it's a vendor
kernel, you'll need to ask the vendor.

> The application has many threads and processes (around 15 total). In
> that 1 thread is time critical and must run every interval confiurable
> as 2, 4 or 8 milli sec etc. There is another thread which transfers
> Images to a FTP or to HMI etc, whoever requested the image.
> With this image transfer enabled my critical process misses the
> deadline and timesout. This critical thread is at RR priority 50. I
> used a HR timer thread with RR pri 55 to wakeup the critical thread
> every 1 milli sec to do the job.
> If image transfer is enabled i see that real time thread misses deadlines.
> 
> All traffic is flowing on common ethernet connection. Does it affect
> in any way ?
> 
> How to find out where the kernel is waiting or delaying to schedule
> the critical thread even though it is at higher RR priority. How to
> measure the latency ?
> 
> I am trying to use Oprofile tool, but got some error in cross
> compiling. Doesn't these tools add their own latency to the original
> problem ?
> 
> Any tools ? which one is better. Please provide your input on this.

I'm not the guy to answer this ;-)  I'll just forward to the appropriate
lists.

hth,

Jason.

[1] http://vger.kernel.org/vger-lists.html#linux-rt-users
--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Need help to measure and tune the latency in Linux RT

[ddegraff]

On 04/30/2013 03:27 AM, Ashoka K wrote:
> Hi,
>
> I have an embedded system running 2.6.33 cross compiled for ARM OMAP
> with 1 GHZ uni-processor system.

A project I am working on is using the same vintage kernel with RT 
patches applied, though on DaVinci (omap l138).

I, myself, however, have not dealt with any of that work, but I will ask 
the other developer if he has any suggestions.


>
> The application has many threads and processes (around 15 total). In
> that 1 thread is time critical and must run every interval confiurable
> as 2, 4 or 8 milli sec etc. There is another thread which transfers
> Images to a FTP or to HMI etc, whoever requested the image.
> With this image transfer enabled my critical process misses the
> deadline and timesout. This critical thread is at RR priority 50. I
> used a HR timer thread with RR pri 55 to wakeup the critical thread
> every 1 milli sec to do the job.
> If image transfer is enabled i see that real time thread misses deadlines.
>
> All traffic is flowing on common ethernet connection. Does it affect
> in any way ?
>
> How to find out where the kernel is waiting or delaying to schedule
> the critical thread even though it is at higher RR priority. How to
> measure the latency ?
>
> I am trying to use Oprofile tool, but got some error in cross
> compiling. Doesn't these tools add their own latency to the original
> problem ?
>
> Any tools ? which one is better. Please provide your input on this.
>
>
> Regards
> Ashoka.  K
> --
> To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Need help to measure and tune the latency in Linux RT

[Stanislav Meduna]

>> This critical thread is at RR priority 50.

Try to put it higher than the irq thread handling your HD hardware.
I've even encountered hardware where I had to put the ATA thread
in the SHED_OTHER domain - see [1] for the thread.

>> I used a HR timer thread with RR pri 55 to wakeup the critical thread
>> every 1 milli sec to do the job.

Which prio is the ksoftirqd running at? The HRTIMER interrupts are
routed through it so it also has to be high enough.

I got burned with this myself [2]

Regards
                                     Stano

[1] http://www.spinics.net/lists/linux-rt-users/msg08711.html
[2] http://www.spinics.net/lists/linux-rt-users/msg08745.html

Re: Need help to measure and tune the latency in Linux RT

[Ashoka K]

Thanks for all the replies.
I am stuck with this issue for more tan a month. Explained this in
more detail at:

http://stackoverflow.com/questions/15805231/need-to-improve-the-linux-performance-for-embedded-system

But with no response from anyone.

I use PREEMPT patch for 2.6.33 vanilla kernel.

I have pasted all the processes running on my system:

  PID USER       VSZ STAT COMMAND
    1 root      3192 S    init
    2 root         0 SW   [kthreadd]
    3 root         0 SW   [ksoftirqd/0]
    4 root         0 SW   [events/0]
    5 root         0 SW   [khelper]
    8 root         0 SW   [async/mgr]
   84 root         0 SW   [sync_supers]
   86 root         0 SW   [bdi-default]
   88 root         0 SW   [kblockd/0]
   91 root         0 SW   [omap2_mcspi]
  116 root         0 SW   [rpciod/0]
  123 root         0 SW   [kswapd0]
  124 root         0 SW   [aio/0]
  125 root         0 SW   [nfsiod]
  126 root         0 SW<  [kslowd001]
  127 root         0 SW<  [kslowd000]
  128 root         0 SW   [crypto/0]
  267 root         0 SW   [mtdblockd]
  351 root         0 SWN  [jffs2_gcd_mtd8]
  359 root      3192 S    /sbin/syslogd
  361 root      3192 S    /usr/sbin/telnetd
  364 root      1996 S    vsftpd
  410 root     43844 S   my_appl
  416 root     56412 S <  my_appl
  417 root     84804 S    my_appl
  418 root     43844 S    my_appl
  419 root     84824 S    my_appl
  430 root     46552 S <  my_appl

The process my_appl is my application processes.

the ksoftirq is running with SCHED_OTHER.

$ chrt -p 3
pid 3's current scheduling policy: SCHED_OTHER
pid 3's current scheduling priority: 0

I will test by changing ksoftirq priority to see how it works.

Regards
Ashoka.  K


On Tue, Apr 30, 2013 at 5:43 PM, Stanislav Meduna <stano@meduna.org> wrote:
>>> This critical thread is at RR priority 50.
>
> Try to put it higher than the irq thread handling your HD hardware.
> I've even encountered hardware where I had to put the ATA thread
> in the SHED_OTHER domain - see [1] for the thread.
>
>>> I used a HR timer thread with RR pri 55 to wakeup the critical thread
>>> every 1 milli sec to do the job.
>
> Which prio is the ksoftirqd running at? The HRTIMER interrupts are
> routed through it so it also has to be high enough.
>
> I got burned with this myself [2]
>
> Regards
>                                      Stano
>
> [1] http://www.spinics.net/lists/linux-rt-users/msg08711.html
> [2] http://www.spinics.net/lists/linux-rt-users/msg08745.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs

Re: Need help to measure and tune the latency in Linux RT

[Ashoka K]

Hello Android Techies and others,

I am still debugging this issue to understand the traces generated. I
am debugging at this level for first time, and now i am bit familiar
with traces.

To summarize again:

I have ARM OMAP embedded system, running vanilla Linux 2.6.33-29 cross
compiled with PREEMPT patch. There is a application, say Thread_1 (RR,
with highest priority) which need to send and receive data every 2ms,
or 4ms or 8 ms etc. (configurable). This data is small and atmost it
will be 1 ethernet frame (1500 bytes). So i expect this application to
execute for very short time but regularly without missing deadlines
(There is deviation allowed with watchdog timeout but in worst cases).
There is another process running (RR, lower pri than Thread_1) on same
system which transfers images on request from outside users/HMI. When
this is enabled and i see a frequesnt timeout for the time critical RT
thread_1. Even if this image transfer is not enabled then also i see
Kernel Oops after long time, say 1 day.
All data is flowing on ethernet.

I had increaed the KSOFTIRQD policy and priority to be more than my
time critical application , but it didn't help much. Now i am trying
to understand the debug traces generated to know where the latency is
coming from.

Any suggestions is greatly appreciated.

Regards
Ashoka.  K


On Mon, May 20, 2013 at 12:32 AM, Android Techies
<android.techies@gmail.com> wrote:
> Hello Ashoka,
> You can try to measure the hrtimer latency and see whether there is a delay
> in hrtimer itself. I will suggest to try and make Oprofile working. If you
> have already figured out a way please let me know as well as I need similar
> stuff.
>
>
>
> On Tue, Apr 30, 2013 at 1:27 AM, Ashoka K <ashok.vinu@gmail.com> wrote:
>>
>> Hi,
>>
>> I have an embedded system running 2.6.33 cross compiled for ARM OMAP
>> with 1 GHZ uni-processor system.
>>
>> The application has many threads and processes (around 15 total). In
>> that 1 thread is time critical and must run every interval confiurable
>> as 2, 4 or 8 milli sec etc. There is another thread which transfers
>> Images to a FTP or to HMI etc, whoever requested the image.
>> With this image transfer enabled my critical process misses the
>> deadline and timesout. This critical thread is at RR priority 50. I
>> used a HR timer thread with RR pri 55 to wakeup the critical thread
>> every 1 milli sec to do the job.
>> If image transfer is enabled i see that real time thread misses deadlines.
>>
>> All traffic is flowing on common ethernet connection. Does it affect
>> in any way ?
>>
>> How to find out where the kernel is waiting or delaying to schedule
>> the critical thread even though it is at higher RR priority. How to
>> measure the latency ?
>>
>> I am trying to use Oprofile tool, but got some error in cross
>> compiling. Doesn't these tools add their own latency to the original
>> problem ?
>>
>> Any tools ? which one is better. Please provide your input on this.
>>
>>
>> Regards
>> Ashoka.  K
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
>>
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at http://www.linux-learn.org/faqs
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-newbie" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.linux-learn.org/faqs