Re: Query regarding 2.6.335 RT and Non-RT performance

Re: Query regarding 2.6.335 RT and Non-RT performance

[Manikandan Ramachandran]

:
>
> Hello All,
>
>     I created a very simple program which has higher priority than normal tasks and runs a tight loop. Under same test environment I ran this program on both non-rt and rt 2.6.33.5 kernel.  To my suprise I see that performance of non-RT kernel is better than RT. non-RT kernel took 3 sec and 366156 usec while RT kernel took about 3 sec and 418011 usec.Can someone please explain why the performance of non-rt kernel is better than rt kernel? From the face of the test result, I feel RT has more overhead,Is there any configuration that I could do to bring down the overhead?
>
> Processor:
> ----------------
> processor       : 0
> cpu             : 7448
> clock           : 996.000000MHz
> revision        : 2.2 (pvr 8004 0202)
> bogomips        : 83.10
> processor       : 1
> cpu             : 7448
> clock           : 996.000000MHz
> revision        : 2.2 (pvr 8004 0202)
> bogomips        : 83.10
>
> CFS optimization:
> --------------------------
> # cat /proc/sys/kernel/sched_rt_runtime_us
> 1000000
> # cat /proc/sys/kernel/sched_rt_period_us
> 1000000
> # cat /proc/sys/kernel/sched_compat_yield
> 1
>
> Test Program:
> ---------------------
>
> main()
> {
>
>     int sched_rr_min,sched_rr_max;
>     struct sched_param scheduling_parameters;
>     struct timeval tv,late_tv;
>     suseconds_t usec_diff,avg_usec = 0;
>     time_t sec_diff, avg_sec = 0;
>     int i;
>     long count = 1;
>
>     sched_rr_min = sched_get_priority_min(SCHED_RR);
>     sched_rr_max = sched_get_priority_max(SCHED_RR);
>     scheduling_parameters.sched_priority = sched_rr_min+4;
>     sched_setscheduler(0, SCHED_RR, &scheduling_parameters);// Run the process with the given priority
>
>     for(i = 0 ; i < 150 ; i++) {
>        gettimeofday(&tv, NULL);
>        while(count > 0){
>         //printf(".");
>         count++;
>        }
>        gettimeofday(&late_tv, NULL);
>        count = 1;
>        sec_diff = (late_tv.tv_sec - tv.tv_sec);
>        avg_sec += sec_diff;
>        usec_diff = ( (late_tv.tv_usec > tv.tv_usec) ? (late_tv.tv_usec - tv.tv_usec) : ( tv.tv_usec - late_tv.tv_usec));
>        avg_usec += usec_diff;
>        printf("Iteration #%d sec %x usec %x\n",i,(sec_diff),(usec_diff));
>     }
>        printf("Average of #%d sec %x usec %x\n",i,(avg_sec/i),(avg_usec)/i);
> }
>
> Partial Result of non-rt kernel:
> -------------------------------------------
>
> Iteration #140 sec 3 usec 3aef8
> Iteration #141 sec 3 usec 3aefe
> Iteration #142 sec 3 usec 3aee4
> Iteration #143 sec 4 usec b935b  [Why there is this periodic bump ??] [Scheduler at work??]
> Iteration #144 sec 3 usec 3aef2
> Iteration #145 sec 3 usec 3aef0
> Iteration #146 sec 3 usec 3aef4
> Iteration #147 sec 4 usec b934b
> Iteration #148 sec 3 usec 3aeed
> Iteration #149 sec 3 usec 3aef9
>
> Partial Result of rt kernel:
> -------------------------------------------
> Iteration #135 sec 3 usec 47328
> Iteration #136 sec 4 usec ac4fd
> Iteration #137 sec 3 usec 48b0b
> Iteration #138 sec 3 usec 4738c
> Iteration #139 sec 4 usec ac4d5
> Iteration #140 sec 3 usec 483cb
> Iteration #141 sec 3 usec 48500
> Iteration #142 sec 4 usec acc49
> Iteration #143 sec 3 usec 47c1f
> Iteration #144 sec 3 usec 478c2
> Iteration #145 sec 3 usec 47e48
> Iteration #146 sec 4 usec ac9b5
> Iteration #147 sec 3 usec 48de4
> Iteration #148 sec 3 usec 46fbe
> Iteration #149 sec 4 usec ac52e
> Average of #150 sec 3 usec 660db
>
> Thanks,
> Mani
>
>
>
>


--
Thanks,
Manik

Think twice about a tree before you take a printout
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Query regarding 2.6.335 RT and Non-RT performance

[Mark Knecht]

On Mon, Aug 16, 2010 at 11:07 PM, Manikandan Ramachandran
<crmanik@gmail.com> wrote:
> :
>>
>> Hello All,
>>
>>     I created a very simple program which has higher priority than normal tasks and runs a tight loop. Under same test environment I ran this program on both non-rt and rt 2.6.33.5 kernel.  To my suprise I see that performance of non-RT kernel is better than RT. non-RT kernel took 3 sec and 366156 usec while RT kernel took about 3 sec and 418011 usec.Can someone please explain why the performance of non-rt kernel is better than rt kernel? From the face of the test result, I feel RT has more overhead,Is there any configuration that I could do to bring down the overhead?
>>

I'm not a programmer so I'll leave it to someone else to comment on
the code. (If at all.)

In my mind the purpose of the rt-kernel is not to 'go faster' but
rather to manage the __latency__ on high priority events. I use
rt-sources for my real-time audio system. I need low latency to my
sound cards. With rt-sources I can set the priority of sound cards
higher than other parts of the hardware system, and with the priority
higher when they need to send or receive audio data it happens in a
known amount of time. Without rt-sources they may want access to the
system but a disk drive or the network can get in the way and cause
delays.

If your program is measuring time around a loop getting time of day
then that's not likely a good measurement. If you measure how much the
interrupt latency changes when a sound card interrupts and asks for
service then that's probably more of where you'll see the value.

Again, not a programmer.

Cheers,
Mark
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Query regarding 2.6.335 RT and Non-RT performance

[David Sommerseth]

On 17/08/10 08:07, Manikandan Ramachandran wrote:
> :
>>
>> Hello All,
>>
>>     I created a very simple program which has higher priority than normal tasks and runs a tight loop. Under same test environment I ran this program on both non-rt and rt 2.6.33.5 kernel.  To my suprise I see that performance of non-RT kernel is better than RT. non-RT kernel took 3 sec and 366156 usec while RT kernel took about 3 sec and 418011 usec.Can someone please explain why the performance of non-rt kernel is better than rt kernel? From the face of the test result, I feel RT has more overhead,Is there any configuration that I could do to bring down the overhead?
>>
>> Processor:
>> ----------------
>> processor       : 0
>> cpu             : 7448
>> clock           : 996.000000MHz
>> revision        : 2.2 (pvr 8004 0202)
>> bogomips        : 83.10
>> processor       : 1
>> cpu             : 7448
>> clock           : 996.000000MHz
>> revision        : 2.2 (pvr 8004 0202)
>> bogomips        : 83.10
>>
>> CFS optimization:
>> --------------------------
>> # cat /proc/sys/kernel/sched_rt_runtime_us
>> 1000000
>> # cat /proc/sys/kernel/sched_rt_period_us
>> 1000000
>> # cat /proc/sys/kernel/sched_compat_yield
>> 1
>>
>> Test Program:
>> ---------------------
>>
>> main()
>> {
>>
>>     int sched_rr_min,sched_rr_max;
>>     struct sched_param scheduling_parameters;
>>     struct timeval tv,late_tv;
>>     suseconds_t usec_diff,avg_usec = 0;
>>     time_t sec_diff, avg_sec = 0;
>>     int i;
>>     long count = 1;
>>
>>     sched_rr_min = sched_get_priority_min(SCHED_RR);
>>     sched_rr_max = sched_get_priority_max(SCHED_RR);
>>     scheduling_parameters.sched_priority = sched_rr_min+4;
>>     sched_setscheduler(0, SCHED_RR, &scheduling_parameters);// Run the process with the given priority
>>
>>     for(i = 0 ; i < 150 ; i++) {
>>        gettimeofday(&tv, NULL);
>>        while(count > 0){
>>         //printf(".");
>>         count++;
>>        }
>>        gettimeofday(&late_tv, NULL);
>>        count = 1;
>>        sec_diff = (late_tv.tv_sec - tv.tv_sec);
>>        avg_sec += sec_diff;
>>        usec_diff = ( (late_tv.tv_usec > tv.tv_usec) ? (late_tv.tv_usec - tv.tv_usec) : ( tv.tv_usec - late_tv.tv_usec));
>>        avg_usec += usec_diff;
>>        printf("Iteration #%d sec %x usec %x\n",i,(sec_diff),(usec_diff));
>>     }
>>        printf("Average of #%d sec %x usec %x\n",i,(avg_sec/i),(avg_usec)/i);
>> }

It seems to me that this test program only tries to measure speed, which
is not something RT kernels will not give you.  As Mark Knecht correctly
pointed out, RT kernels care about latency.  Non-RT kernels are
sometimes referred to as Real-Fast kernels.

Your program will run before all SCHED_OTHER programs.  But again, all
SCHED_FIFO/SCHED_RR tasks which got a higher priority (like many kernel
threads), will preempt your program, which slows it down.  Having that
said, this do not mean that running your program with priority 99 is a
clever thing, in fact, that can block the kernel to really do more
needed stuff, like to take care of I/O traffic, etc, etc.

But to really measure what the RT kernel can give you, try to run
cyclictest in the rt-tests tree [1].  That program will measure your
system latency.  Btw. to really get a good overview over you latency,
run it over a longer time (>several hours).

You can also have peek at a tool call rteval [2] which will run
cyclictest together with some loads and summarise the performance for
you.  During an rteval run, your box will be heavy loaded - but if you
get max latencies < 100us after some hours of test run, consider that
pretty good results.

In general, don't use RT if you want to do things quick.  RT will be
slower than non-RT kernels.  Try building a kernel on a non-RT kernel
and then on a RT kernel, and you'll feel it very well.

But if you care about latency (a program which needs to react within a
certain time scope after it received some work to do), then the RT
kernel will serve you better.


kind regards,

David Sommerseth


[1]
<http://git.kernel.org/?p=linux/kernel/git/clrkwllms/rt-tests.git;a=summary>
[2]
<http://git.kernel.org/?p=linux/kernel/git/clrkwllms/rteval.git;a=summary>

Re: Query regarding 2.6.335 RT and Non-RT performance

[Manikandan Ramachandran]

On Tue, Aug 17, 2010 at 10:19 AM, David Sommerseth <davids@redhat.com> wrote:
> On 17/08/10 08:07, Manikandan Ramachandran wrote:
>> :
>>>
>>> Hello All,
>>>
>>>     I created a very simple program which has higher priority than normal tasks and runs a tight loop. Under same test environment I ran this program on both non-rt and rt 2.6.33.5 kernel.  To my suprise I see that performance of non-RT kernel is better than RT. non-RT kernel took 3 sec and 366156 usec while RT kernel took about 3 sec and 418011 usec.Can someone please explain why the performance of non-rt kernel is better than rt kernel? From the face of the test result, I feel RT has more overhead,Is there any configuration that I could do to bring down the overhead?
>>>
>>> Processor:
>>> ----------------
>>> processor       : 0
>>> cpu             : 7448
>>> clock           : 996.000000MHz
>>> revision        : 2.2 (pvr 8004 0202)
>>> bogomips        : 83.10
>>> processor       : 1
>>> cpu             : 7448
>>> clock           : 996.000000MHz
>>> revision        : 2.2 (pvr 8004 0202)
>>> bogomips        : 83.10
>>>
>>> CFS optimization:
>>> --------------------------
>>> # cat /proc/sys/kernel/sched_rt_runtime_us
>>> 1000000
>>> # cat /proc/sys/kernel/sched_rt_period_us
>>> 1000000
>>> # cat /proc/sys/kernel/sched_compat_yield
>>> 1
>>>
>>> Test Program:
>>> ---------------------
>>>
>>> main()
>>> {
>>>
>>>     int sched_rr_min,sched_rr_max;
>>>     struct sched_param scheduling_parameters;
>>>     struct timeval tv,late_tv;
>>>     suseconds_t usec_diff,avg_usec = 0;
>>>     time_t sec_diff, avg_sec = 0;
>>>     int i;
>>>     long count = 1;
>>>
>>>     sched_rr_min = sched_get_priority_min(SCHED_RR);
>>>     sched_rr_max = sched_get_priority_max(SCHED_RR);
>>>     scheduling_parameters.sched_priority = sched_rr_min+4;
>>>     sched_setscheduler(0, SCHED_RR, &scheduling_parameters);// Run the process with the given priority
>>>
>>>     for(i = 0 ; i < 150 ; i++) {
>>>        gettimeofday(&tv, NULL);
>>>        while(count > 0){
>>>         //printf(".");
>>>         count++;
>>>        }
>>>        gettimeofday(&late_tv, NULL);
>>>        count = 1;
>>>        sec_diff = (late_tv.tv_sec - tv.tv_sec);
>>>        avg_sec += sec_diff;
>>>        usec_diff = ( (late_tv.tv_usec > tv.tv_usec) ? (late_tv.tv_usec - tv.tv_usec) : ( tv.tv_usec - late_tv.tv_usec));
>>>        avg_usec += usec_diff;
>>>        printf("Iteration #%d sec %x usec %x\n",i,(sec_diff),(usec_diff));
>>>     }
>>>        printf("Average of #%d sec %x usec %x\n",i,(avg_sec/i),(avg_usec)/i);
>>> }
>
> It seems to me that this test program only tries to measure speed, which
> is not something RT kernels will not give you.  As Mark Knecht correctly
> pointed out, RT kernels care about latency.  Non-RT kernels are
> sometimes referred to as Real-Fast kernels.
>
> Your program will run before all SCHED_OTHER programs.  But again, all
> SCHED_FIFO/SCHED_RR tasks which got a higher priority (like many kernel
> threads), will preempt your program, which slows it down.  Having that
> said, this do not mean that running your program with priority 99 is a
> clever thing, in fact, that can block the kernel to really do more
> needed stuff, like to take care of I/O traffic, etc, etc.
>
> But to really measure what the RT kernel can give you, try to run
> cyclictest in the rt-tests tree [1].  That program will measure your
> system latency.  Btw. to really get a good overview over you latency,
> run it over a longer time (>several hours).
>
> You can also have peek at a tool call rteval [2] which will run
> cyclictest together with some loads and summarise the performance for
> you.  During an rteval run, your box will be heavy loaded - but if you
> get max latencies < 100us after some hours of test run, consider that
> pretty good results.
>
> In general, don't use RT if you want to do things quick.  RT will be
> slower than non-RT kernels.  Try building a kernel on a non-RT kernel
> and then on a RT kernel, and you'll feel it very well.
>
> But if you care about latency (a program which needs to react within a
> certain time scope after it received some work to do), then the RT
> kernel will serve you better.
>
>
> kind regards,
>
> David Sommerseth
>
>
> [1]
> <http://git.kernel.org/?p=linux/kernel/git/clrkwllms/rt-tests.git;a=summary>
> [2]
> <http://git.kernel.org/?p=linux/kernel/git/clrkwllms/rteval.git;a=summary>
>

Thanks David & Mark for your valuable input!

I'm not expecting RT kernel to be real fast. However I expect latency
of higher priority task to be less in RT kernel. I'll run the tests
that you have suggested and see if there is any inherent issue with my
system.

One more query, with RT patch is it possible for higher priority task
to preempt a lower priority IRQ thread [For eg, IDE] ? If so when does
that happen, during next timer interrupt [Assuming IRQ thread uses
whole schedule slot and doesn't yield between]?
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Query regarding 2.6.335 RT and Non-RT performance

[Nivedita Singhvi]

On 08/17/2010 11:24 AM, Manikandan Ramachandran wrote:

> Thanks David&  Mark for your valuable input!
>
> I'm not expecting RT kernel to be real fast. However I expect latency
> of higher priority task to be less in RT kernel. I'll run the tests
> that you have suggested and see if there is any inherent issue with my
> system.

Actually, a more accurate statement would be "I expect max latency
of higher priority task to be less in RT kernel."

This is true for max latency as reported by you (max, min):

Stock:	418011us	3s
RT:	366156us	3s

So while the min might reflect the best that the system can do, the
max does seem a tad high -- you do need to look into that.

> One more query, with RT patch is it possible for higher priority task
> to preempt a lower priority IRQ thread [For eg, IDE] ? If so when does
> that happen, during next timer interrupt [Assuming IRQ thread uses
> whole schedule slot and doesn't yield between]?

Yes, and when that higher priority task becomes runnable.

thanks,
Nivedita

Re: Query regarding 2.6.335 RT and Non-RT performance

[Manikandan Ramachandran]

On Tue, Aug 17, 2010 at 12:45 PM, Nivedita Singhvi <niv@us.ibm.com> wrote:
> On 08/17/2010 11:24 AM, Manikandan Ramachandran wrote:
>
>> Thanks David&  Mark for your valuable input!
>>
>> I'm not expecting RT kernel to be real fast. However I expect latency
>> of higher priority task to be less in RT kernel. I'll run the tests
>> that you have suggested and see if there is any inherent issue with my
>> system.
>
> Actually, a more accurate statement would be "I expect max latency
> of higher priority task to be less in RT kernel."
>
> This is true for max latency as reported by you (max, min):
>
> Stock:  418011us        3s
> RT:     366156us        3s
>
> So while the min might reflect the best that the system can do, the
> max does seem a tad high -- you do need to look into that.
>
>> One more query, with RT patch is it possible for higher priority task
>> to preempt a lower priority IRQ thread [For eg, IDE] ? If so when does
>> that happen, during next timer interrupt [Assuming IRQ thread uses
>> whole schedule slot and doesn't yield between]?
>
> Yes, and when that higher priority task becomes runnable.
>
> thanks,
> Nivedita
> --

I see a scenario where higher priority task is ready to run but lower
priority IRQ thread is hogging the cpu. Please see the timeline log
below:

GIVE ffff0c1c [Comment: High priority IRQ thread wakes up high
priority process at jiffy 0xffff0c1c]
**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c24##**ffff0c24**##ffff0c24##**ffff0c24**##ffff0c24##
[Comment: Low priority IDE[ide_intr] handler hogs CPU for almost 3 ms]
GOTffff0c26 [Comment: High priority task wakes up after 10 ms!!]

I fail to understand why scheduler failed to schedule high priority
task 10 times in a row. FYI, I have put IDE thread to SCHED_NORMAL
while high priority thread and task to SCHED_FIFO.

-- 
Thanks,
Manik
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Query regarding 2.6.335 RT and Non-RT performance

[Nivedita Singhvi]

On 08/17/2010 03:22 PM, Manikandan Ramachandran wrote:


> I see a scenario where higher priority task is ready to run but lower
> priority IRQ thread is hogging the cpu. Please see the timeline log
> below:
>
> GIVE ffff0c1c [Comment: High priority IRQ thread wakes up high
> priority process at jiffy 0xffff0c1c]
> **ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c24##**ffff0c24**##ffff0c24##**ffff0c24**##ffff0c24##
> [Comment: Low priority IDE[ide_intr] handler hogs CPU for almost 3 ms]
> GOTffff0c26 [Comment: High priority task wakes up after 10 ms!!]
>
> I fail to understand why scheduler failed to schedule high priority
> task 10 times in a row. FYI, I have put IDE thread to SCHED_NORMAL
> while high priority thread and task to SCHED_FIFO.

Not really sure from the above what's going on, what priorities
you are running at, etc.

You might want look at the output of "ps -eLo pid,rtprio,policy,comm"
to see if you can confirm your scheduler attributes.

thanks,
Nivedita

Re: Query regarding 2.6.335 RT and Non-RT performance

[Sven-Thorsten Dietrich]

On Thu, 2010-08-19 at 07:59 -0700, Nivedita Singhvi wrote:
> On 08/17/2010 03:22 PM, Manikandan Ramachandran wrote:
> 
> 
> > I see a scenario where higher priority task is ready to run but lower
> > priority IRQ thread is hogging the cpu. Please see the timeline log
> > below:
> >
> > GIVE ffff0c1c [Comment: High priority IRQ thread wakes up high
> > priority process at jiffy 0xffff0c1c]
> > **ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c24##**ffff0c24**##ffff0c24##**ffff0c24**##ffff0c24##
> > [Comment: Low priority IDE[ide_intr] handler hogs CPU for almost 3 ms]
> > GOTffff0c26 [Comment: High priority task wakes up after 10 ms!!]
> >
> > I fail to understand why scheduler failed to schedule high priority
> > task 10 times in a row. FYI, I have put IDE thread to SCHED_NORMAL
> > while high priority thread and task to SCHED_FIFO.
> 
> Not really sure from the above what's going on, what priorities
> you are running at, etc.
> 
> You might want look at the output of "ps -eLo pid,rtprio,policy,comm"
> to see if you can confirm your scheduler attributes.
> 

Also, use the tracing facilities, as its hard to understand your time
line and the source of it.

I think if your priorities are right, your kernel config is right, and
your measurement tool doesn't interfere, 10 ms latency is incredibly
unlikely.


> thanks,
> Nivedita
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Query regarding 2.6.335 RT and Non-RT performance

[Manikandan Ramachandran]

On Thu, Aug 19, 2010 at 9:06 AM, Sven-Thorsten Dietrich
<thebigcorporation@gmail.com> wrote:
> On Thu, 2010-08-19 at 07:59 -0700, Nivedita Singhvi wrote:
>> On 08/17/2010 03:22 PM, Manikandan Ramachandran wrote:
>>
>>
>> > I see a scenario where higher priority task is ready to run but lower
>> > priority IRQ thread is hogging the cpu. Please see the timeline log
>> > below:
>> >
>> > GIVE ffff0c1c [Comment: High priority IRQ thread wakes up high
>> > priority process at jiffy 0xffff0c1c]
>> > **ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c24##**ffff0c24**##ffff0c24##**ffff0c24**##ffff0c24##
>> > [Comment: Low priority IDE[ide_intr] handler hogs CPU for almost 3 ms]
>> > GOTffff0c26 [Comment: High priority task wakes up after 10 ms!!]
>> >
>> > I fail to understand why scheduler failed to schedule high priority
>> > task 10 times in a row. FYI, I have put IDE thread to SCHED_NORMAL
>> > while high priority thread and task to SCHED_FIFO.
>>
>> Not really sure from the above what's going on, what priorities
>> you are running at, etc.
>>
>> You might want look at the output of "ps -eLo pid,rtprio,policy,comm"
>> to see if you can confirm your scheduler attributes.
>>
>
> Also, use the tracing facilities, as its hard to understand your time
> line and the source of it.
>
> I think if your priorities are right, your kernel config is right, and
> your measurement tool doesn't interfere, 10 ms latency is incredibly
> unlikely.
>
>
>> thanks,
>> Nivedita
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
I see many tracing options. Any trace config in particular?


-- 
Thanks,
Manik
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Query regarding 2.6.335 RT and Non-RT performance

[Sven-Thorsten Dietrich]

  On 08/19/2010 11:15 AM, Manikandan Ramachandran wrote:
> On Thu, Aug 19, 2010 at 9:06 AM, Sven-Thorsten Dietrich
> <thebigcorporation@gmail.com>  wrote:
>> On Thu, 2010-08-19 at 07:59 -0700, Nivedita Singhvi wrote:
>>> On 08/17/2010 03:22 PM, Manikandan Ramachandran wrote:
>>>
>>>
>>>> I see a scenario where higher priority task is ready to run but lower
>>>> priority IRQ thread is hogging the cpu. Please see the timeline log
>>>> below:
>>>>
>>>> GIVE ffff0c1c [Comment: High priority IRQ thread wakes up high
>>>> priority process at jiffy 0xffff0c1c]
>>>> **ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c24##**ffff0c24**##ffff0c24##**ffff0c24**##ffff0c24##
>>>> [Comment: Low priority IDE[ide_intr] handler hogs CPU for almost 3 ms]
>>>> GOTffff0c26 [Comment: High priority task wakes up after 10 ms!!]
>>>>
>>>> I fail to understand why scheduler failed to schedule high priority
>>>> task 10 times in a row. FYI, I have put IDE thread to SCHED_NORMAL
>>>> while high priority thread and task to SCHED_FIFO.
>>> Not really sure from the above what's going on, what priorities
>>> you are running at, etc.
>>>
>>> You might want look at the output of "ps -eLo pid,rtprio,policy,comm"
>>> to see if you can confirm your scheduler attributes.
>>>
>> Also, use the tracing facilities, as its hard to understand your time
>> line and the source of it.
>>
>> I think if your priorities are right, your kernel config is right, and
>> your measurement tool doesn't interfere, 10 ms latency is incredibly
>> unlikely.
>>
>>
>>> thanks,
>>> Nivedita
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> I see many tracing options. Any trace config in particular?
>
>

Steve's latency tracing can generate a time-stamped stack dump of the 
Kernel code-path that blocks the RT task you are trying to run.

Re: Query regarding 2.6.335 RT and Non-RT performance

[gowrishankar]

On Wednesday 18 August 2010 03:52 AM, Manikandan Ramachandran wrote:
> I see a scenario where higher priority task is ready to run but lower
> priority IRQ thread is hogging the cpu. Please see the timeline log
> below:
>
>    
Can you try turning on RCU_CPU_STALL_DETECTION to get any trace
which might have hogged the cpu too long. Also make sure that
sirq-rcu is running at little more prio than your high prio task.

Thanks,
Gowri
> GIVE ffff0c1c [Comment: High priority IRQ thread wakes up high
> priority process at jiffy 0xffff0c1c]
> **ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c21##**ffff0c21**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c22**##ffff0c22##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c23##**ffff0c23**##ffff0c24##**ffff0c24**##ffff0c24##**ffff0c24**##ffff0c24##
> [Comment: Low priority IDE[ide_intr] handler hogs CPU for almost 3 ms]
> GOTffff0c26 [Comment: High priority task wakes up after 10 ms!!]
>
> I fail to understand why scheduler failed to schedule high priority
> task 10 times in a row. FYI, I have put IDE thread to SCHED_NORMAL
> while high priority thread and task to SCHED_FIFO.
>
>    

Re: Query regarding 2.6.335 RT and Non-RT performance

[Mark Knecht]

On Tue, Aug 17, 2010 at 11:24 AM, Manikandan Ramachandran
<crmanik@gmail.com> wrote:
<SNIP>
>
> Thanks David & Mark for your valuable input!
>
> I'm not expecting RT kernel to be real fast. However I expect latency
> of higher priority task to be less in RT kernel. I'll run the tests
> that you have suggested and see if there is any inherent issue with my
> system.
>

It is not that the delay of a high priority task is less in the rt
kernel. It's more that it meets a maximum latency number more often.
It's more _dependable_ in meeting a known latency value than the
standard kernel.

> One more query, with RT patch is it possible for higher priority task
> to preempt a lower priority IRQ thread [For eg, IDE] ? If so when does
> that happen, during next timer interrupt [Assuming IRQ thread uses
> whole schedule slot and doesn't yield between]?

Yes, it is possible for higher priorities to interrupt low priorities.
In a __properly__ functioning system higher priority should always
trump lower priority. The higher runs and then the lower picks up
where it left off.

In your question it seems to me you are assuming a device or it's
software is not 'real time compliant' which has certainly been the
case a few times over the years. If a lower priority device won't
yield - maybe it's in some tight loop that doesn't pay attention to
the kernel? - then that software needs to be fixed or the system won't
produce predictable latencies. Over the years this has become much
less an issue but I'm not sure anything can be guaranteed without
testing.

Again, I'm not a programmer but I've used these kernels for years so I
speak from sort of a practical user POV and not necessarily the most
accurate software technical POV. (I'm a hardware guy)