Re: Issue with SCHED_FIFO app

[Robert Hancock <hancockrwd@gmail.com>]

On 05/11/2010 08:46 PM, Xianghua Xiao wrote:
> On Sun, May 9, 2010 at 11:42 PM, Suresh Rajashekara
> <suresh.raj+linuxomap@gmail.com>  wrote:
>> Hi All,
>>
>> I had a couple of application (with real time priority SCHED_FIFO)
>> which were working fine on 2.6.16. They have started behaving
>> differently on 2.6.29.
>>
>> I will explain my problem briefly.
>>
>> Application A (my main application) is scheduled with SCHED_FIFO and priority 5.
>> Application B (watchdog application) is also scheduled with SCHED_FIFO
>> but with priority 54.
>>
>> A keeps putting the OMAP to sleep and wake up every 4 seconds and
>> again puts it to sleep.
>> B is supposed to be running every 1.25 seconds to kick watchdog, but
>> since A keeps OMAP in sleep for 4 seconds, it should run as soon as
>> OMAP wakes up.
>>
>> Since B is of a higher priority, its supposed to run whenever the OMAP
>> wakes up and then A should again put it back to sleep. This happens
>> perfectly on 2.6.16
>>
>> On 2.6.29, B fails to run when OMAP wakes up and before A puts it back
>> to sleep. B only runs if there is atleast 1.5 seconds of delay between
>> the awake-sleep cycle.
>>
>> On searching the internet, I figured out that CFS (completely fair
>> scheduler) was introduced in 2.6.23, which makes some changes to the
>> RT bandwidth (and many users started facing issues with they
>> applications with SCHED_FIFO). Somewhere on the web I found that
>> issuing
>>
>> echo -1>  /proc/sys/kernel/sched_rt_runtime_us
>>
>> should disable the changes which affects the RT bandwidth. It actually
>> did help to an extent in solving some other problem (not described
>> above. A's IOCTL call return was getting delayed), but this problem
>> still persists.
>>
>> Any pointers to where I should look for the solution.
>>
>> Is there a way I can revert back to the scheduler behavior as it was on 2.6.16?
>>
>> I have disabled CONFIG_GROUP_SCHED and also CONFIG_CGROUPS. I am using
>> 2.6.29 on an OMAP1 platform.
>>
>> Thanks in advance,
>> Suresh
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-omap" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> I have seen similar things while upgrading a 2.6.18 RT kernel to
> 2.6.33 RT, actually exactly when CFS was introduced we found
> performance issues, in that, our main application(a multi-thread
> SCHED_FIFO / SCHED_RR mixed) runs with much higher overhead under CFS.
> In 2.6.18RT, the cpu usage is close to 0% and on newer kernel with
> CFS, the cpu usage is 12% when the application runs idle(i.e. sleeping
> and waiting for input, WCHAN shows sched_timeout or futex_wait). When
> the main application runs with real load, cpu usage gets much worse
> with CFS.
>
> I tried various methods, including the one you described above, and
> made sure no sched_yield is used, etc, still the main application
> spends 6% cpu in user space and 6% in kernel space while at idle. I
> tried BFS schedule and it's actually better, about 8% in user space
> and 0.6% in kernel space while the application runs idle. Again with
> 2.6.18 RT it's nearly 0% cpu usage.

If it's using 6% of CPU in userspace, then it sounds to me like it's not 
really idle. Could be some kind of timing issue that the scheduler 
change exposes?