Re: kernel 2.6.33.7-rt29 problem

[gowrishankar <gowrishankar.m@linux.vnet.ibm.com>]

On Friday 15 October 2010 01:13 AM, Primus Mutasingwa wrote:
> Hello,
>
> I am using a linux kernel 2.6.33.7+preempt rt patch (patch-2.6.33.7-rt29)
> My system is running a user-space application that is made of multiple threads
> with varying priorities.
> This app is also making read/write calls into a kernel space driver that is
> running at priority 99 (very high priority)
> These are priorities fo some of the items running in the system
>       Threads in the app (10 ... 98, FIFO) 
>       Serial console isr (99, FIFO)
>       Driver used by app (99, FIFO)
> Different threads fail at different times (even ones with priority 50 or 98)
>
> System randomly fails after running for anywhere from 1 to 20 minutes.
>    

Could be due to cpu stall detected by rcu which will further request for
NMI to be sent.. your app may block rcu system threads which if fails to
extend the grace time, rcu will assume cpu stalled.. Any such info you saw
in /var/log/messages ? Anyhow, please reduce your threads prio atleast to
80 to give cpu chances to system priority threads.

Thanks,
Gowrishankar
> When it fails, these are the symptoms observed.
> 1. Running function/thread gets preempted and never gets time to run after that.
>     Console locks up, can no longer run ethernet traffic.
>     It is possible to get some console use by changing the scheduler
>     throttling threshold.   Usually it is at 95% and we change it to about 60%.
>   
>     /* from the console  */
>     echo 600000>  /proc/sys/kernel/sched_rt_runtime_us
> 2. When we get console use:
>     Displaying processor status (ps) shows that this thread is running even
> though we
>
>     never returned to the function
>     Using a top utility , it is reported that that particular thread consumes
>     about 95% of the CPU resources.
> 3. Using a JTAG based debugger
>       -- the stack was verified to be ok.
>       -- If we set a breakpoint in the function that fails we never hit the
> breakpoint (we expected function to return after preemption)
>
> I am looking for suggestions on how to resolve this problem. Why doesnt the
> thread run ?
> Why is the system locked up ?
> Is there any tools included in the kernel that can allow me to debug this ?
> I am not aware of an available lttng package available for this kernel version.
>
> Thank you,
>
> P. Mutasingwa
>
> Included is the kernel configuration file (omap3_beagle_defconfig) used in my
> setup
> Sent Separately
>
>
>
>