Re: Soft lock issue with 2.6.33.7-rt29

[Nathan Grennan <linux-rt-users@cygnusx-1.org>]

On 11/19/2010 10:46 AM, Darren Hart wrote:
> On 11/18/2010 02:11 PM, Nathan Grennan wrote:
>> On 11/17/2010 05:26 PM, Darren Hart wrote:
>>> On 11/17/2010 11:11 AM, Nathan Grennan wrote:
>>>> I have been working for weeks to get a stable rt kernel. I had been
>>>> focusing on 2.6.31.6-rt19. It is stable for about four days under 
>>>> stress
>>>> testing before it soft locks. I am using rt19 instead of rt21, because
>>>> rt19 seems to be more stable. The rtmutex issue that seems to still be
>>>> in rt29 is in rt21. I also had to backport the iptables fix to rt19.
>>>>
>>>> I just started looking at 2.6.33.7-rt29 again, since I can reproduce a
>>>> soft lock with it in 10-15 minutes. I have yet to get sysrq output for
>>>> rt19, since it takes four days. The soft lock with rt29 as far as I 
>>>> can
>>>> tell seems to relate to disk i/o.
>>>>
>>>> There are links to two logs of rt29 from a serial console below. They
>>>> include sysrq output like "Show Blocked State" and "Show State". The
>>>> level7 file is with nfsd enable, and level9 is with it disable. So 
>>>> nfsd
>>>> doesn't seem to be the issue.
>>>>
>>>> If any other debugging information is useful or needed, just say the
>>>> word.
>>>
>>> A reproducible test-case is always the first thing we ask for :-) What
>>> is your stress test?
>>
>> I have been able to boil it down the script below. If I just run yes it
>> is fine, if I just run dd, it is fine. If you just run octave, it is
>> fine. Run yes+dd, gets it most of the way there, but will wake up
>> sometimes, off and on. Do all three together and it soft locks. It takes
>> 5-15 minutes. I did it on our main example hardware, which is a server.
>> I have also reproduced it on a desktop. Sometimes sysrq-n, to renice
>> realtime processes, brings it out of it enough you can kill processes 
>> off.
>
>
> Interesting, so you're locking up a preempt-rt kernel with SCHED_OTHER 
> tasks running at the least favorable priority.
>
> Note: nice -n 19 is actually the valid nice value (20 and higher seem 
> to be accepted, but have the same effect as 19). NICE(1)
>
> How many CPUs on your test machine?
>
The server is dual quad-core. The desktop is a quad-core with 
hyperthreading. Both are i7 based.