Soft lock issue with 2.6.33.7-rt29

[Nathan Grennan <ngrennan@willowgarage.com>]

    I have been working for weeks to get a stable rt kernel. I had been 
focusing on 2.6.31.6-rt19. It is stable for about four days under stress 
testing before it soft locks. I am using rt19 instead of rt21, because 
rt19 seems to be more stable. The rtmutex issue that seems to still be 
in rt29 is in rt21. I also had to backport the iptables fix to rt19.

   I just started looking at 2.6.33.7-rt29 again, since I can reproduce 
a soft lock with it in 10-15 minutes. I have yet to get sysrq output for 
rt19, since it takes four days. The soft lock with rt29 as far as I can 
tell seems to relate to disk i/o.

    There are links to two logs of rt29 from a serial console below. 
They include sysrq output like "Show Blocked State" and "Show State". 
The level7 file is with nfsd enable, and level9 is with it disable. So 
nfsd doesn't seem to be the issue.

   If any other debugging information is useful or needed, just say the 
word.


http://proton.cygnusx-1.org/~edgan/kernel-logs/kernel-2.6.33-rt29.level7.log

http://proton.cygnusx-1.org/~edgan/kernel-logs/kernel-2.6.33-rt29.level9.log