Re: troubleshooting/debugging hard locks

[Lee Howard <faxguy@howardsilvan.com>]

Lee Howard wrote:
> Zan Lynx wrote:
>> On Wed, 2008-05-14 at 15:43 -0700, Ray Lee wrote:
>>  
>>> On Wed, May 14, 2008 at 12:27 PM, Lee Howard 
>>> <faxguy@howardsilvan.com> wrote:
>>>     
>>
>>  
>>>>  But, without kernel messages indicating where to look to debug... 
>>>> what is
>>>> the best approach to start troubleshooting and debugging this 
>>>> condition?  Is
>>>> there some general debug feature that can be enabled in the kernel 
>>>> that
>>>> would help hone in on the culprit?
>>>>       
>>> There's something called the NMI watchdog, that will print debugging
>>> messages out if it finds the system has hard locked. The short version
>>> is that you should add "nmi_watchdog=1" (no quotes) to the line in
>>> GRUB that has the kernel options. That assumes you have an APIC on the
>>> system. If that's not the case (you're on Uniprocessor, and no APIC)
>>> then you can try nmi_watchdog=2 instead. That'll only work on some
>>> systems, though.
>>>
>>> Better docs (than my cheesy writeup) are in
>>> Documentation/nmi_watchdog.txt in the kernel source distribution.
>>>     
>>
>> I was once told to add these to the kernel command line as well when
>> using NMI watchdog and they do seem to help it trigger more reliably:
>> "idle=poll nohz=off"
>
> Thank you to both Ray and Zan.  This was very helpful, and I think 
> that it has gotten me what I needed.
>
> "serial8250: too much work for irq16"
>
> Interestingly, now CTRL-SysRq-H will wake it back up... things get 
> running normally afterwards - the hard lock never occurs. 

After further troubleshooting it turns out that the message above was a 
bit of a red herring.  I moved the PCIe modem card from one PCIe slot to 
another so that it would be on a different IRQ (one that wasn't shared 
with the network, ATA, and USB).  The message above does not occur now, 
however, the hard lock is back... and this time with no kernel messages 
at all.

So, I'm back to square one again... I've got:

  nmi_watchdog=1 idle=poll nohz=off

... and still I get the hard lockup and no kernel messages.  How do I 
troubleshoot/debug this?  How do I know where to look?

Thanks,

Lee.