How to debug system freeze (not detected by kernel debug options)

How to debug system freeze (not detected by kernel debug options)

[Jongman Heo]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 843 bytes --]


Dear kernel hackers,

I have a problem in SMP environment, in x86 platform (Intel Atom based embedded system)

In UP, there is no issue, but in SMP, system freezed in tens of minutes (or shorter), if I perform IO test with flash memory and HDD simultaneously (using dd).

I enabled relevant kernel debug options, like LOCKDEP, DETECT_SOFTLOCKUP, DETECT_HUNG_TASK, along with "nmi_watchdog=1".
(Yeah, this is somewhat old kernel, 2.6.35.14).

But no debug message is shown. (I had checked that NMI interrupt count correctly increase.).

Do you have any thoughts what can cause system freeze without being detected by LOCKDEP, watchdog, and other options.

Thanks in advance,
Jongman.ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: How to debug system freeze (not detected by kernel debug options)

[Alan Cox]

On Fri, 09 Nov 2012 02:19:33 +0000 (GMT)
Jongman Heo <jongman.heo@samsung.com> wrote:

> 
> Dear kernel hackers,
> 
> I have a problem in SMP environment, in x86 platform (Intel Atom based embedded system)
> 
> In UP, there is no issue, but in SMP, system freezed in tens of minutes (or shorter), if I perform IO test with flash memory and HDD simultaneously (using dd).
> 
> I enabled relevant kernel debug options, like LOCKDEP, DETECT_SOFTLOCKUP, DETECT_HUNG_TASK, along with "nmi_watchdog=1".
> (Yeah, this is somewhat old kernel, 2.6.35.14).
> 
> But no debug message is shown. (I had checked that NMI interrupt count correctly increase.).
> 
> Do you have any thoughts what can cause system freeze without being detected by LOCKDEP, watchdog, and other options.

Hardware problems, firmware bugs, PATA controller hangs, some
classes of PCI device hang, certain cases where for some reason the crash
is so bad the kernel can't get the message out even though it has detected
the failure.

A good starting point is probably "can you make two identical systems do
it". If you've got a pair of boards which fed the same software set and
have the same flash and hdd crash in the same way its unlikely to be a
faulty board.

You may find it useful to make the NMI timeout handler trigger a directly
detectable event via an I/O port if your platform has a buzzer or LED
directly I/O mapped somewhere.

Failing that the fastest approach may be to use hardware debugging aids
if you have access to them for that platform.

Alan