* How to debug hard lockup issue @ 2012-08-02 3:03 Lin Ming 2012-08-02 7:15 ` Uwe Kleine-König 2012-08-02 8:28 ` Jassi Brar 0 siblings, 2 replies; 8+ messages in thread From: Lin Ming @ 2012-08-02 3:03 UTC (permalink / raw) To: linux-arm-kernel Hi list, I encountered a hard lockup on a armv5 board with 2.6.32 kernel. There is no any message output since it is hard lockup. I tried watchdog, but watchdog can only reset the board. I still can't get any info. The hard lockup is possibly because kernel blocked/looped with interrupts disabled. Is there any non-maskable interrupt I can use to dump the kernel stack? Or any other hint? Thanks, Lin Ming ^ permalink raw reply [flat|nested] 8+ messages in thread
* How to debug hard lockup issue 2012-08-02 3:03 How to debug hard lockup issue Lin Ming @ 2012-08-02 7:15 ` Uwe Kleine-König 2012-08-02 7:59 ` Lin Ming 2012-08-02 8:28 ` Jassi Brar 1 sibling, 1 reply; 8+ messages in thread From: Uwe Kleine-König @ 2012-08-02 7:15 UTC (permalink / raw) To: linux-arm-kernel Hello, On Thu, Aug 02, 2012 at 11:03:31AM +0800, Lin Ming wrote: > I encountered a hard lockup on a armv5 board with 2.6.32 kernel. > There is no any message output since it is hard lockup. > > I tried watchdog, but watchdog can only reset the board. I still can't > get any info. > > The hard lockup is possibly because kernel blocked/looped with > interrupts disabled. > > Is there any non-maskable interrupt I can use to dump the kernel stack? > Or any other hint? Try enabling various debug options. The following come to (my) mind: CONFIG_DETECT_HUNG_TASK CONFIG_PROVE_LOCKING CONFIG_DEBUG_ATOMIC_SLEEP CONFIG_MAGIC_SYSRQ Best regards Uwe -- Pengutronix e.K. | Uwe Kleine-K?nig | Industrial Linux Solutions | http://www.pengutronix.de/ | ^ permalink raw reply [flat|nested] 8+ messages in thread
* How to debug hard lockup issue 2012-08-02 7:15 ` Uwe Kleine-König @ 2012-08-02 7:59 ` Lin Ming 2012-08-02 8:34 ` Uwe Kleine-König 0 siblings, 1 reply; 8+ messages in thread From: Lin Ming @ 2012-08-02 7:59 UTC (permalink / raw) To: linux-arm-kernel On Thu, Aug 2, 2012 at 3:15 PM, Uwe Kleine-K?nig <u.kleine-koenig@pengutronix.de> wrote: > Hello, > > On Thu, Aug 02, 2012 at 11:03:31AM +0800, Lin Ming wrote: >> I encountered a hard lockup on a armv5 board with 2.6.32 kernel. >> There is no any message output since it is hard lockup. >> >> I tried watchdog, but watchdog can only reset the board. I still can't >> get any info. >> >> The hard lockup is possibly because kernel blocked/looped with >> interrupts disabled. >> >> Is there any non-maskable interrupt I can use to dump the kernel stack? >> Or any other hint? > Try enabling various debug options. The following come to (my) mind: > > CONFIG_DETECT_HUNG_TASK > CONFIG_PROVE_LOCKING > CONFIG_DEBUG_ATOMIC_SLEEP > CONFIG_MAGIC_SYSRQ These debug options are already enabled. But it doesn't help. Thanks. ^ permalink raw reply [flat|nested] 8+ messages in thread
* How to debug hard lockup issue 2012-08-02 7:59 ` Lin Ming @ 2012-08-02 8:34 ` Uwe Kleine-König 2012-08-02 9:30 ` Lin Ming 0 siblings, 1 reply; 8+ messages in thread From: Uwe Kleine-König @ 2012-08-02 8:34 UTC (permalink / raw) To: linux-arm-kernel On Thu, Aug 02, 2012 at 03:59:36PM +0800, Lin Ming wrote: > On Thu, Aug 2, 2012 at 3:15 PM, Uwe Kleine-K?nig > <u.kleine-koenig@pengutronix.de> wrote: > > Hello, > > > > On Thu, Aug 02, 2012 at 11:03:31AM +0800, Lin Ming wrote: > >> I encountered a hard lockup on a armv5 board with 2.6.32 kernel. > >> There is no any message output since it is hard lockup. > >> > >> I tried watchdog, but watchdog can only reset the board. I still can't > >> get any info. > >> > >> The hard lockup is possibly because kernel blocked/looped with > >> interrupts disabled. > >> > >> Is there any non-maskable interrupt I can use to dump the kernel stack? > >> Or any other hint? > > Try enabling various debug options. The following come to (my) mind: > > > > CONFIG_DETECT_HUNG_TASK > > CONFIG_PROVE_LOCKING > > CONFIG_DEBUG_ATOMIC_SLEEP > > CONFIG_MAGIC_SYSRQ > > These debug options are already enabled. > But it doesn't help. then if you don't have a hardware debugger handy you can try https://lkml.org/lkml/2012/5/26/83 Best regards Uwe -- Pengutronix e.K. | Uwe Kleine-K?nig | Industrial Linux Solutions | http://www.pengutronix.de/ | ^ permalink raw reply [flat|nested] 8+ messages in thread
* How to debug hard lockup issue 2012-08-02 8:34 ` Uwe Kleine-König @ 2012-08-02 9:30 ` Lin Ming 2012-08-02 9:40 ` Jassi Brar 0 siblings, 1 reply; 8+ messages in thread From: Lin Ming @ 2012-08-02 9:30 UTC (permalink / raw) To: linux-arm-kernel On Thu, Aug 2, 2012 at 4:34 PM, Uwe Kleine-K?nig <u.kleine-koenig@pengutronix.de> wrote: > On Thu, Aug 02, 2012 at 03:59:36PM +0800, Lin Ming wrote: >> On Thu, Aug 2, 2012 at 3:15 PM, Uwe Kleine-K?nig >> <u.kleine-koenig@pengutronix.de> wrote: >> > Hello, >> > >> > On Thu, Aug 02, 2012 at 11:03:31AM +0800, Lin Ming wrote: >> >> I encountered a hard lockup on a armv5 board with 2.6.32 kernel. >> >> There is no any message output since it is hard lockup. >> >> >> >> I tried watchdog, but watchdog can only reset the board. I still can't >> >> get any info. >> >> >> >> The hard lockup is possibly because kernel blocked/looped with >> >> interrupts disabled. >> >> >> >> Is there any non-maskable interrupt I can use to dump the kernel stack? >> >> Or any other hint? >> > Try enabling various debug options. The following come to (my) mind: >> > >> > CONFIG_DETECT_HUNG_TASK >> > CONFIG_PROVE_LOCKING >> > CONFIG_DEBUG_ATOMIC_SLEEP >> > CONFIG_MAGIC_SYSRQ >> >> These debug options are already enabled. >> But it doesn't help. > then if you don't have a hardware debugger handy you can try > > https://lkml.org/lkml/2012/5/26/83 Will try pstore and PostMortem. Thanks. ^ permalink raw reply [flat|nested] 8+ messages in thread
* How to debug hard lockup issue 2012-08-02 9:30 ` Lin Ming @ 2012-08-02 9:40 ` Jassi Brar 2012-08-03 1:23 ` Lin Ming 0 siblings, 1 reply; 8+ messages in thread From: Jassi Brar @ 2012-08-02 9:40 UTC (permalink / raw) To: linux-arm-kernel On 2 August 2012 15:00, Lin Ming <minggr@gmail.com> wrote: > On Thu, Aug 2, 2012 at 4:34 PM, Uwe Kleine-K?nig > <u.kleine-koenig@pengutronix.de> wrote: >> On Thu, Aug 02, 2012 at 03:59:36PM +0800, Lin Ming wrote: >>> On Thu, Aug 2, 2012 at 3:15 PM, Uwe Kleine-K?nig >>> <u.kleine-koenig@pengutronix.de> wrote: >>> > Hello, >>> > >>> > On Thu, Aug 02, 2012 at 11:03:31AM +0800, Lin Ming wrote: >>> >> I encountered a hard lockup on a armv5 board with 2.6.32 kernel. >>> >> There is no any message output since it is hard lockup. >>> >> >>> >> I tried watchdog, but watchdog can only reset the board. I still can't >>> >> get any info. >>> >> >>> >> The hard lockup is possibly because kernel blocked/looped with >>> >> interrupts disabled. >>> >> >>> >> Is there any non-maskable interrupt I can use to dump the kernel stack? >>> >> Or any other hint? >>> > Try enabling various debug options. The following come to (my) mind: >>> > >>> > CONFIG_DETECT_HUNG_TASK >>> > CONFIG_PROVE_LOCKING >>> > CONFIG_DEBUG_ATOMIC_SLEEP >>> > CONFIG_MAGIC_SYSRQ >>> >>> These debug options are already enabled. >>> But it doesn't help. >> then if you don't have a hardware debugger handy you can try >> >> https://lkml.org/lkml/2012/5/26/83 > > Will try pstore and PostMortem. > If the hang is after bootup, pstore is a better option. If the hang is consistent and during boot, dumping __log_buf is the only option. ^ permalink raw reply [flat|nested] 8+ messages in thread
* How to debug hard lockup issue 2012-08-02 9:40 ` Jassi Brar @ 2012-08-03 1:23 ` Lin Ming 0 siblings, 0 replies; 8+ messages in thread From: Lin Ming @ 2012-08-03 1:23 UTC (permalink / raw) To: linux-arm-kernel On Thu, Aug 2, 2012 at 5:40 PM, Jassi Brar <jaswinder.singh@linaro.org> wrote: > On 2 August 2012 15:00, Lin Ming <minggr@gmail.com> wrote: >> On Thu, Aug 2, 2012 at 4:34 PM, Uwe Kleine-K?nig >> <u.kleine-koenig@pengutronix.de> wrote: >>> On Thu, Aug 02, 2012 at 03:59:36PM +0800, Lin Ming wrote: >>>> On Thu, Aug 2, 2012 at 3:15 PM, Uwe Kleine-K?nig >>>> <u.kleine-koenig@pengutronix.de> wrote: >>>> > Hello, >>>> > >>>> > On Thu, Aug 02, 2012 at 11:03:31AM +0800, Lin Ming wrote: >>>> >> I encountered a hard lockup on a armv5 board with 2.6.32 kernel. >>>> >> There is no any message output since it is hard lockup. >>>> >> >>>> >> I tried watchdog, but watchdog can only reset the board. I still can't >>>> >> get any info. >>>> >> >>>> >> The hard lockup is possibly because kernel blocked/looped with >>>> >> interrupts disabled. >>>> >> >>>> >> Is there any non-maskable interrupt I can use to dump the kernel stack? >>>> >> Or any other hint? >>>> > Try enabling various debug options. The following come to (my) mind: >>>> > >>>> > CONFIG_DETECT_HUNG_TASK >>>> > CONFIG_PROVE_LOCKING >>>> > CONFIG_DEBUG_ATOMIC_SLEEP >>>> > CONFIG_MAGIC_SYSRQ >>>> >>>> These debug options are already enabled. >>>> But it doesn't help. >>> then if you don't have a hardware debugger handy you can try >>> >>> https://lkml.org/lkml/2012/5/26/83 >> >> Will try pstore and PostMortem. >> > If the hang is after bootup, pstore is a better option. If the hang is > consistent and during boot, dumping __log_buf is the only option. It's after bootup. After running about 1 day, the system is hard lockup. That's possibly because kernel blocked/looped with interrupts disabled(local_irq_disable). So soft lockup detector won't help because timer interrupt(and all other maskable interrupts) was disabled. And unfortunately, armv5 seems don't support hard lockup detector. Hard lockup detector needs non-maskable interrupt. I'm going to try pstore to get the last function call when hard lockup happens. Thanks, Lin Ming ^ permalink raw reply [flat|nested] 8+ messages in thread
* How to debug hard lockup issue 2012-08-02 3:03 How to debug hard lockup issue Lin Ming 2012-08-02 7:15 ` Uwe Kleine-König @ 2012-08-02 8:28 ` Jassi Brar 1 sibling, 0 replies; 8+ messages in thread From: Jassi Brar @ 2012-08-02 8:28 UTC (permalink / raw) To: linux-arm-kernel On 2 August 2012 08:33, Lin Ming <minggr@gmail.com> wrote: > Hi list, > > I encountered a hard lockup on a armv5 board with 2.6.32 kernel. > There is no any message output since it is hard lockup. > > I tried watchdog, but watchdog can only reset the board. I still can't > get any info. > Maybe take hint from http://www.denx.de/wiki/DULG/LinuxPostMortemAnalysis ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-08-03 1:23 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-08-02 3:03 How to debug hard lockup issue Lin Ming 2012-08-02 7:15 ` Uwe Kleine-König 2012-08-02 7:59 ` Lin Ming 2012-08-02 8:34 ` Uwe Kleine-König 2012-08-02 9:30 ` Lin Ming 2012-08-02 9:40 ` Jassi Brar 2012-08-03 1:23 ` Lin Ming 2012-08-02 8:28 ` Jassi Brar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).