kirkwood openrd: random kernel crashes with git kernel

kirkwood openrd: random kernel crashes with git kernel

[Dieter Kiermaier]

Hi all,

hm - it seems to be suspect but I've got random kernel crashes with latest openrd git kernel from:

http://repo.or.cz/w/linux-2.6/linux-2.6-openrd.git
(BTW: are there any news on bringing the openrd kernel to mainline?)

It looks like there is something wrong in cpu_idle()???
kernel: Linux version 2.6.31-rc9-ge2e09ec (dieter at dk1-linux) (gcc version 4.2.0 20070413 (prerelease)) #10 PREEMPT Thu Oct 1 08:46:47 CEST 2009

Once I got a backtrace like:
------------[ cut here ]------------
WARNING: at arch/arm/kernel/process.c:171 cpu_idle+0x74/0xbc()
Modules linked in:
[<c002a748>] (unwind_backtrace+0x0/0xd8) from [<c003931c>] (warn_slowpath_common+0x4c/0x68)
[<c003931c>] (warn_slowpath_common+0x4c/0x68) from [<c0026844>] (cpu_idle+0x74/0xbc)
[<c0026844>] (cpu_idle+0x74/0xbc) from [<c0008b2c>] (start_kernel+0x22c/0x27c)
[<c0008b2c>] (start_kernel+0x22c/0x27c) from [<00008034>] (0x8034)
---[ end trace fb05d74310f1e273 ]---

Many other times the system freezes and I see in the debugger:

: System
cpsr: 0xffffffff pc: 0xffffffe9
MMU: disabled, D-Cache: disabled, I-Cache: disabled
timeout waiting for SYSCOMP & DBGACK, last DBG_STATUS: 0
timeout waiting for SYSCOMP & DBGACK, last DBG_STATUS: 0
timeout waiting for SYSCOMP & DBGACK, last DBG_STATUS: 0
timeout waiting for SYSCOMP & DBGACK, last DBG_STATUS: 0
timeout waiting for SYSCOMP & DBGACK, last DBG_STATUS: 0
timeout waiting for SYSCOMP & DBGACK, last DBG_STATUS: 0
target state: halted
target halted in Thumb state due to debug-request, current mode: Abort
cpsr: 0x00000037 pc: 0xc046a74a
MMU: disabled, D-Cache: disabled, I-Cache: disabled

I've written yesterday in a email to openrd google group "openrd kernel questions" that I've got problems with enabled EHCI but it looks like
that wasn't the reason because today morning I got the crashes without enabled EHCI!

Have somenone else observed this earlier and how can I go on to track the bug down?

Many thanks,
Dieter

kirkwood openrd: random kernel crashes with git kernel

[Simon Kagstrom]

On Thu, 1 Oct 2009 09:04:18 +0200
Dieter Kiermaier <dk-arm-linux@gmx.de> wrote:

> hm - it seems to be suspect but I've got random kernel crashes with latest openrd git kernel from:
> 
> http://repo.or.cz/w/linux-2.6/linux-2.6-openrd.git
> (BTW: are there any news on bringing the openrd kernel to mainline?)
> 
> It looks like there is something wrong in cpu_idle()???

Try starting with openocd disconnected (i.e., the openocd program
suspended or terminated). I also get random crashes when that is on,
presumably caused by it stopping and resuming the kernel randomly (you
can see that in the openocd terminal window).

// Simon

kirkwood openrd: random kernel crashes with git kernel

[Dieter Kiermaier]

Am Donnerstag 01 Oktober 2009 09:04:44 schrieb Simon Kagstrom:
> On Thu, 1 Oct 2009 09:04:18 +0200
> Dieter Kiermaier <dk-arm-linux@gmx.de> wrote:
> 
> > hm - it seems to be suspect but I've got random kernel crashes with latest openrd git kernel from:
> > 
> > http://repo.or.cz/w/linux-2.6/linux-2.6-openrd.git
> > (BTW: are there any news on bringing the openrd kernel to mainline?)
> > 
> > It looks like there is something wrong in cpu_idle()???
> 
> Try starting with openocd disconnected (i.e., the openocd program
> suspended or terminated). I also get random crashes when that is on,
> presumably caused by it stopping and resuming the kernel randomly (you
> can see that in the openocd terminal window).
> 

:)
It looks like this was the real reason!
I've spent hours of sorting out what causes the crashes but now it's clear why it is almost random
and not depending on my kernel configuration!

Many thanks,
Dieter


> // Simon
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 

kirkwood openrd: random kernel crashes with git kernel

[Simon Kagstrom]

On Thu, 1 Oct 2009 10:07:17 +0200
Dieter Kiermaier <dk-arm-linux@gmx.de> wrote:

> > Try starting with openocd disconnected (i.e., the openocd program
> > suspended or terminated). I also get random crashes when that is on,
> > presumably caused by it stopping and resuming the kernel randomly (you
> > can see that in the openocd terminal window).
>
> It looks like this was the real reason!
> I've spent hours of sorting out what causes the crashes but now it's clear why it is almost random
> and not depending on my kernel configuration!

I guess this is something that should be asked to the openocd people. I
haven't got around to look at it more yet though. It might even be a
known issue I suppose.

Or perhaps the timing difference is triggering some real kernel bug.
Anyway, without OpenOCD, I don't see random kernel crashes anymore.

// Simon

kirkwood openrd: random kernel crashes with git kernel

[Nicolas Pitre]

On Thu, 1 Oct 2009, Simon Kagstrom wrote:

> On Thu, 1 Oct 2009 09:04:18 +0200
> Dieter Kiermaier <dk-arm-linux@gmx.de> wrote:
> 
> > hm - it seems to be suspect but I've got random kernel crashes with latest openrd git kernel from:
> > 
> > http://repo.or.cz/w/linux-2.6/linux-2.6-openrd.git
> > (BTW: are there any news on bringing the openrd kernel to mainline?)
> > 
> > It looks like there is something wrong in cpu_idle()???
> 
> Try starting with openocd disconnected (i.e., the openocd program
> suspended or terminated). I also get random crashes when that is on,
> presumably caused by it stopping and resuming the kernel randomly (you
> can see that in the openocd terminal window).

Also note that openocd cannot work reliably with Linux running on the 
target if you don't use the nohlt kernel cmdline argument.


Nicolas

kirkwood openrd: random kernel crashes with git kernel

[Sascha Silbe]

On Thu, Oct 01, 2009 at 09:04:18AM +0200, Dieter Kiermaier wrote:

> WARNING: at arch/arm/kernel/process.c:171 cpu_idle+0x74/0xbc()
> Modules linked in:
> [<c002a748>] (unwind_backtrace+0x0/0xd8) from [<c003931c>] 
> (warn_slowpath_common+0x4c/0x68)
> [<c003931c>] (warn_slowpath_common+0x4c/0x68) from [<c0026844>] 
> (cpu_idle+0x74/0xbc)
> [<c0026844>] (cpu_idle+0x74/0xbc) from [<c0008b2c>] 
> (start_kernel+0x22c/0x27c)
> [<c0008b2c>] (start_kernel+0x22c/0x27c) from [<00008034>] (0x8034)
Try the patch from [1], it works for me (uptime 5 days now without the 
message above).
Disclaimer: I know next to nothing about Linux internals, so can't judge 
the quality of the patch. Use at your own risk.


[1] http://patchwork.kernel.org/patch/40871/

CU Sascha

-- 
http://sascha.silbe.org/
http://www.infra-silbe.de/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20091001/e932760f/attachment-0001.sig>

kirkwood openrd: random kernel crashes with git kernel

[Simon Kagstrom]

On Thu, 1 Oct 2009 10:12:16 +0200
Sascha Silbe <sascha-ml-devel-openrd-google@silbe.org> wrote:

> On Thu, Oct 01, 2009 at 09:04:18AM +0200, Dieter Kiermaier wrote:
> 
> > WARNING: at arch/arm/kernel/process.c:171 cpu_idle+0x74/0xbc()
> > Modules linked in:
> > [<c002a748>] (unwind_backtrace+0x0/0xd8) from [<c003931c>] 
> > (warn_slowpath_common+0x4c/0x68)
> > [<c003931c>] (warn_slowpath_common+0x4c/0x68) from [<c0026844>] 
> > (cpu_idle+0x74/0xbc)
> > [<c0026844>] (cpu_idle+0x74/0xbc) from [<c0008b2c>] 
> > (start_kernel+0x22c/0x27c)
> > [<c0008b2c>] (start_kernel+0x22c/0x27c) from [<00008034>] (0x8034)
> Try the patch from [1], it works for me (uptime 5 days now without the 
> message above).
> Disclaimer: I know next to nothing about Linux internals, so can't judge 
> the quality of the patch. Use at your own risk.
> 
> [1] http://patchwork.kernel.org/patch/40871/

Nice!

I should add that I've seen this error even without openocd connected,
so this seems to be a real error. Anyway, I guess this will sooner or
later be applied (in some form) upstream, so openrd should benefit
"automatically" from it.

However, with OpenOCD there are other type of crashes which look very
random, so I believe those will stay even with this.

// Simon

kirkwood openrd: random kernel crashes with git kernel

[Dieter Kiermaier]

Am Donnerstag 01 Oktober 2009 10:19:25 schrieb Simon Kagstrom:
> 
> On Thu, 1 Oct 2009 10:12:16 +0200
> Sascha Silbe <sascha-ml-devel-openrd-google@silbe.org> wrote:
> 
> > On Thu, Oct 01, 2009 at 09:04:18AM +0200, Dieter Kiermaier wrote:
> > 
> > > WARNING: at arch/arm/kernel/process.c:171 cpu_idle+0x74/0xbc()
> > > Modules linked in:
> > > [<c002a748>] (unwind_backtrace+0x0/0xd8) from [<c003931c>] 
> > > (warn_slowpath_common+0x4c/0x68)
> > > [<c003931c>] (warn_slowpath_common+0x4c/0x68) from [<c0026844>] 
> > > (cpu_idle+0x74/0xbc)
> > > [<c0026844>] (cpu_idle+0x74/0xbc) from [<c0008b2c>] 
> > > (start_kernel+0x22c/0x27c)
> > > [<c0008b2c>] (start_kernel+0x22c/0x27c) from [<00008034>] (0x8034)
> > Try the patch from [1], it works for me (uptime 5 days now without the 
> > message above).
> > Disclaimer: I know next to nothing about Linux internals, so can't judge 
> > the quality of the patch. Use at your own risk.
> > 
> > [1] http://patchwork.kernel.org/patch/40871/
> 
> Nice!
> 
> I should add that I've seen this error even without openocd connected,
> so this seems to be a real error. Anyway, I guess this will sooner or
> later be applied (in some form) upstream, so openrd should benefit
> "automatically" from it.
> 
> However, with OpenOCD there are other type of crashes which look very
> random, so I believe those will stay even with this.
> 

Many thanks to all off you for your valuable comments!

As far as I've seen on my system there are two different things in conjunction with openOCD:

1. The above adressed issue in cpu_idle path
2. Another different thing which causes slightly random crashes.

So [1.] could be solved by the above mentioned patch - I will try this out.

[2.] ??? I will try to find some information at the openocd list archive...

Dieter


> // Simon
> 
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "OpenRD" group.
> To post to this group, send email to openrd at googlegroups.com
> To unsubscribe from this group, send email to openrd+unsubscribe at googlegroups.com
> For more options, visit this group at http://groups.google.com/group/openrd?hl=en
> -~----------~----~----~----~------~----~------~--~---
> 

kirkwood openrd: random kernel crashes with git kernel

[Nicolas Pitre]

On Thu, 1 Oct 2009, Simon Kagstrom wrote:

> On Thu, 1 Oct 2009 10:12:16 +0200
> Sascha Silbe <sascha-ml-devel-openrd-google@silbe.org> wrote:
> 
> > On Thu, Oct 01, 2009 at 09:04:18AM +0200, Dieter Kiermaier wrote:
> > 
> > > WARNING: at arch/arm/kernel/process.c:171 cpu_idle+0x74/0xbc()
> > > Modules linked in:
> > > [<c002a748>] (unwind_backtrace+0x0/0xd8) from [<c003931c>] 
> > > (warn_slowpath_common+0x4c/0x68)
> > > [<c003931c>] (warn_slowpath_common+0x4c/0x68) from [<c0026844>] 
> > > (cpu_idle+0x74/0xbc)
> > > [<c0026844>] (cpu_idle+0x74/0xbc) from [<c0008b2c>] 
> > > (start_kernel+0x22c/0x27c)
> > > [<c0008b2c>] (start_kernel+0x22c/0x27c) from [<00008034>] (0x8034)
> > Try the patch from [1], it works for me (uptime 5 days now without the 
> > message above).
> > Disclaimer: I know next to nothing about Linux internals, so can't judge 
> > the quality of the patch. Use at your own risk.
> > 
> > [1] http://patchwork.kernel.org/patch/40871/
> 
> Nice!
> 
> I should add that I've seen this error even without openocd connected,
> so this seems to be a real error. Anyway, I guess this will sooner or
> later be applied (in some form) upstream, so openrd should benefit
> "automatically" from it.
> 
> However, with OpenOCD there are other type of crashes which look very
> random, so I believe those will stay even with this.

Please send me information about those when you can.


Nicolas

kirkwood openrd: random kernel crashes with git kernel

[Simon Kagstrom]

On Thu, 01 Oct 2009 12:19:21 -0400 (EDT)
Nicolas Pitre <nico@fluxnic.net> wrote:

> > I should add that I've seen this error even without openocd connected,
> > so this seems to be a real error. Anyway, I guess this will sooner or
> > later be applied (in some form) upstream, so openrd should benefit
> > "automatically" from it.
> > 
> > However, with OpenOCD there are other type of crashes which look very
> > random, so I believe those will stay even with this.
> 
> Please send me information about those when you can.

The nohlt argument seems to fix them, thanks! Without it I get random
memory corruption errors such as this:

   Mount local file[    5.160903] BUG: spinlock wrong owner on CPU#0, rc.sysinit/452
   [    5.166878]  lock: df464af8, .magic: dead4ead, .owner: <none>/-1, .owner_cpu: 0
   [    5.174229] [<c003130c>] (unwind_backtrace+0x0/0xdc) from [<c02c7b10>] (dump_stack+0x18/0x1c)
   [    5.182842] [<c02c7b10>] (dump_stack+0x18/0x1c) from [<c017bc08>] (spin_bug+0x90/0xa4)
   [    5.190837] [<c017bc08>] (spin_bug+0x90/0xa4) from [<c017bc84>] (_raw_spin_unlock+0x68/0xac)
   [    5.199356] [<c017bc84>] (_raw_spin_unlock+0x68/0xac) from [<c02ca8a0>] (_spin_unlock_irq+0x10/0x20)
   [    5.208573] [<c02ca8a0>] (_spin_unlock_irq+0x10/0x20) from [<c00475dc>] (dup_mm+0x278/0x450)
   [    5.217093] [<c00475dc>] (dup_mm+0x278/0x450) from [<c0048004>] (copy_process+0x794/0xddc)
   [    5.225437] [<c0048004>] (copy_process+0x794/0xddc) from [<c00487ac>] (do_fork+0x160/0x314)
   [    5.233870] [<c00487ac>] (do_fork+0x160/0x314) from [<c002e218>] (sys_clone+0x34/0x3c)
   [    5.241867] [<c002e218>] (sys_clone+0x34/0x3c) from [<c002ae80>] (ret_fast_syscall+0x0/0x28)
   systems.
   Segmentation fault
   [    5.264718] Slab corruption: names_cache start=df9f9000, len=4096

although the typical case is that it just hangs without printing out
much.

// Simon

kirkwood openrd: random kernel crashes with git kernel

[Dieter Kiermaier]

Am Freitag 02 Oktober 2009 08:40:00 schrieb Simon Kagstrom:
> 
> On Thu, 01 Oct 2009 12:19:21 -0400 (EDT)
> Nicolas Pitre <nico@fluxnic.net> wrote:
> 
> > > I should add that I've seen this error even without openocd connected,
> > > so this seems to be a real error. Anyway, I guess this will sooner or
> > > later be applied (in some form) upstream, so openrd should benefit
> > > "automatically" from it.
> > > 
> > > However, with OpenOCD there are other type of crashes which look very
> > > random, so I believe those will stay even with this.
> > 
> > Please send me information about those when you can.
> 
> The nohlt argument seems to fix them, thanks! Without it I get random
> memory corruption errors such as this:
>

This is exactly what I've seen on my desk - with nohlt everything is running fine all the time!
Many thanks to the experts :)

Dieter

 
>    Mount local file[    5.160903] BUG: spinlock wrong owner on CPU#0, rc.sysinit/452
>    [    5.166878]  lock: df464af8, .magic: dead4ead, .owner: <none>/-1, .owner_cpu: 0
>    [    5.174229] [<c003130c>] (unwind_backtrace+0x0/0xdc) from [<c02c7b10>] (dump_stack+0x18/0x1c)
>    [    5.182842] [<c02c7b10>] (dump_stack+0x18/0x1c) from [<c017bc08>] (spin_bug+0x90/0xa4)
>    [    5.190837] [<c017bc08>] (spin_bug+0x90/0xa4) from [<c017bc84>] (_raw_spin_unlock+0x68/0xac)
>    [    5.199356] [<c017bc84>] (_raw_spin_unlock+0x68/0xac) from [<c02ca8a0>] (_spin_unlock_irq+0x10/0x20)
>    [    5.208573] [<c02ca8a0>] (_spin_unlock_irq+0x10/0x20) from [<c00475dc>] (dup_mm+0x278/0x450)
>    [    5.217093] [<c00475dc>] (dup_mm+0x278/0x450) from [<c0048004>] (copy_process+0x794/0xddc)
>    [    5.225437] [<c0048004>] (copy_process+0x794/0xddc) from [<c00487ac>] (do_fork+0x160/0x314)
>    [    5.233870] [<c00487ac>] (do_fork+0x160/0x314) from [<c002e218>] (sys_clone+0x34/0x3c)
>    [    5.241867] [<c002e218>] (sys_clone+0x34/0x3c) from [<c002ae80>] (ret_fast_syscall+0x0/0x28)
>    systems.
>    Segmentation fault
>    [    5.264718] Slab corruption: names_cache start=df9f9000, len=4096
> 
> although the typical case is that it just hangs without printing out
> much.
> 
> // Simon
> 
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "OpenRD" group.
> To post to this group, send email to openrd at googlegroups.com
> To unsubscribe from this group, send email to openrd+unsubscribe at googlegroups.com
> For more options, visit this group at http://groups.google.com/group/openrd?hl=en
> -~----------~----~----~----~------~----~------~--~---
> 

kirkwood openrd: random kernel crashes with git kernel

[Nicolas Pitre]

On Fri, 2 Oct 2009, Simon Kagstrom wrote:

> On Thu, 01 Oct 2009 12:19:21 -0400 (EDT)
> Nicolas Pitre <nico@fluxnic.net> wrote:
> 
> > > I should add that I've seen this error even without openocd connected,
> > > so this seems to be a real error. Anyway, I guess this will sooner or
> > > later be applied (in some form) upstream, so openrd should benefit
> > > "automatically" from it.
> > > 
> > > However, with OpenOCD there are other type of crashes which look very
> > > random, so I believe those will stay even with this.
> > 
> > Please send me information about those when you can.
> 
> The nohlt argument seems to fix them, thanks! Without it I get random
> memory corruption errors such as this:
[...]

Good.  Just so that you know the reason: the nohlt argument tells the 
kernel not to use the wait-for-interrupt instruction when idle.  This 
basically turns the idle loop into a busy loop.

The WFI instruction is normally used because it turns the CPU clock off 
to save power. Unfortunately, the clock to the JTAG unit is also turned 
off in that case and JTAG communication goes down the drain when that 
occurs.  Depending on exactly when the WFI mode is entered, if that 
happens in the middle of a JTAG communication then strange things may 
happen.  With nohlt the CPU clock is never stopped.


Nicolas

kirkwood openrd: random kernel crashes with git kernel

[Russell King - ARM Linux]

On Thu, Oct 01, 2009 at 09:04:18AM +0200, Dieter Kiermaier wrote:
> Once I got a backtrace like:
> ------------[ cut here ]------------
> WARNING: at arch/arm/kernel/process.c:171 cpu_idle+0x74/0xbc()
> Modules linked in:
> [<c002a748>] (unwind_backtrace+0x0/0xd8) from [<c003931c>] (warn_slowpath_common+0x4c/0x68)
> [<c003931c>] (warn_slowpath_common+0x4c/0x68) from [<c0026844>] (cpu_idle+0x74/0xbc)
> [<c0026844>] (cpu_idle+0x74/0xbc) from [<c0008b2c>] (start_kernel+0x22c/0x27c)
> [<c0008b2c>] (start_kernel+0x22c/0x27c) from [<00008034>] (0x8034)
> ---[ end trace fb05d74310f1e273 ]---

This is indicative of something causing interrupts to be disabled in
the cpu idle loop after the pm specific idle loop returns.  That could
be a buggy pm_idle implementation, or, as you seem to have identified,
not using your debugger appears to solve the problem.