preempt crash in 2.6.18

preempt crash in 2.6.18

[Rune Torgersen]

[-- Attachment #1: Type: text/plain, Size: 1622 bytes --]


This is from a Freescale 8265 running 2.6.18
Anyone have any idea what this is and how to fix it?

TIPC: Blocking bearer <eth:eth0>
huh, entered a0232b3c with preempt_count 00000102, exited with 00000203?
kernel BUG in __run_timers at kernel/timer.c:453!
Oops: Exception in kernel mode, sig: 5 [#1]
PREEMPT
NIP: A0024574 LR: A0024574 CTR: A0032104
REGS: a02cfd90 TRAP: 0700   Tainted: P       (2.6.18)
MSR: 00029032 <EE,ME,IR,DR>  CR: 44024024  XER: 00000000
TASK = a02900a8[0] 'swapper' THREAD: a02ce000
GPR00: A0024574 A02CFE40 A02900A8 0000004C A02F0E34 00004481 F4915203 00000000
GPR08: 00000008 00000203 DE53FE9C A02CE000 44024084 414404C0 0FFCE000 FFFFFFFF
GPR16: 00000000 0FFC8D68 007FFF00 0FF77600 00000002 00200040 00800000 007FFEA0
GPR24: 00000000 A02F0000 A0290000 A02F61C8 A02CFE50 00000102 A0232B3C A02CE000
NIP [A0024574] run_timer_softirq+0x12c/0x214
LR [A0024574] run_timer_softirq+0x12c/0x214
Call Trace:
[A02CFE40] [A0024574] run_timer_softirq+0x12c/0x214 (unreliable)
[A02CFE70] [A001F38C] __do_softirq+0x80/0xe4
[A02CFEA0] [A0009ACC] do_softirq+0x58/0x60
[A02CFEB0] [A001F450] irq_exit+0x60/0x80
[A02CFEC0] [A0009A28] do_IRQ+0x70/0xbc
[A02CFED0] [A00047B4] ret_from_except+0x0/0x14
[A02CFF90] [A000C9BC] cpu_idle+0xf8/0x108
[A02CFFB0] [A0003790] rest_init+0x4c/0x60
[A02CFFD0] [A02D06A4] start_kernel+0x1b8/0x1f4
[A02CFFF0] [000032C0] 0x32c0
Instruction dump:
7fa3eb78 7fc803a6 83bf000c 4e800021 80df000c 3c60a025 7f86e800 3863eaf4
7fc4f378 7fa5eb78 419e000c 4bff4dfd <0fe00000> 7c0000a6 5400045e 7c000124
Kernel panic - not syncing: Aiee, killing interrupt handler!

[-- Attachment #2: Type: text/html, Size: 2192 bytes --]

Re: preempt crash in 2.6.18

[Vitaly Bordug]

[-- Attachment #1: Type: text/plain, Size: 397 bytes --]

On Fri, 13 Oct 2006 09:16:52 -0500
Rune Torgersen wrote:

> 
> This is from a Freescale 8265 running 2.6.18
> Anyone have any idea what this is and how to fix it?
> 
Yes. Actually, I recall alike thing and used to fix with proper spin-locking if the 
cascade PCI irq, which apparently does do_irq that seems to confuse preempt counters.

Just a pure guess.

Hope it helps.

-Vitaly

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: preempt crash in 2.6.18

[Benjamin Herrenschmidt]

On Sat, 2006-10-14 at 01:21 +0400, Vitaly Bordug wrote:
> On Fri, 13 Oct 2006 09:16:52 -0500
> Rune Torgersen wrote:
> 
> > 
> > This is from a Freescale 8265 running 2.6.18
> > Anyone have any idea what this is and how to fix it?
> > 
> Yes. Actually, I recall alike thing and used to fix with proper spin-locking if the 
> cascade PCI irq, which apparently does do_irq that seems to confuse preempt counters.
> 
> Just a pure guess.

How so ? Cascades shouldn't do do_IRQ with the new irq code anyway
unless this is still arch/ppc, they should do either __do_IRQ or better,
generic_handle_irq().

Ben.

Re: preempt crash in 2.6.18

[Vitaly Bordug]

[-- Attachment #1: Type: text/plain, Size: 863 bytes --]

On Sat, 14 Oct 2006 09:14:38 +1000
Benjamin Herrenschmidt wrote:

> On Sat, 2006-10-14 at 01:21 +0400, Vitaly Bordug wrote:
> > On Fri, 13 Oct 2006 09:16:52 -0500
> > Rune Torgersen wrote:
> > 
> > > 
> > > This is from a Freescale 8265 running 2.6.18
> > > Anyone have any idea what this is and how to fix it?
> > > 
> > Yes. Actually, I recall alike thing and used to fix with proper
> > spin-locking if the cascade PCI irq, which apparently does do_irq
> > that seems to confuse preempt counters.
> > 
> > Just a pure guess.
> 
> How so ? Cascades shouldn't do do_IRQ with the new irq code anyway
> unless this is still arch/ppc, they should do either __do_IRQ or
> better, generic_handle_irq().
> 
To clarify - I was about __do_IRQ and arch/ppc. 
Anyway, cascade for PCI is the first place I'd give a look.

--
Sincerely,

-Vitaly

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: preempt crash in 2.6.18

[Rune Torgersen]

[-- Attachment #1: Type: text/plain, Size: 1141 bytes --]

-----Original Message-----
From: Vitaly Bordug [mailto:vbordug@ru.mvista.com]
 
On Sat, 14 Oct 2006 09:14:38 +1000
Benjamin Herrenschmidt wrote:

> On Sat, 2006-10-14 at 01:21 +0400, Vitaly Bordug wrote:
> > On Fri, 13 Oct 2006 09:16:52 -0500
> > Rune Torgersen wrote:
> > 
> > > 
> > > This is from a Freescale 8265 running 2.6.18
> > > Anyone have any idea what this is and how to fix it?
> > > 
> > Yes. Actually, I recall alike thing and used to fix with proper
> > spin-locking if the cascade PCI irq, which apparently does do_irq
> > that seems to confuse preempt counters.
> > 
> > Just a pure guess.
> 
> How so ? Cascades shouldn't do do_IRQ with the new irq code anyway
> unless this is still arch/ppc, they should do either __do_IRQ or
> better, generic_handle_irq().
> 
To clarify - I was about __do_IRQ and arch/ppc. 
Anyway, cascade for PCI is the first place I'd give a look.

In our case it turned out to be in the TIPC network code. post_timeout was called and didn't unlock a lock.
The patch had been in the TIPC tree for abpout 2 days when we saw the problem.


--
Sincerely,

-Vitaly


[-- Attachment #2: Type: text/html, Size: 1792 bytes --]