[BUG] oops in net_rx_action on 64-bit powerpc

[BUG] oops in net_rx_action on 64-bit powerpc

[Chris Friesen]

I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very similar to a 
Maple board).  The first time I booted I got the first log below via the 
serial console.  I rebooted and got as far as a login prompt.  I was able to 
log in via the serial console, but then got an almost identical oops again, as 
shown in the second log below.

I configed out the gigE drivers for the backplane so the only remaining 
network link was the e100 link used for booting, but the problem remained.

Anyone have any idea what might be causing this?

Thanks,

Chris


Starting xinetd: [  OK  ]
Starting cron: [  OK  ]
Unable to handle kernel paging request for data at address 0x00100108
Faulting instruction address: 0xc00000000028c1cc
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2 Maple
Modules linked in:
NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000
REGS: c00000000fff7b90 TRAP: 0300   Not tainted  (2.6.27-05329-g39076ba)
MSR: 9000000000009032 <EE,ME,IR,DR>  CR: 22ffff24  XER: 20000000
DAR: 0000000000100108, DSISR: 000000000a000000
TASK = c00000017a061080[0] 'swapper' THREAD: c00000017a078000 CPU: 1
GPR00: 0000000000000000 c00000000fff7e10 c00000000059bfe0 0000000000000020
GPR04: 0000000000000001 c000000178179800 c00000000027fda8 0000000000000000
GPR08: 0000000000000000 0000000000200200 0000000000000001 0000000000100100
GPR12: 0000000022ffff22 c0000000005bc500 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 000000000000000a 0000000000000001 0000000000000001
GPR24: c0000000005a2280 c0000000005f5134 00000000fffd9bbe 00000000000000ec
GPR28: c000000006e30c28 0000000000000020 c000000000543440 c00000017a279b40
NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c
LR [c00000000028c13c] .net_rx_action+0x154/0x26c
Call Trace:
[c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c (unreliable)
[c00000000fff7ec0] [c000000000056938] .__do_softirq+0xf8/0x1f4
[c00000000fff7f90] [c000000000024334] .call_do_softirq+0x14/0x24
[c00000017a07b970] [c00000000000bcdc] .do_softirq+0xf0/0x104
[c00000017a07ba10] [c000000000056ae8] .irq_exit+0x70/0x88
[c00000017a07ba90] [c00000000000ba18] .do_IRQ+0x14c/0x244
[c00000017a07bb30] [c000000000004710] hardware_interrupt_entry+0x18/0x1c
--- Exception: 501 at .raw_local_irq_restore+0x38/0x44
      LR = .cpu_idle+0xd8/0x154
[c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154 (unreliable)
[c00000017a07bec0] [c0000000003d4304] .start_secondary+0x310/0x3e8
[c00000017a07bf90] [c0000000000072b4] .start_secondary_prolog+0x10/0x14
Instruction dump:
eb61ffd8 eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 4e800020 e81f0010
7809ffe3 40820038 e93f0008 e97f0000 <f92b0008> f9690000 e95c0008 fb9f0000




root@10:/root> uname -a
Linux 10.41.18.77 2.6.27-05329-g39076ba #1 SMP Tue Oct 21 16:46:06 CST 2008
ppc64 GNU/Linux
root@10:/root> Unable to handle kernel paging request for data at address
0x00100108
Faulting instruction address: 0xc00000000028c1cc
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2 Maple
Modules linked in:
NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000
REGS: c00000000fff7b90 TRAP: 0300   Not tainted  (2.6.27-05329-g39076ba)
MSR: 9000000000009032 <EE,ME,IR,DR>  CR: 22ffff24  XER: 20000000
DAR: 0000000000100108, DSISR: 000000000a000000
TASK = c00000017a061080[0] 'swapper' THREAD: c00000017a078000 CPU: 1
GPR00: 0000000000000000 c00000000fff7e10 c00000000059bfe0 0000000000000020
GPR04: 0000000000000001 0000000000000001 c00000000027fda8 0000000000000000
GPR08: 0000000000000000 0000000000200200 0000000000000001 0000000000100100
GPR12: 0000000022ffff22 c0000000005bc500 0000000000000000 0000000000000000
GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
GPR20: 0000000000000000 000000000000000a 0000000000000001 0000000000000001
GPR24: c0000000005a2280 c0000000005f5134 00000001000387ff 000000000000010c
GPR28: c000000006e30c28 0000000000000020 c000000000543440 c00000017a2b0b40
NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c
LR [c00000000028c13c] .net_rx_action+0x154/0x26c
Call Trace:
[c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c (unreliable)
[c00000000fff7ec0] [c000000000056938] .__do_softirq+0xf8/0x1f4
[c00000000fff7f90] [c000000000024334] .call_do_softirq+0x14/0x24
[c00000017a07b970] [c00000000000bcdc] .do_softirq+0xf0/0x104
[c00000017a07ba10] [c000000000056ae8] .irq_exit+0x70/0x88
[c00000017a07ba90] [c00000000000ba18] .do_IRQ+0x14c/0x244
[c00000017a07bb30] [c000000000004710] hardware_interrupt_entry+0x18/0x1c
--- Exception: 501 at .cpu_idle+0xf0/0x154
      LR = .cpu_idle+0xd8/0x154
[c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154 (unreliable)
[c00000017a07bec0] [c0000000003d4304] .start_secondary+0x310/0x3e8
[c00000017a07bf90] [c0000000000072b4] .start_secondary_prolog+0x10/0x14
Instruction dump:
eb61ffd8 eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 4e800020 e81f0010
7809ffe3 40820038 e93f0008 e97f0000 <f92b0008> f9690000 e95c0008 fb9f0000

RE: [BUG] oops in net_rx_action on 64-bit powerpc

[Brandeburg, Jesse]

Chris Friesen wrote:
> I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very
> similar to a Maple board).  The first time I booted I got the first
> log below via the serial console.  I rebooted and got as far as a
> login prompt.  I was able to log in via the serial console, but then
> got an almost identical oops again, as shown in the second log below.
>=20
> I configed out the gigE drivers for the backplane so the only
> remaining network link was the e100 link used for booting, but the
> problem remained.=20
>=20
> Anyone have any idea what might be causing this?
>=20
> Thanks,
>=20
> Chris
>=20
>=20
> Starting xinetd: [  OK  ]
> Starting cron: [  OK  ]
> Unable to handle kernel paging request for data at address 0x00100108

that 00100108 pattern looks familiar, I'm not much help here, but I think t=
hat had something to do with the list management of the poll_list in a netd=
ev struct.

so now you just have to figure out why someone's netdev struct is becoming =
NULL. :-)

> Faulting instruction address: 0xc00000000028c1cc
> Oops: Kernel access of bad area, sig: 11 [#1]
> SMP NR_CPUS=3D2 Maple
> Modules linked in:
> NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000
> REGS: c00000000fff7b90 TRAP: 0300   Not tainted=20
> (2.6.27-05329-g39076ba)=20
> MSR: 9000000000009032 <EE,ME,IR,DR>  CR: 22ffff24  XER: 20000000
> DAR: 0000000000100108, DSISR: 000000000a000000

> TASK =3D c00000017a061080[0] 'swapper' THREAD: c00000017a078000 CPU: 1
> GPR00: 0000000000000000 c00000000fff7e10 c00000000059bfe0
> 0000000000000020 GPR04: 0000000000000001 c000000178179800
> c00000000027fda8 0000000000000000 GPR08: 0000000000000000
> 0000000000200200 0000000000000001 0000000000100100 GPR12:
> 0000000022ffff22 c0000000005bc500 0000000000000000 0000000000000000
> GPR16: 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000 GPR20: 0000000000000000 000000000000000a
> 0000000000000001 0000000000000001 GPR24: c0000000005a2280
> c0000000005f5134 00000000fffd9bbe 00000000000000ec GPR28:
> c000000006e30c28 0000000000000020 c000000000543440 c00000017a279b40
> NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c =20
> LR [c00000000028c13c] .net_rx_action+0x154/0x26c
> Call Trace:
> [c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c
> (unreliable) [c00000000fff7ec0] [c000000000056938]
> .__do_softirq+0xf8/0x1f4 [c00000000fff7f90] [c000000000024334]
> .call_do_softirq+0x14/0x24 [c00000017a07b970] [c00000000000bcdc]
> .do_softirq+0xf0/0x104 [c00000017a07ba10] [c000000000056ae8]
> .irq_exit+0x70/0x88 [c00000017a07ba90] [c00000000000ba18]
> .do_IRQ+0x14c/0x244 [c00000017a07bb30] [c000000000004710]
> hardware_interrupt_entry+0x18/0x1c --- Exception: 501 at
>       .raw_local_irq_restore+0x38/0x44 LR =3D .cpu_idle+0xd8/0x154
> [c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154
> (unreliable) [c00000017a07bec0] [c0000000003d4304]
> .start_secondary+0x310/0x3e8 [c00000017a07bf90] [c0000000000072b4]
> .start_secondary_prolog+0x10/0x14 Instruction dump:
> eb61ffd8 eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 4e800020
> e81f0010 7809ffe3 40820038 e93f0008 e97f0000 <f92b0008> f9690000
> e95c0008 fb9f0000=20
>=20
>=20
>=20
>=20
> root@10:/root> uname -a
> Linux 10.41.18.77 2.6.27-05329-g39076ba #1 SMP Tue Oct 21 16:46:06
> CST 2008 ppc64 GNU/Linux
> root@10:/root> Unable to handle kernel paging request for data at
> address 0x00100108
> Faulting instruction address: 0xc00000000028c1cc
> Oops: Kernel access of bad area, sig: 11 [#1]
> SMP NR_CPUS=3D2 Maple
> Modules linked in:
> NIP: c00000000028c1cc LR: c00000000028c13c CTR: 0000000000000000
> REGS: c00000000fff7b90 TRAP: 0300   Not tainted=20
> (2.6.27-05329-g39076ba)=20
> MSR: 9000000000009032 <EE,ME,IR,DR>  CR: 22ffff24  XER: 20000000
> DAR: 0000000000100108, DSISR: 000000000a000000
> TASK =3D c00000017a061080[0] 'swapper' THREAD: c00000017a078000 CPU: 1
> GPR00: 0000000000000000 c00000000fff7e10 c00000000059bfe0
> 0000000000000020 GPR04: 0000000000000001 0000000000000001
> c00000000027fda8 0000000000000000 GPR08: 0000000000000000
> 0000000000200200 0000000000000001 0000000000100100 GPR12:
> 0000000022ffff22 c0000000005bc500 0000000000000000 0000000000000000
> GPR16: 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000 GPR20: 0000000000000000 000000000000000a
> 0000000000000001 0000000000000001 GPR24: c0000000005a2280
> c0000000005f5134 00000001000387ff 000000000000010c GPR28:
> c000000006e30c28 0000000000000020 c000000000543440 c00000017a2b0b40
> NIP [c00000000028c1cc] .net_rx_action+0x1e4/0x26c =20
> LR [c00000000028c13c] .net_rx_action+0x154/0x26c
> Call Trace:
> [c00000000fff7e10] [c00000000028c13c] .net_rx_action+0x154/0x26c
> (unreliable) [c00000000fff7ec0] [c000000000056938]
> .__do_softirq+0xf8/0x1f4 [c00000000fff7f90] [c000000000024334]
> .call_do_softirq+0x14/0x24 [c00000017a07b970] [c00000000000bcdc]
> .do_softirq+0xf0/0x104 [c00000017a07ba10] [c000000000056ae8]
> .irq_exit+0x70/0x88 [c00000017a07ba90] [c00000000000ba18]
> .do_IRQ+0x14c/0x244 [c00000017a07bb30] [c000000000004710]
> hardware_interrupt_entry+0x18/0x1c --- Exception: 501 at
>       .cpu_idle+0xf0/0x154 LR =3D .cpu_idle+0xd8/0x154
> [c00000017a07be20] [c000000000012068] .cpu_idle+0x118/0x154
> (unreliable) [c00000017a07bec0] [c0000000003d4304]
> .start_secondary+0x310/0x3e8 [c00000017a07bf90] [c0000000000072b4]
> .start_secondary_prolog+0x10/0x14 Instruction dump:
> eb61ffd8 eb81ffe0 eba1ffe8 ebc1fff0 ebe1fff8 7c0803a6 4e800020
> e81f0010 7809ffe3 40820038 e93f0008 e97f0000 <f92b0008> f9690000
> e95c0008 fb9f0000=20

Re: [BUG] oops in net_rx_action on 64-bit powerpc

[David Miller]

From: "Brandeburg, Jesse" <jesse.brandeburg@intel.com>
Date: Thu, 23 Oct 2008 14:50:06 -0700

> Chris Friesen wrote:
> > I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very
> > similar to a Maple board).  The first time I booted I got the first
> > log below via the serial console.  I rebooted and got as far as a
> > login prompt.  I was able to log in via the serial console, but then
> > got an almost identical oops again, as shown in the second log below.
> > 
> > I configed out the gigE drivers for the backplane so the only
> > remaining network link was the e100 link used for booting, but the
> > problem remained. 
> > 
> > Anyone have any idea what might be causing this?
> > 
> > Thanks,
> > 
> > Chris
> > 
> > 
> > Starting xinetd: [  OK  ]
> > Starting cron: [  OK  ]
> > Unable to handle kernel paging request for data at address 0x00100108
> 
> that 00100108 pattern looks familiar, I'm not much help here, but I think that had something to do with the list management of the poll_list in a netdev struct.
> 
> so now you just have to figure out why someone's netdev struct is becoming NULL. :-)

Usually this is an indication of returning the wrong value from
the driver's ->poll() routine.

Re: [BUG] oops in net_rx_action on 64-bit powerpc

[Chris Friesen]

David Miller wrote:
> From: "Brandeburg, Jesse" <jesse.brandeburg@intel.com> Date: Thu, 23 Oct
> 2008 14:50:06 -0700
> 
>> Chris Friesen wrote:
>>> I tried booting a post 2.6.27 -git on a Motorola ATCA6101 (very similar
>>> to a Maple board).  The first time I booted I got the first log below
>>> via the serial console.  I rebooted and got as far as a login prompt.
>>> I was able to log in via the serial console, but then got an almost
>>> identical oops again, as shown in the second log below.
>>> 
>>> I configed out the gigE drivers for the backplane so the only remaining
>>> network link was the e100 link used for booting, but the problem
>>> remained.
>>> 
>>> Anyone have any idea what might be causing this?
>>> 
>>> Thanks,
>>> 
>>> Chris
>>> 
>>> 
>>> Starting xinetd: [  OK  ] Starting cron: [  OK  ] Unable to handle
>>> kernel paging request for data at address 0x00100108
>> that 00100108 pattern looks familiar, I'm not much help here, but I think
>> that had something to do with the list management of the poll_list in a
>> netdev struct.
>> 
>> so now you just have to figure out why someone's netdev struct is
>> becoming NULL. :-)
> 
> Usually this is an indication of returning the wrong value from the
> driver's ->poll() routine.

Looks like I was wrong before...the remaining ethernet link is an AMD-8111, 
not an e100.  Sorry about that.

I backed out 6ba33ac "amd8111e: delete non NAPI code from the driver".  With 
NAPI disabled, the blade appears stable.  With NAPI enabled, the original 
problem recurred.

So...it would appear that the NAPI code is somehow buggy, and 6ba33ac should 
probably be reverted until the problem is found and fixed.

Chris

Re: [BUG] oops in net_rx_action on 64-bit powerpc

[David Miller]

From: "Chris Friesen" <cfriesen@nortel.com>
Date: Fri, 24 Oct 2008 17:39:00 -0600

> So...it would appear that the NAPI code is somehow buggy, and
> 6ba33ac should probably be reverted until the problem is found and
> fixed.

No I think the problem is simple enough that someone should study the
->poll() routine quickly and audit it's return values.

Re: [BUG] oops in net_rx_action on 64-bit powerpc

[Chris Friesen]

David Miller wrote:
> From: "Chris Friesen" <cfriesen@nortel.com>
> Date: Fri, 24 Oct 2008 17:39:00 -0600
> 
>> So...it would appear that the NAPI code is somehow buggy, and
>> 6ba33ac should probably be reverted until the problem is found and
>> fixed.
> 
> No I think the problem is simple enough that someone should study the
> ->poll() routine quickly and audit it's return values.

Assuming that amd8111e_rx_poll() is the proper routine, there is only one exit 
point, and it returns "num_rx_pkt".  This variable is initialized to zero and 
increments for each packet sent up the stack.

Chris

Re: [BUG] oops in net_rx_action on 64-bit powerpc

[David Miller]

From: "Chris Friesen" <cfriesen@nortel.com>
Date: Fri, 24 Oct 2008 23:42:48 -0600

> David Miller wrote:
> > From: "Chris Friesen" <cfriesen@nortel.com>
> > Date: Fri, 24 Oct 2008 17:39:00 -0600
> > 
> >> So...it would appear that the NAPI code is somehow buggy, and
> >> 6ba33ac should probably be reverted until the problem is found and
> >> fixed.
> > No I think the problem is simple enough that someone should study the
> > ->poll() routine quickly and audit it's return values.
>
> Assuming that amd8111e_rx_poll() is the proper routine, there is
> only one exit point, and it returns "num_rx_pkt".  This variable is
> initialized to zero and increments for each packet sent up the
> stack.

The problematic case in this routine seem to be when the driver
processes exactly "budget" number of packets.

In this case it should not call __netif_rx_complete(), it should instead
use the rx_not_empty label.

Probably the simplest fix is to get rid of the rx_not_empty label and
protect the entire:

	/* Receive descriptor is empty now */
	spin_lock_irqsave(&lp->lock, flags);
	__netif_rx_complete(dev, napi);
	writel(VAL0|RINTEN0, mmio + INTEN0);
	writel(VAL2 | RDMD0, mmio + CMD0);
	spin_unlock_irqrestore(&lp->lock, flags);

code block with a test such as:

	if (rx_pkt_limit > 0)

(yes, greater than zero, not >= 0)

then replace the rx_not_empty goto with a simple break.

Re: [BUG] oops in net_rx_action on 64-bit powerpc

[Chris Friesen]

David Miller wrote:


 > Probably the simplest fix is to get rid of the rx_not_empty label and
 > protect the entire:
 >
 > 	/* Receive descriptor is empty now */
 > 	spin_lock_irqsave(&lp->lock, flags);
 > 	__netif_rx_complete(dev, napi);
 > 	writel(VAL0|RINTEN0, mmio + INTEN0);
 > 	writel(VAL2 | RDMD0, mmio + CMD0);
 > 	spin_unlock_irqrestore(&lp->lock, flags);
 >
 > code block with a test such as:
 >
 > 	if (rx_pkt_limit > 0)
 >
 > (yes, greater than zero, not >= 0)
 >
 > then replace the rx_not_empty goto with a simple break.


Are you sure about that?  Doing that, if we "--rx_pkt_limit < 0" we'll only 
break out of the inner while loop.  We then check then interrupt status 
register and potentially loop through the do/while loop again (maybe 
decrementing rx_pkt_limit again) even though we've used up our budget.

If I leave the label and jump and just add the "rx_pkt_limit > 0" test, it 
seems to work.

Chris


From: Chris Friesen    <cfriesen@nortel.com>
Subject: [PATCH] fix amd8111e rx return code

The amd8111e rx poll routine currently mishandles the case when we process
exactly the number of packets specified in the budget.

This patch is basically as suggested by David Miller.

Signed-off-by: Chris Friesen <cfriesen@nortel.com>

diff --git a/drivers/net/amd8111e.c b/drivers/net/amd8111e.c
index c54967f..ba1be0b 100644
--- a/drivers/net/amd8111e.c
+++ b/drivers/net/amd8111e.c
@@ -833,12 +833,14 @@ static int amd8111e_rx_poll(struct napi_struct *napi, 
int budget)

  	} while(intr0 & RINT0);

-	/* Receive descriptor is empty now */
-	spin_lock_irqsave(&lp->lock, flags);
-	__netif_rx_complete(dev, napi);
-	writel(VAL0|RINTEN0, mmio + INTEN0);
-	writel(VAL2 | RDMD0, mmio + CMD0);
-	spin_unlock_irqrestore(&lp->lock, flags);
+	if (rx_pkt_limit > 0) {
+		/* Receive descriptor is empty now */
+		spin_lock_irqsave(&lp->lock, flags);
+		__netif_rx_complete(dev, napi);
+		writel(VAL0|RINTEN0, mmio + INTEN0);
+		writel(VAL2 | RDMD0, mmio + CMD0);
+		spin_unlock_irqrestore(&lp->lock, flags);
+	}

  rx_not_empty:
  	return num_rx_pkt;

Re: [BUG] oops in net_rx_action on 64-bit powerpc

[David Miller]

From: "Chris Friesen" <cfriesen@nortel.com>
Date: Mon, 27 Oct 2008 18:13:54 -0600

> [PATCH] fix amd8111e rx return code
> 
> The amd8111e rx poll routine currently mishandles the case when we process
> exactly the number of packets specified in the budget.
> 
> This patch is basically as suggested by David Miller.
> 
> Signed-off-by: Chris Friesen <cfriesen@nortel.com>

I had to apply this by hand because your email client heavily
corrupted the patch.

Thanks.

Re: [BUG] oops in net_rx_action on 64-bit powerpc

[Chris Friesen]

David Miller wrote:
> From: "Chris Friesen" <cfriesen@nortel.com>
> Date: Mon, 27 Oct 2008 18:13:54 -0600
> 
> 
>>[PATCH] fix amd8111e rx return code
>>
>>The amd8111e rx poll routine currently mishandles the case when we process
>>exactly the number of packets specified in the budget.
>>
>>This patch is basically as suggested by David Miller.
>>
>>Signed-off-by: Chris Friesen <cfriesen@nortel.com>
> 
> 
> I had to apply this by hand because your email client heavily
> corrupted the patch.

Crap.  Sorry about that, it looks like I forgot to set my line-wrap to 0 
before creating the email.  @#$#% Thunderbird.

Chris