FCC Ethernet startup crash

FCC Ethernet startup crash

[Rune Torgersen]

I've been running into the old problem of the kernel ip stack crasheing
the kernel, because of early packets again.

This has been mentioned on this list several times before, but the
worarounds that used to work doesn't seem to work anymore.
(like changing the order of things in ip_init())

As far as I can tell, it is because the FCC enet drivers starts to
receive packets way before the ip stack is initialized.

Does anybody have any good ideas on how to fix this?

Kernel: 2.6.12-rc2 on a MPC8265
I can almost reliably recreate this by flood-pinging the board while the
kernel is booting.


The crach is as follows:
eth0: FCC ENET Version 0.3, 00:30:d7:00:01:08
eth1: FCC ENET Version 0.3, 00:30:d7:00:01:09
NET: Registered protocol family 2
IP: routing cache hash table of 4096 buckets, 32Kbytes
Oops: kernel access of bad area, sig: 11 [#1]
PREEMPT
NIP: A0141440 LR: A012ACA0 SP: AFFB1D80 REGS: affb1cd0 TRAP: 0300    Not
tainted
MSR: 00009032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
DAR: 00000004, DSISR: 20000000
TASK =3D afde5ae0[1] 'swapper' THREAD: affb0000
Last syscall: 120
GPR00: 00000000 AFFB1D80 AFDE5AE0 A07069E0 AFF44000 A01E2C04 00001032
00000002
GPR08: A01E2C04 A0200000 AFFB0000 00000000 8010C082 000000B1 10000000
007FFF1E
GPR16: 00000000 00000001 007FFF00 0FFFA6F0 00000000 FFFFFFFF 00000003
0FFA0BD0
GPR24: 00000000 FFFB72C3 A01E1324 00000040 AFFB1DF8 A07069E0 A07069E0
A07069E0
Call trace: [a012aca0]  [a012ae58]  [a012afd8]  [a001b30c]  [a001b374]
[a001b464]  [a0005a60]  [a000480c]  [a01fcfcc]  [a01fd2bc]
[a01fddd0]  [a01ec718]  [a01ec7d8]  [a0003a20]  [a0006b44]
Kernel panic - not syncing: Aiee, killing interrupt handler!
 <0>Rebooting in 180 seconds..

Call trace:
$ call2sym
Ready for call trace list.  <ctrl-d> on a blank line when done.

[A0141440] [a012aca0]  [a012ae58]  [a012afd8]  [a001b30c]  [a001b374]
[a001b464]  [a0005a60]  [a000480c]  [a01fcfcc]  [a01fd2bc]
[a01fddd0]  [a01ec718]  [a01ec7d8]  [a0003a20]  [a0006b44]

Processing...

Address         Function

A0141440        ip_rcv
a012aca0        netif_receive_skb
a012ae58        process_backlog
a012afd8        net_rx_action
a001b30c        __do_softirq
a001b374        do_softirq
a001b464        irq_exit
a0005a60        do_IRQ
a000480c        ret_from_except
a01fcfcc        ip_rt_init
a01fd2bc        ip_init
a01ec718        do_initcalls
a01ec7d8        do_basic_setup
a0003a20        init
a0006b44        kernel_thread
[runet@pib linux-innsys]$

Gdb list of excact line=20

(gdb) list *0xA0141440
0xa0141440 is in ip_rcv (net/ipv4/ip_input.c:370).
365              * that it receives, do not try to analyse it.
366              */
367             if (skb->pkt_type =3D=3D PACKET_OTHERHOST)
368                     goto drop;
369
370             IP_INC_STATS_BH(IPSTATS_MIB_INRECEIVES);
371
372             if ((skb =3D skb_share_check(skb, GFP_ATOMIC)) =3D=3D =
NULL) {
373                     IP_INC_STATS_BH(IPSTATS_MIB_INDISCARDS);
374                     goto out;

Rune Torgersen
System Developer
Innovative Systems LLC
1000 Innovative Drive
Mitchell, SD 57301
Ph: 605-995-6120
www.innovsys.com

Re: FCC Ethernet startup crash

[Stefan Nickl]

On Tue, 2005-04-05 at 10:37 -0500, Rune Torgersen wrote:
> I've been running into the old problem of the kernel ip stack crasheing
> the kernel, because of early packets again.
> 
> This has been mentioned on this list several times before, but the
> worarounds that used to work doesn't seem to work anymore.
> (like changing the order of things in ip_init())
> 
> As far as I can tell, it is because the FCC enet drivers starts to
> receive packets way before the ip stack is initialized.
> 
> Does anybody have any good ideas on how to fix this?

Maybe you want to try my version:

http://ozlabs.org/ppc32-patches/patch.pl?id=14

Or is it equal to your (no longer working) solution?

-- 
Stefan Nickl
Kontron Modular Computers

RE: FCC Ethernet startup crash

[Rune Torgersen]

Ahhh....
I see what happened....
Basically only the last part of that patch has been applied to the 2.6
kernels..

fcc_restart gets called both in init_fcc_startup *and* fcc_open.
=20
I'll try to remove the fcc_restart in init_fcc_startup, and see what
happens.


> -----Original Message-----
> From: Stefan Nickl [mailto:Stefan.Nickl@kontron.com]=20
> Sent: Tuesday, April 05, 2005 10:57
> To: Rune Torgersen
> Cc: linuxppc-embedded
> Subject: Re: FCC Ethernet startup crash
>=20
> On Tue, 2005-04-05 at 10:37 -0500, Rune Torgersen wrote:
> > I've been running into the old problem of the kernel ip=20
> stack crasheing
> > the kernel, because of early packets again.
> >=20
> > This has been mentioned on this list several times before, but the
> > worarounds that used to work doesn't seem to work anymore.
> > (like changing the order of things in ip_init())
> >=20
> > As far as I can tell, it is because the FCC enet drivers starts to
> > receive packets way before the ip stack is initialized.
> >=20
> > Does anybody have any good ideas on how to fix this?
>=20
> Maybe you want to try my version:
>=20
> http://ozlabs.org/ppc32-patches/patch.pl?id=3D14
>=20
> Or is it equal to your (no longer working) solution?
>=20
> --=20
> Stefan Nickl
> Kontron Modular Computers
>=20
>=20
>=20
>=20

RE: FCC Ethernet startup crash

[Rune Torgersen]

Thanks....

That whole patch works beautifully.
Had to apply the whole thing (so fcc_restart gets called in the start,
and end of fcc_open) for it to work.

Without calling fcc_restart in the start, I get lots of "eth0: tx queue
full!." messages.

> -----Original Message-----
> From: Stefan Nickl [mailto:Stefan.Nickl@kontron.com]=20
> Sent: Tuesday, April 05, 2005 10:57
> To: Rune Torgersen
> Cc: linuxppc-embedded
> Subject: Re: FCC Ethernet startup crash
>=20
> On Tue, 2005-04-05 at 10:37 -0500, Rune Torgersen wrote:
> > I've been running into the old problem of the kernel ip=20
> stack crasheing
> > the kernel, because of early packets again.
> >=20
> > This has been mentioned on this list several times before, but the
> > worarounds that used to work doesn't seem to work anymore.
> > (like changing the order of things in ip_init())
> >=20
> > As far as I can tell, it is because the FCC enet drivers starts to
> > receive packets way before the ip stack is initialized.
> >=20
> > Does anybody have any good ideas on how to fix this?
>=20
> Maybe you want to try my version:
>=20
> http://ozlabs.org/ppc32-patches/patch.pl?id=3D14
>=20
> Or is it equal to your (no longer working) solution?
>=20
> --=20
> Stefan Nickl
> Kontron Modular Computers
>=20
>=20
>=20
>=20

Re: FCC Ethernet startup crash

[Dan Malek]

On Apr 5, 2005, at 12:41 PM, Rune Torgersen wrote:

> Without calling fcc_restart in the start, I get lots of "eth0: tx queue
> full!." messages.

Do you tftp boot your kernel using the boot rom?  Does the
boot rom disable the Ethernet before calling the kernel?

Thanks.


	-- Dan

RE: FCC Ethernet startup crash

[Rune Torgersen]

> From: Dan Malek [mailto:dan@embeddededge.com]=20
> On Apr 5, 2005, at 12:41 PM, Rune Torgersen wrote:
>=20
> > Without calling fcc_restart in the start, I get lots of=20
> "eth0: tx queue
> > full!." messages.
>=20
> Do you tftp boot your kernel using the boot rom?  Does the
> boot rom disable the Ethernet before calling the kernel?

Yes, I tftp the kernel, using U-Boot (1.1.2 CVS Head as of 11-20-04).
Haven't looked at the code, but I think it disablesd the FCC Ethernets
(Have seen discussions about htat on u-boot mailing list before)

The patch Stefan Nickl gave to me works perfectly, and all it does is
basically move the fcc_restart.
The kernel was still calling fcc_restart twiche per FCC at startup (one
time on init_fcc_startup, and then again in fcc_open), now it just calls
it twice in fcc_open. Seems to work.

Re: FCC Ethernet startup crash

[Pantelis Antoniou]

Rune Torgersen wrote:
>>From: Dan Malek [mailto:dan@embeddededge.com] 
>>On Apr 5, 2005, at 12:41 PM, Rune Torgersen wrote:
>>
>>
>>>Without calling fcc_restart in the start, I get lots of 
>>
>>"eth0: tx queue
>>
>>>full!." messages.
>>
>>Do you tftp boot your kernel using the boot rom?  Does the
>>boot rom disable the Ethernet before calling the kernel?
> 
> 
> Yes, I tftp the kernel, using U-Boot (1.1.2 CVS Head as of 11-20-04).
> Haven't looked at the code, but I think it disablesd the FCC Ethernets
> (Have seen discussions about htat on u-boot mailing list before)
> 

U-boot definately disables the ethernets. Or at least it must.

> The patch Stefan Nickl gave to me works perfectly, and all it does is
> basically move the fcc_restart.
> The kernel was still calling fcc_restart twiche per FCC at startup (one
> time on init_fcc_startup, and then again in fcc_open), now it just calls
> it twice in fcc_open. Seems to work.
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded
> 
> 
> 

Regards

Pantelis

Re: FCC Ethernet startup crash

[Tom Rini]

On Tue, Apr 05, 2005 at 01:53:36PM -0500, Rune Torgersen wrote:
> > From: Dan Malek [mailto:dan@embeddededge.com] 
> > On Apr 5, 2005, at 12:41 PM, Rune Torgersen wrote:
> > 
> > > Without calling fcc_restart in the start, I get lots of 
> > "eth0: tx queue
> > > full!." messages.
> > 
> > Do you tftp boot your kernel using the boot rom?  Does the
> > boot rom disable the Ethernet before calling the kernel?
> 
> Yes, I tftp the kernel, using U-Boot (1.1.2 CVS Head as of 11-20-04).
> Haven't looked at the code, but I think it disablesd the FCC Ethernets
> (Have seen discussions about htat on u-boot mailing list before)
> 
> The patch Stefan Nickl gave to me works perfectly, and all it does is
> basically move the fcc_restart.
> The kernel was still calling fcc_restart twiche per FCC at startup (one
> time on init_fcc_startup, and then again in fcc_open), now it just calls
> it twice in fcc_open. Seems to work.

Would someone please post what's needed vs current'ish 2.6 so we can get
the proper fix up & in? :)  With the Signed-off-by & all that.. Thanks.

-- 
Tom Rini
http://gate.crashing.org/~trini/

Re: FCC Ethernet startup crash

[Stefan Nickl]

[-- Attachment #1: Type: text/plain, Size: 1030 bytes --]

On Thu, 2005-04-07 at 09:36 -0700, Tom Rini wrote:
> > The patch Stefan Nickl gave to me works perfectly, and all it does is
> > basically move the fcc_restart.
> > The kernel was still calling fcc_restart twiche per FCC at startup (one
> > time on init_fcc_startup, and then again in fcc_open), now it just calls
> > it twice in fcc_open. Seems to work.
> 
> Would someone please post what's needed vs current'ish 2.6 so we can get
> the proper fix up & in? :)  With the Signed-off-by & all that.. Thanks.

I took a little longer because I was not happy with the simple
cut'n'paste version. I believe the actual problem is that fcc_restart()
mustn't be called before netif_start_queue(). So the original version
only shortens the race period, it does not actually resolve it.

Unfortunately I'm deprived of suitable hardware as the focus of my work
has shifted in the last months, so I urge everyone out there to give it
a try.

Signed-off-by: Stefan Nickl <Stefan.Nickl@kontron.com>

-- 
Stefan Nickl
Kontron Modular Computers


[-- Attachment #2: fcc_enet_startup_crash_v2.patch --]
[-- Type: text/x-patch, Size: 1828 bytes --]

--- linux-2.5/arch/ppc/8260_io/fcc_enet.c.bak	2005-04-08 09:37:40.000000000 +0200
+++ linux-2.5/arch/ppc/8260_io/fcc_enet.c	2005-04-08 12:01:55.000000000 +0200
@@ -2158,16 +2158,6 @@ init_fcc_startup(fcc_info_t *fip, struct
 	*(volatile uint *)(BCSR_ADDR + 12) &= ~BCSR3_FETHIEN2;
 	*(volatile uint *)(BCSR_ADDR + 12) |=  BCSR3_FETH2_RST;
 #endif
-
-#if defined(CONFIG_USE_MDIO) || defined(CONFIG_TQM8260)
-	/* start in full duplex mode, and negotiate speed
-	 */
-	fcc_restart (dev, 1);
-#else
-	/* start in half duplex mode
-	 */
-	fcc_restart (dev, 0);
-#endif
 }
 
 #ifdef	CONFIG_USE_MDIO
@@ -2372,24 +2362,28 @@ fcc_enet_open(struct net_device *dev)
 	fep->sequence_done = 0;
 	fep->link = 0;
 
-	if (fep->phy) {
-		fcc_restart(dev, 0);	/* always start in half-duplex */
-		mii_do_cmd(dev, fep->phy->ack_int);
-		mii_do_cmd(dev, fep->phy->config);
-		mii_do_cmd(dev, phy_cmd_config);  /* display configuration */
-		while(!fep->sequence_done)
-			schedule();
-
-		mii_do_cmd(dev, fep->phy->startup);
-		netif_start_queue(dev);
-		return 0;		/* Success */
-	}
-	return -ENODEV;		/* No PHY we understand */
+	if (!fep->phy)
+		return -ENODEV;		/* No PHY we understand */
+
+	mii_do_cmd(dev, fep->phy->ack_int);
+	mii_do_cmd(dev, fep->phy->config);
+	mii_do_cmd(dev, phy_cmd_config);  /* display configuration */
+	while(!fep->sequence_done)
+		schedule();
+
+	mii_do_cmd(dev, fep->phy->startup);
 #else
 	fep->link = 1;
-	fcc_restart(dev, 0);	/* always start in half-duplex */
-	netif_start_queue(dev);
-	return 0;					/* Always succeed */
 #endif	/* CONFIG_USE_MDIO */
+
+	netif_start_queue(dev);
+
+#if defined(CONFIG_USE_MDIO) || defined(CONFIG_TQM8260)
+	fcc_restart (dev, 1);	/* start in full duplex mode */
+#else
+	fcc_restart (dev, 0);	/* start in half duplex mode */
+#endif
+
+	return 0;		/* Always succeed */
 }
 

RE: FCC Ethernet startup crash

[Rune Torgersen]

I'm not sure what is going on, but that patch causes me to get "eth0: tx
queue full!." messages as soon as I try to use the ethernet
(Same thing happened if I just tried to remove the fcc_restart from
init_fcc_startup)

> -----Original Message-----
> From: Stefan Nickl [mailto:Stefan.Nickl@kontron.com]=20
> Sent: Friday, April 08, 2005 05:49
> To: Tom Rini
> Cc: Rune Torgersen; linuxppc-embedded
> Subject: Re: FCC Ethernet startup crash
>=20
> On Thu, 2005-04-07 at 09:36 -0700, Tom Rini wrote:
> > > The patch Stefan Nickl gave to me works perfectly, and=20
> all it does is
> > > basically move the fcc_restart.
> > > The kernel was still calling fcc_restart twiche per FCC=20
> at startup (one
> > > time on init_fcc_startup, and then again in fcc_open),=20
> now it just calls
> > > it twice in fcc_open. Seems to work.
> >=20
> > Would someone please post what's needed vs current'ish 2.6=20
> so we can get
> > the proper fix up & in? :)  With the Signed-off-by & all=20
> that.. Thanks.
>=20
> I took a little longer because I was not happy with the simple
> cut'n'paste version. I believe the actual problem is that=20
> fcc_restart()
> mustn't be called before netif_start_queue(). So the original version
> only shortens the race period, it does not actually resolve it.
>=20
> Unfortunately I'm deprived of suitable hardware as the focus=20
> of my work
> has shifted in the last months, so I urge everyone out there=20
> to give it
> a try.
>=20
> Signed-off-by: Stefan Nickl <Stefan.Nickl@kontron.com>
>=20
> --=20
> Stefan Nickl
> Kontron Modular Computers
>=20
>=20