Kernel boot hangs by parallel flood pings

Kernel boot hangs by parallel flood pings

[Steffen Rumler]

I'm using a 2.4.20 kernel (PPC).

When I run multiple 'ping -f' directed to my board
during the kernel boots, sometimes it hangs with
the following messages:


   Memory BAT mapping: BAT2=64Mb, BAT3=32Mb, residual: 84Mb
   Linux version 2.4.20-rthal5 (ru@styx) (gcc version 2.95.3 20010111 (prerelease/franzo/20010111))
   #9 Tue Jan 11 17:09:22 CET 2005
   On node 0 totalpages: 29859
   zone(0): 29859 pages.
   zone(1): 0 pages.
   zone(2): 0 pages.
   Kernel command line: bootdev=aps0 ip=218.1.36.70:218.1.36.47:218.1.36.254:255.255.255.0::eth0:off
   panic=1 mem=119436k
   Calibrating delay loop... 194.15 BogoMIPS
   ...
   NET4: Linux TCP/IP 1.0 for NET4.0
   IP Protocols: ICMP, UDP, TCP
   IP: routing cache ha
                 ^^^^^^^
                   the kernel boot hangs here


I have checked this with the BDI2000 and found that the
kernel loops anywhere inside ip_route_input().

I think this is because the ethernet driver (fcc_enet.c)
enables the hardware too already inside it's init routine (fec_enet_init()).

In this way, frames may be received and forwarded to the upper
layers with netif_rx(), before the TCP/IP was initialized.

I suggest to delay the lines

   /* Enable transmit/receive */
   fccp->fcc_gfmr |= FCC_GFMR_ENR | FCC_GFMR_ENT;

to the open entry point (fcc_enet_open()), which is
triggered by 'ifconfig up' in the following way:


   if(!((fccp->fcc_gfmr) & FCC_GFMR_ENR))
      fccp->fcc_gfmr |= (FCC_GFMR_ENR | FCC_GFMR_ENT);

By the way, fcc_enet_close() does _NOT_ stop the hardware.
Hopefully, this is not a real problem.


Steffen

-- 


--------------------------------------------------------------

Steffen Rumler
ICN CP D NT SW 3
Siemens AG
Hofmannstr. 51                 Email: Steffen.Rumler@siemens.com
D-81359 Munich                 Phone: +49 89 722-44061
Germany                        Fax  : +49 89 722-36703

--------------------------------------------------------------

Re: Kernel boot hangs by parallel flood pings

[Stefan Nickl]

[-- Attachment #1: Type: text/plain, Size: 632 bytes --]

On Wed, 2005-01-12 at 09:38 +0100, Steffen Rumler wrote:
> I'm using a 2.4.20 kernel (PPC).
> 
> When I run multiple 'ping -f' directed to my board
> during the kernel boots, sometimes it hangs with
> the following messages:

> I have checked this with the BDI2000 and found that the
> kernel loops anywhere inside ip_route_input().

This is a long-standing bug in the cpm and cmp2 drivers.
I think it's also present in 2.6 as far as I remember.

I sent my own fix some time ago, you may find it in the
archives as "[patch] Race condition #2 in arch/ppc/cpm2_io/fcc_enet.c".

Greetings,

-- 
Stefan Nickl
Kontron Modular Computers


[-- Attachment #2: fcc_enet-mdio_race.patch --]
[-- Type: text/x-patch, Size: 412 bytes --]

--- linuxppc-2.4/arch/ppc/cpm2_io/fcc_enet.c	2004-08-09 08:32:28.000000000 +0200
+++ fcc_enet.c	2004-11-25 18:28:56.000000000 +0100
@@ -2095,10 +2095,10 @@ mii_send_receive(fcc_info_t *fip, uint c
 		for (i = 0, off = 15; i < 16; i++, off--)
 		{
 			FCC_PDATC_MDC(1);
+			udelay(1);
 			retval <<= 1;
 			if (io->iop_pdatc & fip->fc_mdio)
 				retval++;
-			udelay(1);
 			FCC_PDATC_MDC(0);
 			udelay(1);
 		}

[-- Attachment #3: fcc_enet-startup_race.patch --]
[-- Type: text/x-patch, Size: 969 bytes --]

--- linuxppc-2.4/arch/ppc/cpm2_io/fcc_enet.c	2004-08-09 08:32:28.000000000 +0200
+++ fcc_enet.c	2004-11-25 18:41:22.000000000 +0100
@@ -2008,16 +2008,6 @@ init_fcc_startup(fcc_info_t *fip, struct
         *(volatile uint *)(BCSR_ADDR + 4) &= ~BCSR1_FETHIEN;
         *(volatile uint *)(BCSR_ADDR + 4) |=  BCSR1_FETH_RST;
 #endif
-
-#if defined(CONFIG_USE_MDIO) || defined(CONFIG_TQM8260)
-	/* start in full duplex mode, and negotiate speed
-	 */
-	fcc_restart (dev, 1);
-#else
-	/* start in half duplex mode
-	 */
-	fcc_restart (dev, 0);
-#endif
 }
 
 #ifdef	CONFIG_USE_MDIO
@@ -2155,6 +2145,16 @@ fcc_enet_open(struct net_device *dev)
 {
 	struct fcc_enet_private *fep = dev->priv;
 
+#if defined(CONFIG_USE_MDIO) || defined(CONFIG_TQM8260)
+	/* start in full duplex mode, and negotiate speed
+	 */
+	fcc_restart (dev, 1);
+#else
+	/* start in half duplex mode
+	 */
+	fcc_restart (dev, 0);
+#endif
+
 #ifdef	CONFIG_USE_MDIO
 	fep->sequence_done = 0;
 	fep->link = 0;

RE: Kernel boot hangs by parallel flood pings

[Rune Torgersen]

I have had the problem before (and so has several other people)

This is a reply I got on the old Linux PPC mailinglist last year
(from Gerard Guevel [gguevel@interfaceconcept.com])
---- Original message -----
RE: FCC ethernets not stable

I received the right solution from Jean-Denis Boyer (thanks to him...)
>>
It registers the IPv4 handling functions AFTER initializing the IP
routing
stuff.
Early packets could enter the stack and crash the system.

--- Kernel.old/net/ipv4/ip_output.c     Thu Jun  5 09:00:39 2003
+++ Kernel/net/ipv4/ip_output.c Thu Jun  5 09:00:28 2003
@@ -1087,11 +1087,11 @@

 void __init ip_init(void)
 {
-       dev_add_pack(&ip_packet_type);
-
        ip_rt_init();
        inet_initpeers();

+       dev_add_pack(&ip_packet_type);
+
 #ifdef CONFIG_IP_MULTICAST
        proc_net_create("igmp", 0, ip_mc_procinfo);
 #endif
>>

I never saw this problem after installing this patch.

Regards
Gerard Guevel=20

> -----Original Message-----
> From: linuxppc-embedded-bounces@ozlabs.org=20
> [mailto:linuxppc-embedded-bounces@ozlabs.org] On Behalf Of=20
> Steffen Rumler
> Sent: Wednesday, January 12, 2005 02:38
> To: linuxppc
> Subject: Kernel boot hangs by parallel flood pings
>=20
> I'm using a 2.4.20 kernel (PPC).
>=20
> When I run multiple 'ping -f' directed to my board
> during the kernel boots, sometimes it hangs with
> the following messages:
>=20
>=20
>    Memory BAT mapping: BAT2=3D64Mb, BAT3=3D32Mb, residual: 84Mb
>    Linux version 2.4.20-rthal5 (ru@styx) (gcc version 2.95.3=20
> 20010111 (prerelease/franzo/20010111))
>    #9 Tue Jan 11 17:09:22 CET 2005
>    On node 0 totalpages: 29859
>    zone(0): 29859 pages.
>    zone(1): 0 pages.
>    zone(2): 0 pages.
>    Kernel command line: bootdev=3Daps0=20
> ip=3D218.1.36.70:218.1.36.47:218.1.36.254:255.255.255.0::eth0:off
>    panic=3D1 mem=3D119436k
>    Calibrating delay loop... 194.15 BogoMIPS
>    ...
>    NET4: Linux TCP/IP 1.0 for NET4.0
>    IP Protocols: ICMP, UDP, TCP
>    IP: routing cache ha
>                  ^^^^^^^
>                    the kernel boot hangs here
>=20
>=20
> I have checked this with the BDI2000 and found that the
> kernel loops anywhere inside ip_route_input().
>=20
> I think this is because the ethernet driver (fcc_enet.c)
> enables the hardware too already inside it's init routine=20
> (fec_enet_init()).
>=20
> In this way, frames may be received and forwarded to the upper
> layers with netif_rx(), before the TCP/IP was initialized.
>=20
> I suggest to delay the lines
>=20
>    /* Enable transmit/receive */
>    fccp->fcc_gfmr |=3D FCC_GFMR_ENR | FCC_GFMR_ENT;
>=20
> to the open entry point (fcc_enet_open()), which is
> triggered by 'ifconfig up' in the following way:
>=20
>=20
>    if(!((fccp->fcc_gfmr) & FCC_GFMR_ENR))
>       fccp->fcc_gfmr |=3D (FCC_GFMR_ENR | FCC_GFMR_ENT);
>=20
> By the way, fcc_enet_close() does _NOT_ stop the hardware.
> Hopefully, this is not a real problem.
>=20
>=20
> Steffen
>=20
> --=20
>=20
>=20
> --------------------------------------------------------------
>=20
> Steffen Rumler
> ICN CP D NT SW 3
> Siemens AG
> Hofmannstr. 51                 Email: Steffen.Rumler@siemens.com
> D-81359 Munich                 Phone: +49 89 722-44061
> Germany                        Fax  : +49 89 722-36703
>=20
> --------------------------------------------------------------
>=20
>=20
>=20
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded
>=20
>=20

Re: Kernel boot hangs by parallel flood pings

[Wolfgang Denk]

In message <DCEAAC0833DD314AB0B58112AD99B93B85928F@ismail.innsys.innovsys.com> you wrote:
> I have had the problem before (and so has several other people)
> 
> This is a reply I got on the old Linux PPC mailinglist last year
> (from Gerard Guevel [gguevel@interfaceconcept.com])

According to my records the patch was by Jean-Denis Boyer, 05 Jun
2003.

> I never saw this problem after installing this patch.

I can confirm this, too.

Steffen: this is PatchSet 251 (Date: 2004/06/07 20:05:39) in our  CVS
tree.

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Yes, it's a technical challenge, and  you  have  to  kind  of  admire
people  who go to the lengths of actually implementing it, but at the
same time you wonder about their IQ...
         --  Linus Torvalds in <5phda5$ml6$1@palladium.transmeta.com>

RE: Kernel boot hangs by parallel flood pings

[Ho Lee]

I met the same problem in CPM driver, and there could be many 
solutions. I think the right place to fix is the fec driver. 
The fec driver registers interrupt handler during the 
initialization not when the device is opened, so it may
receive packets even the device is not up and running (also 
before the network subsystem is initialized) My fix was to 
ignore the received packets if the driver is not opened. 

--- linux-2.4.26.orig/arch/ppc/8xx_io/fec.c
+++ linux-2.4.26/arch/ppc/8xx_io/fec.c 
@@ -587,6 +587,9 @@
                goto rx_processing_done;
        }
 
+       if (!netif_running(dev))
+           goto rx_processing_done;
+
        /* Process the incoming frame.
         */
        fep->stats.rx_packets++;


Regards,
Ho

-----Original Message-----
From: linuxppc-embedded-bounces@ozlabs.org
[mailto:linuxppc-embedded-bounces@ozlabs.org] On Behalf Of Rune Torgersen
Sent: Wednesday, January 12, 2005 7:32 AM
To: Steffen Rumler; linuxppc
Subject: RE: Kernel boot hangs by parallel flood pings

--- Kernel.old/net/ipv4/ip_output.c     Thu Jun  5 09:00:39 2003
+++ Kernel/net/ipv4/ip_output.c Thu Jun  5 09:00:28 2003
@@ -1087,11 +1087,11 @@

 void __init ip_init(void)
 {
-       dev_add_pack(&ip_packet_type);
-
        ip_rt_init();
        inet_initpeers();

+       dev_add_pack(&ip_packet_type);
+
 #ifdef CONFIG_IP_MULTICAST
        proc_net_create("igmp", 0, ip_mc_procinfo);
 #endif
>>

RE: Kernel boot hangs by parallel flood pings

[Johns Daniel]

I have had better luck with the following fcc_enet.c patch (against the =
ELDK 2.4.4 kernel):

~~~~~~~~~~~~~
1.) In static void mii_display_status(struct net_device *dev):
=20
 	if (!fep->link && !fep->old_link) {
 		/* Link is still down - don't print anything */
 		return;
 	}
=20
	printk("%s: status: ", dev->name);
=20
 	if (!fep->link) {
+        netif_carrier_off(dev);
		printk("link down");
 	} else {
+        netif_carrier_on(dev);
		printk("link up");
=20
 		switch(s & PHY_STAT_SPMASK) {
~~~~~~~~~~~~~

2.) In fcc_enet_open(struct net_device *dev):

 		mii_do_cmd(dev, fep->phy->startup);
 #ifdef	CONFIG_PM826
 		/* Read the autonegotiation results */
 		mii_do_cmd(dev, fep->phy->ack_int);
 		mii_do_cmd(dev, phy_cmd_relink);
 #endif	/* CONFIG_PM826 */
+        if (fep->link) {
+            netif_carrier_on(dev);
+        } else {
+            netif_carrier_off(dev);
+        }
 		netif_start_queue(dev);
 		return 0;		/* Success */
 	}
 	return -ENODEV;		/* No PHY we understand */
 #else
 	fep->link =3D 1;
~~~~~~~~~~~~~

This communicates the link status to the kernel TCP/IP stack. Obviously, =
this assumes that MDIO is working!

-- Johns Daniel, Advent Networks, Inc.

-----Original Message-----
From:	linuxppc-embedded-bounces@ozlabs.org on behalf of Ho Lee
Sent:	Wed 1/12/2005 6:14 PM
To:	'linuxppc'
Cc:	'Steffen Rumler'
Subject:	RE: Kernel boot hangs by parallel flood pings



-----Original Message-----
From:	linuxppc-embedded-bounces@ozlabs.org on behalf of Ho Lee
Sent:	Wed 1/12/2005 6:14 PM
To:	'linuxppc'
Cc:	'Steffen Rumler'
Subject:	RE: Kernel boot hangs by parallel flood pings

I met the same problem in CPM driver, and there could be many=20
solutions. I think the right place to fix is the fec driver.=20
The fec driver registers interrupt handler during the=20
initialization not when the device is opened, so it may
receive packets even the device is not up and running (also=20
before the network subsystem is initialized) My fix was to=20
ignore the received packets if the driver is not opened.=20

--- linux-2.4.26.orig/arch/ppc/8xx_io/fec.c
+++ linux-2.4.26/arch/ppc/8xx_io/fec.c=20
@@ -587,6 +587,9 @@
                goto rx_processing_done;
        }
=20
+       if (!netif_running(dev))
+           goto rx_processing_done;
+
        /* Process the incoming frame.
         */
        fep->stats.rx_packets++;


Regards,
Ho

-----Original Message-----
From: linuxppc-embedded-bounces@ozlabs.org
[mailto:linuxppc-embedded-bounces@ozlabs.org] On Behalf Of Rune =
Torgersen
Sent: Wednesday, January 12, 2005 7:32 AM
To: Steffen Rumler; linuxppc
Subject: RE: Kernel boot hangs by parallel flood pings

--- Kernel.old/net/ipv4/ip_output.c     Thu Jun  5 09:00:39 2003
+++ Kernel/net/ipv4/ip_output.c Thu Jun  5 09:00:28 2003
@@ -1087,11 +1087,11 @@

 void __init ip_init(void)
 {
-       dev_add_pack(&ip_packet_type);
-
        ip_rt_init();
        inet_initpeers();

+       dev_add_pack(&ip_packet_type);
+
 #ifdef CONFIG_IP_MULTICAST
        proc_net_create("igmp", 0, ip_mc_procinfo);
 #endif
>>


_______________________________________________
Linuxppc-embedded mailing list
Linuxppc-embedded@ozlabs.org
https://ozlabs.org/mailman/listinfo/linuxppc-embedded