sky2 vs netconsole hard hang

sky2 vs netconsole hard hang

[Alexey Dobriyan]

If sky2 interface is netconsoling, then

	ifconfig down; ifconfig up

leads to hard hang after printing

	sky2 eth0: enabling interface

SysRq+b reboots the box, nothing else works (or prints something).

This is with 2.6.28.2 and 2.6.29-rc2-5ee810072175042775e39bdd3eaaa68884c27805.
Most certainly was there since day one.

Haven't investigated further.

[PATCH] sky2: fix hard hang with netconsoling and iface going up

[Alexey Dobriyan]

On Tue, Jan 27, 2009 at 03:01:29PM +0300, Alexey Dobriyan wrote:
> If sky2 interface is netconsoling, then
> 
> 	ifconfig down; ifconfig up
> 
> leads to hard hang after printing
> 
> 	sky2 eth0: enabling interface
> 
> SysRq+b reboots the box, nothing else works (or prints something).
> 
> This is with 2.6.28.2 and 2.6.29-rc2-5ee810072175042775e39bdd3eaaa68884c27805.
> Most certainly was there since day one.
> 
> Haven't investigated further.

Well, duh...


[PATCH] sky2: fix hard hang with netconsoling and iface going up

Printing anything over netconsole before hw is up and running is,
of course, not going to work.

Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
---

 drivers/net/sky2.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -1403,9 +1403,6 @@ static int sky2_up(struct net_device *dev)
 
  	}
 
-	if (netif_msg_ifup(sky2))
-		printk(KERN_INFO PFX "%s: enabling interface\n", dev->name);
-
 	netif_carrier_off(dev);
 
 	/* must be power of 2 */
@@ -1484,6 +1481,9 @@ static int sky2_up(struct net_device *dev)
 	sky2_write32(hw, B0_IMSK, imask);
 
 	sky2_set_multicast(dev);
+
+	if (netif_msg_ifup(sky2))
+		printk(KERN_INFO PFX "%s: enabling interface\n", dev->name);
 	return 0;
 
 err_out:

Re: [PATCH] sky2: fix hard hang with netconsoling and iface going up

[Alexey Dobriyan]

On Tue, Jan 27, 2009 at 04:27:18PM +0300, Alexey Dobriyan wrote:
> On Tue, Jan 27, 2009 at 03:01:29PM +0300, Alexey Dobriyan wrote:
> > If sky2 interface is netconsoling, then
> > 
> > 	ifconfig down; ifconfig up
> > 
> > leads to hard hang after printing
> > 
> > 	sky2 eth0: enabling interface
> > 
> > SysRq+b reboots the box, nothing else works (or prints something).
> > 
> > This is with 2.6.28.2 and 2.6.29-rc2-5ee810072175042775e39bdd3eaaa68884c27805.
> > Most certainly was there since day one.
> > 
> > Haven't investigated further.
> 
> Well, duh...
> 
> 
> [PATCH] sky2: fix hard hang with netconsoling and iface going up
> 
> Printing anything over netconsole before hw is up and running is,
> of course, not going to work.
> 
> Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
> ---
> 
>  drivers/net/sky2.c |    6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> --- a/drivers/net/sky2.c
> +++ b/drivers/net/sky2.c
> @@ -1403,9 +1403,6 @@ static int sky2_up(struct net_device *dev)
>  
>   	}
>  
> -	if (netif_msg_ifup(sky2))
> -		printk(KERN_INFO PFX "%s: enabling interface\n", dev->name);
> -
>  	netif_carrier_off(dev);
>  
>  	/* must be power of 2 */
> @@ -1484,6 +1481,9 @@ static int sky2_up(struct net_device *dev)
>  	sky2_write32(hw, B0_IMSK, imask);
>  
>  	sky2_set_multicast(dev);
> +
> +	if (netif_msg_ifup(sky2))
> +		printk(KERN_INFO PFX "%s: enabling interface\n", dev->name);
>  	return 0;

Oh, this means that most printks on error path in ->ndo_open are broken
and every early enough "I'm up!" printk is broken is well. :-(

Re: [PATCH] sky2: fix hard hang with netconsoling and iface going up

[Stephen Hemminger]

On Tue, 27 Jan 2009 16:35:04 +0300
Alexey Dobriyan <adobriyan@gmail.com> wrote:

> On Tue, Jan 27, 2009 at 04:27:18PM +0300, Alexey Dobriyan wrote:
> > On Tue, Jan 27, 2009 at 03:01:29PM +0300, Alexey Dobriyan wrote:
> > > If sky2 interface is netconsoling, then
> > > 
> > > 	ifconfig down; ifconfig up
> > > 
> > > leads to hard hang after printing
> > > 
> > > 	sky2 eth0: enabling interface
> > > 
> > > SysRq+b reboots the box, nothing else works (or prints something).
> > > 
> > > This is with 2.6.28.2 and 2.6.29-rc2-5ee810072175042775e39bdd3eaaa68884c27805.
> > > Most certainly was there since day one.
> > > 
> > > Haven't investigated further.
> > 
> > Well, duh...
> > 
> > 
> > [PATCH] sky2: fix hard hang with netconsoling and iface going up
> > 
> > Printing anything over netconsole before hw is up and running is,
> > of course, not going to work.
> > 
> > Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
>

Netconsole should be smart enough to not try and print to
a device that is not marked running.

Re: [PATCH] sky2: fix hard hang with netconsoling and iface going up

[David Miller]

From: Stephen Hemminger <shemminger@linux-foundation.org>
Date: Tue, 27 Jan 2009 09:49:28 -0800

> Netconsole should be smart enough to not try and print to
> a device that is not marked running.

I think the hardware is marked running.

This is at the beginning of the driver's ->open() method.

The netpoll layer is in fact checking those states:

static void netpoll_send_skb(struct netpoll *np, struct sk_buff *skb)
{
 ...
	if (!npinfo || !netif_running(dev) || !netif_device_present(dev)) {
		__kfree_skb(skb);
		return;
	}

Re: [PATCH] sky2: fix hard hang with netconsoling and iface going up

[David Miller]

From: Alexey Dobriyan <adobriyan@gmail.com>
Date: Tue, 27 Jan 2009 16:27:18 +0300

> [PATCH] sky2: fix hard hang with netconsoling and iface going up
> 
> Printing anything over netconsole before hw is up and running is,
> of course, not going to work.
> 
> Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>

Alexey, can you get this tested by the reporter to make sure
it really fixes the hang?

Once it's been tested, I'll apply it.

Thank you.

Re: [PATCH] sky2: fix hard hang with netconsoling and iface going up

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Thu, 29 Jan 2009 17:00:43 -0800 (PST)

> From: Alexey Dobriyan <adobriyan@gmail.com>
> Date: Tue, 27 Jan 2009 16:27:18 +0300
> 
> > [PATCH] sky2: fix hard hang with netconsoling and iface going up
> > 
> > Printing anything over netconsole before hw is up and running is,
> > of course, not going to work.
> > 
> > Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
> 
> Alexey, can you get this tested by the reporter to make sure
> it really fixes the hang?
> 
> Once it's been tested, I'll apply it.

Actually, I just read the followup where you indicated that
other debugging printk's in this driver might also have the
same problem.

I'm dropping this until everything is resolved, thanks.

Re: [PATCH] sky2: fix hard hang with netconsoling and iface going up

[Alexey Dobriyan]

On Thu, Jan 29, 2009 at 05:00:43PM -0800, David Miller wrote:
> From: Alexey Dobriyan <adobriyan@gmail.com>
> Date: Tue, 27 Jan 2009 16:27:18 +0300
> 
> > [PATCH] sky2: fix hard hang with netconsoling and iface going up
> > 
> > Printing anything over netconsole before hw is up and running is,
> > of course, not going to work.
> > 
> > Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
> 
> Alexey, can you get this tested by the reporter to make sure
> it really fixes the hang?

I'm the reporter and tester and using this patch successfully. :-)

> Once it's been tested, I'll apply it.

As for other printks, someone with 8139cp should experience the same bug.

Re: [PATCH] sky2: fix hard hang with netconsoling and iface going up

[David Miller]

From: Alexey Dobriyan <adobriyan@gmail.com>
Date: Fri, 30 Jan 2009 14:59:07 +0300

> On Thu, Jan 29, 2009 at 05:00:43PM -0800, David Miller wrote:
> > From: Alexey Dobriyan <adobriyan@gmail.com>
> > Date: Tue, 27 Jan 2009 16:27:18 +0300
> > 
> > > [PATCH] sky2: fix hard hang with netconsoling and iface going up
> > > 
> > > Printing anything over netconsole before hw is up and running is,
> > > of course, not going to work.
> > > 
> > > Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
> > 
> > Alexey, can you get this tested by the reporter to make sure
> > it really fixes the hang?
> 
> I'm the reporter and tester and using this patch successfully. :-)

Then you give me no choice, I have to apply this patch :-)

Thanks!

Re: [PATCH] sky2: fix hard hang with netconsoling and iface going up

[Stephen Hemminger]

On Fri, 30 Jan 2009 13:45:50 -0800 (PST)
David Miller <davem@davemloft.net> wrote:

> From: Alexey Dobriyan <adobriyan@gmail.com>
> Date: Fri, 30 Jan 2009 14:59:07 +0300
> 
> > On Thu, Jan 29, 2009 at 05:00:43PM -0800, David Miller wrote:
> > > From: Alexey Dobriyan <adobriyan@gmail.com>
> > > Date: Tue, 27 Jan 2009 16:27:18 +0300
> > > 
> > > > [PATCH] sky2: fix hard hang with netconsoling and iface going up
> > > > 
> > > > Printing anything over netconsole before hw is up and running is,
> > > > of course, not going to work.
> > > > 
> > > > Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
> > > 
> > > Alexey, can you get this tested by the reporter to make sure
> > > it really fixes the hang?
> > 
> > I'm the reporter and tester and using this patch successfully. :-)
> 
> Then you give me no choice, I have to apply this patch :-)
> 
> Thanks!

Yes, this is a nice harmless change
Acked-by: Stephen Hemminger <shemminger@vyatta.com>

Re: [PATCH] sky2: fix hard hang with netconsoling and iface going up

[Alexey Dobriyan]

On Fri, Jan 30, 2009 at 02:59:07PM +0300, Alexey Dobriyan wrote:
> On Thu, Jan 29, 2009 at 05:00:43PM -0800, David Miller wrote:
> > From: Alexey Dobriyan <adobriyan@gmail.com>
> > Date: Tue, 27 Jan 2009 16:27:18 +0300
> > 
> > > [PATCH] sky2: fix hard hang with netconsoling and iface going up
> > > 
> > > Printing anything over netconsole before hw is up and running is,
> > > of course, not going to work.
> > > 
> > > Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
> > 
> > Alexey, can you get this tested by the reporter to make sure
> > it really fixes the hang?
> 
> I'm the reporter and tester and using this patch successfully. :-)
> 
> > Once it's been tested, I'll apply it.
> 
> As for other printks, someone with 8139cp should experience the same bug.

Strangely enough, 8139cp survives with no message printed and no hang or
anything. But this is with qemu/kvm.

	static int cp_open (struct net_device *dev)
	{
	        struct cp_private *cp = netdev_priv(dev);
	        int rc;

	        if (netif_msg_ifup(cp))
	                printk(KERN_DEBUG "%s: enabling interface\n", dev->name);

	        rc = cp_alloc_rings(cp);
			...

Re: [PATCH] sky2: fix hard hang with netconsoling and iface going up

[Stephen Hemminger]

On Mon, 2 Feb 2009 01:38:53 +0300
Alexey Dobriyan <adobriyan@gmail.com> wrote:

> On Fri, Jan 30, 2009 at 02:59:07PM +0300, Alexey Dobriyan wrote:
> > On Thu, Jan 29, 2009 at 05:00:43PM -0800, David Miller wrote:
> > > From: Alexey Dobriyan <adobriyan@gmail.com>
> > > Date: Tue, 27 Jan 2009 16:27:18 +0300
> > > 
> > > > [PATCH] sky2: fix hard hang with netconsoling and iface going up
> > > > 
> > > > Printing anything over netconsole before hw is up and running is,
> > > > of course, not going to work.
> > > > 
> > > > Signed-off-by: Alexey Dobriyan <adobriyan@gmail.com>
> > > 
> > > Alexey, can you get this tested by the reporter to make sure
> > > it really fixes the hang?
> > 
> > I'm the reporter and tester and using this patch successfully. :-)
> > 
> > > Once it's been tested, I'll apply it.
> > 
> > As for other printks, someone with 8139cp should experience the same bug.
> 
> Strangely enough, 8139cp survives with no message printed and no hang or
> anything. But this is with qemu/kvm.
> 
> 	static int cp_open (struct net_device *dev)
> 	{
> 	        struct cp_private *cp = netdev_priv(dev);
> 	        int rc;
> 
> 	        if (netif_msg_ifup(cp))
> 	                printk(KERN_DEBUG "%s: enabling interface\n", dev->name);
> 
> 	        rc = cp_alloc_rings(cp);
> 			...

Unlike other drivers, sky2 calls its own up routine on resume.
In this case, the netif_running flag is already set.