skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

[Chris Stromsoe]

Within 2 or 3 minutes after issuing

ip link set bond1 mtu 9000

I get one "NETDEV WATCHDOG: eth2: transmit timed out" to the console, and 
then this starts to repeat:

BUG: soft lockup detected on CPU#0!
  [<c0103667>] show_trace_log_lvl+0x19/0x2e
  [<c010368e>] show_trace+0x12/0x14
  [<c010377a>] dump_stack+0x14/0x16
  [<c0134964>] softlockup_tick+0x9f/0xae
  [<c0120f38>] run_local_timers+0x12/0x14
  [<c0120d89>] update_process_times+0x3e/0x63
  [<c010ceb0>] smp_apic_timer_interrupt+0x6f/0x7e
  [<c010337c>] apic_timer_interrupt+0x28/0x30
  [<d088d5b3>] skge_tx_clean+0x1d/0x87 [skge]
  [<d088d663>] skge_tx_timeout+0x46/0x4c [skge]
  [<c02b0854>] dev_watchdog+0x79/0xb9
  [<c0120ec9>] run_timer_softirq+0x10e/0x16b
  [<c011cebb>] __do_softirq+0x65/0xc3
  [<c0104d55>] do_softirq+0x54/0xbb
  =======================

Once the soft lockups start, the machine becomes unresponsive and has to 
be power cycled.

bond1 is a dual-port syskonnect sk-98xx with both ports bonded in 
active-backup mode.

kernel is 2.6.20.1 with the web100 kernel patches from http://web100.org/. 
I'm building a plain 2.6.20.1 without the patches right now and will test 
when it finishes compiling.

lspci for the card shows:

fresno:~ # lspci -vv -s 02:01.0
0000:02:01.0 Ethernet controller: Syskonnect (Schneider & Koch) SK-98xx Gigabit Ethernet Server Adapter (rev 11)
         Subsystem: Syskonnect (Schneider & Koch) SK-9844 Gigabit Ethernet Server Adapter (SK-NET GE-SX dual link)
         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
         Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
         Latency: 64 (5750ns min, 7750ns max), Cache Line Size: 0x08 (32 bytes)
         Interrupt: pin A routed to IRQ 22
         Region 0: Memory at febfc000 (32-bit, non-prefetchable) [size=16K]
         Region 1: I/O ports at e800 [size=256]
         Expansion ROM at febc0000 [disabled] [size=128K]
         Capabilities: [48] Power Management version 1
                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                 Status: D0 PME-Enable- DSel=0 DScale=1 PME-
         Capabilities: [50] Vital Product Data




-Chris

Re: skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

[Stephen Hemminger]

[SKGE]: Fix deadlock in skge_tx_timeout

dev_watchdog() already holds the device lock, don't take it again in
skge_tx_clean().

Signed-off-by: Patrick McHardy <kaber@trash.net>

---
commit 0b1cfafa6f6b8a168d5811d1f65cf540942c52b1
tree 4d3f252d6618adfe812e9da95cd496bb798e7c7b
parent 1ca949299260aa49eeba34ff912e2321c8b1f647
author Patrick McHardy <kaber@trash.net> Sat, 24 Feb 2007 20:05:39 +0100
committer Patrick McHardy <kaber@trash.net> Sat, 24 Feb 2007 20:05:39 +0100

 drivers/net/skge.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/net/skge.c b/drivers/net/skge.c
index e482e7f..4a948c2 100644
--- a/drivers/net/skge.c
+++ b/drivers/net/skge.c
@@ -2575,7 +2575,9 @@ static int skge_down(struct net_device *
 	skge_led(skge, LED_MODE_OFF);
 
 	netif_poll_disable(dev);
+	netif_tx_lock_bh(dev);
 	skge_tx_clean(dev);
+	netif_tx_unlock_bh(dev);
 	skge_rx_clean(skge);
 
 	kfree(skge->rx_ring.start);
@@ -2720,7 +2722,6 @@ static void skge_tx_clean(struct net_dev
 	struct skge_port *skge = netdev_priv(dev);
 	struct skge_element *e;
 
-	netif_tx_lock_bh(dev);
 	for (e = skge->tx_ring.to_clean; e != skge->tx_ring.to_use; e = e->next) {
 		struct skge_tx_desc *td = e->desc;
 		skge_tx_free(skge, e, td->control);
@@ -2729,7 +2730,6 @@ static void skge_tx_clean(struct net_dev
 
 	skge->tx_ring.to_clean = e;
 	netif_wake_queue(dev);
-	netif_tx_unlock_bh(dev);
 }
 
 static void skge_tx_timeout(struct net_device *dev)


--------------010402080104080400040104--
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

[Chris Stromsoe]

Thanks.  That fixes the soft lockup.

I've got another problem now.  The cards I'm using are dual port 
(sk-9844).  I am bonding both ports together.

The card presents as eth2 and eth3.  If I remove eth2 from the bond so 
that eth3 is the active interface, I get a hard lock (nothing prints to 
serial console, sysrq isn't responsive) and have to power cycle.

This is with plain 2.6.20.1.  I also tested using skge.[ch] from the 
current netdev git tree.


-Chris

On Thu, 8 Mar 2007, Stephen Hemminger wrote:

>
> [SKGE]: Fix deadlock in skge_tx_timeout
>
> dev_watchdog() already holds the device lock, don't take it again in
> skge_tx_clean().
>
> Signed-off-by: Patrick McHardy <kaber@trash.net>
>
> ---
> commit 0b1cfafa6f6b8a168d5811d1f65cf540942c52b1
> tree 4d3f252d6618adfe812e9da95cd496bb798e7c7b
> parent 1ca949299260aa49eeba34ff912e2321c8b1f647
> author Patrick McHardy <kaber@trash.net> Sat, 24 Feb 2007 20:05:39 +0100
> committer Patrick McHardy <kaber@trash.net> Sat, 24 Feb 2007 20:05:39 +0100
>
> drivers/net/skge.c |    4 ++--
> 1 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/net/skge.c b/drivers/net/skge.c
> index e482e7f..4a948c2 100644
> --- a/drivers/net/skge.c
> +++ b/drivers/net/skge.c
> @@ -2575,7 +2575,9 @@ static int skge_down(struct net_device *
> 	skge_led(skge, LED_MODE_OFF);
>
> 	netif_poll_disable(dev);
> +	netif_tx_lock_bh(dev);
> 	skge_tx_clean(dev);
> +	netif_tx_unlock_bh(dev);
> 	skge_rx_clean(skge);
>
> 	kfree(skge->rx_ring.start);
> @@ -2720,7 +2722,6 @@ static void skge_tx_clean(struct net_dev
> 	struct skge_port *skge = netdev_priv(dev);
> 	struct skge_element *e;
>
> -	netif_tx_lock_bh(dev);
> 	for (e = skge->tx_ring.to_clean; e != skge->tx_ring.to_use; e = e->next) {
> 		struct skge_tx_desc *td = e->desc;
> 		skge_tx_free(skge, e, td->control);
> @@ -2729,7 +2730,6 @@ static void skge_tx_clean(struct net_dev
>
> 	skge->tx_ring.to_clean = e;
> 	netif_wake_queue(dev);
> -	netif_tx_unlock_bh(dev);
> }
>
> static void skge_tx_timeout(struct net_device *dev)
>
>
> --------------010402080104080400040104--
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

[Stephen Hemminger]

On Thu, 8 Mar 2007 13:31:13 -0800 (PST)
Chris Stromsoe <cbs@cts.ucla.edu> wrote:

> Thanks.  That fixes the soft lockup.
> 
> I've got another problem now.  The cards I'm using are dual port 
> (sk-9844).  I am bonding both ports together.
> 
> The card presents as eth2 and eth3.  If I remove eth2 from the bond so 
> that eth3 is the active interface, I get a hard lock (nothing prints to 
> serial console, sysrq isn't responsive) and have to power cycle.
> 
> This is with plain 2.6.20.1.  I also tested using skge.[ch] from the 
> current netdev git tree.

Which form of bonding failover, there are locking issues with some
of the bonding modes. You should ask on the bonding mailing list.


-- 
Stephen Hemminger <shemminger@linux-foundation.org>

Re: skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

[Chris Stromsoe]

On Thu, 8 Mar 2007, Stephen Hemminger wrote:
> On Thu, 8 Mar 2007 13:31:13 -0800 (PST)
> Chris Stromsoe <cbs@cts.ucla.edu> wrote:
>
>> Thanks.  That fixes the soft lockup.
>>
>> I've got another problem now.  The cards I'm using are dual port
>> (sk-9844).  I am bonding both ports together.
>>
>> The card presents as eth2 and eth3.  If I remove eth2 from the bond so
>> that eth3 is the active interface, I get a hard lock (nothing prints to
>> serial console, sysrq isn't responsive) and have to power cycle.
>>
>> This is with plain 2.6.20.1.  I also tested using skge.[ch] from the
>> current netdev git tree.
>
> Which form of bonding failover, there are locking issues with some
> of the bonding modes. You should ask on the bonding mailing list.

It's active-backup.  Testing with the same setup and e100 works fine. 
I've done a few tests without the bonding module, using the dual-port 
separately.

Testing with bonding and skge:

1) ifenslave bond0 eth2 eth3
    ifenslave -d bond0 eth3
    ifenslave -d bond0 eth2   <-- locks up here

2) ifenslave bond0 eth2 eth3
    ifenslave -d bond0 eth2   <-- locks up here

3) ifenslave bond0 eth3 eth2
    ifenslave -d bond0 eth2
    ifenslave bond0 eth2
    ifenslave bond0 -d eth3
    ifenslave bond0 eth3
    ifenslave -d bond0 eth2   <-- locks up here


Testing without bonding:

1) ip link set mtu 9000 eth2  <-- eth2 is no longer responsive
    ip link set mtu 1500 eth2  <-- eth2 remains unresponsive

2) ifup eth2
    ifdown eth2

    perl -pi -e 's/eth2/eth3/' /etc/network/interfaces

    ifup eth3   <--  locks up here




-Chris

Re: skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

[Jay Vosburgh]

Chris Stromsoe <cbs@cts.ucla.edu> wrote:

>It's active-backup.  Testing with the same setup and e100 works fine. I've
>done a few tests without the bonding module, using the dual-port
>separately.

	Somebody else a couple of weeks ago was having similar issues
running bonding with skge (in 802.3ad mode, in his case) that also
vanished with different hardware.  I don't have any skge hardware, so I
can't test it here.  His problem was a failure in 802.3ad negotiation,
not a system lockup, though.

	If you're running active-backup and not using the ARP monitor
(arp_interval), then I'm not aware of any possible locking problems in
bonding for the kernel version you reference (2.6.20.1).

>1) ip link set mtu 9000 eth2  <-- eth2 is no longer responsive
>   ip link set mtu 1500 eth2  <-- eth2 remains unresponsive
>
>2) ifup eth2
>   ifdown eth2
>
>   perl -pi -e 's/eth2/eth3/' /etc/network/interfaces
>
>   ifup eth3   <--  locks up here

	This would seem to suggest a problem with skge itself, although
there might be some other interaction with bonding that causes the
problems for that case.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

[Chris Stromsoe]

On Thu, 8 Mar 2007, Jay Vosburgh wrote:

> 	If you're running active-backup and not using the ARP monitor 
> (arp_interval), then I'm not aware of any possible locking problems in 
> bonding for the kernel version you reference (2.6.20.1).

I'm not using arp_interval.

On Thu, 8 Mar 2007, Jay Vosburgh wrote:
> Chris Stromsoe <cbs@cts.ucla.edu> wrote:
>
>> 1) ip link set mtu 9000 eth2  <-- eth2 is no longer responsive
>>    ip link set mtu 1500 eth2  <-- eth2 remains unresponsive
>>
>> 2) ifup eth2
>>    ifdown eth2
>>
>>    perl -pi -e 's/eth2/eth3/' /etc/network/interfaces
>>
>>    ifup eth3   <--  locks up here
>
> 	This would seem to suggest a problem with skge itself, although 
> there might be some other interaction with bonding that causes the 
> problems for that case.

In both of the above mentioned cases, I was not using bonding.  That was 
with the skge driver only.


-Chris

Re: skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

[Chris Stromsoe]

On Thu, 8 Mar 2007, Chris Stromsoe wrote:
> On Thu, 8 Mar 2007, Jay Vosburgh wrote:
>> Chris Stromsoe <cbs@cts.ucla.edu> wrote:
>> 
>>> 1) ip link set mtu 9000 eth2  <-- eth2 is no longer responsive
>>>    ip link set mtu 1500 eth2  <-- eth2 remains unresponsive
>>> 
>>> 2) ifup eth2
>>>    ifdown eth2
>>>
>>>    perl -pi -e 's/eth2/eth3/' /etc/network/interfaces
>>>
>>>    ifup eth3   <--  locks up here
>>
>> 	This would seem to suggest a problem with skge itself, although 
>> there might be some other interaction with bonding that causes the 
>> problems for that case.
>
> In both of the above mentioned cases, I was not using bonding.  That was 
> with the skge driver only.

The above tests both work fine with the 2.6.20.1 sk98lin driver loaded as 
"modprobe sk98lin RlmtMode=DualNet".

I can change the MTU, add and remove eth2/eth3 from the bond, and up and 
down the interface.  It also works fine with different hardware (e100, 
e1000, tg3, bnx2).  Running both interfaces alone without the bonding 
driver also works (I can up and down the interfaces with no side-affects).




-Chris

[PATCH] skge: set mac address bonding fix

[Stephen Hemminger]

When bonding does fail over it calls set_mac_address.  When this happens
as the result of another port going down, the phy_mutex that is common to
both ports is held, so it deadlocks. Setting the address doesn't need to do
anything that needs the phy_mutex, it already has the RTNL to protect against
other admin actions.

This change just disables the receiver to avoid any hardware confusion
while address is changing.

Signed-off-by: Stephen Hemminger <shemminger@linux-foundation.org>
---
 drivers/net/skge.c |   28 +++++++++++++++++-----------
 1 files changed, 17 insertions(+), 11 deletions(-)

diff --git a/drivers/net/skge.c b/drivers/net/skge.c
index eea75a4..8fecf1b 100644
--- a/drivers/net/skge.c
+++ b/drivers/net/skge.c
@@ -3275,24 +3275,30 @@ static int skge_set_mac_address(struct net_device *dev, void *p)
 	struct skge_hw *hw = skge->hw;
 	unsigned port = skge->port;
 	const struct sockaddr *addr = p;
+	u16 ctrl;
 
 	if (!is_valid_ether_addr(addr->sa_data))
 		return -EADDRNOTAVAIL;
 
-	mutex_lock(&hw->phy_mutex);
 	memcpy(dev->dev_addr, addr->sa_data, ETH_ALEN);
-	memcpy_toio(hw->regs + B2_MAC_1 + port*8,
-		    dev->dev_addr, ETH_ALEN);
-	memcpy_toio(hw->regs + B2_MAC_2 + port*8,
-		    dev->dev_addr, ETH_ALEN);
 
-	if (hw->chip_id == CHIP_ID_GENESIS)
-		xm_outaddr(hw, port, XM_SA, dev->dev_addr);
-	else {
-		gma_set_addr(hw, port, GM_SRC_ADDR_1L, dev->dev_addr);
-		gma_set_addr(hw, port, GM_SRC_ADDR_2L, dev->dev_addr);
+	/* disable Rx */
+	ctrl = gma_read16(hw, port, GM_GP_CTRL);
+	gma_write16(hw, port, GM_GP_CTRL, ctrl & ~GM_GPCR_RX_ENA);
+
+	memcpy_toio(hw->regs + B2_MAC_1 + port*8, dev->dev_addr, ETH_ALEN);
+	memcpy_toio(hw->regs + B2_MAC_2 + port*8, dev->dev_addr, ETH_ALEN);
+
+	if (netif_running(dev)) {
+		if (hw->chip_id == CHIP_ID_GENESIS)
+			xm_outaddr(hw, port, XM_SA, dev->dev_addr);
+		else {
+			gma_set_addr(hw, port, GM_SRC_ADDR_1L, dev->dev_addr);
+			gma_set_addr(hw, port, GM_SRC_ADDR_2L, dev->dev_addr);
+		}
 	}
-	mutex_unlock(&hw->phy_mutex);
+
+	gma_write16(hw, port, GM_GP_CTRL, ctrl);
 
 	return 0;
 }
-- 
1.5.0.2

Re: [PATCH] skge: set mac address bonding fix

[Chris Stromsoe]

On Mon, 12 Mar 2007, Stephen Hemminger wrote:

> When bonding does fail over it calls set_mac_address.  When this happens 
> as the result of another port going down, the phy_mutex that is common 
> to both ports is held, so it deadlocks. Setting the address doesn't need 
> to do anything that needs the phy_mutex, it already has the RTNL to 
> protect against other admin actions.
>
> This change just disables the receiver to avoid any hardware confusion 
> while address is changing.

Applied, but no change.  For reference, all of the following test cases 
fail with skge:

* modprobe bonding miimon=100 mode=active-backup
   modprobe skge
   ip ad ad 10.0.0.2/24 brd + dev bond0
   ip ro ad 0.0.0.0/0 dev bond0 via 10.0.0.1

The link works fine at this point.

   ip link set bond0 mtu 9000

Causes the link to stop responding completely.  Setting mtu back to 1500 
does not change behavior.

* modprobe bonding miimon=100 mode=active-backup
   modprobe skge
   ifconfig bond0 up
   ifenslave bond0 eth2 eth3
   ifenslave -d bond0 eth2 eth3

That hangs the machine.  No output on serial console.  I have to power 
cycle to restore access.

* modprobe skge
   ifconfig eth2 up
   ifconfig eth3 up
   ifconfig eth3 down
   ifconfig eth2 down

As above, this hangs the machine, and is without the bonding driver 
loaded.



-Chris

Re: [PATCH] skge: set mac address bonding fix

[Stephen Hemminger]

O
> * modprobe bonding miimon=100 mode=active-backup
>    modprobe skge
>    ifconfig bond0 up
>    ifenslave bond0 eth2 eth3
>    ifenslave -d bond0 eth2 eth3
> 
> That hangs the machine.  No output on serial console.  I have to power 
> cycle to restore access.
> 
> * modprobe skge
>    ifconfig eth2 up
>    ifconfig eth3 up
>    ifconfig eth3 down
>    ifconfig eth2 down

Sounds like a dual-port only problem.. Unfortunately, I don't have actual
dual port skge hardware to test.  Try this:



--- test.orig/drivers/net/skge.c
+++ test/drivers/net/skge.c
@@ -671,7 +671,7 @@ static void skge_led(struct skge_port *s
 	struct skge_hw *hw = skge->hw;
 	int port = skge->port;
 
-	mutex_lock(&hw->phy_mutex);
+	spin_lock_bh(&skge->phy_lock);
 	if (hw->chip_id == CHIP_ID_GENESIS) {
 		switch (mode) {
 		case LED_MODE_OFF:
@@ -742,7 +742,7 @@ static void skge_led(struct skge_port *s
 				     PHY_M_LED_MO_RX(MO_LED_ON));
 		}
 	}
-	mutex_unlock(&hw->phy_mutex);
+	spin_unlock_bh(&skge->phy_lock);
 }
 
 /* blink LED's for finding board */
@@ -1316,7 +1316,7 @@ static void xm_phy_init(struct skge_port
 	xm_phy_write(hw, port, PHY_XMAC_CTRL, ctrl);
 
 	/* Poll PHY for status changes */
-	schedule_delayed_work(&skge->link_thread, LINK_HZ);
+	mod_timer(&skge->link_timer, jiffies + LINK_HZ);
 }
 
 static void xm_check_link(struct net_device *dev)
@@ -1391,10 +1391,9 @@ static void xm_check_link(struct net_dev
  * Since internal PHY is wired to a level triggered pin, can't
  * get an interrupt when carrier is detected.
  */
-static void xm_link_timer(struct work_struct *work)
+static void xm_link_timer(unsigned long arg)
 {
-	struct skge_port *skge =
-		container_of(work, struct skge_port, link_thread.work);
+	struct skge_port *skge = (struct skge_port *) arg;
 	struct net_device *dev = skge->netdev;
  	struct skge_hw *hw = skge->hw;
 	int port = skge->port;
@@ -1414,13 +1413,13 @@ static void xm_link_timer(struct work_st
 			goto nochange;
 	}
 
-	mutex_lock(&hw->phy_mutex);
+	spin_lock(&skge->phy_lock);
 	xm_check_link(dev);
-	mutex_unlock(&hw->phy_mutex);
+	spin_unlock(&skge->phy_lock);
 
 nochange:
 	if (netif_running(dev))
-		schedule_delayed_work(&skge->link_thread, LINK_HZ);
+		mod_timer(&skge->link_timer, jiffies + LINK_HZ);
 }
 
 static void genesis_mac_init(struct skge_hw *hw, int port)
@@ -2323,7 +2322,7 @@ static void skge_phy_reset(struct skge_p
 	netif_stop_queue(skge->netdev);
 	netif_carrier_off(skge->netdev);
 
-	mutex_lock(&hw->phy_mutex);
+	spin_lock_bh(&skge->phy_lock);
 	if (hw->chip_id == CHIP_ID_GENESIS) {
 		genesis_reset(hw, port);
 		genesis_mac_init(hw, port);
@@ -2331,7 +2330,7 @@ static void skge_phy_reset(struct skge_p
 		yukon_reset(hw, port);
 		yukon_init(hw, port);
 	}
-	mutex_unlock(&hw->phy_mutex);
+	spin_unlock_bh(&skge->phy_lock);
 
 	dev->set_multicast_list(dev);
 }
@@ -2354,12 +2353,12 @@ static int skge_ioctl(struct net_device 
 		/* fallthru */
 	case SIOCGMIIREG: {
 		u16 val = 0;
-		mutex_lock(&hw->phy_mutex);
+		spin_lock_bh(&skge->phy_lock);
 		if (hw->chip_id == CHIP_ID_GENESIS)
 			err = __xm_phy_read(hw, skge->port, data->reg_num & 0x1f, &val);
 		else
 			err = __gm_phy_read(hw, skge->port, data->reg_num & 0x1f, &val);
-		mutex_unlock(&hw->phy_mutex);
+		spin_unlock_bh(&skge->phy_lock);
 		data->val_out = val;
 		break;
 	}
@@ -2368,14 +2367,14 @@ static int skge_ioctl(struct net_device 
 		if (!capable(CAP_NET_ADMIN))
 			return -EPERM;
 
-		mutex_lock(&hw->phy_mutex);
+		spin_lock_bh(&skge->phy_lock);
 		if (hw->chip_id == CHIP_ID_GENESIS)
 			err = xm_phy_write(hw, skge->port, data->reg_num & 0x1f,
 				   data->val_in);
 		else
 			err = gm_phy_write(hw, skge->port, data->reg_num & 0x1f,
 				   data->val_in);
-		mutex_unlock(&hw->phy_mutex);
+		spin_unlock_bh(&skge->phy_lock);
 		break;
 	}
 	return err;
@@ -2481,12 +2480,12 @@ static int skge_up(struct net_device *de
 		goto free_rx_ring;
 
 	/* Initialize MAC */
-	mutex_lock(&hw->phy_mutex);
+	spin_lock_bh(&skge->phy_lock);
 	if (hw->chip_id == CHIP_ID_GENESIS)
 		genesis_mac_init(hw, port);
 	else
 		yukon_mac_init(hw, port);
-	mutex_unlock(&hw->phy_mutex);
+	spin_unlock_bh(&skge->phy_lock);
 
 	/* Configure RAMbuffers */
 	chunk = hw->ram_size / ((hw->ports + 1)*2);
@@ -2531,7 +2530,7 @@ static int skge_down(struct net_device *
 
 	netif_stop_queue(dev);
 	if (hw->chip_id == CHIP_ID_GENESIS && hw->phy_type == SK_PHY_XMAC)
-		cancel_delayed_work(&skge->link_thread);
+		del_timer_sync(&skge->link_timer);
 
 	skge_write8(skge->hw, SK_REG(skge->port, LNK_LED_REG), LED_OFF);
 	if (hw->chip_id == CHIP_ID_GENESIS)
@@ -3160,28 +3159,29 @@ static void skge_error_irq(struct skge_h
 }
 
 /*
- * Interrupt from PHY are handled in work queue
+ * Interrupt from PHY are handled in tasklet (softirq)
  * because accessing phy registers requires spin wait which might
  * cause excess interrupt latency.
  */
-static void skge_extirq(struct work_struct *work)
+static void skge_extirq(unsigned long arg)
 {
-	struct skge_hw *hw = container_of(work, struct skge_hw, phy_work);
+	struct skge_hw *hw = (struct skge_hw *) arg;
 	int port;
 
-	mutex_lock(&hw->phy_mutex);
 	for (port = 0; port < hw->ports; port++) {
 		struct net_device *dev = hw->dev[port];
-		struct skge_port *skge = netdev_priv(dev);
 
 		if (netif_running(dev)) {
+			struct skge_port *skge = netdev_priv(dev);
+
+			spin_lock(&skge->phy_lock);
 			if (hw->chip_id != CHIP_ID_GENESIS)
 				yukon_phy_intr(skge);
 			else if (hw->phy_type == SK_PHY_BCOM)
 				bcom_phy_intr(skge);
+			spin_unlock(&skge->phy_lock);
 		}
 	}
-	mutex_unlock(&hw->phy_mutex);
 
 	spin_lock_irq(&hw->hw_lock);
 	hw->intr_mask |= IS_EXT_REG;
@@ -3206,7 +3206,7 @@ static irqreturn_t skge_intr(int irq, vo
 	status &= hw->intr_mask;
 	if (status & IS_EXT_REG) {
 		hw->intr_mask &= ~IS_EXT_REG;
-		schedule_work(&hw->phy_work);
+		tasklet_schedule(&hw->ext_task);
 	}
 
 	if (status & (IS_XA1_F|IS_R1_F)) {
@@ -3484,14 +3484,12 @@ static int skge_reset(struct skge_hw *hw
 
 	skge_write32(hw, B0_IMSK, hw->intr_mask);
 
-	mutex_lock(&hw->phy_mutex);
 	for (i = 0; i < hw->ports; i++) {
 		if (hw->chip_id == CHIP_ID_GENESIS)
 			genesis_reset(hw, i);
 		else
 			yukon_reset(hw, i);
 	}
-	mutex_unlock(&hw->phy_mutex);
 
 	return 0;
 }
@@ -3539,6 +3537,8 @@ static struct net_device *skge_devinit(s
 	skge->netdev = dev;
 	skge->hw = hw;
 	skge->msg_enable = netif_msg_init(debug, default_msg);
+	spin_lock_init(&skge->phy_lock);
+
 	skge->tx_ring.count = DEFAULT_TX_RING_SIZE;
 	skge->rx_ring.count = DEFAULT_RX_RING_SIZE;
 
@@ -3555,7 +3555,7 @@ static struct net_device *skge_devinit(s
 	skge->port = port;
 
 	/* Only used for Genesis XMAC */
-	INIT_DELAYED_WORK(&skge->link_thread, xm_link_timer);
+	setup_timer(&skge->link_timer, xm_link_timer, (unsigned long) skge);
 
 	if (hw->chip_id != CHIP_ID_GENESIS) {
 		dev->features |= NETIF_F_IP_CSUM | NETIF_F_SG;
@@ -3637,9 +3637,8 @@ static int __devinit skge_probe(struct p
 	}
 
 	hw->pdev = pdev;
-	mutex_init(&hw->phy_mutex);
-	INIT_WORK(&hw->phy_work, skge_extirq);
 	spin_lock_init(&hw->hw_lock);
+	tasklet_init(&hw->ext_task, &skge_extirq, (unsigned long) hw);
 
 	hw->regs = ioremap_nocache(pci_resource_start(pdev, 0), 0x4000);
 	if (!hw->regs) {
@@ -3725,6 +3724,8 @@ static void __devexit skge_remove(struct
 	dev0 = hw->dev[0];
 	unregister_netdev(dev0);
 
+	tasklet_disable(&hw->ext_task);
+
 	spin_lock_irq(&hw->hw_lock);
 	hw->intr_mask = 0;
 	skge_write32(hw, B0_IMSK, 0);
--- test.orig/drivers/net/skge.h
+++ test/drivers/net/skge.h
@@ -2412,6 +2412,7 @@ struct skge_hw {
 	void __iomem  	     *regs;
 	struct pci_dev	     *pdev;
 	spinlock_t	     hw_lock;
+	struct tasklet_struct ext_task;
 	u32		     intr_mask;
 	struct net_device    *dev[2];
 
@@ -2424,8 +2425,6 @@ struct skge_hw {
 	u32	     	     ram_size;
 	u32	     	     ram_offset;
 	u16		     phy_addr;
-	struct work_struct   phy_work;
-	struct mutex	     phy_mutex;
 };
 
 enum pause_control {
@@ -2447,17 +2446,18 @@ enum pause_status {
 
 
 struct skge_port {
-	u32		     msg_enable;
 	struct skge_hw	     *hw;
 	struct net_device    *netdev;
 	int		     port;
+	u32		     msg_enable;
+	spinlock_t	     phy_lock;
 
 	struct skge_ring     tx_ring;
 	struct skge_ring     rx_ring;
 
 	struct net_device_stats net_stats;
 
-	struct delayed_work  link_thread;
+	struct timer_list    link_timer;
 	enum pause_control   flow_control;
 	enum pause_status    flow_status;
 	u8		     rx_csum;

Re: [PATCH] skge: set mac address bonding fix

[Chris Stromsoe]

On Mon, 12 Mar 2007, Stephen Hemminger wrote:

> Sounds like a dual-port only problem.. Unfortunately, I don't have 
> actual dual port skge hardware to test.  Try this:

That doesn't change the behavior at all.  I tested against 2.6.20.2 with 
skge from git and against 2.6.21-rc3-git7.

-Chris

Re: [PATCH] skge: set mac address bonding fix

[Jeff Garzik]

Stephen Hemminger wrote:
> When bonding does fail over it calls set_mac_address.  When this happens
> as the result of another port going down, the phy_mutex that is common to
> both ports is held, so it deadlocks. Setting the address doesn't need to do
> anything that needs the phy_mutex, it already has the RTNL to protect against
> other admin actions.
> 
> This change just disables the receiver to avoid any hardware confusion
> while address is changing.
> 
> Signed-off-by: Stephen Hemminger <shemminger@linux-foundation.org>

applied, because:

Rather than guessing about whether hardware confusion would occur, it is 
best to disable the receiver (for any hardware) unless we are CERTAIN 
that the hardware RX filter can be updated without stopping RX.

Motivated people should check this on other hardware too.

	Jeff

Re: skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

[Chris Stromsoe]

On Mon, 12 Mar 2007, Chris Stromsoe wrote:
> On Thu, 8 Mar 2007, Chris Stromsoe wrote:
>> On Thu, 8 Mar 2007, Jay Vosburgh wrote:
>>> Chris Stromsoe <cbs@cts.ucla.edu> wrote:
>>> 
>>>> 1) ip link set mtu 9000 eth2  <-- eth2 is no longer responsive
>>>>    ip link set mtu 1500 eth2  <-- eth2 remains unresponsive
>>>> 
>>>> 2) ifup eth2
>>>>    ifdown eth2
>>>>
>>>>    perl -pi -e 's/eth2/eth3/' /etc/network/interfaces
>>>>
>>>>    ifup eth3   <--  locks up here
>>>
>>> 	This would seem to suggest a problem with skge itself, although 
>>> there might be some other interaction with bonding that causes the 
>>> problems for that case.
>> 
>> In both of the above mentioned cases, I was not using bonding.  That 
>> was with the skge driver only.
>
> The above tests both work fine with the 2.6.20.1 sk98lin driver loaded 
> as "modprobe sk98lin RlmtMode=DualNet".
>
> I can change the MTU, add and remove eth2/eth3 from the bond, and up and 
> down the interface.  It also works fine with different hardware (e100, 
> e1000, tg3, bnx2).  Running both interfaces alone without the bonding 
> driver also works (I can up and down the interfaces with no 
> side-affects).

Just an update - it looks like 2.6.20.1 fixed the MTU problem ("1" 
above), but not the other problem (where the machine locks up if the 
second port on the dual-port card is downed).

To recap:

I am use SysKonnect SK-9843 cards.  The sk98lin driver works fine; the 
skge driver does not.  The following sequence of commands locks up the 
server.  The lock is a hard lock; console is not responsive to keyboard 
input or to sysrq.  Nothing is printed on the serial console.


   ip li set eth2 up
   ip li set eth2 down
   ip li set eth3 up


There are no addresses assigned to either interface.  This was done after 
a fresh boot.  It is repeatable.  If I do not down eth2, I can up eth3 
assign addresses, and use both interfaces.

The kernel is fresh from kernel.org and does not have any third party 
patches.

lspci -vv output:

0000:01:0a.0 Ethernet controller: Syskonnect (Schneider & Koch) SK-98xx Gigabit Ethernet Server Adapter (rev 12)
         Subsystem: Syskonnect (Schneider & Koch) SK-9844 Gigabit Ethernet Server Adapter (SK-NET GE-SX dual link)
         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
         Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
         Latency: 64 (5750ns min, 7750ns max), Cache Line Size: 0x08 (32 bytes)
         Interrupt: pin A routed to IRQ 10
         Region 0: Memory at ff8fc000 (32-bit, non-prefetchable) [size=16K]
         Region 1: I/O ports at d800 [size=256]
         Expansion ROM at ff400000 [disabled] [size=128K]
         Capabilities: <available only to root>




-Chris

Re: skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

[Chris Stromsoe]

On Tue, 8 May 2007, Chris Stromsoe wrote:
> On Mon, 12 Mar 2007, Chris Stromsoe wrote:
>> On Thu, 8 Mar 2007, Chris Stromsoe wrote:
>>> On Thu, 8 Mar 2007, Jay Vosburgh wrote:
>>>> Chris Stromsoe <cbs@cts.ucla.edu> wrote:
>>>> 
>>>>> 1) ip link set mtu 9000 eth2  <-- eth2 is no longer responsive
>>>>>    ip link set mtu 1500 eth2  <-- eth2 remains unresponsive
>>>>> 
>>>>> 2) ifup eth2
>>>>>    ifdown eth2
>>>>>
>>>>>    perl -pi -e 's/eth2/eth3/' /etc/network/interfaces
>>>>>
>>>>>    ifup eth3   <--  locks up here
>>>>
>>>> 	This would seem to suggest a problem with skge itself, although there 
>>>> might be some other interaction with bonding that causes the problems for 
>>>> that case.
>>> 
>>> In both of the above mentioned cases, I was not using bonding.  That was 
>>> with the skge driver only.
>> 
>> The above tests both work fine with the 2.6.20.1 sk98lin driver loaded as 
>> "modprobe sk98lin RlmtMode=DualNet".
>> 
>> I can change the MTU, add and remove eth2/eth3 from the bond, and up and 
>> down the interface.  It also works fine with different hardware (e100, 
>> e1000, tg3, bnx2).  Running both interfaces alone without the bonding 
>> driver also works (I can up and down the interfaces with no side-affects).
>
> Just an update - it looks like 2.6.20.1 fixed the MTU problem ("1" above), 
> but not the other problem (where the machine locks up if the second port on 
> the dual-port card is downed).
>
> To recap:
>
> I am use SysKonnect SK-9843 cards.  The sk98lin driver works fine; the skge

I should proof-read first.  The cards are SK-9844s, not SK-9843s.  The 
rest of the prior message is still correct.



-Chris

> driver does not.  The following sequence of commands locks up the server. 
> The lock is a hard lock; console is not responsive to keyboard input or to 
> sysrq.  Nothing is printed on the serial console.
>
>
>  ip li set eth2 up
>  ip li set eth2 down
>  ip li set eth3 up
>
>
> There are no addresses assigned to either interface.  This was done after a 
> fresh boot.  It is repeatable.  If I do not down eth2, I can up eth3 assign 
> addresses, and use both interfaces.
>
> The kernel is fresh from kernel.org and does not have any third party 
> patches.
>
> lspci -vv output:
>
> 0000:01:0a.0 Ethernet controller: Syskonnect (Schneider & Koch) SK-98xx 
> Gigabit Ethernet Server Adapter (rev 12)
>        Subsystem: Syskonnect (Schneider & Koch) SK-9844 Gigabit Ethernet  Server Adapter (SK-NET GE-SX dual link)
>        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
>        Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
>        Latency: 64 (5750ns min, 7750ns max), Cache Line Size: 0x08 (32 bytes)
>        Interrupt: pin A routed to IRQ 10
>        Region 0: Memory at ff8fc000 (32-bit, non-prefetchable) [size=16K]
>        Region 1: I/O ports at d800 [size=256]
>        Expansion ROM at ff400000 [disabled] [size=128K]
>        Capabilities: <available only to root>
>
>
>
>
> -Chris
>

Re: skge- "soft lockup on CPU#0" with mtu=9000 (2.6.20.1 + web100 patch)

[Jay Vosburgh]

Chris Stromsoe <cbs@cts.ucla.edu> wrote:

>Within 2 or 3 minutes after issuing
>
>ip link set bond1 mtu 9000
>
>I get one "NETDEV WATCHDOG: eth2: transmit timed out" to the console, and
>then this starts to repeat:
>
>BUG: soft lockup detected on CPU#0!

	I believe this is the same bug that is fixed by this change:

commit c4f283b1f275e5528c13c119e5cfc80cdba55d00
Author: Jay Vosburgh <fubar@us.ibm.com>
Date:   Wed Feb 28 17:03:20 2007 -0800

    bonding: fix double dev_add_pack
    
        Bonding can erroneously register the same packet_type to receive
    ARPs (for use by ARP validation): once at device open time, and once via
    sysfs.  Since sysfs can change the validate setting (and thus register
    or unregister) at any time, a flag is needed to synchronize with device
    open in order to avoid double registrations, and the simplest place is
    within the packet_type structure itself.  Double unregister is not an
    issue.
    
        Bug reported by Ulrich Oelmann <ulrich.oelmann@web.de>.
    
    Signed-off-by: Jay Vosburgh <fubar@us.ibm.com>
    Signed-off-by: Jeff Garzik <jeff@garzik.org>

diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index ea73ebf..68afcb5 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -3423,6 +3423,9 @@ void bond_register_arp(struct bonding *b
 {
 	struct packet_type *pt = &bond->arp_mon_pt;
 
+	if (pt->type)
+		return;
+
 	pt->type = htons(ETH_P_ARP);
 	pt->dev = NULL; /*bond->dev;XXX*/
 	pt->func = bond_arp_rcv;
@@ -3431,7 +3434,10 @@ void bond_register_arp(struct bonding *b
 
 void bond_unregister_arp(struct bonding *bond)
 {
-	dev_remove_pack(&bond->arp_mon_pt);
+	struct packet_type *pt = &bond->arp_mon_pt;
+
+	dev_remove_pack(pt);
+	pt->type = 0;
 }
 
 /*---------------------------- Hashing Policies -----------------------------*/


	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com