RE: [E1000-devel] [Bugme-new] [Bug 12570] New: Bonding does not work over e1000e.

RE: [E1000-devel] [Bugme-new] [Bug 12570] New: Bonding does not work over e1000e.

[Graham, David]

[-- Attachment #1: Type: text/plain, Size: 10254 bytes --]

Hi Konstantin. 

I have been trying but so far been failed to reproduce the reported problem. 
I have a few questions.

1) While I can't repro your problem, I can see something very similar if I don't load with module load parameter miimon=100. From your /proc/net/bonding/bond1 dumps it looks like you do the right thing, but would you please confirm for me by listing exactly which bonding module params you load with. 

2) At 11:56:01 in the report you "turn on eth2 uplink on the virtual connect bay5", and I see in /proc/net/bonding/bond1 , immediately after that, eth2 still shows MII status *down*, which would be incorrect. Can you confirm that this snippet of the file really is in the correct place in the reported sequence - that is, there is already a problem at this step, and that's where we should look for the problem. 

3) Could you send me your network scripts for the two slaves and the bonding interface itself (on RH systems, I think that's /etc/sysconfig/networking-scripts/ifcfg-*. They should be modeled on the sample info in <kernel>Documentation/networking/bonding.txt, and I'd like to check them.

4) I'm probably not controlling the slave link state in the same way that you are, because in the NEC bladeserver that I'm using, I am bringing the e1000e link-partern ports up & down by using an admins console, using SW I don't understand. I (so far) have not been able to physically disconnect one of the (serdes) links connecting the 82571 without also disconnecting the other, as I have to pull an entire switch module to make the disconnect. Can you give me mmore information on what your system is, and how you can physically disconnect  on link at a time. Then I might be able to get hold of a similar setup and see the problem.

5) I have been testing on 2.6.29-rc3 and on 2.6.28 kernels, not 2.6.29-rc1 which is what you reported the problem on. I think its unlikely that the problem is only on the 2.6.29-rc1 build, but would like to know if you've had a chance to try any other build, and what the results were. Also please let me know if you have tested with any other non-INTEL 1GB interfaces, and if you have *ever* seen bonding work properly on the system you are testing. 

6) While I can't repro your issue yet, I have made some changes very recently to the serdes link detect logic in the e1000e driver. They were written to address a separate issue, and are actually NOT in the kernel that you have been testing. I also can't see how fixing that problem might fix your problem. However, because the fixes do concern serdes link detection, and so does yours, it's probably worth a (long) shot. If you are comfortable trying them out, I have attached them to this email. They are also being queued for upstream, but only after some further local testing.
 
Thanks
Dave

>-----Original Message-----
>From: Andrew Morton [mailto:akpm@linux-foundation.org]
>Sent: Thursday, January 29, 2009 9:53 AM
>To: netdev@vger.kernel.org; e1000-devel@lists.sourceforge.net; bonding-
>devel@lists.sourceforge.net
>Cc: khorenko@parallels.com; bugme-daemon@bugzilla.kernel.org
>Subject: Re: [E1000-devel] [Bugme-new] [Bug 12570] New: Bonding does not
>work over e1000e.
>
>
>(switched to email.  Please respond via emailed reply-to-all, not via the
>bugzilla web interface).
>
>On Thu, 29 Jan 2009 03:12:01 -0800 (PST) bugme-daemon@bugzilla.kernel.org
>wrote:
>
>> http://bugzilla.kernel.org/show_bug.cgi?id=12570
>>
>>            Summary: Bonding does not work over e1000e.
>>            Product: Drivers
>>            Version: 2.5
>>      KernelVersion: 2.6.29-rc1
>>           Platform: All
>>         OS/Version: Linux
>>               Tree: Mainline
>>             Status: NEW
>>           Severity: normal
>>           Priority: P1
>>          Component: Network
>>         AssignedTo: jgarzik@pobox.com
>>         ReportedBy: khorenko@parallels.com
>>
>>
>> Checked (failing) kernel: 2.6.29-rc1
>> Latest working kernel version: unknown
>> Earliest failing kernel version: not checked but probably any. RHEL5
>kernels
>> are also affected.
>>
>> Distribution: Enterprise Linux Enterprise Linux Server release 5.1
>(Carthage)
>>
>> Hardware Environment:
>> lspci:
>> 15:00.0 Ethernet controller: Intel Corporation 82571EB Quad Port Gigabit
>> Mezzanine Adapter (rev 06)
>> 15:00.1 Ethernet controller: Intel Corporation 82571EB Quad Port Gigabit
>> Mezzanine Adapter (rev 06)
>>
>> 15:00.0 0200: 8086:10da (rev 06)
>>         Subsystem: 103c:1717
>>         Flags: bus master, fast devsel, latency 0, IRQ 154
>>         Memory at fdde0000 (32-bit, non-prefetchable) [size=128K]
>>         Memory at fdd00000 (32-bit, non-prefetchable) [size=512K]
>>         I/O ports at 6000 [size=32]
>>         [virtual] Expansion ROM at d1300000 [disabled] [size=512K]
>>         Capabilities: [c8] Power Management version 2
>>         Capabilities: [d0] Message Signalled Interrupts: 64bit+
>Queue=0/0
>> Enable+
>>         Capabilities: [e0] Express Endpoint IRQ 0
>>         Capabilities: [100] Advanced Error Reporting
>>         Capabilities: [140] Device Serial Number 24-d1-78-ff-ff-78-1b-00
>>
>> 15:00.1 0200: 8086:10da (rev 06)
>>         Subsystem: 103c:1717
>>         Flags: bus master, fast devsel, latency 0, IRQ 162
>>         Memory at fdce0000 (32-bit, non-prefetchable) [size=128K]
>>         Memory at fdc00000 (32-bit, non-prefetchable) [size=512K]
>>         I/O ports at 6020 [size=32]
>>         [virtual] Expansion ROM at d1380000 [disabled] [size=512K]
>>         Capabilities: [c8] Power Management version 2
>>         Capabilities: [d0] Message Signalled Interrupts: 64bit+
>Queue=0/0
>> Enable+
>>         Capabilities: [e0] Express Endpoint IRQ 0
>>         Capabilities: [100] Advanced Error Reporting
>>         Capabilities: [140] Device Serial Number 24-d1-78-ff-ff-78-1b-00
>>
>> Problem Description: Bonding does not work over NICs supported by
>e1000e: if
>> you brake/restore physical links of bonding slaves one by one - network
>won't
>> work anymore.
>>
>> Steps to reproduce:
>> 2 NICs supported by e1000e put into bond device (Bonding Mode: fault-
>tolerance
>> (active-backup)).
>> * ping to the outside node is ok
>> * physically brake the link of active bond slave (1)
>> * bond detects the failure, makes another slave (2) active.
>> * ping works fine
>> * restore the connection of (1)
>> * ping works fine
>> * brake the link of (2)
>> * bond detects it, reports that it makes active (1), but
>> * ping _does not_ work anymore
>>
>> Logs:
>> /var/log/messages:
>> Jan 27 11:53:29 host kernel: 0000:15:00.0: eth2: Link is Down
>> Jan 27 11:53:29 host kernel: bonding: bond1: link status definitely down
>for
>> interface eth2, disabling it
>> Jan 27 11:53:29 host kernel: bonding: bond1: making interface eth3 the
>new
>> active one.
>> Jan 27 11:56:37 host kernel: 0000:15:00.0: eth2: Link is Up 1000 Mbps
>Full
>> Duplex, Flow Control: RX/TX
>> Jan 27 11:56:37 host kernel: bonding: bond1: link status definitely up
>for
>> interface eth2.
>> Jan 27 11:57:39 host kernel: 0000:15:00.1: eth3: Link is Down
>> Jan 27 11:57:39 host kernel: bonding: bond1: link status definitely down
>for
>> interface eth3, disabling it
>> Jan 27 11:57:39 host kernel: bonding: bond1: making interface eth2 the
>new
>> active one.
>>
>> What was done + dumps of /proc/net/bonding/bond1:
>> ## 11:52:42
>> ##cat /proc/net/bonding/bond1
>> Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008)
>>
>> Bonding Mode: fault-tolerance (active-backup)
>> Primary Slave: None
>> Currently Active Slave: eth2
>> MII Status: up
>> MII Polling Interval (ms): 100
>> Up Delay (ms): 0
>> Down Delay (ms): 0
>>
>> Slave Interface: eth2
>> MII Status: up
>> Link Failure Count: 0
>> Permanent HW addr: 00:17:a4:77:00:1c
>>
>> Slave Interface: eth3
>> MII Status: up
>> Link Failure Count: 0
>> Permanent HW addr: 00:17:a4:77:00:1e
>>
>> ## 11:53:05 shutdown eth2 uplink on the virtual connect bay5
>> ##cat /proc/net/bonding/bond1
>> Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008)
>>
>> Bonding Mode: fault-tolerance (active-backup)
>> Primary Slave: None
>> Currently Active Slave: eth3
>> MII Status: up
>> MII Polling Interval (ms): 100
>> Up Delay (ms): 0
>> Down Delay (ms): 0
>>
>> Slave Interface: eth2
>> MII Status: down
>> Link Failure Count: 1
>> Permanent HW addr: 00:17:a4:77:00:1c
>>
>> Slave Interface: eth3
>> MII Status: up
>> Link Failure Count: 0
>> Permanent HW addr: 00:17:a4:77:00:1e
>>
>> ## 11:56:01 turn on eth2 uplink on the virtual connect bay5
>> ##cat /proc/net/bonding/bond1
>> Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008)
>>
>> Bonding Mode: fault-tolerance (active-backup)
>> Primary Slave: None
>> Currently Active Slave: eth3
>> MII Status: up
>> MII Polling Interval (ms): 100
>> Up Delay (ms): 0
>> Down Delay (ms): 0
>>
>> Slave Interface: eth2
>> MII Status: down
>> Link Failure Count: 1
>> Permanent HW addr: 00:17:a4:77:00:1c
>>
>> Slave Interface: eth3
>> MII Status: up
>> Link Failure Count: 0
>> Permanent HW addr: 00:17:a4:77:00:1e
>>
>> ## 11:57:22 turn off eth3 uplink on the virtual connect bay5
>> ##cat /proc/net/bonding/bond1
>> Ethernet Channel Bonding Driver: v3.3.0 (June 10, 2008)
>>
>> Bonding Mode: fault-tolerance (active-backup)
>> Primary Slave: None
>> Currently Active Slave: eth2
>> MII Status: up
>> MII Polling Interval (ms): 100
>> Up Delay (ms): 0
>> Down Delay (ms): 0
>>
>> Slave Interface: eth2
>> MII Status: up
>> Link Failure Count: 1
>> Permanent HW addr: 00:17:a4:77:00:1c
>>
>> Slave Interface: eth3
>> MII Status: down
>> Link Failure Count: 1
>> Permanent HW addr: 00:17:a4:77:00:1e
>>
>
>
>--------------------------------------------------------------------------
>----
>This SF.net email is sponsored by:
>SourcForge Community
>SourceForge wants to tell your story.
>http://p.sf.net/sfu/sf-spreadtheword
>_______________________________________________
>E1000-devel mailing list
>E1000-devel@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/e1000-devel

[-- Attachment #2: SerdesSM.patch --]
[-- Type: application/octet-stream, Size: 6332 bytes --]

commit 64621828ee2071dd70740541950c04cd488e2ff7
Author: drgraha1 <drgraha1@drgraha1-t60p.(none)>
Date:   Fri Jan 30 10:41:40 2009 -0800

    Changes the serdes link detection for 82571 & 82572 adapters to a
    state machine. Testing has shown that this both stabiitises link
    reporting) no false positives even on disconnected serdes links)
    and properly restarts autonegotiation when a link parter is returned
    to service.

diff --git a/drivers/net/e1000e/82571.c b/drivers/net/e1000e/82571.c
index 0890162..2f3cb5f 100644
--- a/drivers/net/e1000e/82571.c
+++ b/drivers/net/e1000e/82571.c
@@ -61,6 +61,7 @@
 static s32 e1000_get_phy_id_82571(struct e1000_hw *hw);
 static s32 e1000_setup_copper_link_82571(struct e1000_hw *hw);
 static s32 e1000_setup_fiber_serdes_link_82571(struct e1000_hw *hw);
+static s32 e1000_check_for_serdes_link_82571(struct e1000_hw *hw);
 static s32 e1000_write_nvm_eewr_82571(struct e1000_hw *hw, u16 offset,
 				      u16 words, u16 *data);
 static s32 e1000_fix_nvm_checksum_82571(struct e1000_hw *hw);
@@ -250,7 +251,7 @@ static s32 e1000_init_mac_params_82571(struct e1000_adapter *adapter)
 	case e1000_media_type_internal_serdes:
 		func->setup_physical_interface =
 			e1000_setup_fiber_serdes_link_82571;
-		func->check_for_link = e1000e_check_for_serdes_link;
+		func->check_for_link = e1000_check_for_serdes_link_82571;
 		func->get_link_up_info =
 			e1000e_get_speed_and_duplex_fiber_serdes;
 		break;
@@ -830,6 +831,10 @@ static s32 e1000_reset_hw_82571(struct e1000_hw *hw)
 		hw->dev_spec.e82571.alt_mac_addr_is_present)
 			e1000e_set_laa_state_82571(hw, true);
 
+	/* Reinitialize the 82571 serdes link state machine */
+	if (hw->phy.media_type == e1000_media_type_internal_serdes)
+		hw->mac.serdes_link_state = e1000_serdes_link_down;
+
 	return 0;
 }
 
@@ -1203,6 +1208,131 @@ static s32 e1000_setup_fiber_serdes_link_82571(struct e1000_hw *hw)
 }
 
 /**
+ *  e1000_check_for_serdes_link_82571 - Check for link (Serdes)
+ *  @hw: pointer to the HW structure
+ *
+ *  Checks for link up on the hardware.  If link is not up and we have
+ *  a signal, then we need to force link up.
+ **/
+s32 e1000_check_for_serdes_link_82571(struct e1000_hw *hw)
+{
+	struct e1000_mac_info *mac = &hw->mac;
+	u32 rxcw;
+	u32 ctrl;
+	u32 status;
+	s32 ret_val = 0;
+
+	ctrl = er32(CTRL);
+	status = er32(STATUS);
+	rxcw = er32(RXCW);
+
+	if ((rxcw & E1000_RXCW_SYNCH) && !(rxcw & E1000_RXCW_IV)) {
+
+		/* Receiver is synchronized with no invalid bits.  */
+		switch (mac->serdes_link_state) {
+		case e1000_serdes_link_autoneg_complete:
+			if (!(status & E1000_STATUS_LU)) {
+				/*
+				 * We have lost link, retry autoneg before
+				 * reporting link failure
+				 */
+				mac->serdes_link_state =
+				    e1000_serdes_link_autoneg_progress;
+				hw_dbg(hw, "AN_UP     -> AN_PROG\n");
+			}
+		break;
+
+		case e1000_serdes_link_forced_up:
+			/*
+			 * If we are receiving /C/ ordered sets, re-enable
+			 * auto-negotiation in the TXCW register and disable
+			 * forced link in the Device Control register in an
+			 * attempt to auto-negotiate with our link partner.
+			 */
+			if (rxcw & E1000_RXCW_C) {
+				/* Enable autoneg, and unforce link up */
+				ew32(TXCW, mac->txcw);
+				ew32(CTRL,
+				    (ctrl & ~E1000_CTRL_SLU));
+				mac->serdes_link_state =
+				    e1000_serdes_link_autoneg_progress;
+				hw_dbg(hw, "FORCED_UP -> AN_PROG\n");
+			}
+			break;
+
+		case e1000_serdes_link_autoneg_progress:
+			/*
+			 * If the LU bit is set in the STATUS register,
+			 * autoneg has completed sucessfully. If not,
+			 * try foring the link because the far end may be
+			 * available but not capable of autonegotiation.
+			 */
+			if (status & E1000_STATUS_LU)  {
+				mac->serdes_link_state =
+				    e1000_serdes_link_autoneg_complete;
+				hw_dbg(hw, "AN_PROG   -> AN_UP\n");
+			} else {
+				/*
+				 * Disable autoneg, force link up and
+				 * full duplex, and change state to forced
+				 */
+				ew32(TXCW,
+				    (mac->txcw & ~E1000_TXCW_ANE));
+				ctrl |= (E1000_CTRL_SLU | E1000_CTRL_FD);
+				ew32(CTRL, ctrl);
+
+				/* Configure Flow Control after link up. */
+				ret_val =
+				    e1000e_config_fc_after_link_up(hw);
+				if (ret_val) {
+					hw_dbg(hw, "Error config flow control\n");
+					break;
+				}
+				mac->serdes_link_state =
+				    e1000_serdes_link_forced_up;
+				hw_dbg(hw, "AN_PROG   -> FORCED_UP\n");
+			}
+			mac->serdes_has_link = true;
+			break;
+
+		case e1000_serdes_link_down:
+		default:
+			/* The link was down but the receiver has now gained
+			 * valid sync, so lets see if we can bring the link
+			 * up. */
+			ew32(TXCW, mac->txcw);
+			ew32(CTRL,
+			    (ctrl & ~E1000_CTRL_SLU));
+			mac->serdes_link_state =
+			    e1000_serdes_link_autoneg_progress;
+			hw_dbg(hw, "DOWN      -> AN_PROG\n");
+			break;
+		}
+	} else {
+		if (!(rxcw & E1000_RXCW_SYNCH)) {
+			mac->serdes_has_link = false;
+			mac->serdes_link_state = e1000_serdes_link_down;
+			hw_dbg(hw, "ANYSTATE  -> DOWN\n");
+		} else {
+			/*
+			 * We have sync, and can tolerate one
+			 * invalid (IV) codeword before declaring
+			 * link down, so reread to look again
+			 */
+			udelay(10);
+			rxcw = er32(RXCW);
+			if (rxcw & E1000_RXCW_IV) {
+				mac->serdes_link_state = e1000_serdes_link_down;
+				mac->serdes_has_link = false;
+				hw_dbg(hw, "ANYSTATE  -> DOWN\n");
+			}
+		}
+	}
+
+	return ret_val;
+}
+
+/**
  *  e1000_valid_led_default_82571 - Verify a valid default LED config
  *  @hw: pointer to the HW structure
  *  @data: pointer to the NVM (EEPROM)
diff --git a/drivers/net/e1000e/hw.h b/drivers/net/e1000e/hw.h
index 2d4ce04..de2be1f 100644
--- a/drivers/net/e1000e/hw.h
+++ b/drivers/net/e1000e/hw.h
@@ -459,6 +459,13 @@ enum e1000_smart_speed {
 	e1000_smart_speed_off
 };
 
+enum e1000_serdes_link_state {
+	e1000_serdes_link_down = 0,
+	e1000_serdes_link_autoneg_progress,
+	e1000_serdes_link_autoneg_complete,
+	e1000_serdes_link_forced_up
+};
+
 /* Receive Descriptor */
 struct e1000_rx_desc {
 	__le64 buffer_addr; /* Address of the descriptor's data buffer */
@@ -786,7 +793,8 @@ struct e1000_mac_info {
 	bool get_link_status;
 	bool in_ifs_mode;
 	bool serdes_has_link;
-	bool tx_pkt_filtering;
+	enum e1000_serdes_link_state serdes_link_state;
+       	bool tx_pkt_filtering;
 };
 
 struct e1000_phy_info {

[-- Attachment #3: disable_dmaclkgating.patch --]
[-- Type: application/octet-stream, Size: 1776 bytes --]

commit f189b02ce5bf7233e9aa217b561d0b4ec0a818b3
Author: drgraha1 <drgraha1@drgraha1-t60p.(none)>
Date:   Fri Jan 30 10:55:23 2009 -0800

    Some 82571 adapters were shipped with teh eeprom setting of
    DMA dynamic clock gating enabled. It hsould be disabled, and
    this patch force-disables the setting in the CTRL_EXT
    register (initially mapped form the EEPROM)

diff --git a/drivers/net/e1000e/82571.c b/drivers/net/e1000e/82571.c
index 2f3cb5f..565fd4e 100644
--- a/drivers/net/e1000e/82571.c
+++ b/drivers/net/e1000e/82571.c
@@ -985,6 +985,18 @@ static void e1000_initialize_hw_bits_82571(struct e1000_hw *hw)
 		reg |= E1000_PBA_ECC_CORR_EN;
 		ew32(PBA_ECC, reg);
 	}
+	/*
+	 * Workaround for hardware errata.
+	 * Ensure that DMA Dynamic Clock gating is disabled on 82571 and 82572
+	 */
+
+        if ((hw->mac.type == e1000_82571) ||
+           (hw->mac.type == e1000_82572)) {
+                reg = er32(CTRL_EXT);
+                reg &= ~E1000_CTRL_EXT_DMA_DYN_CLK_EN;
+                ew32(CTRL_EXT, reg);
+        }
+
 
 	/* PCI-Ex Control Registers */
 	if (hw->mac.type == e1000_82574) {
diff --git a/drivers/net/e1000e/defines.h b/drivers/net/e1000e/defines.h
index e6caf29..243aa49 100644
--- a/drivers/net/e1000e/defines.h
+++ b/drivers/net/e1000e/defines.h
@@ -69,6 +69,7 @@
 #define E1000_CTRL_EXT_SDP7_DATA 0x00000080 /* Value of SW Definable Pin 7 */
 #define E1000_CTRL_EXT_EE_RST    0x00002000 /* Reinitialize from EEPROM */
 #define E1000_CTRL_EXT_RO_DIS    0x00020000 /* Relaxed Ordering disable */
+#define E1000_CTRL_EXT_DMA_DYN_CLK_EN 0x00080000 /* DMA Dynamic Clock Gating */
 #define E1000_CTRL_EXT_LINK_MODE_MASK 0x00C00000
 #define E1000_CTRL_EXT_LINK_MODE_PCIE_SERDES  0x00C00000
 #define E1000_CTRL_EXT_EIAME          0x01000000

[-- Attachment #4: RemoveRXSEQ.patch --]
[-- Type: application/octet-stream, Size: 1223 bytes --]

commit f8658955ae28415f1d83ed4c72ed4632a7368e26
Author: drgraha1 <drgraha1@drgraha1-t60p.(none)>
Date:   Fri Jan 30 10:48:17 2009 -0800

    Remove the LSC-like treatment that is given to RXSEQ interrutps.
    It is thought that this was initially added to assist in serdes
    link detection which would not always indicate a link-down interrupt
    when a link partenre was removed from service. The new link detect
    mechanism for serdes no longer requires this.

diff --git a/drivers/net/e1000e/netdev.c b/drivers/net/e1000e/netdev.c
index 91817d0..55b2f6b 100644
--- a/drivers/net/e1000e/netdev.c
+++ b/drivers/net/e1000e/netdev.c
@@ -1152,7 +1152,7 @@ static irqreturn_t e1000_intr_msi(int irq, void *data)
 	 * read ICR disables interrupts using IAM
 	 */
 
-	if (icr & (E1000_ICR_RXSEQ | E1000_ICR_LSC)) {
+	if (icr & E1000_ICR_LSC) {
 		hw->mac.get_link_status = 1;
 		/*
 		 * ICH8 workaround-- Call gig speed drop workaround on cable
@@ -1218,7 +1218,7 @@ static irqreturn_t e1000_intr(int irq, void *data)
 	 * IMC write
 	 */
 
-	if (icr & (E1000_ICR_RXSEQ | E1000_ICR_LSC)) {
+	if (icr & E1000_ICR_LSC) {
 		hw->mac.get_link_status = 1;
 		/*
 		 * ICH8 workaround-- Call gig speed drop workaround on cable

Re: [Bugme-new] [Bug 12570] New: Bonding does not work over e1000e.

[Konstantin Khorenko]

Hello Dave,

thank you for investigating this, i'll try to answer your questions:

> 1) ... if I don't load with module load parameter miimon=100. ... would you please confirm for me by listing exactly which bonding module params you load with.

Yes, surely, /etc/modprobe.conf contains "miimon=100" parameter for all bonding devices on the node, and you are right, if i remove "miimon=100", /proc/net/bonding/bondX will report "MII Polling Interval (ms): 0".

# cat etc/modprobe.conf
alias eth0 bnx2
alias eth1 bnx2
alias eth2 e1000e
alias eth3 e1000e
alias scsi_hostadapter cciss
alias scsi_hostadapter1 lpfc
alias scsi_hostadapter2 usb-storage

alias bond0 bonding
options bond0 miimon=100 mode=1 max_bonds=2

alias bond1 bonding
options bond1 miimon=100 mode=1

# Disable IPV6
alias net-pf-10 off
alias ipv6 off

options ip_conntrack ip_conntrack_disable_ve0=1
######################################################################

> 2) At 11:56:01 in the report you "turn on eth2 uplink on the virtual connect bay5", and I see in /proc/net/bonding/bond1 , immediately after that, eth2 still shows MII status *down*, which would be incorrect. Can you confirm that this snippet of the file really is in the correct place in the reported sequence - that is, there is already a problem at this step, and that's where we should look for the problem.

Well, yes, the order is correct and you noticed right, but:
/var/log/messages were saved on other server so there might be a bit mistiming, but estimate it:

Here is the mix of messages from console and taken from /var/log/messages:
----------
## 11:53:05 shutdown eth2 uplink on the virtual connect bay5
Jan 27 11:53:29 sdencltst01blade12 kernel: 0000:15:00.0: eth2: Link is Down

## 11:56:01 turn on eth2 uplink on the virtual connect bay5
Jan 27 11:56:37 sdencltst01blade12 kernel: 0000:15:00.0: eth2: Link is Up 1000 Mbps Full
Duplex, Flow Control: RX/TX

## 11:57:22 turn off eth3 uplink on the virtual connect bay5
Jan 27 11:57:39 sdencltst01blade12 kernel: 0000:15:00.1: eth3: Link is Down
----------
Deltas are: 24 sec, 36 sec and 17 secs. (i mean the reaction that we can see in /var/log/messages on action from console).
The thus timing is different in no more than 17 secs, then we have: 7 secs, 19 secs and 0 secs deltas.

Well, may be 7 and 19 secs for link detection is too much, but i cannot blame for that, it works at least.
But surely if you think this is a problem, let's also focus on it. (BTW, RHEL5 kernel does not provide correct link status at all, e.g. /proc/net/bonding/bond1 still shows eth2's mii status is UP if we unplug eth2 link.)


> 3) .../etc/sysconfig/networking-scripts/ifcfg-*....
ifcfg-eth2:

DEVICE=eth2
BOOTPROTO=none
ONBOOT=yes
MASTER=bond1
SLAVE=yes
USERCTR=no
HWADDR=00:1E:0B:93:D2:E2

ifcfg-bond1:

DEVICE=bond1
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
IPADDR=xxx
NETMASK=xxx
GATEWAY=xxx

4) physical links managing

HP C7000 blade system with the Virtual Connect switch modules for the server's uplink is used.
The Onboard Administrator for the C7000 is used to power off the switch to simulate the uplink failure for this test.
Yes, the powering off switches off all links connected to the switch, but network cards are connected to different switches - thus we can disconnect links one by one.

> 5) ... if you have *ever* seen bonding work properly on the system you are testing.

Yes, we've seen. The node has 2 Broadcom network cards (eth0 and eth1) which are in bond0 the same node - bonding works fine there.
Ethernet controller: Broadcom Corporation NetXtreme II BCM5708S Gigabit Ethernet (rev 12)
0200: 14e4:16ac (rev 12)
        Subsystem: 103c:703b

6) test kernel with additional patches.
Thank you for the patches, i'll let you know as soon as the kernel with them will be tested.

Thank you!

-- 
Best regards,

Konstantin Khorenko,
Virtuozzo/OpenVZ developer,
Parallels

On 02/03/2009 04:42 AM, Graham, David wrote:
> Hi Konstantin. 
> 
> I have been trying but so far been failed to reproduce the reported problem. 
> I have a few questions.
> 
> 1) While I can't repro your problem, I can see something very similar if I don't load with module load parameter miimon=100. From your /proc/net/bonding/bond1 dumps it looks like you do the right thing, but would you please confirm for me by listing exactly which bonding module params you load with. 
> 
> 2) At 11:56:01 in the report you "turn on eth2 uplink on the virtual connect bay5", and I see in /proc/net/bonding/bond1 , immediately after that, eth2 still shows MII status *down*, which would be incorrect. Can you confirm that this snippet of the file really is in the correct place in the reported sequence - that is, there is already a problem at this step, and that's where we should look for the problem. 
> 
> 3) Could you send me your network scripts for the two slaves and the bonding interface itself (on RH systems, I think that's /etc/sysconfig/networking-scripts/ifcfg-*. They should be modeled on the sample info in <kernel>Documentation/networking/bonding.txt, and I'd like to check them.
> 
> 4) I'm probably not controlling the slave link state in the same way that you are, because in the NEC bladeserver that I'm using, I am bringing the e1000e link-partern ports up & down by using an admins console, using SW I don't understand. I (so far) have not been able to physically disconnect one of the (serdes) links connecting the 82571 without also disconnecting the other, as I have to pull an entire switch module to make the disconnect. Can you give me mmore information on what your system is, and how you can physically disconnect  on link at a time. Then I might be able to get hold of a similar setup and see the problem.
> 
> 5) I have been testing on 2.6.29-rc3 and on 2.6.28 kernels, not 2.6.29-rc1 which is what you reported the problem on. I think its unlikely that the problem is only on the 2.6.29-rc1 build, but would like to know if you've had a chance to try any other build, and what the results were. Also please let me know if you have tested with any other non-INTEL 1GB interfaces, and if you have *ever* seen bonding work properly on the system you are testing. 
> 
> 6) While I can't repro your issue yet, I have made some changes very recently to the serdes link detect logic in the e1000e driver. They were written to address a separate issue, and are actually NOT in the kernel that you have been testing. I also can't see how fixing that problem might fix your problem. However, because the fixes do concern serdes link detection, and so does yours, it's probably worth a (long) shot. If you are comfortable trying them out, I have attached them to this email. They are also being queued for upstream, but only after some further local testing.
>  
> Thanks
> Dave

------------------------------------------------------------------------------
Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM)
software. With Adobe AIR, Ajax developers can use existing skills and code to
build responsive, highly engaging applications that combine the power of local
resources and data with the reach of the web. Download the Adobe AIR SDK and
Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com

RE: [E1000-devel] [Bugme-new] [Bug 12570] New: Bonding does not work over e1000e.

[Graham, David]

Konstantin,
To get closer to your environment, I reconfigured my network, and same kernel & built-in driver that you used, but channel failover still works in my tests. Because this is without the recent serdes link patches that I referred to earlier, that means I don't expect them to be significant to the problem.

But there are some very significant differences in our setups, and I want to align my configuration closer to yours.

1) I am using a different Mezz card, with different EEPROM settings (and so features). Could you please send me "ethtool ethx" and "ethtool -e ethx" settings for the problem interfaces ? I may even spot something incorrect in the programming, but if not, I can probably use all or some of your content to make my card behave more like yours.

2) We have different link parters , and disable link in a different way.
I tried to remove the switch modules as you did, but in my bladeserver system, couldn't. There must be some administrative command to allow the latch to unlock, but I am not familiar with it. I'll keep looking. Do you have the same (failing) result if you take the link partners down administratively from the switch console ?

3) I am testing in a different chassis/backplane
Let's address the simpler differences first, but if we go another round or two without being able to figure this out, and you are prepared to send us one of the systems with the problem for definite root cause analysis, you can contact me off-line from this bz and we'll work the detail.

Dave

FYI: here's more info & log that show's how the failover works OK on my system.

      2.6.29-rc1 blade in bladeserver
      Ping from console 
         |         
      +--------+
      |  bond0 |  static address
      ++------++
       |      |
   +---+--+  ++-----+
   | eth2 |  | eth3 |
   +---+--+  ++-----+
       |      |        Serdes Backplane
       |      |
   +---+--+  ++-----+
   | 5/4  |  | 6/4  | Bladeserver wwitch module/port
   +---+--+  ++-----+
       |      |                     
    +--+------+----+
    |  1GB switch  |  External to bladeserver
    +-----+--------+
          |
    +-----+-------+   
    | ping target |
    +-------------+

Linux localhost.localdomain 2.6.29-rc1 #3 SMP Fri Feb 13 21:31:17 EST 2009
x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# ethtool -i eth2
driver: e1000e
version: 0.3.3.3-k6
firmware-version: 5.6-2
bus-info: 0000:07:00.0
[root@localhost ~]# 

07:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
        Subsystem: NEC Corporation Unknown device 834c
        Flags: bus master, fast devsel, latency 0, IRQ 59
        Memory at dee20000 (32-bit, non-prefetchable) [size=128K]
        Memory at dee00000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at 4000 [size=32]
        [virtual] Expansion ROM at 50200000 [disabled] [size=128K]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable+
        Capabilities: [e0] Express Endpoint IRQ 0
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number d8-25-62-ff-ff-97-16-00
00: 86 80 60 10 47 05 10 00 06 00 00 02 08 00 80 00
10: 00 00 e2 de 00 00 e0 de 01 40 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 33 10 4c 83
30: 00 00 00 00 c8 00 00 00 00 00 00 00 04 01 00 00

07:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
        Subsystem: NEC Corporation Unknown device 834c
        Flags: bus master, fast devsel, latency 0, IRQ 60
        Memory at dee60000 (32-bit, non-prefetchable) [size=128K]
        Memory at dee40000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at 4020 [size=32]
        [virtual] Expansion ROM at 50220000 [disabled] [size=128K]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable+
        Capabilities: [e0] Express Endpoint IRQ 0
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number d8-25-62-ff-ff-97-16-00
00: 86 80 60 10 47 05 10 00 06 00 00 02 08 00 80 00
10: 00 00 e6 de 00 00 e4 de 21 40 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 33 10 4c 83
30: 00 00 00 00 c8 00 00 00 00 00 00 00 0c 02 00 00

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth2
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:16:97:62:25:d8

Slave Interface: eth3
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:16:97:62:25:d9


// Started a ping to 10.0.0.7, all OK, & TX goes out
// physical eth2

// And then I took eth2's link partner out of service from the admin console at the connected switch port 5/4 , [command = oper/port 4/dis]

Feb 17 21:36:15 localhost kernel: e1000e: eth2 NIC Link is Down
Feb 17 21:36:15 localhost kernel: bonding: bond0: link status definitely down
for interface eth2, disabling it
Feb 17 21:36:15 localhost kernel: bonding: bond0: making interface eth3 the
new active one.
(END) 

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth3
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: down
Link Failure Count: 1
Permanent HW addr: 00:16:97:62:25:d8

Slave Interface: eth3
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:16:97:62:25:d9

// Restored the link from switch 5/4 (eth2)
//
Feb 17 21:38:29 localhost kernel: e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Feb 17 21:38:29 localhost kernel: bonding: bond0: link status definitely up
for interface eth2.

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth3
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:16:97:62:25:d8

Slave Interface: eth3
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:16:97:62:25:d9

// All still OK, with pings TX out of eth3. Now take down the link at eth3,
// switch module 6 port 4

Feb 17 21:41:41 localhost kernel: e1000e: eth3 NIC Link is Down
Feb 17 21:41:41 localhost kernel: bonding: bond0: link status definitely down
for interface eth3, disabling it
Feb 17 21:41:41 localhost kernel: bonding: bond0: making interface eth2 the
new active one.


Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth2
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:16:97:62:25:d8

Slave Interface: eth3
MII Status: down
Link Failure Count: 1
Permanent HW addr: 00:16:97:62:25:d9

// Now restore link 3
Feb 17 21:42:56 localhost kernel: e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Feb 17 21:42:56 localhost kernel: bonding: bond0: link status definitely up
for interface eth3.

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth2
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:16:97:62:25:d8

Slave Interface: eth3
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:16:97:62:25:d9

// The ping continued in the background throughout.

Re: [E1000-devel] [Bugme-new] [Bug 12570] New: Bonding does not work over e1000e.

[Konstantin Khorenko]

[-- Attachment #1: Type: text/plain, Size: 5973 bytes --]

Hello David,

sorry for the huge delay, i'll try to answer your questions below.

On 02/17/2009 10:00 PM, Graham, David wrote:
> To get closer to your environment, I reconfigured my network, and same kernel & built-in driver that you used, but channel failover still works in my tests. Because this is without the recent serdes link patches that I referred to earlier, that means I don't expect them to be significant to the problem.

Unfortunately i don't have a direct access to the problematic node thus this takes so long time.
Yesterday at last the kernel 2.6.29-rc4 + patches SerdesSM.patch, disable_dmaclkgating.patch and RemoveRXSEQ.patch was tested and it works fine for the failback!

[root@hostname ~]# uname -a
Linux hostname 2.6.29-rc4.e1000e.ver1 #1 SMP Tue Feb 10 20:47:26 MSK 2009 x86_64 x86_64 x86_64 GNU/Linux
[root@hostname ~]#

##Took down the icbay5 uplink
[root@hostname ~]# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth3
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: down
Link Failure Count: 1
Permanent HW addr: 00:17:a4:77:00:1c

Slave Interface: eth3
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:17:a4:77:00:1e

##Enable icbay5 uplink
[root@hostname ~]# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth3
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: down
Link Failure Count: 1
Permanent HW addr: 00:17:a4:77:00:1c

Slave Interface: eth3
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:17:a4:77:00:1e

## disable icbay6 (this is still working!!!  It used to die right here.
[root@hostname ~]# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth2
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:17:a4:77:00:1c

Slave Interface: eth3
MII Status: down
Link Failure Count: 1
Permanent HW addr: 00:17:a4:77:00:1e


> But there are some very significant differences in our setups, and I want to align my configuration closer to yours.
> 1) I am using a different Mezz card, with different EEPROM settings (and so features). Could you please send me "ethtool ethx" and "ethtool -e ethx" settings for the problem interfaces ? I may even spot something incorrect in the programming, but if not, I can probably use all or some of your content to make my card behave more like yours.

i've attached ethtool.info.gz file with the following commands output:
   # ethtool eth2
   # ethtool eth3
   # ethtool -e eth2
   # ethtool -e eth3
   # ethtool -i eth2
   # ethtool -i eth3

> 2) We have different link parters , and disable link in a different way.
> I tried to remove the switch modules as you did, but in my bladeserver system, couldn't. There must be some administrative command to allow the latch to unlock, but I am not familiar with it. I'll keep looking. Do you have the same (failing) result if you take the link partners down administratively from the switch console ?

Yes. The same failure occurs if we admin down the switch from the virtual connect.
But we have to do it for the whole switch.  Virtual Connect doesn't allow us to disable just one single port.

> FYI: here's more info & log that show's how the failover works OK on my system.
>
>       2.6.29-rc1 blade in bladeserver
>       Ping from console
>          |
>       +--------+
>       |  bond0 |  static address
>       ++------++
>        |      |
>    +---+--+  ++-----+
>    | eth2 |  | eth3 |
>    +---+--+  ++-----+
>        |      |        Serdes Backplane
>        |      |
>    +---+--+  ++-----+
>    | 5/4  |  | 6/4  | Bladeserver wwitch module/port
>    +---+--+  ++-----+
>        |      |
>     +--+------+----+
>     |  1GB switch  |  External to bladeserver
>     +-----+--------+
>           |
>     +-----+-------+
>     | ping target |
>     +-------------+

This is the same topology of our set up. We use the HP C7000 server chassis with the Virtual Connect enet-F module.

> 3) I am testing in a different chassis/backplane
> Let's address the simpler differences first, but if we go another round or two without being able to figure this out, and you are prepared to send us one of the systems with the problem for definite root cause analysis, you can contact me off-line from this bz and we'll work the detail.

Well, unfortunately sending the system for reproduction does not seem as an option, but if you need/want something to check, we can arrange a WebEx session. Please, let me know if this is needed.


Conclusions: well, the latest kernel with your patches does work, thank you very much, David!
Now i have to solve my original problem - to make RHEL5-based (2.6.18-x) kernel working.
At the moment RHEL5 kernel is affected by 2 issues:
1) that one which seems to be fixed by updating the testkernel from 2.6.29-r1 up to rc4 + 3 your patches.
2) when we break a link, mii status is still reported as "up" in /proc/net/bonding/bond1.
   (at the same time bonding changes the active slave to the working one correctly).

i understand there were a lot of changes since 2.6.18, but i still want to try not to replace the e1000e driver completely from the latest mainstream kernel, but to backport the set of patches to fix this exact issue.
Could you please help me pointing the patches that are essential to fix this issue (and probably issue 2)) from your point of view?

Thank you very much!

-- 
Best regards,

Konstantin Khorenko,
PVC/OpenVZ developer,
Parallels

[-- Attachment #2: ethtool.info.gz --]
[-- Type: application/x-gzip, Size: 4905 bytes --]

Re: [Bonding-devel] [Bugme-new] [Bug 12570] New: Bonding does not work over e1000e.

[Andy Gospodarek]

On Tue, Mar 17, 2009 at 10:25 AM, Konstantin Khorenko
<khorenko@parallels.com> wrote:
>
> Conclusions: well, the latest kernel with your patches does work, thank you very much, David!
> Now i have to solve my original problem - to make RHEL5-based (2.6.18-x) kernel working.
> At the moment RHEL5 kernel is affected by 2 issues:
> 1) that one which seems to be fixed by updating the testkernel from 2.6.29-r1 up to rc4 + 3 your patches.
> 2) when we break a link, mii status is still reported as "up" in /proc/net/bonding/bond1.
>   (at the same time bonding changes the active slave to the working one correctly).
>
> i understand there were a lot of changes since 2.6.18, but i still want to try not to replace the e1000e driver completely from the latest mainstream kernel, but to backport the set of patches to fix this exact issue.
> Could you please help me pointing the patches that are essential to fix this issue (and probably issue 2)) from your point of view?
>

I'm sure if you opened a new bug at bugzilla.redhat.com to address
this issue someone would be willing to make sure this was fixed.

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com

Re: [Bonding-devel] [Bugme-new] [Bug 12570] New: Bonding does not work over e1000e.

[Konstantin Khorenko]

Hello Andy,

On 03/17/2009 06:04 PM, Andy Gospodarek wrote:
> I'm sure if you opened a new bug at bugzilla.redhat.com to address
> this issue someone would be willing to make sure this was fixed.

Andy, thank you for the care, we don't forget about you. :)

In fact, i've already opened the bug, here you are:
https://bugzilla.redhat.com/show_bug.cgi?id=483034

So, you saved us one iteration - you already know about the problem, so looking forward for the fix. :)

-- 
Best regards,

Konstantin Khorenko,
PVC/OpenVZ developer,
Parallels

------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com