bonding driver issues: slave interface not coming up

bonding driver issues: slave interface not coming up

[John Jasen]

Hello;

In short, using redhat enterprise 4, we're attempting to bond two tg3
gigabit cards in XOR mode.

These cards are plugged into a cisco 2970 switch, configured with
etherchannel bonding in src-dst-mac, which looks to be mode 2.

The problem area is, upon boot, the first slave NIC does not come up.
Furthermore, running ethtool against the slave yields the following message:

[root@XXX ~]# ethtool eth0
Settings for eth0:
Cannot get device settings: Resource temporarily unavailable
         Supports Wake-on: g
         Wake-on: d
         Current message level: 0x000000ff (255)
         Link detected: no

Now, curiously enough, after boot, logging in and bringing down the
bonding interface, then bringing it back up will activate both
interfaces. ie:

ifdown bond0
ifup bond0

ethtool eth0
[root@XXX ~]# ethtool eth0
Settings for eth0:
         Supported ports: [ MII ]
         Supported link modes:   10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Supports auto-negotiation: Yes
         Advertised link modes:  Not reported
         Advertised auto-negotiation: No
         Speed: 1000Mb/s
         Duplex: Full
         Port: Twisted Pair
         PHYAD: 1
         Transceiver: internal
         Auto-negotiation: off
         Supports Wake-on: g
         Wake-on: d
         Current message level: 0x000000ff (255)
         Link detected: yes

Attempts at debugging that we've taken and their results:

duplex/speed forced on both switch and card: no effect
turning off spanning tree protocol: no effect
enabling fastport: no effect
switching from tg3 to bcm5700 drivers: no effect
attempting to down and up the interface in /etc/rc.d/init.d/network: no
effect
attempting a script that runs after network and irqbalance to do the
same thing: no effect
putting something in rc.local to do the same: works, our current ugly
workaround
breaking the bond, using the interfaces as two independent NICS: works
restarting the bond0 interface manually: works

This problem also manifests on more than one system, which would seem to
preclude possible cable, host, or port problems.

Any ideas, or any further information I can send you to help diagnose?
Any help would be much appreciated.

-- 
-- John E. Jasen (jjasen@realityfailure.org)
-- No one will sorrow for me when I die, because those who would
-- are dead already. -- Lan Mandragoran, The Wheel of Time, New Spring

Re: bonding driver issues: slave interface not coming up

[Jason R. Martin]

On 6/27/05, John Jasen <jjasen@realityfailure.org> wrote:
> 
> Hello;
> 
> In short, using redhat enterprise 4, we're attempting to bond two tg3
> gigabit cards in XOR mode.
> 
> These cards are plugged into a cisco 2970 switch, configured with
> etherchannel bonding in src-dst-mac, which looks to be mode 2.
> 
> The problem area is, upon boot, the first slave NIC does not come up.
> Furthermore, running ethtool against the slave yields the following message:
> 
> [root@XXX ~]# ethtool eth0
> Settings for eth0:
> Cannot get device settings: Resource temporarily unavailable
>          Supports Wake-on: g
>          Wake-on: d
>          Current message level: 0x000000ff (255)
>          Link detected: no
> 
> Now, curiously enough, after boot, logging in and bringing down the
> bonding interface, then bringing it back up will activate both
> interfaces. ie:
> 
> ifdown bond0
> ifup bond0
> 
> ethtool eth0
> [root@XXX ~]# ethtool eth0
> Settings for eth0:
>          Supported ports: [ MII ]
>          Supported link modes:   10baseT/Half 10baseT/Full
>                                  100baseT/Half 100baseT/Full
>                                  1000baseT/Half 1000baseT/Full
>          Supports auto-negotiation: Yes
>          Advertised link modes:  Not reported
>          Advertised auto-negotiation: No
>          Speed: 1000Mb/s
>          Duplex: Full
>          Port: Twisted Pair
>          PHYAD: 1
>          Transceiver: internal
>          Auto-negotiation: off
>          Supports Wake-on: g
>          Wake-on: d
>          Current message level: 0x000000ff (255)
>          Link detected: yes
> 
> Attempts at debugging that we've taken and their results:
> 
> duplex/speed forced on both switch and card: no effect
> turning off spanning tree protocol: no effect
> enabling fastport: no effect
> switching from tg3 to bcm5700 drivers: no effect
> attempting to down and up the interface in /etc/rc.d/init.d/network: no
> effect
> attempting a script that runs after network and irqbalance to do the
> same thing: no effect
> putting something in rc.local to do the same: works, our current ugly
> workaround
> breaking the bond, using the interfaces as two independent NICS: works
> restarting the bond0 interface manually: works
> 
> This problem also manifests on more than one system, which would seem to
> preclude possible cable, host, or port problems.
> 
> Any ideas, or any further information I can send you to help diagnose?
> Any help would be much appreciated.

What's in /proc/net/bonding/bond0?

You'll probably have better luck getting help with this issue on
netdev@vger.kernel.org and bonding-devel@lists.sourceforge.net.

Jason

Re: bonding driver issues: slave interface not coming up

[John Jasen]

On Mon, 27 Jun 2005, Jason R. Martin wrote:

> What's in /proc/net/bonding/bond0?

before a successful ifdown/ifup bond0:

Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)

Bonding Mode: load balancing (xor)
MII Status: up
MII Polling Interval (ms): 1000
Up Delay (ms): 0
Down Delay (ms): 3000

Slave Interface: eth0
MII Status: down
Link Failure Count: 1
Permanent HW addr: 00:09:3d:11:8d:93

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:09:3d:11:8d:94

after a successful ifdown/ifup bond0:
Ethernet Channel Bonding Driver: v2.6.0 (January 14, 2004)

Bonding Mode: load balancing (xor)
MII Status: up
MII Polling Interval (ms): 1000
Up Delay (ms): 0
Down Delay (ms): 3000

Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:09:3d:11:8d:93

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:09:3d:11:8d:94

> You'll probably have better luck getting help with this issue on
> netdev@vger.kernel.org and bonding-devel@lists.sourceforge.net.

Thanks. I will email them shortly.

-- 
-- John E. Jasen (jjasen@realityfailure.org)
-- No one will sorrow for me when I die, because those who would
-- are dead already. -- Lan Mandragoran, The Wheel of Time, New Spring