Problem with 2.4.24 e1000 and keepalived

Problem with 2.4.24 e1000 and keepalived

[Stephan von Krawczynski]

Hello all,

I am looking for confirmation for the following problem.
Setup is a simple pair of routers with 2 nics each, all e1000. If you start a
vrrp setup with keepalived and interface state is down during keepalived
startup, then the failover does not work. If the nics are UP during startup
everything works well. Now the kernel part of the story: the exact same setup
works with tulip cards.
Is there a difference regarding UP/DOWN state handling/events in e1000 and
tulip. e100 and eepro100 show the same problem btw.

Any hints are welcome

Regards,
Stephan

Re: Problem with 2.4.24 e1000 and keepalived

[Willy Tarreau]

Hi Stephan,

On Wed, Jan 07, 2004 at 08:05:56PM +0100, Stephan von Krawczynski wrote:
> Setup is a simple pair of routers with 2 nics each, all e1000. If you start a
> vrrp setup with keepalived and interface state is down during keepalived
> startup, then the failover does not work. If the nics are UP during startup
> everything works well. Now the kernel part of the story: the exact same setup
> works with tulip cards.
> Is there a difference regarding UP/DOWN state handling/events in e1000 and
> tulip. e100 and eepro100 show the same problem btw.

I noticed the exact same problem about 1 year ago with the early 2.4
bonding code and eepro100. At this time, I attributed this to a yet
undiscovered but in the bonding state machine, and could not investigate
much since it was on a remote production machine. Someone went there and
rebooted it and everything went OK. Before the reboot, the switch alredy
detected an UP link, while the bonding code saw it down (using MII at this
time, not ethtool). I recently read one report (here or on keepalived list)
about someone who got the same problem with another eepro100. I wonder
whether there would not be a bug either in the driver or in the chip itself.

What I noticed is that if you load the driver while the cable is unplugged,
and then plug it, the MII status says the link is still down. Unfortunately,
the only e100 I have access to are in prod at a customer's and I really
cannot make tests there.

Cheers,
Willy

Re: Problem with 2.4.24 e1000 and keepalived

[Ben Greear]

Willy Tarreau wrote:
> Hi Stephan,
> 
> On Wed, Jan 07, 2004 at 08:05:56PM +0100, Stephan von Krawczynski wrote:
> 
>>Setup is a simple pair of routers with 2 nics each, all e1000. If you start a
>>vrrp setup with keepalived and interface state is down during keepalived
>>startup, then the failover does not work. If the nics are UP during startup
>>everything works well. Now the kernel part of the story: the exact same setup
>>works with tulip cards.
>>Is there a difference regarding UP/DOWN state handling/events in e1000 and
>>tulip. e100 and eepro100 show the same problem btw.
> 
> 
> I noticed the exact same problem about 1 year ago with the early 2.4
> bonding code and eepro100. At this time, I attributed this to a yet
> undiscovered but in the bonding state machine, and could not investigate
> much since it was on a remote production machine. Someone went there and
> rebooted it and everything went OK. Before the reboot, the switch alredy
> detected an UP link, while the bonding code saw it down (using MII at this
> time, not ethtool). I recently read one report (here or on keepalived list)
> about someone who got the same problem with another eepro100. I wonder
> whether there would not be a bug either in the driver or in the chip itself.
> 
> What I noticed is that if you load the driver while the cable is unplugged,
> and then plug it, the MII status says the link is still down. Unfortunately,
> the only e100 I have access to are in prod at a customer's and I really
> cannot make tests there.

You have to bring the interface 'UP' before it will detect link,
with something like:  ifconfig eth2 up

Could that be the problem?

Ben

> 
> Cheers,
> Willy
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Problem with 2.4.24 e1000 and keepalived

[Willy Tarreau]

Hi Ben,

On Wed, Jan 07, 2004 at 06:45:04PM -0800, Ben Greear wrote:
 
> You have to bring the interface 'UP' before it will detect link,
> with something like:  ifconfig eth2 up

Don't you mean "after" instead of "before" here ? Because the case where
it doesn't work is when everything is set up while the cable is unplugged,
but conversely, if the system goes up with the cable plugged, setting the
interface UP detects the link as UP and works. I believe that the problem
is related to setting the interface UP with nothing plugged into it.

Cheers,
Willy

Re: Problem with 2.4.24 e1000 and keepalived

[Ben Greear]

Willy Tarreau wrote:
> Hi Ben,
> 
> On Wed, Jan 07, 2004 at 06:45:04PM -0800, Ben Greear wrote:
>  
> 
>>You have to bring the interface 'UP' before it will detect link,
>>with something like:  ifconfig eth2 up
> 
> 
> Don't you mean "after" instead of "before" here ? Because the case where
> it doesn't work is when everything is set up while the cable is unplugged,
> but conversely, if the system goes up with the cable plugged, setting the
> interface UP detects the link as UP and works. I believe that the problem
> is related to setting the interface UP with nothing plugged into it.

No, I meant what I said:  You have to tell many drivers to bring the interface
up before they will attempt (or at least report) link negotiation.
You do NOT have to give it an IP address or add any routes to it.

But, I don't know about your particular program, I just suspect it
is related to detecting link state.  I think tg3 detects link when
the interface is not UP, if you have some tg3 nics maybe you could
try with them?

Ben

> 
> Cheers,
> Willy
> 


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Problem with 2.4.24 e1000 and keepalived

[Willy Tarreau]

On Thu, Jan 08, 2004 at 12:07:10AM -0800, Ben Greear wrote:
 
> No, I meant what I said:  You have to tell many drivers to bring the 
> interface
> up before they will attempt (or at least report) link negotiation.
> You do NOT have to give it an IP address or add any routes to it.

ah, OK. No, anyway, it is just a matter of wrongly detecting link state
after the link has been plugged while the interface was already UP, no
matter if an IP was set or not.

> But, I don't know about your particular program, I just suspect it
> is related to detecting link state.  I think tg3 detects link when
> the interface is not UP, if you have some tg3 nics maybe you could
> try with them?

As far as I have tested, tg3 are fine WRT this.

Willy

Re: Problem with 2.4.24 e1000 and keepalived

[Stephan von Krawczynski]

On Wed, 07 Jan 2004 18:45:04 -0800
Ben Greear <greearb@candelatech.com> wrote:

> Willy Tarreau wrote:
> > Hi Stephan,
> > [...]
> > What I noticed is that if you load the driver while the cable is unplugged,
> > and then plug it, the MII status says the link is still down.
> > Unfortunately, the only e100 I have access to are in prod at a customer's
> > and I really cannot make tests there.
> 
> You have to bring the interface 'UP' before it will detect link,
> with something like:  ifconfig eth2 up
> 
> Could that be the problem?
> 
> Ben

Hi Ben,

the situation is like this (exactly this works flawlessly with tulip):

- unplug all interfaces from the switches
- reboot box
- plug in _one_ interface 
- log into the box (yes, network works flawlessly)
- start keepalived
- now plug in rest of the interfaces
- watch keepalived do _nothing_ (seems no UP event shows up)

in comparison to:

- let all interfaces plugged in
- reboot box
- log in
- start keepalived
- watch it work as expected

Regards,
Stephan

Re: Problem with 2.4.24 e1000 and keepalived

[Willy Tarreau]

On Thu, Jan 08, 2004 at 09:14:41AM +0100, Stephan von Krawczynski wrote:
> the situation is like this (exactly this works flawlessly with tulip):
> 
> - unplug all interfaces from the switches
> - reboot box
> - plug in _one_ interface 
> - log into the box (yes, network works flawlessly)
> - start keepalived
> - now plug in rest of the interfaces
> - watch keepalived do _nothing_ (seems no UP event shows up)

I agree with this description, and would add :
  - mii-diag ethX or ethtool ethX report link down

Willy