Ethernet driver link state propagation to ip stack

Ethernet driver link state propagation to ip stack

[Jüri Põldre]

All,

I am experiencing issues with connecting two network adapters to the same
subnet, eg.

eth0 192.168.100.200
eth1 192.168.100.201

The task is to have redundant connections to two different hubs. In case one
link goes down the connection should go through the other. The driver
handles link events with netif_carrier_ok and netif_carrier_on from
linux/netdevice.h. These eventually send messages to networking stack with
netdev_change_state from net/core/dev.c

My question is:  Does the kernel handle  the interface state/routing tables
modifications due to link changing automatically or is there some external
daemon required to do that. Any links are greatly appreciated.


Sincerely,
Jyri Põldre.

Re: Ethernet driver link state propagation to ip stack

[Peter Buckingham]

Hi Juri,

Jüri Põldre wrote:
> I am experiencing issues with connecting two network adapters to the same
> subnet, eg.

I recently had similar problems (specifically with the e1000 driver). 
When one hang goes down a send to the other interface will hang. This 
was when we were using the same socket to send on both interfaces, what 
resulted is that the socket buffer would fill up and cause the send to 
block. If you aren't using the same socket to send out on both 
interfaces you should be fine. the rather large hack that i used to get 
this to work on my system is below. basically it will not enqueue a 
packet when the interface is down. this may well do nasty things to tcp...

peter

---
--- linux-2.6.8-24.3/net/core/dev.c
+++ linux-2.6.8-24.3/net/core/dev.c
@@ -1379,6 +1379,11 @@
  #ifdef CONFIG_NET_CLS_ACT
  	skb->tc_verd = SET_TC_AT(skb->tc_verd,AT_EGRESS);
  #endif
+	if (!netif_carrier_ok(dev)) {
+		rc = NET_XMIT_DROP;
+		goto out_kfree_skb;
+	}
+
  	if (q->enqueue) {
  		/* Grab device queue */
  		spin_lock(&dev->queue_lock);

Re: Ethernet driver link state propagation to ip stack

[Paul Jakma]

On Fri, 14 Jan 2005, Peter Buckingham wrote:

> I recently had similar problems (specifically with the e1000 
> driver). When one hang goes down a send to the other interface will 
> hang. This was when we were using the same socket to send on both 
> interfaces, what resulted is that the socket buffer would fill up 
> and cause the send to block. If you aren't using the same socket to 
> send out on both interfaces you should be fine. the rather large 
> hack that i used to get this to work on my system is below. 
> basically it will not enqueue a packet when the interface is down. 
> this may well do nasty things to tcp...

There's an old thread on netdev, occasionally resurrected, about this 
behaviour btw.

> peter

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
I hear what you're saying but I just don't care.

Re: Ethernet driver link state propagation to ip stack

[Ben Greear]

Jüri Põldre wrote:
> All,
> 
> I am experiencing issues with connecting two network adapters to the same
> subnet, eg.
> 
> eth0 192.168.100.200
> eth1 192.168.100.201
> 
> The task is to have redundant connections to two different hubs. In case one
> link goes down the connection should go through the other. The driver
> handles link events with netif_carrier_ok and netif_carrier_on from
> linux/netdevice.h. These eventually send messages to networking stack with
> netdev_change_state from net/core/dev.c
> 
> My question is:  Does the kernel handle  the interface state/routing tables
> modifications due to link changing automatically or is there some external
> daemon required to do that. Any links are greatly appreciated.

As far as I know, you have to handle this sort of thing in user-space.  You may
also have ARP issues with having two interfaces on the same subnet.  Often people
will use two private, non-related IP addresses and then migrate a virtual IP back
and forth, placing it on the preferred (ie, link-ok) interface.

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Ethernet driver link state propagation to ip stack

[Paul Jakma]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; CHARSET=utf-8; FORMAT=flowed, Size: 1728 bytes --]

On Fri, 14 Jan 2005, [iso-8859-4] Jüri Põldre wrote:

> My question is:  Does the kernel handle the interface state/routing 
> tables modifications due to link changing automatically

Not completely.

The biggest problem is that kernel does not remove its "connected" or 
"subnet" route while link is down. This means that even though kernel 
knows link is down, it will still try route packets out that 
interface.

> or is there some external daemon required to do that. Any links are 
> greatly appreciated.

There is "Netplug" - part of the net-tools package (On fedora core 3 
at least). You can use it to 'ip link set dev .... down' when carrier 
is removed. However, you wont get notified of carrier being inserted 
back - I dont know whether that holds true generally, or whether its 
an e1000 bug. So it's one-shot.

We're looking at adding support to the 'zebra' daemon in Quagga to 
remove connected routes while link is down and add them back when 
link-up again, and hence deal with this properly. Amir is very 
interested in this..

/me grumbles about why oh why the "make kernel add connected routes"
   feature was *ever* added in 2.3 in the first place (cause now
   userspace has "forgotten" how to manage them, so we cant simply undo
   this brain-damage[1] without breaking networking for everyone, sigh)

1. In retrospect at least. Brain-damage really only becomes obvious 
with carrier-sensing drivers, which were very rare back in 2.2 days, 
dont think there was a unified way to report these events to 
userspace either.

> Sincerely,
> Jyri Põldre.

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
Sailors in ships, sail on!  Even while we died, others rode out the storm.

Re: Ethernet driver link state propagation to ip stack

[Amir Guindehi]

Hi,

>> My question is:  Does the kernel handle the interface state/routing 
>> tables modifications due to link changing automatically
> 
> Not completely.
> 
> The biggest problem is that kernel does not remove its "connected" or 
> "subnet" route while link is down. This means that even though kernel 
> knows link is down, it will still try route packets out that interface.

I can confirm this.

>> or is there some external daemon required to do that. Any links are 
>> greatly appreciated.

Try the shell script below. It works for me (tm).

> There is "Netplug" - part of the net-tools package (On fedora core 3 at 
> least). You can use it to 'ip link set dev .... down' when carrier is 
> removed. However, you wont get notified of carrier being inserted back - 
> I dont know whether that holds true generally, or whether its an e1000 
> bug. So it's one-shot.
> 
> We're looking at adding support to the 'zebra' daemon in Quagga to 
> remove connected routes while link is down and add them back when 
> link-up again, and hence deal with this properly. Amir is very 
> interested in this..

Yes, I am. And I found out the following. If I add the following 
one-liner shell script daemon, I get it all working:

while /bin/true; do echo "Check of eth5 ..."; A=`ethtool eth5 | grep 
'Link detected'`; if [ "$A" != "$O" ] ; then C=`echo "$A" | grep 'yes'`; 
if [ -n "$C" ] ; then echo "Adding route";  route add -net 5.5.7.0 
netmask 255.255.255.0 dev eth5; else echo "Removing route"; route del 
-net 5.5.7.0 netmask 255.255.255.0 dev eth5; fi; O="$A"; fi; sleep 1; done

[I have to mention that eth5 is the interface where I pull and reinsert 
the cable]

As you can see I simply try to remove and re-add the connected route 
from user space. The above shell script works for me, and i can pull and 
re-insert my network cable as often as I want while still being able to 
reach the network on eth5 over different routes (received via ospf, I 
run Quagga) while the cable is pulled out and over the local interface 
while the cable is back in.

> /me grumbles about why oh why the "make kernel add connected routes"
>   feature was *ever* added in 2.3 in the first place (cause now
>   userspace has "forgotten" how to manage them, so we cant simply undo
>   this brain-damage[1] without breaking networking for everyone, sigh)

True.

I'm /not/ able to recreate the route as it was before my 'route del' the 
recreated route looks different from the route I remove at first (the 
connected one). I'm nor able to recreate the 'src xxx' flags and 
neighter is it a 'proto kernel' route ;)

So, in my eyes we need:
- A way to create connected routes from user space
or
- A kernel which removes the connected route on link loss and recreates 
it later when the link gets connected again.

Regards
- Amir
-- 
Amir Guindehi, nospam.amir@datacore.ch
DataCore GmbH, Witikonerstrasse 289, 8053 Zurich, Switzerland

Re: Ethernet driver link state propagation to ip stack

[Stefan Seyfried]

Jüri Põldre wrote:

> My question is:  Does the kernel handle  the interface state/routing tables
> modifications due to link changing automatically or is there some external
> daemon required to do that. Any links are greatly appreciated.

You might want to have a look at ifplugd (from memory:
http://0pointer.de/~lennart/). It works very well for me (i ifup/ifdown
my interfaces with it, works fine at least as long as there is only one
cable plugged at a time :-)

Stefan

Re: Ethernet driver link state propagation to ip stack

[Mark Watts]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> All,
>
> I am experiencing issues with connecting two network adapters to the same
> subnet, eg.
>
> eth0 192.168.100.200
> eth1 192.168.100.201
>
> The task is to have redundant connections to two different hubs. In case
> one link goes down the connection should go through the other.

Use the bonding.o driver. This scenario is _exactly_ what it's designed for 
(when mode=1)

Mark.

- -- 
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFB641/Bn4EFUVUIO0RAsN6AKC3SdnVTlnJ8XDB+bn6yIUf563rNwCeJaGc
qwPKo+ucugBZsXtn0Olve2A=
=UQSS
-----END PGP SIGNATURE-----