Link detection

Link detection

[Hasso Tepper]

We (the Quagga project) are planning to rework link detection in Quagga to 
support more platforms (only Linux and Solaris are supported at the moment) 
and to have proper abstraction of link detection in zebra daemon. During 
discussions several questions were raised.

1) What's the proper way to detect carrier in Linux for user space 
application like Quagga?

Till now I thought that IFF_RUNNING flag should be used for that. But some 
time ago I tried to bug Stephen Hemminger about missing IFF_RUNNING flag in 
ip output and got answer that IFF_RUNNING is BSD'ism and shouldn't be used 
any more and /sys/class/net/eth0/carrier should be looked at.

True, IFF_RUNNING is from BSD, but as long as the code in 
net/core/dev.c:dev_get_flags() is there, it gives nice clean way to get 
carrier status from user space (AFAIK it's used in same way in Solaris). 
Especially for applications using rtnetlink - ie. without it there is no 
all info in the netlink message any more. If these applications will 
receive RTM_NEWLINK, they have to poll kernel "look, maybe carrier status 
is changed" every time?

2) There is no difference between "we know that carrier is really on" and 
"interface doesn't support carrier detection" in Linux.

>From users' point of view it might make huge difference - can I rely on info 
I get from OS, or have to run to the equipment to see whether links are 
really up or not? It's not critical, but would be still nice to have.

3) Connected routes in carrier down situations. If address are added to the 
interface, "connected" route is added to the interface as well. Problem is 
that this route is still there if carrier is down, but there is no 
difference for user if just carrier is down or interface is 
administratively down - network behind this interface is still actually 
unreachable.  

It's not only the issue related with routing protocols, but the general one. 
I'm using laptop with two interfaces - ethernet and wireless. If I plug off 
ethernet cable and walk away from my desk, the network on this ethernet 
segment should be still reachable for me via wireless (default route), but 
no ... I have to shut down ethernet interface manually.

There are several scenarios which are affected - building redundant 
firewalls/gateways, computers/equipment with backup links etc. And of 
course routers. We take care of this in Quagga not to take nexthops in down 
network into account, but it doesn't matter if destination of actual 
traffic is this particular network on interface which is actually down.

Before someone proposes that - yes, managing these routes can be done from 
user space in theory (like zebra daemon), but there are several problems 
regarding this.

* Managing connected routes from user space might leave network in 
inconsistent state. You can't create routes with protocol kernel with ip 
utility, so kernel will not remove these routes if address is really 
deleted from interface.
* There are several applications which would be interested in this, but it 
would be dangerous imho if several applications at once will manage 
connected routes in kernel.

And I don't see any actual reason not to manage removal/adding connected 
routes in kernel level if link goes down/up. Or is there any? Why would 
anyone need route to the interface if it's actually known to be unusable?


with my best wishes,

-- 
Hasso Tepper
Elion Enterprises Ltd.
WAN administrator

Re: Link detection

[Thomas Graf]

* Hasso Tepper <200503141435.38227.hasso@estpak.ee> 2005-03-14 14:35
> 1) What's the proper way to detect carrier in Linux for user space 
> application like Quagga?
> 
> True, IFF_RUNNING is from BSD, but as long as the code in 
> net/core/dev.c:dev_get_flags() is there, it gives nice clean way to get 
> carrier status from user space (AFAIK it's used in same way in Solaris). 
> Especially for applications using rtnetlink - ie. without it there is no 
> all info in the netlink message any more. If these applications will 
> receive RTM_NEWLINK, they have to poll kernel "look, maybe carrier status 
> is changed" every time?

You should open a rtnetlink socket and register to the RTMGRP_LINK
multicast group and filter for RTM_NEWLINK with ifi_change & IFF_UP and
then look at ifi_flags & IFF_UP to see whether the interface is now
up or down. You'll receive such a rtnetlink message everytime the
carrier status changes.

> 2) There is no difference between "we know that carrier is really on" and 
> "interface doesn't support carrier detection" in Linux.
> 
> >From users' point of view it might make huge difference - can I rely on info 
> I get from OS, or have to run to the equipment to see whether links are 
> really up or not? It's not critical, but would be still nice to have.

Yes, this is a known weakness. You can use ethtool to work around this a bit.

> 3) Connected routes in carrier down situations. If address are added to the 
> interface, "connected" route is added to the interface as well. Problem is 
> that this route is still there if carrier is down, but there is no 
> difference for user if just carrier is down or interface is 
> administratively down - network behind this interface is still actually 
> unreachable.  

The routes must stay if the interface isn't shut down completely. A loss
of the carrier can be temporarly for just a very short period of time.
What we can do is, like in the neighbour tables, differ between a
adimistrative shutdown and a loss of the carrier with a new flag, would
that help you?

> It's not only the issue related with routing protocols, but the general one. 
> I'm using laptop with two interfaces - ethernet and wireless. If I plug off 
> ethernet cable and walk away from my desk, the network on this ethernet 
> segment should be still reachable for me via wireless (default route), but 
> no ... I have to shut down ethernet interface manually.

I think the reaction upon a loss of the carrier should remain in userspace
to some kind of daemon. The only way that would be half working would be
some kind of timeout representing a watermark which makes a temporary
carrier loss a permanent one (i.e. react on it as it would be a
permanent shutdown). This timer must be disabled by default but can be
activated by the user. My question is, what should happen once the
carrier is found again? Restore the routes from somd kind of backup
tables created while disabling the interface? Leave it to userspace?

> Before someone proposes that - yes, managing these routes can be done from 
> user space in theory (like zebra daemon), but there are several problems 
> regarding this.
> 
> * There are several applications which would be interested in this, but it 
> would be dangerous imho if several applications at once will manage 
> connected routes in kernel.

This is subject to be solved via the use of a netlink daemon taking care
of proper locking. Just needs some more time.

> And I don't see any actual reason not to manage removal/adding connected 
> routes in kernel level if link goes down/up. Or is there any? Why would 
> anyone need route to the interface if it's actually known to be unusable?

Problem #1: Differ between a temporary loss of carrier and a permanent
            failure. Relatively easy from userspace because userspace
	    knows about the strategy.

Problem #2: Strategy upon resume of carrier signal, userspace can simply
            reread its configuration and add the routes again. Kernel
	    must backup them and somehow ensure that they stay
	    consistent. Imagine you delete a route to a certain host but
	    one of your links is down at the moment, the route will be
	    restored once the interface comes back up which is probably
	    unexpected behaviour.

Both issues can be solved with quite some effort though.

Re: Link detection

[Hasso Tepper]

Thomas Graf wrote:
> The routes must stay if the interface isn't shut down completely. A loss
> of the carrier can be temporarly for just a very short period of time.
> What we can do is, like in the neighbour tables, differ between a
> adimistrative shutdown and a loss of the carrier with a new flag, would
> that help you?

In theory yes, if this info is carried over netlink. 

> Problem #1: Differ between a temporary loss of carrier and a permanent
>             failure. Relatively easy from userspace because userspace
>      knows about the strategy.

Is there need for that? There should be knob of course to tune behaviour 
IMHO. Sure there are applications which expect current behaviour. 

> Problem #2: Strategy upon resume of carrier signal, userspace can simply
>             reread its configuration and add the routes again. Kernel
>      must backup them and somehow ensure that they stay
>      consistent. Imagine you delete a route to a certain host but
>      one of your links is down at the moment, the route will be
>      restored once the interface comes back up which is probably
>      unexpected behaviour.

I think that you misunderstood. I don't talk about any routes created by any 
user space applications or by user. These _must_ be untouched of course. 
I'm talking about routes _kernel_ creates if address is added to the 
interface. All other routes are not problem anyway. Even if their next hop 
points to network behind down interface, their next hop is unreachable and 
route shouldn't be used, no?

arena:/home/hasso# ip route | grep 192.168
arena:/home/hasso# ip addr add 192.168.1.1/24 dev eth0
arena:/home/hasso# ip route | grep 192.168
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.1
arena:/home/hasso#

I'm talking only about these routes - created by kernel with link scope.


with my best wishes,

-- 
Hasso Tepper
Elion Enterprises Ltd.
WAN administrator

Re: Link detection

[Thomas Graf]

* Hasso Tepper <200503141605.02959.hasso@estpak.ee> 2005-03-14 16:05
> > The routes must stay if the interface isn't shut down completely. A loss
> > of the carrier can be temporarly for just a very short period of time.
> > What we can do is, like in the neighbour tables, differ between a
> > adimistrative shutdown and a loss of the carrier with a new flag, would
> > that help you?
> 
> In theory yes, if this info is carried over netlink. 

It is carried but not in the way I mentioned. I was wrong with IFF_UP, it
isn't modified. An event is sent upon carrier off and again when the carrier
is found again so you must toggle. Alternatively we can alter a flag in
netif_carrier_(on|off) which is probably a good idea anyway. I'll prepare
a patch for this. Polling sysfs is definitely not the way to go.

> > Problem #1: Differ between a temporary loss of carrier and a permanent
> >             failure. Relatively easy from userspace because userspace
> >      knows about the strategy.
> 
> Is there need for that? There should be knob of course to tune behaviour 
> IMHO. Sure there are applications which expect current behaviour. 

Now that I know you're only taking about implicite routes the problem
is less serious, still a timeout value would be a good thing, one can
always set it to 0 to remove routes immediately or disable it completely.

> I think that you misunderstood. I don't talk about any routes created by any 
> user space applications or by user. These _must_ be untouched of course. 
> I'm talking about routes _kernel_ creates if address is added to the 
> interface. All other routes are not problem anyway. Even if their next hop 
> points to network behind down interface, their next hop is unreachable and 
> route shouldn't be used, no?

Makes a lot more sense now. Yes, the nexthops should be marked unreachable
since the routing cache flushed upon a NETDEV_CHANGE.

What about setting RFT_REJECT/RTN_UNREACHABLE on those routes for the time
the carrier is gone?

Re: Link detection

[Hasso Tepper]

Thomas Graf wrote:
> * Hasso Tepper <200503141605.02959.hasso@estpak.ee> 2005-03-14 16:05
> > I think that you misunderstood. I don't talk about any routes created
> > by any user space applications or by user. These _must_ be untouched of
> > course. I'm talking about routes _kernel_ creates if address is added
> > to the interface. All other routes are not problem anyway. Even if
> > their next hop points to network behind down interface, their next hop
> > is unreachable and route shouldn't be used, no?
>
> Makes a lot more sense now. Yes, the nexthops should be marked
> unreachable since the routing cache flushed upon a NETDEV_CHANGE.
>
> What about setting RFT_REJECT/RTN_UNREACHABLE on those routes for the
> time the carrier is gone?

>From users' point of view I think that any solution is acceptable which 
prevents kernel to actually use these routers if carrier is down. And 
setting flags might be less expensive of course.


with my best wishes,

-- 
Hasso Tepper
Elion Enterprises Ltd.
WAN administrator

Re: Link detection

[Hasso Tepper]

Thomas Graf wrote:
> * Hasso Tepper <200503141435.38227.hasso@estpak.ee> 2005-03-14 14:35
>
> > 1) What's the proper way to detect carrier in Linux for user space
> > application like Quagga?
> >
> > True, IFF_RUNNING is from BSD, but as long as the code in
> > net/core/dev.c:dev_get_flags() is there, it gives nice clean way to get
> > carrier status from user space (AFAIK it's used in same way in
> > Solaris). Especially for applications using rtnetlink - ie. without it
> > there is no all info in the netlink message any more. If these
> > applications will receive RTM_NEWLINK, they have to poll kernel "look,
> > maybe carrier status is changed" every time?
>
> You should open a rtnetlink socket and register to the RTMGRP_LINK
> multicast group and filter for RTM_NEWLINK with ifi_change & IFF_UP and
> then look at ifi_flags & IFF_UP to see whether the interface is now
> up or down. You'll receive such a rtnetlink message everytime the
> carrier status changes.

Forgot to answer this part. We already do all this. IFF_UP doesn't show 
carrier status, but administrative status. IFF_RUNNING shows carrier 
status.


with my best wishes,

-- 
Hasso Tepper
Elion Enterprises Ltd.
WAN administrator

Re: Link detection

[Thomas Graf]

* Hasso Tepper <200503141641.33510.hasso@estpak.ee> 2005-03-14 16:41
> Forgot to answer this part. We already do all this. IFF_UP doesn't show 
> carrier status, but administrative status. IFF_RUNNING shows carrier 
> status.

You're right, forget what I've said.

Re: Link detection

[Thomas Graf]

* Hasso Tepper <200503141641.33510.hasso@estpak.ee> 2005-03-14 16:41
> > > 1) What's the proper way to detect carrier in Linux for user space
> > > application like Quagga?
> > >
> > > True, IFF_RUNNING is from BSD, but as long as the code in
> > > net/core/dev.c:dev_get_flags() is there, it gives nice clean way to get
> > > carrier status from user space (AFAIK it's used in same way in
> > > Solaris). Especially for applications using rtnetlink - ie. without it
> > > there is no all info in the netlink message any more. If these
> > > applications will receive RTM_NEWLINK, they have to poll kernel "look,
> > > maybe carrier status is changed" every time?

I've been looking into it more deeply. Sorry for the wrong information at first,
I thought IFF_RUNNING was unused after the link state was introduced, especially
after seeing IFF_RUNNING being ignored in iproue2 source. I'm fixing up the
drivers still doing it wrong, after that IFF_RUNNING should be accurate and with
no races. The way you're checking carrier at the moment is absolutely correct.

[PATCH] ip link should print NO-CARRIER for !IFF_RUNNING && IFF_UP

[Thomas Graf]

> Till now I thought that IFF_RUNNING flag should be used for that. But some 
> time ago I tried to bug Stephen Hemminger about missing IFF_RUNNING flag in 
> ip output and got answer that IFF_RUNNING is BSD'ism and shouldn't be used 
> any more and /sys/class/net/eth0/carrier should be looked at.

The patch below changes ip link to print NO-CARRIER flag if there is
no carrier and the link is up.

# This is a BitKeeper generated diff -Nru style patch.
#
# ChangeSet
#   2005/03/14 17:12:43+01:00 tgraf@suug.ch 
#   [LINK] Report if link has no carrier
# 
# ip/ipaddress.c
#   2005/03/14 17:12:43+01:00 tgraf@suug.ch +2 -0
#   [LINK] Report if link has no carrier
# 
diff -Nru a/ip/ipaddress.c b/ip/ipaddress.c
--- a/ip/ipaddress.c	2005-03-14 17:13:59 +01:00
+++ b/ip/ipaddress.c	2005-03-14 17:13:59 +01:00
@@ -77,6 +77,8 @@
 void print_link_flags(FILE *fp, unsigned flags, unsigned mdown)
 {
 	fprintf(fp, "<");
+	if (flags & IFF_UP && !(flags & IFF_RUNNING))
+		fprintf(fp, "NO-CARRIER%s", flags ? "," : "");
 	flags &= ~IFF_RUNNING;
 #define _PF(f) if (flags&IFF_##f) { \
                   flags &= ~IFF_##f ; \