Why linux keeps connected routes when link goes down

Why linux keeps connected routes when link goes down

[Ханкин Константин]

Hello!
Why kernel keeps connected routes when link is not in RUNNING state? It's not 
only linux behavior, some other OSes do so, but why?
It's interesting for me because I have a problem with setting up Quagga for 
OSPF support. When link goes down (!IFF_RUNNING), Quagga updates only its own 
routing table. I must use ifplugd to update routes in kernel (actually to 
delete connected route), but I think it is not a right way.
So the main question is: why linux keeps connected routes when link goes down?
PS: ``ip monitor'' reports about link fail after a minute or longer (on 
8139too and 3c905x). On some Acorp cards it doesn't notice link failure. Is it 
a bad driver or crappy cards? On r8169 I have no problems

--
Konstantin
E-mail homecreate@list.ru
Jabber homecreate@jabber.ru

Re: Why linux keeps connected routes when link goes down

[David Miller]

From: Ханкин Константин <homecreate@list.ru>
Date: Thu, 19 Feb 2009 02:03:40 +0500

> So the main question is: why linux keeps connected routes when link
> goes down?

Because the IP addresses are still assigned to the interface.

IP addresses are owned by the "host" rather than specific interfaces
under Linux.  So just bringing an interface down does not disable
IP addresses configured to that interface.

If you want all the routes to go away, explicitly delete the IP
addresses.

Re: Why linux keeps connected routes when link goes down

[Stephen Hemminger]

On Thu, 19 Feb 2009 02:03:40 +0500
Ханкин Константин  <homecreate@list.ru> wrote:

> Hello!
> Why kernel keeps connected routes when link is not in RUNNING state? It's not 
> only linux behavior, some other OSes do so, but why?
> It's interesting for me because I have a problem with setting up Quagga for 
> OSPF support. When link goes down (!IFF_RUNNING), Quagga updates only its own 
> routing table. I must use ifplugd to update routes in kernel (actually to 
> delete connected route), but I think it is not a right way.
> So the main question is: why linux keeps connected routes when link goes down?
> PS: ``ip monitor'' reports about link fail after a minute or longer (on 
> 8139too and 3c905x). On some Acorp cards it doesn't notice link failure. Is it 
> a bad driver or crappy cards? On r8169 I have no problems

There are patches in Vyatta version of Quagga to what you want.
They have been submitted to quagga-developers but the community
process on quagga is much slower than the kernel.

Re: Why linux keeps connected routes when link goes down

[Ханкин Константин]

> IP addresses are owned by the "host" rather than specific interfaces
> under Linux.  So just bringing an interface down does not disable
> IP addresses configured to that interface.
Well. Why routes are not in kernel table when interface is DOWN? And why 
routes are still alive if interface is UP, but not RUNNING?
I think it's right when I can't send anything through interface when there is 
no link. But now I have no link and a route to some network with the highest 
metric

--
Konstantin
E-mail homecreate@list.ru
Jabber homecreate@jabber.ru

Re: Why linux keeps connected routes when link goes down

[Lennart Sorensen]

On Wed, Feb 18, 2009 at 03:52:07PM -0800, David Miller wrote:
> Because the IP addresses are still assigned to the interface.
> 
> IP addresses are owned by the "host" rather than specific interfaces
> under Linux.  So just bringing an interface down does not disable
> IP addresses configured to that interface.
> 
> If you want all the routes to go away, explicitly delete the IP
> addresses.

An interface IP address and the router to the connected subnet are not
the same thing.  Keep the IP address but drop the route to the subnet
you can no longer reach.

By elliminating the route to stuff you can no longer reach, quagga can
do its job to provide an alternative way to reach the subnet.  It can
not do so as long as the kernel insist on providing a now useless route.

-- 
Len Sorensen

Re: Why linux keeps connected routes when link goes down

[David Miller]

From: lsorense@csclub.uwaterloo.ca (Lennart Sorensen)
Date: Fri, 20 Feb 2009 12:19:12 -0500

> By elliminating the route to stuff you can no longer reach, quagga can
> do its job to provide an alternative way to reach the subnet.

This is not necessarily true.

The subnet might be reachable by one of the other interfaces.
Just the same as someone might be able to reach the down
interface's IP address via another one of our interfaces.

The route is tied to the IP address, and the route is therefore
created and destroyed alongside the IP address.

As long as the IP address is active and associated with the
system, so is the route.

Re: Why linux keeps connected routes when link goes down

[Lennart Sorensen]

On Fri, Feb 20, 2009 at 12:15:08PM -0800, David Miller wrote:
> This is not necessarily true.
> 
> The subnet might be reachable by one of the other interfaces.
> Just the same as someone might be able to reach the down
> interface's IP address via another one of our interfaces.

If you assign IPs for the same subnet to two different interfaces in
linux, the results are not particularly useful.  The route contains
the interface associated with the creation of the route, and since
it doesn't go away even if you had the same route for two interfaecs,
it will never use the second one.

> The route is tied to the IP address, and the route is therefore
> created and destroyed alongside the IP address.

That's how it works now.  It wasn't how it worked in linux a long
time ago.  Long ago you had to create the route yourself in userspace
(which was actually to some extent better than the current setup).
The current method is simpler to understand for your typical user though.

> As long as the IP address is active and associated with the
> system, so is the route.

Exactly, and that's the problem when trying to be a router.  You still
want to be able to reach the system through any of the IPs assigned
to it, and you would prefer to still be able to reach any networks
through alternate routes if they exist.  The way linux, BSD and other
unix systems behave is counter productive to routing, and hence not the
way real router OSs work.

Fortunately patching some sense into the linux kernel isn't very hard.

-- 
Len Sorensen

Re: Why linux keeps connected routes when link goes down

[Ханкин Константин]

> Fortunately patching some sense into the linux kernel isn't very hard.

Can I hope that it will be done some day?

Linux is used as router in many places, such behavior is a suicide. Of course, 
one can use ifplugd (so do I), but it's a crap for network OS, don't you think 
so?

Who made linux network subsystem? Can Linus Torvalds say some words about this 
problem?

I'm not a developer, but network administrator. So I can't make a patch. I can 
only find a solution or make it. So please make my work a bit easier

--
Konstantin
E-mail homecreate@list.ru
Jabber homecreate@jabber.ru

Re: Why linux keeps connected routes when link goes down

[Lennart Sorensen]

On Mon, Feb 23, 2009 at 11:02:55PM +0500, ???????????? ???????????????????? wrote:
> Can I hope that it will be done some day?

I don't have much hope that the kernel will ever be changed to behave
as desired for a router.

> Linux is used as router in many places, such behavior is a suicide. Of course, 
> one can use ifplugd (so do I), but it's a crap for network OS, don't you think 
> so?

Well you will need some user space support to handle static routes,
unless you want to make major changes to the kernel route management.
We use zebra to manage static routes when the link goes up or down, with
the kernel simply patched to delete the subnet route when an interface
goes down, and re add it when it comes back up.  All other routes are
handled by zebra.

With the current patch I am using all routes assigned to an interface goes
away when the link goes down, but the kernel only recreates the directly
connected network's route when it comes up.  Any other static routes
have to be recreated by user space, hence the user of zebra for that.

Better and more friendly would be if routes could be marked inactive
and ignored by the kernel for routing decisions when a link went down.
That way multiple routes with different metrics could be used with the
best one who's link is up would be in use.  I find this too much to try
and put into the kernel though, so I will stick to zebra for that task.

> Who made linux network subsystem? Can Linus Torvalds say some words about this 
> problem?
> 
> I'm not a developer, but network administrator. So I can't make a patch. I can 
> only find a solution or make it. So please make my work a bit easier

-- 
Len Sorensen

Re: Why linux keeps connected routes when link goes down

[Stephen Hemminger]

On Mon, 23 Feb 2009 16:14:11 -0500
lsorense@csclub.uwaterloo.ca (Lennart Sorensen) wrote:

> On Mon, Feb 23, 2009 at 11:02:55PM +0500, ???????????? ???????????????????? wrote:
> > Can I hope that it will be done some day?
> 
> I don't have much hope that the kernel will ever be changed to behave
> as desired for a router.
> 
> > Linux is used as router in many places, such behavior is a suicide. Of course, 
> > one can use ifplugd (so do I), but it's a crap for network OS, don't you think 
> > so?
> 
> Well you will need some user space support to handle static routes,
> unless you want to make major changes to the kernel route management.
> We use zebra to manage static routes when the link goes up or down, with
> the kernel simply patched to delete the subnet route when an interface
> goes down, and re add it when it comes back up.  All other routes are
> handled by zebra.
> 
> With the current patch I am using all routes assigned to an interface goes
> away when the link goes down, but the kernel only recreates the directly
> connected network's route when it comes up.  Any other static routes
> have to be recreated by user space, hence the user of zebra for that.
> 
> Better and more friendly would be if routes could be marked inactive
> and ignored by the kernel for routing decisions when a link went down.
> That way multiple routes with different metrics could be used with the
> best one who's link is up would be in use.  I find this too much to try
> and put into the kernel though, so I will stick to zebra for that task.
> 
> > Who made linux network subsystem? Can Linus Torvalds say some words about this 
> > problem?
> > 
> > I'm not a developer, but network administrator. So I can't make a patch. I can 
> > only find a solution or make it. So please make my work a bit easier
> 

Like I said already, it is possible to fix Quagga; we have done it the
patches are available. unfortunately quagga is without a real
software maintenance infrastructure at this point.

Re: Why linux keeps connected routes when link goes down

[David Miller]

From: lsorense@csclub.uwaterloo.ca (Lennart Sorensen)
Date: Mon, 23 Feb 2009 09:38:30 -0500

> On Fri, Feb 20, 2009 at 12:15:08PM -0800, David Miller wrote:
> > The route is tied to the IP address, and the route is therefore
> > created and destroyed alongside the IP address.
> 
> That's how it works now.  It wasn't how it worked in linux a long
> time ago.  Long ago you had to create the route yourself in userspace
> (which was actually to some extent better than the current setup).
> The current method is simpler to understand for your typical user though.

I don't remember Linux ever not creating a subnet route when an IP
address was added to an interface.  And I also don't remember Linux
ever deleting an IP address on interface down.

Maybe it happened back in 0.99pl5 times, but we're talking at least
10 years ago.

Re: Why linux keeps connected routes when link goes down

[David Miller]

From: Ханкин Константин <homecreate@list.ru>
Date: Mon, 23 Feb 2009 23:02:55 +0500

> Linux is used as router in many places, such behavior is a
> suicide. Of course, one can use ifplugd (so do I), but it's a crap
> for network OS, don't you think so?

The default behavior for a general purpose operating system is
to increase the likelyhood of successful communication.

And the way to maximize successful communication is to associate
addresses with the host rather than the interface.

This decision was made at least 10 years ago, and if you think what we
have now is broken just imagine how much struff would explode if we
changed things.

Re: Why linux keeps connected routes when link goes down

[Lennart Sorensen]

On Mon, Feb 23, 2009 at 03:33:53PM -0800, David Miller wrote:
> From: lsorense@csclub.uwaterloo.ca (Lennart Sorensen)
> Date: Mon, 23 Feb 2009 09:38:30 -0500
> 
> > On Fri, Feb 20, 2009 at 12:15:08PM -0800, David Miller wrote:
> > > The route is tied to the IP address, and the route is therefore
> > > created and destroyed alongside the IP address.
> > 
> > That's how it works now.  It wasn't how it worked in linux a long
> > time ago.  Long ago you had to create the route yourself in userspace
> > (which was actually to some extent better than the current setup).
> > The current method is simpler to understand for your typical user though.
> 
> I don't remember Linux ever not creating a subnet route when an IP
> address was added to an interface.  And I also don't remember Linux
> ever deleting an IP address on interface down.

Not IP address.  And no it never deleted the route either.

> Maybe it happened back in 0.99pl5 times, but we're talking at least
> 10 years ago.

I did run 0.99something on my first linux system, so I remember the
transition happening at some point.  I remember distribution's network
scripts creating routes after assigning the IP to an interface.  I don't
remember how long ago that was though.

Now it had nothing to do with link state, it was simply a matter that
setting an ip and netmask on an interface did not create a route the
way newer kernels have done (where newer probably covers most years of
linux kernels).  All route management was userspace's job back then.
In the case of running quagga and trying to do link detect and route
management based on link state, having the one route the kernel auto
creates there is a pain in the ass and it would be nice if it wasn't
there based on some sysctl option.  I haven't found one though.  I would
be happy to have zebra be responsible for the network route for all
interfaces for consistency.  That way all route management without
exception would be a userspace problem.  This would of course not be the
default, since for most normal users this is not the desired behaviour,
but for an advanced router it very much is desired behaviour.

-- 
Len Sorensen