Unplug ethernet cable, the route persists. Why?

Unplug ethernet cable, the route persists. Why?

[Mike Caoco]

Hello,

This may have been discussed, but all search engines couldn't give me a good answer...

I notice that when an interface is up/running, a local route is in the routing table:

$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:13:20:0e:2f:ed  
          inet addr:192.168.1.125  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::213:20ff:fe0e:2fed/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:35984995 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7409151 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3252413825 (3.2 GB)  TX bytes:1340077250 (1.3 GB)

$ ip route
192.168.20.0/24 dev eth0  proto kernel  scope link  src 192.168.20.120
192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.125 
default via 192.168.20.254 dev eth1  metric 100 

After I unplug the cable from eth1, the RUNNING flag disappears, but the route is still there:

$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 00:13:20:0e:2f:ed  
          inet addr:192.168.1.125  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::213:20ff:fe0e:2fed/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:35985023 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7409151 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3252415633 (3.2 GB)  TX bytes:1340077250 (1.3 GB)

$ ip route
192.168.20.0/24 dev eth0  proto kernel  scope link  src 192.168.20.120 
192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.125 
default via 192.168.20.254 dev eth1  metric 100 

And that *prevents* from using the default route to reach 192.168.1/24 subnet after eth1 is out.

I looked at the code, it seems the IFF_RUNNING flag change is ignored in dev_change_flags():

void __dev_notify_flags(struct net_device *dev, unsigned int old_flags)
{
        .....
        if (dev->flags & IFF_UP &&
            (changes & ~(IFF_UP | IFF_PROMISC | IFF_ALLMULTI | IFF_VOLATILE)))
                call_netdevice_notifiers(NETDEV_CHANGE, dev);
}

I searched in the Internet, and saw some people suggest using an application listener (eg, netplug) to remove the route. 

My question is why cannot the kernel remove the route automatically when the link becomes down?  Why should this complexity be pushed to the user to find a program to do that?

Thanks,
Joe


      

Re: Unplug ethernet cable, the route persists. Why?

[Stephen Hemminger]

On Wed, 24 Nov 2010 11:48:03 -0800 (PST)
Mike Caoco <caoco2002@yahoo.com> wrote:

> Hello,
> 
> This may have been discussed, but all search engines couldn't give me a good answer...
> 
> I notice that when an interface is up/running, a local route is in the routing table:
> 
> $ ifconfig eth1
> eth1      Link encap:Ethernet  HWaddr 00:13:20:0e:2f:ed  
>           inet addr:192.168.1.125  Bcast:192.168.1.255  Mask:255.255.255.0
>           inet6 addr: fe80::213:20ff:fe0e:2fed/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:35984995 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:7409151 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000 
>           RX bytes:3252413825 (3.2 GB)  TX bytes:1340077250 (1.3 GB)
> 
> $ ip route
> 192.168.20.0/24 dev eth0  proto kernel  scope link  src 192.168.20.120
> 192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.125 
> default via 192.168.20.254 dev eth1  metric 100 
> 
> After I unplug the cable from eth1, the RUNNING flag disappears, but the route is still there:
> 
> $ ifconfig eth1
> eth1      Link encap:Ethernet  HWaddr 00:13:20:0e:2f:ed  
>           inet addr:192.168.1.125  Bcast:192.168.1.255  Mask:255.255.255.0
>           inet6 addr: fe80::213:20ff:fe0e:2fed/64 Scope:Link
>           UP BROADCAST MULTICAST  MTU:1500  Metric:1
>           RX packets:35985023 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:7409151 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000 
>           RX bytes:3252415633 (3.2 GB)  TX bytes:1340077250 (1.3 GB)
> 
> $ ip route
> 192.168.20.0/24 dev eth0  proto kernel  scope link  src 192.168.20.120 
> 192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.125 
> default via 192.168.20.254 dev eth1  metric 100 
> 
> And that *prevents* from using the default route to reach 192.168.1/24 subnet after eth1 is out.
> 
> I looked at the code, it seems the IFF_RUNNING flag change is ignored in dev_change_flags():
> 
> void __dev_notify_flags(struct net_device *dev, unsigned int old_flags)
> {
>         .....
>         if (dev->flags & IFF_UP &&
>             (changes & ~(IFF_UP | IFF_PROMISC | IFF_ALLMULTI | IFF_VOLATILE)))
>                 call_netdevice_notifiers(NETDEV_CHANGE, dev);
> }
> 
> I searched in the Internet, and saw some people suggest using an application listener (eg, netplug) to remove the route. 
> 
> My question is why cannot the kernel remove the route automatically when the link becomes down?  Why should this complexity be pushed to the user to find a program to do that?
> 

Because there is no reason for the kernel to not expect the link to come back.
It is up to user space to do routing policy. For desktop/laptop users this is
done typically with NetworkManager or Connman; for routers this is done with
Quagga; and for servers use other tools.

If the kernel automatically removed the route, it would cause routing daemons
to recompute the route table (and propagate the change) every time a cable
got pulled or NIC needed to be reset.


-- 

Re: Unplug ethernet cable, the route persists. Why?

[Mike Caoco]

--- On Wed, 11/24/10, Stephen Hemminger <shemminger@vyatta.com> wrote:

> From: Stephen Hemminger <shemminger@vyatta.com>
> Subject: Re: Unplug ethernet cable, the route persists.  Why?
> To: "Mike Caoco" <caoco2002@yahoo.com>
> Cc: "Netdev" <netdev@vger.kernel.org>, "LKML" <linux-kernel@vger.kernel.org>
> Date: Wednesday, November 24, 2010, 12:18 PM
> On Wed, 24 Nov 2010 11:48:03 -0800
> (PST)
> Mike Caoco <caoco2002@yahoo.com>
> wrote:
> 
> > Hello,
> > 
> > This may have been discussed, but all search engines
> couldn't give me a good answer...
> > 
> > I notice that when an interface is up/running, a local
> route is in the routing table:
> > 
> > $ ifconfig eth1
> > eth1      Link encap:Ethernet 
> HWaddr 00:13:20:0e:2f:ed  
> >           inet
> addr:192.168.1.125  Bcast:192.168.1.255 
> Mask:255.255.255.0
> >           inet6
> addr: fe80::213:20ff:fe0e:2fed/64 Scope:Link
> >           UP
> BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
> >           RX
> packets:35984995 errors:0 dropped:0 overruns:0 frame:0
> >           TX
> packets:7409151 errors:0 dropped:0 overruns:0 carrier:0
> >       
>    collisions:0 txqueuelen:1000 
> >           RX
> bytes:3252413825 (3.2 GB)  TX bytes:1340077250 (1.3
> GB)
> > 
> > $ ip route
> > 192.168.20.0/24 dev eth0  proto kernel 
> scope link  src 192.168.20.120
> > 192.168.1.0/24 dev eth1  proto kernel  scope
> link  src 192.168.1.125 
> > default via 192.168.20.254 dev eth1  metric 100 
> > 
> > After I unplug the cable from eth1, the RUNNING flag
> disappears, but the route is still there:
> > 
> > $ ifconfig eth1
> > eth1      Link encap:Ethernet 
> HWaddr 00:13:20:0e:2f:ed  
> >           inet
> addr:192.168.1.125  Bcast:192.168.1.255 
> Mask:255.255.255.0
> >           inet6
> addr: fe80::213:20ff:fe0e:2fed/64 Scope:Link
> >           UP
> BROADCAST MULTICAST  MTU:1500  Metric:1
> >           RX
> packets:35985023 errors:0 dropped:0 overruns:0 frame:0
> >           TX
> packets:7409151 errors:0 dropped:0 overruns:0 carrier:0
> >       
>    collisions:0 txqueuelen:1000 
> >           RX
> bytes:3252415633 (3.2 GB)  TX bytes:1340077250 (1.3
> GB)
> > 
> > $ ip route
> > 192.168.20.0/24 dev eth0  proto kernel 
> scope link  src 192.168.20.120 
> > 192.168.1.0/24 dev eth1  proto kernel  scope
> link  src 192.168.1.125 
> > default via 192.168.20.254 dev eth1  metric 100 
> > 
> > And that *prevents* from using the default route to
> reach 192.168.1/24 subnet after eth1 is out.
> > 
> > I looked at the code, it seems the IFF_RUNNING flag
> change is ignored in dev_change_flags():
> > 
> > void __dev_notify_flags(struct net_device *dev,
> unsigned int old_flags)
> > {
> >         .....
> >         if
> (dev->flags & IFF_UP &&
> >         
>    (changes & ~(IFF_UP | IFF_PROMISC |
> IFF_ALLMULTI | IFF_VOLATILE)))
> >             
>    call_netdevice_notifiers(NETDEV_CHANGE,
> dev);
> > }
> > 
> > I searched in the Internet, and saw some people
> suggest using an application listener (eg, netplug) to
> remove the route. 
> > 
> > My question is why cannot the kernel remove the route
> automatically when the link becomes down?  Why should
> this complexity be pushed to the user to find a program to
> do that?
> > 
> 
> Because there is no reason for the kernel to not expect the
> link to come back.
> It is up to user space to do routing policy. For
> desktop/laptop users this is
> done typically with NetworkManager or Connman; for routers
> this is done with
> Quagga; and for servers use other tools.
> 
> If the kernel automatically removed the route, it would
> cause routing daemons
> to recompute the route table (and propagate the change)
> every time a cable
> got pulled or NIC needed to be reset.
> 

So if you rely on NetworkManager or Connman or Quagga to remove the route, the routing daemons will recompute the route table anyway.  So why cannot this be done in the kernel?  

Even when no NetworkManager/Quagga is present, I think it is a legitimate reason to recompute the route when a cable is unplugged, which should not be a frequent event unless when under error conditions.

Thanks,



      

Re: Unplug ethernet cable, the route persists. Why?

[David Miller]

From: Mike Caoco <caoco2002@yahoo.com>
Date: Wed, 24 Nov 2010 12:29:43 -0800 (PST)

> Even when no NetworkManager/Quagga is present, I think it is a
> legitimate reason to recompute the route when a cable is unplugged,
> which should not be a frequent event unless when under error
> conditions.

Cards periodically reset themselves, faulty switchs flap occaisionally,
this is life and it shouldn't cause route table recomputations across
your entire region.

Also Stephen listed places where such policy should be employed in
userspace, he absolutely did not say they should act that way by
default.

Re: Unplug ethernet cable, the route persists. Why?

[Hans de Bruin]

On 11/24/2010 08:48 PM, Mike Caoco wrote:
> Hello,
>
> This may have been discussed, but all search engines couldn't give me a good answer...
>
> I notice that when an interface is up/running, a local route is in the routing table:
>
> $ ifconfig eth1
> eth1      Link encap:Ethernet  HWaddr 00:13:20:0e:2f:ed
>            inet addr:192.168.1.125  Bcast:192.168.1.255  Mask:255.255.255.0
>            inet6 addr: fe80::213:20ff:fe0e:2fed/64 Scope:Link
>            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>            RX packets:35984995 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:7409151 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:1000
>            RX bytes:3252413825 (3.2 GB)  TX bytes:1340077250 (1.3 GB)
>
> $ ip route
> 192.168.20.0/24 dev eth0  proto kernel  scope link  src 192.168.20.120
> 192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.125
> default via 192.168.20.254 dev eth1  metric 100
>
> After I unplug the cable from eth1, the RUNNING flag disappears, but the route is still there:
>
> $ ifconfig eth1
> eth1      Link encap:Ethernet  HWaddr 00:13:20:0e:2f:ed
>            inet addr:192.168.1.125  Bcast:192.168.1.255  Mask:255.255.255.0
>            inet6 addr: fe80::213:20ff:fe0e:2fed/64 Scope:Link
>            UP BROADCAST MULTICAST  MTU:1500  Metric:1
>            RX packets:35985023 errors:0 dropped:0 overruns:0 frame:0
>            TX packets:7409151 errors:0 dropped:0 overruns:0 carrier:0
>            collisions:0 txqueuelen:1000
>            RX bytes:3252415633 (3.2 GB)  TX bytes:1340077250 (1.3 GB)
>
> $ ip route
> 192.168.20.0/24 dev eth0  proto kernel  scope link  src 192.168.20.120
> 192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.125
> default via 192.168.20.254 dev eth1  metric 100
>
> And that *prevents* from using the default route to reach 192.168.1/24 subnet after eth1 is out.

Well suppose the default route is used, and the source address is of the 
packets stay 192.168.1.125, then there is no way for the peers to 
respond to the packets. I do not know wat sets the source address in 
linux, but on windows it will allway's be the one you do not want.

-- 
Hans

Re: Unplug ethernet cable, the route persists. Why?

[Ben Gamari]

On Wed, 24 Nov 2010 12:29:43 -0800 (PST), Mike Caoco <caoco2002@yahoo.com> wrote:
> So if you rely on NetworkManager or Connman or Quagga to remove the
> route, the routing daemons will recompute the route table anyway.  So
> why cannot this be done in the kernel?

This is policy. In the Linux world we generally strive to separate
policy from mechanism, leaving the former to userspace. This allows
(potentially complex) policy decisions to be made in user-space. The
reason for this is two-fold: First, every line of kernel code introduces
the potentially for a bug and error handling in the kernel is generally
more complex than it is in user-space. Secondly, allowing user-space to
handle policy allows users to do things with the kernel that kernel
developers did not envision. This flexibility is one reason why the
kernel is so suited for running on anything from your cell-phone to 4000
processor big iron.

> Even when no NetworkManager/Quagga is present, I think it is a
> legitimate reason to recompute the route when a cable is unplugged,
> which should not be a frequent event unless when under error
> conditions.

There have to be real, demonstrable benefits for moving policy into the
kernel (i.e. the recent discussions concerning per-session
cgroups). Considering how long the Linux networking subsystem has
existed, I highly doubt there is a good reason to move this sort of
routing policy into the kernel that has not already been discussed.

Cheers,

- Ben

Re: Unplug ethernet cable, the route persists. Why?

[Jarek Poplawski]

Ben Gamari wrote:
> On Wed, 24 Nov 2010 12:29:43 -0800 (PST), Mike Caoco <caoco2002@yahoo.com> wrote:
>> So if you rely on NetworkManager or Connman or Quagga to remove the
>> route, the routing daemons will recompute the route table anyway.  So
>> why cannot this be done in the kernel?
> 
> This is policy. In the Linux world we generally strive to separate
> policy from mechanism, leaving the former to userspace. This allows
> (potentially complex) policy decisions to be made in user-space. The
> reason for this is two-fold: First, every line of kernel code introduces
> the potentially for a bug and error handling in the kernel is generally
> more complex than it is in user-space. Secondly, allowing user-space to
> handle policy allows users to do things with the kernel that kernel
> developers did not envision. This flexibility is one reason why the
> kernel is so suited for running on anything from your cell-phone to 4000
> processor big iron.

Secondly and a half, if you add a specific route you may really
mean it, and prefer not to send at all than use default.

Cheers,
Jarek P.