Routes with unreachable gateways are staying in the routing table and they are functional

Routes with unreachable gateways are staying in the routing table and they are functional

[Fedor Babkin]

Hello,

I noticed that in case a network interface has two addresses assigned (no matter if IPv4 or IPv6) from subnets A and B, the route to reach subnet C via the gateway in A is staying configured and functional when subnet A is removed from the interface. Let me illustrate the behavior with IPv4 example.

Starting point. eth0 has 2 subnets, the route to reach 3-rd subnet via gw in one of them is present:
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:12:34:57 brd ff:ff:ff:ff:ff:ff
    inet 10.20.10.3/24 brd 10.20.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.30.10.3/24 brd 10.30.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe12:3457/64 scope link
       valid_lft forever preferred_lft forever
root@ubuntu14-4-vm:~# ip ro ls
10.11.10.0/24 via 10.30.10.1 dev eth0
10.20.10.0/24 dev eth0  proto kernel  scope link  src 10.20.10.3
10.30.10.0/24 dev eth0  proto kernel  scope link  src 10.30.10.3

Delete 10.30.10.3/24, the indirect route persists:
root@ubuntu14-4-vm:~# ip ro ls
10.11.10.0/24 via 10.30.10.1 dev eth0
10.20.10.0/24 dev eth0  proto kernel  scope link  src 10.20.10.3

Ping to destinations in 10.11.10.0/24 works, unreachable gateway 10.30.10.1 is resolved, even recovers in case of neighbor flushes.
root@ubuntu14-4-vm:~# ping 10.11.10.1
PING 10.11.10.1 (10.11.10.1) 56(84) bytes of data.
64 bytes from 10.11.10.1: icmp_seq=1 ttl=64 time=0.706 ms
root@ubuntu14-4-vm:~# ip n l
10.30.10.1 dev eth0 lladdr 52:54:00:12:34:56 REACHABLE

The above was observed with kernel version 2.6.32 and confirmed with 3.13.0 In case of IPv6 this behavior is more disturbing as there is always LLA assigned, so applications have to orchestrate the removal of unreachable routes. 
If it's an intended implementation, would be helpful to know a reason behind it. Thanks for your support.

Regards,
Fedor

Re: Routes with unreachable gateways are staying in the routing table and they are functional

[Julian Anastasov]

	Hello,

On Fri, 22 Aug 2014, Fedor Babkin wrote:

> Hello,
> 
> I noticed that in case a network interface has two addresses assigned (no matter if IPv4 or IPv6) from subnets A and B, the route to reach subnet C via the gateway in A is staying configured and functional when subnet A is removed from the interface. Let me illustrate the behavior with IPv4 example.
> 
> Starting point. eth0 has 2 subnets, the route to reach 3-rd subnet via gw in one of them is present:
> eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
>     link/ether 52:54:00:12:34:57 brd ff:ff:ff:ff:ff:ff
>     inet 10.20.10.3/24 brd 10.20.10.255 scope global eth0
>        valid_lft forever preferred_lft forever
>     inet 10.30.10.3/24 brd 10.30.10.255 scope global eth0
>        valid_lft forever preferred_lft forever
>     inet6 fe80::5054:ff:fe12:3457/64 scope link
>        valid_lft forever preferred_lft forever
> root@ubuntu14-4-vm:~# ip ro ls
> 10.11.10.0/24 via 10.30.10.1 dev eth0
> 10.20.10.0/24 dev eth0  proto kernel  scope link  src 10.20.10.3
> 10.30.10.0/24 dev eth0  proto kernel  scope link  src 10.30.10.3
> 
> Delete 10.30.10.3/24, the indirect route persists:
> root@ubuntu14-4-vm:~# ip ro ls
> 10.11.10.0/24 via 10.30.10.1 dev eth0
> 10.20.10.0/24 dev eth0  proto kernel  scope link  src 10.20.10.3
> 
> Ping to destinations in 10.11.10.0/24 works, unreachable gateway 10.30.10.1 is resolved, even recovers in case of neighbor flushes.
> root@ubuntu14-4-vm:~# ping 10.11.10.1
> PING 10.11.10.1 (10.11.10.1) 56(84) bytes of data.
> 64 bytes from 10.11.10.1: icmp_seq=1 ttl=64 time=0.706 ms
> root@ubuntu14-4-vm:~# ip n l
> 10.30.10.1 dev eth0 lladdr 52:54:00:12:34:56 REACHABLE
> 
> The above was observed with kernel version 2.6.32 and confirmed with 3.13.0 In case of IPv6 this behavior is more disturbing as there is always LLA assigned, so applications have to orchestrate the removal of unreachable routes. 
> If it's an intended implementation, would be helpful to know a reason behind it. Thanks for your support.

	It is expensive to implement solution for this
problem. If we try to be very strict by handling it
at FIB level (catch route removal) simple operations
like secondary address promotion or route replacement
can lead to cascade of route removals. At the end,
it is again the user who have to take care to add
all lost routes back. So, it is a complex task to
solve.

	Here is recent discussion on such topic:

http://marc.info/?t=139030500700005&r=1&w=2
http://marc.info/?t=139055890700002&r=1&w=2

Regards

--
Julian Anastasov <ja@ssi.bg>

Aw: Re: Routes with unreachable gateways are staying in the routing table and they are functional

[Fedor Babkin]

Hello Julian,

Thanks for your feedback. For IPv4 the situation is clear, you have to configure more that one subnet to the interface in order to start experiencing this issue. However exactly the same issue exists in IPv6, where once you configure a single address, you have to count on effectively having 2 addresses on the interface, due to the presence of a link-local address fe80::xxxx. Moreover with IPv6 stateless address autoconfiguration (RFC 4862), there is more address configuration dynamics in IPv6 comparing to IPv4. I would say this issue has a higher visibility and side-effect potential. Is there anyone looking into this issue from IPv6 perspective?

Thanks,
Fedor


	Hello,

On Fri, 22 Aug 2014, Fedor Babkin wrote:

> Hello,
> 
> I noticed that in case a network interface has two addresses assigned (no matter if IPv4 or IPv6) from subnets A and B, the route to reach subnet C via the gateway in A is staying configured and functional when subnet A is removed from the interface. Let me illustrate the behavior with IPv4 example.
> 
> Starting point. eth0 has 2 subnets, the route to reach 3-rd subnet via gw in one of them is present:
> eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
>     link/ether 52:54:00:12:34:57 brd ff:ff:ff:ff:ff:ff
>     inet 10.20.10.3/24 brd 10.20.10.255 scope global eth0
>        valid_lft forever preferred_lft forever
>     inet 10.30.10.3/24 brd 10.30.10.255 scope global eth0
>        valid_lft forever preferred_lft forever
>     inet6 fe80::5054:ff:fe12:3457/64 scope link
>        valid_lft forever preferred_lft forever
> root@...ntu14-4-vm:~# ip ro ls
> 10.11.10.0/24 via 10.30.10.1 dev eth0
> 10.20.10.0/24 dev eth0  proto kernel  scope link  src 10.20.10.3
> 10.30.10.0/24 dev eth0  proto kernel  scope link  src 10.30.10.3
> 
> Delete 10.30.10.3/24, the indirect route persists:
> root@...ntu14-4-vm:~# ip ro ls
> 10.11.10.0/24 via 10.30.10.1 dev eth0
> 10.20.10.0/24 dev eth0  proto kernel  scope link  src 10.20.10.3
> 
> Ping to destinations in 10.11.10.0/24 works, unreachable gateway 10.30.10.1 is resolved, even recovers in case of neighbor flushes.
> root@...ntu14-4-vm:~# ping 10.11.10.1
> PING 10.11.10.1 (10.11.10.1) 56(84) bytes of data.
> 64 bytes from 10.11.10.1: icmp_seq=1 ttl=64 time=0.706 ms
> root@...ntu14-4-vm:~# ip n l
> 10.30.10.1 dev eth0 lladdr 52:54:00:12:34:56 REACHABLE
> 
> The above was observed with kernel version 2.6.32 and confirmed with 3.13.0 In case of IPv6 this behavior is more disturbing as there is always LLA assigned, so applications have to orchestrate the removal of unreachable routes. 
> If it's an intended implementation, would be helpful to know a reason behind it. Thanks for your support.

	It is expensive to implement solution for this
problem. If we try to be very strict by handling it
at FIB level (catch route removal) simple operations
like secondary address promotion or route replacement
can lead to cascade of route removals. At the end,
it is again the user who have to take care to add
all lost routes back. So, it is a complex task to
solve.

	Here is recent discussion on such topic:

http://marc.info/?t=139030500700005&r=1&w=2
http://marc.info/?t=139055890700002&r=1&w=2

Regards

--
Julian Anastasov
--
 
 

Gesendet: Freitag, 22. August 2014 um 18:28 Uhr
Von: "Julian Anastasov" <ja@ssi.bg>
An: "Fedor Babkin" <fedor.babkin@gmx.net>
Cc: netdev@vger.kernel.org
Betreff: Re: Routes with unreachable gateways are staying in the routing table and they are functional
Hello, On Fri, 22 Aug 2014, Fedor Babkin wrote: > Hello, > > I noticed that in case a network interface has two addresses assigned (no matter if IPv4 or IPv6) from subnets A and B, the route to reach subnet C via the gateway in A is staying configured and functional when subnet A is removed from the interface. Let me illustrate the behavior with IPv4 example. > > Starting point. eth0 has 2 subnets, the route to reach 3-rd subnet via gw in one of them is present: > eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 > link/ether 52:54:00:12:34:57 brd ff:ff:ff:ff:ff:ff > inet 10.20.10.3/24 brd 10.20.10.255 scope global eth0 > valid_lft forever preferred_lft forever > inet 10.30.10.3/24 brd 10.30.10.255 scope global eth0 > valid_lft forever preferred_lft forever > inet6 fe80::5054:ff:fe12:3457/64 scope link > valid_lft forever preferred_lft forever > root@ubuntu14-4-vm:~# ip ro ls > 10.11.10.0/24 via 10.30.10.1 dev eth0 > 10.20.10.0/24 dev eth0 proto kernel scope link src 10.20.10.3 > 10.30.10.0/24 dev eth0 proto kernel scope link src 10.30.10.3 > > Delete 10.30.10.3/24, the indirect route persists: > root@ubuntu14-4-vm:~# ip ro ls > 10.11.10.0/24 via 10.30.10.1 dev eth0 > 10.20.10.0/24 dev eth0 proto kernel scope link src 10.20.10.3 > > Ping to destinations in 10.11.10.0/24 works, unreachable gateway 10.30.10.1 is resolved, even recovers in case of neighbor flushes. > root@ubuntu14-4-vm:~# ping 10.11.10.1 > PING 10.11.10.1 (10.11.10.1) 56(84) bytes of data. > 64 bytes from 10.11.10.1: icmp_seq=1 ttl=64 time=0.706 ms > root@ubuntu14-4-vm:~# ip n l > 10.30.10.1 dev eth0 lladdr 52:54:00:12:34:56 REACHABLE > > The above was observed with kernel version 2.6.32 and confirmed with 3.13.0 In case of IPv6 this behavior is more disturbing as there is always LLA assigned, so applications have to orchestrate the removal of unreachable routes. > If it's an intended implementation, would be helpful to know a reason behind it. Thanks for your support. It is expensive to implement solution for this problem. If we try to be very strict by handling it at FIB level (catch route removal) simple operations like secondary address promotion or route replacement can lead to cascade of route removals. At the end, it is again the user who have to take care to add all lost routes back. So, it is a complex task to solve. Here is recent discussion on such topic: http://marc.info/?t=139030500700005&r=1&w=2 http://marc.info/?t=139055890700002&r=1&w=2 Regards -- Julian Anastasov

Re: Aw: Re: Routes with unreachable gateways are staying in the routing table and they are functional

[Hannes Frederic Sowa]

Hi,

On Di, 2014-08-26 at 09:53 +0200, Fedor Babkin wrote:
> Thanks for your feedback. For IPv4 the situation is clear, you have to configure more that one subnet to the interface in order to start experiencing this issue. However exactly the same issue exists in IPv6, where once you configure a single address, you have to count on effectively having 2 addresses on the interface, due to the presence of a link-local address fe80::xxxx. Moreover with IPv6 stateless address autoconfiguration (RFC 4862), there is more address configuration dynamics in IPv6 comparing to IPv4. I would say this issue has a higher visibility and side-effect potential. Is there anyone looking into this issue from IPv6 perspective?

Actually, it is an error and an implementation problem that we currently
add interface routes when adding ipv6 addresses to an interface. Those
routes should only be allocated when the host receives on-link
information (redirect, router advertisment). So in case of IPv6 routing
information should always be handled separate from routing information.

Also see:
$ git log -G IFA_F_NOPREFIXROUTE net/ipv6/addrconf.c

Greetings,
Hannes