on-link assumption in ipv4 routing cache

on-link assumption in ipv4 routing cache

[Vlad Yasevich]

Hi All

Something that came as a surprise to me is that ipv4 implementation
seems to assume that a destination is on-link when there is
no explicit route to it.

A customer submitted an interesting problem where their SCTP associations
kept getting restarted.  The configuration was as follows:

host A: eth0: 17.17.17.17/24
	routing:
		17.17.17.0/24 dev eth0 on-link
		default dev eth1 10.0.0.1

host B: eth0: 18.18.18.18/24
		18.18.18.0/24 dev eth0 on-link
		default dev eth1 10.0.0.1

There were no routes to the "other" subnet on either host.

The application running on both hosts performed a bind to the specific
address as well as SO_BINDTODEVICE.

The result was that both hosts assumed that the peer was on-link, issued
ARP request/replies and successfully connected.  tcpdump showed only packets
on eth0.

This was somewhat of a surprise since I expected a EHOSTUNRACH error since
there were no routes to the destination and SO_DONTROUTE was not set.

I am really curious as to reason for this behavior?

Thanks
-vlad

p.s.  BTW, the solution to the association restart appeared to be
29e75252da20f3ab9e132c68c9aed156b87beae6 ([IPV4] route cache: Introduce rt_genid
for smooth cache invalidation).  There used to be some kind of a
race between cache flushing and SCTP bottom half attempting to recreate
a cache entry.  My guess is that there were rcu issues, since any
cache updated triggered by user application seem to have worked correctly.
Regardless, it appears to have been fixed in 2.6.25.

Re: on-link assumption in ipv4 routing cache

[David Miller]

From: Vlad Yasevich <vladislav.yasevich@hp.com>
Date: Mon, 20 Oct 2008 21:02:44 -0400

> This was somewhat of a surprise since I expected a EHOSTUNRACH error since
> there were no routes to the destination and SO_DONTROUTE was not set.
> 
> I am really curious as to reason for this behavior?

In general the Linux ipv4 stack tries to do things that make it more
likely for successful communication between two nodes.

This is one such example, another is the choice of using the host
based addressing model rather than the interface based addressing
model.

Alexey Kuznetsov is responsible for most of these decisions, he is
a genius.

Re: on-link assumption in ipv4 routing cache

[Vlad Yasevich]

David Miller wrote:
> From: Vlad Yasevich <vladislav.yasevich@hp.com>
> Date: Mon, 20 Oct 2008 21:02:44 -0400
> 
>> This was somewhat of a surprise since I expected a EHOSTUNRACH error since
>> there were no routes to the destination and SO_DONTROUTE was not set.
>>
>> I am really curious as to reason for this behavior?
> 
> In general the Linux ipv4 stack tries to do things that make it more
> likely for successful communication between two nodes.
> 
> This is one such example, another is the choice of using the host
> based addressing model rather than the interface based addressing
> model.
> 
> Alexey Kuznetsov is responsible for most of these decisions, he is
> a genius.
> 

Hi David

Ok, I've found the code and the explanation, but I think there is
a small bug here that's been around a very long time.  There is absolutely
no checking for the interface state.  This means that if the interface
is brought down, we are still going to attempt to route through it.
That seems broken, since the interface was administratively brought down.

I've actually tried to do this with my test app.  I've set it to connect
over a given interface, brought the interface down, and then issued the
connect().  The result is that the app hung until tcp_syn_retries SYNs have
been issued and then returned error.

I've got no issues against on-link assumption as long as there are some smarts
behind it.  The least we could do is use a running interface.

-vlad

Re: on-link assumption in ipv4 routing cache

[David Miller]

From: Vlad Yasevich <vladislav.yasevich@hp.com>
Date: Thu, 23 Oct 2008 13:47:44 -0400

> Ok, I've found the code and the explanation, but I think there is
> a small bug here that's been around a very long time.  There is absolutely
> no checking for the interface state.  This means that if the interface
> is brought down, we are still going to attempt to route through it.
> That seems broken, since the interface was administratively brought down.
> 
> I've actually tried to do this with my test app.  I've set it to connect
> over a given interface, brought the interface down, and then issued the
> connect().  The result is that the app hung until tcp_syn_retries SYNs have
> been issued and then returned error.
> 
> I've got no issues against on-link assumption as long as there are some smarts
> behind it.  The least we could do is use a running interface.

I agree with you that this behavior is at best sub-optimal.

I'm happy to entertain patches the try to make this
area more sane.