netdev.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* IPv6 path discovery oddities - flushing the routing cache resolves
@ 2013-10-16 10:31 Valentijn Sessink
  2013-10-16 15:48 ` Hannes Frederic Sowa
  0 siblings, 1 reply; 10+ messages in thread
From: Valentijn Sessink @ 2013-10-16 10:31 UTC (permalink / raw)
  To: netdev

Hello list,

I'm experiencing difficulties with IPv6 path discovery. The setup is 
quite simple, a machine with native IPv6, no special routing - let's 
call it the "server" for now. Unfortunately, I seem to loose 
connectivity multiple times a day - and after digging in, I found this 
to be "too big" messages that weren't honored at the server. The network 
consists of something like:

server --- hosting --- others ---- SIXXS tunnel with 1280 MTU --- me.

A "ip -6 route list cache" would show a cached route to my "client", but 
one without MTU. Then after "ip -6 route flush cache", and after trying 
to send a large packet (for example issuing "ps uaxww" on an ssh 
prompt), "ip -6 route list cache" will show a correct MTU.

But after a while, things start to go wrong again, and another "ip -6 
route flush cache" is needed.

The server is running 3.8.0-something (ubuntu 12.04 with a newer kernel).

tcpdump shows that on reception of icmpv6 "too big", nothing happens 
(i.e. the "too big" packet will be sent time and again), and after the 
"ip -6 route flush cache", suddenly the "too big" message is honored.

I saw a couple of path discovery issues on this list, more specifically 
one with the subject "IPv6 path MTU discovery broken" earlier this month 
- but I'm not sure it's the same issue (because the original submitter 
specifically mentions kernels 3.10 and 3.11 and has a much more 
complicated routing table).

The "server" just has:

auto br0
iface br0 inet6 static
   address 2a01:xxxx:xxx:xxxx::2
   netmask 64
   gateway fe80::1
   bridge_ports eth0
   bridge_maxwait 0

Nothing special, no special routes, no routing daemons.

If I can do anything to shine more light on this issue, please tell me so.

Best regards,

Valentijn

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: IPv6 path discovery oddities - flushing the routing cache resolves
  2013-10-16 10:31 IPv6 path discovery oddities - flushing the routing cache resolves Valentijn Sessink
@ 2013-10-16 15:48 ` Hannes Frederic Sowa
  2013-10-17 10:53   ` Valentijn Sessink
  0 siblings, 1 reply; 10+ messages in thread
From: Hannes Frederic Sowa @ 2013-10-16 15:48 UTC (permalink / raw)
  To: Valentijn Sessink; +Cc: netdev

On Wed, Oct 16, 2013 at 12:31:31PM +0200, Valentijn Sessink wrote:
> Hello list,
> 
> I'm experiencing difficulties with IPv6 path discovery. The setup is 
> quite simple, a machine with native IPv6, no special routing - let's 
> call it the "server" for now. Unfortunately, I seem to loose 
> connectivity multiple times a day - and after digging in, I found this 
> to be "too big" messages that weren't honored at the server. The network 
> consists of something like:
> 
> server --- hosting --- others ---- SIXXS tunnel with 1280 MTU --- me.
> 
> A "ip -6 route list cache" would show a cached route to my "client", but 
> one without MTU. Then after "ip -6 route flush cache", and after trying 
> to send a large packet (for example issuing "ps uaxww" on an ssh 
> prompt), "ip -6 route list cache" will show a correct MTU.

Oh, thats another very good hint.

> But after a while, things start to go wrong again, and another "ip -6 
> route flush cache" is needed.
> 
> The server is running 3.8.0-something (ubuntu 12.04 with a newer kernel).
> 
> tcpdump shows that on reception of icmpv6 "too big", nothing happens 
> (i.e. the "too big" packet will be sent time and again), and after the 
> "ip -6 route flush cache", suddenly the "too big" message is honored.
> 
> I saw a couple of path discovery issues on this list, more specifically 
> one with the subject "IPv6 path MTU discovery broken" earlier this month 
> - but I'm not sure it's the same issue (because the original submitter 
> specifically mentions kernels 3.10 and 3.11 and has a much more 
> complicated routing table).

I do think these two issues are connected. Could you send me a the
corresponding ip route output and /proc/net/ipv6_route output for when it
works and mtus are correctly handled and when it does not work?

Thanks,

  Hannes

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: IPv6 path discovery oddities - flushing the routing cache resolves
  2013-10-16 15:48 ` Hannes Frederic Sowa
@ 2013-10-17 10:53   ` Valentijn Sessink
  2013-10-18  3:04     ` Hannes Frederic Sowa
  0 siblings, 1 reply; 10+ messages in thread
From: Valentijn Sessink @ 2013-10-17 10:53 UTC (permalink / raw)
  To: netdev

Hi,

Thanks for your fast response, please see below. Sorry for the rather 
long mail, due to the routing tables. Even though they're really short, 
the output of ipv6_route is extensive...

On 16-10-13 17:48, Hannes Frederic Sowa wrote:
> I do think these two issues are connected. Could you send me a the
> corresponding ip route output and /proc/net/ipv6_route output for when it
> works and mtus are correctly handled and when it does not work?

When it works (please note that the routing cache at first doesn't show 
an MTU, that only happens after a "too big" has been received, hence the 
difference before and after a "ps uaxww"), I have the following - this 
is from last night:

# ip -6 ro li
2a01:4f8:200:546b::/64 dev br0  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev br0  proto kernel  metric 256
fe80::/64 dev vnet0  proto kernel  metric 256
fe80::/64 dev vnet1  proto kernel  metric 256
default via fe80::1 dev br0  metric 1024

# ip -6 ro li cache
2001:7b8:1529:0:cd7a:7681:b261:af09 via fe80::1 dev br0  metric 0
     cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0  metric 0
     cache

# ps uaxww
[... output cut for brevity ...]

# ip -6 ro li cache
2001:7b8:1529:0:cd7a:7681:b261:af09 via fe80::1 dev br0  metric 0
     cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0  metric 0
     cache  expires 577sec mtu 1280

# cat /proc/net/ipv6_route
200107b815290000cd7a7681b261af09 80 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000000 00000001 000062b2 01000023
   br0
20011af8ff030003021966fffe2606dd 80 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000000 00000001 00000002 01400023
   br0
2a0104f80000a0a10000000000020001 80 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000000 00000000 00000001 01000003
   br0
2a0104f80200546b0000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
   br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
  eth0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
   br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet1
00000000000000000000000000000000 00 00000000000000000000000000000000 00
fe800000000000000000000000000001 00000400 00000000 00000000 00000003
   br0
00000000000000000000000000000000 00 00000000000000000000000000000000 00
00000000000000000000000000000000 ffffffff 00000001 0001b230 00200200
    lo
00000000000000000000000000000001 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000037 80200001
    lo
2a0104f80200546b0000000000000002 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000003 000e6bc4 80200001
    lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 0000003b 80200001
    lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000000 80200001
    lo
fe80000000000000fc5400fffe17ee5b 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000000 80200001
    lo
fe80000000000000fc5400fffe497447 80 00000000000000000000000000000000 00
00000000000000000000000000000000 00000000 00000001 00000000 80200001
    lo
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
   br0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
  eth0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00
00000000000000000000000000000000 00000100 00000000 00000000 00000001
vnet1
00000000000000000000000000000000 00 00000000000000000000000000000000 00
00000000000000000000000000000000 ffffffff 00000001 0001b230 00200200
    lo

(Please note, while trying, I found a rather interesting output, showing 
negative time; if it's normal, please ignore:
# ip -statistics -6 ro li cache
2001:1af8:ff03:3:219:66ff:fe26:6dd via fe80::1 dev br0  metric 0
     cache  expires -23sec users 2 used 2 age 637sec mtu 1280)


Anyway, today, suddenly it doesn't work anymore:

# ip -6 ro li
2a01:4f8:200:546b::/64 dev br0  proto kernel  metric 256
fe80::/64 dev eth0  proto kernel  metric 256
fe80::/64 dev br0  proto kernel  metric 256
fe80::/64 dev vnet0  proto kernel  metric 256
fe80::/64 dev vnet1  proto kernel  metric 256
fe80::/64 dev vnet2  proto kernel  metric 256
default via fe80::1 dev br0  metric 1024

# ip -6 ro li cache
2001:7b8:2ff:37a::2 via fe80::1 dev br0  metric 0
     cache
2001:7b8:1529::f via fe80::1 dev br0  metric 0
     cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0  metric 0
     cache
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0  metric 0
     cache
2a01:4f8:200:546b::5 via 2a01:4f8:200:546b::5 dev br0  metric 0
     cache
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0  metric 0
     cache

# ip -statistics -6 ro li cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0  metric 0
     cache  users 4 used 9448
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0  metric 0
     cache  users 4 used 260467 age 0sec
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0  metric 0
     cache  users 1 used 987 age 205sec
# ip -6 ro li cache
2001:7b8:1529:0:213:8fff:feda:140c via fe80::1 dev br0  metric 0
     cache
2001:7b8:1529:0:b845:bb25:5d99:2a07 via fe80::1 dev br0  metric 0
     cache
2a01:4f8:200:546b::2eef via 2a01:4f8:200:546b::2eef dev br0  metric 0
     cache
(The second time suddenly a couple of routes were gone - I don't know 
why). After "cat /proc/net/ipv6_route > /tmp/route-bad" I typed a "route 
-6 flush cache" and TCP started flowing again. Here's ipv6_route:

200107b802ff037a0000000000000002 80 00000000000000000000000000000000 00 
fe800000000000000000000000000001 00000000 00000000 00000006 01000003 
   br0
200107b815290000000000000000000f 80 00000000000000000000000000000000 00 
fe800000000000000000000000000001 00000000 00000000 00000006 01000003 
   br0
200107b81529000002138ffffeda140c 80 00000000000000000000000000000000 00 
fe800000000000000000000000000001 00000000 00000004 000024aa 01000023 
   br0
200107b815290000b845bb255d992a07 80 00000000000000000000000000000000 00 
fe800000000000000000000000000001 00000000 00000004 0003f79f 01000023 
   br0
2a0104f80200546b0000000000000005 80 00000000000000000000000000000000 00 
2a0104f80200546b0000000000000005 00000000 00000000 00000018 01000001 
   br0
2a0104f80200546b0000000000002eef 80 00000000000000000000000000000000 00 
2a0104f80200546b0000000000002eef 00000000 00000001 000003db 01000001 
   br0
2a0104f80200546b0000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
   br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
  eth0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
   br0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet0
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet1
fe800000000000000000000000000000 40 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet2
00000000000000000000000000000000 00 00000000000000000000000000000000 00 
fe800000000000000000000000000001 00000400 00000000 00000000 00000003 
   br0
00000000000000000000000000000000 00 00000000000000000000000000000000 00 
00000000000000000000000000000000 ffffffff 00000001 00064566 00200200 
    lo
00000000000000000000000000000001 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000167 80200001 
    lo
2a0104f80200546b0000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
2a0104f80200546b0000000000000002 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 0000000b 002a4bfe 80200001 
    lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
fe800000000000000000000000000000 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 00300001 
    lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 000001ed 80200001 
    lo
fe80000000000000d63d7efffee306e9 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 80200001 
    lo
fe80000000000000fc5400fffe17ee5b 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 80200001 
    lo
fe80000000000000fc5400fffe497447 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 80200001 
    lo
fe80000000000000fc5400fffedd5a81 80 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000000 00000001 00000000 80200001 
    lo
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
   br0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
  eth0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet0
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet1
ff000000000000000000000000000000 08 00000000000000000000000000000000 00 
00000000000000000000000000000000 00000100 00000000 00000000 00000001 
vnet2
00000000000000000000000000000000 00 00000000000000000000000000000000 00 
00000000000000000000000000000000 ffffffff 00000001 00064566 00200200 
    lo

I hope this helps.

Best regards,

Valentijn

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: IPv6 path discovery oddities - flushing the routing cache resolves
  2013-10-17 10:53   ` Valentijn Sessink
@ 2013-10-18  3:04     ` Hannes Frederic Sowa
  2013-10-18  6:44       ` Valentijn Sessink
  0 siblings, 1 reply; 10+ messages in thread
From: Hannes Frederic Sowa @ 2013-10-18  3:04 UTC (permalink / raw)
  To: Valentijn Sessink; +Cc: netdev

On Thu, Oct 17, 2013 at 12:53:56PM +0200, Valentijn Sessink wrote:
> Hi,
> 
> Thanks for your fast response, please see below. Sorry for the rather 
> long mail, due to the routing tables. Even though they're really short, 
> the output of ipv6_route is extensive...
> 
> On 16-10-13 17:48, Hannes Frederic Sowa wrote:
> >I do think these two issues are connected. Could you send me a the
> >corresponding ip route output and /proc/net/ipv6_route output for when it
> >works and mtus are correctly handled and when it does not work?
> 
> When it works (please note that the routing cache at first doesn't show 
> an MTU, that only happens after a "too big" has been received, hence the 
> difference before and after a "ps uaxww"), I have the following - this 
> is from last night:

Thanks, I needed this to verify I am on the right track replicating this.
2001:1af8:ff03:3:219:66ff:fe26:6dd is the other end of the connection, I
guess?

Greetings,

  Hannes

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: IPv6 path discovery oddities - flushing the routing cache resolves
  2013-10-18  3:04     ` Hannes Frederic Sowa
@ 2013-10-18  6:44       ` Valentijn Sessink
  2013-10-19  8:42         ` Hannes Frederic Sowa
  0 siblings, 1 reply; 10+ messages in thread
From: Valentijn Sessink @ 2013-10-18  6:44 UTC (permalink / raw)
  To: netdev

On 18-10-13 05:04, Hannes Frederic Sowa wrote:
> Thanks, I needed this to verify I am on the right track replicating this.
> 2001:1af8:ff03:3:219:66ff:fe26:6dd is the other end of the connection, I
> guess?

Yes, the working connection (first example) is from
2001:1af8:ff03:3:219:66ff:fe26:6dd. The non-working connection should
have an MTU of 1280 on the 2001:7b8:1529:: subnet connections (those are
tunneled, with the tunnel restricting the MTU).

Best regards,

Valentijn

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: IPv6 path discovery oddities - flushing the routing cache resolves
  2013-10-18  6:44       ` Valentijn Sessink
@ 2013-10-19  8:42         ` Hannes Frederic Sowa
  2013-10-19  8:53           ` Hannes Frederic Sowa
  2013-10-19 10:12           ` Steinar H. Gunderson
  0 siblings, 2 replies; 10+ messages in thread
From: Hannes Frederic Sowa @ 2013-10-19  8:42 UTC (permalink / raw)
  To: Valentijn Sessink; +Cc: netdev, sgunderson

On Fri, Oct 18, 2013 at 08:44:46AM +0200, Valentijn Sessink wrote:
> On 18-10-13 05:04, Hannes Frederic Sowa wrote:
> > Thanks, I needed this to verify I am on the right track replicating this.
> > 2001:1af8:ff03:3:219:66ff:fe26:6dd is the other end of the connection, I
> > guess?
> 
> Yes, the working connection (first example) is from
> 2001:1af8:ff03:3:219:66ff:fe26:6dd. The non-working connection should
> have an MTU of 1280 on the 2001:7b8:1529:: subnet connections (those are
> tunneled, with the tunnel restricting the MTU).

I got access to a nice test box yesterday where I could brute force the
problem in parallel (it was a PITA). This is what I found:

This first patch solves the problem of a complete lockdown of all sockets
towards one ipv6 destination. This can happen if we recheck the ipv6 fib
(expiration is ok) and we get back a rt6_info where we apply the new metrics
information on. After the check the dst entry expires and we do a relookup.
We try to insert the same routing information into the fib which results only
in a call to rt6_clean_expires. Because we don't reset the dst.expires value a
later update of mtu information won't update the expiration time because of
the strange semantics in rt6_update_expires. This patch should fix this.

diff --git a/include/net/ip6_fib.h b/include/net/ip6_fib.h
index 6738f34..3932633 100644
--- a/include/net/ip6_fib.h
+++ b/include/net/ip6_fib.h
@@ -164,6 +164,7 @@ static inline struct inet6_dev *ip6_dst_idev(struct dst_entry *dst)
 
 static inline void rt6_clean_expires(struct rt6_info *rt)
 {
+	rt->dst.expires = 0;
 	rt->rt6i_flags &= ~RTF_EXPIRES;
 }

The second patch resolves the problem that the socket keeps hanging on
outdated mtu information which gets invalidated just after processing. We
need to relookup the destination entry in case the socket expires. This
helps a socket to free the cached dst before applying the mtu information
to an already expired dst which will be reinserted (see above, it will
only call rt6_clean_expires on the dst_entry). This is normally not a
problem, but in the process of the creation of the cloned dst_entry we
end up copying the metric information from the non-DST_CACHEd route to
the dst_entry (ip6_rt_copy/dst_copy_metrics). Because the information are
held in inetpeer storage and the key for the expired dst and the new dst
have the same key we overwrite the metrics store which currently is in
use by two rt6_infos. So we just invalidate the newly installed metrics
information and will use the interface mtu just after the PACKET_TOO_BIG
notification, which leads to hangs of the connection. A flush of the
cached routing entries causes relookups, so this a workaround.

This patch should fix this:

diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index c3130ff..7629022 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -1064,10 +1064,13 @@ static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie)
 	if (rt->rt6i_genid != rt_genid_ipv6(dev_net(rt->dst.dev)))
 		return NULL;
 
-	if (rt->rt6i_node && (rt->rt6i_node->fn_sernum == cookie))
-		return dst;
+	if (!rt->rt6i_node && (rt->rt6i_node->fn_sernum != cookie))
+		return NULL;
 
-	return NULL;
+	if (rt6_check_expired(rt))
+		return NULL;
+
+	return dst;
 }
 
 static struct dst_entry *ip6_negative_advice(struct dst_entry *dst)


I had the patches in test for a few hours on some VMs where I could normally
reproduce this issue within 5 minutes. They are for testing only and I don't
know if they resolve all issues. I also have to check why rt6_update_expires
has such strange expiration update logic.

Steinar and Valentijn could you give them a test drive?

Greetings,

  Hannes

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: IPv6 path discovery oddities - flushing the routing cache resolves
  2013-10-19  8:42         ` Hannes Frederic Sowa
@ 2013-10-19  8:53           ` Hannes Frederic Sowa
  2013-10-19 10:12           ` Steinar H. Gunderson
  1 sibling, 0 replies; 10+ messages in thread
From: Hannes Frederic Sowa @ 2013-10-19  8:53 UTC (permalink / raw)
  To: Valentijn Sessink, netdev, sgunderson

On Sat, Oct 19, 2013 at 10:42:25AM +0200, Hannes Frederic Sowa wrote:
> This patch should fix this:
> 
> diff --git a/net/ipv6/route.c b/net/ipv6/route.c
> index c3130ff..7629022 100644
> --- a/net/ipv6/route.c
> +++ b/net/ipv6/route.c
> @@ -1064,10 +1064,13 @@ static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie)
>  	if (rt->rt6i_genid != rt_genid_ipv6(dev_net(rt->dst.dev)))
>  		return NULL;
>  
> -	if (rt->rt6i_node && (rt->rt6i_node->fn_sernum == cookie))
> -		return dst;
> +	if (!rt->rt6i_node && (rt->rt6i_node->fn_sernum != cookie))
> +		return NULL;

I just rewrote this patch to have a clean diff. I missed to rewrite the '&&'
to '||'.

Correct patch is:

diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index c3130ff..17b3cc1 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -1064,10 +1064,13 @@ static struct dst_entry *ip6_dst_check(struct dst_entry *dst, u32 cookie)
 	if (rt->rt6i_genid != rt_genid_ipv6(dev_net(rt->dst.dev)))
 		return NULL;
 
-	if (rt->rt6i_node && (rt->rt6i_node->fn_sernum == cookie))
-		return dst;
+	if (!rt->rt6i_node || (rt->rt6i_node->fn_sernum != cookie))
+		return NULL;
 
-	return NULL;
+	if (rt6_check_expired(rt))
+		return NULL;
+
+	return dst;
 }
 
 static struct dst_entry *ip6_negative_advice(struct dst_entry *dst)

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: IPv6 path discovery oddities - flushing the routing cache resolves
  2013-10-19  8:42         ` Hannes Frederic Sowa
  2013-10-19  8:53           ` Hannes Frederic Sowa
@ 2013-10-19 10:12           ` Steinar H. Gunderson
  2013-10-19 14:24             ` Valentijn Sessink
  1 sibling, 1 reply; 10+ messages in thread
From: Steinar H. Gunderson @ 2013-10-19 10:12 UTC (permalink / raw)
  To: Valentijn Sessink, netdev

On Sat, Oct 19, 2013 at 10:42:25AM +0200, Hannes Frederic Sowa wrote:
> I got access to a nice test box yesterday where I could brute force the
> problem in parallel (it was a PITA). This is what I found:

Thanks so much for looking into this! It's great that to hear that it's not
just me being weird =) I'll try rebooting the machine in question into a
kernel with these patches later today; given how intermittent it is, it can
of course only show if the patches are not fully working (or, over time,
make it likely that they are).

/* Steinar */
-- 
Homepage: http://www.sesse.net/

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: IPv6 path discovery oddities - flushing the routing cache resolves
  2013-10-19 10:12           ` Steinar H. Gunderson
@ 2013-10-19 14:24             ` Valentijn Sessink
  2013-10-19 20:24               ` Hannes Frederic Sowa
  0 siblings, 1 reply; 10+ messages in thread
From: Valentijn Sessink @ 2013-10-19 14:24 UTC (permalink / raw)
  To: netdev; +Cc: Steinar H. Gunderson

op 19-10-13 12:12, Steinar H. Gunderson schreef:
> Thanks so much for looking into this! It's great that to hear that it's not
> just me being weird =)

I'm not sure the patch addresses weirdness ;-P

On the more serious side: I have this problem with a 3.8.0 kernel 
(Ubuntu 12.04 with updated kernel). I saw that you (Steinar) experience 
this with 3.10 and 3.11, iirc. I'm a little bit worried by the fact that 
this has not been seen for at least six months - and now suddenly pops up.

Does anyone have an idea when this race condition was introduced?

(It could be, that certain patches have been included in Ubuntu 
3.8.0-something but not in an official 3.8).

Best regards,

Valentijn
p.s. I won't be able to test any patches until tuesday - sorry.

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: IPv6 path discovery oddities - flushing the routing cache resolves
  2013-10-19 14:24             ` Valentijn Sessink
@ 2013-10-19 20:24               ` Hannes Frederic Sowa
  0 siblings, 0 replies; 10+ messages in thread
From: Hannes Frederic Sowa @ 2013-10-19 20:24 UTC (permalink / raw)
  To: Valentijn Sessink; +Cc: netdev, Steinar H. Gunderson

On Sat, Oct 19, 2013 at 04:24:59PM +0200, Valentijn Sessink wrote:
> op 19-10-13 12:12, Steinar H. Gunderson schreef:
> >Thanks so much for looking into this! It's great that to hear that it's not
> >just me being weird =)
> 
> I'm not sure the patch addresses weirdness ;-P
> 
> On the more serious side: I have this problem with a 3.8.0 kernel 
> (Ubuntu 12.04 with updated kernel). I saw that you (Steinar) experience 
> this with 3.10 and 3.11, iirc. I'm a little bit worried by the fact that 
> this has not been seen for at least six months - and now suddenly pops up.
> 
> Does anyone have an idea when this race condition was introduced?

Do a bisect. ;)

I had a look at the history but could not find a single commit
to blame. Even reference counting changes (e.g. removal of neighbour
caching in routes) or something in the way the socket release callbacks
are processed by tcp could introduce a way where these problems are more
likely to happen. I wondered, too.

A lot of people of the IPv6 community often state that mtu discovery is broken
because of filtering packet_too_bigs. Maybe most people assumed broken
firewalls and did not actual check if it is the kernels problem. So, thanks
again for reporting this issue.

Greetings,

  Hannes

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2013-10-19 20:24 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-10-16 10:31 IPv6 path discovery oddities - flushing the routing cache resolves Valentijn Sessink
2013-10-16 15:48 ` Hannes Frederic Sowa
2013-10-17 10:53   ` Valentijn Sessink
2013-10-18  3:04     ` Hannes Frederic Sowa
2013-10-18  6:44       ` Valentijn Sessink
2013-10-19  8:42         ` Hannes Frederic Sowa
2013-10-19  8:53           ` Hannes Frederic Sowa
2013-10-19 10:12           ` Steinar H. Gunderson
2013-10-19 14:24             ` Valentijn Sessink
2013-10-19 20:24               ` Hannes Frederic Sowa

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).