Issue with GRE tunnel, created with "local any" on kernel 3.14

Issue with GRE tunnel, created with "local any" on kernel 3.14

[Sergey Popov]

[-- Attachment #1: Type: text/plain, Size: 3909 bytes --]

Hi. I have updated kernel on one of my servers to 3.14 and hit strange
issue:

Simple setup:

serv1(172.30.0.100) <-> serv2(172.30.0.251)

Serv1 is on 3.11.7
Serv2 is on 3.13.6

Gre tunnel(serv1):
ip tunnel add tun_test mode gre remote 172.30.0.251 local any ttl 225
ifconfig tun_test 192.168.0.1/30

Gre tunnel(serv2):
ip tunnel add tun_test mode gre remote 172.30.0.100 local any ttl 225
ifconfig tun_test 192.168.0.2/30

Everything work as expected.

Now, with updated kernel on serv2 i have this:


serv2 ~ # ping 192.168.0.1 -c 10
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.339 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.303 ms

--- 192.168.0.1 ping statistics ---
10 packets transmitted, 2 received, 80% packet loss, time 9002ms
rtt min/avg/max/mdev = 0.303/0.321/0.339/0.018 ms

Huh? tcpdump on serv1 show me really strange thing:

bgp ~ # tcpdump -i lan0 -nn ip proto gre
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lan0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:49:33.338964 IP 172.30.0.251 > 172.30.0.100: GREv0, length 88: IP
192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 1, length 64
16:49:33.339010 IP 172.30.0.100 > 172.30.0.251: GREv0, length 88: IP
192.168.0.1 > 192.168.0.2: ICMP echo reply, id 5259, seq 1, length 64
16:49:34.337945 IP 172.30.0.251 > 172.30.0.100: GREv0, length 88: IP
192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 2, length 64
16:49:34.337988 IP 172.30.0.100 > 172.30.0.251: GREv0, length 88: IP
192.168.0.1 > 192.168.0.2: ICMP echo reply, id 5259, seq 2, length 64
16:49:35.336948 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP
192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 3, length 64
16:49:36.340999 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP
192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 4, length 64
16:49:37.340971 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP
192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 5, length 64
16:49:38.340985 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP
192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 6, length 64
16:49:39.340971 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP
192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 7, length 64
16:49:40.340918 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP
192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 8, length 64
16:49:41.340947 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP
192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 9, length 64
16:49:42.340970 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP
192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 10, length 64
^C
12 packets captured
14 packets received by filter
0 packets dropped by kernel

Huh? 0.0.0.0? o_O

I have tried to up and done tunnels few times. Sometimes 0.0.0.0 is
replaced by correct address in the middle of icmp "session" for one or
two packets, but usually only first one or two packets have correct src
address.

I have tried some(not all of them) kernels from 3.14.2 up to 3.15.5 -
all have the same issue.

If i set "local 172.30.0.251" on serv2 the issue is gone completely.

Looking on changelog i discovered some fixes to gre, but i could not
understand if they can be culprit or not.

MTU on both real network ifaces are 1500, MTU on gre tunnels are 1476,
so i suppose that's right(and not an issue, cause mentioned icmp packets
are too small to hit MTU problem).

I am running Gentoo Linux, but issue is detected in vanilla kernel too,
so it's not distro problem, i suppose.

Can anybody help me to track this issue down?

-- 
Best regards, Sergey Popov
Gentoo developer
Gentoo Desktop-effects project lead
Gentoo Proxy maintainers project lead


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 538 bytes --]

Re: Issue with GRE tunnel, created with "local any" on kernel 3.14

[Dmitry Popov]

On Fri, 18 Jul 2014 17:01:23 +0400
Sergey Popov <pinkbyte@gentoo.org> wrote:

> Hi. I have updated kernel on one of my servers to 3.14 and hit strange
> issue:

> serv1(172.30.0.100) <-> serv2(172.30.0.251)

> Gre tunnel(serv1):
> ip tunnel add tun_test mode gre remote 172.30.0.251 local any ttl 225
> ifconfig tun_test 192.168.0.1/30
> 
> Gre tunnel(serv2):
> ip tunnel add tun_test mode gre remote 172.30.0.100 local any ttl 225
> ifconfig tun_test 192.168.0.2/30

> 16:49:39.340971 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP
> 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 7, length 64

Hi!

It seems we have this issue since 7d442fab0 (ipv4: Cache dst in tunnels).
It is in ip_tunnel_xmit: fl4 is initialized with saddr = 0 in
init_tunnel_flow(&fl4, protocol, dst, ...), net/ipv4/ip_tunnel.c:610,
and isn't changed until iptunnel_xmit(skb->sk, ...), line 677, because 
connected = true and rt is almost always cached.

I have no patch to fix this issue, but as a workaround, you can use 
ip tunnel change tun_test tos inherit.

Re: Issue with GRE tunnel, created with "local any" on kernel 3.14

[Sergey Popov]

[-- Attachment #1: Type: text/plain, Size: 1532 bytes --]

18.07.2014 22:32, Dmitry Popov пишет:
> On Fri, 18 Jul 2014 17:01:23 +0400
> Sergey Popov <pinkbyte@gentoo.org> wrote:
> 
>> Hi. I have updated kernel on one of my servers to 3.14 and hit strange
>> issue:
> 
>> serv1(172.30.0.100) <-> serv2(172.30.0.251)
> 
>> Gre tunnel(serv1):
>> ip tunnel add tun_test mode gre remote 172.30.0.251 local any ttl 225
>> ifconfig tun_test 192.168.0.1/30
>>
>> Gre tunnel(serv2):
>> ip tunnel add tun_test mode gre remote 172.30.0.100 local any ttl 225
>> ifconfig tun_test 192.168.0.2/30
> 
>> 16:49:39.340971 IP 0.0.0.0 > 172.30.0.100: GREv0, length 88: IP
>> 192.168.0.2 > 192.168.0.1: ICMP echo request, id 5259, seq 7, length 64
> 
> Hi!
> 
> It seems we have this issue since 7d442fab0 (ipv4: Cache dst in tunnels).
> It is in ip_tunnel_xmit: fl4 is initialized with saddr = 0 in
> init_tunnel_flow(&fl4, protocol, dst, ...), net/ipv4/ip_tunnel.c:610,
> and isn't changed until iptunnel_xmit(skb->sk, ...), line 677, because 
> connected = true and rt is almost always cached.
> 
> I have no patch to fix this issue, but as a workaround, you can use 
> ip tunnel change tun_test tos inherit.

Thanks, your workaround works as expected if tos value defined
explicitly. Though, I do not understand why: iproute2 documentation says
that default tos value in 'ip tunnel' command is already 'inherit'.

-- 
Best regards, Sergey Popov
Gentoo developer
Gentoo Desktop Effects project lead
Gentoo Qt project lead
Gentoo Proxy maintainers project lead


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 538 bytes --]

Re: Issue with GRE tunnel, created with "local any" on kernel 3.14

[Dmitry Popov]

On Thu, 31 Jul 2014 10:29:00 +0400
Sergey Popov <pinkbyte@gentoo.org> wrote:

> > I have no patch to fix this issue, but as a workaround, you can use 
> > ip tunnel change tun_test tos inherit.
> 
> Thanks, your workaround works as expected if tos value defined
> explicitly. Though, I do not understand why: iproute2 documentation says
> that default tos value in 'ip tunnel' command is already 'inherit'.

man ip-tunnel says default tos is 00, and that's true. If you're about *.tex
files under iproute2.git/doc, they are more than 10 years old, I doubt if they
are maintained. Otherwise I don't know where you found it.

This workaround works because setting tos=inherit disables tunnel route cache,
which is buggy in your release (it doesn't cache source address).

Btw, thanks for the report, David has accepted the patch today, commit
95cb574 (ip_tunnel(ipv4): fix tunnels with "local any remote $remote_ip"), so
I think next kernel releases will have it.