Re: PROBLEM: IPv6 TCP-Connections resetting

Re: PROBLEM: IPv6 TCP-Connections resetting

[Neal Cardwell]

On Fri, Apr 5, 2013 at 11:48 AM, Tetja Rediske <tetja@tetja.de> wrote:
> I tracked it down with 'git bisect' to commit:
>
>   093d04d42fa094f6740bb188f0ad0c215ff61e2c
...

Thanks for the  detailed report!

> 11:52:04.634656 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 136

Would you be able to re-run your tests with a tcpdump command line like:

  tcpdump -v -n -X -s 1600 icmp6

And report the full dump of this first ICMP6 packet in the exchange?

It seems that perhaps the parsing/validation of this packet is failing
somehow, so it would be nice to know exactly what that packet looked
like.

Also, are both sides of your test running 3.9.0-rc5+?

And can you please cc netdev@vger.kernel.org if you follow up with more details?

Thanks!
neal

Re: PROBLEM: IPv6 TCP-Connections resetting

[Eric Dumazet]

On Fri, 2013-04-05 at 17:48 +0200, Tetja Rediske wrote:
> Hi,
> 

CC netdev and Duan Jiong (author of bad commit)

> [1.] One line summary of the problem:
> 
> IPv6 TCP-Connections resetting 
> 
> [2.] Full description of the problem/report:
> 
> In the last weeks we updated some of our systems to a 3.8.4 Kernel.
> Since then sometimes we can't connect to services running IPv6,
> Apache and Openssh tested. 
> 
> We got this on different machines with x86 and x86_64 Kernels. On
> x86_64 it is more random, but on x86 i can reproduce it permanently
> (Just opening any TCP Connection 1st time or after some short delay).
> Connecting quick after the reset again will work as expected. It will
> also work, if you keep another connection open.
> 
> Before I got to the Kernel, I just kept an strace on an userspace
> process, but it did not notice the connection attempt. After this I
> monitored the connection with tcpdump, but nothing unusual.
> 
> Then I did a rollback to the older Kernel and it worked as expected.
> 
> I tracked it down with 'git bisect' to commit:
> 
>   093d04d42fa094f6740bb188f0ad0c215ff61e2c
> 
> I also tested latest git state available.
> 
> [3.] Keywords (i.e., modules, networking, kernel):
> 
> networking, IPv6
> 
> [4.] Kernel information
> [4.1.] Kernel version (from /proc/version): 
> 
>   since commit: 093d04d42fa094f6740bb188f0ad0c215ff61e2c
> 
> [4.2.] Kernel .config file:
> [5.] Most recent kernel version which did not have the bug:
> 
> none
> 
> [6.] Output of Oops.. message (if applicable) with symbolic information
>      resolved (see Documentation/oops-tracing.txt)
> [7.] A small shell script or example program which triggers the
>      problem (if possible)
> [8.] Environment
> [8.1.] Software (add the output of the ver_linux script here)
> 
> Different systems, mostly reproduced on this one:
> 
> Linux dns03.tetja.de 3.9.0-rc5+ #10 SMP Fri Apr 5 16:55:54 CEST 2013
> i686 AMD Athlon(tm) 64 X2 Dual Core Processor 5600+ AuthenticAMD
> GNU/Linux
> 
> Gnu C                  4.4.5
> Gnu make               3.82
> binutils               2.22
> util-linux             2.22.2
> mount                  debug
> module-init-tools      12
> e2fsprogs              1.42
> jfsutils               1.1.15
> reiserfsprogs          3.6.21
> xfsprogs               3.1.10
> Linux C Library        2.15
> Dynamic linker (ldd)   2.15
> Procps                 3.3.4
> Net-tools              1.60_p20120127084908
> Kbd                    1.15.3wip
> Sh-utils               8.20
> Modules Loaded
> 
> Connections looking like this on booth sites:
> 
> 11:52:04.634315 IP6 2a00:1828:0:1::10.51808 >
> 2a00:1828:1000:1102::2.80: Flags [S], seq 103067898, win 5760, options
> [mss 1440,sackOK,TS val 232579708 ecr 0,nop,wscale 7], length 0
> 
> 11:52:04.634354 IP6 2a00:1828:1000:1102::2.80 >
> 2a00:1828:0:1::10.51808: Flags [S.], seq 3352491415, ack 103067899, win
> 14280, options [mss 1440,sackOK,TS val 174797959 ecr
> 232579708,nop,wscale 7], length 0 
> 
> 11:52:04.634656 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 136
> 
> 11:52:04.634715 IP6 2a00:1828:0:1::10.51808 >
> 2a00:1828:1000:1102::2.80: Flags [.], ack 1, win 45, options
> [nop,nop,TS val 232579708 ecr 174797959], length 0
> 
> 11:52:04.634726 IP6 2a00:1828:1000:1102::2.80 >
> 2a00:1828:0:1::10.51808: Flags [R], seq 3352491416, win 0, length 0
> 
> 11:52:04.635027 IP6 2a00:1828:0:1::10.51808 >
> 2a00:1828:1000:1102::2.80: Flags [P.], seq 1:359, ack 1, win 45,
> options [nop,nop,TS val 232579708 ecr 174797959], length 358
> 
> 11:52:04.635037 IP6 2a00:1828:1000:1102::2.80 >
> 2a00:1828:0:1::10.51808: Flags [R], seq 3352491416, win 0, length 0
> 
> 11:52:04.635071 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 112
> 
> 11:52:04.635246 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 112
> 
> Kind Regards and keep up the good work! :)
> 
> Tetja Rediske

Re: PROBLEM: IPv6 TCP-Connections resetting

[Hannes Frederic Sowa]

[no comments from me - only forwarded to netdev@vger.kernel.org, Duan
Jiong and Steffen Klassert]

On Fri, Apr 05, 2013 at 05:48:28PM +0200, Tetja Rediske wrote:
> Hi,
> 
> [1.] One line summary of the problem:
> 
> IPv6 TCP-Connections resetting 
> 
> [2.] Full description of the problem/report:
> 
> In the last weeks we updated some of our systems to a 3.8.4 Kernel.
> Since then sometimes we can't connect to services running IPv6,
> Apache and Openssh tested. 
> 
> We got this on different machines with x86 and x86_64 Kernels. On
> x86_64 it is more random, but on x86 i can reproduce it permanently
> (Just opening any TCP Connection 1st time or after some short delay).
> Connecting quick after the reset again will work as expected. It will
> also work, if you keep another connection open.
> 
> Before I got to the Kernel, I just kept an strace on an userspace
> process, but it did not notice the connection attempt. After this I
> monitored the connection with tcpdump, but nothing unusual.
> 
> Then I did a rollback to the older Kernel and it worked as expected.
> 
> I tracked it down with 'git bisect' to commit:
> 
>   093d04d42fa094f6740bb188f0ad0c215ff61e2c
> 
> I also tested latest git state available.
> 
> [3.] Keywords (i.e., modules, networking, kernel):
> 
> networking, IPv6
> 
> [4.] Kernel information
> [4.1.] Kernel version (from /proc/version): 
> 
>   since commit: 093d04d42fa094f6740bb188f0ad0c215ff61e2c
> 
> [4.2.] Kernel .config file:
> [5.] Most recent kernel version which did not have the bug:
> 
> none
> 
> [6.] Output of Oops.. message (if applicable) with symbolic information
>      resolved (see Documentation/oops-tracing.txt)
> [7.] A small shell script or example program which triggers the
>      problem (if possible)
> [8.] Environment
> [8.1.] Software (add the output of the ver_linux script here)
> 
> Different systems, mostly reproduced on this one:
> 
> Linux dns03.tetja.de 3.9.0-rc5+ #10 SMP Fri Apr 5 16:55:54 CEST 2013
> i686 AMD Athlon(tm) 64 X2 Dual Core Processor 5600+ AuthenticAMD
> GNU/Linux
> 
> Gnu C                  4.4.5
> Gnu make               3.82
> binutils               2.22
> util-linux             2.22.2
> mount                  debug
> module-init-tools      12
> e2fsprogs              1.42
> jfsutils               1.1.15
> reiserfsprogs          3.6.21
> xfsprogs               3.1.10
> Linux C Library        2.15
> Dynamic linker (ldd)   2.15
> Procps                 3.3.4
> Net-tools              1.60_p20120127084908
> Kbd                    1.15.3wip
> Sh-utils               8.20
> Modules Loaded
> 
> Connections looking like this on booth sites:
> 
> 11:52:04.634315 IP6 2a00:1828:0:1::10.51808 >
> 2a00:1828:1000:1102::2.80: Flags [S], seq 103067898, win 5760, options
> [mss 1440,sackOK,TS val 232579708 ecr 0,nop,wscale 7], length 0
> 
> 11:52:04.634354 IP6 2a00:1828:1000:1102::2.80 >
> 2a00:1828:0:1::10.51808: Flags [S.], seq 3352491415, ack 103067899, win
> 14280, options [mss 1440,sackOK,TS val 174797959 ecr
> 232579708,nop,wscale 7], length 0 
> 
> 11:52:04.634656 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 136
> 
> 11:52:04.634715 IP6 2a00:1828:0:1::10.51808 >
> 2a00:1828:1000:1102::2.80: Flags [.], ack 1, win 45, options
> [nop,nop,TS val 232579708 ecr 174797959], length 0
> 
> 11:52:04.634726 IP6 2a00:1828:1000:1102::2.80 >
> 2a00:1828:0:1::10.51808: Flags [R], seq 3352491416, win 0, length 0
> 
> 11:52:04.635027 IP6 2a00:1828:0:1::10.51808 >
> 2a00:1828:1000:1102::2.80: Flags [P.], seq 1:359, ack 1, win 45,
> options [nop,nop,TS val 232579708 ecr 174797959], length 358
> 
> 11:52:04.635037 IP6 2a00:1828:1000:1102::2.80 >
> 2a00:1828:0:1::10.51808: Flags [R], seq 3352491416, win 0, length 0
> 
> 11:52:04.635071 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 112
> 
> 11:52:04.635246 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 112
> 
> Kind Regards and keep up the good work! :)
> 
> Tetja Rediske

Re: PROBLEM: IPv6 TCP-Connections resetting

[Hannes Frederic Sowa]

On Sat, Apr 06, 2013 at 06:35:34AM +0200, Hannes Frederic Sowa wrote:
> [no comments from me - only forwarded to netdev@vger.kernel.org, Duan
> Jiong and Steffen Klassert]

Sorry, I had this in my todo list and haven't seen it got here already. :/

Re: PROBLEM: IPv6 TCP-Connections resetting

[Christoph Paasch]

Hello,

On Saturday 06 April 2013 06:35:34 Hannes Frederic Sowa wrote:
> > [1.] One line summary of the problem:
> > 
> > IPv6 TCP-Connections resetting
> > 
> > [2.] Full description of the problem/report:
> > 
> > In the last weeks we updated some of our systems to a 3.8.4 Kernel.
> > Since then sometimes we can't connect to services running IPv6,
> > Apache and Openssh tested.
> > 
> > We got this on different machines with x86 and x86_64 Kernels. On
> > x86_64 it is more random, but on x86 i can reproduce it permanently
> > (Just opening any TCP Connection 1st time or after some short delay).
> > Connecting quick after the reset again will work as expected. It will
> > also work, if you keep another connection open.
> > 
> > Before I got to the Kernel, I just kept an strace on an userspace
> > process, but it did not notice the connection attempt. After this I
> > monitored the connection with tcpdump, but nothing unusual.
> > 
> > Then I did a rollback to the older Kernel and it worked as expected.
> > 
> > I tracked it down with 'git bisect' to commit:
> >   093d04d42fa094f6740bb188f0ad0c215ff61e2c
> > 
> > I also tested latest git state available.
> > 
> > [3.] Keywords (i.e., modules, networking, kernel):
> > 
> > networking, IPv6
> > 
> > [4.] Kernel information
> > 
> > [4.1.] Kernel version (from /proc/version):
> >   since commit: 093d04d42fa094f6740bb188f0ad0c215ff61e2c
> > 
> > [4.2.] Kernel .config file:
> > [5.] Most recent kernel version which did not have the bug:
> > 
> > none
> > 
> > [6.] Output of Oops.. message (if applicable) with symbolic information
> > 
> >      resolved (see Documentation/oops-tracing.txt)
> > 
> > [7.] A small shell script or example program which triggers the
> > 
> >      problem (if possible)
> > 
> > [8.] Environment
> > [8.1.] Software (add the output of the ver_linux script here)
> > 
> > Different systems, mostly reproduced on this one:
> > 
> > Linux dns03.tetja.de 3.9.0-rc5+ #10 SMP Fri Apr 5 16:55:54 CEST 2013
> > i686 AMD Athlon(tm) 64 X2 Dual Core Processor 5600+ AuthenticAMD
> > GNU/Linux
> > 
> > Gnu C                  4.4.5
> > Gnu make               3.82
> > binutils               2.22
> > util-linux             2.22.2
> > mount                  debug
> > module-init-tools      12
> > e2fsprogs              1.42
> > jfsutils               1.1.15
> > reiserfsprogs          3.6.21
> > xfsprogs               3.1.10
> > Linux C Library        2.15
> > Dynamic linker (ldd)   2.15
> > Procps                 3.3.4
> > Net-tools              1.60_p20120127084908
> > Kbd                    1.15.3wip
> > Sh-utils               8.20
> > Modules Loaded
> > 
> > Connections looking like this on booth sites:
> > 
> > 11:52:04.634315 IP6 2a00:1828:0:1::10.51808 >
> > 2a00:1828:1000:1102::2.80: Flags [S], seq 103067898, win 5760, options
> > [mss 1440,sackOK,TS val 232579708 ecr 0,nop,wscale 7], length 0
> > 
> > 11:52:04.634354 IP6 2a00:1828:1000:1102::2.80 >
> > 2a00:1828:0:1::10.51808: Flags [S.], seq 3352491415, ack 103067899, win
> > 14280, options [mss 1440,sackOK,TS val 174797959 ecr
> > 232579708,nop,wscale 7], length 0
> > 
> > 11:52:04.634656 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> > ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 136
> > 
> > 11:52:04.634715 IP6 2a00:1828:0:1::10.51808 >
> > 2a00:1828:1000:1102::2.80: Flags [.], ack 1, win 45, options
> > [nop,nop,TS val 232579708 ecr 174797959], length 0
> > 
> > 11:52:04.634726 IP6 2a00:1828:1000:1102::2.80 >
> > 2a00:1828:0:1::10.51808: Flags [R], seq 3352491416, win 0, length 0
> > 
> > 11:52:04.635027 IP6 2a00:1828:0:1::10.51808 >
> > 2a00:1828:1000:1102::2.80: Flags [P.], seq 1:359, ack 1, win 45,
> > options [nop,nop,TS val 232579708 ecr 174797959], length 358
> > 
> > 11:52:04.635037 IP6 2a00:1828:1000:1102::2.80 >
> > 2a00:1828:0:1::10.51808: Flags [R], seq 3352491416, win 0, length 0
> > 
> > 11:52:04.635071 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> > ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 112
> > 
> > 11:52:04.635246 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> > ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 112

May it simply be a missing "goto out" in tcp_v6_err (see below patch) ?

Cheers,
Christoph

--------

From: Christoph Paasch <christoph.paasch@uclouvain.be>
Date: Sat, 6 Apr 2013 10:21:01 +0200
Subject: [PATCH] ipv6/tcp: Stop processing ICMPv6 redirect messages

Upon reception of an ICMPv6 Redirect message, we should not continue
inside tcp_v6_err. Otherwise, an error will be reported or request-socks
will be closed.

Adds also some parantheses to respect codingstyle guidelines.

Reported-by: Tetja Rediske <tetja@tetja.de>
Signed-off-by: Christoph Paasch <christoph.paasch@uclouvain.be>
---
 net/ipv6/tcp_ipv6.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 1033d2b..24434c5 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -386,6 +386,7 @@ static void tcp_v6_err(struct sk_buff *skb, struct 
inet6_skb_parm *opt,
 
 		if (dst)
 			dst->ops->redirect(dst, sk, skb);
+		goto out;
 	}
 
 	if (type == ICMPV6_PKT_TOOBIG) {
@@ -441,16 +442,18 @@ static void tcp_v6_err(struct sk_buff *skb, struct 
inet6_skb_parm *opt,
 			sk->sk_error_report(sk);		/* Wake people up to see the error 
(see connect in sock.c) */
 
 			tcp_done(sk);
-		} else
+		} else {
 			sk->sk_err_soft = err;
+		}
 		goto out;
 	}
 
 	if (!sock_owned_by_user(sk) && np->recverr) {
 		sk->sk_err = err;
 		sk->sk_error_report(sk);
-	} else
+	} else {
 		sk->sk_err_soft = err;
+	}
 
 out:
 	bh_unlock_sock(sk);
-- 
1.8.1.227.g44fe835




-- 
IP Networking Lab --- http://inl.info.ucl.ac.be
MultiPath TCP in the Linux Kernel --- http://multipath-tcp.org
UCLouvain
--

Re: PROBLEM: IPv6 TCP-Connections resetting

[Christoph Paasch]

On Saturday 06 April 2013 11:14:44 Christoph Paasch wrote:
> From: Christoph Paasch <christoph.paasch@uclouvain.be>

Argh...
Sorry, my e-mail client messed up the patch. Here the resend:


------

From: Christoph Paasch <christoph.paasch@uclouvain.be>
Date: Sat, 6 Apr 2013 10:21:01 +0200
Subject: [PATCH] ipv6/tcp: Stop processing ICMPv6 redirect messages

Upon reception of an ICMPv6 Redirect message, we should not continue
inside tcp_v6_err. Otherwise, an error will be reported or request-socks
will be closed.

Adds also some parantheses to respect codingstyle guidelines.

Reported-by: Tetja Rediske <tetja@tetja.de>
Signed-off-by: Christoph Paasch <christoph.paasch@uclouvain.be>
---
 net/ipv6/tcp_ipv6.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 1033d2b..24434c5 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -386,6 +386,7 @@ static void tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,
 
 		if (dst)
 			dst->ops->redirect(dst, sk, skb);
+		goto out;
 	}
 
 	if (type == ICMPV6_PKT_TOOBIG) {
@@ -441,16 +442,18 @@ static void tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,
 			sk->sk_error_report(sk);		/* Wake people up to see the error (see connect in sock.c) */
 
 			tcp_done(sk);
-		} else
+		} else {
 			sk->sk_err_soft = err;
+		}
 		goto out;
 	}
 
 	if (!sock_owned_by_user(sk) && np->recverr) {
 		sk->sk_err = err;
 		sk->sk_error_report(sk);
-	} else
+	} else {
 		sk->sk_err_soft = err;
+	}
 
 out:
 	bh_unlock_sock(sk);
-- 
1.8.1.227.g44fe835



-- 
IP Networking Lab --- http://inl.info.ucl.ac.be
MultiPath TCP in the Linux Kernel --- http://multipath-tcp.org
UCLouvain
--

Re: PROBLEM: IPv6 TCP-Connections resetting

[Sergei Shtylyov]

Hello.

On 06-04-2013 13:18, Christoph Paasch wrote:

> From: Christoph Paasch <christoph.paasch@uclouvain.be>
> Date: Sat, 6 Apr 2013 10:21:01 +0200
> Subject: [PATCH] ipv6/tcp: Stop processing ICMPv6 redirect messages

> Upon reception of an ICMPv6 Redirect message, we should not continue
> inside tcp_v6_err. Otherwise, an error will be reported or request-socks
> will be closed.

> Adds also some parantheses to respect codingstyle guidelines.

    When you say "älso", it's often a good indicator there should be another 
patch -- like in this case.

> Reported-by: Tetja Rediske <tetja@tetja.de>
> Signed-off-by: Christoph Paasch <christoph.paasch@uclouvain.be>
> ---
>   net/ipv6/tcp_ipv6.c | 7 +++++--
>   1 file changed, 5 insertions(+), 2 deletions(-)

> diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
> index 1033d2b..24434c5 100644
> --- a/net/ipv6/tcp_ipv6.c
> +++ b/net/ipv6/tcp_ipv6.c
> @@ -386,6 +386,7 @@ static void tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,
>
>   		if (dst)
>   			dst->ops->redirect(dst, sk, skb);
> +		goto out;
>   	}
>
>   	if (type == ICMPV6_PKT_TOOBIG) {
> @@ -441,16 +442,18 @@ static void tcp_v6_err(struct sk_buff *skb, struct inet6_skb_parm *opt,
>   			sk->sk_error_report(sk);		/* Wake people up to see the error (see connect in sock.c) */
>
>   			tcp_done(sk);
> -		} else
> +		} else {
>   			sk->sk_err_soft = err;
> +		}
>   		goto out;
>   	}
>
>   	if (!sock_owned_by_user(sk) && np->recverr) {
>   		sk->sk_err = err;
>   		sk->sk_error_report(sk);
> -	} else
> +	} else {
>   		sk->sk_err_soft = err;
> +	}
>
>   out:
>   	bh_unlock_sock(sk);

WBR, Sergei

Re: PROBLEM: IPv6 TCP-Connections resetting

[Eric Dumazet]

On Sat, 2013-04-06 at 11:14 +0200, Christoph Paasch wrote:
> Hello,
> 
> On Saturday 06 April 2013 06:35:34 Hannes Frederic Sowa wrote:
> > > [1.] One line summary of the problem:
> > > 
> > > IPv6 TCP-Connections resetting
> > > 
> > > [2.] Full description of the problem/report:
> > > 
> > > In the last weeks we updated some of our systems to a 3.8.4 Kernel.
> > > Since then sometimes we can't connect to services running IPv6,
> > > Apache and Openssh tested.
> > > 
> > > We got this on different machines with x86 and x86_64 Kernels. On
> > > x86_64 it is more random, but on x86 i can reproduce it permanently
> > > (Just opening any TCP Connection 1st time or after some short delay).
> > > Connecting quick after the reset again will work as expected. It will
> > > also work, if you keep another connection open.
> > > 
> > > Before I got to the Kernel, I just kept an strace on an userspace
> > > process, but it did not notice the connection attempt. After this I
> > > monitored the connection with tcpdump, but nothing unusual.
> > > 
> > > Then I did a rollback to the older Kernel and it worked as expected.
> > > 
> > > I tracked it down with 'git bisect' to commit:
> > >   093d04d42fa094f6740bb188f0ad0c215ff61e2c
> > > 
> > > I also tested latest git state available.
> > > 
> > > [3.] Keywords (i.e., modules, networking, kernel):
> > > 
> > > networking, IPv6
> > > 
> > > [4.] Kernel information
> > > 
> > > [4.1.] Kernel version (from /proc/version):
> > >   since commit: 093d04d42fa094f6740bb188f0ad0c215ff61e2c
> > > 
> > > [4.2.] Kernel .config file:
> > > [5.] Most recent kernel version which did not have the bug:
> > > 
> > > none
> > > 
> > > [6.] Output of Oops.. message (if applicable) with symbolic information
> > > 
> > >      resolved (see Documentation/oops-tracing.txt)
> > > 
> > > [7.] A small shell script or example program which triggers the
> > > 
> > >      problem (if possible)
> > > 
> > > [8.] Environment
> > > [8.1.] Software (add the output of the ver_linux script here)
> > > 
> > > Different systems, mostly reproduced on this one:
> > > 
> > > Linux dns03.tetja.de 3.9.0-rc5+ #10 SMP Fri Apr 5 16:55:54 CEST 2013
> > > i686 AMD Athlon(tm) 64 X2 Dual Core Processor 5600+ AuthenticAMD
> > > GNU/Linux
> > > 
> > > Gnu C                  4.4.5
> > > Gnu make               3.82
> > > binutils               2.22
> > > util-linux             2.22.2
> > > mount                  debug
> > > module-init-tools      12
> > > e2fsprogs              1.42
> > > jfsutils               1.1.15
> > > reiserfsprogs          3.6.21
> > > xfsprogs               3.1.10
> > > Linux C Library        2.15
> > > Dynamic linker (ldd)   2.15
> > > Procps                 3.3.4
> > > Net-tools              1.60_p20120127084908
> > > Kbd                    1.15.3wip
> > > Sh-utils               8.20
> > > Modules Loaded
> > > 
> > > Connections looking like this on booth sites:
> > > 
> > > 11:52:04.634315 IP6 2a00:1828:0:1::10.51808 >
> > > 2a00:1828:1000:1102::2.80: Flags [S], seq 103067898, win 5760, options
> > > [mss 1440,sackOK,TS val 232579708 ecr 0,nop,wscale 7], length 0
> > > 
> > > 11:52:04.634354 IP6 2a00:1828:1000:1102::2.80 >
> > > 2a00:1828:0:1::10.51808: Flags [S.], seq 3352491415, ack 103067899, win
> > > 14280, options [mss 1440,sackOK,TS val 174797959 ecr
> > > 232579708,nop,wscale 7], length 0
> > > 
> > > 11:52:04.634656 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> > > ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 136
> > > 
> > > 11:52:04.634715 IP6 2a00:1828:0:1::10.51808 >
> > > 2a00:1828:1000:1102::2.80: Flags [.], ack 1, win 45, options
> > > [nop,nop,TS val 232579708 ecr 174797959], length 0
> > > 
> > > 11:52:04.634726 IP6 2a00:1828:1000:1102::2.80 >
> > > 2a00:1828:0:1::10.51808: Flags [R], seq 3352491416, win 0, length 0
> > > 
> > > 11:52:04.635027 IP6 2a00:1828:0:1::10.51808 >
> > > 2a00:1828:1000:1102::2.80: Flags [P.], seq 1:359, ack 1, win 45,
> > > options [nop,nop,TS val 232579708 ecr 174797959], length 358
> > > 
> > > 11:52:04.635037 IP6 2a00:1828:1000:1102::2.80 >
> > > 2a00:1828:0:1::10.51808: Flags [R], seq 3352491416, win 0, length 0
> > > 
> > > 11:52:04.635071 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> > > ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 112
> > > 
> > > 11:52:04.635246 IP6 fe80::92e2:baff:fe00:c120 > 2a00:1828:1000:1102::2:
> > > ICMP6, redirect, 2a00:1828:0:1::10 to 2a00:1828:0:1::10, length 112
> 
> May it simply be a missing "goto out" in tcp_v6_err (see below patch) ?
> 
> Cheers,
> Christoph
> 
> --------
> 
> From: Christoph Paasch <christoph.paasch@uclouvain.be>
> Date: Sat, 6 Apr 2013 10:21:01 +0200
> Subject: [PATCH] ipv6/tcp: Stop processing ICMPv6 redirect messages
> 
> Upon reception of an ICMPv6 Redirect message, we should not continue
> inside tcp_v6_err. Otherwise, an error will be reported or request-socks
> will be closed.
> 
> Adds also some parantheses to respect codingstyle guidelines.
> 
> Reported-by: Tetja Rediske <tetja@tetja.de>
> Signed-off-by: Christoph Paasch <christoph.paasch@uclouvain.be>
> ---
>  net/ipv6/tcp_ipv6.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
> index 1033d2b..24434c5 100644
> --- a/net/ipv6/tcp_ipv6.c
> +++ b/net/ipv6/tcp_ipv6.c
> @@ -386,6 +386,7 @@ static void tcp_v6_err(struct sk_buff *skb, struct 
> inet6_skb_parm *opt,
>  
>  		if (dst)
>  			dst->ops->redirect(dst, sk, skb);
> +		goto out;
>  	}
>  

OK, it seems bug was added in commit
ec18d9a2691d69cd14b48f9b919fddcef28b7f5c
(ipv6: Add redirect support to all protocol icmp error handlers.)

Not sure why Tetja Rediske  bisected to
093d04d42fa094f6740bb188f0ad0c215ff61e2c

Could you send a patch with this single line change (no cleanup), and
a more detailed changelog, once the bug origin is clearly identified ?

Thanks !

Re: PROBLEM: IPv6 TCP-Connections resetting

[Tetja Rediske]

Hi,

> OK, it seems bug was added in commit
> ec18d9a2691d69cd14b48f9b919fddcef28b7f5c
> (ipv6: Add redirect support to all protocol icmp error handlers.)

> Not sure why Tetja Rediske  bisected t
> 093d04d42fa094f6740bb188f0ad0c215ff61e2c

there could be an simple explenation, because i couldn't realy provoke 
it with direct connected machines I did it on one of our least important 
DNS-Server, I did a "bad" everytime the connection fault and did a 
"good" after a some succesful connects (with waiting Time in between). 
If the Bug is based on an ICMPv6 redirect in the meantime a redirect 
could be sent out without me noticing. I did not keep the tcpdump 
running while bisect(?ing?).

I will happily try the patch, when I get to work Monday, weekend is 
mostly for family. ;)

Tetja

Re: PROBLEM: IPv6 TCP-Connections resetting

[Christoph Paasch]

On Saturday 06 April 2013 10:54:39 Eric Dumazet wrote:
> OK, it seems bug was added in commit
> ec18d9a2691d69cd14b48f9b919fddcef28b7f5c
> (ipv6: Add redirect support to all protocol icmp error handlers.)
> 
> Not sure why Tetja Rediske  bisected to
> 093d04d42fa094f6740bb188f0ad0c215ff61e2c

I made a setup to trigger the ICMPv6 Redirect.
Yes, the bug was added by ec18d9a26 (ipv6: Add redirect support to all 
protocol icmp error handlers.), but prior to 093d04d4 (ipv6: Change skb->data 
before using icmpv6_notify() to propagate redirect) the stack did not enter 
tcp_v6_err upon a redirect message.

This, because inside icmpv6_notify, skb->data did not point to the inner IP 
header. So, when icmpv6_notify looks up the protocol-handler, it will not 
match on tcp_v6_err.

> Could you send a patch with this single line change (no cleanup), and
> a more detailed changelog, once the bug origin is clearly identified ?

Yes, will resend.


Cheers,
Christoph

-- 
IP Networking Lab --- http://inl.info.ucl.ac.be
MultiPath TCP in the Linux Kernel --- http://multipath-tcp.org
UCLouvain
--

Re: PROBLEM: IPv6 TCP-Connections resetting

[Tetja Rediske]

Hi,

> I made a setup to trigger the ICMPv6 Redirect.
> Yes, the bug was added by ec18d9a26 (ipv6: Add redirect support to
> all protocol icmp error handlers.), but prior to 093d04d4 (ipv6:
> Change skb->data before using icmpv6_notify() to propagate redirect)
> the stack did not enter tcp_v6_err upon a redirect message.
> 
> This, because inside icmpv6_notify, skb->data did not point to the
> inner IP header. So, when icmpv6_notify looks up the
> protocol-handler, it will not match on tcp_v6_err.

with the goto line I can't see this behaviour anymore.

Thanks!

Tetja