slow tcp connect when using IPsec

slow tcp connect when using IPsec

[Steffen Klassert]

I'm fighting with a strange behaviour since a some days.
When I try to send tcp data over an IPsec tunnel, the tcp connect hangs
for about 20 seconds before it finally sends out the SYN packet.
This happens just on tcp with IPsec. When I bind the connection to
a specific local port, everything works fine. After some time of 
unsuccessful debugging, I bisected this issue down to

commit 5e2b61f78411be25f0b84f97d5b5d312f184dfd1
Author: David S. Miller <davem@davemloft.net>
Date:   Fri Mar 4 21:47:09 2011 -0800

    ipv4: Remove flowi from struct rtable.

Some time and a lot of trace_printks later I found that we set up
the flow informations without source _and_ destination address in
ip_route_newports(). That is because we take the address informations
from the the rt_key_src and rt_key_dst fields of the rtable here
and they appear to be empty. If I restore the behaviour before the bisected
commit by taking the address informations from rt_src and rt_dst the issue
is gone. So now I know why it did not behave as expected, but unfortunately
I still don't know why it magically started to work after 20 seconds...

I'll send the patch that fixed the issue in replay to this mail.

Steffen

[PATCH] route: Take the right src and dst addresses in ip_route_newports

[Steffen Klassert]

When we set up the flow informations in ip_route_newports(),  we take the
address informations from the the rt_key_src and rt_key_dst fields of the
rtable. They appear to be empty. So take the address informations from
rt_src and rt_dst instead. This issue was introduced by
commit 5e2b61f78411be25f0b84f97d5b5d312f184dfd1

Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
---
 include/net/route.h |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/include/net/route.h b/include/net/route.h
index dc10244..f88429c 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -270,8 +270,8 @@ static inline struct rtable *ip_route_newports(struct rtable *rt,
 		struct flowi4 fl4 = {
 			.flowi4_oif = rt->rt_oif,
 			.flowi4_mark = rt->rt_mark,
-			.daddr = rt->rt_key_dst,
-			.saddr = rt->rt_key_src,
+			.daddr = rt->rt_dst,
+			.saddr = rt->rt_src,
 			.flowi4_tos = rt->rt_tos,
 			.flowi4_proto = protocol,
 			.fl4_sport = sport,
-- 
1.7.0.4

Re: slow tcp connect when using IPsec

[David Miller]

From: Steffen Klassert <steffen.klassert@secunet.com>
Date: Fri, 25 Mar 2011 07:41:16 +0100

> commit 5e2b61f78411be25f0b84f97d5b5d312f184dfd1
> Author: David S. Miller <davem@davemloft.net>
> Date:   Fri Mar 4 21:47:09 2011 -0800
> 
>     ipv4: Remove flowi from struct rtable.
> 
> Some time and a lot of trace_printks later I found that we set up
> the flow informations without source _and_ destination address in
> ip_route_newports(). That is because we take the address informations
> from the the rt_key_src and rt_key_dst fields of the rtable here
> and they appear to be empty. If I restore the behaviour before the bisected
> commit by taking the address informations from rt_src and rt_dst the issue
> is gone.

Indeed, it is wrong to use the key values, since they can be
wildcards.  Thanks for tracking this down.

> So now I know why it did not behave as expected, but unfortunately I
> still don't know why it magically started to work after 20
> seconds...

After some time, TCP will mark routing path as having trouble, then it
will relookup the route.  At this point source and dest will no longer
be wildcarded in the socket, and thus neither will be the resulting
route keys in the relooked-up route.

Look for the dst_negative_advice() paths to see where this happens.

Re: [PATCH] route: Take the right src and dst addresses in ip_route_newports

[David Miller]

From: Steffen Klassert <steffen.klassert@secunet.com>
Date: Fri, 25 Mar 2011 07:42:03 +0100

> When we set up the flow informations in ip_route_newports(),  we take the
> address informations from the the rt_key_src and rt_key_dst fields of the
> rtable. They appear to be empty. So take the address informations from
> rt_src and rt_dst instead. This issue was introduced by
> commit 5e2b61f78411be25f0b84f97d5b5d312f184dfd1
> 
> Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>

Applied, thank you.

Re: slow tcp connect when using IPsec

[Steffen Klassert]

On Fri, Mar 25, 2011 at 01:27:49AM -0700, David Miller wrote:
> 
> > So now I know why it did not behave as expected, but unfortunately I
> > still don't know why it magically started to work after 20
> > seconds...
> 
> After some time, TCP will mark routing path as having trouble, then it
> will relookup the route.  At this point source and dest will no longer
> be wildcarded in the socket, and thus neither will be the resulting
> route keys in the relooked-up route.
> 
> Look for the dst_negative_advice() paths to see where this happens.

Ok, I see. Thanks for the explanation.