when having to acquire an SA, ipsec drops the packet

when having to acquire an SA, ipsec drops the packet

[Joy Latten]

IPsec returns EAGAIN when it needs to acquire an SA.
There have been a thread or two about this...
Has there been any info or progress in how best to fix this?

James Morris presented some work/ideas,
http://vger.kernel.org/jmorris_ipsec_sa_resolution_netconf2006.pdf

When using labeled xfrms (xfrms that contain a security context), there
is potential for a greater amount of SAs to be created than when using
regular xfrms. An SA may be created every time a different security
context is encountered in a particular traffic stream. This could be
many if each networking app has its own security context, making current
behavior problematic.

Bugreport 225328 has been opened in the Redhat Bugzilla to address
when having to acquire an SA, ipsec drops the packet.

Regards,
Joy

Re: when having to acquire an SA, ipsec drops the packet

[James Morris]

On Thu, 1 Feb 2007, Joy Latten wrote:

> IPsec returns EAGAIN when it needs to acquire an SA.
> There have been a thread or two about this...
> Has there been any info or progress in how best to fix this?
> 
> James Morris presented some work/ideas,
> http://vger.kernel.org/jmorris_ipsec_sa_resolution_netconf2006.pdf

The status of this is that I started refactoring some core IPv6 code (to 
bring it in line with the IPv4 code, e.g. create an ip6_route_connect() to 
match ip_route_connect(), and manage the packet queuing from there) and 
ran into some IPv6 bugs, and fixed those, but then ran out of time on the 
IPsec stuff.  AFAIK, no other progress has been made.

Generally, the problem is only seen when using the BSD userland, which 
does not proactively maintain SAs.  Openswan does, and general IPsec users 
running it never see the problem, so it's not clear how high the priority 
is for fixing this really is in the general case.  It's quite a lot of 
surgery on core networking to fix a problem which doesn't occur with the 
Linux tools, which seemingly most people use when they're not using 
proprietary and/or userland IPsec stacks.

Non-kernel options include moving to Openswan (which I would hope is 
getting labeled networking support at some stage in any case), or have the 
BSD code proactively maintain SAs.

Also, applications using TCP, and UDP applications with their own session 
management, will resend packets while the SA is being negotiated, which 
I've observed as typically completing before the first resend.  I think 
one of the practical problems with this is that the apps are not expecting 
an EAGAIN and thus decide to crash.

A quick & dirty solution, which is what I think the BSD kernels do, is to 
still drop the packet but just not return an error to the app.  The app 
then just sees a slight delay on the initial connection, as if a DNS 
lookup took a bit longer than usual.

> When using labeled xfrms (xfrms that contain a security context), there
> is potential for a greater amount of SAs to be created than when using
> regular xfrms. An SA may be created every time a different security
> context is encountered in a particular traffic stream. This could be
> many if each networking app has its own security context, making current
> behavior problematic.

Do you have any examples of how many SAs would need to be created on a 
typical system?

It may not be the end of the world if an MLS box has to negotiate a 
whole bunch of SAs when it boots up.


- James
-- 
James Morris
<jmorris@namei.org>

Re: when having to acquire an SA, ipsec drops the packet

[Paul Moore]

On Thursday, February 1 2007 6:44 pm, James Morris wrote:
> On Thu, 1 Feb 2007, Joy Latten wrote:
> > When using labeled xfrms (xfrms that contain a security context), there
> > is potential for a greater amount of SAs to be created than when using
> > regular xfrms. An SA may be created every time a different security
> > context is encountered in a particular traffic stream. This could be
> > many if each networking app has its own security context, making current
> > behavior problematic.
>
> Do you have any examples of how many SAs would need to be created on a
> typical system?
>
> It may not be the end of the world if an MLS box has to negotiate a
> whole bunch of SAs when it boots up.

I agree that having an MLS box spend some extra time when starting the IKE 
daemon is probably not the end of the world.  However, I'm a little concerned 
that it may not be possible to determine a "good" set of SAs to negotiate 
with only the SPD as a reference.

For example, the current SELinux MLS policy has 16 sensitivity levels and 1024 
categories.  Ignoring the TE portion of the SELinux context for the sake of 
clarity you can easily see the large number of unique combinations, with each 
combination requiring a new SA.  Granted, in the majority of these cases only 
a quick mode IKE negotiation would be required, which is much less expensive 
then having to do a full phase-1 negotiation, but due to the large numbers of 
SAs involved I believe it would still be quite a task.  It also should be 
said that this procedure would need to be done for each SPD rule.

I haven't thought about this too much yet, but I suspect proactively creating 
SAs is not going to be a practical solution.

-- 
paul moore
linux security @ hp

Re: when having to acquire an SA, ipsec drops the packet

[David Miller]

From: James Morris <jmorris@namei.org>
Date: Thu, 1 Feb 2007 18:44:48 -0500 (EST)

> A quick & dirty solution, which is what I think the BSD kernels do, is to 
> still drop the packet but just not return an error to the app.  The app 
> then just sees a slight delay on the initial connection, as if a DNS 
> lookup took a bit longer than usual.

I have another idea.

Why don't we just flat-out ignore MSG_DONTWAIT for the socket
visible cases, and handle connect() similarly?

I think this is (just barely) legal, will be simple to implement, and
will leave us with semantics that look like:

1) Sockets never see -EAGAIN due to SA resolution.  They'll just
   pause until the route is resolved, even with O_NONBLOCK or
   MSG_DONTWAIT.

2) Asynchronous contexts such as ICMP replies and firewalling
   will still see the -EAGAIN and simply drop packets.

These sleeps are legal because all of the socket paths involved
have to be able to do lock_socket() (at a minimum) anyways.

Something like this (untested) on the ipv4 side, for example:

diff --git a/include/net/route.h b/include/net/route.h
index 486e37a..a8af632 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -146,7 +146,8 @@ static inline char rt_tos2priority(u8 tos)
 
 static inline int ip_route_connect(struct rtable **rp, __be32 dst,
 				   __be32 src, u32 tos, int oif, u8 protocol,
-				   __be16 sport, __be16 dport, struct sock *sk)
+				   __be16 sport, __be16 dport, struct sock *sk,
+				   int flags)
 {
 	struct flowi fl = { .oif = oif,
 			    .nl_u = { .ip4_u = { .daddr = dst,
@@ -168,7 +169,7 @@ static inline int ip_route_connect(struct rtable **rp, __be32 dst,
 		*rp = NULL;
 	}
 	security_sk_classify_flow(sk, &fl);
-	return ip_route_output_flow(rp, &fl, sk, 0);
+	return ip_route_output_flow(rp, &fl, sk, 1);
 }
 
 static inline int ip_route_newports(struct rtable **rp, u8 protocol,
diff --git a/net/dccp/ipv4.c b/net/dccp/ipv4.c
index 90c74b4..fa2c982 100644
--- a/net/dccp/ipv4.c
+++ b/net/dccp/ipv4.c
@@ -72,7 +72,7 @@ int dccp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
 	tmp = ip_route_connect(&rt, nexthop, inet->saddr,
 			       RT_CONN_FLAGS(sk), sk->sk_bound_dev_if,
 			       IPPROTO_DCCP,
-			       inet->sport, usin->sin_port, sk);
+			       inet->sport, usin->sin_port, sk, 1);
 	if (tmp < 0)
 		return tmp;
 
diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index 8640096..5750a2b 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -1007,7 +1007,7 @@ static int inet_sk_reselect_saddr(struct sock *sk)
 			       RT_CONN_FLAGS(sk),
 			       sk->sk_bound_dev_if,
 			       sk->sk_protocol,
-			       inet->sport, inet->dport, sk);
+			       inet->sport, inet->dport, sk, 0);
 	if (err)
 		return err;
 
diff --git a/net/ipv4/datagram.c b/net/ipv4/datagram.c
index 7b068a8..0072d79 100644
--- a/net/ipv4/datagram.c
+++ b/net/ipv4/datagram.c
@@ -49,7 +49,7 @@ int ip4_datagram_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
 	err = ip_route_connect(&rt, usin->sin_addr.s_addr, saddr,
 			       RT_CONN_FLAGS(sk), oif,
 			       sk->sk_protocol,
-			       inet->sport, usin->sin_port, sk);
+			       inet->sport, usin->sin_port, sk, 1);
 	if (err)
 		return err;
 	if ((rt->rt_flags & RTCF_BROADCAST) && !sock_flag(sk, SOCK_BROADCAST)) {
diff --git a/net/ipv4/raw.c b/net/ipv4/raw.c
index a6c63bb..fed6a1e 100644
--- a/net/ipv4/raw.c
+++ b/net/ipv4/raw.c
@@ -489,7 +489,7 @@ static int raw_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
 		}
 
 		security_sk_classify_flow(sk, &fl);
-		err = ip_route_output_flow(&rt, &fl, sk, !(msg->msg_flags&MSG_DONTWAIT));
+		err = ip_route_output_flow(&rt, &fl, sk, 1);
 	}
 	if (err)
 		goto done;
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index f061ec5..383e4b5 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -191,7 +191,7 @@ int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
 	tmp = ip_route_connect(&rt, nexthop, inet->saddr,
 			       RT_CONN_FLAGS(sk), sk->sk_bound_dev_if,
 			       IPPROTO_TCP,
-			       inet->sport, usin->sin_port, sk);
+			       inet->sport, usin->sin_port, sk, 1);
 	if (tmp < 0)
 		return tmp;
 
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index cfff930..8b54c68 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -629,7 +629,7 @@ int udp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
 					       { .sport = inet->sport,
 						 .dport = dport } } };
 		security_sk_classify_flow(sk, &fl);
-		err = ip_route_output_flow(&rt, &fl, sk, !(msg->msg_flags&MSG_DONTWAIT));
+		err = ip_route_output_flow(&rt, &fl, sk, 1);
 		if (err)
 			goto out;
 

Re: when having to acquire an SA, ipsec drops the packet

[James Morris]

On Sun, 4 Feb 2007, David Miller wrote:

> Something like this (untested) on the ipv4 side, for example:

Looks like it should work.  Will do some testing.


-- 
James Morris
<jmorris@namei.org>

Re: when having to acquire an SA, ipsec drops the packet

[James Morris]

On Mon, 5 Feb 2007, James Morris wrote:

> On Sun, 4 Feb 2007, David Miller wrote:
> 
> > Something like this (untested) on the ipv4 side, for example:
> 
> Looks like it should work.  Will do some testing.

Appears to work well, with a slight delay on the first packet as expected.  
Tested with tcp, udp & icmp.




- James
-- 
James Morris
<jmorris@namei.org>

Re: when having to acquire an SA, ipsec drops the packet

[David Miller]

From: James Morris <jmorris@namei.org>
Date: Mon, 5 Feb 2007 15:34:39 -0500 (EST)

> On Mon, 5 Feb 2007, James Morris wrote:
> 
> > On Sun, 4 Feb 2007, David Miller wrote:
> > 
> > > Something like this (untested) on the ipv4 side, for example:
> > 
> > Looks like it should work.  Will do some testing.
> 
> Appears to work well, with a slight delay on the first packet as expected.  
> Tested with tcp, udp & icmp.

Thanks for testing James.

Last night I put an updated version of my patch into the net-2.6.21
tree, taking care of the ipv6 bits as well.  I attach it below
for completeness.

commit cbc77956da98305f0525c0b40257608f0c02a357
Author: David S. Miller <davem@sunset.davemloft.net>
Date:   Sun Feb 4 21:21:59 2007 -0800

    [IPV4/IPV6]: Always wait for IPSEC SA resolution in socket contexts.
    
    Do this even for non-blocking sockets.  This avoids the silly -EAGAIN
    that applications can see now, even for non-blocking sockets in some
    cases (f.e. connect()).
    
    Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/include/net/route.h b/include/net/route.h
index 486e37a..a8af632 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -146,7 +146,8 @@ static inline char rt_tos2priority(u8 tos)
 
 static inline int ip_route_connect(struct rtable **rp, __be32 dst,
 				   __be32 src, u32 tos, int oif, u8 protocol,
-				   __be16 sport, __be16 dport, struct sock *sk)
+				   __be16 sport, __be16 dport, struct sock *sk,
+				   int flags)
 {
 	struct flowi fl = { .oif = oif,
 			    .nl_u = { .ip4_u = { .daddr = dst,
@@ -168,7 +169,7 @@ static inline int ip_route_connect(struct rtable **rp, __be32 dst,
 		*rp = NULL;
 	}
 	security_sk_classify_flow(sk, &fl);
-	return ip_route_output_flow(rp, &fl, sk, 0);
+	return ip_route_output_flow(rp, &fl, sk, 1);
 }
 
 static inline int ip_route_newports(struct rtable **rp, u8 protocol,
diff --git a/net/dccp/ipv4.c b/net/dccp/ipv4.c
index 90c74b4..fa2c982 100644
--- a/net/dccp/ipv4.c
+++ b/net/dccp/ipv4.c
@@ -72,7 +72,7 @@ int dccp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
 	tmp = ip_route_connect(&rt, nexthop, inet->saddr,
 			       RT_CONN_FLAGS(sk), sk->sk_bound_dev_if,
 			       IPPROTO_DCCP,
-			       inet->sport, usin->sin_port, sk);
+			       inet->sport, usin->sin_port, sk, 1);
 	if (tmp < 0)
 		return tmp;
 
diff --git a/net/dccp/ipv6.c b/net/dccp/ipv6.c
index 6b91a9d..79140b3 100644
--- a/net/dccp/ipv6.c
+++ b/net/dccp/ipv6.c
@@ -1041,7 +1041,7 @@ static int dccp_v6_connect(struct sock *sk, struct sockaddr *uaddr,
 	if (final_p)
 		ipv6_addr_copy(&fl.fl6_dst, final_p);
 
-	err = xfrm_lookup(&dst, &fl, sk, 0);
+	err = xfrm_lookup(&dst, &fl, sk, 1);
 	if (err < 0)
 		goto failure;
 
diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index 8640096..5750a2b 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -1007,7 +1007,7 @@ static int inet_sk_reselect_saddr(struct sock *sk)
 			       RT_CONN_FLAGS(sk),
 			       sk->sk_bound_dev_if,
 			       sk->sk_protocol,
-			       inet->sport, inet->dport, sk);
+			       inet->sport, inet->dport, sk, 0);
 	if (err)
 		return err;
 
diff --git a/net/ipv4/datagram.c b/net/ipv4/datagram.c
index 7b068a8..0072d79 100644
--- a/net/ipv4/datagram.c
+++ b/net/ipv4/datagram.c
@@ -49,7 +49,7 @@ int ip4_datagram_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
 	err = ip_route_connect(&rt, usin->sin_addr.s_addr, saddr,
 			       RT_CONN_FLAGS(sk), oif,
 			       sk->sk_protocol,
-			       inet->sport, usin->sin_port, sk);
+			       inet->sport, usin->sin_port, sk, 1);
 	if (err)
 		return err;
 	if ((rt->rt_flags & RTCF_BROADCAST) && !sock_flag(sk, SOCK_BROADCAST)) {
diff --git a/net/ipv4/raw.c b/net/ipv4/raw.c
index a6c63bb..fed6a1e 100644
--- a/net/ipv4/raw.c
+++ b/net/ipv4/raw.c
@@ -489,7 +489,7 @@ static int raw_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
 		}
 
 		security_sk_classify_flow(sk, &fl);
-		err = ip_route_output_flow(&rt, &fl, sk, !(msg->msg_flags&MSG_DONTWAIT));
+		err = ip_route_output_flow(&rt, &fl, sk, 1);
 	}
 	if (err)
 		goto done;
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index f061ec5..383e4b5 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -191,7 +191,7 @@ int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
 	tmp = ip_route_connect(&rt, nexthop, inet->saddr,
 			       RT_CONN_FLAGS(sk), sk->sk_bound_dev_if,
 			       IPPROTO_TCP,
-			       inet->sport, usin->sin_port, sk);
+			       inet->sport, usin->sin_port, sk, 1);
 	if (tmp < 0)
 		return tmp;
 
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index cfff930..8b54c68 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -629,7 +629,7 @@ int udp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
 					       { .sport = inet->sport,
 						 .dport = dport } } };
 		security_sk_classify_flow(sk, &fl);
-		err = ip_route_output_flow(&rt, &fl, sk, !(msg->msg_flags&MSG_DONTWAIT));
+		err = ip_route_output_flow(&rt, &fl, sk, 1);
 		if (err)
 			goto out;
 
diff --git a/net/ipv6/datagram.c b/net/ipv6/datagram.c
index 5c94fea..ecde301 100644
--- a/net/ipv6/datagram.c
+++ b/net/ipv6/datagram.c
@@ -178,7 +178,7 @@ ipv4_connected:
 	if (final_p)
 		ipv6_addr_copy(&fl.fl6_dst, final_p);
 
-	if ((err = xfrm_lookup(&dst, &fl, sk, 0)) < 0)
+	if ((err = xfrm_lookup(&dst, &fl, sk, 1)) < 0)
 		goto out;
 
 	/* source address lookup done in ip6_dst_lookup */
diff --git a/net/ipv6/raw.c b/net/ipv6/raw.c
index 4ae1b19..f2e883c 100644
--- a/net/ipv6/raw.c
+++ b/net/ipv6/raw.c
@@ -815,7 +815,7 @@ static int rawv6_sendmsg(struct kiocb *iocb, struct sock *sk,
 	if (final_p)
 		ipv6_addr_copy(&fl.fl6_dst, final_p);
 
-	if ((err = xfrm_lookup(&dst, &fl, sk, 0)) < 0)
+	if ((err = xfrm_lookup(&dst, &fl, sk, 1)) < 0)
 		goto out;
 
 	if (hlimit < 0) {
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index c25e930..dcb7b00 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -265,7 +265,7 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr,
 	if (final_p)
 		ipv6_addr_copy(&fl.fl6_dst, final_p);
 
-	if ((err = xfrm_lookup(&dst, &fl, sk, 0)) < 0)
+	if ((err = xfrm_lookup(&dst, &fl, sk, 1)) < 0)
 		goto failure;
 
 	if (saddr == NULL) {
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index f52a5c3..15e5195 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -736,7 +736,7 @@ do_udp_sendmsg:
 	if (final_p)
 		ipv6_addr_copy(&fl.fl6_dst, final_p);
 
-	if ((err = xfrm_lookup(&dst, &fl, sk, 0)) < 0)
+	if ((err = xfrm_lookup(&dst, &fl, sk, 1)) < 0)
 		goto out;
 
 	if (hlimit < 0) {

RE: when having to acquire an SA, ipsec drops the packet

[Venkat Yekkirala]

> Something like this (untested) on the ipv4 side, for example:
> 
> diff --git a/include/net/route.h b/include/net/route.h
> index 486e37a..a8af632 100644
> --- a/include/net/route.h
> +++ b/include/net/route.h
> @@ -146,7 +146,8 @@ static inline char rt_tos2priority(u8 tos)
>  
>  static inline int ip_route_connect(struct rtable **rp, __be32 dst,
>  				   __be32 src, u32 tos, int 
> oif, u8 protocol,
> -				   __be16 sport, __be16 dport, 
> struct sock *sk)
> +				   __be16 sport, __be16 dport, 
> struct sock *sk,
> +				   int flags)
>  {
>  	struct flowi fl = { .oif = oif,
>  			    .nl_u = { .ip4_u = { .daddr = dst,
> @@ -168,7 +169,7 @@ static inline int ip_route_connect(struct 
> rtable **rp, __be32 dst,
>  		*rp = NULL;
>  	}
>  	security_sk_classify_flow(sk, &fl);
> -	return ip_route_output_flow(rp, &fl, sk, 0);
> +	return ip_route_output_flow(rp, &fl, sk, 1);

I guess you meant to pass the new flags param to ip_route_output_flow here?

>  }
> 

Re: when having to acquire an SA, ipsec drops the packet

[David Miller]

From: "Venkat Yekkirala" <vyekkirala@trustedcs.com>
Date: Mon, 5 Feb 2007 14:49:17 -0600

> > Something like this (untested) on the ipv4 side, for example:
> > 
> > diff --git a/include/net/route.h b/include/net/route.h
> > index 486e37a..a8af632 100644
> > --- a/include/net/route.h
> > +++ b/include/net/route.h
> > @@ -146,7 +146,8 @@ static inline char rt_tos2priority(u8 tos)
> >  
> >  static inline int ip_route_connect(struct rtable **rp, __be32 dst,
> >  				   __be32 src, u32 tos, int 
> > oif, u8 protocol,
> > -				   __be16 sport, __be16 dport, 
> > struct sock *sk)
> > +				   __be16 sport, __be16 dport, 
> > struct sock *sk,
> > +				   int flags)
> >  {
> >  	struct flowi fl = { .oif = oif,
> >  			    .nl_u = { .ip4_u = { .daddr = dst,
> > @@ -168,7 +169,7 @@ static inline int ip_route_connect(struct 
> > rtable **rp, __be32 dst,
> >  		*rp = NULL;
> >  	}
> >  	security_sk_classify_flow(sk, &fl);
> > -	return ip_route_output_flow(rp, &fl, sk, 0);
> > +	return ip_route_output_flow(rp, &fl, sk, 1);
> 
> I guess you meant to pass the new flags param to ip_route_output_flow here?

Yes I did, thanks for catching that.

commit a6886040ae6b8c9bfc811bd0dbdb47cfa3f2db29
Author: David S. Miller <davem@davemloft.net>
Date:   Mon Feb 5 13:11:42 2007 -0800

    [IPV4]: Fix thinko in ip_route_connect().
    
    The idea was the pass in the new "flags" parameter down
    to ip_route_output_flow().
    
    Noticed by Venkat Tekkirala.
    
    Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/include/net/route.h b/include/net/route.h
index a8af632..1440bdb 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -169,7 +169,7 @@ static inline int ip_route_connect(struct rtable **rp, __be32 dst,
 		*rp = NULL;
 	}
 	security_sk_classify_flow(sk, &fl);
-	return ip_route_output_flow(rp, &fl, sk, 1);
+	return ip_route_output_flow(rp, &fl, sk, flags);
 }
 
 static inline int ip_route_newports(struct rtable **rp, u8 protocol,

Re: when having to acquire an SA, ipsec drops the packet

[Joy Latten]

I can run some tests with this patch and report any results... 

Regards,
Joy

On Sun, 2007-02-04 at 20:53 -0800, David Miller wrote:
> From: James Morris <jmorris@namei.org>
> Date: Thu, 1 Feb 2007 18:44:48 -0500 (EST)
> 
> > A quick & dirty solution, which is what I think the BSD kernels do, is to 
> > still drop the packet but just not return an error to the app.  The app 
> > then just sees a slight delay on the initial connection, as if a DNS 
> > lookup took a bit longer than usual.
> 
> I have another idea.
> 
> Why don't we just flat-out ignore MSG_DONTWAIT for the socket
> visible cases, and handle connect() similarly?
> 
> I think this is (just barely) legal, will be simple to implement, and
> will leave us with semantics that look like:
> 
> 1) Sockets never see -EAGAIN due to SA resolution.  They'll just
>    pause until the route is resolved, even with O_NONBLOCK or
>    MSG_DONTWAIT.
> 
> 2) Asynchronous contexts such as ICMP replies and firewalling
>    will still see the -EAGAIN and simply drop packets.
> 
> These sleeps are legal because all of the socket paths involved
> have to be able to do lock_socket() (at a minimum) anyways.
> 
> Something like this (untested) on the ipv4 side, for example:
> 
> diff --git a/include/net/route.h b/include/net/route.h
> index 486e37a..a8af632 100644
> --- a/include/net/route.h
> +++ b/include/net/route.h
> @@ -146,7 +146,8 @@ static inline char rt_tos2priority(u8 tos)
>  
>  static inline int ip_route_connect(struct rtable **rp, __be32 dst,
>  				   __be32 src, u32 tos, int oif, u8 protocol,
> -				   __be16 sport, __be16 dport, struct sock *sk)
> +				   __be16 sport, __be16 dport, struct sock *sk,
> +				   int flags)
>  {
>  	struct flowi fl = { .oif = oif,
>  			    .nl_u = { .ip4_u = { .daddr = dst,
> @@ -168,7 +169,7 @@ static inline int ip_route_connect(struct rtable **rp, __be32 dst,
>  		*rp = NULL;
>  	}
>  	security_sk_classify_flow(sk, &fl);
> -	return ip_route_output_flow(rp, &fl, sk, 0);
> +	return ip_route_output_flow(rp, &fl, sk, 1);
>  }
>  
>  static inline int ip_route_newports(struct rtable **rp, u8 protocol,
> diff --git a/net/dccp/ipv4.c b/net/dccp/ipv4.c
> index 90c74b4..fa2c982 100644
> --- a/net/dccp/ipv4.c
> +++ b/net/dccp/ipv4.c
> @@ -72,7 +72,7 @@ int dccp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
>  	tmp = ip_route_connect(&rt, nexthop, inet->saddr,
>  			       RT_CONN_FLAGS(sk), sk->sk_bound_dev_if,
>  			       IPPROTO_DCCP,
> -			       inet->sport, usin->sin_port, sk);
> +			       inet->sport, usin->sin_port, sk, 1);
>  	if (tmp < 0)
>  		return tmp;
>  
> diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
> index 8640096..5750a2b 100644
> --- a/net/ipv4/af_inet.c
> +++ b/net/ipv4/af_inet.c
> @@ -1007,7 +1007,7 @@ static int inet_sk_reselect_saddr(struct sock *sk)
>  			       RT_CONN_FLAGS(sk),
>  			       sk->sk_bound_dev_if,
>  			       sk->sk_protocol,
> -			       inet->sport, inet->dport, sk);
> +			       inet->sport, inet->dport, sk, 0);
>  	if (err)
>  		return err;
>  
> diff --git a/net/ipv4/datagram.c b/net/ipv4/datagram.c
> index 7b068a8..0072d79 100644
> --- a/net/ipv4/datagram.c
> +++ b/net/ipv4/datagram.c
> @@ -49,7 +49,7 @@ int ip4_datagram_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
>  	err = ip_route_connect(&rt, usin->sin_addr.s_addr, saddr,
>  			       RT_CONN_FLAGS(sk), oif,
>  			       sk->sk_protocol,
> -			       inet->sport, usin->sin_port, sk);
> +			       inet->sport, usin->sin_port, sk, 1);
>  	if (err)
>  		return err;
>  	if ((rt->rt_flags & RTCF_BROADCAST) && !sock_flag(sk, SOCK_BROADCAST)) {
> diff --git a/net/ipv4/raw.c b/net/ipv4/raw.c
> index a6c63bb..fed6a1e 100644
> --- a/net/ipv4/raw.c
> +++ b/net/ipv4/raw.c
> @@ -489,7 +489,7 @@ static int raw_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
>  		}
>  
>  		security_sk_classify_flow(sk, &fl);
> -		err = ip_route_output_flow(&rt, &fl, sk, !(msg->msg_flags&MSG_DONTWAIT));
> +		err = ip_route_output_flow(&rt, &fl, sk, 1);
>  	}
>  	if (err)
>  		goto done;
> diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
> index f061ec5..383e4b5 100644
> --- a/net/ipv4/tcp_ipv4.c
> +++ b/net/ipv4/tcp_ipv4.c
> @@ -191,7 +191,7 @@ int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
>  	tmp = ip_route_connect(&rt, nexthop, inet->saddr,
>  			       RT_CONN_FLAGS(sk), sk->sk_bound_dev_if,
>  			       IPPROTO_TCP,
> -			       inet->sport, usin->sin_port, sk);
> +			       inet->sport, usin->sin_port, sk, 1);
>  	if (tmp < 0)
>  		return tmp;
>  
> diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
> index cfff930..8b54c68 100644
> --- a/net/ipv4/udp.c
> +++ b/net/ipv4/udp.c
> @@ -629,7 +629,7 @@ int udp_sendmsg(struct kiocb *iocb, struct sock *sk, struct msghdr *msg,
>  					       { .sport = inet->sport,
>  						 .dport = dport } } };
>  		security_sk_classify_flow(sk, &fl);
> -		err = ip_route_output_flow(&rt, &fl, sk, !(msg->msg_flags&MSG_DONTWAIT));
> +		err = ip_route_output_flow(&rt, &fl, sk, 1);
>  		if (err)
>  			goto out;
>  
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: when having to acquire an SA, ipsec drops the packet

[David Miller]

From: Joy Latten <latten@austin.ibm.com>
Date: Mon, 05 Feb 2007 14:53:39 -0600

> I can run some tests with this patch and report any results... 

Please check out the two most recent patches I posted:

1) Updated core patch with ipv6 side added.
2) Fix for thinko noticed by Venkat.

Thanks.

Re: when having to acquire an SA, ipsec drops the packet

[Joy Latten]

On Thu, 2007-02-01 at 18:44 -0500, James Morris wrote:
> On Thu, 1 Feb 2007, Joy Latten wrote:
> 
> > IPsec returns EAGAIN when it needs to acquire an SA.
> > There have been a thread or two about this...
> > Has there been any info or progress in how best to fix this?
> > 
> > James Morris presented some work/ideas,
> > http://vger.kernel.org/jmorris_ipsec_sa_resolution_netconf2006.pdf
> 
> The status of this is that I started refactoring some core IPv6 code (to 
> bring it in line with the IPv4 code, e.g. create an ip6_route_connect() to 
> match ip_route_connect(), and manage the packet queuing from there) and 
> ran into some IPv6 bugs, and fixed those, but then ran out of time on the 
> IPsec stuff.  AFAIK, no other progress has been made.
> 
> Generally, the problem is only seen when using the BSD userland, which 
> does not proactively maintain SAs.  Openswan does, and general IPsec users 
> running it never see the problem, so it's not clear how high the priority 
> is for fixing this really is in the general case.  It's quite a lot of 
> surgery on core networking to fix a problem which doesn't occur with the 
> Linux tools, which seemingly most people use when they're not using 
> proprietary and/or userland IPsec stacks.
> 
> Non-kernel options include moving to Openswan (which I would hope is 
> getting labeled networking support at some stage in any case), or have the 
> BSD code proactively maintain SAs.
> 
> Also, applications using TCP, and UDP applications with their own session 
> management, will resend packets while the SA is being negotiated, which 
> I've observed as typically completing before the first resend.  I think 
> one of the practical problems with this is that the apps are not expecting 
> an EAGAIN and thus decide to crash.
> 
> A quick & dirty solution, which is what I think the BSD kernels do, is to 
> still drop the packet but just not return an error to the app.  The app 
> then just sees a slight delay on the initial connection, as if a DNS 
> lookup took a bit longer than usual.
> 
> > When using labeled xfrms (xfrms that contain a security context), there
> > is potential for a greater amount of SAs to be created than when using
> > regular xfrms. An SA may be created every time a different security
> > context is encountered in a particular traffic stream. This could be
> > many if each networking app has its own security context, making current
> > behavior problematic.
> 
> Do you have any examples of how many SAs would need to be created on a 
> typical system?
> 
No, but I have to admit, just playing around with it gives me a scary
idea. We have s0-s15 levels, each with c0-c1023 categories. Then I have
noticed, each mapped selinux user is a different context... thus just an
ssh session by a sysadm and a regular user (who is mapped to an selinux
user) can cause 8 SAs to be generated, whereas with regular ipsec, we
only need one... :-(
 
Regards,
Joy

Re: when having to acquire an SA, ipsec drops the packet

[Joy Latten]

>From: Joy Latten <latten@austin.ibm.com>
>Date: Mon, 05 Feb 2007 14:53:39 -0600
>
>> I can run some tests with this patch and report any results... 
>
>Please check out the two most recent patches I posted:
>
>1) Updated core patch with ipv6 side added.
>2) Fix for thinko noticed by Venkat.

Just a quick update. I have patched an lspp kernel and 
latest kernel.org kernel. I have been using them alternately
while doing some minor testing and have not found any problems.
Over the next few days, I plan to run a few stress tests for 
labeled ipsec and regular ipsec and will do so with the patched
lspp kernel.  I will also try with kernel.org kernel.
It may take a few days. 

Regards,
Joy

Re: when having to acquire an SA, ipsec drops the packet

[Joy Latten]

>From: Joy Latten <latten@austin.ibm.com>
>Date: Mon, 05 Feb 2007 14:53:39 -0600
>
>> I can run some tests with this patch and report any results... 
>
>Please check out the two most recent patches I posted:
>
>1) Updated core patch with ipv6 side added.
>2) Fix for thinko noticed by Venkat.

I have been testing this a lot in the lspp kernel.
Plan to test also in upstream kernel.
I am seeing a second ACQUIRE occur while establishing the SAs.

My scenario:
My policy states to use both the ESP and AH protocols (may not
make much sense but this was for testing purposes).  I get double 
SAs with only difference being SPI.

Here is what I see happening... 

1. Trigger first ACQUIRE via ping or netperf.

2. xfrm_lookup() calls xfrm_tmpl_resolv() who calls xfrm_state_find().
   First time around, we need to establish SA, so a minimal SA
   get allocated and put in SAD, timer is set for the minimal SA
   to be ACQUIRED and km_query() gets called.
   
3. xfrm_tmpl_resolv() returns -EAGAIN causing add_wait_queue(&km_waitq, &wait)
   and proceeding code to get called waiting for SA to be established.
   As long as the minimal SA with XFRM_STATE_ACQUIRE is in SAD,
   we keep waiting...
   
4. First set of SAs (one for AH and ESP) for IN direction get inserted in SAD.
 
5. Around the time the set of SAs for OUT direction are to be
   inserted into SAD, I see another ACQUIRE happening.
   
   I have not yet figured out where this second ACQUIRE comes from
   and why it happens. As long as the minimal SA or set of valid outgoing
   SAs exist in SAD, an ACQUIRE should not happen.
   The minimal SA does not get removed from the SAD until the set 
   of SAs for OUT get added and the xfrm_state_lock
   released. And the lock pretty much guarantees no one else can step
   through the SAD until after new SAs are being added...
   and if someone gets the lock to step though SAD before OUT SAs
   are added, minimal SA is still there... 

 6. Since this second ACQUIRE was able to happen, result is identical
    sets of SAs for the traffic stream. SPIs are only difference.
 
 7. Noticed something while pasting log info below.
    Perhaps when outgoing AH SA is added, wake_up(&km_waitq) gets called, 
    lock released, and minimal SA deleted (xfrm_state_add()), 
    xfrm_tmpl_resolv() is called and it looks first for the outgoing
    ESP SA. Since it is not there yet and no minimal SA, then km_query()
    results in an ACQUIRE just before the outgoing ESP SA gets added.

    It would explain why I only see it when both ESP and AH are specified...
    that is if I am thinking correctly... 

Regards,
Joy Latten

>From my log file:

Mar  5 19:10:02 racoon: INFO: initiate new phase 2 negotiation: 9.3.192.210[500]<=>9.3.189.55[500]
Mar  5 19:10:03 racoon: INFO: IPsec-SA established: AH/Transport 9.3.189.55[0]->9.3.192.210[0] spi=137942922(0x838d78a)
Mar  5 19:10:03 racoon: INFO: IPsec-SA established: ESP/Transport 9.3.189.55[0]->9.3.192.210[0] spi=244321490(0xe900cd2)
Mar  5 19:10:03 racoon: INFO: IPsec-SA established: AH/Transport 9.3.192.210[0]->9.3.189.55[0] spi=38721750(0x24ed8d6)
Mar  5 19:10:03 racoon: INFO: initiate new phase 2 negotiation: 9.3.192.210[500]<=>9.3.189.55[500]
Mar  5 19:10:03 racoon: INFO: IPsec-SA established: ESP/Transport 9.3.192.210[0]->9.3.189.55[0] spi=265079770(0xfcccbda)
Mar  5 19:10:05 racoon: INFO: IPsec-SA established: AH/Transport 9.3.189.55[0]->9.3.192.210[0] spi=108627618(0x67986a2)
Mar  5 19:10:05 racoon: INFO: IPsec-SA established: ESP/Transport 9.3.189.55[0]->9.3.192.210[0] spi=182973856(0xae7f5a0)
Mar  5 19:10:05 racoon: INFO: IPsec-SA established: AH/Transport 9.3.192.210[0]->9.3.189.55[0] spi=58486297(0x37c6e19)
Mar  5 19:10:05 racoon: INFO: IPsec-SA established: ESP/Transport 9.3.192.210[0]->9.3.189.55[0] spi=268295215(0xffddc2f)

Re: when having to acquire an SA, ipsec drops the packet

[James Morris]

On Mon, 5 Mar 2007, Joy Latten wrote:

> 5. Around the time the set of SAs for OUT direction are to be
>    inserted into SAD, I see another ACQUIRE happening.
>    
>    I have not yet figured out where this second ACQUIRE comes from
>    and why it happens. As long as the minimal SA or set of valid outgoing
>    SAs exist in SAD, an ACQUIRE should not happen.

I saw something similar to this some time ago when testing various 
failure modes, and discused it with Herbert.

IIRC, there's a larval SA which is not torn down properly by Racoon once 
the full SA is established, and the larval SA keeps resending until it 
times out.



- James
-- 
James Morris
<jmorris@namei.org>

Re: when having to acquire an SA, ipsec drops the packet

[Joy Latten]

On Mon, 2007-03-05 at 22:21 -0500, James Morris wrote:
> On Mon, 5 Mar 2007, Joy Latten wrote:
> 
> > 5. Around the time the set of SAs for OUT direction are to be
> >    inserted into SAD, I see another ACQUIRE happening.
> >    
> >    I have not yet figured out where this second ACQUIRE comes from
> >    and why it happens. As long as the minimal SA or set of valid outgoing
> >    SAs exist in SAD, an ACQUIRE should not happen.
> 
> I saw something similar to this some time ago when testing various 
> failure modes, and discused it with Herbert.
> 
> IIRC, there's a larval SA which is not torn down properly by Racoon once 
> the full SA is established, and the larval SA keeps resending until it 
> times out.
> 
Ok, good to know. 
I thought a bit more about this last night but am not
sure best way to fix it. Perhaps a way to keep larval
SA around until all SAs resulting from xfrm_vec[xfrm_nr]
are established... oh well, just thinking out loud... :-) 


Joy

Re: when having to acquire an SA, ipsec drops the packet

[James Morris]

On Tue, 6 Mar 2007, Joy Latten wrote:

> > I saw something similar to this some time ago when testing various 
> > failure modes, and discused it with Herbert.
> > 
> > IIRC, there's a larval SA which is not torn down properly by Racoon once 
> > the full SA is established, and the larval SA keeps resending until it 
> > times out.
> > 
> Ok, good to know. 
> I thought a bit more about this last night but am not
> sure best way to fix it. Perhaps a way to keep larval
> SA around until all SAs resulting from xfrm_vec[xfrm_nr]
> are established... oh well, just thinking out loud... :-) 

I think the solution, if this actually the problem, is for the userland 
code to maintain the SAs.


- James
-- 
James Morris
<jmorris@namei.org>