"xfrm: Fix the gc threshold value for ipv4" broke my IPSec connections

"xfrm: Fix the gc threshold value for ipv4" broke my IPSec connections

[Damian Pietras]

I've recently upgraded from 3.4.x to 3.10.x and this broke my IPSec
setup in transport mode. The simplest test case is to setup few such
connections with few boxes like this:

spdadd 192.168.1.100 192.168.2.100 any -P out ipsec
           esp/transport//require
           ah/transport//require;

spdadd 192.168.2.100 192.168.1.100 any -P in ipsec
           esp/transport//require
           ah/transport//require;

Then set up an HTTP server on one box and run ab on the other box to
create come TCP connections:

ab -n 10000 -c 50 http://192.168.1.100/

Then the connect() call will very quickly start returning ENOBUFS. I
haven't seen anything wrong with my simple setup (just copy of
ipsec-howto.org in transport mode and pre shared keys) and started
bisecting. That way I found this commit to break my case:

703fb94ec58e0e8769380c2877a8a34aeb5b6c97
xfrm: Fix the gc threshold value for ipv4

Reverting it on 3.10.15 fixes my issue. This seems to be there from 3.7
and I don't really believe such simple case stayed broken for so long.
Em I missing something or there is really a bug?

If smeone is interested in details of this configuration and commands
I'm running, just let me know. This was reproduced with few VMs under XEN.

-- 
Damian Pietras

Re: "xfrm: Fix the gc threshold value for ipv4" broke my IPSec connections

[Eric Dumazet]

On Tue, 2013-10-15 at 22:40 +0200, Damian Pietras wrote:
> I've recently upgraded from 3.4.x to 3.10.x and this broke my IPSec
> setup in transport mode. The simplest test case is to setup few such
> connections with few boxes like this:
> 
> spdadd 192.168.1.100 192.168.2.100 any -P out ipsec
>            esp/transport//require
>            ah/transport//require;
> 
> spdadd 192.168.2.100 192.168.1.100 any -P in ipsec
>            esp/transport//require
>            ah/transport//require;
> 
> Then set up an HTTP server on one box and run ab on the other box to
> create come TCP connections:
> 
> ab -n 10000 -c 50 http://192.168.1.100/
> 
> Then the connect() call will very quickly start returning ENOBUFS. I
> haven't seen anything wrong with my simple setup (just copy of
> ipsec-howto.org in transport mode and pre shared keys) and started
> bisecting. That way I found this commit to break my case:
> 
> 703fb94ec58e0e8769380c2877a8a34aeb5b6c97
> xfrm: Fix the gc threshold value for ipv4
> 
> Reverting it on 3.10.15 fixes my issue. This seems to be there from 3.7
> and I don't really believe such simple case stayed broken for so long.
> Em I missing something or there is really a bug?
> 
> If smeone is interested in details of this configuration and commands
> I'm running, just let me know. This was reproduced with few VMs under XEN.
> 

It looks like you need to tune /proc/sys/net/ipv4/xfrm4_gc_thresh to a
sensible value given your workload.

try :

echo 65536 >/proc/sys/net/ipv4/xfrm4_gc_thresh

Presumably the 1024 default is really too small...

Re: "xfrm: Fix the gc threshold value for ipv4" broke my IPSec connections

[Damian Pietras]

On 15.10.2013 23:02, Eric Dumazet wrote:
>> 703fb94ec58e0e8769380c2877a8a34aeb5b6c97
>> xfrm: Fix the gc threshold value for ipv4
>>
>> Reverting it on 3.10.15 fixes my issue. This seems to be there from 3.7
>> and I don't really believe such simple case stayed broken for so long.
>> Em I missing something or there is really a bug?
>>
>> If smeone is interested in details of this configuration and commands
>> I'm running, just let me know. This was reproduced with few VMs under XEN.
>>
> 
> It looks like you need to tune /proc/sys/net/ipv4/xfrm4_gc_thresh to a
> sensible value given your workload.
> 
> try :
> 
> echo 65536 >/proc/sys/net/ipv4/xfrm4_gc_thresh
> 
> Presumably the 1024 default is really too small...

Now it's working in my test setup, I'm changing it on the production
boxes, thanks!


-- 
Damian Pietras

Re: "xfrm: Fix the gc threshold value for ipv4" broke my IPSec connections

[Eric Dumazet]

On Wed, 2013-10-16 at 00:15 +0200, Damian Pietras wrote:
> On 15.10.2013 23:02, Eric Dumazet wrote:
> >> 703fb94ec58e0e8769380c2877a8a34aeb5b6c97
> >> xfrm: Fix the gc threshold value for ipv4
> >>
> >> Reverting it on 3.10.15 fixes my issue. This seems to be there from 3.7
> >> and I don't really believe such simple case stayed broken for so long.
> >> Em I missing something or there is really a bug?
> >>
> >> If smeone is interested in details of this configuration and commands
> >> I'm running, just let me know. This was reproduced with few VMs under XEN.
> >>
> > 
> > It looks like you need to tune /proc/sys/net/ipv4/xfrm4_gc_thresh to a
> > sensible value given your workload.
> > 
> > try :
> > 
> > echo 65536 >/proc/sys/net/ipv4/xfrm4_gc_thresh
> > 
> > Presumably the 1024 default is really too small...
> 
> Now it's working in my test setup, I'm changing it on the production
> boxes, thanks!
> 
> 

Steffen, what do you think ?

1024 seems really small, given we had much higher values.

(256 K on a 1GB host)

This sysctl also needs an entry in
Documentation/networking/ip-sysctl.txt

Re: "xfrm: Fix the gc threshold value for ipv4" broke my IPSec connections

[Steffen Klassert]

On Tue, Oct 15, 2013 at 03:51:26PM -0700, Eric Dumazet wrote:
> On Wed, 2013-10-16 at 00:15 +0200, Damian Pietras wrote:
> > On 15.10.2013 23:02, Eric Dumazet wrote:
> > >> 703fb94ec58e0e8769380c2877a8a34aeb5b6c97
> > >> xfrm: Fix the gc threshold value for ipv4
> > >>
> > >> Reverting it on 3.10.15 fixes my issue. This seems to be there from 3.7
> > >> and I don't really believe such simple case stayed broken for so long.
> > >> Em I missing something or there is really a bug?
> > >>
> > >> If smeone is interested in details of this configuration and commands
> > >> I'm running, just let me know. This was reproduced with few VMs under XEN.
> > >>
> > > 
> > > It looks like you need to tune /proc/sys/net/ipv4/xfrm4_gc_thresh to a
> > > sensible value given your workload.
> > > 
> > > try :
> > > 
> > > echo 65536 >/proc/sys/net/ipv4/xfrm4_gc_thresh
> > > 
> > > Presumably the 1024 default is really too small...
> > 
> > Now it's working in my test setup, I'm changing it on the production
> > boxes, thanks!
> > 
> > 
> 
> Steffen, what do you think ?
> 
> 1024 seems really small, given we had much higher values.

Sure, we can increase the default value, maybe along with the ipv6 side.
Any recomendation on a good default for both?

> 
> (256 K on a 1GB host)
> 
> This sysctl also needs an entry in
> Documentation/networking/ip-sysctl.txt
> 

Yes, it's undocumented. I'll take care of it.