RE: CPU usage for IPSec in Linux 2.6.38

RE: CPU usage for IPSec in Linux 2.6.38

[Naveen B N (nbn)]

Hi All,
I am using linux 2.6.36 kernel and trying to add 6000 Ikev2/ipsec
tunnels at 20 Ikev2 messages per second , I am using netlink socket
which is set to NON_BLOCKING and i am sending XFRM SPD added for every
successful AUTH message received.

But the problem is after the 4000 tunnels are established, CPU usage
goes to > 90%, which will likely cause dropping of few AUTH response
from responder.

NOTE:
But when I disabled adding SPD messaged via netlink sockets using xfrm
messages, I am able to complete 6000 ikev2 SA negotiation successfully.

So the problem i am seeing is when sending XFRM netlink message > 4000.

Any solutions are or analysis different then the above is appreciated.


Thanks and Regards
Naveen
_______________________________________________
IPsec mailing list
IPsec@ietf.org
https://www.ietf.org/mailman/listinfo/ipsec

RE: [IPsec] CPU usage for IPSec in Linux 2.6.38

[Naveen B N (nbn)]

I think the issue is when adding an entry to SPD 
Database which already contains 4000 entries, because 
The entries have to be checked before adding the 
New entry.

Regards
Naveen  

-----Original Message-----
From: John Lowry [mailto:jlowry@bbn.com] 
Sent: Tuesday, January 10, 2012 8:25 PM
To: Naveen B N (nbn)
Subject: Re: [IPsec] CPU usage for IPSec in Linux 2.6.38

The only time we had troubles like this was with
a Dell 1U that had rotten drivers.  Upgrading to
different kernel and updated drivers fixed the problem.


On Jan 10, 2012, at 8:48 AM, Naveen B N (nbn) wrote:

> Hi All,
> I am using linux 2.6.36 kernel and trying to add 6000 Ikev2/ipsec
> tunnels 
> at 20 Ikev2 messages per second , I am using netlink socket which is
set
> to 
> NON_BLOCKING and i am sending XFRM SPD added for every successful AUTH
> message received.
> 
> But the problem is after the 4000 tunnels are established, CPU usage
> goes 
> to > 90%, which will likely cause dropping of few AUTH response from
> responder.
> 
> NOTE:
> But when I disabled adding SPD messaged via netlink sockets using xfrm
> messages,
> I am able to complete 6000 ikev2 SA negotiation successfully.
> 
> So the problem i am seeing is when sending XFRM netlink message >
4000.
> 
> Any solutions are or analysis different then the above is appreciated.
> 
> 
> Thanks and Regards
> Naveen 
> _______________________________________________
> IPsec mailing list
> IPsec@ietf.org
> https://www.ietf.org/mailman/listinfo/ipsec

RE: CPU usage for IPSec in Linux 2.6.38

[Eric Dumazet]

Le mardi 10 janvier 2012 à 20:23 +0530, Naveen B N (nbn) a écrit :
> Hi All,
> I am using linux 2.6.36 kernel and trying to add 6000 Ikev2/ipsec
> tunnels at 20 Ikev2 messages per second , I am using netlink socket
> which is set to NON_BLOCKING and i am sending XFRM SPD added for every
> successful AUTH message received.
> 
> But the problem is after the 4000 tunnels are established, CPU usage
> goes to > 90%, which will likely cause dropping of few AUTH response
> from responder.
> 
> NOTE:
> But when I disabled adding SPD messaged via netlink sockets using xfrm
> messages, I am able to complete 6000 ikev2 SA negotiation successfully.
> 
> So the problem i am seeing is when sending XFRM netlink message > 4000.
> 
> Any solutions are or analysis different then the above is appreciated.
> 

Most probably something uses a linear search, giving O(N^2) complexity.

You could post "perf top" results while doing these operations.

RE: CPU usage for IPSec in Linux 2.6.38

[Naveen B N (nbn)]

Hi All,
Did anybody try creating IPSec Tunnels > 6000 in Linux
And faced the same problem below.

Regards
Naveen

-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com] 
Sent: Tuesday, January 10, 2012 8:55 PM
To: Naveen B N (nbn)
Cc: netdev@vger.kernel.org
Subject: RE: CPU usage for IPSec in Linux 2.6.38

Le mardi 10 janvier 2012 à 20:23 +0530, Naveen B N (nbn) a écrit :
> Hi All,
> I am using linux 2.6.36 kernel and trying to add 6000 Ikev2/ipsec
> tunnels at 20 Ikev2 messages per second , I am using netlink socket
> which is set to NON_BLOCKING and i am sending XFRM SPD added for every
> successful AUTH message received.
> 
> But the problem is after the 4000 tunnels are established, CPU usage
> goes to > 90%, which will likely cause dropping of few AUTH response
> from responder.
> 
> NOTE:
> But when I disabled adding SPD messaged via netlink sockets using xfrm
> messages, I am able to complete 6000 ikev2 SA negotiation successfully.
> 
> So the problem i am seeing is when sending XFRM netlink message > 4000.
> 
> Any solutions are or analysis different then the above is appreciated.
> 

Most probably something uses a linear search, giving O(N^2) complexity.

You could post "perf top" results while doing these operations.

Re: CPU usage for IPSec in Linux 2.6.38

[David Miller]

From: "Naveen B N (nbn)" <nbn@cisco.com>
Date: Wed, 11 Jan 2012 08:20:12 +0530

> Did anybody try creating IPSec Tunnels > 6000 in Linux
> And faced the same problem below.

The problem is that you must situate your rules according to
certain rules otherwise performance will suffer greatly.

You must:

1) Predominantly use fully specified, non-wildcard, rules.

   These go into a special hash table which approaches complexity
   O(1).

2) If you absolutely must have wildcarded rules, only have an
   extremely small number of them.

   These go onto a linked list which is O(N).

There is no reasonable reason to have thousands of wildcarded
rules.

Thousands of fully specified non-wildcard rules are reasonable,
and what we optimize the IPSEC datastructures for.

Re: CPU usage for IPSec in Linux 2.6.38

[Naveen B N (nbn)]

Hi David,
Thanks for the early reply .

I can't have a wild card has my outer IP address
Are unique for across Ipsec Tunnle > 6000, so 
I this case I have to add all the policies for 
Which the selectors my have a wild card but 
Outer IP address are specific and unique.

So taught of optimizing my just adding the policy 
Entry without the actual checking the previous list 
In kernel.

Is there better way of doing it David.

Thanks and Regards
Naveen

-----Original Message-----
From: David Miller [mailto:davem-fT/PcQaiUtIeIZ0/mPfg9Q@public.gmane.org] 
Sent: Wednesday, January 11, 2012 12:05 PM
To: Naveen B N (nbn)
Cc: ipsec-tools-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org; netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: CPU usage for IPSec in Linux 2.6.38

From: "Naveen B N (nbn)" <nbn-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
Date: Wed, 11 Jan 2012 08:20:12 +0530

> Did anybody try creating IPSec Tunnels > 6000 in Linux
> And faced the same problem below.

The problem is that you must situate your rules according to
certain rules otherwise performance will suffer greatly.

You must:

1) Predominantly use fully specified, non-wildcard, rules.

   These go into a special hash table which approaches complexity
   O(1).

2) If you absolutely must have wildcarded rules, only have an
   extremely small number of them.

   These go onto a linked list which is O(N).

There is no reasonable reason to have thousands of wildcarded
rules.

Thousands of fully specified non-wildcard rules are reasonable,
and what we optimize the IPSEC datastructures for.

------------------------------------------------------------------------------
Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex
infrastructure or vast IT resources to deliver seamless, secure access to
virtual desktops. With this all-in-one solution, easily deploy virtual 
desktops for less than the cost of PCs and save 60% on VDI infrastructure 
costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox

Re: CPU usage for IPSec in Linux 2.6.38

[David Miller]

From: "Naveen B N (nbn)" <nbn-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org>
Date: Wed, 11 Jan 2012 12:39:07 +0530

> Is there better way of doing it David.

You can't depend upon me, and only me, to solve all of your
problems.

I've given you the constraints for rulesets for good performance,
it is your task to make your ruleset comply.

------------------------------------------------------------------------------
Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex
infrastructure or vast IT resources to deliver seamless, secure access to
virtual desktops. With this all-in-one solution, easily deploy virtual 
desktops for less than the cost of PCs and save 60% on VDI infrastructure 
costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox