IPsec performance

IPsec performance

[Miika Komu]

Hi folks,

I compared TCP-over-NULL-ESP vs. plain-TCP throughput with "ttcp" tool and
obtained the following results:

Plain-TCP:	11 megabytes/s
NULL-ESP:	2.5 megabytes/s

Am I doing something wrong? I did the measurements by transferring a 100
megabyte file using vanilla 2.6.7 kernel and IPv6. Seems like my
results are inconsistent at least with measurements published by Linux
Journal:

http://www.linuxjournal.com/article/7840

The main differences with the Linux Journal measurements are that I used a
significantly larger filesize and IPv6. So, I tried to run the same with
IPv4 and got similar results... any ideas?

-- 
Miika Komu              miika@iki.fi          http://www.iki.fi/miika/

Re: IPsec performance

[Miika Komu]

On Mon, 2 May 2005, Miika Komu wrote:

> Hi folks,
>
> I compared TCP-over-NULL-ESP vs. plain-TCP throughput with "ttcp" tool and
> obtained the following results:
>
> Plain-TCP:	11 megabytes/s
> NULL-ESP:	2.5 megabytes/s
>
> Am I doing something wrong? I did the measurements by transferring a 100
> megabyte file using vanilla 2.6.7 kernel and IPv6. Seems like my
> results are inconsistent at least with measurements published by Linux
> Journal:
>
> http://www.linuxjournal.com/article/7840
>
> The main differences with the Linux Journal measurements are that I used a
> significantly larger filesize and IPv6. So, I tried to run the same with
> IPv4 and got similar results... any ideas?

I browsed the archives also and found this one:

http://oss.sgi.com/archives/netdev/2004-12/msg00801.html

Nothing has been done for this?

-- 
Miika Komu              miika@iki.fi          http://www.iki.fi/miika/

Re: IPsec performance

[Patrick McHardy]

On Mon, 2 May 2005, Miika Komu wrote:
>> I compared TCP-over-NULL-ESP vs. plain-TCP throughput with "ttcp" tool and
>> obtained the following results:
>>
>> Plain-TCP:	11 megabytes/s
>> NULL-ESP:	2.5 megabytes/s
>>
> I browsed the archives also and found this one:
>
> http://oss.sgi.com/archives/netdev/2004-12/msg00801.html

null encryption uses a block size of 1, which causes inefficent handling
of the data. I'm unsure if some hack for this specific case is worth it
since its only meant for testing.

Regards
Patrick

Re: IPsec performance

[Miika Komu]

On Tue, 3 May 2005, Patrick McHardy wrote:

> On Mon, 2 May 2005, Miika Komu wrote:
> >> I compared TCP-over-NULL-ESP vs. plain-TCP throughput with "ttcp" tool and
> >> obtained the following results:
> >>
> >> Plain-TCP:	11 megabytes/s
> >> NULL-ESP:	2.5 megabytes/s
> >>
> > I browsed the archives also and found this one:
> >
> > http://oss.sgi.com/archives/netdev/2004-12/msg00801.html
>
> null encryption uses a block size of 1, which causes inefficent handling
> of the data. I'm unsure if some hack for this specific case is worth it
> since its only meant for testing.

Thanks for your response.

However, this does explain why I got only 1.9 megabytes/s with 3DES.

-- 
Miika Komu              miika@iki.fi          http://www.iki.fi/miika/

Re: IPsec performance

[Dave Dillow]

On Tue, 2005-05-03 at 08:25 +0300, Miika Komu wrote:
> However, this does explain why I got only 1.9 megabytes/s with 3DES.

What was your hardware? Without knowing what you're running on, it's
impossible to tell if that is good or bad.

Linux Journal was getting ~5.5MB/s on a PIV 2.2GHz.
I get ~2.1MB/s on a 550MHz Athlon.

Dave

Re: IPsec performance

[Miika Komu]

On Tue, 3 May 2005, Dave Dillow wrote:

> On Tue, 2005-05-03 at 08:25 +0300, Miika Komu wrote:
> > However, this does explain why I got only 1.9 megabytes/s with 3DES.
>
> What was your hardware? Without knowing what you're running on, it's
> impossible to tell if that is good or bad.
>
> Linux Journal was getting ~5.5MB/s on a PIV 2.2GHz.
> I get ~2.1MB/s on a 550MHz Athlon.

Intel Pentium III 500 MHz

-- 
Miika Komu              miika@iki.fi          http://www.iki.fi/miika/

Re: IPsec performance

[Dave Dillow]

On Tue, 2005-05-03 at 09:44 +0300, Miika Komu wrote:
> On Tue, 3 May 2005, Dave Dillow wrote:
> 
> > On Tue, 2005-05-03 at 08:25 +0300, Miika Komu wrote:
> > > However, this does explain why I got only 1.9 megabytes/s with 3DES.
> >
> > What was your hardware? Without knowing what you're running on, it's
> > impossible to tell if that is good or bad.
> >
> > Linux Journal was getting ~5.5MB/s on a PIV 2.2GHz.
> > I get ~2.1MB/s on a 550MHz Athlon.
> 
> Intel Pentium III 500 MHz

3DES is processor intensive. I'd say 1.9MB/s is reasonable for the
hardware you are on.

Re: IPsec performance

[Miika Komu]

On Tue, 3 May 2005, Dave Dillow wrote:

> On Tue, 2005-05-03 at 09:44 +0300, Miika Komu wrote:
> > On Tue, 3 May 2005, Dave Dillow wrote:
> >
> > > On Tue, 2005-05-03 at 08:25 +0300, Miika Komu wrote:
> > > > However, this does explain why I got only 1.9 megabytes/s with 3DES.
> > >
> > > What was your hardware? Without knowing what you're running on, it's
> > > impossible to tell if that is good or bad.
> > >
> > > Linux Journal was getting ~5.5MB/s on a PIV 2.2GHz.
> > > I get ~2.1MB/s on a 550MHz Athlon.
> >
> > Intel Pentium III 500 MHz
>
> 3DES is processor intensive. I'd say 1.9MB/s is reasonable for the
> hardware you are on.

Ok. Thanks for your quick responses!

-- 
Miika Komu              miika@iki.fi          http://www.iki.fi/miika/

Re: IPsec performance

[Dave Dillow]

On Tue, 2005-05-03 at 19:14 +0300, Miika Komu wrote:
> On Tue, 3 May 2005, Dave Dillow wrote:
> > 3DES is processor intensive. I'd say 1.9MB/s is reasonable for the
> > hardware you are on.
> 
> Ok. Thanks for your quick responses!

I should note that I mean reasonable given the current implementation --
you're seeing performance in line with numbers on other machines. I
think it's been mentioned that our 3DES implementation has not been
tuned for x86, so there's probably some performance being left on the
floor. I could be wrong, though.
-- 
Dave Dillow <dave@thedillows.org>

Re: IPsec performance

[David S. Miller]

On Tue, 03 May 2005 15:27:41 -0400
Dave Dillow <dave@thedillows.org> wrote:

> I think it's been mentioned that our 3DES implementation has not been
> tuned for x86, so there's probably some performance being left on the
> floor.

That's right.  We only have an optimized AES implementation in assembler,
and that's for i386 and x86_64 platforms only.

Re: IPsec performance

[jamal]

On Tue, 2005-03-05 at 09:44 +0300, Miika Komu wrote:
> On Tue, 3 May 2005, Dave Dillow wrote:
> 
> > On Tue, 2005-05-03 at 08:25 +0300, Miika Komu wrote:
> > > However, this does explain why I got only 1.9 megabytes/s with 3DES.
> >
> > What was your hardware? Without knowing what you're running on, it's
> > impossible to tell if that is good or bad.
> >
> > Linux Journal was getting ~5.5MB/s on a PIV 2.2GHz.
> > I get ~2.1MB/s on a 550MHz Athlon.
> 
> Intel Pentium III 500 MHz

I tested on 2.4.x based on Davems code backport and contribution from
Herbert > a year ago on a really bad Xeon 2.0Ghz(?), 32 bit bus mboard,
e1000 NIC. I used a 2.4.x kernel because i was trying to compare against
a broadcom SB1250 board which couldnt run 2.6.x - I also had to do a few
tricks to get it to run/compile on MIPS and so may have contributed to
perfomance degradation. 
In any case, the numbers between those two were not very different (even
though the sb1250 was at i think 600Mhz). Unfortunately i lost the
loaner sb1250 shortly after (it was fun to play with). I have my full
results somewhere in another machine which i can lookup, but heres some
summary data i had in my laptop:

used:
A) i) two flows ii) 4 flows iii) 8 flows
all seem to reproduce the same aggregate throughput. 

B) static keying


Tests and results
-----------------

I: Transport mode:

1) Message Authentication: tested with hmac-md5
Should have done sha-1. Next time.

a) 64 byte packets
Input 100Mbps (148.8Kpps) output 112 Kpps

This is about 75% of wire rate (75Mbps).

b) 1400 byte packets
8712 (99% of wire rate) == about 17.4Kpps which is almost
wire rate at 200Mbps at that packet size.
Note we can never get 100% wire rate since the packet header grows
because of the AH insertion.

2) Data encryption mode: 3DES/md5

a) 64 byte packets
Peak achievement:
43Kpps == 30Mbps.

b) 1400 byte packets
Peak achievement:
1.8Kpps == 20Mbps.

Now this result is very interesting. It clearly shows that 3DES
is compute intensive i.e as the packet grows you get slower - quiet
counterintuitive. 
An encryption chip would go along way to help with large packet sizes.
Another observation:
In other words given the synchronous nature of the crypto path, there is
a threshold value where it may be valuable to use a crypto chip.
The effects of this were actually demonstrated by Eugene Surovegin
<ebs@ebshome.net>; long thread of discussion on this on netdev.

We need to find the threshold packet size where encryption starts
overtaking general packet processing i.e where offloading becomes
interesting if we stick with sync scheme.

Another thing for s/ware based crypto is to improve the 3DES
implementation as in assembler coding.

Iam back on/off playing with this, so expect more results Real Soon Now.


cheers,
jamal

ipsec performance

[Andreas Schuldei]

hi!

i experience performance issues with ipsec transport mode with debian
lenny and strongswan, on a stock debian kernel 2.6.26-2-amd64.

the goal is to set up a full mash of several hundred hosts, talking
ipsec with each other, in order to be able to skip firewalls and to be
able to let the hosts be spread out over several sites in a
transparent fashion.

regardless of the cipher (i tried aes and blowfish) the bandwidth
maxes out at about 0.5-0.25 of the expected (unencrypted) value,
without hitting obvious bottlenecks like cpu, disk, or ram.

tcpdump shows packages below the MTU (which is 1500):

20:25:03.313469 IP 78.31.14.86 > 78.31.14.93:
ESP(spi=0xc929dbe8,seq=0x100a87), length 1332
20:25:03.313514 IP 78.31.14.86 > 78.31.14.93:
ESP(spi=0xc929dbe8,seq=0x100a88), length 1476
20:25:03.313529 IP 78.31.14.93 > 78.31.14.86:
ESP(spi=0xc4967810,seq=0x7bcd1), length 68
20:25:03.313557 IP 78.31.14.86 > 78.31.14.93:
ESP(spi=0xc929dbe8,seq=0x100a89), length 1476
20:25:03.313603 IP 78.31.14.86 > 78.31.14.93:
ESP(spi=0xc929dbe8,seq=0x100a8a), length 1332
20:25:03.313605 IP 78.31.14.86 > 78.31.14.93:
ESP(spi=0xc929dbe8,seq=0x100a8a), length 1332
20:25:03.313606 IP 78.31.14.93 > 78.31.14.86:
ESP(spi=0xc4967810,seq=0x7bcd2), length 68
20:25:03.313649 IP 78.31.14.86 > 78.31.14.93:
ESP(spi=0xc929dbe8,seq=0x100a8b), length 1476

how can i inspect window size, fragmentation etc? are there useful
files in /proc or /sys or enlightening ip commands?

/andreas

Re: ipsec performance

[Abhijit Karmarkar]

On Tue, Dec 29, 2009 at 1:09 PM, Andreas Schuldei <schuldei@spotify.com> wrote:
> hi!
>
> i experience performance issues with ipsec transport mode with debian
> lenny and strongswan, on a stock debian kernel 2.6.26-2-amd64.
>
> the goal is to set up a full mash of several hundred hosts, talking
> ipsec with each other, in order to be able to skip firewalls and to be
> able to let the hosts be spread out over several sites in a
> transparent fashion.
>
> regardless of the cipher (i tried aes and blowfish) the bandwidth
> maxes out at about 0.5-0.25 of the expected (unencrypted) value,
> without hitting obvious bottlenecks like cpu, disk, or ram.

you may want try Steffen Klassert's parallel crypto patches (nice work!):

  http://marc.info/?l=linux-kernel&m=126155699817914&w=2

the numbers are impressive. i plan to try them sometime this (or next week).

yes, on the current kernels, the ipsec throughput numbers are around
50% of the non-ipsec case. for me.


>
> tcpdump shows packages below the MTU (which is 1500):
>
> 20:25:03.313469 IP 78.31.14.86 > 78.31.14.93:
> ESP(spi=0xc929dbe8,seq=0x100a87), length 1332
> 20:25:03.313514 IP 78.31.14.86 > 78.31.14.93:
> ESP(spi=0xc929dbe8,seq=0x100a88), length 1476
> 20:25:03.313529 IP 78.31.14.93 > 78.31.14.86:
> ESP(spi=0xc4967810,seq=0x7bcd1), length 68
> 20:25:03.313557 IP 78.31.14.86 > 78.31.14.93:
> ESP(spi=0xc929dbe8,seq=0x100a89), length 1476
> 20:25:03.313603 IP 78.31.14.86 > 78.31.14.93:
> ESP(spi=0xc929dbe8,seq=0x100a8a), length 1332
> 20:25:03.313605 IP 78.31.14.86 > 78.31.14.93:
> ESP(spi=0xc929dbe8,seq=0x100a8a), length 1332
> 20:25:03.313606 IP 78.31.14.93 > 78.31.14.86:
> ESP(spi=0xc4967810,seq=0x7bcd2), length 68
> 20:25:03.313649 IP 78.31.14.86 > 78.31.14.93:
> ESP(spi=0xc929dbe8,seq=0x100a8b), length 1476
>
> how can i inspect window size, fragmentation etc? are there useful
> files in /proc or /sys or enlightening ip commands?
>
> /andreas
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: ipsec performance

[Andreas Schuldei]

On Tue, Dec 29, 2009 at 11:55 PM, Abhijit Karmarkar <awk@google.com> wrote:
> On Tue, Dec 29, 2009 at 1:09 PM, Andreas Schuldei <schuldei@spotify.com> wrote:
>> hi!
>>
>> i experience performance issues with ipsec transport mode with debian
>> lenny and strongswan, on a stock debian kernel 2.6.26-2-amd64.
>>
>> the goal is to set up a full mash of several hundred hosts, talking
>> ipsec with each other, in order to be able to skip firewalls and to be
>> able to let the hosts be spread out over several sites in a
>> transparent fashion.
>>
>> regardless of the cipher (i tried aes and blowfish) the bandwidth
>> maxes out at about 0.5-0.25 of the expected (unencrypted) value,
>> without hitting obvious bottlenecks like cpu, disk, or ram.
>
> you may want try Steffen Klassert's parallel crypto patches (nice work!):
>
>  http://marc.info/?l=linux-kernel&m=126155699817914&w=2
>
> the numbers are impressive. i plan to try them sometime this (or next week).
>
> yes, on the current kernels, the ipsec throughput numbers are around
> 50% of the non-ipsec case. for me.

i have an 8core xeon machine 2.5GHz machine and my throughput of
39Mbyte/s correlates nicely with Steffens 325Mbit/s when i use AES.
when i switch to blowfish the throuput decreases to 27.5Mbyte/s. the
time the cpu spends in kernel code decreases, too, to ~5% (give or
take a coconut). the machine seems idle, almost. where is the
bottleneck? what needs parallelization? (i read steffens mail but i
didnt understand his explanation. could you explain it in laymens
terms?) what i find funny is that the apache process serving the data
uses between 20-95%cpu. how come?

i dont intent do have vpn gateways.  I want every machine to encrypt
its own network traffic. doubling the performance (as steffens patch
seems to do) would help (in the AES case, not for blowfish.). i would
want to actually deploy the stuff soon, though, and i will have a hard
time selling a patch and a homebuild kernel to my colleges.

>> how can i inspect window size, fragmentation etc? are there useful
>> files in /proc or /sys or enlightening ip commands?

is there a way to do this?