unstable 10GBE performance with recent kernels (> 3.0.X)

unstable 10GBE performance with recent kernels (> 3.0.X)

[Stefan Priebe - Profihost AG]

Hello list,

i've discovered very unstable 10GBE performance using recent kernels. 
I'm using some optimized settings mentioned by intel here 
(part:Improving Performance): 
http://downloadmirror.intel.com/5874/eng/README.txt

I'm using Intel X520 cards (ixgbe driver version: 3.9.17-NAPI in all 
tests).

I'm measuring the performance with iperf.

With 3.0.32 i get constant 9,90 Gbit/s in both directions simultaneously.

With 3.4.2 or 3.5.0-rc2 it get sometimes 9,9gbit/s - sometimes 3gbit/s 
or even sometimes only 1gbit/s throughput.

I also tried to change the tcp_congestion_control from cubic to reno, 
bic and highspeed but no change.

I also tried to bisect the issue but there are so many changes in the 
net kernel part that i'm unable to identify the problem as with some 
commits i only get 0-300kb/s performance.

Any ideas?

Greets
Stefan

Re: unstable 10GBE performance with recent kernels (> 3.0.X)

[Eric Dumazet]

On Mon, 2012-06-18 at 11:22 +0200, Stefan Priebe - Profihost AG wrote:
> Hello list,
> 
> i've discovered very unstable 10GBE performance using recent kernels. 
> I'm using some optimized settings mentioned by intel here 
> (part:Improving Performance): 
> http://downloadmirror.intel.com/5874/eng/README.txt
> 

like what settings ?

> I'm using Intel X520 cards (ixgbe driver version: 3.9.17-NAPI in all 
> tests).
> 
> I'm measuring the performance with iperf.
> 
> With 3.0.32 i get constant 9,90 Gbit/s in both directions simultaneously.
> 
> With 3.4.2 or 3.5.0-rc2 it get sometimes 9,9gbit/s - sometimes 3gbit/s 
> or even sometimes only 1gbit/s throughput.
> 
> I also tried to change the tcp_congestion_control from cubic to reno, 
> bic and highspeed but no change.
> 
> I also tried to bisect the issue but there are so many changes in the 
> net kernel part that i'm unable to identify the problem as with some 
> commits i only get 0-300kb/s performance.
> 
> Any ideas?

ethtool -S eth0
ethtool -k eth0
netstat -s   (on sender, on receiver)

single flow , multiple flows ?

Re: unstable 10GBE performance with recent kernels (> 3.0.X)

[Stefan Priebe - Profihost AG]

Am 18.06.2012 11:45, schrieb Eric Dumazet:
> On Mon, 2012-06-18 at 11:22 +0200, Stefan Priebe - Profihost AG wrote:
>> Hello list,
>>
>> i've discovered very unstable 10GBE performance using recent kernels.
>> I'm using some optimized settings mentioned by intel here
>> (part:Improving Performance):
>> http://downloadmirror.intel.com/5874/eng/README.txt
>>
> like what settings ?

Sorry these ones:
http://pastebin.com/raw.php?i=J2XRFAjD

>> Any ideas?
>
> ethtool -S eth0
 From host1 and 2: http://pastebin.com/raw.php?i=0Pp8Vs3Y

> ethtool -k eth0
 From host1 and 2: # ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
ntuple-filters: off
receive-hashing: on

# ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
ntuple-filters: off
receive-hashing: on

> netstat -s   (on sender, on receiver)

 From host1 and 2: http://pastebin.com/raw.php?i=P9T96j0L

> single flow , multiple flows ?

host1: iperf -s
host2: iperf -c ssdstor002 -d -t 600 -i 10;

In this test i got around 7.4Gbit/s

Stefan

Re: unstable 10GBE performance with recent kernels (> 3.0.X)

[Eric Dumazet]

On Mon, 2012-06-18 at 12:05 +0200, Stefan Priebe - Profihost AG wrote:
> Am 18.06.2012 11:45, schrieb Eric Dumazet:
> > On Mon, 2012-06-18 at 11:22 +0200, Stefan Priebe - Profihost AG wrote:
> >> Hello list,
> >>
> >> i've discovered very unstable 10GBE performance using recent kernels.
> >> I'm using some optimized settings mentioned by intel here
> >> (part:Improving Performance):
> >> http://downloadmirror.intel.com/5874/eng/README.txt
> >>
> > like what settings ?
> 
> Sorry these ones:
> http://pastebin.com/raw.php?i=J2XRFAjD
> 

I would remove all this (pretty old and obsolete) stuff and use standard
params.

Only thing you could do is :

ethtool -K eth2 ntuple on

on both machines

Re: unstable 10GBE performance with recent kernels (> 3.0.X)

[Stefan Priebe - Profihost AG]

Am 18.06.2012 12:25, schrieb Eric Dumazet:
> I would remove all this (pretty old and obsolete) stuff and use standard
> params.
OK, thanks. done.

I've one RHEL6 system (using default 2.6.32 kernel) where only using 
"ntuple on" results in just 4-5Gbit/s while the others using 3.5.0-rc2 
are working fine.

> Only thing you could do is :
> ethtool -K eth2 ntuple on
>
> on both machines
Thanks that works great and boosts the performance a lot. What does it 
do? man ethtool doesn't show anything useful.

Is this also recommanded for 1Gb/s?

Could you recommend a tcp_congestion_control module? RHEL6 uses cubic by 
default. Some others use reno or bic.

Stefan

Re: unstable 10GBE performance with recent kernels (> 3.0.X)

[Eric Dumazet]

On Mon, 2012-06-18 at 14:36 +0200, Stefan Priebe - Profihost AG wrote:
> Am 18.06.2012 12:25, schrieb Eric Dumazet:
> > I would remove all this (pretty old and obsolete) stuff and use standard
> > params.
> OK, thanks. done.
> 
> I've one RHEL6 system (using default 2.6.32 kernel) where only using 
> "ntuple on" results in just 4-5Gbit/s while the others using 3.5.0-rc2 
> are working fine.
> 

OK, then you might play with affinities, that's the only thing that
might need sysadmin tuning (irqbalance needs to be disabled)

> > Only thing you could do is :
> > ethtool -K eth2 ntuple on
> >
> > on both machines
> Thanks that works great and boosts the performance a lot. What does it 
> do? man ethtool doesn't show anything useful.
> 

I am not Intel guy, but you might find some info in :

http://downloadmirror.intel.com/14687/eng/README.txt



> Is this also recommanded for 1Gb/s?
> 

Not sure it's supported on ixgb


> Could you recommend a tcp_congestion_control module? RHEL6 uses cubic by 
> default. Some others use reno or bic.

Hard to tell, there is no generic answer. Default should be fine for
most uses.

Re: unstable 10GBE performance with recent kernels (> 3.0.X)

[Stefan Priebe - Profihost AG]

Am 18.06.2012 14:51, schrieb Eric Dumazet:
 > On Mon, 2012-06-18 at 14:36 +0200, Stefan Priebe - Profihost AG wrote:
 >> Am 18.06.2012 12:25, schrieb Eric Dumazet:
 >>> I would remove all this (pretty old and obsolete) stuff and use 
standard
 >>> params.
 >> OK, thanks. done.
 >>
 >> I've one RHEL6 system (using default 2.6.32 kernel) where only using
 >> "ntuple on" results in just 4-5Gbit/s while the others using 3.5.0-rc2
 >> are working fine.
 >>
 >
 > OK, then you might play with affinities, that's the only thing that
 > might need sysadmin tuning (irqbalance needs to be disabled)

I've now used this one:

# script provided by intel
/root/set_irq_affinity.sh eth2
ifconfig eth2 mtu 9000
ethtool -K eth2 ntuple on

irqbalance is off

But with the latest long term stable vanilla kernel 3.0.34.

# iperf -c HOST -t 60

works fine but with -d for using both directions i see around 9.8Gbit/s 
in one direction and only 300kb/s in the other direction.

Stefan

Re: unstable 10GBE performance with recent kernels (> 3.0.X)

[Eric Dumazet]

On Mon, 2012-06-18 at 15:07 +0200, Stefan Priebe - Profihost AG wrote:
> Am 18.06.2012 14:51, schrieb Eric Dumazet:
>  > On Mon, 2012-06-18 at 14:36 +0200, Stefan Priebe - Profihost AG wrote:
>  >> Am 18.06.2012 12:25, schrieb Eric Dumazet:
>  >>> I would remove all this (pretty old and obsolete) stuff and use 
> standard
>  >>> params.
>  >> OK, thanks. done.
>  >>
>  >> I've one RHEL6 system (using default 2.6.32 kernel) where only using
>  >> "ntuple on" results in just 4-5Gbit/s while the others using 3.5.0-rc2
>  >> are working fine.
>  >>
>  >
>  > OK, then you might play with affinities, that's the only thing that
>  > might need sysadmin tuning (irqbalance needs to be disabled)
> 
> I've now used this one:
> 
> # script provided by intel
> /root/set_irq_affinity.sh eth2
> ifconfig eth2 mtu 9000
> ethtool -K eth2 ntuple on
> 
> irqbalance is off
> 
> But with the latest long term stable vanilla kernel 3.0.34.
> 
> # iperf -c HOST -t 60
> 
> works fine but with -d for using both directions i see around 9.8Gbit/s 
> in one direction and only 300kb/s in the other direction.
> 
> Stefan


I would try using regular MTU.

And post "netstat -s" results before/after your iperf.

Re: unstable 10GBE performance with recent kernels (> 3.0.X)

[Stefan Priebe - Profihost AG]

Am 18.06.2012 15:29, schrieb Eric Dumazet:
>> But with the latest long term stable vanilla kernel 3.0.34.
>>
>> # iperf -c HOST -t 60
>>
>> works fine but with -d for using both directions i see around 9.8Gbit/s
>> in one direction and only 300kb/s in the other direction.
>
> I would try using regular MTU.
For 3.5.0-rc2 this works fine but decreases the speed from 9,9Gbit/s to 
9,3Gbit/s. With mtu 9000 i get around 3-4Gbit/s (netstat -s attached below)

With 3.0.34 MTU 9000 works fine with constant 9,9Gbit/s in both directions.

> And post "netstat -s" results before/after your iperf.

Tests with mtu 9000 and 3.5rc2:
Pre: http://pastebin.com/raw.php?i=uBjsFKjs
Post: http://pastebin.com/raw.php?i=TAK8JS6j

Thanks again!

Stefan