stable 10GBE under 3.4.2 or 3.5.0-rc2

stable 10GBE under 3.4.2 or 3.5.0-rc2

[Stefan Priebe - Profihost AG]

Hi list,

i've still problems with stable network speed under recent kernels. With 
3.0.32 i get stable 9.90 Gbit/s in both directions. With 3.4.2 or 
3.5.0-rc2 it drops sometimes down to around 1 Gbit/s. Sadly i've no idea 
when this happen? It works sometimes for minutes at 9.9Gbit/s and then 
suddently only with 3-4Gbit/s or even 1Gbit/s using recent kernels.

I'm using various tunings recommanded by intel (Improving Performance) 
=> http://downloadmirror.intel.com/5874/eng/README.txt

Has anybody a hint for me or additional settings intel does not mention?

Stefan

Re: stable 10GBE under 3.4.2 or 3.5.0-rc2

[Alexandre DERUMIER]

maybe try to play with net.ipv4.tcp_congestion_control ? 



The available algorithms are: 

    • BIC - The default on Gentoo 
    • Reno - The classic TCP protocol. Most OSes use this. 
    • highspeed - HighSpeed TCP: Sally Floyd's suggested algorithm 
    • htcp - Hamilton TCP 
    • hybla - For Satellite Links 
    • scalable - Scalable TCP 
    • vegas - Vegas TCP 
    • westwood - Optimized for lossy networks 

http://lwn.net/Articles/128681/

The High-speed TCP algorithm is optimized for very fat pipes - 10G Ethernet and such. When things are congested, it behaves much like the Reno algorithm. When the congestion window is being increased, however, the high-speed algorithm makes use of a table to pick large increment values. This approach lets the congestion window get very large (i.e. tens of thousands of segments) quickly, and to stay large, without requiring that the network function for long periods of time without a single dropped packet.



----- Mail original -----

De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag> 
À: ceph-devel@vger.kernel.org 
Envoyé: Lundi 18 Juin 2012 09:40:47 
Objet: stable 10GBE under 3.4.2 or 3.5.0-rc2 

Hi list, 

i've still problems with stable network speed under recent kernels. With 
3.0.32 i get stable 9.90 Gbit/s in both directions. With 3.4.2 or 
3.5.0-rc2 it drops sometimes down to around 1 Gbit/s. Sadly i've no idea 
when this happen? It works sometimes for minutes at 9.9Gbit/s and then 
suddently only with 3-4Gbit/s or even 1Gbit/s using recent kernels. 

I'm using various tunings recommanded by intel (Improving Performance) 
=> http://downloadmirror.intel.com/5874/eng/README.txt 

Has anybody a hint for me or additional settings intel does not mention? 

Stefan 
-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 



-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 
	
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: stable 10GBE under 3.4.2 or 3.5.0-rc2

[Stefan Priebe - Profihost AG]

Am 18.06.2012 10:11, schrieb Alexandre DERUMIER:
> maybe try to play with net.ipv4.tcp_congestion_control ?
The default on my machines and even under RHEL6 is cubic. But i've now 
alos tried, reno, bic and highspeed. But it doesn't change anything.

Everything is fine under 3.0.32 and pretty bad under 3.4.X or 3.5rc2.

Stefan

Re: stable 10GBE under 3.4.2 or 3.5.0-rc2

[Alexandre DERUMIER]

Did you use same driver version with the differents kernel version ?


Maybe try to  desactive tso,gso,... with ethtool ?


----- Mail original ----- 

De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Lundi 18 Juin 2012 11:11:16 
Objet: Re: stable 10GBE under 3.4.2 or 3.5.0-rc2 

Am 18.06.2012 10:11, schrieb Alexandre DERUMIER: 
> maybe try to play with net.ipv4.tcp_congestion_control ? 
The default on my machines and even under RHEL6 is cubic. But i've now 
alos tried, reno, bic and highspeed. But it doesn't change anything. 

Everything is fine under 3.0.32 and pretty bad under 3.4.X or 3.5rc2. 

Stefan 




--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: stable 10GBE under 3.4.2 or 3.5.0-rc2

[Stefan Priebe - Profihost AG]

Am 18.06.2012 11:21, schrieb Alexandre DERUMIER:
> Did you use same driver version with the differents kernel version ?

Yes driver: ixgbe version: 3.9.17-NAPI.

> Maybe try to  desactive tso,gso,... with ethtool ?

Already tried that no change and remember it works fine with 3.0.32.

Stefan

Re: stable 10GBE under 3.4.2 or 3.5.0-rc2

[Alexandre DERUMIER]

also , I see a new feature in 3.3

byte_queue_limits 

1.5. Bufferbloat fighting: Byte queue limits

https://lwn.net/Articles/454390/

"Bufferbloat" is a term used to describe the latency and throughput problems caused by excessive buffering trough the several elements of a network connection. Some tools are being developed to help to alleviate these problems, and this feature is one of them.

Byte queue limits are a configurable limit of packet data that can be put in the transmission queue of a network device. As a result one can tune things such that high priority packets get serviced with a reasonable amount of latency whilst not subjecting the hardware queue to emptying when data is available to send. Configuration of the queue limits is in the tx-<n> sysfs directory for the queue under the byte_queue_limits directory.

I see some bug report about spike:

http://www.mail-archive.com/e1000-devel@lists.sourceforge.net/msg05538.html



maybe can you try to play with values in

/sys/class/net/eth0/queues/tx-0/byte_queue_limits/*

----- Mail original ----- 

De: "Alexandre DERUMIER" <aderumier@odiso.com> 
À: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Lundi 18 Juin 2012 11:21:09 
Objet: Re: stable 10GBE under 3.4.2 or 3.5.0-rc2 

Did you use same driver version with the differents kernel version ? 


Maybe try to desactive tso,gso,... with ethtool ? 


----- Mail original ----- 

De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag> 
À: "Alexandre DERUMIER" <aderumier@odiso.com> 
Cc: ceph-devel@vger.kernel.org 
Envoyé: Lundi 18 Juin 2012 11:11:16 
Objet: Re: stable 10GBE under 3.4.2 or 3.5.0-rc2 

Am 18.06.2012 10:11, schrieb Alexandre DERUMIER: 
> maybe try to play with net.ipv4.tcp_congestion_control ? 
The default on my machines and even under RHEL6 is cubic. But i've now 
alos tried, reno, bic and highspeed. But it doesn't change anything. 

Everything is fine under 3.0.32 and pretty bad under 3.4.X or 3.5rc2. 

Stefan 







-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 
	
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: stable 10GBE under 3.4.2 or 3.5.0-rc2

[Mark Nelson]

On 6/18/12 2:40 AM, Stefan Priebe - Profihost AG wrote:
> Hi list,
>
> i've still problems with stable network speed under recent kernels. With
> 3.0.32 i get stable 9.90 Gbit/s in both directions. With 3.4.2 or
> 3.5.0-rc2 it drops sometimes down to around 1 Gbit/s. Sadly i've no idea
> when this happen? It works sometimes for minutes at 9.9Gbit/s and then
> suddently only with 3-4Gbit/s or even 1Gbit/s using recent kernels.
>
> I'm using various tunings recommanded by intel (Improving Performance)
> => http://downloadmirror.intel.com/5874/eng/README.txt
>

Hi Stefan,

Did you ever get a chance to talk to Jason Wang about the commit that 
was causing the problems?  It might be a good idea to report all of this 
upstream and see what they have to say.

Mark

Re: stable 10GBE under 3.4.2 or 3.5.0-rc2

[Stefan Priebe - Profihost AG]

Am 18.06.2012 15:56, schrieb Mark Nelson:
> On 6/18/12 2:40 AM, Stefan Priebe - Profihost AG wrote:
>> Hi list,
>>
>> i've still problems with stable network speed under recent kernels. With
>> 3.0.32 i get stable 9.90 Gbit/s in both directions. With 3.4.2 or
>> 3.5.0-rc2 it drops sometimes down to around 1 Gbit/s. Sadly i've no idea
>> when this happen? It works sometimes for minutes at 9.9Gbit/s and then
>> suddently only with 3-4Gbit/s or even 1Gbit/s using recent kernels.
>>
>> I'm using various tunings recommanded by intel (Improving Performance)
>> => http://downloadmirror.intel.com/5874/eng/README.txt
>>
>
> Hi Stefan,
>
> Did you ever get a chance to talk to Jason Wang about the commit that
> was causing the problems? It might be a good idea to report all of this
> upstream and see what they have to say.

Hi Mark,

yes i talked to him and he told me that his change was partically 
removed in more recent kernel version.

Right now i'm in discussion with eric dumazet at the netdev kernel 
mailinglist. And he gives me some good advices which seem to work well.

I'm still testing and will share the results here.

Stefan