ixgbe unstable performance at 1Gb/s

ixgbe unstable performance at 1Gb/s

[Charles]

Hi,

I hope I am posting on the right mailing-list. If not, sorry; please redirect me
to the right place. Thanks.

I have a new motherboard with integrated Intel X540 10GBase-T. For now, I want
to use it at 1Gb/s.

The bandwidth is only of ~300 Mbit/s (with Iperf). It's actually very unstable
(always varies between 100 to 800 Mbit/s during the transfer).

If I do "ethtool -K eth1 ntuple on", things are a bit better but still very
unstable bandwidth. If I add the kernel parameter "pci=nomsi", everything is
fine (and normal, with the same performance, ~950 MBit/s, as a 1Gbit/s NIC).

I also run the set_irq_affinity.sh script, w/o noticing a difference.

The only weird thing I have found is with ethtoll -S eth1:

    tx_queue_0_packets: 304262
     tx_queue_1_packets: 0
     tx_queue_2_packets: 0
     tx_queue_3_packets: 1
     tx_queue_4_packets: 502939
     tx_queue_5_packets: 677967
     tx_queue_6_packets: 459869
     tx_queue_7_packets: 0
     tx_queue_8_packets: 0
     tx_queue_9_packets: 0
     tx_queue_10_packets: 632626
     tx_queue_11_packets: 1719870
     tx_queue_12_packets: 0
     tx_queue_13_packets: 449811
     tx_queue_14_packets: 0
.......

I am using ArchLinux with kernel 3.5.4. Please ask for more details.

Any help would be appreciated. Thanks

Charles

RE: ixgbe unstable performance at 1Gb/s

[Tantilov, Emil S]

>-----Original Message-----
>From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On
>Behalf Of Charles
>Sent: Monday, September 24, 2012 10:47 AM
>To: netdev@vger.kernel.org
>Subject: ixgbe unstable performance at 1Gb/s
>
>Hi,
>
>I hope I am posting on the right mailing-list. If not, sorry; please
>redirect me
>to the right place. Thanks.
>
>I have a new motherboard with integrated Intel X540 10GBase-T. For now, I
>want
>to use it at 1Gb/s.
>
>The bandwidth is only of ~300 Mbit/s (with Iperf). It's actually very
>unstable
>(always varies between 100 to 800 Mbit/s during the transfer).

Do you by any chance have CONFIG_IXGBE_PTP set in your kernel config?

If so, try disabling it and see if it fixes your performance. 

Thanks,
Emil

Re: ixgbe unstable performance at 1Gb/s

[Charles Vejnar]

Le 25/09/2012 19:58, Tantilov, Emil S a écrit :
>> -----Original Message-----
>> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On
>> Behalf Of Charles
>> Sent: Monday, September 24, 2012 10:47 AM
>> To: netdev@vger.kernel.org
>> Subject: ixgbe unstable performance at 1Gb/s
>>
>> Hi,
>>
>> I hope I am posting on the right mailing-list. If not, sorry; please
>> redirect me
>> to the right place. Thanks.
>>
>> I have a new motherboard with integrated Intel X540 10GBase-T. For now, I
>> want
>> to use it at 1Gb/s.
>>
>> The bandwidth is only of ~300 Mbit/s (with Iperf). It's actually very
>> unstable
>> (always varies between 100 to 800 Mbit/s during the transfer).
> Do you by any chance have CONFIG_IXGBE_PTP set in your kernel config?
>
> If so, try disabling it and see if it fixes your performance.
>
> Thanks,
> Emil
>
Hi,

Thanks for your reply.

I compiled manually the ixgbe module with the default options of my 
distribution (Archlinux). I had the same problem.

I then changed the CONFIG_IXGBE_PTP to no (default is yes) as you 
suggested, and recompiled. The problem disappeared; normal transfer.

Could you please explain why this PTP is causing a problem? How can this 
be fixed without having to recompile the module (ethtool, /sys, bios... 
)? Thanks

Regards,

Charles

RE: ixgbe unstable performance at 1Gb/s

[Tantilov, Emil S]

>-----Original Message-----
>From: Charles Vejnar [mailto:Charles.Vejnar@unige.ch]
>Sent: Wednesday, September 26, 2012 11:33 AM
>To: Tantilov, Emil S; netdev@vger.kernel.org
>Subject: Re: ixgbe unstable performance at 1Gb/s
>
>Le 25/09/2012 19:58, Tantilov, Emil S a écrit :
>>> -----Original Message-----
>>> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
>On
>>> Behalf Of Charles
>>> Sent: Monday, September 24, 2012 10:47 AM
>>> To: netdev@vger.kernel.org
>>> Subject: ixgbe unstable performance at 1Gb/s
>>>
>>> Hi,
>>>
>>> I hope I am posting on the right mailing-list. If not, sorry; please
>>> redirect me
>>> to the right place. Thanks.
>>>
>>> I have a new motherboard with integrated Intel X540 10GBase-T. For now,
>I
>>> want
>>> to use it at 1Gb/s.
>>>
>>> The bandwidth is only of ~300 Mbit/s (with Iperf). It's actually very
>>> unstable
>>> (always varies between 100 to 800 Mbit/s during the transfer).
>> Do you by any chance have CONFIG_IXGBE_PTP set in your kernel config?
>>
>> If so, try disabling it and see if it fixes your performance.
>>
>> Thanks,
>> Emil
>>
>Hi,
>
>Thanks for your reply.
>
>I compiled manually the ixgbe module with the default options of my
>distribution (Archlinux). I had the same problem.
>
>I then changed the CONFIG_IXGBE_PTP to no (default is yes) as you
>suggested, and recompiled. The problem disappeared; normal transfer.
>
>Could you please explain why this PTP is causing a problem? How can this
>be fixed without having to recompile the module (ethtool, /sys, bios...
>)? Thanks

This is actually a bug in the driver. We should have a patch out very soon to address it.

>
>Regards,
>
>Charles

Thanks,
Emil

Re: ixgbe unstable performance at 1Gb/s

[Charles Vejnar]

Le 26/09/2012 22:22, Tantilov, Emil S a écrit :
>> -----Original Message-----
>> From: Charles Vejnar [mailto:Charles.Vejnar@unige.ch]
>> Sent: Wednesday, September 26, 2012 11:33 AM
>> To: Tantilov, Emil S; netdev@vger.kernel.org
>> Subject: Re: ixgbe unstable performance at 1Gb/s
>>
>> Le 25/09/2012 19:58, Tantilov, Emil S a écrit :
>>>> -----Original Message-----
>>>> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
>> On
>>>> Behalf Of Charles
>>>> Sent: Monday, September 24, 2012 10:47 AM
>>>> To: netdev@vger.kernel.org
>>>> Subject: ixgbe unstable performance at 1Gb/s
>>>>
>>>> Hi,
>>>>
>>>> I hope I am posting on the right mailing-list. If not, sorry; please
>>>> redirect me
>>>> to the right place. Thanks.
>>>>
>>>> I have a new motherboard with integrated Intel X540 10GBase-T. For now,
>> I
>>>> want
>>>> to use it at 1Gb/s.
>>>>
>>>> The bandwidth is only of ~300 Mbit/s (with Iperf). It's actually very
>>>> unstable
>>>> (always varies between 100 to 800 Mbit/s during the transfer).
>>> Do you by any chance have CONFIG_IXGBE_PTP set in your kernel config?
>>>
>>> If so, try disabling it and see if it fixes your performance.
>>>
>>> Thanks,
>>> Emil
>>>
>> Hi,
>>
>> Thanks for your reply.
>>
>> I compiled manually the ixgbe module with the default options of my
>> distribution (Archlinux). I had the same problem.
>>
>> I then changed the CONFIG_IXGBE_PTP to no (default is yes) as you
>> suggested, and recompiled. The problem disappeared; normal transfer.
>>
>> Could you please explain why this PTP is causing a problem? How can this
>> be fixed without having to recompile the module (ethtool, /sys, bios...
>> )? Thanks
> This is actually a bug in the driver. We should have a patch out very soon to address it.
>
>> Regards,
>>
>> Charles
> Thanks,
> Emil
Hi Emil,

 > This is actually a bug in the driver. We should have a patch out very 
soon to address it.

OK. Please send a reply to this thread when it's available. I can test 
the patch.

Thanks

Charles