under-performing bonded interfaces

under-performing bonded interfaces

[Simon Chen]

Hello,

I am bonding two 10G interfaces (ixgbe driver) under Debian 6.0.2. The
bonded interface for some reason can only achieve 12Gbps aggregated
throughput. If a single NIC is used, I can get close to 10Gbps.

I've tried different bonding modes (balance-xor, 802.3ad, balance-alb,
balance-tlb), and different xmit hashing policy (layer2, layer2+3,
layer3+4). I've increased all types of kernel parameters for TCP. MTU
on the physical and bonded interface is set to 8000 and 9000. The MTU
on the switch is 9200+.

Instead of nperf (a single server), I also tried my own TCP sender and
receivers.

All those done, still only 12Gbps... How can I really achieve close to 20Gbps?

(I also tried cutting loose the switch in between, and also getting
12G, so not an issue with the switch.)

Thanks.
-Simon

Re: under-performing bonded interfaces

[Ben Greear]

On 11/16/2011 03:44 PM, Simon Chen wrote:
> Hello,
>
> I am bonding two 10G interfaces (ixgbe driver) under Debian 6.0.2. The
> bonded interface for some reason can only achieve 12Gbps aggregated
> throughput. If a single NIC is used, I can get close to 10Gbps.
>
> I've tried different bonding modes (balance-xor, 802.3ad, balance-alb,
> balance-tlb), and different xmit hashing policy (layer2, layer2+3,
> layer3+4). I've increased all types of kernel parameters for TCP. MTU
> on the physical and bonded interface is set to 8000 and 9000. The MTU
> on the switch is 9200+.
>
> Instead of nperf (a single server), I also tried my own TCP sender and
> receivers.
>
> All those done, still only 12Gbps... How can I really achieve close to 20Gbps?
>
> (I also tried cutting loose the switch in between, and also getting
> 12G, so not an issue with the switch.)

How much can you get if you run each of the NIC ports independently
w/out bonding?  Plz send the 'dmesg' messages about ixgbe (ie, how
many lanes, how many GT/s).  What is your processor?

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: under-performing bonded interfaces

[Simon Chen]

If used independently, I can get around 9.8Gbps.

Here is from dmesg:
[   11.386736] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver -
version 3.2.9-k2
[   11.386738] ixgbe: Copyright (c) 1999-2010 Intel Corporation.
[   11.386778] ixgbe 0000:03:00.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
[   11.386788] ixgbe 0000:03:00.0: setting latency timer to 64
[   11.572715] ixgbe 0000:03:00.0: irq 75 for MSI/MSI-X
[   11.572720] ixgbe 0000:03:00.0: irq 76 for MSI/MSI-X
[   11.572728] ixgbe 0000:03:00.0: irq 77 for MSI/MSI-X
[   11.572733] ixgbe 0000:03:00.0: irq 78 for MSI/MSI-X
[   11.572737] ixgbe 0000:03:00.0: irq 79 for MSI/MSI-X
[   11.572741] ixgbe 0000:03:00.0: irq 80 for MSI/MSI-X
[   11.572745] ixgbe 0000:03:00.0: irq 81 for MSI/MSI-X
[   11.572750] ixgbe 0000:03:00.0: irq 82 for MSI/MSI-X
[   11.572755] ixgbe 0000:03:00.0: irq 83 for MSI/MSI-X
[   11.572759] ixgbe 0000:03:00.0: irq 84 for MSI/MSI-X
[   11.572766] ixgbe 0000:03:00.0: irq 85 for MSI/MSI-X
[   11.572770] ixgbe 0000:03:00.0: irq 86 for MSI/MSI-X
[   11.572775] ixgbe 0000:03:00.0: irq 87 for MSI/MSI-X
[   11.572779] ixgbe 0000:03:00.0: irq 88 for MSI/MSI-X
[   11.572783] ixgbe 0000:03:00.0: irq 89 for MSI/MSI-X
[   11.572787] ixgbe 0000:03:00.0: irq 90 for MSI/MSI-X
[   11.572791] ixgbe 0000:03:00.0: irq 91 for MSI/MSI-X
[   11.572795] ixgbe 0000:03:00.0: irq 92 for MSI/MSI-X
[   11.572799] ixgbe 0000:03:00.0: irq 93 for MSI/MSI-X
[   11.572803] ixgbe 0000:03:00.0: irq 94 for MSI/MSI-X
[   11.572807] ixgbe 0000:03:00.0: irq 95 for MSI/MSI-X
[   11.572812] ixgbe 0000:03:00.0: irq 96 for MSI/MSI-X
[   11.572816] ixgbe 0000:03:00.0: irq 97 for MSI/MSI-X
[   11.572820] ixgbe 0000:03:00.0: irq 98 for MSI/MSI-X
[   11.572825] ixgbe 0000:03:00.0: irq 99 for MSI/MSI-X
[   11.572857] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count
= 24, Tx Queue count = 24
[   11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4)
e8:9a:8f:23:42:1a
[   11.572943] ixgbe 0000:03:00.0: MAC: 2, PHY: 8, SFP+: 3, PBA No: FFFFFF-0FF
[   11.572944] ixgbe 0000:03:00.0: PCI-Express bandwidth available for
this card is not sufficient for optimal performance.
[   11.572946] ixgbe 0000:03:00.0: For optimal performance a x8
PCI-Express slot is required.
[   11.573815] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[   11.573833] ixgbe 0000:03:00.1: PCI INT B -> GSI 34 (level, low) -> IRQ 34
[   11.573839] ixgbe 0000:03:00.1: setting latency timer to 64
[   11.743748] ixgbe 0000:03:00.1: irq 100 for MSI/MSI-X
[   11.743753] ixgbe 0000:03:00.1: irq 101 for MSI/MSI-X
[   11.743758] ixgbe 0000:03:00.1: irq 102 for MSI/MSI-X
[   11.743762] ixgbe 0000:03:00.1: irq 103 for MSI/MSI-X
[   11.743769] ixgbe 0000:03:00.1: irq 104 for MSI/MSI-X
[   11.743773] ixgbe 0000:03:00.1: irq 105 for MSI/MSI-X
[   11.743777] ixgbe 0000:03:00.1: irq 106 for MSI/MSI-X
[   11.743781] ixgbe 0000:03:00.1: irq 107 for MSI/MSI-X
[   11.743785] ixgbe 0000:03:00.1: irq 108 for MSI/MSI-X
[   11.743789] ixgbe 0000:03:00.1: irq 109 for MSI/MSI-X
[   11.743793] ixgbe 0000:03:00.1: irq 110 for MSI/MSI-X
[   11.743796] ixgbe 0000:03:00.1: irq 111 for MSI/MSI-X
[   11.743800] ixgbe 0000:03:00.1: irq 112 for MSI/MSI-X
[   11.743804] ixgbe 0000:03:00.1: irq 113 for MSI/MSI-X
[   11.743808] ixgbe 0000:03:00.1: irq 114 for MSI/MSI-X
[   11.743815] ixgbe 0000:03:00.1: irq 115 for MSI/MSI-X
[   11.743819] ixgbe 0000:03:00.1: irq 116 for MSI/MSI-X
[   11.743823] ixgbe 0000:03:00.1: irq 117 for MSI/MSI-X
[   11.743827] ixgbe 0000:03:00.1: irq 118 for MSI/MSI-X
[   11.743831] ixgbe 0000:03:00.1: irq 119 for MSI/MSI-X
[   11.743835] ixgbe 0000:03:00.1: irq 120 for MSI/MSI-X
[   11.743839] ixgbe 0000:03:00.1: irq 121 for MSI/MSI-X
[   11.743843] ixgbe 0000:03:00.1: irq 122 for MSI/MSI-X
[   11.743847] ixgbe 0000:03:00.1: irq 123 for MSI/MSI-X
[   11.743851] ixgbe 0000:03:00.1: irq 124 for MSI/MSI-X
[   11.743882] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count
= 24, Tx Queue count = 24
[   11.743886] ixgbe 0000:03:00.1: (PCI Express:5.0Gb/s:Width x4)
e8:9a:8f:23:42:1b

I have a 24 cores (just showing the last one).

processor       : 23
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
stepping        : 2
cpu MHz         : 2668.000
cache size      : 12288 KB
physical id     : 1
siblings        : 12
core id         : 10
cpu cores       : 6
apicid          : 53
initial apicid  : 53
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64
monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2
popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept
vpid
bogomips        : 5333.52
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

thanks!
-Simon

On Wed, Nov 16, 2011 at 7:01 PM, Ben Greear <greearb@candelatech.com> wrote:
> On 11/16/2011 03:44 PM, Simon Chen wrote:
>>
>> Hello,
>>
>> I am bonding two 10G interfaces (ixgbe driver) under Debian 6.0.2. The
>> bonded interface for some reason can only achieve 12Gbps aggregated
>> throughput. If a single NIC is used, I can get close to 10Gbps.
>>
>> I've tried different bonding modes (balance-xor, 802.3ad, balance-alb,
>> balance-tlb), and different xmit hashing policy (layer2, layer2+3,
>> layer3+4). I've increased all types of kernel parameters for TCP. MTU
>> on the physical and bonded interface is set to 8000 and 9000. The MTU
>> on the switch is 9200+.
>>
>> Instead of nperf (a single server), I also tried my own TCP sender and
>> receivers.
>>
>> All those done, still only 12Gbps... How can I really achieve close to
>> 20Gbps?
>>
>> (I also tried cutting loose the switch in between, and also getting
>> 12G, so not an issue with the switch.)
>
> How much can you get if you run each of the NIC ports independently
> w/out bonding?  Plz send the 'dmesg' messages about ixgbe (ie, how
> many lanes, how many GT/s).  What is your processor?
>
> Thanks,
> Ben
>
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
>
>

Re: under-performing bonded interfaces

[Ben Greear]

On 11/16/2011 04:05 PM, Simon Chen wrote:
> If used independently, I can get around 9.8Gbps.

On each of them concurrently, or just one at a time?

> [   11.572857] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count
> = 24, Tx Queue count = 24
> [   11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4)
> e8:9a:8f:23:42:1a
> [   11.572943] ixgbe 0000:03:00.0: MAC: 2, PHY: 8, SFP+: 3, PBA No: FFFFFF-0FF
> [   11.572944] ixgbe 0000:03:00.0: PCI-Express bandwidth available for
> this card is not sufficient for optimal performance.
> [   11.572946] ixgbe 0000:03:00.0: For optimal performance a x8
> PCI-Express slot is required.

Looks to me like it may be hardware related unless you are certain you
can get both NICs running at high-speed independently of any bonding
code...

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: under-performing bonded interfaces

[Ben Hutchings]

On Wed, 2011-11-16 at 19:05 -0500, Simon Chen wrote:
> If used independently, I can get around 9.8Gbps.
[...]
> [   11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4) e8:9a:8f:23:42:1a
[...]

You need a PCIe 2.0 x8 slot to run 2 10 Gb ports at full speed.  An x4
slot is only good for about 12-13 Gb aggregate throughput (dependent on
packet sizes and other details).

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: under-performing bonded interfaces

[Simon Chen]

Thanks, Ben. That's good discovery...

Are you saying that both 10G NICs are on the same PCIe x4 slot, so
that they're subject to the 12G throughput bottleneck?

I'm gonna verify this with the hardware vendor...

Thanks.
-Simon

On Wed, Nov 16, 2011 at 7:57 PM, Ben Hutchings
<bhutchings@solarflare.com> wrote:
> On Wed, 2011-11-16 at 19:05 -0500, Simon Chen wrote:
>> If used independently, I can get around 9.8Gbps.
> [...]
>> [   11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4) e8:9a:8f:23:42:1a
> [...]
>
> You need a PCIe 2.0 x8 slot to run 2 10 Gb ports at full speed.  An x4
> slot is only good for about 12-13 Gb aggregate throughput (dependent on
> packet sizes and other details).
>
> Ben.
>
> --
> Ben Hutchings, Staff Engineer, Solarflare
> Not speaking for my employer; that's the marketing department's job.
> They asked us to note that Solarflare product names are trademarked.
>
>

Re: under-performing bonded interfaces

[Simon Chen]

Emm, actually two different PCIe slots. But one slot is only
5.0Gbps... Seem pretty clear from the message below, wasn't reading
thoroughly :-(

[   11.572857] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count
= 24, Tx Queue count = 24
[   11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4)
e8:9a:8f:23:42:1a
[   11.572943] ixgbe 0000:03:00.0: MAC: 2, PHY: 8, SFP+: 3, PBA No: FFFFFF-0FF
[   11.572944] ixgbe 0000:03:00.0: PCI-Express bandwidth available for
this card is not sufficient for optimal performance.
[   11.572946] ixgbe 0000:03:00.0: For optimal performance a x8
PCI-Express slot is required.
[   11.573815] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
[   11.573833] ixgbe 0000:03:00.1: PCI INT B -> GSI 34 (level, low) -> IRQ 34
[   11.573839] ixgbe 0000:03:00.1: setting latency timer to 64



On Wed, Nov 16, 2011 at 8:38 PM, Simon Chen <simonchennj@gmail.com> wrote:
> Thanks, Ben. That's good discovery...
>
> Are you saying that both 10G NICs are on the same PCIe x4 slot, so
> that they're subject to the 12G throughput bottleneck?
>
> I'm gonna verify this with the hardware vendor...
>
> Thanks.
> -Simon
>
> On Wed, Nov 16, 2011 at 7:57 PM, Ben Hutchings
> <bhutchings@solarflare.com> wrote:
>> On Wed, 2011-11-16 at 19:05 -0500, Simon Chen wrote:
>>> If used independently, I can get around 9.8Gbps.
>> [...]
>>> [   11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4) e8:9a:8f:23:42:1a
>> [...]
>>
>> You need a PCIe 2.0 x8 slot to run 2 10 Gb ports at full speed.  An x4
>> slot is only good for about 12-13 Gb aggregate throughput (dependent on
>> packet sizes and other details).
>>
>> Ben.
>>
>> --
>> Ben Hutchings, Staff Engineer, Solarflare
>> Not speaking for my employer; that's the marketing department's job.
>> They asked us to note that Solarflare product names are trademarked.
>>
>>
>

Re: under-performing bonded interfaces

[Rick Jones]

On 11/16/2011 05:38 PM, Simon Chen wrote:
> Thanks, Ben. That's good discovery...
>
> Are you saying that both 10G NICs are on the same PCIe x4 slot, so
> that they're subject to the 12G throughput bottleneck?
>
> I'm gonna verify this with the hardware vendor...

Look at the PCI addresses - they are:

0000:03:00.0
0000:03:00.1

The numbers after the "dot" are PCI function numbers, meaning that both 
*ports* of the dual-port NIC are in the same PCIe slot, in this case a 
x4 slot.

more generally, the format is:

DDDD:BB:SS.F

D == DOMAIN; B == BUS; S == SLOT; F == FUNCTION.  If domain, bus and 
slot are the same, but functions differ, the "devices" are in the same slot.

rick jones

Re: under-performing bonded interfaces

[Ben Hutchings]

On Wed, 2011-11-16 at 20:38 -0500, Simon Chen wrote:
> Thanks, Ben. That's good discovery...
> 
> Are you saying that both 10G NICs are on the same PCIe x4 slot, so
> that they're subject to the 12G throughput bottleneck?

I assumed you were using 2 ports on the same board, i.e. the same slot.
If you were using 1 port each of 2 boards then I would have expected
them both to be usable at full speed.  So far as I can remember, PCIe
bridges are usually set up so there isn't contention for bandwidth
between slots.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: under-performing bonded interfaces

[Simon Chen]

Hi folks,

I added an Intel X520 card to both the sender and receiver... Now I
have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of
the PCI bus shouldn't be the bottleneck.

Now the throughput test gives me around 16Gbps in aggregate. Any ideas
how I can push closer to 20G? I don't quite understand where the
bottleneck is now.

Thanks.
-Simon

On Wed, Nov 16, 2011 at 9:51 PM, Ben Hutchings
<bhutchings@solarflare.com> wrote:
> On Wed, 2011-11-16 at 20:38 -0500, Simon Chen wrote:
>> Thanks, Ben. That's good discovery...
>>
>> Are you saying that both 10G NICs are on the same PCIe x4 slot, so
>> that they're subject to the 12G throughput bottleneck?
>
> I assumed you were using 2 ports on the same board, i.e. the same slot.
> If you were using 1 port each of 2 boards then I would have expected
> them both to be usable at full speed.  So far as I can remember, PCIe
> bridges are usually set up so there isn't contention for bandwidth
> between slots.
>
> Ben.
>
> --
> Ben Hutchings, Staff Engineer, Solarflare
> Not speaking for my employer; that's the marketing department's job.
> They asked us to note that Solarflare product names are trademarked.
>

Re: under-performing bonded interfaces

[Stephen Hemminger]

On Wed, 21 Dec 2011 20:26:04 -0500
Simon Chen <simonchennj@gmail.com> wrote:

> Hi folks,
> 
> I added an Intel X520 card to both the sender and receiver... Now I
> have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of
> the PCI bus shouldn't be the bottleneck.
> 
> Now the throughput test gives me around 16Gbps in aggregate. Any ideas
> how I can push closer to 20G? I don't quite understand where the
> bottleneck is now.

In my experience, Intel dual port cards can not run at full speed
when both ports are in use. You need separate slots to hit full
line rate.

Re: under-performing bonded interfaces

[Simon Chen]

Just a bit more info...
I have 24 cores, and the interrupts for each 10G NIC are distributed
to all 24 cores...
I am using the most recent 3.7.17 driver ixgbe driver from Intel.
layer3+4 xmit policy.

-Simon

On Wed, Dec 21, 2011 at 8:26 PM, Simon Chen <simonchennj@gmail.com> wrote:
> Hi folks,
>
> I added an Intel X520 card to both the sender and receiver... Now I
> have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of
> the PCI bus shouldn't be the bottleneck.
>
> Now the throughput test gives me around 16Gbps in aggregate. Any ideas
> how I can push closer to 20G? I don't quite understand where the
> bottleneck is now.
>
> Thanks.
> -Simon
>
> On Wed, Nov 16, 2011 at 9:51 PM, Ben Hutchings
> <bhutchings@solarflare.com> wrote:
>> On Wed, 2011-11-16 at 20:38 -0500, Simon Chen wrote:
>>> Thanks, Ben. That's good discovery...
>>>
>>> Are you saying that both 10G NICs are on the same PCIe x4 slot, so
>>> that they're subject to the 12G throughput bottleneck?
>>
>> I assumed you were using 2 ports on the same board, i.e. the same slot.
>> If you were using 1 port each of 2 boards then I would have expected
>> them both to be usable at full speed.  So far as I can remember, PCIe
>> bridges are usually set up so there isn't contention for bandwidth
>> between slots.
>>
>> Ben.
>>
>> --
>> Ben Hutchings, Staff Engineer, Solarflare
>> Not speaking for my employer; that's the marketing department's job.
>> They asked us to note that Solarflare product names are trademarked.
>>

Re: under-performing bonded interfaces

[Ben Greear]

On 12/21/2011 05:36 PM, Stephen Hemminger wrote:
> On Wed, 21 Dec 2011 20:26:04 -0500
> Simon Chen<simonchennj@gmail.com>  wrote:
>
>> Hi folks,
>>
>> I added an Intel X520 card to both the sender and receiver... Now I
>> have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of
>> the PCI bus shouldn't be the bottleneck.
>>
>> Now the throughput test gives me around 16Gbps in aggregate. Any ideas
>> how I can push closer to 20G? I don't quite understand where the
>> bottleneck is now.
>
> In my experience, Intel dual port cards can not run at full speed
> when both ports are in use. You need separate slots to hit full
> line rate.

We can run 2 ports right at 10Gbps tx + rx using a core-i7 980x
processor and 5gt/s pci-e bus.  This is using a modified version of
pktgen to generate traffic.  We can only push around 6 Gbps tx + rx
when generating tcp traffic to/from user-space, but our tcp generator
is not as optimized for bulk transfer as it could be.

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: under-performing bonded interfaces

[Eric Dumazet]

Le mercredi 21 décembre 2011 à 20:26 -0500, Simon Chen a écrit :
> Hi folks,
> 
> I added an Intel X520 card to both the sender and receiver... Now I
> have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of
> the PCI bus shouldn't be the bottleneck.
> 
> Now the throughput test gives me around 16Gbps in aggregate. Any ideas
> how I can push closer to 20G? I don't quite understand where the
> bottleneck is now.

Could you post some "perf top" or "perf record / report" numbers ?

Re: under-performing bonded interfaces

[Simon Chen]

It's a funny thing again... I left the bandwidth test running
over-night - I used 16 simple senders and receivers.

The bandwidth slowly climbed from around 16G to 19G, which is much
better. I suspect two causes: 1) regular tcp implementation that has
slow-start and aggressive back-off; 2) user-land application...

-Simon

On Thu, Dec 22, 2011 at 12:43 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le mercredi 21 décembre 2011 à 20:26 -0500, Simon Chen a écrit :
>> Hi folks,
>>
>> I added an Intel X520 card to both the sender and receiver... Now I
>> have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of
>> the PCI bus shouldn't be the bottleneck.
>>
>> Now the throughput test gives me around 16Gbps in aggregate. Any ideas
>> how I can push closer to 20G? I don't quite understand where the
>> bottleneck is now.
>
> Could you post some "perf top" or "perf record / report" numbers ?
>
>
>