* under-performing bonded interfaces @ 2011-11-16 23:44 Simon Chen 2011-11-17 0:01 ` Ben Greear 0 siblings, 1 reply; 15+ messages in thread From: Simon Chen @ 2011-11-16 23:44 UTC (permalink / raw) To: netdev Hello, I am bonding two 10G interfaces (ixgbe driver) under Debian 6.0.2. The bonded interface for some reason can only achieve 12Gbps aggregated throughput. If a single NIC is used, I can get close to 10Gbps. I've tried different bonding modes (balance-xor, 802.3ad, balance-alb, balance-tlb), and different xmit hashing policy (layer2, layer2+3, layer3+4). I've increased all types of kernel parameters for TCP. MTU on the physical and bonded interface is set to 8000 and 9000. The MTU on the switch is 9200+. Instead of nperf (a single server), I also tried my own TCP sender and receivers. All those done, still only 12Gbps... How can I really achieve close to 20Gbps? (I also tried cutting loose the switch in between, and also getting 12G, so not an issue with the switch.) Thanks. -Simon ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-11-16 23:44 under-performing bonded interfaces Simon Chen @ 2011-11-17 0:01 ` Ben Greear 2011-11-17 0:05 ` Simon Chen 0 siblings, 1 reply; 15+ messages in thread From: Ben Greear @ 2011-11-17 0:01 UTC (permalink / raw) To: Simon Chen; +Cc: netdev On 11/16/2011 03:44 PM, Simon Chen wrote: > Hello, > > I am bonding two 10G interfaces (ixgbe driver) under Debian 6.0.2. The > bonded interface for some reason can only achieve 12Gbps aggregated > throughput. If a single NIC is used, I can get close to 10Gbps. > > I've tried different bonding modes (balance-xor, 802.3ad, balance-alb, > balance-tlb), and different xmit hashing policy (layer2, layer2+3, > layer3+4). I've increased all types of kernel parameters for TCP. MTU > on the physical and bonded interface is set to 8000 and 9000. The MTU > on the switch is 9200+. > > Instead of nperf (a single server), I also tried my own TCP sender and > receivers. > > All those done, still only 12Gbps... How can I really achieve close to 20Gbps? > > (I also tried cutting loose the switch in between, and also getting > 12G, so not an issue with the switch.) How much can you get if you run each of the NIC ports independently w/out bonding? Plz send the 'dmesg' messages about ixgbe (ie, how many lanes, how many GT/s). What is your processor? Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-11-17 0:01 ` Ben Greear @ 2011-11-17 0:05 ` Simon Chen 2011-11-17 0:07 ` Ben Greear 2011-11-17 0:57 ` Ben Hutchings 0 siblings, 2 replies; 15+ messages in thread From: Simon Chen @ 2011-11-17 0:05 UTC (permalink / raw) To: Ben Greear; +Cc: netdev If used independently, I can get around 9.8Gbps. Here is from dmesg: [ 11.386736] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.2.9-k2 [ 11.386738] ixgbe: Copyright (c) 1999-2010 Intel Corporation. [ 11.386778] ixgbe 0000:03:00.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24 [ 11.386788] ixgbe 0000:03:00.0: setting latency timer to 64 [ 11.572715] ixgbe 0000:03:00.0: irq 75 for MSI/MSI-X [ 11.572720] ixgbe 0000:03:00.0: irq 76 for MSI/MSI-X [ 11.572728] ixgbe 0000:03:00.0: irq 77 for MSI/MSI-X [ 11.572733] ixgbe 0000:03:00.0: irq 78 for MSI/MSI-X [ 11.572737] ixgbe 0000:03:00.0: irq 79 for MSI/MSI-X [ 11.572741] ixgbe 0000:03:00.0: irq 80 for MSI/MSI-X [ 11.572745] ixgbe 0000:03:00.0: irq 81 for MSI/MSI-X [ 11.572750] ixgbe 0000:03:00.0: irq 82 for MSI/MSI-X [ 11.572755] ixgbe 0000:03:00.0: irq 83 for MSI/MSI-X [ 11.572759] ixgbe 0000:03:00.0: irq 84 for MSI/MSI-X [ 11.572766] ixgbe 0000:03:00.0: irq 85 for MSI/MSI-X [ 11.572770] ixgbe 0000:03:00.0: irq 86 for MSI/MSI-X [ 11.572775] ixgbe 0000:03:00.0: irq 87 for MSI/MSI-X [ 11.572779] ixgbe 0000:03:00.0: irq 88 for MSI/MSI-X [ 11.572783] ixgbe 0000:03:00.0: irq 89 for MSI/MSI-X [ 11.572787] ixgbe 0000:03:00.0: irq 90 for MSI/MSI-X [ 11.572791] ixgbe 0000:03:00.0: irq 91 for MSI/MSI-X [ 11.572795] ixgbe 0000:03:00.0: irq 92 for MSI/MSI-X [ 11.572799] ixgbe 0000:03:00.0: irq 93 for MSI/MSI-X [ 11.572803] ixgbe 0000:03:00.0: irq 94 for MSI/MSI-X [ 11.572807] ixgbe 0000:03:00.0: irq 95 for MSI/MSI-X [ 11.572812] ixgbe 0000:03:00.0: irq 96 for MSI/MSI-X [ 11.572816] ixgbe 0000:03:00.0: irq 97 for MSI/MSI-X [ 11.572820] ixgbe 0000:03:00.0: irq 98 for MSI/MSI-X [ 11.572825] ixgbe 0000:03:00.0: irq 99 for MSI/MSI-X [ 11.572857] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 24, Tx Queue count = 24 [ 11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4) e8:9a:8f:23:42:1a [ 11.572943] ixgbe 0000:03:00.0: MAC: 2, PHY: 8, SFP+: 3, PBA No: FFFFFF-0FF [ 11.572944] ixgbe 0000:03:00.0: PCI-Express bandwidth available for this card is not sufficient for optimal performance. [ 11.572946] ixgbe 0000:03:00.0: For optimal performance a x8 PCI-Express slot is required. [ 11.573815] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection [ 11.573833] ixgbe 0000:03:00.1: PCI INT B -> GSI 34 (level, low) -> IRQ 34 [ 11.573839] ixgbe 0000:03:00.1: setting latency timer to 64 [ 11.743748] ixgbe 0000:03:00.1: irq 100 for MSI/MSI-X [ 11.743753] ixgbe 0000:03:00.1: irq 101 for MSI/MSI-X [ 11.743758] ixgbe 0000:03:00.1: irq 102 for MSI/MSI-X [ 11.743762] ixgbe 0000:03:00.1: irq 103 for MSI/MSI-X [ 11.743769] ixgbe 0000:03:00.1: irq 104 for MSI/MSI-X [ 11.743773] ixgbe 0000:03:00.1: irq 105 for MSI/MSI-X [ 11.743777] ixgbe 0000:03:00.1: irq 106 for MSI/MSI-X [ 11.743781] ixgbe 0000:03:00.1: irq 107 for MSI/MSI-X [ 11.743785] ixgbe 0000:03:00.1: irq 108 for MSI/MSI-X [ 11.743789] ixgbe 0000:03:00.1: irq 109 for MSI/MSI-X [ 11.743793] ixgbe 0000:03:00.1: irq 110 for MSI/MSI-X [ 11.743796] ixgbe 0000:03:00.1: irq 111 for MSI/MSI-X [ 11.743800] ixgbe 0000:03:00.1: irq 112 for MSI/MSI-X [ 11.743804] ixgbe 0000:03:00.1: irq 113 for MSI/MSI-X [ 11.743808] ixgbe 0000:03:00.1: irq 114 for MSI/MSI-X [ 11.743815] ixgbe 0000:03:00.1: irq 115 for MSI/MSI-X [ 11.743819] ixgbe 0000:03:00.1: irq 116 for MSI/MSI-X [ 11.743823] ixgbe 0000:03:00.1: irq 117 for MSI/MSI-X [ 11.743827] ixgbe 0000:03:00.1: irq 118 for MSI/MSI-X [ 11.743831] ixgbe 0000:03:00.1: irq 119 for MSI/MSI-X [ 11.743835] ixgbe 0000:03:00.1: irq 120 for MSI/MSI-X [ 11.743839] ixgbe 0000:03:00.1: irq 121 for MSI/MSI-X [ 11.743843] ixgbe 0000:03:00.1: irq 122 for MSI/MSI-X [ 11.743847] ixgbe 0000:03:00.1: irq 123 for MSI/MSI-X [ 11.743851] ixgbe 0000:03:00.1: irq 124 for MSI/MSI-X [ 11.743882] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 24, Tx Queue count = 24 [ 11.743886] ixgbe 0000:03:00.1: (PCI Express:5.0Gb/s:Width x4) e8:9a:8f:23:42:1b I have a 24 cores (just showing the last one). processor : 23 vendor_id : GenuineIntel cpu family : 6 model : 44 model name : Intel(R) Xeon(R) CPU X5650 @ 2.67GHz stepping : 2 cpu MHz : 2668.000 cache size : 12288 KB physical id : 1 siblings : 12 core id : 10 cpu cores : 6 apicid : 53 initial apicid : 53 fpu : yes fpu_exception : yes cpuid level : 11 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnmi flexpriority ept vpid bogomips : 5333.52 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: thanks! -Simon On Wed, Nov 16, 2011 at 7:01 PM, Ben Greear <greearb@candelatech.com> wrote: > On 11/16/2011 03:44 PM, Simon Chen wrote: >> >> Hello, >> >> I am bonding two 10G interfaces (ixgbe driver) under Debian 6.0.2. The >> bonded interface for some reason can only achieve 12Gbps aggregated >> throughput. If a single NIC is used, I can get close to 10Gbps. >> >> I've tried different bonding modes (balance-xor, 802.3ad, balance-alb, >> balance-tlb), and different xmit hashing policy (layer2, layer2+3, >> layer3+4). I've increased all types of kernel parameters for TCP. MTU >> on the physical and bonded interface is set to 8000 and 9000. The MTU >> on the switch is 9200+. >> >> Instead of nperf (a single server), I also tried my own TCP sender and >> receivers. >> >> All those done, still only 12Gbps... How can I really achieve close to >> 20Gbps? >> >> (I also tried cutting loose the switch in between, and also getting >> 12G, so not an issue with the switch.) > > How much can you get if you run each of the NIC ports independently > w/out bonding? Plz send the 'dmesg' messages about ixgbe (ie, how > many lanes, how many GT/s). What is your processor? > > Thanks, > Ben > > > -- > Ben Greear <greearb@candelatech.com> > Candela Technologies Inc http://www.candelatech.com > > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-11-17 0:05 ` Simon Chen @ 2011-11-17 0:07 ` Ben Greear 2011-11-17 0:57 ` Ben Hutchings 1 sibling, 0 replies; 15+ messages in thread From: Ben Greear @ 2011-11-17 0:07 UTC (permalink / raw) To: Simon Chen; +Cc: netdev On 11/16/2011 04:05 PM, Simon Chen wrote: > If used independently, I can get around 9.8Gbps. On each of them concurrently, or just one at a time? > [ 11.572857] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count > = 24, Tx Queue count = 24 > [ 11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4) > e8:9a:8f:23:42:1a > [ 11.572943] ixgbe 0000:03:00.0: MAC: 2, PHY: 8, SFP+: 3, PBA No: FFFFFF-0FF > [ 11.572944] ixgbe 0000:03:00.0: PCI-Express bandwidth available for > this card is not sufficient for optimal performance. > [ 11.572946] ixgbe 0000:03:00.0: For optimal performance a x8 > PCI-Express slot is required. Looks to me like it may be hardware related unless you are certain you can get both NICs running at high-speed independently of any bonding code... Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-11-17 0:05 ` Simon Chen 2011-11-17 0:07 ` Ben Greear @ 2011-11-17 0:57 ` Ben Hutchings 2011-11-17 1:38 ` Simon Chen 1 sibling, 1 reply; 15+ messages in thread From: Ben Hutchings @ 2011-11-17 0:57 UTC (permalink / raw) To: Simon Chen; +Cc: Ben Greear, netdev On Wed, 2011-11-16 at 19:05 -0500, Simon Chen wrote: > If used independently, I can get around 9.8Gbps. [...] > [ 11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4) e8:9a:8f:23:42:1a [...] You need a PCIe 2.0 x8 slot to run 2 10 Gb ports at full speed. An x4 slot is only good for about 12-13 Gb aggregate throughput (dependent on packet sizes and other details). Ben. -- Ben Hutchings, Staff Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-11-17 0:57 ` Ben Hutchings @ 2011-11-17 1:38 ` Simon Chen 2011-11-17 1:45 ` Simon Chen ` (2 more replies) 0 siblings, 3 replies; 15+ messages in thread From: Simon Chen @ 2011-11-17 1:38 UTC (permalink / raw) To: Ben Hutchings; +Cc: Ben Greear, netdev Thanks, Ben. That's good discovery... Are you saying that both 10G NICs are on the same PCIe x4 slot, so that they're subject to the 12G throughput bottleneck? I'm gonna verify this with the hardware vendor... Thanks. -Simon On Wed, Nov 16, 2011 at 7:57 PM, Ben Hutchings <bhutchings@solarflare.com> wrote: > On Wed, 2011-11-16 at 19:05 -0500, Simon Chen wrote: >> If used independently, I can get around 9.8Gbps. > [...] >> [ 11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4) e8:9a:8f:23:42:1a > [...] > > You need a PCIe 2.0 x8 slot to run 2 10 Gb ports at full speed. An x4 > slot is only good for about 12-13 Gb aggregate throughput (dependent on > packet sizes and other details). > > Ben. > > -- > Ben Hutchings, Staff Engineer, Solarflare > Not speaking for my employer; that's the marketing department's job. > They asked us to note that Solarflare product names are trademarked. > > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-11-17 1:38 ` Simon Chen @ 2011-11-17 1:45 ` Simon Chen 2011-11-17 1:45 ` Rick Jones 2011-11-17 2:51 ` Ben Hutchings 2 siblings, 0 replies; 15+ messages in thread From: Simon Chen @ 2011-11-17 1:45 UTC (permalink / raw) To: Ben Hutchings; +Cc: Ben Greear, netdev Emm, actually two different PCIe slots. But one slot is only 5.0Gbps... Seem pretty clear from the message below, wasn't reading thoroughly :-( [ 11.572857] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 24, Tx Queue count = 24 [ 11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4) e8:9a:8f:23:42:1a [ 11.572943] ixgbe 0000:03:00.0: MAC: 2, PHY: 8, SFP+: 3, PBA No: FFFFFF-0FF [ 11.572944] ixgbe 0000:03:00.0: PCI-Express bandwidth available for this card is not sufficient for optimal performance. [ 11.572946] ixgbe 0000:03:00.0: For optimal performance a x8 PCI-Express slot is required. [ 11.573815] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection [ 11.573833] ixgbe 0000:03:00.1: PCI INT B -> GSI 34 (level, low) -> IRQ 34 [ 11.573839] ixgbe 0000:03:00.1: setting latency timer to 64 On Wed, Nov 16, 2011 at 8:38 PM, Simon Chen <simonchennj@gmail.com> wrote: > Thanks, Ben. That's good discovery... > > Are you saying that both 10G NICs are on the same PCIe x4 slot, so > that they're subject to the 12G throughput bottleneck? > > I'm gonna verify this with the hardware vendor... > > Thanks. > -Simon > > On Wed, Nov 16, 2011 at 7:57 PM, Ben Hutchings > <bhutchings@solarflare.com> wrote: >> On Wed, 2011-11-16 at 19:05 -0500, Simon Chen wrote: >>> If used independently, I can get around 9.8Gbps. >> [...] >>> [ 11.572861] ixgbe 0000:03:00.0: (PCI Express:5.0Gb/s:Width x4) e8:9a:8f:23:42:1a >> [...] >> >> You need a PCIe 2.0 x8 slot to run 2 10 Gb ports at full speed. An x4 >> slot is only good for about 12-13 Gb aggregate throughput (dependent on >> packet sizes and other details). >> >> Ben. >> >> -- >> Ben Hutchings, Staff Engineer, Solarflare >> Not speaking for my employer; that's the marketing department's job. >> They asked us to note that Solarflare product names are trademarked. >> >> > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-11-17 1:38 ` Simon Chen 2011-11-17 1:45 ` Simon Chen @ 2011-11-17 1:45 ` Rick Jones 2011-11-17 2:51 ` Ben Hutchings 2 siblings, 0 replies; 15+ messages in thread From: Rick Jones @ 2011-11-17 1:45 UTC (permalink / raw) To: Simon Chen; +Cc: Ben Hutchings, Ben Greear, netdev On 11/16/2011 05:38 PM, Simon Chen wrote: > Thanks, Ben. That's good discovery... > > Are you saying that both 10G NICs are on the same PCIe x4 slot, so > that they're subject to the 12G throughput bottleneck? > > I'm gonna verify this with the hardware vendor... Look at the PCI addresses - they are: 0000:03:00.0 0000:03:00.1 The numbers after the "dot" are PCI function numbers, meaning that both *ports* of the dual-port NIC are in the same PCIe slot, in this case a x4 slot. more generally, the format is: DDDD:BB:SS.F D == DOMAIN; B == BUS; S == SLOT; F == FUNCTION. If domain, bus and slot are the same, but functions differ, the "devices" are in the same slot. rick jones ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-11-17 1:38 ` Simon Chen 2011-11-17 1:45 ` Simon Chen 2011-11-17 1:45 ` Rick Jones @ 2011-11-17 2:51 ` Ben Hutchings 2011-12-22 1:26 ` Simon Chen 2 siblings, 1 reply; 15+ messages in thread From: Ben Hutchings @ 2011-11-17 2:51 UTC (permalink / raw) To: Simon Chen; +Cc: Ben Greear, netdev On Wed, 2011-11-16 at 20:38 -0500, Simon Chen wrote: > Thanks, Ben. That's good discovery... > > Are you saying that both 10G NICs are on the same PCIe x4 slot, so > that they're subject to the 12G throughput bottleneck? I assumed you were using 2 ports on the same board, i.e. the same slot. If you were using 1 port each of 2 boards then I would have expected them both to be usable at full speed. So far as I can remember, PCIe bridges are usually set up so there isn't contention for bandwidth between slots. Ben. -- Ben Hutchings, Staff Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-11-17 2:51 ` Ben Hutchings @ 2011-12-22 1:26 ` Simon Chen 2011-12-22 1:36 ` Stephen Hemminger ` (2 more replies) 0 siblings, 3 replies; 15+ messages in thread From: Simon Chen @ 2011-12-22 1:26 UTC (permalink / raw) To: Ben Hutchings; +Cc: Ben Greear, netdev Hi folks, I added an Intel X520 card to both the sender and receiver... Now I have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of the PCI bus shouldn't be the bottleneck. Now the throughput test gives me around 16Gbps in aggregate. Any ideas how I can push closer to 20G? I don't quite understand where the bottleneck is now. Thanks. -Simon On Wed, Nov 16, 2011 at 9:51 PM, Ben Hutchings <bhutchings@solarflare.com> wrote: > On Wed, 2011-11-16 at 20:38 -0500, Simon Chen wrote: >> Thanks, Ben. That's good discovery... >> >> Are you saying that both 10G NICs are on the same PCIe x4 slot, so >> that they're subject to the 12G throughput bottleneck? > > I assumed you were using 2 ports on the same board, i.e. the same slot. > If you were using 1 port each of 2 boards then I would have expected > them both to be usable at full speed. So far as I can remember, PCIe > bridges are usually set up so there isn't contention for bandwidth > between slots. > > Ben. > > -- > Ben Hutchings, Staff Engineer, Solarflare > Not speaking for my employer; that's the marketing department's job. > They asked us to note that Solarflare product names are trademarked. > ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-12-22 1:26 ` Simon Chen @ 2011-12-22 1:36 ` Stephen Hemminger 2011-12-22 3:31 ` Ben Greear 2011-12-22 2:28 ` Simon Chen 2011-12-22 5:43 ` Eric Dumazet 2 siblings, 1 reply; 15+ messages in thread From: Stephen Hemminger @ 2011-12-22 1:36 UTC (permalink / raw) To: Simon Chen; +Cc: Ben Hutchings, Ben Greear, netdev On Wed, 21 Dec 2011 20:26:04 -0500 Simon Chen <simonchennj@gmail.com> wrote: > Hi folks, > > I added an Intel X520 card to both the sender and receiver... Now I > have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of > the PCI bus shouldn't be the bottleneck. > > Now the throughput test gives me around 16Gbps in aggregate. Any ideas > how I can push closer to 20G? I don't quite understand where the > bottleneck is now. In my experience, Intel dual port cards can not run at full speed when both ports are in use. You need separate slots to hit full line rate. ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-12-22 1:36 ` Stephen Hemminger @ 2011-12-22 3:31 ` Ben Greear 0 siblings, 0 replies; 15+ messages in thread From: Ben Greear @ 2011-12-22 3:31 UTC (permalink / raw) To: Stephen Hemminger; +Cc: Simon Chen, Ben Hutchings, netdev On 12/21/2011 05:36 PM, Stephen Hemminger wrote: > On Wed, 21 Dec 2011 20:26:04 -0500 > Simon Chen<simonchennj@gmail.com> wrote: > >> Hi folks, >> >> I added an Intel X520 card to both the sender and receiver... Now I >> have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of >> the PCI bus shouldn't be the bottleneck. >> >> Now the throughput test gives me around 16Gbps in aggregate. Any ideas >> how I can push closer to 20G? I don't quite understand where the >> bottleneck is now. > > In my experience, Intel dual port cards can not run at full speed > when both ports are in use. You need separate slots to hit full > line rate. We can run 2 ports right at 10Gbps tx + rx using a core-i7 980x processor and 5gt/s pci-e bus. This is using a modified version of pktgen to generate traffic. We can only push around 6 Gbps tx + rx when generating tcp traffic to/from user-space, but our tcp generator is not as optimized for bulk transfer as it could be. Thanks, Ben -- Ben Greear <greearb@candelatech.com> Candela Technologies Inc http://www.candelatech.com ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-12-22 1:26 ` Simon Chen 2011-12-22 1:36 ` Stephen Hemminger @ 2011-12-22 2:28 ` Simon Chen 2011-12-22 5:43 ` Eric Dumazet 2 siblings, 0 replies; 15+ messages in thread From: Simon Chen @ 2011-12-22 2:28 UTC (permalink / raw) To: Ben Hutchings; +Cc: Ben Greear, netdev Just a bit more info... I have 24 cores, and the interrupts for each 10G NIC are distributed to all 24 cores... I am using the most recent 3.7.17 driver ixgbe driver from Intel. layer3+4 xmit policy. -Simon On Wed, Dec 21, 2011 at 8:26 PM, Simon Chen <simonchennj@gmail.com> wrote: > Hi folks, > > I added an Intel X520 card to both the sender and receiver... Now I > have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of > the PCI bus shouldn't be the bottleneck. > > Now the throughput test gives me around 16Gbps in aggregate. Any ideas > how I can push closer to 20G? I don't quite understand where the > bottleneck is now. > > Thanks. > -Simon > > On Wed, Nov 16, 2011 at 9:51 PM, Ben Hutchings > <bhutchings@solarflare.com> wrote: >> On Wed, 2011-11-16 at 20:38 -0500, Simon Chen wrote: >>> Thanks, Ben. That's good discovery... >>> >>> Are you saying that both 10G NICs are on the same PCIe x4 slot, so >>> that they're subject to the 12G throughput bottleneck? >> >> I assumed you were using 2 ports on the same board, i.e. the same slot. >> If you were using 1 port each of 2 boards then I would have expected >> them both to be usable at full speed. So far as I can remember, PCIe >> bridges are usually set up so there isn't contention for bandwidth >> between slots. >> >> Ben. >> >> -- >> Ben Hutchings, Staff Engineer, Solarflare >> Not speaking for my employer; that's the marketing department's job. >> They asked us to note that Solarflare product names are trademarked. >> ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-12-22 1:26 ` Simon Chen 2011-12-22 1:36 ` Stephen Hemminger 2011-12-22 2:28 ` Simon Chen @ 2011-12-22 5:43 ` Eric Dumazet 2011-12-23 15:03 ` Simon Chen 2 siblings, 1 reply; 15+ messages in thread From: Eric Dumazet @ 2011-12-22 5:43 UTC (permalink / raw) To: Simon Chen; +Cc: Ben Hutchings, Ben Greear, netdev Le mercredi 21 décembre 2011 à 20:26 -0500, Simon Chen a écrit : > Hi folks, > > I added an Intel X520 card to both the sender and receiver... Now I > have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of > the PCI bus shouldn't be the bottleneck. > > Now the throughput test gives me around 16Gbps in aggregate. Any ideas > how I can push closer to 20G? I don't quite understand where the > bottleneck is now. Could you post some "perf top" or "perf record / report" numbers ? ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: under-performing bonded interfaces 2011-12-22 5:43 ` Eric Dumazet @ 2011-12-23 15:03 ` Simon Chen 0 siblings, 0 replies; 15+ messages in thread From: Simon Chen @ 2011-12-23 15:03 UTC (permalink / raw) To: Eric Dumazet; +Cc: Ben Hutchings, Ben Greear, netdev It's a funny thing again... I left the bandwidth test running over-night - I used 16 simple senders and receivers. The bandwidth slowly climbed from around 16G to 19G, which is much better. I suspect two causes: 1) regular tcp implementation that has slow-start and aggressive back-off; 2) user-land application... -Simon On Thu, Dec 22, 2011 at 12:43 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > Le mercredi 21 décembre 2011 à 20:26 -0500, Simon Chen a écrit : >> Hi folks, >> >> I added an Intel X520 card to both the sender and receiver... Now I >> have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of >> the PCI bus shouldn't be the bottleneck. >> >> Now the throughput test gives me around 16Gbps in aggregate. Any ideas >> how I can push closer to 20G? I don't quite understand where the >> bottleneck is now. > > Could you post some "perf top" or "perf record / report" numbers ? > > > ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2011-12-23 15:03 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-11-16 23:44 under-performing bonded interfaces Simon Chen 2011-11-17 0:01 ` Ben Greear 2011-11-17 0:05 ` Simon Chen 2011-11-17 0:07 ` Ben Greear 2011-11-17 0:57 ` Ben Hutchings 2011-11-17 1:38 ` Simon Chen 2011-11-17 1:45 ` Simon Chen 2011-11-17 1:45 ` Rick Jones 2011-11-17 2:51 ` Ben Hutchings 2011-12-22 1:26 ` Simon Chen 2011-12-22 1:36 ` Stephen Hemminger 2011-12-22 3:31 ` Ben Greear 2011-12-22 2:28 ` Simon Chen 2011-12-22 5:43 ` Eric Dumazet 2011-12-23 15:03 ` Simon Chen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).