From mboxrd@z Thu Jan 1 00:00:00 1970 From: Simon Chen Subject: Re: under-performing bonded interfaces Date: Wed, 21 Dec 2011 21:28:45 -0500 Message-ID: References: <4EC44ECB.4050201@candelatech.com> <1321491449.2709.90.camel@bwh-desktop> <1321498314.2885.78.camel@deadeye> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Ben Greear , netdev@vger.kernel.org To: Ben Hutchings Return-path: Received: from mail-lpp01m010-f46.google.com ([209.85.215.46]:55962 "EHLO mail-lpp01m010-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752670Ab1LVC2r convert rfc822-to-8bit (ORCPT ); Wed, 21 Dec 2011 21:28:47 -0500 Received: by lahd3 with SMTP id d3so758866lah.19 for ; Wed, 21 Dec 2011 18:28:46 -0800 (PST) In-Reply-To: Sender: netdev-owner@vger.kernel.org List-ID: Just a bit more info... I have 24 cores, and the interrupts for each 10G NIC are distributed to all 24 cores... I am using the most recent 3.7.17 driver ixgbe driver from Intel. layer3+4 xmit policy. -Simon On Wed, Dec 21, 2011 at 8:26 PM, Simon Chen wro= te: > Hi folks, > > I added an Intel X520 card to both the sender and receiver... Now I > have two 10G ports on a PCIe 2.0 x8 slot (5Gx8), so the bandwidth of > the PCI bus shouldn't be the bottleneck. > > Now the throughput test gives me around 16Gbps in aggregate. Any idea= s > how I can push closer to 20G? I don't quite understand where the > bottleneck is now. > > Thanks. > -Simon > > On Wed, Nov 16, 2011 at 9:51 PM, Ben Hutchings > wrote: >> On Wed, 2011-11-16 at 20:38 -0500, Simon Chen wrote: >>> Thanks, Ben. That's good discovery... >>> >>> Are you saying that both 10G NICs are on the same PCIe x4 slot, so >>> that they're subject to the 12G throughput bottleneck? >> >> I assumed you were using 2 ports on the same board, i.e. the same sl= ot. >> If you were using 1 port each of 2 boards then I would have expected >> them both to be usable at full speed. =A0So far as I can remember, P= CIe >> bridges are usually set up so there isn't contention for bandwidth >> between slots. >> >> Ben. >> >> -- >> Ben Hutchings, Staff Engineer, Solarflare >> Not speaking for my employer; that's the marketing department's job. >> They asked us to note that Solarflare product names are trademarked. >>