what is the expeted performance from a dual port 10G card ?

what is the expeted performance from a dual port 10G card ?

[Raz]

Hello
I have been testing 10G ethernet cards , intel ixgbe, chelsio, tehuti
and melanox . All are dual port cards.
I have used pktgen to test it. A single cpu is mapped over a single
interface (no neeed for MQ , which is quite bad anyway).
Results:
Intel gives: 13Gbps , 6.5 from each port.
The rest give ~10 Gbps .
Cards were tested on PCI-E x8 slot, on a supermicro x7DBI and
supermicro 1025W-UR.

Am i to expect ~20Gbps from each card ?
Thank you.

Re: what is the expeted performance from a dual port 10G card ?

[Ben Hutchings]

Raz wrote:
> Hello
> I have been testing 10G ethernet cards , intel ixgbe, chelsio, tehuti
> and melanox . All are dual port cards.
> I have used pktgen to test it. A single cpu is mapped over a single
> interface (no neeed for MQ , which is quite bad anyway).
> Results:
> Intel gives: 13Gbps , 6.5 from each port.
> The rest give ~10 Gbps .
> Cards were tested on PCI-E x8 slot, on a supermicro x7DBI and
> supermicro 1025W-UR.
> 
> Am i to expect ~20Gbps from each card ?

PCI Express provides 2.5 Gbps per lane (can be 5 Gbps in PCIe 2.0 but
that's quite new).  So an x8 slot is limited to 20 Gbps including all
the PCIe protocol overhead, which can be substantial.  You will not
get 20 Gbps out of dual-port cards without doubling the width or lane
speed.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: what is the expeted performance from a dual port 10G card ?

[Raz]

PCI Encoding overhead is 8 bits /10 bits . so 16Gbps supposed to be
the practical limit. I still lack 3 Gbps in the Intel case.
where are they ?

On Fri, Nov 28, 2008 at 3:54 PM, Ben Hutchings
<bhutchings@solarflare.com> wrote:
> Raz wrote:
>> Hello
>> I have been testing 10G ethernet cards , intel ixgbe, chelsio, tehuti
>> and melanox . All are dual port cards.
>> I have used pktgen to test it. A single cpu is mapped over a single
>> interface (no neeed for MQ , which is quite bad anyway).
>> Results:
>> Intel gives: 13Gbps , 6.5 from each port.
>> The rest give ~10 Gbps .
>> Cards were tested on PCI-E x8 slot, on a supermicro x7DBI and
>> supermicro 1025W-UR.
>>
>> Am i to expect ~20Gbps from each card ?
>
> PCI Express provides 2.5 Gbps per lane (can be 5 Gbps in PCIe 2.0 but
> that's quite new).  So an x8 slot is limited to 20 Gbps including all
> the PCIe protocol overhead, which can be substantial.  You will not
> get 20 Gbps out of dual-port cards without doubling the width or lane
> speed.
>
> Ben.
>
> --
> Ben Hutchings, Senior Software Engineer, Solarflare Communications
> Not speaking for my employer; that's the marketing department's job.
> They asked us to note that Solarflare product names are trademarked.
>

Re: what is the expeted performance from a dual port 10G card ?

[Ben Hutchings]

Raz wrote:
> PCI Encoding overhead is 8 bits /10 bits . so 16Gbps supposed to be
> the practical limit.

The 8b10 encoding is already accounted for in the 2.5 Gbps figure;
the raw bit rate is 3.125 Gbps.

I'm thinking about the transaction layer overhead.  Since PCI Express
has few side-band signals there is a protocol a little like TCP running
over the link, which adds a header to each request or response packet
(TLP).  The maximum TLP size is usually quite small so the header
accounts for a fair proportion of the link bandwidth.

> I still lack 3 Gbps in the Intel case. where are they ?

That's for Intel to answer.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: what is the expeted performance from a dual port 10G card ?

[Roland Dreier]

 > The 8b10 encoding is already accounted for in the 2.5 Gbps figure;
 > the raw bit rate is 3.125 Gbps.

No, this is absolutely incorrect for PCI Express.  PCI Express (1.0) raw
signaling is 2.5 gigatransfers/sec, and the overhead of 8b/10b encoding
reduces the data throughput to 2.0 gigabits/sec.  (You may be thinking
of the 10 gigabit ethernet XAUI standard, which uses 8b/10b encoding on
4 lanes running at 3.125 GT/sec to get 10 Gb/sec of data)

In addition, PCI Express transfers are broken up into packets, usually
with very small payloads (128 or 256 bytes are common).  So the packet
header overhead reduces throughput further, and then the transaction
layer adds further overhead.  Then transferring NIC control structures
over the link adds even more overhead.  So achieving 13.something Gb/sec
of real throughput on a PCIe link theoretically capable of 16 Gb/sec
seems pretty good to me.

There are motherboards with PCIe 2.0 slots running at 5.0 GT/sec
available (ie 32 Gb/sec of raw throughput), and I know the Mellanox
ConnectX NIC at least is capable of that as well, so you might be able
to get better performance on such a system.  However, you mention that
most of the NICs top out at 10 Gb/sec in your testing but the Intel NIC
goes higher.  So most likely you are hitting driver limits.  One common
issue is CPU affinity; you might want to play around with setting
interrupt affinity and/or pinning processes to a single CPU.  It might
also be interesting to profile where your CPU time is going.

 - R.

Re: what is the expeted performance from a dual port 10G card ?

[Trent Piepho]

On Sat, 29 Nov 2008, Roland Dreier wrote:
> > The 8b10 encoding is already accounted for in the 2.5 Gbps figure;
> > the raw bit rate is 3.125 Gbps.
>
> In addition, PCI Express transfers are broken up into packets, usually
> with very small payloads (128 or 256 bytes are common).  So the packet
> header overhead reduces throughput further, and then the transaction
> layer adds further overhead.  Then transferring NIC control structures
> over the link adds even more overhead.  So achieving 13.something Gb/sec
> of real throughput on a PCIe link theoretically capable of 16 Gb/sec
> seems pretty good to me.

For transfers with only 4 byte payloads, I get about 6.5 MB/sec over a PCIe
4x link.  For small transfers, the overhead is huge.

Re: what is the expeted performance from a dual port 10G card ?

[Raz]

On Sun, Nov 30, 2008 at 4:55 AM, Trent Piepho <tpiepho@freescale.com> wrote:
> On Sat, 29 Nov 2008, Roland Dreier wrote:
>> > The 8b10 encoding is already accounted for in the 2.5 Gbps figure;
>> > the raw bit rate is 3.125 Gbps.
>>
>> In addition, PCI Express transfers are broken up into packets, usually
>> with very small payloads (128 or 256 bytes are common).  So the packet
>> header overhead reduces throughput further, and then the transaction
>> layer adds further overhead.  Then transferring NIC control structures
>> over the link adds even more overhead.  So achieving 13.something Gb/sec
>> of real throughput on a PCIe link theoretically capable of 16 Gb/sec
>> seems pretty good to me.
I tested it both on PCIE2 and PCIE1 . same results.

> For transfers with only 4 byte payloads, I get about 6.5 MB/sec over a PCIe
> 4x link.  For small transfers, the overhead is huge.
>
Well, you are correct. I used packet size of 1.3k. when I increased to
9K a packet each port
reached ~7Gbps ( I am using pktgen). I used two cards, so I reached
28Gbps. BIOS settings for pci payload were AUTO,so i set it to 256
bytes but it made no difference, still 28Gbps. well, at least I know
what the problem is.
guys, I truly thank you all.

Re: what is the expeted performance from a dual port 10G card ?

[Roland Dreier]

 > I tested it both on PCIE2 and PCIE1 . same results.

if the NIC is not capable of PCIe 2.0 (5 GT/sec) then it wouldn't make a
difference.

 - R.

Re: what is the expeted performance from a dual port 10G card ?

[Rick Jones]

It would be nice to be able to expand this:

http://pcitop.berlios.de/

to a broader number of platforms.

rick jones

Re: what is the expeted performance from a dual port 10G card ?

[Divy Le Ray]

Raz wrote:
> Hello
> I have been testing 10G ethernet cards , intel ixgbe, chelsio, tehuti
> and melanox . All are dual port cards.
> I have used pktgen to test it. A single cpu is mapped over a single
> interface (no neeed for MQ , which is quite bad anyway).
> Results:
> Intel gives: 13Gbps , 6.5 from each port.
> The rest give ~10 Gbps .
> Cards were tested on PCI-E x8 slot, on a supermicro x7DBI and
> supermicro 1025W-UR.
> 
> Am i to expect ~20Gbps from each card ?

Hi Raz,

For the rest, what is the load balance between the ports?
The Chelsio adapter shares the pci link between all ports.

Cheers,
Divy

Re: what is the expeted performance from a dual port 10G card ?

[Raz]

On Sat, Nov 29, 2008 at 8:30 AM, Divy Le Ray <divy@chelsio.com> wrote:
> Raz wrote:
>> Hello
>> I have been testing 10G ethernet cards , intel ixgbe, chelsio, tehuti
>> and melanox . All are dual port cards.
>> I have used pktgen to test it. A single cpu is mapped over a single
>> interface (no neeed for MQ , which is quite bad anyway).
>> Results:
>> Intel gives: 13Gbps , 6.5 from each port.
>> The rest give ~10 Gbps .
>> Cards were tested on PCI-E x8 slot, on a supermicro x7DBI and
>> supermicro 1025W-UR.
>>
>> Am i to expect ~20Gbps from each card ?
>
> Hi Raz,
>
> For the rest, what is the load balance between the ports?
> The Chelsio adapter shares the pci link between all ports.
>
> Cheers,
> Divy
>
All ports share the same bandwidth. in chelsio it is 4.8 Gbps from each port.

Re: what is the expeted performance from a dual port 10G card ?

[Christoph Lameter]

Note also that the performance of NICs may be descreasing over time.
Network latency for myricom drivers for example doubled between .22 and
.28-rc6.