How fast can your 10G go?

How fast can your 10G go?

[Ben Greear]

I've been running some tests on a new Nehalem based system
with a 2 port pci-e x8 10G NIC (ixgbe driver).

When using pktgen, max I can get is about 5.6Gbps tx + rx on both ports.  This is
about 22Gbps across the backplane, so I don't mean to complain :)

However, I'm curious if anyone has gotten any better performance on
some other system?  In particular, it seems that my system is bound by
the bus and/or the NIC.  Would I need to find something like a x16 slot
to have a chance at 10Gbps bi-directional on 2 ports?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: How fast can your 10G go?

[Bill Fink]

On Tue, 19 May 2009, Ben Greear wrote:

> I've been running some tests on a new Nehalem based system
> with a 2 port pci-e x8 10G NIC (ixgbe driver).
> 
> When using pktgen, max I can get is about 5.6Gbps tx + rx on both ports.  This is
> about 22Gbps across the backplane, so I don't mean to complain :)
> 
> However, I'm curious if anyone has gotten any better performance on
> some other system?  In particular, it seems that my system is bound by
> the bus and/or the NIC.  Would I need to find something like a x16 slot
> to have a chance at 10Gbps bi-directional on 2 ports?

We have achieved bidirectional 20-Gbps line rate traffic between a pair
of Nehalem i7 quad-core servers, each with a dual-port 10-GigE Myricom
PCI-E 2.0 x8 SFP+ NIC.  Here's a unidirectional 10-second nuttcp TCP
test (using 9000 byte jumbo frames):

[root@i7raid-1 ~]# ./nuttcp-6.2.6 -N2 -w10m /192.168.101.2/192.168.102.2
23627.0625 MB /  10.01 sec = 19792.4891 Mbps 53 %TX 57 %RX 0.11 msRTT

Your results sound about right for a PCI-E 1.0 x8 slot, which has
16 Gbps full-duplex usable bandwidth, and then subtracting about 25%
PCI-E overhead leaves about 12 Gbps full-duplex or 24 Gbps total
available bandwidth for TCP|UDP data transfers.

						-Bill

Re: How fast can your 10G go?

[Jesper Dangaard Brouer]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1727 bytes --]

On Tue, 19 May 2009, Ben Greear wrote:

> I've been running some tests on a new Nehalem based system
> with a 2 port pci-e x8 10G NIC (ixgbe driver).
>
> When using pktgen, max I can get is about 5.6Gbps tx + rx on both ports. 
> This is about 22Gbps across the backplane, so I don't mean to complain :)
>
> However, I'm curious if anyone has gotten any better performance on
> some other system?

Robert Olsson (and Olof Hagsand and Bengt Gördén) got some better results 
using that exact hardware Intel 82598 chips (and Sun neptune NICs).  They 
made an article called: "Open-source routing at 10Gb/s"

  https://www.iis.se/docs/10G-OS-router_2_.pdf


> In particular, it seems that my system is bound by
> the bus and/or the NIC.  Would I need to find something like a x16 slot
> to have a chance at 10Gbps bi-directional on 2 ports?

Are you doing a single flow bandwidth test?

These NICs designed for multiflow performance. They have hardware RX and 
TX queues to facilitate this, often call multiqueue NICs.  This is also 
what helps us do real parallel processing across CPUs in the network stack 
(since DaveMs multiqueue changes, although not all drivers uses this 
correctly yet...).

Try doing a multi-flow network test, I bet you will see better results. 
(Normally you also need to adjust smp-affinity, but with large frames on 
my Core i7 system its fast enough for 9.5 Gbit/s without tuning, small 
frames is another case)

Cheers,
   Jesper Brouer

--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen
Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------