Re: IPoIB to Ethernet routing performance

From: "Richard Croucher" <richard-jNDFPZUTrfRJpuwtbJ71GdBPR1lH4CV8@public.gmane.org>
To: 'sebastien dugue' <sebastien.dugue-6ktuUTfB/bM@public.gmane.org>,
	'OF EWG' <ewg-ZwoEplunGu1OwGhvXhtEPSCwEArCW2h5@public.gmane.org>
Cc: 'linux-rdma' <linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Subject: Re: IPoIB to Ethernet routing performance
Date: Mon, 6 Dec 2010 10:49:58 -0000	[thread overview]
Message-ID: <00d701cb9533$71c5f2e0$5551d8a0$@com> (raw)
In-Reply-To: <20101206112454.76bb85f1-xRPE6/W2vR9iM9LT7/dT9zWMkbuR3peG@public.gmane.org>

You may be able to improve by doing some OS tuning.  All this data should stay in kernel mode but there are lots of bottlenecks in the TCP/IP stack that limit scalability.  The IPoIB code has not been optimized for this use case.

You don't mention what Server, kernel and OFED distro you are running.

The best performance is achieved using InfiniBand/Ethernet hardware gateways.  Most of these provide virtual Ethernet NICs to InfiniBand hosts, but the Voltaire 4036E does provide a  IPoIB to Ethernet gateway capability.  This is FPGA based so does provide much higher performance than you will achieve using a standard server solution. 

-----Original Message-----
From: ewg-bounces@lists.openfabrics.org [mailto:ewg-bounces@lists.openfabrics.org] On Behalf Of sebastien dugue
Sent: 06 December 2010 10:25
To: OF EWG
Cc: linux-rdma
Subject: [ewg] IPoIB to Ethernet routing performance

  Hi,

  I know this might be off topic, but somebody may have already run into the same
problem before.

  I'm trying to use a server as a router between an IB fabric and an Ethernet network.

  The router is fitted with one ConnectX2 QDR HCA and one dual port Myricom 10G
Ethernet adapter.

  I did some bandwidth measurements using iperf with the following setup:

  +---------+               +---------+               +---------+
  |         |               |         |   10G Eth     |         |
  |         |    QDR IB     |         +---------------+         |
  | client  +---------------+  Router |   10G Eth     |  Server |
  |         |               |         +---------------+         |
  |         |               |         |               |         |
  +---------+               +---------+               +---------+

  However, the routing performance is far from what I would have expected.

  Here are some numbers:

  - 1 IPoIB stream between client and router: 20 Gbits/sec

    Looks OK.

  - 2 Ethernet streams between router and server: 19.5 Gbits/sec

    Looks OK.

  - routing 1 IPoIB stream to 1 Ethernet stream from client to server: 9.8 Gbits/sec

    We manage to saturate the Ethernet link, looks good so far.

  - routing 2 IPoIB streams to 2 Ethernet streams from client to server: 9.3 Gbits/sec

    Argh, even less that when routing a single stream. I would have expected
    a bit more than this.

  Has anybody ever tried to do some routing between an IB fabric and an Ethernet
network and achieved some sensible bandwidth figures?

  Are there some known limitations in what I try to achieve?

  Thanks,

  Sébastien.

_______________________________________________
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg

_______________________________________________
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg