Re: IPoIB performance

[Ezra Kissel <ezkissel-GZvvpLG7cYSVc3sceRu5cw@public.gmane.org>]

On 9/5/2012 3:48 PM, Atchley, Scott wrote:
> On Sep 5, 2012, at 3:06 PM, Christoph Lameter wrote:
>
>> On Wed, 5 Sep 2012, Atchley, Scott wrote:
>>
>>>> AFAICT the network stack is useful up to 1Gbps and
>>>> after that more and more band-aid comes into play.
>>>
>>> Hmm, many 10G Ethernet NICs can reach line rate. I have not yet tested any 40G Ethernet NICs, but I hope that they will get close to line rate. If not, what is the point? ;-)
>>
>> Oh yes they can under restricted circumstances. Large packets, multiple
>> cores etc. With the band-aids….
>
> With Myricom 10G NICs, for example, you just need one core and it can do line rate with 1500 byte MTU. Do you count the stateless offloads as band-aids? Or something else?
>
> I have not tested any 40G NICs yet, but I imagine that one core will not be enough.
>
Since you are using netperf, you might also considering experimenting 
with the TCP_SENDFILE test.  Using sendfile/splice calls can have a 
significant impact for sockets-based apps.

Using 40G NICs (Mellanox ConnectX-3 EN), I've seen our applications hit 
22Gb/s single core/stream while fully CPU bound.  With sendfile/splice, 
there is no issue saturating a 40G link with about 40-50% core 
utilization.  That being said, binding to the right core/node, message 
size and memory alignment, interrupt handling, and proper host/NIC 
tuning all have an impact on the performance.  The state of 
high-performance networking is certainly not plug-and-play.

- ezra
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html