Re: Infiniband 40GB

[Mark Nelson <mark.nelson@inktank.com>]

On 6/4/12 1:22 AM, Hannes Reinecke wrote:
> On 06/03/2012 02:56 PM, Mark Nelson wrote:
>> On 6/3/12 3:10 AM, Stefan Priebe wrote:
>>> Hi List,
>>>
>>> has anybody already tried CEPH over Infiniband 40GB?
>>>
>>> Stefan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>> Hi Stefan,
>>
>> A couple of folks have done DDR IB.  For now you are limited to
>> ipoib though.  If you have the hardware available I'd be really
>> curious what kind of throughput/latencies you see.
>>
> Hehe.
>
> Good luck with that.
>
> We've tried on 10GigE with _disastrous_ results.
> Up to the point where 1GigE was actually _faster_.

Strange!  Do you see good results with something like iperf?  Internally 
we have 10GE on some of our test nodes and I can get up to around 
600MB/s per node during rados bench testing.

> So far we've uncovered two issues:
> - intel_idle was/is seriously broken (we've tried on 3.0-stable,
>    so might've been fixed by now)
> - osd-server is calling 'fsync' on each and every write request.
>    Does wonders for performance ...

For syncfs support, upgrade to a distro with glibc 2.13+ (ie precise). 
I've noticed a significant improvement in our spinning disk performance 
going from oneiric and kernel 3.3 to precise and kernel 3.4.  I think 
part of this is related to the raid drivers for the cards we have in our 
test boxes though.  I'm actually recording blktrace and seekwatcher 
results for all of our tests to specifically look at syncs and disk seek 
behavior...

>
> Cheers,
>
> Hannes

Mark