Re: Request for ceph.conf environment extension

[Mark Nelson <mark.nelson@inktank.com>]

On 05/06/2013 07:36 AM, Gandalf Corvotempesta wrote:
> 2013/5/6 Andreas Friedrich <andreas.friedrich@ts.fujitsu.com>:
>> To enable the LD_PRELOAD mechanism for the Ceph daemons only, a little
>> generic extension in the global section of /etc/ceph/ceph.conf would
>> be helpful, e.g.:
>>
>> [global]
>>          environment = LD_PRELOAD=/usr/lib64/libsdp.so.1
>>
>> The appending patch adds 5 lines in the Bobtail (0.56.6) init script.
>> The init script will then read the environment setting and - if present -
>> call the Ceph daemons with the preceding environment string.
>
> Cool! We are planning the same infrastructure with IB as networking.
> Could you share more details about this?
>   - which performance are you having and on which hardware?
>   - any issue with IB and SDP?
>   - are you able to use the newer rsockets and not SDP? It also has a
> preloader library and is still developed (SDP is deprecated)

Yeah, that's more or less the conclusion I came to as well.  With SDP 
being deprecated, rsockets is looking like an attractive potential 
alternative.  I'll let Sage or someone comment on the patch though.

It would be very interesting to hear how SDP does.  With IPoIB I've 
gotten about 2GB/s on QDR with Ceph, which is roughly also what I can 
get in an ideal round-robin setup with 2 bonded 10GbE links.

Mark