Re: flashcache

[Mark Nelson <mark.nelson@inktank.com>]

On 01/17/2013 07:32 AM, Joseph Glanville wrote:
> On 17 January 2013 20:46, Gandalf Corvotempesta
> <gandalf.corvotempesta@gmail.com>  wrote:
>> 2013/1/16 Mark Nelson<mark.nelson@inktank.com>:
>
>> I don't know if I have to use a single two port IB card (switch
>> redundancy and no card redundancy) or
>> I have to use two single port cards. (or a single one port IB?)
>
> On the topic of IB..
>
> But slightly off-topic all the same..  I would love to attempt getting
> Ceph running on rsockets if I could find the time (alas we don't run
> Ceph).
> rsockets is a fully userland implementation of BSD sockets over RDMA,
> supporting fork and all the usual goodies, in theory unless you are
> using the kernel RBD module (of the kernel FS module etc) you should
> be able to run it on rsockets and enjoy a considerable performance
> increase.
>
> rsockets is available in the librdmacm git up on Open Fabrics and dev
> + support happens on the linux-rdma list.
>

There's been some talk about rsockets on the list before.  I think there 
are a couple of different folks that have tried (succeeded?) in getting 
it working.  barring that, it sounds like if you tune interrupt affinity 
settings and various other bits you can get IPoIB up into the 2GB/s+ 
range which while not RDMA speed, is at least better than 10GbE.

-- 
Mark Nelson
Performance Engineer
Inktank