Re: Request for ceph.conf environment extension

[Mark Nelson <mark.nelson@inktank.com>]

On 05/06/2013 09:18 AM, Jens Kristian Søgaard wrote:
> Hi,
>
>> I'm just supposing, i'm also evaluating the same network topology
>> like you.
>
> I have tested my current setup for a while with just triple-bonded GbE,
> and haven't found the need for more features. I might be wrong ofcourse :-)
>
>> There is a lowcost. 12x 10GbaseT switch from Netgear.
>
> I was looking at the Netgear XS708E which is very low cost compared to
> what the prices were 6 months ago when I looked at it last.

I'm really not up to speed on the capabilities of cheap 10GbE switches. 
I'm sure other people will have comments about what features are 
worthwhile, etc.  Just from a raw performance perspective, you might 
want to make sure that the switch can handle lots of randomly 
distributed traffic between all of the ports well.  I'd expect that it 
shouldn't be too terrible, but who knows.  I expect most IB switches you 
run into should deal with this kind of pattern competently enough for 
Ceph workloads to do OK.  On the front end portion of the network you'll 
always have client<->server communication, so the pattern there will be 
less all-to-all than the backend traffic (more 1st half <-> 2nd half).

On really big deployments, static routing becomes a pretty big issue. 
Dynamic routing, or at least well optimized routes can make a huge 
difference.  ORNL has done some work in this area for their Lustre IB 
networks and I think LLNL is investigating it as well.

Mark

>
> What I don't know is how those prices compare to IB pricing.
>
>> on cluster network and IB to public network). Do you think will be
>> possible to use all ceph services (RBD, RGW, CephFS, QEMU, ...) via
>> SDP/rsockets ?
>
> I haven't tried SDP/rsocket, so I don't know. From what I have
> experienced in the past, these things doesn't seem to be completely
> "drop in and forget about it" - there's always something that is not
> completely compatible with ordinary TCP/IP sockets.
>
>> On ebay there are many IB switches sold at 1000-1500$ with 24 or 36
>> DDR ports. If used with SDP/rsockets you should archieve more or less
>> 18Gbps
>
> That sounds cheap yes, but I assume that is used equipment?
> I was looking at costs for new equipment.
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html