Re: Ideal hardware spec?

[Mark Nelson <mark.nelson@inktank.com>]

On 08/24/2012 01:12 PM, Wido den Hollander wrote:
>
>
> On 08/24/2012 05:05 PM, Mark Nelson wrote:
>>>>
>>>> I'm running Atom D525 (SuperMicro X7SPA-HF) nodes with 4GB of RAM and
>>>> 4 2TB
>>> disks and a 80GB SSD (old X25-M) for journaling.
>>>>
>>>> That works, but what I notice is that under heavy recover the Atoms
>>>> can't
>>> cope with it.
>>>>
>>>> I'm thinking about building a couple of nodes with the AMD Brazos
>>> mainboard, somelike like an Asus E35M1-I.
>>>>
>>>> That is not a serverboard, but it would just be a reference to see
>>>> what it
>>> does.
>>>>
>>>> One of the problems with the Atoms is the 4GB memory limitation, with
>>>> the
>>> AMD Brazos you can use 8GB.
>>>>
>>>> I'm trying to figure out a way to have a really large amount of small
>>>> nodes
>>> for a low price to have
>>>> a massive cluster where the impact of loosing one node is very small.
>>>
>>> Given that "massive" is a relative term, I am as well... but I'm also
>>> trying
>>> to reduce the footprint (power and space) of that "massive" cluster.
>>> I also
>>> want to start small (1/2 rack) and scale as needed.
>>
>> If you do end up testing Brazos processes, please post your results! I
>> think it really depends on what kind of performance you are aiming for.
>> Our stock 2U test boxes have 6-core opterons, and our SC847a has dual
>> 6-core low power Xeon E5s. At 10GbE+ these are probably going to be
>> pushed pretty hard, especially during recovery.
>>
>
> I'm aiming for a Ceph cluster of a couple of hundred TB consisting out
> of 5 or 6 racks full of 1U machines with each 4x 1TB.
>
> Having about ~200 of these nodes all doing not that much work.
>
> If one fails I'd loose 0.5% of my cluster and recovery shouldn't be that
> hard. Assuming here that the node crashes due to hardware failure, not
> being plagued by some Ceph or BTRFS bug cluster-wide :)
>
> Wido

Just based on past experience, I figure the most common causes of 
failure are going to be drive "failure", and controller failure.  Your 
solution mitigates that by just going with tons of 1U nodes with few 
drives.  I'm hoping we can also mitigate it by skipping expanders and 
doing no more than 8 drives per controller.  It does mean you top out at 
like 40-48 drives per node max on most server boards.

Mark