Re: Ideal hardware spec?

[Wido den Hollander <wido@widodh.nl>]

On 08/24/2012 05:05 PM, Mark Nelson wrote:
>>>
>>> I'm running Atom D525 (SuperMicro X7SPA-HF) nodes with 4GB of RAM and
>>> 4 2TB
>> disks and a 80GB SSD (old X25-M) for journaling.
>>>
>>> That works, but what I notice is that under heavy recover the Atoms
>>> can't
>> cope with it.
>>>
>>> I'm thinking about building a couple of nodes with the AMD Brazos
>> mainboard, somelike like an Asus E35M1-I.
>>>
>>> That is not a serverboard, but it would just be a reference to see
>>> what it
>> does.
>>>
>>> One of the problems with the Atoms is the 4GB memory limitation, with
>>> the
>> AMD Brazos you can use 8GB.
>>>
>>> I'm trying to figure out a way to have a really large amount of small
>>> nodes
>> for a low price to have
>>> a massive cluster where the impact of loosing one node is very small.
>>
>> Given that "massive" is a relative term, I am as well... but I'm also
>> trying
>> to reduce the footprint (power and space) of that "massive" cluster.
>> I also
>> want to start small (1/2 rack) and scale as needed.
>
> If you do end up testing Brazos processes, please post your results!  I
> think it really depends on what kind of performance you are aiming for.
>   Our stock 2U test boxes have 6-core opterons, and our SC847a has dual
> 6-core low power Xeon E5s.  At 10GbE+ these are probably going to be
> pushed pretty hard, especially during recovery.
>

I'm aiming for a Ceph cluster of a couple of hundred TB consisting out 
of 5 or 6 racks full of 1U machines with each 4x 1TB.

Having about ~200 of these nodes all doing not that much work.

If one fails I'd loose 0.5% of my cluster and recovery shouldn't be that 
hard. Assuming here that the node crashes due to hardware failure, not 
being plagued by some Ceph or BTRFS bug cluster-wide :)

Wido