Re: high throughput storage server?

[Joe Landman <joe.landman@gmail.com>]

On 02/17/2011 04:47 PM, Stan Hoeppner wrote:
> John Robinson put forth on 2/17/2011 5:07 AM:
>> On 14/02/2011 23:59, Matt Garman wrote:
>> [...]
>>> The requirement is basically this: around 40 to 50 compute machines
>>> act as basically an ad-hoc scientific compute/simulation/analysis
>>> cluster.  These machines all need access to a shared 20 TB pool of
>>> storage.  Each compute machine has a gigabit network connection, and
>>> it's possible that nearly every machine could simultaneously try to
>>> access a large (100 to 1000 MB) file in the storage pool.  In other
>>> words, a 20 TB file store with bandwidth upwards of 50 Gbps.
>>
>> I'd recommend you analyse that requirement more closely. Yes, you have
>> 50 compute machines with GigE connections so it's possible they could
>> all demand data from the file store at once, but in actual use, would they?
>
> This is a very good point and one which I somewhat ignored in my initial
> response, making a silent assumption.  I did so based on personal
> experience, and knowledge of what other sites are deploying.

Well, the application area appears to be high performance cluster 
computing, and the storage behind it.  Its a somewhat more specialized 
version of storage, and not one that a typical IT person runs into 
often.  There are different, some profoundly so, demands placed upon 
such storage.

Full disclosure:  this is our major market, we make/sell products in 
this space, have for a while.  Take what we say with that in your mind 
as a caveat, as it does color our opinions.

The spec's as stated, 50Gb/s ... its rare ... exceptionally rare ... 
that you ever see cluster computing storage requirements stated in such 
terms.  Usually they are stated in the MB/s or GB/s regime.  Using  a 
basic conversion of Gb/s to GB/s, the OP is looking for ~6GB/s support.

Some basic facts about this.

Fibre channel (FC-8 in particular), will give you, at best 1GB/s per 
loop, and that presumes you aren't oversubscribing the loop.  The vast 
majority of designs we see coming from IT shops, do, in fact, badly 
oversubscribe the bandwidth, which causes significant contention on the 
loops.  The Nexsan unit you indicated (they are nominally a competitor 
of ours) is an FC device, though we've heard rumblings that they may 
even allow for SAS direct connections (though that would be quite cost 
ineffective as a SAS JBOD chassis compared to other units, and you still 
have the oversubscription problem).

As I said, high performance storage design is a very ... very ... 
different animal from standard IT storage design.  There are very 
different decision points, and design concepts.

> You don't see many deployed filers on the planet with 5 * 10 GbE front
> end connections.  In fact, today, you still don't see many deployed
> filers with even one 10 GbE front end connection, but usually multiple
> (often but not always bonded) GbE connections.

In this space, high performance cluster storage, this statement is 
incorrect.

Our units (again, not trying to be a commercial here, see .sig if you 
want to converse offline) usually ship with either 2x 10GbE, 2x QDR IB, 
or combinations of these.  QDR IB gets you 3.2 GB/s.  Per port.

In high performance computing storage (again, the focus of the OP's 
questions), this is a reasonable configuration and request.
>
> A single 10 GbE front end connection provides a truly enormous amount of
> real world bandwidth, over 1 GB/s aggregate sustained.  *This is
> equivalent to transferring a full length dual layer DVD in 10 seconds*

Trust me.  This is not *enormous*.  Well, ok ... put another way, we 
architect systems that scale well beyond 10GB/s sustained.  We have nice 
TB sprints and similar sorts of "drag racing" as I call them (c.f. 
http://scalability.org/?p=2912 http://scalability.org/?p=2356 
http://scalability.org/?p=2165  http://scalability.org/?p=1980 
http://scalability.org/?p=1756 )

1 GB/s is nothing magical.  Again, not a commercial, but our DeltaV 
units, running MD raid, achieve 850-900MB/s (0.85-0.9 GB/s) for RAID6.

To get good (great) performance you have to start out with a good 
(great) design.  One that will really optimize the performance on a per 
unit basis.

> Few sites/applications actually need this kind of bandwidth, either
> burst or sustained.  But, this is the system I spec'd for the OP
> earlier.  Sometimes people get caught up in comparing raw bandwidth
> numbers between different platforms and lose sight of the real world
> performance they can get from any one of them.

The sad part is that we often wind up fighting against others "marketing 
numbers".  Our real benchmarks are often comparable to their "strong 
wind a the back" numbers.  Heck, our MD raid numbers often are better 
than others hardware RAID numbers.

Theoretical bandwidth from the marketing docs doesn't matter.  The only 
thing that does matter is having a sound design and implementation at 
all levels.  This is why we do what we do, and why we do use MD raid.

Regards,

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615