Re: high throughput storage server?

[Joe Landman <joe.landman@gmail.com>]

On 2/17/2011 6:49 PM, Stan Hoeppner wrote:
> Joe Landman put forth on 2/17/2011 4:13 PM:
>
>> Well, the application area appears to be high performance cluster
>> computing, and the storage behind it.  Its a somewhat more specialized
>> version of storage, and not one that a typical IT person runs into
>> often.  There are different, some profoundly so, demands placed upon
>> such storage.
>
> The OP's post described an ad hoc collection of 40-50 machines doing
> various types of processing on shared data files.  This is not classical
> cluster computing.  He didn't describe any kind of _parallel_
> processing.  It sounded to me like staged batch processing, the

Semantics at best.  He is doing significant processing, in parallel, 
doing data analysis, in parallel, across a cluster of machines.  Doing 
MPI-IO?  No.  Does not using MPI make this not a cluster?  No.

> bandwidth demands of which are typically much lower than a parallel
> compute cluster.

See his original post.  He posits his bandwidth demands.

>
>> Full disclosure:  this is our major market, we make/sell products in
>> this space, have for a while.  Take what we say with that in your mind
>> as a caveat, as it does color our opinions.
>
> Thanks for the disclosure Joe.
>
>> The spec's as stated, 50Gb/s ... its rare ... exceptionally rare ...
>> that you ever see cluster computing storage requirements stated in such
>> terms.  Usually they are stated in the MB/s or GB/s regime.  Using  a
>> basic conversion of Gb/s to GB/s, the OP is looking for ~6GB/s support.
>
> Indeed.  You typically don't see this kind of storage b/w need outside
> the government labs and supercomputing centers (LLNL, Sandia, NCCS,
> SDSC, etc).  Of course those sites' requirements are quite a bit higher
> than a "puny" 6 GB/s.

Heh ... we see it all the time in compute cluster, large data analysis 
farms etc.  Not at the big labs.

[...]

> McData, etc.  I've not hard of a front end loop being used in many many
> years.  Some storage vendors still use loops on the _back_ end to
> connect FC/SAS/SATA expansion chassis to the head controller, IBM and

I am talking about the back end.

> NetApp come to mind, but it's usually dual loops per chassis, so you're
> looking at ~3 GB/s per expansion chassis using 8 Gbit loops.  One would

2 GB/s assuming FC-8, and 20 lower speed drives are sufficient to 
completely fill 2 GB/s.  So, as I was saying, the design matters.

[...]

> Nexsan doesn't offer direct SAS connection on the big 42/102 drive Beast
> units, only on the Boy units.  The Beast units all use dual or quad FC
> front end ports, with a couple front end GbE iSCSI ports thrown in for
> flexibility.  The SAS Boy units beat all competitors on price/TB, as do
> all the Nexsan products.

As I joked one time, many many years ago "broad sweeping generalizations 
tend to be incorrect".  Yes, it is a recursive joke, but there is a 
serious aspect to it.  Your proffered pricing per TB, which you claim 
Nexsan beats all ... is much higher than ours, and many others.  No, 
they don't beat all, or even many.


> I'd like to note that over subscription isn't intrinsic to a piece of
> hardware.  It's indicative of an engineer or storage architect not
> knowing what the blank he's doing.

Oversubscription and it corresponding resource contention, not to 
mention poor design of other aspects ... yeah, I agree that this is 
indicative of something.  One must question why people continue to 
deploy architectures which don't scale.

>
>> As I said, high performance storage design is a very ... very ...
>> different animal from standard IT storage design.  There are very
>> different decision points, and design concepts.
>
> Depends on the segment of the HPC market.  It seems you're competing in
> the low end of it.  Configurations get a bit exotic at the very high

I noted this about your previous responses, this particular tone you 
take.  I debated for a while responding, until I saw something I simply 
needed to correct.  I'll try not to take your bait.

[...]

> So, again, it really depends on the application(s), as always,
> regardless of whether it's HPC or IT, although there are few purely
> streaming IT workloads, EDL of decision support databases comes to mind,
> but these are usually relatively short duration.  They can still put
> some strain on a SAN if not architected correctly.
>
>>> You don't see many deployed filers on the planet with 5 * 10 GbE front
>>> end connections.  In fact, today, you still don't see many deployed
>>> filers with even one 10 GbE front end connection, but usually multiple
>>> (often but not always bonded) GbE connections.
>>
>> In this space, high performance cluster storage, this statement is
>> incorrect.
>
> The OP doesn't have a high performance cluster.  HPC cluster storage by

Again, semantics.  They are doing massive data ingestion and processing. 
  The view of this is called "big data" in HPC circles and it is *very 
much* an HPC problem.

> accepted definition includes highly parallel workloads.  This is not
> what the OP described.  He described ad hoc staged data analysis.

See above.  If you want to argue semantics, be my guest, I won't be 
party to such a waste of time.  The OP is doing analysis that requires a 
high performance architecture.  The architecture you suggested is not 
one people in the field would likely recommend.

[rest deleted]


--
joe