building a disk server

building a disk server

[Sebastian Kuzminsky]

Hi folks, I want to pick your brains.  I'm building a new disk server
for a small company, about 20 people.  We need about 2 TB now and
maybe another 2 TB in a year or so.  Price is a factor, but
reliability is more important.

I'm planning to build a server-class PC from parts, and use the
standard RAID + LVM setup for growing the disk space in the future. 
Does this seem reasonable?

I'm planning to use SATA disks, because they seem to offer a good
balance between performance, price, and reliability.  I wish Linux
SATA supported SMART and hotplug!  I've had several drive failures
with Maxtor 300 GB PATA and 320 GB SATA disks this year, what's a more
reliable drive people use?  I'd prefer to buy fewer, higher-capacity
drives (300+ GB).  Any experience with the new 500's?

I'm planning to put all the disks in a big tower case like the
CoolerMaster Stacker
<http://www.coolermaster-usa.com/CoolerMaster/ProductList.aspx?catID=614>,
probably using the CoolerMaster 3-to-4 disk adapter.  I'd like dual,
redundant power supplies.  Any particular cases & power supplies work
well for y'all?

Some sort of name-brand motherboard & CPU, about 2 GB RAM, and I'm done!

SATA disk controllers usually come in 2x, 4x, and 8x versions, so
there's a slight advantage to building arrays that use those numbers
of disks.  I've mostly been using 4-disk RAID-5 arrays in the past,
but I'd like more redundancy.  4-disk RAID-6 seems is an option, as is
8-disk RAID-6 with a hot spare.  What else should I be considering?

Am I on the right track here?  I'm worried that I'm just scaling up my
home setup, maybe I'm missing an opportunity to do something smarter
here.

--
Sebastian Kuzminsky

Re: building a disk server

[Brad Dameron]

On Mon, 2005-11-28 at 16:55 -0700, Sebastian Kuzminsky wrote:
> Hi folks, I want to pick your brains.  I'm building a new disk server
> for a small company, about 20 people.  We need about 2 TB now and
> maybe another 2 TB in a year or so.  Price is a factor, but
> reliability is more important.
> 
> I'm planning to build a server-class PC from parts, and use the
> standard RAID + LVM setup for growing the disk space in the future. 
> Does this seem reasonable?
> 
> I'm planning to use SATA disks, because they seem to offer a good
> balance between performance, price, and reliability.  I wish Linux
> SATA supported SMART and hotplug!  I've had several drive failures
> with Maxtor 300 GB PATA and 320 GB SATA disks this year, what's a more
> reliable drive people use?  I'd prefer to buy fewer, higher-capacity
> drives (300+ GB).  Any experience with the new 500's?
> 
> I'm planning to put all the disks in a big tower case like the
> CoolerMaster Stacker
> <http://www.coolermaster-usa.com/CoolerMaster/ProductList.aspx?catID=614>,
> probably using the CoolerMaster 3-to-4 disk adapter.  I'd like dual,
> redundant power supplies.  Any particular cases & power supplies work
> well for y'all?
> 
> Some sort of name-brand motherboard & CPU, about 2 GB RAM, and I'm done!
> 
> SATA disk controllers usually come in 2x, 4x, and 8x versions, so
> there's a slight advantage to building arrays that use those numbers
> of disks.  I've mostly been using 4-disk RAID-5 arrays in the past,
> but I'd like more redundancy.  4-disk RAID-6 seems is an option, as is
> 8-disk RAID-6 with a hot spare.  What else should I be considering?
> 
> Am I on the right track here?  I'm worried that I'm just scaling up my
> home setup, maybe I'm missing an opportunity to do something smarter
> here.
> 
> --
> Sebastian Kuzminsky

Might look at a Areca SATA RAID controller. They support up to 24 ports
and has hardware level RAID capacity expansion. Or if you want to go
cheaper look at the 3ware controller and use it in JBOD. That way you
can get the Smart monitoring and hotplug. 

Here is the server I built with 3TB usable for pretty cheap.
All this was from www.8anet.com

Supermicro SC933T-R760 3u or SC932T-R760 rackmount Chassis with 15 SATA
Hot-Swap drive trays and triple redundant power supplies.
Any motherboard and CPU will do. I would recommend a AMD64 CPU with a
motherboard that has a PCI-X slot on it if possible. I used a Tyan S2468
with dual Athlon 2800's and 2GB. 
A 3ware 9500S-12. Not the 9500S-12MI with this case. Or the new
9550SX-12 which is much faster now.
12 - 300GB Maxtor MaXLine III drives
2 - Western Digital 36GB 10k drives


I use the 2 36GB drives mirrored for the OS since I had the extra slots.
Could of went with a Areca 16 port card instead. But I already had the
3ware laying around. I went with the 300GB Maxtor drives because at the
time they were the ones that had SATAII NCQ (Native Command Queuing) and
16MB cache. This setup is very fast and I use it as a NFS server for
backing up my main servers. I currently have about 20% left out of 3TB.
Time to add another one. 

Hope this helps.

Brad Dameron
SeaTab Software
www.seatab.com  

Re: building a disk server

[Lajber Zoltan]

Hi,

On Mon, 28 Nov 2005, Sebastian Kuzminsky wrote:

> Hi folks, I want to pick your brains.  I'm building a new disk server
> for a small company, about 20 people.  We need about 2 TB now and
> maybe another 2 TB in a year or so.  Price is a factor, but
> reliability is more important.

Take a look at http://www.coraid.com

Bye,
-=Lajbi=----------------------------------------------------------------
 LAJBER Zoltan               Szent Istvan Egyetem,  Informatika Hivatal
 Most of the time, if you think you are in trouble, crank that throttle!

Re: building a disk server

[Sebastian Kuzminsky]

On 11/29/05, Lajber Zoltan <lajbi@lajli.gau.hu> wrote:
> Take a look at http://www.coraid.com

Looks sweet!  Kinda expensive though...

Does anyone have real-world experience with this setup?  How does it
work in practice?  Pretty reliable?  Hotplug & smart?


--
Sebastian Kuzminsky

Re: building a disk server

[Brad Dameron]

On Mon, 2005-11-28 at 22:05 -0700, Sebastian Kuzminsky wrote:
> On 11/28/05, Brad Dameron <brad@seatab.com> wrote:
> > Might look at a Areca SATA RAID controller. They support up to 24 ports
> > and has hardware level RAID capacity expansion. Or if you want to go
> > cheaper look at the 3ware controller and use it in JBOD. That way you
> > can get the Smart monitoring and hotplug.
> 
> Do you use the regular smartd?
> 
> The only hotplug disks I've used have been through USB, and that works
> great now days.  Does SATA behave the same way?
> 
> I prefer JBOD because I trust the Linux software RAID better than the
> hardware vendor's.  Last time I built a system with hardware RAID was
> about 2002 and access to the array was really slow, maybe it's better
> now...  That was a 3Ware 4xPATA controller, dont remember the model
> number.
> 
> 
> > Supermicro SC933T-R760 3u or SC932T-R760 rackmount Chassis with 15 SATA
> > Hot-Swap drive trays and triple redundant power supplies.
> > Any motherboard and CPU will do. I would recommend a AMD64 CPU with a
> > motherboard that has a PCI-X slot on it if possible. I used a Tyan S2468
> > with dual Athlon 2800's and 2GB.
> > A 3ware 9500S-12. Not the 9500S-12MI with this case. Or the new
> > 9550SX-12 which is much faster now.
> > 12 - 300GB Maxtor MaXLine III drives
> > 2 - Western Digital 36GB 10k drives
> 
> Thanks for the list!
> 
> Is Linux aware of the state of your power supplies at all?  It'd be
> nice to get an email when someone trips and pulls out one of the power
> cords....  Do you have each power supply plugged in to a different
> UPS?  :-)
> 

Please don't send to me directly. This could help someone else on the
list. 

3ware has Linux tools to monitor the drives or you. And their 9000
series cards are very fast in a RAID configuration. I actually use RAID5
with 1 hot-spare drive. Their older cards were kind of slow. I have a
few laying around here as well. 

No I do not believe that Linux is aware of the power supplies. They do
however beep when one is not plugged in. And I have seen a card for
windows that you can plug this into and notify you. But not sure of a
Linux version. The 3ware and Areca card's are very fast and reliable.
SATA is the future and expect it to replace SCSI in the next year or so
when SATA-3 comes out at 600MB/s bus and they start coming out with full
10k and 15k drives.

Brad Dameron
SeaTab Software
www.seatab.com

Re: building a disk server

[Jeff Breidenbach]

> I'd prefer to buy fewer, higher-capacity drives (300+ GB).  Any
> experience with the new 500's?

I currently have 3 of the 500GB Hitachi's in a RAID-1 configuration
using linux software RAID. So far, so good.  

In response to someone else's question, the mostly random reads are
pretty well balanced between the disks, judging from atsar reports.
Adding the third drive definitely helped read performance.

-Jeff

Re: building a disk server

[Brad Dameron]

On Tue, 2005-11-29 at 14:26 -0800, Jeff Breidenbach wrote:
> > I'd prefer to buy fewer, higher-capacity drives (300+ GB).  Any
> > experience with the new 500's?
> 
> I currently have 3 of the 500GB Hitachi's in a RAID-1 configuration
> using linux software RAID. So far, so good.  
> 
> In response to someone else's question, the mostly random reads are
> pretty well balanced between the disks, judging from atsar reports.
> Adding the third drive definitely helped read performance.
> 
> -Jeff

With 3 drives I would of opted for a RAID5. More storage and the
performance won't be too bad. Except writes would be somewhat slower.
Probably not to noticable on a desktop.

Brad Dameron
SeaTab Software
www.seatab.com

Re: building a disk server

[Andy Smith]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

On Tue, Nov 29, 2005 at 02:26:24PM -0800, Jeff Breidenbach wrote:
> 
> > I'd prefer to buy fewer, higher-capacity drives (300+ GB).  Any
> > experience with the new 500's?
> 
> I currently have 3 of the 500GB Hitachi's in a RAID-1 configuration
> using linux software RAID. So far, so good.  

Hmm, but currently (at least in UK), 250GB is the sweet spot for
cost/GB in SATA.  So personally I'd prefer a 4+ disk RAID-10, more
spindles will give better performance.

Fewer, larger drives does make life (space, cooling) simpler I
suppose.


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: building a disk server

[Sebastian Kuzminsky]

Ok, I've been looking around some more.  I like the flexibility of
having the disks external to the "head" computer.  Looks like AoE is
not quite ready for the prime time yet.  SCSI seems overpriced for
what you get.  eSATA looks sweet.

Sonnet makes an 8-channel eSATA controller, PCI-X, using the Marvell
6081 chipset.  Supported in 2.6.14.  About $275. 
<http://www.sonnettech.com/product/tempo-x_esata8.html>

Addonics makes a 4-channel eSATA controller, also PCI-X, using the Sil
3124 chipset, also supported in 2.6.14.  About $90. 
<http://addonics.com/products/host_controller/adsa3gx4r-e.asp>

Anyone know of any other PCI-X multi-channel eSATA controllers? 
There's also the option of getting a regular (non-external) SATA
controller and the little SATA-to-eSATA converters, but I'd prefer to
cut down on the number of in-line connectors.


There are lots of eSATA enclosures available, holding from 1 to 8
drives.  Here's just one example:
<http://www.macgurus.com/productpages/sata/hotswapsatakits.php>. 
However, none of the enclosures I've found have redundant power
supplies.  I'm thinking to use RAID-6, and a pile of 2x enclosures, so
if a power supply goes I lose just 2 disks and my array stays up.  If
I can find an 8x enclosure with redundant power supplies that would be
preferable.

Does anyone have any experience with any eSATA enclosures, good or bad?


Comments?


--
Sebastian Kuzminsky

Re: building a disk server

[Bill Davidsen]

Brad Dameron wrote:

>Might look at a Areca SATA RAID controller. They support up to 24 ports
>and has hardware level RAID capacity expansion. Or if you want to go
>cheaper look at the 3ware controller and use it in JBOD. That way you
>can get the Smart monitoring and hotplug. 
>
>Here is the server I built with 3TB usable for pretty cheap.
>All this was from www.8anet.com
>
>Supermicro SC933T-R760 3u or SC932T-R760 rackmount Chassis with 15 SATA
>Hot-Swap drive trays and triple redundant power supplies.
>Any motherboard and CPU will do. I would recommend a AMD64 CPU with a
>motherboard that has a PCI-X slot on it if possible. I used a Tyan S2468
>with dual Athlon 2800's and 2GB. 
>A 3ware 9500S-12. Not the 9500S-12MI with this case. Or the new
>9550SX-12 which is much faster now.
>12 - 300GB Maxtor MaXLine III drives
>2 - Western Digital 36GB 10k drives
>
>
>I use the 2 36GB drives mirrored for the OS since I had the extra slots.
>Could of went with a Areca 16 port card instead. But I already had the
>3ware laying around. I went with the 300GB Maxtor drives because at the
>time they were the ones that had SATAII NCQ (Native Command Queuing) and
>16MB cache. This setup is very fast and I use it as a NFS server for
>backing up my main servers. I currently have about 20% left out of 3TB.
>Time to add another one. 
>  
>

The only part of the hardware I would change is the CPU setup, a single 
dual core setup seems more cost and heat effective now. The controller 
is fine, but that just gets better with time as new stuff comes out.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979