building a large/fast server

building a large/fast server

[Andy Arvai]

Hi,

I'm in the process of designing a large server system. The primary
consideration will be i/o performance (block read and write of large
files and maximum i/o's per second) and a filesystem size of 2TB. I'm
planning to go with 2 3ware 7500-8's, 16 Maxtor 4G160J8's (160GB) and
do software raid0 between the 7500-8's (raid5). The filesystem will
either be Reiser or Ext3. Any suggestions for fast dual cpu
motherboards or comments are welcome. Thanks for any information.

Andy

Re: building a large/fast server

[Joshua Baker-LePain]

On Wed, 29 Jan 2003 at 12:12pm, Andy Arvai wrote

> I'm in the process of designing a large server system. The primary
> consideration will be i/o performance (block read and write of large
> files and maximum i/o's per second) and a filesystem size of 2TB. I'm
> planning to go with 2 3ware 7500-8's, 16 Maxtor 4G160J8's (160GB) and
> do software raid0 between the 7500-8's (raid5). The filesystem will
> either be Reiser or Ext3. Any suggestions for fast dual cpu
> motherboards or comments are welcome. Thanks for any information.

Those boards need to be on separate PCI busses if you want to fully 
utilize them.  That means PIII or Xeon solutions only -- there aren't any 
dual AMD boards with more than one PCI bus yet.  You'll want to look at 
E7500 (or E7501 if they're in the channel yet) based boards.  I've got a 
system on just like that on a Supermicro P4DPE-G2 board that works rather 
nicely.

As for comments, in my testing on a single 3ware board (in hardware 
RAID5), ext3's write speeds sucked.  I'd suggest including XFS in your 
testing.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: building a large/fast server

[Mikael Abrahamsson]

On Wed, 29 Jan 2003, Joshua Baker-LePain wrote:

> As for comments, in my testing on a single 3ware board (in hardware 
> RAID5), ext3's write speeds sucked.  I'd suggest including XFS in your 
> testing.

Will xfs help write speeds with 3ware hardware raid5? You have tested 
this? Just as you have experienced, my 3ware raid5 write speeds are lousy, 
approx 5-15 megs/s.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: building a large/fast server

[Joshua Baker-LePain]

On Wed, 29 Jan 2003 at 9:23pm, Mikael Abrahamsson wrote

> On Wed, 29 Jan 2003, Joshua Baker-LePain wrote:
> 
> > As for comments, in my testing on a single 3ware board (in hardware 
> > RAID5), ext3's write speeds sucked.  I'd suggest including XFS in your 
> > testing.
> 
> Will xfs help write speeds with 3ware hardware raid5? You have tested 
> this? Just as you have experienced, my 3ware raid5 write speeds are lousy, 
> approx 5-15 megs/s.

Here's the output of a bonnie++ run on a single 3ware hardware RAID5 
array.  The system is dual 2.2GHz Xeons on a SuperMicro P4DPE-G2 with 
2GB of RAM.  I'm using the 160GB Maxtors, and I've setup a hot spare (so 
it's a 7 drive array).  I'm running Red Hat 7.3 with kernel 2.4.18-17 
patched with XFS 1.2pre3 from SGI:

Version 1.02c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
$BOX             4G 24222  99 56221  16 31373  13 24478  95 204587  39 394.6   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1901  15 +++++ +++  1099   8   979   8 +++++ +++   960   7
$BOX,4G,24222,99,56221,16,31373,13,24478,95,204587,39,394.6,2,16,1901,15,+++++,+++,1099,8,979,8,+++++,+++,960,7


-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: building a large/fast server

[Mikael Abrahamsson]

On Wed, 29 Jan 2003, Andy Arvai wrote:

> I'm in the process of designing a large server system. The primary
> consideration will be i/o performance (block read and write of large
> files and maximum i/o's per second) and a filesystem size of 2TB. I'm
> planning to go with 2 3ware 7500-8's, 16 Maxtor 4G160J8's (160GB) and
> do software raid0 between the 7500-8's (raid5). The filesystem will
> either be Reiser or Ext3. Any suggestions for fast dual cpu
> motherboards or comments are welcome. Thanks for any information.

1:st up if you want high write speeds:

Don't use the internal RAID5 of the 3wares, use software RAID5. Hmm, if 
you really want high write speeds, skip RAID5 altogether and go for 
mirroring.

If you want to be able to transfer these files quickly over the network, 
skip the SMP and go for the fastest possible single cpu system you can 
get, get one with multiple PCI buses that can handle 66MHz/64bit PCI 
(serverworks chipset for instance). Get a good gigabit network card, I 
personally like the Syskonnect series.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: building a large/fast server

[Joshua Baker-LePain]

On Wed, 29 Jan 2003 at 9:22pm, Mikael Abrahamsson wrote

> On Wed, 29 Jan 2003, Andy Arvai wrote:
> 
> > I'm in the process of designing a large server system. The primary
> > consideration will be i/o performance (block read and write of large
> > files and maximum i/o's per second) and a filesystem size of 2TB. I'm
> > planning to go with 2 3ware 7500-8's, 16 Maxtor 4G160J8's (160GB) and
> > do software raid0 between the 7500-8's (raid5). The filesystem will
> > either be Reiser or Ext3. Any suggestions for fast dual cpu
> > motherboards or comments are welcome. Thanks for any information.
> 
> 1:st up if you want high write speeds:
> 
> Don't use the internal RAID5 of the 3wares, use software RAID5. Hmm, if 
> you really want high write speeds, skip RAID5 altogether and go for 
> mirroring.

That depends on your definition of high, and the application.  Using XFS, 
I can get 55MB/s writing to a single 3ware in hardware RAID5 (bonnie++).  
Stripe across two of those, and you're over 100MB/s (I've tested this, 
seen it, but don't have the data handy), which is more than enough to fill 
a gigabit pipe.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: building a large/fast server

[Illtud Daniel]

Andy,

> I'm in the process of designing a large server system. The primary
> consideration will be i/o performance (block read and write of large
> files and maximum i/o's per second) and a filesystem size of 2TB. I'm
> planning to go with 2 3ware 7500-8's, 16 Maxtor 4G160J8's (160GB) and
> do software raid0 between the 7500-8's (raid5).

I did this with dual Chaparral controllers on Fortra enclosures
with U160 disks. (h/w RAID 5 on the disks, dual U160 scsi 66/64
on the server, stripe both with md RAID 0). Two points:

1. It screamed. I've got all the iozone results here
somewhere for XFS, Reiser, Ext3 & Ext2. Reading is 400-500MB/s
(sometimes more, but that must be cache, since the bus can't
be much faster), writing is 150-250MB/s. IIRC, the filesystems
made little difference. I was going to publish these results,
but had a problem on production testing (ext3, if you're asking):

2. It crashed and burned badly. A disk problem became a controller
problem and one of the RAID 5 arrays went down, the other became
degraded (this is one of 5 similar Chaparral/Fortra setups we've
got, no problem with the others - 24x7x52. But don't touch IBM
ultrastars with a bargepole) which didn't do the md RAID 0 array
any favours. Eventually, we got the RAID 5 arrays up & running
again, and could mount the RAID 0. Everything seemed to be OK, but
accessing files would give i/o errors all over the shop. The
whole caboodle got powered down and I'm waiting for a free couple
of days to attempt a thorough post-mortem. It's been like that
for months now...

My tip - don't stripe two RAID5s on RAID0, cos when things go
wrong, they do so in style, and it's very hard to debug. Nice
and fast if you've got the (SCSI) kit, though.

Anybody interested in an autopsy and get to Aberystwyth would be
very welcome...

-- 
Illtud Daniel                                 illtud.daniel@llgc.org.uk
Uwch Ddadansoddwr Systemau                       Senior Systems Analyst
Llyfrgell Genedlaethol Cymru                  National Library of Wales
Yn siarad drosof fy hun, nid LlGC   -  Speaking personally, not for NLW