large devices (3ware)

large devices (3ware)

[Mikael Abrahamsson]

I am considering a 3ware 7500-12 (12 channel IDE raid controller) and 12 
120 gig drives in a single large RAID5. Am I correct that Linux (latest 
Redhat 7.3 shipped 2.4.18 kernel) can handle devices up to 2 terabyte and 
that this raid5 (1320 gig) would work as expected as one single file 
system (ext3) ? Anyone actually have something similar up and running and 
can verify that it works?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: large devices (3ware)

[Effrem Norwood]

> I am considering a 3ware 7500-12 (12 channel IDE raid controller) and 12 
> 120 gig drives in a single large RAID5. Am I correct that Linux (latest 
> Redhat 7.3 shipped 2.4.18 kernel) can handle devices up to 2 terabyte and 
> that this raid5 (1320 gig) would work as expected as one single file 
> system (ext3) ? Anyone actually have something similar up and running and 
> can verify that it works?

Hi Mikael,

Yes, it will work just fine. We have several of the 3ware 7500-12's here with
exactly the setup you're looking at, save for we use Debian. 3ware makes a
driver for Redhat and the kernel if recompiled with the option has one built
in. It's a solid setup.

Best,

Effrem Norwood

Re: large devices (3ware)

[Maurice Hilarius]

With regards to your message at 02:37 AM 8/21/02, Mikael Abrahamsson. Where 
you stated:

>I am considering a 3ware 7500-12 (12 channel IDE raid controller) and 12
>120 gig drives in a single large RAID5. Am I correct that Linux (latest
>Redhat 7.3 shipped 2.4.18 kernel) can handle devices up to 2 terabyte and
>that this raid5 (1320 gig) would work as expected as one single file
>system (ext3) ? Anyone actually have something similar up and running and
>can verify that it works?


It works just great.
The 3Ware have the new ATA addressing and handle large drives just fine.
Actually the 120GB does not need this newer addressing. Only drives over 
128GB need it.
We use the WD and also the Maxtor 160GB with 3Ware.
You will get a RAID volume with a formatted capacity of about 1.221TB
One thing I can point out is that the "slower" Maxtor 5400rpm drives 
actually produce very similar performance results compared to the "faster" 
7200rpm models. While they spin slower, the track data density is very 
high, and this seems to make up for the slower spin rate to a great degree.
Using a set of 160GB, 5400 rpm drives we get very similar performance to 
the 7200rpm WDs.
Using the 160GB drives we build 16 drive RAID 5 arrays (using 2 of the 
3Ware cards, and can build a 2TB partition and if we use all drive, another 
311MB for a second partition.
Works great.
Here's a bonnie test on 8 of the WD1200-JB, using different options. Read 
the inline notes by the bonnie++ tests for info.
Note that we are using a 4GB file size in the bonnie++ tests, so as to 
avoid artificially high readings due to OS buffers, drive cacheing, etc.:

Hardware:

SuperMicro P4DL6 Motherboard
1024 MB of RAM
3ware 7850 RAID Controller
Dual P4 Xeons 2GHz
8 x 120GB Western Digital c/w 8MB Cache WD1200-JB

raidtab file:

raiddev             /dev/md2
raid-level                  5
nr-raid-disks               8
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
     device          /dev/sda3
     raid-disk     0
     device          /dev/sdb2
     raid-disk     1
     device          /dev/sdc2
     raid-disk     2
     device          /dev/sdd2
     raid-disk     3
     device          /dev/sde2
     raid-disk     4
     device          /dev/sdf2
     raid-disk     5
     device          /dev/sdg2
     raid-disk     6
     device          /dev/sdh2
     raid-disk     7
raiddev             /dev/md0
raid-level                  5
nr-raid-disks               8
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
     device          /dev/sda9
     raid-disk     0
     device          /dev/sdb9
     raid-disk     1
     device          /dev/sdc9
     raid-disk     2
     device          /dev/sdd9
     raid-disk     3
     device          /dev/sde9
     raid-disk     4
     device          /dev/sdf9
     raid-disk     5
     device          /dev/sdg9
     raid-disk     6
     device          /dev/sdh9
     raid-disk     7
raiddev             /dev/md1
raid-level                  5
nr-raid-disks               8
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
     device          /dev/sda8
     raid-disk     0
     device          /dev/sdb8
     raid-disk     1
     device          /dev/sdc8
     raid-disk     2
     device          /dev/sdd8
     raid-disk     3
     device          /dev/sde8
     raid-disk     4
     device          /dev/sdf8
     raid-disk     5
     device          /dev/sdg8
     raid-disk     6
     device          /dev/sdh8
     raid-disk     7
raiddev             /dev/md4
raid-level                  5
nr-raid-disks               8
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
     device          /dev/sda6
     raid-disk     0
     device          /dev/sdb6
     raid-disk     1
     device          /dev/sdc6
     raid-disk     2
     device          /dev/sdd6
     raid-disk     3
     device          /dev/sde6
     raid-disk     4
     device          /dev/sdf6
     raid-disk     5
     device          /dev/sdg6
     raid-disk     6
     device          /dev/sdh6
     raid-disk     7
raiddev             /dev/md5
raid-level                  5
nr-raid-disks               8
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
     device          /dev/sda7
     raid-disk     0
     device          /dev/sdb7
     raid-disk     1
     device          /dev/sdc7
     raid-disk     2
     device          /dev/sdd7
     raid-disk     3
     device          /dev/sde7
     raid-disk     4
     device          /dev/sdf7
     raid-disk     5
     device          /dev/sdg7
     raid-disk     6
     device          /dev/sdh7
     raid-disk     7
raiddev             /dev/md3
raid-level                  5
nr-raid-disks               8
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0
     device          /dev/sda5
     raid-disk     0
     device          /dev/sdb5
     raid-disk     1
     device          /dev/sdc5
     raid-disk     2
     device          /dev/sdd5
     raid-disk     3
     device          /dev/sde5
     raid-disk     4
     device          /dev/sdf5
     raid-disk     5
     device          /dev/sdg5
     raid-disk     6
     device          /dev/sdh5
     raid-disk     7

Test 1:  Command Line: bonnie++ -d /data -s 4g -r 1024 -u root > /root/bonoutb
* This test had the driver from the 7.3 RedHat Install and the RAID was 
still resyncing after the OS install.
*3ware Storage Controller device driver for Linux v1.02.00.020

Version 1.02c       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dynamic-188.hard 4G 15554  99 76414  98 49839  52 22177  99 155182  53 
373.9   2                    ------Sequential Create------ --------Random 
Create--------
                     -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP  /sec %CP
                  16  2101  97 +++++ +++ 29801 100  1878  97 +++++ 
+++  3738  98
dynamic-188.harddata.net,4G,15554,99,76414,98,49839,52,22177,99,155182,53,373.9,2,16,2101,97,+++++,+++,29801,100,1878,97,+++++,+++,3738,98


Test3: Command line: bonnie++ -d /data -s 4g -r 1024 -u root > /root/bonoutd
* This test is the stock driver from the 7.3 RedHat Install and the RAID is 
done resyncing.
*3ware Storage Controller device driver for Linux v1.02.00.020

Version 1.02c       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dynamic-188.hard 4G 17293  99 77746  99 51494  54 22826  98 157093  50 
393.7   3                    ------Sequential Create------ --------Random 
Create--------
                     -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP  /sec %CP
                  16  2081  97 +++++ +++ 32012  99  2200  99 +++++ 
+++  5846 100
dynamic-188.harddata.net,4G,17293,99,77746,99,51494,54,22826,98,157093,50,393.7,3,16,2081,97,+++++,+++,32012,99,2200,99,+++++,+++,5846,100

Test4: Command line: bonnie++ -d /data -s 4096 -r 1024 -u root
* This test is the stock driver from the 7.3 RedHat Install and the RAID is 
done resyncing.
*3ware Storage Controller device driver for Linux v1.02.00.020

Version 1.02c       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dynamic-188.hard 4G 17324  99 82418  99 50897  55 22718  99 154522  46 
389.9   2                    ------Sequential Create------ --------Random 
Create--------
                     -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP  /sec %CP
                  16  2075  99 +++++ +++ 32095  99  2202  98 +++++ 
+++  5848 100
dynamic-188.harddata.net,4G,17324,99,82418,99,50897,55,22718,99,154522,46,389.9,2,16,2075,99,+++++,+++,32095,99,2202,98,+++++,+++,5848,100

Test5: Command line: bonnie++ -d /data -s 4096 -r 1024 -u root
* This test is with the newest driver available from 3ware.
*3ware Storage Controller device driver for Linux v1.02.00.025

Version 1.02c       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dynamic-188.hard 4G 17299  99 82519  99 51073  54 22969  99 158657  46 
394.5   2                    ------Sequential Create------ --------Random 
Create--------
                     -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP  /sec %CP
                  16  2078  98 +++++ +++ 31989  99  2139  98 +++++ 
+++  5129  99
dynamic-188.harddata.net,4G,17299,99,82519,99,51073,54,22969,99,158657,46,394.5,2,16,2078,98,+++++,+++,31989,99,2139,98,+++++,+++,5129,99

Test6: Command line: bonnie++ -d /data -s 4g -r 1024 -u root
* This test is with the newest driver available from 3ware.
*3ware Storage Controller device driver for Linux v1.02.00.025

Version 1.02c       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
dynamic-188.hard 4G 17465  99 78499  99 50943  53 22833  98 150570  44 
384.8   2                    ------Sequential Create------ --------Random 
Create--------
                     -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP  /sec %CP
                  16  2083  98 +++++ +++ 31929  99  2158  99 +++++ 
+++  5029  98
dynamic-188.harddata.net,4G,17465,99,78499,99,50943,53,22833,98,150570,44,384.8,2,16,2083,98,+++++,+++,31929,99,2158,99,+++++,+++,5029,98



With our best regards,

Maurice W. Hilarius       Telephone: 01-780-456-9771
Hard Data Ltd.               FAX:       01-780-456-9772
11060 - 166 Avenue        mailto:maurice@harddata.com
Edmonton, AB, Canada      http://www.harddata.com/
    T5X 1Y3

2.3TB RAID5 NAS server - dual AthlonMP CPU, Linux, $10,995 CAD / $6850 USD

Re: large devices (3ware)

[jlewis]

On Wed, 21 Aug 2002, Effrem Norwood wrote:

> Yes, it will work just fine. We have several of the 3ware 7500-12's here with
> exactly the setup you're looking at, save for we use Debian. 3ware makes a
> driver for Redhat and the kernel if recompiled with the option has one built
> in. It's a solid setup.

What sorts of cases are you using that allow for mounting 12 drives close 
enough to the controller for all the cables to reach?

----------------------------------------------------------------------
 Jon Lewis *jlewis@lewis.org*|  I route
 System Administrator        |  therefore you are
 Atlantic Net                |  
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

Re: large devices (3ware)

[Mike Dresser]

On Wed, 21 Aug 2002 jlewis@lewis.org wrote:

> On Wed, 21 Aug 2002, Effrem Norwood wrote:
>
> > Yes, it will work just fine. We have several of the 3ware 7500-12's here with
> > exactly the setup you're looking at, save for we use Debian. 3ware makes a
> > driver for Redhat and the kernel if recompiled with the option has one built
> > in. It's a solid setup.
>
> What sorts of cases are you using that allow for mounting 12 drives close
> enough to the controller for all the cables to reach?

I was looking at a rack mount case today, that would let me put 23 drives
in a 7" high case.  All those cables should reach with standard 18"
cables.

About 450 bucks for the case, and around 900 for the adapters.

:D

Re: large devices (3ware)

[Mike Dresser]

Gah, I can't count/read.  There is a case that holds 15 SCA or 14 IDE in
3U (5.75") though, and one that holds 15 IDE in 4U (7")

On Wed, 21 Aug 2002, Mike Dresser wrote:

> I was looking at a rack mount case today, that would let me put 23 drives
> in a 7" high case.  All those cables should reach with standard 18"
> cables.

Re: large devices (3ware)

[Maurice Hilarius]

With regards to your message at 01:07 PM 8/21/02, jlewis@lewis.org. Where 
you stated:

> > Yes, it will work just fine. We have several of the 3ware 7500-12's 
> here with
> > exactly the setup you're looking at, save for we use Debian. 3ware makes a
> > driver for Redhat and the kernel if recompiled with the option has one 
> built
> > in. It's a solid setup.
>
>What sorts of cases are you using that allow for mounting 12 drives close
>enough to the controller for all the cables to reach?


We have 3U cases with 8 or 12 hot-swap IDE trays, 4U cases with 16 IDE trays.
Also towers with support for up to 16 IDE trays.
3Ware can support a 36" IDE cable using the right cables.
In our 3U the cables are only 18" long, and in the 4U they are 24" long.



With our best regards,

Maurice W. Hilarius       Telephone: 01-780-456-9771
Hard Data Ltd.               FAX:       01-780-456-9772
11060 - 166 Avenue        mailto:maurice@harddata.com
Edmonton, AB, Canada      http://www.harddata.com/
    T5X 1Y3

2.3TB RAID5 NAS server - dual AthlonMP CPU, Linux, $10,995 CAD / $6850 USD

Re: large devices (3ware)

[Alvin Oga]

On Wed, 21 Aug 2002, Mike Dresser wrote:

> On Wed, 21 Aug 2002 jlewis@lewis.org wrote:
> 

...

> > What sorts of cases are you using that allow for mounting 12 drives close
> > enough to the controller for all the cables to reach?
> 
> I was looking at a rack mount case today, that would let me put 23 drives
> in a 7" high case.  All those cables should reach with standard 18"
> cables.
> 
> About 450 bucks for the case, and around 900 for the adapters.
> 

3ware has modified hardware that allows it to drive longer ide cables
( that exceed the 18" cable length )

3ware has a 8-port ide card ... for 8 master ide disks but its
still on one pci buss ( i think thats the bottle neck )

if you have 12, 26 ide disks... you'd need a 2nd pci slot...but again
on the one pci buss...

8 disks ( 160GB each ) is 1TeraByte of data that can be moving back and
forth on that pci buss...

i think it'd be better to put one ide controller ( like the 3-ware )
into one motherboard... and use a different server for the next 8 disks

-- am looking for benchmarks of 8-24-48 hard disks systems

we have 8-drive 1U chassis  - cabling is an issue... due to 18" maximum
but so far, we havent seen any major problems w/ the 24" and 26" length
cables..
	http://www.Linux-1U.net/Dwg/jpg.sm/c2610.jpg

thanx
alvin

Re: large devices (3ware)

[Trent Piepho]

On Wed, 21 Aug 2002, Mike Dresser wrote:
> 
> I was looking at a rack mount case today, that would let me put 23 drives
> in a 7" high case.  All those cables should reach with standard 18"
> cables.

23!?  I've seen a rackmount case that has 16 drive bays, but not one for 23. 
Is there a web site I can see this case at?

Re: large devices (3ware)

[Alvin Oga]

hi ya

the rackmount cases is typically 17" or 17.5" wide that fits into
the so-called 19" cabinets/racks

after alowing for 1/8" screw holes on the chassis to mount it to the racks

you'd be looking at 16 or 17 ide disks on its side ( 3U cases and the
disks are 4" wide - plenty of room for removable trays )
	- not sure if you can do a real "hot swap ide" yet ??

have fun
alvin

On Wed, 21 Aug 2002, Trent Piepho wrote:

> On Wed, 21 Aug 2002, Mike Dresser wrote:
> > 
> > I was looking at a rack mount case today, that would let me put 23 drives
> > in a 7" high case.  All those cables should reach with standard 18"
> > cables.
> 
> 23!?  I've seen a rackmount case that has 16 drive bays, but not one for 23. 
> Is there a web site I can see this case at?
> 

Re: large devices (3ware)

[Maurice Hilarius]

With regards to your message at 01:53 PM 8/21/02, Alvin Oga. Where you stated:
>3ware has a 8-port ide card ... for 8 master ide disks but its
>still on one pci buss ( i think thats the bottle neck )
And a 12 port card too.

>if you have 12, 26 ide disks... you'd need a 2nd pci slot...but again
>on the one pci buss...
We do this all the time. 2 of the cards. One array.

>i think it'd be better to put one ide controller ( like the 3-ware )
>into one motherboard... and use a different server for the next 8 disks
Don't need it.
Bottleneck is that the card is 64 bit, but only 33MHz.
It's throughput maxes around 160MB/sec.
This is a limitation of the PCI at 64/33


>-- am looking for benchmarks of 8-24-48 hard disks systems

Here is for 2 x 8 port cards, 16 x Maxtor 160GB.
RAID5, on a Tyan S2460 AthlonMP board with a single Duron CPU.
Cards in JBOD mode, using Linux software RAID.

Version 1.02a       ------Sequential Output------ --Sequential Input- --Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
raiddawg         4G 17752  89 78709  44 55648  32 20109  98 162752  67 
224.3   1
                     ------Sequential Create------ --------Random 
Create--------
                     -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
%CP  /sec %CP
                  16  1240  99 +++++ +++ +++++ +++  1284  99 +++++ 
+++  3581  99
raiddawg,4G,17752,89,78709,44,55648,32,20109,98,162752,67,224.3,1,16,1240,99,+++++,+++,+++++,+++,1284,99,+++++,+++,3581,99

With a faster CPU the writes go up higher, typically up to 90MB/sec.
Reads still max around 160MB/sec.
With a single card and 8 drives we get the same type of numbers.
With 2 of the 12 port cards and 24 disks we still get the same numbers.

Note these tests use 4GB file size.
If you use some smaller size you can get higher figures, but is essentially 
bullshit, as these are then influenced by Linux cacheing, drive caches, etc..


With our best regards,

Maurice W. Hilarius       Telephone: 01-780-456-9771
Hard Data Ltd.               FAX:       01-780-456-9772
11060 - 166 Avenue        mailto:maurice@harddata.com
Edmonton, AB, Canada      http://www.harddata.com/
    T5X 1Y3

2.3TB RAID5 NAS server - dual AthlonMP CPU, Linux, $10,995 CAD / $6850 USD

Re: large devices (3ware) - thanx for the info

[Alvin Oga]

hi ya maurice

yes... thankx for the benchmark numbers ...

good that the pci buss is the bottle neck for the disk io transfers
( which would than have one  wonder if the 64bit pic buss
( or the so-called 533FSB is any better 

and yes... the file sizes does affect overall transfer speeds etc

thanx
alvin

and nope... we only use 1U chassis... so we're stuck with one pci card
using the normal off-the-shelf atx stuff

On Wed, 21 Aug 2002, Maurice Hilarius wrote:

> With regards to your message at 01:53 PM 8/21/02, Alvin Oga. Where you stated:
> >3ware has a 8-port ide card ... for 8 master ide disks but its
> >still on one pci buss ( i think thats the bottle neck )
> And a 12 port card too.
> 
> >if you have 12, 26 ide disks... you'd need a 2nd pci slot...but again
> >on the one pci buss...
> We do this all the time. 2 of the cards. One array.
> 
> >i think it'd be better to put one ide controller ( like the 3-ware )
> >into one motherboard... and use a different server for the next 8 disks
> Don't need it.
> Bottleneck is that the card is 64 bit, but only 33MHz.
> It's throughput maxes around 160MB/sec.
> This is a limitation of the PCI at 64/33
> 
> 
> >-- am looking for benchmarks of 8-24-48 hard disks systems
> 
> Here is for 2 x 8 port cards, 16 x Maxtor 160GB.
> RAID5, on a Tyan S2460 AthlonMP board with a single Duron CPU.
> Cards in JBOD mode, using Linux software RAID.
> 
> Version 1.02a       ------Sequential Output------ --Sequential Input- --Random-
>                      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
> raiddawg         4G 17752  89 78709  44 55648  32 20109  98 162752  67 
> 224.3   1
>                      ------Sequential Create------ --------Random 
> Create--------
>                      -Create-- --Read--- -Delete-- -Create-- --Read--- 
> -Delete--
>                files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec 
> %CP  /sec %CP
>                   16  1240  99 +++++ +++ +++++ +++  1284  99 +++++ 
> +++  3581  99
> raiddawg,4G,17752,89,78709,44,55648,32,20109,98,162752,67,224.3,1,16,1240,99,+++++,+++,+++++,+++,1284,99,+++++,+++,3581,99
> 
> With a faster CPU the writes go up higher, typically up to 90MB/sec.
> Reads still max around 160MB/sec.
> With a single card and 8 drives we get the same type of numbers.
> With 2 of the 12 port cards and 24 disks we still get the same numbers.
> 
> Note these tests use 4GB file size.
> If you use some smaller size you can get higher figures, but is essentially 
> bullshit, as these are then influenced by Linux cacheing, drive caches, etc..
> 
> 
> With our best regards,
> 
> Maurice W. Hilarius       Telephone: 01-780-456-9771
> Hard Data Ltd.               FAX:       01-780-456-9772
> 11060 - 166 Avenue        mailto:maurice@harddata.com
> Edmonton, AB, Canada      http://www.harddata.com/
>     T5X 1Y3
> 
> 2.3TB RAID5 NAS server - dual AthlonMP CPU, Linux, $10,995 CAD / $6850 USD
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>