1x 3ware controllers vs. 2x 3ware controllers

1x 3ware controllers vs. 2x 3ware controllers

[Adam Hunt]

I'm looking at building what I guess you could call a budget NAS
array.  I'm looking to start with four 250MB SATA drives on 3ware
8506-4LP controller.  I eventually want to move to eight drives in the
box.  Is there any reason to expect that the performance of eight
drives spread across two 8506-4LP controllers is going to be any worse
then the same eight drives all on one 8506-8?

--adam

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Scott T. Smith]

On Thu, 2004-07-29 at 20:53, Adam Hunt wrote:
> I'm looking at building what I guess you could call a budget NAS
> array.  I'm looking to start with four 250MB SATA drives on 3ware
> 8506-4LP controller.  I eventually want to move to eight drives in the
> box.  Is there any reason to expect that the performance of eight
> drives spread across two 8506-4LP controllers is going to be any worse
> then the same eight drives all on one 8506-8?

are they on the same PCI bus or not?

I've had issues where removing a drive (without notifying the 3ware
card) causes all access to that controller to lock up about 20 seconds
later.  It only locks up though for about 10-15 seconds and then
continues normally, so depending on your demands, that may be just fine
for you.

	Scott

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Luca Berra]

On Thu, Jul 29, 2004 at 08:53:47PM -0700, Adam Hunt wrote:
>I'm looking at building what I guess you could call a budget NAS
>array.  I'm looking to start with four 250MB SATA drives on 3ware
>8506-4LP controller.  I eventually want to move to eight drives in the
>box.  Is there any reason to expect that the performance of eight
>drives spread across two 8506-4LP controllers is going to be any worse
>then the same eight drives all on one 8506-8?

maybe pci bus issues, but you don't tell us the intended layout.

regards,
L.

-- 
Luca Berra -- bluca@comedia.it
        Communication Media & Services S.r.l.
 /"\
 \ /     ASCII RIBBON CAMPAIGN
  X        AGAINST HTML MAIL
 / \

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Watts]

> I'm looking at building what I guess you could call a budget NAS
> array.  I'm looking to start with four 250MB SATA drives on 3ware
> 8506-4LP controller.  I eventually want to move to eight drives in the
> box.  Is there any reason to expect that the performance of eight
> drives spread across two 8506-4LP controllers is going to be any worse
> then the same eight drives all on one 8506-8?

With the performance issues I'm seeing with the 8506-4LP's I have, I wouldn't 
recommend them to anyone currently...

The thread on LKML from a few days ago about mke2fs -j and 8506-4LP says it 
all, but basically and reasonable amount of I/O brings a Dual Opteron system 
to its knees.
And by reasonable I mean copying ISO images from a usb2 drive to a raid 5 (4 x 
250GB maxtor) or even just formatting a 600MB partition.

Mark.

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Watts]

> > I'm looking at building what I guess you could call a budget NAS
> > array.  I'm looking to start with four 250MB SATA drives on 3ware
> > 8506-4LP controller.  I eventually want to move to eight drives in the
> > box.  Is there any reason to expect that the performance of eight
> > drives spread across two 8506-4LP controllers is going to be any worse
> > then the same eight drives all on one 8506-8?
>
> With the performance issues I'm seeing with the 8506-4LP's I have, I
> wouldn't recommend them to anyone currently...
>
> The thread on LKML from a few days ago about mke2fs -j and 8506-4LP says it
> all, but basically and reasonable amount of I/O brings a Dual Opteron
> system to its knees.
> And by reasonable I mean copying ISO images from a usb2 drive to a raid 5
> (4 x 250GB maxtor) or even just formatting a 600MB partition.

Should be 600GB but I haven't had coffee yet...

Mark.

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Scott T. Smith]

On Fri, 2004-07-30 at 00:15, Mark Watts wrote:
> > I'm looking at building what I guess you could call a budget NAS
> > array.  I'm looking to start with four 250MB SATA drives on 3ware
> > 8506-4LP controller.  I eventually want to move to eight drives in the
> > box.  Is there any reason to expect that the performance of eight
> > drives spread across two 8506-4LP controllers is going to be any worse
> > then the same eight drives all on one 8506-8?

oh one thing I forgot -- are you going to use software RAID or hardware
RAID?  Because with hardware RAID, I don't think you can spread a RAID
across two controllers.  That might affect your decision.

> With the performance issues I'm seeing with the 8506-4LP's I have, I wouldn't 
> recommend them to anyone currently...
> 
> The thread on LKML from a few days ago about mke2fs -j and 8506-4LP says it 
> all, but basically and reasonable amount of I/O brings a Dual Opteron system 
> to its knees.
> And by reasonable I mean copying ISO images from a usb2 drive to a raid 5 (4 x 
> 250GB maxtor) or even just formatting a 600MB partition.

I've been able to get nearly 1.5 Gbits/sec off of an 8506-8, with 8
250GB disks...  of course those are all 1 megabyte reads (but from
random locations).  Is your problem related to # of IO's, or size of
transfer?

	Scott

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Watts]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>> With the performance issues I'm seeing with the 8506-4LP's I have, I 
>wouldn't 
>> recommend them to anyone currently...
>> 
>> The thread on LKML from a few days ago about mke2fs -j and 8506-4LP says it 
>> all, but basically and reasonable amount of I/O brings a Dual Opteron 
>system 
>> to its knees.
>> And by reasonable I mean copying ISO images from a usb2 drive to a raid 5 
>(4 x 
>> 250GB maxtor) or even just formatting a 600MB partition.
>
>I've been able to get nearly 1.5 Gbits/sec off of an 8506-8, with 8
>250GB disks...  of course those are all 1 megabyte reads (but from
>random locations).  Is your problem related to # of IO's, or size of
>transfer?

# hdparm -tT /dev/sda

/dev/sda:
 Timing buffer-cache reads:   2232 MB in  2.00 seconds = 1115.61 MB/sec
 Timing buffered disk reads:  190 MB in  3.03 seconds =  62.76 MB/sec

bonnie++ concurs with this and gives us about 22MB/sec write performance.

Basically anything involving I/O is causing the system to crawl.
I can reliably reproduce this by doing:

	mke2fs -j /dev/sda7

Where sda7 is a 600GB partition.

Basically the system load goes orbital and responsivness goes out the window.
gkrellm only shows ~1MB/sec of disk activity and the processors (Dual Opteron) 
are twiddling their thumbs (top shows 100% 'wa' state for both processors), 
yet I can hardly do anything with any open applications.

Kernel is a 2.6.8rc1 kernel.org job.

- -- 
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBCiAKBn4EFUVUIO0RAsmIAJ4zaXUPemotyUDFTjj9soa4ZSD+GwCfYQMg
jxmu5wO9o4XXh+0p54zKVio=
=zWAH
-----END PGP SIGNATURE-----

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Joshua Baker-LePain]

On Fri, 30 Jul 2004 at 11:16am, Mark Watts wrote

> Basically anything involving I/O is causing the system to crawl.
> I can reliably reproduce this by doing:
> 
> 	mke2fs -j /dev/sda7
> 
> Where sda7 is a 600GB partition.
> 
> Basically the system load goes orbital and responsivness goes out the window.
> gkrellm only shows ~1MB/sec of disk activity and the processors (Dual Opteron) 
> are twiddling their thumbs (top shows 100% 'wa' state for both processors), 
> yet I can hardly do anything with any open applications.
> 
> Kernel is a 2.6.8rc1 kernel.org job.

Make sure that your driver and firmware versions match (check the 3ware 
site).  Also, have you tried other FSs?  I get *very* good performance out 
of 7500-8s using XFS on 2.4 (and hardware RAID5+software RAID0).  I 
haven't tried 2.6 yet.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Watts]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> On Fri, 30 Jul 2004 at 11:16am, Mark Watts wrote
>
> > Basically anything involving I/O is causing the system to crawl.
> > I can reliably reproduce this by doing:
> >
> > 	mke2fs -j /dev/sda7
> >
> > Where sda7 is a 600GB partition.
> >
> > Basically the system load goes orbital and responsivness goes out the
> > window. gkrellm only shows ~1MB/sec of disk activity and the processors
> > (Dual Opteron) are twiddling their thumbs (top shows 100% 'wa' state for
> > both processors), yet I can hardly do anything with any open
> > applications.
> >
> > Kernel is a 2.6.8rc1 kernel.org job.
>
> Make sure that your driver and firmware versions match (check the 3ware
> site).  Also, have you tried other FSs?  I get *very* good performance out
> of 7500-8s using XFS on 2.4 (and hardware RAID5+software RAID0).  I
> haven't tried 2.6 yet.

I'm running the latest firmware from www.3ware.com (FE8S 1.05.00.068)
Driver is v1.26.00.039 from kernel 2.6.8rc1

Obviously these aren't the same, but I can only see drivers for RedHat or 
generic 2.4 kernels on the 3ware site, not 2.6.x

Mark.

- -- 
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBCkwJBn4EFUVUIO0RAn4uAKDfFSFhbINbNzy64XJHfcqE3ZxrZwCfWhrE
r9pxnUC81Uwdm3OCc8JzPy8=
=aEJ4
-----END PGP SIGNATURE-----

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Joshua Baker-LePain]

On Fri, 30 Jul 2004 at 2:24pm, Mark Watts wrote

> > Make sure that your driver and firmware versions match (check the 3ware
> > site).  Also, have you tried other FSs?  I get *very* good performance out
> > of 7500-8s using XFS on 2.4 (and hardware RAID5+software RAID0).  I
> > haven't tried 2.6 yet.
> 
> I'm running the latest firmware from www.3ware.com (FE8S 1.05.00.068)
> Driver is v1.26.00.039 from kernel 2.6.8rc1
> 
> Obviously these aren't the same, but I can only see drivers for RedHat or 
> generic 2.4 kernels on the 3ware site, not 2.6.x

Bah.  They have the 2.6 driver under "In Engineering Phase", and it's that 
same version.  For 2.4 they tie certain driver and firmware versions 
together as a "software release", but apparently they aren't doing this 
for 2.6 yet.  Sorry for the noise.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Marc Bevand]

Mark Watts wrote:
> With the performance issues I'm seeing with the 8506-4LP's I have, I wouldn't 
> recommend them to anyone currently...
> 
> The thread on LKML from a few days ago about mke2fs -j and 8506-4LP says it 
> all, but basically and reasonable amount of I/O brings a Dual Opteron system 
> to its knees.
> And by reasonable I mean copying ISO images from a usb2 drive to a raid 5 (4 x 
> 250GB maxtor) or even just formatting a 600MB partition.

I would be interested to see the output of 'vmstat 1' while your system is
so slow.

IMHO you shouldn't draw such conclusion ("reasonable amount of I/O brings a
Dual Opteron system to its knees") from your particular case. The 3 HT links
of the Opteron make this CPU particularly adapted to I/O operations.

Personnaly, on a dual Opteron, I am able to read datas from 4 SATA disks at
about 225 MB/s, with CPU time used at about 32%, awd with a system still
reasonably responsive.

-- 
Marc Bevand                          http://www.epita.fr/~bevand_m
Computer Science School EPITA - System, Network and Security Dept.

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Watts]

> Mark Watts wrote:
> > With the performance issues I'm seeing with the 8506-4LP's I have, I
> > wouldn't recommend them to anyone currently...
> >
> > The thread on LKML from a few days ago about mke2fs -j and 8506-4LP says
> > it all, but basically and reasonable amount of I/O brings a Dual Opteron
> > system to its knees.
> > And by reasonable I mean copying ISO images from a usb2 drive to a raid 5
> > (4 x 250GB maxtor) or even just formatting a 600MB partition.
>
> I would be interested to see the output of 'vmstat 1' while your system is
> so slow.
>
> IMHO you shouldn't draw such conclusion ("reasonable amount of I/O brings a
> Dual Opteron system to its knees") from your particular case. The 3 HT
> links of the Opteron make this CPU particularly adapted to I/O operations.

Well its the only conclusion I *can* come to at this time.
When you move from a UP 1.8Ghz P4 with single EIDE disks to SMP Opteron with 4 
times the ram and a hardware raid card, you tend to assume that performance 
in all areas will go up.
When it doesn't, and you find yourself watching screen redraws while you 
format a 600GB partition (ext3) you do feel the need to blame something :)

I'm all ears for suggestions on what me be wrong or things I can try to 
improve performance.


>
> Personnaly, on a dual Opteron, I am able to read datas from 4 SATA disks at
> about 225 MB/s, with CPU time used at about 32%, awd with a system still
> reasonably responsive.

I'm seeing ~80MB/sec reading from the 3ware raid-5 according to hdparm and 
bonnie++
Write performance is around 25MB/sec according to bonnie++.

Mark.

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Jim Buttafuoco]

I second Marks comment on NOT recommending the 3ware cards to anyone.  

I have a dual XEON 3.2 Ghz system 12 G of ram with a 3ware 8506-8 in it (8 250 G drives).  The hard raid performance 
was very bad.  with the load avg going over 40.  I then switched over to JBOD and software raid.  The IO wait times 
are really high and the performance sucks.  Very hard to explain to my boss where the $20k went. The system is a 
database server (postgres).  I tried both kernel 2.4 and 2.6 with the same problem.  I am now in the process of 
testing the adaptec raid controller.  


Jim
    

---------- Original Message -----------
From: Mark Watts <mrwatts@fast24.co.uk>
To: Marc Bevand <bevand_m@epita.fr>
Cc: linux-raid@vger.kernel.org
Sent: Fri, 30 Jul 2004 17:17:36 +0100
Subject: Re: 1x 3ware controllers vs. 2x 3ware controllers

> > Mark Watts wrote:
> > > With the performance issues I'm seeing with the 8506-4LP's I have, I
> > > wouldn't recommend them to anyone currently...
> > >
> > > The thread on LKML from a few days ago about mke2fs -j and 8506-4LP says
> > > it all, but basically and reasonable amount of I/O brings a Dual Opteron
> > > system to its knees.
> > > And by reasonable I mean copying ISO images from a usb2 drive to a raid 5
> > > (4 x 250GB maxtor) or even just formatting a 600MB partition.
> >
> > I would be interested to see the output of 'vmstat 1' while your system is
> > so slow.
> >
> > IMHO you shouldn't draw such conclusion ("reasonable amount of I/O brings a
> > Dual Opteron system to its knees") from your particular case. The 3 HT
> > links of the Opteron make this CPU particularly adapted to I/O operations.
> 
> Well its the only conclusion I *can* come to at this time.
> When you move from a UP 1.8Ghz P4 with single EIDE disks to SMP Opteron with 4 
> times the ram and a hardware raid card, you tend to assume that performance 
> in all areas will go up.
> When it doesn't, and you find yourself watching screen redraws while you 
> format a 600GB partition (ext3) you do feel the need to blame something :)
> 
> I'm all ears for suggestions on what me be wrong or things I can try to 
> improve performance.
> 
> >
> > Personnaly, on a dual Opteron, I am able to read datas from 4 SATA disks at
> > about 225 MB/s, with CPU time used at about 32%, awd with a system still
> > reasonably responsive.
> 
> I'm seeing ~80MB/sec reading from the 3ware raid-5 according to hdparm and 
> bonnie++
> Write performance is around 25MB/sec according to bonnie++.
> 
> Mark.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
------- End of Original Message -------

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mikael Abrahamsson]

On Fri, 30 Jul 2004, Mark Watts wrote:

> Basically the system load goes orbital and responsivness goes out the window.
> gkrellm only shows ~1MB/sec of disk activity and the processors (Dual Opteron) 
> are twiddling their thumbs (top shows 100% 'wa' state for both processors), 
> yet I can hardly do anything with any open applications.

I have had good results with IRQ affinity on the 3ware 7500 (echo "0" >
/proc/irq/<X>/smp_affinity on my dual pIII machine. Definately makes 3dm
behave better anyway.

And yes, writing to a hardware raid5 on 3ware isn't fast, so I use scsi 
drives for anything that needs fast writes, and then use the 3ware for 
bulk storage (most of it write once read many).

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mikael Abrahamsson]

On Fri, 30 Jul 2004, Jim Buttafuoco wrote:

> I have a dual XEON 3.2 Ghz system 12 G of ram with a 3ware 8506-8 in it
> (8 250 G drives).  The hard raid performance was very bad.  with the
> load avg going over 40.  I then switched over to JBOD and software raid.  
> The IO wait times are really high and the performance sucks.  Very hard
> to explain to my boss where the $20k went. The system is a database
> server (postgres).  I tried both kernel 2.4 and 2.6 with the same
> problem.  I am now in the process of testing the adaptec raid
> controller.

You should never ever ever use raid5 with something that is write 
intensive, such as a database server. Raid5 is a cost compromise that is a 
little of everything, if you want protection and speed at the same time, 
you should use raid1 (and possible raid0 the raid1:s).

I learnt this the hard way running a nntp (news) server.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Joshua Baker-LePain]

On Fri, 30 Jul 2004 at 7:53pm, Jim Buttafuoco wrote

> I second Marks comment on NOT recommending the 3ware cards to anyone.  
> 
> I have a dual XEON 3.2 Ghz system 12 G of ram with a 3ware 8506-8 in it (8 250 G drives).  The hard raid performance 
> was very bad.  with the load avg going over 40.  I then switched over to JBOD and software raid.  The IO wait times 
> are really high and the performance sucks.  Very hard to explain to my boss where the $20k went. The system is a 
> database server (postgres).  I tried both kernel 2.4 and 2.6 with the same problem.  I am now in the process of 
> testing the adaptec raid controller.  

3wares *can* work, and work well.  I've got two 7500-8s on a dual Xeon 
board (Supermicro), each with 8 180GB WD drives (7200 RPM) and running 
hardware RAID5.  Then I do a software RAID0 strips across the two arrays.  
Performance is quite good:

[jlb@buckbeak jlb]$ bonnie++ -s 8192
Version 1.02c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
buckbeak         8G 25283  96 131142  39 83869  32 28038  99 342849  68 444.1   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  2644  21 +++++ +++  2227  17  2548  20 +++++ +++  2118  21
buckbeak,8G,25283,96,131142,39,83869,32,28038,99,342849,68,444.1,1,16,2644,21,+++++,+++,2227,17,2548,20,+++++,+++,2118,21

That's with a 2.4 kernel and XFS.

-- 
Joshua Baker-LePain
Department of Biomedical Engineering
Duke University

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Jon Lewis]

On Sat, 31 Jul 2004, Mikael Abrahamsson wrote:

> You should never ever ever use raid5 with something that is write
> intensive, such as a database server. Raid5 is a cost compromise that is a
> little of everything, if you want protection and speed at the same time,
> you should use raid1 (and possible raid0 the raid1:s).

Hardware RAID5 or software RAID5?  My experience has been that even the
higher end hardware RAID5 cards I've dealt with give nowhere near the
performace you get with software RAID5.  HW RAID5 is a good way to make an
IO intensive system turn into a slug.

> I learnt this the hard way running a nntp (news) server.

The only RAID I'd do / have done with nntp servers is RAID1 for the system
and RAID0 for the spool...but with modern software you don't even need
RAID0 for the spool...you just keep telling it to use more drives for
spools.

----------------------------------------------------------------------
 Jon Lewis                   |  I route
 Senior Network Engineer     |  therefore you are
 Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mikael Abrahamsson]

On Sat, 31 Jul 2004, Jon Lewis wrote:

> Hardware RAID5 or software RAID5?  My experience has been that even the
> higher end hardware RAID5 cards I've dealt with give nowhere near the
> performace you get with software RAID5.  HW RAID5 is a good way to make an
> IO intensive system turn into a slug.

It doesn't matter. On small writes you need to do a lot of reads to get 
the parity correct. If you do a lot of sequencial writes then RAID5 is ok, 
if you do a lot of random writes, then RAID5 is bad. 

HW 3ware raid5 is ok if you just do "cat /dev/null > /dev/sda", you'll get
nice write speeds, but for real life filesystem writes it needs to read a
lot to write the correct parity and this is quite slow. The only thing
software RAID5 solves is that it can use system memory to cache a lot more
than the HW 3ware RAID5 card can, so it can sometimes avoid to read from
the drives before writing parity.

This doesn't change the fact that if you need to do a lot of random writes 
and you need to do it quickly, avoid RAID5:ing large number of drives.
 
-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Watts]

> On Sat, 31 Jul 2004, Jon Lewis wrote:
> > Hardware RAID5 or software RAID5?  My experience has been that even the
> > higher end hardware RAID5 cards I've dealt with give nowhere near the
> > performace you get with software RAID5.  HW RAID5 is a good way to make
> > an IO intensive system turn into a slug.
>
> It doesn't matter. On small writes you need to do a lot of reads to get
> the parity correct. If you do a lot of sequencial writes then RAID5 is ok,
> if you do a lot of random writes, then RAID5 is bad.
>
> HW 3ware raid5 is ok if you just do "cat /dev/null > /dev/sda", you'll get
> nice write speeds, but for real life filesystem writes it needs to read a
> lot to write the correct parity and this is quite slow. The only thing
> software RAID5 solves is that it can use system memory to cache a lot more
> than the HW 3ware RAID5 card can, so it can sometimes avoid to read from
> the drives before writing parity.
>
> This doesn't change the fact that if you need to do a lot of random writes
> and you need to do it quickly, avoid RAID5:ing large number of drives.

So why are scsi raid-5 systems so much better?

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Jurriaan]

From: Mark Watts <mrwatts@fast24.co.uk>
Date: Sat, Jul 31, 2004 at 05:42:38PM +0100
> 
> > This doesn't change the fact that if you need to do a lot of random writes
> > and you need to do it quickly, avoid RAID5:ing large number of drives.
> 
> So why are scsi raid-5 systems so much better?

compared to other scsi raid-arrays, they are not 'better' for small
random writes. If they are better than ide, it's probably because of the
smaller access times for 15k rpm drives.

Jurriaan
-- 
It has a name, among the Voyani.
Lumina Arden. The light that burns.
You will feel its fire, Serra Diora.
	Michelle West - The Broken Crown
Debian (Unstable) GNU/Linux 2.6.8-rc2-mm1 2x6078 bogomips load 0.72

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mikael Abrahamsson]

On Sat, 31 Jul 2004, Jurriaan wrote:

> > So why are scsi raid-5 systems so much better?
> 
> compared to other scsi raid-arrays, they are not 'better' for small
> random writes. If they are better than ide, it's probably because of the
> smaller access times for 15k rpm drives.

And also because scsi drives can do tagged queueing which makes it more 
efficient to do a lot of smaller operations. Historically the SCSI drives 
also had more cache memory which helps the situation, and the scsi 
RAID controllers probably also had more cache memory on them (I know RAID 
systems that have gigabytes of cache memory).

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Gordon Henderson]

On Sun, 1 Aug 2004, Mikael Abrahamsson wrote:

> And also because scsi drives can do tagged queueing which makes it more
> efficient to do a lot of smaller operations. Historically the SCSI drives
> also had more cache memory which helps the situation, and the scsi
> RAID controllers probably also had more cache memory on them (I know RAID
> systems that have gigabytes of cache memory).

What I find amusing these days is trying to work out the "boundary" point
between a "traditional" server with an (external) RAID controller and say
a Linux server with software RAID in a purely fileserving environment (eg.
NFS/Samba, not used for local operations at all) ... Both systems as a
unit provide the same services - ie. filespace at the end of the Ether,
but what are the advantages of one over the other, and why would I ever
want a hardware RAID controller in a PCI slot in a Server PC?

Discuss... ;-)

Gordon

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mikael Abrahamsson]

On Sun, 1 Aug 2004, Gordon Henderson wrote:

> What I find amusing these days is trying to work out the "boundary" point
> between a "traditional" server with an (external) RAID controller and say
> a Linux server with software RAID in a purely fileserving environment (eg.
> NFS/Samba, not used for local operations at all) ... Both systems as a
> unit provide the same services - ie. filespace at the end of the Ether,
> but what are the advantages of one over the other, and why would I ever
> want a hardware RAID controller in a PCI slot in a Server PC?
> 
> Discuss... ;-)

Let's say you have a database server with a large RAID1+0 array. If you do
this in software, you have to transfer data over the PCI bus twice if you
do the raid in software, not so if you do it in hardware.

But I agree, it's like "what is the best car?" It all depends on your
needs and budget. A Porsche 911 is perfect for people who need to go
places fast, it's useless for people who need to haul things.

You always have to optimize everything after your needs, do you need 
highspeed sequencial access, do you need low latency small accesses, do 
you need a lot of fast low latency writes, etc. All these require 
different solutions.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Watts]

> On Sun, 1 Aug 2004, Gordon Henderson wrote:
> > What I find amusing these days is trying to work out the "boundary" point
> > between a "traditional" server with an (external) RAID controller and say
> > a Linux server with software RAID in a purely fileserving environment
> > (eg. NFS/Samba, not used for local operations at all) ... Both systems as
> > a unit provide the same services - ie. filespace at the end of the Ether,
> > but what are the advantages of one over the other, and why would I ever
> > want a hardware RAID controller in a PCI slot in a Server PC?
> >
> > Discuss... ;-)
>
> Let's say you have a database server with a large RAID1+0 array. If you do
> this in software, you have to transfer data over the PCI bus twice if you
> do the raid in software, not so if you do it in hardware.
>
> But I agree, it's like "what is the best car?" It all depends on your
> needs and budget. A Porsche 911 is perfect for people who need to go
> places fast, it's useless for people who need to haul things.
>
> You always have to optimize everything after your needs, do you need
> highspeed sequencial access, do you need low latency small accesses, do
> you need a lot of fast low latency writes, etc. All these require
> different solutions.

Ok, I understand that for many areas, a software raid-5 will thrash the pants 
off of a hardware one.

This doesn't help explain why some of us are experiencing crap performance and 
un-responsive systems whereas others are having none of these problems  with 
the same 3Ware cards.

We have the hardware so we'd like to use it to the best of its ability. 
Obviously something isn't right with our systems - does anyone have any 
insight as to what this might be?

Mark.

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mikael Abrahamsson]

On Sun, 1 Aug 2004, Mark Watts wrote:

> This doesn't help explain why some of us are experiencing crap performance and 
> un-responsive systems whereas others are having none of these problems  with 
> the same 3Ware cards.

On my 3ware equipped system running hw raid5 I get lousy write speeds 
(approx 5-15 megabyte/s) on writes. I get great read speeds. I don't sit 
directly on the console so I have no idea if the system is interactively 
responsive or not, but I notice no problems being logged in via ssh. 

On the other hand, the 3ware volume is only used for bulk data storage,
the system, swap and other often needed data that is accessed and changed
often, I have stored on SCSI drives that are RAID1 or not RAIDed at all.

I would never store the system/swap on a 3ware hw RAID5, that would be 
disasterous for performance. Could you perhaps describe your system setup 
in more detail?
 
-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Watts]

> On Sun, 1 Aug 2004, Mark Watts wrote:
> > This doesn't help explain why some of us are experiencing crap
> > performance and un-responsive systems whereas others are having none of
> > these problems  with the same 3Ware cards.
>
> On my 3ware equipped system running hw raid5 I get lousy write speeds
> (approx 5-15 megabyte/s) on writes. I get great read speeds. I don't sit
> directly on the console so I have no idea if the system is interactively
> responsive or not, but I notice no problems being logged in via ssh.
>
> On the other hand, the 3ware volume is only used for bulk data storage,
> the system, swap and other often needed data that is accessed and changed
> often, I have stored on SCSI drives that are RAID1 or not RAIDed at all.
>
> I would never store the system/swap on a 3ware hw RAID5, that would be
> disasterous for performance. Could you perhaps describe your system setup
> in more detail?

Tyan S2845
Dual Opteron 246 Processors
2GB DDR333 Ram
nVidia FX5700
3Ware 8506-4LP 4 port SATA raid card
4x 250GB Maxtor hdds

RAID Configuration:
4-disk RAID-5 with 64K block size.

/dev/sda1 = / = 20GB
/dev/sda2 = swap = 2048MB
/dev/sda5 = /home = 100GB

All partitions (except swap) formatted as ext3.

Kernel - 2.6.8rc1, SMP for x86_64

Read performance is ok but nothing special.
Writing larger files (anything that takes more than a few seconds) will cause 
the system to 'stutter' as the write occurs.
Longer writes (eg: copying ISO image about or formatting the rest of the 
filesystem (600GB)) cause the system load to go high, very quickly.
System is almost unusable at this stage but load returns to normal as the 
write completes.


Slow write speeds is one thing, practically locking the system up while the 
write occurs is another, unless I'm missing something obvious.

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Jim Buttafuoco]

I understand this, but I need the space.  I have databases that are over 3TB and my customers don't want to pay.  I 
just need to get the biggest bang for my customers bucks.  I know you get what you pay for.  I just didn't expect the 
3ware cards to perform so bad.  I think I would have been better off with 4 SATA controllers in the system and do 
software raid.  For my customers that will pay the big $$$, I am using SCSI raid controllers from ICP (now Intel I 
believe).

I am switching all of my 3ware controllers over to Adaptec SATA raid controllers.  I am getting better performance 
with them.  
Jim




---------- Original Message -----------
From: Mikael Abrahamsson <swmike@swm.pp.se>
To: Jim Buttafuoco <jim@contactbda.com>
Cc: linux-raid@vger.kernel.org
Sent: Sat, 31 Jul 2004 10:49:00 +0200 (CEST)
Subject: Re: 1x 3ware controllers vs. 2x 3ware controllers

> On Fri, 30 Jul 2004, Jim Buttafuoco wrote:
> 
> > I have a dual XEON 3.2 Ghz system 12 G of ram with a 3ware 8506-8 in it
> > (8 250 G drives).  The hard raid performance was very bad.  with the
> > load avg going over 40.  I then switched over to JBOD and software raid.  
> > The IO wait times are really high and the performance sucks.  Very hard
> > to explain to my boss where the $20k went. The system is a database
> > server (postgres).  I tried both kernel 2.4 and 2.6 with the same
> > problem.  I am now in the process of testing the adaptec raid
> > controller.
> 
> You should never ever ever use raid5 with something that is write 
> intensive, such as a database server. Raid5 is a cost compromise that is a 
> little of everything, if you want protection and speed at the same time, 
> you should use raid1 (and possible raid0 the raid1:s).
> 
> I learnt this the hard way running a nntp (news) server.
> 
> -- 
> Mikael Abrahamsson    email: swmike@swm.pp.se
------- End of Original Message -------

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Jim Buttafuoco]

I get the same with dual XEON systems (I have 2 with Supermicro mb).  Reading on the net.  It seems that this is a 
know problem with people who are trying to build big systems.  No solution that I have seen.


---------- Original Message -----------
From: Mark Watts <mrwatts@fast24.co.uk>
To: Mikael Abrahamsson <swmike@swm.pp.se>
Cc: linux-raid@vger.kernel.org
Sent: Sun, 1 Aug 2004 16:01:46 +0100
Subject: Re: 1x 3ware controllers vs. 2x 3ware controllers

> > On Sun, 1 Aug 2004, Mark Watts wrote:
> > > This doesn't help explain why some of us are experiencing crap
> > > performance and un-responsive systems whereas others are having none of
> > > these problems  with the same 3Ware cards.
> >
> > On my 3ware equipped system running hw raid5 I get lousy write speeds
> > (approx 5-15 megabyte/s) on writes. I get great read speeds. I don't sit
> > directly on the console so I have no idea if the system is interactively
> > responsive or not, but I notice no problems being logged in via ssh.
> >
> > On the other hand, the 3ware volume is only used for bulk data storage,
> > the system, swap and other often needed data that is accessed and changed
> > often, I have stored on SCSI drives that are RAID1 or not RAIDed at all.
> >
> > I would never store the system/swap on a 3ware hw RAID5, that would be
> > disasterous for performance. Could you perhaps describe your system setup
> > in more detail?
> 
> Tyan S2845
> Dual Opteron 246 Processors
> 2GB DDR333 Ram
> nVidia FX5700
> 3Ware 8506-4LP 4 port SATA raid card
> 4x 250GB Maxtor hdds
> 
> RAID Configuration:
> 4-disk RAID-5 with 64K block size.
> 
> /dev/sda1 = / = 20GB
> /dev/sda2 = swap = 2048MB
> /dev/sda5 = /home = 100GB
> 
> All partitions (except swap) formatted as ext3.
> 
> Kernel - 2.6.8rc1, SMP for x86_64
> 
> Read performance is ok but nothing special.
> Writing larger files (anything that takes more than a few seconds) will cause 
> the system to 'stutter' as the write occurs.
> Longer writes (eg: copying ISO image about or formatting the rest of the 
> filesystem (600GB)) cause the system load to go high, very quickly.
> System is almost unusable at this stage but load returns to normal as the 
> write completes.
> 
> Slow write speeds is one thing, practically locking the system up while the 
> write occurs is another, unless I'm missing something obvious.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
------- End of Original Message -------

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mikael Abrahamsson]

On Sun, 1 Aug 2004, Jim Buttafuoco wrote:

> I am switching all of my 3ware controllers over to Adaptec SATA raid
> controllers.  I am getting better performance with them.  Jim

I am quite curious about the 16port SATA controller Adaptec has now. Do 
you have any filesystem write figures for this controller under linux and 
ext3?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mikael Abrahamsson]

On Sun, 1 Aug 2004, Mark Watts wrote:

> RAID Configuration:
> 4-disk RAID-5 with 64K block size.
> 
> /dev/sda1 = / = 20GB
> /dev/sda2 = swap = 2048MB
> /dev/sda5 = /home = 100GB
> 
> All partitions (except swap) formatted as ext3.

It would be interesting to see difference if you put the system and swap 
on another drive that is not part of the RAID5.
 
> Kernel - 2.6.8rc1, SMP for x86_64

I use v2.4, haven't used 2.6 with 3ware and SMP at all, so the problem 
might be there. Did you try the smp_affinity thing I emailed about 1-2 
days ago?

> Slow write speeds is one thing, practically locking the system up while the 
> write occurs is another, unless I'm missing something obvious.

Well, if the system needs to page-in and this page-in takes seconds to 
fulfil because the raid5 is full at work and has a large latency queue, 
that might explain it. Does the system do anything else when you're 
copying?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Watts]

> On Sun, 1 Aug 2004, Mark Watts wrote:
> > RAID Configuration:
> > 4-disk RAID-5 with 64K block size.
> >
> > /dev/sda1 = / = 20GB
> > /dev/sda2 = swap = 2048MB
> > /dev/sda5 = /home = 100GB
> >
> > All partitions (except swap) formatted as ext3.
>
> It would be interesting to see difference if you put the system and swap
> on another drive that is not part of the RAID5.

With 2GB ram, I can disable swap entirely if you want, but I don't have 
another drive to put it on.

>
> > Kernel - 2.6.8rc1, SMP for x86_64
>
> I use v2.4, haven't used 2.6 with 3ware and SMP at all, so the problem
> might be there. Did you try the smp_affinity thing I emailed about 1-2
> days ago?

No - The system is at work - I'll try it on Monday.

>
> > Slow write speeds is one thing, practically locking the system up while
> > the write occurs is another, unless I'm missing something obvious.
>
> Well, if the system needs to page-in and this page-in takes seconds to
> fulfil because the raid5 is full at work and has a large latency queue,
> that might explain it. Does the system do anything else when you're
> copying?

No (thats the point!). Playing (or rather skipping) mp3's with xmms is about 
the most it does.

I'm going to try using xfs instead of ext3 on Monday too.

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Gordon Henderson]

On Sun, 1 Aug 2004, Mark Watts wrote:

> I'm going to try using xfs instead of ext3 on Monday too.

I doubt it's a filesystem issue.

Try this - make the array work flat-out - dd into a fil eor something and
look at the process state of 'dd'. See if it's stuck in "D".

Now look at /proc/interrupts...

Look for ERR and MIS counts of non-zero...

I've built some systems in the past that have exhibited similar
performance and what it boiled down to was the motherboard interrupt
controller is basically crap. I "solved" it with one system by
re-arranging the PCI cards and used the on-board IDE controllers (and one
primose plug-in card rather than 2; 4 IDE drives, one per cable) to make
sure everything had it's own interrupt. Eg:

           CPU0       CPU1
  0:  107450604  108441652    IO-APIC-edge  timer
  1:        340        446    IO-APIC-edge  keyboard
  2:          0          0          XT-PIC  cascade
 14:   19756978   19994219    IO-APIC-edge  ide0
 15:   20297959   20492514    IO-APIC-edge  ide1
 16:   21597138   21847832   IO-APIC-level  ide2
 17:   20928098   21157136   IO-APIC-level  ide4
 18:      72819      72311   IO-APIC-level  aic7xxx
 19:  404554707  404245858   IO-APIC-level  aic7xxx, eth0
NMI:          0          0
LOC:  215883196  215883194
ERR:          4
MIS:        395

I still see the occasional MIS interrupt (it's accompanied by an APIC
error - eg:

Jul 14 17:59:25 blue kernel: APIC error on CPU0: 00(02)
Jul 14 17:59:25 blue kernel: APIC error on CPU1: 00(02)

This is a dual Athlon mobo.

Good luck!

Gordon

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Scott T. Smith]

On Sun, 2004-08-01 at 08:06, Jim Buttafuoco wrote:
> I think I would have been better off with 4 SATA controllers in the system and do 
> software raid.

Along these lines... what is the best
multi-SATA-controller-on-a-single-board out there?  I've had issues with
the 3Ware controllers in JBOD mode where one disk failing affects the
rest of the drives on the controller; are there better (i.e. more
independent) ones out there?

	Scott

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Scott T. Smith]

On Sun, 2004-08-01 at 08:24, Mikael Abrahamsson wrote:
> On Sun, 1 Aug 2004, Jim Buttafuoco wrote:
> 
> > I am switching all of my 3ware controllers over to Adaptec SATA raid
> > controllers.  I am getting better performance with them.  Jim
> 
> I am quite curious about the 16port SATA controller Adaptec has now. Do 
> you have any filesystem write figures for this controller under linux and 
> ext3?

One main problem with the 16port controller, at least in JBOD mode, is
that there is not enough PCI bandwidth to fill the disks if you do large
writes.  Thus two separate 8port controllers, on separate PCI busses,
should yield higher performance.

OTOH, if you're doing small ops, or RAID1, it will probably be fine.

(disclaimer -- I haven't tried the Adaptec boards yet, but they're on my
list, along with Highpoint's)

	Scott

Re: 1x 3ware controllers vs. 2x 3ware controllers

[David Greaves]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Scott T. Smith wrote:
| On Sun, 2004-08-01 at 08:24, Mikael Abrahamsson wrote:
|
|>On Sun, 1 Aug 2004, Jim Buttafuoco wrote:
|>
|>
|>>I am switching all of my 3ware controllers over to Adaptec SATA raid
|>>controllers.  I am getting better performance with them.  Jim
|>
|>I am quite curious about the 16port SATA controller Adaptec has now. Do
|>you have any filesystem write figures for this controller under linux and
|>ext3?
|
|
| One main problem with the 16port controller, at least in JBOD mode, is
| that there is not enough PCI bandwidth to fill the disks if you do large
| writes.  Thus two separate 8port controllers, on separate PCI busses,
| should yield higher performance.
|
| OTOH, if you're doing small ops, or RAID1, it will probably be fine.
|
| (disclaimer -- I haven't tried the Adaptec boards yet, but they're on my
| list, along with Highpoint's)
|
| 	Scott

Highpoint have a multi port PCI-X card out (but no open drivers support it)
Are there any others multi-sata PCI-X cards?

I'm thinking about building a 2Tb SW RAID but will wait a few months
because I want PCI-X or similar to get the bandwidth to the gigabit card.

David
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBDURE8LvjTle4P1gRApWbAJ0f5UHlMUsxkx36qSbk59Ak1Vj/cwCcD5hw
AruxO8CSNAM+4ixZioKYfQA=
=cYtx
-----END PGP SIGNATURE-----

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mikael Abrahamsson]

On Sun, 1 Aug 2004, David Greaves wrote:

> I'm thinking about building a 2Tb SW RAID but will wait a few months
> because I want PCI-X or similar to get the bandwidth to the gigabit card.

According to Adaptec, their 16 port card with 64bit 66MHz PCI will 
transfer up to 1.5gigabit/s. That's a tad low I must say, that's only 190 
megabytes/s. I've gotten better read performance off of a 3ware card. Good 
enough for feed a gig card though, especially if you have multiple PCI 
buses.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Watts]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> On Sun, 1 Aug 2004, Mark Watts wrote:
> > RAID Configuration:
> > 4-disk RAID-5 with 64K block size.
> >
> > /dev/sda1 = / = 20GB
> > /dev/sda2 = swap = 2048MB
> > /dev/sda5 = /home = 100GB
> >
> > All partitions (except swap) formatted as ext3.
>
> It would be interesting to see difference if you put the system and swap
> on another drive that is not part of the RAID5.
>
> > Kernel - 2.6.8rc1, SMP for x86_64
>
> I use v2.4, haven't used 2.6 with 3ware and SMP at all, so the problem
> might be there. Did you try the smp_affinity thing I emailed about 1-2
> days ago?

Ok i tried this and it made no difference.


- -- 
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBDgvyBn4EFUVUIO0RAqJ0AKC2CgF/qT32el1xq7JdGntZj2fl0wCfVkRt
9sehETCi1tgs+ol/vRrkYu8=
=Ven5
-----END PGP SIGNATURE-----

what is the best multi-SATA-controller-on-a-single-board out there?

[Tim Small]

Scott T. Smith wrote:

>Along these lines... what is the best
>multi-SATA-controller-on-a-single-board out there?  I've had issues with
>the 3Ware controllers in JBOD mode where one disk failing affects the
>rest of the drives on the controller; are there better (i.e. more
>independent) ones out there?
>  
>
I've used multiple cheap Promise TX4s PCI cards, and Silicon Image 
onboard controllers (e.g Tyan Thunder K8SPro dual Opteron board).  
Haven't had at SATA drives fail yet, so I can't say anything about 
reliability with failed drives.

The Promise cards, are cheap, but only do 66Mhz/32 bit PCI and provide 
four SATA ports, so they may not be well suited to some boards..  Both 
chipsets were used with libata on 2.6 kernels, with a mixture of s/w 
raid5, and s/w raid1.  The only slight irritation is the current lack of 
smartd support (pending libata driver support).

I haven't done much in the way of benchmarking on these machines, (just 
tested linear reads at 42MB/s on 2x 250G/7200 rpm Raid1, and 62MB/s on a 
5x 250G/7200 rpm Raid5), but performance is good enough for the tasks 
that they will be carrying out.  Two of these boxes are not commissioned 
yet, so if anyone would like to have me run some benchmarks, let me know 
what to run, and I'll see what I can do.

BTW, I have just run a linear read test on a similar machine (same 
kernel), but with 2x PATA (similar 250G drives), and get 57MB/s on 
Raid1, so maybe libata performance isn't quite there yet.

Tim.

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Watts]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


> On Sun, 1 Aug 2004, Mark Watts wrote:
> > I'm going to try using xfs instead of ext3 on Monday too.
>
> I doubt it's a filesystem issue.
>
> Try this - make the array work flat-out - dd into a fil eor something and
> look at the process state of 'dd'. See if it's stuck in "D".
>
> Now look at /proc/interrupts...
>
> Look for ERR and MIS counts of non-zero...

Both of those are zero, even when hammering the disks.

Mark.

- -- 
Mark Watts
Senior Systems Engineer
QinetiQ Trusted Information Management
Trusted Solutions and Services group
GPG Public Key ID: 455420ED

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBDjmMBn4EFUVUIO0RAqjKAJ0R+SMjOUFfVQ/QMimcE2pFNukosQCeNYyY
C26gL+Lh/vqG1q75nazqf5E=
=vwnS
-----END PGP SIGNATURE-----

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Gordon Henderson]

On Mon, 2 Aug 2004, Mark Watts wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> > On Sun, 1 Aug 2004, Mark Watts wrote:
> > > I'm going to try using xfs instead of ext3 on Monday too.
> >
> > I doubt it's a filesystem issue.
> >
> > Try this - make the array work flat-out - dd into a fil eor something and
> > look at the process state of 'dd'. See if it's stuck in "D".
> >
> > Now look at /proc/interrupts...
> >
> > Look for ERR and MIS counts of non-zero...
>
> Both of those are zero, even when hammering the disks.

Oh well - another good theory up the spout!

Gordon

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Mark Hahn]

> According to Adaptec, their 16 port card with 64bit 66MHz PCI will 
> transfer up to 1.5gigabit/s. That's a tad low I must say, that's only 190 

I think they're clearly pulling a marketing-weasel on you:
the 1.5 gbps is from sata's peak/theoretical/never data rate.

Re: 1x 3ware controllers vs. 2x 3ware controllers

[Scott T. Smith]

On Mon, 2004-08-02 at 11:02, Mark Hahn wrote:
> > According to Adaptec, their 16 port card with 64bit 66MHz PCI will 
> > transfer up to 1.5gigabit/s. That's a tad low I must say, that's only 190 
> 
> I think they're clearly pulling a marketing-weasel on you:
> the 1.5 gbps is from sata's peak/theoretical/never data rate.

I routinely get 1.5 Gbits/sec on my 3ware, 8 port (8506-8) controller. 
8 SATA, 250GB WD disks (JBOD mode).  Like the Adaptec, it is also a
64bit 66MHz PCI.

Of course, reads are 1 megabyte each (but random on disk, not
sequential), but still, it is possible.

	Scott

Re: what is the best multi-SATA-controller-on-a-single-board out there?

[Jon Lewis]

On Mon, 2 Aug 2004, Tim Small wrote:

> The Promise cards, are cheap, but only do 66Mhz/32 bit PCI and provide
> four SATA ports, so they may not be well suited to some boards..  Both
> chipsets were used with libata on 2.6 kernels, with a mixture of s/w
> raid5, and s/w raid1.  The only slight irritation is the current lack of
> smartd support (pending libata driver support).

With what distros / exactly which kernels?  I've been testing a system
with 2 Promise SATA150 TX4's and 6 Maxtor 200gb SATA drives in a
supermicro (mb/chassis) system with 7 drive hot-swap drive carrier.  I've
been having serious problems and am having trouble telling if its hardware
or software.  I've been trying both Whitebox (basically RH ES 3.0 which
uses a 2.4 kernel and includes libata but doesn't build it by default) and
Fedora Core 2, which supports the SATA hardware by default.  OS gets
installed on smaller drives on the MB's PATA controller.  The big 6
drives are supposed to be SW RAID5.  The system usually locks up while
building the array.  Today it got as far as nearly finishing the mke2fs on
the array when it locked up.

Running badblocks on each 200gb drive individually, I got errors on one of
them once.  When I reran badblocks on that drive, I did not get errors.
I still can't get an array built/formatted (under whitebox today) without
the system locking up.

So, I'm curious who else is using the SATA150 TX4 and with which
kernel/driver and whether they're having or have seen any similar
problems.


----------------------------------------------------------------------
 Jon Lewis                   |  I route
 Senior Network Engineer     |  therefore you are
 Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

RE: what is the best multi-SATA-controller-on-a-single-board out there?

[robin-lists]

> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org 
> [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Jon Lewis
> Sent: 02 August 2004 21:10
> To: Tim Small
> Cc: linux-raid@vger.kernel.org
> Subject: Re: what is the best 
> multi-SATA-controller-on-a-single-board out there?
> 
> On Mon, 2 Aug 2004, Tim Small wrote:
> 
> > The Promise cards, are cheap, but only do 66Mhz/32 bit PCI 
> and provide 
> > four SATA ports, so they may not be well suited to some 
> boards..  Both 
> > chipsets were used with libata on 2.6 kernels, with a 
> mixture of s/w 
> > raid5, and s/w raid1.  The only slight irritation is the 
> current lack 
> > of smartd support (pending libata driver support).
> 
> With what distros / exactly which kernels?  I've been testing 
> a system with 2 Promise SATA150 TX4's and 6 Maxtor 200gb SATA 
> drives in a supermicro (mb/chassis) system with 7 drive 
> hot-swap drive carrier.  I've been having serious problems 
> and am having trouble telling if its hardware or software.  
> I've been trying both Whitebox (basically RH ES 3.0 which 
> uses a 2.4 kernel and includes libata but doesn't build it by 
> default) and Fedora Core 2, which supports the SATA hardware 
> by default.  OS gets installed on smaller drives on the MB's 
> PATA controller.  The big 6 drives are supposed to be SW 
> RAID5.  The system usually locks up while building the array. 
>  Today it got as far as nearly finishing the mke2fs on the 
> array when it locked up.
> 
> Running badblocks on each 200gb drive individually, I got 
> errors on one of them once.  When I reran badblocks on that 
> drive, I did not get errors.
> I still can't get an array built/formatted (under whitebox 
> today) without the system locking up.
> 
> So, I'm curious who else is using the SATA150 TX4 and with 
> which kernel/driver and whether they're having or have seen 
> any similar problems.

Jon,

I've got a very similar setup to you - 2 x SATA150 TX4 with 6 x Maxtor 250GB
drives.

I had exactly the same symptoms and I found it was caused by failing drives.
I think I got a bad batch, or a bad supplier or something because I had
three out of a batch of four fail. I got replacements from Maxtor (3 year
warranty) and have not had any problems since.

When you RMA their drives they ask that you download a utility called
"powermax" and check out the drives before returning them - the utility
gives you a failure code if the drive is bad. Try downloading it and running
the quick test on your drives. I'll give you 5-1 that they will fail.

R.
--
http://robinbowes.com 

Re: [RAID] Re: 1x 3ware controllers vs. 2x 3ware controllers

[Julian Cowley]

On Sun, 1 Aug 2004, Gordon Henderson wrote:
> On Sun, 1 Aug 2004, Mikael Abrahamsson wrote:
> > And also because scsi drives can do tagged queueing which makes it more
> > efficient to do a lot of smaller operations. Historically the SCSI drives
> > also had more cache memory which helps the situation, and the scsi
> > RAID controllers probably also had more cache memory on them (I know RAID
> > systems that have gigabytes of cache memory).
>
> What I find amusing these days is trying to work out the "boundary" point
> between a "traditional" server with an (external) RAID controller and say
> a Linux server with software RAID in a purely fileserving environment (eg.
> NFS/Samba, not used for local operations at all) ... Both systems as a
> unit provide the same services - ie. filespace at the end of the Ether,
> but what are the advantages of one over the other, and why would I ever
> want a hardware RAID controller in a PCI slot in a Server PC?
>
> Discuss... ;-)

Recently I did a survey of this very question (hardware vs. software
RAID) based on the comments from this mailing list:

Software
--------

- CPU must handle operations
- twice the I/O bandwidth when using RAID1
+ non-proprietary disk format
+ open source implementation
- limited or non-existent support for hot-swapping, even with SATA
  (see http://www.redhat.com/archives/fedora-test-list/2004-March/msg01204.html)
- OS-specific format (can't be shared between Linux, Windows, etc.)
+ drives can be anything (ie. a mixture of SATA, PATA, Firewire, USB, etc.)
- disk surface testing must be done manually (7/2004)
- no bad block relocation (7/2004)
- no parity verification (7/2004)
- no mirror verification (7/2004)
+ reputedly, much better performance than hardware raid

Hardware
--------

+ off-loads the CPU
+ I/O bandwidth needed on a RAID1 system is same as single disk
- proprietary disk format (although limited drivers are available for Linux)
- proprietary implementation
+ easy hot-swapping (some controllers even indicate the bad drive with an LED)
+ non-OS-specific (can share between Linux, Windows, etc.)
- some features may not be supported on non-Windows operating systems
+ able to create logical disks that seem like physical disks to the OS
+ bad sector relocation (on the fly?)
- drives must connect to the controller and all must be same type (e.g. SATA)
+ disk surface testing done automatically
+ automatic bad block relocation
+ parity verification
+ mirror verification

Definition of hotswap, was 1x 3ware controllers vs. 2x 3ware controllers

[Scott T. Smith]

On Mon, 2004-08-02 at 18:58, Julian Cowley wrote:
> - limited or non-existent support for hot-swapping, even with SATA
>   (see http://www.redhat.com/archives/fedora-test-list/2004-March/msg01204.html)

This link seems to indicate "hot swap" means telling the machine you're
going to remove the device, and then removing the device.  To me, that's
not really hot swapping; I would prefer to be able to just yank to drive
and let the OS figure it out itself.

What is the common industry definition of hot swapping?

	Scott

Re: what is the best multi-SATA-controller-on-a-single-board out there?

[Tim Small]

Jon Lewis wrote:

>On Mon, 2 Aug 2004, Tim Small wrote:
>  
>
>>The Promise cards, are cheap, but only do 66Mhz/32 bit PCI and provide
>>four SATA ports, so they may not be well suited to some boards..  Both
>>chipsets were used with libata on 2.6 kernels, with a mixture of s/w
>>raid5, and s/w raid1.  The only slight irritation is the current lack of
>>smartd support (pending libata driver support).
>>    
>>
>
>With what distros / exactly which kernels?  I've been testing a system
>  
>
I'm using kernel.org 2.6.8rc2 (also used 2.6.5rc3), with libata on 
Debian/Sarge (not that the distribution should matter, this is all 
kernel side stuff) - both on Xeons, and Opterons, all filesystems are 
ext3.  I was expecting to have some hassle with these systems, but have 
had none up to now (in service a few months).  The revised (minor) 
shortcoming list is:

. Sustained throughput not as high as the PATA based arrays (yet)
. No hot-plug support (yet)
. No smartd support (yet)

Tim.

Software vs. Hardware RAID

[Tim Small]

Julian Cowley wrote:

>Recently I did a survey of this very question (hardware vs. software
>RAID) based on the comments from this mailing list:
>
>Software
>--------
>
>- CPU must handle operations
>- twice the I/O bandwidth when using RAID1
>  
>
Yes (during writes)

>+ non-proprietary disk format
>+ open source implementation
>- limited or non-existent support for hot-swapping, even with SATA
>  (see http://www.redhat.com/archives/fedora-test-list/2004-March/msg01204.html)
>  
>
I've swapped out SCSI drives with software RAID on a live system - it 
isn't 100% smooth, as it triggers a bus reset on these systems, and 
hence about 15 seconds of no-I/O, but the machine did work afterwards, 
and no reboot was required.  For SATA hot-swap, see this article:

http://kerneltrap.org/node/view/3432

>- OS-specific format (can't be shared between Linux, Windows, etc.)
>  
>
Well, you can configure a partition as mirrored using Linux software 
RAID, and then have Windows use the rest of the disk..  Whether you 
could then have Windows use it's own software RAID on the rest of the 
disk, I couldn't say..  As long as you kept access to read-only you 
could probably then read the whole of the fs content from both OS (why 
do you want to run Windows anyway? :o)

>+ drives can be anything (ie. a mixture of SATA, PATA, Firewire, USB, etc.)
>- disk surface testing must be done manually (7/2004)
>  
>
Smartd can automate this e.g. these lines in smartd.conf will tell the 
drives to do an extended self-test at 1am, and 2am on Saturday...

/dev/hda -a -s L/../../6/01 -m root
/dev/hdc -a -s L/../../6/02 -m root

This may catch blocks which are going bad before they become unreadable 
(i.e. when the hardware and/or firmware ECC algorithms are still able to 
reconstruct the data), and cause the drive to silently remap these 
blocks - so these may well save you an array degradation...

>- no bad block relocation (7/2004)
>  
>
Most drives will do this automatically, except in the event of data loss 
(i.e. if it can't reconstruct the correct data, it will just return a 
read error - if you try to write the entire block, it will then remap 
it) - with software RAID, you will end up with a degraded array at the 
moment.  It would be cool if the software raid subsystem would try to 
rewrite individual blocks which have had read failures (assuming it has 
info on the other disks, or n RAM to do this) before marking the whole 
partition as bad, but it doesn't at the moment (AFAIK).

I've had cases (on IBM 75GXP drives <spit>), where two drives in a 
mirror have independently had different unreadable sectors, and the 
hardware RAID controller has kicked out drives, and left the OS with an 
unusable array (although together, both drives have all the data - 
grrr).  If this was software RAID, the same thing would have happened, 
but at least I would have been able to manually copy bad blocks from the 
failed drive using dd, without taking down the OS.

>- no parity verification (7/2004)
>- no mirror verification (7/2004)
>  
>
True, but with the exception of kernel bugs, arrays shouldn't get into 
these states.  Would be a nice feature tho'.

>+ reputedly, much better performance than hardware raid
>  
>
Can be I think, yes.  e.g. I get ~120 MB/Sec linear device reads/writes 
on a 3x 10k rpm 75G (all drives on a single U320 SCSI bus) software 
raid5 array that I've built.  With modern CPUs, the processing overhead 
required for RAID is not highly significant - a bit higher if an array 
is degraded, and on RAID5 writes of course - e.g. see this kernel output 
on a dual Xeon 2.8GHz box:

raid5: using function: pIII_sse (3649.600 MB/sec)

And this on a dual Opteron 248

raid5: using function: generic_sse (6744.000 MB/sec)

so parity calculation is not a serious overhead these days, but the 
extra I/O may be - on the 2.8GHz Xeon box (which is the aforementioned 
3x 10k rpm SCSI machine, running 2.4.26), I see:

Read from RAID5:

119MB/Sec, with 25% kernel CPU usage

Read from RAID5 (degraded array):

127MB/Sec, with 60% kernel CPU usage

>Hardware
>--------
>
>+ off-loads the CPU
>+ I/O bandwidth needed on a RAID1 system is same as single disk
>  
>
again, this is only for writes, you get a similar effect with RAID5 
(e.g. a four disk RAID5 needs 1.25 times the writes)

>- proprietary disk format (although limited drivers are available for Linux)
>- proprietary implementation
>+ easy hot-swapping (some controllers even indicate the bad drive with an LED)
>+ non-OS-specific (can share between Linux, Windows, etc.)
>- some features may not be supported on non-Windows operating systems
>  
>
you can also add "non-Redhat kernels" to this list...

>+ able to create logical disks that seem like physical disks to the OS
>  
>
and associated with this - less trouble with boot loaders (e.g. booting 
from a degraded array as root fs)

>+ bad sector relocation (on the fly?)
>  
>
Depends on the controller  e.g. 3ware does now, but it didn't used to

>- drives must connect to the controller and all must be same type (e.g. SATA)
>+ disk surface testing done automatically
>+ automatic bad block relocation
>+ parity verification
>+ mirror verification
>  
>
You can add a "maybe" to the last four - all depends on the 
implementation, and if you can't get the management software to run on 
your kernel/distribution, then you may not get any of them (or degraded 
array notification!) without using the RAID controller's BIOS.

Add to this another negative - patchy SMART support (only 3ware supports 
smartd pass-through at the moment, AFAIK) - which is useful if you want 
more granularity than "drive good", or "drive bad", e.g. the ability to 
read serial numbers, firmware versions, drive temperatures, SMART error 
log entries, interface errors, remapped block count, spin-up count, 
power-on hours etc. whilst the OS is up and running.

Tim.

Re: Definition of hotswap, was 1x 3ware controllers vs. 2x 3ware controllers

[Tim Small]

Scott T. Smith wrote:

>This link seems to indicate "hot swap" means telling the machine you're
>going to remove the device, and then removing the device.  To me, that's
>not really hot swapping; I would prefer to be able to just yank to drive
>and let the OS figure it out itself.
>
>What is the common industry definition of hot swapping?
>  
>
I'd say the most common definition is the ability to swap-out a failed 
disk whilst power is connected to the system, and the OS remains up - 
sometimes informing the controller, sometimes not..

Re: Software vs. Hardware RAID

[Ricky Beam]

On Tue, 3 Aug 2004, Tim Small wrote:
>This may catch blocks which are going bad before they become unreadable
>(i.e. when the hardware and/or firmware ECC algorithms are still able to
>reconstruct the data), and cause the drive to silently remap these
>blocks - so these may well save you an array degradation...
...
>>- no bad block relocation (7/2004)
>Most drives will do this automatically, except in the event of data loss
...

Don't bet the farm on IDE drives remapping bad sectors.  There are two
Seagate drives on my desk that claim to have remapped the bad sectors
but still return errors for those sectors. (both via the seatools DOS
thing and Linux zeroing the entire (200G) drive.)

Remember: You get what you pay for.

Maybe we should start a class action suit to get SCSI prices down :-)

--Ricky

Re: what is the best multi-SATA-controller-on-a-single-board out there?

[Jon Lewis]

On Tue, 3 Aug 2004, Tim Small wrote:

> I'm using kernel.org 2.6.8rc2 (also used 2.6.5rc3), with libata on
> Debian/Sarge (not that the distribution should matter, this is all
> kernel side stuff) - both on Xeons, and Opterons, all filesystems are

It matters to the extent that some of the distros like to use heavily
patched kernels...so one distro's 2.4.x kernel may be quite different from
another's.  So unless you build custom kernels from the kernel.org
sources, kernel verion _and_ distro is more meaningful.

It seems (from Maxtor's powermax util) that I do have at least one
defective drive...so hopefully that's all the problem was.  I'm building
an array now with the remaining 5 drives that passed the powermax tests.

----------------------------------------------------------------------
 Jon Lewis                   |  I route
 Senior Network Engineer     |  therefore you are
 Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

Re: what is the best multi-SATA-controller-on-a-single-board out there?

[Tim Small]

Jon Lewis wrote:

>some of the distros like to use heavily
>patched kernels...so one distro's 2.4.x kernel may be quite different from
>another's.  So unless you build custom kernels from the kernel.org
>sources, kernel verion _and_ distro is more meaningful.
>
>  
>
Sorry, I wasn't being clear - I interpreted "kernel version" as 
including specifying what distro patches were included.  e.g. Redhat 
EL's 2.4.9, or Debian's 2.4.26, kernel.org's 2.6.8rc2 etc. since you can 
- for example - run the redhat kernel, with Debian's userland e.g. 
http://packages.debian.org/testing/devel/kernel-patch-redhat if you need 
to use some Redhat-only binary driver, or kernel feature).

Tim.

Re: Definition of hotswap, was 1x 3ware controllers vs. 2x 3ware controllers

[Maurice Hilarius]

With regards to your message at 05:55 AM 8/3/04, Tim Small. Where you stated:
>Scott T. Smith wrote:
>
>>This link seems to indicate "hot swap" means telling the machine you're
>>going to remove the device, and then removing the device.  To me, that's
>>not really hot swapping; I would prefer to be able to just yank to drive
>>and let the OS figure it out itself.
>>
>>What is the common industry definition of hot swapping?
>>
>I'd say the most common definition is the ability to swap-out a failed 
>disk whilst power is connected to the system, and the OS remains up - 
>sometimes informing the controller, sometimes not..


I think one has to add to that:

"While allowing the filesystem to remain available to the OS and 
applications, and without the loss or corruption of data, preferably 
without the requirement for manual intervention by a user or administrator."



With our best regards,

Maurice W. Hilarius       Telephone: 01-780-456-9771
Hard Data Ltd.               FAX:       01-780-456-9772
11060 - 166 Avenue        mailto:maurice@harddata.com
Edmonton, AB, Canada      http://www.harddata.com/
    T5X 1Y3