Linux SATA RAID FAQ

Linux SATA RAID FAQ

[Jeff Garzik]

As author of the current Linux SATA driver, I get the brunt of the 
questions and "bug reports" about "Linux doesn't support my hardware 
SATA RAID".  Sigh.  Silly marketing departments.

Thus, I have created a FAQ.  You might sense a theme...

http://www.kernel.org/pub/linux/kernel/people/jgarzik/faq-sata-raid.html

Re: Linux SATA RAID FAQ

[Michael Knigge]

> Thus, I have created a FAQ.  You might sense a theme...
> 
http://www.kernel.org/pub/linux/kernel/people/jgarzik/faq-sata-raid.html


;-) Funny.....

I don't care about RAID, but could you tell which S-ATA 
Controller/Chipset is the best for Hot-Swap?


Thanks,
  Michael

Re: Linux SATA RAID FAQ

[Paul Ionescu]

On Thu, 12 Aug 2004 07:18:10 +0000, Michael Knigge wrote:
> I don't care about RAID, but could you tell which S-ATA 
> Controller/Chipset is the best for Hot-Swap?

Hi Michael,

I think 3ware is the most stable right now, and has his own separate
driver. It is even seen as a SCSI controller by linux, but has SATA or
PATA disks.
Maybe there are some others, but I heard this is stable right now, and
hot-swapping is working on it.

Re: Linux SATA RAID FAQ

[Willy Tarreau]

Hi Jeff,

> http://www.kernel.org/pub/linux/kernel/people/jgarzik/faq-sata-raid.html

I like it. It's fairly simple. I'm always amazed how many people do really
believe that these cards provide hardware RAID !!! The problem is when you
ask a reseller to add a real hardware RAID card in a system you purchase
and you end up with a cheap silicon image... It happened to us once and it's
not funny at all.

Cheers,
Willy

Re: Linux SATA RAID FAQ

[Alan Cox]

On Iau, 2004-08-12 at 14:38, Bernd Eckenfels wrote:
> Speaing of that, does that mean that other OSes (i.e. Windows) are using
> BIOS provided code to do raid, or do they also have raid software drivers
> and the bios is only used on bootup for signature detection and formatting?

Normally BIOS and windows drivers doing their raid. It isn't entirely
that simple. The 3ware is hardware raid as are some of the other high
end devices (eg aacraid sata boards). There are also some low end
devices with part of the raid logic in hardware (some promise) although
I don't believe we use that to the full yet.

I'm currently trying to fix up the IT8212 which is an older PATA board
which does have real h/w raid 0/1

Alan

Re: Linux SATA RAID FAQ

[Bernd Eckenfels]

In article <20040812113413.GA19252@alpha.home.local> you wrote:
> I like it. It's fairly simple. I'm always amazed how many people do really
> believe that these cards provide hardware RAID !!! The problem is when you
> ask a reseller to add a real hardware RAID card in a system you purchase
> and you end up with a cheap silicon image... It happened to us once and it's
> not funny at all.

Speaing of that, does that mean that other OSes (i.e. Windows) are using
BIOS provided code to do raid, or do they also have raid software drivers
and the bios is only used on bootup for signature detection and formatting?

Greetings
Bernd
-- 
eckes privat - http://www.eckes.org/
Project Freefire - http://www.freefire.org/

Re: Linux SATA RAID FAQ

[Dieter Stueken]

Hello Jeff,

thanks for the FAQ. Do you also have a link to
some FAQ not dealing with RAID and hotplug, but
explaining how to simply "coldplug" SATA drives?
I want to use them much in the same way like USB
disks can be used.

I picked up some fragments about /sys/class/scsi_host
scans and some /sbin/sata (script?) but I could not
find any more details or howtos.

Dieter.
-- 
Dieter Stüken, con terra GmbH, Münster
     stueken@conterra.de
     http://www.conterra.de/
     (0)251-7474-501

Re: Linux SATA RAID FAQ

[Alistair John Strachan]

On Thursday 12 August 2004 14:38, you wrote:
> In article <20040812113413.GA19252@alpha.home.local> you wrote:
> > I like it. It's fairly simple. I'm always amazed how many people do
> > really believe that these cards provide hardware RAID !!! The problem is
> > when you ask a reseller to add a real hardware RAID card in a system you
> > purchase and you end up with a cheap silicon image... It happened to us
> > once and it's not funny at all.
>
> Speaing of that, does that mean that other OSes (i.e. Windows) are using
> BIOS provided code to do raid, or do they also have raid software drivers
> and the bios is only used on bootup for signature detection and formatting?
>
> Greetings
> Bernd

Windows also has "software drivers". Since vendors provide drivers 
independently of Microsoft, they can put precisely what they like in them.

-- 
Cheers,
Alistair.

personal:   alistair()devzero!co!uk
university: s0348365()sms!ed!ac!uk
student:    CS/AI Undergraduate
contact:    1F2 55 South Clerk Street,
            Edinburgh. EH8 9PP.

Re: Linux SATA RAID FAQ

[Jeff Garzik]

Alan Cox wrote:
> end devices (eg aacraid sata boards). There are also some low end
> devices with part of the raid logic in hardware (some promise) although
> I don't believe we use that to the full yet.

Nope.  My SATA RAID FAQ mentions the Promise "RAID accelerator" stuff. 
In particular, the Promise SX4 is in fact likely to be faster with the 
_Promise_ RAID driver, rather than the Linux non-RAID driver.

I really like the SX4, and wish Linux could make better use of it...

The SX4 has an on-board DIMM (128M - 2G), through which all data _must_ 
pass.  The data transfer between host and on-board DIMM is a separate 
DMA engine and separate interrupt event from the four ATA DMA engines 
(one per SATA port).  There are several possibilities that are worth 
exploring on this card:

* Caching
* Eliminate PCI bus traffic by sending RAID1/5 writes a _single_ time to 
the card, and then multiplex to multiple attached drives from there
* Offload RAID5 XOR calculations, which becomes quite useful in 
combination with these other features
* Execute RAID1/5 resyncs and parity checks completely on the card

I have full docs and knowledge, but no idea how to hook this stuff into 
Linux md efficiently.

	Jeff

Re: Linux SATA RAID FAQ

[Jeff Garzik]

Jeff Garzik wrote:
> Alan Cox wrote:
> 
>> end devices (eg aacraid sata boards). There are also some low end
>> devices with part of the raid logic in hardware (some promise) although
>> I don't believe we use that to the full yet.
> 
> 
> Nope.  My SATA RAID FAQ mentions the Promise "RAID accelerator" stuff. 

Clarification:  "nope" == "nope, we don't use that to the full yet"


> The SX4 has an on-board DIMM (128M - 2G), through which all data _must_ 
> pass.  The data transfer between host and on-board DIMM is a separate 
> DMA engine and separate interrupt event from the four ATA DMA engines 
> (one per SATA port).  There are several possibilities that are worth 
> exploring on this card:
> 
> * Caching
> * Eliminate PCI bus traffic by sending RAID1/5 writes a _single_ time to 
> the card, and then multiplex to multiple attached drives from there
> * Offload RAID5 XOR calculations, which becomes quite useful in 
> combination with these other features
> * Execute RAID1/5 resyncs and parity checks completely on the card

And one more:  the Promise hardware allows multiple disk transactions to 
be chained together in a sequence, such that, you only receive an 
interrupt when the full sequence is complete (or there is an error). 
You can look at it as either interrupt coalescing, or simply coalescing 
of multiple low-level disk transactions into a single "RAID transaction."

	Jeff

Re: Linux SATA RAID FAQ

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 506 bytes --]

On Thu, 2004-08-12 at 08:33, Jeff Garzik wrote:
> As author of the current Linux SATA driver, I get the brunt of the 
> questions and "bug reports" about "Linux doesn't support my hardware 
> SATA RAID".  Sigh.  Silly marketing departments.
> 
> Thus, I have created a FAQ.  You might sense a theme...
> 
> http://www.kernel.org/pub/linux/kernel/people/jgarzik/faq-sata-raid.html
> 

Would be nice if somebody though could add how to compile dmraid
with make-3.80 :/


-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Linux SATA RAID FAQ

[J. Ryan Earl]

Willy Tarreau wrote:

>I like it. It's fairly simple. I'm always amazed how many people do really
>believe that these cards provide hardware RAID !!! The problem is when you
>ask a reseller to add a real hardware RAID card in a system you purchase
>and you end up with a cheap silicon image... It happened to us once and it's
>not funny at all.
>
On the brightside, md raid5 is often faster than hardware raid5.  At 
least on the 7000 and 8000 series of 3ware hardware; the 9000 series 
looks promising though.  I haven't seen megaraid SATA numbers, and I 
don't know what happened to the SX8.

When the libata Marvell drivers come out, you'll have a cheap upgrade 
path for PCI-X boards if you want fast md raid: 
http://www.supermicro.com/products/accessories/addon/DAC-SATA-MV8.cfm  
$100 to add 8 unbottlenecked SATA ports to your server motherboard.

-ryan

Re: Linux SATA RAID FAQ

[Mark Lord]

Yup SATA is catching on quickly.

I expect to be releasing a new SATA/RAID driver here
within a few weeks as well.  PCI-X, TCQ, H/W RAID 0/1/10, ..

Cheers
-- 
Mark Lord
(hdparm keeper & the original "Linux IDE Guy")

Re: Linux SATA RAID FAQ

[age huisman]

Mark Lord wrote:

> Yup SATA is catching on quickly.
> 
> I expect to be releasing a new SATA/RAID driver here
> within a few weeks as well.  PCI-X, TCQ, H/W RAID 0/1/10, ..

Hi Mark,

TCQ ?   What about NCQ?  :-)

Age

Re: Linux SATA RAID FAQ

[Mark Lord]

age huisman wrote:
> TCQ ?   What about NCQ?  :-)

Both kinds of SATA command-queuing will be included.\,
whatever the name/abbreviation.

hdparm support as well (a first for SATA/SCSI).

Cheers
-- 
Mark Lord
(hdparm keeper & the original "Linux IDE Guy")

Re: Linux SATA RAID FAQ

[Alan Cox]

On Sad, 2004-08-14 at 01:31, J. Ryan Earl wrote:
> On the brightside, md raid5 is often faster than hardware raid5.  At 
> least on the 7000 and 8000 series of 3ware hardware; the 9000 series 
> looks promising though.  I haven't seen megaraid SATA numbers, and I 
> don't know what happened to the SX8.

Be cautious what you measure. One of he problems until you reach PCI-X
is PCI bandwidth. Thus software md5 can look good but the moment its
combined with other PCI activity goes down the pan entirely.

> When the libata Marvell drivers come out, you'll have a cheap upgrade 
> path for PCI-X boards if you want fast md raid: 

Agreed. PCI-X will change a lot of this for boxes that are not very
cpu/memory limited.

Re: Linux SATA RAID FAQ

[Alan Cox]

On Iau, 2004-08-12 at 18:30, Jeff Garzik wrote:
> > The SX4 has an on-board DIMM (128M - 2G), through which all data _must_ 
> > pass.  The data transfer between host and on-board DIMM is a separate 
> > DMA engine and separate interrupt event from the four ATA DMA engines 
> > (one per SATA port).  There are several possibilities that are worth 
> > exploring on this card:
> > 
> > * Caching

Is it battery backed ? If it is battery backed then its useful, if not
then it becomes less useful although not always. The i2o drivers have
some ioctls so you can turn on writeback caching even without battery
backup. While this is suicidal for filesytems its just great for swap..

Re: Linux SATA RAID FAQ

[J. Ryan Earl]

Alan Cox wrote:

>Be cautious what you measure. One of he problems until you reach PCI-X
>is PCI bandwidth. Thus software md5 can look good but the moment its
>combined with other PCI activity goes down the pan entirely.
>  
>
Right, which is why you'd think hardware based RAID would fair better, 
the parity information or mirror'd writes would only require one 
transfer/transaction across the PCI bus.  However, when benchmarking a 
3Ware 4-port ide raid controller (7000) we saw 40-50MB/sec read/write on 
4x160GB drives raid5.  With md raid5 (same controller, HDs, FS, etc)  we 
saw 100MB/sec read, 60MB/sec write.  This was using max 15% CPU on a 
2.4GHz Pentium4, with a 32bit/33MHz PCI bus.

>  
>
>>When the libata Marvell drivers come out, you'll have a cheap upgrade 
>>path for PCI-X boards if you want fast md raid: 
>>    
>>
>
>Agreed. PCI-X will change a lot of this for boxes that are not very
>cpu/memory limited.
>  
>
 From the testing I've done, interconnect bandwidth has always been the 
limiting factor for the md driver.  Using cheap ($~230) PCI-X 
motherboards--http://www.supermicro.com/products/motherboard/P4/E7210/P4SCT+.cfm--with 
dedicated gigE channels, you can make high density/price and 
performance/price NAS type appliances with no bottlenecks.  For <$3K you 
can build a 2TB NAS server that'll keep a 250MB/s gigE link saturated.

-ryan

Re: Linux SATA RAID FAQ

[Jeff Garzik]

On Sat, Aug 14, 2004 at 10:49:24PM +0100, Alan Cox wrote:
> On Iau, 2004-08-12 at 18:30, Jeff Garzik wrote:
> > > The SX4 has an on-board DIMM (128M - 2G), through which all data _must_ 
> > > pass.  The data transfer between host and on-board DIMM is a separate 
> > > DMA engine and separate interrupt event from the four ATA DMA engines 
> > > (one per SATA port).  There are several possibilities that are worth 
> > > exploring on this card:
> > > 
> > > * Caching
> 
> Is it battery backed ? If it is battery backed then its useful, if not
> then it becomes less useful although not always. The i2o drivers have
> some ioctls so you can turn on writeback caching even without battery
> backup. While this is suicidal for filesytems its just great for swap..

Nope not battery backed...

	Jeff

Re: Linux SATA RAID FAQ

[Alan Cox]

On Sul, 2004-08-15 at 21:54, Xavier Bestel wrote:
> Isn't sufficient to have it do ordered writes ? If you power your
> machine off, you'll have things half-written anyway, the only thing
> important with journaled filesystems (and raid5 arrays) is to have
> writes staying between barriers.

True to a point but 2Gb of data going walkies will offend even if the
file system is gloriously intact

Re: Linux SATA RAID FAQ

[Xavier Bestel]

Le sam 14/08/2004 à 23:49, Alan Cox a écrit :
> > > * Caching
> 
> Is it battery backed ? If it is battery backed then its useful, if not
> then it becomes less useful although not always. The i2o drivers have
> some ioctls so you can turn on writeback caching even without battery
> backup. While this is suicidal for filesytems its just great for swap..

Isn't sufficient to have it do ordered writes ? If you power your
machine off, you'll have things half-written anyway, the only thing
important with journaled filesystems (and raid5 arrays) is to have
writes staying between barriers.

	Xav

Re: Linux SATA RAID FAQ

[Jakob Oestergaard]

On Sun, Aug 15, 2004 at 10:54:02PM +0200, Xavier Bestel wrote:
> Le sam 14/08/2004 à 23:49, Alan Cox a écrit :
> > > > * Caching
> > 
> > Is it battery backed ? If it is battery backed then its useful, if not
> > then it becomes less useful although not always. The i2o drivers have
> > some ioctls so you can turn on writeback caching even without battery
> > backup. While this is suicidal for filesytems its just great for swap..
> 
> Isn't sufficient to have it do ordered writes ? If you power your
> machine off, you'll have things half-written anyway, the only thing
> important with journaled filesystems (and raid5 arrays) is to have
> writes staying between barriers.

On a RAID controller with battery backed write-back cache, it can
complete a "sync" operation as soon as the data is in the controller
cache.   This gives a significant performance speedup.

If the cache is not battery backed and power is lost after such a "sync"
operation, then data that the kernel/userspace thought was sync'ed to
disk is actually lost.

This *will* break journalling filesystems and databases.  They work from
the assumption that 'sync' means 'sync' - and if it doesn't, then all
hell breaks lose.

It is common to enforce ordering of writes by issuing a 'sync' of some
data and after the sync completes then starting the writeout of the
'next' data.  On a controller with write-back cache without battery
backup, you could actually risk that your 'next' data were written
before the data you just issued a 'sync' for.

In other words;  write-back cache without battery backup is absolutely
insane, except for some few isolated cases, such as swap-space that Alan
pointed out.

-- 

 / jakob

Re: Linux SATA RAID FAQ

[Bernd Eckenfels]

On Sun, Aug 15, 2004 at 09:51:46PM +0100, Alan Cox wrote:
> True to a point but 2Gb of data going walkies will offend even if the
> file system is gloriously intact

Especially if the data was already visible in the shared filesystem space.

Greetings
Bernd
-- 
  (OO)      -- Bernd_Eckenfels@Mörscher_Strasse_8.76185Karlsruhe.de --
 ( .. )      ecki@{inka.de,linux.de,debian.org}  http://www.eckes.org/
  o--o     1024D/E383CD7E  eckes@IRCNet  v:+497211603874  f:+497211606754
(O____O)  When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl!

Re: Linux SATA RAID FAQ

[Alan Cox]

On Llu, 2004-08-16 at 23:06, Andrew Morton wrote:
> I'm sitting on the vendor's driver for these cards.  How does your work
> differ from this?

It uses the IDE layer instead of badly duplicating it in essence.

> hch questioned why we need the driver at all: just put the card in JBOD
> mode and use s/w raid drivers.  But the thing does have an on-board CPU and
> the idea is that by offloading to that, the data transits the bus just a
> single time.  The developers are off doing some comparative benchmarking at
> present.

On my set up raid1 is materially faster using their processor and raid0
is materially faster not. There are also co-existance issues with
Windows dual boot setups. The PCI single copy makes a big difference on
a 32bit/33Mhz plug in card.

Alan

Re: Linux SATA RAID FAQ

[Andrew Morton]

Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>
> I'm currently trying to fix up the IT8212 which is an older PATA board
>  which does have real h/w raid 0/1

I'm sitting on the vendor's driver for these cards.  How does your work
differ from this?

hch questioned why we need the driver at all: just put the card in JBOD
mode and use s/w raid drivers.  But the thing does have an on-board CPU and
the idea is that by offloading to that, the data transits the bus just a
single time.  The developers are off doing some comparative benchmarking at
present.

Re: Linux SATA RAID FAQ

[Christoph Hellwig]

On Mon, Aug 16, 2004 at 03:06:41PM -0700, Andrew Morton wrote:
> hch questioned why we need the driver at all: just put the card in JBOD
> mode and use s/w raid drivers.  But the thing does have an on-board CPU and
> the idea is that by offloading to that, the data transits the bus just a
> single time.  The developers are off doing some comparative benchmarking at
> present.

Alan's driver can use the chip in raid mode.