Linux: Why software RAID?

Linux: Why software RAID?

[Jeff Garzik]

Mark Perkel wrote:
> Running Linux on an AMD AM2 nVidia chip ser that supports Raid 0
> striping on the motherboard. Just wondering if hardware raid (SATA2) is
> going to be faster that software raid and why? 


First, it sounds like you are confusing motherboard "RAID" with real 
RAID.  There's a FAQ for this sort of thing:

	http://linux-ata.org/faq-sata-raid.html

In particular, your motherboard's Raid 0 striping (a) is not done in 
hardware, and (b) has nothing to do with SATA2.

But anyway, to help answer the question of hardware vs. software RAID, I 
wrote up a page:

	http://linux.yyz.us/why-software-raid.html

Generally, you want software RAID unless your PCI bus (or more rarely, 
your CPU) is getting saturated.  With RAID-0, there is no duplication of 
data, and so, PCI bus and CPU usage should be about the same for 
hardware and software RAID.

	Jeff

Re: Linux: Why software RAID?

[Richard Scobie]

Jeff Garzik wrote:
> Mark Perkel wrote:
> 
>> Running Linux on an AMD AM2 nVidia chip ser that supports Raid 0
>> striping on the motherboard. Just wondering if hardware raid (SATA2) is
>> going to be faster that software raid and why? 
> 

Jeff, on a slightly related note, is the driver status for the NVIDIA as 
reflected on your site, correct for the new nForce 590/570 AM2 chipset?

Regards,

Richard

Re: Linux: Why software RAID?

[Chris Friesen]

Jeff Garzik wrote:

> But anyway, to help answer the question of hardware vs. software RAID, I 
> wrote up a page:
> 
>     http://linux.yyz.us/why-software-raid.html

Just curious...with these guys 
(http://www.bigfootnetworks.com/KillerOverview.aspx) putting linux on a 
PCI NIC to allow them to bypass Windows' network stack, has anyone ever 
considered doing "hardware" raid by using an embedded cpu running linux 
software RAID, with battery-backed memory?

It would theoretically allow you to remain feature-compatible by 
downloading new kernels to your RAID card.

Chris

Re: Linux: Why software RAID?

[H. Peter Anvin]

Chris Friesen wrote:
> Jeff Garzik wrote:
> 
>> But anyway, to help answer the question of hardware vs. software RAID, 
>> I wrote up a page:
>>
>>     http://linux.yyz.us/why-software-raid.html
> 
> Just curious...with these guys 
> (http://www.bigfootnetworks.com/KillerOverview.aspx) putting linux on a 
> PCI NIC to allow them to bypass Windows' network stack, has anyone ever 
> considered doing "hardware" raid by using an embedded cpu running linux 
> software RAID, with battery-backed memory?
> 
> It would theoretically allow you to remain feature-compatible by 
> downloading new kernels to your RAID card.
> 

Yes.  In fact, I have been told by several RAID chip vendors that their 
customers are *strongly* demanding that their chips be able to run Linux 
  md (and still use whatever hardware offload features.)

So it's happening.

	-hpa

Re: Linux: Why software RAID?

[Mattias Wadenstein]

On Wed, 23 Aug 2006, Chris Friesen wrote:

> Jeff Garzik wrote:
>
>> But anyway, to help answer the question of hardware vs. software RAID, I 
>> wrote up a page:
>>
>>     http://linux.yyz.us/why-software-raid.html
>
> Just curious...with these guys 
> (http://www.bigfootnetworks.com/KillerOverview.aspx) putting linux on a PCI 
> NIC to allow them to bypass Windows' network stack, has anyone ever 
> considered doing "hardware" raid by using an embedded cpu running linux 
> software RAID, with battery-backed memory?

I'd expect this to be the reason why md offload support to xor engines and 
whatever turns up. It makes very little sense for a modern server/desktop 
CPU, but for the embedded ones it does.

/Mattias Wadenstein

Re: Linux: Why software RAID?

[Jeff Garzik]

Richard Scobie wrote:
> Jeff Garzik wrote:
>> Mark Perkel wrote:
>>
>>> Running Linux on an AMD AM2 nVidia chip ser that supports Raid 0
>>> striping on the motherboard. Just wondering if hardware raid (SATA2) is
>>> going to be faster that software raid and why? 
>>
> 
> Jeff, on a slightly related note, is the driver status for the NVIDIA as 
> reflected on your site, correct for the new nForce 590/570 AM2 chipset?

Unfortunately I rarely have an idea about how marketing names correlate 
to chipsets.

Do you have a PCI ID (lspci -n)?

	Jeff

Re: Linux: Why software RAID?

[Adam Kropelin]

Jeff Garzik <jeff@garzik.org> wrote:
> But anyway, to help answer the question of hardware vs. software RAID, I 
> wrote up a page:
> 
> 	http://linux.yyz.us/why-software-raid.html
> 
> Generally, you want software RAID unless your PCI bus (or more rarely, 
> your CPU) is getting saturated.  With RAID-0, there is no duplication of 
> data, and so, PCI bus and CPU usage should be about the same for 
> hardware and software RAID.

Hardware RAID can be (!= is) more tolerant of serious drive failures
where a single drive locks up the bus. A high-end hardware RAID card 
may be designed with independent controllers so a single drive failure
cannot take other spindles down with it. The same can be accomplished 
with sw RAID of course if the builder is careful to use multiple PCI 
cards, etc. Sw RAID over your motherboard's onboard controllers leaves
you vulnerable.

--Adam

Re: Linux: Why software RAID?

[Alan Cox]

Ar Iau, 2006-08-24 am 09:07 -0400, ysgrifennodd Adam Kropelin:
> Jeff Garzik <jeff@garzik.org> wrote:
> with sw RAID of course if the builder is careful to use multiple PCI 
> cards, etc. Sw RAID over your motherboard's onboard controllers leaves
> you vulnerable.

Generally speaking the channels on onboard ATA are independant with any
vaguely modern card. And for newer systems well the motherboard tends to
be festooned with random SATA controllers, all separate!

Re: Linux: Why software RAID?

[Adam Kropelin]

On Thu, Aug 24, 2006 at 02:20:50PM +0100, Alan Cox wrote:
> Ar Iau, 2006-08-24 am 09:07 -0400, ysgrifennodd Adam Kropelin:
> > Jeff Garzik <jeff@garzik.org> wrote:
> > with sw RAID of course if the builder is careful to use multiple PCI 
> > cards, etc. Sw RAID over your motherboard's onboard controllers leaves
> > you vulnerable.
> 
> Generally speaking the channels on onboard ATA are independant with any
> vaguely modern card. 

Ahh, I did not know that. Does this apply to master/slave connections on
the same PATA cable as well? I know zero about PATA, but I assumed from
the terminology that master and slave needed to cooperate rather closely.

> And for newer systems well the motherboard tends to
> be festooned with random SATA controllers, all separate!

And how. You can't swing a dead cat without hitting a half-dozen ATA
ports these days. And most of them are those infuriatingly insecure SATA
connectors that pop off when you look at them cross-eyed...

--Adam

Re: Linux: Why software RAID?

[Gordon Henderson]

On Thu, 24 Aug 2006, Adam Kropelin wrote:

> > Generally speaking the channels on onboard ATA are independant with any
> > vaguely modern card.
>
> Ahh, I did not know that. Does this apply to master/slave connections on
> the same PATA cable as well? I know zero about PATA, but I assumed from
> the terminology that master and slave needed to cooperate rather closely.

I don't know much about co-operation between master & slave, but I do know
that a failing PATA IDE drive can take out the other one on the same bus -
or in my case, render it unusable until I removed the dead drive,
whereupon (to my relief) it sprang back into life.

This was many many moons ago before I started to use s/w RAID, but it's
one thing that would kill a multi-disk array, so I've never done it since.

I guess the same could happen on SCSI, but I suspect the interface is a
little better designed...

Gordon

Re: Linux: Why software RAID?

[Mark Lord]

Adam Kropelin wrote:
> On Thu, Aug 24, 2006 at 02:20:50PM +0100, Alan Cox wrote:
>> Generally speaking the channels on onboard ATA are independant with any
>> vaguely modern card. 
> 
> Ahh, I did not know that. Does this apply to master/slave connections on
> the same PATA cable as well?

No, it doesn't.  Except for cards which use special cables,
such as the Pacific Digital ADMA cards (which can even run both
master and slave simultaneously on a cable, though not with
the current Linux drivers).

Cheers

Re: Linux: Why software RAID?

[Alan Cox]

Ar Iau, 2006-08-24 am 07:31 -0700, ysgrifennodd Marc Perkel:
> So - the bottom line answer to my question is that unless you are
> running raid 5 and you have a high powered raid card with cache and
> battery backup that there is no significant speed increase to use
> hardware raid. For raid 0 there is no advantage.
> 
If your raid is entirely on PCI plug in cards and you are doing RAID1
there is a speed up using hardware assisted raid because of the PCI bus
contention. If your controllers are PCI express, on internal high speed
busses (eg chipset controllers) or at least half of them are then
generally there is no win with hardware raid 0/1 and it is often slower.

Re: Linux: Why software RAID?

[Richard Scobie]

Jeff Garzik wrote:
> Richard Scobie wrote:

>> Jeff, on a slightly related note, is the driver status for the NVIDIA 
>> as reflected on your site, correct for the new nForce 590/570 AM2 
>> chipset?
> 
> 
> Unfortunately I rarely have an idea about how marketing names correlate 
> to chipsets.
> 
> Do you have a PCI ID (lspci -n)?

Unfortunately not, as I am researchiung prior to purchase and Google has 
not thrown up anything useful.

However, the chipset numbers for the nForce 590 are C51Xe and MCP55PXE.

I think though, that I have answered the question. According to this 
NVIDIA page containing a download that appears to be the source from the 
kernel, "These drivers have been fully tested with nForce 570/590."

http://www.nvidia.co.uk/object/linux_nforce_1.11_uk.html

Regards,

Richard

Re: Linux: Why software RAID?

[linux]

> Hardware RAID can be (!= is) more tolerant of serious drive failures
> where a single drive locks up the bus. A high-end hardware RAID card 
> may be designed with independent controllers so a single drive failure
> cannot take other spindles down with it. The same can be accomplished 
> with sw RAID of course if the builder is careful to use multiple PCI 
> cards, etc. Sw RAID over your motherboard's onboard controllers leaves
> you vulnerable.

Which is exactly why I *like* SW RAID - I can, and do, have the mirrors
span controllers so a whole controller can fail without taking down
the system.

With HW RAID cards, if your controller dies, you're SOL.

Re: Linux: Why software RAID?

[Raz Ben-Jehuda(caro)]

Furthremore , hw controller are much less feaure rich than sw raid.
many different stripe sizes, stripe cache tunning ....

On 25 Aug 2006 23:50:34 -0400, linux@horizon.com <linux@horizon.com> wrote:
> > Hardware RAID can be (!= is) more tolerant of serious drive failures
> > where a single drive locks up the bus. A high-end hardware RAID card
> > may be designed with independent controllers so a single drive failure
> > cannot take other spindles down with it. The same can be accomplished
> > with sw RAID of course if the builder is careful to use multiple PCI
> > cards, etc. Sw RAID over your motherboard's onboard controllers leaves
> > you vulnerable.
>
> Which is exactly why I *like* SW RAID - I can, and do, have the mirrors
> span controllers so a whole controller can fail without taking down
> the system.
>
> With HW RAID cards, if your controller dies, you're SOL.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Raz

Re: Linux: Why software RAID?

[Dan Williams]

On 8/23/06, H. Peter Anvin <hpa@zytor.com> wrote:
> Chris Friesen wrote:
> > Jeff Garzik wrote:
> >
> >> But anyway, to help answer the question of hardware vs. software RAID,
> >> I wrote up a page:
> >>
> >>     http://linux.yyz.us/why-software-raid.html
> >
> > Just curious...with these guys
> > (http://www.bigfootnetworks.com/KillerOverview.aspx) putting linux on a
> > PCI NIC to allow them to bypass Windows' network stack, has anyone ever
> > considered doing "hardware" raid by using an embedded cpu running linux
> > software RAID, with battery-backed memory?
> >
> > It would theoretically allow you to remain feature-compatible by
> > downloading new kernels to your RAID card.
> >
>
> Yes.  In fact, I have been told by several RAID chip vendors that their
> customers are *strongly* demanding that their chips be able to run Linux
>   md (and still use whatever hardware offload features.)
>
> So it's happening.
Speaking of md with hardware offload features:

http://prdownloads.sourceforge.net/xscaleiop/ols_paper_2006.pdf?download

>         -hpa

Dan

Re: Linux: Why software RAID?

[Bill Davidsen]

Gordon Henderson wrote:

>On Thu, 24 Aug 2006, Adam Kropelin wrote:
>
>  
>
>>>Generally speaking the channels on onboard ATA are independant with any
>>>vaguely modern card.
>>>      
>>>
>>Ahh, I did not know that. Does this apply to master/slave connections on
>>the same PATA cable as well? I know zero about PATA, but I assumed from
>>the terminology that master and slave needed to cooperate rather closely.
>>    
>>
>
>I don't know much about co-operation between master & slave, but I do know
>that a failing PATA IDE drive can take out the other one on the same bus -
>or in my case, render it unusable until I removed the dead drive,
>whereupon (to my relief) it sprang back into life.
>
>This was many many moons ago before I started to use s/w RAID, but it's
>one thing that would kill a multi-disk array, so I've never done it since.
>
>I guess the same could happen on SCSI, but I suspect the interface is a
>little better designed...
>
Until recently I was working with 38 systems using SCSI RAID controllers 
(IBM ServeRAID Ultra320). With several types of SCSI drives I saw 
failures where one drive failed, hung the bus, and caused the next 
command to another drive to fail. At that point I have to force the 
controller to think the 2nd drive failed was okay, and then it would 
recover. I'm told this happens with other hardware, I just haven't 
personally seen it.

 From that standpoint, the SATA on the MB look pretty good!

-- 

bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: Linux: Why software RAID?

[Joel Jaeggli]

Bill Davidsen wrote:
> Alan Cox wrote:
> 
>> Ar Iau, 2006-08-24 am 07:31 -0700, ysgrifennodd Marc Perkel:
>>  
>>
>>> So - the bottom line answer to my question is that unless you are
>>> running raid 5 and you have a high powered raid card with cache and
>>> battery backup that there is no significant speed increase to use
>>> hardware raid. For raid 0 there is no advantage.
>>>
>>>   
>> If your raid is entirely on PCI plug in cards and you are doing RAID1
>> there is a speed up using hardware assisted raid because of the PCI bus
>> contention.
>>
> 
> I would expect to see this with RAID5 as well, for the same reason...

assuming you actually have lots of pci contention that might be a
consideration... if you're sitting on server class hardware with
multiple pci buses or using pci-express cards that won't be a
significant issue.

-- 
------------------------------------------------------------------------
Joel Jaeggli             Unix Consulting              joelja@uoregon.edu
GPG Key Fingerprint:   5C6E 0104 BAF0 40B0 5BD3 C38B F000 35AB B67F 56B2

Re: Linux: Why software RAID?

[Bill Davidsen]

Alan Cox wrote:

>Ar Iau, 2006-08-24 am 07:31 -0700, ysgrifennodd Marc Perkel:
>  
>
>>So - the bottom line answer to my question is that unless you are
>>running raid 5 and you have a high powered raid card with cache and
>>battery backup that there is no significant speed increase to use
>>hardware raid. For raid 0 there is no advantage.
>>
>>    
>>
>If your raid is entirely on PCI plug in cards and you are doing RAID1
>there is a speed up using hardware assisted raid because of the PCI bus
>contention.
>

I would expect to see this with RAID5 as well, for the same reason...

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979