dmraid - where is the raid done?

dmraid - where is the raid done?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1250 bytes --]

Somebody had suggested this list for this question:

I understand that dmraid is just some kind of interface to the
_software_ raid that is provided on a number of SATA interface cards
such as the Promise and Adaptec cards.

Reading http://ata.wiki.kernel.org/index.php/SATA_RAID_FAQ I get the
message loud and clear that this is not real hardware RAID but rather
it's done in the BIOS on the card.

The message there seems to be, if you are going to do it in software
anyway, why not just skip the IO card RAID and just use MD?

The one reason I could think of for using the BIOS provided RAID would
be to reduce the data needing to traverse the PCI bus.  With host-raid
(i.e. MD) every write to disk needs to actually be written over the PCI
bus twice, once for each disk, right?

But do the BIOS RAID cards out there only need the write command and
data once and they do the work of writing it to the two disks on-board
the IO card?

IIUC, when you have a BIOS RAID configured, even though there are two
disks, the operating system only sees one.  This would seem to support
the theory that the write only goes to the controller once and it takes
care of the mirroring, but I will defer to your experience.

Thanx,
b.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: dmraid - where is the raid done?

[Roger Heflin]

Brian J. Murrell wrote:
> Somebody had suggested this list for this question:
> 
> I understand that dmraid is just some kind of interface to the
> _software_ raid that is provided on a number of SATA interface cards
> such as the Promise and Adaptec cards.
> 
> Reading http://ata.wiki.kernel.org/index.php/SATA_RAID_FAQ I get the
> message loud and clear that this is not real hardware RAID but rather
> it's done in the BIOS on the card.
> 
> The message there seems to be, if you are going to do it in software
> anyway, why not just skip the IO card RAID and just use MD?
> 
> The one reason I could think of for using the BIOS provided RAID would
> be to reduce the data needing to traverse the PCI bus.  With host-raid
> (i.e. MD) every write to disk needs to actually be written over the PCI
> bus twice, once for each disk, right?
> 
> But do the BIOS RAID cards out there only need the write command and
> data once and they do the work of writing it to the two disks on-board
> the IO card?

At bootup the bios takes care of booting properly for the fakeraid.

After that the OS driver only reads tables that the BIOS put on the disks,
and figures out which disks are to be raided, and the OS does *ALL* of the work 
just like MD.   The only advantage is the bios taking care of booting from 
either disk, without this if a bad disk still partially there (say it just had 
bad blocks on all of the /boot partition) then the OS won't boot (with MD), but 
may boot with bios raid1/dmraid.

If you set raid in the bios, and don't use dmraid in the OS, at some future
boot the bios raid *CAN* determine that the disks are out of sync
and correct this (and depending on your configuration eliminate your data), I
had a customer that did this and was running R0 on the 2 disks, but the
bios was configured R1 and after a year+ of running R0 the machine crashed
in a bad way and the bios determined there were out of sync and asked to
correct it, and they said yes, and it eliminated their data.

So only bootup is done in the bios, once the machine boots the bios is gone
and no longer doing anything.

> 
> IIUC, when you have a BIOS RAID configured, even though there are two
> disks, the operating system only sees one.  This would seem to support
> the theory that the write only goes to the controller once and it takes
> care of the mirroring, but I will defer to your experience.

I don't remember if dmraid takes care of hiding the 2 devices, if it does
the OS is still aware of them and writing to both of them.

> 
> Thanx,
> b.

Re: dmraid - where is the raid done?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

On Mon, 2008-10-27 at 18:29 -0500, Roger Heflin wrote:
> 
> After that the OS driver only reads tables that the BIOS put on the disks,
> and figures out which disks are to be raided, and the OS does *ALL* of the work 
> just like MD.

Ahhhh.  Really.  That is disappointing.  So in reference to the OS doing
all of the work would that be the kernel driver for the particular RAID
card, such as the Promise or Adaptec fakeraid cards?

b.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: dmraid - where is the raid done?

[Roger Heflin]

Brian J. Murrell wrote:
> On Mon, 2008-10-27 at 18:29 -0500, Roger Heflin wrote:
>> After that the OS driver only reads tables that the BIOS put on the disks,
>> and figures out which disks are to be raided, and the OS does *ALL* of the work 
>> just like MD.
> 
> Ahhhh.  Really.  That is disappointing.  So in reference to the OS doing
> all of the work would that be the kernel driver for the particular RAID
> card, such as the Promise or Adaptec fakeraid cards?
> 
> b.
> 

If it is an out-of-kernel driver, then the work is being done in the 
manufacturer's driver.

If it is dmraid it is being done in the dmraid driver.

If there is no underlying hardware to run the raid on then everything has
to be done in the OS.   All of the real raid cards have at least a
really basic microprocesser on the card that takes care of things.
The higher-end cards have a lot more powerful processors.   In both
cases have a small computer on the card is what makes the hardware
raid cards cost quite a bit more than the fakeraid cards.

                    Roger

Re: dmraid - where is the raid done?

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1271 bytes --]

On Mon, 2008-10-27 at 20:12 -0500, Roger Heflin wrote:
> 
> If it is an out-of-kernel driver, then the work is being done in the 
> manufacturer's driver.

I see.

> If it is dmraid it is being done in the dmraid driver.

Ahhh.  So in the above "out of kernel driver" instance that is a case
where dmraid is not handling it, yes?

> If there is no underlying hardware to run the raid on then everything has
> to be done in the OS.

Indeed.  I had just wondered if these "fakeraid" solutions where at
least fanning out the copy (in the case of mirror) themselves, saving
the doubling of PCI bus traffic.

> All of the real raid cards have at least a
> really basic microprocesser on the card that takes care of things.

Yeah, I'm fairly familiar with the concepts of real RAID hardware.  It
was this new fakeraid I was wondering about.

Thanx for the enlightenment though.  The one last value that I had hope
for in the fakeraid arena seems to be invalid so I have to start to
agree with the documentation I've read and really have to wonder the
value of the fakeraid, especially and until the on-disk formats are
standardized.  Not being able to move fakeraid disks from one
manufacturer to another is a real impediment in my mind.

b.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: dmraid - where is the raid done?

[Mario 'BitKoenig' Holbe]

Brian J. Murrell <brian@interlinx.bc.ca> wrote:
> On Mon, 2008-10-27 at 20:12 -0500, Roger Heflin wrote:
>> If it is dmraid it is being done in the dmraid driver.

There is no such thing like a dmraid driver. dmraid uses the
device-mapper driver (which is why it's called dm*).

> Ahhh.  So in the above "out of kernel driver" instance that is a case
> where dmraid is not handling it, yes?

Well, the "fake"raid in-kernel or "out of kernel" (whatever that means)
drivers have been there far more early. dmraid is a quite new concept
just as device-mapper is.


regards
   Mario
-- 
To err is human. To really foul things up requires a computer.

Re: dmraid - where is the raid done?

[Mario 'BitKoenig' Holbe]

Brian J. Murrell <brian@interlinx.bc.ca> wrote:
> I understand that dmraid is just some kind of interface to the
> _software_ raid that is provided on a number of SATA interface cards
> such as the Promise and Adaptec cards.

Well, to be exact, dmraid creates device-mapper tables depending on the
on-disk-layout of various proprietary Software-RAID solutions. It does
neither co-operate with the respective BIOS-subsystems nor does it
manage the respective on-disk metadata to mark disk-failures, change
layouts etc.

> Reading http://ata.wiki.kernel.org/index.php/SATA_RAID_FAQ I get the
> message loud and clear that this is not real hardware RAID but rather
> it's done in the BIOS on the card.

Most of the proprietary Software-RAID solutions out there are
implemented in software only. All the "BIOS" contains there is some very
basic support to initialize and boot off such disk-conglomerates.

> The message there seems to be, if you are going to do it in software
> anyway, why not just skip the IO card RAID and just use MD?

Yes, that's the much cleaner approach. And, for Linux, the more stable
one as well - since dmraid doesn't manage drive-faults etc. anyways as
noted above.

> The one reason I could think of for using the BIOS provided RAID would
> be to reduce the data needing to traverse the PCI bus.  With host-raid
> (i.e. MD) every write to disk needs to actually be written over the PCI
> bus twice, once for each disk, right?

Most of the proprietary Software-RAID solutions out there don't do this,
especially not the low-cost ones that only support RAID0/RAID1/RAID10.
Some solutions that support RAID5 (not all of them, though) provide
on-chip XOR engines - I would suspect those to be able to spread writes
after the PCI(e) bus.

> IIUC, when you have a BIOS RAID configured, even though there are two
> disks, the operating system only sees one.  This would seem to support
> the theory that the write only goes to the controller once and it takes
> care of the mirroring, but I will defer to your experience.

Well, usually the drivers are the ones that hide single devices for ease
and convenience, not the controllers.


regards
   Mario
-- 
Tower: "Say fuelstate." Pilot: "Fuelstate."
Tower: "Say again." Pilot: "Again."
Tower: "Arghl, give me your fuel!" Pilot: "Sorry, need it by myself..."

Re: dmraid - where is the raid done?

[Dan Williams]

On Mon, Oct 27, 2008 at 4:38 PM, Mario 'BitKoenig' Holbe
<Mario.Holbe@tu-ilmenau.de> wrote:
>> The message there seems to be, if you are going to do it in software
>> anyway, why not just skip the IO card RAID and just use MD?
>
> Yes, that's the much cleaner approach. And, for Linux, the more stable
> one as well - since dmraid doesn't manage drive-faults etc. anyways as
> noted above.
>

...or with the upcoming mdadm-3.0[1] you will be able to have the
stability of MD and the boot support for bios raid.  At least for
Intel platforms.

--
Dan

[1] http://marc.info/?l=linux-raid&m=122221593720893&w=2