Linux RAID metadata format docs?

Linux RAID metadata format docs?

[Jeff Garzik]

Do any docs exist, that describe the on-disk md format?

I'm talking with a vendor that would like to support booting off of 
"Linux RAID" (as they call it), from their card's BIOS, which implies
the vendor would need to add support for the md format to their BIOS.

	Jeff

Re: Linux RAID metadata format docs?

[Matti Aarnio]

On Fri, Nov 14, 2003 at 01:50:13PM -0500, Jeff Garzik wrote:
> Do any docs exist, that describe the on-disk md format?

Not as such.   An  H-file  isn't a document in itself...

> I'm talking with a vendor that would like to support booting off of 
> "Linux RAID" (as they call it), from their card's BIOS, which implies
> the vendor would need to add support for the md format to their BIOS.
> 
> 	Jeff

Problem there is the multitude of partition schemes, and that RAID
superblocks are inside the partitions...

Nevertheless I would love to be able to boot from _any_ of disks
in a bootable RAID1 diskset.  I have just experimented with Highpoint's
RocketRaid 1520, which almost can do that.  It fails if the primary
disk is offline, but succeeds if secondary is out.
I also liked their superblock location - very early in disk.
Negative thing in the HPT BIOS is, that if it detects a fault, it
wants user input instead of just trying starting disks a bit more 
persistently, and continuing _soon_ the boot in degraded RAID1 mode
without any user input.

After all, Sun has been able to do that for a very long time,
it should be possible in PC hardware, too.
(In Suns we have to define a set of boot devices into prom level,
 and there all devices are different, and retain their identity
 forever independent of plugging new interface cards into system,
 but that is another story...)

/Matti Aarnio

Re: Linux RAID metadata format docs?

[Jeff Garzik]

On Fri, Nov 14, 2003 at 09:31:14PM +0200, Matti Aarnio wrote:
> On Fri, Nov 14, 2003 at 01:50:13PM -0500, Jeff Garzik wrote:
> > Do any docs exist, that describe the on-disk md format?
> 
> Not as such.   An  H-file  isn't a document in itself...
> 
> > I'm talking with a vendor that would like to support booting off of 
> > "Linux RAID" (as they call it), from their card's BIOS, which implies
> > the vendor would need to add support for the md format to their BIOS.
> > 
> > 	Jeff
> 
> Problem there is the multitude of partition schemes, and that RAID
> superblocks are inside the partitions...

Yep.  Thought I consider it fairly pointless to support anything but
ms-dos partitions and EFI partitions.  Given limited development
resources, I don't see a huge need for (for example) BSD disklabel
support.


> Nevertheless I would love to be able to boot from _any_ of disks
> in a bootable RAID1 diskset.  I have just experimented with Highpoint's
> RocketRaid 1520, which almost can do that.  It fails if the primary
> disk is offline, but succeeds if secondary is out.
> I also liked their superblock location - very early in disk.
> Negative thing in the HPT BIOS is, that if it detects a fault, it
> wants user input instead of just trying starting disks a bit more 
> persistently, and continuing _soon_ the boot in degraded RAID1 mode
> without any user input.

Yeah, the main point is to be able to root in degraded mode, as well
as regular mode.

	Jeff

Re: Linux RAID metadata format docs?

[Hendrik Visage]

On Fri, Nov 14, 2003 at 09:31:14PM +0200, Matti Aarnio wrote:
> Nevertheless I would love to be able to boot from _any_ of disks
> in a bootable RAID1 diskset.  I have just experimented with Highpoint's
> RocketRaid 1520, which almost can do that.  It fails if the primary
<snip> 
> After all, Sun has been able to do that for a very long time,
> it should be possible in PC hardware, too.

Nope, Sun's booting is achieved in the OBP (Open Boot Prom)
which is arguably more advanced than the current PC BIOSes.

> (In Suns we have to define a set of boot devices into prom level,

Which is specified based on the PCI/SBUS slot the controller is
attached to. One Day(TM) we'll be able to see the PeeCee world 
drop it's ISA slot independence but IRQ/IOport uniqueness, not to
mention it's real mode DOS compatible 0x80, 0x81 harddisk ordering... :(

>  and there all devices are different, and retain their identity
>  forever independent of plugging new interface cards into system,
>  but that is another story...)

That's only a function of the order of entries in /etc/path_to_inst, which is
achievable (to an extend) in linux with modules.conf trickery.

Hendrik

Re: Linux RAID metadata format docs?

[Neil Brown]

On Friday November 14, jgarzik@pobox.com wrote:
> Do any docs exist, that describe the on-disk md format?

Not to my knowledge, beyond source code.

> 
> I'm talking with a vendor that would like to support booting off of 
> "Linux RAID" (as they call it), from their card's BIOS, which implies
> the vendor would need to add support for the md format to their BIOS.

I don't see that knowledge of the RAID format is needed by the bios.

It might be needed by the boot sector, and certainly by the next stage
(whether that be the kernel itself, or grub or whatever).

Based on my limited understanding, the bios should only need:
  1/ to be able to try subsequent drives if the first fails,
  2/ be able to tell the boot sector which drive was used
  3/ to allow the boot sector to indicate failure
 and maybe
  4/ be able to have a dynamically determined boot order
   (so that when one drive fails, it gets put to the bottom of the 
    list).

Clearly 1,2,3 don't need the bios to know anything about RAID layout.  4
could use raid layout information, and an easy interface for the
kernel to tell it when things have changed would be a lot easier and
more general.

I think the simplest approach would be:

 Bios attempts each potential boot device in some order, loading the
 boot sector and telling the boot sector that the device is "0x80".
 If the boot sector fails (e.g. bad code, such as all zeros), it tries
 the next.

 The boot sector tries to load a kernel or second-stage off "0x80" as
 normal.   The kernel or second-stage understands raid and makes sure
 the correct root is assembled.

 Some daemon monitors drives that fail, and attempts to over-write
 sector 0 of any such drive that is a boot-candidate.

This wouldn't quite do the right thing if:
  The "first" drive goes off line for some reason.
  A new kernel/second-stage is installed on remaining drive(s).
  machine is rebooted and "first" drive comes back online during the
  boot.

The "first" drive will look OK, but will not have a current kernel on
it.

Does any of this make sense?

Were you thinking of some RAID level other than RAID1?

NeilBrown

Re: Linux RAID metadata format docs?

[Jeff Garzik]

Neil Brown wrote:
> On Friday November 14, jgarzik@pobox.com wrote:
> 
>>Do any docs exist, that describe the on-disk md format?
> 
> 
> Not to my knowledge, beyond source code.

Thanks.


> I don't see that knowledge of the RAID format is needed by the bios.
[...]
>  The boot sector tries to load a kernel or second-stage off "0x80" as
>  normal.   The kernel or second-stage understands raid and makes sure
>  the correct root is assembled.
> 
>  Some daemon monitors drives that fail, and attempts to over-write
>  sector 0 of any such drive that is a boot-candidate.
> 
> This wouldn't quite do the right thing if:
>   The "first" drive goes off line for some reason.
>   A new kernel/second-stage is installed on remaining drive(s).
>   machine is rebooted and "first" drive comes back online during the
>   boot.
> 
> The "first" drive will look OK, but will not have a current kernel on
> it.
> 
> Does any of this make sense?
> 
> Were you thinking of some RAID level other than RAID1?


All RAID levels, 1 and 5 particularly.  To me, it boils down to a 
choice:  support in the bootloader, and/or support in the BIOS.  If the 
BIOS vendor is willing to support the _Linux_ md format, it seems to me 
that's a more reliable place to handle booting and failover in RAID1 or 
5.  OTOH, while coordinating multiple boot loaders (since one must 
predict behavior of lilo/grub in all raid1/5 failure scenarios) seems 
much more difficult, but ultimate not specific to one vendor.

By tradition the BIOS/firmware is the software that loads the 
bootloader.  It feels like the right thing to me, for that to continue 
even in the face of RAID arrays.  Users of "fake RAID" (i.e. the vendors 
market it as RAID, even though it's really software RAID) are used to 
their RAID arrays being accessible as a BIOS device, as weird as that 
may be :)

	Jeff