bootable RAID, number of member disks limitation

bootable RAID, number of member disks limitation

[Chris Murphy]

I'm finding GRUB2 will boot from md RAID 0, 1, 4, 5, 6, with /boot in the array, up to six member devices. At device 7, GRUB2 fails to boot the array, drops to a grub rescue prompt saying the mduuid couldn't be found.

With Btrfs single and RAID 0 profiles, /boot on Btrfs is bootable for up to 4 devices. At the 5th device in a volume, it's no longer bootable by GRUB2, I get a grub rescue prompt. I get variable results with RAID 10 for some reason where often it works with 5 and 6 disks.

The question is if this is expected, as-designed, and if it's stable? I feel like some subjective (certainly objective limits are OK) limit for number of member devices is acceptable, but needs to be known, so that installers can inhibit or warn users from creating unbootable arrays. And then how to communicate this downstream?


Chris Murphy

Re: bootable RAID, number of member disks limitation

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1684 bytes --]

On 03.01.2013 21:28, Chris Murphy wrote:

> I'm finding GRUB2 will boot from md RAID 0, 1, 4, 5, 6, with /boot in the array, up to six member devices. At device 7, GRUB2 fails to boot the array, drops to a grub rescue prompt saying the mduuid couldn't be found.
> 
> With Btrfs single and RAID 0 profiles, /boot on Btrfs is bootable for up to 4 devices. At the 5th device in a volume, it's no longer bootable by GRUB2, I get a grub rescue prompt. I get variable results with RAID 10 for some reason where often it works with 5 and 6 disks.


> 
> The question is if this is expected, as-designed, and if it's stable? I feel like some subjective (certainly objective limits are OK) limit for number of member devices is acceptable, but needs to be known, so that installers can inhibit or warn users from creating unbootable arrays. And then how to communicate this downstream?

GRUB has no limitation on number on devices other than free memory to
hold structures and 2^32 for IDs. If there is any other limit it's a bug
and please file a bug report with the images in question (no need to
install, just use grub-fstest) However BIOS limits the number of disks
that can be accessed. Theoretically there could be up to 128 (or 256 if
you use floppy numbers as well) accessible through BIOS. The convention
is to use only 16 possible IDs for HDD (0x80-0x8f). In practice many
BIOSes are limited to less (8 is usual limit).

> 
> 
> Chris Murphy
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



-- 
Regards
Vladimir 'φ-coder/phcoder' Serbinenko


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 294 bytes --]

Re: bootable RAID, number of member disks limitation

[Chris Murphy]

On Jan 3, 2013, at 3:00 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:

> GRUB has no limitation on number on devices other than free memory to
> hold structures and 2^32 for IDs. If there is any other limit it's a bug
> and please file a bug report with the images in question (no need to
> install, just use grub-fstest) However BIOS limits the number of disks
> that can be accessed. Theoretically there could be up to 128 (or 256 if
> you use floppy numbers as well) accessible through BIOS. The convention
> is to use only 16 possible IDs for HDD (0x80-0x8f). In practice many
> BIOSes are limited to less (8 is usual limit).

I'm totally unclear on how to use grub-fstest.

Should ls at grub rescue report all attached devices?

With single disk btrfs only (no md RAID), at a grub command prompt, ls reports
(hd0) (hd0,msdos1) (hd1) (hd2) (hd3)

Yet there are 11 devices attached. This is VirtualBox, so this very well may be a vbox limitation. I haven't tried it with KVM's SEABIOS yet.

With md RAID only (no Btrfs), ls reports:
(md/root) (hd0) (hd0,msdos1) (hd1) (hd1,msdos1) (hd2) (hd1,msdos2)

Chris Murphy

Re: bootable RAID, number of member disks limitation

[Chris Murphy]

On Jan 4, 2013, at 12:42 AM, Chris Murphy <chris@colorremedies.com> wrote:

> With md RAID only (no Btrfs), ls reports:
> (md/root) (hd0) (hd0,msdos1) (hd1) (hd1,msdos1) (hd2) (hd1,msdos2)

Also with 11 devices attached.

Chris Murphy

Re: bootable RAID, number of member disks limitation

[Jordan Uggla]

On Thu, Jan 3, 2013 at 11:42 PM, Chris Murphy <chris@colorremedies.com> wrote:
>
> On Jan 3, 2013, at 3:00 PM, Vladimir 'φ-coder/phcoder' Serbinenko <phcoder@gmail.com> wrote:
>
>> GRUB has no limitation on number on devices other than free memory to
>> hold structures and 2^32 for IDs. If there is any other limit it's a bug
>> and please file a bug report with the images in question (no need to
>> install, just use grub-fstest) However BIOS limits the number of disks
>> that can be accessed. Theoretically there could be up to 128 (or 256 if
>> you use floppy numbers as well) accessible through BIOS. The convention
>> is to use only 16 possible IDs for HDD (0x80-0x8f). In practice many
>> BIOSes are limited to less (8 is usual limit).
>
> I'm totally unclear on how to use grub-fstest.

I don't have a GNU/Linux system (other than my phone) to test this
with at the moment, but I think "grub-fstest --diskcount=11
/dev/sd{a..k} ls -- -l" should list all 11 disks, and their
partitions, with information about the filesystem each contains. If
(loop0,msdos1) is one of the devices that makes up this btrfs volume
then you should be able to list files on the filesystem with:

grub-fstest --diskcount=11 /dev/sd{a..k} ls '(loop0,msdos1)/'

>
> Should ls at grub rescue report all attached devices?

Yes, if it doesn't then that's a pretty clear indication that your
boot firmware can't properly handle many drives. You can also try
"grub-install --disk-module=native /dev/sdX" to test if grub can
access all of the drives using its own native disk drivers rather than
relying on the boot firmware.

>
> With single disk btrfs only (no md RAID), at a grub command prompt, ls reports
> (hd0) (hd0,msdos1) (hd1) (hd2) (hd3)
>
> Yet there are 11 devices attached. This is VirtualBox, so this very well may be a vbox limitation. I haven't tried it with KVM's SEABIOS yet.


I am surprised to hear that this problem is appearing with VirtualBox
rather than real hardware. If this is a bug in VirtualBox's BIOS then
a bug report should definitely be filed with them.

-- 
Jordan Uggla (Jordan_U on irc.freenode.net)