grub2 and Linux software RAID devices

grub2 and Linux software RAID devices

[Sam Morris]

[-- Attachment #1: Type: text/plain, Size: 2887 bytes --]

Hi there,

A while ago, I tried grub2 on my Debian system, which has my root
filesystem on a Linux software RAID-1 array. I ran into some problems,
and while they were raised here, nothing really came of them. Robert
Millan suggested I post my problem again to see if anything can be done
to fix it.

I think my problems stem from the Promise IDE controller that my second
disk is connected to. It does not support 48-bit LBA addressing, and so
any attempt to read the end of the disk using BIOS calls will fail. Of
course, once an operating system has loaded its own driver for the
controller, the disk can be read correctly.

Here's what the two disks look like:

        Model: Maxtor 6L300R0 (ide)
        Disk /dev/hdb: 300GB
        Sector size (logical/physical): 512B/512B
        Partition Table: msdos
        
        Number  Start   End    Size   Type     File system  Flags
         1      32.3kB  543MB  543MB  primary                    
         2      543MB   300GB  300GB  primary  ext3         raid 

The first partition is swap, the second is the root filesystem.

The first problem is the operation of the grub-probe partition. Debian's
post-install script runs the following command to determine which
modules to include in the generated core.img file:

        grub-probe --target=partmap --device-map=/boot/grub/device.map /boot/grub

Which fails with the error:

        grub-probe: error: Cannot detect partition map for md0

It appears that grub-probe expects to find a partition table inside the
RAID device, when of course, it is really in its containing
device, /dev/hdb.

Debian's post-install script has actually been written to substitute 'pc
gpt' if the partmap probing fails, so I should still be able to boot up
manually. However, I now hit the second problem: the menu that grub
presents has no text! It seems to have an entry, however, as there is a
highlighted line.

The third problem is that if I press enter, or wait for the timeout to
finish, or the screen blanks, and I get the message "Booting '". The
system then freezes and I have to use the hardware reset switch to
continue (ctrl+alt+del does not work). Pressing 'e', or the up or down
keys also freeze the system, without the "Booting '" message.

I am able to get to the console by pressing 'c'. Here I can attempt to
load the raid module, however I run into a fourth problem: I get the
error, "out of disk": presumably because GRUB is trying to read past the
apparent end of the disk connected to my promise controller.

Jeroen Dekkers previously suggested a patch to suppress the 'out of
disk' error, at
<http://www.mail-archive.com/grub-devel@gnu.org/msg02873.html> but no
one ever committed it.

-- 
Sam Morris
http://robots.org.uk/

PGP key id 1024D/5EA01078
3412 EA18 1277 354B 991B  C869 B219 7FDB 5EA0 1078


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: grub2 and Linux software RAID devices

[Robert Millan]

On Mon, Feb 04, 2008 at 09:36:45PM +0000, Sam Morris wrote:
> Hi there,
> 
> A while ago, I tried grub2 on my Debian system, which has my root
> filesystem on a Linux software RAID-1 array. I ran into some problems,
> and while they were raised here, nothing really came of them. Robert
> Millan suggested I post my problem again to see if anything can be done
> to fix it.
> 
> I think my problems stem from the Promise IDE controller that my second
> disk is connected to. It does not support 48-bit LBA addressing, and so
> any attempt to read the end of the disk using BIOS calls will fail. Of
> course, once an operating system has loaded its own driver for the
> controller, the disk can be read correctly.
> 
> Here's what the two disks look like:
> 
>         Model: Maxtor 6L300R0 (ide)
>         Disk /dev/hdb: 300GB
>         Sector size (logical/physical): 512B/512B
>         Partition Table: msdos
>         
>         Number  Start   End    Size   Type     File system  Flags
>          1      32.3kB  543MB  543MB  primary                    
>          2      543MB   300GB  300GB  primary  ext3         raid 
> 
> The first partition is swap, the second is the root filesystem.
> 
> The first problem is the operation of the grub-probe partition. Debian's
> post-install script runs the following command to determine which
> modules to include in the generated core.img file:
> 
>         grub-probe --target=partmap --device-map=/boot/grub/device.map /boot/grub
> 
> Which fails with the error:
> 
>         grub-probe: error: Cannot detect partition map for md0
> 
> It appears that grub-probe expects to find a partition table inside the
> RAID device, when of course, it is really in its containing
> device, /dev/hdb.

This is a known problem, and I roughly have a solution in mind, but I haven't
been able to reproduce it.  When I try to install Debian with /boot inside an
LVM, the installer hangs.  This option doesn't seem to be supported at all.

Furthermore, how do you boot that system with GRUB Legacy?

If you give me some details on how to reproduce the scheme in which /boot is
behind a lvm/raid abstraction, I could try to get this fixed.

> Debian's post-install script has actually been written to substitute 'pc
> gpt' if the partmap probing fails,

Actually, this was reverted a while ago.

> manually. However, I now hit the second problem: the menu that grub
> presents has no text! It seems to have an entry, however, as there is a
> highlighted line.

Which version did you try?  Is it more recent than 1.96 ?  We fixed bugs
producing this result recently.

> The third problem is that if I press enter, or wait for the timeout to
> finish, or the screen blanks, and I get the message "Booting '". The
> system then freezes and I have to use the hardware reset switch to
> continue (ctrl+alt+del does not work). Pressing 'e', or the up or down
> keys also freeze the system, without the "Booting '" message.

Where does GRUB get its grub.cfg from?  What are its contents? (from the POV
of GRUB;  use 'cat' to determine).

> Jeroen Dekkers previously suggested a patch to suppress the 'out of
> disk' error, at
> <http://www.mail-archive.com/grub-devel@gnu.org/msg02873.html> but no
> one ever committed it.

Ah, I see.  The patch looks correct to me;  only the description you gave
before isn't.

Any objection if I check that in ?  ChangeLog:

	Patch from Jeroen Dekkers.
	* disk/raid.c (grub_raid_scan_device): Reset `grub_errno' on disk
	failure, since succesfuly reading all array members might not be
	required.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call… if you are unable to speak?
(as seen on /.)

Re: grub2 and Linux software RAID devices

[Sam Morris]

On Mon, 04 Feb 2008 23:43:38 +0100, Robert Millan wrote:
> On Mon, Feb 04, 2008 at 09:36:45PM +0000, Sam Morris wrote:
>> Hi there,
>> 
>> A while ago, I tried grub2 on my Debian system, which has my root
>> filesystem on a Linux software RAID-1 array. I ran into some problems,
>> and while they were raised here, nothing really came of them. Robert
>> Millan suggested I post my problem again to see if anything can be done
>> to fix it.
>> 
>> I think my problems stem from the Promise IDE controller that my second
>> disk is connected to. It does not support 48-bit LBA addressing, and so
>> any attempt to read the end of the disk using BIOS calls will fail. Of
>> course, once an operating system has loaded its own driver for the
>> controller, the disk can be read correctly.
>> 
>> Here's what the two disks look like:
>> 
>>         Model: Maxtor 6L300R0 (ide)
>>         Disk /dev/hdb: 300GB
>>         Sector size (logical/physical): 512B/512B Partition Table:
>>         msdos
>>         
>>         Number  Start   End    Size   Type     File system  Flags
>>          1      32.3kB  543MB  543MB  primary
>>          2      543MB   300GB  300GB  primary  ext3         raid
>> 
>> The first partition is swap, the second is the root filesystem.
>> 
>> The first problem is the operation of the grub-probe partition.
>> Debian's post-install script runs the following command to determine
>> which modules to include in the generated core.img file:
>> 
>>         grub-probe --target=partmap --device-map=/boot/grub/device.map
>>         /boot/grub
>> 
>> Which fails with the error:
>> 
>>         grub-probe: error: Cannot detect partition map for md0
>> 
>> It appears that grub-probe expects to find a partition table inside the
>> RAID device, when of course, it is really in its containing device,
>> /dev/hdb.
> 
> This is a known problem, and I roughly have a solution in mind, but I
> haven't been able to reproduce it.  When I try to install Debian with
> /boot inside an LVM, the installer hangs.  This option doesn't seem to
> be supported at all.

Ouch... I have never tried /boot on LVM myself (since grub legacy can't 
handle it). However /boot on RAID works fine, I suggest you try that 
instead.

> 
> Furthermore, how do you boot that system with GRUB Legacy?

As far as grub1 is concerned, /dev/hdb2 is a normal partition containing 
an ext3 filesystem.

> If you give me some details on how to reproduce the scheme in which
> /boot is behind a lvm/raid abstraction, I could try to get this fixed.

It's pretty simple, assuming you are using d-i. When partitioning, 
configure two disks with identical partition layouts (a single partition 
on each is sufficient). Then, tell partman that you want to use them as 
'physical volumes for RAID'. A new option should appear, 'configure 
RAID' (or something similar). Here you can create a RAID1 array using 
both the partitions, which you can use as an ext3 filesystem, mounted 
at /.

>> Debian's post-install script has actually been written to substitute
>> 'pc gpt' if the partmap probing fails,
> 
> Actually, this was reverted a while ago.
> 
>> manually. However, I now hit the second problem: the menu that grub
>> presents has no text! It seems to have an entry, however, as there is a
>> highlighted line.
> 
> Which version did you try?  Is it more recent than 1.96 ?  We fixed bugs
> producing this result recently.

Damn, I installed the version from testing by mistake. I will try again 
with version 1.96+20080203-1.

>> Jeroen Dekkers previously suggested a patch to suppress the 'out of
>> disk' error, at
>> <http://www.mail-archive.com/grub-devel@gnu.org/msg02873.html> but no
>> one ever committed it.
>
> Ah, I see.  The patch looks correct to me;  only the description you
> gave before isn't.

Hmm, I don't understand the difference between your changelog entry and 
my description, but ok. :)

-- 
Sam Morris
http://robots.org.uk/
 
PGP key id 1024D/5EA01078
3412 EA18 1277 354B 991B  C869 B219 7FDB 5EA0 1078

Re: grub2 and Linux software RAID devices

[Sam Morris]

On Mon, 04 Feb 2008 23:43:38 +0100, Robert Millan wrote:
>> Debian's post-install script has actually been written to substitute
>> 'pc gpt' if the partmap probing fails,
> 
> Actually, this was reverted a while ago.
> 
>> manually. However, I now hit the second problem: the menu that grub
>> presents has no text! It seems to have an entry, however, as there is a
>> highlighted line.
> 
> Which version did you try?  Is it more recent than 1.96 ?  We fixed bugs
> producing this result recently.

This now works fine (and looks nice and pretty!) with version 1.96
+20080203-1. :)

>> The third problem is that if I press enter, or wait for the timeout to
>> finish, or the screen blanks, and I get the message "Booting '". The
>> system then freezes and I have to use the hardware reset switch to
>> continue (ctrl+alt+del does not work). Pressing 'e', or the up or down
>> keys also freeze the system, without the "Booting '" message.
> 
> Where does GRUB get its grub.cfg from?  What are its contents? (from the
> POV of GRUB;  use 'cat' to determine).

Version 1.96+20080203-1 behaves differently; I am simply dropped to the 
rescue shell, and I can't load any modules (because root is set to 
(hd1,2) which does not show up when I run 'ls'). I guessed that this is 
because 'pc' is not in the list of modules included when grub-install 
creates core.img.

If I add 'pc' to core.img then things are much healthier: I am still 
dropped into the rescue shell, but I can load the 'normal' module, issue 
the 'normal' command and then I get the (pretty!) menu and am able to 
boot normally. I'm still not sure why I'm dropped into the rescue shell 
in the first place, however. There's a flash of text before the prompt 
appears, but the screen is cleared much to fast to read any of it.

ISTR that when I tried Jeroen Dekkers' patch before, I was no longer 
booted into the rescue shell; but I can't remember (and I don't 
understand why, if I was kicked into the rescue shell becuase grub tried 
to read past the 'end' of my disk, why I can immediatly issue the 
'normal' command to go to the menu and have everything working). Anyway, 
I'll wait to see how things improve after a new grub-pc is uploaded. 
Thanks for your help & advice so far!

-- 
Sam Morris
http://robots.org.uk/
 
PGP key id 1024D/5EA01078
3412 EA18 1277 354B 991B  C869 B219 7FDB 5EA0 1078

Re: grub2 and Linux software RAID devices

[Robert Millan]

On Tue, Feb 05, 2008 at 01:32:46AM +0000, Sam Morris wrote:
> 
> Version 1.96+20080203-1 behaves differently; I am simply dropped to the 
> rescue shell, and I can't load any modules (because root is set to 
> (hd1,2) which does not show up when I run 'ls'). I guessed that this is 
> because 'pc' is not in the list of modules included when grub-install 
> creates core.img.

Did you make sure 'pc' is loaded before 'raid' ?  This was required, at least
for LVM.

> If I add 'pc' to core.img then things are much healthier: I am still 
> dropped into the rescue shell, but I can load the 'normal' module, issue 
> the 'normal' command and then I get the (pretty!) menu and am able to 
> boot normally. I'm still not sure why I'm dropped into the rescue shell 
> in the first place, however. There's a flash of text before the prompt 
> appears, but the screen is cleared much to fast to read any of it.

Please can you take a picture of what you see right after the text is
cleared?

> ISTR that when I tried Jeroen Dekkers' patch before, I was no longer 
> booted into the rescue shell; but I can't remember (and I don't 
> understand why, if I was kicked into the rescue shell becuase grub tried 
> to read past the 'end' of my disk, why I can immediatly issue the 
> 'normal' command to go to the menu and have everything working).

Sounds strange, yes.  I suggest we try to fix the clearing error that
prevents you from reading that text, then see if that helps us make progress.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call… if you are unable to speak?
(as seen on /.)

Re: grub2 and Linux software RAID devices

[Robert Millan]

On Tue, Feb 05, 2008 at 12:44:38AM +0000, Sam Morris wrote:
> 
> Ouch... I have never tried /boot on LVM myself (since grub legacy can't 
> handle it). However /boot on RAID works fine, I suggest you try that 
> instead.

Ah, I see.  What remains puzzling is how do other people manage to boot from
/boot on LVM (we had reports about that).

> It's pretty simple, assuming you are using d-i. When partitioning, 
> configure two disks with identical partition layouts (a single partition 
> on each is sufficient). Then, tell partman that you want to use them as 
> 'physical volumes for RAID'. A new option should appear, 'configure 
> RAID' (or something similar). Here you can create a RAID1 array using 
> both the partitions, which you can use as an ext3 filesystem, mounted 
> at /.

Thank you.  I'll try this as soon I get a free minute.

> >> Jeroen Dekkers previously suggested a patch to suppress the 'out of
> >> disk' error, at
> >> <http://www.mail-archive.com/grub-devel@gnu.org/msg02873.html> but no
> >> one ever committed it.
> >
> > Ah, I see.  The patch looks correct to me;  only the description you
> > gave before isn't.
> 
> Hmm, I don't understand the difference between your changelog entry and 
> my description, but ok. :)

You said unsetting grub_errno ensured the iteration is not stopped, but
grub_errno has no effect on GRUB iterators, only the return value does.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call… if you are unable to speak?
(as seen on /.)

Re: grub2 and Linux software RAID devices

[Sam Morris]

On Tue, 05 Feb 2008 10:35:19 +0100, Robert Millan wrote:

> On Tue, Feb 05, 2008 at 01:32:46AM +0000, Sam Morris wrote:
>> 
>> Version 1.96+20080203-1 behaves differently; I am simply dropped to the
>> rescue shell, and I can't load any modules (because root is set to
>> (hd1,2) which does not show up when I run 'ls'). I guessed that this is
>> because 'pc' is not in the list of modules included when grub-install
>> creates core.img.
> 
> Did you make sure 'pc' is loaded before 'raid' ?  This was required, at
> least for LVM.

Yes, the exact command I ran was: 'grub-mkimage --output=/boot/grub/
core.img --prefix=/boot/grub ext2 pc biosdisk raid _chain'.

>> If I add 'pc' to core.img then things are much healthier: I am still
>> dropped into the rescue shell, but I can load the 'normal' module,
>> issue the 'normal' command and then I get the (pretty!) menu and am
>> able to boot normally. I'm still not sure why I'm dropped into the
>> rescue shell in the first place, however. There's a flash of text
>> before the prompt appears, but the screen is cleared much to fast to
>> read any of it.
> 
> Please can you take a picture of what you see right after the text is
> cleared?

Ok, I discovered that I can use the Pause key to read what's on the 
screen. Unfortunately, it's not very useful... it's just the text printed 
by grub (legacy) as it loads the core.img file!

 Booting 'Chainload into GRUB 2'
 root (hd1,1)
  Filesystem type is ext2fs, partition type 0xfd
 kernel /boot/grub/core.img
   [Multiboot-kludge, loadaddr=0x100000, text-and-data=0x7676, bss=0x0,  
entry=0x1002903]
 savedefault

Then the screen blanks, and the next thing printed is:

 Welcome to GRUB!

 Entering into rescue mode...
 grub rescue>

Whereupon I can enter 'insmod normal' and then 'normal' to get to the 
regular boot menu.

-- 
Sam Morris
http://robots.org.uk/
 
PGP key id 1024D/5EA01078
3412 EA18 1277 354B 991B  C869 B219 7FDB 5EA0 1078

Re: grub2 and Linux software RAID devices

[Sam Morris]

On Wed, 06 Feb 2008 00:49:24 +0000, Sam Morris wrote:

> Then the screen blanks, and the next thing printed is:
> 
>  Welcome to GRUB!
> 
>  Entering into rescue mode...
>  grub rescue>
> 
> Whereupon I can enter 'insmod normal' and then 'normal' to get to the
> regular boot menu.

I modified the grub_load_normal_mode function in kern/main.c to print out 
the value of grub_errno after calling 'grub_dl_load ("normal")'. The 
result was 11, which corresponds to GRUB_ERR_OUT_OF_RANGE.

I think it would be a good idea to modify the code to print out a 
description of the error at this point, or at least to display the error 
number, so that users know why they are being booted into the rescue 
shell.

-- 
Sam Morris
http://robots.org.uk/
 
PGP key id 1024D/5EA01078
3412 EA18 1277 354B 991B  C869 B219 7FDB 5EA0 1078

Re: grub2 and Linux software RAID devices

[Yoshinori K. Okuji]

On Thursday 07 February 2008 00:19, Sam Morris wrote:
> On Wed, 06 Feb 2008 00:49:24 +0000, Sam Morris wrote:
> > Then the screen blanks, and the next thing printed is:
> >
> >  Welcome to GRUB!
> >
> >  Entering into rescue mode...
> >  grub rescue>
> >
> > Whereupon I can enter 'insmod normal' and then 'normal' to get to the
> > regular boot menu.
>
> I modified the grub_load_normal_mode function in kern/main.c to print out
> the value of grub_errno after calling 'grub_dl_load ("normal")'. The
> result was 11, which corresponds to GRUB_ERR_OUT_OF_RANGE.
>
> I think it would be a good idea to modify the code to print out a
> description of the error at this point, or at least to display the error
> number, so that users know why they are being booted into the rescue
> shell.

Clever idea. I think it is crucial to add this kind of code more and more to 
GRUB, because the user often doesn't want to patch and recompile and 
reinstall, again and again.

Okuji

Re: grub2 and Linux software RAID devices

[Robert Millan]

On Mon, Feb 04, 2008 at 11:43:38PM +0100, Robert Millan wrote:
> 
> Any objection if I check that in ?  ChangeLog:
> 
> 	Patch from Jeroen Dekkers.
> 	* disk/raid.c (grub_raid_scan_device): Reset `grub_errno' on disk
> 	failure, since succesfuly reading all array members might not be
> 	required.

Ok, in it goes then.

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call… if you are unable to speak?
(as seen on /.)

Re: grub2 and Linux software RAID devices

[Jan Nieuwenhuizen]

Robert Millan:

> Ah, I see.  What remains puzzling is how do other people manage to boot from
> /boot on LVM (we had reports about that).

Ah, that's possibly what your request is about that I still have
standing out.  I don't think I've ever installed Debian using d-i,
you've got much more uptime when doing it from a running system.

In this case I simply started from my plain Debian install (actually
Ubuntu).  Add another disk, create lvm on that, copy whole system over
to lvm.  Boot new system with / on lvm using old /boot on plain disk.
Some playing with lvm, mkinitrd and grub2 is required here.  

Then play with grub2's /boot device and ordering of modules until it
will also boot from the new copied /boot on lvm.  Trash old plain disk.

Greetings,
Jan.

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond - The music typesetter
http://www.xs4all.nl/~jantien       | http://www.lilypond.org

Re: grub2 and Linux software RAID devices

[Robert Millan]

On Thu, Feb 07, 2008 at 09:35:32PM +0100, Jan Nieuwenhuizen wrote:
> Robert Millan:
> 
> > Ah, I see.  What remains puzzling is how do other people manage to boot from
> > /boot on LVM (we had reports about that).
> 
> Ah, that's possibly what your request is about that I still have
> standing out.  I don't think I've ever installed Debian using d-i,
> you've got much more uptime when doing it from a running system.
> 
> In this case I simply started from my plain Debian install (actually
> Ubuntu).  Add another disk, create lvm on that, copy whole system over
> to lvm.  Boot new system with / on lvm using old /boot on plain disk.
> Some playing with lvm, mkinitrd and grub2 is required here.  
> 
> Then play with grub2's /boot device and ordering of modules until it
> will also boot from the new copied /boot on lvm.  Trash old plain disk.

I don't think I could handle this.  My experience with lvm as a user is
minimal.

If you can provide a disk image that I can use to reproduce the problem in
qemu, that would help me look into the problem(s).

-- 
Robert Millan

<GPLv2> I know my rights; I want my phone call!
<DRM> What use is a phone call… if you are unable to speak?
(as seen on /.)