grub fails boot after update

grub fails boot after update

[Jan Engelhardt]

>From Novell Bugzilla, I gather that XFS has a serious problem with grub. 
Since I'd like to keep XFS for the time being, is there any way to fix 
this issue, or make dead sure that a given file is on disk?
Like ioctl(fd, XFS_FLUSH_I_MEAN_IT)?


---------- Forwarded message ----------
https://bugzilla.novell.com/show_bug.cgi?id=223773

--- Comment #39  2008-07-01 08:44:49 MDT ---
I agree with comment #37: XFS really does suck, especially when it comes to
booting Linux on a PC. Fortunately we do not support it any more for new
installations, an ext2 /boot partition is highly recommended.

The problem is that with XFS, sync(2) returns, but the data isn't synced.
The first time yast calls grub install, grub does not find the new stage1.5,
because it is not on the disk yet, despite a successful sync; thus it modifies
stage2 to do the job. On the second invocation, stage1.5 is found and
installed, but stage2 already is modified.

So once again this isn't a grub bug, but an XFS bug with FS semantics.

Re: grub fails boot after update

[Christoph Hellwig]

sync works perfectly fine on xfs.  Grub just doesn't understand what
sync means, and because of that it's buggy on all filesystems, just
with less a chance on others.   The fix is pretty simple and that is
stopping to try to access the filesystem with it's own driver through
the block device node.

Re: grub fails boot after update

[Eric Sandeen]

Christoph Hellwig wrote:
> sync works perfectly fine on xfs.  Grub just doesn't understand what
> sync means, and because of that it's buggy on all filesystems, just
> with less a chance on others.   The fix is pretty simple and that is
> stopping to try to access the filesystem with it's own driver through
> the block device node.

Aye.

And from the bug:

>> I agree with comment #37: XFS really does suck, especially when it comes to
>> booting Linux on a PC. 

Now that's just inflammatory.  :)

>> Fortunately we do not support it any more for new
>> installations, an ext2 /boot partition is highly recommended.

I didn't read the details of the bug but the conclusion is right though
- grub is busted, just use ext3 on /boot to work around it.

>> The problem is that with XFS, sync(2) returns, but the data isn't synced.
>> The first time yast calls grub install, grub does not find the new stage1.5,
>> because it is not on the disk yet, despite a successful sync; thus it modifies
>> stage2 to do the job. On the second invocation, stage1.5 is found and
>> installed, but stage2 already is modified.
>> 
>> So once again this isn't a grub bug, but an XFS bug with FS semantics.

No, that's wrong as hch said.

(FWIW the issue is that xfs data is safe on disk, metadata is safe in
the log, but grub tries to read the fs directly as if it were frozen and
expects to find metadata at the final spot on disk, .)

Syncing a live filesystem and then thinking you can go read (or worse,
write!) directly from (to) disk is a busted notion in many ways.   It's
the same problem as thinking you can do "sync" and then take a
block-based snapshot.  There's a reason DM for example freezes before this.

There was a bug w/ grub vs. ext3 causing corruption for the exact same
sorts of reasons; it's just a little harder to hit.

This really is grub that is busted, but I'd still just suggest using
ext3 to (mostly) work around the breakage for the foreseeable future.

The other option is to teach grub to always do its io via the filesystem
not the block device while the fs is mounted (IIRC there are various &
sundry non-intuitive commands which actually nudge grub towards or away
from this desired behavior... --with-stage2=/path is one I think,
skipping the "verification" phase (i.e. trying to read the block dev
while mounted) is another)

BTW the patch to "wait 10s for the fs to settle" is pure bunk and will
not definitively fix the problem.  It's not even worth committing IMHO.

-Eric

Re: grub fails boot after update

[Jason White]

On Wed, Jul 02, 2008 at 10:54:38PM -0500, Eric Sandeen wrote:
 
> This really is grub that is busted, but I'd still just suggest using
> ext3 to (mostly) work around the breakage for the foreseeable future.
> 
> The other option is to teach grub to always do its io via the filesystem
> not the block device while the fs is mounted (IIRC there are various &
> sundry non-intuitive commands which actually nudge grub towards or away
> from this desired behavior... --with-stage2=/path is one I think,
> skipping the "verification" phase (i.e. trying to read the block dev
> while mounted) is another)

Does grub 2 (still in development when last I checked) improve on this
situation?

I managed to get Grub 1 installed on machines with XFS root file systems by
running the install from within the grub "shell" environment rather than using
grub-install. Maybe this skips the checks that attempt to read the block
device directly. I also recall that grub-install failed.

Re: grub fails boot after update

[Eric Sandeen]

Jason White wrote:
> On Wed, Jul 02, 2008 at 10:54:38PM -0500, Eric Sandeen wrote:
>  
>> This really is grub that is busted, but I'd still just suggest using
>> ext3 to (mostly) work around the breakage for the foreseeable future.
>>
>> The other option is to teach grub to always do its io via the filesystem
>> not the block device while the fs is mounted (IIRC there are various &
>> sundry non-intuitive commands which actually nudge grub towards or away
>> from this desired behavior... --with-stage2=/path is one I think,
>> skipping the "verification" phase (i.e. trying to read the block dev
>> while mounted) is another)
> 
> Does grub 2 (still in development when last I checked) improve on this
> situation?

No idea.  Does anyone  use grub2?  My impression is that each distro is
currently just maintaining a fork of grub(1).  If grub2 is using more
than "sync and hope" I'd be pleasantly surprised.  :)

> I managed to get Grub 1 installed on machines with XFS root file systems by
> running the install from within the grub "shell" environment rather than using
> grub-install. Maybe this skips the checks that attempt to read the block
> device directly. I also recall that grub-install failed.

Yep, this might be some of the "magic" I talked about w.r.t. behavior of
different grub incantations.

-Eric

Re: grub fails boot after update

[Martin Steigerwald]

[-- Attachment #1: Type: text/plain, Size: 1526 bytes --]


Hi!

Am Donnerstag 03 Juli 2008 schrieb Jason White:
> On Wed, Jul 02, 2008 at 10:54:38PM -0500, Eric Sandeen wrote:
> > This really is grub that is busted, but I'd still just suggest using
> > ext3 to (mostly) work around the breakage for the foreseeable future.
> >
> > The other option is to teach grub to always do its io via the
> > filesystem not the block device while the fs is mounted (IIRC there
> > are various & sundry non-intuitive commands which actually nudge grub
> > towards or away from this desired behavior... --with-stage2=/path is
> > one I think, skipping the "verification" phase (i.e. trying to read
> > the block dev while mounted) is another)
>
> Does grub 2 (still in development when last I checked) improve on this
> situation?
>
> I managed to get Grub 1 installed on machines with XFS root file
> systems by running the install from within the grub "shell" environment
> rather than using grub-install. Maybe this skips the checks that
> attempt to read the block device directly. I also recall that
> grub-install failed.

I use GRUB 1 on XFS ever since. But I also used the GRUB shell directly to 
write it. grub-install doesn't work reliable.

I even use the functionality to save the last booted menu entry 
into /boot/grub/default without a problem.

I agree, reading or writing a mounted filesystem directly is not a good 
idea.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: grub fails boot after update

[Martin Steigerwald]

Hi!

Am Donnerstag 03 Juli 2008 schrieb Jason White:
> On Wed, Jul 02, 2008 at 10:54:38PM -0500, Eric Sandeen wrote:
> > This really is grub that is busted, but I'd still just suggest using
> > ext3 to (mostly) work around the breakage for the foreseeable future.
> >
> > The other option is to teach grub to always do its io via the
> > filesystem not the block device while the fs is mounted (IIRC there
> > are various & sundry non-intuitive commands which actually nudge grub
> > towards or away from this desired behavior... --with-stage2=/path is
> > one I think, skipping the "verification" phase (i.e. trying to read
> > the block dev while mounted) is another)
>
> Does grub 2 (still in development when last I checked) improve on this
> situation?
>
> I managed to get Grub 1 installed on machines with XFS root file
> systems by running the install from within the grub "shell" environment
> rather than using grub-install. Maybe this skips the checks that
> attempt to read the block device directly. I also recall that
> grub-install failed.

I use GRUB 1 on XFS ever since. But I also used the GRUB shell directly to 
write it. grub-install doesn't work reliable.

I even use the functionality to save the last booted menu entry 
into /boot/grub/default without a problem.

I agree, reading or writing a mounted filesystem directly is not a good 
idea.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7