EXT2 corruption -- After running 2.5.46, my root partition cannot be mounted by older kernels

EXT2 corruption -- After running 2.5.46, my root partition cannot be mounted by older kernels

[Miles Lane]

Hi,

I am not sure how to diagnose the problem.  When I try to boot
the latest RH 8.0 kernel, I am informed that some unsupported
extensions are present on /dev/hda12 at boot time.  The partition
fails to mount and the boot process halts.

I have no trouble booting with 2.5.46.

    Miles

Re: EXT2 corruption -- After running 2.5.46, my root partition cannotbe mounted by older kernels

[Andrew Morton]

Miles Lane wrote:
> 
> Hi,
> 
> I am not sure how to diagnose the problem.  When I try to boot
> the latest RH 8.0 kernel, I am informed that some unsupported
> extensions are present on /dev/hda12 at boot time.  The partition
> fails to mount and the boot process halts.
> 
> I have no trouble booting with 2.5.46.
> 

Please send the exact diagnostic message.

Re: EXT2 corruption -- After running 2.5.46, my root partition cannot be mounted by older kernels

[Theodore Ts'o]

On Wed, Nov 06, 2002 at 06:34:45PM -0800, Miles Lane wrote:
>
> I am not sure how to diagnose the problem.  When I try to boot
> the latest RH 8.0 kernel, I am informed that some unsupported
> extensions are present on /dev/hda12 at boot time.  The partition
> fails to mount and the boot process halts.
> 
> I have no trouble booting with 2.5.46.

Send the output of dumpe2fs -h to be sure, but it's almost certainly
one of two things: 

1) You didn't unmount the filesystem cleanly when you previously
booted a kernel with ext3 compiled in , and your 2.4 kernel has ext3
as a module, but you either don't have an initrd or the initrd doesn't
have the ext3 module in it.

2) You managed to enable a new ext3 feature, such as htree, or
extended attributes which was supported in the newer kernel, but not
in the 2.4 kernel.

(1) tends to be the most likely cause, given the confused users who
ask these sorts of questions on th ext3-users mailing list.  As a
result, I've developed a very strong distaste for initrd, and
generally strongly encourage people to compile ext3 and whatever
device drivers you require into the kernel, and to not try to use
initrd.  initrd turns out to be a confusing stumbling block for far
too many users.

						- Ted

Re: EXT2 corruption -- After running 2.5.46, my root partition cannot be mounted by older kernels

[Miles Lane]

Ted Tso wrote:

> Send the output of dumpe2fs -h to be sure, but it's almost certainly
> one of two things: 
> 
> 1) You didn't unmount the filesystem cleanly when you previously
> booted a kernel with ext3 compiled in , and your 2.4 kernel has ext3
> as a module, but you either don't have an initrd or the initrd doesn't
> have the ext3 module in it.

I do also have initrd support compiled into the kernel:

# Block devices
#
...
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_BLK_DEV_INITRD=y

I added this because I have been basing my latest kernel configs
off if the default Redhat 8.0 configuration.  Hmm.  

> 2) You managed to enable a new ext3 feature, such as htree, or
> extended attributes which was supported in the newer kernel, 
> but not in the 2.4 kernel.

I have, indeed, compiled my 2.5.46 kernel with ACL support.
Mea culpa for not studying the effect on the filesystem 
before testing this new code.  I usually try to do compile 
testing on a lot of options, whether or not I exercize the 
code.  I didn't realise that ACL support would modify the 
filesystem whether or not I applied ACLs to a particular file.
Or is that what has happened?

# File systems
#
CONFIG_QUOTA=y
CONFIG_QUOTACTL=y
CONFIG_EXT3_FS=m
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_JBD=y
CONFIG_PROC_FS=y
CONFIG_DEVPTS_FS=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y

>> dumpe2fs -h /dev/hda12
dumpe2fs 1.27 (8-Mar-2002)
Filesystem volume name:   /
Last mounted on:          <not available>
Filesystem UUID:          0a3ccf38-e09c-4ce8-af56-4c086b7adce4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal filetype needs_recovery sparse_super
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1982464
Block count:              3962022
Reserved block count:     198101
Free blocks:              2862850
Free inodes:              1800406
First block:              0
Block size:               4096
Fragment size:            4096
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512
Last mount time:          Wed Nov  6 18:09:02 2002
Last write time:          Wed Nov  6 18:09:02 2002
Mount count:              6
Maximum mount count:      23
Last checked:             Wed Nov  6 16:29:29 2002
Check interval:           15552000 (6 months)
Next check after:         Mon May  5 17:29:29 2003
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               128
Journal UUID:             <none>
Journal inode:            54
Journal device:           0x0000
First orphan inode:       345436

> (1) tends to be the most likely cause, given the confused users who
> ask these sorts of questions on th ext3-users mailing list.  As a
> result, I've developed a very strong distaste for initrd, and
> generally strongly encourage people to compile ext3 and whatever
> device drivers you require into the kernel, and to not try to use
> initrd.  initrd turns out to be a confusing stumbling block for far
> too many users.

I'm not sure how best to proceed.

Thanks very much for your help!

	Miles

Re: EXT2 corruption -- After running 2.5.46, my root partition cannot be mounted by older kernels

[Theodore Ts'o]

On Wed, Nov 06, 2002 at 10:30:06PM -0800, Miles Lane wrote:
> I do also have initrd support compiled into the kernel:
> 
> # Block devices
> #
> ...
> CONFIG_BLK_DEV_RAM=y
> CONFIG_BLK_DEV_RAM_SIZE=4096
> CONFIG_BLK_DEV_INITRD=y
> 
> I added this because I have been basing my latest kernel configs
> off if the default Redhat 8.0 configuration.  Hmm.  

... and you no doubt are compiling ext3 as a module.  Don't do that,
or make sure your initrd image includes the ext3.o module.

> I have, indeed, compiled my 2.5.46 kernel with ACL support.
> Mea culpa for not studying the effect on the filesystem 
> before testing this new code.  I usually try to do compile 
> testing on a lot of options, whether or not I exercize the 
> code.  I didn't realise that ACL support would modify the 
> filesystem whether or not I applied ACLs to a particular file.
> Or is that what has happened?

No, it has nothing to do with ACL support.

Look at your dumpe2fs output:

> >>dumpe2fs -h /dev/hda12
> dumpe2fs 1.27 (8-Mar-2002)
> Filesystem volume name:   /
> Last mounted on:          <not available>
> Filesystem UUID:          0a3ccf38-e09c-4ce8-af56-4c086b7adce4
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic)
> Filesystem features:      has_journal filetype needs_recovery sparse_super
                                                 ^^^^^^^^^^^^^^

There are no ACL or extended attribute features listed in the
dumpe2fs's output.  The only feature which would give ext2 filesystems
heartburn is the needs_recovery feature.  This means either (a) the
filesystem was mounted (using the ext3 filesystem) when you ran
dumpe2fs, or (b) the filesystem was mounted using ext3, but the system
wasn't cleanly shutdown.

This needs_recovery feature is set when the filesystem is mounted as
ext3, and indicates that there may be commited metadata blocks in the
journal which must be copied to their final place on disk before the
filesystem canbe considered consistent.  This feature will stop ext2
filesystems from mounting the filesystem (since the ext2 filesystem
code will check the feature flags, and will stop because it doesn't
understand the needs_revoery flag).

> >(1) tends to be the most likely cause, given the confused users who
> >ask these sorts of questions on th ext3-users mailing list.  As a
> >result, I've developed a very strong distaste for initrd, and
> >generally strongly encourage people to compile ext3 and whatever
> >device drivers you require into the kernel, and to not try to use
> >initrd.  initrd turns out to be a confusing stumbling block for far
> >too many users.

And yup, once again, the confusing initrd feature strikes **again**.

The problem is that your initrd image doesn't include the ext3 module,
and so when you boot your system, the initrd scripts try to mount your
filesystem as ext2, but because it wasn't cleanly shutdown, there the
ext2 filesystem code complains because the needs_recovery feature flag
is set.

Just recompile your 2.4 kernel with ext3 and all device drivers needed
to mount your root filesystem compiled into the kernel, and abandon
using initrd altogether, and you will be a much happier camper.

						- Ted