Re: [Bugme-new] [Bug 11873] New: unable to mount ext3 root filesystem due to htree_dirblock_to_tree

Re: [Bugme-new] [Bug 11873] New: unable to mount ext3 root filesystem due to htree_dirblock_to_tree

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Tue, 28 Oct 2008 05:09:25 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=11873
> 
>            Summary: unable to mount ext3 root filesystem due to
>                     htree_dirblock_to_tree
>            Product: File System
>            Version: 2.5
>      KernelVersion: 2.6.28 rcx git and vanilla trees
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: ext3
>         AssignedTo: akpm@osdl.org
>         ReportedBy: Jimmy.Jazz@gmx.net
>                 CC: Jimmy.Jazz@gmx.net
> 
> 
> Latest working kernel version:2.6.27.3
> Earliest failing kernel version:2.6.28 rc1 git1
> Distribution: gentoo
> Hardware Environment: x86_64 amd
> Software Environment: kernel + initramfs
> Problem Description:
> 
> Since 2.6.28 mount command (busybox) is unable to mount the root file system.
> The error returned is,
> 
> EXT3-fs error (device md_d0p2): htree_dirblock_to_tree: bad entry in directory
> #2: rec_len % 4 != 0 - offset=0, inode=2109536318, rec_len=16058, name_len=111
> 
> (The figures could change)
> 
> 
> To make it work, I need first to mount the file system as an ext2 file system,
> unmount it and recall mount command with the -t ext3 option.
> 
> step 1:
> mount -o ro /dev/md/d0p2 /new-root
> (mount ok with flags ro,errors=continue)
> ls /new-root
> (ls without error)
> umount /new-root
> 
> and then
> 
> step 2:
> mount -t ext3 -o ro /dev/md/d0p2 /new-root
> (mount ok with flags ro,errors=continue,data=ordered)
> ls /new-root
> (ls without error)
> 
> ext2,jbd,ext3,mbcache,crc32 are modules and are loaded from an initramfs file
> system.
> 
> IMHO, it is a problem in mbcache or jbd.
> 
> Also, there is an other problem with that kernel. After creating the /dev node,
> loading md_mod and raid1, mdassemble start as usual the md device, but the
> mount command doesn't recognize the /dev node as valid and abort. 
> I need to stop the md device that has been created earlier and reapply
> mdassemble or mdadm --assemble --scan from initramfs file system. Afterwards
> mount command is able to mount the file system but... like an ext2 file system.
> I'm on step 2 again.
> 
> 
> Steps to reproduce: reboot the kernel, load the initramfs file system and open
> a shell just after the mount command has aborted.
> 

Re: [Bugme-new] [Bug 11873] New: unable to mount ext3 root filesystem due to htree_dirblock_to_tree

[Theodore Tso]

On Tue, Oct 28, 2008 at 10:31:22AM -0700, Andrew Morton wrote:
> > 
> > Since 2.6.28 mount command (busybox) is unable to mount the root file system.
> > The error returned is,
> > 
> > EXT3-fs error (device md_d0p2): htree_dirblock_to_tree: bad entry in directory
> > #2: rec_len % 4 != 0 - offset=0, inode=2109536318, rec_len=16058, name_len=111
> > 

That error means the ext3 filesystem detected an error when responding
to a readdir() request from user space.  So it seems rather strange
that this came from a mount command.  It looks like the mount
succeeded, and then something in userspace did the equivalent of "ls
/*" or "echo /*", and the first part of the directory was totally
garbaled.  (It should have been inode=2, rec_len=12, name_len=1 for
the '.' entry in the root directory.)

> > IMHO, it is a problem in mbcache or jbd.

Why do you think it is a problem with mbcache or jbd?  mbcache is only
used for extended attributes, and jbd would only get invoked once the
filesystem was getting modified.  If there was corruption in the
journal that was corrupting the filesystem when journal was recovered,
the directory corruption would be permanent, and not something that
would magically disappear when you mount the filesystem as ext2
read-only, and then remount as ext3.

This smells like ext3 is getting garbage when first reading in the
filesystem from the md device.  And then when you report that you're
also having mysterious md failures:

> > Also, there is an other problem with that kernel. After creating
> > the /dev node, loading md_mod and raid1, mdassemble start as usual
> > the md device, but the mount command doesn't recognize the /dev
> > node as valid and abort.  I need to stop the md device that has
> > been created earlier and reapply mdassemble or mdadm --assemble
> > --scan from initramfs file system. Afterwards mount command is
> > able to mount the file system but... like an ext2 file system.
> > I'm on step 2 again.
> > 

That smells like a problem with the md device, or the md setup code.

I suspect if you were to arrange to include dumpe2fs or debugfs in
your initrd, you would find that you are reading garbage from the md
device, and dumpe2fs and debugfs would be reporting, "huh?  there's no
ext3 filesystem image here".  I don't think this is a ext3 problem at
all.

							- Ted