fsck 1.39 segfaults while fixing a corrupt inode

fsck 1.39 segfaults while fixing a corrupt inode

[Kosta Kliakhandler]

[-- Attachment #1: Type: text/plain, Size: 819 bytes --]

Hi!

I have a severe problem - I was a fool and made / (root) ext4 (at
least not /home, thank god) and somehow some corruption occured.
While running fsck, it keeps segfaulting, complaining about fast
memory corruption (segfaults always at the same inode, always when I
tell it to fix it). I *think* the issue was addressed in
e2fsprogs-1.40.2 (from what I understood in the changelog) - but I
can't find a patchset for it...
I tried to do the 1.39 patch on the 1.40.2 source but there were more
then a few mismatches and it didn't build. with enough work, I might
be able to build it, but I strongly prefer a ready patch/tarball which
I know will work and I haven't found any on the net.

Can anyone please point me to one or send one to me, or advise on a
different solution?

Thanks in advance,
Kosta.

-- 
Kosta.tk

[-- Attachment #2: error.out --]
[-- Type: text/plain, Size: 3067 bytes --]

#fsck -a /dev/sda3

/ contains a file system with errors, check forced.
/: Inode 7413663 has high 16 bits of extent/index block set
CLEARED.
/: High 16 bits of extent/index block set
CLEARED.
/: Inode 7413663 has corrupt extent index at block 1769235789 (logical 778400627) entry 0
CLEARED.
/: 

*** glibc detected *** fsck.ext4: malloc(): memory corruption (fast): 0x0809b878 ***
======= Backtrace: =========
/lib/libc.so.6[0xb7e43510]
/lib/libc.so.6[0xb7e45c17]
/lib/libc.so.6(malloc+0x7e)[0xb7e46e2e]
/lib/libc.so.6[0xb7e040f0]
/lib/libc.so.6[0xb7e02511]
/lib/libc.so.6[0xb7e01e9c]
/lib/libc.so.6(__dcgettext+0x3d)[0xb7e00fbd]
fsck.ext4[0x805ddea]
fsck.ext4[0x80504dc]
fsck.ext4[0x805186f]
/lib/libext2fs.so.2(block_iterate_extents+0x310)[0xb7f28ad7]
/lib/libext2fs.so.2(ext2fs_block_iterate2+0x1c8)[0xb7f256f5]
fsck.ext4[0x8050e1b]
fsck.ext4[0x8051b4c]
fsck.ext4[0x8051bc7]
/lib/libext2fs.so.2(ext2fs_get_next_inode_full+0x45)[0xb7f2c16e]
fsck.ext4[0x80530bc]
fsck.ext4[0x804e004]
fsck.ext4[0x804dcce]
/lib/libc.so.6(__libc_start_main+0xdc)[0xb7df39bc]
fsck.ext4[0x804b881]
======= Memory map: ========
08048000-08068000 r-xp 00000000 08:03 2785317    /sbin/e2fsck
08068000-0806a000 rw-p 00020000 08:03 2785317    /sbin/e2fsck
0806a000-080e3000 rw-p 0806a000 00:00 0          [heap]
b6f00000-b6f21000 rw-p b6f00000 00:00 0
b6f21000-b7000000 ---p b6f21000 00:00 0
b7099000-b70a3000 r-xp 00000000 08:03 6390674    /usr/lib/gcc/i586-pc-linux-gnu/4.1.2/libgcc_s.so.1
b70a3000-b70a4000 rw-p 00009000 08:03 6390674    /usr/lib/gcc/i586-pc-linux-gnu/4.1.2/libgcc_s.so.1
b70cd000-b7d9e000 rw-p b70cd000 00:00 0
b7d9e000-b7ddd000 r--p 00000000 08:03 7668654    /usr/lib/locale/en_US.utf8/LC_CTYPE
b7ddd000-b7dde000 rw-p b7ddd000 00:00 0
b7dde000-b7f08000 r-xp 00000000 08:03 3018119    /lib/libc-2.6.so
b7f08000-b7f0a000 r--p 00129000 08:03 3018119    /lib/libc-2.6.so
b7f0a000-b7f0b000 rw-p 0012b000 08:03 3018119    /lib/libc-2.6.so
b7f0b000-b7f0e000 rw-p b7f0b000 00:00 0
b7f0e000-b7f10000 r-xp 00000000 08:03 5570963    /lib/libuuid.so.1.2
b7f10000-b7f11000 rw-p 00001000 08:03 5570963    /lib/libuuid.so.1.2
b7f11000-b7f17000 r-xp 00000000 08:03 5570827    /lib/libblkid.so.1.0
b7f17000-b7f18000 rw-p 00006000 08:03 5570827    /lib/libblkid.so.1.0
b7f18000-b7f1a000 r-xp 00000000 08:03 2492999    /lib/libcom_err.so.2.1
b7f1a000-b7f1b000 rw-p 00001000 08:03 2492999    /lib/libcom_err.so.2.1
b7f1b000-b7f34000 r-xp 00000000 08:03 5570863    /lib/libext2fs.so.2.4
b7f34000-b7f35000 rw-p 00019000 08:03 5570863    /lib/libext2fs.so.2.4
b7f35000-b7f36000 rw-p b7f35000 00:00 0
b7f57000-b7f5e000 r--s 00000000 08:03 6557443    /usr/lib/gconv/gconv-modules.cache
b7f5e000-b7f5f000 r--p 00000000 08:03 7668715    /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
b7f5f000-b7f60000 r-xp b7f5f000 00:00 0          [vdso]
b7f60000-b7f7a000 r-xp 00000000 08:03 3018117    /lib/ld-2.6.so
b7f7a000-b7f7b000 r--p 00019000 08:03 3018117    /lib/ld-2.6.so
b7f7b000-b7f7c000 rw-p 0001a000 08:03 3018117    /lib/ld-2.6.so
bfc4e000-bfc64000 rw-p bfc4e000 00:00 0          [stack]

fsck 1.39 segfaults while fixing a corrupt inode

[Kosta Kliakhandler]

[-- Attachment #1: Type: text/plain, Size: 850 bytes --]

Hi!

I have a severe problem - I was a fool and made / (root) ext4 (at
least not /home, thank god) and somehow some corruption occured.
While running fsck, it keeps segfaulting, complaining about fast
memory corruption (segfaults always at the same inode, always when I
tell it to fix it). I *think* the issue was addressed in
e2fsprogs-1.40.2 (from what I understood in the changelog) - but I
can't find a patchset for it...
I tried to do the 1.39 patch on the 1.40.2 source but there were more
then a few mismatches and it didn't build. With enough work, I might
be able to build it, but I strongly prefer a ready patch/tarball which
I would know works, but I haven't found any on the net.

Can anyone please point me to one or send one to me, or advise on a
different solution?

Attached is the error log.

Thanks in advance,
Kosta.

-- 
Kosta.tk

[-- Attachment #2: error.out --]
[-- Type: text/plain, Size: 3067 bytes --]

#fsck -a /dev/sda3

/ contains a file system with errors, check forced.
/: Inode 7413663 has high 16 bits of extent/index block set
CLEARED.
/: High 16 bits of extent/index block set
CLEARED.
/: Inode 7413663 has corrupt extent index at block 1769235789 (logical 778400627) entry 0
CLEARED.
/: 

*** glibc detected *** fsck.ext4: malloc(): memory corruption (fast): 0x0809b878 ***
======= Backtrace: =========
/lib/libc.so.6[0xb7e43510]
/lib/libc.so.6[0xb7e45c17]
/lib/libc.so.6(malloc+0x7e)[0xb7e46e2e]
/lib/libc.so.6[0xb7e040f0]
/lib/libc.so.6[0xb7e02511]
/lib/libc.so.6[0xb7e01e9c]
/lib/libc.so.6(__dcgettext+0x3d)[0xb7e00fbd]
fsck.ext4[0x805ddea]
fsck.ext4[0x80504dc]
fsck.ext4[0x805186f]
/lib/libext2fs.so.2(block_iterate_extents+0x310)[0xb7f28ad7]
/lib/libext2fs.so.2(ext2fs_block_iterate2+0x1c8)[0xb7f256f5]
fsck.ext4[0x8050e1b]
fsck.ext4[0x8051b4c]
fsck.ext4[0x8051bc7]
/lib/libext2fs.so.2(ext2fs_get_next_inode_full+0x45)[0xb7f2c16e]
fsck.ext4[0x80530bc]
fsck.ext4[0x804e004]
fsck.ext4[0x804dcce]
/lib/libc.so.6(__libc_start_main+0xdc)[0xb7df39bc]
fsck.ext4[0x804b881]
======= Memory map: ========
08048000-08068000 r-xp 00000000 08:03 2785317    /sbin/e2fsck
08068000-0806a000 rw-p 00020000 08:03 2785317    /sbin/e2fsck
0806a000-080e3000 rw-p 0806a000 00:00 0          [heap]
b6f00000-b6f21000 rw-p b6f00000 00:00 0
b6f21000-b7000000 ---p b6f21000 00:00 0
b7099000-b70a3000 r-xp 00000000 08:03 6390674    /usr/lib/gcc/i586-pc-linux-gnu/4.1.2/libgcc_s.so.1
b70a3000-b70a4000 rw-p 00009000 08:03 6390674    /usr/lib/gcc/i586-pc-linux-gnu/4.1.2/libgcc_s.so.1
b70cd000-b7d9e000 rw-p b70cd000 00:00 0
b7d9e000-b7ddd000 r--p 00000000 08:03 7668654    /usr/lib/locale/en_US.utf8/LC_CTYPE
b7ddd000-b7dde000 rw-p b7ddd000 00:00 0
b7dde000-b7f08000 r-xp 00000000 08:03 3018119    /lib/libc-2.6.so
b7f08000-b7f0a000 r--p 00129000 08:03 3018119    /lib/libc-2.6.so
b7f0a000-b7f0b000 rw-p 0012b000 08:03 3018119    /lib/libc-2.6.so
b7f0b000-b7f0e000 rw-p b7f0b000 00:00 0
b7f0e000-b7f10000 r-xp 00000000 08:03 5570963    /lib/libuuid.so.1.2
b7f10000-b7f11000 rw-p 00001000 08:03 5570963    /lib/libuuid.so.1.2
b7f11000-b7f17000 r-xp 00000000 08:03 5570827    /lib/libblkid.so.1.0
b7f17000-b7f18000 rw-p 00006000 08:03 5570827    /lib/libblkid.so.1.0
b7f18000-b7f1a000 r-xp 00000000 08:03 2492999    /lib/libcom_err.so.2.1
b7f1a000-b7f1b000 rw-p 00001000 08:03 2492999    /lib/libcom_err.so.2.1
b7f1b000-b7f34000 r-xp 00000000 08:03 5570863    /lib/libext2fs.so.2.4
b7f34000-b7f35000 rw-p 00019000 08:03 5570863    /lib/libext2fs.so.2.4
b7f35000-b7f36000 rw-p b7f35000 00:00 0
b7f57000-b7f5e000 r--s 00000000 08:03 6557443    /usr/lib/gconv/gconv-modules.cache
b7f5e000-b7f5f000 r--p 00000000 08:03 7668715    /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES
b7f5f000-b7f60000 r-xp b7f5f000 00:00 0          [vdso]
b7f60000-b7f7a000 r-xp 00000000 08:03 3018117    /lib/ld-2.6.so
b7f7a000-b7f7b000 r--p 00019000 08:03 3018117    /lib/ld-2.6.so
b7f7b000-b7f7c000 rw-p 0001a000 08:03 3018117    /lib/ld-2.6.so
bfc4e000-bfc64000 rw-p bfc4e000 00:00 0          [stack]

Re: fsck 1.39 segfaults while fixing a corrupt inode

[Andreas Dilger]

On Aug 10, 2007  15:34 +0300, Kosta Kliakhandler wrote:
> I have a severe problem - I was a fool and made / (root) ext4 (at
> least not /home, thank god) and somehow some corruption occured.
> While running fsck, it keeps segfaulting, complaining about fast
> memory corruption (segfaults always at the same inode, always when I
> tell it to fix it). I *think* the issue was addressed in
> e2fsprogs-1.40.2 (from what I understood in the changelog) - but I
> can't find a patchset for it...
> I tried to do the 1.39 patch on the 1.40.2 source but there were more
> then a few mismatches and it didn't build. with enough work, I might
> be able to build it, but I strongly prefer a ready patch/tarball which
> I know will work and I haven't found any on the net.
> 
> Can anyone please point me to one or send one to me, or advise on a
> different solution?

Try ftp://ftp.lustre.org/pub/lustre/other/e2fsprogs/

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: fsck 1.39 segfaults while fixing a corrupt inode

[Theodore Tso]

On Fri, Aug 10, 2007 at 03:41:32PM +0300, Kosta Kliakhandler wrote:
> 
> I have a severe problem - I was a fool and made / (root) ext4 (at
> least not /home, thank god) and somehow some corruption occured.
> While running fsck, it keeps segfaulting, complaining about fast
> memory corruption (segfaults always at the same inode, always when I
> tell it to fix it). I *think* the issue was addressed in
> e2fsprogs-1.40.2 (from what I understood in the changelog)

What, you mean this one?

   A recent change to e2fsck_add_dir_info() to use tdb files to check
   filesystems with a very large number of filesystems had a typo which
   caused us to resize the wrong data structure.  This would cause a
   array overrun leading to malloc pointer corruptions and segfaults.
   Since we normally can very accurately predict how big the the dirinfo
   array needs to be, this bug only got triggered on very badly corrupted
   filesystems.

If so, it couldn't be, since the tdb support was only added in
e2fsprogs 1.40, and you're using the 1.39 patchset, right?  So it has
to be some other problem, and probably a bug which gets triggered when
it runs across a corrupted extent entry.

How big is your root filesystem image?  Unfortunately e2image hasn't
been updated support extents yet (there's a reason I keep telling
people ext4 isn't quite ready for prime time yet...), so we can't use
a compressed e2image file.

							- Ted

Re: fsck 1.39 segfaults while fixing a corrupt inode

[Kosta Kliakhandler]

Thanks man!

I'm now finally writing this from firefox and not from links :)

If you care to know, I found the 1.40.1 patches in the testing dir,
and applied only the extents patch to 1.40.2, and then build it and
ran e2fsck, which worked fine and found and fixed lots of
corruption...

as for the block that was giving problems last time, it now said that
it's position is in -1 instead of 0, so I guess this is what caused
problems..

Best regards,
Kosta.



>  On 8/10/07, Andreas Dilger <adilger@clusterfs.com> wrote:
> >  On Aug 10, 2007  15:34 +0300, Kosta Kliakhandler wrote:
> > > I have a severe problem - I was a fool and made / (root) ext4 (at
> > > least not /home, thank god) and somehow some corruption occured.
> > > While running fsck, it keeps segfaulting, complaining about fast
> > > memory corruption (segfaults always at the same inode, always when I
> > > tell it to fix it). I *think* the issue was addressed in
> > > e2fsprogs-1.40.2 (from what I understood in the changelog) - but I
> >  > can't find a patchset for it...
> > > I tried to do the 1.39 patch on the 1.40.2 source but there were more
> > > then a few mismatches and it didn't build. with enough work, I might
> > > be able to build it, but I strongly prefer a ready patch/tarball which
> > > I know will work and I haven't found any on the net.
> > >
> > > Can anyone please point me to one or send one to me, or advise on a
> > > different solution?
> >
> > Try  ftp://ftp.lustre.org/pub/lustre/other/e2fsprogs/
> >
> > Cheers, Andreas
> > --
> > Andreas Dilger
> > Principal Software Engineer
> > Cluster File Systems, Inc.

Re: fsck 1.39 segfaults while fixing a corrupt inode

[Kosta Kliakhandler]

On 8/10/07, Theodore Tso <tytso@mit.edu> wrote:
> On Fri, Aug 10, 2007 at 03:41:32PM +0300, Kosta Kliakhandler wrote:
> >
> > I have a severe problem - I was a fool and made / (root) ext4 (at
> > least not /home, thank god) and somehow some corruption occured.
> > While running fsck, it keeps segfaulting, complaining about fast
> > memory corruption (segfaults always at the same inode, always when I
> > tell it to fix it). I *think* the issue was addressed in
> > e2fsprogs-1.40.2 (from what I understood in the changelog)
>
> What, you mean this one?
>
>    A recent change to e2fsck_add_dir_info() to use tdb files to check
>    filesystems with a very large number of filesystems had a typo which
>    caused us to resize the wrong data structure.  This would cause a
>    array overrun leading to malloc pointer corruptions and segfaults.
>    Since we normally can very accurately predict how big the the dirinfo
>    array needs to be, this bug only got triggered on very badly corrupted
>    filesystems.
>
> If so, it couldn't be, since the tdb support was only added in
> e2fsprogs 1.40, and you're using the 1.39 patchset, right?  So it has
> to be some other problem, and probably a bug which gets triggered when
> it runs across a corrupted extent entry.
>
> How big is your root filesystem image?  Unfortunately e2image hasn't
> been updated support extents yet (there's a reason I keep telling
> people ext4 isn't quite ready for prime time yet...), so we can't use
> a compressed e2image file.
>
>                                                         - Ted
>

Yes, this was what I thought about... Well, maybe it wasn't that bug,
but what happened to me certainly seemed similar - especially since
after applying the extents patch which andreas supplied, it worked
well.

When I ran the new one, it reported that the problematic inode is in
position -1 and not 0, so I guess this is what caused the problems in
1.39.

Anyway, thanks for the help.

Regards,
Kosta.

-- 
Kosta.tk