Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[InvTraySts]

So long story short, I had a server running that had a processor fail
while powered on, causing the file systems to become corrupt. I
replaced the motherboard, processor and power supply just to be on the
safe side. However, I am at a bit of a loss as to what to do now. I
was working sandeen in the OFTC IRC channel, but, on his
recommendation he suggested me to post something to the mailing list.

Lets start off with one drive at a time (I have 4 that are corrupt).
The specific logical drive in question was in RAID1 on a Dell PERC 5/i
card.
If I try to mount this using:
mount -t ext4 /dev/sda1 /media/tmp

It complains in dmesg with the following output:
685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
comm mount: bad extra_isize (18013 != 256)
[685621.845213] EXT4-fs (sda1): no journal found


However, if I run dumpe2fs -f /dev/sda1 I get the following output:
root@server:~# dumpe2fs -f /dev/sda1
dumpe2fs 1.42.5 (29-Jul-2012)
Filesystem volume name:   root
Last mounted on:          /media/ubuntu/root
Filesystem UUID:          f959e195-[removed]
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index
filetype extent flex_bg sparse_super large_file huge_file uninit_bg
dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         not clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              4849664
Block count:              19398144
Reserved block count:     969907
Free blocks:              17034219
Free inodes:              4592929
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1019
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Sat May 25 14:59:50 2013
Last mount time:          Sat Aug 24 11:04:25 2013
Last write time:          Tue Sep 24 13:55:36 2013
Mount count:              0
Maximum mount count:      -1
Last checked:             Sat Aug 24 16:56:09 2013
Check interval:           0 (<none>)
Lifetime writes:          107 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
Journal backup:           inode blocks
FS Error count:           8
First error time:         Sat Aug 24 13:44:55 2013
First error function:     ext4_iget
First error line #:       3889
First error inode #:      8
First error block #:      0
Last error time:          Tue Sep 24 13:55:36 2013
Last error function:      ext4_iget
Last error line #:        3888
Last error inode #:       8
Last error block #:       0
dumpe2fs: Corrupt extent header while reading journal super block


So I attempted to clone the drive to a 2TB backup drive that is empty,
and currently I am having more problems with the cloned drive than I
am with the original.

sandeen said something about using tune2fs to tell it to remove the
has_journal flag, but I might need some assistance with that.

I would appreciate any help that you could give me, as I know my
chances of recovering data are slim, but I would definitely like to
try and recover as much data as I can.

Thanks
Andrew

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[Eric Sandeen]

On 9/24/13 9:25 PM, InvTraySts wrote:
> So long story short, I had a server running that had a processor fail
> while powered on, causing the file systems to become corrupt. I
> replaced the motherboard, processor and power supply just to be on the
> safe side. However, I am at a bit of a loss as to what to do now. I
> was working sandeen in the OFTC IRC channel, but, on his
> recommendation he suggested me to post something to the mailing list.

Just so we had a record of things.  :)

(also: removing -fsdevel cc:)

> Lets start off with one drive at a time (I have 4 that are corrupt).
> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
> card.
> If I try to mount this using:
> mount -t ext4 /dev/sda1 /media/tmp
> 
> It complains in dmesg with the following output:
> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
> comm mount: bad extra_isize (18013 != 256)
> [685621.845213] EXT4-fs (sda1): no journal found

(FWIW, inode #8 is the journal inode.)

Do you have any idea what happened *first* - did you have any kind of
errors from the raid controller back on Aug 24?

First step is to be sure the storage is in decent shape.  No amount
of fsck or whatnot will fix misconfigured or degraded storage, scrambled
raids, etc...

and if you have 4 "bad" logical drives on that raid, it sure sounds like
something went wrong storage-wise.
 

> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
> root@server:~# dumpe2fs -f /dev/sda1
> dumpe2fs 1.42.5 (29-Jul-2012)
> Filesystem volume name:   root
> Last mounted on:          /media/ubuntu/root
> Filesystem UUID:          f959e195-[removed]
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic)
> Filesystem features:      has_journal ext_attr resize_inode dir_index
> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
> dir_nlink extra_isize
> Filesystem flags:         signed_directory_hash
> Default mount options:    user_xattr acl
> Filesystem state:         not clean with errors
> Errors behavior:          Continue
> Filesystem OS type:       Linux
> Inode count:              4849664
> Block count:              19398144
> Reserved block count:     969907
> Free blocks:              17034219
> Free inodes:              4592929
> First block:              0
> Block size:               4096
> Fragment size:            4096
> Reserved GDT blocks:      1019
> Blocks per group:         32768
> Fragments per group:      32768
> Inodes per group:         8192
> Inode blocks per group:   512
> Flex block group size:    16
> Filesystem created:       Sat May 25 14:59:50 2013
> Last mount time:          Sat Aug 24 11:04:25 2013
> Last write time:          Tue Sep 24 13:55:36 2013
> Mount count:              0
> Maximum mount count:      -1
> Last checked:             Sat Aug 24 16:56:09 2013
> Check interval:           0 (<none>)
> Lifetime writes:          107 GB
> Reserved blocks uid:      0 (user root)
> Reserved blocks gid:      0 (group root)
> First inode:              11
> Inode size:               256
> Required extra isize:     28
> Desired extra isize:      28
> Journal inode:            8
> Default directory hash:   half_md4
> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> Journal backup:           inode blocks
> FS Error count:           8
> First error time:         Sat Aug 24 13:44:55 2013
> First error function:     ext4_iget
> First error line #:       3889
> First error inode #:      8
> First error block #:      0
> Last error time:          Tue Sep 24 13:55:36 2013
> Last error function:      ext4_iget
> Last error line #:        3888
> Last error inode #:       8
> Last error block #:       0
> dumpe2fs: Corrupt extent header while reading journal super block

inode 8 is the journal inode.

> 
> So I attempted to clone the drive to a 2TB backup drive that is empty,
> and currently I am having more problems with the cloned drive than I
> am with the original.

cloned how?  Working on a backup is a good idea, to be sure.

> sandeen said something about using tune2fs to tell it to remove the
> has_journal flag, but I might need some assistance with that.

I had suggested that just because the journal superblock seems
corrupted, and removing & recreating the journal is fairly harmless.

To do so, it'd be tune2fs -O ^has_journal /dev/sda1

But there may well be other problems behind that one.

> I would appreciate any help that you could give me, as I know my
> chances of recovering data are slim, but I would definitely like to
> try and recover as much data as I can.
> 
> Thanks
> Andrew
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[InvTraySts]

Not used to mailing list courtesy so forgive how gmail responds to these...

I didn't have any kind of error on the RAID controller itself. When I
went back for one of the weekends, I went into the RAID controller
BIOS and everything was reported as normal. Of the four logical drives
experiencing problems, only two of them were on the controller, one
was plugged into the motherboard, the last one was plugged into an
add-on SATA card.

I don't know what happened on the 24th of August, all I know is that
it was working fine the previous night, tried to get on the network,
and everything had stopped working (web server, DHCP, bind, samba,
etc). Went down to inspect the machine and noticed that it was running
but there was nothing showing up the monitor when plugging it in. So I
am not sure of the exact events of how it failed, I just know that
after hardware testing, the processor was dead.

I have tried using dd and ddrescue using the following commands:
 dd if=/dev/sdc of=/dev/sdf bs=4096 conv=notrunc,noerror,sync
 ddrescue -vf /dev/sdh /dev/sdf /home/andrew/logfile.txt

On Wed, Sep 25, 2013 at 11:35 AM, Eric Sandeen <sandeen@redhat.com> wrote:
> On 9/24/13 9:25 PM, InvTraySts wrote:
>> So long story short, I had a server running that had a processor fail
>> while powered on, causing the file systems to become corrupt. I
>> replaced the motherboard, processor and power supply just to be on the
>> safe side. However, I am at a bit of a loss as to what to do now. I
>> was working sandeen in the OFTC IRC channel, but, on his
>> recommendation he suggested me to post something to the mailing list.
>
> Just so we had a record of things.  :)
>
> (also: removing -fsdevel cc:)
>
>> Lets start off with one drive at a time (I have 4 that are corrupt).
>> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
>> card.
>> If I try to mount this using:
>> mount -t ext4 /dev/sda1 /media/tmp
>>
>> It complains in dmesg with the following output:
>> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
>> comm mount: bad extra_isize (18013 != 256)
>> [685621.845213] EXT4-fs (sda1): no journal found
>
> (FWIW, inode #8 is the journal inode.)
>
> Do you have any idea what happened *first* - did you have any kind of
> errors from the raid controller back on Aug 24?
>
> First step is to be sure the storage is in decent shape.  No amount
> of fsck or whatnot will fix misconfigured or degraded storage, scrambled
> raids, etc...
>
> and if you have 4 "bad" logical drives on that raid, it sure sounds like
> something went wrong storage-wise.
>
>
>> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
>> root@server:~# dumpe2fs -f /dev/sda1
>> dumpe2fs 1.42.5 (29-Jul-2012)
>> Filesystem volume name:   root
>> Last mounted on:          /media/ubuntu/root
>> Filesystem UUID:          f959e195-[removed]
>> Filesystem magic number:  0xEF53
>> Filesystem revision #:    1 (dynamic)
>> Filesystem features:      has_journal ext_attr resize_inode dir_index
>> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
>> dir_nlink extra_isize
>> Filesystem flags:         signed_directory_hash
>> Default mount options:    user_xattr acl
>> Filesystem state:         not clean with errors
>> Errors behavior:          Continue
>> Filesystem OS type:       Linux
>> Inode count:              4849664
>> Block count:              19398144
>> Reserved block count:     969907
>> Free blocks:              17034219
>> Free inodes:              4592929
>> First block:              0
>> Block size:               4096
>> Fragment size:            4096
>> Reserved GDT blocks:      1019
>> Blocks per group:         32768
>> Fragments per group:      32768
>> Inodes per group:         8192
>> Inode blocks per group:   512
>> Flex block group size:    16
>> Filesystem created:       Sat May 25 14:59:50 2013
>> Last mount time:          Sat Aug 24 11:04:25 2013
>> Last write time:          Tue Sep 24 13:55:36 2013
>> Mount count:              0
>> Maximum mount count:      -1
>> Last checked:             Sat Aug 24 16:56:09 2013
>> Check interval:           0 (<none>)
>> Lifetime writes:          107 GB
>> Reserved blocks uid:      0 (user root)
>> Reserved blocks gid:      0 (group root)
>> First inode:              11
>> Inode size:               256
>> Required extra isize:     28
>> Desired extra isize:      28
>> Journal inode:            8
>> Default directory hash:   half_md4
>> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
>> Journal backup:           inode blocks
>> FS Error count:           8
>> First error time:         Sat Aug 24 13:44:55 2013
>> First error function:     ext4_iget
>> First error line #:       3889
>> First error inode #:      8
>> First error block #:      0
>> Last error time:          Tue Sep 24 13:55:36 2013
>> Last error function:      ext4_iget
>> Last error line #:        3888
>> Last error inode #:       8
>> Last error block #:       0
>> dumpe2fs: Corrupt extent header while reading journal super block
>
> inode 8 is the journal inode.
>
>>
>> So I attempted to clone the drive to a 2TB backup drive that is empty,
>> and currently I am having more problems with the cloned drive than I
>> am with the original.
>
> cloned how?  Working on a backup is a good idea, to be sure.
>
>> sandeen said something about using tune2fs to tell it to remove the
>> has_journal flag, but I might need some assistance with that.
>
> I had suggested that just because the journal superblock seems
> corrupted, and removing & recreating the journal is fairly harmless.
>
> To do so, it'd be tune2fs -O ^has_journal /dev/sda1
>
> But there may well be other problems behind that one.
>
>> I would appreciate any help that you could give me, as I know my
>> chances of recovering data are slim, but I would definitely like to
>> try and recover as much data as I can.
>>
>> Thanks
>> Andrew
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[Eric Sandeen]

On 9/25/13 11:35 AM, InvTraySts wrote:
> Not used to mailing list courtesy so forgive how gmail responds to these...
> 
> I didn't have any kind of error on the RAID controller itself. When I
> went back for one of the weekends, I went into the RAID controller
> BIOS and everything was reported as normal. Of the four logical drives
> experiencing problems, only two of them were on the controller, one
> was plugged into the motherboard, the last one was plugged into an
> add-on SATA card.
> 
> I don't know what happened on the 24th of August, all I know is that
> it was working fine the previous night, tried to get on the network,
> and everything had stopped working (web server, DHCP, bind, samba,
> etc). Went down to inspect the machine and noticed that it was running
> but there was nothing showing up the monitor when plugging it in. So I
> am not sure of the exact events of how it failed, I just know that
> after hardware testing, the processor was dead.

Ok, so pretty extreme hardware failure.

> I have tried using dd and ddrescue using the following commands:
>  dd if=/dev/sdc of=/dev/sdf bs=4096 conv=notrunc,noerror,sync
>  ddrescue -vf /dev/sdh /dev/sdf /home/andrew/logfile.txt

(you said the copy was worse; what was wrong with it?)
(did the dd or ddrescue encounter any errors while doing the copy?)

dd should have worked fine, and you should be able to play with that
image.  First thing I'd try, if it still throws the bad journal
error first, is to tune2fs -O ^has_journal /dev/sdf

Then try to mount and /or run e2fsck on it, see how it fares.
The journal is just the first thing mount's going to look at;
if the corruption is widespread you may just have a cascade of other
errors behind it, but worth a shot.

-Eric

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[InvTraySts]

So for instance, where I can run:
mount /dev/sda1 /media/tmp

It will attempt to mount but complain about either a bad filesystem,
superblock, etc. If I try to mount the cloned drive:
mount: you must specify the filesystem type

Looking back at dmesg the results are different:
[767942.335569] JBD2: Invalid start block of journal: 0
[767942.335572] EXT4-fs (sda1): error loading journal
[767947.740996] EXT3-fs (sdf1): error: can't find ext3 filesystem on dev sdf1.
[767947.741123] EXT4-fs (sdf1): VFS: Can't find ext4 filesystem
[767947.741253] FAT-fs (sdf1): invalid media value (0xad)
[767947.741255] FAT-fs (sdf1): Can't find a valid FAT filesystem
[767947.741403] EXT2-fs (sdf1): error: can't find an ext2 filesystem
on dev sdf1.
[767957.299593] EXT4-fs (sdf1): VFS: Can't find ext4 filesystem

Looking at dumpe2fs for the filesystem on /dev/sdf1 (sdf is the cloned
drive, the actual drive, sda, is in the original e-mail)
root@server:~# dumpe2fs /dev/sdf1
dumpe2fs 1.42.5 (29-Jul-2012)
dumpe2fs: Bad magic number in super-block while trying to open /dev/sdf1
Couldn't find valid filesystem superblock.

dd didn't say it had any errors, and the log file for ddrescue doesn't
appear to give any errors either. I can run it again though. Should
there be any difference in the command that I run this time? I was
under the impression that notrunc,noerror,sync would have been
suitable to clone the drive over.



On Wed, Sep 25, 2013 at 12:50 PM, Eric Sandeen <sandeen@redhat.com> wrote:
> On 9/25/13 11:35 AM, InvTraySts wrote:
>> Not used to mailing list courtesy so forgive how gmail responds to these...
>>
>> I didn't have any kind of error on the RAID controller itself. When I
>> went back for one of the weekends, I went into the RAID controller
>> BIOS and everything was reported as normal. Of the four logical drives
>> experiencing problems, only two of them were on the controller, one
>> was plugged into the motherboard, the last one was plugged into an
>> add-on SATA card.
>>
>> I don't know what happened on the 24th of August, all I know is that
>> it was working fine the previous night, tried to get on the network,
>> and everything had stopped working (web server, DHCP, bind, samba,
>> etc). Went down to inspect the machine and noticed that it was running
>> but there was nothing showing up the monitor when plugging it in. So I
>> am not sure of the exact events of how it failed, I just know that
>> after hardware testing, the processor was dead.
>
> Ok, so pretty extreme hardware failure.
>
>> I have tried using dd and ddrescue using the following commands:
>>  dd if=/dev/sdc of=/dev/sdf bs=4096 conv=notrunc,noerror,sync
>>  ddrescue -vf /dev/sdh /dev/sdf /home/andrew/logfile.txt
>
> (you said the copy was worse; what was wrong with it?)
> (did the dd or ddrescue encounter any errors while doing the copy?)
>
> dd should have worked fine, and you should be able to play with that
> image.  First thing I'd try, if it still throws the bad journal
> error first, is to tune2fs -O ^has_journal /dev/sdf
>
> Then try to mount and /or run e2fsck on it, see how it fares.
> The journal is just the first thing mount's going to look at;
> if the corruption is widespread you may just have a cascade of other
> errors behind it, but worth a shot.
>
> -Eric
>

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[Eric Sandeen]

On 9/25/13 12:09 PM, InvTraySts wrote:
> So for instance, where I can run:
> mount /dev/sda1 /media/tmp
> 
> It will attempt to mount but complain about either a bad filesystem,
> superblock, etc.

that's userspace mount; look at dmesg to see what really went wrong.

> If I try to mount the cloned drive:
> mount: you must specify the filesystem type
> 
> Looking back at dmesg the results are different:
> [767942.335569] JBD2: Invalid start block of journal: 0
> [767942.335572] EXT4-fs (sda1): error loading journal
> [767947.740996] EXT3-fs (sdf1): error: can't find ext3 filesystem on dev sdf1.
> [767947.741123] EXT4-fs (sdf1): VFS: Can't find ext4 filesystem
> [767947.741253] FAT-fs (sdf1): invalid media value (0xad)
> [767947.741255] FAT-fs (sdf1): Can't find a valid FAT filesystem
> [767947.741403] EXT2-fs (sdf1): error: can't find an ext2 filesystem
> on dev sdf1.
> [767957.299593] EXT4-fs (sdf1): VFS: Can't find ext4 filesystem

ok, you did clone it wrong.  Maybe you cloned sdc to sdf1?  What do

# blkid /dev/sdf1

# blkid /dev/sdf

say?

> Looking at dumpe2fs for the filesystem on /dev/sdf1 (sdf is the cloned
> drive, the actual drive, sda, is in the original e-mail)
> root@server:~# dumpe2fs /dev/sdf1
> dumpe2fs 1.42.5 (29-Jul-2012)
> dumpe2fs: Bad magic number in super-block while trying to open /dev/sdf1
> Couldn't find valid filesystem superblock.
> 
> dd didn't say it had any errors, and the log file for ddrescue doesn't
> appear to give any errors either. I can run it again though. Should
> there be any difference in the command that I run this time? I was
> under the impression that notrunc,noerror,sync would have been
> suitable to clone the drive over.

Should be.  I think maybe you must "missed."

-Eric

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[InvTraySts]

Blkid for the drives say:

root@server:~# blkid /dev/sdf
root@server:~#
root@server:~# blkid /dev/sdf1
root@server:~#


root@server:~# blkid /dev/sda1
/dev/sda1: LABEL="root" UUID=remove TYPE="ext4"
root@server:~# blkid /dev/sda
root@server:~#

On Wed, Sep 25, 2013 at 5:45 PM, Eric Sandeen <sandeen@redhat.com> wrote:
> On 9/25/13 12:09 PM, InvTraySts wrote:
>> So for instance, where I can run:
>> mount /dev/sda1 /media/tmp
>>
>> It will attempt to mount but complain about either a bad filesystem,
>> superblock, etc.
>
> that's userspace mount; look at dmesg to see what really went wrong.
>
>> If I try to mount the cloned drive:
>> mount: you must specify the filesystem type
>>
>> Looking back at dmesg the results are different:
>> [767942.335569] JBD2: Invalid start block of journal: 0
>> [767942.335572] EXT4-fs (sda1): error loading journal
>> [767947.740996] EXT3-fs (sdf1): error: can't find ext3 filesystem on dev sdf1.
>> [767947.741123] EXT4-fs (sdf1): VFS: Can't find ext4 filesystem
>> [767947.741253] FAT-fs (sdf1): invalid media value (0xad)
>> [767947.741255] FAT-fs (sdf1): Can't find a valid FAT filesystem
>> [767947.741403] EXT2-fs (sdf1): error: can't find an ext2 filesystem
>> on dev sdf1.
>> [767957.299593] EXT4-fs (sdf1): VFS: Can't find ext4 filesystem
>
> ok, you did clone it wrong.  Maybe you cloned sdc to sdf1?  What do
>
> # blkid /dev/sdf1
>
> # blkid /dev/sdf
>
> say?
>
>> Looking at dumpe2fs for the filesystem on /dev/sdf1 (sdf is the cloned
>> drive, the actual drive, sda, is in the original e-mail)
>> root@server:~# dumpe2fs /dev/sdf1
>> dumpe2fs 1.42.5 (29-Jul-2012)
>> dumpe2fs: Bad magic number in super-block while trying to open /dev/sdf1
>> Couldn't find valid filesystem superblock.
>>
>> dd didn't say it had any errors, and the log file for ddrescue doesn't
>> appear to give any errors either. I can run it again though. Should
>> there be any difference in the command that I run this time? I was
>> under the impression that notrunc,noerror,sync would have been
>> suitable to clone the drive over.
>
> Should be.  I think maybe you must "missed."
>
> -Eric
>

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[Eric Sandeen]

On 9/25/13 6:57 PM, InvTraySts wrote:
> Blkid for the drives say:
> 
> root@server:~# blkid /dev/sdf
> root@server:~#
> root@server:~# blkid /dev/sdf1
> root@server:~#
> 

Jan got it right, I wasn't thinking about dd'ing the entire drive w/ partition table.

Onward! ;)

-Eric

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[Jan Kara]

On Tue 24-09-13 22:25:49, InvTraySts wrote:
> So long story short, I had a server running that had a processor fail
> while powered on, causing the file systems to become corrupt. I
> replaced the motherboard, processor and power supply just to be on the
> safe side. However, I am at a bit of a loss as to what to do now. I
> was working sandeen in the OFTC IRC channel, but, on his
> recommendation he suggested me to post something to the mailing list.
> 
> Lets start off with one drive at a time (I have 4 that are corrupt).
> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
> card.
> If I try to mount this using:
> mount -t ext4 /dev/sda1 /media/tmp
> 
> It complains in dmesg with the following output:
> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
> comm mount: bad extra_isize (18013 != 256)
> [685621.845213] EXT4-fs (sda1): no journal found
> 
> 
> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
> root@server:~# dumpe2fs -f /dev/sda1
> dumpe2fs 1.42.5 (29-Jul-2012)
> Filesystem volume name:   root
> Last mounted on:          /media/ubuntu/root
> Filesystem UUID:          f959e195-[removed]
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic)
> Filesystem features:      has_journal ext_attr resize_inode dir_index
> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
> dir_nlink extra_isize
> Filesystem flags:         signed_directory_hash
> Default mount options:    user_xattr acl
> Filesystem state:         not clean with errors
> Errors behavior:          Continue
> Filesystem OS type:       Linux
> Inode count:              4849664
> Block count:              19398144
> Reserved block count:     969907
> Free blocks:              17034219
> Free inodes:              4592929
> First block:              0
> Block size:               4096
> Fragment size:            4096
> Reserved GDT blocks:      1019
> Blocks per group:         32768
> Fragments per group:      32768
> Inodes per group:         8192
> Inode blocks per group:   512
> Flex block group size:    16
> Filesystem created:       Sat May 25 14:59:50 2013
> Last mount time:          Sat Aug 24 11:04:25 2013
> Last write time:          Tue Sep 24 13:55:36 2013
> Mount count:              0
> Maximum mount count:      -1
> Last checked:             Sat Aug 24 16:56:09 2013
> Check interval:           0 (<none>)
> Lifetime writes:          107 GB
> Reserved blocks uid:      0 (user root)
> Reserved blocks gid:      0 (group root)
> First inode:              11
> Inode size:               256
> Required extra isize:     28
> Desired extra isize:      28
> Journal inode:            8
> Default directory hash:   half_md4
> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> Journal backup:           inode blocks
> FS Error count:           8
> First error time:         Sat Aug 24 13:44:55 2013
> First error function:     ext4_iget
> First error line #:       3889
> First error inode #:      8
> First error block #:      0
> Last error time:          Tue Sep 24 13:55:36 2013
> Last error function:      ext4_iget
> Last error line #:        3888
> Last error inode #:       8
> Last error block #:       0
> dumpe2fs: Corrupt extent header while reading journal super block
  OK, so really journal inode (inode #8) looks toast but superblock looks
OK.

> So I attempted to clone the drive to a 2TB backup drive that is empty,
> and currently I am having more problems with the cloned drive than I
> am with the original.
> 
> sandeen said something about using tune2fs to tell it to remove the
> has_journal flag, but I might need some assistance with that.
  Yes, you can do that with:
tune2fs -f -O ^has_journal /dev/sda1

  Let's see what mount will say after that.

  Another option is to run
debugfs /dev/sda1

  Then you can use ls, cd, and other debugfs commands to move within the
filesystem and investigate things. If that will work, you have a reasonable
chance of getting at least some data back.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[InvTraySts]

And am cloning the drive without the sync parameter this time.
root@server:~# dd if=/dev/sda of=/dev/sdf bs=4096 conv=notrunc,noerror
After it finished, I attempted to run dumpe2fs and it still responds with:
root@server:~# dumpe2fs /dev/sdf1
dumpe2fs 1.42.5 (29-Jul-2012)
dumpe2fs: Bad magic number in super-block while trying to open /dev/sdf1
Couldn't find valid filesystem superblock.


So I went ahead and tried to run the tune2fs command:
root@server:~# tune2fs -f -O ^has_journal /dev/sda1
tune2fs 1.42.5 (29-Jul-2012)
tune2fs: Bad magic number in super-block while trying to open /dev/sda1
Couldn't find valid filesystem superblock.

Which also fails, yet dumpe2fs on /dev/sda1 works fine.


On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
> On Tue 24-09-13 22:25:49, InvTraySts wrote:
>> So long story short, I had a server running that had a processor fail
>> while powered on, causing the file systems to become corrupt. I
>> replaced the motherboard, processor and power supply just to be on the
>> safe side. However, I am at a bit of a loss as to what to do now. I
>> was working sandeen in the OFTC IRC channel, but, on his
>> recommendation he suggested me to post something to the mailing list.
>>
>> Lets start off with one drive at a time (I have 4 that are corrupt).
>> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
>> card.
>> If I try to mount this using:
>> mount -t ext4 /dev/sda1 /media/tmp
>>
>> It complains in dmesg with the following output:
>> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
>> comm mount: bad extra_isize (18013 != 256)
>> [685621.845213] EXT4-fs (sda1): no journal found
>>
>>
>> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
>> root@server:~# dumpe2fs -f /dev/sda1
>> dumpe2fs 1.42.5 (29-Jul-2012)
>> Filesystem volume name:   root
>> Last mounted on:          /media/ubuntu/root
>> Filesystem UUID:          f959e195-[removed]
>> Filesystem magic number:  0xEF53
>> Filesystem revision #:    1 (dynamic)
>> Filesystem features:      has_journal ext_attr resize_inode dir_index
>> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
>> dir_nlink extra_isize
>> Filesystem flags:         signed_directory_hash
>> Default mount options:    user_xattr acl
>> Filesystem state:         not clean with errors
>> Errors behavior:          Continue
>> Filesystem OS type:       Linux
>> Inode count:              4849664
>> Block count:              19398144
>> Reserved block count:     969907
>> Free blocks:              17034219
>> Free inodes:              4592929
>> First block:              0
>> Block size:               4096
>> Fragment size:            4096
>> Reserved GDT blocks:      1019
>> Blocks per group:         32768
>> Fragments per group:      32768
>> Inodes per group:         8192
>> Inode blocks per group:   512
>> Flex block group size:    16
>> Filesystem created:       Sat May 25 14:59:50 2013
>> Last mount time:          Sat Aug 24 11:04:25 2013
>> Last write time:          Tue Sep 24 13:55:36 2013
>> Mount count:              0
>> Maximum mount count:      -1
>> Last checked:             Sat Aug 24 16:56:09 2013
>> Check interval:           0 (<none>)
>> Lifetime writes:          107 GB
>> Reserved blocks uid:      0 (user root)
>> Reserved blocks gid:      0 (group root)
>> First inode:              11
>> Inode size:               256
>> Required extra isize:     28
>> Desired extra isize:      28
>> Journal inode:            8
>> Default directory hash:   half_md4
>> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
>> Journal backup:           inode blocks
>> FS Error count:           8
>> First error time:         Sat Aug 24 13:44:55 2013
>> First error function:     ext4_iget
>> First error line #:       3889
>> First error inode #:      8
>> First error block #:      0
>> Last error time:          Tue Sep 24 13:55:36 2013
>> Last error function:      ext4_iget
>> Last error line #:        3888
>> Last error inode #:       8
>> Last error block #:       0
>> dumpe2fs: Corrupt extent header while reading journal super block
>   OK, so really journal inode (inode #8) looks toast but superblock looks
> OK.
>
>> So I attempted to clone the drive to a 2TB backup drive that is empty,
>> and currently I am having more problems with the cloned drive than I
>> am with the original.
>>
>> sandeen said something about using tune2fs to tell it to remove the
>> has_journal flag, but I might need some assistance with that.
>   Yes, you can do that with:
> tune2fs -f -O ^has_journal /dev/sda1
>
>   Let's see what mount will say after that.
>
>   Another option is to run
> debugfs /dev/sda1
>
>   Then you can use ls, cd, and other debugfs commands to move within the
> filesystem and investigate things. If that will work, you have a reasonable
> chance of getting at least some data back.
>
>                                                                 Honza
> --
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR


On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
> On Tue 24-09-13 22:25:49, InvTraySts wrote:
>> So long story short, I had a server running that had a processor fail
>> while powered on, causing the file systems to become corrupt. I
>> replaced the motherboard, processor and power supply just to be on the
>> safe side. However, I am at a bit of a loss as to what to do now. I
>> was working sandeen in the OFTC IRC channel, but, on his
>> recommendation he suggested me to post something to the mailing list.
>>
>> Lets start off with one drive at a time (I have 4 that are corrupt).
>> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
>> card.
>> If I try to mount this using:
>> mount -t ext4 /dev/sda1 /media/tmp
>>
>> It complains in dmesg with the following output:
>> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
>> comm mount: bad extra_isize (18013 != 256)
>> [685621.845213] EXT4-fs (sda1): no journal found
>>
>>
>> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
>> root@server:~# dumpe2fs -f /dev/sda1
>> dumpe2fs 1.42.5 (29-Jul-2012)
>> Filesystem volume name:   root
>> Last mounted on:          /media/ubuntu/root
>> Filesystem UUID:          f959e195-[removed]
>> Filesystem magic number:  0xEF53
>> Filesystem revision #:    1 (dynamic)
>> Filesystem features:      has_journal ext_attr resize_inode dir_index
>> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
>> dir_nlink extra_isize
>> Filesystem flags:         signed_directory_hash
>> Default mount options:    user_xattr acl
>> Filesystem state:         not clean with errors
>> Errors behavior:          Continue
>> Filesystem OS type:       Linux
>> Inode count:              4849664
>> Block count:              19398144
>> Reserved block count:     969907
>> Free blocks:              17034219
>> Free inodes:              4592929
>> First block:              0
>> Block size:               4096
>> Fragment size:            4096
>> Reserved GDT blocks:      1019
>> Blocks per group:         32768
>> Fragments per group:      32768
>> Inodes per group:         8192
>> Inode blocks per group:   512
>> Flex block group size:    16
>> Filesystem created:       Sat May 25 14:59:50 2013
>> Last mount time:          Sat Aug 24 11:04:25 2013
>> Last write time:          Tue Sep 24 13:55:36 2013
>> Mount count:              0
>> Maximum mount count:      -1
>> Last checked:             Sat Aug 24 16:56:09 2013
>> Check interval:           0 (<none>)
>> Lifetime writes:          107 GB
>> Reserved blocks uid:      0 (user root)
>> Reserved blocks gid:      0 (group root)
>> First inode:              11
>> Inode size:               256
>> Required extra isize:     28
>> Desired extra isize:      28
>> Journal inode:            8
>> Default directory hash:   half_md4
>> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
>> Journal backup:           inode blocks
>> FS Error count:           8
>> First error time:         Sat Aug 24 13:44:55 2013
>> First error function:     ext4_iget
>> First error line #:       3889
>> First error inode #:      8
>> First error block #:      0
>> Last error time:          Tue Sep 24 13:55:36 2013
>> Last error function:      ext4_iget
>> Last error line #:        3888
>> Last error inode #:       8
>> Last error block #:       0
>> dumpe2fs: Corrupt extent header while reading journal super block
>   OK, so really journal inode (inode #8) looks toast but superblock looks
> OK.
>
>> So I attempted to clone the drive to a 2TB backup drive that is empty,
>> and currently I am having more problems with the cloned drive than I
>> am with the original.
>>
>> sandeen said something about using tune2fs to tell it to remove the
>> has_journal flag, but I might need some assistance with that.
>   Yes, you can do that with:
> tune2fs -f -O ^has_journal /dev/sda1
>
>   Let's see what mount will say after that.
>
>   Another option is to run
> debugfs /dev/sda1
>
>   Then you can use ls, cd, and other debugfs commands to move within the
> filesystem and investigate things. If that will work, you have a reasonable
> chance of getting at least some data back.
>
>                                                                 Honza
> --
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[Jan Kara]

On Wed 25-09-13 15:24:34, InvTraySts wrote:
> And am cloning the drive without the sync parameter this time.
> root@server:~# dd if=/dev/sda of=/dev/sdf bs=4096 conv=notrunc,noerror
> After it finished, I attempted to run dumpe2fs and it still responds with:
> root@server:~# dumpe2fs /dev/sdf1
> dumpe2fs 1.42.5 (29-Jul-2012)
> dumpe2fs: Bad magic number in super-block while trying to open /dev/sdf1
> Couldn't find valid filesystem superblock.
  Well, that's likely because the partition table on /dev/sdf didn't get
reread. You can run 'partprobe /dev/sdf' to tell the kernel about the new
partition table.

								Honza

> So I went ahead and tried to run the tune2fs command:
> root@server:~# tune2fs -f -O ^has_journal /dev/sda1
> tune2fs 1.42.5 (29-Jul-2012)
> tune2fs: Bad magic number in super-block while trying to open /dev/sda1
> Couldn't find valid filesystem superblock.
> 
> Which also fails, yet dumpe2fs on /dev/sda1 works fine.
> 
> 
> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
> >> So long story short, I had a server running that had a processor fail
> >> while powered on, causing the file systems to become corrupt. I
> >> replaced the motherboard, processor and power supply just to be on the
> >> safe side. However, I am at a bit of a loss as to what to do now. I
> >> was working sandeen in the OFTC IRC channel, but, on his
> >> recommendation he suggested me to post something to the mailing list.
> >>
> >> Lets start off with one drive at a time (I have 4 that are corrupt).
> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
> >> card.
> >> If I try to mount this using:
> >> mount -t ext4 /dev/sda1 /media/tmp
> >>
> >> It complains in dmesg with the following output:
> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
> >> comm mount: bad extra_isize (18013 != 256)
> >> [685621.845213] EXT4-fs (sda1): no journal found
> >>
> >>
> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
> >> root@server:~# dumpe2fs -f /dev/sda1
> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> Filesystem volume name:   root
> >> Last mounted on:          /media/ubuntu/root
> >> Filesystem UUID:          f959e195-[removed]
> >> Filesystem magic number:  0xEF53
> >> Filesystem revision #:    1 (dynamic)
> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
> >> dir_nlink extra_isize
> >> Filesystem flags:         signed_directory_hash
> >> Default mount options:    user_xattr acl
> >> Filesystem state:         not clean with errors
> >> Errors behavior:          Continue
> >> Filesystem OS type:       Linux
> >> Inode count:              4849664
> >> Block count:              19398144
> >> Reserved block count:     969907
> >> Free blocks:              17034219
> >> Free inodes:              4592929
> >> First block:              0
> >> Block size:               4096
> >> Fragment size:            4096
> >> Reserved GDT blocks:      1019
> >> Blocks per group:         32768
> >> Fragments per group:      32768
> >> Inodes per group:         8192
> >> Inode blocks per group:   512
> >> Flex block group size:    16
> >> Filesystem created:       Sat May 25 14:59:50 2013
> >> Last mount time:          Sat Aug 24 11:04:25 2013
> >> Last write time:          Tue Sep 24 13:55:36 2013
> >> Mount count:              0
> >> Maximum mount count:      -1
> >> Last checked:             Sat Aug 24 16:56:09 2013
> >> Check interval:           0 (<none>)
> >> Lifetime writes:          107 GB
> >> Reserved blocks uid:      0 (user root)
> >> Reserved blocks gid:      0 (group root)
> >> First inode:              11
> >> Inode size:               256
> >> Required extra isize:     28
> >> Desired extra isize:      28
> >> Journal inode:            8
> >> Default directory hash:   half_md4
> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> >> Journal backup:           inode blocks
> >> FS Error count:           8
> >> First error time:         Sat Aug 24 13:44:55 2013
> >> First error function:     ext4_iget
> >> First error line #:       3889
> >> First error inode #:      8
> >> First error block #:      0
> >> Last error time:          Tue Sep 24 13:55:36 2013
> >> Last error function:      ext4_iget
> >> Last error line #:        3888
> >> Last error inode #:       8
> >> Last error block #:       0
> >> dumpe2fs: Corrupt extent header while reading journal super block
> >   OK, so really journal inode (inode #8) looks toast but superblock looks
> > OK.
> >
> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
> >> and currently I am having more problems with the cloned drive than I
> >> am with the original.
> >>
> >> sandeen said something about using tune2fs to tell it to remove the
> >> has_journal flag, but I might need some assistance with that.
> >   Yes, you can do that with:
> > tune2fs -f -O ^has_journal /dev/sda1
> >
> >   Let's see what mount will say after that.
> >
> >   Another option is to run
> > debugfs /dev/sda1
> >
> >   Then you can use ls, cd, and other debugfs commands to move within the
> > filesystem and investigate things. If that will work, you have a reasonable
> > chance of getting at least some data back.
> >
> >                                                                 Honza
> > --
> > Jan Kara <jack@suse.cz>
> > SUSE Labs, CR
> 
> 
> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
> >> So long story short, I had a server running that had a processor fail
> >> while powered on, causing the file systems to become corrupt. I
> >> replaced the motherboard, processor and power supply just to be on the
> >> safe side. However, I am at a bit of a loss as to what to do now. I
> >> was working sandeen in the OFTC IRC channel, but, on his
> >> recommendation he suggested me to post something to the mailing list.
> >>
> >> Lets start off with one drive at a time (I have 4 that are corrupt).
> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
> >> card.
> >> If I try to mount this using:
> >> mount -t ext4 /dev/sda1 /media/tmp
> >>
> >> It complains in dmesg with the following output:
> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
> >> comm mount: bad extra_isize (18013 != 256)
> >> [685621.845213] EXT4-fs (sda1): no journal found
> >>
> >>
> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
> >> root@server:~# dumpe2fs -f /dev/sda1
> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> Filesystem volume name:   root
> >> Last mounted on:          /media/ubuntu/root
> >> Filesystem UUID:          f959e195-[removed]
> >> Filesystem magic number:  0xEF53
> >> Filesystem revision #:    1 (dynamic)
> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
> >> dir_nlink extra_isize
> >> Filesystem flags:         signed_directory_hash
> >> Default mount options:    user_xattr acl
> >> Filesystem state:         not clean with errors
> >> Errors behavior:          Continue
> >> Filesystem OS type:       Linux
> >> Inode count:              4849664
> >> Block count:              19398144
> >> Reserved block count:     969907
> >> Free blocks:              17034219
> >> Free inodes:              4592929
> >> First block:              0
> >> Block size:               4096
> >> Fragment size:            4096
> >> Reserved GDT blocks:      1019
> >> Blocks per group:         32768
> >> Fragments per group:      32768
> >> Inodes per group:         8192
> >> Inode blocks per group:   512
> >> Flex block group size:    16
> >> Filesystem created:       Sat May 25 14:59:50 2013
> >> Last mount time:          Sat Aug 24 11:04:25 2013
> >> Last write time:          Tue Sep 24 13:55:36 2013
> >> Mount count:              0
> >> Maximum mount count:      -1
> >> Last checked:             Sat Aug 24 16:56:09 2013
> >> Check interval:           0 (<none>)
> >> Lifetime writes:          107 GB
> >> Reserved blocks uid:      0 (user root)
> >> Reserved blocks gid:      0 (group root)
> >> First inode:              11
> >> Inode size:               256
> >> Required extra isize:     28
> >> Desired extra isize:      28
> >> Journal inode:            8
> >> Default directory hash:   half_md4
> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> >> Journal backup:           inode blocks
> >> FS Error count:           8
> >> First error time:         Sat Aug 24 13:44:55 2013
> >> First error function:     ext4_iget
> >> First error line #:       3889
> >> First error inode #:      8
> >> First error block #:      0
> >> Last error time:          Tue Sep 24 13:55:36 2013
> >> Last error function:      ext4_iget
> >> Last error line #:        3888
> >> Last error inode #:       8
> >> Last error block #:       0
> >> dumpe2fs: Corrupt extent header while reading journal super block
> >   OK, so really journal inode (inode #8) looks toast but superblock looks
> > OK.
> >
> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
> >> and currently I am having more problems with the cloned drive than I
> >> am with the original.
> >>
> >> sandeen said something about using tune2fs to tell it to remove the
> >> has_journal flag, but I might need some assistance with that.
> >   Yes, you can do that with:
> > tune2fs -f -O ^has_journal /dev/sda1
> >
> >   Let's see what mount will say after that.
> >
> >   Another option is to run
> > debugfs /dev/sda1
> >
> >   Then you can use ls, cd, and other debugfs commands to move within the
> > filesystem and investigate things. If that will work, you have a reasonable
> > chance of getting at least some data back.
> >
> >                                                                 Honza
> > --
> > Jan Kara <jack@suse.cz>
> > SUSE Labs, CR
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[InvTraySts]

(Going to merge these two back, because both of you are actually
helping me, but with the two conversations being segregated like this
it makes it hard to correlate with you both)
The partprobe did work with getting the partition table reread.
After that, the tune2fs sorta worked.
root@server:~# tune2fs -f -O ^has_journal /dev/sdf1
tune2fs 1.42.5 (29-Jul-2012)
root@server:~# mount /dev/sdf1 /media/tmp
root@server:~# ls -l /media/tmp/
total 0

When I try and use the debugfs /dev/sdf1

root@server:~# debugfs /dev/sdf1
debugfs 1.42.5 (29-Jul-2012)
debugfs:  ls
EXT2 directory corrupted
debugfs:  ls /
/: EXT2 directory corrupted


The drive is supposed to be ext4.
root@server:~# dumpe2fs /dev/sdf1
dumpe2fs 1.42.5 (29-Jul-2012)
Filesystem volume name:   root
Last mounted on:          /media/ubuntu/root
Filesystem UUID:         removed
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      ext_attr resize_inode dir_index filetype
extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink
extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         not clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              4849664
Block count:              19398144
Reserved block count:     969907
Free blocks:              17066988
Free inodes:              4592929
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1019
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Sat May 25 14:59:50 2013
Last mount time:          Wed Sep 25 19:59:07 2013
Last write time:          Wed Sep 25 19:59:51 2013
Mount count:              1
Maximum mount count:      -1
Last checked:             Sat Aug 24 16:56:09 2013
Check interval:           0 (<none>)
Lifetime writes:          107 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Default directory hash:   half_md4
Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
Journal backup:           inode blocks
FS Error count:           9
First error time:         Sat Aug 24 13:44:55 2013
First error function:     ext4_iget
First error line #:       3889
First error inode #:      8
First error block #:      0
Last error time:          Wed Sep 25 19:59:12 2013
Last error function:      htree_dirblock_to_tree
Last error line #:        892
Last error inode #:       2
Last error block #:       20037

[...]
[output scrolls very fast, and is very large]
Group 10: (Blocks 327680-360447) [ITABLE_ZEROED]
  Checksum 0xecaa, unused inodes 0
  Block bitmap at 1035 (bg #0 + 1035), Inode bitmap at 1051 (bg #0 + 1051)
  Inode table at 6177-6688 (bg #0 + 6177)
  0 free blocks, 16 free inodes, 0 directories
  Free blocks: 327680, 327683-327685, 327687, 327689-327690,
327692-327693, 327695-327697, 327700-327701, 327703, 327705,
327707-327709, 327711-327715, 327718-327727, 327730-327808,
327810-327823, 327825-327842, 327846-327855, 327857-327875, 327878,
327881

(the total amount of pages that the dumpe2fs comes out with is about
240 pages. something that can't be pasted. If attachments would be
required, I can attach a file with it).

On Wed, Sep 25, 2013 at 5:28 PM, Jan Kara <jack@suse.cz> wrote:
> On Wed 25-09-13 15:24:34, InvTraySts wrote:
>> And am cloning the drive without the sync parameter this time.
>> root@server:~# dd if=/dev/sda of=/dev/sdf bs=4096 conv=notrunc,noerror
>> After it finished, I attempted to run dumpe2fs and it still responds with:
>> root@server:~# dumpe2fs /dev/sdf1
>> dumpe2fs 1.42.5 (29-Jul-2012)
>> dumpe2fs: Bad magic number in super-block while trying to open /dev/sdf1
>> Couldn't find valid filesystem superblock.
>   Well, that's likely because the partition table on /dev/sdf didn't get
> reread. You can run 'partprobe /dev/sdf' to tell the kernel about the new
> partition table.
>
>                                                                 Honza
>
>> So I went ahead and tried to run the tune2fs command:
>> root@server:~# tune2fs -f -O ^has_journal /dev/sda1
>> tune2fs 1.42.5 (29-Jul-2012)
>> tune2fs: Bad magic number in super-block while trying to open /dev/sda1
>> Couldn't find valid filesystem superblock.
>>
>> Which also fails, yet dumpe2fs on /dev/sda1 works fine.
>>
>>
>> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
>> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
>> >> So long story short, I had a server running that had a processor fail
>> >> while powered on, causing the file systems to become corrupt. I
>> >> replaced the motherboard, processor and power supply just to be on the
>> >> safe side. However, I am at a bit of a loss as to what to do now. I
>> >> was working sandeen in the OFTC IRC channel, but, on his
>> >> recommendation he suggested me to post something to the mailing list.
>> >>
>> >> Lets start off with one drive at a time (I have 4 that are corrupt).
>> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
>> >> card.
>> >> If I try to mount this using:
>> >> mount -t ext4 /dev/sda1 /media/tmp
>> >>
>> >> It complains in dmesg with the following output:
>> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
>> >> comm mount: bad extra_isize (18013 != 256)
>> >> [685621.845213] EXT4-fs (sda1): no journal found
>> >>
>> >>
>> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
>> >> root@server:~# dumpe2fs -f /dev/sda1
>> >> dumpe2fs 1.42.5 (29-Jul-2012)
>> >> Filesystem volume name:   root
>> >> Last mounted on:          /media/ubuntu/root
>> >> Filesystem UUID:          f959e195-[removed]
>> >> Filesystem magic number:  0xEF53
>> >> Filesystem revision #:    1 (dynamic)
>> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
>> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
>> >> dir_nlink extra_isize
>> >> Filesystem flags:         signed_directory_hash
>> >> Default mount options:    user_xattr acl
>> >> Filesystem state:         not clean with errors
>> >> Errors behavior:          Continue
>> >> Filesystem OS type:       Linux
>> >> Inode count:              4849664
>> >> Block count:              19398144
>> >> Reserved block count:     969907
>> >> Free blocks:              17034219
>> >> Free inodes:              4592929
>> >> First block:              0
>> >> Block size:               4096
>> >> Fragment size:            4096
>> >> Reserved GDT blocks:      1019
>> >> Blocks per group:         32768
>> >> Fragments per group:      32768
>> >> Inodes per group:         8192
>> >> Inode blocks per group:   512
>> >> Flex block group size:    16
>> >> Filesystem created:       Sat May 25 14:59:50 2013
>> >> Last mount time:          Sat Aug 24 11:04:25 2013
>> >> Last write time:          Tue Sep 24 13:55:36 2013
>> >> Mount count:              0
>> >> Maximum mount count:      -1
>> >> Last checked:             Sat Aug 24 16:56:09 2013
>> >> Check interval:           0 (<none>)
>> >> Lifetime writes:          107 GB
>> >> Reserved blocks uid:      0 (user root)
>> >> Reserved blocks gid:      0 (group root)
>> >> First inode:              11
>> >> Inode size:               256
>> >> Required extra isize:     28
>> >> Desired extra isize:      28
>> >> Journal inode:            8
>> >> Default directory hash:   half_md4
>> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
>> >> Journal backup:           inode blocks
>> >> FS Error count:           8
>> >> First error time:         Sat Aug 24 13:44:55 2013
>> >> First error function:     ext4_iget
>> >> First error line #:       3889
>> >> First error inode #:      8
>> >> First error block #:      0
>> >> Last error time:          Tue Sep 24 13:55:36 2013
>> >> Last error function:      ext4_iget
>> >> Last error line #:        3888
>> >> Last error inode #:       8
>> >> Last error block #:       0
>> >> dumpe2fs: Corrupt extent header while reading journal super block
>> >   OK, so really journal inode (inode #8) looks toast but superblock looks
>> > OK.
>> >
>> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
>> >> and currently I am having more problems with the cloned drive than I
>> >> am with the original.
>> >>
>> >> sandeen said something about using tune2fs to tell it to remove the
>> >> has_journal flag, but I might need some assistance with that.
>> >   Yes, you can do that with:
>> > tune2fs -f -O ^has_journal /dev/sda1
>> >
>> >   Let's see what mount will say after that.
>> >
>> >   Another option is to run
>> > debugfs /dev/sda1
>> >
>> >   Then you can use ls, cd, and other debugfs commands to move within the
>> > filesystem and investigate things. If that will work, you have a reasonable
>> > chance of getting at least some data back.
>> >
>> >                                                                 Honza
>> > --
>> > Jan Kara <jack@suse.cz>
>> > SUSE Labs, CR
>>
>>
>> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
>> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
>> >> So long story short, I had a server running that had a processor fail
>> >> while powered on, causing the file systems to become corrupt. I
>> >> replaced the motherboard, processor and power supply just to be on the
>> >> safe side. However, I am at a bit of a loss as to what to do now. I
>> >> was working sandeen in the OFTC IRC channel, but, on his
>> >> recommendation he suggested me to post something to the mailing list.
>> >>
>> >> Lets start off with one drive at a time (I have 4 that are corrupt).
>> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
>> >> card.
>> >> If I try to mount this using:
>> >> mount -t ext4 /dev/sda1 /media/tmp
>> >>
>> >> It complains in dmesg with the following output:
>> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
>> >> comm mount: bad extra_isize (18013 != 256)
>> >> [685621.845213] EXT4-fs (sda1): no journal found
>> >>
>> >>
>> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
>> >> root@server:~# dumpe2fs -f /dev/sda1
>> >> dumpe2fs 1.42.5 (29-Jul-2012)
>> >> Filesystem volume name:   root
>> >> Last mounted on:          /media/ubuntu/root
>> >> Filesystem UUID:          f959e195-[removed]
>> >> Filesystem magic number:  0xEF53
>> >> Filesystem revision #:    1 (dynamic)
>> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
>> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
>> >> dir_nlink extra_isize
>> >> Filesystem flags:         signed_directory_hash
>> >> Default mount options:    user_xattr acl
>> >> Filesystem state:         not clean with errors
>> >> Errors behavior:          Continue
>> >> Filesystem OS type:       Linux
>> >> Inode count:              4849664
>> >> Block count:              19398144
>> >> Reserved block count:     969907
>> >> Free blocks:              17034219
>> >> Free inodes:              4592929
>> >> First block:              0
>> >> Block size:               4096
>> >> Fragment size:            4096
>> >> Reserved GDT blocks:      1019
>> >> Blocks per group:         32768
>> >> Fragments per group:      32768
>> >> Inodes per group:         8192
>> >> Inode blocks per group:   512
>> >> Flex block group size:    16
>> >> Filesystem created:       Sat May 25 14:59:50 2013
>> >> Last mount time:          Sat Aug 24 11:04:25 2013
>> >> Last write time:          Tue Sep 24 13:55:36 2013
>> >> Mount count:              0
>> >> Maximum mount count:      -1
>> >> Last checked:             Sat Aug 24 16:56:09 2013
>> >> Check interval:           0 (<none>)
>> >> Lifetime writes:          107 GB
>> >> Reserved blocks uid:      0 (user root)
>> >> Reserved blocks gid:      0 (group root)
>> >> First inode:              11
>> >> Inode size:               256
>> >> Required extra isize:     28
>> >> Desired extra isize:      28
>> >> Journal inode:            8
>> >> Default directory hash:   half_md4
>> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
>> >> Journal backup:           inode blocks
>> >> FS Error count:           8
>> >> First error time:         Sat Aug 24 13:44:55 2013
>> >> First error function:     ext4_iget
>> >> First error line #:       3889
>> >> First error inode #:      8
>> >> First error block #:      0
>> >> Last error time:          Tue Sep 24 13:55:36 2013
>> >> Last error function:      ext4_iget
>> >> Last error line #:        3888
>> >> Last error inode #:       8
>> >> Last error block #:       0
>> >> dumpe2fs: Corrupt extent header while reading journal super block
>> >   OK, so really journal inode (inode #8) looks toast but superblock looks
>> > OK.
>> >
>> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
>> >> and currently I am having more problems with the cloned drive than I
>> >> am with the original.
>> >>
>> >> sandeen said something about using tune2fs to tell it to remove the
>> >> has_journal flag, but I might need some assistance with that.
>> >   Yes, you can do that with:
>> > tune2fs -f -O ^has_journal /dev/sda1
>> >
>> >   Let's see what mount will say after that.
>> >
>> >   Another option is to run
>> > debugfs /dev/sda1
>> >
>> >   Then you can use ls, cd, and other debugfs commands to move within the
>> > filesystem and investigate things. If that will work, you have a reasonable
>> > chance of getting at least some data back.
>> >
>> >                                                                 Honza
>> > --
>> > Jan Kara <jack@suse.cz>
>> > SUSE Labs, CR
> --
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[Eric Sandeen]

On 9/25/13 7:08 PM, InvTraySts wrote:
> (Going to merge these two back, because both of you are actually
> helping me, but with the two conversations being segregated like this
> it makes it hard to correlate with you both)
> The partprobe did work with getting the partition table reread.
> After that, the tune2fs sorta worked.
> root@server:~# tune2fs -f -O ^has_journal /dev/sdf1
> tune2fs 1.42.5 (29-Jul-2012)
> root@server:~# mount /dev/sdf1 /media/tmp
> root@server:~# ls -l /media/tmp/
> total 0
> 
> When I try and use the debugfs /dev/sdf1
> 
> root@server:~# debugfs /dev/sdf1
> debugfs 1.42.5 (29-Jul-2012)
> debugfs:  ls
> EXT2 directory corrupted
> debugfs:  ls /
> /: EXT2 directory corrupted

At this point you could try e2fsck on the _copy_, to see what it
can do.

Seems like something quite bad may have happened to your storage,
though, and e2fsck may not be magical for you.

-Eric

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[InvTraySts]

After running e2fsck on the copied drive,
e2fsck -y /dev/sdf1
[long output]
Illegal double indirect block (1162627398) in inode 792606.  CLEARED.
Inode 792606 is too big.  Truncate? yes

Block #2206781 (1047) causes directory to be too big.  CLEARED.
Error storing directory block information (inode=792606, block=0,
num=818574): Memory allocation failed

root: ***** FILE SYSTEM WAS MODIFIED *****
e2fsck: aborted

root: ***** FILE SYSTEM WAS MODIFIED *****
root@server:~#

mount reports an error:
[793486.478664] EXT4-fs error (device sdf1):
htree_dirblock_to_tree:892: inode #2: block 20037: comm ls: bad entry
in directory: inode out of bounds - offset=0(0), inode=1162627398,
rec_len=48, name_len=3
[799147.110694] EXT4-fs (sdf1): ext4_check_descriptors: Checksum for
group 0 failed (32817!=0)
[799147.110698] EXT4-fs (sdf1): group descriptors corrupted!

On Wed, Sep 25, 2013 at 8:59 PM, Eric Sandeen <sandeen@redhat.com> wrote:
> On 9/25/13 7:08 PM, InvTraySts wrote:
>> (Going to merge these two back, because both of you are actually
>> helping me, but with the two conversations being segregated like this
>> it makes it hard to correlate with you both)
>> The partprobe did work with getting the partition table reread.
>> After that, the tune2fs sorta worked.
>> root@server:~# tune2fs -f -O ^has_journal /dev/sdf1
>> tune2fs 1.42.5 (29-Jul-2012)
>> root@server:~# mount /dev/sdf1 /media/tmp
>> root@server:~# ls -l /media/tmp/
>> total 0
>>
>> When I try and use the debugfs /dev/sdf1
>>
>> root@server:~# debugfs /dev/sdf1
>> debugfs 1.42.5 (29-Jul-2012)
>> debugfs:  ls
>> EXT2 directory corrupted
>> debugfs:  ls /
>> /: EXT2 directory corrupted
>
> At this point you could try e2fsck on the _copy_, to see what it
> can do.
>
> Seems like something quite bad may have happened to your storage,
> though, and e2fsck may not be magical for you.
>
> -Eric
>

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[Jan Kara]

On Wed 25-09-13 20:08:38, InvTraySts wrote:
> (Going to merge these two back, because both of you are actually
> helping me, but with the two conversations being segregated like this
> it makes it hard to correlate with you both)
> The partprobe did work with getting the partition table reread.
> After that, the tune2fs sorta worked.
> root@server:~# tune2fs -f -O ^has_journal /dev/sdf1
> tune2fs 1.42.5 (29-Jul-2012)
> root@server:~# mount /dev/sdf1 /media/tmp
> root@server:~# ls -l /media/tmp/
> total 0
> 
> When I try and use the debugfs /dev/sdf1
> 
> root@server:~# debugfs /dev/sdf1
> debugfs 1.42.5 (29-Jul-2012)
> debugfs:  ls
> EXT2 directory corrupted
> debugfs:  ls /
> /: EXT2 directory corrupted
> 
> The drive is supposed to be ext4.
  'EXT2' in the message doesn't mean anything. It is always there. But we
see that even root directory is corrupted. That's bad. You can try what
'stat /' says in debugfs? Possibly also run 'dump / <somefile>' to dump raw
directory data to <somefile> and attach the file here. But also given the
output of e2fsck I'm afraid there won't be much to save on the drive...

								Honza

> root@server:~# dumpe2fs /dev/sdf1
> dumpe2fs 1.42.5 (29-Jul-2012)
> Filesystem volume name:   root
> Last mounted on:          /media/ubuntu/root
> Filesystem UUID:         removed
> Filesystem magic number:  0xEF53
> Filesystem revision #:    1 (dynamic)
> Filesystem features:      ext_attr resize_inode dir_index filetype
> extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink
> extra_isize
> Filesystem flags:         signed_directory_hash
> Default mount options:    user_xattr acl
> Filesystem state:         not clean with errors
> Errors behavior:          Continue
> Filesystem OS type:       Linux
> Inode count:              4849664
> Block count:              19398144
> Reserved block count:     969907
> Free blocks:              17066988
> Free inodes:              4592929
> First block:              0
> Block size:               4096
> Fragment size:            4096
> Reserved GDT blocks:      1019
> Blocks per group:         32768
> Fragments per group:      32768
> Inodes per group:         8192
> Inode blocks per group:   512
> Flex block group size:    16
> Filesystem created:       Sat May 25 14:59:50 2013
> Last mount time:          Wed Sep 25 19:59:07 2013
> Last write time:          Wed Sep 25 19:59:51 2013
> Mount count:              1
> Maximum mount count:      -1
> Last checked:             Sat Aug 24 16:56:09 2013
> Check interval:           0 (<none>)
> Lifetime writes:          107 GB
> Reserved blocks uid:      0 (user root)
> Reserved blocks gid:      0 (group root)
> First inode:              11
> Inode size:               256
> Required extra isize:     28
> Desired extra isize:      28
> Default directory hash:   half_md4
> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> Journal backup:           inode blocks
> FS Error count:           9
> First error time:         Sat Aug 24 13:44:55 2013
> First error function:     ext4_iget
> First error line #:       3889
> First error inode #:      8
> First error block #:      0
> Last error time:          Wed Sep 25 19:59:12 2013
> Last error function:      htree_dirblock_to_tree
> Last error line #:        892
> Last error inode #:       2
> Last error block #:       20037
> 
> [...]
> [output scrolls very fast, and is very large]
> Group 10: (Blocks 327680-360447) [ITABLE_ZEROED]
>   Checksum 0xecaa, unused inodes 0
>   Block bitmap at 1035 (bg #0 + 1035), Inode bitmap at 1051 (bg #0 + 1051)
>   Inode table at 6177-6688 (bg #0 + 6177)
>   0 free blocks, 16 free inodes, 0 directories
>   Free blocks: 327680, 327683-327685, 327687, 327689-327690,
> 327692-327693, 327695-327697, 327700-327701, 327703, 327705,
> 327707-327709, 327711-327715, 327718-327727, 327730-327808,
> 327810-327823, 327825-327842, 327846-327855, 327857-327875, 327878,
> 327881
> 
> (the total amount of pages that the dumpe2fs comes out with is about
> 240 pages. something that can't be pasted. If attachments would be
> required, I can attach a file with it).
> 
> On Wed, Sep 25, 2013 at 5:28 PM, Jan Kara <jack@suse.cz> wrote:
> > On Wed 25-09-13 15:24:34, InvTraySts wrote:
> >> And am cloning the drive without the sync parameter this time.
> >> root@server:~# dd if=/dev/sda of=/dev/sdf bs=4096 conv=notrunc,noerror
> >> After it finished, I attempted to run dumpe2fs and it still responds with:
> >> root@server:~# dumpe2fs /dev/sdf1
> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> dumpe2fs: Bad magic number in super-block while trying to open /dev/sdf1
> >> Couldn't find valid filesystem superblock.
> >   Well, that's likely because the partition table on /dev/sdf didn't get
> > reread. You can run 'partprobe /dev/sdf' to tell the kernel about the new
> > partition table.
> >
> >                                                                 Honza
> >
> >> So I went ahead and tried to run the tune2fs command:
> >> root@server:~# tune2fs -f -O ^has_journal /dev/sda1
> >> tune2fs 1.42.5 (29-Jul-2012)
> >> tune2fs: Bad magic number in super-block while trying to open /dev/sda1
> >> Couldn't find valid filesystem superblock.
> >>
> >> Which also fails, yet dumpe2fs on /dev/sda1 works fine.
> >>
> >>
> >> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
> >> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
> >> >> So long story short, I had a server running that had a processor fail
> >> >> while powered on, causing the file systems to become corrupt. I
> >> >> replaced the motherboard, processor and power supply just to be on the
> >> >> safe side. However, I am at a bit of a loss as to what to do now. I
> >> >> was working sandeen in the OFTC IRC channel, but, on his
> >> >> recommendation he suggested me to post something to the mailing list.
> >> >>
> >> >> Lets start off with one drive at a time (I have 4 that are corrupt).
> >> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
> >> >> card.
> >> >> If I try to mount this using:
> >> >> mount -t ext4 /dev/sda1 /media/tmp
> >> >>
> >> >> It complains in dmesg with the following output:
> >> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
> >> >> comm mount: bad extra_isize (18013 != 256)
> >> >> [685621.845213] EXT4-fs (sda1): no journal found
> >> >>
> >> >>
> >> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
> >> >> root@server:~# dumpe2fs -f /dev/sda1
> >> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> >> Filesystem volume name:   root
> >> >> Last mounted on:          /media/ubuntu/root
> >> >> Filesystem UUID:          f959e195-[removed]
> >> >> Filesystem magic number:  0xEF53
> >> >> Filesystem revision #:    1 (dynamic)
> >> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
> >> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
> >> >> dir_nlink extra_isize
> >> >> Filesystem flags:         signed_directory_hash
> >> >> Default mount options:    user_xattr acl
> >> >> Filesystem state:         not clean with errors
> >> >> Errors behavior:          Continue
> >> >> Filesystem OS type:       Linux
> >> >> Inode count:              4849664
> >> >> Block count:              19398144
> >> >> Reserved block count:     969907
> >> >> Free blocks:              17034219
> >> >> Free inodes:              4592929
> >> >> First block:              0
> >> >> Block size:               4096
> >> >> Fragment size:            4096
> >> >> Reserved GDT blocks:      1019
> >> >> Blocks per group:         32768
> >> >> Fragments per group:      32768
> >> >> Inodes per group:         8192
> >> >> Inode blocks per group:   512
> >> >> Flex block group size:    16
> >> >> Filesystem created:       Sat May 25 14:59:50 2013
> >> >> Last mount time:          Sat Aug 24 11:04:25 2013
> >> >> Last write time:          Tue Sep 24 13:55:36 2013
> >> >> Mount count:              0
> >> >> Maximum mount count:      -1
> >> >> Last checked:             Sat Aug 24 16:56:09 2013
> >> >> Check interval:           0 (<none>)
> >> >> Lifetime writes:          107 GB
> >> >> Reserved blocks uid:      0 (user root)
> >> >> Reserved blocks gid:      0 (group root)
> >> >> First inode:              11
> >> >> Inode size:               256
> >> >> Required extra isize:     28
> >> >> Desired extra isize:      28
> >> >> Journal inode:            8
> >> >> Default directory hash:   half_md4
> >> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> >> >> Journal backup:           inode blocks
> >> >> FS Error count:           8
> >> >> First error time:         Sat Aug 24 13:44:55 2013
> >> >> First error function:     ext4_iget
> >> >> First error line #:       3889
> >> >> First error inode #:      8
> >> >> First error block #:      0
> >> >> Last error time:          Tue Sep 24 13:55:36 2013
> >> >> Last error function:      ext4_iget
> >> >> Last error line #:        3888
> >> >> Last error inode #:       8
> >> >> Last error block #:       0
> >> >> dumpe2fs: Corrupt extent header while reading journal super block
> >> >   OK, so really journal inode (inode #8) looks toast but superblock looks
> >> > OK.
> >> >
> >> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
> >> >> and currently I am having more problems with the cloned drive than I
> >> >> am with the original.
> >> >>
> >> >> sandeen said something about using tune2fs to tell it to remove the
> >> >> has_journal flag, but I might need some assistance with that.
> >> >   Yes, you can do that with:
> >> > tune2fs -f -O ^has_journal /dev/sda1
> >> >
> >> >   Let's see what mount will say after that.
> >> >
> >> >   Another option is to run
> >> > debugfs /dev/sda1
> >> >
> >> >   Then you can use ls, cd, and other debugfs commands to move within the
> >> > filesystem and investigate things. If that will work, you have a reasonable
> >> > chance of getting at least some data back.
> >> >
> >> >                                                                 Honza
> >> > --
> >> > Jan Kara <jack@suse.cz>
> >> > SUSE Labs, CR
> >>
> >>
> >> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
> >> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
> >> >> So long story short, I had a server running that had a processor fail
> >> >> while powered on, causing the file systems to become corrupt. I
> >> >> replaced the motherboard, processor and power supply just to be on the
> >> >> safe side. However, I am at a bit of a loss as to what to do now. I
> >> >> was working sandeen in the OFTC IRC channel, but, on his
> >> >> recommendation he suggested me to post something to the mailing list.
> >> >>
> >> >> Lets start off with one drive at a time (I have 4 that are corrupt).
> >> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
> >> >> card.
> >> >> If I try to mount this using:
> >> >> mount -t ext4 /dev/sda1 /media/tmp
> >> >>
> >> >> It complains in dmesg with the following output:
> >> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
> >> >> comm mount: bad extra_isize (18013 != 256)
> >> >> [685621.845213] EXT4-fs (sda1): no journal found
> >> >>
> >> >>
> >> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
> >> >> root@server:~# dumpe2fs -f /dev/sda1
> >> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> >> Filesystem volume name:   root
> >> >> Last mounted on:          /media/ubuntu/root
> >> >> Filesystem UUID:          f959e195-[removed]
> >> >> Filesystem magic number:  0xEF53
> >> >> Filesystem revision #:    1 (dynamic)
> >> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
> >> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
> >> >> dir_nlink extra_isize
> >> >> Filesystem flags:         signed_directory_hash
> >> >> Default mount options:    user_xattr acl
> >> >> Filesystem state:         not clean with errors
> >> >> Errors behavior:          Continue
> >> >> Filesystem OS type:       Linux
> >> >> Inode count:              4849664
> >> >> Block count:              19398144
> >> >> Reserved block count:     969907
> >> >> Free blocks:              17034219
> >> >> Free inodes:              4592929
> >> >> First block:              0
> >> >> Block size:               4096
> >> >> Fragment size:            4096
> >> >> Reserved GDT blocks:      1019
> >> >> Blocks per group:         32768
> >> >> Fragments per group:      32768
> >> >> Inodes per group:         8192
> >> >> Inode blocks per group:   512
> >> >> Flex block group size:    16
> >> >> Filesystem created:       Sat May 25 14:59:50 2013
> >> >> Last mount time:          Sat Aug 24 11:04:25 2013
> >> >> Last write time:          Tue Sep 24 13:55:36 2013
> >> >> Mount count:              0
> >> >> Maximum mount count:      -1
> >> >> Last checked:             Sat Aug 24 16:56:09 2013
> >> >> Check interval:           0 (<none>)
> >> >> Lifetime writes:          107 GB
> >> >> Reserved blocks uid:      0 (user root)
> >> >> Reserved blocks gid:      0 (group root)
> >> >> First inode:              11
> >> >> Inode size:               256
> >> >> Required extra isize:     28
> >> >> Desired extra isize:      28
> >> >> Journal inode:            8
> >> >> Default directory hash:   half_md4
> >> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> >> >> Journal backup:           inode blocks
> >> >> FS Error count:           8
> >> >> First error time:         Sat Aug 24 13:44:55 2013
> >> >> First error function:     ext4_iget
> >> >> First error line #:       3889
> >> >> First error inode #:      8
> >> >> First error block #:      0
> >> >> Last error time:          Tue Sep 24 13:55:36 2013
> >> >> Last error function:      ext4_iget
> >> >> Last error line #:        3888
> >> >> Last error inode #:       8
> >> >> Last error block #:       0
> >> >> dumpe2fs: Corrupt extent header while reading journal super block
> >> >   OK, so really journal inode (inode #8) looks toast but superblock looks
> >> > OK.
> >> >
> >> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
> >> >> and currently I am having more problems with the cloned drive than I
> >> >> am with the original.
> >> >>
> >> >> sandeen said something about using tune2fs to tell it to remove the
> >> >> has_journal flag, but I might need some assistance with that.
> >> >   Yes, you can do that with:
> >> > tune2fs -f -O ^has_journal /dev/sda1
> >> >
> >> >   Let's see what mount will say after that.
> >> >
> >> >   Another option is to run
> >> > debugfs /dev/sda1
> >> >
> >> >   Then you can use ls, cd, and other debugfs commands to move within the
> >> > filesystem and investigate things. If that will work, you have a reasonable
> >> > chance of getting at least some data back.
> >> >
> >> >                                                                 Honza
> >> > --
> >> > Jan Kara <jack@suse.cz>
> >> > SUSE Labs, CR
> > --
> > Jan Kara <jack@suse.cz>
> > SUSE Labs, CR
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[InvTraySts]

root@server:~# debugfs /dev/sdf1
debugfs 1.42.5 (29-Jul-2012)
debugfs:  stat /
stat: A block group is missing an inode table while reading inode 2
debugfs:  dump /etc/
dump: Usage: dump_inode [-p] <file> <output_file>
debugfs:  dump /etc/passwd /root/passwd.recovery
/etc/passwd: A block group is missing an inode table
debugfs:  ls /etc/
/etc/: A block group is missing an inode table

So now I am getting a different error message, even though literally
nothing has changed other than the time I ran the command. I figured
the passwd file would be the easiest thing to recover, since I don't
remember the exact paths for other application configurations without
doing some digging for the paths and such.

At one point, before the mailing list and me joining the #ext channel,
I was able to see that there was a lost and found folder, I just
couldn't navigate it. So, I am not sure why that has changed either. I
assume logical corruption to the extent it has could cause almost
anything.




On Thu, Sep 26, 2013 at 4:53 AM, Jan Kara <jack@suse.cz> wrote:
> On Wed 25-09-13 20:08:38, InvTraySts wrote:
>> (Going to merge these two back, because both of you are actually
>> helping me, but with the two conversations being segregated like this
>> it makes it hard to correlate with you both)
>> The partprobe did work with getting the partition table reread.
>> After that, the tune2fs sorta worked.
>> root@server:~# tune2fs -f -O ^has_journal /dev/sdf1
>> tune2fs 1.42.5 (29-Jul-2012)
>> root@server:~# mount /dev/sdf1 /media/tmp
>> root@server:~# ls -l /media/tmp/
>> total 0
>>
>> When I try and use the debugfs /dev/sdf1
>>
>> root@server:~# debugfs /dev/sdf1
>> debugfs 1.42.5 (29-Jul-2012)
>> debugfs:  ls
>> EXT2 directory corrupted
>> debugfs:  ls /
>> /: EXT2 directory corrupted
>>
>> The drive is supposed to be ext4.
>   'EXT2' in the message doesn't mean anything. It is always there. But we
> see that even root directory is corrupted. That's bad. You can try what
> 'stat /' says in debugfs? Possibly also run 'dump / <somefile>' to dump raw
> directory data to <somefile> and attach the file here. But also given the
> output of e2fsck I'm afraid there won't be much to save on the drive...
>
>                                                                 Honza
>
>> root@server:~# dumpe2fs /dev/sdf1
>> dumpe2fs 1.42.5 (29-Jul-2012)
>> Filesystem volume name:   root
>> Last mounted on:          /media/ubuntu/root
>> Filesystem UUID:         removed
>> Filesystem magic number:  0xEF53
>> Filesystem revision #:    1 (dynamic)
>> Filesystem features:      ext_attr resize_inode dir_index filetype
>> extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink
>> extra_isize
>> Filesystem flags:         signed_directory_hash
>> Default mount options:    user_xattr acl
>> Filesystem state:         not clean with errors
>> Errors behavior:          Continue
>> Filesystem OS type:       Linux
>> Inode count:              4849664
>> Block count:              19398144
>> Reserved block count:     969907
>> Free blocks:              17066988
>> Free inodes:              4592929
>> First block:              0
>> Block size:               4096
>> Fragment size:            4096
>> Reserved GDT blocks:      1019
>> Blocks per group:         32768
>> Fragments per group:      32768
>> Inodes per group:         8192
>> Inode blocks per group:   512
>> Flex block group size:    16
>> Filesystem created:       Sat May 25 14:59:50 2013
>> Last mount time:          Wed Sep 25 19:59:07 2013
>> Last write time:          Wed Sep 25 19:59:51 2013
>> Mount count:              1
>> Maximum mount count:      -1
>> Last checked:             Sat Aug 24 16:56:09 2013
>> Check interval:           0 (<none>)
>> Lifetime writes:          107 GB
>> Reserved blocks uid:      0 (user root)
>> Reserved blocks gid:      0 (group root)
>> First inode:              11
>> Inode size:               256
>> Required extra isize:     28
>> Desired extra isize:      28
>> Default directory hash:   half_md4
>> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
>> Journal backup:           inode blocks
>> FS Error count:           9
>> First error time:         Sat Aug 24 13:44:55 2013
>> First error function:     ext4_iget
>> First error line #:       3889
>> First error inode #:      8
>> First error block #:      0
>> Last error time:          Wed Sep 25 19:59:12 2013
>> Last error function:      htree_dirblock_to_tree
>> Last error line #:        892
>> Last error inode #:       2
>> Last error block #:       20037
>>
>> [...]
>> [output scrolls very fast, and is very large]
>> Group 10: (Blocks 327680-360447) [ITABLE_ZEROED]
>>   Checksum 0xecaa, unused inodes 0
>>   Block bitmap at 1035 (bg #0 + 1035), Inode bitmap at 1051 (bg #0 + 1051)
>>   Inode table at 6177-6688 (bg #0 + 6177)
>>   0 free blocks, 16 free inodes, 0 directories
>>   Free blocks: 327680, 327683-327685, 327687, 327689-327690,
>> 327692-327693, 327695-327697, 327700-327701, 327703, 327705,
>> 327707-327709, 327711-327715, 327718-327727, 327730-327808,
>> 327810-327823, 327825-327842, 327846-327855, 327857-327875, 327878,
>> 327881
>>
>> (the total amount of pages that the dumpe2fs comes out with is about
>> 240 pages. something that can't be pasted. If attachments would be
>> required, I can attach a file with it).
>>
>> On Wed, Sep 25, 2013 at 5:28 PM, Jan Kara <jack@suse.cz> wrote:
>> > On Wed 25-09-13 15:24:34, InvTraySts wrote:
>> >> And am cloning the drive without the sync parameter this time.
>> >> root@server:~# dd if=/dev/sda of=/dev/sdf bs=4096 conv=notrunc,noerror
>> >> After it finished, I attempted to run dumpe2fs and it still responds with:
>> >> root@server:~# dumpe2fs /dev/sdf1
>> >> dumpe2fs 1.42.5 (29-Jul-2012)
>> >> dumpe2fs: Bad magic number in super-block while trying to open /dev/sdf1
>> >> Couldn't find valid filesystem superblock.
>> >   Well, that's likely because the partition table on /dev/sdf didn't get
>> > reread. You can run 'partprobe /dev/sdf' to tell the kernel about the new
>> > partition table.
>> >
>> >                                                                 Honza
>> >
>> >> So I went ahead and tried to run the tune2fs command:
>> >> root@server:~# tune2fs -f -O ^has_journal /dev/sda1
>> >> tune2fs 1.42.5 (29-Jul-2012)
>> >> tune2fs: Bad magic number in super-block while trying to open /dev/sda1
>> >> Couldn't find valid filesystem superblock.
>> >>
>> >> Which also fails, yet dumpe2fs on /dev/sda1 works fine.
>> >>
>> >>
>> >> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
>> >> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
>> >> >> So long story short, I had a server running that had a processor fail
>> >> >> while powered on, causing the file systems to become corrupt. I
>> >> >> replaced the motherboard, processor and power supply just to be on the
>> >> >> safe side. However, I am at a bit of a loss as to what to do now. I
>> >> >> was working sandeen in the OFTC IRC channel, but, on his
>> >> >> recommendation he suggested me to post something to the mailing list.
>> >> >>
>> >> >> Lets start off with one drive at a time (I have 4 that are corrupt).
>> >> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
>> >> >> card.
>> >> >> If I try to mount this using:
>> >> >> mount -t ext4 /dev/sda1 /media/tmp
>> >> >>
>> >> >> It complains in dmesg with the following output:
>> >> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
>> >> >> comm mount: bad extra_isize (18013 != 256)
>> >> >> [685621.845213] EXT4-fs (sda1): no journal found
>> >> >>
>> >> >>
>> >> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
>> >> >> root@server:~# dumpe2fs -f /dev/sda1
>> >> >> dumpe2fs 1.42.5 (29-Jul-2012)
>> >> >> Filesystem volume name:   root
>> >> >> Last mounted on:          /media/ubuntu/root
>> >> >> Filesystem UUID:          f959e195-[removed]
>> >> >> Filesystem magic number:  0xEF53
>> >> >> Filesystem revision #:    1 (dynamic)
>> >> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
>> >> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
>> >> >> dir_nlink extra_isize
>> >> >> Filesystem flags:         signed_directory_hash
>> >> >> Default mount options:    user_xattr acl
>> >> >> Filesystem state:         not clean with errors
>> >> >> Errors behavior:          Continue
>> >> >> Filesystem OS type:       Linux
>> >> >> Inode count:              4849664
>> >> >> Block count:              19398144
>> >> >> Reserved block count:     969907
>> >> >> Free blocks:              17034219
>> >> >> Free inodes:              4592929
>> >> >> First block:              0
>> >> >> Block size:               4096
>> >> >> Fragment size:            4096
>> >> >> Reserved GDT blocks:      1019
>> >> >> Blocks per group:         32768
>> >> >> Fragments per group:      32768
>> >> >> Inodes per group:         8192
>> >> >> Inode blocks per group:   512
>> >> >> Flex block group size:    16
>> >> >> Filesystem created:       Sat May 25 14:59:50 2013
>> >> >> Last mount time:          Sat Aug 24 11:04:25 2013
>> >> >> Last write time:          Tue Sep 24 13:55:36 2013
>> >> >> Mount count:              0
>> >> >> Maximum mount count:      -1
>> >> >> Last checked:             Sat Aug 24 16:56:09 2013
>> >> >> Check interval:           0 (<none>)
>> >> >> Lifetime writes:          107 GB
>> >> >> Reserved blocks uid:      0 (user root)
>> >> >> Reserved blocks gid:      0 (group root)
>> >> >> First inode:              11
>> >> >> Inode size:               256
>> >> >> Required extra isize:     28
>> >> >> Desired extra isize:      28
>> >> >> Journal inode:            8
>> >> >> Default directory hash:   half_md4
>> >> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
>> >> >> Journal backup:           inode blocks
>> >> >> FS Error count:           8
>> >> >> First error time:         Sat Aug 24 13:44:55 2013
>> >> >> First error function:     ext4_iget
>> >> >> First error line #:       3889
>> >> >> First error inode #:      8
>> >> >> First error block #:      0
>> >> >> Last error time:          Tue Sep 24 13:55:36 2013
>> >> >> Last error function:      ext4_iget
>> >> >> Last error line #:        3888
>> >> >> Last error inode #:       8
>> >> >> Last error block #:       0
>> >> >> dumpe2fs: Corrupt extent header while reading journal super block
>> >> >   OK, so really journal inode (inode #8) looks toast but superblock looks
>> >> > OK.
>> >> >
>> >> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
>> >> >> and currently I am having more problems with the cloned drive than I
>> >> >> am with the original.
>> >> >>
>> >> >> sandeen said something about using tune2fs to tell it to remove the
>> >> >> has_journal flag, but I might need some assistance with that.
>> >> >   Yes, you can do that with:
>> >> > tune2fs -f -O ^has_journal /dev/sda1
>> >> >
>> >> >   Let's see what mount will say after that.
>> >> >
>> >> >   Another option is to run
>> >> > debugfs /dev/sda1
>> >> >
>> >> >   Then you can use ls, cd, and other debugfs commands to move within the
>> >> > filesystem and investigate things. If that will work, you have a reasonable
>> >> > chance of getting at least some data back.
>> >> >
>> >> >                                                                 Honza
>> >> > --
>> >> > Jan Kara <jack@suse.cz>
>> >> > SUSE Labs, CR
>> >>
>> >>
>> >> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
>> >> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
>> >> >> So long story short, I had a server running that had a processor fail
>> >> >> while powered on, causing the file systems to become corrupt. I
>> >> >> replaced the motherboard, processor and power supply just to be on the
>> >> >> safe side. However, I am at a bit of a loss as to what to do now. I
>> >> >> was working sandeen in the OFTC IRC channel, but, on his
>> >> >> recommendation he suggested me to post something to the mailing list.
>> >> >>
>> >> >> Lets start off with one drive at a time (I have 4 that are corrupt).
>> >> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
>> >> >> card.
>> >> >> If I try to mount this using:
>> >> >> mount -t ext4 /dev/sda1 /media/tmp
>> >> >>
>> >> >> It complains in dmesg with the following output:
>> >> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
>> >> >> comm mount: bad extra_isize (18013 != 256)
>> >> >> [685621.845213] EXT4-fs (sda1): no journal found
>> >> >>
>> >> >>
>> >> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
>> >> >> root@server:~# dumpe2fs -f /dev/sda1
>> >> >> dumpe2fs 1.42.5 (29-Jul-2012)
>> >> >> Filesystem volume name:   root
>> >> >> Last mounted on:          /media/ubuntu/root
>> >> >> Filesystem UUID:          f959e195-[removed]
>> >> >> Filesystem magic number:  0xEF53
>> >> >> Filesystem revision #:    1 (dynamic)
>> >> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
>> >> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
>> >> >> dir_nlink extra_isize
>> >> >> Filesystem flags:         signed_directory_hash
>> >> >> Default mount options:    user_xattr acl
>> >> >> Filesystem state:         not clean with errors
>> >> >> Errors behavior:          Continue
>> >> >> Filesystem OS type:       Linux
>> >> >> Inode count:              4849664
>> >> >> Block count:              19398144
>> >> >> Reserved block count:     969907
>> >> >> Free blocks:              17034219
>> >> >> Free inodes:              4592929
>> >> >> First block:              0
>> >> >> Block size:               4096
>> >> >> Fragment size:            4096
>> >> >> Reserved GDT blocks:      1019
>> >> >> Blocks per group:         32768
>> >> >> Fragments per group:      32768
>> >> >> Inodes per group:         8192
>> >> >> Inode blocks per group:   512
>> >> >> Flex block group size:    16
>> >> >> Filesystem created:       Sat May 25 14:59:50 2013
>> >> >> Last mount time:          Sat Aug 24 11:04:25 2013
>> >> >> Last write time:          Tue Sep 24 13:55:36 2013
>> >> >> Mount count:              0
>> >> >> Maximum mount count:      -1
>> >> >> Last checked:             Sat Aug 24 16:56:09 2013
>> >> >> Check interval:           0 (<none>)
>> >> >> Lifetime writes:          107 GB
>> >> >> Reserved blocks uid:      0 (user root)
>> >> >> Reserved blocks gid:      0 (group root)
>> >> >> First inode:              11
>> >> >> Inode size:               256
>> >> >> Required extra isize:     28
>> >> >> Desired extra isize:      28
>> >> >> Journal inode:            8
>> >> >> Default directory hash:   half_md4
>> >> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
>> >> >> Journal backup:           inode blocks
>> >> >> FS Error count:           8
>> >> >> First error time:         Sat Aug 24 13:44:55 2013
>> >> >> First error function:     ext4_iget
>> >> >> First error line #:       3889
>> >> >> First error inode #:      8
>> >> >> First error block #:      0
>> >> >> Last error time:          Tue Sep 24 13:55:36 2013
>> >> >> Last error function:      ext4_iget
>> >> >> Last error line #:        3888
>> >> >> Last error inode #:       8
>> >> >> Last error block #:       0
>> >> >> dumpe2fs: Corrupt extent header while reading journal super block
>> >> >   OK, so really journal inode (inode #8) looks toast but superblock looks
>> >> > OK.
>> >> >
>> >> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
>> >> >> and currently I am having more problems with the cloned drive than I
>> >> >> am with the original.
>> >> >>
>> >> >> sandeen said something about using tune2fs to tell it to remove the
>> >> >> has_journal flag, but I might need some assistance with that.
>> >> >   Yes, you can do that with:
>> >> > tune2fs -f -O ^has_journal /dev/sda1
>> >> >
>> >> >   Let's see what mount will say after that.
>> >> >
>> >> >   Another option is to run
>> >> > debugfs /dev/sda1
>> >> >
>> >> >   Then you can use ls, cd, and other debugfs commands to move within the
>> >> > filesystem and investigate things. If that will work, you have a reasonable
>> >> > chance of getting at least some data back.
>> >> >
>> >> >                                                                 Honza
>> >> > --
>> >> > Jan Kara <jack@suse.cz>
>> >> > SUSE Labs, CR
>> > --
>> > Jan Kara <jack@suse.cz>
>> > SUSE Labs, CR
> --
> Jan Kara <jack@suse.cz>
> SUSE Labs, CR

Re: Fwd: Need help with Data Recovery on Ext4 partitions that became corrupted on running OS

[Jan Kara]

On Thu 26-09-13 13:45:34, InvTraySts wrote:
> root@server:~# debugfs /dev/sdf1
> debugfs 1.42.5 (29-Jul-2012)
> debugfs:  stat /
> stat: A block group is missing an inode table while reading inode 2
> debugfs:  dump /etc/
> dump: Usage: dump_inode [-p] <file> <output_file>
> debugfs:  dump /etc/passwd /root/passwd.recovery
> /etc/passwd: A block group is missing an inode table
> debugfs:  ls /etc/
> /etc/: A block group is missing an inode table
> 
> So now I am getting a different error message, even though literally
> nothing has changed other than the time I ran the command. I figured
> the passwd file would be the easiest thing to recover, since I don't
> remember the exact paths for other application configurations without
> doing some digging for the paths and such.
> 
> At one point, before the mailing list and me joining the #ext channel,
> I was able to see that there was a lost and found folder, I just
> couldn't navigate it. So, I am not sure why that has changed either. I
> assume logical corruption to the extent it has could cause almost
> anything.
  Isn't the fs already modified from fsck? Anyway, the fs really looks
hopelessly damaged so I'm afraid we won't be able to get any data from it.

								Honza

> On Thu, Sep 26, 2013 at 4:53 AM, Jan Kara <jack@suse.cz> wrote:
> > On Wed 25-09-13 20:08:38, InvTraySts wrote:
> >> (Going to merge these two back, because both of you are actually
> >> helping me, but with the two conversations being segregated like this
> >> it makes it hard to correlate with you both)
> >> The partprobe did work with getting the partition table reread.
> >> After that, the tune2fs sorta worked.
> >> root@server:~# tune2fs -f -O ^has_journal /dev/sdf1
> >> tune2fs 1.42.5 (29-Jul-2012)
> >> root@server:~# mount /dev/sdf1 /media/tmp
> >> root@server:~# ls -l /media/tmp/
> >> total 0
> >>
> >> When I try and use the debugfs /dev/sdf1
> >>
> >> root@server:~# debugfs /dev/sdf1
> >> debugfs 1.42.5 (29-Jul-2012)
> >> debugfs:  ls
> >> EXT2 directory corrupted
> >> debugfs:  ls /
> >> /: EXT2 directory corrupted
> >>
> >> The drive is supposed to be ext4.
> >   'EXT2' in the message doesn't mean anything. It is always there. But we
> > see that even root directory is corrupted. That's bad. You can try what
> > 'stat /' says in debugfs? Possibly also run 'dump / <somefile>' to dump raw
> > directory data to <somefile> and attach the file here. But also given the
> > output of e2fsck I'm afraid there won't be much to save on the drive...
> >
> >                                                                 Honza
> >
> >> root@server:~# dumpe2fs /dev/sdf1
> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> Filesystem volume name:   root
> >> Last mounted on:          /media/ubuntu/root
> >> Filesystem UUID:         removed
> >> Filesystem magic number:  0xEF53
> >> Filesystem revision #:    1 (dynamic)
> >> Filesystem features:      ext_attr resize_inode dir_index filetype
> >> extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink
> >> extra_isize
> >> Filesystem flags:         signed_directory_hash
> >> Default mount options:    user_xattr acl
> >> Filesystem state:         not clean with errors
> >> Errors behavior:          Continue
> >> Filesystem OS type:       Linux
> >> Inode count:              4849664
> >> Block count:              19398144
> >> Reserved block count:     969907
> >> Free blocks:              17066988
> >> Free inodes:              4592929
> >> First block:              0
> >> Block size:               4096
> >> Fragment size:            4096
> >> Reserved GDT blocks:      1019
> >> Blocks per group:         32768
> >> Fragments per group:      32768
> >> Inodes per group:         8192
> >> Inode blocks per group:   512
> >> Flex block group size:    16
> >> Filesystem created:       Sat May 25 14:59:50 2013
> >> Last mount time:          Wed Sep 25 19:59:07 2013
> >> Last write time:          Wed Sep 25 19:59:51 2013
> >> Mount count:              1
> >> Maximum mount count:      -1
> >> Last checked:             Sat Aug 24 16:56:09 2013
> >> Check interval:           0 (<none>)
> >> Lifetime writes:          107 GB
> >> Reserved blocks uid:      0 (user root)
> >> Reserved blocks gid:      0 (group root)
> >> First inode:              11
> >> Inode size:               256
> >> Required extra isize:     28
> >> Desired extra isize:      28
> >> Default directory hash:   half_md4
> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> >> Journal backup:           inode blocks
> >> FS Error count:           9
> >> First error time:         Sat Aug 24 13:44:55 2013
> >> First error function:     ext4_iget
> >> First error line #:       3889
> >> First error inode #:      8
> >> First error block #:      0
> >> Last error time:          Wed Sep 25 19:59:12 2013
> >> Last error function:      htree_dirblock_to_tree
> >> Last error line #:        892
> >> Last error inode #:       2
> >> Last error block #:       20037
> >>
> >> [...]
> >> [output scrolls very fast, and is very large]
> >> Group 10: (Blocks 327680-360447) [ITABLE_ZEROED]
> >>   Checksum 0xecaa, unused inodes 0
> >>   Block bitmap at 1035 (bg #0 + 1035), Inode bitmap at 1051 (bg #0 + 1051)
> >>   Inode table at 6177-6688 (bg #0 + 6177)
> >>   0 free blocks, 16 free inodes, 0 directories
> >>   Free blocks: 327680, 327683-327685, 327687, 327689-327690,
> >> 327692-327693, 327695-327697, 327700-327701, 327703, 327705,
> >> 327707-327709, 327711-327715, 327718-327727, 327730-327808,
> >> 327810-327823, 327825-327842, 327846-327855, 327857-327875, 327878,
> >> 327881
> >>
> >> (the total amount of pages that the dumpe2fs comes out with is about
> >> 240 pages. something that can't be pasted. If attachments would be
> >> required, I can attach a file with it).
> >>
> >> On Wed, Sep 25, 2013 at 5:28 PM, Jan Kara <jack@suse.cz> wrote:
> >> > On Wed 25-09-13 15:24:34, InvTraySts wrote:
> >> >> And am cloning the drive without the sync parameter this time.
> >> >> root@server:~# dd if=/dev/sda of=/dev/sdf bs=4096 conv=notrunc,noerror
> >> >> After it finished, I attempted to run dumpe2fs and it still responds with:
> >> >> root@server:~# dumpe2fs /dev/sdf1
> >> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> >> dumpe2fs: Bad magic number in super-block while trying to open /dev/sdf1
> >> >> Couldn't find valid filesystem superblock.
> >> >   Well, that's likely because the partition table on /dev/sdf didn't get
> >> > reread. You can run 'partprobe /dev/sdf' to tell the kernel about the new
> >> > partition table.
> >> >
> >> >                                                                 Honza
> >> >
> >> >> So I went ahead and tried to run the tune2fs command:
> >> >> root@server:~# tune2fs -f -O ^has_journal /dev/sda1
> >> >> tune2fs 1.42.5 (29-Jul-2012)
> >> >> tune2fs: Bad magic number in super-block while trying to open /dev/sda1
> >> >> Couldn't find valid filesystem superblock.
> >> >>
> >> >> Which also fails, yet dumpe2fs on /dev/sda1 works fine.
> >> >>
> >> >>
> >> >> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
> >> >> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
> >> >> >> So long story short, I had a server running that had a processor fail
> >> >> >> while powered on, causing the file systems to become corrupt. I
> >> >> >> replaced the motherboard, processor and power supply just to be on the
> >> >> >> safe side. However, I am at a bit of a loss as to what to do now. I
> >> >> >> was working sandeen in the OFTC IRC channel, but, on his
> >> >> >> recommendation he suggested me to post something to the mailing list.
> >> >> >>
> >> >> >> Lets start off with one drive at a time (I have 4 that are corrupt).
> >> >> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
> >> >> >> card.
> >> >> >> If I try to mount this using:
> >> >> >> mount -t ext4 /dev/sda1 /media/tmp
> >> >> >>
> >> >> >> It complains in dmesg with the following output:
> >> >> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
> >> >> >> comm mount: bad extra_isize (18013 != 256)
> >> >> >> [685621.845213] EXT4-fs (sda1): no journal found
> >> >> >>
> >> >> >>
> >> >> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
> >> >> >> root@server:~# dumpe2fs -f /dev/sda1
> >> >> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> >> >> Filesystem volume name:   root
> >> >> >> Last mounted on:          /media/ubuntu/root
> >> >> >> Filesystem UUID:          f959e195-[removed]
> >> >> >> Filesystem magic number:  0xEF53
> >> >> >> Filesystem revision #:    1 (dynamic)
> >> >> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
> >> >> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
> >> >> >> dir_nlink extra_isize
> >> >> >> Filesystem flags:         signed_directory_hash
> >> >> >> Default mount options:    user_xattr acl
> >> >> >> Filesystem state:         not clean with errors
> >> >> >> Errors behavior:          Continue
> >> >> >> Filesystem OS type:       Linux
> >> >> >> Inode count:              4849664
> >> >> >> Block count:              19398144
> >> >> >> Reserved block count:     969907
> >> >> >> Free blocks:              17034219
> >> >> >> Free inodes:              4592929
> >> >> >> First block:              0
> >> >> >> Block size:               4096
> >> >> >> Fragment size:            4096
> >> >> >> Reserved GDT blocks:      1019
> >> >> >> Blocks per group:         32768
> >> >> >> Fragments per group:      32768
> >> >> >> Inodes per group:         8192
> >> >> >> Inode blocks per group:   512
> >> >> >> Flex block group size:    16
> >> >> >> Filesystem created:       Sat May 25 14:59:50 2013
> >> >> >> Last mount time:          Sat Aug 24 11:04:25 2013
> >> >> >> Last write time:          Tue Sep 24 13:55:36 2013
> >> >> >> Mount count:              0
> >> >> >> Maximum mount count:      -1
> >> >> >> Last checked:             Sat Aug 24 16:56:09 2013
> >> >> >> Check interval:           0 (<none>)
> >> >> >> Lifetime writes:          107 GB
> >> >> >> Reserved blocks uid:      0 (user root)
> >> >> >> Reserved blocks gid:      0 (group root)
> >> >> >> First inode:              11
> >> >> >> Inode size:               256
> >> >> >> Required extra isize:     28
> >> >> >> Desired extra isize:      28
> >> >> >> Journal inode:            8
> >> >> >> Default directory hash:   half_md4
> >> >> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> >> >> >> Journal backup:           inode blocks
> >> >> >> FS Error count:           8
> >> >> >> First error time:         Sat Aug 24 13:44:55 2013
> >> >> >> First error function:     ext4_iget
> >> >> >> First error line #:       3889
> >> >> >> First error inode #:      8
> >> >> >> First error block #:      0
> >> >> >> Last error time:          Tue Sep 24 13:55:36 2013
> >> >> >> Last error function:      ext4_iget
> >> >> >> Last error line #:        3888
> >> >> >> Last error inode #:       8
> >> >> >> Last error block #:       0
> >> >> >> dumpe2fs: Corrupt extent header while reading journal super block
> >> >> >   OK, so really journal inode (inode #8) looks toast but superblock looks
> >> >> > OK.
> >> >> >
> >> >> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
> >> >> >> and currently I am having more problems with the cloned drive than I
> >> >> >> am with the original.
> >> >> >>
> >> >> >> sandeen said something about using tune2fs to tell it to remove the
> >> >> >> has_journal flag, but I might need some assistance with that.
> >> >> >   Yes, you can do that with:
> >> >> > tune2fs -f -O ^has_journal /dev/sda1
> >> >> >
> >> >> >   Let's see what mount will say after that.
> >> >> >
> >> >> >   Another option is to run
> >> >> > debugfs /dev/sda1
> >> >> >
> >> >> >   Then you can use ls, cd, and other debugfs commands to move within the
> >> >> > filesystem and investigate things. If that will work, you have a reasonable
> >> >> > chance of getting at least some data back.
> >> >> >
> >> >> >                                                                 Honza
> >> >> > --
> >> >> > Jan Kara <jack@suse.cz>
> >> >> > SUSE Labs, CR
> >> >>
> >> >>
> >> >> On Wed, Sep 25, 2013 at 12:12 PM, Jan Kara <jack@suse.cz> wrote:
> >> >> > On Tue 24-09-13 22:25:49, InvTraySts wrote:
> >> >> >> So long story short, I had a server running that had a processor fail
> >> >> >> while powered on, causing the file systems to become corrupt. I
> >> >> >> replaced the motherboard, processor and power supply just to be on the
> >> >> >> safe side. However, I am at a bit of a loss as to what to do now. I
> >> >> >> was working sandeen in the OFTC IRC channel, but, on his
> >> >> >> recommendation he suggested me to post something to the mailing list.
> >> >> >>
> >> >> >> Lets start off with one drive at a time (I have 4 that are corrupt).
> >> >> >> The specific logical drive in question was in RAID1 on a Dell PERC 5/i
> >> >> >> card.
> >> >> >> If I try to mount this using:
> >> >> >> mount -t ext4 /dev/sda1 /media/tmp
> >> >> >>
> >> >> >> It complains in dmesg with the following output:
> >> >> >> 685621.845207] EXT4-fs error (device sda1): ext4_iget:3888: inode #8:
> >> >> >> comm mount: bad extra_isize (18013 != 256)
> >> >> >> [685621.845213] EXT4-fs (sda1): no journal found
> >> >> >>
> >> >> >>
> >> >> >> However, if I run dumpe2fs -f /dev/sda1 I get the following output:
> >> >> >> root@server:~# dumpe2fs -f /dev/sda1
> >> >> >> dumpe2fs 1.42.5 (29-Jul-2012)
> >> >> >> Filesystem volume name:   root
> >> >> >> Last mounted on:          /media/ubuntu/root
> >> >> >> Filesystem UUID:          f959e195-[removed]
> >> >> >> Filesystem magic number:  0xEF53
> >> >> >> Filesystem revision #:    1 (dynamic)
> >> >> >> Filesystem features:      has_journal ext_attr resize_inode dir_index
> >> >> >> filetype extent flex_bg sparse_super large_file huge_file uninit_bg
> >> >> >> dir_nlink extra_isize
> >> >> >> Filesystem flags:         signed_directory_hash
> >> >> >> Default mount options:    user_xattr acl
> >> >> >> Filesystem state:         not clean with errors
> >> >> >> Errors behavior:          Continue
> >> >> >> Filesystem OS type:       Linux
> >> >> >> Inode count:              4849664
> >> >> >> Block count:              19398144
> >> >> >> Reserved block count:     969907
> >> >> >> Free blocks:              17034219
> >> >> >> Free inodes:              4592929
> >> >> >> First block:              0
> >> >> >> Block size:               4096
> >> >> >> Fragment size:            4096
> >> >> >> Reserved GDT blocks:      1019
> >> >> >> Blocks per group:         32768
> >> >> >> Fragments per group:      32768
> >> >> >> Inodes per group:         8192
> >> >> >> Inode blocks per group:   512
> >> >> >> Flex block group size:    16
> >> >> >> Filesystem created:       Sat May 25 14:59:50 2013
> >> >> >> Last mount time:          Sat Aug 24 11:04:25 2013
> >> >> >> Last write time:          Tue Sep 24 13:55:36 2013
> >> >> >> Mount count:              0
> >> >> >> Maximum mount count:      -1
> >> >> >> Last checked:             Sat Aug 24 16:56:09 2013
> >> >> >> Check interval:           0 (<none>)
> >> >> >> Lifetime writes:          107 GB
> >> >> >> Reserved blocks uid:      0 (user root)
> >> >> >> Reserved blocks gid:      0 (group root)
> >> >> >> First inode:              11
> >> >> >> Inode size:               256
> >> >> >> Required extra isize:     28
> >> >> >> Desired extra isize:      28
> >> >> >> Journal inode:            8
> >> >> >> Default directory hash:   half_md4
> >> >> >> Directory Hash Seed:      01a8f605-b2bc-41ee-b7b5-11d843ab622f
> >> >> >> Journal backup:           inode blocks
> >> >> >> FS Error count:           8
> >> >> >> First error time:         Sat Aug 24 13:44:55 2013
> >> >> >> First error function:     ext4_iget
> >> >> >> First error line #:       3889
> >> >> >> First error inode #:      8
> >> >> >> First error block #:      0
> >> >> >> Last error time:          Tue Sep 24 13:55:36 2013
> >> >> >> Last error function:      ext4_iget
> >> >> >> Last error line #:        3888
> >> >> >> Last error inode #:       8
> >> >> >> Last error block #:       0
> >> >> >> dumpe2fs: Corrupt extent header while reading journal super block
> >> >> >   OK, so really journal inode (inode #8) looks toast but superblock looks
> >> >> > OK.
> >> >> >
> >> >> >> So I attempted to clone the drive to a 2TB backup drive that is empty,
> >> >> >> and currently I am having more problems with the cloned drive than I
> >> >> >> am with the original.
> >> >> >>
> >> >> >> sandeen said something about using tune2fs to tell it to remove the
> >> >> >> has_journal flag, but I might need some assistance with that.
> >> >> >   Yes, you can do that with:
> >> >> > tune2fs -f -O ^has_journal /dev/sda1
> >> >> >
> >> >> >   Let's see what mount will say after that.
> >> >> >
> >> >> >   Another option is to run
> >> >> > debugfs /dev/sda1
> >> >> >
> >> >> >   Then you can use ls, cd, and other debugfs commands to move within the
> >> >> > filesystem and investigate things. If that will work, you have a reasonable
> >> >> > chance of getting at least some data back.
> >> >> >
> >> >> >                                                                 Honza
> >> >> > --
> >> >> > Jan Kara <jack@suse.cz>
> >> >> > SUSE Labs, CR
> >> > --
> >> > Jan Kara <jack@suse.cz>
> >> > SUSE Labs, CR
> > --
> > Jan Kara <jack@suse.cz>
> > SUSE Labs, CR
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR