Restoring filenames from partly damaged ext4-filesystem

Restoring filenames from partly damaged ext4-filesystem

[Rudolf Zran]

Hi everybody!

I recently damaged an ext4 partition by accident (mistakenly forced
a RAID sync with another partition onto it, which I realized after
about 3% completion). As a result the beginning of the ext4 partition
seems to be overwritten with garbage and refuses to mount.

As you might guess, I'd now like to get as most of my data back (from
the part which hasn't been overwritten, of course :)).

Maybe somebody knows a good method to just "repair" the ext4-structure
from the remaining part of the partition?


Background information:

Operation system:   Debian GNU/Linux 6.0.4 (Squeeze)
Kernel:             Linux 2.6.32-5
Architecture:       x86-amd64
e2fsprogs:          1.41.12
Filesystem type:    ext4, with default settings (mkfs.ext4 /dev/xyz)
Filesystem size:    Around 1TB, filled with around 800GB of data.
Filesystem content: From a lot of ASCII stuff up to files with several
                    GB in size and arbitrary non-standard type.


What I've tried so far:

* First of all, though the source drive is physically fine, I work on
  images of the partition on a spare drive, to experiment with. Everytime
  I make unrecoverable errors to that image, I recopy the original.

* "dumpe2fs -b $SBERR /dev/loop0" does NOT work, for SBERR={32768, 98304,
  163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000,
  7962624} ( see https://pzt.me/1l55 )

* "dumpe2fs -b $SBOK /dev/loop0" DOES (partly) work, for SBOK={11239424,
  20480000, 23887872, 71663616, 78675968, 214990848} ( see https://pzt.me/6txz )

* "mkfs.ext4 -S /dev/loop0" results in a completely empty filesystem

* "fsck.ext4 -b $SBOK -B 4096 -v -n /dev/loop0" shows a lot of errors.
  Filesystem isn't mountable afterwards ( see https://pzt.me/42yk and
  https://pzt.me/3frg )

* "fsck.ext4 -b $SBOK -B 4096 -v -y /dev/loop0" recoveres after a long time.
  Filesystem is mountable. Root is empty besides lost+found folder, which
  contains about 300GB mostly useless data: Millions of files with wrong
  permissions, useless names and some random content.

* photorec from the testdisk package recoveres, luckily!, about 500GB of
  data. Though the content seems to be pretty reasonable, no filenames
  are recovered, since photorec operates without using filesystem knowledge.

Do you see any chances (besides consulting professional recovery companies)
getting the filenames back? I looked into the ext4 specs a bit to figure out
where this information is stored on disk, but before I step with hexedit
through a terabyte of data, I'd rather try some solutions which are maybe
already out there.

Any help is appretiated.

Thanks in advance, Rudolf.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Restoring filenames from partly damaged ext4-filesystem

[Andreas Dilger]

On 2012-02-09, at 9:29 AM, Rudolf Zran wrote:
> I recently damaged an ext4 partition by accident (mistakenly forced
> a RAID sync with another partition onto it, which I realized after
> about 3% completion). As a result the beginning of the ext4 partition
> seems to be overwritten with garbage and refuses to mount.
> 
> As you might guess, I'd now like to get as most of my data back (from
> the part which hasn't been overwritten, of course :)).
> 
> Maybe somebody knows a good method to just "repair" the ext4-structure
> from the remaining part of the partition?
> 
> 
> Background information:
> 
> Operation system:   Debian GNU/Linux 6.0.4 (Squeeze)
> Kernel:             Linux 2.6.32-5
> Architecture:       x86-amd64
> e2fsprogs:          1.41.12
> Filesystem type:    ext4, with default settings (mkfs.ext4 /dev/xyz)
> Filesystem size:    Around 1TB, filled with around 800GB of data.
> Filesystem content: From a lot of ASCII stuff up to files with several
>                     GB in size and arbitrary non-standard type.
> 
> 
> What I've tried so far:
> 
> * First of all, though the source drive is physically fine, I work on
>   images of the partition on a spare drive, to experiment with. Everytime
>   I make unrecoverable errors to that image, I recopy the original.
> 
> * "dumpe2fs -b $SBERR /dev/loop0" does NOT work, for SBERR={32768, 98304,
>   163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000,
>   7962624} ( see https://pzt.me/1l55 )
> 
> * "dumpe2fs -b $SBOK /dev/loop0" DOES (partly) work, for SBOK={11239424,
>   20480000, 23887872, 71663616, 78675968, 214990848} ( see https://pzt.me/6txz )
> 
> * "mkfs.ext4 -S /dev/loop0" results in a completely empty filesystem
> 
> * "fsck.ext4 -b $SBOK -B 4096 -v -n /dev/loop0" shows a lot of errors.
>   Filesystem isn't mountable afterwards ( see https://pzt.me/42yk and
>   https://pzt.me/3frg )
> 
> * "fsck.ext4 -b $SBOK -B 4096 -v -y /dev/loop0" recoveres after a long time.
>   Filesystem is mountable. Root is empty besides lost+found folder, which
>   contains about 300GB mostly useless data: Millions of files with wrong
>   permissions, useless names and some random content.

The ability to recover from something like this depends heavily on where
the directory structure and inodes were located.  Filesystems tend to use
the start of the disk first, because it has the best performance (about 2x
as fast as the end).

> * photorec from the testdisk package recoveres, luckily!, about 500GB of
>   data. Though the content seems to be pretty reasonable, no filenames
>   are recovered, since photorec operates without using filesystem knowledge.
> 
> Do you see any chances (besides consulting professional recovery companies)
> getting the filenames back?

There was an ext3grep tool that some people had success with, but when I
looked at it, it was still fairly complex to use.

> I looked into the ext4 specs a bit to figure out
> where this information is stored on disk, but before I step with hexedit
> through a terabyte of data, I'd rather try some solutions which are maybe
> already out there.
> 
> Any help is appretiated.
> 
> Thanks in advance, Rudolf.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Cheers, Andreas

Re: Restoring filenames from partly damaged ext4-filesystem

[Rudolf Zran]

Hello Andreas!


[ext4 partition overwritten with garbage at the beginning]


>>  * photorec from the testdisk package recoveres, luckily!, about 500GB of
>>    data. Though the content seems to be pretty reasonable, no filenames
>>    are recovered, since photorec operates without using filesystem 
>>   knowledge.
>> 
>>  Do you see any chances (besides consulting professional recovery companies)
>>  getting the filenames back?
> 
> There was an ext3grep tool that some people had success with, but when I
> looked at it, it was still fairly complex to use.

Thanks for the hint, but the problem with ext3grep is that it seems to
rely on an intact filesystem structure. It even has no option to set
another superblock besides the default one and fails with all commands
I tried ( see https://pzt.me/8q6k ).

Bye, Rudolf.

Re: Restoring filenames from partly damaged ext4-filesystem

[Walter Hurry]

On Thu, 09 Feb 2012 16:29:53 +0000, Rudolf Zran wrote:

> Hi everybody!
> 
> I recently damaged an ext4 partition by accident (mistakenly forced a
> RAID sync with another partition onto it, which I realized after about
> 3% completion). As a result the beginning of the ext4 partition seems to
> be overwritten with garbage and refuses to mount.
> 
> As you might guess, I'd now like to get as most of my data back (from
> the part which hasn't been overwritten, of course :)).
> 
> Maybe somebody knows a good method to just "repair" the ext4-structure
> from the remaining part of the partition?
> 
> 
> Background information:
> 
> Operation system:   Debian GNU/Linux 6.0.4 (Squeeze) Kernel:            
> Linux 2.6.32-5 Architecture:       x86-amd64 e2fsprogs:          1.41.12
> Filesystem type:    ext4, with default settings (mkfs.ext4 /dev/xyz)
> Filesystem size:    Around 1TB, filled with around 800GB of data.
> Filesystem content: From a lot of ASCII stuff up to files with several
>                     GB in size and arbitrary
>                     non-standard type.
> 
> 
> What I've tried so far:
> 
> * First of all, though the source drive is physically fine, I work on
>   images of the partition on a spare drive, to experiment with.
>   Everytime I make unrecoverable errors to that image, I recopy the
>   original.
> 
> * "dumpe2fs -b $SBERR /dev/loop0" does NOT work, for SBERR={32768,
> 98304,
>   163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000,
>   7962624} ( see https://pzt.me/1l55 )
> 
> * "dumpe2fs -b $SBOK /dev/loop0" DOES (partly) work, for SBOK={11239424,
>   20480000, 23887872, 71663616, 78675968, 214990848} ( see
>   https://pzt.me/6txz )
> 
> * "mkfs.ext4 -S /dev/loop0" results in a completely empty filesystem
> 
> * "fsck.ext4 -b $SBOK -B 4096 -v -n /dev/loop0" shows a lot of errors.
>   Filesystem isn't mountable afterwards ( see https://pzt.me/42yk and
>   https://pzt.me/3frg )
> 
> * "fsck.ext4 -b $SBOK -B 4096 -v -y /dev/loop0" recoveres after a long
> time.
>   Filesystem is mountable. Root is empty besides lost+found folder,
>   which contains about 300GB mostly useless data: Millions of files
>   with wrong permissions, useless names and some random content.
> 
> * photorec from the testdisk package recoveres, luckily!, about 500GB of
>   data. Though the content seems to be pretty reasonable, no filenames
>   are recovered, since photorec operates without using filesystem
>   knowledge.
> 
> Do you see any chances (besides consulting professional recovery
> companies) getting the filenames back? I looked into the ext4 specs a
> bit to figure out where this information is stored on disk, but before I
> step with hexedit through a terabyte of data, I'd rather try some
> solutions which are maybe already out there.

The best, simplest and safest plan I think, would be simply to format the 
partition and restore from backup.


-- 
To UNSUBSCRIBE, email to debian-user-REQUEST@lists.debian.org 
with a subject of "unsubscribe". Trouble? Contact listmaster@lists.debian.org
Archive: http://lists.debian.org/jh1kc6$nt9$1@dough.gmane.org

Re: Restoring filenames from partly damaged ext4-filesystem

[Bernd Schubert]

On 02/09/2012 10:22 PM, Rudolf Zran wrote:
> Hello Andreas!
>
>
> [ext4 partition overwritten with garbage at the beginning]
>
>
>>>   * photorec from the testdisk package recoveres, luckily!, about 500GB of
>>>     data. Though the content seems to be pretty reasonable, no filenames
>>>     are recovered, since photorec operates without using filesystem
>>>    knowledge.
>>>
>>>   Do you see any chances (besides consulting professional recovery companies)
>>>   getting the filenames back?
>>
>> There was an ext3grep tool that some people had success with, but when I
>> looked at it, it was still fairly complex to use.
>
> Thanks for the hint, but the problem with ext3grep is that it seems to
> rely on an intact filesystem structure. It even has no option to set
> another superblock besides the default one and fails with all commands
> I tried ( see https://pzt.me/8q6k ).
>

I have written some tools in the past to recover the file structure of 
an over-formated ext3/ext4 device based on directory blocks.
With some tweaks it should be able to assign the file#inode_numbers in 
lost+found to a directory structure.
Problem is that I'm rather busy with too many other projects already. Do 
you know C and could you add some code on your own for the lost+found 
assignment? I can assist you, but it is unlikely that I find much time 
do it myself...


Cheers,
Bernd

Re: Restoring filenames from partly damaged ext4-filesystem

[Rudolf Zran]

Hello Bernd!


> I have written some tools in the past to recover the file structure of 
> an over-formated ext3/ext4 device based on directory blocks.
> With some tweaks it should be able to assign the file#inode_numbers in 
> lost+found to a directory structure.
> Problem is that I'm rather busy with too many other projects already. Do 
> you know C and could you add some code on your own for the lost+found 
> assignment? I can assist you, but it is unlikely that I find much time 
> do it myself...

I'm not that fluent in C, but I could just give it a try. Since it's my
own data and this recovery for personal "fun" I can't do something bad
with it :) So, if you could share the code I'd be happy!

Thanks, Rudolf.

Re: Restoring filenames from partly damaged ext4-filesystem

[Theodore Tso]

On Feb 9, 2012, at 11:29 AM, Rudolf Zran wrote:

> Hi everybody!
> 
> I recently damaged an ext4 partition by accident (mistakenly forced
> a RAID sync with another partition onto it, which I realized after
> about 3% completion). As a result the beginning of the ext4 partition
> seems to be overwritten with garbage and refuses to mount.
> 
> As you might guess, I'd now like to get as most of my data back (from
> the part which hasn't been overwritten, of course :)).
> 
> Maybe somebody knows a good method to just "repair" the ext4-structure
> from the remaining part of the partition?\

Have you tried just simply running e2fsck, specifying an alternate superblock?

I'd do this by making a copy of the file system first, of course….

-- Ted


--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Restoring filenames from partly damaged ext4-filesystem

[Rudolf Zran]

Hello Ted!

>>  I recently damaged an ext4 partition by accident
[...]
>>  Maybe somebody knows a good method to just "repair" the
>> ext4-structure from the remaining part of the partition?
> 
> Have you tried just simply running e2fsck, specifying an alternate superblock?

Yes of course I tried, as I wrote in my original post ($SBOK is one
of the intact superblocks at 11239424, 20480000, 23887872, 71663616,
78675968 and 214990848).

With "all answers: no":

>> * "fsck.ext4 -b $SBOK -B 4096 -v -n /dev/loop0" shows a lot of errors.
>>   Filesystem isn't mountable afterwards

# fsck.ext4 -b 214990848 -B 4096 -v -n /dev/loop0
e2fsck 1.41.12 (17-May-2010)
One or more block group descriptor checksums are invalid.  Fix? no
Group descriptor 0 checksum is invalid.  IGNORED.
Group descriptor 1 checksum is invalid.  IGNORED.
...
Group descriptor 7451 checksum is invalid.  IGNORED.
Group descriptor 7452 checksum is invalid.  IGNORED.
/dev/loop0 contains a file system with errors, check forced.
Resize inode not valid.  Recreate? no
Pass 1: Checking inodes, blocks, and sizes
Root inode is not a directory.  Clear? no
Inode 5 should not have EOFBLOCKS_FL set (size 2642074971391648798, lblk -1)
Clear? no
...

# mount -o ro /dev/loop0 /mnt
mount: /dev/loop0: can't read superblock

# dmesg 
[504508.215515] EXT4-fs error (device loop0): ext4_iget: bad extended attribute block 1920620055 in inode #2
[504508.215530] EXT4-fs (loop0): get root inode failed
[504508.215534] EXT4-fs (loop0): mount failed

With "all answers: yes":

>> * "fsck.ext4 -b $SBOK -B 4096 -v -y /dev/loop0" recoveres after a long time.
>>   Filesystem is mountable. Root is empty besides lost+found folder, which
>>   contains about 300GB mostly useless data: Millions of files with wrong
>>   permissions, useless names and some random content.

> I'd do this by making a copy of the file system first, of course….

For sure! :) I solely work on copies.

Thanks, Rudolf.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Restoring filenames from partly damaged ext4-filesystem

[Ted Ts'o]

On Fri, Feb 10, 2012 at 06:36:52PM +0000, Rudolf Zran wrote:
> >> * "fsck.ext4 -b $SBOK -B 4096 -v -y /dev/loop0" recoveres after a long time.
> >>   Filesystem is mountable. Root is empty besides lost+found folder, which
> >>   contains about 300GB mostly useless data: Millions of files with wrong
> >>   permissions, useless names and some random content.
> 
> > I'd do this by making a copy of the file system first, of course….

I would have expected at least some subdirectories from your directory
hierarcy that contained useful content.

Just to set expectations, things that you might do that tried to look
for directory blocks, etc., *might* give you more useful filenames as
opposed to random inode numbers in lost+found, but it's unlikely to
recover any more *files*.  It sounds most of your files were located
in part of the inode table that got smashed, and so short of looking
at each data block to find useful bits, I doubt spending a lot of time
on trying to use or create more recovery tools based on file system
metadata is likely to result in more data getting recovered.

	    	      	     	     - Ted

P.S.  e2fsck -n by definition opens the block device read-only, so
it's not at all surprising you couldn't mount it after e2fsck -n; it
wouldn't have changed anything.

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Restoring filenames from partly damaged ext4-filesystem

[Bernd Schubert]

On 02/10/2012 10:32 PM, Ted Ts'o wrote:
> On Fri, Feb 10, 2012 at 06:36:52PM +0000, Rudolf Zran wrote:
>>>> * "fsck.ext4 -b $SBOK -B 4096 -v -y /dev/loop0" recoveres after a long time.
>>>>    Filesystem is mountable. Root is empty besides lost+found folder, which
>>>>    contains about 300GB mostly useless data: Millions of files with wrong
>>>>    permissions, useless names and some random content.
>>
>>> I'd do this by making a copy of the file system first, of course….
>
> I would have expected at least some subdirectories from your directory
> hierarcy that contained useful content.
>
> Just to set expectations, things that you might do that tried to look
> for directory blocks, etc., *might* give you more useful filenames as
> opposed to random inode numbers in lost+found, but it's unlikely to
> recover any more *files*.  It sounds most of your files were located
> in part of the inode table that got smashed, and so short of looking
> at each data block to find useful bits, I doubt spending a lot of time
> on trying to use or create more recovery tools based on file system
> metadata is likely to result in more data getting recovered.
>

You do not need inode tables to get back a basic directory structure. 
Assigning directory blocks with '.' and '..' and then with real content 
provide the file system structure and also inode-number to name 
translation. Even better would be if secondary blocks also would have 
'.' and '..'.


Cheers,
Bernd
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Restoring filenames from partly damaged ext4-filesystem

[Bernd Schubert]

On 02/10/2012 06:07 PM, Rudolf Zran wrote:
> Hello Bernd!
>
>
>> I have written some tools in the past to recover the file structure of
>> an over-formated ext3/ext4 device based on directory blocks.
>> With some tweaks it should be able to assign the file#inode_numbers in
>> lost+found to a directory structure.
>> Problem is that I'm rather busy with too many other projects already. Do
>> you know C and could you add some code on your own for the lost+found
>> assignment? I can assist you, but it is unlikely that I find much time
>> do it myself...
>
> I'm not that fluent in C, but I could just give it a try. Since it's my
> own data and this recovery for personal "fun" I can't do something bad
> with it :) So, if you could share the code I'd be happy!

I'm going upload the existing code on Sunday. Going to travel tomorrow 
and I need to go to bed now.

Sorry for the day!


Cheers,
Bernd

[SOLVED] (was "Re: Restoring filenames from partly damaged ext4-filesystem")

[Rudolf Zran]

Hi List!

In an offlist reply someone recommended me ext4magic (see
http://developer.berlios.de/projects/ext4magic ).

Like magic it recovered complete directory hierarchies with filenames,
timestamps, even ownership and permissions for more than 300GB of the
deleted data.

I can recommend this to everybody who accidently destroyed parts of
his filesystem. In contrast to fsck ext4magic doesn't try to repair,
 but just extracts the data completely in read only mode (feelslike
"tar xf" actually :).It's easier to use and understand and way more
effective when it comesto recovery. ext4magic shouldn't be missing
in any good recovery collection.

Thanks to all helping me out in this case.

Have a nice weekend, Rudolf.

Re: [SOLVED] (was "Re: Restoring filenames from partly damaged ext4-filesystem")

[Ted Ts'o]

On Sat, Feb 11, 2012 at 05:33:30PM +0000, Rudolf Zran wrote:
> Hi List!
> 
> In an offlist reply someone recommended me ext4magic (see
> http://developer.berlios.de/projects/ext4magic ).

I wasn't familiar with ext4magic, so thanks for recommending it.
After taking a quick look at its wiki page, it looks like worthy
successor to ext3grep (which only works on ext3 file systems).  It
appears from their web page (I haven't had a chance to play with it
yet) that it uses a variety of techniques to recover data, which
combined should work really well.

Thanks for reporting back!

						- Ted

Re: [SOLVED] (was "Re: Restoring filenames from partly damaged ext4-filesystem")

[Andreas Dilger]

On 2012-02-11, at 11:30 AM, Ted Ts'o wrote:
> On Sat, Feb 11, 2012 at 05:33:30PM +0000, Rudolf Zran wrote:
>> Hi List!
>> 
>> In an offlist reply someone recommended me ext4magic (see
>> http://developer.berlios.de/projects/ext4magic ).
> 
> I wasn't familiar with ext4magic, so thanks for recommending it.
> After taking a quick look at its wiki page, it looks like worthy
> successor to ext3grep (which only works on ext3 file systems).  It
> appears from their web page (I haven't had a chance to play with it
> yet) that it uses a variety of techniques to recover data, which
> combined should work really well.

Looks interesting and useful...  I wonder if it makes sense to include
this into e2fsprogs at some point, so that it is available when users
need it most...

Cheers, Andreas

Re: [SOLVED] (was "Re: Restoring filenames from partly damaged ext4-filesystem")

[Ted Ts'o]

On Sat, Feb 11, 2012 at 11:47:14AM -0700, Andreas Dilger wrote:
> 
> Looks interesting and useful...  I wonder if it makes sense to include
> this into e2fsprogs at some point, so that it is available when users
> need it most...

We'll need to talk to the author about that.  At the very least it
would be good to translate the ext4magic wiki into English, so that
people who are doing web searches are more likely to find it.  :-)

       	       	     	 	      - Ted