ext4 undeletion question

ext4 undeletion question

[Michael B. Trausch]

(First things first, please keep my cell phone -- mike-mobile@trausch.us 
-- on the cc list, as I cannot easily read mail on my system since it is, 
more or less, crippled at the moment.)

Alright, so I have found myself in a troublesome situation.  I had a 
directory which I accidentally deleted (instead of an identically-named 
directory in *another* directory) and I need to get it back; it was a 
version control repository directory and contains ~150 revisions of a 
project I have been working on this week.  (Ironically, I was preparing to 
back it up today, heh.)

Anyway, is there _any_ means by which to recover files from an ext4 
filesystem?  The utils for ext2 filesystems don't work (not surprising), 
and I am wondering if there is a way to look for deleted files, knowing 
what their name was, and recover them if they are not yet overwritten.  I 
have my home directory mounted read-only at this point so as to minimize 
the chance of the latter, and discovered my mistake nearly immediately so 
I hope that the data are still intact.

Any help would be greatly appreciated!

   --- Mike

Re: ext4 undeletion question

[Theodore Tso]

On Tue, Apr 28, 2009 at 10:42:39AM -0400, Michael B. Trausch wrote:
> (First things first, please keep my cell phone -- mike-mobile@trausch.us  
> -- on the cc list, as I cannot easily read mail on my system since it is, 
> more or less, crippled at the moment.)
>
> Alright, so I have found myself in a troublesome situation.  I had a  
> directory which I accidentally deleted (instead of an identically-named  
> directory in *another* directory) and I need to get it back; it was a  
> version control repository directory and contains ~150 revisions of a  
> project I have been working on this week.  (Ironically, I was preparing 
> to back it up today, heh.)
>
> Anyway, is there _any_ means by which to recover files from an ext4  
> filesystem?  The utils for ext2 filesystems don't work (not surprising),  
> and I am wondering if there is a way to look for deleted files, knowing  
> what their name was, and recover them if they are not yet overwritten.  I 
> have my home directory mounted read-only at this point so as to minimize  
> the chance of the latter, and discovered my mistake nearly immediately so 
> I hope that the data are still intact.

There is the program "ext3grep" which will look for older versions of
the directory and inode table blocks in the journal.  This can work,
but unfortunately I don't think it's been extended to understand about
the ext4 extent data structure.

						- Ted

Re: ext4 undeletion question

[Michael B. Trausch]

[-- Attachment #1: Type: text/plain, Size: 1337 bytes --]

On Tue, 28 Apr 2009 12:11:06 -0400
Theodore Tso <tytso@mit.edu> wrote:
> There is the program "ext3grep" which will look for older versions of
> the directory and inode table blocks in the journal.  This can work,
> but unfortunately I don't think it's been extended to understand about
> the ext4 extent data structure.

Eh.  Thanks for the mention... gave it a shot, but it seems to fail
nearly immediately:

Tuesday, 2009-Apr-28 at 13:21:41 - mbt@zest - Linux v2.6.29.1
Ubuntu Jaunty:[0-9/10014-0]:undel> sudo ext3grep --restore-all /dev/zestvg/home-retain-undelete
Running ext3grep version 0.10.1
WARNING: I don't know what EXT3_FEATURE_COMPAT_EXT_ATTR is.
ext3grep: ext3grep.cc:119: void run_program(): Assertion `be2le(journal_super_block.s_header.h_magic) == 0xc03b3998U' failed.
zsh: abort      sudo ext3grep --restore-all /dev/zestvg/home-retain-undelete

I guess that means it won't work on an ext4 fs.  :-)

I did create a snapshot of it using LVM (durr, I didn't think of that
before) so the FS is preserved as it was... I just don't know how to go
about digging through it to get the directory that I deleted out.
Hopefully I can figure that out before terribly long, as I am stuck
until I do...

	--- Mike

-- 
Don't fix bugs later; fix them now.
                            --- Steve Maguire

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: ext4 undeletion question

[Greg Freemyer]

On Tue, Apr 28, 2009 at 1:26 PM, Michael B. Trausch <mbt@zest.trausch.us> wrote:
> On Tue, 28 Apr 2009 12:11:06 -0400
> Theodore Tso <tytso@mit.edu> wrote:
>> There is the program "ext3grep" which will look for older versions of
>> the directory and inode table blocks in the journal.  This can work,
>> but unfortunately I don't think it's been extended to understand about
>> the ext4 extent data structure.
>
> Eh.  Thanks for the mention... gave it a shot, but it seems to fail
> nearly immediately:
>
> Tuesday, 2009-Apr-28 at 13:21:41 - mbt@zest - Linux v2.6.29.1
> Ubuntu Jaunty:[0-9/10014-0]:undel> sudo ext3grep --restore-all /dev/zestvg/home-retain-undelete
> Running ext3grep version 0.10.1
> WARNING: I don't know what EXT3_FEATURE_COMPAT_EXT_ATTR is.
> ext3grep: ext3grep.cc:119: void run_program(): Assertion `be2le(journal_super_block.s_header.h_magic) == 0xc03b3998U' failed.
> zsh: abort      sudo ext3grep --restore-all /dev/zestvg/home-retain-undelete
>
> I guess that means it won't work on an ext4 fs.  :-)
>
> I did create a snapshot of it using LVM (durr, I didn't think of that
> before) so the FS is preserved as it was... I just don't know how to go
> about digging through it to get the directory that I deleted out.
> Hopefully I can figure that out before terribly long, as I am stuck
> until I do...
>
>        --- Mike

Mike,

WinHex is a commercial product for doing undeletes that supports
Ext2/3, ReiserFS, Reiser4, and UFS.  You need the Specialist version
to handle ext3.  It might handle ext4.

If not and assuming you have a very ext3 like on disk structure, you
might be able to force it to work anyway.

http://www.x-ways.com/winhex/index-m.html

You can try out the demo and see if you can see if it works.  I don't
know what the limitations of the demo version are.

Also, the support forum is run by one of the main coders (Stefan) of
the software, so he is extremely knowledgeable.   I think he also owns
part of the company, but I'm not sure about that.

Given that ext4 is the coming thing, he is likely to be willing to
work with you to extend his product at least as far as needed to
support you.  Notice he has Reiser4 support which shows he keeps the
software pretty leading edge.

Greg
-- 
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ext4 undeletion question

[Andreas Dilger]

On Apr 28, 2009  13:55 -0400, Greg Freemyer wrote:
> On Tue, Apr 28, 2009 at 1:26 PM, Michael B. Trausch <mbt@zest.trausch.us> wrote:
> > Eh.  Thanks for the mention... gave it a shot, but it seems to fail
> > nearly immediately:
> >
> > Tuesday, 2009-Apr-28 at 13:21:41 - mbt@zest - Linux v2.6.29.1
> > Ubuntu Jaunty:[0-9/10014-0]:undel> sudo ext3grep --restore-all /dev/zestvg/home-retain-undelete
> > Running ext3grep version 0.10.1
> > WARNING: I don't know what EXT3_FEATURE_COMPAT_EXT_ATTR is.
> > ext3grep: ext3grep.cc:119: void run_program(): Assertion `be2le(journal_super_block.s_header.h_magic) == 0xc03b3998U' failed.
> > zsh: abort      sudo ext3grep --restore-all /dev/zestvg/home-retain-undelete

One question is whether you formatted this filesystem as ext4 to start
with, or did you upgrade the filesystem from ext3 and the files existed
before the upgrade?  If the latter it is much more likely that you can
recover.  If the former you need to add EXT4_FEATURE_INCOMPAT_EXTENTS
to the list of supported feature flags in run_program().

If the files were created under ext4 then the ext3grep tool will likely
have to be further modified to understand the extent format, which
might not be too hard if you are willing to poke at the code a bit.
It seems like the ext3grep author is receptive to answering questions
about that tool and we can answer questions about the ext4 extent layout.

At a quick glance, what would be easiest to start is to modify the
print_inode_to() function to check for the EXT4_EXTENT_FL being set
in the inode.flags() field.  Add a new extent() method to the Inode
struct that casts i_blocks[0] to struct ext4_extent_header and then
the i_blocks[3], i_blocks[6], i_blocks[9], i_blocks[12] to struct
ext4_extent if eh_depth == 0, or ext4_extent_index if eh_depth > 0.

The good news is that ext4 usually allocates file blocks contiguously
so if you can find the inodes themselves in the journal you can likely
extract most of the data just by printing the in-inode extents to find
the block ranges and then dumping the file data with 'dd'.

> > I did create a snapshot of it using LVM (durr, I didn't think of that
> > before) so the FS is preserved as it was... I just don't know how to go
> > about digging through it to get the directory that I deleted out.
> > Hopefully I can figure that out before terribly long, as I am stuck
> > until I do...

> WinHex is a commercial product for doing undeletes that supports
> Ext2/3, ReiserFS, Reiser4, and UFS.  You need the Specialist version
> to handle ext3.  It might handle ext4.

It is highly unlikely that this tool will understand ext4 enhancements as-is.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ext4 undeletion question

[Michael B. Trausch]

[-- Attachment #1: Type: text/plain, Size: 1335 bytes --]

On Tue, 28 Apr 2009 13:55:07 -0400
Greg Freemyer <greg.freemyer@gmail.com> wrote:

> WinHex is a commercial product for doing undeletes that supports
> Ext2/3, ReiserFS, Reiser4, and UFS.  You need the Specialist version
> to handle ext3.  It might handle ext4.

Aye, it says with the full forensic version it will handle ext4 (gave
me a dialog saying that it detected ext4, and then closed the window on
me).

Unfortunately, I don't have €176.90 (~USD 230) to purchase it, so it's
not going to be a viable option for me.  At least, if I understand it
correctly, that's the price for the version that I would need.

It is rather nice to see a program that not only runs under Wine well,
but also has functionality that makes it very useful specifically for
Linux when running under Wine.  It also mentions that there are
unsupported flags in the superblock, so maybe support isn't 100%
complete, but if I had the funds I'd try it anyway; I am not that great
a low-level programmer and certainly have never mucked about with
filesystems (other than vfat, but that really doesn't count as a
filesystem in my book...)

	--- Mike

-- 
Real quality means making sure that people are proud of the code they
write, that they're involved and taking it personally.
                            --- Linus Torvalds

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: ext4 undeletion question

[Michael B. Trausch]

[-- Attachment #1: Type: text/plain, Size: 2439 bytes --]

(Andreas, sorry for the dup, forgot to hit reply-all the last time.  My
fault.)

On Tue, 28 Apr 2009 12:53:46 -0600
Andreas Dilger <adilger@sun.com> wrote:

> The good news is that ext4 usually allocates file blocks contiguously
> so if you can find the inodes themselves in the journal you can likely
> extract most of the data just by printing the in-inode extents to find
> the block ranges and then dumping the file data with 'dd'.

So, here's what my current situation is:

  * I deleted a .bzr directory which contains files and additional
    directories.  AFAIK, directories are just "files", but exposed
    differently to the operating system, yes?  So I should be able to,
    if I were to find its inode, find all of its descendants, too, to
    bring them back, right?  I don't have a directory listing from
    before the deletion, and I presume that would make all the
    difference in the world.  Also, I'd need to figure out which one
    would be the correct one; I've deleted other .bzr directories (as
    part of their directory trees) repeatedly in the past on this
    filesystem.

  * This filesystem was in fact created as ext4.  I'd attempted an
    upgrade some time ago, but the upgrade took *ages* and so I aborted
    it early, reformatted, and restored my home directory from a
    tarball on the new filesystem.

  * I know where in the directory tree the .bzr directory had a link,
    obviously.  I don't know if that information is helpful or not,
    though, unless having the inode # of the containing directory is
    useful in some way.

  * I don't know _anything_ about how journaling works, but I do know
    that the directory was about a week old (just young enough to not
    be on a backup; since some hardware changes were made I hadn't
    started my hourly rsnapshot back up since I don't currently have a
    drive to rsnapshot to---d'oh!)

I'd be happy to look into options, though as I mentioned in my previous
message, I am not a solid system-level programmer (at least in my own
personal opinion).  I could probably figure it out, given enough time,
but when it comes to C (or C++) I just plain _suck_.  I am trying to
improve that, though.

	--- Mike

-- 
I don't really know that anybody's proven that a random collection of
people doing their own thing actually creates value.
                            --- Steve Ballmer, 2007

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]