How To Recover Files From ext3 Partition??

How To Recover Files From ext3 Partition??

[UZAIR LAKHANI]

Hello All,

Is there any way to recover the deleted files from a
folder in ext3 partition. Basically in one directory,
I gave the command (rm -rf *) and all the files and
folders of this directory are lost. How to recover
these. I had once found an article regarding
un-deleting ext2 files.

Thanks,
Uzair Lakhani

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: How To Recover Files From ext3 Partition??

[Erik Mouw]

On Thu, May 04, 2006 at 07:18:38AM -0700, UZAIR LAKHANI wrote:
> Is there any way to recover the deleted files from a
> folder in ext3 partition. Basically in one directory,
> I gave the command (rm -rf *) and all the files and
> folders of this directory are lost. How to recover
> these. I had once found an article regarding
> un-deleting ext2 files.

It *should* work the same, though the the last time I tried such a
recovery from and ext3 filesystem, ext3 made it particularly hard cause
it zeros most of the inode on delete (including ext3_inode->i_block[]).
Ext2 leaves much more information behind.

You could try with the undelfs feature in Midnight Commander, but first
be sure to mount the filesystem read-only in order to avoid any further
changes.


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: How To Recover Files From ext3 Partition??

[UZAIR LAKHANI]

--- Erik Mouw <erik@harddisk-recovery.com> wrote:

> On Thu, May 04, 2006 at 07:18:38AM -0700, UZAIR
> LAKHANI wrote:
> > Is there any way to recover the deleted files from
> a
> > folder in ext3 partition. Basically in one
> directory,
> > I gave the command (rm -rf *) and all the files
> and
> > folders of this directory are lost. How to recover
> > these. I had once found an article regarding
> > un-deleting ext2 files.
> 
> It *should* work the same, though the the last time
> I tried such a
> recovery from and ext3 filesystem, ext3 made it
> particularly hard cause
> it zeros most of the inode on delete (including
> ext3_inode->i_block[]).
> Ext2 leaves much more information behind.
> 
> You could try with the undelfs feature in Midnight
> Commander, but first
> be sure to mount the filesystem read-only in order
> to avoid any further
> changes.
> 
> 
> Erik
> 
Hello All,

Thanks for the reply. Yes it is difficult to recover
files from ext3 as compared to ext2. I found this
helpful info from one website.

Q: How can I recover (undelete) deleted files from my
ext3 partition?
Actually, you can't! This is what one of the
developers, Andreas Dilger, said about it:
In order to ensure that ext3 can safely resume an
unlink after a crash, it actually zeros out the block
pointers in the inode, whereas
ext2 just marks these blocks as unused in the block
bitmaps and marks the inode as "deleted" and leaves
the block pointers alone.

Your only hope is to "grep" for parts of your files
that have been deleted and hope for the best.

[Source] =
http://batleth.sapienti-sat.org/projects/FAQs/ext3-faq.html

Additionally where can I get the source for Midnight
Commander.

Thanks,
Uzair Lakhani
> -- 
> +-- Erik Mouw -- www.harddisk-recovery.com -- +31 70
> 370 12 90 --
> | Lab address: Delftechpark 26, 2628 XH, Delft, The
> Netherlands
> -
> To unsubscribe from this list: send the line
> "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at 
> http://vger.kernel.org/majordomo-info.html
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: How To Recover Files From ext3 Partition??

[Erik Mouw]

On Thu, May 04, 2006 at 10:16:18PM -0700, UZAIR LAKHANI wrote:
> --- Erik Mouw <erik@harddisk-recovery.com> wrote:
> > On Thu, May 04, 2006 at 07:18:38AM -0700, UZAIR
> > LAKHANI wrote:
> > > Is there any way to recover the deleted files from
> > a
> > > folder in ext3 partition. Basically in one
> > directory,
> > > I gave the command (rm -rf *) and all the files
> > and
> > > folders of this directory are lost. How to recover
> > > these. I had once found an article regarding
> > > un-deleting ext2 files.
> > 
> > It *should* work the same, though the the last time
> > I tried such a
> > recovery from and ext3 filesystem, ext3 made it
> > particularly hard cause
> > it zeros most of the inode on delete (including
> > ext3_inode->i_block[]).
> > Ext2 leaves much more information behind.
> > 
> > You could try with the undelfs feature in Midnight
> > Commander, but first
> > be sure to mount the filesystem read-only in order
> > to avoid any further
> > changes.
> 
> Thanks for the reply. Yes it is difficult to recover
> files from ext3 as compared to ext2. I found this
> helpful info from one website.
> 
> Q: How can I recover (undelete) deleted files from my
> ext3 partition?
> Actually, you can't! This is what one of the
> developers, Andreas Dilger, said about it:
> In order to ensure that ext3 can safely resume an
> unlink after a crash, it actually zeros out the block
> pointers in the inode, whereas
> ext2 just marks these blocks as unused in the block
> bitmaps and marks the inode as "deleted" and leaves
> the block pointers alone.
> 
> Your only hope is to "grep" for parts of your files
> that have been deleted and hope for the best.
> 
> [Source] =
> http://batleth.sapienti-sat.org/projects/FAQs/ext3-faq.html

Andreas, care to elaborate? AFAICS resetting ->i_mode to 0000, setting
->i_links_count to 0, and setting ->i_dtime should be enough to
uniquely mark an inode as deleted. Doing so would make an occasional
undelete much easier to recover for Joe Sixpack (and for us a lot less
work to reconstruct files by scavenging the whole partition for clues).

> Additionally where can I get the source for Midnight
> Commander.

IIRC it lives somewhere on ftp.gnome.org and/or on ftp.gnu.org. Look
for mc-XXX.tar.gz. It started its life as a text based file manager
similar to the Norton Commander. Later on it was used by the Gnome
project to make a graphical file manager (gmc) which was later on
replaced by Nautilus.


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: How To Recover Files From ext3 Partition??

[Andreas Dilger]

On May 05, 2006  13:18 +0200, Erik Mouw wrote:
> Andreas, care to elaborate? AFAICS resetting ->i_mode to 0000, setting
> ->i_links_count to 0, and setting ->i_dtime should be enough to
> uniquely mark an inode as deleted. Doing so would make an occasional
> undelete much easier to recover for Joe Sixpack (and for us a lot less
> work to reconstruct files by scavenging the whole partition for clues).

Yes, while this would mark the _inode_ unused, it does nothing to
mark the blocks unused in the block bitmaps (of which there may be
many thousand/million in a single file).  So ext3 needs to be able to
mark these bitmap blocks unused when the file is unlinked/truncated.
Otherwise each crash would leak blocks that can't be recovered except
by full e2fsck.

There is another mechanism ext3 could potentially use, wherein it
walks the whole inode in advance of the truncate and creates a
(potentially) very large transaction handle to do the bitmap updates
in a single shot (and also reducing the amount of IO needed for an
unlink by 96%), but nobody has ever cared enough about it to work on
implementing this.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: How To Recover Files From ext3 Partition??

[Stephen C. Tweedie]

Hi,

On Fri, 2006-05-05 at 10:41 -0600, Andreas Dilger wrote:

> There is another mechanism ext3 could potentially use, wherein it
> walks the whole inode in advance of the truncate and creates a
> (potentially) very large transaction handle to do the bitmap updates
> in a single shot (and also reducing the amount of IO needed for an
> unlink by 96%), but nobody has ever cared enough about it to work on
> implementing this.

Trouble is, there's no guarantee that that transaction would actually
fit into the journal.  Most of the time it will, but if it doesn't, then
we deadlock or risk data corruption.  

--Stephen

Re: How To Recover Files From ext3 Partition??

[Erik Mouw]

On Mon, May 08, 2006 at 11:51:34AM +0100, Stephen C. Tweedie wrote:
> On Fri, 2006-05-05 at 10:41 -0600, Andreas Dilger wrote:
> > There is another mechanism ext3 could potentially use, wherein it
> > walks the whole inode in advance of the truncate and creates a
> > (potentially) very large transaction handle to do the bitmap updates
> > in a single shot (and also reducing the amount of IO needed for an
> > unlink by 96%), but nobody has ever cared enough about it to work on
> > implementing this.
> 
> Trouble is, there's no guarantee that that transaction would actually
> fit into the journal.  Most of the time it will, but if it doesn't, then
> we deadlock or risk data corruption.  

Is there some way to determine in advance if a transaction fits into
the journal? If so, we could unlink using Andreas's proposal if the
journal allows, and fall back to the old method if not.


Erik

-- 
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands

Re: How To Recover Files From ext3 Partition??

[Stephen C. Tweedie]

Hi,

On Mon, 2006-05-08 at 14:34 +0200, Erik Mouw wrote:

> > Trouble is, there's no guarantee that that transaction would actually
> > fit into the journal.  Most of the time it will, but if it doesn't, then
> > we deadlock or risk data corruption.  
> 
> Is there some way to determine in advance if a transaction fits into
> the journal?

For truncate/delete, no, not easily.  Or rather, it's possible, but only
for trivially short files.  The trouble is that we can't assume that all
of the file's blocks are on the same block groups, so each block in the
file is potentially an update to a new group descriptor and a new block
bitmap (ie. 2 blocks of journal per block of file.)

That's hugely pessimistic, of course, but it is the genuine worst-case
scenario and we have to be prepared for it.  We only work out that we
need less once we actually start walking the file's indirect tree, at
which point the truncate is already under way.  

We _could_ walk the tree twice, but that would be unnecessarily
expensive, especially for large files.

--Stephen

Re: How To Recover Files From ext3 Partition??

[Theodore Tso]

On Mon, May 08, 2006 at 11:51:34AM +0100, Stephen C. Tweedie wrote:
> On Fri, 2006-05-05 at 10:41 -0600, Andreas Dilger wrote:
> 
> > There is another mechanism ext3 could potentially use, wherein it
> > walks the whole inode in advance of the truncate and creates a
> > (potentially) very large transaction handle to do the bitmap updates
> > in a single shot (and also reducing the amount of IO needed for an
> > unlink by 96%), but nobody has ever cared enough about it to work on
> > implementing this.
> 
> Trouble is, there's no guarantee that that transaction would actually
> fit into the journal.  Most of the time it will, but if it doesn't, then
> we deadlock or risk data corruption.  

The only way to do this would be to figure out how how much space is
needed in advance, see if there's enough room, and if there isn't, to
fall back the current strategy of doing a partial truncate.  A lot of
complexity to speed up unlink and make it possible to recover from
unlinked files, but it's doable.  No one with the skills to implement
it has stepped up to date, however.  :-)

						- Ted

Re: How To Recover Files From ext3 Partition??

[Andreas Dilger]

On May 08, 2006  13:42 +0100, Stephen C. Tweedie wrote:
> On Mon, 2006-05-08 at 14:34 +0200, Erik Mouw wrote:
> 
> > > Trouble is, there's no guarantee that that transaction would actually
> > > fit into the journal.  Most of the time it will, but if it doesn't, then
> > > we deadlock or risk data corruption.  
> > 
> > Is there some way to determine in advance if a transaction fits into
> > the journal?
> 
> For truncate/delete, no, not easily.  Or rather, it's possible, but only
> for trivially short files.  The trouble is that we can't assume that all
> of the file's blocks are on the same block groups, so each block in the
> file is potentially an update to a new group descriptor and a new block
> bitmap (ie. 2 blocks of journal per block of file.)

Actually, the upper limit is the number of block groups in the filesystem.
In many cases this would be a sufficient upper bound that could be checked
without any effort.  Given that the default journal size is now 128MB
(32k blocks) this would guarantee full-file truncates in the worst case
for up (to 32k blocks / 4 / (1 + 1/128 blocks/group)) = 8128 groups = 1016GB
before we even have to consider walking the file.  We could also work out
the worst-case truncate by the number of blocks in the file.

> That's hugely pessimistic, of course, but it is the genuine worst-case
> scenario and we have to be prepared for it.  We only work out that we
> need less once we actually start walking the file's indirect tree, at
> which point the truncate is already under way.  
> 
> We _could_ walk the tree twice, but that would be unnecessarily
> expensive, especially for large files.

That was actually my thought.  I don't think it is expensive, given the
current implementation already has to read all of these blocks from disk
(in reverse order, I might add) and then _write_ to each of the indirect
blocks (1/1024 of the whole file size) to zero them out.

Instead, we could walk the file tree in forward order, doing async readahead
for the indirect, dindirect, tindirect blocks, then a second batch of
readahead for all of the indirect blocks from dindirect, and dindirect blocks
from tindirect, rinse repeat.  In the meantime we could walk the blocks and
count the block groups affected (= transaction size) while waiting for the
next batch of blocks to complete IO.  Since we need to read these blocks in
any case, doing the readahead efficiently will likely improve performance
of this step.

Rather than hurt performance, I think this will actually improve the truncate
performance because we don't need to do ANY indirect block writes, removing
1 block write per 1MB of file space, only doing the (already required) write
to the group descriptor and bitmap.  Given that we try hard to do contiguous
allocations for files this will usually be a relatively small number of
blocks, as few as 1/128MB of the file size (group size limit).  If we do the
file walk, instead of being completely pessimistic, we can also reduce the
pressure on journal flushes, which I suspect would be more costly than the
walk itself.

The fact that this also fixes undelete was actually a side-effect, IMHO.

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.