fs corruption recovery?

fs corruption recovery?

[Kervin Pierre]

Hi,

I install and used 2.4.17 for about a week before my filesystem 
corrupted.  I've tried 'fsck -a' but it complains that there was no 
valid superblock found.

Are there any tools or techniques that will recover data from the 
corrupted filesystem even if there isn't a valid superblock?  Or is 
there a way to write a temporary superblock so I can access the 
information on the disk?

Lastly, if all else fails I'm going to try sending the drive one of 
those 'file recovery companies'.  Does anyone have a recommendation for 
a particular company?  I'm guessing that there'll be a few that wouldn't 
know what to do with a ext3 partition.

thanks,
-Kervin

Re: fs corruption recovery?

[Andreas Dilger]

On Jan 08, 2002  21:52 -0500, Kervin Pierre wrote:
> I install and used 2.4.17 for about a week before my filesystem 
> corrupted.  I've tried 'fsck -a' but it complains that there was no 
> valid superblock found.

Try "e2fsck -B 4096 -b 32768 <device>" instead.

> Are there any tools or techniques that will recover data from the 
> corrupted filesystem even if there isn't a valid superblock?  Or is 
> there a way to write a temporary superblock so I can access the 
> information on the disk?

The ext2 format (includes ext3) has backup superblocks for just this reason.

> Lastly, if all else fails I'm going to try sending the drive one of 
> those 'file recovery companies'.  Does anyone have a recommendation for 
> a particular company?  I'm guessing that there'll be a few that wouldn't 
> know what to do with a ext3 partition.

Is the data really that valuable, and you don't have a backup?  It may
cost you several thousand dollars to do a recovery from such a company.
Yet, it isn't worth doing backups, it appears.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: fs corruption recovery?

[Richard Gooch]

Andreas Dilger writes:
> On Jan 08, 2002  21:52 -0500, Kervin Pierre wrote:
> > I install and used 2.4.17 for about a week before my filesystem 
> > corrupted.  I've tried 'fsck -a' but it complains that there was no 
> > valid superblock found.
> 
> Try "e2fsck -B 4096 -b 32768 <device>" instead.
> 
> > Are there any tools or techniques that will recover data from the 
> > corrupted filesystem even if there isn't a valid superblock?  Or is 
> > there a way to write a temporary superblock so I can access the 
> > information on the disk?
> 
> The ext2 format (includes ext3) has backup superblocks for just this reason.
> 
> > Lastly, if all else fails I'm going to try sending the drive one of 
> > those 'file recovery companies'.  Does anyone have a recommendation for 
> > a particular company?  I'm guessing that there'll be a few that wouldn't 
> > know what to do with a ext3 partition.
> 
> Is the data really that valuable, and you don't have a backup?  It may
> cost you several thousand dollars to do a recovery from such a company.
> Yet, it isn't worth doing backups, it appears.

And these companies don't really do much that you can't do yourself. I
had a failing drive some years ago, where some sectors couldn't be
read. So I tried to dd the raw device to a file elsewhere. Of course,
dd will quit when it has an I/O error. So I wrote a recovery utility
that writes a zero sector if reading the input sector gives an I/O
error. Unfortunately, I couldn't mount the file (too much corruption),
but I was able to use debugfs on it. I got the most important data
back.

While I was waiting for 48 hours for the data to be pulled off (each
time a bad sector was encountered, the drive would retry several
times, with lots of clicking and rattling), I contacted one of these
recovery companies. I wanted to know if they could recover the bad
sectors. I was told no. After some probing, it turns out that all they
do is basically what I was doing. They just charge $2000 for it.

No doubt if you took your drive to your local CIA/KGB/MI6 offices,
they could recover some of those bad sectors. But I hear they charge
their customers quite a lot...

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: fs corruption recovery?

[Kervin Pierre]

Hi,

Thanks for the input.  

Do you still have any of those scripts around? Or can you give me a 
general idea of how you used debugfs to retrieve your files?

I was actually expecting to spend a few hundred instead of a few thousand.

Thanks,
-Kervin

Richard Gooch wrote:

>Andreas Dilger writes:
>
>>On Jan 08, 2002  21:52 -0500, Kervin Pierre wrote:
>>
>>>I install and used 2.4.17 for about a week before my filesystem 
>>>corrupted.  I've tried 'fsck -a' but it complains that there was no 
>>>valid superblock found.
>>>
>>Try "e2fsck -B 4096 -b 32768 <device>" instead.
>>
>>>Are there any tools or techniques that will recover data from the 
>>>corrupted filesystem even if there isn't a valid superblock?  Or is 
>>>there a way to write a temporary superblock so I can access the 
>>>information on the disk?
>>>
>>The ext2 format (includes ext3) has backup superblocks for just this reason.
>>
>>>Lastly, if all else fails I'm going to try sending the drive one of 
>>>those 'file recovery companies'.  Does anyone have a recommendation for 
>>>a particular company?  I'm guessing that there'll be a few that wouldn't 
>>>know what to do with a ext3 partition.
>>>
>>Is the data really that valuable, and you don't have a backup?  It may
>>cost you several thousand dollars to do a recovery from such a company.
>>Yet, it isn't worth doing backups, it appears.
>>
>
>And these companies don't really do much that you can't do yourself. I
>had a failing drive some years ago, where some sectors couldn't be
>read. So I tried to dd the raw device to a file elsewhere. Of course,
>dd will quit when it has an I/O error. So I wrote a recovery utility
>that writes a zero sector if reading the input sector gives an I/O
>error. Unfortunately, I couldn't mount the file (too much corruption),
>but I was able to use debugfs on it. I got the most important data
>back.
>
>While I was waiting for 48 hours for the data to be pulled off (each
>time a bad sector was encountered, the drive would retry several
>times, with lots of clicking and rattling), I contacted one of these
>recovery companies. I wanted to know if they could recover the bad
>sectors. I was told no. After some probing, it turns out that all they
>do is basically what I was doing. They just charge $2000 for it.
>
>No doubt if you took your drive to your local CIA/KGB/MI6 offices,
>they could recover some of those bad sectors. But I hear they charge
>their customers quite a lot...
>
>				Regards,
>
>					Richard....
>Permanent: rgooch@atnf.csiro.au
>Current:   rgooch@ras.ucalgary.ca
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>

Re: fs corruption recovery?

[Richard Gooch]

Kervin Pierre writes:
> Do you still have any of those scripts around? Or can you give me a 
> general idea of how you used debugfs to retrieve your files?

http://www.atnf.csiro.au/~rgooch/linux/

> I was actually expecting to spend a few hundred instead of a few
> thousand.

That was a quote?

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: fs corruption recovery?

[Walter Hofmann]

On Tue, 08 Jan 2002, Kervin Pierre wrote:

> Do you still have any of those scripts around? Or can you give me a 
> general idea of how you used debugfs to retrieve your files?

SuSE has a ddrescure RPM in their distribution which will do what you
need:

wh@aragorn:~$ /usr/local/frodo/bin/dd_rescue
dd_rescue: (fatal): both input and output have to be specified!
 
dd_rescue Version 0.98, garloff@suse.de, GNU GPL
 ($Id: dd_rescue.c,v 1.22 1999/10/19 23:46:25 garloff Exp $)
dd_rescue copies data from one file (or block device) to another
USAGE: dd_rescue [options] infile outfile
Options: -s ipos    start position in  input file (default=0),
         -S opos    start position in output file (def=ipos);
         -b softbs  block size for copy operation (def=16384),
         -B hardbs  fallback block size in case of errs (def=512);
         -e maxerr  exit after maxerr errors (def=0=infinite);
         -m maxxfer maximum amount of data to be transfered (def=0=inf);
         -l logfile name of a file to log errors and summary to
(def="");
         -r         reverse direction copy (def=forward);
         -t         truncate output file (def=no);
         -w         abort on Write errors (def=no);
         -a         spArse file writing (def=no),
         -A         Always write blocks, zeroed if err (def=no);
         -i         interactive: ask before overwriting data (def=no);
         -f         force: skip some sanity checks (def=no);
         -q         quiet operation,
         -v         verbose operation;
         -V         display version and exit;
         -h         display this help and exit.
Note: Sizes may be given in units b(=512), k(=1024) or M(=1024*1024)
bytes
This program is useful to rescue data in case of I/O errors, because
 it does not necessarily aborts or truncates the output.


Walter

Re: fs corruption recovery?

[Ralf Baechle]

On Tue, Jan 08, 2002 at 11:14:04PM -0500, Kervin Pierre wrote:

> Do you still have any of those scripts around? Or can you give me a 
> general idea of how you used debugfs to retrieve your files?

dd if=/dev/broken of=somefile conv=broken

> I was actually expecting to spend a few hundred instead of a few thousand.

Oh, the price tag for recovery of information from drive with heavier
physical damages are quite a bit juicier :-)

  Ralf

Re: fs corruption recovery?

[Helge Hafting]

Richard Gooch wrote:
> 
> Andreas Dilger writes:
[...]
> > Is the data really that valuable, and you don't have a backup?  It may
> > cost you several thousand dollars to do a recovery from such a company.
> > Yet, it isn't worth doing backups, it appears.
> 
> And these companies don't really do much that you can't do yourself. I
> had a failing drive some years ago, where some sectors couldn't be
> read. So I tried to dd the raw device to a file elsewhere. Of course,
> dd will quit when it has an I/O error. So I wrote a recovery utility
> that writes a zero sector if reading the input sector gives an I/O
> error. Unfortunately, I couldn't mount the file (too much corruption),
> but I was able to use debugfs on it. I got the most important data
> back.
> 
> While I was waiting for 48 hours for the data to be pulled off (each
> time a bad sector was encountered, the drive would retry several
> times, with lots of clicking and rattling), I contacted one of these
> recovery companies. I wanted to know if they could recover the bad
> sectors. I was told no. After some probing, it turns out that all they
> do is basically what I was doing. They just charge $2000 for it.
> 
> No doubt if you took your drive to your local CIA/KGB/MI6 offices,
> they could recover some of those bad sectors. But I hear they charge
> their customers quite a lot...

No need for CIA/KGB.  There are companies that do more than this.  
If necessary, they disassemble the drive in a clean room and use 
their own reading equipment.  This allows recovery
from fried drive electronics and broken motors/heads.  And sometimes
(partial) recovery from scratches and other bad sectors.

If you really need this, consider
http://www.ibasuk.com/technology/patan.htm

Helge Hafting

Re: fs corruption recovery?

[Richard Gooch]

Helge Hafting writes:
> No need for CIA/KGB.  There are companies that do more than this.  
> If necessary, they disassemble the drive in a clean room and use 
> their own reading equipment.  This allows recovery
> from fried drive electronics and broken motors/heads.  And sometimes
> (partial) recovery from scratches and other bad sectors.
> 
> If you really need this, consider
> http://www.ibasuk.com/technology/patan.htm

Any idea how much this costs?

				Regards,

					Richard....
Permanent: rgooch@atnf.csiro.au
Current:   rgooch@ras.ucalgary.ca

Re: fs corruption recovery?

[Roy Sigurd Karlsbakk]

> Any idea how much this costs?

I've been using IBAS a few times for faulty drives, where - say - a head
has fallen off or the likes. That usually costs some NOK 15-20,000
(EUR 1900-2500 something). They charge around EUR 250 to diagnose the
problem, and then you get a fixed price.

Check http://www.ibas.com/recovery/

roy

--
Roy Sigurd Karlsbakk, MCSE, MCNE, CLS, LCA

Computers are like air conditioners.
They stop working when you open Windows.

Re: fs corruption recovery?

[Bjorn Wesen]

On Tue, 8 Jan 2002, Richard Gooch wrote:
> And these companies don't really do much that you can't do yourself. I
> had a failing drive some years ago, where some sectors couldn't be
> read. So I tried to dd the raw device to a file elsewhere. Of course,
> dd will quit when it has an I/O error. So I wrote a recovery utility
> that writes a zero sector if reading the input sector gives an I/O
> error. Unfortunately, I couldn't mount the file (too much corruption),
> but I was able to use debugfs on it. I got the most important data
> back.

I thought I'd just chip in with a story about hard disk recovery here.
There is a snippet in the end relating to ext2 at least :)

Once upon a time, there was a program called DiskSalv (or something) for
the Amiga, which did the Right Thing - it tried to extract as many files
as possible from a broken disk, onto _another_ disk. That was great.

Now, 15 years later, it seems as if disk rescue programs still use the
method of trying to fix the broken disk directly. That's all good and well
for the occational missing update due to an OS crash, but should never be
done on a physically broken medium of course, if one cares about the
files..

So when my Windows (TM) FAT32 partition broke physically some months ago,
I was stunned in that I could not seem to find any tool which did the
Right Thing. All tools I found wanted to actually try to fix the partition
instead of just giving up on it and extracting what's possible to extract.

I wound up writing my own FAT32 salvage program in Linux which did exactly
that - guesses parameters even if the boot block is wasted, scans for
possible directories even if the root dir is gone, uses the backup FAT if
the other one is broken.. and then copies all the dirs and files it could
read onto the rescue medium, interactively if wanted. That was great.

Actually just some weeks ago a friend got into the same mess - could find
no program to fix the hard disk, he was ready to give up and format it,
when we thought, hey come put your disk in my computer and we'll fire up
this Linux program which will magically get your files back. Said and
done, he got all his files back even though the boot block and root dirs
were mangled (get this - he had run some program included with WinXP
called "diskfix" or something, which had more or less said "your disk is
broken. I'm setting FAT on it", upon it had quick-formatted it with
_FAT12_ yes FAT12 :) There was a big ROTFL when I discovered that. Nice
utility...

Now, when the same thing happens on an ext2 partition, something similar
should be possible to do, shouldn't it ? Is the "copy the raw data to
another partition" really necessary ?

Thing is, there are like two situations when something on a partition
breaks.. either you just want to get the partition working again and don't
really care about the data, or you care about the data but not the
partition. So if you really want to get your data back, you are willing to
take your chances on things you normally aren't. Exactly how big
corruption can e2fsck face (in the case of a copied raw partition so it
can write to it) and still recover the files ? Does it find loose files
whose directory entries are lost etc ? (impossible in FAT unfortunately)

/BW

Re: fs corruption recovery?

[Alex Bligh - linux-kernel]

--On Tuesday, 08 January, 2002 8:26 PM -0700 Richard Gooch 
<rgooch@ras.ucalgary.ca> wrote:

> I contacted one of these
> recovery companies. I wanted to know if they could recover the bad
> sectors. I was told no. After some probing, it turns out that all they
> do is basically what I was doing. They just charge $2000 for it.

You are talking to the wrong data recovery company. There
are companies in the UK I know of (and I'm sure elsewhere)
who go further, for instance
a) Physical cleaning of drive and electronics (useful if
   a water release fire suppression, or foam fire suppression
   has just washed crap all over it).
b) Swapping controller electronics with known working
   drive (to combat fried electronics or physical damage
   to PCB).
c) Go to clean room, remove platters, clean carefully,
   insert into known working drive
d) Read sectors multiple times varying various parameters
   to build up statistical bitmaps of sectors which fatally
   fail CRC checks.

Now I'm not a DR person but I know 2 people who have worked
for them. I'm sure you can do much of this yourself if you
dare^Wtry, but I think many /do/ do more than keep reading
the same sector multiple times (FWIW if you want this to
work I recommend resetting the drive, and/or powering it
down after every n attempts at a read, then doing a seek
to a random position - this used to help).

--
Alex Bligh

Re: fs corruption recovery?

[Kervin Pierre]

Thanks for the reply,

 >Try "e2fsck -B 4096 -b 32768 <device>" instead.
 >

  e2fsck -B 4096 -b 32768  /dev/hdc1

e2fsck 1.25 (20-Sep-2001)
e2fsck: Attempt to read block from filesystem resulted in short read
while trying to open /dev/hdc1
Could this be a zero-length partition?


Does ext keep a backup of that backup?  :)

Are there any other options?

 >
 >Is the data really that valuable, and you don't have a backup?  It may
 >cost you several thousand dollars to do a recovery from such a company.
 >Yet, it isn't worth doing backups, it appears.
 >
Not the smartest thing I've done, I'll admit. :)

-Kervin

Re: fs corruption recovery?

[H. Peter Anvin]

Followup to:  <3C3BC0FC.30403@fit.edu>
By author:    Kervin Pierre <kpierre@fit.edu>
In newsgroup: linux.dev.kernel
> 
> Thanks for the reply,
> 
>  >Try "e2fsck -B 4096 -b 32768 <device>" instead.
>  >
> 
>   e2fsck -B 4096 -b 32768  /dev/hdc1
> 
> e2fsck 1.25 (20-Sep-2001)
> e2fsck: Attempt to read block from filesystem resulted in short read
> while trying to open /dev/hdc1
> Could this be a zero-length partition?
> 
> Does ext keep a backup of that backup?  :)
> 
> Are there any other options?
> 

You have bad media.  You need to read the disk sector by sector, save
it elsewhere, write it to another drive, and then try to fsck it --
HOWEVER, DON'T FSCK THE ORIGINAL.  Fsck can actually cause serious
damage if it guesses wrong.

The following is a small C program that I just threw together to read
sector by sector and dump the data on stdout.  No guarantees
whatsoever:


#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <unistd.h>
#include <errno.h>

#define SECTORSIZE 512

int read_sector(int fd, void *buf)
{
  char *p = buf;
  int bytes = SECTORSIZE;
  int rv;

  while ( bytes ) {
    if ( (rv = read(fd, p, bytes)) <= 0 ) {
      if ( rv < 0 || errno == EINTR )
	continue;		/* Happens... */
      return 0;			/* Error */
    }
    p += rv;
    bytes -= rv;
  }

  return 1;			/* Success */
}

int main(int argc, char *argv[])
{
  unsigned long long count;
  unsigned long long n;
  int fd;
  char buffer[SECTORSIZE];

  if ( argc != 3 ) {
    fprintf(stderr, "Usage: %s device sectors\n", argv[0]);
    exit(1);
  }

  count = strtoull(argv[2], NULL, 0);
  
  fd = open(argv[1], O_RDONLY);
  if ( fd < 0 ) {
    perror(argv[1]);
    exit(1);
  }
  
  n = 0;
  while ( count-- ) {
    if ( !read_sector(fd, buffer) ) {
      fprintf(stderr, "Sector %llu: %s\n", n, strerror(errno));
      memset(buffer, 0, SECTORSIZE);
    }
    if ( fwrite(buffer, 1, SECTORSIZE, stdout) != SECTORSIZE ) {
      fprintf(stderr, "Sector %llu: output error: %s\n", n, strerror(errno));
      exit(1);
    }
    n++;
  }

  return 0;
}


	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: fs corruption recovery?

[Thomas Capricelli]

On Wednesday 09 January 2002 04:07, Andreas Dilger wrote:
> Try "e2fsck -B 4096 -b 32768 <device>" instead.

	I thought e2fsck was already trying the different superblocks present on the 
device. Why isn't e2fsck smart enought to look for then ? Is this an 
intended purpose ?

	Why do you use the -B option ? How can it be useful to force the block size 
? Especially if this one is different. 

Thanx,
Thomas

Re: fs corruption recovery?

[Andreas Dilger]

On Jan 09, 2002  10:28 +0100, Thomas Capricelli wrote:
> On Wednesday 09 January 2002 04:07, Andreas Dilger wrote:
> > Try "e2fsck -B 4096 -b 32768 <device>" instead.
> 
> I thought e2fsck was already trying the different superblocks present on the 
> device.

Well, yes and no.  Most versions of e2fsck (i.e. every version in existence
unless you have a very recent copy from Ted's BitKeeper repository) will
try possible block sizes, but will not try different block numbers.

> Why isn't e2fsck smart enought to look for then? Is this an intended purpose?

Well, in the most recent versions, it will try a lot harder to try and find
backup superblocks.  It will still not run e2fsck automatically on the device.
There are many reasons why e2fsck may think the superblock is corrupted, but
in fact it isn't:

1) superblock has a new feature which an old e2fsck doesn't understand
2) filesystem is no longer ext2, but may still have the backup superblock
   (e.g. you mkswap on an old ext2 partition, it leaves the old superblock)

> Why do you use the -B option ? How can it be useful to force the block size?
> Especially if this one is different. 

Well, since it is possible to have multiple block sizes for ext2, if you
specify "-b 32768" (i.e. block 32768) for the superblock backup, how does
it know what the blocksize is if you don't tell it that also*.  The old
error message (try -b 8193) assumes that you have a 1kB blocksize.  All
ext2 filesystems larger than 500MB made in the last couple of years really
have 4kB blocksize** so e2fsck is far more likely to find a superblock
backup at 4kB * 32768 than at 1kB * 32768 (especially since there will
never be a backup at 1kB * 32768, but rather 1kB * 32769, and only on
_really_old_ non-sparse ext2 filesystems).

Cheers, Andreas

*) OK, that is a bit of a lie, e2fsck appears to check all valid blocksizes
   when a block number is given, but since one can assume it is a 4kB
   block size, you may as well specify it.
**) 4kB blocks provide _much_ better performance than 1kB blocks, even if
   they waste more space, and also allow for larger filesystems.
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/