Mount fails after reiserfsck --rebuild-tree failure

Mount fails after reiserfsck --rebuild-tree failure

[KAA]

hi there!
I'm running into a real big problem with one of my servers.
Very important data contained on its /dev/md0 which is RAID1
consisting of /dev/hda & /dev/hdc - both 120GB Seagate Barrackuda IV
Yesterday I had one of drive replaced with exactly the same one because 
of it's failure. After that mdadm with no complaints synced array and I 
mounted it without any problems. Later that day I've noticed several 
errors like following one in syslog:
  vs-5150: search_by_key: invalid format found in block 6377300. Fsck?
Mar 17 13:23:44 gate kernel: vs-13070: reiserfs_read_locked_inode: i/o 
failure occurred trying to find stat data of [13247 14
Mar 17 13:23:44 gate kernel: is_tree_node: node level 0 does not match 
to the expected one 1

so I've unmounted array fsck'ed it (unfortunately didn't save output of 
first fsck but it failed and recommended to do --rebuild-tree)
since that time fsck --rebuild-tree /dev/md0 fails with the same error 
all the time:


reiserfsck 3.6.13 (2003 www.namesys.com)

*************************************************************
** Do not  run  the  program  with  --rebuild-tree  unless **
** something is broken and MAKE A BACKUP  before using it. **
** If you have bad sectors on a drive  it is usually a bad **
** idea to continue using it. Then you probably should get **
** a working hard drive, copy the file system from the bad **
** drive  to the good one -- dd_rescue is  a good tool for **
** that -- and only then run this program.                 **
** If you are using the latest reiserfsprogs and  it fails **
** please  email bug reports to reiserfs-list@namesys.com, **
** providing  as  much  information  as  possible --  your **
** hardware,  kernel,  patches,  settings,  all reiserfsck **
** messages  (including version),  the reiserfsck logfile, **
** check  the  syslog file  for  any  related information. **
** If you would like advice on using this program, support **
** is available  for $25 at  www.namesys.com/support.html. **
*************************************************************

Will rebuild the filesystem (/dev/md0) tree
Will put log info to 'rebuild-tree.log'

Do you want to run this program?[N/Yes] (note need to type Yes if you 
do):Yes
Replaying journal..
0 transactions replayed
###########
reiserfsck --rebuild-tree started at Wed Mar 17 18:53:53 2004
###########

Pass 0:
Loading on-disk bitmap .. ok, 18587359 blocks marked used
Skipping 9105 blocks (super block, journal, bitmaps) 18578254 blocks 
will be read
0%....20%....40%....60%....80%....100%                        left 0, 
8914 /sec
         "r5" hash is selected
Flushing..finished
         Read blocks (but not data blocks) 18578254
                 Leaves among those 16268
                 Objectids found 55407

Pass 1 (will try to insert 16268 leaves):
Looking for allocable blocks ..
0%                                                         left 15846, 
422 /sec
The problem has occurred looks like a hardware problem (perhaps
memory). Send us the bug report only if the second run dies at
the same place with the same block number.

build_the_tree: Nothing but leaves are expected. Block 292951 - unknown

Aborted

rebuild-tree.log contains three lines:
####### Pass 0 #######
39673 directory entries were hashed with "r5" hash.
####### Pass 1 #######

is there any way to get my data back?
thanks in advance!

Re: Mount fails after reiserfsck --rebuild-tree failure

[jlewis]

On Wed, 17 Mar 2004, KAA wrote:

>   vs-5150: search_by_key: invalid format found in block 6377300. Fsck?
> Mar 17 13:23:44 gate kernel: vs-13070: reiserfs_read_locked_inode: i/o
> failure occurred trying to find stat data of [13247 14
> Mar 17 13:23:44 gate kernel: is_tree_node: node level 0 does not match
> to the expected one 1

This reminded me, when our server was having serious issues with fs
corruption, once we upgraded to a kernel that the corruption didn't crash,
we got lots of log messages like:

is_tree_node: node level 13624 does not match to the expected one 1
vs-5150: search_by_key: invalid format found in block 8451301. Fsck?
vs-13070: reiserfs_read_inode2: i/o failure occurred trying to find
stat data of [7228813 7505972 0x0 SD]

The trouble is, these errors don't tell you which partition is generating
the messages...so you have to unmount and reiserfsck them all to eliminate
them all.  It'd be nice if they'd at least prepend the device name to the
message, so you knew which partition was in trouble.

One last thing, before I leave the list...has it been considered to
restrict posting to list members?  It seems like I've gotten a couple
spams a day through this mailing list.  I don't think it's too much to ask
of someone seeking support here that they subscribe to the mailing list
before posting their pleas for help.  That's what I did...as I assumed it
was the right way to go.

----------------------------------------------------------------------
 Jon Lewis *jlewis@lewis.org*|  I route
 Senior Network Engineer     |  therefore you are
 Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________

Re: Mount fails after reiserfsck --rebuild-tree failure

[Vitaly Fertman]

Hello, 

> so I've unmounted array fsck'ed it (unfortunately didn't save output of
> first fsck but it failed and recommended to do --rebuild-tree)
> since that time fsck --rebuild-tree /dev/md0 fails with the same error
> all the time:
>
> build_the_tree: Nothing but leaves are expected. Block 292951 - unknown
>
> Aborted
>
> is there any way to get my data back?
> thanks in advance!

You rebuilt your raid and this problem appeared, this really looks 
like a hardware problem, check your raid again please.
If the problem in the code I can check how reiserfsck handles this 
block outside of your raid if you send it to me:
	dd if=/dev/md0 of=block.292951 count=1 bs=4096 skip=292951

-- 
Thanks,
Vitaly Fertman

Re: Mount fails after reiserfsck --rebuild-tree failure

[Vitaly Fertman]

On Thursday 18 March 2004 00:31, Vitaly Fertman wrote:
> Hello,
>
> > so I've unmounted array fsck'ed it (unfortunately didn't save output of
> > first fsck but it failed and recommended to do --rebuild-tree)
> > since that time fsck --rebuild-tree /dev/md0 fails with the same error
> > all the time:
> >
> > build_the_tree: Nothing but leaves are expected. Block 292951 - unknown
> >
> > Aborted
> >
> > is there any way to get my data back?
> > thanks in advance!
>
> You rebuilt your raid and this problem appeared, this really looks
> like a hardware problem, check your raid again please.
> If the problem in the code I can check how reiserfsck handles this
I mean   ... in the reiserfsprogs code ... 

-- 
Thanks,
Vitaly Fertman

Mount fails after reiserfsck --rebuild-tree failure

[Giuliano Luppi]

[-- Attachment #1: Type: text/plain, Size: 1171 bytes --]

Estimated sirs,
after experiencing some problems accessing files on a reiserfs file system
(SuSE linux 8.1, kernel 2.4), I tried reiserfsck –rebuild-tree.
Since the problem resulted to be in the hardware, it failed, leaving me with
an unmountable partition (root inode set to 0)

As suggested in you site’s FAQ (section 2), I tried with the latest
reiserfsprogs (3.6.7-pre1), installed on another HD (with RedHat). The
latter also failed.

Now I know the problems are (a lot of) bad blocks.

I would like to
1.	re-enable mounting the partition, in order to recover some of the data I
still have there (I know, I should backup more often… :-( )
2.	possibly fix the badblocks (as you describe in section 38 of the FAQ), in
order to bring up the system until new hard disks are ready.

The main question is: how can I undo the setting of root inode to 0?
Is there a good way for dealing with faulty hardware with reiserfs?

Thank you very much for your answers

Davide Imbeni

---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.489 / Virus Database: 288 - Release Date: 10/06/2003

Re: Mount fails after reiserfsck --rebuild-tree failure

[Oleg Drokin]

Hello!

On Thu, Jun 19, 2003 at 09:11:29PM +0200, Giuliano Luppi wrote:

> Now I know the problems are (a lot of) bad blocks.
> I would like to
> 1.	re-enable mounting the partition, in order to recover some of the data I
> still have there (I know, I should backup more often? :-( )

This is not possible to "reenable" it.

> 2.	possibly fix the badblocks (as you describe in section 38 of the FAQ), in
> order to bring up the system until new hard disks are ready.

This is not possible in your case, I'd say.
What you can do - is you can copy entire block device to a file/another device
and run rebuild-tree there. Use dd_rescue for copying.

> The main question is: how can I undo the setting of root inode to 0?

There is no such way, if you do not know what was the location of root node.

> Is there a good way for dealing with faulty hardware with reiserfs?

There is no good (reliable) way of dealing with faulty hardware with software, except avoiding
faulty hardware at all costs, I believe.

Bye,
    Oleg

Re: Mount fails after reiserfsck --rebuild-tree failure

[Davide Imbeni]

Hi Oleg,


Oleg Drokin wrote:

> This is not possible in your case, I'd say.
> What you can do - is you can copy entire block device to a file/another device
> and run rebuild-tree there. Use dd_rescue for copying.

I did as you suggested, and it worked fine.

Thank you VERY much for the timely advice!

BR

Davide


-- 
Davide Imbeni
Research Engineer

Ericsson Telecomunicazioni S.p.A.
Radio Network System Management
via Pio La Torre, 14/C
I-20090 Vimodrone
Milano - Italy

Tel: +39 02 26594 692 / +39 051 6115 329
Fax: +39 02 26594 635

Davide.Imbeni@tei.ericsson.se
www.ericsson.com

Re: Mount fails after reiserfsck --rebuild-tree failure

[Alexander Lyamin]

rule number 1,
as soon as you found that your harddisk is about to break - do 
dd_rescue to other disk, ASAP.

then proceed as ussual (--rebuild-tree) on copied image on proven
to be not faulty hardware.

what you did, running rebuild-tree on known-to-be-faulty hdd, is
sure way to get your data WASTED.

Thu, Jun 19, 2003 at 09:11:29PM +0200, Giuliano Luppi wrote:
> Estimated sirs,
> after experiencing some problems accessing files on a reiserfs file system
> (SuSE linux 8.1, kernel 2.4), I tried reiserfsck ?rebuild-tree.
> Since the problem resulted to be in the hardware, it failed, leaving me with
> an unmountable partition (root inode set to 0)
> 
> As suggested in you site?s FAQ (section 2), I tried with the latest
> reiserfsprogs (3.6.7-pre1), installed on another HD (with RedHat). The
> latter also failed.
> 
> Now I know the problems are (a lot of) bad blocks.
> 
> I would like to
> 1.	re-enable mounting the partition, in order to recover some of the data I
> still have there (I know, I should backup more often? :-( )
> 2.	possibly fix the badblocks (as you describe in section 38 of the FAQ), in
> order to bring up the system until new hard disks are ready.
> 
> The main question is: how can I undo the setting of root inode to 0?
> Is there a good way for dealing with faulty hardware with reiserfs?
> 
> Thank you very much for your answers
> 
> Davide Imbeni
> 
> ---
> Outgoing mail is certified Virus Free.
> Checked by AVG anti-virus system (http://www.grisoft.com).
> Version: 6.0.489 / Virus Database: 288 - Release Date: 10/06/2003

-- 
everything zen
lex lyamin