Procedure to recover a bad superblock

Procedure to recover a bad superblock

[John P. New]

Yesterday my UPS decided it didn't like the load it was carrying, and
unceremoniously shut off the power to my computer just as I was logging
into KDE. Upon reboot, I can no longer mount my /home partition. Here is
my system configuration:

Mandrake 9.1
Kernel 2.4.21-0.13mdk
Partitions:
hdg8      /boot      reiserfs
hdg9      /          reiserfs
hde5} md0 /mnt/LM82  reiserfs
hdg1}
hde6} md1 /home      reiserfs
hdg7}

ReiserFS 3.6.25
ReiserFSprogs 3.6.8

I know that the ReiserFS is up and running, because my md0 partition is
mounted and accessible.

I believe the following is the relevant portion of the /var/log/messages
for the next boot after the power was cut:

Aug 16 21:09:23 headoffice kernel: reiserfs: checking transaction log
(device 22:09) ...
Aug 16 21:09:23 headoffice kernel: Warning, log replay starting on
readonly filesystem
Aug 16 21:09:23 headoffice kernel: reiserfs: replayed 20 transactions in
6 seconds
Aug 16 21:09:23 headoffice kernel: Using r5 hash to sort names
Aug 16 21:09:23 headoffice kernel: ReiserFS version 3.6.25
(snip)

Aug 16 21:09:23 headoffice kernel: reiserfs: checking transaction log
(device 22:08) ...
Aug 16 21:09:23 headoffice kernel: reiserfs: replayed 2 transactions in
1 seconds
Aug 16 21:09:23 headoffice kernel: Using r5 hash to sort names
Aug 16 21:09:23 headoffice kernel: ReiserFS version 3.6.25
Aug 16 21:09:23 headoffice kernel: reiserfs: checking transaction log
(device 09:01) ...
Aug 16 21:09:23 headoffice kernel: reiserfs: replayed 4 transactions in
1 seconds
Aug 16 21:09:23 headoffice kernel: attempt to access beyond end of
device
Aug 16 21:09:23 headoffice kernel: 09:01: rw=0, want=1370749188,
(=0x51b3f904), limit=59447552
Aug 16 21:09:23 headoffice kernel: vs-13070: reiserfs_read_inode2: i/o
failure occurred trying to find stat data of [1 2 0x0 SD]
Aug 16 21:09:23 headoffice kernel: invalidate: busy buffer
Aug 16 21:09:23 headoffice last message repeated 2 times
Aug 16 21:09:23 headoffice kernel: reiserfs: checking transaction log
(device 09:00) ...
Aug 16 21:09:23 headoffice kernel: Using r5 hash to sort names
Aug 16 21:09:23 headoffice kernel: ReiserFS version 3.6.25
(snip)

Aug 16 21:09:27 headoffice kernel: read_super_block: can't find a
reiserfs filesystem on (dev 09:01, block 16, size 4096)
Aug 16 21:09:27 headoffice kernel: read_super_block: can't find a
reiserfs filesystem on (dev 09:01, block 2, size 4096)
Aug 16 21:09:27 headoffice mount: mount: wrong fs type, bad option, bad
superblock on /dev/md1,
Aug 16 21:09:27 headoffice mount:        or too many mounted file
systems
(snip)

When I try 'mount /dev/md1 -t reiserfs /home' I get the "mount: wrong fs
type..." error and the "read_super_block: can't find a reiserfs..."
entry in /var/log/messages.

I have done a 'reiserfsck --check --logfile /root/rfscheck.log /dev/md1'
with the following result:

reiserfs_open: the reiserfs superblock cannot be found on /dev/md1.
Failed to open the filesystem.
If the partition table has not been changed, and the partition is
valid  and  it really  contains  a reiserfs  partition,  then the
superblock  is corrupted and you need to run  --rebuild-sb.
Aborted (core dumped)

So, I guess I have a bad superblock on md0. I assume my next step would
be:

reiserfsck --rebuild-sb /dev/md1

Is there anything I should do to protect my data before I run this
command, and is there a possibiltity of file corruption at this point,
or is this a fairly safe step?

Thanks in advance,
-- 
John P. New <jnew@hazelden.ca>

Re: Procedure to recover a bad superblock

[Oleg Drokin]

Hello!

On Sun, Aug 17, 2003 at 10:52:16AM -0400, John P. New wrote:

> Aug 16 21:09:23 headoffice kernel: Using r5 hash to sort names
> Aug 16 21:09:23 headoffice kernel: ReiserFS version 3.6.25
> Aug 16 21:09:23 headoffice kernel: reiserfs: checking transaction log
> (device 09:01) ...
> Aug 16 21:09:23 headoffice kernel: reiserfs: replayed 4 transactions in
> 1 seconds
> Aug 16 21:09:23 headoffice kernel: attempt to access beyond end of
> device
> Aug 16 21:09:23 headoffice kernel: 09:01: rw=0, want=1370749188,
> (=0x51b3f904), limit=59447552

Hm, so you had valid superblock, but the copy in journal was not valid,
do you have some kind of IDE drives with write cache turned on below
your softraid?

> reiserfs_open: the reiserfs superblock cannot be found on /dev/md1.
> Failed to open the filesystem.
> If the partition table has not been changed, and the partition is
> valid  and  it really  contains  a reiserfs  partition,  then the
> superblock  is corrupted and you need to run  --rebuild-sb.
> Aborted (core dumped)
> So, I guess I have a bad superblock on md0. I assume my next step would
> be:
> reiserfsck --rebuild-sb /dev/md1

Yes.
It will ask you few questions and then will create new superblock.

> Is there anything I should do to protect my data before I run this

Well, you may create copy of entire device with dd, if you want.

> command, and is there a possibiltity of file corruption at this point,
> or is this a fairly safe step?

If only superblock is corrupted, that should be pretty safe.

Bye,
    Oleg

Re: Procedure to recover a bad superblock

[John P. New]

Oleg,

Thanks for the response.

As for what ide devices I have below the softraid, I'm not sure what you
mean. I have a CD/RW on hda and a Colorado Travan tape drive on hdb, and
other, non-RAID partitions on hde and hdg, if that's what you mean.

I ran reiserfsck --rebuild-sb /dev/md1, and it asked me what block size
to use. I looked at /var/log/messages, and the first boot after the
power outage log reads:

Aug 16 21:09:27 headoffice kernel: read_super_block: can't find a
reiserfs filesystem on (dev 09:01, block 16, size 4096)
Aug 16 21:09:27 headoffice kernel: read_super_block: can't find a
reiserfs filesystem on (dev 09:01, block 2, size 4096)

In more recent boots (and when I try a manual mount), it reads:

Aug 17 10:28:31 headoffice kernel: read_super_block: can't find a
reiserfs filesystem on (dev 09:01, block 64, size 1024)
Aug 17 10:28:31 headoffice kernel: read_super_block: can't find a
reiserfs filesystem on (dev 09:01, block 8, size 1024)

In other words, different block sizes. for the same device. Should I be
concerned, and what value should I use for reiserfsck?

John


On Mon, 2003-08-18 at 02:19, Oleg Drokin wrote:
> Hello!
> 
> On Sun, Aug 17, 2003 at 10:52:16AM -0400, John P. New wrote:
> 
> > Aug 16 21:09:23 headoffice kernel: Using r5 hash to sort names
> > Aug 16 21:09:23 headoffice kernel: ReiserFS version 3.6.25
> > Aug 16 21:09:23 headoffice kernel: reiserfs: checking transaction log
> > (device 09:01) ...
> > Aug 16 21:09:23 headoffice kernel: reiserfs: replayed 4 transactions in
> > 1 seconds
> > Aug 16 21:09:23 headoffice kernel: attempt to access beyond end of
> > device
> > Aug 16 21:09:23 headoffice kernel: 09:01: rw=0, want=1370749188,
> > (=0x51b3f904), limit=59447552
> 
> Hm, so you had valid superblock, but the copy in journal was not valid,
> do you have some kind of IDE drives with write cache turned on below
> your softraid?
> 
> > reiserfs_open: the reiserfs superblock cannot be found on /dev/md1.
> > Failed to open the filesystem.
> > If the partition table has not been changed, and the partition is
> > valid  and  it really  contains  a reiserfs  partition,  then the
> > superblock  is corrupted and you need to run  --rebuild-sb.
> > Aborted (core dumped)
> > So, I guess I have a bad superblock on md0. I assume my next step would
> > be:
> > reiserfsck --rebuild-sb /dev/md1
> 
> Yes.
> It will ask you few questions and then will create new superblock.
> 
> > Is there anything I should do to protect my data before I run this
> 
> Well, you may create copy of entire device with dd, if you want.
> 
> > command, and is there a possibiltity of file corruption at this point,
> > or is this a fairly safe step?
> 
> If only superblock is corrupted, that should be pretty safe.
> 
> Bye,
>     Oleg

Re: Procedure to recover a bad superblock

[Oleg Drokin]

Hello!

On Mon, Aug 18, 2003 at 07:43:48AM -0400, John P. New wrote:

> As for what ide devices I have below the softraid, I'm not sure what you
> mean. I have a CD/RW on hda and a Colorado Travan tape drive on hdb, and

I mean on what devices do you build your softraid.

> I ran reiserfsck --rebuild-sb /dev/md1, and it asked me what block size
> to use. I looked at /var/log/messages, and the first boot after the
> power outage log reads:

blocksize is always 4k

> In other words, different block sizes. for the same device. Should I be
> concerned, and what value should I use for reiserfsck?

Sure.
Always givre 4k for now.

Bye,
    Oleg

Re: Procedure to recover a bad superblock

[John P. New]

Oleg,

I ran reiserfsck --rebuild-sb /dev/md1 and got (edited):

reiserfsck, 2003 - reiserfsprogs 3.6.8  

Will check superblock and rebuild it if needed
Will put log info to 'stdout'

Do you want to run this program?[N/Yes] (note need to type Yes if you
do):Yes

reiserfs_open: the reiserfs superblock cannot be found on /dev/md1.

what is version of ReiserFS you use[1-4]
        (1)   3.6.x
        (2) >=3.5.9 (introduced in the middle of 1999) (if you use linux
2.2, choose this one)
        (3) < 3.5.9 converted to new format (don't choose if unsure)
        (4) < 3.5.9 (this is very old format, don't choose if unsure)
        (X)   exit
1

Enter block size [4096]:
4096

No journal device was specified. (If journal is not available, re-run
with --no-journal-available).
Is journal standard? (y/n)[y]: y

Did you use resiser(y/n)[n]: n
rebuild-sb: no uuid found, a new uuid generated
(5f705731-5fa9-4c16-b5fe-2b0a64ea6a60)
Reiserfs super block in block 16 on 0x901 of format 3.6 with standard
journal
Count of blocks on the device: 14861888
Number of bitmaps: 454
Blocksize: 4096
Free blocks (count of blocks - used [journal, bitmaps, data, reserved]
blocks): 0
Root block: 0
Filesystem is NOT cleanly umounted
Tree height: 0
Hash function used to sort names: not set
Objectid map size 0, max 972
Journal parameters:
        Device [0x0]
        Magic [0x0]
        Size 8193 blocks (including 1 for journal header) (first block
18)
        Max transaction length 1024 blocks
        Max batch size 900 blocks
        Max commit age 30
Blocks reserved by journal: 0
Fs state field: 0x1:
         some corruptions exist.
sb_version: 2
inode generation number: 0
UUID: 5f705731-5fa9-4c16-b5fe-2b0a64ea6a60
LABEL:
Set flags in SB:
Is this ok ? (y/n)[n]: y
The fs may still be unconsistent. Run reiserfsck --check.



And then I ran reiserfsck --check --logfile /root/rfscheck.log /dev/md1
and got (edited):

reiserfsck, 2003 - reiserfsprogs 3.6.8  
Will read-only check consistency of the filesystem on /dev/md1
Will put log info to '/root/rfscheck.log'

Do you want to run this program?[N/Yes] (note need to type Yes if you
do):Yes
###########
reiserfsck --check started at Mon Aug 18 09:59:43 2003
###########
Replaying journal..
0 transactions replayed
Checking internal tree..

Bad root block 0. (--rebuild-tree did not complete)

Aborted (core dumped)



Now what? Is it time for a --rebuild-tree, or are we past that because
of the 'Bad root block 0. (--rebuild-tree did not complete)' error?

John



On Mon, 2003-08-18 at 09:33, Oleg Drokin wrote:
> Hello!
> 
> On Mon, Aug 18, 2003 at 07:43:48AM -0400, John P. New wrote:
> 
> > As for what ide devices I have below the softraid, I'm not sure what you
> > mean. I have a CD/RW on hda and a Colorado Travan tape drive on hdb, and
> 
> I mean on what devices do you build your softraid.
> 
> > I ran reiserfsck --rebuild-sb /dev/md1, and it asked me what block size
> > to use. I looked at /var/log/messages, and the first boot after the
> > power outage log reads:
> 
> blocksize is always 4k
> 
> > In other words, different block sizes. for the same device. Should I be
> > concerned, and what value should I use for reiserfsck?
> 
> Sure.
> Always givre 4k for now.
> 
> Bye,
>     Oleg

Re: Procedure to recover a bad superblock

[Oleg Drokin]

Hello!

On Mon, Aug 18, 2003 at 10:22:47AM -0400, John P. New wrote:

[...]

> Set flags in SB:
> Is this ok ? (y/n)[n]: y
> The fs may still be unconsistent. Run reiserfsck --check.
> And then I ran reiserfsck --check --logfile /root/rfscheck.log /dev/md1
> and got (edited):

[...]
 
> Bad root block 0. (--rebuild-tree did not complete)
> Aborted (core dumped)

> Now what? Is it time for a --rebuild-tree, or are we past that because
> of the 'Bad root block 0. (--rebuild-tree did not complete)' error?

Well, since your superblock was completely destroyed, we no longer know where rootblock is.
So you need to run reiserfsck --rebuild-tree now

Bye,
    Oleg