superblock error after power failure ---on an old system

superblock error after power failure ---on an old system

[Wierdl Máté]

Hello,

I have an old Dell Poweredge 2500 server running Fedora Core 3. The
server  had one of its power units failed, and (I think) as a result,
the raid arrays got damaged: three of them stayed alive (though
degraded), but one is not accessible. Unfortunately, the server only
has a floppy drive which doesn't work now, and so I cannot get any
files, command outputs directly off the box.  I copy it all from the
screen.

The array I cannot access is md3. It has the shared libraries on it so
many commands do not work. In particular, I cannot have remote access
to the box.

 The output of

mdadm -Es

is (I am not copying the UUIDs)

ARRAY /dev/md0 level=raid1 num-devices=2 devices=/dev/sdb2,/dev/sdc2
ARRAY /dev/md2 level=raid1 num-devices=2 devices=/dev/sda6,/dev/sdc3
ARRAY /dev/md3 level=raid5 num-devices=3 devices=/dev/sda3,/dev/sdb1,/dev/sdc1
ARRAY /dev/md1 level=raid1 num-devices=2 devices=/dev/sda1,/dev/sda2

Also

md0  mounted at /
md2 mounted at /var
md1  mounted at /boot
md3 should be mounted at /home

# cat /proc/mdstat
Personalities: raid1 raid5
md1: active raid1 sda2[1] sda1[0]
         104320 blocks [2/2] [UU]

md2: active raid1 sda6[0]
         513984 blocks [2/1] [U_]

md0: active raid1 sdc2[1]
        1534080 blocks [2/1] [_U]

unused devices: none

# mdadm --detail /dev/md3
mdadm: md device /dev/md3 does not appear to be active

At somebody's suggestion I tried

mdadm --assemble --scan /dev/md0

and got

/dev/sda has no superblock - assembly aborted

Then tried

mdadm  --force --run /dev/md3

and got

failed to run array /dev/md3: Invalid argument

I finally tried

# mdadm --assemble /dev/md3 /dev/sda3 /dev/sdb1 /dev/sdc1
mdadm: device 1 in  /dev/md3 has wrong state superblock, but /dev/sdb1 seems ok

Thanks for any help. I certainly would give remote access to a helping
hand, but unfortunately the box cannot run sshd: the shared libraries
are on the md3 array.

Máté

Máté Wierdl
Dept of Math Sci, Univ of Memphis
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: superblock error after power failure ---on an old system

[Neil Brown]

On Sat, 24 Apr 2010 06:18:57 -0500
Wierdl Máté <wierdlmate@gmail.com> wrote:

> Hello,
> 
> I have an old Dell Poweredge 2500 server running Fedora Core 3. The
> server  had one of its power units failed, and (I think) as a result,
> the raid arrays got damaged: three of them stayed alive (though
> degraded), but one is not accessible. Unfortunately, the server only
> has a floppy drive which doesn't work now, and so I cannot get any
> files, command outputs directly off the box.  I copy it all from the
> screen.
> 
> The array I cannot access is md3. It has the shared libraries on it so
> many commands do not work. In particular, I cannot have remote access
> to the box.
> 
>  The output of
> 
> mdadm -Es
> 
> is (I am not copying the UUIDs)
> 
> ARRAY /dev/md0 level=raid1 num-devices=2 devices=/dev/sdb2,/dev/sdc2
> ARRAY /dev/md2 level=raid1 num-devices=2 devices=/dev/sda6,/dev/sdc3
> ARRAY /dev/md3 level=raid5 num-devices=3 devices=/dev/sda3,/dev/sdb1,/dev/sdc1
> ARRAY /dev/md1 level=raid1 num-devices=2 devices=/dev/sda1,/dev/sda2
> 
> Also
> 
> md0  mounted at /
> md2 mounted at /var
> md1  mounted at /boot
> md3 should be mounted at /home
> 
> # cat /proc/mdstat
> Personalities: raid1 raid5
> md1: active raid1 sda2[1] sda1[0]
>          104320 blocks [2/2] [UU]
> 
> md2: active raid1 sda6[0]
>          513984 blocks [2/1] [U_]
> 
> md0: active raid1 sdc2[1]
>         1534080 blocks [2/1] [_U]
> 
> unused devices: none
> 
> # mdadm --detail /dev/md3
> mdadm: md device /dev/md3 does not appear to be active
> 
> At somebody's suggestion I tried
> 
> mdadm --assemble --scan /dev/md0
> 
> and got
> 
> /dev/sda has no superblock - assembly aborted
> 
> Then tried
> 
> mdadm  --force --run /dev/md3
> 
> and got
> 
> failed to run array /dev/md3: Invalid argument
> 
> I finally tried
> 
> # mdadm --assemble /dev/md3 /dev/sda3 /dev/sdb1 /dev/sdc1
> mdadm: device 1 in  /dev/md3 has wrong state superblock, but /dev/sdb1 seems ok

That looks close.  Just add "--force" to that and it will probably get
you going.

NeilBrown


> 
> Thanks for any help. I certainly would give remote access to a helping
> hand, but unfortunately the box cannot run sshd: the shared libraries
> are on the md3 array.
> 
> Máté
> 
> Máté Wierdl
> Dept of Math Sci, Univ of Memphis
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: superblock error after power failure ---on an old system

[Wierdl Máté]

2010/4/24 Neil Brown <neilb@suse.de>
>
> On Sat, 24 Apr 2010 06:18:57 -0500
> Wierdl Máté <wierdlmate@gmail.com> wrote:
>
> > Hello,
> >
> > I have an old Dell Poweredge 2500 server running Fedora Core 3. The
> > server  had one of its power units failed, and (I think) as a result,
> > the raid arrays got damaged: three of them stayed alive (though
> > degraded), but one is not accessible. Unfortunately, the server only
> > has a floppy drive which doesn't work now, and so I cannot get any
> > files, command outputs directly off the box.  I copy it all from the
> > screen.
> >
> > The array I cannot access is md3. It has the shared libraries on it so
> > many commands do not work. In particular, I cannot have remote access
> > to the box.
> >
> >  The output of
> >
> > mdadm -Es
> >
> > is (I am not copying the UUIDs)
> >
> > ARRAY /dev/md0 level=raid1 num-devices=2 devices=/dev/sdb2,/dev/sdc2
> > ARRAY /dev/md2 level=raid1 num-devices=2 devices=/dev/sda6,/dev/sdc3
> > ARRAY /dev/md3 level=raid5 num-devices=3 devices=/dev/sda3,/dev/sdb1,/dev/sdc1
> > ARRAY /dev/md1 level=raid1 num-devices=2 devices=/dev/sda1,/dev/sda2
> >
> > Also
> >
> > md0  mounted at /
> > md2 mounted at /var
> > md1  mounted at /boot
> > md3 should be mounted at /home
> >
> > # cat /proc/mdstat
> > Personalities: raid1 raid5
> > md1: active raid1 sda2[1] sda1[0]
> >          104320 blocks [2/2] [UU]
> >
> > md2: active raid1 sda6[0]
> >          513984 blocks [2/1] [U_]
> >
> > md0: active raid1 sdc2[1]
> >         1534080 blocks [2/1] [_U]
> >
> > unused devices: none
> >
> > # mdadm --detail /dev/md3
> > mdadm: md device /dev/md3 does not appear to be active
> >
> > At somebody's suggestion I tried
> >
> > mdadm --assemble --scan /dev/md0
> >
> > and got
> >
> > /dev/sda has no superblock - assembly aborted
> >
> > Then tried
> >
> > mdadm  --force --run /dev/md3
> >
> > and got
> >
> > failed to run array /dev/md3: Invalid argument
> >
> > I finally tried
> >
> > # mdadm --assemble /dev/md3 /dev/sda3 /dev/sdb1 /dev/sdc1
> > mdadm: device 1 in  /dev/md3 has wrong state superblock, but /dev/sdb1 seems ok
>
> That looks close.  Just add "--force" to that and it will probably get
> you going.
>
> NeilBrown
>
>

Well, I now get, after rebooting:

/dev/md3: UNEXPECTED INCONSISTENCY; run fsck manually

So I did

fsck -y /dev/md3

and after running for about a minute or so, it gets into this

Too many illegal blocks in inode 8

and

Illegal block #6828 (39485860) in inode 8. CLEARED

Several lines of these, and then again

Too many illegal blocks in inode 8

The block number (#6828) changes but this loop never seems to end. I
aborted this, for now.  When I tried

mount /dev/md3 /home

I got: wrong fs type, bad option, bad superblock on md3

Máté
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: superblock error after power failure ---on an old system

[Wierdl Máté]

> Well, I now get, after rebooting:
>
> /dev/md3: UNEXPECTED INCONSISTENCY; run fsck manually
>
> So I did
>
> fsck -y /dev/md3
>
> and after running for about a minute or so, it gets into this
>
> Too many illegal blocks in inode 8
>
> and
>
> Illegal block #6828 (39485860) in inode 8. CLEARED
>
> Several lines of these, and then again
>
> Too many illegal blocks in inode 8
>
> The block number (#6828) changes but this loop never seems to end. I
> aborted this, for now.  When I tried
>
> mount /dev/md3 /home
>
> I got: wrong fs type, bad option, bad superblock on md3
>
I ran fsck on /dev/md0 again, answered 'n' to the question if I wanted
to clear inode 8, and it gets into the following infinite loop

Directory inode 83161, block 0, offset 0: directory corrupted.
Salvage<y>?

If I answer 'y' then it asks the very same thing about inode 83161,
and if I answer n, then I get

e2fsck aborted.

Máté




-- 
Matyi
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html