ReiserFS and RAID5

ReiserFS and RAID5

[Sven Heinicke]

We had a drive go bad on a RAID5 with reiserfs on it.  The file system
was built with reiserfsprogs-3.x.0h tools and the systems was running
Linux 2.4.13.  As we had an issue it will be updated to the latest
kernel, it had been stable up to now.

A drive failed and left the partition in a funk.  When I ran ls in the
RAID directory it would freeze up the ls (I suspect in a hardware wait
of some kind).  I uncommented the partition from the fstab file then
tied to shut down the system, but that froze up that system and I hit
the reset key.

The system came up, the raid started rebuilding itsself with the spare
drive.  I tried to mount the drive and it didn't mount.  I updated my
reiserfs tools to reiserfsprogs-3.x.0j.  I ran reiserfsck on the
partition, I wish I kept the exact error message but didn't, it said
something was wrong with the tree and segfaulted.  I then ran it with
--rebuild-tree and went home.

The next morning the raid rebuild and the fsck was finished (should of
I waited for the raid rebuild to finish before running reiserfsck?).
I mounted the disk read only.  The df command reported 1% full when
before it was like 35% full.

But, all was not lost.  inspecting the partition all the data seemed
to be good.  We hurriedly we copied the files to another partition,
took another night, oddly one directory didn't get copied. 

Then some testing:

1. umounted to mounted /mnt/raid0 a coupld of time in read only mode.
   (nothing changed).

2. mounted in read-write.
   (nothing changed)

3. touched /mnt/raid0/foo
   (nothing changed)

4. rm /mnt/raid0/foo
   (nothing changed in df).  Lost a whole bunch of data according to
   du and other programs.  Specifically, we were able to copy:

121M	  scoutabout/08Oct01
59G	  scoutabout/21Nov01
38G	  scoutabout/23Jul01
5.4G	  scoutabout/23Jul01Output
65G	  scoutabout/27Nov01
4.1G	  scoutabout/29Jun01

But now the corrupted file system reads:

121M	/mnt/raid0/scoutabout/08Oct01
1.0k	/mnt/raid0/scoutabout/12Nov01
59G	/mnt/raid0/scoutabout/21Nov01
38G	/mnt/raid0/scoutabout/23Jul01
238M	/mnt/raid0/scoutabout/23Jul01Output
65G	/mnt/raid0/scoutabout/27Nov01
4.1G	/mnt/raid0/scoutabout/29Jun01

and that is the state we are in.  At least most of our data is saved.
Did I do anything wrong that might of been able to keep the RAID
stable after the drive crash?  Would reiser developers want me to try
anything on it to help them debug it and make the support more stable.

Thanks,

 Sven

Re: ReiserFS and RAID5

[Stephan von Krawczynski]

On Thu, 24 Jan 2002 13:46:07 -0500 (EST)
Sven Heinicke <sven@research.nj.nec.com> wrote:

> 
> We had a drive go bad on a RAID5 with reiserfs on it.  The file system
> was built with reiserfsprogs-3.x.0h tools and the systems was running
> Linux 2.4.13.  As we had an issue it will be updated to the latest
> kernel, it had been stable up to now.

Would you mind to be a bit more specific about the hardware involved, 
especially the RAID5, SMP etc?
This is not HW-RAID5, is it?

Regards,
Stephan

Re: ReiserFS and RAID5

[Sven Heinicke]

Stephan von Krawczynski writes:
 > On Thu, 24 Jan 2002 13:46:07 -0500 (EST)
 > Sven Heinicke <sven@research.nj.nec.com> wrote:
 > 
 > > 
 > > We had a drive go bad on a RAID5 with reiserfs on it.  The file system
 > > was built with reiserfsprogs-3.x.0h tools and the systems was running
 > > Linux 2.4.13.  As we had an issue it will be updated to the latest
 > > kernel, it had been stable up to now.
 > 
 > Would you mind to be a bit more specific about the hardware involved, 
 > especially the RAID5, SMP etc?
 > This is not HW-RAID5, is it?
 > 


Software RAID .. Yes

System got two:

vendor_id	: GenuineIntel
model name	: Pentium III (Coppermine)
cpu MHz		: 868.671
cache size	: 256 KB

It's an ASLab system with 4 Ultra100 Cards each with 4 80G maxtor
drives.  So with the video and network card the PCI slots are full.
The raid tools are raidtools-0.90-9mdk as shipped with Mandrake 7.2.

Re: ReiserFS and RAID5

[Stephan von Krawczynski]

On Thu, 24 Jan 2002 14:14:30 -0500 (EST)
Sven Heinicke <sven@research.nj.nec.com> wrote:

> Software RAID .. Yes
> 
> System got two:
> 
> vendor_id	: GenuineIntel
> model name	: Pentium III (Coppermine)
> cpu MHz		: 868.671
> cache size	: 256 KB
> 
> It's an ASLab system with 4 Ultra100 Cards each with 4 80G maxtor
> drives.  So with the video and network card the PCI slots are full.
> The raid tools are raidtools-0.90-9mdk as shipped with Mandrake 7.2.

Ok. If I get that right your report means that SW RAID5 is just broken,
because if it wasn't reiserfs must have staid alive, but it just hung.
I guess any other fs would have hung, too.
And afterwards you simply fell into the obvious problem, that reiserfs
must heavily rely on the underlying "hw" and gets completely confused
if the lower layer is trashed for whatever reason.

Is anybody out there that ever survived a hd crash in SW RAID5 config?
(Meaning _without_ need to reboot)

(I only try to figure out what is going on, because I am heavily 
interested in building RAID5 configs myself, and you would not want to
touch components (SW or HW) in this case that you cannot not trust)

Regards,
Stephan

Re: ReiserFS and RAID5

[Simen Thoresen]

>Is anybody out there that ever survived a hd crash in SW RAID5 config?
>(Meaning _without_ need to reboot)

Survived and survived....

I had a drive lose power (flaky connector) in a 4 drive setup (2x hpt370 dual channel controllers, each with a 30G IBM ide-drive on each channel - Using IDE and Raid-patches for 2.2.17). The result was a readable, but unwritable array. A streamed mp3 played flawlessly while the drive 'died', while my download (over smb) just froze.

Rebuilding the array (after a reboot) went ok, but during rebuild the 'new' drive appeared to be drive #5 in the 4-disk set, with drive #3 still dead. 

>(I only try to figure out what is going on, because I am heavily 
>interested in building RAID5 configs myself, and you would not want to
>touch components (SW or HW) in this case that you cannot not trust)
>

Seems to work ok on 2.2.17 at least (had problems with 2.2.19 as detailed previously) - untried with 2.2.20 or 2.4 so far.

It's fun tho :-)

-S
--
Simen Thoresen, Beowulf-cleaner and random artist - close and personal.

Er det ikke rart?
The gnu RART-project on http://valinor.dolphinics.no:1080/~simentt/rart

Re: ReiserFS and RAID5

[Stephan von Krawczynski]

On Thu, 24 Jan 2002 21:29:15 +0100
"Simen Thoresen" <simen-tt@online.no> wrote:

> >Is anybody out there that ever survived a hd crash in SW RAID5 config?
> >(Meaning _without_ need to reboot)
> 
> Survived and survived....
> 
> I had a drive lose power (flaky connector) in a 4 drive setup (2x hpt370 dual channel controllers, each with a 30G IBM ide-drive on each channel - Using IDE and Raid-patches for 2.2.17). The result was a readable, but unwritable array. A streamed mp3 played flawlessly while the drive 'died', while my download (over smb) just froze.
> 
> Rebuilding the array (after a reboot) went ok, but during rebuild the 'new' drive appeared to be drive #5 in the 4-disk set, with drive #3 still dead. 

Why is it not writeable? What about a runtime fallover to a spare disk?

Regards,
Stephan

Re: ReiserFS and RAID5

[Sven Heinicke]

Stephan von Krawczynski writes:
 > On Thu, 24 Jan 2002 21:29:15 +0100
 > "Simen Thoresen" <simen-tt@online.no> wrote:
 > 
 > > >Is anybody out there that ever survived a hd crash in SW RAID5 config?
 > > >(Meaning _without_ need to reboot)
 > > 
 > > Survived and survived....
 > > 
 > > I had a drive lose power (flaky connector) in a 4 drive setup (2x hpt370 dual channel controllers, each with a 30G IBM ide-drive on each channel - Using IDE and Raid-patches for 2.2.17). The result was a readable, but unwritable array. A streamed mp3 played flawlessly while the drive 'died', while my download (over smb) just froze.
 > > 
 > > Rebuilding the array (after a reboot) went ok, but during rebuild the 'new' drive appeared to be drive #5 in the 4-disk set, with drive #3 still dead. 
 > 
 > Why is it not writeable? What about a runtime fallover to a spare disk?
 > 

runtime fallover to the spare disk did not work for me until the
reboot.  And even then after the reboot my data was recoverable, but
as describe by my first post, didn't last for long.

    Sven