Fatal File System Corruption -Software RAID + NFS

Fatal File System Corruption -Software RAID + NFS

[Steven Poulakos]

Hi,

I'm encountering my second wave of reiserfs filesystem corruption with a 
Web/NFS/NIS server.  This time, I had fatal corruption after updating 
the Kernel.  Below are the system specs and the error message that I'm 
receiving.

The system:

Debian (latest unstable)
2.4.22 Kernel  (not patched in any way)
running NFS (from the kernel), which serves home directories to 3 
workstations
Software RAID 1
-2, mirrored 80gig (7200RPM) IBM hard drives
-Each drive is connected to its own channel on a Promise Ultra ATA/100 
Controller (Ultra100 Tx2) card (PCI)  (using ATA 100 cables)
-the two drives are called hda and hdc (contain 3 mirrored RAID partitions)

The problem:

Two of three RAID'ed partitions have been working without errors (/home 
and /usr).  My root partition generated the errors below for many days 
until  detected.  After a reboot, the system would not load.  reiserfsck 
--rebuild-tree could not recover the corruption.  I have assumed that I 
must rebuild the system now, and being able to prevent the errors below 
from happening again will affect how I proceed with the rebuild.  Any 
info for prevent the below errors would be great!

-Steve

Here are snippets of the errors that just repeat over and over...

Nov  9 06:47:34 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekComplete DataRequest }
Nov  9 06:47:34 ctdev kernel:
Nov  9 06:47:34 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:47:34 ctdev kernel:
Nov  9 06:47:34 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:47:34 ctdev kernel:
Nov  9 06:47:39 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:47:39 ctdev kernel:
Nov  9 06:47:39 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:47:39 ctdev kernel:
Nov  9 06:47:44 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:47:44 ctdev kernel:
Nov  9 06:47:44 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
.....(snip).......

.....(snip).......
Nov  9 06:47:59 ctdev kernel:
Nov  9 06:48:05 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:48:05 ctdev kernel:
Nov  9 06:48:05 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:48:05 ctdev kernel:
Nov  9 06:48:05 ctdev kernel: hdc: status timeout: status=0xd0 { Busy }
Nov  9 06:48:05 ctdev kernel:
Nov  9 06:48:05 ctdev kernel: PDC202XX: Secondary channel reset.
Nov  9 06:48:05 ctdev kernel: ide1: reset: success
Nov  9 06:48:10 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:48:10 ctdev kernel:
Nov  9 06:48:10 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:48:10 ctdev kernel:
Nov  9 06:48:10 ctdev kernel: hdc: write_intr error1: nr_sectors=5, 
stat=0xd0
Nov  9 06:48:10 ctdev kernel: hdc: write_intr: status=0xd0 { Busy }
Nov  9 06:48:10 ctdev kernel:
Nov  9 06:48:10 ctdev kernel: PDC202XX: Secondary channel reset.
Nov  9 06:48:10 ctdev kernel: ide1: reset: success
Nov  9 06:48:15 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
.....(snip).......


.....(snip).......
Nov  9 06:56:45 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:56:45 ctdev kernel:
Nov  9 06:56:50 ctdev kernel: hdc: write_intr error1: nr_sectors=51, 
stat=0xd0
Nov  9 06:56:50 ctdev kernel: hdc: write_intr: status=0xd0 { Busy }
Nov  9 06:56:50 ctdev kernel:
Nov  9 06:56:50 ctdev kernel: PDC202XX: Secondary channel reset.
Nov  9 06:56:50 ctdev kernel: ide1: reset: success
Nov  9 06:56:56 ctdev kernel: hdc: write_intr error1: nr_sectors=51, 
stat=0xd0
Nov  9 06:56:56 ctdev kernel: hdc: write_intr: status=0xd0 { Busy }
Nov  9 06:56:56 ctdev kernel:
Nov  9 06:56:56 ctdev kernel: PDC202XX: Secondary channel reset.
Nov  9 06:56:56 ctdev kernel: ide1: reset: success
Nov  9 06:56:56 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:56:56 ctdev kernel:
Nov  9 06:57:01 ctdev kernel: hdc: status error: status=0x58 { 
DriveReady SeekCo
mplete DataRequest }
Nov  9 06:57:01 ctdev kernel:
..... (snip) .....

Re: Fatal File System Corruption -Software RAID + NFS

[Christian Kujau]

Steven Poulakos wrote:
[...]
> Here are snippets of the errors that just repeat over and over...
> 
> Nov  9 06:47:34 ctdev kernel: hdc: status error: status=0x58 { 
> DriveReady SeekComplete DataRequest }
> Nov  9 06:47:34 ctdev kernel:
> Nov  9 06:47:34 ctdev kernel: hdc: status error: status=0x58 { 
> DriveReady SeekCo

hardware errors, i'd say :-(
try to dd / dd_rescue to a working disk, then go on with newest 
reiserfsprogs or /bin/cat...

Christian.
-
BOFH excuse #6:

global warming

Re: Fatal File System Corruption -Software RAID + NFS

[Dan Oglesby]

Christian Kujau wrote:
> Steven Poulakos wrote:
> [...]
> 
>> Here are snippets of the errors that just repeat over and over...
>>
>> Nov  9 06:47:34 ctdev kernel: hdc: status error: status=0x58 { 
>> DriveReady SeekComplete DataRequest }
>> Nov  9 06:47:34 ctdev kernel:
>> Nov  9 06:47:34 ctdev kernel: hdc: status error: status=0x58 { 
>> DriveReady SeekCo
> 
> 
> hardware errors, i'd say :-(
> try to dd / dd_rescue to a working disk, then go on with newest 
> reiserfsprogs or /bin/cat...
> 
> Christian.
> -

I saw these types of errors on my software RAID-5 array when I first 
brought it online.  The array is made up of four 20GB hard drives from 
four different manufacturers, two drives to an IDE channel.  Some of the 
drives didn't want to play nice when sharing the IDE channel.

Using hdparm, I found that not all drives were enabling DMA or 32-bit 
transfers, so I forced all of the hard drives to run 32-bit with DMA. 
This solved the problem.

Another interesting thing I found (while I'm talking about my goofy 
array) was that the system performed MUCH better when I disabled write 
caching on all of the hard drives.

The filesystem is ReiserFS 3.6, of course.

--Dan

Re: Fatal File System Corruption -Software RAID + NFS

[Steven Poulakos]

Dan Oglesby wrote:

> Christian Kujau wrote:
>
>> Steven Poulakos wrote:
>> [...]
>>
>>> Here are snippets of the errors that just repeat over and over...
>>>
>>> Nov  9 06:47:34 ctdev kernel: hdc: status error: status=0x58 { 
>>> DriveReady SeekComplete DataRequest }
>>> Nov  9 06:47:34 ctdev kernel:
>>> Nov  9 06:47:34 ctdev kernel: hdc: status error: status=0x58 { 
>>> DriveReady SeekCo
>>
>>
>>
>> hardware errors, i'd say :-(
>> try to dd / dd_rescue to a working disk, then go on with newest 
>> reiserfsprogs or /bin/cat...
>>
>> Christian.
>> -
>
>
> I saw these types of errors on my software RAID-5 array when I first 
> brought it online.  The array is made up of four 20GB hard drives from 
> four different manufacturers, two drives to an IDE channel.  Some of 
> the drives didn't want to play nice when sharing the IDE channel.
>
> Using hdparm, I found that not all drives were enabling DMA or 32-bit 
> transfers, so I forced all of the hard drives to run 32-bit with DMA. 
> This solved the problem.
>
> Another interesting thing I found (while I'm talking about my goofy 
> array) was that the system performed MUCH better when I disabled write 
> caching on all of the hard drives.
>
> The filesystem is ReiserFS 3.6, of course.
>
> --Dan

Christian above suggested that I'm experiencing hardware problems.  I'm 
now wondering if the problem is with my PCI bus speed, which is running 
at 33MHz.   Since I'm using a Promise Ultra100Tx2 controller card, it 
might not be able to operate fast enough to support UDMA 5.  Is this 
correct? 

Here are some additional specs:
Asus P2B Motherboard (33MHz PCI bus)
Pentium 350
384 MB PC100 memory
I also have one AGP video card in use and two NICs on the PCI bus

Is a 33MHz bus sufficient for running the hard drives in UDMA 5 mode?  
Would you recommend that I upgrade to a 66MHz PCI bus motherboard?  The 
other system specs (from the previous email) are below.

Steve

More system specs:
Debian (latest unstable)
2.4.22 Kernel  (not patched in any way)
running NFS (from the kernel), which serves home directories to 3 
workstations
Software RAID 1
-2, mirrored 80gig (7200RPM) IBM hard drives
-Each drive is connected to its own channel on a Promise Ultra ATA/100 
Controller (Ultra100 Tx2) card (PCI)  (using ATA 100 cables)
-the two drives are called hda and hdc (contain 3 mirrored RAID partitions)

Re: Fatal File System Corruption -Software RAID + NFS

[Christian Kujau]

Steven Poulakos wrote:
> at 33MHz.   Since I'm using a Promise Ultra100Tx2 controller card, it 
> might not be able to operate fast enough to support UDMA 5.  Is this 
> correct?

are you using the promise chipset drivers then? did you try without 
these and instead using plain generic ide drivers?

i dunno much of the innards of any bus-subsystems, but "good drives and 
good cabling" are a must, of course. i myself experience SCSI related 
problems when using high-speed scsi disks in a low-end machine....


Christian.
-- 
BOFH excuse #65:

system needs to be rebooted

Re: Fatal File System Corruption -Software RAID + NFS

[Steven Poulakos]

Thanks, Dan and Christian, for the offered advice (below).  It turns out 
that my problems were hardware related and not with Reiserfs.  One of 
the IDE cables had a small knick in it with exposed wire.  This was 
probably the source of errors that caused the file system corruption.

Unfortunately, I can't be absolutely certain of this because I also 
upgraded my motherboard to one that supports a 66MHz PCI bus.  This 
allows the Promise ATA/100 controller card to work at full speed. 

My newly rebuilt system has been up and running for three days now.  So 
far, there aren't any errors.

Thanks, again!

Steve

Dan Oglesby wrote:

> Christian Kujau wrote:
>
>> Steven Poulakos wrote:
>> [...]
>>
>>> Here are snippets of the errors that just repeat over and over...
>>>
>>> Nov  9 06:47:34 ctdev kernel: hdc: status error: status=0x58 { 
>>> DriveReady SeekComplete DataRequest }
>>> Nov  9 06:47:34 ctdev kernel:
>>> Nov  9 06:47:34 ctdev kernel: hdc: status error: status=0x58 { 
>>> DriveReady SeekCo
>>
>>
>>
>> hardware errors, i'd say :-(
>> try to dd / dd_rescue to a working disk, then go on with newest 
>> reiserfsprogs or /bin/cat...
>>
>> Christian.
>> -
>
>
> I saw these types of errors on my software RAID-5 array when I first 
> brought it online.  The array is made up of four 20GB hard drives from 
> four different manufacturers, two drives to an IDE channel.  Some of 
> the drives didn't want to play nice when sharing the IDE channel.
>
> Using hdparm, I found that not all drives were enabling DMA or 32-bit 
> transfers, so I forced all of the hard drives to run 32-bit with DMA. 
> This solved the problem.
>
> Another interesting thing I found (while I'm talking about my goofy 
> array) was that the system performed MUCH better when I disabled write 
> caching on all of the hard drives.
>
> The filesystem is ReiserFS 3.6, of course.
>
> --Dan

Re: Fatal File System Corruption -Software RAID + NFS

[Hans Reiser]

Steven Poulakos wrote:

>
> Dan Oglesby wrote:
>
>>
>>
>> Another interesting thing I found (while I'm talking about my goofy 
>> array) was that the system performed MUCH better when I disabled 
>> write caching on all of the hard drives.
>
Dan and Chris Mason, if you achieve an understanding of this, let me know.

-- 
Hans