mdadm crashing server?

mdadm crashing server?

[Luke Sheldrick]

Hi all,

Have a niggling problem on my home nas at the 
moment.

I rebuilt a raid 1 array a few nights ago, and 
haven't managed to get the disc's to resync since 
creating a ext3 fs on the array.

It would seem that as soon as they try to resync 
it crashes the server, totally. All networking 
stops, not responsive at the console.

If I keep the array up with a single disc (doesn't 
matter which) then all is fine.

There is another raid on the server using mdadm 
which runs fine.

Any ideas?

Background:
Server running FC8, up to date on testing tree, 
although same problem on stable.
The two disc's are on the tertiary ide (master, 
slave). Both maxtor 250gb.
Server, Dell Poweredge 600sc.

Thanks

Re: mdadm crashing server?

[Keith Roberts]

On Mon, 28 Apr 2008, Luke Sheldrick wrote:

> To: linux-raid@vger.kernel.org
> From: Luke Sheldrick <luke@sheldrick.co.uk>
> Subject: mdadm crashing server?
> 
> Hi all,
>
> Have a niggling problem on my home nas at the moment.
>
> I rebuilt a raid 1 array a few nights ago, and haven't 
> managed to get the disc's to resync since creating a ext3 
> fs on the array.
>
> It would seem that as soon as they try to resync it 
> crashes the server, totally. All networking stops, not 
> responsive at the console.
>
> If I keep the array up with a single disc (doesn't matter 
> which) then all is fine.
>
> There is another raid on the server using mdadm which runs fine.
>
> Any ideas?
>
> Background:
> Server running FC8, up to date on testing tree, although same problem on 
> stable.
> The two disc's are on the tertiary ide (master, slave). Both maxtor 
> 250gb.
> Server, Dell Poweredge 600sc.
>
> Thanks

From the Linux software raid howto: It is not recommended to 
run the same RAID1 array on the same IDE cable as 
master/slave. In fact the howto recommends 1 drive set as 
master on one cable only. One reason being if your master
drive goes down, then the slave may not operate correctly.

Kind Regards,

Keith Roberts

-----------------------------------------------------------------
Websites:
http://www.php-debuggers.net
http://www.karsites.net
http://www.raised-from-the-dead.org.uk

All email addresses are challenge-response protected with
TMDA [http://tmda.net]
-----------------------------------------------------------------

RE: mdadm crashing server?

[Luke Sheldrick]

From the Linux software raid howto: It is not recommended to
>run the same RAID1 array on the same IDE cable as
>master/slave. In fact the howto recommends 1 drive set as
>master on one cable only. One reason being if your master
>drive goes down, then the slave may not operate correctly.


I was think it probably wasn't a good idea.

I changed over one of the disc's from the array, to sit on the secondary IDE. It seems to now die at 1% and reset the sync (not killing the machine this time however)

Below is what is cropping up in the /var/log/messages

Apr 28 09:26:51 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:26:51 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:26:51 nas kernel: ata4.01: cmd c8/00:00:00:61:94/00:00:00:00:00/f0 tag 0 dma 131072 in
Apr 28 09:26:51 nas kernel:          res 51/40:00:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:26:51 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:26:51 nas kernel: ata4.01: error: { UNC }
Apr 28 09:26:51 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:26:51 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:26:51 nas kernel: ata4: EH complete
Apr 28 09:26:53 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:26:53 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:26:53 nas kernel: ata4.01: cmd c8/00:00:00:61:94/00:00:00:00:00/f0 tag 0 dma 131072 in
Apr 28 09:26:53 nas kernel:          res 51/40:00:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:26:53 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:26:53 nas kernel: ata4.01: error: { UNC }
Apr 28 09:26:53 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:26:53 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:26:53 nas kernel: ata4: EH complete
Apr 28 09:26:53 nas kernel: sd 3:0:0:0: [sde] 586072368 512-byte hardware sectors (300069 MB)
Apr 28 09:26:54 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:26:54 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:26:54 nas kernel: ata4.01: cmd c8/00:00:00:61:94/00:00:00:00:00/f0 tag 0 dma 131072 in
Apr 28 09:26:54 nas kernel:          res 51/40:00:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:26:54 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:26:54 nas kernel: ata4.01: error: { UNC }
Apr 28 09:26:54 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:26:54 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:26:54 nas kernel: ata4: EH complete
Apr 28 09:26:54 nas kernel: sd 3:0:0:0: [sde] Write Protect is off
Apr 28 09:26:56 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:26:56 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:26:56 nas kernel: ata4.01: cmd c8/00:00:00:61:94/00:00:00:00:00/f0 tag 0 dma 131072 in
Apr 28 09:26:56 nas kernel:          res 51/40:00:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:26:56 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:26:56 nas kernel: ata4.01: error: { UNC }
Apr 28 09:26:56 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:26:56 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:26:56 nas kernel: ata4: EH complete
Apr 28 09:26:57 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:26:57 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:26:57 nas kernel: ata4.01: cmd c8/00:00:00:61:94/00:00:00:00:00/f0 tag 0 dma 131072 in
Apr 28 09:26:57 nas kernel:          res 51/40:00:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:26:57 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:26:57 nas kernel: ata4.01: error: { UNC }
Apr 28 09:26:57 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:26:57 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:26:57 nas kernel: ata4: EH complete
Apr 28 09:26:57 nas kernel: sd 3:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr 28 09:26:59 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:26:59 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:26:59 nas kernel: ata4.01: cmd c8/00:00:00:61:94/00:00:00:00:00/f0 tag 0 dma 131072 in
Apr 28 09:26:59 nas kernel:          res 51/40:00:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:26:59 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:26:59 nas kernel: ata4.01: error: { UNC }
Apr 28 09:26:59 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:26:59 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:26:59 nas kernel: sd 3:0:1:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
Apr 28 09:26:59 nas kernel: sd 3:0:1:0: [sdf] Sense Key : Medium Error [current] [descriptor]
Apr 28 09:26:59 nas kernel: Descriptor sense data with sense descriptors (in hex):
Apr 28 09:26:59 nas kernel:         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Apr 28 09:26:59 nas kernel:         00 94 61 80
Apr 28 09:26:59 nas kernel: sd 3:0:1:0: [sdf] Add. Sense: Unrecovered read error - auto reallocate failed
Apr 28 09:26:59 nas kernel: end_request: I/O error, dev sdf, sector 9724288
Apr 28 09:26:59 nas kernel: ata4: EH complete
Apr 28 09:26:59 nas kernel: sd 3:0:1:0: [sdf] 490234752 512-byte hardware sectors (251000 MB)
Apr 28 09:26:59 nas kernel: sd 3:0:1:0: [sdf] Write Protect is off
Apr 28 09:26:59 nas kernel: sd 3:0:1:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr 28 09:26:59 nas kernel: sd 3:0:0:0: [sde] 586072368 512-byte hardware sectors (300069 MB)
Apr 28 09:27:01 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:27:01 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:27:01 nas kernel: ata4.01: cmd c8/00:08:80:61:94/00:00:00:00:00/f0 tag 0 dma 4096 in
Apr 28 09:27:01 nas kernel:          res 51/40:08:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:27:01 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:27:01 nas kernel: ata4.01: error: { UNC }
Apr 28 09:27:01 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:27:01 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:27:01 nas kernel: ata4: EH complete
Apr 28 09:27:01 nas kernel: sd 3:0:0:0: [sde] Write Protect is off
Apr 28 09:27:02 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:27:02 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:27:02 nas kernel: ata4.01: cmd c8/00:08:80:61:94/00:00:00:00:00/f0 tag 0 dma 4096 in
Apr 28 09:27:02 nas kernel:          res 51/40:08:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:27:02 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:27:02 nas kernel: ata4.01: error: { UNC }
Apr 28 09:27:02 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:27:02 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:27:02 nas kernel: ata4: EH complete
Apr 28 09:27:04 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:27:04 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:27:04 nas kernel: ata4.01: cmd c8/00:08:80:61:94/00:00:00:00:00/f0 tag 0 dma 4096 in
Apr 28 09:27:04 nas kernel:          res 51/40:08:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:27:04 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:27:04 nas kernel: ata4.01: error: { UNC }
Apr 28 09:27:04 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:27:04 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:27:04 nas kernel: ata4: EH complete
Apr 28 09:27:04 nas kernel: sd 3:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr 28 09:27:05 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:27:05 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:27:05 nas kernel: ata4.01: cmd c8/00:08:80:61:94/00:00:00:00:00/f0 tag 0 dma 4096 in
Apr 28 09:27:05 nas kernel:          res 51/40:08:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:27:05 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:27:05 nas kernel: ata4.01: error: { UNC }
Apr 28 09:27:05 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:27:05 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:27:05 nas kernel: ata4: EH complete
Apr 28 09:27:07 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:27:07 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:27:07 nas kernel: ata4.01: cmd c8/00:08:80:61:94/00:00:00:00:00/f0 tag 0 dma 4096 in
Apr 28 09:27:07 nas kernel:          res 51/40:08:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:27:07 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:27:07 nas kernel: ata4.01: error: { UNC }
Apr 28 09:27:07 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:27:07 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:27:07 nas kernel: ata4: EH complete
Apr 28 09:27:09 nas kernel: ata4.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr 28 09:27:09 nas kernel: ata4.01: BMDMA stat 0x64
Apr 28 09:27:09 nas kernel: ata4.01: cmd c8/00:08:80:61:94/00:00:00:00:00/f0 tag 0 dma 4096 in
Apr 28 09:27:09 nas kernel:          res 51/40:08:80:61:94/40:04:00:00:00/f0 Emask 0x9 (media error)
Apr 28 09:27:09 nas kernel: ata4.01: status: { DRDY ERR }
Apr 28 09:27:09 nas kernel: ata4.01: error: { UNC }
Apr 28 09:27:09 nas kernel: ata4.00: configured for UDMA/33
Apr 28 09:27:09 nas kernel: ata4.01: configured for UDMA/100
Apr 28 09:27:09 nas kernel: sd 3:0:1:0: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
Apr 28 09:27:09 nas kernel: sd 3:0:1:0: [sdf] Sense Key : Medium Error [current] [descriptor]
Apr 28 09:27:09 nas kernel: Descriptor sense data with sense descriptors (in hex):
Apr 28 09:27:09 nas kernel:         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Apr 28 09:27:09 nas kernel:         00 94 61 80
Apr 28 09:27:09 nas kernel: sd 3:0:1:0: [sdf] Add. Sense: Unrecovered read error - auto reallocate failed
Apr 28 09:27:09 nas kernel: end_request: I/O error, dev sdf, sector 9724288
Apr 28 09:27:09 nas kernel: ata4: EH complete
Apr 28 09:27:09 nas kernel: raid1: sdf: unrecoverable I/O read error for block 9722240
Apr 28 09:27:09 nas kernel: md: md0: recovery done.
Apr 28 09:27:09 nas kernel: sd 3:0:1:0: [sdf] 490234752 512-byte hardware sectors (251000 MB)
Apr 28 09:27:09 nas kernel: sd 3:0:1:0: [sdf] Write Protect is off
Apr 28 09:27:09 nas kernel: sd 3:0:1:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr 28 09:27:09 nas kernel: sd 3:0:0:0: [sde] 586072368 512-byte hardware sectors (300069 MB)
Apr 28 09:27:09 nas kernel: sd 3:0:0:0: [sde] Write Protect is off
Apr 28 09:27:09 nas kernel: sd 3:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr 28 09:27:09 nas kernel: sd 3:0:1:0: [sdf] 490234752 512-byte hardware sectors (251000 MB)
Apr 28 09:27:09 nas kernel: sd 3:0:1:0: [sdf] Write Protect is off
Apr 28 09:27:09 nas kernel: sd 3:0:1:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr 28 09:27:09 nas kernel: RAID1 conf printout:
Apr 28 09:27:09 nas kernel:  --- wd:1 rd:2
Apr 28 09:27:09 nas kernel:  disk 0, wo:1, o:1, dev:sda1
Apr 28 09:27:09 nas kernel:  disk 1, wo:0, o:1, dev:sdf1
Apr 28 09:27:09 nas kernel: RAID1 conf printout:
Apr 28 09:27:09 nas kernel:  --- wd:1 rd:2
Apr 28 09:27:09 nas kernel:  disk 1, wo:0, o:1, dev:sdf1
Apr 28 09:27:09 nas kernel: RAID1 conf printout:
Apr 28 09:27:09 nas kernel:  --- wd:1 rd:2
Apr 28 09:27:09 nas kernel:  disk 0, wo:1, o:1, dev:sda1
Apr 28 09:27:09 nas kernel:  disk 1, wo:0, o:1, dev:sdf1
Apr 28 09:27:09 nas kernel: md: recovery of RAID array md0
Apr 28 09:27:09 nas kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Apr 28 09:27:09 nas kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Apr 28 09:27:09 nas kernel: md: using 128k window, over a total of 245111616 blocks.

b0rked disk?

RE: mdadm crashing server?

[Keith Roberts]

On Mon, 28 Apr 2008, Luke Sheldrick wrote:

> To: "linux-raid@vger.kernel.org" <linux-raid@vger.kernel.org>
> From: Luke Sheldrick <luke@sheldrick.co.uk>
> Subject: RE: mdadm crashing server?
> 
>
>> From the Linux software raid howto: It is not recommended to
>> run the same RAID1 array on the same IDE cable as
>> master/slave. In fact the howto recommends 1 drive set as
>> master on one cable only. One reason being if your master
>> drive goes down, then the slave may not operate correctly.
>
>
> I was think it probably wasn't a good idea.
>
> I changed over one of the disc's from the array, to sit on 
> the secondary IDE. It seems to now die at 1% and reset the 
> sync (not killing the machine this time however)
*snip*
> hex): Apr 28 09:27:09 nas kernel:  72 03 11 04 00 00 00 0c 
> 00 0a 80 00 00 00 00 00 Apr 28 09:27:09 nas kernel:  00 94 
> 61 80 Apr 28 09:27:09 nas kernel: sd 3:0:1:0: [sdf] Add. 
> Sense: Unrecovered read error - auto reallocate failed Apr 
> 28 09:27:09 nas kernel: end_request: I/O error, dev sdf, 
> sector 9724288 Apr 28 09:27:09 nas kernel: ata4: EH 
> complete Apr 28 09:27:09 nas kernel: raid1: sdf: 
> unrecoverable I/O read error for block 9722240 Apr 28

Maybe you need to run smartmontools on the disk.

I think auto reallocate failure is a smart disk attribute.

Also the disks seem to be running at different DMA speeds. 
Do they need setting to the same speeds or not?

Kind Regards

Keith Roberts

-----------------------------------------------------------------
Websites:
http://www.php-debuggers.net
http://www.karsites.net
http://www.raised-from-the-dead.org.uk

All email addresses are challenge-response protected with 
TMDA [http://tmda.net] 
-----------------------------------------------------------------