From: Ronny Adsetts <ronny.adsetts@amazinginternet.com>
To: Linux RAID ML <linux-raid@vger.kernel.org>
Subject: RAID mirror, resyncing from bad disk
Date: Sun, 13 Sep 2009 20:19:10 +0100 [thread overview]
Message-ID: <4AAD45AE.6060503@amazinginternet.com> (raw)
[-- Attachment #1: Type: text/plain, Size: 12825 bytes --]
Hi,
I've found myself in a situation that I'm unable to resolve hence this request for help.
A messed up /etc/mdadm/mdadm.conf led to one of the disk in a RAID 1 mirrored pair being kicked from the array. Unfortunately, the disk left in the array has bad sectors so I can't resync the good disk back in so I can fail the bad disk and replace it.
/proc/mdstat looks like this:
Personalities : [raid1]
md0 : active raid1 sdb1[0] sdc1[1]
489856 blocks [2/2] [UU]
md1 : active raid1 sdb2[0] sdc2[2](F)
1951808 blocks [2/1] [U_]
md2 : active raid1 sdb3[2](F) sdc3[1]
114776320 blocks [2/1] [_U]
unused devices: <none>
The problematic array is /dev/md2 and the dying disk is /dev/sdc.
When I try to resync it gets to about 99.2% then gives load of I/O errors in /var/log/kern.log and finally gives up and restarts the sync.
Ideally I just want to tell the system to ignore the bad sector and just resync the array.
Does anyone have any ideas on how I can get this resolved short of reinstalling? This is a production server so I'd like to avoid the downtime if at all possible. (I have all the important stuff backed up on tape; /dev/md2 is mainly system stuff).
Other misc. info:
$ mdadm --version
mdadm - v2.6.2 - 21st May 2007
$ uname -a
Linux vimes 2.6.24-etchnhalf.1-amd64 #1 SMP Sat Aug 15 20:38:41 UTC 2009 x86_64 GNU/Linux
The system is sata disks for the md2 array. This is then an LVM volume which is partitioned up in to about 5 partitions all using the XFS filesystem.
Logs for a resync:
---
Aug 19 12:20:54 vimes mdadm: Rebuild80 event detected on md device /dev/md2
Aug 19 12:38:26 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:26 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:26 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:26 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:26 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:26 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:26 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:26 vimes kernel: ata2: EH complete
Aug 19 12:38:30 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:30 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:30 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:30 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:30 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:30 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:30 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:30 vimes kernel: ata2: EH complete
Aug 19 12:38:33 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:33 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:33 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:33 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:33 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:33 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:33 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:33 vimes kernel: ata2: EH complete
Aug 19 12:38:37 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:37 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:37 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:37 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:37 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:37 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:37 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:37 vimes kernel: ata2: EH complete
Aug 19 12:38:41 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:41 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:41 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:41 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:41 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:41 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:41 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:41 vimes kernel: ata2: EH complete
Aug 19 12:38:45 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:45 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:45 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:45 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:45 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:45 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:45 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor]
Aug 19 12:38:45 vimes kernel: Descriptor sense data with sense descriptors (in hex):
Aug 19 12:38:45 vimes kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Aug 19 12:38:45 vimes kernel: 0d f1 e4 05
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
Aug 19 12:38:45 vimes kernel: end_request: I/O error, dev sdb, sector 233956357
Aug 19 12:38:45 vimes kernel: ata2: EH complete
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] 234441648 512-byte hardware sectors (120034 MB)
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Write Protect is off
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] 234441648 512-byte hardware sectors (120034 MB)
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Write Protect is off
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Aug 19 12:38:49 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:49 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:49 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:38:49 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:38:49 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:49 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:49 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:49 vimes kernel: ata2: EH complete
Aug 19 12:38:52 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:52 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:52 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:38:52 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:38:52 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:52 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:53 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:53 vimes kernel: ata2: EH complete
Aug 19 12:38:56 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:56 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:56 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:38:56 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:38:56 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:56 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:56 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:56 vimes kernel: ata2: EH complete
Aug 19 12:39:00 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:39:00 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:39:00 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:39:00 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:39:00 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:39:00 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:39:00 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:39:00 vimes kernel: ata2: EH complete
Aug 19 12:39:04 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:39:04 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:39:04 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:39:04 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:39:04 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:39:04 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:39:04 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:39:04 vimes kernel: ata2: EH complete
Aug 19 12:39:08 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:39:08 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:39:08 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:39:08 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:39:08 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:39:08 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:39:08 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor]
Aug 19 12:39:08 vimes kernel: Descriptor sense data with sense descriptors (in hex):
Aug 19 12:39:08 vimes kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Aug 19 12:39:08 vimes kernel: 0d f1 e4 05
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
Aug 19 12:39:08 vimes kernel: end_request: I/O error, dev sdb, sector 233956357
Aug 19 12:39:08 vimes kernel: raid1: sdb: unrecoverable I/O read error for block 229072512
Aug 19 12:39:08 vimes kernel: ata2: EH complete
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] 234441648 512-byte hardware sectors (120034 MB)
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Write Protect is off
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] 234441648 512-byte hardware sectors (120034 MB)
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Write Protect is off
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Aug 19 12:39:08 vimes kernel: RAID1 conf printout:
Aug 19 12:39:08 vimes kernel: --- wd:1 rd:2
Aug 19 12:39:08 vimes kernel: disk 0, wo:1, o:1, dev:sda3
Aug 19 12:39:08 vimes kernel: disk 1, wo:0, o:1, dev:sdb3
Aug 19 12:39:08 vimes kernel: RAID1 conf printout:
Aug 19 12:39:08 vimes kernel: --- wd:1 rd:2
Aug 19 12:39:08 vimes kernel: disk 1, wo:0, o:1, dev:sdb3
Aug 19 12:39:08 vimes kernel: RAID1 conf printout:
Aug 19 12:39:08 vimes kernel: --- wd:1 rd:2
Aug 19 12:39:08 vimes kernel: disk 0, wo:1, o:1, dev:sda3
Aug 19 12:39:08 vimes kernel: disk 1, wo:0, o:1, dev:sdb3
Aug 19 12:39:08 vimes kernel: md: recovery of RAID array md2
Aug 19 12:39:08 vimes kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Aug 19 12:39:08 vimes mdadm: RebuildFinished event detected on md device /dev/md2
Aug 19 12:39:08 vimes kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Aug 19 12:39:08 vimes kernel: md: using 128k window, over a total of 114776320 blocks.
Aug 19 12:39:08 vimes mdadm: RebuildStarted event detected on md device /dev/md2
Aug 19 12:51:08 vimes mdadm: Rebuild20 event detected on md device /dev/md2
---
Thanks for any help.
Ronny
--
Ronny Adsetts
Technical Director
Amazing Internet Ltd, London
t: +44 20 8607 9535
f: +44 20 8607 9536
w: www.amazinginternet.com
Registered office: UK House, 82 Heath Road, Twickenham TW1 4BW
Registered in England. Company No. 4042957
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]
next reply other threads:[~2009-09-13 19:19 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-09-13 19:19 Ronny Adsetts [this message]
2009-09-13 20:24 ` RAID mirror, resyncing from bad disk John Robinson
2009-09-13 22:49 ` Majed B.
2009-09-14 9:00 ` Ronny Adsetts
2009-09-14 9:13 ` Ronny Adsetts
2009-09-14 9:23 ` John Robinson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4AAD45AE.6060503@amazinginternet.com \
--to=ronny.adsetts@amazinginternet.com \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.