From: Ronny Adsetts <ronny.adsetts@amazinginternet.com>
To: Linux RAID ML <linux-raid@vger.kernel.org>
Subject: RAID mirror, resyncing from bad disk
Date: Sun, 13 Sep 2009 20:19:10 +0100 [thread overview]
Message-ID: <4AAD45AE.6060503@amazinginternet.com> (raw)
[-- Attachment #1: Type: text/plain, Size: 12825 bytes --]
Hi,
I've found myself in a situation that I'm unable to resolve hence this request for help.
A messed up /etc/mdadm/mdadm.conf led to one of the disk in a RAID 1 mirrored pair being kicked from the array. Unfortunately, the disk left in the array has bad sectors so I can't resync the good disk back in so I can fail the bad disk and replace it.
/proc/mdstat looks like this:
Personalities : [raid1]
md0 : active raid1 sdb1[0] sdc1[1]
489856 blocks [2/2] [UU]
md1 : active raid1 sdb2[0] sdc2[2](F)
1951808 blocks [2/1] [U_]
md2 : active raid1 sdb3[2](F) sdc3[1]
114776320 blocks [2/1] [_U]
unused devices: <none>
The problematic array is /dev/md2 and the dying disk is /dev/sdc.
When I try to resync it gets to about 99.2% then gives load of I/O errors in /var/log/kern.log and finally gives up and restarts the sync.
Ideally I just want to tell the system to ignore the bad sector and just resync the array.
Does anyone have any ideas on how I can get this resolved short of reinstalling? This is a production server so I'd like to avoid the downtime if at all possible. (I have all the important stuff backed up on tape; /dev/md2 is mainly system stuff).
Other misc. info:
$ mdadm --version
mdadm - v2.6.2 - 21st May 2007
$ uname -a
Linux vimes 2.6.24-etchnhalf.1-amd64 #1 SMP Sat Aug 15 20:38:41 UTC 2009 x86_64 GNU/Linux
The system is sata disks for the md2 array. This is then an LVM volume which is partitioned up in to about 5 partitions all using the XFS filesystem.
Logs for a resync:
---
Aug 19 12:20:54 vimes mdadm: Rebuild80 event detected on md device /dev/md2
Aug 19 12:38:26 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:26 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:26 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:26 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:26 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:26 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:26 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:26 vimes kernel: ata2: EH complete
Aug 19 12:38:30 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:30 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:30 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:30 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:30 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:30 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:30 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:30 vimes kernel: ata2: EH complete
Aug 19 12:38:33 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:33 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:33 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:33 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:33 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:33 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:33 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:33 vimes kernel: ata2: EH complete
Aug 19 12:38:37 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:37 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:37 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:37 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:37 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:37 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:37 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:37 vimes kernel: ata2: EH complete
Aug 19 12:38:41 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:41 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:41 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:41 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:41 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:41 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:41 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:41 vimes kernel: ata2: EH complete
Aug 19 12:38:45 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:45 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:45 vimes kernel: ata2.00: cmd 25/00:00:b0:e2:f1/00:04:0d:00:00/e0 tag 0 dma 524288 in
Aug 19 12:38:45 vimes kernel: res 51/40:55:05:e4:f1/40:01:0d:00:00/e0 Emask 0x9 (media error)
Aug 19 12:38:45 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:45 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:45 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor]
Aug 19 12:38:45 vimes kernel: Descriptor sense data with sense descriptors (in hex):
Aug 19 12:38:45 vimes kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Aug 19 12:38:45 vimes kernel: 0d f1 e4 05
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
Aug 19 12:38:45 vimes kernel: end_request: I/O error, dev sdb, sector 233956357
Aug 19 12:38:45 vimes kernel: ata2: EH complete
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] 234441648 512-byte hardware sectors (120034 MB)
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Write Protect is off
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] 234441648 512-byte hardware sectors (120034 MB)
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Write Protect is off
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Aug 19 12:38:45 vimes kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Aug 19 12:38:49 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:49 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:49 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:38:49 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:38:49 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:49 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:49 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:49 vimes kernel: ata2: EH complete
Aug 19 12:38:52 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:52 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:52 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:38:52 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:38:52 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:52 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:53 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:53 vimes kernel: ata2: EH complete
Aug 19 12:38:56 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:38:56 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:38:56 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:38:56 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:38:56 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:38:56 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:38:56 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:38:56 vimes kernel: ata2: EH complete
Aug 19 12:39:00 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:39:00 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:39:00 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:39:00 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:39:00 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:39:00 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:39:00 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:39:00 vimes kernel: ata2: EH complete
Aug 19 12:39:04 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:39:04 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:39:04 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:39:04 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:39:04 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:39:04 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:39:04 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:39:04 vimes kernel: ata2: EH complete
Aug 19 12:39:08 vimes kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Aug 19 12:39:08 vimes kernel: ata2.00: BMDMA stat 0x24
Aug 19 12:39:08 vimes kernel: ata2.00: cmd c8/00:08:00:e4:f1/00:00:00:00:00/ed tag 0 dma 4096 in
Aug 19 12:39:08 vimes kernel: res 51/40:05:05:e4:f1/40:01:0d:00:00/ed Emask 0x9 (media error)
Aug 19 12:39:08 vimes kernel: ata2.00: status: { DRDY ERR }
Aug 19 12:39:08 vimes kernel: ata2.00: error: { UNC }
Aug 19 12:39:08 vimes kernel: ata2.00: configured for UDMA/133
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor]
Aug 19 12:39:08 vimes kernel: Descriptor sense data with sense descriptors (in hex):
Aug 19 12:39:08 vimes kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Aug 19 12:39:08 vimes kernel: 0d f1 e4 05
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
Aug 19 12:39:08 vimes kernel: end_request: I/O error, dev sdb, sector 233956357
Aug 19 12:39:08 vimes kernel: raid1: sdb: unrecoverable I/O read error for block 229072512
Aug 19 12:39:08 vimes kernel: ata2: EH complete
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] 234441648 512-byte hardware sectors (120034 MB)
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Write Protect is off
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] 234441648 512-byte hardware sectors (120034 MB)
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Write Protect is off
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Aug 19 12:39:08 vimes kernel: sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Aug 19 12:39:08 vimes kernel: RAID1 conf printout:
Aug 19 12:39:08 vimes kernel: --- wd:1 rd:2
Aug 19 12:39:08 vimes kernel: disk 0, wo:1, o:1, dev:sda3
Aug 19 12:39:08 vimes kernel: disk 1, wo:0, o:1, dev:sdb3
Aug 19 12:39:08 vimes kernel: RAID1 conf printout:
Aug 19 12:39:08 vimes kernel: --- wd:1 rd:2
Aug 19 12:39:08 vimes kernel: disk 1, wo:0, o:1, dev:sdb3
Aug 19 12:39:08 vimes kernel: RAID1 conf printout:
Aug 19 12:39:08 vimes kernel: --- wd:1 rd:2
Aug 19 12:39:08 vimes kernel: disk 0, wo:1, o:1, dev:sda3
Aug 19 12:39:08 vimes kernel: disk 1, wo:0, o:1, dev:sdb3
Aug 19 12:39:08 vimes kernel: md: recovery of RAID array md2
Aug 19 12:39:08 vimes kernel: md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Aug 19 12:39:08 vimes mdadm: RebuildFinished event detected on md device /dev/md2
Aug 19 12:39:08 vimes kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
Aug 19 12:39:08 vimes kernel: md: using 128k window, over a total of 114776320 blocks.
Aug 19 12:39:08 vimes mdadm: RebuildStarted event detected on md device /dev/md2
Aug 19 12:51:08 vimes mdadm: Rebuild20 event detected on md device /dev/md2
---
Thanks for any help.
Ronny
--
Ronny Adsetts
Technical Director
Amazing Internet Ltd, London
t: +44 20 8607 9535
f: +44 20 8607 9536
w: www.amazinginternet.com
Registered office: UK House, 82 Heath Road, Twickenham TW1 4BW
Registered in England. Company No. 4042957
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 260 bytes --]
next reply other threads:[~2009-09-13 19:19 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-09-13 19:19 Ronny Adsetts [this message]
2009-09-13 20:24 ` RAID mirror, resyncing from bad disk John Robinson
2009-09-13 22:49 ` Majed B.
2009-09-14 9:00 ` Ronny Adsetts
2009-09-14 9:13 ` Ronny Adsetts
2009-09-14 9:23 ` John Robinson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4AAD45AE.6060503@amazinginternet.com \
--to=ronny.adsetts@amazinginternet.com \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).