From: Semyon Enskiy <semyon.enskiy@yandex.ru>
To: linux-raid@vger.kernel.org
Subject: Recovering RAID5 with 2, actually 1, faulty disks.
Date: Mon, 23 Nov 2015 18:28:38 +0300 [thread overview]
Message-ID: <189791448292518@web28m.yandex.ru> (raw)
Hi all,
I am sorry for bad English, it is not my primary language.
Does solution, described here -
"http://marc.info/?l=linux-raid&m=144659416216285&w=2", matches for mine
issue?
Please, read issue description below, I am ready to provide additional info.
RAID1 (md1; boot), RAID10 (md2; swap, root) and
RAID5 (md3; LVM -> ext4 -> data)
are placed on 10 disks, each of them is 4TB WD Red with 3 partitions on it.
All disks model is WDC WD40EFRX-68WT0N0.
sd?1 partitions belongs to md1, sd?2 belongs to md2, sd?3 belongs to md3.
sdj caught IO error and was reinitialized under new minor and in-/dev/ IDs,
in fact partitions on it was dropped from arrays.
sdk1 and sdk2, previous sdj1 and sdj2 accordingly, was readded to their
arrays by me, but md3 stay leave sdj3 as member of array, I can't do
anything with it, only receiving errors like that:
# mdadm /dev/md3 -r detached
mdadm: Cannot find 8:147: No such file or directory
so sdk3 can't be readded.
So I've managed to grow md3 array with command like following, do not
remember why and strictly command:
# mdadm --grow --raid-devices=11 -add /dev/md3 /dev/sdk3
array start to reshaping, but caught second error and reshaping process
stalled with 1536 bytes transferred. Then was executed near two useless
commands, like making sdk3 spare device.
Now md3 consists 11 devices, was 10, has 2 faulty devices, actually 1 - old
sdj3 and sdj3 under new ID - sdk3, which is slightly corrupted by writing
1536 bytes.
Seems, that I should and write data back to source block device to do it
consistent and able to reboot, recreate array with one corrupted device and
resync it.
Which data was transferred at reshaping - raw data from first block device
in the array, including mdadm metadata (/dev/sd?3), or data from array's
summary block device (/dev/md3)?
After this all happened, host was not rebooted, stay online, FS on md3 array
unavailable, smartctl reports (-H argument), that all disks are healthy.
# uname -sr
Linux 3.13.0-52-generic
# strace -p `pgrep -o mdadm` -f
Process 1250 attached
select(5, NULL, NULL, [4], {181, 601546}
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid10 sdk2[12] sdg2[1] sdh2[10] sdi2[5] sdf2[9] sde2[8] sdd2[6] sdc2[4] sdb2[11] sda2[0]
35653120 blocks super 1.2 512K chunks 2 near-copies [10/10] [UUUUUUUUUU]
bitmap: 1/1 pages [4KB], 65536KB chunk
md1 : active raid1 sdk1[9] sdg1[6] sdh1[7] sdi1[8] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
203712 blocks [10/10] [UUUUUUUUUU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md3 : active raid5 sdk3[14](S) sdg3[6] sdh3[11] sdj3[13](F) sdi3[10] sde3[4] sdf3[5] sdd3[3] sdc3[2] sdb3[12] sda3[0]
35095924224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [11/10] [UUUUUUUU_U_]
[>....................] reshape = 0.0% (1536/3899547136) finish=10215829552.5min speed=0K/sec
bitmap: 0/15 pages [0KB], 131072KB chunk
unused devices: <none>
# blkid /dev/sda3 /dev/sdk3
/dev/sda3: UUID="9bdc0939-4838-89af-b67c-199b2aaecb4e" UUID_SUB="fa7c541d-a25b-abd8-14ff-e1c0c6c1578c" LABEL="ubuntu:3" TYPE="linux_raid_member"
/dev/sdk3: UUID="9bdc0939-4838-89af-b67c-199b2aaecb4e" UUID_SUB="c0155214-a9c8-e378-d368-b6f6735048a8" LABEL="ubuntu:3" TYPE="linux_raid_member"
# mdadm --detail /dev/md3
/dev/md3:
Version : 1.2
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 3899547136 (3718.90 GiB 3993.14 GB)
Raid Devices : 11
Total Devices : 11
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Nov 6 16:53:06 2015
State : active, FAILED, reshaping
Active Devices : 9
Working Devices : 10
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Reshape Status : 0% complete
Delta Devices : 1, (10->11)
Name : ubuntu:3
UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Events : 21586
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
12 8 19 1 active sync /dev/sdb3
2 8 35 2 active sync /dev/sdc3
3 8 51 3 active sync /dev/sdd3
4 8 67 4 active sync /dev/sde3
5 8 83 5 active sync /dev/sdf3
6 8 99 6 active sync /dev/sdg3
11 8 115 7 active sync /dev/sdh3
16 0 0 16 removed
10 8 131 9 active sync /dev/sdi3
20 0 0 20 removed
13 8 147 - faulty
14 8 163 - spare /dev/sdk3
# "mdadm --examine" output
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : fa7c541d:a25babd8:14ffe1c0:c6c1578c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : ac275763 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 70afdaad:b0979fb1:1b99ef18:dd58cc3a
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 1c2bf759 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 345b0212:6c3a0ae8:d8ed9273:ecc53573
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 49c30753 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a4b8b425:b0307fe5:9eb8e3ac:ae985311
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 3258f890 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4490843e:d2c5c2e7:d222fa0a:f98b277c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 1656c2d2 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 1d41056f:88149b1a:48e939e5:8c775e23
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : fb26746a - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 375d85e2:9077dd68:bc1327af:8fee4112
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 75b99505 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 67cd2bfa:affa578c:e0c1286a:bd4d2627
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 80c895f3 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a5f2cea8:a89c88a0:0dd3900c:624f0491
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 4fda6fb3 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c0155214:a9c8e378:d368b6f6:735048a8
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 952a55f2 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c0155214:a9c8e378:d368b6f6:735048a8
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 952a55f2 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
# grep'ed smartctl output #1,
/dev/sda
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdb
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdc
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdd
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sde
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdf
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdg
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdh
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdi
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdk
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
# grep 'sd.3' /proc/partitions
8 3 3899678535 sda3
8 19 3899678535 sdb3
8 35 3899678535 sdc3
8 51 3899678535 sdd3
8 67 3899678535 sde3
8 83 3899678535 sdf3
8 99 3899678535 sdg3
8 115 3899678535 sdh3
8 131 3899678535 sdi3
8 163 3899678535 sdk3
# grep'ed smartctl output #2
/dev/sda
9 Power_On_Hours -O--CK 082 082 000 - 13233
/dev/sdb
9 Power_On_Hours -O--CK 085 085 000 - 11051
/dev/sdc
9 Power_On_Hours -O--CK 085 085 000 - 11379
/dev/sdd
9 Power_On_Hours -O--CK 085 085 000 - 11403
/dev/sde
9 Power_On_Hours -O--CK 081 081 000 - 14233
/dev/sdf
9 Power_On_Hours -O--CK 085 085 000 - 11357
/dev/sdg
9 Power_On_Hours -O--CK 085 085 000 - 11382
/dev/sdh
9 Power_On_Hours -O--CK 085 085 000 - 11097
/dev/sdi
9 Power_On_Hours -O--CK 082 082 000 - 13256
/dev/sdk
9 Power_On_Hours -O--CK 089 089 000 - 8413
# dmesg
"http://paste.debian.net/335374/"
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next reply other threads:[~2015-11-23 15:28 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-11-23 15:28 Semyon Enskiy [this message]
2015-11-23 15:35 ` Recovering RAID5 with 2, actually 1, faulty disks Semyon Enskiy
2015-11-23 15:47 ` Wols Lists
2015-11-23 15:57 ` Semyon Enskiy
2015-11-23 16:08 ` Wols Lists
2015-11-23 18:24 ` Phil Turmel
2015-11-25 12:12 ` Semyon Enskiy
2015-11-25 13:22 ` Phil Turmel
2015-11-25 14:48 ` Semyon Enskiy
2015-11-25 14:55 ` Semyon Enskiy
2015-11-25 15:44 ` Phil Turmel
2015-11-25 16:29 ` Semyon Enskiy
2015-11-25 16:58 ` Phil Turmel
2015-11-25 17:32 ` Semyon Enskiy
2015-11-25 17:49 ` Phil Turmel
2015-11-26 13:21 ` Semyon Enskiy
-- strict thread matches above, loose matches on Subject: below --
2015-12-02 13:35 Semyon Enskiy
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=189791448292518@web28m.yandex.ru \
--to=semyon.enskiy@yandex.ru \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox