* Recovering RAID5 with 2, actually 1, faulty disks.
@ 2015-11-23 15:28 Semyon Enskiy
2015-11-23 15:35 ` Semyon Enskiy
2015-11-23 18:24 ` Phil Turmel
0 siblings, 2 replies; 17+ messages in thread
From: Semyon Enskiy @ 2015-11-23 15:28 UTC (permalink / raw)
To: linux-raid
Hi all,
I am sorry for bad English, it is not my primary language.
Does solution, described here -
"http://marc.info/?l=linux-raid&m=144659416216285&w=2", matches for mine
issue?
Please, read issue description below, I am ready to provide additional info.
RAID1 (md1; boot), RAID10 (md2; swap, root) and
RAID5 (md3; LVM -> ext4 -> data)
are placed on 10 disks, each of them is 4TB WD Red with 3 partitions on it.
All disks model is WDC WD40EFRX-68WT0N0.
sd?1 partitions belongs to md1, sd?2 belongs to md2, sd?3 belongs to md3.
sdj caught IO error and was reinitialized under new minor and in-/dev/ IDs,
in fact partitions on it was dropped from arrays.
sdk1 and sdk2, previous sdj1 and sdj2 accordingly, was readded to their
arrays by me, but md3 stay leave sdj3 as member of array, I can't do
anything with it, only receiving errors like that:
# mdadm /dev/md3 -r detached
mdadm: Cannot find 8:147: No such file or directory
so sdk3 can't be readded.
So I've managed to grow md3 array with command like following, do not
remember why and strictly command:
# mdadm --grow --raid-devices=11 -add /dev/md3 /dev/sdk3
array start to reshaping, but caught second error and reshaping process
stalled with 1536 bytes transferred. Then was executed near two useless
commands, like making sdk3 spare device.
Now md3 consists 11 devices, was 10, has 2 faulty devices, actually 1 - old
sdj3 and sdj3 under new ID - sdk3, which is slightly corrupted by writing
1536 bytes.
Seems, that I should and write data back to source block device to do it
consistent and able to reboot, recreate array with one corrupted device and
resync it.
Which data was transferred at reshaping - raw data from first block device
in the array, including mdadm metadata (/dev/sd?3), or data from array's
summary block device (/dev/md3)?
After this all happened, host was not rebooted, stay online, FS on md3 array
unavailable, smartctl reports (-H argument), that all disks are healthy.
# uname -sr
Linux 3.13.0-52-generic
# strace -p `pgrep -o mdadm` -f
Process 1250 attached
select(5, NULL, NULL, [4], {181, 601546}
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid10 sdk2[12] sdg2[1] sdh2[10] sdi2[5] sdf2[9] sde2[8] sdd2[6] sdc2[4] sdb2[11] sda2[0]
35653120 blocks super 1.2 512K chunks 2 near-copies [10/10] [UUUUUUUUUU]
bitmap: 1/1 pages [4KB], 65536KB chunk
md1 : active raid1 sdk1[9] sdg1[6] sdh1[7] sdi1[8] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
203712 blocks [10/10] [UUUUUUUUUU]
bitmap: 0/1 pages [0KB], 65536KB chunk
md3 : active raid5 sdk3[14](S) sdg3[6] sdh3[11] sdj3[13](F) sdi3[10] sde3[4] sdf3[5] sdd3[3] sdc3[2] sdb3[12] sda3[0]
35095924224 blocks super 1.2 level 5, 512k chunk, algorithm 2 [11/10] [UUUUUUUU_U_]
[>....................] reshape = 0.0% (1536/3899547136) finish=10215829552.5min speed=0K/sec
bitmap: 0/15 pages [0KB], 131072KB chunk
unused devices: <none>
# blkid /dev/sda3 /dev/sdk3
/dev/sda3: UUID="9bdc0939-4838-89af-b67c-199b2aaecb4e" UUID_SUB="fa7c541d-a25b-abd8-14ff-e1c0c6c1578c" LABEL="ubuntu:3" TYPE="linux_raid_member"
/dev/sdk3: UUID="9bdc0939-4838-89af-b67c-199b2aaecb4e" UUID_SUB="c0155214-a9c8-e378-d368-b6f6735048a8" LABEL="ubuntu:3" TYPE="linux_raid_member"
# mdadm --detail /dev/md3
/dev/md3:
Version : 1.2
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 3899547136 (3718.90 GiB 3993.14 GB)
Raid Devices : 11
Total Devices : 11
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Nov 6 16:53:06 2015
State : active, FAILED, reshaping
Active Devices : 9
Working Devices : 10
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Reshape Status : 0% complete
Delta Devices : 1, (10->11)
Name : ubuntu:3
UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Events : 21586
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
12 8 19 1 active sync /dev/sdb3
2 8 35 2 active sync /dev/sdc3
3 8 51 3 active sync /dev/sdd3
4 8 67 4 active sync /dev/sde3
5 8 83 5 active sync /dev/sdf3
6 8 99 6 active sync /dev/sdg3
11 8 115 7 active sync /dev/sdh3
16 0 0 16 removed
10 8 131 9 active sync /dev/sdi3
20 0 0 20 removed
13 8 147 - faulty
14 8 163 - spare /dev/sdk3
# "mdadm --examine" output
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : fa7c541d:a25babd8:14ffe1c0:c6c1578c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : ac275763 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 70afdaad:b0979fb1:1b99ef18:dd58cc3a
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 1c2bf759 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 345b0212:6c3a0ae8:d8ed9273:ecc53573
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 49c30753 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a4b8b425:b0307fe5:9eb8e3ac:ae985311
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 3258f890 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4490843e:d2c5c2e7:d222fa0a:f98b277c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 1656c2d2 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 1d41056f:88149b1a:48e939e5:8c775e23
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : fb26746a - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 375d85e2:9077dd68:bc1327af:8fee4112
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 75b99505 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 67cd2bfa:affa578c:e0c1286a:bd4d2627
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 80c895f3 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a5f2cea8:a89c88a0:0dd3900c:624f0491
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 4fda6fb3 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c0155214:a9c8e378:d368b6f6:735048a8
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 952a55f2 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdk3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c0155214:a9c8e378:d368b6f6:735048a8
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 952a55f2 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
# grep'ed smartctl output #1,
/dev/sda
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdb
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdc
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdd
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sde
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdf
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdg
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdh
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
/dev/sdi
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
/dev/sdk
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
# grep 'sd.3' /proc/partitions
8 3 3899678535 sda3
8 19 3899678535 sdb3
8 35 3899678535 sdc3
8 51 3899678535 sdd3
8 67 3899678535 sde3
8 83 3899678535 sdf3
8 99 3899678535 sdg3
8 115 3899678535 sdh3
8 131 3899678535 sdi3
8 163 3899678535 sdk3
# grep'ed smartctl output #2
/dev/sda
9 Power_On_Hours -O--CK 082 082 000 - 13233
/dev/sdb
9 Power_On_Hours -O--CK 085 085 000 - 11051
/dev/sdc
9 Power_On_Hours -O--CK 085 085 000 - 11379
/dev/sdd
9 Power_On_Hours -O--CK 085 085 000 - 11403
/dev/sde
9 Power_On_Hours -O--CK 081 081 000 - 14233
/dev/sdf
9 Power_On_Hours -O--CK 085 085 000 - 11357
/dev/sdg
9 Power_On_Hours -O--CK 085 085 000 - 11382
/dev/sdh
9 Power_On_Hours -O--CK 085 085 000 - 11097
/dev/sdi
9 Power_On_Hours -O--CK 082 082 000 - 13256
/dev/sdk
9 Power_On_Hours -O--CK 089 089 000 - 8413
# dmesg
"http://paste.debian.net/335374/"
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-23 15:28 Recovering RAID5 with 2, actually 1, faulty disks Semyon Enskiy
@ 2015-11-23 15:35 ` Semyon Enskiy
2015-11-23 15:47 ` Wols Lists
2015-11-23 18:24 ` Phil Turmel
1 sibling, 1 reply; 17+ messages in thread
From: Semyon Enskiy @ 2015-11-23 15:35 UTC (permalink / raw)
To: linux-raid@vger.kernel.org
/
Seems, that I should write data back to source block device to do it
consistent and able to recreate array with one corrupted device, reboot,
recreate array, resync faulty device.
/
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-23 15:35 ` Semyon Enskiy
@ 2015-11-23 15:47 ` Wols Lists
2015-11-23 15:57 ` Semyon Enskiy
0 siblings, 1 reply; 17+ messages in thread
From: Wols Lists @ 2015-11-23 15:47 UTC (permalink / raw)
To: Semyon Enskiy, linux-raid@vger.kernel.org
On 23/11/15 15:35, Semyon Enskiy wrote:
> /
> Seems, that I should write data back to source block device to do it
> consistent and able to recreate array with one corrupted device, reboot,
> recreate array, resync faulty device.
> /
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Something I couldn't work out from your post - what are your drives? Are
they proper raid drives, or desktop drives? Is SCTERC (think that's what
it's called) enabled and functioning?
I won't be able to help beyond that, but the experts will want that info...
smartctl -x
Cheers,
Wol
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-23 15:47 ` Wols Lists
@ 2015-11-23 15:57 ` Semyon Enskiy
2015-11-23 16:08 ` Wols Lists
0 siblings, 1 reply; 17+ messages in thread
From: Semyon Enskiy @ 2015-11-23 15:57 UTC (permalink / raw)
To: Wols Lists, linux-raid@vger.kernel.org
> Something I couldn't work out from your post - what are your drives? Are
> they proper raid drives, or desktop drives? Is SCTERC (think that's what
> it's called) enabled and functioning?
>
> I won't be able to help beyond that, but the experts will want that info...
>
> smartctl -x
>
> Cheers,
> Wol
Thanks for your reply, Wol.
As written in first message of this thread, all disks are 4TB WD Red,
"WDC WD40EFRX-68WT0N0", in-kernel timeout for them is 30s.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-23 15:57 ` Semyon Enskiy
@ 2015-11-23 16:08 ` Wols Lists
0 siblings, 0 replies; 17+ messages in thread
From: Wols Lists @ 2015-11-23 16:08 UTC (permalink / raw)
To: Semyon Enskiy, linux-raid@vger.kernel.org
On 23/11/15 15:57, Semyon Enskiy wrote:
>> Something I couldn't work out from your post - what are your drives? Are
>> they proper raid drives, or desktop drives? Is SCTERC (think that's what
>> it's called) enabled and functioning?
>>
>> I won't be able to help beyond that, but the experts will want that info...
>>
>> smartctl -x
>>
>> Cheers,
>> Wol
>
> Thanks for your reply, Wol.
>
> As written in first message of this thread, all disks are 4TB WD Red,
> "WDC WD40EFRX-68WT0N0", in-kernel timeout for them is 30s.
>
Sorry, missed that. That's good.
Cheers,
Wol
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-23 15:28 Recovering RAID5 with 2, actually 1, faulty disks Semyon Enskiy
2015-11-23 15:35 ` Semyon Enskiy
@ 2015-11-23 18:24 ` Phil Turmel
2015-11-25 12:12 ` Semyon Enskiy
1 sibling, 1 reply; 17+ messages in thread
From: Phil Turmel @ 2015-11-23 18:24 UTC (permalink / raw)
To: Semyon Enskiy, linux-raid
Hi Semyon,
On 11/23/2015 10:28 AM, Semyon Enskiy wrote:
> Hi all,
>
> I am sorry for bad English, it is not my primary language.
No worries. Good report.
> Does solution, described here -
> "http://marc.info/?l=linux-raid&m=144659416216285&w=2", matches for mine
> issue?
No, not related. You have v1.2 metadata.
> Please, read issue description below, I am ready to provide additional info.
>
> RAID1 (md1; boot), RAID10 (md2; swap, root) and
> RAID5 (md3; LVM -> ext4 -> data)
> are placed on 10 disks, each of them is 4TB WD Red with 3 partitions on it.
> All disks model is WDC WD40EFRX-68WT0N0.
This is very good. You don't have a timeout mismatch problem.
> sd?1 partitions belongs to md1, sd?2 belongs to md2, sd?3 belongs to md3.
>
> sdj caught IO error and was reinitialized under new minor and in-/dev/ IDs,
> in fact partitions on it was dropped from arrays.
This bothers me. With no timeout issue, this must be a real problem.
Possibilities:
1) Drive, cable, or controller hardware flaw.
2) Transient power supply problem.
3) Kernel bug.
4) Memory bit flip if no ECC Ram.
I've looked at your dmesg, and I'm concerned about the errors on ata12
and ata8. They report the error source as the ata bus. And they are
happening during your check scrubs. That suggests the sustained load
across the entire array during the scrub is stressing some component.
Insufficient power has been the most popular reason for this in the
past. Since it seems to be happening to ata12 the most, with some ata8,
it might just be loose connectors or bad cables.
> sdk1 and sdk2, previous sdj1 and sdj2 accordingly, was readded to their
> arrays by me, but md3 stay leave sdj3 as member of array, I can't do
> anything with it, only receiving errors like that:
> # mdadm /dev/md3 -r detached
> mdadm: Cannot find 8:147: No such file or directory
> so sdk3 can't be readded.
This is likely a kernel bug. A reboot at this point would probably have
cleared it and made --re-add possible.
> So I've managed to grow md3 array with command like following, do not
> remember why and strictly command:
> # mdadm --grow --raid-devices=11 -add /dev/md3 /dev/sdk3
> array start to reshaping, but caught second error and reshaping process
> stalled with 1536 bytes transferred. Then was executed near two useless
> commands, like making sdk3 spare device.
Yeah, growing the array was a mistake. (Where'd that idea come from?)
With mdadm v3.3 or later, you can use --assemble --update=revert-reshape
to cancel this. It is at position 1536, so hasn't actually done any
significant reshaping yet.
--re-add was the correct operation for you to perform. When it failed
due to a busy device, fixing the busy device issue is the correct next
step. But this is no longer possible, as some new metadata has been
written to the device as part of the --grow operation.
The grow operation made no progress due to a kernel bug, I believe. You
have stuck task backtraces in your dmesg for what I believe are
operations directed at the missing but not removed sdj. Not making
progress is a lucky chance, as reshape is not the right answer anyways.
> Now md3 consists 11 devices, was 10, has 2 faulty devices, actually 1 - old
> sdj3 and sdj3 under new ID - sdk3, which is slightly corrupted by writing
> 1536 bytes.
> Seems, that I should and write data back to source block device to do it
> consistent and able to reboot, recreate array with one corrupted device and
> resync it.
No, I think you may really have bad hardware. Check all of your power
and sata cables -- replace the cables to sdj/sdk. Check the voltage on
the drive power rails.
When you are sure power is OK, and all the sata connectors are seated
tightly, reboot into an environment that has mdadm v3.3, and revert the
reshape.
> After this all happened, host was not rebooted, stay online, FS on md3 array
> unavailable, smartctl reports (-H argument), that all disks are healthy.
smartctl's health report is not sufficient info. Please supply the
output of:
for x in /dev/sd[a-z] ; do echo $x ; smartctl -i -A $x ; done
Paste this inline in your next reply, with word-wrap turned off.
Then we might be able to say what's next.
> # dmesg
> "http://paste.debian.net/335374/"
External resources generally don't last as long as this list's archives.
In the future, please paste inline anything important. This list
accepts messages up to ~100k.
Phil
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-23 18:24 ` Phil Turmel
@ 2015-11-25 12:12 ` Semyon Enskiy
2015-11-25 13:22 ` Phil Turmel
0 siblings, 1 reply; 17+ messages in thread
From: Semyon Enskiy @ 2015-11-25 12:12 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
Hi Phil.
Thanks for your suggestions, I have followed them, but RAID5 at md3 not
recovered at yet, maybe you will find time to suggest anything else.
Exactly, power and data connections of SATA disks was not reliable, fixed
this and after boot get no errors in kernel logs.
md3 was replaced with "<ingore>" in mdadm.conf and commented in fstab before
poweroff, things returned back after boot.
# mdadm --assemble --update=revert-reshape /dev/md3
# mdadm --assemble --scan -vv
mdadm: looking for devices for /dev/md1
mdadm: no recogniseable superblock on /dev/dm-1
mdadm: no recogniseable superblock on /dev/dm-0
mdadm: no recogniseable superblock on /dev/sdj4
mdadm: /dev/sdj3 has wrong uuid.
mdadm: /dev/sdj2 has wrong uuid.
mdadm: UUID differs from /dev/md/2.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdj1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdj
mdadm: no recogniseable superblock on /dev/sdi4
mdadm: /dev/sdi3 has wrong uuid.
mdadm: /dev/sdi2 has wrong uuid.
mdadm: UUID differs from /dev/md/2.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdi1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdi
mdadm: no recogniseable superblock on /dev/sdh4
mdadm: /dev/sdh3 has wrong uuid.
mdadm: /dev/sdh2 has wrong uuid.
mdadm: UUID differs from /dev/md/2.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdh1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdh
mdadm: no recogniseable superblock on /dev/md/2
mdadm: no recogniseable superblock on /dev/md1
mdadm: no recogniseable superblock on /dev/sdg4
mdadm: /dev/sdg3 has wrong uuid.
mdadm: /dev/sdg2 has wrong uuid.
mdadm: UUID differs from /dev/md/2.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdg1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdg
mdadm: no recogniseable superblock on /dev/sdb4
mdadm: /dev/sdb3 has wrong uuid.
mdadm: /dev/sdb2 has wrong uuid.
mdadm: UUID differs from /dev/md/2.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdb1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: no recogniseable superblock on /dev/sdf4
mdadm: /dev/sdf3 has wrong uuid.
mdadm: /dev/sdf2 has wrong uuid.
mdadm: UUID differs from /dev/md/2.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdf1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdf
mdadm: no recogniseable superblock on /dev/sde4
mdadm: /dev/sde3 has wrong uuid.
mdadm: /dev/sde2 has wrong uuid.
mdadm: UUID differs from /dev/md/2.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sde1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sde
mdadm: no recogniseable superblock on /dev/sdd4
mdadm: /dev/sdd3 has wrong uuid.
mdadm: /dev/sdd2 has wrong uuid.
mdadm: UUID differs from /dev/md/2.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdd1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdd
mdadm: no recogniseable superblock on /dev/sdc4
mdadm: /dev/sdc3 has wrong uuid.
mdadm: /dev/sdc2 has wrong uuid.
mdadm: UUID differs from /dev/md/2.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdc1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: no recogniseable superblock on /dev/sda4
mdadm: /dev/sda3 has wrong uuid.
mdadm: /dev/sda2 has wrong uuid.
mdadm: UUID differs from /dev/md/2.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sda1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: looking for devices for /dev/md/2
mdadm: no RAID superblock on /dev/dm-1
mdadm: no RAID superblock on /dev/dm-0
mdadm: no RAID superblock on /dev/sdj4
mdadm: /dev/sdj3 has wrong uuid.
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdj2 is busy - skipping
mdadm: no RAID superblock on /dev/sdj1
mdadm: no RAID superblock on /dev/sdj
mdadm: no RAID superblock on /dev/sdi4
mdadm: /dev/sdi3 has wrong uuid.
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdi2 is busy - skipping
mdadm: no RAID superblock on /dev/sdi1
mdadm: no RAID superblock on /dev/sdi
mdadm: no RAID superblock on /dev/sdh4
mdadm: /dev/sdh3 has wrong uuid.
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdh2 is busy - skipping
mdadm: no RAID superblock on /dev/sdh1
mdadm: no RAID superblock on /dev/sdh
mdadm: no RAID superblock on /dev/md/2
mdadm: no RAID superblock on /dev/md1
mdadm: no RAID superblock on /dev/sdg4
mdadm: /dev/sdg3 has wrong uuid.
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdg2 is busy - skipping
mdadm: no RAID superblock on /dev/sdg1
mdadm: no RAID superblock on /dev/sdg
mdadm: no RAID superblock on /dev/sdb4
mdadm: /dev/sdb3 has wrong uuid.
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdb2 is busy - skipping
mdadm: no RAID superblock on /dev/sdb1
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sdf4
mdadm: /dev/sdf3 has wrong uuid.
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdf2 is busy - skipping
mdadm: no RAID superblock on /dev/sdf1
mdadm: no RAID superblock on /dev/sdf
mdadm: no RAID superblock on /dev/sde4
mdadm: /dev/sde3 has wrong uuid.
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sde2 is busy - skipping
mdadm: no RAID superblock on /dev/sde1
mdadm: no RAID superblock on /dev/sde
mdadm: no RAID superblock on /dev/sdd4
mdadm: /dev/sdd3 has wrong uuid.
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdd2 is busy - skipping
mdadm: no RAID superblock on /dev/sdd1
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdc4
mdadm: /dev/sdc3 has wrong uuid.
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sdc2 is busy - skipping
mdadm: no RAID superblock on /dev/sdc1
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sda4
mdadm: /dev/sda3 has wrong uuid.
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/3.
mdadm: /dev/sda2 is busy - skipping
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda
mdadm: looking for devices for /dev/md/3
mdadm: no RAID superblock on /dev/dm-1
mdadm: no RAID superblock on /dev/dm-0
mdadm: no RAID superblock on /dev/sdj4
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/2.
mdadm: /dev/sdj3 is busy - skipping
mdadm: /dev/sdj2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdj1
mdadm: no RAID superblock on /dev/sdj
mdadm: no RAID superblock on /dev/sdi4
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/2.
mdadm: /dev/sdi3 is busy - skipping
mdadm: /dev/sdi2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdi1
mdadm: no RAID superblock on /dev/sdi
mdadm: no RAID superblock on /dev/sdh4
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/2.
mdadm: /dev/sdh3 is busy - skipping
mdadm: /dev/sdh2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdh1
mdadm: no RAID superblock on /dev/sdh
mdadm: no RAID superblock on /dev/md/2
mdadm: no RAID superblock on /dev/md1
mdadm: no RAID superblock on /dev/sdg4
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/2.
mdadm: /dev/sdg3 is busy - skipping
mdadm: /dev/sdg2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdg1
mdadm: no RAID superblock on /dev/sdg
mdadm: no RAID superblock on /dev/sdb4
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/2.
mdadm: /dev/sdb3 is busy - skipping
mdadm: /dev/sdb2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdb1
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sdf4
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/2.
mdadm: /dev/sdf3 is busy - skipping
mdadm: /dev/sdf2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdf1
mdadm: no RAID superblock on /dev/sdf
mdadm: no RAID superblock on /dev/sde4
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/2.
mdadm: /dev/sde3 is busy - skipping
mdadm: /dev/sde2 has wrong uuid.
mdadm: no RAID superblock on /dev/sde1
mdadm: no RAID superblock on /dev/sde
mdadm: no RAID superblock on /dev/sdd4
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/2.
mdadm: /dev/sdd3 is busy - skipping
mdadm: /dev/sdd2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdd1
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdc4
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/2.
mdadm: /dev/sdc3 is busy - skipping
mdadm: /dev/sdc2 has wrong uuid.
mdadm: no RAID superblock on /dev/sdc1
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sda4
mdadm: UUID differs from /dev/md1.
mdadm: UUID differs from /dev/md/2.
mdadm: /dev/sda3 is busy - skipping
mdadm: /dev/sda2 has wrong uuid.
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda
Note the difference in --display and --examine outputs (see below) about
total and delta device numbers, 10 in detail and 11 in examine.
Also note, that sda3 is marked as "spare" device, I am wrote before in first
message, that some useless commands was executed.
# mdadm --add /dev/md3 --re-add /dev/sda3
mdadm: Cannot get array info for /dev/md3
# This is because array is not started?
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2 : active raid10 sdd2[8] sdg2[6] sdh2[4] sdj2[5] sdi2[11] sdb2[1] sdc2[10] sdf2[0] sde2[9] sda2[12]
35653120 blocks super 1.2 512K chunks 2 near-copies [10/10] [UUUUUUUUUU]
bitmap: 1/1 pages [4KB], 65536KB chunk
md3 : inactive sdg3[3](S) sdh3[2](S) sdj3[10](S) sdi3[12](S) sdb3[6](S) sdc3[11](S) sdf3[0](S) sdd3[4](S) sde3[5](S) sda3[14](S)
38995474635 blocks super 1.2
md1 : active raid1 sdd1[4] sdg1[3] sdh1[2] sdj1[8] sdi1[1] sdb1[6] sdc1[7] sdf1[0] sde1[5] sda1[9]
203712 blocks [10/10] [UUUUUUUUUU]
bitmap: 0/1 pages [0KB], 65536KB chunk
# mdadm --detail /dev/md3
/dev/md3:
Version : 1.2
Raid Level : raid0
Total Devices : 10
Persistence : Superblock is persistent
State : inactive
Delta Devices : 1, (-1->0)
New Level : raid5
New Layout : left-symmetric
New Chunksize : 512K
Name : ubuntu:3
UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Events : 21586
Number Major Minor RaidDevice
- 8 3 - /dev/sda3
- 8 19 - /dev/sdb3
- 8 35 - /dev/sdc3
- 8 51 - /dev/sdd3
- 8 67 - /dev/sde3
- 8 83 - /dev/sdf3
- 8 99 - /dev/sdg3
- 8 115 - /dev/sdh3
- 8 131 - /dev/sdi3
- 8 147 - /dev/sdj3
# mdadm --examine /dev/sd?3
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c0155214:a9c8e378:d368b6f6:735048a8
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 952a55f2 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 375d85e2:9077dd68:bc1327af:8fee4112
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 75b99505 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 67cd2bfa:affa578c:e0c1286a:bd4d2627
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 80c895f3 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4490843e:d2c5c2e7:d222fa0a:f98b277c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 1656c2d2 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 1d41056f:88149b1a:48e939e5:8c775e23
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : fb26746a - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : fa7c541d:a25babd8:14ffe1c0:c6c1578c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : ac275763 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a4b8b425:b0307fe5:9eb8e3ac:ae985311
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 3258f890 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 345b0212:6c3a0ae8:d8ed9273:ecc53573
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 49c30753 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 70afdaad:b0979fb1:1b99ef18:dd58cc3a
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 1c2bf759 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a5f2cea8:a89c88a0:0dd3900c:624f0491
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 4fda6fb3 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
# for x in /dev/sd[a-z] ; do echo $x ; smartctl -i -A $x ; done
/dev/sda
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-57-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E1657399
LU WWN Device Id: 5 0014ee 2b4caa93a
Firmware Version: 80.00A80
User Capacity: 4 000 787 030 016 bytes [4,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Nov 25 14:25:50 2015 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 179 179 021 Pre-fail Always - 8033
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 9
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 089 089 000 Old_age Always - 8756
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 9
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 66
194 Temperature_Celsius 0x0022 117 102 000 Old_age Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
/dev/sdb
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-57-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E1643332
LU WWN Device Id: 5 0014ee 25f74cf3c
Firmware Version: 80.00A80
User Capacity: 4 000 787 030 016 bytes [4,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Nov 25 14:25:50 2015 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 182 181 021 Pre-fail Always - 7866
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 19
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 084 084 000 Old_age Always - 11724
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 19
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 5
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 106
194 Temperature_Celsius 0x0022 117 102 000 Old_age Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
/dev/sdc
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-57-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E1649141
LU WWN Device Id: 5 0014ee 25f74a7c9
Firmware Version: 80.00A80
User Capacity: 4 000 787 030 016 bytes [4,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Nov 25 14:25:50 2015 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 185 183 021 Pre-fail Always - 7741
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 14
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 085 085 000 Old_age Always - 11439
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 14
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 84
194 Temperature_Celsius 0x0022 118 105 000 Old_age Always - 34
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
/dev/sdd
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-57-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E0340253
LU WWN Device Id: 5 0014ee 20921e030
Firmware Version: 80.00A80
User Capacity: 4 000 787 030 016 bytes [4,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Nov 25 14:25:50 2015 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 175 173 021 Pre-fail Always - 8241
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 28
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 081 081 000 Old_age Always - 14575
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 28
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 9
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 1496402
194 Temperature_Celsius 0x0022 114 108 000 Old_age Always - 38
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
/dev/sde
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-57-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E1658818
LU WWN Device Id: 5 0014ee 20a1fca11
Firmware Version: 80.00A80
User Capacity: 4 000 787 030 016 bytes [4,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Nov 25 14:25:50 2015 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 185 183 021 Pre-fail Always - 7733
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 15
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 084 084 000 Old_age Always - 11700
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 15
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 101
194 Temperature_Celsius 0x0022 116 109 000 Old_age Always - 36
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
/dev/sdf
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-57-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E1349511
LU WWN Device Id: 5 0014ee 2b4a00452
Firmware Version: 80.00A80
User Capacity: 4 000 787 030 016 bytes [4,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed Nov 25 14:25:50 2015 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 178 176 021 Pre-fail Always - 8100
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 27
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 082 082 000 Old_age Always - 13576
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 27
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 8
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 120
194 Temperature_Celsius 0x0022 117 110 000 Old_age Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
/dev/sdg
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-57-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E1265787
LU WWN Device Id: 5 0014ee 25f483dfb
Firmware Version: 80.00A80
User Capacity: 4 000 787 030 016 bytes [4,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Nov 25 14:25:50 2015 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 55
3 Spin_Up_Time 0x0027 177 175 021 Pre-fail Always - 8141
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 23
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 084 084 000 Old_age Always - 11746
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 23
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 9
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 104
194 Temperature_Celsius 0x0022 115 107 000 Old_age Always - 37
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
/dev/sdh
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-57-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E1639809
LU WWN Device Id: 5 0014ee 25f7425d2
Firmware Version: 80.00A80
User Capacity: 4 000 787 030 016 bytes [4,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Nov 25 14:25:50 2015 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 180 179 021 Pre-fail Always - 7958
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 17
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 084 084 000 Old_age Always - 11722
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 17
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 103
194 Temperature_Celsius 0x0022 115 107 000 Old_age Always - 37
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
/dev/sdi
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-57-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E1639009
LU WWN Device Id: 5 0014ee 25f752df3
Firmware Version: 80.00A80
User Capacity: 4 000 787 030 016 bytes [4,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Nov 25 14:25:50 2015 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 179 177 021 Pre-fail Always - 8016
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 12
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 085 085 000 Old_age Always - 11393
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 12
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 83
194 Temperature_Celsius 0x0022 117 105 000 Old_age Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
/dev/sdj
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.13.0-57-generic] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: WDC WD40EFRX-68WT0N0
Serial Number: WD-WCC4E1228884
LU WWN Device Id: 5 0014ee 209e58132
Firmware Version: 80.00A80
User Capacity: 4 000 787 030 016 bytes [4,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Nov 25 14:25:51 2015 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 175 174 021 Pre-fail Always - 8208
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 27
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 082 082 000 Old_age Always - 13598
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 27
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 8
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 120
194 Temperature_Celsius 0x0022 117 102 000 Old_age Always - 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
END OF MESSAGE
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-25 12:12 ` Semyon Enskiy
@ 2015-11-25 13:22 ` Phil Turmel
2015-11-25 14:48 ` Semyon Enskiy
0 siblings, 1 reply; 17+ messages in thread
From: Phil Turmel @ 2015-11-25 13:22 UTC (permalink / raw)
To: Semyon Enskiy, linux-raid@vger.kernel.org
On 11/25/2015 07:12 AM, Semyon Enskiy wrote:
> Hi Phil.
>
> Thanks for your suggestions, I have followed them, but RAID5 at md3 not
> recovered at yet, maybe you will find time to suggest anything else.
>
> Exactly, power and data connections of SATA disks was not reliable, fixed
> this and after boot get no errors in kernel logs.
> md3 was replaced with "<ingore>" in mdadm.conf and commented in fstab before
> poweroff, things returned back after boot.
>
> # mdadm --assemble --update=revert-reshape /dev/md3
This ^^^ is a mistake. It did nothing because you didn't list all of
the component partitions. Stop md3 and try this part again. If it
still doesn't work, add --force.
> # mdadm --assemble --scan -vv
> Note the difference in --display and --examine outputs (see below) about
> total and delta device numbers, 10 in detail and 11 in examine.
--display is pretty much useless on an inactive array. The --examine
reports are the ones that matter.
> Also note, that sda3 is marked as "spare" device, I am wrote before in first
> message, that some useless commands was executed.
This is from the --add operation you should not have done.
> # mdadm --add /dev/md3 --re-add /dev/sda3
Augh! Don't guess at what to do! And even if the right thing, the
syntax is wrong (never put --add and --re-add together).
> mdadm: Cannot get array info for /dev/md3
> # This is because array is not started?
Yes.
[trim /]
> # mdadm --examine /dev/sd?3
[trim /]
Very good, you have device roles 0-7 & 9, plus one spare, matching the
array state. Put these devices in the --assemble --update command.
> # for x in /dev/sd[a-z] ; do echo $x ; smartctl -i -A $x ; done
[trim /]
Your drives are entirely healthy. No apparent lasting effects from the
problem cables/power.
For reference, your drive name to serial numbers to roles:
/dev/sda3 ==> WD-WCC4E1657399 ==> spare
/dev/sdb3 ==> WD-WCC4E1643332 ==> role 6
/dev/sdc3 ==> WD-WCC4E1649141 ==> role 7
/dev/sdd3 ==> WD-WCC4E0340253 ==> role 4
/dev/sde3 ==> WD-WCC4E1658818 ==> role 5
/dev/sdf3 ==> WD-WCC4E1349511 ==> role 0
/dev/sdg3 ==> WD-WCC4E1265787 ==> role 3
/dev/sdh3 ==> WD-WCC4E1639809 ==> role 2
/dev/sdi3 ==> WD-WCC4E1639009 ==> role 1
/dev/sdj3 ==> WD-WCC4E1228884 ==> role 9
Your next operations are:
mdadm --stop /dev/md3
mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-i]3
If successful, it should begin rebuilding onto /dev/sda3 (role 8).
If the above fails, repeat with --force. If that fails show the output
and do nothing else.
Phil
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-25 13:22 ` Phil Turmel
@ 2015-11-25 14:48 ` Semyon Enskiy
2015-11-25 14:55 ` Semyon Enskiy
0 siblings, 1 reply; 17+ messages in thread
From: Semyon Enskiy @ 2015-11-25 14:48 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
> Your next operations are:
>
> mdadm --stop /dev/md3
>
> mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-i]3
>
> If successful, it should begin rebuilding onto /dev/sda3 (role 8).
>
> If the above fails, repeat with --force. If that fails show the output
> and do nothing else.
>
> Phil
# mdadm --stop /dev/md3
mdadm: stopped /dev/md3
# mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-i]3
mdadm: looking for devices for /dev/md3
mdadm: /dev/sda3 is identified as a member of /dev/md3, slot -1.
mdadm: /dev/sdb3 is identified as a member of /dev/md3, slot 6.
mdadm: /dev/sdc3 is identified as a member of /dev/md3, slot 7.
mdadm: /dev/sdd3 is identified as a member of /dev/md3, slot 4.
mdadm: /dev/sde3 is identified as a member of /dev/md3, slot 5.
mdadm: /dev/sdf3 is identified as a member of /dev/md3, slot 0.
mdadm: /dev/sdg3 is identified as a member of /dev/md3, slot 3.
mdadm: /dev/sdh3 is identified as a member of /dev/md3, slot 2.
mdadm: /dev/sdi3 is identified as a member of /dev/md3, slot 1.
mdadm: added /dev/sdi3 to /dev/md3 as 1
mdadm: added /dev/sdh3 to /dev/md3 as 2
mdadm: added /dev/sdg3 to /dev/md3 as 3
mdadm: added /dev/sdd3 to /dev/md3 as 4
mdadm: added /dev/sde3 to /dev/md3 as 5
mdadm: added /dev/sdb3 to /dev/md3 as 6
mdadm: added /dev/sdc3 to /dev/md3 as 7
mdadm: no uptodate device for slot 16 of /dev/md3
mdadm: no uptodate device for slot 18 of /dev/md3
mdadm: no uptodate device for slot 20 of /dev/md3
mdadm: added /dev/sda3 to /dev/md3 as -1
mdadm: added /dev/sdf3 to /dev/md3 as 0
mdadm: /dev/md3 assembled from 8 drives and 1 spare - not enough to start the array.
# mdadm -v --assemble --update=revert-reshape --force /dev/md3 /dev/sd[a-i]3
mdadm: looking for devices for /dev/md3
mdadm: /dev/sda3 is identified as a member of /dev/md3, slot -1.
mdadm: /dev/sdb3 is identified as a member of /dev/md3, slot 6.
mdadm: /dev/sdc3 is identified as a member of /dev/md3, slot 7.
mdadm: /dev/sdd3 is identified as a member of /dev/md3, slot 4.
mdadm: /dev/sde3 is identified as a member of /dev/md3, slot 5.
mdadm: /dev/sdf3 is identified as a member of /dev/md3, slot 0.
mdadm: /dev/sdg3 is identified as a member of /dev/md3, slot 3.
mdadm: /dev/sdh3 is identified as a member of /dev/md3, slot 2.
mdadm: /dev/sdi3 is identified as a member of /dev/md3, slot 1.
mdadm: added /dev/sdi3 to /dev/md3 as 1
mdadm: added /dev/sdh3 to /dev/md3 as 2
mdadm: added /dev/sdg3 to /dev/md3 as 3
mdadm: added /dev/sdd3 to /dev/md3 as 4
mdadm: added /dev/sde3 to /dev/md3 as 5
mdadm: added /dev/sdb3 to /dev/md3 as 6
mdadm: added /dev/sdc3 to /dev/md3 as 7
mdadm: no uptodate device for slot 16 of /dev/md3
mdadm: no uptodate device for slot 18 of /dev/md3
mdadm: no uptodate device for slot 20 of /dev/md3
mdadm: added /dev/sda3 to /dev/md3 as -1
mdadm: added /dev/sdf3 to /dev/md3 as 0
mdadm: /dev/md3 assembled from 8 drives and 1 spare - not enough to start the array.
# mdadm --examine /dev/sd?3
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c0155214:a9c8e378:d368b6f6:735048a8
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 952aa5ef - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 375d85e2:9077dd68:bc1327af:8fee4112
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 75b9e502 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 67cd2bfa:affa578c:e0c1286a:bd4d2627
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 80c8e5f0 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4490843e:d2c5c2e7:d222fa0a:f98b277c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 165712cf - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 1d41056f:88149b1a:48e939e5:8c775e23
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : fb26c467 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : fa7c541d:a25babd8:14ffe1c0:c6c1578c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : ac27a760 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a4b8b425:b0307fe5:9eb8e3ac:ae985311
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 3259488d - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 345b0212:6c3a0ae8:d8ed9273:ecc53573
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 49c35750 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 70afdaad:b0979fb1:1b99ef18:dd58cc3a
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 1c2c4756 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 11
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 38995471360 (37188.98 GiB 39931.36 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
New Offset : 251904 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a5f2cea8:a89c88a0:0dd3900c:624f0491
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : 1 (10->11)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 4fda6fb3 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-25 14:48 ` Semyon Enskiy
@ 2015-11-25 14:55 ` Semyon Enskiy
2015-11-25 15:44 ` Phil Turmel
0 siblings, 1 reply; 17+ messages in thread
From: Semyon Enskiy @ 2015-11-25 14:55 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
Maybe assemble command should include sdj3 device too?
Like
# mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-j]3
not
# mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-i]3
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-25 14:55 ` Semyon Enskiy
@ 2015-11-25 15:44 ` Phil Turmel
2015-11-25 16:29 ` Semyon Enskiy
0 siblings, 1 reply; 17+ messages in thread
From: Phil Turmel @ 2015-11-25 15:44 UTC (permalink / raw)
To: Semyon Enskiy, linux-raid@vger.kernel.org
On 11/25/2015 09:55 AM, Semyon Enskiy wrote:
> Maybe assemble command should include sdj3 device too?
> Like
> # mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-j]3
> not
> # mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-i]3
Yes, my mistake. :-( Try again, including 'sdj3'.
Phil
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-25 15:44 ` Phil Turmel
@ 2015-11-25 16:29 ` Semyon Enskiy
2015-11-25 16:58 ` Phil Turmel
0 siblings, 1 reply; 17+ messages in thread
From: Semyon Enskiy @ 2015-11-25 16:29 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
>> Maybe assemble command should include sdj3 device too?
>> Like
>> # mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-j]3
>> not
>> # mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-i]3
>
> Yes, my mistake. :-( Try again, including 'sdj3'.
>
> Phil
# mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-j]3
mdadm: looking for devices for /dev/md3
mdadm: superblock on /dev/sdj3 doesn't match others - assembly aborted
# mdadm -v --assemble --update=revert-reshape --force /dev/md3 /dev/sd[a-j]3
mdadm: looking for devices for /dev/md3
mdadm: superblock on /dev/sdj3 doesn't match others - assembly aborted
Note, that --examine in previous message shows delta devices value "-1" for
all members of array, excepting sdj3, which has value "1", without minus
symbol.
According to mdadm wiki 1.2v superblock placed on device after first 4K and
has size 256 Bytes, plus 2 Bytes per device in the array, in current case
this is 276 Bytes. Dumping superblock:
# for i in /dev/sd?3; do dd ibs=1c skip=4000 count=276 if=$i of=`echo $i | tr -d /`-superblock.raw; done
# for i in devsd*.raw; do echo -e "\n$i" && hd -v $i; done > superblock-dump-000.txt
devsda3-superblock.raw
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 fc 4e 2b a9 01 00 00 00 45 00 00 00 00 00 00 00 |.N+.....E.......|
00000070 9b dc 09 39 48 38 89 af b6 7c 19 9b 2a ae cb 4e |...9H8...|..*..N|
00000080 75 62 75 6e 74 75 3a 33 00 00 00 00 00 00 00 00 |ubuntu:3........|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 c8 f6 d0 53 00 00 00 00 05 00 00 00 02 00 00 00 |...S............|
000000b0 00 bc dc d0 01 00 00 00 00 04 00 00 0a 00 00 00 |................|
000000c0 08 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 ff ff ff ff 02 00 00 00 00 04 00 00 00 28 00 00 |.............(..|
000000e0 00 d8 03 00 00 00 00 00 8f e6 dc d0 01 00 00 00 |................|
000000f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 0e 00 00 00 00 00 00 00 c0 15 52 14 a9 c8 e3 78 |..........R....x|
00000110 d3 68 b6 f6 |.h..|
00000114
devsdb3-superblock.raw
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 fc 4e 2b a9 01 00 00 00 45 00 00 00 00 00 00 00 |.N+.....E.......|
00000070 9b dc 09 39 48 38 89 af b6 7c 19 9b 2a ae cb 4e |...9H8...|..*..N|
00000080 75 62 75 6e 74 75 3a 33 00 00 00 00 00 00 00 00 |ubuntu:3........|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 c8 f6 d0 53 00 00 00 00 05 00 00 00 02 00 00 00 |...S............|
000000b0 00 bc dc d0 01 00 00 00 00 04 00 00 0a 00 00 00 |................|
000000c0 08 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 ff ff ff ff 02 00 00 00 00 04 00 00 00 28 00 00 |.............(..|
000000e0 00 d8 03 00 00 00 00 00 8f e6 dc d0 01 00 00 00 |................|
000000f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 06 00 00 00 00 00 00 00 37 5d 85 e2 90 77 dd 68 |........7]...w.h|
00000110 bc 13 27 af |..'.|
00000114
devsdc3-superblock.raw
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 fc 4e 2b a9 01 00 00 00 45 00 00 00 00 00 00 00 |.N+.....E.......|
00000070 9b dc 09 39 48 38 89 af b6 7c 19 9b 2a ae cb 4e |...9H8...|..*..N|
00000080 75 62 75 6e 74 75 3a 33 00 00 00 00 00 00 00 00 |ubuntu:3........|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 c8 f6 d0 53 00 00 00 00 05 00 00 00 02 00 00 00 |...S............|
000000b0 00 bc dc d0 01 00 00 00 00 04 00 00 0a 00 00 00 |................|
000000c0 08 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 ff ff ff ff 02 00 00 00 00 04 00 00 00 28 00 00 |.............(..|
000000e0 00 d8 03 00 00 00 00 00 8f e6 dc d0 01 00 00 00 |................|
000000f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 0b 00 00 00 00 00 00 00 67 cd 2b fa af fa 57 8c |........g.+...W.|
00000110 e0 c1 28 6a |..(j|
00000114
devsdd3-superblock.raw
00000000 4e fc 4f 03 50 51 50 4e fe 4f 02 50 4e 4e fe 4f |N.O.PQPN.O.PNN.O|
00000010 ff 4b ff 49 1b 4a 4e 4d 4a 4d 4b 4c 49 4a 46 47 |.K.I.JNMJMKLIJFG|
00000020 47 49 48 49 48 47 46 45 45 46 46 45 47 48 46 48 |GIHIHGFEEFFEGHFH|
00000030 46 fe 45 fe 46 00 42 fe 46 08 44 45 44 47 46 46 |F.E.F.B.F.DEDGFF|
00000040 45 45 46 fd 45 05 46 45 46 46 45 45 fc 46 03 45 |EEF.E.FEFFEE.F.E|
00000050 46 46 45 fc 46 13 45 44 46 45 44 46 45 45 46 45 |FFE.F.EDFEDFEEFE|
00000060 fc 4e 2b a9 01 00 00 00 45 00 00 00 00 00 00 00 |.N+.....E.......|
00000070 9b dc 09 39 48 38 89 af b6 7c 19 9b 2a ae cb 4e |...9H8...|..*..N|
00000080 75 62 75 6e 74 75 3a 33 00 00 00 00 00 00 00 00 |ubuntu:3........|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 c8 f6 d0 53 00 00 00 00 05 00 00 00 02 00 00 00 |...S............|
000000b0 00 bc dc d0 01 00 00 00 00 04 00 00 0a 00 00 00 |................|
000000c0 08 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 ff ff ff ff 02 00 00 00 00 04 00 00 00 28 00 00 |.............(..|
000000e0 00 d8 03 00 00 00 00 00 8f e6 dc d0 01 00 00 00 |................|
000000f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 04 00 00 00 00 00 00 00 44 90 84 3e d2 c5 c2 e7 |........D..>....|
00000110 d2 22 fa 0a |."..|
00000114
devsde3-superblock.raw
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 fc 4e 2b a9 01 00 00 00 45 00 00 00 00 00 00 00 |.N+.....E.......|
00000070 9b dc 09 39 48 38 89 af b6 7c 19 9b 2a ae cb 4e |...9H8...|..*..N|
00000080 75 62 75 6e 74 75 3a 33 00 00 00 00 00 00 00 00 |ubuntu:3........|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 c8 f6 d0 53 00 00 00 00 05 00 00 00 02 00 00 00 |...S............|
000000b0 00 bc dc d0 01 00 00 00 00 04 00 00 0a 00 00 00 |................|
000000c0 08 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 ff ff ff ff 02 00 00 00 00 04 00 00 00 28 00 00 |.............(..|
000000e0 00 d8 03 00 00 00 00 00 8f e6 dc d0 01 00 00 00 |................|
000000f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 05 00 00 00 00 00 00 00 1d 41 05 6f 88 14 9b 1a |.........A.o....|
00000110 48 e9 39 e5 |H.9.|
00000114
devsdf3-superblock.raw
00000000 3c 39 37 39 3a 3a 39 38 36 39 38 37 34 35 35 36 |<979::9869874556|
00000010 36 31 37 35 35 31 31 35 36 31 34 35 37 35 35 37 |6175511561457557|
00000020 37 35 31 36 36 35 37 38 36 35 34 38 37 37 35 39 |7516657865487759|
00000030 3a 37 3a 3b ff 38 09 3b 3a 3a 3b 3a 3d 3b 3c 3c |:7:;.8.;::;:=;<<|
00000040 40 fe 3d fc 41 03 45 43 43 41 fe 43 ff 46 02 45 |@.=.A.ECCA.C.F.E|
00000050 48 48 fa 4a 14 4b 4a 4e 4a 4d 4d 4f 4f 4e 4e 51 |HH.J.KJNJMMOONNQ|
00000060 fc 4e 2b a9 01 00 00 00 45 00 00 00 00 00 00 00 |.N+.....E.......|
00000070 9b dc 09 39 48 38 89 af b6 7c 19 9b 2a ae cb 4e |...9H8...|..*..N|
00000080 75 62 75 6e 74 75 3a 33 00 00 00 00 00 00 00 00 |ubuntu:3........|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 c8 f6 d0 53 00 00 00 00 05 00 00 00 02 00 00 00 |...S............|
000000b0 00 bc dc d0 01 00 00 00 00 04 00 00 0a 00 00 00 |................|
000000c0 08 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 ff ff ff ff 02 00 00 00 00 04 00 00 00 28 00 00 |.............(..|
000000e0 00 d8 03 00 00 00 00 00 8f e6 dc d0 01 00 00 00 |................|
000000f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 00 00 00 00 00 00 00 00 fa 7c 54 1d a2 5b ab d8 |.........|T..[..|
00000110 14 ff e1 c0 |....|
00000114
devsdg3-superblock.raw
00000000 5e 75 7c 7f 74 7b 81 8a 97 a0 a8 af b0 be c6 da |^u|.t{..........|
00000010 e5 e9 f2 f3 f6 f7 f2 ef ee ed e9 e7 e8 e8 e3 de |................|
00000020 df e1 df e0 da de e2 e0 df d9 dc d5 d3 ce d3 d3 |................|
00000030 d1 ce cb 07 d2 d5 d7 d7 da d7 d7 db fe da 4a d9 |..............J.|
00000040 d6 d6 d5 d5 d6 d2 ce cc d2 d1 cf d0 d2 d4 d9 db |................|
00000050 db dc da db da da d8 da da d7 cf d3 cf ce ce d2 |................|
00000060 fc 4e 2b a9 01 00 00 00 45 00 00 00 00 00 00 00 |.N+.....E.......|
00000070 9b dc 09 39 48 38 89 af b6 7c 19 9b 2a ae cb 4e |...9H8...|..*..N|
00000080 75 62 75 6e 74 75 3a 33 00 00 00 00 00 00 00 00 |ubuntu:3........|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 c8 f6 d0 53 00 00 00 00 05 00 00 00 02 00 00 00 |...S............|
000000b0 00 bc dc d0 01 00 00 00 00 04 00 00 0a 00 00 00 |................|
000000c0 08 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 ff ff ff ff 02 00 00 00 00 04 00 00 00 28 00 00 |.............(..|
000000e0 00 d8 03 00 00 00 00 00 8f e6 dc d0 01 00 00 00 |................|
000000f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 03 00 00 00 00 00 00 00 a4 b8 b4 25 b0 30 7f e5 |...........%.0..|
00000110 9e b8 e3 ac |....|
00000114
devsdh3-superblock.raw
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 fc 4e 2b a9 01 00 00 00 45 00 00 00 00 00 00 00 |.N+.....E.......|
00000070 9b dc 09 39 48 38 89 af b6 7c 19 9b 2a ae cb 4e |...9H8...|..*..N|
00000080 75 62 75 6e 74 75 3a 33 00 00 00 00 00 00 00 00 |ubuntu:3........|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 c8 f6 d0 53 00 00 00 00 05 00 00 00 02 00 00 00 |...S............|
000000b0 00 bc dc d0 01 00 00 00 00 04 00 00 0a 00 00 00 |................|
000000c0 08 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 ff ff ff ff 02 00 00 00 00 04 00 00 00 28 00 00 |.............(..|
000000e0 00 d8 03 00 00 00 00 00 8f e6 dc d0 01 00 00 00 |................|
000000f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 02 00 00 00 00 00 00 00 34 5b 02 12 6c 3a 0a e8 |........4[..l:..|
00000110 d8 ed 92 73 |...s|
00000114
devsdi3-superblock.raw
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 fc 4e 2b a9 01 00 00 00 45 00 00 00 00 00 00 00 |.N+.....E.......|
00000070 9b dc 09 39 48 38 89 af b6 7c 19 9b 2a ae cb 4e |...9H8...|..*..N|
00000080 75 62 75 6e 74 75 3a 33 00 00 00 00 00 00 00 00 |ubuntu:3........|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 c8 f6 d0 53 00 00 00 00 05 00 00 00 02 00 00 00 |...S............|
000000b0 00 bc dc d0 01 00 00 00 00 04 00 00 0a 00 00 00 |................|
000000c0 08 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 ff ff ff ff 02 00 00 00 00 04 00 00 00 28 00 00 |.............(..|
000000e0 00 d8 03 00 00 00 00 00 8f e6 dc d0 01 00 00 00 |................|
000000f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 0c 00 00 00 00 00 00 00 70 af da ad b0 97 9f b1 |........p.......|
00000110 1b 99 ef 18 |....|
00000114
devsdj3-superblock.raw
00000000 8b f5 bf f2 9f 03 b5 cb fd 29 28 17 15 d9 c5 6a |.........)(....j|
00000010 37 52 cb f3 72 47 43 45 3c 44 23 33 2e d2 de fa |7R..rGCE<D#3....|
00000020 e1 d6 e4 da fb f1 7e c9 ec 0c ea e1 c9 f0 f2 f2 |......~.........|
00000030 db cd c9 2f 3a 70 d1 2a ed a0 d0 d1 f9 da 4c cd |.../:p.*......L.|
00000040 b7 05 f1 cb 2d ad 6b b9 db 39 54 8a 92 cb 90 72 |....-.k..9T....r|
00000050 3c 67 65 43 90 fa ba ba b0 b7 dd b4 af a4 a1 b1 |<geC............|
00000060 fc 4e 2b a9 01 00 00 00 45 00 00 00 00 00 00 00 |.N+.....E.......|
00000070 9b dc 09 39 48 38 89 af b6 7c 19 9b 2a ae cb 4e |...9H8...|..*..N|
00000080 75 62 75 6e 74 75 3a 33 00 00 00 00 00 00 00 00 |ubuntu:3........|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 c8 f6 d0 53 00 00 00 00 05 00 00 00 02 00 00 00 |...S............|
000000b0 00 bc dc d0 01 00 00 00 00 04 00 00 0b 00 00 00 |................|
000000c0 08 00 00 00 05 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 01 00 00 00 02 00 00 00 00 04 00 00 00 d8 ff ff |................|
000000e0 00 00 04 00 00 00 00 00 8f be dc d0 01 00 00 00 |................|
000000f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 0a 00 00 00 00 00 00 00 a5 f2 ce a8 a8 9c 88 a0 |................|
00000110 0d d3 90 0c |....|
00000114
END OF MESSAGE
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-25 16:29 ` Semyon Enskiy
@ 2015-11-25 16:58 ` Phil Turmel
2015-11-25 17:32 ` Semyon Enskiy
0 siblings, 1 reply; 17+ messages in thread
From: Phil Turmel @ 2015-11-25 16:58 UTC (permalink / raw)
To: Semyon Enskiy, linux-raid@vger.kernel.org
On 11/25/2015 11:29 AM, Semyon Enskiy wrote:
>>> Maybe assemble command should include sdj3 device too?
>>> Like
>>> # mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-j]3
>>> not
>>> # mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-i]3
>>
>> Yes, my mistake. :-( Try again, including 'sdj3'.
>>
>> Phil
>
> # mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-j]3
> mdadm: looking for devices for /dev/md3
> mdadm: superblock on /dev/sdj3 doesn't match others - assembly aborted
>
> # mdadm -v --assemble --update=revert-reshape --force /dev/md3 /dev/sd[a-j]3
> mdadm: looking for devices for /dev/md3
> mdadm: superblock on /dev/sdj3 doesn't match others - assembly aborted
>
> Note, that --examine in previous message shows delta devices value "-1" for
> all members of array, excepting sdj3, which has value "1", without minus
> symbol.
Hmmm. That's interesting. That means mdadm wrote the revert-reshape
before it finished assembly. Try this:
mdadm --stop /dev/md3
mdadm --assemble --update=revert-reshape /dev/md3 /dev/sdj3
That should fail w/ not enough to start, but apply the revert to that
one drive. Then --stop again, and try with [a-j].
Phil
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-25 16:58 ` Phil Turmel
@ 2015-11-25 17:32 ` Semyon Enskiy
2015-11-25 17:49 ` Phil Turmel
0 siblings, 1 reply; 17+ messages in thread
From: Semyon Enskiy @ 2015-11-25 17:32 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
> Hmmm. That's interesting. That means mdadm wrote the revert-reshape
> before it finished assembly. Try this:
>
> mdadm --stop /dev/md3
>
> mdadm --assemble --update=revert-reshape /dev/md3 /dev/sdj3
>
> That should fail w/ not enough to start, but apply the revert to that
> one drive. Then --stop again, and try with [a-j].
Phil, thanks for your sustaining.
Does you think, that problem is that "Raid Devices : 10", but
"Array State : AAAAAAAA.A.", where the extra, 11th, device is sdj3, which
was rejected from system in past, but record about it leave in superblock?
# mdadm --stop /dev/md3
mdadm: error opening /dev/md3: No such file or directory
# mdadm --assemble --update=revert-reshape /dev/md3 /dev/sdj3
mdadm: /dev/md3 assembled from 1 drive - not enough to start the array.
# mdadm --stop /dev/md3
mdadm: error opening /dev/md3: No such file or directory
# mdadm -v --assemble --update=revert-reshape /dev/md3 /dev/sd[a-j]3
mdadm: looking for devices for /dev/md3
mdadm: /dev/sda3 is identified as a member of /dev/md3, slot -1.
mdadm: /dev/sdb3 is identified as a member of /dev/md3, slot 6.
mdadm: /dev/sdc3 is identified as a member of /dev/md3, slot 7.
mdadm: /dev/sdd3 is identified as a member of /dev/md3, slot 4.
mdadm: /dev/sde3 is identified as a member of /dev/md3, slot 5.
mdadm: /dev/sdf3 is identified as a member of /dev/md3, slot 0.
mdadm: /dev/sdg3 is identified as a member of /dev/md3, slot 3.
mdadm: /dev/sdh3 is identified as a member of /dev/md3, slot 2.
mdadm: /dev/sdi3 is identified as a member of /dev/md3, slot 1.
mdadm: /dev/sdj3 is identified as a member of /dev/md3, slot 9.
mdadm: added /dev/sdi3 to /dev/md3 as 1
mdadm: added /dev/sdh3 to /dev/md3 as 2
mdadm: added /dev/sdg3 to /dev/md3 as 3
mdadm: added /dev/sdd3 to /dev/md3 as 4
mdadm: added /dev/sde3 to /dev/md3 as 5
mdadm: added /dev/sdb3 to /dev/md3 as 6
mdadm: added /dev/sdc3 to /dev/md3 as 7
mdadm: no uptodate device for slot 16 of /dev/md3
mdadm: added /dev/sdj3 to /dev/md3 as 9
mdadm: no uptodate device for slot 20 of /dev/md3
mdadm: added /dev/sda3 to /dev/md3 as -1
mdadm: added /dev/sdf3 to /dev/md3 as 0
mdadm: /dev/md3 assembled from 9 drives and 1 spare - not enough to start the array.
# mdadm -v --assemble --update=revert-reshape --force /dev/md3 /dev/sd[a-j]3
mdadm: looking for devices for /dev/md3
mdadm: /dev/sda3 is identified as a member of /dev/md3, slot -1.
mdadm: /dev/sdb3 is identified as a member of /dev/md3, slot 6.
mdadm: /dev/sdc3 is identified as a member of /dev/md3, slot 7.
mdadm: /dev/sdd3 is identified as a member of /dev/md3, slot 4.
mdadm: /dev/sde3 is identified as a member of /dev/md3, slot 5.
mdadm: /dev/sdf3 is identified as a member of /dev/md3, slot 0.
mdadm: /dev/sdg3 is identified as a member of /dev/md3, slot 3.
mdadm: /dev/sdh3 is identified as a member of /dev/md3, slot 2.
mdadm: /dev/sdi3 is identified as a member of /dev/md3, slot 1.
mdadm: /dev/sdj3 is identified as a member of /dev/md3, slot 9.
mdadm: added /dev/sdi3 to /dev/md3 as 1
mdadm: added /dev/sdh3 to /dev/md3 as 2
mdadm: added /dev/sdg3 to /dev/md3 as 3
mdadm: added /dev/sdd3 to /dev/md3 as 4
mdadm: added /dev/sde3 to /dev/md3 as 5
mdadm: added /dev/sdb3 to /dev/md3 as 6
mdadm: added /dev/sdc3 to /dev/md3 as 7
mdadm: no uptodate device for slot 16 of /dev/md3
mdadm: added /dev/sdj3 to /dev/md3 as 9
mdadm: no uptodate device for slot 20 of /dev/md3
mdadm: added /dev/sda3 to /dev/md3 as -1
mdadm: added /dev/sdf3 to /dev/md3 as 0
mdadm: /dev/md3 assembled from 9 drives and 1 spare - not enough to start the array.
# mdadm --examine /dev/sd?3
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : c0155214:a9c8e378:d368b6f6:735048a8
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 952aa5ef - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : spare
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 375d85e2:9077dd68:bc1327af:8fee4112
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 75b9e502 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 67cd2bfa:affa578c:e0c1286a:bd4d2627
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 80c8e5f0 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4490843e:d2c5c2e7:d222fa0a:f98b277c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 165712cf - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 1d41056f:88149b1a:48e939e5:8c775e23
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : fb26c467 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdf3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : fa7c541d:a25babd8:14ffe1c0:c6c1578c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : ac27a760 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a4b8b425:b0307fe5:9eb8e3ac:ae985311
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 3259488d - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 345b0212:6c3a0ae8:d8ed9273:ecc53573
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 49c35750 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdi3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 70afdaad:b0979fb1:1b99ef18:dd58cc3a
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 1c2c4756 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 9bdc0939:483889af:b67c199b:2aaecb4e
Name : ubuntu:3
Creation Time : Thu Jul 24 16:06:32 2014
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799105167 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 251904 sectors
New Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : a5f2cea8:a89c88a0:0dd3900c:624f0491
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 0
Delta Devices : -1 (11->10)
Update Time : Fri Nov 6 16:53:06 2015
Checksum : 4fdabfb0 - correct
Events : 21586
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAAA.A. ('A' == active, '.' == missing, 'R' == replacing)
END OF MESSAGE
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-25 17:32 ` Semyon Enskiy
@ 2015-11-25 17:49 ` Phil Turmel
2015-11-26 13:21 ` Semyon Enskiy
0 siblings, 1 reply; 17+ messages in thread
From: Phil Turmel @ 2015-11-25 17:49 UTC (permalink / raw)
To: Semyon Enskiy, linux-raid@vger.kernel.org
On 11/25/2015 12:32 PM, Semyon Enskiy wrote:
> Does you think, that problem is that "Raid Devices : 10", but
> "Array State : AAAAAAAA.A.", where the extra, 11th, device is sdj3, which
> was rejected from system in past, but record about it leave in superblock?
No. It just can't get past the fact that your array is double-degraded,
even though it isn't really, once the revert succeeds.
At this point, I don't see a way out without --create. Fortunately, we
have complete details on your array, so we can supply the correct paramters:
mdadm --stop /dev/md3
mdadm --create --assume-clean --chunk=512 --data-offset=262144 \
--level=5 --raid-devices=10 --metadata=1.2 /dev/md3 \
/dev/sd{f,i,h,g,d,e,b,c}3 missing /dev/sdj3
Note that you cannot use [] notation -- it doesn't maintain the order of
the letters. The keyword 'missing' takes the place of /dev/sda3. It's
possible it would be fine, but I'd rather not risk it.
You should be able to access the content at this point. Copy out any
super-critical files. Then add sda3:
mdadm --add /dev/md3 /dev/sda3
This should initiate the rebuild.
Update your mdadm.conf file to use the new UUID, and update your initramfs.
Phil
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
2015-11-25 17:49 ` Phil Turmel
@ 2015-11-26 13:21 ` Semyon Enskiy
0 siblings, 0 replies; 17+ messages in thread
From: Semyon Enskiy @ 2015-11-26 13:21 UTC (permalink / raw)
To: Phil Turmel, linux-raid@vger.kernel.org
Should say firstly, that before committing any changes, 10 MB of /dev/sd?3
head and tail was dumped, so array can be returned to initial state.
> mdadm --create --assume-clean --chunk=512 --data-offset=262144 \
> --level=5 --raid-devices=10 --metadata=1.2 /dev/md3 \
> /dev/sd{f,i,h,g,d,e,b,c}3 missing /dev/sdj3
Done this with overwriting old superblock, no errors occurred.
Then I booted to Debian Live, mdadm.conf got old array description with new
one, so old was commented:
# ARRAY /dev/md/3 metadata=1.2 UUID=9bdc0939:483889af:b67c199b:2aaecb4e name=ubuntu:3
spares=1
ARRAY /dev/md/3 metadata=1.2 UUID=89dc7cd3:a7641d65:fe399415:9cdfd5a5 name=bnt:3
Previously "blkid" returned
/dev/md3: UUID="4bz7kl-2QWk-4YZX-IDmH-IFXL-eM3l-F2dB2n" TYPE="LVM2_member"
now nothing.
Regardless to this I am attempted to start LVM, LVM was waiting dev with
UUID from blkid output, so starting was possible only on recovery mode,
"--activationmode partial", as result "data" logical volume's block dev was
created as in past, but mount not found FS superblock, LVM reported errors
on logical extend at that time. As I remember, the Testdisk utility reported
buffer I/O errors while running on logical volume's block dev, but reported
nothing while running on array's block dev.
Apparently, revert-reshape was unsuccessful, so I'm going to collect more
information before going to forward.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: Recovering RAID5 with 2, actually 1, faulty disks.
@ 2015-12-02 13:35 Semyon Enskiy
0 siblings, 0 replies; 17+ messages in thread
From: Semyon Enskiy @ 2015-12-02 13:35 UTC (permalink / raw)
To: philip, linux-raid@vger.kernel.org
Hello, Phil, thank you for help.
Writing to report, that array and LVM/ext4 on it successfully restored.
# Before rebooting and committing changes, dump of first 10 MB was done,
now restore headers.
# for i in f i h g d e b c a j; do dd if=devsd"$i"3-head-bigger.raw
of=/dev/sd"$i"3; done
# Without revert-reshape assembling.
# mdadm --create --assume-clean --level=5 --raid-devices=10
--metadata=1.2 /dev/md3 /dev/sd{f,i,h,g,d,e,b,c,a,j}3
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5]
[raid4] [raid10]
md3 : active raid5 sdj3[9] sda3[8] sdc3[7] sdb3[6] sde3[5] sdd3[4]
sdg3[3] sdh3[2] sdi3[1] sdf3[0]
35095924224 blocks super 1.2 level 5, 512k chunk, algorithm 2
[10/10] [UUUUUUUUUU]
bitmap: 1/30 pages [4KB], 65536KB chunk
# blkid /dev/md3
/dev/md3: UUID="4bz7kl-2QWk-4YZX-IDmH-IFXL-eM3l-F2dB2n" TYPE="LVM2_member"
# vgchange -aay
# blkid /dev/vg/data
/dev/vg/data: UUID="60e11554-074d-4156-a4d3-ed43318f5446" TYPE="ext4"
# fsck.ext4 /dev/vg/data
e2fsck 1.42.9 (4-Feb-2014)
/dev/vg/data: clean, 464/548112384 files, 4798924497/8769786880 blocks
# mount /dev/vg/data
# echo check > /sys/block/md3/md/sync_action
# mdadm --detail /dev/md3
/dev/md3:
Version : 1.2
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 3899547136 (3718.90 GiB 3993.14 GB)
Raid Devices : 10
Total Devices : 10
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Dec 2 13:40:04 2015
State : active
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : ubuntu:3 (local to host ubuntu)
UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Events : 2
Number Major Minor RaidDevice State
0 8 83 0 active sync /dev/sdf3
1 8 131 1 active sync /dev/sdi3
2 8 115 2 active sync /dev/sdh3
3 8 99 3 active sync /dev/sdg3
4 8 51 4 active sync /dev/sdd3
5 8 67 5 active sync /dev/sde3
6 8 19 6 active sync /dev/sdb3
7 8 35 7 active sync /dev/sdc3
8 8 3 8 active sync /dev/sda3
9 8 147 9 active sync /dev/sdj3
# mdadm --examine /dev/sd?3
/dev/sda3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 074ebbe7:81662e37:cc777096:3d50d481
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : d47becef - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 8
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdb3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 4adc3cf2:9d813d1e:ab8044ab:0cc5afe8
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 41bc13fb - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : ddc9c73b:007f6c0f:35a03e5c:e5fc8157
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9c425653 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 73077551:029e51dc:6d036bf4:b81195a4
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 64142af5 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sde3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 1edcf3a7:9decc271:262271a6:54cff640
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 9e6c2a90 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdf3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 0438b536:b5943d93:70280dd9:70952d14
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 547afaef - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdg3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : e16e0db6:5d8e9016:84d2034b:7519f24d
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 2e15990 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdh3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 32bdd4dd:934c3ba5:adc8bfc2:0f1f323e
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 214f61da - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdi3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : bd0d56ba:78e5ee76:d600d587:ab65cbc9
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 2032ca0e - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
/dev/sdj3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 39b91f6d:7fd12764:825bb342:dd4d14fd
Name : ubuntu:3 (local to host ubuntu)
Creation Time : Wed Dec 2 13:09:01 2015
Raid Level : raid5
Raid Devices : 10
Avail Dev Size : 7799094927 (3718.90 GiB 3993.14 GB)
Array Size : 35095924224 (33470.08 GiB 35938.23 GB)
Used Dev Size : 7799094272 (3718.90 GiB 3993.14 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262056 sectors, after=655 sectors
State : clean
Device UUID : 35d4e323:8e2485cb:2c6492a1:e9f8e215
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Dec 2 15:19:29 2015
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 442bc637 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAAAAA ('A' == active, '.' == missing, 'R' ==
replacing)
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2015-12-02 13:35 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-11-23 15:28 Recovering RAID5 with 2, actually 1, faulty disks Semyon Enskiy
2015-11-23 15:35 ` Semyon Enskiy
2015-11-23 15:47 ` Wols Lists
2015-11-23 15:57 ` Semyon Enskiy
2015-11-23 16:08 ` Wols Lists
2015-11-23 18:24 ` Phil Turmel
2015-11-25 12:12 ` Semyon Enskiy
2015-11-25 13:22 ` Phil Turmel
2015-11-25 14:48 ` Semyon Enskiy
2015-11-25 14:55 ` Semyon Enskiy
2015-11-25 15:44 ` Phil Turmel
2015-11-25 16:29 ` Semyon Enskiy
2015-11-25 16:58 ` Phil Turmel
2015-11-25 17:32 ` Semyon Enskiy
2015-11-25 17:49 ` Phil Turmel
2015-11-26 13:21 ` Semyon Enskiy
-- strict thread matches above, loose matches on Subject: below --
2015-12-02 13:35 Semyon Enskiy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.