* [New Here] Need some help recovering failed Raid-5, Cant seem to re-create the same size array.
@ 2012-01-15 4:33 Steve Vialle
0 siblings, 0 replies; only message in thread
From: Steve Vialle @ 2012-01-15 4:33 UTC (permalink / raw)
To: linux-raid
First off, Hi all.
First time posting to _any_ mailing list so feel free to correct any
misdemeanors on my part :-)
My situation is this:
I decided to convert my operational raid-5 (5x2Tb devices + 1 spare) into a
raid-6 using all 6 devices.
During the reshape, devices 0 & 2 failed due to a dodgy power cable :-(
*REBOOT* && replace cable.
I attempted to assemble & restart the reshape with the 2 devices plugged back
in:
$>mdadm --assemble /dev/md3 /dev/sdg1 /dev/sdf1 /dev/sdi1 /dev/sde1 /dev/sdh1
/dev/sdj1 --backup-file=/root/backup-md3
Got:
mdadm: device 6 in /dev/md3 has wrong state in superblock, but /dev/sdg1 seems
ok
mdadm: accepting backup with timestamp 1326496445 for array with timestamp
1326504140
mdadm: restoring critical section
mdadm: /dev/md3 assembled from 3 drives, 1 rebuilding and 2 spares - not
enough to start the array.
$> mdadm -S /dev/md3
I then set up a dm sandbox with
dmsetup create sdg1 --table "0 3907027120 snapshot /dev/sdg1 /dev/loop0 N 1"
etc. for each drive (I am under the impression this will keep me from
buggering things up further, seems to work as advertised)
Attempted to asemble the array again with --force:
$>mdadm --assemble /dev/md3 /dev/mapper/sdg1 /dev/mapper/sdf1 /dev/mapper/sdi1
/dev/mapper/sde1 /dev/mapper/sdh1 /dev/mapper/sdj1 --backup-file=/root/backup-
md3 --force
Got:
mdadm: clearing FAULTY flag for device 0 in /dev/md3 for /dev/mapper/sdg1
mdadm: Marking array /dev/md3 as 'clean'
mdadm: accepting backup with timestamp 1326496445 for array with timestamp
1326504140
mdadm: restoring critical section
mdadm: /dev/md3 assembled from 3 drives, 1 rebuilding and 2 spares - not
enough to start the array.
Looks like my 2 (Presumably ok) devices in slot 0&2 have been marked as spare?
As I have been unable to find a way to reverse this, I tried recreating the
array with the original raid-5 order & layout:
<https://raid.wiki.kernel.org/articles/r/a/i/RAID_Recovery_d376.html>
System log snippet from last array start prior to interrupted reshape:
md/raid:md3: device sdh1 operational as raid disk 4
md/raid:md3: device sdi1 operational as raid disk 2
md/raid:md3: device sdg1 operational as raid disk 0
md/raid:md3: device sdf1 operational as raid disk 1
md/raid:md3: device sde1 operational as raid disk 3
md/raid:md3: allocated 5320kB
md/raid:md3: raid level 5 active with 5 out of 5 devices, algorithm 2
md3: detected capacity change from 0 to 8001586462720
This gives the device order as sdg1, sdf1, sdi1, sde1, sdh1
I know the chunk size == 128K
The layout == left-symmetric
And the metadata == 1.2
So:
$>mdadm -S /dev/md3
$>mdadm --create --metadata=1.2 --level=5 --chunk=128K --layout=left-symmetric
--raid-devices=5 --assume-clean /dev/md3 /dev/mapper/sdg1 /dev/mapper/sdf1
/dev/mapper/sdi1 /dev/mapper/sde1 /dev/mapper/sdh1
The array starts as expected, but fsck can't find a superblock.
Testdisk finds a superblock:
Partition Start End Size in sectors
ext4 32 0 1 1953512351 1 4 15628098560
But the filesystem appears larger than the array:
Disk /dev/md3 - 8001 GB / 7452 GiB - CHS 1953512192 2 4
Examining logs shows:
Original raid5 > md3: detected capacity change from 0 to 8001586462720
New raid5 > md3: detected capacity change from 0 to 8001585938432
i.e. 524,288 smaller.
Comparing the output of $>mdadm -E [any disk in array]:
Failed raid-6:
/dev/mapper/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4
Array UUID : 06a7fdac:3b560176:9fc05905:34f97822
Name : Damnation:3
Creation Time : Sun Jun 12 14:38:45 2011
Raid Level : raid6
Raid Devices : 6
Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
Array Size : 15628098560 (7452.06 GiB 8001.59 GB)
Used Dev Size : 3907024640 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 4fcba3d0:3219c5d9:e847abe8:c8c93e03
Reshape pos'n : 232030208 (221.28 GiB 237.60 GB)
New Layout : left-symmetric
Update Time : Sat Jan 14 14:22:20 2012
Checksum : f214fb99 - correct
Events : 193807
Layout : left-symmetric-6
Chunk Size : 128K
Device Role : Active device 3
Array State : .A.AAA ('A' == active, '.' == missing)
New raid-5 (Should be the same as above but raid5 / left-symmetric?):
/dev/mapper/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : e6e1fa47:fa23ea0c:4cabf95d:1aa9c21d
Name : (none):3 (local to host (none))
Creation Time : Sun Jan 15 06:03:21 2012
Raid Level : raid5
Raid Devices : 5
Avail Dev Size : 3907025072 (1863.01 GiB 2000.40 GB)
Array Size : 15628097536 (7452.06 GiB 8001.59 GB)
Used Dev Size : 3907024384 (1863.01 GiB 2000.40 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 7d4fe0db:fbd649f7:d62d9df8:39dafe20
Update Time : Sun Jan 15 06:03:21 2012
Checksum : 7823d35e - correct
Events : 0
Layout : left-symmetric
Chunk Size : 128K
Device Role : Active device 3
Array State : AAAAA ('A' == active, '.' == missing)
Note Avail Dev Size is the same, but Array Size 15628097536 < 15628098560 by
1024 and Used Dev Size 3907024384 < 3907024640 by 256
I have tried many combinations of level, layout & stripe size (Though I am
certain I had it right the first time) to no avail.
Any idea where those 256b are going?
I see little hope of recovering the filesystem if the array size != original
layout.
Any help MUCH appreciated, I have scoured the WWW and been to #linux-raid on
freenode.
From what I can tell, recreating the array has worked for others in a similar
situation... But they all managed to create something _identical_ to the
original array.
Thank you for taking the time to read this.
Steve.
<steve_v@orcon.net.nz>
<vevets3@gmail.com>
^ permalink raw reply [flat|nested] only message in thread
only message in thread, other threads:[~2012-01-15 4:33 UTC | newest]
Thread overview: (only message) (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-01-15 4:33 [New Here] Need some help recovering failed Raid-5, Cant seem to re-create the same size array Steve Vialle
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.