* dm-crypt over raid6 unreadable after crash
@ 2011-07-06 16:12 Louis-David Mitterrand
2011-07-06 17:00 ` Phil Turmel
0 siblings, 1 reply; 13+ messages in thread
From: Louis-David Mitterrand @ 2011-07-06 16:12 UTC (permalink / raw)
To: linux-raid
Hi,
After a hardware crash I can no longer open a dm-crypt partition located
directly over a md-raid6 partition. I get this error:
root@grml ~ # cryptsetup isLuks /dev/md1
Device /dev/md1 is not a valid LUKS device
It seems the LUKS header has been shifted a few bytes forward, but looks
otherwise fine to specialists on the dm-crypt mailing list. Normally the
"LUKS" signature should be at 0x00000000
Is there some way that the md layer could have shifted its contents?
Is there a way to shift it back in place?
Thanks,
Here is a hexdum of /dev/md1 done with "hd /dev/md1 | head -n 40"
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00100000 4c 55 4b 53 ba be 00 01 61 65 73 00 00 00 00 00 |LUKS....aes.....|
00100010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100020 00 00 00 00 00 00 00 00 63 62 63 2d 65 73 73 69 |........cbc-essi|
00100030 76 3a 73 68 61 32 35 36 00 00 00 00 00 00 00 00 |v:sha256........|
00100040 00 00 00 00 00 00 00 00 73 68 61 31 00 00 00 00 |........sha1....|
00100050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100060 00 00 00 00 00 00 00 00 00 00 18 00 00 00 00 20 |............... |
00100070 d3 67 34 b9 7b 55 fe 0d e7 cc 5d df 6e ae f7 1f |.g4.{U....].n...|
00100080 96 78 43 c5 d2 55 58 4b a9 2a 73 8b 28 45 6b 8c |.xC..UXK.*s.(Ek.|
00100090 cd 26 45 61 6b f4 5a 58 6d 8e 1b 85 c2 60 4f 7c |.&Eak.ZXm....`O||
001000a0 00 39 48 89 00 00 6d dd 66 37 38 31 66 62 34 30 |.9H...m.f781fb40|
001000b0 2d 37 65 34 31 2d 34 30 63 62 2d 62 65 66 66 2d |-7e41-40cb-beff-|
001000c0 61 39 66 37 61 65 61 63 64 35 63 36 00 00 00 00 |a9f7aeacd5c6....|
001000d0 00 ac 71 f3 00 01 b8 3f f7 9d c2 ef e0 07 78 c3 |..q....?......x.|
001000e0 f6 90 73 57 39 be 8e 79 59 44 a1 23 35 11 6f 6a |..sW9..yYD.#5.oj|
001000f0 6c b0 36 fc 7d 9b 15 0f 00 00 00 08 00 00 0f a0 |l.6.}...........|
00100100 00 00 de ad 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100120 00 00 00 00 00 00 00 00 00 00 01 08 00 00 0f a0 |................|
00100130 00 00 de ad 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100150 00 00 00 00 00 00 00 00 00 00 02 08 00 00 0f a0 |................|
00100160 00 00 de ad 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100180 00 00 00 00 00 00 00 00 00 00 03 08 00 00 0f a0 |................|
00100190 00 00 de ad 00 00 00 00 00 00 00 00 00 00 00 00 |................|
001001a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
001001b0 00 00 00 00 00 00 00 00 00 00 04 08 00 00 0f a0 |................|
001001c0 00 00 de ad 00 00 00 00 00 00 00 00 00 00 00 00 |................|
001001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
001001e0 00 00 00 00 00 00 00 00 00 00 05 08 00 00 0f a0 |................|
001001f0 00 00 de ad 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100210 00 00 00 00 00 00 00 00 00 00 06 08 00 00 0f a0 |................|
00100220 00 00 de ad 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100230 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00100240 00 00 00 00 00 00 00 00 00 00 07 08 00 00 0f a0 |................|
00100250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: dm-crypt over raid6 unreadable after crash 2011-07-06 16:12 dm-crypt over raid6 unreadable after crash Louis-David Mitterrand @ 2011-07-06 17:00 ` Phil Turmel 2011-07-07 9:05 ` Louis-David Mitterrand 0 siblings, 1 reply; 13+ messages in thread From: Phil Turmel @ 2011-07-06 17:00 UTC (permalink / raw) To: Louis-David Mitterrand; +Cc: linux-raid Hi Louis-David, On 07/06/2011 12:12 PM, Louis-David Mitterrand wrote: > Hi, > > After a hardware crash I can no longer open a dm-crypt partition located > directly over a md-raid6 partition. I get this error: > > root@grml ~ # cryptsetup isLuks /dev/md1 > Device /dev/md1 is not a valid LUKS device > > It seems the LUKS header has been shifted a few bytes forward, but looks > otherwise fine to specialists on the dm-crypt mailing list. Normally the > "LUKS" signature should be at 0x00000000 > > Is there some way that the md layer could have shifted its contents? > > Is there a way to shift it back in place? > > Thanks, > > > Here is a hexdum of /dev/md1 done with "hd /dev/md1 | head -n 40" > > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > * > 00100000 4c 55 4b 53 ba be 00 01 61 65 73 00 00 00 00 00 |LUKS....aes.....| > 00100010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > 00100020 00 00 00 00 00 00 00 00 63 62 63 2d 65 73 73 69 |........cbc-essi| > 00100030 76 3a 73 68 61 32 35 36 00 00 00 00 00 00 00 00 |v:sha256........| [...] The offset is precisely 1MB. This is the default data offset for metadata types 1.1 and 1.2 (nowadays). Metadata types 0.90 and 1.0 have a zero offset (the metadata is at the end.) You don't say what your recovery efforts were, but I'd guess you did a "mdadm --create" somewhere in there, and didn't match the original parameters. Or you used an older version of mdadm than was used originally, and therefore got different defaults. Another possibility is that the original array was set up on a 1MB aligned partition, and the array is now using the whole device. This can happen with v0.90 metadata. If so, the original partition table is obviously zeroed out now. Please share more information about what you've done so far. Also show us the output of "mdadm -D /dev/md1" and then "mdadm -E /dev/xxx" for each of its components. The output of "lsdrv"[1] would also be useful for visualizing your setup. Regards, Phil [1] http://github.com/pturmel/lsdrv ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: dm-crypt over raid6 unreadable after crash 2011-07-06 17:00 ` Phil Turmel @ 2011-07-07 9:05 ` Louis-David Mitterrand 2011-07-07 12:41 ` Phil Turmel 0 siblings, 1 reply; 13+ messages in thread From: Louis-David Mitterrand @ 2011-07-07 9:05 UTC (permalink / raw) To: linux-raid On Wed, Jul 06, 2011 at 01:00:43PM -0400, Phil Turmel wrote: > > After a hardware crash I can no longer open a dm-crypt partition located > > directly over a md-raid6 partition. I get this error: > > > > root@grml ~ # cryptsetup isLuks /dev/md1 > > Device /dev/md1 is not a valid LUKS device > > > > It seems the LUKS header has been shifted a few bytes forward, but looks > > otherwise fine to specialists on the dm-crypt mailing list. Normally the > > "LUKS" signature should be at 0x00000000 > > > > Is there some way that the md layer could have shifted its contents? > > > > Here is a hexdum of /dev/md1 done with "hd /dev/md1 | head -n 40" > > > > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > > * > > 00100000 4c 55 4b 53 ba be 00 01 61 65 73 00 00 00 00 00 |LUKS....aes.....| > > 00100010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > > 00100020 00 00 00 00 00 00 00 00 63 62 63 2d 65 73 73 69 |........cbc-essi| > > 00100030 76 3a 73 68 61 32 35 36 00 00 00 00 00 00 00 00 |v:sha256........| Hi Phil, > The offset is precisely 1MB. This is the default data offset for > metadata types 1.1 and 1.2 (nowadays). Metadata types 0.90 and 1.0 > have a zero offset (the metadata is at the end.) > > You don't say what your recovery efforts were, but I'd guess you did a > "mdadm --create" somewhere in there, and didn't match the original > parameters. Or you used an older version of mdadm than was used > originally, and therefore got different defaults. No I did a mdadm-startall with a grml livecd. > Another possibility is that the original array was set up on a 1MB > aligned partition, and the array is now using the whole device. This > can happen with v0.90 metadata. If so, the original partition table > is obviously zeroed out now. > > Please share more information about what you've done so far. Also Nothing appart from assembling the array and failing to decrypt it with cryptsetup. > show us the output of "mdadm -D /dev/md1" /dev/md1: Version : 1.2 Creation Time : Wed Oct 20 21:40:40 2010 Raid Level : raid6 Array Size : 841863168 (802.86 GiB 862.07 GB) Used Dev Size : 140310528 (133.81 GiB 143.68 GB) Raid Devices : 8 Total Devices : 8 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Thu Jul 7 09:44:49 2011 State : active Active Devices : 8 Working Devices : 8 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : grml:1 (local to host grml) UUID : 1434a46a:f2b751cd:8604803c:b545de8c Events : 8292 Number Major Minor RaidDevice State 0 8 130 0 active sync /dev/sdi2 1 8 50 1 active sync /dev/sdd2 2 8 34 2 active sync /dev/sdc2 3 8 82 3 active sync /dev/sdf2 4 8 66 4 active sync /dev/sde2 5 8 146 5 active sync /dev/sdj2 8 8 114 6 active sync /dev/sdh2 7 8 98 7 active sync /dev/sdg2 > and then "mdadm -E /dev/xxx" for each of its components. /dev/sdc2: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 1434a46a:f2b751cd:8604803c:b545de8c Name : grml:1 (local to host grml) Creation Time : Wed Oct 20 21:40:40 2010 Raid Level : raid6 Raid Devices : 8 Avail Dev Size : 280621372 (133.81 GiB 143.68 GB) Array Size : 1683726336 (802.86 GiB 862.07 GB) Used Dev Size : 280621056 (133.81 GiB 143.68 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : 0a10d6c3:8a6f1948:f1a546a4:32f10094 Internal Bitmap : 2 sectors from superblock Update Time : Thu Jul 7 11:00:42 2011 Checksum : 16c1099b - correct Events : 8292 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 2 Array State : AAAAAAAA ('A' == active, '.' == missing) /dev/sdd2: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 1434a46a:f2b751cd:8604803c:b545de8c Name : grml:1 (local to host grml) Creation Time : Wed Oct 20 21:40:40 2010 Raid Level : raid6 Raid Devices : 8 Avail Dev Size : 280621372 (133.81 GiB 143.68 GB) Array Size : 1683726336 (802.86 GiB 862.07 GB) Used Dev Size : 280621056 (133.81 GiB 143.68 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : 5f8c46cb:614354cf:dd91f7c2:f1260b2e Internal Bitmap : 2 sectors from superblock Update Time : Thu Jul 7 11:00:42 2011 Checksum : 5e277b71 - correct Events : 8292 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 1 Array State : AAAAAAAA ('A' == active, '.' == missing) /dev/sde2: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 1434a46a:f2b751cd:8604803c:b545de8c Name : grml:1 (local to host grml) Creation Time : Wed Oct 20 21:40:40 2010 Raid Level : raid6 Raid Devices : 8 Avail Dev Size : 280621372 (133.81 GiB 143.68 GB) Array Size : 1683726336 (802.86 GiB 862.07 GB) Used Dev Size : 280621056 (133.81 GiB 143.68 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : ab27b114:ea95aa0a:9adb310b:c456ee56 Internal Bitmap : 2 sectors from superblock Update Time : Thu Jul 7 11:00:42 2011 Checksum : 5405e2d7 - correct Events : 8292 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 4 Array State : AAAAAAAA ('A' == active, '.' == missing) /dev/sdf2: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 1434a46a:f2b751cd:8604803c:b545de8c Name : grml:1 (local to host grml) Creation Time : Wed Oct 20 21:40:40 2010 Raid Level : raid6 Raid Devices : 8 Avail Dev Size : 280621372 (133.81 GiB 143.68 GB) Array Size : 1683726336 (802.86 GiB 862.07 GB) Used Dev Size : 280621056 (133.81 GiB 143.68 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : 31bc8c85:7b754501:ea0b713e:2714810a Internal Bitmap : 2 sectors from superblock Update Time : Thu Jul 7 11:00:42 2011 Checksum : a24e44a0 - correct Events : 8292 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 3 Array State : AAAAAAAA ('A' == active, '.' == missing) /dev/sdg2: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 1434a46a:f2b751cd:8604803c:b545de8c Name : grml:1 (local to host grml) Creation Time : Wed Oct 20 21:40:40 2010 Raid Level : raid6 Raid Devices : 8 Avail Dev Size : 280621372 (133.81 GiB 143.68 GB) Array Size : 1683726336 (802.86 GiB 862.07 GB) Used Dev Size : 280621056 (133.81 GiB 143.68 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : 4db23afd:d4422390:e39d701e:7223cc9e Internal Bitmap : 2 sectors from superblock Update Time : Thu Jul 7 11:00:42 2011 Checksum : 1d24a95f - correct Events : 8292 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 7 Array State : AAAAAAAA ('A' == active, '.' == missing) /dev/sdh2: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 1434a46a:f2b751cd:8604803c:b545de8c Name : grml:1 (local to host grml) Creation Time : Wed Oct 20 21:40:40 2010 Raid Level : raid6 Raid Devices : 8 Avail Dev Size : 280621372 (133.81 GiB 143.68 GB) Array Size : 1683726336 (802.86 GiB 862.07 GB) Used Dev Size : 280621056 (133.81 GiB 143.68 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : 1f220bf2:1c86fc2b:0e99f2d2:8283497c Internal Bitmap : 2 sectors from superblock Update Time : Thu Jul 7 11:00:42 2011 Checksum : 3fcdb7b5 - correct Events : 8292 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 6 Array State : AAAAAAAA ('A' == active, '.' == missing) /dev/sdi2: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 1434a46a:f2b751cd:8604803c:b545de8c Name : grml:1 (local to host grml) Creation Time : Wed Oct 20 21:40:40 2010 Raid Level : raid6 Raid Devices : 8 Avail Dev Size : 280621372 (133.81 GiB 143.68 GB) Array Size : 1683726336 (802.86 GiB 862.07 GB) Used Dev Size : 280621056 (133.81 GiB 143.68 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : 6acb87a4:3ac53237:1f5fff58:3611a0b0 Internal Bitmap : 2 sectors from superblock Update Time : Thu Jul 7 11:00:42 2011 Checksum : b7e3f3da - correct Events : 8292 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 0 Array State : AAAAAAAA ('A' == active, '.' == missing) /dev/sdj2: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 1434a46a:f2b751cd:8604803c:b545de8c Name : grml:1 (local to host grml) Creation Time : Wed Oct 20 21:40:40 2010 Raid Level : raid6 Raid Devices : 8 Avail Dev Size : 280621372 (133.81 GiB 143.68 GB) Array Size : 1683726336 (802.86 GiB 862.07 GB) Used Dev Size : 280621056 (133.81 GiB 143.68 GB) Data Offset : 2048 sectors Super Offset : 8 sectors State : clean Device UUID : 1515f5a7:b0c78638:8ca1d918:f1fa47d7 Internal Bitmap : 2 sectors from superblock Update Time : Thu Jul 7 11:00:42 2011 Checksum : a3276c28 - correct Events : 8292 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 5 Array State : AAAAAAAA ('A' == active, '.' == missing) > The output of "lsdrv"[1] would also be useful for visualizing your setup. PCI [ata_piix] 00:1f.2 IDE interface: Intel Corporation 82801IB (ICH9) 2 port SATA IDE Controller (rev 02) ├─scsi 0:0:0:0 HL-DT-ST DVD+-RW GH50N {K1LA7D41849} │ └─sr0: [11:0] Partitioned (dos) 224.00m 'grml64-medium_2011.05' │ └─Mounted as /dev/sr0 @ /live/image └─scsi 1:x:x:x [Empty] PCI [mpt2sas] 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02) ├─scsi 2:0:0:0 ATA WDC WD1002FAEX-0 {WD-WCATR1851552} │ └─sdc: [8:32] Partitioned (dos) 931.51g │ ├─sdc1: [8:33] MD raid1 (2/8) 250.98m md0 clean in_sync {2871f814-ceb7-6a88-d8b7-8f6599226e41} │ │ └─md0: [9:0] Partitioned (dos) 250.88m {d1d876e9-6905-4940-bf55-7cdb4b64484f} │ ├─sdc2: [8:34] MD raid6 (2/8) 133.81g md1 clean in_sync 'grml:1' {1434a46a-f2b7-51cd-8604-803cb545de8c} │ │ └─md1: [9:1] Empty/Unknown 802.86g │ └─sdc3: [8:35] MD raid6 (0/8) 797.36g md2 clean in_sync 'zenon:2' {5c037ba3-ca4b-f7b9-eb8f-b01608e1fd3b} │ └─md2: [9:2] (crypto_LUKS) 4.67t {1d30a244-9d40-48e8-925a-1d6c93a45474} │ └─dm-0: [253:0] (xfs) 4.67t {3cad63a0-a586-43e0-bf89-5be9066c884f} │ └─Mounted as /dev/mapper/cmd2 @ /backup ├─scsi 2:0:1:0 ATA WDC WD1002FAEX-0 {WD-WCATR2968402} │ └─sdd: [8:48] Partitioned (dos) 931.51g │ ├─sdd1: [8:49] MD raid1 (3/8) 250.98m md0 clean in_sync {2871f814-ceb7-6a88-d8b7-8f6599226e41} │ ├─sdd2: [8:50] MD raid6 (1/8) 133.81g md1 clean in_sync 'grml:1' {1434a46a-f2b7-51cd-8604-803cb545de8c} │ └─sdd3: [8:51] MD raid6 (5/8) 797.36g md2 clean in_sync 'zenon:2' {5c037ba3-ca4b-f7b9-eb8f-b01608e1fd3b} ├─scsi 2:0:2:0 ATA WDC WD1002FAEX-0 {WD-WCATR1851573} │ └─sde: [8:64] Partitioned (dos) 931.51g │ ├─sde1: [8:65] MD raid1 (7/8) 250.98m md0 clean in_sync {2871f814-ceb7-6a88-d8b7-8f6599226e41} │ ├─sde2: [8:66] MD raid6 (4/8) 133.81g md1 clean in_sync 'grml:1' {1434a46a-f2b7-51cd-8604-803cb545de8c} │ └─sde3: [8:67] MD raid6 (6/8) 797.36g md2 clean in_sync 'zenon:2' {5c037ba3-ca4b-f7b9-eb8f-b01608e1fd3b} ├─scsi 2:0:3:0 ATA WDC WD1002FAEX-0 {WD-WCATR3005506} │ └─sdf: [8:80] Partitioned (dos) 931.51g │ ├─sdf1: [8:81] MD raid1 (0/8) 250.98m md0 clean in_sync {2871f814-ceb7-6a88-d8b7-8f6599226e41} │ ├─sdf2: [8:82] MD raid6 (3/8) 133.81g md1 clean in_sync 'grml:1' {1434a46a-f2b7-51cd-8604-803cb545de8c} │ └─sdf3: [8:83] MD raid6 (4/8) 797.36g md2 clean in_sync 'zenon:2' {5c037ba3-ca4b-f7b9-eb8f-b01608e1fd3b} ├─scsi 2:0:4:0 ATA WDC WD1002FAEX-0 {WD-WCATR3007070} │ └─sdg: [8:96] Partitioned (dos) 931.51g │ ├─sdg1: [8:97] MD raid1 (6/8) 250.98m md0 clean in_sync {2871f814-ceb7-6a88-d8b7-8f6599226e41} │ ├─sdg2: [8:98] MD raid6 (7/8) 133.81g md1 clean in_sync 'grml:1' {1434a46a-f2b7-51cd-8604-803cb545de8c} │ └─sdg3: [8:99] MD raid6 (3/8) 797.36g md2 clean in_sync 'zenon:2' {5c037ba3-ca4b-f7b9-eb8f-b01608e1fd3b} ├─scsi 2:0:5:0 ATA WDC WD1002FAEX-0 {WD-WCATR3004862} │ └─sdh: [8:112] Partitioned (dos) 931.51g │ ├─sdh1: [8:113] MD raid1 (4/8) 250.98m md0 clean in_sync {2871f814-ceb7-6a88-d8b7-8f6599226e41} │ ├─sdh2: [8:114] MD raid6 (6/8) 133.81g md1 clean in_sync 'grml:1' {1434a46a-f2b7-51cd-8604-803cb545de8c} │ └─sdh3: [8:115] MD raid6 (1/8) 797.36g md2 clean in_sync 'zenon:2' {5c037ba3-ca4b-f7b9-eb8f-b01608e1fd3b} ├─scsi 2:0:6:0 ATA WDC WD1002FAEX-0 {WD-WCATR2969087} │ └─sdi: [8:128] Partitioned (dos) 931.51g │ ├─sdi1: [8:129] MD raid1 (1/8) 250.98m md0 clean in_sync {2871f814-ceb7-6a88-d8b7-8f6599226e41} │ ├─sdi2: [8:130] MD raid6 (0/8) 133.81g md1 clean in_sync 'grml:1' {1434a46a-f2b7-51cd-8604-803cb545de8c} │ └─sdi3: [8:131] MD raid6 (7/8) 797.36g md2 clean in_sync 'zenon:2' {5c037ba3-ca4b-f7b9-eb8f-b01608e1fd3b} ├─scsi 2:0:7:0 ATA WDC WD1002FAEX-0 {WD-WCATR2984361} │ └─sdj: [8:144] Partitioned (dos) 931.51g │ ├─sdj1: [8:145] MD raid1 (5/8) 250.98m md0 clean in_sync {2871f814-ceb7-6a88-d8b7-8f6599226e41} │ ├─sdj2: [8:146] MD raid6 (5/8) 133.81g md1 clean in_sync 'grml:1' {1434a46a-f2b7-51cd-8604-803cb545de8c} │ └─sdj3: [8:147] MD raid6 (2/8) 797.36g md2 clean in_sync 'zenon:2' {5c037ba3-ca4b-f7b9-eb8f-b01608e1fd3b} └─scsi 2:x:x:x [Empty] USB [usb-storage] Bus 002 Device 004: ID 0624:0249 Avocent Corp. {20080519} └─scsi 3:0:0:0 iDRAC LCDRIVE └─sda: [8:0] Empty/Unknown 0.00k USB [usb-storage] Bus 002 Device 004: ID 0624:0249 Avocent Corp. {20080519} ├─scsi 4:0:0:0 iDRAC Virtual CD │ └─sr1: [11:1] Empty/Unknown 1.00g └─scsi 4:0:0:1 iDRAC Virtual Floppy └─sdb: [8:16] Empty/Unknown 0.00k Other Block Devices └─loop0: [7:0] (squashfs) 199.44m └─Mounted as /dev/loop0 @ /grml64-medium.squashfs > Regards, Thanks, -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: dm-crypt over raid6 unreadable after crash 2011-07-07 9:05 ` Louis-David Mitterrand @ 2011-07-07 12:41 ` Phil Turmel 2011-07-07 13:09 ` Louis-David Mitterrand 0 siblings, 1 reply; 13+ messages in thread From: Phil Turmel @ 2011-07-07 12:41 UTC (permalink / raw) To: Louis-David Mitterrand; +Cc: linux-raid On 07/07/2011 05:05 AM, Louis-David Mitterrand wrote: > On Wed, Jul 06, 2011 at 01:00:43PM -0400, Phil Turmel wrote: >>> After a hardware crash I can no longer open a dm-crypt partition located >>> directly over a md-raid6 partition. I get this error: >>> >>> root@grml ~ # cryptsetup isLuks /dev/md1 >>> Device /dev/md1 is not a valid LUKS device >>> >>> It seems the LUKS header has been shifted a few bytes forward, but looks >>> otherwise fine to specialists on the dm-crypt mailing list. Normally the >>> "LUKS" signature should be at 0x00000000 >>> >>> Is there some way that the md layer could have shifted its contents? >>> >>> Here is a hexdum of /dev/md1 done with "hd /dev/md1 | head -n 40" >>> >>> 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| >>> * >>> 00100000 4c 55 4b 53 ba be 00 01 61 65 73 00 00 00 00 00 |LUKS....aes.....| >>> 00100010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| >>> 00100020 00 00 00 00 00 00 00 00 63 62 63 2d 65 73 73 69 |........cbc-essi| >>> 00100030 76 3a 73 68 61 32 35 36 00 00 00 00 00 00 00 00 |v:sha256........| > > Hi Phil, > >> The offset is precisely 1MB. This is the default data offset for >> metadata types 1.1 and 1.2 (nowadays). Metadata types 0.90 and 1.0 >> have a zero offset (the metadata is at the end.) >> >> You don't say what your recovery efforts were, but I'd guess you did a >> "mdadm --create" somewhere in there, and didn't match the original >> parameters. Or you used an older version of mdadm than was used >> originally, and therefore got different defaults. > > No I did a mdadm-startall with a grml livecd. Very interesting. >> Another possibility is that the original array was set up on a 1MB >> aligned partition, and the array is now using the whole device. This >> can happen with v0.90 metadata. If so, the original partition table >> is obviously zeroed out now. >> >> Please share more information about what you've done so far. Also > > Nothing appart from assembling the array and failing to decrypt it with > cryptsetup. OK. And it's a v1.2 superblock, so mdadm very likely got it right. >> show us the output of "mdadm -D /dev/md1" > > /dev/md1: > Version : 1.2 > Creation Time : Wed Oct 20 21:40:40 2010 > Raid Level : raid6 > Array Size : 841863168 (802.86 GiB 862.07 GB) > Used Dev Size : 140310528 (133.81 GiB 143.68 GB) > Raid Devices : 8 > Total Devices : 8 > Persistence : Superblock is persistent > > Intent Bitmap : Internal > > Update Time : Thu Jul 7 09:44:49 2011 > State : active > Active Devices : 8 > Working Devices : 8 > Failed Devices : 0 > Spare Devices : 0 > > Layout : left-symmetric > Chunk Size : 512K > > Name : grml:1 (local to host grml) > UUID : 1434a46a:f2b751cd:8604803c:b545de8c > Events : 8292 > > Number Major Minor RaidDevice State > 0 8 130 0 active sync /dev/sdi2 > 1 8 50 1 active sync /dev/sdd2 > 2 8 34 2 active sync /dev/sdc2 > 3 8 82 3 active sync /dev/sdf2 > 4 8 66 4 active sync /dev/sde2 > 5 8 146 5 active sync /dev/sdj2 > 8 8 114 6 active sync /dev/sdh2 > 7 8 98 7 active sync /dev/sdg2 > >> and then "mdadm -E /dev/xxx" for each of its components. > > > /dev/sdc2: > Magic : a92b4efc > Version : 1.2 > Feature Map : 0x1 > Array UUID : 1434a46a:f2b751cd:8604803c:b545de8c > Name : grml:1 (local to host grml) > Creation Time : Wed Oct 20 21:40:40 2010 > Raid Level : raid6 > Raid Devices : 8 > > Avail Dev Size : 280621372 (133.81 GiB 143.68 GB) > Array Size : 1683726336 (802.86 GiB 862.07 GB) > Used Dev Size : 280621056 (133.81 GiB 143.68 GB) > Data Offset : 2048 sectors > Super Offset : 8 sectors > State : clean > Device UUID : 0a10d6c3:8a6f1948:f1a546a4:32f10094 > > Internal Bitmap : 2 sectors from superblock > Update Time : Thu Jul 7 11:00:42 2011 > Checksum : 16c1099b - correct > Events : 8292 > > Layout : left-symmetric > Chunk Size : 512K > > Device Role : Active device 2 > Array State : AAAAAAAA ('A' == active, '.' == missing) [...] No oddities in the above. Both of my speculations are wrong. >> The output of "lsdrv"[1] would also be useful for visualizing your setup. > > PCI [ata_piix] 00:1f.2 IDE interface: Intel Corporation 82801IB (ICH9) 2 port SATA IDE Controller (rev 02) > ├─scsi 0:0:0:0 HL-DT-ST DVD+-RW GH50N {K1LA7D41849} > │ └─sr0: [11:0] Partitioned (dos) 224.00m 'grml64-medium_2011.05' > │ └─Mounted as /dev/sr0 @ /live/image > └─scsi 1:x:x:x [Empty] > PCI [mpt2sas] 02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02) > ├─scsi 2:0:0:0 ATA WDC WD1002FAEX-0 {WD-WCATR1851552} > │ └─sdc: [8:32] Partitioned (dos) 931.51g > │ ├─sdc1: [8:33] MD raid1 (2/8) 250.98m md0 clean in_sync {2871f814-ceb7-6a88-d8b7-8f6599226e41} > │ │ └─md0: [9:0] Partitioned (dos) 250.88m {d1d876e9-6905-4940-bf55-7cdb4b64484f} > │ ├─sdc2: [8:34] MD raid6 (2/8) 133.81g md1 clean in_sync 'grml:1' {1434a46a-f2b7-51cd-8604-803cb545de8c} > │ │ └─md1: [9:1] Empty/Unknown 802.86g > │ └─sdc3: [8:35] MD raid6 (0/8) 797.36g md2 clean in_sync 'zenon:2' {5c037ba3-ca4b-f7b9-eb8f-b01608e1fd3b} > │ └─md2: [9:2] (crypto_LUKS) 4.67t {1d30a244-9d40-48e8-925a-1d6c93a45474} > │ └─dm-0: [253:0] (xfs) 4.67t {3cad63a0-a586-43e0-bf89-5be9066c884f} > │ └─Mounted as /dev/mapper/cmd2 @ /backup So, cryptsetup saw and properly handled /dev/md2. [...] Well. /dev/md1 is assembled correctly as far as I can tell. That make me wonder what else might be in play. First, it would be good to know if the luks data is truly intact at the 1MB offset. As a test, please add a linear device mapper layer that skips the 1MB. Like so: echo 0 1683724288 linear /dev/md1 2048 | dmsetup create mdtest Depending on grml's udev setup, this may prompt you to unlock /dev/mapper/mdtest. Otherwise, use cryptsetup to test it, and then unlock it. Do *NOT* mount yet. Run "fsck -n" to see if it is intact. I also wonder if the md device itself was partitioned, maybe with EFI GPT, and the grml liveCD doesn't support it? (Long shot.) Please show "zcat /proc/config.gz |grep PART" I'm running out of ideas. Phil -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: dm-crypt over raid6 unreadable after crash 2011-07-07 12:41 ` Phil Turmel @ 2011-07-07 13:09 ` Louis-David Mitterrand 2011-07-07 13:32 ` Phil Turmel 0 siblings, 1 reply; 13+ messages in thread From: Louis-David Mitterrand @ 2011-07-07 13:09 UTC (permalink / raw) To: linux-raid On Thu, Jul 07, 2011 at 08:41:36AM -0400, Phil Turmel wrote: > > So, cryptsetup saw and properly handled /dev/md2. > > [...] > > Well. /dev/md1 is assembled correctly as far as I can tell. That > make me wonder what else might be in play. First, it would be good to > know if the luks data is truly intact at the 1MB offset. As a test, > please add a linear device mapper layer that skips the 1MB. Like so: > > echo 0 1683724288 linear /dev/md1 2048 | dmsetup create mdtest That worked fine. I then successfully unlocked mdtest with: cryptsetup luksOpen /dev/mapper/mdtest cmd1 > Depending on grml's udev setup, this may prompt you to unlock > /dev/mapper/mdtest. Otherwise, use cryptsetup to test it, and then > unlock it. Do *NOT* mount yet. Run "fsck -n" to see if it is intact. root@grml /dev/mapper # xfs_check /dev/mapper/cmd1 xfs_check: /dev/mapper/cmd1 is not a valid XFS filesystem (unexpected SB magic number 0xd0f1b462) xfs_check: size check failed xfs_check: WARNING - filesystem uses v1 dirs,limited functionality provided. xfs_check: read failed: Invalid argument xfs_check: data size check failed cache_node_purge: refcount was 1, not zero (node=0x1835a30) xfs_check: cannot read root inode (22) cache_node_purge: refcount was 1, not zero (node=0x1835c80) xfs_check: cannot read realtime bitmap inode (22) xfs_check: size check failed xfs_check: WARNING - filesystem uses v1 dirs,limited functionality provided. xfs_check: read failed: Invalid argument xfs_check: data size check failed bad superblock magic number d0f1b462, giving up > I also wonder if the md device itself was partitioned, maybe with EFI > GPT, and the grml liveCD doesn't support it? (Long shot.) Please > show "zcat /proc/config.gz |grep PART" root@grml /dev/mapper # zcat /proc/config.gz |grep PART CONFIG_PM_STD_PARTITION="" CONFIG_MTD_PARTITIONS=y CONFIG_MTD_REDBOOT_PARTS=m # CONFIG_MTD_REDBOOT_PARTS_UNALLOCATED is not set # CONFIG_MTD_REDBOOT_PARTS_READONLY is not set CONFIG_MTD_AR7_PARTS=m CONFIG_PARTITION_ADVANCED=y CONFIG_ACORN_PARTITION=y # CONFIG_ACORN_PARTITION_CUMANA is not set # CONFIG_ACORN_PARTITION_EESOX is not set CONFIG_ACORN_PARTITION_ICS=y # CONFIG_ACORN_PARTITION_ADFS is not set # CONFIG_ACORN_PARTITION_POWERTEC is not set # CONFIG_ACORN_PARTITION_RISCIX is not set CONFIG_OSF_PARTITION=y CONFIG_AMIGA_PARTITION=y CONFIG_ATARI_PARTITION=y CONFIG_MAC_PARTITION=y CONFIG_MSDOS_PARTITION=y CONFIG_MINIX_SUBPARTITION=y CONFIG_SOLARIS_X86_PARTITION=y CONFIG_LDM_PARTITION=y CONFIG_SGI_PARTITION=y CONFIG_ULTRIX_PARTITION=y CONFIG_SUN_PARTITION=y CONFIG_KARMA_PARTITION=y CONFIG_EFI_PARTITION=y CONFIG_SYSV68_PARTITION=y > I'm running out of ideas. Well thanks a lot for trying at least. At least now I understand how 'linear' can be used. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: dm-crypt over raid6 unreadable after crash 2011-07-07 13:09 ` Louis-David Mitterrand @ 2011-07-07 13:32 ` Phil Turmel 2011-07-08 5:40 ` Luca Berra 0 siblings, 1 reply; 13+ messages in thread From: Phil Turmel @ 2011-07-07 13:32 UTC (permalink / raw) To: linux-raid On 07/07/2011 09:09 AM, Louis-David Mitterrand wrote: > On Thu, Jul 07, 2011 at 08:41:36AM -0400, Phil Turmel wrote: >> >> So, cryptsetup saw and properly handled /dev/md2. >> >> [...] >> >> Well. /dev/md1 is assembled correctly as far as I can tell. That >> make me wonder what else might be in play. First, it would be good to >> know if the luks data is truly intact at the 1MB offset. As a test, >> please add a linear device mapper layer that skips the 1MB. Like so: >> >> echo 0 1683724288 linear /dev/md1 2048 | dmsetup create mdtest > > That worked fine. > > I then successfully unlocked mdtest with: > > cryptsetup luksOpen /dev/mapper/mdtest cmd1 > >> Depending on grml's udev setup, this may prompt you to unlock >> /dev/mapper/mdtest. Otherwise, use cryptsetup to test it, and then >> unlock it. Do *NOT* mount yet. Run "fsck -n" to see if it is intact. > > root@grml /dev/mapper # xfs_check /dev/mapper/cmd1 > xfs_check: /dev/mapper/cmd1 is not a valid XFS filesystem (unexpected SB magic number 0xd0f1b462) > xfs_check: size check failed This is important. When I computed the sector count for the linear mapping, I just took 2048 off the end. You may want to select a sector count that aligns the endpoint. > xfs_check: WARNING - filesystem uses v1 dirs,limited functionality provided. > xfs_check: read failed: Invalid argument > xfs_check: data size check failed > cache_node_purge: refcount was 1, not zero (node=0x1835a30) > xfs_check: cannot read root inode (22) > cache_node_purge: refcount was 1, not zero (node=0x1835c80) > xfs_check: cannot read realtime bitmap inode (22) > xfs_check: size check failed > xfs_check: WARNING - filesystem uses v1 dirs,limited functionality provided. > xfs_check: read failed: Invalid argument > xfs_check: data size check failed > bad superblock magic number d0f1b462, giving up If the md array is assembled with devices out of order, the initialization vectors for all the sectors are likely to be wrong, with no way to examine the contents to try to work out the device order. Of course, I don't know of any way mdadm can screw up the device order with v1+ metadata. >> I also wonder if the md device itself was partitioned, maybe with EFI >> GPT, and the grml liveCD doesn't support it? (Long shot.) Please >> show "zcat /proc/config.gz |grep PART" > > root@grml /dev/mapper # zcat /proc/config.gz |grep PART > CONFIG_PM_STD_PARTITION="" > CONFIG_MTD_PARTITIONS=y > CONFIG_MTD_REDBOOT_PARTS=m > # CONFIG_MTD_REDBOOT_PARTS_UNALLOCATED is not set > # CONFIG_MTD_REDBOOT_PARTS_READONLY is not set > CONFIG_MTD_AR7_PARTS=m > CONFIG_PARTITION_ADVANCED=y > CONFIG_ACORN_PARTITION=y > # CONFIG_ACORN_PARTITION_CUMANA is not set > # CONFIG_ACORN_PARTITION_EESOX is not set > CONFIG_ACORN_PARTITION_ICS=y > # CONFIG_ACORN_PARTITION_ADFS is not set > # CONFIG_ACORN_PARTITION_POWERTEC is not set > # CONFIG_ACORN_PARTITION_RISCIX is not set > CONFIG_OSF_PARTITION=y > CONFIG_AMIGA_PARTITION=y > CONFIG_ATARI_PARTITION=y > CONFIG_MAC_PARTITION=y > CONFIG_MSDOS_PARTITION=y > CONFIG_MINIX_SUBPARTITION=y > CONFIG_SOLARIS_X86_PARTITION=y > CONFIG_LDM_PARTITION=y > CONFIG_SGI_PARTITION=y > CONFIG_ULTRIX_PARTITION=y > CONFIG_SUN_PARTITION=y > CONFIG_KARMA_PARTITION=y > CONFIG_EFI_PARTITION=y ^^^ So that's not it. > CONFIG_SYSV68_PARTITION=y > >> I'm running out of ideas. > > Well thanks a lot for trying at least. At least now I understand how > 'linear' can be used. Sorry I couldn't help more. Phil ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: dm-crypt over raid6 unreadable after crash 2011-07-07 13:32 ` Phil Turmel @ 2011-07-08 5:40 ` Luca Berra 2011-07-10 12:03 ` Louis-David Mitterrand 0 siblings, 1 reply; 13+ messages in thread From: Luca Berra @ 2011-07-08 5:40 UTC (permalink / raw) To: linux-raid On Thu, Jul 07, 2011 at 09:32:38AM -0400, Phil Turmel wrote: >On 07/07/2011 09:09 AM, Louis-David Mitterrand wrote: >> On Thu, Jul 07, 2011 at 08:41:36AM -0400, Phil Turmel wrote: ... >>> echo 0 1683724288 linear /dev/md1 2048 | dmsetup create mdtest >> .... >> >> root@grml /dev/mapper # xfs_check /dev/mapper/cmd1 >> xfs_check: /dev/mapper/cmd1 is not a valid XFS filesystem (unexpected SB magic number 0xd0f1b462) >> xfs_check: size check failed > >This is important. When I computed the sector count for the linear mapping, I just took 2048 off the end. You may want to select a sector count that aligns the endpoint. but the xfs sb should be at sector 0 >If the md array is assembled with devices out of order, the initialization vectors for all the sectors are likely to be wrong, with no way to examine the contents to try to work out the device order. Of course, I don't know of any way mdadm can screw up the device order with v1+ metadata. > >>> I also wonder if the md device itself was partitioned, maybe with EFI >>> GPT, and the grml liveCD doesn't support it? (Long shot.) Please >>> show "zcat /proc/config.gz |grep PART" if the md itself were partitioned i would expect to see a partition table somewhere between the start and the liks header, but from the posted od it is 1M of zeroes. also from the lsdrv i see another array named zenon, while this is named grml (i suppose the hostname of your server is the former), this makes me wonder... i do not have a debian, but from what i found of the man-page the script should be harmless, but. the luks header starts at 1M, which is a multiple of the 512K chunk size, which could be a symptom of mis-ordered devices. did you keep any log when you ran the mdadm-startall? did md1 resync? L. -- Luca Berra -- bluca@comedia.it ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: dm-crypt over raid6 unreadable after crash 2011-07-08 5:40 ` Luca Berra @ 2011-07-10 12:03 ` Louis-David Mitterrand 2011-07-10 12:28 ` Phil Turmel 0 siblings, 1 reply; 13+ messages in thread From: Louis-David Mitterrand @ 2011-07-10 12:03 UTC (permalink / raw) To: linux-raid On Fri, Jul 08, 2011 at 07:40:45AM +0200, Luca Berra wrote: > >This is important. When I computed the sector count for the linear mapping, I just took 2048 off the end. You may want to select a sector count that aligns the endpoint. > but the xfs sb should be at sector 0 So what should I change to the dmsetup command to make it work? ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: dm-crypt over raid6 unreadable after crash 2011-07-10 12:03 ` Louis-David Mitterrand @ 2011-07-10 12:28 ` Phil Turmel 2011-07-10 23:44 ` Stan Hoeppner 2011-07-11 7:54 ` Louis-David Mitterrand 0 siblings, 2 replies; 13+ messages in thread From: Phil Turmel @ 2011-07-10 12:28 UTC (permalink / raw) To: linux-raid On 07/10/2011 08:03 AM, Louis-David Mitterrand wrote: > On Fri, Jul 08, 2011 at 07:40:45AM +0200, Luca Berra wrote: >>> This is important. When I computed the sector count for the linear mapping, I just took 2048 off the end. You may want to select a sector count that aligns the endpoint. >> but the xfs sb should be at sector 0 > > So what should I change to the dmsetup command to make it work? He means (not to put words in Luca's mouth...) that the precise endpoint of the mapped volume shouldn't matter to fsck.xfs. Changing device order in the raid is probably your only hope of recovery. The dmsetup exercise was a blind alley. Luca also pointed out that the problem array is named "grml" which means that it was created with grml, not your original system (zenon). That suggests that "mdadm --create" was used under grml, and the member devices were specified in an order differing from the original install. If that "mdadm --create" didn't include the "--assume-clean" option, then the parity blocks are almost certainly recomputed, and your data destroyed. Otherwise, you can try "mdadm --create --assume-clean" with other combinations of device order to try to find the "right" one. I recommend trying "mdadm --create --assume-clean" with the devices in the same order as shown by lsdrv for the zenon array. Phil ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: dm-crypt over raid6 unreadable after crash 2011-07-10 12:28 ` Phil Turmel @ 2011-07-10 23:44 ` Stan Hoeppner 2011-07-11 7:57 ` Louis-David Mitterrand 2011-07-11 7:54 ` Louis-David Mitterrand 1 sibling, 1 reply; 13+ messages in thread From: Stan Hoeppner @ 2011-07-10 23:44 UTC (permalink / raw) To: Phil Turmel; +Cc: linux-raid On 7/10/2011 7:28 AM, Phil Turmel wrote: > On 07/10/2011 08:03 AM, Louis-David Mitterrand wrote: >> On Fri, Jul 08, 2011 at 07:40:45AM +0200, Luca Berra wrote: >>>> This is important. When I computed the sector count for the linear mapping, I just took 2048 off the end. You may want to select a sector count that aligns the endpoint. >>> but the xfs sb should be at sector 0 >> >> So what should I change to the dmsetup command to make it work? > > He means (not to put words in Luca's mouth...) that the precise endpoint of the mapped volume shouldn't matter to fsck.xfs. Changing device order in the raid is probably your only hope of recovery. The dmsetup exercise was a blind alley. > > Luca also pointed out that the problem array is named "grml" which means that it was created with grml, not your original system (zenon). That suggests that "mdadm --create" was used under grml, and the member devices were specified in an order differing from the original install. If that "mdadm --create" didn't include the "--assume-clean" option, then the parity blocks are almost certainly recomputed, and your data destroyed. Otherwise, you can try "mdadm --create --assume-clean" with other combinations of device order to try to find the "right" one. > > I recommend trying "mdadm --create --assume-clean" with the devices in the same order as shown by lsdrv for the zenon array. Moral of this story: Using live CDs to troubleshoot Linux RAID arrays, LVM volumes, dmcrypt, etc, is tricky at best, fraught with disaster at worst. Always have a "system rescue" CD/DVD built from the particular machine in question at the ready. Most distros have a facility for building such a per machine boot/rescue CD. Such media will have the same kernel and module versions, the system's mdadm configuration, and all other system specific stuff needed to properly troubleshoot and repair. If such rescue media had been used the OP would have likely experienced far less grief, and might already had this problem fixed. -- Stan ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: dm-crypt over raid6 unreadable after crash 2011-07-10 23:44 ` Stan Hoeppner @ 2011-07-11 7:57 ` Louis-David Mitterrand 2011-07-11 11:49 ` Stan Hoeppner 0 siblings, 1 reply; 13+ messages in thread From: Louis-David Mitterrand @ 2011-07-11 7:57 UTC (permalink / raw) To: linux-raid On Sun, Jul 10, 2011 at 06:44:21PM -0500, Stan Hoeppner wrote: > > Moral of this story: Using live CDs to troubleshoot Linux RAID arrays, > LVM volumes, dmcrypt, etc, is tricky at best, fraught with disaster at > worst. Always have a "system rescue" CD/DVD built from the particular > machine in question at the ready. Most distros have a facility for > building such a per machine boot/rescue CD. Such media will have the > same kernel and module versions, the system's mdadm configuration, and > all other system specific stuff needed to properly troubleshoot and repair. > > If such rescue media had been used the OP would have likely experienced > far less grief, and might already had this problem fixed. A live CD like grml can properly auto-start a md array. My problem came from doing a --create instead of an --assemble at some point. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: dm-crypt over raid6 unreadable after crash 2011-07-11 7:57 ` Louis-David Mitterrand @ 2011-07-11 11:49 ` Stan Hoeppner 0 siblings, 0 replies; 13+ messages in thread From: Stan Hoeppner @ 2011-07-11 11:49 UTC (permalink / raw) To: linux-raid On 7/11/2011 2:57 AM, Louis-David Mitterrand wrote: > On Sun, Jul 10, 2011 at 06:44:21PM -0500, Stan Hoeppner wrote: >> >> Moral of this story: Using live CDs to troubleshoot Linux RAID arrays, >> LVM volumes, dmcrypt, etc, is tricky at best, fraught with disaster at >> worst. Always have a "system rescue" CD/DVD built from the particular >> machine in question at the ready. Most distros have a facility for >> building such a per machine boot/rescue CD. Such media will have the >> same kernel and module versions, the system's mdadm configuration, and >> all other system specific stuff needed to properly troubleshoot and repair. >> >> If such rescue media had been used the OP would have likely experienced >> far less grief, and might already had this problem fixed. > > A live CD like grml can properly auto-start a md array. My problem > came from doing a --create instead of an --assemble at some point. What did it not auto-start your md array in this case again? -- Stan ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: dm-crypt over raid6 unreadable after crash 2011-07-10 12:28 ` Phil Turmel 2011-07-10 23:44 ` Stan Hoeppner @ 2011-07-11 7:54 ` Louis-David Mitterrand 1 sibling, 0 replies; 13+ messages in thread From: Louis-David Mitterrand @ 2011-07-11 7:54 UTC (permalink / raw) To: linux-raid On Sun, Jul 10, 2011 at 08:28:27AM -0400, Phil Turmel wrote: > On 07/10/2011 08:03 AM, Louis-David Mitterrand wrote: > > On Fri, Jul 08, 2011 at 07:40:45AM +0200, Luca Berra wrote: > >>> This is important. When I computed the sector count for the linear mapping, I just took 2048 off the end. You may want to select a sector count that aligns the endpoint. > >> but the xfs sb should be at sector 0 > > > > So what should I change to the dmsetup command to make it work? > > He means (not to put words in Luca's mouth...) that the precise > endpoint of the mapped volume shouldn't matter to fsck.xfs. Changing > device order in the raid is probably your only hope of recovery. The > dmsetup exercise was a blind alley. > > Luca also pointed out that the problem array is named "grml" which > means that it was created with grml, not your original system (zenon). > That suggests that "mdadm --create" was used under grml, and the > member devices were specified in an order differing from the original > install. If that "mdadm --create" didn't include the "--assume-clean" > option, then the parity blocks are almost certainly recomputed, and > your data destroyed. Otherwise, you can try "mdadm --create > --assume-clean" with other combinations of device order to try to find > the "right" one. > > I recommend trying "mdadm --create --assume-clean" with the devices in > the same order as shown by lsdrv for the zenon array. That clinched the deal! By reordering the md1 array according to the working 'zenon' md2 array I was able to unlock it with cryptsetup. However the xfs filesystem was too damaged: an xfs_repair sent everything in lost+found. That lsdrv tool is really useful, I'll keep it my box. Thanks, ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2011-07-11 11:49 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-07-06 16:12 dm-crypt over raid6 unreadable after crash Louis-David Mitterrand 2011-07-06 17:00 ` Phil Turmel 2011-07-07 9:05 ` Louis-David Mitterrand 2011-07-07 12:41 ` Phil Turmel 2011-07-07 13:09 ` Louis-David Mitterrand 2011-07-07 13:32 ` Phil Turmel 2011-07-08 5:40 ` Luca Berra 2011-07-10 12:03 ` Louis-David Mitterrand 2011-07-10 12:28 ` Phil Turmel 2011-07-10 23:44 ` Stan Hoeppner 2011-07-11 7:57 ` Louis-David Mitterrand 2011-07-11 11:49 ` Stan Hoeppner 2011-07-11 7:54 ` Louis-David Mitterrand
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox