[dm-crypt] can't open luks device after raid 5 rebuild

[dm-crypt] can't open luks device after raid 5 rebuild

[artificial11000]

Hi guys,

after my encrypted RAID 5 was degraded and rebuilt, I can't open it anymore.

cryptsetup luksOpen /dev/md127 archiv
Geben Sie den Passsatz für /dev/md127 ein:
Kein Schlüssel mit diesem Passsatz verfügbar.

It says that the provided password doesn't match any of the stored keys.

Here's the output of luksDump:

cryptsetup luksDump /dev/md127
LUKS header information for /dev/md127

Version:       	1
Cipher name:   	aes
Cipher mode:   	xts-plain
Hash spec:     	sha1
Payload offset:	4040
MK bits:       	512
MK digest:     	53 24 e5 1d 68 3e f1 5f 96 56 06 23 85 4a b6 89 07 de 52 1d
MK salt:       	5b 2e 4b 48 d8 2f 91 52 90 79 3b 6e 73 13 52 f1
                	8f 9e 35 50 8e 25 9e 92 22 52 48 26 44 d7 bd 6c
MK iterations: 	10
UUID:          	3fbd3c47-8dc3-4541-8960-00c0a80c8ae9

Key Slot 0: ENABLED
	Iterations:         	363564
	Salt:               	32 db 41 40 56 fe 6f a9 c7 e4 c4 5f a0 4c 52 db
	                      	8e df 07 8c 9b 32 86 e6 ac 0c 8a e6 c4 69 31 0f
	Key material offset:	8
	AF stripes:            	4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED


Can anyone help me?

Regards
artificial

[dm-crypt] can't open luks device after raid 5 rebuild

[artificial11000]

Hi guys,

after my encrypted RAID 5 was degraded and rebuilt, I can't open it anymore.

cryptsetup luksOpen /dev/md127 archiv
Geben Sie den Passsatz für /dev/md127 ein:
Kein Schlüssel mit diesem Passsatz verfügbar.

It says that the provided password doesn't match any of the stored keys.

Here's the output of luksDump:

cryptsetup luksDump /dev/md127
LUKS header information for /dev/md127

Version:           1
Cipher name:       aes
Cipher mode:       xts-plain
Hash spec:         sha1
Payload offset:    4040
MK bits:           512
MK digest:         53 24 e5 1d 68 3e f1 5f 96 56 06 23 85 4a b6 89 07 de 
52 1d
MK salt:           5b 2e 4b 48 d8 2f 91 52 90 79 3b 6e 73 13 52 f1
                    8f 9e 35 50 8e 25 9e 92 22 52 48 26 44 d7 bd 6c
MK iterations:     10
UUID:              3fbd3c47-8dc3-4541-8960-00c0a80c8ae9

Key Slot 0: ENABLED
     Iterations:             363564
     Salt:                   32 db 41 40 56 fe 6f a9 c7 e4 c4 5f a0 4c 52 db
                               8e df 07 8c 9b 32 86 e6 ac 0c 8a e6 c4 69 
31 0f
     Key material offset:    8
     AF stripes:                4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED


Can anyone help me?

Regards
artificial

Re: [dm-crypt] can't open luks device after raid 5 rebuild

[Arno Wagner]

Hi,

a RAID degradation and successful rebuild does not have this 
effect. Did you do something else? Or did you do a manual (forced)
rebuild? What exactly did happen to the RAID and what did you 
do to fix it? 

The symptoms you see are typical for a damaged key-slot. The
header resides in a single sector and is not affected. 
If so, a header backup is possibly the  only thing that will 
help you, see the FAQ for details.

Arno

On Sun, Apr 08, 2012 at 03:04:47PM +0200, artificial11000 wrote:
> Hi guys,
> 
> after my encrypted RAID 5 was degraded and rebuilt, I can't open it anymore.
> 
> cryptsetup luksOpen /dev/md127 archiv
> Geben Sie den Passsatz f?r /dev/md127 ein:
> Kein Schl?ssel mit diesem Passsatz verf?gbar.
> 
> It says that the provided password doesn't match any of the stored keys.
> 
> Here's the output of luksDump:
> 
> cryptsetup luksDump /dev/md127
> LUKS header information for /dev/md127
> 
> Version:           1
> Cipher name:       aes
> Cipher mode:       xts-plain
> Hash spec:         sha1
> Payload offset:    4040
> MK bits:           512
> MK digest:         53 24 e5 1d 68 3e f1 5f 96 56 06 23 85 4a b6 89
> 07 de 52 1d
> MK salt:           5b 2e 4b 48 d8 2f 91 52 90 79 3b 6e 73 13 52 f1
>                    8f 9e 35 50 8e 25 9e 92 22 52 48 26 44 d7 bd 6c
> MK iterations:     10
> UUID:              3fbd3c47-8dc3-4541-8960-00c0a80c8ae9
> 
> Key Slot 0: ENABLED
>     Iterations:             363564
>     Salt:                   32 db 41 40 56 fe 6f a9 c7 e4 c4 5f a0 4c 52 db
>                               8e df 07 8c 9b 32 86 e6 ac 0c 8a e6 c4
> 69 31 0f
>     Key material offset:    8
>     AF stripes:                4000
> Key Slot 1: DISABLED
> Key Slot 2: DISABLED
> Key Slot 3: DISABLED
> Key Slot 4: DISABLED
> Key Slot 5: DISABLED
> Key Slot 6: DISABLED
> Key Slot 7: DISABLED
> 
> 
> Can anyone help me?
> 
> Regards
> artificial
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
One of the painful things about our time is that those who feel certainty 
are stupid, and those with any imagination and understanding are filled 
with doubt and indecision. -- Bertrand Russell 

Re: [dm-crypt] can't open luks device after raid 5 rebuild

[artificial11000]

Hi,

as I noticed that I can't access the device, I recreated the raid manually.
  mdadm --create --level=5 -n 3 /dev/md127 /dev/sdb /dev/sdc /dev/sdd

Could this damage the key-slot?
Are the metadata of the RAID relevant to this?
I read that old versions of mdadm created RAIDs with metadata=0.90 and 
the newer versions with metadata=1.20.

artificial

On 04/08/2012 05:47 PM, Arno Wagner wrote:
> Hi,
>
> a RAID degradation and successful rebuild does not have this
> effect. Did you do something else? Or did you do a manual (forced)
> rebuild? What exactly did happen to the RAID and what did you
> do to fix it?
>
> The symptoms you see are typical for a damaged key-slot. The
> header resides in a single sector and is not affected.
> If so, a header backup is possibly the  only thing that will
> help you, see the FAQ for details.
>
> Arno
>
> On Sun, Apr 08, 2012 at 03:04:47PM +0200, artificial11000 wrote:
>> Hi guys,
>>
>> after my encrypted RAID 5 was degraded and rebuilt, I can't open it anymore.
>>
>> cryptsetup luksOpen /dev/md127 archiv
>> Geben Sie den Passsatz f?r /dev/md127 ein:
>> Kein Schl?ssel mit diesem Passsatz verf?gbar.
>>
>> It says that the provided password doesn't match any of the stored keys.
>>
>> Here's the output of luksDump:
>>
>> cryptsetup luksDump /dev/md127
>> LUKS header information for /dev/md127
>>
>> Version:           1
>> Cipher name:       aes
>> Cipher mode:       xts-plain
>> Hash spec:         sha1
>> Payload offset:    4040
>> MK bits:           512
>> MK digest:         53 24 e5 1d 68 3e f1 5f 96 56 06 23 85 4a b6 89
>> 07 de 52 1d
>> MK salt:           5b 2e 4b 48 d8 2f 91 52 90 79 3b 6e 73 13 52 f1
>>                     8f 9e 35 50 8e 25 9e 92 22 52 48 26 44 d7 bd 6c
>> MK iterations:     10
>> UUID:              3fbd3c47-8dc3-4541-8960-00c0a80c8ae9
>>
>> Key Slot 0: ENABLED
>>      Iterations:             363564
>>      Salt:                   32 db 41 40 56 fe 6f a9 c7 e4 c4 5f a0 4c 52 db
>>                                8e df 07 8c 9b 32 86 e6 ac 0c 8a e6 c4
>> 69 31 0f
>>      Key material offset:    8
>>      AF stripes:                4000
>> Key Slot 1: DISABLED
>> Key Slot 2: DISABLED
>> Key Slot 3: DISABLED
>> Key Slot 4: DISABLED
>> Key Slot 5: DISABLED
>> Key Slot 6: DISABLED
>> Key Slot 7: DISABLED
>>
>>
>> Can anyone help me?
>>
>> Regards
>> artificial
>> _______________________________________________
>> dm-crypt mailing list
>> dm-crypt@saout.de
>> http://www.saout.de/mailman/listinfo/dm-crypt
>>
>

Re: [dm-crypt] can't open luks device after raid 5 rebuild

[Ingo Schmitt]

Hi Artificial,

On 04/08/12 18:23, artificial11000 wrote:

> as I noticed that I can't access the device, I recreated the raid manually.
>  mdadm --create --level=5 -n 3 /dev/md127 /dev/sdb /dev/sdc /dev/sdd

You created a new raid that way.

> Could this damage the key-slot?

Yes and all the data on the drives too.

> Are the metadata of the RAID relevant to this?
> I read that old versions of mdadm created RAIDs with metadata=0.90 and
> the newer versions with metadata=1.20.
> 
> artificial
> 
> On 04/08/2012 05:47 PM, Arno Wagner wrote:
>> Hi,
>>
>> a RAID degradation and successful rebuild does not have this
>> effect. Did you do something else? Or did you do a manual (forced)
>> rebuild? What exactly did happen to the RAID and what did you
>> do to fix it?
>>
>> The symptoms you see are typical for a damaged key-slot. The
>> header resides in a single sector and is not affected.
>> If so, a header backup is possibly the  only thing that will
>> help you, see the FAQ for details.
>>
>> Arno
>>
>> On Sun, Apr 08, 2012 at 03:04:47PM +0200, artificial11000 wrote:
>>> Hi guys,
>>>
>>> after my encrypted RAID 5 was degraded and rebuilt, I can't open it
>>> anymore.
>>>
>>> cryptsetup luksOpen /dev/md127 archiv
>>> Geben Sie den Passsatz f?r /dev/md127 ein:
>>> Kein Schl?ssel mit diesem Passsatz verf?gbar.
>>>
>>> It says that the provided password doesn't match any of the stored keys.
>>>
>>> Here's the output of luksDump:
>>>
>>> cryptsetup luksDump /dev/md127
>>> LUKS header information for /dev/md127
>>>
>>> Version:           1
>>> Cipher name:       aes
>>> Cipher mode:       xts-plain
>>> Hash spec:         sha1
>>> Payload offset:    4040
>>> MK bits:           512
>>> MK digest:         53 24 e5 1d 68 3e f1 5f 96 56 06 23 85 4a b6 89
>>> 07 de 52 1d
>>> MK salt:           5b 2e 4b 48 d8 2f 91 52 90 79 3b 6e 73 13 52 f1
>>>                     8f 9e 35 50 8e 25 9e 92 22 52 48 26 44 d7 bd 6c
>>> MK iterations:     10
>>> UUID:              3fbd3c47-8dc3-4541-8960-00c0a80c8ae9
>>>
>>> Key Slot 0: ENABLED
>>>      Iterations:             363564
>>>      Salt:                   32 db 41 40 56 fe 6f a9 c7 e4 c4 5f a0
>>> 4c 52 db
>>>                                8e df 07 8c 9b 32 86 e6 ac 0c 8a e6 c4
>>> 69 31 0f
>>>      Key material offset:    8
>>>      AF stripes:                4000
>>> Key Slot 1: DISABLED
>>> Key Slot 2: DISABLED
>>> Key Slot 3: DISABLED
>>> Key Slot 4: DISABLED
>>> Key Slot 5: DISABLED
>>> Key Slot 6: DISABLED
>>> Key Slot 7: DISABLED
>>>
>>>
>>> Can anyone help me?
>>>
>>> Regards
>>> artificial
>>> _______________________________________________
>>> dm-crypt mailing list
>>> dm-crypt@saout.de
>>> http://www.saout.de/mailman/listinfo/dm-crypt
>>>
>>
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] can't open luks device after raid 5 rebuild

[Heinz Diehl]

On 08.04.2012, artificial11000 wrote: 

>  mdadm --create --level=5 -n 3 /dev/md127 /dev/sdb /dev/sdc /dev/sdd
         ---------

Seems your data is gone, forever. You created a new raid manually,
which means that you deleted all your previous data. 

You have a backup, do you?

Re: [dm-crypt] can't open luks device after raid 5 rebuild

[Salatiel Filho]

On Sun, Apr 8, 2012 at 14:09, Heinz Diehl <htd@fancy-poultry.org> wrote:
> On 08.04.2012, artificial11000 wrote:
>
>>  mdadm --create --level=5 -n 3 /dev/md127 /dev/sdb /dev/sdc /dev/sdd
>         ---------
>
> Seems your data is gone, forever. You created a new raid manually,
> which means that you deleted all your previous data.
>
> You have a backup, do you?
>
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

I disagree. I got some missing superblock sometime in the past while
assembling the array. Recreated it using mdadm -C with the same
parameters for the raid-level and raid-devices and it did not touch my
encrypted data. I was able to mount just fine after that. I always
thought that mdadm would touch only their own superblock when
creating. Though at that time i had a degraded  array and my array is
built on partitions , not the whole disk.


[]'s
Salatiel

Re: [dm-crypt] can't open luks device after raid 5 rebuild

[Arno Wagner]

On Sun, Apr 08, 2012 at 05:07:28PM -0300, Salatiel Filho wrote:
> On Sun, Apr 8, 2012 at 14:09, Heinz Diehl <htd@fancy-poultry.org> wrote:
> > On 08.04.2012, artificial11000 wrote:
> >
> >> ?mdadm --create --level=5 -n 3 /dev/md127 /dev/sdb /dev/sdc /dev/sdd
> > ? ? ? ? ---------
> >
> > Seems your data is gone, forever. You created a new raid manually,
> > which means that you deleted all your previous data.
> >
> > You have a backup, do you?
> >
> > _______________________________________________
> > dm-crypt mailing list
> > dm-crypt@saout.de
> > http://www.saout.de/mailman/listinfo/dm-crypt
> 
> I disagree. I got some missing superblock sometime in the past while
> assembling the array. Recreated it using mdadm -C with the same
> parameters for the raid-level and raid-devices and it did not touch my
> encrypted data. I was able to mount just fine after that. I always
> thought that mdadm would touch only their own superblock when
> creating. 

No, it recreates the parity stripes. It has to, otherwise RAID consistency
checks would fail. If, for any reason, drive/device 
order or stripe size is diferent, this will scramble your data. 
The data is still there unless you write to the array, see my other 
answer. It also overwrites the RAID superblocks.

If everything is the same, the original data will be there though,
because the new parity stripes are exactly at the same positions 
on the same disks as the old ones and created from the same 
non-parity stripes.

> Though at that time i had a degraded  array and my array is
> built on partitions , not the whole disk.

On a degraded array, you should always only (re-)add the missing 
device or a replacement.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
One of the painful things about our time is that those who feel certainty 
are stupid, and those with any imagination and understanding are filled 
with doubt and indecision. -- Bertrand Russell 

Re: [dm-crypt] can't open luks device after raid 5 rebuild

[Arno Wagner]

On Sun, Apr 08, 2012 at 07:09:30PM +0200, Heinz Diehl wrote:
> On 08.04.2012, artificial11000 wrote: 
> 
> >  mdadm --create --level=5 -n 3 /dev/md127 /dev/sdb /dev/sdc /dev/sdd
>          ---------
> 
> Seems your data is gone, forever. You created a new raid manually,
> which means that you deleted all your previous data. 

Not quite. The LUKS header is still there, obviously.
What happens when you create a RAID5 array out of components
of another RAID5 array is the following: They get put
into some order and, in a fashion rotating by stripe-size,
one drive gets overwritten with the parity of the others.
If everything, includiong the drive order is the same
as in the original artray, everything will be fine.

Otherwise, if the stripe size is correct, you will not actually
lose any data, as only one drive was overwritten per stripe
and the original RAID5 data is tolerant against one device loss
(or loss of one stripe per stripe-set). The array still assembles 
incorrectly and you will _not_ be able to re-create the original 
data without major effort. It should be possible though, but
it involves figuring out the current and the old stripe order
and then manually rebuilding the old array out of the old stripes
still there. This is made trickier because the old RAID
superblocks are also gone.

I think could do this, but I would expect several days of coding and
I would need the password (otherwise it is impossible to verify 
correct stripe order and size) and complete drive images of all drives. 
(And, sorry, my rates are likely way above what the data is 
worth to you...) Some RAID recovery software may also be able
to help, but as the data is encrypted, I doubt it, as with encrypted
data, it is not possible to detect whether the assembly order
is right. This would require a specialized LUKS-aware RAID recovery
tool.

> You have a backup, do you?

That would be the next question. The proble is not (only) that
you hosed the LUKS header, but the complete data area.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
One of the painful things about our time is that those who feel certainty 
are stupid, and those with any imagination and understanding are filled 
with doubt and indecision. -- Bertrand Russell 

Re: [dm-crypt] can't open luks device after raid 5 rebuild

[Arno Wagner]

On Sun, Apr 08, 2012 at 06:23:22PM +0200, artificial11000 wrote:
> Hi,
> 
> as I noticed that I can't access the device, I recreated the raid manually.
>  mdadm --create --level=5 -n 3 /dev/md127 /dev/sdb /dev/sdc /dev/sdd
> 
> Could this damage the key-slot?
> Are the metadata of the RAID relevant to this?
> I read that old versions of mdadm created RAIDs with metadata=0.90
> and the newer versions with metadata=1.20.

This may be an additional problem or the only problem. Metadata 0.90 
is at the end of the device. Metadata 1.20 is 4k from the start of 
the device. The other versions are in still other places.
(I have no idea what messed up process created these thorougly
insane changes.)

So, yes, 4k offset is right in the first key-stripe. If so,
then your data is pemanently gone unless you have a header-backup.
In this case it is also possible that the array actually assembled
right and the problem is _only_ the new raid metadata block.

Arno

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
One of the painful things about our time is that those who feel certainty 
are stupid, and those with any imagination and understanding are filled 
with doubt and indecision. -- Bertrand Russell 

Re: [dm-crypt] can't open luks device after raid 5 rebuild

[artificial11000]

Hi guys,

thanks for your effort.
Meanwhile I had to start over and format the whole device.
I have a backup. Allthough it is not really up to date, I can work with it.

Thanks again

artificial

On 04/08/2012 10:35 PM, Arno Wagner wrote:
> On Sun, Apr 08, 2012 at 06:23:22PM +0200, artificial11000 wrote:
>> Hi,
>>
>> as I noticed that I can't access the device, I recreated the raid manually.
>>   mdadm --create --level=5 -n 3 /dev/md127 /dev/sdb /dev/sdc /dev/sdd
>>
>> Could this damage the key-slot?
>> Are the metadata of the RAID relevant to this?
>> I read that old versions of mdadm created RAIDs with metadata=0.90
>> and the newer versions with metadata=1.20.
>
> This may be an additional problem or the only problem. Metadata 0.90
> is at the end of the device. Metadata 1.20 is 4k from the start of
> the device. The other versions are in still other places.
> (I have no idea what messed up process created these thorougly
> insane changes.)
>
> So, yes, 4k offset is right in the first key-stripe. If so,
> then your data is pemanently gone unless you have a header-backup.
> In this case it is also possible that the array actually assembled
> right and the problem is _only_ the new raid metadata block.
>
> Arno
>

Re: [dm-crypt] can't open luks device after raid 5 rebuild

[Arno Wagner]

On Sun, Apr 08, 2012 at 10:43:34PM +0200, artificial11000 wrote:
> Hi guys,
> 
> thanks for your effort.

No problem.

> Meanwhile I had to start over and format the whole device.
> I have a backup. Allthough it is not really up to date, I can work with it.

Good. So you already did something very right.

Arno 




> Thanks again
> 
> artificial
> 
> On 04/08/2012 10:35 PM, Arno Wagner wrote:
> >On Sun, Apr 08, 2012 at 06:23:22PM +0200, artificial11000 wrote:
> >>Hi,
> >>
> >>as I noticed that I can't access the device, I recreated the raid manually.
> >>  mdadm --create --level=5 -n 3 /dev/md127 /dev/sdb /dev/sdc /dev/sdd
> >>
> >>Could this damage the key-slot?
> >>Are the metadata of the RAID relevant to this?
> >>I read that old versions of mdadm created RAIDs with metadata=0.90
> >>and the newer versions with metadata=1.20.
> >
> >This may be an additional problem or the only problem. Metadata 0.90
> >is at the end of the device. Metadata 1.20 is 4k from the start of
> >the device. The other versions are in still other places.
> >(I have no idea what messed up process created these thorougly
> >insane changes.)
> >
> >So, yes, 4k offset is right in the first key-stripe. If so,
> >then your data is pemanently gone unless you have a header-backup.
> >In this case it is also possible that the array actually assembled
> >right and the problem is _only_ the new raid metadata block.
> >
> >Arno
> >
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
One of the painful things about our time is that those who feel certainty 
are stupid, and those with any imagination and understanding are filled 
with doubt and indecision. -- Bertrand Russell