[dm-crypt] dm-crypt hanging on attempts to manipulate an encrypted device

[dm-crypt] dm-crypt hanging on attempts to manipulate an encrypted device

[Pedro Fernandes Macedo]

[-- Attachment #1: Type: text/plain, Size: 3219 bytes --]

I have a 1TB USB disk that I use for backups and has recently hit a snag.
 The encrypted device was working fine, but it now is failing on a weird
way: after plugging it in yesterday and doing a luksOpen, the "automatic
header conversion from 0.99 to 0.991 triggered" message appeared. After
that, every single attempt at opening the device or add keys hangs after the
key is entered. This is a device that was in use all the time and was
working until I unplugged it (for safety) for some changes to my raid
setup.

I've now removed the hard drive from its enclosure and put it directly on
the server, made an image and I am testing now using a loop device and the
image. Here's the interesting bits:

# cryptsetup luksDump /dev/loop1
LUKS header information for /dev/loop1

Version:        1
Cipher name:    aes
Cipher mode:    cbc-essiv:sha256
Hash spec:      sha1
Payload offset: 1032
MK bits:        128
MK digest:      b5 00 fe b8 fc a8 4a 21 9a 89 a2 cf 83 ac 0c de 93 50 cc 32
MK salt:        00 72 37 b0 f1 a1 55 e5 ba 0c 2b 0e 83 cc 54 1d
                8f 43 f5 12 9b 57 05 83 e0 7f 2a d6 3d df 1d d0
MK iterations:  0
UUID:           48d0bd7e-ef95-4e47-bda0-80c5d872087d

Key Slot 0: ENABLED
        Iterations:             148711
        Salt:                   34 eb 8d df 49 8e 5f 24 1c 1f 27 77 e9 fd 3d
b1
                                04 37 ea 66 81 77 8b 96 62 fc 3f a2 c8 c9 c3
87
        Key material offset:    8
        AF stripes:             4000
Key Slot 1: ENABLED
        Iterations:             183406
        Salt:                   73 23 9f 40 5a ea ae 50 7b 6b 59 77 e1 a3 f8
11
                                79 09 eb ce a3 8e e9 05 15 5c 16 bf 18 5a 83
07
        Key material offset:    136
        AF stripes:             4000
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED


Doing an strace of cryptsetup luksOpen:
open("/proc/misc", O_RDONLY)            = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x2aea59b33000
read(3, " 61 autofs\n200 tun\n220 mptctl\n 6"..., 4096) = 86
close(3)                                = 0
munmap(0x2aea59b33000, 4096)            = 0
stat("/dev/mapper/control", {st_mode=S_IFCHR|0600, st_rdev=makedev(10, 62),
...}) = 0
open("/dev/mapper/control", O_RDWR)     = 3
ioctl(3, DM_DEV_REMOVE, 0x6ecdd70)      = 0
close(3)                                = 0
stat("/dev/mapper/temporary-cryptsetup-29295", {st_mode=S_IFBLK|0660,
st_rdev=makedev(253, 0), ...}) = 0
unlink("/dev/mapper/temporary-cryptsetup-29295") = 0

And it hangs there forever, using a whole core for something (that I have
been unable to find yet). Same for AddKey.

This is on Centos 5.4 64 bit, cryptsetup-luks-1.0.3-5.el5 (and no custom
packages; all from the original repository).

Sadly, I dont have backups of the luks header that I can restore. Any ideas
on how to open the device? I am certain both keys are valid (the one in slot
1 is a file I used for scripted mounts of the device and the one in slot 0
is a passphrase that I already tried copy-pasting to avoid any typos and
keymap issues).

--
Pedro Macedo

[-- Attachment #2: Type: text/html, Size: 4002 bytes --]

Re: [dm-crypt] dm-crypt hanging on attempts to manipulate an encrypted device

[Milan Broz]

On 04/18/2010 01:17 PM, Pedro Fernandes Macedo wrote:
> I have a 1TB USB disk that I use for backups and has recently hit a
> snag.  The encrypted device was working fine, but it now is failing on a
> weird way: after plugging it in yesterday and doing a luksOpen,
> the "automatic header conversion from 0.99 to 0.991 triggered" message
> appeared. After that, every single attempt at opening the device or add
> keys hangs after the key is entered. This is a device that was in use
> all the time and was working until I unplugged it (for safety) for some
> changes to my raid setup. 

The automatic upgrade of version indicates that header was created
with old version of cryptsetup...

Do you remember which cryptsetup was used to format device?
(Or version of distro release, if it is CentOS, I'll check it.)

> MK iterations:  0

This is wrong and something very bad happened with the header.
Anyway, if the keyslots are ok, it can be probably still fixed easily.

> This is on Centos 5.4 64 bit, cryptsetup-luks-1.0.3-5.el5 (and no custom
> packages; all from the original repository).

Please can you report this to upstream bugzilla (bugzilla.redhat.com,
cryptsetup-luks component)?

I'll try to help you there, if the code is misinterpreting some old metadata
it must be fixed both upstream and in RHEL (and I can handle both together
if problem is understood).

But I will need binary copy of header
dd if=/dev/<your_luks_device> of=/backup.img bs=512 count=1032

(If you agree, please submit that bug for RHEL5.4 and send me
the header - best in private mail, or encrypted using my gpg key,
id is 0xBB21822F on keyservers)

(and probably I will need your help to test passphrase later)

Milan
--
mbroz@redhat.com

Re: [dm-crypt] dm-crypt hanging on attempts to manipulate an encrypted device

[Pedro Fernandes Macedo]

[-- Attachment #1: Type: text/plain, Size: 2165 bytes --]

On Sun, Apr 18, 2010 at 3:13 PM, Milan Broz <mbroz@redhat.com> wrote:

> On 04/18/2010 01:17 PM, Pedro Fernandes Macedo wrote:
> > I have a 1TB USB disk that I use for backups and has recently hit a
> > snag.  The encrypted device was working fine, but it now is failing on a
> > weird way: after plugging it in yesterday and doing a luksOpen,
> > the "automatic header conversion from 0.99 to 0.991 triggered" message
> > appeared. After that, every single attempt at opening the device or add
> > keys hangs after the key is entered. This is a device that was in use
> > all the time and was working until I unplugged it (for safety) for some
> > changes to my raid setup.
>
> The automatic upgrade of version indicates that header was created
> with old version of cryptsetup...
>
> Do you remember which cryptsetup was used to format device?
>

I'm not sure. I encrypted it for my move about one year ago, using whatever
version of Fedora was the stable at the time.  Then after I moved, I always
used Centos 5.


> (Or version of distro release, if it is CentOS, I'll check it.)
>
> > MK iterations:  0
>
> This is wrong and something very bad happened with the header.
> Anyway, if the keyslots are ok, it can be probably still fixed easily.
>
> > This is on Centos 5.4 64 bit, cryptsetup-luks-1.0.3-5.el5 (and no custom
> > packages; all from the original repository).
>
> Please can you report this to upstream bugzilla (bugzilla.redhat.com,
> cryptsetup-luks component)?
>

https://bugzilla.redhat.com/show_bug.cgi?id=583431 opened.

>
> I'll try to help you there, if the code is misinterpreting some old
> metadata
> it must be fixed both upstream and in RHEL (and I can handle both together
> if problem is understood).
>
> But I will need binary copy of header
> dd if=/dev/<your_luks_device> of=/backup.img bs=512 count=1032
>
> (If you agree, please submit that bug for RHEL5.4 and send me
> the header - best in private mail, or encrypted using my gpg key,
> id is 0xBB21822F on keyservers)
>
>
I'll send you a separate private mail with the key.


> (and probably I will need your help to test passphrase later)
>
> Milan
> --
> mbroz@redhat.com
>

[-- Attachment #2: Type: text/html, Size: 3270 bytes --]

Re: [dm-crypt] dm-crypt hanging on attempts to manipulate an encrypted device

[Milan Broz]

On 04/18/2010 03:13 PM, Milan Broz wrote:
> On 04/18/2010 01:17 PM, Pedro Fernandes Macedo wrote:
>> I have a 1TB USB disk that I use for backups and has recently hit a
>> snag.  The encrypted device was working fine, but it now is failing on a
>> weird way: after plugging it in yesterday and doing a luksOpen,
>> the "automatic header conversion from 0.99 to 0.991 triggered" message
>> appeared. After that, every single attempt at opening the device or add
>> keys hangs after the key is entered. This is a device that was in use
>> all the time and was working until I unplugged it (for safety) for some
>> changes to my raid setup. 
> 
> The automatic upgrade of version indicates that header was created
> with old version of cryptsetup...

For the archive:

seems that it was really unexpected result of automatic update.
("update" is basically conversion of iteration count into network byte order
- big/little endian conversion + some other magic.)

Because master key header digest iteration was always 10 in that problematic
version of cryptsetup, where device was originally formatted, recovery was quite easy.

I removed this automatic update in cryptsetup 1.1.0 (it was not safe as this report
proved), but because it seems that still there is possibility that old metadata
version exists, I'll add some workaround to luksHeaderRestore command.

So in future, general procedure to "update & fix" incompabilities is to perform

crypsetup luksHeaderBackup --header-backup-file <file> <device>
crypsetup luksHeaderRestore --header-backup-file <file> <device>

(currently it wipes possible FS signatures, which can be between header
and keyslots - also known problem with previous metadata handler.)

Milan